[kv] Implement insertIfNotExists on tablet server

[beryllw]

Purpose

Linked issue: close #2482

Brief change log

Tests

API and Format

Documentation

[beryllw]

Will add KeyDecoders for Paimon and Iceberg later.

[platinumhamburg]

LGTM, Just a few minor comments. Overall, this is the implementation with the minimal changes for the current update. Of course, it's not yet performance-optimal—it introduces a notable performance inefficiency: during the PutKV process, the original key is encoded and then immediately decoded again. This could be addressed in a future optimization; I suggest adding a TODO comment for it.

[platinumhamburg]

There is a critical bug that needs to be fixed

[platinumhamburg]

I’ve thought about it carefully, and I think KeyRecordBatch is too specific—especially since it’s meant for a temporary solution we all agree isn’t final. We could just remove it entirely and use a simpler approach instead. But if we’re short on time, it’s not a big deal—it doesn’t add much code—so we can keep it for now and clean it up later.

[wuchong]

@beryllw I left some comments. Especially, I added a concurrent test that reveals the correctness issue and should be fixed.

[wuchong]

Could you add an additional test for concurrent lookups with insert-if-not-exists and auto-increment enabled?

The test should:

• Spawn multiple threads (e.g., 3) that concurrently perform lookupWithInsert operations on the same set of keys (e.g., 100, 200, 300).
• Verify that all threads receive consistent lookup results (i.e., the same inserted values for each key).
• Confirm that exactly 3 changelog entries are written to the log tablet—one per unique key.

This will validate that concurrent put-and-relookup operations behave correctly under contention, ensuring idempotency and consistency when auto-increment and conditional inserts are enabled.

[wuchong]

How much effort would it take to support this? I believe it’s essential—since most clusters will enable lakehouse integration, this will become a common and critical path.

If this requires significant work, please create a blocker issue to track it. And please add tests for this case as well.

[wuchong]

This is incorrect. We should use partial updates to modify only the primary key fields. Otherwise, non-primary-key columns—including auto-increment fields—will be overwritten with null. This issue manifests when multiple threads concurrently call lookupAndInsert on the same keys.

To reproduce this, I’ve added a test: testConcurrentLookupWithInsertIfNotExistsAutoIncrement.

Additionally, we should enforce a validation in PutRequest: when performing a put operation, auto-increment fields must be excluded from the target columns. This ensures that auto-incremented values are never accidentally overwritten during updates. That means INSERT INTO t (id, auto_inc) VALUES ... or INSERT INTO t VALUES ... should fail with explicit error message that the auto increment fields can't be updated.

[xx789633]

Could you please also add a comment here to indicate that there might be a data race and describe how you solve it? The same key might be inserted by another thread after the lookup check passes.