fix(namespace): add primary key to manifest table and fix race condition

[jackye1995]

Summary

• Add unenforced primary key on object_id field in manifest schema to enable bloom filter conflict detection during concurrent MergeInsert operations
• Fix race condition in create_or_get_manifest where multiple concurrent namespace instances could fail when trying to create the manifest table by adding retry logic when manifest creation fails with "already exists" error
• Disable MergeInsert retry (retry_timeout = Duration::ZERO) to return concurrent modification errors immediately instead of silently retrying
• Use proper NamespaceError::ConcurrentModification error type for concurrent modification conflicts
• Move S3 integration tests from DirectoryNamespaceS3Test.java to NamespaceIntegrationTest.java using existing LocalStack configuration

Test plan

• Added test_manifest_schema_has_primary_key to verify Arrow schema metadata and Lance dataset recognition of primary key
• Added test_concurrent_create_drop_different_tables to test concurrent table operations with single manifest
• S3 integration tests moved and consolidated in NamespaceIntegrationTest.java
• All 46 manifest tests pass
• Java code compiles successfully

🤖 Generated with Claude Code

[github-actions]

Code Review

Summary

This PR adds unenforced primary key support to the manifest table for bloom filter conflict detection during concurrent MergeInsert operations, and fixes a race condition in manifest table creation.

P1 Issues

1. Retry logic in describe_table may mask actual errors (lines 1410-1037 in diff)

   The retry loop with exponential backoff waits up to ~1.6s for table visibility, but if the table genuinely doesn't exist, this adds unnecessary latency. The code should differentiate between "table not yet visible" (eventual consistency) vs "table doesn't exist". Currently, this will always delay non-existent table lookups.

2. Missing reload after ensure_manifest_primary_key modifies the dataset (lines 942-992 in diff)

   The ensure_manifest_primary_key function calls update_field_metadata().await which commits a new version, but the dataset_guard still holds the old version. The caller continues using the wrapper without reloading, which could cause version conflicts on subsequent operations. Consider calling checkout_latest() after the metadata update.

3. Direct access to internal wrapper field self.manifest_dataset.0 (line 1027 in diff)

   In describe_table, the code directly accesses .0.write().await on the DatasetConsistencyWrapper, bypassing encapsulation. This should use the wrapper's public methods (get_mut or reload) for consistency and maintainability.

Minor Observations

• The delete_from_manifest change from get_mut() to get() + clone + DeleteBuilder is a reasonable approach for proper conflict handling with the new primary key semantics.

• Good use of double-check locking pattern in ensure_manifest_primary_key.

• Test coverage for concurrent scenarios is comprehensive.