[JingsongLi]

Purpose

PK compaction previously rejected source-backed index manifests or rewrote data without maintaining declared indexes. DE automatic indexing rebuilt a growing tail shard on each small append, and the two table modes used separate configuration namespaces. Share index configuration across PK and append/DE tables, maintain PK indexes with compacted data, and build DE additions independently until a shared UniversalCompaction policy selects adjacent runs to rebuild.

Changes

• Share CoreOptions declaration and field-option resolution across PK and append/DE construction, maintenance, scans, indexed/raw searches and refinement, following Java PR #10914 at merged commit b4342332 (shared resolver, PK payload selection, and lossless Native option forwarding); the resolver and Native build contract also match current upstream master. Use global-index.<type>.index-columns and fields.<column>.index-options, with short field JSON keys such as nlist/dimension and the existing type-level prefixes such as ivf-pq.dimension; support the DLF singular index-column alias, implementation parameters and flat vector field parameters. Retain legacy PK declarations/options and the former Rust family-style vector configuration. Shared JSON replaces legacy JSON and overrides defaults; non-metric legacy conflicts remain errors. Preserve Java JSON insertion order. Resolve vector aliases as one parameter after merging JSON, following the updated Java VectorIndexOptions precedence and optional reader metric default. Validate global declarations when their index path is used without adding generic schema-level index validation; retain existing Rust PK schema prerequisites.
• Remove the legacy PK vector type/metric accessors and rename the remaining internal layout/scalar adapters without a primary_key prefix. Public callers use the shared global_index_* / vector_index_* API. Resolve vector type, options and metric together in one serializable VectorIndexOptions result; remove independent configuration type lookups. Retain the complete old vector configuration (pk-vector.index.columns plus that column's fields.<column>.pk-vector.index.type); an orphan old field type no longer selects an implementation when Rust parses table/query options. Canonical declarations still take precedence. Logical VectorType stays public; query implementations are configured through ordinary declarations/options.
• Native single, batch and hybrid searches use ordinary schema/query/route options, following upstream native-index-options.md. Remove all separate vector type setters and builder/per-query hint state from Rust, Python bindings and stubs. Preserve original field JSON, canonical/legacy/DLF declarations and opaque parameters through captured readers/routes and worker tasks. PK scans select one algorithm from valid payloads (including empty markers), discard stale sources and reject mixed valid formats; configuration is used only with no valid metadata. Keep typed RawVectorSearchSplit.index_type and persisted/index-build API metadata. Query defaults remain distinct from explicit metric constraints; PK raw and indexed reads keep Java table-metric semantics. Untyped DE FULL/DETAIL reads resolve an explicit metric from shared field JSON without selecting a backend; an explicit query metric overrides table JSON and does not parse unused invalid defaults.
• Accept full raw build option maps in Rust: parse per-field JSON before native construction, keep unchanged echoed vector schema entries in the table layer, preserve genuine procedure-over-table precedence without false legacy-JSON conflicts or injected metric defaults, keep qualified procedure typo validation, and retain original options/schema in serialized shards. Static root native aliases remain ignored like Java; standalone native builder parameters remain supported. DiskANN ignores the schema granule switch without rejecting a full schema map. Shared normalization handles JSON-encoded full-text lists including columns/text_fields aliases. Build planning respects captured historical or explicitly empty read views; option overrides preserve those views.
• Remove query implementation option aliases index_type, index-type, vector.index-type and fields.<column>.index-type; stop using an option-map bridge for raw source types. Native training index.type and typed metadata/transport fields remain supported.
• Keep table-mode validation and routing separate from shared configuration. Build one PK payload per partition, bucket, positive level and column from the complete compacted source set, preserving physical NULL/tombstone positions. Publish index additions/retirements atomically with compacted data, changelogs and deletion vectors. Shared PK declarations enter compaction maintenance without triggering a second DE index round.
• Resolve scalar read options per column, including fallback limits, BTree caches and FM read parameters. Bootstrap readers after field overrides so an overridden invalid default does not fail prematurely. Share equal-budget caches across columns and shards. Pass vector query widths through their native parameter names while preserving query-over-table precedence; honor DLF field shard-size overrides in planning and building.
• Share native scalar/composite writers, full-text archives, vector spill/training/serialization, output placement and failure cleanup through table/index_build. Share PK source/level planning and output validation between compaction and commit. Compact, Direct and Overwrite index deltas use one manifest merge path with layout-specific source validation; PK level-local positions remain distinct from global row IDs.
• Project only the indexed PK field. Apply captured and newly discovered lookup deletions before vector schema casts/native validation, then fill ordinal gaps with bounded NULL batches. Scalar and full-text builders retain Java's unfiltered physical-row behavior.
• Share the UniversalCompaction selector between PK data compaction and DE index maintenance. PK keeps its existing byte sizes, ordering, forced L0 rules and output levels. DE uses logical row counts and only merges adjacent ranges in the same partition, bucket, shard and connected row-ID group. Large base runs can remain untouched while small tail runs merge.
• Add independent index policies: global-index.num-sorted-run.compaction-trigger=5, global-index.compaction.size-ratio=1, and global-index.compaction.max-size-amplification-percent=200. Fully qualified keys in a column's index-options JSON override the table policy. Data-compaction settings are independent.
• Retain compatible DE payloads across physical data compaction. Retire overlapping/stale ranges, changed shard boundaries and incompatible vector algorithms, metrics or dimensions with their replacements. Invalid source vectors fail maintenance before publishing mixed definitions; retrying the same commit identity preserves the committed data without duplicating snapshots. A native vector training-configuration constraint can retry once with complete pending groups using the same algorithm, metric and budget. Native I/O/execution failures do not trigger expansion.
• Record the actual scalar/full-text builder schema in the file name and verify the field ID/type before retaining an index. Legacy file names require a complete unchanged type history. This handles stale producers, type round trips, text narrowing and expired source snapshots without changing Java/native formats. Compare parsed full-text configurations using the builder's parser: analyzer changes and unproven metadata rebuild old leaves, while explicit defaults remain compatible. Compatible scalar writer-option changes keep their leaves.
• Preserve Java's different empty-vector contracts: PK publishes source-backed empty segments; all-NULL DE builds produce no payload or marker. Build and query paths share field-option parsing, vector algorithms/metrics and source-type checks.
• Revalidate source sets, schema and captured index membership on publication retries. Clean only private outputs on definite failures, preserve historical files and uncertain publications, and recover duplicate commits before expired-source checks.
• Require captured vector-source deletions to remain deleted on commit, including retained sources outside the task inputs. Permit DV growth and container repacking; reject a rollback that restores omitted positions. Unchanged references and absent captured masks incur no extra payload I/O.
• Reuse captured snapshot/source/index metadata across automatic maintenance tasks and columns. Filter source views with positions into the shared metadata buffer, materializing only at the native builder boundary.

Index declarations are unset by default. write-only=false maintains declared indexes automatically; explicit PK compaction also maintains them with write-only=true.

Tests

• PK integration coverage includes all native families, physical positions, incremental/history retention, HASH/DV coexistence, independent buckets, source/index/schema conflicts, missing/tampered outputs, failure cleanup and idempotent recovery. Java empty payload fixtures exercise query-side source coverage without native file I/O.

• DE LSM regressions cover independent appends through the merge threshold, small-tail selection with a retained large base, per-column overrides, shard boundaries, logical holes, overlap/removal retirement, physical source rewrites, conflict replanning and lost-publication recovery. The incremental regression fails against the former eager-rebuild implementation.

• Native regressions cover scalar payload results, full-text hits/history/allow-lists and scalar FAST/FULL/DETAIL filtering, tiny and sparse IVF/PQ leaves, actual non-NULL training counts, row-ID offsets, budget-triggered expansion and all-NULL recovery. I/O assertions verify leaf builds avoid old base data and produce one final payload.

• Shared selector tests and existing PK planner tests protect the unchanged PK policy. Metadata-I/O and common manifest-merge tests cover shared-buffer retention, source validation and layout-specific overlap rules.

• Review regressions cover scalar type evolution/round trips and legacy schema history, stale producers, source-snapshot expiration, full-text narrowing/analyzer changes/default equivalence, native versus raw FULL/DETAIL results, vector dimension changes and idempotent failure recovery, PK field projection, hidden invalid vector values, casts before native validation, multi-file ordinal gaps at batch sizes 1/8, fully deleted source I/O, and concurrent DV removal/reduction/repacking/growth. Regressions reproduced the type-retention, full-text, vector-definition, hidden-vector and DV-restoration failures before their fixes.

• Unified-option regressions cover declaration/JSON precedence, singular DLF keys, invalid legacy JSON suppressed by shared JSON, typed full-text arrays, insertion-order aliases, deterministic dimension aliases, mode-specific schema guards, per-column BTree fallback with an overridden invalid default, scoped vector query widths and refinement, and native PK/DE builds and reads. PK recovery exercises both old and shared declarations; DE FAST results agree with raw FULL/DETAIL under the same field configuration.

• Resolver regressions cover metadata precedence, stale/malformed PK sources, mixed algorithms, empty markers, complete/incomplete legacy PK configuration, captured query options and shared worker transport without separate type hints.

• Current complete Core: 4,814 passed, 6 ignored. Current PK/append/DE integration matrix: 160 passed. Focused follow-ups verify PK canonical/singular declarations without synthetic old field-type keys, and JSON-encoded full-text collection aliases. The raw full-map vector build regression reproduced the former strict-validator failure against the preceding SDK.

• Current Python binding unit tests: 147 passed, including ordinary builders and absence of the removed setter.

• Current real SDK tests: 41 passed, covering single/batch/hybrid option capture, pickle transport, metadata precedence, removed hint aliases, raw full-map vector construction, and full-text JSON collection aliases. The SDK is rebuilt from this tree and loaded with PyArrow 24. New regression cases cover untyped table/query field-JSON metrics across FULL/DETAIL and single/batch requests, plus IVF/DiskANN full-schema builds and serialized shard reconstruction. Full-text tests cover shared and flat JSON lists, including aliases, both with and without declarations; full-text construction retains complete option normalization. Both behavior failures were reproduced against the preceding SDK before their fixes.

• Native Core regressions cover independent hybrid columns, canonical/singular/legacy configuration, mixed indexed/raw coverage, raw-source and serialized worker reconstruction, PK table/query metric scope, and eager invalid-input validation.

• Earlier in the branch, the full paimon --all-targets --features vortex suite passed: 5,660 passed, 6 ignored across 86 test targets. The latest native API/configuration changes were checked with the complete core/native matrix and Python suites above.

• Passed CARGO_BUILD_JOBS=2 cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings and cargo +1.98.0 fmt --all -- --check.

API, format and documentation

CoreOptions::vector_index_options(Option<&str>, column, default_metric) returns a serializable result containing implementation, merged options and metric. A supplied metadata type wins over declarations; the default flag does not inject an unspecified query metric. Legacy PK type/metric convenience methods and independent type lookup are removed; Rust consumers use the shared index-option result. Remaining layout adapters stay crate-internal. Configuration compatibility remains independent of exposing those methods: Rust retains the complete old PK columns/type pair, while an orphan legacy field type does not select an implementation. There is no separate Native type setter or per-query hint state. Canonical/legacy/DLF declarations use ordinary options; typed persisted/source metadata remains supported.

CompactResult adds optional serialized index_entries; older results still deserialize. First publication requires re-execution when index maintenance is missing, while already published results remain recoverable. PK source metadata uses Java's existing version-1 DataOutput/writeUTF encoding. Native index formats are unchanged.

Updated docs/src/compaction.md describes declarations, shared maintenance boundaries, LSM policies, vector constraints, atomicity, concurrency and recovery.

Legacy PK declarations/field JSON and DLF singular declarations/flat parameters remain accepted. The four removed query type hints must move to canonical declarations in ordinary table/query/route options. Before adopting shared-only PK properties, upgrade readers/writers; mixed-version deployments should retain aligned legacy declarations and field values for older clients. This change does not rewrite stored table options.

[QuakeWang]

LGTM