[Feature] Read source-backed primary-key BTree indexes

[wangyong9999]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Paimon release-2.0.0 (Java) introduced source-backed global scalar indexes for primary-key tables (pk-btree / pk-bitmap). When an index payload covers a data level, the Java batch planner evaluates indexed predicates against the payload and narrows candidate rows down to physical row ranges inside individual data files, instead of scanning the files in full.

paimon-cpp cannot consume these indexes today: a C++ engine scanning a Paimon 2.0 primary-key table falls back to full file scans even for highly selective predicates.

The existing building blocks are not sufficient on their own:

• The BTree global index payload reader/writer is already byte-compatible with Java, but nothing plans against it for primary-key tables.
• The Data Evolution global index read path addresses rows by table-wide row id, while primary-key source-backed indexes address rows by ordinal inside a validated group of source files, later localized to per-file physical positions. The two address spaces are not interchangeable.
• feat: update commit message to version 12 #179 already landed the metadata carrier (GlobalIndexMeta._SOURCE_META, commit-message v12), but nothing decodes or validates the carried source metadata.

Solution

Implement the primary-key source-backed global scalar-index read path on top of the landed metadata carrier, following the Java release-2.0.0 wire and scan semantics:

1. Decode PrimaryKeyIndexSourceMeta v1 using the big-endian layout and Java modified UTF-8 file names, with defensive count validation and trailing-byte rejection.

2. Validate coverage independently for each bucket, indexed field, index type, and positive data level. A level is covered only when exactly one payload exists and its ordered source file names and row counts exactly match all active COMPACT files of that level, with matching field id, index type, payload row count, and row range. Missing, corrupt, duplicate, or mismatched payloads leave that level uncovered and it is scanned normally.

3. Plan ordinary primary-key batch scans against validated groups. Evaluate the indexed part of the predicate once per group, localize group ordinals to file-local physical row positions using source row-count prefixes, and keep deletion files aligned with their data files. An out-of-range group ordinal invalidates all covered files in that group. Invalid file-local positions or results requiring more than 4096 ranges fall back only the affected file.

4. Consume indexed splits in the raw read path by intersecting file-local row ranges with file-index selection and deletion vectors. The reader continues to apply the complete original predicate, so index results cannot change row visibility or query semantics.

5. Wire BTree payload evaluation in this issue. pk-bitmap definitions are resolved but conservatively use normal scans until a dedicated reader is available. Vector and full-text definitions are recognized during definition resolution but are not part of the scalar-index planner.

Anything else?

Per community guidance, no separate PIP is needed because this ports an existing Java capability.

The original three stacked branches have been consolidated into one self-contained implementation in #194.

This issue is scoped to read-path acceleration for ordinary primary-key batch scans. The PR also includes PkSortedIndexFile::Build, which can construct one source-backed BTree payload from value-sorted input for tests and future tooling. Automatic payload construction and maintenance during compaction are not included. Streaming scans, read-optimized scans, Data Evolution indexes, and Bitmap / vector / full-text payload readers are also outside this issue's scope.

Are you willing to submit a PR?

• I'm willing to submit a PR!