perf(read): add bounded late materialization

[gripleaf]

Purpose

Add bounded late materialization for raw-file reads to reduce unnecessary payload-column I/O.

When enabled, the reader:

• Reads predicate columns first and evaluates predicates, deletion vectors, and index bitmaps.
• Reads payload columns only for matching row ranges.
• Preserves row offsets and reader metrics when combining probe and payload readers.
• Falls back to the regular read path when the number of matching rows exceeds the configured threshold.
• Uses the regular read path when late-materialization prerequisites are not satisfied.

The following options are introduced:

• read.late-materialization.enabled
• read.late-materialization.max-match-rows

This change also adds validation for malformed predicate results, invalid bitmap positions, missing payload readers, and inconsistent probe/payload batch sizes.

Tests

• Added unit tests for:

  • Probe and payload column merging.
  • Probe-only reads.
  • Payload bitmap filtering.
  • Row-offset preservation across batches.
  • Reader metrics preservation.
  • Payload batches with different lifetimes.
  • Mismatched probe and payload batch sizes.
  • Missing payload readers and invalid arguments.
  • Default, configured, and invalid late-materialization options.

• Ran paimon-common-test with the relevant late-materialization and metrics tests: 10 tests passed.

• Ran CoreOptionsTest.* and RawFileSplitReadTest.*: 44 tests passed.

• Built paimon-common-test and paimon-core-test in the dev container with GCC 13.3 and -Werror.

• Ran clang-format --dry-run --Werror and git diff --check.

API and Format

This change adds public option-name constants under include/paimon/defs.h.

It does not change the storage format or protocol and remains backward compatible because late materialization is disabled by default.

Documentation

Added user documentation describing the late-materialization options, activation requirements, bounded fallback behavior, and metrics semantics.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[zhf999]

Thanks for your contribution!

The feature is gated on every file having a first_row_id, but the global row id isn't actually used — it's added and immediately subtracted back:

• probe: global = first_row_id + file_row_id
• payload: ToFileSelection translates it right back (from - start, start = first_row_id)

What reaches the format reader is a file-local selection bitmap, and the payload reader is already created per file. So the file-local file_row_id from GetPreviousBatchFileRowId is enough on its own; the global round-trip only exists to reuse the existing global ranges pushdown.

The downside: it unnecessarily restricts an otherwise general optimization to row-tracking append tables (the only files with a first_row_id).

Proposal: implement this purely on top of FileBatchReader, pushing a file-local selection down directly from the probe-collected file_row_ids — no first_row_id or global-row-id dependency. Correctness already relies only on ascending file-local ordering. If global ids are needed for a planned cross-file use, a comment noting that would help.

[zhf999]

Besides, could you also add some unit-test to cover what changed in this PR, including the normal path and fallbacks paths where TryCreateLateMaterializedReader fails to create late-mat reader?

[zhf999]

Thanks a lot for putting this together, @gripleaf! The two-pass probe/payload design is an interesting read, and the bounded-fallback idea gave us some useful new perspectives.

I should mention that we've also been working on late materialization for the read path internally, and our current direction differs somewhat from this PR in a few design aspects. So we'd like to take a bit more time to align on the overall approach before landing changes in this area.

Once our proposal is ready we'll share it publicly and would love to invite you to review it — your input would be very welcome, and hopefully we can converge on something that takes the best of both.

Thanks again for the contribution!