feat(parquet): support vector type storage

[ChaomingZhangCN]

Purpose

Linked issue: #197

Related design: PIP-40: Introduce a new Vector data type

This PR implements the first phase of VECTOR<T, N> support in Paimon C++. It introduces the logical type and schema representation, maps it to Arrow FixedSizeList<T, N>, and provides end-to-end Parquet reads and writes while keeping the physical Parquet representation interoperable with standard readers.

The main changes are:

• Add FieldType::VECTOR and support the SQL-style representation VECTOR<T, N>.

• Add JSON schema serialization and parsing, for example:

  {
    "type": "VECTOR",
    "element": "FLOAT",
    "length": 3
  }

• Support BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, and DOUBLE element types.

• Allow a VECTOR value to be null, but reject null elements inside a non-null VECTOR.

• Validate that the vector length is positive and that every value matches the dimension declared by the schema.

• Support VECTOR values nested in STRUCT, LIST, and MAP using the default MAP layout.

• Preserve nested field IDs and collect nested null-count statistics for VECTOR columns.

• Reject VECTOR columns in primary, partition, and bucket keys.

• Restrict VECTOR data files to Parquet in this phase.

• Reject VECTOR values inside a shared-shredding MAP until the two features can be integrated safely.

Logical and physical representation

flowchart LR
    subgraph WritePath["Write path"]
        W1["Paimon schema VECTOR<T, N>"] --> W2["Arrow logical array FixedSizeList<T, N>"]
        W2 -->|"validate dimension and nulls"| W3["Parquet write array List<T>"]
    end

    subgraph Storage["Physical storage"]
        P[("Parquet data file standard LIST encoding")]
    end

    subgraph ReadPath["Read path"]
        R1["Parquet reader List<T>"] -->|"validate T and N"| R2["Arrow result FixedSizeList<T, N>"]
    end

    W3 --> P --> R1
    E[("Third-party Parquet file ordinary LIST column")] --> R1

Loading

VECTOR has a fixed-size logical representation in Paimon and Arrow, but it is written as a standard Parquet LIST. On read, the converter validates the element type, row length, and nullability before restoring the Arrow FixedSizeList. This also allows a compatible LIST column written by another Parquet implementation to be read as VECTOR when the Paimon schema declares VECTOR<T, N>.

Scope of this phase

flowchart TB
    V["VECTOR<T, N> phase 1"]

    V --> S["Schema and type system"]
    S --> S1["SQL and JSON representation"]
    S --> S2["Arrow FixedSizeList mapping"]
    S --> S3["Dimension and null validation"]

    V --> IO["Parquet I/O"]
    IO --> IO1["FixedSizeList → LIST writes"]
    IO --> IO2["LIST → FixedSizeList reads"]
    IO --> IO3["STRUCT / LIST / MAP nesting"]

    V --> M["Metadata integration"]
    M --> M1["Nested field IDs"]
    M --> M2["Nested null-count statistics"]

    V -.-> F["Follow-up phases"]
    F --> F1["Data Evolution"]
    F --> F2["Dedicated vector-store format"]
    F --> F3["Vector indexes and search"]
    F --> F4["Shared-shredding MAP integration"]

Loading

Data Evolution, a dedicated point-lookup-optimized vector format, vector indexes/search, and shared-shredding MAP integration are intentionally left for follow-up work.

Tests

Added focused coverage for:

• VECTOR SQL/JSON parsing, serialization, supported element types, and invalid definitions.
• Arrow type conversion and schema validation.
• Primary/partition/bucket key restrictions and Parquet-only validation.
• Nested field ID propagation and nested column statistics.
• VECTOR-to-LIST and LIST-to-VECTOR conversion, including nullable vectors, sliced arrays, invalid dimensions, and null elements.
• VECTOR values nested in STRUCT, LIST, and MAP.
• Reading an ordinary third-party Parquet LIST column as VECTOR.
• End-to-end Parquet I/O for FLOAT and nested DOUBLE vectors.
• Table-level append/write/read coverage in WriteAndReadInteTest.TestAppendVector.

Validation performed locally:

• 22 focused unit tests passed across type/JSON handling, Arrow utilities, schema validation, field IDs/statistics, and Parquet conversion/I/O.
• paimon-write-and-read-inte-test and paimon-data-evolution-table-test built successfully.
• All changed files passed the repository pre-commit hooks, including clang-format, C++ lint, CMake format, codespell, and Sphinx lint.
• git diff --check passed.

The table-level VECTOR test could not be executed successfully in the current macOS Debug build because Arrow is statically embedded in multiple dynamic libraries, causing a cross-DSO RTTI std::bad_cast. The existing non-VECTOR TestAppendSimple baseline also fails in the same environment. The test is included for validation in the supported CI/Linux environment.

API and Format

API: Yes.

• Add public FieldType::VECTOR = 18.
• Represent VECTOR columns through Arrow FixedSizeList<T, N> at the C++/Arrow boundary.
• Extend schema parsing, validation, field type utilities, and nested statistics to recognize VECTOR.

Schema protocol: Yes.

• Add the VECTOR JSON type with element and length attributes.
• Add the SQL-style type string VECTOR<T, N>.

Storage format: The change adds VECTOR storage support but does not introduce a new Parquet physical encoding.

• VECTOR is persisted using the standard Parquet LIST representation.
• Existing non-VECTOR files and schemas are unchanged.
• Compatible Parquet LIST data can be restored as VECTOR using the Paimon logical schema.
• Dedicated vector storage and Data Evolution are outside the scope of this PR.

Documentation

Yes. The data types documentation now describes:

• VECTOR<T, N> and its Arrow FixedSizeList mapping.
• Supported element types and nullability rules.
• The standard Parquet LIST representation used in this phase.
• Current restrictions and follow-up scope.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)