From 1c818be1154f0bfee4784a723599cb482fce7fd0 Mon Sep 17 00:00:00 2001 From: Nicholas Jiang Date: Tue, 4 Aug 2026 23:36:42 +0800 Subject: [PATCH] feat(build): support Linux aarch64 builds The architecture-specific build logic keyed on PAIMON_CPU_FLAG and PAIMON_ARMV8_ARCH, neither of which was ever defined, so aarch64 got no -march= at all. TargetArchitecture.cmake now resolves the target architecture once, and PAIMON_AARCH64_MARCH selects the -march= value (default armv8-a; an empty value passes no -march flag). Fixing the target-dependent behavior this exposed changes one result on x86-64 as well: out of range float to integer conversion now follows Java. - Remove dead SIMD build logic: the SSE4.2 CRC32C kernel (Castagnoli, never selected, not the persisted zlib CRC-32, so no on-disk value changes), the ppc AltiVec probe, and the unread PAIMON_HAVE_NEON / ARMV8_CRC / ARMV8_CRYPTO definitions. crc32c_test.cpp now pins the checksum to zlib.crc32 reference values. - Name the package after the host platform, e.g. paimon-cpp-linux-aarch64.tar.gz; add --platform and --print-name to build_and_package.sh. Lumina is prebuilt for linux-x86_64 only and now fails configuration elsewhere with an explicit error. - Run the same CI matrix on x86_64 and aarch64 (clang-tidy excepted), plus a script-tests job for the new CMake-module, packaging-argument and asan_symbolize tests; all jobs are required checks. - Fix char-signedness defects the unsigned-char AArch64 Linux ABI exposed: BinaryString UTF-8 leading-byte classification (NumChars, Substring and IndexOf counted bytes, not characters), tolower and toupper on bytes at or above 0x80, TINYINT min/max aggregation (min(-20, 10) returned 10) and VariantValueToString (-20 printed as 236). - Replace the undefined float/double to integer static_casts, in the cast executor and in the Arrow kernel it called, with JavaFloatingToIntegerCast: NaN to 0, out of range saturating at the int32/int64 bounds, then narrowing to width. Both the literal and the array path use it so stats and data stay consistent. - Link compiler-rt's builtins into the Clang UBSan build: the 128-bit multiplication overflow check calls __muloti4, which libgcc does not provide and Clang does not inline on aarch64, so the sanitizer build did not link there. - Pin asan_symbolize.py stdin/stdout to UTF-8 with surrogateescape so a byte that is not valid UTF-8 no longer truncates the test log. - docs: a supported platform matrix, rules on char signedness and on float to integer conversion, and Java-consistent results in the type change support matrix. --- .asf.yaml | 26 +- .github/workflows/build_and_test.yaml | 52 +++- .github/workflows/release_candidate.yaml | 7 +- README.md | 3 +- build_and_package.sh | 51 ++++ build_support/asan_symbolize.py | 14 +- ci/scripts/build_paimon.sh | 6 + ci/scripts/test_asan_symbolize.sh | 95 ++++++ ci/scripts/test_cmake_modules.sh | 34 +++ ci/scripts/test_packaging_args.sh | 164 ++++++++++ cmake_modules/DefineOptions.cmake | 18 ++ cmake_modules/SetupCxxFlags.cmake | 84 +++--- cmake_modules/TargetArchitecture.cmake | 50 ++++ cmake_modules/ThirdpartyToolchain.cmake | 26 +- cmake_modules/san-config.cmake | 22 ++ .../tests/aarch64_march_option_test.cmake | 115 +++++++ .../tests/target_architecture_test.cmake | 114 +++++++ docs/code-style.md | 20 +- docs/source/building.rst | 60 +++- docs/source/user_guide/read.rst | 16 +- src/paimon/common/data/binary_string.cpp | 27 +- src/paimon/common/data/binary_string_test.cpp | 20 +- src/paimon/common/data/data_define.h | 4 + src/paimon/common/data/data_define_test.cpp | 11 + .../binary_row_partition_computer_test.cpp | 8 +- src/paimon/common/utils/crc32c.cpp | 41 +-- src/paimon/common/utils/crc32c.h | 29 +- src/paimon/common/utils/crc32c_test.cpp | 102 ++++++- .../utils/data_converter_utils_test.cpp | 9 +- .../core/casting/cast_executor_test.cpp | 283 ++++++++++++------ .../numeric_primitive_cast_executor.cpp | 70 +++++ .../casting/numeric_primitive_cast_executor.h | 63 +++- .../compact/aggregate/field_max_agg.h | 12 + .../compact/aggregate/field_min_agg.h | 12 + .../aggregate/field_min_max_agg_test.cpp | 3 + 35 files changed, 1421 insertions(+), 250 deletions(-) create mode 100755 ci/scripts/test_asan_symbolize.sh create mode 100755 ci/scripts/test_cmake_modules.sh create mode 100755 ci/scripts/test_packaging_args.sh create mode 100644 cmake_modules/TargetArchitecture.cmake create mode 100644 cmake_modules/tests/aarch64_march_option_test.cmake create mode 100644 cmake_modules/tests/target_architecture_test.cmake diff --git a/.asf.yaml b/.asf.yaml index 33a74a179..0cc39ed40 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -59,17 +59,31 @@ github: app_slug: -1 - name: "rat-license-check" app_slug: -1 - - name: "asan-ubsan" + - name: "script-tests" app_slug: -1 - - name: "tsan" + - name: "asan-ubsan-x86_64" app_slug: -1 - - name: "clang-debug" + - name: "tsan-x86_64" app_slug: -1 - - name: "clang-release" + - name: "clang-debug-x86_64" app_slug: -1 - - name: "gcc-debug" + - name: "clang-release-x86_64" app_slug: -1 - - name: "gcc-release" + - name: "gcc-debug-x86_64" + app_slug: -1 + - name: "gcc-release-x86_64" + app_slug: -1 + - name: "gcc-debug-aarch64" + app_slug: -1 + - name: "gcc-release-aarch64" + app_slug: -1 + - name: "clang-debug-aarch64" + app_slug: -1 + - name: "asan-ubsan-aarch64" + app_slug: -1 + - name: "clang-release-aarch64" + app_slug: -1 + - name: "tsan-aarch64" app_slug: -1 - name: "gcc8-test" app_slug: -1 diff --git a/.github/workflows/build_and_test.yaml b/.github/workflows/build_and_test.yaml index ab3f22bb0..b19e49069 100644 --- a/.github/workflows/build_and_test.yaml +++ b/.github/workflows/build_and_test.yaml @@ -33,31 +33,69 @@ permissions: contents: read jobs: + script-tests: + name: script-tests + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Checkout paimon-cpp + uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + - name: Run CMake module tests + shell: bash + run: ci/scripts/test_cmake_modules.sh + - name: Run packaging argument tests + shell: bash + run: ci/scripts/test_packaging_args.sh + - name: Run asan_symbolize tests + shell: bash + run: ci/scripts/test_asan_symbolize.sh + build-and-test: name: ${{ matrix.name }} - runs-on: ubuntu-24.04 + runs-on: ${{ matrix.runner || 'ubuntu-24.04' }} timeout-minutes: 120 strategy: fail-fast: false matrix: include: - - name: gcc-release + - name: gcc-release-x86_64 cc: gcc-14 cxx: g++-14 build_args: --build_type Release - - name: clang-release + - name: clang-release-x86_64 build_args: --build_type Release - - name: gcc-debug + - name: gcc-debug-x86_64 cc: gcc-14 cxx: g++-14 - - name: clang-debug + - name: clang-debug-x86_64 fetch_depth: '0' # fetch the PR target branch history for clang-tidy build_args: >- --check_clang_tidy --lint_git_target_commit "origin/${{ github.base_ref || github.event.repository.default_branch }}" - - name: asan-ubsan + - name: asan-ubsan-x86_64 build_args: --enable_asan --enable_ubsan - - name: tsan + - name: tsan-x86_64 + skip_rust: true + build_args: --enable_tsan + - name: gcc-debug-aarch64 + runner: ubuntu-24.04-arm + cc: gcc-14 + cxx: g++-14 + - name: gcc-release-aarch64 + runner: ubuntu-24.04-arm + cc: gcc-14 + cxx: g++-14 + build_args: --build_type Release + - name: clang-debug-aarch64 + runner: ubuntu-24.04-arm + - name: asan-ubsan-aarch64 + runner: ubuntu-24.04-arm + build_args: --enable_asan --enable_ubsan + - name: clang-release-aarch64 + runner: ubuntu-24.04-arm + build_args: --build_type Release + - name: tsan-aarch64 + runner: ubuntu-24.04-arm skip_rust: true build_args: --enable_tsan steps: diff --git a/.github/workflows/release_candidate.yaml b/.github/workflows/release_candidate.yaml index 859b25780..6b5b1ebda 100644 --- a/.github/workflows/release_candidate.yaml +++ b/.github/workflows/release_candidate.yaml @@ -99,7 +99,7 @@ jobs: name: Verify source archive (${{ matrix.compiler }}) if: github.ref_type == 'tag' needs: archive - runs-on: ubuntu-24.04 + runs-on: ${{ matrix.runner || 'ubuntu-24.04' }} timeout-minutes: 180 strategy: fail-fast: false @@ -107,6 +107,7 @@ jobs: compiler: - gcc-14 - clang + - gcc-14-aarch64 include: - compiler: gcc-14 cc: gcc-14 @@ -114,6 +115,10 @@ jobs: - compiler: clang cc: clang cxx: clang++ + - compiler: gcc-14-aarch64 + cc: gcc-14 + cxx: g++-14 + runner: ubuntu-24.04-arm steps: - name: Checkout source uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 diff --git a/README.md b/README.md index a912e067d..9161e6f89 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,8 @@ Paimon C++ currently provides: > Paimon C++ therefore currently treats the `bitmap` global index type as unsupported. The legacy > implementation remains in the codebase pending migration to the Java-compatible format. -Note: Only Linux x86_64 builds are currently supported and verified. +Note: Linux `x86_64` and `aarch64` builds are supported and verified in CI. See the supported +platform matrix in [docs/source/building.rst](docs/source/building.rst) for other platforms. ## Building diff --git a/build_and_package.sh b/build_and_package.sh index f9e671e79..652eaecaa 100755 --- a/build_and_package.sh +++ b/build_and_package.sh @@ -26,6 +26,8 @@ MAKE_CLEAN=false PACKAGE=false CMAKE_OPTIONS=() JOBS="" +PACKAGE_PLATFORM="" +PRINT_NAME=false show_help() { cat << EOF @@ -37,6 +39,12 @@ Options: -c, --clean Clean build directory before building -p, --package Package creation -j, --jobs Number of parallel jobs for building (default: auto-detect) + --platform