feat(fs): add OSS file system - #205
Conversation
Add read-only Alibaba Cloud OSS access through OSS SDK v2 while reusing shared object-store semantics and input streams. Support endpoint and region settings, credentials, session tokens, path-style addressing, and signature versions. Register OSS at the top-level build alongside the existing object-store file systems. Co-authored-by: GPT-5.6 Terra <codex@users.noreply.github.com>
|
I identified an OSS SDK v2 issue where a retried request can retain a stale This PR temporarily applies a small build-time patch to overwrite the header on retry. The issue has been reported to the OSS SDK maintainers, and they will address it upstream soon. |
| @@ -0,0 +1,35 @@ | |||
| /* | |||
| * Copyright 2026-present Alibaba Inc. | |||
| if(PAIMON_ENABLE_JINDO) | ||
| add_definitions(-DPAIMON_ENABLE_JINDO) | ||
| endif() | ||
| if(PAIMON_ENABLE_OSS) |
There was a problem hiding this comment.
could you add oss and s3 options in the docs (docs/source/building.rst and docs/source/build_system.rst)
| PATCH_COMMAND bash -c | ||
| "perl -0pi -e 's/\"-Wall\"\\s+\"-Werror\"\\s+\"-pedantic\"\\s+\"-Wextra\"/\"-Wall\" \"-pedantic\" \"-Wextra\"/g' <SOURCE_DIR>/sdk/CMakeLists.txt" | ||
| COMMAND bash -c | ||
| "perl -0pi -e 's/#if LIBCURL_VERSION_NUM >= 0x074400/#if 0/g' <SOURCE_DIR>/sdk/src/transport/curl/CurlMultiTransport.cpp" | ||
| COMMAND bash -c | ||
| "perl -0pi -e 's/#if LIBCURL_VERSION_NUM >= 0x074200/#if 0/g' <SOURCE_DIR>/sdk/src/transport/curl/CurlMultiTransport.cpp" | ||
| COMMAND bash -c | ||
| "perl -0pi -e 's/request->headers\\.emplace\\(\"Authorization\", credentialHeader\\)\\x3b/request->headers.insert_or_assign(\"Authorization\", credentialHeader)\\x3b/g' <SOURCE_DIR>/sdk/src/signer/SignerV1.cpp" | ||
| COMMAND bash -c | ||
| "perl -0pi -e 's/request->headers\\.emplace\\(\"Authorization\", ss\\.str\\(\\)\\)\\x3b/request->headers.insert_or_assign(\"Authorization\", ss.str())\\x3b/g' <SOURCE_DIR>/sdk/src/signer/SignerV4.cpp" | ||
| ${THIRDPARTY_LOG_OPTIONS} |
There was a problem hiding this comment.
could you make the patch as a diff file (like cmake_modules/jieba.diff), and add the diff file in LICENSE file like ↓
There was a problem hiding this comment.
It will be removed once the fix aliyun/alibabacloud-oss-cpp-sdk-v2@fe7d847 is merged. I plan not to involve the patch in this commit.
2 participants
Purpose
This PR adds a native, read-only OSS filesystem backed by Alibaba Cloud OSS SDK V2.
Motivation
Static linking and distribution
Some integration scenarios require Paimon-cpp to be linked statically into another library, such as a DuckDB extension. JindoSDK is delivered as a separate dynamic library, which makes this integration and distribution model less flexible.
Shutdown reliability in PostgreSQL processes
While integrating PostgreSQL with DuckDB, we observed an intermittent shutdown hang along the PostgreSQL → DuckDB → duckdb-paimon → paimon-cpp path when JindoSDK was loaded through the Paimon filesystem. The captured shutdown stack was:
The process also had a
bvar_samplerthread running. This points to a fork/exit lifecycle issue in JindoSDK's embedded bRPC/bvar dependencies. Since JindoSDK is closed source and brings a relatively large dependency set, it is difficult to investigate or address from Paimon-cpp.An open-source OSS filesystem
Given that OSS SDK V2 is open source, adding an OSS implementation to Paimon-cpp would provide an open and self-contained option for OSS access. The existing object-store abstraction introduced for S3 keeps this addition small and avoids duplicated filesystem logic.
Benchmark
The benchmark directly reads a 133 MiB Parquet data file belonging to a Paimon table in a real OSS environment. Both filesystems were built in a clean Release build, and each workload was run six times. Values are medians.
Open(known status)reopens a file with previously obtained metadata. It avoids an extra metadata request through the status propagation introduced in #189. The range and sequential workloads each issue a single read request.