Repository navigation
[rust] Add a Paimon lake source for UnionRead - #5160
Conversation
251c7d9 to
c263381
Compare
|
cc @fresh-borzoni @beryllw, this PR is now rebased on main and ready for review. It adds Paimon snapshot planning and baseline reading for UnionRead. |
|
One dependency question: this PR uses Paimon 0.3 with Arrow 58, bridged to Fluss's Arrow 59 through the Arrow C Data Interface. Paimon 0.5 now uses Arrow 59.2, which would let us remove the bridge, but requires Rust 1.95 rather than our current 1.91 baseline. Would you prefer keeping the bridge for now, or upgrading Paimon and the minimum Rust version in a separate PR? |
@naivedogger let's go with paimon 0.5 in this PR and drop the bridge |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, left some comments, PTAL
| let children = children | ||
| .iter() | ||
| .map(|child| convert_paimon_predicate(child, builder, fields)) | ||
| .collect::<paimon::Result<Vec<_>>>()?; |
There was a problem hiding this comment.
if one child of an AND can't be converted, e.g. a float comparison, the whole predicate is dropped and we lose the key pushdown as well
for AND we can keep the convertible children, the exact filter runs afterwards anyway, only OR has to stay all or nothing
There was a problem hiding this comment.
AND now keeps the children Paimon can evaluate; OR is pushed down only if every child can be converted. The final Fluss filter still evaluates the full predicate.
| }) | ||
| } | ||
|
|
||
| fn encode_portable_split(split: &DataSplit, table_location: &str) -> Result<String> { |
There was a problem hiding this comment.
files written with data-file.external-paths are outside the table root and fail planning here, though paimon-rust reads them fine
why do we need relative paths in the payload at all?
There was a problem hiding this comment.
Removed the relative-path rewrite. The payload now preserves Paimon’s file paths and their associated deletion files, including files outside the table root.
| }; | ||
| let expected_bucket_id = first.bucket_id; | ||
| let table = | ||
| open_pinned_table(info, catalog, snapshot_id, context.reconcile_primary_key).await?; |
There was a problem hiding this comment.
every read task reopens the table and time travels to the snapshot before reading anything
I wonder if we can cache the pinned table per snapshot in the source
There was a problem hiding this comment.
The source now caches the last pinned table. Concurrent tasks for the same snapshot share one open, and active readers keep their table if the cache moves to another snapshot.
| table.schema().fields(), | ||
| info.row_type(), | ||
| )?; | ||
| if let Some(range) = context.selected_bucket { |
There was a problem hiding this comment.
with selected_bucket every task still plans the whole partition and drops other buckets afterwards, so each bucket worker reads the same manifests
Seems paimon-rust has no public bucket filter yet, do I miss smth or is it a gap?
There was a problem hiding this comment.
Yes, Paimon 0.5 has no public batch scan API for a physical bucket ID. This PR prunes partitions and filters out other buckets before encoding tasks, but workers can still read the same manifests.
|
|
||
| [dependencies] | ||
| arrow_array_58 = { package = "arrow-array", version = "58.4.0", features = ["ffi"], optional = true } | ||
| paimon = { version = "0.3.0", default-features = false, features = ["storage-memory", "storage-fs", "storage-oss", "storage-s3"], optional = true } |
There was a problem hiding this comment.
paimon depends on reqwest with default features, so this feature brings openssl and switches the reqwest client used by fluss-rs from rustls to native-tls
default-features = false here doesn't help since it's paimon's own dependency, probably worth an issue in paimon-rust before the bindings enable it
fluss-rs can protect itself by always asking for its own library explicitly until it's fixed
There was a problem hiding this comment.
Fluss now explicitly selects rustls for S3/OSS data requests. Paimon 0.5 still pulls in native TLS, and OpenDAL’s credential client is separate.
|
@fresh-borzoni, thank you for the helpful review! I upgraded to Paimon 0.5, removed the Arrow bridge, and raised the fluss-lake MSRV to 1.95. I’ve replied inline on the planner and reader changes, including the two paimon-rust follow-ups. PTAL. |
fcbe3bd to
233813d
Compare
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, LGTM, pushed some improvements on top, will merge once CI is green
I was also thinking - shall we move json payload to paimon's own split format?
Closes #5155