Skip to content

[rust] Add a Paimon lake source for UnionRead - #5160

Merged
fresh-borzoni merged 2 commits into
apache:mainfrom
naivedogger:fip48/pr4-paimon-source
Nov 19, 2026
Merged

fresh-borzoni merged 2 commits into
apache:mainfrom
naivedogger:fip48/pr4-paimon-source

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Closes #5155

@naivedogger

naivedogger commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

cc @fresh-borzoni @beryllw, this PR is now rebased on main and ready for review. It adds Paimon snapshot planning and baseline reading for UnionRead.
Would appreciate your review!

@naivedogger

Copy link
Copy Markdown
Contributor Author

One dependency question: this PR uses Paimon 0.3 with Arrow 58, bridged to Fluss's Arrow 59 through the Arrow C Data Interface. Paimon 0.5 now uses Arrow 59.2, which would let us remove the bridge, but requires Rust 1.95 rather than our current 1.91 baseline.

Would you prefer keeping the bridge for now, or upgrading Paimon and the minimum Rust version in a separate PR?

@fresh-borzoni

Copy link
Copy Markdown
Member

One dependency question: this PR uses Paimon 0.3 with Arrow 58, bridged to Fluss's Arrow 59 through the Arrow C Data Interface. Paimon 0.5 now uses Arrow 59.2, which would let us remove the bridge, but requires Rust 1.95 rather than our current 1.91 baseline.

Would you prefer keeping the bridge for now, or upgrading Paimon and the minimum Rust version in a separate PR?

@naivedogger let's go with paimon 0.5 in this PR and drop the bridge

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, left some comments, PTAL

let children = children
.iter()
.map(|child| convert_paimon_predicate(child, builder, fields))
.collect::<paimon::Result<Vec<_>>>()?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if one child of an AND can't be converted, e.g. a float comparison, the whole predicate is dropped and we lose the key pushdown as well
for AND we can keep the convertible children, the exact filter runs afterwards anyway, only OR has to stay all or nothing

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AND now keeps the children Paimon can evaluate; OR is pushed down only if every child can be converted. The final Fluss filter still evaluates the full predicate.

})
}

fn encode_portable_split(split: &DataSplit, table_location: &str) -> Result<String> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

files written with data-file.external-paths are outside the table root and fail planning here, though paimon-rust reads them fine
why do we need relative paths in the payload at all?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the relative-path rewrite. The payload now preserves Paimon’s file paths and their associated deletion files, including files outside the table root.

};
let expected_bucket_id = first.bucket_id;
let table =
open_pinned_table(info, catalog, snapshot_id, context.reconcile_primary_key).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

every read task reopens the table and time travels to the snapshot before reading anything
I wonder if we can cache the pinned table per snapshot in the source

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source now caches the last pinned table. Concurrent tasks for the same snapshot share one open, and active readers keep their table if the cache moves to another snapshot.

table.schema().fields(),
info.row_type(),
)?;
if let Some(range) = context.selected_bucket {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with selected_bucket every task still plans the whole partition and drops other buckets afterwards, so each bucket worker reads the same manifests
Seems paimon-rust has no public bucket filter yet, do I miss smth or is it a gap?

@naivedogger naivedogger Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Paimon 0.5 has no public batch scan API for a physical bucket ID. This PR prunes partitions and filters out other buckets before encoding tasks, but workers can still read the same manifests.

Comment thread fluss-rust/crates/fluss-lake/Cargo.toml Outdated

[dependencies]
arrow_array_58 = { package = "arrow-array", version = "58.4.0", features = ["ffi"], optional = true }
paimon = { version = "0.3.0", default-features = false, features = ["storage-memory", "storage-fs", "storage-oss", "storage-s3"], optional = true }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paimon depends on reqwest with default features, so this feature brings openssl and switches the reqwest client used by fluss-rs from rustls to native-tls
default-features = false here doesn't help since it's paimon's own dependency, probably worth an issue in paimon-rust before the bindings enable it

fluss-rs can protect itself by always asking for its own library explicitly until it's fixed

@naivedogger naivedogger Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fluss now explicitly selects rustls for S3/OSS data requests. Paimon 0.5 still pulls in native TLS, and OpenDAL’s credential client is separate.

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thank you for the helpful review! I upgraded to Paimon 0.5, removed the Arrow bridge, and raised the fluss-lake MSRV to 1.95. I’ve replied inline on the planner and reader changes, including the two paimon-rust follow-ups. PTAL.

@fresh-borzoni
fresh-borzoni force-pushed the fip48/pr4-paimon-source branch from fcbe3bd to 233813d Compare November 19, 2026 00:37

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, LGTM, pushed some improvements on top, will merge once CI is green

I was also thinking - shall we move json payload to paimon's own split format?

@fresh-borzoni
fresh-borzoni merged commit b72c5ea into apache:main Nov 19, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Add a Paimon lake source for UnionRead

2 participants