Skip to content

[rust] Add lake source contracts and UnionRead planning - #5158

Merged
fresh-borzoni merged 7 commits into
apache:mainfrom
naivedogger:fip48/pr3-lake-planning
Nov 16, 2026
Merged

fresh-borzoni merged 7 commits into
apache:mainfrom
naivedogger:fip48/pr3-lake-planning

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Closes #5152

@naivedogger

Copy link
Copy Markdown
Contributor Author

cc @fresh-borzoni @beryllw, this PR adds UnionRead planning and lake source contracts for FIP-48.
Would appreciate your review!

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, looks good, left some comments, PTAL

.then_with(|| left_bucket.cmp(right_bucket))
},
);
for ((partition, bucket_id), planned_lake_bucket) in lake_only_buckets {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lake tasks find their live partition by name, so a value spelled differently ends up here as lake-only
e.g. for a TIMESTAMP partition the Paimon source in #5160 returns 2024-01-01 12:00:00.500, while Fluss names it 2024-01-01-12-00-00_5

for a PK table the lake rows are then read without the tail, Java converts to Fluss names in the source with toFlussPartitionValues, can we do the same in #5160, or compare typed values here so a mismatch can't slip through?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the core already has this conversion, convert_value_of_type in util::partition, but it's crate-private
this PR already exposes BucketingFunction for pruning, maybe expose this one too, so can we use it instead of parsing the path?

@naivedogger naivedogger Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The task contract now requires Fluss partition encoding. The Paimon fix converts typed partition values instead of parsing paths; it is prepared for #5160.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exposed the existing core conversion helpers here. The pending #5160 update reuses them for both planning and task validation.

FlussLakeReadSplit::try_new(
table_info.clone(),
FlussLakeSplitDescriptor {
kv_snapshot_id: match bucket_range.baseline() {

@fresh-borzoni fresh-borzoni Nov 15, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the KV snapshot reader from #5030 is in main now, but nothing calls it for these tasks yet
until it's wired, can execution reject kv_snapshot_id, so a KV task never runs as an empty baseline plus the tail?

@naivedogger naivedogger Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The execution-branch fixes reject KV baselines before any empty-task fast path. They are prepared for #5161/#5162; actual KV snapshot execution remains a follow-up.

}

/// Immutable input to lake planning. The snapshot must never be refreshed.
pub struct LakePlannerContext<'a> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the source doesn't know the partition scope or the selected bucket, so with N one-bucket workers it plans the whole snapshot N times
can we pass both here, so the source can prune?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added partition_scope and selected_bucket to LakePlannerContext. The pending #5160 update applies partition filtering before planning and filters bucket tasks before encoding. paimon-rust has no bucket scan filter yet, so some manifest work remains.

if matches!(predicate, BoundPredicate::AlwaysTrue) {
return None;
}
if !reconcile_primary_key {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why only key predicates on the lake side for PK? the tail is still folded unfiltered, so a lake row is either dropped by key or goes through the exact filter at the end
the stale row Leonard raised on FIP-50 discussions needs the filter on the log side too, on the lake side alone I think it's safe for deduplicate, or did I miss something here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right for deduplicate. Removed the key-only restriction and passed the full predicate to the lake source. The contract requires preserving lake current-view semantics, folding the unfiltered tail, and applying the final exact filter.

}

/// Evaluates predicates only against fields represented by partition values.
pub(crate) struct PartitionPruner {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PartitionPruner and PartitionScope in partition.rs both evaluate a predicate on partition values, one exact and one conservative, and the planner uses both in one plan
can they share one three-valued evaluator?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both now share one three-valued evaluator and the string-equality extraction. Exact scope evaluation propagates errors; conservative pruning keeps partitions when the result is unknown.

self.estimated_size
}

pub(crate) fn validate(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import runs this and then FlussLakeReadSplit::validate on the same task, about 100 lines each with overlapping checks, and planning checks the lake tasks three times before that
can we check source output once and imported tasks once?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the overlapping checks. Source output is validated before internal split construction; imported tasks keep complementary structural and frozen-boundary checks. Internally planned splits skip the redundant validation.

// Cache once per distinct layout, rather than re-encoding keys per bucket.
let mut bucket_pruners = std::collections::HashMap::new();
let mut bucket_may_match = |bucket_id, bucket_count| {
// A Fluss-only context does not carry a lake hash-format contract.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess pruning is off here because the writer hashes by table.datalake.format even when the lake is disabled
could the context keep the bucketing format, so Fluss-only reads get bucket pruning too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabled Fluss-only bucket pruning using the format from identity-checked table metadata. table.datalake.format is immutable, so it can be retained when restoring the frozen table without adding a protocol field.

@naivedogger
naivedogger force-pushed the fip48/pr3-lake-planning branch from 4e681e9 to c11ede3 Compare November 16, 2026 02:17
@naivedogger

naivedogger commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thank you for the helpful review! Updated the planning contracts, pruning, and validation in the latest commit. The Paimon and executor will get fixed in the dependent PRs. PTAL.

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger makes sense, thank you, LGTM 👍

@fresh-borzoni
fresh-borzoni merged commit faeab32 into apache:main Nov 16, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Add lake source contracts and UnionRead planning

2 participants