Repository navigation
[rust] Add lake source contracts and UnionRead planning - #5158
Conversation
340c519 to
beb7c3a
Compare
|
cc @fresh-borzoni @beryllw, this PR adds UnionRead planning and lake source contracts for FIP-48. |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, looks good, left some comments, PTAL
| .then_with(|| left_bucket.cmp(right_bucket)) | ||
| }, | ||
| ); | ||
| for ((partition, bucket_id), planned_lake_bucket) in lake_only_buckets { |
There was a problem hiding this comment.
lake tasks find their live partition by name, so a value spelled differently ends up here as lake-only
e.g. for a TIMESTAMP partition the Paimon source in #5160 returns 2024-01-01 12:00:00.500, while Fluss names it 2024-01-01-12-00-00_5
for a PK table the lake rows are then read without the tail, Java converts to Fluss names in the source with toFlussPartitionValues, can we do the same in #5160, or compare typed values here so a mismatch can't slip through?
There was a problem hiding this comment.
the core already has this conversion, convert_value_of_type in util::partition, but it's crate-private
this PR already exposes BucketingFunction for pruning, maybe expose this one too, so can we use it instead of parsing the path?
There was a problem hiding this comment.
Agreed. The task contract now requires Fluss partition encoding. The Paimon fix converts typed partition values instead of parsing paths; it is prepared for #5160.
There was a problem hiding this comment.
Exposed the existing core conversion helpers here. The pending #5160 update reuses them for both planning and task validation.
| FlussLakeReadSplit::try_new( | ||
| table_info.clone(), | ||
| FlussLakeSplitDescriptor { | ||
| kv_snapshot_id: match bucket_range.baseline() { |
There was a problem hiding this comment.
the KV snapshot reader from #5030 is in main now, but nothing calls it for these tasks yet
until it's wired, can execution reject kv_snapshot_id, so a KV task never runs as an empty baseline plus the tail?
| } | ||
|
|
||
| /// Immutable input to lake planning. The snapshot must never be refreshed. | ||
| pub struct LakePlannerContext<'a> { |
There was a problem hiding this comment.
the source doesn't know the partition scope or the selected bucket, so with N one-bucket workers it plans the whole snapshot N times
can we pass both here, so the source can prune?
There was a problem hiding this comment.
Added partition_scope and selected_bucket to LakePlannerContext. The pending #5160 update applies partition filtering before planning and filters bucket tasks before encoding. paimon-rust has no bucket scan filter yet, so some manifest work remains.
| if matches!(predicate, BoundPredicate::AlwaysTrue) { | ||
| return None; | ||
| } | ||
| if !reconcile_primary_key { |
There was a problem hiding this comment.
why only key predicates on the lake side for PK? the tail is still folded unfiltered, so a lake row is either dropped by key or goes through the exact filter at the end
the stale row Leonard raised on FIP-50 discussions needs the filter on the log side too, on the lake side alone I think it's safe for deduplicate, or did I miss something here?
There was a problem hiding this comment.
You're right for deduplicate. Removed the key-only restriction and passed the full predicate to the lake source. The contract requires preserving lake current-view semantics, folding the unfiltered tail, and applying the final exact filter.
| } | ||
|
|
||
| /// Evaluates predicates only against fields represented by partition values. | ||
| pub(crate) struct PartitionPruner { |
There was a problem hiding this comment.
PartitionPruner and PartitionScope in partition.rs both evaluate a predicate on partition values, one exact and one conservative, and the planner uses both in one plan
can they share one three-valued evaluator?
There was a problem hiding this comment.
Both now share one three-valued evaluator and the string-equality extraction. Exact scope evaluation propagates errors; conservative pruning keeps partitions when the result is unknown.
| self.estimated_size | ||
| } | ||
|
|
||
| pub(crate) fn validate( |
There was a problem hiding this comment.
import runs this and then FlussLakeReadSplit::validate on the same task, about 100 lines each with overlapping checks, and planning checks the lake tasks three times before that
can we check source output once and imported tasks once?
There was a problem hiding this comment.
Removed the overlapping checks. Source output is validated before internal split construction; imported tasks keep complementary structural and frozen-boundary checks. Internally planned splits skip the redundant validation.
| // Cache once per distinct layout, rather than re-encoding keys per bucket. | ||
| let mut bucket_pruners = std::collections::HashMap::new(); | ||
| let mut bucket_may_match = |bucket_id, bucket_count| { | ||
| // A Fluss-only context does not carry a lake hash-format contract. |
There was a problem hiding this comment.
I guess pruning is off here because the writer hashes by table.datalake.format even when the lake is disabled
could the context keep the bucketing format, so Fluss-only reads get bucket pruning too?
There was a problem hiding this comment.
Enabled Fluss-only bucket pruning using the format from identity-checked table metadata. table.datalake.format is immutable, so it can be retained when restoring the frozen table without adding a protocol field.
4e681e9 to
c11ede3
Compare
|
@fresh-borzoni, thank you for the helpful review! Updated the planning contracts, pruning, and validation in the latest commit. The Paimon and executor will get fixed in the dependent PRs. PTAL. |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger makes sense, thank you, LGTM 👍
Closes #5152