Skip to content

[rust] Add bounded append UnionRead execution - #5161

Open
naivedogger wants to merge 6 commits into
apache:mainfrom
naivedogger:fip48/pr5-append-execution
Open

naivedogger wants to merge 6 commits into
apache:mainfrom
naivedogger:fip48/pr5-append-execution

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Closes #5156

@naivedogger

naivedogger commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

cc @fresh-borzoni @beryllw, this PR is now rebased on main and ready for review. It adds bounded Append UnionRead execution with exact filtering and projection.
Would appreciate your review!

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, left some comments, PTAL

let scanner = table_scan
.create_record_batch_log_scanner()
.map_err(|error| execution_client_error("create append-log scanner", error))?;
let reader = RecordBatchLogReader::new_from_ranges(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the partition is dropped after planning, e.g. expired by partition TTL, reading its task never finishes

The scanner ignores PartitionNotExists and unknown bucket errors and keeps retrying, which is fine for streaming but a bounded read should end with DataUnavailable here. Probably the core bounded reader is the right place to fail, wdyt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The core bounded reader now propagates PartitionNotExists when a metadata refresh confirms the partition is gone. UnionRead reports DataUnavailable, including when the partition disappears after the first batch. Streaming scans still retry.

@@ -0,0 +1,468 @@
// Licensed to the Apache Software Foundation (ASF) under one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file isn't run in CI, the rust-integration job only runs test_prepare, can you add it to client-integration.yml?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rust-integration job now runs test_union_read alongside test_prepare.

pub(crate) fn table_info(&self) -> &TableInfo {
&self.table_info
}
pub(crate) fn validate_live_table(&self, live: &TableInfo) -> Result<()> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like FlussLakeReadContext::validate_table with slightly different rules, can we reuse it?

also a dropped table ends up as DataUnavailable but a recreated one as SchemaIncompatible, both mean the frozen source is gone, I'd expect DataUnavailable for both like in the Java capture

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The context and split now share the same table validation. Dropping a table or recreating it under the same name gives DataUnavailable in both paths; a layout change on the same table still gives SchemaIncompatible.

.expect("Failed to build append record batch")
}

#[tokio::test]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

none of the tests read lake data, there is no LakeSource here at all
Mb a stub LakeSource returning fixed batches on a lake-enabled table?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used a stub LakeSource on a lake-enabled table: the lake returns row 1, the Fluss log contains rows 1 and 2, and the frozen range starts at row 2. The reader returns each row exactly once.

/// Construction validates local task compatibility without I/O. Polling
/// refreshes table metadata and opens the frozen inputs. Initialization
/// and read failures are terminal stream errors. Drop cancels the read.
pub fn read_split(&self, split: &FlussLakeReadSplit) -> Result<RecordBatchStream> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

every read_split builds its own resources, so each task refreshes table metadata and is noticeably slower than the same tasks through read_splits
Engines like StarRocks or Doris call it per scan range, I wonder if the reader can keep these resources

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reader now owns the shared resources. Concurrent read_split calls on the same reader or its clones reuse the table metadata and lake source. A new execution attempt gets a fresh reader.

"PK UnionRead execution requires the PK reconciliation executor".into(),
));
}
if split.is_empty() || (scan.lake_only() && split.lake_splits().is_empty()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a lake-only reader returns an empty stream for a task that has only a log tail
Should we reject it instead? lake-only planning never produces such tasks, so getting one here is a mistake

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lake-only reader now rejects any task with a nonempty log range. A genuinely empty task still returns an empty stream.

}

/// Retains cause messages in diagnostics without changing the public error variants.
pub(crate) fn error_message(action: &str, error: &dyn std::error::Error) -> String {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same helper is added in #5160, can we reuse it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Once either #5160 or this PR lands, I’ll rebase the other and keep a single shared error_message helper.

self
}

/// Sets the target output batch size.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it only splits larger batches and never merges smaller ones, so it's rather a max batch size

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed it to with_max_batch_size. Smaller batches pass through unchanged.

table_info.row_type(),
output_projection.as_deref(),
filter,
descriptor.primary_key_indexes(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: lake-only PK reads don't reconcile, do we still need the hidden key columns there?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the extra key columns from this path. A lake-only PK read now requests a key column only if the projection or filter needs it.

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thanks for the review! Updated in b42d93a. Bounded reads now fail when a frozen source disappears, and split reads share resources within an execution attempt. I’ll consolidate the error helper when this PR or #5160 lands. PTAL.

@fresh-borzoni

Copy link
Copy Markdown
Member

@naivedogger let's rebase now

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni Thanks! Rebasing.

@naivedogger
naivedogger force-pushed the fip48/pr5-append-execution branch from b42d93a to eb65d9b Compare November 19, 2026 02:04

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thanks, lgtm, non-blocking comment

///
/// Use the same projection, filter, read mode and batch size that produced
/// the tasks, with a compatible lake backend. Task metadata does not encode
/// these settings; mismatches can produce incomplete results or the wrong

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in Java the splits don't carry the filter either, but Flink ships the same source object to readers, here the engine rebuilds the scan from descriptors, so I wonder if the task should store it and fail on mismatch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Add bounded append UnionRead execution

2 participants