Repository navigation
[rust] Add bounded append UnionRead execution - #5161
naivedogger wants to merge 6 commits into
Conversation
c3d17a7 to
35d5cae
Compare
|
cc @fresh-borzoni @beryllw, this PR is now rebased on main and ready for review. It adds bounded Append UnionRead execution with exact filtering and projection. |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, left some comments, PTAL
| let scanner = table_scan | ||
| .create_record_batch_log_scanner() | ||
| .map_err(|error| execution_client_error("create append-log scanner", error))?; | ||
| let reader = RecordBatchLogReader::new_from_ranges( |
There was a problem hiding this comment.
if the partition is dropped after planning, e.g. expired by partition TTL, reading its task never finishes
The scanner ignores PartitionNotExists and unknown bucket errors and keeps retrying, which is fine for streaming but a bounded read should end with DataUnavailable here. Probably the core bounded reader is the right place to fail, wdyt?
There was a problem hiding this comment.
Agreed. The core bounded reader now propagates PartitionNotExists when a metadata refresh confirms the partition is gone. UnionRead reports DataUnavailable, including when the partition disappears after the first batch. Streaming scans still retry.
| @@ -0,0 +1,468 @@ | |||
| // Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
this file isn't run in CI, the rust-integration job only runs test_prepare, can you add it to client-integration.yml?
There was a problem hiding this comment.
The rust-integration job now runs test_union_read alongside test_prepare.
| pub(crate) fn table_info(&self) -> &TableInfo { | ||
| &self.table_info | ||
| } | ||
| pub(crate) fn validate_live_table(&self, live: &TableInfo) -> Result<()> { |
There was a problem hiding this comment.
this looks like FlussLakeReadContext::validate_table with slightly different rules, can we reuse it?
also a dropped table ends up as DataUnavailable but a recreated one as SchemaIncompatible, both mean the frozen source is gone, I'd expect DataUnavailable for both like in the Java capture
There was a problem hiding this comment.
The context and split now share the same table validation. Dropping a table or recreating it under the same name gives DataUnavailable in both paths; a layout change on the same table still gives SchemaIncompatible.
| .expect("Failed to build append record batch") | ||
| } | ||
|
|
||
| #[tokio::test] |
There was a problem hiding this comment.
none of the tests read lake data, there is no LakeSource here at all
Mb a stub LakeSource returning fixed batches on a lake-enabled table?
There was a problem hiding this comment.
I used a stub LakeSource on a lake-enabled table: the lake returns row 1, the Fluss log contains rows 1 and 2, and the frozen range starts at row 2. The reader returns each row exactly once.
| /// Construction validates local task compatibility without I/O. Polling | ||
| /// refreshes table metadata and opens the frozen inputs. Initialization | ||
| /// and read failures are terminal stream errors. Drop cancels the read. | ||
| pub fn read_split(&self, split: &FlussLakeReadSplit) -> Result<RecordBatchStream> { |
There was a problem hiding this comment.
every read_split builds its own resources, so each task refreshes table metadata and is noticeably slower than the same tasks through read_splits
Engines like StarRocks or Doris call it per scan range, I wonder if the reader can keep these resources
There was a problem hiding this comment.
The reader now owns the shared resources. Concurrent read_split calls on the same reader or its clones reuse the table metadata and lake source. A new execution attempt gets a fresh reader.
| "PK UnionRead execution requires the PK reconciliation executor".into(), | ||
| )); | ||
| } | ||
| if split.is_empty() || (scan.lake_only() && split.lake_splits().is_empty()) { |
There was a problem hiding this comment.
a lake-only reader returns an empty stream for a task that has only a log tail
Should we reject it instead? lake-only planning never produces such tasks, so getting one here is a mistake
There was a problem hiding this comment.
A lake-only reader now rejects any task with a nonempty log range. A genuinely empty task still returns an empty stream.
| } | ||
|
|
||
| /// Retains cause messages in diagnostics without changing the public error variants. | ||
| pub(crate) fn error_message(action: &str, error: &dyn std::error::Error) -> String { |
| self | ||
| } | ||
|
|
||
| /// Sets the target output batch size. |
There was a problem hiding this comment.
nit: it only splits larger batches and never merges smaller ones, so it's rather a max batch size
There was a problem hiding this comment.
Renamed it to with_max_batch_size. Smaller batches pass through unchanged.
| table_info.row_type(), | ||
| output_projection.as_deref(), | ||
| filter, | ||
| descriptor.primary_key_indexes(), |
There was a problem hiding this comment.
nit: lake-only PK reads don't reconcile, do we still need the hidden key columns there?
There was a problem hiding this comment.
Removed the extra key columns from this path. A lake-only PK read now requests a key column only if the projection or filter needs it.
|
@fresh-borzoni, thanks for the review! Updated in b42d93a. Bounded reads now fail when a frozen source disappears, and split reads share resources within an execution attempt. I’ll consolidate the error helper when this PR or #5160 lands. PTAL. |
|
@naivedogger let's rebase now |
|
@fresh-borzoni Thanks! Rebasing. |
b42d93a to
eb65d9b
Compare
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thanks, lgtm, non-blocking comment
| /// | ||
| /// Use the same projection, filter, read mode and batch size that produced | ||
| /// the tasks, with a compatible lake backend. Task metadata does not encode | ||
| /// these settings; mismatches can produce incomplete results or the wrong |
There was a problem hiding this comment.
in Java the splits don't carry the filter either, but Flink ships the same source object to readers, here the engine rebuilds the scan from descriptors, so I wonder if the task should store it and fail on mismatch
Closes #5156