Repository navigation
[rust] Add reusable UnionRead source boundaries - #4887
Conversation
|
@fresh-borzoni @beryllw Could you please take a look at this one when you got a chance? Thanks!🙏 |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you for the PR, a couple of comments and questions, PTAL
| |(id, name, identity, count)| async move { | ||
| let bucket_ids: Vec<i32> = (0..count).collect(); | ||
| let offsets = | ||
| load_server_offsets(admin, table_path, name.as_deref(), &bucket_ids).await?; |
There was a problem hiding this comment.
Partitions are listed first, and their offsets are looked up by name afterwards. If a partition is dropped in between, the whole prepare() fails. Auto-partition retention drops partitions as a matter of course, so long-running services will hit this.
A partition that disappears mid-prepare can be treated as if it was dropped before we started: its lake data is still read from the snapshot, and its log is gone either way. A drop can surface as different errors depending on timing, not only PartitionNotExists. So when a lookup fails, it may be simplest to re-check the partition list by id, skip the partition only if it's really gone, and otherwise return the original error.
There was a problem hiding this comment.
I've updated boundary capture to recheck partition IDs after the offset lookups. A partition's log ranges are omitted only when its original ID is confirmed absent, including after same-name recreation. Other lookup failures still propagate, and lake data remains covered by the pinned snapshot.
| // Without a lake baseline the complete history is required, not merely | ||
| // the retained suffix. Availability is checked after scan-specific pruning | ||
| // so an irrelevant range cannot break a lake-only or pruned read. | ||
| let start_offset = snapshot_offset.unwrap_or(0); |
There was a problem hiding this comment.
For a bucket with no lake offset, the range starts at 0. For log tables that works, since the context also records the earliest offset, so a caller can choose to fail or read what's left. For PK tables it doesn't. If the changelog start was cleaned up, the current state can't be rebuilt from the log, although Fluss still has it in the KV snapshot.
Could a range say what sits under it (lake offset, KV snapshot, or nothing), even if only the lake case is implemented now? The KV snapshot would need to be captured here, before the stop offsets, so the snapshot and the log after it stay consistent. Adding it later would change the format. WDYT?
There was a problem hiding this comment.
Each range now explicitly identifies its baseline as lake or none, including the distinction between a lake seam at zero and no baseline. Without a baseline, reads start at the captured earliest offset, so PK results only cover keys reconstructible from retained logs. KV snapshot fallback remains future work, and readers reject unknown baseline kinds.
There was a problem hiding this comment.
for a PK table without a lake snapshot, once the log start is cleaned up (earliest > 0) reading only the log silently loses keys, main reads KV snapshot + log here
can we fail in this case until the KV snapshot fallback is in? I'll review it shortly, whatever you prefer
There was a problem hiding this comment.
PK reads without a baseline now fail when earliest > 0, pending KV snapshot fallback.
| # } | ||
| ``` | ||
|
|
||
| The context freezes table/schema identity, physical lake table mapping, a readable snapshot and half-open log ranges `[start, stop)`. Scan configuration, including projection and filters, belongs to the subsequent planning layer; preparation always captures table-wide boundaries. Actual per-partition bucket counts are preserved. The context contains no credentials or physical lake-file tasks, is not a global transactional snapshot and does not acquire a retention lease. Its JSON representation is an evolving Rust contract, not a stable cross-language ABI. |
There was a problem hiding this comment.
Engines that plan in Java (Trino, Spark, StarRocks, Doris) can't call prepare(), and the README says not to treat the JSON as cross-language. How do you see those engines getting a context? If a Java producer is planned, it's worth settling the format before later PRs build on it.
There was a problem hiding this comment.
I've added UnionReadBoundaryPlanner.captureReadContext(admin, tablePath), with an overload for partition scope, to capture the boundaries and return the encoded context in one call. Context v1 is now a documented cross-language format, so Java coordinators can produce it without calling Rust or assembling the JSON themselves.
There was a problem hiding this comment.
+1 to the fluss-client capture as a follow-up, without the Flink connector
what I'm missing now is the format: v1 was cross-language, now each engine maps the fields itself
could it come back as a protobuf message before the next PRs build on it? Java and Rust already generate the Fluss protos
There was a problem hiding this comment.
Added a shared Protobuf contract with versioning. Java capture remains a follow-up in fluss-client.
| /// Uses the public Fluss schema JSON, not Rust's internal Schema serde. | ||
| /// No arbitrary catalog/table property map is allowed in this descriptor. | ||
| #[derive(Debug, Clone, Serialize, Deserialize)] | ||
| #[serde(deny_unknown_fields)] |
There was a problem hiding this comment.
Rejecting unknown fields and accepting exactly one version means any new field breaks existing readers. The side that prepares the context and the side that reads it would always have to upgrade together, which is hard during rolling upgrades.
A more forgiving rule would be to ignore unknown optional fields, and to bump the version (or list a required feature) only when a change alters meaning. Also, the strictness is uneven today: unknown fields inside table_bucket are accepted.
There was a problem hiding this comment.
Readers now ignore unknown optional fields while rejecting missing required fields, unsupported versions and unknown semantic variants. Additions are compatible only when ignoring them preserves the read semantics; incompatible changes require a version bump, with readers upgraded before producers use the new format.
| if self.table_path() != &table.table_path | ||
| || self.table_id() != table.table_id | ||
| || self.lake_table_path() != &crate::table::resolve_lake_table_path(table)? | ||
| || self.schema_id() != table.schema_id |
There was a problem hiding this comment.
validate_table fails if a column was added after prepare(), but I don't think a schema change can affect a frozen read. The frozen data already mixes schemas: if a column was added before prepare(), older log rows and older lake files don't have it. So readers have to map each batch to one schema anyway. The Rust scanner does that by column id, and Paimon reads a snapshot with that snapshot's own schema. If readers use the frozen schema rather than the live one, a later added column changes nothing in the frozen read. The same is true when the column is added while prepare() is running, since the schema is read before the offsets.
Could validate_table check only that it's still the same table (table id, lake table), and leave the schema to the frozen one? A schema id is never reused, so comparing the schema JSON isn't needed either.
There was a problem hiding this comment.
Could we avoid rejecting the context when columns are added after prepare()? The read should keep using the schema captured during planning, with data mapped to it by column ID.
There was a problem hiding this comment.
I've removed the live-schema equality check from validate_table() and kept the source-identity and lake-mapping checks. Readers must use the captured schema rather than the latest one; the default execution path shown in #5000 passes that schema to the core scanner for column-ID-based mapping.
There was a problem hiding this comment.
Agreed. The context now remains valid when columns are added, and its captured schema stays unchanged. The reader contract requires using that schema for column-ID-based mapping, rather than reopening the read against the latest schema.
|
|
||
| // Source preparation needs only Fluss: no lake reader, tiering job or S3 fixture. | ||
| #[tokio::test] | ||
| async fn prepare_freezes_table_wide_ranges_without_a_lake_backend() { |
There was a problem hiding this comment.
This is the only end-to-end test, and no workflow runs it, so it could break without anyone noticing. Could it run in client-integration.yml? A partitioned-table case would also help, since most of the logic is in the partition path.
There was a problem hiding this comment.
I've wired test_prepare into the existing rust-integration job in client-integration.yml. The coverage includes partitioned tables with different per-partition bucket counts and scoped capture.
| Ok::<_, FlussLakeError>((id, identity, count, offsets)) | ||
| }, | ||
| )) | ||
| .buffered(8); |
There was a problem hiding this comment.
prepare() always looks up every partition, so a scan that needs one partition still pays for all of them. Table-wide makes sense as the default, since two references to the same table in one query should share one cut.
An optional partition filter would let a single scan avoid the cost, wdyt?
There was a problem hiding this comment.
Could we apply safe partition pruning before fetching log offsets? For a single scan, freezing only the selected splits should be sufficient and would avoid querying unrelated partitions.
There was a problem hiding this comment.
capture_read_context() remains table-wide so multiple scans can share the same boundaries. capture_read_context_with_partition_scope(&PartitionSpec) selects partitions before fetching offsets, with the scope carried in the context and applied to both lake and log inputs.
There was a problem hiding this comment.
This PR now supports explicit partition scope before fetching offsets. Deriving that scope from scan predicates belongs to the subsequent planner implementation, illustrated in #5000: it extracts safe STRING partition equalities, including AND conjuncts, and keeps a broader scope when a predicate cannot be safely converted.
There was a problem hiding this comment.
thanks, this covers equalities. since the context stores the scope as an exact spec, the planner can't narrow dt >= '2026-11-01' later without changing the context
can it be a fluss Predicate on partition columns from the start? the planner can still pass only equalities for now
There was a problem hiding this comment.
Changed the scope to core Predicate, evaluated before offset queries.
| use serde::{Deserialize, Serialize}; | ||
|
|
||
| /// Stable partition identity exposed for scheduling and diagnostics. | ||
| #[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] |
There was a problem hiding this comment.
If the format stays JSON, the Rust variant names ("Unpartitioned", "KeyValues") become part of it, so renaming the enum would change the format.
Maybe set explicit names, e.g. #[serde(rename_all = "snake_case")]?
There was a problem hiding this comment.
I've added explicit serde names for the partition and baseline variants, so renaming a Rust variant will not change the wire format.
| use serde::{Deserialize, Serialize}; | ||
| use std::collections::{HashMap, HashSet}; | ||
|
|
||
| const CONTEXT_VERSION: u32 = 3; |
There was a problem hiding this comment.
+1
Why does the context version start at 3? Could we also define a compatibility and upgrade strategy? Currently, rejecting unknown fields means adding an optional field can break older readers. Compatible additions should ideally work during rolling upgrades, with version bumps reserved for incompatible changes.
There was a problem hiding this comment.
The initial format is now v1; the previous number came from unpublished PoC iterations. Readers now ignore unknown optional fields when doing so preserves read semantics, while rejecting missing required fields and unknown semantic variants. Incompatible changes require a version bump, with readers upgraded before producers emit the new format. During rolling upgrades, producers must continue using the older format until all readers support the new one.
|
Given the roadmap in #4713, could we open the API/interface PR first, with an end-to-end usage example? Seeing how |
|
We may also need to consider historical partitions here. Could we add a TODO comment or open a follow-up issue to track this? |
I've opened draft #5000 as a reference for the full API and usage examples, including ReadContext → Scan → Plan → Split → Reader. It includes supporting implementation beyond this PR's scope to show the complete flow; this PR remains focused on boundary capture and transport. Does this give you enough context to assess the interfaces? |
|
Opened #5001 to track shared historical-log reads. The current design covers expired partitions retained in the lake snapshot, while historical logs still need their own boundary, pruning and reconciliation rules. |
|
@fresh-borzoni @beryllw, thank you for the helpful review! The latest commit addresses your review comments. Draft #5000 provides the full API and usage examples as a reference beyond this PR's scope. I'd appreciate your thoughts on the updated boundary semantics and API design. PTAL! |
|
The latest commit removes the JSON context format in favor of structured Java and Rust objects, leaving serialization and distribution to engine adapters. It also clarifies the PK scope and deferred KV snapshot fallback. @fresh-borzoni, regarding your question about the Java side, I have been reconsidering whether we need a separate API here. StarRocks already uses |
|
I've removed the new Java boundary API from this PR to keep it focused on Rust. Java coordinators can obtain the readable lake snapshot and matching bucket offsets through If a dedicated Java client API would simplify integration without a Flink connector dependency, we can introduce it in a follow-up PR. |
There was a problem hiding this comment.
@naivedogger Thank you, and sorry for the long wait, lgtm overall, couple of minor comments
| .any(|(_, count, buckets)| buckets.len() != *count as usize) | ||
| { | ||
| return Err(invalid_context( | ||
| "context must cover every bucket of each captured live partition", |
There was a problem hiding this comment.
in StarRocks or Doris each reader task gets one bucket, but from_boundary wants every bucket of the partition, so each task would have to carry all of them
can a single range be imported and checked on its own? capture itself can stay table-wide
There was a problem hiding this comment.
Added single-bucket import and validation. Table-wide capture is unchanged.
| /// | ||
| /// Runtime credentials and scan configuration are supplied separately. | ||
| /// Retention availability must still be checked for the required ranges. | ||
| pub fn from_boundary( |
There was a problem hiding this comment.
a StarRocks or Doris backend only gets ids and offsets from the frontend, not the TableInfo from planning time, so it can't call from_boundary as documented
can it take the current TableInfo, check the table id, and load the planned schema with get_table_schema by schema id?
There was a problem hiding this comment.
Import now resolves the captured schema by ID and checks table identity, preserving the original snapshot and offsets.
| # } | ||
| ``` | ||
|
|
||
| The context freezes table/schema identity, physical lake table mapping, a readable snapshot and half-open log ranges `[start, stop)`. Scan configuration, including projection and filters, belongs to the subsequent planning layer; preparation always captures table-wide boundaries. Actual per-partition bucket counts are preserved. The context contains no credentials or physical lake-file tasks, is not a global transactional snapshot and does not acquire a retention lease. Its JSON representation is an evolving Rust contract, not a stable cross-language ABI. |
There was a problem hiding this comment.
+1 to the fluss-client capture as a follow-up, without the Flink connector
what I'm missing now is the format: v1 was cross-language, now each engine maps the fields itself
could it come back as a protobuf message before the next PRs build on it? Java and Rust already generate the Fluss protos
| Ok::<_, FlussLakeError>((id, identity, count, offsets)) | ||
| }, | ||
| )) | ||
| .buffered(8); |
There was a problem hiding this comment.
thanks, this covers equalities. since the context stores the scope as an exact spec, the planner can't narrow dt >= '2026-11-01' later without changing the context
can it be a fluss Predicate on partition columns from the start? the planner can still pass only equalities for now
|
@fresh-borzoni, thank you for the helpful review! I've addressed these comments in the latest commit. Java capture remains a follow-up. PTAL. |
9ac6496 to
cecd5fc
Compare
|
@leonardBang Thanks for the review. I have pushed another commit to reject UnionRead tables with shared historical logs until it is supported. |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, LGTM 👍
|
Hi @fresh-borzoni , @naivedogger , If I’m reading this correctly, the FE and BE exchange this message through the engine’s own task protocol. Was |
|
@wuchong Yes, that’s the intended use case. There’s no planned Fluss client–server RPC use. |
|
@wuchong yes, I asked for protobuf, so Java and Rust generate the same message from what they already build, it only goes between the engine's FE and BE, never over Fluss RPC. @naivedogger #5164 adds a public Java planner and this PR makes the capture/import message the cross-language contract, neither is in FIP-48 yet |
|
@fresh-borzoni Good point. I will update the FIP and post it on the mail thread in a bit. |
|
Thanks @naivedogger @fresh-borzoni for the explanation. I think this looks good. |
Closes #4886