Skip to content

[rust] Add reusable UnionRead source boundaries - #4887

Merged
fresh-borzoni merged 9 commits into
apache:mainfrom
naivedogger:fip48/02-read-context
Nov 11, 2026
Merged

fresh-borzoni merged 9 commits into
apache:mainfrom
naivedogger:fip48/02-read-context

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Closes #4886

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni @beryllw Could you please take a look at this one when you got a chance? Thanks!🙏

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you for the PR, a couple of comments and questions, PTAL

|(id, name, identity, count)| async move {
let bucket_ids: Vec<i32> = (0..count).collect();
let offsets =
load_server_offsets(admin, table_path, name.as_deref(), &bucket_ids).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partitions are listed first, and their offsets are looked up by name afterwards. If a partition is dropped in between, the whole prepare() fails. Auto-partition retention drops partitions as a matter of course, so long-running services will hit this.

A partition that disappears mid-prepare can be treated as if it was dropped before we started: its lake data is still read from the snapshot, and its log is gone either way. A drop can surface as different errors depending on timing, not only PartitionNotExists. So when a lookup fails, it may be simplest to re-check the partition list by id, skip the partition only if it's really gone, and otherwise return the original error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated boundary capture to recheck partition IDs after the offset lookups. A partition's log ranges are omitted only when its original ID is confirmed absent, including after same-name recreation. Other lookup failures still propagate, and lake data remains covered by the pinned snapshot.

// Without a lake baseline the complete history is required, not merely
// the retained suffix. Availability is checked after scan-specific pruning
// so an irrelevant range cannot break a lake-only or pruned read.
let start_offset = snapshot_offset.unwrap_or(0);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a bucket with no lake offset, the range starts at 0. For log tables that works, since the context also records the earliest offset, so a caller can choose to fail or read what's left. For PK tables it doesn't. If the changelog start was cleaned up, the current state can't be rebuilt from the log, although Fluss still has it in the KV snapshot.

Could a range say what sits under it (lake offset, KV snapshot, or nothing), even if only the lake case is implemented now? The KV snapshot would need to be captured here, before the stop offsets, so the snapshot and the log after it stay consistent. Adding it later would change the format. WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each range now explicitly identifies its baseline as lake or none, including the distinction between a lake seam at zero and no baseline. Without a baseline, reads start at the captured earliest offset, so PK results only cover keys reconstructible from retained logs. KV snapshot fallback remains future work, and readers reject unknown baseline kinds.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for a PK table without a lake snapshot, once the log start is cleaned up (earliest > 0) reading only the log silently loses keys, main reads KV snapshot + log here
can we fail in this case until the KV snapshot fallback is in? I'll review it shortly, whatever you prefer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PK reads without a baseline now fail when earliest > 0, pending KV snapshot fallback.

Comment thread fluss-rust/crates/fluss-lake/README.md Outdated
# }
```

The context freezes table/schema identity, physical lake table mapping, a readable snapshot and half-open log ranges `[start, stop)`. Scan configuration, including projection and filters, belongs to the subsequent planning layer; preparation always captures table-wide boundaries. Actual per-partition bucket counts are preserved. The context contains no credentials or physical lake-file tasks, is not a global transactional snapshot and does not acquire a retention lease. Its JSON representation is an evolving Rust contract, not a stable cross-language ABI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Engines that plan in Java (Trino, Spark, StarRocks, Doris) can't call prepare(), and the README says not to treat the JSON as cross-language. How do you see those engines getting a context? If a Java producer is planned, it's worth settling the format before later PRs build on it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added UnionReadBoundaryPlanner.captureReadContext(admin, tablePath), with an overload for partition scope, to capture the boundaries and return the encoded context in one call. Context v1 is now a documented cross-language format, so Java coordinators can produce it without calling Rust or assembling the JSON themselves.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to the fluss-client capture as a follow-up, without the Flink connector
what I'm missing now is the format: v1 was cross-language, now each engine maps the fields itself
could it come back as a protobuf message before the next PRs build on it? Java and Rust already generate the Fluss protos

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a shared Protobuf contract with versioning. Java capture remains a follow-up in fluss-client.

/// Uses the public Fluss schema JSON, not Rust's internal Schema serde.
/// No arbitrary catalog/table property map is allowed in this descriptor.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rejecting unknown fields and accepting exactly one version means any new field breaks existing readers. The side that prepares the context and the side that reads it would always have to upgrade together, which is hard during rolling upgrades.

A more forgiving rule would be to ignore unknown optional fields, and to bump the version (or list a required feature) only when a change alters meaning. Also, the strictness is uneven today: unknown fields inside table_bucket are accepted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readers now ignore unknown optional fields while rejecting missing required fields, unsupported versions and unknown semantic variants. Additions are compatible only when ignoring them preserves the read semantics; incompatible changes require a version bump, with readers upgraded before producers use the new format.

if self.table_path() != &table.table_path
|| self.table_id() != table.table_id
|| self.lake_table_path() != &crate::table::resolve_lake_table_path(table)?
|| self.schema_id() != table.schema_id

@fresh-borzoni fresh-borzoni Oct 27, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate_table fails if a column was added after prepare(), but I don't think a schema change can affect a frozen read. The frozen data already mixes schemas: if a column was added before prepare(), older log rows and older lake files don't have it. So readers have to map each batch to one schema anyway. The Rust scanner does that by column id, and Paimon reads a snapshot with that snapshot's own schema. If readers use the frozen schema rather than the live one, a later added column changes nothing in the frozen read. The same is true when the column is added while prepare() is running, since the schema is read before the offsets.

Could validate_table check only that it's still the same table (table id, lake table), and leave the schema to the frozen one? A schema id is never reused, so comparing the schema JSON isn't needed either.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid rejecting the context when columns are added after prepare()? The read should keep using the schema captured during planning, with data mapped to it by column ID.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the live-schema equality check from validate_table() and kept the source-identity and lake-mapping checks. Readers must use the captured schema rather than the latest one; the default execution path shown in #5000 passes that schema to the core scanner for column-ID-based mapping.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The context now remains valid when columns are added, and its captured schema stays unchanged. The reader contract requires using that schema for column-ID-based mapping, rather than reopening the read against the latest schema.


// Source preparation needs only Fluss: no lake reader, tiering job or S3 fixture.
#[tokio::test]
async fn prepare_freezes_table_wide_ranges_without_a_lake_backend() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only end-to-end test, and no workflow runs it, so it could break without anyone noticing. Could it run in client-integration.yml? A partitioned-table case would also help, since most of the logic is in the partition path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've wired test_prepare into the existing rust-integration job in client-integration.yml. The coverage includes partitioned tables with different per-partition bucket counts and scoped capture.

Ok::<_, FlussLakeError>((id, identity, count, offsets))
},
))
.buffered(8);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prepare() always looks up every partition, so a scan that needs one partition still pays for all of them. Table-wide makes sense as the default, since two references to the same table in one query should share one cut.
An optional partition filter would let a single scan avoid the cost, wdyt?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we apply safe partition pruning before fetching log offsets? For a single scan, freezing only the selected splits should be sufficient and would avoid querying unrelated partitions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

capture_read_context() remains table-wide so multiple scans can share the same boundaries. capture_read_context_with_partition_scope(&PartitionSpec) selects partitions before fetching offsets, with the scope carried in the context and applied to both lake and log inputs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR now supports explicit partition scope before fetching offsets. Deriving that scope from scan predicates belongs to the subsequent planner implementation, illustrated in #5000: it extracts safe STRING partition equalities, including AND conjuncts, and keeps a broader scope when a predicate cannot be safely converted.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, this covers equalities. since the context stores the scope as an exact spec, the planner can't narrow dt >= '2026-11-01' later without changing the context
can it be a fluss Predicate on partition columns from the start? the planner can still pass only equalities for now

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the scope to core Predicate, evaluated before offset queries.

use serde::{Deserialize, Serialize};

/// Stable partition identity exposed for scheduling and diagnostics.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the format stays JSON, the Rust variant names ("Unpartitioned", "KeyValues") become part of it, so renaming the enum would change the format.
Maybe set explicit names, e.g. #[serde(rename_all = "snake_case")]?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added explicit serde names for the partition and baseline variants, so renaming a Rust variant will not change the wire format.

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more Q

use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};

const CONTEXT_VERSION: u32 = 3;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why 3?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Why does the context version start at 3? Could we also define a compatibility and upgrade strategy? Currently, rejecting unknown fields means adding an optional field can break older readers. Compatible additions should ideally work during rolling upgrades, with version bumps reserved for incompatible changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial format is now v1; the previous number came from unpublished PoC iterations. Readers now ignore unknown optional fields when doing so preserves read semantics, while rejecting missing required fields and unknown semantic variants. Incompatible changes require a version bump, with readers upgraded before producers emit the new format. During rolling upgrades, producers must continue using the older format until all readers support the new one.

@beryllw
beryllw self-requested a review October 28, 2026 08:33
@beryllw

beryllw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Given the roadmap in #4713, could we open the API/interface PR first, with an end-to-end usage example? Seeing how ReadContext fits into Scan → Plan → Split → Reader would give us the full picture before reviewing the implementation details.

@beryllw

beryllw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

We may also need to consider historical partitions here. Could we add a TODO comment or open a follow-up issue to track this?

@naivedogger

Copy link
Copy Markdown
Contributor Author

Could we open the API/interface PR first, with an end-to-end usage example?

I've opened draft #5000 as a reference for the full API and usage examples, including ReadContext → Scan → Plan → Split → Reader. It includes supporting implementation beyond this PR's scope to show the complete flow; this PR remains focused on boundary capture and transport. Does this give you enough context to assess the interfaces?

@naivedogger

Copy link
Copy Markdown
Contributor Author

We may also need to consider historical partitions here. Could we add a TODO comment or open a follow-up issue to track this?

Opened #5001 to track shared historical-log reads. The current design covers expired partitions retained in the lake snapshot, while historical logs still need their own boundary, pruning and reconciliation rules.

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni @beryllw, thank you for the helpful review! The latest commit addresses your review comments. Draft #5000 provides the full API and usage examples as a reference beyond this PR's scope. I'd appreciate your thoughts on the updated boundary semantics and API design. PTAL!

@naivedogger

Copy link
Copy Markdown
Contributor Author

The latest commit removes the JSON context format in favor of structured Java and Rust objects, leaving serialization and distribution to engine adapters. It also clarifies the PK scope and deferred KV snapshot fallback.

@fresh-borzoni, regarding your question about the Java side, I have been reconsidering whether we need a separate API here. StarRocks already uses LakeSplitGenerator, which exposes lake tasks and log ranges that could also be adapted for native execution. Do you think a separate boundary-capture API is needed now, or could we reuse the existing planner initially?

@leonardBang
leonardBang self-requested a review November 3, 2026 13:44
@naivedogger

Copy link
Copy Markdown
Contributor Author

I've removed the new Java boundary API from this PR to keep it focused on Rust. Java coordinators can obtain the readable lake snapshot and matching bucket offsets through Admin.getReadableLakeSnapshot(), then capture log bounds with listOffsets(), as the existing LakeSplitGenerator path does. An engine adapter can map these captured inputs into the Rust read context.

If a dedicated Java client API would simplify integration without a Flink connector dependency, we can introduce it in a follow-up PR.

@fresh-borzoni fresh-borzoni left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, and sorry for the long wait, lgtm overall, couple of minor comments

.any(|(_, count, buckets)| buckets.len() != *count as usize)
{
return Err(invalid_context(
"context must cover every bucket of each captured live partition",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in StarRocks or Doris each reader task gets one bucket, but from_boundary wants every bucket of the partition, so each task would have to carry all of them
can a single range be imported and checked on its own? capture itself can stay table-wide

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added single-bucket import and validation. Table-wide capture is unchanged.

///
/// Runtime credentials and scan configuration are supplied separately.
/// Retention availability must still be checked for the required ranges.
pub fn from_boundary(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a StarRocks or Doris backend only gets ids and offsets from the frontend, not the TableInfo from planning time, so it can't call from_boundary as documented
can it take the current TableInfo, check the table id, and load the planned schema with get_table_schema by schema id?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import now resolves the captured schema by ID and checks table identity, preserving the original snapshot and offsets.

Comment thread fluss-rust/crates/fluss-lake/README.md Outdated
# }
```

The context freezes table/schema identity, physical lake table mapping, a readable snapshot and half-open log ranges `[start, stop)`. Scan configuration, including projection and filters, belongs to the subsequent planning layer; preparation always captures table-wide boundaries. Actual per-partition bucket counts are preserved. The context contains no credentials or physical lake-file tasks, is not a global transactional snapshot and does not acquire a retention lease. Its JSON representation is an evolving Rust contract, not a stable cross-language ABI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to the fluss-client capture as a follow-up, without the Flink connector
what I'm missing now is the format: v1 was cross-language, now each engine maps the fields itself
could it come back as a protobuf message before the next PRs build on it? Java and Rust already generate the Fluss protos

Ok::<_, FlussLakeError>((id, identity, count, offsets))
},
))
.buffered(8);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, this covers equalities. since the context stores the scope as an exact spec, the planner can't narrow dt >= '2026-11-01' later without changing the context
can it be a fluss Predicate on partition columns from the start? the planner can still pass only equalities for now

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thank you for the helpful review! I've addressed these comments in the latest commit. Java capture remains a follow-up. PTAL.

@naivedogger
naivedogger force-pushed the fip48/02-read-context branch from 9ac6496 to cecd5fc Compare November 11, 2026 08:23
@naivedogger

Copy link
Copy Markdown
Contributor Author

@leonardBang Thanks for the review. I have pushed another commit to reject UnionRead tables with shared historical logs until it is supported.

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, LGTM 👍

@leonardBang leonardBang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from my side

@wuchong

wuchong commented Nov 12, 2026

Copy link
Copy Markdown
Member

Hi @fresh-borzoni , @naivedogger , FlussApi.proto now defines PbUnionReadContext and the related lake and log-range messages, but I don’t see a Fluss server RPC that takes or returns them. Is the intended use case an engine integration like Doris or StarRocks, where a Java FE uses the Fluss Java SDK to capture the Union Read boundaries and sends them to a C++ BE that uses the Fluss Rust SDK to execute the read?

If I’m reading this correctly, the FE and BE exchange this message through the engine’s own task protocol. Was FlussApi.proto chosen so the Java and Rust SDKs can generate types from the same definition, or is there also a planned use for it in a Fluss client–server RPC?

@naivedogger

Copy link
Copy Markdown
Contributor Author

@wuchong Yes, that’s the intended use case. There’s no planned Fluss client–server RPC use.

@fresh-borzoni

fresh-borzoni commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

@wuchong yes, I asked for protobuf, so Java and Rust generate the same message from what they already build, it only goes between the engine's FE and BE, never over Fluss RPC.
would you prefer it in a separate proto file?

@naivedogger #5164 adds a public Java planner and this PR makes the capture/import message the cross-language contract, neither is in FIP-48 yet
can we update the FIP and post it on the FIP-48 mail thread to let community know?

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni Good point. I will update the FIP and post it on the mail thread in a bit.

@wuchong

wuchong commented Nov 13, 2026

Copy link
Copy Markdown
Member

Thanks @naivedogger @fresh-borzoni for the explanation. I think this looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Add reusable UnionRead source boundaries

5 participants