Skip to content

[client] Capture bounded UnionRead boundaries - #5164

Merged
fresh-borzoni merged 2 commits into
apache:mainfrom
naivedogger:fip48/java-read-boundaries
Nov 18, 2026
Merged

fresh-borzoni merged 2 commits into
apache:mainfrom
naivedogger:fip48/java-read-boundaries

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Closes #5154

@naivedogger
naivedogger marked this pull request as draft November 12, 2026 01:45
@naivedogger
naivedogger force-pushed the fip48/java-read-boundaries branch from e11c666 to f2e917f Compare November 13, 2026 06:09
@naivedogger naivedogger changed the title [client] Capture reusable UnionRead boundaries [client] Capture bounded UnionRead boundaries Nov 13, 2026
@naivedogger
naivedogger marked this pull request as ready for review November 13, 2026 07:09
@naivedogger

Copy link
Copy Markdown
Contributor Author

cc @fresh-borzoni @beryllw, this adds Java-side boundary capture for bounded UnionRead, building on #4887.
Would appreciate your review!

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, left some comments, PTAL

"Scope must reference partition keys with non-null encoded values.");
validateSupported(table);
Predicate predicate =
PredicateBuilder.partition(capturedScope.getSpecMap(), table.getRowType());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same partition name format issue as in #5158, here on the Java side: the spec holds Fluss partition names, but PredicateBuilder.partition parses them with castFromString, so a DOUBLE partition 1_5 throws, for example
can we use PartitionUtils.parseValueOfType, the same parser the server uses, and test a non-string partition key here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to PartitionUtils.parseValueOfType, so the scope uses Fluss partition encoding rather than generic string casting.

context.addLogRange()
.setBucketId(bucketId)
.setBucketCount(count)
.setBaseline(seam == null ? 0 : 1)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the FIP has the Java helper select KV snapshot baselines too, with the same rules as Rust, and #5158 already does it there
is it planned here or with the KV follow-up?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added here: each PK bucket uses a lake baseline, otherwise a persisted KV snapshot, otherwise complete logs. KV snapshots are selected before the stop offsets, using the shared contract from #5158; execution support remains in the KV follow-up.

* inconsistent
*/
public static PbUnionReadContext captureReadContext(
Admin admin, TablePath tablePath, PartitionSpec scope) throws Exception {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can there also be an overload with a Predicate on partition columns, like the Rust capture has since #4887?
StarRocks and Doris prune partitions themselves, and capturing one partition per call gives each partition its own lake snapshot

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a partition-column Predicate overload. It prunes partitions before querying offsets and shares one readable lake snapshot across the capture.

.thenComparingLong(PartitionInfo::getPartitionId));
}
List<PartitionCapture> captures = new ArrayList<>();
for (PartitionInfo partition : selected) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

offsets are fetched one partition at a time, earliest then latest, and each listOffsets call also refreshes metadata and waits for the table info before sending
the Rust capture runs 8 partitions at a time and asks for earliest and latest together, can we do something similar here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capture now runs up to eight partitions concurrently and issues earliest/latest before waiting. The public API stays synchronous; Admin's blocking metadata refresh runs on planning threads.

public final class UnionReadBoundaryPlanner {

private static final int CONTEXT_VERSION = 1;
private static final int COVERAGE_COMPLETE_PARTITIONS = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StarRocks and Doris would send each BE one bucket, can we add the one-bucket helper here too, like for_bucket in Rust?
then engines don't have to build coverage 1 by hand

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added forBucket(context, bucket), selecting by table/partition/bucket identity and setting single-bucket coverage while preserving the frozen bounds and scope.

bucket);
checkArgument(
!table.hasPrimaryKey() || seam != null || begin == 0,
"PK bucket %s has no baseline and its log begins at %s; KV snapshot fallback is required.",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is missing data rather than a bad argument, Rust returns DataUnavailable here, maybe a dedicated exception so engines can tell them apart?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this to UnionReadDataUnavailableException, so missing baseline/history can be distinguished from invalid input.

@naivedogger

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thank you for the helpful review! I've addressed your comments in f0b5d73, including KV baseline capture and the engine-facing helpers. PTAL.

@fresh-borzoni

Copy link
Copy Markdown
Member

@naivedogger Can you fix CI, pls?

@naivedogger
naivedogger force-pushed the fip48/java-read-boundaries branch from f0b5d73 to a09d8a9 Compare November 16, 2026 15:27

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, lgtm overall, additional comments, PTAL

checkArgument(
captured.getTableId() == live.getTableId()
&& captured.getTablePath().equals(live.getTablePath()),
"Table identity changed during boundary capture.");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does an engine tell this from a bad argument? A table recreated during capture and a scope on a non-partition column both throw IllegalArgumentException
can we throw UnionReadDataUnavailableException here, so the engine knows it can retry?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed table identity and lake mapping changes to UnionReadDataUnavailableException, so callers can distinguish an invalidated capture from invalid input.

checkArgument(
snapshots.getTableId() == table.getTableId()
&& Objects.equals(snapshots.getPartitionId(), partitionId),
"KV snapshots belong to a different table or partition.");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, this is the table changing during capture, not a bad argument

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed KV snapshot identity mismatches to UnionReadDataUnavailableException too. The partition ID recheck still excludes deleted or recreated partitions before propagating errors.

Math.min(CAPTURE_CONCURRENCY, partitions.size()),
new ExecutorThreadFactory("union-read-boundary"));
try {
for (int first = 0; first < partitions.size(); first += CAPTURE_CONCURRENCY) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: each batch of 8 waits for its slowest partition, one slow partition in 8 makes capture slower here
how about submitting all partitions to the pool at once, so a free thread takes the next one?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All partitions are now submitted to the fixed-size pool upfront, so a free thread takes the next task. Concurrency remains capped at eight.

* @return an independent context with single-bucket coverage
* @throws IllegalArgumentException if the bucket is outside the captured context
*/
public static PbUnionReadContext forBucket(PbUnionReadContext context, TableBucket bucket) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about partitions that fluss already dropped by retention, but the lake still has? For instance fluss keeps 7 days and paimon keeps a year: the context has ranges only for the 7 days, and each forBucket context reads only its own bucket, so nobody reads the older days

can we add a helper that returns one more context for them: same lake snapshot, every partition in scope except the ones that have a range here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added forRemainingLakePartitions(context, table): it keeps the same snapshot and original scope, excluding the business partitions already covered by log ranges. It returns a lake-only context using the existing partition predicate and wire contract.

@naivedogger

naivedogger commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni, thank you for the helpful review! I have addressed your comments in the latest commit. PTAL.

@fresh-borzoni
fresh-borzoni force-pushed the fip48/java-read-boundaries branch from e40fb23 to 6e0921b Compare November 17, 2026 23:37

@fresh-borzoni fresh-borzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naivedogger Thank you, LGTM 👍

added couple of fixes:

  • remaining lake partitions now use the captured schema, otherwise a column added after capture breaks it
  • partition scope is checked without arrow and once per lake partition, with many live partitions it was the slowest part of planning

Will merge once CI is green

@fresh-borzoni
fresh-borzoni merged commit d37e6c9 into apache:main Nov 18, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[client] Capture reusable UnionRead boundaries

2 participants