Repository navigation
[client] Capture bounded UnionRead boundaries - #5164
Conversation
e11c666 to
f2e917f
Compare
|
cc @fresh-borzoni @beryllw, this adds Java-side boundary capture for bounded UnionRead, building on #4887. |
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, left some comments, PTAL
| "Scope must reference partition keys with non-null encoded values."); | ||
| validateSupported(table); | ||
| Predicate predicate = | ||
| PredicateBuilder.partition(capturedScope.getSpecMap(), table.getRowType()); |
There was a problem hiding this comment.
same partition name format issue as in #5158, here on the Java side: the spec holds Fluss partition names, but PredicateBuilder.partition parses them with castFromString, so a DOUBLE partition 1_5 throws, for example
can we use PartitionUtils.parseValueOfType, the same parser the server uses, and test a non-string partition key here?
There was a problem hiding this comment.
Switched to PartitionUtils.parseValueOfType, so the scope uses Fluss partition encoding rather than generic string casting.
| context.addLogRange() | ||
| .setBucketId(bucketId) | ||
| .setBucketCount(count) | ||
| .setBaseline(seam == null ? 0 : 1) |
There was a problem hiding this comment.
the FIP has the Java helper select KV snapshot baselines too, with the same rules as Rust, and #5158 already does it there
is it planned here or with the KV follow-up?
There was a problem hiding this comment.
Added here: each PK bucket uses a lake baseline, otherwise a persisted KV snapshot, otherwise complete logs. KV snapshots are selected before the stop offsets, using the shared contract from #5158; execution support remains in the KV follow-up.
| * inconsistent | ||
| */ | ||
| public static PbUnionReadContext captureReadContext( | ||
| Admin admin, TablePath tablePath, PartitionSpec scope) throws Exception { |
There was a problem hiding this comment.
can there also be an overload with a Predicate on partition columns, like the Rust capture has since #4887?
StarRocks and Doris prune partitions themselves, and capturing one partition per call gives each partition its own lake snapshot
There was a problem hiding this comment.
Added a partition-column Predicate overload. It prunes partitions before querying offsets and shares one readable lake snapshot across the capture.
| .thenComparingLong(PartitionInfo::getPartitionId)); | ||
| } | ||
| List<PartitionCapture> captures = new ArrayList<>(); | ||
| for (PartitionInfo partition : selected) { |
There was a problem hiding this comment.
offsets are fetched one partition at a time, earliest then latest, and each listOffsets call also refreshes metadata and waits for the table info before sending
the Rust capture runs 8 partitions at a time and asks for earliest and latest together, can we do something similar here?
There was a problem hiding this comment.
Capture now runs up to eight partitions concurrently and issues earliest/latest before waiting. The public API stays synchronous; Admin's blocking metadata refresh runs on planning threads.
| public final class UnionReadBoundaryPlanner { | ||
|
|
||
| private static final int CONTEXT_VERSION = 1; | ||
| private static final int COVERAGE_COMPLETE_PARTITIONS = 0; |
There was a problem hiding this comment.
StarRocks and Doris would send each BE one bucket, can we add the one-bucket helper here too, like for_bucket in Rust?
then engines don't have to build coverage 1 by hand
There was a problem hiding this comment.
Added forBucket(context, bucket), selecting by table/partition/bucket identity and setting single-bucket coverage while preserving the frozen bounds and scope.
| bucket); | ||
| checkArgument( | ||
| !table.hasPrimaryKey() || seam != null || begin == 0, | ||
| "PK bucket %s has no baseline and its log begins at %s; KV snapshot fallback is required.", |
There was a problem hiding this comment.
this is missing data rather than a bad argument, Rust returns DataUnavailable here, maybe a dedicated exception so engines can tell them apart?
There was a problem hiding this comment.
Changed this to UnionReadDataUnavailableException, so missing baseline/history can be distinguished from invalid input.
|
@fresh-borzoni, thank you for the helpful review! I've addressed your comments in f0b5d73, including KV baseline capture and the engine-facing helpers. PTAL. |
|
@naivedogger Can you fix CI, pls? |
f0b5d73 to
a09d8a9
Compare
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, lgtm overall, additional comments, PTAL
| checkArgument( | ||
| captured.getTableId() == live.getTableId() | ||
| && captured.getTablePath().equals(live.getTablePath()), | ||
| "Table identity changed during boundary capture."); |
There was a problem hiding this comment.
how does an engine tell this from a bad argument? A table recreated during capture and a scope on a non-partition column both throw IllegalArgumentException
can we throw UnionReadDataUnavailableException here, so the engine knows it can retry?
There was a problem hiding this comment.
Changed table identity and lake mapping changes to UnionReadDataUnavailableException, so callers can distinguish an invalidated capture from invalid input.
| checkArgument( | ||
| snapshots.getTableId() == table.getTableId() | ||
| && Objects.equals(snapshots.getPartitionId(), partitionId), | ||
| "KV snapshots belong to a different table or partition."); |
There was a problem hiding this comment.
same here, this is the table changing during capture, not a bad argument
There was a problem hiding this comment.
Changed KV snapshot identity mismatches to UnionReadDataUnavailableException too. The partition ID recheck still excludes deleted or recreated partitions before propagating errors.
| Math.min(CAPTURE_CONCURRENCY, partitions.size()), | ||
| new ExecutorThreadFactory("union-read-boundary")); | ||
| try { | ||
| for (int first = 0; first < partitions.size(); first += CAPTURE_CONCURRENCY) { |
There was a problem hiding this comment.
nit: each batch of 8 waits for its slowest partition, one slow partition in 8 makes capture slower here
how about submitting all partitions to the pool at once, so a free thread takes the next one?
There was a problem hiding this comment.
All partitions are now submitted to the fixed-size pool upfront, so a free thread takes the next task. Concurrency remains capped at eight.
| * @return an independent context with single-bucket coverage | ||
| * @throws IllegalArgumentException if the bucket is outside the captured context | ||
| */ | ||
| public static PbUnionReadContext forBucket(PbUnionReadContext context, TableBucket bucket) { |
There was a problem hiding this comment.
what about partitions that fluss already dropped by retention, but the lake still has? For instance fluss keeps 7 days and paimon keeps a year: the context has ranges only for the 7 days, and each forBucket context reads only its own bucket, so nobody reads the older days
can we add a helper that returns one more context for them: same lake snapshot, every partition in scope except the ones that have a range here?
There was a problem hiding this comment.
Added forRemainingLakePartitions(context, table): it keeps the same snapshot and original scope, excluding the business partitions already covered by log ranges. It returns a lake-only context using the existing partition predicate and wire contract.
|
@fresh-borzoni, thank you for the helpful review! I have addressed your comments in the latest commit. PTAL. |
e40fb23 to
6e0921b
Compare
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, LGTM 👍
added couple of fixes:
- remaining lake partitions now use the captured schema, otherwise a column added after capture breaks it
- partition scope is checked without arrow and once per lake partition, with many live partitions it was the slowest part of planning
Will merge once CI is green
Closes #5154