Repository navigation
[core] Add a packed R-tree global index for GEOMETRY - #11238
Merged
JingsongLi merged 17 commits intoNov 18, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add a packed R-tree global index for bounding-box queries on a single
GEOMETRYcolumn in row-tracked Data Evolution append tables. Java callers can build anrtreeindex and usePredicateBuilder.bboxIntersectsto find envelopes that overlap or touch a query rectangle. Empty geometries remain available to non-null queries.The writer packs Hilbert-ordered envelopes into checksummed V1 pages. The reader prunes files and subtrees, coalesces adjacent page reads and searches up to four files concurrently by default. Queries and the byte-bounded cache share the backing array of each contiguous read. The format documentation and golden fixtures cover file layout, row IDs, nulls and empty geometries.
The R-tree serves bounding-box and null predicates. Geometry equality stays on the data filter. For bbox queries, fast mode returns R-tree matches. Full and detail modes also read the uncovered row-ID ranges. A mixed-column spatial disjunction in fast mode is evaluated by the planner so fallback is decided for the whole query.
Vector and full-text top-k queries refine candidates when an R-tree only covers part of a geometry filter. For example, a bbox hit can still fail a geometry equality condition on the same column.
A Flink build shard can produce several R-tree files. Each file reports its own source row count. Comparing the first file's count with the whole shard produced a spurious warning. The build keeps the shard input statistics and each file's row count.
The Paimon extension for Spark 4.1 and 4.2 exposes
ST_Intersects(left, right)for planar geometries. For a positive filter on an indexed column and a constant geometry, the scan pushes the envelope into the R-tree and Spark evaluates exact intersection on the candidates. Both argument orders are supported. Null inputs yield null and incompatible SRIDs raise an error.Tests
During development, at sixteen concurrent files with 20 ms added per physical read, sharing page buffers reduced the median sampled Java query peak from 435.1 to 364.6 MiB. Large-window warm queries took 105.47 and 105.66 ms respectively. This workload used 16 files, an 8 MiB cache, one query client and three independent Corretto 8 JVMs per configuration.