Skip to content

[core] Add a packed R-tree global index for GEOMETRY - #11238

Merged
JingsongLi merged 17 commits into
apache:masterfrom
JingsongLi:codex/java-rtree-global-index
Nov 18, 2026
Merged

JingsongLi merged 17 commits into
apache:masterfrom
JingsongLi:codex/java-rtree-global-index

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add a packed R-tree global index for bounding-box queries on a single GEOMETRY column in row-tracked Data Evolution append tables. Java callers can build an rtree index and use PredicateBuilder.bboxIntersects to find envelopes that overlap or touch a query rectangle. Empty geometries remain available to non-null queries.

The writer packs Hilbert-ordered envelopes into checksummed V1 pages. The reader prunes files and subtrees, coalesces adjacent page reads and searches up to four files concurrently by default. Queries and the byte-bounded cache share the backing array of each contiguous read. The format documentation and golden fixtures cover file layout, row IDs, nulls and empty geometries.

The R-tree serves bounding-box and null predicates. Geometry equality stays on the data filter. For bbox queries, fast mode returns R-tree matches. Full and detail modes also read the uncovered row-ID ranges. A mixed-column spatial disjunction in fast mode is evaluated by the planner so fallback is decided for the whole query.

Vector and full-text top-k queries refine candidates when an R-tree only covers part of a geometry filter. For example, a bbox hit can still fail a geometry equality condition on the same column.

A Flink build shard can produce several R-tree files. Each file reports its own source row count. Comparing the first file's count with the whole shard produced a spurious warning. The build keeps the shard input statistics and each file's row count.

The Paimon extension for Spark 4.1 and 4.2 exposes ST_Intersects(left, right) for planar geometries. For a positive filter on an indexed column and a constant geometry, the scan pushes the envelope into the R-tree and Spark evaluates exact intersection on the candidates. Both argument orders are supported. Null inputs yield null and incompatible SRIDs raise an error.

Tests

  • Ran 47 Spark 4.1 tests and 17 Spark 4.2 tests. Ranked SQL cases cover positive boolean wrappers and geometries whose SRID becomes known during constant folding. The indexed SQL test checks a false-positive envelope, boundary contact, nulls, negation, mismatched SRIDs, and the results of fast and full scans after an unindexed append.
  • Ran 388 Java and Flink tests covering the index reader and writer, predicate handling, Data Evolution, ranked spatial filtering, per-file row counts, and the Flink builder.
  • Verified that the Java writer reproduced the frozen V1 fixture byte for byte, and that Java and Rust could read each other's fixture.
  • Ran the Java benchmark against JTS HPRtree and STRtree on uniform points, clustered boxes and elongated boxes. Ran the Java and Rust file readers on the same 1,048,576 uniform points. Each worker compared complete query result bitmaps with a brute-force scan.
  • Built the documentation site and ran the root Apache RAT check.

During development, at sixteen concurrent files with 20 ms added per physical read, sharing page buffers reduced the median sampled Java query peak from 435.1 to 364.6 MiB. Large-window warm queries took 105.47 and 105.66 ms respectively. This workload used 16 files, an 8 MiB cache, one query client and three independent Corretto 8 JVMs per configuration.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 79b8ec4 into apache:master Nov 18, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants