Skip to content

[core] Add a Java-compatible R-tree global index for GEOMETRY - #1457

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/rtree-index
Nov 18, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/rtree-index

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Add a Rust reader and writer for the packed R-tree global index introduced in apache/paimon#11238. Tables using the native planar GEOMETRY type can build the same V1 files and use their envelopes for filtering.

Brief change log

  • Stream WKB from snapshot-pinned table shards into Hilbert-packed files. Each physical file records its own row count and uses shard-relative row IDs. Manual and automatic builds handle multi-file shards, including cleanup after a partial build fails.
  • Use the manifest envelope for file pruning and read selected pages or row-ID blocks on demand. The reader bounds the page cache, coalesces adjacent reads and shares a file-concurrency limit across queries using the same read context.
  • Add the bbox_intersects(shape, min_x, min_y, max_x, max_y) DataFusion function and push constant bounds through the native predicate API. The rectangle includes its boundary. The index also answers IS NULL and IS NOT NULL, including the distinction between null and empty geometry. FAST searches indexed coverage. FULL and DETAIL also evaluate uncovered rows.

Tests

  • Rust core library suite passed with 4,977 tests and 7 ignored. An initial concurrent run timed out in the existing disk-cache child-process test. The full suite passed on rerun with four test threads.
  • DataFusion library suite passed with 448 tests. The global-index schema-evolution test and both spatial integration tests passed, including the failed-build cleanup case.
  • Compared both Java V1 fixtures byte for byte. Both implementations read the fixtures and reproduce the mixed multilevel file. The Java R-tree suites passed with 54 tests. Rust also covers noncontiguous child pages, sparse 64-bit row IDs, invalid headers and checksums, bounded parallel reads, and partial index coverage.
  • Rustfmt and strict Clippy passed for the core and DataFusion crates.

API and Format

The Rust table builder and DataFusion's create/drop global index procedures accept rtree. The native filter is PredicateBuilder::bbox_intersects. The index operates on planar XY envelopes in the column's coordinate system. Both engines use the existing R-tree V1 file format.

Documentation

Updated the SQL and automatic maintenance guides with usage, search modes, build settings, and read limits.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 637e6c0 into apache:main Nov 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants