Skip to content

[core] Support Java-compatible GEOMETRY and GEOGRAPHY types - #1446

Merged
JingsongLi merged 6 commits into
apache:mainfrom
JingsongLi:codex/java-geospatial-types
Nov 18, 2026
Merged

JingsongLi merged 6 commits into
apache:mainfrom
JingsongLi:codex/java-geospatial-types

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Support Java Paimon's GEOMETRY and GEOGRAPHY types in Rust and DataFusion. Rust can read Java-written Parquet files, append rows, and create tables with the same logical types. Java implementation

Changes

  • Add both schema types with Java's default CRS and five geography edge algorithms. Expose them as Arrow binary fields with GeoArrow WKB metadata. Preserve the declared CRS, edge algorithm, and Java field IDs in Parquet, including nested columns. Arrow extension metadata stores the CRS as a string, matching Java, and Rust also reads PROJJSON objects from other GeoArrow producers.
  • Copy WKB bytes on assignment. The destination column declares how to interpret the coordinates. Applications that need reprojection perform it before writing. DataFusion handles SQL comparisons using its existing binary semantics.
  • Handle geo types in DataFusion DDL, casts, and views. Restore the list element metadata for WKB arrays materialized by INSERT VALUES.

Verification

  • Ran the Rust core library suite (4,957 passed, 7 ignored) and 587 DataFusion library and integration tests, including cross-CRS WKB assignment on copy-on-write and data-evolution tables. The schema tests also cover geospatial columns in clustering.columns and its alias. VECTOR rejects both GEOMETRY and GEOGRAPHY as element types.
  • Verified a Java → Rust → Java Parquet round trip. Java wrote a geometry, a geography using Karney edges, and a geometry array with a null element. Rust read the row and appended a second one. Java verified both IDs, the bytes, and the nested nulls. Inspection of both Parquet footers and the Arrow schema embedded by Rust confirmed that the original PROJJSON text, including its surrounding whitespace, survived the round trip.
  • Workspace all-target checks, strict Clippy for the core and DataFusion crates, and formatting passed.

API and documentation

The Rust schema API exposes GeometryType, GeographyType, and EdgeAlgorithm. Use Parquet to exchange geospatial data with Java. The Rust Row format also supports WKB. SQL data types

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reproducible geospatial metadata preservation issue is detailed inline.

Comment thread crates/paimon/src/arrow/mod.rs Resolved
@JingsongLi
JingsongLi merged commit c552065 into apache:main Nov 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants