Skip to content

[core] Preserve and protect generated-column metadata - #1447

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1442
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1442

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: #1442

Loading a Java schema discarded generationExpression. Rewriting that schema could erase generated-column definitions and permit changes that invalidate stored values. Preserve this metadata throughout Rust schema handling and reject incompatible schema changes before publishing a new version.

Brief change log

  • Preserve optional generationExpression in DataField, field copies, schema builders, Arrow conversion, read/write projections and Map/Variant transformations. Expose stored expressions through the Python DataField wrapper and retain them during DataFusion Variant pushdown.
  • Validate generated columns and their dependencies by stable field ID after nullability normalization. Reject structural changes to protected fields and new generated columns after table creation. When dependencies cannot be resolved, conservatively reject renames, drops and type changes of existing columns while allowing compatible metadata changes.
  • Enforce Java's generated-column constraints during CREATE and ALTER: reject nested generated columns, generated columns with defaults and generated columns used as rowkind.field.
  • Reuse write-time default parsing to validate retained defaults before publishing type changes. For example, narrowing BIGINT DEFAULT 2147483648 to INT now fails without publishing an unusable schema.

Tests

  • Share a Java-compatible schema fixture across schema, catalog and Python tests. Cover JSON round-trips, field copies, builder ID assignment, Arrow metadata and Python access.
  • Cover dependency protection, unknown expression syntax, nullability normalization, nested generated-column constraints and invalid default/type combinations. Catalog regressions verify rejected changes leave the schema version unchanged and the table writable.
  • Extend existing projection, Map/Variant, key-value writer and DataFusion tests for metadata preservation. Verify readers return explicitly stored generated values, including NULL, without recomputing them.

Local validation passed: 212 schema, generated-column, default-parser and filesystem catalog tests; the expanded primary-key metadata test; cargo +1.98.0 fmt --all -- --check; and cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings.

API and Format

Add Rust DataField::generation_expression(), with_generation_expression(...), with_data_type(...) and SchemaBuilder::field(...), plus Python DataField.generation_expression(). Schema JSON retains Java's optional generationExpression; absent and null values serialize without the property. Arrow fields carry expressions under paimon:generationExpression. Data file layouts are unchanged.

Documentation

Document the metadata representation, APIs, schema-evolution rules and Java-compatible constraints in docs/src/getting-started.md. Writers must supply generated values explicitly. Expression evaluation and SQL GENERATED ALWAYS AS authoring remain unsupported.

🤖 Assisted with Codex.

@SteNicholas
SteNicholas force-pushed the PAIMON-1442 branch 2 times, most recently from c15fca1 to 110389f Compare November 18, 2026 05:56
Comment thread crates/paimon/src/spec/schema.rs Resolved
Comment thread crates/paimon/src/spec/schema.rs Resolved
Preserve Java's optional generationExpression in schema JSON and through
field copies, schema builders, Arrow conversion and read/write
projections. Expose stored expressions in the Python DataField wrapper.

Validate generated-column evolution against final fields by stable ID,
after nullability normalization. Reject changes to generated columns or
their dependencies and new generated columns after table creation.
Reject structural changes conservatively when dependencies cannot be
resolved.

Apply generated-column constraints during CREATE and ALTER: reject nested
generated columns, generated columns with defaults and generated columns
used as rowkind.field. Reuse write-time default parsing to validate
retained defaults before publishing type changes.

Cover metadata round-trips, rejected changes without schema publication,
and reads of stored generated values, including NULL. Document that
writers must supply generated values; expression evaluation and SQL
GENERATED ALWAYS AS authoring remain unsupported.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants