Repository navigation
[core] Read legacy schemas with Java-compatible defaults - #1388
suxiaogang223 wants to merge 2 commits into
Conversation
Fixes apache#1384. Apply legacy defaults at the shared serde boundary so historical schema files and resolved C table construction agree with Java.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at de93e11.
Java's legacy version/default boundaries and missing timeMillis handling match this deserializer. Local Rust 1.98.0 validation: 152 schema/schema-change tests, 9 schema-manager tests and 98 C-binding tests pass (259). An additional real filesystem-catalog probe reloads a versionless schema with legacy defaults, reads existing ORC data, appends with an explicitly supported ORC compression setting, and reads both rows successfully. Current-head CI is green and the branch merges cleanly with current main.
A separate public ALTER/RESET probe passes on exact base f91fdb8 and fails on this head because new schema versions retain an old format-version flag; details inline.
| if stored.version <= 2 { | ||
| options | ||
| .entry("file.format".into()) | ||
| .or_insert_with(|| "orc".into()); |
There was a problem hiding this comment.
[P2] Upgrade newly persisted schema versions before reapplying legacy defaults
TableSchema::apply_changes clones the stored version, and the filesystem catalog saves that result without promoting it to CURRENT_VERSION. With this new deserializer, a legal RESET can therefore change meaning only after serialization/reload.
I reproduced through the public catalog: an existing version=2 append table explicitly sets file.format=parquet; write one row, execute SchemaChange::remove_option("file.format"), reload through a fresh FileSystemCatalog, and append another row. On exact base f91fdb8 the reloaded schema remains Parquet and both writes succeed. On this head the new schema is still version 2, this branch reinserts ORC, and the next write fails ORC compression 'zstd' is not supported by the current writer. Even with a supported compression setting this silently switches output format instead of retaining the current Parquet default.
Java's corresponding ALTER path creates a current-version schema, so RESET does not reintroduce the old ORC default. Promote newly created persisted schema versions during evolution, while keeping already resolved legacy defaults stable and leaving read-only copies alone. Add a version-1/2 explicit-Parquet → RESET → fresh-catalog reload/write regression.
Purpose
Linked issue: close #1384
Legacy Java schema JSON can omit
version. Rust rejects it, and accepting the JSON alone would still select the wrong bucket/file-format defaults. Interpret stored schemas using the Java 2.1.0 version boundaries, covering filesystem history and catalog-free C construction. Stamp evolved schemas with the current format version so ALTER/RESET does not reintroduce legacy defaults on reload.Brief change log
TableSchema::apply_changesuseCURRENT_VERSIONfor the new schema ID. Keep previously resolved options, historical files and read-only copies stable.file.format, reload through a new catalog, append again and read both rows. Verify the new schema uses the current version and the original schema file remains byte-for-byte unchanged.Tests
cargo +1.95.0 test --locked -p paimon --lib legacy_schema_: 8 passed.cargo +1.95.0 test --locked -p paimon --lib: 4,866 passed, 6 ignored.cargo +1.95.0 test --locked -p paimon-c --lib: 98 passed.cargo +1.95.0 fmt --all -- --checkandgit diff --check: passed.-D warnings -A clippy::nonminimal_bool. Strict Clippy still reports the same six existing nonminimal-bool errors in unchanged vector-read/merge modules under Rust 1.95; no source lint suppressions were added.Local Cargo builds used debug info disabled and incremental compilation disabled. No new Java runtime fixture was executed; compatibility rules were checked against Java 2.1.0 SchemaSerializer, SchemaManagerUtils and TableSchema construction. Full workspace, Vortex and cross-platform execution are left to CI.
API and Format
Existing public signatures, JSON fields and version constants are unchanged. Stored legacy schemas receive version-dependent defaults when loaded. Newly evolved schema IDs use the existing current format version, matching Java ALTER behavior. Already resolved defaults remain explicit options unless changed by the ALTER operation; historical schema files and read-only option copies retain their versions.
Documentation
Document deserialization and evolution behavior beside the implementation and in regression cases. No new configuration is needed.
AI assistance: Codex helped develop the implementation, tests, and this description.