Repository navigation
[datafusion] Support persistent CREATE TABLE AS SELECT - #1448
Open
SteNicholas wants to merge 1 commit into
Open
SteNicholas wants to merge 1 commit into
SteNicholas wants to merge 1 commit into
Conversation
Materializing a query in Paimon requires a schema declaration and a separate INSERT. A single statement must infer and populate a persistent table from the same resolved query, preserve its identity across concurrent DDL and retain files when publication is uncertain. - Infer native CTAS schemas from analyzed queries, including empty results. Reuse CREATE TABLE validation for partition keys, options and comments, addressing apache#1425. - Resolve sources and SQL functions in the session namespace, preserving quoted identifiers and qualified time-travel references. Execute the same plan and normalize scalar and nested output layouts for writing. - Skip existing native and external targets with IF NOT EXISTS. Publish initial table schemas atomically and validate table identity during snapshot publication through filesystem and REST catalogs. - Retain the table after population errors. Clean up owned output on pre-commit failures and cancellation, finish commits already started and preserve prepared files when publication may have succeeded.
JingsongLi
reviewed
Nov 19, 2026
| let mut options: Vec<_> = schema | ||
| .options() | ||
| .iter() | ||
| .filter(|(key, _)| key.as_str() != "internal.table-creation-id") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #1425
Materializing a query in a persistent Paimon table currently requires a schema declaration and a separate INSERT. This adds native
CREATE TABLE AS SELECTto the DataFusion SQL context, inferring and populating the table from the same resolved query while preserving table identity across concurrent DDL and retaining files when publication is uncertain.Brief change log
Tests
crates/integrations/datafusion/tests/create_table_as.rs: inferred schemas, empty results, nullability, partitioning, options, nested Arrow layouts, time precision, quoted names, cross-catalog destinations, time travel and REST SQL functions. Verify existing-target no-ops and rejected statements preserve catalog and source data.crates/integrations/datafusion/tests/create_table_as/execution.rs: reuse of one source plan, creation races, lookup errors, concurrent writes, dropped or replaced targets, source and write failures, failed or uncertain publication, and cancellation before and during commit. Check table retention, committed rows and cleanup of owned files.read_files_behaviorcoverage for temporary and persistent CTAS. Extend filesystem catalog, FileIO and REST server tests for atomic creation, lifecycle locking and stale-writer rejection, and run existing SQLContext and SQL integration tests against the shared CREATE TABLE validation.API and Format
Add
Catalog::create_table_for_write,TableSchema::creation_id,FileSystemCatalog::commit_snapshotandError::TableReplaced. Persist an internalinternal.table-creation-idschema option for owned writes; data-file and snapshot encodings are unchanged. Filesystem table creation requires atomic initial-schema publication, and concurrent DDL clients must share the database locking protocol. REST catalogs must preserve creation identity and validate it during snapshot publication. CTAS supports native Paimon tables; OR REPLACE and external-engine CTAS remain unsupported.Documentation
Update
docs/src/sql.mdandcrates/integrations/datafusion/README.mdwith CTAS syntax, schema inference, supported time precision, namespace resolution and IF NOT EXISTS behavior. Document table retention, cancellation and uncertain commits, database lifecycle locking, and recovery of abandoned object-storage locks.🤖 Assisted with Codex.