Skip to content

[datafusion] Support persistent CREATE TABLE AS SELECT - #1448

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1425
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1425

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: #1425

Materializing a query in a persistent Paimon table currently requires a schema declaration and a separate INSERT. This adds native CREATE TABLE AS SELECT to the DataFusion SQL context, inferring and populating the table from the same resolved query while preserving table identity across concurrent DDL and retaining files when publication is uncertain.

Brief change log

  • Infer column names, supported types and nullability from the analyzed query, including empty results. Reuse CREATE TABLE validation for partition keys, options and comments. Reject explicit column declarations, unsupported modifiers and time outputs that would lose precision.
  • Resolve source tables and SQL functions in the session namespace independently of the destination. Preserve quoted identifiers and qualified time-travel references, execute the same resolved plan used for inference, and normalize scalar and nested Arrow output layouts for writing.
  • Skip existing native and external targets with IF NOT EXISTS before source planning or execution. Publish initial schemas atomically, bind writers to a unique table creation identity, and coordinate filesystem catalog DDL and snapshot publication with database lifecycle locks. Enforce the same identity check in the filesystem-backed REST server.
  • Retain the created table after population errors and preserve concurrent writers' rows. Clean up owned output after pre-commit failures and cancellation, finish commits already started, and retain prepared files after commit errors because publication may have succeeded.

Tests

  • Integration tests in crates/integrations/datafusion/tests/create_table_as.rs: inferred schemas, empty results, nullability, partitioning, options, nested Arrow layouts, time precision, quoted names, cross-catalog destinations, time travel and REST SQL functions. Verify existing-target no-ops and rejected statements preserve catalog and source data.
  • Execution tests in crates/integrations/datafusion/tests/create_table_as/execution.rs: reuse of one source plan, creation races, lookup errors, concurrent writes, dropped or replaced targets, source and write failures, failed or uncertain publication, and cancellation before and during commit. Check table retention, committed rows and cleanup of owned files.
  • Reuse read_files_behavior coverage for temporary and persistent CTAS. Extend filesystem catalog, FileIO and REST server tests for atomic creation, lifecycle locking and stale-writer rejection, and run existing SQLContext and SQL integration tests against the shared CREATE TABLE validation.

API and Format

Add Catalog::create_table_for_write, TableSchema::creation_id, FileSystemCatalog::commit_snapshot and Error::TableReplaced. Persist an internal internal.table-creation-id schema option for owned writes; data-file and snapshot encodings are unchanged. Filesystem table creation requires atomic initial-schema publication, and concurrent DDL clients must share the database locking protocol. REST catalogs must preserve creation identity and validate it during snapshot publication. CTAS supports native Paimon tables; OR REPLACE and external-engine CTAS remain unsupported.

Documentation

Update docs/src/sql.md and crates/integrations/datafusion/README.md with CTAS syntax, schema inference, supported time precision, namespace resolution and IF NOT EXISTS behavior. Document table retention, cancellation and uncertain commits, database lifecycle locking, and recovery of abandoned object-storage locks.

🤖 Assisted with Codex.

Materializing a query in Paimon requires a schema declaration and a
separate INSERT. A single statement must infer and populate a persistent
table from the same resolved query, preserve its identity across
concurrent DDL and retain files when publication is uncertain.

- Infer native CTAS schemas from analyzed queries, including empty
  results. Reuse CREATE TABLE validation for partition keys, options
  and comments, addressing apache#1425.
- Resolve sources and SQL functions in the session namespace, preserving
  quoted identifiers and qualified time-travel references. Execute the
  same plan and normalize scalar and nested output layouts for writing.
- Skip existing native and external targets with IF NOT EXISTS. Publish
  initial table schemas atomically and validate table identity during
  snapshot publication through filesystem and REST catalogs.
- Retain the table after population errors. Clean up owned output on
  pre-commit failures and cancellation, finish commits already started
  and preserve prepared files when publication may have succeeded.
let mut options: Vec<_> = schema
.options()
.iter()
.filter(|(key, _)| key.as_str() != "internal.table-creation-id")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants