Skip to content

[lake/delta] Support tiering log tables to Delta Lake - #4944

Draft
jx2lee wants to merge 13 commits into
apache:mainfrom
jx2lee:jx2lee/feat/support-log-table-delta
Draft

jx2lee wants to merge 13 commits into
apache:mainfrom
jx2lee:jx2lee/feat/support-log-table-delta

Conversation

@jx2lee

@jx2lee jx2lee commented Oct 30, 2026 •

Copy link
Copy Markdown

Purpose

Linked issue: part of #967. This is phase 1 of the plan proposed in comment. It does not close the issue.

Fluss can tier data to Paimon, Iceberg, Lance and Hudi, but not to Delta Lake. This PR adds a fluss-lake-delta plugin that tiers Fluss log tables into Delta tables. It uses only the public APIs of Delta Kernel 4.4.1, so any Delta reader (Spark, Trino, Databricks external tables) can read the tables.

Primary key tables with union read (phase 2, which needs a FIP first) and catalog-managed tables such as Unity Catalog (phase 3) are left for later PRs.

Brief change log

  • DataLakeFormat.DELTA: Delta Lake has no hash bucketing spec, so Delta uses FlussBucketingFunction and the compacted key encoder/decoder, the same as Lance.
  • DeltaLakeCatalog: creates a Delta table with CreateTableTransactionBuilder, or checks that an existing table's schema matches. It rejects primary key tables, and supports only STRING partition keys.
  • DeltaLakeWriter: converts Fluss rows to Kernel columnar batches (DeltaColumnarBatch, FlussRowToDeltaValues), then writes Parquet with transformLogicalData and generateAppendActions.
  • DeltaLakeCommitter: commits the add actions together with the Fluss bucket offsets, stored as the fluss.tiering domain metadata of the same commit, so data and offsets become visible atomically. getMissingLakeSnapshot reads that domain metadata back to recover a commit that reached Delta but not Fluss.
  • Type mapping: FlussDataTypeToDeltaDataType maps Fluss types to Delta types.
  • CI: adds fluss-lake/fluss-lake-delta to MODULES_LAKE in .github/workflows/stage.sh, so its tests run in the lake stage instead of core.
  • Java 8: like other modules that need Java 11+, compile and tests are skipped with ${skip.on.java8}.

Tests

  • Unit tests:
    • DeltaTieringTest: write and commit for partitioned and non-partitioned log tables.
    • DeltaColumnarBatchTest
    • FlussDataTypeToDeltaDataTypeTest
  • IT case: DeltaTieringITCase runs the full Flink tiering job against a Fluss cluster, then reads the Delta table back.
  • ./mvnw verify -pl fluss-lake/fluss-lake-delta (after ./mvnw install -DskipTests -pl fluss-lake/fluss-lake-delta -am): BUILD SUCCESS
    • Unit: 12 tests, 0 failures
    • IT: DeltaTieringITCase 2 tests, 0 failures
  • LakeStorageTest.testWithPluginManager in fluss-common fails locally on JDK 17 with a Mockito error. It fails the same way on the base commit without this PR, so it is unrelated.

API and Format

Adds delta as a new datalake.format value. No existing API or storage format changes.

Documentation

No documentation in this PR. User docs for Delta Lake (website/docs/streaming-lakehouse/integrate-data-lakes/delta.md) will follow in a separate PR, once the module scope is agreed in #967.


AI-assisted with Claude Code (Claude Opus 5.5). The author reviewed the design, code and tests.

@jx2lee
jx2lee marked this pull request as ready for review October 30, 2026 00:09
@jx2lee
jx2lee force-pushed the jx2lee/feat/support-log-table-delta branch from 5c873fe to f38ef5a Compare November 1, 2026 11:43
jx2lee and others added 12 commits November 17, 2026 10:01
Add a fluss-lake-delta module that tiers Fluss log tables into Delta
Lake tables using Delta Kernel 4.4.1.

- Add DELTA to DataLakeFormat. Delta Lake has no hash bucketing spec,
  so it reuses FlussBucketingFunction and the compacted key
  encoder/decoder, the same as Lance.
- DeltaLakeCatalog creates or validates Delta tables. Primary key
  tables are rejected, and only STRING partition keys are supported.
- DeltaLakeWriter/DeltaLakeCommitter write Parquet files and commit
  them to the Delta log through Delta Kernel.
- Add unit tests and DeltaTieringITCase
Remove the space after the Hudi entry in stage.sh. The unquoted output
was split into words, so Maven treated fluss-lake-delta as a lifecycle
phase and the core and lake stages failed before running tests.

Stop bundling GPL-licensed classes in the shaded jar. Exclude
hadoop-client-api and javax.annotation-api, and keep only the relocated
Guava classes from hadoop-client-runtime, which Parquet needs at runtime.
Add hadoop-mapreduce-client-core as a provided dependency.

Add the NOTICE file and license texts for the bundled dependencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the Paimon and Iceberg catalogs: delta.* custom properties are
passed to the Delta table as they are, every other Fluss property is
written with the fluss. prefix. They are informational only, tiering
never reads them back, but they show how the table is configured in
Fluss.

alterTable now mirrors set/reset of Fluss-side options in one
metadata-only Delta commit instead of ignoring them. Changes are folded
in order so a key set and reset in the same request is not written, and
no commit is made when only unchanged delta.* properties are replayed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
getMissingLakeSnapshot walked back version by version from the latest
snapshot to find the commit that set the Fluss domain metadata. Once log
cleanup removed old commits, loading an older version threw and every
tiering round failed.

Check the version known to Fluss first: if it still holds the latest
domain metadata, no Fluss commit came after it and the walk is skipped.
When the walk reaches a version the log no longer holds, stop at the
oldest version left. It holds the same offsets and can still be loaded.
If the version known to Fluss is gone as well, the same offsets may be
recorded again, which is harmless, while missing a commit would tier its
data twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writers check the column names when they load the table, but
another engine may change the Delta table, e.g. by ALTER TABLE ADD
COLUMNS in Spark, between that and the commit of the round. The Kernel
conflict check only covers changes after the committer loaded the
snapshot, so the round would add files that don't match the table.

Compare the schema and the partition columns of the Delta table with
the Fluss table before committing and fail the round with a message
that names each differing column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tiering doesn't compact Delta tables yet, so the option would be
accepted and silently do nothing. Reject it when the Delta table is
created and when the option is enabled later, and point to compacting
the table with another engine. Disabling it stays allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writer and the committer passed every partition value to Kernel as
a string literal, so only STRING partition keys could be tiered. Parse
the values of the Fluss partition name with PartitionUtils and convert
them to Kernel literals of the column type, which Kernel checks against
the partition column and serializes as the Delta protocol defines.

All Fluss partition key types are supported except BINARY and BYTES,
whose partition values Delta stores as text, and TIME, which Delta
lacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move CloseableIterators to the utils package so that the lake source
can wrap in-memory iterators too, and move the writer and committer
contexts of DeltaTieringTest to DeltaTestUtils for the lake source
tests. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a log table had a lake snapshot, Flink SELECT failed because the
Delta plugin had no LakeSource. Add one, so that a union read takes the
tiered rows from the Delta version recorded by Fluss and the rest from
the Fluss log.

The planner makes one split per data file with bucket -1. A split
carries the Kernel scan state and scan file row as JSON, and the reader
reads the file with Scan#transformPhysicalData, which adds the
partition columns and applies deletion vectors written by other
engines. Partition values are converted back to the format of Fluss
partition names, so the splits match the Fluss partitions. Projection
is pushed down, filters are not yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tier a log table to Delta, stop the tiering job and write more rows,
then check that a Flink batch SELECT returns every row exactly once,
partitioned and not, also with a projection that puts the partition
column first. Without the Delta LakeSource the query fails as reported
on the FIP-62 discussion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jx2lee
jx2lee force-pushed the jx2lee/feat/support-log-table-delta branch from f063bb8 to 5aab7b6 Compare November 17, 2026 02:12
@jx2lee
jx2lee marked this pull request as draft November 17, 2026 03:34
…ss data

The lake source read the version that Fluss registered as its lake
snapshot. When the Fluss table gets no new data, Fluss doesn't commit
again, and other engines may OPTIMIZE the table and VACUUM the files
it replaced. Reading the registered version then fails on deleted files.

The planner now reads the latest version when its fluss.tiering domain
metadata, schema and partition columns equal those of the registered
version. Fluss changes the domain metadata on every commit, and other
engines must not change the data of the table, so both versions hold
the same rows. Otherwise it keeps reading the registered version.

If log cleanup has removed the registered version, the planner can't
compare and fails with a message naming delta.logRetentionDuration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant