Repository navigation
Conversation
jx2lee
marked this pull request as ready for review
October 30, 2026 00:09
jx2lee
force-pushed
the
jx2lee/feat/support-log-table-delta
branch
from
November 1, 2026 11:43
5c873fe to
f38ef5a
Compare
Add a fluss-lake-delta module that tiers Fluss log tables into Delta Lake tables using Delta Kernel 4.4.1. - Add DELTA to DataLakeFormat. Delta Lake has no hash bucketing spec, so it reuses FlussBucketingFunction and the compacted key encoder/decoder, the same as Lance. - DeltaLakeCatalog creates or validates Delta tables. Primary key tables are rejected, and only STRING partition keys are supported. - DeltaLakeWriter/DeltaLakeCommitter write Parquet files and commit them to the Delta log through Delta Kernel. - Add unit tests and DeltaTieringITCase
Remove the space after the Hudi entry in stage.sh. The unquoted output was split into words, so Maven treated fluss-lake-delta as a lifecycle phase and the core and lake stages failed before running tests. Stop bundling GPL-licensed classes in the shaded jar. Exclude hadoop-client-api and javax.annotation-api, and keep only the relocated Guava classes from hadoop-client-runtime, which Parquet needs at runtime. Add hadoop-mapreduce-client-core as a provided dependency. Add the NOTICE file and license texts for the bundled dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the Paimon and Iceberg catalogs: delta.* custom properties are passed to the Delta table as they are, every other Fluss property is written with the fluss. prefix. They are informational only, tiering never reads them back, but they show how the table is configured in Fluss. alterTable now mirrors set/reset of Fluss-side options in one metadata-only Delta commit instead of ignoring them. Changes are folded in order so a key set and reset in the same request is not written, and no commit is made when only unchanged delta.* properties are replayed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
getMissingLakeSnapshot walked back version by version from the latest snapshot to find the commit that set the Fluss domain metadata. Once log cleanup removed old commits, loading an older version threw and every tiering round failed. Check the version known to Fluss first: if it still holds the latest domain metadata, no Fluss commit came after it and the walk is skipped. When the walk reaches a version the log no longer holds, stop at the oldest version left. It holds the same offsets and can still be loaded. If the version known to Fluss is gone as well, the same offsets may be recorded again, which is harmless, while missing a commit would tier its data twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writers check the column names when they load the table, but another engine may change the Delta table, e.g. by ALTER TABLE ADD COLUMNS in Spark, between that and the commit of the round. The Kernel conflict check only covers changes after the committer loaded the snapshot, so the round would add files that don't match the table. Compare the schema and the partition columns of the Delta table with the Fluss table before committing and fail the round with a message that names each differing column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tiering doesn't compact Delta tables yet, so the option would be accepted and silently do nothing. Reject it when the Delta table is created and when the option is enabled later, and point to compacting the table with another engine. Disabling it stays allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writer and the committer passed every partition value to Kernel as a string literal, so only STRING partition keys could be tiered. Parse the values of the Fluss partition name with PartitionUtils and convert them to Kernel literals of the column type, which Kernel checks against the partition column and serializes as the Delta protocol defines. All Fluss partition key types are supported except BINARY and BYTES, whose partition values Delta stores as text, and TIME, which Delta lacks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move CloseableIterators to the utils package so that the lake source can wrap in-memory iterators too, and move the writer and committer contexts of DeltaTieringTest to DeltaTestUtils for the lake source tests. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a log table had a lake snapshot, Flink SELECT failed because the Delta plugin had no LakeSource. Add one, so that a union read takes the tiered rows from the Delta version recorded by Fluss and the rest from the Fluss log. The planner makes one split per data file with bucket -1. A split carries the Kernel scan state and scan file row as JSON, and the reader reads the file with Scan#transformPhysicalData, which adds the partition columns and applies deletion vectors written by other engines. Partition values are converted back to the format of Fluss partition names, so the splits match the Fluss partitions. Projection is pushed down, filters are not yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tier a log table to Delta, stop the tiering job and write more rows, then check that a Flink batch SELECT returns every row exactly once, partitioned and not, also with a projection that puts the partition column first. Without the Delta LakeSource the query fails as reported on the FIP-62 discussion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jx2lee
force-pushed
the
jx2lee/feat/support-log-table-delta
branch
from
November 17, 2026 02:12
f063bb8 to
5aab7b6
Compare
jx2lee
marked this pull request as draft
November 17, 2026 03:34
…ss data The lake source read the version that Fluss registered as its lake snapshot. When the Fluss table gets no new data, Fluss doesn't commit again, and other engines may OPTIMIZE the table and VACUUM the files it replaced. Reading the registered version then fails on deleted files. The planner now reads the latest version when its fluss.tiering domain metadata, schema and partition columns equal those of the registered version. Fluss changes the domain metadata on every commit, and other engines must not change the data of the table, so both versions hold the same rows. Otherwise it keeps reading the registered version. If log cleanup has removed the registered version, the planner can't compare and fails with a message naming delta.logRetentionDuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: part of #967. This is phase 1 of the plan proposed in comment. It does not close the issue.
Fluss can tier data to Paimon, Iceberg, Lance and Hudi, but not to Delta Lake. This PR adds a
fluss-lake-deltaplugin that tiers Fluss log tables into Delta tables. It uses only the public APIs of Delta Kernel 4.4.1, so any Delta reader (Spark, Trino, Databricks external tables) can read the tables.Primary key tables with union read (phase 2, which needs a FIP first) and catalog-managed tables such as Unity Catalog (phase 3) are left for later PRs.
Brief change log
DataLakeFormat.DELTA: Delta Lake has no hash bucketing spec, so Delta usesFlussBucketingFunctionand the compacted key encoder/decoder, the same as Lance.DeltaLakeCatalog: creates a Delta table withCreateTableTransactionBuilder, or checks that an existing table's schema matches. It rejects primary key tables, and supports onlySTRINGpartition keys.DeltaLakeWriter: converts Fluss rows to Kernel columnar batches (DeltaColumnarBatch,FlussRowToDeltaValues), then writes Parquet withtransformLogicalDataandgenerateAppendActions.DeltaLakeCommitter: commits the add actions together with the Fluss bucket offsets, stored as thefluss.tieringdomain metadata of the same commit, so data and offsets become visible atomically.getMissingLakeSnapshotreads that domain metadata back to recover a commit that reached Delta but not Fluss.FlussDataTypeToDeltaDataTypemaps Fluss types to Delta types.fluss-lake/fluss-lake-deltatoMODULES_LAKEin.github/workflows/stage.sh, so its tests run in the lake stage instead of core.${skip.on.java8}.Tests
DeltaTieringTest: write and commit for partitioned and non-partitioned log tables.DeltaColumnarBatchTestFlussDataTypeToDeltaDataTypeTestDeltaTieringITCaseruns the full Flink tiering job against a Fluss cluster, then reads the Delta table back../mvnw verify -pl fluss-lake/fluss-lake-delta(after./mvnw install -DskipTests -pl fluss-lake/fluss-lake-delta -am): BUILD SUCCESSDeltaTieringITCase2 tests, 0 failuresLakeStorageTest.testWithPluginManagerinfluss-commonfails locally on JDK 17 with a Mockito error. It fails the same way on the base commit without this PR, so it is unrelated.API and Format
Adds
deltaas a newdatalake.formatvalue. No existing API or storage format changes.Documentation
No documentation in this PR. User docs for Delta Lake (
website/docs/streaming-lakehouse/integrate-data-lakes/delta.md) will follow in a separate PR, once the module scope is agreed in #967.AI-assisted with Claude Code (Claude Opus 5.5). The author reviewed the design, code and tests.