Skip to content

[core] Wait for end-input maintenance queued after earlier maintenance - #11181

Open
dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:end-input-commit-sync-expire
Open

dragonknightk wants to merge 2 commits into
apache:masterfrom
dragonknightk:end-input-commit-sync-expire

Conversation

@dragonknightk

@dragonknightk dragonknightk commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes issue 11180

With snapshot.expire.execution-mode = async, the end-input commit now queues its maintenance after the maintenance of earlier commits and waits for it before returning. The end-input commit is the last commit of a bounded input. It uses BatchWriteBuilder.COMMIT_IDENTIFIER, and close() follows it. Before this change, the end-input maintenance was left on the async executor, and close() calls shutdownNow(), which interrupts a running expiration or drops one that has not started.

  • The end-input maintenance goes to the same single-thread executor instead of running on the committing thread. The maintenance managers keep state between runs, such as the tag data files cached by ChangelogDeletion and the next snapshot of TagAutoCreation. An end-input maintenance that overlaps an earlier one corrupts this state. Changelog expiration can then delete a data file that a retained tag still needs, and an automatic tag can be skipped.
  • This covers Flink bounded jobs, whose end-input commit uses COMMIT_IDENTIFIER, and the Flink remove-unexisting-files action. The Spark compact procedure from issue 5681 commits through commit(List), which runs maintenance on the caller thread since PR 6593. That path is unchanged.
  • close() is unchanged. Issue 5681 proposed shutdown() instead of shutdownNow(). The maintain thread is a daemon thread, so queued maintenance can still be lost when the process exits after close() returns. Waiting in close() would also block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs.
  • Checkpoint commits of streaming jobs keep the async mode. An expiration that close() interrupts there is picked up by a later expiration.
  • When an earlier maintenance is slow, the end-input commit returns only after it finishes. Sync mode is unchanged.

Tests

  • testEndInputCommitExpiresBeforeClose in SimpleTableTestBase, run by AppendOnlySimpleTableTest, PrimaryKeySimpleTableTest, and MockRESTSimpleTableTest: checkpoint commits keep expiring asynchronously, and after the end-input commit and close() only the latest snapshot remains. It fails on the code before this pull request.
  • TableCommitTest#testEndInputMaintenanceKeepsDataFilesOfRetainedTag: with decoupled changelog retention, an end-input commit made while a checkpoint maintenance expires changelogs keeps a retained tag readable.
  • TableCommitTest#testEndInputMaintenanceCreatesEveryAutomaticTag: with watermark automatic tags, an end-input commit made while a checkpoint maintenance creates tags still gets one tag per snapshot, the same as maintenance that runs one at a time.
  • Both TableCommitTest cases fail when the end-input maintenance runs on the committing thread.

AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

(๑•̀ㅂ•́)و✧

… input

### Purpose

Closes issue 11180

With `snapshot.expire.execution-mode = async`, `TableCommitImpl` now runs maintenance, including snapshot expiration, on the committing thread when the commit identifier is `BatchWriteBuilder.COMMIT_IDENTIFIER`. This identifier marks the last commit of a bounded input, and `close()` follows that commit. Before this change, the maintenance of that commit was queued on the async executor, and `close()` calls `shutdownNow()`, which interrupts a running expiration or drops one that has not started.

- The Spark compact procedure from issue 5681 already expires synchronously since PR 6593, because it commits through `commit(List)`, which marks the commit as batch. A `COMMIT_IDENTIFIER` commit that does not go through `commit(List)` still ran maintenance asynchronously. This covers all Flink bounded commits ending with `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action.
- `close()` is unchanged. Issue 5681 proposed `shutdown()` instead of `shutdownNow()`, but the maintain thread is a daemon thread, so the queued maintenance can still be lost when the process exits after `close()` returns. Waiting for it in `close()` would block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs. Issue 5681 also proposed skipping expiration in the Spark compact procedure, and recommended an internally used IGNORE value of `snapshot.expire.execution-mode` for that. PR 6593 addressed that procedure differently, by expiring synchronously in batch commits.
- The last maintenance runs on the committing thread and may run concurrently with an async maintenance still in progress from the previous checkpoint. Snapshot expiration already tolerates concurrent runs, and tag creation uses `ignoreIfExists`, so this change does not submit that maintenance to the executor and wait.
- Checkpoint commits of streaming jobs keep the async mode. An expiration that `close()` interrupts there is picked up by a later expiration.
- The last commit now behaves as in sync mode: it returns after maintenance, and a maintenance failure is thrown to the caller instead of being logged.

### Tests

- With async mode and the maintain thread kept busy, two checkpoint commits keep the first snapshot, so they still expire asynchronously. The following end-input commit through `filterAndCommit` with `COMMIT_IDENTIFIER` leaves only the latest snapshot after `close()`. `testEndInputCommitExpiresBeforeClose` is in `SimpleTableTestBase` and runs in `AppendOnlySimpleTableTest`, `PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`. It fails on the code before this change.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

The bounded-input cleanup problem is real, but the final maintenance must retain per-committer serialization. All 28 existing commit/end-input tests passed on JDK 11 with normal Maven checks, and this head's CI is green. Two additional real-table concurrency regressions nevertheless reproduce retained-tag data-file deletion and skipped automatic tags, detailed together below because they share the same serialization break.

For the retained-tag case, the same test-only scheduling probe passes both cases on clean merge-base 9b6c997 and fails only the overlapping end-input case on this head. The failing read uses freshly reopened persisted tag metadata. The automatic-tag probe was repeated with normal Maven checks. No production sources were modified for these probes.

Comment on lines 409 to 410
if (batchCommitted || identifier == COMMIT_IDENTIFIER) {
maintain(identifier, doExpire);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Serialize final maintenance with already-running asynchronous maintenance

This direct call can enter the same maintenance managers as the previous checkpoint's executor task. With decoupled changelog retention, the shared ChangelogDeletion tag skipper closes over cachedTagDataFiles: a second pass can replace the retained-tag-1 cache with tag 3 while the first pass is between planning and filtering its deletions. The first pass then deletes a file still required by tag 1.

I reproduced this with six real primary-key snapshots (overwrites at 2 and 4), retained tags at 1 and 3, snapshot retention min/max 1 and changelog retention max 2. Holding the old pass after planning changelog 2, running the final COMMIT_IDENTIFIER pass, then resuming the old pass removes the protected data file. The tag remains registered; reloading its persisted metadata and reading it throws FileNotFoundException. Both cases pass on clean base 9b6c997198; the same-head serialized checkpoint control also retains and reads the file.

There is a second manifestation with watermark automatic tags: three real append commits persist tags for snapshots [1,3] under overlap, versus [1,2,3] in the serialized control, because both runs mutate the same TagAutoCreation.nextSnapshot. ignoreIfExists protects duplicate tag creation, not this progress state.

Please queue the final maintenance behind preceding maintenance and wait for completion, or otherwise serialize access to all of these shared managers. It still needs to finish before close, while preserving the existing ordering of maintenance passes.

### Purpose

Closes issue 11180

With `snapshot.expire.execution-mode = async`, the end-input commit now queues its maintenance after the maintenance of earlier commits and waits for it before returning. The end-input commit is the last commit of a bounded input. It uses `BatchWriteBuilder.COMMIT_IDENTIFIER`, and `close()` follows it. Before this change, the end-input maintenance was left on the async executor, and `close()` calls `shutdownNow()`, which interrupts a running expiration or drops one that has not started.

- The end-input maintenance goes to the same single-thread executor instead of running on the committing thread. The maintenance managers keep state between runs, such as the tag data files cached by `ChangelogDeletion` and the next snapshot of `TagAutoCreation`. An end-input maintenance that overlaps an earlier one corrupts this state. Changelog expiration can then delete a data file that a retained tag still needs, and an automatic tag can be skipped.
- This covers Flink bounded jobs, whose end-input commit uses `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action. The Spark compact procedure from issue 5681 commits through `commit(List)`, which runs maintenance on the caller thread since PR 6593. That path is unchanged.
- `close()` is unchanged. Issue 5681 proposed `shutdown()` instead of `shutdownNow()`. The maintain thread is a daemon thread, so queued maintenance can still be lost when the process exits after `close()` returns. Waiting in `close()` would also block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs.
- Checkpoint commits of streaming jobs keep the async mode. An expiration that `close()` interrupts there is picked up by a later expiration.
- When an earlier maintenance is slow, the end-input commit returns only after it finishes. Sync mode is unchanged.

### Tests

- `testEndInputCommitExpiresBeforeClose` in `SimpleTableTestBase`, run by `AppendOnlySimpleTableTest`, `PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`: checkpoint commits keep expiring asynchronously, and after the end-input commit and `close()` only the latest snapshot remains. It fails on the code before this pull request.
- `TableCommitTest#testEndInputMaintenanceKeepsDataFilesOfRetainedTag`: with decoupled changelog retention, an end-input commit made while a checkpoint maintenance expires changelogs keeps a retained tag readable.
- `TableCommitTest#testEndInputMaintenanceCreatesEveryAutomaticTag`: with watermark automatic tags, an end-input commit made while a checkpoint maintenance creates tags still gets one tag per snapshot, the same as maintenance that runs one at a time.
- Both `TableCommitTest` cases fail when the end-input maintenance runs on the committing thread.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
@dragonknightk dragonknightk changed the title [core] Run maintenance synchronously for the last commit of a bounded input [core] Wait for end-input maintenance queued after earlier maintenance Nov 18, 2026
@dragonknightk

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for the detailed repro, both cases were spot on. I went with your first suggestion: the end-input maintenance is now submitted to the same single-thread executor and the commit waits for it, so it runs after any earlier maintenance and finishes before close(). Checkpoint commits stay async, and the order of maintenance passes is unchanged.

I added two tests based on your scenarios, testEndInputMaintenanceKeepsDataFilesOfRetainedTag and testEndInputMaintenanceCreatesEveryAutomaticTag. Both fail when the end-input maintenance runs on the committing thread. The PR description is updated to match. PTAL ٩(ˊᗜˋ*)و

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants