Repository navigation
[core] Wait for end-input maintenance queued after earlier maintenance - #11181
dragonknightk wants to merge 2 commits into
Conversation
… input ### Purpose Closes issue 11180 With `snapshot.expire.execution-mode = async`, `TableCommitImpl` now runs maintenance, including snapshot expiration, on the committing thread when the commit identifier is `BatchWriteBuilder.COMMIT_IDENTIFIER`. This identifier marks the last commit of a bounded input, and `close()` follows that commit. Before this change, the maintenance of that commit was queued on the async executor, and `close()` calls `shutdownNow()`, which interrupts a running expiration or drops one that has not started. - The Spark compact procedure from issue 5681 already expires synchronously since PR 6593, because it commits through `commit(List)`, which marks the commit as batch. A `COMMIT_IDENTIFIER` commit that does not go through `commit(List)` still ran maintenance asynchronously. This covers all Flink bounded commits ending with `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action. - `close()` is unchanged. Issue 5681 proposed `shutdown()` instead of `shutdownNow()`, but the maintain thread is a daemon thread, so the queued maintenance can still be lost when the process exits after `close()` returns. Waiting for it in `close()` would block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs. Issue 5681 also proposed skipping expiration in the Spark compact procedure, and recommended an internally used IGNORE value of `snapshot.expire.execution-mode` for that. PR 6593 addressed that procedure differently, by expiring synchronously in batch commits. - The last maintenance runs on the committing thread and may run concurrently with an async maintenance still in progress from the previous checkpoint. Snapshot expiration already tolerates concurrent runs, and tag creation uses `ignoreIfExists`, so this change does not submit that maintenance to the executor and wait. - Checkpoint commits of streaming jobs keep the async mode. An expiration that `close()` interrupts there is picked up by a later expiration. - The last commit now behaves as in sync mode: it returns after maintenance, and a maintenance failure is thrown to the caller instead of being logged. ### Tests - With async mode and the maintain thread kept busy, two checkpoint commits keep the first snapshot, so they still expire asynchronously. The following end-input commit through `filterAndCommit` with `COMMIT_IDENTIFIER` leaves only the latest snapshot after `close()`. `testEndInputCommitExpiresBeforeClose` is in `SimpleTableTestBase` and runs in `AppendOnlySimpleTableTest`, `PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`. It fails on the code before this change. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The bounded-input cleanup problem is real, but the final maintenance must retain per-committer serialization. All 28 existing commit/end-input tests passed on JDK 11 with normal Maven checks, and this head's CI is green. Two additional real-table concurrency regressions nevertheless reproduce retained-tag data-file deletion and skipped automatic tags, detailed together below because they share the same serialization break.
For the retained-tag case, the same test-only scheduling probe passes both cases on clean merge-base 9b6c997 and fails only the overlapping end-input case on this head. The failing read uses freshly reopened persisted tag metadata. The automatic-tag probe was repeated with normal Maven checks. No production sources were modified for these probes.
| if (batchCommitted || identifier == COMMIT_IDENTIFIER) { | ||
| maintain(identifier, doExpire); |
There was a problem hiding this comment.
[P1] Serialize final maintenance with already-running asynchronous maintenance
This direct call can enter the same maintenance managers as the previous checkpoint's executor task. With decoupled changelog retention, the shared ChangelogDeletion tag skipper closes over cachedTagDataFiles: a second pass can replace the retained-tag-1 cache with tag 3 while the first pass is between planning and filtering its deletions. The first pass then deletes a file still required by tag 1.
I reproduced this with six real primary-key snapshots (overwrites at 2 and 4), retained tags at 1 and 3, snapshot retention min/max 1 and changelog retention max 2. Holding the old pass after planning changelog 2, running the final COMMIT_IDENTIFIER pass, then resuming the old pass removes the protected data file. The tag remains registered; reloading its persisted metadata and reading it throws FileNotFoundException. Both cases pass on clean base 9b6c997198; the same-head serialized checkpoint control also retains and reads the file.
There is a second manifestation with watermark automatic tags: three real append commits persist tags for snapshots [1,3] under overlap, versus [1,2,3] in the serialized control, because both runs mutate the same TagAutoCreation.nextSnapshot. ignoreIfExists protects duplicate tag creation, not this progress state.
Please queue the final maintenance behind preceding maintenance and wait for completion, or otherwise serialize access to all of these shared managers. It still needs to finish before close, while preserving the existing ordering of maintenance passes.
### Purpose Closes issue 11180 With `snapshot.expire.execution-mode = async`, the end-input commit now queues its maintenance after the maintenance of earlier commits and waits for it before returning. The end-input commit is the last commit of a bounded input. It uses `BatchWriteBuilder.COMMIT_IDENTIFIER`, and `close()` follows it. Before this change, the end-input maintenance was left on the async executor, and `close()` calls `shutdownNow()`, which interrupts a running expiration or drops one that has not started. - The end-input maintenance goes to the same single-thread executor instead of running on the committing thread. The maintenance managers keep state between runs, such as the tag data files cached by `ChangelogDeletion` and the next snapshot of `TagAutoCreation`. An end-input maintenance that overlaps an earlier one corrupts this state. Changelog expiration can then delete a data file that a retained tag still needs, and an automatic tag can be skipped. - This covers Flink bounded jobs, whose end-input commit uses `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action. The Spark compact procedure from issue 5681 commits through `commit(List)`, which runs maintenance on the caller thread since PR 6593. That path is unchanged. - `close()` is unchanged. Issue 5681 proposed `shutdown()` instead of `shutdownNow()`. The maintain thread is a daemon thread, so queued maintenance can still be lost when the process exits after `close()` returns. Waiting in `close()` would also block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs. - Checkpoint commits of streaming jobs keep the async mode. An expiration that `close()` interrupts there is picked up by a later expiration. - When an earlier maintenance is slow, the end-input commit returns only after it finishes. Sync mode is unchanged. ### Tests - `testEndInputCommitExpiresBeforeClose` in `SimpleTableTestBase`, run by `AppendOnlySimpleTableTest`, `PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`: checkpoint commits keep expiring asynchronously, and after the end-input commit and `close()` only the latest snapshot remains. It fails on the code before this pull request. - `TableCommitTest#testEndInputMaintenanceKeepsDataFilesOfRetainedTag`: with decoupled changelog retention, an end-input commit made while a checkpoint maintenance expires changelogs keeps a retained tag readable. - `TableCommitTest#testEndInputMaintenanceCreatesEveryAutomaticTag`: with watermark automatic tags, an end-input commit made while a checkpoint maintenance creates tags still gets one tag per snapshot, the same as maintenance that runs one at a time. - Both `TableCommitTest` cases fail when the end-input maintenance runs on the committing thread. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: KK <37928252+dragonknightk@users.noreply.github.com>
|
@JingsongLi Thanks for the detailed repro, both cases were spot on. I went with your first suggestion: the end-input maintenance is now submitted to the same single-thread executor and the commit waits for it, so it runs after any earlier maintenance and finishes before I added two tests based on your scenarios, |
Purpose
Closes issue 11180
With
snapshot.expire.execution-mode = async, the end-input commit now queues its maintenance after the maintenance of earlier commits and waits for it before returning. The end-input commit is the last commit of a bounded input. It usesBatchWriteBuilder.COMMIT_IDENTIFIER, andclose()follows it. Before this change, the end-input maintenance was left on the async executor, andclose()callsshutdownNow(), which interrupts a running expiration or drops one that has not started.ChangelogDeletionand the next snapshot ofTagAutoCreation. An end-input maintenance that overlaps an earlier one corrupts this state. Changelog expiration can then delete a data file that a retained tag still needs, and an automatic tag can be skipped.COMMIT_IDENTIFIER, and the Flink remove-unexisting-files action. The Spark compact procedure from issue 5681 commits throughcommit(List), which runs maintenance on the caller thread since PR 6593. That path is unchanged.close()is unchanged. Issue 5681 proposedshutdown()instead ofshutdownNow(). The maintain thread is a daemon thread, so queued maintenance can still be lost when the process exits afterclose()returns. Waiting inclose()would also block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs.close()interrupts there is picked up by a later expiration.Tests
testEndInputCommitExpiresBeforeCloseinSimpleTableTestBase, run byAppendOnlySimpleTableTest,PrimaryKeySimpleTableTest, andMockRESTSimpleTableTest: checkpoint commits keep expiring asynchronously, and after the end-input commit andclose()only the latest snapshot remains. It fails on the code before this pull request.TableCommitTest#testEndInputMaintenanceKeepsDataFilesOfRetainedTag: with decoupled changelog retention, an end-input commit made while a checkpoint maintenance expires changelogs keeps a retained tag readable.TableCommitTest#testEndInputMaintenanceCreatesEveryAutomaticTag: with watermark automatic tags, an end-input commit made while a checkpoint maintenance creates tags still gets one tag per snapshot, the same as maintenance that runs one at a time.TableCommitTestcases fail when the end-input maintenance runs on the committing thread.AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.
(๑•̀ㅂ•́)و✧