Repository navigation
Conversation
| @@ -41,9 +46,6 @@ public List<CommitMessage> prepare(List<CommitMessage> messages) { | |||
| result.addAll( | |||
There was a problem hiding this comment.
Now that DataEvolutionCompactGlobalIndexDropper is removed and preparation only emits DELETION_VECTORS_INDEX changes, nothing prepares FileKind.DELETE global-index entries against an older snapshot anymore (DataEvolutionCompactDeletionVectorRewriter#checkNoCompactIndexChanges also rejects pre-existing compact index changes).
Should we also clean up the leftover filter in FileStoreCommitImpl#dataEvolutionChangesProvider (lines 1031–1043, // Replace global-index deletions prepared against an older snapshot.) to new ArrayList<>(changes.indexFiles) and update FileStoreCommitTest#testMaterializedCompactionOnlyRefreshesGlobalIndexInSameBucket accordingly?
There was a problem hiding this comment.
Removed the obsolete filtering. Updated the test to verify that only global indexes in the materialized bucket are deleted, while indexes in other buckets remain intact.
| table, partition(table, "b"), globalIndexFile("untouched-index", new Range(3, 5))); | ||
| if ("before".equals(indexTiming) || fullyDeleted) { | ||
| commitGlobalIndex(table, partitionA, globalIndexFile("old-index", new Range(0, 2))); | ||
| } |
There was a problem hiding this comment.
In the "retry" case, partitionA currently has no global index when snapshot is taken ("old-index" is only committed for "before" and "fully-deleted"). As a result:
- The assertion at lines 1071–1079 is vacuous for
"retry"becausepartitionAhas no global index atsnapshot. - Commit attempt 1 does not produce any global-index
DELETEentry forpartitionA, so the retry does not verify that attempt 1'sDELETEentries are discarded/rebuilt cleanly on attempt 2 without causing a duplicate or missing-fileIllegalStateExceptioninGlobalIndexCombiner#combine.
If we also commit "old-index" before preparation when "retry".equals(indexTiming) and compact it into "concurrent-index" inside tryToWriteAtomic (via indexCompactMessage(partitionA, oldIndex, globalIndexFile("concurrent-index", new Range(0, 2)))), the "retry" case will verify both that attempt 1's stale DELETE old-index is dropped and that "concurrent-index" from the retried snapshot is invalidated.
There was a problem hiding this comment.
Updated the retry case to start with old-index and replace it with concurrent-index during the first commit attempt. It now verifies that the retry regenerates the deletions against the latest snapshot.
Purpose
close #11251
Tests