Skip to content

[core] Remove redundant global-index preparation during DV materialization - #11252

Open
zhoulii wants to merge 2 commits into
apache:masterfrom
zhoulii:improve/de-materialization-global-index-preparation
Open

zhoulii wants to merge 2 commits into
apache:masterfrom
zhoulii:improve/de-materialization-global-index-preparation

Conversation

@zhoulii

@zhoulii zhoulii commented Nov 18, 2026

Copy link
Copy Markdown
Member

Purpose

close #11251

Tests

@@ -41,9 +46,6 @@ public List<CommitMessage> prepare(List<CommitMessage> messages) {
result.addAll(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that DataEvolutionCompactGlobalIndexDropper is removed and preparation only emits DELETION_VECTORS_INDEX changes, nothing prepares FileKind.DELETE global-index entries against an older snapshot anymore (DataEvolutionCompactDeletionVectorRewriter#checkNoCompactIndexChanges also rejects pre-existing compact index changes).

Should we also clean up the leftover filter in FileStoreCommitImpl#dataEvolutionChangesProvider (lines 1031–1043, // Replace global-index deletions prepared against an older snapshot.) to new ArrayList<>(changes.indexFiles) and update FileStoreCommitTest#testMaterializedCompactionOnlyRefreshesGlobalIndexInSameBucket accordingly?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the obsolete filtering. Updated the test to verify that only global indexes in the materialized bucket are deleted, while indexes in other buckets remain intact.

table, partition(table, "b"), globalIndexFile("untouched-index", new Range(3, 5)));
if ("before".equals(indexTiming) || fullyDeleted) {
commitGlobalIndex(table, partitionA, globalIndexFile("old-index", new Range(0, 2)));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "retry" case, partitionA currently has no global index when snapshot is taken ("old-index" is only committed for "before" and "fully-deleted"). As a result:

  1. The assertion at lines 1071–1079 is vacuous for "retry" because partitionA has no global index at snapshot.
  2. Commit attempt 1 does not produce any global-index DELETE entry for partitionA, so the retry does not verify that attempt 1's DELETE entries are discarded/rebuilt cleanly on attempt 2 without causing a duplicate or missing-file IllegalStateException in GlobalIndexCombiner#combine.

If we also commit "old-index" before preparation when "retry".equals(indexTiming) and compact it into "concurrent-index" inside tryToWriteAtomic (via indexCompactMessage(partitionA, oldIndex, globalIndexFile("concurrent-index", new Range(0, 2)))), the "retry" case will verify both that attempt 1's stale DELETE old-index is dropped and that "concurrent-index" from the retried snapshot is invalidated.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the retry case to start with old-index and replace it with concurrent-index during the first commit attempt. It now verifies that the retry regenerates the deletions against the latest snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Remove redundant global index preparation during DV materialization

2 participants