Skip to content

SOLR-18663: Don't leave the update log BUFFERING after an incomplete … - #5359

Open
Neta-Rose wants to merge 1 commit into
apache:mainfrom
Neta-Rose:SOLR-18663
Open

Neta-Rose wants to merge 1 commit into
apache:mainfrom
Neta-Rose:SOLR-18663

Conversation

@Neta-Rose

@Neta-Rose Neta-Rose commented Nov 17, 2026 •

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/SOLR-18663

Description

RecoveryStrategy.doSyncOrReplicateRecovery() puts the update log into BUFFERING on every attempt, but only the success path replays the buffer. When an attempt ends any other way, the update log stays in BUFFERING: cancelled by leader election, max retries, or the "we are now the leader" return.

From then on UpdateLog.preCommit() skips every hard commit, so the tlog never rolls and grows at the full indexing rate while the core looks healthy. The common trigger is a replica promoted to leader mid-recovery: election calls cancelRecovery(), the loop exits via isClosed(), and the new leader is left BUFFERING. I hit this on 9.10.1: TLOG.state=1 on an indexing leader, tlog generation frozen, +33 GB in 12 days. This is the NRT side of SOLR-14679; that fix only covered TLOG replicas.

Solution

The retry loop moves into a private method that returns whether recovery succeeded, and doSyncOrReplicateRecovery() calls UpdateLog.dropBufferedUpdates() in a finally block when it didn't. Two guards:

  • only if this recovery started buffering, so a buffer owned by MIGRATE or a split is left alone (see SOLR-18601)
  • not during shutdown, where UpdateLog.close() keeps the buffer tlog as the "previous recovery did not complete" marker for the next startup

This doesn't change which updates get applied: nothing in the buffer was ever applied, and the next attempt's bufferUpdates() would discard it anyway.

I investigated this bug myself, and used an AI coding assistant during the fix implementation and for parts of the tests; I reviewed and verified every change myself.

Tests

New RecoveryUlogStateTest, one test per exit path plus the two guards:

  • cancelled recovery → ACTIVE
  • replica promoted to leader mid-recovery → ACTIVE, and hard commits roll the tlog again
  • max retries exceeded → ACTIVE
  • recovery on a leader buffering for someone else → buffer untouched
  • shutdown during buffering → buffer tlog kept on disk

On unpatched main the first three fail every time (the promoted leader's tlog id stays at 1 across three hard commits), and each guard test fails if its guard is removed. I also ran the recovery, leader election, tlog, PeerSync, split and MIGRATE suites (the core ones with -Ptests.nightly=true) and the same tests on branch_9x.

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.

@github-actions

Copy link
Copy Markdown

Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

@jd252387

Copy link
Copy Markdown

+1. We've encountered this in production as well, and the fact that indexing continues normally while tlogs silently stop rotating makes this especially difficult to detect.

Interested in a 9.x backport too.

@sarkaramrit1993

Copy link
Copy Markdown
Contributor

Thanks @Neta-Rose for tracking this down and for the clear write-up. Locally the 5 new tests pass, the cancelled/promoted/max-retries ones fail with the fix reverted, and the recovery, tlog, PeerSync, split and MIGRATE suites are green.

One gap: the isInterrupted() guard skips the drop on the main trigger. On leader election, close() calls cancelPrepRecoveryCmd() → FutureTask.cancel(true). sendPrepRecoveryCmd() runs that task on the recovery thread itself, right after bufferUpdates(), so the recovery thread gets interrupted (HttpJettySolrClient keeps the flag). The loop exits via isClosed() and the ulog stays BUFFERING.

The current cancel test calls cancelRecovery() from the replay hook, when no prep request is in flight, so it misses this. Pausing PREPRECOVERY with TestInjection.prepRecoveryOpPauseForever = "true:100" and cancelling from the test thread fails 5/5 on this PR and passes 5/5 without the isInterrupted() clause. Your 5 tests still pass without it, since cc.isShutDown() already covers shutdown.

Suggestion: drop the isInterrupted() check and add that case as a test. Happy to share mine.

@adamsi

adamsi commented Nov 18, 2026

Copy link
Copy Markdown

We've encountered this problem too, please merge this ASAP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants