Repository navigation
Conversation
|
Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).
|
|
+1. We've encountered this in production as well, and the fact that indexing continues normally while tlogs silently stop rotating makes this especially difficult to detect. Interested in a 9.x backport too. |
|
Thanks @Neta-Rose for tracking this down and for the clear write-up. Locally the 5 new tests pass, the cancelled/promoted/max-retries ones fail with the fix reverted, and the recovery, tlog, PeerSync, split and MIGRATE suites are green. One gap: the The current cancel test calls Suggestion: drop the |
|
We've encountered this problem too, please merge this ASAP |
https://issues.apache.org/jira/browse/SOLR-18663
Description
RecoveryStrategy.doSyncOrReplicateRecovery()puts the update log into BUFFERING on every attempt, but only the success path replays the buffer. When an attempt ends any other way, the update log stays in BUFFERING: cancelled by leader election, max retries, or the "we are now the leader" return.From then on
UpdateLog.preCommit()skips every hard commit, so the tlog never rolls and grows at the full indexing rate while the core looks healthy. The common trigger is a replica promoted to leader mid-recovery: election callscancelRecovery(), the loop exits viaisClosed(), and the new leader is left BUFFERING. I hit this on 9.10.1: TLOG.state=1 on an indexing leader, tlog generation frozen, +33 GB in 12 days. This is the NRT side of SOLR-14679; that fix only covered TLOG replicas.Solution
The retry loop moves into a private method that returns whether recovery succeeded, and
doSyncOrReplicateRecovery()callsUpdateLog.dropBufferedUpdates()in afinallyblock when it didn't. Two guards:UpdateLog.close()keeps the buffer tlog as the "previous recovery did not complete" marker for the next startupThis doesn't change which updates get applied: nothing in the buffer was ever applied, and the next attempt's
bufferUpdates()would discard it anyway.I investigated this bug myself, and used an AI coding assistant during the fix implementation and for parts of the tests; I reviewed and verified every change myself.
Tests
New
RecoveryUlogStateTest, one test per exit path plus the two guards:On unpatched main the first three fail every time (the promoted leader's tlog id stays at 1 across three hard commits), and each guard test fails if its guard is removed. I also ran the recovery, leader election, tlog, PeerSync, split and MIGRATE suites (the core ones with
-Ptests.nightly=true) and the same tests on branch_9x.Checklist
Please review the following and check all that apply: