Repository navigation
Conversation
…range Reconnect from an earlier watermark instead of failing with PartialReorgError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…termark Starting over isn't safe: older watermarks may have been pruned, so consumers can't undo the pruned data a full replay would send again, and CDC consumers end up with duplicates. Fail with PartialReorgError instead, leaving the state as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a reorg starts inside a watermark's range,
TransactionalStreamnow does what Amp's docs say (step 4): it undoes back to the last watermark that ends before the reorg, and reconnects from it. Stacked on #35, which makes it see these reorgs on resume.PartialReorgError. It now emits anUndoand reconnects, so Amp re-sends everything that was undone. The rest of the old connection is dropped, including the batch that revealed the reorg.You might want to look at:
PartialReorgErrorbefore it undoes or reconnects, and you have to restart with fresh state. I didn't replay from the start becauseCdcStreamcan't delete batches it already pruned, so consumers would end up with duplicates. A "pruned" marker in the state store would make that safe for streams that never pruned, but it changes the store's interface.[100-199]retained and[0-99]pruned, a reorg at 180 is inside the 128-block window but has nothing to reconnect from. Keeping one more watermark infindPruningPointwould fix it, but it changes pruning, so I left it alone.PartialReorgErroris still public, and gained aninvalidationfield so the stream can recover from it.Undocan be wider than the reorg.cause.invalidationstill names the reorged blocks, whileinvalidatealso covers the earlier transactions.CdcStreamusesinvalidate, but a consumer deleting by the cause's block range would get duplicates of the re-sent blocks before the reorg.UnrecoverableReorgError.Checks:
pnpm lint,pnpm check:tsgo,pnpm test(250 passed, 13 new) andpnpm circularpass.ProtocolStream, and duplicates after pruning with the real transactional and CDC layers. The recovery ones fail without the fix. The ones expectingPartialReorgErrorpass either way, and are there to stop a replay from coming back.CdcStream, stopping, reorging 2 blocks and resuming with[0-58],[59-61]givesDelete, thenInsert [59-62], with one row per block matching the chain (before:PartialReorgError, as inTransactionalStreamfails withPartialReorgErrorwhen resuming after a reorg inside its last watermark #36).[0-64], or withretention: 0and only[68-70]left, givesPartialReorgError, no events and no duplicates.Fixes #36
🤖 Generated with Claude Code