Skip to content

fix(transactional stream): recover from a reorg inside a watermark's range - #37

Draft
benface wants to merge 2 commits into
benface/reorg-fixesfrom
benface/partial-reorg-recovery
Draft

benface wants to merge 2 commits into
benface/reorg-fixesfrom
benface/partial-reorg-recovery

Conversation

@benface

@benface benface commented Nov 3, 2026

Copy link
Copy Markdown

When a reorg starts inside a watermark's range, TransactionalStream now does what Amp's docs say (step 4): it undoes back to the last watermark that ends before the reorg, and reconnects from it. Stacked on #35, which makes it see these reorgs on resume.

  • Resuming after a reorg inside a multi-block watermark failed with PartialReorgError. It now emits an Undo and reconnects, so Amp re-sends everything that was undone. The rest of the old connection is dropped, including the batch that revealed the reorg.

You might want to look at:

  • It still fails when there's no earlier watermark to reconnect from: a reorg inside the first watermark, or when the older ones were pruned. The stream fails with PartialReorgError before it undoes or reconnects, and you have to restart with fresh state. I didn't replay from the start because CdcStream can't delete batches it already pruned, so consumers would end up with duplicates. A "pruned" marker in the state store would make that safe for streams that never pruned, but it changes the store's interface.
  • Reorgs within retention can hit that too. With [100-199] retained and [0-99] pruned, a reorg at 180 is inside the 128-block window but has nothing to reconnect from. Keeping one more watermark in findPruningPoint would fix it, but it changes pruning, so I left it alone.
  • PartialReorgError is still public, and gained an invalidation field so the stream can recover from it.
  • The Undo can be wider than the reorg. cause.invalidation still names the reorged blocks, while invalidate also covers the earlier transactions. CdcStream uses invalidate, but a consumer deleting by the cause's block range would get duplicates of the re-sent blocks before the reorg.
  • Not changed: a reorg at or before the first buffered watermark's first block still fails with UnrecoverableReorgError.

Checks:

  • pnpm lint, pnpm check:tsgo, pnpm test (250 passed, 13 new) and pnpm circular pass.
  • The new tests cover recovery and the no-earlier-watermark failure in the state actor, the reconnect in the stream with a mock ProtocolStream, and duplicates after pruning with the real transactional and CDC layers. The recovery ones fail without the fix. The ones expecting PartialReorgError pass either way, and are there to stop a replay from coming back.
  • Against ampd v0.0.42 over Anvil, through CdcStream, stopping, reorging 2 blocks and resuming with [0-58], [59-61] gives Delete, then Insert [59-62], with one row per block matching the chain (before: PartialReorgError, as in TransactionalStream fails with PartialReorgError when resuming after a reorg inside its last watermark #36).
  • Same setup with only [0-64], or with retention: 0 and only [68-70] left, gives PartialReorgError, no events and no duplicates.

Fixes #36

🤖 Generated with Claude Code

benface and others added 2 commits November 2, 2026 23:24
…range

Reconnect from an earlier watermark instead of failing with PartialReorgError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…termark

Starting over isn't safe: older watermarks may have been pruned, so consumers
can't undo the pruned data a full replay would send again, and CDC consumers
end up with duplicates. Fail with PartialReorgError instead, leaving the state
as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@benface
benface added this pull request to stack #38 November 3, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant