Skip to content

fix: keep reorg replacement rows, detect reorgs on resume, skip keep-alives - #39

Open
glabun002 wants to merge 2 commits into
edgeandnode:mainfrom
glabun002:fix/reorg-rows-resume-keepalive
Open

glabun002 wants to merge 2 commits into
edgeandnode:mainfrom
glabun002:fix/reorg-rows-resume-keepalive

Conversation

@glabun002

Copy link
Copy Markdown

Fixes the three streaming issues the SDK docs list as known issues, plus one undocumented bug. Each fix has a test that
fails on main (1b8cf83) and passes with the fix.

Fixes

1. Rows are lost after a reorg (protocol-stream/service.ts)

Amp sends the replacement blocks' rows in the same record batch that reveals the reorg, but ProtocolStream emitted a
Reorg message in place of that batch's Data. So TransactionalStream and CdcStream never delivered those rows.
It now emits Reorg followed by Data. For a complete range without rows, it emits Reorg followed by Watermark.

2. A reorg while a stream is stopped goes unnoticed (protocol-stream/service.ts)

ProtocolStream started with no previous ranges, so the first batch after a resume was never compared with the
cursor. Rows Amp re-sent after a reorg arrived as plain Data, and clients kept both the orphaned and the replacement
rows. The stream now starts from the resume watermark, so the existing validation and reorg detection check the first
batch against it.

3. Streams fail after 30 seconds (arrow-flight/service.ts)

The empty keep-alive message Amp sends every keep_alive_interval was parsed as a flatbuffer, which threw
RangeError: Offset is outside the bounds of the DataView. Messages without a data header are now skipped.

4. stream: false ran a streaming query (arrow-flight/service.ts, undocumented)

Any defined stream option set amp-stream: true, and the header overrides the query's SETTINGS, so
query(sql, { stream: false }) never ended. The header now carries the option's value.

Tests

  • New test/protocol-stream/stream.test.ts runs the real ProtocolStream over a mock ArrowFlight that replays
    Amp's batches. The existing reorg.test.ts checks a copy of the detection logic, which is how fix 1's bug went
    unnoticed.
  • test/arrow-flight/service.test.ts gains a keep-alive test and an amp-stream header test.
main source + these tests This PR
The two changed test files 5 fail, 88 pass 93 pass
Full suite (pnpm test) not run 240 pass
pnpm check, pnpm check:tsgo, pnpm lint, pnpm circular not run pass

Not run: the Deno test job.

Not in this PR

PartialReorgError still ends the stream when a reorg starts inside a multi-block watermark, like the first one after
a resume. The protocol docs describe a recovery (resume from the cursor before that range), which could be a follow-up.

Once merged, the "Known issue" boxes on the TypeScript SDK's streaming, transactional-streams and cdc-streams docs
pages can go.

Found while building a reorg fuzzer for Amp Lab Week.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL

glabun002 and others added 2 commits November 4, 2026 18:26
…n resume

Two documented known issues in ProtocolStream (and so in TransactionalStream
and CdcStream, which build on it):

1. Rows are lost after a reorg. Amp sends the replacement blocks' rows in the
   same record batch that reveals the reorg, but ProtocolStream emitted a
   `Reorg` message in place of that batch's `Data`. It now emits `Reorg`, then
   `Data` with the batch's rows (or `Watermark`, for a complete range without
   rows).

2. A reorg while a stream is stopped goes unnoticed. ProtocolStream started
   with no previous ranges, so the first batch after a resume was never
   compared with the cursor, and re-sent rows arrived as plain `Data`: clients
   kept both the orphaned and the replacement rows. The stream now starts from
   the resume watermark, so the first batch's ranges are checked against it.

Adds end-to-end tests that run the real ProtocolStream over a mock ArrowFlight;
the existing reorg tests only exercised a copy of the detection logic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL
1. Streams failed after 30 seconds with `RangeError: Offset is outside the
   bounds of the DataView`. While a streaming query waits for new blocks, Amp
   sends an empty keep-alive message every `keep_alive_interval` seconds;
   streamQuery tried to parse its empty data header as a flatbuffer. Messages
   without a data header are now skipped.

2. `stream: false` sent `amp-stream: true`. Any defined `stream` option set the
   header to "true", and the header overrides the query's `SETTINGS`, so
   `query(sql, { stream: false })` ran a streaming query that never ended. The
   header now carries the option's value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL
@benface

benface commented Nov 13, 2026

Copy link
Copy Markdown

Heads-up: this overlaps three of my drafts, #31 (keep-alive frames), #33 (stream: false, plus Utf8View/BinaryView decoding) and #35 (the two reorg fixes, plus making InMemoryStateStore.commit idempotent). The overlapping fixes look like the same approach, and #31 conflicts with yours in arrow-flight/service.ts. Since mine also cover the view types and the idempotent commit, would you be OK landing #31, #33 and #35 and closing this one? Or if you'd rather land yours, I'll rebase the parts yours doesn't cover on top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants