Repository navigation
Conversation
…n resume Two documented known issues in ProtocolStream (and so in TransactionalStream and CdcStream, which build on it): 1. Rows are lost after a reorg. Amp sends the replacement blocks' rows in the same record batch that reveals the reorg, but ProtocolStream emitted a `Reorg` message in place of that batch's `Data`. It now emits `Reorg`, then `Data` with the batch's rows (or `Watermark`, for a complete range without rows). 2. A reorg while a stream is stopped goes unnoticed. ProtocolStream started with no previous ranges, so the first batch after a resume was never compared with the cursor, and re-sent rows arrived as plain `Data`: clients kept both the orphaned and the replacement rows. The stream now starts from the resume watermark, so the first batch's ranges are checked against it. Adds end-to-end tests that run the real ProtocolStream over a mock ArrowFlight; the existing reorg tests only exercised a copy of the detection logic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL
1. Streams failed after 30 seconds with `RangeError: Offset is outside the
bounds of the DataView`. While a streaming query waits for new blocks, Amp
sends an empty keep-alive message every `keep_alive_interval` seconds;
streamQuery tried to parse its empty data header as a flatbuffer. Messages
without a data header are now skipped.
2. `stream: false` sent `amp-stream: true`. Any defined `stream` option set the
header to "true", and the header overrides the query's `SETTINGS`, so
`query(sql, { stream: false })` ran a streaming query that never ended. The
header now carries the option's value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL
|
Heads-up: this overlaps three of my drafts, #31 (keep-alive frames), #33 ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the three streaming issues the SDK docs list as known issues, plus one undocumented bug. Each fix has a test that
fails on
main(1b8cf83) and passes with the fix.Fixes
1. Rows are lost after a reorg (
protocol-stream/service.ts)Amp sends the replacement blocks' rows in the same record batch that reveals the reorg, but
ProtocolStreamemitted aReorgmessage in place of that batch'sData. SoTransactionalStreamandCdcStreamnever delivered those rows.It now emits
Reorgfollowed byData. For a complete range without rows, it emitsReorgfollowed byWatermark.2. A reorg while a stream is stopped goes unnoticed (
protocol-stream/service.ts)ProtocolStreamstarted with no previous ranges, so the first batch after a resume was never compared with thecursor. Rows Amp re-sent after a reorg arrived as plain
Data, and clients kept both the orphaned and the replacementrows. The stream now starts from the resume watermark, so the existing validation and reorg detection check the first
batch against it.
3. Streams fail after 30 seconds (
arrow-flight/service.ts)The empty keep-alive message Amp sends every
keep_alive_intervalwas parsed as a flatbuffer, which threwRangeError: Offset is outside the bounds of the DataView. Messages without a data header are now skipped.4.
stream: falseran a streaming query (arrow-flight/service.ts, undocumented)Any defined
streamoption setamp-stream: true, and the header overrides the query'sSETTINGS, soquery(sql, { stream: false })never ended. The header now carries the option's value.Tests
test/protocol-stream/stream.test.tsruns the realProtocolStreamover a mockArrowFlightthat replaysAmp's batches. The existing
reorg.test.tschecks a copy of the detection logic, which is how fix 1's bug wentunnoticed.
test/arrow-flight/service.test.tsgains a keep-alive test and anamp-streamheader test.mainsource + these testspnpm test)pnpm check,pnpm check:tsgo,pnpm lint,pnpm circularNot run: the Deno test job.
Not in this PR
PartialReorgErrorstill ends the stream when a reorg starts inside a multi-block watermark, like the first one aftera resume. The protocol docs describe a recovery (resume from the cursor before that range), which could be a follow-up.
Once merged, the "Known issue" boxes on the TypeScript SDK's streaming, transactional-streams and cdc-streams docs
pages can go.
Found while building a reorg fuzzer for Amp Lab Week.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JhDzg7TanevyCxwQKacLpL