Skip to content

fix(arrow flight): only stream when stream is true, decode Utf8View and BinaryView - #33

Draft
benface wants to merge 2 commits into
mainfrom
benface/stream-false-utf8view
Draft

benface wants to merge 2 commits into
mainfrom
benface/stream-false-utf8view

Conversation

@benface

@benface benface commented Nov 2, 2026

Copy link
Copy Markdown

streamQuery ran a streaming query whenever stream was set, even to false, and couldn't decode results with Utf8View columns, which Amp returns for casts to VARCHAR. It now only sends amp-stream for stream: true, and decodes Utf8View and BinaryView: the schema parser knows both types, record batches read their variadic buffer counts, and new readers handle inline and out-of-line values.

  • stream: false ran a streaming query that never ended. It now runs a normal query.
  • Results with a Utf8View column (like CAST(x AS VARCHAR) or concat) failed with ParseSchemaError (InvalidArrowDataTypeError, type 24), and BinaryView ones with type 23. Both now decode.

You might prefer:

  • Sending amp-stream: false for stream: false, instead of leaving the header out.
  • Supporting the other newer types too. ListView, LargeListView and RunEndEncoded still fail with InvalidArrowDataTypeError.
  • Handling view-typed dictionary values in decodeDictionaryBatch. streamQuery doesn't decode dictionary batches yet (the TODO in service.ts), so I left it.

Checks:

  • pnpm lint, pnpm check:tsgo, pnpm test (237 passed, 4 new) and pnpm circular pass.
  • The new tests check the amp-stream header for stream: false and true, and roundtrip Utf8View and BinaryView columns, including nested ones and values spread over several data buffers. They fail without the fix.
  • PyArrow-written IPC fixtures decode too: values at the 12/13-byte inline boundary, Unicode, nulls, nested lists and structs, and sliced arrays across data buffers.
  • Against ampd v0.0.42 over Anvil, stream: false ends with all rows, and stream: true still streams. CAST(block_num AS VARCHAR), concat (long and short values, nulls) and arrow_cast(…, 'BinaryView') all decode and match their source columns.

Fixes #32

🤖 Generated with Claude Code

benface and others added 2 commits November 2, 2026 15:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Arrow Flight queries: stream: false still streams, and Utf8View results don't decode

1 participant