Repository navigation
[core] Stream Avro data reads with bounded buffering - #1449
Open
SteNicholas wants to merge 1 commit into
Open
SteNicholas wants to merge 1 commit into
SteNicholas wants to merge 1 commit into
Conversation
SteNicholas
force-pushed
the
PAIMON-1444
branch
2 times, most recently
from
November 18, 2026 07:19
0a7d4c0 to
952581d
Compare
JingsongLi
reviewed
Nov 18, 2026
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
The 56 existing Avro tests passed locally. The inline comments cover full-scan costs, executor compatibility, and cleanup on cancellation or error.
Avro scans fetched the entire file and retained every decoded record before yielding the first Arrow batch. Even a short LIMIT paid the full I/O and decoding cost, and row selection needed a file-wide mask. - Parse the writer schema once per file and feed selected Avro blocks to the same reader as the consumer requests batches, addressing apache#1444. - Start with 64 KiB of read-ahead. Fetch known payloads in requests of up to 8 MiB and grow the window to that size after the first batch of an unrestricted scan. - Apply inclusive ranges to physical row positions. Skip unselected block payloads while checking sync markers, stop after the last selected row and keep predicate columns for exact residual filtering. - Decode one batch of physical rows at a time on Tokio's blocking pool or inline on other executors. Release exhausted read buffers, abort queued work on drop and free scan state on completion or error.
SteNicholas
force-pushed
the
PAIMON-1444
branch
from
November 18, 2026 21:01
0bb6a96 to
b7600be
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #1444.
Avro data-file scans fetched the entire file and decoded every record before yielding the first Arrow batch. Stream OCF blocks on demand so a prefix scan can stop after bounded in-flight work and memory depends on the active block, batch and read-ahead window.
Brief change log
apache_avro::Reader.Tests
Verified at
b7600be6with Rust 1.98.0:cargo +1.98.0 test --locked --offline -p paimon --lib: 4,989 passed, 7 ignored.cargo +1.98.0 test --locked --offline -p paimon --lib avro: 83 passed.cargo +1.98.0 fmt --all -- --check: passed.cargo +1.98.0 clippy --locked --offline -p paimon --lib --tests -- -D warnings: passed.The streaming tests cover bounded first-batch I/O and full-scan request counts; backpressure; cancellation during pending reads, ready reads and sparse decoding; queued-task cancellation; and reader/buffer release while an errored stream remains alive. Non-Tokio coverage uses memory-backed FileIO and
futures::executor::block_on.Result coverage includes all six supported codecs, full scans and cross-block ranges, empty selections and files, zero-width records and empty blocks, missing/reordered fields, numeric widening, nullable unions, nested/logical values, and predicates on unprojected columns. Corrupt payloads, mismatched record counts, malformed/truncated framing and injected storage failures return stream errors. Related cases share existing fixtures and assertions.
Full-scan and LIMIT measurements
These measurements were recorded for the revisions listed below; the revised reader measured here is
0bb6a961. They have not been rerun for the current head,b7600be6.On a 15.61 MiB file, the larger full-scan window reduced range reads from 250 to 4 and raised median peak RSS from 19.0 to 28.6 MiB relative to the original streaming revision. Both streaming revisions served
LIMIT 32with 128 KiB of input.Fixture and measurement method
AvroFormatReaderfrom the eager base (ddcad6bd), the original streaming revision (952581de), and the revised reader (0bb6a961). Use identical release compiler settings, requested fields, batch size, andFileReadimplementation for all three. These timings are for a warm local file on arm64 macOS 26.6.2 with Rust 1.98.0.apache_avro::Writer, the null codec, a 65,536-byte block target, and a sync marker of sixteen0x07bytes. The schema containsid(long),payload(eight-digit zero-padded ID followed by 128xcharacters), and 96 long fields namedcol_2throughcol_97, each containing the ID. The file contains 250 blocks and 16,367,638 bytes.FileRead::readand perform a seek and exact read on that file handle. Measure from before stream construction through stream drop. Record the first batch when it reaches the consumer and usewait4for process peak RSS. Warm up each reader and workload once, then rotate reader order across five fresh processes per combination. The table gives medians, with min–max for elapsed time.API and Format
The
FormatFileReaderinterface and Avro OCF encoding remain unchanged. I/O and decoding become lazy, so their errors surface while polling the stream. Batches end at block boundaries and may shrink after selection or filtering.Memory includes the header/schema, compressed and decompressed block, output/conversion buffers, row ranges and bounded read-ahead. A single large block, header or value can dominate memory. Dropping a stream cancels queued work; a running task may finish its current decompression and physical batch. Skipped payloads and undecoded suffix records are not validated.
Documentation
benchmarks/avro/README.mddocuments block traversal, selection, memory bounds, read-ahead and cancellation, and links to the measurements above. The duplicate eager-reader benchmark helper and its module declaration have been removed.🤖 Assisted with Codex.