Skip to content

[core] Stream Avro data reads with bounded buffering - #1449

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1444
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1444

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #1444.

Avro data-file scans fetched the entire file and decoded every record before yielding the first Arrow batch. Stream OCF blocks on demand so a prefix scan can stop after bounded in-flight work and memory depends on the active block, batch and read-ahead window.

Brief change log

  • Read the header lazily, parse the writer schema once per file, and feed successive selected blocks to the same apache_avro::Reader.
  • Start with 64 KiB of read-ahead. Fetch known payloads in requests of up to 8 MiB, and grow the read-ahead window to 8 MiB after the first batch of an unrestricted scan. Explicit row ranges retain the smaller window for skipping irrelevant payloads.
  • Apply inclusive ranges to original physical row positions without a file-sized mask. Skip unselected block payloads while checking framing and sync markers, and stop after the final selected row. Preserve projection widening and exact residual filtering.
  • Run schema parsing, decompression, datum decoding, Arrow conversion and filtering on Tokio's blocking pool, with an inline fallback for other executors. Bound each decode task by physical rows, including discarded rows, and yield between immediately ready reads.
  • Abort queued decode tasks on drop and release their captured input even while blocking workers are busy. Release exhausted storage windows and drop scan state before yielding a terminal error or a batch that exhausts the file or selection.

Tests

Verified at b7600be6 with Rust 1.98.0:

  • cargo +1.98.0 test --locked --offline -p paimon --lib: 4,989 passed, 7 ignored.
  • cargo +1.98.0 test --locked --offline -p paimon --lib avro: 83 passed.
  • cargo +1.98.0 fmt --all -- --check: passed.
  • cargo +1.98.0 clippy --locked --offline -p paimon --lib --tests -- -D warnings: passed.

The streaming tests cover bounded first-batch I/O and full-scan request counts; backpressure; cancellation during pending reads, ready reads and sparse decoding; queued-task cancellation; and reader/buffer release while an errored stream remains alive. Non-Tokio coverage uses memory-backed FileIO and futures::executor::block_on.

Result coverage includes all six supported codecs, full scans and cross-block ranges, empty selections and files, zero-width records and empty blocks, missing/reordered fields, numeric widening, nullable unions, nested/logical values, and predicates on unprojected columns. Corrupt payloads, mismatched record counts, malformed/truncated framing and injected storage failures return stream errors. Related cases share existing fixtures and assertions.

Full-scan and LIMIT measurements

These measurements were recorded for the revisions listed below; the revised reader measured here is 0bb6a961. They have not been rerun for the current head, b7600be6.

On a 15.61 MiB file, the larger full-scan window reduced range reads from 250 to 4 and raised median peak RSS from 19.0 to 28.6 MiB relative to the original streaming revision. Both streaming revisions served LIMIT 32 with 128 KiB of input.

Workload Reader Elapsed ms, median (min–max) First batch ms Consumed rows/s Range reads Bytes read Peak RSS MiB
Full scan Eager 143.87 (131.67–152.51) 77.12 278,023 1 16,367,638 418.8
Full scan Original streaming 140.00 (125.78–147.47) 1.09 285,705 250 16,367,638 19.0
Full scan Revised streaming 113.87 (102.51–116.52) 1.06 351,286 4 16,367,638 28.6
LIMIT 32 Eager 121.17 (114.86–128.81) 81.52 264 1 16,367,638 418.5
LIMIT 32 Original streaming 1.12 (1.09–1.22) 1.09 28,605 2 131,072 18.3
LIMIT 32 Revised streaming 1.04 (1.01–1.16) 1.00 30,859 2 131,072 18.3
Fixture and measurement method
  • Compare the production AvroFormatReader from the eager base (ddcad6bd), the original streaming revision (952581de), and the revised reader (0bb6a961). Use identical release compiler settings, requested fields, batch size, and FileRead implementation for all three. These timings are for a warm local file on arm64 macOS 26.6.2 with Rust 1.98.0.
  • Generate 40,000 records for IDs 0 through 39,999 using apache_avro::Writer, the null codec, a 65,536-byte block target, and a sync marker of sixteen 0x07 bytes. The schema contains id (long), payload (eight-digit zero-padded ID followed by 128 x characters), and 96 long fields named col_2 through col_97, each containing the ID. The file contains 250 blocks and 16,367,638 bytes.
  • Project all 98 fields with a 1,024-row batch target, no predicates, and no row ranges. Consume all batches for a full scan. For the prefix, count 32 rows and drop the stream immediately. The first batch held 1,024 rows for the eager reader and 217 for each streaming reader.
  • Open the file before timing. Count one range for each FileRead::read and perform a seek and exact read on that file handle. Measure from before stream construction through stream drop. Record the first batch when it reaches the consumer and use wait4 for process peak RSS. Warm up each reader and workload once, then rotate reader order across five fresh processes per combination. The table gives medians, with min–max for elapsed time.

API and Format

The FormatFileReader interface and Avro OCF encoding remain unchanged. I/O and decoding become lazy, so their errors surface while polling the stream. Batches end at block boundaries and may shrink after selection or filtering.

Memory includes the header/schema, compressed and decompressed block, output/conversion buffers, row ranges and bounded read-ahead. A single large block, header or value can dominate memory. Dropping a stream cancels queued work; a running task may finish its current decompression and physical batch. Skipped payloads and undecoded suffix records are not validated.

Documentation

benchmarks/avro/README.md documents block traversal, selection, memory bounds, read-ahead and cancellation, and links to the measurements above. The duplicate eager-reader benchmark helper and its module declaration have been removed.

🤖 Assisted with Codex.

@SteNicholas
SteNicholas force-pushed the PAIMON-1444 branch 2 times, most recently from 0a7d4c0 to 952581d Compare November 18, 2026 07:19

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 56 existing Avro tests passed locally. The inline comments cover full-scan costs, executor compatibility, and cleanup on cancellation or error.

Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Avro scans fetched the entire file and retained every decoded record
before yielding the first Arrow batch. Even a short LIMIT paid the
full I/O and decoding cost, and row selection needed a file-wide mask.

- Parse the writer schema once per file and feed selected Avro blocks
  to the same reader as the consumer requests batches, addressing apache#1444.
- Start with 64 KiB of read-ahead. Fetch known payloads in requests of
  up to 8 MiB and grow the window to that size after the first batch of
  an unrestricted scan.
- Apply inclusive ranges to physical row positions. Skip unselected
  block payloads while checking sync markers, stop after the last
  selected row and keep predicate columns for exact residual filtering.
- Decode one batch of physical rows at a time on Tokio's blocking pool
  or inline on other executors. Release exhausted read buffers, abort
  queued work on drop and free scan state on completion or error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Stream Avro data reads with bounded buffering

2 participants