From b7600be65002c72bb36b05f6e88023a998ab2b0f Mon Sep 17 00:00:00 2001 From: Nicholas Jiang Date: Tue, 17 Nov 2026 23:19:34 -0800 Subject: [PATCH] [core] Stream Avro data-file reads with bounded buffering Avro scans fetched the entire file and retained every decoded record before yielding the first Arrow batch. Even a short LIMIT paid the full I/O and decoding cost, and row selection needed a file-wide mask. - Parse the writer schema once per file and feed selected Avro blocks to the same reader as the consumer requests batches, addressing #1444. - Start with 64 KiB of read-ahead. Fetch known payloads in requests of up to 8 MiB and grow the window to that size after the first batch of an unrestricted scan. - Apply inclusive ranges to physical row positions. Skip unselected block payloads while checking sync markers, stop after the last selected row and keep predicate columns for exact residual filtering. - Decode one batch of physical rows at a time on Tokio's blocking pool or inline on other executors. Release exhausted read buffers, abort queued work on drop and free scan state on completion or error. --- Cargo.lock | 1 + benchmarks/avro/README.md | 80 ++ crates/paimon/Cargo.toml | 2 +- crates/paimon/src/arrow/format/avro.rs | 750 ++++++------ .../src/arrow/format/avro/streaming_tests.rs | 1009 +++++++++++++++++ crates/paimon/src/arrow/format/avro_write.rs | 13 +- crates/paimon/src/arrow/format/mod.rs | 10 + crates/paimon/src/arrow/format/mosaic.rs | 12 +- crates/paimon/src/table/data_file_reader.rs | 10 +- 9 files changed, 1535 insertions(+), 352 deletions(-) create mode 100644 benchmarks/avro/README.md create mode 100644 crates/paimon/src/arrow/format/avro/streaming_tests.rs diff --git a/Cargo.lock b/Cargo.lock index e3ec5e217..d69d178c8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7519,6 +7519,7 @@ dependencies = [ "futures-core", "futures-io", "futures-sink", + "futures-util", "pin-project-lite", "tokio", ] diff --git a/benchmarks/avro/README.md b/benchmarks/avro/README.md new file mode 100644 index 000000000..cd653f3d7 --- /dev/null +++ b/benchmarks/avro/README.md @@ -0,0 +1,80 @@ + + +# Avro data-file streaming + +The data-file reader visits Avro Object Container File (OCF) blocks in +physical row order, following Java Paimon's `AvroBulkFormat`. It decompresses +one selected block at a time. A block can yield several Arrow batches, each +with at most the requested number of rows. Row selection and residual +filtering can make batches smaller. The reader includes unprojected predicate +columns when evaluating residual filters. + +Row ranges use original, zero-based physical positions, with inclusive ends. +Block record counts allow whole irrelevant payloads to be skipped. The reader +still checks framing and sync markers. Reaching a selected record inside a +block requires decoding preceding records in that block. This is sequential +access, without a random-access index. Scanning stops after the final selected +row. Corruption in consumed framing, markers, payloads or records is returned +as a stream error. Unread blocks, skipped payloads and undecoded suffix records +are not validated. A header-only OCF is empty. A zero-byte file is malformed. +An empty row selection performs no I/O. + +## Memory, backpressure and cancellation + +Storage reads and datum decoding start on the first poll. At most one storage +request is pending. The initial read-ahead window is 64 KiB. When framing +gives the size of a selected payload, the reader fetches the remaining bytes +in requests of up to 8 MiB, including for the first block. After the first +batch, an unrestricted scan also reads ahead up to 8 MiB to reduce sequential +range GETs. A `LIMIT` that needs another block can trigger this larger +read-ahead. Explicit row ranges keep the 64 KiB read-ahead window so the +reader can skip irrelevant payloads. + +Under Tokio, schema parsing, decompression, datum decoding, Arrow conversion +and residual filtering run on the blocking pool. Other executors run this +work inline. Each decode task processes at most one batch of physical +records, including rows discarded by row selection. Dropping the stream +cancels a pending read or aborts a queued decode and frees its input even +while every blocking worker is busy. A running task can finish its current +decompression and batch. Only polling the stream can start another read or +decode. Reads yield between buffer refills, including on cache hits. +The stream releases the reader and decoder before returning a terminal error +or a batch that exhausts the file or the selected ranges. + +Peak memory includes the read-ahead window, the header and schema, the +compressed and decompressed block, row ranges, and one batch of Avro values +and Arrow conversion buffers. The reader releases each storage window when +it consumes the last byte. It parses the writer schema once per file and +releases the encoded block once the decoder has copied its payload and marker. +Only rows selected by position accumulate as Avro values. Arrow conversion +releases those values before residual filtering. The decoder can retain +capacity from a previous large block. A large block, header or value can +dominate memory. Consumers and storage caches can retain additional buffers. + +## Comparing eager and streaming reads + +The comparison uses a 40,000-row file with 98 columns, the null codec and a +64 KiB block target. Project all columns with a 1,024-row batch target. +Consume the full stream for throughput. For `LIMIT 32`, drop the stream as +soon as the consumer has 32 rows. Time from reader construction through +stream drop, recording time to first batch, consumed rows per second, storage +requests and bytes read. Run each sample in a fresh process for peak RSS. +The fixture recipe, cache policy and before/after measurements are in +[PR #1449](https://github.com/apache/paimon-rust/pull/1449). diff --git a/crates/paimon/Cargo.toml b/crates/paimon/Cargo.toml index 3b881b543..25d0f8f8c 100644 --- a/crates/paimon/Cargo.toml +++ b/crates/paimon/Cargo.toml @@ -139,7 +139,7 @@ arrow-select = { workspace = true } arrow-string = { workspace = true } futures = "0.3" crossbeam-channel = "0.5" -tokio-util = { workspace = true, features = ["compat", "io-util"] } +tokio-util = { workspace = true, features = ["compat", "io-util", "rt"] } parquet = { workspace = true, features = ["async", "zstd", "lz4", "snap"] } orc-rust = "0.9.0" async-stream = "0.3.6" diff --git a/crates/paimon/src/arrow/format/avro.rs b/crates/paimon/src/arrow/format/avro.rs index c1f1ce37c..eb3385dc8 100644 --- a/crates/paimon/src/arrow/format/avro.rs +++ b/crates/paimon/src/arrow/format/avro.rs @@ -15,8 +15,9 @@ // specific language governing permissions and limitations // under the License. -use super::{FilePredicates, FormatFileReader}; +use super::{panic_payload_message, FilePredicates, FormatFileReader}; use crate::arrow::build_target_arrow_schema; +use crate::arrow::residual::{filter_record_batch_by_predicates, widen_scan_fields}; use crate::io::FileRead; use crate::spec::{DataField, DataType, IntType, MapType, RowType}; use crate::table::{ArrowRecordBatchStream, RowRange}; @@ -32,14 +33,24 @@ use arrow_array::{ }; use arrow_buffer::{BooleanBuffer, NullBuffer, OffsetBuffer, ScalarBuffer}; use arrow_schema::SchemaRef; -use async_stream::try_stream; use async_trait::async_trait; -use futures::StreamExt; -use std::sync::Arc; - +use bytes::{Buf, Bytes}; +use futures::{stream, StreamExt, TryFutureExt}; +use std::collections::VecDeque; +use std::io::Read; +use std::panic::{catch_unwind, AssertUnwindSafe}; +use std::sync::{Arc, Mutex}; +use tokio_util::task::AbortOnDropHandle; + +/// Reads Avro object container files on demand, one block at a time. pub(crate) struct AvroFormatReader; const DEFAULT_BATCH_SIZE: usize = 8192; +const INITIAL_READ_AHEAD: u64 = 64 * 1024; +const MAX_READ_SIZE: u64 = 8 * 1024 * 1024; + +#[cfg(test)] +mod streaming_tests; #[async_trait] impl FormatFileReader for AvroFormatReader { @@ -52,66 +63,429 @@ impl FormatFileReader for AvroFormatReader { batch_size: Option, row_selection: Option>, ) -> crate::Result { - // NOTE: Avro OCF requires sequential reading, so we load the entire file into memory. - // This is fine for typical Paimon data files but may be problematic for very large files. - let file_bytes = reader.read(0..file_size).await?; - - // Widen the decoded schema to include any predicate columns that are not - // in the projection, so residual filtering can see them. DataFileReader - // projects the returned batch to `read_fields` by name, dropping extras. - let scan_fields = crate::arrow::residual::widen_scan_fields(read_fields, predicates); + let scan_fields = widen_scan_fields(read_fields, predicates); let target_schema = build_target_arrow_schema(&scan_fields)?; - let batch_size = batch_size.unwrap_or(DEFAULT_BATCH_SIZE); - // Own the predicates so the returned 'static stream does not borrow the - // caller's `&FilePredicates` (FilePredicates is not `Clone`; rebuild it). + let batch_size = batch_size.unwrap_or(DEFAULT_BATCH_SIZE).max(1); let predicates = predicates.map(|fp| FilePredicates { predicates: fp.predicates.clone(), row_filter_factory: None, file_fields: fp.file_fields.clone(), }); - // Collect Avro records directly as apache_avro::Value, avoiding intermediate conversion. - let all_records: Vec = Reader::new(&file_bytes[..]) - .map_err(|e| Error::UnexpectedError { - message: format!("Failed to open Avro file: {e}"), - source: Some(Box::new(e)), - })? - .collect::, _>>() - .map_err(|e| Error::UnexpectedError { - message: format!("Failed to deserialize Avro record: {e}"), - source: Some(Box::new(e)), + let context = AvroReadContext { + fields: scan_fields, + schema: target_schema, + predicates, + batch_size, + }; + let selection = AvroRowSelection::new(row_selection); + if selection.is_finished() { + return Ok(stream::empty().boxed()); + } + + Ok(async move { + let mut input = BufferedAvroInput::new(reader, file_size); + let (header, marker) = input.read_header().await?; + let decoder = + run_decode_task(move || AvroBatchDecoder::new(header, selection, context)).await?; + let scan = AvroScan { + input, + marker, + decoder, + }; + Ok(stream::try_unfold(Some(scan), AvroScan::next_batch)) + } + .try_flatten_stream() + .boxed()) + } +} + +struct AvroScan { + input: BufferedAvroInput, + marker: [u8; 16], + decoder: AvroBatchDecoder, +} + +impl AvroScan { + fn has_more_to_scan(&self) -> bool { + !self.decoder.selection.is_finished() + && (self.decoder.row_position < self.decoder.block_end + || self.input.offset < self.input.file_size) + } + + // Owning the scan here lets us drop it before yielding a terminal error + // or the batch that exhausts the file or row selection. + async fn next_batch(state: Option) -> crate::Result)>> { + let Some(mut scan) = state else { + return Ok(None); + }; + while scan.has_more_to_scan() { + if scan.decoder.row_position == scan.decoder.block_end { + scan.read_next_selected_block().await?; + continue; + } + + let batch; + (scan.decoder, batch) = run_decode_task(move || scan.decoder.decode_batch()).await?; + if let Some(batch) = batch { + if scan.decoder.selection.ranges.is_none() { + scan.input.read_ahead = MAX_READ_SIZE; + } + let next = scan.has_more_to_scan().then_some(scan); + return Ok(Some((batch, next))); + } + } + Ok(None) + } + + async fn read_next_selected_block(&mut self) -> crate::Result<()> { + let mut encoded = Vec::new(); + while self.input.offset < self.input.file_size { + encoded.clear(); + let count = self.input.read_length(&mut encoded).await?; + let size = self.input.read_length(&mut encoded).await?; + let block_end = self + .decoder + .row_position + .checked_add(count) + .ok_or_else(|| avro_invalid("Avro row position overflow"))?; + let selected = count > 0 && self.decoder.selection.overlaps_block(block_end); + let payload_to_read = if selected { + size + } else { + self.input.advance(size)?; + 0 + }; + self.input + .read_into(payload_to_read + self.marker.len() as u64, &mut encoded) + .await?; + if !encoded.ends_with(&self.marker) { + return Err(avro_invalid("Avro sync marker mismatch")); + } + self.decoder.block_end = block_end; + if selected { + *self.decoder.input.lock().unwrap() = encoded.into(); + break; + } + self.decoder.row_position = block_end; + } + Ok(()) + } +} + +struct AvroReadContext { + fields: Vec, + schema: SchemaRef, + predicates: Option, + batch_size: usize, +} + +/// The Avro reader owns its synchronous input. Sharing the current bytes lets +/// the scan supply successive blocks while keeping the parsed writer schema. +struct AvroBlockInput { + bytes: Arc>, +} + +impl Read for AvroBlockInput { + fn read(&mut self, output: &mut [u8]) -> std::io::Result { + let mut bytes = self.bytes.lock().unwrap(); + let count = output.len().min(bytes.len()); + bytes.copy_to_slice(&mut output[..count]); + if bytes.is_empty() { + *bytes = Bytes::new(); + } + Ok(count) + } +} + +struct AvroBatchDecoder { + reader: Reader<'static, AvroBlockInput>, + input: Arc>, + row_position: u64, + block_end: u64, + selection: AvroRowSelection, + context: AvroReadContext, +} + +impl AvroBatchDecoder { + fn new( + header: Vec, + selection: AvroRowSelection, + context: AvroReadContext, + ) -> crate::Result { + let input = Arc::new(Mutex::new(header.into())); + let reader = Reader::new(AvroBlockInput { + bytes: input.clone(), + }) + .map_err(avro_decode_error)?; + Ok(Self { + reader, + input, + row_position: 0, + block_end: 0, + selection, + context, + }) + } + + fn decode_batch(mut self) -> crate::Result<(Self, Option)> { + let physical_rows = + (self.block_end - self.row_position).min(self.context.batch_size as u64); + let mut records = Vec::new(); + // Count discarded rows toward the batch limit so sparse scans stay cancellable. + for _ in 0..physical_rows { + if self.selection.is_finished() { + break; + } + let record = self + .reader + .next() + .ok_or_else(|| avro_invalid("Truncated Avro block"))? + .map_err(avro_decode_error)?; + if self.selection.select_row(self.row_position) { + records.push(record); + } + self.row_position += 1; + } + let batch = if records.is_empty() { + None + } else { + let context = &self.context; + let batch = records_to_batch(&records, &context.fields, &context.schema)?; + drop(records); + Some(match context.predicates.as_ref() { + Some(fp) => filter_record_batch_by_predicates(batch, fp, &context.fields)?, + None => batch, + }) + }; + Ok((self, batch)) + } +} + +async fn run_decode_task( + task: impl FnOnce() -> crate::Result + Send + 'static, +) -> crate::Result { + let Ok(runtime) = tokio::runtime::Handle::try_current() else { + // Yield between physical batches even when a sparse selection produces no output. + tokio::task::yield_now().await; + // apache-avro's bzip2 and xz decoders can panic on corrupt payloads. + return catch_unwind(AssertUnwindSafe(task)).map_err(|panic| Error::UnexpectedError { + message: format!( + "Avro decoding task panicked: {}", + panic_payload_message(panic) + ), + source: None, + })?; + }; + // An aborted task can remain in Tokio's blocking queue. The stream owns + // its captured input until a worker takes it. + let pending = Arc::new(Mutex::new(Some(task))); + let queued = Arc::downgrade(&pending); + let handle = runtime.spawn_blocking(move || { + let task = queued + .upgrade() + .and_then(|pending| pending.lock().unwrap().take()) + .ok_or_else(|| Error::UnexpectedError { + message: "Avro decoding task was canceled".to_string(), + source: None, })?; + task() + }); + AbortOnDropHandle::new(handle) + .await + .map_err(|error| Error::UnexpectedError { + message: "Avro decoding task failed".to_string(), + source: Some(Box::new(error)), + })? +} + +fn avro_invalid(message: &str) -> Error { + Error::DataInvalid { + message: message.to_string(), + source: None, + } +} - // Apply row selection filtering. - let records: Vec = match row_selection { - Some(ref ranges) => { - let total_rows = all_records.len(); - let mask = ranges_to_mask(total_rows, ranges); - all_records +fn avro_decode_error(error: apache_avro::Error) -> Error { + Error::UnexpectedError { + message: format!("Failed to decode Avro file: {error}"), + source: Some(Box::new(error)), + } +} + +/// Inclusive physical row ranges, consumed in ascending order. +struct AvroRowSelection { + ranges: Option>, +} + +impl AvroRowSelection { + fn new(ranges: Option>) -> Self { + Self { + ranges: ranges.map(|ranges| { + crate::table::merge_row_ranges(ranges) .into_iter() - .enumerate() - .filter(|(i, _)| mask[*i]) - .map(|(_, r)| r) + .filter_map(|range| range.intersect_inclusive(0, i64::MAX)) .collect() - } - None => all_records, + }), + } + } + + fn is_finished(&self) -> bool { + self.ranges.as_ref().is_some_and(VecDeque::is_empty) + } + + fn overlaps_block(&self, block_end: u64) -> bool { + self.ranges.as_ref().is_none_or(|ranges| { + ranges + .front() + .is_some_and(|range| (range.from() as u64) < block_end) + }) + } + + fn select_row(&mut self, position: u64) -> bool { + let Some(ranges) = &mut self.ranges else { + return true; + }; + let Some(range) = ranges.front() else { + return false; }; + let selected = (range.from() as u64) <= position; + if range.to() as u64 == position { + ranges.pop_front(); + } + selected + } +} + +/// The buffer starts at the current file offset, including after a skipped payload. +struct BufferedAvroInput { + reader: Box, + file_size: u64, + offset: u64, + buffer: Bytes, + read_ahead: u64, +} + +impl BufferedAvroInput { + fn new(reader: Box, file_size: u64) -> Self { + Self { + reader, + file_size, + offset: 0, + buffer: Bytes::new(), + read_ahead: INITIAL_READ_AHEAD, + } + } + + fn ensure_available(&self, len: u64) -> crate::Result<()> { + if len > self.file_size - self.offset { + return Err(avro_invalid("Truncated Avro file")); + } + Ok(()) + } - Ok(try_stream! { - for chunk in records.chunks(batch_size) { - let batch = records_to_batch(chunk, &scan_fields, &target_schema)?; - let batch = match predicates.as_ref() { - Some(fp) => crate::arrow::residual::filter_record_batch_by_predicates( - batch, - fp, - &scan_fields, - )?, - None => batch, - }; - yield batch; + async fn fill(&mut self, needed: u64) -> crate::Result<()> { + if self.buffer.is_empty() { + self.ensure_available(1)?; + let read_size = needed.clamp(self.read_ahead, MAX_READ_SIZE); + let end = self.offset.saturating_add(read_size).min(self.file_size); + self.buffer = self.reader.read(self.offset..end).await?; + if self.buffer.len() as u64 != end - self.offset { + return Err(avro_invalid("Short read from Avro file")); } + // Give cancellation a chance between immediately ready cached reads. + tokio::task::yield_now().await; } - .boxed()) + Ok(()) + } + + async fn read_into(&mut self, len: u64, out: &mut Vec) -> crate::Result<()> { + self.ensure_available(len)?; + let mut remaining = usize::try_from(len) + .map_err(|_| avro_invalid("Avro length exceeds addressable memory"))?; + out.try_reserve(remaining) + .map_err(|error| Error::UnexpectedError { + message: "Failed to allocate Avro buffer".to_string(), + source: Some(Box::new(error)), + })?; + while remaining > 0 { + self.fill(remaining as u64).await?; + let count = remaining.min(self.buffer.len()); + out.extend_from_slice(&self.buffer[..count]); + self.advance(count as u64)?; + remaining -= count; + } + Ok(()) + } + + fn advance(&mut self, len: u64) -> crate::Result<()> { + self.ensure_available(len)?; + if len < self.buffer.len() as u64 { + self.buffer.advance(len as usize); + } else { + // An empty slice would still own the exhausted storage window. + self.buffer = Bytes::new(); + } + self.offset += len; + Ok(()) + } + + async fn read_long(&mut self, encoded: &mut Vec) -> crate::Result { + let mut value = 0_u64; + for shift in (0..70).step_by(7) { + self.fill(1).await?; + let byte = self.buffer[0]; + self.advance(1)?; + encoded.push(byte); + if shift == 63 && byte > 1 { + return Err(avro_invalid("Avro long overflow")); + } + value |= u64::from(byte & 0x7f) << shift; + if byte & 0x80 == 0 { + return Ok((value >> 1) as i64 ^ -((value & 1) as i64)); + } + } + Err(avro_invalid("Invalid Avro long")) + } + + async fn read_length(&mut self, encoded: &mut Vec) -> crate::Result { + u64::try_from(self.read_long(encoded).await?) + .map_err(|_| avro_invalid("Negative Avro count or length")) + } + + async fn read_header(&mut self) -> crate::Result<(Vec, [u8; 16])> { + let mut header = Vec::new(); + self.read_into(4, &mut header).await?; + if header.as_slice() != b"Obj\x01" { + return Err(avro_invalid("Invalid Avro magic bytes")); + } + // OCF metadata is a map. Preserve its encoding, including + // negative-count map blocks and arbitrary binary user metadata. + loop { + let count = self.read_long(&mut header).await?; + if count == 0 { + break; + } + let metadata_end = if count < 0 { + let len = self.read_length(&mut header).await?; + self.ensure_available(len)?; + Some(self.offset + len) + } else { + None + }; + for _ in 0..count.unsigned_abs() { + for _ in 0..2 { + let len = self.read_length(&mut header).await?; + self.read_into(len, &mut header).await?; + } + } + if metadata_end.is_some_and(|end| end != self.offset) { + return Err(avro_invalid("Invalid Avro metadata block length")); + } + } + self.read_into(16, &mut header).await?; + let marker = *header.last_chunk().unwrap(); + Ok((header, marker)) } } @@ -198,26 +572,6 @@ fn value_as_bytes(v: &Value) -> Option<&[u8]> { } } -// --------------------------------------------------------------------------- -// Row ranges → boolean mask -// --------------------------------------------------------------------------- - -fn ranges_to_mask(total_rows: usize, ranges: &[RowRange]) -> Vec { - let mut mask = vec![false; total_rows]; - if total_rows == 0 { - return mask; - } - let file_end = total_rows as i64 - 1; - for r in ranges { - let from = r.from().max(0) as usize; - let to = (r.to().min(file_end) as usize).min(total_rows - 1); - for item in mask.iter_mut().take(to + 1).skip(from) { - *item = true; - } - } - mask -} - // --------------------------------------------------------------------------- // Avro records → Arrow RecordBatch conversion // --------------------------------------------------------------------------- @@ -920,31 +1274,6 @@ mod tests { assert!(get_field_at(&record, None).is_none()); } - // ----------------------------------------------------------------------- - // ranges_to_mask - // ----------------------------------------------------------------------- - - #[test] - fn test_ranges_to_mask_single_range() { - let ranges = vec![RowRange::new(1, 3)]; - let mask = ranges_to_mask(5, &ranges); - assert_eq!(mask, vec![false, true, true, true, false]); - } - - #[test] - fn test_ranges_to_mask_full_range() { - let ranges = vec![RowRange::new(0, 4)]; - let mask = ranges_to_mask(5, &ranges); - assert_eq!(mask, vec![true, true, true, true, true]); - } - - #[test] - fn test_ranges_to_mask_multiple_ranges() { - let ranges = vec![RowRange::new(0, 0), RowRange::new(3, 4)]; - let mask = ranges_to_mask(5, &ranges); - assert_eq!(mask, vec![true, false, false, true, true]); - } - // ----------------------------------------------------------------------- // build_column + records_to_batch // ----------------------------------------------------------------------- @@ -1363,235 +1692,4 @@ mod tests { let batch = records_to_batch(&records, &fields, &schema).unwrap(); assert_eq!(batch.num_rows(), 0); } - - // ----------------------------------------------------------------------- - // Exact residual predicate filtering on read - // ----------------------------------------------------------------------- - - use crate::spec::{Datum, Predicate, PredicateOperator}; - - fn leaf( - index: usize, - data_type: DataType, - op: PredicateOperator, - literals: Vec, - ) -> Predicate { - Predicate::Leaf { - column: format!("c{index}"), - index, - data_type, - op, - literals, - } - } - - /// Write the given records into an in-memory Avro OCF file, returning the - /// encoded bytes. The Avro schema mirrors `age: long, name: string`. - fn write_avro_age_name(rows: &[(i64, &str)]) -> Vec { - use apache_avro::{Codec, Schema, Writer}; - - let schema = Schema::parse_str( - r#"{"type": "record", "name": "row", "fields": [ - {"name": "age", "type": "long"}, - {"name": "name", "type": "string"} - ]}"#, - ) - .unwrap(); - let mut writer = Writer::with_codec(&schema, Vec::new(), Codec::Null); - for (age, name) in rows { - let mut record = apache_avro::types::Record::new(&schema).unwrap(); - record.put("age", *age); - record.put("name", *name); - writer.append(record).unwrap(); - } - writer.into_inner().unwrap() - } - - #[tokio::test] - async fn avro_reader_applies_exact_residual_filter_int() { - use crate::btree::test_util::BytesFileRead; - use crate::spec::{BigIntType, VarCharType}; - use futures::TryStreamExt; - - let bytes = write_avro_age_name(&[(10, "a"), (20, "b"), (30, "c"), (40, "d"), (50, "e")]); - - let age = DataField::new(0, "age".to_string(), DataType::BigInt(BigIntType::new())); - let name = DataField::new( - 1, - "name".to_string(), - DataType::VarChar(VarCharType::new(50).unwrap()), - ); - let read_fields = vec![age.clone(), name.clone()]; - let file_fields = vec![age.clone(), name.clone()]; - - // age > 25 → [30, 40, 50]. - let predicates = FilePredicates { - predicates: vec![leaf( - 0, - DataType::BigInt(BigIntType::new()), - PredicateOperator::Gt, - vec![Datum::Long(25)], - )], - row_filter_factory: None, - file_fields, - }; - - let batches = AvroFormatReader - .read_batch_stream( - Box::new(BytesFileRead(bytes.clone().into())), - bytes.len() as u64, - &read_fields, - Some(&predicates), - None, - None, - ) - .await - .unwrap() - .try_collect::>() - .await - .unwrap(); - - let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); - assert_eq!(total_rows, 3); - let ages: Vec = batches - .iter() - .flat_map(|b| { - b.column(0) - .as_any() - .downcast_ref::() - .unwrap() - .values() - .to_vec() - }) - .collect(); - assert_eq!(ages, vec![30, 40, 50]); - } - - #[tokio::test] - async fn avro_reader_filters_on_non_projected_predicate_column() { - use crate::btree::test_util::BytesFileRead; - use crate::spec::{BigIntType, VarCharType}; - use futures::TryStreamExt; - - let bytes = write_avro_age_name(&[(10, "a"), (20, "b"), (30, "c"), (40, "d"), (50, "e")]); - - let age = DataField::new(0, "age".to_string(), DataType::BigInt(BigIntType::new())); - let name = DataField::new( - 1, - "name".to_string(), - DataType::VarChar(VarCharType::new(50).unwrap()), - ); - // Project ONLY `name`; the predicate is on `age`, which is NOT projected. - let read_fields = vec![name.clone()]; - let file_fields = vec![age.clone(), name.clone()]; - - // age > 25 → rows c, d, e (age is a BigInt/long, so literal is Datum::Long). - let predicates = FilePredicates { - predicates: vec![leaf( - 0, - DataType::BigInt(BigIntType::new()), - PredicateOperator::Gt, - vec![Datum::Long(25)], - )], - row_filter_factory: None, - file_fields, - }; - - let batches = AvroFormatReader - .read_batch_stream( - Box::new(BytesFileRead(bytes.clone().into())), - bytes.len() as u64, - &read_fields, - Some(&predicates), - None, - None, - ) - .await - .unwrap() - .try_collect::>() - .await - .unwrap(); - - // Assert on the FILTERED rows/values, not an exact column set (the batch - // may contain the extra `age` column — DataFileReader projects it away). - let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); - assert_eq!(total_rows, 3); - let names: Vec = batches - .iter() - .flat_map(|b| { - let col = b - .column_by_name("name") - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - (0..col.len()) - .map(|i| col.value(i).to_string()) - .collect::>() - }) - .collect(); - assert_eq!(names, vec!["c", "d", "e"]); - } - - #[tokio::test] - async fn avro_reader_applies_exact_residual_filter_like() { - use crate::btree::test_util::BytesFileRead; - use crate::spec::{BigIntType, VarCharType}; - use futures::TryStreamExt; - - let bytes = write_avro_age_name(&[ - (10, "apple"), - (20, "banana"), - (30, "apricot"), - (40, "cherry"), - ]); - - let age = DataField::new(0, "age".to_string(), DataType::BigInt(BigIntType::new())); - let name = DataField::new( - 1, - "name".to_string(), - DataType::VarChar(VarCharType::new(50).unwrap()), - ); - let read_fields = vec![age.clone(), name.clone()]; - - // name LIKE 'a%' → ["apple", "apricot"]. - let predicates = FilePredicates { - predicates: vec![leaf( - 1, - DataType::VarChar(VarCharType::new(50).unwrap()), - PredicateOperator::Like, - vec![Datum::String("a%".to_string())], - )], - row_filter_factory: None, - file_fields: vec![age, name], - }; - - let batches = AvroFormatReader - .read_batch_stream( - Box::new(BytesFileRead(bytes.clone().into())), - bytes.len() as u64, - &read_fields, - Some(&predicates), - None, - None, - ) - .await - .unwrap() - .try_collect::>() - .await - .unwrap(); - - let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); - assert_eq!(total_rows, 2); - let names: Vec = batches - .iter() - .flat_map(|b| { - let col = b.column(1).as_any().downcast_ref::().unwrap(); - (0..col.len()) - .map(|i| col.value(i).to_string()) - .collect::>() - }) - .collect(); - assert_eq!(names, vec!["apple", "apricot"]); - } } diff --git a/crates/paimon/src/arrow/format/avro/streaming_tests.rs b/crates/paimon/src/arrow/format/avro/streaming_tests.rs new file mode 100644 index 000000000..ed185323e --- /dev/null +++ b/crates/paimon/src/arrow/format/avro/streaming_tests.rs @@ -0,0 +1,1009 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::*; +use crate::spec::{BigIntType, Datum, DoubleType, Predicate, PredicateOperator, VarCharType}; +use apache_avro::{from_avro_datum, Codec, Schema, Writer}; +use futures::TryStreamExt; +use std::io::Cursor; +use std::ops::Range; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Mutex; + +#[derive(Default)] +struct ReadStats { + requests: Mutex>>, + last_window: Mutex, + dropped: AtomicBool, +} + +impl ReadStats { + fn bytes_read(&self) -> u64 { + self.requests + .lock() + .unwrap() + .iter() + .map(|r| r.end - r.start) + .sum() + } +} + +struct CountingRead { + bytes: Vec, + stats: Arc, + fail_at: Option, +} + +impl Drop for CountingRead { + fn drop(&mut self) { + self.stats.dropped.store(true, Ordering::SeqCst); + } +} + +#[async_trait] +impl FileRead for CountingRead { + async fn read(&self, range: Range) -> crate::Result { + self.stats.requests.lock().unwrap().push(range.clone()); + if self.fail_at.is_some_and(|offset| range.end > offset) { + return Err(avro_invalid("injected Avro read failure")); + } + let start = (range.start as usize).min(self.bytes.len()); + let end = (range.end as usize).min(self.bytes.len()); + let window = Bytes::copy_from_slice(&self.bytes[start..end]); + *self.stats.last_window.lock().unwrap() = window.clone(); + Ok(window) + } +} + +fn counting_read(bytes: Vec, fail_at: Option) -> (Box, Arc) { + let stats = Arc::new(ReadStats::default()); + ( + Box::new(CountingRead { + bytes, + stats: stats.clone(), + fail_at, + }), + stats, + ) +} + +fn fields() -> Vec { + vec![ + DataField::new(0, "id".into(), DataType::BigInt(BigIntType::new())), + DataField::new( + 1, + "payload".into(), + DataType::VarChar(VarCharType::string_type()), + ), + ] +} + +fn schema() -> Schema { + Schema::parse_str( + r#"{"type":"record","name":"row","fields":[ + {"name":"id","type":"long"}, {"name":"payload","type":"string"} + ]}"#, + ) + .unwrap() +} + +fn record(id: i64, payload_size: usize) -> Value { + Value::Record(vec![ + ("id".into(), Value::Long(id)), + ( + "payload".into(), + Value::String(format!("{id:08}{}", "x".repeat(payload_size))), + ), + ]) +} + +fn supported_codecs() -> [Codec; 6] { + [ + Codec::Null, + Codec::Deflate(Default::default()), + Codec::Snappy, + Codec::Zstandard(Default::default()), + Codec::Bzip2(Default::default()), + Codec::Xz(Default::default()), + ] +} + +fn file(codec: Codec, blocks: usize, rows_per_block: usize, payload_size: usize) -> Vec { + let schema = schema(); + let mut writer = Writer::builder() + .schema(&schema) + .writer(Vec::new()) + .codec(codec) + .block_size(2 * 1024 * 1024) + .marker([7; 16]) + .build(); + for block in 0..blocks { + for row in 0..rows_per_block { + writer + .append(record((block * rows_per_block + row) as i64, payload_size)) + .unwrap(); + } + writer.flush().unwrap(); + } + writer.into_inner().unwrap() +} + +struct BlockLocation { + start: usize, + payload: Range, + end: usize, +} + +fn layout(bytes: &[u8]) -> (usize, Vec) { + let mut cursor = Cursor::new(bytes); + drop(Reader::new(&mut cursor).unwrap()); + let header_end = cursor.position() as usize; + let mut blocks = Vec::new(); + while (cursor.position() as usize) < bytes.len() { + let start = cursor.position() as usize; + from_avro_datum(&Schema::Long, &mut cursor, None).unwrap(); + let Value::Long(size) = from_avro_datum(&Schema::Long, &mut cursor, None).unwrap() else { + unreachable!(); + }; + let payload_start = cursor.position() as usize; + let payload_end = payload_start + size as usize; + blocks.push(BlockLocation { + start, + payload: payload_start..payload_end, + end: payload_end + 16, + }); + cursor.set_position((payload_end + 16) as u64); + } + (header_end, blocks) +} + +async fn stream( + bytes: Vec, + batch_size: usize, + selection: Option>, +) -> (ArrowRecordBatchStream, Arc) { + let size = bytes.len() as u64; + let (input, stats) = counting_read(bytes, None); + let stream = AvroFormatReader + .read_batch_stream(input, size, &fields(), None, Some(batch_size), selection) + .await + .unwrap(); + (stream, stats) +} + +fn ids(batches: &[RecordBatch]) -> Vec { + batches + .iter() + .flat_map(|batch| { + batch + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .iter() + .map(|id| id.expect("non-null id")) + }) + .collect() +} + +async fn terminal_error(mut stream: ArrowRecordBatchStream, stats: &ReadStats) -> Error { + let error = stream + .by_ref() + .try_for_each(|_| async { Ok(()) }) + .await + .unwrap_err(); + assert!(stats.dropped.load(Ordering::SeqCst)); + { + let window = stats.last_window.lock().unwrap(); + assert!( + window.is_empty() || window.is_unique(), + "the errored stream retained its storage window" + ); + } + assert!(stream.try_next().await.unwrap().is_none()); + error +} + +#[tokio::test] +async fn dropping_before_or_after_the_first_batch_stops_io() { + let bytes = file(Codec::Null, 8, 16, 8192); + let (_, blocks) = layout(&bytes); + let size = bytes.len() as u64; + let (unpolled, stats) = stream(bytes.clone(), 3, None).await; + drop(unpolled); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert_eq!(stats.bytes_read(), 0); + + for batch_size in [3, DEFAULT_BATCH_SIZE] { + let (mut stream, stats) = stream(bytes.clone(), batch_size, None).await; + assert_eq!(stats.bytes_read(), 0, "stream construction must be lazy"); + let first = stream.try_next().await.unwrap().unwrap(); + assert_eq!( + ids(&[first]), + (0..batch_size.min(16) as i64).collect::>() + ); + let read = stats.bytes_read(); + assert!(read < size); + assert!(read <= blocks[0].end as u64 + INITIAL_READ_AHEAD); + assert_eq!(stats.requests.lock().unwrap()[0], 0..INITIAL_READ_AHEAD); + tokio::task::yield_now().await; + assert_eq!(stats.bytes_read(), read); + drop(stream); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(stats.last_window.lock().unwrap().is_unique()); + tokio::task::yield_now().await; + assert_eq!(stats.bytes_read(), read); + } +} + +#[test] +fn sparse_selection_remains_cancellable_with_buffered_input_without_tokio() { + futures::executor::block_on(async { + let bytes = file(Codec::Null, 1, 256, 8); + let size = bytes.len() as u64; + let selection = Some(vec![RowRange::new(0, 0), RowRange::new(255, 255)]); + let (mut stream, stats) = stream(bytes, 4, selection).await; + assert_eq!(ids(&[stream.try_next().await.unwrap().unwrap()]), [0]); + assert_eq!(stats.bytes_read(), size); + + let mut next = Box::pin(stream.try_next()); + assert!(futures::poll!(&mut next).is_pending()); + assert!(futures::poll!(&mut next).is_pending()); + drop(next); + drop(stream); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(stats.last_window.lock().unwrap().is_unique()); + assert_eq!(stats.requests.lock().unwrap().len(), 1); + }); +} + +#[tokio::test] +async fn full_scan_coalesces_reads_for_the_first_block_and_subsequent_blocks() { + let bytes = file(Codec::Null, 12, 128, 8192); + let size = bytes.len() as u64; + let (mut stream, stats) = stream(bytes, 512, None).await; + let first = stream.try_next().await.unwrap().unwrap(); + let prefix_bytes = stats.bytes_read(); + let prefix_requests = stats.requests.lock().unwrap().len(); + assert!(prefix_bytes < size); + assert_eq!(prefix_requests, 2, "header and first 1 MiB block"); + assert!( + stats.last_window.lock().unwrap().is_unique(), + "the stream retained the exhausted storage window" + ); + + let mut rows = first.num_rows(); + while rows < 12 * 128 { + rows += stream.try_next().await.unwrap().unwrap().num_rows(); + } + assert_eq!(rows, 12 * 128); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(stream.try_next().await.unwrap().is_none()); + assert_eq!(stats.bytes_read(), size); + let requests = stats.requests.lock().unwrap(); + assert_eq!( + (requests.len() - prefix_requests) as u64, + (size - prefix_bytes).div_ceil(MAX_READ_SIZE) + ); + assert!(requests[prefix_requests..] + .iter() + .all(|r| r.end - r.start <= MAX_READ_SIZE)); +} + +#[cfg(feature = "storage-memory")] +#[test] +fn reads_java_avro_from_memory_without_tokio() { + futures::executor::block_on(async { + let bytes = include_bytes!("../../../../testdata/avro_codec/java_bzip2.avro"); + let size = bytes.len() as u64; + let io = crate::io::FileIOBuilder::new("memory").build().unwrap(); + let path = "memory:/without-tokio.avro"; + let mut output = io.new_output(path).unwrap().writer().await.unwrap(); + output.write(Bytes::from_static(bytes)).await.unwrap(); + output.close().await.unwrap(); + let reader = io.new_input(path).unwrap().reader().await.unwrap(); + let batches = AvroFormatReader + .read_batch_stream(Box::new(reader), size, &fields()[..1], None, Some(2), None) + .await + .unwrap() + .try_collect::>() + .await + .unwrap(); + assert_eq!(ids(&batches), [-7, 0, i64::MAX]); + }); +} + +#[test] +fn dropping_decode_future_releases_and_aborts_queued_work() { + let runtime = tokio::runtime::Builder::new_current_thread() + .max_blocking_threads(1) + .build() + .unwrap(); + let task_ran = Arc::new(AtomicBool::new(false)); + runtime.block_on(async { + let (started, running) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::channel(); + let occupied = tokio::task::spawn_blocking(move || { + started.send(()).unwrap(); + let _ = wait.recv(); + }); + running.await.unwrap(); + let worker_flag = task_ran.clone(); + let mut decode = Box::pin(run_decode_task(move || { + worker_flag.store(true, Ordering::SeqCst); + Ok(()) + })); + assert!(futures::poll!(&mut decode).is_pending()); + drop(decode); + assert_eq!( + Arc::strong_count(&task_ran), + 1, + "queued decode retains its input" + ); + release.send(()).unwrap(); + occupied.await.unwrap(); + tokio::task::spawn_blocking(|| {}).await.unwrap(); + }); + assert!(!task_ran.load(Ordering::SeqCst)); +} + +#[tokio::test] +async fn dropping_during_pending_or_ready_reads_stops_further_io() { + struct NotifyingRead { + stats: Arc, + entered: Arc, + bytes: Option, + } + impl Drop for NotifyingRead { + fn drop(&mut self) { + self.stats.dropped.store(true, Ordering::SeqCst); + } + } + #[async_trait] + impl FileRead for NotifyingRead { + async fn read(&self, range: Range) -> crate::Result { + let count = { + let mut requests = self.stats.requests.lock().unwrap(); + requests.push(range.clone()); + requests.len() + }; + if self.bytes.is_none() || count == 2 { + self.entered.notify_one(); + } + match &self.bytes { + Some(bytes) => Ok(bytes.slice(range.start as usize..range.end as usize)), + None => futures::future::pending().await, + } + } + } + let bytes = Bytes::from(file( + Codec::Null, + 1, + 1, + (MAX_READ_SIZE + 2 * INITIAL_READ_AHEAD) as usize, + )); + for ready in [false, true] { + let stats = Arc::new(ReadStats::default()); + let entered = Arc::new(tokio::sync::Notify::new()); + let mut stream = AvroFormatReader + .read_batch_stream( + Box::new(NotifyingRead { + stats: stats.clone(), + entered: entered.clone(), + bytes: ready.then(|| bytes.clone()), + }), + bytes.len() as u64, + &fields(), + None, + Some(2), + None, + ) + .await + .unwrap(); + tokio::select! { + biased; + result = stream.try_next() => panic!("unexpected read completion: {result:?}"), + _ = entered.notified() => {} + } + drop(stream); + assert!(stats.dropped.load(Ordering::SeqCst)); + let requests = stats.requests.lock().unwrap(); + if ready { + assert_eq!( + requests[..], + [ + 0..INITIAL_READ_AHEAD, + INITIAL_READ_AHEAD..INITIAL_READ_AHEAD + MAX_READ_SIZE + ] + ); + } else { + assert_eq!(requests.len(), 1); + assert_eq!(requests[0], 0..INITIAL_READ_AHEAD); + } + } +} + +#[tokio::test] +async fn full_scans_and_cross_block_ranges_preserve_physical_order_for_every_codec() { + for codec in supported_codecs() { + for payload_size in [32, INITIAL_READ_AHEAD as usize + 17] { + let bytes = file(codec, 5, 7, payload_size); + let expected_records = Reader::new(bytes.as_slice()) + .unwrap() + .collect::, _>>() + .unwrap(); + let fields = fields(); + let schema = build_target_arrow_schema(&fields).unwrap(); + let expected = records_to_batch(&expected_records, &fields, &schema).unwrap(); + let (full, _) = stream(bytes.clone(), 3, None).await; + let batches = full.try_collect::>().await.unwrap(); + assert!(batches.iter().all(|batch| batch.num_rows() <= 3)); + assert_eq!( + arrow_select::concat::concat_batches(&schema, &batches).unwrap(), + expected + ); + + let ranges = vec![ + RowRange::new(19, 23), + RowRange::new(5, 9), + RowRange::new(7, 11), + RowRange::new(-4, -1), + RowRange::new(-1, 0), + RowRange::new(33, i64::MAX), + ]; + let (selected, _) = stream(bytes, 3, Some(ranges)).await; + let batches = selected.try_collect::>().await.unwrap(); + assert_eq!( + ids(&batches), + [ + vec![0], + (5..=11).collect(), + (19..=23).collect(), + vec![33, 34] + ] + .concat() + ); + } + } +} + +#[tokio::test] +async fn projection_preserves_missing_fields_numeric_widening_and_null_second_unions() { + let schema = Schema::parse_str( + r#"{"type":"record","name":"projected","fields":[ + {"name":"skipped","type":"string"}, + {"name":"value","type":["int","null"]}, + {"name":"number","type":"float"} + ]}"#, + ) + .unwrap(); + let mut writer = Writer::with_codec(&schema, Vec::new(), Codec::Zstandard(Default::default())); + for row in 0..6 { + let value = if row % 2 == 0 { + Value::Union(0, Box::new(Value::Int(row))) + } else { + Value::Union(1, Box::new(Value::Null)) + }; + writer + .append(Value::Record(vec![ + ("skipped".into(), Value::String(format!("row-{row}"))), + ("value".into(), value), + ("number".into(), Value::Float(row as f32 + 0.5)), + ])) + .unwrap(); + if row % 2 == 1 { + writer.flush().unwrap(); + } + } + let bytes = writer.into_inner().unwrap(); + let fields = vec![ + DataField::new(2, "number".into(), DataType::Double(DoubleType::new())), + DataField::new(3, "missing".into(), DataType::Int(IntType::new())), + DataField::new(1, "value".into(), DataType::BigInt(BigIntType::new())), + ]; + let arrow_schema = build_target_arrow_schema(&fields).unwrap(); + let expected = RecordBatch::try_new( + arrow_schema.clone(), + vec![ + Arc::new(Float64Array::from(vec![0.5, 1.5, 2.5, 3.5, 4.5, 5.5])), + Arc::new(Int32Array::new_null(6)), + Arc::new(Int64Array::from(vec![ + Some(0), + None, + Some(2), + None, + Some(4), + None, + ])), + ], + ) + .unwrap(); + let size = bytes.len() as u64; + let (reader, _) = counting_read(bytes, None); + let batches = AvroFormatReader + .read_batch_stream(reader, size, &fields, None, Some(1), None) + .await + .unwrap() + .try_collect::>() + .await + .unwrap(); + assert_eq!( + arrow_select::concat::concat_batches(&arrow_schema, &batches).unwrap(), + expected + ); +} + +#[tokio::test] +async fn selection_skips_whole_block_payloads_and_stops_at_the_last_range() { + let mut bytes = file(Codec::Null, 8, 16, 8192); + let (_, blocks) = layout(&bytes); + bytes[blocks[1].payload.clone()].fill(0xff); + let ranges = vec![RowRange::new(0, 0), RowRange::new(48, 50)]; + let (selected, stats) = stream(bytes.clone(), 4, Some(ranges.clone())).await; + let batches = selected.try_collect::>().await.unwrap(); + assert_eq!(ids(&batches), [0, 48, 49, 50]); + { + let requests = stats.requests.lock().unwrap(); + for block in &blocks[1..3] { + let middle = block.payload.start as u64 + INITIAL_READ_AHEAD; + assert!(!requests.iter().any(|range| range.contains(&middle))); + } + assert!(requests + .iter() + .all(|range| range.end <= blocks[3].end as u64 + INITIAL_READ_AHEAD)); + } + bytes[blocks[2].payload.end] ^= 1; + let (selected, stats) = stream(bytes, 4, Some(ranges)).await; + let error = terminal_error(selected, &stats).await; + assert!(error.to_string().contains("sync marker mismatch")); +} + +#[tokio::test] +async fn empty_selections_and_header_only_files_release_the_reader() { + let bytes = file(Codec::Null, 1, 4, 8); + let (header_end, _) = layout(&bytes); + for ranges in [vec![], vec![RowRange::new(-5, -1)]] { + let (mut empty, stats) = stream(bytes.clone(), 2, Some(ranges)).await; + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(empty.try_next().await.unwrap().is_none()); + assert_eq!(stats.bytes_read(), 0); + } + for (bytes, ranges) in [ + (bytes[..header_end].to_vec(), None), + (bytes, Some(vec![RowRange::new(10, 20)])), + ] { + let (mut empty, stats) = stream(bytes, 2, ranges).await; + assert!(empty.try_next().await.unwrap().is_none()); + assert!(stats.dropped.load(Ordering::SeqCst)); + } +} + +#[tokio::test] +async fn empty_projection_counts_regular_and_zero_width_records() { + fn zero_width_blocks(counts: &[i64]) -> Vec { + let schema = Schema::parse_str(r#"{"type":"record","name":"empty","fields":[]}"#).unwrap(); + let mut bytes = Writer::new(&schema, Vec::new()).into_inner().unwrap(); + let marker: [u8; 16] = *bytes.last_chunk().unwrap(); + for &count in counts { + bytes.extend(apache_avro::to_avro_datum(&Schema::Long, Value::Long(count)).unwrap()); + bytes.push(0); + bytes.extend(marker); + } + bytes + } + + let large_block = zero_width_blocks(&[i64::MAX, 1]); + for (bytes, batch_size, position, expected_rows) in [ + (file(Codec::Null, 1, 4, 8), 0, None, 4), + (zero_width_blocks(&[4]), 0, None, 4), + (zero_width_blocks(&[0, 2, 0, 2]), 0, None, 4), + (large_block.clone(), usize::MAX, Some(0), 1), + (large_block, usize::MAX, Some(i64::MAX), 1), + ] { + let size = bytes.len() as u64; + let (reader, stats) = counting_read(bytes, None); + let selection = position.map(|row| vec![RowRange::new(row, row)]); + let mut stream = AvroFormatReader + .read_batch_stream(reader, size, &[], None, Some(batch_size), selection) + .await + .unwrap(); + for _ in 0..expected_rows { + let batch = stream.try_next().await.unwrap().unwrap(); + assert_eq!((batch.num_columns(), batch.num_rows()), (0, 1)); + } + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(stream.try_next().await.unwrap().is_none()); + } +} + +#[tokio::test] +async fn predicates_preserve_projection_and_selected_physical_positions() { + let bytes = file(Codec::Null, 5, 7, 8); + let size = bytes.len() as u64; + let fields = fields(); + for (index, op, literal, matching_rows) in [ + ( + 0, + PredicateOperator::Gt, + Datum::Long(8), + (9..35).collect::>(), + ), + (0, PredicateOperator::Gt, Datum::Long(i64::MAX), vec![]), + ( + 1, + PredicateOperator::Like, + Datum::String("0000002%".into()), + (20..30).collect(), + ), + ( + 1, + PredicateOperator::Like, + Datum::String("missing%".into()), + vec![], + ), + ] { + let predicates = FilePredicates { + predicates: vec![Predicate::Leaf { + column: fields[index].name().to_string(), + index, + data_type: fields[index].data_type().clone(), + op, + literals: vec![literal], + }], + row_filter_factory: None, + file_fields: fields.clone(), + }; + let without_predicate_column = if index == 0 { + &fields[1..] + } else { + &fields[..1] + }; + for read_fields in [&fields[..], without_predicate_column] { + let schema = build_target_arrow_schema(read_fields).unwrap(); + for selection in [ + None, + Some(vec![RowRange::new(5, 11), RowRange::new(20, 22)]), + ] { + let expected_records: Vec<_> = matching_rows + .iter() + .copied() + .filter(|&row| { + selection.as_ref().is_none_or(|ranges| { + ranges + .iter() + .any(|range| row >= range.from() && row <= range.to()) + }) + }) + .map(|row| record(row, 8)) + .collect(); + let (reader, _) = counting_read(bytes.clone(), None); + let batches = AvroFormatReader + .read_batch_stream( + reader, + size, + read_fields, + Some(&predicates), + Some(3), + selection, + ) + .await + .unwrap() + .map_ok(|batch| { + let columns: Vec<_> = read_fields + .iter() + .map(|field| batch.schema_ref().index_of(field.name()).unwrap()) + .collect(); + batch.project(&columns).unwrap() + }) + .try_collect::>() + .await + .unwrap(); + assert_eq!( + arrow_select::concat::concat_batches(&schema, &batches).unwrap(), + records_to_batch(&expected_records, read_fields, &schema).unwrap() + ); + } + } + } +} + +#[tokio::test] +async fn nested_and_logical_values_match_eager_conversion() { + use crate::spec::{ArrayType, DateType, DecimalType, TimestampType}; + let schema = Schema::parse_str( + r#"{"type":"record","name":"nested","fields":[ + {"name":"row","type":["null",{"type":"record","name":"child","fields":[ + {"name":"items","type":{"type":"array","items":["null","int"]}} + ]}]}, + {"name":"other","type":"child"}, + {"name":"map","type":{"type":"map","values":"long"}}, + {"name":"date","type":{"type":"int","logicalType":"date"}}, + {"name":"time","type":{"type":"long","logicalType":"timestamp-micros"}}, + {"name":"amount","type":{"type":"bytes","logicalType":"decimal","precision":10,"scale":2}} + ]}"#, + ) + .unwrap(); + let child_type = |items_id| { + DataType::Row(RowType::new(vec![DataField::new( + items_id, + "items".into(), + DataType::Array(ArrayType::new(DataType::Int(IntType::new()))), + )])) + }; + let fields = vec![ + DataField::new(0, "row".into(), child_type(5)), + DataField::new(6, "other".into(), child_type(7)), + DataField::new( + 1, + "map".into(), + DataType::Map(MapType::new( + DataType::VarChar(VarCharType::string_type()), + DataType::BigInt(BigIntType::new()), + )), + ), + DataField::new(2, "date".into(), DataType::Date(DateType::new())), + DataField::new( + 3, + "time".into(), + DataType::Timestamp(TimestampType::new(6).unwrap()), + ), + DataField::new( + 4, + "amount".into(), + DataType::Decimal(DecimalType::new(10, 2).unwrap()), + ), + ]; + let mut writer = Writer::with_codec(&schema, Vec::new(), Codec::Snappy); + for row in 0..9 { + let child = Value::Record(vec![( + "items".into(), + Value::Array(vec![ + Value::Union(0, Box::new(Value::Null)), + Value::Union(1, Box::new(Value::Int(row))), + ]), + )]); + writer + .append(Value::Record(vec![ + ( + "row".into(), + if row % 2 == 0 { + Value::Union(0, Box::new(Value::Null)) + } else { + Value::Union(1, Box::new(child.clone())) + }, + ), + ("other".into(), child), + ( + "map".into(), + Value::Map(std::collections::HashMap::from([( + "key".into(), + Value::Long(row as i64), + )])), + ), + ("date".into(), Value::Date(row)), + ( + "time".into(), + Value::TimestampMicros(row as i64 * 1_000_001), + ), + ( + "amount".into(), + Value::Decimal(apache_avro::Decimal::from(vec![row as u8])), + ), + ])) + .unwrap(); + if row % 3 == 2 { + writer.flush().unwrap(); + } + } + let bytes = writer.into_inner().unwrap(); + let records = Reader::new(bytes.as_slice()) + .unwrap() + .collect::, _>>() + .unwrap(); + let arrow_schema = build_target_arrow_schema(&fields).unwrap(); + let expected = records_to_batch(&records, &fields, &arrow_schema).unwrap(); + let size = bytes.len() as u64; + let (reader, _) = counting_read(bytes, None); + let batches = AvroFormatReader + .read_batch_stream(reader, size, &fields, None, Some(2), None) + .await + .unwrap() + .try_collect::>() + .await + .unwrap(); + assert_eq!( + arrow_select::concat::concat_batches(&arrow_schema, &batches).unwrap(), + expected + ); +} + +#[tokio::test] +async fn corrupt_consumed_blocks_and_truncated_framing_return_errors() { + let good = file(Codec::Null, 3, 4, 16); + let (header_end, blocks) = layout(&good); + let mut cases = vec![ + Vec::new(), + good[..3].to_vec(), + good[..header_end - 1].to_vec(), + ]; + let mut bad_magic = good.clone(); + bad_magic[0] ^= 1; + cases.push(bad_magic); + let mut bad_codec = good.clone(); + let codec_offset = good[..header_end] + .windows(4) + .position(|bytes| bytes == b"null") + .unwrap(); + bad_codec[codec_offset..codec_offset + 4].copy_from_slice(b"nope"); + cases.push(bad_codec); + let mut missing_schema = b"Obj\x01\x00".to_vec(); + missing_schema.extend([7; 16]); + cases.push(missing_schema); + for end in [ + blocks[0].start + 1, + blocks[0].payload.end - 1, + blocks[0].end - 1, + ] { + cases.push(good[..end].to_vec()); + } + let mut bad_marker = good.clone(); + bad_marker[blocks[0].payload.end] ^= 1; + cases.push(bad_marker); + let mut too_many_records = good.clone(); + too_many_records[blocks[0].start] = 5 << 1; + cases.push(too_many_records); + let mut negative_count = good[..header_end].to_vec(); + negative_count.push(1); + cases.push(negative_count); + let mut negative_size = good[..header_end].to_vec(); + negative_size.extend([2, 1]); + cases.push(negative_size); + let mut overflowing_count = good[..header_end].to_vec(); + overflowing_count.extend([0xff; 10]); + cases.push(overflowing_count); + let mut partial_next_count = good.clone(); + partial_next_count.push(0x80); + cases.push(partial_next_count); + for bytes in cases { + let (stream, stats) = stream(bytes, 2, None).await; + terminal_error(stream, &stats).await; + } +} + +#[test] +fn corrupt_payloads_return_errors_on_both_executors() { + async fn check_codecs() { + for codec in supported_codecs() { + let mut bytes = file(codec, 2, 4, 16); + let (_, blocks) = layout(&bytes); + bytes[blocks[0].payload.clone()].fill(0xff); + let (stream, stats) = stream(bytes, 2, None).await; + terminal_error(stream, &stats).await; + } + } + + futures::executor::block_on(check_codecs()); + tokio::runtime::Builder::new_current_thread() + .build() + .unwrap() + .block_on(check_codecs()); +} + +#[tokio::test] +async fn storage_errors_and_short_reads_release_the_reader() { + let bytes = file(Codec::Null, 2, 16, 8192); + let size = bytes.len() as u64; + for (bytes, fail_at) in [ + (bytes.clone(), Some(0)), + (bytes.clone(), Some(INITIAL_READ_AHEAD)), + (bytes.clone(), Some(3 * INITIAL_READ_AHEAD)), + (bytes[..100].to_vec(), None), + ] { + let (reader, stats) = counting_read(bytes, fail_at); + let stream = AvroFormatReader + .read_batch_stream(reader, size, &fields(), None, Some(2), None) + .await + .unwrap(); + let error = terminal_error(stream, &stats).await; + let expected = if fail_at.is_some() { + "injected Avro read failure" + } else { + "Short read from Avro file" + }; + assert!(matches!(error, Error::DataInvalid { message, .. } if message == expected)); + } +} + +#[tokio::test] +async fn corruption_is_reported_only_when_the_affected_block_or_record_is_consumed() { + let bytes = file(Codec::Null, 3, 4, 32); + let (_, blocks) = layout(&bytes); + let mut bad_marker = bytes.clone(); + bad_marker[blocks[1].payload.end] ^= 1; + let mut bad_record = bytes; + let record_size = apache_avro::to_avro_datum(&schema(), record(0, 32)) + .unwrap() + .len(); + let third_record = blocks[0].payload.start + 2 * record_size; + bad_record[third_record..third_record + record_size].fill(0xff); + for (bytes, prefix_rows) in [(bad_marker, 4), (bad_record, 2)] { + let (mut selected, stats) = stream( + bytes.clone(), + prefix_rows, + Some(vec![RowRange::new(0, prefix_rows as i64 - 1)]), + ) + .await; + assert_eq!( + ids(&[selected.try_next().await.unwrap().unwrap()]), + (0..prefix_rows as i64).collect::>() + ); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(selected.try_next().await.unwrap().is_none()); + let (mut stream, stats) = stream(bytes, prefix_rows, None).await; + assert_eq!( + ids(&[stream.try_next().await.unwrap().unwrap()]), + (0..prefix_rows as i64).collect::>() + ); + assert!(stream.try_next().await.is_err()); + assert!(stats.dropped.load(Ordering::SeqCst)); + assert!(stream.try_next().await.unwrap().is_none()); + } +} + +#[tokio::test] +async fn metadata_blocks_support_binary_values_and_validate_lengths() { + let schema = schema(); + let mut metadata = std::collections::HashMap::new(); + metadata.insert( + "avro.schema".into(), + Value::Bytes(serde_json::to_vec(&schema).unwrap()), + ); + metadata.insert( + "user.binary".into(), + Value::Bytes(vec![0xff; INITIAL_READ_AHEAD as usize + 17]), + ); + let encoded = + apache_avro::to_avro_datum(&Schema::map(Schema::Bytes), Value::Map(metadata)).unwrap(); + let entries = &encoded[1..encoded.len() - 1]; + for (negative_count, length_adjustment) in [(false, 0), (true, 0), (true, -1), (true, 1)] { + let mut bytes = b"Obj\x01".to_vec(); + let count = if negative_count { -2 } else { 2 }; + bytes.extend(apache_avro::to_avro_datum(&Schema::Long, Value::Long(count)).unwrap()); + if negative_count { + bytes.extend( + apache_avro::to_avro_datum( + &Schema::Long, + Value::Long(entries.len() as i64 + length_adjustment), + ) + .unwrap(), + ); + } + bytes.extend_from_slice(entries); + bytes.push(0); + bytes.extend([7; 16]); + let mut writer = Writer::append_to(&schema, bytes, [7; 16]); + writer.append(record(42, 16)).unwrap(); + let (stream, stats) = stream(writer.into_inner().unwrap(), 2, None).await; + if length_adjustment == 0 { + assert_eq!(ids(&stream.try_collect::>().await.unwrap()), [42]); + } else { + let error = terminal_error(stream, &stats).await; + assert!(error.to_string().contains("metadata block length")); + } + } +} diff --git a/crates/paimon/src/arrow/format/avro_write.rs b/crates/paimon/src/arrow/format/avro_write.rs index d52ef6006..9cf680410 100644 --- a/crates/paimon/src/arrow/format/avro_write.rs +++ b/crates/paimon/src/arrow/format/avro_write.rs @@ -577,6 +577,8 @@ mod tests { assert!(writer.num_bytes() > first_size); assert_eq!(writer.in_progress_size(), 0); let size = Box::new(writer).close().await.unwrap().file_size; + let expected = + arrow_select::concat::concat_batches(&first.schema(), [&first, &second]).unwrap(); let bytes = file_io.new_input(path).unwrap().read().await.unwrap(); assert_eq!(size, bytes.len() as u64); @@ -611,13 +613,10 @@ mod tests { .try_collect::>() .await .unwrap(); - assert_eq!(decoded.iter().map(RecordBatch::num_rows).sum::(), 3); - let ids = decoded[0] - .column(0) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(ids.iter().collect::>(), vec![Some(1), None, Some(3)]); + assert_eq!( + arrow_select::concat::concat_batches(&expected.schema(), &decoded).unwrap(), + expected + ); } #[test] diff --git a/crates/paimon/src/arrow/format/mod.rs b/crates/paimon/src/arrow/format/mod.rs index b51f403a8..1edc54631 100644 --- a/crates/paimon/src/arrow/format/mod.rs +++ b/crates/paimon/src/arrow/format/mod.rs @@ -49,6 +49,16 @@ use async_trait::async_trait; use std::collections::{HashMap, VecDeque}; use std::sync::Arc; +fn panic_payload_message(panic: Box) -> String { + if let Some(message) = panic.downcast_ref::<&str>() { + (*message).to_string() + } else if let Some(message) = panic.downcast_ref::() { + message.clone() + } else { + "unknown panic payload".to_string() + } +} + /// Predicates with the file-level field context needed for pushdown. /// Only used by formats that support predicate pushdown (e.g. Parquet). pub(crate) struct FilePredicates { diff --git a/crates/paimon/src/arrow/format/mosaic.rs b/crates/paimon/src/arrow/format/mosaic.rs index 0a9d7e7ee..f2b68a082 100644 --- a/crates/paimon/src/arrow/format/mosaic.rs +++ b/crates/paimon/src/arrow/format/mosaic.rs @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -use super::{timestamp_millis_schema, FilePredicates, FormatFileReader}; +use super::{panic_payload_message, timestamp_millis_schema, FilePredicates, FormatFileReader}; use crate::arrow::build_target_arrow_schema; use crate::arrow::filtering::{ predicates_may_match_with_schema, remap_predicates_to_file, StatsAccessor, @@ -514,16 +514,6 @@ fn read_planned_row_group( take_row_slices(batch, plan.selected_slices.as_deref(), read_schema) } -fn panic_payload_message(panic: Box) -> String { - if let Some(message) = panic.downcast_ref::<&str>() { - (*message).to_string() - } else if let Some(message) = panic.downcast_ref::() { - message.clone() - } else { - "unknown panic payload".to_string() - } -} - fn apply_mosaic_residual( batch: RecordBatch, predicates: Option<&FilePredicates>, diff --git a/crates/paimon/src/table/data_file_reader.rs b/crates/paimon/src/table/data_file_reader.rs index fb10d20ae..4fd25de50 100644 --- a/crates/paimon/src/table/data_file_reader.rs +++ b/crates/paimon/src/table/data_file_reader.rs @@ -969,13 +969,9 @@ impl DataFileReader { /// engine-supplied bucket split does — hands them over directly rather than /// expanding them into positions this would only coalesce back. /// - /// The emitted rows are always exactly the selected ones, but what that saves is - /// the format's business, and it differs: mosaic skips a row group before - /// touching its column data, parquet skips pages through the offset index, - /// `.row` prunes blocks. Avro is the exception — its reader loads the whole file - /// and deserializes every record before applying the selection, so there a - /// narrow selection saves only what comes after decoding: Arrow column - /// materialization, and whatever the caller does per row. + /// Mosaic prunes row groups, Parquet uses its page offset index, and the row + /// format prunes blocks. Avro skips blocks outside the selected ranges and + /// decodes records sequentially within each selected block. pub(super) fn read_single_file_stream_local_ranges( &self, split: &DataSplit,