Repository navigation
[python] Stream raw vector batches for local single-query search - #11254
wangzhigang1999 wants to merge 3 commits into
Conversation
7a64f9c to
d290df0
Compare
| for block in batches: | ||
| # Bound the Arrow take as well as the scoring matrix for wide vectors. | ||
| blocks = batches if score_candidates is None else _slice_batches(batches, _score_block_size(query_vector)) | ||
| for block in blocks: |
There was a problem hiding this comment.
In _parallel_raw_batches (line 902), del batch, future is used after yield batch so the just-scored RecordBatch can be freed before waiting on the next read. However, in Python a for block in blocks: loop keeps block, row_ids, and vectors bound in _score_raw_batches's frame while blocks.__next__() is executing to fetch the next batch (and similarly batch remains bound in _slice_batches on line 1137 while batches.__next__() runs).
Because block / vectors (and batch in _slice_batches) share the underlying Arrow buffers, the previous RecordBatch stays pinned in memory throughout the next batch read until the loop variable is reassigned.
Consider explicitly deleting the loop references at the end of each iteration (matching del batch, row_ids, vectors in _score_raw_batch_queries on line 868), and doing del batch at the end of the outer loop in _slice_batches:
for block in blocks:
row_ids = block.column(SpecialFields.ROW_ID.name).to_pylist()
if score_candidates is not None:
positions = [i for i, row_id in enumerate(row_ids)
if row_id in score_candidates]
if not positions:
del block, row_ids
continue
vectors = block.column(self._vector_column.name).take(positions)
row_ids = [row_ids[i] for i in positions]
else:
vectors = block.column(self._vector_column.name)
for row_id, score in zip(
row_ids, _iter_arrow_scores(vectors, query_vector, metric)
):
if score is not None:
if reject_nan and math.isnan(score):
raise ValueError("Ray vector search cannot rank NaN scores.")
_offer_score(top_k_heap, self._limit, row_id, score)
del block, row_ids, vectorsThere was a problem hiding this comment.
Thanks for catching this. I could reproduce the previous Arrow buffer still being held when the reader was asked for the next batch. Fixed in eafb7eb. The regression test checks the buffer lifetime with filtering off, with a matching candidate, and with no matches.
| ) -> "_BatchStream": | ||
| """Open on the caller thread and register cleanup before submitting the first read.""" | ||
| reader_scope = streams.enter_context(ExitStack()) | ||
| reader = reader_scope.enter_context(_open_raw_batch_reader(table_read, splits)) |
There was a problem hiding this comment.
When a split group yields multiple batches (>= 2 batches across its splits/row groups), combining _read_one_batch_ahead with submitting next(batches, None) after yield batch has two side effects:
- First-batch latency & eager second-batch read: On the initial
executor.submit(next, batches, None),_read_one_batch_aheadreadscurrent = next(reader, None)(batch 0) and then immediately entersfor following in reader:to read batch 1 before yielding batch 0. The caller cannot start scoring batch 0 until two full batches have been read by that worker, and if scoring fails on the first batch, every multi-batch worker has already read a second batch. - No I/O–compute overlap within a split group: Because
pending[executor.submit(next, batches, None)] = batcheshappens afteryield batch, the worker thread for that split group sits idle while the caller scoresbatch(holdingfollowingin its suspended generator frame). Then, once scoring finishes, the nextnext(batches, None)call cannot return the already-bufferedfollowingbatch until it finishes reading the subsequent batch from storage.
Would it be simpler to have batches yield one batch per next() call (closing reader_scope in finally on exhaustion) and submit executor.submit(next, batches, None) before yield batch? That way batch 0 can be scored after a single batch read, reading batch k + 1 (or hitting EOF and closing the reader after the last batch) overlaps on the worker thread while the caller scores batch k, and each split group still has at most 1 batch being scored + 1 read in flight.
There was a problem hiding this comment.
I tried this ordering. You’re right about the overlap: on split groups with multiple batches, submitting the next read before scoring brought the first batch to the scorer 6–24% sooner across the local Parquet and OSS cases. The caller gets its Top-K after the scan finishes, though, and median end-to-end times changed by less than 5% in all six cases (five measured runs each).
The deciding case for me was 32 single-batch splits on OSS. Both versions passed 20/20 runs under a 6 GiB virtual address space limit, but median peak RSS rose from 299 to 347 MiB with early submission. The current lookahead reaches EOF and closes each reader before handing its last batch to the scorer. With early submission, that close can still be waiting on a worker. Memory results varied by layout (early submission did save 98 MiB on the large local table), so I’m keeping the current ordering for this PR to preserve early close on small splits. Thanks for suggesting the comparison.
Closes #11253
Purpose
PyPaimon's local executor currently collects all raw vectors for a single query before it scores the first batch. This change passes Arrow batches directly to the existing scorer, which updates one Top-K heap throughout the scan.
With parallel reads, each split group has at most one read in flight and one batch of lookahead. The calling thread scores whichever batch is ready. If scoring fails, it waits for outstanding reads before closing the readers.
Tests
The new tests cover parallel Top-K results, bounded read-ahead, candidate filtering, and reader cleanup after errors. The regression run passed 202 tests and skipped one S3 checksum test that requires PyArrow 22+. Both versions matched the published L2 Top-10 for GIST1M queries 0, 499 and 999. Dataset
Both versions read the same unindexed GIST1M snapshot (1M rows × 960 dimensions, 12 splits). The table shows medians for query 0 over three runs after one warmup per version, comparing master
d90a902dwith the patch.Ray workers already score one split at a time from a batch reader, and their peak RSS stayed close across versions. The Ray rows report the largest scoring worker's peak and include the time to fetch the ten hits.