Skip to content

[python] Stream raw vector batches for local single-query search - #11254

Open
wangzhigang1999 wants to merge 3 commits into
apache:masterfrom
wangzhigang1999:zhigang/python-vector-stream-search
Open

wangzhigang1999 wants to merge 3 commits into
apache:masterfrom
wangzhigang1999:zhigang/python-vector-stream-search

Conversation

@wangzhigang1999

Copy link
Copy Markdown
Contributor

Closes #11253

Purpose

PyPaimon's local executor currently collects all raw vectors for a single query before it scores the first batch. This change passes Arrow batches directly to the existing scorer, which updates one Top-K heap throughout the scan.

With parallel reads, each split group has at most one read in flight and one batch of lookahead. The calling thread scores whichever batch is ready. If scoring fails, it waits for outstanding reads before closing the readers.

Tests

The new tests cover parallel Top-K results, bounded read-ahead, candidate filtering, and reader cleanup after errors. The regression run passed 202 tests and skipped one S3 checksum test that requires PyArrow 22+. Both versions matched the published L2 Top-10 for GIST1M queries 0, 499 and 999. Dataset

Both versions read the same unindexed GIST1M snapshot (1M rows × 960 dimensions, 12 splits). The table shows medians for query 0 over three runs after one warmup per version, comparing master d90a902d with the patch.

Executor Storage Peak scoring-process RSS (MiB), before → after Query time (s), before → after
Local (2 CPUs) Local Parquet 4,865 → 795 (-83.7%) 12.90 → 8.57 (-33.6%)
Local (2 CPUs) OSS 5,158 → 683 (-86.8%) 13.25 → 9.47 (-28.5%)
Ray (2 workers) Worker-local Parquet 268 → 268 (0.0%) 11.69 → 11.54 (-1.3%)
Ray (2 workers) OSS 282 → 279 (-1.1%) 18.35 → 18.75 (+2.2%)

Ray workers already score one split at a time from a batch reader, and their peak RSS stayed close across versions. The Ray rows report the largest scoring worker's peak and include the time to fetch the ten hits.

@wangzhigang1999
wangzhigang1999 force-pushed the zhigang/python-vector-stream-search branch from 7a64f9c to d290df0 Compare November 18, 2026 15:50
for block in batches:
# Bound the Arrow take as well as the scoring matrix for wide vectors.
blocks = batches if score_candidates is None else _slice_batches(batches, _score_block_size(query_vector))
for block in blocks:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _parallel_raw_batches (line 902), del batch, future is used after yield batch so the just-scored RecordBatch can be freed before waiting on the next read. However, in Python a for block in blocks: loop keeps block, row_ids, and vectors bound in _score_raw_batches's frame while blocks.__next__() is executing to fetch the next batch (and similarly batch remains bound in _slice_batches on line 1137 while batches.__next__() runs).

Because block / vectors (and batch in _slice_batches) share the underlying Arrow buffers, the previous RecordBatch stays pinned in memory throughout the next batch read until the loop variable is reassigned.

Consider explicitly deleting the loop references at the end of each iteration (matching del batch, row_ids, vectors in _score_raw_batch_queries on line 868), and doing del batch at the end of the outer loop in _slice_batches:

        for block in blocks:
            row_ids = block.column(SpecialFields.ROW_ID.name).to_pylist()
            if score_candidates is not None:
                positions = [i for i, row_id in enumerate(row_ids)
                             if row_id in score_candidates]
                if not positions:
                    del block, row_ids
                    continue
                vectors = block.column(self._vector_column.name).take(positions)
                row_ids = [row_ids[i] for i in positions]
            else:
                vectors = block.column(self._vector_column.name)
            for row_id, score in zip(
                row_ids, _iter_arrow_scores(vectors, query_vector, metric)
            ):
                if score is not None:
                    if reject_nan and math.isnan(score):
                        raise ValueError("Ray vector search cannot rank NaN scores.")
                    _offer_score(top_k_heap, self._limit, row_id, score)
            del block, row_ids, vectors

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I could reproduce the previous Arrow buffer still being held when the reader was asked for the next batch. Fixed in eafb7eb. The regression test checks the buffer lifetime with filtering off, with a matching candidate, and with no matches.

) -> "_BatchStream":
"""Open on the caller thread and register cleanup before submitting the first read."""
reader_scope = streams.enter_context(ExitStack())
reader = reader_scope.enter_context(_open_raw_batch_reader(table_read, splits))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a split group yields multiple batches (>= 2 batches across its splits/row groups), combining _read_one_batch_ahead with submitting next(batches, None) after yield batch has two side effects:

  1. First-batch latency & eager second-batch read: On the initial executor.submit(next, batches, None), _read_one_batch_ahead reads current = next(reader, None) (batch 0) and then immediately enters for following in reader: to read batch 1 before yielding batch 0. The caller cannot start scoring batch 0 until two full batches have been read by that worker, and if scoring fails on the first batch, every multi-batch worker has already read a second batch.
  2. No I/O–compute overlap within a split group: Because pending[executor.submit(next, batches, None)] = batches happens after yield batch, the worker thread for that split group sits idle while the caller scores batch (holding following in its suspended generator frame). Then, once scoring finishes, the next next(batches, None) call cannot return the already-buffered following batch until it finishes reading the subsequent batch from storage.

Would it be simpler to have batches yield one batch per next() call (closing reader_scope in finally on exhaustion) and submit executor.submit(next, batches, None) before yield batch? That way batch 0 can be scored after a single batch read, reading batch k + 1 (or hitting EOF and closing the reader after the last batch) overlaps on the worker thread while the caller scores batch k, and each split group still has at most 1 batch being scored + 1 read in flight.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this ordering. You’re right about the overlap: on split groups with multiple batches, submitting the next read before scoring brought the first batch to the scorer 6–24% sooner across the local Parquet and OSS cases. The caller gets its Top-K after the scan finishes, though, and median end-to-end times changed by less than 5% in all six cases (five measured runs each).

The deciding case for me was 32 single-batch splits on OSS. Both versions passed 20/20 runs under a 6 GiB virtual address space limit, but median peak RSS rose from 299 to 347 MiB with early submission. The current lookahead reaches EOF and closes each reader before handing its last batch to the scorer. With early submission, that close can still be waiting on a worker. Memory results varied by layout (early submission did save 98 MiB on the large local table), so I’m keeping the current ordering for this PR to preserve early close on small splits. Thanks for suggesting the comparison.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Reduce peak memory for local vector search in PyPaimon

2 participants