Repository navigation
feat: add a portable centroid router for independently built vector shards - #131
Merged
Merged
Conversation
|
Benchmark run · success · attempt 1 Vector index benchmark🟢 No alerts (5/5 indexes)
🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup. Absolute metrics & environmentMedians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.
Base |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A Spark build can shuffle vectors by a global coarse center and build an independent vector index in each reducer. At query time the same centers select the files to open. This PR adds that outer routing operation to Rust and Java/JNI. The downstream Spark integration and its SIFT1M benchmarks are in apache/paimon#11249. The original sharding proposal is PIP-47.
API and storage
planTrainingSample(vectorCount). For every shard type, Spark workers select those input rows from their own splits, and the training task receives them in reservoir slot order. DiskANN's seed and memory budget apply inside individual shards to PQ training and graph construction.addTrainingVectors. CallfinishTraining()and transfer the coarse centers withCentroidRouter.fromTraining(coarse). Broadcast the serialized router. Workers restore it and shuffle each vector and its absolute row ID byassignCentroid(vector). The versioned payload contains the dimension, metric, center count and coarse center coordinates.findNearestCentroids(query, width). The outer width and the local IVF probe count or DiskANN graph search width can be set independently. Merge the local top-k results using their scores.The router takes ownership of ordinary IVF-FLAT training. The Rust entry points are
VectorIndexTrainer::plan_training_sampleandCentroidRouter::from_training. The same router serves every local index type. Existing shard formats and the public ordinary reader and writer APIs are preserved. A 0.7 prerelease snapshot that wrote the earlier shared-model preview requires rebuilding its centroid generation with the new JAR. Current API examples and the router wire format are indocs/api.htmlandcore/STORAGE_FORMAT.md.Verification
Downstream Spark performance
DiskANN was rebuilt on SIFT1M (1M rows, 128D L2, Apple M4 Pro, Spark local[4], four native threads). The two layouts used the same graph degree (32), build search-list size (64), 32 PQ subquantizers and F16 raw vectors. Range searched all 64 shards with graph width 16. Centroid searched ten of 64 shards with graph width 128. The same 2,000 held-out query IDs from the earlier calibration were used for each run. Each query opened a fresh reader with the OS page cache warm. Reads are logical MiB per query.
The first three rounds summarize the median full build, P95 and reads per layout. The 05cbd9e runs used the earlier Builder API with the same training, routing and graph configuration. On the ordinary Trainer API in c0dbe10, both layouts were rebuilt from scratch and scored on the same 2,000 held-out queries. The Centroid-minus-Range recall difference was -0.025 percentage points (paired 95% bootstrap interval [-0.185, +0.130]). All three earlier comparisons passed the preselected matching rule. Build timing includes Spark sampling, training, shuffle, writing and commit.
The last pair used the downstream unified IVF-FLAT sampler and the same native runtime, build settings and held-out query IDs. Both layouts were rebuilt once. Centroid minus Range was -0.105 recall percentage points with a paired 95% bootstrap interval of [-0.260, +0.055], meeting the existing matching rule. Compared with the previous DiskANN-specific outer sample, the change in Centroid recall was -0.080 points [-0.235, +0.075]. Timings for each of those two Centroid models come from separate runs.