Skip to content

feat: add a portable centroid router for independently built vector shards - #131

Merged
JingsongLi merged 10 commits into
apache:mainfrom
JingsongLi:codex/centroid-sharding
Nov 18, 2026
Merged

JingsongLi merged 10 commits into
apache:mainfrom
JingsongLi:codex/centroid-sharding

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

A Spark build can shuffle vectors by a global coarse center and build an independent vector index in each reducer. At query time the same centers select the files to open. This PR adds that outer routing operation to Rust and Java/JNI. The downstream Spark integration and its SIFT1M benchmarks are in apache/paimon#11249. The original sharding proposal is PIP-47.

API and storage

  1. Create an ordinary IVF-FLAT trainer with the outer routing options and call planTrainingSample(vectorCount). For every shard type, Spark workers select those input rows from their own splits, and the training task receives them in reservoir slot order. DiskANN's seed and memory budget apply inside individual shards to PQ training and graph construction.
  2. Feed those rows in slot order into an ordinary IVF-FLAT trainer using addTrainingVectors. Call finishTraining() and transfer the coarse centers with CentroidRouter.fromTraining(coarse). Broadcast the serialized router. Workers restore it and shuffle each vector and its absolute row ID by assignCentroid(vector). The versioned payload contains the dimension, metric, center count and coarse center coordinates.
  3. For each populated shard, train an ordinary IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ or DiskANN index using the local vectors. Use the normal writer and reader. IVF trains its own inner lists and quantizer. DiskANN trains its own PQ and builds its own graph. The caller maps outer center IDs to files, handles empty slots and publishes the index generation.
  4. Route a query with findNearestCentroids(query, width). The outer width and the local IVF probe count or DiskANN graph search width can be set independently. Merge the local top-k results using their scores.

The router takes ownership of ordinary IVF-FLAT training. The Rust entry points are VectorIndexTrainer::plan_training_sample and CentroidRouter::from_training. The same router serves every local index type. Existing shard formats and the public ordinary reader and writer APIs are preserved. A 0.7 prerelease snapshot that wrote the earlier shared-model preview requires rebuilding its centroid generation with the new JAR. Current API examples and the router wire format are in docs/api.html and core/STORAGE_FORMAT.md.

Verification

  • On SIFT1M with 64 outer centers and a training ratio of 1, the downstream distributed sampling path for IVF-FLAT, IVF-PQ, IVF-SQ, IVF-RQ and DiskANN selected identical ordinals and vector bytes for all 65,536 rows. The serialized routers matched an independent streaming IVF-FLAT trainer fed all one million rows, and all families matched on 10,000 assignments and 100,000 ordered probes. DiskANN also matched with seed 93 and a 4 MiB local memory budget. The historical per-DiskANN sampling path retained 50,000 rows with default settings and 2,563 with that budget. Existing stored routers are read as before.
  • Verified Rust workspace tests, Clippy with warnings denied, Rust formatting and Java 8 packaged-JAR smoke, native validation and handle safety checks. The Java integration test routes with one model, builds all five shard types using standard writers, and retrieves their absolute row IDs with standard readers.
  • Verified the downstream Spark SQL integration with all five index types, independent two-level search widths, distributed sampling, filtering, empty shards and range rebuilds.

Downstream Spark performance

DiskANN was rebuilt on SIFT1M (1M rows, 128D L2, Apple M4 Pro, Spark local[4], four native threads). The two layouts used the same graph degree (32), build search-list size (64), 32 PQ subquantizers and F16 raw vectors. Range searched all 64 shards with graph width 16. Centroid searched ten of 64 shards with graph width 128. The same 2,000 held-out query IDs from the earlier calibration were used for each run. Each query opened a fresh reader with the OS page cache warm. Reads are logical MiB per query.

Code and layout Recall@10 Build s P95 ms Files/query MiB/query
12612fe Range (3 rounds) 99.460% 102.75 51.70 64 82.65
12612fe Centroid (3 rounds) 99.435% 114.74 12.53 11 17.00
05cbd9e Range (final rerun) 99.460% 101.43 48.76 64 82.53
05cbd9e Centroid (final rerun) 99.435% 106.25 10.60 11 16.79
c0dbe10 Range (ordinary Trainer) 99.460% 95.07 53.17 64 82.71
c0dbe10 Centroid (DiskANN outer sample) 99.435% 102.49 9.87 11 16.77
51250ac Range 99.460% 98.13 46.75 64 82.49
51250ac Centroid (IVF-FLAT outer sample) 99.355% 106.25 13.26 11 16.98

The first three rounds summarize the median full build, P95 and reads per layout. The 05cbd9e runs used the earlier Builder API with the same training, routing and graph configuration. On the ordinary Trainer API in c0dbe10, both layouts were rebuilt from scratch and scored on the same 2,000 held-out queries. The Centroid-minus-Range recall difference was -0.025 percentage points (paired 95% bootstrap interval [-0.185, +0.130]). All three earlier comparisons passed the preselected matching rule. Build timing includes Spark sampling, training, shuffle, writing and commit.

The last pair used the downstream unified IVF-FLAT sampler and the same native runtime, build settings and held-out query IDs. Both layouts were rebuilt once. Centroid minus Range was -0.105 recall percentage points with a paired 95% bootstrap interval of [-0.260, +0.055], meeting the existing matching rule. Compared with the previous DiskANN-specific outer sample, the change in Centroid recall was -0.080 points [-0.235, +0.075]. Timings for each of those two Centroid models come from separate runs.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Benchmark run · success · attempt 1

Vector index benchmark

🟢 No alerts (5/5 indexes)

Index Recall@10 (Δ) Single QPS Δ Batch QPS Δ Status
IVF_FLAT 100.00% (+0.00 pp) +0.7% -0.3% 🟢 OK
IVF_SQ 99.26% (+0.00 pp) -0.9% +0.2% 🟢 OK
IVF_PQ 46.60% (+0.00 pp) -1.5% -3.3% 🟢 OK
IVF_RQ 90.91% (+0.00 pp) -0.8% -1.7% 🟢 OK
DISKANN 78.32% (+0.00 pp) -0.2% -0.3% 🟢 OK

🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup.

Absolute metrics & environment

Medians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.

Index Single QPS Batch QPS P95 (µs) Build (ms) Process RSS (MiB) Size (MiB)
IVF_FLAT 78,694.12 → 79,241.52 90,566.97 → 90,261.33 15.73 → 15.71 25.00 → 24.00 19.87 → 19.87 2.47 → 2.47
IVF_SQ 62,462.46 → 61,909.77 117,086.27 → 117,263.72 18.08 → 18.26 18.00 → 18.00 19.87 → 19.87 0.67 → 0.67
IVF_PQ 16,758.81 → 16,501.36 141,332.49 → 136,691.80 74.49 → 74.85 189.00 → 189.50 19.87 → 19.87 0.17 → 0.17
IVF_RQ 11,408.90 → 11,312.19 49,154.93 → 48,329.65 109.27 → 108.89 32.00 → 32.00 19.99 → 19.99 0.52 → 0.52
DISKANN 10,577.74 → 10,558.88 20,056.82 → 19,999.12 113.57 → 113.58 1,082.00 → 1,079.00 20.12 → 20.12 3.22 → 3.22

Base 252f77c43bd655ce4e742a6e374e06cda7dcaf6d · PR merge 3d6e11a5da226a2e9f3a12c3e50a7a3c7d7a99a0
rustc 1.100.0 (0ef1bc386 2026-11-09) · Linux-7.0.0-1017-azure-x86_64-with-glibc2.39 · AMD EPYC 7763 64-Core Processor
6 samples/version/index, alternating execution order; 2 threads; 10k × 64D vectors; 2,048 queries; ≥1s per timed mode.
RSS is the process lifetime peak up to build completion, including dataset/ground truth.
Raw CSVs, sample ranges, workload settings and logs are in the workflow artifact.

@JingsongLi JingsongLi changed the title feat: support portable centroid shards for all vector index types feat: add a portable centroid router for independently built vector shards Nov 18, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e794291 into apache:main Nov 18, 2026
10 checks passed
JingsongLi added a commit that referenced this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants