Skip to content

[spark] Support centroid-based sharding for vector global indexes - #11249

Draft
JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/centroid-sharding
Draft

JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/centroid-sharding

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add opt-in centroid-based sharding for Spark vector global indexes on Data Evolution append tables. Range-based sharding is the default. With centroid sharding, a query can open just the files assigned to the nearest global centers. PIP-47

Spark pins a snapshot, counts non-null vectors per split and plans a deterministic sample with an ordinary IVF-FLAT trainer. Each map task selects rows from its own splits. A single task trains the coarse centers from those rows in reservoir order. Spark broadcasts the router, shuffles by centroid ID and builds an independent IVF or DiskANN index per populated shard. Readers use the persisted router to select files and merge the local results by score.

Vector index plugins opt into routing by implementing CentroidVectorGlobalIndexer. The routing model and shard entries use the existing GlobalIndexMeta.indexMeta bytes. A model entry records its generation ID, shard count and exact source row ranges. A shard entry wraps the local writer’s opaque metadata with its generation ID and centroid ID. The local reader receives the original bytes. The trained centers are stored once in the model file, and sourceMeta continues to identify the Data Evolution source snapshot.

Native dependency. This branch uses paimon-vector-index-java:0.7.0-SNAPSHOT with the router and sample-planning APIs from apache/paimon-vector-index#131. Local tests used a JAR built from the companion PR. The Apache snapshot 0.7.0-20261118.003701-2 lacks CentroidRouter and VectorIndexTrainer.planTrainingSample. This PR will switch to a published version containing both APIs after the companion PR merges. Readers need centroid support before this layout is enabled.

Tests

  • 314 local tests passed across Paimon core, the native-backed Java indexer and Spark 3. Coverage includes metadata and commit compatibility, split-local sampling, all five index types, shard pruning, range rebuilds and post-filtering with absolute row IDs in a partition whose first row ID is nonzero.
  • Clean builds using the companion native JAR passed 22 focused JDK 8 tests. All seven Spark centroid integration cases passed on each CI combination tested locally (Spark 3 on JDK 8 with Scala 2.12 and 2.13, and Spark 4 on JDK 17 with Scala 2.13).
  • On SIFT1M with 64 outer centers and a sampling ratio of 1, all five index types selected the same 65,536 rows. The serialized router matched streaming IVF-FLAT training on all one million rows, including when DiskANN used seed 93 and a 4 MiB local build budget.

Benchmark results

Each cell shows Range → Centroid at matched Recall@10. These measurements predate the Java metadata refactor. SIFT1M contains one million 128-dimensional L2 vectors. Runs used an Apple M4 Pro, Spark local[4] and four native threads. Build time includes sampling, training, shuffle, index writes and commit. Each timed query opened a fresh reader with a warm OS page cache. File counts include the router, and MiB measures logical reads per query.

Index Recall@10 (%) Build (s) Query P95 (ms) Files/query MiB/query
IVF-FLAT 99.000 → 99.220 3.32 → 6.87 9.22 → 5.41 64 → 25 67.18 → 37.16
IVF-SQ 98.015 → 97.950 3.81 → 4.64 7.58 → 3.38 64 → 11 22.54 → 11.03
IVF-PQ 72.886 → 73.094 66.80 → 67.41 33.08 → 1.69 64 → 5 14.98 → 1.11
IVF-RQ 90.075 → 90.550 3.44 → 5.89 6.20 → 2.09 64 → 13 13.04 → 3.46
DiskANN 99.460 → 99.355 98.13 → 106.25 46.75 → 13.26 64 → 11 82.49 → 16.98

The four IVF results come from earlier runs with independently trained shard models. Build time, P95 and reads are medians over three rounds. IVF-PQ used 32-byte codes in both layouts and 3,500 held-out queries. Each other family used 2,000 held-out queries. All five comparisons met the prespecified matching rule, which requires an absolute mean recall difference of at most one percentage point and a paired 95% bootstrap lower bound above −1 point.

DiskANN was rebuilt once per layout using the unified IVF-FLAT outer sampler. Both builds used 64 shards, graph degree 32, build search width 64, 32 PQ subquantizers and F16 raw vectors. Range searched 64 shards with graph search width 16. Centroid searched 10 shards with width 128. The recall difference was −0.105 percentage points (paired 95% interval [−0.260, +0.055]). Centroid took 8.12 s longer to build and reduced query P95 from 46.75 ms to 13.26 ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant