Repository navigation
[spark] Support centroid-based sharding for vector global indexes - #11249
Draft
JingsongLi wants to merge 5 commits into
Draft
JingsongLi wants to merge 5 commits into
JingsongLi wants to merge 5 commits into
Conversation
JingsongLi
marked this pull request as draft
November 18, 2026 11:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add opt-in centroid-based sharding for Spark vector global indexes on Data Evolution append tables. Range-based sharding is the default. With centroid sharding, a query can open just the files assigned to the nearest global centers. PIP-47
Spark pins a snapshot, counts non-null vectors per split and plans a deterministic sample with an ordinary IVF-FLAT trainer. Each map task selects rows from its own splits. A single task trains the coarse centers from those rows in reservoir order. Spark broadcasts the router, shuffles by centroid ID and builds an independent IVF or DiskANN index per populated shard. Readers use the persisted router to select files and merge the local results by score.
Vector index plugins opt into routing by implementing
CentroidVectorGlobalIndexer. The routing model and shard entries use the existingGlobalIndexMeta.indexMetabytes. A model entry records its generation ID, shard count and exact source row ranges. A shard entry wraps the local writer’s opaque metadata with its generation ID and centroid ID. The local reader receives the original bytes. The trained centers are stored once in the model file, andsourceMetacontinues to identify the Data Evolution source snapshot.Native dependency. This branch uses
paimon-vector-index-java:0.7.0-SNAPSHOTwith the router and sample-planning APIs from apache/paimon-vector-index#131. Local tests used a JAR built from the companion PR. The Apache snapshot0.7.0-20261118.003701-2lacksCentroidRouterandVectorIndexTrainer.planTrainingSample. This PR will switch to a published version containing both APIs after the companion PR merges. Readers need centroid support before this layout is enabled.Tests
Benchmark results
Each cell shows Range → Centroid at matched Recall@10. These measurements predate the Java metadata refactor. SIFT1M contains one million 128-dimensional L2 vectors. Runs used an Apple M4 Pro, Spark
local[4]and four native threads. Build time includes sampling, training, shuffle, index writes and commit. Each timed query opened a fresh reader with a warm OS page cache. File counts include the router, and MiB measures logical reads per query.The four IVF results come from earlier runs with independently trained shard models. Build time, P95 and reads are medians over three rounds. IVF-PQ used 32-byte codes in both layouts and 3,500 held-out queries. Each other family used 2,000 held-out queries. All five comparisons met the prespecified matching rule, which requires an absolute mean recall difference of at most one percentage point and a paired 95% bootstrap lower bound above −1 point.
DiskANN was rebuilt once per layout using the unified IVF-FLAT outer sampler. Both builds used 64 shards, graph degree 32, build search width 64, 32 PQ subquantizers and F16 raw vectors. Range searched 64 shards with graph search width 16. Centroid searched 10 shards with width 128. The recall difference was −0.105 percentage points (paired 95% interval [−0.260, +0.055]). Centroid took 8.12 s longer to build and reduced query P95 from 46.75 ms to 13.26 ms.