Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions docs/docs/multimodal-table/global-index/vector.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,7 @@ Search-time options are passed with each vector search request:
| Option | Default | Description |
|---|---|---|
| `ivf.nprobe` | Automatic | Explicit number of IVF clusters to probe. When omitted, paimon-vindex derives the width from the index, `top_k`, and filter selectivity. |
| `centroid.nprobe` | Automatic | Number of centroid shards to search. Within each selected shard, `ivf.nprobe` controls the IVF lists to search and `diskann.l_search` controls the DiskANN graph search width. |
| `ivf.max_initial_filter_expansion_factor` | Disabled | Positive integer limiting filter-driven expansion of the initial automatic IVF probe width. A factor of `1` disables initial filter expansion. Progressive retries may still probe more clusters when fewer than `top_k` filtered results are found. |
| `ivf.refine_factor` | Disabled | Retrieves `top_k * refine_factor` IVF candidates and reranks them with the original vectors stored in the Paimon table. It is most useful for compressed indexes such as `ivf-pq`, `ivf-sq`, and `ivf-rq` when recall is more important than latency. |
| `ivf_pq.batch_table_reuse` | `auto` | IVF-PQ batch search distance-table reuse mode: `auto`, `on`, or `off`. Other index types and scalar searches ignore it. |
Expand Down Expand Up @@ -463,6 +464,118 @@ CALL sys.create_global_index(
Pass the same option keys to Python's `create_global_index(..., options={...})`.
Use full 768-dimensional query vectors when searching either of these indexes.

## Centroid-Based Sharding in Spark

Spark can build `ivf-flat-centroid`, `ivf-pq-centroid`, `ivf-sq-centroid`, `ivf-rq-centroid`
and `diskann-centroid` on Data Evolution append tables. Each build writes one routing model
for the partition and one file per centroid. The table must enable row tracking and
Data Evolution, have no primary key, and use `bucket=-1`.

```sql
ALTER TABLE db.model_embeddings SET TBLPROPERTIES (
'ivf-pq-centroid.dimension' = '768',
'ivf-pq-centroid.distance.metric' = 'cosine'
);

CALL sys.create_global_index(
table => 'db.model_embeddings',
index_column => 'embedding',
index_type => 'ivf-pq-centroid',
partitions => 'dt="2026-11-17"',
options => 'ivf-pq-centroid.nlist=256'
);
```

Use the registered index type in both `index_type` and the option prefix.
The ordinary names (`ivf-pq`, for example) select range-based sharding.
Drop the previous index on the vector column before switching types, including
when different partitions are built in separate calls. A centroid
build pins one snapshot. Spark map tasks count non-null vectors in their input
splits. Using those counts and the configured training ratio, the driver plans
the exact row ordinals that the outer IVF-FLAT trainer would retain. Each
map task reads its own splits and shuffles selected vectors by reservoir slot.
A single task trains the routing model on those vectors in the original training
order. The driver broadcasts the model for the subsequent centroid shuffle.
Sampling requires at least one
non-null vector. Each call builds a complete partition. An unpartitioned table
counts as one partition. Repeating the call atomically replaces the
partition's existing model and shards of the requested type.

Queries load the routing model, select centroids with `centroid.nprobe`, and open
only their shard files. Inside each selected shard, `ivf.nprobe` controls how many
local IVF lists to search. For DiskANN, `diskann.l_search` controls the graph search.
Automatic probe sizing and filtered search remain available.
The trained outer centers live in the routing model file. In the manifest, each
centroid index entry stores its routing descriptor in `GlobalIndexMeta.indexMeta`.
For a shard, this descriptor identifies the model and centroid and wraps the local
index writer’s metadata. The local reader receives those original bytes.

Centroid shards store absolute row IDs. The model entry records the exact source row
ranges so gaps belonging to other partitions remain eligible for raw search in
`vector-index.search-mode=detail` or `chunk`. The default `fast` mode searches
only indexed rows.
Persist the metric and dimension on the table when using raw fallback so indexed
and unindexed ranges use the same distance definition.
Batch queries share each selected shard across the queries routed to it.

For all four IVF types, Spark shuffles vectors by their assigned routing centroid.
Each populated shard trains an independent IVF index on the vectors it receives.
For IVF-PQ, this gives each shard its own inner centroids and PQ codebook. The outer
`<index-type>.nlist` sets the number of shard slots. The optional
`<index-type>.centroid.local-nlist` sets the number of IVF lists per shard and
defaults to automatic sizing from that shard's vector count.

For example, build with
`ivf-pq-centroid.nlist=64,ivf-pq-centroid.centroid.local-nlist=32`.
A query with `map('centroid.nprobe','4','ivf.nprobe','8')` selects four shard
files and searches eight local lists in each populated shard. Both levels
support automatic probe sizing. Empty local shards have a manifest row count
of zero and queries skip them.

DiskANN uses the same outer router as IVF. Each populated centroid uses the
ordinary DiskANN trainer, writer and reader. The shard trains its PQ codebook
and builds its graph from the rows it receives. DiskANN’s seed and memory budget
apply to this local build. The outer sample and coarse centers use the same
IVF-FLAT training options as the IVF shard types. An empty shard uses an
eight-byte `PCEMPTY1` marker with manifest row count zero. Queries skip that file. Empty slots remain in the complete
generation so centroid identifiers stay stable.

For example, use `index_type => 'diskann-centroid'` and
`options => 'diskann-centroid.nlist=64'`, then query with
`map('centroid.nprobe','8','diskann.l_search','64')`. Persist
`diskann-centroid.dimension` and `diskann-centroid.metric` on the table for raw fallback.

If source rows change while the build is running, including through compaction,
the commit fails and the build must be retried. Appends outside its source ranges
can proceed. Maintenance that reassigns row IDs invalidates the complete index
generation and requires a rebuild.

This build mode and its reader currently require the Java/Spark implementation
and a matching native library with centroid model APIs. Install the vector plugin on readers
and on jobs that plan, commit or maintain these indexes. The plugin interprets manifest metadata without loading JNI.
For indexes built with the pre-release `shard=centroid-based` option, drop the old index with
that build before upgrading, then rebuild with the new type. Older readers and Python index
builders do not support this layout; drop the centroid index before downgrading
or rebuilding it through those clients. The model file and each populated shard carry
training metadata, so measure model-loading cost and total index size alongside
recall, latency, and the number of opened files.

For quantized IVF indexes, each range or centroid shard fits its own codebooks
or bounds. Compare performance at a common recall target
using separate calibration and validation queries. To measure the effect of
quantization alone, probe every outer centroid and every local list. If it remains below the target,
increasing `centroid.nprobe` alone cannot remove quantization error. For IVF-RQ,
calibrate `rq.bits`; for DiskANN, calibrate both outer probes and local graph
search width. A shared routing model does not guarantee equal recall.

Calibrate `ivf-pq-centroid.nlist` and `ivf-pq-centroid.pq.m` together with the query probe width.
Increasing `nlist` creates more centroid files, each populated file carrying its own
local model. Increasing `pq.m` uses more encoded bytes per vector;
for the current 8-bit PQ codes, `pq.m=64` uses twice the code bytes of
`pq.m=32`. OPQ and larger training limits are also workload-dependent choices,
not recall guarantees. Record the resolved build options, total index size,
query file reads, and achieved recall when comparing configurations.

## Per-Field Options

The options above can also be set at the table level (in `TBLPROPERTIES`), where they are shared
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.paimon.globalindex;

import org.apache.paimon.predicate.FieldRef;

import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;

/** Base for vector-only readers: scalar predicates remain unavailable for index evaluation. */
public abstract class AbstractVectorGlobalIndexReader implements GlobalIndexReader {
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitIsNotNull(FieldRef fieldRef) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitIsNull(FieldRef fieldRef) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitStartsWith(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitEndsWith(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitContains(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLessThan(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterOrEqual(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitNotEqual(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLessOrEqual(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitEqual(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitGreaterThan(
FieldRef fieldRef, Object literal) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitIn(
FieldRef fieldRef, List<Object> literals) {
return CompletableFuture.completedFuture(Optional.empty());
}

@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitNotIn(
FieldRef fieldRef, List<Object> literals) {
return CompletableFuture.completedFuture(Optional.empty());
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.paimon.globalindex;

import org.apache.paimon.globalindex.io.GlobalIndexFileWriter;
import org.apache.paimon.utils.Range;

import java.io.IOException;
import java.io.Serializable;
import java.util.Iterator;
import java.util.List;

/** A vector indexer that can train and restore a router for independently built shards. */
public interface CentroidVectorGlobalIndexer extends VectorGlobalIndexer {

/** Persist the trained router and return the model entry with implementation-owned metadata. */
ResultEntry writeRoutingModel(
GlobalIndexFileWriter io,
String name,
byte[] payload,
String modelId,
int nlist,
List<Range> coverage)
throws IOException;

/** Return non-null input ordinals in the order required for training. */
long[] planGlobalTrainingSample(long vectorCount);

/** Train on ordered, preselected batches using the full non-null corpus size. */
VectorTrainingModel trainGlobalModel(long vectorCount, Iterator<TrainingBatch> batches);

VectorGlobalModelTrainer createGlobalModelTrainer();

VectorTrainingModel restoreTrainingModel(byte[] payload);

/** A row-major batch selected by the distributed sampler. */
final class TrainingBatch implements Serializable {
private static final long serialVersionUID = 1L;
public final float[] vectors;
public final int count;

public TrainingBatch(float[] vectors, int count) {
this.vectors = vectors;
this.count = count;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.paimon.globalindex;

import org.apache.paimon.utils.Range;

import java.util.List;

/**
* A centroid index groups all files for one field in a partition into a single generation. Its
* reader accepts and returns absolute table row IDs. Committers and scan planners can inspect the
* generation through this factory without opening files or initializing the native runtime.
*/
public interface CentroidVectorIndexFactory extends GlobalIndexerFactory {
/** Exact indexed row ranges for a model entry, or an empty list for a shard entry. */
List<Range> sourceRowRanges(byte[] indexMeta);

/** Validate that these entries form one complete generation and return its source coverage. */
List<Range> validateGeneration(List<byte[]> indexMetas);
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,20 @@

import java.util.List;

/** Index writer for single-column global index with relative row id (from 0 to rowCnt - 1). */
/**
* Single-column global index writer. Range writers consume relative row IDs; centroid vector
* writers consume absolute Paimon row IDs, as recorded in their index metadata.
*/
public interface GlobalIndexSingleColumnWriter extends GlobalIndexWriter {

/**
* Write the indexed key and its related relative row id to the index file.
* Write the indexed key and its row ID in the writer's declared encoding.
*
* @param key nullable index key
* @param relativeRowId local row id calculated by {@code rowId - rangeStart}
* @param rowId relative ID calculated by {@code absoluteRowId - rangeStart} for range writers,
* or the absolute row ID for a centroid vector writer
*/
void write(@Nullable Object key, long relativeRowId);
void write(@Nullable Object key, long rowId);

/**
* Finishes a writer whose normalized entries cover {@code sourceRowCount} source rows.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.annotation.Nullable;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -73,6 +75,18 @@ private static GlobalIndexerFactory find(String type) {
return null;
}

/** Look up the layout without constructing an indexer or interpreting its metadata. */
@Nullable
public static CentroidVectorIndexFactory centroidFactory(String type) {
GlobalIndexerFactory factory = find(type);
if (factory == null && type.endsWith("-centroid")) {
throw new IllegalStateException("Missing centroid index plugin for type: " + type);
}
return factory instanceof CentroidVectorIndexFactory
? (CentroidVectorIndexFactory) factory
: null;
}

public static GlobalIndexerFactory load(String type) {
GlobalIndexerFactory globalIndexerFactory = find(type);
if (globalIndexerFactory == null) {
Expand Down
Loading
Loading