Skip to content

[core] Support multiple external paths for global index - #11248

Open
lszskye wants to merge 2 commits into
apache:masterfrom
lszskye:support_multi_global_index_external_path
Open

lszskye wants to merge 2 commits into
apache:masterfrom
lszskye:support_multi_global_index_external_path

Conversation

@lszskye

@lszskye lszskye commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

Allow global-index.external-path to configure multiple root directories,
for example s3://bucket-a/index,s3://bucket-b/index. New index files are
distributed across these directories in round-robin order.

  • Separate new-file path allocation from existing-file path resolution.
  • Allocate each file's full path once and carry it through writing,
    ResultEntry, and commit metadata generation.
  • Preserve round-robin state across file rotation and synchronize shared
    path allocation.
  • Update index implementations, adapters, tests, and documentation to use
    the full-path contract consistently.

Tests

  • Configuration parsing, round-robin distribution, and concurrent allocation.
  • Allocated-path preservation through writing and commit metadata generation,
    including reads after external-path configuration changes.
  • Targeted Bitmap, BTree, FMIndex, primary-key index, Spark, and ES index tests.
  • Multiple external-directory cleanup in FileSystemCatalogTest and
    CachingCatalogTest.

FileStoreTable table, String indexType, List<DataField> indexFields, Options options)
throws IOException {
return createIndexWriter(
createGlobalIndexFileReadWrite(table), indexType, indexFields, options);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In SortedGlobalIndexWriter, fileReadWrite is cached across rotated files so that createGlobalIndexFileReadWrite(table) is not called per file (since table.store().pathFactory().globalIndexFileFactory() creates a new RoundRobinExternalPathProvider whose starting cursor is randomized via ThreadLocalRandom).

However, in GenericIndexTopoBuilder.BuildIndexOperator#processElement, a single operator subtask can process multiple IndexedSplit shards and still calls createIndexWriter(table, ...) on every shard. That recreates a new RoundRobinExternalPathProvider with a fresh random starting position for each shard instead of round-robining across shards processed by the same subtask.

Could we also initialize a transient GlobalIndexFileReadWrite once in BuildIndexOperator#open() and pass it to createIndexWriter(fileReadWrite, ...) (just like SortedGlobalIndexWriter)?

List<Path> paths = new ArrayList<>();
for (String value : pathString.split(",")) {
Path path = new Path(value.trim());
checkArgument(path.toUri().getScheme() != null, "scheme should not be null: %s", path);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CoreOptions#globalIndexExternalPaths trims each comma-separated path (value.trim()) and ignores "" (and CoreOptionsTest explicitly tests values with spaces and trailing commas like "file:///a, file:///b", " file:///a/ ", and "").

However, during dropTable, AbstractCatalog#getSchemaExternalPaths splits the raw option value directly via:

.flatMap(externalPath -> Arrays.stream(externalPath.split(",")))
.map(Path::new)

without trimming or filtering empty strings. If global-index.external-path is configured with spaces around the comma (e.g., "file:///a, file:///b") or "", new Path(" file:///b") / new Path("") in AbstractCatalog#getSchemaExternalPaths will throw IllegalArgumentException (URISyntaxException: Illegal character in scheme name at index 0 / Can not create a Path from an empty string) when dropping the table.

Should we update AbstractCatalog#getSchemaExternalPaths to .map(String::trim).filter(s -> !s.isEmpty()).map(Path::new) so dropTable handles the same inputs accepted by CoreOptions#globalIndexExternalPaths?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants