Repository navigation
Conversation
| FileStoreTable table, String indexType, List<DataField> indexFields, Options options) | ||
| throws IOException { | ||
| return createIndexWriter( | ||
| createGlobalIndexFileReadWrite(table), indexType, indexFields, options); |
There was a problem hiding this comment.
In SortedGlobalIndexWriter, fileReadWrite is cached across rotated files so that createGlobalIndexFileReadWrite(table) is not called per file (since table.store().pathFactory().globalIndexFileFactory() creates a new RoundRobinExternalPathProvider whose starting cursor is randomized via ThreadLocalRandom).
However, in GenericIndexTopoBuilder.BuildIndexOperator#processElement, a single operator subtask can process multiple IndexedSplit shards and still calls createIndexWriter(table, ...) on every shard. That recreates a new RoundRobinExternalPathProvider with a fresh random starting position for each shard instead of round-robining across shards processed by the same subtask.
Could we also initialize a transient GlobalIndexFileReadWrite once in BuildIndexOperator#open() and pass it to createIndexWriter(fileReadWrite, ...) (just like SortedGlobalIndexWriter)?
| List<Path> paths = new ArrayList<>(); | ||
| for (String value : pathString.split(",")) { | ||
| Path path = new Path(value.trim()); | ||
| checkArgument(path.toUri().getScheme() != null, "scheme should not be null: %s", path); |
There was a problem hiding this comment.
CoreOptions#globalIndexExternalPaths trims each comma-separated path (value.trim()) and ignores "" (and CoreOptionsTest explicitly tests values with spaces and trailing commas like "file:///a, file:///b", " file:///a/ ", and "").
However, during dropTable, AbstractCatalog#getSchemaExternalPaths splits the raw option value directly via:
.flatMap(externalPath -> Arrays.stream(externalPath.split(",")))
.map(Path::new)without trimming or filtering empty strings. If global-index.external-path is configured with spaces around the comma (e.g., "file:///a, file:///b") or "", new Path(" file:///b") / new Path("") in AbstractCatalog#getSchemaExternalPaths will throw IllegalArgumentException (URISyntaxException: Illegal character in scheme name at index 0 / Can not create a Path from an empty string) when dropping the table.
Should we update AbstractCatalog#getSchemaExternalPaths to .map(String::trim).filter(s -> !s.isEmpty()).map(Path::new) so dropTable handles the same inputs accepted by CoreOptions#globalIndexExternalPaths?
Purpose
Allow
global-index.external-pathto configure multiple root directories,for example
s3://bucket-a/index,s3://bucket-b/index. New index files aredistributed across these directories in round-robin order.
ResultEntry, and commit metadata generation.path allocation.
the full-path contract consistently.
Tests
including reads after external-path configuration changes.
FileSystemCatalogTestandCachingCatalogTest.