Skip to content

[core] Support opt-in partition fallback for filesystem append reads - #1458

Draft
taoran92 wants to merge 1 commit into
apache:mainfrom
taoran92:feature/core-partition-fallback
Draft

taoran92 wants to merge 1 commit into
apache:mainfrom
taoran92:feature/core-partition-fallback

Conversation

@taoran92

@taoran92 taoran92 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #1455.

Let Rust callers fill missing partitions from the branch named by scan.fallback-branch when batch-reading a partitioned append-only table through the filesystem catalog. Reads stay on the current branch by default.

Brief change log

  • Add ReadBuilder::with_partition_fallback() for latest-snapshot batch reads. Plan current-branch partition ownership before applying data filters and use the backup only for unowned partitions.
  • Route each fallback split through its source branch on the reader. Exclude shadowed backup partitions before applying a split limit.

Tests

The partition_fallback_test integration target passed on the merge with main. Its overlap case returns ID 1 by default, IDs 1 and 2 with fallback, and zero rows for id = 99. It also covers transported splits, projection, limits, empty branches, and schema changes.

API and Format

Call with_partition_fallback() on the builder used for planning and reading. Split transport uses Java's existing FallbackDataSplit frame (type 6). Table data and manifest formats are unchanged.

Documentation

The new builder method documents the opt-in behavior, supported tables, and scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support opt-in partition fallback for Rust batch reads

1 participant