Skip to content

[core] Support batch scans by physical bucket ID - #1452

Open
naivedogger wants to merge 1 commit into
apache:mainfrom
naivedogger:physical-bucket-scan
Open

naivedogger wants to merge 1 commit into
apache:mainfrom
naivedogger:physical-bucket-scan

Conversation

@naivedogger

Copy link
Copy Markdown

Purpose

Closes #1450.

A worker assigned a physical Paimon bucket currently has to plan a broader scan and discard splits for other buckets. A worker shard does not identify a physical bucket: its contents depend on the table and scan mode.

Brief change log

Add TableScan::with_bucket(id). Snapshot scans prune manifests by bucket bounds and filter entries in the remaining manifests. Incremental scans apply the selection after validating entries, so it cannot hide an unsupported delete.

Tests

The scan test selects bucket 2 from manifests covering [0, 1] and [2, 3]. It checks the returned file and verifies that only the second manifest's entries were read. The shard test covers both orders of the conflicting builder calls.

API and Format

The new method applies to Paimon tables. Combining it with with_shard returns an error because a shard describes work distribution and need not correspond to one physical bucket. Existing scans and the storage format are unchanged.

Documentation

The scan API documents the scope of bucket selection and its interaction with sharding.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 105/105
AI-Contributed/UT: 0/0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support batch scans by physical bucket ID

1 participant