Skip to content

[python] Add list-files & file command support for Python CLI. - #11154

Open
fengruotj wants to merge 2 commits into
apache:masterfrom
fengruotj:feat/pypaimon-cli-file
Open

fengruotj wants to merge 2 commits into
apache:masterfrom
fengruotj:feat/pypaimon-cli-file

Conversation

@fengruotj

Copy link
Copy Markdown
Contributor

Purpose

Implements the file commands proposed in #11073 (https://github.com/apache/paimon/issues/11073).

Adds read-only file inspection to the PyPaimon CLI:

  • paimon table list-files <database.table> lists live data files referenced by the latest or a specified snapshot, including the snapshot ID and file metadata. It supports --snapshot-id, --bucket, and --limit, with table output by default or JSON output.
  • paimon table file <database.table> --file reads a Parquet file at a path or filesystem URI and displays its schema and footer metadata, including row groups and column statistics. It supports JSON output by default or table output, with schema and metadata displayed as indented, multiline JSON.

These commands inspect metadata without scanning data records or modifying the table. The file command does not query snapshots or manifests. CLI documentation and output examples are included.

Also renames table rollback --snapshot to --snapshot-id for consistency with the other table commands.

Tests

  • paimon-python/pypaimon/tests/cli_table_test.py: 90 passed, including file inspection regression tests.
  • flake8 for the changed Python files: passed.

@fengruotj fengruotj changed the title [python] Add list-files & file by file support for Python CLI. [python] Add list-files & file command support for Python CLI. Nov 14, 2026
table = _open_schema_table(args)
try:
status = table.file_io.get_file_status(args.file)
if status.type != pafs.FileType.File:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Med]
list-files and manifest emit _metadata_file_path(table, entry), which preserves file: schemes and percent-escaped partition directories (e.g., file:/warehouse/db.db/t/pt=a%2Fb/bucket-0/data-0.parquet). Passing args.file directly to table.file_io causes LocalFileIO._to_file() to URL-decode %2F into / and fail with FileNotFoundError.

We should convert the path with to_file_io_path(args.file) before calling file_io (matching DataFileMeta.physical_path()):

        from pypaimon.utils.path import to_file_io_path

        io_path = to_file_io_path(args.file)
        status = table.file_io.get_file_status(io_path)
        file_format = _data_file_format(basename(status.path)).lower()
        if file_format != 'parquet':
            raise ValueError(
                f"Unsupported data file format '{file_format}' for '{args.file}'; only parquet is supported.")

        with table.file_io.new_input_stream(io_path) as stream:

@fengruotj fengruotj Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Akash3121 Thanks for reviewing! Agreed. I've added a local change to preserve the emitted metadata path at both FileIO calls, with regression tests for the public list-files/manifest-to-file round trip on escaped partitions and for ordinary encoded file URIs.

Comment thread paimon-python/pypaimon/cli/cli_table.py Outdated
rollback_target = rollback_parser.add_mutually_exclusive_group(required=True)
rollback_target.add_argument(
'--snapshot', type=int, default=None,
'--snapshot-id', type=int, default=None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unifying this option name with --snapshot-id used by get, snapshot, list-manifests, and list-files makes the CLI much more consistent, and argparse sets args.snapshot_id automatically.

Two minor suggestions here:

  1. Keep --snapshot as a backward-compatible alias ('--snapshot-id', '--snapshot') so existing scripts using paimon table rollback ... --snapshot <id> don't break.
  2. Could this also use type=_positive_int like the --snapshot-id option on get (line 1471), snapshot (line 1501), list-manifests (line 1531), and list-files (line 1564)?
Suggested change
'--snapshot-id', type=int, default=None,
'--snapshot-id', '--snapshot', type=int, default=None,

@fengruotj fengruotj Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Akash3121 Thanks for reviewing! I've added --snapshot as an explicit alias and switched to _positive_int.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

I independently confirmed the existing path round-trip finding on bdd9b2a142 with a real filesystem catalog and committed Parquet data: warehouse configured as a file URI, partition value a/b, then public list-files --format json followed by file --file <exact emitted file_path>. The latter exits 1 because the literal pt=a%2Fb directory is URL-decoded; inspecting the same file using its physical local path succeeds. This breaks the documented diagnostic workflow. Please preserve the metadata path semantics at both FileIO calls and add this public command-to-command regression, while retaining support for ordinary encoded filesystem URIs.

90 CLI tests passed (108 subtests); project flake8 and git diff --check passed. The explicit legacy rollback --snapshot 1 command also succeeded: argparse currently accepts it as an unambiguous abbreviation, so I did not find a compatibility failure there. No additional independent code finding.

@fengruotj

fengruotj commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for reviewing! Agreed. I’ve updated both FileIO calls to preserve the metadata path semantics, with regression tests for the public list-files → file round trip on escaped partitions and for ordinary encoded filesystem URIs.

@JingsongLi

Copy link
Copy Markdown
Contributor

Please make sure the test passed.

@fengruotj
fengruotj force-pushed the feat/pypaimon-cli-file branch from 886ac2b to 93ffe40 Compare November 16, 2026 13:54

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

The CLI inspection commands have operational value. On 93ffe4095b, all 94 CLI tests passed, including the real escaped-partition list-files/file round trip and both rollback aliases; project-configured flake8 and diff checks passed. A separate public-CLI A/B probe reproduces one new URI-selection regression below. No source files were changed for the probe.

Comment thread paimon-python/pypaimon/cli/cli_table.py Outdated
io_path = to_file_io_path(args.file)
fallback_path = args.file
# Paimon emits literal file:/ paths; standard file:/// URIs are URL-encoded.
if args.file.startswith('file:///'):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Disambiguate localhost URIs before opening the file

Only file:/// receives URI-decoding precedence here. For a standard file://localhost/.../literal%20file.parquet URI, to_file_io_path instead tries the physical literal%20file.parquet first. If that file also exists beside the intended literal file.parquet, the stat succeeds and no fallback occurs, so the command silently reports the other file's schema/footer while echoing the original URI.

I reproduced this with two real Parquet files containing one and two rows respectively. The decoded physical path and file:///.../literal%20file.parquet both report two rows, but the equivalent localhost URI reports one. With the same files and URI, the previous bdd9b2a142 command reports the correct two rows; the regression is introduced by this revision.

Please make metadata-literal paths and standard URIs unambiguous, or reject inputs whose two interpretations identify different existing files. Simply reversing localhost precedence can break metadata paths that retain that authority, so preserve the metadata round-trip tests and add this coexistence case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JingsongLi Thanks for reviewing! Agreed. I’ve updated the command to reject ambiguous inputs before opening the file, with regression tests for coexisting literal and decoded paths, including file://localhost/. The public list-files → file round-trip tests and ordinary encoded filesystem URI coverage are preserved.

@fengruotj
fengruotj force-pushed the feat/pypaimon-cli-file branch 2 times, most recently from 39c333b to 4ba27f7 Compare November 17, 2026 03:11

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS (one remaining P2).

The latest change fixes the previously reported lowercase file-URI ambiguity, and the inspection commands have operational value. All 94 CLI tests pass on this head, including the real list-files/file round trip and rollback aliases; current CI is green. An additional real public-CLI probe still finds the scheme-case variant below. It uses real catalog/FileIO/Parquet files; no production sources were modified.

Comment thread paimon-python/pypaimon/cli/cli_table.py Outdated
status = None
missing = None
# Check both literal Paimon paths and encoded file URIs without guessing their meaning.
for path in dict.fromkeys((to_file_io_path(args.file), args.file)):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Check file URI schemes case-insensitively for ambiguity

to_file_io_path only recognizes lowercase file:, while LocalFileIO's urlparse normalizes the URI scheme. Consequently FILE:/FiLe: produce one candidate here and skip the new ambiguity check. I created real collision%20file.parquet (one row) and collision file.parquet (two rows) and called the public CLI with equivalent encoded URIs. Lowercase file:///.../collision%20file.parquet correctly exits 1 with Ambiguous --file; FILE:///... and FiLe:///... both exit 0 and report the decoded file's two rows. Scheme casing should not silently change file selection. Normalize/check the scheme case-insensitively before constructing literal and decoded candidates (or explicitly reject the unsupported form), and add these variants to the coexisting-path regression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JingsongLi Thanks for reviewing! Agreed. I’ve updated the command to normalize the file: scheme case-insensitively before checking both path candidates, while preserving the rest of the path. The coexisting-path regression tests now cover FILE: and FiLe:, including the localhost form.

@fengruotj
fengruotj force-pushed the feat/pypaimon-cli-file branch 3 times, most recently from 81770ed to 42becbb Compare November 18, 2026 09:47
@fengruotj
fengruotj force-pushed the feat/pypaimon-cli-file branch from 42becbb to 598cb35 Compare November 18, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants