Repository navigation
Conversation
| table = _open_schema_table(args) | ||
| try: | ||
| status = table.file_io.get_file_status(args.file) | ||
| if status.type != pafs.FileType.File: |
There was a problem hiding this comment.
[Med]
list-files and manifest emit _metadata_file_path(table, entry), which preserves file: schemes and percent-escaped partition directories (e.g., file:/warehouse/db.db/t/pt=a%2Fb/bucket-0/data-0.parquet). Passing args.file directly to table.file_io causes LocalFileIO._to_file() to URL-decode %2F into / and fail with FileNotFoundError.
We should convert the path with to_file_io_path(args.file) before calling file_io (matching DataFileMeta.physical_path()):
from pypaimon.utils.path import to_file_io_path
io_path = to_file_io_path(args.file)
status = table.file_io.get_file_status(io_path)
file_format = _data_file_format(basename(status.path)).lower()
if file_format != 'parquet':
raise ValueError(
f"Unsupported data file format '{file_format}' for '{args.file}'; only parquet is supported.")
with table.file_io.new_input_stream(io_path) as stream:There was a problem hiding this comment.
@Akash3121 Thanks for reviewing! Agreed. I've added a local change to preserve the emitted metadata path at both FileIO calls, with regression tests for the public list-files/manifest-to-file round trip on escaped partitions and for ordinary encoded file URIs.
| rollback_target = rollback_parser.add_mutually_exclusive_group(required=True) | ||
| rollback_target.add_argument( | ||
| '--snapshot', type=int, default=None, | ||
| '--snapshot-id', type=int, default=None, |
There was a problem hiding this comment.
Unifying this option name with --snapshot-id used by get, snapshot, list-manifests, and list-files makes the CLI much more consistent, and argparse sets args.snapshot_id automatically.
Two minor suggestions here:
- Keep
--snapshotas a backward-compatible alias ('--snapshot-id', '--snapshot') so existing scripts usingpaimon table rollback ... --snapshot <id>don't break. - Could this also use
type=_positive_intlike the--snapshot-idoption onget(line 1471),snapshot(line 1501),list-manifests(line 1531), andlist-files(line 1564)?
| '--snapshot-id', type=int, default=None, | |
| '--snapshot-id', '--snapshot', type=int, default=None, |
There was a problem hiding this comment.
@Akash3121 Thanks for reviewing! I've added --snapshot as an explicit alias and switched to _positive_int.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
I independently confirmed the existing path round-trip finding on bdd9b2a142 with a real filesystem catalog and committed Parquet data: warehouse configured as a file URI, partition value a/b, then public list-files --format json followed by file --file <exact emitted file_path>. The latter exits 1 because the literal pt=a%2Fb directory is URL-decoded; inspecting the same file using its physical local path succeeds. This breaks the documented diagnostic workflow. Please preserve the metadata path semantics at both FileIO calls and add this public command-to-command regression, while retaining support for ordinary encoded filesystem URIs.
90 CLI tests passed (108 subtests); project flake8 and git diff --check passed. The explicit legacy rollback --snapshot 1 command also succeeded: argparse currently accepts it as an unambiguous abbreviation, so I did not find a compatibility failure there. No additional independent code finding.
|
@JingsongLi Thanks for reviewing! Agreed. I’ve updated both FileIO calls to preserve the metadata path semantics, with regression tests for the public |
|
Please make sure the test passed. |
886ac2b to
93ffe40
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The CLI inspection commands have operational value. On 93ffe4095b, all 94 CLI tests passed, including the real escaped-partition list-files/file round trip and both rollback aliases; project-configured flake8 and diff checks passed. A separate public-CLI A/B probe reproduces one new URI-selection regression below. No source files were changed for the probe.
| io_path = to_file_io_path(args.file) | ||
| fallback_path = args.file | ||
| # Paimon emits literal file:/ paths; standard file:/// URIs are URL-encoded. | ||
| if args.file.startswith('file:///'): |
There was a problem hiding this comment.
[P2] Disambiguate localhost URIs before opening the file
Only file:/// receives URI-decoding precedence here. For a standard file://localhost/.../literal%20file.parquet URI, to_file_io_path instead tries the physical literal%20file.parquet first. If that file also exists beside the intended literal file.parquet, the stat succeeds and no fallback occurs, so the command silently reports the other file's schema/footer while echoing the original URI.
I reproduced this with two real Parquet files containing one and two rows respectively. The decoded physical path and file:///.../literal%20file.parquet both report two rows, but the equivalent localhost URI reports one. With the same files and URI, the previous bdd9b2a142 command reports the correct two rows; the regression is introduced by this revision.
Please make metadata-literal paths and standard URIs unambiguous, or reject inputs whose two interpretations identify different existing files. Simply reversing localhost precedence can break metadata paths that retain that authority, so preserve the metadata round-trip tests and add this coexistence case.
There was a problem hiding this comment.
@JingsongLi Thanks for reviewing! Agreed. I’ve updated the command to reject ambiguous inputs before opening the file, with regression tests for coexisting literal and decoded paths, including file://localhost/. The public list-files → file round-trip tests and ordinary encoded filesystem URI coverage are preserved.
39c333b to
4ba27f7
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS (one remaining P2).
The latest change fixes the previously reported lowercase file-URI ambiguity, and the inspection commands have operational value. All 94 CLI tests pass on this head, including the real list-files/file round trip and rollback aliases; current CI is green. An additional real public-CLI probe still finds the scheme-case variant below. It uses real catalog/FileIO/Parquet files; no production sources were modified.
| status = None | ||
| missing = None | ||
| # Check both literal Paimon paths and encoded file URIs without guessing their meaning. | ||
| for path in dict.fromkeys((to_file_io_path(args.file), args.file)): |
There was a problem hiding this comment.
[P2] Check file URI schemes case-insensitively for ambiguity
to_file_io_path only recognizes lowercase file:, while LocalFileIO's urlparse normalizes the URI scheme. Consequently FILE:/FiLe: produce one candidate here and skip the new ambiguity check. I created real collision%20file.parquet (one row) and collision file.parquet (two rows) and called the public CLI with equivalent encoded URIs. Lowercase file:///.../collision%20file.parquet correctly exits 1 with Ambiguous --file; FILE:///... and FiLe:///... both exit 0 and report the decoded file's two rows. Scheme casing should not silently change file selection. Normalize/check the scheme case-insensitively before constructing literal and decoded candidates (or explicitly reject the unsupported form), and add these variants to the coexisting-path regression.
There was a problem hiding this comment.
@JingsongLi Thanks for reviewing! Agreed. I’ve updated the command to normalize the file: scheme case-insensitively before checking both path candidates, while preserving the rest of the path. The coexisting-path regression tests now cover FILE: and FiLe:, including the localhost form.
81770ed to
42becbb
Compare
42becbb to
598cb35
Compare
Purpose
Implements the file commands proposed in #11073 (https://github.com/apache/paimon/issues/11073).
Adds read-only file inspection to the PyPaimon CLI:
These commands inspect metadata without scanning data records or modifying the table. The file command does not query snapshots or manifests. CLI documentation and output examples are included.
Also renames table rollback --snapshot to --snapshot-id for consistency with the other table commands.
Tests