Skip to content

patchedast: locate multi-line f-strings, replacement fields and match-sequence delimiters by parser coordinates - #882

Open
marlon-costa-dc wants to merge 9 commits into
python-rope:masterfrom
marlon-costa-dc:pr/parser-coordinates
Open

marlon-costa-dc wants to merge 9 commits into
python-rope:masterfrom
marlon-costa-dc:pr/parser-coordinates

Conversation

@marlon-costa-dc

Copy link
Copy Markdown
Contributor

Description

Three patchedast defects found by walking a ~6,300-file Python 3.13 codebase with get_patched_ast(source, True) and requiring write_ast() to reproduce each file exactly:

  1. Multi-line replacement fields (PEP 701). x = (f"{a}{\n b\n}{c}").upper() raised MismatchedTokenError: _JoinedStr located the literal with the textual string pattern, whose short-string form excludes newlines. The literal's region now comes from the node's own coordinates (ast_adapter), implicit concatenation included.
  2. Replacement field brace in a plain string of the same concatenation. In "({})," f"{a}" the field's { was searched for as text and matched the { inside the plain part, so the FormattedValue region started at ( and write_ast duplicated source. Since Python 3.12 the parser locates each FormattedValue at its own {; the walker starts there when the coordinate points at one (older parsers give the enclosing literal's position, which is left untouched). This also fixes the case covered by test_extract_method_f_string_false_format_value_in_regular_string, whose @expectedFailure is removed.
  3. Grouped first pattern in a match sequence (related to patchedast is currently not handling some match-case statement syntaxes #623). case (("a" | None), None): warned Unexpected character in MatchSequence's opening_paren <((> and duplicated the source on write, because every character before the first pattern was taken as the delimiter. The sequence's delimiters are now the bracket pair that encloses the whole node, found by tokenizing its segment; case (1), (2):, whose outer parentheses belong to its elements, is also handled.

Checklist

  • I have added tests that prove my fix is effective: three new tests in ropetest/refactor/patchedasttest.py fail on current master and pass here; the previously expected-failure extract test passes.
  • I have updated CHANGELOG.md

🤖 Generated with Claude Code

marlon-costa-dc and others added 3 commits September 25, 2026 12:28
A PEP 701 replacement field may span lines, which the textual string
pattern cannot match, so JoinedStr raised MismatchedTokenError; its
region now comes from the node's own coordinates, implicit concatenation
included. A MatchSequence took every character before its first pattern
as its opening delimiter, so a grouped first pattern (`((a | b), c)`)
produced `((`; the delimiters are now the bracket pair that encloses the
whole node, found by tokenizing its segment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In an implicit concatenation mixing f-strings and plain strings, the
field's opening brace was searched for as text and matched a "{" inside
a plain part (`"({})," f"{a}"`), shifting the field's region and
duplicating source on write. Since Python 3.12 the parser locates each
FormattedValue at its own "{"; the walker now starts there when the
coordinate points at one. test_extract_method_f_string_false_format_value
_in_regular_string, marked as an expected failure for this bug, passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marlon-costa-dc added a commit to marlon-costa-dc/rope that referenced this pull request Sep 25, 2026
Adds the parser-coordinate patchedast fixes (upstream PR python-rope#882): multi-line
f-string replacement fields, replacement-field braces in plain parts of a
concatenation, and grouped first patterns in match sequences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lieryan lieryan added this to the 1.16.0 milestone Sep 26, 2026
@lieryan lieryan added the patchedast-string Issues related to patchedast when parsing string syntax. label Sep 30, 2026
@marlon-costa-dc

Copy link
Copy Markdown
Contributor Author

Reworked the residual of this PR on top of current master (post-#896): the f-string bounding via parser coordinates landed through #896's ast_adapter regions, so what remains valuable here is the deterministic delimiter matching for MatchSequence — _get_surrounding_parens tokenizes the node segment and treats the sequence as delimited only when the bracket opening its first character closes at its last character (case (1), (2), grouped first patterns), removing the please report! RuntimeWarning heuristic.

Available for comparison at dc/pr-882 with the three tests from this PR adapted to the current master (patchedasttest 172 passed on 3.13).

@mcepl

mcepl commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

The patched-AST cleanup in #896 overlaps with the f-string parts of this PR, but the MatchSequence delimiter work still appears useful.

Could you update this PR against current master and narrow it to the remaining MatchSequence behavior? Please retain regression tests for grouped first patterns, such as case (1), (2), and verify both source regions and exact source round trips.

The dc/pr-882 branch linked in the discussion may be a useful starting point. Please also check whether any f-string regression still fails on master before dropping it. If one does, we should track it separately rather than assume the cleanup fixed it.

Once the remaining change is isolated, a fresh CI matrix should give us much more useful results.

@mcepl
mcepl requested review from lieryan and mcepl and removed request for lieryan October 26, 2026 21:11
marlon-costa-dc and others added 3 commits October 31, 2026 16:03
The f-string coordinate work of this pull request landed through python-rope#896,
so the merge takes master's patchedast, its tests and the changelog, and
narrows the pull request to the remaining MatchSequence delimiter fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_get_surrounding_parens stripped the text around the first and last
patterns and emitted a please-report RuntimeWarning whenever that
text was not a single bracket, which happens for grouping parentheses
of an element (case (("a" | b), c)) and for sequences whose elements
are separately parenthesized (case (1), (2)). Tokenize the segment
instead: the sequence has its own delimiters exactly when the bracket
opening its first character closes at its last character.

(cherry picked from commit 431061b)
The multiline replacement field case already passes on master after python-rope#896.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marlon-costa-dc

Copy link
Copy Markdown
Contributor Author

Updated against current master and narrowed to the MatchSequence delimiter behaviour, as requested:

  • The f-string coordinate parts are dropped, since patchedast cleanup and refactoring #896 covers them. I checked the remaining f-string case from the previous version (a multiline replacement field, x = (f"{a}{\n b\n}{c}").upper()): it passes on master with the exact source round trip, so there is nothing to track separately.
  • _get_surrounding_parens now tokenizes the node segment. The sequence has its own delimiters exactly when the bracket opening its first character closes at its last character. This removes the "please report" RuntimeWarning heuristic.
  • Regression tests cover grouped first patterns (case (("a" | None), None):) and separately parenthesized elements (case (1), (2):), checking both the MatchSequence source region and the exact round trip through write_ast. On master, case (1), (2): raises MismatchedTokenError: Token <)> at (2, 17) cannot be matched.

Tested locally on Python 3.10, 3.11, 3.13 and 3.14.

@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 95.48%. Comparing base (ff8a306) to head (ebdd40e).

Files with missing lines Patch % Lines
rope/refactor/patchedast.py 94.44% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #882      +/-   ##
==========================================
+ Coverage   95.44%   95.48%   +0.03%     
==========================================
  Files         134      134              
  Lines       26910    26936      +26     
==========================================
+ Hits        25685    25719      +34     
+ Misses       1225     1217       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mcepl mcepl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The narrowed MatchSequence change looks good to me. The regression tests cover the grouped-pattern and separately parenthesized-element cases, and the checks pass. Please update the PR description to remove the f-string fixes, which are no longer part of this diff and landed via #896.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patchedast-string Issues related to patchedast when parsing string syntax.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants