Skip to content

Fix patchedast aborting on ''' inside a double-quoted f-string - #844

Open
marlon-costa-dc wants to merge 9 commits into
python-rope:masterfrom
marlon-costa-dc:fix/joinedstr-triple-quote-in-body
Open

marlon-costa-dc wants to merge 9 commits into
python-rope:masterfrom
marlon-costa-dc:fix/joinedstr-triple-quote-in-body

Conversation

@marlon-costa-dc

Copy link
Copy Markdown
Contributor

Description

patchedast._JoinedStr.end_quote_char() selects the closing delimiter by max(len(quote)) over every quote character found anywhere in the f-string body. When a ''' sequence appears inside a double-quoted f-string, e.g.:

f"cfg = '''\n{value}"

the ''' in the body is picked as the delimiter instead of the actual closing ". That produces a reconstructed token that consume_joined_string() cannot locate, raising:

ValueError: substring not found

which surfaces as MismatchedTokenError and aborts rename / inline for the entire module — a single such f-string anywhere in a project breaks refactoring of that file.

Root cause

end_quote_char() chose the quote by length precedence rather than by which delimiter the f-string literally ends with. The opening quote (start_quote_char) was already correct; the closing detection was not symmetric.

Fix

Pick the delimiter the f-string literally ends with. QUOTE_CHARS is ordered longest-first (""", ''', ", '), so triple-quoted strings are still matched before single-character quotes, and the closing quote now always matches the opening one regardless of quote characters in the body.

Reproducer (before the fix)

import ast
from rope.refactor import patchedast
patchedast.patch_ast(ast.parse('f"a = \'\'\'{y}"\n'), 'f"a = \'\'\'{y}"\n')
# ValueError: substring not found

Testing

  • New regression test test_handling_format_strings_with_triple_quote_in_body in ropetest/refactor/patchedasttest.py (red before, green after).
  • Full suites green: ropetest/refactor/ = 944 passed, 7 skipped, 1 xfailed.
  • patchedasttest.py = 141 passed, 6 skipped.

Checklist

  • I have added tests that prove my fix is effective
  • I have updated CHANGELOG.md
  • I have made corresponding changes to user documentation for new features (n/a — bug fix)
  • I have made corresponding changes to library documentation for API changes (n/a — no API change)

end_quote_char() picked the closing delimiter by max(len(quote)) over every
quote char found anywhere in the f-string body, so a ''' sequence inside a
double-quoted f-string (e.g. f"cfg = '''\n{value}") was selected as the
delimiter. That produced a reconstructed token consume_joined_string() could
not locate, raising `ValueError: substring not found` -> MismatchedTokenError,
which aborts rename/inline for the entire module.

Pick the delimiter the f-string literally ends with (QUOTE_CHARS is ordered
longest-first, so """/''' win over "/' correctly) so the closing quote matches
the opening one regardless of quote characters appearing in the body.

Adds a regression test; full patchedast, rename and inline suites stay green.
Copilot AI review requested due to automatic review settings July 22, 2026 13:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes patchedast f-string reconstruction when the f-string body contains quote sequences (notably ''') that previously confused _JoinedStr.end_quote_char(), causing consume_joined_string() failures and aborting refactorings for the whole module.

Changes:

  • Update _JoinedStr.end_quote_char() to select the closing delimiter based on what the source literally ends with (rather than the longest quote found in the body).
  • Add a regression test covering a double-quoted f-string containing ''' inside its body.
  • Document the fix in CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
rope/refactor/patchedast.py Fixes f-string closing delimiter detection to avoid mismatches caused by quote sequences in the f-string body.
ropetest/refactor/patchedasttest.py Adds a regression test proving the bug and preventing future regressions.
CHANGELOG.md Notes the bug fix and its impact on refactorings.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread rope/refactor/patchedast.py Resolved
Addresses review: end_quote_char fell back to the last character when no
quote delimiter matched, contradicting its intent and masking parsing
bugs. It now raises MismatchedTokenError naming the line, like the other
token-matching failures in patchedast.
@lieryan lieryan added the patchedast-string Issues related to patchedast when parsing string syntax. label Sep 30, 2026
@marlon-costa-dc

Copy link
Copy Markdown
Contributor Author

Rebased this fix onto current master (post-#896): the old end_quote_char shape is gone, but the bug persists — the post-#896 rfind-based version still picks the longest quote found anywhere in the body, so a ''' inside a double-quoted f-string returns a body suffix as its own delimiter.

The updated implementation reads the delimiter from whatever quote sequence actually terminates the region and raises MismatchedTokenError when none does (keeping the fail-loudly intent of the last commit here), plus the original regression test adapted to the current children layout. Available for comparison at dc/pr-844 (single commit, applies cleanly on master).

@marlon-costa-dc

Copy link
Copy Markdown
Contributor Author

Merged current master into this branch; the changelog entry now sits under the upcoming release.

The bug is still present on master. With the regression test's source, f"abc = '''{a}", get_patched_ast(source, True) raises ValueError: substring not found in consume_joined_string, because the closing delimiter is taken from the longest quote sequence in the body. With this branch, the test passes on Python 3.10, 3.11, 3.13 and 3.14.

@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.45%. Comparing base (ff8a306) to head (f9b43a7).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #844   +/-   ##
=======================================
  Coverage   95.44%   95.45%           
=======================================
  Files         134      134           
  Lines       26910    26924   +14     
=======================================
+ Hits        25685    25699   +14     
  Misses       1225     1225           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marlon-costa-dc added a commit to marlon-costa-dc/rope that referenced this pull request Nov 7, 2026
Our open python-rope/rope PRs carry tests that the fork release absorbed
in behaviour but not in tests. Each passes against the current code:
- python-rope#908 test_assignment_hints_ignore_names_that_are_not_assigned
- python-rope#909 test_scope_region_rejects_decorator_without_source_positions,
       test_scope_region_rejects_decorator_without_source_marker
- python-rope#920 test_hint_param_on_a_hierarchy_python_refuses
- python-rope#910 test_find_module_skips_python_path_files
- python-rope#844 test_format_string_region_without_closing_quote_is_rejected

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patchedast-string Issues related to patchedast when parsing string syntax.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants