Repository navigation
🐛 Preserve inline child edits when converting syntax trees to tokens - #453
chrisjsewell merged 8 commits into
Conversation
The token stream a tree was built from sees the edits; another tree built from the same tokens keeps its own nodes and writes its own children back.
…n place (#464) #453 made `SyntaxTreeNode.to_tokens()` keep edits to the children of inline and image nodes by assigning the rebuilt child list to the shared Token objects. That also changed the token stream the tree was built from, and a list returned earlier could change again after another tree built from the same tokens called `to_tokens()`. A token is now replaced by `token.copy(children=...)` only when its rebuilt children differ by identity from its current ones. Edits still reach the output, the original tokens are left unchanged, and an unedited round trip still returns the same Token objects and child lists. A copied image token makes the enclosing inline token's children differ, so it is copied too. The edit check compares the rebuilt children by identity, so a child replaced by an equal but distinct token still reaches the output. The docstring and `docs/using.md` drop the shared-state caveats and the deepcopy advice. They now say that the tree's `children` are the source of truth, so a change made directly to a token's `children` list does not appear in the output, and that `to_tokens()` does not modify the token stream the tree was built from, although nodes share their `Token` objects with it.
|
Merged in 4d7bf9e, thanks @KennyMcSimpson! Before merging I merged One follow-up since: as merged, |
SyntaxTreeNode.to_tokens()includes changes to block children but silently drops changes below inline and image nodes. For example, clearing an inline node's children still renders the original paragraph, and clearing image children retains the original alternative text.Rebuild token child lists from the current tree while keeping the original token objects. Add regression coverage for removal, reordering, nested edits and empty inline replacement, document the synchronization, and add a changelog entry.
Validation: the final regression tests produce 3 failures and 1 passing round-trip control on the original implementation. With this change, the complete Python 3.12 suite passes (1,035 tests), and all pre-commit hooks pass, including Ruff and strict mypy. Tests ran in UTF-8 mode on Windows.
Prepared with OpenAI Codex assistance; no independent human review is claimed.