Skip to content

👌 Skip the substitutions in normalize when there is nothing to replace - #465

Open
g4lb wants to merge 1 commit into
executablebooks:masterfrom
g4lb:perf-normalize
Open

g4lb wants to merge 1 commit into
executablebooks:masterfrom
g4lb:perf-normalize

Conversation

@g4lb

@g4lb g4lb commented Nov 10, 2026

Copy link
Copy Markdown

normalize ran two regex substitutions on every input. The newline regex matched every "\n" and rebuilt the string, also when the input had no "\r". Now normalize runs the newline substitution only when the input contains "\r", and the regex matches only "\r\n?". str.replace now replaces NULL characters, and only when the input contains one. The output does not change.

Benchmark

Interleaved A/B, 20 to 30 rounds per side, Mann-Whitney test, Apple M5:

CPython 3.11 CPython 3.14 PyPy 3.11
parse of benchmarking/samples/spec.md -5.7% -5.8% -22.1%
render of spec.md -5.2% -5.1% -15.1%
render of one 10 KB paragraph -45.5% -45.9% -24.2%
parse of spec.md with CRLF line endings -2.4% -2.4% -1.7%

I measured no slowdown on 31 benchmarks, including empty input, CR-only and NUL-heavy input, all presets, renderer.render and SyntaxTreeNode. A first version that used only str.replace was 3.7% slower on PyPy for CRLF input, so this version keeps the regex for "\r".

Safety

  • Tokens, HTML, env and the syntax tree are identical on 1,003,726 inputs: the CommonMark spec, the port fixtures, the benchmark samples and random documents. I compared the commonmark, default, gfm-like, zero and js-default presets, and setups with typographer, breaks and mdit-py-plugins.
  • The full test suite passes on CPython 3.11, 3.14 and PyPy 3.11. pre-commit (ruff, ruff-format, mypy) is clean.
  • Added test_normalize for CRLF, CR, mixed line endings and NULL characters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant