Skip to content

Fix #739 (Add a lookahead buffer to TokenList::Stream) - #740

Merged
danmar merged 7 commits into
masterfrom
fix-739
Nov 7, 2026
Merged

danmar merged 7 commits into
masterfrom
fix-739

Conversation

@danmar

@danmar danmar commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

dmcppcheck and others added 4 commits November 7, 2026 20:10
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@dmcppcheck dmcppcheck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review

I found no problems. Inputs with a lone CR (\r) used to drop the last token (#define A 1\rA\r gave empty output) or report a false "No newline at end of file". The PR fixes this. UTF-16 files read through FileStream with CR or CRLF newlines used to hang the base build until it was killed; with the PR they are processed correctly. With LF, CRLF or EOF without a newline, the output is the same as base, and speed did not change. One cleanup remains: Stream::good() is no longer called anywhere, but every stream still has to implement it.

Testing

Build and test suites

  • Built make -j8 testrunner simplecpp CXXOPTS=-Werror in pr/ with no warnings, and built base/ to compare.
  • ./testrunner passed with no failures.
  • python3 run-tests.py: 176 tests, 0 failed (21 skipped, 15 todo).
  • python3 -m pytest integration_test.py: 128 passed.
  • ./selfcheck.sh exited with 0.

Scratch files, PR compared with base and gcc -E -P

Input PR Base gcc
printf '#define A 1\rA\r' 1 empty output plus a false "No newline at end of file" 1
printf 'x\r' x false "No newline at end of file" —
printf 'x /\r* c *\r/ y\r/\r/ z\r' / and / z on their own lines, like gcc drops the last two lines —
printf '#if 1 //x\\\r#endif\rq\r' q, the same as the PR and base give for the LF and CRLF versions false "failed to evaluate #if condition" —
UTF-16 LE and BE files with BOM and mixed \r\n / \r / \n newlines (made with Python) correct output hung until it was killed —

Same as base

  • Files ending without a newline: abc, #error abc, "abc", a /, 1'2.
  • a\r\r\nb.
  • An unterminated raw string (R"x(ab).
  • A UTF-16 file with an odd number of bytes.

Speed

  • A 120k-line file took about 1.17 s with both base and PR.

Not covered by the PR

  • a\\\rb (backslash before a lone CR) gives a b, while gcc splices the lines into ab. This is not new: base gives the same, and LF input a\\\nb also gives a b.
  • The new unit tests only use in-memory input. Nothing tests the UTF-16 file case through FileStream that used to hang.

Automated review of 3530830 by Claude. The pull request was built and tested locally.

Comment thread simplecpp.cpp Resolved
dmcppcheck and others added 3 commits November 7, 2026 21:52
Calling them from outside Stream would skip the lookahead buffer,
UTF-16 decoding and CR handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danmar
danmar merged commit bccf566 into master Nov 7, 2026
47 checks passed
@danmar
danmar deleted the fix-739 branch November 7, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants