[pfultz2]

A function-like macro name that only becomes a call after its neighbours were substituted was never expanded. This fixes #226.

#define EMPTY()
#define EVAL(...) __VA_ARGS__
#define HELLO() "Hello World"
EVAL ( HELLO EMPTY() () )

simplecpp produced HELLO () where gcc and clang produce "Hello World". The same gap broke the DEFER/OBSTRUCT/EVAL idioms (cloak.h style), so iteration macros like REPEAT and WHILE never terminated correctly.

• Rescan substituted arguments. After an argument is pre-expanded and substituted into the replacement list, the result is rescanned together with the tokens that follow it, so a name followed by a ( that came from a different argument or from the replacement list is expanded as a call.
• Model the rescan cursor. A new Token::skippedByRescan flag records that the rescan already passed a function-like name without finding (. A ( produced later by another expansion does not turn it into a call, until the token is substituted into a new replacement list and rescanned again. This keeps EVAL(HELLO DEFER(EMPTY)() ()) at HELLO (), matching gcc.
• One trailing-call loop. expandTrailingCalls is the only place that expands a trailing macro name, with a parameter-aware lookahead for arguments that come from a parameter. Macro::expand marks its own trailing token when a call expanded to nothing, and the argument pre-expansion clears the flag at the substitution point. The flag sits next to whitespaceahead, so sizeof(Token) is unchanged.
• Common path unchanged. Arguments without any macro name skip the extra rescan.

[pfultz2]

@danmar Can you review this?

[dmcppcheck]

AI review

I found no wrong output that this PR introduces. In about 22,000 fuzzed inputs and in all my hand-written cases it never gave output that differs from gcc where base matched gcc. It fixes a lot: DEFER/OBSTRUCT/EVAL with REPEAT now gives 0 1 2 3 like gcc.

There are two new findings:

• Slower on nested macro arguments. An argument that contains any macro name is rescanned in full after substitution (simplecpp.cpp:2331-2332). The extra rescan runs even when the argument holds no deferred call. On deeply nested arguments this makes simplecpp about 3x slower. On a synthetic file heavy with nested calls it is about 25% slower. cppcheck preprocesses every file with simplecpp, so this is worth measuring or reducing.
• One case that is still wrong. It is not a regression, because base gives the same output. When a parameter's expansion starts a call that only the replacement-list tokens after it complete, the call is not expanded. With #define A(f,x) f x 1), A(F, (LP)) gives F(()1) where gcc and clang give [()1].

Testing

Build and suites (pr/): make -j8 testrunner simplecpp CXXOPTS=-Werror builds with no warnings. All of these pass:

• ./testrunner: exit 0
• python3 run-tests.py: 0 failed, 15 todo; macro_rescan_varargs.c now passes
• python3 -m pytest integration_test.py: 128 passed
• ./selfcheck.sh: exit 0

I also built base/.

Differential tests against gcc -E -P and clang -E -P, with whitespace removed before comparing. All of the following match gcc, and many were wrong in base:

• deferred calls through EMPTY() and E inside ID/EVAL
• full cloak-style EVAL(REPEAT(4, M, ~))
• the unexpanded REPEAT_INDIRECT () results at one and two levels
• F x where x is a parameter that is empty or that starts with (
• f x y (z) with empty parameters
• self-referential macros (f(x) f(x+1), g/h that call each other)
• # and ##, and __VA_OPT__
• unbalanced ( in object-like and function-like bodies (#define X F EMPTY() (1 then X ))
• the flag at top level (F EMPTY() (1), ID(F) E (4))

Fuzzing: I generated random token sequences over about 20 macros, including deferring and self-referencing ones. There were about 22,000 inputs in total, and I compared pr, base and gcc for each. There were 0 cases where base matched gcc and pr did not. The only mismatches were cases where base was wrong too, mostly "Wrong number of parameters" errors that both versions give.

Gap that is still there (base gives the same output):

#define ID(x) x
#define F(x) [x]
#define LP (
#define A(f,x) f x 1)
a: A(F, (LP))

/work/pr/simplecpp t11.c gives a: F ( ( ) 1 ), while gcc and clang give a: [()1].

Performance: I timed time ./simplecpp file on generated files. The outputs of base and pr are identical.

• 100 lines of x = ID(ID(...ID(F(0) F(1) ... F(49))...));:
  • depth 30: base 0.043s, pr 0.108s
  • depth 60: base 0.060s, pr 0.195s
  • depth 120: base 0.115s, pr 0.384s
• 3,000 lines of L4(ID(ID(F(j)))) + ID(F)(1), where L4 nests A(x) = F(ID(x)) ID(F(x)) four times: base 1.88s, pr 2.33s.
• On simplecpp.cpp itself I saw no measurable difference.

---

Automated review of 15f01b8 by Claude. The pull request was built and tested locally.

[dmcppcheck]

AI review

This rescan runs for every argument that contains any macro name, even when pre-expansion left nothing to find. The usual ID(ID(F(1) F(2) ...)) style then copies the whole substituted argument once more at every nesting level. I measured about 3x on deeply nested arguments (depth 120: 0.115s with base, 0.384s with this PR) and about 25% on a synthetic file heavy with macro calls; the output was identical. cppcheck preprocesses every file with this, so could the rescan run only when the substituted tokens still have a function-like macro name that pre-expansion did not expand?

[dmcppcheck]

AI review

This rescan only sees the parameter's expansion. When the expansion opens a call that the replacement list closes later, the call is not found. With #define LP (, #define F(x) [x] and #define A(f,x) f x 1), A(F, (LP)) gives F ( ( ) 1 ), while gcc and clang give [()1]. Base gives the same output, so this is not a regression. It is in the area that this PR handles, and there is no test for it.

[dmcppcheck]

This pull request has merge conflicts that must be resolved.

[dmcppcheck]

AI review

New regression: gatherCallAfter macro-expands a parameter that is an operand of ##. When the function-like name comes from a parameter (#define B(f,a,x) f a ## x), the PR now expands a as an ordinary argument and consumes it before the paste is done. The result is a call that gcc does not make, and in one case a token is dropped: B(F, (3) E, Z) gives [3]Z where gcc and clang give [3] EZ. Base gave F ( 3 ) EZ, which is also wrong but keeps every token. Base already had this bug when the name is written literally in the replacement list (F a ## x). The PR carries it over to names that come from a parameter.

The two earlier points are still open: the extra rescan cost on nested arguments (C3) and the A(F, (LP)) gap (C4). Apart from the ## case I found no new wrong output.

Earlier comments

#	Comment	By	Where	Status
1	Every argument that contains a macro name is rescanned again: up to ~3x slower on nested arguments	AI review	simplecpp.cpp:2332	⚠️ Open
2	A(F, (LP)) with f x 1) is not expanded like gcc (call completed by the replacement list)	AI review	simplecpp.cpp:2268	⚠️ Open (false negative)

@pfultz2, please fix, resolve or reply to the open comments: #1, #2.
For #2 (false negative), it is enough to create a GitHub issue and link it here. It does not need to be fixed in this PR.

Testing

Build and suites (pr/): make -j8 testrunner simplecpp CXXOPTS=-Werror builds cleanly. All suites pass:

• ./testrunner: exit 0
• python3 run-tests.py: 0 failed
• python3 -m pytest integration_test.py: 128 passed
• ./selfcheck.sh: exit 0

I also built base/ to compare.

Hand-written cases: these match gcc. They cover DEFER/EMPTY across ID/EVAL, object-like macros that expand to nothing between a name and (, empty and (-starting parameters, #x, CAT(a, ), and __VA_ARGS__ with , ##.

Fuzzing: about 9,000 gcc-valid random token sequences over 20 macros (##, #, __VA_OPT__, LP/RP, self-reference, DEFER/OBSTRUCT). No case where base matched gcc and pr did not, and no crash or hang. The generator never puts a parameter before ## inside a definition, which is the case below.

Regression with ## (scratch/p3.c, p4.c):

#define F(x) [x]
#define E
#define ID(x) x
#define B(f,a,x) f a ## x
1: B(F, (1) E, Z)
#define A(f,x,y) f x ## y
3: A(F, E (3), )
4: ID(A(F, E (4), ))

/work/pr/simplecpp p3.c:

1 : [ 1 ]Z
3 : [ 3 ]
4 : [ 4 ]

/work/base/simplecpp p3.c:

1 : F ( 1 ) EZ
3 : F E ( 3 )
4 : F E ( 4 )

gcc -E -P and clang -E -P:

1: [1] EZ
3: F (3)
4: [4]

For line 1 the PR drops the E of the pasted EZ. For line 3 it calls F, which gcc does not. When F is written literally in the replacement list (#define B2(a,x) F a ## x, B2((2) E, Z)), base already gives the same wrong [ 2 ]Z.

Earlier points rechecked:

• C4: A(F, (LP)) still gives F ( ( ) 1 ); gcc gives [()1].
• C3: depth-60 nested ID(...) file: base 0.066s, pr 0.211s.

---

Automated review of e1fe1ab by Claude. The pull request was built and tested locally.

[dmcppcheck]

AI review

after can be the left operand of ##. Here it is macro-expanded like an ordinary argument and consumed, before Macro::expand gets to the paste. When the macro name comes from a parameter, that is new in this PR:

#define F(x) [x]
#define E
#define B(f,a,x) f a ## x
#define A(f,x,y) f x ## y
B(F, (1) E, Z)
A(F, E (3), )

• pr gives [ 1 ]Z and [ 3 ]
• gcc and clang give [1] EZ and F (3)
• base gives F ( 1 ) EZ and F E ( 3 )

In the first line the pasted EZ loses its E. The old code had the same problem only when the name is written literally in the replacement list (F a ## x). A parameter that is followed by ## (or that is the operand of #) should not be expanded here. Stop the lookahead at it instead. A test with ## after the parameter would cover this.

[dmcppcheck]

AI review

I found no new problems. The ## regression from C6/C7 is still there: B(F, (1) E, Z) with #define B(f,a,x) f a ## x gives [ 1 ]Z, while gcc and clang give [1] EZ. The other two earlier points are fixed now. On the nested ID(...) benchmark from C3, pr runs as fast as base and gives identical output. A(F, (LP)) now gives [ ( ) 1 ] like gcc and has a test (define28, code7). Apart from the ## case, every input that I tried for the new rescan and the skippedByRescan flag gives the same output as gcc and clang. That includes many cases where base was wrong, and #if ID(ONE EMPTY() ()) == 1, which base rejected with an error.

Earlier comments

#	Comment	By	Where	Status
1	Rescan of every argument with a macro name makes nested arguments ~3x slower	AI review	simplecpp.cpp:2332	✅ Fixed
2	A(F, (LP)) with f x 1) is not expanded like gcc	AI review	simplecpp.cpp:2268	✅ Fixed
3	gatherCallAfter expands a parameter that is an operand of ## (B(F,(3) E,Z) drops E)	AI review	general	⚠️ Open
4	Parameter before ## is expanded and consumed by the lookahead; B(F, (1) E, Z) gives [ 1 ]Z	AI review	simplecpp.cpp:2269	⚠️ Open

@pfultz2, please fix, resolve or reply to the open comments: #3, #4.

Testing

Build and suites (pr/): make -j8 testrunner simplecpp CXXOPTS=-Werror builds with no warnings. All suites pass:

• ./testrunner: 0 failed
• python3 run-tests.py: 0 failed, 14 todo
• python3 -m pytest integration_test.py: 128 passed
• ./selfcheck.sh: exit 0

I built base/ too, to compare.

Hand-written cases in scratch/q.c, r.c, s.c, u.c and v.c, about 60 lines in total, compared with gcc -E -P and clang -E -P. They cover:

• F E (1) and F EMPTY() (1) at top level and inside ID/EVAL
• DEFER and double DEFER, and a deferred name that comes from a parameter (TWO(DEFER(F), DEFER(ID)((4))))
• object-like macros that end in EMPTY()
• H(F) (6), where the rescan passes a trailing name
• calls split across lines
• LP/RP from parameters (A(F, LP 2 RP))
• #x after the call
• __VA_ARGS__
• a deferred call inside #if

All of these match gcc. In about 20 of them base was wrong.

Recursion: EVAL(R(0)) with R(x) x DEFER(R_I)()(x+1), a self-referential S() S EMPTY() () and an OBSTRUCT loop all give the same output as gcc after removing whitespace. pr takes 0.31 s here and base 0.004 s, but base stops expanding and gives the wrong result. There is no hang.

Performance (C3): 100 lines of ID( nested 120 deep around F(0) … F(49). base takes 0.118 s and pr 0.112 s, with identical output.

Still open (C7):

#define F(x) [x]
#define E
#define B(f,a,x) f a ## x
1: B(F, (1) E, Z)
#define A(f,x,y) f x ## y
3: A(F, E (3), )

/work/pr/simplecpp p3.c gives 1 : [ 1 ]Z and 3 : [ 3 ]. gcc gives 1: [1] EZ and 3: F (3).

Same in base, so not a regression: #define C(f,x,y) f x ## y 5) with C(F,,(5)) gives F ( 5 ) 5 ). gcc gives [5] 5). This is the other side of the same ## lookahead in gatherCallAfter.

---

Automated review of 08c7c43 by Claude. The pull request was built and tested locally.