[danmar]

Part of #742 (draft, work in progress).

Adds a macro expander that uses Prosser's hide-set algorithm, next to the old Macro::expand(). It is off by default and selected with DUI::hideSetExpansion (simplecpp option -hideset-expansion). testrunner runs every test with both expansions; HIDESET_EXPANSION_TODO() skips a test that the new expansion can't handle yet.

How well the new macro expansion works

clang and gcc preprocessor testsuites (run-tests.py; a test passes when the output is the same as clang's or gcc's):

Testsuite	Tests	Old expansion	New expansion
clang	136	124	126
gcc	19	16	16
total	155	140	142

• No test that passes with the old expansion fails with the new one.
• macro_paste_hard.c and macro_rescan_varargs.c fail with the old expansion (todo list of run-tests.py) and pass with the new one.
• The 13 tests that fail with the new expansion also fail with the old one.

testrunner: 323 tests, all run with both expansions. With the new expansion 6 are skipped:

Not handled yet	Tests
the old Token::macro and location rules	4
the comma before an empty __VA_ARGS__ is removed	1
an unterminated macro call	1

Real code: for 1,687 files (/usr/include, cppcheck lib and test, simplecpp testsuite) the output is the same with both expansions, except for 14 files of the clang/gcc testsuites. In 11 of them the new expansion gives clang's or gcc's result and the old one doesn't; in the other 3 both are wrong in the same way.

Correct with the new expansion, wrong with the old one (the results are the same as gcc's; the tests use TODO_OLD_EXPANSION_ASSERT_EQUALS()):

Input	gcc / new	old
#define f(x) x / f(f)(1)	f(1)	1
#define R() / R at the end of the file	R	error: wrong number of parameters
ELLIP_FUNC(F LPAREN 'a', 'b' RPAREN) (clang macro_rescan_varargs.c)	'a' + 'b'	F ( 'a' , 'b' )
#define r(x,y) x ## y / { r(,) } (C11 6.10.3.5 EXAMPLE 3 and 5)	{ }	an empty token between the braces
#define A x######x / A (gcc pr58844-1.c)	xx	x## x
#define F(...) <__VA_OPT__(x)> / F(E), E expands to nothing	<>	<x>
#define F(a,...) #__VA_OPT__(a b) / F(X,2) with #define X 1	"1 b"	"X" b
#define F(a,...) a ## __VA_OPT__(b c) ## d / F(x)	xd	x# "d"
#define HASH # / HASH define foo bar	# define foo bar on line 2	# on line 1 (the location of the #define)

simplecpp issues fixed by the new macro expansion

The examples in these issues give the same result as gcc with the new expansion; they are added as tests (TODO_OLD_EXPANSION_ASSERT_EQUALS() where the old expansion is wrong).

Issue	Example	Old	New / gcc
#226, #713	EVAL ( HELLO EMPTY() () )	HELLO ( )	"Hello World"
#713	EVAL(REPEAT(4, M, ~)) (cloak.h idioms)	REPEAT ( 3 , M , ~ ) 3	0 1 2 3
#656	PAR(PREFIX_ ## kind, (__VA_ARGS__))	PREFIX_SCALAR ( int , x )	int x
#394	MACRO_COMPOSE(FOO, BAR(1)) with A ## B	FOOBAR ( 1 )	1
#259	PREFIX##NAME##EXTENSION with empty arguments	error: invalid ##	int foo ;
#378	a ## ## c (consecutive ##)	x# "y"	xy
#458	# __VA_OPT__(# x), # __VA_OPT__(x ## __VA_ARGS__)	"x" "Hello" world	"\"x\"" "Helloworld"
#491	#define HASH # / HASH define foo bar	# on line 1	# define foo bar on line 2

For the #713 iteration idioms, a hide set only applies while an expansion is rescanned: a macro name that is found in the hide set of its own macro is never replaced ("painted", C11 6.10.3.4p2), but other tokens can be expanded again when they are rescanned later, for example as part of an argument.

Not fixed by this PR: defined(X ## Y) with parentheses in #if (#653, #218, #75) is not handled by either expansion; it can be fixed in findDefinedOperand(), which both use. An unterminated macro call (#414) is still not an error.

Commits

Preparation, can be split out into separate PRs:

• Regression tests: C11 6.10.3.5 examples, C23 __VA_OPT__ examples, invalid ##, and Token::macro, expanded token locations and MacroUsage (used by cppcheck). Known bugs use the new TODO_ASSERT_EQUALS().
• Functions broken out of the old expansion and used by the new one: the ## checks, stringify(), the argument count check in getMacroParameters(), expandBuiltin(), findDefinedOperand()/definedMacroName(), copyMacroCall(), setLocation(), and combineWithNext() in combineOperators().
• Fixes: ## of a name or number with a string literal is an error (cat(a,"s")); # keeps the space after an operator of two characters (str(a == 0)); a # from a macro at the start of an output line gets a space before it, as gcc and clang do.

Hide sets and the new expansion:

• Token::mExpandedFrom becomes the hide set mHideSet, with macro names instead of Macro pointers.
• HideSetExpander, added step by step: object-like and function-like macros, __VA_ARGS__ and the built-in macros; then #, ## (with placemarkers, as in C11 6.10.3.3), defined in a replacement list, and __VA_OPT__ (C23 6.10.5.1). A hide set only applies while the expansion is rescanned.

All commits except the fixes above have no behaviour change with the default (old) expansion.

Still to do

• the comma before an empty __VA_ARGS__ (MSVC-like, done by the old expansion)
• an unterminated macro call
• decide the Token::macro and location rules and follow them (cppcheck uses them)
• test in cppcheck with hideSetExpansion
• compare the speed

🤖 Generated with Claude Code

[danmar]

@chrchr-github @pfultz2 @web-flow @glankk @ludviggunne @wjakobsson I do not plan to merge this PR "as is" . This is a proof-of-concept PR written by AI. But can you please give me feedback about the "philosophy".

I want to restructure macro expansion in simplecpp to follow the standard better and use best-practices for C preprocessors. When I wrote simplecpp I did not have a complete understanding of macro expansion. Issue #742 has a section "Problems with the current macro expansion" that explains the problems.

If we agree about the goal and philosophy then I will try to start splitting out some refactorings from this PR that are small and safe. I will thereby reduce the size of this PR. And then when it's smaller we can make up a plan about how we convert macro expansion in simplecpp as safely as possible.
Testing the changes on a large number of Debian packages before we convert would seem logical to me.
We could also test some certain repos that we know are difficult.. I know we have problems with zephyr for instance.