Skip to content

gh-129724: Intern code object names before merging them with constants - #161457

Draft
gpshead wants to merge 1 commit into
python:mainfrom
gpshead:names-tuple-intern
Draft

gpshead wants to merge 1 commit into
python:mainfrom
gpshead:names-tuple-intern

Conversation

@gpshead

@gpshead gpshead commented Nov 6, 2026

Copy link
Copy Markdown
Member

The compiler merges the co_names and local variable names tuples of a code object with its constant cache, so they can be the same object as an equal tuple constant of an earlier function. _PyCode_New() then interns the items of that shared tuple in place: if an equal string was already interned in the process, the constant's string is replaced by it, otherwise the constant's own string becomes the interned one. Which constants are the same object, and so the bytes marshal writes, depended on what was interned before. This change interns the names in makecode() before the merge.

It needs a name that is not interned yet and is not an ASCII identifier, since the parser interns identifiers and the compiler interns ASCII identifier-like string constants: a private name of a class with a non-ASCII name, which each compilation mangles into a new string, or an identifier of an AST passed to compile() that the parser did not create.

import subprocess, sys
src = '''\
class \xc9:
    def g(self):
        return ("_\xc9__x",), "_\xc9__x"
    def f(self):
        return self.__x
'''
child = f'''
import hashlib, marshal, sys
if sys.argv[1] == "interned":
    keep = sys.intern("_\\xc9" + "__x")
print(hashlib.sha256(marshal.dumps(compile({src!r}, "t", "exec"))).hexdigest()[:12])
'''
for state in "fresh", "interned":
    print(state, subprocess.check_output([sys.executable, "-c", child, state], text=True).strip())

On main the two digests differ, and compiling the same source twice in one process also gives different bytes. The free-threaded build was not affected, because it interns all string constants.

Part of gh-129724.

…nstants

makecode() merges the co_names and co_localsplusnames tuples with the
constant cache, so they can become an equal tuple constant of another
code object, and _PyCode_New() then interns the items of that tuple in
place.  If an equal string was already interned in the process, the
string of the constant was replaced by it; otherwise the string of the
constant itself became the interned one.  The constants which are the
same object, and so the marshal output, depended on the process state.

The parser interns identifiers and the compiler interns the ASCII
identifier-like string constants, so this needs a name which is not
interned yet and is not an ASCII identifier: a private name of a class
with a non-ASCII name, which each compilation mangles into a new string,
or an identifier of an AST passed to compile() which was not created by
the parser.

Intern the names before the merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant