Skip to content

Build a new Solution for each Python test case - #802

Open
TKkusa wants to merge 2 commits into
sysprog21:mainfrom
TKkusa:python-solution-per-case
Open

TKkusa wants to merge 2 commits into
sysprog21:mainfrom
TKkusa:python-solution-per-case

Conversation

@TKkusa

@TKkusa TKkusa commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

The Python driver ran every test case against one Solution built before the case loop, so an answer that keeps state on self started each case with what the earlier cases left behind. On Grid Segment Net Yield, the answer in the issue, with self.best set in __init__, showed Test results · 2/6, every case after the second returning 42; it now passes 6 of 6, as it already did in C++. An __init__ that raises, such as self.memo = {}[0], used to read as Could not find bestRouteYield in your code - keep the starter signature.; each case now fails with KeyError: 0.

Approach

The driver keeps its setup check that Solution defines the entry method and builds the instance inside each case's try, as the C++ and Java harnesses in web/compiler-explorer.js already do. Class judges and answers written as a plain function take the same path as before, and a test pins each. Another test makes sure each case runs on an instance __init__ built for it alone: every case reports how many times __init__ has run, whether self is the newest object it ran on, and the attributes self already had. An answer that does heavy work in __init__ now repeats it for every case, inside the same 5 s run budget and counted in that case's timeMs, which is also how C++ and Java construct.

pythonDriver is exported from web/runners.js so tests/browser/python-driver.test.js can run it on CPython from node --test, as suggested on #783. The tests skip only when the interpreter, python3 or the one PYTHON names, is missing; one that is there but does not start, exits nonzero or hangs fails them with what --version showed. A fix for #783 would need the same export, so whichever change lands second rebases onto the other. The fresh-state test also asserts that the judge's cases still expose a shared instance, so reordering the bank's cases cannot leave it passing on the old driver.

Validation

  • node --test tests/browser/python-driver.test.js: # pass 6, # fail 0. Before the fix the three tests of per-case state and construction fail (cases 3 to 6 got 42; a setup error instead of KeyError: 0). A driver that shares one instance, hands each case a copy.copy or copy.deepcopy of it, calls __init__ on it again (with or without clearing its __dict__), builds every instance before the loop, skips __init__, or repeats, stops at or returns the first constructor error fails at least one test. With PYTHON pointing at a missing interpreter all six skip; with a non-executable one or /bin/false all six fail with the probe's message.
  • Pyodide in Chromium through runBrowserTests: the issue's answer passes 6/6 and the raising __init__ gives KeyError: 0 in every case. I also checked both by hand in an offline Practice interview.
  • ./scripts/test.sh browser at the first head (e3d54427, the same web/runners.js): pass 2224, fail 0, with 1393 tests skipped because Playwright 1.65's Chromium was not installed here. The suites that run Python in the page (runners, graph-clone-identity, candidate-cases, task-runner, specialized-tree-cycles) and the new file, rerun with an installed Playwright 1.64.1 Chromium: pass 192, fail 0, 22 Java skips.
  • ./scripts/test.sh portable: review-memory-tests, git-hooks and shell-syntax fail here identically on a clean main (no gh in WSL, commentflow, shfmt and ruff not installed, shellcheck 0.11.0). ./scripts/test.sh rust passes here. Not run: Java (no javac).

Close #799


Summary by cubic

Builds a fresh Solution for each Python test case so answers that keep state on self no longer leak state between cases, matching what C++ and Java already do. On Grid Segment Net Yield an answer that set self.best in __init__ previously passed 2 of 6 cases, returning 42 after the second case; it now passes 6 of 6. A constructor that raises now fails each case with the real error (KeyError: 0) instead of Could not find bestRouteYield in your code - keep the starter signature.

  • Moves Solution() construction inside each case's try; the setup check that the entry method exists is unchanged.
  • One test pins that each case runs on an instance __init__ built for it alone: each case reports the construction count, whether it's the newest instance, and pre-existing attributes — a driver that shares, copies, re-initializes, or prebuilds an instance fails it.
  • Exports pythonDriver from web/runners.js so tests/browser/python-driver.test.js can run it on CPython, skipping when python3 is absent. The new tests also cover the paths the fix keeps: a missing entry method is still a setup error, class judges keep one shared instance per case, and plain-function answers still run.

Closes #799.

Written for commit 409fa1b. Summary will update on new commits.

View guided diff

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

@TKkusa

TKkusa commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

gates (browser 3/3) failed on the task editor scroll check with [900, 2276, 2276], the intermittent failure reported in #654 and addressed by #793. The last commit changes only tests/browser/python-driver.test.js, and the same shard passed on the three earlier heads with the same web/ tree. Rerunning the failed job should clear it.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase the current branch onto the upstream default branch and resolve any conflicts. Run /codetrial-review in AI agents (use $codetrial-review in Codex) and resolve all identified issues. Rework the series into minimal, logically coherent commits, squash closely related commits, and ensure all commit messages follow the project’s rules.

Comment thread tests/browser/python-driver.test.js Resolved
Comment thread tests/browser/python-driver.test.js Resolved
The Python driver built one Solution for the whole run, while the C++
and Java harnesses build one inside each case, so an answer that keeps
state on self started each case with what the earlier cases left: on
Grid Segment Net Yield the usual Binary Tree Maximum Path Sum answer
passed 2 of 6, every later case returning 42, though it passes all 6 in
C++. Solution() also ran inside the try that reports a missing method,
so an __init__ that raised read as "Could not find bestRouteYield in
your code". Build it inside each case's try instead, after the setup
check that the class defines the entry, so the work __init__ does now
counts toward each case's timeMs and the run's 5 s budget. The driver
is exported so a new test runs it on CPython from node --test, as asked
on sysprog21#783: besides the reported answer, it checks that every case runs on
an object __init__ built for it alone and that a constructor error fails
only its own case, and it skips only when the interpreter is missing.

Close sysprog21#799

Change-Id: I2f53fef2bae477241d763a5249f772980b6d2856
The previous commit reworked the branch of the Python driver that builds
the Solution an answer's entry is called on. Pin on CPython what it
keeps around that branch: a Solution without the entry method is still a
setup error, a class judge's methods within one case share an instance
while the next case starts a new one, and an answer written as a plain
module-level function still runs, so a later change to that block cannot
break them unnoticed.

Change-Id: I1e9a7fc3a96c92b8fac0ee06abe2eff8160aabc3
@TKkusa
TKkusa force-pushed the python-solution-per-case branch from 1bc3700 to 409fa1b Compare November 22, 2026 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python tests carry Solution state from one case to the next

2 participants