Repository navigation
Cover overshooting final hops in the judge - #805
jengyoyoAo wants to merge 2 commits into
Conversation
Existing cases let an incorrect final-hop subtraction pass. Add cases that reject it while preserving correct solutions. Close sysprog21#797 Change-Id: I7804cb50bc1eceab8678fb93d2d608eae9ddc682
| } | ||
| }); | ||
|
|
||
| test("jump-game-ii cases reject subtracting a final hop unconditionally", () => { |
There was a problem hiding this comment.
This repeats the shape of jump-game cases reject each hop-by-hop shortcut (look each label up, assert.ok(pinned), compare the wrong answer with expected), and the open judge-gap PRs repeat either it (#779) or the anagram test's any-case form (#777, #761). As a follow-up beyond this PR, one helper and one table would make the next judge fix a data-only change: a row per shortcut, checked against every pinned case, or against any case when the row names no label. Run over the anagram, jump-game and this test's shortcuts at this head it passes all 8, and it fails exactly where these tests do when the three overshooting cases are removed, when one of them is changed to an input the shortcut gets right, or when the anagram's expected: false cases are dropped.
import { isDeepStrictEqual } from "node:util";
const assertCasesReject = (id, name, wrong, labels) => {
const { cases } = judges[id];
const caught = ({ input, expected }) =>
!isDeepStrictEqual(wrong(...input), expected);
if (!labels) {
assert.ok(cases.some(caught), `the built-in ${id} cases must reject ${name}`);
return;
}
for (const label of labels) {
const pinned = cases.find((row) => row.label === label);
assert.ok(pinned, `the built-in cases must keep "${label}"`);
assert.ok(caught(pinned), `"${label}" must reject ${name}`);
}
};
for (const [id, name, wrong, labels] of SHORTCUTS) {
test(`${id} cases reject ${name}`, () => assertCasesReject(id, name, wrong, labels));
}A new shortcut still needs its wrong solution written, but then the row is all it needs.
There was a problem hiding this comment.
Thanks! A shared helper and table would make these tests easier to maintain. I'll leave that for a follow-up and keep this PR focused on #797.
| "isomorphic-strings": "65993b8137658baf9c57dd17847f3edc30d51481e3194e05940ce91e2e8ff26b", | ||
| "jump-game": "a01ddbb3aea588dc8fa943d5c4655459d7502b7477eae92d6cb025aba249cb80", | ||
| "jump-game-ii": "e08cc621d3a69f16d95b48bd1f8c6212b33638cc8d48a69c79abac1aa5d3c44a", | ||
| "jump-game-ii": "ffa7555442cd9130d1694fd58031b279308860cdea7b82e3f8b648641c62d2c0", |
There was a problem hiding this comment.
Every judge fix edits this digest by hand, and unlike tests/golden/task-prompts.json (UPDATE_PROMPT_GOLDEN=1) and the problem golden (UPDATE_PROBLEM_GOLDEN=1), docs/adding-a-problem.md gives no way to refresh it or says it needs one. As a follow-up beyond this PR, an UPDATE_POSED_GOLDEN=1 switch on test_posed_bank_golden_matches_python_judges_starters_and_notes in tests/test_task_package.py could write it with the posed and compact the test already uses, which reproduce this value (ffa75554…) from the bank at this head. every_public_judge_and_starter_matches_the_instructor_preview in tests/unit/tasks/exercise.rs would still check the Rust side against the written file, and the change would still show in the diff for review.
There was a problem hiding this comment.
Thanks for the suggestion. I'll leave this as a follow-up and keep this PR focused on #797.
Reject incorrect expected values as well as the reported shortcut. Check each pinned case with a correct greedy solution. Change-Id: Ic0cf849942c7d6ab30d2a1b57c4d88957f844544
|
Some review comments on this pull request are not settled yet. Please refine the change per the review comments listed below, reply on each thread with what you changed or why you did not, and mark each thread resolved yourself once it is settled (Resolve conversation, at the bottom of the thread). The pull request stays blocked while review threads are open. Not addressed at the current head:
Checked at 13ea6cb. |
chenyu50
left a comment
There was a problem hiding this comment.
Reviewed d7ea5de on macOS arm64 with Node v26.13.0.
Focused validation:
$ node --test --test-name-pattern='jump-game-ii' tests/browser/compiler-explorer.test.js
1 passed, 0 failed, 0 skipped.
$ python3 scripts/gen-problems.py --check
Passed.
The reported final-hop subtraction is already rejected at this head. The two inline comments suggest additional test coverage:
- A solution that rejects zero-valued destinations when
nums.length > 1still passes all eight built-in cases. Changing the two-pad case to[2,0], with expected result1, rejects that shortcut while preserving the final-hop subtraction check. - With the existing correct-solution assertion retained, replacing the third case with
[4,1,1,1]andexpected: 1leaves the current regression passing. The proposed first-hop-only assertion passes with the original case and fails after that replacement, preserving coverage of overshooting after the first hop.
| "input": [ | ||
| [ | ||
| 2, | ||
| 1 |
There was a problem hiding this comment.
As a small extension of this case, all eight cases still accept an implementation that returns -1 when nums.length > 1 && nums.at(-1) === 0, then otherwise uses the correct greedy algorithm.
Replacing this 1 with 0 gives [2,0] -> 1. It still rejects the reported final-hop subtraction and also checks that reaching a zero-valued destination is valid.
| 1 | |
| 0 |
| for (const label of [ | ||
| "two pads with room to spare", | ||
| "three pads with room to spare", | ||
| "range grows after the first pad", |
There was a problem hiding this comment.
With the existing correct-solution assertion in place, keeping this label but replacing its case with [4,1,1,1] and expected: 1 still leaves the regression green and removes coverage of overshooting after the first hop.
The variant (nums) => nums.length === 1 ? 0 : nums[0] >= nums.length - 1 ? 1 : reported(nums) passes the other seven cases but fails the current [1,3,1,1] case.
Check that named variant against this pinned case as well, so the later-hop coverage cannot be replaced by another first-hop case.
One concrete approach is to add this after byLabel is built, before the existing loop:
const firstHopOnly = (nums) =>
nums.length === 1
? 0
: nums[0] >= nums.length - 1
? 1
: reported(nums);
const later = byLabel["range grows after the first pad"];
assert.ok(later);
assert.notEqual(
firstHopOnly(later.input[0]),
later.expected,
"must reject a first-hop-only repair",
);This passes with the current [1,3,1,1] case. Replacing that case with [4,1,1,1] and expected: 1 makes this assertion fail, preserving the later-hop coverage.
Problem and behavior
The existing Drone Pad Hopping cases let a solution that always subtracts one final hop pass. For example, it returns 0 for
[2,1], where the answer is 1.Approach
Add three cases where the reachable range extends past the final pad, along with a regression test that checks each case rejects the reported solution. Regenerate the judge file and update the contract pins and golden digest.
Validation
node --test tests/browser/compiler-explorer.test.js: 108 passed, 0 failed, 0 skipped.python3 scripts/gen-problems.py --check: passed../scripts/test.sh: completed in an ARM64 Linux VM with exit code 0 and no failed gates. Used ShellCheck 0.9.0 to match the Ubuntu 24.04 CI package.Chromium/ffmpeg-dependent tests, cargo-audit, cargo-deny, and actionlint were skipped because the required tools were unavailable. The x86_64-only Clang-fetch test was also skipped.
Close #797
Summary by cubic
Closes a gap in the
jump-game-iijudge that let incorrect solutions pass by always subtracting a final hop, such as returning 0 for[2, 1]when the answer is 1.Written for commit d7ea5de. Summary will update on new commits.