Repository navigation
Credit only the restatement a candidate stated - #700
KentChen1112 wants to merge 2 commits into
Conversation
31e4744 to
82d59fa
Compare
jserv
left a comment
There was a problem hiding this comment.
Keep Git commit messages concise. Omit unnecessary details such as “This is report prompt 27 and bundle 48; live prompt 35, rubric 1, and report schema 4 are unchanged.”
95001bf to
b766431
Compare
|
Shortened the commit message to two paragraphs: the problem, and what the rule now credits and counts as a gap. The version numbers, band values and test location are dropped from it; they remain in the description and docs/interview-contract-versions.md. Amended in b766431; the code is unchanged from 95001bf. |
b766431 to
6cd5993
Compare
6cd5993 to
ab0026d
Compare
ab0026d to
9e1c025
Compare
9e1c025 to
18aa506
Compare
18aa506 to
46d2170
Compare
46d2170 to
f7bb8b3
Compare
f7bb8b3 to
d6f8174
Compare
d6f8174 to
6637bdd
Compare
6637bdd to
0e1fa75
Compare
0e1fa75 to
fcfb84f
Compare
A report scored Repeat 90/100 and praised a restatement that misstated the output and never said the sale must follow the purchase. The brief gives the report model the private contract, nothing kept it apart from what the candidate said, and the interviewer's "That's right" added unearned credit. A restatement is now judged as it stands when the candidate first moves on from it, counting what they said, wrote, corrected or settled with their own question by then, never what the code or the board implies. A comment counts only when a repeat phase row quotes it, and writing on the board only when the Repeat checkpoint image shows it; writing added after that image is a known limit and earns no credit. A misstatement, a left-out input or output, or a relevant rule posed on screen that it still lacks is a gap tied to the rubric's bands; a rule only the contract or the constraints hold is not. The interviewer's confirmation earns no credit. Change-Id: Ib6bcdfb53dbd448c9d6d2ab3c0adfe6c7acc41c0
fcfb84f to
c354c96
Compare
c354c96 to
b32b05b
Compare
|
Reworked as requested: rebased onto main (no conflicts), ran /codetrial-review and resolved its findings, killed the surviving mutants with new assertions, and squashed the series into two commits: 7411c63 (report grounding) and b32b05b (Repeat reopening). The large-change review for b32b05b is recorded. Earlier "Fixed in" hashes now map to these two commits. |
b32b05b to
9ca3282
Compare
|
|
||
| // Then only a comment adds to it: speech is the restatement already | ||
| // recorded, and a quote a Repeat row already holds adds nothing. | ||
| if again && (source != EvidenceSource::EditorSnapshot || restatement_quoted(state, quote)) { |
There was a problem hiding this comment.
This dates only a Repeat that is already recorded. While Repeat is unrecorded, again is false, so a comment typed after Example or Algorithm was recorded in an earlier answer still becomes the first repeat row: a candidate who goes straight to an example and adds # sell after buy once a same-day bug shows up while coding gets that rule quoted for Repeat, and the editor rule then credits it as written before they moved on, since it asks only that a repeat row quote the comment. Refuse an EditorSnapshot quote for Repeat whenever a step other than Repeat is already recorded, not only when Repeat is, and add a case beside a_comment_typed_after_the_next_step_does_not_reopen_repeat that records Example with no Repeat and expects the comment to record nothing. A late spoken restatement can still be recorded, and the report rule already gives it no credit for what the step before it showed.
| .filter(|item| item.phase == FrameworkPhase::Repeat) | ||
| .map(|item| { | ||
| item.summary | ||
| .rsplit_once("Quote: ") |
There was a problem hiding this comment.
A candidate comment can itself contain Quote: , for example # Quote: the sale must follow the buy day. When the judge quotes that whole comment again, rsplit_once starts after the marker inside the candidate text, so restatement_quoted misses the earlier row. If the model gives the same quote a different summary, record_evidence adds it again instead of deduplicating it. Keep the quote separately or encode the boundary unambiguously, and add a repeated-quote test containing the marker.
ChenYuHsuan0719
left a comment
There was a problem hiding this comment.
Checked at 9ca3282a with a probe test added to tests/agent/restatement_comments.rs, using the Single Resale Window Python starter as the template and a recorded spoken restatement:
as you go sell after buy: recorded as a second Repeat row, sourceeditor_snapshotsell after buy: refusedthink out loud as you go: refused
| "the final code,\nor the test account", | ||
| "as a comment in\n the code that a `repeat` phase row in the rolling assessment \ | ||
| quotes\n (a pause-time note does not date a comment, and a comment under six \ | ||
| words is\n not recorded on its own and counts only when a row quotes it with \ |
There was a problem hiding this comment.
The six-word floor this sentence rests on can be met with the starter's own comment. quote_is_grounded refuses an editor quote only when the starter alone holds it, so with a short rule typed under the starter's comment (125 of the 151 Python starters have that line):
# Think out loud as you go!
# sell after buythe quote as you go sell after buy is recorded for Repeat, while sell after buy alone is refused. A short rule comment typed in the usual place is therefore recorded with words the candidate did not write, though this sentence and the bundle 49 row say it is not recorded on its own; either leave the starter's comment lines out before the comments are joined for an editor quote, or drop the claim here and in the row.
|
Some review comments on this pull request are not settled yet. Please refine the change per the review comments listed below, reply on each thread with what you changed or why you did not, and mark each thread resolved yourself once it is settled (Resolve conversation, at the bottom of the thread). The pull request stays blocked while review threads are open. Not addressed at the current head:
Checked at 9ca3282. |
9ca3282 to
da0cb66
Compare
The report credits a rule written as an editor comment only when a repeat phase row quotes it, but the phase judge skipped Repeat once it was recorded, so a comment typed during Repeat after a correct restatement never got that row. While Repeat is the only step recorded at an editor, the judge may now record it again from a comment that adds an input, output or rule, and its prompt says so only then. A comment counts for Repeat, first or added, only while no other step is recorded, this answer's included in any order, so one typed after an example or an approach earns nothing; a late spoken restatement still counts. At a whiteboard the Repeat checkpoint counts only when it is the first checkpoint image, since a Repeat recorded late has its image taken after the candidate moved on. A comment already quoted is matched as the row stores it, cut to 110 characters, so a long one is not recorded twice. The row's quote is found by the double quotes around it, one inside stored as a single quote, since a comment or the judge's summary may also hold "Quote: ". The judge's six-word floor applies to its quote, and the comment lines are read joined with the starter's left out, so a rule comment under six words counts only when a row quotes it with lines the candidate typed around it; the editor rule says so. Close sysprog21#613 Change-Id: I4ad384cf18cf4ad2f5e7dae604385820d9542747
da0cb66 to
d19dce1
Compare
Problem and behavior
A report scored Repeat 90/100 and listed "You clearly restated the problem requirements" as a communication strength after the candidate misstated the output as the maximum price and never said that the sale must come after the purchase (reproduced on cfcd576, details in #613). The report brief carries the scenario contract, which requires
i < j, but the scoring rubrics asked only whether the candidate restated the problem, and the interviewer's "That's right" right after the restatement was a second source of unearned credit. After this change a restatement is judged as it stands when the candidate first moves on from it to any other step, counting what they stated in their own words, wrote as a comment in the code or on the board, corrected themselves, or settled with their own question by then; a misstated input, output or rule, a left-out input or output, or a relevant rule the scenario posed on screen that it still lacks is a gap, not a strength; and the interviewer's confirmation or rephrasing earns no credit.Approach
A bounded report-prompt change following #506: one grounding rule placed beside the rule #506 added, and "accurately restated" in both the editor and whiteboard scoring rubrics. The contract is the private specification the interviewer never reads out, so a rule only the contract holds, such as Single Resale Window's zero-profit case, is not a gap;
i < jstill is, because the screen says "selling on a later day". One timing boundary decides what the restatement holds: the moment the candidate first moves on from it to any other step. Before then, a question, a self-correction or a written comment counts, so asking or correcting during Repeat is never penalized; after then, nothing does, so a rule found by asking while coding, by fixing a same-day bug, or by restating late does not erase the gap, and nothing is inferred from the code or the board. A new{written}value renders "as a comment in the code that a rolling-assessment note records for Repeat" at an editor and "on the board" with the condition that the Repeat checkpoint shows it at a whiteboard, since neither the final code nor a later board image shows when something was written. So that the editor note can exist, a second commit keeps Repeat open to the phase judge while it is the latest recorded step, for a comment that adds to the restatement, and closes it once another step is recorded (live prompt 37). A gap is tied to rubric 1's anchors: it puts Repeat below 90-100, one left-out rule with the rest right is a minor gap (75-89), and a left-out input or output, or more than one left-out rule, is a material gap (60-74). A restatement that only leaves parts out is still a correct partial one and completes the step, as the phase judge holds, so it does not reach 0-39; a misstated input, output or rule is placed by the anchors, which reserve 0-39 for directly observed incorrect work. The phase judge reads an editor comment but at a whiteboard only the transcript, so a rule written only on the board is credited by the report and not seen by the phase judge. This is report prompt 27, live prompt 37 and bundle 49; rubric 1 and report schema 4 are unchanged, and the ledger and schema stay out of scope. The regression test lives in a new tests/agent/report_grounding.rs because tests/agent/prompts.rs is at its line ceiling.Validation
cargo test --test agentpasses (321) on this tree, including the newcontract_is_not_credited_as_candidate_restatement: its first assertions are preconditions showing the brief carries the contract, the on-screen "selling on a later day" and the interviewer's confirmation, and its clause loop is the regression check that both surfaces' rules carry each clause with{written}and{material}filled in. The rule was checked by reading against a table of candidate behaviors: complete restatement, a contract-only rule left out, the on-screen rule left out, a question during Repeat, a question after moving on or after skipping to code,j = i + 1in the final code, a self-correction during Repeat, an uncorrected misstatement, an input or output settled by a question, and a late restatement after a failed run. The test shows the rule reaches every report reviewer; it does not show that the model heeds it. One model run at report prompt 27 on the first version of this rule, before the review changes, with the same scripted restatement, scored Repeat 70 against 90 at bundle 44, and the communication strength narrowed from "clearly restated the problem requirements" to "clearly restated the goal", but the improvements still did not name the gap, and Example also fell from 85 to 70, so one pair of runs cannot separate the rule's effect from run-to-run variance; the rule as it now stands has not been run against the model. Known limit: a question lost to speech recognition, with only Jim's answer in the transcript, may still read as a gap../scripts/test.sh rustpasses on this tree, which the pre-push gate receipt requires; it skipped desktop clippy and tests (no libwebkit2gtk) and cargo-deny. The full gate's review-memory-tests and shell-syntax still fail locally, identically on a clean main in files this change does not touch: the local GitHub CLI 2.46.0 lacks attestation flags, and the local shellcheck 0.11.0 reports SC2319 at scripts/git-post-commit.sh:13; both passed in CI on the first head. Not checked locally: one browser test that needs ffmpeg and actionlint.Close #613