Skip to content

Credit only the restatement a candidate stated - #700

Open
KentChen1112 wants to merge 2 commits into
sysprog21:mainfrom
KentChen1112:report-restatement-credit
Open

KentChen1112 wants to merge 2 commits into
sysprog21:mainfrom
KentChen1112:report-restatement-credit

Conversation

@KentChen1112

@KentChen1112 KentChen1112 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

A report scored Repeat 90/100 and listed "You clearly restated the problem requirements" as a communication strength after the candidate misstated the output as the maximum price and never said that the sale must come after the purchase (reproduced on cfcd576, details in #613). The report brief carries the scenario contract, which requires i < j, but the scoring rubrics asked only whether the candidate restated the problem, and the interviewer's "That's right" right after the restatement was a second source of unearned credit. After this change a restatement is judged as it stands when the candidate first moves on from it to any other step, counting what they stated in their own words, wrote as a comment in the code or on the board, corrected themselves, or settled with their own question by then; a misstated input, output or rule, a left-out input or output, or a relevant rule the scenario posed on screen that it still lacks is a gap, not a strength; and the interviewer's confirmation or rephrasing earns no credit.

Approach

A bounded report-prompt change following #506: one grounding rule placed beside the rule #506 added, and "accurately restated" in both the editor and whiteboard scoring rubrics. The contract is the private specification the interviewer never reads out, so a rule only the contract holds, such as Single Resale Window's zero-profit case, is not a gap; i < j still is, because the screen says "selling on a later day". One timing boundary decides what the restatement holds: the moment the candidate first moves on from it to any other step. Before then, a question, a self-correction or a written comment counts, so asking or correcting during Repeat is never penalized; after then, nothing does, so a rule found by asking while coding, by fixing a same-day bug, or by restating late does not erase the gap, and nothing is inferred from the code or the board. A new {written} value renders "as a comment in the code that a rolling-assessment note records for Repeat" at an editor and "on the board" with the condition that the Repeat checkpoint shows it at a whiteboard, since neither the final code nor a later board image shows when something was written. So that the editor note can exist, a second commit keeps Repeat open to the phase judge while it is the latest recorded step, for a comment that adds to the restatement, and closes it once another step is recorded (live prompt 37). A gap is tied to rubric 1's anchors: it puts Repeat below 90-100, one left-out rule with the rest right is a minor gap (75-89), and a left-out input or output, or more than one left-out rule, is a material gap (60-74). A restatement that only leaves parts out is still a correct partial one and completes the step, as the phase judge holds, so it does not reach 0-39; a misstated input, output or rule is placed by the anchors, which reserve 0-39 for directly observed incorrect work. The phase judge reads an editor comment but at a whiteboard only the transcript, so a rule written only on the board is credited by the report and not seen by the phase judge. This is report prompt 27, live prompt 37 and bundle 49; rubric 1 and report schema 4 are unchanged, and the ledger and schema stay out of scope. The regression test lives in a new tests/agent/report_grounding.rs because tests/agent/prompts.rs is at its line ceiling.

Validation

cargo test --test agent passes (321) on this tree, including the new contract_is_not_credited_as_candidate_restatement: its first assertions are preconditions showing the brief carries the contract, the on-screen "selling on a later day" and the interviewer's confirmation, and its clause loop is the regression check that both surfaces' rules carry each clause with {written} and {material} filled in. The rule was checked by reading against a table of candidate behaviors: complete restatement, a contract-only rule left out, the on-screen rule left out, a question during Repeat, a question after moving on or after skipping to code, j = i + 1 in the final code, a self-correction during Repeat, an uncorrected misstatement, an input or output settled by a question, and a late restatement after a failed run. The test shows the rule reaches every report reviewer; it does not show that the model heeds it. One model run at report prompt 27 on the first version of this rule, before the review changes, with the same scripted restatement, scored Repeat 70 against 90 at bundle 44, and the communication strength narrowed from "clearly restated the problem requirements" to "clearly restated the goal", but the improvements still did not name the gap, and Example also fell from 85 to 70, so one pair of runs cannot separate the rule's effect from run-to-run variance; the rule as it now stands has not been run against the model. Known limit: a question lost to speech recognition, with only Jim's answer in the transcript, may still read as a gap. ./scripts/test.sh rust passes on this tree, which the pre-push gate receipt requires; it skipped desktop clippy and tests (no libwebkit2gtk) and cargo-deny. The full gate's review-memory-tests and shell-syntax still fail locally, identically on a clean main in files this change does not touch: the local GitHub CLI 2.46.0 lacks attestation flags, and the local shellcheck 0.11.0 reports SC2319 at scripts/git-post-commit.sh:13; both passed in CI on the first head. Not checked locally: one browser test that needs ffmpeg and actionlint.

Close #613

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as outdated.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 31e4744 to 82d59fa Compare November 14, 2026 15:16
@KentChen1112

Copy link
Copy Markdown
Contributor Author

Rebased onto main at 55805ca. Upstream's cd24ac8 took bundle 47 for live prompt 35, so this change is now bundle 48 with live prompt 35 and report prompt 27; the bundle 47 row is kept exactly as upstream wrote it. All four review comments are addressed in 82d59fa.

@jserv
jserv dismissed their stale review November 14, 2026 15:25

No merge conflicts at the current head.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep Git commit messages concise. Omit unnecessary details such as “This is report prompt 27 and bundle 48; live prompt 35, rubric 1, and report schema 4 are unchanged.”

jserv

This comment was marked as resolved.

Urdan0117

This comment was marked as resolved.

jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch 2 times, most recently from 95001bf to b766431 Compare November 16, 2026 14:27
@KentChen1112

Copy link
Copy Markdown
Contributor Author

Shortened the commit message to two paragraphs: the problem, and what the rule now credits and counts as a gap. The version numbers, band values and test location are dropped from it; they remain in the description and docs/interview-contract-versions.md. Amended in b766431; the code is unchanged from 95001bf.

jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from b766431 to 6cd5993 Compare November 17, 2026 12:09
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 6cd5993 to ab0026d Compare November 17, 2026 12:39
jserv

This comment was marked as resolved.

jserv

This comment was marked as outdated.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from ab0026d to 9e1c025 Compare November 18, 2026 14:29
@KentChen1112

Copy link
Copy Markdown
Contributor Author

Rebased onto main at fbc8bff. Upstream's #707 took bundle 48 for live prompt 36, so this change is now bundle 49 with live prompt 36 and report prompt 27; the bundle 48 row is kept exactly as upstream wrote it. The open thread on the clarification clause is addressed in 9e1c025.

@jserv
jserv dismissed their stale review November 18, 2026 14:35

No merge conflicts at the current head.

jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 9e1c025 to 18aa506 Compare November 18, 2026 15:13
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 18aa506 to 46d2170 Compare November 19, 2026 15:25
jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 46d2170 to f7bb8b3 Compare November 21, 2026 09:03
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from f7bb8b3 to d6f8174 Compare November 21, 2026 09:38
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from d6f8174 to 6637bdd Compare November 21, 2026 10:12
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 6637bdd to 0e1fa75 Compare November 21, 2026 10:55
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 0e1fa75 to fcfb84f Compare November 21, 2026 11:20
jserv

This comment was marked as resolved.

jserv

This comment was marked as outdated.

A report scored Repeat 90/100 and praised a restatement that misstated
the output and never said the sale must follow the purchase. The brief
gives the report model the private contract, nothing kept it apart from
what the candidate said, and the interviewer's "That's right" added
unearned credit.

A restatement is now judged as it stands when the candidate first moves
on from it, counting what they said, wrote, corrected or settled with
their own question by then, never what the code or the board implies.
A comment counts only when a repeat phase row quotes it, and writing on
the board only when the Repeat checkpoint image shows it; writing added
after that image is a known limit and earns no credit. A misstatement,
a left-out input or output, or a relevant rule posed on screen that it
still lacks is a gap tied to the rubric's bands; a rule only the
contract or the constraints hold is not. The interviewer's confirmation
earns no credit.

Change-Id: Ib6bcdfb53dbd448c9d6d2ab3c0adfe6c7acc41c0
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from fcfb84f to c354c96 Compare November 21, 2026 13:31
jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from c354c96 to b32b05b Compare November 21, 2026 14:09
@KentChen1112

Copy link
Copy Markdown
Contributor Author

Reworked as requested: rebased onto main (no conflicts), ran /codetrial-review and resolved its findings, killed the surviving mutants with new assertions, and squashed the series into two commits: 7411c63 (report grounding) and b32b05b (Repeat reopening). The large-change review for b32b05b is recorded. Earlier "Fixed in" hashes now map to these two commits.

jserv

This comment was marked as resolved.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from b32b05b to 9ca3282 Compare November 21, 2026 14:49
Comment thread src/agent/phase_judge.rs Outdated

// Then only a comment adds to it: speech is the restatement already
// recorded, and a quote a Repeat row already holds adds nothing.
if again && (source != EvidenceSource::EditorSnapshot || restatement_quoted(state, quote)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dates only a Repeat that is already recorded. While Repeat is unrecorded, again is false, so a comment typed after Example or Algorithm was recorded in an earlier answer still becomes the first repeat row: a candidate who goes straight to an example and adds # sell after buy once a same-day bug shows up while coding gets that rule quoted for Repeat, and the editor rule then credits it as written before they moved on, since it asks only that a repeat row quote the comment. Refuse an EditorSnapshot quote for Repeat whenever a step other than Repeat is already recorded, not only when Repeat is, and add a case beside a_comment_typed_after_the_next_step_does_not_reopen_repeat that records Example with no Repeat and expects the comment to record nothing. A late spoken restatement can still be recorded, and the report rule already gives it no credit for what the step before it showed.

Comment thread src/agent/phase_judge.rs Outdated
.filter(|item| item.phase == FrameworkPhase::Repeat)
.map(|item| {
item.summary
.rsplit_once("Quote: ")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A candidate comment can itself contain Quote: , for example # Quote: the sale must follow the buy day. When the judge quotes that whole comment again, rsplit_once starts after the marker inside the candidate text, so restatement_quoted misses the earlier row. If the model gives the same quote a different summary, record_evidence adds it again instead of deduplicating it. Keep the quote separately or encode the boundary unambiguously, and add a repeated-quote test containing the marker.

@ChenYuHsuan0719 ChenYuHsuan0719 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked at 9ca3282a with a probe test added to tests/agent/restatement_comments.rs, using the Single Resale Window Python starter as the template and a recorded spoken restatement:

  • as you go sell after buy: recorded as a second Repeat row, source editor_snapshot
  • sell after buy: refused
  • think out loud as you go: refused

"the final code,\nor the test account",
"as a comment in\n the code that a `repeat` phase row in the rolling assessment \
quotes\n (a pause-time note does not date a comment, and a comment under six \
words is\n not recorded on its own and counts only when a row quotes it with \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The six-word floor this sentence rests on can be met with the starter's own comment. quote_is_grounded refuses an editor quote only when the starter alone holds it, so with a short rule typed under the starter's comment (125 of the 151 Python starters have that line):

        # Think out loud as you go!
        # sell after buy

the quote as you go sell after buy is recorded for Repeat, while sell after buy alone is refused. A short rule comment typed in the usual place is therefore recorded with words the candidate did not write, though this sentence and the bundle 49 row say it is not recorded on its own; either leave the starter's comment lines out before the comments are joined for an editor quote, or drop the claim here and in the row.

@jserv

jserv commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

Some review comments on this pull request are not settled yet. Please refine the change per the review comments listed below, reply on each thread with what you changed or why you did not, and mark each thread resolved yourself once it is settled (Resolve conversation, at the bottom of the thread). The pull request stays blocked while review threads are open.

Not addressed at the current head:

Checked at 9ca3282.

@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from 9ca3282 to da0cb66 Compare November 22, 2026 05:45
The report credits a rule written as an editor comment only when a
repeat phase row quotes it, but the phase judge skipped Repeat once it
was recorded, so a comment typed during Repeat after a correct
restatement never got that row. While Repeat is the only step recorded
at an editor, the judge may now record it again from a comment that
adds an input, output or rule, and its prompt says so only then.
A comment counts for Repeat, first or added, only while no other step
is recorded, this answer's included in any order, so one typed after an
example or an approach earns nothing; a late spoken restatement still
counts. At a whiteboard the Repeat checkpoint counts only when it is
the first checkpoint image, since a Repeat recorded late has its image
taken after the candidate moved on.

A comment already quoted is matched as the row stores it, cut to 110
characters, so a long one is not recorded twice. The row's quote is
found by the double quotes around it, one inside stored as a single
quote, since a comment or the judge's summary may also hold "Quote: ".
The judge's six-word floor applies to its quote, and the comment lines
are read joined with the starter's left out, so a rule comment under
six words counts only when a row quotes it with lines the candidate
typed around it; the editor rule says so.

Close sysprog21#613

Change-Id: I4ad384cf18cf4ad2f5e7dae604385820d9542747
@KentChen1112
KentChen1112 force-pushed the report-restatement-credit branch from da0cb66 to d19dce1 Compare November 22, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report credits an inaccurate problem restatement as a strength

4 participants