[ako]

What

The last two pieces of the agent dev-loop benchmark proposal: the trend chart and the alert.

A night's table compares two binaries on one runner. It says whether last night's commits made a step slower, and cannot show a step that got a few percent slower many nights running. Nothing read the records on the benchmarks branch at all.

• scripts/agentbench/trend reads every record and writes TREND.md and trend.json. Per session: a Mermaid chart of the session-weighted total over the last 30 runs, and a table with a row per step (latest, run before, median of the seven before, the night's A/B ratio, a sparkline of the last 14). The scale session also charts check-one and exec-batch per statement. GitHub renders the file in place on the branch, so nothing is hosted (Open Question 6).
• The bench job commits both files with the record, uploads them as an artifact, and opens or updates one issue labelled benchmark-regression when there is something to report. It comments only when the set of alerts changed, and never closes the issue.
• make bench-trend RECORDS=<dir> builds it locally.

The alerts

Alert	When
slower-two-runs	wall ratio against the night's baseline above 1.20 in the latest run and the one before
drift	median of the last 3 runs more than 1.30 × the median of the 7 before; needs 10 runs and a step of at least 100 ms
counts-increased	more unit files read, or more writes landed, than the baseline in the same run; first night
wrote-on-re-exec	a settled script changed a unit file

Drift compares different nights, so different machines. The two records that exist differ by about a quarter on every step from the runner alone (describe × 0.70 night to night, × 0.91 against its own baseline). So each run is first corrected by the median, over all steps, of its time against each step's own median. The limit of that: a change that slows every step alike looks like a slow machine. The nightly A/B is what catches that one.

The thresholds are flags and are provisional. The proposal's plan was to set them from two weeks of data; there are two records.

Verified

• Unit tests for each rule, each with a control in the same test (one slow night → nothing, two → alert; a slow runner → nothing, the same runner plus one regressed step → that step only). Three mutations of the rules each fail the test that covers them (runner correction disabled, the previous-night condition removed, > changed to != on counts).
• Ran the tool on the two real records from the benchmarks branch; rendered the generated Mermaid in Chromium to confirm the charts parse and the line is visible.
• Ran the workflow step's shell, extracted from the YAML, against a local bare remote, including a push that loses a race to another run: it refetches, rebuilds the trend with the other run's record in it, and pushes (4 runs in the resulting TREND.md).
• A workflow_dispatch of the bench job from this branch (run 43391783249) passed: the trend step ran on the runner, the agent-benchmark-trend artifact holds TREND.md, trend.json and the alerts file, the issue step was skipped, and the benchmarks branch is unchanged (still at e52592d4e).

Not verified

• The issue step (gh issue create/edit/comment, the issues: write permission) has only been syntax-checked. It runs on main only, so its first real run is after merge, and only when an alert fires.
• The push of TREND.md from the real runner; the same shell was run against a local remote only.

🤖 Generated with Claude Code

https://claude.ai/code/session_011oK82u4HRCKfX2BBvhgsNm