Skip to content

Confirm benchmark verdicts and run A/A nightly - #543

Open
xalestar wants to merge 5 commits into
sysprog21:mainfrom
xalestar:bench-aa
Open

xalestar wants to merge 5 commits into
sysprog21:mainfrom
xalestar:bench-aa

Conversation

@xalestar

@xalestar xalestar commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Intent: Report a slower or faster benchmark verdict only when a second set of pairs gives it again, let a reviewer measure a pull request again by adding the bench label, and measure main on a schedule: nightly A/A, and weekly against the main of 4 weeks earlier. This is PR 3 of #503, without the two group thresholds.

This PR:

  • tests/bench-suite.sh takes BENCH_METRICS, the metrics to run.
  • scripts/bench-paired.sh measures 6 more pairs of the flagged metrics and adds them to the report; bench-compare.py and bench-report.py keep a verdict only when that second set gives it again; otherwise it reads not confirmed, or lost on head when the second set's head lost a value, so a bench-compare.py --paired run without a second set gives no slower or faster. A metric is flagged when it reads slower or faster at BENCH_FLAG_THRESHOLD: 0 for a pull_request run, which from a fork may not get the repository variable (the current GitHub docs on variables do not say), for an A/A run, whose second set is threshold data, and for the weekly run; report, the threshold of the verdicts, for the other runs, so while the repository variable is unset they measure no second set. The second set is best effort: when a round of it or its comparison fails, or it stops before its first pair, the report says so and Tier 3 still runs; a run that posts no comment shows that only in its job summary.
  • The second set is cut at BENCH_CONFIRM_DEADLINE (1080 s; docs/testing.md has the exact rule), so Tier 3 still fits in the 25 minute step. On the first CI run that left room only for pairs of the flagged metrics under about 51 s each, calibration readings included; when a pair takes longer the set is cut, and a metric that reads slower at the configured threshold reads not confirmed.
  • benchmark.yml adds the bench label trigger and the two schedule lines. Two choices differ from Benchmark each PR against its base commit and chart main on elfuse-bench #503: the weekly base comes from git rev-list -1 --first-parent --before='4 weeks ago', because without --first-parent it can be a commit of a branch merged later, and schedule runs keep their artifacts 90 days instead of 14, so the A/A data is still there when the thresholds are set from it. bench-report.yml comments on a labeled pull request even when the report is clean, then takes the label off (also when it leaves the comment to a later merge ref's report), so adding it again measures again, and it ignores a run whose jobs were all skipped.

If the two sets were independent, an unchanged metric would pass both about 0.05% of the time (2 x (1/64)^2), not the 0.1% #503 gives; the nightly A/A data will show the real rate.

Left out: the two group thresholds (they need A/A data, so BENCH_REGRESSION_THRESHOLD stays unset), the runs against a deliberately slowed branch, and creating the bench label.

Testing:

  • make check-bench-report passes under Python 3 and 3.9 at each commit; make check-format and scripts/check-commit-log.sh pass.
  • Each new self-test and stub-test case fails when the code it covers is broken.
  • End to end, a head 50 ms slower at startup read slower in both sets (5.06x, then 5.11x).

Not tested: No test runs three BENCH_METRICS paths of tests/bench-suite.sh, because they need an elfuse build: a tier left out and the lat_fs drop were checked on local runs, and ERR in place of the no-results stop was not run. On GitHub Actions only the opened path has run (run 44875161536, nothing flagged), and on an earlier revision (939253c), so the scope step's choice of BENCH_FLAG_THRESHOLD and the best-effort second set have not run there. The bench label does not exist yet, the schedule lines run only on main, and bench-report.yml runs from main, so those changes, the label removal among them, are checked by reading. The second set's timing on CI.

Refs #503

cubic-dev-ai[bot]

This comment was marked as resolved.

@xalestar
xalestar force-pushed the bench-aa branch 2 times, most recently from ecf6ddc to 5588c93 Compare November 21, 2026 08:50
cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread scripts/bench-compare.py Resolved
Comment thread .github/workflows/bench-report.yml Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread .github/workflows/bench-report.yml Resolved
Comment thread .github/workflows/bench-report.yml Resolved
Comment thread scripts/bench-paired.sh Resolved
Comment thread scripts/bench-paired.sh Resolved
A slower or faster verdict is to be measured again before it is listed,
and a whole round for one metric costs about 40 s on the CI host.
BENCH_METRICS names the metrics to run, as the comparison flattens them.
A tier with none of them is left out the way BENCH_APPS=0 leaves out
Tier 2, without the skipped mark that records a missing prerequisite,
and an unknown name exits 2 instead of giving results without it. One
lat_fs run times both create and delete, so the one not named is dropped
from the results.

With a subset, a Tier-1 metric that fails every attempt is recorded as
ERR instead of stopping the run: one named metric failing is a result,
not a broken environment.

tests/test-bench-paired.sh runs the suite as far as the name check,
which needs no elfuse build: an unknown name exits 2, and known names of
all three tiers pass it. Checked locally: a run of lat_syscall.null and
lat_fs.create wrote those two and no skipped mark in 11 s; zstd_ms and
startup_ms on a staged sysroot wrote those two; unknown names exited 2;
a full round without BENCH_METRICS wrote every Tier-1 and Tier-2 metric.

Refs sysprog21#503
At 6 pairs, an interval excludes 1 by chance in 3.1% of runs per metric,
so in a run of 20 unchanged metrics some row reads slower or faster
about 47% of the time before the threshold filters it.

bench-compare.py now takes a second set of rounds (--confirm-base,
--confirm-head) and keeps a slower or faster verdict only when that set
gives the same one; otherwise the metric reads "not confirmed", which
posts no comment. A metric whose second-set head lost a value reads
"lost on head" whatever the first set gave, so a head failure in the
second set posts a comment even with no threshold set. The schema 3
report carries the second set under an optional "confirm" key, because
bench-report.py recomputes every verdict from the report; the reporter
on main reads only rounds and metrics, so it keeps rendering such a
report, and a report without the key renders here with each slower or
faster verdict not confirmed.

--flagged writes the metrics that read slower or faster at
--flag-threshold, for the runner to measure again: a fraction, none for
no metric, or report for the threshold of the verdicts, so that a
workflow can pass BENCH_REGRESSION_THRESHOLD on to one parser. At the
default of 0 the list holds every verdict any threshold gives. A report
without a second set can say why (confirm_missing: not flagged, failed,
or out of time), and a slower or faster metric then reads not confirmed
with that reason, in the comment and in the bench-compare.py table. A
failed second set prints changes, with or without a threshold, so that a
head build that broke there is reported.

The self-test covers the second set agreeing in both directions,
disagreeing, missing, with too few pairs and losing a head value; each
way a confirm block can be malformed, one at a time; the block read
through the command line; a 2% slowdown and a 2% speedup flagged with no
threshold, and a head ERR in their second set read as lost on head; and
the refusals of bench-compare.py, more than 32 second-set rounds among
them. For the flag threshold, 10%, none and report with no threshold
flag neither change, and 1% and report at a threshold of .01 flag both.
For the reasons, it covers a failed second set listed with no threshold,
each reason in the rows and the rule of the comment, a reason in the
bench-compare.py table, and a reason next to a second set refused. Three
mutations of the rule, four of the flag threshold and six of the reasons
each fail it.

Refs sysprog21#503
scripts/bench-paired.sh now writes the first report and the flagged
metrics, those that read slower or faster at BENCH_FLAG_THRESHOLD
(default 0), then measures BENCH_PAIRS more pairs of only those metrics
(BENCH_METRICS) and writes the report again with that set, which a
slower or faster verdict has to agree with. The second set is best
effort: when a round or the comparison of it fails, or it stops before
its first pair, the report says so (failed or out of time) and the
script goes on to Tier 3, so a head that broke there shows in the report
instead of in a failed job. It runs as a subshell of its own, because
set -e is off inside a function called after || or in an if. A set cut
later is written with the pairs it has, and below 6 pairs it confirms no
verdict.

benchmark.yml sets BENCH_FLAG_THRESHOLD to 0 for a pull_request run,
which from a fork may not get the repository variable, and for an aa
dispatch, whose second set is data for the thresholds. Other runs get
report, the threshold of the verdicts, so while the variable is unset
they skip up to about 306 s on the exclusive host lock and give up the
one change such a second set could make: a head that loses a value there
reads lost on head. The word report stands in for the value because
bench-paired.sh checks a number more strictly than the reporter does and
would refuse .05.

No pair of the second set starts at or past BENCH_CONFIRM_DEADLINE
(1080 s), nor one that, with the pairs after it, each as long as the
last one, would end there; pair 1 has no earlier pair to estimate from,
so it can end past the deadline by its own length. The first CI run
(44836576102) took 12.9 min for 14 rounds and 3.7 min for Tier 3, so a
second set ending by 18 min leaves Tier 3 about 3 min of margin in the
25 minute step, and about 1.5 min after a pair 1 that ends late. After a
first set that long, the second set has about 306 s: 6 pairs fit while a
pair, calibration readings and image attaches included, takes under
51 s.

The stub test covers a flagged metric measured again alone, a run into
the same directory that keeps none of its second-set files,
BENCH_FLAG_THRESHOLD at 50%, at none and at report with a threshold of
.05, a second set stopped before its first pair and after one pair, a
rerun in the second set held to its own deadline, and a failing
second-set round that ends the set there, is recorded, and still lets
Tier 3 run; it takes about 48 s instead of about 24 s. Dropping the
projected end from the cut fails it, and so do holding that rerun to the
first set's deadline, running the set after ||, dropping its own detach,
and not passing the flag threshold. A head wrapper sleeping 50 ms read
applications.startup_ms 5.06x slower, its second set 5.11x, and the
verdict stayed slower.

Refs sysprog21#503
A pull request is measured once, when it is opened or marked ready, and
review can change it after that run. Adding the bench label to an open
pull request, draft or not, now measures its merge ref again whatever
the change touches. bench-report.yml takes the label off once it has
written the comment, or left it to a later merge ref's report (its
pull-requests: write token allows that), so adding it again repeats the
run, and a later run without it updates that comment. Only users with
triage access can add a label.

A run for another label measures nothing, but in the per-commit
concurrency group it would replace a bench run still pending there, so
it gets a group of its own. Its jobs are all skipped, which gives the
run the conclusion "skipped" (as bench-report.yml's own skipped run on
main shows), which bench-report.yml would take for a failed run with no
report and rewrite an earlier comment to say so. It now leaves the
comment alone, as for a draft, and gives the report job of such a run a
concurrency group of its own, so that it cannot replace the pending
report job of a real run. While the pull request holds the label,
bench-report.yml also creates the comment, when there is none yet, for a
clean report or for a run that failed or was cancelled without one,
because the person who added it asked for a result and the workflow_run
payload does not say which action started the run.

Taking the label off and adding it again while one run is in progress
and another is pending replaces the pending one, whose report job then
writes that the benchmark did not complete before the run in progress
reports. A cancelled run's comment therefore no longer carries the ref
time, so the report of the earlier merge ref, which comes later, still
replaces it. A failed run's comment keeps it, so the report of an
earlier merge ref cannot replace the failure of a later one.

Refs sysprog21#503
The thresholds are to come from A/A runs on the CI host, and only a
workflow_dispatch with aa makes one today. A nightly schedule run now
measures the main tip on both sides, so every night adds the ratios and
intervals of an unchanged build to that data, and a change of the host
shows up in them.

A merge run compares one merge, so ten merges of +2% each all pass and
add up to +22%. A weekly schedule run measures the main tip against the
last main tip from 4 weeks earlier, found with git rev-list
--first-parent so that it is never a commit of a branch merged later.

Both run Tier 1 and Tier 2 without the path filter and post no comment,
and both flag at a zero threshold, so the nightly artifacts also show
how often a second set agrees by chance, and the weekly ones hold a
second set for the metrics they flag. Their artifacts are kept 90 days
instead of 14, so the A/A data is still there when the thresholds are
set from it. The scope step, run locally on upstream/main, picked the
tip on both sides for the nightly line, e9fdbbb (2026-10-22) for the
weekly one, and failed on an unknown line.

Refs sysprog21#503

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/bench-compare.py">

<violation number="1" location="scripts/bench-compare.py:260">
P2: `missing_reason` labels any metric absent from the second set as `not flagged`, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.</violation>
</file>

<file name=".github/workflows/benchmark.yml">

<violation number="1" location=".github/workflows/benchmark.yml:221">
P2: Pull-request runs force a zero flag threshold even when `BENCH_REGRESSION_THRESHOLD` is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use `report` when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread scripts/bench-compare.py
) -> str | None:
"""Return why the metric has no second set, or None when the reason is
not recorded: a second set without it means it was not flagged."""
return "not flagged" if confirm and name not in confirm else missing

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: missing_reason labels any metric absent from the second set as not flagged, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/bench-compare.py, line 260:

<comment>`missing_reason` labels any metric absent from the second set as `not flagged`, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.</comment>

<file context>
@@ -246,6 +253,20 @@ def env_threshold(*, blank_is_unset: bool) -> float | None:
+                   ) -> str | None:
+    """Return why the metric has no second set, or None when the reason is
+    not recorded: a second set without it means it was not flagged."""
+    return "not flagged" if confirm and name not in confirm else missing
+
+
</file context>

# BENCH_REGRESSION_THRESHOLD; bench-paired.sh checks a copied
# number more strictly and would refuse .05.
case "$EVENT/$aa" in
pull_request/* | schedule/* | */true) flag_threshold=0 ;;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Pull-request runs force a zero flag threshold even when BENCH_REGRESSION_THRESHOLD is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use report when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/benchmark.yml, line 221:

<comment>Pull-request runs force a zero flag threshold even when `BENCH_REGRESSION_THRESHOLD` is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use `report` when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.</comment>

<file context>
@@ -208,7 +209,19 @@ jobs:
+          # BENCH_REGRESSION_THRESHOLD; bench-paired.sh checks a copied
+          # number more strictly and would refuse .05.
+          case "$EVENT/$aa" in
+            pull_request/* | schedule/* | */true) flag_threshold=0 ;;
+            *) flag_threshold=report ;;
+          esac
</file context>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants