Repository navigation
Conversation
ecf6ddc to
5588c93
Compare
2148bb2 to
c45a8ad
Compare
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
A slower or faster verdict is to be measured again before it is listed, and a whole round for one metric costs about 40 s on the CI host. BENCH_METRICS names the metrics to run, as the comparison flattens them. A tier with none of them is left out the way BENCH_APPS=0 leaves out Tier 2, without the skipped mark that records a missing prerequisite, and an unknown name exits 2 instead of giving results without it. One lat_fs run times both create and delete, so the one not named is dropped from the results. With a subset, a Tier-1 metric that fails every attempt is recorded as ERR instead of stopping the run: one named metric failing is a result, not a broken environment. tests/test-bench-paired.sh runs the suite as far as the name check, which needs no elfuse build: an unknown name exits 2, and known names of all three tiers pass it. Checked locally: a run of lat_syscall.null and lat_fs.create wrote those two and no skipped mark in 11 s; zstd_ms and startup_ms on a staged sysroot wrote those two; unknown names exited 2; a full round without BENCH_METRICS wrote every Tier-1 and Tier-2 metric. Refs sysprog21#503
At 6 pairs, an interval excludes 1 by chance in 3.1% of runs per metric, so in a run of 20 unchanged metrics some row reads slower or faster about 47% of the time before the threshold filters it. bench-compare.py now takes a second set of rounds (--confirm-base, --confirm-head) and keeps a slower or faster verdict only when that set gives the same one; otherwise the metric reads "not confirmed", which posts no comment. A metric whose second-set head lost a value reads "lost on head" whatever the first set gave, so a head failure in the second set posts a comment even with no threshold set. The schema 3 report carries the second set under an optional "confirm" key, because bench-report.py recomputes every verdict from the report; the reporter on main reads only rounds and metrics, so it keeps rendering such a report, and a report without the key renders here with each slower or faster verdict not confirmed. --flagged writes the metrics that read slower or faster at --flag-threshold, for the runner to measure again: a fraction, none for no metric, or report for the threshold of the verdicts, so that a workflow can pass BENCH_REGRESSION_THRESHOLD on to one parser. At the default of 0 the list holds every verdict any threshold gives. A report without a second set can say why (confirm_missing: not flagged, failed, or out of time), and a slower or faster metric then reads not confirmed with that reason, in the comment and in the bench-compare.py table. A failed second set prints changes, with or without a threshold, so that a head build that broke there is reported. The self-test covers the second set agreeing in both directions, disagreeing, missing, with too few pairs and losing a head value; each way a confirm block can be malformed, one at a time; the block read through the command line; a 2% slowdown and a 2% speedup flagged with no threshold, and a head ERR in their second set read as lost on head; and the refusals of bench-compare.py, more than 32 second-set rounds among them. For the flag threshold, 10%, none and report with no threshold flag neither change, and 1% and report at a threshold of .01 flag both. For the reasons, it covers a failed second set listed with no threshold, each reason in the rows and the rule of the comment, a reason in the bench-compare.py table, and a reason next to a second set refused. Three mutations of the rule, four of the flag threshold and six of the reasons each fail it. Refs sysprog21#503
scripts/bench-paired.sh now writes the first report and the flagged metrics, those that read slower or faster at BENCH_FLAG_THRESHOLD (default 0), then measures BENCH_PAIRS more pairs of only those metrics (BENCH_METRICS) and writes the report again with that set, which a slower or faster verdict has to agree with. The second set is best effort: when a round or the comparison of it fails, or it stops before its first pair, the report says so (failed or out of time) and the script goes on to Tier 3, so a head that broke there shows in the report instead of in a failed job. It runs as a subshell of its own, because set -e is off inside a function called after || or in an if. A set cut later is written with the pairs it has, and below 6 pairs it confirms no verdict. benchmark.yml sets BENCH_FLAG_THRESHOLD to 0 for a pull_request run, which from a fork may not get the repository variable, and for an aa dispatch, whose second set is data for the thresholds. Other runs get report, the threshold of the verdicts, so while the variable is unset they skip up to about 306 s on the exclusive host lock and give up the one change such a second set could make: a head that loses a value there reads lost on head. The word report stands in for the value because bench-paired.sh checks a number more strictly than the reporter does and would refuse .05. No pair of the second set starts at or past BENCH_CONFIRM_DEADLINE (1080 s), nor one that, with the pairs after it, each as long as the last one, would end there; pair 1 has no earlier pair to estimate from, so it can end past the deadline by its own length. The first CI run (44836576102) took 12.9 min for 14 rounds and 3.7 min for Tier 3, so a second set ending by 18 min leaves Tier 3 about 3 min of margin in the 25 minute step, and about 1.5 min after a pair 1 that ends late. After a first set that long, the second set has about 306 s: 6 pairs fit while a pair, calibration readings and image attaches included, takes under 51 s. The stub test covers a flagged metric measured again alone, a run into the same directory that keeps none of its second-set files, BENCH_FLAG_THRESHOLD at 50%, at none and at report with a threshold of .05, a second set stopped before its first pair and after one pair, a rerun in the second set held to its own deadline, and a failing second-set round that ends the set there, is recorded, and still lets Tier 3 run; it takes about 48 s instead of about 24 s. Dropping the projected end from the cut fails it, and so do holding that rerun to the first set's deadline, running the set after ||, dropping its own detach, and not passing the flag threshold. A head wrapper sleeping 50 ms read applications.startup_ms 5.06x slower, its second set 5.11x, and the verdict stayed slower. Refs sysprog21#503
A pull request is measured once, when it is opened or marked ready, and review can change it after that run. Adding the bench label to an open pull request, draft or not, now measures its merge ref again whatever the change touches. bench-report.yml takes the label off once it has written the comment, or left it to a later merge ref's report (its pull-requests: write token allows that), so adding it again repeats the run, and a later run without it updates that comment. Only users with triage access can add a label. A run for another label measures nothing, but in the per-commit concurrency group it would replace a bench run still pending there, so it gets a group of its own. Its jobs are all skipped, which gives the run the conclusion "skipped" (as bench-report.yml's own skipped run on main shows), which bench-report.yml would take for a failed run with no report and rewrite an earlier comment to say so. It now leaves the comment alone, as for a draft, and gives the report job of such a run a concurrency group of its own, so that it cannot replace the pending report job of a real run. While the pull request holds the label, bench-report.yml also creates the comment, when there is none yet, for a clean report or for a run that failed or was cancelled without one, because the person who added it asked for a result and the workflow_run payload does not say which action started the run. Taking the label off and adding it again while one run is in progress and another is pending replaces the pending one, whose report job then writes that the benchmark did not complete before the run in progress reports. A cancelled run's comment therefore no longer carries the ref time, so the report of the earlier merge ref, which comes later, still replaces it. A failed run's comment keeps it, so the report of an earlier merge ref cannot replace the failure of a later one. Refs sysprog21#503
The thresholds are to come from A/A runs on the CI host, and only a workflow_dispatch with aa makes one today. A nightly schedule run now measures the main tip on both sides, so every night adds the ratios and intervals of an unchanged build to that data, and a change of the host shows up in them. A merge run compares one merge, so ten merges of +2% each all pass and add up to +22%. A weekly schedule run measures the main tip against the last main tip from 4 weeks earlier, found with git rev-list --first-parent so that it is never a commit of a branch merged later. Both run Tier 1 and Tier 2 without the path filter and post no comment, and both flag at a zero threshold, so the nightly artifacts also show how often a second set agrees by chance, and the weekly ones hold a second set for the metrics they flag. Their artifacts are kept 90 days instead of 14, so the A/A data is still there when the thresholds are set from it. The scope step, run locally on upstream/main, picked the tip on both sides for the nightly line, e9fdbbb (2026-10-22) for the weekly one, and failed on an unknown line. Refs sysprog21#503
There was a problem hiding this comment.
2 issues found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/bench-compare.py">
<violation number="1" location="scripts/bench-compare.py:260">
P2: `missing_reason` labels any metric absent from the second set as `not flagged`, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.</violation>
</file>
<file name=".github/workflows/benchmark.yml">
<violation number="1" location=".github/workflows/benchmark.yml:221">
P2: Pull-request runs force a zero flag threshold even when `BENCH_REGRESSION_THRESHOLD` is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use `report` when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| ) -> str | None: | ||
| """Return why the metric has no second set, or None when the reason is | ||
| not recorded: a second set without it means it was not flagged.""" | ||
| return "not flagged" if confirm and name not in confirm else missing |
There was a problem hiding this comment.
P2: missing_reason labels any metric absent from the second set as not flagged, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/bench-compare.py, line 260:
<comment>`missing_reason` labels any metric absent from the second set as `not flagged`, even when that metric was flagged. Only a metric that was not flagged should get that label. Consider distinguishing the flagged-but-unmeasured case, for example with a separate reason.</comment>
<file context>
@@ -246,6 +253,20 @@ def env_threshold(*, blank_is_unset: bool) -> float | None:
+ ) -> str | None:
+ """Return why the metric has no second set, or None when the reason is
+ not recorded: a second set without it means it was not flagged."""
+ return "not flagged" if confirm and name not in confirm else missing
+
+
</file context>
| # BENCH_REGRESSION_THRESHOLD; bench-paired.sh checks a copied | ||
| # number more strictly and would refuse .05. | ||
| case "$EVENT/$aa" in | ||
| pull_request/* | schedule/* | */true) flag_threshold=0 ;; |
There was a problem hiding this comment.
P2: Pull-request runs force a zero flag threshold even when BENCH_REGRESSION_THRESHOLD is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use report when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/benchmark.yml, line 221:
<comment>Pull-request runs force a zero flag threshold even when `BENCH_REGRESSION_THRESHOLD` is unset, so ordinary interval noise launches a second set that cannot confirm a slower/faster verdict. This can spend roughly the remaining 306-second confirmation budget under the exclusive host lock; use `report` when no verdict threshold is configured, while retaining zero-threshold runs for A/A data collection.</comment>
<file context>
@@ -208,7 +209,19 @@ jobs:
+ # BENCH_REGRESSION_THRESHOLD; bench-paired.sh checks a copied
+ # number more strictly and would refuse .05.
+ case "$EVENT/$aa" in
+ pull_request/* | schedule/* | */true) flag_threshold=0 ;;
+ *) flag_threshold=report ;;
+ esac
</file context>
Intent: Report a
slowerorfasterbenchmark verdict only when a second set of pairs gives it again, let a reviewer measure a pull request again by adding thebenchlabel, and measuremainon a schedule: nightly A/A, and weekly against themainof 4 weeks earlier. This is PR 3 of #503, without the two group thresholds.This PR:
tests/bench-suite.shtakesBENCH_METRICS, the metrics to run.scripts/bench-paired.shmeasures 6 more pairs of the flagged metrics and adds them to the report;bench-compare.pyandbench-report.pykeep a verdict only when that second set gives it again; otherwise it readsnot confirmed, orlost on headwhen the second set's head lost a value, so abench-compare.py --pairedrun without a second set gives noslowerorfaster. A metric is flagged when it readsslowerorfasteratBENCH_FLAG_THRESHOLD: 0 for apull_requestrun, which from a fork may not get the repository variable (the current GitHub docs on variables do not say), for an A/A run, whose second set is threshold data, and for the weekly run;report, the threshold of the verdicts, for the other runs, so while the repository variable is unset they measure no second set. The second set is best effort: when a round of it or its comparison fails, or it stops before its first pair, the report says so and Tier 3 still runs; a run that posts no comment shows that only in its job summary.BENCH_CONFIRM_DEADLINE(1080 s;docs/testing.mdhas the exact rule), so Tier 3 still fits in the 25 minute step. On the first CI run that left room only for pairs of the flagged metrics under about 51 s each, calibration readings included; when a pair takes longer the set is cut, and a metric that readsslowerat the configured threshold readsnot confirmed.benchmark.ymladds thebenchlabel trigger and the twoschedulelines. Two choices differ from Benchmark each PR against its base commit and chart main on elfuse-bench #503: the weekly base comes fromgit rev-list -1 --first-parent --before='4 weeks ago', because without--first-parentit can be a commit of a branch merged later, and schedule runs keep their artifacts 90 days instead of 14, so the A/A data is still there when the thresholds are set from it.bench-report.ymlcomments on a labeled pull request even when the report is clean, then takes the label off (also when it leaves the comment to a later merge ref's report), so adding it again measures again, and it ignores a run whose jobs were all skipped.If the two sets were independent, an unchanged metric would pass both about 0.05% of the time (2 x (1/64)^2), not the 0.1% #503 gives; the nightly A/A data will show the real rate.
Left out: the two group thresholds (they need A/A data, so
BENCH_REGRESSION_THRESHOLDstays unset), the runs against a deliberately slowed branch, and creating thebenchlabel.Testing:
make check-bench-reportpasses under Python 3 and 3.9 at each commit;make check-formatandscripts/check-commit-log.shpass.slowerin both sets (5.06x, then 5.11x).Not tested: No test runs three
BENCH_METRICSpaths oftests/bench-suite.sh, because they need an elfuse build: a tier left out and thelat_fsdrop were checked on local runs, andERRin place of the no-results stop was not run. On GitHub Actions only theopenedpath has run (run 44875161536, nothing flagged), and on an earlier revision (939253c), so the scope step's choice ofBENCH_FLAG_THRESHOLDand the best-effort second set have not run there. Thebenchlabel does not exist yet, theschedulelines run only onmain, andbench-report.ymlruns frommain, so those changes, the label removal among them, are checked by reading. The second set's timing on CI.Refs #503