Repository navigation
Conversation
jserv
reviewed
Nov 21, 2026
xalestar
force-pushed
the
bench-dashboard
branch
2 times, most recently
from
November 21, 2026 17:30
a878a14 to
b53323f
Compare
xalestar
force-pushed
the
bench-dashboard
branch
from
November 21, 2026 17:34
b53323f to
779a931
Compare
jserv
reviewed
Nov 21, 2026
scripts/bench-dashboard.py reads the rounds scripts/bench-paired.sh leaves and prints the list that github-action-benchmark takes with its customSmallerIsBetter tool. Each Tier 1 and Tier 2 metric gets the median of its head rounds and, under the same name with " (base)" appended, the median of its base rounds. A host that runs slow for one job then moves both series at the same merge, where a code change moves the head series alone. Tier 3 runs on the head build only and gets one point, from head-oci.json. The rounds go through bench-compare.py's flatten(), so a point carries the metric name of the comparison report. An ERR row, a zero and a tier the round records as skipped count as missing, as they do there, and a metric missing from every round of a side gets no point on that side. Each side's median takes every round of that side with a value, where the comparison keeps only the rounds in which both sides have one, so the two medians can differ. Given the dashboard's data.js and a commit, the script leaves out the points data.js already holds for that commit, so a rerun adds only what an earlier attempt did not publish. A data.js it cannot read or parse leaves no point out, because a point published twice costs less than a point lost. It is Python rather than jq because its self-test runs in make check: the bench scripts already need python3, and the host requirements in docs/testing.md do not list jq. Refs sysprog21#503
A push run of benchmark.yml ends with a publish job that turns the bench-rounds artifact into points and pushes them to the main branch of sysprog21/elfuse-bench, for its GitHub Pages dashboard. Only a push reaches it, so a pull request or dispatch run never lands on the chart, and only once report.json shows that every pair ran; a run whose Tier 3 failed still adds its Tier 1 and Tier 2 points. The action appends an entry even for a commit it already holds, so the job first fetches data.js and has the script leave out the points it holds for the pushed commit. "Re-run failed jobs" after a red bench job then adds only the Tier 3 points the first attempt lacked; skipping the whole commit would drop them. A fetch that fails, stalls past its 60 s limit, or returns a copy cached from before that attempt publishes the points a second time. github-action-benchmark is third-party, so it is pinned to the v1.22.2 commit and only its step gets ELFUSE_BENCH_TOKEN. While that secret is unset the job prints a notice and passes, so main stays green until the token is added. The job has no concurrency group, because the action re-clones the data branch and retries when another publish pushed first. Refs sysprog21#503
xalestar
force-pushed
the
bench-dashboard
branch
from
November 21, 2026 18:13
779a931 to
7e6d567
Compare
jserv
reviewed
Nov 22, 2026
| if: ${{ steps.points.outputs.ready == 'true' }} | ||
| # A third-party action, so it is pinned to a commit and is the only | ||
| # step that gets the token. | ||
| uses: benchmark-action/github-action-benchmark@4322e5726e6334590d251fc4f92bec0efafc45dc # v1.22.2 |
Contributor
There was a problem hiding this comment.
The dashboard sorts commits by the action entry’s date, which is the time the action runs, not the merge order. If an older benchmark finishes or is rerun after a newer merge has published, the chart places the older commit later and can attribute a performance change to the wrong merge. Please order the dashboard by the measured commits’ position on main (including when a rerun adds missing metrics), rather than publication time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent: Add each measured merge to
mainto the elfuse-bench dashboard (https://sysprog21.github.io/elfuse-bench/): per metric, the median of the head rounds, and the median of the base rounds as a second series, so a shift of the runner shows as both series moving at the same merge. This is PR 4 of #503.This PR:
scripts/bench-dashboard.pyturns thebench-roundsartifact into thecustomSmallerIsBetterinput ofbenchmark-action/github-action-benchmark. For each Tier 1 and Tier 2 metric it gives the median of thehead-N.jsonrounds under the metric's name in the comparison report, and the median of thebase-N.jsonrounds under that name with(base)appended; for each Tier 3 metric, its median inhead-oci.json. As inbench-compare.py, an ERR row, a zero and a tier the round records as skipped count as missing, and a metric missing from every round of a side gets no point on that side. A side's median takes every round of that side with a value, where the comparison keeps only the rounds in which both sides have one. It reads onlybase-N.jsonandhead-N.json, so theconfirm-*rounds of Confirm benchmark verdicts and run A/A nightly #543 do not change the points. Given the dashboard'sdata.jsand a commit, it leaves out the pointsdata.jsalready holds for that commit. Its self-test runs inmake check-bench-report.benchmark.ymladds apublishjob for push runs only, so the label and schedule runs of Confirm benchmark verdicts and run A/A nightly #543 never reach the chart. It adds no point whenreport.jsonis missing (the comparison did not complete), and still adds Tier 1 and Tier 2 when Tier 3 failed. The action appends an entry even for a commit it already holds, so the job first fetchesdata.js(10 s to connect, 60 s in all) and publishes only the points it lacks: "Re-run failed jobs" after a failed Tier 3 adds just the Tier 3 points. A fetch that fails, or returns a copy raw.githubusercontent.com cached before the first attempt (it keeps one for up to 5 minutes), publishes the points a second time. The action is pinned to commit 4322e57 (v1.22.2), and the step that runs it is the only one that gets the token. There is no concurrency group: when two publishes race, the one whose push is rejected clones again and retries inside the action.The script is Python, as #503 now says, because its self-test runs in
make check, the bench scripts already need python3, and the host requirements indocs/testing.mddo not listjq. The separatelat_syscall nullseries that #503 asks for needs no code, because the dashboard draws one chart per metric.#543 edits the same workflow. This PR only appends a job and a header line there, and
git merge-treewith #543's branch reports no conflict.Setup: elfuse-bench now has a
mainbranch served by GitHub Pages, with an emptydata.jsand anindex.htmlthat draws eachX (base)series on the chart ofX, one point per commit. What is left needs an elfuse admin: the repository secretELFUSE_BENCH_TOKEN, a fine-grained token with Contents read and write onsysprog21/elfuse-benchonly. Until it is set, the job prints a notice and passes, so no merge goes red.Testing:
make check-bench-reportpasses under Python 3.14 and 3.9 at each commit;make check-formatandscripts/check-commit-log.shpass.make checkpasses except thetest-busyboxnslookup case, which fails the same way on an upstream/main build.meta.envcheck, no rounding, keeping a zero, ignoring a skipped tier, nodata.jsfilter, matching another commit's entry, and failing on an unreadabledata.js.bench-roundsartifact of run 44836576102, the script gives 34 points (15 metrics with head and base, and 4 Tier 3). Each Tier 1 and Tier 2 value equals the median of that side's list inreport.json; the script rounds to 12 significant digits, which drops only float noise such as 1.0261999999999998.publishjob, taken from the YAML and run locally on that artifact: with every point already indata.jsit prints the notice and publishes nothing; with Tier 1 and 2 already there, the 4 Tier 3 points; with the commit absent from the livedata.js, 34 points; on a 404, and on a server that accepts and never answers (time limit lowered to 3 s), the warning and 34 points.Not tested: The
publishjob has not run on GitHub Actions, because it runs only on a push tomain; the notice while the secret is unset is checked by reading.actionlintwas not run. The dashboard page has not been opened in a browser; its script was run with a stub DOM on made-up data.Refs #503