Skip to content

Chart each measured merge on elfuse-bench - #545

Open
xalestar wants to merge 2 commits into
sysprog21:mainfrom
xalestar:bench-dashboard
Open

xalestar wants to merge 2 commits into
sysprog21:mainfrom
xalestar:bench-dashboard

Conversation

@xalestar

@xalestar xalestar commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Intent: Add each measured merge to main to the elfuse-bench dashboard (https://sysprog21.github.io/elfuse-bench/): per metric, the median of the head rounds, and the median of the base rounds as a second series, so a shift of the runner shows as both series moving at the same merge. This is PR 4 of #503.

This PR:

  • scripts/bench-dashboard.py turns the bench-rounds artifact into the customSmallerIsBetter input of benchmark-action/github-action-benchmark. For each Tier 1 and Tier 2 metric it gives the median of the head-N.json rounds under the metric's name in the comparison report, and the median of the base-N.json rounds under that name with (base) appended; for each Tier 3 metric, its median in head-oci.json. As in bench-compare.py, an ERR row, a zero and a tier the round records as skipped count as missing, and a metric missing from every round of a side gets no point on that side. A side's median takes every round of that side with a value, where the comparison keeps only the rounds in which both sides have one. It reads only base-N.json and head-N.json, so the confirm-* rounds of Confirm benchmark verdicts and run A/A nightly #543 do not change the points. Given the dashboard's data.js and a commit, it leaves out the points data.js already holds for that commit. Its self-test runs in make check-bench-report.
  • benchmark.yml adds a publish job for push runs only, so the label and schedule runs of Confirm benchmark verdicts and run A/A nightly #543 never reach the chart. It adds no point when report.json is missing (the comparison did not complete), and still adds Tier 1 and Tier 2 when Tier 3 failed. The action appends an entry even for a commit it already holds, so the job first fetches data.js (10 s to connect, 60 s in all) and publishes only the points it lacks: "Re-run failed jobs" after a failed Tier 3 adds just the Tier 3 points. A fetch that fails, or returns a copy raw.githubusercontent.com cached before the first attempt (it keeps one for up to 5 minutes), publishes the points a second time. The action is pinned to commit 4322e57 (v1.22.2), and the step that runs it is the only one that gets the token. There is no concurrency group: when two publishes race, the one whose push is rejected clones again and retries inside the action.

The script is Python, as #503 now says, because its self-test runs in make check, the bench scripts already need python3, and the host requirements in docs/testing.md do not list jq. The separate lat_syscall null series that #503 asks for needs no code, because the dashboard draws one chart per metric.

#543 edits the same workflow. This PR only appends a job and a header line there, and git merge-tree with #543's branch reports no conflict.

Setup: elfuse-bench now has a main branch served by GitHub Pages, with an empty data.js and an index.html that draws each X (base) series on the chart of X, one point per commit. What is left needs an elfuse admin: the repository secret ELFUSE_BENCH_TOKEN, a fine-grained token with Contents read and write on sysprog21/elfuse-bench only. Until it is set, the job prints a notice and passes, so no merge goes red.

Testing:

  • make check-bench-report passes under Python 3.14 and 3.9 at each commit; make check-format and scripts/check-commit-log.sh pass. make check passes except the test-busybox nslookup case, which fails the same way on an upstream/main build.
  • Each self-test case fails when the code it covers is broken. 14 mutations were each caught: a match on the round file names without the start or the end anchor, mean instead of median, dropping a metric that only base measured, accepting an empty list, no unit check, no base-round check, no meta.env check, no rounding, keeping a zero, ignoring a skipped tier, no data.js filter, matching another commit's entry, and failing on an unreadable data.js.
  • On the bench-rounds artifact of run 44836576102, the script gives 34 points (15 metrics with head and base, and 4 Tier 3). Each Tier 1 and Tier 2 value equals the median of that side's list in report.json; the script rounds to 12 significant digits, which drops only float noise such as 1.0261999999999998.
  • The points step of the publish job, taken from the YAML and run locally on that artifact: with every point already in data.js it prints the notice and publishes nothing; with Tier 1 and 2 already there, the 4 Tier 3 points; with the commit absent from the live data.js, 34 points; on a 404, and on a server that accepts and never answers (time limit lowered to 3 s), the warning and 34 points.

Not tested: The publish job has not run on GitHub Actions, because it runs only on a push to main; the notice while the secret is unset is checked by reading. actionlint was not run. The dashboard page has not been opened in a browser; its script was run with a stub DOM on made-up data.

Refs #503

@jserv
jserv requested a review from Max042004 November 21, 2026 17:13
cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread .github/workflows/benchmark.yml Resolved
@xalestar
xalestar force-pushed the bench-dashboard branch 2 times, most recently from a878a14 to b53323f Compare November 21, 2026 17:30
cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread .github/workflows/benchmark.yml Resolved
scripts/bench-dashboard.py reads the rounds scripts/bench-paired.sh
leaves and prints the list that github-action-benchmark takes with its
customSmallerIsBetter tool. Each Tier 1 and Tier 2 metric gets the
median of its head rounds and, under the same name with " (base)"
appended, the median of its base rounds. A host that runs slow for one
job then moves both series at the same merge, where a code change moves
the head series alone. Tier 3 runs on the head build only and gets one
point, from head-oci.json.

The rounds go through bench-compare.py's flatten(), so a point carries
the metric name of the comparison report. An ERR row, a zero and a
tier the round records as skipped count as missing, as they do there,
and a metric missing from every round of a side gets no point on that
side. Each side's median takes every round of that side with a value,
where the comparison keeps only the rounds in which both sides have
one, so the two medians can differ.

Given the dashboard's data.js and a commit, the script leaves out the
points data.js already holds for that commit, so a rerun adds only what
an earlier attempt did not publish. A data.js it cannot read or parse
leaves no point out, because a point published twice costs less than a
point lost.

It is Python rather than jq because its self-test runs in make check:
the bench scripts already need python3, and the host requirements in
docs/testing.md do not list jq.

Refs sysprog21#503
A push run of benchmark.yml ends with a publish job that turns the
bench-rounds artifact into points and pushes them to the main branch of
sysprog21/elfuse-bench, for its GitHub Pages dashboard. Only a push
reaches it, so a pull request or dispatch run never lands on the
chart, and only once report.json shows that every pair ran; a run whose
Tier 3 failed still adds its Tier 1 and Tier 2 points.

The action appends an entry even for a commit it already holds, so the
job first fetches data.js and has the script leave out the points it
holds for the pushed commit. "Re-run failed jobs" after a red bench job
then adds only the Tier 3 points the first attempt lacked; skipping the
whole commit would drop them. A fetch that fails, stalls past its 60 s
limit, or returns a copy cached from before that attempt publishes the
points a second time.

github-action-benchmark is third-party, so it is pinned to the v1.22.2
commit and only its step gets ELFUSE_BENCH_TOKEN. While that secret is
unset the job prints a notice and passes, so main stays green until the
token is added. The job has no concurrency group, because the action
re-clones the data branch and retries when another publish pushed
first.

Refs sysprog21#503
cubic-dev-ai[bot]

This comment was marked as resolved.

if: ${{ steps.points.outputs.ready == 'true' }}
# A third-party action, so it is pinned to a commit and is the only
# step that gets the token.
uses: benchmark-action/github-action-benchmark@4322e5726e6334590d251fc4f92bec0efafc45dc # v1.22.2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dashboard sorts commits by the action entry’s date, which is the time the action runs, not the merge order. If an older benchmark finishes or is rerun after a newer merge has published, the chart places the older commit later and can attribute a performance change to the wrong merge. Please order the dashboard by the measured commits’ position on main (including when a rerun adds missing metrics), rather than publication time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants