Skip to content

Migrate nightly benchmark result persistence to dedicated branch - #28

Merged
poshul merged 2 commits into
mainfrom
feat/benchmark-results-persistence
Nov 17, 2026
Merged

poshul merged 2 commits into
mainfrom
feat/benchmark-results-persistence

Conversation

@nej1gotnochill

@nej1gotnochill nej1gotnochill commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Nightly package-mode runs #37–#41 computed their benchmark results successfully and then failed at persistence: the workflow tried to commit the generated result JSONs to main, which the "Protect Main" ruleset only accepts through a pull request. The benchmark computations themselves are not affected — the reporting pipeline simply never received their output.

Generated benchmark data is not reviewable source code, so this change moves it to a dedicated benchmark-results branch, and Pages now assembles the report from code and pinned reference data on main plus generated history from benchmark-results.

Design intent and rationale

  • Nightly persistence (.github/workflows/nightly-benchmark.yml): dispatches benchmark.yml with runtime_source=package, identifies the run from the returned workflow_run_id, downloads only the two expected result artifacts, and validates them before staging. Both JSONs must carry matching package identity metadata (filename, sha256, archive_dir, date) with non-empty string values. Staging happens outside the checkouts. Persistence commits exactly two result files, skips them when they are already byte-identical, asserts the commit contains exactly those two paths, rebases before pushing, and retries only when the remote tip has actually moved. It never force-pushes, and Pages is dispatched only after a successful push, so a stale site cannot look fresh.
  • Pages assembly (.github/workflows/pages.yml): checks out main for code and pinned reference data and benchmark-results for generated history, assembles site-results/ from the generated OpenMS and OpenSwath results plus benchmark/results/tools from main, then renders with --results-dir site-results. The frozen pre-migration result trees on main are deliberately not copied, so no run can be counted twice. A missing data branch, empty history or missing tool reference data fails the workflow instead of publishing an incomplete report.
  • Historical recovery (benchmark/recover_nightly_results.py): a dry-run-by-default utility that recovers results from retained GitHub Actions artifacts. It validates artifact IDs, result JSONs, run IDs, timestamps and matching package metadata, and it refuses collisions with main, incomplete or differing result pairs, mixed-package pairs and duplicate run IDs. With --apply it creates one local commit per recovered run on an existing clean benchmark-results checkout; it never pushes and never creates branches.
  • Tests: adds unit tests for the recovery utility plus persistence and Pages assembly harnesses registered in pr-checks.yml. The shell harnesses execute the relevant workflow steps extracted from the YAML against a controlled git test double, covering persistence, failure handling, idempotency and Pages assembly without touching a real remote.

Recovered historical results

The five affected runs (#37–#41) were recovered from retained artifacts and pushed to benchmark-results as five separate commits:

Nightly run Benchmark run ID Recovery commit Package
#37 42333314804 4641bc0 OpenMS-3.7.0-pre-nightly-2026-11-06-Debian-Linux-x86_64.deb
#38 42442257487 9bd9929 OpenMS-3.7.0-pre-nightly-2026-11-06-Debian-Linux-x86_64.deb
#39 42555927417 f5577d7 OpenMS-3.7.0-pre-nightly-2026-11-08-Debian-Linux-x86_64.deb
#40 42748636332 328eb67 OpenMS-3.7.0-pre-nightly-2026-11-09-Debian-Linux-x86_64.deb
#41 42951989482 7ae1a8f OpenMS-3.7.0-pre-nightly-2026-11-10-Debian-Linux-x86_64.deb

This produced ten result JSON files (five openms/smoke-* and five openswath_dia/openms/openswath_dia-*). Every run's two files were verified to share one package identity. #37 and #38 legitimately share the 2026-11-06 package because no newer nightly was published that weekend.

Validation and test evidence

Run by the author on the Windows development environment against this branch's content, before the commit was created:

  • python -m pytest benchmark/test_recover_nightly_results.py -v — 38 passed, plus 13 subtests.
  • bash tests/test_persist_step.sh — 24/24 passed.
  • bash tests/test_pages_assembly.sh — 8/8 passed.
  • python -m py_compile benchmark/recover_nightly_results.py benchmark/test_recover_nightly_results.py — passed.
  • git diff --check — clean.
  • YAML parsing of all three modified workflows — YAML OK.
  • actionlint — no findings in the earlier run with ShellCheck integration disabled (ShellCheck itself is not installed on the author's machine, so this is not a ShellCheck result).
  • A real GitHub CLI dry-run against the retained artifacts for #37–#41 — all five runs validated, no repository changes made. This dry-run predates the current artifact-download implementation, which is covered by mocked unit tests only.

The pytest run reported a .pytest_cache permission warning; all tests nevertheless passed.

Local run screenshots Screenshot 2026-11-15 133049 Screenshot 2026-11-15 133100 Screenshot 2026-11-15 133108 Screenshot 2026-11-15 133127 Screenshot 2026-11-15 133222 Screenshot 2026-11-15 133350 Screenshot 2026-11-15 133806 Screenshot 2026-11-15 133908 Screenshot 2026-11-15 133916 Screenshot 2026-11-15 133939

Known limitations

  • The nightly workflow has not been verified end to end in real GitHub Actions after this migration: first post-merge persistence, Pages dispatch and deployed report freshness still need verification. No deployment is claimed here.
  • The local harnesses exercise the workflow steps with test doubles and do not replace a live Actions run.
  • If gh ever returns a non-ZIP body with a zero exit code, extraction raises zipfile.BadZipFile instead of the utility's RecoveryError; it still fails closed, but with a traceback rather than a clean message.

Checklist

  • Make sure that you are listed in the AUTHORS file
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing unit tests pass locally with my changes
  • Updated or added python bindings for changed or new classes (Tick if no updates were necessary.)

This repository has no AUTHORS file, so the first item is not applicable. There are no Python bindings here, so the last item is ticked as not necessary.

AI use (see the OpenMS AI Policy)

  • I have read the OpenMS AI Policy, and I understand, have reviewed and tested every change in this PR and can explain it in my own words
  • AI use in this PR (tick one):
    • None
    • Assisted (e.g. completion, suggestions, small edits); commits carry an Assisted-by: trailer
    • Substantial (e.g. agentic tools wrote significant parts); commits carry a Co-authored-by: trailer, and the description above explains the design intent and rationale

Substantial agentic AI assistance was used for the implementation and testing of this contribution.

  • Implementation and test development: an agentic coding assistant (Freebuff) wrote the workflow migration, the recovery utility, both test harnesses and their validation.
  • Design input and review: ChatGPT provided design discussion and independent review of the diff; it did not author the code.
  • Human responsibility: the author directed the work, reviewed and tested the changes, and takes responsibility for understanding and maintaining the contribution.

The commit carries a Co-authored-by: trailer for the agentic tool and an Assisted-by: ChatGPT trailer, per the policy. No AI tool produced the benchmark results themselves, and no AI tool performed the live deployment.

No need to edit the CHANGELOG file: entries are generated automatically once per week from merged changes.

How can I get additional information on failed tests during CI

Click to expand If your PR is failing you can check out
  • The details of the action statuses at the end of the PR or the "Checks" tab.
  • http://cdash.seqan.de/index.php?project=OpenMS and look for your PR. Use the "Show filters" capability on the top right to search for your PR number.
    If you click in the column that lists the failed tests you will get detailed error messages.

Advanced commands (admins / reviewer only)

Click to expand
  • /reformat (experimental) applies the clang-format style changes as additional commit. Note: your branch must have a different name (e.g., yourrepo:feature/XYZ) than the receiving branch (e.g., OpenMS:develop). Otherwise, reformat fails to push.

⚠️ Note: Once you opened a PR try to minimize the number of pushes to it as every push will trigger CI (automated builds and test) and is rather heavy on our infrastructure (e.g., if several pushes per day are performed).

The tests conducted on the local build are as follows:-
Screenshot 2026-11-15 133916
Screenshot 2026-11-15 133939

Screenshot 2026-11-15 133222 Screenshot 2026-11-15 133350 Screenshot 2026-11-15 133806 Screenshot 2026-11-15 133908 Screenshot 2026-11-15 133049 Screenshot 2026-11-15 133100 Screenshot 2026-11-15 133108 Screenshot 2026-11-15 133127 Screenshot 2026-11-15 133215

Nightly package-mode runs #37-#41 computed their results and then failed at
persistence: the generated result JSONs were committed to `main`, which the
Protect Main ruleset only accepts through a pull request. Generated benchmark
data is not reviewable source, so it moves to a dedicated `benchmark-results`
branch and Pages assembles history from both branches.

- nightly-benchmark: dispatch benchmark.yml with runtime_source=package, trust
  only the returned workflow_run_id, download just the two result artifacts and
  validate them before staging (package runtime, run_id, and one identical
  identity.runtime.package on both arms). Stage outside every checkout, commit
  exactly two files with cmp -s idempotency and a two-path assertion, rebase
  before pushing, retry only when the remote tip moved, never force-push, and
  dispatch Pages only after a successful push.
- pages: check out main for code and pinned reference data plus benchmark-results
  for generated history, assemble site-results from both, never copy main's
  frozen pre-split run trees, and fail loudly on a missing branch or empty
  history instead of publishing an empty report.
- recovery: recover_nightly_results.py re-imports retained artifacts into the
  data branch, dry-run by default, refusing main collisions, partial, differing,
  mixed-package and duplicate run IDs; --apply commits per run and never pushes.
- tests: recover_nightly_results.py unit tests plus two harnesses that execute
  the real workflow steps against a PATH-shadowed git double, registered in
  pr-checks.

The five affected runs (#37-#41) were recovered separately onto
benchmark-results as commits 4641bc0, 9bd9929, f5577d7, 328eb67 and 7ae1a8f.

Co-authored-by: Codebuff <noreply@codebuff.com>
Assisted-by: ChatGPT
@nej1gotnochill
nej1gotnochill requested review from a team and a balanced review from Copilot November 15, 2026 09:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4a2f2521-5549-4f18-8174-3d1617aaa642
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@poshul

poshul commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Written by an AI agent using GPT-6. (Reviewed by Sam)

Reviewed commit f606f99. I found three issues:

  1. [P1] Preserve the existing benchmark history during Pages assembly.
    pages.yml, lines 106–110 copies generated history exclusively from benchmark-results. That branch currently contains 10 recovered result files; main contains 25 older result files, with no overlapping paths. Executing the assembly step confirms the renderer receives only the 10 recovered results, silently dropping all 25 existing results from the report. Please migrate the older history before switching sources, or assemble a union with duplicate detection.
  2. [P2] Make the documented recovery procedure executable.
    recover_nightly_results.py, lines 45–49 instructs users to switch to benchmark-results and then execute the script. That branch contains only result JSONs, so switching removes the script. I reproduced the command failing with “No such file or directory.” Please document a separate code checkout and data worktree, invoking the script with --repo-dir pointing to the latter, and update the README accordingly.
  3. [P2] Check collisions against the actual result paths on main.
    recover_nightly_results.py, lines 385–391 checks paths such as main:openms/smoke-.json, but results on main reside under benchmark/results/. Consequently, the advertised collision guard accepts an existing main result. I reproduced this in an isolated repository. Please prepend benchmark/results/ when checking main and use that layout in the collision tests.
    Validation: all seven Python suites and four shell harnesses passed locally. These findings were reproduced separately; no live Actions deployment was performed.

@nej1gotnochill

Copy link
Copy Markdown
Collaborator Author

Thank you for the review sir , I am working on fixing the same and will do the necessary changes by today itself ,, meanwhile can you please answer the queries I had related to the self hosted runner , I will be really thankful for the same , thank you and have an amazing day !

Co-authored-by: Codebuff <noreply@codebuff.com>
Assisted-by: ChatGPT
@nej1gotnochill

Copy link
Copy Markdown
Collaborator Author

@poshul Hello sir , thank you for the review. I have now addressed all the three issues in the latest commit. The fixes now cover Pages history preservation , the recovery documentation and main branch collision detection.

All relevant tests passed:-

  • Recovery tests:- 39 passed
  • Pages assembly:- 18 tests passed
  • Persistence harness:- 24/24 passed

The changes have been pushed to the PR. Live end to end Pages deployment validation is still pending. Thanks a ton for the feedback on the same , really appreciate it and have an amazing day ahead !

@poshul
poshul merged commit 7ae0dfa into main Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants