Repository navigation
Migrate nightly benchmark result persistence to dedicated branch - #28
Conversation
Nightly package-mode runs #37-#41 computed their results and then failed at persistence: the generated result JSONs were committed to `main`, which the Protect Main ruleset only accepts through a pull request. Generated benchmark data is not reviewable source, so it moves to a dedicated `benchmark-results` branch and Pages assembles history from both branches. - nightly-benchmark: dispatch benchmark.yml with runtime_source=package, trust only the returned workflow_run_id, download just the two result artifacts and validate them before staging (package runtime, run_id, and one identical identity.runtime.package on both arms). Stage outside every checkout, commit exactly two files with cmp -s idempotency and a two-path assertion, rebase before pushing, retry only when the remote tip moved, never force-push, and dispatch Pages only after a successful push. - pages: check out main for code and pinned reference data plus benchmark-results for generated history, assemble site-results from both, never copy main's frozen pre-split run trees, and fail loudly on a missing branch or empty history instead of publishing an empty report. - recovery: recover_nightly_results.py re-imports retained artifacts into the data branch, dry-run by default, refusing main collisions, partial, differing, mixed-package and duplicate run IDs; --apply commits per run and never pushes. - tests: recover_nightly_results.py unit tests plus two harnesses that execute the real workflow steps against a PATH-shadowed git double, registered in pr-checks. The five affected runs (#37-#41) were recovered separately onto benchmark-results as commits 4641bc0, 9bd9929, f5577d7, 328eb67 and 7ae1a8f. Co-authored-by: Codebuff <noreply@codebuff.com> Assisted-by: ChatGPT
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
|
Written by an AI agent using GPT-6. (Reviewed by Sam) Reviewed commit f606f99. I found three issues:
|
|
Thank you for the review sir , I am working on fixing the same and will do the necessary changes by today itself ,, meanwhile can you please answer the queries I had related to the self hosted runner , I will be really thankful for the same , thank you and have an amazing day ! |
Co-authored-by: Codebuff <noreply@codebuff.com> Assisted-by: ChatGPT
|
@poshul Hello sir , thank you for the review. I have now addressed all the three issues in the latest commit. The fixes now cover Pages history preservation , the recovery documentation and main branch collision detection. All relevant tests passed:-
The changes have been pushed to the PR. Live end to end Pages deployment validation is still pending. Thanks a ton for the feedback on the same , really appreciate it and have an amazing day ahead ! |
Description
Nightly package-mode runs #37–#41 computed their benchmark results successfully and then failed at persistence: the workflow tried to commit the generated result JSONs to
main, which the "Protect Main" ruleset only accepts through a pull request. The benchmark computations themselves are not affected — the reporting pipeline simply never received their output.Generated benchmark data is not reviewable source code, so this change moves it to a dedicated
benchmark-resultsbranch, and Pages now assembles the report from code and pinned reference data onmainplus generated history frombenchmark-results.Design intent and rationale
.github/workflows/nightly-benchmark.yml): dispatchesbenchmark.ymlwithruntime_source=package, identifies the run from the returnedworkflow_run_id, downloads only the two expected result artifacts, and validates them before staging. Both JSONs must carry matching package identity metadata (filename,sha256,archive_dir,date) with non-empty string values. Staging happens outside the checkouts. Persistence commits exactly two result files, skips them when they are already byte-identical, asserts the commit contains exactly those two paths, rebases before pushing, and retries only when the remote tip has actually moved. It never force-pushes, and Pages is dispatched only after a successful push, so a stale site cannot look fresh..github/workflows/pages.yml): checks outmainfor code and pinned reference data andbenchmark-resultsfor generated history, assemblessite-results/from the generated OpenMS and OpenSwath results plusbenchmark/results/toolsfrommain, then renders with--results-dir site-results. The frozen pre-migration result trees onmainare deliberately not copied, so no run can be counted twice. A missing data branch, empty history or missing tool reference data fails the workflow instead of publishing an incomplete report.benchmark/recover_nightly_results.py): a dry-run-by-default utility that recovers results from retained GitHub Actions artifacts. It validates artifact IDs, result JSONs, run IDs, timestamps and matching package metadata, and it refuses collisions withmain, incomplete or differing result pairs, mixed-package pairs and duplicate run IDs. With--applyit creates one local commit per recovered run on an existing cleanbenchmark-resultscheckout; it never pushes and never creates branches.pr-checks.yml. The shell harnesses execute the relevant workflow steps extracted from the YAML against a controlledgittest double, covering persistence, failure handling, idempotency and Pages assembly without touching a real remote.Recovered historical results
The five affected runs (#37–#41) were recovered from retained artifacts and pushed to
benchmark-resultsas five separate commits:4641bc0OpenMS-3.7.0-pre-nightly-2026-11-06-Debian-Linux-x86_64.deb9bd9929OpenMS-3.7.0-pre-nightly-2026-11-06-Debian-Linux-x86_64.debf5577d7OpenMS-3.7.0-pre-nightly-2026-11-08-Debian-Linux-x86_64.deb328eb67OpenMS-3.7.0-pre-nightly-2026-11-09-Debian-Linux-x86_64.deb7ae1a8fOpenMS-3.7.0-pre-nightly-2026-11-10-Debian-Linux-x86_64.debThis produced ten result JSON files (five
openms/smoke-*and fiveopenswath_dia/openms/openswath_dia-*). Every run's two files were verified to share one package identity. #37 and #38 legitimately share the 2026-11-06 package because no newer nightly was published that weekend.Validation and test evidence
Run by the author on the Windows development environment against this branch's content, before the commit was created:
python -m pytest benchmark/test_recover_nightly_results.py -v— 38 passed, plus 13 subtests.bash tests/test_persist_step.sh— 24/24 passed.bash tests/test_pages_assembly.sh— 8/8 passed.python -m py_compile benchmark/recover_nightly_results.py benchmark/test_recover_nightly_results.py— passed.git diff --check— clean.YAML OK.actionlint— no findings in the earlier run with ShellCheck integration disabled (ShellCheck itself is not installed on the author's machine, so this is not a ShellCheck result).The pytest run reported a
.pytest_cachepermission warning; all tests nevertheless passed.Local run screenshots
Known limitations
ghever returns a non-ZIP body with a zero exit code, extraction raiseszipfile.BadZipFileinstead of the utility'sRecoveryError; it still fails closed, but with a traceback rather than a clean message.Checklist
This repository has no
AUTHORSfile, so the first item is not applicable. There are no Python bindings here, so the last item is ticked as not necessary.AI use (see the OpenMS AI Policy)
Assisted-by:trailerCo-authored-by:trailer, and the description above explains the design intent and rationaleSubstantial agentic AI assistance was used for the implementation and testing of this contribution.
The commit carries a
Co-authored-by:trailer for the agentic tool and anAssisted-by: ChatGPTtrailer, per the policy. No AI tool produced the benchmark results themselves, and no AI tool performed the live deployment.No need to edit the CHANGELOG file: entries are generated automatically once per week from merged changes.
How can I get additional information on failed tests during CI
Click to expand
If your PR is failing you can check outIf you click in the column that lists the failed tests you will get detailed error messages.
Advanced commands (admins / reviewer only)
Click to expand
/reformat(experimental) applies the clang-format style changes as additional commit. Note: your branch must have a different name (e.g., yourrepo:feature/XYZ) than the receiving branch (e.g., OpenMS:develop). Otherwise, reformat fails to push.The tests conducted on the local build are as follows:-

