Skip to content

[RF][HS3] Add an elective test with the HS3 documents on HEPData - #24228

Draft
cburgard wants to merge 4 commits into
root-project:masterfrom
cburgard:hs3-hepdata-test
Draft

cburgard wants to merge 4 commits into
root-project:masterfrom
cburgard:hs3-hepdata-test

Conversation

@cburgard

@cburgard cburgard commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

This Pull request:

Adds an elective test that runs the HS3 documents published on HEPData through RooFit: import, likelihood values, round trip and, optionally, fits. It serves as a regression test of the HS3 import with real analyses and, with the fits, as a benchmark.
It only extends the tests; it changes no library code.

Important

Stacked on #23839 (the HS3 conformance test suite): the first commit of this PR is the commit of #23839, the last two commits are new. Draft until #23839 is merged; it then needs a rebase.

Changes or fixes:

The documents on HEPData are the ones that users of RooFit actually have, and most were written by older versions of ROOT. They show whether RooFit still reads what it once wrote, which no other test does: recently added checks made some of them unreadable without any test noticing (see #24201). Being real analyses, they are also realistic benchmarks.

First commit: the elective test.

  • New test roofit/hs3/test/test_hs3_hepdata.py. It asks HEPData for all records with resources of the type "HS3" (analysis:hs3) and downloads them.
  • Elective, not for the CI. The test is enabled with the new option -Dtest_roofit_hs3hepdata=ON (default OFF, requires -Dtesting=ON and -Dpyroot=ON). It is deliberately not added to the CI configuration: HEPData must not be queried on every build.
  • Cache. Everything that is downloaded is kept in a cache directory (HS3HEPDATA_CACHE, by default in the build directory) and not downloaded again. HS3HEPDATA_OFFLINE=1 runs on the cache without contacting HEPData; HS3HEPDATA_SELECT=<regex> restricts the test to matching documents.
  • Access through hepdata-cli. HEPData answers requests of generic scripted clients with 403, so the test uses HEPData's own Python client (a new Python dependency of this test only). Its public download command only covers tables, so the resource files are fetched with the request helper of the package (hepdata_cli.resilient_requests), which is not a documented interface.

Second commit: the documents run through the HS3 test suite, with fits (asked for in the review, to make the test usable as a benchmark).

  • Fixtures in roofit/hs3/test/hs3hepdata, in the layout of the HS3 conformance suite but without the HS3 documents: those are downloaded when the test runs, and manifest.json pins them by their hashes. hepdata.json lists where each document is found.

  • Checks of the suite: the import with the pdf and data set of every analysis, and 2·(NLL − NLL(reference)) at fixed points along the parameters of interest, with the full likelihood of the ModelConfig (global observables and external constraints).

  • On top: the round trip (export and import again) and, with -DHS3HEPDATA_FITS=ON or HS3HEPDATA_FITS=1, a fit per analysis:

    • the best fit, Hesse and Minos uncertainties of the parameters of interest and of the free normalization factors;
    • a profile likelihood scan;
    • stored in expected_fit.json, a format of this test, since the suite has no check for fits.

    Every fit names the parameter point of the document it starts from and its Minuit strategy. HS3HEPDATA_EVAL_BACKEND selects the evaluation backend, e.g. codegen.

  • Timings. The time of every step (import, round trip, creating the likelihood, minimization, Hesse, Minos, profile scans) is written to timings.json in the cache directory.

  • Generating the fixtures: hs3hepdata_generate_fixtures.py. The fits can be generated in parallel, one process per document.

  • Expected-failure list roofit/hs3/test/hs3hepdata_expected_failures.txt, now per check, with the same rules as the list of the conformance suite: a listed check that starts to pass, or an entry that matches no check, fails the test.

  • Docs: the section "Published HS3 documents" in roofit/doc/developers/roofit_hs3.md.

Note

The expected values are the ones that RooFit computed when the fixtures were generated. For now, they are assumed to be correct, which every fixture says. The test is therefore a regression test: the fits tell whether RooFit still finds the same minimum, not whether it is the published one. A comparison with the published results is in progress and will lead to corrections of single fixtures, e.g. where a document needs another starting point or Minuit strategy.

Size: the fixtures are 854 files with 4.7 MB in total (expected.json, expected_fit.json and metadata.json per document, plus the manifest). If that is too much for the repository, they could move to a separate repository like the fixtures of the conformance suite.

Current result

HEPData lists 17 records with 35 HS3 resources, which contain 283 HS3 documents (all written by ROOT 6.30 to 6.37, HS3 0.1.90 and 0.2).

  • Without fits: against the master commit that this branch is based on, the 1310 checks pass, except for 9 expected failures. These belong to four documents:

    Document Fails at Reason
    2905253/workspace-postFit_PTV.json, 2905253/workspace-postFit_STXS.json import two histosys modifiers of the same name in a sample
    2689657/ratio_likelihood.json, 2689657/separate_likelihood.json export a formula b*b among the factors of a sample is written as two normfactor modifiers of the same name

    [RF][HS3] Follow the HS3 0.3 specification fixes (HS3 #154) #24201 fixes all four: with it, all 283 documents pass the round trip. Whichever of the two PRs is merged second has to empty the list.

  • Fits: 279 of the 283 documents have one. The two documents of 2905253 above are not imported, and two documents of 2910761 (likelihood-stxs-smeft, likelihood-dphijj-cp) don't converge (Minuit status −1).

Since the test is not pinned to a state of HEPData, a new record can make it fail without a change in RooFit. That is intended for a test that is run by hand, but it is another reason not to run it in the CI.

Checklist:

  • tested changes locally
  • updated the docs (if necessary)

Local testing (Linux, Python 3.14): the test file was run directly, with PyROOT of an existing ROOT build and the HS3 library compiled from the base of this branch.

  • All 283 documents, offline on a cache, without fits: 1310 checks pass, 9 expected failures, in about 13 minutes with a memory peak of 4.2 GB.
  • Online through hepdata-cli: the search (same 35 resources) and the download of one resource that was removed from the cache.
  • With the HS3 library of [RF][HS3] Follow the HS3 0.3 specification fixes (HS3 #154) #24201: the listed checks are reported as unexpected successes, as intended.
  • The fit fixtures were generated on a computing cluster, one job per document, with a ROOT nightly of the master branch: about 4 CPU hours in total, a median of 21 s and a maximum of 25 minutes per document.

Not tested:

  • The fit check of the test itself against these fixtures.
  • The CMake side: the options test_roofit_hs3hepdata and HS3HEPDATA_FITS, the registration of the test with ROOT_ADD_PYUNITTEST, the check for hepdata_cli and jsonschema, and the timeout of the test have not been configured or run through ctest.
  • A complete download of all resources by the test itself: the cache was filled from an earlier manual download, to not fetch 250 MB from HEPData twice.
  • Other platforms.

AI disclosure: Claude Code (claude-fable-5-2 and claude-opus-5-5) wrote the test, the fixtures, the documentation and this description and ran the tests, at the direction of the author. The commits carry Assisted-by lines. This draft is pending the author's review of the full diff.

🤖 Generated with Claude Code

…list

The HS3 conformance suite was split upstream: the fixtures stay in
HS3TestSuite, while the runner moved to HS3TestSuiteCode with a new
backend plugin interface. Fetch and pin both repositories, at their
current master HEADs:

  HS3TestSuite      9f799b27f63b40d1ba96ff66d14d61787892606d
  HS3TestSuiteCode  36aeb1c67b14dbbd9d9e9b79a4bad561c5e6dac7

Adapt the RooFit backend to the new interface (multi-parameter scans,
likelihood targets, pdf_scan and function_scan checks).

If the repositories can't be reached at configuration time, fail with
a message explaining the alternatives instead of a FetchContent error.

Add roofit/hs3/test/hs3testsuite_expected_failures.txt. Checks listed
there, and checks marked xfail for RooFit in the upstream manifest, run
as expected failures. Unexpected successes and stale entries fail the
test, so the list can only shrink. It starts out empty: all checks pass
except rf209_anaconv, which is already xfail upstream.

Document the option, the pins and the list in roofit_hs3.md.

Assisted-by: claude-code:claude-opus-5-5
The HS3 documents that experiments publish on HEPData are the ones that
users of RooFit actually have. Most were written by older versions of
ROOT, so they show whether RooFit still reads what it once wrote: two
checks that were added recently made three of them unreadable without
any test noticing.

The new test test_hs3_hepdata.py asks HEPData for all records with
resources of the type "HS3", downloads them, and checks for every HS3
document that RooFit can import it, export the result and import that
again. It does not compare values of the models.

The test is enabled with the new option -Dtest_roofit_hs3hepdata=ON,
which is off by default and deliberately not enabled in the CI: HEPData
must not be queried on every build. Downloads are kept in a cache
directory and not repeated; HS3HEPDATA_OFFLINE=1 runs on the cache
alone. HEPData is accessed with its own client, the Python package
hepdata-cli, because it refuses requests of other scripted clients.

Documents that are known to fail are listed in
hs3hepdata_expected_failures.txt, like for the conformance test suite.
Of the 283 documents that HEPData lists at the moment, four are on the
list: two that are not imported (two histosys modifiers of the same
name in a sample) and two that are imported but can't be exported (a
normfactor that is applied twice to a sample).

Assisted-by: claude-code:claude-fable-5-2
@guitargeek

guitargeek commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What a dream, all open likelihoods as regression tests!

I can only encourage you to continue working on this PR 🙂

Note I'm not starting the CI yet, because it's currently overloaded, and this is a draft PR.

@cburgard

Copy link
Copy Markdown
Contributor Author

No worries, this is not at all urgent. I just used this to validate my work on #24201 and figured that since I have the code anyway, I might just as well push it.

One place where I need your decision: do you want this to stay a pure IO test, or do you want the test to also run likelihood scans, which would make it considerably more thorough but also painstakingly slow?

I'm happy to prepare it either way, just say the word.

@guitargeek

Copy link
Copy Markdown
Contributor

I'd strongly prefer at least having the option to also run likelihood scans, because this would make the validation suite also serve as a benchmark suite. Like this I can for example validate RooFit AD performance on real-world analyses or benchmark some ideas I have for Minuit 2 to accelerate likelihood scans by re-using more minimizer state between the points.

cburgard and others added 2 commits November 18, 2026 17:01
…fits

The test with the HS3 documents on HEPData only checked that RooFit
imports and exports them. To make it a validation and a benchmark of
real analyses, it now uses the runner and the fixture layout of the HS3
conformance test suite, and can fit the likelihoods.

* Fixtures in roofit/hs3/test/hs3hepdata, in the layout of the suite but
  without the HS3 documents: those are downloaded from HEPData when the
  test runs, and the manifest pins them by their hashes. hepdata.json
  lists where each document is found.
* With the suite: the import with the pdf and data set of every
  analysis, and 2*(NLL - NLL(reference)) at fixed points along the
  parameters of interest, with the full likelihood of the ModelConfig
  (global observables and external constraints).
* On top: the round trip (export and import again) and, with
  -DHS3HEPDATA_FITS=ON or HS3HEPDATA_FITS=1, a fit per analysis: best
  fit, Hesse and Minos uncertainties of the parameters of interest and
  of the free normalization factors, and a profile likelihood scan
  (expected_fit.json, a format of this test). Every fit names the
  parameter point of the document it starts from and its Minuit
  strategy. The time of every step is written to timings.json.
* hs3hepdata_generate_fixtures.py generates the fixtures; the fits can
  be generated in parallel, one process per document.

The expected values are the ones that RooFit computes: they are assumed
to be correct for now, which every fixture says. A first comparison with
the published results (PDG rounding) is not part of this commit.

Four of the 283 documents have no fit, because the minimization does not
converge. The expected failures are now listed per check.

Assisted-by: claude-code:claude-fable-5-2
…settings

A comparison of the fit fixtures with the published results of the
analyses showed fits that are not the ones the papers quote:

* Background-only documents of 2816994 fitted the signal strengths, which
  the paper's background-only fit holds at zero. A fit can now hold
  parameters at a value ("fixed_parameters"); with them, the normalization
  factors agree with the paper.
* The VLQ documents of 2936806 are signal-plus-background models, while
  the paper quotes the normalization factors of the background-only fit.
  A fixture can now have several fits of the same data set; these get an
  additional fit with the signal strength at zero.
* 2689657: at Minuit strategy 2, Hesse fails and the fixtures had no
  uncertainties at all. At strategy 1, Minos works; the fit check
  compares the Minos intervals, since the covariance matrix is not
  accurate.
* 3130442: Minos had failed on one side of a parameter when the fixture
  was generated; regenerated, the interval agrees with the paper.

The settings of the fits are now read from hs3hepdata/fit_settings.json,
by INSPIRE id or test_id, each with its reason, and written to the
fixtures as before.

Assisted-by: claude-code:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants