Repository navigation
Conversation
…list The HS3 conformance suite was split upstream: the fixtures stay in HS3TestSuite, while the runner moved to HS3TestSuiteCode with a new backend plugin interface. Fetch and pin both repositories, at their current master HEADs: HS3TestSuite 9f799b27f63b40d1ba96ff66d14d61787892606d HS3TestSuiteCode 36aeb1c67b14dbbd9d9e9b79a4bad561c5e6dac7 Adapt the RooFit backend to the new interface (multi-parameter scans, likelihood targets, pdf_scan and function_scan checks). If the repositories can't be reached at configuration time, fail with a message explaining the alternatives instead of a FetchContent error. Add roofit/hs3/test/hs3testsuite_expected_failures.txt. Checks listed there, and checks marked xfail for RooFit in the upstream manifest, run as expected failures. Unexpected successes and stale entries fail the test, so the list can only shrink. It starts out empty: all checks pass except rf209_anaconv, which is already xfail upstream. Document the option, the pins and the list in roofit_hs3.md. Assisted-by: claude-code:claude-opus-5-5
The HS3 documents that experiments publish on HEPData are the ones that users of RooFit actually have. Most were written by older versions of ROOT, so they show whether RooFit still reads what it once wrote: two checks that were added recently made three of them unreadable without any test noticing. The new test test_hs3_hepdata.py asks HEPData for all records with resources of the type "HS3", downloads them, and checks for every HS3 document that RooFit can import it, export the result and import that again. It does not compare values of the models. The test is enabled with the new option -Dtest_roofit_hs3hepdata=ON, which is off by default and deliberately not enabled in the CI: HEPData must not be queried on every build. Downloads are kept in a cache directory and not repeated; HS3HEPDATA_OFFLINE=1 runs on the cache alone. HEPData is accessed with its own client, the Python package hepdata-cli, because it refuses requests of other scripted clients. Documents that are known to fail are listed in hs3hepdata_expected_failures.txt, like for the conformance test suite. Of the 283 documents that HEPData lists at the moment, four are on the list: two that are not imported (two histosys modifiers of the same name in a sample) and two that are imported but can't be exported (a normfactor that is applied twice to a sample). Assisted-by: claude-code:claude-fable-5-2
|
What a dream, all open likelihoods as regression tests! I can only encourage you to continue working on this PR 🙂 Note I'm not starting the CI yet, because it's currently overloaded, and this is a draft PR. |
|
No worries, this is not at all urgent. I just used this to validate my work on #24201 and figured that since I have the code anyway, I might just as well push it. One place where I need your decision: do you want this to stay a pure IO test, or do you want the test to also run likelihood scans, which would make it considerably more thorough but also painstakingly slow? I'm happy to prepare it either way, just say the word. |
|
I'd strongly prefer at least having the option to also run likelihood scans, because this would make the validation suite also serve as a benchmark suite. Like this I can for example validate RooFit AD performance on real-world analyses or benchmark some ideas I have for Minuit 2 to accelerate likelihood scans by re-using more minimizer state between the points. |
…fits The test with the HS3 documents on HEPData only checked that RooFit imports and exports them. To make it a validation and a benchmark of real analyses, it now uses the runner and the fixture layout of the HS3 conformance test suite, and can fit the likelihoods. * Fixtures in roofit/hs3/test/hs3hepdata, in the layout of the suite but without the HS3 documents: those are downloaded from HEPData when the test runs, and the manifest pins them by their hashes. hepdata.json lists where each document is found. * With the suite: the import with the pdf and data set of every analysis, and 2*(NLL - NLL(reference)) at fixed points along the parameters of interest, with the full likelihood of the ModelConfig (global observables and external constraints). * On top: the round trip (export and import again) and, with -DHS3HEPDATA_FITS=ON or HS3HEPDATA_FITS=1, a fit per analysis: best fit, Hesse and Minos uncertainties of the parameters of interest and of the free normalization factors, and a profile likelihood scan (expected_fit.json, a format of this test). Every fit names the parameter point of the document it starts from and its Minuit strategy. The time of every step is written to timings.json. * hs3hepdata_generate_fixtures.py generates the fixtures; the fits can be generated in parallel, one process per document. The expected values are the ones that RooFit computes: they are assumed to be correct for now, which every fixture says. A first comparison with the published results (PDG rounding) is not part of this commit. Four of the 283 documents have no fit, because the minimization does not converge. The expected failures are now listed per check. Assisted-by: claude-code:claude-fable-5-2
…settings
A comparison of the fit fixtures with the published results of the
analyses showed fits that are not the ones the papers quote:
* Background-only documents of 2816994 fitted the signal strengths, which
the paper's background-only fit holds at zero. A fit can now hold
parameters at a value ("fixed_parameters"); with them, the normalization
factors agree with the paper.
* The VLQ documents of 2936806 are signal-plus-background models, while
the paper quotes the normalization factors of the background-only fit.
A fixture can now have several fits of the same data set; these get an
additional fit with the signal strength at zero.
* 2689657: at Minuit strategy 2, Hesse fails and the fixtures had no
uncertainties at all. At strategy 1, Minos works; the fit check
compares the Minos intervals, since the covariance matrix is not
accurate.
* 3130442: Minos had failed on one side of a parameter when the fixture
was generated; regenerated, the interval agrees with the paper.
The settings of the fits are now read from hs3hepdata/fit_settings.json,
by INSPIRE id or test_id, each with its reason, and written to the
fixtures as before.
Assisted-by: claude-code:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This Pull request:
Adds an elective test that runs the HS3 documents published on HEPData through RooFit: import, likelihood values, round trip and, optionally, fits. It serves as a regression test of the HS3 import with real analyses and, with the fits, as a benchmark.
It only extends the tests; it changes no library code.
Important
Stacked on #23839 (the HS3 conformance test suite): the first commit of this PR is the commit of #23839, the last two commits are new. Draft until #23839 is merged; it then needs a rebase.
Changes or fixes:
The documents on HEPData are the ones that users of RooFit actually have, and most were written by older versions of ROOT. They show whether RooFit still reads what it once wrote, which no other test does: recently added checks made some of them unreadable without any test noticing (see #24201). Being real analyses, they are also realistic benchmarks.
First commit: the elective test.
roofit/hs3/test/test_hs3_hepdata.py. It asks HEPData for all records with resources of the type "HS3" (analysis:hs3) and downloads them.-Dtest_roofit_hs3hepdata=ON(default OFF, requires-Dtesting=ONand-Dpyroot=ON). It is deliberately not added to the CI configuration: HEPData must not be queried on every build.HS3HEPDATA_CACHE, by default in the build directory) and not downloaded again.HS3HEPDATA_OFFLINE=1runs on the cache without contacting HEPData;HS3HEPDATA_SELECT=<regex>restricts the test to matching documents.hepdata-cli. HEPData answers requests of generic scripted clients with 403, so the test uses HEPData's own Python client (a new Python dependency of this test only). Its publicdownloadcommand only covers tables, so the resource files are fetched with the request helper of the package (hepdata_cli.resilient_requests), which is not a documented interface.Second commit: the documents run through the HS3 test suite, with fits (asked for in the review, to make the test usable as a benchmark).
Fixtures in
roofit/hs3/test/hs3hepdata, in the layout of the HS3 conformance suite but without the HS3 documents: those are downloaded when the test runs, andmanifest.jsonpins them by their hashes.hepdata.jsonlists where each document is found.Checks of the suite: the import with the pdf and data set of every analysis, and 2·(NLL − NLL(reference)) at fixed points along the parameters of interest, with the full likelihood of the ModelConfig (global observables and external constraints).
On top: the round trip (export and import again) and, with
-DHS3HEPDATA_FITS=ONorHS3HEPDATA_FITS=1, a fit per analysis:expected_fit.json, a format of this test, since the suite has no check for fits.Every fit names the parameter point of the document it starts from and its Minuit strategy.
HS3HEPDATA_EVAL_BACKENDselects the evaluation backend, e.g.codegen.Timings. The time of every step (import, round trip, creating the likelihood, minimization, Hesse, Minos, profile scans) is written to
timings.jsonin the cache directory.Generating the fixtures:
hs3hepdata_generate_fixtures.py. The fits can be generated in parallel, one process per document.Expected-failure list
roofit/hs3/test/hs3hepdata_expected_failures.txt, now per check, with the same rules as the list of the conformance suite: a listed check that starts to pass, or an entry that matches no check, fails the test.Docs: the section "Published HS3 documents" in
roofit/doc/developers/roofit_hs3.md.Note
The expected values are the ones that RooFit computed when the fixtures were generated. For now, they are assumed to be correct, which every fixture says. The test is therefore a regression test: the fits tell whether RooFit still finds the same minimum, not whether it is the published one. A comparison with the published results is in progress and will lead to corrections of single fixtures, e.g. where a document needs another starting point or Minuit strategy.
Size: the fixtures are 854 files with 4.7 MB in total (
expected.json,expected_fit.jsonandmetadata.jsonper document, plus the manifest). If that is too much for the repository, they could move to a separate repository like the fixtures of the conformance suite.Current result
HEPData lists 17 records with 35 HS3 resources, which contain 283 HS3 documents (all written by ROOT 6.30 to 6.37, HS3 0.1.90 and 0.2).
Without fits: against the master commit that this branch is based on, the 1310 checks pass, except for 9 expected failures. These belong to four documents:
2905253/workspace-postFit_PTV.json,2905253/workspace-postFit_STXS.jsonhistosysmodifiers of the same name in a sample2689657/ratio_likelihood.json,2689657/separate_likelihood.jsonb*bamong the factors of a sample is written as twonormfactormodifiers of the same name[RF][HS3] Follow the HS3 0.3 specification fixes (HS3 #154) #24201 fixes all four: with it, all 283 documents pass the round trip. Whichever of the two PRs is merged second has to empty the list.
Fits: 279 of the 283 documents have one. The two documents of 2905253 above are not imported, and two documents of 2910761 (
likelihood-stxs-smeft,likelihood-dphijj-cp) don't converge (Minuit status −1).Since the test is not pinned to a state of HEPData, a new record can make it fail without a change in RooFit. That is intended for a test that is run by hand, but it is another reason not to run it in the CI.
Checklist:
Local testing (Linux, Python 3.14): the test file was run directly, with PyROOT of an existing ROOT build and the HS3 library compiled from the base of this branch.
hepdata-cli: the search (same 35 resources) and the download of one resource that was removed from the cache.Not tested:
test_roofit_hs3hepdataandHS3HEPDATA_FITS, the registration of the test withROOT_ADD_PYUNITTEST, the check forhepdata_cliandjsonschema, and the timeout of the test have not been configured or run throughctest.AI disclosure: Claude Code (claude-fable-5-2 and claude-opus-5-5) wrote the test, the fixtures, the documentation and this description and ran the tests, at the direction of the author. The commits carry
Assisted-bylines. This draft is pending the author's review of the full diff.🤖 Generated with Claude Code