Skip to content

[RF][HS3] Refactor after the HS3 0.3 changes to reduce lines of code - #24261

Open
guitargeek wants to merge 22 commits into
root-project:masterfrom
guitargeek:hs3-spec-fixes-followup
Open

guitargeek wants to merge 22 commits into
root-project:masterfrom
guitargeek:hs3-spec-fixes-followup

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

This Pull request:

Follows up on #24201, which added a lot of code to RooFitHS3 in a short time, in the spirit of #24231: it reduces roofit/hs3/inc and roofit/hs3/src by about 820 lines (1917 added, 2735 deleted) while keeping the JSON layer independent of RooFit. The tests are unchanged and pass at every commit.

Changes or fixes:

  • Typed combined distributions: a combined_dist is now part of the
    typed document and shares its representation with combined data sets, so the parser, the writer and the
    validation handle both with the same code.
  • Order of the import: a document is validated as it was written, then its variates and constraints are checked, and only then it is adapted and imported. The adaptations (auxiliary channels read as constraint terms, in-line numbers as point data) move from RooJSONFactoryWSTool to the JSON layer.
  • Importers and exporters: all are registered in the same way (static_execute.h is removed), and 13 classes that are read and written field by field are described once for both directions.
  • Simplifications in the tool, the HistFactory code, the parser, the writer, the validation, the upgrade of older documents and JSONNode, one commit per area.
  • Fix: JSONNode::val() kept only six significant digits of a number and truncated large integers, so literal numbers in a document lost precision on import.

What changes for users:

  • A document that fails the checks of its variates or constraints, or whose auxiliary components are inconsistent, is rejected before anything is imported into the workspace.
  • RooFit no longer warns about components without data when it reads back auxiliary measurements that it wrote.
  • A user's own importer for combined_dist is not called for the distributions of a document anymore, and an importer registered for the replacement of a deprecated type key is now also used for the deprecated key.
  • The names of the built-in importers and exporters, as printImporters() and removeImporters() see them, changed.
  • Literal constants of a RooGaussModel are written as numbers.
  • Public headers: RooJSONFactoryWSTool::CombinedData and the operator that streams node.children() into a std::vector are removed (see the release notes); RooJSONFactoryWSTool::channelConstraints() is new.

Assisted-by: claude-code:claude-fable-5-2. The AI wrote the commits and their messages and tested them locally.

A "combined_dist" was a generic object, read and written through its
JSON node, although both layers have to know what it consists of: the
variates analysis read the node in six places, the export of the
auxiliary measurements appended components to the node of a
distribution that was already written, and the import kept a list of
components for the importer to leave out, behind two public functions
of the tool that only this importer called.

It is now a part of the document like the combined data sets, with
which it shares its form: HS3::Combined<Ref_t> has the columns and the
components of both, and the parser, the writer and the validation
handle them with the same code. The checks of the columns and the keys
that the importer did while it built the RooSimultaneous are the ones
of the validation, with the same messages.

* Import: the tool builds the RooSimultaneous from the typed
  distribution (HS3::buildCombinedDist()). The components that are read
  as constraint terms are taken out of the distribution together with
  their data sets and labels, instead of being skipped by the importer.
  isAuxiliaryChannelReadAsConstraint() and otherLikelihoodTerms() of the
  tool, readSelectionColumns() and readSelectionKey() are gone, and
  constraintsOfChannel() and channelWithConstraints() are local to the
  tool. The importer that is registered for "combined_dist" remains for
  a distribution that comes without a document: it parses the node,
  checks it with HS3::validateCombinedDist(), and builds the same way.
  The labels of the keys are then checked against the categories of
  the workspace when the RooSimultaneous is built.
* Export: exportSimultaneous() fills the typed distribution, which is
  written at the end of the export, when it has all of its components.

What changes:
* Whether the likelihoods list the constraints of HistFactory modifiers
  (HS3::unlistedConstraints()) is checked on the document as it was
  written, not after the auxiliary measurements were moved out of the
  combined distributions. Before, the check saw the moved data sets, but
  the unchanged node of the distribution, so that RooFit warned about
  components without data when it read back a document that it wrote
  with config().exportConstraintData. With
  config().importConstraintsAsChannels, the check saw the data of a
  term already moved to its component: a distribution that is a
  component without data and a term with data is now noted in this mode
  as well, and its contributions are counted as the document has them.
  The check follows products, so it now guards against circular ones,
  which are otherwise only found when the distributions are imported.
* A user's own importer for "combined_dist" is not called for the
  distributions of a document anymore.
* The diagnostics of a combined_dist come from the parser and the
  validation, with the path in the document. For one that comes without
  a document, errors start with its name, and a fragment that brings
  its own "distributions" along gets a warning that this component is
  not known, as for a histfactory_dist.

Assisted-by: claude-code:claude-fable-5-2
constraintsOfChannel() took a list of candidate terms and returned the
ones that constrain a channel. Of its two callers, the one that puts the
constraints of a ModelConfig back into the channels only ever passed a
single term, and went on to handle a list of them: a set of the terms
that found a channel, compared with the list by size, and a list of
(product, term) pairs. Inside, the function looped over a list with the
candidate as its only entry.

It is now the predicate isConstraintOfChannel(). The caller decides for
one constraint which products get it, and the combined distribution
collects the constraints of a channel and wraps it itself, which is
what channelWithConstraints() did.

Assisted-by: claude-code:claude-fable-5-2
The tool and the variates analysis looked up data sets, distributions,
domains and likelihoods of a document by name in about twenty places,
each with its own loop, std::find_if() or lambda, often followed by a
std::get_if() for the kind of object.

HS3::findByName() does the lookup in any section, and HS3::find<Body_t>()
gives the object if it is of the wanted kind, e.g.
find<CombinedData>(doc.data, name). Data::numEntries() replaces the three
places that asked whether a data set has entries by looking for
unbinned data without them.

The likelihood of an analysis is not looked up with an error anymore:
the validation has reported an analysis without one.

Assisted-by: claude-code:claude-fable-5-2
…oved

A document was changed before it was validated: the components of
combined distributions that RooFit reads as constraint terms were taken
out of it by a function of the tool, so that the validation saw the
result and not what was written. Whatever needed the document as it
was (which components have data) had to be computed before and
corrected afterwards, in a member of the tool.

The document is now validated as it was read. The function moves to the
typed layer as HS3::readAuxiliaryChannelsAsConstraints(), which is the
only thing besides validate() that may change a ValidatedHS3Document,
and which reports what it did instead of printing it: it only deals
with the document, and uses nothing of RooFit. The components that have
data are those of the document that is built, which the state of the
import takes from it.

What changes: a document whose auxiliary components are not consistent
is rejected, e.g. if the label of such a component is not one of the
categories of its column, or if its data set doesn't exist. Before,
these components were taken out before anything looked at them. The
warnings of the function come after those of the validation.

Assisted-by: claude-code:claude-fable-5-2
A number given in-line in the "data" of a likelihood, or a point data
set without axes, is the value of the single variate of the paired
distribution. The tool worked that out in the middle of the import, and
then carried the result along: the data sets made from numbers in a
list of their own with a map from (likelihood, index of the term) to
their names, and from there in a second map by term for the import of
an analysis, and the variates of the point data sets in a map for the
function that loads data sets.

HS3::resolvePointData() now changes the validated document instead: the
numbers become point data sets, and the point data sets get their axis.
What builds the RooFit objects only sees likelihood terms that refer to
a data set or have none, and point data with an axis.

The errors are the same, but they are reported before anything is
imported, not after the distributions.

Assisted-by: claude-code:claude-fable-5-2
That the terms of the likelihoods have distinct variates, and list the
constraints of their HistFactory modifiers, was checked after the
variables, functions, distributions and data sets of the document were
imported. A document that fails these checks left all of that in the
workspace. Since the constraints have to be checked on the document as
it was written, their check was also split: computed early, before the
auxiliary measurements were moved, and reported late.

Both are checks of the document alone, so they now come right after
the validation, in checkVariates(). Nothing is imported from a document
that fails them.

Assisted-by: claude-code:claude-fable-5-2
Simplifications in the functions of the tool that export a workspace.
What is written doesn't change.

* The export of a ModelConfig listed the channels of its pdf twice, with
  the same loop over the states of a RooSimultaneous and the same
  replacement of the components whose observables are renamed: once for
  their names, and once in constraintsOfChannels() for the pdfs. The
  first loop went over the terms of the likelihood, which only has the
  pdf of the ModelConfig at that point. channelsOf() gives the channels
  once, for both.
* The export of a combined data set returned its name and columns in a
  RooJSONFactoryWSTool::CombinedData, and the tool collected these in a
  list for the export of the ModelConfigs, which need the combined data
  sets with the columns of their pdf. The document that is being
  exported has the same information. The public struct was only used by
  these private functions and is removed, which the release notes now
  mention.
* Both callers of exportVariable() passed `storeConstant = true`, so the
  parameter is removed. exportVariables() only looped over a set for one
  of them, which now has the loop.
* exportData() filled an HS3::Data in three branches and appended it to
  the document in each of them. The weights of unbinned data were
  collected together with a flag that says whether any of them differs
  from one, which is now asked of the collected weights.
* The top-level pdfs and the top-level functions were collected by two
  copies of the same loop: topLevelObjects() does it for both.
* isTopLevel() had two comments, one above the other.
* Flags that were set in a loop or an if, only to be tested right
  after, are replaced by the condition.
* The export of point data recorded the domain of its variable, which
  the export of any data set does for all variables before.

Assisted-by: claude-code:claude-fable-5-2
Simplifications in the functions of the tool that import a document.
What is imported doesn't change.

* The data sets that are loaded from a document were looked up by name
  in five places: with std::find_if(), with std::any_of(), and with
  loops that set a flag or skip all entries but one. findDataset() does
  it for all of them. The import of an analysis also kept the names of
  the distributions of the likelihood next to the list of its terms,
  which have them.
* The specializations of requestImpl() look for an object in the
  workspace, import what the document has under its name, and look
  again. Each of them spelled out the second look after every import,
  with its own early return. They now import what is missing and end
  with the lookup. The order in which the sections of the document are
  tried is the same.
* importJSONElement() set `isVariable` and `isData` from the "type" of
  the element, which it looked up three times, to then branch on them.
  It now branches on the type.
* The import of a document, the import of a single variable and the
  export each defined a local struct whose destructor resets the state
  of the tool. StateScope does it for the three of them.
* toNumber() returned a struct with a value and a flag that says whether
  the value is valid. It now returns a std::optional.

Assisted-by: claude-code:claude-fable-5-2
Simplifications in JSONFactories_HistFactory.cxx. What is read and
written doesn't change, with the one exception that the first point
names.

* isShapeModifierFunction() checked by hand that the variables of a
  ParamHistFunc are the observables of the channel: the same number, and
  each one found by name. That is what RooAbsCollection::equals() does
  (same size, and the same names in any order). The only difference is
  a ParamHistFunc that has the same variable more than once, which no
  longer counts as a function over the observables when the channel
  happens to have as many. Such a function has no defined binning
  anyway.
* splitTopLevelProduct() kept a flag for having seen a `*` at the top
  level, which is the same as having collected a factor, and copied
  each factor into a temporary before stripping its parentheses.
  stripOuterParens() tracked with a flag and two breaks whether the
  outer parentheses enclose the whole expression, and re-checked the
  loop condition inside the loop. That question is now a lambda, and
  the loop strips while its answer is yes. Both functions give the same
  result as before for three million random strings of parentheses,
  operators, names and numbers.
* The struct SampleElements only existed as one local variable of
  buildSample(), the only function that fills and reads it. Its members
  are now local variables of that function, with comments that say what
  they are collected for.
* createPHF() compared the parameters of an existing ParamHistFunc with
  the ones of the modifier in a loop over their addresses, driven by a
  flag. Both lists only hold objects of the workspace, where a name
  stands for one object, so the same parameters are the same names in
  the same order: RooAbsCollection::hasSameLayout(), as for the
  observables of the histograms on export. Checked by hand that
  modifiers of the same name with the parameters in another order, or
  with another parameter, are still rejected.
* The helpers that identify the constant inverse bin widths of
  HistFactory before ROOT 6.26 repeated what their caller had already
  done or made sure of. binVolumes() collected the nodes of the channel
  a second time; it now takes the nodes that readChannel() collects
  anyway. isLegacyBinWidthFactor() looked for the "ORIGNAME:" attribute
  on the RooRealSumPdf and then on the nodes of the channel, of which
  the sum is the first one. It got the workspace as a pointer and
  checked it for null twice, but the workspace of the tool always
  exists: it is a reference now, also in readChannel().
* A factor of a formula that is a product is a parameter of the
  formula, an explicitly written normsys, or something else that
  becomes a formula of its own. The loop over the factors handled the
  first two with early `continue`s; the alternatives are now the
  branches of one if-else chain. This also fixes a typo in a comment.
* buildHistFactoryDist() created the RooDataHists of all samples up
  front and kept them in a vector, in order to read the yields and
  their variances back from them for the relative uncertainties of the
  staterror gammas. The numbers are those of the document: the variance
  of a bin is the square of its error, or for a sample without "errors"
  in a document with implied Poisson uncertainties, its content (which
  is what RooDataHist::weightSquared() falls back to without errors).
  The sums are now taken from the samples directly, and buildSample()
  creates the RooDataHist of its sample where it needs it. The gammas,
  their constraints and the other objects of the workspace are the same
  to the last digit as before for documents of HS3 0.2 and 0.3 with and
  without errors (also negative and zero ones), for each of the
  channel-level constraint types.
* The sums and relative uncertainties from which the errors of the
  samples are reconstructed on export were kept by bin number starting
  at one, with a separate counter while reading the gammas and `- 1`
  wherever the contents of a sample were accessed. They are indexed
  like the contents now, with the position of the gamma among the
  parameters of the modifier.
* exportNamedConstraints() passed an empty parameter name for a gamma
  constraint that has no entry in the parameters of its modifier. That
  can't happen: readChannel() appends the parameter and the constraint
  of each gamma in the same loop, for shapefactor and staterror
  modifiers alike, so the two lists have the same length.
* createNominal() took the range of the global observable
  "nom_<parameter>" as two arguments, and both of its callers passed
  -10 and 10. The range is now part of the function.
* parseOverallModifierFormula() returned a FormulaNormSys whose
  parameter is null if the factor of the formula is no explicitly
  written normsys, and its caller checked that member. It now returns a
  std::optional, so that "nothing recognized" is not a state of the
  struct. The export of formulas with and without such factors, with
  spaces, exponents, duplicates and nested parentheses, gives the same
  JSON as before.

Assisted-by: claude-code:claude-fable-5-2
…annel

The constraint terms that a document adds to a HistFactory channel were
passed to HS3::buildHistFactoryDist() by its caller. For a channel that
the document doesn't interpret, the caller is the importer that is
registered for "histfactory_dist", which doesn't know them, so the tool
recognized that importer in its loop over the importers of a type, and
called HS3::importHistFactoryNode() with the constraints instead.

The builder now asks the tool with channelConstraints(), like it asks
for the HS3 version of the document. The tool treats all importers the
same again, and what importHistFactoryNode() did is the body of the
importer.

toContinuousAxis() leaves HS3Builder.h as well: it was declared there,
but only the tool uses it.

Assisted-by: claude-code:claude-fable-5-2
* The translation of ternary operators into ifelse() had three loops
  that walk over an expression while counting brackets: one to split at
  the commas, one to find the first '?', and one to find the ':' that
  belongs to it. The latter two also skipped the scope operator "::".
  They are now one function, findTopLevel(), which returns the first of
  the given characters that is not enclosed in brackets. Each search
  restarts behind a character that was found outside of brackets, so
  restarting with a depth of zero is the same as carrying the depth
  along. Skipping "::" when looking for commas makes no difference,
  since the skipped character is a colon.
* ternariesToIfElse() and ternaryPartToIfElse() returned whether the
  translation was possible, and the result in an output parameter. They
  now return a std::optional<std::string>, which saves the declarations
  of the outputs at every call and the clearing of the output
  parameter. The three operands of a ternary operator are now all
  translated before the first failure is noticed, instead of stopping
  at the first one that fails. The functions have no side effects, so
  the result is the same.
* extractArguments() was a state machine over the characters, with the
  start index of the current identifier doubling as the state and the
  handling of an identifier at the end of the string written out a
  second time. It now reads an identifier as a whole where one starts,
  and then decides whether it is an argument. Its documentation said
  that arguments contain no digits and are returned as a vector; they
  may contain digits after the first letter and are returned as a set.

No change in behaviour. For findTopLevel() and extractArguments(), the
old and the new functions agree on 3 million random expressions.

Assisted-by: claude-code:claude-fable-5-2
Simplifications in JSONFactories_RooFitCore.cxx. What is read and
written doesn't change. What RooFit::JSONIO::printImporters() lists
does, as the first two points say.

* An importer returns false to decline an object, so that the next
  importer that is registered for the type key is tried. Only the two
  importers for "step" and the one for histograms do that; all others
  ended in "return true". These are now void functions, and the adapter
  to the RooFit::JSONIO::Importer interface returns true for them, so
  that the return type of an import function says whether it can
  decline. The adapter is a template of the function type for this, so
  the names of its instances, as printImporters() prints them, now
  include that type.
* A deprecated type key had two importers: a DeprecatedKeyImporter,
  which warned the first time and declined, and after it the importer
  of the replacement. The class had a constructor and three members for
  what is a warning in front of a function call. Since the adapter for
  import functions takes any callable, the two are now one importer: a
  closure that warns the first time that the key is read and then calls
  the import function. The warning and the way the object is read are
  the same; printImporters() now lists one importer for a deprecated
  key instead of two.
* The includes of RooCategory, RooSimultaneous and RooSuperCategory are
  removed: the importer of a "combined_dist" that comes without a
  document doesn't need them, since the tool builds the
  RooSimultaneous.
* importAddPdf() counted the summands and coefficients of a
  mixture_dist in the branch for documents with an "extended" flag,
  again in the branch for documents without one, and a third time,
  spelled out with has_child() and num_children(), for the warning
  about mixtures without coefficients. It also compared the HS3 version
  twice. The counts and the version check are now taken once, before
  the branches. The keys were read on every path before, so the report
  of unknown components is not affected.
* The struct LognormalParameters and the function lognormalParameters()
  translated the "mu" and "sigma" of a lognormal_dist into what to
  create the RooLognormal from. Their only user is importLogNormal(),
  also for the components of array distributions, which are read
  through the importer of their type. It now does this itself, in the
  same way as importExponential() does for its coefficient.
* A pdf as a component of an array distribution was described by two
  structs: ArrayComponent (type, observable and fields), internal to
  JSONFactories_RooFitCore.cxx and returned by arrayComponentOf(), and
  ArrayComponentInfo (type and observable), declared in JSONIOUtils.h
  and returned by arrayComponentInfo(), which called arrayComponentOf()
  and copied the two members. ArrayComponentInfo now has the fields as
  well, and arrayComponentInfo() is the function that was
  arrayComponentOf(), with its optional `tool` that writes the
  functions that a field names. It moved to the end of the file, next
  to writeArrayDist(), because it is declared in the header and can't
  be in the anonymous namespace. The callers in the other files only
  use the type and the observable and don't change.
* arrayComponentName() appended "_constraint" to the name of an
  observable and was only called by importArrayDist(), whose comment
  already says that a component is named after its observable.
* importParamHistFunc() had two copies of the code that checks the
  order of the parameters and creates the ParamHistFunc: one for a
  function without "axes", with the variables themselves, and one with
  the binning variables that are read from the "axes". The only
  difference is the list of variables, which is now chosen first. The
  warning about missing axes is put together as a string instead of
  with a std::stringstream; its text is the same, including the line
  break at its end.
* The exporters of RooFormulaVar and RooGenericPdf, and the one of
  RooTFnBinding, each had their own loop that refuses expressions that
  refer to a pdf, with the same error message and the same comment
  about the rule of HS3. It is now one function. For the formula
  classes it takes dependents(), which is the list that getParameter()
  indexes.
* CrystalBall::hasAny() was a hand-written loop that returns at the
  first key that the node has, which is what std::any_of does.

Assisted-by: claude-code:claude-fable-5-2
…ld once

Thirteen classes had an importer that requests one object or one list
of objects per key and passes them to the constructor, and an exporter
that writes the same keys from the getters: RooAddition, RooArgusBG,
RooBifurGauss, RooCBShape, RooChebychev, RooEffProd, RooGamma,
RooGaussModel, RooPowerSum, RooProdPdf, RooProjectedPdf, RooRealSumFunc
and RooTruthModel. The keys and their order were written down twice per
class, in two functions that had to agree.

Such a class is now registered with registerFields(), which takes one
field per constructor argument: object(key, getter) or list(key,
getter), with the type to request the objects as if it is not
RooAbsReal. The importer and the exporter follow from the fields. This
is plain C++: the getters are pointers to member functions, and the
compiler checks that the constructor of the class takes what the fields
request. Classes that need anything else keep their functions.

The fields are requested in the order of the former importers, which
the braces of the constructor call guarantee, with the same types and
the same error messages, and the exporters write the same references.
For this, the adapter for export functions takes any callable, like the
one for import functions.

Two of the classes shared their functions with something that is not
read field by field:

* importRealSum() and exportRealSum() were templates for RooRealSumFunc
  and RooRealSumPdf, with an "if constexpr" each for the "extended" flag
  that only the pdf has. They are plain functions for RooRealSumPdf
  now.
* importCrystalBall() read the single-sided Crystal Ball, and the
  double-sided one that "crystalball_dist" also was before HS3 0.3.
  What remains of it is the importer for the documents before HS3 0.3,
  which is registered in front and declines a "crystalball_dist"
  without the fields of two tails, in the same way as the two importers
  of "step" share their type key. The same objects are created from the
  same fields, with the same warning and error messages.
  RooFit::JSONIO::printImporters() lists two importers for
  "crystalball_dist" now.

What changes:

* The exporter of RooGaussModel wrote the names of its mean and sigma.
  For a literal constant, e.g. RooConst(0), that is the string "0",
  which refers to an object that the document doesn't have. They are
  now written with writeReference() like the references of most other
  classes, so a literal constant becomes a number. Nothing changes for
  other parameters. RooFit can read such a document as little as
  before: the importer requests the mean and sigma as variables.
* The registration of a deprecated type key named its replacement, for
  the warning, and the import function of the replacement, which had to
  be the one that the replacement is registered with. Now it only names
  the replacement, and the importer of the deprecated key tries the
  importers that are registered for the replacement when it reads an
  object. An importer that a user registers for the replacement, e.g.
  for "argus_dist", is therefore also used for the deprecated key
  "ARGUS_dist", in the order of the importers of the replacement.
  Before, the deprecated key was always read by the built-in function.
  This is what "read like its replacement" should mean, and the
  built-in importers behave as before.

Assisted-by: claude-code:claude-fable-5-2
* static_execute.h defined STATIC_EXECUTE(), which wraps a function in
  a struct with a static member whose constructor calls it, to run code
  when the library is loaded. It was used twice, each time with a
  lambda. The initializer of a variable at namespace scope does the
  same, and the header is removed.
* The importers and exporters of PiecewiseInterpolation and
  FlexibleInterpVar were four classes, each with the boilerplate of the
  Importer or Exporter interface around one function: a key() with a
  static string, a cast of the argument, and a `return true`. The
  functions of JSONFactories_RooFitCore.cxx are registered through
  adapters that take care of this. The adapters move to JSONIOUtils.h,
  so that both files can use them, and the four classes become
  functions that are registered with registerType(). Nothing else is
  registered for these two types and classes, so the order in which
  their importers and exporters are tried doesn't change.
  HistFactoryImporter and HistFactoryStreamer stay classes: the first
  is recognized by its type, and the second declines the products and
  sums that are no HistFactory channels.
* An exporter can be registered by the name of its class, because the
  TClass might not exist yet when a library is loaded. These exporters
  were kept in a second map by class name, in vectors with a nullptr
  separator between the ones with top priority and the others, which
  were then spliced around the exporters registered so far. They are
  now kept in a list in the order of their registration, and
  exporters() registers them with their TClass in that order. This
  gives the same order of exporters for a class as before: each one
  goes to the front or the back of what was registered before it. The
  only difference is for two different names of the same class (e.g. a
  typedef): their exporters used to be added in the alphabetical order
  of the names, and are now added in the order of their registration,
  like all others.

Assisted-by: claude-code:claude-fable-5-2
* The steps that upgrade older documents all visit the structs in some
  array of the tree: the modifiers of the samples of a distribution,
  the axes of the data sets, the axes of the objects of three sections,
  the distributions. Each one spelled out the same checks (is there
  such a component, is it an array, is the element a struct), the
  index, and the cursors that give the paths in diagnostics. The new
  forEachStruct() does this in one place, and the steps only say what
  they do with an element. The same elements are visited with the same
  cursors and indices as before, so the upgraded documents and the
  diagnostics don't change. warnBoundsWithEdges() now takes the tree as
  non-const like the other steps, because the helper is shared with the
  steps that change it; it still only reads.
* upgradeMissingInterpolation() took seven arguments to write the
  implicit interpolation code of older documents into a modifier that
  has none, and then called upgradeInterpolationCode() a second time to
  translate the code that it just wrote. The modifier now gets the
  implicit code right before the one call that translates integer
  codes, with the same condition and the same warning.

Assisted-by: claude-code:claude-fable-5-2
Simplifications in HS3Writer.cxx. What is written doesn't change.

* The writer had two helpers for arrays: writeRefs() for a list of
  references, and writeList() for the lists of the interpolation
  functions, which wrote its elements with writeValue(). writeValue()
  now also has an overload for a reference and for a list of anything
  it can write, like readValue() in the parser, and replaces the other
  two.
* writeRooFitInternal() had seven blocks of the same shape, one for
  each map of the RooFit-internal section: if the map is not empty,
  create the component and write each value under its name.
  writeInternal() does this for any map, with a writeValue() overload
  for each type of value. The function then lists the components and
  their keys just like readMisc() does in the parser.
* Eight components were written with the same three lines: if the
  std::optional has a value, write it under its key. writeOptional()
  says that in one line, as the Reader of the parser does for reading.
* Every branch of writeModifier() started by calling appendModifier()
  with its own name and with the string of its type. The types of the
  modifiers were already listed in typeOf(), which HS3Document.cxx used
  for diagnostics. typeOf() is now declared in HS3Document.h, and
  writeModifier() writes the name and the type once before it visits
  the modifier, so that the branches only write what is specific to
  them. A custom modifier has nothing else.
* writeNumber() was declared in HS3Document.h, but only HS3Writer.cxx
  uses it: the RooFit layer writes its numbers with writeContents() or
  directly. It is local to the writer now.
* The writer wrote misc.ROOT_internal.combined_distributions and
  combined_datasets, and the "combined_data_name" of a ModelConfig.
  These are how older versions of RooFit wrote a RooSimultaneous and
  its data: only the parser fills them, and the export never does, so
  the writer never had anything to write there.

Assisted-by: claude-code:claude-fable-5-2
* The components of misc.ROOT_internal were read with readMap(), which
  handled a missing or null component, checked for a struct and read
  each value. The "dict" of the attributes spelled out the same loop
  again. A readValue() overload for std::map now reads a struct with
  one value per name, next to the one for std::vector, and the Reader
  takes care of the rest: Reader::optional() for the components of
  ROOT_internal, which has exactly the semantics of readMap(), and
  Reader::operator() for the "dict". The diagnostics are the same. The
  only difference is that after a "dict" that is not a struct, which is
  an error, the other components of the attributes are still read and
  can give errors of their own.
* The describe() functions of the normsys and histosys modifiers were
  identical: they have the same components, and only the type of the
  variations "hi" and "lo" differs, which the Reader resolves by
  itself. describeSys() is now a template that is used for both.
* readValue() for an interpolation first checked with the JSON node
  that "type", "in" and "out" exist, and at the end created a Reader
  and asked it for the same three components again, only to have them
  count as read so that the other ones are reported as unknown. The
  Reader is now created up front and used for the check itself. It
  reports unknown components only if there was no error after it was
  created, which is when the function used to get to the end.
* ParseOptions::requireMetadata was true by default, and nothing ever
  set it to false: fragments of a document are read with parseData()
  and the other functions for single objects, not with parseDocument().
  A document without "hs3_version" in its metadata is therefore always
  an error, and the option is removed.
* readDistribution() read a HistFactory distribution into a local
  variable and moved it into the variant afterwards. It is now read
  into the variant directly, as describe() does for the kinds of data
  and of modifiers.
* When the validation compares the ranges of the data axes with the
  domains of an analysis, it took a missing "min" or "max" of a domain
  axis from its "edges". There is no such axis in a document: the
  parser already fills in the bounds of an axis from its edges, and the
  export only gives an axis a binning if it has both bounds
  (toDomainAxis() in RooJSONFactoryWSTool.cxx). These are the only two
  places that create the axes of a domain with a binning.
* That check of the data axes against the domains repeated the same
  call for the axes of point data, and for the continuous and the
  binned axes of the other kinds of data. It is now one generic lambda
  that takes any axis with a range.
* checkHistFactory() defined a lambda to check that the constraint
  distribution of a modifier exists, ninety lines before the only place
  that calls it. The two lines are now in checkSingleParameter(), next
  to the other checks of a modifier with a single parameter.

Assisted-by: claude-code:claude-fable-5-2
* set_map() and set_seq() were copies of each other that differed in
  the type of the container and in one word of the error message. They
  now share setContainerType(), which also absorbs
  isResettingPossible(), a fourteen-line way of saying "null or an
  empty string". The comment now says where the empty strings come
  from. The behaviour and the error messages are the same.
* find() and get() made a reference to *this before using it, and the
  variadic get() called set_map() on the next node right before calling
  get() on it, which does the same. They are now the single expressions
  that they stand for, and say in a comment what they do. Two "public:"
  labels in a section of the class that is already public go as well.
* JSONInterface.h had two operators to fill a std::vector: one from a
  node, which checks that the node is an array, and one from the view
  of the children of a const node, which the first one called and
  nothing else used: not in ROOT, and not in the importers and
  exporters of CMS combine. The loop is now in the operator for the
  node. This removes `vector << node.children()` from a public header
  (the namespace is RooFit::Detail). `vector << node` does the same
  with the check, which the release notes now say.

Assisted-by: claude-code:claude-fable-5-2
…udes

* The tool passed itself to writeArrayDist() together with the map of
  the observables that are recorded for constraint terms, which the
  function can ask the tool for with constraintObservable().
* The documentation of RooJSONFactoryWSTool described a function
  writedoc(), which doesn't exist. A few comments of the tool said what
  the next line does in other words ("collect all datasets"), or
  described what the code did before it was changed ("if this is a
  toplevel pdf, also create a modelConfig for it" at the start of the
  import of an analysis).
* The comment at the top of HS3Writer.cxx named testHS3Document.cxx,
  which was never part of the repository. The round trips are tested
  through the RooJSONFactoryWSTool in testRooFitHS3.cxx.
* Nothing uses std::numeric_limits in HS3Writer.cxx, std::exception in
  HS3Parser.cxx, std::string_view in HS3Schema.h or RooAbsPdf in
  JSONIO.cxx anymore.

Assisted-by: claude-code:claude-fable-5-2
readAuxiliaryChannelsAsConstraints() copies a data set that would
otherwise be a component of two combined data sets. It keeps pointers
into the data sets of the document while it does, so it reserved room
for the copies up front, and silently made no copy once that room was
used up. The copies are now collected and added after the loop.

Assisted-by: claude-code:claude-fable-5-2
…tion

Two checks of a HistFactory distribution were made while its RooFit
objects were built, although they only look at the document: a
modifier with a single parameter can only have the constraint type
"Gauss", and a staterror modifier can't have "LogNormal". They are now
part of the validation, with the same messages, so that such a
document is rejected with all its other problems before anything is
created.

Assisted-by: claude-code:claude-fable-5-2
JSONNode::val() wrote a floating-point number with the default precision
of the streams, which is six significant digits, and it cast an integer
to a 32-bit one. The importers read what a key refers to as a string,
because it is usually the name of an object, and the tool takes a string
that spells a number for a literal constant. So the numbers of a
document arrived in the workspace like this:

    "sigma": 1.23456789   ->  1.23457
    "mean": 123456789.5   ->  123457000
    "mean": 10000000000   ->  1410065408

Now val() uses as many digits as it takes for the string to read back as
the same number. It starts with the six digits, so the string of a
number that six digits spell exactly is the same as before, and nothing
changes for the code that compares such strings or prints them.

This also covers the importers outside of ROOT, which had to avoid
val() for numbers.

The names of the literal constants stay as they are, because
RooFit::RooConst() gives them a name with six digits. Two different
numbers that agree in these digits are therefore still one object in
the workspace, as they are in a workspace that is filled without HS3.

Assisted-by: claude-code:claude-fable-5-2
@github-actions

Copy link
Copy Markdown

Test Results

    22 files      22 suites   3d 14h 0m 37s ⏱️
 3 946 tests  3 946 ✅ 0 💤 0 ❌
77 321 runs  77 321 ✅ 0 💤 0 ❌

Results for commit c0d2618.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant