Repository navigation
Match each test annotation at most once in compare_annotations - #606
Conversation
tompollard
left a comment
There was a problem hiding this comment.
Thanks, this looks good, and the explanation of the fallback made it easy to follow!
I ran my own comparison with wider windows (50,000 random cases, windows up to 30 samples). main matched a test annotation twice in about 18% of them, which can make fp negative and positive predictivity greater than 1. With this PR there were none, and every case where main was already correct gave identical matches. Record 100 with xqrs_detect is unchanged too.
199295a to
c324ba1
Compare
|
Sorry, another conflict on changes.rst. This will be a continued annoyance so perhaps we need a better way of dealing with the file. |
|
lol yes, will see this |
c324ba1 to
b33b53a
Compare
When a reference annotation lost a contested test annotation to the next reference, it fell back to the previous test annotation, checking only that the previous reference hadn't taken it. If that reference was unmatched, a test annotation matched earlier could be matched again, giving a negative false positive count and a PPV above 1. Track the last matched test annotation instead. Matches are made in increasing order, so the fallback is only used if it comes after it.
b33b53a to
717f332
Compare
|
Moved each changelog entry next to a related one instead of all at the end, so the open PRs shouldn't conflict with each other anymore, whatever order they go in. Longer term, one file per change (like towncrier) would avoid this entirely. Happy to set that up if you want. |
Thanks again @TheNetherWatcher. I don't know towncrier, but it sounds like a good idea to me. |
|
Nice, I'll put together a small PR for it once these are in, so it doesn't clash with them. |
As discussed in #606: each change now adds its own file in `changelog.d/` instead of editing `docs/changes.rst`, so pull requests no longer conflict on the changelog. - Fragments are plain `.rst` files named after the change (for example `rr2ann-fixes.rst`), in the same bold-heading format as now. `changelog.d/README.rst` explains this. - At release time, `uv run towncrier build --version X.Y.Z --date "Mon YYYY"` adds a `Version X.Y.Z (Mon YYYY)` section to `docs/changes.rst` and deletes the fragments. Entries are sorted by file name. - The 25 existing unreleased entries are moved into `changelog.d/` unchanged. I checked that `towncrier build --draft` reproduces them exactly, and ran a real build on a copy: the docs build passes with `-W` both before and after. - `towncrier` is pinned in the `dev` group, like Ruff. AGENTS.md and DEVELOPING.md are updated. One trade-off: the unreleased changes no longer show on the docs site until a release. If you'd like them there, `sphinxcontrib-towncrier` can render a draft section, but that touches `docs/conf.py`, so I'd do it after #616. #619 and #583 add entries to `docs/changes.rst`, so they'd need to move them into a fragment after this is merged (or the other way round, whichever goes in first).
compare_annotationscan match one test annotation to two reference annotations:When a reference annotation loses a contested test annotation to the next reference,
comparefalls back to the previous test annotation. It only checks that the previous reference didn't take it, so if that reference was unmatched, a test annotation matched earlier on can be matched again.Fix: keep track of the last matched test annotation. Matches are made in increasing order, so the fallback is used only if it comes after that one. This replaces the old check (which it covers).
Checking that nothing else changes:
mainwhenevermaindoesn't match a test annotation twice (about 6% of the cases did). With the fix, none do.Tests: the example above, plus a randomized check that matches are unique, in order and within the window, and that
fpis never negative. Both fail onmain.pytest -m "not network"and the docs build pass.This is separate from #566, which handles empty inputs in
_calc_stats.