Skip to content

Match each test annotation at most once in compare_annotations - #606

Merged
tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:comparitor-double-match
Oct 31, 2026
Merged

tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:comparitor-double-match

Conversation

@TheNetherWatcher

Copy link
Copy Markdown
Contributor

compare_annotations can match one test annotation to two reference annotations:

c = processing.compare_annotations(np.array([0, 2, 4, 5]), np.array([1, 5]), window_width=6)
c.matching_sample_nums  # [0, -1, 0, 1]: test annotation 0 is used twice
c.fp, c.positive_predictivity  # -1, 1.5

When a reference annotation loses a contested test annotation to the next reference, compare falls back to the previous test annotation. It only checks that the previous reference didn't take it, so if that reference was unmatched, a test annotation matched earlier on can be matched again.

Fix: keep track of the last matched test annotation. Matches are made in increasing order, so the fallback is used only if it comes after that one. This replaces the old check (which it covers).

Checking that nothing else changes:

  • On 200,000 random small cases, the new code gives exactly the same matches as main whenever main doesn't match a test annotation twice (about 6% of the cases did). With the fix, none do.
  • On record 100 (gqrs and xqrs against the reference beats, two window sizes), the matches are identical.

Tests: the example above, plus a randomized check that matches are unique, in order and within the window, and that fp is never negative. Both fail on main. pytest -m "not network" and the docs build pass.

This is separate from #566, which handles empty inputs in _calc_stats.

@tompollard tompollard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good, and the explanation of the fallback made it easy to follow!

I ran my own comparison with wider windows (50,000 random cases, windows up to 30 samples). main matched a test annotation twice in about 18% of them, which can make fp negative and positive predictivity greater than 1. With this PR there were none, and every case where main was already correct gave identical matches. Record 100 with xqrs_detect is unchanged too.

@tompollard

Copy link
Copy Markdown
Member

Sorry, another conflict on changes.rst. This will be a continued annoyance so perhaps we need a better way of dealing with the file.

@TheNetherWatcher

TheNetherWatcher commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

lol yes, will see this

When a reference annotation lost a contested test annotation to the next
reference, it fell back to the previous test annotation, checking only
that the previous reference hadn't taken it. If that reference was
unmatched, a test annotation matched earlier could be matched again,
giving a negative false positive count and a PPV above 1.

Track the last matched test annotation instead. Matches are made in
increasing order, so the fallback is only used if it comes after it.
@TheNetherWatcher

TheNetherWatcher commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Moved each changelog entry next to a related one instead of all at the end, so the open PRs shouldn't conflict with each other anymore, whatever order they go in.

Longer term, one file per change (like towncrier) would avoid this entirely. Happy to set that up if you want.

@tompollard

Copy link
Copy Markdown
Member

For the longer term, one option is a file per change (for example docs/changes/606.rst), combined into changes.rst at release time, as towncrier does.

Thanks again @TheNetherWatcher. I don't know towncrier, but it sounds like a good idea to me.

@TheNetherWatcher

Copy link
Copy Markdown
Contributor Author

Nice, I'll put together a small PR for it once these are in, so it doesn't clash with them.

@tompollard
tompollard merged commit adf00ae into MIT-LCP:main Oct 31, 2026
18 checks passed
@TheNetherWatcher
TheNetherWatcher deleted the comparitor-double-match branch November 1, 2026 06:12
tompollard added a commit that referenced this pull request Nov 1, 2026
As discussed in #606: each change now adds its own file in
`changelog.d/` instead of editing `docs/changes.rst`, so pull requests
no longer conflict on the changelog.

- Fragments are plain `.rst` files named after the change (for example
`rr2ann-fixes.rst`), in the same bold-heading format as now.
`changelog.d/README.rst` explains this.
- At release time, `uv run towncrier build --version X.Y.Z --date "Mon
YYYY"` adds a `Version X.Y.Z (Mon YYYY)` section to `docs/changes.rst`
and deletes the fragments. Entries are sorted by file name.
- The 25 existing unreleased entries are moved into `changelog.d/`
unchanged. I checked that `towncrier build --draft` reproduces them
exactly, and ran a real build on a copy: the docs build passes with `-W`
both before and after.
- `towncrier` is pinned in the `dev` group, like Ruff. AGENTS.md and
DEVELOPING.md are updated.

One trade-off: the unreleased changes no longer show on the docs site
until a release. If you'd like them there, `sphinxcontrib-towncrier` can
render a draft section, but that touches `docs/conf.py`, so I'd do it
after #616.

#619 and #583 add entries to `docs/changes.rst`, so they'd need to move
them into a fragment after this is merged (or the other way round,
whichever goes in first).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants