Skip to content

Speed up decoding of annotation files in rdann - #627

Merged
tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:perf/rdann-decode
Nov 8, 2026
Merged

tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:perf/rdann-decode

Conversation

@TheNetherWatcher

Copy link
Copy Markdown
Contributor

proc_ann_bytes and its helpers decode an annotation file by indexing the Nx2 NumPy byte array one element at a time (filebytes[bpi, 1], np.int64(...), .astype("i1")), and build each aux note with "".join(chr(c) for c in ...). Each of those makes a NumPy scalar, which is slow in a Python loop.

This converts the byte pairs to a list once with .tolist() and does the rest with plain ints. Signed subtype/num bytes are converted with a small helper, and aux notes are decoded with bytes(...).decode("latin-1"), which gives the same characters as chr() for 0-255. Since no NumPy scalars are involved any more, the decoding no longer depends on NumPy's scalar type promotion rules, which caused the OverflowError in #493 and #522 (already worked around on main with np.int64).

The output is unchanged: I compared every field (sample, symbol, subtype, chan, num, aux_note, fs, label_store, custom labels, and dtypes) for all 15 annotation files in sample-data, with and without sampto, against main.

Benchmark (median, Apple Silicon, alternating runs of main and this branch; the machine was busy, so the absolute numbers are noisy):

file main this PR speedup
rdann 100.atr 4.10 ms 2.01 ms 2.0x
rdann 1003.atr 6.24 ms 3.31 ms 1.9x
rdann 12726.wabp (many aux notes) 16.5 ms 6.4 ms 2.6x
rdann 03700181.gqrsl 2.99 ms 1.64 ms 1.8x
rdann, 100,000 beats with some aux notes (written with wrann) 176 ms 76 ms 2.3x
proc_ann_bytes only, 12726.wabp 22.4 ms 7.6 ms 2.9x

Tests: added a round-trip test for subtype, chan and num values (including negative subtypes). uv run pytest -m "not network" passes, tests/test_annotation.py passes with network tests, ruff format/ruff check are clean, and the docs build without warnings. Added a changelog entry.

Open PRs #491 and #498 also touch annotation.py, but only how the file is opened and read into bytes, not the decoding, so they shouldn't conflict beyond nearby lines.

@tompollard tompollard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good to me. I checked the decoding against main: rdann converts subtype/chan/num with lists_to_int_arrays and samples to int64 afterwards, so the output dtypes are unchanged, _to_signed_char matches .astype("i1"), and latin-1 gives the same characters as chr() for 0–255. Existing tests already cover SKIP pairs (huge.qrs, 12726.anI) and odd-length aux notes (1003.atr), and the new round-trip test covers signed subtypes.

@tompollard
tompollard merged commit 263624c into MIT-LCP:main Nov 8, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants