Skip to content

Support bounded curation and full-source replay for long recordings - #1428

Draft
rossant wants to merge 179 commits into
masterfrom
feature/long-recording-curation
Draft

rossant wants to merge 179 commits into
masterfrom
feature/long-recording-curation

Conversation

@rossant

@rossant rossant commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Curate long recordings using bounded shared-window evidence, then replay saved decisions against every source spike into a new full sorting. The source stays unchanged.

phy prepare-large SOURCE/params.py BOUNDED
phy template-gui BOUNDED/params.py
phy apply-curation SOURCE/params.py BOUNDED CURATED
  • Merge, label, source-amplitude split, Undo/Redo, Save/reopen; preserve SLAY review decisions and metadata through replay and later rounds.
  • Ctrl-click multiple SLAY suggestions into one deduplicated Merge workspace; accept the combined result as one undoable operation.
  • Restore parents in Merge Mode recursively expands recorded merge ancestry in one new action: 1+2→100, then 100+3→101, restores equivalents of 1,2,3 together. Later unrelated edits remain intact; one Undo reverses the whole restoration. Fresh IDs preserve unambiguous history and original identities appear in Parent.
  • Restored components are real editable clusters. Exclude a component, add Similarity candidates and merge using standard Merge operations; leaving Merge Mode keeps them live. The scrollable workspace supports more than 16 components. Quality/custom metadata start unclassified.
  • SLAY returns to its accepted display state when the complete recorded accepted membership is remerged. Partial reassembly remains historical; extra membership is accepted-modified. Historical reviews remain unchanged, including Save/reopen and Undo/Redo.
  • Right-click sampled correlograms to transfer candidates. Table handlers verify the Qt mouse-event class before mouse-specific calls, covering the reported context-menu exception. Secondary Inspect parents (read-only) retains immediate-parent comparison.
  • Full-duration counts/rates and trends, optional prepared indexes, background preparation and CLI progress/ETA/cancellation, bounded workers, faster lookup/histogram passes, parallel hashes and verified independent-copy fallback. Default: one worker.

Updated narrated 58-second, 30 fps demonstration · User guide · Restoration guide · Validation, screenshots and media provenance

Local evidence: both copied real-data walkthroughs passed all 12 checkpoints, including recursive membership, later-edit preservation, exact green SLAY status, one Undo, Save/reopen and unchanged-source hashes. 706 core + 833 application tests passed, 19 skipped, with normal exits; final followups passed 147 ordinary integration tests and 122 Supervisor tests, including rollback-cache failure recovery. Lint, formatting, docs and source/wheel builds pass. Validation uses PyQt5-sip 12.20.1, proposed separately in #1442 for the diagnosed upstream shutdown bug; this branch's lockfile remains unchanged. CI is excluded.

Independent original-label/raw-amplitude verification checked all 469,725,567 assignments in each of two full-source replays (one/four workers), following an amplitude split, two nested merges, recursive restoration, exclusion/addition/remerge and later unrelated edits. All 45 metadata fields / 31,545 cells, counts, reviews, lineage and cumulative provenance survive replay/reprepare. All 140 input files retain hashes and stats. The scientific run is attributed to 03377575; subsequent UI-only proof changes are separately hashed/AST-checked and tested. The final ordinary alias and rollback-display fixes have focused regressions.

Still draft: focused user testing remains. Restoration stops at genuine split or unknown ancestry boundaries; corrupt known ancestry or any missing retained leaf evidence rejects before mutation. New version-4 recursive events preserve the existing version-3 immediate-parent replay meaning. Historical restoration across a fresh preparation remains unavailable; restored live children replay/reprepare normally. Read-only comparison remains limited to 16 immediate parents. Arbitrary bounded lasso/feature splits remain outside scope. The converted fixture's clock/exposure provenance is unverified; this is engineering validation, and the newer 5.24-billion-spike recording is unavailable locally. The event-class guard addresses the logged exception path without establishing the cause of every instability.

@rossant

rossant commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Implemented the latest feedback:

  • Restore parents now expands the whole recorded merge chain in one undoable action, while preserving later edits.
  • Restored IDs remain fresh; Parent shows the original IDs.
  • SLAY returns to green when the complete accepted pair is remerged, including Undo/Redo and save/reopen.
  • Added event-class guards for the Qt exception in the supplied log.

Validated ordinary and bounded real-data workflows, 706 core + 833 application tests, and independent checks of every assignment in two 469.7-million-spike full-source replays. Additional focused regressions cover legacy identities and rollback display state. CI was excluded.

Updated narrated demonstration and validation.

# Conflicts:
#	docs/changelog.md
#	docs/clustering.md
#	phy/cluster/supervisor.py
#	phy/cluster/tests/test_supervisor.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant