Merge workspace, amplitude threshold splits, and curation workflow improvements

[rossant]

Draft: this is an in-progress phy 2.2 feature branch shared for manual testing. Please do not merge until the workflows below have received release acceptance.

Summary

This PR introduces explicit merge workspaces, AIND/SpikeInterface merge-proposition review, amplitude-threshold splitting, and the selection/history model needed to make those curation operations reversible and internally consistent. It also includes related GUI, performance, shutdown, documentation, and installation improvements.

Manual Merge workflow

• Press V with at least one Cluster View row selected to enter Merge mode.
• Merge View stages the blue reference first, followed by other Cluster View selections and selected Similarity candidates.
• Transfer rows between Merge and Similarity views with Control-right-click or drag-and-drop. Drag within Merge View to reorder staged units.
• Cluster View is dimmed and read-only but remains scrollable. Similarity exploration, sorting, filtering, Control+Space, and multi-selection remain available.
• Scientific views use Merge View order first, then selected Similarity rows in visible table order. Existing colors remain fixed throughout the workspace.
• Press G to merge all staged and selected candidates. After a manual merge, Merge mode closes and the result becomes the sole Cluster View selection, ready for quality assignment or explicit entry into another merge.
• Press V, choose Cancel Merge Mode, or close Merge View to cancel and restore the exact entry workspace, including selections, sort/filter state, ordering, and scroll context.
• Merge View is reused rather than destroyed between sessions. Its dock area, tab relationship, floating geometry, size, and visibility persist across manual/proposition actions and application restarts.
• Control+Z after a commit restores the exact pre-G Merge workspace; redo reapplies the merge.

Merge Propositions workflow

Template GUI reads the standard AIND/SpikeInterface format-version 2 merges list from dataset-local curation.json.

• The persistent Merge Propositions view is a button-free review queue.
• Source entries receive concise one-based labels (P1, P2, …) from their original curation.json position, including invalid entries. Filtering and review-state changes never renumber them; internal hash keys remain the durable identity. Hierarchical labels such as P12.1 are reserved for future persisted propositions derived from source P12; this release does not synthesize derived rows.
• Rows show all IDs for propositions of up to four units. Larger rows show first, second, …, last (count). A supplied new_unit_id appears as ⇒ new_unit_id.
• The selected-row background alone identifies the active review, while foreground colors indicate accepted, accepted-modified, rejected, stale, or invalid states. Blue text remains reserved for the merge reference cluster. Hovering exposes the full ordered IDs, blue/reference unit, status, stable internal key, and reason.
• Clicking a pending row immediately opens it in Merge View. It replaces any active manual or proposition workspace.
• Moving between propositions updates only the affected queue rows in place, preserving the Merge Propositions dock position and size, table layout, selection, and scroll context.
• Clicking a completed, stale, or invalid row cancels an active workspace and highlights the row without opening it.
• The ordinary Merge View remains authoritative during review: candidates can be added, removed, and reordered. G records accepted_modified when the committed IDs differ from the source proposition.
• In Correlogram View, Control-right-click a staged cluster to remove it during proposition review. Removing the reference promotes the next staged cluster while keeping the proposition active; the last staged cluster remains protected because Merge mode requires a reference.
• After a successful proposition merge, the next pending proposition opens automatically in the visible table order captured before the merge. Manual merges and failed proposition merges never advance the queue.
• Cancelling leaves a proposition pending. Overlapping propositions whose source units disappear become stale and are never guessed or remapped through descendants.
• Reject and reset are undoable. Undo/redo restore the exact before/after proposition workspace, including an automatically opened next proposition.
• Proposition shortcuts are suppressed while a text input has focus.

curation.json remains producer-owned and is never overwritten. Accepted/rejected decisions and the actual applied/result IDs are saved atomically to dataset-local curation_review.json, coupled to the normal dirty/save lifecycle and guarded by a source-file hash.

Merge Proposition shortcuts

Action	Shortcut
Next pending proposition, wrapping in visible order	Alt+Down
Previous pending proposition, wrapping in visible order	Alt+Up
Reject active proposition and advance	Alt+Backspace
Reset highlighted completed review and reopen	Alt+Shift+Backspace
Commit the staged merge	G
Cancel Merge mode	V
Undo / redo	Control+Z / Control+Shift+Z or Control+Y

Because Alt+Up/Alt+Down now navigate propositions, Trace View scaling moves to Control+Alt+Up/Control+Alt+Down.

Amplitude threshold splitting

• With exactly one cluster selected, Alt-right-drag in Amplitude View to preview a horizontal threshold.
• Matching waveform traces receive the same transient preview.
• Press K to split all eligible spikes strictly below the threshold. The preview is sampled for responsiveness, while the committed split evaluates every eligible spike.
• Clear the preview with Control-right-click or View > Clear amplitude split threshold.
• Empty and whole-cluster threshold selections are not committed.

Selection, wizard, and history behavior

• The first blue Cluster View selection is the explicit Similarity reference.
• In Normal mode, scientific views follow selected Cluster and Similarity rows in visible table order. Sorting/filtering changes presentation order without recoloring existing selections.
• A normal click or Space/Shift+Space replaces the Similarity candidate and reuses the first candidate color; Control/Shift multi-selection preserves existing color slots.
• Backspace clears Similarity candidates and their Normal-mode color reservations.
• Control+Space selects the first eligible Similarity rows in configurable batches; repeated use advances through later batches.
• Skip Noise and MUA applies to wizard navigation and batch selection and persists across sessions.
• Merge, split, metadata, proposition accept/reject/reset, undo, and redo restore complete selection and table workflow context rather than only cluster IDs.

Other user-visible changes

• Correlogram View supports Control-right-click deselection with display-aligned matrix hit-testing. A diagonal cell removes that exact cluster; an off-diagonal cell removes the non-primary cluster when the pair contains the current Cluster View selection or staged Merge reference.
• Firing Rate View now reports spikes per second and uses a rendered bin count consistent with its configuration.
• Amplitude and Firing Rate views share a seconds/minutes/hours recording-time preference with concise tick labels.
• Save is enabled only for dirty curation, the title shows unsaved state, and successful saves are confirmed in the status bar.
• Help > Show shortcuts and commands opens a searchable in-GUI shortcut reference, including plugin actions.
• View menus consistently put content actions before Auto-update, Screenshot, and Close; view creation is grouped under View > Add view.
• Native tables show sort direction, row-wide hover, stable row sizing/column widths, and improved drag previews.
• Waveform, Amplitude, and Correlogram views support fixed per-cluster or shared sampling budgets and defer hidden-tab rendering.
• Merge/assignment operations use maintained per-cluster indexes and batch expensive table fitting.
• GUI/Supervisor/table/dock callbacks are released on close, addressing the intermittent Qt shutdown crash seen while repeatedly opening datasets.
• The branch identifies as 2.2.0.dev0; installation documentation follows the uv-first workflow.

How to test

Use a separate checkout and a copy of a dataset because curation writes into the dataset directory.

git clone https://github.com/cortex-lab/phy.git phy-merge-workflow
cd phy-merge-workflow
git switch feature/merge-view-workflow
uv sync --dev
uv pip install --python .venv "phylib @ git+https://github.com/cortex-lab/phylib.git@agent/fix-template-less-curation-reload"

For the compact example dataset:

git clone https://github.com/kwikteam/phy-data.git
cd phy-data/template
../../phy-merge-workflow/.venv/bin/phy template-gui params.py

Suggested checks:

1. Exercise manual Merge mode, row transfers/reordering, cancel by both V and dock close, commit, undo, and redo.
2. Add a format-version 2 curation.json containing several merges, including overlapping propositions and one with new_unit_id. Exercise click-to-review, every proposition shortcut, edited acceptance, automatic advancement, cancellation, reject/reset, and undo/redo. In Correlogram View, also remove the staged reference and another staged candidate from diagonal and off-diagonal cells and verify the exact cluster is removed.
3. Save and inspect curation_review.json; confirm the input curation.json is unchanged. Repeat after replacing the source file to exercise hash mismatch handling.
4. Exercise the Amplitude threshold preview, waveform linkage, clear actions, split, and undo/redo.
5. Verify Normal and Merge-mode colors/order while clicking, using wizard shortcuts, sorting, filtering, and changing the blue reference.
6. Open and close multiple datasets/windows and confirm clean process shutdown.

Return to master

Close phy first, then restore the default branch and environment:

cd phy-merge-workflow
git switch master
git pull --ff-only
uv sync --dev

Keep any source changes before switching branches. Dataset curation output is separate from the source checkout; restore the dataset from backup if you want to discard saved curation.

Validation

• make lint
• make format-check
• make doc-check
• make test-full: 472 core/GUI tests and 153 application tests passed; aggregate coverage 97%.
• Latest targeted validation: 122 template/selection/supervisor tests passed, along with focused Correlogram and Merge Proposition regressions covering all 20 displayed diagonal cells, current-best removal, hidden selections, staged-reference promotion, and exact candidate removal.
• Focused regressions cover selection intent/order/colors, merge lifecycle and failure atomicity, proposition parsing/persistence/staleness, exact undo/redo workspaces, Qt callback cleanup, amplitude thresholds, tables, views, axes, and generated documentation.

[goatsofnaxos]

ProbeView does not appear to update in this version.

[goatsofnaxos]

After performing a merge the resulting unit should appear in the MergeView workspace and be selected (i.e. become the blue unit). This will permit evaluation for further merges or quality.

[goatsofnaxos]

Got a bunch of warnings when first opened and interacted with the GUI (apologies, I failed to save these). I then saved, closed, and re-launched and it was not able to initialize at all:

Last login: Mon Aug  3 17:27:21 on ttys000
➜  ~ source "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/bin/activate"

(phy) ➜  ~ phy template-gui /Users/#######/params.py
17:43:21.471 [W] model:626            Unreferenced clusters found in spike_clusters (generally not a problem)
17:43:24.919 [E] __init__:57          An error has occurred (AttributeError): 'NoneType' object has no attribute 'cols'
Traceback (most recent call last):
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/bin/phy", line 10, in <module>
    sys.exit(phycli())
             ~~~~~~^^
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/core.py", line 1569, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/core.py", line 1490, in main
    rv = self.invoke(ctx)
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/core.py", line 1970, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/core.py", line 1353, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/core.py", line 907, in invoke
    return callback(*args, **kwargs)
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/venv-2.2.0.dev0-20260803T1534/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/phy/apps/__init__.py", line 164, in cli_template_gui
    template_gui(params_path, **kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/#######/Desktop/phy-source/phy-2.2.0.dev0-20260803T1534/phy/apps/template/gui.py", line 208, in template_gui
    model = load_model(params_path)
  File "/Users/#######/Desktop/phy-source/phylib-2.7.1.dev0-20260723T1706/phylib/io/model.py", line 1441, in load_model
    return TemplateModel(**get_template_params(params_path))
  File "/Users/#######/Desktop/phy-source/phylib-2.7.1.dev0-20260723T1706/phylib/io/model.py", line 340, in __init__
    self._load_data()
    ~~~~~~~~~~~~~~~^^
  File "/Users/#######/Desktop/phy-source/phylib-2.7.1.dev0-20260723T1706/phylib/io/model.py", line 418, in _load_data
    self.sparse_templates.cols is None:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'cols'

QWidget: Must construct a QApplication before a QWidget
[1]    16977 abort      phy template-gui 

[rossant]

ProbeView does not appear to update in this version.

It does update on my computer. Perhaps try to remove this view and add it again?

[rossant]

@goatsofnaxos Thanks for the traceback and the follow-up Merge-mode report. I reproduced two separate problems, and both fixes are now available.

The restart failure (AttributeError: 'NoneType' object has no attribute 'cols') was a phylib loading bug. It is fixed on phylib master in cortex-lab/phylib@9bcbdad. Your saved spike_clusters.npy should still contain your curation work.

I also updated this PR in 4ba56d5d to fix the manual Merge-mode continuation and the Error when executing action Merge path. After pressing G in manual Merge mode:

• the merged unit remains in Merge View as the sole blue reference;
• Similarity View is recomputed for it;
• dragging a candidate from Similarity View into Merge View works for a chained merge;
• G can commit the chained merge without the action error;
• Control+Z restores the exact pre-merge workspace and redo restores the continuation;
• press V (shown as Exit Merge Mode) to return to Normal mode before assigning quality.

To retest both fixes, activate the same environment used to run phy and install the updated branches:

pip install --upgrade "git+https://github.com/cortex-lab/phylib.git@master"
pip install --upgrade "git+https://github.com/cortex-lab/phy.git@feature/merge-view-workflow"

Then restart phy and reopen the dataset. Please let me know if either the loading error or the Merge-mode workflow problem persists.

[goatsofnaxos]

ProbeView does not appear to update in this version.

It does update on my computer. Perhaps try to remove this view and add it again?

It's still not working after removing/adding. Also it only shows one electrode instead of the number shown in the waveformview.

[goatsofnaxos]

@goatsofnaxos Thanks for the traceback and the follow-up Merge-mode report. I reproduced two separate problems, and both fixes are now available.

The restart failure (AttributeError: 'NoneType' object has no attribute 'cols') was a phylib loading bug. It is fixed on phylib master in cortex-lab/phylib@9bcbdad. Your saved spike_clusters.npy should still contain your curation work.

I also updated this PR in 4ba56d5d to fix the manual Merge-mode continuation and the Error when executing action Merge path. After pressing G in manual Merge mode:

• the merged unit remains in Merge View as the sole blue reference;
• Similarity View is recomputed for it;
• dragging a candidate from Similarity View into Merge View works for a chained merge;
• G can commit the chained merge without the action error;
• Control+Z restores the exact pre-merge workspace and redo restores the continuation;
• press V (shown as Exit Merge Mode) to return to Normal mode before assigning quality.

To retest both fixes, activate the same environment used to run phy and install the updated branches:

pip install --upgrade "git+https://github.com/cortex-lab/phylib.git@master"
pip install --upgrade "git+https://github.com/cortex-lab/phy.git@feature/merge-view-workflow"

Then restart phy and reopen the dataset. Please let me know if either the loading error or the Merge-mode workflow problem persists.

The errors are gone but still get warnings every time I select the next unit, see attached log.

warning_log.txt

[goatsofnaxos]

• the merged unit remains in Merge View as the sole blue reference

I think from a UX perspective it would be most intuitive if the newly created unit post-merge become exclusively selected in the ClusterView. In other words we move the focus to ClusterView away from MergePropositionsView, so that we can focus on assessing the quality of the merged unit, and consider whether more units need to be merged into it.

[rossant]

@goatsofnaxos Thanks — I traced the Probe View issue and the repeated waveform warnings to the same template-less dataset path.

This dataset has a stored spike-waveform subset but no usable templates.npy. Two coordinated fixes are now ready:

• phylib derives the waveform sample count from the stored subset, so those waveforms remain accessible after saving and reopening the dataset: Load stored waveform subsets without templates phylib#63
• phy derives cluster-specific channel rankings from the stored waveforms instead of falling back to channel zero. Probe View and Waveform View therefore use the actual channels represented by each cluster.

I also implemented the requested manual-merge behavior:

• after G, manual Merge mode closes;
• the newly merged unit becomes the sole Cluster View selection;
• quality can be assigned immediately;
• press V again to start another merge with that unit;
• undo restores the exact pre-merge workspace and redo returns to Normal mode with the result selected.

Merge propositions retain their automatic advancement behavior.

Validation completed with 472 core/GUI tests and 153 application tests passing, plus lint, formatting, and documentation checks.

To retest:

pip install --upgrade "git+https://github.com/cortex-lab/phylib.git@agent/fix-template-less-curation-reload"
pip install --upgrade "git+https://github.com/cortex-lab/phy.git@feature/merge-view-workflow"

Please let me know whether the warnings, channel-zero Probe View behavior, or post-merge workflow persist with these revisions.