Skip to content

Fix novel silently saving no model on multiome/swap - #62

Merged
rcannood merged 1 commit into
mainfrom
fix/novel-lsi-nan
Aug 13, 2026
Merged

Fix novel silently saving no model on multiome/swap#62
rcannood merged 1 commit into
mainfrom
fix/novel-lsi-nan

Conversation

@rcannood

Copy link
Copy Markdown
Member

novel scored 12/16 in the run_2026-08-01_11-48-20 benchmark. The four missing scores are both multiome/swap datasets (2021 + 2022), where novel_predict died on:

FileNotFoundError: '_viash_par/input_model_1/log_cp10k.novel.novel_train.output/tensor.pt'

What actually happened

novel_train exited 0 and published a model directory with no model in it.

The chain, straight from the run logs:

  • An empty ATAC cell produces a constant lsi row, so the per-cell z-score in lsiTransformer.transform() divides 0 by 0 -- helper_functions.py:78: RuntimeWarning: invalid value encountered in divide.

  • That nan reaches rmse(), and if best_score > rmse(...) is always False for nan, so torch.save() never runs.

  • train_and_valid() then prints its untouched sentinel and returns normally:

    bmmc_multiome/swap    best rmse:  100000     <- never saved
    pbmc_multiome/swap    best rmse:  100000     <- never saved
    bmmc_multiome/normal  best rmse:  0.47737852 <- fine
    

So the train step looked green, and the failure only surfaced one step later in predict.

Changes

  • Guard the zero-variance divide in lsiTransformer.transform().
  • Track the best score as None/nan-aware rather than a 100000 sentinel, and raise if it never became finite -- a training step that saves no model should fail, not exit 0.

Verification

  • viash test src/methods/novel/novel_train/config.vsh.yaml: 1/1 passed

  • viash test src/methods/novel/novel_predict/config.vsh.yaml: 1/1 passed

  • The fixture is bmmc_cite (ADT), so it does not exercise the lsi path that failed. Checked that separately against the real lsiTransformer with an all-zero cell present:

    LSI output (60, 20): 0 NaN entries
    empty cell row all-finite: True
    

The helper_functions.py:33 "divide by zero" warning also present in the run is left alone: empty rows have no stored values, so the inf never multiplies anything and no nan reaches the output (0 NaN above, with that warning still firing).

An empty ATAC cell gives a constant lsi row, so the per-cell z-score divided 0 by 0.
The resulting nan rmse never beat best_score, so tensor.pt was never written and
novel_predict failed on a train step that had exited 0.
@rcannood
rcannood merged commit 7085ef1 into main Aug 13, 2026
3 checks passed
@rcannood
rcannood deleted the fix/novel-lsi-nan branch August 13, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant