[ieivanov]

Why

Most steps requested their init estimate verbatim on every retry. A retry only gets more resources from retry_memory / retry_time in modules/common.nf, and most processes did not use them. So an estimate that was too low failed every attempt the same way and aborted the run once maxRetries ran out.

The v9 debug run of 2026_08_11_A549_SEC61B_DENV hit exactly this in its pyramid step: every attempt asked for 16 GB and was OOM-killed with 137.

Before this PR:

process	time	memory
run_flat_field, run_deskew, run_apply_inv_tf, run_virtual_stain, run_concatenate, run_segment	retry_time	fixed estimate
compute_transfer_function	fixed 30 min	fixed 64 GB
run_virtual_stain_preprocess	fixed 1 h	retry_memory
run_track	fixed 2 h, 1 retry	fixed estimate
QC, registration	retry_time	retry_memory

What changes

• Memory: every SLURM process listed above now uses retry_memory. It doubles memory after an OOM kill (exit 137) and keeps the request after a preemption.
• Time: compute_transfer_function, run_virtual_stain_preprocess and run_track now use retry_time instead of a constant, so a wall-time kill doubles the next attempt. This matters most for track: it retries only once and ultrack cannot resume, so a second timeout with the same 2 h was certain.
• Memory cap: retry_memory caps a doubled request at 244 GB, the second node tier of preempted.

node memory on preempted	nodes
120000 MiB (cpu-c-[1-4])	4
250000 MiB (gpu-sm01/02)	40
≥ 500000 MiB	78

• 244 GiB = 249856 MiB, so a capped retry fits 118 of the 122 nodes and sbatch never refuses it. A refusal is exit 1, which would terminate the run.
• A request whose init estimate is already above the cap is kept, never lowered.
• Trade-off: a retry near the cap takes a whole gpu-sm node's memory and idles its GPU while it runs.
• GPU memory is out of scope. A CUDA OOM exits 1 and is not retried by these helpers.

project.nf and pyramid.nf exist only on their feature branches and get the same retry_memory switch there: #424 (a84a480) and #430 (f1966b2). #430 picks up the cap once it merges main.

Testing

• nextflow run nextflow/tests/retry_resources.nf passes. It adds four cap cases: 128 GB doubles to 244 GB, a request at the cap stays at the cap, a 400 GB base is kept, and an OOM retry of a 400 GB request stays at 400 GB.
• Every changed module compiles under Nextflow 26.04 (an include-only script that loads each changed process).
• Pre-commit hooks pass.

🤖 Generated with Claude Code