Skip to content

Adjust power-of-two memory requests for cloud/HPC headroom - #286

Merged
johandahlberg merged 2 commits into
devfrom
cursor/adjust-memory-headroom
Nov 18, 2026
Merged

johandahlberg merged 2 commits into
devfrom
cursor/adjust-memory-headroom

Conversation

@johandahlberg

@johandahlberg johandahlberg commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adjust exact power-of-two process memory requests slightly downward so tasks fit onto standard cloud/HPC hosts after OS, kernel, and container agent (e.g. AWS Batch / ECS) memory reservations, both on initial attempts and * task.attempt retries:

  • conf/cells_8k.config:
    • PIXELATOR_AGGREGATE: 256.GB → 240.GB (fits 256 GiB hosts on attempt 1, 512 GiB on attempt 2, 768 GiB on attempt 3, 1024 GiB on attempt 4; DuckDB --memory with memory_factor = 0.75 becomes 184320M on attempt 1)
    • PIXELATOR_GRAPH: 512.GB → 480.GB (fits 512 GiB hosts on attempt 1, 1024 GiB on attempt 2)
    • PIXELATOR_GRAPH_COLLAPSE: 64.GB → 60.GB (fits 64 GiB 4xlarge / 128 GiB 8xlarge hosts)
    • PIXELATOR_DENOISE: 256.GB → 240.GB (32 CPUs / 240 GB fits 32-vCPU / 256 GiB 8xlarge hosts cleanly)
    • EXPERIMENT_SUMMARY: 64.GB → 60.GB
  • conf/cells_1k.config:
    • PIXELATOR_AGGREGATE: 64.GB → 60.GB
    • PIXELATOR_GRAPH: 512.GB → 480.GB
    • PIXELATOR_GRAPH_COLLAPSE: 32.GB → 30.GB
    • EXPERIMENT_SUMMARY: 64.GB → 60.GB
  • conf/base.config:
    • process_high: 64.GB → 60.GB (inherited by PIXELATOR_SAMPLE_CALLING, and PIXELATOR_DENOISE under cells_1k)
    • process_high_memory: 128.GB → 120.GB

PR checklist

  • This comment contains a description of changes (with reason).
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • CHANGELOG.md is updated.

johandahlberg and others added 2 commits November 18, 2026 11:04
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅

Posted for pipeline commit 2db1e18

+| ✅ 224 tests passed       |+
#| ❔   9 tests were ignored |#
Details

❔ Tests ignored:

  • files_exist - File is ignored: assets/multiqc_config.yml
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_exist - File is ignored: tests/default.nf.test
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-pixelator_logo_light.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • multiqc_config - multiqc_config

✅ Tests passed:

Run details

  • nf-core/tools version 4.2.2
  • Run at 2026-11-18 10:06:59

@johandahlberg
johandahlberg marked this pull request as ready for review November 18, 2026 10:13
@johandahlberg
johandahlberg requested a review from a team as a code owner November 18, 2026 10:13
@johandahlberg
johandahlberg requested a review from Aratz November 18, 2026 10:13
@johandahlberg
johandahlberg merged commit b4ee9fa into dev Nov 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants