Skip to content

Dev fix resource tags - #486

Open
migrau wants to merge 3 commits into
devfrom
dev-fix-resource-tags
Open

Dev fix resource tags#486
migrau wants to merge 3 commits into
devfrom
dev-fix-resource-tags

Conversation

@migrau

@migrau migrau commented Aug 20, 2026

Copy link
Copy Markdown
Member

Fix Nextflow resource tiers (CPU/memory) and error handling based on real trace data

Cleans up conf/base.config so process resource requests actually reflect measured usage, fixing two classes of bugs found while validating a resources_test run: processes silently running with cpus=1 (never parallelizing), and several processes so under-provisioned on memory that they either failed permanently after exhausting retries or wasted time retrying from a guaranteed-to-fail baseline.

Changes:

  • Consolidate ad-hoc CPU settings into 4 documented tiers (cpu_low/cpu_medium/cpu_high labels + untouched 1-cpu default) and apply them across ~80 modules that were silently defaulting to cpus=1.
  • Bump memory for processes verified via trace peak_rss to be under-provisioned: OMEGA/OMEGAMULTI ESTIMATOR/ESTIMATORGLOBALLOC (peak ~24GB vs old 4GB×attempt ceiling of 16GB), ONCODRIVE3D_RUN (peak ~40GB, had no override at all), FILTERNANOSEQNOISE and MUTDENSITYALL:MUTDENSITY (both were guaranteed to fail/retry on the first attempt).
  • Fix ONCODRIVEFMLSNVS cohort-level runs (all_samples, Ismale0/1, Samplelocation*) getting OOM-killed by Slurm (confirmed via .command.err, not a timeout despite matching the 6h wall-time default) — decouple cpus from cpu_high's attempt-scaling so memory retries actually gain headroom per core instead of both scaling in lockstep.
  • Add errorStrategy = 'ignore' for DNA2PROTEINMAPPING and a retry-then-ignore strategy for PLOTSATURATION, matching known-flaky behavior instead of requiring per-run overrides in the Seqera launch config.

###Test plan:

  • Re-run and confirm ONCODRIVEFMLSNVS, ESTIMATOR*, ONCODRIVE3D_RUN, FILTERNANOSEQNOISE and MUTDENSITYALL:MUTDENSITY complete without FAILED/ABORTED tasks
  • Spot-check %cpu in the new trace to confirm no tier is now over-provisioned

migrau added 3 commits August 7, 2026 08:49
…ling cpus from attempt-scaling so retries gain real headroom
@migrau migrau self-assigned this Aug 20, 2026
@migrau
migrau requested a review from FerriolCalvet August 20, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant