From 21dfbf167ea690aec53ddf92b16120fec668e609 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 29 Sep 2026 13:05:14 +0200 Subject: [PATCH 1/6] Bump sched-client to 0.96.0 and lib-httpx to 2.7.1 [ci fast] sched-client 0.96.0 is built against lib-httpx 2.7.1, and nf-seqera keeps the single lib-httpx declaration in step with it. Assisted-by: Claude Code (Claude Opus 5.5) Signed-off-by: Paolo Di Tommaso --- gradle.properties | 2 +- plugins/nf-seqera/build.gradle | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/gradle.properties b/gradle.properties index c30425530a..bdb25c0c0b 100644 --- a/gradle.properties +++ b/gradle.properties @@ -18,7 +18,7 @@ jacksonCloudSdkVersion=2.18.9 jaxbApiVersion=2.4.0-b180830.0359 jimfsVersion=1.2 kryoVersion=2.24.0 -libHttpxVersion=2.7.0 +libHttpxVersion=2.7.1 libTraceVersion=0.1.0 logbackVersion=1.5.38 nprVersion=0.27.0 diff --git a/plugins/nf-seqera/build.gradle b/plugins/nf-seqera/build.gradle index dcb6d9c9b8..7f1195b5e5 100644 --- a/plugins/nf-seqera/build.gradle +++ b/plugins/nf-seqera/build.gradle @@ -53,7 +53,7 @@ dependencies { compileOnly "org.pf4j:pf4j:${pf4jVersion}" // intelligent compute scheduler client - api 'io.seqera:sched-client:0.79.3' + api 'io.seqera:sched-client:0.96.0' // Single lib-httpx declaration, kept in step with the version sched-client is built // against. SchedHxClient overrides shouldRetryOnException(HttpRequest, Throwable), an // overload added in 2.6.0: on an older lib-httpx the override binds to nothing and the From 6fa0d669fc628db5f9e474771fe059172b4fcca5 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 29 Sep 2026 13:05:14 +0200 Subject: [PATCH 2/6] Add seqera.executor.machineRequirement.onDemandCostRatio option Set the run's cost limit on the spot to on-demand fallback (spotFirst only). Unset keeps the scheduler's per-cloud default, 0 removes the limit, and a positive value replaces the default. The scheduler reads the ratio at run level only, so it is sent on run creation and rejected as a process hint. Requires a sched-client that includes seqeralabs/sched#1400. Assisted-by: Claude Code (Claude Opus 5.5) Signed-off-by: Paolo Di Tommaso --- docs/executor/seqera.mdx | 31 +++++++++++++++++++ docs/reference/config/seqera.mdx | 13 ++++++++ .../config/MachineRequirementOpts.groovy | 14 +++++++++ .../src/main/io/seqera/util/HintHelper.groovy | 7 ++++- .../io/seqera/util/SchemaMapperUtil.groovy | 8 ++++- .../test/io/seqera/util/HintHelperTest.groovy | 13 ++++++-- .../test/io/seqera/util/MapperUtilTest.groovy | 23 ++++++++++++++ 7 files changed, 105 insertions(+), 4 deletions(-) diff --git a/docs/executor/seqera.mdx b/docs/executor/seqera.mdx index 42a8e3bbaa..266fad39cc 100644 --- a/docs/executor/seqera.mdx +++ b/docs/executor/seqera.mdx @@ -71,6 +71,37 @@ process hello { See the [`seqera` scope][config-seqera] for the full config reference. +## On-demand fallback cost limit + + + +With `provisioning = 'spotFirst'`, the scheduler falls back to on-demand when spot capacity runs out, but only while on-demand is not too expensive. It compares the highest on-demand hourly price of the instance types selected for a task with their highest spot price. When on-demand exceeds the limit, the task is retried on spot instead, and fails if spot capacity stays unavailable. Its error message starts with `on-demand fallback refused`. + +The scheduler applies a default limit for each cloud provider. Use `onDemandCostRatio` to set a different limit for the whole run: + +```groovy +seqera { + executor { + machineRequirement { + provisioning = 'spotFirst' + onDemandCostRatio = 2.5 // fall back only while on-demand costs at most 2.5x spot + } + } +} +``` + +Leaving `onDemandCostRatio` unset and setting it to `0` are different: + +| `onDemandCostRatio` | Behavior | +| --- | --- | +| Not set | The scheduler's default limit for the cloud provider applies. | +| `0` | No limit. Tasks always fall back to on-demand, whatever the price. | +| Greater than `0` | The run's own limit, used instead of the scheduler's default. | + +`0` does not disable the fallback. To never fall back to on-demand, set `provisioning = 'spot'`. + +`onDemandCostRatio` applies to the whole run, so it isn't supported as a process hint. Setting it also stops the run from using the compute environment's warm pool. + ## Disk support When a process specifies the [disk][process-disk] directive, the Seqera executor provisions storage for the task container. It supports two disk allocation strategies: diff --git a/docs/reference/config/seqera.mdx b/docs/reference/config/seqera.mdx index 4aeb87d74a..1b227f3215 100644 --- a/docs/reference/config/seqera.mdx +++ b/docs/reference/config/seqera.mdx @@ -103,6 +103,18 @@ List of acceptable EC2 instance families. For example, `['m5', 'c5', 'r5']`. The maximum number of spot retry attempts before falling back to on-demand. Only used when `provisioning` is `'spot'` or `'spotFirst'`. +##### `seqera.executor.machineRequirement.onDemandCostRatio` + + + +The cost limit on the spot to on-demand fallback for the whole run. Only used when `provisioning` is `'spotFirst'`. A task falls back to on-demand only while the highest on-demand hourly price of the instance types selected for it is at most this multiple of their highest spot price. Otherwise the task is retried on spot and fails if spot capacity stays unavailable. For example, `onDemandCostRatio = 2.5`. + +- Not set (default): the scheduler's default limit for the cloud provider applies. +- `0`: no limit. Tasks always fall back to on-demand, whatever the price. This does not disable the fallback; use `provisioning = 'spot'` to never fall back to on-demand. +- Greater than `0`: the run's own limit, used instead of the scheduler's default. + +The value must not be negative. This option applies to the whole run and can't be set as a process hint. See [On-demand fallback cost limit][seqera-executor-cost-limit]. + ##### `seqera.executor.machineRequirement.provisioning` The instance provisioning mode. Can be `'spot'`, `'ondemand'`, or `'spotFirst'`. @@ -142,3 +154,4 @@ The multiplier for exponential backoff (default: `2.0`). [config-tower-autoLabels]: ./tower#towerautolabels [process-resource-labels]: ../process/directives/resource-labels [seqera-executor]: ../../executor/seqera +[seqera-executor-cost-limit]: ../../executor/seqera#on-demand-fallback-cost-limit diff --git a/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy index 887c99e2e0..f2d4b1b58a 100644 --- a/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy @@ -43,6 +43,15 @@ class MachineRequirementOpts implements ConfigScope { """) final Integer maxSpotAttempts + @ConfigOption + @Description(""" + Cost limit on the spot to on-demand fallback for the whole run. With `spotFirst`, a task falls + back to on-demand only while on-demand costs at most this multiple of the spot price. When not + set, the scheduler's default for the cloud provider applies. Set to `0` to remove the limit, so + tasks always fall back whatever the price. Only used when provisioning is `spotFirst`. + """) + final Double onDemandCostRatio + @ConfigOption @Description(""" List of acceptable machine type patterns. Supports exact types (e.g., `t3.small`), @@ -112,6 +121,7 @@ class MachineRequirementOpts implements ConfigScope { MachineRequirementOpts(Map opts) { this.provisioning = opts.provisioning as String this.maxSpotAttempts = opts.maxSpotAttempts as Integer + this.onDemandCostRatio = opts.onDemandCostRatio as Double this.machineTypes = (opts.machineTypes ?: opts.machineFamilies) as List this.diskType = opts.diskType as String this.diskThroughputMiBps = opts.diskThroughputMiBps as Integer @@ -133,6 +143,10 @@ class MachineRequirementOpts implements ConfigScope { return maxSpotAttempts } + Double getOnDemandCostRatio() { + return onDemandCostRatio + } + List getMachineTypes() { return machineTypes } diff --git a/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy b/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy index 7ac5ec6fdc..7f15eb9be4 100644 --- a/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy @@ -47,8 +47,11 @@ class HintHelper { /** Recognized top-level (non machineRequirement) task hint keys. */ static final Set TASK_KEYS = Set.of(PREDICTION_MODEL_KEY) + /** machineRequirement options the scheduler reads for the whole run only, so not valid as task hints. */ + static final Set RUN_ONLY_KEYS = Set.of(MR_PREFIX + 'onDemandCostRatio') + static final Set KNOWN_KEYS = Collections.unmodifiableSet( - (MR_FIELDS.collect { MR_PREFIX + it.name } + (TASK_KEYS as List)).toSet() + (MR_FIELDS.collect { MR_PREFIX + it.name }.findAll { !RUN_ONLY_KEYS.contains(it) } + (TASK_KEYS as List)).toSet() ) private static final String SUPPORTED_KEYS_MSG = @@ -83,6 +86,8 @@ class HintHelper { if( key.startsWith(PREFIX) ) { final stripped = key.substring(PREFIX.length()) + if( RUN_ONLY_KEYS.contains(stripped) ) + throw new IllegalArgumentException("Seqera Platform hint '${key}' is not supported — it applies to the whole run, set it with the 'seqera.executor.${stripped}' config option") if( !KNOWN_KEYS.contains(stripped) ) throw new IllegalArgumentException("Unknown Seqera Platform hint: '${key}' — supported keys are: ${SUPPORTED_KEYS_MSG}") prefixed.put(stripped, entry.value) diff --git a/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy index 3eead3f3c3..75716ee2a1 100644 --- a/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy @@ -64,11 +64,17 @@ class SchemaMapperUtil { return null final diskReq = toDiskRequirement(opts.diskSize, opts) final capacityMode = toEcsCapacityMode(opts.capacityMode) - if (!opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode) + // the cost ratio is read by the scheduler at run level only; null check rather than + // truthiness, because an explicit 0 (no limit) must be sent + final costRatio = opts.onDemandCostRatio + if (costRatio != null && costRatio < 0) + throw new IllegalArgumentException("Invalid onDemandCostRatio value: ${costRatio} -- the value must be zero or a positive number") + if (!opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode && costRatio == null) return null new MachineRequirement() .provisioning(toProvisioningModel(opts.provisioning)) .maxSpotAttempts(opts.maxSpotAttempts) + .onDemandCostRatio(costRatio) .machineTypes(opts.machineTypes) .disk(diskReq) .capacityMode(capacityMode) diff --git a/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy b/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy index 26efe36798..cec8f87bd4 100644 --- a/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy +++ b/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy @@ -186,15 +186,24 @@ class HintHelperTest extends Specification { } def 'should derive known keys from MachineRequirementOpts declared fields'() { - expect: 'KNOWN_KEYS covers every declared field of MachineRequirementOpts' + expect: 'KNOWN_KEYS covers every declared field of MachineRequirementOpts, except the run-only ones' HintHelper.KNOWN_KEYS.size() > 0 for( final field : MachineRequirementOpts.declaredFields ) { if( field.synthetic || java.lang.reflect.Modifier.isStatic(field.modifiers) || field.name.startsWith('$') || field.name == 'metaClass' ) continue - assert HintHelper.KNOWN_KEYS.contains("machineRequirement.${field.name}".toString()) + final key = "machineRequirement.${field.name}".toString() + assert HintHelper.KNOWN_KEYS.contains(key) != HintHelper.RUN_ONLY_KEYS.contains(key) } } + def 'should reject a run-only option as a task hint'() { + when: + HintHelper.extractSeqeraHints(['seqera/machineRequirement.onDemandCostRatio': 2]) + then: + def e = thrown(IllegalArgumentException) + e.message.contains("'seqera.executor.machineRequirement.onDemandCostRatio' config option") + } + def 'should error on unknown seqera hint'() { when: HintHelper.extractSeqeraHints(['seqera/machineRequirement.unknownField': 'value']) diff --git a/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy index 05f3903a37..7f02d5d2ae 100644 --- a/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy +++ b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy @@ -44,6 +44,29 @@ class MapperUtilTest extends Specification { SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([:])) == null } + def 'should send the on-demand cost ratio only when set, including zero' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([onDemandCostRatio: RATIO])) + + then: + result?.onDemandCostRatio == EXPECTED + + where: + RATIO | EXPECTED + null | null // absent: the scheduler's per-cloud default applies + 0 | 0d // explicit zero: no limit, must not be dropped + 2.5 | 2.5d + } + + def 'should reject a negative on-demand cost ratio' () { + when: + SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([onDemandCostRatio: -1])) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('onDemandCostRatio') + } + def 'should map all fields' () { when: def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([ From 4f2717fc0ddf32f2efc477b8ae85e76e859e2519 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 29 Sep 2026 13:09:55 +0200 Subject: [PATCH 3/6] Use sched-client 0.96.0-SNAPSHOT for onDemandCostRatio [ci fast] Temporary until sched publishes the release that includes seqeralabs/sched#1400. Assisted-by: Claude Code (Claude Opus 5.5) Signed-off-by: Paolo Di Tommaso --- plugins/nf-seqera/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/nf-seqera/build.gradle b/plugins/nf-seqera/build.gradle index 7f1195b5e5..cefc5eaab2 100644 --- a/plugins/nf-seqera/build.gradle +++ b/plugins/nf-seqera/build.gradle @@ -53,7 +53,7 @@ dependencies { compileOnly "org.pf4j:pf4j:${pf4jVersion}" // intelligent compute scheduler client - api 'io.seqera:sched-client:0.96.0' + api 'io.seqera:sched-client:0.96.0-SNAPSHOT' // Single lib-httpx declaration, kept in step with the version sched-client is built // against. SchedHxClient overrides shouldRetryOnException(HttpRequest, Throwable), an // overload added in 2.6.0: on an older lib-httpx the override binds to nothing and the From 48ab6b3c7806273fc9747d486d52d778dd69f428 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 29 Sep 2026 14:02:02 +0200 Subject: [PATCH 4/6] Allow onDemandCostRatio as a per-process hint [ci fast] The scheduler now resolves the ratio per task, like provisioning: the config value is the run default and the seqera/machineRequirement.onDemandCostRatio hint overrides it for a process. Send it on each task's machine requirement and stop rejecting the hint. Signed-off-by: Paolo Di Tommaso Assisted-by: Claude Code (Claude Opus 5.5) --- docs/executor/seqera.mdx | 20 +++++++++++++--- docs/reference/config/seqera.mdx | 6 ++--- .../config/MachineRequirementOpts.groovy | 8 +++---- .../src/main/io/seqera/util/HintHelper.groovy | 7 +----- .../io/seqera/util/SchemaMapperUtil.groovy | 24 ++++++++++++++----- .../test/io/seqera/util/HintHelperTest.groovy | 23 +++++++++--------- .../test/io/seqera/util/MapperUtilTest.groovy | 9 ++++--- 7 files changed, 61 insertions(+), 36 deletions(-) diff --git a/docs/executor/seqera.mdx b/docs/executor/seqera.mdx index 266fad39cc..c656583ccf 100644 --- a/docs/executor/seqera.mdx +++ b/docs/executor/seqera.mdx @@ -38,6 +38,7 @@ The following [hints][process-hints] are supported: - `machineRequirement.diskType` - `machineRequirement.machineTypes` - `machineRequirement.maxSpotAttempts` +- `machineRequirement.onDemandCostRatio` - `machineRequirement.provisioning` - `predictionModel` @@ -77,7 +78,7 @@ See the [`seqera` scope][config-seqera] for the full config reference. With `provisioning = 'spotFirst'`, the scheduler falls back to on-demand when spot capacity runs out, but only while on-demand is not too expensive. It compares the highest on-demand hourly price of the instance types selected for a task with their highest spot price. When on-demand exceeds the limit, the task is retried on spot instead, and fails if spot capacity stays unavailable. Its error message starts with `on-demand fallback refused`. -The scheduler applies a default limit for each cloud provider. Use `onDemandCostRatio` to set a different limit for the whole run: +The scheduler applies a default limit for each cloud provider. Use `onDemandCostRatio` to set a different limit for all the tasks of the run: ```groovy seqera { @@ -96,11 +97,24 @@ Leaving `onDemandCostRatio` unset and setting it to `0` are different: | --- | --- | | Not set | The scheduler's default limit for the cloud provider applies. | | `0` | No limit. Tasks always fall back to on-demand, whatever the price. | -| Greater than `0` | The run's own limit, used instead of the scheduler's default. | +| Greater than `0` | This limit, used instead of the scheduler's default. | `0` does not disable the fallback. To never fall back to on-demand, set `provisioning = 'spot'`. -`onDemandCostRatio` applies to the whole run, so it isn't supported as a process hint. Setting it also stops the run from using the compute environment's warm pool. +Like `provisioning`, the limit can be overridden for a single process with a hint. For example, to let a long-running process always fall back to on-demand: + +```nextflow +process align { + hints 'seqera/machineRequirement.onDemandCostRatio': 0 + + script: + """ + your_command --here + """ +} +``` + +Tasks with different limits run on separate clusters. Tasks that set a limit, `0` included, do not use the compute environment's warm pool. ## Disk support diff --git a/docs/reference/config/seqera.mdx b/docs/reference/config/seqera.mdx index 1b227f3215..36c2a9aca2 100644 --- a/docs/reference/config/seqera.mdx +++ b/docs/reference/config/seqera.mdx @@ -107,13 +107,13 @@ The maximum number of spot retry attempts before falling back to on-demand. Only -The cost limit on the spot to on-demand fallback for the whole run. Only used when `provisioning` is `'spotFirst'`. A task falls back to on-demand only while the highest on-demand hourly price of the instance types selected for it is at most this multiple of their highest spot price. Otherwise the task is retried on spot and fails if spot capacity stays unavailable. For example, `onDemandCostRatio = 2.5`. +The cost limit on the spot to on-demand fallback. Only used when `provisioning` is `'spotFirst'`. A task falls back to on-demand only while the highest on-demand hourly price of the instance types selected for it is at most this multiple of their highest spot price. Otherwise the task is retried on spot and fails if spot capacity stays unavailable. For example, `onDemandCostRatio = 2.5`. - Not set (default): the scheduler's default limit for the cloud provider applies. - `0`: no limit. Tasks always fall back to on-demand, whatever the price. This does not disable the fallback; use `provisioning = 'spot'` to never fall back to on-demand. -- Greater than `0`: the run's own limit, used instead of the scheduler's default. +- Greater than `0`: this limit, used instead of the scheduler's default. -The value must not be negative. This option applies to the whole run and can't be set as a process hint. See [On-demand fallback cost limit][seqera-executor-cost-limit]. +The value must not be negative. It is the default for every task of the run; use the `machineRequirement.onDemandCostRatio` hint to override it for a single process. See [On-demand fallback cost limit][seqera-executor-cost-limit]. ##### `seqera.executor.machineRequirement.provisioning` diff --git a/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy index f2d4b1b58a..bb65dd1662 100644 --- a/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy @@ -45,10 +45,10 @@ class MachineRequirementOpts implements ConfigScope { @ConfigOption @Description(""" - Cost limit on the spot to on-demand fallback for the whole run. With `spotFirst`, a task falls - back to on-demand only while on-demand costs at most this multiple of the spot price. When not - set, the scheduler's default for the cloud provider applies. Set to `0` to remove the limit, so - tasks always fall back whatever the price. Only used when provisioning is `spotFirst`. + Cost limit on the spot to on-demand fallback. With `spotFirst`, a task falls back to on-demand + only while on-demand costs at most this multiple of the spot price. When not set, the + scheduler's default for the cloud provider applies. Set to `0` to remove the limit, so tasks + always fall back whatever the price. Only used when provisioning is `spotFirst`. """) final Double onDemandCostRatio diff --git a/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy b/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy index 7f15eb9be4..7ac5ec6fdc 100644 --- a/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/util/HintHelper.groovy @@ -47,11 +47,8 @@ class HintHelper { /** Recognized top-level (non machineRequirement) task hint keys. */ static final Set TASK_KEYS = Set.of(PREDICTION_MODEL_KEY) - /** machineRequirement options the scheduler reads for the whole run only, so not valid as task hints. */ - static final Set RUN_ONLY_KEYS = Set.of(MR_PREFIX + 'onDemandCostRatio') - static final Set KNOWN_KEYS = Collections.unmodifiableSet( - (MR_FIELDS.collect { MR_PREFIX + it.name }.findAll { !RUN_ONLY_KEYS.contains(it) } + (TASK_KEYS as List)).toSet() + (MR_FIELDS.collect { MR_PREFIX + it.name } + (TASK_KEYS as List)).toSet() ) private static final String SUPPORTED_KEYS_MSG = @@ -86,8 +83,6 @@ class HintHelper { if( key.startsWith(PREFIX) ) { final stripped = key.substring(PREFIX.length()) - if( RUN_ONLY_KEYS.contains(stripped) ) - throw new IllegalArgumentException("Seqera Platform hint '${key}' is not supported — it applies to the whole run, set it with the 'seqera.executor.${stripped}' config option") if( !KNOWN_KEYS.contains(stripped) ) throw new IllegalArgumentException("Unknown Seqera Platform hint: '${key}' — supported keys are: ${SUPPORTED_KEYS_MSG}") prefixed.put(stripped, entry.value) diff --git a/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy index 75716ee2a1..bae1b3c14d 100644 --- a/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy +++ b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy @@ -64,11 +64,7 @@ class SchemaMapperUtil { return null final diskReq = toDiskRequirement(opts.diskSize, opts) final capacityMode = toEcsCapacityMode(opts.capacityMode) - // the cost ratio is read by the scheduler at run level only; null check rather than - // truthiness, because an explicit 0 (no limit) must be sent - final costRatio = opts.onDemandCostRatio - if (costRatio != null && costRatio < 0) - throw new IllegalArgumentException("Invalid onDemandCostRatio value: ${costRatio} -- the value must be zero or a positive number") + final costRatio = toOnDemandCostRatio(opts) if (!opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode && costRatio == null) return null new MachineRequirement() @@ -123,19 +119,35 @@ class SchemaMapperUtil { final effectiveDiskSize = diskSize ?: opts?.diskSize final diskReq = toDiskRequirement(effectiveDiskSize, opts) final capacityMode = toEcsCapacityMode(opts?.capacityMode) + final costRatio = toOnDemandCostRatio(opts) // return null if no settings - if (!arch && !provisioning && !maxSpotAttempts && !machineTypes && !diskReq && !snapshotEnabled && !capacityMode) + if (!arch && !provisioning && !maxSpotAttempts && !machineTypes && !diskReq && !snapshotEnabled && !capacityMode && costRatio == null) return null new MachineRequirement() .arch(arch) .provisioning(toProvisioningModel(provisioning)) .maxSpotAttempts(maxSpotAttempts) + .onDemandCostRatio(costRatio) .machineTypes(machineTypes) .disk(diskReq) .snapshotEnabled(snapshotEnabled ? Boolean.TRUE : null) .capacityMode(capacityMode) } + /** + * Validates the spot to on-demand fallback cost ratio. Compared against null rather than by + * truthiness, because an explicit {@code 0} (no limit) is a real value that must be sent. + * + * @param opts the config options (can be null) + * @return the ratio, or null when not set + */ + static Double toOnDemandCostRatio(MachineRequirementOpts opts) { + final ratio = opts?.onDemandCostRatio + if (ratio != null && ratio < 0) + throw new IllegalArgumentException("Invalid onDemandCostRatio value: ${ratio} -- the value must be zero or a positive number") + return ratio + } + /** * Maps a disk size to DiskRequirement API object. * Uses config options if provided, otherwise defaults to Fusion recommended settings: diff --git a/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy b/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy index cec8f87bd4..682e86a23f 100644 --- a/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy +++ b/plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy @@ -67,6 +67,16 @@ class HintHelperTest extends Specification { result.maxSpotAttempts == 3 } + def 'should overlay onDemandCostRatio hint over the config value, zero included'() { + given: + def base = new MachineRequirementOpts([onDemandCostRatio: 3.25]) + + when: + def result = HintHelper.overlayHints(base, ['seqera/machineRequirement.onDemandCostRatio': 0]) + then: 'a task 0 (no limit) is a real value, not "unset"' + result.onDemandCostRatio == 0d + } + def 'should overlay machineTypes as list'() { given: def base = new MachineRequirementOpts([:]) @@ -186,24 +196,15 @@ class HintHelperTest extends Specification { } def 'should derive known keys from MachineRequirementOpts declared fields'() { - expect: 'KNOWN_KEYS covers every declared field of MachineRequirementOpts, except the run-only ones' + expect: 'KNOWN_KEYS covers every declared field of MachineRequirementOpts' HintHelper.KNOWN_KEYS.size() > 0 for( final field : MachineRequirementOpts.declaredFields ) { if( field.synthetic || java.lang.reflect.Modifier.isStatic(field.modifiers) || field.name.startsWith('$') || field.name == 'metaClass' ) continue - final key = "machineRequirement.${field.name}".toString() - assert HintHelper.KNOWN_KEYS.contains(key) != HintHelper.RUN_ONLY_KEYS.contains(key) + assert HintHelper.KNOWN_KEYS.contains("machineRequirement.${field.name}".toString()) } } - def 'should reject a run-only option as a task hint'() { - when: - HintHelper.extractSeqeraHints(['seqera/machineRequirement.onDemandCostRatio': 2]) - then: - def e = thrown(IllegalArgumentException) - e.message.contains("'seqera.executor.machineRequirement.onDemandCostRatio' config option") - } - def 'should error on unknown seqera hint'() { when: HintHelper.extractSeqeraHints(['seqera/machineRequirement.unknownField': 'value']) diff --git a/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy index 7f02d5d2ae..a7e97de552 100644 --- a/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy +++ b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy @@ -45,11 +45,14 @@ class MapperUtilTest extends Specification { } def 'should send the on-demand cost ratio only when set, including zero' () { - when: - def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([onDemandCostRatio: RATIO])) + when: 'mapped for the run (the default for its tasks) and for a task' + def opts = new MachineRequirementOpts([onDemandCostRatio: RATIO]) + def runReq = SchemaMapperUtil.toMachineRequirement(opts) + def taskReq = SchemaMapperUtil.toMachineRequirement(opts, null, null, false, 0) then: - result?.onDemandCostRatio == EXPECTED + runReq?.onDemandCostRatio == EXPECTED + taskReq?.onDemandCostRatio == EXPECTED where: RATIO | EXPECTED From a3e4adf1109fea932052f21bed8371466b234e38 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 30 Sep 2026 10:05:41 +0200 Subject: [PATCH 5/6] Update docs/executor/seqera.mdx [ci skip] Co-authored-by: Chris Hakkaart Signed-off-by: Paolo Di Tommaso --- docs/executor/seqera.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/executor/seqera.mdx b/docs/executor/seqera.mdx index c656583ccf..4c02db5ba8 100644 --- a/docs/executor/seqera.mdx +++ b/docs/executor/seqera.mdx @@ -76,9 +76,9 @@ See the [`seqera` scope][config-seqera] for the full config reference. -With `provisioning = 'spotFirst'`, the scheduler falls back to on-demand when spot capacity runs out, but only while on-demand is not too expensive. It compares the highest on-demand hourly price of the instance types selected for a task with their highest spot price. When on-demand exceeds the limit, the task is retried on spot instead, and fails if spot capacity stays unavailable. Its error message starts with `on-demand fallback refused`. +With `provisioning = 'spotFirst'`, the scheduler falls back to on-demand instances when spot capacity runs out, as long as on-demand stays within a cost limit. The limit is the maximum ratio of the highest on-demand hourly price to the highest spot price, across the instance types selected for a task. If the ratio exceeds the limit, the scheduler retries the task on spot instead. If spot capacity stays unavailable, the task fails with an error message that starts with `on-demand fallback refused`. -The scheduler applies a default limit for each cloud provider. Use `onDemandCostRatio` to set a different limit for all the tasks of the run: +The scheduler applies a default limit for each cloud provider. To set a different limit for all tasks in the run, use `onDemandCostRatio`: ```groovy seqera { @@ -91,17 +91,17 @@ seqera { } ``` -Leaving `onDemandCostRatio` unset and setting it to `0` are different: +Leaving `onDemandCostRatio` unset is not the same as setting it to `0`: | `onDemandCostRatio` | Behavior | | --- | --- | | Not set | The scheduler's default limit for the cloud provider applies. | -| `0` | No limit. Tasks always fall back to on-demand, whatever the price. | -| Greater than `0` | This limit, used instead of the scheduler's default. | +| `0` | No limit. Tasks always fall back to on-demand, regardless of price. | +| Greater than `0` | The scheduler uses this value instead of its default limit. | -`0` does not disable the fallback. To never fall back to on-demand, set `provisioning = 'spot'`. +Setting `0` does not disable the fallback. To disable it, set `provisioning = 'spot'`. -Like `provisioning`, the limit can be overridden for a single process with a hint. For example, to let a long-running process always fall back to on-demand: +To override the limit for a single process, use a hint, as with `provisioning`. For example, to let a long-running process always fall back to on-demand: ```nextflow process align { @@ -114,7 +114,7 @@ process align { } ``` -Tasks with different limits run on separate clusters. Tasks that set a limit, `0` included, do not use the compute environment's warm pool. +Tasks with different limits run on separate clusters. Tasks that set a limit, including `0`, do not use the compute environment's warm pool. ## Disk support From 16defe0b8c51e2a2dc311c4a8eb01f28252183c0 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 30 Sep 2026 10:05:51 +0200 Subject: [PATCH 6/6] Update docs/reference/config/seqera.mdx [ci skip] Co-authored-by: Chris Hakkaart Signed-off-by: Paolo Di Tommaso --- docs/reference/config/seqera.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/config/seqera.mdx b/docs/reference/config/seqera.mdx index 36c2a9aca2..1025fe8e09 100644 --- a/docs/reference/config/seqera.mdx +++ b/docs/reference/config/seqera.mdx @@ -107,13 +107,13 @@ The maximum number of spot retry attempts before falling back to on-demand. Only -The cost limit on the spot to on-demand fallback. Only used when `provisioning` is `'spotFirst'`. A task falls back to on-demand only while the highest on-demand hourly price of the instance types selected for it is at most this multiple of their highest spot price. Otherwise the task is retried on spot and fails if spot capacity stays unavailable. For example, `onDemandCostRatio = 2.5`. +The cost limit for falling back from spot to on-demand. Only used when `provisioning` is `'spotFirst'`. Across the instance types selected for a task, the scheduler compares the highest on-demand hourly price with the highest spot price. The task falls back to on-demand only while that ratio is at most this value. Otherwise, the scheduler retries the task on spot. If spot capacity stays unavailable, the task fails. For example, `onDemandCostRatio = 2.5`. - Not set (default): the scheduler's default limit for the cloud provider applies. -- `0`: no limit. Tasks always fall back to on-demand, whatever the price. This does not disable the fallback; use `provisioning = 'spot'` to never fall back to on-demand. -- Greater than `0`: this limit, used instead of the scheduler's default. +- `0`: no limit. Tasks always fall back to on-demand, regardless of price. This does not disable the fallback. To never fall back to on-demand, set `provisioning = 'spot'`. +- Greater than `0`: the scheduler uses this value instead of its default limit. -The value must not be negative. It is the default for every task of the run; use the `machineRequirement.onDemandCostRatio` hint to override it for a single process. See [On-demand fallback cost limit][seqera-executor-cost-limit]. +The value must not be negative. It applies to every task in the run. To override it for a single process, use the `machineRequirement.onDemandCostRatio` hint. See [On-demand fallback cost limit][seqera-executor-cost-limit]. ##### `seqera.executor.machineRequirement.provisioning`