Add seqera.executor.machineRequirement.onDemandCostRatio option

docs/executor/seqera.mdx

@@ -38,6 +38,7 @@ The following [hints][process-hints] are supported:
 - `machineRequirement.diskType`
 - `machineRequirement.machineTypes`
 - `machineRequirement.maxSpotAttempts`
+- `machineRequirement.onDemandCostRatio`
 - `machineRequirement.provisioning`
 - `predictionModel`
 
@@ -71,6 +72,50 @@ process hello {
 
 See the [`seqera` scope][config-seqera] for the full config reference.
 
+## On-demand fallback cost limit
+
+<AddedInVersion version="26.10" />
+
+With `provisioning = 'spotFirst'`, the scheduler falls back to on-demand instances when spot capacity runs out, as long as on-demand stays within a cost limit. The limit is the maximum ratio of the highest on-demand hourly price to the highest spot price, across the instance types selected for a task. If the ratio exceeds the limit, the scheduler retries the task on spot instead. If spot capacity stays unavailable, the task fails with an error message that starts with `on-demand fallback refused`.
+
+The scheduler applies a default limit for each cloud provider. To set a different limit for all tasks in the run, use `onDemandCostRatio`:
+
+```groovy
+seqera {
+    executor {
+        machineRequirement {
+            provisioning = 'spotFirst'
+            onDemandCostRatio = 2.5    // fall back only while on-demand costs at most 2.5x spot
+        }
+    }
+}
+```
+
+Leaving `onDemandCostRatio` unset is not the same as setting it to `0`:
+
+| `onDemandCostRatio` | Behavior |
+| --- | --- |
+| Not set | The scheduler's default limit for the cloud provider applies. |
+| `0` | No limit. Tasks always fall back to on-demand, regardless of price. |
+| Greater than `0` | The scheduler uses this value instead of its default limit. |
+
+Setting `0` does not disable the fallback. To disable it, set `provisioning = 'spot'`.
+
+To override the limit for a single process, use a hint, as with `provisioning`. For example, to let a long-running process always fall back to on-demand:
+
+```nextflow
+process align {
+    hints 'seqera/machineRequirement.onDemandCostRatio': 0
+
+    script:
+    """
+    your_command --here
+    """
+}
+```
+
+Tasks with different limits run on separate clusters. Tasks that set a limit, including `0`, do not use the compute environment's warm pool.
+
 ## Disk support
 
 When a process specifies the [disk][process-disk] directive, the Seqera executor provisions storage for the task container. It supports two disk allocation strategies:

docs/reference/config/seqera.mdx

@@ -103,6 +103,18 @@ List of acceptable EC2 instance families. For example, `['m5', 'c5', 'r5']`.
 
 The maximum number of spot retry attempts before falling back to on-demand. Only used when `provisioning` is `'spot'` or `'spotFirst'`.
 
+##### `seqera.executor.machineRequirement.onDemandCostRatio`
+
+<AddedInVersion version="26.10" />
+
+The cost limit for falling back from spot to on-demand. Only used when `provisioning` is `'spotFirst'`. Across the instance types selected for a task, the scheduler compares the highest on-demand hourly price with the highest spot price. The task falls back to on-demand only while that ratio is at most this value. Otherwise, the scheduler retries the task on spot. If spot capacity stays unavailable, the task fails. For example, `onDemandCostRatio = 2.5`.
+
+- Not set (default): the scheduler's default limit for the cloud provider applies.
+- `0`: no limit. Tasks always fall back to on-demand, regardless of price. This does not disable the fallback. To never fall back to on-demand, set `provisioning = 'spot'`.
+- Greater than `0`: the scheduler uses this value instead of its default limit.
+
+The value must not be negative. It applies to every task in the run. To override it for a single process, use the `machineRequirement.onDemandCostRatio` hint. See [On-demand fallback cost limit][seqera-executor-cost-limit].
+
 ##### `seqera.executor.machineRequirement.provisioning`
 
 The instance provisioning mode. Can be `'spot'`, `'ondemand'`, or `'spotFirst'`.
@@ -142,3 +154,4 @@ The multiplier for exponential backoff (default: `2.0`).
 [config-tower-autoLabels]: ./tower#towerautolabels
 [process-resource-labels]: ../process/directives/resource-labels
 [seqera-executor]: ../../executor/seqera
+[seqera-executor-cost-limit]: ../../executor/seqera#on-demand-fallback-cost-limit

gradle.properties

@@ -18,7 +18,7 @@ jacksonCloudSdkVersion=2.18.9
 jaxbApiVersion=2.4.0-b180830.0359
 jimfsVersion=1.2
 kryoVersion=2.24.0
-libHttpxVersion=2.7.0
+libHttpxVersion=2.7.1
 libTraceVersion=0.1.0
 logbackVersion=1.5.38
 nprVersion=0.27.0

plugins/nf-seqera/build.gradle

@@ -53,7 +53,7 @@ dependencies {
     compileOnly "org.pf4j:pf4j:${pf4jVersion}"
 
     // intelligent compute scheduler client
-    api 'io.seqera:sched-client:0.79.3'
+    api 'io.seqera:sched-client:0.96.0-SNAPSHOT'
     // Single lib-httpx declaration, kept in step with the version sched-client is built
     // against. SchedHxClient overrides shouldRetryOnException(HttpRequest, Throwable), an
     // overload added in 2.6.0: on an older lib-httpx the override binds to nothing and the

plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy

@@ -43,6 +43,15 @@ class MachineRequirementOpts implements ConfigScope {
     """)
     final Integer maxSpotAttempts
 
+    @ConfigOption
+    @Description("""
+        Cost limit on the spot to on-demand fallback. With `spotFirst`, a task falls back to on-demand
+        only while on-demand costs at most this multiple of the spot price. When not set, the
+        scheduler's default for the cloud provider applies. Set to `0` to remove the limit, so tasks
+        always fall back whatever the price. Only used when provisioning is `spotFirst`.
+    """)
+    final Double onDemandCostRatio
+
     @ConfigOption
     @Description("""
         List of acceptable machine type patterns. Supports exact types (e.g., `t3.small`),
@@ -112,6 +121,7 @@ class MachineRequirementOpts implements ConfigScope {
     MachineRequirementOpts(Map opts) {
         this.provisioning = opts.provisioning as String
         this.maxSpotAttempts = opts.maxSpotAttempts as Integer
+        this.onDemandCostRatio = opts.onDemandCostRatio as Double
         this.machineTypes = (opts.machineTypes ?: opts.machineFamilies) as List<String>
         this.diskType = opts.diskType as String
         this.diskThroughputMiBps = opts.diskThroughputMiBps as Integer
@@ -133,6 +143,10 @@ class MachineRequirementOpts implements ConfigScope {
         return maxSpotAttempts
     }
 
+    Double getOnDemandCostRatio() {
+        return onDemandCostRatio
+    }
+
     List<String> getMachineTypes() {
         return machineTypes
     }

plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy

@@ -64,11 +64,13 @@ class SchemaMapperUtil {
             return null
         final diskReq = toDiskRequirement(opts.diskSize, opts)
         final capacityMode = toEcsCapacityMode(opts.capacityMode)
-        if (!opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode)
+        final costRatio = toOnDemandCostRatio(opts)
+        if (!opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode && costRatio == null)
             return null
         new MachineRequirement()
             .provisioning(toProvisioningModel(opts.provisioning))
             .maxSpotAttempts(opts.maxSpotAttempts)
+            .onDemandCostRatio(costRatio)
             .machineTypes(opts.machineTypes)
             .disk(diskReq)
             .capacityMode(capacityMode)
@@ -117,19 +119,35 @@ class SchemaMapperUtil {
         final effectiveDiskSize = diskSize ?: opts?.diskSize
         final diskReq = toDiskRequirement(effectiveDiskSize, opts)
         final capacityMode = toEcsCapacityMode(opts?.capacityMode)
+        final costRatio = toOnDemandCostRatio(opts)
         // return null if no settings
-        if (!arch && !provisioning && !maxSpotAttempts && !machineTypes && !diskReq && !snapshotEnabled && !capacityMode)
+        if (!arch && !provisioning && !maxSpotAttempts && !machineTypes && !diskReq && !snapshotEnabled && !capacityMode && costRatio == null)
             return null
         new MachineRequirement()
             .arch(arch)
             .provisioning(toProvisioningModel(provisioning))
             .maxSpotAttempts(maxSpotAttempts)
+            .onDemandCostRatio(costRatio)
             .machineTypes(machineTypes)
             .disk(diskReq)
             .snapshotEnabled(snapshotEnabled ? Boolean.TRUE : null)
             .capacityMode(capacityMode)
     }
 
+    /**
+     * Validates the spot to on-demand fallback cost ratio. Compared against null rather than by
+     * truthiness, because an explicit {@code 0} (no limit) is a real value that must be sent.
+     *
+     * @param opts the config options (can be null)
+     * @return the ratio, or null when not set
+     */
+    static Double toOnDemandCostRatio(MachineRequirementOpts opts) {
+        final ratio = opts?.onDemandCostRatio
+        if (ratio != null && ratio < 0)
+            throw new IllegalArgumentException("Invalid onDemandCostRatio value: ${ratio} -- the value must be zero or a positive number")
+        return ratio
+    }
+
     /**
      * Maps a disk size to DiskRequirement API object.
      * Uses config options if provided, otherwise defaults to Fusion recommended settings:

plugins/nf-seqera/src/test/io/seqera/util/HintHelperTest.groovy

@@ -67,6 +67,16 @@ class HintHelperTest extends Specification {
         result.maxSpotAttempts == 3
     }
 
+    def 'should overlay onDemandCostRatio hint over the config value, zero included'() {
+        given:
+        def base = new MachineRequirementOpts([onDemandCostRatio: 3.25])
+
+        when:
+        def result = HintHelper.overlayHints(base, ['seqera/machineRequirement.onDemandCostRatio': 0])
+        then: 'a task 0 (no limit) is a real value, not "unset"'
+        result.onDemandCostRatio == 0d
+    }
+
     def 'should overlay machineTypes as list'() {
         given:
         def base = new MachineRequirementOpts([:])

plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy

@@ -44,6 +44,32 @@ class MapperUtilTest extends Specification {
         SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([:])) == null
     }
 
+    def 'should send the on-demand cost ratio only when set, including zero' () {
+        when: 'mapped for the run (the default for its tasks) and for a task'
+        def opts = new MachineRequirementOpts([onDemandCostRatio: RATIO])
+        def runReq = SchemaMapperUtil.toMachineRequirement(opts)
+        def taskReq = SchemaMapperUtil.toMachineRequirement(opts, null, null, false, 0)
+
+        then:
+        runReq?.onDemandCostRatio == EXPECTED
+        taskReq?.onDemandCostRatio == EXPECTED
+
+        where:
+        RATIO   | EXPECTED
+        null    | null      // absent: the scheduler's per-cloud default applies
+        0       | 0d        // explicit zero: no limit, must not be dropped
+        2.5     | 2.5d
+    }
+
+    def 'should reject a negative on-demand cost ratio' () {
+        when:
+        SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([onDemandCostRatio: -1]))
+
+        then:
+        def e = thrown(IllegalArgumentException)
+        e.message.contains('onDemandCostRatio')
+    }
+
     def 'should map all fields' () {
         when:
         def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([