From 109429524ffe8e9a4520ded46361b98614e26e5a Mon Sep 17 00:00:00 2001 From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Date: Tue, 21 Jul 2026 18:25:46 +0100 Subject: [PATCH 1/6] docs: Rewrite Azure Batch tutorial for native Batch Forge support Batch Forge now provisions most advanced Azure Batch topologies natively (separate head/worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking). The manual-setup tutorials taught Nextflow-config workarounds that are now redundant, and the Nextflow autopool option was misleading since Platform disables allowPoolCreation. - Restructure both tutorials as a native-first Batch Forge walkthrough - Keep genuinely manual content: low-priority (spot) worker nodes (Forge is dedicated-only) and pre-existing/custom pools - Remove deprecated autopool options; replace with a deprecation note - Add reference cross-links from compute-envs/azure-batch.md to the tutorial Scope: cloud + enterprise latest only; versioned snapshots untouched. Generated by Claude Code --- .../docs/compute-envs/azure-batch.md | 4 + .../manual-azure-batch-setup.md | 345 +++------------- .../compute-envs/azure-batch.md | 4 + .../manual-azure-batch-setup.md | 380 +++--------------- 4 files changed, 111 insertions(+), 622 deletions(-) diff --git a/platform-cloud/docs/compute-envs/azure-batch.md b/platform-cloud/docs/compute-envs/azure-batch.md index d10d7a9a3f..953b41d8b8 100644 --- a/platform-cloud/docs/compute-envs/azure-batch.md +++ b/platform-cloud/docs/compute-envs/azure-batch.md @@ -230,6 +230,10 @@ There are two ways to create an Azure Batch compute environment in Seqera Platfo - [**Batch Forge**](#batch-forge): Automatically creates Azure Batch resources. - [**Manual**](#manual): For using existing Azure Batch resources. +:::tip +For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, Entra authentication, and private networking), see the [Azure Batch setup tutorial](../enterprise/advanced-topics/manual-azure-batch-setup). +::: + ### VM size considerations Azure Batch requires you to select an appropriate VM size for your compute environment. There are a number of considerations when selecting VM sizes — See [Sizes for virtual machines in Azure][az-vm-sizes] for more information. diff --git a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md index f797663825..b50318e856 100644 --- a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -6,7 +6,9 @@ description: "A tutorial for using Azure Batch with Seqera Platform" This guide details how to set up an Azure Batch and Seqera Cloud account to run a workflow in a Seqera Azure Batch compute environment. -It begins with the simplest possible setup and then details more complex environment configuration options. +Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This tutorial walks the native Batch Forge path end-to-end, then covers the cases that still require a manually created pool. + +For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. ## Prerequisites @@ -41,31 +43,11 @@ In Seqera Cloud: - Create a new account. - [Create a new organization and workspace][create-org-workspace]. -- Add a GitHub credential the workspace to prevent API rate-limiting issues with GitHub. - -## Compute environment and pipeline configuration - -### Option 1. Azure Batch with Seqera Batch Forge - -**Behavior**: - -- Seqera Platform will submit a Nextflow job and task to this pool. -- The Nextflow job will execute and submit each task to the same node pool on Azure Batch. -- The node pool will autoscale up and down based on the number of waiting tasks. - -**Advantages**: +- Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -- Simple to set up. -- Low cost. -- Autoscales for number of waiting tasks. +## Part 1. Azure Batch with Batch Forge -**Disadvantages**: - -- The Nextflow job will submit each task to the same node pool on Azure Batch, which can cause bottlenecks. -- Because the processes require larger resources than the head node, you often have oversized machines running Nextflow or undersized machines running processes. -- Dedicated nodes only. - -The first configuration is a simple Azure Batch compute environment created with Batch Forge. This environment uses the same Batch pool for both the Nextflow head job and task nodes. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. First, add the Azure Batch account credentials to Seqera Platform: @@ -76,21 +58,25 @@ First, add the Azure Batch account credentials to Seqera Platform: 1. Enter the Batch account name and key, and Storage account name and key. 1. Select **Create** to save the credentials. -Seqera now has the credentials needed to access your Azure Batch and Storage accounts and make the necessary changes. - Next, create a compute environment with Batch Forge: 1. Go to the **Compute Environments** tab and select **Add Compute Environment**. -1. Enter a name such as `1-azure-batch-forge`. +1. Enter a name such as `azure-batch-forge`. 1. Select Azure Batch from the **Provider** drop-down. 1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. +1. Set **Config mode** to **Batch Forge**. 1. For **VMs type**, select `standard_e2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. +1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. 1. All other options can be left default. Select **Create** to save the compute environment. +:::tip +To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). Set **VMs count** to `1` for the head pool. This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +::: + Add the `nextflow-hello` pipeline to your workspace: [Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: @@ -101,176 +87,20 @@ Add the `nextflow-hello` pipeline to your workspace: Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. -### Option 2. Use a separate node and head pool on Seqera Platform - -**Behavior**: - -- Seqera Platform will submit a Nextflow job and task to the first pool. -- The Nextflow job will execute and submit each task to the second pool. -- Both pools will autoscale up and down based on the number of waiting tasks. - -**Advantages**: - -- The processes are not bottlenecked by the head node. -- You can set the worker nodes to use a different VM size than the head node. - -**Disadvantages**: - -- More complex to set up. -- Still fairly inflexible. -- You have to wait a long time for nodes to autoscale up and down in response to the work. - -This configuration separates head and task nodes into different Batch pools. - -To create a separate node pool to run all the processes: - -1. Create another compute environment in Seqera Platform, exactly as before: - - **Name**: `2-azure-batch-low-priority` or similar - - **Platform**: Azure Batch - - **Credentials**: `azure-keys` - - **Region**: As before - - **Work directory**: As before - - **VMs type**: `standard_e2ds_v5` - - **VMs count**: `4` -1. Note the compute environment ID, which is the first item on the compute environment page. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and create a new pool. -1. Find the pool called `tower-pool-${id}`, where `${id}` is the ID you noted earlier. -1. Select **Scale**. -1. Select **Evaluate**, then **Save**. - -You can now run Nextflow on the first pool, but execute all the processes on the second pool. - -1. On the pipeline launch page, duplicate the existing pipeline, but do not save it yet. -1. Under advanced options, add the following configuration block to the `nextflow.config` text input: - - ```nextflow - process.queue = 'tower-pool-${id}' - ``` - - :::info - Remember to replace `${id}` with the ID of the compute environment you created earlier! - ::: - -1. Save the pipeline as `hello-world-worker`. - -Select **Launch** next to the `hello-world-worker` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 3. Configure the head pool with a hot node - -**Behavior**: - -- A "hot" head node is left running. -- The head node will run Nextflow as soon as the work is created. -- The worker node pool will autoscale up and down based on the number of waiting tasks. - -**Advantages**: - -- The latency of the pipeline is reduced. - -**Disadvantages**: - -- The always-on head node incurs additional cost. - -This configuration separates the head and task pools as before and leaves a single head node up and running to make the response time faster. - -To create the compute environment with a persistent head node: - -1. Get the ID of the first node pool (`1-azure-batch-forge`). -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID you made a note of earlier. -1. Select **Scale**. -1. In the line `targetPoolSize = max(0, min($targetVMs, 4));`, change the `0` to `1`. -1. Select **Evaluate**, then **Save**. - -The node pool will increase to a minimum of 1 node. Now, when you make adjustments to the pipeline, the head node will not be scaled down. - -Select **Launch** next to the `hello-world-worker` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. With this run, it should respond much faster. The _latency_ of the pipeline has improved, although the overall run time will be similar. This effect is more substantial on larger production pipelines. - -:::tip -If you do not wish to continue paying for the head node, scale the node pool back down by replacing the original autoscale formula (`targetPoolSize = max(0, min($targetVMs, 4))`). -You can also delete the compute environment in Platform, which will delete the head node. -::: - -### Option 4. Use the Nextflow autopool feature - -**Behavior**: - -- Seqera will submit a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job will create pools in the Azure Batch account based on the pipeline's requirements. -- The pools are called `nf-pool-${id}`, where `${id}` is a unique identifier for the pool. -- The pools are created with the VM size specified in the Nextflow config. -- The pools are created with the autoscale settings specified in the Nextflow config. - -:::info -Nextflow will create a range of pools based on resource sizes and try to reuse them for similar tasks. This means that if you run a process with different CPU, memory, or machineType, it will create a new pool for that process. -::: - -**Advantages**: - -- Nextflow handles the creation and management of pools. -- You can create flexible pools with the correct VM size and autoscale settings. -- The pools are highly configurable via Nextflow configuration. - -**Disadvantages**: - -- You may be overly specific and end up with a lot of pools, which can exhaust your quota for the maximum number of pools. - -With the autopool feature, Nextflow automatically creates and manages Azure Batch pools based on your pipeline's requirements. - -To configure your pipeline to use Nextflow autopool: - -1. Duplicate the `hello-world-worker` pipeline to a new pipeline called `hello-world-autopool`. -1. Update your Nextflow config to use autopool mode: - -```groovy -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - } - } - } -} -``` - -3. Save the pipeline. - -Select **Launch** next to the `hello-world-autopool` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 5. Use Entra authentication - -**Behavior**: - -- Seqera authenticates to Azure Batch and Azure Storage using a service principal. -- It submits a job and task to the Azure Batch service using the service principal. -- The task runs Nextflow, which authenticates to Azure Batch and Azure Storage using the managed identity. -- All processes run on the head node as in the first example. +## Part 2. Use Entra authentication **Advantages**: - No keys or short access tokens are exchanged, increasing security. - A service principal can have very granular permissions, so you can grant it only the permissions it needs. - Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed IDs can have different permissions, so different compute environments can have different scoped permissions. - -**Disadvantages**: +- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. -- The setup is quite complicated with room for error. -- Errors can be harder to troubleshoot. - -Seqera can utilize an Azure Entra service principal to authenticate and access Azure Batch for job execution and Azure Storage for data management, and Nextflow can authenticate to Azure services using a managed identity. This method offers enhanced security compared to access keys, but must run on Azure infrastructure. +Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. -#### Create a service principal for Seqera to use for authentication +### Create a service principal for Seqera to use for authentication 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). @@ -283,143 +113,63 @@ In Seqera: 1. Add the Service Principal ID, Tenant ID and Client Secret. 1. Select **Create** to save the credentials. -#### Create a managed identity for Nextflow to use for authentication +### Create a managed identity for Nextflow to use for authentication -Back in the Azure Portal: +In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. 1. Note the managed identity client ID for later. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID of the head node pool created earlier. -1. Select **Identity**. -1. Select **Add User Assigned Identity**. -1. Select the managed identity created earlier. -1. Select **Add**. - -Processes running on this pool can now use the managed identity to authenticate to Azure Batch and Storage. In Seqera: -1. Add a new compute environment with the name `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the pool you added the managed identity to earlier (`tower-pool-${id}`). -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the `hello-world-autopool` pipeline and save it as `hello-world-entra-mi` but use the new compute environment. - -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -The pipeline will run as before, but using the managed identity to authenticate to Azure Batch and Storage. No keys or storage required. - -:::note -You can also use User Subscription mode instead of Batch Managed here, but this is beyond the scope of this tutorial. -::: +1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -### Option 6. Use a node pool attached to a VNet +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -**Behavior**: +Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. -- Each node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. -- All other behavior is as normal. +## Part 3. Attach the pool to a virtual network (VNet) **Advantages**: - Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. +- Exchange of data can be faster and cheaper than routing over the public internet. -**Disadvantages**: +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. -- It requires fairly complicated setup. -- If security is too restrictive, it can fail silently and be unable to report the error state. - -It is common to attach Azure Batch pools to a virtual network. This is useful to connect to other resources in the same VNet or place things behind enhanced security. Seqera Platform does not support this feature directly, so you must manually create an Azure Batch pool. - -See [Create a Nextflow-compatible Azure Batch pool](../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool) to create an Azure Batch pool manually that is compatible with Seqera and Nextflow. Use the following settings: - -- Name & ID: `3-azure-batch-vnet` -- Add the managed identity created earlier as a user-assigned managed identity. -- VMs type: `standard_e2ds_v5` -- Use the autoscale formula described in the documentation, with a minimum size of 0 and a maximum size of 4. -- For Virtual network, create a new virtual network with the default subnet. You can add this to a new resource group here. - -In practice, you are more likely to connect an Azure Batch Node pool to an existing virtual network that is connected to other resources, such as Seqera Platform or the Azure Storage Account. In this instance, connecting it to a VNet with public internet access will route the network traffic via the virtual network while still allowing you to perform every action. - -Back in Seqera Platform, add a new Azure Batch compute environment: +:::note +VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +::: -1. Add a new compute environment with the name `3-azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Credentials**, select the service principal credentials. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the Compute pool name `3-azure-batch-vnet`. -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the **original** `hellow-world` pipeline and save it as `hello-world-vnet`. - -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -The pipeline runs as before, but it will run on the node pool attached to the VNet. It will resemble a normal Azure Batch pipeline run. - -Using this technique allows you to run pipelines on Azure Batch with more restrictive networking and security requirements. - -### Option 7. Use a node pool attached to a VNet with worker nodes attached to the same VNet - -**Behavior**: +1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: -- We use a separate head node pool to run Nextflow, along with automatically created Nextflow autoscale pools to run processes. -- Each worker node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. - -**Advantages**: - -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. -- Additionally, you get the advantages of using worker nodes with autopools. - -**Disadvantages**: - -- The set up is very complicated now and errors are likely to occur. -- Errors can be hard to troubleshoot. - -Finally, you can combine some of the previous approaches. Nextflow can create and modify Azure Batch pools based on the pipeline requirements. You can also attach Azure Batch pools to a VNet. Next, attach the worker nodes to the same VNet. - -To achieve this, the following requirements must be met: + ``` + /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} + ``` -- The pipeline must be launched on the node pool attached to the VNet. -- The managed identity must be used to authenticate to Azure Batch and Storage. -- The managed identity must have permissions to create resources attached to the VNet. -- Nextflow creates node pools attached to the VNet. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -Do the following: +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. -1. Duplicate the `hello-world-entra-mi` pipeline, but modify the compute environment to `3-azure-batch-vnet` and change the pipeline name to `hello-world-vnet`. -1. Check the virtual network string under the pool details in the Azure Portal, under the **Network Configuration** section. The value should be a Subnet ID, such as `/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}`. Save this value. -1. Change the Nextflow configuration under the **Advanced** tab to include a virtual network with the autopools: +## Advanced. Use a pre-existing Azure Batch pool -```nextflow -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - virtualNetwork = '/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}' - } - } - } -} -``` +Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. +See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. -The pipeline runs as before, but using the managed identity to authenticate to Azure Batch and Storage. It also creates worker pools attached to the VNet. +:::note +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +::: -### Clear up resources +## Clear up resources Once you have completed setup and workflow execution, you can delete the pipelines and compute environments from Seqera. @@ -432,3 +182,6 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [nextflow-working-directory]: https://docs.seqera.io/nextflow/cache-and-resume#work-directory [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad +[azure-batch-reference]: ../../compute-envs/azure-batch.md +[azure-batch-entra]: ../../compute-envs/azure-batch.md#entra-service-principal-and-managed-identity +[azure-batch-manual-pool]: ../../compute-envs/azure-batch.md#create-a-nextflow-compatible-azure-batch-pool diff --git a/platform-enterprise_docs/compute-envs/azure-batch.md b/platform-enterprise_docs/compute-envs/azure-batch.md index 8976517d73..005caff02b 100644 --- a/platform-enterprise_docs/compute-envs/azure-batch.md +++ b/platform-enterprise_docs/compute-envs/azure-batch.md @@ -235,6 +235,10 @@ There are two ways to create an Azure Batch compute environment in Platform: - [**Batch Forge**](#batch-forge): Automatically creates Azure Batch resources. - [**Manual**](#manual): For using existing Azure Batch resources. +:::tip +For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, low-priority nodes, Entra authentication, and private networking), see the [Azure Batch walkthrough](../enterprise/advanced-topics/manual-azure-batch-setup). +::: + ### VM size considerations Azure Batch requires you to select an appropriate VM size for your compute environment. There are a number of considerations when selecting VM sizes. See [Sizes for virtual machines in Azure][azure-vm-sizes-overview] for more information. diff --git a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md index 1c0afdb704..8e1754e03e 100644 --- a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -6,7 +6,9 @@ description: "A tutorial for using advanced features of Azure Batch with Seqera This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. -The first step indicates how to configure a simple Azure Batch compute environment on Azure and Seqera Platform, however beyond that is not required for most users and is only recommended for those who need to customize their compute environments. +Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then the cases that still require manual Azure configuration, such as low-priority (spot) nodes and pre-existing pools. + +For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. :::info Prerequisites - An Azure account with sufficient permissions to create resources. @@ -43,31 +45,11 @@ In Seqera Cloud: - Create a new account. - [Create a new organization and workspace][create-org-workspace]. -- Add a GitHub credential the workspace to prevent API rate-limiting issues with GitHub. - -## Compute environment and pipeline configuration - -### Option 1. Azure Batch with Seqera Batch Forge - -**Behavior**: +- Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -- Seqera Platform will submit a Nextflow job and task to this pool. -- The Nextflow job will execute and submit each task to the same node pool on Azure Batch. -- The node pool will autoscale up and down based on the number of waiting tasks. +## Part 1. Azure Batch with Batch Forge -**Advantages**: - -- Simple to set up. -- Low cost. -- Autoscales for number of waiting tasks. - -**Disadvantages**: - -- The Nextflow job will submit each task to the same node pool on Azure Batch, which can cause bottlenecks. -- Because the processes require larger resources than the head node, you often have oversized machines running Nextflow or undersized machines running processes. -- Dedicated nodes only. - -The first configuration is a simple Azure Batch compute environment created with Batch Forge. This environment uses the same Batch pool for both the Nextflow head job and task nodes. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. First, add the Azure Batch account credentials to Seqera Platform: @@ -78,21 +60,25 @@ First, add the Azure Batch account credentials to Seqera Platform: 1. Enter the Batch account name and key, and Storage account name and key. 1. Select **Create** to save the credentials. -Seqera now has the credentials needed to access your Azure Batch and Storage accounts and make the necessary changes. - Next, create a compute environment with Batch Forge: 1. Go to the **Compute Environments** tab and select **Add Compute Environment**. -1. Enter a name such as `1-azure-batch-forge`. +1. Enter a name such as `azure-batch-forge`. 1. Select Azure Batch from the **Provider** drop-down. 1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. +1. Set **Config mode** to **Batch Forge**. 1. For **VMs type**, select `standard_e2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. +1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. 1. All other options can be left default. Select **Create** to save the compute environment. +:::tip +To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +::: + Add the `nextflow-hello` pipeline to your workspace: [Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: @@ -103,222 +89,37 @@ Add the `nextflow-hello` pipeline to your workspace: Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. -### Option 2. Use a separate node and head pool on Seqera Platform - -**Behavior**: - -- Seqera Platform will submit a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job will execute and submit each task to the second pool, which uses low-priority VMs. -- Both pools will autoscale up and down based on the number of waiting tasks. +## Part 2. Use low-priority (spot) worker nodes **Advantages**: -- The processes are not bottlenecked by the head node. -- You can set the worker nodes to use a different VM size than the head node. -- Cheaper nodes for work than for running Nextflow. +- Low-priority (spot) VMs are cheaper than dedicated VMs. +- You keep the head node on dedicated VMs while running the cheaper work on the worker pool. **Disadvantages**: -- More complex to set up. -- Still fairly inflexible. -- You have to wait a long time for nodes to autoscale up and down in response to the work. - -This configuration separates head and task nodes into different Batch pools. - -To create a separate node pool to run all the processes: - -1. Create another compute environment in Seqera Platform, exactly as before: - - **Name**: `2-azure-batch-low-priority` or similar - - **Platform**: Azure Batch - - **Credentials**: `azure-keys` - - **Region**: As before - - **Pipeline work directory**: As before - - **VMs type**: `standard_e2ds_v5` - - **VMs count**: `4` -1. Note the compute environment ID, which is the first item on the compute environment page. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID you made a note of earlier. -1. Select **Scale**. -1. An Autoscale formula is displayed. On the second-to-last line, there will be a line that starts with `$TargetDedicatedNodes`. Change this string to `$TargetLowPriorityNodes`. -1. Select **Evaluate**, then **Save**. - -You have created a new node pool that uses low-priority VMs, which are cheaper than dedicated VMs. You can now run Nextflow on the first pool, but execute all the processes on the second pool. - -1. On the pipeline launch page, duplicate the existing pipeline, but do not save it yet. -1. Under advanced options, add the following configuration block to the `nextflow.config` text input: - - ```nextflow - process.queue = 'tower-pool-${id}' - ``` - - :::info - Remember to replace `${id}` with the ID of the compute environment you created earlier! - ::: - -1. Save the pipeline as `hello-world-low-priority`. - -Select **Launch** next to the `hello-world-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 3. Configure the head pool with a hot node - -**Behavior**: - -- A "hot" head node is left running. -- The head node will run Nextflow as soon as the work is created. -- The worker node pool will autoscale up and down based on the number of waiting tasks. - -**Advantages**: - -- The latency of the pipeline is reduced. - -**Disadvantages**: - -- The always-on head node incurs additional cost. - -This configuration separates the head and task pools as before and leaves a single head node up and running to make the response time faster. +- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. -To create the compute environment with a persistent head node: +Batch Forge provisions dedicated nodes only. To use low-priority (spot) nodes for the compute tasks, edit the autoscale formula of the Forge-created worker pool to target low-priority nodes: -1. Get the ID of the first node pool (`1-azure-batch-forge`). 1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID you made a note of earlier. +1. Go to the **Pools** tab and find the worker pool called `tower-pool-{envId}-worker`, where `{envId}` is the compute environment ID (shown at the top of the compute environment page in Seqera). 1. Select **Scale**. -1. In the line `targetPoolSize = max(0, min($targetVMs, 4));`, change the `0` to `1`. +1. In the autoscale formula, find the line that starts with `$TargetDedicatedNodes` and change it to `$TargetLowPriorityNodes`. 1. Select **Evaluate**, then **Save**. -The node pool will increase to a minimum of 1 node. Now, when you make adjustments to the pipeline, the head node will not be scaled down. - -Select **Launch** next to the `hello-world-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. With this run, it should respond much faster. The _latency_ of the pipeline has improved, although the overall run time will be similar. This effect is more substantial on larger production pipelines. - -:::tip -If you do not wish to continue paying for the head node, scale the node pool back down by replacing the original autoscale formula (`targetPoolSize = max(0, min($targetVMs, 4))`). -You can also delete the compute environment in Platform, which will delete the head node. -::: - -### Option 4. Use the Nextflow autopool feature - -**Behavior**: - -- Seqera will submit a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job will create pools in the Azure Batch account based on the pipeline's requirements. -- The pools are called `nf-pool-${id}`, where `${id}` is a unique identifier for the pool. -- The pools are created with the VM size specified in the Nextflow config. -- The pools are created with the autoscale settings specified in the Nextflow config. - -:::info -Nextflow will create a range of pools based on resource sizes and try to reuse them for similar tasks. This means that if you run a process with different CPU, memory, or machineType, it will create a new pool for that process. -::: - -**Advantages**: - -- Nextflow handles the creation and management of pools. -- You can create flexible pools with the correct VM size and autoscale settings. -- The pools are highly configurable via Nextflow configuration. - -**Disadvantages**: - -- You may be overly specific and end up with a lot of pools, which can exhaust your quota for the maximum number of pools. -- This configuration does not use low-priority nodes. - -With the autopool feature, Nextflow automatically creates and manages Azure Batch pools based on your pipeline's requirements. - -To configure your pipeline to use Nextflow autopool: +The worker pool now provisions low-priority VMs, while the head pool continues to use dedicated VMs. Launch the `nextflow-hello` pipeline again to run the compute tasks on low-priority nodes. -1. Duplicate the `hello-world-low-priority` pipeline to a new pipeline called `hello-world-autopool`. -1. Update your Nextflow config to use autopool mode: - -```groovy -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - } - } - } -} -``` - -3. Save the pipeline. - -Select **Launch** next to the `hello-world-autopool` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 5. Use Nextflow autopool feature with low-priority nodes - -**Behavior**: - -- Seqera submits a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job creates pools in the Azure Batch account based on the pipeline's requirements. -- The pools are named `nf-pool-${id}`, where `${id}` is a unique identifier for the pool. -- The pools are created with the VM size specified in the Nextflow config. -- The pools are created with the autoscale settings specified in the Nextflow config. -- These pools use low-priority nodes. It achieves this by modifying the autoscale formula. - -**Advantages**: - -- Nextflow handles the creation and management of pools. -- You can create flexible pools with the correct VM size and autoscale settings. -- This configuration uses low-priority nodes. - -**Disadvantages**: - -- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. - -To configure your pipeline to use Nextflow autopool with low-priority nodes: - -1. Duplicate the `hello-world-autopool` pipeline to a new pipeline called `hello-world-autopool-low-priority`. -1. Update your Nextflow config to use low-priority nodes: - -```groovy -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - } - } - } -} -``` - -3. Save the pipeline. - -Select **Launch** next to the `hello-world-autopool-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 6. Use Entra authentication - -**Behavior**: - -- Seqera authenticates to Azure Batch and Azure Storage using a service principal. -- It submits a job and task to the Azure Batch service using the service principal. -- The task runs Nextflow, which authenticates to Azure Batch and Azure Storage using the managed identity. -- All processes run on the head node as in the first example. +## Part 3. Use Entra authentication **Advantages**: - No keys or short access tokens are exchanged, increasing security. - A service principal can have very granular permissions, so you can grant it only the permissions it needs. - Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed IDs can have different permissions, so different compute environments can have different scoped permissions. - -**Disadvantages**: - -- The setup is quite complicated with room for error. -- Errors can be harder to troubleshoot. +- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. -Seqera can utilize an Azure Entra service principal to authenticate and access Azure Batch for job execution and Azure Storage for data management, and Nextflow can authenticate to Azure services using a managed identity. This method offers enhanced security compared to access keys, but must run on Azure infrastructure. +Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. @@ -337,139 +138,63 @@ In Seqera: #### Create a managed identity for Nextflow to use for authentication -Back in the Azure Portal: +In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. 1. Note the managed identity client ID for later. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID of the head node pool created earlier. -1. Select **Identity**. -1. Select **Add User Assigned Identity**. -1. Select the managed identity created earlier. -1. Select **Add**. - -Processes running on this pool can now use the managed identity to authenticate to Azure Batch and Storage. In Seqera: -1. Add a new compute environment with the name `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the pool you added the managed identity to earlier (`tower-pool-${id}`). -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the `hello-world-autopool-low-priority` pipeline and save it as `hello-world-entra-mi`. +1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -The pipeline will run as before, but using the managed identity to authenticate to Azure Batch and Storage. No keys or storage required. +Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. :::note You can also use User Subscription mode instead of Batch Managed here, but this is beyond the scope of this tutorial. ::: -### Option 7. Use a node pool attached to a VNet - -**Behavior**: - -- Each node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. -- All other behavior is as normal. +## Part 4. Attach the pool to a virtual network (VNet) **Advantages**: - Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. - -**Disadvantages**: - -- It requires fairly complicated setup. -- If security is too restrictive, it can fail silently and be unable to report the error state. - -It is common to attach Azure Batch pools to a virtual network. This is useful to connect to other resources in the same VNet or place things behind enhanced security. Seqera Platform does not support this feature directly, so you must manually create an Azure Batch pool. - -See [Create a Nextflow-compatible Azure Batch pool](../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool) to create an Azure Batch pool manually that is compatible with Seqera and Nextflow. Use the following settings: - -- Name & ID: `3-azure-batch-vnet` -- Add the managed identity created earlier as a user-assigned managed identity. -- VMs type: `standard_e2ds_v5` -- Use the autoscale formula described in the documentation, with a minimum size of 0 and a maximum size of 4. -- For Virtual network, create a new virtual network with the default subnet. You can add this to a new resource group here. +- Exchange of data can be faster and cheaper than routing over the public internet. -In practice, you are more likely to connect an Azure Batch Node pool to an existing virtual network that is connected to other resources, such as Seqera Platform or the Azure Storage Account. In this instance, connecting it to a VNet with public internet access will route the network traffic via the virtual network while still allowing you to perform every action. +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. -Back in Seqera Platform, add a new Azure Batch compute environment: +:::note +VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +::: -1. Add a new compute environment with the name `3-azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Credentials**, select the service principal credentials. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the Compute pool name `3-azure-batch-vnet`. -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the **original** `hellow-world` pipeline and save it as `hello-world-vnet`. - -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -The pipeline runs as before, but it will run on the node pool attached to the VNet. It will resemble a normal Azure Batch pipeline run. - -Using this technique allows you to run pipelines on Azure Batch with more restrictive networking and security requirements. - -### Option 8. Use a node pool attached to a VNet with worker nodes attached to the same VNet +1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: -**Behavior**: - -- We use a separate head node pool to run Nextflow, along with automatically created Nextflow autoscale pools to run processes. -- Each worker node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. - -**Advantages**: - -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. -- Additionally, you get the advantages of using worker nodes with autopools. - -**Disadvantages**: - -- The set up is very complicated now and errors are likely to occur. -- Errors can be hard to troubleshoot. - -Finally, you can combine some of the previous approaches. Nextflow can create and modify Azure Batch pools based on the pipeline requirements. You can also attach Azure Batch pools to a VNet. Next, attach the worker nodes to the same VNet. - -To achieve this, the following requirements must be met: + ``` + /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} + ``` -- The pipeline must be launched on the node pool attached to the VNet. -- The managed identity must be used to authenticate to Azure Batch and Storage. -- The managed identity must have permissions to create resources attached to the VNet. -- Nextflow creates node pools attached to the VNet. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -Do the following: +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. -1. Duplicate the `hello-world-entra-mi` pipeline, but modify the compute environment to `3-azure-batch-vnet` and change the pipeline name to `hello-world-vnet`. -1. Check the virtual network string under the pool details in the Azure Portal, under the **Network Configuration** section. The value should be a Subnet ID, such as `/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}`. Save this value. -1. Change the Nextflow configuration under the **Advanced** tab to include a virtual network with the autopools: +## Advanced. Use a pre-existing Azure Batch pool -```nextflow -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - virtualNetwork = '/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}' - } - } - } -} -``` +Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. +See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. -The pipeline runs as before, but using the managed identity to authenticate to Azure Batch and Storage. It also creates worker pools attached to the VNet. +:::note +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +::: ### Clear up resources @@ -484,3 +209,6 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [nextflow-working-directory]: https://docs.seqera.io/nextflow/cache-and-resume#work-directory [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad +[azure-batch-reference]: ../../compute-envs/azure-batch +[azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity +[azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool From df72370fc07ee2c94fc5d5acdad7a4a6ad1d7de4 Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Wed, 22 Jul 2026 10:49:57 +0200 Subject: [PATCH 2/6] docs: Port Azure Batch tutorial rewrite to versioned 26.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies the same native-Batch-Forge rewrite to the versioned 26.1 snapshot so the current release surface matches master. Preserves the versioned-tree convention of using an absolute URL for the Seqera CLI installation link. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) --- .../version-26.1/compute-envs/azure-batch.md | 4 + .../manual-azure-batch-setup.md | 384 +++--------------- 2 files changed, 60 insertions(+), 328 deletions(-) diff --git a/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md b/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md index 74227fe567..62a5fca9e2 100644 --- a/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md +++ b/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md @@ -235,6 +235,10 @@ There are two ways to create an Azure Batch compute environment in Platform: - [**Batch Forge**](#batch-forge): Automatically creates Azure Batch resources. - [**Manual**](#manual): For using existing Azure Batch resources. +:::tip +For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, low-priority nodes, Entra authentication, and private networking), see the [Azure Batch walkthrough](../enterprise/advanced-topics/manual-azure-batch-setup). +::: + ### VM size considerations Azure Batch requires you to select an appropriate VM size for your compute environment. There are a number of considerations when selecting VM sizes. See [Sizes for virtual machines in Azure][azure-vm-sizes-overview] for more information. diff --git a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md index b34ea3b682..64b9b7c2f0 100644 --- a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -6,7 +6,9 @@ description: "A tutorial for using advanced features of Azure Batch with Seqera This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. -The first step indicates how to configure a simple Azure Batch compute environment on Azure and Seqera Platform, however beyond that is not required for most users and is only recommended for those who need to customize their compute environments. +Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then the cases that still require manual Azure configuration, such as low-priority (spot) nodes and pre-existing pools. + +For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. :::info Prerequisites - An Azure account with sufficient permissions to create resources. @@ -43,31 +45,11 @@ In Seqera Cloud: - Create a new account. - [Create a new organization and workspace][create-org-workspace]. -- Add a GitHub credential the workspace to prevent API rate-limiting issues with GitHub. - -## Compute environment and pipeline configuration - -### Option 1. Azure Batch with Seqera Batch Forge - -**Behavior**: +- Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -- Seqera Platform will submit a Nextflow job and task to this pool. -- The Nextflow job will execute and submit each task to the same node pool on Azure Batch. -- The node pool will autoscale up and down based on the number of waiting tasks. +## Part 1. Azure Batch with Batch Forge -**Advantages**: - -- Simple to set up. -- Low cost. -- Autoscales for number of waiting tasks. - -**Disadvantages**: - -- The Nextflow job will submit each task to the same node pool on Azure Batch, which can cause bottlenecks. -- Because the processes require larger resources than the head node, you often have oversized machines running Nextflow or undersized machines running processes. -- Dedicated nodes only. - -The first configuration is a simple Azure Batch compute environment created with Batch Forge. This environment uses the same Batch pool for both the Nextflow head job and task nodes. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. First, add the Azure Batch account credentials to Seqera Platform: @@ -78,21 +60,25 @@ First, add the Azure Batch account credentials to Seqera Platform: 1. Enter the Batch account name and key, and Storage account name and key. 1. Select **Create** to save the credentials. -Seqera now has the credentials needed to access your Azure Batch and Storage accounts and make the necessary changes. - Next, create a compute environment with Batch Forge: 1. Go to the **Compute Environments** tab and select **Add Compute Environment**. -1. Enter a name such as `1-azure-batch-forge`. +1. Enter a name such as `azure-batch-forge`. 1. Select Azure Batch from the **Provider** drop-down. -1. Sellect your `azure-keys` credentials. +1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. +1. Set **Config mode** to **Batch Forge**. 1. For **VMs type**, select `standard_e2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. +1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. 1. All other options can be left default. Select **Create** to save the compute environment. +:::tip +To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +::: + Add the `nextflow-hello` pipeline to your workspace: [Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: @@ -103,222 +89,37 @@ Add the `nextflow-hello` pipeline to your workspace: Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. -### Option 2. Use a separate node and head pool on Seqera Platform - -**Behavior**: - -- Seqera Platform will submit a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job will execute and submit each task to the second pool, which uses low-priority VMs. -- Both pools will autoscale up and down based on the number of waiting tasks. +## Part 2. Use low-priority (spot) worker nodes **Advantages**: -- The processes are not bottlenecked by the head node. -- You can set the worker nodes to use a different VM size than the head node. -- Cheaper nodes for work than for running Nextflow. +- Low-priority (spot) VMs are cheaper than dedicated VMs. +- You keep the head node on dedicated VMs while running the cheaper work on the worker pool. **Disadvantages**: -- More complex to set up. -- Still fairly inflexible. -- You have to wait a long time for nodes to autoscale up and down in response to the work. - -This configuration separates head and task nodes into different Batch pools. - -To create a separate node pool to run all the processes: - -1. Create another compute environment in Seqera Platform, exactly as before: - - **Name**: `2-azure-batch-low-priority` or similar - - **Platform**: Azure Batch - - **Credentials**: `azure-keys` - - **Region**: As before - - **Pipeline work directory**: As before - - **VMs type**: `standard_e2ds_v5` - - **VMs count**: `4` -1. Note the compute environment ID, which is the first item on the compute environment page. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID you made a note of earlier. -1. Select **Scale**. -1. An Autoscale formula is displayed. On the second-to-last line, there will be a line that starts with `$TargetDedicatedNodes`. Change this string to `$TargetLowPriorityNodes`. -1. Select **Evaluate**, then **Save**. - -You have created a new node pool that uses low-priority VMs, which are cheaper than dedicated VMs. You can now run Nextflow on the first pool, but execute all the processes on the second pool. - -1. On the pipeline launch page, duplicate the existing pipeline, but do not save it yet. -1. Under advanced options, add the following configuration block to the `nextflow.config` text input: - - ```nextflow - process.queue = 'tower-pool-${id}' - ``` - - :::info - Remember to replace `${id}` with the ID of the compute environment you created earlier! - ::: - -1. Save the pipeline as `hello-world-low-priority`. - -Select **Launch** next to the `hello-world-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 3. Configure the head pool with a hot node - -**Behavior**: - -- A "hot" head node is left running. -- The head node will run Nextflow as soon as the work is created. -- The worker node pool will autoscale up and down based on the number of waiting tasks. - -**Advantages**: - -- The latency of the pipeline is reduced. - -**Disadvantages**: - -- The always-on head node incurs additional cost. - -This configuration separates the head and task pools as before and leaves a single head node up and running to make the response time faster. +- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. -To create the compute environment with a persistent head node: +Batch Forge provisions dedicated nodes only. To use low-priority (spot) nodes for the compute tasks, edit the autoscale formula of the Forge-created worker pool to target low-priority nodes: -1. Get the ID of the first node pool (`1-azure-batch-forge`). 1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID you made a note of earlier. +1. Go to the **Pools** tab and find the worker pool called `tower-pool-{envId}-worker`, where `{envId}` is the compute environment ID (shown at the top of the compute environment page in Seqera). 1. Select **Scale**. -1. In the line `targetPoolSize = max(0, min($targetVMs, 4));`, change the `0` to `1`. +1. In the autoscale formula, find the line that starts with `$TargetDedicatedNodes` and change it to `$TargetLowPriorityNodes`. 1. Select **Evaluate**, then **Save**. -The node pool will increase to a minimum of 1 node. Now, when you make adjustments to the pipeline, the head node will not be scaled down. - -Select **Launch** next to the `hello-world-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. With this run, it should respond much faster. The _latency_ of the pipeline has improved, although the overall run time will be similar. This effect is more substantial on larger production pipelines. - -:::tip -If you do not wish to continue paying for the head node, scale the node pool back down by replacing the original autoscale formula (`targetPoolSize = max(0, min($targetVMs, 4))`). -You can also delete the compute environment in Platform, which will delete the head node. -::: - -### Option 4. Use the Nextflow autopool feature - -**Behavior**: - -- Seqera will submit a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job will create pools in the Azure Batch account based on the pipeline's requirements. -- The pools are called `nf-pool-${id}`, where `${id}` is a unique identifier for the pool. -- The pools are created with the VM size specified in the Nextflow config. -- The pools are created with the autoscale settings specified in the Nextflow config. - -:::info -Nextflow will create a range of pools based on resource sizes and try to reuse them for similar tasks. This means that if you run a process with different CPU, memory, or machineType, it will create a new pool for that process. -::: - -**Advantages**: - -- Nextflow handles the creation and management of pools. -- You can create flexible pools with the correct VM size and autoscale settings. -- The pools are highly configurable via Nextflow configuration. - -**Disadvantages**: - -- You may be overly specific and end up with a lot of pools, which can exhaust your quota for the maximum number of pools. -- This configuration does not use low-priority nodes. - -With the autopool feature, Nextflow automatically creates and manages Azure Batch pools based on your pipeline's requirements. - -To configure your pipeline to use Nextflow autopool: +The worker pool now provisions low-priority VMs, while the head pool continues to use dedicated VMs. Launch the `nextflow-hello` pipeline again to run the compute tasks on low-priority nodes. -1. Duplicate the `hello-world-low-priority` pipeline to a new pipeline called `hello-world-autopool`. -1. Update your Nextflow config to use autopool mode: - -```groovy -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - } - } - } -} -``` - -3. Save the pipeline. - -Select **Launch** next to the `hello-world-autopool` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 5. Use Nextflow autopool feature with low-priority nodes - -**Behavior**: - -- Seqera submits a Nextflow job and task to the first pool, which uses dedicated VMs. -- The Nextflow job creates pools in the Azure Batch account based on the pipeline's requirements. -- The pools are named `nf-pool-${id}`, where `${id}` is a unique identifier for the pool. -- The pools are created with the VM size specified in the Nextflow config. -- The pools are created with the autoscale settings specified in the Nextflow config. -- These pools use low-priority nodes. It achieves this by modifying the autoscale formula. - -**Advantages**: - -- Nextflow handles the creation and management of pools. -- You can create flexible pools with the correct VM size and autoscale settings. -- This configuration uses low-priority nodes. - -**Disadvantages**: - -- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. - -To configure your pipeline to use Nextflow autopool with low-priority nodes: - -1. Duplicate the `hello-world-autopool` pipeline to a new pipeline called `hello-world-autopool-low-priority`. -1. Update your Nextflow config to use low-priority nodes: - -```groovy -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - } - } - } -} -``` - -3. Save the pipeline. - -Select **Launch** next to the `hello-world-autopool-low-priority` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -### Option 6. Use Entra authentication - -**Behavior**: - -- Seqera authenticates to Azure Batch and Azure Storage using a service principal. -- It submits a job and task to the Azure Batch service using the service principal. -- The task runs Nextflow, which authenticates to Azure Batch and Azure Storage using the managed identity. -- All processes run on the head node as in the first example. +## Part 3. Use Entra authentication **Advantages**: - No keys or short access tokens are exchanged, increasing security. -- A service prinicipal can have very granular permissions, so you can grant it only the permissions it needs. +- A service principal can have very granular permissions, so you can grant it only the permissions it needs. - Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed IDs can have different permissions, so different compute environments can have different scoped permissions. - -**Disadvantages**: - -- The setup is quite complicated with room for error. -- Errors can be harder to troubleshoot. +- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. -Seqera can utilize an Azure Entra service principal to authenticate and access Azure Batch for job execution and Azure Storage for data management, and Nextflow can authenticate to Azure services using a managed identity. This method offers enhanced security compared to access keys, but must run on Azure infrastructure. +Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. @@ -337,139 +138,63 @@ In Seqera: #### Create a managed identity for Nextflow to use for authentication -Back in the Azure Portal: +In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. 1. Note the managed identity client ID for later. -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the pool called `tower-pool-${id}`, where `${id}` is the ID of the head node pool created earlier. -1. Select **Identity**. -1. Select **Add User Assigned Identity**. -1. Select the managed identity created earlier. -1. Select **Add**. - -Processes running on this pool can now use the managed identity to authenticate to Azure Batch and Storage. In Seqera: -1. Add a new compute environment with the name `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the pool you added the managed identity to earlier (`tower-pool-${id}`). -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the `hello-world-autopool-low-priority` pipeline and save it as `hello-world-entra-mi`. +1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -The pipeline will run as before, but using the managed identity to authenticate to Azure Batch and Storage. No keys or storage required. +Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. :::note You can also use User Subscription mode instead of Batch Managed here, but this is beyond the scope of this tutorial. ::: -### Option 7. Use a node pool attached to a VNet - -**Behavior**: - -- Each node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. -- All other behaviour is as normal. +## Part 4. Attach the pool to a virtual network (VNet) **Advantages**: - Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. - -**Disadvantages**: - -- It requires fairly complicated setup. -- If security is too restrictive, it can fail silently and be unable to report the error state. - -It is common to attach Azure Batch pools to a virtual network. This is useful to connect to other resources in the same VNet or place things behind enhanced security. Seqera Platform does not support this feature directly, so you must manually create an Azure Batch pool. - -See [Create a Nextflow-compatible Azure Batch pool](../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool) to create an Azure Batch pool manually that is compatible with Seqera and Nextflow. Use the following settings: - -- Name & ID: `3-azure-batch-vnet` -- Add the managed identity created earlier as a user-assigned managed identity. -- VMs type: `standard_e2ds_v5` -- Use the autoscale formula described in the documentation, with a minimum size of 0 and a maximum size of 4. -- For Virtual network, create a new virtual network with the default subnet. You can add this to a new resource group here. +- Exchange of data can be faster and cheaper than routing over the public internet. -In practice, you are more likely to connect an Azure Batch Node pool to an existing virtual network that is connected to other resources, such as Seqera Platform or the Azure Storage Acccount. In this instance, connecting it to a VNet with public internet access will route the network traffic via the virtual network while still allowing you to perform every action. +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. -Back in Seqera Platform, add a new Azure Batch compute environment: +:::note +VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +::: -1. Add a new compute environment with the name `3-azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. For **Credentials**, select the service principal credentials. -1. For **Config mode**, select Manual. -1. For **Compute pool**, select the Compute pool name `3-azure-batch-vnet`. -1. For **Managed Identity Client ID**, enter the client ID of the managed identity created earlier. - -Duplicate the **original** `hellow-world` pipeline and save it as `hello-world-vnet`. - -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. - -The pipeline runs as before, but it will run on the node pool attached to the VNet. It will resemble a normal Azure Batch pipeline run. - -Using this technique allows you to run pipelines on Azure Batch with more restrictive networking and security requirements. - -### Option 8. Use a node pool attached to a VNet with worker nodes attached to the same VNet +1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: -**Behavior**: - -- We use a separate head node pool to run Nextflow, along with automatically created Nextflow autoscale pools to run processes. -- Each worker node is attached to the VNet and uses the security and networking rules of that virtual network subnetwork. - -**Advantages**: - -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than other services. -- Additionally, you get the advantages of using worker nodes with autopools. - -**Disadvantages**: - -- The set up is very complicated now and errors are likely to occur. -- Errors can be hard to troubleshoot. - -Finally, you can combine some of the previous approaches. Nextflow can create and modify Azure Batch pools based on the pipeline requirements. You can also attach Azure Batch pools to a VNet. Next, attach the worker nodes to the same VNet. - -To achieve this, the following requirements must be met: + ``` + /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} + ``` -- The pipeline must be launched on the node pool attached to the VNet. -- The managed identity must be used to authenticate to Azure Batch and Storage. -- The managed identity must have permissions to create resources attached to the VNet. -- Nextflow creates node pools attached to the VNet. +1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). -Do the following: +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. -1. Duplicate the `hello-world-entra-mi` pipeline, but modfiy the compute environment to `3-azure-batch-vnet` and change the pipeline name to `hello-world-vnet`. -1. Check the virtual network string under the pool details in the Azure Portal, under the **Network Configuration** section. The value should be a Subnet ID, such as `/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}`. Save this value. -1. Change the Nextflow configuration under the **Advanced** tab to include a virtual network with the autopools: +## Advanced. Use a pre-existing Azure Batch pool -```nextflow -process.queue = "auto" -process.machineType = "Standard_E*d_v5" -azure { - batch { - autoPoolMode = true - allowPoolCreation = true - pools { - auto { - autoscale = true - vmCount = 1 - maxVmCount = 4 - virtualNetwork = '/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Network/virtualNetworks/${vnetName}/subnets/${subnetName}' - } - } - } -} -``` +Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. -Select **Launch** next to the `hello-world-vnet` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. +See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. -The pipeline runs as before, but using the managed identity to authenticate to Azure Batch and Storage. It also creates worker pools attached to the VNet. +:::note +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +::: ### Clear up resources @@ -484,3 +209,6 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [nextflow-working-directory]: https://docs.seqera.io/nextflow/cache-and-resume#work-directory [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad +[azure-batch-reference]: ../../compute-envs/azure-batch +[azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity +[azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool From c23beb330c51e9f5fb570e59ae6538b34432e7d3 Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Tue, 28 Jul 2026 22:47:16 +0200 Subject: [PATCH 3/6] Update manual-azure-batch-setup.md Signed-off-by: Justine Geffen --- .../docs/enterprise/advanced-topics/manual-azure-batch-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md index b50318e856..0af48629ec 100644 --- a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -182,6 +182,6 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [nextflow-working-directory]: https://docs.seqera.io/nextflow/cache-and-resume#work-directory [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad -[azure-batch-reference]: ../../compute-envs/azure-batch.md +[azure-batch-reference]: ../../compute-envs/azure-batch [azure-batch-entra]: ../../compute-envs/azure-batch.md#entra-service-principal-and-managed-identity [azure-batch-manual-pool]: ../../compute-envs/azure-batch.md#create-a-nextflow-compatible-azure-batch-pool From 0a0b22adab357a9a2688f8b79e6b63aee833b70a Mon Sep 17 00:00:00 2001 From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Date: Thu, 13 Aug 2026 18:03:56 +0100 Subject: [PATCH 4/6] docs(azure): correct Batch Forge guidance Clarify spot pool support, managed identity resource IDs, head pool autoscaling, and VNet permissions. --- .../manual-azure-batch-setup.md | 10 +++--- .../manual-azure-batch-setup.md | 35 ++++++------------- .../manual-azure-batch-setup.md | 35 ++++++------------- 3 files changed, 26 insertions(+), 54 deletions(-) diff --git a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md index 0af48629ec..3cfeacaf8c 100644 --- a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -74,7 +74,7 @@ Next, create a compute environment with Batch Forge: 1. All other options can be left default. Select **Create** to save the compute environment. :::tip -To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). Set **VMs count** to `1` for the head pool. This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. ::: Add the `nextflow-hello` pipeline to your workspace: @@ -119,14 +119,14 @@ In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID for later. +1. Note the managed identity client ID and resource ID for later. In Seqera: 1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. 1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. @@ -146,6 +146,8 @@ To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** w VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: +Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. + 1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. @@ -161,7 +163,7 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and sele ## Advanced. Use a pre-existing Azure Batch pool -Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. +Some scenarios still require a manually created pool, such as custom VM images, spot nodes, or Azure Batch features that Batch Forge does not expose. Batch Forge does not support spot nodes. In these cases, create the pool yourself and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this tutorial. See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. diff --git a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md index 8e1754e03e..ce027db158 100644 --- a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -6,7 +6,7 @@ description: "A tutorial for using advanced features of Azure Batch with Seqera This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. -Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then the cases that still require manual Azure configuration, such as low-priority (spot) nodes and pre-existing pools. +Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then identifies the cases that still require manually created pools. For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. @@ -28,7 +28,6 @@ In the Azure Portal: 1. For **Quota Type**, select **Batch**, then select **Next**. 1. Select **Enter Details**, then choose the **Location** as the region of your Batch account. 1. Select **EDv5 Series**. -1. Select **Spot/low-priority vCPUS (all Series)**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. @@ -37,7 +36,6 @@ Increase each value to a minimum of the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 -- **Spot/low-priority vCPUS (all Series)**: 192 ### Set up Seqera Cloud @@ -76,7 +74,7 @@ Next, create a compute environment with Batch Forge: 1. All other options can be left default. Select **Create** to save the compute environment. :::tip -To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. ::: Add the `nextflow-hello` pipeline to your workspace: @@ -89,26 +87,11 @@ Add the `nextflow-hello` pipeline to your workspace: Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. -## Part 2. Use low-priority (spot) worker nodes +## Part 2. Use spot nodes -**Advantages**: - -- Low-priority (spot) VMs are cheaper than dedicated VMs. -- You keep the head node on dedicated VMs while running the cheaper work on the worker pool. - -**Disadvantages**: - -- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. - -Batch Forge provisions dedicated nodes only. To use low-priority (spot) nodes for the compute tasks, edit the autoscale formula of the Forge-created worker pool to target low-priority nodes: - -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the worker pool called `tower-pool-{envId}-worker`, where `{envId}` is the compute environment ID (shown at the top of the compute environment page in Seqera). -1. Select **Scale**. -1. In the autoscale formula, find the line that starts with `$TargetDedicatedNodes` and change it to `$TargetLowPriorityNodes`. -1. Select **Evaluate**, then **Save**. - -The worker pool now provisions low-priority VMs, while the head pool continues to use dedicated VMs. Launch the `nextflow-hello` pipeline again to run the compute tasks on low-priority nodes. +:::note +Batch Forge does not support spot nodes. To use spot nodes, create an Azure Batch pool manually and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this walkthrough. +::: ## Part 3. Use Entra authentication @@ -142,14 +125,14 @@ In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID for later. +1. Note the managed identity client ID and resource ID for later. In Seqera: 1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. 1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. @@ -173,6 +156,8 @@ To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** w VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: +Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. + 1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. diff --git a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md index 64b9b7c2f0..aaf658cb19 100644 --- a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -6,7 +6,7 @@ description: "A tutorial for using advanced features of Azure Batch with Seqera This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. -Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then the cases that still require manual Azure configuration, such as low-priority (spot) nodes and pre-existing pools. +Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then identifies the cases that still require manually created pools. For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. @@ -28,7 +28,6 @@ In the Azure Portal: 1. For **Quota Type**, select **Batch**, then select **Next**. 1. Select **Enter Details**, then choose the **Location** as the region of your Batch account. 1. Select **EDv5 Series**. -1. Select **Spot/low-priority vCPUS (all Series)**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. @@ -37,7 +36,6 @@ Increase each value to a minimum of the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 -- **Spot/low-priority vCPUS (all Series)**: 192 ### Set up Seqera Cloud @@ -76,7 +74,7 @@ Next, create a compute environment with Batch Forge: 1. All other options can be left default. Select **Create** to save the compute environment. :::tip -To reduce pipeline latency, disable **Autoscale** on the head pool so a head node stays running (fixed scale). This incurs additional cost for the always-on node but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. ::: Add the `nextflow-hello` pipeline to your workspace: @@ -89,26 +87,11 @@ Add the `nextflow-hello` pipeline to your workspace: Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. -## Part 2. Use low-priority (spot) worker nodes +## Part 2. Use spot nodes -**Advantages**: - -- Low-priority (spot) VMs are cheaper than dedicated VMs. -- You keep the head node on dedicated VMs while running the cheaper work on the worker pool. - -**Disadvantages**: - -- Spot and low-priority nodes can be preempted, which can cause the pipeline to fail. - -Batch Forge provisions dedicated nodes only. To use low-priority (spot) nodes for the compute tasks, edit the autoscale formula of the Forge-created worker pool to target low-priority nodes: - -1. In the Azure Portal, go to the Batch account you created earlier. -1. Go to the **Pools** tab and find the worker pool called `tower-pool-{envId}-worker`, where `{envId}` is the compute environment ID (shown at the top of the compute environment page in Seqera). -1. Select **Scale**. -1. In the autoscale formula, find the line that starts with `$TargetDedicatedNodes` and change it to `$TargetLowPriorityNodes`. -1. Select **Evaluate**, then **Save**. - -The worker pool now provisions low-priority VMs, while the head pool continues to use dedicated VMs. Launch the `nextflow-hello` pipeline again to run the compute tasks on low-priority nodes. +:::note +Batch Forge does not support spot nodes. To use spot nodes, create an Azure Batch pool manually and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this walkthrough. +::: ## Part 3. Use Entra authentication @@ -142,14 +125,14 @@ In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID for later. +1. Note the managed identity client ID and resource ID for later. In Seqera: 1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID of the managed identity created earlier. +1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. 1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. @@ -173,6 +156,8 @@ To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** w VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: +Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. + 1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. For **Location**, select the same region as your Batch account. From adda104d88a8d9032d23564e3a000a184b2d273f Mon Sep 17 00:00:00 2001 From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Date: Fri, 14 Aug 2026 15:36:15 +0100 Subject: [PATCH 5/6] docs(azure): address Batch review comments Clarify required service principal roles and manual pool networking. Move allocation-mode guidance to the manual pool section and align Azure Batch links across tutorial variants. --- .../advanced-topics/manual-azure-batch-setup.md | 14 +++++++++++--- .../advanced-topics/manual-azure-batch-setup.md | 12 ++++++++---- .../advanced-topics/manual-azure-batch-setup.md | 12 ++++++++---- 3 files changed, 27 insertions(+), 11 deletions(-) diff --git a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md index 3cfeacaf8c..529ee26cb1 100644 --- a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -105,7 +105,9 @@ See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). 1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). -1. [Add to Seqera credentials](../../compute-envs/azure-batch.md#entra-service-principal-and-managed-identity) +1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity) + +The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. In Seqera: @@ -165,8 +167,14 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and sele Some scenarios still require a manually created pool, such as custom VM images, spot nodes, or Azure Batch features that Batch Forge does not expose. Batch Forge does not support spot nodes. In these cases, create the pool yourself and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this tutorial. +Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. + See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +:::note +Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. +::: + :::note The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. ::: @@ -185,5 +193,5 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad [azure-batch-reference]: ../../compute-envs/azure-batch -[azure-batch-entra]: ../../compute-envs/azure-batch.md#entra-service-principal-and-managed-identity -[azure-batch-manual-pool]: ../../compute-envs/azure-batch.md#create-a-nextflow-compatible-azure-batch-pool +[azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity +[azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool diff --git a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md index ce027db158..fa61c195b5 100644 --- a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -113,6 +113,8 @@ See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in 1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). 1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity). +The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. + In Seqera: 1. Add new credentials with the name `entra-keys` and select the Azure **Provider**. @@ -139,10 +141,6 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. -:::note -You can also use User Subscription mode instead of Batch Managed here, but this is beyond the scope of this tutorial. -::: - ## Part 4. Attach the pool to a virtual network (VNet) **Advantages**: @@ -175,8 +173,14 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and sele Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. +Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. + See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +:::note +Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. +::: + :::note The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. ::: diff --git a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md index aaf658cb19..76e711d92c 100644 --- a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -113,6 +113,8 @@ See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in 1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). 1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity). +The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. + In Seqera: 1. Add new credentials with the name `entra-keys` and select the Azure **Provider**. @@ -139,10 +141,6 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. -:::note -You can also use User Subscription mode instead of Batch Managed here, but this is beyond the scope of this tutorial. -::: - ## Part 4. Attach the pool to a virtual network (VNet) **Advantages**: @@ -175,8 +173,14 @@ Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and sele Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. +Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. + See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +:::note +Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. +::: + :::note The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. ::: From 9c4f1025fbd4d39db51ed8c6211b66001839f43c Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Thu, 20 Aug 2026 15:09:52 +0200 Subject: [PATCH 6/6] docs: deslop the Azure Batch tutorial Apply the CTRT tutorial conventions and strip AI patterns from the three manual-azure-batch-setup.md pages and the azure-batch.md cross-link tip. - Rename the page to an active verb + noun title, unified across Cloud and Enterprise; rewrite the frontmatter description. - Convert prerequisites to the house `:::info[**Prerequisites**]` admonition with a `You need the following:` lead-in and noun-phrase bullets. - Promote the setup sections to H2 and fix the H2 -> H4 heading jumps. - Rename Part 1 to an active-verb heading and merge the thin low-priority section into the pre-existing-pool part; renumber the parts. - Replace the "Advantages:" bold-lead bullet blocks with lead-in sentences. - Align UI labels with compute-envs/azure-batch.md (Add compute environment, Region, Provider, Entra tab, Client ID / Tenant ID / Client secret). - Use "pipeline"/"run" for Platform concepts instead of "workflow". - Cut self-referential openers, ", so" joins, future tense, and padding. Co-Authored-By: Claude Opus 5 (1M context) --- .../docs/compute-envs/azure-batch.md | 2 +- .../manual-azure-batch-setup.md | 183 +++++++++-------- .../compute-envs/azure-batch.md | 2 +- .../manual-azure-batch-setup.md | 191 +++++++++--------- .../version-26.1/compute-envs/azure-batch.md | 2 +- .../manual-azure-batch-setup.md | 191 +++++++++--------- 6 files changed, 290 insertions(+), 281 deletions(-) diff --git a/platform-cloud/docs/compute-envs/azure-batch.md b/platform-cloud/docs/compute-envs/azure-batch.md index ee53eef81c..b9f299a8d9 100644 --- a/platform-cloud/docs/compute-envs/azure-batch.md +++ b/platform-cloud/docs/compute-envs/azure-batch.md @@ -232,7 +232,7 @@ There are two ways to create an Azure Batch compute environment in Seqera Platfo - [**Manual**](#manual): For using existing Azure Batch resources. :::tip -For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, Entra authentication, and private networking), see the [Azure Batch setup tutorial](../enterprise/advanced-topics/manual-azure-batch-setup). +For an end-to-end walkthrough of Azure Batch topologies (separate head and worker pools, Entra authentication, and private networking), see [Set up advanced Azure Batch compute environments](../enterprise/advanced-topics/manual-azure-batch-setup). ::: ### VM size considerations diff --git a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md index 529ee26cb1..2d845c0699 100644 --- a/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-cloud/docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -1,43 +1,45 @@ --- -title: Azure Batch compute environment setup -headline: "Azure Batch compute environment tutorial" -description: "A tutorial for using Azure Batch with Seqera Platform" +title: Set up advanced Azure Batch compute environments +headline: "Set up advanced Azure Batch compute environments" +description: "Build Azure Batch compute environments with separate head and worker pools, Entra authentication, and private networking." --- -This guide details how to set up an Azure Batch and Seqera Cloud account to run a workflow in a Seqera Azure Batch compute environment. +Batch Forge provisions most Azure Batch topologies for you: separate head and worker pools, dedicated head job resources, autoscaling, Microsoft Entra authentication, and private networking. Each part below adds one capability to the setup before it. Work through them in order. The final part covers the cases that still need a pool you create yourself. -Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This tutorial walks the native Batch Forge path end-to-end, then covers the cases that still require a manually created pool. +For every compute environment field, see the [Azure Batch][azure-batch-reference] reference. -For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. +:::info[**Prerequisites**] -## Prerequisites +You need the following: -- An Azure account with sufficient permissions to create resources. -- [Azure CLI][install-azure-cli] -- [Seqera Platform CLI][install-seqera-cli] +- An Azure account with permissions to create resources. +- The [Azure CLI][install-azure-cli]. +- The [Seqera Platform CLI][install-seqera-cli]. -### Set up Azure Batch +::: + +## Set up Azure Batch In the Azure Portal: 1. Create an Azure Storage account with the default settings. -1. In the Azure Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. -1. Create a new Azure Batch account. Use Batch Managed for now, with the default settings. Use the same region as your Storage account and attach the Storage account to the Batch account when prompted. +1. In the Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. +1. Create an Azure Batch account with the default settings and the Batch Managed pool allocation mode. Use the same region as your Storage account, and attach the Storage account to the Batch account when prompted. 1. On the Azure Batch page, select **Quotas**. 1. Select **Request Quota Increase**. 1. For **Quota Type**, select **Batch**, then select **Next**. -1. Select **Enter Details**, then choose the **Location** as the region of your Batch account. +1. Select **Enter Details**, then set **Location** to the region of your Batch account. 1. Select **EDv5 Series**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. -Increase each value to a minimum of the following: +Increase each value to at least the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 -### Set up Seqera Cloud +## Set up Seqera Cloud In Seqera Cloud: @@ -45,147 +47,151 @@ In Seqera Cloud: - [Create a new organization and workspace][create-org-workspace]. - Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -## Part 1. Azure Batch with Batch Forge +## Part 1. Create a Batch Forge compute environment -Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates two pools, one for the Nextflow head job and one for compute tasks, named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`. The head node does not compete with tasks for resources, and you can size each pool independently. Both pools autoscale based on the number of waiting tasks. -First, add the Azure Batch account credentials to Seqera Platform: +Add the Azure Batch account credentials to Seqera Platform: -1. In the Azure portal, go to the Batch account you created and note the Batch account name and region. -1. Go to the **Keys** tab to find the primary access keys for the Batch account and Storage account. -1. In your Seqera Platform workspace, go to the **Credentials** tab and select **Add credentials**. -1. Enter a credential name such as `azure-keys` and select Azure from the **Provider** drop-down. -1. Enter the Batch account name and key, and Storage account name and key. -1. Select **Create** to save the credentials. +1. In the Azure Portal, open the Batch account you created and note the Batch account name and region. +1. On the **Keys** tab, find the primary access keys for the Batch account and the Storage account. +1. In your Seqera Platform workspace, select the **Credentials** tab, then select **Add credentials**. +1. Enter a credential name such as `azure-keys`, then select **Azure** for **Provider**. +1. Enter the Batch account name and key, and the Storage account name and key. +1. Select **Create**. -Next, create a compute environment with Batch Forge: +Create the compute environment: -1. Go to the **Compute Environments** tab and select **Add Compute Environment**. +1. Select **Compute Environments**, then select **Add compute environment**. 1. Enter a name such as `azure-batch-forge`. -1. Select Azure Batch from the **Provider** drop-down. +1. For **Provider**, select **Azure Batch**. 1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. 1. Set **Config mode** to **Batch Forge**. -1. For **VMs type**, select `standard_e2ds_v5`. +1. For **VMs type**, select `Standard_E2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. -1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. -1. All other options can be left default. Select **Create** to save the compute environment. +1. (Optional) Under **Head job resources**, set **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. +1. Leave the remaining options at their defaults, then select **Create**. :::tip -To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. One head node then stays running while the worker pool autoscales independently. An always-on head node costs more but responds faster. The difference is most noticeable on large production pipelines. ::: -Add the `nextflow-hello` pipeline to your workspace: +[Add a pipeline][add-pipeline] named `nextflow-hello` from your workspace Launchpad with the following settings: -[Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: - -- Select your Azure Batch compute environment from the drop-down. +- Select your Azure Batch compute environment. - For **Pipeline to launch**, enter `https://github.com/nextflow-io/hello`. -- For **Work directory**, enter a subdirectory in the `az://work` container in your Storage account. +- For **Work directory**, enter a subdirectory of the `az://work` container in your Storage account. -Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. +On the Launchpad, select **Launch** next to the pipeline name, complete the launch form, and start the run. -## Part 2. Use Entra authentication +## Part 2. Authenticate with Microsoft Entra -**Advantages**: +Instead of access keys, Seqera Platform can authenticate to Azure Batch and Storage with a Microsoft Entra service principal, and Nextflow can authenticate with a managed identity. You create the service principal and managed identity in Azure, then select them in the compute environment form. Batch Forge configures the pool identity for you. -- No keys or short access tokens are exchanged, increasing security. -- A service principal can have very granular permissions, so you can grant it only the permissions it needs. -- Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. +Entra authentication has the following advantages over access keys: -Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. +- No keys or short-lived access tokens are exchanged. +- A service principal holds only the permissions you grant it. +- A managed identity scoped to a single resource keeps the Nextflow head job's permissions narrow. +- Different compute environments can use different managed identities, each with its own permissions. -See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. +For more detail, see [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation. -### Create a service principal for Seqera to use for authentication +### Create a service principal for Seqera Platform + +In the Azure Portal: 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). -1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). -1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity) +1. [Create a client secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret), then note the client ID, tenant ID, and client secret value. + +The service principal needs the following role assignments: + +- **Azure Batch Data Contributor** on the Batch account. +- **Storage Blob Data Contributor** on the Storage account. +- **Managed Identity Operator** on each managed identity. +- **Network Contributor** on the virtual network (VNet), when you use one. -The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. +For details, see [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. -In Seqera: +In Seqera Platform: -1. Add new credentials with the name `entra-keys` and select the Azure **Provider**. -1. Add the Service Principal ID, Tenant ID and Client Secret. -1. Select **Create** to save the credentials. +1. Select the **Credentials** tab, then select **Add credentials**. +1. Enter the name `entra-keys` and select **Azure** for **Provider**. +1. Select the **Entra** tab, then enter the **Client ID**, **Tenant ID**, and **Client secret**. +1. Select **Create**. -### Create a managed identity for Nextflow to use for authentication +### Create a managed identity for Nextflow In the Azure Portal: -1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) -1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID and resource ID for later. +1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp). +1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). For the Nextflow requirements, see [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments). +1. Note the managed identity client ID and resource ID. -In Seqera: +In Seqera Platform: -1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Select the **Region** of your Batch account. +1. In the managed identity fields, enter the client ID and resource ID for both the head and worker pools. You can use the same managed identity for both pools. +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. - -## Part 3. Attach the pool to a virtual network (VNet) +On the Launchpad, select **Launch** next to `hello-world-entra-mi` and start the run. The pipeline runs as before, but authenticates to Azure Batch and Storage with the managed identity. No keys are exchanged. -**Advantages**: +## Part 3. Attach the pool to a virtual network -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than routing over the public internet. +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when you create the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, and the worker nodes inherit the same networking. -To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. +A private VNet restricts network access to the subnet you specify, and data transfer inside the VNet can be faster and cheaper than routing over the public internet. :::note -VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +The **Subnet ID** field is only available when you select Entra credentials. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: -Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. +Before you create the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. -1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: +1. Select the **Region** of your Batch account. +1. For **Subnet ID**, enter the full Azure Resource Manager (ARM) subnet resource ID: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). -Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the pipeline on the pool attached to the VNet. -## Advanced. Use a pre-existing Azure Batch pool +## Part 4. Use a pre-existing Azure Batch pool -Some scenarios still require a manually created pool, such as custom VM images, spot nodes, or Azure Batch features that Batch Forge does not expose. Batch Forge does not support spot nodes. In these cases, create the pool yourself and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this tutorial. +Some setups still need a pool you create yourself: low-priority (spot) worker nodes, custom VM images, or Azure Batch features that Batch Forge does not expose. Batch Forge provisions dedicated (on-demand) nodes only. In these cases, create the pool in Azure and select it in a **Manual** compute environment. -Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. +Manual pools can use the Batch Managed or User Subscription pool allocation mode. Batch Forge supports Batch Managed only. -See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +Manual mode does not attach a subnet to a pre-existing pool. Configure the VNet and subnet on each pool as you create it. -:::note -Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. -::: +For the full pool configuration (identity, OS image, autoscale formula, start task, and networking), see [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference. Then create a Manual compute environment that points at the pool name. + +To run compute tasks on low-priority nodes, create one dedicated pool and one low-priority pool, then route compute tasks to the low-priority pool with the `process.queue` directive. See [Use separate Batch pools for head and compute nodes][azure-separate-pools]. :::note -The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and unused by Seqera Platform. Use the Batch Forge head and worker pools instead. ::: ## Clear up resources -Once you have completed setup and workflow execution, you can delete the pipelines and compute environments from Seqera. +After the setup and the runs are complete, delete the pipelines and compute environments from Seqera Platform. -In Azure, you can delete the Batch account, which will delete all pools, jobs, and tasks. You can then delete the Storage account. +In Azure, deleting the Batch account deletes all its pools, jobs, and tasks. You can then delete the Storage account. -If you wish to keep the Azure resources, you can remove each pool within a Batch account and mark any active jobs as terminated to free up any quotas on your Azure Batch account. +To keep the Azure resources, remove each pool in the Batch account and mark any active jobs as terminated to free up quota on the Batch account. [install-azure-cli]: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli [install-seqera-cli]: https://docs.seqera.io/platform-cli/installation @@ -195,3 +201,4 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [azure-batch-reference]: ../../compute-envs/azure-batch [azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity [azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool +[azure-separate-pools]: ../../troubleshooting_and_faqs/azure_troubleshooting#use-separate-batch-pools-for-head-and-compute-nodes diff --git a/platform-enterprise_docs/compute-envs/azure-batch.md b/platform-enterprise_docs/compute-envs/azure-batch.md index 36b0b49754..8b88918d48 100644 --- a/platform-enterprise_docs/compute-envs/azure-batch.md +++ b/platform-enterprise_docs/compute-envs/azure-batch.md @@ -236,7 +236,7 @@ There are two ways to create an Azure Batch compute environment in Platform: - [**Manual**](#manual): For using existing Azure Batch resources. :::tip -For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, low-priority nodes, Entra authentication, and private networking), see the [Azure Batch walkthrough](../enterprise/advanced-topics/manual-azure-batch-setup). +For an end-to-end walkthrough of Azure Batch topologies (separate head and worker pools, Entra authentication, and private networking), see [Set up advanced Azure Batch compute environments](../enterprise/advanced-topics/manual-azure-batch-setup). ::: ### VM size considerations diff --git a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md index fa61c195b5..4f3ab37718 100644 --- a/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_docs/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -1,197 +1,197 @@ --- -title: Azure Batch walkthrough -headline: "Azure Batch walkthrough" -description: "A tutorial for using advanced features of Azure Batch with Seqera Platform" +title: Set up advanced Azure Batch compute environments +headline: "Set up advanced Azure Batch compute environments" +description: "Build Azure Batch compute environments with separate head and worker pools, Entra authentication, and private networking." --- -This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. +Batch Forge provisions most Azure Batch topologies for you: separate head and worker pools, dedicated head job resources, autoscaling, Microsoft Entra authentication, and private networking. Each part below adds one capability to the setup before it. Work through them in order. The final part covers the cases that still need a pool you create yourself. -Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then identifies the cases that still require manually created pools. +For every compute environment field, see the [Azure Batch][azure-batch-reference] reference. -For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. +:::info[**Prerequisites**] + +You need the following: + +- An Azure account with permissions to create resources. +- The [Azure CLI][install-azure-cli]. +- The [Seqera Platform CLI][install-seqera-cli]. -:::info Prerequisites -- An Azure account with sufficient permissions to create resources. -- [Azure CLI][install-azure-cli] -- [Seqera Platform CLI][install-seqera-cli] ::: -### Set up Azure Batch +## Set up Azure Batch In the Azure Portal: 1. Create an Azure Storage account with the default settings. -1. In the Azure Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. -1. Create a new Azure Batch account. Use Batch Managed for now, with the default settings. Use the same region as your Storage account and attach the Storage account to the Batch account when prompted. +1. In the Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. +1. Create an Azure Batch account with the default settings and the Batch Managed pool allocation mode. Use the same region as your Storage account, and attach the Storage account to the Batch account when prompted. 1. On the Azure Batch page, select **Quotas**. 1. Select **Request Quota Increase**. 1. For **Quota Type**, select **Batch**, then select **Next**. -1. Select **Enter Details**, then choose the **Location** as the region of your Batch account. +1. Select **Enter Details**, then set **Location** to the region of your Batch account. 1. Select **EDv5 Series**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. -Increase each value to a minimum of the following: +Increase each value to at least the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 -### Set up Seqera Cloud +## Set up Seqera Platform -In Seqera Cloud: +In Seqera Platform: - Create a new account. - [Create a new organization and workspace][create-org-workspace]. - Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -## Part 1. Azure Batch with Batch Forge +## Part 1. Create a Batch Forge compute environment -Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates two pools, one for the Nextflow head job and one for compute tasks, named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`. The head node does not compete with tasks for resources, and you can size each pool independently. Both pools autoscale based on the number of waiting tasks. -First, add the Azure Batch account credentials to Seqera Platform: +Add the Azure Batch account credentials to Seqera Platform: -1. In the Azure portal, go to the Batch account you created and note the Batch account name and region. -1. Go to the **Keys** tab to find the primary access keys for the Batch account and Storage account. -1. In your Seqera Platform workspace, go to the **Credentials** tab and select **Add credentials**. -1. Enter a credential name such as `azure-keys` and select Azure from the **Provider** drop-down. -1. Enter the Batch account name and key, and Storage account name and key. -1. Select **Create** to save the credentials. +1. In the Azure Portal, open the Batch account you created and note the Batch account name and region. +1. On the **Keys** tab, find the primary access keys for the Batch account and the Storage account. +1. In your Seqera Platform workspace, select the **Credentials** tab, then select **Add credentials**. +1. Enter a credential name such as `azure-keys`, then select **Azure** for **Provider**. +1. Enter the Batch account name and key, and the Storage account name and key. +1. Select **Create**. -Next, create a compute environment with Batch Forge: +Create the compute environment: -1. Go to the **Compute Environments** tab and select **Add Compute Environment**. +1. Select **Compute Environments**, then select **Add compute environment**. 1. Enter a name such as `azure-batch-forge`. -1. Select Azure Batch from the **Provider** drop-down. +1. For **Provider**, select **Azure Batch**. 1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. 1. Set **Config mode** to **Batch Forge**. -1. For **VMs type**, select `standard_e2ds_v5`. +1. For **VMs type**, select `Standard_E2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. -1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. -1. All other options can be left default. Select **Create** to save the compute environment. +1. (Optional) Under **Head job resources**, set **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. +1. Leave the remaining options at their defaults, then select **Create**. :::tip -To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. One head node then stays running while the worker pool autoscales independently. An always-on head node costs more but responds faster. The difference is most noticeable on large production pipelines. ::: -Add the `nextflow-hello` pipeline to your workspace: +[Add a pipeline][add-pipeline] named `nextflow-hello` from your workspace Launchpad with the following settings: -[Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: - -- Select your Azure Batch compute environment from the drop-down. +- Select your Azure Batch compute environment. - For **Pipeline to launch**, enter `https://github.com/nextflow-io/hello`. -- For **Work directory**, enter a subdirectory in the `az://work` container in your Storage account. - -Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. +- For **Work directory**, enter a subdirectory of the `az://work` container in your Storage account. -## Part 2. Use spot nodes +On the Launchpad, select **Launch** next to the pipeline name, complete the launch form, and start the run. -:::note -Batch Forge does not support spot nodes. To use spot nodes, create an Azure Batch pool manually and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this walkthrough. -::: +## Part 2. Authenticate with Microsoft Entra -## Part 3. Use Entra authentication +Instead of access keys, Seqera Platform can authenticate to Azure Batch and Storage with a Microsoft Entra service principal, and Nextflow can authenticate with a managed identity. You create the service principal and managed identity in Azure, then select them in the compute environment form. Batch Forge configures the pool identity for you. -**Advantages**: +Entra authentication has the following advantages over access keys: -- No keys or short access tokens are exchanged, increasing security. -- A service principal can have very granular permissions, so you can grant it only the permissions it needs. -- Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. +- No keys or short-lived access tokens are exchanged. +- A service principal holds only the permissions you grant it. +- A managed identity scoped to a single resource keeps the Nextflow head job's permissions narrow. +- Different compute environments can use different managed identities, each with its own permissions. -Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. +For more detail, see [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation. -See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. +### Create a service principal for Seqera Platform -#### Create a service principal for Seqera to use for authentication +In the Azure Portal: 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). -1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). -1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity). +1. [Create a client secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret), then note the client ID, tenant ID, and client secret value. + +The service principal needs the following role assignments: -The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. +- **Azure Batch Data Contributor** on the Batch account. +- **Storage Blob Data Contributor** on the Storage account. +- **Managed Identity Operator** on each managed identity. +- **Network Contributor** on the virtual network (VNet), when you use one. -In Seqera: +For details, see [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. -1. Add new credentials with the name `entra-keys` and select the Azure **Provider**. -1. Add the Service Principal ID, Tenant ID and Client Secret. -1. Select **Create** to save the credentials. +In Seqera Platform: -#### Create a managed identity for Nextflow to use for authentication +1. Select the **Credentials** tab, then select **Add credentials**. +1. Enter the name `entra-keys` and select **Azure** for **Provider**. +1. Select the **Entra** tab, then enter the **Client ID**, **Tenant ID**, and **Client secret**. +1. Select **Create**. + +### Create a managed identity for Nextflow In the Azure Portal: -1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) -1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID and resource ID for later. +1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp). +1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). For the Nextflow requirements, see [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments). +1. Note the managed identity client ID and resource ID. -In Seqera: +In Seqera Platform: -1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Select the **Region** of your Batch account. +1. In the managed identity fields, enter the client ID and resource ID for both the head and worker pools. You can use the same managed identity for both pools. +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. +On the Launchpad, select **Launch** next to `hello-world-entra-mi` and start the run. The pipeline runs as before, but authenticates to Azure Batch and Storage with the managed identity. No keys are exchanged. -## Part 4. Attach the pool to a virtual network (VNet) +## Part 3. Attach the pool to a virtual network -**Advantages**: +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when you create the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, and the worker nodes inherit the same networking. -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than routing over the public internet. - -To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. +A private VNet restricts network access to the subnet you specify, and data transfer inside the VNet can be faster and cheaper than routing over the public internet. :::note -VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +The **Subnet ID** field is only available when you select Entra credentials. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: -Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. +Before you create the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. -1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: +1. Select the **Region** of your Batch account. +1. For **Subnet ID**, enter the full Azure Resource Manager (ARM) subnet resource ID: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). -Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the pipeline on the pool attached to the VNet. -## Advanced. Use a pre-existing Azure Batch pool +## Part 4. Use a pre-existing Azure Batch pool -Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. +Some setups still need a pool you create yourself: low-priority (spot) worker nodes, custom VM images, or Azure Batch features that Batch Forge does not expose. Batch Forge provisions dedicated (on-demand) nodes only. In these cases, create the pool in Azure and select it in a **Manual** compute environment. -Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. +Manual pools can use the Batch Managed or User Subscription pool allocation mode. Batch Forge supports Batch Managed only. -See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +Manual mode does not attach a subnet to a pre-existing pool. Configure the VNet and subnet on each pool as you create it. -:::note -Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. -::: +For the full pool configuration (identity, OS image, autoscale formula, start task, and networking), see [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference. Then create a Manual compute environment that points at the pool name. + +To run compute tasks on low-priority nodes, create one dedicated pool and one low-priority pool, then route compute tasks to the low-priority pool with the `process.queue` directive. See [Use separate Batch pools for head and compute nodes][azure-separate-pools]. :::note -The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and unused by Seqera Platform. Use the Batch Forge head and worker pools instead. ::: -### Clear up resources +## Clear up resources -Once you have completed setup and workflow execution, you can delete the pipelines and compute environments from Seqera. +After the setup and the runs are complete, delete the pipelines and compute environments from Seqera Platform. -In Azure, you can delete the Batch account, which will delete all pools, jobs, and tasks. You can then delete the Storage account. +In Azure, deleting the Batch account deletes all its pools, jobs, and tasks. You can then delete the Storage account. -If you wish to keep the Azure resources, you can remove each pool within a Batch account and mark any active jobs as terminated to free up any quotas on your Azure Batch account. +To keep the Azure resources, remove each pool in the Batch account and mark any active jobs as terminated to free up quota on the Batch account. [install-azure-cli]: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli [install-seqera-cli]: /platform-cli/installation @@ -201,3 +201,4 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [azure-batch-reference]: ../../compute-envs/azure-batch [azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity [azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool +[azure-separate-pools]: ../../troubleshooting_and_faqs/azure_troubleshooting#use-separate-batch-pools-for-head-and-compute-nodes diff --git a/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md b/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md index 62a5fca9e2..8ff9376293 100644 --- a/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md +++ b/platform-enterprise_versioned_docs/version-26.1/compute-envs/azure-batch.md @@ -236,7 +236,7 @@ There are two ways to create an Azure Batch compute environment in Platform: - [**Manual**](#manual): For using existing Azure Batch resources. :::tip -For a worked, end-to-end walkthrough of Azure Batch topologies (separate pools, low-priority nodes, Entra authentication, and private networking), see the [Azure Batch walkthrough](../enterprise/advanced-topics/manual-azure-batch-setup). +For an end-to-end walkthrough of Azure Batch topologies (separate head and worker pools, Entra authentication, and private networking), see [Set up advanced Azure Batch compute environments](../enterprise/advanced-topics/manual-azure-batch-setup). ::: ### VM size considerations diff --git a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md index 76e711d92c..e6d844b2f9 100644 --- a/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md +++ b/platform-enterprise_versioned_docs/version-26.1/enterprise/advanced-topics/manual-azure-batch-setup.md @@ -1,197 +1,197 @@ --- -title: Azure Batch walkthrough -headline: "Azure Batch walkthrough" -description: "A tutorial for using advanced features of Azure Batch with Seqera Platform" +title: Set up advanced Azure Batch compute environments +headline: "Set up advanced Azure Batch compute environments" +description: "Build Azure Batch compute environments with separate head and worker pools, Entra authentication, and private networking." --- -This guide details how to set up more complex Azure Batch compute environments with Seqera Platform. It begins with the simplest possible setup before adding complexity, therefore it is designed to be performed stepwise. +Batch Forge provisions most Azure Batch topologies for you: separate head and worker pools, dedicated head job resources, autoscaling, Microsoft Entra authentication, and private networking. Each part below adds one capability to the setup before it. Work through them in order. The final part covers the cases that still need a pool you create yourself. -Seqera Platform now provisions most advanced Azure Batch topologies natively through Batch Forge, including separate head and worker pools, dedicated head job resources, autoscaling, Entra authentication, and private networking. This walkthrough covers the native Batch Forge path, then identifies the cases that still require manually created pools. +For every compute environment field, see the [Azure Batch][azure-batch-reference] reference. -For a field-by-field reference of every compute environment option, see [Azure Batch][azure-batch-reference]. +:::info[**Prerequisites**] + +You need the following: + +- An Azure account with permissions to create resources. +- The [Azure CLI][install-azure-cli]. +- The [Seqera Platform CLI][install-seqera-cli]. -:::info Prerequisites -- An Azure account with sufficient permissions to create resources. -- [Azure CLI][install-azure-cli] -- [Seqera Platform CLI][install-seqera-cli] ::: -### Set up Azure Batch +## Set up Azure Batch In the Azure Portal: 1. Create an Azure Storage account with the default settings. -1. In the Azure Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. -1. Create a new Azure Batch account. Use Batch Managed for now, with the default settings. Use the same region as your Storage account and attach the Storage account to the Batch account when prompted. +1. In the Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. +1. Create an Azure Batch account with the default settings and the Batch Managed pool allocation mode. Use the same region as your Storage account, and attach the Storage account to the Batch account when prompted. 1. On the Azure Batch page, select **Quotas**. 1. Select **Request Quota Increase**. 1. For **Quota Type**, select **Batch**, then select **Next**. -1. Select **Enter Details**, then choose the **Location** as the region of your Batch account. +1. Select **Enter Details**, then set **Location** to the region of your Batch account. 1. Select **EDv5 Series**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. -Increase each value to a minimum of the following: +Increase each value to at least the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 -### Set up Seqera Cloud +## Set up Seqera Platform -In Seqera Cloud: +In Seqera Platform: - Create a new account. - [Create a new organization and workspace][create-org-workspace]. - Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. -## Part 1. Azure Batch with Batch Forge +## Part 1. Create a Batch Forge compute environment -Batch Forge creates and manages the Azure Batch pools for you. By default it creates **separate pools** for the Nextflow head job and compute tasks (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`), so the head node does not compete with tasks for resources and each pool can be sized independently. Both pools autoscale based on the number of waiting tasks. +Batch Forge creates and manages the Azure Batch pools for you. By default it creates two pools, one for the Nextflow head job and one for compute tasks, named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`. The head node does not compete with tasks for resources, and you can size each pool independently. Both pools autoscale based on the number of waiting tasks. -First, add the Azure Batch account credentials to Seqera Platform: +Add the Azure Batch account credentials to Seqera Platform: -1. In the Azure portal, go to the Batch account you created and note the Batch account name and region. -1. Go to the **Keys** tab to find the primary access keys for the Batch account and Storage account. -1. In your Seqera Platform workspace, go to the **Credentials** tab and select **Add credentials**. -1. Enter a credential name such as `azure-keys` and select Azure from the **Provider** drop-down. -1. Enter the Batch account name and key, and Storage account name and key. -1. Select **Create** to save the credentials. +1. In the Azure Portal, open the Batch account you created and note the Batch account name and region. +1. On the **Keys** tab, find the primary access keys for the Batch account and the Storage account. +1. In your Seqera Platform workspace, select the **Credentials** tab, then select **Add credentials**. +1. Enter a credential name such as `azure-keys`, then select **Azure** for **Provider**. +1. Enter the Batch account name and key, and the Storage account name and key. +1. Select **Create**. -Next, create a compute environment with Batch Forge: +Create the compute environment: -1. Go to the **Compute Environments** tab and select **Add Compute Environment**. +1. Select **Compute Environments**, then select **Add compute environment**. 1. Enter a name such as `azure-batch-forge`. -1. Select Azure Batch from the **Provider** drop-down. +1. For **Provider**, select **Azure Batch**. 1. Select your `azure-keys` credentials. 1. Select the **Region** of your Batch account. 1. Select the `az://work` container in your Storage account. 1. Set **Config mode** to **Batch Forge**. -1. For **VMs type**, select `standard_e2ds_v5`. +1. For **VMs type**, select `Standard_E2ds_v5`. 1. For **VMs count**, select 4. 1. Enable **Autoscale** and **Dispose resources**. -1. (Optional) Under **Head job resources**, set a separate **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. -1. All other options can be left default. Select **Create** to save the compute environment. +1. (Optional) Under **Head job resources**, set **Head VM type**, **Head job CPUs**, and **Head job memory** to size the head node independently of the compute tasks. +1. Leave the remaining options at their defaults, then select **Create**. :::tip -To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. This keeps one head node running while the worker pool continues to autoscale independently. The always-on head node incurs additional cost but improves response time, which is more noticeable on larger production pipelines. +To reduce pipeline latency, expand **Head job resources**, disable **Autoscale**, and set **VMs count** to `1`. One head node then stays running while the worker pool autoscales independently. An always-on head node costs more but responds faster. The difference is most noticeable on large production pipelines. ::: -Add the `nextflow-hello` pipeline to your workspace: +[Add a pipeline][add-pipeline] named `nextflow-hello` from your workspace Launchpad with the following settings: -[Add a pipeline][add-pipeline] from your workspace Launchpad with the following settings: - -- Select your Azure Batch compute environment from the drop-down. +- Select your Azure Batch compute environment. - For **Pipeline to launch**, enter `https://github.com/nextflow-io/hello`. -- For **Work directory**, enter a subdirectory in the `az://work` container in your Storage account. - -Select **Launch** next to the pipeline name in your workspace Launchpad to complete the launch form and launch the workflow. +- For **Work directory**, enter a subdirectory of the `az://work` container in your Storage account. -## Part 2. Use spot nodes +On the Launchpad, select **Launch** next to the pipeline name, complete the launch form, and start the run. -:::note -Batch Forge does not support spot nodes. To use spot nodes, create an Azure Batch pool manually and select it in a **Manual** compute environment. Spot pool configuration is outside the scope of this walkthrough. -::: +## Part 2. Authenticate with Microsoft Entra -## Part 3. Use Entra authentication +Instead of access keys, Seqera Platform can authenticate to Azure Batch and Storage with a Microsoft Entra service principal, and Nextflow can authenticate with a managed identity. You create the service principal and managed identity in Azure, then select them in the compute environment form. Batch Forge configures the pool identity for you. -**Advantages**: +Entra authentication has the following advantages over access keys: -- No keys or short access tokens are exchanged, increasing security. -- A service principal can have very granular permissions, so you can grant it only the permissions it needs. -- Managed identities can be scoped to a specific resource, so the Nextflow head job has very restricted permissions. -- Different managed identities can have different permissions, so different compute environments can have different scoped permissions. +- No keys or short-lived access tokens are exchanged. +- A service principal holds only the permissions you grant it. +- A managed identity scoped to a single resource keeps the Nextflow head job's permissions narrow. +- Different compute environments can use different managed identities, each with its own permissions. -Instead of access keys, Seqera can authenticate to Azure Batch and Storage using an Entra service principal, and Nextflow can authenticate using a managed identity. Batch Forge configures the pool identity for you. You create the service principal and managed identity in Azure, then select them in the compute environment form. +For more detail, see [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation. -See [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation for more information. +### Create a service principal for Seqera Platform -#### Create a service principal for Seqera to use for authentication +In the Azure Portal: 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). -1. [Get the Service Principal ID, Tenant ID, and Client Secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret). -1. [Add to Seqera credentials](../../compute-envs/azure-batch#entra-service-principal-and-managed-identity). +1. [Create a client secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret), then note the client ID, tenant ID, and client secret value. + +The service principal needs the following role assignments: -The service principal requires **Azure Batch Data Contributor** on the Batch account, **Storage Blob Data Contributor** on the Storage account, **Managed Identity Operator** on each managed identity, and, when using a VNet, **Network Contributor** on the VNet. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference for details. +- **Azure Batch Data Contributor** on the Batch account. +- **Storage Blob Data Contributor** on the Storage account. +- **Managed Identity Operator** on each managed identity. +- **Network Contributor** on the virtual network (VNet), when you use one. -In Seqera: +For details, see [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. -1. Add new credentials with the name `entra-keys` and select the Azure **Provider**. -1. Add the Service Principal ID, Tenant ID and Client Secret. -1. Select **Create** to save the credentials. +In Seqera Platform: -#### Create a managed identity for Nextflow to use for authentication +1. Select the **Credentials** tab, then select **Add credentials**. +1. Enter the name `entra-keys` and select **Azure** for **Provider**. +1. Select the **Entra** tab, then enter the **Client ID**, **Tenant ID**, and **Client secret**. +1. Select **Create**. + +### Create a managed identity for Nextflow In the Azure Portal: -1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp) -1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). See [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments) for Nextflow requirements. -1. Note the managed identity client ID and resource ID for later. +1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp). +1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). For the Nextflow requirements, see [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments). +1. Note the managed identity client ID and resource ID. -In Seqera: +In Seqera Platform: -1. Add a new Batch Forge compute environment named `entra-mi` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `entra-mi`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. Under the managed identity fields, enter the client ID and resource ID of the managed identity created earlier for both the head and worker pools. You can use the same managed identity for both pools. -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Select the **Region** of your Batch account. +1. In the managed identity fields, enter the client ID and resource ID for both the head and worker pools. You can use the same managed identity for both pools. +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. -Select **Launch** next to the `hello-world-entra-mi` pipeline in your workspace Launchpad to complete the launch form and launch the workflow. The pipeline runs as before, but uses the managed identity to authenticate to Azure Batch and Storage. No keys are exchanged. +On the Launchpad, select **Launch** next to `hello-world-entra-mi` and start the run. The pipeline runs as before, but authenticates to Azure Batch and Storage with the managed identity. No keys are exchanged. -## Part 4. Attach the pool to a virtual network (VNet) +## Part 3. Attach the pool to a virtual network -**Advantages**: +To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when you create the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, and the worker nodes inherit the same networking. -- Security can be increased by restricting the virtual network subnet. -- Exchange of data can be faster and cheaper than routing over the public internet. - -To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when creating the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, so worker nodes inherit the same networking automatically. +A private VNet restricts network access to the subnet you specify, and data transfer inside the VNet can be faster and cheaper than routing over the public internet. :::note -VNet/subnet configuration requires Entra credentials. The **Subnet ID** field is only available when Entra credentials are selected. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. +The **Subnet ID** field is only available when you select Entra credentials. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: -Before creating the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. +Before you create the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. -1. Add a new Batch Forge compute environment named `azure-batch-vnet` and select the Azure Batch **Provider** type. +1. Add a new Batch Forge compute environment named `azure-batch-vnet`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. -1. For **Location**, select the same region as your Batch account. -1. For **Subnet ID**, enter the full Azure ARM subnet resource ID: +1. Select the **Region** of your Batch account. +1. For **Subnet ID**, enter the full Azure Resource Manager (ARM) subnet resource ID: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` -1. Configure the remaining fields as in [Part 1](#part-1-azure-batch-with-batch-forge). +1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). -Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the workflow on the pool attached to the VNet. +Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the pipeline on the pool attached to the VNet. -## Advanced. Use a pre-existing Azure Batch pool +## Part 4. Use a pre-existing Azure Batch pool -Some scenarios still require a manually created pool, such as custom VM images or Azure Batch features that Batch Forge does not expose. In these cases, create the pool yourself and select it in a **Manual** compute environment. +Some setups still need a pool you create yourself: low-priority (spot) worker nodes, custom VM images, or Azure Batch features that Batch Forge does not expose. Batch Forge provisions dedicated (on-demand) nodes only. In these cases, create the pool in Azure and select it in a **Manual** compute environment. -Unlike Batch Forge, Manual mode does not attach a subnet to pre-existing pools. Configure the VNet/subnet on each pool when you create it. +Manual pools can use the Batch Managed or User Subscription pool allocation mode. Batch Forge supports Batch Managed only. -See [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference for the full pool configuration (identity, OS image, autoscale formula, start task, and networking), then create a Manual compute environment that points at the pool name. +Manual mode does not attach a subnet to a pre-existing pool. Configure the VNet and subnet on each pool as you create it. -:::note -Manual pools can use Batch Managed or User Subscription allocation mode. Batch Forge supports Batch Managed only. -::: +For the full pool configuration (identity, OS image, autoscale formula, start task, and networking), see [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference. Then create a Manual compute environment that points at the pool name. + +To run compute tasks on low-priority nodes, create one dedicated pool and one low-priority pool, then route compute tasks to the low-priority pool with the `process.queue` directive. See [Use separate Batch pools for head and compute nodes][azure-separate-pools]. :::note -The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and not used by Seqera Platform. Use the Batch Forge separate-pool model described above instead. +The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and unused by Seqera Platform. Use the Batch Forge head and worker pools instead. ::: -### Clear up resources +## Clear up resources -Once you have completed setup and workflow execution, you can delete the pipelines and compute environments from Seqera. +After the setup and the runs are complete, delete the pipelines and compute environments from Seqera Platform. -In Azure, you can delete the Batch account, which will delete all pools, jobs, and tasks. You can then delete the Storage account. +In Azure, deleting the Batch account deletes all its pools, jobs, and tasks. You can then delete the Storage account. -If you wish to keep the Azure resources, you can remove each pool within a Batch account and mark any active jobs as terminated to free up any quotas on your Azure Batch account. +To keep the Azure resources, remove each pool in the Batch account and mark any active jobs as terminated to free up quota on the Batch account. [install-azure-cli]: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli [install-seqera-cli]: https://docs.seqera.io/platform-cli/installation @@ -201,3 +201,4 @@ If you wish to keep the Azure resources, you can remove each pool within a Batch [azure-batch-reference]: ../../compute-envs/azure-batch [azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity [azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool +[azure-separate-pools]: ../../troubleshooting_and_faqs/azure_troubleshooting