diff --git a/adr/20260930-conda-layered-builds.md b/adr/20260930-conda-layered-builds.md index 370085056..a113d05dd 100644 --- a/adr/20260930-conda-layered-builds.md +++ b/adr/20260930-conda-layered-builds.md @@ -1,8 +1,8 @@ -# Layered Conda Builds (`conda/micromamba:v3`) +# Layered Conda Builds (`conda/micromamba:v2-fast`) ## Summary -This document describes the `conda/micromamba:v3` build template, which ships the Conda environment of a Docker image as multiple layers grouped by package, instead of one large layer. The template installs the environment exactly like `conda/micromamba:v2`. In the same build step, a small static Go tool (`condasplit`) moves the installed files into 32 fixed layer slots, each holding at most 500 MB before compression. The template is opt-in: existing templates, the default template and existing container ids are unchanged. +This document describes the `conda/micromamba:v2-fast` build template, which ships the Conda environment of a Docker image as multiple layers grouped by package, instead of one large layer. The template installs the environment exactly like `conda/micromamba:v2`. In the same build step, a small static Go tool (`condasplit`) moves the installed files into 32 fixed layer slots, each holding at most 500 MB before compression. The template is opt-in: existing templates, the default template and existing container ids are unchanged. ## Context @@ -29,8 +29,8 @@ For the reference environment (`bioconda::gatk4=4.6.2.0` and `bioconda::gcnvkern ### How it works -1. User submits a container request with `buildTemplate: "conda/micromamba:v3"` -2. Wave renders the v3 Dockerfile, naming the tool image configured with `wave.build.condasplit-image`, both as its BuildKit frontend (`# syntax=`, first line) and as the image mounted into the install step +1. User submits a container request with `buildTemplate: "conda/micromamba:v2-fast"` +2. Wave renders the v2-fast Dockerfile, naming the tool image configured with `wave.build.condasplit-image`, both as its BuildKit frontend (`# syntax=`, first line) and as the image mounted into the install step 3. Two-stage build executes: - **Stage 1 (build)**: Runs the `conda/micromamba:v2` install commands and prints the conda lock. A lock-file URL is first added to the stage with `ADD` and installed from the local copy, because micromamba can't read an explicit lock file from a URL. In the same `RUN`, it runs `condasplit` from the mounted tool image, which moves every file of `/opt/conda` (except `pkgs/`) into at most 32 directories `/layers/00`, `/layers/01`, …, each rooted at `/`, and prints the layer plan - **Stage 2 (prod)**: Starts from `{{base_image}}` and adds each layer directory as its own layer with `COPY --link` @@ -49,13 +49,13 @@ For the reference environment (`bioconda::gatk4=4.6.2.0` and `bioconda::gcnvkern ### 1. New Versioned Template -**Decision:** Added `conda/micromamba:v3` (`BuildTemplate.CONDA_MICROMAMBA_V3`) instead of changing `conda/micromamba:v2`. `BuildTemplate.defaultTemplate()` still returns `conda/micromamba:v2`. +**Decision:** Added `conda/micromamba:v2-fast` (`BuildTemplate.CONDA_MICROMAMBA_V2_FAST`) instead of changing `conda/micromamba:v2`. `BuildTemplate.defaultTemplate()` still returns `conda/micromamba:v2`. | Value | Conda layers | Default | |-------|--------------|---------| | `conda/micromamba:v1` | 1 (single-stage) | No | | `conda/micromamba:v2` | 1 | Yes (`CONDA` packages) | -| `conda/micromamba:v3` | Up to 32, one per used slot | No | +| `conda/micromamba:v2-fast` | Up to 32, one per used slot | No | | `conda/pixi:v1` | 1 | No | **Rationale:** @@ -155,7 +155,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \ ### 7. Package Cache Dropped -**Decision:** `conda/micromamba:v3` excludes `/opt/conda/pkgs` from the image. The other templates keep it. +**Decision:** `conda/micromamba:v2-fast` excludes `/opt/conda/pkgs` from the image. The other templates keep it. **Rationale:** - Extracted packages are not needed at runtime @@ -164,7 +164,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \ ### 8. Docker Only -**Decision:** A `conda/micromamba:v3` request with `format: "sif"` fails with HTTP 400 `Build template 'conda/micromamba:v3' does not support Singularity format`. There are no Singularity template files. +**Decision:** A `conda/micromamba:v2-fast` request with `format: "sif"` fails with HTTP 400 `Build template 'conda/micromamba:v2-fast' does not support Singularity format`. There are no Singularity template files. **Rationale:** - A SIF image is a single squashfs file, so layering doesn't apply @@ -180,7 +180,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \ ### 10. BuildKit Frontend Repeating the Layer `COPY` Line -**Decision:** The v3 Dockerfile starts with `# syntax={{layers_image}}`, which makes the tool image its BuildKit frontend. The frontend (`condasplit/frontend/`, entrypoint `/condasplit-frontend`) builds the `build` stage, lists `/layers`, repeats the `COPY --link --from=build /layers/NN/ /` line for each directory and builds the whole Dockerfile. Both builds are delegated to the built-in Dockerfile frontend. +**Decision:** The v2-fast Dockerfile starts with `# syntax={{layers_image}}`, which makes the tool image its BuildKit frontend. The frontend (`condasplit/frontend/`, entrypoint `/condasplit-frontend`) builds the `build` stage, lists `/layers`, repeats the `COPY --link --from=build /layers/NN/ /` line for each directory and builds the whole Dockerfile. Both builds are delegated to the built-in Dockerfile frontend. **Rationale:** - The image has exactly one layer per layer directory. BuildKit adds a layer for every `COPY`, even an empty one, so 32 fixed `COPY` lines gave a small environment such as `bioconda::samtools` 30 empty layers @@ -206,7 +206,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \ | Value | Constant | Description | |-------|----------|-------------| -| `conda/micromamba:v3` | `BuildTemplate.CONDA_MICROMAMBA_V3` | Layered multi-stage Micromamba build, Docker only | +| `conda/micromamba:v2-fast` | `BuildTemplate.CONDA_MICROMAMBA_V2_FAST` | Layered multi-stage Micromamba build, Docker only | No new request fields. The template accepts the same inputs as `conda/micromamba:v2`: conda file, conda lock file URL, custom `baseImage`/`containerImage`, `mambaImage`, `basePackages` and `commands`. @@ -214,20 +214,20 @@ No new request fields. The template accepts the same inputs as `conda/micromamba | Condition | Response | |-----------|----------| -| `format: "sif"` | HTTP 400 `Build template 'conda/micromamba:v3' does not support Singularity format` | -| Non-`CONDA` package type | HTTP 400 `Package type '' not supported by 'conda/micromamba:v3' build template` | +| `format: "sif"` | HTTP 400 `Build template 'conda/micromamba:v2-fast' does not support Singularity format` | +| Non-`CONDA` package type | HTTP 400 `Package type '' not supported by 'conda/micromamba:v2-fast' build template` | ## Configuration Changes | Property | Default | Description | |----------|---------|-------------| -| `wave.build.condasplit-image` | `public.cr.seqera.io/wave/condasplit:v1` | Tool image mounted into `conda/micromamba:v3` builds | +| `wave.build.condasplit-image` | `public.cr.seqera.io/wave/condasplit:v1` | Tool image mounted into `conda/micromamba:v2-fast` builds | -Each tool release gets a new immutable tag (`v1`, `v2`, …), taken from `condasplit/VERSION`, and the config value may also pin a digest. The tag stays readable because it appears in the v3 container file; a SHA-256 of the tool sources goes into the `io.seqera.condasplit.source` image label instead, and the release fails when an existing tag has different sources. The image reference is part of the v3 container file, so a tool upgrade changes container ids for v3 images only. Enterprise installs without access to `public.cr.seqera.io` mirror the image and set this property. +Each tool release gets a new immutable tag (`v1`, `v2`, …), taken from `condasplit/VERSION`, and the config value may also pin a digest. The tag stays readable because it appears in the v2-fast container file; a SHA-256 of the tool sources goes into the `io.seqera.condasplit.source` image label instead, and the release fails when an existing tag has different sources. The image reference is part of the v2-fast container file, so a tool upgrade changes container ids for v2-fast images only. Enterprise installs without access to `public.cr.seqera.io` mirror the image and set this property. ## Template Comparison -| Aspect | `conda/micromamba:v2` | `conda/micromamba:v3` | +| Aspect | `conda/micromamba:v2` | `conda/micromamba:v2-fast` | |--------|-----------------------|-----------------------| | Build stages | 2 | 2 | | Conda environment layers | 1 | One per used slot, up to 32, at most 500 MB each before compression | @@ -254,7 +254,7 @@ Reference environment built with BuildKit v0.25.2, gzip compression and OCI medi |----------|-------| | API Models | `BuildTemplate.java` | | Configuration | `BuildConfig.groovy`, `application.yml` | -| Templates | `conda-micromamba-v3/dockerfile-conda-file.txt` | +| Templates | `conda-micromamba-v2-fast/dockerfile-conda-file.txt` | | Helpers | `TemplateUtils.java`, `CondaHelper.groovy`, `ContainerHelper.groovy` | | Controller | `ContainerController.groovy` | | Services | `ContainerInspectServiceImpl.groovy` | @@ -285,14 +285,14 @@ Reference environment built with BuildKit v0.25.2, gzip compression and OCI medi ### Neutral - `conda/micromamba:v2` remains the default template - Tuning the grouping parameters requires a new template version -- A tool upgrade changes container ids of `conda/micromamba:v3` images only +- A tool upgrade changes container ids of `conda/micromamba:v2-fast` images only - The same tool could later serve `conda/pixi` environments, which also have `conda-meta/` ## Follow-ups - **Production default**: At the production release, switch the default of `wave.build.condasplit-image` to `public.cr.seqera.io/wave/condasplit:v1` in `BuildConfig.groovy`, `application.yml` and `docs/install/reference.md` -- **Nextflow**: Add `conda/micromamba:v3` to the `wave.build.template` description in `plugins/nf-wave/src/main/io/seqera/wave/plugin/config/WaveConfig.groovy`, and correct its stated default to `conda/micromamba:v2` -- **Wave CLI**: Add `conda/micromamba:v3` to the `--build-template` help text in `app/src/main/java/io/seqera/wave/cli/App.java` +- **Nextflow**: Add `conda/micromamba:v2-fast` to the `wave.build.template` description in `plugins/nf-wave/src/main/io/seqera/wave/plugin/config/WaveConfig.groovy`, and correct its stated default to `conda/micromamba:v2` +- **Wave CLI**: Add `conda/micromamba:v2-fast` to the `--build-template` help text in `app/src/main/java/io/seqera/wave/cli/App.java` ## References diff --git a/condasplit/Dockerfile b/condasplit/Dockerfile index f402d9158..28905cadf 100644 --- a/condasplit/Dockerfile +++ b/condasplit/Dockerfile @@ -9,5 +9,5 @@ RUN go test ./... \ FROM scratch COPY --from=build /condasplit /condasplit-frontend / -# the image is also the BuildKit frontend of the conda/micromamba:v3 template, see frontend/ +# the image is also the BuildKit frontend of the conda/micromamba:v2-fast template, see frontend/ ENTRYPOINT ["/condasplit-frontend"] diff --git a/condasplit/README.md b/condasplit/README.md index 2781341d9..48ed94b55 100644 --- a/condasplit/README.md +++ b/condasplit/README.md @@ -1,6 +1,6 @@ # condasplit -A small static tool used by the `conda/micromamba:v3` build template to ship a conda +A small static tool used by the `conda/micromamba:v2-fast` build template to ship a conda environment as several image layers instead of one large layer. It runs in the same `RUN` step that installs the environment. It moves every file of the @@ -84,7 +84,7 @@ condasplit --src DIR --out DIR --slots N | `--own-layer-size` | 50000000 | Packages at least this size get a layer to themselves | | `--exclude` | none | Path relative to `--src` left in place and not layered (repeatable) | -The `conda/micromamba:v3` template runs: +The `conda/micromamba:v2-fast` template runs: ``` /opt/wave-tools/condasplit --src /opt/conda --out /layers \ @@ -126,7 +126,7 @@ and cross-compiles a static binary (`CGO_ENABLED=0`) from the pinned `golang` im Each release gets a new immutable tag (`v1`, `v2`, ...), set in the `VERSION` file. Wave pins the exact tag with the `wave.build.condasplit-image` setting, so a new tool version -changes the v3 container file and therefore the container id. Never overwrite an existing tag. +changes the v2-fast container file and therefore the container id. Never overwrite an existing tag. **Stage** (default registry of the `Makefile`, only for tests and local development): @@ -140,7 +140,7 @@ make all `public.cr.seqera.io/wave/condasplit:` on the next Wave release commit (`[release]` on `master`). Then point the default of `wave.build.condasplit-image` to the new tag. -The tag is always the `VERSION` value, never a checksum: Wave writes it into the v3 container +The tag is always the `VERSION` value, never a checksum: Wave writes it into the v2-fast container file and the build logs, where `condasplit:v1` is readable and a checksum isn't. The checksum goes into an `io.seqera.condasplit.source` image label instead: a SHA-256 of the tracked files of this directory, the Markdown docs excluded. `make release` uses it to make sure a tag always holds diff --git a/condasplit/frontend/main.go b/condasplit/frontend/main.go index ec5bb9942..627e89d81 100644 --- a/condasplit/frontend/main.go +++ b/condasplit/frontend/main.go @@ -16,7 +16,7 @@ * along with this program. If not, see . */ -// condasplit-frontend is the BuildKit frontend of the conda/micromamba:v3 build template. +// condasplit-frontend is the BuildKit frontend of the conda/micromamba:v2-fast build template. // Wave writes the Dockerfile before the environment is known, so the template has one // COPY line for the layer directories: this frontend builds the install stage first, // then builds the image with that line repeated for each directory condasplit created. @@ -41,7 +41,7 @@ import ( "github.com/moby/buildkit/util/appcontext" ) -// the COPY line of the layer directories in the conda/micromamba:v3 template +// the COPY line of the layer directories in the conda/micromamba:v2-fast template var layersLine = regexp.MustCompile(`(?m)^COPY --link --from=build /layers/NN/ /\n`) func main() { diff --git a/docs/api.md b/docs/api.md index 98aec4887..d07f721b9 100644 --- a/docs/api.md +++ b/docs/api.md @@ -223,7 +223,7 @@ Returns the name of the container request made available by Wave. | `pixiImage` | Name of the Docker image used for Pixi package manager (e.g., `ghcr.io/prefix-dev/pixi:latest`). | | `cranOpts` | CRAN build options (when `type` is `CRAN`). | | `rImage` | Name of the R Docker image used to build CRAN containers (e.g., `rocker/r-ver:4.4.1`). | -| `buildTemplate` | The build template to use for container builds. Supported values: `conda/micromamba:v1` (single-stage build using Micromamba 1.x), `conda/pixi:v1` (multi-stage build using Pixi), `conda/micromamba:v2` (multi-stage build using Micromamba 2.x), `conda/micromamba:v3` (multi-stage build using Micromamba 2.x that splits the Conda environment into multiple image layers, Docker only, see [Layered Conda builds](#layered-conda-builds)), `cran/installr:v1` (CRAN packages). The multi-stage templates produce smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build with `conda/pixi:v1` and `conda/micromamba:v2`). Default: `conda/micromamba:v2` for `CONDA` packages, `cran/installr:v1` for `CRAN` packages. | +| `buildTemplate` | The build template to use for container builds. Supported values: `conda/micromamba:v1` (single-stage build using Micromamba 1.x), `conda/pixi:v1` (multi-stage build using Pixi), `conda/micromamba:v2` (multi-stage build using Micromamba 2.x), `conda/micromamba:v2-fast` (multi-stage build using Micromamba 2.x that splits the Conda environment into multiple image layers, Docker only, see [Layered Conda builds](#layered-conda-builds)), `cran/installr:v1` (CRAN packages). The multi-stage templates produce smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build with `conda/pixi:v1` and `conda/micromamba:v2`). Default: `conda/micromamba:v2` for `CONDA` packages, `cran/installr:v1` for `CRAN` packages. | | `nameStrategy` | The name strategy to be used to create the name of the container built by Wave. Options: `none`, `tagPrefix`, `imageSuffix`. | | #### Response @@ -493,7 +493,7 @@ Get status of build against `buildId` passed as path variable } ``` -- Create Docker image with Micromamba v3 template (layered multi-stage build): +- Create Docker image with `conda/micromamba:v2-fast` template (layered multi-stage build): **Request** @@ -503,7 +503,7 @@ Get status of build against `buildId` passed as path variable --data '{ "containerPlatform": "linux/amd64", "format": "docker", - "buildTemplate": "conda/micromamba:v3", + "buildTemplate": "conda/micromamba:v2-fast", "packages":{ "type": "CONDA", "entries": ["gatk4=4.6.2.0", "gcnvkernel=0.9"], @@ -620,7 +620,7 @@ Get status of build against `buildId` passed as path variable ``` :::note -Multi-stage build templates (`conda/pixi:v1`, `conda/micromamba:v2`, and `conda/micromamba:v3`) create optimized Docker container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools. +Multi-stage build templates (`conda/pixi:v1`, `conda/micromamba:v2`, and `conda/micromamba:v2-fast`) create optimized Docker container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools. ::: :::important @@ -633,7 +633,7 @@ As a result, the `baseImage` option has no effect on Singularity builds. It only #### Layered Conda builds -The `conda/micromamba:v3` template installs the Conda environment with the same steps as `conda/micromamba:v2`, then splits it into multiple image layers instead of shipping it as one large layer. Use it when a registry, proxy, or CDN cache rejects or doesn't cache layers larger than 512 MB, or to let container runtimes download and unpack the environment in parallel. +The `conda/micromamba:v2-fast` template installs the Conda environment with the same steps as `conda/micromamba:v2`, then splits it into multiple image layers instead of shipping it as one large layer. Use it when a registry, proxy, or CDN cache rejects or doesn't cache layers larger than 512 MB, or to let container runtimes download and unpack the environment in parallel. The environment is split by Conda package: @@ -649,7 +649,7 @@ The build log shows the layer plan between the `>> CONDA_LAYERS_START` and `<< C The container content is the same as `conda/micromamba:v2` produces for the same environment, except that the unused package cache (`/opt/conda/pkgs`) is dropped, which also makes the image smaller. The template supports the same inputs as `conda/micromamba:v2`: Conda packages, Conda environment files, Conda lock file URLs, and the `condaOpts` options. Custom `commands` run after the Conda layers and add their own layers. :::important -`conda/micromamba:v3` builds Docker images only. Singularity requests (`format: "sif"`) are rejected with HTTP 400 and the message `Build template 'conda/micromamba:v3' does not support Singularity format`. +`conda/micromamba:v2-fast` builds Docker images only. Singularity requests (`format: "sif"`) are rejected with HTTP 400 and the message `Build template 'conda/micromamba:v2-fast' does not support Singularity format`. ::: ### GET `/v1alpha1/builds/{buildId}/status` diff --git a/docs/cli/use-cases.md b/docs/cli/use-cases.md index 08cc6c4c7..629467192 100644 --- a/docs/cli/use-cases.md +++ b/docs/cli/use-cases.md @@ -94,7 +94,7 @@ The Wave CLI supports build templates for creating container images from Conda p |-----------------------|------------------------------------------------------------------------------------------------------------------| | `conda/micromamba:v1` | Single-stage build using Micromamba v1. The final image includes the package manager. | | `conda/micromamba:v2` | Multi-stage build using Micromamba 2.x (default). Produces smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build). | -| `conda/micromamba:v3` | Multi-stage build using Micromamba 2.x that splits the Conda environment into multiple layers of at most 500 MB each before compression, with rare exceptions. Docker only. | +| `conda/micromamba:v2-fast` | Multi-stage build using Micromamba 2.x that splits the Conda environment into multiple layers of at most 500 MB each before compression, with rare exceptions. Docker only. | | `conda/pixi:v1` | Multi-stage build using [Pixi][pixi] package manager. Produces smaller images with faster dependency resolution (Singularity still uses a single-stage build). | **Related CLI arguments** @@ -119,9 +119,9 @@ As a result, the `baseImage` option has no effect on Singularity builds. It only **Layered Conda builds** -The `conda/micromamba:v3` template installs the environment like `conda/micromamba:v2`, then splits it into multiple image layers grouped by Conda package. Large packages get their own layer, small packages are grouped together, and every layer holds at most 500 MB of files before compression, so compressed layers stay under 512 MB. The only exceptions are a single file larger than 500 MB, which gets a layer to itself, and layers merged when the environment needs more than 32 layers. The build log shows a warning in both cases. Use it when your registry or proxy rejects large layers, or to speed up image pulls. The build log shows which packages went into each layer, unless the build reuses the environment install step from the build cache. +The `conda/micromamba:v2-fast` template installs the environment like `conda/micromamba:v2`, then splits it into multiple image layers grouped by Conda package. Large packages get their own layer, small packages are grouped together, and every layer holds at most 500 MB of files before compression, so compressed layers stay under 512 MB. The only exceptions are a single file larger than 500 MB, which gets a layer to itself, and layers merged when the environment needs more than 32 layers. The build log shows a warning in both cases. Use it when your registry or proxy rejects large layers, or to speed up image pulls. The build log shows which packages went into each layer, unless the build reuses the environment install step from the build cache. -The `conda/micromamba:v3` template builds Docker images only. Wave rejects requests that combine it with `--singularity`. See [Layered Conda builds](../features/container-builds.mdx#layered-conda-builds) for more details. +The `conda/micromamba:v2-fast` template builds Docker images only. Wave rejects requests that combine it with `--singularity`. See [Layered Conda builds](../features/container-builds.mdx#layered-conda-builds) for more details. **Example usage** @@ -142,12 +142,12 @@ wave \ --build-template conda/micromamba:v2 ``` -Build a container with the Conda environment split into multiple layers using the Micromamba v3 template: +Build a container with the Conda environment split into multiple layers using the `conda/micromamba:v2-fast` template: ```bash wave \ --conda-package bioconda::gatk4=4.6.2.0 \ - --build-template conda/micromamba:v3 + --build-template conda/micromamba:v2-fast ``` Build a Singularity container using Pixi: diff --git a/docs/features/container-builds.mdx b/docs/features/container-builds.mdx index 4508363d5..ca7f1f4e7 100644 --- a/docs/features/container-builds.mdx +++ b/docs/features/container-builds.mdx @@ -159,7 +159,7 @@ By default, Wave builds Conda images with the `conda/micromamba:v2` template, wh Three multi-stage templates are available: - **`conda/micromamba:v2`** uses Micromamba and produces an `environment.lock` file. This is the default template for Conda builds. -- **`conda/micromamba:v3`** builds like `conda/micromamba:v2`, then splits the environment into multiple image layers. It produces Docker images only. See [Layered Conda builds](#layered-conda-builds). +- **`conda/micromamba:v2-fast`** builds like `conda/micromamba:v2`, then splits the environment into multiple image layers. It produces Docker images only. See [Layered Conda builds](#layered-conda-builds). - **`conda/pixi:v1`** uses [Pixi](https://pixi.sh/) and produces a `pixi.lock` file. Conda lock files are not supported through this template. Pixi support was added in Wave 1.31.0. Select a template with the `buildTemplate` field on the build request, the `--build-template` option of the Wave CLI, or the `wave.build.template` setting in Nextflow. The `conda/micromamba:v2` and `conda/pixi:v1` templates accept Conda packages and Conda environment files, and both produce Docker and Singularity images. @@ -168,7 +168,7 @@ Select a template with the `buildTemplate` field on the build request, the `--bu The `conda/micromamba:v1`, `conda/micromamba:v2`, and `conda/pixi:v1` templates ship the whole Conda environment as a single image layer, which can reach several gigabytes. Some registries, proxies, and CDN caches reject or don't cache layers larger than 512 MB. Container runtimes also download and unpack a single layer over one connection, even when they could fetch several layers in parallel. -The `conda/micromamba:v3` template installs the environment as `conda/micromamba:v2` does, then splits it into layers grouped by Conda package: +The `conda/micromamba:v2-fast` template installs the environment as `conda/micromamba:v2` does, then splits it into layers grouped by Conda package: - Each package of 50 MB or more gets its own layer, and a package larger than 500 MB is split across several layers. - Smaller packages are grouped together into shared layers. @@ -182,10 +182,10 @@ To build with this template using the Wave CLI: ```bash wave \ --conda-package bioconda::gatk4=4.6.2.0 \ - --build-template conda/micromamba:v3 + --build-template conda/micromamba:v2-fast ``` -`conda/micromamba:v3` is opt-in. Existing templates and the default template are unchanged, so existing images keep their build IDs. Singularity builds are not supported with this template, and Wave rejects them. See [Layered Conda builds](../api.md#layered-conda-builds) in the API reference for details. +`conda/micromamba:v2-fast` is opt-in. Existing templates and the default template are unchanged, so existing images keep their build IDs. Singularity builds are not supported with this template, and Wave rejects them. See [Layered Conda builds](../api.md#layered-conda-builds) in the API reference for details. ### Build from CRAN packages diff --git a/docs/install/reference.md b/docs/install/reference.md index 947ea5cb7..395600833 100644 --- a/docs/install/reference.md +++ b/docs/install/reference.md @@ -304,7 +304,7 @@ Configure how Wave builds container images and manages build logs. Options include: `uncompressed`, `estargz`, and `zstd`. `wave.build.condasplit-image` *(optional)* -: Container image that supplies the `condasplit` tool used by `conda/micromamba:v3` builds to split the Conda environment into multiple image layers (default: `public.cr.seqera.io/wave/condasplit:v1`). +: Container image that supplies the `condasplit` tool used by `conda/micromamba:v2-fast` builds to split the Conda environment into multiple image layers (default: `public.cr.seqera.io/wave/condasplit:v1`). The same image is also the BuildKit frontend of these builds, named by the `# syntax=` directive of the generated Dockerfile. It adds one image layer for each layer directory created by `condasplit`. The image is used by the build only and is not included in the built image. If your installation can't access the default image, mirror it to a registry your build service can pull from and set this option to the mirrored image. diff --git a/specs/002-conda-micromamba-v3-layers/spec.md b/specs/002-conda-micromamba-v3-layers/spec.md index 39451b42a..819d35d2e 100644 --- a/specs/002-conda-micromamba-v3-layers/spec.md +++ b/specs/002-conda-micromamba-v3-layers/spec.md @@ -1,4 +1,4 @@ -# Layered Conda Builds (`conda/micromamba:v3`) +# Layered Conda Builds (`conda/micromamba:v2-fast`) **Branch**: `002-conda-micromamba-v3-layers` · **PR**: [#1146](https://github.com/seqeralabs/wave/pull/1146) · **Created**: 2026-09-30 **Status**: Implemented, pending staging validation and the production image release @@ -7,7 +7,7 @@ ## Summary -`conda/micromamba:v3` is a new, opt-in build template. It installs the Conda environment exactly like `conda/micromamba:v2`, then ships it as several image layers grouped by package, each at most 500 MB before compression, instead of one multi-GB layer. Two small Go programs make this work. Both ship in one tool image, `wave/condasplit`: +`conda/micromamba:v2-fast` is a new, opt-in build template. It installs the Conda environment exactly like `conda/micromamba:v2`, then ships it as several image layers grouped by package, each at most 500 MB before compression, instead of one multi-GB layer. Two small Go programs make this work. Both ship in one tool image, `wave/condasplit`: - **`condasplit`** runs inside the install step and moves the installed files into one directory per layer. - **`condasplit-frontend`** is a BuildKit frontend. It turns the template's single `COPY` line into one `COPY`, and so one image layer, per directory. @@ -30,18 +30,18 @@ Every Conda template ships the environment as a single layer. For the reference - No new services, API fields or persistence. **Non-goals.** -- Singularity: a SIF image is one squashfs file, so v3 rejects it with HTTP 400. +- Singularity: a SIF image is one squashfs file, so v2-fast rejects it with HTTP 400. - Changing v1, v2 or pixi. -- Making v3 the default. +- Making v2-fast the default. - Sharing package layers across images. - Showing the layer plan in the UI. -## How a v3 build works +## How a v2-fast build works | Component | Role | |---|---| -| Wave | Renders the Dockerfile from the v3 template and starts the build with the usual `buildctl` arguments | -| v3 Dockerfile | Two stages. The first line `# syntax=` selects the frontend. The `build` stage installs the environment and runs `condasplit`. The `prod` stage has one `COPY --link --from=build /layers/NN/ /` line | +| Wave | Renders the Dockerfile from the v2-fast template and starts the build with the usual `buildctl` arguments | +| v2-fast Dockerfile | Two stages. The first line `# syntax=` selects the frontend. The `build` stage installs the environment and runs `condasplit`. The `prod` stage has one `COPY --link --from=build /layers/NN/ /` line | | `condasplit-frontend` | The image's entrypoint. BuildKit runs it for the `# syntax=` directive. It builds the `build` stage, lists `/layers`, repeats the `COPY` line for each directory and builds the image. It delegates both builds to the built-in Dockerfile frontend | | `condasplit` | Mounted read-only into the install `RUN`. It moves `/opt/conda` (except `pkgs/`) into `/layers/00`, `/layers/01`, …, each rooted at `/`, and prints the layer plan | | BuildKit | Unchanged, rootless `v0.25.2` in a Kubernetes pod. It pulls the tool image once as the frontend and once for the mount | @@ -57,8 +57,8 @@ sequenceDiagram participant S as build stage (mamba image) participant R as Registry - C->>W: container request, buildTemplate conda/micromamba:v3 - W->>W: render the v3 Dockerfile (conda.yml or lock file URL) + C->>W: container request, buildTemplate conda/micromamba:v2-fast + W->>W: render the v2-fast Dockerfile (conda.yml or lock file URL) W-->>C: target image and build id, the build runs asynchronously W->>B: buildctl build --frontend dockerfile.v0 (usual arguments) B->>D: load the Dockerfile @@ -75,7 +75,7 @@ sequenceDiagram B->>R: push ``` -**The template** (`src/main/resources/templates/conda-micromamba-v3/dockerfile-conda-file.txt`) is v2's install step plus the frontend directive, the tool call and a new final stage: +**The template** (`src/main/resources/templates/conda-micromamba-v2-fast/dockerfile-conda-file.txt`) is v2's install step plus the frontend directive, the tool call and a new final stage: ```dockerfile # syntax={{layers_image}} @@ -106,7 +106,7 @@ ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH" ## Key decisions -**D1. A new template version.** The container id hashes the generated Dockerfile, so changing v2 would change every existing id and trigger rebuilds. v3 is opt-in, and `defaultTemplate()` still returns v2. The tool parameters are literal in the template, so tuning them means a new template version. +**D1. A new template.** The container id hashes the generated Dockerfile, so changing v2 would change every existing id and trigger rebuilds. v2-fast is opt-in, and `defaultTemplate()` still returns v2. The tool parameters are literal in the template, so tuning them means a new template version. **D2. Split inside the install step, by moving files.** `condasplit` runs in the same `RUN` as `micromamba install`, so every installed file is in the same writable layer: - `rename()` only rewrites directory entries, which takes seconds (2.8 s for 88,231 files). @@ -141,7 +141,7 @@ A final check fails the build if anything but directories and `pkgs/` is left be The frontend is its own Go module, so its `github.com/moby/buildkit` dependency (v0.25.2, the production version) never reaches the tool. Tags are immutable (`v1`, `v2`, …), so a new tool version means a new container id. -**D8. Lock file URLs are added, not passed to micromamba.** `micromamba install -f ` downloads only YAML environment files and treats an explicit lock URL as a local path. Wave's `/v1alpha1/builds//condalock` endpoint serves explicit locks, so v3 uses `ADD ` and installs from the local copy. The file name is kept, since micromamba reads the format from the extension, and falls back to `conda.lock`. v1 and v2 have the same bug but stay unchanged to keep their container ids (follow-up). +**D8. Lock file URLs are added, not passed to micromamba.** `micromamba install -f ` downloads only YAML environment files and treats an explicit lock URL as a local path. Wave's `/v1alpha1/builds//condalock` endpoint serves explicit locks, so v2-fast uses `ADD ` and installs from the local copy. The file name is kept, since micromamba reads the format from the extension, and falls back to `conda.lock`. v1 and v2 have the same bug but stay unchanged to keep their container ids (follow-up). **D9. The layer plan goes in the build log.** `condasplit` prints it between `>> CONDA_LAYERS_START` and `<< CONDA_LAYERS_END`, after the unchanged conda lock markers, so lock extraction works as for v2: @@ -170,7 +170,7 @@ condasplit --src DIR --out DIR --slots N [--max-layer-size BYTES] [--own-layer-s | `--slots` | Maximum number of layers; above it the smallest are merged | | `--max-layer-size` | Cap per layer, the sum of file sizes before compression (default 500,000,000) | | `--own-layer-size` | Packages at least this size get their own layers (default 50,000,000) | -| `--exclude` | Paths relative to `--src` left in place (v3 passes `pkgs`) | +| `--exclude` | Paths relative to `--src` left in place (v2-fast passes `pkgs`) | **Steps:** 1. Walk the prefix without following symlinks, skipping `--exclude` paths. @@ -186,16 +186,16 @@ The prefix directory and its parents are `0755 root:root` in every layer, as v2' | Area | Change | |---|---| -| `wave-api` `BuildTemplate` | `CONDA_MICROMAMBA_V3 = "conda/micromamba:v3"` | +| `wave-api` `BuildTemplate` | `CONDA_MICROMAMBA_V2_FAST = "conda/micromamba:v2-fast"` | | `BuildConfig`, `application.yml` | `wave.build.condasplit-image`, default `public.cr.seqera.io/wave/condasplit:v1` | -| `ContainerHelper`, `ContainerController` | Route v3 to `CondaHelper.containerFileV3`, passing the configured image. Singularity gets HTTP 400 | -| `CondaHelper` | `containerFileV3`. It shares the CONDA-type check, the v1 mamba-image override and the base image handling with v2 (`micromambaV2Opts`) | -| `TemplateUtils` | `condaToDockerFileUsingV3(lockFile, channels, opts, layersImage)` renders the template, with a `null` lock file for the conda-file input | +| `ContainerHelper`, `ContainerController` | Route v2-fast to `CondaHelper.containerFileV2Fast`, passing the configured image. Singularity gets HTTP 400 | +| `CondaHelper` | `containerFileV2Fast`. It shares the CONDA-type check, the v1 mamba-image override and the base image handling with v2 (`micromambaV2Opts`) | +| `TemplateUtils` | `condaToDockerFileUsingV2Fast(lockFile, channels, opts, layersImage)` renders the template, with a `null` lock file for the conda-file input | | `ContainerInspectServiceImpl` | Collects `RUN --mount ... from=` images for the build credentials, ignoring stage names | | `condasplit/` | Tool and tests, `frontend/` module, `Dockerfile` (`FROM scratch`, both binaries, frontend entrypoint), `Makefile`, `README.md` | | `.github/workflows/build-condasplit.yml` | Go tests on pull requests; on a Wave `[release]` commit, publishes the `condasplit/VERSION` tag. It skips a published tag with the same sources checksum (`io.seqera.condasplit.source` label) and fails for one with different sources | -Unchanged: `BuildStrategy` (`buildctl` arguments, cache, compression), `BuildLogServiceImpl`, persistence, `MultiPlatformBuildService` (one build per platform), scanning, mirroring and freeze. v3 produces its own Dockerfile, so it gets its own container ids and never collides with images from other templates. +Unchanged: `BuildStrategy` (`buildctl` arguments, cache, compression), `BuildLogServiceImpl`, persistence, `MultiPlatformBuildService` (one build per platform), scanning, mirroring and freeze. v2-fast produces its own Dockerfile, so it gets its own container ids and never collides with images from other templates. ## Acceptance and verification @@ -212,7 +212,7 @@ The builds used Wave-rendered Dockerfiles on the production BuildKit (`v0.25.2-r | Build not slower than v2 | 146 s against 283 s; export 19 s against 125 s | | Tool absent from the image | Yes | | Existing templates unchanged | 23 golden files for v1, v2, pixi and cran, byte-identical | -| Singularity or non-CONDA v3 requests | HTTP 400 | +| Singularity or non-CONDA v2-fast requests | HTTP 400 | | Tool image under 10 MB compressed | 8.0 MB amd64, 7.2 MB arm64 | | Smoke tests | GATK 4.6.2.0, gcnvkernel 0.9, samtools 1.24 on arm64 and amd64 | @@ -225,7 +225,7 @@ Tests: ## Rollout 1. Publish `public.cr.seqera.io/wave/condasplit:v1`, Wave's default: the `build-condasplit` workflow does it on the next Wave `[release]` commit. `public.cr.stage-seqera.io/wave/condasplit` is only for tests and local development. -2. Validate on the stage cluster: Nextflow with `wave.build.template = 'conda/micromamba:v3'`, the reference environment, and pull speed against v2. +2. Validate on the stage cluster: Nextflow with `wave.build.template = 'conda/micromamba:v2-fast'`, the reference environment, and pull speed against v2. 3. Release as opt-in. Enterprise installs mirror the image and set `wave.build.condasplit-image`. ## Risks and open items @@ -238,7 +238,7 @@ Tests: | Oversized files | A single file over 500 MB still produces a blob over 512 MB (warned, build continues) | | Custom `mambaImage` with files under `/opt/conda` | overlayfs copies them up on `rename()`: metadata is kept, but hardlinks among those files are split | | Not exercised in a real build | Slot overflow, oversized files, custom `commands`/`baseImage`/`mambaImage` (unit and render tests only), and a multi-platform request through Wave | -| Follow-ups | v3 in the Nextflow and `wave-cli` help text; an issue for the v1/v2 lock URL bug (D8) | +| Follow-ups | v2-fast in the Nextflow and `wave-cli` help text; an issue for the v1/v2 lock URL bug (D8) | ## Alternatives considered diff --git a/src/main/groovy/io/seqera/wave/configuration/BuildConfig.groovy b/src/main/groovy/io/seqera/wave/configuration/BuildConfig.groovy index 7c45bc9cc..05867c0b4 100644 --- a/src/main/groovy/io/seqera/wave/configuration/BuildConfig.groovy +++ b/src/main/groovy/io/seqera/wave/configuration/BuildConfig.groovy @@ -50,7 +50,7 @@ class BuildConfig { String singularityImageInit /** - * The image providing the {@code condasplit} tool used by the {@code conda/micromamba:v3} + * The image providing the {@code condasplit} tool used by the {@code conda/micromamba:v2-fast} * build template to split the Conda environment into multiple image layers */ @Value('${wave.build.condasplit-image:`public.cr.seqera.io/wave/condasplit:v1`}') diff --git a/src/main/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImpl.groovy b/src/main/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImpl.groovy index cd58d41dc..e61497d63 100644 --- a/src/main/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImpl.groovy +++ b/src/main/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImpl.groovy @@ -149,7 +149,7 @@ class ContainerInspectServiceImpl implements ContainerInspectService { result.add(repo) else // images mounted with `RUN --mount=...,from=` are pulled by the build too, - // e.g. the condasplit tool image of `conda/micromamba:v3` mirrored to a private registry + // e.g. the condasplit tool image of `conda/micromamba:v2-fast` mirrored to a private registry result.addAll(findMountRepositories(line.trim())) } return result diff --git a/src/main/groovy/io/seqera/wave/util/CondaHelper.groovy b/src/main/groovy/io/seqera/wave/util/CondaHelper.groovy index b1ef9af62..3b5a79377 100644 --- a/src/main/groovy/io/seqera/wave/util/CondaHelper.groovy +++ b/src/main/groovy/io/seqera/wave/util/CondaHelper.groovy @@ -31,12 +31,12 @@ import static TemplateUtils.condaPackagesToDockerFile import static TemplateUtils.condaPackagesToDockerFileUsingV2 import static TemplateUtils.condaPackagesToSingularityFile import static TemplateUtils.condaPackagesToSingularityFileV2 -import static TemplateUtils.condaToDockerFileUsingV3 +import static TemplateUtils.condaToDockerFileUsingV2Fast /** * Helper class for Conda/Micromamba container builds. * Supports the legacy v1 template, the v2 (MICROMAMBA_V2) template and the layered - * v3 (MICROMAMBA_V3) template. + * v2-fast (MICROMAMBA_V2_FAST) template. * * @author Paolo Di Tommaso */ @@ -98,7 +98,7 @@ class CondaHelper { } /** - * Generate a Dockerfile using the Micromamba v3 template. The Conda environment is installed + * Generate a Dockerfile using the `conda/micromamba:v2-fast` template. The Conda environment is installed * as with the v2 template and then split into multiple image layers by the {@code condasplit} tool. * Only supports CONDA package type and Docker format. Supports both lock files and environment files. * @@ -108,13 +108,13 @@ class CondaHelper { * @return The generated Dockerfile content * @throws BadRequestException if package type is not CONDA */ - static String containerFileV3(PackagesSpec spec, String containerImage, String layersImage) { - final opts = micromambaV2Opts(spec, containerImage, 'conda/micromamba:v3') - return condaToDockerFileUsingV3(tryGetLockFile(spec.entries), spec.channels, opts, layersImage) + static String containerFileV2Fast(PackagesSpec spec, String containerImage, String layersImage) { + final opts = micromambaV2Opts(spec, containerImage, 'conda/micromamba:v2-fast') + return condaToDockerFileUsingV2Fast(tryGetLockFile(spec.entries), spec.channels, opts, layersImage) } /** - * The Conda options of the templates based on micromamba 2.x, i.e. v2 and v3 + * The Conda options of the templates based on micromamba 2.x, i.e. v2 and v2-fast * * @throws BadRequestException if package type is not CONDA */ diff --git a/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy b/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy index 7d7d85add..7c20a6a80 100644 --- a/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy +++ b/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy @@ -35,7 +35,7 @@ import io.seqera.wave.service.request.TokenData import org.yaml.snakeyaml.Yaml import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V1 import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V2 -import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V3 +import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V2_FAST import static io.seqera.wave.api.BuildTemplate.CONDA_PIXI_V1 import static io.seqera.wave.api.BuildTemplate.CRAN_INSTALLR_V1 import static io.seqera.wave.service.builder.BuildFormat.SINGULARITY @@ -55,7 +55,7 @@ class ContainerHelper { * Dispatches to the appropriate helper based on build template and package type. * * @param req The container token request - * @param condasplitImage The image providing the {@code condasplit} tool used by the {@code conda/micromamba:v3} template + * @param condasplitImage The image providing the {@code condasplit} tool used by the {@code conda/micromamba:v2-fast} template * @return The generated container file content */ static String containerFileFromRequest(SubmitContainerTokenRequest req, String condasplitImage) { @@ -66,10 +66,10 @@ class ContainerHelper { if( spec.type == PackagesSpec.Type.CONDA && (!req.buildTemplate || req.buildTemplate==CONDA_MICROMAMBA_V2) ) { return CondaHelper.containerFileV2(spec, req.containerImage, singularity) } - if( req.buildTemplate == CONDA_MICROMAMBA_V3 ) { + if( req.buildTemplate == CONDA_MICROMAMBA_V2_FAST ) { if( singularity ) - throw new BadRequestException("Build template '${CONDA_MICROMAMBA_V3}' does not support Singularity format") - return CondaHelper.containerFileV3(spec, req.containerImage, condasplitImage) + throw new BadRequestException("Build template '${CONDA_MICROMAMBA_V2_FAST}' does not support Singularity format") + return CondaHelper.containerFileV2Fast(spec, req.containerImage, condasplitImage) } if( req.buildTemplate == CONDA_MICROMAMBA_V1 ) { return CondaHelper.containerFile(spec, singularity) diff --git a/src/main/java/io/seqera/wave/util/TemplateUtils.java b/src/main/java/io/seqera/wave/util/TemplateUtils.java index aaac83778..275868d91 100644 --- a/src/main/java/io/seqera/wave/util/TemplateUtils.java +++ b/src/main/java/io/seqera/wave/util/TemplateUtils.java @@ -130,12 +130,12 @@ static public String condaPackagesToSingularityFileV2(String packages, List condaChannels, CondaOpts opts, String layersImage) { + static public String condaToDockerFileUsingV2Fast(String lockFile, List condaChannels, CondaOpts opts, String layersImage) { final Map binding = new HashMap<>(); binding.put("layers_image", layersImage); binding.put("conda_file_add", "COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml"); @@ -148,7 +148,7 @@ static public String condaToDockerFileUsingV3(String lockFile, List cond binding.put("conda_file_add", "# micromamba can't read an explicit lock file from a URL, add it to the build stage\nADD " + lockFile + " " + path); binding.put("conda_file_opts", channels.stream().map(it -> "-c " + it + " ").collect(Collectors.joining()) + "-f " + path); } - return condaFileTemplateV2("/templates/conda-micromamba-v3/dockerfile-conda-file.txt", opts, binding); + return condaFileTemplateV2("/templates/conda-micromamba-v2-fast/dockerfile-conda-file.txt", opts, binding); } /** diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index 96a483913..0c6a75848 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -85,7 +85,7 @@ wave: buildkit-image: "public.cr.seqera.io/wave/buildkit:v0.25.2-rootless" singularity-image: "public.cr.seqera.io/wave/singularity:v4.2.1-r4" singularity-image-init: "public.cr.seqera.io/wave/busybox:latest" - # image providing the `condasplit` tool used by the `conda/micromamba:v3` build template + # image providing the `condasplit` tool used by the `conda/micromamba:v2-fast` build template condasplit-image: "public.cr.seqera.io/wave/condasplit:v1" # note changing this should be matching the micronaut.server.idle-timeout (see above) timeout: 900s diff --git a/src/main/resources/templates/conda-micromamba-v3/dockerfile-conda-file.txt b/src/main/resources/templates/conda-micromamba-v2-fast/dockerfile-conda-file.txt similarity index 100% rename from src/main/resources/templates/conda-micromamba-v3/dockerfile-conda-file.txt rename to src/main/resources/templates/conda-micromamba-v2-fast/dockerfile-conda-file.txt diff --git a/src/test/groovy/io/seqera/wave/controller/ContainerControllerTest.groovy b/src/test/groovy/io/seqera/wave/controller/ContainerControllerTest.groovy index 567cb86a7..82809f5cc 100644 --- a/src/test/groovy/io/seqera/wave/controller/ContainerControllerTest.groovy +++ b/src/test/groovy/io/seqera/wave/controller/ContainerControllerTest.groovy @@ -617,7 +617,7 @@ class ContainerControllerTest extends Specification { } } - def 'should create response with conda packages using micromamba v3 template' () { + def 'should create response with conda packages using `conda/micromamba:v2-fast` template' () { given: def dockerAuth = Mock(ContainerInspectServiceImpl) def freeze = new FreezeServiceImpl( inspectService: dockerAuth) @@ -631,7 +631,7 @@ class ContainerControllerTest extends Specification { addressResolver: addressResolver, containerService: tokenService, persistenceService: persistence, validationService: validationService, serverUrl: 'http://wave.com') and: def packagesSpec = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: ['https://foo.com/lock.yml'], channels: ['conda-forge']) - def req = new SubmitContainerTokenRequest(packages: packagesSpec, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3, freeze: true, buildRepository: 'docker.io/foo', towerAccessToken: '123') + def req = new SubmitContainerTokenRequest(packages: packagesSpec, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST, freeze: true, buildRepository: 'docker.io/foo', towerAccessToken: '123') def user = new User(email: 'foo@bar.com', userName: 'foo') def id = PlatformId.of(user, req) and: @@ -648,17 +648,17 @@ class ContainerControllerTest extends Specification { response.status.code == 200 and: buildConfig.condasplitImage == 'public.cr.seqera.io/wave/condasplit:v1' - build.buildTemplate == BuildTemplate.CONDA_MICROMAMBA_V3 + build.buildTemplate == BuildTemplate.CONDA_MICROMAMBA_V2_FAST build.containerFile.contains("RUN --mount=type=bind,from=${buildConfig.condasplitImage},source=/,target=/opt/wave-tools \\\n") build.containerFile.contains('COPY --link --from=build /layers/NN/ /\n') } - def 'should reject micromamba v3 template with singularity format' () { + def 'should reject `conda/micromamba:v2-fast` template with singularity format' () { given: def controller = new ContainerController(inspectService: Mock(ContainerInspectServiceImpl), buildConfig: buildConfig, validationService: validationService) and: def packagesSpec = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: ['bwa=0.7.15'], channels: ['bioconda']) - def req = new SubmitContainerTokenRequest(format: 'sif', packages: packagesSpec, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3, freeze: true, buildRepository: 'docker.io/foo', towerAccessToken: '123') + def req = new SubmitContainerTokenRequest(format: 'sif', packages: packagesSpec, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST, freeze: true, buildRepository: 'docker.io/foo', towerAccessToken: '123') def user = new User(email: 'foo@bar.com', userName: 'foo') when: @@ -666,7 +666,7 @@ class ContainerControllerTest extends Specification { then: def e = thrown(BadRequestException) - e.message == "Build template 'conda/micromamba:v3' does not support Singularity format" + e.message == "Build template 'conda/micromamba:v2-fast' does not support Singularity format" } def 'should create multi-platform singularity response' () { diff --git a/src/test/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImplTest.groovy b/src/test/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImplTest.groovy index 0d889b3fa..b819ae5d6 100644 --- a/src/test/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImplTest.groovy +++ b/src/test/groovy/io/seqera/wave/service/inspect/ContainerInspectServiceImplTest.groovy @@ -116,14 +116,14 @@ class ContainerInspectServiceImplTest extends Specification { 'RUN --network=none --mount=type=cache,target=/c --mount=type=bind,from=bar:1,target=/o \\' | ['bar:1'] } - def 'should find repos in micromamba v3 container file' () { + def 'should find repos in micromamba v2-fast container file' () { given: def LAYERS = 'public.cr.seqera.io/wave/condasplit:v1' def CONDA_OPTS = new CondaOpts([mambaImage: 'mambaorg/micromamba:2.1.1', baseImage: 'ubuntu:24.04']) expect: // the ADD line of a lock file adds no repository - ContainerInspectServiceImpl.findRepositories(TemplateUtils.condaToDockerFileUsingV3(LOCK_FILE, ['bioconda'], CONDA_OPTS, LAYERS)) + ContainerInspectServiceImpl.findRepositories(TemplateUtils.condaToDockerFileUsingV2Fast(LOCK_FILE, ['bioconda'], CONDA_OPTS, LAYERS)) == ['mambaorg/micromamba:2.1.1', LAYERS, 'ubuntu:24.04'] where: diff --git a/src/test/groovy/io/seqera/wave/util/CondaHelperTest.groovy b/src/test/groovy/io/seqera/wave/util/CondaHelperTest.groovy index 18c797b4b..182522c2b 100644 --- a/src/test/groovy/io/seqera/wave/util/CondaHelperTest.groovy +++ b/src/test/groovy/io/seqera/wave/util/CondaHelperTest.groovy @@ -290,38 +290,38 @@ class CondaHelperTest extends Specification { ex.message.contains("not supported by 'conda/micromamba:v2' build template") } - // === containerFileV3 (micromamba v3) tests === + // === containerFileV2Fast (micromamba v2-fast) tests === static final private String LAYERS_IMAGE = 'public.cr.seqera.io/wave/condasplit:v1' - def 'should create v3 docker file with packages'() { + def 'should create v2-fast docker file with packages'() { given: def CHANNELS = ['conda-forge', 'bioconda'] def PACKAGES = ['bwa=0.7.15', 'salmon=1.1.1'] def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS) when: - def result = CondaHelper.containerFileV3(packages, null, LAYERS_IMAGE) + def result = CondaHelper.containerFileV2Fast(packages, null, LAYERS_IMAGE) then: // the packages go into the conda.yml file of the build context - result == TemplateUtils.condaToDockerFileUsingV3(null, CHANNELS, CondaOpts.v2(), LAYERS_IMAGE) + result == TemplateUtils.condaToDockerFileUsingV2Fast(null, CHANNELS, CondaOpts.v2(), LAYERS_IMAGE) } - def 'should create v3 docker file with lock file'() { + def 'should create v2-fast docker file with lock file'() { given: def CHANNELS = ['conda-forge', 'bioconda'] def PACKAGES = ['https://foo.com/lock.yml'] def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS) when: - def result = CondaHelper.containerFileV3(packages, null, LAYERS_IMAGE) + def result = CondaHelper.containerFileV2Fast(packages, null, LAYERS_IMAGE) then: - result == TemplateUtils.condaToDockerFileUsingV3('https://foo.com/lock.yml', CHANNELS, CondaOpts.v2(), LAYERS_IMAGE) + result == TemplateUtils.condaToDockerFileUsingV2Fast('https://foo.com/lock.yml', CHANNELS, CondaOpts.v2(), LAYERS_IMAGE) } - def 'should use custom base image in v3'() { + def 'should use custom base image in v2-fast'() { given: def CHANNELS = ['conda-forge'] def PACKAGES = ['bwa=0.7.15'] @@ -329,7 +329,7 @@ class CondaHelperTest extends Specification { def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS, condaOpts: CONDA_OPTS) expect: - CondaHelper.containerFileV3(packages, CONTAINER_IMAGE, LAYERS_IMAGE).contains("FROM ${EXPECTED} AS prod") + CondaHelper.containerFileV2Fast(packages, CONTAINER_IMAGE, LAYERS_IMAGE).contains("FROM ${EXPECTED} AS prod") where: CONTAINER_IMAGE | EXPECTED @@ -337,7 +337,7 @@ class CondaHelperTest extends Specification { 'ubuntu:22.04' | 'ubuntu:22.04' // container image takes precedence } - def 'should use custom condaOpts and layers image in v3'() { + def 'should use custom condaOpts and layers image in v2-fast'() { given: def CHANNELS = ['conda-forge'] def PACKAGES = ['bwa=0.7.15'] @@ -350,14 +350,14 @@ class CondaHelperTest extends Specification { def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS, condaOpts: CONDA_OPTS) when: - def result = CondaHelper.containerFileV3(packages, null, 'my.registry.io/wave/condasplit:v2@sha256:1234') + def result = CondaHelper.containerFileV2Fast(packages, null, 'my.registry.io/wave/condasplit:v2@sha256:1234') then: - result == TemplateUtils.condaToDockerFileUsingV3(null, CHANNELS, CONDA_OPTS, 'my.registry.io/wave/condasplit:v2@sha256:1234') + result == TemplateUtils.condaToDockerFileUsingV2Fast(null, CHANNELS, CONDA_OPTS, 'my.registry.io/wave/condasplit:v2@sha256:1234') result.contains('RUN --mount=type=bind,from=my.registry.io/wave/condasplit:v2@sha256:1234,source=/,target=/opt/wave-tools') } - def 'should override a v1 mamba image with the v2 default in v3'() { + def 'should override a v1 mamba image with the v2 default in v2-fast'() { given: def CHANNELS = ['conda-forge'] def PACKAGES = ['bwa=0.7.15'] @@ -365,7 +365,7 @@ class CondaHelperTest extends Specification { def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS, condaOpts: CONDA_OPTS) when: - def result = CondaHelper.containerFileV3(packages, null, LAYERS_IMAGE) + def result = CondaHelper.containerFileV2Fast(packages, null, LAYERS_IMAGE) then: result.contains("FROM ${EXPECTED} AS build") @@ -378,15 +378,15 @@ class CondaHelperTest extends Specification { 'quay.io/custom/base:latest' | 'quay.io/custom/base:latest' // custom image kept } - def 'should throw exception for non-CONDA package type in v3'() { + def 'should throw exception for non-CONDA package type in v2-fast'() { given: def packages = new PackagesSpec(type: PackagesSpec.Type.CRAN, entries: ['dplyr']) when: - CondaHelper.containerFileV3(packages, null, LAYERS_IMAGE) + CondaHelper.containerFileV2Fast(packages, null, LAYERS_IMAGE) then: def ex = thrown(BadRequestException) - ex.message == "Package type 'CRAN' not supported by 'conda/micromamba:v3' build template" + ex.message == "Package type 'CRAN' not supported by 'conda/micromamba:v2-fast' build template" } } diff --git a/src/test/groovy/io/seqera/wave/util/ContainerHelperTest.groovy b/src/test/groovy/io/seqera/wave/util/ContainerHelperTest.groovy index 110021898..61a2c3763 100644 --- a/src/test/groovy/io/seqera/wave/util/ContainerHelperTest.groovy +++ b/src/test/groovy/io/seqera/wave/util/ContainerHelperTest.groovy @@ -919,9 +919,9 @@ class ContainerHelperTest extends Specification { '''.stripIndent() } - // === build with micromamba v3 tests + // === build with micromamba v2-fast tests - def 'should create conda docker file with packages and micromamba v3'() { + def 'should create conda docker file with packages and micromamba v2-fast'() { given: def CHANNELS = ['conda-forge', 'bioconda'] def PACKAGES = ['bwa=0.7.15', 'salmon=1.1.1'] @@ -930,16 +930,16 @@ class ContainerHelperTest extends Specification { entries: PACKAGES, channels: CHANNELS) and: - def req = new SubmitContainerTokenRequest(packages:packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3) + def req = new SubmitContainerTokenRequest(packages:packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST) when: def result = ContainerHelper.containerFileFromRequest(req, LAYERS_IMAGE) then: - result == CondaHelper.containerFileV3(new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS), null, LAYERS_IMAGE) + result == CondaHelper.containerFileV2Fast(new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: PACKAGES, channels: CHANNELS), null, LAYERS_IMAGE) } - def 'should create conda docker file with lock file, container image and micromamba v3'() { + def 'should create conda docker file with lock file, container image and micromamba v2-fast'() { given: def CHANNELS = ['conda-forge', 'bioconda'] def PACKAGES = ['https://foo.com/lock.yml'] @@ -950,40 +950,40 @@ class ContainerHelperTest extends Specification { channels: CHANNELS, condaOpts: CONDA_OPTS) and: - def req = new SubmitContainerTokenRequest(packages:packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3, containerImage: 'debian:12') + def req = new SubmitContainerTokenRequest(packages:packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST, containerImage: 'debian:12') when: def result = ContainerHelper.containerFileFromRequest(req, 'my.registry.io/wave/condasplit:v2') then: - result == CondaHelper.containerFileV3(packages, 'debian:12', 'my.registry.io/wave/condasplit:v2') + result == CondaHelper.containerFileV2Fast(packages, 'debian:12', 'my.registry.io/wave/condasplit:v2') result.contains('FROM debian:12 AS prod\n') } - def 'should reject singularity format with micromamba v3'() { + def 'should reject singularity format with micromamba v2-fast'() { given: def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: ['bwa=0.7.15'], channels: ['bioconda']) - def req = new SubmitContainerTokenRequest(packages: packages, format: 'sif', buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3) + def req = new SubmitContainerTokenRequest(packages: packages, format: 'sif', buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST) when: ContainerHelper.containerFileFromRequest(req, LAYERS_IMAGE) then: def e = thrown(BadRequestException) - e.message == "Build template 'conda/micromamba:v3' does not support Singularity format" + e.message == "Build template 'conda/micromamba:v2-fast' does not support Singularity format" } - def 'should reject non-conda package type with micromamba v3'() { + def 'should reject non-conda package type with micromamba v2-fast'() { given: def packages = new PackagesSpec(type: PackagesSpec.Type.CRAN, entries: ['dplyr']) - def req = new SubmitContainerTokenRequest(packages: packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V3) + def req = new SubmitContainerTokenRequest(packages: packages, buildTemplate: BuildTemplate.CONDA_MICROMAMBA_V2_FAST) when: ContainerHelper.containerFileFromRequest(req, LAYERS_IMAGE) then: def e = thrown(BadRequestException) - e.message == "Package type 'CRAN' not supported by 'conda/micromamba:v3' build template" + e.message == "Package type 'CRAN' not supported by 'conda/micromamba:v2-fast' build template" } def 'should create cran docker file with packages'() { @@ -1123,7 +1123,7 @@ class ContainerHelperTest extends Specification { def req = new SubmitContainerTokenRequest(packages: PACKAGES, buildTemplate: TEMPLATE, format: FORMAT, containerImage: CONTAINER_IMAGE) and: - // golden files captured from the release preceding 'conda/micromamba:v3' + // golden files captured from the release preceding 'conda/micromamba:v2-fast' def expected = this.class.getResource("/golden/${NAME}.txt").getText('UTF-8') when: diff --git a/src/test/groovy/io/seqera/wave/util/TemplateUtilsTest.groovy b/src/test/groovy/io/seqera/wave/util/TemplateUtilsTest.groovy index e25e5e2ef..f435812b6 100644 --- a/src/test/groovy/io/seqera/wave/util/TemplateUtilsTest.groovy +++ b/src/test/groovy/io/seqera/wave/util/TemplateUtilsTest.groovy @@ -955,7 +955,7 @@ class TemplateUtilsTest extends Specification { } /* ********************************************************************************* - * Micromamba v3 template tests + * `conda/micromamba:v2-fast` template tests * * Same build stage as the v2 template, plus the `condasplit` tool (mounted from the * layers image) moving the environment into at most 32 layer directories, which the @@ -965,7 +965,7 @@ class TemplateUtilsTest extends Specification { static final private String LAYERS_IMAGE = 'public.cr.seqera.io/wave/condasplit:v1' - def 'should create dockerfile using micromamba v3 template from conda file' () { + def 'should create dockerfile using `conda/micromamba:v2-fast` template from conda file' () { given: def CONDA_OPTS = new CondaOpts([ mambaImage: 'mambaorg/micromamba:2.1.1', @@ -974,7 +974,7 @@ class TemplateUtilsTest extends Specification { ]) expect: - TemplateUtils.condaToDockerFileUsingV3(null, null, CONDA_OPTS, LAYERS_IMAGE) == '''\ + TemplateUtils.condaToDockerFileUsingV2Fast(null, null, CONDA_OPTS, LAYERS_IMAGE) == '''\ # syntax=public.cr.seqera.io/wave/condasplit:v1 FROM mambaorg/micromamba:2.1.1 AS build USER root @@ -1007,12 +1007,12 @@ class TemplateUtilsTest extends Specification { '''.stripIndent() } - def 'should add the remote lock file to the micromamba v3 build stage' () { + def 'should add the remote lock file to the micromamba v2-fast build stage' () { given: def CONDA_OPTS = new CondaOpts([mambaImage: 'mambaorg/micromamba:2.1.1', baseImage: 'ubuntu:24.04']) when: - def result = TemplateUtils.condaToDockerFileUsingV3(LOCK_URL, ['conda-forge', 'bioconda'], CONDA_OPTS, LAYERS_IMAGE) + def result = TemplateUtils.condaToDockerFileUsingV2Fast(LOCK_URL, ['conda-forge', 'bioconda'], CONDA_OPTS, LAYERS_IMAGE) def lines = result.readLines() then: @@ -1046,7 +1046,7 @@ class TemplateUtilsTest extends Specification { 'https://foo.com/my%20env.yml' | 'conda.lock' } - def 'should render micromamba v3 #VARIANT template with commands' () { + def 'should render micromamba v2-fast #VARIANT template with commands' () { given: def CONDA_OPTS = new CondaOpts([ mambaImage: 'mambaorg/micromamba:2.1.1', @@ -1056,7 +1056,7 @@ class TemplateUtilsTest extends Specification { ]) when: - def result = TemplateUtils.condaToDockerFileUsingV3(VARIANT=='conda-file' ? null : 'https://foo.com/lock.yml', ['bioconda'], CONDA_OPTS, LAYERS_IMAGE) + def result = TemplateUtils.condaToDockerFileUsingV2Fast(VARIANT=='conda-file' ? null : 'https://foo.com/lock.yml', ['bioconda'], CONDA_OPTS, LAYERS_IMAGE) def lines = result.readLines() then: diff --git a/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java b/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java index 8b2050bd3..ebed73395 100644 --- a/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java +++ b/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java @@ -43,7 +43,7 @@ public final class BuildTemplate { * Build template for Micromamba v2-based multi-stage builds producing * the Conda environment as multiple image layers (Docker format only) */ - public static final String CONDA_MICROMAMBA_V3 = "conda/micromamba:v3"; + public static final String CONDA_MICROMAMBA_V2_FAST = "conda/micromamba:v2-fast"; /** * Build template for R/CRAN package builds diff --git a/wave-api/src/test/groovy/io/seqera/wave/api/BuildTemplateTest.groovy b/wave-api/src/test/groovy/io/seqera/wave/api/BuildTemplateTest.groovy index c8a7206c9..b07420be3 100644 --- a/wave-api/src/test/groovy/io/seqera/wave/api/BuildTemplateTest.groovy +++ b/wave-api/src/test/groovy/io/seqera/wave/api/BuildTemplateTest.groovy @@ -40,7 +40,7 @@ class BuildTemplateTest extends Specification { BuildTemplate.CONDA_PIXI_V1 == 'conda/pixi:v1' BuildTemplate.CONDA_MICROMAMBA_V1 == 'conda/micromamba:v1' BuildTemplate.CONDA_MICROMAMBA_V2 == 'conda/micromamba:v2' - BuildTemplate.CONDA_MICROMAMBA_V3 == 'conda/micromamba:v3' + BuildTemplate.CONDA_MICROMAMBA_V2_FAST == 'conda/micromamba:v2-fast' BuildTemplate.CRAN_INSTALLR_V1 == 'cran/installr:v1' } }