Skip to content

Rename the conda/micromamba:v3 template to conda/micromamba:v2-fast #1151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pditommaso merged 1 commit into from
Oct 2, 2026
Merged

Rename the conda/micromamba:v3 template to conda/micromamba:v2-fast #1151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions adr/20260930-conda-layered-builds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Layered Conda Builds (`conda/micromamba:v3`)
# Layered Conda Builds (`conda/micromamba:v2-fast`)

## Summary

This document describes the `conda/micromamba:v3` build template, which ships the Conda environment of a Docker image as multiple layers grouped by package, instead of one large layer. The template installs the environment exactly like `conda/micromamba:v2`. In the same build step, a small static Go tool (`condasplit`) moves the installed files into 32 fixed layer slots, each holding at most 500 MB before compression. The template is opt-in: existing templates, the default template and existing container ids are unchanged.
This document describes the `conda/micromamba:v2-fast` build template, which ships the Conda environment of a Docker image as multiple layers grouped by package, instead of one large layer. The template installs the environment exactly like `conda/micromamba:v2`. In the same build step, a small static Go tool (`condasplit`) moves the installed files into 32 fixed layer slots, each holding at most 500 MB before compression. The template is opt-in: existing templates, the default template and existing container ids are unchanged.

## Context

Expand All @@ -29,8 +29,8 @@ For the reference environment (`bioconda::gatk4=4.6.2.0` and `bioconda::gcnvkern

### How it works

1. User submits a container request with `buildTemplate: "conda/micromamba:v3"`
2. Wave renders the v3 Dockerfile, naming the tool image configured with `wave.build.condasplit-image`, both as its BuildKit frontend (`# syntax=`, first line) and as the image mounted into the install step
1. User submits a container request with `buildTemplate: "conda/micromamba:v2-fast"`
2. Wave renders the v2-fast Dockerfile, naming the tool image configured with `wave.build.condasplit-image`, both as its BuildKit frontend (`# syntax=`, first line) and as the image mounted into the install step
3. Two-stage build executes:
- **Stage 1 (build)**: Runs the `conda/micromamba:v2` install commands and prints the conda lock. A lock-file URL is first added to the stage with `ADD` and installed from the local copy, because micromamba can't read an explicit lock file from a URL. In the same `RUN`, it runs `condasplit` from the mounted tool image, which moves every file of `/opt/conda` (except `pkgs/`) into at most 32 directories `/layers/00`, `/layers/01`, …, each rooted at `/`, and prints the layer plan
- **Stage 2 (prod)**: Starts from `{{base_image}}` and adds each layer directory as its own layer with `COPY --link`
Expand All @@ -49,13 +49,13 @@ For the reference environment (`bioconda::gatk4=4.6.2.0` and `bioconda::gcnvkern

### 1. New Versioned Template

**Decision:** Added `conda/micromamba:v3` (`BuildTemplate.CONDA_MICROMAMBA_V3`) instead of changing `conda/micromamba:v2`. `BuildTemplate.defaultTemplate()` still returns `conda/micromamba:v2`.
**Decision:** Added `conda/micromamba:v2-fast` (`BuildTemplate.CONDA_MICROMAMBA_V2_FAST`) instead of changing `conda/micromamba:v2`. `BuildTemplate.defaultTemplate()` still returns `conda/micromamba:v2`.

| Value | Conda layers | Default |
|-------|--------------|---------|
| `conda/micromamba:v1` | 1 (single-stage) | No |
| `conda/micromamba:v2` | 1 | Yes (`CONDA` packages) |
| `conda/micromamba:v3` | Up to 32, one per used slot | No |
| `conda/micromamba:v2-fast` | Up to 32, one per used slot | No |
| `conda/pixi:v1` | 1 | No |

**Rationale:**
Expand Down Expand Up @@ -155,7 +155,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \

### 7. Package Cache Dropped

**Decision:** `conda/micromamba:v3` excludes `/opt/conda/pkgs` from the image. The other templates keep it.
**Decision:** `conda/micromamba:v2-fast` excludes `/opt/conda/pkgs` from the image. The other templates keep it.

**Rationale:**
- Extracted packages are not needed at runtime
Expand All @@ -164,7 +164,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \

### 8. Docker Only

**Decision:** A `conda/micromamba:v3` request with `format: "sif"` fails with HTTP 400 `Build template 'conda/micromamba:v3' does not support Singularity format`. There are no Singularity template files.
**Decision:** A `conda/micromamba:v2-fast` request with `format: "sif"` fails with HTTP 400 `Build template 'conda/micromamba:v2-fast' does not support Singularity format`. There are no Singularity template files.

**Rationale:**
- A SIF image is a single squashfs file, so layering doesn't apply
Expand All @@ -180,7 +180,7 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \

### 10. BuildKit Frontend Repeating the Layer `COPY` Line

**Decision:** The v3 Dockerfile starts with `# syntax={{layers_image}}`, which makes the tool image its BuildKit frontend. The frontend (`condasplit/frontend/`, entrypoint `/condasplit-frontend`) builds the `build` stage, lists `/layers`, repeats the `COPY --link --from=build /layers/NN/ /` line for each directory and builds the whole Dockerfile. Both builds are delegated to the built-in Dockerfile frontend.
**Decision:** The v2-fast Dockerfile starts with `# syntax={{layers_image}}`, which makes the tool image its BuildKit frontend. The frontend (`condasplit/frontend/`, entrypoint `/condasplit-frontend`) builds the `build` stage, lists `/layers`, repeats the `COPY --link --from=build /layers/NN/ /` line for each directory and builds the whole Dockerfile. Both builds are delegated to the built-in Dockerfile frontend.

**Rationale:**
- The image has exactly one layer per layer directory. BuildKit adds a layer for every `COPY`, even an empty one, so 32 fixed `COPY` lines gave a small environment such as `bioconda::samtools` 30 empty layers
Expand All @@ -206,28 +206,28 @@ RUN --mount=type=bind,from={{layers_image}},source=/,target=/opt/wave-tools \

| Value | Constant | Description |
|-------|----------|-------------|
| `conda/micromamba:v3` | `BuildTemplate.CONDA_MICROMAMBA_V3` | Layered multi-stage Micromamba build, Docker only |
| `conda/micromamba:v2-fast` | `BuildTemplate.CONDA_MICROMAMBA_V2_FAST` | Layered multi-stage Micromamba build, Docker only |

No new request fields. The template accepts the same inputs as `conda/micromamba:v2`: conda file, conda lock file URL, custom `baseImage`/`containerImage`, `mambaImage`, `basePackages` and `commands`.

### New Error Responses

| Condition | Response |
|-----------|----------|
| `format: "sif"` | HTTP 400 `Build template 'conda/micromamba:v3' does not support Singularity format` |
| Non-`CONDA` package type | HTTP 400 `Package type '<type>' not supported by 'conda/micromamba:v3' build template` |
| `format: "sif"` | HTTP 400 `Build template 'conda/micromamba:v2-fast' does not support Singularity format` |
| Non-`CONDA` package type | HTTP 400 `Package type '<type>' not supported by 'conda/micromamba:v2-fast' build template` |

## Configuration Changes

| Property | Default | Description |
|----------|---------|-------------|
| `wave.build.condasplit-image` | `public.cr.seqera.io/wave/condasplit:v1` | Tool image mounted into `conda/micromamba:v3` builds |
| `wave.build.condasplit-image` | `public.cr.seqera.io/wave/condasplit:v1` | Tool image mounted into `conda/micromamba:v2-fast` builds |

Each tool release gets a new immutable tag (`v1`, `v2`, …), taken from `condasplit/VERSION`, and the config value may also pin a digest. The tag stays readable because it appears in the v3 container file; a SHA-256 of the tool sources goes into the `io.seqera.condasplit.source` image label instead, and the release fails when an existing tag has different sources. The image reference is part of the v3 container file, so a tool upgrade changes container ids for v3 images only. Enterprise installs without access to `public.cr.seqera.io` mirror the image and set this property.
Each tool release gets a new immutable tag (`v1`, `v2`, …), taken from `condasplit/VERSION`, and the config value may also pin a digest. The tag stays readable because it appears in the v2-fast container file; a SHA-256 of the tool sources goes into the `io.seqera.condasplit.source` image label instead, and the release fails when an existing tag has different sources. The image reference is part of the v2-fast container file, so a tool upgrade changes container ids for v2-fast images only. Enterprise installs without access to `public.cr.seqera.io` mirror the image and set this property.

## Template Comparison

| Aspect | `conda/micromamba:v2` | `conda/micromamba:v3` |
| Aspect | `conda/micromamba:v2` | `conda/micromamba:v2-fast` |
|--------|-----------------------|-----------------------|
| Build stages | 2 | 2 |
| Conda environment layers | 1 | One per used slot, up to 32, at most 500 MB each before compression |
Expand All @@ -254,7 +254,7 @@ Reference environment built with BuildKit v0.25.2, gzip compression and OCI medi
|----------|-------|
| API Models | `BuildTemplate.java` |
| Configuration | `BuildConfig.groovy`, `application.yml` |
| Templates | `conda-micromamba-v3/dockerfile-conda-file.txt` |
| Templates | `conda-micromamba-v2-fast/dockerfile-conda-file.txt` |
| Helpers | `TemplateUtils.java`, `CondaHelper.groovy`, `ContainerHelper.groovy` |
| Controller | `ContainerController.groovy` |
| Services | `ContainerInspectServiceImpl.groovy` |
Expand Down Expand Up @@ -285,14 +285,14 @@ Reference environment built with BuildKit v0.25.2, gzip compression and OCI medi
### Neutral
- `conda/micromamba:v2` remains the default template
- Tuning the grouping parameters requires a new template version
- A tool upgrade changes container ids of `conda/micromamba:v3` images only
- A tool upgrade changes container ids of `conda/micromamba:v2-fast` images only
- The same tool could later serve `conda/pixi` environments, which also have `conda-meta/`

## Follow-ups

- **Production default**: At the production release, switch the default of `wave.build.condasplit-image` to `public.cr.seqera.io/wave/condasplit:v1` in `BuildConfig.groovy`, `application.yml` and `docs/install/reference.md`
- **Nextflow**: Add `conda/micromamba:v3` to the `wave.build.template` description in `plugins/nf-wave/src/main/io/seqera/wave/plugin/config/WaveConfig.groovy`, and correct its stated default to `conda/micromamba:v2`
- **Wave CLI**: Add `conda/micromamba:v3` to the `--build-template` help text in `app/src/main/java/io/seqera/wave/cli/App.java`
- **Nextflow**: Add `conda/micromamba:v2-fast` to the `wave.build.template` description in `plugins/nf-wave/src/main/io/seqera/wave/plugin/config/WaveConfig.groovy`, and correct its stated default to `conda/micromamba:v2`
- **Wave CLI**: Add `conda/micromamba:v2-fast` to the `--build-template` help text in `app/src/main/java/io/seqera/wave/cli/App.java`

## References

Expand Down
2 changes: 1 addition & 1 deletion condasplit/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ RUN go test ./... \

FROM scratch
COPY --from=build /condasplit /condasplit-frontend /
# the image is also the BuildKit frontend of the conda/micromamba:v3 template, see frontend/
# the image is also the BuildKit frontend of the conda/micromamba:v2-fast template, see frontend/
ENTRYPOINT ["/condasplit-frontend"]
8 changes: 4 additions & 4 deletions condasplit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# condasplit

A small static tool used by the `conda/micromamba:v3` build template to ship a conda
A small static tool used by the `conda/micromamba:v2-fast` build template to ship a conda
environment as several image layers instead of one large layer.

It runs in the same `RUN` step that installs the environment. It moves every file of the
Expand Down Expand Up @@ -84,7 +84,7 @@ condasplit --src DIR --out DIR --slots N
| `--own-layer-size` | 50000000 | Packages at least this size get a layer to themselves |
| `--exclude` | none | Path relative to `--src` left in place and not layered (repeatable) |

The `conda/micromamba:v3` template runs:
The `conda/micromamba:v2-fast` template runs:

```
/opt/wave-tools/condasplit --src /opt/conda --out /layers \
Expand Down Expand Up @@ -126,7 +126,7 @@ and cross-compiles a static binary (`CGO_ENABLED=0`) from the pinned `golang` im

Each release gets a new immutable tag (`v1`, `v2`, ...), set in the `VERSION` file. Wave
pins the exact tag with the `wave.build.condasplit-image` setting, so a new tool version
changes the v3 container file and therefore the container id. Never overwrite an existing tag.
changes the v2-fast container file and therefore the container id. Never overwrite an existing tag.

**Stage** (default registry of the `Makefile`, only for tests and local development):

Expand All @@ -140,7 +140,7 @@ make all
`public.cr.seqera.io/wave/condasplit:<VERSION>` on the next Wave release commit (`[release]`
on `master`). Then point the default of `wave.build.condasplit-image` to the new tag.

The tag is always the `VERSION` value, never a checksum: Wave writes it into the v3 container
The tag is always the `VERSION` value, never a checksum: Wave writes it into the v2-fast container
file and the build logs, where `condasplit:v1` is readable and a checksum isn't. The checksum
goes into an `io.seqera.condasplit.source` image label instead: a SHA-256 of the tracked files of
this directory, the Markdown docs excluded. `make release` uses it to make sure a tag always holds
Expand Down
4 changes: 2 additions & 2 deletions condasplit/frontend/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/

// condasplit-frontend is the BuildKit frontend of the conda/micromamba:v3 build template.
// condasplit-frontend is the BuildKit frontend of the conda/micromamba:v2-fast build template.
// Wave writes the Dockerfile before the environment is known, so the template has one
// COPY line for the layer directories: this frontend builds the install stage first,
// then builds the image with that line repeated for each directory condasplit created.
Expand All @@ -41,7 +41,7 @@ import (
"github.com/moby/buildkit/util/appcontext"
)

// the COPY line of the layer directories in the conda/micromamba:v3 template
// the COPY line of the layer directories in the conda/micromamba:v2-fast template
var layersLine = regexp.MustCompile(`(?m)^COPY --link --from=build /layers/NN/ /\n`)

func main() {
Expand Down
12 changes: 6 additions & 6 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ Returns the name of the container request made available by Wave.
| `pixiImage` | Name of the Docker image used for Pixi package manager (e.g., `ghcr.io/prefix-dev/pixi:latest`). |
| `cranOpts` | CRAN build options (when `type` is `CRAN`). |
| `rImage` | Name of the R Docker image used to build CRAN containers (e.g., `rocker/r-ver:4.4.1`). |
| `buildTemplate` | The build template to use for container builds. Supported values: `conda/micromamba:v1` (single-stage build using Micromamba 1.x), `conda/pixi:v1` (multi-stage build using Pixi), `conda/micromamba:v2` (multi-stage build using Micromamba 2.x), `conda/micromamba:v3` (multi-stage build using Micromamba 2.x that splits the Conda environment into multiple image layers, Docker only, see [Layered Conda builds](#layered-conda-builds)), `cran/installr:v1` (CRAN packages). The multi-stage templates produce smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build with `conda/pixi:v1` and `conda/micromamba:v2`). Default: `conda/micromamba:v2` for `CONDA` packages, `cran/installr:v1` for `CRAN` packages. |
| `buildTemplate` | The build template to use for container builds. Supported values: `conda/micromamba:v1` (single-stage build using Micromamba 1.x), `conda/pixi:v1` (multi-stage build using Pixi), `conda/micromamba:v2` (multi-stage build using Micromamba 2.x), `conda/micromamba:v2-fast` (multi-stage build using Micromamba 2.x that splits the Conda environment into multiple image layers, Docker only, see [Layered Conda builds](#layered-conda-builds)), `cran/installr:v1` (CRAN packages). The multi-stage templates produce smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build with `conda/pixi:v1` and `conda/micromamba:v2`). Default: `conda/micromamba:v2` for `CONDA` packages, `cran/installr:v1` for `CRAN` packages. |
| `nameStrategy` | The name strategy to be used to create the name of the container built by Wave. Options: `none`, `tagPrefix`, `imageSuffix`. | |

#### Response
Expand Down Expand Up @@ -493,7 +493,7 @@ Get status of build against `buildId` passed as path variable
}
```

- Create Docker image with Micromamba v3 template (layered multi-stage build):
- Create Docker image with `conda/micromamba:v2-fast` template (layered multi-stage build):

**Request**

Expand All @@ -503,7 +503,7 @@ Get status of build against `buildId` passed as path variable
--data '{
"containerPlatform": "linux/amd64",
"format": "docker",
"buildTemplate": "conda/micromamba:v3",
"buildTemplate": "conda/micromamba:v2-fast",
"packages":{
"type": "CONDA",
"entries": ["gatk4=4.6.2.0", "gcnvkernel=0.9"],
Expand Down Expand Up @@ -620,7 +620,7 @@ Get status of build against `buildId` passed as path variable
```

:::note
Multi-stage build templates (`conda/pixi:v1`, `conda/micromamba:v2`, and `conda/micromamba:v3`) create optimized Docker container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools.
Multi-stage build templates (`conda/pixi:v1`, `conda/micromamba:v2`, and `conda/micromamba:v2-fast`) create optimized Docker container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools.
:::

:::important
Expand All @@ -633,7 +633,7 @@ As a result, the `baseImage` option has no effect on Singularity builds. It only

#### Layered Conda builds

The `conda/micromamba:v3` template installs the Conda environment with the same steps as `conda/micromamba:v2`, then splits it into multiple image layers instead of shipping it as one large layer. Use it when a registry, proxy, or CDN cache rejects or doesn't cache layers larger than 512 MB, or to let container runtimes download and unpack the environment in parallel.
The `conda/micromamba:v2-fast` template installs the Conda environment with the same steps as `conda/micromamba:v2`, then splits it into multiple image layers instead of shipping it as one large layer. Use it when a registry, proxy, or CDN cache rejects or doesn't cache layers larger than 512 MB, or to let container runtimes download and unpack the environment in parallel.

The environment is split by Conda package:

Expand All @@ -649,7 +649,7 @@ The build log shows the layer plan between the `>> CONDA_LAYERS_START` and `<< C
The container content is the same as `conda/micromamba:v2` produces for the same environment, except that the unused package cache (`/opt/conda/pkgs`) is dropped, which also makes the image smaller. The template supports the same inputs as `conda/micromamba:v2`: Conda packages, Conda environment files, Conda lock file URLs, and the `condaOpts` options. Custom `commands` run after the Conda layers and add their own layers.

:::important
`conda/micromamba:v3` builds Docker images only. Singularity requests (`format: "sif"`) are rejected with HTTP 400 and the message `Build template 'conda/micromamba:v3' does not support Singularity format`.
`conda/micromamba:v2-fast` builds Docker images only. Singularity requests (`format: "sif"`) are rejected with HTTP 400 and the message `Build template 'conda/micromamba:v2-fast' does not support Singularity format`.
:::

### GET `/v1alpha1/builds/{buildId}/status`
Expand Down
Loading
Loading