diff --git a/.github/actions/setup-mkosi/action.yml b/.github/actions/setup-mkosi/action.yml new file mode 100644 index 00000000..d709e6b1 --- /dev/null +++ b/.github/actions/setup-mkosi/action.yml @@ -0,0 +1,38 @@ +# Set up an ubuntu-24.04 runner to build a moose image with mkosi. Shared by +# ci-cloud-image.yml (the build, boot proofs and publish) and os-lock-bump.yml +# (the scheduled OS package lock bump, #560), so both build with the same mkosi +# and the same host tools, and a bump resolves exactly what CI will check. +name: Set up mkosi +description: Relax the userns restriction, install the image-build tools and mkosi v26. +runs: + using: composite + steps: + # mkosi 26 builds rootless: it unshares a user namespace and drops + # capabilities inside it. Ubuntu 24.04 ships + # kernel.apparmor_restrict_unprivileged_userns=1, which hands an unconfined + # process a userns with no CAP_SETPCAP, so mkosi's PR_CAPBSET_DROP returns + # EPERM and the build dies at sandbox bring-up (#189). Relaxing the knob is + # safe on an ephemeral runner. + - name: Relax AppArmor unprivileged-userns restriction + shell: bash + run: | + knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns + echo "before: $(cat "$knob" 2>/dev/null || echo 'n/a')" + sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 + echo "after: $(cat "$knob")" + + # Host tooling mkosi needs to bootstrap a Debian tools tree and build + # rootless (uidmap/mmdebstrap/keyring; dosfstools/mtools for the ESP; + # squashfs-tools/e2fsprogs for the root; systemd-container for the nspawn + # build sandbox; libpam0g-dev for the slim host-agent-real CGO build). + - name: Install mkosi and image-build tooling + shell: bash + run: | + sudo apt-get update + sudo apt-get install -y \ + uidmap mmdebstrap debian-archive-keyring \ + dosfstools mtools squashfs-tools e2fsprogs systemd-container \ + libpam0g-dev + git clone --depth=1 --branch v26 \ + https://github.com/systemd/mkosi.git /tmp/mkosi-src + sudo ln -sf /tmp/mkosi-src/bin/mkosi /usr/local/bin/mkosi diff --git a/.github/workflows/ci-cloud-image.yml b/.github/workflows/ci-cloud-image.yml index e5f936a8..1841d757 100644 --- a/.github/workflows/ci-cloud-image.yml +++ b/.github/workflows/ci-cloud-image.yml @@ -5,15 +5,31 @@ # separate, downstream concern that consumes this asset; this workflow does not # do it and holds no provider credentials. # -# Three triggers, three intents: +# Two release lines publish from here (BUILD.md # Versioning, DECISIONS.md +# 2026-10-01), each behind its own switch: +# - the OS line (`VERSION`, git tag `vX.Y.Z`): the disk image, attached to the +# `vX.Y.Z` GitHub Release. Gated on SHOULD_PUBLISH_OS. +# - the control-plane line (`CONTROL_PLANE_VERSION`, git tag +# `control-plane-vX.Y.Z`): the brain and UI images pushed to ghcr, tagged +# `v` + `latest`. Gated on SHOULD_PUBLISH_CP. +# Both lines share the one build and the one boot proof: a control-plane-only +# release builds the disk image too, boots it, and pushes the images that image +# baked, so a control plane is only ever published after it was proven to boot. +# +# Triggers and their intents: # - `workflow_call` — the AUTOMATED release path. `release.yml` runs on every -# push to `main`; when it finds `VERSION` names a version with no matching -# tag yet, it creates the tag + GitHub Release itself, THEN calls this -# workflow with `publish: true` (`secrets: inherit`) to build and publish -# the image for that same commit. This is the path a release actually -# takes now — read on for why it isn't the `push: tags` trigger below. -# - `push: tags: v*` — the MANUAL escape hatch: a human pushing a `vX.Y.Z` -# tag by hand (e.g. re-cutting an image for an already-tagged release +# push to `main`; when it finds a version file names a version with no +# matching tag yet, it creates the tag + GitHub Release itself, THEN calls +# this workflow with `publish_os` and/or `publish_control_plane` set +# (`secrets: inherit`) to build and publish for that same commit. This is +# the path a release actually takes now — read on for why it isn't the +# `push: tags` trigger below. +# - `push: tags: v*` — the MANUAL escape hatch for the OS line: a human +# pushing a `vX.Y.Z` tag by hand. It publishes the OS release files only +# (the disk image and the signed bundle, #562), never the control-plane images (a `v*` tag is an OS release now). A +# `control-plane-v*` tag does not match `v*` and triggers nothing; re-publish +# the control plane with workflow_dispatch instead (`publish=false +# publish_control_plane=true`) (e.g. re-cutting an image for an already-tagged release # without touching `release.yml`). Deliberately KEPT alongside # `workflow_call` rather than replaced by it, for a reason that is easy to # "fix" into a bug if you don't know it: a tag created by `release.yml` @@ -26,10 +42,17 @@ # triggers can never fire for the same tag-creation event, so there is no # double-build risk from keeping both — `push: tags` only ever fires for a # human-originated tag push (a PAT or a person's own `git push --tags`), -# never for the auto-tag `release.yml` creates. +# never for the auto-tag `release.yml` creates. Since #562 that OS +# release is the whole OS set: the disk image AND the signed bundle, so it +# needs the release root CA committed and the `os-release` signer, like +# any OS release (docs/dev/rauc-signing.md). # - `workflow_dispatch` — manual, on-demand, ad hoc ref. The `publish` input -# still governs whether the run publishes (default true); `publish=false` -# gives a build-only smoke run. Unchanged from before tags existed. +# governs whether the run publishes BOTH lines (default true); +# `publish=false` gives a build-only smoke run. To re-publish one line only, +# pass `publish=false` with `publish_os=true` or `publish_control_plane=true`. +# Each publish step refuses to overwrite a release cut from another commit, +# and adds only what is missing: it never replaces an asset on a Release or +# an image tag on ghcr that is already published (see the guards). # - `pull_request` (path-filtered, #389) — the UPDATER GUARD. A PR that touches # the control-plane updater runs this lane automatically and boots ONLY the # `update` scenario (see the boot-test step). It publishes nothing: no tag @@ -41,14 +64,29 @@ # updater lands on `dev` and stays green until someone runs this workflow by # hand (the same argument that put the `access` boot in the gate). # -# The three publish-capable triggers share one `publish` input name/shape (`workflow_dispatch` and -# `workflow_call` inputs both surface as the same `inputs.*` context), which -# is why `SHOULD_PUBLISH` below needs no per-trigger special-casing for the -# call path: `inputs.publish` already carries the right value regardless of -# whether a human or `release.yml` set it. +# `workflow_dispatch` and `workflow_call` inputs both surface as the same +# `inputs.*` context. Dispatch has a `publish` switch for both lines plus one per +# line; the call has one per line. An input a trigger does not declare reads as null, which is +# falsy, so each SHOULD_PUBLISH_* below simply ORs the inputs that may set it. +# +# A publish is a deliberate act, not a per-push one. # -# Either way the full build takes ~10 min and a publish is a deliberate act, -# not a per-push one. +# Shape (the CI speed-up, docs/progress/ci-cloud-image-speedup.md; `sign` from +# #562). Four jobs: +# - `build` runs every assert, builds the production image and the boot-proof +# image once, and uploads them as artifacts. It also picks the boots. +# - `boot` is a matrix with one job per boot group and firmware, all at once, +# each booting the downloaded boot-proof image. `unseeded seeded` (and +# `frozen`, when it is asked for) stay one group: they share one disk. +# - `sign` runs only when the OS line publishes, after every `boot` job +# passed. It alone enters the `os-release` environment and re-signs the OS +# update bundle with the release signer; it does nothing else. +# - `publish` runs only when a line publishes, and only after `build` and +# every `boot` job passed (and `sign`, when the OS line publishes). It publishes the image and the control-plane +# images that `build` made, the same bytes the boots proved. +# A full run takes about 12 min instead of about 27, and uses more runner +# minutes in total (about 34 instead of about 27). The repo is public, so +# standard runners cost nothing. # # Closes the gap #189/#238 carried: until now the cloud image had NO automated # build — `make build-cloud-image` ran only by hand, and on an Ubuntu 24.04 box @@ -91,13 +129,23 @@ on: # - internal/api/system*.go POST /api/v1/system/update, the trigger the # boot drives # - dev/cloud/** the lane and its in-guest assertions - # - this workflow + # - dev/control-plane/**, dev/release/ghcr-resolve.sh, Makefile + # the control-plane bundle both images bake (#566) + # - dev/os-lock/** the OS package lock (#560). A PR that touches + # it runs the FULL boot list, not only `update` + # (see "Choose the boots"): a lock bump changes + # every package the boots stand on. + # - the shared mkosi setup action, and this workflow pull_request: # `ready_for_review` is not in the default set and is needed here: the job # skips draft PRs (see the job's `if`), so marking one ready has to be what # starts the run. Without it a PR opened as a draft would never build. types: [opened, synchronize, reopened, ready_for_review] - branches: [dev, main] + # hotfix/** (#560): an OS lock bump for a patch release opens its PR into + # hotfix/X.Y.Z (docs/dev/contributing.md # OS patch releases from a lock + # bump). It touches dev/os-lock/, so it gets the lock check and the full + # boot list like a bump into dev. + branches: [dev, main, "hotfix/**"] paths: - "internal/hostagent/**" - "cmd/host-agent-real/**" @@ -105,13 +153,30 @@ on: - "internal/lifecycle/lifecycle.go" - "internal/api/system*.go" - "dev/cloud/**" + - "dev/os-lock/**" + # The control-plane bundle the images bake (#566): the pull, the record + # and the make targets that run them. + - "dev/control-plane/**" + - "dev/release/ghcr-resolve.sh" + - "Makefile" + - ".github/actions/setup-mkosi/**" - ".github/workflows/ci-cloud-image.yml" workflow_dispatch: inputs: publish: - description: "Attach the built image to the tagged GitHub Release as an asset" + description: "Publish both lines: attach the disk image to the v Release and push the control-plane images to ghcr" type: boolean default: true + # One line at a time, for a re-publish of just that line. Use with + # publish=false, or `publish` turns both lines on anyway. + publish_os: + description: "With publish=false: attach the disk image to the v Release only" + type: boolean + default: false + publish_control_plane: + description: "With publish=false: push the control-plane images to ghcr only" + type: boolean + default: false # Run only some boots, for a quick proof of one scenario (#531), for # example `remap`. Empty runs the full gate list below. The names are the # ones dev/cloud/run-cloud-tests.sh knows, and it refuses any other, so a @@ -123,70 +188,154 @@ on: description: "Boots to run, space separated (empty = the full gate list)" type: string default: "" + # Which brain and UI the images bake (#566, BUILD.md # Versioning). + # `local` builds them from this commit; `released` pulls the last + # released control plane (CONTROL_PLANE_VERSION) from ghcr by digest, + # which is what an OS-only release bakes. Use `released` with + # publish=false to see, before cutting an OS release, that the OS boots + # with the control plane it will ship. Only a run that publishes nothing + # reads it: an OS-only release always bakes `released`, a run that + # publishes the control plane always bakes `local`, and asking for + # `released` there fails at once. + control_plane: + description: "Brain and UI to bake: local (build this commit) or released (ghcr, v)" + type: choice + options: [local, released] + default: local workflow_call: inputs: - publish: - description: "Attach the built image to the tagged GitHub Release as an asset" + publish_os: + description: "OS release: attach the built disk image to the v GitHub Release" type: boolean - default: true + default: false + publish_control_plane: + description: "Control-plane release: push the brain and UI images to ghcr as v" + type: boolean + default: false + # The OS lock bump (os-lock-bump.yml, #560) calls this workflow to boot + # its bot/os-lock branch when no PR can be opened for it. A called + # workflow checks out the CALLER's ref by default, so the bump names the + # branch here. A run with a ref must not publish (asserted below): the + # publish steps describe github.sha, which is not the commit built. + ref: + description: "Git ref to build instead of the caller's (the OS lock bump branch)" + type: string + default: "" -# contents: write is the minimum grant for `gh release upload`/`gh release -# create` in the release-asset step below. packages: write is what lets the -# job's own GITHUB_TOKEN push the control-plane images to ghcr (# 370) — still -# no long-lived registry credential anywhere in this lane. Neither grant is -# used on a pull_request run: both steps that need them are gated on -# SHOULD_PUBLISH, which is false there (and asserted so, below). A PR from a -# fork gets a read-only token regardless — it still builds and boots. +# Read only by default. Every job sets its own permissions: `build` and `boot` +# read only, and `publish` alone gets the two write grants. contents: write is +# the minimum grant for `gh release upload`/`gh release create` in the +# release-asset step, and packages: write is what lets the job's own +# GITHUB_TOKEN push the control-plane images to ghcr (#370), so there is still +# no long-lived registry credential anywhere in this lane. The publish job does +# not run on a pull_request run: SHOULD_PUBLISH is false there (and asserted +# so, in `build`). A PR from a fork gets a read-only token regardless, and it +# still builds and boots. A caller (release.yml, os-lock-bump.yml) must still +# grant both, since a called workflow can only narrow what its caller grants. permissions: - contents: write - packages: write + contents: read -# Cancel superseded runs on the same ref. +# Cancel superseded runs on the same ref. A call with `ref` (the OS lock bump) +# is grouped by that ref, not by the caller's: the caller's ref is `dev`, and a +# shared group would let a bump cancel an unrelated run on `dev`, or the reverse. +# +# A run that publishes has a group of its own (#562), marked `publish` +# before the ref (`build` for every other run), and is never cancelled. The +# marker comes first so no branch name, such as `main-publish`, can land in +# the other group: a cancelled Release upload or ghcr push +# leaves a half-published set that the next run then refuses. GitHub reads +# `cancel-in-progress` from the NEW run, so one shared group would let a +# build-only run on the same ref cancel a release in flight. With two groups a +# build-only run can never cancel a release or wait behind one, and two +# releases on one ref queue one after the other. Same expression as +# SHOULD_PUBLISH below; the workflow `env` is not readable here. concurrency: - group: ci-cloud-image-${{ github.ref }} - cancel-in-progress: true + group: ci-cloud-image-${{ ((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os || inputs.publish_control_plane) && 'publish' || 'build' }}-${{ inputs.ref || github.ref }} + cancel-in-progress: ${{ !((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os || inputs.publish_control_plane) }} + +# Single source of truth for "should this run publish", one per line, computed +# once for the whole workflow and read by every job. SHOULD_PUBLISH is "either +# line", for the asserts that must hold whatever is published. +# +# OS: a real `v*` tag push always publishes the disk image (that's the +# point of tagging an OS release); otherwise `publish` (dispatch) or +# `publish_os` (dispatch or call). On a `push` event no input is defined, which is why +# the tag-push half of the OR is needed. +# Control plane: only `publish` (dispatch) or `publish_control_plane` +# (dispatch or call). A `v*` tag push is an OS release and never +# pushes the control-plane images. +# Each expression ends in `&& 'true' || 'false'` so the env value is always +# the string true or false, never an empty string from a null input. +# +# `startsWith(github.ref, 'refs/tags/')` is LOAD-BEARING, not +# belt-and-braces. In a reusable workflow the `github` context is the +# CALLER's, not this workflow's: when release.yml (triggered by +# `push: branches: [main]`) calls us, github.event_name here is `push` and +# github.ref is `refs/heads/main`. Keyed on event_name alone this would +# collapse to "any call from a push-triggered workflow publishes", +# ignoring `publish` entirely — a future caller passing publish=false +# would publish anyway. Keyed on the ref, the only path that publishes +# without an explicit input is a genuine tag push. +# +# MOOSE_BUILD_CACHE comes from the same expression as SHOULD_PUBLISH, inverted. +# A run that publishes nothing builds the hosted Caddy image with the GitHub +# Actions layer cache (`gha`); the brain and UI always build plain. A run where either line +# publishes gets `none`: it reads no cache and builds what ships fresh, so no +# layer another run wrote can reach a release. The Makefile reads it. +env: + SHOULD_PUBLISH_OS: ${{ ((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os) && 'true' || 'false' }} + SHOULD_PUBLISH_CP: ${{ (inputs.publish || inputs.publish_control_plane) && 'true' || 'false' }} + SHOULD_PUBLISH: ${{ ((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os || inputs.publish_control_plane) && 'true' || 'false' }} + MOOSE_BUILD_CACHE: ${{ ((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os || inputs.publish_control_plane) && 'none' || 'gha' }} + # The RAUC keyring the images bake as /etc/rauc/keyring.pem (#562, + # dev/cloud/rauc.sh): the release root (dev/release/rauc/release-ca.pem) when + # this run publishes the OS line, a throwaway root on every other run. Same + # expression as SHOULD_PUBLISH_OS. + MOOSE_RAUC_KEYRING: ${{ ((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish || inputs.publish_os) && 'release' || 'throwaway' }} + # Which brain and UI both images bake (#566, dev/cloud/stage-control-plane.sh, + # BUILD.md # Versioning). The rule, in order: + # 1. The control plane publishes (SHOULD_PUBLISH_CP): `local`. This run + # cuts that release, its images are not on ghcr yet, and the publish job + # pushes the very images the boots ran. A merge that bumps both files + # lands here too, so its disk image bakes the control plane released + # beside it. + # 2. Only the OS publishes (SHOULD_PUBLISH_OS): `released`, the last + # released control plane from ghcr by digest, so the image never bakes + # unreleased brain or UI code under the last released number. + # 3. Nothing publishes: the dispatch input `control_plane`, else `local`, so + # a PR or a lock bump boots the brain of its own commit. + MOOSE_CONTROL_PLANE_SOURCE: ${{ (inputs.publish || inputs.publish_control_plane) && 'local' || (((github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish_os) && 'released' || (inputs.control_plane == 'released' && 'released' || 'local')) }} jobs: - cloud-image: - name: Build + seeded-boot test + publish hosted cloud image + build: + name: Build the hosted cloud image and the boot-proof image # Skip draft PRs. This job builds a whole OS image, so every push to a draft - # would spend ~12-15 min of runner on work nobody is reading yet. Marking the + # would spend runner time on work nobody is reading yet. Marking the # PR ready fires `ready_for_review` and the run starts then. Only affects # pull_request runs — a tag push, a dispatch or the release call has no draft - # field and passes the first half of the condition. + # field and passes the first half of the condition. The boot and publish + # jobs need this one, so they skip with it. if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }} - # Single source of truth for "should this run publish", computed once and - # read by the release-asset step below. - # A real tag push always publishes (that's the point of tagging a release); - # a workflow_dispatch OR workflow_call run publishes iff its `publish` - # input says so. On a `push` event `inputs.publish` is undefined, which is - # why the tag-push half of the OR is needed — `inputs.publish` alone would - # evaluate falsy and silently skip publishing on every tagged release. - # `workflow_call` and `workflow_dispatch` both populate the SAME `inputs` - # context under the same input name (`publish`), so `release.yml` calling - # this workflow with `publish: true` flows through the `inputs.publish` - # half identically to a human running `workflow_dispatch publish=true`. - # - # `startsWith(github.ref, 'refs/tags/')` is LOAD-BEARING, not - # belt-and-braces. In a reusable workflow the `github` context is the - # CALLER's, not this workflow's: when release.yml (triggered by - # `push: branches: [main]`) calls us, github.event_name here is `push` and - # github.ref is `refs/heads/main`. Keyed on event_name alone this would - # collapse to "any call from a push-triggered workflow publishes", - # ignoring `publish` entirely — a future caller passing publish=false - # would publish anyway. Keyed on the ref, the only path that publishes - # without an explicit input is a genuine tag push. - env: - SHOULD_PUBLISH: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) || inputs.publish }} + # This job runs mkosi as root with a lot of third-party code, and it + # publishes nothing, so it gets read access only. The write grants are on + # the publish job alone. + permissions: + contents: read + outputs: + matrix: ${{ steps.boots.outputs.matrix }} + should_publish: ${{ env.SHOULD_PUBLISH }} + should_publish_os: ${{ env.SHOULD_PUBLISH_OS }} + # The sha256 of the bundle this job built (#562). The `sign` job signs + # only a bundle with this digest. + bundle_sha256: ${{ steps.bundle.outputs.sha256 }} # Pin 24.04 explicitly: this is the exact environment that reproduces the # mkosi-sandbox EPERM (#189), so the fix has to be proven here, not on a # floating ubuntu-latest. runs-on: ubuntu-24.04 - # Two mkosi builds (production image + QEMU test image), the QEMU boots (eight - # on a push/dispatch, just `update` on a PR; see MOOSE_CLOUD_BOOTS below), and - # the compress-and-attach publish run ~10 min in practice; the 120 cap is a - # generous ceiling so a one-off slow runner is never killed mid-build. - timeout-minutes: 120 + # Two mkosi builds (production image + QEMU test image) and the uploads; + # a full build is about 8 min, and the cap is a generous ceiling so a + # one-off slow runner is never killed mid-build. + timeout-minutes: 60 steps: # persist-credentials: false matters more now that a PR can start this job. # The job holds `packages: write` (the ghcr push below needs it), and by @@ -201,6 +350,19 @@ jobs: - uses: actions/checkout@v5 with: persist-credentials: false + ref: ${{ inputs.ref || '' }} + + - name: Assert a run that builds another ref publishes nothing + if: ${{ inputs.ref != '' && inputs.ref != null }} + env: + REF_INPUT: ${{ inputs.ref }} + run: | + set -euo pipefail + if [ "${SHOULD_PUBLISH}" = "true" ]; then + echo "::error::ref='${REF_INPUT}' builds a ref other than the caller's, so this run must not publish." + exit 1 + fi + echo "building ref '${REF_INPUT}', SHOULD_PUBLISH=${SHOULD_PUBLISH}: nothing will be published" # A PR run must never publish. That already falls out of SHOULD_PUBLISH # (a pull_request event has no tag ref and no `publish` input, so both @@ -235,10 +397,39 @@ jobs: fi echo "boots='${BOOTS_INPUT}', SHOULD_PUBLISH=${SHOULD_PUBLISH}: nothing will be published" + # The control-plane source is decided by the publish switches when a + # line publishes (MOOSE_CONTROL_PLANE_SOURCE above). A dispatch that asks + # for `released` while it publishes the control plane would get `local` + # instead, so refuse it rather than quietly bake something else. + - name: Assert a control-plane release does not ask for the released control plane + if: ${{ inputs.control_plane == 'released' && env.SHOULD_PUBLISH_CP == 'true' }} + run: | + echo "::error::control_plane=released with a control-plane publish: a control-plane release bakes and pushes the images it builds. Drop one of the two." + exit 1 + + # Say what this run will publish, per line, before the build. It costs + # nothing and makes a wrong switch visible in the log at once. + - name: What this run publishes + run: | + echo "OS (disk image on the v$(tr -d '[:space:]' < VERSION) Release): ${SHOULD_PUBLISH_OS}" + echo "Control plane (ghcr brain + ui as v$(tr -d '[:space:]' < CONTROL_PLANE_VERSION)): ${SHOULD_PUBLISH_CP}" + echo "Brain and UI baked into the images: ${MOOSE_CONTROL_PLANE_SOURCE} (released = v$(tr -d '[:space:]' < CONTROL_PLANE_VERSION) from ghcr by digest)" + echo "Image build layer cache: ${MOOSE_BUILD_CACHE} (none on any run that publishes)" + echo "RAUC keyring in the image: ${MOOSE_RAUC_KEYRING} (release only when the OS line publishes)" + + # An OS release bakes the release root CA into the image and has its + # bundle signed by the release signer (#562, docs/dev/rauc-signing.md). + # Without the committed root there is nothing to bake, so fail before the + # ~10 min build. The signer secrets are checked in the `sign` job, the + # only job that may read them. + - name: Assert an OS release has the release root CA + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + run: ./dev/release/require-release-ca.sh + # Fail FAST on a mistagged release, before the ~10min build below runs. - # VERSION (repo root) is the single source of truth for the monorepo's - # one version (BUILD.md # Versioning, DECISIONS.md 2026-07-16); a `v*` - # tag push is only a real release if it matches. workflow_dispatch has no + # VERSION (repo root) is the source of truth for the moose (OS) release + # (BUILD.md # Versioning, DECISIONS.md 2026-10-01); a `v*` tag push is + # only a real OS release if it matches. workflow_dispatch has no # tag (github.ref_name is the branch/ref it ran against), so this only # runs — and only can fail — on a genuine `push: tags: v*` trigger. # @@ -265,41 +456,44 @@ jobs: fi echo "tag ${{ github.ref_name }} matches VERSION" - # mkosi 26 builds rootless: it unshares a user namespace and drops - # capabilities inside it. Ubuntu 24.04 ships - # kernel.apparmor_restrict_unprivileged_userns=1, which hands an unconfined - # process a userns with no CAP_SETPCAP, so mkosi's PR_CAPBSET_DROP returns - # EPERM and the build dies at sandbox bring-up (#189). Relaxing the knob is - # safe on an ephemeral runner. - - name: Relax AppArmor unprivileged-userns restriction - run: | - knob=/proc/sys/kernel/apparmor_restrict_unprivileged_userns - echo "before: $(cat "$knob" 2>/dev/null || echo 'n/a')" - sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 - echo "after: $(cat "$knob")" - - # Host tooling mkosi needs to bootstrap a Debian tools tree and build - # rootless (uidmap/mmdebstrap/keyring; dosfstools/mtools for the ESP; - # squashfs-tools/e2fsprogs for the root; systemd-container for the nspawn - # build sandbox; libpam0g-dev for the slim host-agent-real CGO build). - - name: Install mkosi and image-build tooling + # The QEMU packages, downloaded ONCE here, on a fresh runner before + # anything else is installed, so the set is exactly what a fresh boot + # runner lacks. Each boot job installs them from the artifact instead of + # the Ubuntu mirror: with ten boot jobs at once, one slow mirror would + # otherwise hold up the whole run (a boot job once spent 6 min fetching + # these 46 MB at 135 kB/s). + - name: Download the QEMU packages for the boot jobs run: | + set -euo pipefail + debs="${RUNNER_TEMP}/qemu-debs" + mkdir -p "${debs}/partial" sudo apt-get update - sudo apt-get install -y \ - uidmap mmdebstrap debian-archive-keyring \ - dosfstools mtools squashfs-tools e2fsprogs systemd-container \ - libpam0g-dev - git clone --depth=1 --branch v26 \ - https://github.com/systemd/mkosi.git /tmp/mkosi-src - sudo ln -sf /tmp/mkosi-src/bin/mkosi /usr/local/bin/mkosi + sudo apt-get install -y --download-only -o Dir::Cache::Archives="${debs}" qemu-system-x86 qemu-utils ovmf + sudo rm -rf "${debs}/partial" "${debs}/lock" + sudo chmod -R a+r "${debs}" + ls "${debs}" | wc -l + - name: Upload the QEMU packages + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-qemu-debs + path: ${{ runner.temp }}/qemu-debs/*.deb + compression-level: 0 + retention-days: 3 + if-no-files-found: error + + # The userns knob (#189), the image-build tools and mkosi v26. Shared with + # the OS lock bump (os-lock-bump.yml), so both build the same way. + - uses: ./.github/actions/setup-mkosi # QEMU + UEFI firmware for the seeded-boot test lane (run-cloud-tests.sh). # GitHub-hosted Linux runners expose /dev/kvm; the lane uses it when present # and falls back to TCG otherwise (slower, still correct). - name: Install QEMU + OVMF for the seeded-boot test run: | - sudo apt-get install -y qemu-system-x86 qemu-utils ovmf - echo "kvm: $(ls -l /dev/kvm 2>/dev/null || echo 'none — test lane will use TCG')" + sudo DEBIAN_FRONTEND=noninteractive dpkg -i "${RUNNER_TEMP}"/qemu-debs/*.deb + echo "kvm: $(ls -l /dev/kvm 2>/dev/null || echo 'none: test lane will use TCG')" - name: Versions run: | @@ -323,24 +517,23 @@ jobs: echo "go-bin=${go_bin}" >> "$GITHUB_OUTPUT" echo "go: ${go_bin} ($("$go_bin" version))" - # 1. Build the production image that will be published. A fresh checkout has - # no .dev/ cache, so make rebuilds the control-plane images (brain/ui) - # from source — this is what removes the stale-cached-brain failure mode. - - name: Build the hosted cloud image (+ lean check) - run: make build-cloud-image - - # 2. SEEDED-BOOT GATE. Boot the image under QEMU with a current-schema seed + # The boots are chosen BEFORE the build, so a typo in the `boots` input + # fails at once, not after the ~10 min build. + # + # SEEDED-BOOT GATE. Boot the image under QEMU with a current-schema seed # and assert the brain ingests it (cloud-assertions.sh 'seeded' boot). A # brain that rejects the live seed fails here -> nothing is published. The # test image reuses the control-plane bundle built in step 1, so it # exercises the same brain that ships. - # Scope to unseeded + seeded + bios + access + update + ssh + remap, each - # guarding a regression that must never reach publish: + # Scope to unseeded + seeded + access + update + ssh + remap, each + # guarding a regression that must never reach publish. Since #561 every + # one of them runs twice, under UEFI and again under legacy BIOS, each + # firmware in its own boot job here (MOOSE_CLOUD_FIRMWARES, the lane's default + # "uefi bios"). A box landed on a Hetzner CX (Intel) type boots under + # legacy BIOS (#277), and the A/B slot switch must work the same there. + # Every boot also checks the A/B layout (cloud-assertions.sh 1b). # - unseeded + seeded: the regression this gate was built for — the built # image's brain accepting the current seed schema. - # - bios (#277): re-boots the same image under legacy BIOS (SeaBIOS). A box - # landed on a Hetzner CX (Intel) type is unbootable otherwise, and the - # OVMF-only lane cannot catch it. # - access (#308): the ONLY real-Caddy proof of the hosted per-app # forward-auth gate + whole-Cookie-header strip (#305/#306) — a gate that is # ON BY DEFAULT for every hosted app (docs/specs/DECISIONS.md 2026-07-08 @@ -384,8 +577,8 @@ jobs: # ON A PULL REQUEST the list narrows to `update` alone (#389). That boot is # standalone — its own fresh overlay and its own box-id — so it needs none # of the others, and the ones it drops each guard a publish-time regression - # that a PR cannot ship. One boot keeps the PR cost close to the build cost - # (the build dominates), while still failing a PR that breaks the updater. + # that a PR cannot ship. One boot (two jobs, one per firmware) keeps the PR + # cost close to the build cost, while still failing a PR that breaks the updater. # A PR that wants the whole list can still get it: `gh workflow run # "CI / Cloud image" --ref -f publish=false`. A manual run can # also pick its own list with the `boots` input, for example @@ -397,12 +590,503 @@ jobs: # under `sudo -E`, and sudo's secure_path resets PATH, so `go` on the runner's # PATH is invisible to the script; pass it explicitly (run-cloud-tests.sh # honours ${GO:-}, and -E preserves it). - - name: Boot test (${{ github.event_name == 'pull_request' && 'control-plane update only' || inputs.boots || 'seed schema + BIOS/UEFI + access modes + control-plane update + SSH + userns tiers' }}) + # Which boots run. A PR boots only `update` (the updater guard, #389), + # unless it touches the OS package lock: a lock bump changes every package + # the boots stand on, so it runs the full gate list (#560). Any other run + # takes the `boots` input or the full list. The PR's file list comes from + # the compare API, so the checkout can stay shallow. + - name: Choose the boots + id: boots + env: + GH_TOKEN: ${{ github.token }} + EVENT: ${{ github.event_name }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + BOOTS_INPUT: ${{ inputs.boots }} + run: | + set -euo pipefail + full="unseeded seeded access update ssh remap os-update os-revert" + if [ "$EVENT" = "pull_request" ]; then + # The compare API needs only `contents: read`, which this job has. The + # PR files API would need `pull-requests: read`, and asking for it + # here would break release.yml's call, which does not grant it. + files="$(gh api "repos/${GITHUB_REPOSITORY}/compare/${BASE_SHA}...${HEAD_SHA}" --jq '.files[].filename')" + if printf '%s\n' "$files" | grep -q '^dev/os-lock/'; then + list="$full"; why="this PR changes the OS package lock" + else + list="update"; why="control-plane update only" + fi + else + list="${BOOTS_INPUT:-$full}"; why="full gate list or the boots input" + fi + echo "boots: ${list} (${why})" + + # One boot job per group and firmware. unseeded, seeded and frozen + # share one disk overlay (frozen reboots the disk seeded provisioned), + # so they stay one group; every other boot takes its own overlay and + # gets its own job. The names are checked here, before any boot job + # starts; run-cloud-tests.sh checks them again. + # set -f: the list is split on spaces, never expanded as a glob. + set -f + chain=""; groups=() + for b in $list; do + case "$b" in + unseeded|seeded|frozen) case " $chain " in *" $b "*) ;; *) chain="${chain:+$chain }$b" ;; esac ;; + access|update|ssh|remap|os-update|os-revert) case " ${groups[*]-} " in *" $b "*) ;; *) groups+=("$b") ;; esac ;; + bios) echo "::error::'bios' is no longer a boot: every boot runs under both firmwares"; exit 1 ;; + *) echo "::error::unknown boot '$b' (known: unseeded seeded frozen access update ssh remap os-update os-revert)"; exit 1 ;; + esac + done + # A run that publishes the OS bakes only the release root into the + # boot-proof image, so the throwaway-signed test bundle cannot install + # (#563). os-update then runs its refusal half only, and os-revert, + # which needs an install, is left out. + if [ "$SHOULD_PUBLISH_OS" = true ]; then + kept=(); for g in "${groups[@]}"; do [ "$g" = os-revert ] || kept+=("$g"); done; groups=("${kept[@]}") + echo "an OS release run: os-update runs its refusal half only, os-revert is left out" + fi + # The OS update boots need the test bundle; the build makes it only + # when one of them runs (about 1.5 min). + os_test=false + case " ${groups[*]} " in *" os-update "*|*" os-revert "*) os_test=true ;; esac + echo "os_test=${os_test}" >> "$GITHUB_OUTPUT" + if [ -n "$chain" ]; then groups=("$chain" "${groups[@]}"); fi + [ "${#groups[@]}" -gt 0 ] || { echo "::error::the boot list '${list}' names no boot"; exit 1; } + matrix="$(printf '%s\n' "${groups[@]}" | jq -Rnc '[inputs] as $g | {include: [$g[] as $b | ("uefi", "bios") as $f | {boots: $b, firmware: $f}]}')" + echo "matrix: ${matrix}" + echo "matrix=${matrix}" >> "$GITHUB_OUTPUT" + + # Layer cache for the hosted Caddy image build, on a run that publishes + # nothing (MOOSE_BUILD_CACHE, workflow env above). The Makefile's Caddy + # build reads and writes the GitHub Actions cache through + # BuildKit, which needs the runtime token and cache URLs a `run` step + # does not see; the first step exposes them. The second makes the + # docker-container builder the Makefile names, under `sudo -E` like the + # build itself, so both find the same builder config. A cache is scoped + # to a branch: a run reads its own branch's entries and those of `dev`. + # Pinned by commit: this third-party action runs in a job that builds + # what CI boots. + - name: Expose the Actions cache to the image builds + if: ${{ env.MOOSE_BUILD_CACHE == 'gha' }} + uses: crazy-max/ghaction-github-runtime@04d248b84655b509d8c44dc1d6f990c879747487 # v4.0.0 + - name: Create the builder for the layer cache + if: ${{ env.MOOSE_BUILD_CACHE == 'gha' }} + run: sudo -E docker buildx create --name moose-cache --driver docker-container --bootstrap + + # 1. Build the production image that will be published. A fresh checkout has + # no .dev/ cache, so make rebuilds the control-plane images (brain/ui) + # from source — this is what removes the stale-cached-brain failure mode. + - name: Build the hosted cloud image (+ lean check + OS package lock check) + env: + GO: ${{ steps.go.outputs.go-bin }} + run: | + set -euo pipefail + make build-cloud-image + # What the image that ships baked (#566). The boot-proof build below + # must bake the same pair; the step after it checks. + cp .dev/control-plane/control-plane.env "${RUNNER_TEMP}/control-plane.env" + { + echo "### Control plane baked into the image (${MOOSE_CONTROL_PLANE_SOURCE})" + echo "" + echo '```' + grep '^MOOSE_BAKED_' "${RUNNER_TEMP}/control-plane.env" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + # The OS update bundle (#562, BUILD.md # 1b # The bundle): slot A of the + # image just built, as a verity RAUC bundle signed with the throwaway key, + # checked against the keyring and config read back out of the slot, with + # a wrong-key check and a rehearsal of the sign job's re-signing. It + # writes the bundle size to the job summary. About 1 min. + - name: Build the OS update bundle (+ keyring and signature checks) + id: bundle + env: + GO: ${{ steps.go.outputs.go-bin }} + run: | + set -euo pipefail + sudo -E ./dev/cloud/build-bundle.sh .dev/cloud/moose-cloud.raw .dev/cloud/bundle + sum="$(sha256sum .dev/cloud/bundle/moose-cloud.raucb | cut -d' ' -f1)" + echo "bundle sha256: ${sum}" + echo "sha256=${sum}" >> "$GITHUB_OUTPUT" + + # The package list this build resolved to (#560). When the lock check + # fails, this is the file to compare with dev/os-lock/cloud-packages.lock, + # or to commit when the change was on purpose. Uploaded on failure too, + # which is when it matters. + - name: Upload the resolved package list + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-packages-lock + path: | + .dev/cloud/cloud-packages.lock + .dev/cloud/lean-check.txt + if-no-files-found: ignore + + + # 1b. The boot-proof image: the production image plus the test harness + # (dev/cloud/test/). It reuses the control-plane bundle built in step 1, + # so the boots exercise the same brain that ships. It also reuses the + # tools tree step 1 just made from the same settings + # (MOOSE_MKOSI_TOOLS_TREE), which saves about 40 s. Then it converts the + # image to qcow2 once, for every boot job to download: the qcow2 skips + # the empty space, so it is about half the size of the raw. + - name: Build the boot-proof image + env: + GO: ${{ steps.go.outputs.go-bin }} + MOOSE_MKOSI_TOOLS_TREE: ${{ github.workspace }}/dev/cloud/mkosi.tools + run: | + set -euo pipefail + sudo -E ./dev/cloud/test/bootstrap.sh + # The boots must run the brain and UI the shipped image baked (#566): + # the same record, so the same image IDs (cloud-assertions.sh 5a then + # checks the running containers against it). + if ! cmp "${RUNNER_TEMP}/control-plane.env" dev/cloud/mkosi.extra.wiring/usr/lib/moose/control-plane.env; then + echo "::error::the boot-proof image baked another control plane than the image that ships" + diff "${RUNNER_TEMP}/control-plane.env" dev/cloud/mkosi.extra.wiring/usr/lib/moose/control-plane.env || true + exit 1 + fi + echo "boot-proof image bakes the same control plane as the image that ships" + qemu-img convert -f raw -O qcow2 .dev/cloud-boot/moose-cloud.raw .dev/cloud-boot/moose-cloud-boot.qcow2 + ls -l .dev/cloud-boot/moose-cloud-boot.qcow2 + + # The test-only OS bundle the os-update and os-revert boots install + # (#563): the boot-proof slot repacked fast (gzip level 1) with a + # host-agent one patch release up, signed with the throwaway key. Only + # when one of those boots runs. + - name: Build the OS test bundle + if: ${{ steps.boots.outputs.os_test == 'true' }} + env: + GO: ${{ steps.go.outputs.go-bin }} + run: sudo -E ./dev/cloud/test/build-os-test-bundle.sh .dev/cloud-boot/moose-cloud.raw .dev/cloud-boot/os-test + - name: Upload the OS test bundle + if: ${{ steps.boots.outputs.os_test == 'true' }} + uses: actions/upload-artifact@v4 + with: + overwrite: true + name: cloud-image-os-test-bundle + path: .dev/cloud-boot/os-test + compression-level: 0 + retention-days: 3 + if-no-files-found: error + + # The squashfs inside is already xz, so zipping it again only costs time. + # Three days is enough to re-run a failed boot or publish job. + - name: Upload the boot-proof image + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-boot-qcow2 + path: .dev/cloud-boot/moose-cloud-boot.qcow2 + compression-level: 0 + retention-days: 3 + if-no-files-found: error + + # What the publish job publishes, uploaded only when that line publishes: + # the production raw this job built, and the brain and UI tarballs that + # step 1 baked into both images. The publish job loads those tarballs, so + # it pushes the very images the boots ran, not a rebuild. The raw is + # mostly empty space, so the fastest zip level still shrinks it a lot. + - name: Upload the production image for the publish job + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-os-raw + path: .dev/cloud/moose-cloud.raw + compression-level: 1 + retention-days: 3 + if-no-files-found: error + # The bundle and the slot's own RAUC config and keyring, for the `sign` + # job to re-sign and check. Already squashfs-xz inside, so no zip. + - name: Upload the OS update bundle for the sign job + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-os-bundle + path: | + .dev/cloud/bundle/moose-cloud.raucb + .dev/cloud/bundle/image-etc/system.conf + .dev/cloud/bundle/image-etc/keyring.pem + compression-level: 0 + retention-days: 3 + if-no-files-found: error + # A control-plane release always bakes `local` (MOOSE_CONTROL_PLANE_SOURCE), + # so these are the images this commit built and the boots ran. Checked, + # not trusted: pushing a pulled release under a new number would publish + # old bytes as a new release. + - name: Make the control-plane tarballs readable for the upload + if: ${{ env.SHOULD_PUBLISH_CP == 'true' }} + run: | + set -euo pipefail + grep -qx 'MOOSE_BAKED_CP_SOURCE=local' .dev/control-plane/control-plane.env \ + || { echo "::error::a control-plane release must bake the images it builds, not ${MOOSE_CONTROL_PLANE_SOURCE}"; exit 1; } + sudo chmod a+r .dev/control-plane/moose-brain.tar .dev/control-plane/moose-ui.tar .dev/control-plane/control-plane.env + - name: Upload the control-plane images for the publish job + if: ${{ env.SHOULD_PUBLISH_CP == 'true' }} + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-control-plane + path: | + .dev/control-plane/moose-brain.tar + .dev/control-plane/moose-ui.tar + .dev/control-plane/control-plane.env + compression-level: 1 + retention-days: 3 + if-no-files-found: error + + # Every boot group under each firmware, all at once, on its own runner. The + # longest boot (`update`, about 2.7 min) sets the wall time, so this is the + # split that gives the shortest run: one job per firmware would take about + # 18 min, one job per boot running both firmwares about 15, this about 12. It + # costs each job about a minute of setup and one image download, which is + # free on a public repo. fail-fast is off, so a red run shows every failing + # boot at once instead of the first one. + boot: + name: Boot ${{ matrix.boots }} (${{ matrix.firmware }}) + needs: build + # The boots publish nothing and need no write access. + permissions: + contents: read + runs-on: ubuntu-24.04 + # The update and remap boots allow themselves up to 25 min of verdict + # wait each (run-cloud-tests.sh VERDICT_TIMEOUT); remap is two boots. + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.build.outputs.matrix) }} + steps: + # Same reasoning as the build job: no git credentials on disk. + - uses: actions/checkout@v5 + with: + persist-credentials: false + ref: ${{ inputs.ref || '' }} + + # QEMU + UEFI firmware for the boot lane (run-cloud-tests.sh), from the + # packages the build job downloaded. dpkg, not apt: given local files + # whose versions are also in the mirror, apt downloads them again. If + # the runner image moved since the build job ran and the set no longer + # fits, dpkg fails and the step falls back to the mirror. Each + # matrix job is its own ubuntu-24.04 runner, which exposes /dev/kvm the + # way the single job did; the lane uses KVM when present and falls back + # to TCG otherwise (slower, still correct). The line below shows which. + - name: Download the QEMU packages + uses: actions/download-artifact@v4 + with: + name: cloud-image-qemu-debs + path: ${{ runner.temp }}/qemu-debs + - name: Install QEMU + OVMF + run: | + if ! sudo DEBIAN_FRONTEND=noninteractive dpkg -i "${RUNNER_TEMP}"/qemu-debs/*.deb; then + echo "::warning::the QEMU packages from the build job did not install; falling back to the mirror" + sudo apt-get update + sudo apt-get install -y qemu-system-x86 qemu-utils ovmf + fi + echo "kvm: $(ls -l /dev/kvm 2>/dev/null || echo 'none: test lane will use TCG')" + + # Absolute path to the runner's go, for the host-side assertion mint + # (see GO in the gate notes above). Resolved as a step output so a runner + # image that moves go doesn't silently fall back to the script's guesses. + - name: Resolve go (for the assertion mint) + id: go + run: | + go_bin="$(command -v go)" + echo "go-bin=${go_bin}" >> "$GITHUB_OUTPUT" + echo "go: ${go_bin} ($("$go_bin" version))" + + - name: Download the boot-proof image + uses: actions/download-artifact@v4 + with: + name: cloud-image-boot-qcow2 + path: .dev/cloud-boot + - name: Download the OS test bundle + if: ${{ matrix.boots == 'os-update' || matrix.boots == 'os-revert' }} + uses: actions/download-artifact@v4 + with: + name: cloud-image-os-test-bundle + path: .dev/cloud-boot/os-test + + # MOOSE_CLOUD_QCOW2 makes the lane boot the downloaded image and build + # nothing. The boot list reaches the script through an env var, never + # the shell text, so the `boots` input cannot inject a command. + - name: Boot test (${{ matrix.boots }}, ${{ matrix.firmware }}) env: - MOOSE_CLOUD_BOOTS: ${{ github.event_name == 'pull_request' && 'update' || inputs.boots || 'unseeded seeded bios access update ssh remap' }} + MOOSE_CLOUD_QCOW2: ${{ github.workspace }}/.dev/cloud-boot/moose-cloud-boot.qcow2 + MOOSE_CLOUD_BOOTS: ${{ matrix.boots }} + MOOSE_CLOUD_FIRMWARES: ${{ matrix.firmware }} + MOOSE_CLOUD_OS_BUNDLE_DIR: ${{ github.workspace }}/.dev/cloud-boot/os-test + MOOSE_CLOUD_OS_REFUSE_ONLY: ${{ env.SHOULD_PUBLISH_OS }} GO: ${{ steps.go.outputs.go-bin }} run: make test-cloud-qemu + # Re-sign the OS update bundle with the release signer (#562, DECISIONS.md + # 2026-10-02 key custody). Only when the OS line publishes, and only after + # the build and every boot job passed. This job does nothing else: it is the + # only job that enters the `os-release` environment, whose secrets + # RAUC_SIGNING_CERT and RAUC_SIGNING_KEY only `main` and `v*` tags may use, so + # a dispatch from any other branch with publish_os set fails here and never + # reads the key. It writes nothing anywhere but its own artifact: the + # publish job attaches what it signed. + sign: + name: Sign the OS update bundle + needs: [build, boot] + if: ${{ needs.boot.result == 'success' && needs.build.outputs.should_publish_os == 'true' }} + runs-on: ubuntu-24.04 + timeout-minutes: 20 + environment: os-release + permissions: + contents: read + outputs: + signed_sha256: ${{ steps.sign.outputs.sha256 }} + steps: + - uses: actions/checkout@v5 + with: + persist-credentials: false + ref: ${{ inputs.ref || '' }} + + - name: Download the OS update bundle + uses: actions/download-artifact@v4 + with: + name: cloud-image-os-bundle + path: .dev/cloud/bundle + + # The signature vouches for what the build job produced, so sign only + # the bundle whose digest the build job reported. The build job itself + # could still produce a bad bundle; the box's check of the bundle digest + # its update target names (#563) is the control that covers that. + - name: Check the bundle is the one the build job made + env: + WANT: ${{ needs.build.outputs.bundle_sha256 }} + run: | + set -euo pipefail + got="$(sha256sum .dev/cloud/bundle/moose-cloud.raucb | cut -d' ' -f1)" + if [ -z "$WANT" ] || [ "$got" != "$WANT" ]; then + echo "::error::the downloaded bundle has sha256 ${got}, but the build job reported '${WANT}'; refusing to sign" + exit 1 + fi + echo "bundle sha256 ${got} matches the build job" + + # dev/release/sign-bundle.sh refuses when a secret is empty, when the + # image keyring is not the committed release root, or when that keyring + # accepts the throwaway-signed bundle; it writes the bundle only when the + # result verifies against the image's own config and keyring. + - name: Sign it with the release signer + id: sign + env: + RAUC_SIGNING_CERT: ${{ secrets.RAUC_SIGNING_CERT }} + RAUC_SIGNING_KEY: ${{ secrets.RAUC_SIGNING_KEY }} + run: | + set -euo pipefail + ./dev/release/sign-bundle.sh .dev/cloud/bundle/moose-cloud.raucb .dev/cloud/bundle/image-etc .dev/cloud/bundle/moose-signed.raucb + sum="$(sha256sum .dev/cloud/bundle/moose-signed.raucb | cut -d' ' -f1)" + echo "signed bundle sha256: ${sum}" + echo "sha256=${sum}" >> "$GITHUB_OUTPUT" + + - name: Upload the signed bundle for the publish job + uses: actions/upload-artifact@v4 + with: + # A re-run of this job uploads again; replace, do not fail with 409. + overwrite: true + name: cloud-image-os-bundle-signed + path: .dev/cloud/bundle/moose-signed.raucb + compression-level: 0 + retention-days: 3 + if-no-files-found: error + + # Only when a line publishes, and only after the build and every boot job + # passed. `needs` waits for all of them. The `if` says it in so many words: + # the boot matrix must have succeeded as a whole (one red, skipped or + # cancelled boot job makes it fail), and a line must publish. A red boot + # skips this job, so nothing is published from an image that did not prove + # itself. + # + # When the OS line publishes it also waits for `sign` and needs it green. + # `!cancelled()` lets this job run when `sign` was skipped (a + # control-plane-only publish), which `needs` alone would not. + publish: + name: Publish the hosted cloud image and the control plane + needs: [build, boot, sign] + if: ${{ !cancelled() && needs.boot.result == 'success' && needs.build.outputs.should_publish == 'true' && (needs.sign.result == 'success' || (needs.sign.result == 'skipped' && needs.build.outputs.should_publish_os != 'true')) }} + runs-on: ubuntu-24.04 + timeout-minutes: 30 + # The only job with write access: contents for the Release asset, packages + # for the ghcr push. + permissions: + contents: write + packages: write + steps: + - uses: actions/checkout@v5 + with: + persist-credentials: false + ref: ${{ inputs.ref || '' }} + + - name: Download the production image + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + uses: actions/download-artifact@v4 + with: + name: cloud-image-os-raw + path: .dev/cloud + + # The bundle the `sign` job signed, checked against the digest it + # reported. + - name: Download the signed OS update bundle + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + uses: actions/download-artifact@v4 + with: + name: cloud-image-os-bundle-signed + path: .dev/cloud/bundle + - name: Check the signed bundle is the one the sign job made + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} + env: + WANT: ${{ needs.sign.outputs.signed_sha256 }} + run: | + set -euo pipefail + got="$(sha256sum .dev/cloud/bundle/moose-signed.raucb | cut -d' ' -f1)" + if [ -z "$WANT" ] || [ "$got" != "$WANT" ]; then + echo "::error::the signed bundle has sha256 ${got}, but the sign job reported '${WANT}'; refusing to attach it" + exit 1 + fi + echo "signed bundle sha256 ${got} matches the sign job" + + # The brain and UI tarballs the build job baked into both images. Loading + # them gives back moose-brain:dev and moose-ui:dev with the same image IDs + # the boots ran, so the push below publishes the proven images. + - name: Download the control-plane images the boots ran + if: ${{ env.SHOULD_PUBLISH_CP == 'true' }} + uses: actions/download-artifact@v4 + with: + name: cloud-image-control-plane + path: .dev/control-plane + - name: Load them into Docker + if: ${{ env.SHOULD_PUBLISH_CP == 'true' }} + run: | + set -euo pipefail + docker load -i .dev/control-plane/moose-brain.tar + docker load -i .dev/control-plane/moose-ui.tar + docker image inspect --format '{{.RepoTags}} {{.Id}}' moose-brain:dev moose-ui:dev + # The loaded images must be the ones the boots ran (#566): the image + # IDs the build recorded, which every boot checked the running + # containers against (cloud-assertions.sh step 5a). + . .dev/control-plane/control-plane.env + [ "${MOOSE_BAKED_CP_SOURCE}" = local ] || { echo "::error::the bundle is '${MOOSE_BAKED_CP_SOURCE}', not a build of this commit"; exit 1; } + for pair in "moose-brain:${MOOSE_BAKED_BRAIN_ID}" "moose-ui:${MOOSE_BAKED_UI_ID}"; do + name="${pair%%:*}"; want="${pair#*:}" + # The config digest of what Docker now holds, read the same way the + # build read it (bundle-record.sh), so the image store of this + # runner does not matter. + got="$(docker save "${name}:dev" | tar -xOf - manifest.json | python3 -c 'import json,re,sys; c=json.load(sys.stdin)[0]["Config"]; print("sha256:"+re.search(r"[0-9a-f]{64}",c).group(0))')" + [ "$got" = "$want" ] || { echo "::error::${name}:dev loaded as ${got}, but the boots ran ${want}"; exit 1; } + echo "${name}:dev is ${want}, the image the boots ran" + done + # ---- attach the image to the tagged GitHub Release (only when requested) ---- # Every release should carry its own built artifact, not just source — @@ -412,22 +1096,22 @@ jobs: # a publish emits: the `.raw` built in step 1 lives under `.dev/cloud/`; # we xz it here and attach it plus a checksum to the Release. # - # Naming follows BUILD.md # 6's existing per-release artifact scheme - # (`moose-vX.Y.Z-amd64.`, one `vX.Y.Z` from the repo `VERSION` - # file) — `.raw.xz` names the actual shipped format (raw disk image, + # Naming follows BUILD.md # 6's per-release artifact scheme + # (`moose-vX.Y.Z-amd64.`, the OS release `vX.Y.Z` from the repo + # `VERSION` file) — `.raw.xz` names the actual shipped format (raw disk image, # xz-compressed) rather than the `.qcow2` that section's cloud-image # bullet names, since this lane's build produces `.raw` (mkosi # `Format=disk`), not a qcow2 conversion. # # Tag resolution: `v$(cat VERSION)` matches the tag on both paths that - # reach here with SHOULD_PUBLISH=true — a human tag push (asserted + # reach here with SHOULD_PUBLISH_OS=true — a human tag push (asserted # equal to VERSION by the earlier gate step) and the workflow_call from # the release automation (which tags from VERSION by construction # before calling this workflow). The create-if-missing fallback only # matters for a workflow_dispatch publish run against a commit whose # VERSION hasn't been tagged yet. - name: Attach image + checksum to the GitHub Release - if: ${{ env.SHOULD_PUBLISH == 'true' }} + if: ${{ env.SHOULD_PUBLISH_OS == 'true' }} env: GH_TOKEN: ${{ github.token }} run: | @@ -450,6 +1134,25 @@ jobs: echo "asset: $asset ($(du -h "$asset" | cut -f1))" cat "$checksum" + # The OS update bundle (#562), signed by the release signer in the + # `sign` job, beside the image it was cut from. + bundle="moose-${tag}-amd64.raucb" + bundle_sum="${bundle}.sha256" + cp -- .dev/cloud/bundle/moose-signed.raucb "$bundle" + sha256sum -- "$bundle" > "$bundle_sum" + cat "$bundle_sum" + a_bytes="$(stat -c %s "$asset")"; b_bytes="$(stat -c %s "$bundle")" + { + echo "### OS release files (${tag})" + echo "" + echo "| File | Bytes | Share of the release download |" + echo "|---|---|---|" + awk -v a="$a_bytes" -v b="$b_bytes" -v an="$asset" -v bn="$bundle" 'BEGIN { + printf "| `%s` | %d | %.1f%% |\n", an, a, a * 100 / (a + b) + printf "| `%s` | %d | %.1f%% |\n", bn, b, b * 100 / (a + b) + printf "| both | %d | 100%% |\n", a + b }' + } >> "$GITHUB_STEP_SUMMARY" + api="https://api.github.com/repos/${GITHUB_REPOSITORY}" gh_curl() { curl -sS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" "$@"; } @@ -479,11 +1182,15 @@ jobs: *) echo "::error::release lookup for ${tag} returned HTTP ${status}; aborting rather than treating it as absent"; exit 1 ;; esac - # --clobber: a legitimate re-run against this same commit's tag (e.g. - # retrying a failed publish) replaces the asset instead of erroring on - # "already exists". - gh release upload "$tag" "$asset" "$checksum" --clobber - echo "attached $asset + $checksum to release $tag" + # The image, the bundle and their checksums are ONE set (#562), and + # a published asset is never overwritten. The script detects and + # refuses a mixed set: all present is left, none is uploaded in one + # call, and any other mix (an upload that stopped + # halfway, or a Release cut before the bundle existed) refuses with + # the `gh release delete-asset` lines a person runs before + # re-running. The rules and why live in the script, which + # dev/release/publish_test.go tests with a stub gh. + ./dev/release/attach-image.sh "$tag" "$asset" "$checksum" "$bundle" "$bundle_sum" # ---- publish the control-plane images to ghcr (only when requested) ---- @@ -494,14 +1201,15 @@ jobs: # reprovisioning from a fresh image. # # Why this lane and not a separate, faster workflow: the images pushed - # here are the SAME local images `make build-cloud-image` already built - # and baked into the disk image that the seeded-boot proof just booted. + # here are the SAME images `make build-cloud-image` built in the build + # job and baked into the disk image the boot jobs just booted (loaded + # above from the tarballs that were baked). # Re-building them in a standalone job would publish artifacts that are # merely equivalent, not identical, and would let a red boot proof # coexist with a green publish. Gating on SHOULD_PUBLISH *after* the boot # steps means we can only ever publish a control plane that was proven to - # boot. The cost — publish waits on a ~10 min build plus three QEMU boots - # — is accepted deliberately. + # boot. The cost, publish waiting on the build and every boot job, is + # accepted deliberately. # # ORDERING: this step runs LAST, after the Release-asset attach above. # Every step in this job is a hard sequential gate (nothing here sets @@ -515,6 +1223,13 @@ jobs: # red and is retried on its own, while the release still carries its # image. # + # VERSIONING (BUILD.md # Versioning, DECISIONS.md 2026-10-01): the images + # are the control-plane line. Their number is CONTROL_PLANE_VERSION, and + # the image tag is `v`: the shape every release before the + # split used, and the one the private control plane resolves digests by. + # The git tag and GitHub Release for the same release are named + # `control-plane-v`, which is what the guard below checks. + # # Images are PUBLIC by design (BUILD.md # 6): the brain and UI are built # from this public repo, and every secret a box holds is per-box and # seeded at provision time, never baked into an image. A private registry @@ -523,48 +1238,123 @@ jobs: # them to public is a one-time manual change in the package settings and # cannot be done from here. - name: Publish control-plane images to ghcr - if: ${{ env.SHOULD_PUBLISH == 'true' }} + if: ${{ env.SHOULD_PUBLISH_CP == 'true' }} env: GH_TOKEN: ${{ github.token }} run: | set -euo pipefail - moose_version="$(cat VERSION)" - tag="v${moose_version}" + cp_version="$(tr -d '[:space:]' < CONTROL_PLANE_VERSION)" + git_tag="control-plane-v${cp_version}" + tag="v${cp_version}" registry="ghcr.io/${GITHUB_REPOSITORY_OWNER,,}" - # Same guard as the Release-asset step below, and duplicated rather - # than shared on purpose: SHOULD_PUBLISH is also true for a manual - # workflow_dispatch, which can run against ANY commit while VERSION - # still names an already-shipped release. Pushing then would overwrite - # a released version's images with ones built from another source. + # Same guard as the Release-asset step above, and duplicated rather + # than shared on purpose: SHOULD_PUBLISH_CP is also true for a manual + # workflow_dispatch, which can run against ANY commit while + # CONTROL_PLANE_VERSION still names an already-shipped release. + # Pushing then would overwrite a released version's images with ones + # built from another source. The guard reads the control-plane GIT + # tag, since the image tag carries no prefix. api="https://api.github.com/repos/${GITHUB_REPOSITORY}" tag_commit="$(curl -sS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ - "${api}/commits/${tag}" | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("sha","") if isinstance(d,dict) else "")')" + "${api}/commits/${git_tag}" | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("sha","") if isinstance(d,dict) else "")')" if [ -n "$tag_commit" ] && [ "$tag_commit" != "$GITHUB_SHA" ]; then - echo "::notice::tag ${tag} points at ${tag_commit}, not this build (${GITHUB_SHA}); skipping ghcr push to avoid clobbering released images" + echo "::notice::tag ${git_tag} points at ${tag_commit}, not this build (${GITHUB_SHA}); skipping ghcr push to avoid clobbering released images" exit 0 fi + owner="${GITHUB_REPOSITORY_OWNER,,}" + + # With no git tag at all (a dispatch run ahead of any release), the + # image tag may still be published already, by a release before the + # split or by an earlier dispatch. Check the brain AND the UI before + # pushing either, and refuse if either is there or the registry gives + # no clear answer. release.yml makes the same check before it tags + # (dev/release/decide.sh). + if [ -z "$tag_commit" ]; then + rc=0 + ./dev/release/ghcr-tag-exists.sh "${owner}/brain" "${owner}/ui" "$tag" || rc=$? + case "$rc" in + 0) echo "::error::no git tag ${git_tag}, but ${tag} is already published for the brain or the UI. Refusing to overwrite it; bump CONTROL_PLANE_VERSION."; exit 1 ;; + 1) ;; + *) echo "::error::could not tell whether ${tag} is published for the brain and the UI (exit ${rc}); refusing to push"; exit 1 ;; + esac + fi + echo "${GH_TOKEN}" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin # `latest` advances on the stable channel per BUILD.md # 6. Boxes pull # by digest, not by either tag; the tags are for humans and for the # offline `docker load` path. echo "### Control-plane images" >> "$GITHUB_STEP_SUMMARY" + # Push only an image whose tag is not published yet. With the git tag + # at this commit, an image already there was pushed by an earlier + # attempt at this same release (a resume or a retry), and a rebuild + # is not byte-identical, so pushing again would change what the tag + # names after something may already have resolved it. To replace one + # on purpose, delete the tag on ghcr first. An unclear answer + # refuses, as everywhere else. + # + # `latest` is REPAIRED for such an image, not left alone: an attempt + # that pushed `vX.Y.Z` and died before `latest` would otherwise leave + # `latest` behind for good, and the brain and the UI could end with + # different `latest`. dev/release/ghcr-retag.sh points `latest` at + # the manifest `vX.Y.Z` already names, byte for byte, so nothing is + # rebuilt or re-pushed and both tags carry the same digest. Only when + # this run publishes this version: the git tag points at this commit + # (with no git tag, an existing image tag already refused above), + # and only when this version is the newest (move_latest, above). + # `latest` only ever moves FORWARD. Re-running an older tagged + # control-plane release passes the commit guard above, and a dispatch + # can publish any version, so neither may point `latest` back at + # older images. `latest` moves (by a push or by the repair below) + # only when this version is the newest `control-plane-v*` tag by + # semver. A tag list that cannot be read leaves `latest` alone, + # with a warning, and still publishes the version tag: a stale + # `latest` is a smaller harm than a missing release. + move_latest=false + rc=0 + ./dev/release/is-newest.sh control-plane-v "$cp_version" || rc=$? + case "$rc" in + 0) move_latest=true ;; + 1) echo "::notice::control-plane ${cp_version} is not the newest control-plane release; leaving latest where it is" ;; + *) echo "::warning::could not tell whether control-plane ${cp_version} is the newest release (exit ${rc}); leaving latest where it is" ;; + esac + echo "move latest: ${move_latest}" + for spec in "moose-brain:dev=brain" "moose-ui:dev=ui"; do local_img="${spec%%=*}" remote="${registry}/${spec##*=}" + rc=0 + ./dev/release/ghcr-tag-exists.sh "${owner}/${spec##*=}" "$tag" || rc=$? + case "$rc" in + 0) echo "::notice::${remote}:${tag} is already published; leaving it (delete the tag on ghcr first to replace it)" + if [ "$tag_commit" = "$GITHUB_SHA" ] && [ "$move_latest" = "true" ]; then + digest="$(GHCR_USER="${GITHUB_ACTOR}" GHCR_TOKEN="${GH_TOKEN}" \ + ./dev/release/ghcr-retag.sh "${owner}/${spec##*=}" "$tag" latest)" + echo "pointed ${remote}:latest at ${remote}:${tag} (${digest})" + echo "- \`${remote}:${tag}\`: already published, \`latest\` re-pointed to it (\`${digest}\`)" >> "$GITHUB_STEP_SUMMARY" + else + echo "- \`${remote}:${tag}\`: already published, left as it is" >> "$GITHUB_STEP_SUMMARY" + fi + continue ;; + 1) ;; + *) echo "::error::could not tell whether ${remote}:${tag} is published (exit ${rc}); refusing to push"; exit 1 ;; + esac + docker tag "$local_img" "${remote}:${tag}" - docker tag "$local_img" "${remote}:latest" # Read the digest out of the push output rather than from # `docker inspect .RepoDigests`: after pushing two tags of the same # image, RepoDigests holds an entry per tag and index 0 is not # guaranteed to be the one we want. The push line is authoritative. push_log="$(docker push "${remote}:${tag}" | tee /dev/stderr)" - docker push "${remote}:latest" + if [ "$move_latest" = "true" ]; then + docker tag "$local_img" "${remote}:latest" + docker push "${remote}:latest" + fi digest="$(printf '%s\n' "$push_log" | grep -oE 'sha256:[0-9a-f]{64}' | tail -n1)" [ -n "$digest" ] || { echo "::error::could not resolve pushed digest for ${remote}:${tag}"; exit 1; } diff --git a/.github/workflows/ci-go.yml b/.github/workflows/ci-go.yml index 5f15fabc..8059d360 100644 --- a/.github/workflows/ci-go.yml +++ b/.github/workflows/ci-go.yml @@ -15,7 +15,8 @@ name: CI / Go on: pull_request: - branches: [dev, main] + # hotfix/** gates an OS lock bump into a patch-release branch (#560). + branches: [dev, main, "hotfix/**"] paths: - "**.go" - "go.mod" @@ -23,6 +24,9 @@ on: - "Makefile" - "VERSION" - "api/**" + # The lock files have no .go in them, but dev/os-lock/oslock's test checks + # they fit together, so a lock bump PR must run it (#560). + - "dev/os-lock/**" - ".github/workflows/ci-go.yml" # Cancel superseded runs on the same ref (e.g. force-pushes to a PR branch). diff --git a/.github/workflows/os-lock-bump.yml b/.github/workflows/os-lock-bump.yml new file mode 100644 index 00000000..d9c8508e --- /dev/null +++ b/.github/workflows/os-lock-bump.yml @@ -0,0 +1,475 @@ +# Move the OS package lock forward (#560, BUILD.md # 1b # The OS package lock). +# +# Every day this takes the newest snapshot.debian.org timestamp and the newest +# Docker versions within each pin's current major, builds the lean hosted image +# with them, and compares the package list it resolved with the committed +# dev/os-lock/cloud-packages.lock. When no package changed, nothing happens. +# When one did, it commits the three lock files on one bot branch and names +# every changed package and version in the commit, the PR title and the PR body. +# +# The bot branch keeps human work. A person may push to it (for example the +# fix to dev/cloud/expected-packages.txt that a new package needs) or open its +# PR by hand. So the branch is never deleted and never force-pushed: +# +# - It does not exist (the first bump, or its PR merged and the branch was +# deleted): it is made from the base branch. +# - It exists: the base is merged into it, the lock is resolved on the +# result, and a new commit is added only when the lock files changed. A +# merge conflict stops the run with a summary; nothing is discarded. +# +# Two jobs, so the token never meets the build. `resolve` builds the image as +# root with a lot of third-party code and holds no secret. `handover` builds +# nothing: it takes only the three lock files and the PR title and body from +# `resolve` (an artifact), commits them and pushes. +# +# How the change gets reviewed is chosen in `handover`, by one secret: +# +# - OS_LOCK_BOT_TOKEN is set and works: the branch is pushed with it, and ONE +# PR into the base branch is opened, or updated in place when it is open. +# A PR opened with a person's token starts CI like any human PR, so +# `CI / Cloud image` runs the lock check and the full boot list on it (a PR +# that touches dev/os-lock/ gets the full list, into dev or hotfix/**). +# - It is not set, it is set but rejected (expired, revoked, no access), or +# it pushed the branch but cannot open or edit PRs (no Pull requests: +# write; then the branch is not pushed again): +# the org does not let GitHub Actions open PRs (#484), and a push made with +# GITHUB_TOKEN starts no other workflow. So the branch is pushed with +# GITHUB_TOKEN, this workflow runs the boot proofs itself by calling +# ci-cloud-image.yml on that branch, and it writes the diff and an "open a +# PR" link to the job summary and to one tracking issue. A rejected token is +# said loudly in both places ("renew it"). A missing secret is never a +# failure. +# +# The token (read in exactly one place: the `BOT_TOKEN` env of the "Push and +# hand over" step in the `handover` job) is a fine-grained personal access +# token for onmoose/os only, with Contents: read and write and Pull requests: +# read and write. It belongs to a person and expires (at most one year), so it +# must be renewed. A GitHub App token is the cleaner later replacement: mint it +# in a step before that one (actions/create-github-app-token) and pass it as +# BOT_TOKEN. Nothing else in this file changes. See docs/dev/contributing.md +# # OS package lock bumps. +# +# Hotfix use (an OS patch release cut from main, docs/dev/contributing.md +# # OS patch releases from a lock bump): dispatch with base=hotfix/X.Y.Z to +# resolve the lock for that branch's package list. Only `dev` and +# `hotfix/X.Y.Z` are accepted. +name: OS lock bump + +on: + schedule: + # Daily. snapshot.debian.org takes a snapshot about every six hours, and a + # Debian security fix should reach `dev` within a day. + - cron: "17 5 * * *" + workflow_dispatch: + inputs: + base: + description: "Branch to bump the lock on: dev, or hotfix/X.Y.Z for an OS patch release" + type: string + default: dev + +permissions: + contents: read + +# One bump per base at a time; a second one waits rather than racing the push. +concurrency: + group: os-lock-bump-${{ inputs.base || 'dev' }} + cancel-in-progress: false + +jobs: + resolve: + name: Resolve the newest lock + runs-on: ubuntu-24.04 + timeout-minutes: 60 + # Read only, and no secret anywhere in this job. + permissions: + contents: read + env: + BASE: ${{ inputs.base || 'dev' }} + outputs: + changed: ${{ steps.compare.outputs.changed }} + branch: ${{ steps.base.outputs.branch }} + base_sha: ${{ steps.start.outputs.base_sha }} + branch_sha: ${{ steps.start.outputs.branch_sha }} + steps: + - name: Check the base branch + id: base + run: | + set -euo pipefail + case "$BASE" in + dev) branch="bot/os-lock" ;; + hotfix/[0-9]*.[0-9]*.[0-9]*) branch="bot/os-lock-${BASE//\//-}" ;; + *) echo "::error::base '${BASE}' is not dev or hotfix/X.Y.Z"; exit 1 ;; + esac + echo "branch=${branch}" >> "$GITHUB_OUTPUT" + echo "base ${BASE}, bot branch ${branch}" + + - uses: actions/checkout@v5 + with: + ref: ${{ env.BASE }} + fetch-depth: 0 + persist-credentials: false + + # Resolve on what the PR will hold: the bot branch with the base merged + # in, so a person's edits on the branch (a package list change, say) are + # part of the build. The repo is public, so reading the branch needs no + # credential. + - name: Start from the bot branch when it exists + id: start + env: + BRANCH: ${{ steps.base.outputs.branch }} + run: | + set -euo pipefail + base_sha="$(git rev-parse HEAD)" + branch_sha="" + if git ls-remote --exit-code --heads origin "refs/heads/${BRANCH}" >/dev/null; then + git fetch -q origin "+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}" + branch_sha="$(git rev-parse "origin/${BRANCH}")" + git config user.name "moose os-lock bot" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git checkout -q -B "$BRANCH" "$branch_sha" + if ! git merge -q --no-edit "$base_sha"; then + git merge --abort || true + { + echo "### OS lock bump stopped: merge conflict" + echo + echo "\`${BASE}\` does not merge cleanly into \`${BRANCH}\`. Nothing was changed or discarded." + echo "Merge \`${BASE}\` into \`${BRANCH}\` by hand (or delete the branch if it holds no work you need), then re-run this workflow." + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::${BASE} does not merge cleanly into ${BRANCH}; merge it by hand, then re-run" + exit 1 + fi + echo "bot branch ${BRANCH} exists at ${branch_sha}; merged ${BASE} (${base_sha}) into it" + else + echo "bot branch ${BRANCH} does not exist; starting from ${BASE} (${base_sha})" + fi + echo "base_sha=${base_sha}" >> "$GITHUB_OUTPUT" + echo "branch_sha=${branch_sha}" >> "$GITHUB_OUTPUT" + + # The "did anything change" baseline is what the branch holds now. + # The PR text compares with the base, because that is what the PR + # changes as a whole. + cp dev/os-lock/cloud-packages.lock "$RUNNER_TEMP/branch-packages.lock" + git show "${base_sha}:dev/os-lock/cloud-packages.lock" > "$RUNNER_TEMP/old-packages.lock" + git show "${base_sha}:dev/os-lock/third-party.lock" > "$RUNNER_TEMP/old-third-party.lock" + git show "${base_sha}:dev/os-lock/debian-snapshot" | tr -d '[:space:]' > "$RUNNER_TEMP/old-snapshot" + + - uses: ./.github/actions/setup-mkosi + + # The newest snapshot of the main archive. mkosi's own latest-snapshot verb + # reads the same field; debian-security at that timestamp resolves to its + # newest snapshot at or before it. + - name: Move the snapshot and the Docker pins forward + id: move + run: | + set -euo pipefail + new_ts="$(curl -fsSL --retry 5 https://snapshot.debian.org/mr/timestamp/ \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["result"]["debian"][-1])')" + printf '%s\n' "$new_ts" | grep -Eq '^[0-9]{8}T[0-9]{6}Z$' || { echo "::error::snapshot.debian.org gave '$new_ts'"; exit 1; } + + curl -fsSL --retry 5 https://download.docker.com/linux/debian/dists/trixie/stable/binary-amd64/Packages.gz \ + | gunzip > "$RUNNER_TEMP/docker-Packages" + go run ./dev/os-lock/oslock newest-pins dev/os-lock/third-party.lock "$RUNNER_TEMP/docker-Packages" \ + > "$RUNNER_TEMP/third-party.lock" + + printf '%s\n' "$new_ts" > dev/os-lock/debian-snapshot + cp "$RUNNER_TEMP/third-party.lock" dev/os-lock/third-party.lock + + echo "old_ts=$(cat "$RUNNER_TEMP/old-snapshot")" >> "$GITHUB_OUTPUT" + echo "new_ts=${new_ts}" >> "$GITHUB_OUTPUT" + echo "snapshot: $(cat "$RUNNER_TEMP/old-snapshot") -> ${new_ts}" + diff "$RUNNER_TEMP/old-third-party.lock" dev/os-lock/third-party.lock || true + + # Record mode: the build writes its resolved list to + # dev/os-lock/cloud-packages.lock instead of failing on it, and a failed + # lean check is kept for the PR body instead of stopping the bump. + - name: Build the lean image and record its package list + env: + MOOSE_OS_LOCK_RECORD: "1" + run: make build-cloud-image + + - name: Compare with the bot branch + id: compare + env: + NEW_TS: ${{ steps.move.outputs.new_ts }} + OLD_TS: ${{ steps.move.outputs.old_ts }} + run: | + set -euo pipefail + if diff -q <(grep -v '^#' "$RUNNER_TEMP/branch-packages.lock") <(grep -v '^#' dev/os-lock/cloud-packages.lock) >/dev/null; then + echo "changed=false" >> "$GITHUB_OUTPUT" + echo "No package changed at snapshot ${NEW_TS}; nothing to do." | tee -a "$GITHUB_STEP_SUMMARY" + exit 0 + fi + echo "changed=true" >> "$GITHUB_OUTPUT" + + # trixie-security at the new snapshot, to mark which changes are + # security fixes (the 7-day release rule). + curl -fsSL --retry 5 "https://snapshot.debian.org/archive/debian-security/${NEW_TS}/dists/trixie-security/main/binary-amd64/Packages.xz" \ + | xz -d > "$RUNNER_TEMP/security-Packages" + + mkdir -p "$RUNNER_TEMP/bump" + oslock() { go run ./dev/os-lock/oslock "$@"; } + oslock title -security "$RUNNER_TEMP/security-Packages" "$RUNNER_TEMP/old-packages.lock" dev/os-lock/cloud-packages.lock \ + > "$RUNNER_TEMP/bump/title.txt" + { + echo "Moves the OS package lock forward (\`BUILD.md\` # 1b # The OS package lock). Made by \`.github/workflows/os-lock-bump.yml\`." + echo + echo "- Debian snapshot: \`${OLD_TS}\` to \`${NEW_TS}\`" + if ! diff -q "$RUNNER_TEMP/old-third-party.lock" dev/os-lock/third-party.lock >/dev/null; then + echo "- Docker pins: moved to the newest version within each major (\`dev/os-lock/third-party.lock\`)" + fi + echo + oslock diff -security "$RUNNER_TEMP/security-Packages" "$RUNNER_TEMP/old-packages.lock" dev/os-lock/cloud-packages.lock + if [ -f .dev/cloud/lean-check.txt ] && grep -q "LEAN CHECK FAILED" .dev/cloud/lean-check.txt; then + echo + echo "**The lean check failed for this bump.** Update \`dev/cloud/expected-packages.txt\` on this branch after review; the next bump keeps that commit:" + echo + echo '```' + cat .dev/cloud/lean-check.txt + echo '```' + fi + echo + echo "Merging this puts the change on \`${BASE}\`, not on any box. It ships with the next OS release (\`docs/dev/contributing.md\` # Release model)." + } > "$RUNNER_TEMP/bump/body.md" + cp dev/os-lock/debian-snapshot dev/os-lock/third-party.lock dev/os-lock/cloud-packages.lock "$RUNNER_TEMP/bump/" + cat "$RUNNER_TEMP/bump/title.txt" + cat "$RUNNER_TEMP/bump/body.md" >> "$GITHUB_STEP_SUMMARY" + + # Only these five files cross into the job that holds the token. + - name: Hand the lock files and PR text to the push job + if: ${{ steps.compare.outputs.changed == 'true' }} + uses: actions/upload-artifact@v4 + with: + name: os-lock-bump + path: | + ${{ runner.temp }}/bump/debian-snapshot + ${{ runner.temp }}/bump/third-party.lock + ${{ runner.temp }}/bump/cloud-packages.lock + ${{ runner.temp }}/bump/title.txt + ${{ runner.temp }}/bump/body.md + if-no-files-found: error + + handover: + name: Push and open the PR (or the tracking issue) + needs: resolve + if: ${{ needs.resolve.outputs.changed == 'true' }} + runs-on: ubuntu-24.04 + timeout-minutes: 15 + permissions: + contents: write # push the bot branch with GITHUB_TOKEN on the no-token path + issues: write # the tracking issue on that same path + pull-requests: read # find a PR a person opened by hand + env: + BASE: ${{ inputs.base || 'dev' }} + BRANCH: ${{ needs.resolve.outputs.branch }} + BASE_SHA: ${{ needs.resolve.outputs.base_sha }} + BRANCH_SHA: ${{ needs.resolve.outputs.branch_sha }} + outputs: + pushed: ${{ steps.handover.outputs.pushed }} + mode: ${{ steps.handover.outputs.mode }} + issue: ${{ steps.handover.outputs.issue }} + steps: + # This job builds nothing and runs no code from the repo: it only commits + # the five files from `resolve`. persist-credentials stays false, and + # each push names its token explicitly. + - uses: actions/checkout@v5 + with: + ref: ${{ env.BASE_SHA }} + fetch-depth: 0 + persist-credentials: false + + - uses: actions/download-artifact@v4 + with: + name: os-lock-bump + path: ${{ runner.temp }}/bump + + - name: Push and hand over + id: handover + env: + # The ONLY place the bot token is read. Swap in a GitHub App token + # here later; nothing else changes. + BOT_TOKEN: ${{ secrets.OS_LOCK_BOT_TOKEN }} + GH_DEFAULT_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + title="$(cat "$RUNNER_TEMP/bump/title.txt")" + body="$RUNNER_TEMP/bump/body.md" + repo_url="https://github.com/${GITHUB_REPOSITORY}.git" + git config user.name "moose os-lock bot" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + echo "pushed=false" >> "$GITHUB_OUTPUT" + + stop() { + { echo "### OS lock bump stopped"; echo; echo "$1"; } >> "$GITHUB_STEP_SUMMARY" + echo "::error::$1" + exit 1 + } + + # 1. Which token. A set but rejected token falls back, loudly. + mode=notoken + token_note="" + if [ -n "$BOT_TOKEN" ]; then + if GH_TOKEN="$BOT_TOKEN" gh api "repos/${GITHUB_REPOSITORY}" --jq .full_name >/dev/null 2>&1; then + mode=token + else + token_note="**OS_LOCK_BOT_TOKEN is set but rejected (expired, revoked or without access to this repo). Renew it** (\`docs/dev/contributing.md\` # OS package lock bumps). This run fell back to the no-token path." + echo "::warning::OS_LOCK_BOT_TOKEN is set but rejected; renew it. Falling back to the no-token path." + fi + fi + + # 2. The branch: keep it and its human commits. Merge the base in, + # exactly as `resolve` did; make it from the base only when absent. + if git ls-remote --exit-code --heads origin "refs/heads/${BRANCH}" >/dev/null; then + git fetch -q origin "+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}" + git checkout -q -B "$BRANCH" "origin/${BRANCH}" + if ! git merge -q --no-edit "$BASE_SHA"; then + git merge --abort || true + stop "\`${BASE}\` does not merge cleanly into \`${BRANCH}\`. Nothing was changed or discarded. Merge it by hand, then re-run this workflow." + fi + else + [ -z "$BRANCH_SHA" ] || stop "\`${BRANCH}\` existed when the lock was resolved and is gone now. Re-run this workflow." + git checkout -q -B "$BRANCH" "$BASE_SHA" + fi + + # 3. Apply the lock files; add a commit only if they changed. The + # commit message is the PR title and body, so a PR opened by hand + # from the compare link is filled in by GitHub. + cp "$RUNNER_TEMP/bump/debian-snapshot" "$RUNNER_TEMP/bump/third-party.lock" "$RUNNER_TEMP/bump/cloud-packages.lock" dev/os-lock/ + git add dev/os-lock/debian-snapshot dev/os-lock/third-party.lock dev/os-lock/cloud-packages.lock + if git diff --cached --quiet; then + echo "The lock files on ${BRANCH} already match; nothing to push." | tee -a "$GITHUB_STEP_SUMMARY" + exit 0 + fi + { echo "$title"; echo; cat "$body"; } | git commit -q -F - + + # 4. Push. Never a force-push, so a person's commits on the branch + # stay. The bump only ever adds commits on top. + push_with() { + git -c "http.extraheader=AUTHORIZATION: basic $(printf 'x-access-token:%s' "$1" | base64 -w0)" \ + push "$repo_url" "HEAD:refs/heads/${BRANCH}" 2> "$RUNNER_TEMP/push.err" + } + explain_refusal() { + cat "$RUNNER_TEMP/push.err" >&2 + if grep -qi "workflow" "$RUNNER_TEMP/push.err"; then + stop "GitHub refused the push of \`${BRANCH}\` because it brings in a change to a file under \`.github/workflows/\` from \`${BASE}\`, and the token may not write workflows. Give OS_LOCK_BOT_TOKEN Workflows: read and write, or merge \`${BASE}\` into \`${BRANCH}\` by hand, then re-run." + fi + if grep -qiE "non-fast-forward|fetch first|rejected" "$RUNNER_TEMP/push.err"; then + stop "\`${BRANCH}\` moved while this run worked on it. Nothing was overwritten. Re-run this workflow." + fi + } + pushed_with=OS_LOCK_BOT_TOKEN + if [ "$mode" = token ] && ! push_with "$BOT_TOKEN"; then + explain_refusal + token_note="**OS_LOCK_BOT_TOKEN is set but its push was refused (expired, revoked or without Contents: write). Renew it** (\`docs/dev/contributing.md\` # OS package lock bumps). This run fell back to the no-token path." + echo "::warning::the push with OS_LOCK_BOT_TOKEN was refused; renew it. Falling back to the no-token path." + mode=notoken + fi + if [ "$mode" = notoken ]; then + pushed_with=GITHUB_TOKEN + push_with "$GH_DEFAULT_TOKEN" || { explain_refusal; stop "The push of \`${BRANCH}\` with GITHUB_TOKEN failed; see the log."; } + fi + echo "pushed=true" >> "$GITHUB_OUTPUT" + + # 5a. The token path: one PR, opened or updated in place. A token + # that can push but not open or edit PRs (no Pull requests: + # write) gets here too, so a failed PR call falls back to the + # no-token hand-over below. The branch is already pushed, so + # nothing is pushed again; the boot proofs and the tracking + # issue run instead. + pr_with_token() { + local pr url + pr="$(GH_TOKEN="$BOT_TOKEN" gh pr list --repo "$GITHUB_REPOSITORY" --head "$BRANCH" --base "$BASE" --state open --json number --jq '.[0].number // empty')" || return 1 + if [ -n "$pr" ]; then + GH_TOKEN="$BOT_TOKEN" gh pr edit "$pr" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$body" || return 1 + echo "Updated PR #${pr}." >> "$GITHUB_STEP_SUMMARY" + else + url="$(GH_TOKEN="$BOT_TOKEN" gh pr create --repo "$GITHUB_REPOSITORY" --base "$BASE" --head "$BRANCH" --title "$title" --body-file "$body")" || return 1 + echo "Opened ${url}." >> "$GITHUB_STEP_SUMMARY" + fi + } + if [ "$mode" = token ]; then + if pr_with_token; then + echo "mode=token" >> "$GITHUB_OUTPUT" + exit 0 + fi + token_note="**OS_LOCK_BOT_TOKEN pushed the branch but cannot open or edit PRs. Give it Pull requests: read and write** (\`docs/dev/contributing.md\` # OS package lock bumps). This run fell back to the no-token hand-over: the boot proofs run in this workflow, and the PR must be opened by hand." + echo "::warning::OS_LOCK_BOT_TOKEN cannot open or edit PRs; give it Pull requests: read and write. Falling back to the no-token hand-over." + mode=notoken + fi + + # 5b. The no-token hand-over: a summary and one tracking issue, and + # the next job boots the branch. Reached with no usable token, or + # after 5a fell back (then the branch went up with the token). + echo "mode=notoken" >> "$GITHUB_OUTPUT" + export GH_TOKEN="$GH_DEFAULT_TOKEN" + pr="$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$BRANCH" --base "$BASE" --state open --json number --jq '.[0].number // empty')" + compare="https://github.com/${GITHUB_REPOSITORY}/compare/${BASE}...${BRANCH}?expand=1" + if [ -n "$pr" ]; then + where="PR #${pr} is open from \`${BRANCH}\` and now has the new commit. This workflow boots the branch in its next job, so the boot proofs run even when the PR's own CI does not (a push made with GITHUB_TOKEN starts none)." + else + where="No PR is open from \`${BRANCH}\` yet. Open it here: ${compare} (GitHub fills in the title and body from the newest commit)." + fi + { + echo + echo "### Hand-over" + echo + [ -z "$token_note" ] || { echo "$token_note"; echo; } + echo "Pushed \`${BRANCH}\` with ${pushed_with}. ${where}" + } >> "$GITHUB_STEP_SUMMARY" + + issue_title="OS lock bump ready for ${BASE}" + { + [ -z "$token_note" ] || { echo "$token_note"; echo; } + echo "The scheduled OS lock bump found package changes and pushed \`${BRANCH}\`. ${where}" + echo + echo "Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID} (its boot proofs run on the branch too; the result is added below)." + echo + echo "## ${title}" + echo + cat "$body" + } > "$RUNNER_TEMP/issue.md" + issue="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"${issue_title}\"" --json number,title \ + --jq "map(select(.title == \"${issue_title}\")) | .[0].number // empty")" + if [ -n "$issue" ]; then + gh issue edit "$issue" --repo "$GITHUB_REPOSITORY" --body-file "$RUNNER_TEMP/issue.md" + else + url="$(gh issue create --repo "$GITHUB_REPOSITORY" --title "$issue_title" --body-file "$RUNNER_TEMP/issue.md")" + issue="${url##*/}" + fi + echo "issue=${issue}" >> "$GITHUB_OUTPUT" + echo "Tracking issue: #${issue}" >> "$GITHUB_STEP_SUMMARY" + + # Only on the no-token path: nothing else would boot the branch, because a + # push made with GITHUB_TOKEN starts no workflow. On the token path the PR's + # own `CI / Cloud image` run does this, into dev and into hotfix/** alike. + boot-proofs: + name: Boot proofs on the bump branch + needs: [resolve, handover] + if: ${{ needs.handover.outputs.pushed == 'true' && needs.handover.outputs.mode == 'notoken' }} + # The called workflow asks for these two at its top level, and a called + # workflow can only narrow what its caller grants. Nothing is published: + # both publish inputs are false, and a run with `ref` asserts it. + permissions: + contents: write + packages: write + uses: ./.github/workflows/ci-cloud-image.yml + with: + ref: ${{ needs.resolve.outputs.branch }} + publish_os: false + publish_control_plane: false + + report: + name: Add the boot result to the tracking issue + needs: [handover, boot-proofs] + if: ${{ always() && needs.handover.outputs.mode == 'notoken' && needs.handover.outputs.issue != '' }} + runs-on: ubuntu-24.04 + permissions: + issues: write + steps: + - name: Comment + env: + GH_TOKEN: ${{ github.token }} + ISSUE: ${{ needs.handover.outputs.issue }} + RESULT: ${{ needs.boot-proofs.result }} + run: | + gh issue comment "$ISSUE" --repo "$GITHUB_REPOSITORY" --body \ + "Boot proofs on the bump branch: **${RESULT}** (${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})." diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6de1c7fb..2af2c7de 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,54 +1,64 @@ -# Auto-tag `main` when a merge bumps the repo-root `VERSION` file, and — only -# on that bump — cut the release: a GitHub Release whose notes are the release -# PR's own body plus the generated commit list, then the hosted cloud-image -# build+publish. +# Auto-tag `main` when a merge bumps a version file, and, only on that bump, +# cut that line's release: a GitHub Release whose notes are the release PR's own +# body plus the generated commit list, then the hosted cloud-image build, which +# publishes what that line ships. # -# The VERSION bump IS the release trigger. `docs/dev/contributing.md` # -# Release model: the maintainer bumps VERSION in the dev->main PR; everything -# from here down is automatic. A dev->main merge that does NOT bump VERSION -# (docs-only, a hotfix that doesn't ship a new version yet, etc.) is a silent -# no-op — this workflow still runs, sees the tag for the current VERSION -# already exists, and exits clean without tagging, releasing, or building -# anything. That no-op MUST stay green, not red: most merges to main won't -# bump VERSION, and a failing/red run on every ordinary merge trains people to -# ignore this workflow's status. +# TWO version lines, one per update stream (BUILD.md # Versioning, +# DECISIONS.md 2026-10-01), decided independently on every push: # -# Ordering: tag + GitHub Release FIRST, image build+publish SECOND. Rationale +# - `VERSION` -> git tag `vX.Y.Z`, the moose (OS) release. Publishes the disk +# image, attached to the `vX.Y.Z` GitHub Release. +# - `CONTROL_PLANE_VERSION` -> git tag `control-plane-vX.Y.Z`, the control +# plane (moose-brain + moose-ui). Publishes the two ghcr images, tagged +# `vX.Y.Z` (+ `latest`) with the control-plane number: that image tag shape +# is what the private control plane resolves digests by, so only the git tag +# and the GitHub Release carry the `control-plane-` prefix. Its GitHub +# Release is created with `--latest=false`, so GitHub's "Latest release" (and +# its download link) stays on the OS release and its disk image. +# +# A bump of either file IS that line's release trigger (`docs/dev/contributing.md` +# # Release model). A merge that bumps both cuts both, in one cloud-image run. A +# merge that bumps neither is a silent no-op: this workflow still runs, sees both +# tags already exist, and exits clean. That no-op MUST stay green, not red: most +# merges to main won't bump anything, and a red run on every ordinary merge +# trains people to ignore this workflow's status. +# +# The decision per line lives in dev/release/decide.sh, so it can be tested off +# main (dev/release/decide_test.go runs it against throwaway git repos in +# `make check`). It has four outcomes: release; resume, when the tag already +# points at this commit because an earlier run of this workflow for the same +# push tagged it and then failed; no-op; and a hard error for a bump to a version +# tagged at another commit. For the control plane it also refuses a version +# whose ghcr image tag is already published for the brain or the UI, so a +# release never overwrites released images with new bytes under the same +# number. A failed run is recovered by re-running this workflow. +# +# Ordering: tags + GitHub Releases FIRST, image build+publish SECOND. Rationale # (deliberately chosen over the reverse): # - A git tag names a source commit's version; it doesn't assert "and an -# image was published from it." Those are separable facts, and BUILD.md's -# Versioning section already treats the tag as the source-of-truth marker -# for "this commit is vX.Y.Z" independent of any downstream artifact. +# image was published from it." Those are separable facts. # - If the tag/release step ran AFTER a successful image build instead, a -# build failure — or the runner dying mid-build, which can happen on any -# job (the build is ~10min) — would leave NO tag and NO release for a commit whose -# VERSION was already bumped and merged to main. The next push to main -# (if VERSION isn't bumped again) would see the same untagged VERSION and -# retry the SAME multi-hour build every time until it succeeds, or worse, -# a following VERSION bump would race past the stuck one and the earlier -# version would simply never get tagged. +# build failure, or the runner dying mid-build, would leave NO tag and NO +# release for a commit whose version file was already bumped and merged to +# main. The next push to main would retry the SAME build every time until it +# succeeds, or a following bump would race past the stuck one and the +# earlier version would simply never get tagged. # - Tagging first means a failed/slow image build is a visibly separate, -# independently retriable problem (re-run this workflow, or invoke -# `ci-cloud-image.yml` directly via workflow_dispatch against the tag) — -# without it blocking the source-level fact that vX.Y.Z exists and what -# commit it points at. A tag that exists whose image build later fails is -# a strictly easier incident to recover from (retry the build against a -# known-good, already-published tag/release) than a successful image -# published under no tag at all (which commit is it? what do you call it? -# nothing else in the release process can refer to it). +# independently retriable problem (invoke `ci-cloud-image.yml` directly via +# workflow_dispatch against the tagged commit), without it blocking the +# source-level fact that the version exists and what commit it points at. # # THE TRAP this design avoids: a tag pushed with the default GITHUB_TOKEN does -# NOT fire another workflow's `push: tags` trigger — GitHub suppresses events -# from GITHUB_TOKEN-authored pushes to prevent recursive workflow runs. So -# this workflow does NOT rely on `ci-cloud-image.yml`'s `push: tags: v*` -# trigger picking up the tag it pushes (it would silently never build or -# publish). Instead it CALLS `ci-cloud-image.yml` directly as a reusable -# workflow (`workflow_call`) with `publish: true`. `ci-cloud-image.yml`'s -# `push: tags` trigger is kept as-is alongside `workflow_call` — it's the -# manual escape hatch for a human pushing a tag by hand, and because the -# GITHUB_TOKEN tag push here never fires it, the two paths cannot double-build -# the same release. See the header comment on `ci-cloud-image.yml` for the -# full reasoning. +# NOT fire another workflow's `push: tags` trigger. GitHub suppresses events +# from GITHUB_TOKEN-authored pushes to prevent recursive workflow runs. So this +# workflow does NOT rely on `ci-cloud-image.yml`'s `push: tags: v*` trigger +# picking up the tag it pushes (it would silently never build or publish). +# Instead it CALLS `ci-cloud-image.yml` directly as a reusable workflow +# (`workflow_call`) with one publish switch per line. `ci-cloud-image.yml`'s +# `push: tags` trigger is kept alongside `workflow_call` as the manual escape +# hatch for a human pushing an OS tag by hand; because the GITHUB_TOKEN tag push +# here never fires it, the two paths cannot double-build the same release. See +# the header comment on `ci-cloud-image.yml` for the full reasoning. name: Release on: @@ -67,9 +77,9 @@ concurrency: jobs: tag: - name: Tag + GitHub Release (only if VERSION was bumped) + name: Tag + GitHub Release (only for a bumped version file) runs-on: ubuntu-latest - # `contents: write` to push the tag and create the Release, and + # `contents: write` to push the tags and create the Releases, and # `pull-requests: read` to look up the release PR whose body becomes the # Release notes. Set at job level: the workflow-level grant is # `contents: write` only, which leaves every other scope at `none`. @@ -77,134 +87,161 @@ jobs: contents: write pull-requests: read outputs: - created: ${{ steps.decide.outputs.created }} - tag: ${{ steps.decide.outputs.tag }} + os_created: ${{ steps.decide.outputs.os_created }} + os_tag: ${{ steps.decide.outputs.os_tag }} + cp_created: ${{ steps.decide.outputs.cp_created }} + cp_tag: ${{ steps.decide.outputs.cp_tag }} steps: - uses: actions/checkout@v5 with: - # Full history + tags: the decide step below compares VERSION against + # Full history + tags: the decision compares each version file against # the previous main commit and resolves where an existing tag points. # The default depth-1 checkout can do neither. fetch-depth: 0 - # Decide whether this merge is a release. THREE outcomes, not two: + # Decide both lines before tagging either. If one line is a hard error, + # nothing is tagged, so a merge that bumps both never half-releases. + # The control-plane line also checks that its ghcr image tag is not + # already published, for the brain AND the UI (anonymous read; the images + # are public). Either one published refuses, and so does an unclear + # registry answer. # - # - no tag for VERSION -> cut the release. - # - tag exists, VERSION same -> ordinary merge; clean no-op, green. - # - tag exists, VERSION bumped -> hard error. - # - # That third case is what a plain "does the tag exist?" check gets wrong, - # and it is not hypothetical. This workflow tags BEFORE building (see the - # header), so a release whose image build fails leaves the tag behind. If - # the fix is then merged without deleting that tag, a bare existence - # check sees the tag, exits green, and the release silently never - # happens — you merge a release PR, everything is green, and there is no - # tag movement, no Release, and no image. Bumping VERSION to a version - # that is already tagged is always a real release that cannot be cut, so - # say so loudly and say what to do about it. - - name: Decide whether this merge is a release + # A line whose tag already points at this commit is a RESUME + # (`tag_exists=true`): an earlier run of this workflow for this same push + # tagged it and then failed. Re-running finishes the job: no second tag, + # a Release only if it is missing, and the publish steps add only what is + # missing. + - name: Decide which lines this merge releases id: decide + env: + BEFORE: ${{ github.event.before }} run: | set -euo pipefail - want="v$(cat VERSION)" - echo "tag=$want" >> "$GITHUB_OUTPUT" + ./dev/release/decide.sh VERSION v "$BEFORE" | sed 's/^/os_/' > /tmp/os.out + owner="${GITHUB_REPOSITORY_OWNER,,}" + RELEASE_IMAGE_CHECK="./dev/release/ghcr-tag-exists.sh ${owner}/brain ${owner}/ui" \ + ./dev/release/decide.sh CONTROL_PLANE_VERSION control-plane-v "$BEFORE" | sed 's/^/cp_/' > /tmp/cp.out + cat /tmp/os.out /tmp/cp.out | tee -a "$GITHUB_OUTPUT" - if ! git ls-remote --exit-code --tags origin "refs/tags/$want" >/dev/null 2>&1; then - echo "created=true" >> "$GITHUB_OUTPUT" - echo "release: tag $want does not exist yet — cutting a release at ${GITHUB_SHA}" - exit 0 - fi + # An OS release ships a bundle signed by the release signer, which needs + # the offline root CA committed (#562, docs/dev/rauc-signing.md). Without + # it the cloud-image build fails at its first step, so stop HERE, before + # anything is tagged: otherwise a merge that bumps both lines would tag + # and create both Releases and then push no control-plane images either, + # because both lines publish from the one cloud-image run. The OS line is + # never dropped quietly: the merge must wait for the key setup, or bump + # only CONTROL_PLANE_VERSION. + - name: Refuse an OS release without the release root CA + if: steps.decide.outputs.os_created == 'true' + run: ./dev/release/require-release-ca.sh - # The tag exists. Did THIS push change VERSION? github.event.before is - # main's previous head; on the very first push, a force-push, or a - # commit where VERSION didn't yet exist it may be absent/unreadable, - # in which case we fall back to the old skip behaviour but say so. - before="${{ github.event.before }}" - prev="" - if [ -n "$before" ] \ - && [ "$before" != "0000000000000000000000000000000000000000" ] \ - && git cat-file -e "${before}^{commit}" 2>/dev/null; then - prev="$(git show "${before}:VERSION" 2>/dev/null | tr -d '[:space:]')" || prev="" - fi - cur="$(tr -d '[:space:]' < VERSION)" - - if [ -n "$prev" ] && [ "$prev" != "$cur" ]; then - at="$(git rev-parse --short "refs/tags/$want" 2>/dev/null || echo unknown)" - echo "::error::VERSION was bumped ($prev -> $cur) but tag $want already exists (at $at). Refusing to skip a real release silently. Delete the stale tag and its GitHub Release and re-run this workflow, or bump VERSION again." - exit 1 - fi - - if [ -z "$prev" ]; then - echo "::warning::could not read VERSION at ${before:-} to confirm whether this merge bumped it; assuming it did not, since tag $want already exists." - fi - - echo "created=false" >> "$GITHUB_OUTPUT" - echo "skip: tag $want already exists and VERSION was not bumped by this merge — nothing to release" - - # Tag the merge commit and open a GitHub Release. This uses the default - # GITHUB_TOKEN, which is exactly why ci-cloud-image.yml cannot rely on - # its `push: tags` trigger to notice this tag — see the header comment - # above and on ci-cloud-image.yml. + # Tag the merge commit and open a GitHub Release, once per line that this + # merge releases. This uses the default GITHUB_TOKEN, which is exactly why + # ci-cloud-image.yml cannot rely on its `push: tags` trigger to notice the + # tag (see the header comment above and on ci-cloud-image.yml). # # The release PR's body becomes the Release notes, with the generated # commit list appended under it (the API appends when both a body and # generate_release_notes are sent). The commit list alone is a poor - # answer to "should I upgrade?" — it lists 45 subjects and buries the - # three that matter. The PR body is where the person cutting the release - # already wrote that summary, so use it rather than asking them to write - # it twice. + # answer to "should I upgrade?". The PR body is where the person cutting + # the release already wrote that summary, so use it rather than asking + # them to write it twice. A merge that releases both lines uses the same + # body for both. # # It falls back to the generated notes alone when the body is missing or - # too short to be a summary. That matters: a release PR opened with a - # one-line body would otherwise produce notes strictly worse than the - # automatic ones. The threshold is deliberately low — it only screens out - # an empty or throwaway body, not a terse but real summary. - - name: Create tag + GitHub Release - if: steps.decide.outputs.created == 'true' + # too short to be a summary. The threshold is deliberately low: it only + # screens out an empty or throwaway body, not a terse but real summary. + # + # The generated list starts at the previous tag on the SAME line + # (`--notes-start-tag`), so a control-plane Release never lists commits + # since the last OS tag, and the other way round. + - name: Create tags + GitHub Releases + if: steps.decide.outputs.os_created == 'true' || steps.decide.outputs.cp_created == 'true' env: GH_TOKEN: ${{ github.token }} - TAG: ${{ steps.decide.outputs.tag }} + OS_CREATED: ${{ steps.decide.outputs.os_created }} + OS_TAG: ${{ steps.decide.outputs.os_tag }} + OS_PREV: ${{ steps.decide.outputs.os_prev_tag }} + OS_TAGGED: ${{ steps.decide.outputs.os_tag_exists }} + CP_CREATED: ${{ steps.decide.outputs.cp_created }} + CP_TAG: ${{ steps.decide.outputs.cp_tag }} + CP_PREV: ${{ steps.decide.outputs.cp_prev_tag }} + CP_TAGGED: ${{ steps.decide.outputs.cp_tag_exists }} run: | set -euo pipefail - git tag "$TAG" "$GITHUB_SHA" - git push origin "$TAG" # The PR this merge came from, if any. A direct push to main has none, # which is a normal case here, not an error. body="$(gh api "repos/${GITHUB_REPOSITORY}/commits/${GITHUB_SHA}/pulls" \ --jq '.[0].body // ""' 2>/dev/null || echo "")" - if [ "${#body}" -ge 200 ]; then printf '%s\n' "$body" > /tmp/release-notes.md echo "notes: using the release PR body (${#body} chars) with the generated list appended" - gh release create "$TAG" --title "$TAG" --target "$GITHUB_SHA" \ - --notes-file /tmp/release-notes.md --generate-notes else - echo "::warning::release PR body is missing or too short (${#body} chars) — falling back to generated notes only. Write a summary in the release PR body so the Release page says what changed." - gh release create "$TAG" --title "$TAG" --target "$GITHUB_SHA" --generate-notes + echo "::warning::release PR body is missing or too short (${#body} chars), falling back to generated notes only. Write a summary in the release PR body so the Release page says what changed." fi + release() { # TAG PREV TAGGED [extra gh args...] + local tag="$1" prev="$2" tagged="$3"; shift 3 + local args=(--title "$tag" --target "$GITHUB_SHA" --generate-notes "$@") + if [ -n "$prev" ]; then args+=(--notes-start-tag "$prev"); fi + if [ -f /tmp/release-notes.md ]; then args+=(--notes-file /tmp/release-notes.md); fi + if [ "$tagged" = "true" ]; then + echo "resume: tag $tag already points at this commit; not tagging again" + else + git tag "$tag" "$GITHUB_SHA" + git push origin "$tag" + fi + # Create the Release only on a definite "not found". Any other + # failure aborts, rather than be misread as absence. + local status + status="$(gh api -i "repos/${GITHUB_REPOSITORY}/releases/tags/${tag}" 2>/dev/null | head -n1 | awk '{print $2}' || true)" + case "$status" in + 200) echo "resume: Release $tag already exists; leaving it as it is" ;; + 404) gh release create "$tag" "${args[@]}" ;; + *) echo "::error::Release lookup for $tag returned '${status:-no answer}'; aborting rather than treating it as absent"; exit 1 ;; + esac + } + + # The OS release keeps GitHub's automatic "Latest" choice: it is the + # one with the disk image a person downloads. The control-plane + # Release never takes "Latest". + if [ "$OS_CREATED" = "true" ]; then release "$OS_TAG" "$OS_PREV" "$OS_TAGGED"; fi + if [ "$CP_CREATED" = "true" ]; then release "$CP_TAG" "$CP_PREV" "$CP_TAGGED" --latest=false; fi + - name: Step summary env: - TAG: ${{ steps.decide.outputs.tag }} - CREATED: ${{ steps.decide.outputs.created }} + OS_CREATED: ${{ steps.decide.outputs.os_created }} + OS_TAG: ${{ steps.decide.outputs.os_tag }} + CP_CREATED: ${{ steps.decide.outputs.cp_created }} + CP_TAG: ${{ steps.decide.outputs.cp_tag }} + CP_VERSION: ${{ steps.decide.outputs.cp_version }} run: | { - if [ "$CREATED" = "true" ]; then - echo "### Released ${TAG}" + if [ "$OS_CREATED" = "true" ]; then + echo "### Released ${OS_TAG} (OS)" echo "" - echo "VERSION was bumped by this merge. Tag \`${TAG}\` pushed, GitHub Release created, and the hosted cloud-image build+publish was triggered (see the \`cloud-image\` job/workflow for its progress)." + echo "\`VERSION\` was bumped by this merge. Tag pushed, GitHub Release created, and the disk image will be built and attached by the \`cloud-image\` job." else - echo "### Skipped — \`${TAG}\` already exists" + echo "### OS: skipped, \`${OS_TAG}\` already exists" + fi + echo "" + if [ "$CP_CREATED" = "true" ]; then + echo "### Released ${CP_TAG} (control plane)" echo "" - echo "This merge to main did not bump \`VERSION\`. No tag, no release, no image build." + echo "\`CONTROL_PLANE_VERSION\` was bumped by this merge. Tag pushed, GitHub Release created, and the \`cloud-image\` job will push the brain and UI images to ghcr as \`v${CP_VERSION}\` after the boot proof." + else + echo "### Control plane: skipped, \`${CP_TAG}\` already exists" fi } >> "$GITHUB_STEP_SUMMARY" cloud-image: name: Build + publish hosted cloud image needs: tag - if: needs.tag.outputs.created == 'true' + # Either line released. A control-plane-only release still builds the disk + # image and runs the full boot proof, because the images it pushes are the + # ones baked into that image and proven to boot; it only skips the attach. + if: needs.tag.outputs.os_created == 'true' || needs.tag.outputs.cp_created == 'true' # A called reusable workflow can only NARROW the permissions its caller # grants, never widen them. This workflow's top-level `permissions:` is # `contents: write` only, which sets every other scope — including @@ -223,5 +260,6 @@ jobs: packages: write uses: ./.github/workflows/ci-cloud-image.yml with: - publish: true + publish_os: ${{ needs.tag.outputs.os_created == 'true' }} + publish_control_plane: ${{ needs.tag.outputs.cp_created == 'true' }} secrets: inherit diff --git a/CLAUDE.md b/CLAUDE.md index 850f5fd6..faa77f4b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Two phrases that constrain a lot of design: A running moose is five processes/artifacts. Three are Go, one is JavaScript, one is a container we don't write. - **`moose-brain`** (`cmd/brain/`, `internal/`) — the control-plane daemon. One Go binary: owns SQLite state, the REST+SSE API, the app lifecycle, and the Caddy config. Drives Docker via the `docker compose` CLI. -- **`host-agent`** — the privileged side. Two binaries. `cmd/host-agent/` is the **fake** used by the inner dev loop: it speaks the real `BRAIN_HOST_PROTOCOL.md` wire format over a real UNIX socket, but the host ops are stubbed in memory. `cmd/host-agent-real/` is the real one, and most of it works now: PAM verify, user management, `/proc` sampling, disk and RAM, journal streaming, service health, reboot-required, time zone, Avahi discovery, the first-boot brain launch, and the control-plane update path. Not wired yet: LUKS/TPM, apt, and NetworkManager config. It also builds a slim `hosted` variant (`go build -tags hosted`) for the cloud image. See `docs/architecture.md` # Components for the current split. +- **`host-agent`** — the privileged side. Two binaries. `cmd/host-agent/` is the **fake** used by the inner dev loop: it speaks the real `BRAIN_HOST_PROTOCOL.md` wire format over a real UNIX socket, but the host ops are stubbed in memory. `cmd/host-agent-real/` is the real one, and most of it works now: PAM verify, user management, `/proc` sampling, disk and RAM, journal streaming, service health, reboot-required, time zone, Avahi discovery, the first-boot brain launch, and the control-plane update path. Not wired yet: LUKS/TPM, NetworkManager config, and the A/B OS update (#486, designed in `UPDATES.md` # 1 and `BUILD.md` # 1b; there is no apt on the update path). It also builds a slim `hosted` variant (`go build -tags hosted`) for the cloud image. See `docs/architecture.md` # Components for the current split. - **`web-ui`** (`web-ui/`) — Vue 3 + Vite + TanStack Query dashboard. Talks only to the brain. - **Caddy** (`dev/`) — reverse proxy. Terminates `*.local` and routes to app containers + the brain, configured live by the brain via Caddy's admin API. Subdomain routing per app, never path-based (browser same-origin policy is the reason). - **SQLite** — the brain's only persistent store (`internal/store/`). @@ -61,7 +61,7 @@ The inner/outer boundary is also the **cross-platform / Linux-only** boundary. T - `make test-nopam` — full suite minus the PAM package, for when you don't have `libpam0g-dev` (i.e. off Linux). - `make clean` — stop dev Caddy, remove moose containers/networks, wipe `.dev/state`. -**Building the hosted cloud image is a CI job: don't build it locally.** `make build-cloud-image` / `make test-cloud-qemu` need root + `/dev/kvm` + mkosi and take ~10+ min; a local mkosi build is fragile and easy to get wrong (a broken build is what once produced a phantom "`:443` doesn't bind" hunt). Instead trigger the **`CI / Cloud image`** GitHub Action: `gh workflow run "CI / Cloud image" --ref -f publish=false` builds the image and runs the QEMU boot-proof (`unseeded seeded bios access update ssh remap` boots; add `-f boots="remap"` to run only some) **without** publishing anything. Only `publish=true` (the default) publishes anything. Since #352 that means two things: it **attaches the compressed image and its checksum to the tagged GitHub Release**, and it **pushes the brain and UI images to ghcr**. It uploads to no hosting provider and holds no provider login. Publishing is a deliberate act, not a test. See [`docs/dev/hosted-boot-proof.md`](docs/dev/hosted-boot-proof.md) for reading the result and debugging a red boot. +**Building the hosted cloud image is a CI job: don't build it locally.** `make build-cloud-image` / `make test-cloud-qemu` need root + `/dev/kvm` + mkosi and take ~10+ min; a local mkosi build is fragile and easy to get wrong (a broken build is what once produced a phantom "`:443` doesn't bind" hunt). Instead trigger the **`CI / Cloud image`** GitHub Action: `gh workflow run "CI / Cloud image" --ref -f publish=false` builds the image and runs the QEMU boot-proof (`unseeded seeded bios access update ssh remap` boots; add `-f boots="remap"` to run only some) **without** publishing anything. Only `publish=true` (the default) publishes anything. Since #352 that means two things: it **attaches the compressed image and its checksum to the tagged GitHub Release**, and it **pushes the brain and UI images to ghcr**. Since #559 those are two release lines (the OS and the control plane), and dispatch also takes `publish_os` and `publish_control_plane` to publish just one of them, for example `-f publish=false -f publish_control_plane=true` to re-publish only the control plane. A published file or version tag is never overwritten; only `latest` moves, and only forward (`BUILD.md` # 6). It uploads to no hosting provider and holds no provider login. Publishing is a deliberate act, not a test. See [`docs/dev/hosted-boot-proof.md`](docs/dev/hosted-boot-proof.md) for reading the result and debugging a red boot. **Prerequisites for the inner loop:** Docker + `docker compose`, Node 20+, Go 1.23+, host port `:80` free (dev Caddy binds it so `.local` works portless), and `avahi-daemon` running on Linux (so `.local` names resolve under `make dev`). The full Go test suite additionally needs `libpam0g-dev` on Linux; see `docs/dev/running-locally.md`. @@ -92,7 +92,7 @@ Small set of rules. Codified now so we don't have to back them out later. - **Consumer-side interfaces.** Interfaces live in the package that *uses* them, not the package that implements them. `lifecycle.DockerDriver` lives in `internal/lifecycle/`, not in a hypothetical `internal/docker/`. Provider packages export concrete types only. Exception: a single interface shared by three or more consumers can move to the provider, but default to consumer-side until that's true. - **Layer boundaries.** `internal/lifecycle` is the transaction owner; only `cmd/brain` and `internal/api` may import it. `internal/store` is the persistence boundary; only `internal/lifecycle`, `internal/api`, `internal/auth`, `internal/audit`, and `cmd/brain` may import it. Anything else reaching in is breaking the model — push the call through the right seam instead. - **`log/slog` is the only logger.** No `"log"` imports, no `fmt.Println` for diagnostics. Structured fields, not interpolated strings: `slog.Info("app installed", "instance_id", id)`, not `slog.Info(fmt.Sprintf("installed %s", id))`. The default handler is set in `cmd/brain/main.go`; use `slog.Default()` (the package-level functions) — don't thread `*slog.Logger` through constructors. -- **Standard structured fields.** Use these key names so journalctl/jq filters stay stable: `instance_id`, `manifest_id`, `slug`, `service`, `image`, `host`, `upstream`, `step`, `err`, `output`, `user_id`, `username`, `role`, `action`, `actor_user_id`, `target_kind`, `target_id`, `retry_after`, `retry_after_s`, `name`, `uid`, `iface`, `interfaces`, `src`, `dir`, `profile`, `box_id`, `zone`, `exposure`, `trusted_proxies`, `brain`, `ui`, `minimum_host_agent`, `keys`, `state_dir`, `job_id`, `window`, `url`, `from`, `count`, `dropped`, `tier`, `remap_base`, `gid`, `image_user`. `host` is a machine or upstream hostname only — a single network interface name is `iface`, a list of them is `interfaces` (never overload `host` for either). `src` is a source filesystem path (bind-source, folder-source); `dir` is a relative bind dir path. `profile` is the resolved environment profile (`appliance`|`hosted`, `ENVIRONMENT.md`). `box_id` is the hosted box's provisioned identity (`ENVIRONMENT.md` # Provisioning). `zone` is an IANA time-zone name (`TIME.md`, host-agent set-timezone). `exposure` is an app's per-instance access mode (`restricted`|`public`, `ENVIRONMENT.md` #306). `trusted_proxies` is the configured set of proxies whose `X-Forwarded-For` the brain reads when deriving a client IP (`BRAIN_UI_PROTOCOL.md` # Rate limiting & abuse). `brain` and `ui` are the two control-plane versions a release names, and `minimum_host_agent` the host-agent version it requires (`RELEASE_MANIFEST.md`); use them for versions, not for image refs — an image ref is `image`. `keys` is how many signing keys a build accepts, and `state_dir` a state directory path (`src` stays for a source path being read or bound). `job_id` is a host-agent job id (`internal/hostagent/jobs.go`), and `window` the configured update window (`UPDATES.md` # 8.4). `url` is an HTTP endpoint the box reads, and `from` says which of several configured sources a setting came from, one of `answer` (the control plane's update-target answer), `seed`, `env`, `default` (`UPDATES.md` # 8.4); where there are only two sources, the boolean `from_` form is used instead (`from_ledger`). `dropped` lists what the box read but could not use, from catalog data it reads leniently (AI provider entries, manifest `role` and `requires` keys), and `count` is how many there were. `tier` is an instance's user-namespace tier (`default`|`caps`|`image`|`host`, `APP_ISOLATION.md` # User-namespace tiers), and `remap_base` the first host id of the Docker remap range (0 for none). `image_user` is the user an image sets (its `Config.User`, a name or a number, as the image wrote it), which the image tier resolves to ids. `name` is an app instance's display name (it rides alongside `instance_id`, never instead of it), and `uid` a numeric Unix user id — the allocated app-service identity, a resolved home owner, or an image user's id. `gid` is its numeric group id. `retry_after` and `retry_after_s` come from the two throttles that `AUTH.md` # Rate limiting keeps apart on purpose. Both are correct. Do not merge them. `retry_after` is the login backoff's wait, written as a duration string (`internal/api/auth.go`); that path sends no `Retry-After` header, by design. `retry_after_s` is the general request limiter's wait, written as a whole number of seconds; it matches that limiter's `retry_after_s` JSON field and the `Retry-After` header it sets (`internal/api/ratelimit.go`, `BRAIN_UI_PROTOCOL.md` # 429 contract). To find every throttled request you must search for both keys. That is the price of the split, not a bug. Adding a new recurring field? Add it here. +- **Standard structured fields.** Use these key names so journalctl/jq filters stay stable: `instance_id`, `manifest_id`, `slug`, `service`, `image`, `host`, `upstream`, `step`, `err`, `output`, `user_id`, `username`, `role`, `action`, `actor_user_id`, `target_kind`, `target_id`, `retry_after`, `retry_after_s`, `name`, `uid`, `iface`, `interfaces`, `src`, `dir`, `profile`, `box_id`, `zone`, `exposure`, `trusted_proxies`, `brain`, `ui`, `minimum_host_agent`, `keys`, `state_dir`, `job_id`, `window`, `url`, `from`, `count`, `dropped`, `tier`, `remap_base`, `gid`, `image_user`, `os`, `slot`, `digest`. `host` is a machine or upstream hostname only — a single network interface name is `iface`, a list of them is `interfaces` (never overload `host` for either). `src` is a source filesystem path (bind-source, folder-source); `dir` is a relative bind dir path. `profile` is the resolved environment profile (`appliance`|`hosted`, `ENVIRONMENT.md`). `box_id` is the hosted box's provisioned identity (`ENVIRONMENT.md` # Provisioning). `zone` is an IANA time-zone name (`TIME.md`, host-agent set-timezone). `exposure` is an app's per-instance access mode (`restricted`|`public`, `ENVIRONMENT.md` #306). `trusted_proxies` is the configured set of proxies whose `X-Forwarded-For` the brain reads when deriving a client IP (`BRAIN_UI_PROTOCOL.md` # Rate limiting & abuse). `brain` and `ui` are the two control-plane versions a release names, and `minimum_host_agent` the host-agent version it requires (`RELEASE_MANIFEST.md`); use them for versions, not for image refs — an image ref is `image`. `keys` is how many signing keys a build accepts, and `state_dir` a state directory path (`src` stays for a source path being read or bound). `job_id` is a host-agent job id (`internal/hostagent/jobs.go`), and `window` the configured update window (`UPDATES.md` # 8.4). `url` is an HTTP endpoint the box reads, and `from` says which of several configured sources a setting came from, one of `answer` (the control plane's update-target answer), `seed`, `env`, `default` (`UPDATES.md` # 8.4); where there are only two sources, the boolean `from_` form is used instead (`from_ledger`). `dropped` lists what the box read but could not use, from catalog data it reads leniently (AI provider entries, manifest `role` and `requires` keys), and `count` is how many there were. `tier` is an instance's user-namespace tier (`default`|`caps`|`image`|`host`, `APP_ISOLATION.md` # User-namespace tiers), and `remap_base` the first host id of the Docker remap range (0 for none). `image_user` is the user an image sets (its `Config.User`, a name or a number, as the image wrote it), which the image tier resolves to ids. `os` is an OS (moose) release version, as `brain` and `ui` are control-plane versions; `slot` is an A/B OS slot (`A`|`B`, `BUILD.md` # 1b); `digest` is an OS bundle's sha256 (`UPDATES.md` # 1). `name` is an app instance's display name (it rides alongside `instance_id`, never instead of it), and `uid` a numeric Unix user id — the allocated app-service identity, a resolved home owner, or an image user's id. `gid` is its numeric group id. `retry_after` and `retry_after_s` come from the two throttles that `AUTH.md` # Rate limiting keeps apart on purpose. Both are correct. Do not merge them. `retry_after` is the login backoff's wait, written as a duration string (`internal/api/auth.go`); that path sends no `Retry-After` header, by design. `retry_after_s` is the general request limiter's wait, written as a whole number of seconds; it matches that limiter's `retry_after_s` JSON field and the `Retry-After` header it sets (`internal/api/ratelimit.go`, `BRAIN_UI_PROTOCOL.md` # 429 contract). To find every throttled request you must search for both keys. That is the price of the split, not a bug. Adding a new recurring field? Add it here. - **Typed errors at boundaries, not everywhere.** Define a sentinel/typed error only when a *consumer* needs to discriminate (HTTP status, retry decision, UI text). `store.ErrNotFound` exists because the API maps it to 404. Don't pre-declare error types speculatively. - **No premature abstraction.** Don't introduce an interface, factory, or DI container until at least two concrete consumers exist. It bites hardest in Go where every extra interface is import-graph weight. - **`internal/` for everything except `cmd/`.** No `pkg/`. Anything inside `internal/` is private to this module by Go's own rules — no public API surface to maintain. diff --git a/CONTROL_PLANE_VERSION b/CONTROL_PLANE_VERSION new file mode 100644 index 00000000..04a373ef --- /dev/null +++ b/CONTROL_PLANE_VERSION @@ -0,0 +1 @@ +0.16.0 diff --git a/Makefile b/Makefile index 51a7cb52..37b765fe 100644 --- a/Makefile +++ b/Makefile @@ -11,12 +11,15 @@ DEV_DIR := .dev STATE_DIR := $(DEV_DIR)/state AGENT_SOCK := $(abspath $(DEV_DIR)/agent.sock) -# Build identity (BUILD.md # Versioning): one repo VERSION for the whole -# monorepo, plus the git commit a build was cut from — two stamped fields, no -# "-dev" suffix logic (DECISIONS.md 2026-07-16). VERSION is read from the repo -# root; the commit falls back to "unknown" outside a git checkout (e.g. a -# container build context with no .git) rather than failing the build. +# Build identity (BUILD.md # Versioning): two version lines, one per update +# stream (DECISIONS.md 2026-10-01). VERSION is the moose (OS) release and stamps +# host-agent; CONTROL_PLANE_VERSION is the control-plane release and stamps the +# brain (and labels both control-plane images). Every build also stamps the git +# commit it was cut from, with no "-dev" suffix logic. The commit falls back to +# "unknown" outside a git checkout (e.g. a container build context with no .git) +# rather than failing the build. MOOSE_VERSION := $(shell cat $(CURDIR)/VERSION) +CONTROL_PLANE_VERSION := $(shell cat $(CURDIR)/CONTROL_PLANE_VERSION) MOOSE_COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown) # The minisign public keys a host-agent build accepts for the appliance release # manifest (RELEASE_MANIFEST.md # Signing). Comma-separated base64 key lines. @@ -28,6 +31,10 @@ MOOSE_RELEASE_KEYS ?= LDFLAGS := -X github.com/onmoose/os/internal/version.Version=$(MOOSE_VERSION) \ -X github.com/onmoose/os/internal/version.Commit=$(MOOSE_COMMIT) \ -X github.com/onmoose/os/internal/hostagent/relmanifest.BakedKeys=$(MOOSE_RELEASE_KEYS) +# The brain is on the control-plane line, so it gets its own version stamp. It +# takes no release keys: those are host-agent's. +BRAIN_LDFLAGS := -X github.com/onmoose/os/internal/version.Version=$(CONTROL_PLANE_VERSION) \ + -X github.com/onmoose/os/internal/version.Commit=$(MOOSE_COMMIT) export MOOSE_AGENT_SOCK := $(AGENT_SOCK) export MOOSE_STATE_DIR := $(STATE_DIR) @@ -38,7 +45,7 @@ export MOOSE_STATE_DIR := $(STATE_DIR) # store offline. To boot against a local snapshot instead, see `make dev-app`. export MOOSE_CATALOG_CACHE_DIR := ./.dev/catalog-cache -.PHONY: build host-agent brain host-agent-real host-agent-real-hosted brain-image ui-image control-plane-images caddy-acmedns-image build-cloud-image check check-web fmt fmt-check vet test test-nopam test-caddy test-avahi test-netstate test-health test-usermgr test-usermgr-nspawn test-boot-chain-nspawn test-medium-qemu test-cloud-qemu run-agent run-brain net caddy caddy-down ui dev dev-app seed-catalog stop openapi openapi-check clean check-state-owner help +.PHONY: build host-agent brain host-agent-real host-agent-real-hosted brain-image ui-image control-plane-images control-plane-released control-plane-third-party caddy-acmedns-image build-cloud-image check check-web fmt fmt-check vet test test-nopam test-caddy test-avahi test-netstate test-health test-usermgr test-usermgr-nspawn test-boot-chain-nspawn test-medium-qemu test-cloud-qemu run-agent run-brain net caddy caddy-down ui dev dev-app seed-catalog stop openapi openapi-check clean check-state-owner help # msteinert/pam v2.1.0 uses RTLD_NEXT, a GNU extension that requires # _GNU_SOURCE at C compile time. Apply globally; harmless to non-cgo builds. @@ -53,6 +60,7 @@ help: @echo "make check-web - pre-PR gate for frontend changes: web-ui typecheck + build" @echo "make clean - stop apps, remove dev state" @echo "make control-plane-images - build moose-brain + moose-ui images and docker-save the control-plane bundle to .dev/" + @echo "make control-plane-released - the same bundle with the last released brain + UI, pulled from ghcr by digest" @echo "make caddy-acmedns-image - build the hosted Caddy (stock Caddy + the caddy-dns/acmedns module)" @echo "make dev - all three foreground procs in one terminal (recommended); Go edits rebuild + restart the brain" @echo "make dev-app APP= [STORE=../store] - boot ONE store app under curation: seed its catalog snapshot, then make dev with an inert catalog URL" @@ -182,7 +190,7 @@ host-agent-real-hosted: $(GO) build -tags hosted -ldflags "$(LDFLAGS)" -o $(DEV_DIR)/host-agent-real-hosted ./cmd/host-agent-real brain: - $(GO) build -ldflags "$(LDFLAGS)" -o $(DEV_DIR)/brain ./cmd/brain + $(GO) build -ldflags "$(BRAIN_LDFLAGS)" -o $(DEV_DIR)/brain ./cmd/brain # ---- Control-plane images (M0, #163) ----------------------------------- # Build the two moose OCI images and `docker save` them — together with the two @@ -207,8 +215,28 @@ include dev/control-plane/images.lock CADDY_TAG := $(firstword $(subst @, ,$(CADDY_IMAGE))) PROXY_TAG := $(firstword $(subst @, ,$(PROXY_IMAGE))) +# BuildKit layer cache for the hosted Caddy build only (caddy-acmedns-image, +# #486). Every input of that build is pinned, so its cache hits on every run. +# The brain and UI copy the whole tree and rebuild on each commit, so a cache +# cost them more to upload than it saved; they always build plain. The default +# is a plain `docker build` with no cache. CI sets MOOSE_BUILD_CACHE=gha on a +# run that publishes nothing (ci-cloud-image.yml): the Caddy build then goes +# through the docker-container builder named by MOOSE_BUILD_CACHE_BUILDER and +# reads and writes the GitHub Actions cache. A run that publishes sets it to +# `none`, so what ships is always built fresh, with no cache read. A failed +# cache write never fails the build (ignore-error). +MOOSE_BUILD_CACHE ?= none +MOOSE_BUILD_CACHE_BUILDER ?= moose-cache +ifeq ($(MOOSE_BUILD_CACHE),gha) +docker_build = docker buildx build --builder $(MOOSE_BUILD_CACHE_BUILDER) --load \ + --cache-from type=gha,scope=$(1) --cache-to type=gha,mode=max,scope=$(1),ignore-error=true +else +docker_build = docker build +endif + brain-image: docker build -f cmd/brain/Dockerfile --build-arg MOOSE_COMMIT=$(MOOSE_COMMIT) \ + --build-arg CONTROL_PLANE_VERSION=$(CONTROL_PLANE_VERSION) \ --build-arg BRAIN_BUILDER_IMAGE=$(BRAIN_BUILDER_IMAGE) \ --build-arg BRAIN_RUNTIME_IMAGE=$(BRAIN_RUNTIME_IMAGE) \ -t $(BRAIN_IMAGE) . @@ -217,21 +245,38 @@ brain-image: # for both, not two pins to keep level. ui-image: docker build -f web-ui/Dockerfile \ + --build-arg MOOSE_COMMIT=$(MOOSE_COMMIT) \ + --build-arg CONTROL_PLANE_VERSION=$(CONTROL_PLANE_VERSION) \ --build-arg UI_BUILDER_IMAGE=$(UI_BUILDER_IMAGE) \ --build-arg UI_RUNTIME_IMAGE=$(CADDY_IMAGE) \ -t $(UI_IMAGE) web-ui -control-plane-images: brain-image ui-image +control-plane-images: brain-image ui-image control-plane-third-party + @rm -f $(CP_IMAGE_DIR)/control-plane.env + docker save $(BRAIN_IMAGE) -o $(CP_IMAGE_DIR)/moose-brain.tar + docker save $(UI_IMAGE) -o $(CP_IMAGE_DIR)/moose-ui.tar + ./dev/control-plane/bundle-record.sh $(CP_IMAGE_DIR) local $(CONTROL_PLANE_VERSION) + @echo "saved control-plane image bundle to $(CP_IMAGE_DIR)/" + +# The same bundle with the LAST RELEASED brain and UI instead of a build of +# this commit (#566, BUILD.md # Versioning): ghcr.io/onmoose/{brain,ui} at +# v$(CONTROL_PLANE_VERSION), resolved to digests once and pulled by digest, then +# saved under the same local names. An OS-only release bakes this, so the pair +# in the image is a real control-plane release. Needs only Docker and network. +control-plane-released: control-plane-third-party + ./dev/control-plane/pull-released.sh $(CP_IMAGE_DIR) $(CONTROL_PLANE_VERSION) + @echo "saved control-plane image bundle (released $(CONTROL_PLANE_VERSION)) to $(CP_IMAGE_DIR)/" + +# The two third-party images the bundle carries, pulled by digest from the pin +# file and saved under their plain tags. Shared by both bundles above. +control-plane-third-party: @mkdir -p $(CP_IMAGE_DIR) docker pull $(CADDY_IMAGE) docker pull $(PROXY_IMAGE) docker tag $(CADDY_IMAGE) $(CADDY_TAG) docker tag $(PROXY_IMAGE) $(PROXY_TAG) - docker save $(BRAIN_IMAGE) -o $(CP_IMAGE_DIR)/moose-brain.tar - docker save $(UI_IMAGE) -o $(CP_IMAGE_DIR)/moose-ui.tar docker save $(CADDY_TAG) -o $(CP_IMAGE_DIR)/caddy.tar docker save $(PROXY_TAG) -o $(CP_IMAGE_DIR)/docker-socket-proxy.tar - @echo "saved control-plane image bundle to $(CP_IMAGE_DIR)/" # The hosted profile's Caddy: stock Caddy plus the caddy-dns/acmedns module, for # the wildcard cert's ACME DNS-01 (ENVIRONMENT.md # Networking & discovery). Both @@ -239,7 +284,7 @@ control-plane-images: brain-image ui-image # Dockerfile carries no unpinned default — build it through this target, not `docker build` by hand. # dev/cloud/stage-control-plane.sh calls it, then docker-saves the result. caddy-acmedns-image: - docker build \ + $(call docker_build,moose-caddy-acmedns) \ --build-arg CADDY_ACMEDNS_BUILDER_IMAGE=$(CADDY_ACMEDNS_BUILDER_IMAGE) \ --build-arg CADDY_ACMEDNS_BASE_IMAGE=$(CADDY_ACMEDNS_BASE_IMAGE) \ --build-arg CADDY_ACMEDNS_MODULE=$(CADDY_ACMEDNS_MODULE) \ @@ -386,7 +431,7 @@ dev: check-state-owner build caddy @mkdir -p $(STATE_DIR) @cd web-ui && [ -d node_modules ] || npm install @trap 'kill 0' INT TERM EXIT; \ - (GO="$(GO)" DEV_DIR="$(DEV_DIR)" LDFLAGS="$(LDFLAGS)" ./dev/dev-go.sh) & \ + (GO="$(GO)" DEV_DIR="$(DEV_DIR)" LDFLAGS="$(LDFLAGS)" BRAIN_LDFLAGS="$(BRAIN_LDFLAGS)" ./dev/dev-go.sh) & \ (cd web-ui && npm run dev 2>&1 | sed -u 's/^/[ui] /') & \ wait diff --git a/VERSION b/VERSION index a5510516..04a373ef 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.15.0 +0.16.0 diff --git a/api/openapi.json b/api/openapi.json index 25b25f0c..dc57c0b3 100644 --- a/api/openapi.json +++ b/api/openapi.json @@ -2888,6 +2888,96 @@ ], "type": "object" }, + "OSOutcomeDTO": { + "additionalProperties": false, + "properties": { + "at": { + "type": "string" + }, + "from": { + "type": "string" + }, + "id": { + "type": "string" + }, + "outcome": { + "enum": [ + "good", + "reverted" + ], + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "id", + "outcome", + "version", + "at" + ], + "type": "object" + }, + "OSReleaseDTO": { + "additionalProperties": false, + "properties": { + "bundle_sha256": { + "type": "string" + }, + "bundle_url": { + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "version", + "bundle_url", + "bundle_sha256" + ], + "type": "object" + }, + "OSUpdateDTO": { + "additionalProperties": false, + "properties": { + "detail": { + "type": "string" + }, + "last": { + "$ref": "#/components/schemas/OSOutcomeDTO" + }, + "running": { + "type": "string" + }, + "slot": { + "type": "string" + }, + "state": { + "enum": [ + "unsupported", + "none", + "refused", + "current", + "installing", + "installed", + "waiting", + "rebooting", + "held", + "failed" + ], + "type": "string" + }, + "target": { + "$ref": "#/components/schemas/OSReleaseDTO" + } + }, + "required": [ + "state" + ], + "type": "object" + }, "Parse-custom-overlayRequest": { "additionalProperties": false, "properties": { @@ -3555,6 +3645,12 @@ "host_agent_version": { "type": "string" }, + "os_slot": { + "type": "string" + }, + "os_version": { + "type": "string" + }, "ui_image": { "type": "string" }, @@ -3656,6 +3752,9 @@ "from": { "type": "string" }, + "os": { + "$ref": "#/components/schemas/OSUpdateDTO" + }, "profile": { "type": "string" }, @@ -6125,7 +6224,7 @@ "description": "Error" } }, - "summary": "What this box is running: brain version and commit, host-agent version, UI image" + "summary": "What this box is running: brain version and commit, host-agent version, UI image, OS version and slot" } }, "/api/v1/users": { diff --git a/api/openapi.yaml b/api/openapi.yaml index 9d288981..9aeb61c4 100644 --- a/api/openapi.yaml +++ b/api/openapi.yaml @@ -2048,6 +2048,71 @@ components: - summary - read type: object + OSOutcomeDTO: + additionalProperties: false + properties: + at: + type: string + from: + type: string + id: + type: string + outcome: + enum: + - good + - reverted + type: string + version: + type: string + required: + - id + - outcome + - version + - at + type: object + OSReleaseDTO: + additionalProperties: false + properties: + bundle_sha256: + type: string + bundle_url: + type: string + version: + type: string + required: + - version + - bundle_url + - bundle_sha256 + type: object + OSUpdateDTO: + additionalProperties: false + properties: + detail: + type: string + last: + $ref: "#/components/schemas/OSOutcomeDTO" + running: + type: string + slot: + type: string + state: + enum: + - unsupported + - none + - refused + - current + - installing + - installed + - waiting + - rebooting + - held + - failed + type: string + target: + $ref: "#/components/schemas/OSReleaseDTO" + required: + - state + type: object Parse-custom-overlayRequest: additionalProperties: false properties: @@ -2523,6 +2588,10 @@ components: type: string host_agent_version: type: string + os_slot: + type: string + os_version: + type: string ui_image: type: string version: @@ -2595,6 +2664,8 @@ components: type: string from: type: string + os: + $ref: "#/components/schemas/OSUpdateDTO" profile: type: string running: @@ -4109,7 +4180,7 @@ paths: schema: $ref: "#/components/schemas/ErrorModel" description: Error - summary: "What this box is running: brain version and commit, host-agent version, UI image" + summary: "What this box is running: brain version and commit, host-agent version, UI image, OS version and slot" /api/v1/users: get: operationId: list-users diff --git a/cmd/brain/Dockerfile b/cmd/brain/Dockerfile index 22f9f732..629a98c8 100644 --- a/cmd/brain/Dockerfile +++ b/cmd/brain/Dockerfile @@ -21,13 +21,15 @@ WORKDIR /src COPY go.mod go.sum ./ RUN go mod download COPY . . -# Build identity (BUILD.md # Versioning): VERSION is in the build context (repo -# root), but .git is excluded (.dockerignore — never ship dev/VCS state into the +# Build identity (BUILD.md # Versioning): the brain is on the control-plane +# line, so it is stamped from CONTROL_PLANE_VERSION, not from VERSION (which is +# the moose OS release and stamps host-agent). The file is in the build context +# (repo root), but .git is excluded (.dockerignore — never ship dev/VCS state into the # image), so the commit can't be resolved with `git rev-parse` in here. The # Makefile's brain-image target passes it in as a build-arg instead. ARG MOOSE_COMMIT=unknown RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w \ - -X github.com/onmoose/os/internal/version.Version=$(cat VERSION) \ + -X github.com/onmoose/os/internal/version.Version=$(cat CONTROL_PLANE_VERSION) \ -X github.com/onmoose/os/internal/version.Commit=${MOOSE_COMMIT}" \ -o /out/moose-brain ./cmd/brain @@ -53,6 +55,14 @@ RUN set -eux; \ # Must equal internal/protocol.Major; bump both together with the /v URL # prefix. LABEL moose.protocol.major="1" +# The control-plane release this image belongs to, and its commit, as standard +# OCI labels, so `docker inspect` answers "which build is this" without running +# it. `make brain-image` passes both from the same CONTROL_PLANE_VERSION file and +# commit the binary above is stamped with. +ARG CONTROL_PLANE_VERSION=unknown +ARG MOOSE_COMMIT=unknown +LABEL org.opencontainers.image.version="${CONTROL_PLANE_VERSION}" \ + org.opencontainers.image.revision="${MOOSE_COMMIT}" COPY --from=build /out/moose-brain /usr/local/bin/moose-brain # State dir and catalog are bind-mounts the supervisor provides at runtime; the # brain reads its config from MOOSE_* env (cmd/brain/main.go # loadConfig). diff --git a/cmd/brain/main.go b/cmd/brain/main.go index 00a4c6c9..0bd95f33 100644 --- a/cmd/brain/main.go +++ b/cmd/brain/main.go @@ -63,7 +63,7 @@ func main() { showVersion := flag.Bool("version", false, "print the version and exit") flag.Parse() if *showVersion { - fmt.Println(version.String()) + fmt.Println(version.ControlPlaneString()) return } @@ -81,6 +81,7 @@ func main() { if err := os.MkdirAll(cfg.stateDir, 0o755); err != nil { fatal("create state dir", "err", err) } + writeFloorFile(cfg.stateDir) st, err := store.Open(filepath.Join(cfg.stateDir, "moose.db")) if err != nil { @@ -295,6 +296,10 @@ func main() { checkAgentVersion(pollCtx, host, healthMgr, auditor, notifier, bus) go versionCheckPollLoop(pollCtx, host, healthMgr, auditor, notifier, bus, cfg.healthPollPeriod) + // Stream A outcomes (#563): one admin notification per OS update, read + // off host-agent's update-target report on the same cadence. + go osOutcomeLoop(pollCtx, host, st, notifier, cfg.healthPollPeriod) + // Locus-C brain-DB integrity check (HEALTH.md # Detector catalog): PRAGMA // integrity_check at boot + every 6h, reconciling brain-db-corrupt. Runs // entirely on its own goroutine — the boot run is inside the loop, never @@ -837,15 +842,14 @@ func notificationPruneLoop(ctx context.Context, st *store.Store, interval time.D // works with — the box-side stand-in for the release manifest's // minimum_host_agent field (UPDATES.md # 7 Compatibility matrix, // RELEASE_MANIFEST.md # Manifest schema) until that manifest is actually wired -// up. It is deliberately NOT the same value as this brain's own -// internal/version.Version: with one repo version (DECISIONS.md 2026-07-16), -// host-agent and brain still ship from the same commit, but host-agent updates -// ride apt on their own cadence and can lag the brain by up to 24h during a -// rollout (UPDATES.md # 2) — an update ordering where host-agent updates -// *first*, so a brain running a newer build than the agent it's paired with is -// the normal, expected mid-rollout state, not a mismatch. Bump this by hand -// only when a real host-agent<->brain protocol break lands (i.e. an agent -// older than this genuinely can't serve this brain), not on every release. +// up. host-agent carries the moose (OS) version, so this reads as "the oldest +// moose release this control plane runs on". It is deliberately NOT this +// brain's own internal/version.Version: the brain is on the control-plane line +// (CONTROL_PLANE_VERSION) and host-agent on the OS line (VERSION), two numbers +// that move on their own (BUILD.md # Versioning, DECISIONS.md 2026-10-01). A +// newer agent is always fine. Bump this by hand only when a real +// host-agent<->brain protocol break lands (an agent older than this genuinely +// can't serve this brain), not on every release. const minimumAgentVersion = "0.4.0" // agentVersionAcceptable reports whether reportedVersion is new enough to diff --git a/cmd/brain/osupdate.go b/cmd/brain/osupdate.go new file mode 100644 index 00000000..0efa7a6f --- /dev/null +++ b/cmd/brain/osupdate.go @@ -0,0 +1,89 @@ +package main + +import ( + "context" + "log/slog" + "os" + "path/filepath" + "time" + + "github.com/onmoose/os/internal/notify" + "github.com/onmoose/os/internal/protocol" +) + +// This file is the brain's part of stream A (UPDATES.md # 1, #563): it tells +// host-agent which OS floor this control plane needs, and it tells admins what +// an OS update did. host-agent does the update itself. + +// floorFileName is where the brain writes minimumAgentVersion, in its state +// directory. host-agent reads it to refuse an OS target below the running +// control plane's floor (UPDATES.md # 1): booting a host-agent the brain +// refuses to work with would leave a box nobody can reach. +const floorFileName = "minimum-host-agent" + +// writeFloorFile writes the floor. A failure is a warning: host-agent then has +// no floor to check, which is the state of a box before #563. +func writeFloorFile(stateDir string) { + path := filepath.Join(stateDir, floorFileName) + if err := os.WriteFile(path, []byte(minimumAgentVersion+"\n"), 0o644); err != nil { + slog.Warn("could not write the control-plane floor for host-agent; it will not check OS targets against it", "err", err, "state_dir", stateDir) + } +} + +// osOutcomeReader is the slice of the host client the outcome check needs. +type osOutcomeReader interface { + SystemUpdateTarget(ctx context.Context) (protocol.UpdateTarget, error) +} + +// notificationLookup says whether a notification was ever raised for a key. +type notificationLookup interface { + HasNotification(dedupKey string) (bool, error) +} + +// osOutcomeNotifier is the slice of the notifier the check needs. +type osOutcomeNotifier interface { + OSUpdateOutcome(outcomeID, outcome, version, from string) bool +} + +// osOutcomeMaxAge bounds which outcomes still get a notification. host-agent +// keeps the last outcome for good, and notifications are pruned after 90 days, +// so without a bound an old outcome would be announced again after its row was +// pruned. A box whose brain was down for a week after an update loses that one +// notification; the outcome is still on the update-target read. +const osOutcomeMaxAge = 7 * 24 * time.Hour + +// checkOSOutcome raises one admin notification per OS update outcome. +func checkOSOutcome(ctx context.Context, host osOutcomeReader, seen notificationLookup, n osOutcomeNotifier, now time.Time) { + t, err := host.SystemUpdateTarget(ctx) + if err != nil || t.OS == nil || t.OS.Last == nil || t.OS.Last.ID == "" { + return + } + last := t.OS.Last + at, err := time.Parse(time.RFC3339, last.At) + if err != nil || now.Sub(at) > osOutcomeMaxAge { + return + } + done, err := seen.HasNotification(notify.OSUpdateDedupKey(last.ID)) + if err != nil { + slog.Warn("os update: could not check for an earlier notification; trying on the next poll", "err", err) + return + } + if done { + return + } + if n.OSUpdateOutcome(last.ID, last.Outcome, last.Version, last.From) { + slog.Info("os update: notified admins of the outcome", "os", last.Version) + } +} + +// osOutcomeLoop runs checkOSOutcome on the health-poll cadence. +func osOutcomeLoop(ctx context.Context, host osOutcomeReader, seen notificationLookup, n osOutcomeNotifier, interval time.Duration) { + for { + checkOSOutcome(ctx, host, seen, n, time.Now()) + select { + case <-ctx.Done(): + return + case <-time.After(interval): + } + } +} diff --git a/cmd/brain/osupdate_test.go b/cmd/brain/osupdate_test.go new file mode 100644 index 00000000..4d29db9e --- /dev/null +++ b/cmd/brain/osupdate_test.go @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/onmoose/os/internal/protocol" +) + +type fakeOutcomeHost struct{ t protocol.UpdateTarget } + +func (f fakeOutcomeHost) SystemUpdateTarget(context.Context) (protocol.UpdateTarget, error) { + return f.t, nil +} + +type fakeSeen map[string]bool + +func (f fakeSeen) HasNotification(k string) (bool, error) { return f[k], nil } + +type fakeOutcomeNotifier struct{ raised []string } + +func (f *fakeOutcomeNotifier) OSUpdateOutcome(id, outcome, version, from string) bool { + f.raised = append(f.raised, id+" "+outcome+" "+version+" "+from) + return true +} + +func TestCheckOSOutcome(t *testing.T) { + now := time.Date(2026, 10, 3, 9, 0, 0, 0, time.UTC) + last := &protocol.OSOutcome{ID: "os-0.15.1-1", Outcome: "reverted", Version: "0.15.1", From: "0.15.0", At: "2026-10-03T03:20:00Z"} + host := fakeOutcomeHost{protocol.UpdateTarget{OS: &protocol.OSUpdate{State: "held", Last: last}}} + + n := &fakeOutcomeNotifier{} + checkOSOutcome(context.Background(), host, fakeSeen{}, n, now) + if len(n.raised) != 1 || n.raised[0] != "os-0.15.1-1 reverted 0.15.1 0.15.0" { + t.Fatalf("want one notification, got %v", n.raised) + } + + n = &fakeOutcomeNotifier{} + checkOSOutcome(context.Background(), host, fakeSeen{"os-update:os-0.15.1-1": true}, n, now) + if len(n.raised) != 0 { + t.Fatalf("an outcome already notified was raised again: %v", n.raised) + } + + n = &fakeOutcomeNotifier{} + checkOSOutcome(context.Background(), host, fakeSeen{}, n, now.Add(8*24*time.Hour)) + if len(n.raised) != 0 { + t.Fatalf("an old outcome was raised: %v", n.raised) + } + + n = &fakeOutcomeNotifier{} + checkOSOutcome(context.Background(), fakeOutcomeHost{}, fakeSeen{}, n, now) + if len(n.raised) != 0 { + t.Fatalf("no OS part must raise nothing: %v", n.raised) + } +} + +func TestWriteFloorFile(t *testing.T) { + dir := t.TempDir() + writeFloorFile(dir) + b, err := os.ReadFile(filepath.Join(dir, floorFileName)) + if err != nil { + t.Fatal(err) + } + if strings.TrimSpace(string(b)) != minimumAgentVersion { + t.Fatalf("got %q", b) + } +} diff --git a/cmd/host-agent-real/main.go b/cmd/host-agent-real/main.go index fe6286ec..30e7598d 100644 --- a/cmd/host-agent-real/main.go +++ b/cmd/host-agent-real/main.go @@ -41,6 +41,7 @@ import ( "github.com/onmoose/os/internal/hostagent/brainlaunch" "github.com/onmoose/os/internal/hostagent/controlplane" "github.com/onmoose/os/internal/hostagent/cpupdate" + "github.com/onmoose/os/internal/hostagent/sshaccess" "github.com/onmoose/os/internal/profile" "github.com/onmoose/os/internal/protocol" "github.com/onmoose/os/internal/version" @@ -80,6 +81,25 @@ func main() { a, cleanup := buildAgent() defer cleanup() + // After a Debian major the /etc tidy-up keeps the sshd drop-in but drops the + // unit's enable links (BUILD.md # 1b, rule 4), and leaves a marker. Only + // then is sshd turned back on for the accounts the drop-in names. On any + // other start host-agent leaves sshd's run state alone. + if _, err := os.Stat(sshaccess.MajorTidiedMarker); err == nil { + if sm, ok := a.SSH.(*sshaccess.Manager); ok { + if on, err := sm.EnsureOnAtStart(); err != nil { + slog.Warn("could not turn sshd on after a Debian-major tidy-up; trying again at the next start", "err", err) + } else { + if on { + slog.Info("sshd turned on after a Debian-major tidy-up: accounts have SSH on") + } + if err := os.Remove(sshaccess.MajorTidiedMarker); err != nil { + slog.Warn("could not remove the tidy-up marker", "err", err) + } + } + } + } + // The brain's launch config is built once and used twice: to launch the // brain at boot, and as the base of every control-plane update. Reusing it // is the point — an updated brain has to be identical to a first-boot one @@ -138,7 +158,15 @@ func main() { // prompt differ, and both come from the build-tagged updateTargetSource. // Started last for the same reason the poll is: nothing about booting waits // on it. - stopTarget := startUpdateTarget(brainCfg, a, poller) + // Stream A (#563). Boot decides first whether this boot is an OS trial, + // whatever the update loop does below: a trial has to be decided even on + // a box whose update target is unusable. + osApp := osUpdateApplier(osUpdateDeps{agent: a, brainCfg: brainCfg}) + if osApp != nil { + osApp.Boot(context.Background()) + a.OS = osApp + } + stopTarget := startUpdateTarget(brainCfg, a, poller, osApp) defer stopTarget() slog.Info("host-agent-real listening", "sock", sockPath) diff --git a/cmd/host-agent-real/osupdate.go b/cmd/host-agent-real/osupdate.go new file mode 100644 index 00000000..979f518e --- /dev/null +++ b/cmd/host-agent-real/osupdate.go @@ -0,0 +1,114 @@ +package main + +import ( + "context" + "errors" + "fmt" + "log/slog" + "os" + "path/filepath" + "time" + + "github.com/onmoose/os/internal/hostagent" + "github.com/onmoose/os/internal/hostagent/brainlaunch" + "github.com/onmoose/os/internal/hostagent/cpupdate" + "github.com/onmoose/os/internal/hostagent/osupdate" + "github.com/onmoose/os/internal/hostagent/updatetarget" + "github.com/onmoose/os/internal/version" +) + +// Stream A's settings (#563). Not build-tagged, so `go vet` and `go test` +// see them; only the call that builds the applier is (osupdate_hosted.go). +const ( + // envOSURLPrefix is the expected start of every bundle URL. The default + // is the GitHub Releases path of this repo (updatetarget.DefaultOSURLPrefix). + // The boot lane points it at a server inside the guest. + envOSURLPrefix = "MOOSE_UPDATE_OS_URL_PREFIX" + // envOSTrialTimeout bounds a trial boot's wait for the brain, as a Go + // duration. The default is osupdate.DefaultTrialTimeout. + envOSTrialTimeout = "MOOSE_OS_TRIAL_TIMEOUT" +) + +// osURLPrefix is the bundle URL prefix in force. +func osURLPrefix() string { + if v := os.Getenv(envOSURLPrefix); v != "" { + return v + } + return updatetarget.DefaultOSURLPrefix +} + +// osTrialTimeout is the trial bound in force. An unreadable value warns and +// falls back: a wrong bound only makes a trial shorter or longer. +func osTrialTimeout() time.Duration { + v := os.Getenv(envOSTrialTimeout) + if v == "" { + return osupdate.DefaultTrialTimeout + } + d, err := time.ParseDuration(v) + if err != nil || d <= 0 { + slog.Warn("os update: "+envOSTrialTimeout+" is not a positive duration; using the default", "err", err) + return osupdate.DefaultTrialTimeout + } + // Under the image's 15 min timer, so a slow but healthy slot is decided + // by host-agent and never reverted by the timer. + if d > osupdate.MaxTrialTimeout { + slog.Warn("os update: "+envOSTrialTimeout+" is longer than the image's trial timer allows; using the longest allowed", "err", fmt.Errorf("%s > %s", d, osupdate.MaxTrialTimeout)) + return osupdate.MaxTrialTimeout + } + return d +} + +// osUpdateDeps is what the applier is built from. +type osUpdateDeps struct { + agent *hostagent.Agent + brainCfg brainlaunch.Config +} + +// agentJobs adapts the agent's job lock to the applier's Jobs seam, and maps +// the lock's refusal to osupdate.ErrBusy. +type agentJobs struct{ a *hostagent.Agent } + +func (j agentJobs) StartJob(kind string, d time.Duration, fn func(ctx context.Context) error) (string, error) { + job, err := j.a.StartJob(kind, d, fn) + if hostagent.IsJobRunning(err) { + return "", fmt.Errorf("%w: %v", osupdate.ErrBusy, err) + } + if err != nil { + return "", err + } + return job.ID, nil +} + +// newOSApplier builds the applier. The health check is the brain's /healthz on +// its container address, the same probe the control-plane update uses +// (cpupdate), retried until the trial's deadline: on a fresh boot the brain +// container may not have an address yet. +func newOSApplier(d osUpdateDeps) *osupdate.Applier { + docker := cpupdate.NewCLIDocker() + prober := cpupdate.HTTPProber{} + healthy := func(ctx context.Context) error { + var last error + for { + ip, err := docker.ContainerIP(ctx, d.brainCfg.ContainerName) + if err == nil { + if err = prober.WaitServing(ctx, fmt.Sprintf("http://%s:8080/healthz", ip)); err == nil { + return nil + } + } + last = err + select { + case <-ctx.Done(): + return errors.Join(ctx.Err(), last) + case <-time.After(2 * time.Second): + } + } + } + return &osupdate.Applier{ + RAUC: osupdate.CLIRAUC{}, + Jobs: agentJobs{d.agent}, + Version: version.Version, + FloorFile: filepath.Join(d.brainCfg.StateDir, "minimum-host-agent"), + Healthy: healthy, + TrialTimeout: osTrialTimeout(), + } +} diff --git a/cmd/host-agent-real/osupdate_appliance.go b/cmd/host-agent-real/osupdate_appliance.go new file mode 100644 index 00000000..2b52be9b --- /dev/null +++ b/cmd/host-agent-real/osupdate_appliance.go @@ -0,0 +1,10 @@ +//go:build !hosted + +package main + +import "github.com/onmoose/os/internal/hostagent/osupdate" + +// osUpdateApplier is nil on the appliance build: its image is not in the A/B +// layout yet (#564), so there is no other slot to install into. The update +// loop then reports stream A as unsupported. +func osUpdateApplier(osUpdateDeps) *osupdate.Applier { return nil } diff --git a/cmd/host-agent-real/osupdate_hosted.go b/cmd/host-agent-real/osupdate_hosted.go new file mode 100644 index 00000000..a54e25ed --- /dev/null +++ b/cmd/host-agent-real/osupdate_hosted.go @@ -0,0 +1,10 @@ +//go:build hosted + +package main + +import "github.com/onmoose/os/internal/hostagent/osupdate" + +// osUpdateApplier builds stream A's applier on the hosted build, the one +// profile whose image is in the A/B layout today (BUILD.md # 1b). The +// appliance gets it with #564. +func osUpdateApplier(r osUpdateDeps) *osupdate.Applier { return newOSApplier(r) } diff --git a/cmd/host-agent-real/updatetarget.go b/cmd/host-agent-real/updatetarget.go index e0d246e6..cd7534bf 100644 --- a/cmd/host-agent-real/updatetarget.go +++ b/cmd/host-agent-real/updatetarget.go @@ -7,6 +7,7 @@ import ( "github.com/onmoose/os/internal/hostagent" "github.com/onmoose/os/internal/hostagent/brainlaunch" + "github.com/onmoose/os/internal/hostagent/osupdate" "github.com/onmoose/os/internal/hostagent/relmanifest" "github.com/onmoose/os/internal/hostagent/updatetarget" ) @@ -20,7 +21,7 @@ import ( // validation, the window, the apply, the failure handling — is one loop shared by // both profiles. A second copy of any of it, per profile, is what UPDATES.md # 8 // means by "we only build it once". -func startUpdateTarget(brainCfg brainlaunch.Config, a *hostagent.Agent, poller *relmanifest.Poller) func() { +func startUpdateTarget(brainCfg brainlaunch.Config, a *hostagent.Agent, poller *relmanifest.Poller, osApp *osupdate.Applier) func() { window, windowFrom := updateWindow() // What the box is running, read fresh on every socket read. Built here // whatever happens below: a box that will not update itself still has to be @@ -53,6 +54,7 @@ func startUpdateTarget(brainCfg brainlaunch.Config, a *hostagent.Agent, poller * // box could not build. autoApply is left false on purpose: nothing // on this box will apply anything, whatever profile it is. profile: buildProfile, + os: osReporter(osApp), } return func() {} } @@ -64,10 +66,16 @@ func startUpdateTarget(brainCfg brainlaunch.Config, a *hostagent.Agent, poller * Repos: repositories(), // The box's own window. An answer that names one wins over it, so this // is the fallback the loop uses until a source has an opinion. - Window: window, - WindowFrom: windowFrom, - AutoApply: src.AutoApply, - Profile: src.Profile, + Window: window, + WindowFrom: windowFrom, + AutoApply: src.AutoApply, + Profile: src.Profile, + OSURLPrefix: osURLPrefix(), + } + // Set only when there is an applier: a nil *Applier in the interface + // field would not read as nil. + if osApp != nil { + loop.OS = osApp } // The socket read (GET /v1/system/update-target) is served from this loop's // snapshot. Set before Run, so a read that arrives during the first tick @@ -80,6 +88,7 @@ func startUpdateTarget(brainCfg brainlaunch.Config, a *hostagent.Agent, poller * windowFrom: windowFrom, profile: src.Profile, autoApply: src.AutoApply, + os: osReporter(osApp), } ctx, cancel := context.WithCancel(context.Background()) go loop.Run(ctx) diff --git a/cmd/host-agent-real/updatetargetreport.go b/cmd/host-agent-real/updatetargetreport.go index 785d6520..93526704 100644 --- a/cmd/host-agent-real/updatetargetreport.go +++ b/cmd/host-agent-real/updatetargetreport.go @@ -3,6 +3,7 @@ package main import ( "time" + "github.com/onmoose/os/internal/hostagent/osupdate" "github.com/onmoose/os/internal/hostagent/updatetarget" "github.com/onmoose/os/internal/protocol" ) @@ -39,6 +40,57 @@ type updateTargetReport struct { // build-tagged source that built the loop. profile string autoApply bool + // os reads stream A's facts that the loop does not hold: the running OS + // release and slot, and the last outcome, which outlive a reboot. Nil on + // a box that cannot update its OS. + os osFacts +} + +// osFacts is the slice of osupdate.Applier the report reads. +type osFacts interface { + Running() (version, slot string) + Last() *protocol.OSOutcome + Peek(rel updatetarget.OSRelease) (state, detail string, ok bool) +} + +// osReporter returns the applier as osFacts, or a nil interface for a nil +// applier, so a box with no applier reports no OS facts rather than calling a +// nil pointer. +func osReporter(a *osupdate.Applier) osFacts { + if a == nil { + return nil + } + return a +} + +// readOS assembles stream A's part of the report. +func (r updateTargetReport) readOS() *protocol.OSUpdate { + if r.os == nil { + return &protocol.OSUpdate{State: protocol.OSUpdateUnsupported, Detail: "this host-agent cannot update the OS"} + } + out := &protocol.OSUpdate{State: protocol.OSUpdateNone, Last: r.os.Last()} + out.Running, out.Slot = r.os.Running() + if r.loop == nil { + out.Detail = "this box has no update loop, so it will not update its OS" + return out + } + s := r.loop.Snapshot().OS + if s.State != "" { + out.State, out.Detail = s.State, s.Detail + } + if s.Target != nil { + // The loop decides once per tick; the job it started may have ended + // since. Only the states a job moves between are refreshed: current, + // refused, held and the rest are the loop's to say. + switch s.State { + case protocol.OSUpdateInstalling, protocol.OSUpdateInstalled, protocol.OSUpdateWaiting, protocol.OSUpdateFailed: + if st, d, ok := r.os.Peek(*s.Target); ok { + out.State, out.Detail = st, d + } + } + out.Target = &protocol.OSRelease{Version: s.Target.Version, BundleURL: s.Target.BundleURL, BundleSHA256: s.Target.BundleSHA256} + } + return out } // Read assembles the report. It never fails: every way this can go wrong is a @@ -46,6 +98,7 @@ type updateTargetReport struct { // say so in a form the dashboard can render. func (r updateTargetReport) Read() protocol.UpdateTarget { out := protocol.UpdateTarget{ + OS: r.readOS(), From: r.from, Window: r.window.String(), WindowFrom: r.windowFrom, diff --git a/cmd/host-agent-real/updatetargetreport_test.go b/cmd/host-agent-real/updatetargetreport_test.go index 342afb8b..a5e7451a 100644 --- a/cmd/host-agent-real/updatetargetreport_test.go +++ b/cmd/host-agent-real/updatetargetreport_test.go @@ -287,3 +287,55 @@ func TestReport_RefusedTargetIsNotApplicable(t *testing.T) { t.Fatalf("target = %+v, want the refused answer verbatim", got.Target) } } + +// stubOS is stream A's facts for the report. +type stubOS struct { + peekState string +} + +func (s stubOS) Running() (string, string) { return "0.15.0", "A" } +func (s stubOS) Last() *protocol.OSOutcome { + return &protocol.OSOutcome{ID: "os-0.15.0-1", Outcome: protocol.OSOutcomeGood, Version: "0.15.0", At: "2026-10-01T03:10:00Z"} +} +func (s stubOS) Peek(updatetarget.OSRelease) (string, string, bool) { + return s.peekState, "", s.peekState != "" +} + +type stubOSApplier struct{ state string } + +func (stubOSApplier) Running() (string, string) { return "0.15.0", "A" } +func (stubOSApplier) Floor() string { return "" } +func (a stubOSApplier) Apply(updatetarget.OSRelease, bool, time.Time) updatetarget.OSDecision { + return updatetarget.OSDecision{State: a.state} +} + +func TestReportOS(t *testing.T) { + // No applier: stream A is reported, as unsupported. + r := updateTargetReport{loop: tickedLoop(t, stubSource{target: goodOffer()}, stubRunning{brain: runningBrain, ui: runningUI})} + if got := r.Read().OS; got == nil || got.State != protocol.OSUpdateUnsupported { + t.Fatalf("no applier: %+v", got) + } + + tgt := goodOffer() + tgt.OS = []updatetarget.OSRelease{{Version: "0.15.1", BundleURL: updatetarget.DefaultOSURLPrefix + "v0.15.1/b.raucb", BundleSHA256: strings.Repeat("e", 64)}} + l := &updatetarget.Loop{Source: stubSource{target: tgt}, Current: stubRunning{brain: runningBrain, ui: runningUI}, + Profile: "hosted", OS: stubOSApplier{state: protocol.OSUpdateInstalling}} + l.Tick(context.Background()) + + // The tick said installing; the job has finished since, and Peek says so. + r = updateTargetReport{loop: l, running: stubRunning{brain: runningBrain, ui: runningUI}, os: stubOS{peekState: protocol.OSUpdateInstalled}} + got := r.Read().OS + if got.State != protocol.OSUpdateInstalled || got.Running != "0.15.0" || got.Slot != "A" || got.Target == nil || got.Target.Version != "0.15.1" || got.Last == nil { + t.Fatalf("report: %+v", got) + } + // Peek with nothing newer leaves the loop's word. + r.os = stubOS{} + if got := r.Read().OS; got.State != protocol.OSUpdateInstalling { + t.Fatalf("without a newer state: %+v", got) + } + // A box with no loop still says what it runs. + r = updateTargetReport{disabledErr: "bad seed", os: stubOS{}} + if got := r.Read().OS; got.State != protocol.OSUpdateNone || got.Running != "0.15.0" || got.Detail == "" { + t.Fatalf("disabled: %+v", got) + } +} diff --git a/cmd/host-agent-real/wiring_hosted.go b/cmd/host-agent-real/wiring_hosted.go index 9854f9e0..73e32d86 100644 --- a/cmd/host-agent-real/wiring_hosted.go +++ b/cmd/host-agent-real/wiring_hosted.go @@ -66,7 +66,9 @@ func buildAgent() (*hostagent.Agent, func()) { a.Resources = rampressure.New() // One diskusage.Reporter satisfies both disk seams: DataDisk() for the // install-plan free_bytes (Disk) and Disks() for the Storage bars (DiskSpace). - du := diskusage.New() + // NewHosted measures the state partition, not the 1 GiB OS slot at / + // (BUILD.md # 1b). + du := diskusage.NewHosted() a.Disk = du a.DiskSpace = du a.Reboot = rebootrequired.New() diff --git a/cmd/host-agent/updatetarget.go b/cmd/host-agent/updatetarget.go index 7cf33429..4f5b864d 100644 --- a/cmd/host-agent/updatetarget.go +++ b/cmd/host-agent/updatetarget.go @@ -45,6 +45,8 @@ func fakeUpdateTarget() protocol.UpdateTarget { PublishedAt: "2026-09-01T10:00:00Z", } + out.OS = fakeOSUpdate() + want := os.Getenv("MOOSE_FAKE_UPDATE_TARGET") switch want { case "", protocol.UpdateTargetNone: @@ -71,3 +73,36 @@ func fakeUpdateTarget() protocol.UpdateTarget { } return out } + +// fakeOSUpdate is stream A's canned part (#563). The fake has no slots, so the +// honest default is "unsupported". MOOSE_FAKE_OS_UPDATE picks another state, +// so a dashboard can be built against the shapes a hosted box sends. +func fakeOSUpdate() *protocol.OSUpdate { + want := os.Getenv("MOOSE_FAKE_OS_UPDATE") + if want == "" || want == protocol.OSUpdateUnsupported { + return &protocol.OSUpdate{State: protocol.OSUpdateUnsupported, Detail: "the fake host-agent has no OS slots"} + } + out := &protocol.OSUpdate{ + State: want, + Running: "0.15.0", + Slot: "A", + Target: &protocol.OSRelease{ + Version: "0.15.1", + BundleURL: "https://github.com/onmoose/os/releases/download/v0.15.1/moose-v0.15.1-amd64.raucb", + BundleSHA256: "5555555555555555555555555555555555555555555555555555555555555555", + }, + } + switch want { + case protocol.OSUpdateCurrent: + out.Running = "0.15.1" + out.Last = &protocol.OSOutcome{ID: "os-0.15.1-1790000000", Outcome: protocol.OSOutcomeGood, Version: "0.15.1", From: "0.15.0", At: "2026-10-03T03:12:00Z"} + case protocol.OSUpdateHeld: + out.Detail = "this release was already tried tonight; the next window tries again" + out.Last = &protocol.OSOutcome{ID: "os-0.15.1-1790000000", Outcome: protocol.OSOutcomeReverted, Version: "0.15.1", From: "0.15.0", At: "2026-10-03T03:20:00Z"} + case protocol.OSUpdateFailed, protocol.OSUpdateRefused: + out.Detail = "the bundle's sha256 is 0000..., not the 5555... the update target names; refusing it" + case protocol.OSUpdateNone: + out.Target = nil + } + return out +} diff --git a/dev/cloud/bootstrap.sh b/dev/cloud/bootstrap.sh index 3f932522..a625239d 100755 --- a/dev/cloud/bootstrap.sh +++ b/dev/cloud/bootstrap.sh @@ -16,14 +16,20 @@ # 2. Stage the first-boot wiring into mkosi.extra.wiring/ via the shared # dev/cloud/stage-control-plane.sh (slim host-agent + units + control-plane # bundle + seed materializer; the SAME staging the test lane runs). -# 3. Stage Docker's apt repo (trixie pocket) into mkosi.pkgmngr/ so docker-ce -# resolves at build time (build-host network only; the VM never apt-installs). -# 4. `mkosi build` → a raw GPT disk image under .dev/cloud/. +# 3. Stage the OS package lock (#560, dev/os-lock/os-lock.sh) into +# mkosi.pkgmngr/: Docker's apt repo (trixie pocket) with exact version pins, +# trixie-updates from the snapshot, and apt retries (build-host network only; +# the VM never apt-installs). +# 4. `mkosi build --snapshot ` → a raw GPT disk +# image under .dev/cloud/. # 5. Assert the built package manifest matches the checked-in expected set # (expected-packages.txt) exactly — any off-list package (bloat, e.g. # recommends drift) or unexpectedly-dropped package fails the build, not # just a hardcoded appliance cut list (#238). Then source-sanity-check the -# committed ExtraTrees marker reads `hosted`. +# committed ExtraTrees marker reads `hosted`. Then assert the resolved +# package list, versions included, equals dev/os-lock/cloud-packages.lock. +# 6. Check the OS slot budget (#561): the squashfs in slot A fills at most 60% +# of its 1 GiB partition (dev/cloud/slot-budget.sh). # # Needs root: it builds the control-plane image bundle (docker) and chowns build # artifacts back to the caller; mkosi itself runs as the caller (it auto-escalates @@ -167,16 +173,14 @@ mkdir -p "$WORK" . "${CLOUD_DIR}/stage-control-plane.sh" stage_control_plane -# --- 3. Docker apt repo for the build's package manager.# The build host has network; the VM never apt-installs Docker (baked). trixie -# pocket — the cloud image is Release=trixie (the test lane uses bookworm). -rm -rf "$PKGMNGR" -mkdir -p "$PKGMNGR/etc/apt/keyrings" "$PKGMNGR/etc/apt/sources.list.d" -curl -fsSL https://download.docker.com/linux/debian/gpg \ - -o "$PKGMNGR/etc/apt/keyrings/docker.asc" -chmod a+r "$PKGMNGR/etc/apt/keyrings/docker.asc" -cat > "$PKGMNGR/etc/apt/sources.list.d/docker.list" <<'EOF' -deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable -EOF +# --- 3. the OS package lock (#560): the build's apt sources and pins, from +# dev/os-lock/. Debian comes from snapshot.debian.org at the locked timestamp, +# Docker from its own repo at the pinned versions (the trixie pocket — the cloud +# image is Release=trixie). Build-host network only; the VM never apt-installs. +# shellcheck source=dev/os-lock/os-lock.sh +. "${REPO_ROOT}/dev/os-lock/os-lock.sh" +stage_os_lock_apt "$PKGMNGR" +OS_LOCK_SNAPSHOT="$(os_lock_snapshot)" # --- 4. build. mkosi runs as the caller (it auto-sudos for privileged disk ops); # re-own the staged trees first. NOT $CP_BUNDLE (shared with the medium lane; mkosi @@ -205,9 +209,9 @@ fi if [ -n "$CALLER" ]; then sudo -u "$CALLER" env "MKOSI_INTERPRETER=$MKOSI_INTERPRETER" \ - "$MKOSI_BIN" --directory "$CLOUD_DIR" --force build + "$MKOSI_BIN" --directory "$CLOUD_DIR" --snapshot "$OS_LOCK_SNAPSHOT" --force build else - MKOSI_INTERPRETER="$MKOSI_INTERPRETER" "$MKOSI_BIN" --directory "$CLOUD_DIR" --force build + MKOSI_INTERPRETER="$MKOSI_INTERPRETER" "$MKOSI_BIN" --directory "$CLOUD_DIR" --snapshot "$OS_LOCK_SNAPSHOT" --force build fi # --- 5. assert lean @@ -227,7 +231,13 @@ fi # names only, so version bumps don't churn it. Regenerate it by pasting the # reported sets when a package change is intended (see expected-packages.txt). EXPECTED="${CLOUD_DIR}/expected-packages.txt" -python3 - "$MANIFEST" "$EXPECTED" <<'PY' +# +# In the bump workflow's record mode (MOOSE_OS_LOCK_RECORD=1) a failed lean +# check does not stop the build: the bump still needs the resolved list to show +# in its PR. The result goes to .dev/cloud/lean-check.txt for the PR body, and +# the PR's own CI run then fails the check, so a person reviews the new set. +lean_rc=0 +python3 - "$MANIFEST" "$EXPECTED" > "$WORK/lean-check.txt" 2>&1 <<'PY' || lean_rc=$? import json, re, sys manifest_path, expected_path = sys.argv[1], sys.argv[2] @@ -276,6 +286,20 @@ if unexpected or missing: print(f"lean check passed — manifest matches expected-packages.txt exactly " f"({len(installed)} packages)") PY +cat "$WORK/lean-check.txt" +if [ "$lean_rc" -ne 0 ]; then + if [ "${MOOSE_OS_LOCK_RECORD:-}" = "1" ]; then + echo "lean check failed (rc=$lean_rc); going on because MOOSE_OS_LOCK_RECORD=1" >&2 + else + exit "$lean_rc" + fi +fi + +# OS package lock (#560): the resolved package list must equal the committed +# dev/os-lock/cloud-packages.lock, so two builds of one commit install the same +# versions. The resolved list is written to .dev/cloud/cloud-packages.lock either +# way (CI uploads it). +os_lock_check "$MANIFEST" "$WORK/cloud-packages.lock" # Source-sanity check: verify the committed ExtraTrees source file reads `hosted` # so a stale or accidentally blanked file fails fast before the next build. @@ -289,4 +313,10 @@ echo "source-sanity check passed: ExtraTrees source $MARKER reads 'hosted'" IMAGE_OUT="$(ls -1 "$WORK"/*.raw 2>/dev/null | head -n1 || true)" [ -n "$CALLER" ] && [ -n "$IMAGE_OUT" ] && \ chown "$CALLER":"$(id -gn "$CALLER")" "$IMAGE_OUT" 2>/dev/null || true + +# The OS slot budget (#561, BUILD.md # 1b # Disk budget): the squashfs in slot A +# must fill at most 60% of its fixed 1 GiB, or the build fails. +# shellcheck source=dev/cloud/slot-budget.sh +. "${CLOUD_DIR}/slot-budget.sh" +slot_budget_check "$IMAGE_OUT" echo "hosted cloud image built: ${IMAGE_OUT:-}" diff --git a/dev/cloud/build-bundle.sh b/dev/cloud/build-bundle.sh new file mode 100755 index 00000000..98637da4 --- /dev/null +++ b/dev/cloud/build-bundle.sh @@ -0,0 +1,197 @@ +#!/usr/bin/env bash +# Build the OS update bundle from the image that ships (#562, BUILD.md # 1b +# # The bundle), check it, and rehearse the sign job's signing. +# +# sudo -E dev/cloud/build-bundle.sh IMAGE.raw OUTDIR +# +# IMAGE.raw is the production image dev/cloud/bootstrap.sh built. The bundle is +# one RAUC bundle in the verity format, holding slot A of that image: the same +# bytes as slot A in the published disk image, cut to the squashfs's own size +# (dev/cloud/slotbudget -extract). Its manifest takes `compatible` from the +# image's own /etc/rauc/system.conf, `version` from VERSION and `build` from the +# git commit. +# +# It is always signed here with the throwaway signer (dev/cloud/rauc.sh). A run +# that publishes the OS re-signs it with the release signer in the sign job +# (dev/release/sign-bundle.sh), so the release key never meets this build, +# which runs mkosi as root with a lot of third-party code. +# +# Checks, all against the keyring and system.conf read back OUT OF the slot +# (unsquashfs), not the copies in the repo: +# 1. the keyring the slot carries is the one this build staged +# (MOOSE_RAUC_KEYRING: the throwaway root, or dev/release/rauc/release-ca.pem); +# 2. the bundle verifies against the throwaway root; +# 3. with the throwaway keyring, the image's own config accepts the bundle; +# with the release keyring, it must REFUSE it: an image that ships may +# never trust the throwaway signer; +# 4. a wrong key: a bundle-shaped check against an unrelated CA is refused; +# 5. a rehearsal of dev/release/sign-bundle.sh with a throwaway "release" CA, +# so the sign job's script runs on every build, not only on a release; +# 6. the image's account and group ids, read from the sysusers.d file the +# build generates, match dev/os-lock/cloud-accounts.lock. +# +# Writes to OUTDIR: moose-cloud.raucb, image-etc/{system.conf,keyring.pem} and +# bundle-info.txt, and a size table to $GITHUB_STEP_SUMMARY when set. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +# shellcheck source=dev/cloud/rauc.sh +. "${REPO_ROOT}/dev/cloud/rauc.sh" + +image="$(realpath "${1:?usage: build-bundle.sh IMAGE.raw OUTDIR}")" +out="${2:?usage: build-bundle.sh IMAGE.raw OUTDIR}" +mode="$(rauc_keyring_mode)" +GO="${GO:-$(command -v go || true)}" +[ -n "$GO" ] || { echo "build-bundle: go not found (set GO)" >&2; exit 1; } +CALLER="${SUDO_USER:-}" + +rm -rf "$out" +mkdir -p "$out/bundle" "$out/image-etc" +out="$(realpath "$out")" + +# The throwaway signer signs every bundle here. The image build already made +# it; this only makes it when the image was built elsewhere. +rauc_throwaway_ca +cp "$RAUC_THROWAWAY_DIR/root-ca.pem" "$out/throwaway-root.pem" +cp "$RAUC_THROWAWAY_DIR/signer.pem" "$out/throwaway-signer.pem" +cp "$RAUC_THROWAWAY_DIR/signer.key" "$out/throwaway-signer.key" + +# The keyring this build staged, for check 1. +if [ "$mode" = "release" ]; then + rauc_require_release_ca + staged="$RAUC_RELEASE_CA" +else + staged="$RAUC_THROWAWAY_DIR/root-ca.pem" +fi + +# The wrong key for check 4, and a throwaway "release" CA for the rehearsal. +RAUC_CA_NAME="moose WRONG KEY (check)" RAUC_CA_ROOT_DAYS=2 RAUC_CA_NO_PASSPHRASE=1 \ + "${REPO_ROOT}/dev/release/rauc-ca.sh" root "$out/wrong" >/dev/null +RAUC_CA_NAME="moose REHEARSAL release (check)" RAUC_CA_ROOT_DAYS=2 RAUC_CA_SIGNER_DAYS=2 RAUC_CA_NO_PASSPHRASE=1 \ + "${REPO_ROOT}/dev/release/rauc-ca.sh" root "$out/rehearsal" >/dev/null +RAUC_CA_NAME="moose REHEARSAL release (check)" RAUC_CA_SIGNER_DAYS=2 \ + "${REPO_ROOT}/dev/release/rauc-ca.sh" signer "$out/rehearsal" signer >/dev/null + +# The slot image. +# Built as the caller, so root never owns the caller's Go build cache, into a +# directory the caller owns. +tooldir="$(mktemp -d)" +tool="$tooldir/slotbudget" +if [ -n "$CALLER" ]; then + chown "$CALLER" "$tooldir" + sudo -u "$CALLER" "$GO" build -C "$REPO_ROOT" -o "$tool" ./dev/cloud/slotbudget +else + "$GO" build -C "$REPO_ROOT" -o "$tool" ./dev/cloud/slotbudget +fi +"$tool" -extract "$out/bundle/rootfs.img" "$image" >/dev/null +rm -rf "$tooldir" + +version="$(tr -d '[:space:]' < "${REPO_ROOT}/VERSION")" +if [ -n "$CALLER" ]; then + build="$(sudo -u "$CALLER" git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)" +else + build="$(git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)" +fi + +t0=$(date +%s) +rauc_run "$out" ' + # The image'"'"'s own RAUC config and keyring, read back out of the slot. + unsquashfs -cat bundle/rootfs.img etc/rauc/system.conf > image-etc/system.conf + unsquashfs -cat bundle/rootfs.img etc/rauc/keyring.pem > image-etc/keyring.pem + unsquashfs -cat bundle/rootfs.img usr/lib/sysusers.d/moose-image-accounts.conf > image-etc/accounts.conf + compatible="$(sed -n "s/^compatible=//p" image-etc/system.conf | head -n1)" + [ -n "$compatible" ] || { echo "the slot'"'"'s /etc/rauc/system.conf names no compatible" >&2; exit 1; } + grep -qx "check-purpose=codesign" image-etc/system.conf || { echo "the slot'"'"'s system.conf does not ask for check-purpose=codesign" >&2; exit 1; } + cat > bundle/manifest.raucm < bundle-info.txt + cat bundle-info.txt + + echo "check 3: the image'"'"'s own config and keyring ('"$mode"')" + if [ "'"$mode"'" = "release" ]; then + if rauc --conf=$conf info moose-cloud.raucb >/dev/null 2>&1; then + echo "an image built for release ACCEPTS a bundle signed with the throwaway key; refusing" >&2; exit 1 + fi + echo "ok: the release keyring refuses the throwaway-signed bundle" + else + rauc --conf=$conf info moose-cloud.raucb >/dev/null + echo "ok: the image keyring accepts the bundle" + fi + + echo "check 4: a wrong key is refused" + if rauc --conf=$conf --keyring=wrong/root-ca.pem info moose-cloud.raucb >/dev/null 2>&1; then + echo "a CA that never signed this bundle accepted it" >&2; exit 1 + fi + echo "ok: the wrong key is refused" +' +echo "bundle container: $(( $(date +%s) - t0 )) s" + +echo "check 1: the slot carries the keyring this build staged (${mode})" +cmp -s "$out/image-etc/keyring.pem" "$staged" || { echo "the slot's /etc/rauc/keyring.pem is not the ${mode} keyring this build staged" >&2; exit 1; } +echo "ok: $(openssl x509 -in "$out/image-etc/keyring.pem" -noout -subject)" + +echo "check 6: the image's account ids match dev/os-lock/cloud-accounts.lock" +# A box keeps an account's uid and gid for life: its /etc/passwd is in the +# /etc upper layer, and its files on the state partition are owned by number. +# So an image whose packages gave an account another id would leave a box's +# files owned by the wrong account (BUILD.md # 1b, rule 3). The lock is edited +# by hand: a new account is added, a changed id is a bug to fix in the build. +awk '$1 == "g" { print "g", $2, $3 } $1 == "u" { split($3, id, ":"); print "u", $2, id[1], id[2] }' \ + "$out/image-etc/accounts.conf" | LC_ALL=C sort > "$out/image-etc/cloud-accounts.lock" +lock="${REPO_ROOT}/dev/os-lock/cloud-accounts.lock" +if ! diff -u <(grep -v '^#' "$lock" 2>/dev/null | LC_ALL=C sort) "$out/image-etc/cloud-accounts.lock" > "$out/accounts.diff"; then + echo "the image's accounts differ from dev/os-lock/cloud-accounts.lock ('-' the lock, '+' the image):" >&2 + cat "$out/accounts.diff" >&2 + echo "A new account: add its line to the lock. A changed id: the image must keep the old id, because every box keeps it." >&2 + exit 1 +fi +echo "ok: $(grep -c . "$out/image-etc/cloud-accounts.lock") accounts and groups, ids as locked" + +echo "check 5: rehearse the sign job's signing with a throwaway release CA" +mkdir -p "$out/rehearsal/image-etc" +sed 's/^path=.*/path=keyring.pem/' "$out/image-etc/system.conf" > "$out/rehearsal/image-etc/system.conf" +cp "$out/rehearsal/root-ca.pem" "$out/rehearsal/image-etc/keyring.pem" +RAUC_SIGNING_CERT="$(cat "$out/rehearsal/signer.pem")" RAUC_SIGNING_KEY="$(cat "$out/rehearsal/signer.key")" \ + MOOSE_RAUC_REHEARSAL=1 \ + "${REPO_ROOT}/dev/release/sign-bundle.sh" "$out/moose-cloud.raucb" "$out/rehearsal/image-etc" "$out/rehearsal/signed.raucb" +echo "ok: the rehearsal signed and verified the bundle" + +# Only what the next jobs need stays. +rm -rf "$out/wrong" "$out/rehearsal" "$out/throwaway-signer.key" "$out/throwaway-signer.pem" +rm -f "$out/bundle/rootfs.img" + +slot=$((1 << 30)) +img_bytes="$(sed -n 's/.*Size: .*(\([0-9]*\) bytes).*/\1/p' "$out/bundle-info.txt" | head -n1)" +b_bytes="$(stat -c %s "$out/moose-cloud.raucb")" +pct() { awk -v a="$1" -v b="$2" 'BEGIN { printf "%.1f%%", a * 100 / b }'; } +mb() { awk -v a="$1" 'BEGIN { printf "%.1f MB", a / 1e6 }'; } +{ + echo "### OS update bundle (#562)" + echo "" + echo "| | Bytes | Size | Share of the 1 GiB slot |" + echo "|---|---|---|---|" + echo "| Slot image in the bundle | ${img_bytes} | $(mb "$img_bytes") | $(pct "$img_bytes" "$slot") |" + echo "| Bundle (\`.raucb\`, verity, signed) | ${b_bytes} | $(mb "$b_bytes") | $(pct "$b_bytes" "$slot") |" + echo "" + echo "Keyring in the image: \`${mode}\`. Signed here with the throwaway key; a release re-signs it in the sign job." +} | tee -a "${GITHUB_STEP_SUMMARY:-/dev/null}" +[ -n "$CALLER" ] && chown -R "$CALLER":"$(id -gn "$CALLER")" "$out" 2>/dev/null || true +echo "bundle: $out/moose-cloud.raucb" diff --git a/dev/cloud/cloud-assertions.sh b/dev/cloud/cloud-assertions.sh index 6135b99e..d08428a5 100755 --- a/dev/cloud/cloud-assertions.sh +++ b/dev/cloud/cloud-assertions.sh @@ -151,7 +151,65 @@ fail() { # serial diag, then kill it and keep the run artifacts. exit 1 } +# Every file in the /etc upper layer must be one a Debian-major tidy-up knows +# (BUILD.md # 1b, rule 4): on the keep list (/usr/lib/moose/etc-keep.list, and +# this lane's own list in etc-keep.d/), an account file, a pinned file, or a +# link sshd's run state makes, which host-agent makes again at start. A file +# none of these covers would be lost at a major without anyone choosing that, +# so it fails the boot here first. +etc_upper_check() { + local up=/state/etc/upper pats p pat hit bad="" f + pats="$(cat /usr/lib/moose/etc-keep.list /usr/lib/moose/etc-keep.d/*.list 2>/dev/null | sed -e 's/#.*//' -e 's/[[:space:]]*$//' -e '/^$/d')" + pats="$pats +passwd +group +shadow +gshadow +passwd- +group- +shadow- +gshadow- +subuid- +subgid- +.pwd.lock +docker/daemon.json +login.defs +systemd/system/multi-user.target.wants/ssh.service +systemd/system/sshd.service +rc[0-6S].d/[SK][0-9][0-9]ssh" + while IFS= read -r p; do + [ -n "$p" ] || continue + hit="" + while IFS= read -r pat; do + # shellcheck disable=SC2053 + if [[ $p == $pat || $p == $pat/* ]]; then hit=1; break; fi + done <<<"$pats" + [ -n "$hit" ] || bad="$bad $p" + done < <(cd "$up" && find . -mindepth 1 ! -type d -printf '%P\n' 2>/dev/null) + [ -z "$bad" ] || fail "the /etc upper layer holds files no tidy-up rule covers (add them to /usr/lib/moose/etc-keep.list or give them a rule):$bad" + # Every account and group the image made is on the box, with the image's + # id (BUILD.md # 1b, rule 3; systemd-sysusers adds a missing one at boot). + f=/usr/lib/sysusers.d/moose-image-accounts.conf + [ -s "$f" ] || fail "no $f in the slot" + while read -r kind name id _; do + case "$kind" in + u) [ "$(getent passwd "$name" | cut -d: -f3)" = "${id%%:*}" ] || bad="$bad user:$name(${id%%:*}/$(getent passwd "$name" | cut -d: -f3))" ;; + g) [ "$(getent group "$name" | cut -d: -f3)" = "$id" ] || bad="$bad group:$name($id/$(getent group "$name" | cut -d: -f3))" ;; + esac + done < "$f" + [ -z "$bad" ] || fail "accounts the image made are missing on the box or have another id (image/box):$bad" + echo "cloud-assertions: /etc upper layer: $(cd "$up" && find . -mindepth 1 ! -type d | wc -l) files, all covered by a tidy-up rule; the image's $(grep -c '^[ug] ' "$f") accounts and groups are on the box with their ids" +} ok() { + etc_upper_check + # Last gate, every boot (#561): the slot is read-only, so anything that + # still writes to it fails. Catch it whether it failed a unit or only + # logged. Container logs are left out: an app's own read-only filesystem + # is not the slot. + failed_units="$(systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | tr '\n' ' ')" + [ -z "${failed_units// /}" ] || fail "failed units at the end of the boot: $failed_units" + rofs="$(journalctl -b --no-pager -o json 2>/dev/null | grep -v '"CONTAINER_NAME"' | grep -i 'read-only file system' | head -n 5 | cut -c1-600)" + [ -z "$rofs" ] || fail "something tried to write to the read-only slot: $rofs" emit "PASS" # Clean poweroff so the brain's SQLite writes (the persisted box-id) flush to # the qcow2 overlay before the harness boots the next scenario over it. --no-block @@ -162,6 +220,44 @@ ok() { echo "cloud-assertions: starting boot-proof checks (mode=${MODE})" +# The slot GRUB booted (#563). Every boot but the OS update boots runs on slot A; +# those two move the box between slots, one stage per boot, and keep the stage +# on the state partition. +BOOTED="$(sed -n 's/.*rauc\.slot=\([AB]\).*/\1/p' /proc/cmdline)" +OS_STATE_DIR=/var/lib/moose/test/os +os_stage() { cat "$OS_STATE_DIR/stage" 2>/dev/null || echo 1; } +set_os_stage() { mkdir -p "$OS_STATE_DIR" && echo "$1" > "$OS_STATE_DIR/stage" && sync; } +case "$MODE" in +os-update|os-revert) ;; +*) [ "$BOOTED" = A ] || fail "layout: booted slot '$BOOTED', want A (only the OS update boots leave slot A)" ;; +esac + +# The broken slot of the os-revert boot (#563). host-agent is kept from starting +# on it (a drop-in the first stage planted), which is the case where nothing but +# the image's own safety net can take the box back. So this stage checks only +# that, and waits: every other check of this script would fail on a box with no +# host-agent, which is the point of the scenario. moose-os-trial.timer (90 s in +# this image) must reboot the box; the next stage runs on slot A. +if [ "$MODE" = os-revert ] && [ "$(os_stage)" = 2 ]; then + [ "$BOOTED" = B ] || fail "os-revert: stage 2 should boot slot B, booted '$BOOTED' (did GRUB skip the new slot?)" + boot_env="$(journalctl -u moose-test-grubenv.service -b --no-pager -o cat 2>/dev/null | grep 'grubenv at boot' | tail -1)" + grep -qw B_TRY=1 <<<"$boot_env" || fail "os-revert: GRUB did not save B_TRY=1 for the trial boot of slot B (${boot_env:-no grubenv at boot line})" + [ -e /var/lib/moose/os-update/trial-B ] || fail "os-revert: no trial marker for slot B on its trial boot" + for _ in $(seq 1 20); do systemctl is-active -q host-agent.service && break; sleep 1; done + systemctl is-active -q host-agent.service && fail "os-revert: host-agent runs on the slot it was meant to be kept off" + systemctl list-timers --all --no-pager 2>/dev/null | grep -q moose-os-trial.timer || fail "os-revert: moose-os-trial.timer is not scheduled on the trial boot" + echo "cloud-assertions: os-revert: on slot B, host-agent cannot start, trial marker present; grubenv: $(grub-editenv /efi/grub/grubenv list 2>&1 | tr '\n' ' '); waiting for the safety net to reboot the box" + # The revert direction of a Debian-major tidy-up (BUILD.md # 1b, rule 4): + # record a major one above this slot's, as a newer major would have, so + # slot A must tidy the upper layer when the safety net takes the box back. + rec_major="$(( $( . /usr/lib/os-release && echo "${VERSION_ID%%.*}" ) + 1 ))" + echo "$rec_major" > /state/etc/.moose-debian-major && sync + echo "cloud-assertions: os-revert: recorded Debian $rec_major on the state partition before the revert" + set_os_stage 3 + sleep 300 + fail "os-revert: the safety net never rebooted the box off the broken slot. $boot_env. grubenv now: $(grub-editenv /efi/grub/grubenv list 2>&1 | tr '\n' ' ') timer: $(systemctl list-timers --all --no-pager 2>&1 | grep moose-os-trial) service: $(journalctl -u moose-os-trial.service -b --no-pager 2>&1 | tail -5 | tr '\n' ' ')" +fi + # --- 1. no control-plane unit has failed. # NOTE: we deliberately do NOT gate on `systemctl is-system-running == running`: # this script runs as a boot-transaction unit (WantedBy=multi-user.target), so @@ -175,26 +271,161 @@ for u in docker.service systemd-networkd.service host-agent.service moose-load-i grep -qx "$u" <<<"$failed" && fail "control-plane unit failed: $u (failed: $(tr '\n' ' ' <<<"$failed"))" done -# --- 1b. root grown to fill the provider disk. moose-grow-root.service runs -# systemd-repart at boot to extend the baked 8 GiB root partition to the whole -# disk, then runs systemd-growfs directly to grow the ext4 inside it (issue: a -# hosted box left on 8 GiB has docker image storage + the brain's SQLite store -# sharing that volume, so one app install fills it and 500s login). This QEMU -# boot-proof disk is fixed-size with no spare space, so both steps are a no-op -# here — but the unit must still complete cleanly, which proves systemd-repart -# and systemd-growfs are present in the lean image and the unit is wired. Real -# full-disk growth (partition AND filesystem) can only be proven on a live -# provider box (the cloud on-ramp), not this lane — a prior version of this unit -# passed this exact boot-proof while only growing the partition and leaving the -# filesystem at 8 GiB, because the growfs step was missing. -command -v systemd-repart >/dev/null 2>&1 || fail "systemd-repart missing from the lean image — moose-grow-root cannot grow the root disk" -[ -x /usr/lib/systemd/systemd-growfs ] || fail "systemd-growfs missing from the lean image — moose-grow-root cannot grow the root filesystem" -grow_state="$(systemctl is-active moose-grow-root.service 2>&1 || true)" -# Assert the unit actually completed (active, held by RemainAfterExit) — not merely -# "not failed". An inactive/unknown state means the .wants symlink was dropped or the -# unit was skipped, i.e. the grow never ran; that must fail the proof, not pass it. -[ "$grow_state" = active ] || fail "moose-grow-root.service did not complete successfully (state=$grow_state): $(journalctl -u moose-grow-root.service -b --no-pager 2>/dev/null | tail -10)" -echo "cloud-assertions: root-grow unit ok (state=$grow_state; systemd-repart + systemd-growfs present and wired — this lane cannot prove real growth, only that both steps ran)" +# --- 1b. the A/B layout (#561, BUILD.md # 1b). Every boot, both firmwares. +# The image carries 3 partitions (ESP 128 MiB, BIOS boot, slot A: a 1 GiB +# squashfs); the initramfs (state-setup, via the +# initramfs-tools hook) made slot B and the state partition at first boot, +# grew the state partition to fill the disk (the harness gives the box a disk +# far bigger than the image), and set up the /etc overlay, the pinned files and +# the bind mounts before systemd started. The slot is read-only. +layout_fail() { fail "layout: $*"; } +root_src="$(findmnt -no SOURCE / 2>/dev/null)" +root_disk="/dev/$(lsblk -no PKNAME "$root_src" 2>/dev/null | head -n1)" +[ -b "$root_disk" ] || layout_fail "cannot find the disk under / (source '$root_src')" +nparts="$(lsblk -nr -o TYPE "$root_disk" | grep -c '^part$')" +[ "$nparts" = 5 ] || layout_fail "$root_disk has $nparts partitions, want 5 (ESP, BIOS boot, slot A, slot B, state): $(lsblk -nr -o NAME,SIZE,PARTLABEL "$root_disk" | tr '\n' ';')" +for pair in "1:esp" "3:moose-slot-a" "4:moose-slot-b" "5:moose-state"; do + n="${pair%%:*}"; want="${pair#*:}" + got="$(lsblk -nr -o PARTN,PARTLABEL "$root_disk" | awk -v n="$n" '$1==n{print $2}')" + [ "$got" = "$want" ] || layout_fail "partition $n is labelled '$got', want '$want'" +done +esp_bytes="$(lsblk -bnr -o PARTN,SIZE "$root_disk" | awk '$1==1{print $2}')" +slot_a_bytes="$(lsblk -bnr -o PARTN,SIZE "$root_disk" | awk '$1==3{print $2}')" +slot_b_bytes="$(lsblk -bnr -o PARTN,SIZE "$root_disk" | awk '$1==4{print $2}')" +[ "$esp_bytes" = 134217728 ] || layout_fail "the ESP is $esp_bytes bytes, want 128 MiB" +[ "$slot_a_bytes" = 1073741824 ] && [ "$slot_b_bytes" = 1073741824 ] \ + || layout_fail "slots are A=$slot_a_bytes B=$slot_b_bytes bytes, want 1 GiB each" +if [ "$BOOTED" = B ]; then booted_uuid=21212121-2121-4121-8121-212121212121; booted_idx=1; else booted_uuid=20202020-2020-4020-8020-202020202020; booted_idx=0; fi +[ "$root_src" = "/dev/disk/by-partuuid/$booted_uuid" ] || [ "$(lsblk -no PARTUUID "$root_src")" = "$booted_uuid" ] \ + || layout_fail "/ is $root_src, not slot $BOOTED" +# Grown: the state partition reaches the end of the disk, and its ext4 fills it. +disk_bytes="$(lsblk -bdn -o SIZE "$root_disk")" +state_dev="$(findmnt -no SOURCE /state 2>/dev/null)" +[ -n "$state_dev" ] || layout_fail "/state is not mounted" +state_part_bytes="$(lsblk -bdn -o SIZE "$state_dev")" +state_fs_bytes="$(df -B1 --output=size /state | tail -n1 | tr -d ' ')" +# ESP 128 MiB + BIOS 1 MiB + two 1 GiB slots + GPT and alignment. +used_before_state=$(( (128 + 1 + 2048 + 4) * 1024 * 1024 )) +[ "$state_part_bytes" -ge $(( disk_bytes - used_before_state )) ] \ + || layout_fail "state partition is $state_part_bytes bytes on a $disk_bytes byte disk: it did not grow to the end" +# ext4 reserves its metadata and journal, so the filesystem is a little smaller. +[ "$state_fs_bytes" -ge $(( state_part_bytes * 95 / 100 )) ] \ + || layout_fail "state ext4 is $state_fs_bytes bytes in a $state_part_bytes byte partition: it was not grown" +echo "cloud-assertions: layout: $nparts partitions on $root_disk (${disk_bytes} bytes), ESP 128 MiB, slots 1 GiB each, state partition $state_dev ${state_part_bytes} bytes with ext4 ${state_fs_bytes} bytes" +# The slot is a read-only squashfs, and it is the slot GRUB chose. +findmnt -no OPTIONS / | tr ',' '\n' | grep -qx ro || layout_fail "/ is not mounted read-only: $(findmnt -no OPTIONS /)" +[ "$(findmnt -no FSTYPE /)" = squashfs ] || layout_fail "/ is $(findmnt -no FSTYPE /), want squashfs" +# A squashfs keeps file owners as the build saw them. If the build had mapped +# them wrong, sudo would be neither root's nor setuid. +[ "$(stat -c '%u %a' /usr/bin/sudo)" = "0 4755" ] || layout_fail "/usr/bin/sudo is $(stat -c '%u %a' /usr/bin/sudo), want root-owned 4755" +# The measured numbers for the disk budget (BUILD.md # 1b # Disk budget): the +# squashfs's own size from its superblock (bytes_used, offset 40), the ESP's +# use, and the state partition's use at this point of the boot. +slot_a_dev=/dev/disk/by-partuuid/20202020-2020-4020-8020-202020202020 +sq_bytes="$(dd if="$slot_a_dev" bs=96 count=1 2>/dev/null | od -An -t u8 -j 40 -N 8 | tr -d ' ')" +[ -n "$sq_bytes" ] && [ "$sq_bytes" -gt 0 ] || layout_fail "cannot read the squashfs size from slot A's superblock" +esp_used="$(df -B1 --output=used /efi | tail -n1 | tr -d ' ')" +state_used="$(df -B1 --output=used /state | tail -n1 | tr -d ' ')" +echo "cloud-assertions: layout: measured: squashfs ${sq_bytes} bytes ($(( sq_bytes * 1000 / slot_a_bytes / 10 )).$(( sq_bytes * 1000 / slot_a_bytes % 10 ))% of the slot), ESP used ${esp_used} of ${esp_bytes} bytes, state partition used ${state_used} of ${state_fs_bytes} bytes" +cmdline="$(cat /proc/cmdline)" +for w in "rauc.slot=$BOOTED" panic=10 ro psi=1 BOOT_IMAGE=/usr/lib/moose/boot/vmlinuz; do + grep -qw -- "$w" <<<"$cmdline" || layout_fail "kernel command line lacks '$w': $cmdline" +done +echo "cloud-assertions: layout: / is slot $BOOTED read-only, booted by GRUB from the slot's own kernel ($cmdline)" +# The slot's initramfs can find a provider's disk. Hetzner Cloud presents the +# boot disk as virtio-SCSI, which needs virtio_scsi and the SCSI disk driver +# sd_mod; without sd_mod no /dev/sda appears and the boot hangs in the +# initramfs (diagnosed once on a live Hetzner box, when mkosi built the +# initrd). Debian's initramfs-tools (MODULES=most) carries them; this lane +# boots virtio-blk, so only this check would notice them gone. +initrd_mods="$(lsinitramfs /usr/lib/moose/boot/initrd.img 2>/dev/null)" || layout_fail "lsinitramfs cannot read /usr/lib/moose/boot/initrd.img" +for m in sd_mod virtio_scsi virtio_blk squashfs overlay ext4; do + grep -qE "/${m}\.ko(\.[a-z]+)?$" <<<"$initrd_mods" || layout_fail "the slot's initramfs has no ${m} module" +done +echo "cloud-assertions: layout: the slot's initramfs carries sd_mod, virtio_scsi, virtio_blk, squashfs, overlay and ext4" +# /etc is an overlay with its upper layer on the state partition. +etc_opts="$(findmnt -no FSTYPE,OPTIONS /etc)" +grep -q '^overlay' <<<"$etc_opts" && grep -q 'upperdir=/state/etc/upper' <<<"$etc_opts" \ + || layout_fail "/etc is not the overlay on /state: $etc_opts" +# Every bind mount is the state partition. +state_devno="$(stat -c %d /state)" +for d in /home /var/lib/docker /var/lib/containerd /var/lib/moose /var/log /var/lib/systemd /var/lib/rauc /var/lib/sudo /srv/moose; do + [ "$(stat -c %d "$d")" = "$state_devno" ] || layout_fail "$d is not on the state partition ($(findmnt -no SOURCE,TARGET "$d" | tr '\n' ' '))" +done +# User files live under /srv/moose, as on the appliance; the databases have +# their own bind mounts and never sit under it (ENVIRONMENT.md # Storage +# (hosted)). +findmnt -no SOURCE /home | grep -q '\[/srv/moose/home\]$' || layout_fail "/home is not bound from the state partition's srv/moose/home: $(findmnt -no SOURCE /home)" +findmnt -no SOURCE /var/lib/moose | grep -q '\[/var/lib/moose\]$' || layout_fail "/var/lib/moose is not its own bind mount: $(findmnt -no SOURCE /var/lib/moose)" +findmnt -no FSTYPE /var/tmp | grep -qx tmpfs || layout_fail "/var/tmp is not a tmpfs" +echo "cloud-assertions: layout: /etc overlay and the bind mounts are on the state partition, /var/tmp is a tmpfs" +# The per-box state is on the state partition: the four pinned files (the +# userns-remap), the SSH host keys and machine-id, all in the upper layer. +up=/state/etc/upper +for f in docker/daemon.json subuid subgid login.defs; do + [ -s "$up/$f" ] || layout_fail "pinned file /etc/$f is not in the upper layer" +done +grep -q '"userns-remap": "moose-remap"' "$up/docker/daemon.json" || layout_fail "pinned daemon.json lost the remap" +grep -q '^moose-remap:' "$up/subuid" && grep -q '^moose-remap:' "$up/subgid" || layout_fail "pinned subuid/subgid lost the moose-remap range" +[ -e /state/etc/.moose-pinned ] || layout_fail "no pin marker on the state partition" +for _ in $(seq 1 30); do [ -s "$up/ssh/ssh_host_ed25519_key" ] && break; sleep 1; done +for k in ssh_host_ed25519_key ssh_host_ecdsa_key ssh_host_rsa_key; do + [ -s "$up/ssh/$k" ] || layout_fail "SSH host key $k is not on the state partition" +done +[ -s "$up/machine-id" ] && [ "$(cat "$up/machine-id")" = "$(cat /etc/machine-id)" ] \ + || layout_fail "machine-id is not on the state partition (upper '$(cat "$up/machine-id" 2>/dev/null)', /etc '$(cat /etc/machine-id)')" +echo "cloud-assertions: layout: pinned daemon.json/subuid/subgid/login.defs, SSH host keys and machine-id ($(cat /etc/machine-id)) are on the state partition" +# The bootloader side: RAUC reads the slot config and the grubenv, and the +# box reboots rather than waits in emergency or rescue mode. +rauc_out="$(rauc status 2>&1)" || layout_fail "rauc status failed: $(tail -n3 <<<"$rauc_out" | tr '\n' ' ')" +grep -qi "booted from: *rootfs.${booted_idx} (${BOOTED})" <<<"$rauc_out" || layout_fail "rauc does not see slot $BOOTED as booted: $(tr '\n' ' ' <<<"$rauc_out")" +# The keyring a bundle must chain to (#562) comes from the slot, never from the +# state partition: a new image's keyring must reach the box, so the box never +# writes it into the /etc upper layer. +[ -s /etc/rauc/keyring.pem ] || layout_fail "no /etc/rauc/keyring.pem: rauc install would refuse every bundle" +[ ! -e "$up/rauc/keyring.pem" ] || layout_fail "/etc/rauc/keyring.pem is in the /etc upper layer, so it no longer follows the image" +grep -qx 'check-purpose=codesign' /etc/rauc/system.conf || layout_fail "/etc/rauc/system.conf does not ask for check-purpose=codesign" +echo "cloud-assertions: layout: rauc keyring from the slot: $(openssl x509 -in /etc/rauc/keyring.pem -noout -subject 2>/dev/null || head -c 40 /etc/rauc/keyring.pem)" +# Read once, then match: `grub-editenv list | grep -q` under pipefail fails +# when grep exits before grub-editenv has written everything (run 37063254219). +grubenv_now="$(grub-editenv /efi/grub/grubenv list 2>&1)" +grep -qx "${BOOTED}_OK=1" <<<"$grubenv_now" || layout_fail "grubenv does not mark slot $BOOTED good: $(tr '\n' ' ' <<<"$grubenv_now")" +for u in emergency.service rescue.service; do + systemctl cat "$u" 2>/dev/null | grep -q 'systemctl --no-block reboot' || layout_fail "$u has no reboot drop-in" +done +# A slot that hangs (#486 point 3, BUILD.md # 1b # As built): systemd feeds a +# hardware watchdog, so a hung PID 1 or kernel resets the box. This lane is +# QEMU q35, like a Hetzner Cloud VM, so it has the ICH9 TCO watchdog +# (iTCO_wdt) that a real box has. +wd_dev="$(systemctl show -p WatchdogDevice --value)" +wd_sec="$(systemctl show -p RuntimeWatchdogUSec --value)" +[ -n "$wd_dev" ] && [ -e /sys/class/watchdog/watchdog0 ] \ + || layout_fail "no hardware watchdog in use (WatchdogDevice='$wd_dev', /sys/class/watchdog: $(ls /sys/class/watchdog 2>/dev/null | tr '\n' ' '))" +[ "$wd_sec" = 1min ] || layout_fail "RuntimeWatchdogUSec is '$wd_sec', want 1min" +# systemd logs this before journald runs, so it is in the kernel log (the +# probe of run 37358065628 found it there and not under _PID=1). +wd_log="$(dmesg 2>/dev/null; journalctl -k -b --no-pager -o cat 2>/dev/null; journalctl -b _PID=1 --no-pager -o cat 2>/dev/null)" +grep -q 'Using hardware watchdog' <<<"$wd_log" \ + || layout_fail "systemd does not feed the hardware watchdog (no 'Using hardware watchdog' in the boot's log)" +echo "cloud-assertions: layout: systemd feeds the hardware watchdog $wd_dev ($(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null)) with a $wd_sec timeout" +dmesg 2>/dev/null | grep -q 'moose-state: bind mounts done' || journalctl -k -b --no-pager 2>/dev/null | grep -q 'moose-state: bind mounts done' \ + || layout_fail "no 'moose-state: bind mounts done' in the kernel log" +# state-setup looked for the state partition on the boot disk only. +journalctl -k -b --no-pager 2>/dev/null | grep -q "moose-state: boot disk is $root_disk " \ + || dmesg 2>/dev/null | grep -q "moose-state: boot disk is $root_disk " \ + || layout_fail "state-setup did not report $root_disk as the boot disk" +[ "$(lsblk -no PKNAME "$state_dev")" = "$(basename "$root_disk")" ] || layout_fail "the state partition $state_dev is not on the boot disk $root_disk" +echo "cloud-assertions: layout: rauc sees slot $BOOTED booted, grubenv has ${BOOTED}_OK=1, emergency and rescue reboot" +boot_env="$(journalctl -u moose-test-grubenv.service -b --no-pager -o cat 2>/dev/null | grep 'grubenv at boot' | tail -1)" +echo "cloud-assertions: layout: $boot_env" +# GRUB saved the booted slot's try flag before it booted it (#575), under both +# firmwares. This is what skips a slot that dies before userspace. +grep -qw "${BOOTED}_TRY=1" <<<"$boot_env" || layout_fail "GRUB did not save ${BOOTED}_TRY=1 for the slot it booted (${boot_env:-no grubenv at boot line})" +# No firmware wrote the ESP (#575). OVMF without a writable VARS store saves +# its variables to an NvVars file there, and that write undid GRUB's. +mountpoint -q /efi || layout_fail "ESP not mounted at /efi, so the NvVars check would prove nothing" +[ ! -e /efi/NvVars ] || layout_fail "the firmware wrote /efi/NvVars: the harness gave OVMF no writable VARS store, and that write undoes GRUB's grubenv save" +echo "cloud-assertions: layout: GRUB saved ${BOOTED}_TRY=1 before booting slot $BOOTED; no NvVars on the ESP" # --- 1c. the baked host-agent carries a real build stamp (BUILD.md # Versioning: # "every build stamps two fields"). An unstamped build reports internal/version's @@ -261,6 +492,30 @@ for c in $want; do grep -qw "$c" <<<"$running" || fail "control-plane container '$c' not running after 120s (have: $running)" done +# --- 5a. the brain and UI that run are the ones the image baked (#566). The +# build records which control plane it baked, and the image ID each one loads +# as (dev/control-plane/bundle-record.sh): a build of this commit, or the last +# released pair pulled from ghcr by digest. Checking the RUNNING containers, not +# only the loaded images, is what makes this boot proof a proof of those bytes. +# Docker's image ID here is the sha256 of the image config, because the remap +# keeps Docker on its classic store (BUILD.md # 1). +cp_record=/usr/lib/moose/control-plane.env +[ -f "$cp_record" ] || fail "no ${cp_record}: the image does not say which control plane it baked" +baked_source="$(. "$cp_record"; echo "${MOOSE_BAKED_CP_SOURCE:-}")" +baked_version="$(. "$cp_record"; echo "${MOOSE_BAKED_CP_VERSION:-}")" +for pair in "moose-brain:$(. "$cp_record"; echo "${MOOSE_BAKED_BRAIN_ID:-}")" "moose-ui:$(. "$cp_record"; echo "${MOOSE_BAKED_UI_ID:-}")"; do + c="${pair%%:*}"; want_id="${pair#*:}" + [ -n "$want_id" ] || fail "${cp_record} names no image ID for ${c}" + img_id="$(docker image inspect -f '{{.Id}}' "${c}:dev" 2>/dev/null || true)" + run_id="$(docker inspect -f '{{.Image}}' "$c" 2>/dev/null || true)" + [ "$img_id" = "$want_id" ] || fail "baked ${c}:dev loaded as '${img_id}', but the image recorded ${want_id} (${baked_source} ${baked_version})" + [ "$run_id" = "$want_id" ] || fail "container ${c} runs image '${run_id}', not the baked ${want_id} (${baked_source} ${baked_version})" +done +echo "cloud-assertions: control plane baked: ${baked_source} ${baked_version}; moose-brain and moose-ui run the recorded images" +case "$baked_source" in + released) echo "layout: control plane baked from ghcr: $(. "$cp_record"; echo "${MOOSE_BAKED_BRAIN_REF} ${MOOSE_BAKED_UI_REF}")" ;; +esac + # --- 5b. container stdout is readable through journald by CONTAINER_NAME — the # EXACT query host-agent-real's per-app log tail runs # (internal/hostagent/journalsource: `journalctl CONTAINER_NAME=`). @@ -517,6 +772,7 @@ access) DASH_HOST="$(json_str "$SEED" box_id).onmoose.io" ;; update) DASH_HOST="$(json_str "$SEED" box_id).onmoose.io" ;; ssh) DASH_HOST="$(json_str "$SEED" box_id).onmoose.io" ;; remap|remap-reboot) DASH_HOST="$(json_str "$SEED" box_id).onmoose.io" ;; +os-update|os-revert) DASH_HOST="$(json_str "$SEED" box_id).onmoose.io" ;; esac echo "cloud-assertions: probing control plane at Host=$DASH_HOST (mode=$MODE)" @@ -1273,7 +1529,7 @@ ssh) echo "cloud-assertions: enabling SSH with no key refused (422), :22 still closed" # --- 4. add a key and turn SSH on, in one save. - KEYFILE=/root/.moose-ssh-lane + KEYFILE=/run/moose-ssh-lane # /run, because the OS slot is read-only (BUILD.md # 1b) rm -f "$KEYFILE" "${KEYFILE}.pub" ssh-keygen -t ed25519 -N '' -C 'moose-cloud-lane' -f "$KEYFILE" >/dev/null 2>&1 \ || fail "ssh: ssh-keygen failed (is openssh-client in the image?)" @@ -1925,7 +2181,7 @@ remap|remap-reboot) # checks on what came back. The ids the second boot needs are left in # REMAP_STATE on the overlay. Each boot signs in with its own single-use # owner assertion, so no session crosses the reboot. - REMAP_STATE=/var/lib/moose-remap-test/state + REMAP_STATE=/var/lib/moose/test/remap-state # on the state partition, so it survives the reboot [ -f "$SEED" ] || fail "remap: $MODE mode but $SEED absent (seed materializer did not run?)" box_id="$(json_str "$SEED" box_id)" [ -n "$box_id" ] || fail "remap: could not read box_id from $SEED" @@ -2183,6 +2439,337 @@ remap|remap-reboot) echo "cloud-assertions: remap-reboot: every tier checked again after a real reboot of the same disk; the caps-tier app kept its data (token $rs_token)" fi ;; +os-update|os-revert) + # THE A/B OS UPDATE (#563, UPDATES.md # 1). host-agent reads an OS target + # from its update-target source, downloads the bundle, checks its sha256 + # against the target, has RAUC install it into slot B, switches in the + # window, and on the next boot keeps or gives up the new slot. The bundle + # is the boot-proof slot repacked with a host-agent one patch release up + # (dev/cloud/test/build-os-test-bundle.sh), on a read-only second disk, and + # served to host-agent by a file server inside the guest. One stage per + # boot; the stage is on the state partition, so it survives the reboots. + # os-update: 1 (slot A) wrong digest refused, install ahead of the + # window, switch; 2 (slot B) marked good, everything intact. + # os-revert: 1 (slot A) install and switch, host-agent kept off slot B; + # 2 (slot B) handled at the top of this script; 3 (slot A) + # reverted on its own, everything intact, then slot B made + # active again with a kernel panic planted for it (#575); + # 4 (slot A) GRUB skipped slot B after it panicked. + [ -f "$SEED" ] || fail "$MODE mode but $SEED absent" + box_id="$(json_str "$SEED" box_id)" + apex="${box_id}.onmoose.io" + os_cred="$(tr -d '\r\n' < "${CREDENTIALS_DIRECTORY:-/nonexistent}/moose.os_test" 2>/dev/null || true)" + IFS=: read -r os_mode os_ver os_sum <<<"$os_cred" + [ -n "$os_ver" ] && [ -n "$os_sum" ] || fail "$MODE: moose.os_test credential missing or malformed ('$os_cred')" + base_ver="$(/usr/lib/moose/host-agent-real --version | awk '{print $2}')" + stage="$(os_stage)" + target_dir=/var/lib/moose/test-target + ha_log() { journalctl -u host-agent.service -b --no-pager -o short-unix 2>&1; } + wait_ha() { # PATTERN SECONDS WHAT + for _i in $(seq 1 "$2"); do ha_log | grep -q -- "$1" && return 0; sleep 1; done + fail "$MODE: $3 within $2 s: $(ha_log | grep -i 'os update\|update target' | tail -8 | cut -c1-400 | tr '\n' ' ')" + } + # The admin read of what host-agent decided (BRAIN_UI_PROTOCOL: admin-only). + os_read() { sed '1,/^\r*$/d' <<<"$(full_get /api/v1/system/update-target "$apex" "$session_cookie" 2>/dev/null || true)" | tr -d '\r'; } + os_field() { grep -o "\"os\":{.*" <<<"$1" | json_str_of "$(cat)" "$2"; } + rauc_slot_version() { rauc status --detailed --output-format=json 2>/dev/null | grep -o "\"bootname\":\"$1\"[^]]*" | grep -o '"version":"[^"]*"' | head -1 | cut -d'"' -f4; } + grubvar() { grub-editenv /efi/grub/grubenv list | sed -n "s/^$1=//p"; } + boot_gate() { + local fu ro + fu="$(systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | tr '\n' ' ')" + [ -z "${fu// /}" ] || fail "$MODE: failed units before the switch: $fu" + ro="$(journalctl -b --no-pager -o json 2>/dev/null | grep -v '"CONTAINER_NAME"' | grep -i 'read-only file system' | head -n 3 | cut -c1-400)" + [ -z "$ro" ] || fail "$MODE: something tried to write to the read-only slot: $ro" + } + write_os_target() { # SHA256 WINDOW + printf '{"version":"os-test","window":"%s","os":[{"version":"%s","bundle_url":"http://127.0.0.1:5001/bundles/os-test.raucb","bundle_sha256":"%s"}]}\n' \ + "$2" "$os_ver" "$1" > "$target_dir/target.json" + } + facts() { # the per-box state an OS update must not touch + echo "shadow=$(grep "^${owner}:" /etc/shadow | cut -d: -f1-2)" + echo "hostkey=$(ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub 2>/dev/null | awk '{print $2}')" + echo "machineid=$(cat /etc/machine-id)" + echo "data=$(cat "/home/${owner}/os-update-data.txt" 2>/dev/null)" + echo "groups=$(id -nG "$owner" 2>/dev/null | tr ' ' '\n' | sort | tr '\n' ' ')" + } + + # 4. os-revert, last stage: slot B died before userspace (#575). The + # test initramfs hook crashed the kernel on slot B (dev/cloud/test/ + # bootstrap.sh), panic=10 rebooted, and GRUB skipped slot B because + # it had saved B_TRY=1 before booting it. Without that save, slot B + # would boot again and panic again for ever, and no verdict would come. + if [ "$MODE" = os-revert ] && [ "$stage" = 4 ]; then + [ "$BOOTED" = A ] || fail "os-revert: stage 4 booted slot '$BOOTED', want A: GRUB did not skip slot B after it panicked" + [ -e /state/moose-test/panicked-slot-B ] || fail "os-revert: no panic note from slot B, so slot B never ran its initramfs: GRUB did not boot it" + [ "$(grubvar B_OK)" = 1 ] && [ "$(grubvar B_TRY)" = 1 ] \ + || fail "os-revert: after the panic the grubenv should still have B_OK=1 B_TRY=1: $(grub-editenv /efi/grub/grubenv list | tr '\n' ' ')" + rm -f /state/moose-test/panic-slot-B /state/moose-test/panicked-slot-B + echo "cloud-assertions: os-revert: PANIC SKIP OK (slot B panicked in its initramfs, panic=10 rebooted, GRUB skipped slot B with B_TRY=1 and booted slot A)" + boot_gate + ok + fi + + if [ "$stage" = 1 ]; then + [ "$BOOTED" = A ] || fail "$MODE: the first boot is on slot '$BOOTED', want A" + sso_token="$(tr -d '\r\n' < "${CREDENTIALS_DIRECTORY:-/nonexistent}/moose.sso_token" 2>/dev/null || true)" + [ -n "$sso_token" ] || fail "$MODE: moose.sso_token credential missing" + sso_resp="$(full_get "/_moose/sso?token=${sso_token}" "$apex" 2>/dev/null || true)" + grep -q ' 303' <<<"$(status_of "$sso_resp")" || fail "$MODE: SSO landing did not 303: status='$(status_of "$sso_resp")'" + session_cookie="$(cookie_val "$sso_resp" moose_session)" + [ -n "$session_cookie" ] || fail "$MODE: no moose_session cookie from the SSO landing" + owner="$(json_str_of "$(full_get /api/v1/me "$apex" "$session_cookie" 2>/dev/null || true)" username)" + [ -n "$owner" ] && id -u "$owner" >/dev/null 2>&1 || fail "$MODE: the SSO owner '$owner' has no host account" + if :; then + # SSH on for the owner, so the faked major below must bring sshd + # back on slot B (BUILD.md # 1b, rule 4: the tidy-up keeps the + # drop-in, drops the enable links, and host-agent enables sshd once). + # A known password first, for the elevation gate, as the ssh boot + # does (harness setup, not a product path). + printf '%s:%s\n' "$owner" 'moose-cloud-lane-owner-pw' | chpasswd || fail "$MODE: could not set a known password for '$owner'" + el="$(full_send POST /api/v1/auth/elevate "$apex" "$session_cookie" '{"password":"moose-cloud-lane-owner-pw"}' 2>/dev/null)" + grep -q ' 200' <<<"$(status_of "$el")" || fail "$MODE: elevate as the owner failed: status='$(status_of "$el")'" + rm -f /run/moose-os-update-key /run/moose-os-update-key.pub + ssh-keygen -t ed25519 -N '' -C 'moose-os-update' -f /run/moose-os-update-key >/dev/null 2>&1 || fail "$MODE: ssh-keygen failed" + on="$(full_send PUT /api/v1/me/ssh "$apex" "$session_cookie" \ + "{\"enabled\":true,\"keys\":[{\"public_key\":\"$(tr -d '\n' < /run/moose-os-update-key.pub)\",\"label\":\"os-update\"}]}" 2>/dev/null)" + grep -q ' 200' <<<"$(status_of "$on")" || fail "$MODE: turning SSH on for the owner failed: status='$(status_of "$on")'" + [ "$(systemctl is-enabled ssh.service 2>/dev/null)" = enabled ] || fail "$MODE: ssh.service is not enabled after the owner turned SSH on" + if [ "$MODE" = os-revert ]; then + # An admin turns sshd off by hand while the owner keeps SSH on + # in the drop-in. The tidy-up on the way back must not turn it + # on again (no marker for host-agent). + systemctl disable --now ssh.service >/dev/null 2>&1 || fail "$MODE: could not turn sshd off by hand" + [ "$(systemctl is-enabled ssh.service 2>/dev/null)" != enabled ] || fail "$MODE: ssh.service still enabled after disable" + echo "cloud-assertions: $MODE: SSH on for '$owner' in the drop-in, sshd turned off by hand before the faked major" + else + echo "cloud-assertions: $MODE: SSH on for '$owner' (ssh.service enabled) before the faked major" + fi + fi + head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' > "/home/${owner}/os-update-data.txt" + chown "$owner" "/home/${owner}/os-update-data.txt" + mkdir -p "$OS_STATE_DIR" && chmod 700 "$OS_STATE_DIR" + app_id="" + if [ "$MODE" = os-update ] && [ "$os_mode" = full ]; then + # An app, so "apps intact" means a container that comes back. + resp="$(full_send POST /api/v1/apps "$apex" "$session_cookie" '{"manifest_id":"whoami","scope":"personal"}' 2>/dev/null)" + job="$(json_str_of "$resp" job_id)" + [ -n "$job" ] || fail "$MODE: install whoami did not start: status='$(status_of "$resp")'" + st=""; jr="" + for _i in $(seq 1 300); do + jr="$(full_get "/api/v1/jobs/${job}" "$apex" "$session_cookie" 2>/dev/null || true)" + st="$(json_str_of "$jr" status)" + case "$st" in completed|failed|cancelled|stalled) break ;; esac + sleep 1 + done + [ "$st" = completed ] || fail "$MODE: install whoami ended '$st'" + app_id="$(json_str_of "$jr" instance_id)" + docker ps --filter "label=moose.instance_id=${app_id}" --filter status=running --format '{{.Names}}' | grep -q . \ + || fail "$MODE: whoami has no running container" + fi + printf 'session_cookie=%q\nowner=%q\napp_id=%q\nbase_ver=%q\n' "$session_cookie" "$owner" "$app_id" "$base_ver" > "$OS_STATE_DIR/before" + facts > "$OS_STATE_DIR/facts" + echo "cloud-assertions: $MODE: owner '$owner' signed in, data written${app_id:+, whoami installed ($app_id)}; running OS $base_ver on slot A" + + # The bundle disk, the in-guest source, and host-agent pointed at it. + mkdir -p "$target_dir/bundles" + bdev="$(blkid -L moose-os-test 2>/dev/null || true)" + [ -n "$bdev" ] || fail "$MODE: no disk labelled moose-os-test (the harness's bundle disk)" + mount -o ro "$bdev" "$target_dir/bundles" || fail "$MODE: cannot mount the bundle disk $bdev" + [ "$(sha256sum "$target_dir/bundles/os-test.raucb" | cut -d' ' -f1)" = "$os_sum" ] || fail "$MODE: the bundle on the disk is not the one the harness named" + mkdir -p /etc/systemd/system/host-agent.service.d + cat > /etc/systemd/system/host-agent.service.d/30-os-update-test.conf <<'UNIT' +[Service] +Environment=MOOSE_UPDATE_OS_URL_PREFIX=http://127.0.0.1:5001/ +UNIT + if [ "$MODE" = os-revert ]; then + # The broken slot: host-agent is kept from starting on slot B + # only, so the old slot still runs it after the revert. + cat > /etc/systemd/system/host-agent.service.d/90-os-revert-test.conf <<'UNIT' +[Service] +ExecStartPre=/bin/sh -c 'if grep -qw rauc.slot=B /proc/cmdline; then echo "os-revert test: host-agent is kept off slot B"; exit 1; fi' +UNIT + fi + systemctl daemon-reload + caddy_image="$(docker inspect -f '{{.Config.Image}}' moose-caddy 2>/dev/null || true)" + docker rm -f moose-test-target >/dev/null 2>&1 || true + # A shut window: twelve hours from now, one minute wide. + shut="$(printf '%02d:00-%02d:01' $(( (10#$(date +%H) + 12) % 24 )) $(( (10#$(date +%H) + 12) % 24 )))" + write_os_target "$(printf '%064d' 0)" "$shut" + # --restart: the box reads its target again after each reboot, and the + # last stage checks what it decided then. + docker run -d --restart unless-stopped --name moose-test-target -p 127.0.0.1:5001:80 -v "$target_dir":/srv:ro "$caddy_image" \ + caddy file-server --root /srv --listen :80 >/dev/null 2>&1 || fail "$MODE: could not start the in-guest file server" + for _i in $(seq 1 60); do grep -q ' 200' <<<"$(http_status_addr 127.0.0.1 5001 /target.json 2>/dev/null)" && break; sleep 1; done + + if [ "$MODE" = os-update ]; then + # 1a. A WRONG DIGEST. The signature would pass; the digest the + # target names does not, so RAUC never sees the bundle. + systemctl restart host-agent.service || fail "$MODE: could not restart host-agent" + wait_ha "update target names; refusing it" 180 "host-agent did not refuse a bundle whose digest the target does not name" + [ -z "$(rauc_slot_version B)" ] || fail "$MODE: slot B holds '$(rauc_slot_version B)' after a refused download" + # host-agent logs the refusal just before it writes the state, so + # one read can still see 'installing'. Poll, as 1b does. + st=""; for _i in $(seq 1 30); do st="$(os_field "$(os_read)" state)"; [ "$st" = failed ] && break; sleep 1; done + [ "$st" = failed ] || fail "$MODE: the update-target read says os.state '$st' after a wrong digest, want failed" + echo "cloud-assertions: os-update: WRONG DIGEST OK (refused before RAUC saw it; slot B untouched; os.state=failed)" + + if [ "$os_mode" = refuse ]; then + # 1b-refuse. A run that publishes the OS: the image trusts only + # the release root, so the throwaway-signed bundle must be + # refused by RAUC even with the right digest. + write_os_target "$os_sum" "$shut" + systemctl restart host-agent.service + wait_ha 'install failed.*rauc install' 300 "RAUC did not refuse the throwaway-signed bundle" + [ -z "$(rauc_slot_version B)" ] || fail "$MODE: an image with the release keyring INSTALLED a throwaway-signed bundle" + echo "cloud-assertions: os-update: RELEASE KEYRING OK (the right digest, but RAUC refused the throwaway signature; slot B untouched)" + boot_gate + ok + fi + fi + + # 1b. INSTALL AHEAD OF THE WINDOW. The right digest and a shut + # window: the bundle goes into slot B, and the boot order stays. + write_os_target "$os_sum" "$shut" + systemctl restart host-agent.service + wait_ha "installed into the other slot" 300 "the bundle was not installed into slot B" + [ "$(rauc_slot_version B)" = "$os_ver" ] || fail "$MODE: slot B holds '$(rauc_slot_version B)', want $os_ver" + [ "$(grubvar ORDER)" = "A B" ] || fail "$MODE: the boot order moved outside the window: ORDER='$(grubvar ORDER)'" + for _i in $(seq 1 30); do [ "$(os_field "$(os_read)" state)" = installed ] && break; sleep 1; done + [ "$(os_field "$(os_read)" state)" = installed ] || fail "$MODE: os.state is '$(os_field "$(os_read)" state)', want installed" + t_dl="$(ha_log | grep 'downloading the bundle' | tail -1 | awk '{print $1}')" + t_got="$(ha_log | grep 'bundle downloaded and its digest matches' | tail -1 | awk '{print $1}')" + t_in="$(ha_log | grep 'installed into the other slot' | tail -1 | awk '{print $1}')" + echo "cloud-assertions: $MODE: INSTALL AHEAD OK (slot B holds $os_ver, ORDER still 'A B', os.state=installed); measured: download+digest $(awk -v a="$t_dl" -v b="$t_got" 'BEGIN{printf "%.1f", b-a}') s, rauc install $(awk -v a="$t_got" -v b="$t_in" 'BEGIN{printf "%.1f", b-a}') s, bundle $(stat -c %s "$target_dir/bundles/os-test.raucb") bytes" + + # 1c. THE SWITCH. An open window: host-agent puts slot B first and + # reboots. The next stage runs on slot B. + boot_gate + if [ "$MODE" = os-update ]; then + # A faked Debian major (BUILD.md # 1b, rule 4): the state partition + # says the box last ran Debian 12, so slot B must tidy the /etc + # upper layer before it mounts it. An admin's own edit goes to the + # attic; the owner, the password hash, the host keys, machine-id and + # the sudo membership must come through (stage 2 checks them). + echo 12 > /state/etc/.moose-debian-major + echo "# an admin's own edit (os-update boot)" > /etc/moose-test-hand-edit.conf + sync + fi + date +%s > "$OS_STATE_DIR/switch-at" + set_os_stage 2 + write_os_target "$os_sum" "00:00-23:59" + systemctl restart host-agent.service + echo "cloud-assertions: $MODE: window open; waiting for host-agent to switch to slot B and reboot" + sleep 900 + fail "$MODE: host-agent never switched slots and rebooted: $(ha_log | grep -i 'os update' | tail -6 | tr '\n' ' ')" + fi + + # The last stage: back on a slot that must be healthy and marked good. + # shellcheck disable=SC1090 + . "$OS_STATE_DIR/before" + if [ "$MODE" = os-update ]; then + [ "$stage" = 2 ] && [ "$BOOTED" = B ] || fail "os-update: stage $stage booted slot '$BOOTED', want stage 2 on slot B" + wait_ha "the new slot is healthy and marked good" 300 "host-agent did not mark the new slot good" + [ ! -e /var/lib/moose/os-update/trial-B ] || fail "os-update: the trial marker for slot B is still there" + [ "$(grubvar ORDER)" = "B A" ] && [ "$(grubvar B_OK)" = 1 ] && [ "$(grubvar B_TRY)" = 0 ] \ + || fail "os-update: grubenv after mark-good: $(grub-editenv /efi/grub/grubenv list | tr '\n' ' ')" + [ "$(/usr/lib/moose/host-agent-real --version | awk '{print $2}')" = "$os_ver" ] || fail "os-update: slot B's host-agent is not $os_ver" + want_outcome=good; want_state=current; want_ver="$os_ver" + t_good="$(ha_log | grep 'the new slot is healthy and marked good' | tail -1 | awk '{print $1}')" + slot_major="$( . /usr/lib/os-release && echo "${VERSION_ID%%.*}" )" + [ "$(cat /state/etc/.moose-debian-major 2>/dev/null)" = "$slot_major" ] \ + || fail "os-update: the state partition records Debian '$(cat /state/etc/.moose-debian-major 2>/dev/null)', want $slot_major after the tidy-up" + # dmesg first, as the layout checks read the moose-state lines: the + # journal's kernel log was empty here once (run 37374562648). + tidy_log="$(dmesg 2>/dev/null; journalctl -k -b --no-pager -o cat 2>/dev/null)" + grep -q "moose-state: tidied /etc for Debian 12 to $slot_major" <<<"$tidy_log" \ + || fail "os-update: slot B did not tidy the /etc upper layer for the faked major: $(grep 'moose-state' <<<"$tidy_log" | tail -5 | tr '\n' ' ')" + [ ! -e /etc/moose-test-hand-edit.conf ] || fail "os-update: the admin's own /etc edit survived the faked major" + attic_edit="$(ls -d /state/etc/attic/*-debian-12-to-"$slot_major"/moose-test-hand-edit.conf 2>/dev/null | head -1)" + [ -n "$attic_edit" ] || fail "os-update: the admin's own /etc edit is not in the attic: $(ls /state/etc/attic 2>&1 | tr '\n' ' ')" + # sshd comes back: the drop-in was kept, its enable links were not, and + # host-agent enabled the unit once because of the tidy-up marker. + grep -qE "^AllowUsers .*\b${owner}\b" /etc/ssh/sshd_config.d/moose-allowed.conf 2>/dev/null \ + || fail "os-update: the sshd drop-in does not name '$owner' after the faked major: $(cat /etc/ssh/sshd_config.d/moose-allowed.conf 2>&1 | tr '\n' ' ')" + for _i in $(seq 1 60); do [ "$(systemctl is-enabled ssh.service 2>/dev/null)" = enabled ] && break; sleep 1; done + [ "$(systemctl is-enabled ssh.service 2>/dev/null)" = enabled ] \ + || fail "os-update: ssh.service is '$(systemctl is-enabled ssh.service 2>&1)' after the faked major, want enabled: $(ha_log | grep -i 'sshd\|tidy' | tail -3 | tr '\n' ' ')" + ha_log | grep -q "sshd turned on after a Debian-major tidy-up" || fail "os-update: host-agent did not log turning sshd on after the tidy-up" + [ ! -e /state/etc/.moose-major-tidied ] || fail "os-update: host-agent did not remove the tidy-up marker" + echo "cloud-assertions: os-update: SSHD BACK OK (drop-in names '$owner', ssh.service enabled again by host-agent after the tidy-up, marker removed)" + echo "cloud-assertions: os-update: MAJOR TIDY OK (faked Debian 12 to $slot_major: the upper layer was rebuilt, the admin's edit is in $(dirname "$attic_edit"), $(grep -o 'kept [0-9]* files' <<<"$tidy_log" | tail -1))" + echo "cloud-assertions: os-update: SWITCH OK (booted slot B, marked good, grubenv ORDER='B A' B_OK=1 B_TRY=0); measured: switch to marked good $(awk -v a="$(cat "$OS_STATE_DIR/switch-at")" -v b="$t_good" 'BEGIN{printf "%.0f", b-a}') s, the reboot included" + note_pat="updated its system to $os_ver" + else + [ "$stage" = 3 ] && [ "$BOOTED" = A ] || fail "os-revert: stage $stage booted slot '$BOOTED', want stage 3 back on slot A" + wait_ha "the box went back to the old slot" 120 "host-agent did not record the revert" + # And it was the image's safety net that did it, the case under test. + wait_ha "the image's safety net rebooted the new slot" 10 "the revert was not made by moose-os-trial.timer" + [ ! -e /var/lib/moose/os-update/trial-B ] || fail "os-revert: the trial marker for slot B is still there" + [ "$(grubvar B_OK)" = 0 ] && [ "$(grubvar A_OK)" = 1 ] || fail "os-revert: grubenv after the revert: $(grub-editenv /efi/grub/grubenv list | tr '\n' ' ')" + [ "$(/usr/lib/moose/host-agent-real --version | awk '{print $2}')" = "$base_ver" ] || fail "os-revert: back on slot A, host-agent is not $base_ver" + want_outcome=reverted; want_state=held; want_ver="$base_ver" + # The revert direction of the tidy-up: slot B recorded a newer major, + # so slot A tidied the upper layer on the way back. + slot_major="$( . /usr/lib/os-release && echo "${VERSION_ID%%.*}" )" + [ "$(cat /state/etc/.moose-debian-major 2>/dev/null)" = "$slot_major" ] \ + || fail "os-revert: the state partition records Debian '$(cat /state/etc/.moose-debian-major 2>/dev/null)' back on slot A, want $slot_major" + tidy_log="$(dmesg 2>/dev/null; journalctl -k -b --no-pager -o cat 2>/dev/null)" + grep -q "moose-state: tidied /etc for Debian $(( slot_major + 1 )) to $slot_major: .* sshd was off" <<<"$tidy_log" \ + || fail "os-revert: slot A did not tidy the upper layer on the way back (or saw sshd on): $(grep 'moose-state' <<<"$tidy_log" | tail -4 | tr '\n' ' ')" + ls -d /state/etc/attic/*-debian-$(( slot_major + 1 ))-to-"$slot_major" >/dev/null 2>&1 || fail "os-revert: no attic entry for the tidy-up on the way back" + # sshd was off by hand, so no marker, and host-agent left it off. + [ ! -e /state/etc/.moose-major-tidied ] || fail "os-revert: the tidy-up left the sshd marker although sshd was off" + grep -qE "^AllowUsers .*\b${owner}\b" /etc/ssh/sshd_config.d/moose-allowed.conf 2>/dev/null || fail "os-revert: the sshd drop-in lost '$owner' on the way back" + sleep 5 + [ "$(systemctl is-enabled ssh.service 2>/dev/null)" != enabled ] || fail "os-revert: ssh.service is enabled again after the tidy-up, but the admin had turned it off" + echo "cloud-assertions: os-revert: MAJOR TIDY BACK OK (Debian $(( slot_major + 1 )) to $slot_major on slot A, attic entry made, sshd left off as the admin set it)" + echo "cloud-assertions: os-revert: REVERT OK (slot B never came up; back on slot A, B marked bad, A good); measured: switch to back on slot A $(( $(date +%s) - $(cat "$OS_STATE_DIR/switch-at") )) s, two reboots and the safety net included" + note_pat="did not work, so moose went back" + fi + # What the brain reports, and what the box kept. + me="" + for _i in $(seq 1 90); do me="$(status_of "$(full_get /api/v1/me "$apex" "$session_cookie" 2>/dev/null || true)")"; grep -q ' 200' <<<"$me" && break; sleep 1; done + grep -q ' 200' <<<"$me" || fail "$MODE: the owner's session does not work after the OS move (status '$me')" + ver_body="$(full_get /api/v1/system/version "$apex" "$session_cookie" 2>/dev/null || true)" + [ "$(json_str_of "$ver_body" os_version)" = "$want_ver" ] && [ "$(json_str_of "$ver_body" os_slot)" = "$BOOTED" ] \ + || fail "$MODE: /api/v1/system/version reports os_version '$(json_str_of "$ver_body" os_version)' os_slot '$(json_str_of "$ver_body" os_slot)', want $want_ver $BOOTED" + rd=""; for _i in $(seq 1 60); do rd="$(os_read)"; [ "$(os_field "$rd" state)" = "$want_state" ] && break; sleep 2; done + [ "$(os_field "$rd" state)" = "$want_state" ] && [ "$(os_field "$rd" outcome)" = "$want_outcome" ] \ + || fail "$MODE: the update-target read says os.state '$(os_field "$rd" state)' outcome '$(os_field "$rd" outcome)', want $want_state $want_outcome: $(grep -o '"os":{.*' <<<"$rd" | cut -c1-400)" + now_facts="$(facts)" + [ "$now_facts" = "$(cat "$OS_STATE_DIR/facts")" ] \ + || fail "$MODE: per-box state changed across the OS move: before [$(tr '\n' ' ' < "$OS_STATE_DIR/facts")] now [$(tr '\n' ' ' <<<"$now_facts")]" + if [ -n "$app_id" ]; then + for _i in $(seq 1 180); do docker ps --filter "label=moose.instance_id=${app_id}" --filter status=running --format '{{.Names}}' | grep -q . && break; sleep 1; done + docker ps --filter "label=moose.instance_id=${app_id}" --filter status=running --format '{{.Names}}' | grep -q . \ + || fail "$MODE: the whoami app did not come back after the OS move" + fi + echo "cloud-assertions: $MODE: INTACT OK (owner '$owner', session, password hash, SSH host key, machine-id, data file${app_id:+ and the whoami app} unchanged; os_version $want_ver on slot $BOOTED; os.state=$want_state, last outcome $want_outcome)" + nb="" + for _i in $(seq 1 150); do nb="$(full_get /api/v1/notifications "$apex" "$session_cookie" 2>/dev/null || true)"; grep -q "$note_pat" <<<"$nb" && break; sleep 2; done + grep -q "$note_pat" <<<"$nb" || fail "$MODE: no admin notification '$note_pat': $(sed '1,/^\r*$/d' <<<"$nb" | cut -c1-400)" + echo "cloud-assertions: $MODE: NOTIFY OK (admins were told: '$note_pat')" + if [ "$MODE" = os-revert ]; then + # 3b. A SLOT THAT DIES BEFORE USERSPACE (#575). Make slot B active + # again (RAUC sets B_OK=1 B_TRY=0 and puts it first) and plant + # a kernel panic for it. The next stage must come up on slot A. + # Slot A must be marked good first, or GRUB would have no slot + # to fall back to but the fallback rule. + for _i in $(seq 1 60); do [ "$(grubvar A_TRY)" = 0 ] && break; sleep 1; done + [ "$(grubvar A_OK)" = 1 ] && [ "$(grubvar A_TRY)" = 0 ] || fail "os-revert: slot A is not marked good before the panic test: $(grub-editenv /efi/grub/grubenv list | tr '\n' ' ')" + mkdir -p /state/moose-test + rm -f /state/moose-test/panicked-slot-B + touch /state/moose-test/panic-slot-B + rauc status mark-active other >/dev/null 2>&1 || fail "os-revert: rauc status mark-active other failed" + [ "$(grubvar ORDER)" = "B A" ] && [ "$(grubvar B_OK)" = 1 ] && [ "$(grubvar B_TRY)" = 0 ] \ + || fail "os-revert: after mark-active the grubenv is $(grub-editenv /efi/grub/grubenv list | tr '\n' ' ')" + set_os_stage 4 + echo "cloud-assertions: os-revert: slot B is active again with a kernel panic planted for it; rebooting" + systemctl reboot + sleep 600 + fail "os-revert: the box did not reboot into the panic test" + fi + ;; *) fail "unknown assert mode '$MODE'" ;; diff --git a/dev/cloud/expected-packages.txt b/dev/cloud/expected-packages.txt index 89253e7e..be00574a 100644 --- a/dev/cloud/expected-packages.txt +++ b/dev/cloud/expected-packages.txt @@ -28,14 +28,23 @@ # fix/decision is tracked there; until then it is part of the lean set, not a # leak. # -# grub-common + grub-pc-bin (and their transitive libs — dmsetup, gettext-base, -# libbrotli1, libdevmapper1.02.1, libefiboot1t64, libefivar1t64, libfreetype6, -# libfuse3-4, libpng16-16t64) are in the set because the image is dual-firmware -# (#277): it boots UEFI via systemd-boot AND legacy BIOS via GRUB -# (BiosBootloader=grub), so it comes up on providers that present either firmware -# (Hetzner CX/Intel boots legacy BIOS; CPX/AMD boots UEFI). grub is the BIOS -# bootloader only — the UEFI path stays systemd-boot, so the systemd-boot* entries -# below remain. +# grub-common + grub-pc-bin + grub-efi-amd64-bin (+ grub-efi-amd64-unsigned, and +# the transitive libs: dmsetup, gettext-base, libbrotli1, libdevmapper1.02.1, +# libefiboot1t64, libefivar1t64, libfreetype6, libfuse3-4, libpng16-16t64) are in +# the set because GRUB boots the A/B image under both firmwares (#561; #277): +# legacy BIOS (Hetzner CX) and UEFI (CPX), from one grub.cfg and grubenv. +# systemd-boot left the image with #561. +# +# rauc + rauc-service (#561) carry the slot config for the A/B update (#562, +# #563). They bring glib (libglib2.0-0t64, libffi8, libatomic1), json-glib, +# libnl-3, and libcurl3t64-gnutls with its TLS and HTTP stack (libgnutls30t64, +# libnettle8t64, libhogweed6t64, libp11-kit0, libtasn1-6, libidn2-0, +# libunistring5, libpsl5t64, libnghttp2-14, libnghttp3-9, libngtcp2-16, +# libngtcp2-crypto-gnutls8, librtmp1, libssh2-1t64, libldap2, libsasl2-2, +# libsasl2-modules-db). +# +# e2fsprogs (+ libext2fs2t64, libss2, #561) formats and checks the state +# partition at boot; the image had no mkfs.ext4 before. # # The concrete kernel-image package (linux-image--amd64) is deliberately # NOT listed: its name carries the ABI version, which changes on every kernel @@ -71,19 +80,22 @@ docker-ce-cli docker-compose-plugin dpkg dracut-install +e2fsprogs findutils gawk gcc-14-base gettext-base grep grub-common +grub-efi-amd64-bin +grub-efi-amd64-unsigned grub-pc-bin gzip hostname +init-system-helpers initramfs-tools initramfs-tools-bin initramfs-tools-core -init-system-helpers iproute2 iptables klibc-utils @@ -91,22 +103,24 @@ kmod less libacl1 libapparmor1 +libatomic1 libattr1 -libaudit1 libaudit-common +libaudit1 libblkid1 libbpf1 libbrotli1 libbsd0 libbz2-1.0 +libc-bin libc6 +libcap-ng0 libcap2 libcap2-bin -libcap-ng0 -libc-bin libcbor0.10 libcom-err2 libcrypt1 +libcurl3t64-gnutls libdb5.3t64 libdbus-1-3 libdebconfclient0 @@ -116,16 +130,24 @@ libefiboot1t64 libefivar1t64 libelf1t64 libexpat1 +libext2fs2t64 libfdisk1 +libffi8 libfido2-1 libfreetype6 libfuse3-4 libgcc-s1 +libglib2.0-0t64 libgmp10 +libgnutls30t64 libgssapi-krb5-2 +libhogweed6t64 +libidn2-0 libip4tc2 libip6tc2 libjansson4 +libjson-glib-1.0-0 +libjson-glib-1.0-common libk5crypto3 libkeyutils1 libklibc @@ -133,6 +155,7 @@ libkmod2 libkrb5-3 libkrb5support0 liblastlog2-2 +libldap2 liblzma5 libmd0 libmnl0 @@ -140,34 +163,50 @@ libmount1 libmpfr6 libncursesw6 libnetfilter-conntrack3 +libnettle8t64 libnfnetlink0 libnftables1 libnftnl11 -libpam0g +libnghttp2-14 +libnghttp3-9 +libngtcp2-16 +libngtcp2-crypto-gnutls8 +libnl-3-200 +libnl-genl-3-200 +libp11-kit0 libpam-modules libpam-modules-bin libpam-runtime +libpam0g libpcre2-8-0 libpng16-16t64 libproc2-0 +libpsl5t64 libreadline8t64 +librtmp1 +libsasl2-2 +libsasl2-modules-db libseccomp2 libselinux1 -libsemanage2 libsemanage-common +libsemanage2 libsepol2 libsigsegv2 libsmartcols1 libsqlite3-0 +libss2 +libssh2-1t64 libssl3t64 -libsystemd0 libsystemd-shared +libsystemd0 +libtasn1-6 libtext-charwidth-perl libtext-wrapi18n-perl libtinfo6 -libtirpc3t64 libtirpc-common +libtirpc3t64 libudev1 +libunistring5 libuuid1 libwrap0 libwtmpdb0 @@ -191,15 +230,14 @@ openssl-provider-legacy passwd perl-base procps +rauc +rauc-service readline-common runit-helper sed sensible-utils sudo systemd -systemd-boot -systemd-boot-efi -systemd-boot-tools systemd-repart systemd-sysv sysvinit-utils diff --git a/dev/cloud/mkosi.conf b/dev/cloud/mkosi.conf index 1b1bc192..45b15950 100644 --- a/dev/cloud/mkosi.conf +++ b/dev/cloud/mkosi.conf @@ -26,6 +26,12 @@ # disk image; the qcow2 + serial boot proof are the test lane (dev/cloud/test/). [Distribution] +# Every package version is locked (#560, BUILD.md # 1b # The OS package lock). +# bootstrap.sh passes --snapshot from dev/os-lock/debian-snapshot, so Debian and +# debian-security come from snapshot.debian.org at that moment, and the build +# fails if the resolved list differs from dev/os-lock/cloud-packages.lock. It is +# a CLI flag, not Snapshot= here, so the timestamp lives in one file under +# dev/os-lock/ that the bump workflow rewrites. Distribution=debian # Debian 13 "Trixie" — the locked product base (BUILD.md # 1), NOT the test # lane's bookworm. The QEMU test lane pins bookworm for its proven LUKS/TPM @@ -40,6 +46,15 @@ OutputDirectory=../../.dev/cloud # package set matches the checked-in expected-packages.txt exactly (the lean # guard, #238) without mounting the image — bootstrap.sh parses it post-build. ManifestFormat=json +# 512-byte sectors, so the 128 MiB ESP can be FAT32 (#561). systemd-repart always +# formats an ESP as FAT32, and for an image file it uses 4096-byte filesystem +# sectors unless told otherwise. FAT32 needs at least 65525 clusters, which 4K +# sectors only reach at about 260 MB; at 128 MiB the ESP got too few, and OVMF +# saw no filesystem on it and fell through to its shell (CI runs 36925538434 and +# 36927908164). With 512-byte sectors FAT32 needs only about 36 MB (mkosi's own +# ESP sizing says the same). The GPT already used 512-byte sectors, so nothing +# else moves. Hosted VM disks use 512-byte logical sectors. +SectorSize=512 [Build] CacheDirectory=../../.dev/cloud/cache @@ -61,60 +76,31 @@ ToolsTreeRelease=trixie # which a moving dev build silently re-enabled and pulled `nftables` in as # `iptables`' (docker-ce's hard dep) recommend, breaking the lean check (#237). # Pinning restores the leanness this profile shipped with (#218) and keeps the -# explicit-listing of the recommend-only deps below (systemd-boot-efi, tzdata) +# explicit-listing of the recommend-only deps below (tzdata) # the *only* way those reach the image — not a coincidence of defaults. WithRecommends=no -Bootable=yes -Bootloader=systemd-boot -# Dual-firmware boot (#277). systemd-boot is UEFI-only, so a UEFI-only image hangs -# at SeaBIOS's "Booting from Hard Disk" on a provider that boots the VM under -# legacy BIOS — the Hetzner CX (Intel) line does exactly this, while CPX (AMD) -# presents UEFI and boots the ESP fine. BiosBootloader=grub adds a GRUB BIOS boot -# path ALONGSIDE systemd-boot's UEFI path (mkosi treats the two independently): -# grub-bios-setup embeds core.img in the BIOS Boot Partition (mkosi.repart/ -# 05-bios.conf) and writes boot.img to the protective MBR, and mkosi appends a -# `grub_platform==pc` menuentry to the ESP grub.cfg that loads the SAME kernel + -# initrds + root=PARTUUID cmdline. The UEFI/systemd-boot path is untouched. -# Requires grub-common (grub-mkimage) + grub-pc-bin (i386-pc modules + Debian's -# /usr/lib/grub/i386-pc/grub-bios-setup) in Packages below; no grub-efi-*. -BiosBootloader=grub -# psi=1 is REQUIRED on the product image (BUILD.md # 1): Debian builds the -# kernel CONFIG_PSI=y but CONFIG_PSI_DEFAULT_DISABLED=y, so without it the -# `ram-pressure` health detector (HEALTH.md) reads zeros = a false all-clear. -# The test lane omits it; the product image must not (#205/C2 asserts PSI is -# live). console=ttyS0 last → /dev/console is the serial port the cloud -# hypervisor captures for headless boot diagnostics. rw: mutable Debian (A/B -# immutable is deferred, BUILD.md # 2). -# -# root=PARTUUID pins the root explicitly rather than relying on systemd's -# gpt-auto root discovery. gpt-auto needs the boot loader to set the -# LoaderDevicePartUUID EFI variable, which is not set in this firmware/boot path -# (the initrd's gpt-auto-generator logs "EFI loader partition unknown, exiting" -# and switch-root then fails into emergency mode). The PARTUUID is pinned in -# mkosi.repart/10-root.conf so it is deterministic. (The medium lane likewise -# passes an explicit root= — via rd.luks for its encrypted root.) -KernelCommandLine=console=tty0 console=ttyS0 psi=1 rw root=PARTUUID=20202020-2020-4020-8020-202020202020 -# Separate modules-initrd so the initrd finds a virtio root disk + mounts ext4 -# on a cloud hypervisor. The LUKS/dm-crypt cipher modules the test lane carries -# are intentionally absent — there is no LUKS root here (# Storage: no LUKS/TPM; -# encryption-at-rest is provider/KMS-keyed, out of this slice). +# GRUB on both firmwares (#561, BUILD.md # 1b # Boot; DECISIONS.md 2026-10-01). +# The A/B slot switch and its automatic revert are RAUC's GRUB backend, and they +# must work the same under UEFI (Hetzner CPX) and legacy BIOS (Hetzner CX, +# #277), so GRUB is the boot loader for both: x86_64-efi on the ESP, and +# i386-pc as core.img in the BIOS boot partition (mkosi.repart/05-bios.conf). +# Both read one grub.cfg (mkosi.extra/efi/grub/grub.cfg) and one grubenv on the +# ESP. grub.cfg boots the kernel and initramfs from inside the chosen slot, so +# no UKI: UnifiedKernelImages=no. # -# sd_mod is REQUIRED even though it is not a virtio.* match: a cloud hypervisor -# that presents the boot disk as virtio-SCSI (Hetzner Cloud) enumerates the LUN -# via virtio_scsi + scsi_mod (both pulled by virtio.*/deps), but without the SCSI -# *disk* upper-level driver (sd_mod) no /dev/sda block device is created, so the -# root=PARTUUID= device never appears and boot hangs the 90s device timeout into -# the initramfs emergency shell. `mkosi qemu` attaches the disk as virtio-blk -# (/dev/vda, driven by virtio_blk — a virtio.* match), so the boot-proof lane -# never exercises the SCSI path; the gap only surfaces on a real virtio-scsi -# cloud (diagnosed against a live Hetzner Cloud box). -KernelModulesInitrd=yes -KernelModulesInitrdInclude=virtio.* -KernelModulesInitrdInclude=sd_mod -KernelModulesInitrdInclude=ext4 -KernelModulesInitrdInclude=jbd2 -KernelModulesInitrdInclude=mbcache -KernelModulesInitrdInclude=crc32c.* +# Bootable=auto, not yes, and on purpose. mkosi installs GRUB either way. But +# with "yes" it also wants a kernel in /usr/lib/modules to copy to the ESP with +# its own initrd and menu entries, and it fails the build when there is none. +# The postinst moves the kernel into the slot's /usr/lib/moose/boot/ instead, +# so the ESP holds no kernel at all and stays small (mkosi.repart/00-esp.conf). +# The slot's own initramfs (Debian's initramfs-tools, built in the postinst) is +# the one that boots; mkosi's KernelModulesInitrd and KernelCommandLine +# settings would only shape the entries it no longer writes, so they are gone. +# The command line lives in grub.cfg. +Bootable=auto +Bootloader=grub +BiosBootloader=grub +UnifiedKernelImages=no # The lean cloud-VM minimum (ENVIRONMENT.md # Assumption inventory). Explicitly # NOT here (the cuts — kept out by bootstrap.sh's exact-match guard against expected-packages.txt): @@ -126,33 +112,28 @@ Packages= linux-image-amd64 systemd systemd-sysv - systemd-boot -# The actual EFI executables (/usr/lib/systemd/boot/efi/systemd-bootx64.efi) that -# mkosi's Bootloader=systemd-boot installs to the ESP. On trixie these live in a -# separate systemd-boot-efi package that `systemd-boot` only *Recommends*, and -# we pin recommends off (WithRecommends=no above) — so without listing it -# explicitly the build fails with "a systemd-boot binary was not found". (The medium lane is -# bookworm, whose systemd-boot packaging bundles the binary, so it never hit this.) - systemd-boot-efi -# GRUB for the BIOS boot path (#277; BiosBootloader=grub above). grub-pc-bin -# carries the i386-pc target modules AND grub-bios-setup (Debian installs it at -# /usr/lib/grub/i386-pc/grub-bios-setup, which is exactly where mkosi looks for -# it); grub-common carries grub-mkimage, which builds core.img. Deliberately NO -# grub-efi-* — the UEFI path stays systemd-boot, so nothing about it changes. -# These two pull a small set of transitive libs (libfreetype6, libpng16-16t64, -# libfuse3-4, libdevmapper1.02.1, libefi{boot,var}1t64, libbrotli1, dmsetup, -# gettext-base) — all captured in expected-packages.txt so the lean check stays -# exact. +# GRUB for both firmwares (#561; Bootloader=grub + BiosBootloader=grub above). +# grub-common carries grub-mkimage and grub-editenv (RAUC's GRUB backend calls +# it); grub-pc-bin the i386-pc modules and grub-bios-setup (Debian installs it +# at /usr/lib/grub/i386-pc/grub-bios-setup, where mkosi looks); grub-efi-amd64-bin +# the x86_64-efi modules. systemd-boot left the image with #561. grub-common grub-pc-bin -# systemd-repart, run at boot by moose-grow-root.service to grow the baked 8 GiB -# root partition + ext4 to fill the (much larger) provider disk. On trixie the -# systemd-repart binary is split out of the `systemd` package into its own -# `systemd-repart` package, so the lean image must depend on it explicitly (it -# pulls only libfdisk1 on top of what systemd already brings — no libcryptsetup, -# so the appliance's cryptsetup cut is preserved). Without the grow, a provisioned -# box runs on 8 GiB and one app install fills the shared docker+SQLite volume and -# 500s login (ENVIRONMENT.md # Storage). + grub-efi-amd64-bin +# RAUC with its slot config (/etc/rauc/system.conf). Debian splits it in two: +# `rauc` is the client, and `rauc-service` the D-Bus daemon that `rauc status` +# and `rauc install` talk to (the spike found this, ab-update-engine-spike.md). +# Nothing installs a bundle yet (#562, #563). + rauc + rauc-service +# mkfs.ext4 and e2fsck for the state partition: systemd-repart formats it at +# first boot, and state-setup checks it before every mount. The image had no +# e2fsprogs before #561, because the build host makes the root's ext4. + e2fsprogs +# systemd-repart, run in the initramfs on every boot (/usr/lib/moose/state-setup) +# against /usr/lib/repart.d/: it makes slot B and the state partition at first +# boot and grows the state partition when the provider disk grew. On trixie the +# binary is its own package. systemd-repart ca-certificates dbus @@ -205,6 +186,8 @@ Packages= # repo (BUILD.md # Docker package source), wired in via the auto-detected # mkosi.pkgmngr/ tree that bootstrap.sh populates with Docker's sources.list + # signing key (trixie pocket). Build-time repo only; the VM never apt-installs. +# The exact versions are pinned in dev/os-lock/third-party.lock (#560); keep this +# list and that file in step (dev/os-lock/oslock's test checks both ways). docker-ce docker-ce-cli containerd.io @@ -237,24 +220,20 @@ ExtraTrees=mkosi.extra # firstboot) is dev/cloud/mkosi.postinst.chroot, auto-detected for both lanes. ExtraTrees=mkosi.extra.wiring -# mkosi.repart/ defines a minimal unencrypted ESP + ext4 root (no LUKS/TPM — -# # Storage: encryption-at-rest is provider/KMS-keyed). It is REQUIRED, not -# optional: with no repart definitions systemd-repart grows the root to fill the -# whole build disk (a ~1 TiB root → ~17 GiB of ext4 inode tables written at build -# time → "No space left on device"). The root is fixed at 8 GiB (sparse — the raw -# only consumes written blocks; GrowFileSystem fills the partition) to leave first -# boot's `docker load` of the baked control-plane images runtime headroom, same -# sizing as the medium lane. The test lane additionally encrypts its root (LUKS); -# this profile does not. (C1b shipped no repart on an untested assumption that the -# mkosi default was a sane bounded layout; #205/C2's first real build corrected it.) +# The A/B layout (#561, BUILD.md # 1b # Partition layout). mkosi.repart/ is the +# BUILD-TIME layout: the ESP (128 MiB), the BIOS boot partition (1 MiB) and +# slot A, a fixed 1 GiB read-only squashfs (xz), unencrypted (# Storage: +# encryption-at-rest is provider/KMS-keyed). It is REQUIRED, not optional: with +# no repart definitions systemd-repart grows the root to fill the whole build +# disk. The build fails when the squashfs fills more than 60% of the slot +# (dev/cloud/slotbudget, run by bootstrap.sh and test/bootstrap.sh). # -# That 8 GiB is only the BUILD-TIME size. On a real provider VM the disk is far -# larger (e.g. 80 GB), so the image is grown to fill it at BOOT: moose-grow-root.service -# runs systemd-repart against the RUNTIME definition in mkosi.extra/usr/lib/repart.d/, -# which extends the root partition to the whole disk and online-grows the ext4 -# (ENVIRONMENT.md # Storage). Keeping the two repart definitions separate is -# deliberate: the build-time one pins 8 GiB so the shipped raw is small; the -# runtime one has no cap so the booted box uses all of the disk. +# The RUNTIME layout is mkosi.extra/usr/lib/repart.d/. systemd-repart reads it +# in the initramfs on every boot (/usr/lib/moose/state-setup): it adds slot B +# (1 GiB, empty) and the state partition (the rest of the disk, ext4) at first +# boot, and grows the state partition when the provider disk grew. The root +# never grows; the state partition does, and everything the box writes lives +# there (ENVIRONMENT.md # Storage (hosted)). # Root password intentionally unset (mkosi leaves /etc/shadow with no password). # There is no interactive console login surface in hosted v1 (SSH off; operator diff --git a/dev/cloud/mkosi.extra/efi/grub/grub.cfg b/dev/cloud/mkosi.extra/efi/grub/grub.cfg new file mode 100644 index 00000000..fa78275a --- /dev/null +++ b/dev/cloud/mkosi.extra/efi/grub/grub.cfg @@ -0,0 +1,100 @@ +# moose A/B boot (BUILD.md # 1b # Boot). One config for UEFI and legacy BIOS: +# GRUB x86_64-efi (from the ESP) and GRUB i386-pc (core.img in the BIOS boot +# partition) both read this file and the grubenv beside it. +# +# Slot choice is RAUC's GRUB backend, close to RAUC's reference script +# (contrib/grub.conf): take the first slot in ORDER with _OK=1 and +# _TRY=0, and set its TRY to 1 before booting it. A booted slot that is +# healthy clears TRY with `rauc status mark-good` (#563). A slot that never +# gets there keeps TRY=1, so the next boot skips it. One attempt per update. +# +# Until #563 nothing marks a slot good, so slot A's TRY is set on one boot and +# reset by the fallback below on the next: the box always boots slot A. +# +# This file is the whole menu: the ESP holds no kernel, so mkosi adds no +# entries of its own (dev/cloud/mkosi.conf, Bootable=auto). +set timeout=0 +set default=0 + +insmod regexp +insmod loadenv +insmod test +# Each slot is a squashfs compressed with xz (dev/cloud/mkosi.repart/ +# 10-slot-a.conf). squash4 with xzio is how GRUB reads the kernel and the +# initramfs from inside it. The modules come from the ESP, under the prefix. +insmod part_gpt +insmod squash4 +insmod xzio + +# $root is the ESP here. Its disk is the boot disk, whatever the firmware calls +# it, so the slots are partitions 3 and 4 of that same disk. +regexp --set=1:moose_disk '^([^,]+),' "$root" + +set ORDER="A B" +set A_OK=0 +set B_OK=0 +set A_TRY=0 +set B_TRY=0 +load_env + +set moose_found=0 +for SLOT in $ORDER; do + if [ "$SLOT" == "A" ]; then + INDEX=0 + OK=$A_OK + TRY=$A_TRY + A_TRY=1 + fi + if [ "$SLOT" == "B" ]; then + INDEX=1 + OK=$B_OK + TRY=$B_TRY + B_TRY=1 + fi + if [ "$OK" -eq 1 -a "$TRY" -eq 0 ]; then + default=$INDEX + moose_found=1 + break + fi +done + +# No slot is left to try. RAUC's script falls to a rescue slot here; there is +# none, so boot the known-good slot rather than stop at a GRUB prompt that +# nobody will ever see. The known-good slot is the LAST good slot in ORDER: +# an install puts the new slot first and the slot it came from last. Only that +# slot's attempt is reset. A slot that failed keeps its TRY=1, so it is never +# booted again until a new install into it resets the flag (#563). Today slot +# B is empty (B_OK=0), so this falls back to slot A. +if [ "$moose_found" -eq 0 ]; then + moose_fallback="" + for SLOT in $ORDER; do + if [ "$SLOT" == "A" -a "$A_OK" -eq 1 ]; then moose_fallback=A; fi + if [ "$SLOT" == "B" -a "$B_OK" -eq 1 ]; then moose_fallback=B; fi + done + if [ "$moose_fallback" == "A" ]; then A_TRY=0; default=0; fi + if [ "$moose_fallback" == "B" ]; then B_TRY=0; default=1; fi +fi + +save_env A_TRY A_OK B_TRY B_OK ORDER + +# ro: a slot is never written by the box. psi=1 for the ram-pressure detector +# (BUILD.md # 1 # Kernel cmdline). panic=10: a kernel panic, or a failure in +# the initramfs, reboots after 10 s instead of waiting for ever (UPDATES.md # 1 +# "a slot that hangs must still revert"). fsck.mode=skip: the slot is a +# read-only squashfs with nothing to check, and the initramfs would otherwise +# try to fsck it and stop; state-setup checks the state partition itself +# (e2fsck -p) before it mounts it. console=ttyS0 last, so /dev/console is the +# serial port the provider captures. +set moose_cmdline="ro console=tty0 console=ttyS0 psi=1 panic=10 fsck.mode=skip" + +menuentry "moose slot A" { + set root=($moose_disk,gpt3) + linux /usr/lib/moose/boot/vmlinuz root=PARTUUID=20202020-2020-4020-8020-202020202020 $moose_cmdline rauc.slot=A + initrd /usr/lib/moose/boot/initrd.img +} + +menuentry "moose slot B" { + set root=($moose_disk,gpt4) + linux /usr/lib/moose/boot/vmlinuz root=PARTUUID=21212121-2121-4121-8121-212121212121 $moose_cmdline rauc.slot=B + initrd /usr/lib/moose/boot/initrd.img +} diff --git a/dev/cloud/mkosi.extra/etc/fstab b/dev/cloud/mkosi.extra/etc/fstab new file mode 100644 index 00000000..a3422cbf --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/fstab @@ -0,0 +1,10 @@ +# moose hosted A/B image (BUILD.md # 1b). The root is a read-only slot, and +# the state partition, the /etc overlay and the bind mounts are set up in the +# initramfs before systemd starts (/usr/lib/moose/state-setup), so none of +# them are here. +# +# The ESP, where GRUB and the grubenv live. RAUC reads and writes the grubenv. +PARTLABEL=esp /efi vfat umask=0077,nofail 0 2 +# /var/tmp has no home on the state partition: nothing in it has to outlive a +# reboot on this box, and the slot is read-only. +tmpfs /var/tmp tmpfs mode=1777,nosuid,nodev 0 0 diff --git a/dev/cloud/mkosi.extra/etc/initramfs-tools/scripts/local-bottom/moose-state b/dev/cloud/mkosi.extra/etc/initramfs-tools/scripts/local-bottom/moose-state new file mode 100755 index 00000000..5c8e5256 --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/initramfs-tools/scripts/local-bottom/moose-state @@ -0,0 +1,37 @@ +#!/bin/sh +# moose early state setup (BUILD.md # 1b). initramfs-tools runs this after it +# has mounted the booted slot read-only at ${rootmnt}, and before it hands +# over to systemd. The work is done by /usr/lib/moose/state-setup, which lives +# in the slot and runs chrooted into it, so this hook stays small and every +# slot carries its own setup. The mounts it makes under ${rootmnt} move with +# the root when initramfs-tools switches to it. +# +# A failure panics. With panic= on the kernel command line, initramfs-tools +# then reboots instead of waiting at a shell nobody sees (UPDATES.md # 1). +PREREQ="" +prereqs() { echo "$PREREQ"; } +case "$1" in + prereqs) prereqs; exit 0 ;; +esac + +. /scripts/functions + +# The setup needs the kernel's views and a writable /run inside the chroot. +# They are mounted fresh (not moved), and unmounted again before +# initramfs-tools moves its own /dev, /proc, /sys and /run into the root. +mount -t proc proc "${rootmnt}/proc" +mount -t sysfs sysfs "${rootmnt}/sys" +mount -t devtmpfs devtmpfs "${rootmnt}/dev" +mount -t tmpfs -o mode=0755 tmpfs "${rootmnt}/run" + +rc=0 +chroot "${rootmnt}" /usr/lib/moose/state-setup || rc=$? + +umount "${rootmnt}/run" +umount "${rootmnt}/dev" +umount "${rootmnt}/sys" +umount "${rootmnt}/proc" + +if [ "$rc" -ne 0 ]; then + panic "moose: state setup failed (rc=$rc), see the moose-state lines above" +fi diff --git a/dev/cloud/mkosi.extra/etc/rauc/system.conf b/dev/cloud/mkosi.extra/etc/rauc/system.conf new file mode 100644 index 00000000..ce1310ff --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/rauc/system.conf @@ -0,0 +1,39 @@ +# RAUC slot config for the hosted A/B image (BUILD.md # 1b). The bundle and the +# keyring are built (#562); host-agent installs, switches and marks good (#563, +# internal/hostagent/osupdate). +[system] +compatible=moose-hosted-x86_64 +bootloader=grub +grubenv=/efi/grub/grubenv +# On the state partition (a bind mount, BUILD.md # 1b), so RAUC's status +# survives a slot swap. +data-directory=/var/lib/rauc +bundle-formats=-plain +# An install writes the other slot and leaves the boot order alone. host-agent +# installs ahead of the update window and switches (`rauc status mark-active +# other`) only inside it (UPDATES.md # 1, #563), so a reboot for any other +# reason between the two never boots the new slot early. +activate-installed=false + +# The CA a bundle's signer must chain to (#562, BUILD.md # 1b # The bundle). +# keyring.pem is not in this tree: dev/cloud/stage-control-plane.sh stages it +# into the generated wiring tree. An image built to publish carries the +# release root (dev/release/rauc/release-ca.pem); every other build carries a +# throwaway root (dev/cloud/rauc.sh). A relative path is read next to this +# file. The signer carries the codeSigning purpose (dev/release/rauc-ca.sh); +# without check-purpose RAUC asks for the S/MIME purpose and refuses it. +[keyring] +path=keyring.pem +check-purpose=codesign + +# A slot holds a read-only squashfs (BUILD.md # 1b), which RAUC writes to the +# partition as it is, block for block: type=raw. +[slot.rootfs.0] +device=/dev/disk/by-partuuid/20202020-2020-4020-8020-202020202020 +type=raw +bootname=A + +[slot.rootfs.1] +device=/dev/disk/by-partuuid/21212121-2121-4121-8121-212121212121 +type=raw +bootname=B diff --git a/dev/cloud/mkosi.extra/etc/systemd/system.conf.d/10-moose-watchdog.conf b/dev/cloud/mkosi.extra/etc/systemd/system.conf.d/10-moose-watchdog.conf new file mode 100644 index 00000000..d948ede1 --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/systemd/system.conf.d/10-moose-watchdog.conf @@ -0,0 +1,7 @@ +# A slot that hangs must still revert (UPDATES.md # 1): systemd feeds the +# hardware watchdog, and a hung PID 1 or kernel gets a reset. Hetzner Cloud VMs +# (QEMU q35) have the ICH9 TCO watchdog, which the generic kernel's iTCO_wdt +# drives; the reset comes about twice this value after the last feed +# (BUILD.md # 1b # As built). The boot lane checks systemd uses it. +[Manager] +RuntimeWatchdogSec=60s diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/docker.service.d/10-moose-grow-root.conf b/dev/cloud/mkosi.extra/etc/systemd/system/docker.service.d/10-moose-grow-root.conf deleted file mode 100644 index cb2beaae..00000000 --- a/dev/cloud/mkosi.extra/etc/systemd/system/docker.service.d/10-moose-grow-root.conf +++ /dev/null @@ -1,9 +0,0 @@ -# Fail-closed on the boot-time root grow. moose-grow-root.service extends the -# baked 8 GiB root partition + ext4 to fill the whole provider disk before any -# write-heavy service starts. docker image storage shares that one volume with the -# brain's SQLite store, so if the grow failed the root is still 8 GiB and a single -# app install can fill it and 500 login. Requires= refuses to start docker on an -# un-grown root rather than run into that. (The grow unit's own Before=docker.service -# already orders the grow ahead of us; this only adds the hard dependency.) -[Unit] -Requires=moose-grow-root.service diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/emergency.service.d/10-moose-reboot.conf b/dev/cloud/mkosi.extra/etc/systemd/system/emergency.service.d/10-moose-reboot.conf new file mode 100644 index 00000000..86e63d26 --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/systemd/system/emergency.service.d/10-moose-reboot.conf @@ -0,0 +1,8 @@ +# A slot that hangs must still revert (UPDATES.md # 1). Nobody sits at a +# hosted box's console, and the root account is locked, so the stock +# emergency shell waits for ever and boot counting never gets a second boot. +# Reboot instead, after 10 s so the reason reaches the serial console. +[Service] +ExecStartPre= +ExecStart= +ExecStart=-/bin/sh -c 'echo "moose: emergency mode, rebooting in 10 s" > /dev/console; sleep 10; exec /usr/bin/systemctl --no-block reboot' diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/host-agent.service.d/10-moose-grow-root.conf b/dev/cloud/mkosi.extra/etc/systemd/system/host-agent.service.d/10-moose-grow-root.conf deleted file mode 100644 index 1177ea14..00000000 --- a/dev/cloud/mkosi.extra/etc/systemd/system/host-agent.service.d/10-moose-grow-root.conf +++ /dev/null @@ -1,12 +0,0 @@ -# Fail-closed on the boot-time root grow (hosted image only). host-agent launches -# the brain, which writes the SQLite store on the shared root volume. If -# moose-grow-root.service failed, the root is still the baked 8 GiB and a single -# app install can fill that shared docker+SQLite volume and 500 login — so refuse -# to start on an un-grown root rather than run into it. This lives in the tracked -# hosted ExtraTree (mkosi.extra/), NOT in the shared dist/systemd/host-agent.service -# — the appliance has no moose-grow-root.service, so the Requires= must be scoped to -# the hosted profile via a drop-in. moose-grow-root.service's own Before=host-agent.service -# supplies the ordering; this only adds the hard dependency. -[Unit] -Requires=moose-grow-root.service -After=moose-grow-root.service diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/moose-grow-root.service b/dev/cloud/mkosi.extra/etc/systemd/system/moose-grow-root.service deleted file mode 100644 index 4ce60363..00000000 --- a/dev/cloud/mkosi.extra/etc/systemd/system/moose-grow-root.service +++ /dev/null @@ -1,74 +0,0 @@ -# Hosted cloud image — grow the root partition + filesystem to fill the provider -# disk on boot. Runs systemd-repart against the definition in -# /usr/lib/repart.d/50-moose-grow-root.conf, which extends the baked 8 GiB root -# partition to the whole disk, then runs systemd-growfs directly to grow the ext4 -# inside it. Without this a provisioned box keeps the image's 8 GiB root on a much -# larger provider disk; docker image storage and the brain's SQLite store share -# that volume, so a single catalog-app install can fill it, and once full the -# brain's first store write (the SSO replay-nonce) fails and login 500s — the box -# looks offline though it is up. ENVIRONMENT.md # Storage. -# -# The filesystem grow is an explicit second step, not a side effect of -# GrowFileSystem=yes in the repart definition. systemd-repart only extends the -# partition and sets the GPT GROWFS attribute bit on it; actually growing the -# filesystem is normally left to systemd-growfs-root.service, which -# systemd-gpt-auto-generator wires up only when the kernel is booted with -# root=gpt-auto. Hetzner (and cloud providers generally) boot with an explicit -# root=PARTUUID=..., so that generator never engages and systemd-growfs-root.service -# never runs — verified live: a real provider box grew its partition to 75.8 GiB -# but its ext4 stayed at the baked 8 GiB, because nothing ever called growfs. This -# unit calls it directly instead of relying on that generator path. -# -# Ordered Before=docker.service and host-agent.service so the disk is already full -# before docker pulls any image layer and before host-agent launches the brain, the -# two things that write to the shared volume. ConditionVirtualization=!container so -# it is a no-op inside a container (no disk to repartition). The QEMU boot-proof -# disk is fixed-size with no spare space, so both steps are a harmless no-op there; -# real full-disk growth can only be exercised on a live provider box. -# -# Enabled by dev/cloud/mkosi.postinst.chroot (a .wants symlink) for both the lean -# production image and the boot-proof test lane that Include=..'s it. -# -# NB: the stock systemd-repart.service (shipped by the systemd-repart package, and -# statically enabled by it) already runs the very same /usr/lib/repart.d/*.conf at -# boot, so the partition grow effectively runs twice. That is harmless — -# systemd-repart is idempotent; a second pass on an already-grown disk is a no-op. -# This custom unit's ExecStart is redundant with the stock unit for that reason; -# it is kept so the ordering/gating and the conditional growfs call live in one -# place. Consolidating onto a stock-unit drop-in that only contributes the -# ordering/gating (and the growfs step) is a possible later cleanup, not required now. -[Unit] -Description=moose — grow the root partition and filesystem to fill the provider disk -Documentation=man:systemd-repart.service(8) -Documentation=man:systemd-growfs(8) -Documentation=https://github.com/onmoose/os/blob/main/docs/specs/ENVIRONMENT.md -ConditionVirtualization=!container -Before=docker.service host-agent.service - -[Service] -Type=oneshot -RemainAfterExit=yes -# 76 = no root block device found; 77 = no GPT partition table. On a boot where -# there is simply nothing to grow, systemd-repart still succeeds (exit 0); these -# codes only cover the "no disk to act on" cases, which are not a failure here. -# -# growfs only runs when repart exited 0 (an actual GPT root partition was found -# and acted on) — NOT via a separate ExecStartPost. systemd runs ExecStartPost -# regardless of *which* accepted exit status ExecStart returned, and ExecStartPost -# has no visibility into that status (no $EXIT_STATUS — that's ExecStopPost-only), -# so an unconditional ExecStartPost=systemd-growfs would also run on the 76/77 -# "nothing to grow" cases, have nothing valid to grow, and fail the unit — turning -# a harmless no-op into a hard boot failure via the Requires= drop-ins on -# docker.service/host-agent.service (docs/progress/hosted-grow-root-disk.md). -# Passing 76/77 straight through as success (without calling growfs) keeps -# that invariant. -ExecStart=/bin/sh -c '\ - /usr/bin/systemd-repart --dry-run=no; rc=$?; \ - case "$rc" in \ - 0) /usr/lib/systemd/systemd-growfs / ;; \ - 76|77) exit 0 ;; \ - *) exit "$rc" ;; \ - esac' - -[Install] -WantedBy=multi-user.target diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.service b/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.service new file mode 100644 index 00000000..c2a746c5 --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.service @@ -0,0 +1,10 @@ +[Unit] +Description=moose: reboot to the other slot if a new OS slot was never marked good +Documentation=https://github.com/onmoose/os/blob/main/docs/specs/UPDATES.md +# Only on a slot under trial. The script checks the marker itself too. +ConditionPathExistsGlob=/var/lib/moose/os-update/trial-* + +[Service] +Type=oneshot +ExecStart=/usr/lib/moose/os-trial-check +StandardOutput=journal+console diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.timer b/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.timer new file mode 100644 index 00000000..39fc79ea --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/systemd/system/moose-os-trial.timer @@ -0,0 +1,13 @@ +[Unit] +Description=moose: check that a new OS slot was marked good (UPDATES.md # 1) +Documentation=https://github.com/onmoose/os/blob/main/docs/specs/UPDATES.md + +[Timer] +# Later than host-agent's own trial deadline (MOOSE_OS_TRIAL_TIMEOUT, 10 min), +# so a host-agent that is alive always decides first. This only catches a +# host-agent that never started or hung. +OnBootSec=15min +AccuracySec=10s + +[Install] +WantedBy=timers.target diff --git a/dev/cloud/mkosi.extra/etc/systemd/system/rescue.service.d/10-moose-reboot.conf b/dev/cloud/mkosi.extra/etc/systemd/system/rescue.service.d/10-moose-reboot.conf new file mode 100644 index 00000000..848da730 --- /dev/null +++ b/dev/cloud/mkosi.extra/etc/systemd/system/rescue.service.d/10-moose-reboot.conf @@ -0,0 +1,8 @@ +# A slot that hangs must still revert (UPDATES.md # 1). Nobody sits at a +# hosted box's console, and the root account is locked, so the stock +# emergency shell waits for ever and boot counting never gets a second boot. +# Reboot instead, after 10 s so the reason reaches the serial console. +[Service] +ExecStartPre= +ExecStart= +ExecStart=-/bin/sh -c 'echo "moose: rescue mode, rebooting in 10 s" > /dev/console; sleep 10; exec /usr/bin/systemctl --no-block reboot' diff --git a/dev/cloud/mkosi.extra/usr/lib/moose/etc-keep.list b/dev/cloud/mkosi.extra/usr/lib/moose/etc-keep.list new file mode 100644 index 00000000..51bc35ac --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/moose/etc-keep.list @@ -0,0 +1,26 @@ +# The files in the /etc upper layer that a Debian-major tidy-up keeps as they +# are (BUILD.md # 1b, rule 4; state-setup step 3a). One shell glob per line, +# relative to /etc; a directory keeps everything under it. Everything else in +# the upper layer moves to the attic at a major, except the account files +# (merged) and the pinned daemon.json and login.defs (taken again from the slot +# when that keeps the box's remap). The boot lane fails a boot whose upper +# layer holds a file this list, the account files, the pinned files and the +# links sshd's run state makes do not cover (cloud-assertions.sh), so a new +# file moose writes into /etc must be added here, or given a regeneration rule. +# +# The box's identity. +machine-id +ssh/ssh_host_* +# What users set: SSH access (the drop-in is the enabled set) and keys, and +# the time zone. +ssh/sshd_config.d/moose-allowed.conf +ssh/moose-authorized-keys +localtime +# The box's remap range, pinned at first boot (rule 2). +subuid +subgid +# Appliance only (#564): the LUKS recovery key, the data-drive marker and the +# network connections, WiFi included. +moose/secrets +moose/data-drive.enrolled +NetworkManager/system-connections diff --git a/dev/cloud/mkosi.extra/usr/lib/moose/os-trial-check b/dev/cloud/mkosi.extra/usr/lib/moose/os-trial-check new file mode 100755 index 00000000..3a040cee --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/moose/os-trial-check @@ -0,0 +1,35 @@ +#!/bin/sh +# The safety net for an OS update trial (UPDATES.md # 1, #563), run by +# moose-os-trial.timer some minutes after boot. +# +# host-agent writes /var/lib/moose/os-update/trial- before it switches to +# a new slot, and removes it once the new slot is healthy and marked good +# (internal/hostagent/osupdate). If the marker for the slot this box booted is +# still there now, the new slot never got that far: host-agent did not start, +# or it hung. host-agent cannot reboot the box then, so this does. GRUB skips a +# slot still on trial (TRY=1), so the next boot is the old slot. +# +# Nothing else: a boot that is not on trial has no marker, so this does nothing +# on every normal boot, and on a box that never updated its OS. +set -u +slot="$(sed -n 's/.*rauc\.slot=\([AB]\).*/\1/p' /proc/cmdline)" +[ -n "$slot" ] || exit 0 +[ -e "/var/lib/moose/os-update/trial-${slot}" ] || exit 0 +# The marker alone decides, not the grubenv. The grubenv's TRY flag is written +# by GRUB through the firmware, and a firmware can lose that write: OVMF +# without a VARS store did, by saving its variables to the ESP (#575). Trusting +# TRY=0 as "already good" would then leave a hung slot running for ever. The +# marker is on the state partition, written and removed by host-agent only, +# which removes it, with retries, right after it marks the slot good. +# Once per slot: if this script already rebooted this slot and the box came +# back to it (the mark-bad failed, and GRUB did not skip the slot), rebooting +# again would only loop. Stay up for a person. +if [ -e "/var/lib/moose/os-update/safety-net-${slot}" ]; then + echo "moose-os-trial: slot ${slot} was already rebooted once by this safety net and booted again; not rebooting again" + exit 0 +fi +echo "moose-os-trial: slot ${slot} was not marked good in time; rebooting so the box goes back to the other slot" +# A note for the old slot's host-agent, so the revert says which path made it. +date -u +%Y-%m-%dT%H:%M:%SZ > "/var/lib/moose/os-update/safety-net-${slot}" && sync +rauc status mark-bad booted || echo "moose-os-trial: mark-bad failed; rebooting anyway, GRUB skips a slot still on trial" +systemctl reboot diff --git a/dev/cloud/mkosi.extra/usr/lib/moose/state-setup b/dev/cloud/mkosi.extra/usr/lib/moose/state-setup new file mode 100755 index 00000000..d74a3e08 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/moose/state-setup @@ -0,0 +1,343 @@ +#!/bin/bash +# moose early state setup (BUILD.md # 1b # What a slot holds). Runs on every +# boot, in the initramfs, before systemd starts: the initramfs-tools hook +# /etc/initramfs-tools/scripts/local-bottom/moose-state mounts the booted slot +# read-only at $rootmnt and runs this script chrooted into it, so the tools it +# uses are the slot's own and it works the same in every slot. +# +# 1. systemd-repart makes slot B and the state partition on the first boot, +# and grows the state partition on a later boot when the disk grew. +# 2. The state partition is checked and mounted at /state, and its ext4 is +# grown to fill the partition. +# 3. When the slot's Debian major is not the one the box last booted, the +# /etc upper layer is tidied first (rule 4 below). Then /etc becomes an +# overlay: the slot's /etc below, /state/etc/upper above. +# 4. On the first boot only, the four pinned files are copied up and never +# follow the image again. +# 5. The bind mounts from the state partition. When a state directory does +# not exist yet, the slot's directory is copied into it first (copy once). +# +# Any failure is fatal: the hook panics, and panic=10 reboots the box, so a +# broken slot never hangs and never comes up without its state. +set -euo pipefail +export PATH=/usr/sbin:/usr/bin:/sbin:/bin +# The slot is read-only, and /tmp and /var/tmp are still the slot's own +# directories here (their tmpfs comes later, from systemd and /etc/fstab). +# systemd-repart and mkfs need a writable temp dir, so use the /run tmpfs +# the initramfs hook mounted (run 36930171023: "Failed to create temporary +# file: Read-only file system"). +mkdir -p /run/moose-state-tmp +export TMPDIR=/run/moose-state-tmp + +log() { echo "moose-state: $*" > /dev/kmsg 2>/dev/null || echo "moose-state: $*" >&2; } +die() { log "FAILED: $*"; exit 1; } + +STATE=/state +STATE_LABEL=moose-state + +# The four pinned files (BUILD.md # 1b, rule 2). They keep a box's +# userns-remap fixed for life (BUILD.md # User-namespace remap). +PINNED="docker/daemon.json subuid subgid login.defs" + +# The bind mounts from the state partition (BUILD.md # 1b, the inventory), as +# TARGET:SOURCE, with SOURCE relative to the state partition. The slot has an +# empty or default directory at each target; the build makes the ones the +# packages do not. +# +# Two rules shape this list (ENVIRONMENT.md # Storage (hosted)): +# - The databases stay on the local disk, each on a bind mount of its own: +# /var/lib/moose (the brain's SQLite, state/instances, state/services) and +# /var/lib/docker. They are never under /srv/moose. +# - User files live under /srv/moose, as on the appliance (STORAGE.md # Data +# drive(s)): /home comes from srv/moose/home. So a later storage pool at +# /srv/moose can take in the users' files without moving the databases. +# Nothing builds that pool yet. +BINDS="srv/moose:srv/moose home:srv/moose/home +var/lib/moose:var/lib/moose var/lib/docker:var/lib/docker +var/lib/containerd:var/lib/containerd var/log:var/log +var/lib/systemd:var/lib/systemd var/lib/rauc:var/lib/rauc +var/lib/sudo:var/lib/sudo" + +# --- 1. partitions, on the boot disk only. The boot disk is the disk that +# holds the booted slot: the device behind / (the slot's squashfs), then its +# parent disk through sysfs. repart and the state lookup below both use this +# one disk, so another disk that happens to carry a partition labelled +# moose-state (for example one taken from another box) is never repaired, +# grown or mounted. Idempotent: on a disk that is already laid out and has not +# grown, repart changes nothing. +root_majmin="$(stat -c '%Hd:%Ld' /)" || die "cannot stat /" +root_sys="$(readlink -f "/sys/dev/block/$root_majmin")" || die "no sysfs entry for / ($root_majmin)" +[ -e "$root_sys/partition" ] || die "/ ($root_majmin) is not a partition" +disk="/dev/$(basename "$(dirname "$root_sys")")" +[ -b "$disk" ] || die "the boot disk $disk is not a block device" +log "boot disk is $disk (/ is $root_majmin)" + +if ! out="$(systemd-repart --dry-run=no --no-pager "$disk" 2>&1)"; then + while IFS= read -r line; do log "repart: $line"; done <<< "$out" + die "systemd-repart failed" +fi +log "systemd-repart done" + +# --- 2. the state partition, on the boot disk and nowhere else. A new +# partition can take a moment to show up. No match on the boot disk is fatal; +# there is no fallback to another disk. +dev="" +for _ in $(seq 1 100); do + # The disk's partitions from sysfs, and each one's GPT name read from the + # disk by blkid's low-level probe. Not lsblk: it takes PARTLABEL from the + # udev database, which this chroot's fresh /run does not have (run + # 36940136412 found no match that way). + matches="" + for part_sys in /sys/block/"$(basename "$disk")"/*/partition; do + [ -e "$part_sys" ] || continue + part="/dev/$(basename "$(dirname "$part_sys")")" + name="$(blkid -p -o value -s PART_ENTRY_NAME "$part" 2>/dev/null || true)" + [ "$name" = "$STATE_LABEL" ] && matches="$matches $part" + done + matches="$(echo $matches | tr ' ' '\n')" + if [ "$(printf '%s\n' "$matches" | grep -c .)" -gt 1 ]; then + die "more than one partition labelled $STATE_LABEL on $disk: $(echo $matches)" + fi + dev="$matches" + [ -n "$dev" ] && [ -b "$dev" ] && break + dev="" + sleep 0.1 +done +[ -n "$dev" ] || die "no partition labelled $STATE_LABEL on the boot disk $disk" + +# -p fixes what is safe to fix. 1 means it fixed something, which is fine +# before a mount; 4 and up means it could not. +rc=0 +e2fsck -p "$dev" >/dev/null 2>&1 || rc=$? +[ "$rc" -lt 4 ] || die "e2fsck $dev exited $rc" + +mount -t ext4 -o rw,noatime "$dev" "$STATE" || die "mount $dev on $STATE failed" +# The partition may have grown (repart above). Grow the ext4 to fill it, online. +/usr/lib/systemd/systemd-growfs "$STATE" >/dev/null 2>&1 || die "systemd-growfs $STATE failed" +log "state partition $dev mounted on $STATE" + +# --- 3a. a Debian major (BUILD.md # 1b, rule 4). A file in the upper layer +# hides the slot's copy, so after a move to another Debian major it would stay +# at the old major's version under the new major's packages. So when the +# slot's major is not the one recorded on the state partition, the upper layer +# is rebuilt before the overlay is mounted, from three things: +# - the files moose keeps (/usr/lib/moose/etc-keep.list, and any list in +# /usr/lib/moose/etc-keep.d/), copied as they are: the box's identity and +# its users' settings; +# - the account files, merged: the slot's own entries, plus the box's +# entries the slot does not have, with group members joined; +# - the pinned files, taken again from the slot when that keeps the box's +# remap as it was, and kept as they are otherwise. +# Everything else (an admin's own edits, files a package wrote at run time) +# moves to an attic on the state partition, where it can still be read. It +# runs in both directions, so a revert to the older major is tidied the same +# way. The old upper layer moves to the attic whole, so nothing is lost. +MAJOR_FILE="$STATE/etc/.moose-debian-major" +# Set by a swap that took place; host-agent reads it once (below). +TIDIED_MARKER="$STATE/etc/.moose-major-tidied" +# Debian testing and sid have no VERSION_ID. Then this step is skipped: no +# tidy-up and no record. It never stops a boot. +slot_major="$( . /usr/lib/os-release 2>/dev/null; v="${VERSION_ID:-}"; echo "${v%%.*}" )" || slot_major="" + +keep_patterns() { + local f + for f in /usr/lib/moose/etc-keep.list /usr/lib/moose/etc-keep.d/*.list; do + [ -f "$f" ] || continue + sed -e 's/#.*//' -e 's/[[:space:]]*$//' -e '/^$/d' "$f" + done +} + +# merge_accounts FILE SLOT BOX: the slot's file with the box's entries the slot +# lacks. passwd: the slot's line wins (the slot's packages own it). shadow and +# gshadow: the box's password field wins, so a password set on the box stays. +# group and gshadow: members are joined. +merge_accounts() { + awk -F: -v OFS=: -v kind="$1" ' + function join(a, b, n, i, m, out, seen) { + out = ""; split("", seen) + n = split(a "," b, m, ",") + for (i = 1; i <= n; i++) if (m[i] != "" && !(m[i] in seen)) { seen[m[i]] = 1; out = out (out == "" ? "" : ",") m[i] } + return out + } + FNR == NR { box[$1] = $0; order[++nb] = $1; next } + { + name = $1; slot[name] = 1 + if (name in box) { + split(box[name], b, ":") + if (kind == "passwd" && b[3] != $3) printf "moose-state: account %s has uid %s on the box and %s in the slot\n", name, b[3], $3 > "/dev/stderr" + if (kind == "group" && b[3] != $3) printf "moose-state: group %s has gid %s on the box and %s in the slot\n", name, b[3], $3 > "/dev/stderr" + if (kind == "shadow") $2 = b[2] + if (kind == "group") $4 = join($4, b[4]) + if (kind == "gshadow") { $2 = b[2]; $3 = join($3, b[3]); $4 = join($4, b[4]) } + } + print + } + END { for (i = 1; i <= nb; i++) if (!(order[i] in slot)) print box[order[i]] } + ' "$3" "$2" +} + +# tidy_build FROM TO: build the new upper layer at upper.moose-new. It never +# dies: it returns non-zero on any failure, and the caller boots with the +# upper layer as it was. A bug here must not stop the box from booting, on +# either slot, because both slots run this same step. +tidy_build() { + local up="$STATE/etc/upper" new="$STATE/etc/upper.moose-new" pat p f line + rm -rf "$new" && mkdir -p "$new" && chmod 0755 "$new" || return 1 + # The files moose keeps, as they are (whiteouts and opaque dirs included). + # A pattern with no match is skipped: nullglob drops a glob, and the test + # drops a plain path the box does not have. + ( + cd "$up" || exit 1 + shopt -s nullglob dotglob + for pat in $(keep_patterns); do + for p in $pat; do + [ -e "$p" ] || [ -L "$p" ] || continue + cp -a --parents "$p" "$new/" || exit 1 + done + done + ) || { log "tidy-up: copying the kept files failed"; return 1; } + # The account files, merged. + for f in passwd group shadow gshadow; do + [ -f "$up/$f" ] || continue + merge_accounts "$f" "/etc/$f" "$up/$f" > "$new/$f" 2> "$TMPDIR/merge.log" \ + || { log "tidy-up: merging /etc/$f failed"; return 1; } + while IFS= read -r line; do log "${line#moose-state: }"; done < "$TMPDIR/merge.log" + chown --reference="/etc/$f" "$new/$f" && chmod --reference="/etc/$f" "$new/$f" || return 1 + done + # The pinned files (rule 2). subuid and subgid are the box's remap range: + # they are on the keep list. daemon.json and login.defs are taken again from + # the slot when that keeps the box's remap, and kept as they are otherwise. + mkdir -p "$new/docker" || return 1 + if [ "$(grep -o '"userns-remap"[^,}]*' /etc/docker/daemon.json 2>/dev/null)" = "$(grep -o '"userns-remap"[^,}]*' "$up/docker/daemon.json" 2>/dev/null)" ]; then + cp -p /etc/docker/daemon.json "$new/docker/daemon.json" || return 1 + else + cp -a "$up/docker/daemon.json" "$new/docker/daemon.json" || return 1 + log "tidy-up: the slot's daemon.json would change this box's remap, so the box's copy stays" + fi + if grep -Eq '^SUB_UID_COUNT[[:space:]]+0$' /etc/login.defs && grep -Eq '^SUB_GID_COUNT[[:space:]]+0$' /etc/login.defs; then + cp -p /etc/login.defs "$new/login.defs" || return 1 + else + cp -a "$up/login.defs" "$new/login.defs" || return 1 + log "tidy-up: the slot's login.defs does not set SUB_UID_COUNT and SUB_GID_COUNT to 0, so the box's copy stays" + fi +} + +# tidy_swap FROM TO: the old upper layer goes to the attic and the new one +# takes its place. A power cut between the two renames is finished on the next +# boot (below); one before the first rename starts the tidy-up again. +tidy_swap() { + local up="$STATE/etc/upper" new="$STATE/etc/upper.moose-new" attic + attic="$STATE/etc/attic/$(date -u +%Y%m%dT%H%M%SZ)-debian-$1-to-$2" + mkdir -p "$(dirname "$attic")" || return 1 + # Was sshd on? Its enable link is in the old layer only when the box turned + # it on (the image ships it off). Read before the swap moves the layer. + local sshd_on="" + [ -L "$up/systemd/system/multi-user.target.wants/ssh.service" ] && sshd_on=1 + mv "$up" "$attic" || return 1 + if ! mv "$new" "$up"; then + # Put the old layer back. If even that fails, the copy at + # upper.moose-new is kept: the caller never removes it while upper is + # missing, and the next boot finishes the swap from it. + [ -d "$up" ] || mv "$attic" "$up" || log "tidy-up: could not put the old upper layer back from $attic" + return 1 + fi + rm -rf "$STATE/etc/work" + mkdir -p "$STATE/etc/work" + # For host-agent: sshd's enable links went to the attic (BUILD.md # 1b, + # rule 4), so it turns sshd back on once, at its next start. Only when sshd + # was on: an admin who turned it off by hand keeps it off. A marker that is + # already there stays: it comes from a swap a power cut stopped before the + # record, and this repeat sees the tidied layer, which has no sshd link. + if [ -n "$sshd_on" ]; then + touch "$TIDIED_MARKER" 2>/dev/null || log "tidy-up: could not leave $TIDIED_MARKER for host-agent" + fi + log "tidied /etc for Debian $1 to $2: kept $(find "$up" ! -type d | wc -l) files, sshd was $([ -n "$sshd_on" ] && echo on || echo off), the old upper layer is in $attic" +} + +# A tidy-up a power cut stopped: between the two renames, finish it; before +# them, the half-built copy goes and the tidy-up runs again. +if [ -d "$STATE/etc/upper.moose-new" ]; then + if [ -d "$STATE/etc/upper" ]; then + rm -rf "$STATE/etc/upper.moose-new" + else + mv "$STATE/etc/upper.moose-new" "$STATE/etc/upper" + log "finished a tidy-up of /etc that a reboot cut short" + fi +fi +box_major="$(cat "$MAJOR_FILE" 2>/dev/null || true)" +record_major=1 +if [ -z "$slot_major" ]; then + record_major="" + log "the slot's /usr/lib/os-release has no VERSION_ID; no Debian-major tidy-up of /etc on this boot" +elif [ -n "$box_major" ] && [ "$box_major" != "$slot_major" ] && [ -e "$STATE/etc/.moose-pinned" ] && [ -d "$STATE/etc/upper" ]; then + if tidy_build "$box_major" "$slot_major" && tidy_swap "$box_major" "$slot_major"; then + : + else + # The box boots with the upper layer as it was, and the next boot + # tries again, because the old major stays recorded. The half-built + # copy is removed only while the old layer is in place; with no upper + # layer it is all the box has, and the next boot's check above uses it. + if [ -d "$STATE/etc/upper" ]; then + rm -rf "$STATE/etc/upper.moose-new" + else + # Both renames back failed. Booting would mount an empty upper + # layer: no users, host keys or machine-id. Stop instead; the next + # boot moves upper.moose-new into place (the check above). + die "tidy-up: no /etc upper layer after a failed swap" + fi + record_major="" + log "tidy-up of /etc for Debian $box_major to $slot_major FAILED; booting with the upper layer as it was" + fi +fi +# Recorded last, so a tidy-up a power cut stopped runs again. A box without the +# record (a new box, or one made before the tidy-up shipped) gets the slot's. +# A failed write (a full state partition) is logged and the boot goes on. +if [ -n "$record_major" ] && [ "$box_major" != "$slot_major" ]; then + if ! { mkdir -p "$STATE/etc" && echo "$slot_major" > "$MAJOR_FILE.new" && mv -f "$MAJOR_FILE.new" "$MAJOR_FILE"; }; then + log "could not record Debian major $slot_major in $MAJOR_FILE; the next boot tries again" + fi +fi +log "Debian major of the slot: $slot_major; recorded on the state partition: $(cat "$MAJOR_FILE" 2>/dev/null)" + +# --- 3. /etc as an overlay. Every change the box makes in /etc (users, +# passwords, SSH host keys, machine-id, the rendered sshd drop-ins) lands in +# the upper layer. A file the box never touched still comes from the slot. +mkdir -p "$STATE/etc/upper" "$STATE/etc/work" +chmod 0755 "$STATE/etc/upper" +mount -t overlay overlay \ + -o "lowerdir=/etc,upperdir=$STATE/etc/upper,workdir=$STATE/etc/work" /etc \ + || die "overlay on /etc failed" +log "/etc is an overlay (upper on $STATE/etc/upper)" + +# --- 4. pinned files, first boot only. Copying a file onto itself through the +# overlay puts it in the upper layer, where it hides every later slot's copy. +if [ ! -e "$STATE/etc/.moose-pinned" ]; then + for f in $PINNED; do + [ -e "/etc/$f" ] || die "pinned file /etc/$f is not in the image" + cp -p "/etc/$f" "/etc/$f.moose-pin" + mv -f "/etc/$f.moose-pin" "/etc/$f" + log "pinned /etc/$f" + done + touch "$STATE/etc/.moose-pinned" +fi + +# --- 5. bind mounts. Copy once: when the state directory does not exist yet, +# the slot's directory (with whatever the image baked into it, such as the +# control-plane images under /var/lib/moose) is copied into it. From then on +# the state copy is the one in use, and a newer slot's baked copy stays +# hidden. The copy goes to a temporary name first, so a power cut mid-copy is +# retried whole on the next boot. +for pair in $BINDS; do + d="${pair%%:*}" + src="$STATE/${pair#*:}" + [ -d "/$d" ] || die "/$d is missing from the slot (the build makes every bind target)" + if [ ! -d "$src" ]; then + mkdir -p "$(dirname "$src")" + rm -rf "$src.moose-new" + cp -a "/$d" "$src.moose-new" + mv "$src.moose-new" "$src" + log "$src made from the slot's /$d" + fi + mount --bind "$src" "/$d" || die "bind $src on /$d failed" +done +log "bind mounts done: $(echo $BINDS)" diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/00-esp.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/00-esp.conf new file mode 100644 index 00000000..b523da80 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/repart.d/00-esp.conf @@ -0,0 +1,8 @@ +# Runtime layout, read by systemd-repart in the initramfs on every boot +# (/usr/lib/moose/state-setup; BUILD.md # 1b). These first three match the +# partitions the image already has, by type and in order, and change nothing. +# The build-time layout is dev/cloud/mkosi.repart/. +[Partition] +Type=esp +SizeMinBytes=128M +SizeMaxBytes=128M diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/05-bios.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/05-bios.conf new file mode 100644 index 00000000..c6f41a0a --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/repart.d/05-bios.conf @@ -0,0 +1,4 @@ +[Partition] +Type=21686148-6449-6e6f-744e-656564454649 +SizeMinBytes=1M +SizeMaxBytes=1M diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/10-slot-a.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/10-slot-a.conf new file mode 100644 index 00000000..1ba70af1 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/repart.d/10-slot-a.conf @@ -0,0 +1,6 @@ +# Slot A, already in the image (a 1 GiB squashfs). No Format=, so repart +# never treats it as empty. +[Partition] +Type=root +SizeMinBytes=1G +SizeMaxBytes=1G diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/20-slot-b.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/20-slot-b.conf new file mode 100644 index 00000000..a3d21a22 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/repart.d/20-slot-b.conf @@ -0,0 +1,10 @@ +# Slot B. Not in the image: made empty at first boot, right after slot A, at +# the same fixed 1 GiB. The first OS update writes a squashfs into it (RAUC, +# #563). The PARTUUID is pinned because RAUC's system.conf and grub.cfg name +# it. +[Partition] +Type=root +Label=moose-slot-b +UUID=21212121-2121-4121-8121-212121212121 +SizeMinBytes=1G +SizeMaxBytes=1G diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/30-state.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/30-state.conf new file mode 100644 index 00000000..b8e58fd4 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/repart.d/30-state.conf @@ -0,0 +1,11 @@ +# The state partition: everything the box writes that must outlive a slot +# swap (BUILD.md # 1b). Made and formatted at first boot. It has no size cap, +# so it takes the rest of the disk, and when the provider disk grows repart +# extends it on the next boot; state-setup then grows the ext4 inside it with +# systemd-growfs. This replaces moose-grow-root: the root no longer grows. +# linux-generic, not the "var" type, so systemd's gpt-auto never mounts it on +# its own. +[Partition] +Type=linux-generic +Label=moose-state +Format=ext4 diff --git a/dev/cloud/mkosi.extra/usr/lib/repart.d/50-moose-grow-root.conf b/dev/cloud/mkosi.extra/usr/lib/repart.d/50-moose-grow-root.conf deleted file mode 100644 index 7e3e73e5..00000000 --- a/dev/cloud/mkosi.extra/usr/lib/repart.d/50-moose-grow-root.conf +++ /dev/null @@ -1,23 +0,0 @@ -# Hosted cloud image — RUNTIME grow of the root partition to fill the provider -# disk. Read by systemd-repart at boot (moose-grow-root.service), NOT at build: -# mkosi's build-time layout lives in dev/cloud/mkosi.repart/ and pins the root at -# a fixed 8 GiB so the shipped image stays small (sparse). A provider VM has a far -# larger disk (e.g. 80 GB) than that baked 8 GiB, and nothing grew the partition -# onto it — so the box ran on 8 GiB, docker image storage + the brain's SQLite -# store shared it, and one ordinary app install filled it and 500'd login. -# -# systemd-repart matches this definition to the single existing root partition (by -# Type=root) and, because no SizeMaxBytes caps it, extends it to consume all free -# space on the disk. GrowFileSystem=yes sets the GPT GROWFS attribute bit on the -# partition, which is what systemd-growfs-root.service normally watches for — but -# that unit is only wired up by systemd-gpt-auto-generator when booted with -# root=gpt-auto, which cloud providers (Hetzner included) do not use; they pass an -# explicit root=PARTUUID=... instead. So on a real provider box this flag alone -# does not grow the filesystem: moose-grow-root.service calls systemd-growfs -# directly as a second step (see that unit) rather than relying on this flag being -# noticed by anything. No Format= on purpose: the partition already holds the -# populated ext4, and omitting Format keeps systemd-repart from ever treating it -# as empty. The ESP has no definition here, so it is left untouched. -[Partition] -Type=root -GrowFileSystem=yes diff --git a/dev/cloud/mkosi.extra/usr/lib/sysusers.d/moose.conf b/dev/cloud/mkosi.extra/usr/lib/sysusers.d/moose.conf new file mode 100644 index 00000000..470e3d62 --- /dev/null +++ b/dev/cloud/mkosi.extra/usr/lib/sysusers.d/moose.conf @@ -0,0 +1,16 @@ +# moose's own system users and groups (BUILD.md # 1b, rule 3). From +# sysusers.d, never from useradd at build time: the box's /etc/passwd is in +# the /etc overlay's upper layer once it has changed, so a user a new image +# added with useradd would be hidden. systemd-sysusers adds any that are +# missing on every boot. +# +# moose (3000) is the primary group of every box user (FIRST_RUN.md # +# Identity). moose-app (2000) and moose-shared (2001) are the app-runtime +# identities host-agent resolves by name (APP_ISOLATION.md # Runtime identity). +# moose-remap owns Docker's userns-remap range in /etc/subuid and /etc/subgid +# (BUILD.md # User-namespace remap); its own id does not matter. +g moose 3000 +g moose-app 2000 +g moose-shared 2001 +u moose-app 2000:moose-app "moose app runtime" /nonexistent /usr/sbin/nologin +u moose-remap - "moose userns-remap range owner" /nonexistent /usr/sbin/nologin diff --git a/dev/cloud/mkosi.postinst.chroot b/dev/cloud/mkosi.postinst.chroot index 20114dcb..2e4d21f1 100755 --- a/dev/cloud/mkosi.postinst.chroot +++ b/dev/cloud/mkosi.postinst.chroot @@ -8,12 +8,17 @@ # - enable docker + containerd, systemd-networkd, host-agent, the first-boot # image loader, and the provisioning-seed materializer (.wants symlinks — # `systemctl enable` can't run in the chroot); -# - provision the moose / moose-app / moose-shared identities host-agent + -# the brain resolve by name (FIRST_RUN.md, APP_ISOLATION.md # Runtime identity); +# - the moose / moose-app / moose-shared / moose-remap identities host-agent + +# the brain resolve by name come from /usr/lib/sysusers.d/moose.conf, not +# from useradd here (BUILD.md # 1b rule 3); # - bring the single virtio NIC up via systemd-networkd DHCP — the minimal # cloud-native path, NO NetworkManager (ENVIRONMENT.md # Networking, # Boot); # - pre-commit machine-id + mask systemd-firstboot so the interactive wizard -# never blocks boot. +# never blocks boot; +# - the A/B layout (#561): the kernel and initramfs inside the slot, GRUB's +# modules and the first grubenv on the ESP, the read-only slot's mount +# points, and the jobs that would write to the slot masked (the end of +# this file). # # Deliberately NOT here (hosted cuts, ENVIRONMENT.md # How the profile is realized): # NetworkManager, Avahi, the LUKS/TPM enrollment of the medium lane. sshd used to @@ -154,36 +159,23 @@ EOF # are committed static files under mkosi.extra/ (etc/moose/metadata-firewall.nft). enable_unit /etc/systemd/system/moose-metadata-firewall.service moose-metadata-firewall.service -# Grow the root partition + ext4 to fill the provider disk on boot (systemd-repart -# against /usr/lib/repart.d/50-moose-grow-root.conf). The image bakes a fixed 8 GiB -# root (mkosi.repart/) so the shipped raw stays small; a provider VM's disk is much -# larger, and without this the box would run on 8 GiB — docker image storage + the -# brain's SQLite store share the volume, so one app install could fill it and 500 -# login. Ordered Before=docker/host-agent (in the unit) so the disk is full before -# anything writes to it. Static unit under mkosi.extra/; systemd-repart ships in the -# systemd-repart package (added to the lean set in mkosi.conf). ENVIRONMENT.md # Storage. -enable_unit /etc/systemd/system/moose-grow-root.service moose-grow-root.service - -# moose primary group (GID 3000, FIRST_RUN.md # Identity) — new users are created -# with `useradd --gid moose`; the brain's /setup needs it present. moose-app / -# moose-shared (2000/2001, APP_ISOLATION.md # Runtime identity) are the well-known -# app-runtime identities host-agent resolves by name. Idempotent. -getent group moose >/dev/null || groupadd -g 3000 moose -getent group moose-app >/dev/null || groupadd -g 2000 moose-app -getent group moose-shared >/dev/null || groupadd -g 2001 moose-shared -getent passwd moose-app >/dev/null || useradd -r -u 2000 -g moose-app \ - -M -d /nonexistent -s /usr/sbin/nologin moose-app +# The OS update trial's safety net (UPDATES.md # 1, #563): reboots a box whose +# new slot was never marked good, when host-agent itself cannot. A timer, so +# it goes under timers.target. +mkdir -p /etc/systemd/system/timers.target.wants +ln -sf /etc/systemd/system/moose-os-trial.timer /etc/systemd/system/timers.target.wants/moose-os-trial.timer +chmod 0755 /usr/lib/moose/os-trial-check + # Docker's daemon-wide userns-remap (#530, BUILD.md # User-namespace remap). -# daemon.json names moose-remap, and dockerd reads that account's range from +# daemon.json names moose-remap (made by /usr/lib/sysusers.d/moose.conf, with +# the other moose identities; BUILD.md # 1b rule 3), and dockerd reads that account's range from # /etc/subuid and /etc/subgid. The range is host ids 1000000 to 1065535, far above # every moose band (2000/2001, app services from 2100, users from 3000). It is # written here, before any user exists. SUB_UID_COUNT 0 and SUB_GID_COUNT 0 in # login.defs stop useradd from giving each new user a range of its own, so no # account can use newuidmap to reach the remap range. The appliance image does # the same in dev/test-qemu/mkosi.postinst.chroot. Idempotent. -getent passwd moose-remap >/dev/null || useradd -r -U -M -d /nonexistent \ - -s /usr/sbin/nologin moose-remap for f in /etc/subuid /etc/subgid; do touch "$f" grep -q '^moose-remap:' "$f" || echo 'moose-remap:1000000:65536' >> "$f" @@ -213,3 +205,92 @@ chmod 0755 /usr/local/bin/moose-seed-materialize.sh 2>/dev/null || true # Mask systemd-firstboot.service so the interactive locale/timezone wizard never # blocks sysinit (the empty machine-id is not enough to suppress it alone). systemctl --root=/ mask systemd-firstboot.service >/dev/null 2>&1 || true + +# --------------------------------------------------------------------------- +# The A/B layout (#561, BUILD.md # 1b). The image carries the ESP, the BIOS boot +# partition and slot A; slot B and the state partition come at first boot. + +# The kernel and a Debian initramfs inside the slot, where grub.cfg reads them +# (/usr/lib/moose/boot/). A slot is one self-contained unit: the kernel always +# matches the root it boots. The initramfs is Debian's initramfs-tools (already +# in the image: linux-image-amd64 depends on it), because its local-bottom hook +# (/etc/initramfs-tools/scripts/local-bottom/moose-state) runs the early state +# setup before systemd starts. +# +# The kernel is MOVED, not copied. mkosi copies every kernel it finds in +# /usr/lib/modules// (or /boot) to the ESP, with an initrd of its own and +# menu entries after grub.cfg's. Nothing boots those, and they would need a +# much larger ESP. With the kernel moved, mkosi finds none (mkosi.conf, +# Bootable=auto), and the ESP holds only GRUB (mkosi.repart/00-esp.conf). +chmod 0755 /etc/initramfs-tools/scripts/local-bottom/moose-state /usr/lib/moose/state-setup +kver="$(ls /usr/lib/modules | sort -V | tail -n1)" +mkdir -p /usr/lib/moose/boot +if [ -e "/usr/lib/modules/$kver/vmlinuz" ]; then + mv "/usr/lib/modules/$kver/vmlinuz" /usr/lib/moose/boot/vmlinuz +else + mv "/boot/vmlinuz-$kver" /usr/lib/moose/boot/vmlinuz +fi +rm -f /boot/vmlinuz-* /usr/lib/modules/*/vmlinuz +# The initramfs mounts the slot (squashfs), and state-setup mounts the state +# partition (ext4) and the /etc overlay. Those mounts happen inside the +# initramfs, so their modules must be in it, whatever MODULES= picks. +for m in squashfs overlay ext4; do + grep -qx "$m" /etc/initramfs-tools/modules || echo "$m" >> /etc/initramfs-tools/modules +done +mkinitramfs -o /usr/lib/moose/boot/initrd.img "$kver" +# The kernel's own postinst made a second initramfs in /boot. Nothing reads it. +rm -f /boot/initrd.img-* +ls -l /usr/lib/moose/boot/ + +# GRUB modules on the ESP, so grub.cfg can insmod squash4, xzio, regexp, +# loadenv and test. mkosi's core images embed only a short list. About 3 MB +# per firmware. +mkdir -p /efi/grub +for p in i386-pc x86_64-efi; do + rm -rf "/efi/grub/$p" + cp -r "/usr/lib/grub/$p" "/efi/grub/$p" +done + +# The first grubenv (RAUC's GRUB backend): slot A holds this image and is good, +# slot B is empty. +grub-editenv /efi/grub/grubenv create +grub-editenv /efi/grub/grubenv set ORDER="A B" A_OK=1 A_TRY=0 B_OK=0 B_TRY=0 + +# The slot is mounted read-only, so every mount point must already exist: the +# state partition's, the bind targets (state-setup refuses a missing one), the +# ESP's and /var/tmp's. /opt/containerd/{bin,lib} are made by containerd when +# it starts if they are missing, so they are made here instead. +mkdir -p /state /efi /home /srv/moose /var/log /var/tmp \ + /var/lib/docker /var/lib/containerd /var/lib/moose /var/lib/systemd \ + /var/lib/rauc /var/lib/sudo /opt/containerd/bin /opt/containerd/lib +# dbus's tmpfiles rule makes this link at boot when it is missing. +mkdir -p /var/lib/dbus +ln -sf /etc/machine-id /var/lib/dbus/machine-id + +# The slot replaces these jobs, or they would write to it: +# - systemd-repart.service: state-setup runs repart in the initramfs, before +# systemd, on every boot. One owner. +# - ldconfig.service: the slot ships its own ld.so.cache. A copy written into +# the /etc upper layer would outlive the slot it describes, and after a +# revert it would name libraries of the newer slot. +# - systemd-update-done.service: it stamps /var/.updated on the read-only +# slot. Without it, the ConditionNeedsUpdate jobs (systemd-sysusers, the +# journal catalog) run on every boot, which is what an image that can swap +# under /etc needs anyway. +systemctl --root=/ mask systemd-repart.service ldconfig.service systemd-update-done.service >/dev/null + +# Every account and group the image's packages made, as a sysusers.d file +# (BUILD.md # 1b, rule 3). The box's /etc/passwd and /etc/group are in the +# /etc upper layer and hide the slot's, so an account a later image's packages +# make at build time would be missing on the box. systemd-sysusers runs on +# every boot (systemd-update-done is masked above) and adds any of these the +# box lacks, with the image's ids. It changes nothing on a box that has them. +# The build checks the ids against dev/os-lock/cloud-accounts.lock +# (dev/cloud/build-bundle.sh), because a box keeps an account's id for life. +{ + echo "# Generated at build time from the image's /etc/group and /etc/passwd" + echo "# (dev/cloud/mkosi.postinst.chroot). Do not edit." + awk -F: '{ printf "g %s %s\n", $1, $3 }' /etc/group + awk -F: '{ printf "u %s %s:%s %s %s %s\n", $1, $3, $4, ($5 == "" ? "-" : "\"" $5 "\""), $6, $7 }' /etc/passwd + awk -F: '$4 != "" { n = split($4, m, ","); for (i = 1; i <= n; i++) printf "m %s %s\n", m[i], $1 }' /etc/group +} > /usr/lib/sysusers.d/moose-image-accounts.conf diff --git a/dev/cloud/mkosi.repart/00-esp.conf b/dev/cloud/mkosi.repart/00-esp.conf index 716aa7e6..ba57a340 100644 --- a/dev/cloud/mkosi.repart/00-esp.conf +++ b/dev/cloud/mkosi.repart/00-esp.conf @@ -1,7 +1,17 @@ +# The ESP (BUILD.md # 1b, partition 1). It holds GRUB for UEFI, GRUB's +# modules for both firmwares, the one grub.cfg both firmwares read, and the +# grubenv where RAUC keeps the slot order. The label is what /etc/fstab mounts +# at /efi, where RAUC finds the grubenv. +# +# 128 MiB, and only /efi is copied in. The kernel and initramfs live inside +# each slot, never here: the postinst moves the kernel out of +# /usr/lib/modules, so mkosi finds no kernel to copy to the ESP and writes no +# menu entries of its own (mkosi.conf, Bootable=auto). What lands here is about +# 4 to 7 MB, so 128 MiB leaves room for GRUB to grow. [Partition] Type=esp +Label=esp Format=vfat -CopyFiles=/boot:/ CopyFiles=/efi:/ -SizeMinBytes=512M -SizeMaxBytes=512M +SizeMinBytes=128M +SizeMaxBytes=128M diff --git a/dev/cloud/mkosi.repart/05-bios.conf b/dev/cloud/mkosi.repart/05-bios.conf index f647267e..543ed770 100644 --- a/dev/cloud/mkosi.repart/05-bios.conf +++ b/dev/cloud/mkosi.repart/05-bios.conf @@ -1,18 +1,9 @@ -# BIOS Boot Partition (#277). Required for the GRUB BIOS boot path that -# BiosBootloader=grub (mkosi.conf) adds so the image boots on legacy-BIOS -# providers (Hetzner CX / Intel) as well as UEFI ones (CPX / AMD). On a GPT disk -# there is no MBR gap for GRUB's core.img, so it must live in a partition of this -# well-known type; grub-bios-setup embeds core.img here and writes boot.img to the -# protective MBR. mkosi's own default-for-BIOS partitioning lays down exactly this -# (05-bios, 1 MiB, this type GUID) — we replicate it because providing -# mkosi.repart/ at all takes over partitioning fully (00-esp, 05-bios, 10-root). -# -# The ESP (00-esp.conf) stays 512M: mkosi's default bumps the ESP to 1G only for -# grub-EFI images (which duplicate each kernel/initrd onto the ESP because they -# can't use UKIs). Here the UEFI path is systemd-boot and the kernel sits on the -# ESP once — the BIOS grub.cfg menuentry references those same files — so 512M is -# unchanged and ample. No Format=: the partition holds raw core.img, not a -# filesystem. +# BIOS Boot Partition (#277; BUILD.md # 1b, partition 2). On a GPT disk there +# is no MBR gap for GRUB's core.img, so it lives in a partition of this +# well-known type: grub-bios-setup embeds core.img here and writes boot.img to +# the protective MBR. Legacy BIOS (Hetzner CX) boots through it; UEFI boots +# GRUB from the ESP. Both read the same grub.cfg on the ESP. No Format=: the +# partition holds raw core.img, not a filesystem. [Partition] Type=21686148-6449-6e6f-744e-656564454649 SizeMinBytes=1M diff --git a/dev/cloud/mkosi.repart/10-root.conf b/dev/cloud/mkosi.repart/10-root.conf deleted file mode 100644 index 41c20924..00000000 --- a/dev/cloud/mkosi.repart/10-root.conf +++ /dev/null @@ -1,16 +0,0 @@ -[Partition] -Type=root -Format=ext4 -CopyFiles=/ -# Fix the root partition at exactly 8 GiB (Min == Max). mkosi sizes the disk to -# fit the declared partitions, so this is effectively fixed today; pinning Max too -# makes "8 GiB" true even if a partition is added before root or systemd-repart is -# pointed at a larger disk. Sparse on disk — the raw only consumes written blocks. -SizeMinBytes=8G -SizeMaxBytes=8G -# Pin the GPT partition UUID so the kernel cmdline's root=PARTUUID=... (mkosi.conf) -# is deterministic — the initrd mounts the root by this UUID rather than relying -# on systemd gpt-auto discovery (which needs an EFI loader variable that is not -# set in this boot path). GrowFileSystem defaults on, so the ext4 fills the -# partition; the PARTUUID is stable across the grow. -UUID=20202020-2020-4020-8020-202020202020 diff --git a/dev/cloud/mkosi.repart/10-slot-a.conf b/dev/cloud/mkosi.repart/10-slot-a.conf new file mode 100644 index 00000000..99f1b367 --- /dev/null +++ b/dev/cloud/mkosi.repart/10-slot-a.conf @@ -0,0 +1,32 @@ +# Slot A (BUILD.md # 1b, partition 3): one whole root, kernel and initramfs +# included (/usr/lib/moose/boot/). Slot B and the state partition are not in +# the image: systemd-repart makes them at first boot +# (mkosi.extra/usr/lib/repart.d/). +# +# A read-only squashfs, compressed with xz. GRUB 2.12 must read the kernel +# and initramfs from inside the slot, and squashfs-xz is the one compressed +# format it reads (it has no erofs driver, and no zstd for squashfs). The slot +# is never written by the box, so a read-only filesystem costs nothing. +# +# 1 GiB, fixed for the life of the box: slot B is made right after slot A, so +# a slot can never grow later. The build fails when the squashfs fills more +# than 60% of it (dev/cloud/slotbudget, BUILD.md # 1b # Disk budget). +# +# The PARTUUID is pinned because grub.cfg boots root=PARTUUID=... (no gpt-auto, +# which needs an EFI loader variable this boot path does not set) and RAUC's +# system.conf names the slot by it. It is the UUID the single root had before +# the A/B layout. +# +# /efi is the ESP's mount point on the booted box, so only its contents are +# left out; the empty directory stays, because the slot is read-only and +# cannot make it at boot. +[Partition] +Type=root +Label=moose-slot-a +UUID=20202020-2020-4020-8020-202020202020 +Format=squashfs +Compression=xz +CopyFiles=/ +ExcludeFiles=/efi/ +SizeMinBytes=1G +SizeMaxBytes=1G diff --git a/dev/cloud/ovmf.sh b/dev/cloud/ovmf.sh new file mode 100644 index 00000000..c710cc57 --- /dev/null +++ b/dev/cloud/ovmf.sh @@ -0,0 +1,33 @@ +# Sourced by dev/cloud/run-cloud-tests.sh and dev/cloud/test/bootstrap.sh: the +# one list of OVMF firmware paths the hosted boot lane accepts (#575). +# +# OVMF is picked as a pair: a CODE image and the VARS template that matches it +# (the sizes must match). The VARS store is required. Without a writable one, +# OVMF saves its variables to an NvVars file on the ESP at every boot, and +# that write loses GRUB's grubenv save (run-cloud-tests.sh has the details). +# +# ovmf_find sets OVMF_CODE and OVMF_VARS_TEMPLATE to the first pair it can read, +# and returns 1 when there is none. +ovmf_find() { + local pair + OVMF_CODE="" + OVMF_VARS_TEMPLATE="" + for pair in /usr/share/OVMF/OVMF_CODE_4M.fd:/usr/share/OVMF/OVMF_VARS_4M.fd \ + /usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd \ + /usr/share/edk2/x64/OVMF_CODE.4m.fd:/usr/share/edk2/x64/OVMF_VARS.4m.fd \ + /usr/share/edk2/ovmf/OVMF_CODE.fd:/usr/share/edk2/ovmf/OVMF_VARS.fd \ + /usr/share/edk2-ovmf/x64/OVMF_CODE.fd:/usr/share/edk2-ovmf/x64/OVMF_VARS.fd; do + if [ -r "${pair%%:*}" ] && [ -r "${pair#*:}" ]; then + OVMF_CODE="${pair%%:*}" + OVMF_VARS_TEMPLATE="${pair#*:}" + return 0 + fi + done + return 1 +} + +# ovmf_wanted: true when the firmware list (MOOSE_CLOUD_FIRMWARES, default +# "uefi bios") has UEFI in it. A BIOS-only run never starts OVMF. +ovmf_wanted() { + case " ${MOOSE_CLOUD_FIRMWARES:-uefi bios} " in *" uefi "*) return 0 ;; *) return 1 ;; esac +} diff --git a/dev/cloud/rauc.sh b/dev/cloud/rauc.sh new file mode 100644 index 00000000..0054731b --- /dev/null +++ b/dev/cloud/rauc.sh @@ -0,0 +1,119 @@ +# shellcheck shell=bash +# Shared RAUC helpers for the OS update bundle (#562, BUILD.md # 1b # The +# bundle, docs/dev/rauc-signing.md). Sourced, not run, by +# dev/cloud/stage-control-plane.sh (the keyring both images bake), +# dev/cloud/build-bundle.sh (the build job) and dev/release/sign-bundle.sh (the +# sign job). +# +# Expects REPO_ROOT set by the sourcing script. +# +# Which keyring an image bakes, as /etc/rauc/keyring.pem, is set by +# MOOSE_RAUC_KEYRING: +# throwaway (the default) the root of a throwaway CA, made once per checkout +# under .dev/rauc/throwaway/ with dev/release/rauc-ca.sh. A box +# from such an image trusts only bundles this checkout signed. Every +# run that publishes no OS image uses it, and so does a local build. +# release dev/release/rauc/release-ca.pem, the public cert of the offline +# root CA. ci-cloud-image.yml sets it on every run that publishes +# the OS line. It fails when the file is not committed yet. + +# MOOSE_RAUC_RELEASE_CA_FILE points the checks at another file: for the tests +# in dev/release/rauc_test.go only. +RAUC_RELEASE_CA="${MOOSE_RAUC_RELEASE_CA_FILE:-${REPO_ROOT}/dev/release/rauc/release-ca.pem}" +RAUC_THROWAWAY_DIR="${REPO_ROOT}/.dev/rauc/throwaway" +RAUC_HOWTO="docs/dev/rauc-signing.md" + +# rauc_keyring_mode prints throwaway or release, or fails on any other value. +rauc_keyring_mode() { + case "${MOOSE_RAUC_KEYRING:-throwaway}" in + throwaway|release) printf '%s\n' "${MOOSE_RAUC_KEYRING:-throwaway}" ;; + *) echo "MOOSE_RAUC_KEYRING must be throwaway or release, got '${MOOSE_RAUC_KEYRING}'" >&2; return 1 ;; + esac +} + +# rauc_require_release_ca fails, naming the how-to, when the release root CA is +# not committed, is not a CA certificate, has expired, or is one of the +# throwaway or check CAs this repo's scripts make (its subject says so). +# release.yml runs it (dev/release/require-release-ca.sh) before it tags an OS +# release, and every OS publish run before it builds. +rauc_require_release_ca() { + if [ ! -f "$RAUC_RELEASE_CA" ]; then + echo "no release root CA: dev/release/rauc/release-ca.pem is not committed yet, so no OS release can be built or published. The maintainer makes the root offline and commits its public cert; see ${RAUC_HOWTO}." >&2 + return 1 + fi + if ! openssl x509 -in "$RAUC_RELEASE_CA" -noout -ext basicConstraints 2>/dev/null | grep -q 'CA:TRUE'; then + echo "dev/release/rauc/release-ca.pem is not a CA certificate; see ${RAUC_HOWTO}." >&2 + return 1 + fi + if openssl x509 -in "$RAUC_RELEASE_CA" -noout -subject | grep -Eqi 'THROWAWAY|REHEARSAL|WRONG KEY'; then + echo "dev/release/rauc/release-ca.pem is a throwaway or check CA ($(openssl x509 -in "$RAUC_RELEASE_CA" -noout -subject)), not the offline release root; see ${RAUC_HOWTO}." >&2 + return 1 + fi + if ! openssl x509 -in "$RAUC_RELEASE_CA" -noout -checkend 0 >/dev/null; then + echo "dev/release/rauc/release-ca.pem has expired ($(openssl x509 -in "$RAUC_RELEASE_CA" -noout -enddate)); replace the root, see ${RAUC_HOWTO}." >&2 + return 1 + fi +} + +# rauc_throwaway_ca makes the throwaway CA and its signer once, under +# .dev/rauc/throwaway/, and reuses them after that, so the lean image and the +# boot-proof image of one build bake the same keyring. Short lifetimes: a +# throwaway is never meant to sign anything a week later. +rauc_throwaway_ca() { + local d="$RAUC_THROWAWAY_DIR" + if [ -f "$d/root-ca.pem" ] && [ -f "$d/signer.pem" ] && [ -f "$d/signer.key" ]; then + return 0 + fi + rm -rf "$d" + mkdir -p "$(dirname "$d")" + RAUC_CA_NAME="moose THROWAWAY (not for release)" RAUC_CA_ROOT_DAYS=30 RAUC_CA_SIGNER_DAYS=30 RAUC_CA_NO_PASSPHRASE=1 \ + "${REPO_ROOT}/dev/release/rauc-ca.sh" root "$d" >/dev/null + RAUC_CA_NAME="moose THROWAWAY (not for release)" RAUC_CA_ROOT_DAYS=30 RAUC_CA_SIGNER_DAYS=30 \ + "${REPO_ROOT}/dev/release/rauc-ca.sh" signer "$d" signer >/dev/null + if [ -n "${CALLER:-}" ]; then chown -R "$CALLER":"$(id -gn "$CALLER")" "${REPO_ROOT}/.dev/rauc" 2>/dev/null || true; fi +} + +# rauc_stage_keyring WIRING: write the keyring the image bakes to +# WIRING/etc/rauc/keyring.pem (/etc/rauc/system.conf names it). +rauc_stage_keyring() { + local wiring="$1" mode + mode="$(rauc_keyring_mode)" || return 1 + mkdir -p "$wiring/etc/rauc" + if [ "$mode" = "release" ]; then + rauc_require_release_ca || return 1 + cp "$RAUC_RELEASE_CA" "$wiring/etc/rauc/keyring.pem" + else + rauc_throwaway_ca || return 1 + cp "$RAUC_THROWAWAY_DIR/root-ca.pem" "$wiring/etc/rauc/keyring.pem" + fi + chmod 0644 "$wiring/etc/rauc/keyring.pem" + echo "rauc keyring: ${mode} ($(openssl x509 -in "$wiring/etc/rauc/keyring.pem" -noout -subject))" +} + +# rauc_run WORKDIR SCRIPT: run SCRIPT with bash in a throwaway container that +# holds the same RAUC the image ships: the pinned debian:trixie-slim +# (BRAIN_RUNTIME_IMAGE in dev/control-plane/images.lock), with rauc at the +# version dev/os-lock/cloud-packages.lock names, from snapshot.debian.org at +# the locked timestamp, and squashfs-tools. WORKDIR is mounted at /w. +# +# The snapshot is fetched over plain http: the slim image has no CA store, and +# apt checks every index against Debian's archive key, which the image does +# carry, so the transport needs no TLS for integrity. +rauc_run() { + local work="$1" script="$2" img ts rv + img="$(awk -F= '$1=="BRAIN_RUNTIME_IMAGE"{print $2}' "${REPO_ROOT}/dev/control-plane/images.lock")" + ts="$(tr -d '[:space:]' < "${REPO_ROOT}/dev/os-lock/debian-snapshot")" + rv="$(awk '$1=="rauc"{print $2}' "${REPO_ROOT}/dev/os-lock/cloud-packages.lock")" + [ -n "$img" ] && [ -n "$ts" ] && [ -n "$rv" ] || { echo "rauc_run: could not read the base image, snapshot or rauc version from the locks" >&2; return 1; } + docker run --rm -v "$work:/w" -w /w -e TS="$ts" -e RV="$rv" "$img" bash -euo pipefail -c ' + rm -f /etc/apt/sources.list.d/* + cat > /etc/apt/sources.list </dev/null + echo "in the container: $(rauc --version)" + '"$script" +} diff --git a/dev/cloud/run-cloud-tests.sh b/dev/cloud/run-cloud-tests.sh index 1dfa52c6..020a411c 100755 --- a/dev/cloud/run-cloud-tests.sh +++ b/dev/cloud/run-cloud-tests.sh @@ -9,9 +9,11 @@ # hosted — "the disk IS the installed system", ENVIRONMENT.md # Provisioning), PLUS # the seed delivery + wizard the medium lane has no analogue for. # -# Three sequential UEFI boots over ONE persisted qcow2 overlay (so the brain's -# box-id + first admin carry boot→boot), then a fourth legacy-BIOS smoke boot on -# its own overlay (#277), one virtio NIC with restrict=on (air-gapped — the seed +# Every boot runs twice, once under UEFI (OVMF) and once under legacy BIOS +# (SeaBIOS), each firmware on its own overlays (#561; #277 is why BIOS matters). +# Per firmware: three sequential boots over ONE persisted qcow2 overlay (so the +# brain's box-id + first admin carry boot→boot), then the boots that each take +# their own fresh overlay. One virtio NIC with restrict=on (air-gapped: the seed # arrives over SMBIOS, never the network), serial-log capture per boot. The in-VM # self-check (cloud-assertions.sh, run by moose-cloud-assertions. # service) reads which scenario to assert from a `moose.assert` SMBIOS credential, @@ -28,9 +30,6 @@ # boot 3 frozen a DIFFERENT seed B delivered, same overlay → the brain ignores # it (identity frozen in SQLite); the dashboard + /api still serve # under box_id A and the brain does not re-ingest -# boot 4 bios the SAME image re-booted under legacy BIOS (SeaBIOS, no OVMF) -# on its own overlay → proves the dual-firmware image (grub BIOS -# + systemd-boot UEFI) boots where a UEFI-only one hung (#277) # access own overlay + box-id, seeded with a TEST-PORTAL key the harness # holds → a real owner session, then the per-app forward-auth # access modes end-to-end through real Caddy (#308), and the @@ -72,7 +71,7 @@ VERSION="$(git -C "$REPO_ROOT" describe --tags --always --dirty 2>/dev/null || e QCOW2="${WORK}/moose-${VERSION}-amd64.qcow2" RUN_DIR="$(mktemp -d -t moose-cloud.XXXXXX)" -OVERLAY="${RUN_DIR}/overlay.qcow2" # writable, persisted across the three boots +OVERLAY="" # set per firmware and per boot below (new_overlay) QEMU_SERIAL="${RUN_DIR}/serial.log" # set per-phase by run_boot QEMU_PID="" VERDICT="" @@ -95,11 +94,16 @@ BOX_ID_SSH=heron-birch # The remap boots (#531) share ONE overlay of their own and one box-id: the # second boot is a reboot of the disk the first one installed apps on. BOX_ID_REMAP=moss-lynx +# The OS update boots (#563) each provision their OWN box on a fresh overlay: +# they write slot B and reboot the box between slots, so no other boot may see +# that disk. +BOX_ID_OS_UPDATE=fern-stoat +BOX_ID_OS_REVERT=wren-maple -# Which boots to run, space-separated (unseeded seeded frozen bios access). +# Which boots to run, space-separated (unseeded seeded frozen access update ssh remap). # Default: all. # A subset lets a caller run only the boots it needs — notably the cloud-image -# publish gate, which runs "unseeded seeded bios access" to prove the built image's +# publish gate, which runs "unseeded seeded access update ssh remap" to prove the built image's # brain accepts the current seed schema (the regression that gate exists for) and # leaves out the frozen-identity boot. Frozen is orthogonal to that gate (it # checks that a re-delivered seed is IGNORED on a later boot) and has shown a @@ -108,10 +112,7 @@ BOX_ID_REMAP=moss-lynx # publish gate. Order still holds: frozen reuses the overlay seeded leaves # behind, so run seeded whenever frozen runs. # -# Two boots stand outside that ordering, each on its OWN fresh overlay: -# - `bios` (#277) re-boots the image under legacy BIOS (SeaBIOS) instead of UEFI, -# so providers that only do legacy BIOS are covered; no ordering dependency, and -# the publish gate includes it directly. +# These boots stand outside that ordering, each on its OWN fresh overlay: # - `access` (#308, #469) proves the per-app forward-auth access modes end-to-end # (restricted gate, owner proxy-through, public, whole-Cookie-header strip) and # the hosted confirm step that opens the re-auth window without a password. It @@ -135,7 +136,25 @@ BOX_ID_REMAP=moss-lynx # checks them all again. One name runs both, because the second needs what # the first left on the disk. # All of these are in the gate: see ci-cloud-image.yml. -BOOTS="${MOOSE_CLOUD_BOOTS:-unseeded seeded frozen bios access update ssh remap}" +# - `os-update` and `os-revert` (#563) prove the A/B OS update on the booted +# image. `os-update`: a wrong digest is refused, the bundle installs into +# slot B ahead of the window, the box switches in the window, boots slot B +# and marks it good, with the owner, an app, the SSH host keys and the data +# intact. `os-revert`: the same bundle, but host-agent cannot start on +# slot B, so the image's safety-net timer reboots the box and it comes back +# on slot A on its own; then slot B is made active again and crashes its +# kernel in the initramfs, and GRUB must skip it (#575). Each reboots its +# box inside one QEMU run, so +# neither passes -no-reboot. Both need the test bundle from +# dev/cloud/test/build-os-test-bundle.sh (MOOSE_CLOUD_OS_BUNDLE_DIR). +BOOTS="${MOOSE_CLOUD_BOOTS:-unseeded seeded frozen access update ssh remap os-update os-revert}" +# Which firmwares run the boots (#561): by default every boot runs under UEFI and +# again under legacy BIOS. `bios` used to be a boot of its own (one un-seeded +# boot under SeaBIOS); it is a firmware now. +FIRMWARES="${MOOSE_CLOUD_FIRMWARES:-uefi bios}" +for f in $FIRMWARES; do + case "$f" in uefi|bios) ;; *) echo "unknown firmware '$f' in MOOSE_CLOUD_FIRMWARES='$FIRMWARES' (known: uefi bios)" >&2; exit 1 ;; esac +done should_run() { case " $BOOTS " in *" $1 "*) return 0 ;; *) return 1 ;; esac; } # Refuse a name this script does not know. The workflow lets a person type the # list (its `boots` input), and a typo would otherwise run nothing and pass. @@ -144,8 +163,9 @@ boot_count=0 for b in $BOOTS; do boot_count=$((boot_count + 1)) case "$b" in - unseeded|seeded|frozen|bios|access|update|ssh|remap) ;; - *) echo "unknown boot '$b' in MOOSE_CLOUD_BOOTS='$BOOTS' (known: unseeded seeded frozen bios access update ssh remap)" >&2; exit 1 ;; + unseeded|seeded|frozen|access|update|ssh|remap|os-update|os-revert) ;; + bios) echo "'bios' is no longer a boot: every boot runs under both firmwares (MOOSE_CLOUD_FIRMWARES, default 'uefi bios')" >&2; exit 1 ;; + *) echo "unknown boot '$b' in MOOSE_CLOUD_BOOTS='$BOOTS' (known: unseeded seeded frozen access update ssh remap os-update os-revert)" >&2; exit 1 ;; esac done [ "$boot_count" -gt 0 ] || { echo "MOOSE_CLOUD_BOOTS='$BOOTS' names no boot" >&2; exit 1; } @@ -165,6 +185,9 @@ dump_serial() { grep -niE 'cloud-assertions|moose|docker|caddy|brain|host-agent|networkd|fail' "$QEMU_SERIAL" 2>/dev/null | tail -40 >&2 || true # The whole diag block too. The tail above keeps only 40 lines, and a red # boot with many steps (the remap boots, #531) cuts the brain log out of it. + # GRUB's own messages (OVMF mirrors the EFI console to the serial port). + echo "--- serial: GRUB errors ---" >&2 + grep -aE '^error:|grub.*error|save_env|grubenv' "$QEMU_SERIAL" 2>/dev/null | head -20 >&2 || true echo "--- serial: diag block ---" >&2 sed -n '/=== MOOSE_CLOUD_DIAG ===/,/=== END MOOSE_CLOUD_DIAG ===/p' "$QEMU_SERIAL" 2>/dev/null | grep -v '^-A \|^:\|^\*' | cut -c1-2000 >&2 || true echo "--- serial: tail 30 ---" >&2 @@ -195,6 +218,16 @@ if [ "${EUID:-$(id -u)}" -ne 0 ]; then exit 1 fi +# --- 1 + 2. The image to boot. MOOSE_CLOUD_QCOW2 names a qcow2 that is already +# built, and then this script builds nothing. CI uses it: one job builds the +# boot-proof image once, and each boot job downloads it (ci-cloud-image.yml). +# Without it, this script builds the image and converts it, as below. +if [ -n "${MOOSE_CLOUD_QCOW2:-}" ]; then + [ -f "$MOOSE_CLOUD_QCOW2" ] || { echo "MOOSE_CLOUD_QCOW2='${MOOSE_CLOUD_QCOW2}' is not a file" >&2; exit 1; } + QCOW2="$(cd "$(dirname "$MOOSE_CLOUD_QCOW2")" && pwd)/$(basename "$MOOSE_CLOUD_QCOW2")" + mkdir -p "$WORK" + echo "booting the prebuilt image ${QCOW2} (MOOSE_CLOUD_QCOW2); not building" +else # --- 1. build (own canary gate; fast when current). "${REPO_ROOT}/dev/cloud/test/bootstrap.sh" @@ -203,29 +236,46 @@ fi echo "converting raw -> qcow2 cloud artifact: $(basename "$QCOW2")" qemu-img convert -f raw -O qcow2 "$IMAGE_OUT" "$QCOW2" [ -n "$CALLER" ] && chown "$CALLER":"$(id -gn "$CALLER" 2>/dev/null || echo "$CALLER")" "$QCOW2" 2>/dev/null || true +fi + +# Writable overlays backed by the pristine artifact; the base is never written. +# Each is DISK_SIZE, far bigger than the image (about 1.13 GiB: the 128 MiB ESP, +# the BIOS boot partition and the 1 GiB slot A), the way a provider disk is. +# First boot fills the rest with slot B (1 GiB) and the state partition, and +# the layout check (cloud-assertions.sh 1b) proves the state partition grew to +# the end of it. +DISK_SIZE="${MOOSE_CLOUD_DISK_SIZE:-24G}" +new_overlay() { # PATH + qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$1" "$DISK_SIZE" >/dev/null +} -# A single writable overlay backed by the pristine artifact, reused across all -# three boots so the box-id + first admin the seeded boot writes survive into the -# frozen-identity boot. The base artifact is never written. -qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$OVERLAY" >/dev/null -# --- 3. resolve OVMF firmware (varies by distro). One VARS copy, reused across -# boots so the EFI state persists like a real machine power-cycle. +# --- 3. resolve OVMF firmware (varies by distro): a CODE image and the VARS +# template that matches it, as a pair. One VARS copy, reused across boots so +# the EFI state persists like a real machine power-cycle. +# +# The VARS store is required (#575). Without a writable one (the combined +# /usr/share/ovmf/OVMF.fd attached read-only, which is what this lane used to +# pick on Ubuntu 24.04), OVMF keeps its variables in memory and saves them to +# an `NvVars` file on the ESP through its own FAT driver at every boot. GRUB's +# save of the TRY flag did not survive that: GRUB read its write back, but +# the initramfs found the grubenv sectors as they were before GRUB ran (runs +# 37061482623 and 37064755329). With a VARS store there is no NvVars, and the +# TRY flag stays. A real UEFI keeps +# its variables in flash and never writes the ESP. layout checks there is no +# NvVars on the ESP (cloud-assertions.sh), so this cannot come back silently. +# The pair list is dev/cloud/ovmf.sh, shared with the bootstrap preflight. +# A BIOS-only run (MOOSE_CLOUD_FIRMWARES=bios) never starts OVMF, so it does +# not need it. +# shellcheck source=dev/cloud/ovmf.sh +. "${REPO_ROOT}/dev/cloud/ovmf.sh" OVMF_CODE="" -for cand in /usr/share/OVMF/OVMF_CODE.fd /usr/share/ovmf/OVMF.fd \ - /usr/share/OVMF/OVMF.fd /usr/share/edk2-ovmf/x64/OVMF_CODE.fd; do - [ -r "$cand" ] && { OVMF_CODE="$cand"; break; } -done -[ -n "$OVMF_CODE" ] || { echo "OVMF code firmware not found" >&2; exit 1; } -OVMF_VARS_TEMPLATE="" -for cand in /usr/share/OVMF/OVMF_VARS.fd /usr/share/ovmf/OVMF_VARS.fd \ - /usr/share/edk2-ovmf/x64/OVMF_VARS.fd; do - [ -r "$cand" ] && { OVMF_VARS_TEMPLATE="$cand"; break; } -done OVMF_VARS="" -if [ -n "$OVMF_VARS_TEMPLATE" ]; then +if ovmf_wanted; then + ovmf_find || { echo "OVMF firmware not found: need a CODE image and its VARS template (package: ovmf)" >&2; exit 1; } OVMF_VARS="${RUN_DIR}/OVMF_VARS.fd" cp "$OVMF_VARS_TEMPLATE" "$OVMF_VARS" + echo "OVMF: code ${OVMF_CODE}, vars from ${OVMF_VARS_TEMPLATE}" fi ACCEL=tcg @@ -332,9 +382,11 @@ run_boot() { -device "virtio-net-pci,netdev=n0,mac=52:54:00:c1:0d:01" -smbios "type=11,value=io.systemd.credential:moose.assert=${mode}" "$@" - -no-reboot ) - if [ "$firmware" = uefi ] && [ -n "$OVMF_VARS" ]; then + # A boot that reboots its own box (the OS update boots) keeps QEMU up + # across the reboot; every other boot ends QEMU on a reboot. + if [ -z "${KEEP_REBOOTS:-}" ]; then qemu_args+=( -no-reboot ); fi + if [ "$firmware" = uefi ]; then qemu_args+=( -drive "if=pflash,format=raw,file=${OVMF_VARS}" ) fi @@ -391,6 +443,9 @@ run_boot() { # skipped, a proof that never ran — was indistinguishable from one that # held. These lines are the evidence, and they belong in the CI log. grep -o 'cloud-assertions:.*' "$QEMU_SERIAL" 2>/dev/null | tr -d '\r' | sed 's/^/ /' || true + # GRUB's own error lines, if any (#563: under UEFI GRUB does not + # save the try flag; this is where it would say why). + grep -aE '^error:|save_env|grubenv' "$QEMU_SERIAL" 2>/dev/null | tr -d '\r' | head -10 | sed 's/^/ grub: /' || true # On PASS the guest powers itself off (cloud-assertions.sh ok()); wait # for QEMU to exit so the overlay write (box-id + admin) flushes before # the next boot reads it. Bounded — kill if the clean shutdown hangs. @@ -409,12 +464,21 @@ run_boot() { esac } +# --- 4. every boot, once per firmware (#561). The provider's server type decides +# the firmware (ENVIRONMENT.md # Boot (hosted)), so the whole list runs under UEFI +# (OVMF) and again under legacy BIOS (SeaBIOS, QEMU's own firmware, as Hetzner CX +# presents it). Each firmware gets its own overlays, so the two never share a disk. +for FIRMWARE in $FIRMWARES; do +OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}.qcow2" +new_overlay "$OVERLAY" +echo "=== firmware ${FIRMWARE}: boots ${BOOTS} ===" + # --- 4. boot 1: un-seeded. No seed credential → the brain stays unprovisioned and # GET /_moose/sso returns 503 and /setup returns 403 (the SSO gate is armed but # closed — never the appliance's open empty-box behavior). Also the standalone C2 # control-plane-up proof. if should_run unseeded; then -if ! run_boot "unseeded" "unseeded"; then +if ! run_boot "${FIRMWARE}-unseeded" "unseeded"; then echo "cloud gate proof: ${VERDICT}" >&2 exit 1 fi @@ -426,7 +490,7 @@ fi # (disabled on hosted). The ingested box-id A persists on the overlay. The positive # owner-create + wizard path needs the portal private key (cloud on-ramp). if should_run seeded; then -if ! run_boot "seeded" "seeded" -smbios "type=11,value=$(seed_cred "$BOX_ID_A")"; then +if ! run_boot "${FIRMWARE}-seeded" "seeded" -smbios "type=11,value=$(seed_cred "$BOX_ID_A")"; then echo "cloud gate proof: ${VERDICT}" >&2 exit 1 fi @@ -439,34 +503,13 @@ fi # re-ingest. Proves a re-delivered or changed seed cannot re-key a provisioned box # (MOOSE_NETWORK.md frozen identity). if should_run frozen; then -if ! run_boot "frozen" "frozen:${BOX_ID_A}" -smbios "type=11,value=$(seed_cred "$BOX_ID_B")"; then +if ! run_boot "${FIRMWARE}-frozen" "frozen:${BOX_ID_A}" -smbios "type=11,value=$(seed_cred "$BOX_ID_B")"; then echo "cloud gate proof: ${VERDICT}" >&2 exit 1 fi echo "boot 3 OK — frozen identity held across reboot (re-delivered seed B ignored, box_id still ${BOX_ID_A})" fi -# --- 7. boot 4: legacy-BIOS smoke (#277). The three boots above all run under UEFI -# (OVMF). This one boots the SAME image under QEMU's built-in SeaBIOS — the legacy- -# BIOS firmware a Hetzner CX (Intel) VM presents, where a UEFI-only image hangs at -# "Booting from Hard Disk" and never reaches userspace. It proves the image's grub -# BIOS boot path (BiosBootloader=grub + the BIOS Boot Partition) actually boots to -# a running control plane. It reuses the un-seeded assertion (control plane up, SSO -# gate armed) as the "did it boot and come up" proof — the firmware path is what's -# under test here, not provisioning — on its OWN fresh overlay so it can run -# independently of (and never perturb) the UEFI provisioning sequence above. -if should_run bios; then -BIOS_OVERLAY="${RUN_DIR}/overlay-bios.qcow2" -qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$BIOS_OVERLAY" >/dev/null -OVERLAY="$BIOS_OVERLAY" -FIRMWARE=bios -if ! run_boot "bios" "unseeded"; then - echo "cloud BIOS boot proof: ${VERDICT}" >&2 - exit 1 -fi -echo "boot 4 OK — image boots under legacy BIOS (SeaBIOS), control plane up" -fi - # --- 8. access boot: per-app forward-auth access modes end-to-end (#308). Its OWN # fresh overlay + box-id, seeded with a TEST-PORTAL key so the box can mint a real # owner session (the positive path the box-only SSO gate above can't reach). The @@ -501,17 +544,16 @@ if should_run access; then # independent of the A/B identity the shared overlay carries. OVERLAY and FIRMWARE # are run_boot's globals, and the bios boot above leaves them pointing at ITS # overlay under SeaBIOS — so set both explicitly here rather than inheriting them. - ACCESS_OVERLAY="${RUN_DIR}/overlay-access.qcow2" - qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$ACCESS_OVERLAY" >/dev/null + ACCESS_OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}-access.qcow2" + new_overlay "$ACCESS_OVERLAY" OVERLAY="$ACCESS_OVERLAY" - FIRMWARE=uefi # Wider outer ceiling: the access scenario's in-guest work (SSO + app install + # exposure toggle) adds ~180s of worst-case internal poll time over the shared # prechecks, which alone can approach the 480s default under CI's TCG-only QEMU. # The two folder-app installs (#519) add up to ~240s more. VERDICT_TIMEOUT=960 - if ! run_boot "access" "access" \ + if ! run_boot "${FIRMWARE}-access" "access" \ -smbios "type=11,value=$(seed_cred_keyed "$BOX_ID_ACCESS" "$ACCESS_KEY")" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$ACCESS_TOKEN" | base64 -w0)" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token2=$(printf '%s' "$ACCESS_TOKEN2" | base64 -w0)" \ @@ -555,17 +597,16 @@ if should_run update; then # Same explicit-globals reasoning as the access boot: OVERLAY and FIRMWARE are # run_boot's globals and the boots above leave them pointing elsewhere. - UPDATE_OVERLAY="${RUN_DIR}/overlay-update.qcow2" - qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$UPDATE_OVERLAY" >/dev/null + UPDATE_OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}-update.qcow2" + new_overlay "$UPDATE_OVERLAY" OVERLAY="$UPDATE_OVERLAY" - FIRMWARE=uefi # The widest ceiling in the lane, and it is not padding: on top of the shared # prechecks this boot loads and starts a registry, commits and pushes two images, # then runs TWO full update transactions — the second of which spends a 60s health # wait failing on purpose before it reverts. Under CI's TCG-only QEMU that adds up. VERDICT_TIMEOUT=1500 - if ! run_boot "update" "update" \ + if ! run_boot "${FIRMWARE}-update" "update" \ -smbios "type=11,value=$(seed_cred_keyed "$BOX_ID_UPDATE" "$UPDATE_KEY" "$UPDATE_TARGET_URL")" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$UPDATE_TOKEN" | base64 -w0)"; then echo "cloud update proof: ${VERDICT}" >&2 @@ -601,10 +642,9 @@ if should_run ssh; then # Same explicit-globals reasoning as the access and update boots: OVERLAY and # FIRMWARE are run_boot's globals and the boots above leave them pointing # elsewhere. - SSH_OVERLAY="${RUN_DIR}/overlay-ssh.qcow2" - qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$SSH_OVERLAY" >/dev/null + SSH_OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}-ssh.qcow2" + new_overlay "$SSH_OVERLAY" OVERLAY="$SSH_OVERLAY" - FIRMWARE=uefi # Wider than the default for the same reason as the access boot: on top of the # shared prechecks this one drives SSO, an elevation, a key add, three sshd @@ -612,7 +652,7 @@ if should_run ssh; then # a systemctl round-trip through host-agent, and under CI's TCG-only QEMU those # add up well past the 480s default. VERDICT_TIMEOUT=900 - if ! run_boot "ssh" "ssh" \ + if ! run_boot "${FIRMWARE}-ssh" "ssh" \ -smbios "type=11,value=$(seed_cred_keyed "$BOX_ID_SSH" "$SSH_KEY_B64")" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$SSH_TOKEN" | base64 -w0)"; then echo "cloud ssh proof: ${VERDICT}" >&2 @@ -646,10 +686,9 @@ if should_run remap; then # Same explicit-globals reasoning as the access boot: OVERLAY and FIRMWARE # are run_boot's globals and the boots above leave them pointing elsewhere. - REMAP_OVERLAY="${RUN_DIR}/overlay-remap.qcow2" - qemu-img create -f qcow2 -b "$QCOW2" -F qcow2 "$REMAP_OVERLAY" >/dev/null + REMAP_OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}-remap.qcow2" + new_overlay "$REMAP_OVERLAY" OVERLAY="$REMAP_OVERLAY" - FIRMWARE=uefi # The first boot loads postgres:16 and runs five installs, one of them with # a managed Postgres spin-up, then a recreate and a second round of checks. @@ -657,7 +696,7 @@ if should_run remap; then # so it gets the update boot's ceiling. The reboot only waits for the apps # to come back. VERDICT_TIMEOUT=1500 - if ! run_boot "remap" "remap" \ + if ! run_boot "${FIRMWARE}-remap" "remap" \ -smbios "type=11,value=$(seed_cred_keyed "$BOX_ID_REMAP" "$REMAP_KEY")" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$REMAP_TOKEN" | base64 -w0)"; then echo "cloud remap proof: ${VERDICT}" >&2 @@ -665,7 +704,7 @@ if should_run remap; then fi echo "boot remap OK: one app per userns tier installed and checked, the caps-tier data kept across a recreate (box_id=${BOX_ID_REMAP})" VERDICT_TIMEOUT=900 - if ! run_boot "remap-reboot" "remap-reboot" \ + if ! run_boot "${FIRMWARE}-remap-reboot" "remap-reboot" \ -smbios "type=11,value=$(seed_cred_keyed "$BOX_ID_REMAP" "$REMAP_KEY")" \ -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$REMAP_TOKEN2" | base64 -w0)"; then echo "cloud remap reboot proof: ${VERDICT}" >&2 @@ -674,6 +713,73 @@ if should_run remap; then echo "boot remap-reboot OK: every tier checked again after a real reboot of the same disk (box_id=${BOX_ID_REMAP})" fi -echo "cloud end-to-end: PASS (boots: ${BOOTS})" +# --- 12. the OS update boots (#563). Each takes its own fresh overlay and +# box-id, a test-portal key and one owner assertion (the update-target read is +# admin-only), and a second, read-only disk holding the test bundle: the box is +# air-gapped, so the bundle reaches it as an ext4 image that the guest mounts +# and serves to host-agent from a file server inside the guest. The disk is +# made here, from MOOSE_CLOUD_OS_BUNDLE_DIR, with mke2fs -d (no root needed for +# that part). The guest reboots between slots inside one QEMU run. +# +# MOOSE_CLOUD_OS_REFUSE_ONLY=true runs only the refusal half of os-update: on +# a run that publishes the OS, the boot-proof image trusts only the release +# root, so the throwaway-signed test bundle must be refused (the maintainer's +# call for #563). +os_boot() { # NAME BOX_ID + local name="$1" box="$2" dir="${MOOSE_CLOUD_OS_BUNDLE_DIR:-}" mint key token disk sum ver mode + # A local run that built the image here makes the bundle from it too, once. + if [ -z "$dir" ] && [ -f "$IMAGE_OUT" ]; then + dir="${WORK}/os-test" + [ "$dir/os-test.raucb" -nt "$IMAGE_OUT" ] || GO="$GO" "${REPO_ROOT}/dev/cloud/test/build-os-test-bundle.sh" "$IMAGE_OUT" "$dir" + MOOSE_CLOUD_OS_BUNDLE_DIR="$dir" + fi + [ -n "$GO" ] && [ -x "$GO" ] || { echo "$name boot needs go to mint the owner assertion; none found (\$GO='${GO:-}')" >&2; exit 1; } + [ -n "$dir" ] && [ -f "$dir/os-test.raucb" ] && [ -f "$dir/os-test.sha256" ] && [ -f "$dir/os-test.version" ] || { + echo "$name boot needs the test bundle: set MOOSE_CLOUD_OS_BUNDLE_DIR to the output of dev/cloud/test/build-os-test-bundle.sh" >&2 + exit 1 + } + mapfile -t mint < <(mint_owner_assertion "$box") || true + key="${mint[0]:-}"; token="${mint[1]:-}" + [ -n "$key" ] && [ -n "$token" ] || { echo "$name boot: failed to mint the owner assertion" >&2; exit 1; } + disk="${RUN_DIR}/os-bundle-${FIRMWARE}-${name}.img" + local bytes; bytes="$(stat -c %s "$dir/os-test.raucb")" + mke2fs -q -t ext4 -L moose-os-test -d "$dir" "$disk" "$(( bytes / 1048576 + 64 ))M" + sum="$(tr -d '[:space:]' < "$dir/os-test.sha256")" + ver="$(tr -d '[:space:]' < "$dir/os-test.version")" + mode=full + [ "${MOOSE_CLOUD_OS_REFUSE_ONLY:-false}" = true ] && mode=refuse + + OVERLAY="${RUN_DIR}/overlay-${FIRMWARE}-${name}.qcow2" + new_overlay "$OVERLAY" + VERDICT_TIMEOUT=1500 + KEEP_REBOOTS=1 + if ! run_boot "${FIRMWARE}-${name}" "$name" \ + -drive "file=${disk},if=virtio,format=raw,readonly=on" \ + -smbios "type=11,value=$(seed_cred_keyed "$box" "$key" "http://127.0.0.1:5001/target.json")" \ + -smbios "type=11,value=io.systemd.credential.binary:moose.sso_token=$(printf '%s' "$token" | base64 -w0)" \ + -smbios "type=11,value=io.systemd.credential:moose.os_test=${mode}:${ver}:${sum}"; then + KEEP_REBOOTS="" + echo "cloud ${name} proof: ${VERDICT}" >&2 + exit 1 + fi + KEEP_REBOOTS="" +} +if should_run os-update; then + os_boot os-update "$BOX_ID_OS_UPDATE" + echo "boot os-update OK: the OS update installed into slot B, the box switched, booted slot B and marked it good, with its owner, app, host keys and data intact (box_id=${BOX_ID_OS_UPDATE})" +fi +if should_run os-revert; then + os_boot os-revert "$BOX_ID_OS_REVERT" + # The last stage (#575): slot B crashed in its initramfs. The guest checks + # it came back on slot A; the serial log shows the crash itself. + grep -aq 'moose-test: slot B panics before userspace' "$QEMU_SERIAL" && grep -aq 'Kernel panic' "$QEMU_SERIAL" \ + || { echo "cloud os-revert proof: no kernel panic of slot B on the serial console" >&2; exit 1; } + echo "boot os-revert OK: slot B never came up, the safety net rebooted the box and it went back to slot A on its own; then slot B panicked before userspace and GRUB skipped it (box_id=${BOX_ID_OS_REVERT})" +fi + +echo "firmware ${FIRMWARE}: every boot OK (${BOOTS})" +done + +echo "cloud end-to-end: PASS (boots: ${BOOTS}; firmwares: ${FIRMWARES})" echo "qcow2 cloud artifact: ${QCOW2}" exit 0 diff --git a/dev/cloud/slot-budget.sh b/dev/cloud/slot-budget.sh new file mode 100644 index 00000000..f398bf68 --- /dev/null +++ b/dev/cloud/slot-budget.sh @@ -0,0 +1,35 @@ +# shellcheck shell=bash +# The OS slot budget (#561, BUILD.md # 1b # Disk budget). Sourced by +# dev/cloud/bootstrap.sh (the image that ships), which fails when the squashfs +# in slot A fills more than 60% of its fixed 1 GiB partition, and by +# dev/cloud/test/bootstrap.sh (the boot-proof image), which only reports: that +# image also bakes test-only images (postgres:16, a registry), which never ship. +# +# Expects these set by the sourcing script: REPO_ROOT, CALLER (may be empty), GO. + +# slot_budget_check IMAGE.raw [MAX_PERCENT] [TITLE]: build dev/cloud/slotbudget +# as the caller (so root never owns the caller's Go build cache) and run it on +# IMAGE. MAX_PERCENT defaults to the budget, 60. In CI the table also goes to +# the job summary, under TITLE. +# +# With MOOSE_OS_LOCK_RECORD=1 (the lock bump's record build) an over-budget +# image does not stop the build, the same as the lean check: the bump still +# needs its resolved list, and the bump PR's own CI run fails on it. +slot_budget_check() { + local image="$1" max="${2:-60}" title="${3:-OS slot budget (the image that ships)}" tool rc=0 + [ -f "$image" ] || { echo "slot budget: no image at '$image'" >&2; return 1; } + tool="$(dirname "$image")/slotbudget" + if [ -n "${CALLER:-}" ]; then + sudo -u "$CALLER" "$GO" build -C "$REPO_ROOT" -o "$tool" ./dev/cloud/slotbudget || return 1 + else + "$GO" build -C "$REPO_ROOT" -o "$tool" ./dev/cloud/slotbudget || return 1 + fi + "$tool" -max-percent "$max" -title "$title" ${GITHUB_STEP_SUMMARY:+-summary "$GITHUB_STEP_SUMMARY"} "$image" || rc=$? + if [ "$rc" -ne 0 ]; then + if [ "$rc" -eq 1 ] && [ "${MOOSE_OS_LOCK_RECORD:-}" = "1" ]; then + echo "slot budget failed; going on because MOOSE_OS_LOCK_RECORD=1" >&2 + return 0 + fi + return "$rc" + fi +} diff --git a/dev/cloud/slotbudget/main.go b/dev/cloud/slotbudget/main.go new file mode 100644 index 00000000..f0f823d1 --- /dev/null +++ b/dev/cloud/slotbudget/main.go @@ -0,0 +1,355 @@ +// Command slotbudget checks the hosted image's OS slot against its budget +// (BUILD.md # 1b # Disk budget, #561). +// +// An OS slot is a fixed 1 GiB partition that holds a read-only squashfs. Its +// size is fixed for the life of a box, because slot B is made right after +// slot A at first boot. So the build fails long before the slot is full: when +// the squashfs in slot A fills more than 60% of the partition, the build stops +// and someone has to make room or change the budget on purpose. +// +// slotbudget [-max-percent 60] [-summary FILE] [-title TEXT] [-extract FILE] IMAGE.raw +// +// It reads the raw disk image directly: the GPT, then the squashfs superblock +// at the start of the partition labelled moose-slot-a. It needs no loop +// device and no root. It prints the partition table with each partition's +// share of the smallest hosted disk (40 GB, a Hetzner CX23) and the slot's +// measured content, and appends the same table to FILE when -summary is given +// (CI passes $GITHUB_STEP_SUMMARY). +// +// With -extract FILE it also writes the slot's squashfs to FILE, cut to the +// squashfs's own size rounded up to 4 KiB (the block size mksquashfs pads +// to), never past the partition. That is the slot image the OS update bundle +// carries (dev/cloud/build-bundle.sh, BUILD.md # 1b # The bundle, #562): the +// same bytes as slot A in the disk image. It is written only when the slot +// is within budget. +// +// Exit status: 0 within budget, 1 over budget or not the slot GRUB can read, +// 2 when the image cannot be read. +package main + +import ( + "bytes" + "encoding/binary" + "errors" + "flag" + "fmt" + "io" + "os" + "strings" + "unicode/utf16" +) + +const ( + sectorSize = 512 + + // slotLabel is slot A's GPT partition name (dev/cloud/mkosi.repart/10-slot-a.conf). + slotLabel = "moose-slot-a" + + // smallestDisk is the disk of the smallest hosted box, a Hetzner CX23: + // 40 GB. Every cost is shown as a share of it, so no cost looks smaller + // than it is on the box where it hurts most. + smallestDisk = 40_000_000_000 + + squashfsMagic = 0x73717368 // "hsqs", little-endian + squashfsXZ = 4 // the superblock's compression id for xz +) + +// partition is one used GPT entry. +type partition struct { + Number int + Name string + Start uint64 // byte offset + Size uint64 // bytes +} + +// squashfs is what the slot check needs from a squashfs superblock. +type squashfs struct { + BytesUsed uint64 + Compression uint16 +} + +// report is the result of one check. +type report struct { + Partitions []partition + Slot partition + Content squashfs + MaxPercent int +} + +func (r report) percent() float64 { + return float64(r.Content.BytesUsed) * 100 / float64(r.Slot.Size) +} + +func (r report) overBudget() bool { + return r.Content.BytesUsed*100 > r.Slot.Size*uint64(r.MaxPercent) +} + +func main() { + maxPercent := flag.Int("max-percent", 60, "fail when the squashfs fills more than this share of the slot") + summary := flag.String("summary", "", "append the Markdown table to this file (for $GITHUB_STEP_SUMMARY)") + title := flag.String("title", "OS slot budget", "the heading of the table in -summary") + extract := flag.String("extract", "", "write the slot's squashfs to this file (the OS update bundle's image)") + flag.Usage = func() { + fmt.Fprintln(os.Stderr, "usage: slotbudget [-max-percent N] [-summary FILE] [-title TEXT] [-extract FILE] IMAGE.raw") + flag.PrintDefaults() + } + flag.Parse() + if flag.NArg() != 1 { + flag.Usage() + os.Exit(2) + } + + r, err := check(flag.Arg(0), *maxPercent) + if err != nil { + fmt.Fprintln(os.Stderr, "slotbudget:", err) + var bad badSlotError + if errors.As(err, &bad) { + os.Exit(1) + } + os.Exit(2) + } + + table := r.table() + fmt.Print(table) + if *summary != "" { + if err := appendFile(*summary, "### "+*title+"\n\n"+table+"\n"); err != nil { + fmt.Fprintln(os.Stderr, "slotbudget: write summary:", err) + os.Exit(2) + } + } + if r.overBudget() { + fmt.Fprintf(os.Stderr, "slotbudget: OVER BUDGET. The squashfs in %s is %s, %.1f%% of the %s slot; the budget is %d%%.\n"+ + "A slot is fixed for the life of a box, so make the image smaller, or raise the budget on purpose (BUILD.md # 1b # Disk budget).\n", + r.Slot.Name, human(r.Content.BytesUsed), r.percent(), human(r.Slot.Size), r.MaxPercent) + os.Exit(1) + } + fmt.Printf("slot budget ok: the squashfs is %.1f%% of the slot (budget %d%%)\n", r.percent(), r.MaxPercent) + if *extract != "" { + n, err := extractSlot(flag.Arg(0), r, *extract) + if err != nil { + fmt.Fprintln(os.Stderr, "slotbudget: extract:", err) + os.Exit(2) + } + fmt.Printf("slot image written: %s, %s\n", *extract, human(n)) + } +} + +// slotImageAlign is the block size mksquashfs pads a squashfs to by default. +const slotImageAlign = 4096 + +// slotImageSize is how many bytes of the slot make its image: the squashfs's +// bytes_used rounded up to 4 KiB, never past the end of the partition. +func slotImageSize(r report) uint64 { + n := (r.Content.BytesUsed + slotImageAlign - 1) / slotImageAlign * slotImageAlign + if n > r.Slot.Size { + n = r.Slot.Size + } + return n +} + +// extractSlot copies slot A's image out of the disk image at path into dst, +// and returns how many bytes it wrote. +func extractSlot(path string, r report, dst string) (uint64, error) { + src, err := os.Open(path) + if err != nil { + return 0, err + } + defer src.Close() + out, err := os.Create(dst) + if err != nil { + return 0, err + } + n := slotImageSize(r) + // A section reader stops at the end of the file with no error, so an + // image cut short inside the slot would give a short image that still + // looks fine. Count the bytes. + got, err := io.Copy(out, io.NewSectionReader(src, int64(r.Slot.Start), int64(n))) + if err == nil && uint64(got) != n { + err = fmt.Errorf("the image ends inside %s: copied %d of %d bytes", slotLabel, got, n) + } + if err != nil { + out.Close() + os.Remove(dst) + return 0, err + } + if err := out.Close(); err != nil { + return 0, err + } + return n, nil +} + +// badSlotError marks an image that was read fine but whose slot is wrong: +// missing, not a squashfs, or not compressed the way GRUB can read. +type badSlotError struct{ msg string } + +func (e badSlotError) Error() string { return e.msg } + +// check reads IMAGE and measures slot A. +func check(path string, maxPercent int) (report, error) { + if maxPercent <= 0 || maxPercent > 100 { + return report{}, fmt.Errorf("-max-percent must be 1 to 100, got %d", maxPercent) + } + f, err := os.Open(path) + if err != nil { + return report{}, err + } + defer f.Close() + + parts, err := readGPT(f) + if err != nil { + return report{}, err + } + r := report{Partitions: parts, MaxPercent: maxPercent} + found := false + for _, p := range parts { + if p.Name == slotLabel { + r.Slot, found = p, true + break + } + } + if !found { + return report{}, badSlotError{fmt.Sprintf("no partition labelled %s in %s", slotLabel, path)} + } + sq, err := readSquashfs(f, r.Slot.Start) + if err != nil { + return report{}, err + } + if sq.Compression != squashfsXZ { + return report{}, badSlotError{fmt.Sprintf("%s is a squashfs with compression id %d, not xz (%d); GRUB 2.12 reads no other compressed squashfs", slotLabel, sq.Compression, squashfsXZ)} + } + if sq.BytesUsed > r.Slot.Size { + return report{}, badSlotError{fmt.Sprintf("%s: the squashfs says it is %d bytes, more than its %d byte partition", slotLabel, sq.BytesUsed, r.Slot.Size)} + } + r.Content = sq + return r, nil +} + +// readGPT returns the used entries of the GPT in r, assuming 512-byte sectors +// (a raw image mkosi wrote). +func readGPT(r io.ReaderAt) ([]partition, error) { + hdr := make([]byte, 92) + if _, err := r.ReadAt(hdr, sectorSize); err != nil { + return nil, fmt.Errorf("read GPT header: %w", err) + } + if string(hdr[0:8]) != "EFI PART" { + return nil, errors.New("no GPT header at sector 1") + } + entriesLBA := binary.LittleEndian.Uint64(hdr[72:80]) + count := binary.LittleEndian.Uint32(hdr[80:84]) + entrySize := binary.LittleEndian.Uint32(hdr[84:88]) + if entrySize < 128 || count == 0 || count > 1024 { + return nil, fmt.Errorf("GPT header has %d entries of %d bytes", count, entrySize) + } + buf := make([]byte, int(count)*int(entrySize)) + if _, err := r.ReadAt(buf, int64(entriesLBA)*sectorSize); err != nil { + return nil, fmt.Errorf("read GPT entries: %w", err) + } + var parts []partition + zero := make([]byte, 16) + for i := 0; i < int(count); i++ { + e := buf[i*int(entrySize) : (i+1)*int(entrySize)] + if bytes.Equal(e[0:16], zero) { + continue + } + first := binary.LittleEndian.Uint64(e[32:40]) + last := binary.LittleEndian.Uint64(e[40:48]) + if last < first { + return nil, fmt.Errorf("GPT entry %d ends before it starts", i+1) + } + parts = append(parts, partition{ + Number: i + 1, + Name: gptName(e[56:128]), + Start: first * sectorSize, + Size: (last - first + 1) * sectorSize, + }) + } + if len(parts) == 0 { + return nil, errors.New("the GPT has no partitions") + } + return parts, nil +} + +// gptName decodes a GPT partition name: UTF-16LE, NUL-padded. +func gptName(b []byte) string { + u := make([]uint16, 0, len(b)/2) + for i := 0; i+1 < len(b); i += 2 { + c := binary.LittleEndian.Uint16(b[i : i+2]) + if c == 0 { + break + } + u = append(u, c) + } + return string(utf16.Decode(u)) +} + +// readSquashfs reads the squashfs 4.0 superblock at off. +func readSquashfs(r io.ReaderAt, off uint64) (squashfs, error) { + sb := make([]byte, 96) + if _, err := r.ReadAt(sb, int64(off)); err != nil { + return squashfs{}, fmt.Errorf("read squashfs superblock: %w", err) + } + if binary.LittleEndian.Uint32(sb[0:4]) != squashfsMagic { + return squashfs{}, badSlotError{fmt.Sprintf("%s does not start with a squashfs superblock", slotLabel)} + } + if major := binary.LittleEndian.Uint16(sb[28:30]); major != 4 { + return squashfs{}, badSlotError{fmt.Sprintf("%s is squashfs version %d, want 4", slotLabel, major)} + } + return squashfs{ + Compression: binary.LittleEndian.Uint16(sb[20:22]), + BytesUsed: binary.LittleEndian.Uint64(sb[40:48]), + }, nil +} + +// table renders the partitions, each as a share of the smallest disk, and the +// slot's content against its budget. +func (r report) table() string { + var b strings.Builder + b.WriteString("| # | Partition | Size | Share of 40 GB | Content | Headroom |\n") + b.WriteString("|---|---|---|---|---|---|\n") + var reserved uint64 + for _, p := range r.Partitions { + reserved += p.Size + name := p.Name + if name == "" { + name = "(no label)" + } + content, headroom := "", "" + if p.Number == r.Slot.Number { + content = fmt.Sprintf("squashfs %s (%.1f%% of the slot)", human(r.Content.BytesUsed), r.percent()) + budget := p.Size * uint64(r.MaxPercent) / 100 + if r.Content.BytesUsed <= budget { + headroom = fmt.Sprintf("%s to the %d%% budget", human(budget-r.Content.BytesUsed), r.MaxPercent) + } else { + headroom = fmt.Sprintf("%s OVER the %d%% budget", human(r.Content.BytesUsed-budget), r.MaxPercent) + } + } + fmt.Fprintf(&b, "| %d | %s | %s | %s | %s | %s |\n", p.Number, name, human(p.Size), share(p.Size), content, headroom) + } + fmt.Fprintf(&b, "| | in the image | %s | %s | | |\n", human(reserved), share(reserved)) + return b.String() +} + +func share(n uint64) string { + return fmt.Sprintf("%.2f%%", float64(n)*100/smallestDisk) +} + +// human prints n in MiB below 1 GiB and in GiB from there, with the byte count. +func human(n uint64) string { + const mib, gib = 1 << 20, 1 << 30 + if n >= gib { + return fmt.Sprintf("%.2f GiB (%d bytes)", float64(n)/gib, n) + } + return fmt.Sprintf("%.1f MiB (%d bytes)", float64(n)/mib, n) +} + +func appendFile(path, s string) error { + f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return err + } + if _, err := f.WriteString(s); err != nil { + f.Close() + return err + } + return f.Close() +} diff --git a/dev/cloud/slotbudget/main_test.go b/dev/cloud/slotbudget/main_test.go new file mode 100644 index 00000000..f08b3ac6 --- /dev/null +++ b/dev/cloud/slotbudget/main_test.go @@ -0,0 +1,307 @@ +package main + +import ( + "bufio" + "encoding/binary" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "unicode/utf16" +) + +const gib = 1 << 30 + +// testPart is one partition of a synthetic image. +type testPart struct { + name string + sizeMiB uint64 + squashfs bool + bytesUsed uint64 + compressID uint16 +} + +// writeImage writes a sparse raw disk image with a GPT holding parts, in +// order, and a squashfs superblock at the start of each squashfs partition. +// It is the shape mkosi writes, not a full GPT: no protective MBR, no backup +// header and no CRCs, which slotbudget does not read. +func writeImage(t *testing.T, parts []testPart) string { + t.Helper() + path := filepath.Join(t.TempDir(), "image.raw") + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + + const entries, entrySize = 128, 128 + hdr := make([]byte, 92) + copy(hdr, "EFI PART") + binary.LittleEndian.PutUint64(hdr[72:], 2) // entries start at LBA 2 + binary.LittleEndian.PutUint32(hdr[80:], entries) + binary.LittleEndian.PutUint32(hdr[84:], entrySize) + if _, err := f.WriteAt(hdr, sectorSize); err != nil { + t.Fatal(err) + } + + lba := uint64(2048) + for i, p := range parts { + e := make([]byte, entrySize) + e[0] = 0xAA // any non-zero type GUID + first := lba + last := first + p.sizeMiB*2048 - 1 + binary.LittleEndian.PutUint64(e[32:], first) + binary.LittleEndian.PutUint64(e[40:], last) + for j, c := range utf16.Encode([]rune(p.name)) { + binary.LittleEndian.PutUint16(e[56+2*j:], c) + } + if _, err := f.WriteAt(e, int64(2*sectorSize+i*entrySize)); err != nil { + t.Fatal(err) + } + if p.squashfs { + sb := make([]byte, 96) + binary.LittleEndian.PutUint32(sb[0:], squashfsMagic) + binary.LittleEndian.PutUint16(sb[20:], p.compressID) + binary.LittleEndian.PutUint16(sb[28:], 4) + binary.LittleEndian.PutUint64(sb[40:], p.bytesUsed) + if _, err := f.WriteAt(sb, int64(first*sectorSize)); err != nil { + t.Fatal(err) + } + } + lba = last + 1 + } + if err := f.Truncate(int64(lba * sectorSize)); err != nil { + t.Fatal(err) + } + return path +} + +// hostedImage is the hosted layout: ESP 128 MiB, BIOS boot 1 MiB, slot A 1 GiB. +func hostedImage(t *testing.T, used uint64, compression uint16) string { + return writeImage(t, []testPart{ + {name: "esp", sizeMiB: 128}, + {name: "", sizeMiB: 1}, + {name: slotLabel, sizeMiB: 1024, squashfs: true, bytesUsed: used, compressID: compression}, + }) +} + +// The measured image from CI run 36909222361: a 435 MB squashfs-xz in a 1 GiB +// slot is 40.5%, inside the 60% budget. +func TestWithinBudget(t *testing.T) { + r, err := check(hostedImage(t, 435_000_000, squashfsXZ), 60) + if err != nil { + t.Fatal(err) + } + if r.overBudget() { + t.Fatalf("435 MB in 1 GiB reported over a 60%% budget (%.1f%%)", r.percent()) + } + if r.Slot.Number != 3 || r.Slot.Size != gib { + t.Fatalf("slot = %+v, want partition 3 of 1 GiB", r.Slot) + } + if p := r.percent(); p < 40 || p > 41 { + t.Fatalf("percent = %.2f, want about 40.5", p) + } + tbl := r.table() + for _, want := range []string{"| 3 | moose-slot-a | 1.00 GiB", "2.68%", "40.5% of the slot", "to the 60% budget", "(no label)", "| | in the image |"} { + if !strings.Contains(tbl, want) { + t.Errorf("table lacks %q:\n%s", want, tbl) + } + } +} + +// One byte over 60% fails. Exactly 60% does not. +func TestOverBudgetBoundary(t *testing.T) { + limit := uint64(gib) * 60 / 100 + r, err := check(hostedImage(t, limit, squashfsXZ), 60) + if err != nil { + t.Fatal(err) + } + if r.overBudget() { + t.Fatal("exactly 60% must pass") + } + r, err = check(hostedImage(t, limit+1, squashfsXZ), 60) + if err != nil { + t.Fatal(err) + } + if !r.overBudget() { + t.Fatal("one byte over 60% must fail") + } + if !strings.Contains(r.table(), "OVER the 60% budget") { + t.Fatalf("table does not say OVER:\n%s", r.table()) + } +} + +// GRUB 2.12 reads only xz among compressed squashfs; anything else is a slot +// that does not boot, so it fails the check, not only the budget. +func TestNotXZFails(t *testing.T) { + const zstd = 6 + _, err := check(hostedImage(t, 100_000_000, zstd), 60) + var bad badSlotError + if !errors.As(err, &bad) || !strings.Contains(err.Error(), "not xz") { + t.Fatalf("err = %v, want a bad slot error about xz", err) + } +} + +func TestNotSquashfsFails(t *testing.T) { + img := writeImage(t, []testPart{ + {name: "esp", sizeMiB: 128}, + {name: slotLabel, sizeMiB: 1024}, // an ext4 slot has no hsqs magic + }) + _, err := check(img, 60) + var bad badSlotError + if !errors.As(err, &bad) || !strings.Contains(err.Error(), "squashfs superblock") { + t.Fatalf("err = %v, want a bad slot error about the superblock", err) + } +} + +func TestNoSlotFails(t *testing.T) { + img := writeImage(t, []testPart{{name: "esp", sizeMiB: 128}}) + _, err := check(img, 60) + var bad badSlotError + if !errors.As(err, &bad) { + t.Fatalf("err = %v, want a bad slot error", err) + } +} + +func TestNoGPTIsAReadError(t *testing.T) { + path := filepath.Join(t.TempDir(), "empty.raw") + if err := os.WriteFile(path, make([]byte, 4096), 0o644); err != nil { + t.Fatal(err) + } + _, err := check(path, 60) + var bad badSlotError + if err == nil || errors.As(err, &bad) { + t.Fatalf("err = %v, want a read error (exit 2), not a bad slot", err) + } +} + +// repartSizes reads Label=, Format=, Compression= and the two size bounds +// from a systemd-repart definition. +func repartSizes(t *testing.T, path string) map[string]string { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + kv := map[string]string{} + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, "[") { + continue + } + k, v, ok := strings.Cut(line, "=") + if ok { + kv[k] = v + } + } + if err := sc.Err(); err != nil { + t.Fatal(err) + } + return kv +} + +// The committed layout is the one this tool and BUILD.md # 1b # Disk budget +// describe: a 1 GiB squashfs-xz slot A and a 128 MiB ESP in the image, and the +// runtime definitions repart reads at every boot agree with them, so repart +// never tries to move or resize what the image already has. Slot B is the +// same size as slot A. +func TestCommittedLayout(t *testing.T) { + build := filepath.Join("..", "mkosi.repart") + runtime := filepath.Join("..", "mkosi.extra", "usr", "lib", "repart.d") + + slotA := repartSizes(t, filepath.Join(build, "10-slot-a.conf")) + for k, want := range map[string]string{"Label": slotLabel, "Format": "squashfs", "Compression": "xz", "SizeMinBytes": "1G", "SizeMaxBytes": "1G"} { + if slotA[k] != want { + t.Errorf("mkosi.repart/10-slot-a.conf %s=%q, want %q", k, slotA[k], want) + } + } + esp := repartSizes(t, filepath.Join(build, "00-esp.conf")) + if esp["SizeMinBytes"] != "128M" || esp["SizeMaxBytes"] != "128M" { + t.Errorf("mkosi.repart/00-esp.conf sizes = %q/%q, want 128M", esp["SizeMinBytes"], esp["SizeMaxBytes"]) + } + for file, want := range map[string]string{"00-esp.conf": "128M", "10-slot-a.conf": "1G", "20-slot-b.conf": "1G"} { + kv := repartSizes(t, filepath.Join(runtime, file)) + if kv["SizeMinBytes"] != want || kv["SizeMaxBytes"] != want { + t.Errorf("repart.d/%s sizes = %q/%q, want %s", file, kv["SizeMinBytes"], kv["SizeMaxBytes"], want) + } + } + state := repartSizes(t, filepath.Join(runtime, "30-state.conf")) + if state["Label"] != "moose-state" || state["SizeMaxBytes"] != "" { + t.Errorf("repart.d/30-state.conf = %v, want label moose-state and no size cap", state) + } +} + +// The bundle's image is slot A's squashfs, cut at bytes_used rounded up to +// 4 KiB: the first bytes of the partition, and nothing after them. +func TestExtractSlot(t *testing.T) { + const used = 10_000_001 // not a multiple of 4 KiB + img := hostedImage(t, used, squashfsXZ) + r, err := check(img, 60) + if err != nil { + t.Fatal(err) + } + // A marker inside the image part and one just past it. + f, err := os.OpenFile(img, os.O_RDWR, 0) + if err != nil { + t.Fatal(err) + } + want := uint64(10_002_432) // 2442 blocks of 4 KiB + if _, err := f.WriteAt([]byte("in"), int64(r.Slot.Start+want-2)); err != nil { + t.Fatal(err) + } + if _, err := f.WriteAt([]byte("out"), int64(r.Slot.Start+want)); err != nil { + t.Fatal(err) + } + f.Close() + + dst := filepath.Join(t.TempDir(), "rootfs.img") + n, err := extractSlot(img, r, dst) + if err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(dst) + if err != nil { + t.Fatal(err) + } + if n != want || uint64(len(got)) != want { + t.Fatalf("extracted %d bytes (file %d), want %d", n, len(got), want) + } + if binary.LittleEndian.Uint32(got[0:4]) != squashfsMagic { + t.Fatal("the extracted image does not start with the squashfs superblock") + } + if string(got[len(got)-2:]) != "in" { + t.Fatalf("the image ends with %q, want the marker from inside the squashfs", got[len(got)-2:]) + } +} + +// A squashfs that fills its slot to the last byte is never cut past the +// partition. +func TestSlotImageSizeStopsAtThePartition(t *testing.T) { + r := report{Slot: partition{Size: gib}, Content: squashfs{BytesUsed: gib - 1}} + if n := slotImageSize(r); n != gib { + t.Fatalf("size = %d, want %d", n, gib) + } +} + +// An image cut short inside the slot fails the extract and leaves no file, +// rather than writing a short slot image that looks fine. +func TestExtractSlotFailsOnAShortImage(t *testing.T) { + img := hostedImage(t, 10_000_000, squashfsXZ) + r, err := check(img, 60) + if err != nil { + t.Fatal(err) + } + if err := os.Truncate(img, int64(r.Slot.Start+5_000_000)); err != nil { + t.Fatal(err) + } + dst := filepath.Join(t.TempDir(), "rootfs.img") + if _, err := extractSlot(img, r, dst); err == nil || !strings.Contains(err.Error(), "ends inside") { + t.Fatalf("err = %v, want a short-copy error", err) + } + if _, err := os.Stat(dst); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("a failed extract left %s behind (%v)", dst, err) + } +} diff --git a/dev/cloud/stage-control-plane.sh b/dev/cloud/stage-control-plane.sh index b1ec1978..33403d52 100755 --- a/dev/cloud/stage-control-plane.sh +++ b/dev/cloud/stage-control-plane.sh @@ -11,7 +11,9 @@ # - the slim host-agent (-tags hosted, #204) + its unit + the cloud brain drop-in # - the PAM stack host-agent-real's verify-password needs # - the control-plane image bundle (brain/ui/proxy + the acmedns Caddy) + the -# first-boot loader unit (reused verbatim from the medium lane) +# first-boot loader unit (reused verbatim from the medium lane). The brain +# and UI are a build of this commit, or the last released pair from ghcr +# (MOOSE_CONTROL_PLANE_SOURCE, #566), recorded in /usr/lib/moose/control-plane.env # - the control-plane compose + caddy.json at the same host path the brain sees # - the first-boot provisioning-seed materializer + its unit # @@ -69,17 +71,49 @@ stage_control_plane() { # --- slim host-agent (-tags hosted, #204). stage_build_go "$hostagent_bin" "${REPO_ROOT}/cmd/host-agent-real/" -tags hosted - # --- control-plane image bundle. Rebuild only when absent (or forced via - # MOOSE_REBUILD_CP=1): `make control-plane-images` re-runs the brain (Go) + UI - # (Vue) docker builds, regenerating ~13 GB of BuildKit cache each time. The - # images don't change while iterating on the boot wiring, so reuse the tarballs. + # --- control-plane image bundle (#566, BUILD.md # Versioning). Which brain + # and UI the image bakes is MOOSE_CONTROL_PLANE_SOURCE: + # local (default) a build of this commit: `make control-plane-images`. + # Every run that publishes nothing, so a PR boots its own brain, + # and a control-plane release, whose images are not on ghcr yet + # (the publish job pushes the very ones the boots ran). + # released the last released control plane, CONTROL_PLANE_VERSION, pulled + # from ghcr by digest: `make control-plane-released`. An OS-only + # release bakes this, so the pair in the image is a real + # control-plane release and not unreleased code under its number. + # The bundle records where it came from (control-plane.env). Reuse it only + # when that matches what is asked for, or MOOSE_REBUILD_CP=1 forces it: + # `make control-plane-images` re-runs the brain (Go) + UI (Vue) docker + # builds, regenerating ~13 GB of BuildKit cache each time, and the images + # don't change while iterating on the boot wiring. Both bootstraps of one CI + # run stage from the same bundle, so the image that ships and the image the + # boots run bake the same bytes. + local cp_source="${MOOSE_CONTROL_PLANE_SOURCE:-local}" cp_version have="" + cp_version="$(tr -d '[:space:]' < "${REPO_ROOT}/CONTROL_PLANE_VERSION")" + case "$cp_source" in local|released) ;; *) + echo "MOOSE_CONTROL_PLANE_SOURCE='${cp_source}' (want local or released)" >&2; return 1 ;; + esac + if [ -f "$CP_BUNDLE/control-plane.env" ]; then + have="$(. "$CP_BUNDLE/control-plane.env" && echo "${MOOSE_BAKED_CP_SOURCE:-} ${MOOSE_BAKED_CP_VERSION:-}")" + fi + # The same source AND the same version: a local build stamps the brain + # with CONTROL_PLANE_VERSION, so a bump must rebuild it too. + local want="${cp_source} ${cp_version}" if [ "${MOOSE_REBUILD_CP:-0}" = "1" ] || ! ls "$CP_BUNDLE"/moose-brain.tar "$CP_BUNDLE"/moose-ui.tar \ - "$CP_BUNDLE"/caddy.tar "$CP_BUNDLE"/docker-socket-proxy.tar >/dev/null 2>&1; then - echo "building + saving control-plane image bundle (docker)..." - make -C "$REPO_ROOT" control-plane-images + "$CP_BUNDLE"/caddy.tar "$CP_BUNDLE"/docker-socket-proxy.tar >/dev/null 2>&1 \ + || [ "$have" != "$want" ]; then + if [ "$cp_source" = released ]; then + echo "pulling the released control plane ${cp_version} from ghcr by digest..." + make -C "$REPO_ROOT" control-plane-released + else + echo "building + saving control-plane image bundle (docker)..." + make -C "$REPO_ROOT" control-plane-images + fi else - echo "reusing existing control-plane image bundle (set MOOSE_REBUILD_CP=1 to force)" + echo "reusing existing control-plane image bundle (${have}; set MOOSE_REBUILD_CP=1 to force)" fi + echo "control plane baked into the image:" + grep '^MOOSE_BAKED_' "$CP_BUNDLE/control-plane.env" # --- stage mkosi.extra.wiring/ (generated; gitignored). rm -rf "$WIRING" @@ -124,6 +158,10 @@ EOF # plane integration). A tenant box is air-gapped at boot, so every image is a # local tarball; the VM never pulls. cp "$CP_BUNDLE"/*.tar "$WIRING/var/lib/moose/control-plane-images/" + # The bundle's record, in the slot: which control plane this image bakes + # and the image ID each one loads as. The boot lane checks the running + # brain and UI against it (cloud-assertions.sh step 5a). + cp "$CP_BUNDLE/control-plane.env" "$WIRING/usr/lib/moose/control-plane.env" # Hosted-only Caddy swap: the wildcard cert needs the caddy-dns/acmedns module # (ACME DNS-01 — os #207/C3b), which stock caddy:2-alpine lacks. Build the # xcaddy recipe and docker-save it OVER the *staged* caddy.tar — not the shared @@ -163,4 +201,12 @@ EOF cp "${REPO_ROOT}/dev/cloud/moose-seed-materialize.sh" "$WIRING/usr/local/bin/moose-seed-materialize.sh" chmod 0755 "$WIRING/usr/local/bin/moose-seed-materialize.sh" cp "${REPO_ROOT}/dev/cloud/moose-seed.service" "$WIRING/etc/systemd/system/" + + # The RAUC keyring, /etc/rauc/keyring.pem (#562): the release root on a + # build that publishes the OS, a throwaway root otherwise + # (MOOSE_RAUC_KEYRING, dev/cloud/rauc.sh). A release build without the + # committed release root fails here, before mkosi runs. + # shellcheck source=dev/cloud/rauc.sh + . "${REPO_ROOT}/dev/cloud/rauc.sh" + rauc_stage_keyring "$WIRING" } diff --git a/dev/cloud/test/bootstrap.sh b/dev/cloud/test/bootstrap.sh index d3be6ba0..44109f46 100755 --- a/dev/cloud/test/bootstrap.sh +++ b/dev/cloud/test/bootstrap.sh @@ -33,7 +33,17 @@ WIRING="${CLOUD_DIR}/mkosi.extra.wiring" # shared production wiring (ExtraTree o PKGMNGR="${TEST_DIR}/mkosi.pkgmngr" CP_BUNDLE="${REPO_ROOT}/.dev/control-plane" CANARY="${WORK}/.cloud-boot-ready" -CANARY_VERSION="v25" # bump when staging/mkosi.conf/repart changes require a clean rebuild +CANARY_VERSION="v30" # bump when staging/mkosi.conf/repart changes require a clean rebuild +# A change to the OS package lock (#560) must rebuild too, so all three lock +# files are part of the canary. The resolved list is in it as well: a re-run +# after only the list changed must not exit early and skip os_lock_check below. +# The keyring mode is in it too (#562): a build for release bakes another +# /etc/rauc/keyring.pem. +CANARY_VERSION="${CANARY_VERSION}-rauc-${MOOSE_RAUC_KEYRING:-throwaway}" +# And where the brain and UI come from (#566): a build of this commit, or the +# last released pair from ghcr, at which control-plane version. +CANARY_VERSION="${CANARY_VERSION}-cp-${MOOSE_CONTROL_PLANE_SOURCE:-local}-$(tr -d '[:space:]' < "${REPO_ROOT}/CONTROL_PLANE_VERSION")" +CANARY_VERSION="${CANARY_VERSION}-lock-$(cat "${REPO_ROOT}/dev/os-lock/debian-snapshot" "${REPO_ROOT}/dev/os-lock/third-party.lock" "${REPO_ROOT}/dev/os-lock/cloud-packages.lock" | sha256sum | cut -c1-12)" IMAGE_OUT="${WORK}/moose-cloud.raw" if [ "${EUID:-$(id -u)}" -ne 0 ]; then @@ -60,13 +70,14 @@ for tool in mkosi qemu-system-x86_64 qemu-img curl python3 docker; do done # host-agent-real is a CGO binary (PAM verify is kept in hosted); needs the headers. [ -f /usr/include/security/pam_appl.h ] || missing+=("libpam0g-dev (PAM headers for host-agent-real)") -# OVMF (UEFI firmware) — location varies by distro. -OVMF_CODE="" -for cand in /usr/share/OVMF/OVMF_CODE.fd /usr/share/ovmf/OVMF.fd \ - /usr/share/OVMF/OVMF.fd /usr/share/edk2-ovmf/x64/OVMF_CODE.fd; do - [ -r "$cand" ] && { OVMF_CODE="$cand"; break; } -done -[ -n "$OVMF_CODE" ] || missing+=("ovmf (UEFI firmware — package: ovmf)") +# OVMF (UEFI firmware): a CODE image and its VARS template, as a pair, from the +# list in dev/cloud/ovmf.sh, which run-cloud-tests.sh uses too (#575). Only +# when UEFI is one of the firmwares the boots run under. +# shellcheck source=dev/cloud/ovmf.sh +. "${CLOUD_DIR}/ovmf.sh" +if ovmf_wanted && ! ovmf_find; then + missing+=("ovmf (UEFI firmware with its VARS template, package: ovmf)") +fi if [ ${#missing[@]} -gt 0 ]; then cat >&2 < "$EXTRA/usr/lib/moose/etc-keep.d/boot-lane.list" <<'EOF' +# The boot lane's own writes into /etc (dev/cloud/test/bootstrap.sh). +systemd/system/host-agent.service.d +EOF + +# The OS update trial's safety net fires after 90 s here instead of 15 min +# (#563), so the os-revert boot does not sit out a quarter of an hour. A +# healthy trial boot marks its slot good about 17 s after the switch in CI +# (run 37031756819), which the os-update boot proves under the same setting. +# The image that ships keeps 15 min. +# What GRUB left in the grubenv for this boot, logged before host-agent can +# mark anything (#563): GRUB sets the booted slot's TRY=1, and the boot lane +# checks it did, under both firmwares (#575). RequiresMountsFor: the ESP is +# `nofail` in fstab, so local-fs.target does not wait for it, and a read +# before the mount found no grubenv (run 37063254219). +cat > "$EXTRA/etc/systemd/system/moose-test-grubenv.service" <<'EOF' +[Unit] +Description=moose test: log the grubenv GRUB left for this boot +Before=host-agent.service +After=local-fs.target +RequiresMountsFor=/efi +DefaultDependencies=no + +[Service] +Type=oneshot +ExecStart=/bin/sh -c 'echo "grubenv at boot: $(grub-editenv /efi/grub/grubenv list | tr "\n" " ")"' + +[Install] +WantedBy=multi-user.target +EOF +# A slot that dies before userspace (#575), for the last stage of the +# os-revert boot. This initramfs hook runs right after moose-state, so the +# state partition is mounted at ${rootmnt}/state. When the state partition +# holds moose-test/panic-slot- for the booted slot, it leaves a note and +# crashes the kernel: a real kernel panic, before systemd, which panic=10 +# turns into a reboot. GRUB must then skip the slot, because it saved the +# slot's TRY=1 before it booted it. The hook is in the boot-proof image only, +# so in the test bundle too (it is this image's slot), never in the image +# that ships. mkosi copies the extra trees before any postinst runs, and +# dev/cloud/mkosi.postinst.chroot builds the initramfs, so the hook is in it. +# If it ever is not, slot B boots in full and stage 4 fails on the slot it +# booted. +mkdir -p "$EXTRA/etc/initramfs-tools/scripts/local-bottom" +cat > "$EXTRA/etc/initramfs-tools/scripts/local-bottom/moose-test-panic" <<'EOF' +#!/bin/sh +PREREQ="moose-state" +prereqs() { echo "$PREREQ"; } +case "$1" in + prereqs) prereqs; exit 0 ;; +esac +. /scripts/functions +slot="" +for arg in $(cat /proc/cmdline); do + case "$arg" in rauc.slot=*) slot="${arg#rauc.slot=}" ;; esac +done +[ -n "$slot" ] || exit 0 +[ -e "${rootmnt}/state/moose-test/panic-slot-${slot}" ] || exit 0 +echo "moose-test: slot ${slot} panics before userspace, as the os-revert boot asked" > /dev/kmsg +echo "${slot}" > "${rootmnt}/state/moose-test/panicked-slot-${slot}" +echo s > /proc/sysrq-trigger +sleep 2 +echo c > /proc/sysrq-trigger +# Not reached when the crash works. If it did not, fail the initramfs instead: +# panic= reboots that too. +panic "moose-test: slot ${slot}: the kernel crash did not happen" +EOF +chmod 0755 "$EXTRA/etc/initramfs-tools/scripts/local-bottom/moose-test-panic" + +mkdir -p "$EXTRA/etc/systemd/system/moose-os-trial.timer.d" +cat > "$EXTRA/etc/systemd/system/moose-os-trial.timer.d/10-cloud-test.conf" <<'EOF' +[Timer] +OnBootSec= +OnBootSec=90s +EOF # --- 3b. app-install fixtures for the access-mode e2e (#308) — TEST-LANE ONLY. The # access boot installs whoami air-gapped and drives the per-app forward-auth access @@ -258,15 +348,13 @@ docker pull "$POSTGRES_REF" docker tag "$POSTGRES_REF" postgres:16 docker save postgres:16 -o "$EXTRA/var/lib/moose/test-images/postgres-16.tar" -# --- 4. Docker apt repo for the build's package manager (trixie pocket — the -# cloud image is Release=trixie). Build-host network only; the VM never apt-installs. -rm -rf "$PKGMNGR" -mkdir -p "$PKGMNGR/etc/apt/keyrings" "$PKGMNGR/etc/apt/sources.list.d" -curl -fsSL https://download.docker.com/linux/debian/gpg -o "$PKGMNGR/etc/apt/keyrings/docker.asc" -chmod a+r "$PKGMNGR/etc/apt/keyrings/docker.asc" -cat > "$PKGMNGR/etc/apt/sources.list.d/docker.list" <<'EOF' -deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable -EOF +# --- 4. the OS package lock (#560): the same apt sources, snapshot and pins as +# the lean build (dev/os-lock/os-lock.sh), so this image installs the versions +# the shipped image does. Build-host network only; the VM never apt-installs. +# shellcheck source=dev/os-lock/os-lock.sh +. "${REPO_ROOT}/dev/os-lock/os-lock.sh" +stage_os_lock_apt "$PKGMNGR" +OS_LOCK_SNAPSHOT="$(os_lock_snapshot)" # --- 5. mkosi build (from dev/cloud/test; Include=.. pulls in the production base # + its wiring). Re-own the staged trees + work dir to the caller; mkosi runs as @@ -294,13 +382,33 @@ if [ -z "$MKOSI_INTERPRETER" ] && [ -n "$CALLER_HOME" ]; then fi [ -n "$MKOSI_INTERPRETER" ] || { echo "mkosi needs python >=3.10; none found" >&2; exit 1; } +# MOOSE_MKOSI_TOOLS_TREE (opt-in): use this tools tree instead of building the +# default one. CI sets it to dev/cloud/mkosi.tools, which the lean build made a +# minute earlier in the same job from the same settings, and saves about 40 s. +# Off by default: a local tree from an older build is not checked for being +# current when it is named by path. +tools_tree_args=() +if [ -n "${MOOSE_MKOSI_TOOLS_TREE:-}" ]; then + [ -d "$MOOSE_MKOSI_TOOLS_TREE" ] || { echo "MOOSE_MKOSI_TOOLS_TREE='${MOOSE_MKOSI_TOOLS_TREE}' is not a directory" >&2; exit 1; } + echo "using the tools tree ${MOOSE_MKOSI_TOOLS_TREE} (MOOSE_MKOSI_TOOLS_TREE)" + tools_tree_args=(--tools-tree "$MOOSE_MKOSI_TOOLS_TREE") +fi + if [ -n "$CALLER" ]; then sudo -u "$CALLER" env "MKOSI_INTERPRETER=$MKOSI_INTERPRETER" \ - "$MKOSI_BIN" --directory "$TEST_DIR" --force build + "$MKOSI_BIN" --directory "$TEST_DIR" "${tools_tree_args[@]}" --snapshot "$OS_LOCK_SNAPSHOT" --force build else - MKOSI_INTERPRETER="$MKOSI_INTERPRETER" "$MKOSI_BIN" --directory "$TEST_DIR" --force build + MKOSI_INTERPRETER="$MKOSI_INTERPRETER" "$MKOSI_BIN" --directory "$TEST_DIR" "${tools_tree_args[@]}" --snapshot "$OS_LOCK_SNAPSHOT" --force build fi +# The boot-proof image is a second, separate build of the same package set, so +# it must resolve to the same committed lock as the lean one. In CI this is the +# "two builds of one commit install the same versions" proof (#560). Never in +# record mode: only the lean build records. +TEST_MANIFEST="$(ls -1 "$WORK"/*.manifest 2>/dev/null | head -n1 || true)" +[ -n "$TEST_MANIFEST" ] || { echo "no package manifest under $WORK" >&2; exit 1; } +MOOSE_OS_LOCK_RECORD= os_lock_check "$TEST_MANIFEST" "$WORK/cloud-packages.lock" + # mkosi writes to OutputDirectory=.dev/cloud-boot. Confirm the raw exists. if [ ! -f "$IMAGE_OUT" ]; then for cand in "${WORK}/moose-cloud.raw" "${WORK}/moose-cloud"; do @@ -313,6 +421,14 @@ if [ ! -f "$IMAGE_OUT" ]; then exit 1 fi +# The slot budget, reported but not gated (#561, dev/cloud/slot-budget.sh). The +# 60% gate is on the image that ships (dev/cloud/bootstrap.sh); this one also +# carries test-only images, so it only has to fit its 1 GiB slot, which +# systemd-repart already enforces when it builds the image. +# shellcheck source=dev/cloud/slot-budget.sh +. "${CLOUD_DIR}/slot-budget.sh" +slot_budget_check "$IMAGE_OUT" 100 "OS slot use (the boot-proof image, with test-only images; not gated)" + echo -n "$CANARY_VERSION" > "$CANARY" [ -n "$CALLER" ] && chown "$CALLER":"$(id -gn "$CALLER")" "$CANARY" "$IMAGE_OUT" 2>/dev/null || true echo "cloud boot-proof image ready at $IMAGE_OUT" diff --git a/dev/cloud/test/build-os-test-bundle.sh b/dev/cloud/test/build-os-test-bundle.sh new file mode 100755 index 00000000..bfa02945 --- /dev/null +++ b/dev/cloud/test/build-os-test-bundle.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# Build the test-only OS bundle the os-update and os-revert boots install (#563). +# +# sudo -E dev/cloud/test/build-os-test-bundle.sh BOOT-PROOF.raw OUTDIR +# +# The bundle holds slot A of the boot-proof image, repacked with one change: a +# host-agent stamped one patch release above VERSION. So on slot B the box +# reports that version, and the update loop sees it as current. Nothing in it +# ever ships: it is signed with this checkout's throwaway key (dev/cloud/rauc.sh), +# which only a boot-proof image of the same build trusts. +# +# Kept cheap on purpose (the maintainer's call for #563): the slot is repacked +# with gzip at level 1, not xz, which GRUB also reads; the baked image tarballs +# under /var/lib/moose are left out, because a slot's /var/lib/moose is copied +# to the state partition only at a box's first boot and slot B never has one. +# +# Writes OUTDIR/os-test.raucb, OUTDIR/os-test.sha256 (the digest) and +# OUTDIR/os-test.version (the version in the bundle and in its host-agent). +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)" +# shellcheck source=dev/cloud/rauc.sh +. "${REPO_ROOT}/dev/cloud/rauc.sh" + +image="$(realpath "${1:?usage: build-os-test-bundle.sh BOOT-PROOF.raw OUTDIR}")" +out="${2:?usage: build-os-test-bundle.sh BOOT-PROOF.raw OUTDIR}" +GO="${GO:-$(command -v go || true)}" +[ -n "$GO" ] || { echo "build-os-test-bundle: go not found (set GO)" >&2; exit 1; } +CALLER="${SUDO_USER:-}" +t0=$(date +%s) + +rm -rf "$out" +mkdir -p "$out/work" +out="$(realpath "$out")" +[ -n "$CALLER" ] && chown -R "$CALLER" "$out" + +base="$(tr -d '[:space:]' < "${REPO_ROOT}/VERSION")" +IFS=. read -r maj min pat <<<"$base" +next="${maj}.${min}.$((pat + 1))" + +as_caller() { if [ -n "$CALLER" ]; then sudo -u "$CALLER" "$@"; else "$@"; fi; } + +# The host-agent for the new slot, built the way stage-control-plane.sh builds +# the baked one, with the next patch number. +commit="$(as_caller git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)" +as_caller env CGO_ENABLED=1 CGO_CFLAGS=-D_GNU_SOURCE "$GO" build -C "$REPO_ROOT" -tags hosted \ + -ldflags "-X github.com/onmoose/os/internal/version.Version=${next} -X github.com/onmoose/os/internal/version.Commit=${commit}" \ + -o "$out/work/host-agent-real" ./cmd/host-agent-real/ +as_caller "$GO" build -C "$REPO_ROOT" -o "$out/work/slotbudget" ./dev/cloud/slotbudget +"$out/work/slotbudget" -extract "$out/work/slot.img" "$image" >/dev/null +rm -f "$out/work/slotbudget" + +rauc_throwaway_ca +cp "$RAUC_THROWAWAY_DIR/root-ca.pem" "$RAUC_THROWAWAY_DIR/signer.pem" "$RAUC_THROWAWAY_DIR/signer.key" "$out/work/" + +rauc_run "$out/work" ' + s=$(date +%s) + unsquashfs -q -n -d root slot.img + rm -f slot.img + echo "unsquashfs: $(( $(date +%s) - s )) s" + install -m 0755 host-agent-real root/usr/lib/moose/host-agent-real + # Left empty, not removed: state-setup binds onto these paths. + find root/var/lib/moose -mindepth 1 -delete + s=$(date +%s) + mkdir -p bundle + mksquashfs root bundle/rootfs.img -comp gzip -Xcompression-level 1 -noappend -quiet + echo "mksquashfs (gzip level 1): $(( $(date +%s) - s )) s, $(stat -c %s bundle/rootfs.img) bytes" + compatible="$(sed -n "s/^compatible=//p" root/etc/rauc/system.conf | head -n1)" + cp root/etc/rauc/system.conf system.conf + rm -rf root + cat > bundle/manifest.raucm </dev/null +' +mv "$out/work/os-test.raucb" "$out/os-test.raucb" +rm -rf "$out/work" +sha256sum "$out/os-test.raucb" | cut -d' ' -f1 > "$out/os-test.sha256" +echo "$next" > "$out/os-test.version" +[ -n "$CALLER" ] && chown -R "$CALLER":"$(id -gn "$CALLER")" "$out" 2>/dev/null || true +bytes="$(stat -c %s "$out/os-test.raucb")" +echo "os test bundle: ${next}, ${bytes} bytes, sha256 $(cat "$out/os-test.sha256"), built in $(( $(date +%s) - t0 )) s" +{ + echo "### OS test bundle (#563, boot-proof only)" + echo "" + echo "Version ${next}, $(awk -v a="$bytes" 'BEGIN { printf "%.1f MB", a / 1e6 }'), built in $(( $(date +%s) - t0 )) s. Slot repacked with gzip level 1; never shipped." +} >> "${GITHUB_STEP_SUMMARY:-/dev/null}" diff --git a/dev/cloud/test/mkosi.postinst.chroot b/dev/cloud/test/mkosi.postinst.chroot index 7f44c74d..c2c1f57e 100755 --- a/dev/cloud/test/mkosi.postinst.chroot +++ b/dev/cloud/test/mkosi.postinst.chroot @@ -21,6 +21,7 @@ enable_unit() { # polls it, and writes its verdict to /dev/console for the QEMU harness to grep. # Test-lane only — the production image ships no assertions oneshot. enable_unit /etc/systemd/system/moose-cloud-assertions.service moose-cloud-assertions.service +enable_unit /etc/systemd/system/moose-test-grubenv.service moose-test-grubenv.service # Make the staged self-check executable (ExtraTrees preserves mode, but be # defensive — a clobbered mode would otherwise fail silently at boot). diff --git a/dev/cloud/test/moose-cloud-assertions.service b/dev/cloud/test/moose-cloud-assertions.service index 7c271f18..ee44a6ad 100644 --- a/dev/cloud/test/moose-cloud-assertions.service +++ b/dev/cloud/test/moose-cloud-assertions.service @@ -35,6 +35,7 @@ ImportCredential=moose.sso_token # other boot, like the first. ImportCredential=moose.sso_token2 ImportCredential=moose.sso_token3 +ImportCredential=moose.os_test # The script polls the control plane up (docker load + brain bootstrap + compose # up race this unit) and can run for minutes; the default 90s start timeout would # kill it mid-poll before it writes a verdict. No timeout — the QEMU harness owns diff --git a/dev/control-plane/bundle-record.sh b/dev/control-plane/bundle-record.sh new file mode 100755 index 00000000..547d22ea --- /dev/null +++ b/dev/control-plane/bundle-record.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# Write the record of a control-plane image bundle (#566): where the brain and +# UI tarballs came from, and the image ID each one loads as. +# +# bundle-record.sh local +# bundle-record.sh released +# +# holds moose-brain.tar and moose-ui.tar (`make control-plane-images` or +# `make control-plane-released`). The record goes to /control-plane.env. +# dev/cloud/stage-control-plane.sh reads it to decide whether the bundle can be +# reused, and bakes it into the image as /usr/lib/moose/control-plane.env, +# where the boot lane checks the running brain and UI against it. +# +# The image ID is the sha256 of the image config, read from the tarball's own +# manifest.json, so it does not depend on which image store the build host's +# Docker uses. A box loads the tarball into Docker's classic store (the +# user-namespace remap turns the containerd store off, BUILD.md # 1), where the +# image ID is exactly that digest. +set -euo pipefail + +usage() { echo "usage: bundle-record.sh local | released " >&2; exit 2; } +[ "$#" -ge 3 ] || usage +dir="$1"; source="$2"; version="$3" +brain_ref=""; ui_ref="" +case "$source" in + local) [ "$#" -eq 3 ] || usage ;; + released) [ "$#" -eq 5 ] || usage; brain_ref="$4"; ui_ref="$5" ;; + *) usage ;; +esac +case "$version" in + [0-9]*.[0-9]*.[0-9]*) ;; + *) echo "bundle-record: '${version}' is not a version" >&2; exit 1 ;; +esac + +# config_id TAR: "sha256:" of the one image in TAR. Docker 25+ writes +# "Config": "blobs/sha256/", older Docker ".json". +config_id() { + tar -xOf "$1" manifest.json | python3 -c ' +import json, re, sys +m = json.load(sys.stdin) +if len(m) != 1: + sys.exit("want one image in the tarball, found %d" % len(m)) +c = m[0]["Config"] +h = re.fullmatch(r"(?:blobs/sha256/)?([0-9a-f]{64})(?:\.json)?", c) +if not h: + sys.exit("cannot read a config digest from %r" % c) +print("sha256:" + h.group(1))' +} + +brain_id="$(config_id "$dir/moose-brain.tar")" +ui_id="$(config_id "$dir/moose-ui.tar")" + +tmp="$dir/.control-plane.env.tmp" +cat > "$tmp" < [] +# +# Resolves ghcr.io//brain and ghcr.io//ui at image tag v +# to their digests ONCE (dev/release/ghcr-resolve.sh), pulls each by digest, +# tags it with the local name the box runs (moose-brain:dev, moose-ui:dev: the +# brain drop-in and the control-plane compose name these, and a box never +# pulls them), and docker-saves it into . Then it checks that each +# tarball holds the image config the registry named, and writes the bundle +# record (bundle-record.sh) with the two pinned refs. +# +# defaults to onmoose. Called by `make control-plane-released`, which +# passes CONTROL_PLANE_VERSION: the last released control plane, so an OS +# release bakes a pair that really is a control-plane release (BUILD.md +# # Versioning). Fails when the tag is not on ghcr, which is the case when the +# control-plane release of that number did not publish: finish it first. +set -euo pipefail + +[ "$#" -ge 2 ] && [ "$#" -le 3 ] || { echo "usage: pull-released.sh []" >&2; exit 2; } +dir="$1"; version="$2"; owner="${3:-onmoose}" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +resolve="${here}/../release/ghcr-resolve.sh" + +mkdir -p "$dir" +# A failed pull must not leave a record that names what is no longer there. +rm -f "$dir/control-plane.env" + +declare -A ref want +for name in brain ui; do + read -r digest config < <("$resolve" "${owner}/${name}" "v${version}") || { + echo "pull-released: cannot resolve ghcr.io/${owner}/${name}:v${version}; is control-plane ${version} published?" >&2 + exit 1 + } + remote="ghcr.io/${owner}/${name}@${digest}" + echo "control plane ${version}: ${name} is ${remote} (config ${config})" + docker pull --platform linux/amd64 "$remote" + docker tag "$remote" "moose-${name}:dev" + docker save "moose-${name}:dev" -o "$dir/moose-${name}.tar" + ref[$name]="$remote" + want[$name]="$config" +done + +bash "${here}/bundle-record.sh" "$dir" released "$version" "${ref[brain]}" "${ref[ui]}" + +# The tarballs must hold the configs the registry named, or the record (and +# the boot check that reads it) would describe other bytes than ghcr serves. +# shellcheck disable=SC1091 +. "$dir/control-plane.env" +for pair in "brain:${MOOSE_BAKED_BRAIN_ID}" "ui:${MOOSE_BAKED_UI_ID}"; do + name="${pair%%:*}"; got="${pair#*:}" + if [ "$got" != "${want[$name]}" ]; then + rm -f "$dir/control-plane.env" + echo "pull-released: moose-${name}.tar holds config ${got}, but ghcr names ${want[$name]}" >&2 + exit 1 + fi +done +echo "pull-released: the bundle holds control plane ${version} as published" diff --git a/dev/dev-go.sh b/dev/dev-go.sh index 3e8462d3..f03fa34e 100755 --- a/dev/dev-go.sh +++ b/dev/dev-go.sh @@ -26,6 +26,9 @@ set -uo pipefail GO=${GO:-go} DEV_DIR=${DEV_DIR:-.dev} LDFLAGS=${LDFLAGS:-} +# The brain is stamped from CONTROL_PLANE_VERSION, host-agent from VERSION +# (BUILD.md # Versioning), so each gets its own flags. +BRAIN_LDFLAGS=${BRAIN_LDFLAGS:-} WATCH_DIRS=${MOOSE_DEV_WATCH_DIRS:-cmd internal} DEBOUNCE=${MOOSE_DEV_DEBOUNCE:-10} POLL=${MOOSE_DEV_POLL:-2} @@ -84,7 +87,7 @@ build_go() { mkdir -p "$STAGE" touch "$STAMP" $GO build -ldflags "$LDFLAGS" -o "$STAGE/host-agent" ./cmd/host-agent || return 1 - $GO build -ldflags "$LDFLAGS" -o "$STAGE/brain" ./cmd/brain || return 1 + $GO build -ldflags "$BRAIN_LDFLAGS" -o "$STAGE/brain" ./cmd/brain || return 1 } # -newermt with a relative time is a GNU find extension. Without it the loop diff --git a/dev/os-lock/cloud-accounts.lock b/dev/os-lock/cloud-accounts.lock new file mode 100644 index 00000000..f9eebdc3 --- /dev/null +++ b/dev/os-lock/cloud-accounts.lock @@ -0,0 +1,73 @@ +# The account and group ids of the hosted image (BUILD.md # 1b, rule 3). +# A box keeps them for life, so dev/cloud/build-bundle.sh fails a build whose +# image differs. Edited by hand: add a new account's line; never change an id. +# "g NAME GID" and "u NAME UID GID", sorted. +g _ssh 101 +g adm 4 +g audio 29 +g backup 34 +g bin 2 +g cdrom 24 +g clock 994 +g daemon 1 +g dialout 20 +g dip 30 +g disk 6 +g docker 991 +g fax 21 +g floppy 25 +g games 60 +g input 996 +g irc 39 +g kmem 15 +g kvm 993 +g list 38 +g lp 7 +g mail 8 +g man 12 +g messagebus 997 +g news 9 +g nogroup 65534 +g operator 37 +g plugdev 46 +g proxy 13 +g render 992 +g root 0 +g sasl 45 +g sgx 995 +g shadow 42 +g src 40 +g staff 50 +g sudo 27 +g sys 3 +g systemd-journal 999 +g systemd-network 998 +g tape 26 +g tty 5 +g users 100 +g utmp 43 +g uucp 10 +g video 44 +g voice 22 +g www-data 33 +u _apt 42 65534 +u backup 34 34 +u bin 2 2 +u daemon 1 1 +u games 5 60 +u irc 39 39 +u list 38 38 +u lp 7 7 +u mail 8 8 +u man 6 12 +u messagebus 997 997 +u news 9 9 +u nobody 65534 65534 +u proxy 13 13 +u root 0 0 +u sshd 990 65534 +u sync 4 65534 +u sys 3 3 +u systemd-network 998 998 +u uucp 10 10 +u www-data 33 33 diff --git a/dev/os-lock/cloud-packages.lock b/dev/os-lock/cloud-packages.lock new file mode 100644 index 00000000..6cab590a --- /dev/null +++ b/dev/os-lock/cloud-packages.lock @@ -0,0 +1,197 @@ +# The package list the hosted cloud image resolved to (dev/cloud/mkosi.conf), +# installed from the snapshot in debian-snapshot and the pins in third-party.lock. +# Generated from mkosi's JSON manifest by dev/os-lock/oslock; do not edit by hand. +# The build fails when it resolves to a different list (BUILD.md # 1b # The OS +# package lock). One package per line: name version architecture. +adduser 3.152 all +base-files 13.8+deb13u7 amd64 +base-passwd 3.6.7 amd64 +bash 5.2.37-2+b10 amd64 +bsdutils 1:2.41.5-0+deb13u1 amd64 +ca-certificates 20250419 all +containerd.io 2.3.6-1~debian.13~trixie amd64 +coreutils 9.7-3 amd64 +cpio 2.15+dfsg-2 amd64 +dash 0.5.12-12 amd64 +dbus-bin 1.16.2-2 amd64 +dbus-daemon 1.16.2-2 amd64 +dbus-session-bus-common 1.16.2-2 all +dbus-system-bus-common 1.16.2-2 all +dbus 1.16.2-2 amd64 +debconf 1.5.91 all +debianutils 5.23.2 amd64 +diffutils 1:3.10-4 amd64 +dmsetup 2:1.02.205-2 amd64 +docker-ce-cli 5:29.8.2-1~debian.13~trixie amd64 +docker-ce 5:29.8.2-1~debian.13~trixie amd64 +docker-compose-plugin 5.6.0-1~debian.13~trixie amd64 +dpkg 1.22.22 amd64 +dracut-install 106-6 amd64 +e2fsprogs 1.47.2-3+b12 amd64 +findutils 4.10.0-3 amd64 +gawk 1:5.2.1-2+b1 amd64 +gcc-14-base 14.2.0-19 amd64 +gettext-base 0.23.1-2 amd64 +grep 3.11-4 amd64 +grub-common 2.12-9+deb13u2 amd64 +grub-efi-amd64-bin 2.12-9+deb13u2 amd64 +grub-efi-amd64-unsigned 2.12-9+deb13u2 amd64 +grub-pc-bin 2.12-9+deb13u2 amd64 +gzip 1.13-1+deb13u1 amd64 +hostname 3.25 amd64 +init-system-helpers 1.69~deb13u1 all +initramfs-tools-bin 0.148.4 amd64 +initramfs-tools-core 0.148.4 all +initramfs-tools 0.148.4 all +iproute2 6.15.0-1 amd64 +iptables 1.8.11-2 amd64 +klibc-utils 2.0.14-1 amd64 +kmod 34.2-2 amd64 +less 668-1 amd64 +libacl1 2.3.2-2+b1 amd64 +libapparmor1 4.1.0-1 amd64 +libatomic1 14.2.0-19 amd64 +libattr1 1:2.5.2-3 amd64 +libaudit-common 1:4.0.2-2+deb13u1 all +libaudit1 1:4.0.2-2+deb13u1 amd64 +libblkid1 2.41.5-0+deb13u1 amd64 +libbpf1 1:1.5.0-3 amd64 +libbrotli1 1.1.0-2+b7 amd64 +libbsd0 0.12.2-2 amd64 +libbz2-1.0 1.0.8-6 amd64 +libc-bin 2.41-12+deb13u4 amd64 +libc6 2.41-12+deb13u4 amd64 +libcap-ng0 0.8.5-4+b1 amd64 +libcap2-bin 1:2.75-10+deb13u1+b3 amd64 +libcap2 1:2.75-10+deb13u1+b3 amd64 +libcbor0.10 0.10.2-2 amd64 +libcom-err2 1.47.2-3+b12 amd64 +libcrypt1 1:4.4.38-1 amd64 +libcurl3t64-gnutls 8.14.1-2+deb13u5 amd64 +libdb5.3t64 5.3.28+dfsg2-9 amd64 +libdbus-1-3 1.16.2-2 amd64 +libdebconfclient0 0.280 amd64 +libdevmapper1.02.1 2:1.02.205-2 amd64 +libedit2 3.1-20250104-1 amd64 +libefiboot1t64 38-3.1+b1 amd64 +libefivar1t64 38-3.1+b1 amd64 +libelf1t64 0.192-4 amd64 +libexpat1 2.8.3-1~deb13u1 amd64 +libext2fs2t64 1.47.2-3+b12 amd64 +libfdisk1 2.41.5-0+deb13u1 amd64 +libffi8 3.4.8-2 amd64 +libfido2-1 1.15.0-1+b1 amd64 +libfreetype6 2.13.3+dfsg-1+deb13u1 amd64 +libfuse3-4 3.17.2-3 amd64 +libgcc-s1 14.2.0-19 amd64 +libglib2.0-0t64 2.84.4-3~deb13u5 amd64 +libgmp10 2:6.3.0+dfsg-3 amd64 +libgnutls30t64 3.8.9-3+deb13u4 amd64 +libgssapi-krb5-2 1.21.3-5+deb13u1 amd64 +libhogweed6t64 3.10.1-1 amd64 +libidn2-0 2.3.8-2 amd64 +libip4tc2 1.8.11-2 amd64 +libip6tc2 1.8.11-2 amd64 +libjansson4 2.14-2+b3 amd64 +libjson-glib-1.0-0 1.10.6+ds-2 amd64 +libjson-glib-1.0-common 1.10.6+ds-2 all +libk5crypto3 1.21.3-5+deb13u1 amd64 +libkeyutils1 1.6.3-6 amd64 +libklibc 2.0.14-1 amd64 +libkmod2 34.2-2 amd64 +libkrb5-3 1.21.3-5+deb13u1 amd64 +libkrb5support0 1.21.3-5+deb13u1 amd64 +liblastlog2-2 2.41.5-0+deb13u1 amd64 +libldap2 2.6.10+dfsg-1 amd64 +liblzma5 5.8.1-1+deb13u1 amd64 +libmd0 1.1.0-2+b1 amd64 +libmnl0 1.0.5-3 amd64 +libmount1 2.41.5-0+deb13u1 amd64 +libmpfr6 4.2.2-1 amd64 +libncursesw6 6.5+20250216-2 amd64 +libnetfilter-conntrack3 1.1.0-1 amd64 +libnettle8t64 3.10.1-1 amd64 +libnfnetlink0 1.0.2-3 amd64 +libnftables1 1.1.3-1 amd64 +libnftnl11 1.2.9-1 amd64 +libnghttp2-14 1.64.0-1.1+deb13u1 amd64 +libnghttp3-9 1.8.0-1 amd64 +libngtcp2-16 1.11.0-1+deb13u1 amd64 +libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1 amd64 +libnl-3-200 3.7.0-2 amd64 +libnl-genl-3-200 3.7.0-2 amd64 +libp11-kit0 0.25.5-3 amd64 +libpam-modules-bin 1.7.0-5 amd64 +libpam-modules 1.7.0-5 amd64 +libpam-runtime 1.7.0-5 all +libpam0g 1.7.0-5 amd64 +libpcre2-8-0 10.46-1~deb13u3 amd64 +libpng16-16t64 1.6.48-1+deb13u6 amd64 +libproc2-0 2:4.0.4-9 amd64 +libpsl5t64 0.21.2-1.1+b1 amd64 +libreadline8t64 8.2-6 amd64 +librtmp1 2.4+20151223.gitfa8646d.1-2+b5 amd64 +libsasl2-2 2.1.28+dfsg1-9 amd64 +libsasl2-modules-db 2.1.28+dfsg1-9 amd64 +libseccomp2 2.6.0-2 amd64 +libselinux1 3.8.1-1 amd64 +libsemanage-common 3.8.1-1 all +libsemanage2 3.8.1-1 amd64 +libsepol2 3.8.1-1 amd64 +libsigsegv2 2.14-1+b2 amd64 +libsmartcols1 2.41.5-0+deb13u1 amd64 +libsqlite3-0 3.46.1-7+deb13u2 amd64 +libss2 1.47.2-3+b12 amd64 +libssh2-1t64 1.11.1-1+deb13u2 amd64 +libssl3t64 3.5.7-1~deb13u3 amd64 +libsystemd-shared 257.13-1~deb13u1 amd64 +libsystemd0 257.13-1~deb13u1 amd64 +libtasn1-6 4.20.0-2+deb13u1 amd64 +libtext-charwidth-perl 0.04-11+b4 amd64 +libtext-wrapi18n-perl 0.06-10 all +libtinfo6 6.5+20250216-2 amd64 +libtirpc-common 1.3.6+ds-1 all +libtirpc3t64 1.3.6+ds-1 amd64 +libudev1 257.13-1~deb13u1 amd64 +libunistring5 1.3-2 amd64 +libuuid1 2.41.5-0+deb13u1 amd64 +libwrap0 7.6.q-36 amd64 +libwtmpdb0 0.73.0-3+deb13u1 amd64 +libxtables12 1.8.11-2 amd64 +libzstd1 1.5.7+dfsg-1 amd64 +linux-base 4.12.1 all +linux-image-6.12.111+deb13-amd64 6.12.111-1 amd64 +linux-image-amd64 6.12.111-1 amd64 +login.defs 1:4.17.4-2 all +logsave 1.47.2-3+b12 amd64 +mawk 1.3.4.20250131-1 amd64 +mount 2.41.5-0+deb13u1 amd64 +ncurses-base 6.5+20250216-2 all +ncurses-bin 6.5+20250216-2 amd64 +netbase 6.5 all +nftables 1.1.3-1 amd64 +openssh-client 1:10.0p1-7+deb13u4 amd64 +openssh-server 1:10.0p1-7+deb13u4 amd64 +openssh-sftp-server 1:10.0p1-7+deb13u4 amd64 +openssl-provider-legacy 3.5.7-1~deb13u3 amd64 +openssl 3.5.7-1~deb13u3 amd64 +passwd 1:4.17.4-2 amd64 +perl-base 5.40.1-6+deb13u1 amd64 +procps 2:4.0.4-9 amd64 +rauc-service 1.13-3+deb13u1 all +rauc 1.13-3+deb13u1 amd64 +readline-common 8.2-6 all +runit-helper 2.16.4 all +sed 4.9-2+deb13u1 amd64 +sensible-utils 0.0.25 all +sudo 1.9.16p2-3+deb13u2 amd64 +systemd-repart 257.13-1~deb13u1 amd64 +systemd-sysv 257.13-1~deb13u1 amd64 +systemd 257.13-1~deb13u1 amd64 +sysvinit-utils 3.14-4 amd64 +tar 1.35+dfsg-3.1 amd64 +tzdata 2026c-0+deb13u1 all +ucf 3.0052 all +udev 257.13-1~deb13u1 amd64 +util-linux 2.41.5-0+deb13u1 amd64 +zlib1g 1:1.3.dfsg+really1.3.1-1+b1 amd64 diff --git a/dev/os-lock/debian-snapshot b/dev/os-lock/debian-snapshot new file mode 100644 index 00000000..1860aa21 --- /dev/null +++ b/dev/os-lock/debian-snapshot @@ -0,0 +1 @@ +20261003T083111Z diff --git a/dev/os-lock/os-lock.sh b/dev/os-lock/os-lock.sh new file mode 100644 index 00000000..9750ac56 --- /dev/null +++ b/dev/os-lock/os-lock.sh @@ -0,0 +1,116 @@ +# Shared helpers for the OS package lock (BUILD.md # 1b # The OS package lock, +# #560). Sourced, not run, by dev/cloud/bootstrap.sh (the lean image that ships) +# and dev/cloud/test/bootstrap.sh (the boot-proof image), so both builds install +# the same versions from the same sources. +# +# Expects these set by the sourcing script: REPO_ROOT, CALLER (may be empty), GO. +# +# What the lock pins, and how: +# - Debian: dev/os-lock/debian-snapshot is a snapshot.debian.org timestamp. +# mkosi gets it as --snapshot, so the main archive and debian-security come +# from snapshot.debian.org at that moment (mkosi v26 does both). mkosi leaves +# out trixie-updates in snapshot mode, so it is added back here from the same +# snapshot: the image keeps the suites it had before the lock. +# - Docker: its repo is not in the snapshot. dev/os-lock/third-party.lock names +# exact versions; each becomes an apt pin (Pin-Priority 1001). +# - The result: dev/os-lock/cloud-packages.lock, checked after the build by +# os_lock_check. +# +# There is no fallback to a live mirror when snapshot.debian.org is slow or down: +# that would break the lock. apt retries (Acquire::Retries), and then the build +# fails. A failed release is re-run (docs/dev/contributing.md # Release model). + +OS_LOCK_DIR="${REPO_ROOT}/dev/os-lock" + +# os_lock_snapshot prints the locked timestamp, or fails on a malformed file. +os_lock_snapshot() { + local ts + ts="$(tr -d '[:space:]' < "${OS_LOCK_DIR}/debian-snapshot")" + if ! printf '%s' "$ts" | grep -Eq '^[0-9]{8}T[0-9]{6}Z$'; then + echo "dev/os-lock/debian-snapshot does not hold a snapshot.debian.org timestamp (got '$ts')" >&2 + return 1 + fi + printf '%s\n' "$ts" +} + +# stage_os_lock_apt PKGMNGR_DIR: write the build's apt sources, pins and retry +# setting into an mkosi.pkgmngr tree (mkosi uses it as apt's /etc/apt). Replaces +# the tree, so nothing stale survives. +stage_os_lock_apt() { + local pkgmngr="$1" ts + ts="$(os_lock_snapshot)" || return 1 + rm -rf "$pkgmngr" + mkdir -p "$pkgmngr/etc/apt/keyrings" "$pkgmngr/etc/apt/sources.list.d" \ + "$pkgmngr/etc/apt/preferences.d" "$pkgmngr/etc/apt/apt.conf.d" + + # Docker's repo, trixie pocket. Build-host network only; the VM never + # apt-installs (BUILD.md # Docker package source). + curl -fsSL --retry 5 https://download.docker.com/linux/debian/gpg \ + -o "$pkgmngr/etc/apt/keyrings/docker.asc" + chmod a+r "$pkgmngr/etc/apt/keyrings/docker.asc" + cat > "$pkgmngr/etc/apt/sources.list.d/docker.list" <<'EOF' +deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable +EOF + + # trixie-updates from the same snapshot (mkosi drops it in snapshot mode). + cat > "$pkgmngr/etc/apt/sources.list.d/debian-updates.list" < "$pref" + while IFS= read -r line || [ -n "$line" ]; do + line="${line%%#*}" + line="$(printf '%s' "$line" | tr -d '[:space:]')" + [ -n "$line" ] || continue + name="${line%%=*}" + ver="${line#*=}" + if [ -z "$name" ] || [ -z "$ver" ] || [ "$name" = "$line" ]; then + echo "dev/os-lock/third-party.lock: '$line' is not package=version" >&2 + return 1 + fi + printf 'Package: %s\nPin: version %s\nPin-Priority: 1001\n\n' "$name" "$ver" >> "$pref" + done < "${OS_LOCK_DIR}/third-party.lock" + + # snapshot.debian.org can be slow; retry a fetch before failing the build. + cat > "$pkgmngr/etc/apt/apt.conf.d/80-moose-os-lock" <<'EOF' +Acquire::Retries "5"; +EOF + echo "OS lock: Debian snapshot ${ts}, $(grep -c '^Package:' "$pref") third-party pins" +} + +# os_lock_tool OUT: build dev/os-lock/oslock to OUT, as the caller so root never +# owns the caller's Go build cache. +os_lock_tool() { + local out="$1" + if [ -n "${CALLER:-}" ]; then + sudo -u "$CALLER" "$GO" build -C "$REPO_ROOT" -o "$out" ./dev/os-lock/oslock + else + "$GO" build -C "$REPO_ROOT" -o "$out" ./dev/os-lock/oslock + fi +} + +# os_lock_check MANIFEST OUT_LOCK: write the resolved list for MANIFEST to +# OUT_LOCK, then compare it with the committed dev/os-lock/cloud-packages.lock. +# +# With MOOSE_OS_LOCK_RECORD=1 the committed file is replaced instead of checked. +# Only the bump workflow sets it (.github/workflows/os-lock-bump.yml), right +# after it moved the snapshot or a pin; every other build checks. +os_lock_check() { + local manifest="$1" out="$2" tool + tool="$(dirname "$out")/oslock" + os_lock_tool "$tool" || return 1 + "$tool" normalize "$manifest" > "$out" || return 1 + if [ "${MOOSE_OS_LOCK_RECORD:-}" = "1" ]; then + cp "$out" "${OS_LOCK_DIR}/cloud-packages.lock" + [ -n "${CALLER:-}" ] && chown "$CALLER":"$(id -gn "$CALLER")" "${OS_LOCK_DIR}/cloud-packages.lock" 2>/dev/null + echo "OS lock: recorded $(grep -vc '^#' "$out") packages to dev/os-lock/cloud-packages.lock (MOOSE_OS_LOCK_RECORD=1)" + return 0 + fi + if [ ! -f "${OS_LOCK_DIR}/cloud-packages.lock" ]; then + echo "dev/os-lock/cloud-packages.lock is missing. The resolved list is at $out." >&2 + return 1 + fi + "$tool" check "$manifest" "${OS_LOCK_DIR}/cloud-packages.lock" +} diff --git a/dev/os-lock/oslock/main.go b/dev/os-lock/oslock/main.go new file mode 100644 index 00000000..3ebea3ef --- /dev/null +++ b/dev/os-lock/oslock/main.go @@ -0,0 +1,551 @@ +// Command oslock reads and writes the OS package lock in dev/os-lock/ +// (BUILD.md # 1b # The OS package lock, #560). +// +// The lock has three files. debian-snapshot holds the snapshot.debian.org +// timestamp the image installs from. third-party.lock pins each package from a +// repo that is not in that snapshot (Docker's) to an exact version. And +// cloud-packages.lock is the package list the hosted image resolved to, one +// "name version architecture" line per package, made from mkosi's JSON +// manifest. The build compares a fresh manifest with that file and fails when +// they differ; the scheduled bump workflow (.github/workflows/os-lock-bump.yml) +// writes a new one and names the changes in its PR. +// +// Subcommands: +// +// oslock normalize MANIFEST.json print the lock file for a manifest +// oslock check MANIFEST.json LOCK exit 1, with the changes, if they differ +// oslock title [-security P] OLD NEW one-line PR title naming the changes +// oslock diff [-security P] OLD NEW Markdown table of the changes +// oslock newest-pins LOCK DOCKER_PACKAGES third-party.lock moved to the newest +// version within each pin's major +// +// P is a Debian Packages index of trixie-security at the new snapshot. A changed +// package whose new version is in it is marked as a security update. +package main + +import ( + "bufio" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "sort" + "strings" +) + +func main() { + if err := run(os.Args[1:], os.Stdout); err != nil { + var mismatch errMismatch + if errors.As(err, &mismatch) { + fmt.Fprint(os.Stderr, string(mismatch)) + os.Exit(1) + } + fmt.Fprintln(os.Stderr, "oslock:", err) + os.Exit(2) + } +} + +// errMismatch is the one error a caller acts on: the build resolved to a +// different package list than the committed lock. Its text is the report. +type errMismatch string + +func (e errMismatch) Error() string { return string(e) } + +func run(args []string, out io.Writer) error { + if len(args) == 0 { + return errors.New("usage: oslock normalize|check|title|diff|newest-pins ...") + } + cmd, rest := args[0], args[1:] + switch cmd { + case "normalize": + if len(rest) != 1 { + return errors.New("usage: oslock normalize MANIFEST.json") + } + pkgs, err := readManifest(rest[0]) + if err != nil { + return err + } + _, err = io.WriteString(out, formatLock(pkgs)) + return err + case "check": + if len(rest) != 2 { + return errors.New("usage: oslock check MANIFEST.json LOCK") + } + got, err := readManifest(rest[0]) + if err != nil { + return err + } + want, err := readLockFile(rest[1]) + if err != nil { + return err + } + changes := compare(want, got) + if len(changes) == 0 { + fmt.Fprintf(out, "package lock check passed: %d packages match %s\n", len(got), rest[1]) + return nil + } + var b strings.Builder + fmt.Fprintf(&b, "PACKAGE LOCK CHECK FAILED: the build resolved to a different package list than %s (%d changes).\n", rest[1], len(changes)) + for _, c := range changes { + fmt.Fprintf(&b, " %s\n", c.String()) + } + b.WriteString("\nThe image installs from the snapshot in dev/os-lock/debian-snapshot and the pins in\n" + + "dev/os-lock/third-party.lock, so this list must not move by itself. If you changed the\n" + + "package list, the snapshot or a pin on purpose, update the lock with the resolved list\n" + + "this build wrote to .dev/cloud/cloud-packages.lock (CI uploads it as an artifact).\n") + return errMismatch(b.String()) + case "title", "diff": + fs := flag.NewFlagSet(cmd, flag.ContinueOnError) + security := fs.String("security", "", "Debian Packages index of trixie-security at the new snapshot") + if err := fs.Parse(rest); err != nil { + return err + } + if fs.NArg() != 2 { + return fmt.Errorf("usage: oslock %s [-security PACKAGES] OLD NEW", cmd) + } + old, err := readLockFile(fs.Arg(0)) + if err != nil { + return err + } + cur, err := readLockFile(fs.Arg(1)) + if err != nil { + return err + } + sec := map[string]map[string]bool{} + if *security != "" { + f, err := os.Open(*security) + if err != nil { + return err + } + defer f.Close() + if sec, err = parsePackagesIndex(f); err != nil { + return err + } + } + changes := compare(old, cur) + markSecurity(changes, sec) + if cmd == "title" { + _, err = fmt.Fprintln(out, title(changes)) + } else { + _, err = io.WriteString(out, markdown(changes)) + } + return err + case "newest-pins": + if len(rest) != 2 { + return errors.New("usage: oslock newest-pins LOCK DOCKER_PACKAGES") + } + lock, err := os.ReadFile(rest[0]) + if err != nil { + return err + } + f, err := os.Open(rest[1]) + if err != nil { + return err + } + defer f.Close() + index, err := parsePackagesIndex(f) + if err != nil { + return err + } + next, err := newestPins(string(lock), index) + if err != nil { + return err + } + _, err = io.WriteString(out, next) + return err + } + return fmt.Errorf("unknown subcommand %q", cmd) +} + +// pkg is one installed package. The key is name plus architecture, because a +// multi-arch package can be installed once per architecture. +type pkg struct { + Name, Version, Arch string +} + +func (p pkg) key() string { return p.Name + ":" + p.Arch } + +func readManifest(path string) ([]pkg, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var m struct { + Packages []struct { + Name string `json:"name"` + Version string `json:"version"` + Architecture string `json:"architecture"` + } `json:"packages"` + } + if err := json.Unmarshal(data, &m); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + if len(m.Packages) == 0 { + return nil, fmt.Errorf("%s: the manifest lists no packages", path) + } + var pkgs []pkg + for _, p := range m.Packages { + if p.Name == "" || p.Version == "" || p.Architecture == "" { + return nil, fmt.Errorf("%s: a package entry is missing its name, version or architecture", path) + } + pkgs = append(pkgs, pkg{p.Name, p.Version, p.Architecture}) + } + return pkgs, nil +} + +const lockHeader = `# The package list the hosted cloud image resolved to (dev/cloud/mkosi.conf), +# installed from the snapshot in debian-snapshot and the pins in third-party.lock. +# Generated from mkosi's JSON manifest by dev/os-lock/oslock; do not edit by hand. +# The build fails when it resolves to a different list (BUILD.md # 1b # The OS +# package lock). One package per line: name version architecture. +` + +func formatLock(pkgs []pkg) string { + sorted := append([]pkg(nil), pkgs...) + sort.Slice(sorted, func(i, j int) bool { return sorted[i].key() < sorted[j].key() }) + var b strings.Builder + b.WriteString(lockHeader) + for _, p := range sorted { + fmt.Fprintf(&b, "%s %s %s\n", p.Name, p.Version, p.Arch) + } + return b.String() +} + +func readLockFile(path string) ([]pkg, error) { + f, err := os.Open(path) + if err != nil { + return nil, err + } + defer f.Close() + pkgs, err := parseLock(f) + if err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + return pkgs, nil +} + +// parseLock reads a lock file. Comments and blank lines are ignored, so a +// change to the header is never a package change. +func parseLock(r io.Reader) ([]pkg, error) { + var pkgs []pkg + sc := bufio.NewScanner(r) + n := 0 + for sc.Scan() { + n++ + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + f := strings.Fields(line) + if len(f) != 3 { + return nil, fmt.Errorf("line %d: want \"name version architecture\", got %q", n, line) + } + pkgs = append(pkgs, pkg{f[0], f[1], f[2]}) + } + return pkgs, sc.Err() +} + +type change struct { + Name, Arch string + From, To string // empty From: added; empty To: removed + Security bool +} + +func (c change) String() string { + switch { + case c.From == "": + return fmt.Sprintf("added %s %s", c.Name, c.To) + case c.To == "": + return fmt.Sprintf("removed %s %s", c.Name, c.From) + } + return fmt.Sprintf("changed %s %s to %s", c.Name, c.From, c.To) +} + +// compare lists what moved from old to cur, sorted by package name. +func compare(old, cur []pkg) []change { + before := map[string]pkg{} + for _, p := range old { + before[p.key()] = p + } + after := map[string]pkg{} + for _, p := range cur { + after[p.key()] = p + } + var changes []change + for k, p := range after { + o, ok := before[k] + switch { + case !ok: + changes = append(changes, change{Name: p.Name, Arch: p.Arch, To: p.Version}) + case o.Version != p.Version: + changes = append(changes, change{Name: p.Name, Arch: p.Arch, From: o.Version, To: p.Version}) + } + } + for k, o := range before { + if _, ok := after[k]; !ok { + changes = append(changes, change{Name: o.Name, Arch: o.Arch, From: o.Version}) + } + } + sort.Slice(changes, func(i, j int) bool { + if changes[i].Name != changes[j].Name { + return changes[i].Name < changes[j].Name + } + return changes[i].Arch < changes[j].Arch + }) + return changes +} + +func markSecurity(changes []change, sec map[string]map[string]bool) { + for i := range changes { + if changes[i].To != "" && sec[changes[i].Name][changes[i].To] { + changes[i].Security = true + } + } +} + +const maxTitle = 120 + +// title names the changes, security updates first, and stops at about +// maxTitle characters with a count of the rest. +func title(changes []change) string { + if len(changes) == 0 { + return "OS lock: no package changes" + } + ordered := append([]change(nil), changes...) + sort.SliceStable(ordered, func(i, j int) bool { return ordered[i].Security && !ordered[j].Security }) + prefix := "OS lock: " + if ordered[0].Security { + prefix = "OS lock (security): " + } + t := prefix + for i, c := range ordered { + var part string + switch { + case c.From == "": + part = fmt.Sprintf("add %s %s", c.Name, c.To) + case c.To == "": + part = fmt.Sprintf("remove %s", c.Name) + default: + part = fmt.Sprintf("%s %s to %s", c.Name, c.From, c.To) + } + sep := "" + if i > 0 { + sep = ", " + } + more := fmt.Sprintf(" and %d more", len(ordered)-i) + if i > 0 && len(t)+len(sep)+len(part) > maxTitle { + return t + more + } + t += sep + part + } + return t +} + +func markdown(changes []change) string { + var b strings.Builder + if len(changes) == 0 { + b.WriteString("No package changed.\n") + return b.String() + } + nsec, nadd := 0, 0 + for _, c := range changes { + if c.Security { + nsec++ + } + if c.From == "" || c.To == "" { + nadd++ + } + } + if nsec > 0 { + fmt.Fprintf(&b, "**Security:** %d of the %d changed packages come from `trixie-security`. "+ + "Release rule: cut an OS patch release with this change within 7 days "+ + "(`docs/dev/contributing.md` # Release model).\n\n", nsec, len(changes)) + } else { + b.WriteString("No change comes from `trixie-security`, so this ships with the next normal OS release.\n\n") + } + b.WriteString("| Package | From | To | Source |\n|---|---|---|---|\n") + for _, c := range changes { + from, to, src := c.From, c.To, "" + if from == "" { + from = "(new)" + } + if to == "" { + to = "(removed)" + } + if c.Security { + src = "trixie-security" + } + fmt.Fprintf(&b, "| `%s` | `%s` | `%s` | %s |\n", c.Name, from, to, src) + } + if nadd > 0 { + b.WriteString("\nA package was added or removed. The lean check fails until " + + "`dev/cloud/expected-packages.txt` lists the new set, so a person must review it.\n") + } + return b.String() +} + +// parsePackagesIndex reads a Debian Packages index into name -> versions. +func parsePackagesIndex(r io.Reader) (map[string]map[string]bool, error) { + idx := map[string]map[string]bool{} + sc := bufio.NewScanner(r) + sc.Buffer(make([]byte, 1024*1024), 16*1024*1024) + name := "" + for sc.Scan() { + line := sc.Text() + switch { + case line == "": + name = "" + case strings.HasPrefix(line, "Package:"): + name = strings.TrimSpace(strings.TrimPrefix(line, "Package:")) + case strings.HasPrefix(line, "Version:") && name != "": + v := strings.TrimSpace(strings.TrimPrefix(line, "Version:")) + if idx[name] == nil { + idx[name] = map[string]bool{} + } + idx[name][v] = true + } + } + return idx, sc.Err() +} + +// newestPins rewrites a third-party.lock ("package=version" lines, comments +// kept as they are) so each pin names the newest version in index with the +// same epoch and the same major version. A new major is a manual edit. +func newestPins(lock string, index map[string]map[string]bool) (string, error) { + var b strings.Builder + lines := strings.SplitAfter(lock, "\n") + for _, raw := range lines { + line := strings.TrimSpace(raw) + if line == "" || strings.HasPrefix(line, "#") { + b.WriteString(raw) + continue + } + name, ver, ok := strings.Cut(line, "=") + if !ok || name == "" || ver == "" { + return "", fmt.Errorf("third-party pin %q is not package=version", line) + } + if len(index[name]) == 0 { + return "", fmt.Errorf("package %s is not in the third-party index", name) + } + best := ver + for cand := range index[name] { + if sameMajor(cand, ver) && compareVersions(cand, best) > 0 { + best = cand + } + } + b.WriteString(name + "=" + best + "\n") + } + return b.String(), nil +} + +// sameMajor reports whether two Debian versions share an epoch and the first +// number of the upstream version ("5:29.8.2-1~debian.13~trixie" is 5 and 29). +func sameMajor(a, b string) bool { + ea, ua, _ := splitVersion(a) + eb, ub, _ := splitVersion(b) + return ea == eb && leadingNumber(ua) == leadingNumber(ub) && leadingNumber(ua) != "" +} + +func leadingNumber(s string) string { + i := 0 + for i < len(s) && s[i] >= '0' && s[i] <= '9' { + i++ + } + return s[:i] +} + +// splitVersion splits a Debian version into epoch, upstream and revision. +func splitVersion(v string) (epoch, upstream, revision string) { + epoch = "0" + if e, rest, ok := strings.Cut(v, ":"); ok { + epoch, v = e, rest + } + if i := strings.LastIndex(v, "-"); i >= 0 { + return epoch, v[:i], v[i+1:] + } + return epoch, v, "" +} + +// compareVersions compares two Debian versions the way dpkg does: epoch as a +// number, then upstream and revision by dpkg's string-and-number rule, where +// "~" sorts before everything, even the end of the string. +func compareVersions(a, b string) int { + ea, ua, ra := splitVersion(a) + eb, ub, rb := splitVersion(b) + if c := compareNumber(ea, eb); c != 0 { + return c + } + if c := comparePart(ua, ub); c != 0 { + return c + } + return comparePart(ra, rb) +} + +func compareNumber(a, b string) int { + a = strings.TrimLeft(a, "0") + b = strings.TrimLeft(b, "0") + if len(a) != len(b) { + if len(a) < len(b) { + return -1 + } + return 1 + } + return strings.Compare(a, b) +} + +func isDigit(c byte) bool { return c >= '0' && c <= '9' } + +func order(c byte) int { + switch { + case c == '~': + return -1 + case isDigit(c): + return 0 + case (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'): + return int(c) + } + return int(c) + 256 +} + +func comparePart(a, b string) int { + for a != "" || b != "" { + // The non-digit run. + for (a != "" && !isDigit(a[0])) || (b != "" && !isDigit(b[0])) { + ac, bc := 0, 0 + if a != "" { + ac = order(a[0]) + } + if b != "" { + bc = order(b[0]) + } + if ac != bc { + if ac < bc { + return -1 + } + return 1 + } + if a != "" { + a = a[1:] + } + if b != "" { + b = b[1:] + } + } + // The digit run. + i := 0 + for i < len(a) && isDigit(a[i]) { + i++ + } + j := 0 + for j < len(b) && isDigit(b[j]) { + j++ + } + if c := compareNumber(a[:i], b[:j]); c != 0 { + return c + } + a, b = a[i:], b[j:] + } + return 0 +} diff --git a/dev/os-lock/oslock/main_test.go b/dev/os-lock/oslock/main_test.go new file mode 100644 index 00000000..3786284d --- /dev/null +++ b/dev/os-lock/oslock/main_test.go @@ -0,0 +1,319 @@ +package main + +import ( + "bufio" + "bytes" + "errors" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +func TestCompareVersions(t *testing.T) { + cases := []struct { + a, b string + want int + }{ + {"1.0", "1.0", 0}, + {"1.0", "1.1", -1}, + {"1.10", "1.9", 1}, + {"1.0~rc1", "1.0", -1}, + {"1.0~rc1", "1.0~rc2", -1}, + {"1:1.0", "2.0", 1}, + {"5:29.8.2-1~debian.13~trixie", "5:29.8.10-1~debian.13~trixie", -1}, + {"5:29.8.2-1~debian.13~trixie", "5:28.5.2-1~debian.13~trixie", 1}, + {"3.5.7-1~deb13u2", "3.5.7-1~deb13u3", -1}, + {"3.5.7-1", "3.5.7-1~deb13u3", 1}, + {"1.0a", "1.0", 1}, + {"1.0+b1", "1.0", 1}, + {"2.41.5-0+deb13u1", "2.41-5", 1}, + {"1.0-1", "1.0", 1}, + } + for _, c := range cases { + if got := compareVersions(c.a, c.b); got != c.want { + t.Errorf("compareVersions(%q, %q) = %d, want %d", c.a, c.b, got, c.want) + } + if got := compareVersions(c.b, c.a); got != -c.want { + t.Errorf("compareVersions(%q, %q) = %d, want %d", c.b, c.a, got, -c.want) + } + } +} + +func TestSameMajor(t *testing.T) { + if !sameMajor("5:29.8.2-1~debian.13~trixie", "5:29.0.0-1~debian.13~trixie") { + t.Error("29.8.2 and 29.0.0 should share a major") + } + if sameMajor("5:29.8.2-1~debian.13~trixie", "5:30.0.0-1~debian.13~trixie") { + t.Error("29 and 30 are different majors") + } + if sameMajor("2.3.6-1", "1:2.3.6-1") { + t.Error("a different epoch is a different major") + } +} + +func writeFile(t *testing.T, dir, name, body string) string { + t.Helper() + p := filepath.Join(dir, name) + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// A synthetic mkosi manifest in the published shape (mkosi/manifest.py). +const manifest = `{ + "manifest_version": 1, + "config": {"name": "moose-cloud", "distribution": "debian", "architecture": "x86-64"}, + "packages": [ + {"type": "deb", "name": "zlib1g", "version": "1:1.3", "architecture": "amd64"}, + {"type": "deb", "name": "openssl", "version": "3.5.7-1~deb13u2", "architecture": "amd64"}, + {"type": "deb", "name": "tzdata", "version": "2026b-0+deb13u1", "architecture": "all"} + ], + "extension": {} +}` + +func TestNormalizeIsSortedAndRoundTrips(t *testing.T) { + dir := t.TempDir() + m := writeFile(t, dir, "image.manifest", manifest) + var out bytes.Buffer + if err := run([]string{"normalize", m}, &out); err != nil { + t.Fatal(err) + } + body := out.String() + if !strings.HasPrefix(body, "# ") { + t.Errorf("the lock should start with its header comment, got %q", body) + } + want := "openssl 3.5.7-1~deb13u2 amd64\ntzdata 2026b-0+deb13u1 all\nzlib1g 1:1.3 amd64\n" + if !strings.HasSuffix(body, want) { + t.Errorf("normalize body:\n%s\nwant it to end with:\n%s", body, want) + } + lock := writeFile(t, dir, "cloud-packages.lock", body) + out.Reset() + if err := run([]string{"check", m, lock}, &out); err != nil { + t.Fatalf("check of a manifest against its own lock failed: %v", err) + } +} + +func TestCheckReportsEveryKindOfChange(t *testing.T) { + dir := t.TempDir() + m := writeFile(t, dir, "image.manifest", manifest) + lock := writeFile(t, dir, "cloud-packages.lock", "# header\n"+ + "openssl 3.5.7-1~deb13u1 amd64\n"+ + "tzdata 2026b-0+deb13u1 all\n"+ + "bash 5.2 amd64\n") + err := run([]string{"check", m, lock}, &bytes.Buffer{}) + var mm errMismatch + if !errors.As(err, &mm) { + t.Fatalf("want a mismatch, got %v", err) + } + for _, want := range []string{ + "changed openssl 3.5.7-1~deb13u1 to 3.5.7-1~deb13u2", + "added zlib1g 1:1.3", + "removed bash 5.2", + "3 changes", + } { + if !strings.Contains(string(mm), want) { + t.Errorf("report does not say %q:\n%s", want, mm) + } + } +} + +func TestCheckRefusesAnEmptyManifest(t *testing.T) { + dir := t.TempDir() + m := writeFile(t, dir, "image.manifest", `{"packages": []}`) + lock := writeFile(t, dir, "cloud-packages.lock", "") + err := run([]string{"check", m, lock}, &bytes.Buffer{}) + var mm errMismatch + if err == nil || errors.As(err, &mm) { + t.Fatalf("an empty manifest must be an error, not a pass or a package diff: %v", err) + } +} + +const securityIndex = `Package: openssl +Version: 3.5.7-1~deb13u3 +Architecture: amd64 + +Package: libssl3t64 +Version: 3.5.7-1~deb13u3 +Architecture: amd64 +` + +func TestTitleAndDiffNameTheChanges(t *testing.T) { + dir := t.TempDir() + old := writeFile(t, dir, "old.lock", "openssl 3.5.7-1~deb13u2 amd64\ntzdata 2026b-0+deb13u1 all\ncurl 8.0 amd64\n") + cur := writeFile(t, dir, "new.lock", "openssl 3.5.7-1~deb13u3 amd64\ntzdata 2026c-0+deb13u1 all\ncurl 8.0 amd64\n") + sec := writeFile(t, dir, "Packages", securityIndex) + + var out bytes.Buffer + if err := run([]string{"title", "-security", sec, old, cur}, &out); err != nil { + t.Fatal(err) + } + want := "OS lock (security): openssl 3.5.7-1~deb13u2 to 3.5.7-1~deb13u3, tzdata 2026b-0+deb13u1 to 2026c-0+deb13u1\n" + if out.String() != want { + t.Errorf("title = %q, want %q", out.String(), want) + } + + out.Reset() + if err := run([]string{"diff", "-security", sec, old, cur}, &out); err != nil { + t.Fatal(err) + } + body := out.String() + for _, w := range []string{ + "1 of the 2 changed packages come from `trixie-security`", + "| `openssl` | `3.5.7-1~deb13u2` | `3.5.7-1~deb13u3` | trixie-security |", + "| `tzdata` | `2026b-0+deb13u1` | `2026c-0+deb13u1` | |", + } { + if !strings.Contains(body, w) { + t.Errorf("diff does not contain %q:\n%s", w, body) + } + } + if strings.Contains(body, "curl") { + t.Errorf("an unchanged package is in the diff:\n%s", body) + } +} + +func TestTitleWithoutSecurityAndWithManyChanges(t *testing.T) { + var oldB, newB strings.Builder + for _, n := range []string{"aaaa", "bbbb", "cccc", "dddd", "eeee", "ffff", "gggg"} { + oldB.WriteString(n + " 1.0-1 amd64\n") + newB.WriteString(n + " 1.0-2 amd64\n") + } + dir := t.TempDir() + old := writeFile(t, dir, "old.lock", oldB.String()) + cur := writeFile(t, dir, "new.lock", newB.String()) + var out bytes.Buffer + if err := run([]string{"title", old, cur}, &out); err != nil { + t.Fatal(err) + } + got := strings.TrimSpace(out.String()) + if !strings.HasPrefix(got, "OS lock: aaaa 1.0-1 to 1.0-2, ") || !strings.Contains(got, " more") { + t.Errorf("title = %q", got) + } + if len(got) > maxTitle+len(" and 99 more") { + t.Errorf("title is %d characters, too long: %q", len(got), got) + } +} + +func TestDiffFlagsAddedPackagesForReview(t *testing.T) { + dir := t.TempDir() + old := writeFile(t, dir, "old.lock", "curl 8.0 amd64\n") + cur := writeFile(t, dir, "new.lock", "curl 8.0 amd64\nlibnew1 1.0 amd64\n") + var out bytes.Buffer + if err := run([]string{"diff", old, cur}, &out); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "| `libnew1` | `(new)` | `1.0` |") || !strings.Contains(out.String(), "expected-packages.txt") { + t.Errorf("diff:\n%s", out.String()) + } + if !strings.Contains(out.String(), "next normal OS release") { + t.Errorf("a bump with no security change should say it waits for the next release:\n%s", out.String()) + } +} + +const dockerIndex = `Package: docker-ce +Version: 5:29.8.2-1~debian.13~trixie + +Package: docker-ce +Version: 5:29.10.0-1~debian.13~trixie + +Package: docker-ce +Version: 5:30.0.0-1~debian.13~trixie + +Package: containerd.io +Version: 2.3.6-1~debian.13~trixie + +Package: containerd.io +Version: 1.7.27-1 +` + +func TestNewestPinsStaysInTheMajor(t *testing.T) { + dir := t.TempDir() + lock := writeFile(t, dir, "third-party.lock", "# comment kept\ncontainerd.io=2.3.6-1~debian.13~trixie\ndocker-ce=5:29.8.2-1~debian.13~trixie\n") + idx := writeFile(t, dir, "Packages", dockerIndex) + var out bytes.Buffer + if err := run([]string{"newest-pins", lock, idx}, &out); err != nil { + t.Fatal(err) + } + want := "# comment kept\ncontainerd.io=2.3.6-1~debian.13~trixie\ndocker-ce=5:29.10.0-1~debian.13~trixie\n" + if out.String() != want { + t.Errorf("newest-pins:\n%s\nwant:\n%s", out.String(), want) + } +} + +func TestNewestPinsRefusesAPackageMissingFromTheIndex(t *testing.T) { + dir := t.TempDir() + lock := writeFile(t, dir, "third-party.lock", "docker-buildx-plugin=0.1-1\n") + idx := writeFile(t, dir, "Packages", dockerIndex) + if err := run([]string{"newest-pins", lock, idx}, &bytes.Buffer{}); err == nil { + t.Fatal("a pin for a package the repo does not have must be an error") + } +} + +// The committed lock must hang together: a valid timestamp, one pin per Docker +// package the image installs, each pinned version in the resolved list, and +// the resolved list in the exact form normalize writes. A bad hand edit fails +// here, in make check, instead of in a 10 minute image build. +func TestCommittedLockIsConsistent(t *testing.T) { + snap, err := os.ReadFile("../debian-snapshot") + if err != nil { + t.Fatal(err) + } + if !regexp.MustCompile(`^\d{8}T\d{6}Z$`).MatchString(strings.TrimSpace(string(snap))) { + t.Errorf("debian-snapshot %q is not a snapshot.debian.org timestamp like 20261001T082322Z", snap) + } + + pins := map[string]string{} + f, err := os.Open("../third-party.lock") + if err != nil { + t.Fatal(err) + } + defer f.Close() + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + name, ver, ok := strings.Cut(line, "=") + if !ok || name == "" || ver == "" { + t.Fatalf("third-party.lock line %q is not package=version", line) + } + pins[name] = ver + } + + conf, err := os.ReadFile("../../cloud/mkosi.conf") + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"docker-ce", "docker-ce-cli", "containerd.io", "docker-compose-plugin"} { + if pins[name] == "" { + t.Errorf("third-party.lock has no pin for %s, which dev/cloud/mkosi.conf installs from Docker's repo", name) + } + if !regexp.MustCompile(`(?m)^\s+` + regexp.QuoteMeta(name) + `\s*$`).Match(conf) { + t.Errorf("third-party.lock pins %s, but dev/cloud/mkosi.conf does not install it", name) + } + } + + body, err := os.ReadFile("../cloud-packages.lock") + if err != nil { + t.Fatal(err) + } + pkgs, err := parseLock(bytes.NewReader(body)) + if err != nil { + t.Fatal(err) + } + if string(body) != formatLock(pkgs) { + t.Error("cloud-packages.lock is not in the form oslock normalize writes (sorted, with the header); regenerate it, do not edit it by hand") + } + resolved := map[string]string{} + for _, p := range pkgs { + resolved[p.Name] = p.Version + } + for name, ver := range pins { + if resolved[name] != ver { + t.Errorf("third-party.lock pins %s=%s, but cloud-packages.lock has %q", name, ver, resolved[name]) + } + } +} diff --git a/dev/os-lock/third-party.lock b/dev/os-lock/third-party.lock new file mode 100644 index 00000000..3db04b41 --- /dev/null +++ b/dev/os-lock/third-party.lock @@ -0,0 +1,17 @@ +# Exact versions of the packages the OS image installs from apt repos that are +# not in the Debian snapshot (BUILD.md # 1b # The OS package lock). Today that is +# Docker's repo, download.docker.com, trixie pocket (dev/os-lock/os-lock.sh). +# +# One "package=version" line per package, in apt's own syntax. The build turns +# each line into an apt pin (preferences.d, Pin-Priority 1001), so apt installs +# exactly this version. A version that is gone from the repo makes apt fall back +# to the newest one, and the package lock check (cloud-packages.lock) then fails +# the build, so a missing pin never ships quietly. +# +# The scheduled bump (.github/workflows/os-lock-bump.yml) moves each pin to the +# newest version within its current major. A move to a new major is a manual +# edit here, in its own PR. Keep the four Docker packages on one release train. +containerd.io=2.3.6-1~debian.13~trixie +docker-ce=5:29.8.2-1~debian.13~trixie +docker-ce-cli=5:29.8.2-1~debian.13~trixie +docker-compose-plugin=5.6.0-1~debian.13~trixie diff --git a/dev/release/attach-image.sh b/dev/release/attach-image.sh new file mode 100755 index 00000000..3620787d --- /dev/null +++ b/dev/release/attach-image.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Attach an OS release's files to its GitHub Release as ONE set, and never +# overwrite a published asset. +# +# attach-image.sh [...] +# +# ci-cloud-image.yml passes four files: the disk image, its checksum, the RAUC +# bundle and its checksum (#562). A rebuild is not byte-identical, so the files +# of one build only belong with each other: a checksum only matches its own +# file, and the bundle carries the same slot A as the image of its build. +# - all on the Release: an earlier attempt finished; leave them (exit 0). +# - none: upload all of them in one call (exit 0). +# - only some: an earlier attempt stopped halfway, or the Release was cut +# before the set grew (a Release with the image but no bundle). Uploading +# the rest from this build would mix two builds, so refuse (exit 1) and tell +# a person which assets to delete before re-running. Nothing published is +# deleted here. +# No --clobber: if an asset appears between the check and the upload, the +# upload fails instead of overwriting it. +# +# Needs `gh` with GH_TOKEN; ci-cloud-image.yml calls it after making sure the +# Release exists. +set -euo pipefail + +usage="usage: attach-image.sh [...]" +tag="${1:?$usage}" +shift +[ "$#" -ge 2 ] || { echo "$usage" >&2; exit 2; } + +existing="$(gh release view "$tag" --json assets --jq '.assets[].name')" +has() { grep -qxF "$1" <<<"$existing"; } + +present=() +missing=() +for f in "$@"; do + name="$(basename -- "$f")" + if has "$name"; then present+=("$name"); else missing+=("$name"); fi +done + +if [ "${#missing[@]}" -eq 0 ]; then + echo "::notice::release ${tag} already has ${present[*]}; leaving them (delete them all first to replace them)" +elif [ "${#present[@]}" -eq 0 ]; then + gh release upload "$tag" "$@" + echo "attached ${missing[*]} to release ${tag}" +else + deletes="" + for name in "${present[@]}"; do deletes="${deletes}gh release delete-asset ${tag} ${name} -y; "; done + echo "::error::release ${tag} has ${present[*]} but not ${missing[*]}, so an earlier upload stopped halfway (or the Release was cut before this set grew). This build cannot add the rest: a rebuild has different bytes, so the files would not belong together. Delete the stray assets and re-run: ${deletes}" + exit 1 +fi diff --git a/dev/release/controlplane_test.go b/dev/release/controlplane_test.go new file mode 100644 index 00000000..65e31af6 --- /dev/null +++ b/dev/release/controlplane_test.go @@ -0,0 +1,212 @@ +package release + +import ( + "archive/tar" + "crypto/sha256" + "encoding/hex" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// fakeRegistry writes a stub curl that serves manifests from dir: a request for +// manifests/ answers with the file dir/, and HTTP 404 +// when there is none. dir/.digest, when present, is sent as the +// Docker-Content-Digest header instead of the body's real digest. +func fakeRegistry(t *testing.T, dir string) string { + t.Helper() + bin := t.TempDir() + stub := `#!/usr/bin/env bash +out=""; head=""; url="" +while [ "$#" -gt 0 ]; do + case "$1" in + -o) out="$2"; shift 2 ;; + -D) head="$2"; shift 2 ;; + -H|-w) shift 2 ;; + -*) shift ;; + *) url="$1"; shift ;; + esac +done +case "$url" in + *ghcr.io/token*) echo '{"token":"t"}'; exit 0 ;; + *ghcr.io/v2/*/manifests/*) + ref="${url##*/manifests/}"; f="` + "${FAKE_DIR}" + `/${ref//:/_}" + if [ ! -f "$f" ]; then : > "$out"; : > "$head"; printf 404; exit 0; fi + cp "$f" "$out" + if [ -f "$f.digest" ]; then d="$(cat "$f.digest")"; else d="sha256:$(sha256sum "$f" | cut -d' ' -f1)"; fi + printf 'HTTP/2 200\r\ndocker-content-digest: %s\r\n\r\n' "$d" > "$head" + printf 200; exit 0 ;; +esac +exit 1 +` + if err := os.WriteFile(filepath.Join(bin, "curl"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + return bin +} + +func digestOf(b []byte) string { + s := sha256.Sum256(b) + return "sha256:" + hex.EncodeToString(s[:]) +} + +func hex64(c byte) string { return strings.Repeat(string(c), 64) } + +func runResolve(t *testing.T, bin, dir, ref string) (string, int) { + t.Helper() + script, err := filepath.Abs("ghcr-resolve.sh") + if err != nil { + t.Fatal(err) + } + cmd := exec.Command("bash", script, "onmoose/brain", ref) + cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "FAKE_DIR="+dir) + out, err := cmd.Output() + rc := 0 + if ee, ok := err.(*exec.ExitError); ok { + rc = ee.ExitCode() + } else if err != nil { + t.Fatal(err) + } + return strings.TrimSpace(string(out)), rc +} + +// ghcr-resolve.sh pins a tag to the digest of the bytes the registry served, +// takes the amd64 entry of an index, and fails on anything it cannot pin. +func TestGHCRResolve(t *testing.T) { + dir := t.TempDir() + bin := fakeRegistry(t, dir) + write := func(name string, b []byte) { + if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil { + t.Fatal(err) + } + } + + cfg := "sha256:" + hex64('c') + single := []byte(`{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"digest":"` + cfg + `"},"layers":[]}`) + write("v1.0.0", single) + if out, rc := runResolve(t, bin, dir, "v1.0.0"); rc != 0 || out != digestOf(single)+" "+cfg { + t.Errorf("single manifest: rc=%d out=%q, want %q", rc, out, digestOf(single)+" "+cfg) + } + + // An index: the linux/amd64 entry, fetched by its digest. + cfg2 := "sha256:" + hex64('d') + plat := []byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"digest":"` + cfg2 + `"},"layers":[]}`) + pd := digestOf(plat) + write(strings.ReplaceAll(pd, ":", "_"), plat) + index := []byte(fmt.Sprintf(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[`+ + `{"digest":"sha256:%s","platform":{"os":"linux","architecture":"arm64"}},`+ + `{"digest":"%s","platform":{"os":"linux","architecture":"amd64"}}]}`, hex64('a'), pd)) + write("v2.0.0", index) + if out, rc := runResolve(t, bin, dir, "v2.0.0"); rc != 0 || out != pd+" "+cfg2 { + t.Errorf("index: rc=%d out=%q, want %q", rc, out, pd+" "+cfg2) + } + + // An index with no amd64 image. + write("v3.0.0", []byte(`{"manifests":[{"digest":"sha256:`+hex64('a')+`","platform":{"os":"linux","architecture":"arm64"}}]}`)) + if out, rc := runResolve(t, bin, dir, "v3.0.0"); rc == 0 { + t.Errorf("index without amd64: want a failure, got %q", out) + } + + // The registry's digest header disagrees with the bytes. + write("v4.0.0", single) + write("v4.0.0.digest", []byte("sha256:"+hex64('e'))) + if out, rc := runResolve(t, bin, dir, "v4.0.0"); rc == 0 { + t.Errorf("digest mismatch: want a failure, got %q", out) + } + + // Not published. + if out, rc := runResolve(t, bin, dir, "v9.9.9"); rc == 0 { + t.Errorf("missing tag: want a failure, got %q", out) + } + + // A manifest with no config digest. + write("v5.0.0", []byte(`{"schemaVersion":2,"config":{}}`)) + if out, rc := runResolve(t, bin, dir, "v5.0.0"); rc == 0 { + t.Errorf("no config: want a failure, got %q", out) + } +} + +// writeImageTar writes a docker-save style tarball whose manifest.json names +// config as its Config entry. +func writeImageTar(t *testing.T, path, config string) { + t.Helper() + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + tw := tar.NewWriter(f) + body := []byte(`[{"Config":"` + config + `","RepoTags":["moose-brain:dev"],"Layers":[]}]`) + if err := tw.WriteHeader(&tar.Header{Name: "manifest.json", Mode: 0o644, Size: int64(len(body))}); err != nil { + t.Fatal(err) + } + if _, err := tw.Write(body); err != nil { + t.Fatal(err) + } + if err := tw.Close(); err != nil { + t.Fatal(err) + } +} + +// bundle-record.sh reads each image ID from the tarball's own manifest, in the +// layout of Docker 25+ and of older Docker, and records where the pair came from. +func TestBundleRecord(t *testing.T) { + script, err := filepath.Abs("../control-plane/bundle-record.sh") + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + writeImageTar(t, filepath.Join(dir, "moose-brain.tar"), "blobs/sha256/"+hex64('b')) + writeImageTar(t, filepath.Join(dir, "moose-ui.tar"), hex64('f')+".json") + + out, err := exec.Command("bash", script, dir, "released", "0.15.0", + "ghcr.io/onmoose/brain@sha256:"+hex64('1'), "ghcr.io/onmoose/ui@sha256:"+hex64('2')).CombinedOutput() + if err != nil { + t.Fatalf("bundle-record: %v\n%s", err, out) + } + got, err := os.ReadFile(filepath.Join(dir, "control-plane.env")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{ + "MOOSE_BAKED_CP_SOURCE=released\n", + "MOOSE_BAKED_CP_VERSION=0.15.0\n", + "MOOSE_BAKED_BRAIN_REF=ghcr.io/onmoose/brain@sha256:" + hex64('1') + "\n", + "MOOSE_BAKED_UI_REF=ghcr.io/onmoose/ui@sha256:" + hex64('2') + "\n", + "MOOSE_BAKED_BRAIN_ID=sha256:" + hex64('b') + "\n", + "MOOSE_BAKED_UI_ID=sha256:" + hex64('f') + "\n", + } { + if !strings.Contains(string(got), want) { + t.Errorf("record lacks %q:\n%s", want, got) + } + } + + // local takes no refs, and records empty ones. + out, err = exec.Command("bash", script, dir, "local", "0.16.0").CombinedOutput() + if err != nil { + t.Fatalf("bundle-record local: %v\n%s", err, out) + } + got, _ = os.ReadFile(filepath.Join(dir, "control-plane.env")) + if !strings.Contains(string(got), "MOOSE_BAKED_CP_SOURCE=local\n") || !strings.Contains(string(got), "MOOSE_BAKED_BRAIN_REF=\n") { + t.Errorf("local record:\n%s", got) + } + + // Refused: a bad source, refs on local, a non-version, a tarball it cannot read. + for _, args := range [][]string{ + {dir, "other", "0.16.0"}, + {dir, "local", "0.16.0", "a", "b"}, + {dir, "released", "0.16.0"}, + {dir, "local", "latest"}, + } { + if out, err := exec.Command("bash", append([]string{script}, args...)...).CombinedOutput(); err == nil { + t.Errorf("bundle-record %v: want a refusal, got\n%s", args[1:], out) + } + } + writeImageTar(t, filepath.Join(dir, "moose-ui.tar"), "not-a-digest") + if out, err := exec.Command("bash", script, dir, "local", "0.16.0").CombinedOutput(); err == nil { + t.Errorf("unreadable config: want a refusal, got\n%s", out) + } +} diff --git a/dev/release/decide.sh b/dev/release/decide.sh new file mode 100755 index 00000000..4021a7f6 --- /dev/null +++ b/dev/release/decide.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# Decide whether this push to main cuts a release on one version line. +# +# decide.sh +# +# moose has two version lines (BUILD.md # Versioning, DECISIONS.md 2026-10-01), +# and .github/workflows/release.yml calls this once for each: +# +# decide.sh VERSION v "$before" # the OS release +# decide.sh CONTROL_PLANE_VERSION control-plane-v "$before" # the control plane +# +# It runs in the repo's working tree, which must hold full history and tags +# (release.yml checks out with fetch-depth: 0). It writes key=value lines to +# stdout, ready to append to $GITHUB_OUTPUT, and explains itself on stderr: +# +# created=true|false this commit releases this line: publish what it ships +# tag_exists=true|false the git tag is already there (at this commit). Then +# release.yml does not tag again, creates the GitHub +# Release only if it is missing, and the publish steps +# add only what is missing. +# tag= the tag the file names +# version= the file's contents +# prev_tag= the newest earlier tag on this line, or empty. The +# Release notes start from it, so a control-plane Release +# never lists commits since the last OS tag. +# +# FOUR outcomes: +# +# - no tag for the file's version -> cut the release. +# - tag exists at THIS commit -> resume: the release was cut +# here already, so carry on and +# publish what is still missing. +# - tag exists elsewhere, file unchanged -> ordinary merge; clean no-op. +# - tag exists elsewhere, file bumped -> hard error (exit 1). +# +# The resume case is what makes a half-done run recoverable. A run that tags the +# OS line and then fails on the control-plane tag, or on either GitHub Release, +# leaves a tag at this commit; re-running the workflow must finish the job, not +# trip over the tag it made itself. A tag at another commit is never resumed. +# +# The last case is what a plain "does the tag exist?" check gets wrong. +# release.yml tags BEFORE building, so a release whose image build fails leaves +# the tag behind. If the fix is then merged without deleting that tag, a bare +# existence check sees the tag, exits green, and the release silently never +# happens. Bumping a version file to a version that is already tagged is always +# a real release that cannot be cut, so say so loudly and say what to do. +# +# One more refusal, for a line that publishes images under a tag shape it shares +# with an older line. The control plane pushes ghcr images tagged v, the +# shape every release before the split used (the private control plane resolves +# digests by that tag). If RELEASE_IMAGE_CHECK is set, it is run with v +# as its last argument when the git tag is missing: exit 0 means an image tag +# already exists (release.yml checks the brain and the UI in one call), and +# the release is refused rather than overwrite released images with new bytes +# under the same number. Exit 1 means it does not exist. Anything else is an +# error, never "missing". +# +# RELEASE_REMOTE names the remote to read tags from (default: origin), and +# RELEASE_SHA the commit this run releases (default: HEAD; release.yml checks out +# GITHUB_SHA, so the two agree there). +set -euo pipefail + +file="${1:?usage: decide.sh }" +prefix="${2:?usage: decide.sh }" +before="${3:-}" +remote="${RELEASE_REMOTE:-origin}" +sha="$(git rev-parse "${RELEASE_SHA:-HEAD}^{commit}")" + +err() { echo "::error::$*" >&2; exit 1; } + +[ -f "$file" ] || err "$file is missing at the repo root" +cur="$(tr -d '[:space:]' < "$file")" +[[ "$cur" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || err "$file holds '$cur', not a plain X.Y.Z version" +want="${prefix}${cur}" + +echo "tag=$want" +echo "version=$cur" + +# The newest earlier tag on this line. The pattern is anchored on the full tag +# shape, so the OS prefix "v" never picks up a "control-plane-v" tag. +prev_tag="$(git ls-remote --tags --refs "$remote" "refs/tags/${prefix}*" \ + | sed 's|.*refs/tags/||' \ + | grep -E "^${prefix}[0-9]+\.[0-9]+\.[0-9]+$" \ + | grep -vxF "$want" \ + | sort -V \ + | tail -n1 || true)" +echo "prev_tag=$prev_tag" + +# The commit the tag points at, peeled, so an annotated tag compares as its +# commit. ls-remote lists the peeled "^{}" line after the tag's own line. +tag_commit="$(git ls-remote --tags "$remote" "refs/tags/$want" "refs/tags/$want^{}" | tail -n1 | cut -f1)" + +if [ -z "$tag_commit" ]; then + if [ -n "${RELEASE_IMAGE_CHECK:-}" ]; then + rc=0 + $RELEASE_IMAGE_CHECK "v${cur}" || rc=$? + case "$rc" in + 0) err "$file names $cur, and there is no git tag $want, but the image tag v${cur} is already published. Releasing it would overwrite released images with new bytes under the same number. Bump $file, or, if those images really are this release, tag $want at the commit they were built from." ;; + 1) ;; + *) err "could not tell whether the image tag v${cur} is already published (check exited $rc). Refusing to release rather than risk overwriting it; re-run once the registry answers." ;; + esac + fi + echo "created=true" + echo "tag_exists=false" + echo "release: tag $want does not exist yet; cutting a release" >&2 + exit 0 +fi + +if [ "$tag_commit" = "$sha" ]; then + echo "created=true" + echo "tag_exists=true" + echo "resume: tag $want already points at this commit; carrying on with what is not yet published" >&2 + exit 0 +fi + +# The tag exists. Did THIS push change the file? `before` is main's previous +# head; on the very first push, a force-push, or a commit where the file didn't +# yet exist it may be absent/unreadable, in which case we fall back to the +# skip behaviour but say so. +prev="" +if [ -n "$before" ] \ + && [ "$before" != "0000000000000000000000000000000000000000" ] \ + && git cat-file -e "${before}^{commit}" 2>/dev/null; then + prev="$(git show "${before}:${file}" 2>/dev/null | tr -d '[:space:]')" || prev="" +fi + +if [ -n "$prev" ] && [ "$prev" != "$cur" ]; then + at="$(git rev-parse --short "refs/tags/$want" 2>/dev/null || echo unknown)" + err "$file was bumped ($prev -> $cur) but tag $want already exists (at $at). Refusing to skip a real release silently. Delete the stale tag and its GitHub Release and re-run this workflow, or bump $file again." +fi + +if [ -z "$prev" ]; then + echo "::warning::could not read $file at ${before:-} to confirm whether this merge bumped it; assuming it did not, since tag $want already exists." >&2 +fi + +echo "created=false" +echo "tag_exists=true" +echo "skip: tag $want already exists and $file was not bumped by this merge; nothing to release on this line" >&2 diff --git a/dev/release/decide_test.go b/dev/release/decide_test.go new file mode 100644 index 00000000..c3534f08 --- /dev/null +++ b/dev/release/decide_test.go @@ -0,0 +1,304 @@ +// Package release tests the release decision script, dev/release/decide.sh, +// which .github/workflows/release.yml runs on every push to main. The workflow +// itself only runs on main, so this is where its logic is exercised: each case +// builds a throwaway "origin" repo and a clone, sets up tags and version files, +// and runs the script the way the workflow does. +package release + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +type repo struct { + t *testing.T + clone string +} + +func run(t *testing.T, dir string, name string, args ...string) string { + t.Helper() + cmd := exec.Command(name, args...) + cmd.Dir = dir + cmd.Env = append(os.Environ(), + "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.invalid", + "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.invalid", + "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("%s %v: %v\n%s", name, args, err, out) + } + return strings.TrimSpace(string(out)) +} + +func newRepo(t *testing.T) *repo { + t.Helper() + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not installed") + } + root := t.TempDir() + origin := filepath.Join(root, "origin.git") + clone := filepath.Join(root, "clone") + run(t, root, "git", "init", "-q", "--bare", origin) + run(t, root, "git", "clone", "-q", origin, clone) + run(t, clone, "git", "checkout", "-q", "-b", "main") + return &repo{t: t, clone: clone} +} + +// commit writes the given files and commits them, returning the new sha. +func (r *repo) commit(files map[string]string) string { + r.t.Helper() + for name, body := range files { + if err := os.WriteFile(filepath.Join(r.clone, name), []byte(body+"\n"), 0o644); err != nil { + r.t.Fatal(err) + } + run(r.t, r.clone, "git", "add", name) + } + run(r.t, r.clone, "git", "commit", "-q", "--allow-empty", "-m", "c") + run(r.t, r.clone, "git", "push", "-q", "origin", "main") + return run(r.t, r.clone, "git", "rev-parse", "HEAD") +} + +func (r *repo) tag(name, sha string) { + r.t.Helper() + run(r.t, r.clone, "git", "tag", name, sha) + run(r.t, r.clone, "git", "push", "-q", "origin", name) +} + +type result struct { + out map[string]string + ok bool + logs string +} + +// decide runs the script as release.yml does, with an optional image check. +func (r *repo) decide(file, prefix, before, imageCheck string) result { + r.t.Helper() + script, err := filepath.Abs("decide.sh") + if err != nil { + r.t.Fatal(err) + } + cmd := exec.Command("bash", script, file, prefix, before) + cmd.Dir = r.clone + cmd.Env = append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "RELEASE_IMAGE_CHECK="+imageCheck) + var stdout, stderr strings.Builder + cmd.Stdout, cmd.Stderr = &stdout, &stderr + err = cmd.Run() + res := result{out: map[string]string{}, ok: err == nil, logs: stderr.String()} + for _, line := range strings.Split(stdout.String(), "\n") { + if k, v, found := strings.Cut(line, "="); found { + res.out[k] = v + } + } + return res +} + +func TestBumpOfOneLineReleasesOnlyThatLine(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"VERSION": "0.15.0", "CONTROL_PLANE_VERSION": "0.15.0"}) + r.tag("v0.15.0", base) + r.tag("control-plane-v0.15.0", base) + + // Only CONTROL_PLANE_VERSION moves. + r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.16.0"}) + osLine := r.decide("VERSION", "v", base, "") + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "") + if !osLine.ok || osLine.out["created"] != "false" { + t.Fatalf("OS line: want a clean no-op, got ok=%v out=%v\n%s", osLine.ok, osLine.out, osLine.logs) + } + if !cp.ok || cp.out["created"] != "true" || cp.out["tag"] != "control-plane-v0.16.0" { + t.Fatalf("control-plane line: want a release of control-plane-v0.16.0, got ok=%v out=%v\n%s", cp.ok, cp.out, cp.logs) + } + // The notes start from the last tag on the same line, never the OS tag. + if cp.out["prev_tag"] != "control-plane-v0.15.0" { + t.Errorf("control-plane prev_tag = %q, want control-plane-v0.15.0", cp.out["prev_tag"]) + } + if osLine.out["prev_tag"] != "" { + t.Errorf("OS prev_tag = %q, want empty: the only v* tag is the current one, and control-plane-v* must not count", osLine.out["prev_tag"]) + } +} + +func TestBumpOfVersionReleasesOnlyTheOS(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"VERSION": "0.15.0", "CONTROL_PLANE_VERSION": "0.15.0"}) + r.tag("v0.15.0", base) + r.tag("control-plane-v0.15.0", base) + r.commit(map[string]string{"VERSION": "0.15.1"}) + + osLine := r.decide("VERSION", "v", base, "") + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "") + if !osLine.ok || osLine.out["created"] != "true" || osLine.out["tag"] != "v0.15.1" || osLine.out["prev_tag"] != "v0.15.0" { + t.Fatalf("OS line: want a release of v0.15.1 from v0.15.0, got ok=%v out=%v\n%s", osLine.ok, osLine.out, osLine.logs) + } + if !cp.ok || cp.out["created"] != "false" { + t.Fatalf("control-plane line: want a clean no-op, got ok=%v out=%v\n%s", cp.ok, cp.out, cp.logs) + } +} + +func TestBumpToAnAlreadyTaggedVersionFailsLoudly(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.15.0"}) + r.tag("control-plane-v0.15.0", base) + r.tag("control-plane-v0.16.0", base) // left behind by a failed earlier release + r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.16.0"}) + + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "") + if cp.ok { + t.Fatalf("want a hard error for a bump to an already-tagged version, got out=%v", cp.out) + } + if !strings.Contains(cp.logs, "already exists") { + t.Errorf("error should say the tag already exists; got:\n%s", cp.logs) + } +} + +func TestPublishedImageTagIsNeverOverwritten(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.15.0"}) + + // No control-plane-v0.15.0 git tag yet, but the image tag v0.15.0 is + // published (the first release before the split). Exit 0 = exists. + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "true") + if cp.ok { + t.Fatalf("want a refusal when the image tag is already published, got out=%v", cp.out) + } + if !strings.Contains(cp.logs, "already published") { + t.Errorf("error should say the image tag is already published; got:\n%s", cp.logs) + } + + // The registry cannot answer (exit 2): refuse, never read it as missing. + cp = r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "sh -c 'exit 2'") + if cp.ok { + t.Fatalf("want a refusal when the image check errors, got out=%v", cp.out) + } + + // The image tag is free (exit 1): release. + cp = r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "false") + if !cp.ok || cp.out["created"] != "true" { + t.Fatalf("want a release when the image tag is free, got ok=%v out=%v\n%s", cp.ok, cp.out, cp.logs) + } +} + +func TestFileThatIsNotAVersionIsRefused(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"CONTROL_PLANE_VERSION": "v0.15.0"}) + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "") + if cp.ok { + t.Fatalf("want a refusal for a file that is not X.Y.Z, got out=%v", cp.out) + } +} + +func TestFirstPushWithNoPreviousFileWarnsAndSkips(t *testing.T) { + r := newRepo(t) + before := r.commit(map[string]string{"VERSION": "0.15.0"}) + r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.15.0"}) + // The one-time hand tag sits at an older commit, as control-plane-v0.15.0 + // does at v0.15.0. + r.tag("control-plane-v0.15.0", before) + + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", before, "") + if !cp.ok || cp.out["created"] != "false" { + t.Fatalf("want a skip when the tag exists and the file is new, got ok=%v out=%v\n%s", cp.ok, cp.out, cp.logs) + } + if !strings.Contains(cp.logs, "::warning::") { + t.Errorf("want a warning that the previous value could not be read; got:\n%s", cp.logs) + } +} + +// A run that tagged one line and then died (on the other line's tag, or on a +// GitHub Release) must be resumable: a re-run sees its own tag at this commit +// and carries on, instead of tripping the "bumped to an already-tagged +// version" error. A tag at any other commit is never resumed. +func TestTagAtThisCommitResumes(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"VERSION": "0.15.0", "CONTROL_PLANE_VERSION": "0.15.0"}) + r.tag("v0.15.0", base) + r.tag("control-plane-v0.15.0", base) + head := r.commit(map[string]string{"VERSION": "0.16.0", "CONTROL_PLANE_VERSION": "0.16.0"}) + r.tag("v0.16.0", head) // the first run got this far + + // The image check would refuse, but a resume must not run it: the images + // may be half-pushed by the run being resumed, and that is what it finishes. + osLine := r.decide("VERSION", "v", base, "true") + if !osLine.ok || osLine.out["created"] != "true" || osLine.out["tag_exists"] != "true" { + t.Fatalf("OS line: want a resume (created, tag_exists), got ok=%v out=%v\n%s", osLine.ok, osLine.out, osLine.logs) + } + cp := r.decide("CONTROL_PLANE_VERSION", "control-plane-v", base, "false") + if !cp.ok || cp.out["created"] != "true" || cp.out["tag_exists"] != "false" { + t.Fatalf("control-plane line: want a fresh release, got ok=%v out=%v\n%s", cp.ok, cp.out, cp.logs) + } + + // The same tag, but this run releases a later commit: a hard error, as + // for any bump to a version tagged elsewhere. + r.commit(map[string]string{"VERSION": "0.16.0", "README": "x"}) + if again := r.decide("VERSION", "v", base, ""); again.ok { + t.Fatalf("want a hard error for a tag at another commit, got out=%v", again.out) + } +} + +// An annotated tag at this commit resumes too: the tag object is not the +// commit, so the script must compare the peeled commit. +func TestAnnotatedTagAtThisCommitResumes(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"VERSION": "0.15.0"}) + head := r.commit(map[string]string{"VERSION": "0.16.0"}) + run(t, r.clone, "git", "tag", "-a", "-m", "m", "v0.16.0", head) + run(t, r.clone, "git", "push", "-q", "origin", "v0.16.0") + res := r.decide("VERSION", "v", base, "") + if !res.ok || res.out["created"] != "true" || res.out["tag_exists"] != "true" { + t.Fatalf("want a resume for an annotated tag, got ok=%v out=%v\n%s", res.ok, res.out, res.logs) + } +} + +// ghcr-tag-exists.sh with a stub curl: one call checks several repositories, +// any one published means "exists", and an unclear answer is never "missing". +func TestImageCheckAcrossRepos(t *testing.T) { + bin := t.TempDir() + stub := `#!/usr/bin/env bash +for a in "$@"; do + case "$a" in + *ghcr.io/token*) echo '{"token":"t"}'; exit 0 ;; + *ghcr.io/v2/*/manifests/*) + name="${a#https://ghcr.io/v2/}"; name="${name%%/manifests/*}"; name="${name//\//_}" + var="FAKE_${name}"; printf '%s' "${!var:-404}"; exit 0 ;; + esac +done +exit 1 +` + if err := os.WriteFile(filepath.Join(bin, "curl"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + script, err := filepath.Abs("ghcr-tag-exists.sh") + if err != nil { + t.Fatal(err) + } + cases := []struct { + name string + brain, ui string + wantExitRC int + }{ + {"both missing", "404", "404", 1}, + {"only the ui is published", "404", "200", 0}, + {"only the brain is published", "200", "404", 0}, + {"ui unclear, brain missing", "404", "500", 2}, + {"ui unclear, brain published", "200", "500", 0}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + cmd := exec.Command("bash", script, "onmoose/brain", "onmoose/ui", "v0.16.0") + cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), + "FAKE_onmoose_brain="+c.brain, "FAKE_onmoose_ui="+c.ui) + err := cmd.Run() + rc := 0 + if ee, ok := err.(*exec.ExitError); ok { + rc = ee.ExitCode() + } else if err != nil { + t.Fatal(err) + } + if rc != c.wantExitRC { + t.Errorf("exit = %d, want %d", rc, c.wantExitRC) + } + }) + } +} diff --git a/dev/release/ghcr-resolve.sh b/dev/release/ghcr-resolve.sh new file mode 100755 index 00000000..ffb9cf37 --- /dev/null +++ b/dev/release/ghcr-resolve.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Resolve a published ghcr image tag to its digests (#566). Anonymous and +# read-only: the moose images are public (BUILD.md # 6), so no login is needed +# and none is used. +# +# ghcr-resolve.sh +# +# Prints one line: " ". The manifest digest is +# what a box pulls by (`docker pull ghcr.io/@`). +# The config digest is the image ID Docker gives that image once it is loaded, +# which is how the boot lane checks that a box runs exactly these bytes. +# +# The tag is read once, here, and never again: the build pulls by the digest +# this prints, so a tag that moves during the build cannot change what is +# baked. When the tag names an index (a multi-platform image), the linux/amd64 +# entry is taken, because that is the only platform moose builds. +# +# The digest is computed from the bytes the registry served, not taken from a +# header, and must match the header when the registry sends one. Any HTTP +# answer other than 200 fails the script: a release build must never bake +# something it could not pin. +set -euo pipefail + +[ "$#" -eq 2 ] || { echo "usage: ghcr-resolve.sh " >&2; exit 2; } +repo="$1"; ref="$2" + +accept="application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json" + +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT + +token="$(curl -fsS "https://ghcr.io/token?scope=repository:${repo}:pull" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])' 2>/dev/null)" || { + echo "ghcr-resolve: could not get an anonymous token for ${repo}" >&2 + exit 1 +} + +# fetch REF: the manifest at REF into $work/body, its headers into $work/head. +# Prints the sha256 of the body as "sha256:". +fetch() { + local r="$1" status sum want + status="$(curl -sS -o "$work/body" -D "$work/head" -w '%{http_code}' \ + -H "Authorization: Bearer ${token}" -H "Accept: ${accept}" \ + "https://ghcr.io/v2/${repo}/manifests/${r}")" || status="000" + if [ "$status" != 200 ]; then + echo "ghcr-resolve: ${repo}:${r} returned HTTP ${status}" >&2 + return 1 + fi + sum="sha256:$(sha256sum "$work/body" | cut -d' ' -f1)" + want="$(tr -d '\r' < "$work/head" | awk -F': ' 'tolower($1) == "docker-content-digest" {print $2}' | tail -n1)" + if [ -n "$want" ] && [ "$want" != "$sum" ]; then + echo "ghcr-resolve: ${repo}:${r} sent ${sum}, but the registry says ${want}" >&2 + return 1 + fi + printf '%s\n' "$sum" +} + +# kind: "index ", "manifest ", or fails. +kind() { + python3 - "$work/body" <<'PY' +import json, sys +d = json.load(open(sys.argv[1])) +mt = d.get("mediaType", "") +if "manifests" in d: + for m in d["manifests"]: + p = m.get("platform") or {} + if p.get("os") == "linux" and p.get("architecture") == "amd64": + print("index", m["digest"]) + sys.exit(0) + sys.exit("no linux/amd64 image in the index") +cfg = (d.get("config") or {}).get("digest", "") +if not cfg.startswith("sha256:") or len(cfg) != 71: + sys.exit("manifest has no config digest (mediaType %r)" % mt) +print("manifest", cfg) +PY +} + +digest="$(fetch "$ref")" +read -r k v < <(kind) || { echo "ghcr-resolve: cannot read the manifest of ${repo}:${ref}" >&2; exit 1; } +if [ "$k" = index ]; then + digest="$(fetch "$v")" + [ "$digest" = "$v" ] || { echo "ghcr-resolve: ${repo}@${v} came back as ${digest}" >&2; exit 1; } + read -r k v < <(kind) || { echo "ghcr-resolve: cannot read the manifest of ${repo}@${digest}" >&2; exit 1; } + [ "$k" = manifest ] || { echo "ghcr-resolve: ${repo}@${digest} is an index inside an index" >&2; exit 1; } +fi +case "$digest" in sha256:????????????????????????????????????????????????????????????????) ;; *) + echo "ghcr-resolve: bad digest '${digest}'" >&2; exit 1 ;; +esac +printf '%s %s\n' "$digest" "$v" diff --git a/dev/release/ghcr-retag.sh b/dev/release/ghcr-retag.sh new file mode 100755 index 00000000..54b848df --- /dev/null +++ b/dev/release/ghcr-retag.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Point one ghcr tag at the manifest another tag already names, byte for byte, +# so both tags carry the same digest. Nothing is rebuilt or re-pushed: the +# manifest is read and written back under the new tag. +# +# GHCR_USER= GHCR_TOKEN= ghcr-retag.sh +# +# ci-cloud-image.yml uses it to repair `latest` when a run that pushed `vX.Y.Z` +# died before `latest`. It prints the digest on stdout and checks it: after the +# write, must answer with the same digest as , or it exits 1. +set -euo pipefail + +repo="${1:?usage: ghcr-retag.sh }" +src="${2:?usage: ghcr-retag.sh }" +dst="${3:?usage: ghcr-retag.sh }" +: "${GHCR_USER:?GHCR_USER is required}" "${GHCR_TOKEN:?GHCR_TOKEN is required}" + +accept="application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json" +api="https://ghcr.io/v2/${repo}/manifests" + +token="$(curl -fsS -u "${GHCR_USER}:${GHCR_TOKEN}" "https://ghcr.io/token?scope=repository:${repo}:pull,push" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])' 2>/dev/null)" || { + echo "ghcr-retag: could not get a push token for ${repo}" >&2 + exit 1 +} + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +curl -fsS -H "Authorization: Bearer ${token}" -H "Accept: ${accept}" \ + -D "$tmp/headers" -o "$tmp/manifest" "${api}/${src}" || { + echo "ghcr-retag: could not read ${repo}:${src}" >&2 + exit 1 +} +ctype="$(grep -i '^content-type:' "$tmp/headers" | tail -n1 | cut -d: -f2- | tr -d ' \r')" +[ -n "$ctype" ] || { echo "ghcr-retag: ${repo}:${src} answered with no content type" >&2; exit 1; } +digest="sha256:$(sha256sum "$tmp/manifest" | cut -d' ' -f1)" + +curl -fsS -X PUT -H "Authorization: Bearer ${token}" -H "Content-Type: ${ctype}" \ + --data-binary "@$tmp/manifest" -o /dev/null "${api}/${dst}" || { + echo "ghcr-retag: could not write ${repo}:${dst}" >&2 + exit 1 +} + +# Read it back: the registry's own digest for must be the one we wrote. +got="$(curl -fsS -I -H "Authorization: Bearer ${token}" -H "Accept: ${accept}" "${api}/${dst}" \ + | grep -i '^docker-content-digest:' | tail -n1 | cut -d: -f2- | tr -d ' \r')" || got="" +if [ "$got" != "$digest" ]; then + echo "ghcr-retag: ${repo}:${dst} answers '${got:-nothing}', want ${digest}" >&2 + exit 1 +fi +echo "$digest" diff --git a/dev/release/ghcr-tag-exists.sh b/dev/release/ghcr-tag-exists.sh new file mode 100755 index 00000000..dea9d850 --- /dev/null +++ b/dev/release/ghcr-tag-exists.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Does a ghcr image tag exist, in any of the given repositories? Anonymous, +# read-only: the moose images are public (BUILD.md # 6), so no login is needed +# and none is used. +# +# ghcr-tag-exists.sh [...] +# +# The tag comes last, so dev/release/decide.sh can append it to a configured +# command. release.yml passes the brain and the UI in one call, because the two +# are always pushed together and either one already published is a reason to +# stop. +# +# Exit 0: the tag exists in at least one repository. Exit 1: it is missing from +# all of them (a definite 404 each). Exit 2: no "exists", and at least one +# repository gave some other answer. The caller must treat 2 as "unknown", +# never as "missing", or a registry outage would read as permission to publish +# over a released tag. +set -euo pipefail + +[ "$#" -ge 2 ] || { echo "usage: ghcr-tag-exists.sh [...] " >&2; exit 2; } +tag="${!#}" +repos=("${@:1:$#-1}") + +accept="application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json" + +# one REPO -> 0 exists, 1 missing, 2 unknown +one() { + local repo="$1" token status + token="$(curl -fsS "https://ghcr.io/token?scope=repository:${repo}:pull" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])' 2>/dev/null)" || { + echo "ghcr-tag-exists: could not get an anonymous token for ${repo}" >&2 + return 2 + } + status="$(curl -sS -o /dev/null -w '%{http_code}' -I \ + -H "Authorization: Bearer ${token}" -H "Accept: ${accept}" \ + "https://ghcr.io/v2/${repo}/manifests/${tag}")" || status="000" + case "$status" in + 200) echo "ghcr-tag-exists: ${repo}:${tag} is published" >&2; return 0 ;; + 404) return 1 ;; + *) echo "ghcr-tag-exists: ${repo}:${tag} lookup returned HTTP ${status}" >&2; return 2 ;; + esac +} + +result=1 +for repo in "${repos[@]}"; do + rc=0 + one "$repo" || rc=$? + case "$rc" in + 0) exit 0 ;; + 1) ;; + *) result=2 ;; + esac +done +exit "$result" diff --git a/dev/release/is-newest.sh b/dev/release/is-newest.sh new file mode 100755 index 00000000..8929b953 --- /dev/null +++ b/dev/release/is-newest.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Is the newest release on a line, by semver, among the line's tags? +# +# is-newest.sh +# +# ci-cloud-image.yml asks this before it moves the control-plane images' +# `latest` tag, so re-running or dispatching an older control-plane release can +# never point `latest` back at older images: +# +# is-newest.sh control-plane-v 0.16.0 +# +# Exit 0: no tag on the line is newer (equal is fine: that is this release's own +# tag). Exit 1: a newer tag exists. Exit 2: the tag list could not be read, or +# is not X.Y.Z. The caller treats 2 as "do not touch latest", never as +# "newest". Tags of another shape on the line (a suffix, a typo) are ignored, so +# they can neither block nor move `latest`. +# +# Tags are read from RELEASE_REMOTE (default: origin). The repo is public, so +# this needs no credentials. +set -euo pipefail + +prefix="${1:?usage: is-newest.sh }" +ver="${2:?usage: is-newest.sh }" +remote="${RELEASE_REMOTE:-origin}" + +[[ "$ver" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "is-newest: '$ver' is not X.Y.Z" >&2; exit 2; } + +list="$(git ls-remote --tags --refs "$remote" "refs/tags/${prefix}*")" || { + echo "is-newest: could not read the ${prefix}* tags from ${remote}" >&2 + exit 2 +} + +newest="$(printf '%s\n' "$list" \ + | sed -n "s|.*refs/tags/${prefix}\([0-9]*\.[0-9]*\.[0-9]*\)$|\1|p" \ + | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' \ + | sort -V | tail -n1 || true)" + +if [ -z "$newest" ]; then + exit 0 +fi +top="$(printf '%s\n%s\n' "$newest" "$ver" | sort -V | tail -n1)" +if [ "$top" = "$ver" ]; then + exit 0 +fi +echo "is-newest: ${prefix}${newest} is newer than ${prefix}${ver}" >&2 +exit 1 diff --git a/dev/release/publish_test.go b/dev/release/publish_test.go new file mode 100644 index 00000000..7a7c5a6e --- /dev/null +++ b/dev/release/publish_test.go @@ -0,0 +1,263 @@ +package release + +import ( + "crypto/sha256" + "encoding/hex" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// stubBin writes an executable stub called name into a fresh dir and returns +// the dir, to put first on PATH. +func stubBin(t *testing.T, name, body string) string { + t.Helper() + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil { + t.Fatal(err) + } + return dir +} + +func runScript(t *testing.T, env []string, script string, args ...string) (int, string) { + t.Helper() + abs, err := filepath.Abs(script) + if err != nil { + t.Fatal(err) + } + cmd := exec.Command("bash", append([]string{abs}, args...)...) + cmd.Dir = t.TempDir() + cmd.Env = append(os.Environ(), env...) + out, err := cmd.CombinedOutput() + if ee, ok := err.(*exec.ExitError); ok { + return ee.ExitCode(), string(out) + } else if err != nil { + t.Fatal(err) + } + return 0, string(out) +} + +// The image and its checksum are one pair: both present is left, neither is +// uploaded together, and exactly one refuses, naming the stray to delete. A +// published asset is never uploaded over and never deleted. +func TestAttachImageTreatsImageAndChecksumAsOnePair(t *testing.T) { + gh := `#!/usr/bin/env bash +case "$1 $2" in + "release view") printf '%s' "$FAKE_ASSETS" | tr ',' '\n'; exit 0 ;; + "release upload") shift 3; echo "$*" >> "$FAKE_LOG"; exit 0 ;; +esac +echo "unexpected gh call: $*" >&2; exit 3 +` + bin := stubBin(t, "gh", gh) + const img, sum = "moose-v0.16.0-amd64.raw.xz", "moose-v0.16.0-amd64.raw.xz.sha256" + cases := []struct { + name string + assets string + wantRC int + wantUpload string // what upload was called with, or "" for none + wantInOut string + }{ + {"neither: upload both together", "", 0, img + " " + sum, "attached"}, + {"both: leave them", img + "," + sum, 0, "", "already has"}, + {"only the image: refuse, name it", img, 1, "", "gh release delete-asset v0.16.0 " + img + " -y"}, + {"only the checksum: refuse, name it", sum, 1, "", "gh release delete-asset v0.16.0 " + sum + " -y"}, + {"other assets only: upload both", "notes.txt", 0, img + " " + sum, "attached"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + log := filepath.Join(t.TempDir(), "uploads") + rc, out := runScript(t, []string{"PATH=" + bin + ":" + os.Getenv("PATH"), "FAKE_ASSETS=" + c.assets, "FAKE_LOG=" + log}, + "attach-image.sh", "v0.16.0", img, sum) + if rc != c.wantRC { + t.Fatalf("exit = %d, want %d\n%s", rc, c.wantRC, out) + } + if !strings.Contains(out, c.wantInOut) { + t.Errorf("output should contain %q:\n%s", c.wantInOut, out) + } + got, _ := os.ReadFile(log) + if strings.TrimSpace(string(got)) != c.wantUpload { + t.Errorf("uploads = %q, want %q", strings.TrimSpace(string(got)), c.wantUpload) + } + }) + } +} + +// An OS release attaches four files as one set (#562): the image, the bundle +// and their checksums. All present is left, none is uploaded in one call, and +// any other mix refuses and names every published one to delete, including a +// Release cut before the bundle existed (the image pair only). +func TestAttachImageTreatsTheOSReleaseAsOneSet(t *testing.T) { + gh := `#!/usr/bin/env bash +case "$1 $2" in + "release view") printf '%s' "$FAKE_ASSETS" | tr ',' '\n'; exit 0 ;; + "release upload") shift 3; echo "$*" >> "$FAKE_LOG"; exit 0 ;; +esac +echo "unexpected gh call: $*" >&2; exit 3 +` + bin := stubBin(t, "gh", gh) + const ( + img = "moose-v0.16.0-amd64.raw.xz" + isum = img + ".sha256" + bnd = "moose-v0.16.0-amd64.raucb" + bsum = bnd + ".sha256" + ) + all := []string{img, isum, bnd, bsum} + cases := []struct { + name string + assets string + wantRC int + wantUpload string + wantInOut []string + }{ + {"none: upload all four together", "", 0, strings.Join(all, " "), []string{"attached"}}, + {"all four: leave them", strings.Join(all, ","), 0, "", []string{"already has"}}, + {"image pair only (cut before #562): refuse", img + "," + isum, 1, "", []string{"delete-asset v0.16.0 " + img + " -y", "delete-asset v0.16.0 " + isum + " -y", "not " + bnd + " " + bsum}}, + {"bundle only: refuse", bnd, 1, "", []string{"delete-asset v0.16.0 " + bnd + " -y"}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + log := filepath.Join(t.TempDir(), "uploads") + rc, out := runScript(t, []string{"PATH=" + bin + ":" + os.Getenv("PATH"), "FAKE_ASSETS=" + c.assets, "FAKE_LOG=" + log}, + "attach-image.sh", append([]string{"v0.16.0"}, all...)...) + if rc != c.wantRC { + t.Fatalf("exit = %d, want %d\n%s", rc, c.wantRC, out) + } + for _, w := range c.wantInOut { + if !strings.Contains(out, w) { + t.Errorf("output should contain %q:\n%s", w, out) + } + } + got, _ := os.ReadFile(log) + if strings.TrimSpace(string(got)) != c.wantUpload { + t.Errorf("uploads = %q, want %q", strings.TrimSpace(string(got)), c.wantUpload) + } + }) + } + // One file is not a set. + if rc, _ := runScript(t, []string{"PATH=" + bin + ":" + os.Getenv("PATH")}, "attach-image.sh", "v0.16.0", img); rc != 2 { + t.Errorf("one file: exit = %d, want 2", rc) + } +} + +// ghcr-retag.sh copies the source tag's manifest to the destination tag byte +// for byte, with its content type, and checks the registry then reports the +// same digest. A registry that answers with another digest fails the run. +func TestRetagCopiesTheManifestByteForByte(t *testing.T) { + // A stub registry: GET serves $FAKE_MANIFEST, PUT stores the body and its + // content type, HEAD answers with the digest of what was stored (or + // $FAKE_HEAD_DIGEST, to simulate a registry that disagrees). + curl := `#!/usr/bin/env bash +method=GET headers="" out="" data="" ctype="" url="" +while [ $# -gt 0 ]; do + case "$1" in + -X) method="$2"; shift ;; + -I) method=HEAD ;; + -D) headers="$2"; shift ;; + -o) out="$2"; shift ;; + -u) shift ;; + --data-binary) data="${2#@}"; shift ;; + -H) case "$2" in Content-Type:*) ctype="${2#Content-Type: }";; esac; shift ;; + -*) ;; + *) url="$1" ;; + esac + shift +done +case "$url" in + *ghcr.io/token*) echo '{"token":"t"}'; exit 0 ;; +esac +case "$method" in + GET) + printf 'HTTP/2 200\r\ncontent-type: application/vnd.oci.image.index.v1+json\r\n\r\n' > "$headers" + printf '%s' "$FAKE_MANIFEST" > "$out" ;; + PUT) + echo "$url" > "$FAKE_DIR/put_url"; echo "$ctype" > "$FAKE_DIR/put_ctype"; cp "$data" "$FAKE_DIR/put_body" ;; + HEAD) + d="${FAKE_HEAD_DIGEST:-sha256:$(sha256sum "$FAKE_DIR/put_body" | cut -d' ' -f1)}" + printf 'HTTP/2 200\r\ndocker-content-digest: %s\r\n\r\n' "$d" ;; +esac +` + bin := stubBin(t, "curl", curl) + manifest := `{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[]}` + sum := sha256.Sum256([]byte(manifest)) + want := "sha256:" + hex.EncodeToString(sum[:]) + + dir := t.TempDir() + env := []string{"PATH=" + bin + ":" + os.Getenv("PATH"), "FAKE_DIR=" + dir, "FAKE_MANIFEST=" + manifest, + "GHCR_USER=u", "GHCR_TOKEN=t"} + rc, out := runScript(t, env, "ghcr-retag.sh", "onmoose/brain", "v0.16.0", "latest") + if rc != 0 { + t.Fatalf("exit = %d, want 0\n%s", rc, out) + } + if strings.TrimSpace(out) != want { + t.Errorf("printed digest = %q, want %q", strings.TrimSpace(out), want) + } + body, _ := os.ReadFile(filepath.Join(dir, "put_body")) + if string(body) != manifest { + t.Errorf("PUT body = %q, want the source manifest unchanged", body) + } + ctype, _ := os.ReadFile(filepath.Join(dir, "put_ctype")) + if strings.TrimSpace(string(ctype)) != "application/vnd.oci.image.index.v1+json" { + t.Errorf("PUT content type = %q, want the source's", ctype) + } + url, _ := os.ReadFile(filepath.Join(dir, "put_url")) + if !strings.HasSuffix(strings.TrimSpace(string(url)), "/v2/onmoose/brain/manifests/latest") { + t.Errorf("PUT went to %q, want .../onmoose/brain/manifests/latest", url) + } + + // The registry reports a different digest after the write: fail. + rc, out = runScript(t, append(env, "FAKE_HEAD_DIGEST=sha256:"+strings.Repeat("0", 64)), + "ghcr-retag.sh", "onmoose/brain", "v0.16.0", "latest") + if rc == 0 { + t.Fatalf("want a failure when the registry disagrees on the digest, got 0\n%s", out) + } +} + +// is-newest.sh guards `latest`: it moves only for the newest control-plane +// version by semver, so an older release re-run or dispatched never moves it +// back. An unreadable tag list is "unknown" (exit 2), never "newest". +func TestIsNewestGuardsLatest(t *testing.T) { + r := newRepo(t) + base := r.commit(map[string]string{"CONTROL_PLANE_VERSION": "0.9.0"}) + r.tag("control-plane-v0.9.0", base) + r.tag("control-plane-v0.10.0", base) + r.tag("control-plane-v99.0.0-rc1", base) // not X.Y.Z: ignored + r.tag("v99.0.0", base) // the OS line: does not count + + newest := func(prefix, ver, remote string) int { + t.Helper() + script, err := filepath.Abs("is-newest.sh") + if err != nil { + t.Fatal(err) + } + cmd := exec.Command("bash", script, prefix, ver) + cmd.Dir = r.clone + cmd.Env = append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "RELEASE_REMOTE="+remote) + err = cmd.Run() + if ee, ok := err.(*exec.ExitError); ok { + return ee.ExitCode() + } else if err != nil { + t.Fatal(err) + } + return 0 + } + cases := []struct { + name, prefix, ver, remote string + want int + }{ + {"an older version never moves latest", "control-plane-v", "0.9.0", "origin", 1}, + {"semver, not text: 0.10.0 is newer than 0.9.0", "control-plane-v", "0.10.0", "origin", 0}, + {"a version newer than every tag", "control-plane-v", "0.11.0", "origin", 0}, + {"no tags on the line yet", "nothing-v", "0.1.0", "origin", 0}, + {"not X.Y.Z", "control-plane-v", "0.10", "origin", 2}, + {"unreadable tag list", "control-plane-v", "0.10.0", filepath.Join(t.TempDir(), "missing.git"), 2}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := newest(c.prefix, c.ver, c.remote); got != c.want { + t.Errorf("exit = %d, want %d", got, c.want) + } + }) + } +} diff --git a/dev/release/rauc-ca.sh b/dev/release/rauc-ca.sh new file mode 100755 index 00000000..c92042c6 --- /dev/null +++ b/dev/release/rauc-ca.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +# Make the RAUC root CA and issue bundle signers from it (#562, +# docs/dev/rauc-signing.md, DECISIONS.md 2026-10-02 key custody). +# +# rauc-ca.sh root DIR make DIR/root-ca.key and DIR/root-ca.pem +# rauc-ca.sh signer DIR NAME issue DIR/NAME.key and DIR/NAME.pem from the root in DIR +# rauc-ca.sh show FILE.pem print a certificate's subject, dates and purpose +# +# The maintainer runs `root` and `signer` OFFLINE, on a machine that is not +# CI. The root key never leaves that machine: it is written encrypted +# (AES-256), and openssl asks for its passphrase. root-ca.pem is public: it is +# committed as dev/release/rauc/release-ca.pem and baked into every OS image as +# /etc/rauc/keyring.pem. A signer's key and cert go into the `os-release` +# GitHub Environment as RAUC_SIGNING_KEY and RAUC_SIGNING_CERT. +# +# CI uses the same script for the throwaway CA of a run that publishes no OS +# (dev/cloud/rauc.sh), with RAUC_CA_NO_PASSPHRASE=1 and short lifetimes. That +# is the only reason the knobs below exist; never set them for the real root. +# +# Knobs (environment): +# RAUC_CA_NAME the CN prefix (default "moose OS release") +# RAUC_CA_ROOT_DAYS the root's lifetime (default 7300, 20 years) +# RAUC_CA_SIGNER_DAYS a signer's lifetime (default 1095, 3 years) +# RAUC_CA_NO_PASSPHRASE 1 writes the root key unencrypted (throwaway only) +# +# Keys are EC P-256. A signer carries the codeSigning purpose, which the image's +# /etc/rauc/system.conf asks for (check-purpose=codesign). +set -euo pipefail + +name="${RAUC_CA_NAME:-moose OS release}" +root_days="${RAUC_CA_ROOT_DAYS:-7300}" +signer_days="${RAUC_CA_SIGNER_DAYS:-1095}" +umask 077 + +# A failed run removes what it made, so the next run can start again (a root +# key left behind without its cert would make `root` refuse the directory). +created=() +cleanup() { + local rc=$? + if [ "$rc" -ne 0 ] && [ "${#created[@]}" -gt 0 ]; then + rm -f -- "${created[@]}" + echo "rauc-ca: failed; removed the partial files: ${created[*]}" >&2 + fi + exit "$rc" +} +trap cleanup EXIT + +ext() { # the x509v3 extensions both certificates use + cat <<'EOF' +[root] +basicConstraints=critical,CA:true,pathlen:0 +keyUsage=critical,keyCertSign,cRLSign +subjectKeyIdentifier=hash +[signer] +basicConstraints=critical,CA:false +keyUsage=critical,digitalSignature +extendedKeyUsage=critical,codeSigning +subjectKeyIdentifier=hash +authorityKeyIdentifier=keyid +EOF +} + +cmd="${1:-}" +case "$cmd" in +root) + dir="${2:?usage: rauc-ca.sh root DIR}" + mkdir -p "$dir" + if [ -e "$dir/root-ca.key" ] || [ -e "$dir/root-ca.pem" ]; then + echo "rauc-ca: $dir already holds a root; refusing to overwrite it" >&2 + exit 1 + fi + enc=(-aes256) + if [ "${RAUC_CA_NO_PASSPHRASE:-}" = "1" ]; then enc=(); fi + created=("$dir/root-ca.key" "$dir/root-ca.pem") + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 "${enc[@]}" -out "$dir/root-ca.key" + openssl req -x509 -new -key "$dir/root-ca.key" -sha256 -days "$root_days" \ + -subj "/CN=${name} root CA $(date -u +%Y)" \ + -config <(printf '[req]\ndistinguished_name=dn\n[dn]\n'; ext) -extensions root \ + -out "$dir/root-ca.pem" + chmod 0644 "$dir/root-ca.pem" + echo "root CA: $dir/root-ca.pem (public; commit it as dev/release/rauc/release-ca.pem)" + echo "root key: $dir/root-ca.key (secret; keep it offline)" + ;; +signer) + dir="${2:?usage: rauc-ca.sh signer DIR NAME}" + sname="${3:?usage: rauc-ca.sh signer DIR NAME}" + case "$sname" in *[!A-Za-z0-9._-]*|"") echo "rauc-ca: signer NAME may use only letters, digits, . _ -" >&2; exit 2 ;; esac + [ -f "$dir/root-ca.key" ] && [ -f "$dir/root-ca.pem" ] || { echo "rauc-ca: no root in $dir; run rauc-ca.sh root $dir first" >&2; exit 1; } + if [ -e "$dir/$sname.key" ] || [ -e "$dir/$sname.pem" ]; then + echo "rauc-ca: $dir already holds a signer called $sname; pick a new name" >&2 + exit 1 + fi + csr="$(mktemp)" + created=("$dir/$sname.key" "$dir/$sname.pem" "$csr") + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out "$dir/$sname.key" + openssl req -new -key "$dir/$sname.key" -subj "/CN=${name} signer ${sname}" -out "$csr" + openssl x509 -req -in "$csr" -CA "$dir/root-ca.pem" -CAkey "$dir/root-ca.key" -sha256 \ + -set_serial "0x$(openssl rand -hex 16)" -days "$signer_days" \ + -extfile <(ext) -extensions signer -out "$dir/$sname.pem" + chmod 0644 "$dir/$sname.pem" + rm -f "$csr" + # openssl 3.2 and later know the codesign purpose; older ones check the + # chain, and the purpose is then read from the cert itself. + if ! openssl verify -purpose codesign -x509_strict -CAfile "$dir/root-ca.pem" "$dir/$sname.pem" 2>/dev/null; then + openssl verify -x509_strict -CAfile "$dir/root-ca.pem" "$dir/$sname.pem" + openssl x509 -in "$dir/$sname.pem" -noout -ext extendedKeyUsage | grep -q "Code Signing" \ + || { echo "rauc-ca: $dir/$sname.pem lacks the codeSigning purpose" >&2; exit 1; } + fi + echo "signer cert: $dir/$sname.pem (the RAUC_SIGNING_CERT secret)" + echo "signer key: $dir/$sname.key (the RAUC_SIGNING_KEY secret; delete this copy once it is stored)" + ;; +show) + f="${2:?usage: rauc-ca.sh show FILE.pem}" + openssl x509 -in "$f" -noout -subject -issuer -startdate -enddate -ext basicConstraints,extendedKeyUsage + ;; +*) + sed -n '2,8p' "$0" | sed 's/^# \{0,1\}//' + exit 2 + ;; +esac diff --git a/dev/release/rauc/release-ca.pem b/dev/release/rauc/release-ca.pem new file mode 100644 index 00000000..5c099e83 --- /dev/null +++ b/dev/release/rauc/release-ca.pem @@ -0,0 +1,11 @@ +-----BEGIN CERTIFICATE----- +MIIBljCCAT2gAwIBAgIUaWHuSr5KJQQ5ZZkwZGyax2dDlWgwCgYIKoZIzj0EAwIw +KDEmMCQGA1UEAwwdbW9vc2UgT1MgcmVsZWFzZSByb290IENBIDIwMjYwHhcNMjYx +MDA2MTE1MTI4WhcNNDYxMDAxMTE1MTI4WjAoMSYwJAYDVQQDDB1tb29zZSBPUyBy +ZWxlYXNlIHJvb3QgQ0EgMjAyNjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABDSa +FJInR2BA7zyYTUtNcrmVFUyaACtt/6ApGqJ3/rRoszMOL6YWmzD4kyF3nAntmTjK +9zOYhBa0N8qSCbtnevGjRTBDMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/ +BAQDAgEGMB0GA1UdDgQWBBTDP9fxBAN+9kVaqz4GWPJ1nx52CDAKBggqhkjOPQQD +AgNHADBEAiApaTTpYiet+OcJleEyKbguRI82JD6jxD8sLbRaqC2HxAIgXi1onHxq +QeCi/EaneScAmcCNN/Lisb9OKMWxR54CHok= +-----END CERTIFICATE----- diff --git a/dev/release/rauc_test.go b/dev/release/rauc_test.go new file mode 100644 index 00000000..b26aeb3a --- /dev/null +++ b/dev/release/rauc_test.go @@ -0,0 +1,170 @@ +package release + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// rauc-ca.sh makes a root CA and issues a signer with the codeSigning purpose +// from it, and refuses to overwrite either. The maintainer runs it offline for +// the release root; CI runs it for the throwaway root (#562). +func TestRaucCAMakesARootAndASigner(t *testing.T) { + if _, err := exec.LookPath("openssl"); err != nil { + t.Skip("openssl not installed") + } + dir := filepath.Join(t.TempDir(), "ca") + env := []string{"RAUC_CA_NO_PASSPHRASE=1", "RAUC_CA_NAME=moose test"} + + if rc, out := runScript(t, env, "rauc-ca.sh", "root", dir); rc != 0 { + t.Fatalf("root: exit %d\n%s", rc, out) + } + if rc, out := runScript(t, env, "rauc-ca.sh", "signer", dir, "s1"); rc != 0 { + t.Fatalf("signer: exit %d\n%s", rc, out) + } + for _, f := range []string{"root-ca.pem", "root-ca.key", "s1.pem", "s1.key"} { + if _, err := os.Stat(filepath.Join(dir, f)); err != nil { + t.Fatalf("missing %s: %v", f, err) + } + } + // The keys are private to their owner; the certs are public. + for f, want := range map[string]os.FileMode{"root-ca.key": 0o600, "s1.key": 0o600, "root-ca.pem": 0o644, "s1.pem": 0o644} { + st, _ := os.Stat(filepath.Join(dir, f)) + if st.Mode().Perm() != want { + t.Errorf("%s mode = %o, want %o", f, st.Mode().Perm(), want) + } + } + + out, err := exec.Command("openssl", "x509", "-in", filepath.Join(dir, "s1.pem"), "-noout", "-ext", "extendedKeyUsage,basicConstraints").CombinedOutput() + if err != nil { + t.Fatalf("read signer: %v\n%s", err, out) + } + if !strings.Contains(string(out), "Code Signing") || !strings.Contains(string(out), "CA:FALSE") { + t.Errorf("signer lacks codeSigning or is a CA:\n%s", out) + } + out, err = exec.Command("openssl", "x509", "-in", filepath.Join(dir, "root-ca.pem"), "-noout", "-ext", "basicConstraints").CombinedOutput() + if err != nil || !strings.Contains(string(out), "CA:TRUE") { + t.Errorf("root is not a CA: %v\n%s", err, out) + } + if out, err := exec.Command("openssl", "verify", "-CAfile", filepath.Join(dir, "root-ca.pem"), filepath.Join(dir, "s1.pem")).CombinedOutput(); err != nil { + t.Errorf("signer does not chain to the root: %v\n%s", err, out) + } + + if rc, out := runScript(t, env, "rauc-ca.sh", "root", dir); rc != 1 || !strings.Contains(out, "refusing to overwrite") { + t.Errorf("second root: exit %d, want 1 and a refusal\n%s", rc, out) + } + if rc, out := runScript(t, env, "rauc-ca.sh", "signer", dir, "s1"); rc != 1 || !strings.Contains(out, "pick a new name") { + t.Errorf("second s1: exit %d, want 1 and a refusal\n%s", rc, out) + } + if rc, _ := runScript(t, env, "rauc-ca.sh", "signer", dir, "../evil"); rc != 2 { + t.Errorf("signer name with a slash: exit %d, want 2", rc) + } +} + +// sign-bundle.sh refuses before it starts any container when the release +// signer is not there, and when the image does not carry the committed release +// root (or there is none committed yet), naming the how-to. +func TestSignBundleRefusesWithoutTheReleaseSetup(t *testing.T) { + if _, err := exec.LookPath("openssl"); err != nil { + t.Skip("openssl not installed") + } + tmp := t.TempDir() + in := filepath.Join(tmp, "in.raucb") + etc := filepath.Join(tmp, "etc") + if err := os.MkdirAll(etc, 0o755); err != nil { + t.Fatal(err) + } + for _, f := range []string{in, filepath.Join(etc, "system.conf"), filepath.Join(etc, "keyring.pem")} { + if err := os.WriteFile(f, []byte("not real\n"), 0o644); err != nil { + t.Fatal(err) + } + } + // A docker that fails loudly: no refusal below may get that far. + bin := stubBin(t, "docker", "#!/bin/sh\necho DOCKER-RAN; exit 9\n") + path := "PATH=" + bin + ":" + os.Getenv("PATH") + out := filepath.Join(tmp, "out.raucb") + + rc, o := runScript(t, []string{path, "RAUC_SIGNING_CERT=", "RAUC_SIGNING_KEY="}, "sign-bundle.sh", in, etc, out) + if rc != 1 || !strings.Contains(o, "no release signer") || !strings.Contains(o, "docs/dev/rauc-signing.md") { + t.Errorf("no secrets: exit %d, want 1 naming the how-to\n%s", rc, o) + } + + rc, o = runScript(t, []string{path, "RAUC_SIGNING_CERT=cert", "RAUC_SIGNING_KEY=key"}, "sign-bundle.sh", in, etc, out) + if _, err := os.Stat("rauc/release-ca.pem"); err != nil { + // No release root committed yet: refused, naming the how-to. + if rc != 1 || !strings.Contains(o, "no release root CA") || !strings.Contains(o, "docs/dev/rauc-signing.md") { + t.Errorf("no release root: exit %d, want 1 naming the how-to\n%s", rc, o) + } + } else if rc != 1 || !strings.Contains(o, "not dev/release/rauc/release-ca.pem") { + // A root is committed: an image keyring that is not it is refused. + t.Errorf("wrong image keyring: exit %d, want 1\n%s", rc, o) + } + if strings.Contains(o, "DOCKER-RAN") { + t.Errorf("a refusal started a container:\n%s", o) + } + if _, err := os.Stat(out); err == nil { + t.Error("a refusal wrote a bundle") + } +} + +// A root run that fails part way removes the key it wrote, so a second run is +// not refused by a half-made root. +func TestRaucCAFailedRunLeavesNothing(t *testing.T) { + if _, err := exec.LookPath("openssl"); err != nil { + t.Skip("openssl not installed") + } + dir := filepath.Join(t.TempDir(), "ca") + if rc, out := runScript(t, []string{"RAUC_CA_NO_PASSPHRASE=1", "RAUC_CA_ROOT_DAYS=not-a-number"}, "rauc-ca.sh", "root", dir); rc == 0 || !strings.Contains(out, "removed the partial files") { + t.Fatalf("bad lifetime: exit %d, want a failure that cleans up\n%s", rc, out) + } + if left, _ := filepath.Glob(filepath.Join(dir, "*")); len(left) != 0 { + t.Fatalf("a failed root left %v", left) + } + if rc, out := runScript(t, []string{"RAUC_CA_NO_PASSPHRASE=1"}, "rauc-ca.sh", "root", dir); rc != 0 { + t.Fatalf("retry: exit %d\n%s", rc, out) + } +} + +// release.yml refuses to tag an OS release unless the committed release root +// is a real one: present, a CA, not a throwaway or check CA (#562). +func TestRequireReleaseCA(t *testing.T) { + if _, err := exec.LookPath("openssl"); err != nil { + t.Skip("openssl not installed") + } + tmp := t.TempDir() + mk := func(name, cn string) string { + dir := filepath.Join(tmp, name) + if rc, out := runScript(t, []string{"RAUC_CA_NO_PASSPHRASE=1", "RAUC_CA_NAME=" + cn}, "rauc-ca.sh", "root", dir); rc != 0 { + t.Fatalf("make %s: exit %d\n%s", name, rc, out) + } + return filepath.Join(dir, "root-ca.pem") + } + good := mk("real", "moose OS release") + throwaway := mk("throwaway", "moose THROWAWAY (not for release)") + cases := []struct { + name, file string + wantRC int + want string + }{ + {"missing", filepath.Join(tmp, "none.pem"), 1, "docs/dev/rauc-signing.md"}, + {"throwaway", throwaway, 1, "throwaway or check CA"}, + {"real", good, 0, "release root CA: subject=CN"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + rc, out := runScript(t, []string{"MOOSE_RAUC_RELEASE_CA_FILE=" + c.file}, "require-release-ca.sh") + if rc != c.wantRC || !strings.Contains(out, c.want) { + t.Errorf("exit %d, want %d with %q\n%s", rc, c.wantRC, c.want, out) + } + }) + } + // A signer is not a CA. + if rc, out := runScript(t, []string{"RAUC_CA_NO_PASSPHRASE=1"}, "rauc-ca.sh", "signer", filepath.Join(tmp, "real"), "s"); rc != 0 { + t.Fatalf("signer: %d\n%s", rc, out) + } + if rc, out := runScript(t, []string{"MOOSE_RAUC_RELEASE_CA_FILE=" + filepath.Join(tmp, "real", "s.pem")}, "require-release-ca.sh"); rc != 1 || !strings.Contains(out, "not a CA certificate") { + t.Errorf("signer as root: exit %d\n%s", rc, out) + } +} diff --git a/dev/release/require-release-ca.sh b/dev/release/require-release-ca.sh new file mode 100755 index 00000000..36a2375f --- /dev/null +++ b/dev/release/require-release-ca.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Fail, naming docs/dev/rauc-signing.md, when no OS release can be signed +# because dev/release/rauc/release-ca.pem is missing, is not a CA, has expired, +# or is a throwaway (#562, dev/cloud/rauc.sh rauc_require_release_ca). +# +# release.yml runs it before it tags or creates any Release for a merge that +# bumps VERSION, so an OS release cut before the maintainer's key setup stops +# with nothing tagged, on either line. ci-cloud-image.yml runs it first on any +# run that publishes the OS line. +set -euo pipefail +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +# shellcheck source=dev/cloud/rauc.sh +. "${REPO_ROOT}/dev/cloud/rauc.sh" +if ! msg="$(rauc_require_release_ca 2>&1)"; then + echo "::error::${msg}" + exit 1 +fi +echo "release root CA: $(openssl x509 -in "$RAUC_RELEASE_CA" -noout -subject -enddate | tr '\n' ' ')" diff --git a/dev/release/sign-bundle.sh b/dev/release/sign-bundle.sh new file mode 100755 index 00000000..2023be79 --- /dev/null +++ b/dev/release/sign-bundle.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# Re-sign the OS update bundle with the release signer (#562, +# docs/dev/rauc-signing.md, DECISIONS.md 2026-10-02 key custody). +# +# sign-bundle.sh IN.raucb IMAGE_ETC OUT.raucb +# +# Run by the `sign` job of ci-cloud-image.yml, the only job that sees the +# release signer: RAUC_SIGNING_CERT and RAUC_SIGNING_KEY, secrets of the +# `os-release` GitHub Environment, passed in as environment variables (PEM +# text). IN.raucb is the bundle the build job made and signed with its +# throwaway key. IMAGE_ETC holds system.conf and keyring.pem as the build job +# read them back out of the image's slot. +# +# It refuses, before it signs anything: +# - when either secret is empty (the environment or its secrets are missing); +# - when the image keyring is not dev/release/rauc/release-ca.pem, so the +# image that ships trusts exactly the committed release root; +# - when the image keyring ACCEPTS the throwaway-signed IN.raucb: an image +# that ships may never trust the throwaway key. +# Then it replaces the signature (`rauc resign`; the payload stays byte for +# byte the same) and checks OUT.raucb against the image's own config and +# keyring. A bundle that does not verify is never written to OUT. +# +# `rauc resign --no-verify`: re-checking a verity payload needs a loop device +# and exclusive access, which the container has not got (CI run 37005541036). +# The signature of IN is still checked against the throwaway root by the build +# job, and OUT is fully checked here. +# +# MOOSE_RAUC_REHEARSAL=1 (dev/cloud/build-bundle.sh only): IMAGE_ETC holds a +# throwaway "release" CA, so the committed release CA is not compared. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +# shellcheck source=dev/cloud/rauc.sh +. "${REPO_ROOT}/dev/cloud/rauc.sh" + +in="$(realpath "${1:?usage: sign-bundle.sh IN.raucb IMAGE_ETC OUT.raucb}")" +etc="$(realpath "${2:?usage: sign-bundle.sh IN.raucb IMAGE_ETC OUT.raucb}")" +dst="${3:?usage: sign-bundle.sh IN.raucb IMAGE_ETC OUT.raucb}" + +if [ -z "${RAUC_SIGNING_CERT:-}" ] || [ -z "${RAUC_SIGNING_KEY:-}" ]; then + echo "::error::no release signer: RAUC_SIGNING_CERT or RAUC_SIGNING_KEY is empty. They are secrets of the os-release GitHub Environment, which only main and v* tags may use. Refusing to publish a bundle signed with the throwaway key. See ${RAUC_HOWTO}." >&2 + exit 1 +fi +[ -f "$etc/system.conf" ] && [ -f "$etc/keyring.pem" ] || { echo "sign-bundle: $etc must hold system.conf and keyring.pem" >&2; exit 1; } +if [ "${MOOSE_RAUC_REHEARSAL:-}" != "1" ]; then + rauc_require_release_ca + cmp -s "$etc/keyring.pem" "$RAUC_RELEASE_CA" || { echo "::error::the image's /etc/rauc/keyring.pem is not dev/release/rauc/release-ca.pem; this image was not built for release" >&2; exit 1; } +fi + +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT +chmod 0700 "$work" +mkdir "$work/etc" +cp "$etc/system.conf" "$etc/keyring.pem" "$work/etc/" +ln "$in" "$work/in.raucb" 2>/dev/null || cp "$in" "$work/in.raucb" +printf '%s\n' "$RAUC_SIGNING_CERT" > "$work/signer.pem" +( umask 077; printf '%s\n' "$RAUC_SIGNING_KEY" > "$work/signer.key" ) + +rauc_run "$work" ' + if rauc --conf=etc/system.conf info in.raucb >/dev/null 2>&1; then + echo "::error::the image keyring accepts the throwaway-signed bundle; refusing to sign" >&2; exit 1 + fi + echo "ok: the image keyring refuses the throwaway-signed input" + rauc --conf=etc/system.conf resign --no-verify --cert=signer.pem --key=signer.key --signing-keyring=etc/keyring.pem in.raucb out.raucb + rauc --conf=etc/system.conf info out.raucb > out-info.txt + echo "ok: the re-signed bundle verifies against the image keyring" + sed -n "/^Certificate Chain:/,\$p" out-info.txt + rm -f signer.key +' +mv "$work/out.raucb" "$dst" +echo "signed bundle: $dst" diff --git a/docs/README.md b/docs/README.md index 5aed2140..f4a8ea4c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -54,6 +54,7 @@ doesn't duplicate the list. `moose-brain` (unit → store → lifecycle-with-fakes → API → integration → e2e). Companion to `specs/TESTING.md`, which covers boot-level lanes. - [`dev/hosted-boot-proof.md`](dev/hosted-boot-proof.md) — the "where we are now" runbook for hosted HTTPS bring-up (seed → `EnsureWildcardTLS` → `:443` bind, the two-path cert model) and the `dev/cloud/` boot-proof: the brain-log milestones to grep, a symptom → where-to-look table, and how to run the lane (CI vs local). Read this before chasing a red cloud boot-proof or a "`:443` doesn't bind" report. Operational companion to `specs/TESTING.md` # Hosted cloud variant and `specs/ENVIRONMENT.md` # Networking & discovery. +- [`dev/rauc-signing.md`](dev/rauc-signing.md) — the maintainer's how-to for the OS update bundle's keys: make the offline root CA, issue a CI signer, create the `os-release` environment and its two secrets, commit `release-ca.pem`, rotate the signer and replace the root (#562, `specs/DECISIONS.md` 2026-10-02). - [`dev/code-review.md`](dev/code-review.md) — the review standard: what to read before reviewing, 12 lenses (correctness, security, spec fidelity, Go discipline, audit completeness, test coverage, documentation honesty, scope, migration safety, error quality, dependencies, commit hygiene), severity levels, and output format. Used by the pre-PR self-review step and the dedicated review agent. - [`dev/authoring-apps-with-an-agent.md`](dev/authoring-apps-with-an-agent.md) — a reusable agent prompt that turns an upstream `docker-compose.yml` (or a GitHub repo) into a Door-1 catalog app: rewrites the compose to pass admission, rewires env vars to moose's injected values, resolves image digests, and writes the `apps//` package into the private `store` repo. The author-side adaptation tool for growing the catalog; the run ends at an open store PR. - [`dev/catalog-import-gaps.md`](dev/catalog-import-gaps.md) — append-only ledger of platform gaps surfaced during catalog imports (an env var moose can't inject, a secret it can't generate). Import agents capture findings here; a human triages and promotes recurring/severe gaps into `specs/NEXT.md`. Capture, not the design backlog — the two are kept distinct on purpose. diff --git a/docs/architecture.md b/docs/architecture.md index c53dbc67..17d7d8a7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,7 +14,7 @@ one is JavaScript, one is a container we don't write. |---|---|---|---| | **`moose-brain`** | `cmd/brain/`, `internal/` | The control-plane daemon. Owns SQLite state, the REST+SSE API, the app lifecycle, and the Caddy config. One Go binary. | Real | | **`host-agent` (fake)** | `cmd/host-agent/` | Privileged side used in the inner dev loop. Speaks the real `BRAIN_HOST_PROTOCOL.md` wire format over a UNIX socket; the host operations themselves (Avahi, LUKS, PAM, apt) are stubbed in memory. | **Fake** (real wire, canned ops) | -| **`host-agent-real`** | `cmd/host-agent-real/`, `internal/hostagent/` | The real privileged binary. Seam-injected reporters: PAM password verify (`pamverifier`), `/proc` system sampling (`procsource`), disk usage, RAM pressure, journal streaming, service health, reboot-required flag, user manager, system time-zone setter (`timezone`, `timedatectl set-timezone` — the first-run wizard's Step 3, wired in both build profiles), per-account SSH access (`sshaccess`, #463 — renders `sshd_config.d/moose-allowed.conf` whole from the enabled set, validates with `sshd -t`, and starts or stops sshd so :22 is open only while someone has SSH on; wired in both build profiles). Discovery is real: per-LAN-interface Avahi announcements (`avahipublisher`) driven by the NetworkManager LAN set (`netstate`), with an avahi-daemon.conf allowlist sync and IP-change replay. Seeds the brain's Docker transport then launches the brain container on startup (`brainlaunch`: `EnsureTransport` creates `moose-ingress` + runs the `docker-socket-proxy`; `Launch` docker-loads the bundled image if absent, lockstep `moose.protocol.major` OCI-label check, `docker run --restart unless-stopped` on the ingress net with `DOCKER_HOST` at the proxy). Before the launch it makes sure the household shared tree `/srv/moose/shared` exists as `root:moose-shared` `02770` (`brainlaunch.EnsureSharedTree`) and mounts it into the brain at the same path, so the brain can prepare shared folder sources (#519). Personal folder sources are prepared by host-agent itself, over `POST /v1/users/{username}/prepare-folder` (`usermgr`, a walk that never follows a symlink), so the brain never mounts `/home`. Host ops not yet wired: LUKS/TPM, apt, NM configuration (WiFi setup, `/v1/network/*`). A build-tagged slim **`hosted`** profile (`go build -tags hosted`, #204/C1c) compiles the discovery/NetworkManager stack out for the cloud image — `avahipublisher`/`netstate` unwired, no-op publisher, nil `Net` — keeping the same PAM/user-mgmt/health-system/brain-launch seams (`cmd/host-agent-real/wiring_appliance.go` vs `wiring_hosted.go`). | Partial — see "What is not built yet" | +| **`host-agent-real`** | `cmd/host-agent-real/`, `internal/hostagent/` | The real privileged binary. Seam-injected reporters: PAM password verify (`pamverifier`), `/proc` system sampling (`procsource`), disk usage, RAM pressure, journal streaming, service health, reboot-required flag, user manager, system time-zone setter (`timezone`, `timedatectl set-timezone` — the first-run wizard's Step 3, wired in both build profiles), per-account SSH access (`sshaccess`, #463 — renders `sshd_config.d/moose-allowed.conf` whole from the enabled set, validates with `sshd -t`, and starts or stops sshd so :22 is open only while someone has SSH on; wired in both build profiles). Discovery is real: per-LAN-interface Avahi announcements (`avahipublisher`) driven by the NetworkManager LAN set (`netstate`), with an avahi-daemon.conf allowlist sync and IP-change replay. Seeds the brain's Docker transport then launches the brain container on startup (`brainlaunch`: `EnsureTransport` creates `moose-ingress` + runs the `docker-socket-proxy`; `Launch` docker-loads the bundled image if absent, lockstep `moose.protocol.major` OCI-label check, `docker run --restart unless-stopped` on the ingress net with `DOCKER_HOST` at the proxy). Before the launch it makes sure the household shared tree `/srv/moose/shared` exists as `root:moose-shared` `02770` (`brainlaunch.EnsureSharedTree`) and mounts it into the brain at the same path, so the brain can prepare shared folder sources (#519). Personal folder sources are prepared by host-agent itself, over `POST /v1/users/{username}/prepare-folder` (`usermgr`, a walk that never follows a symlink), so the brain never mounts `/home`. The A/B OS update on the hosted build (`osupdate`, #563: RAUC install into the other slot, switch in the window, trial boot, revert). Host ops not yet wired: LUKS/TPM, NM configuration (WiFi setup, `/v1/network/*`). A build-tagged slim **`hosted`** profile (`go build -tags hosted`, #204/C1c) compiles the discovery/NetworkManager stack out for the cloud image — `avahipublisher`/`netstate` unwired, no-op publisher, nil `Net` — keeping the same PAM/user-mgmt/health-system/brain-launch seams (`cmd/host-agent-real/wiring_appliance.go` vs `wiring_hosted.go`). | Partial — see "What is not built yet" | | **Caddy** | `dev/caddy.json`, `dev/docker-compose.yml` | Reverse proxy. Terminates `*.local` (appliance) or `*..onmoose.io` over real Let's Encrypt HTTPS (hosted, via a custom acme-dns build) and routes to app containers + the brain. Configured live by the brain via Caddy's admin API. | Real (container) | | **`web-ui`** | `web-ui/` | Vue 3 + Vite + TanStack Query dashboard. Talks only to the brain. Tailwind 4 with the Oatmeal `@theme` tokens; `reka-ui` + `cn()` are present as shadcn-vue scaffolding, but the owned components in `components/ui/` (`Button`, `Heading`) are hand-written from the Oatmeal patterns, not pulled through the shadcn CLI (#261). The catalog install is a set of steps, one need per step with the saved accounts listed and one picked, and the last step installs; an app with no steps installs from the App page (`INSTALL_STEPS.md`, `DASHBOARD.md` # Install authorization). Internal code architecture: [`dev/web-ui.md`](dev/web-ui.md). | Real | | **SQLite** | `$STATE_DIR/moose.db` | The brain's only persistent store. Schema + queries in `internal/store/`. | Real | @@ -107,7 +107,7 @@ the `host-agent-real` row in # Components and by # What is not built yet. | `applog` | Per-app log fan-out (`BRAIN_UI_PROTOCOL.md` Pattern C, `LOGGING.md` # Per-app logs). Sits between host-agent's single upstream follow per instance and the dashboard's many SSE readers, and owns the reconnect contract host-agent deliberately does not: a ~256 KiB ring buffer, replay from `Last-Event-ID`, one `{"lost":true}` marker when a position was evicted, and a linger so a quick reconnect reuses the warm buffer. One ref-counted `Hub` per instance — zero idle cost when nobody is watching. | `api`, `cmd/brain` | | `systemlive` | The live system-resources stream (`BRAIN_UI_PROTOCOL.md` Pattern C stream 3, `LOCAL_ANALYTICS.md`). Ref-counted upstream poller: the first SSE subscriber starts a 1 Hz poll of host-agent's raw cumulative counters, each poll is diffed into rates and fanned out, the last unsubscribe stops it. Same zero-idle-cost shape as `applog`. | `api`, `cmd/brain` | | `storageverify` | The canary + enrollment-marker check behind the `moose-storage-verify` reporter (`BOOT.md` # The storage-ready target, `STORAGE.md` # Storage canary). Split out of `cmd/` only so the check is unit-testable against a tempdir root; the binary is a thin shell that writes findings to `/run/moose/health/storage.json`. **Not a brain package** — it is imported by `cmd/moose-storage-verify` alone. | `cmd/moose-storage-verify` | -| `version` | The moose build identity: `Version` (repo `VERSION` file) and `Commit` (git sha), stamped at build time via `-ldflags -X` (`Makefile`, `BUILD.md` # Versioning). Dumb — vars + a `String()`, no logic. | `api`, `hostagent`, `cmd/brain`, `cmd/host-agent`, `cmd/host-agent-real` | +| `version` | A binary's build identity: `Version` (its release line's version file) and `Commit` (git sha), stamped at build time via `-ldflags -X` (`Makefile`, `cmd/brain/Dockerfile`, `BUILD.md` # Versioning). host-agent is stamped from `VERSION` (the moose OS release), the brain from `CONTROL_PLANE_VERSION`. Dumb: vars, `String()` (OS form) and `ControlPlaneString()` (brain form), no logic. | `api`, `hostagent`, `cmd/brain`, `cmd/host-agent`, `cmd/host-agent-real` | **Cross-cutting invariants:** @@ -173,7 +173,60 @@ So this doc isn't read as a claim about the finished product: wherever Docker puts volumes. - **Boot, install ISO, updates.** The `mkosi` image build (`BUILD.md` # 2; proven in the test lane, not yet the production ISO) and stream A - (`unattended-upgrades` + the apt repo) are spec-only. **Stream B — the + (an A/B OS image with RAUC and GRUB, `UPDATES.md` # 1 and `BUILD.md` + # 1b, #486) is half built. **The hosted image is in the A/B layout (#561):** + the 128 MiB ESP, a BIOS boot partition and slot A, a read-only 1 GiB + squashfs (xz) holding the kernel and initramfs, in the image; slot B (1 GiB) + and a state partition (the rest of the disk, grown on every boot) made by + `systemd-repart` in the initramfs at first boot. The OS reserves 5.7% of a + 40 GB disk, and the build fails when the squashfs fills more than 60% of its + slot (`dev/cloud/slotbudget`). GRUB on both firmwares from one `grub.cfg` + and `grubenv`; the `/etc` overlay, the four pinned files and the bind mounts + set up by an initramfs-tools hook (`dev/cloud/mkosi.extra/usr/lib/moose/state-setup`); + `rauc` and `rauc-service` with the slot config and a keyring + (`/etc/rauc/keyring.pem`). **Every OS release builds a signed RAUC bundle + (#562):** slot A of the image that ships, in the verity format, built by + `dev/cloud/build-bundle.sh` and checked against the keyring read back out of + the slot, re-signed in a `sign` job of its own (the only job that enters the + `os-release` environment, `dev/release/sign-bundle.sh`; an offline root CA, + `docs/dev/rauc-signing.md`) and attached beside the image. host-agent's + hosted build reports the state partition as its "System" volume + (`diskusage.NewHosted`). **A hosted box updates its OS (#563):** + `internal/hostagent/osupdate` downloads the bundle the update target names, + checks its sha256 against the target, has RAUC install it into the other + slot ahead of the window, switches and reboots inside it (`os-install` and + `os-switch` jobs under the one job lock, after stream B), and on the next + boot marks the slot good once the brain answers, or reboots back. An image + timer (`moose-os-trial.timer`) reboots a slot whose host-agent never + started. Only the first boot after a switch is on trial. The brain reports + `os_version`/`os_slot` and stream A's decision, and raises one admin + notification per outcome. Proven by the `os-update` and `os-revert` boots + under both firmwares, and `os-revert` also proves that GRUB skips a slot + whose kernel panics before userspace (#575). **A Debian major tidies the + `/etc` upper layer** (`BUILD.md` # 1b, rule 4): `state-setup` keeps the + files on `/usr/lib/moose/etc-keep.list`, merges the account files, takes the + pinned files again from the slot when the remap stays, and moves the rest to + an attic, in both directions; the `os-update` boot fakes a major, and every + boot fails on an upper-layer file no rule covers. The image's accounts are a + generated `sysusers.d` file whose ids the build checks against + `dev/os-lock/cloud-accounts.lock`, and host-agent turns sshd back on once, + at its first start after a tidy-up, when the drop-in names an account + (`sshaccess.EnsureOnAtStart`, gated by `/state/etc/.moose-major-tidied`). systemd + feeds the hardware watchdog a Hetzner VM has (ICH9 TCO), checked on every + boot. Not built: the OS part of the private control plane's + answer (described in `docs/progress/host-agent-os-update.md`), so no + production box moves its OS yet, and the appliance layout (#564). **The OS + package lock is built (#560)** for the hosted image: `dev/os-lock/` holds a + snapshot.debian.org timestamp, exact Docker pins and the resolved package + list, both cloud builds fail when they resolve to a different list, and + `os-lock-bump.yml` moves it forward daily. The appliance lane is not locked + yet (`BUILD.md` # 1b # The OS package lock). **An OS-only release bakes + the last released control plane (#566):** the brain and UI of + `v`, pulled from ghcr by digest + (`make control-plane-released`); a control-plane release and every run + that publishes nothing bake a build of the commit. The image records the + pair in `/usr/lib/moose/control-plane.env`, and every boot checks the + running brain and UI against it (`BUILD.md` # Versioning). **Stream B — the control-plane update — is half built.** A box declares its brain/UI pair in two files (`internal/hostagent/controlplane`: the staged compose plus an `images.json` ledger), the apply/health-check/revert transaction exists diff --git a/docs/dev/contributing.md b/docs/dev/contributing.md index e817e6dd..cadef906 100644 --- a/docs/dev/contributing.md +++ b/docs/dev/contributing.md @@ -154,17 +154,31 @@ If the box-side half is not happening now, open an issue here describing it in b ## Release model -Feature work always branches off `dev` and PRs into `dev` — that's covered above. Releases are a separate, maintainer-only step layered on top, and **the `VERSION` bump is what makes a merge a release** — everything past that point is automatic: - -- The maintainer bumps the repo-root `VERSION` file to the new `X.Y.Z` (BUILD.md # Versioning: one repo version for the whole monorepo, not independent per-component SemVer — DECISIONS.md 2026-07-16) as part of preparing the release, and opens a PR from `dev` into `main`. A dev->main PR **is** a release candidate — but only if it bumps `VERSION`; a dev->main merge that doesn't touch `VERSION` ships nothing (see below). +Feature work always branches off `dev` and PRs into `dev` — that's covered above. Releases are a separate, maintainer-only step layered on top. moose has **two release lines**, one per update stream (`../specs/BUILD.md` # Versioning, `../specs/DECISIONS.md` 2026-10-01), and **a version-file bump is what makes a merge a release** on that line. Everything past that point is automatic: + +| Line | File | Git tag + GitHub Release | What it publishes | +|---|---|---|---| +| moose (OS) | `VERSION` | `vX.Y.Z` | the disk image and the signed OS update bundle, attached to the Release as `moose-vX.Y.Z-amd64.raw.xz` + `.sha256` and `moose-vX.Y.Z-amd64.raucb` + `.sha256` (#562) | +| control plane (brain + UI) | `CONTROL_PLANE_VERSION` | `control-plane-vX.Y.Z` | `ghcr.io/onmoose/brain` and `ghcr.io/onmoose/ui`, image tag `vX.Y.Z` + `latest` | + +- **An OS release needs the bundle signer** (#562). Its root CA (`dev/release/rauc/release-ca.pem`) must be committed, and the `os-release` GitHub Environment must hold `RAUC_SIGNING_CERT` and `RAUC_SIGNING_KEY`. Setting them up, and rotating them, is `rauc-signing.md`. The exact rule: + - **A merge that bumps `VERSION` while `release-ca.pem` is missing (or not a valid release root) tags nothing.** `release.yml` stops before it tags or creates any Release, on either line, even when the same merge also bumps `CONTROL_PLANE_VERSION`. The OS line is never dropped quietly. Commit the root first, or release the control plane alone by bumping only `CONTROL_PLANE_VERSION`. + - **A merge that bumps only `CONTROL_PLANE_VERSION` does not need the root or the signer**, and releases as before. + - **With the root committed but a secret missing**, the tags and the Releases are made, and the `sign` job of the cloud-image run fails, so nothing is published on either line. Add the secrets and re-run `release.yml`: it resumes. +- **To cut an OS release**, bump `VERSION`. **To cut a control-plane release**, bump `CONTROL_PLANE_VERSION`. Bump both to cut both. Each file holds the **last released** version of its line. A dev->main PR **is** a release candidate, but only if it bumps one of them; a dev->main merge that bumps neither ships nothing. +- The control-plane image tag has **no** `control-plane-` prefix, on purpose: the private control plane resolves digests by the `vX.Y.Z` image tag. Only the git tag and the GitHub Release carry the prefix. - Both `ci-go.yml` and `ci-web.yml` gate the dev->main PR the same way they gate every feature PR, so a release can't merge with a broken build or a stale OpenAPI/TS client. -- Once the dev->main PR merges, `.github/workflows/release.yml` runs on every push to `main`. It reads `VERSION` and checks whether a `vX.Y.Z` tag for it already exists: - - **If the tag already exists** (this merge didn't bump `VERSION`), the run is a clean no-op — no tag, no release, no image build. This is the common case for most `main` pushes and is expected to stay green. - - **If the tag doesn't exist** (this merge bumped `VERSION`), the workflow tags the merge commit `vX.Y.Z`, creates a GitHub Release for it, and then triggers the hosted cloud-image build+publish (`ci-cloud-image.yml`) for that same commit with publishing enabled. The Release notes are the **release PR's own body**, with the generated commit list appended under it; if that body is missing or too short to be a summary, the notes fall back to the generated list alone. So the summary you write in the release PR is what people read on the Release page — write it for someone deciding whether to upgrade, not for someone reading commits. -- The cloud-image build is invoked directly as a reusable workflow (`workflow_call`), not via `ci-cloud-image.yml`'s `push: tags` trigger — a tag pushed with the default `GITHUB_TOKEN` (as `release.yml` does) does not fire another workflow's tag-push trigger, so relying on that event would silently tag a release and never build or publish it. `ci-cloud-image.yml`'s `push: tags: v*` trigger still exists as a manual escape hatch for a human pushing a tag by hand; see that workflow's header comment for the full reasoning. -- A tagged release always ships an image stamped with that same version, runs the full seeded-boot gate, and attaches the image to the GitHub Release as `moose-vX.Y.Z-amd64.raw.xz` + a `.sha256` sidecar. That Release asset is the only published artifact — the provider-snapshot upload was removed in #352, so a release no longer pushes to any hosting provider. `workflow_dispatch` on `ci-cloud-image.yml` remains available for manual build-only or build+publish runs outside the release flow (see the workflow's header comment). - -Contributors never push directly to `main`; the tag and the GitHub Release are created automatically by `release.yml`, not by hand. +- Once the dev->main PR merges, `.github/workflows/release.yml` runs on every push to `main` and decides each line on its own (`dev/release/decide.sh`): + - **If the tag for the file's version already exists** and this merge did not change the file, that line is a clean no-op. This is the common case and is expected to stay green. + - **If the tag doesn't exist**, the workflow tags the merge commit, creates a GitHub Release for it, and then runs the hosted cloud-image build (`ci-cloud-image.yml`) for that same commit with that line's publish switch on. The Release notes are the **release PR's own body**, with the commit list since the previous tag on the same line appended under it; if that body is missing or too short to be a summary, the notes fall back to the generated list alone. So the summary you write in the release PR is what people read on the Release page — write it for someone deciding whether to upgrade, not for someone reading commits. A control-plane Release is created with `--latest=false`, so GitHub's "Latest release" stays on the OS release with the disk image. + - **If the tag already points at this merge commit**, an earlier run for this same push got that far and then failed. The run carries on: it does not tag again, creates the Release only if it is missing, and publishes only what is not published yet. **To recover a failed release, re-run `release.yml`.** + - **If the file was bumped to a version tagged at another commit**, the run fails and says so. It never skips a real release silently. + - **If the control-plane version's image tag is already on ghcr**, for the brain or the UI, with no `control-plane-vX.Y.Z` git tag, the run fails before tagging anything, rather than overwrite released images with new bytes under the same number. +- **Every release runs the full boot proof.** A control-plane-only release still builds the disk image and boots it, then pushes the images that image baked and skips only the attach of the OS release files. An OS-only release attaches the disk image and the signed bundle (#562) and pushes nothing to ghcr. A release that bumps both does both from one run. +- The cloud-image build is invoked directly as a reusable workflow (`workflow_call`), not via `ci-cloud-image.yml`'s `push: tags` trigger — a tag pushed with the default `GITHUB_TOKEN` (as `release.yml` does) does not fire another workflow's tag-push trigger, so relying on that event would silently tag a release and never build or publish it. `ci-cloud-image.yml`'s `push: tags: v*` trigger still exists as a manual escape hatch for a human pushing an OS tag by hand (it publishes the OS release files only: the disk image and the signed bundle with their checksums, never the control-plane images; like any OS release it needs the release root CA committed and the `os-release` signer, `rauc-signing.md`); see that workflow's header comment for the full reasoning. `workflow_dispatch` on `ci-cloud-image.yml` remains available for manual build-only runs (`-f publish=false`), build+publish runs of both lines (`publish=true`, the default), or one line (`-f publish=false -f publish_control_plane=true`, or `publish_os=true`). A publish never replaces an asset already on a Release or an image tag already on ghcr; delete it first to replace it on purpose. The four OS files are one set: a Release holding some of them (for example one cut before #562, with the image but no bundle) refuses, and the error lists the `gh release delete-asset` lines to run first. +- **Which brain and UI the disk image bakes** (#566, `../specs/BUILD.md` # Versioning). An OS-only release bakes the **last released** control plane, pulled from ghcr by digest at `v`, and its boot proof runs against those images. A control-plane release, or a release that bumps both files, bakes the brain and UI it builds and releases. So when `main` holds control-plane changes that the OS release's boot checks rely on, an OS-only release fails at the boot proof: release the control plane first, or bump both files in the same PR. To check ahead, dispatch `gh workflow run "CI / Cloud image" --ref -f publish=false -f control_plane=released`. + +Contributors never push directly to `main`; the tags and the GitHub Releases are created automatically by `release.yml`, not by hand. **The one exception is a one-time step** for the control-plane line's first tag. `CONTROL_PLANE_VERSION` starts at `0.15.0`, and the `v0.15.0` images on ghcr were built at the `v0.15.0` commit. So tag that commit `control-plane-v0.15.0` once, before the first dev->main merge after #559 lands: `git tag control-plane-v0.15.0 v0.15.0 && git push origin control-plane-v0.15.0`. Until that tag exists, `release.yml` refuses on every push to `main`, because the `v0.15.0` image tag is already published. ### Cutting a release, step by step @@ -174,7 +188,8 @@ The mechanics above say what happens automatically. This is the part a person do # 1. Land the version bump on dev, like any other change. git checkout dev && git pull git checkout -b release/X.Y.Z -echo "X.Y.Z" > VERSION +echo "X.Y.Z" > VERSION # an OS release +echo "X.Y.Z" > CONTROL_PLANE_VERSION # a control-plane release (either, or both) # commit, PR into dev, merge. # 2. Cut the release branch and bring main back into it. @@ -185,22 +200,94 @@ git merge origin/main # expect conflicts; see below # 3. After the release PR merges, carry main back into dev. git checkout dev && git pull -git merge origin/main # resolve VERSION to dev's value; see below +git merge origin/main # resolve the version files to dev's values; see below git push ``` -`VERSION` conflicts on every release, because `main` still holds the previous number. **Always resolve it to the new `X.Y.Z`** — that file is the release trigger, so resolving it the other way ships nothing. Other conflicts are ordinary content: keep both sides unless they genuinely contradict. +A bumped version file conflicts on every release, because `main` still holds the previous number. **Always resolve it to the new `X.Y.Z`**: that file is the release trigger, so resolving it the other way ships nothing. Other conflicts are ordinary content: keep both sides unless they genuinely contradict. Then open the PR from `release/X.Y.Z` into `main`, let `ci-go.yml` and `ci-web.yml` gate it, and merge. `release.yml` does the rest. **Why step 3 is needed at all:** anything that landed on `main` without going through `dev` is missing from `dev`, and the next `dev` -> `main` PR then conflicts. The release merge itself is always one such commit, so `dev` falls behind at every release even when nothing else slipped in. Step 3 is what keeps the next release starting from a clean `dev`. Do it right after the release PR merges: the work only grows if left. -`VERSION` conflicts here too, in the other direction. **Resolve it to `dev`'s value**, the version being worked towards, not the one `main` just released. Resolving it the other way walks the version backwards on `dev`. +The version files conflict here too, in the other direction. **Resolve each to `dev`'s value**, the version being worked towards, not the one `main` just released. Resolving it the other way walks the version backwards on `dev`. -Anything *other* than the release merge showing up in step 3 means a PR was opened against `main` directly. That is the mistake to catch, not to clean up after: **every PR targets `dev`**, including docs-only and gap-ledger changes. If you find one open, retarget it to `dev` rather than merging it. +Anything *other* than the release merge showing up in step 3 means a PR was opened against `main` directly. That is the mistake to catch, not to clean up after: **every PR targets `dev`**, including docs-only and gap-ledger changes. If you find one open, retarget it to `dev` rather than merging it. The one exception is an OS patch release cut from `main` for a lock bump, below. This used to be a `sync-dev.yml` workflow that opened the `main` -> `dev` PR by itself. It was removed in favour of the step above: it needed an org-wide "Actions may create pull requests" permission that the org does not grant, so every run failed and the PR had to be opened by hand anyway (#484). +### OS package lock bumps + +The OS image installs every package at a locked version (`../specs/BUILD.md` # 1b # The OS package lock): a Debian snapshot timestamp, exact Docker pins and the resolved package list, all in `dev/os-lock/`. `.github/workflows/os-lock-bump.yml` moves them forward **daily**. When no package changed, it does nothing. When one did, it commits the three files on the `bot/os-lock` branch, with a title and body that name every changed package and version and mark the ones from `trixie-security`. + +**The bot branch keeps your work.** The bump never deletes `bot/os-lock` and never force-pushes it. When the branch exists, the bump merges `dev` into it, resolves the lock on the result, and adds a commit only when the lock files changed. So a fix you push to the branch (such as the `expected-packages.txt` change below) stays, and a PR you opened from it stays open. When `dev` does not merge cleanly into the branch, the run stops with a summary and changes nothing: merge `dev` in by hand and re-run. The branch is made fresh from `dev` only when it does not exist, for example after its PR merged and the branch was deleted. + +**Two jobs, so the token never meets the build.** `resolve` builds the image (as root, with a lot of third-party code) and holds no secret. `handover` builds nothing: it gets only the three lock files and the PR title and body from `resolve` as an artifact, commits them and pushes. + +What `handover` does next depends on one secret: + +- **`OS_LOCK_BOT_TOKEN` is set and works.** It pushes the branch with it and opens one PR into `dev`, or updates that PR in place when it is open. CI runs on it like on any human PR, and because it touches `dev/os-lock/`, `CI / Cloud image` runs the lock check and the full boot list. +- **It is not set, or it is set but rejected.** The org does not let Actions open PRs, and a push made with `GITHUB_TOKEN` starts no workflow. So the workflow pushes the branch with `GITHUB_TOKEN`, runs the boot proofs itself (it calls `ci-cloud-image.yml` on the branch, publishing nothing), and puts the diff and an "open a PR" link in the job summary and in one open issue titled "OS lock bump ready for dev". Open the PR from that link: GitHub fills in the title and body from the newest commit. A missing secret is not a failure. +- **It pushes, but cannot open or edit PRs.** The branch is already pushed with `OS_LOCK_BOT_TOKEN`, and the run does not push again. From there it does the same hand-over as above: the boot proofs, the summary and the tracking issue, which open with a note to give the token **Pull requests: read and write**. Because this push was made with the token and not with `GITHUB_TOKEN`, it does start workflows: if a bump PR is already open, its CI runs on the new commit as well. + +**The token.** A maintainer makes a **fine-grained personal access token** for `onmoose/os` only, with **Contents: read and write** and **Pull requests: read and write**, and stores it as the repo secret `OS_LOCK_BOT_TOKEN`. It is read in one place, the `BOT_TOKEN` env of the "Push and hand over" step in the `handover` job of `os-lock-bump.yml`. It belongs to a person and **expires** (at most one year), so put the renewal in a calendar. An expired or revoked token is still a set secret, so the workflow checks it with an API call before it pushes, and also falls back if the push with it is refused. Either way it takes the no-token path and says **"OS_LOCK_BOT_TOKEN is set but rejected ... Renew it"** at the top of the job summary and the tracking issue. A token that can push but cannot open or edit PRs (no **Pull requests: write**) pushes the branch, then the PR call fails: the run then takes the no-token hand-over without pushing again (boot proofs in the workflow, summary, tracking issue) and says **"OS_LOCK_BOT_TOKEN pushed the branch but cannot open or edit PRs. Give it Pull requests: read and write"**. A GitHub App token is the cleaner later replacement: mint it in a step before that one (`actions/create-github-app-token`) and pass it as `BOT_TOKEN`. Nothing else changes. + +**Workflow files.** When `dev` changed a file under `.github/workflows/` since the last bump, merging `dev` into the bot branch brings that change in, and GitHub refuses the push unless the token may write workflows. `GITHUB_TOKEN` never may. The run then stops and says so. Give the token **Workflows: read and write**, or merge `dev` into the branch by hand, and re-run. + +**A bump that adds or drops a package** fails the lean check on its PR, on purpose. Review the new set and update `dev/cloud/expected-packages.txt` on the bot branch. To bump by hand (for example to test a change), dispatch the workflow: `gh workflow run "OS lock bump" --ref dev`. + +**When the lock check fails on your own PR,** the build resolved to a different list than `dev/os-lock/cloud-packages.lock`. That only happens when you changed the package list in `dev/cloud/mkosi.conf`, the snapshot or a pin. Take the resolved list from the run's `cloud-packages-lock` artifact and commit it with your change. + +### OS patch releases from a lock bump + +Merging a lock bump puts the change on `dev`, not on any box. It ships with an OS release (`VERSION`). **A hosted box installs an OS release once its update target names it** (#563, `../specs/UPDATES.md` # 1). Until the control plane sends the OS part of that answer, and on the appliance until #564, an OS release reaches only new boxes; a running box keeps the OS it was built with. + +- **A bump that changes any package from `trixie-security` is released within 7 days.** The bump PR's body says so when it applies. +- **Any other bump ships with the next normal release.** + +`dev` may hold unreleased work that is not ready, so a lock-only release is cut **from `main`**, not from `dev`. This is the one PR into `main` that does not come from `dev`: + +```bash +# 1. A hotfix branch from what is released. +git fetch origin +git checkout -b hotfix/X.Y.Z origin/main # X.Y.Z: the next patch number +git push -u origin hotfix/X.Y.Z + +# 2. Re-run the bump against it, so the lock is resolved for main's package list +# (a cherry-picked lock from dev may name packages main does not install). +gh workflow run "OS lock bump" --ref dev -f base=hotfix/X.Y.Z +# It opens (or links) a PR from bot/os-lock-hotfix-X.Y.Z into hotfix/X.Y.Z. +# CI / Go and CI / Cloud image run on PRs into hotfix/** too, so that PR gets +# the lock check and the full boot list. Merge it once they are green. + +# 3. Check that the OS boots with the control plane the release will bake. +# An OS-only release bakes the LAST RELEASED control plane from ghcr (#566), +# but the PR CI below builds the brain and UI from the branch. This run bakes +# the released pair instead and boots the full list: +gh workflow run "CI / Cloud image" --ref hotfix/X.Y.Z -f publish=false -f control_plane=released +# If os-update or os-revert go red here, the released brain is too old for +# the boot checks: cut a control-plane release first (see below). This check +# reads CONTROL_PLANE_VERSION from the hotfix branch, so once that release +# has merged into main, run `git merge origin/main` on hotfix/X.Y.Z (it brings +# the new CONTROL_PLANE_VERSION; do not bump it here by hand) and re-run this. + +# 4. Bump VERSION on the hotfix branch and open the release PR into main. +git pull +echo "X.Y.Z" > VERSION +# commit, push, open a PR from hotfix/X.Y.Z into main. Its CI runs the full boot +# list (the PR touches dev/os-lock/). Merge it; release.yml tags vX.Y.Z and +# attaches the image. + +# 5. Carry main into dev, as after every release. +git checkout dev && git pull +git merge origin/main +git push +``` + +**The next OS patch release needs a control-plane release first.** The last released control plane, 0.15.0, is older than #563, so its brain cannot report an OS update: with it baked, the `os-update` and `os-revert` boots go red and the release stops at the boot proof. Release a control plane that carries #563 before, or bump `CONTROL_PLANE_VERSION` with the release (a hotfix branch from `main` only can if `main` already holds #563). Step 3 shows the result before anything is tagged. + +In step 5, resolve `VERSION` to the new `X.Y.Z`: that is now the last released version, and `dev` did not have it. Resolve `dev/os-lock/` to `dev`'s side when it is newer; the next daily bump moves `dev` forward anyway. Step 2 needs `dev/os-lock/` and the shared mkosi setup on `main`, so it works from the first release after #560. + ## Definition of done — checklist - [ ] Behavior works in the inner loop (`make dev`), and integration-tested against the real system if it touches one. diff --git a/docs/dev/hosted-boot-proof.md b/docs/dev/hosted-boot-proof.md index 32556a64..763ed5e3 100644 --- a/docs/dev/hosted-boot-proof.md +++ b/docs/dev/hosted-boot-proof.md @@ -18,7 +18,7 @@ Net: a provisioned box logs both milestones, binds `:443`, and serves every `_TRY=1` in the grubenv GRUB left for this boot (#575; the `layout: grubenv at boot:` line, logged by `moose-test-grubenv.service` before host-agent can mark the slot good), no `NvVars` file on the ESP, the reboot drop-ins on emergency and rescue, and a hardware watchdog that systemd feeds with a 1 min timeout (the `layout: systemd feeds the hardware watchdog` line; this lane is QEMU `q35`, so it has the ICH9 TCO watchdog a Hetzner VM has). It also prints the measured numbers for the disk budget (`BUILD.md` # 1b # Disk budget) in one `layout: measured:` line: the squashfs size read from slot A's superblock, the ESP's use, and the state partition's use. A red boot here is the image or the initramfs, not the control plane: look for `moose-state:` lines on the serial console (the initramfs setup logs to the kernel log), or a `Kernel panic`/reboot right after them. A box that never reaches the serial verdict and shows the UEFI shell (or `BdsDxe: failed to load`) did not find GRUB on the ESP: check the ESP's FAT (runs 36925538434 and 36927908164 had a FAT32 with too few clusters, fixed by `SectorSize=512` in `dev/cloud/mkosi.conf`). **The build checks the slot budget too**: the job summary has an "OS slot budget" table from `dev/cloud/slotbudget`, and the lean build fails when the squashfs in slot A is over 60% of the slot. **The build makes the OS update bundle too** (#562, `BUILD.md` # 1b # The bundle): the step "Build the OS update bundle" runs `dev/cloud/build-bundle.sh`, which prints `check 1` to `check 6` and an `ok:` line for each, and adds an "OS update bundle" size table to the job summary. A red `check 1` means the slot carries another keyring than the build staged (look at the `rauc keyring:` line of the image build). A red `check 3` on a release run means the image would trust the throwaway signer, which must never ship. "unsuitable certificate purpose" means a signer without the codeSigning purpose, or a `system.conf` without `check-purpose=codesign`. "no release root CA" or "no release signer" on a run that publishes the OS means the maintainer's setup in `rauc-signing.md` is not done yet. **Every boot checks which control plane it runs** (#566, `BUILD.md` # Versioning), in step 5a of `cloud-assertions.sh`: the running `moose-brain` and `moose-ui` containers must run the image IDs the image recorded in `/usr/lib/moose/control-plane.env`. The line is `control plane baked: `, and a released bake also prints `layout: control plane baked from ghcr:` with the two pinned refs. The build job's summary shows the same record ("Control plane baked into the image"), and the boot-proof build fails when its record differs from the image that ships. A run that publishes only the OS bakes `released`; to see that path on a branch, add `-f control_plane=released`. A red "cannot resolve ghcr.io/onmoose/brain:vX.Y.Z" in the image build means `CONTROL_PLANE_VERSION` names a control-plane release that is not on ghcr. **Every boot also ends with a gate** (`ok()`): no unit may have failed, and no host process may have hit `Read-only file system`, because the slot is read-only and a write the inventory missed must fail here rather than on a box. The gate also checks the `/etc` upper layer (`BUILD.md` # 1b, rules 3 and 4): every file in it must be on a keep list (`/usr/lib/moose/etc-keep.list`, or the lane's own `etc-keep.d/boot-lane.list` for its host-agent drop-ins), an account file, a pinned file or an sshd link, and every account in the image's `moose-image-accounts.conf` must be on the box with the same id. A red "holds files no tidy-up rule covers" means moose or a test wrote a new file into `/etc`: add it to the keep list, or give it a rule. **The `os-update` boot fakes a Debian major**: before the switch it records Debian 12 on the state partition and plants an admin's edit in `/etc`. On slot B it wants the `moose-state: tidied /etc for Debian 12 to 13` kernel log line, the major recorded as 13, the edit in the attic and gone from `/etc`, and the owner's password hash, host key, `machine-id` and groups unchanged (`MAJOR TIDY OK`). The owner turned SSH on before the switch, so slot B also wants the drop-in kept and `ssh.service` enabled again by host-agent (`SSHD BACK OK`). **The `os-revert` boot fakes the revert direction:** slot B records Debian 14 before the safety net reboots it, and slot A must tidy 14 to 13, with sshd, turned off by hand before the switch, left off (`MAJOR TIDY BACK OK`). **The build checks the account ids** (`check 6` of `build-bundle.sh`): the image's accounts must match `dev/os-lock/cloud-accounts.lock`. A red check prints the difference; add a new account's line by hand, and fix the build when an id moved. + **Every boot checks the userns-remap** (#530, `BUILD.md` # User-namespace remap) before its scenario runs. The serial log shows the two `docker info` lines (`security options: [... "name=userns" ...]` and `storage driver: overlay2, root dir: /var/lib/docker/1000000.1000000`) and then `userns-remap on (...)`. A red boot at this step means the image lost part of the remap setup, or the control plane no longer splits across it (proxy and brain in the host user namespace, Caddy and `moose-ui` remapped). The `access` boot also installs the `imageuser` fixture and checks that it runs remapped as its images' own users. The `remap` boots check the other tiers, one app each (the row above). Their serial lines start `cloud-assertions: remap [remap]:` and `cloud-assertions: remap [remap-reboot]:`, one per tier, and each names what it saw, so a red one says which tier and which field was wrong. **Air-gapped means config-apply, not a real cert.** The lane cannot reach acme-dns or Let's Encrypt, so the `seeded` scenario proves the brain *applies* the issuer + binds `:443`, not that a cert was obtained. Real DNS-01 issuance + a live `*.` cert are verified on a real provider box on a real network — deliberately outside this lane (the whole "config that looks right but never issues" class is invisible air-gapped; the `certificates.automate` unit test in `internal/caddy/caddy_test.go` is the closest static guard). @@ -60,6 +64,28 @@ Reading a red one: The diag block dumped on failure carries `images.json`, the compose image lines, the snapshot dir listing and host-agent's update log lines, so a red update boot should be diagnosable from the serial log alone. +## The OS update boots (`os-update`, `os-revert`) + +The two boots of #563. Each one moves its box between slots **inside one QEMU run**: they are the only boots that do not pass `-no-reboot`, and the in-guest script keeps its stage in `/var/lib/moose/test/os/stage` on the state partition. The serial log of one job therefore holds two boots (`os-update`) or three (`os-revert`). + +- **The test bundle** is made in the build job, only when one of these boots runs: `dev/cloud/test/build-os-test-bundle.sh` repacks the boot-proof slot with a host-agent one patch release above `VERSION`, with gzip level 1 (fast, and GRUB reads it), leaves out the baked tarballs under `/var/lib/moose` (a slot's copy is only read at a box's first boot), and signs it with the throwaway key. It is the artifact `cloud-image-os-test-bundle`. The harness puts it on an ext4 image (`mke2fs -d`, label `moose-os-test`) attached as a second read-only disk; the guest mounts it under the in-guest file server on `127.0.0.1:5001`, and the seed points the box's update target there. A host-agent drop-in sets `MOOSE_UPDATE_OS_URL_PREFIX=http://127.0.0.1:5001/`. +- **The answer carries only an OS part.** The control-plane part is then refused, which is how the lane proves the two streams are judged apart. Restarting host-agent forces a tick, as the `update` boot does. +- **On a run that publishes the OS** (`MOOSE_CLOUD_OS_REFUSE_ONLY=true`), the boot-proof image trusts only the release root, so `os-update` checks the wrong digest and then that RAUC refuses the throwaway-signed bundle with the right digest, and stops there. `os-revert` is left out of the matrix. + +Reading a red one: + +| Symptom | Most likely cause | Where to look | +|---------|-------------------|---------------| +| `no disk labelled moose-os-test` | The harness did not attach the bundle disk, or `MOOSE_CLOUD_OS_BUNDLE_DIR` was empty. | The boot job's "Download the OS test bundle" step. | +| `the bundle was not installed into slot B` | `rauc install` failed: signature, compatible, or the image does not fit the slot. | host-agent's `os update: install failed` line, which carries RAUC's own error. | +| `the boot order moved outside the window` | RAUC activated the slot on install: `activate-installed=false` missing from `system.conf`. | `/etc/rauc/system.conf` in the image. | +| `stage 2 booted slot 'A'` | GRUB did not boot the new slot after the switch, so the switch's grubenv was wrong. | host-agent's `after mark-active the grubenv is` line in the stage-1 boot. | +| `host-agent did not mark the new slot good` on `os-update` | The brain did not answer `/healthz` on slot B, or the safety net (90 s) fired first. | The serial log for `moose-os-trial: slot B was not marked good`; host-agent's `os update` lines. | +| `the safety net never rebooted the box` on `os-revert` | `moose-os-trial.timer` is not enabled, or its marker path changed. | `systemctl list-timers` in the stage-2 diag; `/usr/lib/moose/os-trial-check`. | +| `GRUB did not save _TRY=1` (any boot) or `GRUB did not save B_TRY=1` (`os-revert` stage 2) | GRUB's `save_env` did not reach the disk. Under UEFI, first suspect the firmware: OVMF with no VARS store writes `NvVars` to the ESP and loses GRUB's write (#575). | The `OVMF: code ..., vars from ...` line of the boot job; the `the firmware wrote /efi/NvVars` failure; `grub.cfg`'s `save_env` line. | +| `stage 4 booted slot 'B'` or `no panic note from slot B` on `os-revert` | The panic hook did not run: it is not in the initramfs, or the state partition was not mounted when it ran. | `lsinitramfs` of the boot-proof slot for `scripts/local-bottom/moose-test-panic`; the stage-4 serial log for `moose-test: slot B panics`. | +| `stage 4 booted slot 'B'` and no verdict for a long time | GRUB booted slot B again after it panicked: `B_TRY=1` was not saved. | The `layout: grubenv at boot:` lines; this is the bug #575 fixed. | + ## When it's red: where to look The serial log is the primary artifact: `.dev/cloud-boot/last-serial.log` locally, or the "Seeded-boot test" step log in CI. On failure `cloud-assertions.sh` dumps a `=== MOOSE_CLOUD_DIAG ===` block (docker ps, networks, iptables, and **`moose-brain` log tail**). Grep it for the milestone lines above. @@ -75,9 +101,15 @@ A **broken image build** presents as several of these at once (e.g. `:443` refus ## How to run it -- **It can also start on its own.** A PR into `dev` or `main` that touches the updater — `internal/hostagent/**`, `cmd/host-agent-real/**`, `internal/lifecycle/controlplane*.go` or `lifecycle.go`, `internal/api/system*.go`, `dev/cloud/**`, or the workflow itself — runs `CI / Cloud image` automatically and boots **only** the `update` scenario (#389). It publishes nothing (the run asserts that before it builds). So a PR in those paths can turn this lane red without anyone asking for it; read the run the same way as a manual one. Expect ~12–15 min, most of it the image build. Draft PRs are skipped — marking the PR ready starts the run. A PR that needs the other boots can still run them by hand with the command below. -- **CI (preferred: no local root/KVM, no image push):** `gh workflow run "CI / Cloud image" --ref -f publish=false`. Builds the image, then runs the `unseeded seeded bios access update ssh remap` boots under QEMU (`remap` is two boots). Add `-f boots="remap"` (any space-separated list of boot names) to run only those; the script refuses a name it does not know, so a typo fails instead of running nothing. `publish=true` (the default) additionally **attaches the compressed image + checksum to the tagged GitHub Release and pushes the brain + UI images to ghcr** (#352 removed the old provider-snapshot upload, so the lane holds no hosting-provider credential), so only do that deliberately. Runtime ~10 min for the build, plus the boots (the `update` boot is the longest: it runs two full update transactions, one of which spends a 60s health wait failing on purpose). -- **Local:** `sudo make test-cloud-qemu` (needs root + `/dev/kvm`). Scope boots with `MOOSE_CLOUD_BOOTS="seeded"` to reproduce the wildcard path alone, `"bios"` for the legacy-BIOS boot alone, `"update"` for the updater proof alone, `"remap"` for the two remap boots alone, or the default `"unseeded seeded frozen bios access update ssh remap"` for the full run. +- **It can also start on its own.** A PR into `dev` or `main` that touches the updater (`internal/hostagent/**`, `cmd/host-agent-real/**`, `internal/lifecycle/controlplane*.go` or `lifecycle.go`, `internal/api/system*.go`, `dev/cloud/**`, or the workflow itself) runs `CI / Cloud image` automatically and boots **only** the `update` scenario (#389), under both firmwares. It publishes nothing (the run asserts that before it builds). So a PR in those paths can turn this lane red without anyone asking for it; read the run the same way as a manual one. Expect about 11 min: about 8 for the build, then the two `update` boot jobs at once (a little more on a branch's first run, while the Caddy layer cache fills). Draft PRs are skipped — marking the PR ready starts the run. A PR that needs the other boots can still run them by hand with the command below. +- **CI (preferred: no local root/KVM, no image push):** `gh workflow run "CI / Cloud image" --ref -f publish=false`. Builds the image, then runs the `unseeded seeded access update ssh remap os-update os-revert` boots under QEMU (`remap` is two boots; the OS boots reboot their box inside one run), under UEFI and again under legacy BIOS. Add `-f boots="remap"` (any space-separated list of boot names) to run only those; the script refuses a name it does not know, so a typo fails instead of running nothing. `publish=true` (the default) additionally **attaches the compressed image, the signed OS update bundle and their checksums to the `v` GitHub Release (#562; it needs the release root CA and the `os-release` signer, `rauc-signing.md`) and pushes the brain + UI images to ghcr as `v`** (#352 removed the old provider-snapshot upload, so the lane holds no hosting-provider credential), so only do that deliberately. To publish one release line only, pass `-f publish=false` with `-f publish_os=true` or `-f publish_control_plane=true` (#559). `-f control_plane=released` makes a build-only run bake the last released brain and UI from ghcr, the way an OS-only release does, instead of a build of the branch (#566). Either way the publish steps only add what is missing: they never replace an asset already on the Release or an image tag already on ghcr. A Release with only one of the image and its checksum refuses until a person deletes the stray asset. A full run takes about 12 min. +- **How a run is laid out** (`../progress/ci-cloud-image-speedup.md`; `sign` from #562). Four jobs: + - `build` runs every assert, picks the boots (a typo in `boots` fails here, before the build), builds the production image, then the boot-proof image, and uploads the boot-proof image as a qcow2 artifact (`cloud-image-boot-qcow2`). On a run that publishes, it also uploads what the publish job needs. It also downloads the QEMU packages once, on a fresh runner, and hands them to the boot jobs (`cloud-image-qemu-debs`), so one slow Ubuntu mirror cannot hold up ten jobs. This job takes about 8 min, and the resolved package list (`cloud-packages-lock`) is uploaded here. + - `boot` is a matrix: one job per boot group and firmware, named like `Boot update (bios)`, all running at once. `unseeded seeded` is one group because those boots share one disk; `frozen` joins it when you ask for it. Every other boot is its own group. So the full list is 14 jobs, and `-f boots="remap"` is 2. The longest is `update`, about 2.6 min of boot. A red boot does not cancel the others, so one run shows every failing boot. **To debug a red boot, open that one job's log**: it holds the serial dump for that boot only. + - `publish` runs only when a line publishes, and only after `build` and every `boot` job passed. When the OS line publishes it first waits for `sign`, and attaches the image, the bundle `sign` re-signed and their checksums as one set (#562; `dev/release/attach-image.sh` detects and refuses a mixed set). For the control plane it loads the brain and UI tarballs the image baked and pushes those, so it pushes the images the boots ran, not a rebuild. + - `sign` runs only when the OS line publishes, after every `boot` job passed. It is the only job that enters the `os-release` environment, and it does nothing else: it checks the bundle's sha256 against the one the build job reported, re-signs it with the release signer (`dev/release/sign-bundle.sh`) and hands it to `publish`. A red `sign` with "no release signer" means the environment's secrets are missing; on a branch other than `main` or a `v*` tag, the environment refuses the job before any step runs. +- **Layer cache.** A run that publishes nothing builds the hosted Caddy image with the GitHub Actions layer cache (`MOOSE_BUILD_CACHE=gha`, set in the workflow from the same expression as the publish switches). Its inputs are all pinned, so the cache hits on every run after a branch's first. A run that publishes builds it fresh, with no cache read. The brain and UI always build plain: they rebuild on every commit, so a cache cost more to upload than it saved. +- **Local:** `sudo make test-cloud-qemu` (needs root + `/dev/kvm`). Scope boots with `MOOSE_CLOUD_BOOTS="seeded"` to reproduce the wildcard path alone, `"update"` for the updater proof alone, `"remap"` for the two remap boots alone, `"os-update"` or `"os-revert"` for the OS update boots (they need `MOOSE_CLOUD_OS_BUNDLE_DIR`, the output of `dev/cloud/test/build-os-test-bundle.sh`), or the default `"unseeded seeded frozen access update ssh remap os-update os-revert"` for the full run. `MOOSE_CLOUD_FIRMWARES="bios"` (or `"uefi"`) runs them under one firmware only; `bios` is no longer a boot name, and the script refuses it. `MOOSE_CLOUD_QCOW2=` boots a qcow2 that is already built and builds nothing (this is how each CI boot job runs). ## Related history (frozen snapshots — background, not the current view) diff --git a/docs/dev/rauc-signing.md b/docs/dev/rauc-signing.md new file mode 100644 index 00000000..046adc4c --- /dev/null +++ b/docs/dev/rauc-signing.md @@ -0,0 +1,92 @@ +# Signing OS update bundles + +Every OS release ships a RAUC bundle, `moose-vX.Y.Z-amd64.raucb`, next to the disk image (`../specs/BUILD.md` # 1b # The bundle). A box installs a bundle only if its signer chains to the root CA in the box's own `/etc/rauc/keyring.pem`. This page is the maintainer's how-to for the keys. The decision behind it is `../specs/DECISIONS.md` 2026-10-02 (key custody for the OS bundle). + +## The shape + +- **The root CA lives offline**, with the maintainer. Its private key never touches CI, a server or this repo. Its public cert is committed as `dev/release/rauc/release-ca.pem`, and every image built to publish bakes it as `/etc/rauc/keyring.pem`. +- **A signer, issued by the root, lives in CI.** Its cert and key are the secrets `RAUC_SIGNING_CERT` and `RAUC_SIGNING_KEY` of the GitHub Environment `os-release`. Only `main` and `v*` tags may use that environment, and only the `sign` job of `CI / Cloud image` enters it. That job does nothing else: it runs after every boot passed, checks the bundle, re-signs it and hands it to the publish job. The build job, which runs mkosi as root, never sees the key. +- **What the signature vouches for.** The build job builds the bundle, signs it with the throwaway key and reports its sha256. The `sign` job re-signs only a bundle with that digest. So the release signature says "this is what the build job of a release run produced", and no more: a build job that was made to produce a bad bundle would get it signed. The control for that is on the box: it installs only the bundle whose digest its update target names (`../specs/UPDATES.md` # 1, built by #563), and that digest comes from the cloud control plane (hosted) or the minisign-signed release manifest (appliance), not from this CI. +- **Runs that publish no OS image use a throwaway root.** It is made once per checkout under `.dev/rauc/throwaway/` and never stored anywhere else. Such an image trusts only bundles signed in that checkout, so a stray CI artifact trusts nothing real. +- **There are no CRLs.** A CRL expires, and a box that was offline past its expiry could then never update again. A leaked signer is handled by rotation (below). On top of the signature, a box only installs the bundle whose digest its update target names (`../specs/UPDATES.md` # 1), so a signer key on its own cannot push an update to a box. + +All keys are EC P-256. A signer carries the codeSigning purpose, and the image's `/etc/rauc/system.conf` asks for it (`check-purpose=codesign`). + +**What happens before the setup is done** (`dev/release/require-release-ca.sh` checks that the root is present, is a CA, has not expired and is not one of the throwaway or check CAs CI makes): + +- **A merge that bumps `VERSION` while `release-ca.pem` is missing tags nothing.** `release.yml` stops before it tags or creates any Release, on either line, even when the same merge bumps `CONTROL_PLANE_VERSION` too. The OS line is never dropped quietly. Commit the root first (# 4), or release only the control plane by bumping only `CONTROL_PLANE_VERSION`. +- **A merge that bumps only `CONTROL_PLANE_VERSION` needs neither the root nor the signer.** +- **The root committed, but a secret missing:** `release.yml` tags and creates the Releases, then the `sign` job fails with "no release signer" and the publish job does not run, so nothing is published on either line. Add the secrets (# 3) and re-run `release.yml`; it resumes. +- **A dispatch** with `publish_os` (or a hand-pushed `v*` tag) fails at its first step without the root, and in `sign` without the secrets. + +## 1. Make the root CA (once, offline) + +On a machine that is not CI, with this repo checked out and `openssl` installed: + +```bash +dev/release/rauc-ca.sh root ~/moose-rauc-ca +``` + +openssl asks for a passphrase twice. It encrypts `root-ca.key` with it (AES-256). The root is valid for 20 years. You get: + +- `~/moose-rauc-ca/root-ca.key`: **secret**. Keep it offline, with a backup (for example a second encrypted USB stick in another place), and the passphrase in your password manager. Losing it means the next root change needs two releases instead of one (# 5). +- `~/moose-rauc-ca/root-ca.pem`: public. + +## 2. Issue a signer (once now, then at each rotation) + +```bash +dev/release/rauc-ca.sh signer ~/moose-rauc-ca signer-2026 +``` + +openssl asks for the root's passphrase. The signer is valid for 3 years. Give each signer a new name. The script checks that it chains to the root and carries the codeSigning purpose. `dev/release/rauc-ca.sh show ~/moose-rauc-ca/signer-2026.pem` prints its dates. + +## 3. Create the environment and its two secrets + +In GitHub: **Settings > Environments > New environment**, named `os-release`. + +- **Deployment branches and tags:** "Selected branches and tags", with the branch rule `main` and the tag rule `v*`. No required reviewer: the merged release PR is the approval. +- **Environment secrets:** `RAUC_SIGNING_CERT` holds the whole text of `signer-2026.pem`, and `RAUC_SIGNING_KEY` the whole text of `signer-2026.key`, both with their `-----BEGIN` and `-----END` lines. + +Or with the CLI: + +```bash +gh api -X PUT repos/onmoose/os/environments/os-release \ + -F deployment_branch_policy[protected_branches]=false \ + -F deployment_branch_policy[custom_branch_policies]=true +gh api -X POST repos/onmoose/os/environments/os-release/deployment-branch-policies -f name=main -f type=branch +gh api -X POST repos/onmoose/os/environments/os-release/deployment-branch-policies -f name='v*' -f type=tag +gh secret set RAUC_SIGNING_CERT --env os-release < ~/moose-rauc-ca/signer-2026.pem +gh secret set RAUC_SIGNING_KEY --env os-release < ~/moose-rauc-ca/signer-2026.key +``` + +Then delete the local copy of the signer's key: `shred -u ~/moose-rauc-ca/signer-2026.key`. A new signer is cheap to issue; a key that lingers on a laptop is not. + +Create the environment **before** the first OS release. GitHub creates an environment that a job names but that does not exist yet, with no branch rule, the first time such a job runs. + +## 4. Commit the root's public cert + +```bash +mkdir -p dev/release/rauc +cp ~/moose-rauc-ca/root-ca.pem dev/release/rauc/release-ca.pem +``` + +Commit it through a normal PR into `dev`. From the next OS release on, the image bakes it and the bundle is signed by the release signer. + +## 5. Rotate or replace + +- **Rotate the signer** (every year or two, and at once if the key may have leaked): issue a new one (# 2) and replace both secrets (# 3). Boxes change nothing, because they trust the root. The old signer's cert stays valid until it expires, so if it leaked, the update-target digest is what keeps it from reaching a box; ship the next release soon so the fleet moves on. +- **Replace the root** (it leaked, or it nears its 20-year end). Two OS releases: + 1. Make the new root (# 1, a new directory) and put **both** roots into `dev/release/rauc/release-ca.pem` (old first, then new; RAUC trusts every CA in the file). Release it. This release is still signed by the signer in CI, which the old root issued: it needs no use of either root's key. Wait until the fleet runs this release. + 2. Issue a signer from the new root, replace the secrets, drop the old root from the file, and release again. + + **If the old root's key is lost,** this still works, as long as the signer in CI is still valid: step 1 uses only that signer. What a lost root key blocks is issuing a new signer from it. So if the old root's key is lost and the current signer has also expired or leaked, no bundle the fleet trusts can be signed any more, and every box needs a new image (re-provisioning on hosted). Keep the backup, and start a root change well before the signer expires. +- **A box that missed a release** still trusts what its own slot carries. So a root change must reach the whole fleet before the old root leaves the file. + +## Where it is in the code + +- `dev/release/rauc-ca.sh`: the root and signer commands above. CI uses it for the throwaway root. +- `dev/cloud/rauc.sh`: which keyring an image bakes (`MOOSE_RAUC_KEYRING`, set by `ci-cloud-image.yml`), and the pinned RAUC container. +- `dev/cloud/build-bundle.sh`: the build job. It makes the bundle with the throwaway signer and checks it against the keyring read back out of the slot. +- `dev/release/require-release-ca.sh`: the check `release.yml` runs before it tags an OS release, and every OS publish run first. +- `dev/release/sign-bundle.sh`: the `sign` job. It refuses without the secrets, without the committed root, or when the image would trust the throwaway signer, then re-signs and checks the result. +- `dev/release/attach-image.sh`: the publish job attaches the image, the bundle and their checksums. It detects and refuses a mixed set (some of the four already on the Release) and never overwrites one. diff --git a/docs/progress/README.md b/docs/progress/README.md index 5c38ae5e..881ed6b7 100644 --- a/docs/progress/README.md +++ b/docs/progress/README.md @@ -20,11 +20,12 @@ The implementation slice queue, ordered. Each item links back to the progress en This is the **maintainer's critical-path** queue. Work carved off for **parallel contributors** lives in [GitHub Issues](https://github.com/onmoose/os/issues) (some items there are pulled from these "what's next" follow-ups). The two are kept from overlapping on purpose. See [`../dev/contributing.md`](../dev/contributing.md) for the contributor loop. -1. **Persistent journal.** [container-logs-journald-driver.md](container-logs-journald-driver.md) made container logs reach journald, but neither image creates `/var/log/journal` or sets `Storage=persistent`, so the journal is volatile and app-log scrollback is lost on every reboot — `LOGGING.md` # Per-app logs promises "scrollback up to journald's cap". Ship the journald drop-in from `LOGGING.md` # Tuning (`Storage=persistent`, `SystemMaxUse=1G`, `RuntimeMaxUse=128M`) on both images, sized against the root partition that grows at first boot. Also restores the size-based backpressure # Tuning assumes now that docker's per-unit rate limit is off. -2. **GPU + device capacity enforcement.** `install-permissions-enforcement.md` deferred `gpu` enforcement and device-existence validation (both need a host hardware-introspection endpoint). A 422 from the brain will surface correctly in the UI via the existing `dialogError` path ([install-consent-ui.md](install-consent-ui.md)) once the host endpoint lands. See `NEXT.md` # GPU. -3. **Tag the store manifests with `role` and `requires`.** [ai-bindings-after-install.md](ai-bindings-after-install.md) finished the `os` side of the install setup plan, but no real app shows the LLM provider pickers until its manifest in `onmoose/store` carries roles (`APP_MANIFEST.md` # D4 # Roles and requires, `docs/dev/authoring-apps-with-an-agent.md`). The last step of `INSTALL_SETUP.md`. -4. **Try the install setup work on a running box.** [ai-bindings-after-install.md](ai-bindings-after-install.md) and [install-setup-doc-gaps.md](install-setup-doc-gaps.md): the settings screen, the LLM providers screen, a key change and a user delete against a running brain with a real app, in a browser. -5. **Finish the install steps.** [install-steps.md](install-steps.md) built steps 1 to 3 of `INSTALL_STEPS.md`. Left: a real send with the Gmail and iCloud presets, a test of the built flow with 3 to 5 non-technical people, then step 4 (`recommends`) and tagging the store apps. +1. **A/B OS update (#486).** [ab-os-update-design.md](ab-os-update-design.md) designed it; [control-plane-version-line.md](control-plane-version-line.md) (#559) and [os-package-lock.md](os-package-lock.md) (#560) built the version split and the package lock. [hosted-ab-layout.md](hosted-ab-layout.md) (#561) built the hosted image in the A/B layout with the state inventory, and [rauc-bundle.md](rauc-bundle.md) (#562) a signed RAUC bundle per OS release. Before the next OS release the maintainer sets up the offline root CA and the `os-release` signer (`../dev/rauc-signing.md`); until then `release.yml` tags nothing for a `VERSION` bump. [host-agent-os-update.md](host-agent-os-update.md) (#563) made host-agent apply OS updates on hosted: install into the other slot, switch in the window, trial boot, revert. [uefi-grub-try.md](uefi-grub-try.md) (#575) fixed the boot lane's OVMF so GRUB's try flag holds under UEFI, and proves a slot that panics before userspace is skipped. [bake-released-control-plane.md](bake-released-control-plane.md) (#566) made an OS-only release bake the last released control plane from ghcr; the released 0.15.0 control plane is older than #563, so the next OS release must also bump `CONTROL_PLANE_VERSION` (or follow a control-plane release). **Next:** the maintainer adds the `os` part to the control plane's update-target answer (the exact wire is in that entry); until then no production box moves its OS. Then #486's proof on a provisioned box, and #564 (appliance). +2. **Persistent journal.** [container-logs-journald-driver.md](container-logs-journald-driver.md) made container logs reach journald, but neither image creates `/var/log/journal` or sets `Storage=persistent`, so the journal is volatile and app-log scrollback is lost on every reboot, while `LOGGING.md` # Per-app logs promises "scrollback up to journald's cap". Ship the journald drop-in from `LOGGING.md` # Tuning (`Storage=persistent`, `SystemMaxUse=1G`, `RuntimeMaxUse=128M`) on both images, sized against the partition `/var/log` lives on (on hosted, the state partition that grows at first boot, since #561). Also restores the size-based backpressure # Tuning assumes now that docker's per-unit rate limit is off. +3. **GPU + device capacity enforcement.** `install-permissions-enforcement.md` deferred `gpu` enforcement and device-existence validation (both need a host hardware-introspection endpoint). A 422 from the brain will surface correctly in the UI via the existing `dialogError` path ([install-consent-ui.md](install-consent-ui.md)) once the host endpoint lands. See `NEXT.md` # GPU. +4. **Tag the store manifests with `role` and `requires`.** [ai-bindings-after-install.md](ai-bindings-after-install.md) finished the `os` side of the install setup plan, but no real app shows the LLM provider pickers until its manifest in `onmoose/store` carries roles (`APP_MANIFEST.md` # D4 # Roles and requires, `docs/dev/authoring-apps-with-an-agent.md`). The last step of `INSTALL_SETUP.md`. +5. **Try the install setup work on a running box.** [ai-bindings-after-install.md](ai-bindings-after-install.md) and [install-setup-doc-gaps.md](install-setup-doc-gaps.md): the settings screen, the LLM providers screen, a key change and a user delete against a running brain with a real app, in a browser. +6. **Finish the install steps.** [install-steps.md](install-steps.md) built steps 1 to 3 of `INSTALL_STEPS.md`. Left: a real send with the Gmail and iCloud presets, a test of the built flow with 3 to 5 non-technical people, then step 4 (`recommends`) and tagging the store apps. ## Entry template @@ -303,3 +304,16 @@ Oldest first; append new entries to the bottom. | [fake-agent-remap-base.md](fake-agent-remap-base.md) — Closes #548, after [userns-remap-capability.md](userns-remap-capability.md). **The fake host-agent now reports `remap_base`** from the `moose-remap` lines of `/etc/subuid` and `/etc/subgid`, read by the real agent's code (`usermgr.ReadRemapBase`, which `LinuxUserManager.RemapBase` now calls). No line: no field, as before, so `make dev` still refuses these apps (and #544 marks them not installable). A hand-made range: reported, so the dev brain can boot `root_setup` and `image_user` apps there. A bad line: an error, like the real agent. `running-locally.md` # Booting apps that need the remap says how, and the authoring guide points there. **Gaps:** not run on a remapped machine; the unprivileged dev brain cannot give bind dirs or managed-service data to the range, so those apps need the brain as root there | done | | [fixture-remap-fields.md](fixture-remap-fields.md) — Closes the fixture gap of [hide-remap-apps.md](hide-remap-apps.md). `internal/catalog/testdata/snapshot.json` is refreshed from the catalog publisher's own output, which now sends `root_setup` and `image_user` on each browse record; only the `version` token and the two keys on `alpha-notes` changed. `TestNoUnmodeledFields` now covers both, and `TestParseFixtureSnapshot` checks they reach the parsed record. **Gaps:** the live catalog sends them only after the store release | done | | [claude-md-skip-small.md](claude-md-skip-small.md) — Closes #553. The "Skip small issues" rule in `CLAUDE.md` # Working style now covers proposals and questions, not only issues. Only a big improvement, a breaking change for users, a real bug or a security risk is raised; everything else is dropped, with no "one line at most" allowance. A small open question is never repeated across replies, and an item the user says to ignore is dropped for good. It still lets through a design decision the task needs and the findings of a requested review. Docs only | done | +| [ab-update-engine-spike.md](ab-update-engine-spike.md) — Closes #485. A spike, with throwaway code on the never-merged branch `test/485-ab-update-spike`. **RAUC and systemd-sysupdate** each built a small trixie box (mkosi v26) with two 3G root slots and a data partition that first boot makes and grows. In QEMU each box updated to v2, then to a broken v3 that fails its health gate, and came back to v2 on its own with no console. **All four proofs pass for RAUC under UEFI and legacy BIOS (one GRUB config for both), and for sysupdate under UEFI.** sysupdate fails under BIOS: no native rollback, because systemd-boot is UEFI-only. Per-box state (the userns-remap files, password hash, SSH host key, `machine-id`) survives the swap through an `/etc` overlay on the data partition and a short pinned list, so the #486 rule holds. New system users must come from `sysusers.d`. Found: Debian puts RAUC's daemon in `rauc-service`, and trixie's `systemd-import` writes a `.zst` through without unpacking it and reports success. UEFI-only hosting would mean CPX22 at €19.49 instead of CX23 at €5.49. **Recommendation: RAUC with GRUB**, whole-root slots, verity slots and a LUKS data partition on the appliance. **Gaps:** QEMU only; a slot that hangs instead of rebooting never reverts on either engine; Secure Boot, verity and signing not tried. Docs only | done | +| [ab-os-update-design.md](ab-os-update-design.md) — First slice of #486, after [ab-update-engine-spike.md](ab-update-engine-spike.md). No code. **Stream A is an A/B OS image on both profiles, built with RAUC and GRUB**; `apt` and `unattended-upgrades` leave the update path, and `host-agent` ships inside the image. Two maintainer calls: **appliance OS updates apply and reboot in the window with no prompt**, and **a moose release is the OS release while the control plane gets its own version** (`CONTROL_PLANE_VERSION`, `control-plane-vX.Y.Z`), partly reversing 2026-07-16. `UPDATES.md` # 1 and # 2 are rewritten as the update transaction; new `BUILD.md` # 1b holds the layout (two 4 GiB slots, a state partition with the `/etc` overlay, four pinned files now including `login.defs`), the signed bundle and the **OS package lock** (a Debian snapshot timestamp, Docker pinned beside it, bot bump PRs). `NEXT.md` gains six open points to prove before shipping. #486 is split into #559 to #564. **Gaps:** nothing built; the version split is documented, not implemented (#559) | done | +| [control-plane-version-line.md](control-plane-version-line.md) — Closes #559, a slice of #486. **Two version lines are built.** `VERSION` is the moose (OS) release and stamps host-agent; a new `CONTROL_PLANE_VERSION` (starting at `0.15.0`) stamps the brain and labels both control-plane images. `moose-brain --version` now prints `moose control plane X.Y.Z (g)`, host-agent keeps `moose X.Y.Z (g)`. `release.yml` decides each line on its own through `dev/release/decide.sh` (tested against throwaway git repos in `make check`): a `VERSION` bump tags `vX.Y.Z` and attaches the disk image, a `CONTROL_PLANE_VERSION` bump tags `control-plane-vX.Y.Z` (Release with `--latest=false`) and pushes ghcr images tagged `vX.Y.Z` with the control-plane number, because the private control plane resolves digests by that tag. A control-plane-only release still runs the full boot proof and skips only the attach. A control-plane version whose image tag is already published is refused rather than overwritten, so `control-plane-v0.15.0` must be tagged once by hand at `v0.15.0`. **Gaps:** the disk image still bakes the same commit's control plane (#566); About shows the control-plane number as moose until #509 | done | +| [os-package-lock.md](os-package-lock.md) — Closes #560, a slice of #486. **Every package in the hosted image is locked.** `dev/os-lock/` holds a snapshot.debian.org timestamp (mkosi v26 `--snapshot`, `debian-security` included, `trixie-updates` added back from the same snapshot), exact pins for the four Docker packages (apt `Pin-Priority: 1001`), and the resolved list of 163 packages, made from mkosi's manifest by the new `dev/os-lock/oslock` tool. Both cloud builds (the shipped image and the boot-proof image) fail when they resolve to a different list, so CI proves two builds of one commit install the same versions. `os-lock-bump.yml` moves the lock forward daily and names every changed package; with the `OS_LOCK_BOT_TOKEN` secret it opens one PR into `dev`, without it it boots the branch itself and leaves an "open a PR" link in the summary and a tracking issue. A PR touching `dev/os-lock/` runs the full boot list. **Decided:** a lock-only OS patch release is cut from `main` as `hotfix/X.Y.Z`, within 7 days when a `trixie-security` package changed. **Gaps:** the token path is checked only by the first bump after merge; the appliance lane and the mkosi tools tree are not locked (#564) | done | +| [docs-after-version-split.md](docs-after-version-split.md) — Docs-only sweep after [os-package-lock.md](os-package-lock.md). Fixes three statements #559 and #560 made wrong: `CLAUDE.md` no longer lists apt as unwired (it names the A/B OS update) and describes the per-line publish inputs; `BRAIN_HOST_PROTOCOL.md` # Versioning is a `minimumAgentVersion` floor, not "brain N talks to host-agent N", and apt leaves its scope list; # Up next gains the A/B OS update with #561 next. Docs only | done | +| [hosted-ab-layout.md](hosted-ab-layout.md) — Closes #561, a slice of #486, after [docs-after-version-split.md](docs-after-version-split.md). **The hosted image is built in the A/B layout:** a 128 MiB ESP, a BIOS boot partition and slot A, a 1 GiB read-only squashfs-xz holding the kernel and initramfs, in the image; slot B (1 GiB) and a state partition (ext4, the rest of the disk, grown every boot) made by `systemd-repart` in the initramfs at first boot. The OS reserves 2.28 GB, 5.7% of a 40 GB disk. The maintainer rejected the designed 4 GiB ext4 slots (`DECISIONS.md` 2026-10-01). GRUB boots both firmwares from one `grub.cfg` and `grubenv`; the ESP holds no kernel (`Bootable=auto`) and is built with 512-byte sectors so 128 MiB can be FAT32. An initramfs-tools hook sets up the `/etc` overlay, four pinned files and the bind mounts; databases keep their own bind mounts and `/home` comes from `srv/moose/home`. `rauc` with the slot config, no keyring. **`dev/cloud/slotbudget`** fails the build when the squashfs fills over 60% of its slot (now 435.3 MB, 40.5%). host-agent's hosted build reports the state partition as "System". Every boot runs under UEFI and legacy BIOS and checks the layout; run 36932849491 passed all 14. **Gaps:** no bundle, install or mark-good (#562, #563); QEMU only; the control-plane tarballs are copied to the state partition (383 MB); the private smoke test needs a disk of 20 GB or more | done | +| [ci-cloud-image-speedup.md](ci-cloud-image-speedup.md) — A slice of #486, after [hosted-ab-layout.md](hosted-ab-layout.md). **`CI / Cloud image` builds once and boots in parallel: a full run drops from 27.4 to 12.2 min** (run 37001109373). One `build` job makes both images and uploads the boot-proof image as a qcow2; a `boot` matrix runs one job per boot group and firmware (10 for the full list, `unseeded seeded` kept together because they share a disk); a `publish` job, the only one with write access, runs only after every boot passed and pushes the control-plane tarballs the image baked. Every publish guard, the PR `update`-only rule, the lock-bump full-list rule and the `boots` input are kept. The boot-proof build reuses the production tools tree (`MOOSE_MKOSI_TOOLS_TREE`, about 40 s), the QEMU packages are downloaded once and installed with `dpkg` (one slow mirror once cost a job 6 min), and runs that publish nothing build the pinned hosted Caddy image with a BuildKit `gha` layer cache (`MOOSE_BUILD_CACHE`; the brain and UI build plain, because their cache cost more than it saved). Runner minutes go from 27 to about 34, free on a public repo. **Gaps:** every new branch starts with an empty Caddy cache (about 1.2 min on its first run); no apt cache (measured at about 15 s, dropped by decision); the boot-proof image is still a second mkosi build; the publish job has not run yet | done | +| [rauc-bundle.md](rauc-bundle.md) — Closes #562, a slice of #486, after [hosted-ab-layout.md](hosted-ab-layout.md) and [ci-cloud-image-speedup.md](ci-cloud-image-speedup.md). **Every OS release builds a signed RAUC bundle** (`moose-vX.Y.Z-amd64.raucb`, verity format): slot A of the image that ships, cut out by `dev/cloud/slotbudget -extract`, bundled by `dev/cloud/build-bundle.sh` with the box's own RAUC version from the snapshot, and checked against the `system.conf` and keyring read back out of the slot (the staged keyring, the throwaway root, a release keyring that must refuse the throwaway signer, a wrong key, and a rehearsal of the publish job's signing). The image bakes `/etc/rauc/keyring.pem` with `check-purpose=codesign`: the release root on an OS publish run, a throwaway root otherwise. **Key custody decided** (`DECISIONS.md` 2026-10-02): an offline root CA (`dev/release/rauc-ca.sh`, `docs/dev/rauc-signing.md`) and a signer in the `os-release` environment, so only a `sign` job of its own reads the signer, re-signs with `rauc resign` a bundle whose sha256 matches the build job's, and hands it to the publish job, which attaches the image, the bundle and their checksums as one set of four (it detects and refuses a mixed set). `release.yml` tags nothing for a `VERSION` bump until `release-ca.pem` is committed. **Numbers:** bundle 438.5 MB, 40.8% of the slot and half of an 874 MB OS release; about 1 min more per run. **Gaps:** no release path has run yet (no `release-ca.pem` committed; `release.yml` tags no OS release until the maintainer's setup); no `rauc install` proof (#563) | done | +| [host-agent-os-update.md](host-agent-os-update.md) — Closes #563, a slice of #486, after [hosted-ab-layout.md](hosted-ab-layout.md) and [rauc-bundle.md](rauc-bundle.md). **host-agent applies OS updates on hosted.** The update-target answer gains an optional `os` list; host-agent picks the next release (never skipping a minor, at most one minor back, never below the control plane's floor), downloads the bundle, checks its sha256 against the answer before RAUC sees it, installs it into the other slot ahead of the window (`activate-installed=false`), switches inside it after stream B (checking the grubenv reads `OK=1 TRY=0`), and on the next boot marks the slot good once the brain answers or reboots back; an image timer reboots a slot whose host-agent never started. Only the first boot after a switch is on trial. Stream A is reported on the version and update-target reads, and admins get one notification per outcome. New `os-update` and `os-revert` boots prove both paths under UEFI and BIOS, in every full run. Gaps: the private control plane does not send the `os` part yet (wire described for the maintainer), QEMU only, appliance waits for #564 | done | +| [uefi-grub-try.md](uefi-grub-try.md) — Closes #575, a slice of #486, after [host-agent-os-update.md](host-agent-os-update.md). **GRUB's try flag now survives under UEFI in the boot lane.** The cause was the harness, not GRUB or the image: it ran OVMF with no writable VARS store (Ubuntu 24.04 renamed the files), so OVMF saved its variables to an `NvVars` file on the ESP at every boot and lost GRUB's `save_env` write (traced in runs 37061482623, 37063254219 and 37064755329: GRUB wrote and read back `A_TRY=1`, the initramfs read `A_TRY=0`). The harness now needs a CODE and VARS pair. Every boot checks GRUB saved `_TRY=1` and the ESP has no `NvVars`, under both firmwares, and `os-revert` gains a last stage: slot B made active again crashes its kernel in the initramfs (a test-only hook), and GRUB must skip it. The trial timer stays on the marker, not the grubenv. No image change, no disk cost. **Gaps:** QEMU only, a real UEFI provider box not checked for `NvVars`; the appliance medium lane has the same OVMF fallback (#564) | done | +| [bake-released-control-plane.md](bake-released-control-plane.md) — Part of #566 (kept open until the released path is green), a slice of #486, after [control-plane-version-line.md](control-plane-version-line.md). **An OS-only release bakes the last released control plane:** the brain and UI of `v`, resolved to digests once and pulled from ghcr by digest (`make control-plane-released`, `dev/release/ghcr-resolve.sh`). A control-plane release, or one that bumps both files, bakes the pair it builds and releases; a PR, a lock bump or a dispatch bakes a build of its commit (`-f control_plane=released` checks the release path ahead). The image records the pair in `/usr/lib/moose/control-plane.env`, and every boot checks the running brain and UI against it. The released run (37345392679) baked 0.15.0 and passed the six original gate boots under both firmwares; `os-update` and `os-revert` fail with it because the 0.15.0 brain predates #563, so the next OS release must also release the control plane. | +| [ab-hang-and-major.md](ab-hang-and-major.md) — A slice of #486, after [hosted-ab-layout.md](hosted-ab-layout.md) and [host-agent-os-update.md](host-agent-os-update.md). Settles points 3 and 5 of `NEXT.md` # A/B OS image. **A slot that hangs:** real Hetzner cx23 VMs (Intel and AMD hosts) are QEMU `q35` with the ICH9 TCO watchdog; the image's generic kernel drives it (`iTCO_wdt`), and with PID 1 frozen the VM reset after 118 s. No image change; every boot checks systemd feeds it. Gaps: the reset comes after about twice the 60 s timeout, and a hang in GRUB or the initramfs is not covered. **A Debian major:** when the slot's major differs from the one the state partition records, in either direction, the initramfs tidies the `/etc` upper layer: the keep list (`etc-keep.list`) stays, the account files are merged, the pinned files are taken again from the slot when the remap stays, the rest goes to an attic; a failure never stops a boot. host-agent turns sshd back on once after a tidy-up when the drop-in names an account. The image's accounts are a generated `sysusers.d` file, with ids checked against `dev/os-lock/cloud-accounts.lock`. The `os-update` boot fakes a major, and every boot fails on an upper-layer file no rule covers. It must ship in a Debian 13 release before the first 14 release | done | +| [os-switch-undo-marker.md](os-switch-undo-marker.md) — A fix to [host-agent-os-update.md](host-agent-os-update.md) (#563), part of #486, found by Greptile on the 0.16.0 release PR (#583). **A failed switch undo keeps the trial marker:** `doSwitch` removed the new slot's trial marker even when it could not put the booted slot first again, so the new slot could boot next with no trial. The marker now goes only after the booted slot is first again; `Boot` already removes a marker for a switch that never took effect. Unit test only | done | diff --git a/docs/progress/ab-hang-and-major.md b/docs/progress/ab-hang-and-major.md new file mode 100644 index 00000000..28c919e8 --- /dev/null +++ b/docs/progress/ab-hang-and-major.md @@ -0,0 +1,97 @@ +# A slot that hangs, and a Debian major across the /etc overlay + +- **Status:** done +- **Date:** 2026-10-05 +- **Specs touched:** `docs/specs/BUILD.md`, `docs/specs/UPDATES.md`, `docs/specs/NEXT.md`, `docs/specs/DECISIONS.md`, `docs/architecture.md`, `docs/dev/hosted-boot-proof.md` + +A slice of #486. It settles points 3 and 5 of `NEXT.md` # A/B OS image, which [ab-os-update-design.md](ab-os-update-design.md) opened and [hosted-ab-layout.md](hosted-ab-layout.md) narrowed: whether a Hetzner VM has a watchdog, so a hung slot still reboots and reverts, and what happens to the `/etc` upper layer when a box moves to another Debian major. + +## What was done + +### Point 3: Hetzner Cloud has a hardware watchdog + +Checked on real Hetzner Cloud VMs, not in QEMU. Three `cx23` servers (the smallest box), stock Debian 13, two locations on AMD EPYC-Rome hosts and one on an Intel Skylake host, all booting legacy BIOS. Each lived a few minutes and was deleted with its SSH key; the API then listed no server, key, primary IP or snapshot made by the probe. The cost was about €0.035 (three started hours of a cx23). + +- **The device.** The VM is QEMU's `q35` machine. Its ICH9 chipset (`8086:2918`, the LPC bridge at `00:1f.0`) has a TCO watchdog timer. There is no `i6300esb` and no ACPI `WDAT` table. The same chipset is there on the Intel and the AMD hosts. +- **Stock Debian shows nothing.** Hetzner's Debian image runs the `-cloud` kernel, which has no `iTCO_wdt` module: no `/dev/watchdog`, `wdctl` finds no device, `WatchdogDevice=` is empty. +- **The moose kernel drives it.** With `linux-image-amd64`, the kernel moose ships, `lpc_ich` and `iTCO_wdt` load on their own: `Found a ICH9 TCO device (Version=2, TCOBASE=0x0660)`, `initialized. heartbeat=30 sec (nowayout=0)`, `/dev/watchdog0` with identity `iTCO_wdt`. With `RuntimeWatchdogSec=60s`, as the image sets, systemd logs `Using hardware watchdog 'iTCO_wdt', version 2, device /dev/watchdog0` and `Watchdog running with a hardware timeout of 1min.` The box stayed up 150 s with systemd feeding it. +- **It fires.** Opened once and never fed (`echo 1 > /dev/watchdog`, 30 s timeout, `watchdog did not stop!`): the Intel VM reset after about 50 to 65 s, the AMD VM after 59.5 s. The journal of the boot that reset ends with no shutdown. +- **A real hang is covered.** With systemd feeding it at 60 s, PID 1 was frozen with `ptrace` (`PTRACE_SEIZE` and `PTRACE_INTERRUPT`, state `t (tracing stop)`, no sysrq). The VM reset 118.3 s after the freeze, with no shutdown in the journal. +- **The boot lane has it too.** It is `q35` as well. A probe run (37358065628, `unseeded`, both firmwares) logged the same `iTCO_wdt` lines, so `BUILD.md`'s "none in the QEMU lane" was wrong. + +What changed: + +- **No image change.** The kernel already has the driver and `10-moose-watchdog.conf` already sets 60 s. Only its comment changed. +- **Every boot checks it** (`cloud-assertions.sh`, layout): `WatchdogDevice` set, `/sys/class/watchdog/watchdog0` there, `RuntimeWatchdogUSec=1min`, and systemd's `Using hardware watchdog` line in the boot's log. systemd logs that line before journald runs, so it is in the kernel log, not under `_PID=1`. +- **No hang proof in CI** (the maintainer's call). The Hetzner probe is the proof; the boot lane only checks the device is fed. +- **Two gaps, in `BUILD.md` # 1b # As built.** The reset comes after about twice the timeout (the TCO timer fires on its second expiry), so about 2 minutes on a box. A hang in GRUB or the initramfs is not covered, only a crash through `panic=10`; the watchdog is not started earlier because a long `e2fsck` or `systemd-growfs` of a big state partition would reset the box again and again. +- **The appliance fallback** is recorded for #564: `softdog` only where a machine has no hardware watchdog. It covers a hung PID 1, not a kernel locked up with interrupts off, and nothing before systemd. + +### Point 5: a Debian major tidies the /etc upper layer + +The maintainer chose option A (`DECISIONS.md` 2026-10-05). + +- **`state-setup` step 3a** (`dev/cloud/mkosi.extra/usr/lib/moose/state-setup`). It reads the slot's Debian major from `/usr/lib/os-release` and the box's from `/state/etc/.moose-debian-major`. When they differ, before `/etc` is mounted, it builds a new upper layer beside the old one: + - the files on `/usr/lib/moose/etc-keep.list` (and any `etc-keep.d/*.list`), copied with `cp -a --parents`, whiteouts included; + - `passwd`, `group`, `shadow`, `gshadow` merged by `awk`: the slot's lines in order, then the box's lines the slot lacks. In `shadow` and `gshadow` the box's password wins; in `group` and `gshadow` the members are joined. An id that differs between box and slot is logged; + - `daemon.json` from the slot when its `userns-remap` is the box's, `login.defs` from the slot when it still sets `SUB_UID_COUNT 0` and `SUB_GID_COUNT 0`; the box's copy otherwise, with a log line. + Then the old upper layer is renamed into `/state/etc/attic/