diff --git a/README.md b/README.md index c2a95fa..cadcfb0 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ An ordinary machine is trusted as you: it can read and write your home, includin ## Is it for you? -nsl needs an x86-64 Linux host with KVM, systemd-vmspawn, QEMU with UEFI firmware, virtiofsd and membership in the `kvm` group; Waypipe adds desktop windows. `nsl doctor` checks each requirement. [Install →](https://frostyard.github.io/nsl/getting-started/install/) +nsl needs an x86-64 Linux host with KVM, systemd-vmspawn from systemd 259 or newer (as in Fedora 44 and Ubuntu 26.04 LTS), QEMU with UEFI firmware, virtiofsd and membership in the `kvm` group; Waypipe adds desktop windows. `nsl doctor` checks each requirement. [Install →](https://frostyard.github.io/nsl/getting-started/install/) It is **pre-release**. v0.4.0 was the first release of the current design; v0.3.0 and earlier are a retired prototype. The tested host is Snow Linux 13 with systemd 261.2, QEMU 10.0.13, virtiofsd 1.13.2 and GNOME Wayland. Windows are Wayland only, host file edits produce no inotify events in machines, and idle machines stop even when a service inside them is busy. [Limits and troubleshooting →](https://frostyard.github.io/nsl/reference/limits/) diff --git a/docs/adr/0005-vmspawn-and-nspawn-images.md b/docs/adr/0005-vmspawn-and-nspawn-images.md index 3a4a383..376dfea 100644 --- a/docs/adr/0005-vmspawn-and-nspawn-images.md +++ b/docs/adr/0005-vmspawn-and-nspawn-images.md @@ -7,11 +7,15 @@ The [vmspawn comparison](../plans/vmspawn-comparison.md) established a complete workflow on Snow with systemd-vmspawn and QEMU: rootless KVM and vsock, commands, localhost forwarding and Waypipe. It passed 20 restart trials, started faster than the Lima prototype, and returned freed anonymous memory to the host. +vmspawn's flags change between systemd releases. qcow2 disks (`--image-format`, a format in `--extra-drive`) and `--user` arrived in 260; `--firmware=describe` and a disk type in `--extra-drive` arrived in 261. Fedora 44 and Ubuntu 26.04 LTS ship systemd 259 and Debian 13 ships 257; no stable distribution ships 260 ([#54](https://github.com/frostyard/nsl/issues/54)). + ## Decision Run every nsl VM with systemd-vmspawn and QEMU under a systemd user unit. Lima remains only an image-build tool. - **Launch** through the rootless named-device-descriptor path: open `/dev/kvm` and `/dev/vhost-vsock` under the `kvm` group, then run vmspawn in an unprivileged user namespace that maps the user to themselves. +- **Host floor:** systemd 259. Launch passes only flags that vmspawn 259 accepts. It omits `--user`: the namespace keeps the user's UID, so every release from 259 on already runs vmspawn in the user's service manager. `doctor` and launch fail on an older vmspawn and name both versions. +- **Disks** are raw files, because vmspawn 259 attaches no other format. The root is a copy of the cached VM image, cloned where the filesystem shares extents, otherwise copied without its holes and zero blocks, then extended to 16 GiB. The data disk is a sparse file that grows by extension. vmspawn names the raw format to QEMU, so nothing a guest writes to a disk can make QEMU open another file. - **Group membership:** doctor and launch check the host account database before invoking `sg` or `newgrp`, so a non-member gets instructions to join `kvm` instead of a password prompt. Use host `getent` and `id` for NSS lookups: release binaries disable cgo, so Go's `os/user` would read only local files. Query the named account's groups so membership added since login takes effect immediately. - **Identity:** supply the VM's ID, the host UID and primary GID, and a unique public SSH key through a systemd credential. The VM binds that identity on first boot and rejects a different one later. Private keys stay on the host. - **Uniqueness:** give each VM its own unit names and vsock CID, and control a unit only after its description names the VM's ID. @@ -22,7 +26,10 @@ Run every nsl VM with systemd-vmspawn and QEMU under a systemd user unit. Lima r ## Consequences - nsl owns lifecycle, readiness and forwarding behavior, which need failure and multi-VM tests. -- Host systemd-vmspawn, vhost-vsock access through `kvm`, and user namespaces are prerequisites. The tested baseline is systemd 261 on x86-64. +- Host systemd-vmspawn from systemd 259 or newer, vhost-vsock access through `kvm`, and user namespaces are prerequisites. Tested hosts run systemd 261 and 262 on x86-64. Each new floor needs an acceptance run on a host that ships it. +- `doctor` cannot ask vmspawn 259 which firmware it would select. It lists vmspawn's firmware descriptors and applies the selection rules of systemd 259 to 262 itself. +- Without shared extents, as on ext4, each fresh root writes the image's non-zero blocks, about 1 GB of the 2.9 GB image: at creation, after `update` and on `recover`, for the shared VM and each isolated one. On btrfs and XFS the copy shares the image's blocks. +- `qemu-img` is no longer a prerequisite. A raw disk has no metadata to verify, so the data disk check is its ownership, type and size. - Generic images configure the host's numeric UID and GID at boot without rebuilding. - Host file changes do not produce inotify events in the guest; watchers need polling or guest-native files. @@ -31,6 +38,9 @@ Run every nsl VM with systemd-vmspawn and QEMU under a systemd user unit. Lima r - **Keep Lima as the runtime:** mature integration, but the measured vmspawn path justified owning the missing pieces. - **libvirt or Incus:** capable VM managers that add a daemon and their own image and network models. - **Another implementation language:** no demonstrated need; Go manages the required tools and state. +- **Require systemd 261:** keeps qcow2 overlays and `--firmware=describe`, but excludes Fedora 44 and Ubuntu 26.04 LTS. Requiring 260 gains no distribution. +- **qcow2 on newer systemd, raw on 259:** two disk formats and two launch paths to test, for a saving only on hosts without shared extents. +- **Serve qcow2 to vmspawn 259 as a raw file through `qemu-storage-daemon`'s FUSE export:** adds a daemon and FUSE to every disk access. ## References diff --git a/docs/adr/0008-offline-storage-management.md b/docs/adr/0008-offline-storage-management.md index 178d7d2..579a108 100644 --- a/docs/adr/0008-offline-storage-management.md +++ b/docs/adr/0008-offline-storage-management.md @@ -22,7 +22,7 @@ Lifecycle commands that waited on a lock reject a replacement ID. `resize [NAME] --disk GiB`: - grows the stopped VM's data disk; shrinking is forbidden; -- records the pending target before invoking `qemu-img`, then syncs and verifies the disk before committing the new size; +- records the pending target before extending the disk file, then syncs and verifies the disk before committing the new size; - is completed by a repeated `resize` or by `recover`, and VM start refuses pending growth. The VM grows its filesystem at the next boot ([ADR-0010](0010-explicit-guest-root-growth.md)). The CLI reports virtual capacity separately from filesystem growth. diff --git a/docs/design/lifecycle.md b/docs/design/lifecycle.md index d87e8f2..ecccecd 100644 --- a/docs/design/lifecycle.md +++ b/docs/design/lifecycle.md @@ -10,7 +10,7 @@ nsl runs one VM per state directory, the shared VM, and one more for each isolat flowchart LR CLI[nsl CLI] --> Unit[VM user unit] Unit --> VM[vmspawn + QEMU/KVM] - VMImage[Signed VM image] --> Root[Replaceable root overlay] + VMImage[Signed VM image] --> Root[Replaceable root] Root --> VM Data[Data disk: machines + state] --> VM Cache[Verified image cache] -->|read-only virtiofs| VM @@ -28,14 +28,14 @@ flowchart LR | Path | Content | | --- | --- | -| `vm/` | The shared VM: its record, root overlay, data disk, SSH keypair, pinned host key, SSH configuration and boot credential. | +| `vm/` | The shared VM: its record, root, data disk, SSH keypair, pinned host key, SSH configuration and boot credential. | | `isolated/NAME/` | The same for isolated machine NAME's VM. | | `machines/NAME.json` | A machine record: ID, trust tier, account, image build and digest. | | `default` | The default machine's name, when there is one. | | `removing/NAME` | A machine whose removal is in progress; the name stays reserved. | | `images/`, `delivery/` | The verified image cache and authenticated catalogue history. | -A VM record holds the VM's ID, owner, UID and GID, the image build its root overlay came from and any pending image, the memory and CPUs in effect, and the data disk's committed and pending capacity. Runtime sockets live under `/run/user/UID/nsl/ID`, so long state paths do not lengthen socket names. +A VM record holds the VM's ID, owner, UID and GID, the image build its root came from and any pending image, the memory and CPUs in effect, and the data disk's committed and pending capacity. Runtime sockets live under `/run/user/UID/nsl/ID`, so long state paths do not lengthen socket names. Ownership, file type and permissions are checked before any change. The VM ID yields its unit names and a stable vsock CID, and creation avoids CIDs used by other VMs in the same state directory. Launch refuses a CID that already answers. Unit descriptions must name the ID before nsl controls a unit. @@ -43,7 +43,7 @@ A manager lock serializes image import, name allocation and the default. Per-VM ## The VM -The root is a qcow2 overlay on a cached, verified VM image. It holds no user state. `nsl update` records a pending image, and the next start discards the overlay and creates a new one from that image. The data disk is a separate qcow2 whose btrfs filesystem holds the machines and a state subvolume. That subvolume keeps the VM's binding, SSH host keys and machine records, so a new root keeps the pinned host key and every machine. +The root is a raw copy of a cached, verified VM image, extended to 16 GiB. The copy shares the image's extents where the filesystem can (btrfs, XFS); elsewhere it writes only the image's data, leaving its holes and zero blocks as holes. It holds no user state. `nsl update` records a pending image, and the next start discards the root and copies a new one from that image. The data disk is a separate sparse raw file whose btrfs filesystem holds the machines and a state subvolume. Both disks are raw because systemd 259's vmspawn attaches no other format ([ADR-0005](../adr/0005-vmspawn-and-nspawn-images.md)). That subvolume keeps the VM's binding, SSH host keys and machine records, so a new root keeps the pinned host key and every machine. vmspawn passes the `nsl.vm` credential: the VM ID, role, host UID and GID, public key, autostart setting, shares and aliases. On first boot the VM formats a blank data disk and records the binding. On later boots it rejects a credential that differs, and it grows the data filesystem to the disk before readiness. @@ -51,15 +51,15 @@ Resources come from the configuration when the VM starts, and the record keeps w ## Launch and readiness -The user needs KVM and vhost-vsock access through the `kvm` group. Doctor and launch share a membership check: `getent group kvm` resolves the group's GID through NSS, and `id -G -- USER` returns the account's current group IDs, including its primary group. This works in static release builds and recognizes membership added since login ([ADR-0005](../adr/0005-vmspawn-and-nspawn-images.md)). Missing membership produces a command for an administrator to run; lookup failures report their cause. Neither invokes the group switch when the check fails. A fixed internal command opens the devices under that group, restores the account's primary group, and enters an unprivileged user namespace that maps the user's UID and GID to themselves. It verifies the devices and passes them to vmspawn as named file descriptors. Capabilities stay scoped to that namespace. nsl changes no host permissions, groups, packages or sudoers. +The user needs KVM and vhost-vsock access through the `kvm` group. Doctor and launch share a membership check: `getent group kvm` resolves the group's GID through NSS, and `id -G -- USER` returns the account's current group IDs, including its primary group. This works in static release builds and recognizes membership added since login ([ADR-0005](../adr/0005-vmspawn-and-nspawn-images.md)). Missing membership produces a command for an administrator to run; lookup failures report their cause. Neither invokes the group switch when the check fails. A fixed internal command opens the devices under that group, restores the account's primary group, and enters an unprivileged user namespace that maps the user's UID and GID to themselves. It verifies the devices and passes them to vmspawn as named file descriptors. Capabilities stay scoped to that namespace. Because the namespace keeps the user's UID, vmspawn runs in the user's service manager without `--user`, which systemd 259 lacks. nsl changes no host permissions, groups, packages or sudoers. -The unit `nsl-UID-vm-ID.service` runs vmspawn with the root overlay, the data disk as an extra drive, the virtiofs shares, the read-only image cache and the credential. A newly launched VM gets up to 90 seconds to become ready. Readiness is the agent's `identity` answer over authenticated vsock SSH. The host checks it for the VM's ID, UID, GID and role, and for the image descriptor's protocols and architecture, even when the unit was already running. Failures keep the disks and point to `logs` and `recover`. Forwarding starts only after readiness. +The unit `nsl-UID-vm-ID.service` runs vmspawn with the root, the data disk as an extra drive, the virtiofs shares, the read-only image cache and the credential. A newly launched VM gets up to 90 seconds to become ready. Readiness is the agent's `identity` answer over authenticated vsock SSH. The host checks it for the VM's ID, UID, GID and role, and for the image descriptor's protocols and architecture, even when the unit was already running. Failures keep the disks and point to `logs` and `recover`. Forwarding starts only after readiness. The shared VM starts on first use. At boot it starts every machine unless `autostart` is false, or unless it was started only to create, import, export or remove a machine. It stops itself when no machine is running, and `shutdown` stops it at once. A stop asks the VM to power off, waits up to 30 seconds, then stops the owned unit. The VM's idle monitor makes both idle decisions, so the host runs no background process. It stops a running machine after `idle_timeout` without `nsl-run-*` sessions, Waypipe clients or a `start` or `run` request. It powers the VM off 60 seconds after the last machine stopped and the last agent request ended. Every agent request holds a shared lock that the monitor takes exclusively, and keeps, before powering off. A command that arrives then sees the VM stop, and its readiness loop starts it again. -`recover` stops the owned runtime, completes pending growth, checks the data disk without repairing it, and starts the VM again from a fresh root overlay, since the root holds nothing that must survive. It keeps keys, pinned host keys and machines. It cannot rebuild deleted keys or repair a corrupt filesystem, and it is not a backup. +`recover` stops the owned runtime, completes pending growth, checks the data disk's ownership, type and capacity without repairing it, and starts the VM again from a fresh root, since the root holds nothing that must survive. It keeps keys, pinned host keys and machines. It cannot rebuild deleted keys or repair a corrupt filesystem, and it is not a backup. ## Machines @@ -87,7 +87,7 @@ Machines that are not isolated see the user's home, `/run/media/USER` and `/mnt` `remove --yes` requires a stopped machine and the manager and machine locks. It renames the record into `removing/NAME`, and the agent deletes the subvolume and VM-side record. The host record goes last, so a repeated command resumes. -`resize --disk` grows a stopped VM's data disk. It records the target before invoking `qemu-img`, then syncs and verifies the disk and commits the new capacity. `recover` or a repeated resize completes interrupted growth. The VM grows the filesystem at its next boot, and shrinking is unsupported. See [ADR-0008](../adr/0008-offline-storage-management.md). +`resize --disk` grows a stopped VM's data disk. It records the target before extending the disk file, then syncs and verifies the disk and commits the new capacity. `recover` or a repeated resize completes interrupted growth. The VM grows the filesystem at its next boot, and shrinking is unsupported. See [ADR-0008](../adr/0008-offline-storage-management.md). ## Export and import diff --git a/docs/plans/shared-vm-implementation.md b/docs/plans/shared-vm-implementation.md index 50eb0ed..3195ec2 100644 --- a/docs/plans/shared-vm-implementation.md +++ b/docs/plans/shared-vm-implementation.md @@ -458,6 +458,37 @@ and `nsl_linux_arm64.tar.gz` with `checksums.txt`, and a cask whose URLs keep the tag and pin each archive's SHA256. The guide's URLs resolve from the first release after this change. +**systemd 259 floor, 2026-10-08 ([#54](https://github.com/frostyard/nsl/issues/54)):** +launch now passes only flags vmspawn 259 accepts, and both VM disks are raw +([ADR-0005](../adr/0005-vmspawn-and-nspawn-images.md)). Doctor and launch +refuse an older vmspawn; doctor then checks the firmware through +`--firmware=list`. `probe-vm.py` passed all nine checks on VM image r10 +(`sha256:9c3d68d7a1b963a6a482adee846b71282da1f721458f11de1db9ca295f42ea7a`, +`build/image/evidence/r10-raw-disks-probe.json`) with raw disks and without +`--user`: a 9.5 s first boot, and refusal and binding failing readiness in 9.1 +and 10.6 s. Host: kernel 7.2.8+deb14-amd64, systemd 262 (262-1), QEMU 10.0.13, +virtiofsd 1.13.2. `probe-machines.py --isolated`, with Debian r5 and Fedora 44 +r4 machines and an isolated Debian r5 machine +(`build/image/evidence/r10-raw-disks-machines.json`), passed every check but +one item: Fedora's tally could not reinstall `tzdata`, because the cached r4 +image's version has left Fedora's repositories; its time zone link was correct. +Doctor chose `/usr/share/OVMF/OVMF_CODE_4M.fd`, as vmspawn 262's +`--firmware=describe --secure-boot=no` does. `make ci` and `make site` passed. + +Root cost on the same host's NVMe btrfs, with the image in page cache: cloning +the 2.9 GB image took 130 µs. The copy used without shared extents, which +skips holes and zero blocks, took 0.34–0.39 s and allocated 965 MB; writing +every block took 0.7 s and 2.7 GB. No ext4 filesystem was available; the +Ubuntu 26.04 run below measures it. + +Still required before a release documents 259 as supported: on Fedora 44 +(systemd 259.9), `probe-vm.py` and `probe-machines.py --isolated` must pass, +covering virtiofs shares, the read-only image cache, `update`, `recover`, +`resize` and an isolated machine. vmspawn 259 starts virtiofsd through +`systemd-socket-activate` instead of in a namespace, which no run has covered +yet. An Ubuntu 26.04 run on ext4 should record `recover`'s time for the shared +VM and for an isolated machine. + ## Requirements carried from the experiment Each item was found by a failing check and must not regress. diff --git a/docs/specs/cli.md b/docs/specs/cli.md index a8e6e5f..8e28e7a 100644 --- a/docs/specs/cli.md +++ b/docs/specs/cli.md @@ -90,7 +90,9 @@ Machine names start with a lowercase ASCII letter, then lowercase letters, digit - A forwarder user unit per VM, `nsl-UID-vm-ID-ports.service`, bound to the VM's unit, MUST poll the agent's `listeners` once a second. It forwards host `127.0.0.1:PORT` to VM `127.0.0.1:PORT` for listeners on `127.0.0.1`, `0.0.0.0` or `::`, and to `[::1]:PORT` for listeners only on `::1`. `nsl ports` MUST show each port's machine and state, `forwarded` or `conflict` with the error. - Resource limits MUST come from the configuration: one budget for the shared VM, and one for each isolated machine's VM. - An isolated machine's VM MUST be created with the machine, from the VM image selected for the shared VM, and MUST be deleted with it: `remove` stops it and deletes its disks. A failed creation MUST leave no isolated VM. Every VM has its own ID, units, runtime directory, keys and forwarder, and a vsock CID no other VM of the state directory uses. -- `doctor` MUST fail when systemd-vmspawn finds no firmware for the firmware flags a VM launch passes. It asks vmspawn to describe that firmware with the same flags, because listing firmware succeeds when there is none. +- `doctor` and VM launch MUST fail when systemd-vmspawn comes from a systemd older than 259, naming the installed and required versions ([ADR-0005](../adr/0005-vmspawn-and-nspawn-images.md)). `doctor` MUST report that before the firmware check, and launch MUST refuse before changing any state. +- `doctor` MUST fail when systemd-vmspawn lists no firmware that launch's `--secure-boot=no` selects. It reads the descriptors vmspawn lists, in vmspawn's order, and accepts the first that systemd 259 to 262 would all select: UEFI with an NVRAM template, an x86-64 q35 target, and neither Secure Boot nor enrolled keys. vmspawn 259 cannot describe its choice, and the list succeeds when there is no firmware. +- VM launch MUST pass only flags that systemd 259's vmspawn accepts, and both VM disks MUST be raw. - `doctor` and VM launch MUST check `kvm` membership in the host account database before invoking `sg` or `newgrp`. Missing membership MUST produce instructions to join the group and skip the group switch; lookup failures MUST be reported separately. Membership added since login MUST be recognized. Doctor MUST still probe device access when membership is confirmed. ### State, ownership and locking diff --git a/docs/specs/vm-image.md b/docs/specs/vm-image.md index fa56e39..4568cae 100644 --- a/docs/specs/vm-image.md +++ b/docs/specs/vm-image.md @@ -10,8 +10,8 @@ The [agent protocol](agent.md) covers commands and machine operations. [Machine | Disk | Content | Lifetime | | --- | --- | --- | -| Root | The image, as a 16 GiB qcow2 overlay the host creates for each VM image build; the root filesystem grows into it. | Replaced when the VM image changes and by `nsl recover`. Holds no user state. | -| Data | A btrfs filesystem labelled `nsl-data`, with subvolumes `machines` at `/var/lib/machines` and `state` at `/var/lib/nsl`. | Kept for the VM's lifetime; grows offline and never shrinks. | +| Root | A raw copy of the image, extended to 16 GiB, that the host makes for each VM image build; the root filesystem grows into it. | Replaced when the VM image changes and by `nsl recover`. Holds no user state. | +| Data | A sparse raw file holding a btrfs filesystem labelled `nsl-data`, with subvolumes `machines` at `/var/lib/machines` and `state` at `/var/lib/nsl`. | Kept for the VM's lifetime; grows offline and never shrinks. | The `state` subvolume holds everything that must survive a root replacement: @@ -42,7 +42,7 @@ vmspawn passes the credential `nsl.vm`, a JSON object: The host launches the VM through the rootless device-descriptor path of [the lifecycle design](../design/lifecycle.md#launch-and-readiness): -- the root overlay and the data disk as an extra drive; +- the root, and the data disk as an extra drive, both raw; - each share read-write through virtiofs, at its `/mnt/host` path; - the host's verified machine-image cache read-only at `/var/cache/nsl/images`; - user-mode networking, vsock with the VM's CID, and the credential; @@ -95,8 +95,8 @@ The host launches the VM through the rootless device-descriptor path of [the lif ### Root replacement -- A new root overlay from a different image build MUST keep the VM's binding, SSH host keys, machines and records, so the host's pinned host key still matches. -- The VM MUST NOT store user or machine state on the root. The host MAY discard the root overlay whenever the VM is stopped. +- A new root from a different image build MUST keep the VM's binding, SSH host keys, machines and records, so the host's pinned host key still matches. +- The VM MUST NOT store user or machine state on the root. The host MAY discard the root whenever the VM is stopped. ### Host files @@ -126,7 +126,7 @@ The host launches the VM through the rootless device-descriptor path of [the lif | Formatting | A blank data disk becomes `nsl-data` with both subvolumes. | | Refusal | A data disk with an existing signature (the probe writes a swap header) fails readiness quickly and stays byte-identical. | | Binding | A credential with a different `id` or `uid` fails readiness. | -| Root replacement | After a new root overlay, the pinned host key, `identity.json` and a marker machine survive. | +| Root replacement | After a new root, the pinned host key, `identity.json` and a marker machine survive. | | Growth | After the host grows the data disk, the filesystem reports the new size. | | Allowlist | Each share is mounted at its `/mnt/host` path, and nothing else is. | | Ownership | Host files show the host UID and GID; writes by the account and by root land as the host user; root cannot write a root-owned host directory. | diff --git a/isolated_test.go b/isolated_test.go index 3464be8..0b26cbc 100644 --- a/isolated_test.go +++ b/isolated_test.go @@ -48,7 +48,7 @@ func TestIsolatedMachineGetsItsOwnVM(t *testing.T) { } args := strings.Join(a.launchArgs(v, &cred), " ") if strings.Contains(args, "--bind=") || !strings.Contains(args, "--cpus=2") || !strings.Contains(args, "--ram=2G") || - !strings.Contains(args, "--image="+v.dir+"/root.qcow2") { + !strings.Contains(args, "--image="+v.dir+"/root.raw") { t.Fatal(args) } launched := 0 @@ -238,7 +238,7 @@ func TestIsolatedRemovalRequiresConfirmedStatus(t *testing.T) { if err := a.remove([]string{m.Name, "--yes"}); err == nil { t.Fatal("removed a machine without confirming its state") } - if _, err := os.Stat(filepath.Join(v.dir, "data.qcow2")); err != nil { + if _, err := os.Stat(filepath.Join(v.dir, "data.raw")); err != nil { t.Fatal("lost data disk:", err) } for _, req := range f.requests { diff --git a/main_test.go b/main_test.go index 25f7415..4889a3a 100644 --- a/main_test.go +++ b/main_test.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "crypto/sha256" - "encoding/binary" "encoding/hex" "encoding/json" "errors" @@ -14,7 +13,6 @@ import ( "os" "path/filepath" "reflect" - "strconv" "strings" "sync" "testing" @@ -28,7 +26,7 @@ type call struct { Args []string } -// fakeRunner emulates systemd user units, vmspawn's firmware description, qemu-img, ssh-keygen and the agent. +// fakeRunner emulates systemd user units, vmspawn's version and firmware list, ssh-keygen and the agent. type fakeRunner struct { mu sync.Mutex // helpers call the runner from several goroutines calls []call @@ -40,27 +38,12 @@ type fakeRunner struct { identity func(*protocol.Identity) agent func(req *protocol.Request, stdin io.Reader, stdout io.Writer) error requests []*protocol.Request - failResize bool - failCheck bool - onResize func() - firmware string // vmspawn's firmware description; empty when the host has none + vmspawnVersion string // vmspawn --version's first line + firmware string // vmspawn --firmware=list: descriptor paths, one per line kvmGroup string // getent's group record accountGroups string // id -G USER; independent of the session's groups } -func qcow2(path string, gibs int64, backing bool) error { - h := make([]byte, 104) - copy(h, "QFI\xfb") - binary.BigEndian.PutUint32(h[4:], 3) - if backing { - binary.BigEndian.PutUint64(h[8:], 104) - } - binary.BigEndian.PutUint32(h[20:], 16) - binary.BigEndian.PutUint64(h[24:], uint64(gibs*gib)) - binary.BigEndian.PutUint32(h[100:], 104) - return os.WriteFile(path, h, 0644) -} - func (f *fakeRunner) run(ctx context.Context, in io.Reader, out, stderr io.Writer, env []string, bin string, args ...string) error { f.mu.Lock() f.calls = append(f.calls, call{bin, append([]string{}, args...)}) @@ -81,11 +64,12 @@ func (f *fakeRunner) run(ctx context.Context, in io.Reader, out, stderr io.Write _, err := io.WriteString(out, f.accountGroups) return err case "systemd-vmspawn": - if f.firmware == "" { - fmt.Fprintln(stderr, "Failed to find OVMF config: No such file or directory") - return errors.New("exit status 1") + // Like vmspawn, the list succeeds with no firmware. + answer := f.firmware + if args[0] == "--version" { + answer = f.vmspawnVersion + "\n+PAM +AUDIT\n" } - _, err := io.WriteString(out, f.firmware) + _, err := io.WriteString(out, answer) return err case "ssh-keygen": path := args[len(args)-1] @@ -93,25 +77,6 @@ func (f *fakeRunner) run(ctx context.Context, in io.Reader, out, stderr io.Write return err } return os.WriteFile(path+".pub", []byte("ssh-ed25519 AAAA nsl-vm\n"), 0644) - case "qemu-img": - switch args[0] { - case "create": - size, _ := strconv.ParseInt(strings.TrimSuffix(args[len(args)-1], "G"), 10, 64) - return qcow2(args[len(args)-2], size, strings.Contains(strings.Join(args, " "), " -b ")) - case "check": - if f.failCheck { - return errors.New("injected disk check failure") - } - case "resize": - if f.onResize != nil { - f.onResize() - } - if f.failResize { - return errors.New("injected resize failure") - } - size, _ := strconv.ParseInt(strings.TrimSuffix(args[len(args)-1], "G"), 10, 64) - return qcow2(args[len(args)-2], size, false) - } case "systemctl": unit := args[2] switch args[1] { @@ -221,7 +186,8 @@ func (f *fakeRunner) ran(bin string, prefix ...string) int { func testApp(t *testing.T) (*app, *fakeRunner) { t.Helper() - f := &fakeRunner{states: map[string]string{}, descriptions: map[string]string{}, kvmGroup: "kvm:x:993:u\n", accountGroups: "1000 993\n"} + f := &fakeRunner{states: map[string]string{}, descriptions: map[string]string{}, kvmGroup: "kvm:x:993:u\n", accountGroups: "1000 993\n", + vmspawnVersion: "systemd 262 (262-1)"} host := t.TempDir() t.Setenv("HOME", filepath.Join(host, "home", "u")) t.Setenv("XDG_CONFIG_HOME", filepath.Join(host, "config")) @@ -331,15 +297,15 @@ func TestUpdateSelectsAnImageForTheNextStart(t *testing.T) { if st, err := os.Stat(cached); err != nil || st.Mode().Perm() != 0444 { t.Fatal(err) } - if f.ran("qemu-img", "create", "-q", "-f", "qcow2", "-F", "raw", "-b", cached) != 1 { - t.Fatal(f.calls) + if got := rootStart(t, v, "vm image"); got != "vm image" { + t.Fatalf("the root starts with %q, not the image", got) } // A second image waits for the next start and never touches the running VM. image, digest := localImage(t, "newer vm image") if err := a.execute([]string{"update", "--image", image, "--digest", digest}); err != nil { t.Fatal(err) } - if v, _ = a.loadVM(); v.PendingImage != strings.TrimPrefix(digest, "sha256:") || f.ran("qemu-img", "create", "-q", "-f", "qcow2", "-F", "raw") != 1 { + if v, _ = a.loadVM(); v.PendingImage != strings.TrimPrefix(digest, "sha256:") || rootStart(t, v, "vm image") != "vm image" { t.Fatal("update changed a running VM") } var out bytes.Buffer @@ -354,6 +320,35 @@ func TestUpdateSelectsAnImageForTheNextStart(t *testing.T) { if v, _ = a.runningVM(true); v.Image != strings.TrimPrefix(digest, "sha256:") || v.PendingImage != "" { t.Fatalf("%+v", v) } + if got := rootStart(t, v, "newer vm image"); got != "newer vm image" { + t.Fatalf("the root starts with %q, not the new image", got) + } +} + +// rootStart checks that a VM's root is a private file of the root's capacity +// and returns as many of its first bytes as want has. +func rootStart(t *testing.T, v *vmRecord, want string) string { + t.Helper() + root := filepath.Join(v.dir, "root.raw") + if st, err := os.Stat(root); err != nil || st.Size() != rootGiB*gib || st.Mode().Perm() != 0600 { + t.Fatal("root:", st, err) + } + return diskStart(t, root, len(want)) +} + +// diskStart reads a disk's first n bytes; disks are too large to read whole. +func diskStart(t *testing.T, path string, n int) string { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + b := make([]byte, n) + if _, err = io.ReadFull(f, b); err != nil { + t.Fatal(err) + } + return string(b) } func TestUpdateRefusesBadImages(t *testing.T) { @@ -411,7 +406,7 @@ func TestLaunchArgumentsAndCredential(t *testing.T) { } args := strings.Join(a.launchArgs(v, &cred), " ") for _, want := range []string{ - "--image=" + v.dir + "/root.qcow2", "--extra-drive=qcow2:virtio-blk:" + v.dir + "/data.qcow2", + "--image=" + v.dir + "/root.raw", "--extra-drive=" + v.dir + "/data.raw", "--bind-ro=" + a.home + "/images/machines:/var/cache/nsl/images", "--bind=" + home + ":/mnt/host" + home, "--load-credential=nsl.vm:" + v.dir + "/nsl.vm", "--cpus=8", "--ram=8G", "--register=no", "--secure-boot=no", } { @@ -419,6 +414,11 @@ func TestLaunchArgumentsAndCredential(t *testing.T) { t.Fatalf("missing %s in %s", want, args) } } + for _, arg := range a.launchArgs(v, &cred) { + if arg == "--user" || strings.HasPrefix(arg, "--image-format") || strings.HasPrefix(arg, "--firmware") { + t.Fatalf("launch passes %s, which systemd 259's vmspawn rejects", arg) + } + } } func TestIsolatedHostFilesStayOutOfTheCacheShare(t *testing.T) { @@ -494,7 +494,11 @@ func TestLockRejectsAReplacedVM(t *testing.T) { } func TestResizeGrowsTheStoppedDataDisk(t *testing.T) { - a, f, _ := startedVM(t) + a, f, v := startedVM(t) + disk := filepath.Join(v.dir, "data.raw") + if got, err := diskGiB(disk); err != nil || got != defaultDataGiB { + t.Fatal(got, err) + } if err := a.resize([]string{"--disk", "200"}); err == nil || !strings.Contains(err.Error(), "running") { t.Fatal(err) } @@ -502,53 +506,78 @@ func TestResizeGrowsTheStoppedDataDisk(t *testing.T) { if err := a.resize([]string{"--disk", "64"}); err == nil || !strings.Contains(err.Error(), "shrinking") { t.Fatal(err) } - f.onResize = func() { - if r, _ := a.loadVM(); r.ResizeTarget != 200 { - t.Fatal("resized before recording the target") - } - } - f.failResize = true + // Growth that cannot write the disk fails after recording its target. + os.Chmod(disk, 0400) if err := a.resize([]string{"--disk", "200"}); err == nil { t.Fatal("hid a failed resize") } + if r, _ := a.loadVM(); r.ResizeTarget != 200 { + t.Fatal("did not record the target before growing") + } + if got, _ := diskGiB(disk); got != defaultDataGiB { + t.Fatal("a failed resize changed the disk to", got) + } if _, err := a.runningVM(true); err == nil || !strings.Contains(err.Error(), "recover") { t.Fatal("started with pending growth:", err) } if err := a.resize([]string{"--disk", "300"}); err == nil || !strings.Contains(err.Error(), "pending growth to 200") { t.Fatal(err) } - f.failResize = false + os.Chmod(disk, 0600) f.vm, _ = a.loadVM() if err := a.recover(nil); err != nil { t.Fatal(err) } - v, _ := a.loadVM() + v, _ = a.loadVM() if v.DataGiB != 200 || v.ResizeTarget != 0 { t.Fatalf("%+v", v) } - if got, err := diskGiB(filepath.Join(v.dir, "data.qcow2")); err != nil || got != 200 { + if got, err := diskGiB(disk); err != nil || got != 200 { t.Fatal(got, err) } + // Growth interrupted after extending the disk commits without extending it again. + a.shutdown() + v.ResizeTarget = 256 + if err := a.saveVM(v); err != nil { + t.Fatal(err) + } + os.Truncate(disk, 256*gib) + if err := a.resize([]string{"--disk", "256"}); err != nil { + t.Fatal(err) + } + if v, _ = a.loadVM(); v.DataGiB != 256 || v.ResizeTarget != 0 { + t.Fatalf("%+v", v) + } } func TestRecoverRebuildsTheRootAndKeepsTheDataDisk(t *testing.T) { - a, f, v := startedVM(t) - data, _ := os.ReadFile(filepath.Join(v.dir, "data.qcow2")) - before := f.ran("qemu-img", "create", "-q", "-f", "qcow2", "-F", "raw") + a, _, v := startedVM(t) + root, data := filepath.Join(v.dir, "root.raw"), filepath.Join(v.dir, "data.raw") + for _, p := range []string{root, data} { + f, err := os.OpenFile(p, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + f.WriteAt([]byte("used"), 0) + f.Close() + } if err := a.recover(nil); err != nil { t.Fatal(err) } - if f.ran("qemu-img", "create", "-q", "-f", "qcow2", "-F", "raw") != before+1 { - t.Fatal("root not rebuilt") + if got := rootStart(t, v, "vm image"); got != "vm image" { + t.Fatalf("root not rebuilt: it starts with %q", got) } - if after, _ := os.ReadFile(filepath.Join(v.dir, "data.qcow2")); !bytes.Equal(data, after) { + if got := diskStart(t, data, 4); got != "used" { t.Fatal("data disk changed") } a.shutdown() - f.failCheck = true + os.Truncate(data, gib+1) if err := a.recover(nil); err == nil || !strings.Contains(err.Error(), "preserved") { t.Fatal(err) } + if st, err := os.Stat(data); err != nil || st.Size() != gib+1 { + t.Fatal("recover changed a disk that failed its check") + } } func TestPendingRestartIsReported(t *testing.T) { diff --git a/scripts/probe-vm.py b/scripts/probe-vm.py index 41ab8d3..4575192 100755 --- a/scripts/probe-vm.py +++ b/scripts/probe-vm.py @@ -154,33 +154,27 @@ def host_files(self, home): def refusal(self): home = self.fresh('refusal') - data = home/'vm/data.qcow2' - raw = home/'vm/signed.raw' + data = home/'vm/data.raw' # A Linux swap signature: any existing signature must be refused, and this # one needs no formatting tool on an atomic host. - with raw.open('wb') as f: - f.truncate(128 << 30) + with data.open('r+b') as f: f.seek(1024) f.write((1).to_bytes(4, 'little') + ((128 << 30) // 4096 - 1).to_bytes(4, 'little') + bytes(4) + uuid.uuid4().bytes) f.seek(4096 - 10) f.write(b'SWAPSPACE2') - data.unlink() - subprocess.run(['qemu-img', 'convert', '-q', '-f', 'raw', '-O', 'qcow2', str(raw), str(data)], check=True) - raw.unlink() - data.chmod(0o600) - before = hashlib.sha256(data.read_bytes()).hexdigest() + before = signature(data) began = time.monotonic() r = self.cli(home, 'recover', check=False) seconds = round(time.monotonic() - began, 2) journal = self.unit_journal(home) self.cli(home, 'shutdown', check=False) - after = hashlib.sha256(data.read_bytes()).hexdigest() + after = signature(data) self.record('refusal', r.returncode != 0 and before == after and 'refusing to format' in journal, seconds=seconds, error=r.stderr.strip()[-200:], unchanged=before == after) def binding(self, bound): home = self.fresh('binding') - shutil.copyfile(bound/'vm/data.qcow2', home/'vm/data.qcow2') + sparse_copy(bound/'vm/data.raw', home/'vm/data.raw') vm = json.loads((home/'vm/vm.json').read_text()) vm['data_gib'] = json.loads((bound/'vm/vm.json').read_text())['data_gib'] (home/'vm/vm.json').write_text(json.dumps(vm)) @@ -192,6 +186,32 @@ def binding(self, bound): self.record('binding', r.returncode != 0 and 'does not match' in journal, seconds=seconds, error=r.stderr.strip()[-200:]) +def signature(path): + """Hash a sparse disk's size and allocated data without reading its holes.""" + h = hashlib.sha256() + with path.open('rb') as f: + fd = f.fileno() + size = os.fstat(fd).st_size + h.update(size.to_bytes(8, 'little')) + offset = 0 + while offset < size: + try: + start = os.lseek(fd, offset, os.SEEK_DATA) + except OSError: # no data after offset + break + offset = os.lseek(fd, start, os.SEEK_HOLE) + h.update(start.to_bytes(8, 'little')) + f.seek(start) + for chunk in iter(lambda: f.read(min(1 << 20, offset - f.tell())), b''): + h.update(chunk) + return h.hexdigest() + + +def sparse_copy(source, target): + """Replace target with a copy of a sparse disk, keeping its holes.""" + subprocess.run(['cp', '--reflink=auto', '--sparse=always', str(source), str(target)], check=True) + + def version(*argv): try: return subprocess.run(argv, capture_output=True, text=True, timeout=10).stdout.splitlines()[0].strip() diff --git a/site/content/concepts/how-it-works.md b/site/content/concepts/how-it-works.md index 36ccd6c..b28be35 100644 --- a/site/content/concepts/how-it-works.md +++ b/site/content/concepts/how-it-works.md @@ -10,7 +10,7 @@ Your ordinary machines run as systemd-nspawn containers inside one shared VM. Ea flowchart LR CLI[nsl CLI] --> Unit[VM user unit] Unit --> VM[vmspawn + QEMU/KVM] - VMImage[Signed VM image] --> Root[Replaceable root overlay] + VMImage[Signed VM image] --> Root[Replaceable root] Root --> VM Data[Data disk: machines + state] --> VM Cache[Verified image cache] -->|read-only virtiofs| VM @@ -32,8 +32,8 @@ flowchart LR The shared VM boots the signed nsl VM image with systemd-vmspawn and QEMU/KVM, under a user unit named `nsl-UID-vm-ID.service`. It has two disks: -- **The root**, a qcow2 overlay on the cached VM image. It holds no user state, so `nsl update` and `nsl recover` can replace it at the next start. -- **The data disk**, a qcow2 image with a btrfs filesystem. It holds every machine and the VM's own state: its identity, its SSH host keys and the machine records. +- **The root**, a copy of the cached VM image. It holds no user state, so `nsl update` and `nsl recover` can replace it at the next start. +- **The data disk**, a sparse file with a btrfs filesystem. It holds every machine and the VM's own state: its identity, its SSH host keys and the machine records. The VM starts when you first need it and stops when idle. By default it can use half the host's memory and every host CPU, much like WSL. You can change those limits in [`nsl.conf`](../reference/configuration.md). The machines share the VM's resources, so adding a machine doesn't reserve another VM's worth of memory. diff --git a/site/content/getting-started/install.md b/site/content/getting-started/install.md index 1709a69..7d6ca97 100644 --- a/site/content/getting-started/install.md +++ b/site/content/getting-started/install.md @@ -10,9 +10,9 @@ nsl is a single binary that runs as your user. You'll need an x86-64 Linux host | Requirement | Used for | | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | -| `systemd-vmspawn`, `systemd-run`, `systemctl` and a user systemd manager | Running each VM as a user unit | -| QEMU (`qemu-system-x86_64`, `qemu-img`) with KVM | The VM and its disks | -| UEFI firmware without Secure Boot, with a QEMU firmware descriptor (Debian's `ovmf`) | Booting the VM | +| systemd 259 or newer: `systemd-vmspawn`, `systemd-run`, `systemctl` and a user systemd manager | Running each VM as a user unit | +| QEMU (`qemu-system-x86_64`) with KVM | The VM | +| UEFI firmware without Secure Boot, with a QEMU firmware descriptor (`ovmf`, or Fedora's `edk2-ovmf`) | Booting the VM | | virtiofsd at `/usr/libexec/virtiofsd` | Sharing your files and the image cache with the VM | | OpenSSH (`ssh`, `ssh-keygen`) and `/usr/lib/systemd/systemd-ssh-proxy` | Reaching the VM's agent over vsock | | `sg` or util-linux `newgrp` (on Debian 14, both from `util-linux-extra`), and util-linux `unshare` | Opening the KVM devices through your `kvm` membership | @@ -102,7 +102,7 @@ OK kvm group membership For a `MISSING` result, follow [troubleshooting](../reference/limits.md#troubleshooting). -`doctor` prints `OK` or `MISSING` for each tool and for the UEFI firmware systemd-vmspawn would use. It checks device access in your current session, looks up your `kvm` membership in the host account database, then tries KVM and vsock access through that group. It also checks user namespaces and the user systemd manager. A missing VM prerequisite makes it exit with an error. +`doctor` prints `OK` or `MISSING` for each tool, for the systemd version and for the UEFI firmware systemd-vmspawn would use. Fedora 44 and Ubuntu 26.04 LTS ship systemd 259; Debian 13 ships 257, which is too old. It checks device access in your current session, looks up your `kvm` membership in the host account database, then tries KVM and vsock access through that group. It also checks user namespaces and the user systemd manager. A missing VM prerequisite makes it exit with an error. Having firmware installed isn't enough on its own. systemd-vmspawn needs a QEMU firmware descriptor in `/usr/share/qemu/firmware` or `/etc/qemu/firmware`; without one, `doctor` reports the firmware as missing. Waypipe is optional, but you'll need it to display machine windows on the host. diff --git a/site/content/guides/storage.md b/site/content/guides/storage.md index 3aa8a79..105b7bc 100644 --- a/site/content/guides/storage.md +++ b/site/content/guides/storage.md @@ -10,7 +10,7 @@ Machines, disks and keys live under `NSL_HOME`, which defaults to `$XDG_DATA_HOM | Path in `NSL_HOME` | Content | | --- | --- | -| `vm/` | The shared VM: its record, root overlay, data disk, keys and boot credential | +| `vm/` | The shared VM: its record, root, data disk, keys and boot credential | | `isolated/NAME/` | The same for isolated machine NAME's VM | | `machines/NAME.json` | A machine's record: ID, trust tier, account and image | | `machines/NAME.ssh/` | The machine's key for [`ssh-config`](editors.md) | @@ -21,7 +21,7 @@ nsl checks the owner, type and permissions of these files before changing anythi ## The data disk -Each VM has two disks. nsl can replace the root overlay from the cached VM image. The data disk needs more care: it holds your machines and the VM's own state. +Each VM has two disks. The root is a copy of the cached VM image, which nsl can replace. On btrfs and XFS the copy shares the image's blocks; on other filesystems, such as ext4, each VM's root takes about 1 GB. The data disk needs more care: it holds your machines and the VM's own state. The data disk starts at 128 GiB of virtual capacity and takes host space only as it fills. Machines in a VM share it. `nsl list` shows each VM's capacity. diff --git a/site/content/reference/limits.md b/site/content/reference/limits.md index dd1f93d..acfe071 100644 --- a/site/content/reference/limits.md +++ b/site/content/reference/limits.md @@ -19,7 +19,7 @@ description: Known limits of nsl, and what to check when something goes wrong. ## Troubleshooting `nsl doctor` fails -: A tool, the UEFI firmware, a device or a permission is missing. Install the tool or firmware with your host's own tools. If doctor reports `MISSING kvm group membership`, ask an administrator to run the displayed command to add you to `kvm`, then retry nsl; no new login is needed. nsl changes none of these itself. +: A tool, the UEFI firmware, a device or a permission is missing, or systemd is older than 259. Install the tool or firmware with your host's own tools; `MISSING systemd 259 or newer` needs a newer host release. If doctor reports `MISSING kvm group membership`, ask an administrator to run the displayed command to add you to `kvm`, then retry nsl; no new login is needed. nsl changes none of these itself. Bare `nsl` fails and lists machines : There is no default machine, for example after removing it. Run `nsl default NAME`, or `nsl -m NAME`. diff --git a/state.go b/state.go index 2bddde5..2b4fe07 100644 --- a/state.go +++ b/state.go @@ -413,21 +413,26 @@ func (a *app) prepareVM(v *vmRecord) error { if err := atomicWrite(filepath.Join(v.dir, "ssh.config"), []byte(config), 0600); err != nil { return err } - data := filepath.Join(v.dir, "data.qcow2") + data := filepath.Join(v.dir, "data.raw") if _, err := os.Lstat(data); errors.Is(err, os.ErrNotExist) { + // A sparse raw file, which the VM formats at its first boot. f, err := os.CreateTemp(v.dir, ".data-*") if err != nil { return err } - f.Close() defer os.Remove(f.Name()) - if err = a.call(nil, a.err, "qemu-img", "create", "-q", "-f", "qcow2", f.Name(), fmt.Sprintf("%dG", v.DataGiB)); err != nil { - return err + err = f.Truncate(int64(v.DataGiB) * gib) + if err == nil { + err = f.Sync() } - if err = os.Chmod(f.Name(), 0600); err != nil { + if closeErr := f.Close(); err == nil { + err = closeErr + } + if err != nil { return err } - if err = os.Rename(f.Name(), data); err != nil { + // Link rather than rename, so an existing disk is never replaced. + if err = os.Link(f.Name(), data); err != nil { return err } } else if err != nil { diff --git a/storage.go b/storage.go index 80b3c74..6250477 100644 --- a/storage.go +++ b/storage.go @@ -1,9 +1,7 @@ package main import ( - "bytes" "crypto/sha256" - "encoding/binary" "encoding/hex" "errors" "flag" @@ -12,92 +10,35 @@ import ( "os" "path/filepath" "strings" + "syscall" "text/tabwriter" ) -// standaloneDisk checks a qcow2 disk before QEMU opens it: no backing file, -// encryption, external data or snapshots, and the expected capacity. -// See https://www.qemu.org/docs/master/interop/qcow2.html. -func standaloneDisk(path string, diskGiB int) error { - f, err := os.Open(path) - if err != nil { - return err - } - defer f.Close() - h := make([]byte, 104) - if _, err = io.ReadFull(f, h); err != nil { - return err - } - u32 := func(i int) uint32 { return binary.BigEndian.Uint32(h[i : i+4]) } - u64 := func(i int) uint64 { return binary.BigEndian.Uint64(h[i : i+8]) } - if !bytes.Equal(h[:4], []byte{'Q', 'F', 'I', 0xfb}) || (u32(4) != 2 && u32(4) != 3) { - return errors.New("data disk must be qcow2 version 2 or 3") - } - if u64(8) != 0 || u32(16) != 0 || u32(32) != 0 || u32(60) != 0 { - return errors.New("data disk must have no backing file, encryption or internal snapshots") - } - if u32(20) < 9 || u32(20) > 21 || u64(24) != uint64(int64(diskGiB)*gib) { - return errors.New("data disk size or cluster geometry mismatch") - } - header := uint32(72) - if u32(4) == 3 { - // Only the compression and extended-L2 feature bits are allowed. - if u64(72) & ^uint64(24) != 0 || u64(88)&2 != 0 { - return errors.New("data disk is dirty, corrupt or uses unsupported/external data features") - } - header = u32(100) - } - cluster := uint32(1) << u32(20) - if header < 72 || (u32(4) == 3 && header < 104) || header%8 != 0 || header > cluster-8 { - return errors.New("invalid qcow2 header length") - } - data := make([]byte, cluster) - if _, err = f.ReadAt(data, 0); err != nil && err != io.EOF { - return err - } - for off := uint64(header); off+8 <= uint64(cluster); { - kind := binary.BigEndian.Uint32(data[off : off+4]) - size := uint64(binary.BigEndian.Uint32(data[off+4 : off+8])) - if kind == 0 { - return nil - } - if kind == 0x44415441 || kind == 0xe2792aca || kind == 0x0537be77 { - return errors.New("external disk reference or encryption extension in the data disk") - } - off += 8 + ((size + 7) &^ uint64(7)) - if off > uint64(cluster) { - return errors.New("invalid qcow2 extension length") - } - } - return errors.New("unterminated qcow2 extensions") -} - +// diskGiB returns a raw disk's capacity in whole GiB. func diskGiB(path string) (int, error) { - f, err := os.Open(path) + st, err := os.Lstat(path) if err != nil { return 0, err } - defer f.Close() - var header [32]byte - if _, err = io.ReadFull(f, header[:]); err != nil { - return 0, err + if size := st.Size(); st.Mode().IsRegular() && size >= gib && size <= 4096*gib && size%gib == 0 { + return int(size / gib), nil } - size := binary.BigEndian.Uint64(header[24:]) - if string(header[:4]) != "QFI\xfb" || size < uint64(gib) || size > uint64(4096*gib) || size%uint64(gib) != 0 { - return 0, errors.New("invalid qcow2 capacity") - } - return int(size / uint64(gib)), nil + return 0, errors.New("invalid data disk capacity") } +// checkDataDisk checks the data disk before QEMU opens it: a private regular +// file with the committed capacity. vmspawn attaches it as raw, so nothing on +// the disk can make QEMU open another file. func (a *app) checkDataDisk(v *vmRecord) error { - disk := filepath.Join(v.dir, "data.qcow2") + disk := filepath.Join(v.dir, "data.raw") if err := privateFile(disk, a.uid, 0077); err != nil { return err } - if err := standaloneDisk(disk, v.DataGiB); err != nil { - return err + size, err := diskGiB(disk) + if err == nil && size != v.DataGiB { + err = fmt.Errorf("it has %d GiB, not %d", size, v.DataGiB) } - if err := a.call(nil, a.err, "qemu-img", "check", "-q", "-f", "qcow2", disk); err != nil { + if err != nil { return fmt.Errorf("data disk check failed; the disk is preserved for inspection: %w", err) } return nil @@ -169,7 +110,7 @@ func (a *app) finishGrowth(v *vmRecord) error { if err := a.requireStopped(v); err != nil { return err } - disk := filepath.Join(v.dir, "data.qcow2") + disk := filepath.Join(v.dir, "data.raw") if err := privateFile(disk, a.uid, 0077); err != nil { return err } @@ -180,29 +121,27 @@ func (a *app) finishGrowth(v *vmRecord) error { if current != v.DataGiB && current != v.ResizeTarget { return errors.New("data disk size differs from both committed and pending capacity; inspect before continuing") } - if err = standaloneDisk(disk, current); err != nil { - return err - } - if err = a.call(nil, a.err, "qemu-img", "check", "-q", "-f", "qcow2", disk); err != nil { + f, err := os.OpenFile(disk, os.O_RDWR|syscall.O_NOFOLLOW, 0) + if err != nil { return err } + // Extending a raw disk only appends a hole; the VM grows its filesystem. if current != v.ResizeTarget { - if err = a.call(nil, a.err, "qemu-img", "resize", "-q", "-f", "qcow2", disk, fmt.Sprintf("%dG", v.ResizeTarget)); err != nil { - return err - } + err = f.Truncate(int64(v.ResizeTarget) * gib) } - f, err := os.OpenFile(disk, os.O_RDWR, 0) - if err != nil { - return err + if err == nil { + err = f.Sync() } - err = f.Sync() if closeErr := f.Close(); err == nil { err = closeErr } if err != nil { return err } - if err = standaloneDisk(disk, v.ResizeTarget); err != nil { + if current, err = diskGiB(disk); err == nil && current != v.ResizeTarget { + err = errors.New("the data disk did not reach its pending capacity") + } + if err != nil { return err } v.DataGiB, v.ResizeTarget = v.ResizeTarget, 0 diff --git a/vm.go b/vm.go index 68c33a5..9706d33 100644 --- a/vm.go +++ b/vm.go @@ -21,6 +21,7 @@ import ( "time" "github.com/frostyard/nsl/internal/protocol" + "golang.org/x/sys/unix" ) func vmUnit(v *vmRecord) string { return fmt.Sprintf("nsl-%d-vm-%s.service", v.Owner, v.ID) } @@ -275,7 +276,7 @@ func (a *app) writeCredential(v *vmRecord, c *config, autostart bool) error { } func (a *app) runtimeFiles(v *vmRecord) error { - for _, p := range []string{"data.qcow2", "ssh.config", "keys/identity"} { + for _, p := range []string{"data.raw", "ssh.config", "keys/identity"} { if err := privateFile(filepath.Join(v.dir, p), a.uid, 0077); err != nil { return err } @@ -290,10 +291,13 @@ func (a *app) vmImagePath(digest string) string { return filepath.Join(a.home, "images", "vm", digest+".raw") } -// rootOverlay gives the VM a fresh root from the selected image when the +// rootGiB is the root's capacity; the root filesystem grows into it. +const rootGiB = 16 + +// ensureRoot gives the VM a fresh root from the selected image when the // image changed or no root exists. The root holds no user state. -func (a *app) rootOverlay(v *vmRecord) error { - root := filepath.Join(v.dir, "root.qcow2") +func (a *app) ensureRoot(v *vmRecord) error { + root := filepath.Join(v.dir, "root.raw") selected := v.PendingImage if selected == "" { selected = v.Image @@ -312,12 +316,10 @@ func (a *app) rootOverlay(v *vmRecord) error { if err = privateFile(image, a.uid, 0222); err != nil { return fmt.Errorf("selected VM image: %w", err) } - temporary := filepath.Join(v.dir, ".root.qcow2") + temporary := filepath.Join(v.dir, ".root.raw") os.Remove(temporary) - if err = a.call(nil, a.err, "qemu-img", "create", "-q", "-f", "qcow2", "-F", "raw", "-b", image, temporary, "16G"); err != nil { - return err - } - if err = os.Chmod(temporary, 0600); err != nil { + if err = copyRoot(image, temporary); err != nil { + os.Remove(temporary) return err } if err = os.Rename(temporary, root); err != nil { @@ -327,6 +329,85 @@ func (a *app) rootOverlay(v *vmRecord) error { return a.saveVM(v) } +// copyRoot writes a VM image to a new root file, sharing the image's extents +// where the filesystem can (btrfs, XFS) and copying its data elsewhere, then +// extends it to the root's capacity. +func copyRoot(image, path string) error { + in, err := os.Open(image) + if err != nil { + return err + } + defer in.Close() + st, err := in.Stat() + if err != nil { + return err + } + if st.Size() > rootGiB*gib { + return fmt.Errorf("the VM image is larger than the %d GiB root", rootGiB) + } + out, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return err + } + if err = unix.IoctlFileClone(int(out.Fd()), int(in.Fd())); err != nil { + err = copyData(out, in, st.Size()) + } + if err == nil { + err = out.Truncate(rootGiB * gib) + } + if err == nil { + err = out.Sync() + } + if closeErr := out.Close(); err == nil { + err = closeErr + } + return err +} + +// copyData copies in's first size bytes to the same offsets in out, a new +// file. It skips in's holes and zero blocks, which stay holes in out: most of +// a VM image's allocated blocks are zeros. +func copyData(out, in *os.File, size int64) error { + const block = 4096 + buf, zero := make([]byte, 1<<20), make([]byte, block) + isZero := func(b []byte) bool { return bytes.Equal(b, zero[:len(b)]) } + for offset := int64(0); offset < size; { + start, err := unix.Seek(int(in.Fd()), offset, unix.SEEK_DATA) + if errors.Is(err, unix.ENXIO) { + return nil // only a hole remains + } + if err != nil { + return err + } + if offset, err = unix.Seek(int(in.Fd()), start, unix.SEEK_HOLE); err != nil { + return err + } + for at := start; at < offset; { + n, err := in.ReadAt(buf[:min(int64(len(buf)), offset-at)], at) + if err != nil { + return err + } + // Write each run of blocks that are not all zeros. + for b := 0; b < n; { + for b < n && isZero(buf[b:min(b+block, n)]) { + b += block + } + run := b + for b < n && !isZero(buf[b:min(b+block, n)]) { + b += block + } + if b = min(b, n); b > run { + if _, err = out.WriteAt(buf[run:b], at+int64(run)); err != nil { + return err + } + } + } + at += int64(n) + } + } + return nil +} + // startVM launches the VM if needed and waits for authenticated readiness. A // VM that powers off while idle is started again. autostart lets the VM start // every machine; commands that only manage machines start it without. @@ -392,6 +473,9 @@ func (a *app) launchVM(v *vmRecord, state string, autostart bool) error { if err := a.requireKVMGroup(); err != nil { return fmt.Errorf("missing VM prerequisite: %w", err) } + if _, err := a.requireSystemd(); err != nil { + return fmt.Errorf("missing VM prerequisite: %w", err) + } c, err := a.loadConfig() if err != nil { return err @@ -401,7 +485,7 @@ func (a *app) launchVM(v *vmRecord, state string, autostart bool) error { return err } } - if err = a.rootOverlay(v); err != nil { + if err = a.ensureRoot(v); err != nil { return err } if err = a.writeCredential(v, c, autostart); err != nil { @@ -599,7 +683,7 @@ func (a *app) recover(args []string) error { return err } // The root holds no user state, so a fresh one repairs it. - if err = os.Remove(filepath.Join(v.dir, "root.qcow2")); err != nil && !errors.Is(err, os.ErrNotExist) { + if err = os.Remove(filepath.Join(v.dir, "root.raw")); err != nil && !errors.Is(err, os.ErrNotExist) { return err } if v.PendingImage == "" { @@ -645,20 +729,18 @@ func (a *app) devices(args []string) error { return c.Run() } -// vmFirmware selects the VM's firmware. doctor asks vmspawn to describe the -// firmware these flags select, so the check matches what launch requests. -func vmFirmware() []string { return []string{"--secure-boot=no"} } - func (a *app) launchArgs(v *vmRecord, cred *protocol.Credential) []string { - args := []string{"--user", "--no-ask-password", "--keep-unit", "--register=no", - "--image=" + filepath.Join(v.dir, "root.qcow2"), "--image-format=qcow2", "--machine=nsl-" + v.ID, + // Every flag here is one systemd 259's vmspawn accepts (ADR-0005). Without + // --user, vmspawn still runs in the user's scope: the launch namespace + // keeps the user's UID. + args := []string{"--no-ask-password", "--keep-unit", "--register=no", + "--image=" + filepath.Join(v.dir, "root.raw"), "--machine=nsl-" + v.ID, "--cpus=" + strconv.Itoa(v.CPUs), "--ram=" + strconv.Itoa(v.Memory) + "G", "--kvm=yes", "--vsock=yes", "--vsock-cid=" + strconv.FormatUint(uint64(cid(v)), 10), "--tpm=no", "--network-user-mode", "--notify-ready=no", "--pass-ssh-key=no", "--console=read-only", "--load-credential=nsl.vm:" + filepath.Join(v.dir, "nsl.vm"), - "--extra-drive=qcow2:virtio-blk:" + filepath.Join(v.dir, "data.qcow2"), + "--extra-drive=" + filepath.Join(v.dir, "data.raw"), "--secure-boot=no", "--bind-ro=" + a.machineImages() + ":" + protocol.ImageShare} - args = append(args, vmFirmware()...) // The credential names no shares for an isolated VM. for _, s := range cred.Shares { args = append(args, "--bind="+s.Source+":/mnt/host"+s.Source) @@ -720,32 +802,112 @@ func (a *app) launch(args []string) error { return syscall.Exec(bin, append([]string{bin}, a.launchArgs(v, &cred)...), env) } -// firmware returns the UEFI image vmspawn selects for launch. The check asks vmspawn -// to describe its choice, because --firmware=list succeeds even without firmware. -func (a *app) firmware() (string, error) { - b, err := a.capture(5*time.Second, "systemd-vmspawn", append([]string{"--firmware=describe"}, vmFirmware()...)...) +// minSystemd is the oldest systemd whose vmspawn accepts every launch flag +// (ADR-0005). +const minSystemd = 259 + +// vmspawnVersion returns the systemd release of the host's vmspawn, from a +// first line such as "systemd 259 (259.9-1.fc44)" or "systemd 262~rc3 (...)". +func (a *app) vmspawnVersion() (int, error) { + b, err := a.capture(5*time.Second, "systemd-vmspawn", "--version") if err != nil { - return "", err + return 0, err + } + line, _, _ := strings.Cut(string(b), "\n") + fields := strings.Fields(line) + if len(fields) < 2 || fields[0] != "systemd" { + return 0, fmt.Errorf("unexpected systemd-vmspawn version %q", line) + } + release := fields[1] + if i := strings.IndexFunc(release, func(r rune) bool { return r < '0' || r > '9' }); i >= 0 { + release = release[:i] + } + version, err := strconv.Atoi(release) + if err != nil { + return 0, fmt.Errorf("unexpected systemd-vmspawn version %q", line) } - var d struct { - Mapping struct { - Executable struct { - Filename string `json:"filename"` - } `json:"executable"` - } `json:"mapping"` + return version, nil +} + +// requireSystemd returns vmspawn's systemd release, refusing one older than +// minSystemd, whose vmspawn rejects launch's flags. +func (a *app) requireSystemd() (int, error) { + version, err := a.vmspawnVersion() + if err == nil && version < minSystemd { + err = fmt.Errorf("systemd-vmspawn is from systemd %d", version) } - if err = json.Unmarshal(b, &d); err != nil { + if err != nil { + return 0, fmt.Errorf("systemd %d or newer (%w)", minSystemd, err) + } + return version, nil +} + +// firmwareDescriptor holds the fields of a QEMU firmware descriptor that +// vmspawn selects firmware by. +type firmwareDescriptor struct { + InterfaceTypes []string `json:"interface-types"` + Mapping struct { + Executable struct { + Filename string `json:"filename"` + } `json:"executable"` + NVRAMTemplate struct { + Filename string `json:"filename"` + } `json:"nvram-template"` + } `json:"mapping"` + Targets []struct { + Architecture string `json:"architecture"` + Machines []string `json:"machines"` + } `json:"targets"` + Features []string `json:"features"` +} + +// launchable reports whether systemd 259 to 262 would all select the firmware +// for launch's --secure-boot=no: UEFI with an NVRAM template, an x86-64 q35 +// target, and neither Secure Boot nor enrolled keys. +func (d *firmwareDescriptor) launchable() bool { + if !slices.Contains(d.InterfaceTypes, "uefi") || d.Mapping.Executable.Filename == "" || d.Mapping.NVRAMTemplate.Filename == "" || + slices.Contains(d.Features, "secure-boot") || slices.Contains(d.Features, "enrolled-keys") { + return false + } + for _, t := range d.Targets { + // Targets name machine globs such as "pc-q35-*"; vmspawn runs q35. + if t.Architecture == "x86_64" && slices.ContainsFunc(t.Machines, func(m string) bool { return strings.Contains(m, "q35") }) { + return true + } + } + return false +} + +// firmware returns the UEFI image vmspawn selects for launch. vmspawn 259 +// cannot describe its choice, and --firmware=list succeeds even without +// firmware, so doctor reads the descriptors vmspawn lists, in its order, and +// returns the first launchable one. +func (a *app) firmware() (string, error) { + b, err := a.capture(5*time.Second, "systemd-vmspawn", "--firmware=list") + if err != nil { return "", err } - if d.Mapping.Executable.Filename == "" { - return "", errors.New("the firmware description names no executable") + for _, path := range strings.Split(string(b), "\n") { + if path == "" { + continue + } + f, err := os.Open(path) + if err != nil { + continue // vmspawn skips descriptors it cannot load, too + } + var d firmwareDescriptor + err = json.NewDecoder(io.LimitReader(f, 1<<20)).Decode(&d) + f.Close() + if err == nil && d.launchable() { + return d.Mapping.Executable.Filename, nil + } } - return d.Mapping.Executable.Filename, nil + return "", errors.New("vmspawn lists no x86-64 UEFI firmware without Secure Boot") } func (a *app) doctor() error { failed, vmspawn := false, false - for _, tool := range []string{"systemd-vmspawn", "systemd-run", "systemctl", "qemu-system-x86_64", "qemu-img", "ssh", "ssh-keygen", "getent", "id", a.groupSwitch, "unshare", "/usr/libexec/virtiofsd", "/usr/lib/systemd/systemd-ssh-proxy"} { + for _, tool := range []string{"systemd-vmspawn", "systemd-run", "systemctl", "qemu-system-x86_64", "ssh", "ssh-keygen", "getent", "id", a.groupSwitch, "unshare", "/usr/libexec/virtiofsd", "/usr/lib/systemd/systemd-ssh-proxy"} { p, err := exec.LookPath(tool) if err != nil { failed = true @@ -759,10 +921,17 @@ func (a *app) doctor() error { } } // Without vmspawn there is nothing to ask; its MISSING line already fails doctor. + // Its version comes first, so an old vmspawn is not mistaken for missing firmware. if vmspawn { + if version, err := a.requireSystemd(); err != nil { + failed = true + fmt.Fprintln(a.out, "MISSING", err) + } else { + fmt.Fprintf(a.out, "OK systemd %d\n", version) + } if p, err := a.firmware(); err != nil { failed = true - fmt.Fprintln(a.out, "MISSING UEFI firmware (vmspawn found no x86_64 firmware without Secure Boot; install ovmf)") + fmt.Fprintf(a.out, "MISSING UEFI firmware (%v; install ovmf)\n", err) } else { fmt.Fprintln(a.out, "OK", p) } diff --git a/vm_test.go b/vm_test.go index d4521f6..8850289 100644 --- a/vm_test.go +++ b/vm_test.go @@ -4,10 +4,12 @@ import ( "bytes" "context" "errors" + "io" "os" "path/filepath" "slices" "strings" + "syscall" "testing" ) @@ -85,58 +87,154 @@ func TestInGroupRestoresThePrimaryGroupWithEitherTool(t *testing.T) { } } -func TestDoctorChecksTheFirmwareLaunchSelects(t *testing.T) { - const describe = `{"description":"UEFI firmware for x86_64, without Secure Boot","mapping":{"device":"flash","executable":{"filename":"/usr/share/OVMF/OVMF_CODE_4M.fd","format":"raw"}}}` - const missing = "MISSING UEFI firmware (vmspawn found no x86_64 firmware without Secure Boot; install ovmf)\n" +func TestDoctorChecksSystemdAndTheFirmwareLaunchSelects(t *testing.T) { + dir := t.TempDir() + base := `{"description":"UEFI firmware","interface-types":["uefi"],"mapping":{"device":"flash","executable":{"filename":"/fw/NAME.fd","format":"raw"},"nvram-template":{"filename":"/fw/VARS.fd","format":"raw"}},"targets":[{"architecture":"x86_64","machines":["pc-i440fx-*","pc-q35-*"]}],"features":["acpi-s3"],"tags":[]}` + // descriptor writes a firmware descriptor, replacing pairs of strings in base. + descriptor := func(name string, replace ...string) string { + body := strings.ReplaceAll(base, "NAME", name) + for i := 0; i+1 < len(replace); i += 2 { + body = strings.Replace(body, replace[i], replace[i+1], 1) + } + path := filepath.Join(dir, name+".json") + if err := os.WriteFile(path, []byte(body), 0644); err != nil { + t.Fatal(err) + } + return path + } + list := func(paths ...string) string { return strings.Join(paths, "\n") + "\n" } + enrolled := descriptor("enrolled", `"acpi-s3"`, `"acpi-s3","enrolled-keys","secure-boot"`) + secure := descriptor("secure", `"acpi-s3"`, `"acpi-s3","secure-boot"`) + unusable := list(enrolled, secure, + descriptor("bios", `"uefi"`, `"bios"`), + descriptor("arm", `"x86_64"`, `"aarch64"`), + descriptor("i440fx", `,"pc-q35-*"`, ``), + descriptor("stateless", `,"nvram-template":{"filename":"/fw/VARS.fd","format":"raw"}`, ``), + descriptor("truncated", `"tags":[]}`, ``), + filepath.Join(dir, "absent.json")) + plain := descriptor("plain") + const current = "systemd 262 (262-1)" + const missing = "MISSING UEFI firmware (vmspawn lists no x86-64 UEFI firmware without Secure Boot; install ovmf)\n" for _, c := range []struct { - name string - vmspawn bool - firmware string - want, not string + name string + vmspawn bool + version, firmware string + want, not []string }{ - {"present", true, describe, "OK /usr/share/OVMF/OVMF_CODE_4M.fd\n", missing}, - {"missing", true, "", missing, "OK /usr/share/OVMF"}, - {"no vmspawn", false, describe, "MISSING systemd-vmspawn\n", "UEFI"}, // nothing to ask + {"first launchable", true, current, unusable + list(plain, descriptor("later")), []string{"OK systemd 262\n", "OK /fw/plain.fd\n"}, []string{"UEFI", "later"}}, + {"secure boot only", true, current, list(enrolled, secure), []string{missing}, []string{"OK /fw"}}, + {"no firmware", true, current, "", []string{missing}, []string{"OK /fw"}}, + {"systemd 259", true, "systemd 259 (259.9-1.fc44)", list(plain), []string{"OK systemd 259\n", "OK /fw/plain.fd\n"}, nil}, + {"systemd 258", true, "systemd 258 (258.1-1.fc43)", list(plain), []string{"MISSING systemd 259 or newer (systemd-vmspawn is from systemd 258)\n", "OK /fw/plain.fd\n"}, []string{"OK systemd"}}, + {"release candidate", true, "systemd 262~rc3 (262~rc3-1.fc45)", list(plain), []string{"OK systemd 262\n"}, nil}, + {"unknown version", true, "vmspawn 1.0", list(plain), []string{`MISSING systemd 259 or newer (unexpected systemd-vmspawn version "vmspawn 1.0")`}, []string{"OK systemd"}}, + {"no vmspawn", false, current, list(plain), []string{"MISSING systemd-vmspawn\n"}, []string{"UEFI", "OK systemd", "OK /fw"}}, // nothing to ask } { t.Run(c.name, func(t *testing.T) { a, f := testApp(t) var out bytes.Buffer - a.out, f.firmware = &out, c.firmware - dir := t.TempDir() + a.out, f.vmspawnVersion, f.firmware = &out, c.version, c.firmware + bin := t.TempDir() if c.vmspawn { - if err := os.WriteFile(filepath.Join(dir, "systemd-vmspawn"), nil, 0755); err != nil { + if err := os.WriteFile(filepath.Join(bin, "systemd-vmspawn"), nil, 0755); err != nil { t.Fatal(err) } } - t.Setenv("PATH", dir) + t.Setenv("PATH", bin) // The other host checks pass; only vmspawn answers from the fake. f.blocking = func(ctx context.Context, bin string, args []string) (bool, error) { return bin != "systemd-vmspawn" && bin != "getent" && bin != "id", nil } // The test PATH lacks the other tools, so doctor fails in every case; - // the firmware line is what differs. + // the systemd and firmware lines are what differ. if err := a.doctor(); err == nil { t.Fatal("doctor passed without its tools") } - if !strings.Contains(out.String(), c.want) || strings.Contains(out.String(), c.not) { - t.Fatalf("doctor printed:\n%s", out.String()) + for _, want := range c.want { + if !strings.Contains(out.String(), want) { + t.Fatalf("want %q in:\n%s", want, out.String()) + } } - describes := 0 + for _, not := range c.not { + if strings.Contains(out.String(), not) { + t.Fatalf("unexpected %q in:\n%s", not, out.String()) + } + } + var asked [][]string for _, call := range f.calls { if call.Bin == "systemd-vmspawn" { - describes++ - if !slices.Equal(call.Args, []string{"--firmware=describe", "--secure-boot=no"}) { - t.Fatalf("vmspawn ran with %q", call.Args) - } + asked = append(asked, call.Args) } } - if (describes == 1) != c.vmspawn { - t.Fatalf("vmspawn described firmware %d times", describes) + if want := [][]string{{"--version"}, {"--firmware=list"}}; c.vmspawn != slices.EqualFunc(asked, want, slices.Equal) || (!c.vmspawn && len(asked) != 0) { + t.Fatalf("vmspawn ran with %q", asked) } }) } } +func TestLaunchRefusesAnOldSystemdBeforeChangingState(t *testing.T) { + a, f := testApp(t) + image, digest := localImage(t, "vm image") + if err := a.execute([]string{"update", "--image", image, "--digest", digest}); err != nil { + t.Fatal(err) + } + v, err := a.loadVM() + if err != nil { + t.Fatal(err) + } + f.vmspawnVersion = "systemd 258 (258.1-1.fc43)" + err = a.launchVM(v, "inactive", true) + if err == nil || err.Error() != "missing VM prerequisite: systemd 259 or newer (systemd-vmspawn is from systemd 258)" { + t.Fatal(err) + } + if f.ran("systemd-run") != 0 { + t.Fatal("launched the VM") + } + if _, err := os.Lstat(filepath.Join(v.dir, "root.raw")); !errors.Is(err, os.ErrNotExist) { + t.Fatal("made a root:", err) + } +} + +func TestRootCopySkipsTheImagesHolesAndZeros(t *testing.T) { + dir := t.TempDir() + image := filepath.Join(dir, "image.raw") + f, err := os.Create(image) + if err != nil { + t.Fatal(err) + } + // Data, 32 MiB of written zeros, a hole, data, and a trailing hole. + f.WriteAt([]byte("first"), 0) + f.WriteAt(make([]byte, 32<<20), 4096) + f.WriteAt([]byte("second"), 64<<20+100) + f.Truncate(128 << 20) + f.Close() + root := filepath.Join(dir, "root.raw") + if err = copyRoot(image, root); err != nil { + t.Fatal(err) + } + b, _ := os.ReadFile(image) + got, err := os.Open(root) + if err != nil { + t.Fatal(err) + } + defer got.Close() + copied := make([]byte, len(b)) + if _, err = io.ReadFull(got, copied); err != nil || !bytes.Equal(copied, b) { + t.Fatal("the root differs from the image", err) + } + st, _ := got.Stat() + if st.Size() != rootGiB*gib || st.Mode().Perm() != 0600 { + t.Fatal(st.Size(), st.Mode()) + } + if allocated := st.Sys().(*syscall.Stat_t).Blocks * 512; allocated > 16<<20 { + t.Fatalf("the root allocates %d bytes; it copied the image's holes or zeros", allocated) + } + if err = copyRoot(image, root); !errors.Is(err, os.ErrExist) { + t.Fatal("replaced an existing file:", err) + } +} + func TestKVMGroupMembershipBeforeDoctorAndLaunch(t *testing.T) { for _, c := range []struct { name, group, groups, fail, want string