Skip to content

Image publishes have failed since 2026-09-08: Cargo.lock on main is unusable with --locked #155

Closed

Description

@juan-malbeclabs

Every GHCR publish of doublezero-edge-connect has failed since 2026-09-08. As a result ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta is still the 2026-09-07 build, pinned to the 0.38.0 client base — it missed the doublezero v0.39.0 release and the doublezero:mainnet-beta-0.39.0-1 base image published on 2026-09-10.

The fix is already open and green: #151. It has been waiting on review since 2026-09-07.

Symptom

Both the event-driven rebuild and the daily poll backstop fail in the Docker build step:

#20 0.570 error: cannot update the lock file /src/Cargo.lock because --locked was passed to prevent this
ERROR: failed to build: failed to solve: process "/bin/sh -c cargo build --release --locked -p doublezero-edge-connect ..." did not complete successfully: exit code: 101

Failing runs (all the same error):

When Workflow Run
2026-09-08 14:16 dispatch 34237239761
2026-09-09 05:30 poll 34315124137
2026-09-10 05:30 poll 34441265726
2026-09-10 05:43 dispatch (mainnet-beta 0.39.0) 34442208828
2026-09-10 15:10 dispatch 34493872900

The upstream side is healthy — releaser.client for client/v0.39.0, the doublezero base image publishes and the cross-repo repository_dispatch all succeeded. The failure is entirely here.

Root cause

Six Dependabot PRs merged into main within about 30 seconds on 2026-09-07 without being rebased onto one another:

Each lock was correct against the base it was resolved against, and git merged them line by line with no conflict. The result on main (e258968) is a lock Cargo cannot use:

  • 2 ambiguous dependency refs — rand 0.9.4 and rand_chacha 0.9.0 name a bare "rand_core" while both rand_core 0.9.5 and rand_core 0.10.1 are locked. Cargo cannot resolve a bare name against two versions, so it must rewrite the lock, which --locked forbids.
  • 6 orphan entries reachable from nothing, resurrected by the merge: block-buffer 0.10.4, cpufeatures 0.2.17, crypto-common 0.1.7, digest 0.10.7, generic-array 0.14.7, version_check 0.9.5.

Reproduces at e258968:

$ cargo metadata --locked --format-version 1
error: cannot update the lock file ... because --locked was passed to prevent this

Why CI didn't catch it

.github/workflows/rust.yml runs cargo build/clippy/test --workspace without --locked, so it silently re-resolves and stays green. The Dockerfile's cargo build --release --locked (Dockerfile:50) is the only enforcement — so the first symptom is a failed image publish days later, not a red CI run.

Fix

  • deps: pin past the yanked chacha20 0.10.1 #151 — merge this. It regenerates the lock (removes exactly the six orphans, qualifies the two bare rand_core refs as rand_core 0.9.5) and bumps chacha20 0.10.1 → 0.10.2, the yanked crate that is currently failing cargo-deny on every PR in this repo. Verified at its head 91e8ec5: 0 ambiguous refs, 0 orphans. Green since 2026-09-07, REVIEW_REQUIRED. Merging it triggers release.docker.edge-connect.yml, which republishes all three variants against the current bases.
  • ci: build, lint and test with --locked #157 — add --locked to the three cargo invocations in rust.yml, so a lock Cargo would have to rewrite fails the PR that introduces it instead of a release. Merge after deps: pin past the yanked chacha20 0.10.1 #151.

Neither closes the underlying hole: two Dependabot PRs whose locks are each valid against their own base still merge cleanly into an invalid one. Requiring "branches must be up to date before merging" (or a merge queue) on main is the branch-protection setting that prevents it, and needs someone with repo admin.

Verification once #151 lands

docker buildx imagetools inspect ghcr.io/malbeclabs/doublezero:mainnet-beta-0.39.0-1 \
  --format '{{ .Manifest.Digest }}'
docker buildx imagetools inspect ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta \
  --format '{{ index .Image.Config.Labels "org.opencontainers.image.base.digest" }}'
# the two digests must match

Activity

  1. juan-malbeclabs commented on Sep 18, 2026

    @juan-malbeclabs
    ContributorAuthor

    Resolved by #151, which regenerated the lock file, and #157, which put --locked in front of build, lint and test so the lock and the manifest cannot drift apart again without a red check before the publish rather than inside it.

    Publishes have been green since: release.docker.edge-connect on main succeeded on 2026-09-11 and again on 2026-09-18, and the mainnet-beta tag in GHCR was last pushed 2026-09-18 15:17 — so the image is no longer the 2026-09-07 build pinned to the 0.38.0 client base. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions