Stale bundled Go toolchain causes false checksum mismatch on modules with a nested vendor/ dir (x/mod/zip, golang/go#37397)

[karlkfi]

Describe the bug

Athens' bundled Go toolchain is old enough to contain a golang.org/x/mod/zip bug (golang/go#37397, fixed in Go 1.24) that strips files from a module zip when they sit inside any directory named vendor, even when that directory is not a real module vendor tree (e.g. a test fixture). Because Athens with GOPROXY=direct rebuilds the module zip locally with its bundled go, the rebuilt zip is missing that file, hashes differently from the notarized zip on sum.golang.org, and Athens rejects a valid, untampered module. The client just sees a 404.

This is a distinct root cause from the re-tag/force-push class (#2144, #1470, #1881). The two produce the identical error message but have opposite fixes, and they can be told apart with one command (see below).

Error Message

github.com/onsi/ginkgo/v2@v2.32.0: verifying module: checksum mismatch
	downloaded: h1:OPshXBha3Pavij2LzHWXX30meRBGIucLwh97bHLjLpk=
	sum.golang.org: h1:Hw7s2pVrQo/8Yz5N77qdnpHaoc+c6cC9WIV1Jce+J6E=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.

To Reproduce

github.com/onsi/ginkgo/v2@v2.32.0 declares go 1.25.0 and contains a test fixture at
integration/_fixtures/focused_with_vendor_fixture/vendor/vendored.go.

Build the module zip from VCS (exactly what Athens does with GOPROXY=direct) under two toolchains and compare the h1: hash:

$ printf 'module p\n\ngo 1.20\n' > go.mod

# Athens v0.15.1 bundles go1.20.14:
$ GOTOOLCHAIN=local GOSUMDB=off GOPROXY=direct <go1.20.14> mod download github.com/onsi/ginkgo/v2@v2.32.0
$ cat $GOMODCACHE/cache/download/.../v2.32.0.ziphash
h1:OPshXBha3Pavij2LzHWXX30meRBGIucLwh97bHLjLpk=   # == what Athens rejects

# current Go (1.24+):
$ GOTOOLCHAIN=local GOSUMDB=off GOPROXY=direct go mod download github.com/onsi/ginkgo/v2@v2.32.0
$ cat $GOMODCACHE/cache/download/.../v2.32.0.ziphash
h1:Hw7s2pVrQo/8Yz5N77qdnpHaoc+c6cC9WIV1Jce+J6E=   # == sum.golang.org

Diffing the two zips, they are byte-identical except for exactly one file, which go1.20.14 drops:

github.com/onsi/ginkgo/v2@v2.32.0/integration/_fixtures/focused_with_vendor_fixture/vendor/vendored.go

How to distinguish this from a re-tagged module (#2144 / #1470 / #1881): build the version from VCS with a current go and compare to sum.golang.org.

• Hashes match (as above) → the source is fine; the mismatch is purely Athens' old bundled toolchain. This bug.
• Hashes still differ → the VCS tag was moved after publication; nothing Athens can do but surface it. That is Verifying module: checksum mismatch #2144 (I verified compose-spec/compose-go/v2@v2.10.2 reproduces its "downloaded" hash from VCS with current Go — a genuine re-tag, not this bug).

Expected behavior

Valid public modules containing a nested vendor/ directory resolve through Athens without a checksum mismatch.

Root cause & fix

• x/mod/zip's isVendoredPackage mis-classified files directly under a vendor dir as vendored and excluded them; fixed in Go 1.24 (golang/go#37397). Athens v0.15.1 bundles go1.20.14, which predates the fix.
• Compounding it: go1.20.14 also predates go-directive enforcement / toolchain switching (Go 1.21+), so instead of erroring on a go 1.25.0 module (or switching to a ≥1.25 toolchain that has the fix), it silently proceeds and produces the short zip.
• Fix: build/base the Athens image on a current Go (≥ 1.24) and run with GOTOOLCHAIN=auto so a module's required toolchain is honored. This whole class of toolchain-skew mismatches then disappears. Related: Very slow when go.mod contains a new go version (go1.23) than in athens image (go1.22) - not caching #2000 (same "image Go older than module's go directive" skew, observed as slowness).
• Secondary UX: this failure reaches the client as a bare 404; the actual "checksum mismatch" is only in Athens' logs. Surfacing it would save a lot of diagnosis time.

Note: switching the default GOPROXY to proxy.golang.org (as I proposed in #2143) is not the right fix — it only masks this by skipping the local rebuild, breaks air-gapped operators, and does nothing for the re-tag class. Closing that PR in favor of this.

Environment

• OS: Linux 64bit
• Go version : go1.20.14 (bundled in the Athens image)
• Proxy version : v0.15.1
• Storage : fs

[matt0x6F]

Thanks for opening this issue. I was able to reproduce and validate the conditions you highlighted. Athens has already been on a greater toolchain than 1.24 for a while now. It is possible that there are poisoned packages still sitting stale in a users cache and triggering a 404.

Checksum verification is not part of the role of the proxy (or the download protocol writ large), thus the proxy has no way to signal the client (the compiler) about checksum failures. That is the role of the SumDB and thus something we cannot really patch.

My suggested fix is to write a flag that runs module verification across the stored modules. This would be a one-time clean up of poisoned modules.

An alternative is to run this verification in-flight, however, we would need to balance verification with performance and we would likely need to cache already-verified modules.

[matt0x6F]

I did some further digging and was able to nail the affected versions down to v0.12.0 to v0.15.1. Packages downloaded during those versions could be impacted.

My disposition remains the same: the best fix is one that the user can opt into. I think that's a set of flags that report on checksum mismatches with the SumDB and redownload the modules. In Kubernetes we'd run these as a Job and on a metal system we'd run it as a command.

Users that adopted Athens after v0.15.1 (eg: v0.15.2 forward) were using a Go toolchain that included the fix and thus do not need to deal with this at all.