Repository navigation
ci: fuzz weekly, measure the tree's branch and condition coverage, and date the linter count - #1558
Merged
Merged
Conversation
|
Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jmrplens
changed the title
github-actions
Bot
added
the
mcp
…rrency cap .golangci.yml and the static analysis page explained run.concurrency with "across 226 packages and 55 linters", which read as a count of today's configuration. It is the measurement of July 2026 behind the cap: the module had 226 packages then, and the 55 were the 52 linters and 3 formatters the file enabled, counted together. The module has 306 packages now, and the configuration enables 52 linters, which golangci-lint linters lists, and the same 3 formatters. Both places now say when the measurement was taken and what it counted, so nothing in the tree contradicts the 52 linters the OpenSSF Best Practices answer for the warnings criterion cites. The dash in the comment goes with the sentence it was in. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…al/, and total each runner The repository measures condition coverage with gobco package by package, and had recorded a figure for three packages. The OpenSSF Best Practices gold criterion test_branch_coverage80 asks for one over the tree, measured with a FLOSS tool, and nothing could take it: the workflow measured the packages a dispatch named, one job each, and its summary added nothing up. plan takes -tree: every package the go command lists for ./cmd/... and ./internal/..., dealt round twenty shards, the first to the first shard and so on round again, so the slow packages of one directory land in different jobs, and each shard is a job on each runner named "tree NN of 20". A package list beside -tree is refused. A cell of either kind now carries the packages it measures and the name its job takes. measure is what each job runs: scripts/coverage-conditions.sh through bash over each of its packages, one after another, streaming the output to the log and leaving, per package, gobco.txt and then run.env in a directory of the records named after it. It replaces the shell that wrote one record per job, so the record's writer sits beside its reader and their tests hold each to the other. A package the script could not measure fails the job once the job's other packages are measured. summarize adds a total per runner: how many of its planned packages were measured, the outcomes gobco counted as evaluated of those there are, and that share rounded down to a tenth of a percent, so 799 of 1000 reads 79.9%. A package that was not measured is in no total, and the summary says why with the lines of its output that do: the script's own, and the panic, the failing test, the test package that did not build or set up, or the compiler position gobco's run printed before them, which is the only place the cause of the failures the tree meets is written. A figure too large for an int is refused rather than wrapped. tree is a whole -tree dispatch on the machine at hand: it lists the tree, measures each package, -parallel at a time, and writes the same summary with its system named local. make coverage-conditions-tree runs it, so the figure can be taken again without a dispatch. conditions.yml gains the tree input, names each job by the name its cell carries, runs measure in its measuring step, and bounds a shard's job at two hours while a job of a named package keeps the hour it had. The command reference, the repository map, the development guide and the trees in CLAUDE.md and the Copilot instructions describe the four subcommands. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
go test runs a fuzz target's seed corpus as one more test and varies no input unless it is given -fuzz, and nothing here gave it, so the eleven targets in five packages (FuzzMeterFor, FuzzEvaluate_IsSoundAgainstGitLab, FuzzRefuseUnreadable and the rest) only ever ran on their seeds. The OpenSSF Best Practices answer for dynamic_analysis rested on the race detector and the e2e suite for want of a tool that varies its inputs. .github/workflows/fuzz.yml runs each target with go test -run '^$' -fuzz '^<target>$', one job per target: for ten minutes on Wednesdays, which no other weekly workflow runs on, and when dispatched, and for one on a pull request that changes the workflow or its plan. GitHub offers a workflow for dispatch only once it is on the default branch, so the pull request that adds it can run it through that trigger alone; a later change could be dispatched from its branch, and the trigger runs it without anybody having to. No other pull request runs it, and it stays out of ci.yml so it gates nothing. A failing input is what go test writes under the package's testdata/fuzz/<target>, and the job uploads that directory as its artifact. The targets are not listed in the workflow. cmd/fuzz_targets lists every package with the toolchain's own go, parses the test files the go command builds for each, and prints a matrix cell per top-level function the go command reads as a fuzz target, so a target added anywhere is fuzzed from the next run on, and a plan that found none fails instead of passing with nothing fuzzed. Its tests hold the workflow to the matrix. The static analysis page gains a Fuzzing section next to the race detector, with how to reproduce a failing input and keep it as a regression seed, and the command reference, the repository map and the trees in CLAUDE.md and the Copilot instructions name the command. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
The OpenSSF Best Practices criterion test_branch_coverage80 asks what share of the branches the tests take, and every figure here came from gobco's default mode, which counts something else: every atomic condition, each operand of && and || and each comparison wherever it is written, a return statement as much as an if. gobco 1.3.4 also has -branch, which counts the whole controlling condition of each if and for and each case of a switch, and nothing else, which is what the criterion names. scripts/coverage-conditions.sh reads GOBCO_MODE: condition, the default and what every run measured before, or branch, which passes -branch to gobco and reads the report's Branch coverage figure. Any other value is refused with status 2 before gobco runs, and a report without the figure of its mode is refused as a 0/0 is. The gate reads either mode's report alike, since gobco names a branch taken one way in the words it uses for a condition. measure_conditions takes -mode on plan, summarize and tree, refused as the script would refuse it. summarize reads each record's figure in the words of its mode, so a record of the other mode is one it cannot use, and heads the summary with the mode and what its figures count; tree hands the mode to the script beside the gate. measure takes none, since the script reads the mode from the environment the job gives it. conditions.yml gains the mode input and passes it to the plan, the measuring step and the summary, and make coverage-conditions-tree takes CONDITIONS_MODE. The testing page, the command reference and the script's header say what each mode counts, and the repository map and the trees in CLAUDE.md and the Copilot instructions name the branch coverage beside the condition coverage. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
cmd/fuzz_targets is a package of its own, so CLAUDE.md counts it: 307 in the module and 334 directories holding tracked Go files, the same twenty-seven left out of go list. The testing reference that counts its test functions is regenerated once, at the top of the stack. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…tree The OpenSSF Best Practices gold criterion test_branch_coverage80 asks what share of the code's branches the tests cover, measured with a FLOSS tool, and the testing page had gobco figures for three packages, all in its condition mode. It now has the tree's in both modes, taken on 26 October 2026 on linux/amd64 with the tree subcommand four packages at a time, on this branch once its Go code was final: 265 of the 296 packages below cmd/ and internal/ measured, 44936 of their 44954 branch outcomes taken and 53412 of their 53432 condition outcomes evaluated, 99.9% each, rounded down. The eighteen branch outcomes left are eighteen decisions taken one way in nine packages, thirteen of them the error branch of a call no test makes fail, and the twenty condition outcomes are the same places and two operands no decision holds whole. The page says what each mode counts, and why a decision whose every condition went both ways has gone both ways itself, so that over the same packages the branch figure leaves out no more than the condition figure does, and gives both ways to take each figure again. The 31 packages not measured are named with the limit of gobco's each meets: twenty-seven hand symbols from an export_test.go to their external test package, three have an external test package that does not import the package, for which gobco writes its bridge with an empty import path, and internal/completions has tests that read its own source for case labels gobco's copy has rewritten, which its tests could stop doing. None of them fails under go test. The page says each share is of the packages measured, and that the tree's as a whole is not known until gobco reads those 31. The workflow section names the mode input and the artifact a shard uploads, and says what the summary quotes for a package not measured. The counts beside the measurement are corrected: the paragraph on export_test.go and scripts/check-spec-conditions.sh cite twenty-two of the 149 sub-packages of internal/tools that carry an action_specs.go, which is what that gate reads, where they carried twenty-five of 169. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…ve it #1498 renamed internal/toolutil's fuzz target FuzzLogMarkupRawOffset to FuzzScanLogMarkup_RawOffset, after the function it fuzzes, so on the rebased branch the discovery test over the repository found eleven targets and failed on the one it still named the old way. The test now names the target as the tree spells it; the discovery itself and the workflow, which list the targets rather than name them, are unchanged. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…rebase The branch and condition figures were taken on 26 October over this branch when it sat on main at ed18d7d, and the page said they were taken on the branch once its Go code was final. The branch has since been rebased onto a later main, which brought packages of its own, so the page now says which tree was measured and that the packages main gained since are in neither total. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…s copy writes it TestComplete_EveryCaseOfBothSwitchesIsDriven and TestCompletionBackingActions_StaysAlignedWithBothSurfaces read the case labels of completePromptArg and completeResourceArg out of completions.go. gobco measures the package from a copy in which a tagged switch compares the argument with each label inside a call of its own, GobcoCover(n, gobco0 == "label"), and the tests run there too, so the parser found no label that was a literal on its own, both tests failed, and the whole-tree measurement could not take the package's figures. The label is now looked for inside its expression, which reads the same labels from the source and from gobco's copy. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
… package is held to TestInitialization_PrecedesGoPackages_InEveryBinary reads the imports of every file of the package in its directory and holds them to what go/packages depends on. gobco measures the package from a copy into which it writes files of its own, gobco_fixed.go and gobco_no_testmain_test.go, which imports testing, and the test runs there too, so it failed on an import that is gobco's instrumentation and no binary of the repository links. A file named gobco_ is now left out of what it reads. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
…t read The whole-tree measurement could not measure 31 packages, and 30 of them for two limits of gobco v1.3.4, its latest release, in how it reads an external test package (package x_test). gobco type-checks that package through the source importer of go/importer, which builds the package under test from its non-test files alone, so a name the package's own test files declare for it is undefined there and resolveTypes panics before anything is measured (panic: action_specs_test.go:49:27: undefined: todos.PublishedActionIDs). That was 27 packages, each handing names over in an export_test.go (rillig/gobco#42, with the fix in review as rillig/gobco#43, upstream-bugs entry 97). And gobco bridges the external test package to the instrumented one through an import of the package it finds in the directory, so an external test package that imports nothing of the package, as awardemoji, freezeperiods and groupmarkdownuploads do, gets a bridge with an empty import path, which go test refuses at setup (rillig/gobco#33, with a fix in review as rillig/gobco#39, entry 140). scripts/coverage-conditions.sh now stages both shapes in its copy of the module. Each of the package's own test files that declares no test, benchmark, fuzz test or example becomes a file of the package, renamed without its _test suffix and opened with //go:build gc: the go command satisfies that constraint, so the file is compiled into the test binary as before and its names reach the external test package, and gobco's narrower build context does not, so gobco leaves it uninstrumented, as it leaves a test file. Any constraint of its own holds already, since the file builds here, and its lines are blanked. An external test package that imports nothing of the package is given a test file that imports it and reads one name declared for that purpose in a file of the package behind the same constraint. Neither change moves a position gobco reports. A package with neither shape is run where it is, as before. go list now also gives the package's name and import path, its own test files and what its external tests import. Those lines come before the files the package builds, since $(...) drops the newlines a listing ends with and read fails at the end of its input. Upstream-bugs entry 97 recorded the decision of 2026-10-06 to leave this to gobco. The maintainer decided on 2026-10-27 to keep this staging instead, which supersedes that decision, since the figure over the whole tree that the OpenSSF Best Practices gold criterion test_branch_coverage80 asks for has waited on a release that has not come. The staging is temporary: it goes when gobco publishes a release with the fixes of rillig/gobco#43 (for rillig/gobco#42) and rillig/gobco#39 (for rillig/gobco#33). Entry 97 records the decision and the workaround, the script and the testing page say it is temporary, and entry 140 is new. Tests first: against the previous script the two new staging cases fail on the stand-in gobco's panic and setup failure, which a third case shows the stand-in reproduces in place, and a fourth holds that an external test package gobco can read is still measured where it is. 34 tests pass. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
The whole-tree run of 26 October could not measure 31 packages, and the page said the share of the tree was not known until gobco read them. The staging of an external test package and the two test changes before this commit make every one readable, and the page now records them: measured on 27 October 2026 on linux/amd64 over this branch on main at 8f339f4, the 31 take 3773 of their 3778 branch outcomes and evaluate 4113 of their 4120 condition outcomes, 99.8% each. The five branch outcomes left are four per-action metadata decorations in internal/tools/resourceevents and a comparison with the one tool name the dispatch of internal/tools/workitems is handed; the seven condition outcomes are those five and two operands of workitems' clear guard whose decisions go both ways without them. Over all 296 packages the two runs add up to 48709 of 48732 branch outcomes and 57525 of 57552 condition outcomes, 99.9% each, which the page says is the sum of two trees measured a day apart. Signed-off-by: José M. Requena Plens <mail@jmrp.io>
|
jmrplens
added a commit
that referenced
this pull request
Oct 28, 2026
…ndency and regenerate the testing reference (#1564) Refs #1269, #1114, #1243, #1129 This is the top of the stack that carries the OpenSSF Best Practices work of #1269 and the supply-chain layers below it (#1114, #1243, #1269's Scorecard workflow and #1129's image scan). The layers below leave the committed generated artifacts stale under the freshness deferral, and this one regenerates them once, on the tree the merge puts on `main`. It also carries the fixes the review bots' threads on the layers asked for, each in a commit of its own (Review fixes, below); it changes no handler. The whole stack sits on `main` at `8f339f4e8`, onto which I rebased it on 27 October 2026, layer by layer and in order. Every commit of it is signed and carries a `Signed-off-by` trailer: the layers written before the maintainer adopted the Developer Certificate of Origin got theirs from `git rebase --signoff`. Where a layer met `main`'s configuration stack (#1518 to #1525) or the Orbit fixtures' move (#1532) the conflicts kept both sides, and generated files took `main`'s side to be regenerated here. The supply-chain stack's own regeneration, `c-c1-regen`, conflicted with `main`'s testing reference, held nothing else, and was dropped, so `issue-1129-trivy` is now the top of that part. The header rewrite of `bp-file-headers` was made again by `make fix-file-headers` on the rebased tree rather than rebased. ## What it regenerates Only the testing reference moves, and the action record gains the one file a review fix reads. The layers and the review fixes add 271 test functions, 266 under `cmd/` and five in `test/e2e/http`, 41 test files under `cmd/` and seven packages, so `docs/development/testing/testing.md` gains them in its totals, its naming breakdown, its distribution table and the `cmd` coverage table. I refreshed the counts with `go run ./cmd/gen_testing_docs/ -skip-coverage`, which carries every recorded coverage figure forward and writes `n/a` for a package no coverage run has measured. The six new packages that have statements (`cmd/audit_dco`, `cmd/audit_file_headers`, `cmd/audit_image_vulns`, `cmd/audit_site_headers`, `cmd/fuzz_targets` and `cmd/reproduce_release`) carry the 100.0% `go test -covermode=count -cover` reports for each, and `cmd/internal/fileheader`, which holds constants alone, keeps `n/a`. A second `-skip-coverage` run keeps every figure. Every other generator of `make update-all` runs on this tree and writes nothing: the brand assets, the action grants, the token footprint, the site stats, the llms files, the per-domain tool reference in both languages, the LobeHub manifest, the model corpus ledger and results page, the action catalog manifest, the benchmark charts and the e2e coverage record page, and the table formatter after them. The request inventory, recorded again over the whole unit suite on this tree, does not move (1680 rows, 976 paths, 1090 of 1090 catalog actions owned by a package that recorded a request). `go run ./cmd/audit_supply_chain -record` reads the 32 pinned actions again from GitHub and writes only the Dockerfile of `hadolint/hadolint-action`, which the review fix for #1552 reads. The package and file counts in `CLAUDE.md` match the tree: 311 packages in the module, 338 directories holding tracked Go files, 182 under `internal/tools`, 788 non-test and 488 test Go files there. ## Checks on this head With the freshness comparisons checked (`GITLAB_MCP_TEST_SNAPSHOT_PARITY` unset): - `go test ./... -count=1`: 301 packages pass, recording the request inventory as they run, and `go run ./cmd/gen_request_inventory/ -check` passes after them (1680 rows, 976 paths). The coverage job's own command passed with a total of 100.0% of statements on the head before the review fixes, and `cmd/audit_supply_chain`, the one package whose code they change, is at 100.0% of statements, 1532 of 1532 conditions and 1118 of 1118 branches under gobco, with gremlins leaving no mutant of the five files they change lived, uncovered or timed out. - Every `make` target the generated artifacts job runs, 45 of them with `check-closed-servers`, `check-supply-chain`, `check-e2e-static` and `audit-catalog-first`, passes; `check-openplugin`, `check-server-json-packages` (the eight declared packages), `check-file-headers` (3434 source files) and `check-dco` (all 73 commits from `8f339f4e8`) pass locally, since the first two need `jq` and the network and the last two read git. - `make analyze` on the remote runner: 30 of 32 steps pass. Step 6, markdownlint, has no `npx` there, and step 32, the file headers, has no `.git`; both pass locally, markdownlint over `"**/*.{md,mdx}"`. - `make test-e2e-http` (780 tests, the 8 that need Docker for a proxy or a balancer skipped) and `make test-e2e-stdio` (277 tests) pass. - The server binary and the e2e suite's compiles of the compile job, and the type-check job's vets for windows/amd64, darwin/arm64 and linux/arm64, pass. - Locally: the repository scripts' 517 Python tests, `make check-em-dash` against `main`, `make check-plan-untracked`, `make check-mcpb`, the site's `i18n:check`, `facts:check`, `chips:check`, `llms:check`, `version:check`, `build` and `lint`, and `node scripts/check-doc-links.mjs` (350 files, 8267 anchors) pass. - Every layer's head below this one passes `go build ./...`, `go vet ./...` and `go test ./... -count=1` with the freshness comparisons deferred, as CI runs them. - `make check-pr-description` passes on this text and on every layer's. ## The release rehearsals Three layers change what `release.yml` does, and each is rehearsed from its own branch once the stack is pushed: `issue-1129-trivy` for the supply-chain layers (the PyPI publish and the image scan), `bp-release-notes` for the written summary preflight requires, and `bp-reproducible-builds` for the rebuild compared byte for byte. `gh workflow run release.yml --ref <branch>` builds, smoke-tests and dry-runs every publisher from that branch and publishes nothing. Its race gate runs the unit suite without the deferral, and the packages whose tests compare a committed artifact pass that way on each of the three branches and on `bp-image-base`. The review fix to `scripts/publish-pypi.sh`, whose `--dry-run` the rehearsal's PyPI job runs, lives on this branch, so the PyPI half of the `issue-1129-trivy` rehearsal no longer exercises the script `main` will carry, and `bp-regen` is rehearsed as well for it. ## After the first CI round The first CI round of the stack asked for fixes in ten layers, each made in the layer that owns it and folded into the commit it belongs to, or added as a commit of that layer where it is new work, and every layer above rebased on it: - SonarCloud: the duplicated literals of #1550, #1552 and #1557, the nested anonymous structs of #1553, the eight-parameter functions of #1558, the tool run by its name from PATH in #1557, #1558, #1562 and #1563 (each now resolved with `exec.LookPath`, as `cmd/gen_orbit_record` does), and the test variables of #1562. - GitGuardian on #1551: the stand-in OIDC token of the PyPI publish tests is built at run time, and no commit of that layer carries the literal. - `windows-latest` on #1555: a file in the release notes directory's place is refused on every system. - The whole-tree condition measurement of #1558, which could not measure 31 packages: `scripts/coverage-conditions.sh` stages the two external test package shapes gobco v1.3.4 cannot read, two tests read the same thing in gobco's copy as in the source, and the 31 are measured and recorded. I decided on 27 October to keep that staging, which supersedes my decision of 6 October to leave the limit to gobco, until gobco publishes a release with the fixes; upstream-bugs entries 97 and 140, the script and the testing page say it is temporary. - #1561 gained the hardened mode the maintainer decided on: `--allow-plaintext-http` defaults to `true`, and `false` refuses a plaintext listener off loopback, as register row ADM-015. This head regenerates the testing reference once more for the tests that round added. On every layer's head, `go build ./...`, `go vet ./...` and `go test ./... -count=1` pass with the freshness comparisons deferred, and every package whose code that round changed is at 100.0% of statements, with every condition and branch gobco counts there taken and no mutant lived or uncovered beyond the one constant initializer of `cmd/server/listen.go` the layer below names. ## Review fixes The review bots' threads on #1550 to #1562 found ten real gaps, and I fixed each here, at the top, in a commit of its own, so no layer below changes and only this pull request runs CI again; every thread, declined ones included, is answered with the commit or with why it does not apply. In `cmd/audit_supply_chain` (#1550, #1552, #1555): a GoReleaser template is a value only the run knows and a build's overrides and a flags string are read, an absolute `--config`, `dir` or `dist` stays absolute and is refused, a tag-named image is refused in a job whose token neither it nor its workflow declares, the images a container action's Dockerfile builds from are held to their digest, read as BuildKit binds the file's stages, parser directives and heredocs (the record gains `hadolint/hadolint-action`'s Dockerfile, whose base is pinned), a release note crediting an assistant as its author is refused, and release note headings and fences are read as GitHub renders them. `TestClient_StatelessAnswersCarryNoSession` holds the stateless initialize to a JSON-RPC result (#1550), `scripts/publish-pypi.sh` publishes exactly the six platform wheels, each a regular file (#1551), the install pages say `make check-repeatable` does not build the universal macOS binary (#1557), and the Code of Conduct offers the OpenPGP key for a report sent by email (#1560). Each testable fix came with a test that failed before it, `cmd/audit_supply_chain` stays at 100.0% of statements, and the testing reference is refreshed for the seven test functions they add. ## Dependencies This layer also moves every dependency to its newest release, with no cooldown, which fixes this PR's CI failure: golang.org/x/net v0.62.0 fixes GO-2026-6860 and GO-2026-6866 in x/net/html, and check-binary-vulns now reports no advisory on the six binaries. The other Go moves are tiktoken-go/tokenizer v0.10.0 (whose Codec interface gained two methods that the cmd/audit_tokens test doubles now implement), gqlparser v2.5.64, grpc v1.85.0, grpc-gateway v2.33.0 and their indirect dependencies. The builder image moves to the rebuilt golang:1.27.3-alpine3.24 digest. Five actions move: setup-python v7.1.0, markdownlint-cli2-action v25.0.0, login-action v4.7.0, codeql-action v4.39.1 and sonarqube-scan-action v8.4.0, with pinned-actions.json re-recorded. The release tools move to GoReleaser v2.19.0, cosign v3.1.4, oras 1.3.6 and mcp-publisher v1.8.6, the NuGet validation to the .NET 10.0.403 SDK (both dnx facts the install page relies on re-verified), and the npm validation and publishing to Node 24. The PyPI upload's lock is recompiled (platformdirs 4.13.7, pyasn1 0.6.5). The site moves to pnpm 12.16.1, Node 24.22.1 and the patch releases of astro, starlight, markdown-remark, @astrojs/check and typescript-eslint. The telemetry example moves to collector 0.164.0 and otel-lgtm 0.38.1, the balancer e2e to HAProxy 3.4, and the e2e Bitbucket fixture to 10.5, which now starts with basic authentication force-allowed: Bitbucket 10 blocks it by default, and both the fixture's setup script and GitLab's importer authenticate with it. Left out on purpose: client-go stays at v3.30.0 because it moves in its own stack (the server's major tracks it, and a bump moves the served surface and the fine-grained table); the GitLab CE/EE images stay pinned because 19.5 is still GitLab's newest release and the fine-grained table records it; Node 26 is not taken because no 26.x is LTS and it drops Corepack; TypeScript 7 is refused by the peer ranges of @astrojs/check and typescript-eslint. The release tools, the SDK and the builder image were rehearsed with `release.yml` on this branch before merging. ## Before this merges - **#1517.** It is the outside contribution open on 27 October, based on `main` at `e3e09d290`. Its 18 changed source files merge with the top of this stack without a conflict; the file it adds has no header and its six commits no sign-off, which the header and DCO layers say how to handle. Milestone: 3.4.0. --------- Signed-off-by: José M. Requena Plens <mail@jmrp.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1542
Refs #1269
Three answers of the OpenSSF Best Practices form rested on evidence weaker than they claimed. dynamic_analysis had the race detector and the e2e suite but no tool that varies its inputs: the module carries eleven fuzz targets and
go testonly ever ran their seed corpora. test_branch_coverage80 asks for the share of the code's branches the suite covers, measured with a FLOSS tool, and the repository had gobco figures for three packages, none for the tree, and none in the mode that counts branches. warnings cites the linters the configuration enables, and.golangci.ymland the static analysis page said 55 where it enables 52. This pull request gives each of the three the evidence it lacked; whether each answer becomes Met is the maintainer's call, under Before this merges..github/workflows/fuzz.ymlfuzzes every target withgo test -run '^$' -fuzz '^<target>$', one job per target: for ten minutes on Wednesdays at 05:43 UTC, a day no other weekly workflow takes, and when dispatched, and for one minute on a pull request that changes the workflow or its plan. GitHub offers a workflow for dispatch only once its file is on the default branch, so this pull request can runfuzz.ymlthrough that trigger alone; a later change could be dispatched from its branch, and the trigger runs it without anybody having to. No other pull request runs it, and it stays out ofci.yml, so it gates nothing. When a target fails,go testwrites the input under the package'stestdata/fuzz/<target>/, and the job uploads that directory asfuzz-<package>-<target>. The static analysis page gains a Fuzzing section beside the race detector, with how to reproduce such an input and keep it as a regression seed.The targets are not written into the workflow.
cmd/fuzz_targetslists every package with the toolchain's own go, parses the internal and external test files the go command builds for each, and prints a matrix cell per top-level function the go command reads as a fuzz target (Fuzz, orFuzzfollowed by anything but a lower-case letter, soFuzzyis not one, and a method is not either). A target added anywhere is fuzzed from the next run on, and a plan that finds none fails rather than passing with nothing fuzzed. Its tests hold the workflow to the matrix: the triggers, the read-only token, the anchored-fuzz, the fuzz time per trigger and the path the upload takes.cmd/measure_conditionstakes the whole tree.plan -treelists every package of./cmd/...and./internal/...and deals them round twenty shards, the first package to the first shard, the second to the second and so on round again, so the slow packages of one directory land in different jobs; each shard is a job on each runner. The shell that wrote one record per job is now themeasuresubcommand, which runsscripts/coverage-conditions.shover each package of its job one after another, streams the output to the log and leaves each package'sgobco.txtandrun.envin a directory of its own, so the writer of a record sits beside its reader and their tests hold each to the other.summarizeadds each runner's total over the packages it measured, with the share rounded down to a tenth of a percent so it never reads higher than it is (799 of 1000 is 79.9%); a package that was not measured is in no total, and the summary quotes the lines of its output that say why: the script's own, and the panic, the failing test, the test package that did not build or set up, or the compiler position gobco's run printed before them.treeruns a whole-treedispatch on the machine at hand and writes the same summary, andmake coverage-conditions-treeruns it, so the figures can be taken again without a dispatch.conditions.ymlgains thetreeinput, names each job after its package or shard, and gives a shard's job two hours while a job of a named package keeps the hour it had.gobco counts in one of two modes, and every figure here had come from its default, which counts every atomic condition: each operand of
&&and||and each comparison wherever it is written, areturnstatement as much as anif. That is not what the criterion names. gobco 1.3.4 also has-branch, which counts the whole controlling condition of eachifandforand each case of aswitch, and nothing else.scripts/coverage-conditions.shnow readsGOBCO_MODE,conditionby default orbranch, refuses any other value before gobco runs, and refuses a report without the figure of its mode as it refuses a0/0.measure_conditionstakes-modeonplan,summarizeandtree, reads each record's figure in the words of its mode and heads the summary with it;conditions.ymlgains themodeinput and passes it to the plan, the measuring step and the summary, andmake coverage-conditions-treetakesCONDITIONS_MODE.I took both figures with that command on 26 October 2026, on linux/amd64, on this branch once its Go code was final, when it sat on
mainated18d7d12, four packages at a time on ten cores of a machine other work was loading too. The stack has since been rebased onto a latermain, which brought packages of its own, so the page says which tree was measured and that those packages are in neither total; the only Go change on the branch since is the fuzz target name below. Branch coverage: 265 of the 296 packages belowcmd/andinternal/measured, 44936 of their 44954 outcomes taken, 99.9% rounded down, in an hour and twelve minutes. Condition coverage: the same 265, 53412 of 53432 outcomes evaluated, 99.9%, in an hour and twenty-four minutes.docs/development/testing/README.mdrecords both with the date and both ways to take them again, and says what each mode counts and why the branch figure can leave out no more outcomes than the condition figure over the same packages: Go stops evaluating&&and||once the outcome is known, so a decision whose every condition went both ways has gone both ways itself.The eighteen branch outcomes left are eighteen decisions taken one way in nine packages. Thirteen are the error branch of a call no test makes fail (three on a random read, two on a request build, a stat, a write, a schema build, a catalog build, a marshal, an encode, an unmarshal and a marshal back), two are per-action metadata decorations of the kind
make check-spec-conditionsexists to find, and three are cases no test reaches: a decoding with no candidate to try, a URL elicitation in a session negotiated at 2026-07-28 or later, and an input schema without itsvalueproperty. The twenty condition outcomes left are the same places, the elicitation one at its second operand, and two operands incmd/audit_edition_tierthat no decision holds whole.That run could not measure 31 packages, and none of them fails under
go test: twenty-seven hand symbols from anexport_test.goto their external test package, which gobco type-checks against the non-test files alone (rillig/gobco#42); three have an external test package that does not import the package, for which gobco writes its bridge with an empty import path (rillig/gobco#33); and two have tests that read, in gobco's copy, what gobco rewrote there:internal/completionsthe case labels of its switches, andinternal/testutil/serialtypecheckthe imports of every file of its package, gobco's own among them. gobco v1.3.4 is still its latest release and fixes neither, soscripts/coverage-conditions.shnow stages both shapes, and the two tests read the same thing in the copy as in the source (After the first CI round, below). Measured that way on 27 October, on the rebased branch, the 31 take 3773 of their 3778 branch outcomes and evaluate 4113 of their 4120 condition outcomes, so all 296 packages are measured, 48709 of 48732 branch outcomes and 57525 of 57552 condition outcomes over both runs, 99.9% each, from two trees a day apart; the page says so. The counts beside it are corrected too: twenty-two of the 149 sub-packages ofinternal/toolsthat carry anaction_specs.gowere in theexport_test.gostate, where the page andscripts/check-spec-conditions.shsaid twenty-five of 169..golangci.ymland the static analysis page now say what the sentence behind the concurrency cap measured and when: July 2026, when the module had 226 packages and the configuration enabled 52 linters and 3 formatters, which the 55 had counted together. Today it enables the same 52 and 3 (golangci-lint linterson v2.15.0), andgit grep -n "55 linters"finds nothing.How I checked it
cmd/fuzz_targets: its eleven tests failed against stubs before the command existed, and pass with 100.0% of statements;GOBCO_GATE=all bash scripts/coverage-conditions.sh ./cmd/fuzz_targetsreads 40/40, andmake coverage-mutants PKG=./cmd/fuzz_targetskilled 24 mutants with none lived, uncovered or timed out.go run ./cmd/fuzz_targetson this tree plans the eleven targets in five packages.cmd/measure_conditions, the tree subcommands: their tests were written first, but what I saw fail before the code existed was the build (undefined: treePackages,runMeasure,measurePackage,runScriptand the rest), not an assertion, so the proof that those tests decide the behavior is the mutation run below rather than a red run.cmd/measure_conditions, the reasons a summary gives, the job bound and the mode: those tests were written first and run against the code before them, with stubs for the new signatures where the mode needed them, and failed on their assertions (every new case of the mode, the four new shapes of a reason, and the measure job's bound read as 120 where the test wants the expression). The package passes with 100.0% of statements; gobco reads 270/270 in condition mode and 226/226 in branch mode; gremlins killed 166 mutants with none lived, uncovered or timed out.scripts/coverage-conditions.sh: four new cases inscripts/coverage_conditions_sh_test.py(the mode reaching gobco as-branchbefore the tags and the test flags, staged or not; a value it cannot read refused before anything runs; a report without the figure of its mode refused; the gate reading a branch report) failed thirteen subtests against the script before the mode, and the whole suite, thirty tests, passes, as doscripts/check_spec_conditions_sh_test.pyand shellcheck on both scripts.go test -fuzzfor one minute, as the pull request trigger runs them (cmd/server's in 143 s with the build of its instrumented binary, the others in 60 to 72 s).go test ./... -count=1passed at the head before the rebase with the freshness comparisons on and with them deferred, and passes at the rebased head with them deferred, as CI runs a layer below the top. After the rebase it first failed incmd/fuzz_targets: Read a timestamped job log as GitLab's viewer does: no header, and a line written in pieces whole again #1498 renamedinternal/toolutil's target toFuzzScanLogMarkup_RawOffset, and the discovery test named it the old way, so the test names it as the tree spells it now. golangci-lint is clean on both commands, actionlint v1.7.7 acceptsconditions.ymlandfuzz.yml(the measure job's bound is the expression${{ inputs.tree && 120 || 60 }}), andmake check-supply-chain(the new workflow's actions are pinned to the commits the others use),check-test-file-names,check-test-subtests,check-test-goroutines,check-godocs,check-dead-consts,check-doc-tool-names,check-tenancy,check-md-escaping,check-sdk-context,check-action-idsandcheck-llmspass.go run ./cmd/gen_testing_docs/ --checkandcheck-site-statspass, and so donode scripts/check-doc-links.mjs, markdownlint on every changed Markdown file, the table formatter's--check,make check-em-dashover this layer andmake check-pr-descriptionover this title and description.Before this merges
These are the maintainer's, and nothing above claims them:
fuzz.ymlthrough its paths trigger, eleven jobs of one minute each, which should pass; the first ten-minute run is the first Wednesday after the merge, or a dispatch then.conditions.ymlis already onmain, sogh workflow run conditions.yml --ref bp-analysis -f tree=true -f systems=ubuntu-latest -f gate=all -f mode=branch, and the same with-f mode=condition, should run this branch's version. Its first dispatch, before the fixes below (runs 40467477742 and 40467482058), failed sixteen shards each on the 31 packages gobco could not read; the next should measure all 296, and a shard fails only where a package's figure leaves outcomes untaken undergate=all, which is a measurement, not a failure to measure.fuzz.ymland its first scheduled run, and warnings the 52 linters of.golangci.yml. For test_branch_coverage80 the evidence is the branch coverage of 99.9% over all 296 packages, from the two runs above; whether that answers the criterion as Met is the maintainer's judgment.cmd/audit_edition_tierconditions beside them, and the five ininternal/tools/resourceeventsandinternal/tools/workitemsthe second run found, with the two operands of theworkitemsclear guard beside them, each a test case or a declaration.cmd/fuzz_targetsrow the 100.0%go test -coverreports.On the rebased stack
I rebased the stack onto
mainat8f339f4e8on 27 October 2026. The checks above ran on this layer before that rebase, except where they say otherwise. On the rebased head, with the two commits the rebase called for (the fuzz target's new name and the dated measurement),go build ./...,go vet ./...andgo test ./... -count=1pass with the freshness comparisons deferred, as CI runs every layer below the top, and every commit carries aSigned-off-bytrailer. The committed generated artifacts are regenerated once at the top of the stack, which runs every gate over the combined tree.After the first CI round
runScriptranbashby its name (S4036), and now resolves it withexec.LookPathfirst and runs the absolute path, ascmd/gen_orbit_recordandcmd/gen_model_resultsalready run git; a PATH holding no bash is reported as the lookup's failure, which a new test holds.measurePackageandmeasureAlltook eight parameters each (S107); the runner, the module root, the system, the directory of records and the environment are now onemeasurervalue. Both are folded into the commit that added the subcommands.cmd/measure_conditionsstays at 100.0% of statements, 272 of 272 conditions and 228 of 228 branches, and gremlins kills the 38 mutants ofmeasure.goandtree.gowith none lived or uncovered.scripts/coverage-conditions.shnow stages a package with an external test package that gobco cannot read: each of the package's own test files that declares no test, benchmark, fuzz test or example becomes a file of the package in the copy, renamed without its_testsuffix and opened with//go:build gc, which the go command satisfies and gobco's narrower build context does not, so it is compiled into the test binary as before, its names reach the external tests, and gobco leaves it uninstrumented, as it leaves a test file; and an external test package that imports nothing of the package is given a test file that imports it, beside the one name it reads in a file of the package behind the same constraint. No position gobco reports moves, and a package with neither shape is still measured where it is.internal/completions's tests now look for a case label inside its expression, which reads the same labels from gobco'sGobcoCover(n, gobco0 == "label"), andinternal/testutil/serialtypecheck's test leaves gobco's owngobco_*.gofiles out of the imports it holds the package to. Of four new cases inscripts/coverage_conditions_sh_test.py, the two staging cases fail against the previous script, on a stand-in gobco that panics on the undefined name and fails setup on the bridge as gobco does, a third shows the stand-in fails that way on the package in place, and a fourth holds that an external test package gobco can read is still measured where it is; its 34 tests pass. On 27 October 2026 I decided to keep this staging, which supersedes my decision of 6 October, recorded in upstream-bugs entry 97 and issue 1099, to leave theexport_test.golimit to gobco with no workaround in the staging script. The staging is temporary: it goes when gobco publishes a release with the fixes of Type-check black box tests as go test builds them rillig/gobco#43 (for A black box test that uses a name from export_test.go makes gobco panic with "undefined" rillig/gobco#42) and Resolve black box test import path from go.mod rillig/gobco#39 (for Finding original package path in directory with_testpackage may fails rillig/gobco#33). Entry 97 records that decision and the workaround, entry 140 is new, for the bridge, and the script, the Makefile and the testing page say the staging is temporary.