Audit scripts/test.sh against the test tree: four coverage gaps in two days, each of which hid a real defect

[lmoresi]

CI does not run all the tests, and the gaps are not random — every one found in the last two days was hiding a defect that the missing test would have caught.

The four

1. The tier-A gate does not cover the solver batches. pytest -m "level_1 and tier_a" was green while test_1005_TransientDarcyCartesian failed on CI (PR A rank with no cells must answer every global mesh query like its peers (#405) #557) — a TypeError from a 0-d ndarray reaching a sympy comparison. Local gate green is not CI green, and we treated it as if it were.
2. test_1018_* matched no batch glob — the entire rotated free-slip suite, including the Rotated-path field copy-back drops inhomogeneous essential (Dirichlet) values #497 regression guard. Fixed as CI: the rotated free-slip suite (test_1018_*) is in no scripts/test.sh batch — its regression tests never run #504 (PR Quickfix batch: clone() arguments (#498), print_table keywords (#499), and 56 orphaned test files rejoined to CI (#504) #532), which also found 56 orphaned files and an inline comment that swallowed a batch's || status=1 so its failures could never fail CI.
3. tests/parallel/test_10*py was commented out in scripts/test.sh, so test_1017 and test_1062–test_1068 had executed at no rank count since they were written. Enabled in PR Rotated free-slip: weight the nodal normal by the facet measure the assembly integrates over (#560) #561; it immediately surfaced seven partition-independence failures (Constrained free-slip is partition-dependent (3.4%): boundary normals accumulated rank-locally — and the rotated tests compare against a golden recorded on another host #564), one of them a genuine 3.4% defect.
4. test_106* is deliberately unbatched — and that is where the serial default-normal consumers live. It is the one suite that would have caught the Complete the boundary-normal sum across ranks, and make the partition-independence tests self-referential (#564) #568 review's B1: mesh.boundary_normal() returning normals wrong by ≈5.1° on a 3-D spherical shell in serial.

The ask

A one-off audit of scripts/test.sh against the actual tree, and a mechanism so it cannot drift again:

• enumerate every tests/**/test_*.py and every tests/parallel/** file, and map it to the batch that runs it (or to an explicit, commented decision not to);
• for each file that is deliberately excluded, state the reason in the script itself — runtime, known-slow, requires a resource — so an exclusion is a decision rather than an accident;
• add a check (a test, or a step in the workflow) that FAILS when a test file matches no batch and carries no exclusion marker. Without that, this recurs — three of the four above were introduced by someone adding files and not noticing the globs;
• audit the batch lines for swallowed exit codes (the CI: the rotated free-slip suite (test_1018_*) is in no scripts/test.sh batch — its regression tests never run #504 instance) so a failing batch cannot pass silently.

Why this is worth doing properly

Each of the four gaps cost a defect reaching development or surviving there — a silent-wrong-flux class, a 3.4% partition dependence, a 5° normal error on spherical shells, and a crash. The tests to catch all four already existed. They just never ran.

Underworld development team with AI support from Claude Code