fix(counts): one manifest for every published figure, and the education package that delivered nothing - #96
Merged
Merged
Conversation
…fted The README states that the published figures are generated from the tree and checked in CI, so they cannot drift from the code. Two classes of figure were outside that guarantee. The count of Rego files including tests was hand-maintained. It read 196 while the tree held 198, in the same sentence that claims it cannot drift. It was not in coverage.json at all, so no check could reach it. generate-coverage.sh now counts the test files with the same predicate it uses for policies and emits `rego_files_including_tests`, and check-version-refs.sh enforces it. The translated front pages were never checked. All four said 91 policies, the previous count, including in the alt text for a diagram that renders 92. A reader arriving in Chinese, Japanese, Korean or Hindi was given a different number than a reader arriving in English, and the alt text is the string a screen reader announces, so nobody looking at the page would see the disagreement. The English check script already carries a comment about numbers in alt text drifting for exactly this reason; the translations had the bug it describes. AGENTS.md and CLAUDE.md said 85 policies and 124 files against an actual 92 and 198. These are read by Claude Code and Codex before either touches the tree, so a stale count there propagates into new work. Corrected: README 196 to 198; four i18n READMEs 91 to 92 and 196 to 198; AGENTS.md 85/124 to 92/198; CLAUDE.md 85 to 92. Each is now enforced by check-version-refs.sh, which runs in CI. A count failure also reported itself as a version failure, naming VERSION in the error while the disagreement was in the coverage data. The message now names both figures. Verification: check-version-refs.sh passes on 15 enforced references, and fails with a named surface when AGENTS.md is reverted to 85. opa check, regal lint (198 files, no violations), generate-coverage.sh --check, check-deprecations, check-anchors and check-test-coverage all pass.
…assed
equitable_admissions_systems.rego and unbiased_automated_grading.rego share the
package industry_specific.education.v1.fairness_and_equity, and each defined a
complete rule named `thresholds` with a different value: the admissions policy
using demographic_parity and equalized_odds, the grading policy using
equal_opportunity_difference, average_odds_difference and disparate_impact.
A complete rule that produces multiple outputs is an evaluation-time conflict.
OPA raises
eval_conflict_error: complete rules must not produce multiple outputs
and fails the entire package. A consumer querying
data.industry_specific.education.v1.fairness_and_equity receives nothing, so all
three policies in that package, including digital_divide_mitigation which had no
part in the collision, delivered no verdict at all. Measured against AICertify's
demo contract, the education framework returned 4 verdicts where 5 packages
declare one.
Neither existing gate could see it. `opa check` is static, and the conflict is
an evaluation-time condition. `opa test` queries individual rules, and the two
definitions only conflict when evaluated together, which is what a query for the
package itself does. Both passed: opa check clean, opa test 842/842.
The gap is between how this library tests itself and how it is used. Tests ask
for one rule; a consumer asks for the package. scripts/check-eval-conflicts.sh
now asks each of the 85 policy packages the consumer's question and fails on an
evaluation-time error, and runs in CI.
`is_within_threshold` collided the same way and was not raising an error. Rego
merges same-name, same-arity functions in a package as alternative definitions,
so an admissions metric could be judged by a grading policy's rule. Both are now
named for the policy that owns them.
Verification: the package evaluates and returns both threshold sets. opa check
passes, regal lint reports no violations across 198 files, opa test passes
843/843 including a new test that references both threshold sets in one
expression, which is what forces them to be evaluated together. Reintroducing
the collision fails check-eval-conflicts.sh with the offending package and line
named, while opa check and opa test both still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits. The first closes the count-drift gap; the second fixes three policies that reached no verdict while every gate passed.
Counts generated from one manifest
The README states that published figures are generated from the tree and checked in CI, so they cannot drift. Two classes of figure sat outside that guarantee.
The file total was never in the manifest. "196 Rego files including tests" was hand-maintained against an actual 198, inside the sentence claiming it cannot drift.
coverage.jsonhad no field for it, so no check could reach it.generate-coverage.shnow counts test files with the same predicate it uses for policies and emitsrego_files_including_tests.The translated front pages were never checked. All four said 91 policies, including in the alt text for a diagram rendering 92 — the string a screen reader announces, so nobody looking at the page would see the disagreement.
check-version-refs.shalready carries a comment about alt-text drift for exactly this reason; the translations had the bug it describes.AGENTS.mdsaid 85 policies and 124 files against 92 and 198, andCLAUDE.mdsaid 85. Claude Code and Codex read these before touching the tree, so a stale count propagates into new work.15 references are now enforced. Reverting
AGENTS.mdto 85 fails the check by name.Three education policies delivered no verdict
equitable_admissions_systems.regoandunbiased_automated_grading.regoshare the packageindustry_specific.education.v1.fairness_and_equity, and each defined a complete rule namedthresholdswith a different value. OPA raiseswhich fails the entire package. All three policies in it returned nothing, including
digital_divide_mitigation, which had no part in the collision. Measured against AICertify's demo contract, education returned 4 verdicts where 5 packages declare one.Neither existing gate could see it.
opa checkis static and the conflict is an evaluation-time condition.opa testqueries individual rules, and the two definitions only conflict when evaluated together — which is what a query for the package does. Both passed:opa checkclean,opa test842/842.The gap is between how this library tests itself and how it is used. Tests ask for one rule; a consumer asks for
data.<package>.scripts/check-eval-conflicts.shnow asks each of the 85 policy packages the consumer's question, and runs in CI.is_within_thresholdcollided the same way without raising an error: Rego merges same-name, same-arity functions in a package as alternative definitions, so an admissions metric could be judged by a grading policy's rule. Both are now named for the policy that owns them.Verification
opa checkpasses;regal lintreports no violations across 198 filesopa test843/843, including a new test that references both threshold sets in one expression, which is what forces them to be evaluated togethercheck-version-refs.shpasses on 15 enforced references and fails by name when one is revertedcheck-eval-conflicts.shwith the package and line named, whileopa checkandopa testboth still pass