Four metadata blocks did not list what the policy reads - #95
Merged
Conversation
`aicertify explain eu_ai_act` reported 142 declared fields and 13 measured. The playground manifest reports 150 and 14. Neither was wrong about its own source: the manifest unions an AST scan with the metadata comment block, while `explain` reads the comment block alone. The eight fields in the difference are read by the code and were never declared in the block. transparency.rego no RequiredMetrics block at all (4 fields) risk_management.rego no RequiredMetrics block at all (4 fields) robustness.rego missing system.continues_to_learn_after_deployment record_keeping.rego missing logs.sectoral_minimum_months The block is part of the spec, per CLAUDE.md, and it is the only field list a consumer reading one file can see. Both counts now agree at 150 declared and 14 measured. The union in the manifest is what hid this: a field present in either source appears in the output, so a block can rot indefinitely without any published number moving.
The blocks are a source for coverage.json, so changing them without regenerating left the derived file behind. CI's --check mode caught it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aicertify explain eu_ai_actreports 142 declared fields and 13 measured. The playground manifest reports 150 and 14.Neither is wrong about its own source. The manifest unions an AST scan with the
RequiredMetrics:comment block;explainreads the comment block alone. The eight fields in the difference are read by the code and were never declared in the block.transparency.regoRequiredMetricsblock at all (4 fields)risk_management.regoRequiredMetricsblock at all (4 fields)robustness.regosystem.continues_to_learn_after_deploymentrecord_keeping.regologs.sectoral_minimum_monthsVerified by copying the four edited policies into AICertify's pinned submodule and re-running
explain: 150 declared, 14 measured, an exact match with the manifest.The block is part of the spec per
CLAUDE.md, and it is the only field list someone reading a single policy file can see.Worth a follow-up: the union is what hid this. A field present in either source appears in the manifest, so a comment block can rot indefinitely without any published number moving. A gate comparing the block against the AST scan would catch the next one;
scripts/extract-input-fields.shalready does the extraction.