feat: a lab, a blind benchmark with an auditable trail, and the gate that runs it - #9
Open
darklordVirtual wants to merge 4 commits into
Open
feat: a lab, a blind benchmark with an auditable trail, and the gate that runs it#9darklordVirtual wants to merge 4 commits into
darklordVirtual wants to merge 4 commits into
Conversation
"Built on REMORA, consumed as a hash-pinned release rather than a dependency on its master" was the second sentence a reader met. It is jargon before the reader knows what the product is, and it restated what the REMORA integration section said a screen further down — the same fact in two places, which is how two places start disagreeing. The opening is plain again: "Powered by REMORA." docs/pinned-core.md is now the one home for it: what is pinned and why, the current release, commit and version, how verification works and what each of the three checks protects against, why the text digests are content digests rather than byte digests, the upgrade procedure, and the one authoring dependency the pin does not cover. The README's integration section is three lines and a link. operations.md's upgrade section points at the same document instead of restating it. ## The document is bound to the lock A reference that quotes a commit hash goes stale the moment the pin moves, and a stale reference is worse than none because it looks authoritative. test_pin_manifest_agreement.py now asserts the document names the current release, commit, version and wheel, every pinned artifact, and the prerelease status. Verified it can fail: zeroing the commit in the document turns the suite red on exactly that assertion, and restoring it turns it green. A pin bump that forgets the document now fails in CI rather than misleading the next reader. README is 172 lines. 105 contract tests pass.
…that runs it Four things, and the last one is why the first three are worth having. Supply chain. Base images are digest-pinned; Python dependencies are exact versions in docker/requirements.lock used as a constraint set. The databases sit on an `internal: true` network — measured, not assumed: control-plane-db has no default route, and the three containers that publish a port do have egress, which docs/security-model.md now states per container instead of claiming blanket isolation. Console. The ledger reads the signed audit chain directly on a SELECT-only credential, so it shows the governance record rather than a health banner and four work orders while 165 assessments sat unread one network away. Assurance moved from a page into a strip, because it qualifies everything else. Found and fixed: the drawer was permanently open ([hidden] lost to .drawer on source order), and every inline style was silently dropped by the CSP — the DOM keeps the attribute, the style never applies, no console error. Contrast measured in both modes for the first time; three tokens were below AA and are now above. Lab. A separate service, image and port that holds every role token and can act. Separate because the console reports `console_access: read-only` about itself, and a process that can approve its own proposals cannot credibly say that. Choosing a role selects a credential and grants nothing: the operator attempting to approve gets 403 from the control plane, and there is a test asserting the refusal arrives in the engine's words. Benchmark. Scenarios in benchmarks/suites/, answers in benchmarks/keys/, and the prediction pass never opens a key — enforced by a test that makes load_key throw. Keys are sealed to a suite digest; a broken seal is not scored at all, because grading changed questions against old answers is a number with no meaning. Every run records the chain positions it created, and those can be re-read and confirmed: `run.py bench-verify`. Forging one entry hash is detected. Writing the answers first paid for itself immediately. A key predicted that a value outside a declared `enum` would be refused; the engine assessed it. Probing every clause gave the real shape — structural clauses (required, additionalProperties, type) are enforced and value clauses (enum, pattern) are not. Also found: one signed read authority permits reading every record the tool can reach. Both are upstream, both in docs/limitations.md, both kept as known gaps so the day either closes the run says so. The gate. `run.py verify` and CI now run the benchmark and check its report against the chain. Those steps were absent for a day: every other check proves the product RUNS, none proved it DECIDES correctly, so a policy regression would have shipped green. Verified by injecting one — exit 3. Also: bootstrap_env tops up missing secrets instead of leaving an existing install without them; core_schema records which core release initialised a volume, because REMORA has no migrations and CREATE TABLE IF NOT EXISTS is a no-op against a volume that already has the tables. 342 tests. 18/22 on the sealed key, 0 regressions, 4 known gaps.
It listed four targets and described verify as 'pin + contract tests + end-to-end tests', which stopped being true when the benchmark joined the gate. The first place anyone looks for what a runner does is the runner.
Without workflow_dispatch the only way to ask 'does CI still work?' is to push a commit and infer the answer from whether a run appears, which is a poor diagnostic and a worse habit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8 — the base retargets to
mainwhen that merges.What this is
Four things, and the fourth is why the first three matter.
Supply chain. Base images digest-pinned; Python dependencies exact in
docker/requirements.lock. The databases sit on aninternal: truenetwork — measured rather than assumed.control-plane-dbhas no default route; the three containers that publish a port do have egress, anddocs/security-model.mdnow says so per container instead of claiming blanket isolation.Console. The ledger reads the signed audit chain directly on a SELECT-only credential. It previously showed a health banner and four work orders while 165 assessments, 10 binding refusals and 9 voided approvals sat unread one network away. Assurance moved from a page into an always-visible strip, because it qualifies everything else.
Lab. A separate service, image and port that holds every role token and can act. Separate because the console reports
console_access: read-onlyabout itself, and a process that can approve its own proposals cannot credibly say that.Benchmark. Blind scoring against a sealed key, with an audit trail that can be checked against the chain.
Three bugs the work found
[hidden]and.drawerhave equal specificity; source order won. It ate a third of the viewport, empty.style-src 'self'keeps the attribute in the DOM and never applies it. No console error. The decision-mix bar had correct percentages in the markup and rendered 40px wide. Fixed via CSSOM; the CSP was not weakened.paper-faintmeasured 3.4:1 in light and 3.5:1 in dark, used for 10px labels. The chain spine was 1.7:1 while carrying information.Two upstream findings, both from writing answers first
accept/grounded_read_acceptwith the sameintent_authority_hash.required,additionalPropertiesandtypeare;enumandpatternare recorded and ignored.The second was found blind: the key predicted a refusal, the engine assessed it, and probing every clause afterwards gave the structural/value split. Both are in
docs/limitations.mdand kept as known gaps, so the day either closes the run says so loudly.The gate
run.py verifyand CI now run the benchmark and verify its report against the audit chain. Those steps were absent: every other check proves the product runs, none proved it decides correctly. Verified by injecting a regression — exit 3.Verification
Not verified
Narrow-viewport rendering.
resize_windowhad no effect in this environment, so the media queries and.scroll-xwrapping are covered by tests but never seen. Light mode is now measured for contrast but likewise never rendered.One 5/6 scenario run earlier today, never reproduced across six subsequent cold starts and never captured.