Skip to content

feat: a lab, a blind benchmark with an auditable trail, and the gate that runs it - #9

Open
darklordVirtual wants to merge 4 commits into
mainfrom
feat/lab-and-benchmarks
Open

feat: a lab, a blind benchmark with an auditable trail, and the gate that runs it#9
darklordVirtual wants to merge 4 commits into
mainfrom
feat/lab-and-benchmarks

Conversation

@darklordVirtual

Copy link
Copy Markdown
Owner

Stacked on #8 — the base retargets to main when that merges.

What this is

Four things, and the fourth is why the first three matter.

Supply chain. Base images digest-pinned; Python dependencies exact in docker/requirements.lock. The databases sit on an internal: true network — measured rather than assumed. control-plane-db has no default route; the three containers that publish a port do have egress, and docs/security-model.md now says so per container instead of claiming blanket isolation.

Console. The ledger reads the signed audit chain directly on a SELECT-only credential. It previously showed a health banner and four work orders while 165 assessments, 10 binding refusals and 9 voided approvals sat unread one network away. Assurance moved from a page into an always-visible strip, because it qualifies everything else.

Lab. A separate service, image and port that holds every role token and can act. Separate because the console reports console_access: read-only about itself, and a process that can approve its own proposals cannot credibly say that.

Benchmark. Blind scoring against a sealed key, with an audit trail that can be checked against the chain.

Three bugs the work found

The drawer was permanently open [hidden] and .drawer have equal specificity; source order won. It ate a third of the viewport, empty.
Every inline style was silently dropped style-src 'self' keeps the attribute in the DOM and never applies it. No console error. The decision-mix bar had correct percentages in the markup and rendered 40px wide. Fixed via CSSOM; the CSP was not weakened.
Three colour tokens below AA The light palette had never been rendered. paper-faint measured 3.4:1 in light and 3.5:1 in dark, used for 10px labels. The chain spine was 1.7:1 while carrying information.

Two upstream findings, both from writing answers first

  • A read authority does not bind to the record it names. Holding WO-1201 and reading WO-1202, WO-1203 or WO-1150 all return accept/grounded_read_accept with the same intent_authority_hash.
  • Value constraints in a signed schema are not enforced. required, additionalProperties and type are; enum and pattern are recorded and ignored.

The second was found blind: the key predicted a refusal, the engine assessed it, and probing every clause afterwards gave the structural/value split. Both are in docs/limitations.md and kept as known gaps, so the day either closes the run says so loudly.

The gate

run.py verify and CI now run the benchmark and verify its report against the audit chain. Those steps were absent: every other check proves the product runs, none proved it decides correctly. Verified by injecting a regression — exit 3.

Verification

  • 342 tests (121 contract + 221 e2e)
  • 6/6 scenarios, stable across six cold starts
  • 18/22 on the sealed key, 0 regressions, 4 known gaps
  • audit trail: 25 entries confirmed against the chain; a forged hash is detected

Not verified

Narrow-viewport rendering. resize_window had no effect in this environment, so the media queries and .scroll-x wrapping are covered by tests but never seen. Light mode is now measured for contrast but likewise never rendered.

One 5/6 scenario run earlier today, never reproduced across six subsequent cold starts and never captured.

"Built on REMORA, consumed as a hash-pinned release rather than a dependency
on its master" was the second sentence a reader met. It is jargon before the
reader knows what the product is, and it restated what the REMORA integration
section said a screen further down — the same fact in two places, which is how
two places start disagreeing.

The opening is plain again: "Powered by REMORA."

docs/pinned-core.md is now the one home for it: what is pinned and why, the
current release, commit and version, how verification works and what each of
the three checks protects against, why the text digests are content digests
rather than byte digests, the upgrade procedure, and the one authoring
dependency the pin does not cover.

The README's integration section is three lines and a link. operations.md's
upgrade section points at the same document instead of restating it.

## The document is bound to the lock

A reference that quotes a commit hash goes stale the moment the pin moves, and
a stale reference is worse than none because it looks authoritative.
test_pin_manifest_agreement.py now asserts the document names the current
release, commit, version and wheel, every pinned artifact, and the prerelease
status.

Verified it can fail: zeroing the commit in the document turns the suite red
on exactly that assertion, and restoring it turns it green. A pin bump that
forgets the document now fails in CI rather than misleading the next reader.

README is 172 lines. 105 contract tests pass.
…that runs it

Four things, and the last one is why the first three are worth having.

Supply chain. Base images are digest-pinned; Python dependencies are exact
versions in docker/requirements.lock used as a constraint set. The databases
sit on an `internal: true` network — measured, not assumed: control-plane-db
has no default route, and the three containers that publish a port do have
egress, which docs/security-model.md now states per container instead of
claiming blanket isolation.

Console. The ledger reads the signed audit chain directly on a SELECT-only
credential, so it shows the governance record rather than a health banner and
four work orders while 165 assessments sat unread one network away. Assurance
moved from a page into a strip, because it qualifies everything else. Found
and fixed: the drawer was permanently open ([hidden] lost to .drawer on source
order), and every inline style was silently dropped by the CSP — the DOM keeps
the attribute, the style never applies, no console error. Contrast measured in
both modes for the first time; three tokens were below AA and are now above.

Lab. A separate service, image and port that holds every role token and can
act. Separate because the console reports `console_access: read-only` about
itself, and a process that can approve its own proposals cannot credibly say
that. Choosing a role selects a credential and grants nothing: the operator
attempting to approve gets 403 from the control plane, and there is a test
asserting the refusal arrives in the engine's words.

Benchmark. Scenarios in benchmarks/suites/, answers in benchmarks/keys/, and
the prediction pass never opens a key — enforced by a test that makes load_key
throw. Keys are sealed to a suite digest; a broken seal is not scored at all,
because grading changed questions against old answers is a number with no
meaning. Every run records the chain positions it created, and those can be
re-read and confirmed: `run.py bench-verify`. Forging one entry hash is
detected.

Writing the answers first paid for itself immediately. A key predicted that a
value outside a declared `enum` would be refused; the engine assessed it.
Probing every clause gave the real shape — structural clauses (required,
additionalProperties, type) are enforced and value clauses (enum, pattern) are
not. Also found: one signed read authority permits reading every record the
tool can reach. Both are upstream, both in docs/limitations.md, both kept as
known gaps so the day either closes the run says so.

The gate. `run.py verify` and CI now run the benchmark and check its report
against the chain. Those steps were absent for a day: every other check proves
the product RUNS, none proved it DECIDES correctly, so a policy regression
would have shipped green. Verified by injecting one — exit 3.

Also: bootstrap_env tops up missing secrets instead of leaving an existing
install without them; core_schema records which core release initialised a
volume, because REMORA has no migrations and CREATE TABLE IF NOT EXISTS is a
no-op against a volume that already has the tables.

342 tests. 18/22 on the sealed key, 0 regressions, 4 known gaps.
@darklordVirtual
darklordVirtual changed the base branch from docs/pinned-core to main August 6, 2026 20:20
It listed four targets and described verify as 'pin + contract tests +
end-to-end tests', which stopped being true when the benchmark joined the
gate. The first place anyone looks for what a runner does is the runner.
Without workflow_dispatch the only way to ask 'does CI still work?' is to
push a commit and infer the answer from whether a run appears, which is a poor
diagnostic and a worse habit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant