[ndisidore]

Adds fast-check property tests for the Workshop backend and for gatekeeper-kit. The end-to-end properties in packages/integration-tests replay generated sequences of real client operations over RPC and check the result against a small model; every run uses a fresh account, so runs and shrink attempts can't see each other's state. The in-isolate properties in packages/workshop-backend drive the modules directly inside workerd, and the two in packages/gatekeeper-kit run under Node. No kernel, shared-API or kit source changes; this is tests, the fixture gatekeeper and test wiring.

End to end:

• Approvals: staged writes are decided in any order, with and without an auto-approval rule, while the provider may fail one apply, from staging or from any decision on. After every step the action log, the fixture gatekeeper's state and the mapping between their ids agree; a write the drain failed stays pending and is never skipped past; a failed approval can be retried or denied (Deny always withdraws a gatekeeper action #857); every apply attempt reaches the provider exactly once; deciding an already decided write applies nothing. Pinned examples cover deciding a manual write queued ahead of auto-approvable ones, the path Resume agent turns after auto-approval, and fix retryAgent and Scheduler docs #599 fixed. The fixture gatekeeper now also reports finished apply attempts, so the property never decides a write whose attempt is still in flight
• Observer re-verification: a collaborator opens a shared workspace one to three times while its connections' credentials lapse and are repaired, including during the one re-prompt. Each open checks the prompts and the bindings named by a terminal error. Across the opens, every verified pass registers the collaborator at its binding, no failed open de-registers anything, and every registration names the one observer id the first open reserved (Share one observer identity across concurrent admissions #1001)
• Attachments: chats are created, replied to and deleted, with shared and same-named files. Every attachment id returns its exact bytes in the chat that holds it and nowhere else, including a fresh chat and a deleted chat
• Code changes: two editors insert and delete lines from stale revisions, as exact ranges the way CodeEditor submits them, and resend changes after the other's have landed. The stream numbers changes 1..n, every surviving line appears exactly once, a resend gets its original acknowledgement and isn't applied again, and accepting the draft commits the converged text

In-isolate, workshop-backend:

• Sharing: Share-dialog action sequences (invite, create, copy, open and revoke links, remove with some people kept) against a reachability model. Each person's role, the collaborator list and live links match after every step, and remove/revoke previews match what actually changes
• Blueprint releases: generated gadget file trees round-trip through pack and install, the release's tree matches the one the workspace's git cache writes, and the release id depends on the files alone
• Streamed tool input: writeFile, editFile and executeCode arguments split at arbitrary points, inside escapes and surrogate pairs included, keep the streaming parser and the code-preview managers in step with what the input so far determines
• Turn resume: real agent turns stage one to three awaited writes, and approvals, rejections, the "always approve" rule, failed applies and workspace restarts are replayed over the session's RPC interface. A suspended turn makes exactly one more model request, when the last write it waits on is approved, and none after a rejection (Resume agent turns after auto-approval, and fix retryAgent and Scheduler docs #599, Resume approved agent turns through existing durable execution records #797, wait for every connection request before resuming the agent #320, Require an auto-approval verdict on every gatekeeper action #941). The approvals property above never has a turn waiting, so it does not reach this path
• Two chats about one gadget: edit, save, discard, ask the agent, accept, take the update a stale accept offers, and revert, in any order. Each chat's content cache and every fold of its log read its own changes on what it last merged; an accept is stale only when the chat changed something and is behind the head; and it commits exactly the chat's content (Convert backing storage to git, change sync to OT, editor widget to CodeMirror #275, Change blueprint format to git and enable receiving blueprint updates #659, Fix revert handling for tool calls in the same step as the reverted edit #522, Fix(compaction): Keep a chat's compaction summary when its changes are reverted #683)
• Scope-widening restarts: what an owner does while collaborators have the workspace open (connections, bindings as permanent or as a chat's proposal, accepts, hooks, spawners), through the owner's client interface. Two steps go around it: an AI model connection is created with addGatekeeper directly, since the model gatekeeper's entrypoint is not among the test Worker's exports, and a hook is bound the way a gadget binds one, which is not an owner call. A restart is scheduled exactly when an open session's verification scope widens, as docs/observers.md defines it; what the session gained is unusable until then; and each gadget lists the model's bindings (Observer verification: fix coverage bookkeeping, restart sessions when scope widens #380, Keep pre-existing observer registrations when re-verification fails #344, add support for single-gadget sharing #894, Let the agent ask the user for an agent spawner #808)
• Unread counts: two users and the agent in three chats across two workspaces, while handoffs to the users are lost, repeated and delayed and objects reset. Stored rows, the queries, the summaries and delta-folding subscribers match docs/user-unread-state.md after every step (Add durable unread summaries and message fanout APIs #837, Add unread inbox, web badges, and native badge consumers #838, Connect unread totals to native badges and durable push replacements #842)

gatekeeper-kit:

• Action set: submissions, approvals and denials in the SharePoint shape (a creation, items inside it, edits of them) while the provider succeeds, fails retryably, refuses or times out. Journal records, the pending projection, handler dispatches, release hooks and refused submissions match a model after every step, and no pending action waits on a reference nothing can still bind. Six vendor action sets go through defineActions; the fixture gatekeeper above stages by hand and does not
• Credentials: two chats on one connection with single-use refresh tokens, under fast-check's scheduler, while the token expires, the user reconnects or disconnects and the provider revokes the grant. No operation runs under two connections or replays a write, the stored grant is the last one connected, and every minted token is stored, discarded or refused by connect(). The test fails if its schedules never reach a reconnect during a refresh, a discarded mint or a second attempt

FAST_CHECK_SEED pins the generated cases. The shared options live in @gadgets/scripts/property-runs; CI sets the seed to 1, every generated test task declares it in cache.env (the shared vitest task does, pinned by vitest-task.test.ts), workshop-backend's vitest config forwards it into workerd, and the env-passthrough guard accounts for it. fast-check is a catalog entry. When it's unset, running vitest directly explores a fresh seed.

Left out on purpose: concurrent decisions on one write (already pinned as an expected failure in workshop-agent-actions.test.ts); an apply that fails while the step that staged its write still runs, whose turn carries on by design (see #drainAutoApprovalsAndResume); in-file merges between the two chats, which each edit their own file so the property needs no merge oracle; and, in the kit, concurrent submit and resolve, authority fences and each vendor's staged reconnect. A scheduled job that runs the suites on fresh seeds is left for a follow-up: CI pins the seed, so new cases are otherwise drawn only on a developer's machine. Three of the four newer backend properties extend their file's existing helpers (a trailing chatId parameter, two helper splits, one shared fake-gatekeeper helper).

Cost: the approvals property takes 4-5s per seed and the observer file 6-7s. In workshop-backend's unit suite the first three in-isolate properties add about 1s, and the four newer ones take about 5s, 4s, 5s and 8s, in four test files that run in parallel. The kit's two take about 3s together locally (the action set 2-3s of it) and took 6s and 11s on two CI runs. With 3 Vitest workers, the full integration suite took 67.1s against 64.6s before the observer property and apply failures were added (one run each); it has not been timed since.

Tested: each property caught every bug planted in the source to check it, with FAST_CHECK_SEED=1. For the earlier properties: drain skipping past a failed apply, a provider failure recorded as applied, a failed open rolling back its registrations, an observer id minted again after a failed open, a terminal error naming the first pass's failures, four sharing bugs, three release-tree bugs and three streaming-split bugs. For the newer ones, 24 more: resuming while a write is still pending, resuming despite a rejection, a queued auto-approvable write not suspending its turn; a revert keeping the merged commit, reverted changes still folded, an update dropping the chat's own edits, an accept never stale, a discard keeping undeclared pins; a bind dropping its widening check, a newly reached gadget not counting as a widening, gained connections left usable, enableHook() skipping the scope check, a hook's enable and its disable never written back; a redelivered own message reading itself again, an older handoff lowering the count, a deleted chat coming back, failed handoffs never retried; a denial not retiring dependents, a dispatch against an unbound reference, an applied action dispatched again; a refresh overwriting a reconnect, a retry across connections, and a non-replayable operation replayed.

On the final code, pnpm lint passes (lint, script types, configs:check and the workspace build), as do the scripts suite with both env guards (370 tests), gatekeeper-kit's Node and workerd projects (777 and 127 tests) and the two workshop-backend test files changed last (91 tests, at seed 1 and at a fresh seed). Workshop-backend's full unit and integration suites last ran locally before the final two commits (2,553 and 3 tests, at seed 1), and the full integration-tests suite was not rerun locally after the rebase onto main; CI runs both here. The observer property's model predated #1001's shared observer identities and failed on the first CI run after the rebase; the commit after the kit's brings it up to date. On the next run the kit's action-set property, which had no timeout of its own, passed vitest's 5s default on a slower runner (9.7s, against 4.7s the run before); both kit properties now have the 120s timeout the backend ones have.

[github-actions]

Preview: pr897-feat-fuzz-test

https://pr897-feat-fuzz-test-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what these runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar (gpt-6-luna)	100% → 90%	−10 pp	p = 1.00	95% 0 pp	5.5 → 4.4	29.5 → 26.6
chess (gpt-6-luna)	90%	0 pp	p = 1.00	98% 0 pp	10.5 → 10.1	60.1 → 53.9
compaction (claude-sonnet-5-5)	100%	0 pp	p = 1.00	90% 0 pp	1.9 → 2.1	43.0
compaction (gpt-6-luna)	100%	0 pp	p = 1.00	93% 0 pp	1.6 → 2.0	43.0
incident-desk (gpt-6-luna)	100% → 100% (3 run errors)	candidate run errors	—	96%	4.9 → 4.0	31.7 → 26.0
ticket-triage-agent (gpt-6-luna)	100%	0 pp	p = 1.00	96% 0 pp	0.2	2.0 → 1.6
worker-logs (gpt-6-luna)	80% → 90%	+10 pp	p = 1.00	94% → 95% +1 pp	4.0 → 5.1	19.0 → 22.3

Failed checks

Task	Check	Failed
change-calendar (gpt-6-luna)	t1 schedules-and-lists-windows	0/10 → 1/10
change-calendar (gpt-6-luna)	t1 rejects-invalid-windows-without-changing-anything	0/10 → 1/10
change-calendar (gpt-6-luna)	t1 overlap-is-per-service-and-touching-is-allowed	0/10 → 1/10
chess (gpt-6-luna)	t1 starts-from-the-standard-position	1/10 → 0/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-on-the-hard-positions	1/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-on-perft-positions	1/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-through-random-games	1/10 → 0/10
chess (gpt-6-luna)	t1 the-game-is-shared-across-connections	1/10 → 0/10
worker-logs (gpt-6-luna)	t1 ingests-resets-and-summarises-per-worker	2/10 → 0/10
worker-logs (gpt-6-luna)	t1 hourly-buckets-cover-every-hour-including-empty-ones	2/10 → 0/10
worker-logs (gpt-6-luna)	t1 ranges-are-half-open-and-filter-by-worker	2/10 → 0/10
worker-logs (gpt-6-luna)	t2 existing-data-and-summary-survive	0/8 → 1/10
worker-logs (gpt-6-luna)	t2 filters-combine-across-colo-route-and-worker	0/8 → 1/10
worker-logs (gpt-6-luna)	t2 breaks-down-by-colo-and-ranks-slow-routes	0/8 → 1/10
worker-logs (gpt-6-luna)	t2 reset-and-ingest-still-work	0/8 → 1/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

No comparable pass-rate change exceeded noise: change-calendar 100% → 90%, chess 90% → 90%, worker-logs 80% → 90%, with triage and both compaction cases still passing; incident-desk is non-comparable after three candidate run errors. Prompt-cache hits and breaks did not move significantly, while cost/time moved inconsistently: calendar averaged $0.0236 → $0.0214 and 5.5 → 4.4 minutes, versus worker-logs $0.0168 → $0.0192 and 4.0 → 5.1 minutes. Incident-desk’s aborts lost the most runs, chess consumed the most steps (60.1 → 53.9), and single-run Sonnet compaction was most expensive ($1.7746 → $1.8221).

⚪ VERDICT: NO REGRESSION FROM THIS PR

The diff adds integration property tests and seed/cache configuration without changing agent behavior or eval checks, so it does not explain the isolated failures or the within-noise performance differences.

Triage

Failure modes

• Opaque agent aborts · incident-desk 3/10 · harness bug · this PR: no — trials 2 and 3 abort during turn 2 after successful editFile calls; trial 6 aborts in turn 1 after successful createGadget. All expose only “Sorry, something went wrong,” with no failed check or tool rejection explaining the underlying infrastructure error.
• Broken timestamp parsing · change-calendar 1/10 · model error · this PR: no — trial 2, turn 1’s writeFile(server.js) passes the one-based month directly to Date.UTC, then rejects the resulting month mismatch. Every booking returns INVALID_RANGE; subsequent edits adjust validation order without fixing the parser. Main passed 10/10.
• Black castling uses out-of-board coordinates · chess 1/10 · model error · this PR: no — trial 9, turn 1’s writeFile(server.js) mixes absolute black-square indices with file offsets, adding the rank offset twice and returning e8c15/e8g15. Later editFile calls leave that arithmetic intact; main also failed 1/10, but on calling .map() on a subscriber Set.
• Filter extension reassigns constants · worker-logs 1/10 · model error · this PR: no — trial 1, turn 2’s writeFile(server.js) declares const args and then assigns [sql, args] = addFilters(...) in every analytics method. The agent reports completion without exercising those methods, so all four turn-2 checks throw; main’s two failures were different, earlier SQL mistakes.

Tool errors

• editFile: No matching text was found in the file. · change-calendar 2 → 0, chess 10 → 2, incident-desk 0 → 2, worker-logs 1 → 1 · model error — quoted text differed in escaping or whitespace from the actual source; candidate chess trial 2 and worker-logs trial 8 recover after removing extra regex escaping.
• editFile: Multiple matches were found. The text to match must be unique. · change-calendar 2 → 0, chess 1 → 0, incident-desk 9 → 2 · model error — agents selected repeated helper calls, regex fragments, or SQL column lists despite the unique-match requirement; adding surrounding context allowed recovery.
• editFile: Validation failed · change-calendar 1 → 1, chess 1 → 4, incident-desk 0 → 4, worker-logs 1 → 0 · model error — calls omitted required fields, malformed replacement, or supplied a tool_uses batch to a single editFile; corrected individual calls succeeded.
• readFile: File does not exist. · change-calendar 3 → 0, incident-desk 2 → 0 · model error — main tried reading files before creating them, including server.js and client.js immediately after creating an empty gadget, despite the prompt explaining that new gadgets have no files.
• executeCode: This step already made code changes · worker-logs 1 → 0 · tool design — main trial 1 batched server edits and RPC execution in the same model step. The execution guard correctly rejected it with recovery instructions, but the tool description does not advertise this restriction beforehand.

What to do

• No eval-driven change is required for this PR. Optionally rerun packages/workshop-evals/evals/incident-desk.eval.ts with backend diagnostics retained to identify the three opaque infrastructure aborts.
• Optional follow-up in packages/workshop-backend/src/agent.ts: document in EXECUTE_CODE_TOOL_DESCRIPTION that execution must occur in a separate model step after file edits.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

Review: 1 findings.

Posted 1 actionable inline comment.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[clanky-the-clanker]

Superseded by the review of c377b934.

This review, of 083586c8

✅ Approved — This pull request only adds tests and test wiring, and no problems were found in code quality, security, performance or documentation.

_{The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.}

---

✅ All checks passed

🔒 Security: ✅ LGTM

Test-only changes and test fixture wiring; nothing security-relevant.

---

📖 AGENTS.md: ✅ LGTM

No issues found.

---

📋 Code Quality: ✅ LGTM

Read every file the change touches. The test helpers the new properties call exist, the seed is fingerprinted in cache.env and forwarded into workerd, the lockfile matches the fast-check additions, and the fixture gatekeeper's settled/attempts ordering is correct.

---

⚡ Performance: ✅ LGTM

No issues found.

---

📝 Documentation: ✅ LGTM

No issues found.

---

_{Reviewed commit: 083586c8}

[clanky-the-clanker]

Superseded by the review of f4d997fd.

This review, of c377b934

✅ Approved — This pull request adds only tests and test wiring. The new changes (owner-interface hook steps and the GatekeeperHookLoopback re-export) are consistent with the code they drive, and no issues were found.

_{The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.}

---

✅ All checks passed

🔒 Security: ✅ LGTM

The change is test-only plus test fixture wiring, and nothing in it affects security.

---

📖 AGENTS.md: ✅ LGTM

No issues found.

---

📋 Code Quality: ✅ LGTM

test-worker.ts now re-exports GatekeeperHookLoopback, which src/server.ts exports. That re-export is needed because OverseerClientInterface.enableHook builds the loopback through ctx.exports, and the test pool takes ctx.exports from the test worker's own exports.

observer-scope-restart.test.ts now drives the owner's real enableHook, disableHook and deleteHook, and its model matches what those methods do:

• Hook ids are read before bindHook allocates them.
• Enabling an already enabled hook does nothing, and neither does disabling a disabled one.
• Deleting a hook removes its record.

Replaying after a restart skips join steps before from, so the replay always finishes.

---

⚡ Performance: ✅ LGTM

No issues found.

---

📝 Documentation: ✅ LGTM

No issues found.

---

_{Reviewed commit: c377b934}

[ask-bonk]

Review: 1 findings.

Posted 1 actionable inline comment.

github run

[Maximo-Guk]

🤖 Comment

P3 Nothing checks that gatekeeper-kit and integration-tests declare the seed Location: scripts/env-passthrough.test.ts:101–102 The comment says each of the three packages declares FAST_CHECK_SEED, but the guard holds only workshop-backend to it, through the read in its vitest.config.ts. The other two read the seed only via property-runs.ts, and the scripts area is skipped by the "declares every forwarded variable" test. I removed "FAST_CHECK_SEED" from packages/gatekeeper-kit/vite.config.ts:19 to check: the guard stayed green, and FAST_CHECK_SEED=abc vp run test passed where it otherwise fails with "must be an integer", so the cached task had stripped the seed. CI would then run that suite on a fresh seed whenever it runs, and nothing would say so. Suggest declaring the seed once in the shared task's env (scripts/vitest-task-vite-config.ts:260), which vitest-task.test.ts already pins. That also removes the three per-package cache overrides.

P3 The turn-resume property's logging is nearly half the test log Location: packages/workshop-backend/tests/durable-decision-resume.test.ts:995–998 This property silences no logging, where the chat-reroot and unread-state properties and both kit ones do. In the last CI run it printed 995 log blocks, about 16,800 of the 37,200 lines in the workshop-backend test job's log: the agent.run.* and agent.step.persist.* records of every generated turn, and with each the same four failures (instance.instructions.read.failed, formats.agent.list.failed, connectable.vendors.list.failed, singleton.capsules.ensure.failed). A failing run's counterexample would be somewhere in the middle of that. Suggest silencing the console for the property's duration, as the others do.

P3 Fresh seeds are only ever drawn on a developer's machine Location: .github/workflows/ci.yml:134–140 CI pins the seed to 1 and nothing else runs these suites, so new cases come only from a direct vitest run or from the first uncached pnpm test after a change. A counterexample found that way looks like a failure in whatever that developer was changing. Suggest a scheduled job that runs the three packages' test uncached with the seed unset, so that exploration reports to whoever owns the properties. Not blocking: the suites held on every seed I tried (gatekeeper-kit's two properties on 1–41, the eight changed workshop-backend test files on 1–33, the two integration property files on 1–31).

Nits

The header of property-runs.ts leaves out gatekeeper-kit Location: scripts/property-runs.ts:1–3 It names integration-tests and workshop-backend as the suites sharing these options. The kit's two properties use them too.

fast-check is not a catalog entry Location: packages/gatekeeper-kit/package.json:52 The same ^4.10.2 is repeated in the package.json of integration-tests and workshop-backend. A catalog: entry, as vitest has, keeps the three on one version.

[clanky-the-clanker]

✅ Approved — This pull request adds only tests and test wiring. The new seed declaration on the shared vitest task, its pinning test and the catalog entry for fast-check match the code they drive, and no issues were found.

_{The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.}

---

✅ All checks passed

🔒 Security: ✅ LGTM

The changes are test code and test wiring only. They contain no secrets and open no new trust boundary.

---

📖 AGENTS.md: ✅ LGTM

No issues found.

---

📋 Code Quality: ✅ LGTM

The seed setup is consistent:

• FAST_CHECK_SEED is declared in the shared vitest task's cache.env (scripts/vitest-task-vite-config.ts), and vitest-task.test.ts pins it.
• workshop-backend forwards the seed into workerd as a binding, with nodejs_compat enabled.
• CI sets the seed to 1.

The fixture gatekeeper's apply-attempts control reads settled before attempts, so it can never report an attempt as settled while it is still in flight. The models in the approvals, observer, resume, action-set and credentials properties match the behaviour they describe.

---

⚡ Performance: ✅ LGTM

The property runs are time-bounded by interruptAfterTimeLimit: 90_000 and a 120s test timeout, and the seed is pinned in CI. No production path is affected.

---

📝 Documentation: ✅ LGTM

No issues found.

---

_{Reviewed commit: f4d997fd}

[ask-bonk]

LGTM!

github run