Repository navigation
feat(profiling): exercise Tokenjuice agent tool results - #362
vincentkoc wants to merge 5 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: blocked before merge. Reviewed September 24, 2026, 11:51 AM ET / 15:51 UTC (Revision 12). ClawSweeper reviewWhat this changesAdds an opt-in Tokenjuice profiling workload that compares raw and compacted Git tool results across embedded agent turns, with archive checks, fixture tests, documentation, and a temporary Windows diagnostic job. Regression provenancePossible regression — suspected (failure trace). No predecessor PR is attributed. Merge readiness⛔ Blocked before merge - 6 items remain This PR remains useful because current main has the shared scoped-activation contract but lacks the Tokenjuice workload. The added native fixtures have failed on hosted Windows, and the integrated head has no validated matched-pair receipt, so it is not ready to merge. Priority: P2 Review scores
Verification
How this fits togetherCrabpot’s resource campaign reads configured plugin scenarios and runs their adapters against a prepared OpenClaw host. This adapter sends synthetic model requests through real Gateway agent turns and records whether Tokenjuice changes the tool result seen by the model. flowchart LR
A[Scenario configuration] --> B[Pinned plugin archive]
B --> C[Gateway agent turns]
D[Synthetic Git repository] --> C
C --> E[Loopback model provider]
E --> F{Raw or compacted result?}
F --> G[Statistics and workload receipt]
Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Keep the workload fixture-driven and use the shared campaign runner to verify pinned archive identity, real Gateway model-visible compaction, statistics, and owned cleanup in a qualified isolated run. Do we have a high-confidence way to reproduce the issue? Not applicable to the requested new workload. The earlier-head hosted Windows fixture failure is documented, but the latest diagnostic job’s failure has no readable log here to establish its cause. Is this the best way to solve the issue? Yes, the opt-in adapter uses Crabpot’s existing campaign and scoped-activation contracts. The solution still needs native Windows qualification and an integrated real-Gateway receipt before it can land. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 9ceee4211b2d. LabelsLabel changes: No label changes. Label justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (11 earlier review cycles; latest 8 shown)
|
f09f9b0 to
02ccb61
Compare
02ccb61 to
19bcf36
Compare
4205705 to
9697f43
Compare
What Problem This Solves
Crabpot has no workload proof for Tokenjuice's lazy tool-result middleware; Gateway catalog registration alone cannot show that the model received reduced output.
User Impact
Adds the opt-in
tokenjuice-exec-status-v1scenario: matched disabled/enabled embedded agent turns, real Git-status tool calls, a strict synthetic loopback provider and explicit resource observations. The default bundled campaign does not execute this external scenario; ordinary CI may still download other fixture sources.Why This Change Was Made
Both cases install identical pinned source-built archive bytes and activate the ordinary OpenAI provider. Each runs one first-use turn and five warm turns in fresh sessions against the same synthetic repository with 32 modified tracked files. The fixture requests exactly one exec and checks its output in the second model request before returning a terminal marker. The control must receive raw Git output; the enabled case must receive shorter, marked output and six default statistics records. Automatic Activity recaps and utility-model selection are disabled equally; unexpected requests fail.
The middleware uses a caller-owned handle. Exact Gateway catalog observations remain scoped to the OpenAI dependency; actual model-visible output and statistics prove the target's work. Dependency activation earns no separate workload credit. Both cases use the landed shared workload, scoped-activation, cleanup and receipt owners; this change adds no second Gateway or measurement engine.
The adapter requires source-built
@openclaw/tokenjuice2026.9.5 with bundledtokenjuice0.8.4. It rejects the same-version registry wrapper's different dependency closure. Native archive preparation and installation are outside measured Gateway phases. Source/build provenance and archive identity are separate from host/runtime qualification.Git isolation now uses an empty ordinary file under the existing fixture root. Git for Windows rejects Node's null-device path when used as
GIT_CONFIG_GLOBAL; the file is created before any Git child starts and remains inside the fixture's existing cleanup ownership. No startup deadline, cleanup guard or controller environment is changed.The latest diagnostic change keeps
status,signal,errorandcleanupErrorunchanged. Failure text may include only eight exact known owner reasons and an unsigned 32-bit native error code; arbitrary messages, stack, arguments, environment, output and paths are excluded. This distinguishes known supervisor/adapter outcomes without claiming an internal parse or compile stage.Within the Tokenjuice test file, unconfirmed cleanup retains the exact failed fixture root and blocks later native callbacks before invocation. Joined failures and unrelated roots still receive normal cleanup. Other test files continue; this is not whole-job fail-fast or proof of process extinction. Original minimal environments, tar-before-Git order and deadlines remain unchanged, with no inherited-environment warmup or shared command-owner modification.
Evidence
Landing held. This candidate improves failure diagnostics and test fixture ownership; it does not fix the unresolved Windows startup failure or establish closure for prior failed owners. The latest published required Windows job, at
cdb4f5e, failed. The new diagnostic head is awaiting hosted qualification. Successful native readiness and authoritative cleanup, plus a validated final-head isolated workload receipt, remain required.18747a87f3122e7e82c8a46b416f0210e7e8514e: 35/35 focused builtin tests passed, with no failures, cancellations or skips, in 551.592 milliseconds. This includes real Git/tar setup and the new cleanup-cause privacy regression. Syntax, diff whitespace and public-data checks passed. Exact-head native review completed with one accepted P1: the unchanged Windows native owner still fails readiness/cleanup in the required fixture path. No additional defect in the diagnostic probe was reported; this is not a clean landing review.Retained earlier evidence
cdb4f5e25b0f9f04528e2c37a42768192203c546preserves the owned workload integrated onto frozen main407299b17af54c48ffc117c2536097f6c40bcc6e, the fixture-owned Git configuration repair, and adds bounded diagnostics and test-local fixture retention/admission. All landed shared owners, Session Share's late-fault repair, existing scenarios, scheduled workflow and generated reports are preserved. The command owner, startup/cleanup budgets and environment forwarding are unchanged.cdb4f5e, 34/34 focused builtin Tokenjuice tests passed, no failures or skips, on Node 26.7.0/macOS in 1.555 seconds for this candidate. They cover real Git/tar setup, loopback Responses, strict outcomes, diagnostic privacy, retained uncertain roots, rejected later native admission and normal cleanup after joined failures. The broader 217-test result belongs to prior5198b26; it is not a rerun on this diagnostic candidate.cdb4f5enative review completed with exit 1 and one P1: fix Windows owner startup before admitting these native fixtures. It cites the existing hosted readiness/cleanup failure; that landing blocker is accepted and unresolved. No additional defect in the diagnostic or test-retention change was reported. The review is not claimed clean, and the retained failure does not identify an internal startup stage or establish a production root cause. ClawSweeper Revision 10 covers exactcdb4f5ewith no additional code or security findings. Its before-merge requirements and Rank-up moves retain successful final-head hosted Windows tar/Git readiness and authoritative cleanup, plus a validated isolated matched-pair receipt; the draft hold remains until both are satisfied. Both requirements remain unsatisfied.cdb4f5e, the required Windows job failed: 604 passed, 2 failed, 27 skipped, 0 cancelled out of 633 tests, on Node 24.20.0 x64 / Windows Server 2025 (10.0.26100). Initial tar packing failed after 14.044 seconds with the unchanged four outcome fields:status: null,signal: null,error: EOWNERSTART,cleanupError: EOWNERCLEANUP. The preserved owner reasons are exactlycommand supervisor startup timed outandWindows helper closure was not observed; both native error codes are null. The subsequent Git test failed atnative fixture admission blocked after unconfirmed cleanupin 9.344 milliseconds, before its native callback: that test did not attempt Git initialization. All three new diagnostic-privacy, uncertain-root/admission and joined-failure regressions passed. Canonical Windows bootstrap, cancellation, compiler and Job controls also passed, but they do not establish closure of the failed tar owner. The log does not identify an internal startup cause or prove process extinction. Complete log SHA-256:2997ba8b82cc2894e0405263e8f374199b0ad074bcf0dc3d58f48aa885401612. This changed-source diagnostic run is not an unchanged CI retry or a startup repair.5198b26, the required Windows job failed on Node 24.20.0 / Windows Server 2025: 601 passed, 2 failed, 27 skipped, 0 cancelled. The archive-input test failed at initial tar packing (12.046 seconds), and the real Git fixture failed at initial Git initialization (12.050 seconds). Both returnedstatus: null,signal: null,error: EOWNERSTART,cleanupError: EOWNERCLEANUP; neither successful command readiness nor authoritative fixture-owner closure was confirmed. The complete job log SHA-256 isddf68e1d1c13bbcedf380170ac316a6c78777cd5f093709ba9700dafe7287a59. Canonical Windows owner controls passed in that job, but those separate controls do not establish closure for these failed fixtures. The config-path correction does not repair this distinct startup failure. No unchanged CI rerun or waiver was used.git --versioneach returned 128 with the same complete 52-byte error identifying the null-device config path as invalid. A fresh initialization changing onlyGIT_CONFIG_GLOBALto an existing empty file returned 0 and produced the expected branch HEAD. All three canonical command owners joined cleanly; exact native provider release was verified. A preceding bounded probe also completed tar and Git ownership in both environment arms, without reproducing the hosted startup failure. These observations establish the config-path defect, not a fix forEOWNERSTARTorEOWNERCLEANUP. They ran on Server 2022, while the failed hosted job used Server 2025; actual Git version output was unavailable because the config error also preventedgit --version.88d4d73ef7dc7efb5bd421597257eb5bef7ee5a664d9c4aa6e8d53d8145f07b4. Its source is frozen OpenClaw5708fd83b080a345021ce376d7e532eb38877913; all eleven recorded plugin source hashes match the planned59a734b7358b6ccb62449b74346ff422a9e3fef7host source. This does not relabel the historical package build as a fresh 59a build.9697f435and the 59a host ended with native client exit 124 at the five-minute sync completion-marker guard. The client joined and the exact task-owned lease reached stopped/completed, but no runtime artifact was delivered and no canonical receipt validation ran. Controller startup and workload execution remain unknown: native artifact collection occurs only after exit 0. A successful final-head matched pair and validated receipt remain required. No current-head Gateway, full-campaign or performance qualification is claimed.Historical proof and unresolved failures
9697f435, the required Windows job failed: 599 passed, 2 failed, 27 skipped, 0 cancelled. The added archive-input test failed at initial tar packing, and the real Git fixture failed at initial Git initialization. Both command results werestatus: null,signal: null,error: EOWNERSTART,cleanupError: EOWNERCLEANUP; neither successful readiness nor terminal cleanup was confirmed. The completed job log SHA-256 isf2b7ee6707cd822d1971021f57f3f7bea3e26a6b9a7c9eab0f15e2cf1d168279. No unchanged CI rerun or waiver was used.Three earlier paired runs at
04d515312cfb8691a39695a98d7b411b6f562275and one confirmation pair at7983100b7e01d0070c01c9e01de2e6197de1595ccompleted the selected workload on host5708fd83b080a345021ce376d7e532eb38877913with harnessc029367d73b8539a36de75e94830ca36b4894831. The confirmation receipt SHA-256 is55d3cc212df8546d554fffa5d11e847ff9326305ff2f9b7b004c3051b30bc1ee. These receipts retain their original source/runtime identities and do not qualify this integrated consumer.At prior head
42057057a87c84e909fb84c06c2baa4a5b941fc1, the required Windows job and advisory Windows job failed during fixture tar/Git setup withEOWNERSTARTandEOWNERCLEANUP, before Tokenjuice loaded. Readiness and terminal cleanup were not confirmed. The separate frozen-base main check failed in unchanged Inspector import-time bootstrap with the same startup/cleanup uncertainty; its dashboard writer was skipped. This integration does not diagnose or repair those failures, infer a leak or waive new-head checks.Fixture Impact
Fixture Verification
npm test— exact-head required Windows failed as described above; landing remains heldnode scripts/sync-fixtures.mjs --checknode scripts/run-contract-smoke.mjsNotes
985 added lines across seven files: adapter 385, configuration 8, tests and diagnostic fixture 524, temporary CI job 21, and docs/changelog 47. Production growth adds the strict provider fixture, native package admission and output/statistics observer while reusing the existing runner. The original diagnostic increment adds 19 production lines; this update adds one allowlisted cleanup-cause field to retain authoritative owner facts; test ownership adds no production process-control path.
Gateway process metrics exclude Git children and the mock provider; wall time includes their work. Signed matched deltas include whole-agent noise and do not allocate cost to reducer functions. One Git-status path does not establish other reducers, Codex behavior, real-provider behavior, peaks, leaks, stable budgets or in-process handle disposal. The full source inventory remains the denominator, and no complete four-scenario campaign has run on this candidate. Measured workloads use no production credentials or external services; no release or deployment is included.