Repository navigation
fix: keep skipped required evidence incomplete - #268
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs real behavior proof before merge. What this changesCount required skipped evidence as unfulfilled when computing ledger completeness, missing counts, and passing-record verdicts. Example: A passing record contains command:proof:1 with evidenceStatus skipped and evidenceRequired true.
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Merge readiness⛔ Blocked before merge - 2 items remain This PR fixes a current ledger defect, but leaves report proof summaries inconsistent with the corrected ledger. Priority: P2 Before merge
Findings
Tests
Agent review detailsHow this fits togetherKova finalizes OpenClaw validation records by converting command and collector evidence into a ledger, gating the verdict, and projecting the result into JSON and human-readable reports. flowchart LR
A[OpenClaw validation evidence] --> B[Record finalization]
B --> C[Evidence ledger]
C --> D[Verdict gating]
D --> E[Report summary]
E --> F[JSON and terminal reports]
Technical reviewBest possible solution: Keep the ledger fix and align the existing report proof aggregation with required-skip semantics. Do we have a high-confidence way to reproduce the issue? The linked synthetic input follows a source-verifiable path that preserves skipped status and bypasses main's missing-evidence predicates; no target code was executed during this read-only review. Is this the best way to solve the issue? The shared predicate is a focused ledger repair, but report aggregation must recognize the same unfulfilled requirement to avoid contradictory output. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 8c8764f443ec. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceAcceptance criteria:
What I checked:
Preflight sourceSource unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified. Root-cause clusterRelationship: Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed November 13, 2026, 7:35 AM ET / 12:35 UTC. |
Co-authored-by: Leonardo Bohac <leonardo.bohac@gmail.com>
Co-authored-by: Leonardo Bohac <leonardo.bohac@gmail.com>
Co-authored-by: Leonardo Bohac <leonardo.bohac@gmail.com>
steipete
left a comment
There was a problem hiding this comment.
Verified the required/optional skip behavior through the shipped CLI, including whole-record DRY-RUN and SKIPPED controls. The full Linux gate and exact-head Linux/macOS CI pass, with no actionable P0–P2 findings remaining from independent review.
Closes #267
What Problem This Solves
A required skipped evidence entry can leave an executed Kova record marked PASS, even though its proof obligation was not fulfilled. Updating the ledger alone also leaves the report contradictory: INCOMPLETE alongside complete proof and zero missing requirements.
User Impact
Executed required skips produce INCOMPLETE with matching ledger, terminal, and JSON proof counts. Optional skips remain nonblocking. Whole-record DRY-RUN and SKIPPED outcomes retain their existing proof counts and do not acquire false blocking findings.
Why This Change Was Made
The ledger and report projections use the same record-aware unfulfilled-evidence predicate. Skipped observations remain intact; only executed required skips count as missing obligations. This includes command, snapshot, invariant, artifact, cleanup, and channel-capability evidence.
Evidence
The original required-skip case reproduced on main. A six-case regression through the shipped report CLI passes for required/optional evidence across executed, dry-run, and wholly skipped records. The added dry-run control reproduced and caught the earlier false blocking-finding edge case. Codex autoreview now reports no actionable P0–P2 findings.
The full Linux check suite and release archive packaging passed on the corrected candidate. Exact-head cross-platform CI remains the final gate before merge.