Skip to content

fix: keep skipped required evidence incomplete - #268

Merged
steipete merged 7 commits into
openclaw:mainfrom
llbohacll:fix/required-skipped-evidence
Nov 21, 2026
Merged

steipete merged 7 commits into
openclaw:mainfrom
llbohacll:fix/required-skipped-evidence

Conversation

@llbohacll

@llbohacll llbohacll commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #267

What Problem This Solves

A required skipped evidence entry can leave an executed Kova record marked PASS, even though its proof obligation was not fulfilled. Updating the ledger alone also leaves the report contradictory: INCOMPLETE alongside complete proof and zero missing requirements.

User Impact

Executed required skips produce INCOMPLETE with matching ledger, terminal, and JSON proof counts. Optional skips remain nonblocking. Whole-record DRY-RUN and SKIPPED outcomes retain their existing proof counts and do not acquire false blocking findings.

Why This Change Was Made

The ledger and report projections use the same record-aware unfulfilled-evidence predicate. Skipped observations remain intact; only executed required skips count as missing obligations. This includes command, snapshot, invariant, artifact, cleanup, and channel-capability evidence.

Evidence

The original required-skip case reproduced on main. A six-case regression through the shipped report CLI passes for required/optional evidence across executed, dry-run, and wholly skipped records. The added dry-run control reproduced and caught the earlier false blocking-finding edge case. Codex autoreview now reports no actionable P0–P2 findings.

The full Linux check suite and release archive packaging passed on the corrected candidate. Exact-head cross-platform CI remains the final gate before merge.

@clawsweeper

clawsweeper Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. labels Nov 13, 2026
@clawsweeper

clawsweeper Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge.

What this changes

Count required skipped evidence as unfulfilled when computing ledger completeness, missing counts, and passing-record verdicts.

Example: A passing record contains command:proof:1 with evidenceStatus skipped and evidenceRequired true.

  • Before: The record remains PASS, ledger completeness is complete, and requiredMissing is 0.
  • After: The record becomes INCOMPLETE, ledger completeness is incomplete, and requiredMissing is 1; the entry remains skipped.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The narrow ledger repair has useful regression coverage, but report projection remains inconsistent and supplied validation does not meet the real-behavior proof gate.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is necessary before merge. See Before merge.
Patch quality 🦪 silver shellfish (2/6) 1 actionable review finding remain.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Consumers can accept a PASS record even though declared required proof was skipped.
Reason: Treating an unfulfilled requirement as incomplete restores the existing fail-closed evidence contract without adding configuration or capabilities.

Merge readiness

⛔ Blocked before merge - 2 items remain

This PR fixes a current ledger defect, but leaves report proof summaries inconsistent with the corrected ledger.

Priority: P2
Reviewed head: 7fb30032865069c169203cc51eb093ba4800f528

Before merge

  • Add real behavior proof - The supplied ledger API reproduction and isolated consumer checks exercise synthetic records in process; no supplied artifact exercises the changed verdict and report output through the shipped Kova entry point. After adding proof, update the PR body, then request a new review with @clawsweeper re-review. The PR author or someone with repository write access can request it.
  • Align report proof summaries with required skipped evidence (P2) - For the linked reproduction, passing the patched record to buildReportSummary produces a contradiction: the record ledger is incomplete with requiredMissing=1, but report.proof.completeness is {complete:1}, requiredMissing is 0, and missingRequired is empty. src/reporting/summary.mjs:369-395 recomputes these fields from entry statuses and recognizes only missing; this patch deliberately retains skipped. Update that existing report projection and its regression coverage so terminal and JSON summaries agree with the corrected ledger while preserving optional skips.

Findings

  • [P2] Align report proof summaries with required skipped evidence — src/evidence-ledger.mjs:28

Tests

  • Missing end-to-end proof: Provide redacted terminal output or logs from the shipped Kova reporting path showing a required skip becomes INCOMPLETE with matching proof counts, plus an optional-skip control.
Agent review details

How this fits together

Kova finalizes OpenClaw validation records by converting command and collector evidence into a ledger, gating the verdict, and projecting the result into JSON and human-readable reports.

flowchart LR
  A[OpenClaw validation evidence] --> B[Record finalization]
  B --> C[Evidence ledger]
  C --> D[Verdict gating]
  D --> E[Report summary]
  E --> F[JSON and terminal reports]
Loading

Technical review

Best possible solution:

Keep the ledger fix and align the existing report proof aggregation with required-skip semantics.

Do we have a high-confidence way to reproduce the issue?

The linked synthetic input follows a source-verifiable path that preserves skipped status and bypasses main's missing-evidence predicates; no target code was executed during this read-only review.

Is this the best way to solve the issue?

The shared predicate is a focused ledger repair, but report aggregation must recognize the same unfulfilled requirement to avoid contradictory output.

Full review comments:

  • [P2] Align report proof summaries with required skipped evidence — src/evidence-ledger.mjs:28
    For the linked reproduction, passing the patched record to buildReportSummary produces a contradiction: the record ledger is incomplete with requiredMissing=1, but report.proof.completeness is {complete:1}, requiredMissing is 0, and missingRequired is empty. src/reporting/summary.mjs:369-395 recomputes these fields from entry statuses and recognizes only missing; this patch deliberately retains skipped. Update that existing report projection and its regression coverage so terminal and JSON summaries agree with the corrected ledger while preserving optional skips.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 8c8764f443ec.

Provenance checked

  • Required-evidence gating and completeness keeps the original intent (f863110: Prevent passing records when required evidence is missing.)
  • Required-missing ledger counts keeps the original intent (820b4bf: Provide machine-readable evidence metadata and status counts.)
  • Incomplete proof classification keeps the original intent (fix(evidence): harden publish and run contracts #55: Fail closed on incomplete run proof and prevent reports from overstating evidence.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

Acceptance criteria:

  • node bin/kova.mjs self-check
  • npm run test:snapshots
  • Verify buildReportSummary reports incomplete proof and requiredMissing=1 for an executed required-skip record, while an optional skip remains complete.

What I checked:

  • Exact introduced change: The pinned diff changes all three ledger predicates, documents the semantics, and adds required/optional skip controls across six evidence categories. (src/evidence-ledger.mjs:28, 7fb300328650)
  • Current main still has the defect: The default branch remains at the pinned base; its gate selects required missing entries but excludes skipped entries. (src/evidence-ledger.mjs:28, 8c8764f443ec)
  • Downstream report inconsistency: Report proof aggregation independently counts only missing statuses, so required skipped evidence still produces complete proof, zero requiredMissing, and no missingRequired entry. (src/reporting/summary.mjs:371, 7fb300328650)
  • Production integration and optional cleanup: Terminal record finalization attaches and gates the ledger; intentionally retained environments already mark skipped cleanup evidence optional. (src/run/finalize-record.mjs:82, 8c8764f443ec)
  • Original gating intent: The verified introduction patch added required-evidence gating; subsequent fail-closed work in fix(evidence): harden publish and run contracts #55 explicitly aimed to prevent reports from overstating proof. (src/evidence-ledger.mjs:20, f863110ac2ec)
  • Discussion and supplied validation: The complete PR body describes synthetic ledger API and isolated consumer checks; comments contain no additional runtime artifact. The author explicitly distinguishes these checks from a native producer incident. (7fb300328650)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #267
Summary: This PR directly addresses the required-skip contract defect documented by the linked issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

  • add P2: Required skipped proof can falsely appear complete; the demonstrated impact is validation verdict integrity with limited established reachability.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Label justifications:

  • P2: Required skipped proof can falsely appear complete; the demonstrated impact is validation verdict integrity with limited established reachability.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 7:35 AM ET / 12:35 UTC.

@llbohacll
llbohacll requested a review from a team as a code owner November 13, 2026 18:35

@steipete steipete left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the required/optional skip behavior through the shipped CLI, including whole-record DRY-RUN and SKIPPED controls. The full Linux gate and exact-head Linux/macOS CI pass, with no actionable P0–P2 findings remaining from independent review.

@steipete
steipete merged commit c584335 into openclaw:main Nov 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Required skipped evidence can leave records marked PASS

2 participants