Skip to content

Required skipped evidence can leave records marked PASS #267

Closed

Description

@llbohacll

Problem

A declared required evidence entry can be marked skipped while the evidence
ledger remains complete and the record remains PASS. A consumer using the
record verdict can therefore accept a result whose required proof never ran.

Reproduced through the ledger API on 8c8764f443ecc703ec6d035fc322c2b0b3c9e96b.
This is an API contract reproduction with synthetic metrics; the observation does
not depend on a native scenario currently emitting a required skip.

Reproduction

Run from the repository root:

node --input-type=module <<'JS'
import { attachEvidenceLedger, applyEvidenceLedgerGating } from './src/evidence-ledger.mjs';
const record = {
  status: 'PASS',
  phases: [{ id: 'proof', commands: ['inspect'], results: [
    { status: 0, evidenceStatus: 'skipped', evidenceRequired: true }
  ] }],
  finalMetrics: {
    error: null,
    service: { gatewayState: 'running' },
    health: { ok: true, durationMs: 5 },
    healthSamples: [{ ok: true, durationMs: 5 }],
    healthSummary: { count: 1, failureCount: 0 }
  }
};
attachEvidenceLedger(record);
applyEvidenceLedgerGating(record);
console.log(JSON.stringify({status:record.status,completeness:record.evidenceLedger.completeness,
  requiredMissing:record.evidenceLedger.summary.requiredMissing,
  skipped:record.evidenceLedger.summary.byStatus.skipped}));
JS

Actual:

{"status":"PASS","completeness":"complete","requiredMissing":0,"skipped":1}

Expected: INCOMPLETE, completeness: "incomplete", requiredMissing: 1,
while retaining the entry's original skipped status. An optional skip should
continue to allow PASS; whole-record DRY-RUN and SKIPPED should retain their
existing meaning.

Activity

  1. added
    clawsweeper:intake:bugPreliminary submission classification; full source review pending.
    P2Normal priority bug or improvement with limited blast radius.
    clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    and removed
    clawsweeper:intake:bugPreliminary submission classification; full source review pending.
    on Nov 13, 2026
  2. clawsweeper commented on Nov 13, 2026

    @clawsweeper

    Codex review: this still needs some work.

    Summary
    Current main still permits required skipped evidence to leave a record passing, and the open linked fix PR requires this issue to remain open.

    Reproducibility: Source inspection confirms explicit skipped input survives normalization and bypasses all three missing-evidence predicates; the supplied reproduction was not executed during this read-only review.

    Next step
    Review #268 and verify its required-skip regression fails on main and passes with the fix before merging.

    Review details

    Best possible solution:

    Required skipped proof should count as unfulfilled across ledger summaries and verdict gating while retaining its original status.

    Do we have a high-confidence way to reproduce the issue?

    Source inspection confirms explicit skipped input survives normalization and bypasses all three missing-evidence predicates; the supplied reproduction was not executed during this read-only review.

    Is this the best way to solve the issue?

    The linked PR's shared missing-or-skipped predicate addresses the narrow defect and includes optional-skip and unexecuted-record controls, subject to its separate review and validation.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • The report establishes an API contract defect with synthetic metrics; a native scenario producing required skipped evidence has not been established.

    Codex review notes: model internal, reasoning medium; reviewed against 8c8764f443ec.

    Label changes

    Label changes:

    • add P2: The source-proven contract defect can falsely accept incomplete validation evidence, with native scenario reachability still unestablished.
    • add impact:other: Validation verdict integrity is affected: consumers can accept PASS without declared required proof.
    • add issue-rating: 🦞 diamond lobster: Current issue advisory state selects this label.
    • add clawsweeper:source-repro: Current issue advisory state selects this label.
    • add clawsweeper:linked-pr-open: Current issue advisory state selects this label.
    • add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.

    Label justifications:

    • P2: The source-proven contract defect can falsely accept incomplete validation evidence, with native scenario reachability still unestablished.
    • impact:other: Validation verdict integrity is affected: consumers can accept PASS without declared required proof.
    Evidence reviewed

    What I checked:

    • Current default branch: GitHub identifies main as the default branch and its tip matches the review checkout. (8c8764f443ec)
    • Source establishes the defect: The gate selects only required entries marked missing; completeness and requiredMissing use the same exclusion of skipped entries. commandStatus preserves explicit skipped input. (src/evidence-ledger.mjs:28, 8c8764f443ec)
    • Contract and production integration: The documented contract prevents passing records with missing required proof; terminal scenario finalization attaches the ledger and applies its gate. (src/run/finalize-record.mjs:82, 8c8764f443ec)
    • Existing coverage and optional skip behavior: Current self-checks cover missing required proof; explicit retained-environment cleanup is represented as optional skipped evidence. (src/selfcheck/evidence-ledger.mjs:44, 8c8764f443ec)
    • Open fix owns the work: fix: keep skipped required evidence incomplete #268 is open and unmerged, explicitly closes this issue, and changes all three ledger predicates with required/optional skip regression coverage. (7fb300328650)
    • Feature history: GitHub commit patches verify shakkernerd introduced required-evidence gating and vincentkoc later expanded fail-closed proof validation; local historical blob reads failed, so ownership uses REST commit evidence. (src/evidence-ledger.mjs, f863110ac2ec)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge; those execution lanes are retired.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

    Reviewed November 13, 2026, 7:25 AM ET / 12:25 UTC.

  3. steipete commented on Nov 21, 2026

    @steipete
    Contributor

    Fixed in #268, with thanks to @llbohacll.

    The original required-skip case reproduced on main: an executed record remained PASS with complete proof and zero missing requirements. It now becomes INCOMPLETE with one missing requirement, while the original skipped observation is retained.

    Six tests through the shipped report CLI verify required and optional skips for executed, DRY-RUN, and SKIPPED records. Optional skips stay nonblocking; whole-record plans and skips retain zero missing-proof counts and zero blocking findings. The full Linux check suite and archive packaging passed, followed by exact-head cross-platform CI.

    Merge commit: c584335. CI: https://github.com/openclaw/Kova/actions/runs/44852615079.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priority bug or improvement with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions