Repository navigation
Closed
Labels
P2Normal priority bug or improvement with limited blast radius.Normal priority bug or improvement with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Problem
A declared required evidence entry can be marked skipped while the evidence
ledger remains complete and the record remains PASS. A consumer using the
record verdict can therefore accept a result whose required proof never ran.
Reproduced through the ledger API on 8c8764f443ecc703ec6d035fc322c2b0b3c9e96b.
This is an API contract reproduction with synthetic metrics; the observation does
not depend on a native scenario currently emitting a required skip.
Reproduction
Run from the repository root:
node --input-type=module <<'JS'
import { attachEvidenceLedger, applyEvidenceLedgerGating } from './src/evidence-ledger.mjs';
const record = {
status: 'PASS',
phases: [{ id: 'proof', commands: ['inspect'], results: [
{ status: 0, evidenceStatus: 'skipped', evidenceRequired: true }
] }],
finalMetrics: {
error: null,
service: { gatewayState: 'running' },
health: { ok: true, durationMs: 5 },
healthSamples: [{ ok: true, durationMs: 5 }],
healthSummary: { count: 1, failureCount: 0 }
}
};
attachEvidenceLedger(record);
applyEvidenceLedgerGating(record);
console.log(JSON.stringify({status:record.status,completeness:record.evidenceLedger.completeness,
requiredMissing:record.evidenceLedger.summary.requiredMissing,
skipped:record.evidenceLedger.summary.byStatus.skipped}));
JSActual:
{"status":"PASS","completeness":"complete","requiredMissing":0,"skipped":1}Expected: INCOMPLETE, completeness: "incomplete", requiredMissing: 1,
while retaining the entry's original skipped status. An optional skip should
continue to allow PASS; whole-record DRY-RUN and SKIPPED should retain their
existing meaning.
Reacted by clawsweeper
Metadata
Metadata
Assignees
Labels
P2Normal priority bug or improvement with limited blast radius.Normal priority bug or improvement with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Activity
clawsweeper commentedon Nov 13, 2026
Codex review: this still needs some work.
Summary
Current main still permits required skipped evidence to leave a record passing, and the open linked fix PR requires this issue to remain open.
Reproducibility: Source inspection confirms explicit skipped input survives normalization and bypasses all three missing-evidence predicates; the supplied reproduction was not executed during this read-only review.
Next step
Review #268 and verify its required-skip regression fails on main and passes with the fix before merging.
Review details
Best possible solution:
Required skipped proof should count as unfulfilled across ledger summaries and verdict gating while retaining its original status.
Do we have a high-confidence way to reproduce the issue?
Source inspection confirms explicit skipped input survives normalization and bypasses all three missing-evidence predicates; the supplied reproduction was not executed during this read-only review.
Is this the best way to solve the issue?
The linked PR's shared missing-or-skipped predicate addresses the narrow defect and includes optional-skip and unexecuted-record controls, subject to its separate review and validation.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against 8c8764f443ec.
Label changes
Label changes:
P2: The source-proven contract defect can falsely accept incomplete validation evidence, with native scenario reachability still unestablished.impact:other: Validation verdict integrity is affected: consumers can accept PASS without declared required proof.issue-rating: 🦞 diamond lobster: Current issue advisory state selects this label.clawsweeper:source-repro: Current issue advisory state selects this label.clawsweeper:linked-pr-open: Current issue advisory state selects this label.clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.Label justifications:
P2: The source-proven contract defect can falsely accept incomplete validation evidence, with native scenario reachability still unestablished.impact:other: Validation verdict integrity is affected: consumers can accept PASS without declared required proof.Evidence reviewed
What I checked:
src/evidence-ledger.mjs:28, 8c8764f443ec)src/run/finalize-record.mjs:82, 8c8764f443ec)src/selfcheck/evidence-ledger.mjs:44, 8c8764f443ec)src/evidence-ledger.mjs, f863110ac2ec)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed November 13, 2026, 7:25 AM ET / 12:25 UTC.
steipete commentedon Nov 21, 2026
Fixed in #268, with thanks to @llbohacll.
The original required-skip case reproduced on main: an executed record remained PASS with complete proof and zero missing requirements. It now becomes INCOMPLETE with one missing requirement, while the original skipped observation is retained.
Six tests through the shipped report CLI verify required and optional skips for executed, DRY-RUN, and SKIPPED records. Optional skips stay nonblocking; whole-record plans and skips retain zero missing-proof counts and zero blocking findings. The full Linux check suite and archive packaging passed, followed by exact-head cross-platform CI.
Merge commit: c584335. CI: https://github.com/openclaw/Kova/actions/runs/44852615079.