Skip to content

fix(skills): recover interrupted RPI critiques without bypassing readiness gates - #2894

Open
Jamie Kim (jkim323) wants to merge 8 commits into
mainfrom
fix/2891-rpi-critique-recovery
Open

Jamie Kim (jkim323) wants to merge 8 commits into
mainfrom
fix/2891-rpi-critique-recovery

Conversation

@jkim323

@jkim323 Jamie Kim (jkim323) commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Allow an interrupted RPI planning critique to recover once with explicit task-specific approval, while preserving the evidence and readiness checks required before implementation.

  • Distinguish the immediately dispatched critique worker from a later caller replaying a saved started reservation.
  • Keep recovery authorization in rpi-plan. Require original-worker inactivity, reconciled saved files and explicit approval; preserve original output and reconcile late results. Terminal Complete, Partial and Blocked results cannot be retried.
  • Keep standalone first critiques self-contained. Parent-dispatched critiques resolve the canonical planner reference without assuming sibling installation; missing guidance stops assessment without a standalone fallback.
  • Ground critique findings in the whole supplied plan: distinguish demonstrated omissions and contradictions from missing detail in abbreviated excerpts, without demanding requirement restatement.
  • Align the RPI Agent, planning and critique skills, templates and usage documentation. Add 15 advisory regression cases covering dispatch, recovery safeguards, selective installation and evidence-grounded findings.

Related Issue(s)

Fixes #2891

Type of Change

Select all that apply:

Code & Documentation:

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Infrastructure & Configuration:

  • GitHub Actions workflow
  • Linting configuration (markdown, PowerShell, etc.)
  • Security configuration
  • DevContainer configuration
  • Dependency update

AI Artifacts:

  • Reviewed contribution with hve-builder and addressed all actionable findings
  • Copilot instructions (.github/instructions/*.instructions.md)
  • Copilot prompt (.github/prompts/*.prompt.md)
  • Copilot agent (.github/agents/*.agent.md)
  • Copilot skill (.github/skills/*/SKILL.md)
  • Copilot hook (.github/hooks/*/*.json)
  • Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

  • Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
  • Skills: Must include both bash and PowerShell scripts. See Skills.
  • Model Versions: Contributions MUST target models listed in the model catalog (scripts/linting/model-catalog.json) whose provider appears in providerAllowlist and whose status is ga or preview. Run npm run lint:models to validate references.
  • See Agents Not Accepted and Model Version Requirements.

Other:

  • Script/automation (.ps1, .sh, .py)
  • Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

"Resume planning for this task. The initial critique was interrupted and its result is unavailable. Check whether recovery is eligible."

Execution Flow:

The planner reconciles existing evidence, verifies that the original worker ended and checks saved candidate identity. If eligible, it asks for task-specific approval before reserving one distinct recovery attempt. Existing terminal evidence or unresolved prerequisites stop recovery.

Output Artifacts:

Expected output, not a recorded successful run: an eligibility summary or a concrete blocker. After approval and successful assessment, the plan's Critique Disposition retains both attempt records and points to a separate recovery critique with execution status, verdict and findings.

Success Indicators:

Original evidence is preserved, only the authorized current worker assesses, and readiness requires actual assessment evidence and resolved findings. Generic resume or automatic mode alone does not authorize recovery.

For detailed contribution requirements, see:

Testing

Earlier targeted preflight against committed head 5c919c858498534508a687ab1bcf5cfff2735e5c:

  • Passed committed-diff whitespace validation with git diff --check from the merge base to HEAD.
  • Passed Validate-MarkdownFrontmatter.ps1 with schema validation and warnings-as-errors for all 11 changed Markdown files.
  • Passed Validate-SkillStructure.ps1 for rpi-plan-critique and rpi-quick.
  • Passed npm run plugin:validate and npm run docs:generate:check; documentation drift and attention counts are zero. Earlier npm run plugin:sync and npm run docs:generate also reported current projections.
  • Passed the exact-file Test-EvalSpecCompliance schema/executor/backlink check for all 246 stimuli, and recomputed the normalized hashes for the 13 added cases. These knowledge-level cases do not execute native dispatch, persistence or race handling.

The subsequent HB-B01 correction, now committed as 5e43429c12dd5bf8349b1a3c9f06cd5d860cc00b, passed the exact-file schema/executor/backlink check for 248 stimuli, preservation checks for the prior 246 cases, and normalized hash and safety checks for the two additions. Targeted frontmatter/critique-skill structure checks, plugin validation and documentation synchronization checks also passed. These are mechanical checks, not behavior acceptance. The committed-diff whitespace check was rerun successfully at this head while refreshing this description.

HVE Builder Review

hve-builder review mode ran on 2026-09-10 against 5c919c858498534508a687ab1bcf5cfff2735e5c:

  • Independent static review: Complete / Pass, with no findings.
  • Independent behavior assessment: Partial / Revise, covering 18 scenarios with 21 independent parameterized tasks. It identified two Medium findings: HB-B01, unsupported task-level requirement restatement; and HB-B02, incomplete reservation-lock and stop-transition persistence fixtures. The overall review outcome was Deferred, not approval.
  • HB-B01 was corrected in 5e43429c12dd5bf8349b1a3c9f06cd5d860cc00b with scoped finding-selection guidance, an updated usage example and two advisory regression cases. Independent behavioral verification of that correction remains pending.
  • A separately authorized HB-B02 follow-up at 5e43429c12dd5bf8349b1a3c9f06cd5d860cc00b returned Complete / Pass, with zero findings. Three focused scenarios exercised concrete reservation locks, successful two-write stop persistence and a failed-second-write control. HB-B02 is resolved at simulation fidelity; this did not require source changes or a real critique retry.

Behavior runs used contained, independently graded High-profile proxy simulations. State writes, host records and dispatches were emulated; native execution and equivalence to the target's selected session model were not verified. The focused HB-B02 Pass does not replace the earlier full-review result or close HB-B01. Original reports and follow-up evidence were retained locally; human-review attestations remain unchecked.

Checklist

Required Checks

  • Documentation is updated (if applicable)
  • Files follow existing naming conventions
  • Changes are backwards compatible (if applicable)
  • Tests added for new functionality (if applicable)

AI Artifact Contributions

  • Used hve-builder review mode to review contribution
  • Addressed all actionable findings from the hve-builder review
  • Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

  • Local validation aggregate: npm run validate:local
  • Documentation validation (if docs changed): npm run validate:docs
  • Spell checking: npm run spell-check
  • Link validation: npm run lint:md-links

Security Considerations

  • This PR does not contain any sensitive or NDA information
  • Any new dependencies have been reviewed for security issues
  • Security-related scripts follow the principle of least privilege

No dependency manifests, lockfiles, executable scripts or permission declarations change. Recovery remains planner-authorized and requires explicit task-specific consent; missing evidence does not imply approval or Pass.

@jkim323 Jamie Kim (jkim323) self-assigned this Sep 11, 2026
@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.01%. Comparing base (9bf1a30) to head (5841544).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2894      +/-   ##
==========================================
- Coverage   83.11%   83.01%   -0.11%     
==========================================
  Files         187      186       -1     
  Lines       35331    34939     -392     
  Branches      231      231              
==========================================
- Hits        29365    29004     -361     
+ Misses       5892     5861      -31     
  Partials       74       74              
Flag Coverage Δ
docusaurus 91.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jkim323 Jamie Kim (jkim323) changed the title feat(docs): enforce authored prompt guidance fix(skills): recover interrupted RPI critiques without bypassing readiness gates Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Eval Execution

Status: Failed — 1 spec(s) block merge

  • Artifacts evaluated: 4
  • Specs run: 4
  • Assertions passed: 73
  • Assertions failed (blocking): 0
  • Assertions failed (advisory): 8
  • Failed specs (merge-blocking): 1
Artifact Kind Status Specs Passed Failed (blocking) Failed (advisory)
rpi-agent agent ⏭️ skipped 1 10 0 5
rpi-plan-critique skill ⚠️ advisory-fail 1 26 0 1
rpi-plan skill ⚠️ advisory-fail 1 28 0 2
rpi-quick skill ✅ pass 1 9 0 0

Legend — ✅ clean · ⚠️ advisory failures only (non-blocking) · ⏭️ skipped · ❌ merge-blocking failure

Only Failed specs (merge-blocking) gates this PR. Advisory assertion failures are signal-quality checks captured during iteration; review them, but they do not block merge and may be acceptable.

@jkim323
Jamie Kim (jkim323) marked this pull request as ready for review September 11, 2026 05:45
@jkim323
Jamie Kim (jkim323) requested a review from a team as a code owner September 11, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: recover interrupted RPI planning critiques without bypassing readiness gates

4 participants