Skip to content

fix(ci): ci --config with a plugin config skips requested runtime capture - #125

Merged
vincentkoc merged 2 commits into
openclaw:mainfrom
KrasimirKralev:fix/ci-config-runtime
Nov 4, 2026
Merged

vincentkoc merged 2 commits into
openclaw:mainfrom
KrasimirKralev:fix/ci-config-runtime

Conversation

@KrasimirKralev

@KrasimirKralev KrasimirKralev commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What Problem This Solves

plugin-inspector ci --config plugin-inspector.config.json --runtime --mock-sdk --allow-execute reports PASS and exits 0 but never runs runtime capture, so a CI job that asked for runtime registration checks passes without them.

  • Problem: when ci gets --config, runCiCompatibilityReport (src/cli.js) always takes the fixture-suite path (inspectCompatibilityFixtureSet) and never calls runPluginCheck, even when the file is a plugin-root config like the one init writes. --runtime, --mock-sdk and --allow-execute (all listed for ci in --help) are dropped without a word.
  • What did NOT change: fixture-suite configs (a top-level fixtures array) take exactly the same path as before; ci without --config, check, inspect and report are untouched; no report field or finding code changes. Without --runtime the plugin-root ci --config output is identical to main (same report JSON apart from generatedAt, same file list).
  • Requested by a maintainer? No, found while checking that ci honours the flags its help lists.

User Impact

Plugin authors who point ci at their plugin-inspector.config.json and ask for --runtime now get the runtime capture report (plugin-inspector-runtime-capture.json/.md), the same as ci without --config, instead of a PASS that silently skipped it.

Why This Change Was Made

  • What changed: runCiCompatibilityReport loads the --config file as before; when it is a plugin-root config (isPluginRootConfig) it now runs runPluginCheck with that config path and the config's directory as the plugin root (the same root and report directory the old path used), so runtime capture and the config's capture settings apply. Fixture-suite configs keep the fixture path.
  • Root cause: the --config branch was written for fixture-suite configs; loadInspectorConfig also accepts plugin-root configs (it normalises them into a one-fixture set), so they were sent down the fixture path that has no runtime capture.
  • Missing detection / guardrail: the existing ci --config tests never passed --runtime. A new case in test/cli.test.js runs ci --config <plugin-root config> --runtime --mock-sdk --allow-execute and expects one captured registration; with the src/cli.js change reverted the whole suite goes from 835 passing to 834 passing / 1 failing (only the new case). npm test 835/835 and npm run release:contents pass on the branch. CHANGELOG line under ## Unreleased.

Evidence

  • Behavior or issue addressed: ci --config <plugin-root config> --runtime skipping runtime capture while exiting 0.
  • Real environment tested: Linux, Node 24, npm ci --ignore-scripts; a scratch plugin root whose entry registers one tool, with its config written by plugin-inspector init; upstream/main 8ec7d35 in a worktree versus this branch.
  • Exact steps or command run after this patch: node src/cli.js ci --config plugin-inspector.config.json --no-openclaw --runtime --mock-sdk --allow-execute from the plugin root, then a look at reports/.
  • Evidence after fix:
$ node src/cli.js init --plugin-root <plugin>
wrote plugin-inspector.config.json

# upstream/main 8ec7d35
$ node src/cli.js ci --config plugin-inspector.config.json --no-openclaw --runtime --mock-sdk --allow-execute
Status: PASS
Breakages: 0
exit 0
$ ls reports | grep runtime-capture
(none)

# this branch
$ node src/cli.js ci --config plugin-inspector.config.json --no-openclaw --runtime --mock-sdk --allow-execute
Status: PASS
Breakages: 0
exit 0
$ ls reports | grep runtime-capture
plugin-inspector-runtime-capture.json
plugin-inspector-runtime-capture.md
runtime capture: capturedCount 1, registrations 1

# control: same flags without --config (plugin-root mode), upstream/main
Status: PASS
plugin-inspector-runtime-capture.json
capturedCount 1
  • Observed result after fix: running node src/cli.js ci --config … with --runtime on main writes no runtime capture at all; on this branch it writes plugin-inspector-runtime-capture.json/.md with the one registration, matching what ci already does without --config (the control).
  • Crabpot source-mode smoke: crabpot 20689ea with every plugin submodule checked out, CRABPOT_PLUGIN_INSPECTOR_CLI=source CRABPOT_PLUGIN_INSPECTOR_DIR=<checkout> npm run plugin-inspector:smoke, run once against this branch and once against upstream/main 8ec7d35. The smoke drives report --config <fixture suite>, which this change does not touch, and both runs print the same result line for line (Status: FAIL, Fixtures: 59, Breakages: 35, the same top findings; only the Node PID in an experimental-feature warning differs). The 35 missing-expected-seam breakages are the same on main in this local setup, so they are not caused by this PR.
  • What was not tested: the smoke's other modes (--resources, --retirements, packed package).
  • Proof limitations or environment constraints: none beyond the above.

@KrasimirKralev
KrasimirKralev requested a review from a team as a code owner October 30, 2026 07:06
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. labels Oct 30, 2026
@clawsweeper

clawsweeper Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

Route plugin-root CI configs through the existing plugin check so requested runtime capture runs.

Example: Run plugin-inspector ci --config plugin-inspector.config.json --no-openclaw --runtime --mock-sdk --allow-execute for a plugin registering one tool.

  • Before: CI reports PASS without producing runtime-capture artifacts.
  • After: CI produces runtime-capture JSON and Markdown recording one captured entrypoint and one registration.

Review scores

Measure Result What it means
Overall readiness 🦞 diamond lobster (5/6) A focused repair with shipped-CLI behavior evidence, a meaningful regression case, preserved report contracts, and completed comparative downstream smoke validation.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The contributor's Linux/Node 24 terminal trace exercises the shipped CI CLI using an init-generated config and explicit runtime consent, showing runtime artifacts with one registration after the repair and none on main; the no-config control already captures successfully.
Patch quality 🦞 diamond lobster (5/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A plugin author requests runtime registration checks in CI and receives PASS even though those checks never run.
Reason: Honoring advertised runtime flags and existing capture settings restores the established contract without adding an option or public API.

Merge readiness

✅ Ready for maintainer review

This PR fixes a current CI routing bug, and no actionable defects remain; the updated Crabpot comparison resolves the previous review request.

Priority: P2
Reviewed head: 60e289cbc68662099eea359db3ab00ebf99b6ee5

Before merge

None.

Findings

None.

Agent review details

How this fits together

The CLI accepts plugin-root or fixture-suite configs, dispatches compatibility inspection and optional runtime capture, and writes compatibility reports and CI artifacts.

flowchart TD
CLI[CI command] --> Config[Load and classify config]
Config --> Suite[Fixture suite inspection]
Config --> Plugin[Existing plugin check]
Plugin --> Consent[Execution consent check]
Consent --> Capture[Runtime capture]
Suite --> Reports[Compatibility and CI artifacts]
Plugin --> Reports
Capture --> Runtime[Runtime artifacts]
Loading

Technical review

Best possible solution:

Land the focused routing repair using the existing config discriminator and plugin-check owner.

Do we have a high-confidence way to reproduce the issue?

Current-main source deterministically bypasses runtime capture for explicit configs, corroborated by the contributor's before/after CLI trace; this read-only review ran no target code.

Is this the best way to solve the issue?

The patch repairs dispatch at its existing owner, preserves fixture-suite behavior and output roots, and avoids duplicating runtime capture.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 8ec7d359fd78.

Provenance checked

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

  • Current released behavior: The main explicit-config branch returns after fixture inspection and bypasses runPluginCheck; v0.4.5 resolves to this same main revision. (src/cli.js:274, 8ec7d359fd78)
  • Focused dispatch repair: The introduced diff preserves fixture-suite handling and sends plugin-root configs through runPluginCheck with the resolved config path and original config directory. (src/cli.js:279, 60e289cbc686)
  • Existing capture and consent owner: runPluginCheck honors CLI overrides and config capture settings, rejects execution without consent before invoking runtime capture, and propagates capture failures. (src/api.js:222, 60e289cbc686)
  • Contributor behavior and downstream evidence: The full PR body records Linux/Node 24 shipped-CLI before/after output and a no-config control, plus Crabpot source-mode runs against both revisions with identical 59-fixture, 35-breakage baseline results; the previously requested smoke has therefore been performed. (60e289cbc686)
  • Regression coverage: The added test invokes the shipped CI CLI from outside the plugin directory and reads the resulting capture artifact; the contributor reports that reverting only the production repair makes this case fail. (test/cli.test.js:340, 60e289cbc686)
  • Original intent and feature history: The merged fix(cli): wire ci command to ci summary artifacts #13 established compatibility-backed CI summaries; feat(cli): streamline plugin root checks #10 established explicit runtime flags and config-driven capture. Local log and blame identify shared ownership. Some historical API blobs were unavailable to local show, pickaxe, and blame; GitHub commit patches supplied the original capture rationale. (src/cli.js:277, 8ec7d359fd78)

Likely related people:

  • Solomon Neas: Raw commit 93e52ad adds src/cli.js:145 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 93e52adab2ba; files: src/cli.js)
  • Vincent Koc: Raw commit 0fc9697 adds src/cli.js:127 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 0fc9697a14cd; files: src/cli.js)
  • Patrick Erichsen: Raw commit ea1100d adds src/cli.js:274 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: ea1100dfffd3; files: src/cli.js)

Review metrics

Metric Value Why it matters
Change scope Production +17/-8 lines; tests +27; changelog +2 The small dispatch change reuses existing owners, and the CLI regression case covers the previously skipped path.

Labels

Label changes:

  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

  • P2: Explicit-config CI jobs can silently omit requested runtime checks, with impact limited to this invocation mode.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)
  • reviewed 2026-10-30T07:19:08.600Z sha 60e289c :: needs changes before merge. :: none

Reviewed October 30, 2026, 3:26 AM ET / 07:26 UTC (Revision 2).

@KrasimirKralev

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. clawsweeper:intake:bug Preliminary submission classification; full source review pending. labels Oct 30, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Maintainer validation at d9346d2:

  • Merged current main 8ebe4f8 into this branch; only CHANGELOG.md conflicted, and both entries were retained. The final diff is still the original CLI routing fix, regression test, and changelog entry.
  • Local Node 26 suite: 848/849 passed; the process-profile stalled-ps test hit a missing PID-file timing failure. Its isolated retry passed all 28 tests. Package contents validation passed.
  • Crabpot source-mode smoke was run against both candidate and current main using the explicit source override. Both reported 60 fixtures and 83 breakages; full JSON reports are identical after removing generatedAt. These existing fixture failures are not introduced by this change. The source override exercises this revision without changing the published package pin.
  • Awaiting the refreshed hosted checks before squash merge. No release or publish is authorized.

@vincentkoc
vincentkoc merged commit 6eb9115 into openclaw:main Nov 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants