Skip to content

fix: stabilize resource-profile measurement tests - #94

Merged
steipete merged 2 commits into
mainfrom
codex/oss-s6-plugin-inspector-ci
Sep 30, 2026
Merged

steipete merged 2 commits into
mainfrom
codex/oss-s6-plugin-inspector-ci

Conversation

@steipete

@steipete steipete commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes two intermittent failures in the resource-profile tests: exact floating-point subtraction from a live clock, and an allocation assertion that treats net process memory growth as gross allocation size.

User Impact

CI tests respect the resource metrics they measure. Production resource timing, memory accounting, and report precision are unchanged.

Why This Change Was Made

The JSON transport test already synthesizes resource changes. Set its elapsed-time baseline to the exactly representable fractional value 100.25, preserving the exact 10 ms assertion without depending on process startup timing.

The live test can collect pre-existing buffers during its CPU workload. Account for those observed baseline bytes in the net-growth lower bound, while still requiring positive growth and at least the retained 8 MiB allocation in the loaded snapshot. Keep the full 8 MiB disposal bound and include measured snapshots in assertion failures.

Evidence

@clawsweeper

clawsweeper Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 30, 2026
@clawsweeper

clawsweeper Bot commented Sep 30, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed September 30, 2026, 1:03 AM ET / 05:03 UTC.

ClawSweeper review

What this changes

The resource snapshot JSON transport test uses an exactly representable elapsed-time baseline so its 10 ms delta assertion is deterministic.

Regression provenance

Regression provenance — verified: source commit e2f36f1cdf94 by Vincent Koc; canonical PR #91 (blame-to-merge-commit).

Merge readiness

✅ Ready for maintainer review

Keep this PR open. Current main still contains the flaky exact-equality assertion, and the one-line fixture change addresses its floating-point cause while retaining a separate live timing test.

Priority: P2
Reviewed head: 674317459ce1361de8c8d4de4155274a6109290a

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, source-supported test repair with reported repeated validation; the hosted check's generic failure annotation does not establish a patch defect.
Proof confidence 🐚 platinum hermit (4/6) Not applicable: The changed owner is the JSON transport test, and the PR reports ten after-fix Node 22 runs plus a full package check on Crabbox. Repository admin access supports the trusted-author proof exemption; no stored-data contract changes.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The changed owner is the JSON transport test, and the PR reports ten after-fix Node 22 runs plus a full package check on Crabbox. Repository admin access supports the trusted-author proof exemption; no stored-data contract changes.
Evidence reviewed 7 items Current-main assertion: Main captures a live elapsed timestamp, adds 10, then requires the computed wall-time delta to equal exactly 10.
Introduced test change: The pinned PR delta changes only the test baseline to 100.25; the measured-child test still checks a positive live wall-time delta.
Production calculation: The production helper computes wall time by subtracting snapshot elapsed counters; this PR does not change that helper.
Findings None None.
Security None None.

How this fits together

Plugin Inspector captures timing and resource counters inside a measured process, then compares snapshots for runtime reports. This test sends snapshots through JSON and checks the resulting delta.

flowchart LR
A[Process resources] --> B[Snapshot capture]
B --> C[Synthetic elapsed baseline]
C --> D[JSON transport]
D --> E[Resource delta]
E --> F[Exact assertion]
B --> G[Live timing test]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

None.

Technical review

Best possible solution:

Keep the JSON transport test on a deterministic synthetic clock while retaining the separate measured-child test for live elapsed time.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: current main adds 10 to a live floating-point timestamp and then requires subtraction to equal exactly 10. I did not re-run the test in this read-only review.

Is this the best way to solve the issue?

Yes. An exact synthetic baseline fixes the assertion's precision dependency without changing production timing or removing live timing coverage.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a0c66caff88f.

Labels

Label changes:

  • add P2: This is a narrow test-reliability bug with limited impact outside CI.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The changed owner is the JSON transport test, and the PR reports ten after-fix Node 22 runs plus a full package check on Crabbox. Repository admin access supports the trusted-author proof exemption; no stored-data contract changes.

Label justifications:

  • P2: This is a narrow test-reliability bug with limited impact outside CI.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The changed owner is the JSON transport test, and the PR reports ten after-fix Node 22 runs plus a full package check on Crabbox. Repository admin access supports the trusted-author proof exemption; no stored-data contract changes.

Evidence

What I checked:

  • Current-main assertion: Main captures a live elapsed timestamp, adds 10, then requires the computed wall-time delta to equal exactly 10. (test/resource-profile.test.js:69, a0c66caff88f)
  • Introduced test change: The pinned PR delta changes only the test baseline to 100.25; the measured-child test still checks a positive live wall-time delta. (test/resource-profile.test.js:57, 674317459ce1)
  • Production calculation: The production helper computes wall time by subtracting snapshot elapsed counters; this PR does not change that helper. (src/resource-profile.js:30, a0c66caff88f)
  • Feature history: The merged resource-profiling change introduced the test and its exact wall-time assertion; the recorded parent lacked this test file. (test/resource-profile.test.js:69, e2f36f1cdf94)
  • Reported validation: The captured PR body reports a failing main run and ten passing Node 22 resource-profile runs plus a full package check on AWS Crabbox. These after-fix results are reported without an attached transcript. (674317459ce1)
  • Hosted check limitation: The PR's Node 22 Check failed, but its available annotation reports only exit code 1 and does not identify a failing assertion or an introduced regression. (674317459ce1)

Likely related people:

  • Vincent Koc: Raw commit e2f36f1 adds test/resource-profile.test.js:69 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: e2f36f1cdf94; files: test/resource-profile.test.js)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete steipete changed the title fix: stabilize resource-profile elapsed-time test fix: stabilize resource-profile measurement tests Sep 30, 2026
@clawsweeper

clawsweeper Bot commented Sep 30, 2026

Copy link
Copy Markdown

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix: stabilize resource-profile measurement tests This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@steipete
steipete merged commit 19f481f into main Sep 30, 2026
5 of 6 checks passed
@steipete
steipete deleted the codex/oss-s6-plugin-inspector-ci branch September 30, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant