Skip to content

test(model-info): cover cache usage observation boundaries - #532

Merged
tt-a1i merged 2 commits into
openpi-dev:mainfrom
ooiuuii:test/cache-usage-evidence-156
Sep 12, 2026
Merged

test(model-info): cover cache usage observation boundaries#532
tt-a1i merged 2 commits into
openpi-dev:mainfrom
ooiuuii:test/cache-usage-evidence-156

Conversation

@ooiuuii

@ooiuuii ooiuuii commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Refs #156; this does not close the umbrella issue. The diagnostic core already shipped in #372. Remaining evidence needs to distinguish provider field normalization, OpenPI's existing observational contract, and genuine multi-product Session experiments.

Value

Make the current cache observations reviewable and regression-tested without presenting cache misses as verified TTL expiry, billing evidence, or a proven reason for cache invalidation.

Approach

  • Add a source-pinned matrix for Pi 0.85.1 (d981de1229ef899957bbe968bc8dcda02a21f477) and a governed research record distinguishing facts, synthetic evidence, and unfinished experiments.
  • Add 16 deterministic cases to the existing model-info tests: implicit/unknown provider boundaries, exact warm-prefix threshold, normalized prompt arithmetic, malformed counters, consumed correlations, and Session restart versus persisted usage.
  • Keep all production code, provider classification, configuration, dependencies, and UI unchanged. Tests deliberately exercise the existing implementation; this is not a runtime bug-fix claim.

Validation

Base: OpenPI 5bf2fe29e52801d79826c2eb573be403f53285e5. Candidate: b409b2b3963f2d017864d1e6e55ac0104c4a7860. Local Windows, Node 24.19.0, Bun 1.3.14, frozen lockfile.

Proof: deterministic observation contract

node --test --experimental-strip-types tests/extensions/model-info/cache-diagnostics.test.ts tests/extensions/model-info/index.test.ts tests/extensions/model-info/session-metrics.test.ts
bun run check
bun run test
  • Focused model-info tests: 39/39 passed, including 23 pre-existing and 16 added. The checked-in fixtures are small, hand-authored vectors using the actual tracker and extension event harness.
  • bun run check: passed (format, lint, typecheck, configuration/discipline checks, Web build).
  • git diff --check and new local documentation links: passed.
  • Full bun run test: failed locally, not claimed green. Initial Node group: 1,430 passed, 4 failed, 10 skipped. The four failures were in unchanged git-info/process, setup/integration, shared/setup-config, and workflows/narrator tests (timeouts/Windows file-watcher assertion). The same four files passed 41/41 when rerun serially; that does not establish a unique cause or erase the full-run failure.
  • Separately executed remaining Vitest group: 9 files / 130 tests passed. The Windows background-terminal serial group had 7 observed failures and then hung; it was stopped after verifying its test-process identity. Its remainder is unverified, not a passing full suite. No unrelated process-lifecycle fixes were folded into this PR.
  • Scoped Codex review via autoreview --mode local --engine codex --prompt-file <frozen-scope>: clean, no actionable findings. An earlier attempt ended on a usage limit without a verdict; the completed retry reviewed the same four-file candidate.

Limits / still open

No provider calls, real OpenPI/Bare Pi/OMP Session recordings, cost/latency experiment, false-positive-rate measurement, or verified TTL cause is represented by these fixtures. Those #156 acceptance items remain open. No user chats, credentials, or raw Session files are published. Upstream CI remains the separate repository-wide validation gate.

Impact

  • User-visible behavior: none.
  • Model-visible context/tools: none.
  • Runtime/lifecycle and persisted config/data: unchanged; existing behavior is only documented and tested.
  • Compatibility/risk: no peer/lock changes and no new dependencies. This is an independently useful test/documentation slice based on main, with no dependency on the sidebar PR. It intentionally preserves observability: 从累计 cache 命中率升级到逐 turn 的 prompt-cache 失效诊断 #156's remaining evidence boundary.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 11, 2026
@ooiuuii

ooiuuii commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

CI receipt for b409b2b3963f2d017864d1e6e55ac0104c4a7860 (run 34587171375):

  • Linux Node 22.19.0 / 24 / 26 each passed bun run check, the full Node suite (1,541 passed, 0 failed, 1 skipped) and Vitest (130 passed). The new model-info cases passed.
  • The separate Web E2E job passed.
  • The Windows job passed, including both process lifecycle and full-suite isolation. The CI run is now complete; its overall result is still failure because of the packaging step below.
  • All three Linux jobs then failed in Smoke-test packed Web CLI with locked Pi host, at npm install, not in the test suites. npm selected @effect/platform-node@4.0.0-rc.114 and reported ETARGET: No matching version found for effect@^4.0.0-rc.114. Node 24 job log.

This PR changes only the two model-info test files and the research record/index; package.json, lockfile, installation commands, and CI are unchanged. The failure points to floating packed-install dependency resolution outside this contribution. I have not changed the dependency policy, retried the workflow, or represented the overall CI as green. The package-install gate still needs resolution separately.

@tt-a1i tt-a1i left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head 1db7963 after updating to the Effect cohort fix. Standards: no actionable findings. Spec: the added synthetic tests and source-scoped cache usage record distinguish observation from verified causes, and explicitly leave real comparative Session evidence open. This adds no runtime or model behavior. No P0/P1 found. Merge remains conditional on all required current-head CI.

@tt-a1i
tt-a1i merged commit c616d19 into openpi-dev:main Sep 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants