Skip to content

Add pluggable TUI renderer backends (mattervis.native-v1 default) - #241

Merged
SchrodingersCattt merged 3 commits into
mainfrom
cursor/pluggable-tui-renderers-69fb
Oct 25, 2026
Merged

SchrodingersCattt merged 3 commits into
mainfrom
cursor/pluggable-tui-renderers-69fb

Conversation

@SchrodingersCattt

@SchrodingersCattt SchrodingersCattt commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #240

Summary

MatterVis now has one TUI frame-renderer contract. Static CLI frames and live session frames both go through render_picture. The built-in backend id is mattervis.native-v1. It still calls compose_frame with the host crystal and camera, so the default picture stays byte-for-byte on the frozen fixtures. state_fingerprint does not include renderer identity, and default observations do not grow a renderer field.

An operator selects another backend with TerminalSession(..., renderer=...) or mat-vis tui --renderer ID --renderer-config FILE. Backends are installed wheels in the mattervis.tui.renderers entry-point group. A missing id, wrong API, bad pin, unsupported option, malformed frame, or plugin exception fails closed. Dispatch does not draw the native picture instead. Renderer choice is episode configuration, not a set_display action.

External render methods receive RenderRequest only. The host builds RenderedFrame and the hashes. CrystalIR, MolCrysKit chemistry, gold, and scorer fields are not on that request. observation_profile="model" is opt-in and strips the source title, chemistry warnings, and formula/composition scope fields.

examples/mattervis_mark_renderer is a separate package (example.marks-v1). It draws depth digits and bond dots and never element symbols. It is not copied into mat_viewer.

Picture bytes are the cross-CPython byte-identity contract for the default native backend. Viewport floats can differ by one ULP across interpreters, which changes state_fingerprint. The frozen episode is compared with that tolerance. Within one process, reset still restores the fingerprint and a camera change still replaces it. Viewport values are not rounded before hashing, so default fingerprints stay on the historical algorithm.

Acceptance

  • Frozen 80×24 and 100×36 frames (ascii7 and Unicode, depth on/off), molecule and framework pictures, a boundary CIF-style case, and water match the current compositor byte for byte.
  • The observe → orbit → set_display → reset episode keeps exact frame text and frame.sha256, legacy keys, and within-run fingerprint equality. The full JSONL document is not required to be byte-identical across CPython versions.
  • Static and active paths share one dispatch and one request. The native adapter is the only renderer that sees the host crystal, and only inside dispatch.
  • The installed marks backend changes the screen for the same source, viewport, camera, and actions. Camera state and state_fingerprint stay aligned with native.
  • Missing or incompatible backends, pin mismatches, crashes, and malformed frames fail closed.
  • Canary and poisoned-chemistry tests keep hidden chemistry out of the request and out of model-profile observations. Bond image translations and graph-level provenance (provided / inferred / unspecified) are preserved. provided_only disables inferred edges without mutating the source graph.
  • ascii7 external frames are printable 7-bit text. Session screen hashes stay the padded-screen hash. Replay identity is replay_manifest() plus, on non-default episodes, input_digest, view_digest, and unpadded content_sha256.
  • Changing the renderer does not change action names, orbit/reset meaning, or the allowed JSONL tool set. MatterVis still does not own task identity or a scorer.
  • Public contract: docs/agents/tui_renderer_api.md. CLI notes and developer dispatch notes are updated. Polar-orbit tests stay in the suite.

What was and was not validated

Validated here: default compositor picture byte identity, fail-closed selection, charset bounds, camera and action stability across backends, and canaries for hidden chemistry fields.

Not scientifically validated: an external renderer can still infer chemistry from coordinates. This change does not validate a benchmark, a scorer, a task split, or a human reading of either picture. A Python Protocol is not a sandbox; untrusted renderers need a separate process. This does not implement research harness issue 17 and does not close #163. No subprocess renderer is shipped.

Test plan

  • python -m pytest tests/tui/test_tui_renderer_backend.py tests/tui/test_tui_session.py tests/tui/test_tui_polar_escape.py
  • python -m pytest tests/ -n 4 --dist loadfile -m "not browser" on Python 3.12
  • Full tests on Python 3.10, 3.11, and 3.12, plus Camera WebGL, on 1c89bcc
Open in Web Open in Cursor 

cursoragent and others added 2 commits October 25, 2026 14:05
Keep compose_frame as the built-in mattervis.native-v1 backend and send both static and session frames through one display request. External backends are selected by entry point and fail closed, without changing default JSONL fingerprints.

Co-authored-by: Ming-Yu Guo <guomy26@mail2.sysu.edu.cn>
Resolve the CLI renderer in the terminal CLI helper and attach it to the controller after construction so cli.py and the view controller stay within their line caps.

Co-authored-by: Ming-Yu Guo <guomy26@mail2.sysu.edu.cn>

@SchrodingersCattt SchrodingersCattt left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs-changes (CI not green; do not merge)

Head 50a93099. Author PR → COMMENT only (cannot self-APPROVE).

CI (9 checks)

Check Result
Fast 3.12 (min / latest) success
Build / Minimal smoke / Benchmark success
Full tests Python 3.12 success
Full tests Python 3.11 failure
Full tests Python 3.10 failure
Camera WebGL regression still in progress at review time

Both Full 3.10 and 3.11: 1 failed, 1981 passed, 49 skipped.

Blocker

tests/tui/test_tui_renderer_backend.py::test_frozen_water_and_episode

Exact string compare of episode_ascii7_80x24.jsonl fails on 3.10/3.11. Diff is not picture bytes (frame.sha256 / screen text match the golden). It is viewport float ULP in the JSONL observation:

  • golden (likely frozen on 3.12): y_max / y_min …1549
  • 3.10/3.11: same values …15488

That shifts state_fingerprint and fails assert episode == golden.

This is a real cross-version stability bug in the acceptance golden, not a flake. Default native picture identity still looks fine; the frozen episode JSONL is over-specified.

Concrete fix list (for cloud agent)

  1. Stop byte-comparing the full episode JSONL across CPython versions in CI.
  2. Keep strong checks that already match #240:
    • frozen frame .txt / index.json sha256
    • within-run: orbit changes fingerprint; reset restores fingerprint + frame.sha256
    • legacy observation keys; no renderer / observation_profile on default
    • display toggles after set_display
  3. If an episode golden is kept: compare with a float-tolerant JSON walk (or round viewport scale/bounds before fingerprint materialization), and document that picture bytes are the byte-identity contract, not every float bit in viewport.
  4. Re-run Full on 3.10 and 3.11 (and wait for WebGL) before asking for re-review / merge.

#240 acceptance map (code review; blocked by CI)

Criterion Assessment
Native default byte-identical / state_fingerprint unchanged Frames + default session fingerprint logic look correct; episode golden over-asserts float bits
Single dispatch static + active Yes — both call render_picture
Entry points; fail-closed; no silent native fallback Yes — reserved mattervis.native-v1, pin/API/malformed/crash covered
No CrystalIR/chemistry/gold on RenderRequest; model profile opt-in Yes — assert_public_payload, canaries, observation_profile="model"
External package separate (examples/mattervis_mark_renderer) Yes — own wheel + entry point
Docs docs/agents/tui_renderer_api.md + CLI/dev notes
Tests: golden / fail-closed / canary / polar suite retained Present; episode golden fails on 3.10/3.11

Nits (non-blocking)

  • resolve_renderer_binding assigns stored_backend twice (redundant, harmless).
  • Plugin RuntimeError propagates raw (still fail-closed; no native substitute).

Verdict: needs-changes. Fix the episode golden / comparison for 3.10+3.11, get Full green on those, then re-request review. Will not squash-merge until CI is 9/9 success.

Picture text and frame.sha256 stay exact. Viewport floats may differ by one ULP, which changes state_fingerprint, so the episode trace is checked with a float tolerance and within-run fingerprint equality.

Co-authored-by: Ming-Yu Guo <guomy26@mail2.sysu.edu.cn>
@cursor

cursor Bot commented Oct 25, 2026

Copy link
Copy Markdown

Ready for re-review.

The 3.10/3.11 failure was the frozen episode JSONL, not the picture. y_min/y_max can differ by one ULP across CPython builds, and that changes state_fingerprint while frame.text and frame.sha256 stay the same.

The episode compare now keeps picture bytes and frame.sha256 exact, allows that float tolerance, and still checks within-run fingerprint equality, reset, and legacy keys. Viewport values are not rounded before hashing, so the default fingerprint algorithm is unchanged.

CI on 1c89bcc is green: Full tests 3.10, 3.11, and 3.12, plus Camera WebGL.

@SchrodingersCattt SchrodingersCattt left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — approve-ready (author PR → COMMENT only)

Head 1c89bcc. Prior blocker (Full 3.10/3.11 test_frozen_water_and_episode) is fixed.

CI on this head (9/9 success)

Check Result
Fast 3.12 (min / latest) success
Build / Minimal smoke / Benchmark success
Full tests Python 3.10 / 3.11 / 3.12 success
Camera WebGL regression success

Fix check

_assert_episode_matches_golden + _assert_json_close: picture / frame.sha256 stay exact; viewport floats tolerate ULP; state_fingerprint not cross-CPython byte-compared; within-run orbit≠reset==initial fingerprint still asserted. Hash algorithm unchanged (no pre-hash rounding). Unit test_episode_golden_tolerates_viewport_ulp_not_picture_bytes covers the contract.

#240 acceptance (unchanged from prior map; now unblocked by CI)

Native default frames + single render_picture dispatch, fail-closed entry points, chemistry-free RenderRequest, model profile opt-in, separate marks example package, docs, polar suite retained — all still look good.

Verdict: merge-ready. Squashing next (Closes #240).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants