Record original MT3 real-checkpoint golden MIDI for three backends

CLAUDE.md

@@ -14,8 +14,9 @@ independent MT3 (music transcription) model implementations behind one
 (`mt3_infer/api.py`: `transcribe()`, `load_model()`, `list_models()`,
 `get_model_info()`, `download_model()`). Entry point: `mt3_infer/__init__.py`.
 
-**Scope / status:** shipped on PyPI, all three PyTorch backends (`mr_mt3`,
-`mt3_pytorch`, `yourmt3`) working. Magenta MT3 (JAX/Flax) is intentionally
+**Scope / status:** shipped on PyPI. `mr_mt3` and `mt3_pytorch` work with
+real weights under Transformers 5.13.1; `yourmt3` works with real weights
+under 4.43.4 but fails under 5.13.1. Magenta MT3 (JAX/Flax) is intentionally
 not wrapped — see README's Scope section; the `jax` extra in
 `pyproject.toml` is commented out, not a committed roadmap item. No
 training path, no batch/streaming/ONNX API exist in this repo; if that
@@ -61,15 +62,16 @@ not a modification of working vendored code.
 | `mt3_pytorch` | 2,193 | none upstream (frozen, see above) | Works; transformers v4.44+/v5 compat fixed (commit 83180b7) |
 | `yourmt3` | ~10,400 (post-strip) | Apache-2.0 | Main worktree; see gotcha below |
 
-**Gotcha: yourmt3 needs an older `transformers`.** As of this writing the
-default install resolves `transformers>=4.35.0` (currently 4.57.x in a fresh
-`uv sync`), which works for `mr_mt3` and `mt3_pytorch` but breaks
-`yourmt3`'s vendored T5 forward pass (`cache_position` ends up `None` ->
-`TypeError` in `transformers/models/t5/modeling_t5.py`). This is a
-pre-existing gap, not something touched by the Lightning-shim work -- commit
-83180b7 fixed the same class of transformers-API-drift issue for `mr_mt3`
-and `mt3_pytorch` but never touched `yourmt3/model/t5mod.py`. Until someone
-patches that, testing yourmt3 end-to-end needs an older pin, e.g.:
+**Gotcha: yourmt3 needs an older `transformers`.** The current resolved
+5.13.1 works for `mr_mt3` and `mt3_pytorch` but fails in
+`yourmt3/model/perceiver_mod.py` because the base class no longer supplies
+`get_head_mask`. A diagnostic-only restoration of that old method advanced
+to a separate tuple-index failure in `yourmt3/model/t5mod.py`. The untouched
+default branch successfully transcribed real audio under 4.43.4; its MIDI,
+audio/checkpoint hashes, and environment are committed in
+`tests/golden/mt3/`. Replay with `MT3_REAL_GOLDEN=1` and
+`MT3_CHECKPOINT_DIR=/path/to/.mt3_checkpoints` on the recorded CPU profile.
+Until compatibility is repaired, testing yourmt3 end-to-end needs 4.43.4:
 ```bash
 uv run --with "transformers==4.43.4" python your_script.py
 ```

README.md

@@ -95,9 +95,11 @@ YourMT3's [YourMT3+ paper](https://arxiv.org/abs/2407.04822) (MLSP 2024).
 - **CLI Tool**: `mt3-infer` command-line interface
 - **Reproducible**: Pinned dependencies, verified checkpoints
 
-**Current status:** all three PyTorch-based backends (MR-MT3, MT3-PyTorch,
-YourMT3) are implemented, tested, and installable from PyPI today. What's
-not yet supported: the original Magenta MT3 (JAX/Flax) backend is not
+**Current status:** MR-MT3 and MT3-PyTorch transcribe with real checkpoints
+under the currently resolved Transformers 5.13.1. YourMT3 transcribes under
+Transformers 4.43.4 but fails under 5.13.1 (`get_head_mask` is missing); its
+newer-version compatibility work is pending. The original Magenta MT3
+(JAX/Flax) backend is not
 wrapped (see Scope), and there's no batch-processing API, ONNX export, or
 streaming inference — transcription is single-file, in-process, one model
 at a time.
@@ -398,6 +400,13 @@ Extra smoke tests and tooling live in `examples/diagnostics/`:
 
 Run them via `uv run python examples/diagnostics/<script>.py` after setting any needed environment variables.
 
+An opt-in, SHA-verified real-checkpoint baseline for all three backends is
+recorded in `tests/golden/mt3/`. Set `MT3_REAL_GOLDEN=1` and
+`MT3_CHECKPOINT_DIR` to replay it with `pytest tests/test_real_golden.py` on
+the recorded CPU profile. YourMT3's original baseline uses Transformers
+4.43.4; the test intentionally fails under the current 5.13.1 until its
+compatibility is repaired. See the fixture README for exact commands.
+
 See also `docs/TROUBLESHOOTING.md` (local-only, not on GitHub) for common issues.
 
 ---

pyproject.toml

@@ -140,6 +140,7 @@ python_classes = ["Test*"]
 python_functions = ["test_*"]
 markers = [
     "network: requires internet access (checkpoint URL/repo liveness checks); excluded by default, run with -m network",
+    "real_weights: opt-in real-checkpoint CPU transcription against committed original MIDI",
 ]
 addopts = [
     "-v",

tests/golden/mt3/README.md

@@ -0,0 +1,29 @@
+# MT3 real-checkpoint baseline
+
+These MIDI files and `manifest.json` were captured from untouched default
+commit `3675ad8` on a four-second excerpt of the tracked HappySounds drum WAV.
+The CPU profile was Python 3.10.18, Torch 2.13.0+cu130, NumPy 2.2.6, and
+librosa 0.11.0, with four Torch threads. `mr_mt3` and `mt3_pytorch` used
+Transformers 5.13.1; `yourmt3` used 4.43.4 because the default 5.13.1
+environment fails before inference. YourMT3's reference MIDI was reproduced
+twice with the same SHA-256. Each checkpoint hash is recorded in the manifest.
+
+To replay without downloading, point to the three-checkpoint cache and opt in:
+
+```bash
+MT3_CHECKPOINT_DIR=/path/to/.mt3_checkpoints MT3_REAL_GOLDEN=1 \
+  python -m pytest tests/test_real_golden.py -q
+```
+
+The recorder writes a candidate and its full provenance outside the repo:
+
+```bash
+MT3_CHECKPOINT_DIR=/path/to/.mt3_checkpoints \
+  python -m tools.record_mt3_real_golden mr_mt3 --output /tmp/mt3-candidate
+```
+
+Repeat for `mt3_pytorch` and `yourmt3`. Run YourMT3 under Transformers
+4.43.4 when recreating the original baseline; compare a newer-version repair
+against that same MIDI. Review each candidate and its JSON against the
+committed source/head and manifest before replacing any fixture. The original
+fixture stays immutable during compatibility work.