ICL voice cloning randomly degenerates (no EOS, 24s garbage) — root cause: ref_text not covering full ref_audio; confirmed not a port bug via official PyTorch A/B #921

Summary

ICL voice cloning (ref_audio + ref_text) randomly produces degenerate output: ~24s of noisy garbage instead of the expected ~5.6s, the decoder never emits EOS, and step-by-step logprob instrumentation shows it falls into a single-token attractor (token 302 sampled 59 consecutive times at >90% probability; EOS median rank 495/3072, entering top-50 in only 6 of 300 steps). When it happens, no sampling parameter rescues it (temperature 0.2–0.9, top_k, top_p, repetition_penalty all ineffective or worse).

Root cause found: it is not a port bug — it is misaligned ICL prefill caused by ref_text not covering the full reference audio. With text tokens aligned to only the first ~6s of a 10s clip while codec frames span the whole clip, the model sits at the edge of its input distribution and randomly collapses into the degenerate attractor.

Evidence (official PyTorch A/B)

I ran the official Qwen/Qwen3-TTS inference package (qwen_tts, pinned transformers==4.57.3, official Qwen/Qwen3-TTS-12Hz-1.7B-Base weights, CPU) with the same text, same ref_audio, same truncated ref_text:

The official implementation flies off more consistently than the MLX port under the same misaligned input (0/3 vs 1/3 success) — so the port is faithful, arguably slightly more robust. Everything else I compared line-by-line matched: prefill construction (text+codec_pad, codec+tts_pad, sum-of-codebooks, role tokens, codec_prefix offset), decode loop, RoPE position ids (official get_rope_index degenerates to 1D cumsum, same as MLX).

Fix / usage guidance

The fix is correct usage, not code: ref_text must be a complete transcription of ref_audio (or omitted so the built-in auto-transcription path handles it). Verified stable with 5/5 runs on stock qwen3_tts.py.

Possible upstream improvements (optional):

1. Document the requirement prominently in the voice-cloning docs/example.
2. Runtime sanity check: estimate spoken duration of ref_text vs ref_audio duration and warn on large mismatch.
3. Route "ref_audio without ref_text" through STT automatically (already exists in generate.py for some paths).

Possibly related: #910 (pace acceleration) and #914 — misaligned prefill may produce a spectrum of pathologies depending on how truncated the text is; my truncation was severe (~60% coverage), producing collapse rather than drift.

Environment

• mlx-audio PyPI 0.5.0 (stock qwen3_tts.py, no local patches)
• mlx-community/Qwen3-TTS-12Hz-1.7B-Base-bf16 (4.2G)
• M2 Ultra, macOS
• Official A/B: Qwen/Qwen3-TTS repo qwen_tts package, transformers==4.57.3, official Qwen/Qwen3-TTS-12Hz-1.7B-Base weights