Voxtral TTS: decorative single quotes can add hallucinated speech

[psiberfunk]

Voxtral TTS: decorative single quotes can add hallucinated speech

Environment

• Hardware: Apple M5 Pro (arm64)
• OS: macOS 26.6.2 (build 25G83)
• Python: 3.12.14
• mlx-audio: 0.5.0
• mlx: 0.32.2
• mlx-metal: 0.32.2
• mistral-common: 1.11.7
• Model: mlx-community/Voxtral-4B-TTS-2603-mlx-bf16
• Model revision: dd85c02adbae551f5bb29ded35ee60ccdfb90927
• Voice: neutral_male
• Seed: 42
• max_tokens: 6144
• Non-streaming model.generate() call

Reproduction

import wave

import mlx.core as mx
import numpy as np
from mlx_audio.tts.utils import load_model

model = load_model(
    "mlx-community/Voxtral-4B-TTS-2603-mlx-bf16",
    revision="dd85c02adbae551f5bb29ded35ee60ccdfb90927",
)

def render(text: str, path: str) -> None:
    mx.random.seed(42)
    results = list(
        model.generate(text=text, voice="neutral_male", max_tokens=6144)
    )
    audio = np.concatenate([np.asarray(result.audio) for result in results])
    pcm = (np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2")
    with wave.open(path, "wb") as out:
        out.setnchannels(1)
        out.setsampwidth(2)
        out.setframerate(results[0].sample_rate)
        out.writeframes(pcm.tobytes())
    print(len(audio) / results[0].sample_rate, results[0].token_count)

render("'Nothing taught by force remains in the mind.'", "quoted.wav")
render("Nothing taught by force remains in the mind.", "unquoted.wav")

Observed

On this machine, the quoted input produces 50 audio tokens / 4.00 s. Local
Whisper large-v3-turbo transcribes it as:

Nothing taught by force remains in the mind and not twice.

The extra clause is audible. The only text change in the comparison control is
removal of the two outer single quotation marks. That control produces 36 audio
tokens / 2.88 s and transcribes as the intended sentence exactly:

Nothing taught by force remains in the mind.

Expected

Decorative outer quotation marks should not add speech. Both calls should
speak the requested sentence once.

Scope note

I also investigated a separate long-utterance loudness problem, but a generic
39.2-second control did not reproduce it strongly enough to include in this
report. This issue is intentionally limited to the verified quote-repetition
case. Please test the same reproduction against the 4-bit checkpoint as well.

[linear-code]

NEY-94

[psiberfunk]

Audio controls

Attached are three 24 kHz mono WAVs generated from the exact generic sentence
in the issue:

1. public_repro_variant_voxtral_bf16_quote_variant_a.wav — current mlx-audio BF16 route, quoted input. This is the failing 50-token / 4.00 s output with the audible added clause.
2. public_repro_variant_voxtral_bf16_quote_variant_a_unquoted.wav — same current route, same text/voice/seed, with only the two outer single quotation marks removed. This is the clean 36-token / 2.88 s control.
3. public_repro_independent_original_bf16_quoted.wav — same quoted text, voice and seed through an independent local MLX implementation using Mistral's original BF16 checkpoint. It outputs the requested sentence once in 2.88 s.

The third file is a useful route-level A/B: it makes the current mlx-audio
implementation and/or converted checkpoint the practical suspect. It does
not isolate which of those two components is responsible, and it is not a
CUDA/vLLM reference result.