Summary
With clone() / edit(), generated audio is frequently garbled (sounds like noise or scrambled speech), sometimes near-silence that never terminates. We traced all of it to how the audio-codebook token stream is produced and decoded in tts.py _generate:
No chunk-alignment guarantee on the output stream. The vocoder (cosyvoice.py _reshape) parses the mixed stream positionally, assuming every 5-token group is exactly [vq02, vq02, vq06, vq06, vq06] (vq02 = tokens 65536–66559, vq06 = 66560–73727). But the model sometimes prepends one or three spurious vq06 tokens at generation start, so every frame is misparsed → the entire utterance decodes to garbage. The repo does no realignment check.
BatchEncoding treated as a list of ints (crash + broken cap). token_ids = tokenizer.apply_chat_template(..., tokenize=True) returns a BatchEncoding (dict-like); iterating it yields string keys, so the <audio_N> filter at tts.py:278 raises TypeError: '<=' not supported between 'int' and 'str' under transformers 4.57 (the repo's own pin). The same confusion silently disables the max_tokens cap: max(1, max_tokens - len(token_ids)) computes len() of the dict (≈2 keys), so the subtraction is a no-op and the cap is never applied as intended.
Greedy/argmax never terminates reliably. With temperature 0 (or unseeded sampling), generation can loop for thousands of tokens on a near-silent 8-token attractor ({65545, 65719, 66033, 66167, 66225, 66289, 66420, 67677} — measured 2791 tokens, zero <|EOT|> hits). The model only stops when sampling randomly lands on id 3 <|EOT|>; greedy decoding effectively has no stop.
Reproduce (clone path)
python tts_infer.py --edit_type clone --source_text "The quick brown fox jumps over the lazy dog."
--audio_prompt <ref.wav> --ref_text
Run repeatedly (same input). Observed across draws:
clean speech (~90% of runs with good sampling settings),
fully garbled audio (rotated interleave: chunks classify as (6,6,6,2,2) or (6,2,2,6,6)),
near-silent loop that only stops at max_tokens under greedy.
Environment
torch 2.13.0+cu130, vLLM 0.26 nightly (wheels.vllm.ai commit 30b34171b), transformers 4.57.x, Python 3.12, Ubuntu 24.04 container
GPU: RTX 5060 Ti 16 GB (Blackwell sm_120), weights: Step-Audio-EditX-AWQ-4bit
Verified with full token-stream dumps + whisper ASR round-trip
(Note: we could not test the repo's pinned stack — vllm 0.14/torch 2.9.1 has no sm_120 support. The BatchEncoding crash depends only on transformers ≥4.54ish and should reproduce on the pinned stack; the rotation/EOS behaviors are model-level and likely stack-independent.)
Suggested fix (what we shipped locally)
At the _generate choke point, realign the stream to whole chunks before handing it to the vocoder:
if len(output_token_ids) > 5:
def _cls(t):
t = t - 65536
return '2' if 0 <= t < 1024 else ('6' if 1024 <= t < 8192 else 'X')
def _score(k):
s = output_token_ids[k:]
n = len(s) // 5
if n == 0: return 0.0
return sum(''.join(_cls(s[i5:i5+5][j]) for j in range(5)) == '22666'
for i in range(n)) / n
best = max(range(5), key=_score)
if best != 0 and _score(best) > _score(0):
output_token_ids = output_token_ids[best:] # drop leading offset
output_token_ids = output_token_ids[:len(output_token_ids) // 5 * 5] # whole chunks
Plus: coerce the BatchEncoding (if hasattr(token_ids, "input_ids"): token_ids = list(token_ids["input_ids"])) — which also restores the intended max_tokens cap.
Impact
Zero-shot TTS and audio editing from this repo are unusable out of the box on ~10%+ of draws (garbage) and ~100% under greedy (non-terminating silence). Docs suggest EN/ZH/JA/KO are supported; output quality on supported languages is otherwise good when the stream happens to align.
Summary
With clone() / edit(), generated audio is frequently garbled (sounds like noise or scrambled speech), sometimes near-silence that never terminates. We traced all of it to how the audio-codebook token stream is produced and decoded in tts.py _generate:
No chunk-alignment guarantee on the output stream. The vocoder (cosyvoice.py _reshape) parses the mixed stream positionally, assuming every 5-token group is exactly [vq02, vq02, vq06, vq06, vq06] (vq02 = tokens 65536–66559, vq06 = 66560–73727). But the model sometimes prepends one or three spurious vq06 tokens at generation start, so every frame is misparsed → the entire utterance decodes to garbage. The repo does no realignment check.
BatchEncoding treated as a list of ints (crash + broken cap). token_ids = tokenizer.apply_chat_template(..., tokenize=True) returns a BatchEncoding (dict-like); iterating it yields string keys, so the <audio_N> filter at tts.py:278 raises TypeError: '<=' not supported between 'int' and 'str' under transformers 4.57 (the repo's own pin). The same confusion silently disables the max_tokens cap: max(1, max_tokens - len(token_ids)) computes len() of the dict (≈2 keys), so the subtraction is a no-op and the cap is never applied as intended.
Greedy/argmax never terminates reliably. With temperature 0 (or unseeded sampling), generation can loop for thousands of tokens on a near-silent 8-token attractor ({65545, 65719, 66033, 66167, 66225, 66289, 66420, 67677} — measured 2791 tokens, zero <|EOT|> hits). The model only stops when sampling randomly lands on id 3 <|EOT|>; greedy decoding effectively has no stop.
Reproduce (clone path)
python tts_infer.py --edit_type clone --source_text "The quick brown fox jumps over the lazy dog."
--audio_prompt <ref.wav> --ref_text
Run repeatedly (same input). Observed across draws:
clean speech (~90% of runs with good sampling settings),
fully garbled audio (rotated interleave: chunks classify as (6,6,6,2,2) or (6,2,2,6,6)),
near-silent loop that only stops at max_tokens under greedy.
Environment
torch 2.13.0+cu130, vLLM 0.26 nightly (wheels.vllm.ai commit 30b34171b), transformers 4.57.x, Python 3.12, Ubuntu 24.04 container
GPU: RTX 5060 Ti 16 GB (Blackwell sm_120), weights: Step-Audio-EditX-AWQ-4bit
Verified with full token-stream dumps + whisper ASR round-trip
(Note: we could not test the repo's pinned stack — vllm 0.14/torch 2.9.1 has no sm_120 support. The BatchEncoding crash depends only on transformers ≥4.54ish and should reproduce on the pinned stack; the rotation/EOS behaviors are model-level and likely stack-independent.)
Suggested fix (what we shipped locally)
At the _generate choke point, realign the stream to whole chunks before handing it to the vocoder:
if len(output_token_ids) > 5:
def _cls(t):
t = t - 65536
return '2' if 0 <= t < 1024 else ('6' if 1024 <= t < 8192 else 'X')
def _score(k):
s = output_token_ids[k:]
n = len(s) // 5
if n == 0: return 0.0
return sum(''.join(_cls(s[i5:i5+5][j]) for j in range(5)) == '22666'
for i in range(n)) / n
best = max(range(5), key=_score)
if best != 0 and _score(best) > _score(0):
output_token_ids = output_token_ids[best:] # drop leading offset
output_token_ids = output_token_ids[:len(output_token_ids) // 5 * 5] # whole chunks
Plus: coerce the BatchEncoding (if hasattr(token_ids, "input_ids"): token_ids = list(token_ids["input_ids"])) — which also restores the intended max_tokens cap.
Impact
Zero-shot TTS and audio editing from this repo are unusable out of the box on ~10%+ of draws (garbage) and ~100% under greedy (non-terminating silence). Docs suggest EN/ZH/JA/KO are supported; output quality on supported languages is otherwise good when the stream happens to align.