feat: add CosyVoice3 MLX implementation - #861
walkoncross wants to merge 7 commits into
Conversation
|
@walkoncross Thanks for the submission! This needs some cleanup, I left a few comments inline. Also, the examples should just live in the README.md, we don't want model-specific artifacts in there. If you can make those changes we can consider this. |
lucasnewman
left a comment
There was a problem hiding this comment.
See comments above.
Remove CosyVoice3 porting notes from runtime code, move model-specific arguments out of the shared generate interface, and align tests and examples with the reviewer feedback. Co-Authored-By: Claude <noreply@anthropic.com>
|
Hi @lucasnewman, thanks for the review! All the feedback has been addressed:
I've also resolved the corresponding review threads. Could you take another look when you get a chance? |
57884d1 to
5272f21
Compare
|
Please sign every commit, then update the PR. You can review the commits on the commits tab and follow GitHub's commit-signing guide if needed. |
|
Hi @lucasnewman — a quick follow-up on this. All four review threads from your earlier pass are now resolved, and the corresponding fixes are in:
I also fixed the CI pre-commit failure flagged earlier: the branch hadn't been formatted against the pinned black/isort versions. I ran One thing I can't do from my side: the Docs and Tests and Checks workflows on this fork PR are stuck in |
|
Review the failing CI output and run the corresponding command from |
Summary
Adds a full MLX port of CosyVoice3 (
mlx_audio/tts/models/cosyvoice3/), following the existing model contract used by other TTS models in this repo (e.g. chatterbox, spark).instruct2) generation, matching the referenceCosyVoice3Python implementation's behavior.spk2infosystem for caching/reusing zero-shot speaker prompts (--add_spk,--save_spkinfo,--load_spkinfo).CosyVoice3Tokenizerspecial-token registration for fine-grained control tokens.convert.pyso official CosyVoice3 checkpoints can be converted to MLX weights./CosyVoice/cosyvoice): causal HiFT rewrite, DiT RoPE pairing/application order, CFM noise seeding to remove pitch bias, sample-rate handling for timbre cloning, prompt-feature/silence-token alignment with the reference, and a defensive post-hoc fix for a rare stochastic LLM-sampling tail-hiss artifact (acoustic-signature detection + fade, since the exact triggering token sequence could not be reliably reproduced after 100+ attempts — seeREADME.mdunder this model's## Statussection for the full investigation).## Statussection tracking known issues/investigations.45 commits, squash-mergeable if preferred. See individual commit messages for a detailed history of the implementation and each fix.
Test plan
pytest mlx_audio/tts/tests/test_cosyvoice3.py— 31 passed (2 pre-existing failures unrelated to this branch, confirmed viagit stashagainstmain:TestLLM::test_autoregressive_decode,TestModelIntegration::test_generate_end_to_end_with_mock_frontend, bothAttributeErroron an incompleteMockFrontEndtest double)generate()validated against real CosyVoice3-0.5B checkpoint weights for zero-shot, cross-lingual, and instruct2 modes