How to run Devstral and mistral-vibe with MLX?

[codrut-popescu]

So, I have downloaded Devstral-2-123B-Instruct-2512-4bit (I am running on a MBP M4 Max/128 GB)

cd ~
python -m venv mlx-env
source ~/mlx-env/bin/activate
pip install mlx-lm

Download Devstral-2-123B-Instruct-2512-4bit
python -c "from huggingface_hub import snapshot_download snapshot_download('mlx-community/Devstral-2-123B-Instruct-2512-4bit',local_dir='/Users/codrut/devstral-2-4bit-mlx')"
Run
mlx_lm.chat --model ~/devstral-2-4bit-mlx
Start as a server

mkdir -p ~/.cache/huggingface/hub
mlx_lm.server --model ~/devstral-2-4bit-mlx --port 8080 

In another terminal window I have installed mistral-vibe

curl -LsSf https://mistral.ai/vibe/install.sh | bash
vibe

in my ~/.vibe/config.toml

I have added

active_model = "mlx-local"

[[providers]]
name = "mlx"
api_base = "http://127.0.0.1:8080/v1"
api_key_env_var = ""
api_style = "openai"
backend = "generic"

[[models]]
name = "devstral-2-4bit-mlx"
provider = "mlx"
alias = "mlx-local"
temperature = 0.2
input_price = 0.0
output_price = 0.0

The problem seems to be that the answer from LLM is very very slow

in the server terminal I can see

2026-02-07 22:00:13,345 - WARNING - Received tools but model does not support tool calling. If you think this is an error, file an issue here: https://github.com/ml-explore/mlx-lm/issues
127.0.0.1 - - [07/Feb/2026 22:00:13] "POST /v1/chat/completions HTTP/1.1" 200 -
2026-02-07 22:01:02,124 - INFO - Prompt processing progress: 2048/6540
2026-02-07 22:01:52,304 - INFO - Prompt processing progress: 4096/6540
2026-02-07 22:02:44,796 - INFO - Prompt processing progress: 6144/6540
2026-02-07 22:02:56,113 - INFO - Prompt processing progress: 6539/6540

is there any solution to this?

[codrut-popescu]

Anyone here?
What does this "Received tools but model does not support tool calling" mean? That mlx supports only dense models?
Please!

[awni]

This means we haven't yet added support for tool calling for that specific model. So if you are using it as a coding agent with mlx_lm.server it won't work yet since the server won't properly parse the tools.

[codrut-popescu]

Thanks for replying. Interesting. Does that mean that there are models which are supported?

[awni]

Yes, GLM, Kimi, M2, Qwen Coder are a few that I know are supported. But we can also add support for Devstral tool calling. I will take a look at it.

[codrut-popescu]

working great in v0.30.7
Thank you sooo much!

[rpopescu]

@codrut-popescu salut!
is this working for you now?
I'm having trouble getting this to run...

I'm using the optiq mlx wrapper because it can speak anthropic too:
optiq serve --model mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4 --log-level DEBUG

latest packages are installed:

$ pip3 list| grep mlx
mlx                0.32.0
mlx-lm             0.31.3
mlx-metal          0.32.0
mlx-optiq          0.4.0

agent is Mistral Vibe v2.20.0

I can't even get it to read a file:

> read AGENTS.md                                                                                                                                                                                                                                                                                     
───────
                                                                                                                                                                                                                                                                                                     
  I will read the AGENTS.md file to understand the project instructions. 
⠙ Generating… (5m29s Esc/Ctrl+C to interrupt)

I'd really appreciate any insight into this, thanks.

[codrut-popescu]

@rpopescu salut! I don't use Devstral anymore so I can't test this myself, but a couple of thoughts:

My original problem was that mlx_lm.server didn't parse Devstral's tool calls, and that was fixed in mlx-lm's server in v0.30.7 (thanks @awni). You're not using mlx_lm.server though — optiq serve is its own server with its own OpenAI/Anthropic endpoints, so it is responsible for parsing the model's tool-call output. The mlx-lm fix doesn't necessarily apply to your setup.

Your symptom looks exactly like what I had: the model says "I will read the AGENTS.md file...", then presumably emits a Devstral-format tool call ([TOOL_CALLS]...) that never gets converted into a proper tool_calls response, so Vibe just waits while the model keeps generating. That would explain the 5+ min "Generating…" on a 24B model, which is otherwise way too slow.

To isolate it, try stock mlx_lm.server with the config from my first post:

mlx_lm.server --model mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4 --port 8080

and in ~/.vibe/config.toml point the provider at http://127.0.0.1:8080/v1 with api_style = "openai" (full config in my original post above).

• If it works with mlx_lm.server → the bug is in optiq's Devstral tool-call handling, and you should report it on their tracker (with your DEBUG log).
• If it still hangs → check the mlx_lm.server log: do you see the "Received tools but model does not support tool calling" warning, and what prompt/generation speed does it report?

Also worth checking in your optiq DEBUG log: are tokens actually streaming during those 5 minutes, and do you see raw [TOOL_CALLS] text in the model output? That would confirm the parsing theory. You can also test directly with a curl to /v1/chat/completions including a small tools array and check whether the response contains a tool_calls field or just plain text.

For reference, my setup worked with plain mlx_lm.server + Vibe once mlx-lm hit 0.30.7 — no wrapper needed.

[rpopescu]

Thanks! I realise that when I took your approach above, I hadn't put the model name in vibe's config fully so it returned a 404 when trying to get it from huggingface, even though the model was here. Now it works, and it skips downloading it.
HTTP Request: GET https://huggingface.co/api/models/mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4/revision/main "HTTP/1.1 200 OK"

So now they speak to each-other, but not well; the model calls read_file on AGENTS.md and gets nothing from vibe, which is not doing anything at all.
Repeating the instruction seems to work, the model spits back the contents of AGENTS.md and continues to write:

Write a concise plan and stop. [this is the last line in my AGENTS.md]

IMPORTANT: this context may or may not be relevant to your tasks. You should act on these guidelines if they are relevant to your task. You should act on these guidelines if they are relevant to your task. You should act on these guidelines if they are relevant to your task. You should act on these guidelines if they are [...]"

This feels hopeless. MiMo & opencode cannot talk to this model because they send several prompts which are rejected, which is why I tried optiq/anthropic mode. That works for a bit then the laptop reboots :)

What are you using instead, if you don't mind? Thanks.

[codelion]

The issue is #1307. Mistral and Devstral have no tool-call end marker, so the call ends at EOS. The state machine treats that as a stop, and the final flush only fires if prev_state == "tool", so the call is discarded. The client gets an empty message with finish_reason: "stop" and non-zero completion_tokens, which is why the agent keeps waiting.

It is not MXFP4 specific. tool_call_end_tokens is empty for all Devstral builds, so any quant hits it.

Two related shape rules also affect agents on Mistral, and both surface as a 404: it rejects two same-role turns in a row, and a user turn directly after tool results. Most agent harnesses produce both at some point.

All three are fixed in optiq serve as of 0.4.2, so Vibe with Devstral works through it: https://mlx-optiq.com/docs/integrations/mistral-vibe

[rpopescu]

Thank you! It's much better, but the warning below persists, and tool usage (reading files) stops working after 1 step:

[transformers] The tokenizer you are loading from '/Users/radu/.cache/huggingface/hub/models--mlx-community--mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4/snapshots/6d2da15c2f5afdcdd73becadb30d80f95ae17c97' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

running with optiq serve --model mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4 --log-level DEBUG

$ pip3 list | grep mlx
mlx                0.32.0
mlx-audio          0.4.4
mlx-lm             0.31.3
mlx-metal          0.32.0
mlx-optiq          0.4.2
mlx-vlm            0.6.3

[codelion]

Tool calls dropping after one step. Mistral/Devstral emit tool calls as [TOOL_CALLS]name[ARGS]{…} with no closing marker, so the call ends on EOS. mlx_lm.server's state machine has no exit from its tool state except an end marker or a stop, so it reads that EOS as a stop: finish_reason comes back stop and the buffered call is dropped. In an agent loop that's exactly "calls a tool once, then can't continue."

optiq serve (which wraps mlx_lm.server) handles this by redirecting the tool state's stop edge to normal, so the call is captured and the turn continues. If you installed optiq but are launching mlx_lm.server directly, that's the gap. Serve with optiq serve --model <your-devstral> and point Mistral Vibe at it. The fix is scoped to empty end markers, so Qwen/GLM formats are untouched.

The regex warning is cosmetic here. fix_mistral_regex=True silences it, but tokenization is byte-identical with or without it on code, tool-call JSON, and normal text. It isn't what breaks multi-step tool use.

[rpopescu]

I am not running mlx_lm.server; as I have written in the previous message:

running with optiq serve --model mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4 --log-level DEBUG

[codelion]

On the latest version of mlx-optiq I do not get this issue with mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4, the tool call is captured, not dropped:

$ optiq serve --model mlx-community/mistralai_Devstral-Small-2-24B-Instruct-2512-MLX-MXFP4 \
              --log-level DEBUG --port 8090
[optiq.serve] rotating-cache merge fix installed
[optiq.serve] concurrency cap: 8 decode / 2 prefill in parallel
127.0.0.1 - - "GET /v1/models HTTP/1.1" 200 -

# Devstral emits  [TOOL_CALLS]read_file[ARGS]{"path": "AGENTS.md"}</s>  — terminated by EOS, no end marker.
$ curl -s http://localhost:8090/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "devstral",
    "messages": [{"role":"user","content":"Read the file AGENTS.md so you can follow its instructions."}],
    "tools": [{"type":"function","function":{"name":"read_file",
        "parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}}],
    "tool_choice":"auto"
}'

# Response — finish_reason is tool_calls, and the call is parsed (stock mlx_lm.server returns an
# empty message with finish_reason "stop" here and drops it):
{
  "choices": [{
    "index": 0,
    "finish_reason": "tool_calls",
    "message": {
      "role": "assistant",
      "tool_calls": [{
        "id": "d6c73827-fd4d-4c03-a217-45037bee807b",
        "type": "function",
        "function": { "name": "read_file", "arguments": "{\"path\": \"AGENTS.md\"}" }
      }]
    }
  }],
  "usage": { "prompt_tokens": 76, "completion_tokens": 15, "total_tokens": 91 }
}