DeepSeek-V4-Pro-0813 TP4 + DSpark on four M3 Ultra (mlx.launch)

[guruswami-ai]

DeepSeek-V4-Pro-0813 TP4 + DSpark on four M3 Ultra (mlx.launch)

Report from a four-node M3 Ultra 512 GB mesh (Thunderbolt 5, JACCL). Full write-up with tables: we can paste more detail or open a gist if useful.

Result: one interactive stream of Pro-0813 (1.65 T / 49 B active, official MXFP4 experts + MXFP8 attn, MTP kept) at 15–23 tok/s decode with the checkpoint's own DSpark drafter, out to 32 768 prompt tokens. An OpenAI-compatible server in front of mlx.launch is enough for DeepSeek Harness (dsh) to run a real tool loop (failing unit tests → model edits → tests pass).

This is the first trillion-class open-weight pack we have run on this mesh that is usable as an agent. Previous 2 T-class attempts either did not fit or collapsed into tokenizer boilerplate.

Why this is an mlx-lm issue

Stock mlx-lm 0.31.3 has no deepseek_v4. We used the omlx 0.5.7 overlay (deepseek_v4 + mlx_lm_mtp) as a library under mlx.launch, not the omlx product server.

PRs already open: #1189, #1233. Until one lands, anyone reproducing this is on a patched tree.

mlx_lm.server refuses draft models in distributed mode. DSpark only arms inside the patched GenerationBatch.next(). stream_generate / generate_step never activate MTP. The working driver is BatchGenerator.

Model.shard() does not shard self.mtp. The three DSpark stages need the same head/expert split as the backbone or TP4 desynchronises.

Sampled TP4 without mx.random.seed on every rank parked at 49/48/45/47 tokens and killed JACCL (Send failed … -12). Shared seed is mandatory.

Numbers (14 August 2026, world=4, gather fallback, FAST_SYNCH=0)

cell	decode tok/s	accept	notes
greedy math	22.9	96%	17*23= → 391
greedy prose	14.8	70%	the chat number
sampled prose	17.1	83%	T=1.0 / top_p=0.95, shared seed
ctx 4 096	17.4	82%	TTFT 14 s
ctx 16 384	14.7	76%	TTFT 59 s
ctx 32 768	12.8	79%	TTFT 122 s
batch B=2 / B=4	16.5 / 28.6 agg.	—	DSpark off

Without DSpark the same math smoke is 5 tok/s. With it, 19 tok/s / 96% accept on the first 128-token run.

B>1 is backbone only (_omlx_mtp_rowwise_unsupported on DSpark). One DSpark user is faster than four backbone users.

What blocked long context

Near 4 k prompt: ValueError: Unsupported DSpark physical-ring GEMM layout. omlx's dspark_ring_gemm is compiled for 64 heads. Pro TP4 has 32 heads/rank. Python enables that kernel once pooled KV exceeds index_topk 1024. Hiding the symbol forces the gather path in SparseCompressedAttention. That unlocked 4 k / 16 k / 32 k. A 32-head native kernel compiled but lowered accept (70% → 56%) and then tore JACCL at 16 k. Serving stays on gather.

Instability

JACCL Recv/Send -12 on a later generate in the same mlx.launch process is still common. First generate after load usually works. Second HTTP turn or second BatchGenerator sometimes kills a rank. mlx.launch then reports exit 0. After a crash, leftover ranks plus a hung ibv_devinfo need a full-fleet reboot and one fresh mlx.distributed_config --auto-setup.

Asks

1. Land deepseek_v4 and keep MTP tensors loadable.
2. Either teach mlx_lm.server distributed DSpark, or document that fused draft is BatchGenerator only.
3. Shared RNG for distributed sampled decode.
4. Anyone with a 32-head ring GEMM that does not tank accept: we will A/B it.

Happy to share scripts (convert, shard_mtp, BatchGenerator bench, OpenAI serve with an arm barrier). Not claiming this is upstream-ready. It works well enough that a person can sit in DeepSeek Harness and get code back. That is new for us at this scale on Apple Silicon.

[guruswami-ai]

Recipe and scripts (convert / shard_mtp / BatchGenerator smoke / OpenAI serve). No converted weights — rebuild from deepseek-ai/DeepSeek-V4-Pro-0813.

https://huggingface.co/guruswami-ai/deepseek-v4-pro-0813-mlx-tp4-recipe

[guruswami-ai]

Interim: singleton serve, second generate, prefix trim

The OpenAI endpoint (BatchGenerator + DSpark, gather ring) has now taken three sequential HTTP turns in one mlx.launch after a ~2.5 h idle. All three finished (greedy 32-token math, DSpark on). Turns 2 and 3 used prompt-cache trim of the previous completion.

The JACCL Recv/Send -12 that killed later generates was an undrained BatchGenerator.close(). mx.synchronize() before close() is the workaround. Detail and the cell matrix are on mlx #4247.

Stock mlx_lm.server still cannot do fused DSpark in distributed mode. This path is still the omlx overlay as a library. Native 32-head ring GEMM is still unused (accept drop).

Recipe (scripts, no weights): https://huggingface.co/guruswami-ai/deepseek-v4-pro-0813-mlx-tp4-recipe

[guruswami-ai]

Follow-up, 15 August 2026

Same stack (world=4, gather, FAST_SYNCH=0, omlx 0.5.7 as a library, BatchGenerator). A few things in the original post are now stale.

Context

One process, cap 131 072, 32 new tokens, greedy. Decode stays ~12 tok/s. Prefill is what dies with length.

prompt	TTFT	prefill tok/s	decode tok/s	accept
8 098	26 s	314	11.5	59%
32 360	112 s	290	12.3	70%
64 711	299 s	217	14.8	83%
97 062	507 s	192	12.0	75%
129 339	736 s	176	12.2	76%

129 k is usable if you can wait twelve minutes for the first token. It is not comfortable.

Prefix cache and streaming

PoolingCache still cannot large-trim. Chasing a better trim is the wrong next step. What works is an append-only tape of post-template token ids: a dsh-style follow-up logs cache=append/29 and prefills 8 tokens instead of 37. Hash the tape, not the JSON messages and not the official HF encoder. reasoning_content is passed back only on tool-call turns (same as official dsh).

SSE is no longer a replay of a finished completion. Ranks stay in lockstep on each token. One framing bug blocked stock urllib: BaseHTTPRequestHandler defaults to HTTP/1.0 + keep-alive with no Content-Length, so the client sits on read-until-close. HTTP/1.1 + Connection: close made a 31-event 2+2 stream finish in 2.1 s with [DONE]. dsh sends stream: true; this matters.

Still required

• BatchGenerator, not stream_generate.
• Shared mx.random.seed on every rank.
• Shard mtp.* yourself; Model.shard() does not.
• Hide the 64-head dspark_ring_gemm on 32-head/rank Pro TP4.

Asks from the first post still stand. Happy to paste scripts. Not claiming this is upstream-ready.

[guruswami-ai]

Parking Pro-0813 TP4 as a daily driver (17 August 2026). The load path and short-prompt benches in this thread still hold. Long-lived JACCL dies independently of the model: ml-explore/mlx#4319 and Apple FB24371487. Full note on mlx discussion 4247.