Skip to content

streaming.md

Latest commit

 

History

History
315 lines (252 loc) · 15.9 KB

streaming.md

File metadata and controls

315 lines (252 loc) · 15.9 KB
Raw

Models larger than memory

gmlx can run a mixture-of-experts model whose file is larger than your Mac's memory. It keeps the parts that every token reads in memory and reads the routed experts from disk as each token needs them. A model of about 200 billion parameters runs this way on a Mac with 64 GB.

gmlx validate GLM-5.2-UD-IQ3_XXS-00001-of-00006.gguf          # can this Mac stream it?
gmlx run GLM-5.2-UD-IQ3_XXS-00001-of-00006.gguf --stream-experts

For a file in several parts, name the first part. Decoding runs at the speed of the SSD, and a few tokens per second is normal, so stream only a model that does not fit. Keep the GGUF on the internal SSD, because an external drive slows decoding in proportion to its latency.

Check whether a model fits

gmlx validate reads only the GGUF header, so it works on a local file or on Hugging Face before you download anything. For a MoE file it prints a streaming plan for this Mac. This plan is for Kimi-K3 UD-Q2_K_XL, an 861 GB file, on a Mac with 128 GB:

  streaming: every-token weights 62.2 GB, routed experts 799.1 GB (92 layers, 896 experts, 16 per token), prefill ring 22.7 GB
    every-token by group: attention 31.8 GB, shared experts 12.9 GB, dense ffn and routers 8.2 GB, ...
    this Mac: 128 GB RAM, ceiling 97.9 GB, KV room 29.9 GB at 32768 tokens (22.3 GB of it the prefill chunk and its pool), host floor 9.6 GB, host overhead 10.7 GB
    decode arena 5.9 GB (under 1% of the experts); a cold token reads about 14.3 GB of experts
    prefill ring 22.7 GB exceeds the arena; prefill runs from the page cache
    => streams with --stream-experts (server: stream: experts)

A verdict is a line that starts with =>. The plan's verdict is one of these:

Verdict What to do
=> streams with --stream-experts (server: stream: experts) Run it. The line above gives the decode arena's share of the experts, and a larger share decodes faster.
=> streams, but no decode arena (...); expect slow decode Runs, slowly. When a model cannot stream lists ways to free memory for the arena.
=> cannot stream: every-token weights ... + KV room ... exceed the ... ceiling by ... Pick another quant, as When a model cannot stream describes.
=> the whole file fits in RAM; streaming is optional Run it without --stream-experts.
=> loads with its experts in memory; --stream-experts streams them Run it without --stream-experts. Its lookup tables are read from the file, and the line above gives their size.

When a load without --stream-experts could keep the experts in memory, the last line says what it does with the memory free now:

Line What a load does
=> with the memory free now, its experts stay in memory Keeps the experts in memory.
=> with the memory free now, auto MTP is off: ... Keeps the experts in memory and decodes without the MTP drafter.
=> with the memory free now, its tables are offloaded to disk ... Reads its lookup tables from the file to keep the experts in memory.
=> the experts stream because ... Streams the experts. When a MoE streams its experts gives the ways to keep them in memory.

gmlx pull --list puts the recommended models that run from the SSD on this Mac under their own heading, and a pull of one says so before it downloads. run, chat and serve do not check the plan. They try to load a model that cannot stream, so run validate first. A dense model larger than RAM has no experts to stream and needs a smaller quant.

gmlx validate --json gives the same numbers under stream. There, experts_in_memory_now is false when the experts would stream, auto_mtp_off_now is true when the load would leave out its MTP drafter, and tables_from_file_now is true when it would read its lookup tables from the file to keep the experts in memory. gmlx doctor names any stream: experts entry in your configuration file that cannot stream.

Run a streamed model

Add --stream-experts on run or chat. In the configuration file, set stream on the model entry:

models:
  glm:
    path: GLM-5.2-UD-IQ3_XXS-00001-of-00006.gguf
    stream: experts
    overrides: {load: {kv_bits: 8}}

Generation starts within seconds whatever the file's size, because the load maps the file instead of reading it. For long contexts, add a quantized KV cache with --kv-bits 8 or kv_bits: 8.

The decode arena is wired memory that holds the most used experts. At the end of a run, gmlx prints how often a token found its experts there, on a [stream] decode feeder arena hit rate: line. run and chat print it with -v, and the server always logs it.

--stream-experts and stream: experts always stream the experts, and a lookup table, such as a per-layer embedding table, streams with them. On a model that fits in memory, the decode arena holds at most half of the experts and the rest stay on disk, so such a model decodes faster without the flag.

Placement What stays on the GPU Use it for
--stream-experts, stream: experts Attention, routers, shared experts and the KV cache Most cases, including long contexts, chat and a server with other models
--stream-cpu, stream: cpu Nothing. The whole model runs on the CPU from the page cache. A model on its own server. The decode feeder and its settings do not apply.

When a MoE streams its experts

A MoE model streams its experts in three cases:

  • --stream-experts or stream: experts asks for it.
  • The model is larger than its memory budget, 90% of the GPU working set. On a Mac with 16 GB the budget is 10.3 GB, so gpt-oss-20b, whose weights take 12.1 GB, streams its experts without the flag.
  • The experts fit the budget but not the memory free at load. Other apps' memory counts as free, since macOS compresses it to make room. Memory held wired, such as another model's while it generates, does not.

A model idle in another process, such as a second gmlx serve, holds no wired memory, so a load beside it counts that memory as free. llama-server keeps its model wired for 3 minutes after its last request. When both models generate, they can run the Mac out of memory. One gmlx serve that serves both models counts each of them.

In the last two cases, run and chat print why:

[stream] the experts stream because the model is over its memory budget: holding them takes 12.1 GB, past the 10.3 GB budget. GMLX_OVERCOMMIT=1 keeps them in memory unless the memory the kernel can reclaim runs out while they wire.

A model whose lookup tables alone bring it under the budget, such as Qwen3.8-Flash-Next UD-Q4_K_XL on a Mac with 128 GB, reads those tables from the file and keeps its experts in memory without the flag, and a folder scan leaves its entry without stream: experts. A model under the budget does the same when the memory free at load holds it only without its tables. Each load that reads its tables from the file says so:

[stream] per-layer embedding table of 28.8 GB offloaded to disk

When the memory free at load does not hold the experts even without the tables, they stream, and serve logs why. An entry that an older scan marked stream: experts streams them until gmlx sync-models drops the key, and serve warns about it at its start.

To see which case applies before a load, and to keep the experts in memory anyway:

gmlx validate model.gguf
GMLX_OVERCOMMIT=1 gmlx run model.gguf

Each way to keep the experts in memory has a cost:

Way What it costs
Quit another model or other apps, then load again Nothing. It does not help a model over its budget.
GMLX_OVERCOMMIT=1 The Mac may swap, and a 16 GB Mac can stop responding. The cannot fit: check of run and chat then judges against RAM.
Raise the GPU limit, as The GPU memory limit describes Less memory for macOS and other apps. It raises the budget, not the memory free now.

Under GMLX_OVERCOMMIT=1, a load that needs more than the Mac's RAM still streams or is refused: a MoE streams its experts, and run, chat and the upsampler of image generate refuse another model with cannot fit:. Experts that stay in memory still stream if memory runs out while they wire.

When the experts fit in memory but an MTP drafter that no flag and no speculative: true asked for does not fit beside them, the load keeps the experts and decodes without the drafter:

[mtp] auto MTP is off: the experts stay in memory, and the 2.8 GB MTP drafter does not fit beside them. --speculative streams the experts to keep it.

When a model cannot stream

A MoE file holds two kinds of tensors. The routed experts stream from disk, so their size sets the decode speed but not whether the model fits. Everything else is read by every token and stays in memory: attention, shared experts, dense layers, routers, embeddings and the output head. These every-token weights, plus room for a 32768-token KV cache, must fit under the memory ceiling, which is close to the GPU working set that macOS allows.

To make a model fit, or to give it a decode arena:

  • Pick a quant with smaller every-token tensors. A smaller expert quant does not change the fit. The every-token by group line shows which tensors are large, and a quant that keeps attention and the shared experts at Q8 doubles them.
  • Shrink the KV room. GMLX_STREAM_KV_CTX sets how many tokens it holds, and --kv-bits 8 halves the KV cache part. The safe context is then shorter.
  • Shrink the prefill chunk with --prefill-step-size, or prefill_step_size on the server. The KV room holds the chunk's activations beside a 2 GiB buffer pool, and a narrower chunk gives the room of its activations to the arena. Prefill reads every expert once per chunk, so it is slower.
  • Raise the GPU limit, as The GPU memory limit describes.
  • Quit other models and apps. The arena takes what is left of the RAM at load, and it shrinks when macOS reports memory pressure.
GMLX_STREAM_KV_CTX=16384 gmlx validate model-00001-of-00004.gguf

Streaming measurements gives the ceiling for each RAM size and how the memory is divided.

The lossless settings

Every setting that speeds up streaming without changing the output is on by default for stream: experts. The one exception is lookahead prestage on GLM-5.2, where it costs more than it saves. Streaming measurements lists the settings and how to turn each one off. These two have a flag:

  • --stream-fast-disk {auto,on,off}, or the stream_fast_disk key, sets how hard decoding reads ahead. auto tests the drive at load and reads ahead harder on a drive of about 5 GB/s or faster.
  • --no-gpu-keepwarm turns off a tiny kernel that keeps the GPU clock high between disk reads. It costs a few watts while decoding, so turn it off on battery.

The lossy settings

Five settings trade some output quality for decoding speed. None is on by default, and they act only on a model with a stream placement.

Setting Flag Config key What it does Acts on
Expert cap --moe-experts K moe_experts Each token uses K experts. Prefill and decoding
Expert-mass --moe-expert-mass P moe_expert_mass Each token keeps the fewest experts that cover share P of the gate weight. Prefill and decoding
Miss-shed --moe-miss-shed P moe_miss_shed Drops experts that are not in the arena, lowest first, while the rest cover share P. Decoding
Keeper prestage --moe-prestage keepers moe_prestage Reads ahead only the experts that miss-shed would keep. Needs miss-shed. Decoding
Layer-shed --moe-layer-shed P moe_layer_shed Skips a layer's experts with probability P, keeping a shared expert or dense MLP. Without one, Muse Spark keeps its top expert and other models skip nothing. Decoding

The expert cap and expert-mass combine. This keeps at most 6 experts per token, then keeps the fewest of those 6 that cover 90% of their gate weight:

gmlx run model-00001-of-00004.gguf --stream-experts --moe-experts 6 --moe-expert-mass 0.9

Choose a setting from the arena hit rate, which the [stream] decode feeder arena hit rate: line prints:

  1. When the hit rate is low, use miss-shed. It costs quality only on the reads that would wait for the disk.
  2. When the hit rate is high and a few experts carry most of the gate weight, use expert-mass.
  3. When the weight is spread evenly, only layer-shed helps. Check first that GPU keep-warm is on.

--moe-expert-probe shows how much expert-mass would save. It keeps the trained routing and prints two tables at exit, for decoding and for prefill, with the experts kept and the gate weight dropped at each P. Choose P from the decoding table.

gmlx run model-00001-of-00004.gguf --stream-experts --moe-expert-probe

To check a setting, compare its output with a lossless run on the same prompt, at the same seed and with your sampling settings. Streaming measurements has a full procedure and the values that passed it on four models.

Features with streaming

Combination Result
A placement with --mmproj Media with other features gives the result for each placement.
--stream-experts with --speculative Works. Automatic speculation stays off under streaming.
stream with speculative on a server entry The model loads streamed, without speculation.
A lossy setting with speculative decoding Automatic speculation turns off. Chat refuses an explicit --speculative.

Serving a streamed model

Send one request at a time to a server that streams a model. A second request at the same time slows both, because fewer experts are found in the arena. Prefill is not affected.

The prefill_feeder and decode_feeder keys turn the two feeders off for a model.

Residency of a streamed model

A stream: experts entry counts against server.budget_gb as its every-token weights, its arena and its prefill ring. A stream: cpu entry counts at the full size of its file.

The arena takes what the ceiling leaves, so a streamed model can fill the budget by itself. To keep a second model loaded beside it, set GMLX_DECODE_ARENA_GB to an arena size in GiB that leaves room for both. A load beside a streamed model leaves that model's KV room and released prefill ring free under the memory ceiling, and those with the host floor and host overhead its arena was sized around free in RAM, or the server defers the load. Its weights count there in full, even when they are already in the page cache, since the GPU wires the pages it reads. Streaming also lowers the wired memory limit for the rest of the process, so a dense model loaded later wires only the pages it reads.