README.md

gmlx

gmlx runs GGUF models on Apple Silicon. It runs the K-quant and IQ-quant files that the community publishes, unchanged, on Metal kernels for Apple's MLX framework. One command, gmlx, chats with a model in the terminal, serves it over OpenAI and Anthropic compatible APIs, connects coding agents to it and talks to it by voice.

These pages are the starting points:

• To run your first model, follow the Quickstart.
• To serve models to apps and agents, read Configuration and Agents and chat apps.
• To run an agent in a container, read Container mode and Custom agents.
• To look up a flag, a key or an endpoint, use the CLI reference, Configuration or the HTTP API.
• To work on gmlx itself, start with Internals.

Getting started

• Installation: Homebrew, uv and pip, the optional features, upgrading and removal.
• Quickstart: A first model, the server, a request and a connected client.
• Image generation: Choosing an image model, a first image on the command line, the API, and LoRAs.
• Migrating from other tools: What carries over from llama.cpp, Ollama and LM Studio.

Serving

• Configuration: gmlx.yaml, its models, its profiles and how a request gets its settings.
• Menu bar app: Server status and controls in the macOS menu bar.
• Speech, embeddings and rerank: The services a server can host beside chat models.
• Image API: The OpenAI image routes, their request fields and their errors.
• ComfyUI: The gmlx node pack, which runs the server's image models in ComfyUI graphs.
• RAG pipelines: Retrieval with the embeddings and rerank services.
• Structured decisions: A probability for each answer to a fixed set of questions.

Agents and containers

• Agents and chat apps: Claude Code, other coding agents and Open WebUI, set up by gmlx launch.
• Container mode: A client in an Apple container that sees only the folders you share.
• Container access: Shares, the private home, volumes, forwarded ports, pasted files and SSH in a container.
• Container sessions: Second terminals, background apps, stopping sessions and logs.
• Custom container images: Packages, Containerfiles, ready-made images and services for container mode.
• Container recipes: pi packages, dsh plugins, tool servers, a headless browser, Postgres and web search for Open WebUI.
• Custom agents: A program of your own that gmlx launch runs in a container against the server.
• Agent examples: A chat page, a notebook server and coding agents in the background, ready to copy.
• Container security: What a client in a container can still reach on the Mac and the server, and the limits of a session.

Clients

• Chat: The terminal chat client, its commands, sessions and themes.
• Voice chat: Talking to a model with gmlx talk.
• Assistant: Tools and long-term memory for chat, voice and served models.

Models

• Supported architectures: The GGUF architectures gmlx loads, with their caveats.
• Vision and audio: Multimodal models and their mmproj files.
• Granite Switch: IBM's chat model with built-in safety and decision adapters, and audio through its speech companion.
• Models larger than memory: Mixture-of-experts models that stream their experts from disk.
• Long context: Running a model past its trained context with rope scaling.
• LoRA adapters: Training an adapter and serving several on one base model.
• Distillation: Teaching a small model a document or a larger model's behavior.
• Distillation walkthrough: One distillation task from prompts to a measured adapter.
• Distillation troubleshooting: Reading the training lines and eval report, and fixing a run that goes wrong.

Performance

• Performance tuning: What makes a model fast, measuring, and choosing a quant.
• Speculative decoding: Faster decoding with a drafter, with the same output.
• Prompt cache: Skipping prefill for prompts the server has seen.
• Concurrent requests: Batching, admission pacing and shared prompts.
• Capacity and metrics: Planning how many requests fit, and the server's live metrics.
• Memory and the KV cache: How much memory a model and its context take.
• KV cache quantization: Storing the context in fewer bits.
• Benchmarks: gmlx against llama.cpp on the same files, with the method.

Help

• Troubleshooting: gmlx doctor, common failures, and where gmlx keeps its files.
• Glossary: The terms these pages use.

Reference

• CLI reference: Every command and flag.
• Configuration: Every key of gmlx.yaml.
• Family defaults: The sampling defaults and intents of each model family.
• HTTP API: The endpoints and request features.
• Structured decisions reference: Every field of a decision request and response.
• Distillation reference: The flags of every gmlx distill action.
• ComfyUI reference: The launch options of ComfyUI, how the gmlx nodes find the server and what they refuse.
• Environment variables: The variables a user can set.
• Python API: Using gmlx from Python.

Development

• Internals: How gmlx works inside.
• Adding a GGUF architecture: What supporting a new model family involves.
• Upstream defects: Defects of stock dependencies that gmlx guards against, each with its repro and its test.
• Contributing: Development setup, tests and commit style.
• Changelog: What changed in each release.