Anllm

Anllm is a multi-family dense inference engine (Qwen3 ≤8B, Llama 3.x ≤8B, Int4) for Apple Silicon: a production-grade serving stack (continuous batching, chunked prefill, paged KV cache) with hand-written Metal kernels(needed for optimization in future)

Status

v0.0.1 — Milestone M0 (foundation). See the roadmap below.

Milestone	Scope	State
M0	Package scaffold, audited core modules ported, CI	✅
M1	Unified Qwen3  and Llama 3.x model, generation loop, audit P0 fixes	🚧
M2	Continuous batching + OpenAI-compatible server	⏳
M3	Metal kernels default-on, benchmarks, PyPI release	⏳

Install

Requires macOS on Apple Silicon and Python 3.10–3.12. Activate your conda env (e.g. conda activate aisystem), then:

git clone <repo-url> && cd anllm
pip install -r requirements.txt
pip install -e .

# Required to download models from HuggingFace
pip install huggingface_hub

See Supported models for how to download a checkpoint and run it.

Hardware baseline

Developed against M4 / 16 GB unified memory.

Model	Fit on 16 GB	Notes
Qwen3-0.6B / 1.7B	Yes	Fast iteration targets
Llama 3.2 1B / 3B	Yes	Dense, untied embeddings
Qwen3-4B (4-bit)	Yes (~2.5 GB)	Primary development target
Qwen3-8B / Llama 3.1/3.2 8B (4-bit)	Tight	Upper bound for local dev
Qwen3-14B+, Llama 4	No(only due to device config limitation(16GB RAM))	Out of scope for local-first v0

Supported models

The engine dispatches on model_type from each checkpoint's config.json, so any MLX 4-bit Int4 build of the following dense causal-LM families loads with no code changes. All of the models below fit on 16 GB unified memory.

Qwen3 (dense, ≤8B)

Model	HuggingFace ID	Disk (4-bit)	Fit on 16 GB
Qwen3 0.6B	mlx-community/Qwen3-0.6B-4bit	~0.5 GB	✅
Qwen3 1.7B	mlx-community/Qwen3-1.7B-4bit	~1 GB	✅
Qwen3 4B	mlx-community/Qwen3-4B-4bit	~2.5 GB	✅
Qwen3 8B	mlx-community/Qwen3-8B-4bit	~4.6 GB	✅

Llama 3.x (dense, ≤8B)

Model	HuggingFace ID	Disk (4-bit)	Fit on 16 GB
Llama 3.2 1B	mlx-community/Llama-3.2-1B-Instruct-4bit	~0.7 GB	✅
Llama 3.2 1B (base)	mlx-community/Llama-3.2-1B-4bit	~0.7 GB	✅
Llama 3.2 3B	mlx-community/Llama-3.2-3B-Instruct-4bit	~1.8 GB	✅
Llama 3.2 3B (base)	mlx-community/Llama-3.2-3B-4bit	~1.8 GB	✅
Llama 3.1 8B	mlx-community/Meta-Llama-3.1-8B-Instruct-4bit / -8B-4bit	~4.5 GB	✅

Larger dense models (Qwen3-14B, Llama-3.3-70B) and MoE / multimodal architectures (Llama 4, Gemma 4-E2B) are not supported on this setup. Qwen3-14B, Llama-3.3-70B are dense and architecturally loadable but exceed 16 GB; Llama 4 / Gemma 4 are MoE / multimodal and outside the engine's scope.

1. Download a model (HuggingFace CLI)

pip install huggingface_hub

# Qwen3 ≤8B
huggingface-cli download mlx-community/Qwen3-0.6B-4bit
huggingface-cli download mlx-community/Qwen3-1.7B-4bit
huggingface-cli download mlx-community/Qwen3-4B-4bit
huggingface-cli download mlx-community/Qwen3-8B-4bit

# Llama 3.x ≤8B
huggingface-cli download mlx-community/Llama-3.2-1B-Instruct-4bit
huggingface-cli download mlx-community/Llama-3.2-3B-Instruct-4bit
huggingface-cli download mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

Models cache under ~/.cache/huggingface/hub/. Once downloaded, pass the repo ID (or a local path) to anllm.

2. Run it interactively / one-shot (anllm run)

# Interactive chat REPL
python -m anllm.cli run --model mlx-community/Qwen3-4B-4bit

# One-shot completion (positional argument is the prompt)
python -m anllm.cli run "Name three primary colors" --model mlx-community/Qwen3-4B-4bit

# Pick any supported model
python -m anllm.cli run "Hello" --model mlx-community/Llama-3.2-3B-Instruct-4bit
python -m anllm.cli run "Hello" --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

3. Serve it via the OpenAI-compatible API (anllm serve)

python -m anllm.cli serve --model mlx-community/Qwen3-4B-4bit --host 127.0.0.1 --port 8402

Then call it with any OpenAI SDK (base_url=http://127.0.0.1:8000/v1):

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

# Non-streaming
resp = client.chat.completions.create(
    model="mlx-community/Qwen3-4B-4bit",
    messages=[{"role": "user", "content": "Name three primary colors."}],
    max_tokens=24,
)
print(resp.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="mlx-community/Qwen3-4B-4bit",
    messages=[{"role": "user", "content": "Write a poem on AI"}],
    max_tokens=200,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Endpoints: GET /v1/models, POST /v1/chat/completions, POST /v1/completions, GET /healthz — both streaming and non-streaming are supported.

Layout

anllm/
├── core/     attention, RoPE, RMSNorm, MLP, quantization, fused-kernel dispatch
├── cache/    dense KV cache + paged KV pool (block tables, snapshot/rollback)
├── engine/   sampling, generation, scheduling (M1/M2)
└── server/   OpenAI-compatible HTTP API (M2)

Native Metal kernels (ext/, M1) accelerate RMSNorm, RoPE, SwiGLU, quantized matmul, and paged attention; every pure-Python path works without them.

License

Apache-2.0. Anllm is a product of learning from skyzh/tiny-llm book and mlx documentations NOTICE.