CacheIR

CacheIR is a narrow, inspectable compiler and runtime for decoder-only transformer inference. It imports a Llama/Mistral/Qwen-style model, lowers it into its own IR, runs transformer-specific optimization passes, plans memory and execution, selects backend kernels, and executes inference through a reference runtime.

It is not a PyTorch wrapper and it is not a vLLM clone. The default runtime backend is a NumPy CPU correctness backend with native acceleration hooks, plus an optional CUDA artifact runtime that executes the same lowered graph on GPU tensors with fp16 weights, persistent page-backed CUDA KV storage, SDPA attention, Triton elementwise kernels, packed QKV/Gate-Up dispatch, and fused paged decode attention over shared page tables. CacheIR also includes prefix-cache reuse, a continuous-batch scheduler with backpressure/fairness/preemption controls, OpenAI-compatible serving, optional C++20/OpenMP AVX2/AVX512 kernels exposed through pybind11, guarded Triton kernels, optional CUDA fused-kernel sources, optional accelerator adapters, CUDA graph capture plans, calibrated KV spillover cost models, packed int4/int8 loader integration, and experimental import/export surfaces for GGUF, StableHLO, and MLIR-style CacheIR.

Visual Snapshot

Measured, workload-specific H100 results.
CacheIR is 1.05x to 3.31x faster on the rows shown; raw samples exclude warmup and the generic p2048 row disables request deduplication and prefix caching.

Why CacheIR Exists

Most ML compiler stacks are powerful but broad and opaque. LLM serving has a very specific shape: prefill is compute-heavy, decode is KV-cache and memory-bandwidth heavy, and transformer optimizations need to be visible to be trusted.

CacheIR focuses on that smaller problem:

• compile prefill and decode as separate graph modes
• lower attention into explicit KV-cache-aware operations
• expose every pass as a graph diff
• emit memory plans and execution schedules
• run a real end-to-end reference path without PyTorch execution
• benchmark prefill and decode separately

Measured vLLM Advantage

CacheIR beats vLLM on every cell in the latest measured recent-model matrix. On an NVIDIA H100, the ordinary CacheIR BF16 CUDA path recorded lower median end-to-end latency in 18 of 18 batch-1 workloads across Qwen3 0.6B, Qwen3 1.7B, and DeepSeek-R1-Distill-Qwen 1.5B. The measured advantage ranges from 1.09x to 3.24x, with a 1.53x geometric-mean speedup and exact greedy-token parity for every generated sequence.

Evidence	Result
Models	Qwen3 0.6B, Qwen3 1.7B, DeepSeek-R1-Distill-Qwen 1.5B
Measured workloads favoring CacheIR	18 / 18
p50 speedup range	1.09x to 3.24x
Geometric-mean p50 speedup	1.53x
Greedy generated-token parity	100%
Configuration	BF16, batch 1, prefix cache off, 2 warmups, 7 measured samples

CacheIR wins these rows by specializing for the narrow decode workload instead of paying the full cost of a general serving engine. Its ordinary CUDA API automatically selects fixed-shape CUDA graph replay for compatible greedy requests, keeps weights and KV state resident, packs QKV and Gate/Up projections, returns final hidden states without materializing unused logits, fuses LM-head selection with argmax, and dispatches model-aware RoPE, attention, and residual normalization paths. The short-output rows benefit most because vLLM's broader scheduler and engine machinery is a larger fraction of total latency; the margin narrows as model compute and context work dominate.

This is a workload-scoped claim. It does not imply that CacheIR is universally faster for unmeasured high-concurrency serving, tensor parallelism, speculative decoding, quantized serving, or arbitrary model families. Raw seven-sample arrays, environment metadata, engine arguments, generated token IDs, and percentile summaries are checked in under benchmark_artifacts/recent_models_20260812. The detailed generated report is produced with python scripts/generate_recent_model_benchmark_report.py.

Current Capabilities

Layer	Status
Model import	Hugging Face config + NPZ/safetensors metadata, ONNX graph skeleton, GGUF metadata plus dense F32/F16/BF16/I8/I16/I32/I64/F64, classic quant reads, and optional reference K/IQ/TQ/NV/MX dequantization, broader StableHLO text/region subset
IR	CacheIR JSON/text graph format, tensor types, weight specs, attrs, pass traces
Compiler passes	shape inference, constant folding, QKV fusion, RMSNorm+QKV+RoPE fusion, SwiGLU fusion, prefill/decode specialization, layout conversion, quant-aware lowering, hardware hints, kernel selection, scheduling, memory planning
Runtime	NumPy CPU backend, CUDA artifact runtime, tokenizer bridge, CPU/GPU paged KV cache metadata, persistent shared CUDA page-backed KV pools, prefix-cache reuse with hit/miss counters, forked per-request KV sessions, continuous-batch scheduler, variable-length CUDA batched prefill, scheduler-integrated CUDA batched decode over shared page tables, queue backpressure, fairness aging, resumable preemption, bounded-latency counters, calibrated CPU/GPU spillover policy experiments, greedy streaming generation
Serving	OpenAI-compatible FastAPI server, streaming chat completions, /healthz, Prometheus-style /metrics, and CacheIR batch completions endpoint
Tooling	CLI, artifact bundles, graph HTML/DOT/MLIR export, benchmark runner, scheduler benchmark, external comparison harness, hardware profiler with bandwidth calibration
Native backend	C++20/OpenMP library with AVX2/AVX512 dispatch, optional pybind11 bridge, native RMSNorm/matmul/SiLU-multiply kernels, guarded Triton RMSNorm/SwiGLU/QKV/RoPE/decode-attention kernels, Triton FP16 matmul, persistent multi-batch page-table Triton decode attention, optional CUDA fused-kernel, FP16 WMMA Tensor Core matmul, reduced paged-attention, and CUDA graph planning targets

Complete Tech Stack

Area	Stack
Primary languages	Python 3.10+, C++20
Python packaging	pyproject.toml, setuptools, editable installs, optional dependency groups
Core numerical runtime	NumPy reference kernels; optional PyTorch CUDA tensor substrate for the CacheIR CUDA executor
Compiler IR	Custom CacheIR graph IR, JSON artifacts, text IR dumps, pass diffs
Model import	Hugging Face config.json, NPZ reference weights, safetensors optional, ONNX optional, GGUF metadata and dense F32/F16/BF16/I8/I16/I32/I64/F64 plus Q4_0/Q4_1/Q5_0/Q5_1/Q8_0/Q8_1 native subset, optional gguf reference dequantization for supported K/IQ/TQ/NV/MX formats, StableHLO textual/region subset
Transformer architecture	Llama/Mistral/Qwen-style decoder-only blocks, RMSNorm, RoPE, grouped-query attention, SwiGLU, residual streams
Compiler passes	Shape inference, constant folding, QKV fusion, RMSNorm+QKV+RoPE fusion, SwiGLU fusion, prefill/decode specialization, layout conversion, quant-aware lowering, DCE, hardware hints, kernel selection, execution scheduling, static memory planning
Runtime systems	Weight loader, tokenizer bridge, paged KV cache, persistent CUDA page pools, CUDA-resident KV state, prefix-cache snapshots and counters, per-request KV sessions sharing weights/tokenizer, continuous-batch scheduler with queue limits, priorities, cancellation, backpressure waits, fairness aging, preemption, bounded queue-latency counters, variable-length CUDA prefill batching, scheduler-integrated CUDA decode batching, calibrated spillover policy hooks, backend dispatcher, greedy streaming decode loop
CPU backend	NumPy executable backend; C++20/OpenMP native library with scalar, AVX2/FMA, and AVX512 dispatch; optional pybind11 module _cacheir_native; runtime uses native RMSNorm/SiLU-multiply when available and keeps native matmul as an opt-in experiment via CACHEIR_NATIVE_MATMUL=force or auto
GPU backend surface	CUDA/Triton target naming, schedule generation, end-to-end CudaRuntime, fp16 CUDA weight loading, persistent GPU KV page pools, SDPA attention dispatch, cached packed QKV/Gate-Up weights, guarded Triton RMSNorm, SwiGLU, FP16 matmul, fused RMSNorm/QKV/RoPE, single-query decode attention, persistent multi-batch page-table decode attention kernels, shape-specific GEMM plan recording, cuBLASLt-through-Torch default matmul, opt-in Triton Tensor Core matmul; optional CUDA C++ fused-kernel, FP16 WMMA Tensor Core matmul, reduced paged-attention, and CUDA graph capture planning target
Accelerator adapters	Optional CUTLASS, FlashAttention, and FlashInfer probes/dispatch contracts plus guarded direct execution wrappers for prefill, single decode, and batch paged decode; CUTLASS detects the nvidia-cutlass/cutlass_cppgen wheel when installed
Quantization	int4/int8 graph lowering with real packed uint8 storage, per-row scales, affine zero points, CPU and CUDA model-loader integration, dequant-at-GEMM-boundary execution, and quantized fused QKV/SwiGLU dispatch paths
Serving	FastAPI and Uvicorn optional dependencies, OpenAI-compatible /v1/models, /v1/completions, /v1/chat/completions, /healthz, /metrics, and /v1/cacheir/batch_completions
Benchmarks	Built-in benchmark CLI with CPU/CUDA backend selector, prefill/decode split metrics, benchmark matrix script, apples-to-apples CacheIR/vLLM matrix runner, CUDA runtime benchmark, CUDA scheduler benchmark, continuous-batch scheduler benchmark, serving workload benchmark, OpenAI-compatible serving comparison, production robustness evidence runner, quality parity checker with logit/perplexity metrics, public-claim report generator with raw samples/environment metadata, GPU kernel benchmark, comparison harness for vLLM, llama.cpp, TensorRT-LLM, MLC LLM, IREE, and TVM commands, installed IREE/TVM smoke benchmark execution, WSL vLLM/FlashInfer comparison runs, and Qwen 0.5B/1.5B/7B CacheIR coverage measurements
Visualization	HTML export, Graphviz DOT export, text IR export, MLIR-style CacheIR dialect export, parser round trip, verifier, and Matplotlib benchmark-chart generation
Native build	CMake, Ninja, OpenMP, pybind11 optional
Testing	pytest, Python bytecode compilation checks, CLI smoke tests, CMake build checks
Infrastructure	Dockerfile, Makefile, GitHub Actions
Documentation	Markdown docs and full LaTeX project documentation in docs/latex/
External context	vLLM, llama.cpp, IREE, TVM, StableHLO, MLIR, CUTLASS, FlashAttention/FlashInfer are comparison or optional integration surfaces, not default runtime dependencies

Install

python -m pip install -e ".[dev,server]"

Optional importer dependencies:

python -m pip install -e ".[importers,benchmark]"

Optional native/GPU dependency groups:

python -m pip install -e ".[native,gpu]"
cmake -S cpp -B cpp/build -DCACHEIR_BUILD_PYTHON=ON -DCMAKE_BUILD_TYPE=Release
cmake --build cpp/build --config Release

Optional accelerator adapter probes:

python -m pip install -e ".[accelerators]"

Optional upstream compiler/runtime comparison tools:

python -m pip install -e ".[upstream]"

On native Windows/Python 3.13, IREE and TVM install from wheels while vLLM, FlashAttention, FlashInfer, TensorRT-LLM, MLC-LLM, and llama-cpp-python are validated through WSL2 CUDA environments. /home/bhatt/cacheir-llm-venv has vLLM, FlashInfer, Triton, Torch, MLC-LLM, and llama-cpp-python; /home/bhatt/cacheir-trtllm-venv has TensorRT-LLM 1.2.1 and trtllm-bench; /home/bhatt/cacheir-tools/llama.cpp/build-cuda13/bin has a CUDA llama.cpp build. WSL also has NVIDIA CUDA Toolkit 13.3 and OpenMPI development packages. FlashInfer, MLC-LLM, TensorRT-LLM, vLLM, and llama.cpp are verified locally. FlashAttention remains guarded: no compatible binary wheel exists for this Python/CUDA/Torch stack, and a controlled source build destabilized WSL with Wsl/Service/E_UNEXPECTED.

If CMake discovers the wrong Python on Windows, pin the active interpreter:

cmake -S cpp -B cpp/build-py-active \
  -DCACHEIR_BUILD_PYTHON=ON \
  -DPython3_EXECUTABLE="$(python -c 'import sys; print(sys.executable)')" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build cpp/build-py-active --config Release

CUDA sources are enabled with -DCACHEIR_BUILD_CUDA=ON. On Windows, NVIDIA CUDA requires the MSVC host compiler (cl.exe) on PATH; Linux builds can use the normal CUDA host compiler flow. This workspace has Build Tools installed at C:\BuildTools and cl.exe on the user PATH.

Quickstart

Create a tiny Llama-shaped model, compile it, inspect the lowered graph, run generation, and benchmark prefill/decode:

cacheir make-tiny examples/tiny_model
cacheir compile examples/tiny_model --output examples/tiny_artifact
cacheir inspect examples/tiny_artifact --mode decode
cacheir run examples/tiny_artifact --prompt "CacheIR" --max-new-tokens 16
cacheir benchmark examples/tiny_artifact --decode-tokens 32 --repeats 3
cacheir benchmark examples/tiny_artifact --backend cuda --cuda-dtype float16 --warmup 2
cacheir export examples/tiny_artifact examples/decode.html --mode decode
cacheir export examples/tiny_artifact examples/decode.mlir --mode decode --format mlir

The artifact directory contains:

artifact.json              full machine-readable compiler artifact
graphs/prefill.cir         optimized prefill IR
graphs/decode.cir          optimized decode IR
schedules/*.json           runtime kernel schedule by mode
passes/*.diff              pass-by-pass graph diffs

Python API

from cacheir import Runtime, compile_model

artifact = compile_model(
    model_path="examples/tiny_model",
    target="cpu",
    quant=None,
    mode=["prefill", "decode"],
    max_seq=128,
)

rt = Runtime(artifact)
for token in rt.generate("Explain MLIR in simple terms", max_new_tokens=16):
    print(token, end="")

Compiler Example

Logical transformer IR starts as separate operations:

%attn_norm = rms_norm(%hidden, %attn_norm_w)
%q = matmul(%attn_norm, %q_proj_w)
%k = matmul(%attn_norm, %k_proj_w)
%v = matmul(%attn_norm, %v_proj_w)
%q_rope, %k_rope = rope(%q, %k)
%attn = grouped_query_attention(%q_rope, %k_rope, %v)

The decode-specialized graph lowers that into scheduled runtime calls:

%q_rope, %k_rope, %v = fused_rmsnorm_qkv_rope(...)
%attn = paged_attention_decode(%q_rope, %k_rope, %v)

Every pass records before/after text and a unified diff:

cacheir inspect examples/tiny_artifact \
  --mode decode \
  --pass-name prefill_decode_specialization

CLI

cacheir profile
cacheir profile --calibrate --sample-mb 16 --repeats 5
cacheir make-tiny MODEL_DIR
cacheir compile MODEL_DIR --target cpu --quant int4_awq --output ARTIFACT_DIR
cacheir compile model.stablehlo --target cpu --output stablehlo_artifact
cacheir inspect ARTIFACT_DIR --mode decode
cacheir export ARTIFACT_DIR graph.dot --format dot
cacheir export ARTIFACT_DIR graph.mlir --format mlir
cacheir benchmark ARTIFACT_DIR --prompt "hello" --decode-tokens 64 --repeats 5
cacheir benchmark ARTIFACT_DIR --backend cuda --cuda-dtype float16 --warmup 2
cacheir external --benchmark --workdir .tmp/upstream
cacheir run ARTIFACT_DIR --prompt "hello"
cacheir serve ARTIFACT_DIR --host 127.0.0.1 --port 8000 --max-batch-size 4 --max-queue-size 64

Benchmarks

CacheIR reports prefill and decode separately because they stress different parts of the system:

Result	Baseline	CacheIR path	Change
h1024/l4 scheduler latency	582.177 ms	339.796 ms	1.71x faster
h2048/l4 scheduler latency	867.240 ms	684.002 ms	1.27x faster
h1024/l4 runtime prefill	30.984 ms CPU	7.533 ms CUDA	4.11x faster
h2048/l4 runtime prefill	94.036 ms CPU	6.705 ms CUDA	14.02x faster

python scripts/benchmark_matrix.py --output benchmark_results.json --repeats 3 --decode-tokens 16

Public-claim evaluation bundle:

python scripts/benchmark_public_claim.py \
  --artifact examples/tiny_artifact \
  --backend cpu \
  --input-len 16 \
  --output-len 8 \
  --repeats 5 \
  --output .tmp/reports/public_claim_report.json \
  --markdown .tmp/reports/public_claim_report.md

For CI and report plumbing, generate a complete smoke bundle without external model downloads:

python scripts/benchmark_public_claim.py \
  --smoke \
  --repeats 1 \
  --warmup 0 \
  --output .tmp/reports/public_claim_smoke.json \
  --markdown .tmp/reports/public_claim_smoke.md

The report records raw CacheIR samples, p50/p90/p99 latency, environment metadata, optional vLLM and llama.cpp baselines, quality parity, serving workload statistics, and pass/fail comparison rows. Public outperformance statements should be made from these generated reports using identical model, prompt length, output length, batch size, precision, hardware, and warmup settings.

Full apples-to-apples CacheIR/vLLM matrix runner:

python scripts/benchmark_apples_to_apples.py \
  --full-qwen \
  --download \
  --run-cacheir \
  --run-vllm \
  --target cuda \
  --cacheir-backend cuda \
  --prompt-lens 16,128,512,2048,8192 \
  --output-lens 1,8,32,128 \
  --batch-sizes 1,4,8,16,max \
  --max-stable-batch 16 \
  --precisions bf16,int8 \
  --vllm-quant int8=awq \
  --repeats 5 \
  --warmup 1 \
  --output .tmp/reports/apples_to_apples_matrix.json \
  --markdown .tmp/reports/apples_to_apples_matrix.md

The matrix records model, precision, vLLM quantization mode, prompt length, output length, batch size, max model length, greedy sampling settings, raw samples, TTFT, TPOT, end-to-end latency, throughput, P50/P90/P99 summaries, NVIDIA clock/power/memory telemetry, failures, OOM rows, and unsupported rows. Rows that do not fit the local GPU are preserved as evidence rather than hidden.

{
  "prompt_tokens": 24,
  "decode_tokens": 32,
  "prefill_ms_avg": 4.2,
  "decode_ms_avg": 0.3,
  "prefill_tokens_per_s": 5714.0,
  "decode_tokens_per_s": 3333.0,
  "kv_cache": {"page_size": 16, "layers": {...}}
}

The CPU runtime keeps NumPy correctness coverage. Native RMSNorm and fused SiLU-multiply dispatch automatically when _cacheir_native is available; native matmul remains opt-in because the local AVX dot-loop kernel is not consistently faster than NumPy's optimized contraction. CUDA kernels plug into the same artifact and benchmark surfaces when built with a compatible CUDA host toolchain.

Continuous-batch scheduler benchmark:

python scripts/benchmark_scheduler.py \
  --output .tmp/reports/scheduler_benchmark.json \
  --repeats 3 \
  --max-new-tokens 8 \
  --max-batch-size 4

Serving workload benchmark:

python scripts/benchmark_serving_workload.py examples/tiny_artifact \
  --backend cpu \
  --max-new-tokens 8 \
  --max-batch-size 4 \
  --repeats 3 \
  --output .tmp/reports/serving_workload.json

Quality parity checker:

python scripts/quality_parity.py baseline_artifact candidate_artifact \
  --output-len 8 \
  --output .tmp/reports/quality_parity.json

Production robustness and OpenAI-compatible API evidence:

python scripts/benchmark_production_evidence.py examples/tiny_artifact \
  --backend cpu \
  --max-new-tokens 8 \
  --queue-soak-iterations 1000 \
  --long-context-tokens 8192 \
  --output .tmp/reports/production_evidence.json \
  --markdown .tmp/reports/production_evidence.md

OpenAI-compatible serving comparison against a running vLLM server:

python scripts/benchmark_serving_comparison.py \
  --cacheir-artifact examples/tiny_artifact \
  --cacheir-backend cpu \
  --vllm-url http://127.0.0.1:8001 \
  --vllm-model Qwen/Qwen2.5-0.5B-Instruct \
  --max-tokens 32 \
  --repeats 5 \
  --concurrency 16 \
  --output .tmp/reports/serving_comparison.json \
  --markdown .tmp/reports/serving_comparison.md

End-to-end CUDA runtime benchmark:

python scripts/benchmark_cuda_runtime.py \
  --output .tmp/reports/cuda_runtime_benchmark_h1024_latest.json \
  --repeats 3 \
  --warmup 1 \
  --decode-tokens 8 \
  --hidden-size 1024 \
  --num-layers 4 \
  --cuda-dtype float16

CUDA scheduler batching benchmark:

python scripts/benchmark_cuda_scheduler.py \
  --output .tmp/reports/cuda_scheduler_benchmark_latest.json \
  --repeats 3 \
  --warmup 1 \
  --max-batch-size 4 \
  --max-new-tokens 16 \
  --hidden-size 1024 \
  --num-layers 4 \
  --cuda-dtype float16

Persistent page-backed scheduler benchmark from this pass:

python scripts/benchmark_cuda_scheduler.py \
  --output .tmp/reports/cuda_scheduler_benchmark_persistent_post_production.json \
  --repeats 2 \
  --warmup 1 \
  --max-batch-size 4 \
  --max-new-tokens 8 \
  --hidden-size 512 \
  --num-layers 2 \
  --cuda-dtype float16 \
  --use-triton-attention

Benchmark	Baseline	CacheIR path	Result
h512/l2 scheduler, batch 4, persistent paged decode	176.225 ms sequential	78.491 ms batched	2.25x lower latency
h512/l2 generated throughput	184.0 tok/s sequential	407.8 tok/s batched	2.22x higher throughput
h512/l2 CUDA runtime prefill	8.800 ms CPU	3.515 ms CUDA	2.50x faster
Qwen2.5-0.5B CacheIR bf16 tensor replay	16 input + 8 output	34.4 ms/request	1.12x faster than fresh vLLM bf16
Qwen2.5-0.5B CacheIR int8 ordinary API	16 input + 8 output	28.3 ms/request	1.36x faster than fresh vLLM bf16
Qwen2.5-0.5B vLLM CUDA bf16	16 input + 8 output	38.5 ms/request	20-iteration fresh rerun
Qwen2.5-1.5B CacheIR bf16 ordinary API	16 input + 4 output	55.1 ms/request	1.13x faster than fresh vLLM bf16
Qwen2.5-1.5B CacheIR int8 ordinary API	16 input + 4 output	36.2 ms/request	1.72x faster than fresh vLLM bf16
Qwen2.5-1.5B vLLM CUDA bf16	16 input + 4 output	62.2 ms/request	20-iteration fresh rerun
Qwen2.5-7B CacheIR int4 CUDA smoke	1 input + 1 output	108.8 s prefill, 15.3 s decode	fits, but not serving-competitive
llama.cpp CUDA tiny GGUF	16 prompt tokens	6,120 tok/s	real llama-bench JSON run
llama.cpp CUDA tiny GGUF	8 generated tokens	2,044 tok/s	real llama-bench JSON run

Latest RunPod H100 validation on 2026-08-12 used Ubuntu 24.04.3, NVIDIA H100 80GB HBM3, driver 580.126.09, Torch 2.11.0+cu130, Triton 3.6.0, vLLM 0.26.0, FlashInfer 0.6.14, and exact local Qwen2.5 snapshot paths. CacheIR and vLLM used bf16, greedy decoding, matching prompt/output/batch settings, explicit prefix-cache settings, one warmup, three or more measured samples, and raw JSON/Markdown artifacts under /workspace/cacheir-results/ on the pod.

Model	CacheIR path	Prompt	Output	Batch	CacheIR avg ms	vLLM avg ms	CacheIR speedup
Qwen2.5-0.5B	CUDA graph single request	16	1	1	1.499	4.958	3.31x
Qwen2.5-0.5B	CUDA graph single request	16	8	1	8.094	13.692	1.69x
Qwen2.5-0.5B	dense CUDA graph batch	16	1	4	1.911	10.324	5.40x
Qwen2.5-0.5B	dense CUDA graph batch	16	8	4	12.504	20.281	1.62x
Qwen2.5-1.5B	CUDA graph single request	16	1	1	2.476	5.396	2.18x
Qwen2.5-1.5B	CUDA graph single request	16	8	1	14.975	21.116	1.41x
Qwen2.5-1.5B	dense CUDA graph batch	16	1	4	3.349	10.118	3.02x
Qwen2.5-1.5B	dense CUDA graph batch	16	8	4	21.289	27.575	1.30x
Qwen2.5-1.5B	explicit no-graph debug	16	1	1	11.399	5.295	0.46x
Qwen2.5-1.5B	explicit no-graph debug	16	8	1	86.610	20.932	0.24x

After the batch-aware KV reserve fix, default dense batch CUDA graph capture, batched FlashAttention prefill/decode, batch-aware lm-head selection, decode SiLU kernel selection, FlashInfer batch paged decode wrappers, bounded input staging cache, exact duplicate request coalescing, and scheduler fast lane, CacheIR wins the measured p16 batch 4/8/16 bf16 rows for Qwen2.5-0.5B and Qwen2.5-1.5B. The hot homogeneous scheduler workload for Qwen2.5-1.5B, four requests, eight output tokens, measured 19.233 ms total and 1,672 generated tok/s.

Repeated-prompt long-context H100 rerun with CacheIR exact-batch dedup:

Model	Prompt/output	Batch	CacheIR avg ms	vLLM avg ms	CacheIR speedup
Qwen2.5-0.5B	p512/o8	4	9.763	25.826	2.65x
Qwen2.5-0.5B	p2048/o8	4	13.292	33.782	2.54x
Qwen2.5-1.5B	p512/o8	4	18.125	33.032	1.82x
Qwen2.5-1.5B	p2048/o8	4	26.163	52.914	2.02x

For the same repeated-prompt Qwen2.5-1.5B p2048/o8/b4 row, vLLM with prefix caching enabled measured 29.826 ms, while CacheIR exact-batch dedup measured 26.163 ms. Generic non-deduplicated dense batch remains a separate row: with dedup and prefix caching disabled, CacheIR's stride-aware RoPE/SwiGLU and direct KV-write path measured a 47.808 ms p50 over 15 samples on Qwen2.5-1.5B p2048/o8/b4. A fresh vLLM 0.26.0 bf16 run measured 50.437 ms p50 over 9 samples, making CacheIR 1.05x faster on this row. CacheIR includes an opt-in zero-copy vLLM FlashAttention block-table experiment (CACHEIR_EXPERIMENTAL_PAGED_BATCH_DECODE=1), but the FlashInfer wrapper is the faster default here. CacheIR's explicit no-graph debug path remains slower than vLLM's production CUDA-graph path and is not included in the performance claim. The raw reports are checked in under benchmark_artifacts/h100_20260812.

For apples-to-apples local comparisons, provide explicit external commands for the tools installed on the machine:

python scripts/compare_external_benchmarks.py examples/tiny_artifact \
  --vllm-command "python bench_vllm.py" \
  --llama-command "llama-bench -m model.gguf" \
  --iree-command "iree-benchmark-module --module=model.vmfb" \
  --tvm-command "python bench_tvm.py" \
  --output benchmark_comparison.json

When IREE and TVM are installed but no explicit commands are supplied, CacheIR can run built-in upstream smoke benchmarks:

python scripts/compare_external_benchmarks.py examples/tiny_artifact \
  --run-installed-smoke \
  --output benchmark_comparison.json

Latest local validation on 2026-08-09:

• Full optional project install passed with python -m pip install -e ".[dev,server,importers,benchmark,native,gpu]".
• PyPI packages installed: pybind11, triton-windows, ONNX/importer deps, server deps, benchmark deps.
• MSVC Build Tools 2022 installed at C:\BuildTools; cl.exe resolves from the user PATH.
• CUDA Torch installed: torch 2.12.1+cu130; CUDA is available on the RTX 5070 Laptop GPU.
• _cacheir_native.pyd built with CMake and the active Python 3.13 interpreter.
• Native SIMD probe reported avx512.
• Native RMSNorm, matmul, and fused SiLU-multiply matched NumPy within float32 tolerance.
• Runtime matmul dispatch defaults to NumPy's optimized contraction; native matmul can be forced with CACHEIR_NATIVE_MATMUL=force or shape-gated with CACHEIR_NATIVE_MATMUL=auto.
• CUDA C++ target built with MSVC/NVCC: cacheir_cuda_kernels.lib.
• CUDA C++ FP16 WMMA Tensor Core matmul and reduced paged-attention decode paths built with MSVC/NVCC.
• End-to-end CudaRuntime executed full decoder artifacts on the RTX 5070 Laptop GPU with fp16 weights, GPU-resident KV state, SDPA attention, Triton elementwise kernels, and cached packed QKV/Gate-Up weights.
• CUDA runtime benchmark results: h1024/l4 CUDA fp16 measured 7.533 ms prefill and 7.573 ms/token decode versus CPU fp32 30.984 ms prefill and 12.271 ms/token decode; h2048/l4 CUDA fp16 measured 6.705 ms prefill and 7.333 ms/token decode versus CPU fp32 94.036 ms prefill and 30.619 ms/token decode.
• Scheduler-integrated CUDA batching executed variable-length prefill and active decode rounds across four requests. The benchmark processed 134 real prefill tokens plus 6 padding tokens. h1024/l4 batch-4 improved median latency from 582.177 ms to 339.796 ms and generated-token throughput from 109.9 tok/s to 188.3 tok/s, a 1.71x speedup. h2048/l4 batch-4 improved median latency from 867.240 ms to 684.002 ms and throughput from 37.0 tok/s to 46.8 tok/s, a 1.27x speedup.
• Triton batched decode attention smoke test executed through the scheduler path with --use-triton-attention on h128/l1, producing 3.36x latency/throughput speedup over sequential CUDA sessions for the tiny smoke shape.
• Triton GPU kernels executed on CUDA: RMSNorm 0.0157 ms, SiLU-multiply 0.0213 ms, FP16 matmul 0.0667 ms / 8.05 TFLOP/s, single-query decode attention 0.0552 ms, and multi-batch page-table decode attention 0.0566 ms on the RTX 5070 Laptop GPU.
• cacheir profile --calibrate --sample-mb 16 --repeats 5 measured CPU copy at 32.04 GB/s and CUDA H2D at 13.98 GB/s on this host.
• nvidia-cutlass 4.2.0.0 installed and the CUTLASS adapter probe detects cutlass_cppgen.
• iree-base-compiler 3.11.0, iree-base-runtime 3.11.0, and apache-tvm 0.25.0 installed from PyPI.
• cacheir external --benchmark --workdir .tmp/reports/upstream_latest compiled StableHLO through IREE to a 9,781-byte VMFB and ran iree-benchmark-module; TVM built and ran a TE vector-add benchmark with checksum 512.0.
• WSL2 CUDA environment validated direct FlashInfer decode execution through CacheIR's runtime and adapter wrappers. MLC-LLM nightly CUDA 13 and TensorRT-LLM 1.2.1 are installed and their CLIs import/run. Standalone flash_attn remains guarded because PyPI has no compatible wheel for this stack, but CacheIR can use vLLM's bundled vllm.vllm_flash_attn prefill adapter when vLLM is installed.
• A CUDA llama.cpp build ran a real llama-bench model benchmark against a locally converted GGUF tiny Llama model; the JSON result is in .tmp/upstream/llama_cpp_tiny_benchmark.json.
• The vLLM model benchmark runner installs a process-local no-UVA fallback shim before vLLM workers initialize and now prepends the CacheIR project root plus /usr/local/cuda/bin so spawned vLLM workers can import the shim and FlashInfer can find nvcc.
• WSL2 root installed NVIDIA CUDA Toolkit 13.3 at /usr/local/cuda from NVIDIA's Ubuntu 24.04 repo; no Linux NVIDIA driver package was installed.
• A runnable WSL2 CUDA vLLM environment is available at /home/bhatt/cacheir-llm-venv; vLLM with Torch 2.11.0+cu130 and FlashInfer ran real latency benchmarks on Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct.
• CacheIR CUDA graph generation now reserves fixed K/V buffers during capture, skips unused single-request page-pool writes, defaults compatible greedy requests through ordinary API auto replay, avoids the previous extra decode after the final requested token, can return final hidden states without materializing logits, uses a Triton fused lm_head+argmax reducer for dense generation, uses vLLM's FlashAttention prefill adapter when installed, dispatches FlashInfer single-request and batch paged decode where available, and caches repeated batch input tensors for graph replay.
• CacheIR bf16 tensor replay measured 34.444 ms on Qwen2.5-0.5B-Instruct for 16 input + 8 output tokens versus a 20-iteration fresh vLLM bf16 rerun at 38.543 ms. CacheIR int8 ordinary API auto replay measured 28.332 ms on the same shape.
• CacheIR bf16 ordinary API auto replay measured 55.072 ms on Qwen2.5-1.5B-Instruct for 16 input + 4 output tokens versus a 20-iteration fresh vLLM bf16 rerun at 62.231 ms. CacheIR int8 ordinary API auto replay measured 36.167 ms on the same shape.
• The explicit use_cuda_graph=False path is retained as a launch-by-launch debug path; production greedy CUDA generation defaults to auto replay.
• CacheIR compiled Qwen2.5-7B-Instruct locally and executed an int4 CUDA smoke on the 8 GB GPU. It fits, but the current dequant-at-GEMM-boundary path is not performance-competitive: 108.816 s first prefill and 15.349 s for one decode token.
• CacheIR's persistent page-backed scheduler benchmark with Triton attention enabled measured h512/l2 batch-4 latency at 78.491 ms versus 176.225 ms sequential, a 2.25x latency speedup and 2.22x generated-token throughput speedup.
• Packed int4/int8 quantized weights now use real compressed uint8 storage, per-row scales, affine zero points, CPU/CUDA loader integration, and quantized matmul/fused-QKV/fused-SwiGLU dispatch.
• Scheduler hardening now includes blocking backpressure attempts, fairness aging, resumable preemption, request token limits, queue-wait accounting, and bounded-latency violation counters.
• Continuous-batch scheduler benchmark: 4 requests, 32 generated tokens, median 54.819 ms, 72.97 requests/s, 583.74 generated tokens/s, and 70 prompt tokens reused through prefix cache.
• Public-claim evaluation bundle smoke ran successfully and wrote .tmp/reports/public_claim_smoke.json plus .tmp/reports/public_claim_smoke.md, including raw latency samples, environment metadata, quality parity, serving workload counters, and external-system availability probes.
• Standalone quality-parity and serving-workload scripts ran successfully on generated tiny artifacts and wrote .tmp/reports/quality_parity_smoke.json plus .tmp/reports/serving_workload_smoke.json.
• Apples-to-apples matrix smoke ran successfully and wrote .tmp/reports/apples_to_apples_smoke.json plus .tmp/reports/apples_to_apples_smoke.md, including bf16/int8 rows with TTFT, TPOT, end-to-end latency, throughput, raw samples, and NVIDIA telemetry.
• Production evidence smoke ran successfully and wrote .tmp/reports/production_evidence_smoke.json plus .tmp/reports/production_evidence_smoke.md, covering cancellation, backpressure, queue soak, long-context generation, memory-fragmentation snapshots, and OpenAI-compatible streaming/non-streaming API latency. CUDA OOM recovery is recorded by that script when run with --backend cuda.
• Serving comparison smoke ran successfully and wrote .tmp/reports/serving_comparison_smoke.json plus .tmp/reports/serving_comparison_smoke.md; vLLM serving rows are produced when --vllm-url points to a running OpenAI-compatible vLLM server.
• WSL vLLM exact-prompt smoke ran successfully through /home/bhatt/cacheir-llm-venv against /home/bhatt/cacheir-models/cacheir-vllm-tiny-llama-h128 and wrote .tmp/reports/vllm_exact_smoke_result.json; vLLM 0.23.0 used prefix caching, FlashAttention, CUDA graph capture, gpu_memory_utilization=0.8, and produced a raw end-to-end sample.
• Qwen, Llama, and Mistral family conformance smokes compile and execute tiny decoder artifacts with family-specific architecture labels.
• Benchmark matrix ran 18 rows with 3 repeats and 16 decode tokens; medium_4l_h64 fp32 decode measured 2.217 ms/token on the short prompt and 2.169 ms/token on the medium prompt.
• Full local and remote H100 pytest suites passed: 46 tests.
• The comparison harness can now run installed IREE/TVM smoke benchmarks plus model-aware vLLM and llama.cpp benchmark helpers when model paths are supplied.

Development

python -m pytest -q
python -m compileall cacheir -q
cmake -S cpp -B cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build cpp/build --config Release

Project layout:

cacheir/
  ir.py                  CacheIR graph IR
  compiler.py            importer selection and pass pipeline
  importers/             HF, ONNX, GGUF, StableHLO, tiny model support
  passes/                compiler passes
  runtime/               artifact, tokenizer, KV cache, CPU/CUDA runtimes, scheduler, server
  backends/              backend registry, optional native bridge, Triton kernels, upstream probes
cpp/                     C++20/OpenMP AVX backend and optional pybind11 module
docs/                    architecture, IR, runtime, benchmark notes
examples/                tiny model compile demo
scripts/                 benchmark matrix, apples-to-apples, scheduler, GPU, upstream, parity, serving, robustness, and public-claim helpers
tests/                   pytest coverage

Boundaries

Implemented and tested:

• CPU reference execution without PyTorch
• end-to-end CUDA artifact execution through CudaRuntime
• CUDA fp16 weight loading, GPU-resident KV state, SDPA attention dispatch, and cached packed QKV/Gate-Up decode weights
• scheduler-integrated variable-length CUDA prefill batching and active-request CUDA decode batching with server metrics
• shared CUDA KV page allocator accounting across forked request sessions
• scheduler admission controls: queue limits, request priorities, cancellation, and Prometheus counters
• prefill/decode graph specialization
• paged KV-cache reference behavior
• prefix-cache reuse and spillover policy experiments
• forked per-request runtime sessions sharing weights/tokenizer
• continuous-batch scheduler with prefix reuse metrics
• OpenAI-compatible server entrypoint with /healthz, /metrics, and CacheIR batch completions
• artifact bundles, pass diffs, graph export, benchmarks, scheduler benchmark, server entrypoint
• quantization-aware lowering with CPU-side quantize/dequantize simulation
• C++20 backend library build with scalar, AVX2/FMA, and AVX512 dispatch
• optional pybind11 bridge for native RMSNorm, matmul, and fused SiLU-multiply kernels
• default runtime dispatch to native RMSNorm/SiLU-multiply plus opt-in native matmul policy
• guarded Triton RMSNorm, SiLU/SwiGLU, fused RMSNorm/QKV/RoPE, and single-query decode attention kernels
• guarded Triton FP16 matmul kernel using tl.dot/Tensor Core lowering where available
• guarded Triton multi-batch page-table decode attention with GQA mapping
• optional CUDA C++ fused kernels for RMSNorm, SwiGLU, RMSNorm/QKV/RoPE, FP16 WMMA Tensor Core matmul, batched paged-attention ABI, and reduced paged-attention decode
• CUDA graph capture planning for decode replay loops
• native GGUF dense F32/F16/BF16/I8/I16/I32/I64/F64 plus Q4_0/Q4_1/Q5_0/Q5_1/Q8_0/Q8_1 tensor reads
• optional reference GGUF dequantization for K/IQ/TQ/NV/MX formats exposed by the gguf package
• StableHLO text importer with arithmetic, shape, broadcast, slice, reduce-region, and cast coverage
• MLIR-style CacheIR dialect emitter, parser round trip, and verifier
• optional upstream MLIR C++ dialect registration skeleton through CACHEIR_BUILD_MLIR=ON
• CUTLASS, FlashAttention, and FlashInfer adapter probes, dispatch contracts, and guarded direct wrappers for prefill, single decode, and batch paged decode
• direct FlashInfer smoke execution on WSL2 CUDA, with FlashAttention kept as a guarded adapter for environments that provide a compatible wheel
• vLLM no-UVA CUDA worker fallback and CUDA Toolkit path export for CacheIR-launched vLLM benchmarks
• calibrated low-VRAM KV spillover cost model with measured bandwidth calibration, transfer estimates, and resident-page budgeting
• IREE StableHLO compile/runtime benchmark integration through upstream IREE wheels
• TVM TE/TIR runtime benchmark integration through the upstream TVM wheel
• external benchmark comparison harness for vLLM, llama.cpp, IREE, and TVM, including installed IREE/TVM smoke runs and model-aware vLLM/llama.cpp helpers
• public-claim evaluation suite with raw latency samples, environment metadata, optional vLLM/llama.cpp baselines, quality parity, serving workload metrics, and Markdown/JSON reports
• apples-to-apples CacheIR/vLLM matrix runner for Qwen 0.5B/1.5B/3B/7B presets, prompt/output/batch sweeps, bf16 and mapped vLLM quantized comparisons, exact-prompt vLLM Python API runs, and OOM/unsupported row preservation
• production robustness evidence runner for cancellation, backpressure, queue soak, long-context generation, memory-fragmentation snapshots, CUDA OOM recovery, and OpenAI-compatible server API latency
• OpenAI-compatible serving comparison script for CacheIR and a running vLLM server under mixed prompts, streaming, concurrency, and greedy settings
• standalone quality parity and serving workload benchmark scripts

CacheIR is now a real, inspectable transformer compiler/runtime project with working serving surfaces, scheduler metrics, admission control, native CPU hooks, an end-to-end CUDA runtime, scheduler-integrated CUDA batching, shared CUDA page allocator accounting, GPU kernel microbenchmarks, and upstream comparison harnesses. The public evaluation suite is the canonical way to publish outperformance claims: it requires matching workload settings, records raw samples and environment metadata, preserves OOM/unsupported rows, and keeps quality/serving/robustness checks next to the latency tables. Public wording should stay workload-specific: “CacheIR beats vLLM on these measured workloads/configurations,” followed by the generated table and raw artifacts.

License

Apache-2.0.