Naive Quantization Is a Trap: The Real Way to Speed Up Edge AI on Arm

Track: Physical AI / Edge AI — Arm Create: AI Optimization Challenge 2026 Hardware: Raspberry Pi 5 Model B (Cortex-A76, aarch64, NEON + asimddp dot-product instructions)

TL;DR

Everyone says "quantize to INT8 for edge inference." We tested that claim honestly on real Arm hardware and found a trap most tutorials don't mention:

Approach	Result
Naive NumPy INT8 matmul (no optimized kernel)	100-300x SLOWER than FP32
ONNX Runtime INT8 (real Arm NEON dot-product kernels)	up to 6.18x FASTER than FP32, 4x smaller model

Quantization only helps if the runtime actually has an Arm-optimized integer kernel. Without one, you silently make your model much worse. This repo shows both the failure mode and the fix, with reproducible numbers on a Raspberry Pi 5.

Experiment 1: The Trap (benchmark.py)

Manual INT8 quantization (scale + round + clip) of random matrices, multiplied via NumPy's default int32 matmul path, compared to NumPy's native (BLAS-backed) float32 matmul.

n=  128  fp32=   0.0781 ms  int8=   3.5890 ms  speedup=0.02x
n=  256  fp32=   0.5047 ms  int8=  43.7837 ms  speedup=0.01x
n=  512  fp32=   5.1859 ms  int8= 580.2396 ms  speedup=0.01x
n= 1024  fp32=  33.7103 ms  int8=9494.7164 ms  speedup=0.00x

Why: NumPy's float32/float64 matmul calls a highly optimized BLAS library (OpenBLAS, which uses NEON SIMD on Arm). Its int32 matmul has no such optimized path — it falls back to a generic, non-vectorized loop. Quantizing without the right kernel makes things worse, not better. This is invisible until you actually measure it.

Experiment 2: The Fix (onnx_benchmark.py)

Same idea (INT8 quantization of an edge-classifier-style 4-layer MLP, 784->512x4->10, similar to a typical Arm-deployed vision/audio head), but run through ONNX Runtime, which does have genuine Arm-optimized integer kernels (MLAS/XNNPACK, using the asimddp NEON dot-product instructions present on this CPU).

batch=  1  fp32=  0.5421 ms  int8=  0.0878 ms  speedup=6.18x
batch=  8  fp32=  0.6242 ms  int8=  0.1782 ms  speedup=3.50x
batch= 32  fp32=  1.7421 ms  int8=  1.4972 ms  speedup=1.16x

Model size also drops from 9.5 MB (FP32) to 1.2 MB (dynamic INT8), a 4x reduction — meaningful for flash-constrained embedded deployments.

Why it works this time: ONNX Runtime's CPU execution provider dispatches INT8 GEMM ops to kernels specifically written for Arm NEON dot-product instructions, not a generic fallback loop.

Takeaway for Arm developers

• Don't assume "INT8 = faster" on Arm. Verify with your actual runtime.
• The speedup is largest at low batch size (batch=1: 6.18x) — exactly the common edge-inference case (single sample, low latency, on-device). It shrinks toward memory-bandwidth-bound regimes at larger batches (batch=32: 1.16x).
• If you're prototyping a quantization pipeline in raw NumPy before deploying, your prototype numbers can be actively misleading. Benchmark on the real deployment runtime.

Hardware

{
  "platform": "Linux-6.12.34+rpt-rpi-2712-aarch64-with-glibc2.42",
  "machine": "aarch64",
  "processor": "unknown",
  "python": "3.13.14",
  "numpy": "2.5.1",
  "cpu_model": "Raspberry Pi 5 Model B Rev 1.1",
  "cpu_features": "fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp"
}

Reproduce

python3 -m venv .venv
.venv/bin/pip install numpy onnx onnxruntime
.venv/bin/python benchmark.py         # Experiment 1: the trap
.venv/bin/python build_model.py       # builds model_fp32.onnx
.venv/bin/python -c "from onnxruntime.quantization import quantize_dynamic, QuantType; \
  quantize_dynamic('model_fp32.onnx', 'model_int8.onnx', weight_type=QuantType.QInt8)"
.venv/bin/python onnx_benchmark.py    # Experiment 2: the fix

Raw results: benchmark_results.json, onnx_benchmark_results.json

Files

• benchmark.py — naive NumPy INT8 vs FP32 matmul benchmark (the trap)
• build_model.py — builds a representative 4-layer edge-classifier MLP directly in ONNX
• onnx_benchmark.py — real ONNX Runtime FP32 vs INT8 inference benchmark (the fix)
• model_fp32.onnx / model_int8.onnx — the two model variants benchmarked
• *_results.json — raw benchmark output for both experiments