Independent 4-bit quantization benchmarks for small instruct models
The order matters more than the individual tricks. Teams routinely reach for quantization first because it is the famous one, then discover their real bottleneck was the serving stack — and quantization on a bad stack is measurable work for no user-visible gain.
Work down this list in order. Stop when your latency and cost targets are met; everything below that point is optional complexity.
Before changing anything, record: tokens/sec at your real batch size, p50 and p95 latency, peak VRAM, and your actual prompt/response length distribution. Most published benchmarks — including ours — use batch 1 and a fixed prompt shape. If your traffic is batched or your prompts are 10× longer, published rankings may not transfer. A benchmark is a hypothesis about your workload, not a measurement of it.
Moving from a naive transformers.generate() loop to a real serving engine (vLLM, TGI, SGLang, llama.cpp for CPU/edge) typically buys more than any quantization step, because you gain continuous batching and PagedAttention-style KV management at once. If you are running a generation loop in a web handler, this is your bottleneck — not the weights.
Evidence from our own data: every GPTQ row in the benchmark fell back to a dequantize-in-PyTorch path rather than an optimized kernel, and that single stack detail cost more throughput than the choice between quantization methods. The stack dominates.
Now it is worth doing. What our measurements say for small instruct models on rentable GPUs:
(ΔPPL ≈ +0.58 to +0.95 across every configuration), and peak VRAM was a flat 4.57 GB for a 1.5B model versus 6.48 GB at FP16.
far more (worst +2.07), and its memory use was unstable on the fallback path.
ΔPPL was +0.72 with wikitext2 calibration versus +1.38 with chat-style calibration at the same sample count, while raising samples from 32 to 512 moved it only 0.05–0.11. AWQ was comparatively indifferent. Calibrate on text that resembles what you actually serve.
above the FP16 baseline. Measure it.
If your prompts share a long common prefix — a system prompt, a few-shot block, a retrieved document reused across turns — prefix caching removes that prefill work entirely on repeat requests. This is close to free and is frequently the second-largest win after the serving stack for RAG and chat-style workloads.
The KV cache, not the weights, is usually what limits your concurrency. Levers, cheapest first: cap max_model_len to what you actually serve; enable KV quantization (FP8 is widely supported and typically near-lossless); check whether your model supports grouped-query attention. Halving KV memory often raises usable concurrency more than shrinking the weights does.
A small draft model proposes tokens that the large model verifies in parallel. Real gains on latency-bound single-stream workloads; often negative value on throughput-bound batched serving, where the extra compute competes with real requests. Last on the list because it adds a second model to operate, and its benefit is the most workload-dependent of anything here.
automatically transfer to T4 or A10 — the memory-bandwidth ratios differ.
toolchain alone moving FP16 throughput by up to 27.8% on identical weights and identical hardware, with the sign flipping between configurations. If a vendor's baseline is not your baseline, their speedup is not your speedup.
including in our own tables — and it is not your users' experience. Add a task-specific evaluation on your own held-out prompts before shipping.
---
This playbook is the method behind the Inference Cost & Quantization Audit. Everything cited here is measured and published: rows and logs, exact quantized weights.