Independent 4-bit quantization benchmarks for small instruct models
Every number on this site comes from bench/rows.csv — 161 rows, 34 columns, one row per (model × quantization method × calibration set × calibration size × seed × GPU tier). 154 rows are status=ok; 7 are status=failed and are published with their error strings. Run structure and timings come from bench/driver.log; the pinned parameters below are quoted from the notes column of the rows themselves and from bench/quant_bench_app.py.
Read the limitations section before reading any table.
Models (exact HF ids, as recorded in the model column).
| Model | params | Rows | ok | failed |
|---|---|---|---|---|
Qwen/Qwen2.5-1.5B-Instruct | 1.5B | 76 | 76 | 0 |
HuggingFaceTB/SmolLM2-1.7B-Instruct | 1.7B | 76 | 76 | 0 |
deepgrove/Bonsai | 0.5B | 9 | 2 | 7 |
Only deepgrove/Bonsai is revision-pinned (5d836f3cbcd11c8ac54d920c7230cda04fbffdac); it is also the only model loaded with trust_remote_code=True.
Quantization methods (quant_method / bits).
| Method | Rows | ok | Description as recorded |
|---|---|---|---|
gptq | 75 | 72 | 4-bit, gptqmodel |
awq | 75 | 72 | 4-bit, autoawq |
none-fp16 | 11 | 10 | fp16 reference rows — no quantization |
There is one fp16 reference row per (model × stack × tier): 6 rows recorded on the AWQ stack, 4 on the GPTQ stack, 1 failed. The fp16 rows are the baseline that ppl_delta_vs_fp16 is computed against; they are not a separate "method".
No INT8 run and no ternary/BitNet-class quantization run exist in this data — 0 rows.
GPU tiers (gpu_tier). A10 — 84 rows; T4 — 77 rows. No L4 rows exist. The app requests Modal gpu="A10G" and gpu="T4"; driver.log preflight recorded the device names NVIDIA A10 and Tesla T4. Each quantized artifact was produced once on the A10 and then evaluated on both tiers.
Calibration sets and sizes (calibration_set / calibration_size).
| Set | n=32 | n=128 | n=512 |
|---|---|---|---|
wikitext2-train | 24 rows | 30 rows | 24 rows |
openhermes-2.5 | 24 rows | 24 rows | 24 rows |
The 11 fp16 rows carry calibration_set=none, calibration_size=0. The wikitext2-train n=128 column has 6 extra rows because that is the only cell Bonsai was run in (3 AWQ + 3 GPTQ, all failed).
calibration_size is a count of sampled text windows, not tokens. Windows are drawn with a seeded RNG from a tokenized pool: wikitext-2-raw-v1 train (first 8,000,000 characters) or 4,000 rendered OpenHermes-2.5 conversations (also capped at 8,000,000 characters).
Seeds. 3 seeds — 0, 1, 2 — on every quantized cell. Row counts by seed: 61 / 50 / 50 (seed 0 is higher because all 11 fp16 rows record seed=0). The seed varies the calibration-window draw only; it does not re-run the fp16 baseline or the eval.
Design. 2 models × 2 methods × 2 calibration sets × 3 sizes × 3 seeds = 72 distinct quantized artifacts, each evaluated on 2 tiers = 144 quantized rows, plus 11 fp16 rows and 6 Bonsai failure rows = 161. driver.log shows SmolLM2 was gated behind Qwen (gate smollm2/gptq: qwen A10 ok=19/19 -> PROCEED).
verified_date: 63 rows dated 2026-07-30, 98 rows dated 2026-07-31.
GPTQ — verbatim from notes on all 72 successful GPTQ rows:
> GPTQ g128 desc_act=True damp=0.01 calib_len=2048
i.e. 4 bits, group size 128, desc_act=True, damp_percent=0.01, calibration sequence length 2048 tokens. calib_seq_len=2048 on all 72 rows.
AWQ — verbatim from notes on all 75 AWQ rows:
> AWQ zero_point g128 GEMM calib_len=512; autoawq 0.2.9 (upstream-deprecated; pinned to its last-tested torch2.6.0/transformers4.51.3 combo)
i.e. 4 bits, group size 128, zero_point=True, GEMM version, calibration sequence length 512 tokens. calib_seq_len=512 on all 75 rows. Artifacts are loaded with fuse_layers=False (_awq_load in bench/quant_bench_app.py).
Perplexity protocol — verbatim from notes on all rows that have notes:
> ppl: wikitext2-test + OpenHermes-heldout, ctx2048 non-overlapping, token-capped per tier
Salesforce/wikitext wikitext-2-raw-v1 test split (ppl_*_wt2), and aheld-out OpenHermes-2.5 slice (ppl_*_chat). The OpenHermes pool is split deterministically: conversations 1–10,000 are the calibration pool, 10,001–12,000 are the eval slice; the eval uses the first 400 of those. Calibration and chat-eval text are disjoint by construction.
ppl_ctx_len=2048 on all 161 rows), non-overlapping windows —stride equals context, no sliding window.
PPL_CAPS in bench/quant_bench_app.py): A10 — 250,000 tokenswikitext / 60,000 chat; T4 — 120,000 / 40,000.
ppl_delta_vs_fp16 = ppl_quant_wt2 − ppl_fp16_wt2, always within the same stack andthe same tier. Recomputing this from the columns reproduces the published value on all 144 quantized rows (0 mismatches).
Throughput protocol — verbatim from notes:
> tok/s: greedy b1 64->256 median-of-3 (incl. prefill)
do_sample=False), batch_size=1 on all 161 rows.prompt_tokens=64; gen_tokens=256 new tokens withmin_new_tokens=max_new_tokens=256 on all 161 rows.
generate() call between torch.cuda.synchronize() calls, soprefill is included. tokens_per_sec = 256 ÷ median wall-clock seconds.
VRAM. vram_gb is torch.cuda.max_memory_allocated() / 2**30, with the peak counter reset after the model is on the GPU so the weights are counted. It is peak torch-allocated memory, not process or nvidia-smi memory.
colab_fit. Computed only on T4 rows, as vram_gb < 15.0. All 77 T4 rows read yes; all 84 A10 rows read n/a. No Colab session was run.
The two stacks do not share a Python environment, and the versions recorded per row differ accordingly. gptqmodel 7.3.2 requires transformers ≥ 5.4 and torch ≥ 2.8; autoawq 0.2.9 is deprecated upstream and was pinned to its last-tested combination. Two Modal images were built for this reason.
| Stack | torch_version | transformers_version | quantlib | load_backend | Rows |
|---|---|---|---|---|---|
| GPTQ | 2.8.0+cu128 | 5.14.1 | gptqmodel-7.3.2 | torch-fallback | 72 quantized |
| AWQ | 2.6.0+cu124 | 4.51.3 | autoawq-0.2.9 | gemm | 72 quantized |
| fp16 (AWQ stack) | 2.6.0+cu124 | 4.51.3 | autoawq-0.2.9 | hf-fp16 | 6 |
| fp16 (GPTQ stack) | 2.8.0+cu128 | 5.14.1 | gptqmodel-7.3.2 | hf-fp16 | 4 |
Three failed rows carry the AWQ stack versions; four failed rows have empty version fields because the batch call died before the in-container version probe ran. driver.log preflight lines record the same version triples on both A10 and T4.
This means cross-stack throughput comparison is confounded by the stack itself. Same model, same tier, same fp16 weights: Qwen/Qwen2.5-1.5B-Instruct on A10 measures 45.38 tok/s on the AWQ-stack image and 35.52 tok/s on the GPTQ-stack image. Perplexity is much less sensitive — the same two rows read 9.6394 and 9.6394 wikitext ppl. Treat tokens_per_sec as a within-stack, within-tier quantity.
deepgrove/Bonsai, all on A10, two distinct failure modes.wikitext2-train n=128, seeds 0/1/2. Quantization itselfcompleted (quant_time_s 125.5 / 116.1 / 114.2 s), but reloading the saved artifact raised OSError("/cache/quants/bonsai-0p5b__awq__wikitext2__n128__s{0,1,2} does not appear to have a file named modeling_qllama.py …") — the custom remote-code modeling file was not written next to the quantized weights, so the artifact could not be loaded and no eval numbers exist. Because the row never produced an artifact reference, there are no T4 counterparts to these three rows.
wikitext2-train n=128, seeds 0/1/2, plus the Bonsaifp16-on-the-GPTQ-stack baseline row. The whole A10 batch call died with a10 batch call failed: RuntimeError("ImportError: cannot import name 'LossKwargs' from 'transformers.utils' …") — Bonsai's remote modeling code against transformers 5.14.1. No quantization was attempted; quant_time_s, notes, and all version fields are empty on these rows.
There are no quantized Bonsai numbers in this dataset. 2. Every GPTQ row ran on load_backend=torch-fallback — all 36 A10 rows and all 36 T4 rows. The optimized-kernel load path raised, and the loader fell back to BACKEND.TORCH. The GPTQ tokens_per_sec figures therefore measure a slow dequantize-in-PyTorch path, not an optimized GPTQ kernel, on both tiers. They are not an upper bound on what GPTQ can do on this hardware, and they should not be read as a kernel benchmark. The same fallback shows up in memory: GPTQ peak VRAM reaches 9.94 GB for Qwen2.5-1.5B on A10, above that model's 6.48 GB fp16 baseline, and varies within a cell (6.04–9.94 GB) in a way the AWQ rows do not (constant 4.57 GB). 3. The Bonsai rows carry their own caveat in notes, and it is load-bearing: > ternary weights but ops run in 16-bit (NOT end-to-end ternary inference); base model is not instruction-tuned
This appears on 5 of the 9 Bonsai rows (the other 4 have empty notes — see item 1). Nothing here demonstrates end-to-end ternary inference, and the Bonsai perplexity numbers come from a base model that was never instruction-tuned, evaluated partly on a chat-formatted held-out set. 4. webgpu_runnable is [UNVERIFIED] on all 161 rows. It is a hardcoded placeholder. No browser, WebGPU, or WebLLM measurement was made. Likewise there are no Jetson, ARM, or any other edge-device measurements in this data — the dispatch asked for them; they were not run. 5. Seeds: 3 per quantized cell, 1 for every baseline. Each (model, method, calibration set, size, tier) cell has exactly 3 rows, seeds 0/1/2 — and the seed changes only which calibration windows are drawn. The fp16 baseline is a single measurement per (model, stack, tier), so every ppl_delta_vs_fp16 inherits one unreplicated reference number. Within a row, tokens_per_sec is a median of 3 timed generations inside one process — it is not 3 independent loads, and there is no cross-process variance estimate for VRAM or perplexity. 6. Tier rows are not independent runs. Each artifact was quantized once on the A10; the A10 and T4 rows for that artifact are two evaluations of the same weights. 7. Perplexity is not comparable across tiers. The token caps differ (250k/60k on A10, 120k/40k on T4), so the same fp16 model reads 9.6394 on A10 and 9.0767 on T4. Only deltas within one tier and one stack are meaningful. Absolute ppl is also non-overlapping stride-2048, which is not the sliding-window protocol many published perplexity numbers use — do not compare these absolute values to numbers from elsewhere. 8. Throughput is one point on the workload surface: one fixed prompt, 64 prompt tokens, 256 greedy new tokens, batch size 1, prefill included. Different batch sizes, prompt lengths, or samplers would give different numbers, and the ranking between configurations is not guaranteed to survive that change. 9. colab_fit was never tested in Colab. It is a vram_gb < 15.0 threshold on T4 rows, nothing more. 10. Only 4 of the 72 quantized artifacts are published. The four canonical ones — each model × each method at wikitext2-train n=128 seed 0 — are downloadable from Mohaaxa/quantbench-artifacts, so the headline rows can be checked against the exact weights that produced them. The other 68 remain on the Modal volume and are re-derivable only by re-running the sweep. Note also that the notes column still reads artifact modal vol quant-bench:/quants/<id> (not pushed to HF yet) on all 144 quantized rows — that string was written during the run and is now stale for those four; hf_link is empty on 151 of 161 rows, and on the 10 fp16 rows it points at the base model, not a quantized artifact. 11. Base checkpoints for Qwen and SmolLM2 are not revision-pinned — those runs resolved whatever the default branch was on 2026-07-30/31. Only Bonsai has a pinned commit. 12. rows.csv is not a record of every attempt. driver.log shows two pre-run infrastructure failures whose rows were deliberately purged before the surviving run: 76 rows (marker='DeserializationError') and 38 rows (marker='torchvision'), 114 rows total. Those were environment failures that produced no model measurements, but the CSV is the surviving run, not the full history. The log is published alongside it. 13. Cost figures in the log are estimates, not invoices. driver.log ends with 161 rows; spend est $26.78; wall 0.9h; ran to completion; that dollar figure is the driver's own ledger multiplied by an assumed preemptible rate table (A10 $1.10/h, T4 $0.59/h), and the wall 0.9h refers only to the final resumed driver session.
Three efforts cover neighbouring ground. None of them measures what this benchmark measures, and one of them beats us on a dimension we do not claim.
Intel — Low-bit Open LLM Leaderboard (HuggingFace Space, live; newest commit ≈2026-07-10). Ranks community-submitted quantized models on 0-shot HellaSwag / MMLU / PIQA accuracy, run through EleutherAI's lm-evaluation-harness, across GPTQ, AWQ, AutoRound, bitsandbytes, llama.cpp and AQLM at 2–8 bits. It reports no throughput, latency or tokens/sec column of any kind, and the evaluation hardware is a submitter's choice (RTX 4090 by default, with A100/H100/H200/B200 options) — no T4 or A10. Calibration settings are not exposed as a parameter. It does publish the weights it quantizes itself, so downloadable artifacts are not a point of difference against it.
Lee et al., IJCAI 2025 — Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant (arXiv:2409.11055). Evaluates GPTQ, AWQ, SmoothQuant and FP8 across 12 instruction-tuned models from 1B to 405B on 13 accuracy benchmarks plus MT-Bench, using an H100 / A100 / RTX 6000 / A6000 cluster. Accuracy only — no throughput numbers. Calibration is left at each tool's default corpus and sample count, which confounds the calibration corpus with the method, and each configuration is run at a single fixed seed.
Kurtic et al. — "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization (arXiv:2411.02355, Neural Magic / Red Hat AI). Over 500,000 evaluations of FP8, INT8 and INT4 across the Llama-3.1 family, spanning academic benchmarks, LLM-judge and real-world tasks, with vLLM latency, queries-per-second and cost-per-query measured on A6000, A100 and H100. It varies calibration quality but never sweeps calibration size, and it compares against a single shared BF16 baseline rather than a per-toolchain one. It publishes checkpoints.
What is left uncovered, and is therefore what this benchmark is for: throughput on GPUs you can actually rent cheaply (T4 at $0.59/hr, A10 at $1.10/hr) rather than datacenter cards; the calibration axis swept as a controlled variable (two corpora × n=32/128/512 × 3 seeds); and a separate fp16 baseline per toolchain, which is what exposed the finding that the toolchain alone moves measured fp16 throughput by up to 27.8% on identical weights.
The sweep is a single Modal app, bench/quant_bench_app.py. Its entrypoints, quoted from the module docstring:
modal run --detach bench/quant_bench_app.py::launch
modal run bench/quant_bench_app.py::status
modal run bench/quant_bench_app.py::download # writes bench/rows.csv locallylaunch() spawns the driver and prints a fourth command for stopping it:
modal app stop quant-benchEntrypoint signatures (do not add flags that are not here):
launch(spend_ceiling_usd: float = 31.0, wall_budget_h: float = 20.0) — spawns thedrive function, which runs preflights on both stacks × both tiers, then works through the batch plan under a spend ceiling and a wall-clock budget.
status() — prints the driver log from the Modal volume.download(dest: str = "bench") — pulls rows.csv and the logs into a local directory.What the run needs:
quant-bench if missing; bothcontainer images are declared inline in the file with pinned pip versions, so there is no separate requirements file to install.
Salesforce/wikitext (wikitext-2-raw-v1, train + test) andteknium/OpenHermes-2.5. If OpenHermes is unavailable the run continues and the chat perplexity columns are left empty.
gpu="A10G" for quantize + first eval, gpu="T4" for the second eval pass;preemptible defaults only — the file sets no non-preemptible or region-pinning flags.
The run is resumable and idempotent: one JSON file per finished row under /cache/rows/, a BENCH_DONE marker per quantized artifact directory, and rows.csv rebuilt deterministically from the row files. Re-running skips any row that already exists, so a partial reproduction can be resumed rather than restarted.
Corrections and reproduction reports are welcome — a row that cannot be reproduced should be reported and will be marked or withdrawn.