QuantBench

Independent 4-bit quantization benchmarks for small instruct models

How to read this. Lower Δ PPL vs fp16 is better (quality lost to quantization, measured on wikitext2-test). tok/s is greedy, batch size 1, 64→256 tokens, median of 3, prefill included — it is a comparable number across rows here, not a claim about optimized serving throughput. Each fp16 baseline is stack-specific, so deltas are always computed against the baseline from the same quantization stack. Full protocol on the methodology page.

ModelMethodGPUCalibrationSeedΔ PPL vs fp16tok/sVRAM (GB)Artifact (GB)BackendQuant lib
7 configurations that failed (kept on purpose)
Failures are results too: they say which stack/model combinations do not work today, and why. They are excluded from the leaderboard above.
ModelMethodGPUCalibrationSeedError
deepgrove/BonsaiawqA10wikitext2-train n=1280OSError("/cache/quants/bonsai-0p5b__awq__wikitext2__n128__s0 does not appear to have a file named modeling_qllama.py. Checkout 'https://huggingface.co//cache/quants/bonsai-0p5b__awq__wikitext2__n128__s0/tree/main' for available files.")
deepgrove/BonsaiawqA10wikitext2-train n=1281OSError("/cache/quants/bonsai-0p5b__awq__wikitext2__n128__s1 does not appear to have a file named modeling_qllama.py. Checkout 'https://huggingface.co//cache/quants/bonsai-0p5b__awq__wikitext2__n128__s1/tree/main' for available files.")
deepgrove/BonsaiawqA10wikitext2-train n=1282OSError("/cache/quants/bonsai-0p5b__awq__wikitext2__n128__s2 does not appear to have a file named modeling_qllama.py. Checkout 'https://huggingface.co//cache/quants/bonsai-0p5b__awq__wikitext2__n128__s2/tree/main' for available files.")
deepgrove/Bonsainone-fp16A10none n=00a10 batch call failed: RuntimeError("ImportError: cannot import name 'LossKwargs' from 'transformers.utils' (/usr/local/lib/python3.11/site-packages/transformers/utils/__init__.py)")
deepgrove/BonsaigptqA10wikitext2-train n=1280a10 batch call failed: RuntimeError("ImportError: cannot import name 'LossKwargs' from 'transformers.utils' (/usr/local/lib/python3.11/site-packages/transformers/utils/__init__.py)")
deepgrove/BonsaigptqA10wikitext2-train n=1281a10 batch call failed: RuntimeError("ImportError: cannot import name 'LossKwargs' from 'transformers.utils' (/usr/local/lib/python3.11/site-packages/transformers/utils/__init__.py)")
deepgrove/BonsaigptqA10wikitext2-train n=1282a10 batch call failed: RuntimeError("ImportError: cannot import name 'LossKwargs' from 'transformers.utils' (/usr/local/lib/python3.11/site-packages/transformers/utils/__init__.py)")