Quantization Tradeoff — Qwen2.5-1.5B on a 4 GB GPU

Measures the real cost of quantization across three dimensions — size, speed, and quality — for the same model at Q4_K_M / Q8_0 / FP16, on a 4 GB RTX 3050.

Headline finding

Q4_K_M runs 3.3× faster (87.7 vs 26.8 tok/s) at 1/3 the size (986 MB vs 3 GB) than FP16, with no measurable accuracy loss on the eval below. On consumer hardware, 4-bit quantization is essentially free — which is exactly why 4-bit K-quants are the default for GPU-poor inference.

Results

quant	bits/weight	size	throughput	avg latency	quality (15-Q eval)
Q4_K_M	4	986 MB	87.7 t/s	0.53s	13/15 (87%)
Q8_0	8	1.6 GB	45.5 t/s	0.99s	13/15 (87%)
FP16	16	3.0 GB	26.8 t/s	1.88s	13/15 (87%)

Why smaller = faster (the key insight)

LLM inference is memory-bandwidth-bound, not compute-bound. Every token requires streaming all the model's weights from VRAM to the compute units — so fewer bits per weight = less data to move = faster. Throughput scales roughly with bits-per-weight (Q4 ≈ 2× Q8 ≈ ~3× FP16). Quantization is primarily a bandwidth win, not a compute win.

Honest caveat (the limit of this measurement)

The quality eval is 15 easy multiple-choice questions — all three models even got the same 2 wrong (a model limit, not a quant one). So the honest claim is "no measurable quality loss on this eval," not "no quality loss, ever." Quantization degradation shows up on harder tasks — multi-step reasoning, long context, edge cases — that a small MC quiz doesn't probe. A rigorous version would run hundreds of MMLU questions and report perplexity.

Run it

ollama pull qwen2.5:1.5b-instruct-q4_K_M
ollama pull qwen2.5:1.5b-instruct-q8_0
ollama pull qwen2.5:1.5b-instruct-fp16
pip install httpx matplotlib

python3 benchmark_quants.py   # speed per quant level
python3 eval_quants.py        # quality per quant level (deterministic, temperature=0)
python3 plot_quants.py        # -> quant_tradeoff.png

Stack

Python · httpx · matplotlib · Ollama · Qwen2.5-1.5B (Q4_K_M / Q8_0 / FP16)