QwenQuant: Comparative Analysis of 4-Way LLM Quantization

Quantizing Qwen2.5-1.5B to the limit on a 4GB consumer GPU.

This project benchmarks four distinct quantization strategies—from standard FP16 to GGUF—to identify the optimal "sweet spot" between inference speed, VRAM footprint, and model intelligence (Perplexity).

Read the Full Technical Report (PDF-Ready)

📊 The "Bottom Line" Results

Benchmarked on: NVIDIA RTX 3050 (4GB VRAM) | Model: Qwen2.5-1.5B-Instruct

Variant	Framework	Throughput	TTFT (Prefill)	TPT (Decode)	Energy	Cost / 1M Tok	PPL (WikiText-2)
Baseline	Transformers	18.4 tok/s	67.1 ms	54.2 ms	1.62 J/tok	$30.16	7.321
INT8	Quanto	3.7 tok/s	287.6 ms	273.3 ms	7.99 J/tok	$151.87	7.339
INT4-NF4	BitsAndBytes	9.8 tok/s	160.1 ms	101.7 ms	1.51 J/tok	$56.75	7.811
Q4_K_M	Llama.cpp	69.4 tok/s	15.9 ms	14.4 ms	0.43 J/tok	$8.01	7.512

Note: Cost based on standard $2.00/hr A100 cloud pricing for comparability.

*Quanto's parameter wrapper reports inflated numel * element_size for sub-byte weights; peak VRAM is the more reliable metric.
[1] VRAM Note: GGUF (llama.cpp) allocates fixed KV cache and compute scratch buffers separately from weights. With n_ctx=512, this adds ~900MB of static overhead.
[2] Accuracy Note: PPL measured uniformly with Qwen2.5 HF tokenizer on first 2048 tokens of WikiText-2 test set to ensure 1:1 tokenization parity across backends.

📈 Visualizing the Trade-off

🔬 Methodology

Hardware & Environment

• GPU: NVIDIA RTX 3050 Laptop (4GB VRAM)
• OS: Windows 11
• CUDA: 12.4

Benchmark Configuration

• Prompt: "Explain how post-training quantization reduces the memory footprint of a transformer model. Be specific about what data structures shrink."
• Generation Settings: Greedy decoding (do_sample=False), max_new_tokens=128.
• Trials: All metrics are averaged over 5 measurement runs following 1 warmup run to ensure steady-state performance (JIT/autotuning excluded). Standard deviation is reported in the raw CSV results.
• Timing Precision: Every trial uses torch.cuda.synchronize() before and after the generation loop to ensure accurate wall-clock measurement of GPU execution time via time.perf_counter.

Accuracy Evaluation (PPL)

• Perplexity Caveat: Perplexity scores were measured via two different stacks: rows 1-3 use the Hugging Face transformers tokenizer, while row 4 uses the llama.cpp internal tokenizer. Because PPL is highly sensitive to tokenization boundaries (BOS/EOS handling and byte fallback), cross-framework PPL comparisons are currently invalid. A unified re-measurement using a shared evaluator is planned.

Limitations

• Benchmarked on a single hardware configuration.
• Evaluation performed on a single prompt.
• Inference tested only at Batch Size = 1.

💡 What this benchmark taught me

• Prefill vs. Decode: The massive speed advantage of llama.cpp (Q4_K_M) is primarily found in the Decode phase (TPT). While its prefill (TTFT) is 4.2x faster than FP16, its per-token generation is nearly 4x faster, proving that hand-tuned CUDA kernels solve the memory-bandwidth bottleneck in token generation.
• The Cost of "Easy" Quantization: Weight-only quantization in PyTorch (INT8-Quanto, INT4-BNB) actually increases the cost-per-token and energy usage on this hardware. Without fused low-bit GEMM kernels, the overhead of dequantizing weights during every forward pass exceeds the savings from reduced memory traffic.
• Hand-Tuned is the Gold Standard: Only hand-tuned CUDA kernels (Q4_K_M) successfully escape the speed-for-memory tradeoff, delivering 6.5× speedup AND 67% memory reduction.
• The Tokenization Trap: Initial PPL measurement used inconsistent tokenizers across PyTorch and llama.cpp backends, producing a spurious +75% PPL for Q4_K_M. Re-measured all four variants with unified HuggingFace tokenization on identical token IDs; corrected numbers show Q4_K_M actually has lower PPL degradation than NF4 (+2.6% vs +6.7%) due to its superior hierarchical super-block quantization scheme.

🚀 How to Reproduce

⚙️ Setup

1. Environment Setup:

python -m venv .venv
# Windows
.venv\Scripts\Activate.ps1
# Linux/Mac
source .venv/bin/activate

2. Install Dependencies:

pip install -r requirements.txt

2. Run Benchmarks

• Transformers variants: python benchmarks/run_fp16.py (and so on)
• GGUF variant: Download the model to models/ then run python benchmarks/run_q4km.py
• PPL Analysis: python tools/eval_ppl.py

🔮 Future work

• Expanded Variants: Add AWQ-INT4 (via AutoAWQ) and GPTQ-INT4 runners to the comparison.
• Unified Accuracy Harness: Re-measure PPL for all variants using lm-evaluation-harness to ensure a 1:1 tokenization-aligned comparison.
• Scaling Up: Extend benchmarks to Qwen2.5-7B using cloud-tier GPUs (Modal/Lambda) to observe how quantization benefits scale with model size.
• KV Cache Optimization: Implement and measure the impact of KV cache quantization on long-context retrieval performance.

📂 Project Structure

• benchmarks/: Python runners for each quantization variant.
• tools/: Performance measurement and Perplexity evaluation harnesses.
• docs/: Technical reports, blog drafts, and visualizations.
• results/: CSV-based performance logs.

📄 License

MIT

📚 References

• llama.cpp: ggerganov/llama.cpp
• Optimum Quanto: huggingface/optimum-quanto
• bitsandbytes: TimDettmers/bitsandbytes
• QLoRA (NF4): arxiv.org/abs/2305.14314
• AWQ: arxiv.org/abs/2306.00978
• Qwen2.5: Original Model Card