model-serving-benchmark

Overview

Benchmark comparing three LLM serving strategies — Naive, KV-Cache, and Continuous Batching — measuring throughput, TTFT, and client latency under increasing concurrent load.

For architecture and design rationale, see DESIGN.md.

---

Why This Matters

All three servers use the same model (GPT-2) and hardware (RTX 2070). The only difference is the serving strategy.

The results show that batching strategy is the dominant factor in serving performance — not the model or the hardware.

---

Models and Hardware

Spec	Value
Model	GPT-2 (117M)
GPU	NVIDIA RTX 2070
VRAM	8.6 GB
Framework	PyTorch 2.13 + FastAPI

---

Servers

Server A — Naive

• No KV-cache
• No batching
• Recomputes full attention every token
• One request at a time

Server B — KV-Cache

• Uses past_key_values
• Reduces per-token cost vs naive
• Still one request at a time

Server C — Continuous Batching

• Collects requests in a queue
• Runs batched prefill and decode together
• All requests in the batch share a single forward pass
• Simplified version of what production engines do

---

Results

Throughput (tok/s)

Concurrency	Naive	KV-Cache	Batched	Batched gain
1	157	144	162	1.0×
2	105	120	355	3.4×
4	49	52	707	14.5×
8	42	46	920	21.8×

TTFT — Time to First Token (ms)

Concurrency	Naive	KV-Cache	Batched
1	19	22	22
2	18	18	7
4	82	79	7
8	205	187	8

Client Latency (ms)

Concurrency	Naive	KV-Cache	Batched
1	203	222	198
2	609	531	180
4	2619	2445	181
8	6116	5602	231

Batched server extended scaling (64 output tokens)

Concurrency	Throughput (tok/s)	TTFT (ms)	Client latency (ms)
1	175	14.7	365
2	378	7.1	338
4	735	6.8	348
8	966	8.1	455
16	1912	6.8	522
32	4521	8.9	445

---

Key Findings

Finding 1 — Naive and KV-Cache collapse under load

Without batching, each request waits for the previous one to finish. Client latency grows from 200ms at concurrency=1 to 6100ms at concurrency=8.

KV-cache alone provides negligible improvement over naive under concurrent load.

Finding 2 — Batching keeps TTFT constant

The batched server maintains 7–9ms TTFT regardless of concurrency. Naive and KV-cache TTFT grows linearly with load (19ms → 205ms).

Finding 3 — Throughput scales with load in the batched server

Server	conc=1	conc=8	Scaling
naive	157 tok/s	42 tok/s	−73%
kvcache	144 tok/s	46 tok/s	−68%
batched	162 tok/s	920 tok/s	+469%

Finding 4 — KV-cache value requires batching to materialize

KV-cache reduces per-token compute cost. But without batching, the GPU sits idle between requests. Batching fills that idle time with other requests — that is when the KV-cache investment pays off.

Finding 5 — Batched server reaches 4521 tok/s at concurrency=32

25.8× the single-request throughput, with TTFT stable at 8.9ms. This is 108× the throughput of the naive server at its worst point.

---

Plots

File	Description
plots/main_comparison.png	3-server comparison: throughput, TTFT, latency
plots/batched_extended_scaling.png	Batched server scaling to conc=32
plots/speedup_over_naive.png	Speedup of kvcache and batched vs naive

---

Results Files

File	Description
results/naive_results.csv	Naive server benchmark
results/kvcache_results.csv	KV-cache server benchmark
results/batched_results.csv	Batched server benchmark (conc 1–8)
results/batched_extended_results.csv	Batched server extended (conc 1–32)

---

Repository Structure

model-serving-benchmark/
├── server/
│   ├── naive_server.py
│   ├── kvcache_server.py
│   └── batched_server.py
├── load_test.py
├── run_all_benchmarks.py
├── run_batched_extended.py
├── analyze_results.py
├── README.md
├── DESIGN.md
├── LICENSE
├── requirements.txt
├── results/
└── plots/

---

How to Run

1. Setup

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

2. Run all three servers sequentially

python3 run_all_benchmarks.py

3. Extended batched server scaling

python3 run_batched_extended.py

4. Generate plots

python3 analyze_results.py

5. Run a single server manually

uvicorn server.naive_server:app --host 0.0.0.0 --port 8001
uvicorn server.kvcache_server:app --host 0.0.0.0 --port 8002
uvicorn server.batched_server:app --host 0.0.0.0 --port 8003

---

Limitations

• Single model (GPT-2); results scale qualitatively to larger models
• Simplified batching — no dynamic sequence length management or preemption
• Same prompt repeated; real workloads have heterogeneous prompt lengths
• No persistent KV-cache across requests (no prefix reuse)
• Python-level batching window (10ms); production engines use kernel-level scheduling

---

References

• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (2022)
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)