Serving Benchmark — Qwen2.5-1.5B on a 4 GB GPU

Measures the real latency and throughput of a locally-served LLM, then finds the throughput-vs-latency tradeoff under concurrent load.

Headline finding

On a 4 GB RTX 3050 serving Qwen2.5-1.5B via Ollama, throughput peaks at ~113 tok/s at concurrency 4 (+35% vs sequential), but tail latency (p99) grows from 0.77s → 4.02s. Beyond concurrency 4 the card is oversubscribed — throughput drops with no latency benefit. Optimal concurrency ≈ 4.

Results

Sequential (warmed up): 84 tok/s · p50 0.51s · p95 0.68s · p99 0.76s

Under concurrent load:

concurrency	throughput	p50	p95	p99
1	84 t/s	0.53s	0.75s	0.77s
4	113 t/s	1.35s	1.82s	1.87s
8	105 t/s	3.34s	3.89s	4.02s

What the metrics mean

• Latency — time for one request. Throughput — tokens generated per second.
• p50 / p95 / p99 — percentiles. The average hides the slow tail; p95/p99 are what your unhappiest users actually feel. (In an early run, a p99 of 3.8s hid behind a 0.67s average.)
• Warmup — the first request pays a one-time model-load (cold-start) cost. The benchmark fires and discards a few warmup requests so the measured numbers are reproducible.

The tradeoff (the lesson)

Concurrency raises throughput (the GPU overlaps work) — up to a sweet spot. Past it, oversubscription degrades both throughput and latency. This is the core knob inference engineers tune (batch size / max concurrency): you can't just crank it.

Run it

ollama pull qwen2.5:1.5b
pip install httpx matplotlib
python3 benchmark.py     # sequential: latency, throughput, percentiles (with warmup)
python3 load_test.py     # concurrency sweep -> the tradeoff table
python3 plot.py          # -> tradeoff.png

Stack

Python · httpx (async) · matplotlib · Ollama · Qwen2.5-1.5B