Simulation benchmark for canary deployment strategies in LLM model updates.
When deploying a new model version, replacing the baseline immediately is risky. A canary deployment routes a fraction of traffic to the candidate model, monitors quality and latency, and only completes rollout if all guardrails pass.
Which rollout policy best balances safety and deployment progress when replacing one LLM version with another in production?
The portfolio covers the full serving lifecycle except model replacement:
- model-routing-complexity-bench: routes by difficulty, same model version
- serving-cost-model-v2: cost differences between model sizes
- slo-aware-autoscaling-sim: capacity during transitions
- graceful-degradation-bench: fallback if quality degrades under pressure
- model-ab-testing-sim: safe rollout of new model versions
In production, replacing a model is risky. A new checkpoint may degrade quality, increase latency, or raise error rates. Canary deployment mitigates this by routing a small traffic fraction to the candidate before full rollout.
The best strategy depends on whether the candidate model is good or bad:
| Scenario | Safest policy | Best rollout-capable policy |
|---|---|---|
| good_upgrade | progressive_canary | progressive_canary |
| bad_quality | no_rollout_baseline_only | fixed_canary_5pct |
| bad_latency | no_rollout_baseline_only | fixed_canary_1pct |
| marginal_regression | no_rollout_baseline_only | progressive_canary |
It exposes 100% of users to any regression with no detection capability. Safety score: 22-28 vs 86-87 for all canary policies.
| Policy | Bad exposure | Rollback count | Score |
|---|---|---|---|
| progressive_canary | 5.1% | 4 retries | 55.6 |
| progressive_canary_strict | 1.57% | 1 hard stop | 86.9 |
Retrying after a failed stage turns one regression into repeated user exposure.
Fixed canary at 1% or 5% limits bad exposure to under 1% and correctly rolls back bad candidates. It is the best monitoring strategy.
But it never completes a rollout of a good candidate.
It is the only rollout-capable policy that both validates and promotes the candidate. Full rollout completes at approximately request 1,200.
Fixed canary stays at its fraction indefinitely.
1% and 5% fixed canary produce nearly identical safety scores because the guardrail check fires at the same first batch of candidate requests.
A candidate slightly worse than baseline is correctly promoted by progressive canary if it stays below all configured guardrail thresholds. Guardrail calibration is as important as the rollout strategy itself.
| Policy | Role | Can complete rollout | Best for |
|---|---|---|---|
| instant_rollout | anti-pattern baseline | yes | nothing in production |
| no_rollout_baseline_only | safety ceiling | no | zero-risk environments |
| fixed_canary_1pct | regression monitoring | no | detecting bad candidates |
| fixed_canary_5pct | regression monitoring | no | detecting bad candidates faster |
| progressive_canary | progressive deployment | yes | promoting good candidates |
| progressive_canary_strict | safe progressive deployment | yes | promoting with hard safety guarantee |
| Scenario | Candidate | Description |
|---|---|---|
| good_upgrade | 7b_v2_good | better quality, lower latency, lower cost |
| bad_quality | 7b_v2_bad_quality | quality regression, higher error rate |
| bad_latency | 7b_v2_bad_latency | 55% slower decode |
| marginal_regression | 7b_v2_marginal | slightly worse but below guardrail thresholds |
cd ~/dev/model-ab-testing-sim
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 10 seconds. No GPU required.
results/
summary.csv
stage_log.csv
decision_map.csv
deployment_decision_map.csv
safest_policy_map.csv
policy_roles.csv
plots/
01_safety_score.png
02_bad_exposure.png
03_exposure_vs_safety.png
04_strict_vs_normal.png
05_rollout_speed.png
06_rollout_value_score.png
07_safety_vs_rollout.png
model-ab-testing-sim/
+-- src/
| +-- config.py
| +-- workload.py
| +-- simulator.py
| +-- bench.py
| +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt
This project is a calibrated simulation, not a live serving system.
Each request is routed to either the baseline or candidate model based on the current canary fraction. Guardrails are checked periodically against the observed metrics delta between baseline and candidate traffic.
The benchmark separates two distinct objectives:
Safety objective:
Which policy minimizes bad user exposure?
Answer: never roll out (zero deployment value).
Practical answer: fixed canary at 1-5 percent.
Deployment objective:
Which rollout-capable policy best balances safety and progress?
Answer: fixed canary for bad candidates,
progressive canary for good candidates.
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
If you need a default canary deployment strategy:
- start with fixed_canary_5pct for initial validation
- transition to progressive_canary_strict once validation passes
- never use instant rollout in production
- calibrate guardrail thresholds carefully
- treat marginal regressions as a threshold configuration problem
- design.md -- detailed design rationale and modeling assumptions
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza 2026