Skip to content

Add SLI/SLO definitions and burn-rate alerts for Tier-1 services#9

Draft
bdevierno1 wants to merge 1 commit into
mainfrom
add-slo-definitions-and-burn-rate-alerts
Draft

Add SLI/SLO definitions and burn-rate alerts for Tier-1 services#9
bdevierno1 wants to merge 1 commit into
mainfrom
add-slo-definitions-and-burn-rate-alerts

Conversation

@bdevierno1

Copy link
Copy Markdown
Owner

Summary

  • docs/slo-policy.md — In-repo SLO policy: service tier classification (all three services are Tier 1 per the org policy; no Tier-0 payments/auth exists in this app), SLI definitions sourced from the existing /api/metrics Prometheus endpoint (vecna_operations_total), SLO targets (vecna-api 99.9 %, vecna-agent 99.0 %, vecna-frontend 99.9 %), error-budget table, Green/Yellow/Red band thresholds, operational constraints, and review triggers.
  • monitoring/alerts/slo-burn-rate.yml — Prometheus multi-window burn-rate rules for vecna-api and vecna-agent. Six recording rules (5 m, 30 m, 1 h, 6 h, 3 d error-rate windows) plus 6 alert rules covering all three budget states: Red (≥ 14.4 × / 1 h + 5 m guard), Yellow (≥ 6 × / 6 h + 30 m guard), Green-elevated (≥ 3 × / 3 d + 6 h guard). Every alert links to the runbook.
  • docs/runbooks/slo-burn-rate-runbook.md — Response runbook: alert reference table, diagnosis commands against SQLite and /api/health/deep, per-severity actions (Red = page + freeze deploys, Yellow = business-hours page + ticket, Green = sprint ticket).

What is not in scope

  • Latency SLI (vecna_operation_duration_seconds histogram): the backend does not currently emit duration histograms. Noted in slo-policy.md §2.1; tracked for follow-up.
  • vecna-frontend + vecna-desktop alert rules: both require a Blackbox Exporter probe configured in the Prometheus scrape config, which is out of scope for this PR. SLO targets for both are defined in the policy doc.
  • Prometheus scrape config / prometheus.yml: not present in this repo. The monitoring/alerts/slo-burn-rate.yml is a drop-in rule_files: entry or Kubernetes PrometheusRule CR.

Test plan

  • grep -c '^+' <(git diff main...) — confirm ≤ 400 added lines (actual: 385).
  • Paste monitoring/alerts/slo-burn-rate.yml into Prometheus rule linter or run promtool check rules monitoring/alerts/slo-burn-rate.yml — should pass with no errors.
  • Confirm every alert annotation contains runbook_url pointing to docs/runbooks/slo-burn-rate-runbook.md.
  • Review docs/slo-policy.md against org policy (file_id a86fb303) — targets and budget minutes match pinned memory.

Rollback

Delete the three new files and the monitoring/ directory. No existing code changed.


🤖 Generated with Vecna

Co-Authored-By: Vecna noreply@vecna.ai

- docs/slo-policy.md: service tier classification, SLI definitions,
  SLO targets (vecna-api 99.9%, vecna-agent 99.0%, vecna-frontend 99.9%),
  error-budget table, Green/Yellow/Red band thresholds, operational
  constraints, and SLO review triggers.
- monitoring/alerts/slo-burn-rate.yml: Prometheus multi-window burn-rate
  rules for vecna-api and vecna-agent covering all three budget states
  (Red ≥14.4×/1h, Yellow ≥6×/6h, Green-elevated ≥3×/3d). Every alert
  links to the runbook.
- docs/runbooks/slo-burn-rate-runbook.md: response runbook stub with
  alert reference table, diagnosis commands, and per-severity actions.

Co-Authored-By: Vecna <noreply@vecna.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant