docs: Tier-0 incident response hardening (rollback runbooks, SEV1/SEV2, postmortem template)#8
Draft
bdevierno1 wants to merge 1 commit into
Draft
docs: Tier-0 incident response hardening (rollback runbooks, SEV1/SEV2, postmortem template)#8bdevierno1 wants to merge 1 commit into
bdevierno1 wants to merge 1 commit into
Conversation
- docs/runbooks/production-rollback-runbook.md — per-service rollback steps for vecna-api, vecna-runner, vecna-db; kubectl rollout history/ undo/status for K8s path; non-reversible DB migration escalation rule (page DB owner + on-call DBA, do NOT auto-rollback schema) - docs/runbooks/sev1-response.md — SEV1 definition, rollback-first actions, IC role, exec comms every 30 min, recovery criteria, anchor URL for alert annotations - docs/runbooks/sev2-response.md — SEV2 definition, rollback-first actions, IC role, hourly comms cadence, recovery criteria, anchor URL - docs/runbooks/postmortem-template.md — blameless postmortem template covering timeline, contributing factors, impact scope, detection/ response delta, action items with owner+due-date, anti-blame checklist; required for every SEV1/SEV2; cross-linked from both SEV runbooks Co-Authored-By: Vecna <noreply@vecna.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/runbooks/production-rollback-runbook.md— Tier-0 rollback runbook coveringvecna-api(systemd + kubectl paths),vecna-runner(stuck-operation recovery), andvecna-db(SQLite schema migration escalation). Implements the golden rule (rollback before debug) and the non-reversible DB migration rule (page DB owner + on-call DBA; do NOT auto-rollback schema). Includes post-rollback 5-item verification checklist.docs/runbooks/sev1-response.md— SEV1 definition (customer-wide outage / data loss), 5-min acknowledge SLA, rollback-first action sequence, IC role and exec comms cadence (every 30 min), recovery criteria, anchor URL (#runbook-sev1) for alert annotation embedding.docs/runbooks/sev2-response.md— SEV2 definition (error rate > 5%, p99 > 3× SLO), IC role, hourly comms cadence, escalation ladder, anchor URL (#runbook-sev2) for alert annotation embedding.docs/runbooks/postmortem-template.md— Blameless postmortem template required for every SEV1/SEV2 within 3 business days. Covers incident header, impact scope, timeline, detection/response deltas, root cause + contributing factors, action items (owner + due date), anti-blame checklist, and sign-off table. Cross-linked from both SEV runbooks.Out of scope
Test plan
#runbook-sev1,#runbook-sev2,#runbook-rollback,#postmortem-templateare in the rendered documentsvecna-apidev instance:git checkout <sha>→pip install -r requirements.txt→uvicornrestart →curl /api/health/deeppassesschema_migrate.pyis additive-only (no DROP COLUMN) — verified in code review🤖 Generated with Vecna
Co-Authored-By: Vecna noreply@vecna.ai