docs(incidents): record six production incidents and cross-link the rules - #918
Closed
spashii wants to merge 1 commit into
Closed
docs(incidents): record six production incidents and cross-link the rules#918spashii wants to merge 1 commit into
spashii wants to merge 1 commit into
Conversation
…ules Six production failures were recorded only in one engineer's private assistant memory. This moves them into echo/docs/incidents/ as one short page each: rule first, then mechanism, fix, and prevention. - gitops env removal took prod down (2026-07-02): gitops main syncs to prod in minutes, prod code only on a release tag - Argo self-heal reverts kubectl scale/patch - directus-sync is_indexed trap: prod-only DROP INDEX abort - echo-agent missing from the ci.yml build matrix - event-loop corruption in Dramatiq actors (why the AGENTS.md rules exist) - validate a dependency migration by execution, not construction Every claim was checked against the tree: ci.yml carries the dbr-echo-agent matrix entry, the conversation_chunk.timestamp snapshot has is_indexed: true, and settings.py uses extra="ignore" throughout. The rules themselves stay where they already live. AGENTS.md and branching_and_releases.md gain links to the write-ups rather than copies of them. Matching human-zone notes for the three infrastructure incidents are prepared in echo-gitops/AGENTS.md but left uncommitted, since that clone's main branch deploys to production. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #917, which was cut from a shared working tree and picked up three unrelated commits while carrying none of the incident files. This branch is off main with only the ten intended files.
What this is
Six production failures that lived only in one engineer's head are now written into the repository, next to the code that would prevent them, and cross-linked from the AGENTS.md files where the rule already exists.
gitops-env-removal-release-gate.md— gitops main reaches production in minutes, production code only advances on a release tag. Removing a still-required key crash-loops production. 2026-07-02.argo-self-heal-reverts-manual-changes.md—kubectl scale/patchgets reverted.agent-missing-from-build-matrix.md— a chart workload with no CI matrix entry sits in ImagePullBackOff, invisibly, because Argo reports synced and the service has no ingress.directus-sync-is-indexed.md— snapshotis_indexed: falseover a differently-named live index aborts the wholeschema/apply, on production only.dramatiq-actor-event-loops.md— why the no-asyncio-in-actors rule exists.validate-execution-not-construction.md— a dependency upgrade that passed construction and failed execution.Note on tone
These are written as a record of what happened and why, not as blame and not as ceremony. No invented timestamps or severities: timing comes from git history, with the one exception of the 2026-07-02 date, which is known.