docs(incidents): record six production incidents and cross-link the rules

[spashii]

Six serious production incidents were recorded only in one engineer's private assistant memory. This moves them into the repository, in the place the next person would actually look.

Where things went, and why

echo/docs/incidents/ is the new home. echo/docs/ is the engineering-notes tree (ADRs, plans, backlogs), which is the right altitude for these. The repo-root docs/ is the published customer site at docs.dembrane.com and was deliberately left alone.

One short page per incident, each leading with the rule so a skimmer gets it in one line, then the mechanism, the fix, and what prevents a repeat. README.md indexes them and states when a new page is warranted.

Cross-links, not copies. Where a rule already exists it stays where it is and gains a link to the write-up:

• echo/AGENTS.md: three lines under "Branching & Deployment", one under "Dramatiq & Async Rules", one narrow exception under "Directus Rules"
• echo/server/AGENTS.md: a pointer from "Background task design" to the event-loop write-up
• echo/docs/branching_and_releases.md: the release-step checklist now distinguishes adding an env key (safe) from removing one (release-gated)

The Dramatiq rules in particular were already well covered in echo/AGENTS.md, so nothing there was duplicated. The incident page exists to explain why those rules are worded as they are, since the short form reads as superstition without the failure behind it.

The six

Page	Rule
gitops-env-removal-release-gate.md	Never remove config from gitops before the code that reads it has shipped to a release tag
argo-self-heal-reverts-manual-changes.md	Never kubectl scale/patch production; Argo reverts it
directus-sync-is-indexed.md	A snapshot is_indexed: false over a live index named differently aborts the whole schema/apply, on production only
agent-missing-from-build-matrix.md	Every chart image reference needs a CI matrix entry that produces that tag
dramatiq-actor-event-loops.md	Never create and close a loop per call in a worker
validate-execution-not-construction.md	Validate a dependency migration by running the real path

Verification

Everything was checked against the tree before writing. All three requested checks passed:

• .github/workflows/ci.yml carries the dbr-echo-agent matrix entry (added in feat(agentic): foundation slices — agent image on main, v2 authz, design doc #745, with the reason recorded inline)
• echo/directus/sync/snapshot/fields/conversation_chunk/timestamp.json has is_indexed: true (fix(directus): mark conversation_chunk.timestamp as indexed in schema snapshot #724)
• echo/AGENTS.md already covers the Dramatiq rules; run_async_in_new_loop still exists and now runs on a shared background loop

One nuance worth flagging: run_async_in_new_loop no longer creates a new loop despite its name. Since #712 and #718 it submits to a single long-lived loop on a real OS thread. The blanket phrasing "no asyncio in Dramatiq actors" is therefore about loop lifecycle, not about async code being banned from workers. The write-up says so plainly, and echo/AGENTS.md gained a line calling out the misnomer.

Two live facts fell out of the verification and are recorded because they are actionable:

• DISABLE_REDACTION and FEATURE_FLAGS__ENABLE_CHAT_AUTO_SELECT are still set in the gitops values files even though neither field exists in settings.py on main. That is correct, not drift, and they should not be tidied away until a release containing refactor(chat): remove auto-select in favor of agentic chat #744 is live.
• The production echo-agent deployment stays broken until a release containing feat(agentic): foundation slices — agent image on main, v2 authz, design doc #745 goes out.

GitOps

echo/echo-gitops/ is a separate clone whose main branch deploys to production, and it is gitignored by this repo, so nothing from it is in this PR. Matching notes for the three infrastructure incidents were written into its AGENTS.md, inside the human-notes zone below the auto-generated marker, which the file's own self-healing policy preserves verbatim across regeneration. That change is left uncommitted in the gitops clone and needs a separate PR there.

🤖 Generated with Claude Code