docs(incidents): record six production incidents and cross-link the rules

echo/AGENTS.md

@@ -116,6 +116,9 @@ See [docs/branching_and_releases.md](docs/branching_and_releases.md) for the ful
 - **Releases**: tagged from `main` every ~2 weeks → auto-deploys to production
 - **Hotfixes**: branch off release tag → fix → new release → cherry-pick back into `main`
 - Always check for Directus data migrations before deploying. See [docs/database_migrations.md](docs/database_migrations.md)
+- **Config removals are release-gated.** GitOps `main` syncs to production in minutes, but production code only advances on a release tag. Remove the field from `settings.py`, ship the tag, then remove the key from gitops. See [docs/incidents/gitops-env-removal-release-gate.md](docs/incidents/gitops-env-removal-release-gate.md)
+- Never `kubectl scale` / `kubectl patch` production. Argo self-heal reverts it. See [docs/incidents/argo-self-heal-reverts-manual-changes.md](docs/incidents/argo-self-heal-reverts-manual-changes.md)
+- Adding a service to the Helm chart? Add its image to the `ci.yml` build matrix in the same change. See [docs/incidents/agent-missing-from-build-matrix.md](docs/incidents/agent-missing-from-build-matrix.md)
 
 ## Architecture
 
@@ -155,6 +158,7 @@ Long-running progress streams via Server-Sent Events backed by Redis pub/sub (re
 - `SkipRetryOnUnrecoverableError` middleware skips retries for `TypeError`, `SyntaxError`, `AttributeError`, `ImportError`, `NotImplementedError`
 - To invoke async code from a Dramatiq actor: `run_async_in_new_loop` from `dembrane.async_helpers`. Never `asyncio.run` (clashes with nested loops)
 - Wrap blocking I/O in async endpoints with `run_in_thread_pool` from `dembrane.async_helpers` (Directus, service-layer, S3, token counting). Don't wrap already-async calls (e.g. `rag.aquery`)
+- `run_async_in_new_loop` is a misnomer: it submits to ONE long-lived loop on a real OS thread, it does not make a loop. Never create/close a loop per call. Why these rules exist: [docs/incidents/dramatiq-actor-event-loops.md](docs/incidents/dramatiq-actor-event-loops.md)
 
 ### LLM Model Groups
 
@@ -184,6 +188,8 @@ Which group powers which feature is non-obvious, so don't downgrade silently.
 3. Pull the schema: `cd directus && bash sync.sh -u http://directus:8055 -e admin@dembrane.com -p admin pull`
 4. Commit the snapshot JSON under `directus/sync/snapshot/`. That is the source of truth; the one-shot migration script does not need to be committed
 
+Narrow exception: a field's `schema.is_indexed` must be `true` in the snapshot whenever the column is indexed on production, even though a `pull` from a local database writes `false`. Getting this wrong aborts the entire `schema/apply` on production and nowhere else. See [docs/incidents/directus-sync-is-indexed.md](docs/incidents/directus-sync-is-indexed.md)
+
 ### Python DirectusClient
 
 - `create_item` / `update_item` return `{"data": {...}}`. **MUST** unwrap with `["data"]`

echo/docs/branching_and_releases.md

@@ -39,7 +39,7 @@ Until we can fully synchronize the Directus schema push and server deployment, p
 2. **Pre-release checks**:
    - Check for new env vars (`settings.py` fields, `config.ts` exports)
    - Run Directus data migrations if needed (see [database_migrations.md](database_migrations.md))
-   - Update deployment env vars in the GitOps repo if needed
+   - Update deployment env vars in the GitOps repo if needed. **Adding** a key is safe any time; **removing** one is release-gated, because GitOps `main` reaches production immediately while the code does not. See [incidents/gitops-env-removal-release-gate.md](incidents/gitops-env-removal-release-gate.md)
 3. **Tag and release** — create a GitHub release from a commit on `main`
 4. The release triggers **auto-deployment to production**:
    - Backend: new image tags are picked up by the GitOps repo (`dembrane/echo-gitops`, Argo CD auto-sync)

echo/docs/incidents/README.md

@@ -0,0 +1,37 @@
+# Production incidents
+
+Short write-ups of production failures whose cause was not obvious from reading the
+code. Each page leads with the rule, then explains the mechanism precisely, what the
+fix was, and how to avoid a repeat.
+
+These are engineering notes, not reports. No severities, no timelines, no blame. Where
+timing matters, use `git log` / `git blame` on the commits cited rather than trusting a
+date written here.
+
+## When to add a page here
+
+Add one when a failure meets both tests:
+
+- the cause would surprise a competent reader of the code, and
+- the preventing rule is not already stated somewhere a person would look
+
+If the rule already lives in an `AGENTS.md`, link to that rule from here and link back
+from the rule. Do not restate it in two places.
+
+## Index
+
+| Incident | Rule in one line |
+| --- | --- |
+| [gitops-env-removal-release-gate.md](gitops-env-removal-release-gate.md) | Never remove config from gitops before the code that reads it has shipped to a production release tag. |
+| [argo-self-heal-reverts-manual-changes.md](argo-self-heal-reverts-manual-changes.md) | Never `kubectl scale` or `kubectl patch` production. Argo reverts it. Change capacity in gitops values. |
+| [directus-sync-is-indexed.md](directus-sync-is-indexed.md) | A snapshot field with `is_indexed: false` over a live column indexed under a different name aborts the whole `schema/apply` on production. |
+| [agent-missing-from-build-matrix.md](agent-missing-from-build-matrix.md) | Every deployment referencing an image tag needs a build-matrix entry that produces that tag. Nothing else notices when one does not. |
+| [dramatiq-actor-event-loops.md](dramatiq-actor-event-loops.md) | Never create and close an event loop per call inside a worker. Go through `run_async_in_new_loop`, which owns one long-lived loop. |
+| [validate-execution-not-construction.md](validate-execution-not-construction.md) | A dependency migration is validated by running the real path end to end, never by confirming imports and constructors work. |
+
+## Related
+
+- [../branching_and_releases.md](../branching_and_releases.md): environments, release tags, gitops
+- [../database_migrations.md](../database_migrations.md): Directus schema and data migrations
+- `../../AGENTS.md`: the cross-cutting rules these incidents produced
+- `../../echo-gitops/AGENTS.md`: infrastructure notes, human-notes zone at the bottom

echo/docs/incidents/agent-missing-from-build-matrix.md

@@ -0,0 +1,91 @@
+# echo-agent was never built for production
+
+**Rule: every deployment that references an image tag needs a matching entry in the
+`ci.yml` build matrix that actually produces that tag. Nothing in the pipeline notices
+when one is missing, and an internal ClusterIP service can stay broken indefinitely
+without a single external symptom.**
+
+## What happened
+
+The `echo-agent` deployment in the `echo-prod` namespace sat in `ImagePullBackOff` for
+weeks with no ready pods and therefore no service endpoints. The agentic chat backend
+was unavailable in production for the entire period.
+
+## Mechanism
+
+Two workflows build images, and they tag differently:
+
+- `.github/workflows/deploy-testing.yml` built `dbr-echo-agent`, tagging it with a
+  short 7-character SHA plus `testing`
+- `.github/workflows/ci.yml`, job `ci-build-and-push-servers`, built only
+  `dbr-echo-directus` and `dbr-echo-server`, tagged with the full 40-character
+  `github.sha`
+
+Production and Echo Next both deploy by full commit SHA, so their agent deployments
+requested `dbr-echo-agent:<full-sha>`. That tag was never produced by anything, because
+the only workflow that built the agent used the short form on a different branch. The
+registry's newest agent tags were `testing` and a short SHA, both stale.
+
+Nothing failed loudly:
+
+- CI was green. It builds the images in its matrix and has no knowledge of what the
+  Helm chart references.
+- Argo reported the application synced. The manifests matched git exactly; the
+  container simply could not pull.
+- The service is `ClusterIP` on port 8001 and is not on any ingress. Production ingress
+  exposes only `directus.dembrane.com` and `api.dembrane.com`, and the dashboard and
+  portal front ends are on Vercel. So every external health check stayed green.
+
+The gap between "declared in the chart" and "produced by CI" had no owner, and the one
+component affected happened to be invisible from outside the cluster.
+
+## Fix
+
+`dbr-echo-agent` was added to the `ci.yml` matrix in
+`feat(agentic): foundation slices — agent image on main, v2 authz, design doc`
+(d9e245cf, PR #745). It is present today, with the reason recorded inline so the entry
+does not get pruned as redundant:
+
+```yaml
+# Agent service. Also built by deploy-testing.yml for the testing
+# env; without this entry no main-SHA agent tag ever exists, so the
+# echo-next and prod agent deployments ImagePullBackOff.
+- name: dbr-echo-agent
+  # Repo-root context: the image bakes docs/ as the agent's
+  # read-only knowledge corpus.
+  context: .
+  dockerfile: echo/agent/Dockerfile
+  tag: dbr-echo-agent
+```
+
+The build context is the repository root, not `echo/agent/`, because the image bakes
+the docs corpus in as the agent's read-only knowledge base.
+
+Production only picks this up on a release tag, so the deployment stays broken until a
+release containing d9e245cf goes out.
+
+## Preventing a repeat
+
+- When adding a service to `helm/echo/`, add its image to the `ci.yml` matrix in the
+  same change. The chart reference and the build entry are one unit.
+- When adding an image to `deploy-testing.yml`, check whether `ci.yml` needs it too.
+  The two workflows serve different environments and neither implies the other.
+- For an internal service with no ingress, add a check that fails visibly. Green
+  external probes prove nothing about a `ClusterIP` workload.
+
+A quick audit at any time:
+
+```bash
+# image repositories the chart references
+grep -rn "repository:" echo-gitops/helm/echo/values.yaml
+# images CI actually builds
+grep -n "tag: dbr-" .github/workflows/ci.yml
+# what exists in the registry
+doctl registry repository list-tags dbr-echo-agent
+```
+
+## See also
+
+- [gitops-env-removal-release-gate.md](gitops-env-removal-release-gate.md): the same
+  code-versus-config skew, from the other direction
+- [../agentic-chat-design.md](../agentic-chat-design.md)

echo/docs/incidents/argo-self-heal-reverts-manual-changes.md

@@ -0,0 +1,67 @@
+# Manual kubectl changes to production get reverted
+
+**Rule: never `kubectl scale` or `kubectl patch` a production workload. Argo CD
+self-heal reverts it to the git-declared state within minutes. Capacity changes go
+through `helm/echo/values-prod.yaml` in `echo-gitops`.**
+
+## What happened
+
+During a transcription backlog, `kubectl scale deploy echo-worker-cpu --replicas=10`
+was run against the production cluster. Argo reset the deployment to its declared
+replica count on the next sync. Nothing about the backlog changed, and the time spent
+looked like progress while producing none.
+
+## Mechanism
+
+`argo/echo-prod.yaml` declares:
+
+```yaml
+syncPolicy:
+  automated:
+    prune: true
+    selfHeal: true
+```
+
+`selfHeal` means Argo continuously compares live cluster state against the rendered
+manifests from gitops `main` and reconciles any difference it finds, whatever produced
+it. A hand-edited replica count or resource limit is drift, so it is undone. `prune`
+means a live object with no manifest behind it is deleted outright.
+
+The behaviour differs by resource:
+
+- deployments without an HPA (Neo4j, for example) have their `spec.replicas` reverted
+  hard
+- deployments with an HPA are managed by the autoscaler within the bounds declared in
+  gitops, so a manual scale is either overwritten by Argo or immediately re-evaluated
+  by the HPA, and in both cases the bounds in git win
+
+## What to do instead
+
+Change the declaration and let Argo apply it:
+
+1. Edit replicas, resources, or HPA bounds in `helm/echo/values-prod.yaml` (or the
+   relevant template under `helm/echo/templates/`).
+2. Commit to gitops `main`, push, and watch the sync.
+
+Note the ceiling before assuming a replica bump helps: the production node pool sits
+at its Terraform limit, so raising replicas past available capacity yields `Pending`
+pods with `FailedScheduling: Insufficient cpu/memory`. Real headroom comes from
+Terraform (a larger pool) or from reclaiming capacity by removing an unused workload
+through gitops.
+
+## What is still fine by hand
+
+Self-heal only reconciles declared object state, so diagnosis is unaffected:
+
+- read-only `kubectl get` / `describe` / `logs`
+- `kubectl exec` into a running pod, including a one-off enqueue or a backfill script
+- `kubectl port-forward` for Loki or Grafana
+
+Deleting a pod is also fine when the intent is a restart: the replica set recreates it
+and the declared state never changed.
+
+## See also
+
+- [gitops-env-removal-release-gate.md](gitops-env-removal-release-gate.md): the other
+  half of the same fact, that gitops `main` is production
+- `echo-gitops/AGENTS.md`, recipe "Tune worker CPU scaling"

echo/docs/incidents/directus-sync-is-indexed.md

@@ -0,0 +1,89 @@
+# directus-sync push fails on production with a DROP INDEX for an index that does not exist
+
+**Rule: if a column is indexed on production, the snapshot field file must say
+`is_indexed: true`. A `false` there against a live index named by any other convention
+makes Directus emit a `DROP INDEX` for a name that was never created, and the entire
+`schema/apply` aborts.**
+
+## What happened
+
+`cd echo/directus && bash sync.sh ... push` against production returned a 500:
+
+```
+drop index "conversation_chunk_timestamp_index" - index "conversation_chunk_timestamp_index" does not exist
+```
+
+Nothing applied. The same push succeeded against local and testing.
+
+## Mechanism
+
+`sync.sh push` ends in `POST /schema/apply`. Directus builds the diff by introspecting
+the live database and comparing it against the committed snapshot under
+`echo/directus/sync/snapshot/`.
+
+Directus introspection reports `is_indexed: true` for a column if *any* index covers
+it, regardless of that index's name. Production carries hand-applied performance
+indexes using an `idx_*` naming convention (for `conversation_chunk.timestamp`, indexes
+such as `idx_chunk_conv_timestamp` and `idx_chunk_recent_non_upload`). Those indexes
+are not represented anywhere in the repository, so a `pull` from a database that lacks
+them writes `is_indexed: false` into the snapshot.
+
+That leaves the diff as `is_indexed: true -> false` on production, which Directus
+translates into "drop the index on this column". The knex layer does not look up which
+index actually exists. It generates the name from its own convention,
+`{table}_{field}_index`, so it emits
+`DROP INDEX "conversation_chunk_timestamp_index"`. No such index exists. Postgres
+errors, and because `schema/apply` runs in a single transaction the whole push rolls
+back, including every unrelated collection and field change in the same snapshot.
+
+Local and testing never hit this: they have no `idx_*` indexes, so the column is
+genuinely unindexed, snapshot `false` matches live `false`, and no index statement is
+generated at all. The failure is production-only by construction.
+
+## Fix
+
+Set `is_indexed: true` in the snapshot field file. For
+`conversation_chunk.timestamp` this is
+`echo/directus/sync/snapshot/fields/conversation_chunk/timestamp.json`, done in
+`fix(directus): mark conversation_chunk.timestamp as indexed in schema snapshot`
+(4635762b, PR #724):
+
+```json
+"schema": {
+  ...
+  "is_indexed": true,
+```
+
+Source `true` matches live `true`, the diff disappears, and no index DDL is generated.
+The file is committed with `is_indexed: true` today.
+
+This is one of the narrow cases where editing snapshot JSON by hand is correct. The
+standing rule in `AGENTS.md` is never to hand-write these files, because they should be
+produced by `sync.sh pull` after running an idempotent migration script. That rule
+still holds for structure. It cannot hold here, since a `pull` from any environment
+without the production perf indexes reintroduces the `false`.
+
+## What not to do
+
+Creating `conversation_chunk_timestamp_index` on production so that the `DROP` has a
+target does clear the immediate 500, but the push then drops it while the `idx_*`
+indexes keep the column indexed. The next `pull`/`push` cycle regenerates the same
+mismatch, so the failure returns on every sync.
+
+## Applying it to the next field
+
+When a push fails on production with `drop index ... does not exist`:
+
+1. Read the table and column out of the index name in the error.
+2. Confirm the column really is indexed on production
+   (`\d <table>` or a query against `pg_indexes`), under whatever name.
+3. Set `is_indexed: true` on that field's snapshot file and push again.
+
+Directus system tables are not affected. Only `directus_users` is present in the
+snapshot; `directus_activity`, `directus_revisions` and `directus_presets` are not, so
+their indexes never take part in `schema/apply`.
+
+## See also
+
+- `../../AGENTS.md`, "Directus Rules (Critical)"
+- [../database_migrations.md](../database_migrations.md)