docs(ops): runbook for triaging ERROR lines in the OODA daemon log

[rysweet]

Investigation: root cause of recent ERROR lines in ooda.log

This PR delivers the investigation of the ERROR line(s) in the daemon logs
(~/.simard/ooda.log and ~/.simard/state/ooda.log) and adds an operator
runbook capturing the findings + remediation.

The four ERROR signatures found (located & quoted)

1. Legacy DB-backup — "DB file does not exist" (10× on 2026-05-21)

   ERROR: DB backup failed 3..12 consecutive times — ... 'cognitive-memory' failed to initialize: DB file does not exist, nothing to back up

   • Root cause: the old native lbug-WAL file-copy backup opened the store at a path with no DB file. It escalated WARN→ERROR after 3 consecutive misses.
   • Status: resolved by design — that path was removed (De-fork Phase 2b: make the amplihack-memory-lib library the SOLE cognitive-memory backend for Simard and DELETE Simard's native fork. The library is now complete (persistent lbug GraphStore, all memor #2307) and replaced by the verified backup (Resilience: cognitive-memory backups stopped Jun 20 (wrong path) + recurring main-store WAL corruption #2420), which logs only WARN: verified backup FAILED on error. Current daemon logs verified backup OK every cycle.

2. Legacy DB-backup — "No space left on device (os error 28)" (2026-05-29)

   ERROR: DB backup failed 3 consecutive times — ... 'backup-copy-tmp': No space left on device (os error 28)

   • Root cause: environmental ENOSPC — home partition was full; WAL write, checkpoint, and backup-copy all failed. Same legacy path.
   • Impact then: backups stopped and the OODA cycle errored (cognitive-memory WAL unwritable).
   • Status: historical; disk now at 74% and managed by tiered disk-health (emergency cleanup + disk-health-check recipe + reclaim).

3. recipe_runner_rs::runner ... check-and-clean failed (2× on 2026-07-20, the most recent ERROR token)

   [simard] WARN: disk health check failed: ... [... ERROR recipe_runner_rs::runner] Step 'check-and-clean' failed: ... amplihack copilot failed (exit 1)

   • Key finding: this is a daemon-level WARN. The ERROR token belongs to the child recipe-runner-rs process whose stderr the older build embedded into the WARN body — the primary cause of false-positive "recent ERROR in ooda.log" telemetry alerts (the trigger for this very investigation).
   • Root cause of the underlying failure: the disk-health recipe's single check-and-clean agent step failed because the amplihack copilot agent binary exited 1 — a transient agent-invocation failure.
   • Impact: minimal/self-limiting; best-effort check re-runs next cycle and never aborts OODA. Current daemon logs disk health recipe: OK and the reworked trigger (Rework Simard disk-health to remove the forbidden JSON-emit->Rust-parse->Rust-act pattern, per the brief at /home/azureuser/.copilot/session-state/465c3cbf-6e73-43e5-b917-1b011291725b/files/ws3-disk-h #4722) records by exit status without embedding child stdout.

4. Legacy python-bridge — "Suite progressive failed" (state/ooda.log, 2026-04-17)

   __main__ ERROR Suite progressive failed ... BrokenPipeError: [Errno 32] Broken pipe

   • Root cause: BrokenPipeError — the retired python bridge_server tried to flush a progressive-suite result to a stdout pipe whose reader had already exited (during teardown).
   • Status: legacy; state/ooda.log has not been written since a mid-May shutdown. The live daemon is the Rust OODA loop.

Scope & impact assessment

All four signatures are historical/legacy or transient; none indicate a live fault in the current daemon (cycle #2645 healthy, verified backups OK, disk-health OK, disk at 74%). No code fix is warranted — the code paths that emitted signatures 1–4 are already removed/reworked in the current tree.

Remediation delivered

Adds docs/howto/triage-ooda-log-error-lines.md (+ mkdocs nav): maps each signature to root cause, current status, impact, and remediation, with a decision checklist and guidance to key telemetry on daemon-emitted severity (the [simard] ERROR: tag) rather than a bare ERROR substring — which avoids the false positives caused by nested child-process log lines.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com