Persistent/event-driven Reliability Governor follow-up

[Svaag]

Context

Production rollout of the timer-driven Reliability Governor is complete as of 2026-06-30.

Observed rollout state:

• Governor timer is enabled and running on the loop VM.
• Dry run and first live passes were conservative: no unexpected loop:approved labels.
• Missing Knowledge context routed work to loop:knowledge-gap.
• NOC LHP fetches worked for the observed handoffs and produced concrete CaseService payload hashes.
• Engineering daemon continued consuming only loop:approved; after Governor routing, approved count was zero.

Remaining work

1. Rollout hardening

• Make loop protocol label creation part of the managed rollout/runbook so first live pass cannot fail on missing loop:knowledge-gap / terminal labels.
• Review idempotency around comment/CDR writes after partial failures. The first production attempt wrote an audit record/comment before label mutation failed because labels were missing.
• Decide whether duplicate GitHub comments should be de-duplicated or left as audit evidence.

2. Knowledge-context recovery

• Debug why production Knowledge MCP/context returns missing with no included_refs for current queue items.
• Make the Governor's knowledge-gap output actionable enough for Knowledge Loop/humans to repair the missing context.
• Confirm stale/missing/contradictory Knowledge continues to deny conservatively.
• Re-run a production dry run after Knowledge context starts returning usable refs.

3. Persistent/event-driven Governor design and implementation

• Add durable inbox/outbox storage for Governor work and side effects.
• Add GitHub webhook wakeups instead of relying only on timer polling.
• Add NOC, Knowledge, and Engineering callback ingestion.
• Move callback handling to reconciliation-based authorization instead of direct callback actions.
• Preserve human merge as mandatory and keep Engineering daemon consuming only loop:approved.
• Make GitHub comments, labels, and local CDR persistence idempotent and replay-safe.
• Add reconciliation for missed webhook/callback events.
• Add metrics/logging/alerts for queue depth, decision counts, callback failures, CDR writes, and label transition failures.

Acceptance criteria

• Governor can resume safely after process restart or partial GitHub/API failure.
• Every external mutation has a durable outbox record and can be retried without duplicate unsafe effects.
• Webhook/callback events wake the Governor, but authorization is still derived from reconciled GitHub/Knowledge/NOC state.
• No callback directly applies loop:approved or causes Engineering execution without Governor reconciliation.
• CDRs remain the audit source for decisions and include enough identity material to detect stale LHP/Knowledge inputs.
• Engineering daemon remains label-gated on loop:approved; human merge remains mandatory.

Non-goals

• Do not relax the conservative Knowledge-deny policy.
• Do not introduce auto-merge.
• Do not let NOC/Knowledge/Engineering callbacks directly authorize work.

[hyrule-engineering-loop]

Reliability Governor Decision

• role: staff_sre_autonomous_operations
• decision: knowledge_gap
• next_loop: knowledge
• handoff_contract: knowledge_context_pack
• source: knowledge
• intent: legal / tier 4
• capability: none
• knowledge: missing / unknown
• reasons: Knowledge policy result is deny; Knowledge context returned no included_refs

{
  "affected_assets": [
    "AS215932/engineering-loop"
  ],
  "affected_customers": [],
  "affected_services": [],
  "allowed_paths": [],
  "authority_text_hash": "d28df48831d667acfcf3a26481d9640ee1b61fff21f5c06f80be09c93ba158aa",
  "blast_radius": "legal/commercial",
  "controlled_loops": [
    "engineering",
    "noc",
    "knowledge"
  ],
  "created_at": "2026-06-30T02:31:38.299132+00:00",
  "denial_reasons": [
    "Knowledge policy result is deny",
    "Knowledge context returned no included_refs"
  ],
  "expected_paths": [
    "legal/"
  ],
  "forbidden_paths": [],
  "governor_name": "Reliability Governor",
  "governor_role": "staff_sre_autonomous_operations",
  "handoff_contract": "knowledge_context_pack",
  "intent_type": "legal",
  "issue_id": "AS215932/engineering-loop#29",
  "issue_number": 29,
  "issue_text_hash": "65338112699440dac1802e2301e84b237074aedbea7bd7498725150a57e79672",
  "knowledge_authority_level": "unknown",
  "knowledge_context_pack_id": "ctx_f0760ef2d69f44faf2dba49ce90249f0",
  "knowledge_export_version": "local-20260623120725",
  "knowledge_status": "missing",
  "labels_to_add": [
    "loop:knowledge-gap"
  ],
  "labels_to_remove": [
    "loop:intake",
    "loop:candidate",
    "loop:approved",
    "loop:needs-context",
    "loop:needs-human"
  ],
  "lhp": null,
  "matched_capability": null,
  "next_loop": "knowledge",
  "policy_rules": [
    "deny stale, contradictory, missing, or errored Knowledge context"
  ],
  "record_id": "2defd2ff41b649c8e445",
  "repo": "AS215932/engineering-loop",
  "required_checks": [],
  "risk_tier": 4,
  "rollback_plan": "",
  "routing_decision": "knowledge_gap",
  "schema_version": "reliability-governor.cdr.v1",
  "source": "knowledge",
  "storage_path": "/var/lib/engineering-loop/reliability-governor/as215932-engineering-loop-29-2defd2ff41b649c8e445.json",
  "verification_method": ""
}