feat(observability): WS-6a + WS-6c — D1's attribution stamp exists as a substrate

ai-company-brain/work_plan.md

@@ -73,7 +73,7 @@ gap; calendar P3 was found already shipped (with revised roll-over semantics).
 | WS-3 | **Isolation ladder** (BO-7 / HH-6 / B6 Tier 1→2, `tool_scope` deny, T2 for non-first-party agents) | `permissions_sandbox_b6.md` + `agent_platform_hardening_2026-07.md` Part 1 | 🟢 Tier 1 | Tier 1 container flags partially landed 2026-07-27 (competitive log) — reconcile B6 first. T2 is its own sub-project; required before Agent Workshop opens to non-engineers. **OWNER-GATE:** `AGENT_PERMISSION_MODE=enforce` flip. |
 | WS-4 | **Event-bus consumer + durable queue** (BO-20) | checklist §BO-20 | 🟢 | `ingestion.worker` confirmed absent. Prereq for WS-11 Slice 4 and multi-channel triggers. |
 | WS-5 | **CI gates real** (BO-17/BO-18) | checklist §F | 🟡 Docs | Un-gate evals, blocking gitleaks, coverage floor. ~~AGENT-SAFE~~ → **mixed: the highest-value item is a GitHub *settings* change an agent cannot make.** **Audited 2026-08-01 → NO-GO**: §F has zero testable "done when" ("per the existing plan", "a few green PRs", "for foundation packages"), its ratchet-plan anchor points at a path that moved to `specs/archive/` (3 stale citations live *in the workflow files*), and BO-17 reads ☐ while half of it shipped (blocking ruff-correctness + xenon, a frontend tsc/vitest job, gitleaks, per-PR health). **THE MISSING ITEM — why the 2026-08-01 F821 escape happened, in no doc today:** (1) `main` has **no branch protection** (`gh api …/branches/main/protection` → 404) — every "blocking" gate in these YAMLs is decorative; (2) commits pushed straight to main get **zero check-runs** (`15c8933f` had none); (3) `deploy.yml:56-58` lints with the *non-blocking full* `ruff check .`, **not** the `--select F821,…` correctness gate, so deploy went green over a broken tree; (4) PR #318's `pr-check` **failed on that exact F821 and merged anyway**. **Slice when specced (BO-17a "main-guard"):** add a `correctness` job to `deploy.yml` on push-to-main running the `--select` gate, deliberately NOT in the deploy job's `needs:` — loud, not blocking. AGENT-SAFE. **OWNER-GATE:** enabling branch protection / required checks, wiring any gate into `needs:`, removing `skip_tests`; BO-18's purge+rotation is WS-2's, not this row's. Refuted two long-standing beliefs: pr-check **does** cover the frontend, and it **does** run on non-main branches. |
-| WS-6 | **Observability wiring + attribution** (BO-5 + decision D1) | `observability_e2.md` §6.7 | 🟡 Docs | OTel exporter (absent from uv.lock), Langfuse wiring, durable cost table, and the (run, member, agent, instance) attribution stamp — prereq for WS-16. **Audited 2026-08-01 → NO-GO on the doc contract, not the work.** The cited "§6.7" is an unnumbered recommendation memo with **zero acceptance criteria**, stale pre-restructure paths (`apps/gateway/…`), no gate labels, and D1's attribution stamp **appears in no spec at all**. Verified drift: `agent_run.{prompt,completion,total}_tokens` are read by `/debug/runs` + `/observability/runs` but **never written** (`run_trace.py::_persist_row` is the only INSERT) — the spec claims they carry cost attribution; they are always NULL. Cost today lives only in Redis day-hashes (45-day TTL) plus opt-in `audit_event`; the durable precedent to generalise is `app_audit` (mig 114), already used for a monthly budget in `routes/apps/runtime.py`. Only 2 of D1's 4 fields reach the choke point (`v1_compat` reads `x-cc-agent`/`x-cc-source`; run_id + member are dropped) and `instance` exists nowhere — reuse `_resolve_agent_instance()` (`executor.py:917`), do not invent a second key. **To unblock:** add a numbered `## 7. WS-6 — open work` to `observability_e2.md` with per-item "done when" + AGENT-SAFE/OWNER-GATE labels, fix the 5 stale paths, correct the token-column claim. Then the dispatchable slice is the **D1 stamp + a durable `llm_call` table** (agent-safe); Langfuse/OTel/`LLM_USAGE_AUDIT` are OWNER-GATE (§6). |
+| WS-6 | **Observability wiring + attribution** (BO-5 + decision D1) | `observability_e2.md` **§7** | 🟡 partial | **Docs gate CLEARED** (PR #319 added the numbered §7 with nine lettered tickets WS-6a–i, per-item done-whens and gate labels). **Re-audited 2026-08-02 → GO-NARROWED to WS-6a+WS-6c only.** ✅ **BUILT 2026-08-02, pending review:** D1's attribution stamp exists as a substrate — `instance` joins `_RUN_CONTEXT_KEYS`/`bind_run_context`, resolved once in `run_agent_stream` via a **second additive bind** after `load_agent` (the early bind stays: it is what correlates a failure *during* load; moving it would trade 5 fields for 1), and `_emit_usage` carries the full (run, member, agent, instance) tuple with **zero call-site changes** — it arrives by inheritance via `activity._INHERIT`. Shared agents produce an **absent key, never `''`** (double-guarded + pinned). `refresh_run_presence()` patches `cc:activity:live:{run_id}` after the late bind, so `/observability/active` + `/roster` carry it; interim `by_instance` cost dimension added to the Redis rollup. **Nothing durable is written yet** — logs + Redis feed only. **🔴 WS-6b/6d/6e HELD, still NO-GO:** WS-6b's security amendment names *no workable mechanism* — `bind_run_context` has one call site (`executor.py`), contextvars do not cross the HTTP hop to `v1_compat`, and `agent_run` rows are written at the run *boundary* so a mid-run join finds nothing. **The only mechanism the code supports at request time is the presence key `cc:activity:live:{run_id}`**, which for the orchestrator path carries a server-established `user`; §7 must name it (or name another) before WS-6b dispatches. WS-6e has no token source (`build_run_trace_row` is pure over events+folded) so it sequences *after* WS-6b, not independently; WS-6d additionally waits on the retention/PII answer (Q3). **Two recorded asymmetries** — the `phase="start"` event predates the bind, and **a delegated sub-run inherits the caller's partition** while its blobs key to `''`, so WS-6d must not treat `instance` as a foreign key onto `agent_blob.instance`. **OWNER-GATE:** WS-6f/g/h/i (Langfuse keys, `--profile obs`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `LLM_USAGE_AUDIT`, the MAF telemetry kill switch) — all now listed in §6. |
 | WS-7 | **Memory activation + search** (BO-21 → BO-22) | checklist §C + `llm_caching_memory.md` | 🔴 | **OWNER-GATE:** flipping `MEM0_ENABLED`/`GRAPHITI_ENABLED` in prod (cost + latent findings in `agent_platform_hardening` Part 5). `acb_search` (BO-22) after. |
 
 ### Platform

apps/services/orchestrator/AGENTS.md

@@ -22,7 +22,8 @@ and streams chat responses as AG-UI events.
 6. All agents must go through MAF -- no raw Copilot SDK paths for business execution
 7. mutation_runner.py runs inside the Docker sandbox -- uses Copilot SDK directly (by design)
 8. workflow_tools.py exposes published Workflows-app workflows to every agent as a three-tool trio (`list_workflows` / `run_workflow` / `get_workflow_run`) — the sibling of app_tools.py, injected through the same `_tool_injection` gated pipeline. Calls go in-process to `gateway.routes.workflows.service` (the same entrypoints the Run button and API use), so concurrency caps, run history, and the approval gates inside a workflow bind agent-triggered runs identically; a run paused at a Human-approval node reports itself as waiting in the approvals inbox — an agent cannot bypass the gate. Spec: ai-company-brain/specs/workflows_app.md F13
-9. _tool_injection.py's static tool collection lives in `_collect_injectable_platform_tools()` (WS-23 S1 — the exact import chain `_inject_agent_tools` always ran, extracted verbatim; injection behavior unchanged). It is the read-only introspection seam the skills catalog (`acb_skills.skill_families` + gateway `GET /integrations/skills`) builds on, and `tests/unit/test_skills_registry.py` drift-fails if an injected tool is missing from the family registry — register any newly injected tool there in exactly one family. WS-23 S2: `_resolve_injected_scope(tool_scope, disabled_families=…)` intersects admin skill toggles (`agent_skill_setting`, loaded once per run by `_load_disabled_skill_families` — same best-effort sync-DB mechanism as app grants/MCP) with the declared scope; the core floor survives anything, the `workflows` toggle is honored at the trio's append site, and NO rows means byte-identical pre-S2 behavior (`tests/unit/test_skill_toggle_enforcement.py` pins all three rules).
+9. Run correlation is bound in **two** places in run_agent_stream, deliberately. `bind_run_context(run_id, thread_id, agent, user, source)` fires at the run boundary — BEFORE `load_agent`, so a failure during load is still correlated — and `_bind_run_instance(instance, run_id)` tops up `instance` (the tenant partition from `_resolve_agent_instance`) immediately after the load, because that value needs `loaded.config` and cannot exist earlier. Binds are additive; the single `clear_run_context()` in the `finally` unbinds every key. Do not "simplify" this into one bind: moving the first one late loses correlation on load failures, and dropping the second silently un-attributes every personal/team agent's spend. The same call patches the live presence key via `acb_common.refresh_run_presence` — the `phase="start"` event predates the load, so `/observability/active` and `/observability/roster` would otherwise never show a partition for any run. `_run_sub_agent_streaming` deliberately does **neither**: a delegated sub-run inherits the caller's `instance` (known asymmetry — spec §7 WS-6a) because `bind_context` cannot unbind and a correct fix needs save/restore around the sub-run. Spec: ai-company-brain/specs/observability_e2.md §7 (WS-6a)
+10. _tool_injection.py's static tool collection lives in `_collect_injectable_platform_tools()` (WS-23 S1 — the exact import chain `_inject_agent_tools` always ran, extracted verbatim; injection behavior unchanged). It is the read-only introspection seam the skills catalog (`acb_skills.skill_families` + gateway `GET /integrations/skills`) builds on, and `tests/unit/test_skills_registry.py` drift-fails if an injected tool is missing from the family registry — register any newly injected tool there in exactly one family. WS-23 S2: `_resolve_injected_scope(tool_scope, disabled_families=…)` intersects admin skill toggles (`agent_skill_setting`, loaded once per run by `_load_disabled_skill_families` — same best-effort sync-DB mechanism as app grants/MCP) with the declared scope; the core floor survives anything, the `workflows` toggle is honored at the trio's append site, and NO rows means byte-identical pre-S2 behavior (`tests/unit/test_skill_toggle_enforcement.py` pins all three rules).
 
 ## Work Guidance
 

apps/services/orchestrator/orchestrator/executor.py

@@ -937,6 +937,49 @@ def _resolve_agent_instance(
         return ""
 
 
+def _bind_run_instance(instance: str, run_id: str = "") -> None:
+    """Top up this run's correlation context with its tenant partition (WS-6a).
+
+    A SECOND, additive :func:`acb_common.bind_run_context` call rather than a
+    moved one — deliberately. The run boundary binds run_id/thread_id/agent/
+    user/source *before* ``load_agent`` so a failure DURING load is still
+    correlated, but :func:`_resolve_agent_instance` cannot run until the config
+    is loaded. Binding only ``instance`` here leaves every earlier field in
+    place and keeps the blast radius to one key.
+
+    ``''`` (the shared partition) binds nothing — bind_run_context only takes
+    non-empty values — so a shared agent's context has no ``instance`` key,
+    which is exactly how the store spells "shared" (migration 136). Never
+    raises: attribution must not be able to fail a run.
+
+    The live presence key is patched too, given *run_id*. It was written from
+    the ``phase="start"`` event body, published before the load and therefore
+    without an instance — so without this refresh ``active_runs()``,
+    ``/observability/active`` and the office roster could never show a
+    partition for ANY run. Patching the snapshot beats re-publishing a start
+    event, which every stream consumer would read as a second activation.
+
+    **Scope — this is the partition of the run that RESOLVED it.** A delegated
+    sub-run (``_run_sub_agent_streaming``) resolves none and cannot unbind the
+    caller's, so its events inherit the caller's ``instance`` while its blobs
+    are written under the shared partition. Known asymmetry, recorded in
+    ``specs/observability_e2.md`` §7 WS-6a; do not read the stamp as proof
+    that a given event's blobs live in that partition.
+    """
+    try:
+        from acb_common import bind_run_context
+        bind_run_context(instance=instance)
+    except Exception:
+        pass
+    if not instance or not run_id:
+        return
+    try:
+        from acb_common import refresh_run_presence
+        refresh_run_presence(run_id, instance=instance)
+    except Exception:
+        pass
+
+
 def _resolve_effective_agent_dir(
     agent_dir: Path,
     agent_config: dict[str, Any],
@@ -2153,7 +2196,10 @@ async def run_agent_stream(
     # Bind run_id/thread_id/agent/user into structlog contextvars so EVERY log
     # line this run emits (across all tiers + injected tools on this context)
     # carries them — the thing that makes "show me all logs for run X / agent Y"
-    # possible. Cleared in the finally below.
+    # possible. Cleared in the finally below. The fifth field of decision D1's
+    # stamp, `instance`, cannot be resolved yet (it needs loaded.config) and is
+    # topped up by _bind_run_instance right after the load — this bind stays
+    # here so a failure DURING the load is still correlated.
     _corr_source = "chat"
     _corr_user = ""
     try:
@@ -2322,6 +2368,11 @@ def _respond_input_apply(command: dict[str, Any]) -> bool:
             _agent_instance = _resolve_agent_instance(
                 loaded.config, agent_name, _corr_user,
             )
+            # Attribution (WS-6a): every log line and every model activation
+            # from here on carries the partition this run is executing in, and
+            # the presence snapshot published at start (which predates the
+            # load, so it has no instance) is patched to match.
+            _bind_run_instance(_agent_instance, run_id)
             _effective_ws = _resolve_effective_agent_dir(
                 loaded.agent_dir, loaded.config,
                 session_override=_session_ws,

packages/AGENTS.md

@@ -8,7 +8,7 @@ Reusable Python packages shared across all CommandCenter services.
 - acb_llm/ -- LiteLLM integration, unified credential store (LLM + integration keys), model routing
 - acb_memory/ -- Mem0 and Graphiti memory providers. `compartments.py` owns the scope vocabulary (`scope_key`/`scope_kind`: a bare email, `prefs:<email>`, `room:<thread_id>`, `agent:<name>`, `org:global`) and `resolve_clearance` — WHICH compartments a run may read and the ONE it may write (spec: docs/multiplayer/memory-clearance.md §3). Solo resolves to exactly the three scopes and the write target it always had; a shared run swaps the actor's private compartment for the room's and keeps their prefs. The point is that an excluded compartment's scope key is never passed to `search()` — a boundary, not a request in a system prompt. Dependency-free on purpose: the CALLER decides `shared` (the gateway has `resolve_room_access`), which keeps acb_memory below acb_auth and the gateway in the import graph. `session_cache` keys on the clearance fingerprint as well as the thread — without it a thread cached while solo keeps serving the owner's private block after it is shared.
 - acb_graph/ -- Postgres entity graph (SQLAlchemy sessions)
-- acb_common/ -- Shared settings, logging, activity/cost feed, utilities
+- acb_common/ -- Shared settings, logging, activity/cost feed, utilities. `_log.py` owns the run-correlation stamp: `_RUN_CONTEXT_KEYS` = `(run_id, thread_id, agent, user, source, instance)` — decision D1's attribution four-tuple plus the thread. Binds are **additive and non-empty-only**, so a caller may top up one field later (the executor resolves `instance` only after the agent config loads) and the shared partition `''` binds nothing rather than an empty value. `activity.py::_INHERIT` copies the same keys onto any event whose emitter omits them — which is why a model call inside a run is attributed with no change at its call site. Extend those two tuples together or attribution silently half-lands (`tests/unit/test_observability.py::test_inherit_and_run_context_keys_match` drift-fails if you don't). `instance` names the partition of the run that RESOLVED it — inheritance means a delegated sub-run carries its caller's key while writing blobs under the shared partition, so the stamp is not a join key onto `agent_blob` without knowing which run emitted it. The presence key (`cc:activity:live:{run_id}`) is written once from the `phase="start"` body, so a field resolved after start reaches `/observability/active` only via `activity.refresh_run_presence`. Spec: ai-company-brain/specs/observability_e2.md §7
 - acb_audit/ -- Audit event recording
 - acb_auth/ -- Authentication, roles, and org access control. Two guard styles coexist: the original coarse `require_role(UserRole.EXECUTIVE)` (unchanged) and `require_permission("feature:whatsapp")`, backed by DB roles + per-user allow/deny overrides (`permissions.py` is pure and testable; `access.py` does the I/O with a 60s cache). `require_authenticated(public=...)` is the app-wide default-deny guard (BO-2 #1) — authentication, not authorization. Two DISTINCT secrets: `GATEWAY_INTERNAL_TOKEN` is service identity (grants everything, never handed to agents) and `LITELLM_MASTER_KEY` is the /v1 API key agents hold, checked only by `require_llm_api_auth`. Spec: ai-company-brain/specs/org_access_control.md
 

packages/acb_common/acb_common/__init__.py

@@ -12,6 +12,7 @@
     publish_activity,
     read_activity_since,
     recent_activity,
+    refresh_run_presence,
 )
 from acb_common.settings import Settings, get_settings
 
@@ -28,4 +29,5 @@
     "publish_activity",
     "read_activity_since",
     "recent_activity",
+    "refresh_run_presence",
 ]