[sdk] Native support for LangChain/LangGraph, CrewAI, LlamaIndex and Pydantic AI, on a real identity layer by SiddarthAA · Pull Request #730 · FailproofAI/failproofai

SiddarthAA · 2026-08-19T14:44:49Z

What

The Python SDK was a capture surface you had to operate by hand: 15 keyword-only
emit methods, every one requiring session_id= and agent_id=, and nothing
propagating them. There was no ambient session anywhere in the package — no
decorator, no context manager, no contextvars. skill/SKILL.md stated that as a
deliberate contract, and skill/references/integration.md shipped a ~60-line
contextvars wrapper as markdown for customers to paste into their own
codebase: the SDK asking its users to write the missing half of the SDK.

So every customer on LangGraph, CrewAI or LlamaIndex hand-wrote the same
adapter, got thread propagation wrong, and abandoned it halfway — the exact
failure that reference doc opens by describing.

Now, with the call site unchanged:

import failproofai_sdk

failproofai_sdk.configure(environment="prod")
failproofai_sdk.instrument()          # auto-detects frameworks already imported

graph.invoke({"messages": [...]})     # unchanged

Ported from FailproofAI/agenteye#503,
reconciled against the eleven SDK bugs fixed on feat/fp-cli since that branch
was cut. Ported rather than copied: #503's _writer.py is 80 lines against
our ~450, so a straight copy would have silently reverted every one of them.

Why this is not the same as emitting events by hand

Measured, not asserted. The same task, same model, same tool, instrumented two
ways:

	manual (~8 lines of event calls)	`instrument()` (1 line)
events	4	14
event types	4	8
tool events	0	2
durations	0	6
token counts	0	2

The manual version reported one model_request/model_response pair. The
agent made two LLM calls. It recorded zero tool events for a run whose
entire point was calling a tool.

That is not carelessness — it is the ceiling of the approach. graph.invoke()
is one call seen from outside; the ReAct loop, the tool dispatch, the second
round-trip and the per-node timings all happen inside it. You cannot instrument
what you cannot see, and you can only emit what you remember to emit.

What is here

Ambient identity (_context.py, _scopes.py, _runtime.py) — session(),
agent() and tool_call() under both with and async with, current(), and
propagate() for thread hand-off. session_id/agent_id became optional on
all 15 methods, falling back to context, without breaking a single existing
call — which is why the golden wire-format bytes are untouched.

Four adapters — LangChain/LangGraph, CrewAI, LlamaIndex, Pydantic AI. Each
is a translation table over one shared RunTracker, emitting only the existing
15 event types, so nothing fans out to the server, collector, CLI or the stored
schema.

AutoGen is deliberately absent: autogen-core 0.7.5 last shipped 2025-09-30
with no commits since, and Microsoft's forward path is a separate package. The
live product is AG2, a different distribution whose middleware has no global
auto-instrument hook.

request_id on the two model events. The dashboard pairs model events on
it, but no SDK method accepted one and no doc mentioned it — so every
integration written to our own documentation emitted unpairable model events,
including demo-agent/mock_agent.py.

Three decisions where #503 was right and the current branch was wrong

Pairing keys are session-scoped, never agent-scoped. feat/fp-cli had
just tightened _pending keys to kind:session:agent:id to stop two
concurrent sessions colliding. That over-tightened: once a framework runs
tools inside sub-agents, a tool_use opened under planner and closed under
worker is the ORDINARY case, and an agent-scoped key makes those pairs miss
entirely — silently dropping duration_ms for exactly the nested runs that
most need it. The rule that survives both: include what makes the id unique
(kind, session), exclude what can legitimately change between the two events
(the agent).
TypeError, not ValueError, for missing or mistyped identity. It is
what a caller got before identity became optional, so code catching one keeps
working.
Field validation runs before identity resolution. A reserved **field is
a fault in the call itself and reads the same from anywhere; the identity
error depends on where the call was made from.

One decision where the current branch was right and #503 was not

#503 fixes the _track_pending KeyError — the same race found independently
here — with a threading.Lock. That lock is kept out: a lock held at the
instant of a fork() is inherited locked by a thread that does not exist in the
child, which is the exact hazard _writer rebuilds its Event and lock to
avoid. The tolerant, lock-free eviction on this branch fixes the same crash with
no fork edge.

Zero dependencies survives, and the test got stronger

The adapters import the frameworks they adapt — there is no other way to
subclass a callback base class. But integrations/__init__ resolves them by
string through importlib.import_module at call time, so:

the source scan is scoped to core modules with a per-file allowlist, and a new
file under integrations/ is scanned like core code until it is named there;
the promise is now asserted at runtime: a fresh interpreter imports the
package and must have no framework in sys.modules;
[project.dependencies] is still empty, and CI still installs the built wheel
with --no-deps.

An eager adapter import is not a style problem — it makes import failproofai_sdk raise ImportError on every machine without that framework.
Verified by planting one.

Framework extras carry upper bounds, because without one a clean build a year
from now pulls the next major, the callback API shifts, and the adapter stops
receiving events while raising nothing. There is deliberately no [all].

Verification

Not "the tests pass" — the couplings here fail silently, so each layer was
checked against something that could disagree.

Layer	Result
SDK suite	644 pass without frameworks, 855 with them (was 429)
Adapter suites, real frameworks	langchain 50, crewai 49, llama_index 44, pydantic_ai 68
Every commit, independently	426 → 503 → 503 → 617 → 638 → 640, by clean extraction
Python matrix	3.10 / 3.11 / 3.12 / 3.13 / 3.14, contract tests enforced
Live end-to-end	all four frameworks → daemon → dashboard `/v1/events` → events store

The live run used a real model. Captured and confirmed in the store:

lab-langgraph    44 events   5 agents   14 types
lab-llamaindex   28 events   2 agents    8 types
lab-pydantic-ai  16 events   4 agents    6 types
lab-crewai       14 events   3 agents    6 types

All 15 event types were confirmed end to end, with promoted columns
populated (tool_name, model, duration_ms, tokens), the framework field
on every event, and parent_id nesting intact. 18 batches uploaded, 0
failed.

Beyond the happy path, per framework: a tool that raises is recorded on
tool_result (4/4), the async path is captured (3/3 where async applies), and
concurrent sessions across threads stay isolated.

The four quickstarts in examples/ were executed against a real model
before being committed; parsing is not evidence that an example works.

Not checked

Streaming (.astream), CrewAI flows, LlamaIndex workflows beyond
FunctionAgent, provider retry/rate-limit paths, long-run memory behaviour, and
uninstrument() round-trips under load.

For the reviewer

FailproofAI is now public API that users type — capabilities=[FailproofAI()],
FailproofAITracer, FailproofAICrewListener. Renamed from #503's AgentEye*.
Worth an explicit yes on the naming before it ships.

AGENTEYE_HOME, AGENTEYE_ENVIRONMENT and ~/.agenteye are untouched — they
are a contract with two separately-released daemons. AGENTEYE_STRICT was new
in #503 and nothing else reads it, so it became FAILPROOFAI_SDK_STRICT.

#503 also carried two dashboard fixes (executionGraph.ts, sessionSummary.ts).
Those live in the AgentEye repo and are not in this PR; they need a
companion change there.

Hermes review

Field	Value
Status	Changes requested
Reviewed commit	`28044ff26c7a0a5cee5e93ae0a17d283cb9c5d8a`
Policy revision	`1d8f31d926828f3bae215c58f5b35baa44acbff0`
Model	`gpt-5.6-terra`
Duration	513s
Updated	2026-08-20T10:11:28.270810692+00:00

Summary

Two correctness issues remain in the new SDK identity and Pydantic AI integration paths; the explicit Pydantic API is completely inert without auto-instrumentation.

Changes

Adds ambient SDK identity scopes and optional event identities.
Adds auto-instrumentation adapters for LangChain, CrewAI, LlamaIndex, and Pydantic AI.
Extends collector delivery health reporting and enables delivery when only SDK batches are configured.
Adds integration documentation, examples, tests, and stacked-branch CI coverage.

Validation

Passed docker run --rm -v /review/input/workspace:/work -w /work/sdk/python python:3.12-slim sh -lc 'python -m pip install -q -e ".[dev]" && pytest -q' — SDK test suite completed in an isolated Python 3.12 container. (25s)
Passed docker run --rm -v /review/input/workspace:/work -w /work/sdk/python python:3.12-slim sh -lc 'python -m pip install -q -e ".[dev,pydantic-ai]" && pytest -q tests/integrations/test_pydantic_ai.py' — Pydantic AI adapter suite completed in an isolated Python 3.12 container; it does not cover standalone explicit capability use. (25s)

Findings

High/High Explicit Pydantic AI capabilities are inert without instrument() — The documented standalone API constructs capabilities=[FailproofAI()] at pydantic_ai.py:559, but wrap_run returns directly to the framework whenever module-global _enabled is false (:605-606). _enabled is initialized false and is set only by _Adapter.install() (:913-926), which is reached through instrument(). The model and tool wrappers have the same gate. The existing explicit-capability test uses the instrumented fixture, so it exercises only the enabled state. (sdk/python/failproofai_sdk/integrations/pydantic_ai.py:605)

1 advisory finding

Medium/High Falsy supplied session IDs are silently replaced — session._enter() chooses self._requested or an inherited/generated ID (_scopes.py:97), and agent._enter() does the same (:197). Thus a supplied empty string or 0 is not validated as invalid identity; it is replaced by an unrelated generated session, or by an enclosing session when one exists. A containerized reproduction showed session("") and agent(session_id=0) both entering with generated UUIDs. (sdk/python/failproofai_sdk/_scopes.py:97)

Open questions

None.

Policy overrides

None.

`_pending` correlates a start event with its end so `duration_ms` can be measured. The key it uses has been wrong twice, in opposite directions. Bare ids were the first mistake: tool pairs keyed on `tool_call_id` and hook pairs on `hook_id` shared one flat keyspace, so a caller whose tool call and hook happened to share an id — not exotic, both are frequently the harness's own step id — got a `hook_completed` that consumed the `tool_use` timestamp and then a `tool_result` with no duration at all. Adding the session fixed a second, real collision: `_pending` lives on one process-wide namespace, so two concurrent sessions collided on any shared step id. Starting `step-1` in session A and then in B overwrote A's timestamp; A's result reported B's interval and B's reported none. Adding the AGENT as well was over-tightening, and this commit removes it. Once a framework runs tools inside sub-agents — LangGraph and CrewAI both do — a `tool_use` opened under `planner` and closed under `worker` is the ORDINARY case, and an agent-scoped key makes those pairs miss entirely, silently dropping `duration_ms` for exactly the nested runs that most need it. The rule that survives both: include what makes the id unique (kind, session), exclude what can legitimately change between the two events (the agent). A session cannot change under a pair; an agent can. Applied to all four pair types, since a `human_wait` answered by a supervisor and an `agent_pause` resumed by another agent are the same shape. These are correlation keys only — never emitted, never leaving the process — so no wire format changes. Only `duration_ms` changes, in the colliding cases, from a fabricated or missing value to a correct one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every event method took `session_id` and `agent_id` as required keyword arguments and nothing propagated them, so instrumenting a real agent meant threading two ids through every function that might emit. That is the diff nobody wants to review, and it is why `skill/references/integration.md` shipped a ~60-line contextvars wrapper AS MARKDOWN for customers to paste into their own codebase — the SDK asking users to write the missing half of the SDK. Three scopes bind identity on contextvars instead: with failproofai_sdk.session() as sid: with failproofai_sdk.agent("planner", goal=q): # agent_start/end with failproofai_sdk.tool_call("search") as t: # tool_use/result t.output = search(q) All three work under `with` and `async with` — an agent framework is half-async, and `@contextmanager` supports only the former, so these are plain classes whose async pair delegates to the sync pair. No scope awaits anything (`submit()` is a deque append), so the delegation is not a lie. `session_id`/`agent_id` are now OPTIONAL on all 15 methods, resolved from the scope when omitted. Existing call sites are untouched and still pass ids explicitly, which is why the golden wire-format bytes are unchanged. Details that are load-bearing rather than incidental: * The agent stack is a TUPLE. A `ContextVar[list]` is shared by reference across tasks and threads, so `.append()` in one mutates what every other sees — the cross-run mixing contextvars exist to prevent, wearing a contextvars costume. It passes every single-threaded test. * `propagate(fn)` snapshots VALUES rather than using `copy_context().run`. A `Context` cannot be entered twice, so the copy_context form crashes the caller's worker on any reuse — `pool.map`, a retried submit — and mutations inside `ctx.run` persist, leaking one call's agent stack into the next. * `agent()` emits `error` strictly BEFORE `agent_end`, because the dashboard closes the span at `agent_end` and anything after it is attributed to nothing. A cancellation closes as `cancelled`, not `failed` — a cancelled run is not an error, and marking it one pollutes the Errors surface. * Identity is validated AFTER resolution, never before. Validating first would reject every ambient call; resolving without validating would restore the silent skip, since ingest drops an event whose `session_id` is not a JSON string and answers `200 OK` with `{"accepted":0,"skipped":1}`. * Unresolvable identity raises TypeError, not ValueError — the wrong type, or a missing required argument, which is exactly what a caller got before this change. Code catching TypeError keeps working. * Field validation runs before identity resolution: a reserved `**field` is a fault in the call itself and reads the same from anywhere, so it gives a stable message; the identity error depends on where the call was made from. `conftest.py` gains the suite-wide isolation this makes necessary: a per-test spool, restored process globals, and an assertion that a test leaking a scope FAILS rather than quietly misattributing every event after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The dashboard pairs a model request with its response on `request_id`, but no SDK method accepted one and no doc mentioned it. So every integration written to our own documentation emitted model events that cannot be paired — including `demo-agent/mock_agent.py`, our own reference implementation. Optional on both methods, and appended LAST in the ordered field list, so an event that omits it serialises byte-for-byte as before. That matters twice over: `test_wire_format.py` freezes those bytes, and ingest's dedup key hashes the canonical payload, so a reordering would stop retried batches collapsing and surface as duplicate rows rather than as an error. Only the two model events carry it. The other thirteen have nothing to pair with, and a field most event types cannot use is a field people fill in wrongly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…x, Pydantic AI failproofai_sdk.configure(environment="prod") failproofai_sdk.instrument() # auto-detects what is already imported graph.invoke({"messages": [...]}) # unchanged Each adapter is a translation table over one shared `RunTracker`, emitting only the existing 15 event types — nothing fans out to the server, collector, CLI or the stored schema. WHY THIS IS NOT THE SAME AS EMITTING BY HAND. Measured on one task, same model, same tool: hand-written instrumentation produced 4 events and 4 types; the adapter produced 14 events and 8 types. The manual version reported ONE model_request/model_response pair for a run that made TWO LLM calls, and zero tool events for a run whose entire point was calling a tool. That is not carelessness, it is the ceiling: `graph.invoke()` is one call from outside, and the ReAct loop, the tool dispatch, the second round-trip and the per-node timings all happen inside it. You cannot instrument what you cannot see. AutoGen is deliberately absent. `autogen-core` 0.7.5 last shipped 2025-09-30 with no commits since, and Microsoft's forward path is a separate package; the live product is AG2, a different distribution whose middleware has no global auto-instrument hook. ZERO DEPENDENCIES SURVIVES THIS, and the test got stronger rather than weaker. The adapters import the frameworks they adapt — there is no other way to subclass a callback base class — but `integrations/__init__` resolves them by STRING through `importlib.import_module` at call time. So the source scan is now scoped to core modules with a per-file allowlist, and the promise is asserted at runtime instead: a fresh interpreter imports the package and must have no framework in `sys.modules`. An eager import is not a style problem, it makes `import failproofai_sdk` raise ImportError on every machine without that framework — verified by planting one. Framework extras carry upper bounds. Without one, a clean build a year from now pulls the next major, the callback API shifts, and the adapter stops receiving events while raising nothing — an empty dashboard, not a traceback. There is deliberately no `[all]`: an extra installing four agent frameworks at once is a resolver problem handed to somebody who wanted a telemetry library. Verified against the real frameworks, not mocks: 211 adapter tests (langchain 50, crewai 49, llama_index 44, pydantic_ai 68), plus live runs of all four against a real model, each reaching the daemon and the events store. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`examples/*_quickstart.py` — about 30 lines each, the thing somebody runs in their first five minutes. All four were executed against a real model before being committed; parsing is not evidence that an example works. `tests/test_examples.py` guards them, because nothing else can: they need a framework and an API key, so they cannot run in unit CI, which is exactly why they rot. It checks they parse, that they call API this package actually exports, that each imports only the framework its own extra installs, that the extra they name exists — and that they demonstrate the ergonomics they exist to demonstrate. An example that threads `session_id=` by hand teaches the manual path the scopes were built to remove, so that fails the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`skill/SKILL.md` §3 opened with "There is no ambient session. No decorator, no context manager, no contextvar, no `set_session()`." An agent reads that as the contract and writes against it, so leaving it would have been worse than not documenting the scopes at all — a skill is instructions somebody executes. - `skill/references/frameworks.md` — new. Per-framework mapping tables, what every adapter guarantees, how to mix adapters with hand-written events, how to verify one, and what to do for a framework not on the list. - `README.md` — the frameworks and scopes sections, ahead of the manual event reference, because that is now the order people meet them in. - `skill/SKILL.md` — §3 rewritten to describe the scopes, and pointed at the new reference rather than at the wrapper customers used to paste in by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-19T14:44:58Z

Thanks @SiddarthAA for your contribution to Failproof AI! 🙌

We'd love to discuss your PR and welcome you to our community: https://discord.befailproof.ai/

coderabbitai · 2026-08-19T14:45:04Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8d1498a0-c424-48ed-a907-a8590d0ae068

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Every PR carries an entry, and this one touched nothing outside sdk/python until now — which is exactly how a release note goes missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hermes-exosphere · 2026-08-19T14:47:08Z

Hermes


Status	Reviewed
Verdict	Changes requested
Head	`28044ff26c7a`
Rounds	3 of 5

Two correctness issues remain in the new SDK identity and Pydantic AI integration paths; the explicit Pydantic API is completely inert without auto-instrumentation.

What this changes

flowchart LR
    n0AmbientidentityAPI["+ Ambient identity API"]
    n1Eventemission["~ Event emission"]
    n2Integrationregistry["+ Integration registry"]
    n3PydanticAIadapter["+ Pydantic AI adapter"]
    n4Otherframeworkadapters["+ Other framework adapters"]
    n5Collectordelivery["~ Collector delivery"]
    n6Daemoncollectorlifecycle["~ Daemon collector lifecycle"]
    n7CIandSDKdocumentation["~ CI and SDK documentation"]
    n0AmbientidentityAPI -- "supplies omitted identities" --> n1Eventemission
    n2Integrationregistry -- "loads and enables" --> n3PydanticAIadapter
    n2Integrationregistry -- "loads and enables" --> n4Otherframeworkadapters
    n3PydanticAIadapter -- "emits run telemetry" --> n1Eventemission
    n4Otherframeworkadapters -- "emits framework telemetry" --> n1Eventemission
    n1Eventemission -- "writes SDK spool batches" --> n5Collectordelivery
    n6Daemoncollectorlifecycle -- "starts delivery tasks" --> n5Collectordelivery
    n5Collectordelivery -- "documents delivery behavior" --> n7CIandSDKdocumentation

Rounds

Round	Reviewed	Commits in this round	Verdict
1	`6047ade3dae5`	`ae0b61de0c94` `79abe63ae53c` `56473dce2101` `8e884637c7b9` `0bf8670f25b3` `6da96c33aca0` `69aa34e88bcf` `6047ade3dae5`	Changes requested — F1
2	`6231cf447f98`	`c4f813540a03` `5c86102e17e3` `acd6d14d2597` `23f8cf3787af` `ecbbce005ab6` `0bdb73d26b66` `6231cf447f98`	Changes requested — F1
3	`28044ff26c7a`	`2367fc25049a` `a312b71d3e7f` `6aad9339227d` `2e2ab3c8dfd2` `9d75568f8932` `a087ff69d5c9` `a1dd9a745130` `a518e0612433` `a4eb8c382c27` `bf42a6565e8a` `4d3dd1d184ed` `bc67a1bce1f4` `1a1be733cbb4` `7b6013afe2b6` `1147ead5e189` `dab16eed4854` `e7ec9285f5bc` `8b60238e0f15` `7870fb34f897` `61fa5eea646c` `66274c9421de` `c6d035c74c73` `28044ff26c7a`	Changes requested — F1

Findings

Open

F1 Explicit Pydantic AI capabilities are inert without instrument() (sdk/python/failproofai_sdk/integrations/pydantic_ai.py) — round 1
F2 Falsy supplied session IDs are silently replaced (sdk/python/failproofai_sdk/_scopes.py) — round 1

@hermes-exosphere help lists every command. This comment is maintained in place — I rewrite it after each review rather than posting a new one.

…ey replaced Three gaps found by diffing this branch against the upstream PR it was ported from (FailproofAI/agenteye#503), of which the first would have failed CI. **uv.lock was stale.** `pyproject` grew `pytest-asyncio` and five framework extras; the lockfile had none of them, so `uv sync --locked --extra dev` — the exact command the `failproofai-sdk` CI job runs — failed with "the lockfile needs to be updated". Regenerated: +5941/-94, which is most of the line-count difference between the two PRs and an omission rather than a saving. All five framework extras resolve, and `uv sync --locked --extra pydantic-ai` installs. **`integration.md` still shipped the wrapper.** Its "## The wrapper" section was ~60 lines of contextvars scaffolding for customers to paste into their own codebase — the thing `session()`/`agent()`/`tool_call()` now are. Worse than redundant: it taught `contextvars.copy_context().run` for thread hand-off, which `_context.propagate` documents as broken, because a `Context` cannot be entered by two threads at once and so the copy-context form crashes the caller's worker on any reuse — `pool.map`, a retried submit. That is now a "do not reach for this" warning next to `propagate()`. **`events.md`** picks up the scopes and the four `human_*` events alongside them. Both ported files re-introduced a bug this branch had already fixed: their lifecycle brackets catch `Exception`, and `asyncio.CancelledError` inherits from `BaseException`, so a cancelled tool emits `tool_use` with no `tool_result` and a cancelled run gets no `agent_end` at all. `tests/test_skill_snippets.py` caught both on the way in, which is the whole reason it parses every fenced block rather than trusting review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hermes-exosphere

Hermes found blocking issues that should be addressed.

Review coverage was incomplete, but the concrete blocking findings below are sufficient to request changes.

High: Explicit Pydantic AI capabilities are inert unless auto-instrumentation is also enabled

Rule: COR-001
Location: sdk/python/failproofai_sdk/integrations/pydantic_ai.py:394
Evidence: The adapter documents capabilities=[FailproofAI()] as a standalone usage path (lines 345-353), but all three capability wrappers immediately return the underlying handler while module-global _enabled is false (for example lines 394-396). _enabled starts false and is only set true by _Adapter.install() (lines 671-684). Thus a user following the explicit-capability API without first calling instrument("pydantic_ai") gets no telemetry at all. The explicit-capability test runs under the instrumented fixture, so it cannot expose this path.
Required change: Make explicit FailproofAI() instances active independently of registry installation; keep any uninstall disablement scoped to auto-injected instances. Add a test that constructs Agent(..., capabilities=[FailproofAI()]) without calling instrument() and asserts emitted events.

1 advisory finding

Medium/High Falsy invalid session IDs are converted into unrelated generated sessions — session._enter() selects the requested ID with self._requested or ... (line 97), and agent._enter() does the same (line 197). Consequently session("") and agent(..., session_id=0) do not raise identity validation errors; they generate UUID sessions instead. A containerized reproduction printed generated UUIDs for both cases. This silently splits telemetry from the caller's intended session rather than preserving the event API's invalid-identity failure behavior. (sdk/python/failproofai_sdk/_scopes.py:97)

hermes-exosphere · 2026-08-19T14:57:37Z

+
+    # -- run --------------------------------------------------------------
+
+    async def wrap_run(self, ctx, *, handler):


Hermes — High/High (COR-001): Explicit Pydantic AI capabilities are inert unless auto-instrumentation is also enabled

The adapter documents capabilities=[FailproofAI()] as a standalone usage path (lines 345-353), but all three capability wrappers immediately return the underlying handler while module-global _enabled is false (for example lines 394-396). _enabled starts false and is only set true by _Adapter.install() (lines 671-684). Thus a user following the explicit-capability API without first calling instrument("pydantic_ai") gets no telemetry at all. The explicit-capability test runs under the instrumented fixture, so it cannot expose this path.

Required change: Make explicit FailproofAI() instances active independently of registry installation; keep any uninstall disablement scoped to auto-injected instances. Add a test that constructs Agent(..., capabilities=[FailproofAI()]) without calling instrument() and asserts emitted events.

hermes-exosphere · 2026-08-19T14:57:38Z

+        self._agent_token: "contextvars.Token | None" = None
+
+    def _enter(self) -> str:
+        sid = self._requested or _context.session_id() or uuid.uuid4().hex


Hermes — Medium/High (COR-001): Falsy invalid session IDs are converted into unrelated generated sessions

session._enter() selects the requested ID with self._requested or ... (line 97), and agent._enter() does the same (line 197). Consequently session("") and agent(..., session_id=0) do not raise identity validation errors; they generate UUID sessions instead. A containerized reproduction printed generated UUIDs for both cases. This silently splits telemetry from the caller's intended session rather than preserving the event API's invalid-identity failure behavior.

Required change: Distinguish omission from a supplied value with is None, then validate supplied session IDs before binding them. Cover empty strings and falsy non-string values with unit tests.

A LangChain run with no parent is the session's root, and the adapter turned every root into an `agent_start`/`agent_end` pair — including a root whose own `run_type` is `chat_model`, which is exactly what a direct `ChatOpenAI(...).invoke(...)` outside any graph produces. So that call emitted an agent span and NOTHING ELSE: no `model_request`, no `model_response`, and therefore no model name, no input or output tokens and no latency, while the trace still looked populated and nothing raised. It is not an edge case — a classifier, a summariser and a one-shot rewrite are all shaped like this, and a supervisor that delegates to graphs and then writes its own summary hits it on the summary. That is where this was found. `_start_root` now also dispatches the leaf starter for a leaf-typed root, and `_on_end` closes the leaf before the agent — the dashboard closes the span at `agent_end`, so a `model_response` emitted after it is attributed to nothing. Purely additive: a chain-typed root is untouched. Five tests, four of which fail when the fix is reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

…d it `crewai.flow.runtime` emits `HumanFeedbackRequestedEvent` before it blocks on a person and `HumanFeedbackReceivedEvent` after the answer. The adapter subscribed to neither, so the entire wait was an unexplained gap in the trace and the session's active duration absorbed it. LangChain and LlamaIndex both map their HITL surface onto the same four events; crewai now does too — `human_wait` + `agent_pause`, then `agent_resume` + `human_input`, in that order, because only the first pair carries the prompt and the answer and only the second feeds paused time. Two things surfaced while fixing it, each of which would have left the fix silently inert: * The adapter resolved event classes against `crewai.events.event_types` alone, and the flow events are not in it — they are lazily re-exported from `crewai.events`. The lookup returned None, the capability probe disabled that one hook, and nothing failed. `event_class()` now tries both namespaces, and the anti-drift test resolves through it rather than through a namespace of its own: asserting against the narrower one is what let the gap exist. * crewai sets NO correlation id on either event — `request_id` is None on both and `started_event_id` is None on the received one — so pairing on it raised a TypeError inside the customer's event bus. The join is now `request_id` (which the enterprise async provider does set, and which can interleave), then `(flow_name, method_name)`, then the most recently opened pause, which is sound only because a console prompt blocks. Feedback for a request we never saw records the answer but deliberately withholds `agent_resume`: closing a pause that never opened subtracts a pausedMs interval that was never added. Six tests, all six failing when reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

The docstring said an integration naming its counters something unusual would show blank token columns, which reads as an exotic case. The common case is worse and was undocumented: `FunctionAgent` — the agent API LlamaIndex documents — calls `astream_chat`, and `llama-index-llms-openai` does not send `stream_options={"include_usage": True}`, so the provider never emits the usage chunk and `LLMChatEndEvent.response.raw` has no `usage` key to find. Verified by spying on the dispatcher directly against llama-index-core 0.14.23: every `LLMChatEndEvent` in a `FunctionAgent` run arrives with usage absent, so every token count on the default agent path is null and no instrumentation can recover a number the framework never received. The one-argument user-side fix is now stated in the docstring. Measured on the same run: `(None, None)` becomes `(148, 17)`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

Documentation and examples were two directories kept in agreement by hand, and the docs half was MDX — which renders as raw JSX tags anywhere except a Mintlify build, so on disk and on GitHub it read as broken markup. Both are now one Markdown tree, a directory per framework holding the guide somebody reads and the `examples/` they run: docs/<framework>/README.md docs/<framework>/examples/*.py Five directories — langgraph, crewai, llama_index, pydantic_ai, and manual for an agent with no framework, which also carries the three-seam recipe for any unsupported one and states why AutoGen has no adapter. Every guide follows one shape: install and supported range, the three-line integration, how the adapter attaches, a full framework-concept-to-event mapping, a complete copy-pasteable program, span naming, session resolution, every `instrument()` option, a real captured event payload, and pitfalls written as symptom then cause then fix. The pitfalls are the ones that actually cost time here: construct Pydantic AI agents AFTER `instrument()` or they carry no capability and record nothing, with no error; `create_react_agent` aborts the graph on a raising tool unless the tool node sets `handle_tool_errors`; LlamaIndex needs one `stream_options` argument or every token count is null; and never read the spool to verify anything, because a running `failproofaid` deletes each batch within milliseconds and the read races it. Eleven example scripts, every one executed against a live model before shipping, including a supervisor delegating to two workers (38 events, 5 agents) and a bare OpenAI tool-calling loop instrumented by hand (14 events, no framework). Each ends by printing the event stream it produced, captured by tapping the writer in-process rather than reading the spool, for the reason above. `test_examples.py` becomes `test_docs.py` and walks the whole tree: a framework with an adapter and no directory fails, a guide linking to an example that does not exist fails, a guide or example naming SDK API that does not exist fails, and an example threading `session_id=` by hand fails — checked over the AST, so the manual guide can still explain the argument its whole purpose is to replace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

The site had one page for the Python SDK, titled "Custom agents", documenting the pre-open-source API: `import failproofai` rather than `failproofai_sdk`, a private wheel install, `session_id`/`agent_id` required on every call, no `instrument()` and no adapters. It also carried a claim that is no longer true — that tool and hook ids share one process-wide pending map and must be globally unique across both namespaces. Keys are `tool:{session}:{id}` and `hook:{session}:{id}` now, scoped by kind and session. Adds a Frameworks section under Start here: an index that leads with all five integrations, then a page each for LangChain/LangGraph, CrewAI, LlamaIndex, Pydantic AI, and custom agents, plus a How it works page covering the data model, who mints which id, when a session ends, and how events reach Cloud — the questions no per-framework page can answer. The four framework pages share one section order, so a reader who learns one can skim the next: Install, Instrument, What gets recorded, Example, Name your spans, Control the session, Options, Human in the loop, Common problems, Next. Two diagrams, both the same orientation: the pair structure on the index and the delivery pipeline on How it works. Everything else is tables — a decision tree forced into a flowchart sprawls, and a pipeline table can carry a "runs in" column a diagram cannot. The old page is retitled "Python SDK reference", keeps the reference material that belongs in a reference tab, fixes the import name and install, and points at the new guides. `reference/overview` gains a card so the Integrations tab keeps an entry point. Every code sample on these pages was extracted and run against a live model before shipping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

The published site is a second, hand-maintained copy of claims about this package, and nothing checked it. It had already drifted: it named `capture_content` for CrewAI and `session_id` for LlamaIndex, neither of which those adapters read, and told readers to verify a Pydantic AI install by printing `agent.capabilities`, which raises AttributeError — Pydantic AI merges the list into one `root_capability`. None of that produced an error for a reader. `instrument()` passes one dict to every adapter and drops unknown keys by design, so a wrong option is silently ignored: no error, no effect. Only a test catches it. Parses each adapter's own source for the options it really reads, compares them against every documented `instrument()` call, pins the Pydantic verification snippet to `root_capability`, checks every SDK name the pages mention, and asserts the four framework pages share one section order and are each explicit about human-in-the-loop rather than silent. Same shape as `test_spool_contract.py` and the CLI's `test_fp_home_contract.py`: read the other side's source, skip when it is genuinely absent (an installed sdist has no docs site), and fail when `FAILPROOFAI_SDK_REQUIRE_CONTRACT` says the repository should be there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

hermes-exosphere

Hermes found blocking issues that should be addressed.

Review coverage was incomplete, but the concrete blocking findings below are sufficient to request changes.

High: Explicit Pydantic AI capabilities emit no telemetry unless instrument() is also called

Rule: COR-001
Location: sdk/python/failproofai_sdk/integrations/pydantic_ai.py:395
Evidence: FailproofAI() is documented as a standalone capabilities=[FailproofAI()] integration, but all three wrappers return the underlying handler while module-global _enabled is false (for example wrap_run at line 395). _enabled starts false and is only set by _Adapter.install() at line 684. The explicit-capability test at line 647 receives the instrumented fixture, so it only tests the auto-instrumented state.
Required change: Make explicit FailproofAI() instances active independently of registry installation, while limiting uninstall disablement to auto-injected instances. Add a test that constructs an agent with capabilities=[FailproofAI()] without calling instrument() and asserts emitted events.

1 advisory finding

Medium/High Falsy supplied session IDs are replaced with generated or inherited IDs — session._enter() and agent._enter() select the requested ID with self._requested or ... (lines 97 and 197). Therefore session("") and agent("a", session_id=0) do not reach identity validation; a nested scope can inherit another session and an unbound scope generates a UUID. A nested-container reproduction printed generated UUIDs for both inputs. (sdk/python/failproofai_sdk/_scopes.py:97)

`CollectorConfig::is_enabled()` gated the whole collector on `sessions || hooks`, and `collector_tasks()` returns early when it is false. On a machine with a credential and both capture sources off, the daemon therefore started no spool watcher and no sweeper, logged nothing, and every batch `failproofai-sdk` wrote into `custom-agents/events/` sat on disk forever — no error on either side, and an unread spool is indistinguishable from an idle one. Those two settings gate the daemon's own capture sources, and each is checked again where its source is registered, so leaving them off still starts neither. What they must not gate is delivery: the spool also carries events the user's own instrumented agents produced. `is_enabled()` is now `ingest.is_some()`. An unconfigured machine still starts no thread and no runtime. Verified live against a daemon on an isolated FAILPROOFAI_HOME with `{"sessions":false,"hooks":false}`: before, silence; after, `collector started tasks=3` and a pre-existing batch delivered by the sweeper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`instrument()` with no argument instruments every framework already in `sys.modules`. Called above the `import langchain` line — the natural place for a setup call — it finds nothing, installs nothing, returns `()` and raises nothing. The process then runs with the SDK imported, the adapter apparently installed, and zero events emitted. The message naming the exact fix already existed, at `logger.debug`, which no default logging config shows. So the one mistake that costs a user all of their telemetry was the one mistake we said nothing about. Now `logger.warning`, and only on the path where somebody explicitly asked for instrumentation and got none. The regression test empties the registry for its duration rather than trusting that no earlier test imported a real framework: tests/integrations/ runs first and imports all four, which would otherwise make this test install them for real and leak `_ACTIVE` into every test after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

SKILL.md's durability section opened by calling SIGTERM "not exotic — it is every rolling deploy", every `docker stop`, every Kubernetes eviction, and then told the reader "Python's default handler exits, so `atexit` *does* run". CPython installs no handler for SIGTERM. `signal.getsignal(SIGTERM)` is `SIG_DFL`, the OS terminates the process where it stands, and the atexit flush never runs. Measured: a child that queues 20 events and sends itself SIGTERM writes zero of them. The readers most likely to act on that paragraph are the ones deploying into a container, i.e. exactly the population it reassured wrongly. The text now states the real behaviour and ships the handler that fixes it — flush_now() then sys.exit(128 + signum), which unwinds so an open agent() scope still emits its agent_end before the flush. Two subprocess tests execute both halves. The bare case asserts events are still lost, so if the SDK ever installs its own handler the recipe is flagged as obsolete instead of quietly standing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Ingest splits `environment` on commas to build its filter facets, so it skips any line containing one — the whole line, not the field — and answers 200 with {"accepted":0,"skipped":N}. The daemon then deletes the batch it delivered. The result: no exception in the agent, nothing in its output, and a dashboard session list that looks exactly like an agent nobody ran. Measured against the running stack: AGENTEYE_ENVIRONMENT="prod,eu" produced accepted:0, skipped:1. `failproofaid` has always refused a comma in `collector.environment` for this exact reason. The SDK writes the same field into every event and never checked. configure(environment=...) now raises, naming the fix. The env var warns and falls back to "dev" instead: it is read lazily inside to_dict() on whatever event is next, so raising there would take the caller's agent down from a line of telemetry. Landing under a visibly wrong environment beats vanishing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…vents Found by driving real crews through a live gateway and reading the rows back out of the events store, not by inspection. Each fix was reverted individually against its new test to prove the test fails without it. 1. Hierarchical delegation was flattened. `_tool_start` emitted `tool_use` but never noted the tool as a node, and CrewAI parents a delegated coworker's whole AgentExecutionStartedEvent on the `delegate_work_to_coworker` TOOL event — so `_parent_key` missed it and fell back to `_roots[-1]`. Manager and both coworkers came out as siblings of each other under the crew. They now nest: coworker -> manager -> crew. 2. `FlowFailedEvent` was not in TABLE. A Flow whose method raises emits it and never emits `FlowFinishedEvent`, so the flow's `agent_start` was never closed and the session read `ongoing` forever — in a long-lived process, permanently. 3. A Crew kicked off inside a Flow method became a SECOND session. `_hook_start` did not note the flow-method span, so `on_crew_started` read "no parent" and minted a new root. One logical run, two unlinked sessions, no parent_id on either. 4. Cross-session leak through the process-global `_roots[-1]` fallback. With two crews open, any event whose parent span was gone — closed, or evicted at `_MAX_NODES` — landed in the OTHER run. Reproduced: an orphan tool emitted from crew Alpha's thread was recorded against crew Bravo. Root selection now matches the ambient session. 5. `Task(human_input=True)` recorded nothing at all. CrewAI has two HITL surfaces and only the Flow `@human_feedback` one is on the event bus; `SyncHumanInputProvider._prompt_input` calls `input()` and emits no event of any kind, so the entire human wait was billed as active agent time. Now the full human_wait/human_input/agent_pause/ agent_resume quartet: a real 38s wait measures as 37878ms paused inside a ~43s agent span. This is the adapter's only patch — narrow seam, staticmethod descriptor restored on uninstrument with an identity check, double-patch guarded, exceptions re-raised verbatim. 6. `Agent.kickoff()` (LiteAgent) had no agent span at all — the three LiteAgentExecution events were unmapped. With no ambient session it recorded ZERO rows; with one, everything landed under `agent_id=main`. Also corrects a stale docstring: `_parent_key` claimed async_execution tasks arrive with `parent_event_id=None` because a ThreadPoolExecutor drops contextvars. Measured against 1.15.16 — false; only the two root events have a null parent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Found by driving real LangChain runs through a live gateway and reading the rows back out of the events store. Each fix is covered by a test that fails when its hunk is reverted, and each has a counterweight test that fails when the fix is pushed too far. 1. Concurrent roots under one session id were mistaken for a HITL resume, and events were DROPPED. `_start_root` reused an existing session's agent whenever that agent was still open — which is also true of two roots that merely overlap in time: `.batch()` (langchain-core opens one root run per input), a top-level `RunnableParallel`, or two web requests carrying one conversation id, i.e. the documented `failproofai_sdk_session_id` stitching key. The second root got no `agent_start`, its work was relabelled with the first root's `agent_id`, the first root to finish closed the shared agent, and every later event from the other root resolved to nothing and was dropped — a real model call, with its tokens and its latency, gone behind one WARNING line. `.batch()` of three recorded 8 rows and one agent pair; it now records 12 and three. The test is the whole fix in miniature: a `threading.Barrier` forces both roots open at once, because without it the race passes against the bug about half the time. `open_pauses` is the discriminator. A genuinely paused run always has one — `_end_root` skips `agent_end` exactly when it is non-empty, and `_suspend` is the only thing that fills it — so it separates the two cases precisely. 2. A root run that is itself a leaf double-reported its failure. `_on_end` returned before setting `session.reported_error`, so a failing top-level `tool.invoke()`/`llm.invoke()` emitted `tool_result.error` AND a standalone `error`: one failure counted twice, while the same failure one Runnable deeper counted once. 3. `uninstrument()` did not stop recording when the trace env var was exported before `instrument()`. A configure hook cannot be deregistered, so teardown means "make the hook produce nothing" — but clearing the ContextVar only reaches contexts derived from the caller's, and the env var is deliberately left alone when the process set it. Either hole leaves `_configure` building live tracers: a full run was recorded after teardown. Now a `_State.enabled` kill switch, checked at the two entry points that gate everything else. 4. `tool_result.output` was a Python repr, and a quietly-failed tool had no error at all. A tool handed the LLM's `ToolCall` dict — what `bind_tools` produces and what every modern tool loop does — returns a `ToolMessage`, which rendered as `ToolMessage(content='37000000', name=…)` instead of `37000000`. And `ToolMessage.status == "error"` leaves `run.error` empty, so a tool whose exception the framework converted into a message for the model had NO representation: `is_error` 0, a green span, and the exception text sitting in a field nobody filters on. 5. Every Errors-surface row read `ValueError: ValueError: …`. `error` is the one event carrying `error_type` as its own field and the server composes `summary` as "<error_type>: <message>", but the adapter passed `_error_text`, which prefixes the type. The other three adapters pass a bare `str(exc)`; this makes the fourth agree. `agent_end.summary` keeps the prefixed form — it has no other column to say it in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`_core._truncate` and `_size` dispatched on the concrete `dict`, `list` and `tuple`. That missed every mapping a framework actually hands us which is not literally a dict — `MappingProxyType`, which is what `model_json_schema()` and any frozen config returns, `ChainMap`, and any third-party mapping type — and those fell through to the branch at the bottom that renders an object with no JSON shape via `repr`. The result is in the events store. A crewai `model_request` carries tools[0].function.parameters.properties.from_unit = "{'title': 'From Unit', 'type': 'string'}" a JSON string holding a Python repr. `JSONExtract` over it returns nothing, so the field is unqueryable rather than merely ugly — and a tool's declared schema is exactly what you go to a model_request to read. Both functions now dispatch on `collections.abc.Mapping` and `Sequence`/`Set`. `str` and `bytes` are handled before either check, so a string cannot be exploded into a list of characters, and an object that is neither a mapping nor a sequence still reprs — both pinned by a counterweight test, since widening the check could otherwise leave the repr branch dead. `_size` moves with it: a size computed off `repr` for a value `_truncate` will expand into JSON budgets the wrong number, and the budget decides which fields survive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Four behaviours changed on this branch and the docs still described the old ones. Each of these is the page a reader lands on when the thing goes wrong, so a stale answer there costs more than elsewhere. - python-sdk reference, `environment`: says it must not contain a comma and why. Ingest splits the field on commas for its facets and skips every event whose label has one, so the run vanishes with no error. `configure()` now refuses it, and the page says so at the row a reader is looking at when they choose a label. - python-sdk reference, shutdown: "hard process termination can lose events" was true and useless — it did not say that SIGTERM is one, and SIGTERM is the one you meet, on every rolling deploy and `docker stop`. Now names it, explains that CPython runs no handler so `atexit` never fires, and ships the handler that fixes it. - how-it-works, auto-detection: calling `instrument()` above the framework import records nothing at all, which is the single most expensive ordering mistake available and was documented nowhere. The page now says where to put the call and that a warning is logged. - crewai: the HITL section described one surface; CrewAI has two, and `Task(human_input=True)` — which emits no event at all and is covered by wrapping CrewAI's input provider — is the more common one. The event table also gains `Agent.kickoff()` and states the nesting rules the adapter now produces: a crew inside a flow method nests under it, and a delegated coworker nests under its manager. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Found by driving real LlamaIndex workflows through a live gateway and reading the rows back out of the events store. Each fix has a test that fails when its hunk is reverted. 1. AgentWorkflow handoffs were flattened into one agent. AgentWorkflow does not run its agents as nested workflows, so attribution from the span tree alone collapsed a two-agent crew into a single `agent_id="AgentWorkflow"` — 382 events under one label in the audited run. The real names existed only in the payload extra `fw_agent_name`, which is not a groupable column, so the delegation structure was unreadable on every dashboard surface. Each distinct `current_agent_name` now opens a nested agent under the workflow. The name is sticky because a `ToolCall` step carries none, so a `call_tool` keeps the agent that asked for it; a `name == root.agent_id` guard stops a standalone FunctionAgent nesting inside itself, and an A->B->A round trip opens the first agent again as a second, correctly closed turn. 2. A user-cancelled run was reported `outcome="success"`. `cancel_run()` does not drop the span — the runtime catches its own `WorkflowCancelledByUser` and exits the span cleanly with `result=None`. Rather than infer cancellation from a null result, the adapter now reads the framework's own `SpanCancelledEvent`, dispatched with the exact span id immediately before that exit. The run closes `cancelled` with no `error` event, because a stop button is not a failure, and the in-flight step flips from `success` to `cancelled` with it. That event is deliberately outside `_HANDLED_EVENTS`, since the drift test walks only `llama_index.core.instrumentation.events.*` — so it ships with a drift guard of its own, which fails if the class is renamed or moves, or if `span_id` leaves its fields. 3. A failed `agent_end` carried no `summary`. `summary` is a promoted column and the only place a run's outcome is read; the reason lived only on the failing step's `hook_completed` payload, and vanished entirely under `steps=False`. Now carried on both the exception and the timeout paths, matching the LangChain adapter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`normalize_agent_id` exists because `agent_id` is a `LowCardinality(String)` and the primary facet on every dashboard surface, so a per-run value in it degrades the column and fills the filter dropdown with one entry per run. It only caught a value that was an id ALL THE WAY THROUGH. `agent-<uuid>`, `crew_<uuid>`, `task-3f9a1c2b-…` — a readable name carrying a per-run suffix — went straight through. That is the shape frameworks actually produce, and it is the exact one the CrewAI page already warns about ("a role containing a UUID, timestamp, or per-run suffix"), so the guard was missing by far the more common route to the thing it prevents. The id portion is now stripped and the readable part kept: `agent-<uuid>` becomes `agent`, not `main` — collapsing it would discard the only meaningful token in the label. Dashed UUIDs are matched as a substring before the segment pass, or splitting on separators would break the most standard shape of all into five pieces that are individually innocent. A value with nothing left after stripping falls back to the default, which is what the caller wanted for a bare id anyway; a name where nothing was stripped is returned unchanged, separators included, so this cannot quietly rename every `node_a_b` in a process to `node a b`. Counterweight cases cover `agent-v2`, `step-3`, `node_a1b2` and `deadbeef`, which must all survive untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

- llamaindex: the event table and the naming section described a world where an `AgentWorkflow` was one agent. It is now one span per agent that takes a turn, parented to the workflow, so a handoff reads as two agents; a handoff back opens a second turn rather than reopening the first. The table also gains the cancel row — `cancel_run()` closes `cancelled` with no `error`, because a stop button is not a failure — and says that a failed `agent_end` now names what killed the run. - how-it-works: "Keep `agent_id` low cardinality" was an instruction with no explanation and no statement of what the SDK does about it. It now says why (it is a `LowCardinality` column and the primary facet), what adapters strip on your behalf, and — the part that actually matters to a reader — that the guard applies to labels the FRAMEWORK chose, not to an `agent_id` you pass yourself, which is taken exactly as given. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`collector.redact` scrubs credential-shaped strings — `sk-…`, `ghp_…` — and it is applied in `SpoolWriter::push`, where the daemon writes the events it captures itself. Batches the Python SDK writes go into the same spool directory without passing through that writer, so the daemon ships them byte for byte. Verified against the running stack: a `tool_use` whose `input.command` held `Authorization: Bearer sk-…`, and a `tool_result` holding a `ghp_…`, both arrived in the events store intact — while the daemon's own captures of the same strings are scrubbed by default. Nothing claimed otherwise, which is the problem: the asymmetry is invisible, two events on one delivery path are treated differently by who wrote them, and `redact` sits under `collector` where it reads like a machine-wide policy. A reader who sets it and assumes coverage is wrong and has no way to find out. The behaviour is deliberate — rewriting an SDK payload in transit would mean the events you receive are not the events you emitted — so this documents it and points at the two controls that do work: `capture_content=False`, and not passing the secret to `input=`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The health file answers "is each source producing events" and cannot answer "is anything arriving". A source's job ends when it writes a batch into the spool — the POST, the server's verdict and the parking of what would not go all happen after that — and the SDK's batches have no source entry at all, because `failproofai-sdk` writes them into the spool from the user's own process. So a machine shipping nothing but SDK events wrote a file with an empty, perfectly healthy-looking `sources` map whether ingest was storing every event or discarding all of them. That is not hypothetical. Ingest answers `200` with `{"accepted":N,"skipped":M}` and the daemon deletes the batch either way, so one systematically malformed field discards every event on the machine while every layer reports success. This audit found two such fields. The only trace was an ERROR line in the daemon's log — journald on a real install, which nobody reads until they already suspect a problem. `collector-health.json` gains a `delivery` section carrying the counters the `Uploader` already kept: accepted, skipped, batches fully skipped, and the timestamp of the last upload the server accepted. Verified live — a batch whose `environment` held a comma moved the file to `skipped: 2, batches_fully_skipped: 1`. The section is omitted, not zeroed, when there is no uploader: all-zero counters and "this daemon has no credential" are different facts and must not render the same. The counters are read through to the `Uploader` rather than copied at attach time, since it outlives any supervised task restart — a snapshot would freeze the file at "nothing has happened yet", which reads exactly like a healthy idle machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both example headers taught that a crewai task shows up as its own `hook_triggered`/`hook_completed` pair, `research_crew.py` at length: "task boundaries as hook pairs — crewai tasks are hooks, not nested agents, deliberately". They are not hooks either. The adapter emits nothing for a task on purpose, which `crewai.mdx` states correctly: a task IS the agent execution that runs it, so recording both would double every row and render them as siblings. The task's identity rides along on that agent's events as `fw_task_id` / `fw_task_name`. Checked against the rows rather than the code: across three real crew sessions the only `hook_triggered` is `length_guardrail` — a guardrail — while every one of those sessions carries `fw_task_name` on the agent's events. Re-ran both examples afterwards; neither produces a single hook event, and `research_crew.py` shows exactly the two `agent_id`s its header promises. These are the files a reader copies, so a false claim here is one they carry into their own instrumentation and then cannot find in the dashboard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Found by driving real Pydantic AI runs through a live gateway across all seven ways of driving an agent, and reading the rows back out of the events store. Each fix was reverted individually against its new test. 1. A cancelled leaf closed AFTER the agent it belongs to, and sometimes not at all. `wrap_run` returns before `wrap_tool_execute` / `wrap_model_request` do on the cancellation path: the graph awaits a gather of tool tasks, so the run body unwinds the moment that future is cancelled while each tool task's `CancelledError` lands a loop iteration later. Measured: `agent_end` at .565709 with the matching `tool_result` at .566689. The dashboard closes the agent span at `agent_end`, so anything after it is attributed to nothing — this adapter's own comments say so three times, and a sibling test already asserts that ordering for the model path. In some interleavings the ambient identity was gone by then and the late leaf was dropped outright, leaving a `tool_use` with no `tool_result` at all. Still-open leaves are now closed before `agent_end`, marked `fw_incomplete`, and the real handler becomes a no-op when it finally unwinds. 2. `uninstrument()` during a live run emitted two `agent_end`s for one `agent_start` — `cancelled` from teardown, then `success` from the run five seconds later, with the `tool_result` stranded between them. Whichever closes first now wins. 3. `tool_result.output` was a Python repr of an envelope. A tool returning `ToolReturn` recorded the whole repr, burying the answer next to `metadata` the model is documented never to see; pydantic models and dataclasses recorded as `Weather(city='Faro', celsius=21)`. Unwrapped via the objects' own `model_dump` / `dataclasses.asdict` — deliberately NOT by importing `pydantic_core`, which would put a third-party import in a package whose zero-dependency promise is enforced by a test and a `--no-deps` CI install. 4. A streamed `model_response.duration_ms` is the CONSUMER's time, and said nothing about it. On identical calls (23 in / 7 out both times): 2556ms with no consumer delay, 4059ms with 1.5s of sleep per delta — 1503ms of UI time inside the model's latency. The handler only returns when the caller leaves `async with agent.run_stream(...)`, and no earlier hook is overridable without switching `agent.run()` into streaming mode. The number cannot be made honest, only identifiable, so `fw_streaming` now rides on the response as well as the request — the request carries no `duration_ms` to exclude. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Neither is a Mapping or a Sequence, so both fell to the branch that renders an object with no JSON shape. A tool's argument model, its structured return, a settings object on a model request — every one of them reached the events store looking like `Weather(city='Faro', celsius=21)`: a Python repr inside a JSON string, which `JSONExtract` cannot read and the dashboard cannot filter on. Every framework hands us these, and the adapters had started solving it one at a time — the pydantic_ai adapter unwraps `ToolReturn` and its models in the commit before this one. Doing it once here means an adapter that has not thought about it still records something readable. The unwrap is deliberately SHALLOW. `dataclasses.asdict` and `model_dump` both recurse and both copy, so on a large object they duplicate the whole tree before `_truncate` gets to decide it only wanted the first 8 KB. Reading the top level and handing it back lets the existing walk apply the field limit, the item cap and the depth cap on the way down, exactly as it does for a dict. Guarded, because all of this runs the caller's own code — a validator, a property behind `getattr`. Anything that raises falls through to `repr`, which is what happened before this existed, so the worst case is the old behaviour rather than an exception in someone's agent loop. `model_dump` and not `dict`: pydantic v2 names it distinctively, while half the objects in a typical process have some attribute called `dict`. And a CLASS is excluded explicitly — `dataclasses.is_dataclass` is true of the class as well as its instances, and `fields()` on the class would render a type as though it were data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…oval Found by driving real LangGraph runs — StateGraph, subgraphs two levels deep, ReAct loops, interrupts across two processes — through a live gateway and reading the rows back out of the events store. Every fix has a test that fails when its hunk is reverted, and the two that could be pushed too far have counterweights that fail when they are. 1-3. `_node_of` claimed runs that were not the node. It matched on `run.name == metadata["langgraph_node"]`, and BOTH sides of that are strings the user chooses. Three distinct silent failures, one cause: - `add_node("lookup_population", ToolNode([...]))` recorded NO `tool_use` or `tool_result` at all. The arguments, the result and the LLM's own `tool_call_id` were dropped, and two hook pairs appeared where the tool should have been. Naming a node after the tool it runs is the obvious thing to do. - `add_node("ChatOpenAI", ...)` recorded no `model_request` or `model_response` — model name, both token counts and latency gone. - An inner runnable whose `run_name` matched the node key, or `sub.compile(name="child")` under `add_node("child", sub)`, emitted TWO hook pairs per visit: node counts doubled, apparent latency halved. A node's own run must now also be a non-leaf `run_type` and carry no `seq:step:` tag. Verified against 1.2.11: whatever you hand `add_node`, the node's own run is a `chain` tagged `graph:step:N`, and the thing you handed it runs beneath tagged `seq:step:N`. Both conditions are exclusions, so a tag-convention change upstream degrades to duplicate spans rather than to none — `_node_of` gates `hook_triggered` and `_ensure_subgraph_agent` both. 4. A run that merely OVERLAPPED a pause fabricated the human's approval. `_start_root` read "this session has an open pause and its agent is still open" as a resume — a window that lasts as long as the human takes. Any other run carrying that session id inside it (a second request on one conversation id, a background summariser, a different graph) got no `agent_start`, had its nodes folded into the paused span, and emitted `agent_resume` + `human_input` with an EMPTY response, closing the pause and reporting success. The dashboard then shows an approval that no human gave. A resume must now also look like one: LangGraph continues an interrupted thread only via `Command(...)` or `None`, both shaped unlike fresh state. 5. A cross-process resume never closed the pause — which is the real deployment shape. Two processes against one checkpointer: the first emitted `human_wait` + `agent_pause`, the second emitted nothing, so every cross-process approval left its session reporting "still waiting on a human" forever and `pausedMs` never closed. The fix rests on three facts verified against the framework rather than assumed: `Interrupt.id` is `xxh3_128(checkpoint_ns)` and the interrupted task's namespace is byte-identical across the two invocations, so the second process reconstructs the id with no shared state; `on_resume` fires once per Pregel level, deepest last, which is what excludes a subgraph host; and only a level's first superstep re-runs interrupted tasks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…framework in the no-op warning Three things the LangGraph pass surfaced that belong outside the adapter. - The langchain page now says a node's own run is identified by its SHAPE, not its name, so `add_node("lookup_population", ToolNode(...))` records the tool. That naming used to make the tool's events vanish, and it is the obvious thing to type, so the page should say plainly that it is safe. - Streamed token counts need `ChatOpenAI(stream_usage=True)`. OpenAI only sends usage on a streamed response when asked, so without it `model_response` carries no tokens — the adapter records what the framework gives it, and there is nothing to record. Measured both ways: NULL tokens without the flag, 13/13 with it. Users were reading that absence as a bug in the adapter. - `Command(resume=...)` is noted as correlating on the `Interrupt.id` including across processes, which is the deployment shape and the one the fix in the previous commit was about. Also: the "nothing was instrumented" warning suggested `instrument('crewai')` regardless of what was installed. It now lists every name the call would have accepted — a reader not using CrewAI had to work out for themselves whether that line was a suggestion or a diagnosis. Its test moves from emptying the registry to pointing detection at an unimportable module, so the list of valid names is real and asserted rather than rendered as nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Keeps this branch stacked on its base after #702 took main's changes. `main` is now an ancestor transitively, so #730 stays mergeable into #702 and #702 into main. One conflict, `docs/docs.json`: main restructured the navigation with an extra nesting level per language (#725), in the same "Start here" group this branch added the integration pages to. Resolved by taking main's structure — the rest of the file already follows it — and putting the `Plug in your agents` group back inside it. All 62 pages the English navigation references were then checked to exist on disk, which is what the docs job validates and what the nav-pruning fix in #725 was about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`disabling_collection_stops_it_and_re_enabling_starts_it_again` exists for `--disconnect`: a machine that has left its organisation must stop shipping without a restart. It simulated that by flipping `collector.hooks` to false, which is not what `--disconnect` does — that clears the ingest credential (`clearIngestCredential` in cloud-enrollment-cli.ts) — and which no longer disables anything, because `hooks` gates the daemon's own capture source and deliberately does not gate delivery of the batches the SDK writes. So the test now removes and restores the credential. That is the real lever for the scenario it was written about, and a stronger assertion than the proxy it replaces. A companion pins what replaced the old behaviour: with both capture sources off the daemon still starts the spool watcher — without which it is a process that reports healthy and delivers nothing — and still starts no hook-activity source, so `hooks = false` keeps meaning what an operator sets it for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`ci.yml` triggers on `pull_request` into `main`. #730 targets `feat/fp-cli`, so thirty commits of SDK work ran no unit tests, no build, no lint and no docs check — the only status it produced was the daemon cross-compile, and only because it touched `crates/`. A pull request that cannot go red is not a reviewed pull request. Turning it on immediately found what it had been missing: the `fp-cli` and `failproofai-sdk` jobs declare no `timeout-minutes`, which main made mandatory in #726 and asserts in `release-pipeline.test.ts`. Those two jobs predate the rule and had never been run against it, so #702 would have gone red the moment it merged into main. Both are bounded now, at 10 minutes — a `uv sync` plus pytest, across two interpreters and five. Also rewraps the daemon-skew warning in `fp-reset.ts`. "denies every tool call" is the consequence that message exists to state, and it was split across two hand-wrapped lines, so the test asserting the phrase failed against that branch of the message while the text read perfectly to a human. The other two branches of the same warning keep the phrase whole; this one now matches them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hermes-exosphere

Hermes found blocking issues that should be addressed.

Review coverage was incomplete, but the concrete blocking findings below are sufficient to request changes.

High: Explicit Pydantic AI capabilities are inert without instrument()

Rule: COR-001
Location: sdk/python/failproofai_sdk/integrations/pydantic_ai.py:605
Evidence: The documented standalone API constructs capabilities=[FailproofAI()] at pydantic_ai.py:559, but wrap_run returns directly to the framework whenever module-global _enabled is false (:605-606). _enabled is initialized false and is set only by _Adapter.install() (:913-926), which is reached through instrument(). The model and tool wrappers have the same gate. The existing explicit-capability test uses the instrumented fixture, so it exercises only the enabled state.
Required change: Make an explicitly constructed FailproofAI capability active independently of registry installation, while retaining a separate disable mechanism for auto-injected capabilities after uninstrument(). Add a test that runs an agent with only capabilities=[FailproofAI()] and asserts the emitted events.

1 advisory finding

Medium/High Falsy supplied session IDs are silently replaced — session._enter() chooses self._requested or an inherited/generated ID (_scopes.py:97), and agent._enter() does the same (:197). Thus a supplied empty string or 0 is not validated as invalid identity; it is replaced by an unrelated generated session, or by an enclosing session when one exists. A containerized reproduction showed session("") and agent(session_id=0) both entering with generated UUIDs. (sdk/python/failproofai_sdk/_scopes.py:97)

Merging main and then #730 into this branch left `1.0.1-beta.2` with two `### Docs` headings — the union kept both sides' sections rather than folding them together. Entries are unchanged and all 758 are still there; they now sit under one heading, in the order they were already in. Checked against the base rather than by eye: main carries 8 pre-existing duplicate subsections in older versions, this branch had 9, and it is back to 8 — so the only one removed is the one this PR introduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

SiddarthAA and others added 6 commits August 19, 2026 20:01

docs(changelog): record the SDK framework adapters (#730)

69aa34e

Every PR carries an entry, and this one touched nothing outside sdk/python until now — which is exactly how a release note goes missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hermes-exosphere requested changes Aug 19, 2026

View reviewed changes

SiddarthAA and others added 7 commits August 20, 2026 13:13

docs(changelog): record the adapter fixes and the integration guide

6231cf4

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q645koFdi3k9eZHk9b1qLj

hermes-exosphere requested changes Aug 20, 2026

View reviewed changes

SiddarthAA and others added 10 commits August 20, 2026 14:08

SiddarthAA and others added 13 commits August 20, 2026 14:27

docs(changelog): record the audit's daemon, core and adapter fixes

7b6013a

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs(changelog): record the langgraph, pydantic_ai and core fixes

61fa5ee

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hermes-exosphere requested changes Aug 20, 2026

View reviewed changes

SiddarthAA merged commit ee735f5 into feat/fp-cli Aug 20, 2026
24 checks passed

SiddarthAA deleted the sdk/framework-adapters branch August 20, 2026 10:23

Navigation Menu

Uh oh!

[sdk] Native support for LangChain/LangGraph, CrewAI, LlamaIndex and Pydantic AI, on a real identity layer - #730

[sdk] Native support for LangChain/LangGraph, CrewAI, LlamaIndex and Pydantic AI, on a real identity layer#730
SiddarthAA merged 38 commits into
feat/fp-clifrom
sdk/framework-adapters

SiddarthAA commented Aug 19, 2026 •

edited by hermes-exosphere

Loading

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

coderabbitai Bot commented Aug 19, 2026 •

edited

Loading

Review skipped

Uh oh!

hermes-exosphere commented Aug 19, 2026 •

edited

Loading

Uh oh!

hermes-exosphere left a comment

Uh oh!

hermes-exosphere Aug 19, 2026

Uh oh!

hermes-exosphere Aug 19, 2026

Uh oh!

hermes-exosphere left a comment

Uh oh!

hermes-exosphere left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		# -- run --------------------------------------------------------------

		async def wrap_run(self, ctx, *, handler):

Uh oh!

Conversation

SiddarthAA commented Aug 19, 2026 • edited by hermes-exosphere Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why this is not the same as emitting events by hand

What is here

Three decisions where #503 was right and the current branch was wrong

One decision where the current branch was right and #503 was not

Zero dependencies survives, and the test got stronger

Verification

Not checked

For the reviewer

Hermes review

Summary

Changes

Validation

Findings

Open questions

Policy overrides

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

coderabbitai Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

hermes-exosphere commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hermes

What this changes

Rounds

Findings

Uh oh!

hermes-exosphere left a comment

Choose a reason for hiding this comment

High: Explicit Pydantic AI capabilities are inert unless auto-instrumentation is also enabled

Uh oh!

hermes-exosphere Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

hermes-exosphere Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

hermes-exosphere left a comment

Choose a reason for hiding this comment

High: Explicit Pydantic AI capabilities emit no telemetry unless instrument() is also called

Uh oh!

hermes-exosphere left a comment

Choose a reason for hiding this comment

High: Explicit Pydantic AI capabilities are inert without instrument()

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SiddarthAA commented Aug 19, 2026 •

edited by hermes-exosphere

Loading

coderabbitai Bot commented Aug 19, 2026 •

edited

Loading

hermes-exosphere commented Aug 19, 2026 •

edited

Loading