[AAASM-4809] ✅ (test): Drive real pydantic_ai/agno/llamaindex hooks in fail-open conformance#277
Merged
Conversation
…nformance The pinned/shipped Pydantic AI (>=0.3.0) intercepts tool execution at AbstractToolset.call_tool, not the <0.3.0 Tool._run hook. The conformance cell drove _apply_tool_run_patch, exercising a wrapper the shipped framework never runs. Point it at _apply_toolset_call_tool_patch with the real (self, name, tool_args, ctx, tool) signature and a fake toolset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agno driver drove only the sync FunctionCall.execute wrapper, leaving the async aexecute chokepoint — agno's primary async tool path — unexercised. Drive both entry points and add _assert_sync_and_async_agree so an async-only fail-open regression cannot hide behind a passing sync path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The llamaindex driver drove only the sync FunctionTool.call wrapper; acall — the primary modern path awaited by FunctionAgent / ReActAgent via AgentWorkflow — went unexercised. Apply _apply_tool_acall_patch and drive both, reusing the sync/async agreement guard so the async path is genuinely governed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
Author
🤖 Claude Code — PR reviewScope (AAASM-4809, LOW): ✅ fixed conformance-matrix blind spots — pydantic_ai driver now drives the real
Verdict: ready to merge pending Pioneer approval. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The fail-open conformance matrix (
test/unit/adapters/failopen_conformance.py) had coverage blind spots: three of its adapter cells drove a wrapper that the shipped framework never actually runs, or exercised only the sync sibling of an async chokepoint. This makes those cells green while the real interception path stays untested._apply_tool_run_patch(the pydantic-ai<0.3.0Tool._runhook). On the pinned/shipped pydantic-ai (>=0.3.0),Tool._rundoes not exist and production interception isAbstractToolset.call_tool(patch.py:348, signature(self, name, tool_args, ctx, tool)). The cell drove a wrapper that never runs on the shipped framework. Now points at_apply_toolset_call_tool_patchwith a fake toolset and the real signature.FunctionCall.aexecute(patch.py:208) was unexercised; the driver drove only the syncexecute. Now drives both.FunctionTool.acall(patch.py:303; awaited byFunctionAgent/ReActAgentviaAgentWorkflow) was unexercised; the driver drove only synccall. Now drives both.A shared
_assert_sync_and_async_agreeguard requires an adapter's sync and async wrappers to reach the same verdict per cell, so an async-only fail-open (or fail-closed) regression cannot hide behind a passing sync path.Type of Change
Breaking Changes
Related Issues
Testing
The conformance suite is its own regression guard:
pytest test/unit/adapters/test_failopen_conformance.py— 361 passed, and the pydantic_aicall_tool, agnoaexecute, and llamaindexacallwrappers are now genuinely driven across every scenario × mode. Verified out-of-band that each async path blocks under enforce and runs under observe, and that the agreement guard catches a sync/async divergence.ruff check,black, andmypyall clean.Checklist