test(cli): prove the assert_tool_call argument matchers can fail - #271
Merged
Conversation
AminChirazi
force-pushed
the
feat/falsifiability-arg-matchers
branch
2 times, most recently
from
July 31, 2026 04:04
c91ef8d to
d0727b0
Compare
The tool-NAME layer already had a red path: a flow demanding a tool the agent never calls refuses the trace. Nothing proved the `where <path> <matcher>` clauses can fail at all. That is the layer carrying the most weight and the least evidence. docs/agent-testing.md calls argument assertions "usually where the bugs are", and names chained arguments -- threading one tool's result into the next call -- as the behaviour multi-step agents actually get wrong. Yet every `where` clause in the suite asserts an argument the model was always going to produce, so each passes whether the matcher works or not. One committed guilty call covers both halves of the vocabulary. The right tool with the wrong city violates a value matcher (`where city equals Nairobi`, the value is Mombasa) and a presence matcher (`where city is absent`, the key is there) at once. Both refuse the record, and the helper asserts neither minted a trace -- a presence matcher that could not fail would let a flow assert the absence of anything at all. Both matchers behaved. Closes #247 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AminChirazi
force-pushed
the
feat/falsifiability-arg-matchers
branch
from
July 31, 2026 04:11
d0727b0 to
dda1416
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 2, criterion 6. Closes #247. Based on
main.The gap
The tool-NAME layer already has a red path —
a_failing_assertion_refuses_the_trace(agent_flow_e2e.rs:778) demands a tool the agent never calls, and record refuses the trace. Nothing proved thewhere <path> <matcher>clauses can fail at all.That is the layer carrying the most weight and the least evidence.
docs/agent-testing.mdcalls argument assertions "usually where the bugs are", and names chained arguments — threading one tool's result into the next call — as the behaviour multi-step agents actually get wrong. Yet everywhereclause in the suite asserts an argument the model was always going to produce, so each passes whether the matcher works or not.One guilty call, both halves of the vocabulary
The fixture is the right tool with the wrong city, and it violates two assertions at once:
where city equals Nairobi— a value matcher; the value is Mombasawhere city is absent— a presence matcher; the key is thereBoth refuse the record, and the helper asserts neither minted a trace. The presence case is worth its own test: a presence matcher that could not fail would let a flow assert the absence of anything at all and stay green.
Result
Both matchers behaved. No false green.
Verified locally
fmt --check= 0,clippy --all-targets -D warnings= 0, both tests = 0 — each captured directly, never through a pipe.🤖 Generated with Claude Code