Skip to content

test(cli): prove the assert_tool_call argument matchers can fail - #271

Merged
AminChirazi merged 1 commit into
mainfrom
feat/falsifiability-arg-matchers
Jul 31, 2026
Merged

test(cli): prove the assert_tool_call argument matchers can fail#271
AminChirazi merged 1 commit into
mainfrom
feat/falsifiability-arg-matchers

Conversation

@AminChirazi

Copy link
Copy Markdown
Contributor

Milestone 2, criterion 6. Closes #247. Based on main.

The gap

The tool-NAME layer already has a red path — a_failing_assertion_refuses_the_trace (agent_flow_e2e.rs:778) demands a tool the agent never calls, and record refuses the trace. Nothing proved the where <path> <matcher> clauses can fail at all.

That is the layer carrying the most weight and the least evidence. docs/agent-testing.md calls argument assertions "usually where the bugs are", and names chained arguments — threading one tool's result into the next call — as the behaviour multi-step agents actually get wrong. Yet every where clause in the suite asserts an argument the model was always going to produce, so each passes whether the matcher works or not.

One guilty call, both halves of the vocabulary

The fixture is the right tool with the wrong city, and it violates two assertions at once:

  • where city equals Nairobi — a value matcher; the value is Mombasa
  • where city is absent — a presence matcher; the key is there

Both refuse the record, and the helper asserts neither minted a trace. The presence case is worth its own test: a presence matcher that could not fail would let a flow assert the absence of anything at all and stay green.

Result

Both matchers behaved. No false green.

Verified locally

fmt --check = 0, clippy --all-targets -D warnings = 0, both tests = 0 — each captured directly, never through a pipe.

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings July 30, 2026 21:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@AminChirazi
AminChirazi force-pushed the feat/falsifiability-arg-matchers branch 2 times, most recently from c91ef8d to d0727b0 Compare July 31, 2026 04:04
The tool-NAME layer already had a red path: a flow demanding a tool the
agent never calls refuses the trace. Nothing proved the `where <path>
<matcher>` clauses can fail at all.

That is the layer carrying the most weight and the least evidence.
docs/agent-testing.md calls argument assertions "usually where the bugs
are", and names chained arguments -- threading one tool's result into the
next call -- as the behaviour multi-step agents actually get wrong. Yet every
`where` clause in the suite asserts an argument the model was always going to
produce, so each passes whether the matcher works or not.

One committed guilty call covers both halves of the vocabulary. The right
tool with the wrong city violates a value matcher (`where city equals
Nairobi`, the value is Mombasa) and a presence matcher (`where city is
absent`, the key is there) at once. Both refuse the record, and the helper
asserts neither minted a trace -- a presence matcher that could not fail
would let a flow assert the absence of anything at all.

Both matchers behaved.

Closes #247

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AminChirazi
AminChirazi force-pushed the feat/falsifiability-arg-matchers branch from d0727b0 to dda1416 Compare July 31, 2026 04:11
@AminChirazi
AminChirazi merged commit 74e31df into main Jul 31, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Falsifiability: assert_tool_call red-path proof

3 participants