Skip to content

test(cli): prove call-order tolerance without losing the strictness - #275

Merged
AminChirazi merged 1 commit into
mainfrom
feat/falsifiability-call-order
Jul 31, 2026
Merged

test(cli): prove call-order tolerance without losing the strictness#275
AminChirazi merged 1 commit into
mainfrom
feat/falsifiability-call-order

Conversation

@AminChirazi

Copy link
Copy Markdown
Contributor

Milestone 2, criterion 7. Closes #258. Based on main.

The gap

0.8.0 dropped positional cassette matching. goose issues its task call and a session-title call concurrently and does not wait, so whichever landed first at record decided the order, and a positional matcher reported a divergence when nothing about the agent had changed. As docs/agent-testing.md puts it: order between concurrent calls is not something the agent guarantees, so a recording cannot assert it either.

Nothing exercised the new behaviour, in either direction.

Two proofs, and the second keeps the first honest

Tolerance two INDEPENDENT calls recorded a-then-b replay b-then-a, and the run still passes
Discrimination an agent that changes what it SENDS still diverges

Without the discrimination proof, "order-tolerant matching" is indistinguishable from "the request is not checked at all". A matcher that accepted anything would satisfy the tolerance proof perfectly — and every cassette in the product would stop proving anything about what was sent.

That is the shape a relaxation turns into a false green: the loosening is correct, and nothing pins how far it goes. So the pair ships together rather than the first alone.

The fixture

two-call-agent.py makes two calls that are deliberately independent of each other. That independence is the premise of the whole proof — it is what makes their relative order meaningless and the tolerance correct, rather than merely convenient. The header says so, and says not to make them depend on each other.

Two knobs live in files beside the script rather than the environment — the convention mcp_stdio_e2e.rs already uses — so record and replay differ without a race.

Result

Both proofs behave. No false green.

Verified locally

Replay runs with no model reachable, per invariant 1. Tests pass both with --test-threads=1 and in parallel (checked deliberately, since CI runs cargo test --workspace without that flag). fmt --check = 0, clippy --all-targets -D warnings = 0. Each exit code captured directly.

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings July 30, 2026 23:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@AminChirazi
AminChirazi force-pushed the feat/falsifiability-call-order branch from 77d69f0 to e2fc37d Compare July 31, 2026 04:03
0.8.0 dropped positional cassette matching. goose issues its task call and a
session-title call concurrently and does not wait, so whichever landed first
at record decided the order, and a positional matcher reported a divergence
when nothing about the agent had changed. Order between concurrent calls is
not something an agent guarantees, so a recording cannot assert it either.

Nothing exercised the new behaviour, in either direction.

Two proofs, and the second is what keeps the first honest.

Tolerance: two INDEPENDENT calls recorded a-then-b replay b-then-a, and the
run still passes. Discrimination: an agent that changes what it SENDS still
diverges.

Without the discrimination proof, "order-tolerant matching" is
indistinguishable from "the request is not checked at all" -- a matcher that
accepted anything would satisfy the tolerance proof perfectly, and every
cassette in the product would stop proving anything about what was sent.
That is the shape a relaxation turns into a false green, and it is why the
pair ships together rather than the first alone.

The fixture's two calls are deliberately independent of each other. That
independence is the premise: it is what makes their relative order
meaningless and the tolerance correct, rather than a convenience.

Replay runs with no model reachable, per invariant 1.

Closes #258

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AminChirazi
AminChirazi force-pushed the feat/falsifiability-call-order branch from e2fc37d to ef4fd50 Compare July 31, 2026 04:04
@AminChirazi
AminChirazi merged commit 7377b7e into main Jul 31, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Falsifiability regression fixture: concurrent-call order wrongly asserted

3 participants