test(cli): prove call-order tolerance without losing the strictness - #275
Merged
Conversation
AminChirazi
force-pushed
the
feat/falsifiability-call-order
branch
from
July 31, 2026 04:03
77d69f0 to
e2fc37d
Compare
0.8.0 dropped positional cassette matching. goose issues its task call and a session-title call concurrently and does not wait, so whichever landed first at record decided the order, and a positional matcher reported a divergence when nothing about the agent had changed. Order between concurrent calls is not something an agent guarantees, so a recording cannot assert it either. Nothing exercised the new behaviour, in either direction. Two proofs, and the second is what keeps the first honest. Tolerance: two INDEPENDENT calls recorded a-then-b replay b-then-a, and the run still passes. Discrimination: an agent that changes what it SENDS still diverges. Without the discrimination proof, "order-tolerant matching" is indistinguishable from "the request is not checked at all" -- a matcher that accepted anything would satisfy the tolerance proof perfectly, and every cassette in the product would stop proving anything about what was sent. That is the shape a relaxation turns into a false green, and it is why the pair ships together rather than the first alone. The fixture's two calls are deliberately independent of each other. That independence is the premise: it is what makes their relative order meaningless and the tolerance correct, rather than a convenience. Replay runs with no model reachable, per invariant 1. Closes #258 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AminChirazi
force-pushed
the
feat/falsifiability-call-order
branch
from
July 31, 2026 04:04
e2fc37d to
ef4fd50
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 2, criterion 7. Closes #258. Based on
main.The gap
0.8.0 dropped positional cassette matching. goose issues its task call and a session-title call concurrently and does not wait, so whichever landed first at record decided the order, and a positional matcher reported a divergence when nothing about the agent had changed. As
docs/agent-testing.mdputs it: order between concurrent calls is not something the agent guarantees, so a recording cannot assert it either.Nothing exercised the new behaviour, in either direction.
Two proofs, and the second keeps the first honest
Without the discrimination proof, "order-tolerant matching" is indistinguishable from "the request is not checked at all". A matcher that accepted anything would satisfy the tolerance proof perfectly — and every cassette in the product would stop proving anything about what was sent.
That is the shape a relaxation turns into a false green: the loosening is correct, and nothing pins how far it goes. So the pair ships together rather than the first alone.
The fixture
two-call-agent.pymakes two calls that are deliberately independent of each other. That independence is the premise of the whole proof — it is what makes their relative order meaningless and the tolerance correct, rather than merely convenient. The header says so, and says not to make them depend on each other.Two knobs live in files beside the script rather than the environment — the convention
mcp_stdio_e2e.rsalready uses — so record and replay differ without a race.Result
Both proofs behave. No false green.
Verified locally
Replay runs with no model reachable, per invariant 1. Tests pass both with
--test-threads=1and in parallel (checked deliberately, since CI runscargo test --workspacewithout that flag).fmt --check= 0,clippy --all-targets -D warnings= 0. Each exit code captured directly.🤖 Generated with Claude Code