[false-green] MCP lane loses the final call when the agent kills the stand-in - #274
[false-green] MCP lane loses the final call when the agent kills the stand-in#274AminChirazi wants to merge 3 commits into
Conversation
…stand-in
EVIDENCE, NOT A FIX. This test is expected to FAIL. Do not merge.
0.8.0 fixed "the MCP stand-in no longer loses its recording when the agent
kills it" -- the lane had been written only at stdin EOF, so an abrupt kill
lost everything. The lane is now persisted after every captured call.
It is not enough. An agent that kills the stand-in immediately after reading
a response loses THAT call from the lane, while every earlier call survives.
Observed, with the real server's own log as the oracle:
server received: initialize, notifications/initialized, tools/list,
tools/call get_weather
lane recorded: initialize, tools/list
`record` exited 0 and minted a trace. So a tool the agent demonstrably
invoked is absent from the evidence, and nothing said so.
That is the defect class this milestone exists to eliminate, in the artifact
the MCP boundary's claims rest on. A lane that silently runs short understates
what the agent did, and every assertion evaluated over it inherits the
omission.
The fixture is an agent identical to the suite's polite one but for its last
act: proc.kill() instead of stdin.close(); wait(). Every existing MCP test
closes politely, which is exactly the path the 0.8.0 defect did not affect.
Reported rather than fixed: the brief forbids self-authorising a fix for a
defect the same run discovered.
Refs #257
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rebased onto main, and the CI noise is goneCI was failing twice, and only one of those was the point. Now rebased onto
The single remaining failure is the assertion that is the finding: So CI red on this PR now means exactly one thing, which is what an evidence PR needs to mean. Still not a merge candidateUnchanged: this is evidence for #273, not a fix. The test is expected to fail until the underlying defect is addressed, and the fix is a separate change — the brief forbids the finder self-authorising it. When the fix lands, this test should go green and stay in the suite as the permanent guard. #273 also raises the question worth more than the fix: whether an incomplete lane should be detectable at all, rather than silently minted. |
05461d1 to
f07f2f0
Compare
Formatting only. The single intended CI failure on this branch is the assertion that documents the finding, not a lint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Superseded by #279, which merged as This PR was evidence, not a fix: the test in it was expected to fail, and did — on this box and on GitHub's runners. #279 fixes the defect it documented and carries this exact test cherry-picked unchanged, where it now passes. The test was never adjusted to fit the fix. Verified on The evidence did its job — the branch can go. |
DRAFT — evidence, not a fix. The test in this PR is expected to FAIL. Do not merge as-is.
Discovery from the Milestone 2 falsifiability suite while writing #257. Full write-up in #273.
The finding
0.8.0 fixed "the MCP stand-in no longer loses its recording when the agent kills it" by persisting the lane after every captured call. An agent that kills the stand-in immediately after reading a response still loses that call.
With the real server's own request log as the oracle:
initialize,notifications/initialized,tools/list,tools/call get_weatherinitialize,tools/listrecordexited 0 and minted a trace. A tool the agent demonstrably invoked is absent from the evidence.Why it matters
CHARTER §5 ranks a false green as the only defect class that destroys the product's value. The lane is the record of what the agent did at the tool boundary; one that silently runs short understates it, and every assertion over that lane inherits the omission — including guard-path claims that a tool was never called.
What is in this PR
tests/falsifiability/fixtures/mcp-killing-agent.py— identical to the suite's polite agent but for its last act:proc.kill()instead ofstdin.close(); wait()crates/flowproof-cli/tests/falsifiability_mcp_kill.rs— the red-path proof, which currently fails ontools/callThe test deliberately checks the server log first, so an exchange that never happened cannot pass as a vacuously "preserved" lane.
Why nothing caught it
mcp_stdio_e2e.rshas no kill or signal coverage — every test closes stdin politely and waits, exactly the path the 0.8.0 defect did not affect. The fix shipped with no test that could fail for its own bug: the same pattern 0.9.1's credential leak was described in, "working by inspection rather than by measurement."Next
Per the brief, the fix is a separate change I must not self-authorise. When it lands, this test should go green and stay in the suite as the permanent guard. #273 also raises whether an incomplete lane should be detectable at all rather than silently minted.
🤖 Generated with Claude Code