Skip to content

Falsifiability: assert_tool_call red-path proof #247

Description

@AminChirazi

Prove assert_tool_call can fail. Fixture: a cassette whose tool-call sequence omits or reorders the required call, plus one whose argument at an asserted dotted path diverges.

Definition of done

  1. A NEW fixture under tests/falsifiability/fixtures/ that violates the
    property under test.
  2. A test invoking the real assertion path end-to-end as a library call
    (CHARTER §2: every operation is a library call). When a process exit code
    must be checked, capture it directly — never through a pipe.
  3. Two checks at two layers: the FAIL verdict, and the non-zero exit code.
  4. One appended line in docs/how-flowproof-tests-flowproof.md naming the
    assertion, the fixture, and the defect class it guards.

Hard bars

The Adversary will refuse these even with green CI:

  • Do not modify any existing test, committed fixture, assertion implementation,
    or anything under scripts/gate/. Committed cassettes are ground truth
    (§2) — craft new fixtures; never edit a committed cassette into a violation.
  • If your red-path test PASSES where you expected FAIL, you have found a real
    false-green defect. That is a RESULT, not a task. Report it with the fixture
    attached for the Ledger keeper; the fix is a separate issue you must not
    self-authorise.
  • Fixtures are deterministic and offline. Replay makes zero LLM calls and
    requires no API key.

Charter: Milestone 2 criteria 6–8 (amendment 2026-07-30). Brief:
docs/loop/milestones/m2-falsifiability.md. Parent: #61.

Metadata

Metadata

Assignees

No one assigned

    Labels

    readyA loop Builder may pick this uptesting

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions