audit --since: a control turning capability-error is not treated as a regression

[AminChirazi]

Found while writing the falsifiability proofs for #254. Not fixed — this is a design question, and per the Milestone 2 brief a discovery is reported rather than self-authorised.

What the code does

crates/flowproof-replay/src/runrecord.rs:

pub fn is_regression(&self) -> bool {
    !self.removed.is_empty() || self.changed.iter().any(|c| c.new == ControlVerdict::Fail)
}

So audit --since exits non-zero for a removed control, or one that turned fail. A control going pass → capability-error exits zero.

Why it might matter

capability-error means "this control could not be certified here". The agent-testing docs are emphatic that this must never read as a pass — assert_no_egress on a non-Linux host fails as a capability error specifically so it does not "pass vacuously".

But at the diff layer that distinction disappears. A control that was being enforced yesterday and cannot be certified today — a runner without seccomp, a driver that lost a capability, a suite moved to a host where containment is unavailable — passes audit --since clean. Under the evidence positioning, "we can no longer certify this" is arguably exactly what a regression gate exists to surface.

Why it might be intended

capability-error is not a failure of the system under test, and treating it as a regression would make the gate red on legitimate platform moves — running the same suite on macOS, for instance. That is a real cost and may be why it reads the way it does.

The decision

Three options, roughly:

1. Leave it. Capability loss is an environment fact, not a control regression.
2. Treat pass → capability-error as a regression. Strictest, and consistent with "never pass vacuously".
3. Report it as a distinct third category — neither silent nor build-failing — so a reviewer sees "controls no longer certifiable" without the gate going red on a platform move.

Whichever is chosen, the falsifiability suite should gain a proof pinning it, so the behaviour stops being implicit.

Context: #254, PR #262.

[AminChirazi]

Decided (founder, 2026-07-30): treat pass -> capability-error as a regression. Option 2. Rationale: consistent with the rule the agent-testing docs already insist on, that a capability error must never read as a pass. A control that can no longer be certified is a control you have lost, and the gate exists to say so.

Accepted cost, stated so nobody re-litigates it later: running an existing suite on a host without seccomp will now turn the build red. That is intended. If it proves noisy in practice the answer is to fix the host or narrow the control, not to soften the gate.

Scope for whoever takes this:

1. crates/flowproof-replay/src/runrecord.rs — is_regression() gains the new == ControlVerdict::CapabilityError case alongside Fail.
2. A falsifiability proof pinning it, in the style of falsifiability_audit_since.rs: a record pair where a control goes pass -> capability-error must exit NON-ZERO and name the control. Fixture under tests/falsifiability/fixtures/audit-since/.
3. Check audit_record_e2e.rs still passes unchanged — it exercises removed and pass->fail only, so it should, but confirm rather than assume.
4. CHANGELOG entry in the repo's voice: what was wrong, why it mattered, what holds now.

Labelled ready.