Skip to content

Phase 1: M0 experiment register, eval harness scaffold, opengrep acquisition - #4

Merged
tom-snyder merged 2 commits into
mainfrom
feat/m0-scaffold
Aug 7, 2026
Merged

Phase 1: M0 experiment register, eval harness scaffold, opengrep acquisition#4
tom-snyder merged 2 commits into
mainfrom
feat/m0-scaffold

Conversation

@tom-snyder

Copy link
Copy Markdown
Member

Summary

The three Milestone 0 packets that need no external data and no inference — M0.1 (experiment register), M0.2 (Python eval harness), M0.7 (opengrep acquisition). Dispatched as three parallel workers with disjoint write scopes, then a critic. Every file landed inside its packet's declared scope; nothing under plan/ was touched.

The one property that matters

eval/register.yaml is what M0.18 reads to decide whether Anvil's detection-model tier exists at all. Two of its rows — EXP-01 (advisory-permutation ablation) and EXP-02 (code-metrics baseline) — can delete that tier entirely.

So one property is load-bearing above all others: a row whose experiment has not run must be impossible to confuse with one that passed. decision is required, non-nullable, with no empty-string and no absent representation. Every unrun row carries the explicit sentinel UNRESOLVED.

Two defects the critic found by mutation — both fixed here

1. The guard had a hole. "An unrun row stays UNRESOLVED" covered not_started/in_progress/blocked and omitted deferred. EXP-05 could be set to PASS and validation accepted it — precisely the "a missing decision reads as a pass" failure the rule exists to prevent. Fixed, and verified by mutating a deferred row and watching it get rejected.

2. Nothing ever ran the schema. The register was validated once, by hand, at authoring time. A schema nobody runs is a comment. eval/tests/test_register_schema.py now runs it, with one mutation test per unrun status, plus malformed-decision and deleted-row cases.

CI gains an eval-harness job, which closes the gap that let both of the above ship green: there was no Python job of any kind, so eval/'s tests could not fail a build no matter what they said.

Escalated, not worked around

M0.7 measured AikidoSec/opengrep-rules at the pinned commit and found TWO rules — both languages: [yaml], both restricted by paths.include to .github/workflows/**. Neither does taint analysis; neither examines application source in any language Anvil targets.

I confirmed this independently against the GitHub API: the repo is 8 KB, and rules/ holds exactly github_workflow_prompt_injection and npm_staged_publishing_missing.

00-SPINE.md §S4 assigns Lane B's source-recall tier to that corpus. If it stands, any target repo without GitHub Actions workflows yields zero candidates, and the adjudicator has nothing to adjudicate. This is a component-selection question for the spine, not something a worker may decide — so it is reported rather than patched around, and INSTR-01 should not be read as a recall measurement until it resolves.

Honest gaps

  • The opengrep end-to-end path is UNVERIFIED. Nothing was downloaded or executed (my scope limit). Its four tests SKIP with an explicit reason naming the missing artefact, and smoke.py exits 2 — a code reserved for "artefact absent", distinct from pass/fail — rather than reporting an empty result.
  • The cross-family critic did not run. openrouter/openai/gpt-oss-20b:free exited cleanly after 88 s having emitted zero tokens; the reroute also failed. 00-ROUTING.md makes cross-family critique a hard requirement for licence conclusions, so that gate is unfilled for M0.7's licence findings and is not being represented as satisfied.
  • Workers flagged their own unverified transcriptions (baselines taken from the plan's restatement rather than re-read from research/). Those are recorded in the packet reports.

Testing

pip install -e eval/     ok
pytest tests/ -q         24 passed
register validates       0 errors, 14/14 ids, each exactly once
mutation: deferred+PASS  REJECTED  (was ACCEPTED before this PR)

…isition

M0.1, M0.2 and M0.7 -- the Milestone 0 packets that need no external data and no
inference. Dispatched as three parallel workers with disjoint write scopes, then
a critic. Every file landed inside its packet's declared scope; nothing under
plan/ was touched.

M0.1 -- experiment register (eval/register.yaml + schema)
This is the artifact M0.18 reads to decide whether Anvil's detection-model tier
exists at all, so one property is load-bearing above all others: a row whose
experiment has not run must be impossible to confuse with one that passed.
`decision` is therefore required, non-nullable, with no empty-string and no
absent representation; every unrun row carries the explicit sentinel UNRESOLVED.
All fourteen rows present (EXP-01..EXP-12, INSTR-01, S12-RTT).

M0.2 -- pure-Python eval harness, src layout, PEP 621. eval/ carries no Go code:
S12 permits Python in exactly three places and this is one of them.

M0.7 -- opengrep acquisition machinery: pinned engine v1.26.0 at a tagged commit,
sha256-gated fetch, subprocess wrapper, fixtures. Per the orchestrator's scope
limit nothing was downloaded or executed, so the end-to-end smoke path is
UNVERIFIED and says so: its four tests SKIP with an explicit reason naming the
missing artefact, and smoke.py exits 2 -- a code reserved for "artefact absent"
and distinct from pass/fail -- rather than reporting an empty result.

Two defects the critic found by mutation, both fixed here

1. The register schema's "an unrun row stays UNRESOLVED" guard covered
   not_started/in_progress/blocked and omitted `deferred`. EXP-05 could be set
   to PASS and validation accepted it -- precisely the "a missing decision reads
   as a pass" failure the rule exists to prevent. Guard extended; verified by
   mutating a deferred row and watching it get rejected.

2. Nothing in the repository ever validated register.yaml against
   register.schema.json. The check had been run once, by hand, at authoring
   time. A schema nobody runs is a comment. eval/tests/test_register_schema.py
   now runs it, with one mutation test per unrun status asserting a PASS cannot
   be smuggled onto an unrun row, plus the malformed-decision and deleted-row
   cases.

CI gains an eval-harness job, closing the gap that let both of the above ship
green: there was no Python job of any kind, so eval/'s tests could not fail a
build no matter what they said.

Evidence: 24 tests pass locally after `pip install -e eval/`; register validates
with 0 errors; all fourteen ids present exactly once; mutation of a deferred row
to PASS is rejected.

Not resolved here, escalated instead: M0.7 measured AikidoSec/opengrep-rules at
the pinned commit and found TWO rules, both languages:[yaml], both restricted by
paths.include to .github/workflows/**. Independently confirmed against the
GitHub API: the repo is 8 KB, rules/ holds exactly those two directories. S4
assigns Lane B's source-recall tier to that corpus. This is a component-selection
question for the spine, not something a worker may decide, and it is reported to
the owner rather than worked around.
The job was added to close the critic finding that CI had no Python job at
all, so eval/ tests could not fail a build no matter what they said. Adding it
immediately found two things, which is the point.

1. `pip install -e .` does not bring pytest. Core deps are deliberately minimal
   -- pytest/ruff live in the [dev] extra, and the heavy [stats]/[models] extras
   belong to the packets that need them (M0.6, M0.10, M0.15). Install [dev].

2. ruff reported 11 E501 violations in worker-written code. Fixed by wrapping,
   not by relaxing the rule: five were long f-strings the formatter will not
   break on its own, so they were wrapped by hand.

Also added a guard the skip behaviour makes necessary. M0.7 deliberately ships
four tests that SKIP because nothing downloads the opengrep engine in CI, and
that is a legitimate outcome -- but a suite that silently degraded to all-skips
would still report green. The job now asserts the register mutation suite
actually reported passing tests.

Evidence: ruff clean; 58 passed, 4 skipped (the 4 being the UNVERIFIED opengrep
smoke path, each naming the missing artefact and the command to enable it).
@tom-snyder
tom-snyder merged commit fdebc0d into main Aug 7, 2026
4 checks passed
@tom-snyder
tom-snyder deleted the feat/m0-scaffold branch August 7, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant