Phase 1: M0 experiment register, eval harness scaffold, opengrep acquisition - #4
Merged
Conversation
…isition M0.1, M0.2 and M0.7 -- the Milestone 0 packets that need no external data and no inference. Dispatched as three parallel workers with disjoint write scopes, then a critic. Every file landed inside its packet's declared scope; nothing under plan/ was touched. M0.1 -- experiment register (eval/register.yaml + schema) This is the artifact M0.18 reads to decide whether Anvil's detection-model tier exists at all, so one property is load-bearing above all others: a row whose experiment has not run must be impossible to confuse with one that passed. `decision` is therefore required, non-nullable, with no empty-string and no absent representation; every unrun row carries the explicit sentinel UNRESOLVED. All fourteen rows present (EXP-01..EXP-12, INSTR-01, S12-RTT). M0.2 -- pure-Python eval harness, src layout, PEP 621. eval/ carries no Go code: S12 permits Python in exactly three places and this is one of them. M0.7 -- opengrep acquisition machinery: pinned engine v1.26.0 at a tagged commit, sha256-gated fetch, subprocess wrapper, fixtures. Per the orchestrator's scope limit nothing was downloaded or executed, so the end-to-end smoke path is UNVERIFIED and says so: its four tests SKIP with an explicit reason naming the missing artefact, and smoke.py exits 2 -- a code reserved for "artefact absent" and distinct from pass/fail -- rather than reporting an empty result. Two defects the critic found by mutation, both fixed here 1. The register schema's "an unrun row stays UNRESOLVED" guard covered not_started/in_progress/blocked and omitted `deferred`. EXP-05 could be set to PASS and validation accepted it -- precisely the "a missing decision reads as a pass" failure the rule exists to prevent. Guard extended; verified by mutating a deferred row and watching it get rejected. 2. Nothing in the repository ever validated register.yaml against register.schema.json. The check had been run once, by hand, at authoring time. A schema nobody runs is a comment. eval/tests/test_register_schema.py now runs it, with one mutation test per unrun status asserting a PASS cannot be smuggled onto an unrun row, plus the malformed-decision and deleted-row cases. CI gains an eval-harness job, closing the gap that let both of the above ship green: there was no Python job of any kind, so eval/'s tests could not fail a build no matter what they said. Evidence: 24 tests pass locally after `pip install -e eval/`; register validates with 0 errors; all fourteen ids present exactly once; mutation of a deferred row to PASS is rejected. Not resolved here, escalated instead: M0.7 measured AikidoSec/opengrep-rules at the pinned commit and found TWO rules, both languages:[yaml], both restricted by paths.include to .github/workflows/**. Independently confirmed against the GitHub API: the repo is 8 KB, rules/ holds exactly those two directories. S4 assigns Lane B's source-recall tier to that corpus. This is a component-selection question for the spine, not something a worker may decide, and it is reported to the owner rather than worked around.
The job was added to close the critic finding that CI had no Python job at all, so eval/ tests could not fail a build no matter what they said. Adding it immediately found two things, which is the point. 1. `pip install -e .` does not bring pytest. Core deps are deliberately minimal -- pytest/ruff live in the [dev] extra, and the heavy [stats]/[models] extras belong to the packets that need them (M0.6, M0.10, M0.15). Install [dev]. 2. ruff reported 11 E501 violations in worker-written code. Fixed by wrapping, not by relaxing the rule: five were long f-strings the formatter will not break on its own, so they were wrapped by hand. Also added a guard the skip behaviour makes necessary. M0.7 deliberately ships four tests that SKIP because nothing downloads the opengrep engine in CI, and that is a legitimate outcome -- but a suite that silently degraded to all-skips would still report green. The job now asserts the register mutation suite actually reported passing tests. Evidence: ruff clean; 58 passed, 4 skipped (the 4 being the UNVERIFIED opengrep smoke path, each naming the missing artefact and the command to enable it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The three Milestone 0 packets that need no external data and no inference — M0.1 (experiment register), M0.2 (Python eval harness), M0.7 (opengrep acquisition). Dispatched as three parallel workers with disjoint write scopes, then a critic. Every file landed inside its packet's declared scope; nothing under
plan/was touched.The one property that matters
eval/register.yamlis what M0.18 reads to decide whether Anvil's detection-model tier exists at all. Two of its rows — EXP-01 (advisory-permutation ablation) and EXP-02 (code-metrics baseline) — can delete that tier entirely.So one property is load-bearing above all others: a row whose experiment has not run must be impossible to confuse with one that passed.
decisionis required, non-nullable, with no empty-string and no absent representation. Every unrun row carries the explicit sentinelUNRESOLVED.Two defects the critic found by mutation — both fixed here
1. The guard had a hole. "An unrun row stays UNRESOLVED" covered
not_started/in_progress/blockedand omitteddeferred. EXP-05 could be set toPASSand validation accepted it — precisely the "a missing decision reads as a pass" failure the rule exists to prevent. Fixed, and verified by mutating a deferred row and watching it get rejected.2. Nothing ever ran the schema. The register was validated once, by hand, at authoring time. A schema nobody runs is a comment.
eval/tests/test_register_schema.pynow runs it, with one mutation test per unrun status, plus malformed-decision and deleted-row cases.CI gains an
eval-harnessjob, which closes the gap that let both of the above ship green: there was no Python job of any kind, soeval/'s tests could not fail a build no matter what they said.Escalated, not worked around
M0.7measuredAikidoSec/opengrep-rulesat the pinned commit and found TWO rules — bothlanguages: [yaml], both restricted bypaths.includeto.github/workflows/**. Neither does taint analysis; neither examines application source in any language Anvil targets.I confirmed this independently against the GitHub API: the repo is 8 KB, and
rules/holds exactlygithub_workflow_prompt_injectionandnpm_staged_publishing_missing.00-SPINE.md§S4 assigns Lane B's source-recall tier to that corpus. If it stands, any target repo without GitHub Actions workflows yields zero candidates, and the adjudicator has nothing to adjudicate. This is a component-selection question for the spine, not something a worker may decide — so it is reported rather than patched around, andINSTR-01should not be read as a recall measurement until it resolves.Honest gaps
smoke.pyexits 2 — a code reserved for "artefact absent", distinct from pass/fail — rather than reporting an empty result.openrouter/openai/gpt-oss-20b:freeexited cleanly after 88 s having emitted zero tokens; the reroute also failed.00-ROUTING.mdmakes cross-family critique a hard requirement for licence conclusions, so that gate is unfilled for M0.7's licence findings and is not being represented as satisfied.research/). Those are recorded in the packet reports.Testing