value)", path, line, text)
+ }
+ key := goldenKey(parts[0], parts[1])
+ if _, dup := g.values[key]; dup {
+ t.Fatalf("%s:%d: duplicate golden row for %s/%s", path, line, parts[0], parts[1])
+ }
+ g.values[key] = parts[2]
+ }
+ if err := sc.Err(); err != nil {
+ t.Fatalf("reading golden %s: %v", path, err)
+ }
+ if len(g.values) == 0 {
+ t.Fatalf("%s: golden has no rows; an empty golden asserts nothing", path)
+ }
+ return g
+}
+
+// digest returns the golden digest for label, validating its shape. A golden
+// carrying a truncated or uppercase digest would silently weaken the gate,
+// so it is rejected here rather than compared.
+func (g *goldenFile) digest(t *testing.T, label string) string {
+ t.Helper()
+ v, ok := g.values[goldenKey(goldenKindDigest, label)]
+ if !ok {
+ t.Fatalf("%s: no golden digest for %q; this case is unguarded", g.path, label)
+ }
+ g.seen[goldenKey(goldenKindDigest, label)] = true
+ if err := ValidateDigest(v); err != nil {
+ t.Fatalf("%s: golden digest for %q is malformed: %v", g.path, label, err)
+ }
+ return v
+}
+
+func (g *goldenFile) ordinal(t *testing.T, label string) int {
+ t.Helper()
+ v, ok := g.values[goldenKey(goldenKindOrdinal, label)]
+ if !ok {
+ t.Fatalf("%s: no golden ordinal for %q", g.path, label)
+ }
+ g.seen[goldenKey(goldenKindOrdinal, label)] = true
+ n, err := strconv.Atoi(v)
+ if err != nil || n < 0 {
+ t.Fatalf("%s: golden ordinal for %q is %q, want a non-negative base-10 integer", g.path, label, v)
+ }
+ return n
+}
+
+// assertFullyConsumed is the other half of the coverage guarantee: it fails if
+// the golden holds a row no assertion touched, which is what would happen if a
+// fixture case were deleted while its golden row stayed behind.
+func (g *goldenFile) assertFullyConsumed(t *testing.T) {
+ t.Helper()
+ var unused []string
+ for key := range g.values {
+ if !g.seen[key] {
+ parts := strings.SplitN(key, "\x00", 2)
+ unused = append(unused, parts[0]+"/"+parts[1])
+ }
+ }
+ if len(unused) > 0 {
+ sort.Strings(unused)
+ t.Errorf("%s: %d golden row(s) were never checked: %v\n"+
+ "Either a fixture case was removed without removing its golden row, or "+
+ "this test is skipping cases.", g.path, len(unused), unused)
+ }
+}
+
+func goldenPathFor(fixturePath string) string {
+ return strings.TrimSuffix(fixturePath, filepath.Ext(fixturePath)) + ".golden"
+}
+
+// ---------------------------------------------------------------------------
+// The main corpus: eight tier fixtures and every mutation
+// ---------------------------------------------------------------------------
+
+// TestConformanceMainCorpusMatchesIndependentGoldens is R.16's stop condition:
+// every corpus fixture's Go-computed digest equals its independently computed
+// golden, exactly, with no fixture skipped.
+func TestConformanceMainCorpusMatchesIndependentGoldens(t *testing.T) {
+ fixtures := loadCorpus(t)
+ if len(fixtures) == 0 {
+ t.Fatal("no fixtures loaded; the fixed corpus is mandatory (plan/00-SPINE.md S6)")
+ }
+
+ for _, f := range fixtures {
+ t.Run(f.ID, func(t *testing.T) {
+ g := loadGoldenFile(t, goldenPathFor(f.path))
+
+ // The field list first. A wrong field ORDER produces a valid-looking
+ // 64-hex digest that is silently incompatible; comparing the list
+ // says WHICH field moved instead of only that something did.
+ fields, err := fieldsFor(f.Tier, f.Input)
+ if err != nil {
+ t.Fatalf("building fields: %v", err)
+ }
+ if !reflect.DeepEqual(fields, f.HashedFields) {
+ t.Fatalf("hashed field list drifted from the hand-derived fixture.\n go: %q\n fixture: %q",
+ fields, f.HashedFields)
+ }
+
+ got, err := Digest(fields...)
+ if err != nil {
+ t.Fatalf("hashing: %v", err)
+ }
+ want := g.digest(t, "base")
+ if got != want {
+ t.Errorf("CONFORMANCE FAILURE: Go and the independent oracle disagree.\n"+
+ " go: %s\n oracle: %s\n"+
+ "Two producers emitting different digests breaks regression matching "+
+ "silently and forever (plan/00-SPINE.md S6). Do NOT re-seal the golden: "+
+ "a changed digest is an anvil-fp/v2 event with a dual-write migration "+
+ "(FINGERPRINT-SPEC.md section 0).", got, want)
+ }
+
+ // The fixture's own expected_digest and the oracle's golden were
+ // derived by two different routes and must agree. If they ever
+ // diverge, one of the two derivations is wrong and neither can be
+ // trusted as the reference.
+ if f.ExpectedDigest != want {
+ t.Errorf("the fixture's expected_digest and the oracle's golden disagree.\n"+
+ " fixture expected_digest: %s\n oracle golden: %s",
+ f.ExpectedDigest, want)
+ }
+
+ for _, m := range f.Mutations {
+ label := "mutation:" + m.Name
+ md, err := digestFor(f.Tier, m.Input)
+ if err != nil {
+ t.Fatalf("mutation %q: %v", m.Name, err)
+ }
+ if mw := g.digest(t, label); md != mw {
+ t.Errorf("mutation %q: Go %s, oracle %s\n%s", m.Name, md, mw, m.Description)
+ }
+ }
+
+ g.assertFullyConsumed(t)
+ })
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The derived corpus: values an implementation must COMPUTE, not be given
+// ---------------------------------------------------------------------------
+
+// jsonSastCandidateInput is deliberately NOT jsonSastInput: it has no `ordinal`
+// field, and it is decoded with DisallowUnknownFields. A derived fixture that
+// tried to supply a pre-computed ordinal therefore fails to decode, which is
+// the whole point of the Appendix Z4 corpus.
+type jsonSastCandidateInput struct {
+ TargetID string `json:"target_id"`
+ RuleIDVersioned string `json:"rule_id_versioned"`
+ RepoRelPath string `json:"repo_rel_path"`
+ EnclosingSymbolPath string `json:"enclosing_symbol_path"`
+ Snippet string `json:"snippet"`
+}
+
+type derivedCandidate struct {
+ Name string `json:"name"`
+ Description string `json:"description"`
+ Line int `json:"line,omitempty"`
+ Column int `json:"column,omitempty"`
+ Input json.RawMessage `json:"input"`
+
+ // ExpectedOrdinal is the ordinal the fixture author derived by hand from
+ // FINGERPRINT-SPEC.md section 4. It is a pointer so that "0" and "absent"
+ // are distinguishable — a missing ordinal on a SAST candidate must fail,
+ // not silently assert 0.
+ ExpectedOrdinal *int `json:"expected_ordinal,omitempty"`
+
+ // CanonicalRoute is the derived route the fixture expects, stated in the
+ // clear so a failure reads "segment X templated when it should not have"
+ // rather than "the digest moved".
+ CanonicalRoute string `json:"canonical_route,omitempty"`
+
+ HashedFields []string `json:"hashed_fields"`
+}
+
+type derivedFixture struct {
+ ID string `json:"id"`
+ Kind string `json:"kind"`
+ Resolves []string `json:"resolves"`
+ Description string `json:"description"`
+ Notes []string `json:"notes,omitempty"`
+ Candidates []derivedCandidate `json:"candidates"`
+
+ path string
+}
+
+const (
+ derivedKindSastOrdinalBatch = "sast_ordinal_batch"
+ derivedKindDastCases = "dast_cases"
+)
+
+func loadDerivedCorpus(t *testing.T) []derivedFixture {
+ t.Helper()
+
+ paths, err := filepath.Glob(filepath.Join(derivedCorpusDir, "*.json"))
+ if err != nil {
+ t.Fatalf("globbing %s: %v", derivedCorpusDir, err)
+ }
+ if len(paths) == 0 {
+ t.Fatalf("no fixtures in %s; FINGERPRINT-SPEC.md Appendix Z4 is then re-opened — "+
+ "the ordinal grouping key would be exercised by nothing at all", derivedCorpusDir)
+ }
+ sort.Strings(paths)
+
+ out := make([]derivedFixture, 0, len(paths))
+ for _, p := range paths {
+ b, err := os.ReadFile(p)
+ if err != nil {
+ t.Fatalf("reading %s: %v", p, err)
+ }
+ var f derivedFixture
+ if err := strictUnmarshal(b, &f); err != nil {
+ t.Fatalf("decoding %s: %v", p, err)
+ }
+ if f.ID == "" || f.Kind == "" || len(f.Candidates) == 0 || len(f.Resolves) == 0 {
+ t.Fatalf("%s: a derived fixture must declare id, kind, resolves and at least one candidate", p)
+ }
+ f.path = p
+ out = append(out, f)
+ }
+ return out
+}
+
+// TestConformanceDerivedCorpusMatchesIndependentGoldens drives the derived
+// corpus through the SAME two-sided comparison as the main one, but on values
+// the fixture withholds: the SAST ordinals are computed by AssignSastOrdinals
+// from a shuffled batch, and the DAST routes by CanonicalRouteTemplate from
+// concrete input.
+func TestConformanceDerivedCorpusMatchesIndependentGoldens(t *testing.T) {
+ for _, f := range loadDerivedCorpus(t) {
+ t.Run(f.ID, func(t *testing.T) {
+ g := loadGoldenFile(t, goldenPathFor(f.path))
+ switch f.Kind {
+ case derivedKindSastOrdinalBatch:
+ checkSastOrdinalBatch(t, f, g)
+ case derivedKindDastCases:
+ checkDastCases(t, f, g)
+ default:
+ t.Fatalf("%s: unknown derived fixture kind %q", f.path, f.Kind)
+ }
+ g.assertFullyConsumed(t)
+ })
+ }
+}
+
+func checkSastOrdinalBatch(t *testing.T, f derivedFixture, g *goldenFile) {
+ t.Helper()
+
+ cands := make([]SastCandidate, 0, len(f.Candidates))
+ for _, c := range f.Candidates {
+ var in jsonSastCandidateInput
+ if err := strictUnmarshal(c.Input, &in); err != nil {
+ t.Fatalf("%s: candidate %q: %v\n"+
+ "(A candidate that supplies a pre-computed `ordinal` fails here on purpose: "+
+ "FINGERPRINT-SPEC.md Appendix Z4 exists because every main-corpus SAST fixture "+
+ "does exactly that, leaving section 4 untested.)", f.path, c.Name, err)
+ }
+ if c.ExpectedOrdinal == nil {
+ t.Fatalf("%s: candidate %q has no expected_ordinal", f.path, c.Name)
+ }
+ cands = append(cands, SastCandidate{
+ Input: SastInput{
+ TargetID: in.TargetID,
+ RuleIDVersioned: in.RuleIDVersioned,
+ RepoRelPath: in.RepoRelPath,
+ EnclosingSymbolPath: in.EnclosingSymbolPath,
+ Snippet: in.Snippet,
+ },
+ Line: c.Line,
+ Column: c.Column,
+ })
+ }
+
+ // THE ordinal is derived here, from the batch, exactly as a producer must.
+ assigned, err := AssignSastOrdinals(cands)
+ if err != nil {
+ t.Fatalf("%s: AssignSastOrdinals: %v", f.path, err)
+ }
+ if len(assigned) != len(f.Candidates) {
+ t.Fatalf("%s: AssignSastOrdinals returned %d inputs for %d candidates",
+ f.path, len(assigned), len(f.Candidates))
+ }
+
+ for i, c := range f.Candidates {
+ label := "candidate:" + c.Name
+ got := assigned[i]
+
+ if got.Ordinal != *c.ExpectedOrdinal {
+ t.Errorf("%s: candidate %q: derived ordinal %d, fixture says %d\n%s",
+ f.path, c.Name, got.Ordinal, *c.ExpectedOrdinal, c.Description)
+ }
+ if wantOrd := g.ordinal(t, label); got.Ordinal != wantOrd {
+ t.Errorf("%s: candidate %q: Go derived ordinal %d, the independent oracle derived %d.\n"+
+ "The section 4 grouping key or its ordering rule is being read two different ways.",
+ f.path, c.Name, got.Ordinal, wantOrd)
+ }
+
+ fields, err := SastFields(got)
+ if err != nil {
+ t.Fatalf("%s: candidate %q: SastFields: %v", f.path, c.Name, err)
+ }
+ if !reflect.DeepEqual(fields, c.HashedFields) {
+ t.Fatalf("%s: candidate %q: hashed field list drifted from the hand-derived fixture.\n"+
+ " go: %q\n fixture: %q", f.path, c.Name, fields, c.HashedFields)
+ }
+
+ d, err := Digest(fields...)
+ if err != nil {
+ t.Fatalf("%s: candidate %q: hashing: %v", f.path, c.Name, err)
+ }
+ if want := g.digest(t, label); d != want {
+ t.Errorf("%s: candidate %q: CONFORMANCE FAILURE.\n go: %s\n oracle: %s",
+ f.path, c.Name, d, want)
+ }
+ }
+}
+
+func checkDastCases(t *testing.T, f derivedFixture, g *goldenFile) {
+ t.Helper()
+
+ for _, c := range f.Candidates {
+ label := "case:" + c.Name
+ var in jsonDastInput
+ if err := strictUnmarshal(c.Input, &in); err != nil {
+ t.Fatalf("%s: case %q: %v", f.path, c.Name, err)
+ }
+
+ if got := CanonicalRouteTemplate(in.RouteTemplate); got != c.CanonicalRoute {
+ t.Errorf("%s: case %q: CanonicalRouteTemplate(%q) = %q, fixture says %q\n%s",
+ f.path, c.Name, in.RouteTemplate, got, c.CanonicalRoute, c.Description)
+ }
+
+ fields, err := DastFields(DastInput{
+ TargetID: in.TargetID,
+ RuleIDVersioned: in.RuleIDVersioned,
+ HTTPMethod: in.HTTPMethod,
+ RouteTemplate: in.RouteTemplate,
+ InjectionPoint: InjectionPoint(in.InjectionPoint),
+ ParamName: in.ParamName,
+ EvidenceSignal: EvidenceSignal(in.EvidenceSignal),
+ })
+ if err != nil {
+ t.Fatalf("%s: case %q: DastFields: %v", f.path, c.Name, err)
+ }
+ if !reflect.DeepEqual(fields, c.HashedFields) {
+ t.Fatalf("%s: case %q: hashed field list drifted from the hand-derived fixture.\n"+
+ " go: %q\n fixture: %q", f.path, c.Name, fields, c.HashedFields)
+ }
+
+ d, err := Digest(fields...)
+ if err != nil {
+ t.Fatalf("%s: case %q: hashing: %v", f.path, c.Name, err)
+ }
+ if want := g.digest(t, label); d != want {
+ t.Errorf("%s: case %q: CONFORMANCE FAILURE.\n go: %s\n oracle: %s",
+ f.path, c.Name, d, want)
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The gate on the gate
+// ---------------------------------------------------------------------------
+
+// TestConformanceEveryFixtureHasAGolden is separate from the comparison tests
+// on purpose. Those tests iterate fixtures and would simply not run for a
+// fixture nobody added a golden for if the lookup were ever made lenient; this
+// one asserts the file exists, so "the corpus grew and the gate did not" is a
+// failure rather than an omission.
+func TestConformanceEveryFixtureHasAGolden(t *testing.T) {
+ var fixturePaths []string
+ for _, f := range loadCorpus(t) {
+ fixturePaths = append(fixturePaths, f.path)
+ }
+ for _, f := range loadDerivedCorpus(t) {
+ fixturePaths = append(fixturePaths, f.path)
+ }
+
+ for _, p := range fixturePaths {
+ gp := goldenPathFor(p)
+ if _, err := os.Stat(gp); err != nil {
+ t.Errorf("%s has no committed golden at %s: %v\n"+
+ "Run `python scripts/compute_golden_fingerprints.py --write` and review the "+
+ "result BY HAND against FINGERPRINT-SPEC.md before committing it.", p, gp, err)
+ }
+ }
+
+ goldens, err := filepath.Glob(filepath.Join(corpusDir, "*.golden"))
+ if err != nil {
+ t.Fatalf("globbing goldens: %v", err)
+ }
+ derivedGoldens, err := filepath.Glob(filepath.Join(derivedCorpusDir, "*.golden"))
+ if err != nil {
+ t.Fatalf("globbing derived goldens: %v", err)
+ }
+ if got, want := len(goldens)+len(derivedGoldens), len(fixturePaths); got != want {
+ t.Errorf("found %d golden files for %d fixtures; an orphaned golden guards nothing "+
+ "and a missing one leaves a fixture unguarded", got, want)
+ }
+}
+
+// TestConformanceDerivedCorpusClosesAppendixZ4 fails if the Appendix Z4 gap is
+// ever re-opened by deleting or gutting the ordinal batch. Z4 is the entry
+// Appendix Z itself calls "the one that matters most": the grouping key is what
+// keeps a finding's identity stable when an unrelated edit moves it, and before
+// this corpus existed it was exercised by nothing.
+func TestConformanceDerivedCorpusClosesAppendixZ4(t *testing.T) {
+ var batches int
+ for _, f := range loadDerivedCorpus(t) {
+ if f.Kind != derivedKindSastOrdinalBatch {
+ continue
+ }
+ batches++
+
+ var resolvesZ4 bool
+ for _, z := range f.Resolves {
+ if z == "Z4" {
+ resolvesZ4 = true
+ }
+ }
+ if !resolvesZ4 {
+ t.Errorf("%s: a %s fixture must declare that it resolves Z4", f.path, f.Kind)
+ }
+
+ // The properties that make this fixture actually exercise section 4,
+ // rather than being a batch of ten singleton groups that would pass
+ // with any grouping key at all.
+ var maxOrdinal, nonZero int
+ targets, rules, paths := map[string]bool{}, map[string]bool{}, map[string]bool{}
+ var shuffled bool
+ prevLine := -1
+ for _, c := range f.Candidates {
+ if c.ExpectedOrdinal == nil {
+ t.Fatalf("%s: candidate %q has no expected_ordinal", f.path, c.Name)
+ }
+ if *c.ExpectedOrdinal > maxOrdinal {
+ maxOrdinal = *c.ExpectedOrdinal
+ }
+ if *c.ExpectedOrdinal > 0 {
+ nonZero++
+ }
+ var in jsonSastCandidateInput
+ if err := strictUnmarshal(c.Input, &in); err != nil {
+ t.Fatalf("%s: candidate %q: %v", f.path, c.Name, err)
+ }
+ targets[in.TargetID] = true
+ rules[in.RuleIDVersioned] = true
+ paths[CanonicalRepoRelPath(in.RepoRelPath)] = true
+ if prevLine >= 0 && c.Line < prevLine {
+ shuffled = true
+ }
+ prevLine = c.Line
+ }
+
+ if maxOrdinal < 2 {
+ t.Errorf("%s: highest derived ordinal is %d; a group of at least three is needed "+
+ "before the ordering rule's line/column/batch-index tiers can all be distinguished",
+ f.path, maxOrdinal)
+ }
+ if nonZero < 2 {
+ t.Errorf("%s: only %d candidate(s) derive a non-zero ordinal; the batch is not "+
+ "exercising grouping", f.path, nonZero)
+ }
+ if len(targets) < 2 || len(rules) < 2 || len(paths) < 2 {
+ t.Errorf("%s: the batch varies %d target_id(s), %d rule(s) and %d canonical path(s); "+
+ "each component of the section 4 grouping key must be varied or a wrong key "+
+ "still passes", f.path, len(targets), len(rules), len(paths))
+ }
+ if !shuffled {
+ t.Errorf("%s: the batch is in ascending line order, so a producer that ignored the "+
+ "ordering rule entirely and numbered candidates in batch order would still pass",
+ f.path)
+ }
+ }
+ if batches == 0 {
+ t.Fatalf("no %s fixture in %s: FINGERPRINT-SPEC.md Appendix Z4 is re-opened",
+ derivedKindSastOrdinalBatch, derivedCorpusDir)
+ }
+}
+
+// TestConformanceOracleIsAnIndependentOfflineImplementation is the structural
+// half of R.16's "demonstrably two independent code paths" requirement. The
+// substantive half is that the oracle is a different LANGUAGE implementing a
+// specification document; what a test can check mechanically is that it has no
+// route back to the Go it gates.
+//
+// The check is deliberately narrow and honest about it: it enumerates the
+// script's imports and requires them to be a subset of an allowlist of pure
+// standard-library modules. `subprocess`, `ctypes`, `importlib` and anything
+// else that could invoke or load the implementation under test are therefore
+// excluded by construction rather than by grepping for banned words, which a
+// docstring mentioning them would trip.
+func TestConformanceOracleIsAnIndependentOfflineImplementation(t *testing.T) {
+ b, err := os.ReadFile(oracleScriptPath)
+ if err != nil {
+ t.Fatalf("reading the oracle at %s: %v\n"+
+ "R.16 requires the golden values to come from an offline re-implementation of "+
+ "FINGERPRINT-SPEC.md that is not this package. Without it the goldens have no "+
+ "provenance and the conformance gate is circular.", oracleScriptPath, err)
+ }
+ src := string(b)
+
+ if ext := filepath.Ext(oracleScriptPath); ext == ".go" {
+ t.Fatalf("the oracle must not be Go: %s", oracleScriptPath)
+ }
+ if !strings.Contains(src, "FINGERPRINT-SPEC.md") {
+ t.Errorf("%s never mentions FINGERPRINT-SPEC.md; the oracle must implement the "+
+ "specification document, not the summary in plan/40-record-and-storage.md, which "+
+ "CRITIQUE-01 finding 1 proved insufficient to reproduce the SAST goldens",
+ oracleScriptPath)
+ }
+
+ allowed := map[string]bool{
+ "__future__": true, "argparse": true, "hashlib": true, "json": true,
+ "re": true, "sys": true, "unicodedata": true, "pathlib": true,
+ }
+ // Matched against whole lines that are syntactically import statements, so
+ // prose in the module docstring ("... from its written specification ...")
+ // is not mistaken for one.
+ importRes := []*regexp.Regexp{
+ regexp.MustCompile(`(?m)^import\s+([A-Za-z_][A-Za-z0-9_.]*)\s*$`),
+ regexp.MustCompile(`(?m)^from\s+([A-Za-z_][A-Za-z0-9_.]*)\s+import\s`),
+ }
+ found := map[string]bool{}
+ for _, re := range importRes {
+ for _, m := range re.FindAllStringSubmatch(src, -1) {
+ mod := strings.SplitN(m[1], ".", 2)[0]
+ found[mod] = true
+ if !allowed[mod] {
+ t.Errorf("%s imports %q, which is not on the oracle's allowlist. The oracle must "+
+ "not be able to execute, load or read the implementation it gates — that is "+
+ "what makes the goldens independent evidence rather than a restatement of the Go.",
+ oracleScriptPath, mod)
+ }
+ }
+ }
+ if !found["hashlib"] {
+ t.Errorf("%s does not import hashlib; it cannot be computing SHA-256 itself", oracleScriptPath)
+ }
+
+ // A Go filename literal would mean the oracle opens Go source.
+ for _, bad := range []string{`.go"`, `.go'`} {
+ if strings.Contains(src, bad) {
+ t.Errorf("%s contains a Go filename literal (%s); the oracle must not read Go source",
+ oracleScriptPath, bad)
+ }
+ }
+}
+
+// TestConformanceGoldensAreNotDerivableFromThisPackage records, as an
+// executable note, that nothing in this package can regenerate a golden.
+// R.2's fingerprint_test.go carries the same prohibition in prose; this makes
+// the absence checkable, because the cheapest way to make a failing digest
+// green is always to re-seal, and the change that was supposed to be caught
+// then ships silently.
+func TestConformanceGoldensAreNotDerivableFromThisPackage(t *testing.T) {
+ // Scoped to the two files that touch the corpus and the goldens. It does
+ // NOT police the rest of the package: another test may have a legitimate
+ // reason to write a temporary file, and a gate that fires on unrelated work
+ // gets deleted rather than obeyed.
+ owners := []string{"fingerprint_test.go", "fingerprint_conformance_test.go"}
+ banned := regexp.MustCompile(`os\.(WriteFile|Create)\s*\(`)
+ for _, p := range owners {
+ b, err := os.ReadFile(p)
+ if err != nil {
+ t.Fatalf("reading %s: %v", p, err)
+ }
+ if loc := banned.FindString(string(b)); loc != "" {
+ t.Errorf("%s contains %q: neither the corpus lock nor the conformance gate may "+
+ "write a file. A golden this package can "+
+ "regenerate proves nothing — the cheapest way to make a changed digest green "+
+ "would be to re-seal, and the change this gate exists to catch would ship "+
+ "silently.", p, loc)
+ }
+ }
+}
+
+// TestConformanceCorpusCoverageIsMeaningful pins the total number of compared
+// values, so that a change which quietly reduces coverage — deleting a
+// mutation, dropping a fixture — has to be an explicit edit to this number
+// rather than an invisible loss of assertions.
+func TestConformanceCorpusCoverageIsMeaningful(t *testing.T) {
+ var digests, ordinals int
+ count := func(path string) {
+ g := loadGoldenFile(t, path)
+ for key := range g.values {
+ if strings.HasPrefix(key, goldenKindDigest+"\x00") {
+ digests++
+ } else if strings.HasPrefix(key, goldenKindOrdinal+"\x00") {
+ ordinals++
+ }
+ }
+ }
+ for _, f := range loadCorpus(t) {
+ count(goldenPathFor(f.path))
+ }
+ for _, f := range loadDerivedCorpus(t) {
+ count(goldenPathFor(f.path))
+ }
+
+ const (
+ minDigests = 81 // 8 base + 49 mutations + 10 batch candidates + 14 route cases
+ minOrdinals = 10 // one per Appendix Z4 batch candidate
+ )
+ if digests < minDigests {
+ t.Errorf("only %d digests are compared against the independent oracle, want at least %d; "+
+ "coverage went down", digests, minDigests)
+ }
+ if ordinals < minOrdinals {
+ t.Errorf("only %d derived ordinals are compared, want at least %d; FINGERPRINT-SPEC.md "+
+ "Appendix Z4 coverage went down", ordinals, minOrdinals)
+ }
+ t.Logf("anvil-fp/v1 conformance: %d digests and %d derived ordinals compared against "+
+ "%s", digests, ordinals, oracleScriptPath)
+}
diff --git a/internal/record/readpath.go b/internal/record/readpath.go
new file mode 100644
index 0000000..c55fd5c
--- /dev/null
+++ b/internal/record/readpath.go
@@ -0,0 +1,1267 @@
+// The three-tier read path the coding agent consumes (step R.13).
+//
+// # The three tiers, and why they are tiers
+//
+// research/18-unified-audit-record.md ("Size — the three-tier read path"):
+//
+// Tier 0 — Manifest (always read, target <= 8 KB). Audit metadata, per-half
+// seal status, counts, the deterministic read order, and inverted
+// indexes. Results are EXTERNALISED, not inlined.
+// Tier 1 — Task cards (the unit the agent actually reads, ~1,500–2,500
+// tokens each). One self-contained JSON per finding, DERIVED from
+// the record.
+// Tier 2 — Blobs (fetched on demand, content-addressed by `sha256:` digest).
+// Full response bodies, long taint paths, whole-file contents.
+//
+// The tiers exist because a 128k-context coding model that reads the whole
+// SARIF record reads nothing else. Tier 0 tells it what exists and in what
+// order to work; Tier 1 is what fits in its context; Tier 2 is what it fetches
+// only if it turns out to need it.
+//
+// # Budgets are enforced here, not documented here
+//
+// MaxTier0ManifestBytes and MaxTier1CardTokens are declared in contract.go.
+// This file MEASURES the marshalled bytes and refuses to emit anything over
+// budget: over-budget output degrades deterministically by spilling the
+// largest optional structures to Tier-2 blobs, in a fixed order, and only an
+// EXPLICIT, LOGGED override (Reader.AllowOversizeTier0 /
+// .AllowOversizeTier1) may produce an oversized tier. R.13's forbidden
+// actions: "Do not exceed the 8KB Tier-0 manifest budget or the
+// ~1,500–2,500 token Tier-1 card budget without an explicit, logged override."
+//
+// Nothing is ever silently dropped. Every shrink step records a Spill naming
+// the field and the `sha256:` reference to the bytes that left the tier.
+//
+// MEASURED, so the limit is a fact rather than a hope: the nine-finding
+// fixture in readpath_test.go produces a 3,008-byte manifest, and a
+// 409-finding record produces an 8,063-byte manifest — 98% of the budget —
+// with all four shrink steps taken, carrying the first 40 card refs inline and
+// the remaining 369 behind one Tier-2 reference. The crossover is around fifty
+// findings: beyond that the whole materialised read order stops fitting
+// alongside the envelope, and the tail (never the head) moves to Tier 2, where
+// TierSpill names it, counts it and content-addresses it. The read order is
+// never LOST, and it is never RE-DERIVED by the consumer; the part that does
+// not fit is fetched.
+//
+// # The read order is deterministic and is not the model's to choose
+//
+// DefaultReadOrder() — clusters, then SAST-only by rank, then DAST-only by
+// rank — is the only order this file emits. Correlated clusters come first
+// because they carry runtime proof; within every bucket the sort is total
+// (rank desc, evidence-class strength, finding id) so repeated calls on the
+// same input produce byte-identical output.
+//
+// # Two gates this file will not open
+//
+// 1. THE READ GATE. A half's results are readable only when its
+// `anvil/status` is HalfStatusSealed (R.6, and IMPLEMENTATION-PLAN.md §6
+// ruling G5: "`sealed` is load-bearing … the hard read gate") AND the
+// audit has not expired. BOTH ARMS, ASKED IN ONE PLACE: this file calls
+// sealing.go's HalfReadGate and never re-derives readability from
+// `run.Properties.Status`. See sealing.go's read-gate section for the four
+// separate bypasses that made that rule necessary.
+//
+// Cards are built only from readable halves. The manifest still REPORTS
+// the unreadable half — count, status and the gate's own refusal reason —
+// because a consumer that cannot see the half exists is a consumer that
+// reads "no DAST findings" as "no dynamic vulnerabilities", which is
+// research/23 Risk #1.
+//
+// 2. THE HOST GATE. plan/00-SPINE.md S7 makes the host agent read-only, "no
+// package manager in a mutating mode, not behind a flag", so
+// `remediable_by_agent` is false for every host finding
+// (IMPLEMENTATION-PLAN.md §6, S7). contract.go's validator enforces that
+// on the RECORD. This file enforces it again on the READ PATH, because
+// the record's validator is the producer's gate and a card is what the
+// agent actually receives: a host finding is never handed out as
+// actionable, even if a malformed record claims it is.
+//
+// # Masking is a precondition
+//
+// BuildTaskCards refuses a record that has not been through R.8's masker.
+// plan/00-SPINE.md S7 names the DAST response body "the highest-risk field —
+// up to 32 KB of attacker-controlled bytes fed to a repo-credentialed agent",
+// and the read path is precisely the step that does the feeding.
+//
+// Sources: research/18-unified-audit-record.md ("Size — the three-tier read
+// path", the annotated Tier-1 task card); research/24-coding-agent-consumption
+// .md ("What the audit record must carry"); plan/40-record-and-storage.md
+// (R.13); plan/00-SPINE.md S1, S6, S7.
+package record
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "sort"
+ "strings"
+ "sync"
+ "time"
+)
+
+// ---------------------------------------------------------------------------
+// Budgets and the token approximation
+// ---------------------------------------------------------------------------
+
+// ApproxBytesPerToken is the byte-to-token ratio this package uses to measure
+// a task card against MaxTier1CardTokens.
+//
+// It is DELIBERATELY PESSIMISTIC. research/18 records the budget as an
+// estimate rather than a measurement ("Token budget for task cards is an
+// estimate, not a measurement"), and R.13's expected output schema asks for
+// "a token-count approximation, not a hard requirement on an exact
+// tokenizer". Real BPE tokenizers land between 3 and 4 bytes per token on
+// dense JSON carrying source code; choosing 3 means this package's count is an
+// UPPER bound on every mainstream tokenizer, so a card that passes here passes
+// on the real one. Choosing 4 would have made the check optimistic, which is
+// the failure mode that matters: an under-counted card silently blows the
+// agent's context and the run degrades with no error anywhere.
+const ApproxBytesPerToken = 3
+
+// MaxTier1CardBytes is MaxTier1CardTokens expressed in bytes at
+// ApproxBytesPerToken. It lands at 7,500 bytes, just under research/18's
+// independent "target <= 8 KB each" figure for the same object — the two
+// numbers were derived from different sides of the same budget and agree,
+// which is the only reason to trust either.
+const MaxTier1CardBytes = MaxTier1CardTokens * ApproxBytesPerToken
+
+// MaxAdvisoryExcerptBytes is research/24's "<=800 tokens" advisory-excerpt cap
+// in bytes at the same ratio.
+const MaxAdvisoryExcerptBytes = MaxAdvisoryExcerptTokens * ApproxBytesPerToken
+
+// CardVersion is the task-card shape version, `cardVersion` in research/18's
+// annotated card. It is NOT the record's SchemaVersion: a card is derived, and
+// the projection may change shape without the record changing at all.
+const CardVersion = "1.0.0"
+
+// Default Tier-1 and Tier-2 path prefixes, written into Index.TaskCards and
+// Index.Blobs.
+const (
+ DefaultTaskCardPrefix = "cards/"
+ DefaultBlobPrefix = "blobs/"
+)
+
+// ApproxTokens converts a byte length to the approximate token count this
+// package budgets against. See ApproxBytesPerToken for why the ratio is
+// pessimistic.
+func ApproxTokens(byteLen int) int {
+ if byteLen <= 0 {
+ return 0
+ }
+ return (byteLen + ApproxBytesPerToken - 1) / ApproxBytesPerToken
+}
+
+// ---------------------------------------------------------------------------
+// Sources and sinks
+// ---------------------------------------------------------------------------
+
+// RecordSource supplies the assembled, masked record for an audit id.
+//
+// R.13 depends only on R.1 and R.2, so this file holds no database handle and
+// no store import: the store (R.4/R.5) satisfies this interface from the
+// outside, and so does a test. The dependency runs from the store to the read
+// path, never back.
+type RecordSource interface {
+ Record(auditID string) (*SARIFLog, error)
+}
+
+// RecordSourceFunc adapts a function to RecordSource.
+type RecordSourceFunc func(auditID string) (*SARIFLog, error)
+
+// Record implements RecordSource.
+func (f RecordSourceFunc) Record(auditID string) (*SARIFLog, error) { return f(auditID) }
+
+// RecordMap is an in-memory RecordSource keyed by audit id.
+type RecordMap map[string]*SARIFLog
+
+// Record implements RecordSource.
+func (m RecordMap) Record(auditID string) (*SARIFLog, error) {
+ l, ok := m[auditID]
+ if !ok || l == nil {
+ return nil, fmt.Errorf("record: no audit record for audit id %q", auditID)
+ }
+ return l, nil
+}
+
+// BlobSink persists one Tier-2 blob. Ref is the `sha256:<64 hex>` reference
+// written into the tier that spilled it.
+//
+// NewReader ALWAYS installs one (Reader.RetainedBlobs), so the default path
+// never produces a spill whose bytes nothing holds. A caller that owns real
+// Tier-2 storage replaces it. See Reader.Blobs.
+type BlobSink func(ref string, content []byte) error
+
+// ---------------------------------------------------------------------------
+// Errors
+// ---------------------------------------------------------------------------
+
+// BudgetError reports a tier that could not be brought under its budget by
+// spilling, and for which no explicit override was configured.
+type BudgetError struct {
+ Tier string // "tier-0 manifest" or "tier-1 task card"
+ Subject string // audit id or finding id
+ Bytes int
+ Budget int
+ Tokens int
+ MaxTok int
+}
+
+func (e *BudgetError) Error() string {
+ return fmt.Sprintf(
+ "record: %s for %q is %d bytes (~%d tokens) after every shrink step, over the %d-byte (%d-token) budget; "+
+ "set Reader.AllowOversize with a reason to emit it anyway (R.13 requires the override to be explicit and logged)",
+ e.Tier, e.Subject, e.Bytes, e.Tokens, e.Budget, e.MaxTok)
+}
+
+// ---------------------------------------------------------------------------
+// Tier 0 — the manifest
+// ---------------------------------------------------------------------------
+
+// Manifest is Tier 0: the first and only thing the coding agent reads before
+// it decides what to read next. Target <= MaxTier0ManifestBytes.
+//
+// It is DERIVED. Nothing here is authoritative; the record is. Where a
+// manifest and the record disagree, the record wins and the manifest is a
+// stale projection to be rebuilt.
+type Manifest struct {
+ // ManifestVersion is the shape version of this projection, independent of
+ // the record's SchemaVersion.
+ ManifestVersion string `json:"manifestVersion"`
+
+ SchemaVersion string `json:"anvil/schemaVersion"`
+ AuditID string `json:"anvil/auditId"`
+ State State `json:"anvil/state"`
+ Version int `json:"anvil/version"`
+ CreatedAt string `json:"anvil/createdAt"`
+
+ Target ManifestTarget `json:"anvil/target"`
+ Deadline ManifestClaim `json:"anvil/deadline"`
+
+ // DastStatus is carried verbatim and is never absent. DynamicallyScanned
+ // Clean is DastStatus.MeansDynamicallyScannedClean() precomputed, so a
+ // consumer cannot arrive at "no dynamic vulnerabilities" by testing
+ // `dastStatus != "completed_findings"` — the naive check research/23
+ // Risk #1 exists to prevent.
+ DastStatus DastStatus `json:"anvil/dastStatus"`
+ DynamicallyScannedClean bool `json:"anvil/dynamicallyScannedClean"`
+
+ // Halves reports BOTH halves, readable or not. An unreadable half is
+ // reported with its result count so that "0 cards from the DAST half" and
+ // "the DAST half has not sealed" are visibly different observations.
+ Halves []ManifestHalf `json:"anvil/halves"`
+
+ // Index is contract.go's Tier-0 index: counts, the deterministic read
+ // order, and the inverted indexes. Any index dropped to stay under budget
+ // is null here and named in Spills.
+ Index Index `json:"anvil/index"`
+
+ // Cards is the read order, materialised: one entry per emitted task card,
+ // in the exact order BuildTaskCards returns them.
+ Cards []CardRef `json:"anvil/cards"`
+
+ // Spills names every structure moved out of this tier to stay under
+ // budget, with the `sha256:` reference to its bytes. Never empty when
+ // something was dropped, and nothing is ever dropped without an entry.
+ Spills []TierSpill `json:"anvil/spills,omitempty"`
+
+ // Override is non-nil only when the manifest exceeded its budget after
+ // every shrink step AND the caller explicitly authorised that.
+ Override *BudgetOverride `json:"anvil/budgetOverride,omitempty"`
+
+ // Bytes and Tokens are what this manifest measured at.
+ Bytes int `json:"anvil/manifestBytes"`
+ Tokens int `json:"anvil/manifestTokens"`
+
+ // Blobs are the Tier-2 bytes this manifest spilled, keyed by reference.
+ // NOT serialised: they are the thing that left the tier, and writing them
+ // back into it would defeat the spill.
+ Blobs map[string][]byte `json:"-"`
+}
+
+// ManifestTarget is the trimmed `anvil/target`. Both G4+G7 fields survive:
+// provenance (what happened when we tried to run the target) and provisioning
+// (which path produced one) are different measurements and the agent needs
+// both to know what it is looking at.
+type ManifestTarget struct {
+ RepoURL string `json:"repoUrl"`
+ Ref string `json:"ref"`
+ Commit string `json:"commit"`
+ Subpath string `json:"subpath,omitempty"`
+ RuntimeBaseURL string `json:"runtimeBaseUrl,omitempty"`
+ Provenance TargetProvenance `json:"provenance"`
+ Provisioning TargetProvisioning `json:"provisioning"`
+}
+
+// ManifestClaim is the claim clock, flattened. DeadlineAt is the claim
+// timeout, never a retention or confidentiality guarantee (SECRETS.md).
+type ManifestClaim struct {
+ DeadlineAt string `json:"deadlineAt"`
+ ClaimTimeoutSeconds int `json:"claimTimeoutSeconds"`
+}
+
+// ManifestHalf is one half's seal state as the agent must see it.
+type ManifestHalf struct {
+ Half Half `json:"half"`
+ Status HalfStatus `json:"status"`
+ SealedAt string `json:"sealedAt,omitempty"`
+
+ // Readable is sealing.go's HalfReadGate, and is never re-derived here: a
+ // half is readable when its status is exactly HalfStatusSealed AND the
+ // audit has not expired. A half may be TERMINAL without being READABLE — a
+ // skipped DAST half is finished and unreadable at once — and a cleanly
+ // SEALED half is unreadable once the claim window closes, because the
+ // reaper has dropped the payload the cards would be built from.
+ Readable bool `json:"readable"`
+
+ // ReadRefusal is the gate's own reason, present exactly when Readable is
+ // false. It is what keeps "the half never sealed" and "the audit expired
+ // holding a sealed half" from arriving at the consumer as the same
+ // observation — the manifest already reports Status and the envelope
+ // already reports State, but a consumer should not have to join them.
+ ReadRefusal string `json:"readRefusal,omitempty"`
+
+ // Results is how many results the half carries in the record, whether or
+ // not any card was emitted for them.
+ Results int `json:"results"`
+
+ // Cards is how many cards were emitted from this half. It is 0 whenever
+ // Readable is false.
+ Cards int `json:"cards"`
+
+ Tool string `json:"tool,omitempty"`
+
+ // Coverage is the DAST half's probed/inventory pair. Carried as the pair
+ // and never as a bare ratio, for the reason DastCoverage documents.
+ Coverage *ManifestCoverage `json:"coverage,omitempty"`
+}
+
+// ManifestCoverage is DastCoverage reduced to what Tier 0 can afford.
+type ManifestCoverage struct {
+ ProbedCount int `json:"probedCount"`
+ InventoryUnionCount int `json:"inventoryUnionCount"`
+ EndpointCoverage float64 `json:"endpointCoverage"`
+}
+
+// CardRef is one entry in the read order.
+type CardRef struct {
+ FindingID string `json:"findingId"`
+
+ // Card is the Tier-1 path, Index.TaskCards + a filesystem-safe form of
+ // FindingID.
+ Card string `json:"card"`
+
+ // Bucket is which of DefaultReadOrder()'s three buckets this finding came
+ // from. It is not decoration: it is how a consumer verifies the order it
+ // was handed is the order R.13 promises.
+ Bucket string `json:"bucket"`
+
+ Half Half `json:"half"`
+ EvidenceClass EvidenceClass `json:"evidenceClass"`
+ Rank float64 `json:"rank"`
+ ClusterID string `json:"clusterId,omitempty"`
+
+ // Actionable is the coding agent's gate. False for every host finding,
+ // always — see this file's header, gate 2.
+ Actionable bool `json:"actionable"`
+}
+
+// TierSpill is one structure moved out of a tier to a Tier-2 blob.
+type TierSpill struct {
+ // Field names what left, in the tier's own vocabulary.
+ Field string `json:"field"`
+ // Ref is the `sha256:<64 lowercase hex>` reference to the bytes.
+ Ref string `json:"ref"`
+ // Bytes is the length of the spilled JSON. Items, where the spilled
+ // structure is a collection, is how many entries it held.
+ Bytes int `json:"bytes"`
+ Items int `json:"items,omitempty"`
+}
+
+// BudgetOverride records an explicit decision to emit an over-budget tier. It
+// exists so that "we exceeded the budget" is a fact in the artifact rather
+// than a silence.
+type BudgetOverride struct {
+ Reason string `json:"reason"`
+ Bytes int `json:"bytes"`
+ Budget int `json:"budget"`
+}
+
+// ---------------------------------------------------------------------------
+// The reader
+// ---------------------------------------------------------------------------
+
+// Reader builds the three tiers from a RecordSource. The zero value is not
+// usable — it has no source; use NewReader.
+type Reader struct {
+ // Source resolves an audit id to its assembled, masked record.
+ Source RecordSource
+
+ // TaskCardPrefix and BlobPrefix are the Tier-1 and Tier-2 path prefixes
+ // written into Index. Empty means the defaults.
+ TaskCardPrefix string
+ BlobPrefix string
+
+ // Blobs is called for every Tier-2 spill. NewReader installs the Reader's
+ // own in-memory retainer here; a caller with real Tier-2 storage replaces
+ // it, and only a caller that has deliberately set it to nil gets the old
+ // behaviour of a spill with nowhere to land.
+ //
+ // WHY THE DEFAULT IS NOT NIL (CRITIQUE-03 M3, consequence 2). The spilled
+ // bytes are returned in Manifest.Blobs / TaskCard.Blobs, both of which are
+ // `json:"-"`. A caller that marshals the manifest and drops the struct —
+ // the obvious thing to do with a projection — therefore shipped a Tier-0
+ // manifest whose most load-bearing content, the materialised read order,
+ // was a dangling `sha256:` reference. The hazard was documented; the
+ // default walked straight into it. It now takes an explicit `rd.Blobs =
+ // nil` to reach.
+ Blobs BlobSink
+
+ // retained backs the default sink. It is per-Reader and unbounded, which
+ // is why DrainRetainedBlobs exists: a long-lived Reader projecting many
+ // audits should drain after each one.
+ //
+ // It is a POINTER to a locked struct rather than a map plus a mutex field,
+ // for two reasons: a Reader stays copyable (a mutex field would make every
+ // `rd := *other` a vet copylocks error), and a Reader shared by two
+ // goroutines building two audits' tiers cannot race on the retainer. Every
+ // other field of Reader is configuration, written once before use.
+ retained *blobRetainer
+
+ // AllowOversizeTier0 and AllowOversizeTier1 are the explicit, logged
+ // overrides R.13 requires before an over-budget tier may be emitted. The
+ // string IS the log: it is the reason, it is recorded in the emitted
+ // tier's BudgetOverride, and an empty string means "not authorised", which
+ // is the default.
+ AllowOversizeTier0 string
+ AllowOversizeTier1 string
+
+ // RequireMasked defaults to true via NewReader. Setting it false skips
+ // AssertMasked, which is only ever correct for a caller that has already
+ // run it.
+ RequireMasked bool
+}
+
+// NewReader returns a Reader over src with the safe defaults: masking
+// required, no oversize override authorised, default tier prefixes, and a
+// Tier-2 blob sink that retains every spill on the Reader.
+//
+// The sink is a default, not a policy: a caller with durable Tier-2 storage
+// assigns its own BlobSink to Reader.Blobs and this one is never called.
+func NewReader(src RecordSource) *Reader {
+ rd := &Reader{
+ Source: src,
+ TaskCardPrefix: DefaultTaskCardPrefix,
+ BlobPrefix: DefaultBlobPrefix,
+ RequireMasked: true,
+ retained: &blobRetainer{blobs: map[string][]byte{}},
+ }
+ rd.Blobs = rd.retained.put
+ return rd
+}
+
+// blobRetainer is the in-memory Tier-2 store behind NewReader's default sink.
+type blobRetainer struct {
+ mu sync.Mutex
+ blobs map[string][]byte
+}
+
+// put is the default BlobSink. It never fails: an in-memory map cannot refuse
+// a write, and a sink that could would make the default path able to fail in a
+// way the caller did not ask for.
+func (r *blobRetainer) put(ref string, content []byte) error {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ if r.blobs == nil {
+ r.blobs = map[string][]byte{}
+ }
+ r.blobs[ref] = content
+ return nil
+}
+
+func (r *blobRetainer) snapshot(drain bool) map[string][]byte {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ out := make(map[string][]byte, len(r.blobs))
+ for ref, content := range r.blobs {
+ out[ref] = content
+ }
+ if drain {
+ r.blobs = map[string][]byte{}
+ }
+ return out
+}
+
+// RetainedBlobs returns every Tier-2 blob this Reader's default sink has
+// retained, keyed by `sha256:` reference.
+//
+// It is a copy of the map (the byte slices are shared, and are never mutated
+// after a spill), so a caller can iterate it while building another tier.
+// Empty when the caller supplied its own BlobSink — the Reader retains nothing
+// it did not write.
+func (rd *Reader) RetainedBlobs() map[string][]byte {
+ if rd == nil || rd.retained == nil {
+ return map[string][]byte{}
+ }
+ return rd.retained.snapshot(false)
+}
+
+// DrainRetainedBlobs returns RetainedBlobs and forgets them, so a Reader used
+// for many audits does not accumulate every blob it ever spilled.
+func (rd *Reader) DrainRetainedBlobs() map[string][]byte {
+ if rd == nil || rd.retained == nil {
+ return map[string][]byte{}
+ }
+ return rd.retained.snapshot(true)
+}
+
+func (rd *Reader) cardPrefix() string {
+ if rd.TaskCardPrefix == "" {
+ return DefaultTaskCardPrefix
+ }
+ return rd.TaskCardPrefix
+}
+
+func (rd *Reader) blobPrefix() string {
+ if rd.BlobPrefix == "" {
+ return DefaultBlobPrefix
+ }
+ return rd.BlobPrefix
+}
+
+func (rd *Reader) load(auditID string) (*SARIFLog, error) {
+ if rd == nil || rd.Source == nil {
+ return nil, fmt.Errorf("record: Reader has no RecordSource; use NewReader")
+ }
+ l, err := rd.Source.Record(auditID)
+ if err != nil {
+ return nil, err
+ }
+ if l == nil {
+ return nil, fmt.Errorf("record: RecordSource returned a nil record for audit id %q", auditID)
+ }
+ if l.Properties.AuditID != auditID {
+ return nil, fmt.Errorf("record: RecordSource returned audit %q for audit id %q; "+
+ "the audit identity is the join key and a mismatch means the wrong record was fetched",
+ l.Properties.AuditID, auditID)
+ }
+ if rd.RequireMasked {
+ if err := AssertMasked(l); err != nil {
+ return nil, fmt.Errorf("record: refusing to build the read path for audit %q: %w "+
+ "(00-SPINE.md S7: the read path feeds a repo-credentialed agent; masking is R.8's step and runs before this one)",
+ auditID, err)
+ }
+ }
+ return l, nil
+}
+
+// BuildManifest builds the Tier-0 manifest for auditID.
+//
+// R.13's expected output schema names `BuildManifest(auditID string)
+// (Manifest, error)`. It is a method rather than a package-level function
+// because the record has to come from somewhere and the alternative — a
+// package-level default source — is exactly the kind of hidden global state
+// that makes two callers disagree about which record they read.
+func (rd *Reader) BuildManifest(auditID string) (Manifest, error) {
+ l, err := rd.load(auditID)
+ if err != nil {
+ return Manifest{}, err
+ }
+ return rd.ManifestFromLog(l)
+}
+
+// BuildTaskCards builds the Tier-1 task cards for auditID, in the
+// deterministic read order.
+func (rd *Reader) BuildTaskCards(auditID string) ([]TaskCard, error) {
+ l, err := rd.load(auditID)
+ if err != nil {
+ return nil, err
+ }
+ return rd.CardsFromLog(l)
+}
+
+// ---------------------------------------------------------------------------
+// The deterministic read order
+// ---------------------------------------------------------------------------
+
+// Read-order bucket names. These are the members of DefaultReadOrder(), which
+// contract.go owns; TestReadOrderBucketsMatchTheContract asserts the two
+// cannot drift.
+const (
+ BucketClusters = "clusters"
+ BucketSastByRank = "sastByRank"
+ BucketDastByRank = "dastByRank"
+)
+
+// orderedResult is one result in read order, with the coordinates needed to
+// point back into the record.
+type orderedResult struct {
+ runIndex int
+ resultIndex int
+ run *Run
+ result *Result
+ bucket string
+ clusterID string
+ rank float64
+}
+
+// evidenceClassStrength returns the index of e in EvidenceClassValues(), which
+// contract.go documents as "descending evidence strength — which is also the
+// default rank order". Lower is stronger. An unknown class sorts last rather
+// than panicking: the read path is not the place to reject a record.
+func evidenceClassStrength(e EvidenceClass) int {
+ for i, v := range EvidenceClassValues() {
+ if v == e {
+ return i
+ }
+ }
+ return len(EvidenceClassValues())
+}
+
+// resultRank reads SARIF's `result.rank` — PRIORITY, not confidence and not
+// severity (contract.go, ResultProperties.Confidence). An absent rank sorts
+// below every present one instead of defaulting to 0, which would silently
+// promote unranked findings past genuinely rank-0 ones.
+func resultRank(r *Result) float64 {
+ if r.Rank != nil {
+ return *r.Rank
+ }
+ return -1
+}
+
+// IsHostFinding reports whether r is a host-package finding, by EITHER of the
+// two fields that can say so.
+//
+// Both are checked because they are set by different producers and a record in
+// which only one says "host" is a record whose host-ness is still true. The
+// read path's job is to never hand such a finding to an agent that cannot fix
+// it (plan/00-SPINE.md S7: the host agent is read-only).
+func IsHostFinding(r *Result) bool {
+ return r.Properties.Detector.Kind == DetectorKindHost ||
+ r.Properties.EvidenceClass == EvidenceClassHost
+}
+
+// readOrder returns every READABLE result in the one order R.13 permits:
+// correlated clusters first, then SAST-only by rank, then DAST-only by rank.
+//
+// Results from a half the read gate refuses are omitted entirely. THE GATE IS
+// sealing.go's HalfReadGate AND NOTHING ELSE. This function used to ask
+// IsReadableHalfStatus(run.Properties.Status) directly, which is only the
+// status arm; CRITIQUE-03 M1 reproduced the consequence — an EXPIRED audit
+// yielded nine cards, six of them actionable, against a claim window that had
+// already closed and handoff rows already subject to ReclaimExpired.
+func (rd *Reader) readOrder(l *SARIFLog) []orderedResult {
+ var clustered, sastOnly, dastOnly []orderedResult
+
+ for ri := range l.Runs {
+ run := &l.Runs[ri]
+ if HalfReadGate(l.Properties.AuditID, halfSealOfRun(l, run)) != nil {
+ continue
+ }
+ for si := range run.Results {
+ res := &run.Results[si]
+ o := orderedResult{
+ runIndex: ri,
+ resultIndex: si,
+ run: run,
+ result: res,
+ rank: resultRank(res),
+ }
+ switch {
+ case res.Properties.Correlation != nil:
+ o.bucket = BucketClusters
+ o.clusterID = res.Properties.Correlation.ClusterID
+ clustered = append(clustered, o)
+ case res.Properties.Half == HalfDast:
+ o.bucket = BucketDastByRank
+ dastOnly = append(dastOnly, o)
+ default:
+ o.bucket = BucketSastByRank
+ sastOnly = append(sastOnly, o)
+ }
+ }
+ }
+
+ out := make([]orderedResult, 0, len(clustered)+len(sastOnly)+len(dastOnly))
+ out = append(out, orderClusters(clustered)...)
+ sortByRank(sastOnly)
+ out = append(out, sastOnly...)
+ sortByRank(dastOnly)
+ out = append(out, dastOnly...)
+ return out
+}
+
+// sortByRank is the total order inside a bucket: rank descending, then
+// evidence strength, then finding id. The finding-id tie-break is what makes
+// the order STABLE rather than merely deterministic-for-this-input — two
+// findings with identical rank and class still cannot swap between calls.
+func sortByRank(rs []orderedResult) {
+ sort.SliceStable(rs, func(i, j int) bool { return lessByRank(rs[i], rs[j]) })
+}
+
+func lessByRank(a, b orderedResult) bool {
+ if a.rank != b.rank {
+ return a.rank > b.rank
+ }
+ as, bs := evidenceClassStrength(a.result.Properties.EvidenceClass),
+ evidenceClassStrength(b.result.Properties.EvidenceClass)
+ if as != bs {
+ return as < bs
+ }
+ return a.result.Properties.FindingID < b.result.Properties.FindingID
+}
+
+// orderClusters groups the clustered results by cluster id and emits whole
+// clusters, strongest cluster first.
+//
+// A cluster is emitted CONTIGUOUSLY and never merged: both members survive as
+// separate cards, because the SAST finding owns the file and line and the DAST
+// finding owns the proof (research/18, "link, never merge"). Within a cluster
+// the SAST member comes first — it is the one carrying the code the agent has
+// to edit.
+func orderClusters(rs []orderedResult) []orderedResult {
+ if len(rs) == 0 {
+ return nil
+ }
+ byCluster := map[string][]orderedResult{}
+ for _, o := range rs {
+ byCluster[o.clusterID] = append(byCluster[o.clusterID], o)
+ }
+
+ ids := make([]string, 0, len(byCluster))
+ for id := range byCluster {
+ ids = append(ids, id)
+ members := byCluster[id]
+ sort.SliceStable(members, func(i, j int) bool {
+ a, b := members[i], members[j]
+ if a.result.Properties.Half != b.result.Properties.Half {
+ return a.result.Properties.Half == HalfSast
+ }
+ return lessByRank(a, b)
+ })
+ byCluster[id] = members
+ }
+
+ // Cluster order: best member's rank descending, then cluster id. Ranging
+ // over the map above collected the ids in map order; this sort is what
+ // makes the result deterministic, and it is a total order because cluster
+ // ids are unique keys. The best rank is precomputed rather than derived
+ // inside the comparator, so the comparator is a pure comparison and cannot
+ // be quadratic in cluster size.
+ best := make(map[string]float64, len(byCluster))
+ for id, members := range byCluster {
+ top := members[0].rank
+ for _, m := range members {
+ if m.rank > top {
+ top = m.rank
+ }
+ }
+ best[id] = top
+ }
+ sort.SliceStable(ids, func(i, j int) bool {
+ bi, bj := best[ids[i]], best[ids[j]]
+ if bi != bj {
+ return bi > bj
+ }
+ return ids[i] < ids[j]
+ })
+
+ out := make([]orderedResult, 0, len(rs))
+ for _, id := range ids {
+ out = append(out, byCluster[id]...)
+ }
+ return out
+}
+
+// ---------------------------------------------------------------------------
+// Tier 0 assembly
+// ---------------------------------------------------------------------------
+
+// ManifestFromLog builds the Tier-0 manifest from an already-loaded record.
+func (rd *Reader) ManifestFromLog(l *SARIFLog) (Manifest, error) {
+ if l == nil {
+ return Manifest{}, fmt.Errorf("record: ManifestFromLog got a nil *SARIFLog")
+ }
+ p := &l.Properties
+ order := rd.readOrder(l)
+ clusters := clustersOf(order)
+
+ m := Manifest{
+ ManifestVersion: CardVersion,
+ SchemaVersion: p.SchemaVersion,
+ AuditID: p.AuditID,
+ State: p.State,
+ Version: p.Version,
+ CreatedAt: formatTime(p.CreatedAt),
+ Target: ManifestTarget{
+ RepoURL: p.Target.RepoURL,
+ Ref: p.Target.Ref,
+ Commit: p.Target.Commit,
+ Subpath: p.Target.Subpath,
+ RuntimeBaseURL: p.Target.RuntimeBaseURL,
+ Provenance: p.Target.Provenance,
+ Provisioning: p.Target.Provisioning,
+ },
+ Deadline: ManifestClaim{
+ DeadlineAt: formatTime(p.Deadline.DeadlineAt),
+ ClaimTimeoutSeconds: p.Deadline.ClaimTimeoutSeconds,
+ },
+ DastStatus: p.DastStatus,
+ DynamicallyScannedClean: p.DastStatus.MeansDynamicallyScannedClean(),
+ Blobs: map[string][]byte{},
+ }
+
+ cardsPerHalf := map[Half]int{}
+ counts := IndexCounts{}
+ byCluster := map[string][]string{}
+ byCwe := map[string][]string{}
+ byPath := map[string][]string{}
+ clusterSeen := map[string]bool{}
+
+ for _, o := range order {
+ fid := o.result.Properties.FindingID
+ actionable, _ := cardActionable(o.result, clusters[o.clusterID])
+ m.Cards = append(m.Cards, CardRef{
+ FindingID: fid,
+ Card: rd.cardPath(fid),
+ Bucket: o.bucket,
+ Half: o.result.Properties.Half,
+ EvidenceClass: o.result.Properties.EvidenceClass,
+ Rank: o.rank,
+ ClusterID: o.clusterID,
+ // cardActionable, never isActionable: the manifest's read order and
+ // the card must agree, and they only agree if they ask one
+ // function. TestManifestReadOrderMatchesTheCards asserts it.
+ Actionable: actionable,
+ })
+ cardsPerHalf[o.result.Properties.Half]++
+ counts.Total++
+ switch o.result.Properties.Half {
+ case HalfSast:
+ counts.Sast++
+ case HalfDast:
+ counts.Dast++
+ }
+ if o.clusterID != "" {
+ byCluster[o.clusterID] = append(byCluster[o.clusterID], fid)
+ if !clusterSeen[o.clusterID] {
+ clusterSeen[o.clusterID] = true
+ counts.Clusters++
+ }
+ } else {
+ counts.Unclustered++
+ }
+ for _, taxon := range taxonIDs(o.result) {
+ byCwe[taxon] = append(byCwe[taxon], fid)
+ }
+ if path := primaryPath(o.result); path != "" {
+ byPath[path] = append(byPath[path], fid)
+ }
+ }
+
+ for ri := range l.Runs {
+ run := &l.Runs[ri]
+ // The SAME gate readOrder applied, so a half can never be reported
+ // readable while contributing no cards, or vice versa.
+ seal := halfSealOfRun(l, run)
+ h := ManifestHalf{
+ Half: run.Properties.Half,
+ Status: run.Properties.Status,
+ Readable: seal.Readable(),
+ ReadRefusal: halfReadRefusal(seal),
+ Results: len(run.Results),
+ Cards: cardsPerHalf[run.Properties.Half],
+ Tool: run.Tool.Driver.Name,
+ }
+ if run.Properties.SealedAt != nil {
+ h.SealedAt = formatTime(*run.Properties.SealedAt)
+ }
+ if c := run.Properties.DastCoverage; c != nil {
+ h.Coverage = &ManifestCoverage{
+ ProbedCount: c.ProbedCount,
+ InventoryUnionCount: c.InventoryUnionCount,
+ EndpointCoverage: c.EndpointCoverage,
+ }
+ }
+ m.Halves = append(m.Halves, h)
+ }
+
+ m.Index = Index{
+ Counts: counts,
+ ReadOrder: DefaultReadOrder(),
+ ByCluster: emptyToNil(byCluster),
+ ByCwe: emptyToNil(byCwe),
+ ByPath: emptyToNil(byPath),
+ TaskCards: rd.cardPrefix(),
+ Blobs: rd.blobPrefix(),
+ }
+
+ if err := rd.fitManifest(&m); err != nil {
+ return Manifest{}, err
+ }
+ return m, nil
+}
+
+// fitManifest brings the manifest under MaxTier0ManifestBytes by spilling, in
+// a FIXED order, the structures that can be reconstructed from Tier 2.
+//
+// The order is least-to-most load-bearing for the agent's next action:
+//
+// byPath — a convenience index; every card names its own path.
+// byCwe — a convenience index; every card names its own taxa.
+// byCluster — the cluster membership; the cards still carry cluster ids.
+// cards — the materialised read order. Dropped LAST, because without it
+// the agent has to reconstruct the order, and R.13 forbids any
+// order but this one.
+//
+// Nothing is deleted: each step writes the structure to a Tier-2 blob and
+// records a TierSpill naming the field and the `sha256:` reference.
+//
+// # The three index steps are all-or-nothing; the read order is NOT
+//
+// An index is a lookup table: half a `byPath` map is not a smaller index, it
+// is an index that silently lies about which paths it knows, so those three
+// steps still remove the WHOLE field.
+//
+// The read order is a LIST, and half a list in read order is exactly the first
+// half of the work. So the `anvil/cards` step is PARTIAL (spillCardTail): it
+// keeps as many card refs inline as the remaining budget affords, in read
+// order, and spills only the tail — with the spilled count in TierSpill.Items,
+// which the type has carried since it was written.
+//
+// CRITIQUE-03 M3 measured what the all-or-nothing version cost: at the
+// crossover — around fifty findings — the manifest went from just over 8,192
+// bytes to about 1,776 in one step, and roughly 78% of the Tier-0 budget then
+// sat unused while the agent had to fetch Tier 2 before it could start on the
+// FIRST finding. The budget was never exceeded and the order was never lost,
+// so it was a utilisation defect rather than a correctness one; it is now
+// fixed rather than documented. TestTier0PartialSpillUsesTheBudget asserts the
+// budget is USED at nine sizes, not merely respected.
+func (rd *Reader) fitManifest(m *Manifest) error {
+ steps := []struct {
+ field string
+ take func(*Manifest) (any, int, bool)
+ }{
+ {"anvil/index.byPath", func(m *Manifest) (any, int, bool) {
+ v := m.Index.ByPath
+ if len(v) == 0 {
+ return nil, 0, false
+ }
+ m.Index.ByPath = nil
+ return v, len(v), true
+ }},
+ {"anvil/index.byCwe", func(m *Manifest) (any, int, bool) {
+ v := m.Index.ByCwe
+ if len(v) == 0 {
+ return nil, 0, false
+ }
+ m.Index.ByCwe = nil
+ return v, len(v), true
+ }},
+ {"anvil/index.byCluster", func(m *Manifest) (any, int, bool) {
+ v := m.Index.ByCluster
+ if len(v) == 0 {
+ return nil, 0, false
+ }
+ m.Index.ByCluster = nil
+ return v, len(v), true
+ }},
+ }
+
+ size, err := measureManifest(m)
+ if err != nil {
+ return err
+ }
+ for _, st := range steps {
+ if size <= MaxTier0ManifestBytes {
+ break
+ }
+ payload, items, ok := st.take(m)
+ if !ok {
+ continue
+ }
+ if err := rd.spill(&m.Spills, m.Blobs, st.field, payload, items); err != nil {
+ return err
+ }
+ if size, err = measureManifest(m); err != nil {
+ return err
+ }
+ }
+
+ // The last step, and the only partial one.
+ if size > MaxTier0ManifestBytes && len(m.Cards) > 0 {
+ if size, err = rd.spillCardTail(m); err != nil {
+ return err
+ }
+ }
+
+ if size > MaxTier0ManifestBytes {
+ if rd.AllowOversizeTier0 == "" {
+ return &BudgetError{
+ Tier: "tier-0 manifest", Subject: m.AuditID,
+ Bytes: size, Budget: MaxTier0ManifestBytes,
+ Tokens: ApproxTokens(size), MaxTok: ApproxTokens(MaxTier0ManifestBytes),
+ }
+ }
+ m.Override = &BudgetOverride{
+ Reason: rd.AllowOversizeTier0,
+ Bytes: size,
+ Budget: MaxTier0ManifestBytes,
+ }
+ if _, err = measureManifest(m); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// spillCardTail is the `anvil/cards` shrink step. It keeps the longest PREFIX
+// of the read order that fits the remaining Tier-0 budget and spills the rest
+// to one Tier-2 blob, returning the manifest's size afterwards.
+//
+// A PREFIX, not a sample: the read order is the order R.13 requires the agent
+// to work in, so the refs worth keeping inline are the ones it needs FIRST.
+// The spilled blob holds the tail alone, so `m.Cards` followed by the blob's
+// contents is the whole order, once, in order — a consumer never has to
+// reconcile two overlapping copies, and TierSpill.Items says exactly how many
+// entries are on the other side of the reference.
+//
+// The search is a binary search over trial measurements rather than an
+// estimate from an average ref size, because the thing being fitted is the
+// MARSHALLED manifest: card refs differ in length (finding ids, cluster ids,
+// bucket names), the spill entry it is competing with grows its own decimal
+// fields, and measureManifest re-stamps `anvil/manifestBytes` on every call.
+// A trial measures what all three do together. Each trial marshals the tail,
+// so the step costs O(log n) marshals; nothing is persisted until the size is
+// settled, so a rejected trial never reaches the BlobSink.
+func (rd *Reader) spillCardTail(m *Manifest) (int, error) {
+ all := m.Cards
+ if len(all) == 0 {
+ return measureManifest(m)
+ }
+
+ // trial measures m as it WOULD stand with the first keep refs inline and
+ // the remaining ones spilled, then restores m exactly as it found it.
+ trial := func(keep int) (int, []byte, error) {
+ raw, err := json.Marshal(all[keep:])
+ if err != nil {
+ return 0, nil, fmt.Errorf("record: spilling anvil/cards to a Tier-2 blob failed: %w", err)
+ }
+ m.Cards = cardPrefix(all, keep)
+ m.Spills = append(m.Spills, TierSpill{
+ Field: "anvil/cards", Ref: BlobRef(raw), Bytes: len(raw), Items: len(all) - keep,
+ })
+ size, err := measureManifest(m)
+ m.Spills = m.Spills[:len(m.Spills)-1]
+ m.Cards = all
+ return size, raw, err
+ }
+
+ // keep == len(all) is not a candidate: this step is only reached because
+ // the manifest is over budget with the whole order inline, and a spill
+ // step that spills nothing would record a TierSpill for zero items.
+ lo, hi, best := 0, len(all)-1, -1
+ for lo <= hi {
+ mid := (lo + hi) / 2
+ size, _, err := trial(mid)
+ if err != nil {
+ return 0, err
+ }
+ if size <= MaxTier0ManifestBytes {
+ best, lo = mid, mid+1
+ } else {
+ hi = mid - 1
+ }
+ }
+ // best < 0 means not even an empty `anvil/cards` fits; spill the whole
+ // order and let the caller raise the BudgetError or apply the override.
+ // That is the old all-or-nothing behaviour, reached only when the
+ // envelope alone is over budget.
+ keep := best
+ if keep < 0 {
+ keep = 0
+ }
+
+ _, raw, err := trial(keep)
+ if err != nil {
+ return 0, err
+ }
+ m.Cards = cardPrefix(all, keep)
+ if err := rd.spillBytes(&m.Spills, m.Blobs, "anvil/cards", raw, len(all)-keep); err != nil {
+ return 0, err
+ }
+ return measureManifest(m)
+}
+
+// cardPrefix returns the first keep refs, and nil rather than an empty slice
+// when keep is zero: `"anvil/cards": null` is how a manifest that carries no
+// inline read order has always spelled it, and a consumer distinguishing `null`
+// from `[]` should not start seeing a new spelling because the step became
+// partial.
+func cardPrefix(all []CardRef, keep int) []CardRef {
+ if keep <= 0 {
+ return nil
+ }
+ return all[:keep]
+}
+
+// measureManifest marshals m, stamps the measurement into m, and returns the
+// size m ACTUALLY has once stamped.
+//
+// The stamping is why this is not one line: writing the byte count into the
+// object changes the object's byte count. Re-measuring after each stamp
+// converges — the only thing that changes is the decimal width of two integers
+// that are already close to their final value — and the loop bounds it. It
+// matters because the budget check must run against the size the manifest ends
+// up with, not the size it had before it described itself; a few bytes of
+// self-report are exactly how a check like this ends up passing on an object
+// that is over budget on disk.
+func measureManifest(m *Manifest) (int, error) {
+ size, err := measure(m)
+ if err != nil {
+ return 0, err
+ }
+ for i := 0; i < 8; i++ {
+ m.Bytes, m.Tokens = size, ApproxTokens(size)
+ next, err := measure(m)
+ if err != nil {
+ return 0, err
+ }
+ if next == size {
+ return size, nil
+ }
+ size = next
+ }
+ m.Bytes, m.Tokens = size, ApproxTokens(size)
+ return measure(m)
+}
+
+// ---------------------------------------------------------------------------
+// Spilling to Tier 2
+// ---------------------------------------------------------------------------
+
+// BlobRef returns the `sha256:<64 lowercase hex>` reference for content. It is
+// the same spelling R.8's masker writes, deliberately: one content-addressing
+// scheme, or a consumer has to guess which one it is looking at.
+func BlobRef(content []byte) string {
+ sum := sha256.Sum256(content)
+ return "sha256:" + hex.EncodeToString(sum[:])
+}
+
+func (rd *Reader) spill(dst *[]TierSpill, blobs map[string][]byte, field string, payload any, items int) error {
+ raw, err := json.Marshal(payload)
+ if err != nil {
+ return fmt.Errorf("record: spilling %s to a Tier-2 blob failed: %w", field, err)
+ }
+ return rd.spillBytes(dst, blobs, field, raw, items)
+}
+
+// spillBytes is spill for content that is already bytes. A body spills as the
+// bytes themselves, not as a JSON string containing them: R.8's masker
+// content-addresses the raw masked body, and two spellings of the same blob
+// would content-address to two different digests.
+func (rd *Reader) spillBytes(dst *[]TierSpill, blobs map[string][]byte, field string, raw []byte, items int) error {
+ ref := BlobRef(raw)
+ if blobs != nil {
+ blobs[ref] = raw
+ }
+ if rd.Blobs != nil {
+ if err := rd.Blobs(ref, raw); err != nil {
+ return fmt.Errorf("record: persisting Tier-2 blob %s for %s failed: %w", ref, field, err)
+ }
+ }
+ *dst = append(*dst, TierSpill{Field: field, Ref: ref, Bytes: len(raw), Items: items})
+ return nil
+}
+
+// ---------------------------------------------------------------------------
+// Small shared helpers
+// ---------------------------------------------------------------------------
+
+func measure(v any) (int, error) {
+ raw, err := json.Marshal(v)
+ if err != nil {
+ return 0, fmt.Errorf("record: measuring a read-path tier failed: %w", err)
+ }
+ return len(raw), nil
+}
+
+func emptyToNil(m map[string][]string) map[string][]string {
+ if len(m) == 0 {
+ return nil
+ }
+ return m
+}
+
+// formatTime renders a timestamp in the one form the whole read path uses:
+// RFC 3339 in UTC, which is what time.Time's own JSON encoding produces.
+//
+// A zero time renders as the empty string rather than as year 1, which a
+// consumer would otherwise parse as a real timestamp.
+func formatTime(t time.Time) string {
+ if t.IsZero() {
+ return ""
+ }
+ return t.UTC().Format(time.RFC3339Nano)
+}
+
+// cardPath is the Tier-1 path for a finding id.
+//
+// Finding ids are `sast:8c1e4b0f…` in research/18's own example, and a colon
+// is not a legal path character on Windows — where this project is being
+// developed. The sanitisation is a total, deterministic function so the same
+// finding always lands at the same path on every platform.
+func (rd *Reader) cardPath(findingID string) string {
+ return rd.cardPrefix() + SanitizeCardFilename(findingID) + ".json"
+}
+
+// SanitizeCardFilename maps a finding id onto a filesystem-safe, deterministic
+// basename. Every byte outside [A-Za-z0-9._-] becomes '-'.
+func SanitizeCardFilename(findingID string) string {
+ var b strings.Builder
+ b.Grow(len(findingID))
+ for i := 0; i < len(findingID); i++ {
+ c := findingID[i]
+ switch {
+ case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9',
+ c == '.', c == '_', c == '-':
+ b.WriteByte(c)
+ default:
+ b.WriteByte('-')
+ }
+ }
+ if b.Len() == 0 {
+ return "unnamed"
+ }
+ return b.String()
+}
+
+// taxonIDs returns the result's taxon ids (CWE, in practice), deduplicated and
+// sorted so the inverted index is stable.
+func taxonIDs(r *Result) []string {
+ if len(r.Taxa) == 0 {
+ return nil
+ }
+ seen := map[string]bool{}
+ var out []string
+ for _, t := range r.Taxa {
+ if t.ID == "" || seen[t.ID] {
+ continue
+ }
+ seen[t.ID] = true
+ out = append(out, t.ID)
+ }
+ sort.Strings(out)
+ return out
+}
+
+// primaryPath returns the first physical location's artifact URI, which is
+// SARIF's own notion of "the file this result is about".
+func primaryPath(r *Result) string {
+ for _, loc := range r.Locations {
+ if loc.PhysicalLocation != nil && loc.PhysicalLocation.ArtifactLocation.URI != "" {
+ return loc.PhysicalLocation.ArtifactLocation.URI
+ }
+ }
+ return ""
+}
diff --git a/internal/record/readpath_test.go b/internal/record/readpath_test.go
new file mode 100644
index 0000000..7ea2e7a
--- /dev/null
+++ b/internal/record/readpath_test.go
@@ -0,0 +1,3819 @@
+package record
+
+import (
+ "bytes"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "go/ast"
+ "go/parser"
+ "go/printer"
+ "go/scanner"
+ "go/token"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "testing"
+ "time"
+)
+
+// ---------------------------------------------------------------------------
+// Fixture
+//
+// One realistic record: a correlated cluster (one SAST + one DAST finding),
+// several SAST-only findings including an SCA finding and a HOST finding, and
+// one DAST-only finding. That is exactly R.13's stop condition.
+//
+// The fixture is built, then VALIDATED against contract.go, then MASKED by
+// R.8. Both steps are deliberate: a fixture that could not survive the
+// producer's own gates would prove nothing about the read path, and the read
+// path refuses an unmasked record by design.
+// ---------------------------------------------------------------------------
+
+const (
+ rpAuditID = "0198e2c1-6a4b-7d3e-9f10-2b7c5d8a4e11"
+ rpClusterID = "9f2c7a10-4e88-4d1b-b6c2-1a5f77e40d3e"
+)
+
+func rpDigest(seed byte) string {
+ b := make([]byte, FingerprintDigestHexLen)
+ const hex = "0123456789abcdef"
+ for i := range b {
+ b[i] = hex[(int(seed)+i*7)%16]
+ }
+ return string(b)
+}
+
+func rpFloat(f float64) *float64 { return &f }
+
+func rpTime() time.Time { return time.Date(2026, 8, 7, 9, 0, 0, 0, time.UTC) }
+
+// rpSastResult is the common shape of a static finding.
+func rpSastResult(id string, rank float64, ec EvidenceClass, v Verdict, remediable bool, path string, seed byte) Result {
+ return Result{
+ RuleID: "anvil.sql-injection",
+ Kind: KindFail,
+ Level: LevelError,
+ Rank: rpFloat(rank),
+ Message: Message{Text: fmt.Sprintf(
+ "Fix the tainted value reaching the sink in %s and keep the existing behaviour.", path)},
+ Locations: []Location{{
+ PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: path},
+ Region: &Region{
+ StartLine: 412, EndLine: 414,
+ Snippet: &Snippet{Text: " query = \"SELECT id FROM users WHERE name = '\" + username + \"'\"\n cur.execute(query)"},
+ },
+ ContextRegion: &Region{
+ StartLine: 404, EndLine: 421,
+ Snippet: &Snippet{Text: "def authenticate(conn, username, password):\n ...\n return None"},
+ },
+ },
+ LogicalLocations: []LogicalLocation{{FullyQualifiedName: "app.db.authenticate", Kind: "function"}},
+ }},
+ Taxa: []ReportingDescriptorReference{{ID: "CWE-89"}},
+ PartialFingerprints: map[string]string{
+ PartialFingerprintAnvilFindingID: rpDigest(seed),
+ PartialFingerprintPrimaryLocationLineHash: "7a1c9e0b4d2f6813",
+ },
+ Properties: ResultProperties{
+ FindingID: id,
+ Half: HalfSast,
+ Confidence: 0.88,
+ Verdict: v,
+ RemediableByAgent: remediable,
+ Reasoning: "The username parameter reaches execute() by string concatenation with no parameterisation.",
+ Detector: DetectorRef{
+ Kind: DetectorKindSast, Model: "anvil-sast", Revision: "2026.07.1",
+ },
+ EvidenceClass: ec,
+ Trust: TrustAssertion{Default: TrustUntrusted},
+ Locus: &Locus{ProximityClass: "same_symbol"},
+ PatchContext: &PatchContext{
+ Language: "python3.12", Framework: "flask", DBDriver: "sqlite3",
+ EditableFiles: []string{path},
+ TestCommand: "pytest tests/test_db.py -k authenticate",
+ },
+ },
+ }
+}
+
+// rpFixtureLog builds the base record. It is deliberately NOT masked or
+// validated here: rpFixture does both, and a few tests need a record that
+// fails one of them.
+func rpFixtureLog() *SARIFLog {
+ created := rpTime()
+ sealed := created.Add(20 * time.Minute)
+
+ s1 := rpSastResult("sast:0001", 97.0, EvidenceClassSastReachable, VerdictTruePositive, true, "app/db.py", 1)
+ s1.CorrelationGUID = rpClusterID
+ s1.CodeFlows = []CodeFlow{{ThreadFlows: []ThreadFlow{{Locations: []ThreadFlowLocation{
+ {Location: Location{PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: "app/routes.py"},
+ Region: &Region{StartLine: 91, Snippet: &Snippet{Text: "username = request.json[\"username\"]"}},
+ }}},
+ {Location: Location{PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: "app/db.py"},
+ Region: &Region{StartLine: 414, Snippet: &Snippet{Text: "cur.execute(query)"}},
+ }}},
+ }}}}}
+ s1.Properties.Correlation = &Correlation{
+ ClusterID: rpClusterID, Role: HalfSast, Peers: []string{"dast:0101"},
+ Signals: []SignalWeight{
+ {Name: CorrelationSignalResponseStackTrace, Weight: 0.7, Detail: "stack trace names app/db.py:414"},
+ {Name: CorrelationSignalParameterName, Weight: 0.24, Detail: "username"},
+ },
+ Confidence: 0.94, Verified: true,
+ VerificationMethod: "response stack trace inside the static region",
+ }
+ s1.Properties.Advisory = &AdvisoryContext{
+ IDs: []string{"CWE-89"}, SourceFeed: "cwe", SnapshotDigest: "sha256:aa",
+ AsOf: created.Add(-48 * time.Hour), StalenessSeconds: 172800,
+ Excerpt: &TrustedString{
+ Text: "Use a parameterised query; sqlite3 accepts a params tuple as execute()'s second argument.",
+ Trust: TrustUntrusted,
+ },
+ }
+
+ s2 := rpSastResult("sast:0002", 61.0, EvidenceClassSastReachable, VerdictTruePositive, true, "app/routes.py", 2)
+ s3 := rpSastResult("sast:0003", 40.0, EvidenceClassSastStaticOnly, VerdictInsufficientContext, true, "app/util.py", 3)
+ s4 := rpSastResult("sast:0004", 40.0, EvidenceClassSastStaticOnly, VerdictTruePositive, true, "app/view.py", 4)
+ s7 := rpSastResult("sast:0007", 20.0, EvidenceClassSastStaticOnly, VerdictFalsePositive, true, "app/old.py", 7)
+
+ sca := rpSastResult("sca:0005", 55.0, EvidenceClassSCA, VerdictTruePositive, true, "pom.xml", 5)
+ sca.RuleID = "anvil.vulnerable-dependency"
+ sca.Properties.Detector.Kind = DetectorKindSCA
+ sca.Properties.Risk = &Risk{
+ CvssV4Base: rpFloat(10.0), EpssScore: rpFloat(0.97), EpssPercentile: rpFloat(0.999),
+ KevMember: true, KevRansomwareUse: true,
+ }
+
+ // The HOST finding. remediable_by_agent is false and must stay false:
+ // 00-SPINE.md S7 makes the host agent read-only.
+ host := rpSastResult("host:0006", 30.0, EvidenceClassHost, VerdictTruePositive, false, "", 6)
+ host.RuleID = "anvil.host-package"
+ host.Locations = nil
+ host.Taxa = nil
+ host.CodeFlows = nil
+ host.Properties.Detector.Kind = DetectorKindHost
+ host.Properties.PatchContext = nil
+ host.Properties.Locus = nil
+ delete(host.PartialFingerprints, PartialFingerprintPrimaryLocationLineHash)
+
+ d1 := rpDastResult("dast:0101", 97.0, 101)
+ d1.CorrelationGUID = rpClusterID
+ d1.Properties.Correlation = &Correlation{
+ ClusterID: rpClusterID, Role: HalfDast, Peers: []string{"sast:0001"},
+ Signals: []SignalWeight{
+ {Name: CorrelationSignalResponseStackTrace, Weight: 0.7},
+ {Name: CorrelationSignalRouteTable, Weight: 0.24},
+ },
+ Confidence: 0.94, Verified: true,
+ }
+ d2 := rpDastResult("dast:0102", 70.0, 102)
+
+ sastRun := Run{
+ Tool: Tool{Driver: ToolComponent{
+ Name: "anvil-sast", Version: "2026.07.1",
+ Rules: []ReportingDescriptor{{
+ ID: "anvil.sql-injection", Name: "SqlInjection",
+ ShortDescription: &Message{Text: "Tainted input reaches a SQL sink."},
+ HelpURI: "https://cwe.mitre.org/data/definitions/89.html",
+ }},
+ }},
+ AutomationDetails: RunAutomationDetails{ID: "sast/1", CorrelationGUID: rpAuditID},
+ Results: []Result{s1, s2, s3, s4, sca, host, s7},
+ Properties: RunProperties{
+ Half: HalfSast, Status: HalfStatusSealed, SealedAt: &sealed,
+ AdvisorySnapshot: &AdvisorySnapshot{
+ FeedIDs: []string{"cwe"}, SnapshotDigest: "sha256:aa", ScrapedAt: created,
+ },
+ },
+ }
+
+ dastSealed := created.Add(35 * time.Minute)
+ dastRun := Run{
+ Tool: Tool{Driver: ToolComponent{Name: "anvil-dast", Version: "2026.07.1"}},
+ AutomationDetails: RunAutomationDetails{ID: "dast/1", CorrelationGUID: rpAuditID},
+ Results: []Result{d1, d2},
+ Properties: RunProperties{
+ Half: HalfDast, Status: HalfStatusSealed, SealedAt: &dastSealed,
+ RouteTableDigest: "sha256:7de1a0",
+ RuntimeTarget: &RuntimeTarget{
+ BaseURL: "https://staging.payments.internal", AuthProfileRef: "anvil.dast.yaml@a91c3f2",
+ Scope: []string{"/api/**"}, Excluded: []string{"/api/admin/**"},
+ },
+ DastCoverage: &DastCoverage{
+ ProbedCount: 31, InventoryUnionCount: 50, EndpointCoverage: 0.62,
+ InventoryProvenanceMix: map[InventoryProvenance]int{
+ InventoryProvenanceRuntimeSpec: 40, InventoryProvenanceCrawl: 10,
+ },
+ ConfirmedCount: 40, CandidateCount: 10,
+ },
+ },
+ }
+
+ return &SARIFLog{
+ Schema: SARIFSchemaURI, Version: SARIFVersion,
+ Runs: []Run{sastRun, dastRun},
+ Properties: AuditProperties{
+ SchemaVersion: SchemaVersion,
+ AuditID: rpAuditID,
+ State: StateBothSealed,
+ Version: 1,
+ CreatedAt: created,
+ Target: Target{
+ RepoURL: "https://github.com/example/payments", Ref: "refs/heads/main",
+ Commit: "3f2a1c0d", RuntimeBaseURL: "https://staging.payments.internal",
+ Provenance: TargetProvenanceBootedClean, Provisioning: TargetProvisioningEphemeralManifest,
+ },
+ Trigger: Trigger{
+ Kind: "push", PolicyID: "p-1", PolicyRef: "anvil.yaml@a91c3f2",
+ ConfigSource: "repo", Actor: "ci", ResolvedAt: created,
+ },
+ Deadline: Deadline{
+ DeadlineAt: created.Add(DefaultClaimTimeoutSeconds * time.Second),
+ ClaimTimeoutSeconds: DefaultClaimTimeoutSeconds,
+ },
+ Index: Index{ReadOrder: DefaultReadOrder()},
+ DastStatus: DastStatusCompletedFindings,
+ },
+ }
+}
+
+func rpDastResult(id string, rank float64, seed byte) Result {
+ return Result{
+ RuleID: "anvil.sqli-error-based",
+ Kind: KindFail,
+ Level: LevelError,
+ Rank: rpFloat(rank),
+ Message: Message{Text: "POST /api/login returns 500 with a database error for a quote payload."},
+ Taxa: []ReportingDescriptorReference{{ID: "CWE-89"}},
+ WebRequest: &WebRequest{
+ Method: "POST", Target: "https://staging.payments.internal/api/login",
+ Headers: map[string]string{"Content-Type": "application/json"},
+ Body: &ArtifactContent{Text: `{"username":"' OR '1'='1' -- ","password":"x"}`},
+ },
+ WebResponse: &WebResponse{
+ StatusCode: 500, ReasonPhrase: "Internal Server Error",
+ Headers: map[string]string{"Content-Type": "text/html"},
+ Body: &ArtifactContent{Text: "sqlite3.OperationalError: unrecognized token
"},
+ },
+ PartialFingerprints: map[string]string{PartialFingerprintAnvilFindingID: rpDigest(seed)},
+ Properties: ResultProperties{
+ FindingID: id,
+ Half: HalfDast,
+ Confidence: 0.97,
+ Verdict: VerdictTruePositive,
+ RemediableByAgent: true,
+ Reasoning: "A quote in the username flips the response from 401 to 500 with a driver error.",
+ Detector: DetectorRef{Kind: DetectorKindDast, Model: "anvil-dast", Revision: "2026.07.1"},
+ EvidenceClass: EvidenceClassDastConfirmed,
+ Trust: TrustAssertion{Default: TrustUntrusted},
+ Repro: &Repro{
+ Curl: "curl -sS -X POST https://staging.payments.internal/api/login -H 'Content-Type: application/json' --data-raw '{}'",
+ InjectionPoint: ReproInjection{Kind: InjectionPointBody, Name: "username"},
+ Payload: "' OR '1'='1' -- ",
+ PayloadEncoding: "utf8",
+ Baseline: &ReproBaseline{StatusCode: 401, LatencyMs: 12},
+ ObservedSignal: ReproSignal{
+ Kind: EvidenceSignalDBErrorString,
+ Match: &TrustedString{Text: "sqlite3.OperationalError: unrecognized token", Trust: TrustUntrusted},
+ BodySha256: "sha256:5c0d",
+ },
+ ExpectedAfterFix: &ReproExpectation{
+ StatusCode: 401, MustNotContain: []string{"OperationalError", "Traceback"},
+ },
+ Env: ReproEnv{Sanitizers: []string{}, AslrEnabled: true, Arch: "amd64", OS: "linux"},
+ },
+ },
+ }
+}
+
+// rpFixture builds, validates and masks the fixture. mutate runs before both
+// gates so a test can shape the record and still get the real pipeline.
+func rpFixture(t *testing.T, mutate func(*SARIFLog)) *SARIFLog {
+ t.Helper()
+ l := rpFixtureLog()
+ if mutate != nil {
+ mutate(l)
+ }
+ if err := l.Validate(); err != nil {
+ t.Fatalf("fixture does not satisfy contract.go's own validator: %v", err)
+ }
+ if err := MaskRecord(l); err != nil {
+ t.Fatalf("masking the fixture failed: %v", err)
+ }
+ if err := AssertMasked(l); err != nil {
+ t.Fatalf("fixture is not masked after MaskRecord: %v", err)
+ }
+ return l
+}
+
+func rpReader(t *testing.T, l *SARIFLog) *Reader {
+ t.Helper()
+ return NewReader(RecordMap{l.Properties.AuditID: l})
+}
+
+func rpFindingIDs(cards []TaskCard) []string {
+ out := make([]string, len(cards))
+ for i, c := range cards {
+ out[i] = c.FindingID
+ }
+ return out
+}
+
+func rpMarshal(t *testing.T, v any) []byte {
+ t.Helper()
+ raw, err := json.Marshal(v)
+ if err != nil {
+ t.Fatalf("marshal: %v", err)
+ }
+ return raw
+}
+
+// ---------------------------------------------------------------------------
+// The read order
+// ---------------------------------------------------------------------------
+
+// The bucket names are declared in this package but the ORDER is contract.go's
+// DefaultReadOrder(). If the two ever disagree, the read path silently stops
+// being the order R.13 mandates, so the disagreement is a test failure.
+func TestReadOrderBucketsMatchTheContract(t *testing.T) {
+ want := []string{BucketClusters, BucketSastByRank, BucketDastByRank}
+ got := DefaultReadOrder()
+ if len(got) != len(want) {
+ t.Fatalf("DefaultReadOrder() = %q, this package's buckets are %q", got, want)
+ }
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("DefaultReadOrder()[%d] = %q, this package's bucket is %q", i, got[i], want[i])
+ }
+ }
+}
+
+func TestReadOrderIsClustersThenSastByRankThenDastByRank(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+
+ want := []string{
+ // The cluster first, SAST member before DAST member: the SAST finding
+ // owns the file and line the agent has to edit.
+ "sast:0001", "dast:0101",
+ // Then SAST-only by rank. sast:0003 and sast:0004 are tied at 40, and
+ // the finding-id tie-break puts 0003 first.
+ "sast:0002", "sca:0005", "sast:0003", "sast:0004", "host:0006", "sast:0007",
+ // Then DAST-only.
+ "dast:0102",
+ }
+ got := rpFindingIDs(cards)
+ if strings.Join(got, ",") != strings.Join(want, ",") {
+ t.Fatalf("read order\n got: %q\nwant: %q", got, want)
+ }
+
+ // The bucket sequence must never interleave.
+ seen := map[string]int{}
+ last := -1
+ for i, c := range cards {
+ idx := -1
+ for j, b := range DefaultReadOrder() {
+ if b == c.Bucket {
+ idx = j
+ }
+ }
+ if idx < 0 {
+ t.Fatalf("card %d has bucket %q, which is not in DefaultReadOrder()", i, c.Bucket)
+ }
+ if idx < last {
+ t.Fatalf("card %d (%s) is in bucket %q after a later bucket: buckets must not interleave",
+ i, c.FindingID, c.Bucket)
+ }
+ last = idx
+ seen[c.Bucket]++
+ }
+ for _, b := range DefaultReadOrder() {
+ if seen[b] == 0 {
+ t.Errorf("bucket %q produced no cards; the fixture is supposed to exercise all three", b)
+ }
+ }
+
+ // Within each rank bucket, rank must be non-increasing.
+ for i := 1; i < len(cards); i++ {
+ if cards[i].Bucket != cards[i-1].Bucket || cards[i].Bucket == BucketClusters {
+ continue
+ }
+ if cards[i].Rank > cards[i-1].Rank {
+ t.Errorf("%s (rank %v) sorted after %s (rank %v) in bucket %q",
+ cards[i].FindingID, cards[i].Rank, cards[i-1].FindingID, cards[i-1].Rank, cards[i].Bucket)
+ }
+ }
+}
+
+// Determinism, twice over: repeated calls on the same input, and the same
+// input presented in a different order. The second is the one that matters —
+// a sort that is merely deterministic-for-this-slice still reorders when the
+// producer emits results in a different sequence, and the agent's read order
+// would silently change between two scans of the same repository.
+func TestReadOrderIsStableAcrossCallsAndInputPermutations(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := rpReader(t, l)
+
+ first, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ firstManifest, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ wantCards := string(rpMarshal(t, first))
+ wantManifest := string(rpMarshal(t, firstManifest))
+
+ for i := 0; i < 5; i++ {
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards call %d: %v", i, err)
+ }
+ if got := string(rpMarshal(t, cards)); got != wantCards {
+ t.Fatalf("task cards differ between call 0 and call %d", i+1)
+ }
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest call %d: %v", i, err)
+ }
+ if got := string(rpMarshal(t, m)); got != wantManifest {
+ t.Fatalf("manifest differs between call 0 and call %d", i+1)
+ }
+ }
+
+ reversed := rpFixture(t, func(l *SARIFLog) {
+ for ri := range l.Runs {
+ rs := l.Runs[ri].Results
+ for i, j := 0, len(rs)-1; i < j; i, j = i+1, j-1 {
+ rs[i], rs[j] = rs[j], rs[i]
+ }
+ }
+ })
+ permuted, err := rpReader(t, reversed).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards on the permuted record: %v", err)
+ }
+ if got, want := rpFindingIDs(permuted), rpFindingIDs(first); strings.Join(got, ",") != strings.Join(want, ",") {
+ t.Fatalf("reversing the record's result order changed the read order\n got: %q\nwant: %q", got, want)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Tier 0 — the 8 KB budget, measured
+// ---------------------------------------------------------------------------
+
+// The budget is not assumed. The manifest is marshalled and its real byte
+// length is compared against MaxTier0ManifestBytes.
+func TestManifestFitsTier0BudgetOnARealisticRecord(t *testing.T) {
+ l := rpFixture(t, nil)
+ m, err := rpReader(t, l).BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+
+ raw := rpMarshal(t, m)
+ t.Logf("tier-0 manifest: %d bytes (~%d tokens) of the %d-byte budget, %d findings",
+ len(raw), ApproxTokens(len(raw)), MaxTier0ManifestBytes, m.Index.Counts.Total)
+
+ if len(raw) > MaxTier0ManifestBytes {
+ t.Fatalf("manifest is %d bytes, over the %d-byte Tier-0 budget", len(raw), MaxTier0ManifestBytes)
+ }
+ if m.Bytes != len(raw) {
+ t.Errorf("manifest reports %d bytes but marshals to %d; the self-report must be the measurement",
+ m.Bytes, len(raw))
+ }
+ if m.Tokens != ApproxTokens(len(raw)) {
+ t.Errorf("manifest reports %d tokens, want %d", m.Tokens, ApproxTokens(len(raw)))
+ }
+ if len(m.Spills) != 0 {
+ t.Errorf("a nine-finding record should fit Tier 0 with nothing spilled, got %d spills", len(m.Spills))
+ }
+ if m.Override != nil {
+ t.Errorf("manifest carries a budget override it should not need: %+v", m.Override)
+ }
+
+ if m.Index.Counts.Total != 9 || m.Index.Counts.Sast != 7 || m.Index.Counts.Dast != 2 {
+ t.Errorf("counts = %+v, want total 9 / sast 7 / dast 2", m.Index.Counts)
+ }
+ if m.Index.Counts.Clusters != 1 || m.Index.Counts.Unclustered != 7 {
+ t.Errorf("counts = %+v, want 1 cluster and 7 unclustered", m.Index.Counts)
+ }
+ if m.Index.TaskCards != DefaultTaskCardPrefix || m.Index.Blobs != DefaultBlobPrefix {
+ t.Errorf("tier prefixes = %q/%q, want %q/%q",
+ m.Index.TaskCards, m.Index.Blobs, DefaultTaskCardPrefix, DefaultBlobPrefix)
+ }
+ if len(m.Index.ByCluster[rpClusterID]) != 2 {
+ t.Errorf("byCluster[%s] = %q, want both members", rpClusterID, m.Index.ByCluster[rpClusterID])
+ }
+ if len(m.Index.ByPath["app/db.py"]) != 1 || len(m.Index.ByCwe["CWE-89"]) == 0 {
+ t.Errorf("inverted indexes are not populated: byPath=%v byCwe=%v", m.Index.ByPath, m.Index.ByCwe)
+ }
+}
+
+// A big record still fits, because the manifest degrades by SPILLING rather
+// than by truncating. Nothing is lost: every dropped structure is a Tier-2
+// blob named in Spills.
+func TestLargeRecordManifestStaysUnderBudgetBySpilling(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ for i := 0; i < 400; i++ {
+ r := rpSastResult(fmt.Sprintf("sast:9%03d", i), float64(500-i),
+ EvidenceClassSastStaticOnly, VerdictTruePositive, true,
+ fmt.Sprintf("app/pkg%02d/module%03d.py", i%20, i), byte(i))
+ l.Runs[0].Results = append(l.Runs[0].Results, r)
+ }
+ })
+
+ m, err := rpReader(t, l).BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ raw := rpMarshal(t, m)
+ t.Logf("tier-0 manifest on a 409-finding record: %d bytes, %d spills", len(raw), len(m.Spills))
+
+ if len(raw) > MaxTier0ManifestBytes {
+ t.Fatalf("manifest is %d bytes, over the %d-byte budget even after spilling", len(raw), MaxTier0ManifestBytes)
+ }
+ if m.Override != nil {
+ t.Fatalf("manifest took an override it did not need: %+v", m.Override)
+ }
+ if len(m.Spills) == 0 {
+ t.Fatalf("a 409-finding record must have spilled something to fit 8 KB")
+ }
+
+ // The spill order is fixed: convenience indexes first, the read order last.
+ wantOrder := []string{"anvil/index.byPath", "anvil/index.byCwe", "anvil/index.byCluster", "anvil/cards"}
+ for i, s := range m.Spills {
+ if s.Field != wantOrder[i] {
+ t.Fatalf("spill %d is %q, want %q (the shrink order is fixed and is not the map's)",
+ i, s.Field, wantOrder[i])
+ }
+ }
+
+ // Nothing was dropped: every spill resolves to retained bytes that
+ // round-trip, and the counts still describe all 409 findings.
+ for _, s := range m.Spills {
+ blob, ok := m.Blobs[s.Ref]
+ if !ok {
+ t.Fatalf("spill %s references blob %s, which was not retained", s.Field, s.Ref)
+ }
+ if BlobRef(blob) != s.Ref {
+ t.Fatalf("blob for %s does not hash to its own reference", s.Field)
+ }
+ if len(blob) != s.Bytes {
+ t.Errorf("spill %s reports %d bytes, blob is %d", s.Field, s.Bytes, len(blob))
+ }
+ if !json.Valid(blob) {
+ t.Errorf("spill %s is not valid JSON", s.Field)
+ }
+ }
+ if m.Index.Counts.Total != 409 {
+ t.Errorf("counts.total = %d, want 409: spilling must not change what the manifest counts",
+ m.Index.Counts.Total)
+ }
+ // The read order survives in full, as an inline PREFIX plus a spilled
+ // TAIL: `m.Cards` then the blob is the whole order, once, in order.
+ // CRITIQUE-03 M3: this step used to be all-or-nothing, which spent the
+ // budget it had just freed on nothing.
+ for _, s := range m.Spills {
+ if s.Field != "anvil/cards" {
+ continue
+ }
+ var tail []CardRef
+ if err := json.Unmarshal(m.Blobs[s.Ref], &tail); err != nil {
+ t.Fatalf("spilled read order does not unmarshal: %v", err)
+ }
+ if len(tail) != s.Items {
+ t.Errorf("spill reports %d items, blob holds %d; TierSpill.Items is how a "+
+ "consumer knows how much order is on the other side of the reference",
+ s.Items, len(tail))
+ }
+ if len(m.Cards)+len(tail) != 409 {
+ t.Errorf("%d inline refs + %d spilled = %d, want all 409: a partial spill "+
+ "may move the tail, never lose it", len(m.Cards), len(tail), len(m.Cards)+len(tail))
+ }
+ if len(m.Cards) == 0 {
+ t.Error("the whole read order spilled; the step is partial and the budget freed " +
+ "by the three index spills must be spent on inline refs")
+ }
+ // The PREFIX is what the agent works first, so the cluster — which
+ // DefaultReadOrder puts first — must be inline, not fetched.
+ if len(m.Cards) < 2 || m.Cards[0].FindingID != "sast:0001" || m.Cards[1].FindingID != "dast:0101" {
+ t.Errorf("the inline prefix does not start with the cluster: %v", rpCardIDs(m.Cards))
+ }
+ // And the tail resumes exactly where the prefix stopped: concatenating
+ // them must reproduce the order BuildTaskCards emits.
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ joined := append(rpCardIDs(m.Cards), rpCardIDs(tail)...)
+ if len(joined) != len(cards) {
+ t.Fatalf("read order is %d refs but %d cards were emitted", len(joined), len(cards))
+ }
+ for i := range cards {
+ if cards[i].FindingID != joined[i] {
+ t.Fatalf("prefix+tail entry %d is %q, but the card at that position is %q; "+
+ "the two halves of a partial spill must join back into ONE order",
+ i, joined[i], cards[i].FindingID)
+ }
+ }
+ }
+}
+
+// rpCardIDs is the finding ids of a read order, in order.
+func rpCardIDs(refs []CardRef) []string {
+ out := make([]string, 0, len(refs))
+ for _, r := range refs {
+ out = append(out, r.FindingID)
+ }
+ return out
+}
+
+// An over-budget Tier 0 that cannot be shrunk is an ERROR, not a silently
+// oversized manifest — and it becomes legal only with an explicit reason,
+// which is then recorded in the artifact.
+func TestOversizeTier0NeedsAnExplicitLoggedOverride(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ // A repo URL nothing can spill: it is envelope, not index.
+ l.Properties.Target.RepoURL = "https://example.invalid/" + strings.Repeat("a", 9000)
+ })
+
+ rd := rpReader(t, l)
+ if _, err := rd.BuildManifest(rpAuditID); err == nil {
+ t.Fatal("an over-budget manifest with no override must be an error")
+ } else {
+ var be *BudgetError
+ if !errors.As(err, &be) {
+ t.Fatalf("want a *BudgetError, got %T: %v", err, err)
+ }
+ if be.Budget != MaxTier0ManifestBytes {
+ t.Errorf("BudgetError.Budget = %d, want %d", be.Budget, MaxTier0ManifestBytes)
+ }
+ }
+
+ rd.AllowOversizeTier0 = "R.13 evidence test: envelope alone exceeds the budget"
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("with an explicit override, BuildManifest must succeed: %v", err)
+ }
+ if m.Override == nil {
+ t.Fatal("the override must be recorded in the manifest, not merely honoured")
+ }
+ if m.Override.Reason != rd.AllowOversizeTier0 {
+ t.Errorf("override reason = %q, want %q", m.Override.Reason, rd.AllowOversizeTier0)
+ }
+ if m.Override.Budget != MaxTier0ManifestBytes || m.Override.Bytes <= MaxTier0ManifestBytes {
+ t.Errorf("override does not record the overrun honestly: %+v", m.Override)
+ }
+}
+
+// TestTier0PartialSpillUsesTheBudget is CRITIQUE-03 M3 part 1's regression
+// test, and it asserts the thing the previous shrink policy got wrong: not
+// that the budget is RESPECTED — the all-or-nothing version respected it while
+// throwing away 78% of it — but that the budget is USED.
+//
+// The measurement that motivated the fix, at nine sizes: below the crossover
+// nothing spills and utilisation climbs to 98%; at the crossover the whole
+// read order left in one step and utilisation fell to 22%, where it stayed for
+// every larger record. An agent reading a 409-finding manifest therefore had
+// to fetch Tier 2 before it could start on the FIRST finding, against a Tier-0
+// budget that was three-quarters empty.
+//
+// The floor below is deliberately loose (85%). It is a REGRESSION bound, not
+// the measured figure: this test must fail when the step goes back to
+// all-or-nothing, and must not fail because a card ref grew a field and the
+// last ref no longer fits. The measured figures are logged on every run.
+func TestTier0PartialSpillUsesTheBudget(t *testing.T) {
+ // The floor applies only once something has spilled: a nine-finding
+ // record fits in 3,008 bytes and there is nothing to fill the rest WITH.
+ const floorPct = 85.0
+
+ for _, extra := range []int{0, 10, 20, 30, 40, 50, 80, 120, 400} {
+ t.Run(fmt.Sprintf("extra%d", extra), func(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ for i := 0; i < extra; i++ {
+ r := rpSastResult(fmt.Sprintf("sast:8%03d", i), 30.0, EvidenceClassSastStaticOnly,
+ VerdictTruePositive, true, fmt.Sprintf("app/svc/mod%d/file%d.py", i%16, i), byte(i%251))
+ r.PartialFingerprints[PartialFingerprintAnvilFindingID] = rpDigest(byte(30 + i%220))
+ l.Runs[0].Results = append(l.Runs[0].Results, r)
+ }
+ })
+ rd := rpReader(t, l)
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+
+ total := 0
+ for i := range l.Runs {
+ total += len(l.Runs[i].Results)
+ }
+ raw := rpMarshal(t, m)
+ used := 100 * float64(len(raw)) / float64(MaxTier0ManifestBytes)
+ t.Logf("findings=%3d bytes=%5d (budget %d, used %.0f%%) inline cards=%3d spilled=%3d",
+ total, len(raw), MaxTier0ManifestBytes, used, len(m.Cards), rpSpilledCards(m))
+
+ if len(raw) > MaxTier0ManifestBytes {
+ t.Fatalf("manifest is %d bytes, over the %d-byte budget", len(raw), MaxTier0ManifestBytes)
+ }
+ if m.Override != nil {
+ t.Fatalf("manifest took an override it did not need: %+v", m.Override)
+ }
+ // Nothing is ever lost, at any size.
+ if got := len(m.Cards) + rpSpilledCards(m); got != total {
+ t.Errorf("%d inline + %d spilled card refs = %d, want one per finding (%d)",
+ len(m.Cards), rpSpilledCards(m), got, total)
+ }
+ if len(m.Spills) == 0 {
+ return
+ }
+ if used < floorPct {
+ t.Errorf("only %.0f%% of the %d-byte Tier-0 budget is used after spilling %d card "+
+ "refs; the read order spills PARTIALLY so the budget freed by a spill is spent "+
+ "on the refs the agent reads first, not left empty",
+ used, MaxTier0ManifestBytes, rpSpilledCards(m))
+ }
+ // The prefix is non-empty exactly when the envelope leaves room,
+ // which at every size here it does.
+ if len(m.Cards) == 0 {
+ t.Errorf("no card ref survived inline at %d findings; that is the all-or-nothing "+
+ "behaviour this test exists to prevent", total)
+ }
+ })
+ }
+}
+
+// rpSpilledCards is how many card refs left Tier 0, per the spill ledger.
+func rpSpilledCards(m Manifest) int {
+ n := 0
+ for _, s := range m.Spills {
+ if s.Field == "anvil/cards" {
+ n += s.Items
+ }
+ }
+ return n
+}
+
+// ---------------------------------------------------------------------------
+// Tier 1 — the token budget, measured
+// ---------------------------------------------------------------------------
+
+func TestEveryTaskCardIsWithinTheTokenBudget(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ if len(cards) == 0 {
+ t.Fatal("no cards")
+ }
+ for _, c := range cards {
+ raw := rpMarshal(t, c)
+ tok := ApproxTokens(len(raw))
+ t.Logf("card %-12s %5d bytes ~%4d tokens", c.FindingID, len(raw), tok)
+ if tok > MaxTier1CardTokens {
+ t.Errorf("card %s is ~%d tokens, over the %d-token Tier-1 budget",
+ c.FindingID, tok, MaxTier1CardTokens)
+ }
+ if len(raw) > MaxTier1CardBytes {
+ t.Errorf("card %s is %d bytes, over the %d-byte budget", c.FindingID, len(raw), MaxTier1CardBytes)
+ }
+ if c.Bytes != len(raw) || c.Tokens != tok {
+ t.Errorf("card %s self-reports %d bytes/%d tokens but measures %d/%d",
+ c.FindingID, c.Bytes, c.Tokens, len(raw), tok)
+ }
+ if c.Override != nil {
+ t.Errorf("card %s took a budget override it should not need: %+v", c.FindingID, c.Override)
+ }
+ }
+}
+
+// A finding whose evidence is enormous still produces a card inside the
+// budget, by spilling in the documented order — code last, because a card
+// without its snippet cannot do its one job.
+func TestOversizedEvidenceSpillsInsteadOfBlowingTheCardBudget(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ r := &l.Runs[0].Results[0] // sast:0001 — the one with a code flow
+ pl := r.Locations[0].PhysicalLocation
+ pl.Region.Snippet.Text = strings.Repeat("x = compute(a, b)\n", 900)
+ pl.ContextRegion.Snippet.Text = strings.Repeat("# context line\n", 900)
+ r.Properties.Reasoning = strings.Repeat("because ", 900)
+ r.CodeFlows[0].ThreadFlows[0].Locations[0].Location.PhysicalLocation.Region.Snippet.Text =
+ strings.Repeat("taint step ", 400)
+ })
+
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ var card *TaskCard
+ for i := range cards {
+ if cards[i].FindingID == "sast:0001" {
+ card = &cards[i]
+ }
+ }
+ if card == nil {
+ t.Fatal("sast:0001 produced no card")
+ }
+
+ raw := rpMarshal(t, card)
+ if len(raw) > MaxTier1CardBytes {
+ t.Fatalf("card is %d bytes after spilling, over the %d-byte budget", len(raw), MaxTier1CardBytes)
+ }
+ if len(card.Spills) == 0 {
+ t.Fatal("an oversized card must spill, not truncate silently")
+ }
+
+ spilled := map[string]TierSpill{}
+ for _, s := range card.Spills {
+ spilled[s.Field] = s
+ blob, ok := card.Blobs[s.Ref]
+ if !ok {
+ t.Fatalf("spill %s references blob %s, which was not retained", s.Field, s.Ref)
+ }
+ if BlobRef(blob) != s.Ref {
+ t.Fatalf("blob for %s does not hash to its own reference", s.Field)
+ }
+ }
+ for _, want := range []string{"/static/reasoning", "/static/taintPath", "/static/context/text"} {
+ if _, ok := spilled[want]; !ok {
+ t.Errorf("expected %s to spill before the code snippet did", want)
+ }
+ }
+ if _, ok := spilled["/static/code"]; ok {
+ // The code may spill, but only after everything else has. Assert the
+ // order rather than forbidding it.
+ if card.Spills[len(card.Spills)-1].Field != "/static/code" {
+ t.Errorf("the code snippet spilled before something else; it must be last")
+ }
+ }
+ // The spilled bytes are recoverable in full.
+ if s, ok := spilled["/static/context/text"]; ok {
+ if got := string(card.Blobs[s.Ref]); !strings.HasPrefix(got, "# context line") {
+ t.Errorf("spilled context does not round-trip: %.40q", got)
+ }
+ }
+}
+
+// Bodies never exceed R.8's inline caps on a card either. The card budget is
+// smaller than both caps, so this holds a fortiori — which is the point: prove
+// it rather than assume the arithmetic.
+func TestCardsNeverInlineABodyPastR8sCaps(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ d := &l.Runs[1].Results[1]
+ d.WebRequest.Body.Text = strings.Repeat("a", 20*1024)
+ d.WebResponse.Body.Text = strings.Repeat("b", 100*1024)
+ })
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ for _, c := range cards {
+ if c.Dynamic == nil {
+ continue
+ }
+ if n := len(c.Dynamic.RequestBody); n > MaxInlineRequestBodyBytes {
+ t.Errorf("card %s inlines %d request-body bytes, cap is %d", c.FindingID, n, MaxInlineRequestBodyBytes)
+ }
+ if n := len(c.Dynamic.ResponseExcerpt); n > MaxInlineResponseBodyBytes {
+ t.Errorf("card %s inlines %d response bytes, cap is %d", c.FindingID, n, MaxInlineResponseBodyBytes)
+ }
+ // A field the inline cap already spilled must not spill a second time
+ // when the token ladder drops it: one field, one Tier-2 reference.
+ seen := map[string]int{}
+ for _, s := range c.Spills {
+ seen[s.Field]++
+ }
+ for field, n := range seen {
+ if n > 1 {
+ t.Errorf("card %s spilled %s %d times", c.FindingID, field, n)
+ }
+ }
+ }
+}
+
+// The cap itself, exercised directly: a body over its limit keeps a prefix, an
+// in-band pointer, and spills the whole thing to a content-addressed blob.
+func TestInlineCapKeepsAPrefixAndSpillsTheRemainder(t *testing.T) {
+ rd := NewReader(RecordMap{})
+ c := &TaskCard{FindingID: "dast:0101", Blobs: map[string][]byte{}}
+ full := strings.Repeat("q", 3*MaxInlineRequestBodyBytes)
+ body := full
+
+ if err := rd.capCardText(c, "/dynamic/requestBody", &body, MaxInlineRequestBodyBytes); err != nil {
+ t.Fatalf("capCardText: %v", err)
+ }
+ if len(body) > MaxInlineRequestBodyBytes {
+ t.Fatalf("capped body is %d bytes, cap is %d", len(body), MaxInlineRequestBodyBytes)
+ }
+ if len(c.Spills) != 1 {
+ t.Fatalf("want exactly one spill, got %d", len(c.Spills))
+ }
+ s := c.Spills[0]
+ if !strings.Contains(body, s.Ref) {
+ t.Errorf("the truncated body must carry the sha256: reference in band, got %.80q", body)
+ }
+ if got := string(c.Blobs[s.Ref]); got != full {
+ t.Errorf("the spilled blob is not the full body (%d of %d bytes)", len(got), len(full))
+ }
+ if !strings.HasPrefix(s.Ref, "sha256:") || len(s.Ref) != len("sha256:")+FingerprintDigestHexLen {
+ t.Errorf("spill reference %q is not sha256:<64 hex>", s.Ref)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The host gate
+// ---------------------------------------------------------------------------
+
+// 00-SPINE.md S7: the host agent is read-only. The record's validator already
+// rejects a host finding marked remediable; this proves the READ PATH does not
+// hand one out as actionable even when the record is wrong.
+func TestHostFindingIsNeverHandedOutAsActionable(t *testing.T) {
+ l := rpFixture(t, nil)
+
+ // The well-formed case first.
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ var host *TaskCard
+ for i := range cards {
+ if cards[i].FindingID == "host:0006" {
+ host = &cards[i]
+ }
+ }
+ if host == nil {
+ t.Fatal("the host finding produced no card; it must still be REPORTED, just never actioned")
+ }
+ if host.Actionable || host.RemediableByAgent {
+ t.Errorf("host card is actionable=%t remediable=%t, both must be false",
+ host.Actionable, host.RemediableByAgent)
+ }
+ if len(host.ActionBlockers) == 0 || !strings.Contains(host.ActionBlockers[0], "read-only") {
+ t.Errorf("the host card must say why it is not actionable, got %q", host.ActionBlockers)
+ }
+ for _, c := range ActionableTaskCards(cards) {
+ if c.FindingID == "host:0006" {
+ t.Fatal("ActionableTaskCards handed out the host finding")
+ }
+ }
+
+ // Now the malformed case: a record claiming a host finding is remediable.
+ // contract.go rejects it, so it is built directly rather than through
+ // rpFixture's validate step — which is exactly the situation the read
+ // path's own clamp exists for.
+ broken := rpFixtureLog()
+ for i := range broken.Runs[0].Results {
+ r := &broken.Runs[0].Results[i]
+ if r.Properties.FindingID == "host:0006" {
+ r.Properties.RemediableByAgent = true
+ }
+ }
+ if err := broken.Validate(); err == nil {
+ t.Fatal("contract.go should still reject a remediable host finding; the fixture is not testing what it claims")
+ }
+ if err := MaskRecord(broken); err != nil {
+ t.Fatalf("masking: %v", err)
+ }
+ brokenCards, err := rpReader(t, broken).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards on the malformed record: %v", err)
+ }
+ for _, c := range brokenCards {
+ if c.FindingID != "host:0006" {
+ continue
+ }
+ if c.RemediableByAgent {
+ t.Error("the read path propagated remediableByAgent=true for a HOST finding")
+ }
+ if c.Actionable {
+ t.Error("the read path handed out a HOST finding as actionable")
+ }
+ }
+ for _, c := range ActionableTaskCards(brokenCards) {
+ if IsHostFinding(&broken.Runs[0].Results[5]) && c.FindingID == "host:0006" {
+ t.Fatal("ActionableTaskCards handed out a host finding from a malformed record")
+ }
+ }
+}
+
+func TestVerdictGatesActionability(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ want := map[string]bool{
+ "sast:0001": true, // true_positive
+ "sast:0003": false, // insufficient_context -> report-only
+ "sast:0007": false, // false_positive -> dropped by the pipeline
+ "host:0006": false, // host -> read-only agent
+ }
+ got := map[string]bool{}
+ for _, c := range cards {
+ got[c.FindingID] = c.Actionable
+ }
+ for id, w := range want {
+ if got[id] != w {
+ t.Errorf("card %s actionable = %t, want %t", id, got[id], w)
+ }
+ }
+ // Report-only findings are still CARDS. research/24: never silently
+ // dropped.
+ if len(cards) != 9 {
+ t.Errorf("got %d cards, want all 9 findings represented", len(cards))
+ }
+}
+
+// ---------------------------------------------------------------------------
+// What a card must carry
+// ---------------------------------------------------------------------------
+
+func TestCardCarriesTheNonNegotiableHandoffFields(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ byID := map[string]TaskCard{}
+ for _, c := range cards {
+ byID[c.FindingID] = c
+ }
+
+ sast := byID["sast:0001"]
+ checks := []struct {
+ field string
+ ok bool
+ }{
+ {"finding_id", sast.FindingID == "sast:0001"},
+ {"fingerprint.anvilFindingId", len(sast.Fingerprint.AnvilFindingID) == FingerprintDigestHexLen},
+ {"fingerprint.primary_location_line_hash", sast.Fingerprint.PrimaryLocationLineHash != ""},
+ {"evidence_class", sast.EvidenceClass == EvidenceClassSastReachable},
+ {"locus.path", sast.Locus.Path == "app/db.py"},
+ {"locus.start_line", sast.Locus.StartLine == 412},
+ {"locus.end_line", sast.Locus.EndLine == 414},
+ {"locus.enclosing_symbol", sast.Locus.EnclosingSymbol == "app.db.authenticate"},
+ {"locus.proximity_class", sast.Locus.ProximityClass == "same_symbol"},
+ {"advisory_excerpt", sast.Advisory != nil && sast.Advisory.Excerpt != ""},
+ {"group_id (reserved, empty on a fresh record)", sast.GroupID == ""},
+ {"dast.reproduction (via the cluster peer)", sast.Dynamic != nil && sast.Dynamic.Curl != ""},
+ {"static code", sast.Static != nil && sast.Static.Code != ""},
+ {"taint path", sast.Static != nil && len(sast.Static.TaintPath) == 2},
+ {"constraints", sast.Constraints != nil && sast.Constraints.TestCommand != ""},
+ {"writeBackTo", sast.WriteBackTo == "/runs/0/results/0/fixes"},
+ {"consumption class", sast.ConsumptionClass == ConsumptionClassStaticOnly},
+ }
+ for _, c := range checks {
+ if !c.ok {
+ t.Errorf("card sast:0001 is missing or wrong: %s", c.field)
+ }
+ }
+
+ sca := byID["sca:0005"]
+ if sca.Risk == nil || !sca.Risk.KevMember || sca.Risk.EpssScore == nil {
+ t.Errorf("card sca:0005 must carry risk.*: %+v", sca.Risk)
+ }
+
+ dast := byID["dast:0102"]
+ if dast.Dynamic == nil {
+ t.Fatal("card dast:0102 has no dynamic section")
+ }
+ if dast.Dynamic.Env == nil || dast.Dynamic.Env.Sanitizers == nil {
+ t.Error("a reproduction must carry its sanitizer state: a crash under ASan is a different claim")
+ }
+ if dast.Dynamic.ExpectedAfterFix == nil || dast.Dynamic.InjectionPoint != "body:username" {
+ t.Errorf("dynamic section is incomplete: %+v", dast.Dynamic)
+ }
+ if dast.ConsumptionClass != ConsumptionClassRequiresDynamicConfirmation {
+ t.Errorf("a dast_confirmed finding must be %q, got %q",
+ ConsumptionClassRequiresDynamicConfirmation, dast.ConsumptionClass)
+ }
+ if dast.Trust.Default != TrustUntrusted {
+ t.Errorf("a card's default trust must be %q, got %q", TrustUntrusted, dast.Trust.Default)
+ }
+ if got := dast.Trust.Fields["/task"]; got != TrustAnvilGenerated {
+ t.Errorf("the card's own task text is Anvil-generated, got %q", got)
+ }
+}
+
+// Link, never merge. Both cluster members keep their own card; each card
+// carries the peer's evidence as a convenience, and neither claims a merge.
+func TestClusterMembersAreLinkedNeverMerged(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ byID := map[string]TaskCard{}
+ for _, c := range cards {
+ byID[c.FindingID] = c
+ }
+ sast, dast := byID["sast:0001"], byID["dast:0101"]
+
+ for _, c := range []TaskCard{sast, dast} {
+ if c.Correlation == nil {
+ t.Fatalf("card %s lost its correlation", c.FindingID)
+ }
+ if c.Correlation.Merged {
+ t.Errorf("card %s claims merged=true", c.FindingID)
+ }
+ if c.Correlation.ClusterID != rpClusterID {
+ t.Errorf("card %s cluster = %q", c.FindingID, c.Correlation.ClusterID)
+ }
+ if !c.Correlation.Verified {
+ t.Errorf("card %s: a stack-trace signal is present, so verified should survive", c.FindingID)
+ }
+ }
+ if len(sast.Correlation.Peers) != 1 || sast.Correlation.Peers[0] != "dast:0101" {
+ t.Errorf("sast peers = %q", sast.Correlation.Peers)
+ }
+ if len(dast.Correlation.Peers) != 1 || dast.Correlation.Peers[0] != "sast:0001" {
+ t.Errorf("dast peers = %q", dast.Correlation.Peers)
+ }
+ // The SAST card owns the file and line; the DAST card owns the proof; each
+ // card sees both, and the record still holds two independent results.
+ if sast.Static == nil || sast.Dynamic == nil {
+ t.Error("the SAST card should carry its own static evidence and the peer's reproduction")
+ }
+ if dast.Static == nil || dast.Dynamic == nil {
+ t.Error("the DAST card should carry its own reproduction and the peer's file and line")
+ }
+ if dast.Locus.Path != "app/db.py" {
+ t.Errorf("the DAST card's locus should come from the SAST peer, got %q", dast.Locus.Path)
+ }
+ if n := len(l.Runs[0].Results) + len(l.Runs[1].Results); n != 9 {
+ t.Errorf("the record must still hold both findings independently, got %d results", n)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Gates the read path will not open
+// ---------------------------------------------------------------------------
+
+// R.6's read gate: only HalfStatusSealed opens a half. An unsealed half yields
+// no cards, and the manifest still SAYS the half exists — otherwise "no DAST
+// cards" and "no dynamic vulnerabilities" become the same observation, which
+// is research/23 Risk #1.
+func TestUnsealedHalfYieldsNoCardsButIsStillReported(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ l.Runs[1].Properties.Status = HalfStatusRunning
+ l.Runs[1].Properties.SealedAt = nil
+ l.Properties.State = StateSastSealed
+ l.Properties.DastStatus = DastStatusRunning
+ })
+
+ rd := rpReader(t, l)
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ for _, c := range cards {
+ if c.Half == HalfDast {
+ t.Errorf("card %s came from a half whose status is %q, not %q",
+ c.FindingID, HalfStatusRunning, HalfStatusSealed)
+ }
+ }
+
+ // The SAST member of the cluster survives — the correlation is a fact
+ // recorded on ITS result, and link-never-merge means it does not depend on
+ // the peer being present. What must NOT happen is the peer's evidence
+ // reaching the card through the cluster projection: that would walk around
+ // the seal gate rather than through it.
+ var clustered *TaskCard
+ for i := range cards {
+ if cards[i].FindingID == "sast:0001" {
+ clustered = &cards[i]
+ }
+ }
+ if clustered == nil {
+ t.Fatal("the cluster's readable member vanished when its peer became unreadable")
+ }
+ if clustered.Bucket != BucketClusters || clustered.ClusterID != rpClusterID {
+ t.Errorf("clustered card = bucket %q cluster %q", clustered.Bucket, clustered.ClusterID)
+ }
+ if clustered.Dynamic != nil {
+ t.Error("the card carries the unsealed peer's dynamic evidence: the seal gate was bypassed via the cluster")
+ }
+
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ var dastHalf *ManifestHalf
+ for i := range m.Halves {
+ if m.Halves[i].Half == HalfDast {
+ dastHalf = &m.Halves[i]
+ }
+ }
+ if dastHalf == nil {
+ t.Fatal("the manifest dropped the unreadable half entirely; the consumer must be able to see it exists")
+ }
+ if dastHalf.Readable || dastHalf.Cards != 0 {
+ t.Errorf("dast half readable=%t cards=%d, want false/0", dastHalf.Readable, dastHalf.Cards)
+ }
+ if dastHalf.Results != 2 {
+ t.Errorf("the manifest must report the %d withheld results, got %d", 2, dastHalf.Results)
+ }
+ if m.DynamicallyScannedClean {
+ t.Error("dastStatus is running; nothing may report this target as dynamically scanned clean")
+ }
+}
+
+// Only completed_clean means "dynamically scanned, nothing found". Every other
+// value, including the ones that also carry zero findings, must not be read
+// that way.
+func TestManifestNeverClaimsScannedCleanForAnyOtherDastStatus(t *testing.T) {
+ for _, s := range DastStatusValues() {
+ s := s
+ t.Run(string(s), func(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ l.Properties.DastStatus = s
+ if s == DastStatusNotRun || s == DastStatusSkippedNoManifest {
+ // Both mean the DAST half produced nothing; the record's
+ // own state machine then requires the half to be gone.
+ l.Runs = l.Runs[:1]
+ }
+ })
+ m, err := rpReader(t, l).BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ want := s == DastStatusCompletedClean
+ if m.DynamicallyScannedClean != want {
+ t.Errorf("dastStatus %q: dynamicallyScannedClean = %t, want %t",
+ s, m.DynamicallyScannedClean, want)
+ }
+ if m.DastStatus != s {
+ t.Errorf("the manifest must carry dastStatus verbatim, got %q", m.DastStatus)
+ }
+ })
+ }
+}
+
+// The read path feeds a repo-credentialed agent. An unmasked record does not
+// get through it.
+func TestUnmaskedRecordIsRefused(t *testing.T) {
+ l := rpFixtureLog()
+ l.Runs[1].Results[0].WebRequest.Headers["Authorization"] = "Bearer ghp_averyrealisticlookingtoken"
+
+ rd := NewReader(RecordMap{rpAuditID: l})
+ if _, err := rd.BuildTaskCards(rpAuditID); err == nil {
+ t.Fatal("BuildTaskCards accepted a record carrying a live bearer token")
+ } else if !strings.Contains(err.Error(), "masking") && !strings.Contains(err.Error(), "unmasked") {
+ t.Logf("refusal message: %v", err)
+ }
+ if _, err := rd.BuildManifest(rpAuditID); err == nil {
+ t.Fatal("BuildManifest accepted an unmasked record")
+ }
+
+ if err := MaskRecord(l); err != nil {
+ t.Fatalf("masking: %v", err)
+ }
+ if _, err := rd.BuildTaskCards(rpAuditID); err != nil {
+ t.Fatalf("after masking, the same record must be accepted: %v", err)
+ }
+}
+
+func TestSourceMismatchIsRefused(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := NewReader(RecordMap{"some-other-audit": l})
+ if _, err := rd.BuildManifest("some-other-audit"); err == nil {
+ t.Fatal("a source returning the wrong audit must be refused: the audit id is the join key")
+ }
+ if _, err := NewReader(RecordMap{}).BuildManifest(rpAuditID); err == nil {
+ t.Fatal("a missing audit must be an error, not an empty manifest")
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The card is derived; the record wins
+// ---------------------------------------------------------------------------
+
+func TestCheckAgainstRecordAcceptsDerivedCardsAndRejectsContradictions(t *testing.T) {
+ l := rpFixture(t, nil)
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+
+ byID := map[string]*Result{}
+ for ri := range l.Runs {
+ for si := range l.Runs[ri].Results {
+ r := &l.Runs[ri].Results[si]
+ byID[r.Properties.FindingID] = r
+ }
+ }
+ for _, c := range cards {
+ if err := c.CheckAgainstRecord(byID[c.FindingID]); err != nil {
+ t.Errorf("a freshly derived card disagrees with its own record: %v", err)
+ }
+ }
+
+ // A card may be LESS permissive than the record. That is the host clamp,
+ // and it is not a contradiction.
+ lenient := cards[0]
+ lenient.Actionable = false
+ lenient.RemediableByAgent = false
+ if err := lenient.CheckAgainstRecord(byID[lenient.FindingID]); err != nil {
+ t.Errorf("withholding an action must be legal, got: %v", err)
+ }
+
+ // A card may never be MORE permissive.
+ host := byID["host:0006"]
+ forged := TaskCard{
+ FindingID: "host:0006", EvidenceClass: host.Properties.EvidenceClass,
+ Verdict: host.Properties.Verdict, Half: host.Properties.Half,
+ Confidence: host.Properties.Confidence,
+ Fingerprint: CardFingerprint{AnvilFindingID: host.PartialFingerprints[PartialFingerprintAnvilFindingID]},
+ RemediableByAgent: true, Actionable: true,
+ }
+ err = forged.CheckAgainstRecord(host)
+ if err == nil {
+ t.Fatal("a card granting an action the record forbids must be reported")
+ }
+ if !strings.Contains(err.Error(), "read-only") || !strings.Contains(err.Error(), "the record wins") {
+ t.Errorf("the error should name the rule and the precedence, got: %v", err)
+ }
+
+ // A drifted field is a contradiction, not a rounding difference.
+ drifted := cards[0]
+ drifted.EvidenceClass = EvidenceClassHost
+ if err := drifted.CheckAgainstRecord(byID[drifted.FindingID]); err == nil {
+ t.Error("a card whose evidenceClass differs from the record must be reported")
+ }
+}
+
+// The manifest's read order and the cards are the same order, by construction.
+// They are built by two calls and a consumer pairs them by index, so a drift
+// between them would silently hand the agent one order and one set of paths.
+func TestManifestReadOrderMatchesTheCards(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := rpReader(t, l)
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ if len(m.Cards) != len(cards) {
+ t.Fatalf("manifest lists %d cards, BuildTaskCards returned %d", len(m.Cards), len(cards))
+ }
+ for i := range cards {
+ if m.Cards[i].FindingID != cards[i].FindingID {
+ t.Fatalf("position %d: manifest says %q, cards say %q",
+ i, m.Cards[i].FindingID, cards[i].FindingID)
+ }
+ if m.Cards[i].Bucket != cards[i].Bucket || m.Cards[i].Actionable != cards[i].Actionable {
+ t.Errorf("position %d: manifest and card disagree on bucket/actionable", i)
+ }
+ if want := DefaultTaskCardPrefix + SanitizeCardFilename(cards[i].FindingID) + ".json"; m.Cards[i].Card != want {
+ t.Errorf("card path = %q, want %q", m.Cards[i].Card, want)
+ }
+ if cards[i].Position != i {
+ t.Errorf("card %s reports position %d, is at %d", cards[i].FindingID, cards[i].Position, i)
+ }
+ }
+ // A colon is not a legal Windows path character, and finding ids carry
+ // one.
+ if strings.ContainsAny(m.Cards[0].Card, `:*?"<>|`) {
+ t.Errorf("card path %q is not filesystem-safe", m.Cards[0].Card)
+ }
+}
+
+func TestApproxTokensIsPessimistic(t *testing.T) {
+ // The estimate must never UNDER-count: an under-counted card blows the
+ // agent's context with no error anywhere.
+ if ApproxBytesPerToken > 3 {
+ t.Errorf("ApproxBytesPerToken = %d; anything above 3 makes the budget check optimistic",
+ ApproxBytesPerToken)
+ }
+ if ApproxTokens(0) != 0 || ApproxTokens(1) != 1 || ApproxTokens(3) != 1 || ApproxTokens(4) != 2 {
+ t.Errorf("ApproxTokens rounds wrongly: %d %d %d %d",
+ ApproxTokens(0), ApproxTokens(1), ApproxTokens(3), ApproxTokens(4))
+ }
+ if MaxTier1CardBytes != MaxTier1CardTokens*ApproxBytesPerToken {
+ t.Errorf("MaxTier1CardBytes is not derived from the contract's token budget")
+ }
+}
+
+// Every enum-valued field the read path emits comes from contract.go. This
+// catches the failure mode that produced the ten section-6 defects: a second
+// copy of a literal is a second definition.
+func TestReadPathEmitsOnlyFrozenEnumLiterals(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := rpReader(t, l)
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ if err := ValidateState(string(m.State)); err != nil {
+ t.Error(err)
+ }
+ if err := ValidateDastStatus(string(m.DastStatus)); err != nil {
+ t.Error(err)
+ }
+ if err := ValidateTargetProvenance(string(m.Target.Provenance)); err != nil {
+ t.Error(err)
+ }
+ if err := ValidateTargetProvisioning(string(m.Target.Provisioning)); err != nil {
+ t.Error(err)
+ }
+ for _, h := range m.Halves {
+ if err := ValidateHalf(string(h.Half)); err != nil {
+ t.Error(err)
+ }
+ if err := ValidateHalfStatus(string(h.Status)); err != nil {
+ t.Error(err)
+ }
+ }
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ for _, c := range cards {
+ if err := ValidateVerdict(string(c.Verdict)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ if err := ValidateEvidenceClass(string(c.EvidenceClass)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ if err := ValidateHalf(string(c.Half)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ if err := ValidateConsumptionClass(string(c.ConsumptionClass)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ if err := ValidateTrust(string(c.Trust.Default)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ if c.Dynamic != nil && c.Dynamic.ObservedSignal != "" {
+ if err := ValidateEvidenceSignal(string(c.Dynamic.ObservedSignal)); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ }
+ if c.Correlation != nil {
+ for _, s := range c.Correlation.Signals {
+ if err := ValidateCorrelationSignal(s); err != nil {
+ t.Errorf("card %s: %v", c.FindingID, err)
+ }
+ }
+ }
+ }
+}
+
+// ===========================================================================
+// THE READ GATE — CRITIQUE-03 B1/M1, and the test that is supposed to stop
+// bypass number five
+// ===========================================================================
+//
+// FOUR separate authors have now written their own answer to "may a consumer
+// read this half's results?" and four got it wrong in four different ways
+// (sealing.go's read-gate section lists them: CRITIQUE-02 M2 and M3,
+// CRITIQUE-03 B1 and M1). Patching each bypass as it is found is a losing
+// game, so the countermeasure is this section rather than any one fix:
+//
+// - sealing.go answers the question in ONE function body, halfReadRefusal,
+// reached through HalfReadGate / HalfSeal.Readable;
+// - TestEveryResultBearingEntryPointIsGated enumerates every exported entry
+// point that can hand out a half's results and asserts each one refuses an
+// unsealed half AND an expired audit;
+// - gateAuditedEntryPoints, the list that test drives off, is reconciled
+// against the package source, so an entry point that returns results and
+// is not in the list is a test failure rather than a silence.
+//
+// If you are here because that test failed on a function you just added:
+// route it through HalfReadGate and add it to gateAuditedEntryPoints with a
+// probe. If you believe it genuinely cannot leak a half's results, add it with
+// an `exempt` reason — but write the reason, because "it obviously cannot" is
+// what the last four authors also believed.
+
+// gateScenario is one record-plus-Sealer pair in which NO half is readable.
+// Every probe below must hand out ZERO results for every scenario.
+//
+// The record and the Sealer describe the SAME audit, because the two halves of
+// this package's read surface — the record-side projections (readpath.go,
+// taskcard.go, sarif_github.go) and the in-memory Sealer — are exactly the two
+// places the gate has been bypassed, and a test that covered only one would
+// have missed two of the four historical bugs.
+type gateScenario struct {
+ name string
+ auditID string
+ log *SARIFLog
+ sealer *Sealer
+}
+
+// gateProbe calls one exported entry point and returns the number of a half's
+// results it handed out. Anything above zero is a bypass.
+type gateProbe func(t *testing.T, sc gateScenario) int
+
+// gateEntry is one exported entry point that CAN return a half's results.
+//
+// THE LIST BELOW IS THE BEHAVIOURAL HALF OF THE GUARD: it RUNS each entry
+// point against every state in which nothing may be read and counts what came
+// back. It is maintained by hand, and by hand alone it would go stale the
+// moment someone adds an entry point — which is what the SOURCE half is for:
+//
+// TestResultReachingEntryPointsAreGated reads the package's AST and
+// fails when an exported entry point can reach a half's results without
+// the same call graph reaching the gate. It replaced a whitelist of return
+// TYPES that three synthesised leaks walked straight past.
+// TestReadabilityAnsweringEntryPointsAreProbed
+// fails when an exported entry point hands out a HalfSeal or an AuditSeal
+// — the readability answer itself — without a line in this table.
+//
+// So: adding a new way to read a half's results, without adding a line here,
+// fails the suite.
+type gateEntry struct {
+ // name is "Func" or "Recv.Method", exactly as the source spells it. It is
+ // what the source reconciliation matches on.
+ name string
+
+ // probe is the assertion. Exactly one of probe and exempt is set.
+ probe gateProbe
+
+ // exempt records why this entry point cannot hand out an ungated result,
+ // for the ones where that is structurally true. A reason, never a shrug.
+ exempt string
+}
+
+// gateAuditedEntryPoints is the maintained list. See gateEntry.
+func gateAuditedEntryPoints() []gateEntry {
+ return []gateEntry{
+ // ---- readpath.go / taskcard.go: the record-side projections -------
+ {name: "Reader.BuildManifest", probe: func(t *testing.T, sc gateScenario) int {
+ m, err := NewReader(RecordMap{sc.auditID: sc.log}).BuildManifest(sc.auditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ return gateManifestExposure(t, m)
+ }},
+ {name: "Reader.ManifestFromLog", probe: func(t *testing.T, sc gateScenario) int {
+ m, err := NewReader(RecordMap{sc.auditID: sc.log}).ManifestFromLog(sc.log)
+ if err != nil {
+ t.Fatalf("ManifestFromLog: %v", err)
+ }
+ return gateManifestExposure(t, m)
+ }},
+ {name: "Reader.BuildTaskCards", probe: func(t *testing.T, sc gateScenario) int {
+ cards, err := NewReader(RecordMap{sc.auditID: sc.log}).BuildTaskCards(sc.auditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ return len(cards)
+ }},
+ {name: "Reader.CardsFromLog", probe: func(t *testing.T, sc gateScenario) int {
+ cards, err := NewReader(RecordMap{sc.auditID: sc.log}).CardsFromLog(sc.log)
+ if err != nil {
+ t.Fatalf("CardsFromLog: %v", err)
+ }
+ return len(cards)
+ }},
+ {
+ name: "ActionableTaskCards",
+ exempt: "a filter over a []TaskCard the caller already holds. It cannot reach a " +
+ "record or a Sealer, so the only cards it can return are cards a gated entry " +
+ "point above already emitted. If it ever grows a RecordSource parameter, " +
+ "delete this exemption and give it a probe.",
+ },
+
+ // ---- sarif_github.go: the most externally visible consumer --------
+ {name: "ProjectForGitHub", probe: func(t *testing.T, sc gateScenario) int {
+ files, err := ProjectForGitHub(sc.log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ n := 0
+ for _, f := range files {
+ n += f.ResultCount
+ for _, run := range f.Log.Runs {
+ n += len(run.Results)
+ }
+ }
+ // The loss must also be COUNTABLE, not merely absent: CRITIQUE-03
+ // B1's probe found zero drops recorded in every unsealed case, so
+ // the leak was invisible as well as permitted.
+ loss := GitHubLossOf(files)
+ if loss == nil {
+ t.Fatal("no loss ledger reachable from a fully-withheld projection")
+ }
+ if loss.DropCounts[GitHubDropHalfNotReadable] != loss.SourceResultCount {
+ t.Errorf("%d of %d withheld results were ledgered under %q; a withheld result "+
+ "that is not counted is a silent drop\n%s",
+ loss.DropCounts[GitHubDropHalfNotReadable], loss.SourceResultCount,
+ GitHubDropHalfNotReadable, loss.Summary())
+ }
+ return n
+ }},
+
+ // ---- sealing.go: the in-memory consumer gate ----------------------
+ {name: "Sealer.ReadHalf", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ for _, half := range HalfValues() {
+ seal, err := sc.sealer.ReadHalf(sc.auditID, half)
+ if err == nil {
+ n++
+ } else if !errors.Is(err, ErrHalfNotSealed) {
+ t.Errorf("ReadHalf(%s) refused with %v, want a *ReadGateError", half, err)
+ }
+ if seal.Readable() {
+ n++
+ }
+ }
+ return n
+ }},
+ {name: "ReadHalf", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ for _, half := range HalfValues() {
+ seal, err := ReadHalf(sc.auditID, half)
+ if err == nil {
+ n++
+ }
+ if seal.Readable() {
+ n++
+ }
+ }
+ return n
+ }},
+ {name: "Sealer.Inspect", probe: func(t *testing.T, sc gateScenario) int {
+ s, ok := sc.sealer.Inspect(sc.auditID)
+ if !ok {
+ t.Fatalf("Inspect(%q) does not know the audit the scenario registered", sc.auditID)
+ }
+ return gateReadableHalves(s)
+ }},
+ {name: "Inspect", probe: func(t *testing.T, sc gateScenario) int {
+ s, ok := Inspect(sc.auditID)
+ if !ok {
+ t.Fatalf("package Inspect(%q) does not know the audit the scenario registered", sc.auditID)
+ }
+ return gateReadableHalves(s)
+ }},
+ {name: "Sealer.BeginAudit", probe: gateProbeBeginAudit},
+ {name: "BeginAudit", probe: gateProbeBeginAudit},
+
+ // ---- entry points the source reconciliation does not require, listed
+ // anyway because they answer the readability question even though
+ // their return types are plain bools. The table may be a superset of
+ // what the source demands; it may never be a subset.
+ {name: "Sealer.ReadyForConsumption", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ sast, dast := sc.sealer.ReadyForConsumption(sc.auditID)
+ if sast {
+ n++
+ }
+ if dast {
+ n++
+ }
+ return n
+ }},
+ {name: "ReadyForConsumption", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ sast, dast := ReadyForConsumption(sc.auditID)
+ if sast {
+ n++
+ }
+ if dast {
+ n++
+ }
+ return n
+ }},
+ {name: "HalfSeal.Readable", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ for i := range sc.log.Runs {
+ if halfSealOfRun(sc.log, &sc.log.Runs[i]).Readable() {
+ n++
+ }
+ }
+ return n
+ }},
+ {name: "HalfReadGate", probe: func(t *testing.T, sc gateScenario) int {
+ n := 0
+ for i := range sc.log.Runs {
+ if HalfReadGate(sc.auditID, halfSealOfRun(sc.log, &sc.log.Runs[i])) == nil {
+ n++
+ }
+ }
+ return n
+ }},
+ }
+}
+
+// gateProbeBeginAudit covers both spellings of BeginAudit. A freshly begun
+// audit has produced nothing, so neither half may be readable — including the
+// DAST half a core-`anvil` install seals immediately as skipped, which is
+// terminal and unreadable at once.
+func gateProbeBeginAudit(t *testing.T, sc gateScenario) int {
+ t.Helper()
+ s := NewSealer()
+ seal, err := s.BeginAudit(AuditConfig{
+ AuditID: sc.auditID + "-begin", StartedAt: rpTime(), ClaimTimeoutSeconds: 3600,
+ })
+ if err != nil {
+ t.Fatalf("BeginAudit: %v", err)
+ }
+ return gateReadableHalves(seal)
+}
+
+func gateReadableHalves(s AuditSeal) int {
+ n := 0
+ if s.Sast.Readable() {
+ n++
+ }
+ if s.Dast.Readable() {
+ n++
+ }
+ return n
+}
+
+// gateManifestExposure counts what a manifest handed out: card refs (each of
+// which points at a Tier-1 card built from a half's results) plus any half the
+// manifest CLAIMS is readable.
+//
+// It deliberately does NOT count ManifestHalf.Results. The manifest must keep
+// reporting that an unreadable half exists and how many results it is holding
+// back — otherwise "no DAST findings" and "the DAST half never sealed" arrive
+// as the same observation, which is research/23 Risk #1 and is a worse bug
+// than the one this test guards.
+func gateManifestExposure(t *testing.T, m Manifest) int {
+ t.Helper()
+ n := len(m.Cards)
+ for _, h := range m.Halves {
+ if h.Readable {
+ n++
+ }
+ if !h.Readable && h.ReadRefusal == "" {
+ t.Errorf("half %s is unreadable but the manifest gives no reason; "+
+ "'never sealed' and 'the audit expired' must not arrive as the same observation", h.Half)
+ }
+ if h.Cards != 0 {
+ t.Errorf("half %s reports %d cards", h.Half, h.Cards)
+ }
+ }
+ return n
+}
+
+// gateScenarios builds the states in which NOTHING may be read. Each one is a
+// real record contract.go's own Validate() accepts, plus a Sealer driven to
+// the matching state through its real transitions.
+func gateScenarios(t *testing.T) []gateScenario {
+ t.Helper()
+
+ // (1) Neither half has sealed. This is the arm CRITIQUE-03 B1 found the
+ // GitHub projection ignoring entirely.
+ unsealed := gateScenario{name: "no half has sealed", auditID: rpAuditID + "-unsealed"}
+ unsealed.log = rpFixture(t, func(l *SARIFLog) {
+ l.Properties.AuditID = unsealed.auditID
+ for i := range l.Runs {
+ l.Runs[i].Properties.Status = HalfStatusRunning
+ l.Runs[i].Properties.SealedAt = nil
+ l.Runs[i].AutomationDetails.CorrelationGUID = unsealed.auditID
+ }
+ l.Properties.State = StateCollecting
+ l.Properties.DastStatus = DastStatusRunning
+ })
+ unsealed.sealer = gateSealer(t, unsealed.auditID, false)
+
+ // (2) Both halves are TERMINAL but neither is READABLE. A failed half is
+ // not a clean half; §6 keeps completed_failed distinct from
+ // completed_partial precisely because a half that CRASHED is not a half
+ // that covered part of the surface.
+ failed := gateScenario{name: "both halves failed", auditID: rpAuditID + "-failed"}
+ failed.log = rpFixture(t, func(l *SARIFLog) {
+ l.Properties.AuditID = failed.auditID
+ for i := range l.Runs {
+ l.Runs[i].Properties.Status = HalfStatusFailed
+ l.Runs[i].Properties.SealedAt = nil
+ l.Runs[i].AutomationDetails.CorrelationGUID = failed.auditID
+ }
+ l.Properties.State = StateCollecting
+ l.Properties.DastStatus = DastStatusCompletedFailed
+ })
+ failed.sealer = gateSealer(t, failed.auditID, false)
+
+ // (3) Both halves sealed cleanly and the audit then EXPIRED. This is the
+ // arm CRITIQUE-03 M1 found readpath.go ignoring: the claim window has
+ // closed, the reaper drops the payload, and the handoff rows behind any
+ // card are subject to ReclaimExpired — so an agent handed an actionable
+ // card here has nowhere legal to land its work.
+ expired := gateScenario{name: "the audit expired holding two sealed halves", auditID: rpAuditID + "-expired"}
+ expired.log = rpFixture(t, func(l *SARIFLog) {
+ l.Properties.AuditID = expired.auditID
+ for i := range l.Runs {
+ l.Runs[i].AutomationDetails.CorrelationGUID = expired.auditID
+ }
+ l.Properties.State = StateExpired
+ })
+ expired.sealer = gateSealer(t, expired.auditID, true)
+
+ return []gateScenario{unsealed, failed, expired}
+}
+
+// gateSealer registers auditID on a fresh Sealer AND on the package-level
+// DefaultSealer — the package-level ReadHalf/Inspect/ReadyForConsumption are
+// exported entry points in their own right and are probed through the real
+// global, not a stand-in.
+//
+// expire drives the audit through seal-both-then-ExpireIfDue using a claim
+// clock that is already spent, so no clock is patched and no other test's view
+// of time moves.
+func gateSealer(t *testing.T, auditID string, expire bool) *Sealer {
+ t.Helper()
+ cfg := AuditConfig{AuditID: auditID, StartedAt: rpTime(), ClaimTimeoutSeconds: 3600, DastEnabled: true}
+ if expire {
+ cfg.StartedAt = time.Now().Add(-72 * time.Hour)
+ cfg.ClaimTimeoutSeconds = 1
+ }
+
+ local := NewSealer()
+ for _, s := range []*Sealer{local, DefaultSealer()} {
+ if _, err := s.BeginAudit(cfg); err != nil {
+ t.Fatalf("BeginAudit(%q): %v", auditID, err)
+ }
+ if !expire {
+ continue
+ }
+ for _, half := range HalfValues() {
+ if err := s.SealHalf(auditID, half, HalfStatusSealed); err != nil {
+ t.Fatalf("SealHalf(%q, %s): %v", auditID, half, err)
+ }
+ }
+ done, err := s.ExpireIfDue(auditID)
+ if err != nil || !done {
+ t.Fatalf("ExpireIfDue(%q) = %t, %v; the scenario needs an expired audit", auditID, done, err)
+ }
+ }
+ t.Cleanup(func() { DefaultSealer().Forget(auditID) })
+ return local
+}
+
+// TestEveryResultBearingEntryPointIsGated is the regression test for the
+// PATTERN rather than for any one of the four bypasses. Every exported way to
+// obtain a half's results, driven against every state in which no half may be
+// read, must hand out nothing.
+func TestEveryResultBearingEntryPointIsGated(t *testing.T) {
+ entries := gateAuditedEntryPoints()
+ probed := 0
+ for _, sc := range gateScenarios(t) {
+ for _, e := range entries {
+ if e.probe == nil {
+ continue
+ }
+ probed++
+ t.Run(sc.name+"/"+e.name, func(t *testing.T) {
+ if n := e.probe(t, sc); n != 0 {
+ t.Errorf("%s handed out %d of the audit's results while %s. "+
+ "The read gate is sealing.go's HalfReadGate and it is not this "+
+ "entry point's to re-derive; see sealing.go's read-gate section.",
+ e.name, n, sc.name)
+ }
+ })
+ }
+ }
+ if probed == 0 {
+ t.Fatal("no entry point was probed; the table drives the whole test and it is empty")
+ }
+}
+
+// TestReadGateOpensOnASealedAudit is the other half of the assertion above: a
+// gate that refuses everything is not a gate, it is a wall. The same surfaces,
+// against a fully sealed live audit, must hand results OUT.
+func TestReadGateOpensOnASealedAudit(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := NewReader(RecordMap{rpAuditID: l})
+
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ if len(m.Cards) == 0 {
+ t.Error("a fully sealed audit produced no read order; the gate refuses everything")
+ }
+ for _, h := range m.Halves {
+ if !h.Readable {
+ t.Errorf("half %s is not readable on a fully sealed, live audit", h.Half)
+ }
+ if h.ReadRefusal != "" {
+ t.Errorf("half %s is readable but carries a refusal reason %q", h.Half, h.ReadRefusal)
+ }
+ }
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ if len(cards) == 0 {
+ t.Error("a fully sealed audit produced no task cards")
+ }
+
+ openID := rpAuditID + "-open"
+ s := gateSealer(t, openID, false)
+ for _, half := range HalfValues() {
+ if err := s.SealHalf(openID, half, HalfStatusSealed); err != nil {
+ t.Fatalf("SealHalf(%s): %v", half, err)
+ }
+ }
+ for _, half := range HalfValues() {
+ seal, err := s.ReadHalf(openID, half)
+ if err != nil {
+ t.Errorf("ReadHalf(%s) on a sealed live audit: %v", half, err)
+ }
+ if !seal.Readable() {
+ t.Errorf("ReadHalf(%s) returned a seal that says it is not readable", half)
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// THE SOURCE GUARD — reachability, not a whitelist of today's return types
+// ---------------------------------------------------------------------------
+//
+// # What defeated the previous version of this guard, twice
+//
+// It flagged an exported function whose RETURN TYPE base name appeared in a
+// hardcoded ten-entry map: Manifest, ManifestHalf, CardRef, TaskCard,
+// GitHubSarifFile, GitHubSARIFLog, GitHubRun, GitHubResult, HalfSeal,
+// AuditSeal. `Result`, `Run`, `SARIFLog`, `[]string` and `[]byte` were not in
+// it — and they are the most natural spellings of "hand out a half's results".
+// A re-verifier added three entry points and the suite stayed green both
+// times:
+//
+// func (rd *Reader) LeakResults(l *SARIFLog) []Result
+// func (rd *Reader) LeakLoci(l *SARIFLog) []string
+// func LeakProjectionBytes(l *SARIFLog) ([]byte, error)
+//
+// They handed out nine results, nine loci and 16.6 KB of record bytes from a
+// half that had never sealed. A whitelist of today's return types cannot catch
+// tomorrow's function, because the leak is not in the type: `[]byte` is a
+// perfectly good way to hand out a record.
+//
+// # What this asks instead
+//
+// A BEHAVIOURAL question, answered from the package's own AST: can this
+// exported entry point reach a half's results, and if it can, does the same
+// call graph also reach the read gate?
+//
+// reaches results — the entry point, or an unexported function it calls
+// within this package, reads `.Results` or `.Runs`; or the
+// entry point is handed (or hands back) a whole *SARIFLog,
+// a whole *Run, or Results themselves. The second arm is
+// what catches LeakProjectionBytes, which can marshal the
+// record it was given without ever spelling `.Results`.
+// reaches the gate — the same call graph CALLS HalfReadGate,
+// halfReadRefusal, Readable or readOrder, AND USES what
+// that call returned. A mention is not a call and a
+// discarded result is not obedience; see
+// "WHAT 'REACHES THE GATE' MEANS" below.
+//
+// # WHAT "REACHES THE GATE" MEANS — A CALL WHOSE RESULT IS USED
+//
+// This used to be a MENTION: marks() set reachesGate on any *ast.Ident whose
+// name was in gateGateNames. An adversary ran sixteen attacks at the guard and
+// that one definition lost it two of them outright:
+//
+// ATTACK 9 call HalfReadGate, assign the error to `_`, return the results
+// anyway. A call, obeyed by nobody.
+// ATTACK 10 never touch the gate at all; declare a local variable NAMED
+// readOrder and return every result. Not even a call.
+//
+// So the analysis now requires BOTH halves:
+//
+// - a CALL. gateCalleeName resolves the callee of an *ast.CallExpr — an
+// identifier, or the selector of a method call — and only a callee whose
+// name is in gateGateNames counts. A bare identifier, a struct field
+// called Readable, a local variable named readOrder, a string constant:
+// none of them are calls, so none of them count.
+// - a USED RESULT. gateDiscardedCalls marks every call whose result is
+// thrown away: a call in statement position, a call under `go` or
+// `defer`, and a call assigned entirely to the blank identifier. A
+// discarded call does not count. CHECKING the error is what obeying the
+// gate means; `_ = HalfReadGate(id, seal)` checks nothing.
+//
+// One case is deliberately left to the Go compiler rather than duplicated
+// here: `err := HalfReadGate(...)` followed by no use of `err` does not
+// compile ("declared and not used"), so this analysis does not need to chase
+// it. `err = HalfReadGate(...)` into an already-declared err does compile, and
+// is not caught — see the KNOWN LIMITS section.
+//
+// # HOW HONEST IS THIS? IT IS A HEURISTIC, NOT A PROOF
+//
+// Stated plainly so nobody mistakes a green run for a guarantee:
+//
+// - The call graph is resolved by NAME. A call `x.Foo()` is treated as
+// reaching every method named Foo in the package, because this test does
+// not type-check. That over-approximates in both directions: it can decide
+// a function reaches results when its real callee does not, and it can
+// decide a function reaches the gate when the method that actually runs
+// is a different Foo.
+// - It follows CallExprs only. A function reached through a func-typed
+// STRUCT FIELD, a func value in a map, an interface method, or reflection
+// is invisible to it — `rd.Blobs(ref, raw)` is already such a call.
+// Package-level func-typed VARIABLES initialised with a func literal ARE
+// followed, both as entry points and as callees; that was attacks 11 and
+// 13 and it is closed.
+// - Reaching the gate in the call graph is not the same as OBEYING it with
+// the RIGHT seal. The call must now be a call and its result must be
+// used, but nothing here checks that the seal handed to the gate is the
+// seal of the half whose results are being returned. That is attacks 14
+// and 15, and they are open: read the KNOWN LIMITS section before you
+// trust a green run.
+// - `readOrder` counts as reaching the gate, so if readOrder ITSELF were
+// rewritten to ask one arm — which is precisely what CRITIQUE-03 M1 was —
+// this test would still pass. MEASURED, by putting that defect back: this
+// guard stayed green and three others went red, including
+// TestReadGateArmsAppearOnlyInsideTheGate, which is the test that owns
+// that hazard. The three guards are a set, and none of them is the
+// whole answer.
+//
+// Its value is not completeness. Its value is that the OBVIOUS bypass — the
+// one someone actually writes, which is a new exported function that walks
+// `l.Runs` and returns what it finds — cannot be added silently. All four
+// historical bypasses in sealing.go's list were exactly that shape, and so
+// were all three of the re-verifier's.
+//
+// TestTheSourceGuardCatchesTheLeaksThatDefeatedItsPredecessor synthesises
+// those three signatures and asserts this analysis flags each one, so the
+// guard is never again shipped without having been seen to fail.
+// TestTheSourceGuardCatchesTheAdversarysDefeats does the same for the six
+// adversarial shapes that beat it afterwards.
+
+// ===========================================================================
+// KNOWN LIMITS OF THE SOURCE GUARD — A NON-EXHAUSTIVE LIST OF OPEN HOLES
+// ===========================================================================
+//
+// READ THIS BEFORE YOU TRUST A GREEN RUN. Everything below is a hole that is
+// OPEN, with the reasoning written down. A guard whose limits are written down
+// is a tool; one whose limits are implied is a trap.
+//
+// THIS LIST IS NOT A CENSUS, AND TREATING IT AS ONE IS THE MISTAKE THIS
+// PARAGRAPH EXISTS TO PREVENT. An earlier draft of this section listed two
+// attacks and read as though it were complete. A second adversary then found
+// three more in one sitting (recorded as NEW A/B/C below) and judged the
+// section "adequate for 14 and 15 specifically, but FALSE AS A CENSUS". That
+// judgement was correct. Assume there are further holes not listed here.
+//
+// WHAT THIS GUARD IS FOR, stated plainly so it is not over-trusted:
+// it catches ACCIDENTAL bypass. That is not a small thing — FIVE independent
+// authors re-derived the read gate locally and all five got it wrong, none of
+// them adversarially, and CRITIQUE-02 and CRITIQUE-03 caught four of those in
+// shipped code. Every shape someone writes by mistake is caught: the
+// unexported-helper walk, the callback form, the struct-containing-results,
+// the count-only projection, and marshalling the record you were handed
+// without ever naming a field.
+//
+// WHAT THIS GUARD IS NOT: a security boundary. Obedience is matched BY NAME,
+// so a caller can mint its own — see NEW C. Static reachability cannot
+// distinguish calling the gate from obeying it about the right seal for the
+// right halves. Do not cite a green run as evidence that no exported function
+// can leak a half's results. It is not that, and it cannot be made into that
+// without a different technique (go/types + SSA def-use, or the runtime
+// redesign described under attack 15, which is the one worth doing).
+//
+// THREE FURTHER OPEN HOLES, found by the second adversary after the six
+// cheap ones were closed. Not fixed; recorded so the list is honest.
+//
+// NEW A — satisfy "the error is used" while ignoring the refusal.
+// gateDiscardedCalls only marks statement-position calls, go/defer, and
+// whole-blank assignment. Three shapes slip past, all compiling:
+// A1 if err := HalfReadGate(...); err != nil { log.Printf(...) } then
+// return every result. The error is checked, the branch is taken, and
+// the function proceeds anyway. This is the single most LIKELY of all
+// the open holes to occur by accident rather than by intent.
+// A2 _ = fmt.Sprintf("%v", HalfReadGate(...)) — the blank-assign marker
+// tags the outer call; the inner gate call counts as used.
+// A3 errs := []error{HalfReadGate(...)}; _ = errs
+//
+// NEW B — a method promoted from an EMBEDDED unexported type. Enumeration
+// admits an unexported receiver only when an exported function RETURNS that
+// type. Embedding is a third route into a caller's hands and neither
+// enumeration inspects it: an unexported type with a leaking method,
+// embedded in an exported zero-value-usable struct, needs no constructor.
+// Proven from OUTSIDE the package: an external package obtained results
+// through it and the guard never enumerated the method at all.
+//
+// NEW C — mint your own obedience by name. The callee is resolved to a bare
+// string and matched against gateGateNames; nothing checks the callee IS the
+// gate. C2 is the cheap form and it directly contradicts the fix for attack
+// 10: attack 10 was "declare a local variable named readOrder", the fix
+// demanded a CALL, so make the local variable a func and call it —
+// readOrder := func(*SARIFLog) []orderedResult { return nil }
+// One pair of parentheses over the attack the fix was written against.
+// Closing this needs the callee to resolve to a package-level declaration,
+// not to any identifier that happens to carry the name.
+//
+// The body-hash allowlist survived every attempt, including collision and
+// staleness. Its one real weakness is procedural rather than static: the
+// failure message prints the new hash, which invites a copy-paste that
+// re-grants the exemption without anyone re-reading the reason.
+//
+// TestResultReachingEntryPointsAreGated answers exactly one question:
+//
+// does the call graph of this exported entry point CALL the read gate and
+// USE what the call returned?
+//
+// It does NOT answer, and cannot answer, "did it call the gate about the
+// RIGHT half, with the RIGHT seal, and honour the answer for ALL of the
+// results it went on to return". Those are dataflow questions — which value
+// flowed into which parameter, and which values flowed out — and an AST
+// reachability walk that pretended to answer them would hand out exactly the
+// false confidence this whole exercise exists to avoid. Two attacks live in
+// that gap. Both were run against this guard. Both won. Both still win.
+//
+// ---------------------------------------------------------------------------
+// LIMIT 1 (adversary attack 14) — THE FABRICATED SEAL
+// ---------------------------------------------------------------------------
+//
+// Call the gate. Check the error. Return the results. But hand the gate a
+// HalfSeal you built yourself rather than the seal of the half you are about
+// to read:
+//
+// func (rd *Reader) LeakViaFabricatedSeal(l *SARIFLog) []Result {
+// seal := HalfSeal{Half: HalfSast, Status: HalfStatusSealed, ...}
+// if err := HalfReadGate(l.Properties.AuditID, seal); err != nil {
+// return nil
+// }
+// var out []Result
+// for ri := range l.Runs {
+// out = append(out, l.Runs[ri].Results...)
+// }
+// return out
+// }
+//
+// To this guard that is a textbook obedient function: a real *ast.CallExpr on
+// HalfReadGate, the returned error consumed by an `if`, an early return on
+// refusal. Every structural property it checks is satisfied. The gate was
+// asked a question about a half that does not exist and it answered honestly.
+//
+// WHAT A READER MUST NOT CONCLUDE: that a green
+// TestResultReachingEntryPointsAreGated means the gate was consulted ABOUT the
+// data returned. It means a gate call happened somewhere in the call graph and
+// its result was not thrown away. The seal's provenance is unchecked.
+//
+// WHAT WOULD ACTUALLY CATCH IT: dataflow. The HalfSeal argument at the call
+// site must be traced to its definition and required to originate from
+// halfSealOfRun (or Sealer.HalfSeal) applied to the same record the results
+// are read from — a def-use chain, needing go/types and SSA, not an
+// ast.Inspect. Or, cheaper and stronger, a RUNTIME assertion: give HalfSeal an
+// unexported provenance field that only halfSealOfRun and the Sealer can set,
+// and have HalfReadGate refuse any seal without it. A fabricated composite
+// literal then cannot be handed to the gate at all, and the hole closes in the
+// production code rather than in a test that inspects it.
+//
+// ---------------------------------------------------------------------------
+// LIMIT 2 (adversary attack 15) — OBEY FOR ONE HALF, RETURN BOTH
+// ---------------------------------------------------------------------------
+//
+// Call the gate. Check the error. Obey it — for the SAST half. Then return
+// every result in the record, DAST included:
+//
+// func (rd *Reader) LeakViaPartialObedience(l *SARIFLog) []Result {
+// for ri := range l.Runs {
+// if l.Runs[ri].Properties.Half != HalfSast {
+// continue
+// }
+// if err := HalfReadGate(l.Properties.AuditID,
+// halfSealOfRun(l, &l.Runs[ri])); err != nil {
+// return nil
+// }
+// }
+// var out []Result
+// for ri := range l.Runs {
+// out = append(out, l.Runs[ri].Results...)
+// }
+// return out
+// }
+//
+// Here the seal is genuine — halfSealOfRun over the real record — so even the
+// provenance idea above would not fire. The defect is that the SET of halves
+// the gate was consulted about is smaller than the SET of halves whose results
+// were returned. An unsealed DAST half walks out behind a sealed SAST half's
+// permission.
+//
+// WHAT A READER MUST NOT CONCLUDE: that a gated entry point is gated for every
+// half it can return. This guard counts gate calls; it does not pair them with
+// the results that leave. One honest gate call covers an entry point that
+// hands out ten halves.
+//
+// WHAT WOULD ACTUALLY CATCH IT: dataflow again, and a harder instance —
+// correlating the loop that queries the gate with the loop that accumulates
+// the output, per half. Statically that is a per-index dependence analysis.
+// The practical answer is not static at all: make the gate the only route to a
+// result. If the results of a half were reachable only through a value that
+// HalfReadGate returns — a readable-half handle, rather than a permission slip
+// checked beside a []Result the caller already had — then returning the DAST
+// results would require a DAST handle, and the compiler would demand a second
+// gate call. That is a change to readpath.go's shape, not to this test.
+//
+// ---------------------------------------------------------------------------
+// SMALLER OPEN EDGES, LISTED SO THEY ARE NOT DISCOVERIES LATER
+// ---------------------------------------------------------------------------
+//
+// - `err = HalfReadGate(...)` into an ALREADY-DECLARED err, never read
+// afterwards, counts as a used result here. The `:=` form of the same
+// mistake does not compile, so the compiler carries most of this; the `=`
+// form would need liveness analysis and does not get it.
+// - A func-typed STRUCT FIELD or map entry is still not followed.
+// `rd.Blobs(ref, raw)` is already such a call. Package-level func-literal
+// VARIABLES are followed (attacks 11 and 13); fields are not.
+// - A func-typed package-level variable declared WITHOUT a literal —
+// `var Hook ReadFunc` assigned at init — has no body to index and is
+// invisible.
+// - The allowlist body hash covers the body, not the SIGNATURE and not the
+// functions the body calls. Moving the leak one level down, into an
+// unexported helper an allowlisted function already called, changes the
+// helper rather than the allowlisted body. The reachability analysis is
+// what covers that direction, and only for entry points it enumerates.
+//
+// ---------------------------------------------------------------------------
+// WHAT IS ACTUALLY LOAD-BEARING TODAY
+// ---------------------------------------------------------------------------
+//
+// The behavioural half of the pair, TestEveryResultBearingEntryPointIsGated,
+// RUNS every listed entry point against records in which no half is readable
+// and counts what came back. It would catch both attacks above — on the entry
+// points that are IN gateAuditedEntryPoints. That list is maintained by hand.
+// The source guard's whole job is to notice when something is missing from it.
+//
+// So the honest summary of the pair is: the source guard says "a new exported
+// function that reaches results without asking the gate cannot be added
+// silently", and the behavioural guard says "the entry points we know about
+// hand out nothing when nothing is readable". Neither says "no exported
+// function can leak a half's results". Nothing in this package says that.
+
+// gateResultAccess are the field reads that mean "this body can get at a
+// half's findings". `.Runs` is included because reaching the runs is how a
+// caller reaches the results inside them, and a function that walks `l.Runs`
+// and stops short of `.Results` still holds every half in the record.
+var gateResultAccess = map[string]bool{"Results": true, "Runs": true}
+
+// gateGateNames are the four spellings of "this call graph asked the read
+// gate". readOrder is included because it is readpath.go's own gated entry to
+// the results and every card is built from what it returns.
+//
+// These are CALLEE names, matched against the function position of an
+// *ast.CallExpr and nowhere else. A local variable named readOrder, a struct
+// field named Readable, or a comment naming HalfReadGate is not a call and
+// does not count; that was adversary attack 10.
+var gateGateNames = map[string]bool{
+ "HalfReadGate": true,
+ "halfReadRefusal": true,
+ "Readable": true,
+ "readOrder": true,
+}
+
+// gateRecordTypes are the types that CONTAIN a half's results. An entry point
+// handed one, or handing one back, can project the results out of it with no
+// field access this analysis would otherwise see.
+var gateRecordTypes = map[string]bool{"SARIFLog": true, "Run": true, "Result": true}
+
+// gateSourceIndex is the parsed package: every function body, indexed the two
+// ways a call site can name one.
+//
+// "Function body" includes package-level VARIABLES initialised with a func
+// literal — `var Foo = func(l *SARIFLog) []Result { ... }`. Such a variable is
+// a callable value with a body, so it is synthesised into an *ast.FuncDecl and
+// indexed exactly like a declared function. That is what closes adversary
+// attacks 11 (hide the results read behind a package-level func value) and 13
+// (an exported package-level func-typed variable, which is public API in every
+// sense that matters to a caller).
+type gateSourceIndex struct {
+ fset *token.FileSet // positions, for printing bodies to hash
+ decl map[string]*ast.FuncDecl // "Func" or "Recv.Method" -> body
+ file map[string]string // the same key -> base filename
+ byName map[string][]string // plain function name -> keys
+ byMethod map[string][]string // method name -> keys (any receiver)
+ funcVar map[string]bool // key was a package-level func-literal var
+ keys []string // every key, sorted
+}
+
+// gateParseSource parses the package's non-test files into a gateSourceIndex.
+func gateParseSource(t *testing.T) *gateSourceIndex {
+ t.Helper()
+ files, fset := gatePackageFiles(t)
+ return gateIndexFiles(t, files, fset)
+}
+
+// gatePackageFiles parses every non-test file of the package, keyed by path,
+// and returns the FileSet those files were parsed into. The FileSet is needed
+// because the allowlist hashes function BODIES, and printing a body back to
+// source requires the positions it was parsed with.
+func gatePackageFiles(t *testing.T) (map[string]*ast.File, *token.FileSet) {
+ t.Helper()
+ fset := token.NewFileSet()
+ pkgs, err := parser.ParseDir(fset, ".", func(fi os.FileInfo) bool {
+ return !strings.HasSuffix(fi.Name(), "_test.go")
+ }, 0)
+ if err != nil {
+ t.Fatalf("parsing the package source: %v", err)
+ }
+ if len(pkgs) == 0 {
+ t.Fatal("parsed no packages; this test asserts nothing unless it reads the source")
+ }
+ out := map[string]*ast.File{}
+ for _, pkg := range pkgs {
+ for path, file := range pkg.Files {
+ out[path] = file
+ }
+ }
+ return out, fset
+}
+
+// gateIndexFiles builds the call-graph index over a set of parsed files. It
+// takes the files rather than reading the directory so that
+// TestTheSourceGuardCatchesTheLeaksThatDefeatedItsPredecessor and
+// TestTheSourceGuardCatchesTheAdversarysDefeats can index the real package
+// PLUS a synthetic leak file and run the identical analysis.
+func gateIndexFiles(t *testing.T, files map[string]*ast.File, fset *token.FileSet) *gateSourceIndex {
+ t.Helper()
+ idx := &gateSourceIndex{
+ fset: fset,
+ decl: map[string]*ast.FuncDecl{},
+ file: map[string]string{},
+ byName: map[string][]string{},
+ byMethod: map[string][]string{},
+ funcVar: map[string]bool{},
+ }
+ add := func(path string, key string, fn *ast.FuncDecl) {
+ idx.decl[key] = fn
+ idx.file[key] = filepath.Base(path)
+ idx.keys = append(idx.keys, key)
+ }
+ for path, file := range files {
+ for _, d := range file.Decls {
+ switch d := d.(type) {
+ case *ast.FuncDecl:
+ if d.Body == nil {
+ continue
+ }
+ key := d.Name.Name
+ if d.Recv != nil && len(d.Recv.List) == 1 {
+ recv := gateBaseTypeName(d.Recv.List[0].Type)
+ if recv == "" {
+ continue
+ }
+ key = recv + "." + d.Name.Name
+ idx.byMethod[d.Name.Name] = append(idx.byMethod[d.Name.Name], key)
+ } else {
+ idx.byName[key] = append(idx.byName[key], key)
+ }
+ add(path, key, d)
+ case *ast.GenDecl:
+ // `var Foo = func(...) {...}` is a function with a name, a
+ // signature and a body. Index it as one — attacks 11 and 13.
+ if d.Tok != token.VAR {
+ continue
+ }
+ for _, spec := range d.Specs {
+ vs, ok := spec.(*ast.ValueSpec)
+ if !ok {
+ continue
+ }
+ for i, name := range vs.Names {
+ if i >= len(vs.Values) {
+ continue
+ }
+ lit, ok := vs.Values[i].(*ast.FuncLit)
+ if !ok || lit.Body == nil || name.Name == "_" {
+ continue
+ }
+ key := name.Name
+ if _, dup := idx.decl[key]; dup {
+ continue
+ }
+ idx.byName[key] = append(idx.byName[key], key)
+ idx.funcVar[key] = true
+ add(path, key, &ast.FuncDecl{
+ Name: name,
+ Type: lit.Type,
+ Body: lit.Body,
+ })
+ }
+ }
+ }
+ }
+ }
+ sort.Strings(idx.keys)
+ return idx
+}
+
+// callees returns the package-local functions the body of key can call,
+// resolved by name. See the honesty section above: this is an
+// over-approximation, and deliberately so.
+func (idx *gateSourceIndex) callees(key string) []string {
+ fn := idx.decl[key]
+ if fn == nil {
+ return nil
+ }
+ var out []string
+ ast.Inspect(fn.Body, func(n ast.Node) bool {
+ call, ok := n.(*ast.CallExpr)
+ if !ok {
+ return true
+ }
+ switch f := call.Fun.(type) {
+ case *ast.Ident:
+ out = append(out, idx.byName[f.Name]...)
+ case *ast.SelectorExpr:
+ // `json.Marshal` resolves to nothing: byMethod holds only this
+ // package's own methods.
+ out = append(out, idx.byMethod[f.Sel.Name]...)
+ }
+ return true
+ })
+ return out
+}
+
+// gateBodyMarks is what one function body was seen to do, before any call
+// graph is followed.
+type gateBodyMarks struct {
+ readsResults bool
+ reachesGate bool
+}
+
+func (idx *gateSourceIndex) marks(key string) gateBodyMarks {
+ var m gateBodyMarks
+ fn := idx.decl[key]
+ if fn == nil {
+ return m
+ }
+ discarded := gateDiscardedCalls(fn.Body)
+ ast.Inspect(fn.Body, func(n ast.Node) bool {
+ switch e := n.(type) {
+ case *ast.SelectorExpr:
+ if gateResultAccess[e.Sel.Name] {
+ m.readsResults = true
+ }
+ case *ast.CallExpr:
+ // A CALL to the gate, whose ANSWER is used. Neither half is
+ // optional: a mention is attack 10 and a discarded error is
+ // attack 9.
+ if gateGateNames[gateCalleeName(e.Fun)] && !discarded[e] {
+ m.reachesGate = true
+ }
+ }
+ return true
+ })
+ return m
+}
+
+// gateCalleeName is the name a call site uses for its callee: `f()` -> "f",
+// `x.f()` -> "f", `f[T]()` -> "f". Anything else — a call through a func
+// literal, an index into a map of funcs, a conversion — returns "", which no
+// gate name matches.
+//
+// This is the ONLY place a gate name may be recognised. Matching bare
+// identifiers is what let a local variable named readOrder pass for obedience.
+func gateCalleeName(fun ast.Expr) string {
+ switch f := fun.(type) {
+ case *ast.Ident:
+ return f.Name
+ case *ast.SelectorExpr:
+ return f.Sel.Name
+ case *ast.IndexExpr:
+ return gateCalleeName(f.X)
+ case *ast.IndexListExpr:
+ return gateCalleeName(f.X)
+ case *ast.ParenExpr:
+ return gateCalleeName(f.X)
+ }
+ return ""
+}
+
+// gateDiscardedCalls returns every call in the body whose result is thrown
+// away. A gate call in this set is not obedience:
+//
+// - a call in statement position — `HalfReadGate(id, seal)` on its own line;
+// - a call under `go` or `defer`, whose result is unreachable by
+// construction;
+// - a call assigned entirely to the blank identifier — `_ = HalfReadGate(...)`,
+// which is adversary attack 9, and `_, _ = f()`.
+//
+// The 1:1 assignment form `a, _ := f(), g()` is handled per position, so the
+// g() there is discarded and the f() is not.
+//
+// Assigning to a NAMED variable counts as used. The case that leaves —
+// assigning to a named variable and then never reading it — is caught by the
+// Go compiler for `:=` ("declared and not used") and is documented as open for
+// `=` in the KNOWN LIMITS section.
+func gateDiscardedCalls(body *ast.BlockStmt) map[*ast.CallExpr]bool {
+ out := map[*ast.CallExpr]bool{}
+ mark := func(e ast.Expr) {
+ if c, ok := e.(*ast.CallExpr); ok {
+ out[c] = true
+ }
+ }
+ allBlank := func(lhs []ast.Expr) bool {
+ for _, e := range lhs {
+ id, ok := e.(*ast.Ident)
+ if !ok || id.Name != "_" {
+ return false
+ }
+ }
+ return len(lhs) > 0
+ }
+ isBlank := func(e ast.Expr) bool {
+ id, ok := e.(*ast.Ident)
+ return ok && id.Name == "_"
+ }
+ ast.Inspect(body, func(n ast.Node) bool {
+ switch s := n.(type) {
+ case *ast.ExprStmt:
+ mark(s.X)
+ case *ast.GoStmt:
+ out[s.Call] = true
+ case *ast.DeferStmt:
+ out[s.Call] = true
+ case *ast.AssignStmt:
+ if len(s.Rhs) == 1 {
+ // One call feeding every LHS: discarded only if no LHS
+ // keeps anything.
+ if allBlank(s.Lhs) {
+ mark(s.Rhs[0])
+ }
+ return true
+ }
+ for i := range s.Rhs {
+ if i < len(s.Lhs) && isBlank(s.Lhs[i]) {
+ mark(s.Rhs[i])
+ }
+ }
+ }
+ return true
+ })
+ return out
+}
+
+// gateIsGateFunc reports whether a call-graph key IS one of the gate
+// functions: "HalfReadGate", "HalfSeal.Readable", "Reader.readOrder".
+func gateIsGateFunc(key string) bool {
+ name := key
+ if i := strings.LastIndex(key, "."); i >= 0 {
+ name = key[i+1:]
+ }
+ return gateGateNames[name]
+}
+
+// reach walks the call graph from key and returns the union of every body's
+// marks, key's own included.
+//
+// THE GATE'S OWN BODY IS NOT TRAVERSED. halfReadRefusal calls the gate's
+// arms, and HalfReadGate calls halfReadRefusal and uses its answer — so a
+// closure that descended into them would mark EVERY caller of HalfReadGate as
+// gate-reaching, including one that calls it and throws the error away. That
+// is precisely adversary attack 9, and it is why obedience is established at
+// the CALL SITE and nowhere else: a used call to the gate, in a body in this
+// closure. What the gate does internally is the gate's business.
+//
+// The root is always marked, even when the root itself is a gate function, so
+// that a leak named `Readable` cannot buy an exemption with its own name.
+func (idx *gateSourceIndex) reach(key string) gateBodyMarks {
+ var out gateBodyMarks
+ seen := map[string]bool{key: true}
+ queue := []string{key}
+ for len(queue) > 0 {
+ cur := queue[0]
+ queue = queue[1:]
+ m := idx.marks(cur)
+ out.readsResults = out.readsResults || m.readsResults
+ out.reachesGate = out.reachesGate || m.reachesGate
+ for _, next := range idx.callees(cur) {
+ if seen[next] || gateIsGateFunc(next) {
+ continue
+ }
+ seen[next] = true
+ queue = append(queue, next)
+ }
+ }
+ return out
+}
+
+// handlesRecord reports whether the entry point's own signature takes or
+// returns a whole record, a whole run, or results — the arm that catches a
+// function which marshals what it was handed without ever naming a field.
+func gateHandlesRecord(fn *ast.FuncDecl) bool {
+ fields := []*ast.Field{}
+ if fn.Recv != nil {
+ fields = append(fields, fn.Recv.List...)
+ }
+ if fn.Type.Params != nil {
+ fields = append(fields, fn.Type.Params.List...)
+ }
+ if fn.Type.Results != nil {
+ fields = append(fields, fn.Type.Results.List...)
+ }
+ for _, f := range fields {
+ if gateRecordTypes[gateBaseTypeName(f.Type)] {
+ return true
+ }
+ }
+ return false
+}
+
+// gateVerdict is what the analysis concludes about one entry point.
+type gateVerdict int
+
+const (
+ // gateNoResults: nothing in this call graph can get at a half's findings.
+ gateNoResults gateVerdict = iota
+ // gateGated: it can, and the same call graph asks the read gate.
+ gateGated
+ // gateUngated: it can, and nothing in the call graph asks the gate. This
+ // is the finding.
+ gateUngated
+)
+
+// classify is THE analysis. Both the guard and its own negative control call
+// it, so the control cannot pass against a different code path than the one
+// that ships.
+func (idx *gateSourceIndex) classify(key string) gateVerdict {
+ fn := idx.decl[key]
+ if fn == nil {
+ return gateNoResults
+ }
+ r := idx.reach(key)
+ if !r.readsResults && !gateHandlesRecord(fn) {
+ return gateNoResults
+ }
+ if r.reachesGate {
+ return gateGated
+ }
+ return gateUngated
+}
+
+// gateHandedOutTypes returns the set of UNEXPORTED type names that an exported
+// function or method hands back to a caller outside the package.
+//
+// A caller who holds such a value can call every exported method on it, and
+// the receiver's case is irrelevant to that: `func NewThing() *thing` makes
+// every exported method on `thing` public API. Adversary attack 12 was exactly
+// this — an exported method on an unexported receiver, handed out by an
+// exported constructor — and the previous entry-point scan skipped it on the
+// strength of the receiver's lower-case letter.
+func (idx *gateSourceIndex) gateHandedOutTypes() map[string]bool {
+ out := map[string]bool{}
+ for _, key := range idx.keys {
+ fn := idx.decl[key]
+ if !fn.Name.IsExported() {
+ continue
+ }
+ if fn.Recv != nil && len(fn.Recv.List) == 1 {
+ // Only count returns from something a caller can already reach.
+ if recv := gateBaseTypeName(fn.Recv.List[0].Type); !ast.IsExported(recv) {
+ continue
+ }
+ }
+ if fn.Type.Results == nil {
+ continue
+ }
+ for _, res := range fn.Type.Results.List {
+ if name := gateBaseTypeName(res.Type); name != "" && !ast.IsExported(name) {
+ out[name] = true
+ }
+ }
+ }
+ return out
+}
+
+// exportedEntryPoints returns the keys of every exported function, method and
+// package-level func-literal variable reachable from outside the package,
+// sorted.
+//
+// "Reachable from outside" is deliberately broader than "the receiver type is
+// exported": see gateHandedOutTypes for attack 12, and gateSourceIndex for
+// attack 13.
+func (idx *gateSourceIndex) exportedEntryPoints() []string {
+ handedOut := idx.gateHandedOutTypes()
+ var out []string
+ for _, key := range idx.keys {
+ fn := idx.decl[key]
+ if !fn.Name.IsExported() {
+ continue
+ }
+ if fn.Recv != nil {
+ recv := gateBaseTypeName(fn.Recv.List[0].Type)
+ // A method on an unexported type is a public bypass exactly when
+ // an exported function hands that type out. If nobody can obtain
+ // the receiver, nobody can call the method.
+ if !ast.IsExported(recv) && !handedOut[recv] {
+ continue
+ }
+ }
+ out = append(out, key)
+ }
+ return out
+}
+
+// gateExemption is one allowlist entry: why this entry point is not a leak,
+// and the hash of the BODY that claim was made about.
+//
+// AN ALLOWLIST ENTRY IS A CLAIM ABOUT A BODY, NOT ABOUT A NAME. Adversary
+// attack 16 was to leave the name alone and rewrite what the function does:
+// the exemption was written for one implementation and silently inherited by
+// another. The hash makes that impossible — change the body and the claim
+// expires, the suite goes red, and the author has to re-read the reason and
+// re-justify it against the code that is actually there now.
+type gateExemption struct {
+ // reason is the justification. A reason, never a shrug.
+ reason string
+
+ // body is gateBodyHash of the function body at the moment the reason was
+ // written. Re-run the test after an intentional change: the failure
+ // message carries the new hash.
+ body string
+}
+
+// gateBodyHash is the identity of a function BODY: the body printed back to Go
+// source, re-tokenised, and the TOKEN STREAM hashed, truncated to 16 hex
+// characters.
+//
+// What it covers and what it does not, so the failure message can be trusted:
+//
+// - it covers statements, expressions, names, literals and structure —
+// anything that changes what the function DOES;
+// - it does not cover comments, whitespace, indentation, blank lines or line
+// endings, because the hash is over tokens. Re-wording a comment inside an
+// allowlisted function does not expire its exemption, and neither does
+// gofmt or a CRLF checkout. A hash that expired on every doc edit would be
+// deleted by the third person it annoyed, and then attack 16 would be open
+// again with nobody noticing;
+// - it does not cover the SIGNATURE. A signature change that makes the
+// function newly result-reaching is caught by the analysis itself, and one
+// that does not is not interesting.
+func (idx *gateSourceIndex) gateBodyHash(key string) string {
+ fn := idx.decl[key]
+ if fn == nil || fn.Body == nil || idx.fset == nil {
+ return ""
+ }
+ var buf bytes.Buffer
+ if err := (&printer.Config{Mode: printer.RawFormat, Tabwidth: 8}).
+ Fprint(&buf, idx.fset, fn.Body); err != nil {
+ return ""
+ }
+ src := buf.Bytes()
+
+ // Re-tokenise the printed body. Everything the compiler ignores — layout
+ // and comments — is discarded here, so only a change in what the code says
+ // can change the hash.
+ var fs token.FileSet
+ var sc scanner.Scanner
+ sc.Init(fs.AddFile("", fs.Base(), len(src)), src, nil, 0)
+ var stream strings.Builder
+ for {
+ _, tok, lit := sc.Scan()
+ if tok == token.EOF {
+ break
+ }
+ stream.WriteString(tok.String())
+ if lit != "" {
+ stream.WriteByte('\x1f')
+ stream.WriteString(lit)
+ }
+ stream.WriteByte('\x1e')
+ }
+ sum := sha256.Sum256([]byte(stream.String()))
+ return hex.EncodeToString(sum[:])[:16]
+}
+
+// gateUngatedAllowlist names the exported entry points that CAN reach a half's
+// results but legitimately do not ask the read gate, each with the reason and
+// the hash of the body that reason was written about.
+//
+// An allowlist with reasons is a different object from a whitelist of types.
+// The whitelist said "these shapes are interesting"; a new shape walked past
+// it. This says "these named function BODIES have been looked at, and here is
+// why each one is not a leak" — a new function is not on it, so a new function
+// is flagged, and a rewritten body is not the body that was looked at.
+//
+// TestResultReachingEntryPointsAreGated fails if an entry stops matching a
+// real, still-ungated, still-result-reaching function, so the list cannot rot
+// into a blanket: delete the function and the entry must go with it. It fails
+// again if the body no longer hashes to the recorded value.
+func gateUngatedAllowlist() map[string]gateExemption {
+ return map[string]gateExemption{
+ // ---- the SOURCE side: these hand the record IN --------------------
+ "RecordMap.Record": {body: "25391599143667fe", reason: "the RecordSource a Reader loads FROM. It hands a record in, to a " +
+ "read path that then applies the gate; it is the gate's input, not its output. " +
+ "Callers outside this package supply their own RecordSource and this one is a " +
+ "test/in-memory convenience."},
+ "RecordSourceFunc.Record": {body: "c62370d3dae715ab", reason: "the func adapter for RecordSource, same direction and same " +
+ "reason as RecordMap.Record: it returns whatever the caller's own function returns."},
+
+ // ---- the PRODUCER side: these run before a half is readable -------
+ "SARIFLog.Validate": {body: "5e4bedd55119ae6e", reason: "contract.go's producer-side validator. It walks every run to check " +
+ "the record is well-formed and returns only an error; validation must work on a " +
+ "record NO half of which has sealed yet, so gating it would make an unsealed " +
+ "record unvalidatable."},
+ "MaskRecord": {body: "1d1496b8964a1f30", reason: "R.8's masker, which runs BEFORE the read path and is the precondition " +
+ "Reader.load asserts. It mutates the record in place and returns only an error. " +
+ "Masking an unsealed half is exactly what it is for."},
+ "Masker.Mask": {body: "3b1f1725f1f6decc", reason: "the same masker with a report. The report counts what was masked; it " +
+ "carries no finding content out of a half."},
+ "AssertMasked": {body: "930534f795ec1a22", reason: "the masking precondition itself, called by Reader.load before anything " +
+ "is projected. It answers 'has R.8 run', not 'may this half be read'."},
+
+ // ---- pure functions over data the caller ALREADY holds ------------
+ "Result.ExternalStringPointers": {body: "b598635db55a205e", reason: "a pure accessor on a Result the caller already holds. " +
+ "It cannot obtain one: whoever calls it got the Result from somewhere, and that " +
+ "somewhere is what the gate covers."},
+ "ValidateResultTrust": {body: "df1846aeedb2981a", reason: "a validator over one caller-held Result, returning only an error."},
+ "IsHostFinding": {body: "eaf63949394f43ee", reason: "a pure predicate over one caller-held Result. It reads two enum fields " +
+ "and returns a bool."},
+ "Correlate": {body: "2f5772b9f31ae65b", reason: "the correlation engine (R.12). It is handed two []Result by the PRODUCER, " +
+ "before either half is consumable, and returns clusters rather than results. " +
+ "Gating it would mean no audit could ever be correlated."},
+ "CorrelateWithEvidence": {body: "58f2c8a2f9b29cc0", reason: "the same engine with the evidence bundle; same direction and " +
+ "same reason as Correlate."},
+ "TaskCard.CheckAgainstRecord": {body: "3dcdcd66f256a036", reason: "the card's own agreement check against the Result it was " +
+ "built from — a card the gate already emitted, compared with a result the caller " +
+ "already holds. It returns only an error."},
+
+ // ---- already-projected output ------------------------------------
+ "GitHubSarifFile.WithinCaps": {body: "cd3276a9b07eb740", reason: "a cap check over an ALREADY-PROJECTED GitHub file. " +
+ "ProjectForGitHub applies the gate; what survives into a GitHubSarifFile is what " +
+ "the gate let through, and counting it again cannot un-withhold anything."},
+ }
+}
+
+// TestResultReachingEntryPointsAreGated is the guard. See the long section
+// above for what it asks, and for the three ways it can be fooled.
+func TestResultReachingEntryPointsAreGated(t *testing.T) {
+ idx := gateParseSource(t)
+ allow := gateUngatedAllowlist()
+
+ entries := idx.exportedEntryPoints()
+ if len(entries) == 0 {
+ t.Fatal("the source scan found no exported entry points at all; the scan is broken " +
+ "and this test is guarding nothing")
+ }
+
+ var reaching, gated []string
+ allowed := map[string]bool{}
+ for _, key := range entries {
+ verdict := idx.classify(key)
+ if verdict == gateNoResults {
+ continue
+ }
+ reaching = append(reaching, key)
+ if verdict == gateGated {
+ gated = append(gated, key)
+ continue
+ }
+ if ex, ok := allow[key]; ok {
+ allowed[key] = true
+ if strings.TrimSpace(ex.reason) == "" {
+ t.Errorf("%s is allowlisted with an empty reason; an allowlist without reasons "+
+ "is the whitelist this guard replaced", key)
+ }
+ // ATTACK 16: the exemption was granted to a BODY. If the body
+ // changed, the exemption did not survive it.
+ got := idx.gateBodyHash(key)
+ switch {
+ case got == "":
+ t.Errorf("%s is allowlisted but its body could not be hashed; the exemption "+
+ "cannot be checked and must not be trusted", key)
+ case strings.TrimSpace(ex.body) == "":
+ t.Errorf("%s is allowlisted with no body hash. An allowlist entry is a claim "+
+ "about a BODY, not about a name. Record body: %q.", key, got)
+ case ex.body != got:
+ t.Errorf("%s (%s) is allowlisted, but its body has CHANGED since the exemption "+
+ "was written.\n"+
+ " recorded body hash %s\n"+
+ " current body hash %s\n"+
+ " The reason on file was written about the old implementation:\n"+
+ " %s\n"+
+ " Re-read the function as it is NOW and decide again whether it can leak\n"+
+ " a half's results. If it still cannot, update the body hash to %s and\n"+
+ " say so in the reason. If it can, route it through HalfReadGate instead.\n"+
+ " An allowlist entry is a claim about a BODY, not about a name.",
+ key, idx.file[key], ex.body, got, ex.reason, got)
+ }
+ continue
+ }
+ t.Errorf("%s (%s) can reach a half's results and never reaches the read gate.\n"+
+ " Route it through HalfReadGate — or halfSealOfRun + HalfReadGate for a\n"+
+ " record-side caller — and add a probe to gateAuditedEntryPoints; or, if it\n"+
+ " structurally cannot leak, add it to gateUngatedAllowlist WITH THE REASON.\n"+
+ " Five authors have now derived the read gate locally and five got it wrong;\n"+
+ " see sealing.go's read-gate section.", key, idx.file[key])
+ }
+
+ // An allowlist entry that no longer names a flagged function is a lie
+ // about the code, and the next person to read it inherits the lie.
+ for key := range allow {
+ if allowed[key] {
+ continue
+ }
+ switch {
+ case idx.decl[key] == nil:
+ t.Errorf("gateUngatedAllowlist names %q, which no longer exists in the package. "+
+ "Delete the entry: an allowlist that outlives its function is a standing "+
+ "exemption nobody has read.", key)
+ default:
+ t.Errorf("gateUngatedAllowlist names %q, which this guard no longer flags "+
+ "(it now reaches the gate, or no longer reaches results). Delete the entry: "+
+ "a stale exemption is how the next real one gets waved through.", key)
+ }
+ }
+
+ if len(reaching) == 0 {
+ t.Fatal("no exported entry point was found to reach a half's results, which cannot be " +
+ "true of this package; the analysis is broken and the guard is inert")
+ }
+ if len(gated) == 0 {
+ t.Fatal("no result-reaching entry point was found to reach the gate either, which cannot " +
+ "be true while readOrder exists; the analysis is broken and would pass anything")
+ }
+ t.Logf("source reachability: %d exported entry points, %d can reach a half's results "+
+ "(%d reach the gate, %d allowlisted)", len(entries), len(reaching), len(gated), len(allowed))
+ t.Logf(" gated: %s", strings.Join(gated, ", "))
+}
+
+// gateIndexWithProbe indexes the REAL package plus one synthetic source file
+// and returns the index and the live allowlist. Both negative controls use it,
+// so both run the identical analysis against the shipping code — a control
+// that indexed only the synthetic file would prove nothing about the guard
+// that ships.
+func gateIndexWithProbe(t *testing.T, src string) (*gateSourceIndex, map[string]gateExemption) {
+ t.Helper()
+ files, fset := gatePackageFiles(t)
+ probe, err := parser.ParseFile(fset, "zz_gate_leak_probe.go", src, 0)
+ if err != nil {
+ t.Fatalf("parsing the synthetic probe file: %v", err)
+ }
+ files["zz_gate_leak_probe.go"] = probe
+ return gateIndexFiles(t, files, fset), gateUngatedAllowlist()
+}
+
+// gateLeakProbeSource is the re-verifier's three leak functions, verbatim in
+// shape, as a synthetic source file in this package.
+//
+// They are the three that DEFEATED the previous guard — a type whitelist that
+// did not name Result, []string or []byte — while handing out nine results,
+// nine loci and 16.6 KB of record bytes from a half that had never sealed.
+// They live here, as source text rather than as compiled code, so that the
+// guard is exercised against them on EVERY run and cannot be shipped again
+// without having been seen to fail. A guard that has never failed has not been
+// tested; this one fails three times per run, on purpose.
+//
+// Each is written the way the leak would actually be written: LeakResults and
+// LeakLoci walk `l.Runs` and take what they find; LeakProjectionBytes never
+// names a field at all and simply marshals the record it was handed, which is
+// the case the field-access arm alone would miss.
+const gateLeakProbeSource = `package record
+
+import "encoding/json"
+
+func (rd *Reader) LeakResults(l *SARIFLog) []Result {
+ var out []Result
+ for ri := range l.Runs {
+ out = append(out, l.Runs[ri].Results...)
+ }
+ return out
+}
+
+func (rd *Reader) LeakLoci(l *SARIFLog) []string {
+ var out []string
+ for ri := range l.Runs {
+ for si := range l.Runs[ri].Results {
+ if p := primaryPath(&l.Runs[ri].Results[si]); p != "" {
+ out = append(out, p)
+ }
+ }
+ }
+ return out
+}
+
+func LeakProjectionBytes(l *SARIFLog) ([]byte, error) {
+ return json.Marshal(l)
+}
+`
+
+// TestTheSourceGuardCatchesTheLeaksThatDefeatedItsPredecessor is the negative
+// control for TestResultReachingEntryPointsAreGated.
+//
+// It indexes the REAL package plus gateLeakProbeSource and runs the identical
+// classification — idx.classify, the same function the guard calls — over the
+// three synthetic entry points. Each must come back gateUngated, and none may
+// appear in the allowlist.
+//
+// This is what the previous guard never had. It was a type whitelist that had
+// never been shown to reject anything, so nobody noticed that the shapes it
+// listed were not the shapes a leak takes.
+func TestTheSourceGuardCatchesTheLeaksThatDefeatedItsPredecessor(t *testing.T) {
+ idx, allow := gateIndexWithProbe(t, gateLeakProbeSource)
+
+ // The control is only meaningful if the real package still classifies
+ // correctly alongside the synthetic file.
+ if got := idx.classify("Reader.ManifestFromLog"); got != gateGated {
+ t.Fatalf("with the leak file indexed, Reader.ManifestFromLog classifies as %v, want "+
+ "gateGated; the control is measuring something other than the real analysis", got)
+ }
+
+ for _, key := range []string{"Reader.LeakResults", "Reader.LeakLoci", "LeakProjectionBytes"} {
+ t.Run(key, func(t *testing.T) {
+ if idx.decl[key] == nil {
+ t.Fatalf("%s is not in the index; the synthetic file was not read", key)
+ }
+ if _, ok := allow[key]; ok {
+ t.Fatalf("%s is in gateUngatedAllowlist, which would let the control pass "+
+ "without the analysis doing anything", key)
+ }
+ switch idx.classify(key) {
+ case gateUngated:
+ // The guard would report this one. Correct.
+ case gateGated:
+ t.Errorf("%s classifies as gateGated: the analysis believes it asks the read "+
+ "gate. It does not — it walks the record and returns what it finds.", key)
+ case gateNoResults:
+ t.Errorf("%s classifies as gateNoResults: the analysis cannot see that it "+
+ "reaches a half's results. This is EXACTLY the hole the type whitelist "+
+ "had, and %s handed out a half's data through it.", key, key)
+ }
+ })
+ }
+}
+
+// ===========================================================================
+// THE ADVERSARY'S SIX CLOSED DEFEATS — permanent negative controls
+// ===========================================================================
+//
+// An adversary ran sixteen attacks at this guard and won eight. Six of the
+// eight are closed, and each closed shape lives below as source text so the
+// guard is re-defeated-and-caught on every run. The two that are NOT closed —
+// the fabricated seal and partial obedience — are documented in KNOWN LIMITS
+// above and deliberately have no probe here, because a probe that passed would
+// be a lie about what this analysis can see.
+//
+// Every function below is written the way the bypass would actually be
+// written. None of them is exotic; that is the point.
+const gateAdversaryProbeSource = `package record
+
+// ATTACK 9 — call the gate, assign the answer to the blank identifier, hand
+// out the results anyway. A call that nobody obeys.
+func (rd *Reader) LeakBlankErrGate(l *SARIFLog) []Result {
+ _ = HalfReadGate(l.Properties.AuditID, halfSealOfRun(l, &l.Runs[0]))
+ var out []Result
+ for ri := range l.Runs {
+ out = append(out, l.Runs[ri].Results...)
+ }
+ return out
+}
+
+// ATTACK 9 (variant) — the same, with the call in statement position, which
+// discards the error without even naming it.
+func (rd *Reader) LeakStatementGate(l *SARIFLog) []Result {
+ HalfReadGate(l.Properties.AuditID, halfSealOfRun(l, &l.Runs[0]))
+ var out []Result
+ for ri := range l.Runs {
+ out = append(out, l.Runs[ri].Results...)
+ }
+ return out
+}
+
+// ATTACK 10 — never touch the gate at all. Declare locals NAMED after it and
+// return every result. Against a mention-matching analysis this reads as
+// obedient; it does not call anything.
+func (rd *Reader) LeakNamedLocals(l *SARIFLog) []Result {
+ readOrder := 0
+ Readable := true
+ HalfReadGate := "asked"
+ var out []Result
+ for ri := range l.Runs {
+ readOrder++
+ if Readable && HalfReadGate != "" {
+ out = append(out, l.Runs[ri].Results...)
+ }
+ }
+ return out
+}
+
+// ATTACK 11 — hide the results read behind a package-level func VALUE. The
+// exported method's own signature names no record type; everything
+// interesting happens inside a variable.
+var hiddenLociRead = func(l *SARIFLog) []string {
+ var out []string
+ for ri := range l.Runs {
+ for si := range l.Runs[ri].Results {
+ out = append(out, l.Runs[ri].Results[si].RuleID)
+ }
+ }
+ return out
+}
+
+func (rd *Reader) LeakLociViaFuncValue() []string {
+ return hiddenLociRead(rd.loaded)
+}
+
+// ATTACK 12 — an exported method on an UNEXPORTED receiver, handed out by an
+// exported constructor. The lower-case receiver is cosmetic: a caller who
+// holds the value can call the method.
+type leakHandle struct{ log *SARIFLog }
+
+func NewLeakHandle(l *SARIFLog) *leakHandle { return &leakHandle{log: l} }
+
+func (h *leakHandle) Loci() []string {
+ var out []string
+ for ri := range h.log.Runs {
+ for si := range h.log.Runs[ri].Results {
+ out = append(out, h.log.Runs[ri].Results[si].RuleID)
+ }
+ }
+ return out
+}
+
+// ATTACK 13 — an exported package-level func-typed VARIABLE. Public API in
+// every sense a caller cares about, and invisible to a scan that only walks
+// FuncDecls.
+var LeakExportedFuncVar = func(l *SARIFLog) []Result {
+ var out []Result
+ for ri := range l.Runs {
+ out = append(out, l.Runs[ri].Results...)
+ }
+ return out
+}
+
+// POSITIVE CONTROL — an entry point that actually obeys. It must classify as
+// gated, or the new call-and-use rule has been tightened into a rule that
+// nothing can satisfy, and the guard would be flagging the whole package.
+func (rd *Reader) HonestGatedRead(l *SARIFLog) []Result {
+ var out []Result
+ for ri := range l.Runs {
+ if err := HalfReadGate(l.Properties.AuditID, halfSealOfRun(l, &l.Runs[ri])); err != nil {
+ continue
+ }
+ out = append(out, l.Runs[ri].Results...)
+ }
+ return out
+}
+`
+
+// TestTheSourceGuardCatchesTheAdversarysDefeats is the negative control for
+// the six attacks that beat the previous version of this analysis.
+//
+// It indexes the REAL package plus gateAdversaryProbeSource and runs the
+// identical classification — idx.classify and idx.exportedEntryPoints, the
+// same functions the guard calls. Attacks 9, 10, 11, 12 and 13 must come back
+// gateUngated; the honest one must come back gateGated, because a guard that
+// flags everything is as useless as one that flags nothing.
+func TestTheSourceGuardCatchesTheAdversarysDefeats(t *testing.T) {
+ idx, allow := gateIndexWithProbe(t, gateAdversaryProbeSource)
+
+ // The control is only meaningful if the real package still classifies
+ // correctly alongside the synthetic file.
+ if got := idx.classify("Reader.ManifestFromLog"); got != gateGated {
+ t.Fatalf("with the adversary file indexed, Reader.ManifestFromLog classifies as %v, "+
+ "want gateGated; the control is measuring something other than the real analysis",
+ got)
+ }
+
+ entry := map[string]bool{}
+ for _, key := range idx.exportedEntryPoints() {
+ entry[key] = true
+ }
+
+ defeats := []struct {
+ key string
+ attack string
+ what string
+ mustBeEP bool // the attack is about ENUMERATION, not just classification
+ }{
+ {"Reader.LeakBlankErrGate", "9",
+ "calls HalfReadGate and assigns the error to the blank identifier", false},
+ {"Reader.LeakStatementGate", "9v",
+ "calls HalfReadGate in statement position, discarding the error", false},
+ {"Reader.LeakNamedLocals", "10",
+ "never calls the gate; it declares locals named after it", false},
+ {"Reader.LeakLociViaFuncValue", "11",
+ "reads the results inside a package-level func value", false},
+ {"leakHandle.Loci", "12",
+ "is an exported method on an unexported type handed out by NewLeakHandle", true},
+ {"LeakExportedFuncVar", "13",
+ "is an exported package-level func-typed variable", true},
+ }
+
+ for _, d := range defeats {
+ t.Run("attack"+d.attack+"_"+d.key, func(t *testing.T) {
+ if idx.decl[d.key] == nil {
+ t.Fatalf("%s is not in the index; the synthetic file was not read, or "+
+ "gateIndexFiles no longer indexes this declaration form", d.key)
+ }
+ if _, ok := allow[d.key]; ok {
+ t.Fatalf("%s is in gateUngatedAllowlist, which would let the control pass "+
+ "without the analysis doing anything", d.key)
+ }
+ if d.mustBeEP && !entry[d.key] {
+ t.Fatalf("%s %s, but exportedEntryPoints does not enumerate it, so the guard "+
+ "never classifies it at all. Attack %s is open again.",
+ d.key, d.what, d.attack)
+ }
+ switch idx.classify(d.key) {
+ case gateUngated:
+ // The guard would report this one. Correct.
+ case gateGated:
+ t.Errorf("%s classifies as gateGated: the analysis believes it asks the read "+
+ "gate. It does not — it %s. Attack %s is open again.",
+ d.key, d.what, d.attack)
+ case gateNoResults:
+ t.Errorf("%s classifies as gateNoResults: the analysis cannot see that it "+
+ "reaches a half's results, although it %s. Attack %s is open again.",
+ d.key, d.what, d.attack)
+ }
+ })
+ }
+
+ // The other half of the claim: obedience is still recognisable.
+ t.Run("positive_control_HonestGatedRead", func(t *testing.T) {
+ if got := idx.classify("Reader.HonestGatedRead"); got != gateGated {
+ t.Errorf("Reader.HonestGatedRead classifies as %v, want gateGated. It calls "+
+ "HalfReadGate and checks the error in an if-statement, which is what obeying "+
+ "the gate looks like. If this fails, the call-and-use rule rejects real "+
+ "obedience and the guard is now noise.", got)
+ }
+ })
+}
+
+// gateAllowlistHashProbeSource and gateAllowlistHashProbeRewritten are the same
+// allowlisted-looking function before and after adversary attack 16: the name
+// is untouched, the body is not.
+const gateAllowlistHashProbeSource = `package record
+
+// ExemptLooking is the shape of an allowlisted entry point: it walks the
+// record and returns only an error, so "it carries no finding content out of a
+// half" is a true reason to write beside it.
+func ExemptLooking(l *SARIFLog) error {
+ for ri := range l.Runs {
+ if len(l.Runs[ri].Results) == 0 {
+ return errors.New("empty half")
+ }
+ }
+ return nil
+}
+`
+
+const gateAllowlistHashProbeRewritten = `package record
+
+// ExemptLooking is the shape of an allowlisted entry point: it walks the
+// record and returns only an error, so "it carries no finding content out of a
+// half" is a true reason to write beside it.
+func ExemptLooking(l *SARIFLog) error {
+ for ri := range l.Runs {
+ if len(l.Runs[ri].Results) == 0 {
+ return errors.New("empty half")
+ }
+ leaked = append(leaked, l.Runs[ri].Results...)
+ }
+ return nil
+}
+`
+
+const gateAllowlistHashProbeRecommented = `package record
+
+// ExemptLooking: the comment was rewritten and nothing else. The exemption
+// must survive this, or every doc edit becomes a false alarm and the hash gets
+// deleted by the third person it annoys.
+func ExemptLooking(l *SARIFLog) error {
+ for ri := range l.Runs {
+ // a half with no results is not a half
+ if len(l.Runs[ri].Results) == 0 {
+ return errors.New("empty half")
+ }
+ }
+ return nil
+}
+`
+
+// TestAnAllowlistEntryIsAClaimAboutABodyNotAName is the negative control for
+// adversary attack 16.
+//
+// The attack: the allowlist matches by NAME, so rewrite the body of an
+// allowlisted function and inherit an exemption that was written about
+// something else. gateUngatedAllowlist now records the hash of the body the
+// reason was written about, and TestResultReachingEntryPointsAreGated fails
+// when the body no longer hashes to it.
+//
+// This test asserts the two properties that mechanism needs to be worth
+// having: a changed body changes the hash, and a changed COMMENT does not. The
+// second matters as much as the first — a hash that expires on every doc edit
+// is a hash somebody deletes.
+func TestAnAllowlistEntryIsAClaimAboutABodyNotAName(t *testing.T) {
+ hashOf := func(src string) string {
+ t.Helper()
+ fset := token.NewFileSet()
+ file, err := parser.ParseFile(fset, "zz_gate_hash_probe.go", src, parser.ParseComments)
+ if err != nil {
+ t.Fatalf("parsing the synthetic hash probe: %v", err)
+ }
+ idx := gateIndexFiles(t, map[string]*ast.File{"zz_gate_hash_probe.go": file}, fset)
+ h := idx.gateBodyHash("ExemptLooking")
+ if h == "" {
+ t.Fatal("gateBodyHash returned nothing for the probe; the hash cannot be trusted")
+ }
+ return h
+ }
+
+ original := hashOf(gateAllowlistHashProbeSource)
+
+ if got := hashOf(gateAllowlistHashProbeRewritten); got == original {
+ t.Errorf("rewriting the BODY of an allowlisted function did not change its hash "+
+ "(both %s). Attack 16 is open: an exemption written about one implementation "+
+ "would be inherited by another.", got)
+ }
+ if got := hashOf(gateAllowlistHashProbeRecommented); got != original {
+ t.Errorf("rewriting only a COMMENT changed the body hash (%s -> %s). Every doc edit "+
+ "would expire an exemption, which is how the mechanism gets deleted instead of "+
+ "maintained.", original, got)
+ }
+
+ // And the live allowlist must actually carry hashes: an entry with an
+ // empty or placeholder body hash is a name-only claim wearing the costume.
+ for key, ex := range gateUngatedAllowlist() {
+ if strings.TrimSpace(ex.body) == "" {
+ t.Errorf("gateUngatedAllowlist[%q] has no body hash; it is a claim about a name",
+ key)
+ }
+ if strings.Trim(ex.body, "0") == "" {
+ t.Errorf("gateUngatedAllowlist[%q] has placeholder body hash %q; record the real "+
+ "one", key, ex.body)
+ }
+ }
+}
+
+// TestReadabilityAnsweringEntryPointsAreProbed keeps the behavioural table
+// honest for the entry points the reachability analysis cannot speak to: the
+// ones that return a SEAL rather than results.
+//
+// HalfSeal and AuditSeal ARE the readability answer — HalfSeal.Readable is the
+// gate as a bool — so a new exported function returning one is a new way to
+// answer "may this half be read", and it must be probed against the
+// no-half-is-readable scenarios rather than merely reviewed.
+//
+// This is a coverage check on gateAuditedEntryPoints, not a leak check; the
+// leak check is TestResultReachingEntryPointsAreGated above.
+func TestReadabilityAnsweringEntryPointsAreProbed(t *testing.T) {
+ audited := map[string]bool{}
+ for _, e := range gateAuditedEntryPoints() {
+ if (e.probe == nil) == (e.exempt == "") {
+ t.Errorf("entry %q must have exactly one of probe and exempt", e.name)
+ }
+ if audited[e.name] {
+ t.Errorf("entry %q is listed twice", e.name)
+ }
+ audited[e.name] = true
+ }
+
+ idx := gateParseSource(t)
+ seal := map[string]bool{"HalfSeal": true, "AuditSeal": true}
+
+ found := 0
+ for _, key := range idx.exportedEntryPoints() {
+ fn := idx.decl[key]
+ if fn.Type.Results == nil {
+ continue
+ }
+ answers := false
+ for _, res := range fn.Type.Results.List {
+ if seal[gateBaseTypeName(res.Type)] {
+ answers = true
+ }
+ }
+ if !answers {
+ continue
+ }
+ found++
+ if !audited[key] {
+ t.Errorf("%s (%s) hands out a seal, which IS the readability answer, but is not in "+
+ "gateAuditedEntryPoints. Add a probe asserting it refuses every scenario in "+
+ "gateScenarios, or an `exempt` reason saying why it cannot answer wrongly.",
+ key, idx.file[key])
+ }
+ }
+ if found == 0 {
+ t.Fatal("no exported entry point returns a HalfSeal or an AuditSeal, which cannot be " +
+ "true of this package; the scan is broken")
+ }
+ t.Logf("seal-returning entry points: %d found, all probed", found)
+}
+
+// gateBaseTypeName reduces a type expression to the identifier a caller would
+// name: *T, []T, map[K]T and pkg.T all reduce to T.
+func gateBaseTypeName(e ast.Expr) string {
+ switch t := e.(type) {
+ case *ast.StarExpr:
+ return gateBaseTypeName(t.X)
+ case *ast.ArrayType:
+ return gateBaseTypeName(t.Elt)
+ case *ast.MapType:
+ return gateBaseTypeName(t.Value)
+ case *ast.Ellipsis:
+ return gateBaseTypeName(t.Elt)
+ case *ast.Ident:
+ return t.Name
+ case *ast.SelectorExpr:
+ return t.Sel.Name
+ }
+ return ""
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 M1 — an expired audit is not readable, and says so distinctly
+// ---------------------------------------------------------------------------
+
+// readOrder and ManifestFromLog computed readability as
+// IsReadableHalfStatus(status) ALONE, ignoring l.Properties.State, so an
+// EXPIRED audit was fully readable: nine cards, six of them actionable,
+// against a claim window that had already closed. The handoff rows behind
+// those cards are subject to ReclaimExpired, so the agent's work would have
+// had nowhere legal to land.
+//
+// The manifest must still REPORT the expired half and its result count, for
+// the same reason it reports an unsealed one, so "expired" and "empty" stay
+// distinguishable.
+func TestExpiredAuditYieldsNoCardsAndSaysWhy(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) { l.Properties.State = StateExpired })
+ rd := rpReader(t, l)
+
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ if len(cards) != 0 {
+ t.Errorf("an expired audit produced %d task cards; the claim window has closed and "+
+ "ReclaimExpired owns the handoff rows behind them", len(cards))
+ }
+
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ if len(m.Cards) != 0 {
+ t.Errorf("the manifest listed %d card refs for an expired audit", len(m.Cards))
+ }
+ if m.Index.Counts.Total != 0 {
+ t.Errorf("counts.total = %d, want 0", m.Index.Counts.Total)
+ }
+ for _, h := range m.Halves {
+ if h.Readable {
+ t.Errorf("half %s reports readable on an expired audit; sealing.go's HalfSeal.Readable "+
+ "says false for the same (status, state) pair, and two exported readiness paths "+
+ "giving two answers is a gate that is only advisory", h.Half)
+ }
+ if h.Status != HalfStatusSealed {
+ t.Errorf("half %s reports status %q; the fixture sealed both halves and expiry "+
+ "must not rewrite what happened", h.Half, h.Status)
+ }
+ // "expired" and "empty" must not arrive as the same observation.
+ if h.Results == 0 {
+ t.Errorf("half %s reports 0 results; the withheld results must still be counted", h.Half)
+ }
+ if !strings.Contains(h.ReadRefusal, "claim timeout") {
+ t.Errorf("half %s refusal is %q, want the expiry reason and not the unsealed one",
+ h.Half, h.ReadRefusal)
+ }
+ }
+
+ // The unsealed refusal is a DIFFERENT sentence, so a consumer can tell
+ // "this half never sealed" from "the audit expired holding a sealed half"
+ // without joining Status against State itself.
+ unsealed := rpFixture(t, func(l *SARIFLog) {
+ l.Runs[1].Properties.Status = HalfStatusRunning
+ l.Runs[1].Properties.SealedAt = nil
+ l.Properties.State = StateSastSealed
+ l.Properties.DastStatus = DastStatusRunning
+ })
+ um, err := rpReader(t, unsealed).BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ for _, h := range um.Halves {
+ if h.Half != HalfDast {
+ continue
+ }
+ if strings.Contains(h.ReadRefusal, "claim timeout") {
+ t.Errorf("an unsealed half reports the expiry refusal %q", h.ReadRefusal)
+ }
+ if h.ReadRefusal == "" {
+ t.Error("an unsealed half reports no refusal reason at all")
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 M3 (consequence 2) — a spill is never a dangling reference
+// ---------------------------------------------------------------------------
+
+// NewReader left Reader.Blobs nil, so the spilled bytes existed only in
+// Manifest.Blobs, which is `json:"-"`. A caller that marshalled the manifest
+// and dropped the struct — the obvious thing to do with a projection — shipped
+// a Tier-0 manifest whose most load-bearing content, the materialised read
+// order, was a `sha256:` reference to bytes nobody held.
+func TestSpilledBlobsSurviveDroppingTheManifest(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ for i := 0; i < 400; i++ {
+ l.Runs[0].Results = append(l.Runs[0].Results, rpSastResult(
+ fmt.Sprintf("sast:9%03d", i), float64(500-i),
+ EvidenceClassSastStaticOnly, VerdictTruePositive, true,
+ fmt.Sprintf("app/pkg%02d/module%03d.py", i%20, i), byte(i)))
+ }
+ })
+
+ rd := NewReader(RecordMap{rpAuditID: l})
+ if rd.Blobs == nil {
+ t.Fatal("NewReader left Reader.Blobs nil; every spill it produces is a reference to " +
+ "bytes that live only in a struct the caller was told not to serialise")
+ }
+
+ // Everything a caller keeps: the marshalled manifest and the Reader. The
+ // Manifest struct itself is deliberately not retained past this point.
+ refs := map[string]int{}
+ raw := func() []byte {
+ m, err := rd.BuildManifest(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildManifest: %v", err)
+ }
+ if len(m.Spills) == 0 {
+ t.Fatal("a 409-finding record must spill something for this test to mean anything")
+ }
+ for _, s := range m.Spills {
+ refs[s.Ref] = s.Bytes
+ }
+ return rpMarshal(t, m)
+ }()
+
+ var shipped Manifest
+ if err := json.Unmarshal(raw, &shipped); err != nil {
+ t.Fatalf("the shipped manifest does not round-trip: %v", err)
+ }
+ if len(shipped.Blobs) != 0 {
+ t.Error("the spilled bytes came back through the serialised manifest; the spill did nothing")
+ }
+
+ retained := rd.RetainedBlobs()
+ for _, s := range shipped.Spills {
+ blob, ok := retained[s.Ref]
+ if !ok {
+ t.Fatalf("spill %s references blob %s, which the Reader did not retain: "+
+ "the shipped manifest carries a dangling reference", s.Field, s.Ref)
+ }
+ if BlobRef(blob) != s.Ref {
+ t.Errorf("retained blob for %s does not hash to its own reference", s.Field)
+ }
+ if len(blob) != s.Bytes {
+ t.Errorf("spill %s reports %d bytes, the retained blob is %d", s.Field, s.Bytes, len(blob))
+ }
+ }
+
+ // Draining is how a long-lived Reader avoids accumulating every blob it
+ // ever spilled.
+ drained := rd.DrainRetainedBlobs()
+ if len(drained) != len(retained) {
+ t.Errorf("DrainRetainedBlobs returned %d blobs, RetainedBlobs had %d", len(drained), len(retained))
+ }
+ if len(rd.RetainedBlobs()) != 0 {
+ t.Error("DrainRetainedBlobs did not clear the retainer")
+ }
+
+ // A caller with real Tier-2 storage replaces the sink, and then the Reader
+ // retains nothing it did not write.
+ written := map[string][]byte{}
+ own := NewReader(RecordMap{rpAuditID: l})
+ own.Blobs = func(ref string, content []byte) error {
+ written[ref] = content
+ return nil
+ }
+ if _, err := own.BuildManifest(rpAuditID); err != nil {
+ t.Fatalf("BuildManifest with a caller-supplied sink: %v", err)
+ }
+ if len(written) == 0 {
+ t.Error("a caller-supplied BlobSink was never called")
+ }
+ if len(own.RetainedBlobs()) != 0 {
+ t.Error("the Reader retained blobs the caller was already persisting")
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 m1 — a borrowed locus is labelled, and does not carry the action
+// ---------------------------------------------------------------------------
+
+// Both members of one cluster were independently actionable and pointed at the
+// same line, so one defect produced two patch tasks writing into two different
+// `fixes` arrays and charging the budget twice. The DAST member also presented
+// its SAST peer's file and line as its own locus, which is inference reported
+// as observation.
+func TestBorrowedLocusIsLabelledAndWithholdsTheAction(t *testing.T) {
+ l := rpFixture(t, nil)
+ rd := rpReader(t, l)
+ cards, err := rd.BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ byID := map[string]TaskCard{}
+ for _, c := range cards {
+ byID[c.FindingID] = c
+ }
+ sast, dast := byID["sast:0001"], byID["dast:0101"]
+
+ // The card stays self-contained: the DAST member still SEES the file and
+ // the line, because an agent reading it needs to know where the defect is.
+ if dast.Locus.Path != "app/db.py" || dast.Static == nil {
+ t.Fatalf("the DAST card lost the peer's static evidence: locus=%+v static=%v",
+ dast.Locus, dast.Static != nil)
+ }
+ // ...but it is labelled as the peer's observation, not this finding's.
+ if dast.Locus.BorrowedFrom != "sast:0001" {
+ t.Errorf("dast card locus.borrowedFrom = %q, want %q: a DAST probe did not observe "+
+ "app/db.py:412, the correlation concluded it, and a card that does not say so "+
+ "presents inference as observation", dast.Locus.BorrowedFrom, "sast:0001")
+ }
+ if sast.Locus.BorrowedFrom != "" {
+ t.Errorf("the SAST card claims a borrowed locus (%q); it observed its own",
+ sast.Locus.BorrowedFrom)
+ }
+
+ // Exactly one member of the cluster carries the patch task.
+ if !sast.Actionable {
+ t.Error("the SAST member observed the line and must carry the action")
+ }
+ if dast.Actionable {
+ t.Error("both cluster members are actionable: one defect, two patch tasks, two handoff " +
+ "rows charged against the budget R.11's reservation is dividing")
+ }
+ if len(dast.ActionBlockers) == 0 {
+ t.Fatal("the withheld card gives no reason; 'not actionable' is never unexplained")
+ }
+ if !strings.Contains(strings.Join(dast.ActionBlockers, " "), "sast:0001") {
+ t.Errorf("the blocker does not name the peer that carries the action: %q", dast.ActionBlockers)
+ }
+
+ // Withholding is the ONE legal direction of divergence, so it is not a
+ // disagreement with the record.
+ for i := range l.Runs[1].Results {
+ if l.Runs[1].Results[i].Properties.FindingID != "dast:0101" {
+ continue
+ }
+ if err := dast.CheckAgainstRecord(&l.Runs[1].Results[i]); err != nil {
+ t.Errorf("withholding an action the record allows must not be reported as a "+
+ "disagreement: %v", err)
+ }
+ }
+
+ // An UNCLUSTERED DAST finding observed nothing to borrow and is unaffected.
+ if lone := byID["dast:0102"]; lone.Locus.BorrowedFrom != "" || !lone.Actionable {
+ t.Errorf("an unclustered DAST card was withheld too: borrowedFrom=%q actionable=%t",
+ lone.Locus.BorrowedFrom, lone.Actionable)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 m2 — the card does not take `verified` on trust
+// ---------------------------------------------------------------------------
+
+// `verified` is an S7 gate of the same class as the host gate, and the host
+// gate is enforced a third time on the card precisely because the card is what
+// the agent receives. `verified` was copied across without re-checking the
+// signals, so a malformed record put an unearned verification in front of the
+// agent and CheckAgainstRecord — whose stated job is to name every
+// contradiction — stayed silent.
+func TestCardDoesNotTakeCorrelationVerifiedOnTrust(t *testing.T) {
+ l := rpFixtureLog()
+ co := l.Runs[0].Results[0].Properties.Correlation
+ co.Signals = []SignalWeight{
+ {Name: CorrelationSignalCweMatch, Weight: 0.5, Detail: "CWE-89"},
+ {Name: CorrelationSignalParameterName, Weight: 0.5, Detail: "username"},
+ }
+ // Two signals and not CWE-only, so the link itself is legal; what is not
+ // legal is claiming verification off them.
+ if err := l.Validate(); err == nil {
+ t.Fatal("the fixture must be a record contract.go REJECTS; if Validate() accepts it, " +
+ "this test is no longer exercising the malformed-producer path")
+ }
+ if err := MaskRecord(l); err != nil {
+ t.Fatalf("MaskRecord: %v", err)
+ }
+
+ cards, err := NewReader(RecordMap{rpAuditID: l}).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ var card *TaskCard
+ for i := range cards {
+ if cards[i].FindingID == "sast:0001" {
+ card = &cards[i]
+ }
+ }
+ if card == nil || card.Correlation == nil {
+ t.Fatal("the clustered card vanished")
+ }
+ if card.Correlation.Verified {
+ t.Errorf("the card asserts verified=true off signals %v; only %q or %q earns it, and "+
+ "confidence alone never qualifies (00-SPINE.md S7)",
+ card.Correlation.Signals, CorrelationSignalResponseStackTrace, CorrelationSignalRerunFlip)
+ }
+
+ // And a hand-edited card that grants it is reported as a disagreement.
+ card.Correlation.Verified = true
+ err = card.CheckAgainstRecord(&l.Runs[0].Results[0])
+ if err == nil {
+ t.Fatal("CheckAgainstRecord stayed silent on an unearned verified; naming every " +
+ "contradiction is its whole job")
+ }
+ if !strings.Contains(err.Error(), "verified") {
+ t.Errorf("the disagreement does not name the field: %v", err)
+ }
+
+ // The legitimate case still passes both ways: a stack-trace signal earns it.
+ good := rpFixture(t, nil)
+ goodCards, err := rpReader(t, good).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ for _, c := range goodCards {
+ if c.FindingID == "sast:0001" && !c.Correlation.Verified {
+ t.Error("a responseStackTrace signal is present; the clamp must not withhold an earned verification")
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 m3 — a card names peers the reader cannot fetch
+// ---------------------------------------------------------------------------
+
+// When one half has not sealed its results correctly produce no cards, but the
+// other half's card still named them as peers with `verified: true`. A card is
+// documented as self-contained, and that one asserted a verified link to
+// evidence the read gate had not opened.
+func TestCardMarksPeersTheReadGateWithheld(t *testing.T) {
+ l := rpFixture(t, func(l *SARIFLog) {
+ l.Runs[1].Properties.Status = HalfStatusRunning
+ l.Runs[1].Properties.SealedAt = nil
+ l.Properties.State = StateSastSealed
+ l.Properties.DastStatus = DastStatusRunning
+ })
+ cards, err := rpReader(t, l).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+
+ have := map[string]bool{}
+ for _, c := range cards {
+ have[c.FindingID] = true
+ }
+ var clustered *TaskCard
+ for i := range cards {
+ if cards[i].FindingID == "sast:0001" {
+ clustered = &cards[i]
+ }
+ }
+ if clustered == nil || clustered.Correlation == nil {
+ t.Fatal("the cluster's readable member lost its correlation")
+ }
+
+ // The link is not deleted — "not linked" and "linked to something not yet
+ // readable" are different facts — but the unfetchable peer is named.
+ if len(clustered.Correlation.Peers) == 0 {
+ t.Fatal("the link was deleted rather than marked")
+ }
+ if got := clustered.Correlation.PeersUnreadable; len(got) != 1 || got[0] != "dast:0101" {
+ t.Errorf("peersUnreadable = %q, want [dast:0101]", got)
+ }
+ for _, p := range clustered.Correlation.Peers {
+ if have[p] {
+ continue
+ }
+ found := false
+ for _, u := range clustered.Correlation.PeersUnreadable {
+ if u == p {
+ found = true
+ }
+ }
+ if !found {
+ t.Errorf("peer %q is named on a card, has no card of its own, and is not marked unreadable", p)
+ }
+ }
+ if !strings.Contains(clustered.Correlation.Caveat, "dast:0101") {
+ t.Errorf("the caveat does not say which peer cannot be fetched: %q", clustered.Correlation.Caveat)
+ }
+
+ // On a fully sealed audit every peer is fetchable and nothing is marked.
+ good, err := rpReader(t, rpFixture(t, nil)).BuildTaskCards(rpAuditID)
+ if err != nil {
+ t.Fatalf("BuildTaskCards: %v", err)
+ }
+ for _, c := range good {
+ if c.Correlation != nil && len(c.Correlation.PeersUnreadable) != 0 {
+ t.Errorf("card %s marks peers unreadable on a fully sealed audit: %q",
+ c.FindingID, c.Correlation.PeersUnreadable)
+ }
+ }
+}
diff --git a/internal/record/sarif_github.go b/internal/record/sarif_github.go
new file mode 100644
index 0000000..ef7fc23
--- /dev/null
+++ b/internal/record/sarif_github.go
@@ -0,0 +1,1634 @@
+package record
+
+// sarif_github.go — R.14, the reduced-SARIF GitHub code-scanning projection.
+//
+// ===========================================================================
+// THIS FILE IS THE SINGLE DEFINITION OF THE GITHUB PROJECTION
+// ===========================================================================
+//
+// Every GitHub code-scanning cap, every strip rule and every drop rule Anvil
+// applies lives HERE and nowhere else. Any area that needs to emit SARIF to
+// GitHub calls ProjectForGitHub; it does not re-derive a limit, re-spell a
+// cap, or re-implement the filter.
+//
+// This is not style. plan/IMPLEMENTATION-PLAN.md §6 closed TEN confirmed
+// cross-area defects whose single shared shape was "two areas that could not
+// see each other each defined the same vocabulary from their own side, and no
+// step was ever assigned to reconcile them." A second copy of `25000` in
+// another package is a second definition of GitHub's contract, and it will
+// drift the same way `dast_status` drifted into two disjoint enums.
+//
+// See GitHubProjectionOwner below, which records that ownership in code so it
+// survives the plan documents.
+//
+// ===========================================================================
+// THE READ GATE APPLIES HERE TOO, AND IT IS CALLED, NOT RE-DERIVED
+// ===========================================================================
+//
+// plan/IMPLEMENTATION-PLAN.md §6 ruling G5: "`sealed` is load-bearing: R.6
+// makes it the hard read gate ('do not allow a consumer to read a half's
+// results before that half's `status` equals `sealed`')". GitHub code scanning
+// is the most externally visible consumer Anvil has, so it is the LAST place
+// that gate may be skipped — and CRITIQUE-03 B1 found this file skipping it
+// entirely, projecting every run in `l.Runs` unconditionally.
+//
+// A run whose half sealing.go's HalfReadGate refuses contributes no results,
+// and each of them is ledgered under GitHubDropHalfNotReadable with the half's
+// `anvil/status` and the audit's `anvil/state`. The gate is CALLED — this file
+// does not ask `src.Properties.Status == HalfStatusSealed` for itself. Four
+// authors have now re-derived that question locally and four got it wrong in
+// four different ways; sealing.go's read-gate section lists them.
+//
+// AssertMasked is a precondition for the same structural reason readpath.go
+// makes it one: every surface MaskRecord covers is independently stripped
+// below, so no leak is demonstrable through this file TODAY, which means the
+// safety rests entirely on the strip list staying exhaustive.
+//
+// ===========================================================================
+// WHY THE PROJECTION IS LOSSY, AND WHY THE LOSS IS ENUMERATED
+// ===========================================================================
+//
+// research/18-unified-audit-record.md, "What Anvil loses by choosing SARIF":
+//
+// "GitHub throws away the DAST half. webRequest, webResponse, taxonomies,
+// provenance and property bags are not in GitHub's supported-property list
+// [S2], and a DAST-only result has no startLine to satisfy GitHub's
+// location requirement [S2]. Design rule: the GitHub upload is a
+// projection, not the record."
+//
+// and research/18 Risk #6:
+//
+// "GitHub silently discards the entire DAST half... If anyone treats the
+// GitHub UI as the audit, they will believe Anvil found only static
+// issues."
+//
+// Risk #6 is a risk about a HUMAN BELIEF, and no amount of correct filtering
+// addresses it. What addresses it is making the loss countable: every result
+// this projection refuses to upload is recorded in GitHubProjectionLoss with
+// its finding id and a reason, and every field it strips is counted by kind.
+// A consumer can ask "what did GitHub not get, and why" and receive a total,
+// enumerable answer instead of a shrug.
+//
+// Two distinct kinds of loss are tracked separately, because they answer
+// different questions:
+//
+// - GitHubDropReason — a whole RESULT never reaches GitHub. GitHub could
+// not render it as an alert at all.
+// - GitHubStripReason — a result reaches GitHub with FIELDS removed. The
+// alert exists but carries less than the record does.
+//
+// ===========================================================================
+// WHY THE STRIPPING IS EXPLICIT AND NOT LEFT TO GITHUB
+// ===========================================================================
+//
+// GitHub accepts any valid SARIF 2.1.0 file and ignores every property
+// outside its supported list [S2]. It would therefore "work" to upload the
+// whole record and let GitHub discard the DAST half itself. This packet
+// forbids that, and the reason is size, not tidiness: the ignored bytes still
+// count against the 10 MB gzip limit, so relying on silent ignoring converts
+// a display no-op into an upload REJECTION on exactly the large audits that
+// most need to be uploaded. Stripping is therefore done here, before the byte
+// count is taken.
+//
+// The projection also never emits an `anvil/*` property bag. That is enforced
+// structurally, not by zeroing fields: the wire types below (GitHubSARIFLog,
+// GitHubRun, GitHubResult, GitHubLocation) simply have no `properties`
+// member, so no future edit to AuditProperties, RunProperties or
+// ResultProperties can leak one into a GitHub upload. Reusing SARIFLog with
+// its properties zeroed would have emitted `"properties":{"anvil/findingId":
+// "", ...}` — a bag of empty anvil keys, which is the forbidden thing.
+//
+// ===========================================================================
+// WHAT COUNTS AS "A PHYSICAL CODE LOCATION" — STRICTER THAN Validate()
+// ===========================================================================
+//
+// contract.go has an unexported hasPhysicalCodeLocation used by Validate to
+// decide whether primaryLocationLineHash is REQUIRED. It asks a different
+// question than this file asks, and it deliberately answers it more loosely:
+// it is satisfied by any region with startLine > 0, including research/18's
+// annotated DAST result, whose endpoint location carries
+// `"region": {"startLine": 1, ...}` described in the source as a
+// "placeholder so GitHub can render it".
+//
+// That placeholder does not survive contact with GitHub. GitHub resolves
+// `artifactLocation.uri` against the repository root; an absolute URI such as
+// `https://staging.payments.internal/api/login` resolves to no file, so the
+// alert cannot render, and uploading it additionally publishes an internal
+// hostname to GitHub. So this file asks its own, stricter question — is this
+// a REPO-RELATIVE path with a real start line? — and records the difference
+// as GitHubDropLocationNotRepoRelative rather than letting a DAST result
+// through on a placeholder.
+//
+// This is not a second definition of a shared predicate. Validate answers
+// "must this record carry the hash?"; isRepoCodeLocation answers "can GitHub
+// render this?". Merging them would break Validate for legitimately
+// endpoint-located DAST results.
+//
+// ===========================================================================
+// WHAT THIS FILE DOES NOT DO: it does not COMPUTE primaryLocationLineHash
+// ===========================================================================
+//
+// It filters ON that key and drops results that lack it
+// (GitHubDropNoPrimaryLocationLineHash). It does not fill it in.
+//
+// This is a live, unresolved ownership question and it is deliberately left
+// visible rather than absorbed. fingerprint.go says the key "is owned by the
+// GitHub projection (R.14)"; plan/40-record-and-storage.md's Record Field
+// Contract names the producer as the fingerprint engine; and
+// internal/record/CRITIQUE-01.md's MAJOR 3 records that the disagreement is
+// unruled and that no code in the tree produces the value today. R.14's
+// packet scopes this file to the projection and says nothing about producing
+// a fingerprint.
+//
+// Implementing it here anyway would be the §6 defect exactly: an identity
+// hash defined in two places, silently disagreeing, with regression matching
+// failing forever and nothing to catch it. So the gap is surfaced at runtime
+// instead — a record whose results lack the key produces a projection that
+// drops every one of them and says so, by name and count.
+//
+// ===========================================================================
+// SHARDING POLICY: ONE RUN PER FILE
+// ===========================================================================
+//
+// research/18: "keep any single emitted SARIF projection under 25,000
+// results/run and 10 MB gzipped so the GitHub path never fails [S2]. A
+// full-repo audit exceeding that must shard by run, not truncate."
+//
+// This file shards by run and puts exactly one run in each file. GitHub
+// permits 20 runs per file; emitting one makes GitHubMaxRunsPerFile
+// unreachable by construction rather than by arithmetic, keeps every file as
+// small as possible against the binding 10 MB constraint, and keeps a shard
+// attributable to exactly one half of the audit.
+//
+// Sharding a run forces one further change, and it is not cosmetic: GitHub
+// keys an analysis on `runAutomationDetails.id`, so two shards uploaded under
+// the same id would make the second REPLACE the first and silently lose half
+// the alerts. Shards after the first therefore receive a distinct, derived id
+// (see shardAutomationID) and have `automationDetails.guid` cleared, because
+// a guid identifies one run object and copying it across shards asserts
+// something false. Both are recorded as strips.
+//
+// ===========================================================================
+// TRUNCATION IS NEVER SILENT AND NEVER FIRST
+// ===========================================================================
+//
+// Results are never truncated to fit; the run is split until the parts fit.
+// The one case that cannot be split is a SINGLE result whose own file exceeds
+// 10 MB gzipped. That result is dropped with
+// GitHubDropExceedsFileSizeCap rather than being allowed to make the whole
+// projection fail, and the guarantee "no returned file exceeds a documented
+// cap on any input" is preserved. If even a result-free run exceeds the cap
+// (tool metadata alone over 10 MB), ProjectForGitHub returns an error: at
+// that point nothing legal can be emitted and pretending otherwise would be
+// worse.
+
+import (
+ "bytes"
+ "compress/gzip"
+ "encoding/json"
+ "fmt"
+ "strings"
+)
+
+// GitHubProjectionOwner records, in code, which step owns this logic, in the
+// same spirit as AreaMappingOwners in contract.go: the plan documents are not
+// compiled and a later area cannot grep them from its own package.
+//
+// Area O's O.9 (the GitHub upload step) CONSUMES ProjectForGitHub. It does
+// not re-implement the caps, the strip list or the shard rule. A second
+// implementation of a documented external limit is a second definition of
+// that limit.
+const GitHubProjectionOwner = "R.14 — internal/record/sarif_github.go is the single definition of the " +
+ "GitHub code-scanning projection: its caps, its strip list, its drop rules and its shard policy. " +
+ "O.9 calls ProjectForGitHub; it does not fork this logic (plan/IMPLEMENTATION-PLAN.md §6)."
+
+// ---------------------------------------------------------------------------
+// GitHub's documented limits
+//
+// Every number here is quoted from research/18-unified-audit-record.md,
+// "What GitHub actually accepts — hard numbers", sourced to [S2] (GitHub's
+// SARIF-support documentation, graded A but explicitly "vendor-specific
+// behaviour; limits can change without notice"). They are declared once,
+// here, so that a change to GitHub's documentation is a one-line change in
+// one package.
+// ---------------------------------------------------------------------------
+
+const (
+ // GitHubMaxResultsPerRun is GitHub's "25,000 results per run" limit.
+ GitHubMaxResultsPerRun = 25000
+
+ // GitHubDisplayedResultsPerRun is the number of results GitHub actually
+ // DISPLAYS per run, "only the top 5,000 by severity". It is not a cap
+ // and nothing here enforces it; it is declared so a consumer reporting
+ // "we uploaded 25,000 alerts" can also say how many a human will see.
+ GitHubDisplayedResultsPerRun = 5000
+
+ // GitHubMaxRunsPerFile is GitHub's "20 runs per file" limit. This file
+ // emits one run per file, so the limit is unreachable by construction;
+ // it is still checked by WithinCaps, because an invariant that is only
+ // true by construction stops being true the moment the construction
+ // changes.
+ GitHubMaxRunsPerFile = 20
+
+ // GitHubRunsPerProjectedFile is Anvil's own, stricter policy. See the
+ // sharding-policy section of this file's header.
+ GitHubRunsPerProjectedFile = 1
+
+ // GitHubMaxGzipBytes is GitHub's "10 MB gzip-compressed file" limit.
+ // This is the binding constraint in practice: a full-repo audit hits it
+ // long before it hits 25,000 results.
+ GitHubMaxGzipBytes = 10 * 1024 * 1024
+
+ // GitHubMaxRulesPerRun is GitHub's "25,000 rules per run" limit. A
+ // projected run emits only the rules its own results reference, and a
+ // run carries at most GitHubMaxResultsPerRun results, so distinct rules
+ // can never exceed distinct results and this cap is unreachable. Checked
+ // anyway by WithinCaps.
+ GitHubMaxRulesPerRun = 25000
+
+ // GitHubMaxToolExtensionsPerRun is GitHub's "100 tool extensions per
+ // run" limit. contract.go's Tool carries a driver and no extensions, so
+ // Anvil emits zero. Declared for completeness of the cap table.
+ GitHubMaxToolExtensionsPerRun = 100
+
+ // GitHubMaxLocationsPerResult is GitHub's "1,000 locations per result
+ // (100 displayed)" limit. Enforced by truncation, counted as a strip.
+ GitHubMaxLocationsPerResult = 1000
+
+ // GitHubMaxThreadFlowLocationsPerResult is GitHub's "10,000 thread-flow
+ // locations per result (top 1,000 displayed)" limit, counted across
+ // every code flow of one result. Enforced by truncation, counted as a
+ // strip.
+ GitHubMaxThreadFlowLocationsPerResult = 10000
+)
+
+// ---------------------------------------------------------------------------
+// Loss vocabulary — why a whole result never reached GitHub
+// ---------------------------------------------------------------------------
+
+// GitHubDropReason names, in one closed vocabulary, every reason this
+// projection refuses to upload a result.
+//
+// Lowercase snake_case, matching plan/IMPLEMENTATION-PLAN.md §6: "Lowercase
+// snake_case is the record's convention throughout."
+//
+// These are NOT record enums. They never appear in a record, a column or an
+// `anvil/*` bag; they are the projection's own diagnostic vocabulary, owned
+// by R.14 inside area 40. Nothing outside this package may declare a second
+// set of them.
+type GitHubDropReason string
+
+const (
+ // GitHubDropHalfNotReadable: the result's HALF did not pass the read gate
+ // — its `anvil/status` is not `sealed`, or the audit has expired. This
+ // gate is evaluated per RUN and outranks every per-result reason below,
+ // because none of them is a question worth asking about results a consumer
+ // is not permitted to read at all.
+ //
+ // plan/IMPLEMENTATION-PLAN.md §6 ruling G5: "`sealed` is load-bearing: R.6
+ // makes it the hard read gate ('do not allow a consumer to read a half's
+ // results before that half's `status` equals `sealed`')". CRITIQUE-03 B1
+ // found this file asking nobody: it projected every run unconditionally,
+ // so a `running` half's provisional findings and a `failed` half's
+ // crash-truncated output were both publishable to GitHub code scanning —
+ // the most externally visible consumer Anvil has, read by humans and by
+ // branch-protection rules, and keyed on `runAutomationDetails.id` so that
+ // a premature upload is REPLACED by the real one after the seal.
+ GitHubDropHalfNotReadable GitHubDropReason = "half_not_readable"
+
+ // GitHubDropNoLocations: the result carries no `locations[]` at all.
+ // GitHub requires `message.text`, `locations[]` and
+ // `partialFingerprints` on every result.
+ GitHubDropNoLocations GitHubDropReason = "no_locations"
+
+ // GitHubDropNoPhysicalLocation: the result's PRIMARY location
+ // (`locations[0]`) carries no `physicalLocation`. A logical location
+ // alone cannot render as a code-scanning alert.
+ GitHubDropNoPhysicalLocation GitHubDropReason = "no_physical_location"
+
+ // GitHubDropLocationNotRepoRelative: the primary location's
+ // `artifactLocation.uri` is not a repository-relative path — it is an
+ // absolute URI (`https://…`, a DAST endpoint), an absolute filesystem
+ // path (a host finding), or escapes the root with `..`. GitHub resolves
+ // the uri against the repository root, so such a location renders no
+ // alert; uploading it would also publish an internal hostname.
+ //
+ // This is the reason a correlated DAST finding is dropped even though it
+ // carries research/18's `startLine: 1` placeholder region.
+ GitHubDropLocationNotRepoRelative GitHubDropReason = "location_not_repo_relative"
+
+ // GitHubDropNoStartLine: the primary location is a repository file but
+ // its region has no positive `startLine`. GitHub requires one.
+ GitHubDropNoStartLine GitHubDropReason = "no_start_line"
+
+ // GitHubDropNoPrimaryLocationLineHash: `partialFingerprints` does not
+ // carry PartialFingerprintPrimaryLocationLineHash, the only partial
+ // fingerprint GitHub reads. Uploading without it makes GitHub mint a
+ // duplicate alert on every scan, so the result is withheld instead.
+ //
+ // A projection in which EVERY result carries this reason is the visible
+ // form of the unresolved producer question described in this file's
+ // header. See CRITIQUE-01 MAJOR 3.
+ GitHubDropNoPrimaryLocationLineHash GitHubDropReason = "no_primary_location_line_hash"
+
+ // GitHubDropNoMessageText: `message.text` is empty or blank. GitHub
+ // requires it, and an alert with no text is not worth an upload slot.
+ GitHubDropNoMessageText GitHubDropReason = "no_message_text"
+
+ // GitHubDropExceedsFileSizeCap: this single result, alone in a file,
+ // still exceeds GitHubMaxGzipBytes. It cannot be sharded any further,
+ // so it is dropped rather than being permitted to break the cap.
+ GitHubDropExceedsFileSizeCap GitHubDropReason = "exceeds_file_size_cap"
+)
+
+// GitHubDropReasonValues returns every drop reason, in evaluation order.
+// The order is meaningful: a result that fails several checks is recorded
+// under the FIRST reason in this slice that it fails, so the reported reason
+// is deterministic and does not depend on struct traversal order.
+func GitHubDropReasonValues() []GitHubDropReason {
+ return []GitHubDropReason{
+ GitHubDropHalfNotReadable,
+ GitHubDropNoLocations,
+ GitHubDropNoPhysicalLocation,
+ GitHubDropLocationNotRepoRelative,
+ GitHubDropNoStartLine,
+ GitHubDropNoPrimaryLocationLineHash,
+ GitHubDropNoMessageText,
+ GitHubDropExceedsFileSizeCap,
+ }
+}
+
+// Valid reports whether r is a member of the closed drop vocabulary.
+func (r GitHubDropReason) Valid() bool { return inEnum(r, GitHubDropReasonValues()) }
+
+// Explain returns the GitHub rule that forces this drop, so a consumer asking
+// "why was this not uploaded" gets the external constraint and not just a
+// token.
+func (r GitHubDropReason) Explain() string {
+ switch r {
+ case GitHubDropHalfNotReadable:
+ return "this half has not passed Anvil's read gate (anvil/status is not \"" + string(HalfStatusSealed) +
+ "\", or the audit has expired); §6 G5 makes that gate hard, and provisional or withdrawn " +
+ "findings must not reach a third-party alert feed"
+ case GitHubDropNoLocations:
+ return "GitHub requires locations[] on every result; this result has none"
+ case GitHubDropNoPhysicalLocation:
+ return "GitHub renders alerts from locations[0].physicalLocation; this result's primary location has none"
+ case GitHubDropLocationNotRepoRelative:
+ return "GitHub resolves artifactLocation.uri against the repository root; an absolute URI or path (a DAST endpoint or a host file) resolves to no file and cannot render"
+ case GitHubDropNoStartLine:
+ return "GitHub requires region.startLine on the primary location"
+ case GitHubDropNoPrimaryLocationLineHash:
+ return "GitHub de-duplicates alerts using only partialFingerprints." +
+ PartialFingerprintPrimaryLocationLineHash +
+ "; without it every scan mints duplicate alerts, so the result is withheld"
+ case GitHubDropNoMessageText:
+ return "GitHub requires message.text on every result"
+ case GitHubDropExceedsFileSizeCap:
+ return fmt.Sprintf("this single result alone exceeds GitHub's %d-byte gzip file limit and cannot be sharded further",
+ GitHubMaxGzipBytes)
+ }
+ return "unknown drop reason"
+}
+
+// ---------------------------------------------------------------------------
+// Loss vocabulary — what was removed from results that DID reach GitHub
+// ---------------------------------------------------------------------------
+
+// GitHubStripReason names every field-level loss. A stripped field means the
+// alert exists on GitHub but carries less than the record does; the record
+// remains the audit.
+type GitHubStripReason string
+
+const (
+ // GitHubStripWebRequest / GitHubStripWebResponse: SARIF §3.27.14/15, the
+ // DAST evidence slots. Outside GitHub's supported-property list, and the
+ // response body is the highest-risk field in the record
+ // (plan/00-SPINE.md S7) — there is no reason to ship it to a third party
+ // that will not display it.
+ GitHubStripWebRequest GitHubStripReason = "web_request"
+ GitHubStripWebResponse GitHubStripReason = "web_response"
+
+ // GitHubStripRunTaxonomies / GitHubStripResultTaxa /
+ // GitHubStripRuleRelationships / GitHubStripDriverTaxa: the
+ // taxonomies-as-relationships mechanism (CWE). Unsupported by GitHub,
+ // and the three parts are stripped TOGETHER because result.taxa and
+ // rule.relationships reference run.taxonomies by index — keeping either
+ // without the array would emit a dangling index.
+ GitHubStripRunTaxonomies GitHubStripReason = "run_taxonomies"
+ GitHubStripResultTaxa GitHubStripReason = "result_taxa"
+ GitHubStripRuleRelationships GitHubStripReason = "rule_relationships"
+ GitHubStripDriverTaxa GitHubStripReason = "driver_taxa"
+
+ // GitHubStripResultProvenance: SARIF §3.48 regression history.
+ // Unsupported by GitHub; Anvil's own store is the regression record.
+ GitHubStripResultProvenance GitHubStripReason = "result_provenance"
+
+ // GitHubStripResultFixes: SARIF §3.27.30 proposed patches. Withheld
+ // deliberately: plan/00-SPINE.md S7 is "Never auto-merge. Propose only",
+ // and a fix rendered in a third-party UI as an accept-here button is not
+ // the proposal path Anvil owns.
+ GitHubStripResultFixes GitHubStripReason = "result_fixes"
+
+ // GitHubStripAuditProperties / GitHubStripRunProperties /
+ // GitHubStripResultProperties / GitHubStripLocationProperties: the
+ // `anvil/*` bags. Structurally impossible to emit — the projected types
+ // have no properties member — and counted here so the loss is visible
+ // rather than merely absent.
+ GitHubStripAuditProperties GitHubStripReason = "audit_anvil_properties"
+ GitHubStripRunProperties GitHubStripReason = "run_anvil_properties"
+ GitHubStripResultProperties GitHubStripReason = "result_anvil_properties"
+ GitHubStripLocationProperties GitHubStripReason = "location_anvil_properties"
+
+ // GitHubStripExternalPropertyFileReferences: SARIF §3.15. GitHub does
+ // not fetch external property files, so leaving the reference in place
+ // would advertise results that never arrive.
+ GitHubStripExternalPropertyFileReferences GitHubStripReason = "external_property_file_references"
+
+ // GitHubStripPartialFingerprintKey: a partialFingerprints key other than
+ // the two identity keys the projection keeps
+ // (PartialFingerprintPrimaryLocationLineHash, which GitHub reads, and
+ // PartialFingerprintAnvilFindingID, which is what lets a GitHub alert be
+ // traced back to a record finding).
+ GitHubStripPartialFingerprintKey GitHubStripReason = "partial_fingerprint_key"
+
+ // GitHubStripRelatedLocationNotRepoRelative: a relatedLocation that is
+ // not a repository file — typically the cross-half pointer at a DAST
+ // endpoint. Dropped for the same reason as a primary endpoint location.
+ GitHubStripRelatedLocationNotRepoRelative GitHubStripReason = "related_location_not_repo_relative"
+
+ // GitHubStripSecondaryLocationNotRepoRelative: a NON-primary entry in
+ // locations[] that is not a repository file. The primary location is
+ // never re-chosen — see the projectResult comment.
+ GitHubStripSecondaryLocationNotRepoRelative GitHubStripReason = "secondary_location_not_repo_relative"
+
+ // GitHubStripThreadFlowLocationNotRepoRelative: a code-flow step whose
+ // location is not a repository file.
+ GitHubStripThreadFlowLocationNotRepoRelative GitHubStripReason = "thread_flow_location_not_repo_relative"
+
+ // GitHubStripCodeFlowEmptied: a code flow every step of which was
+ // stripped. An empty threadFlow is not valid SARIF, so the flow goes.
+ GitHubStripCodeFlowEmptied GitHubStripReason = "code_flow_emptied"
+
+ // GitHubStripLocationsOverCap / GitHubStripThreadFlowLocationsOverCap:
+ // truncation forced by GitHubMaxLocationsPerResult and
+ // GitHubMaxThreadFlowLocationsPerResult. Counted in LOCATIONS removed,
+ // not results.
+ GitHubStripLocationsOverCap GitHubStripReason = "locations_over_cap"
+ GitHubStripThreadFlowLocationsOverCap GitHubStripReason = "thread_flow_locations_over_cap"
+
+ // GitHubStripUnreferencedRule: a rule descriptor that NO shard of its
+ // source run delivers, because no surviving result anywhere in that run
+ // references it. Dropping it is what keeps GitHubMaxRulesPerRun
+ // unreachable and the file small.
+ //
+ // Counted against the SHARD SET, once per source run — never per shard.
+ // See tallyRuleLoss.
+ GitHubStripUnreferencedRule GitHubStripReason = "unreferenced_rule"
+
+ // GitHubStripDuplicateRule: a second `reportingDescriptor` for a rule id
+ // already emitted. SARIF's rule array is a set keyed by id, so the
+ // duplicate cannot be carried; it was previously dropped with no tally at
+ // all, which made it the one silent loss in this file.
+ GitHubStripDuplicateRule GitHubStripReason = "duplicate_rule_descriptor"
+
+ // GitHubStripRunGUIDOnShard: `automationDetails.guid` cleared on the
+ // second and later shards of one run, because a guid identifies a single
+ // run object.
+ GitHubStripRunGUIDOnShard GitHubStripReason = "run_guid_cleared_on_shard"
+)
+
+// GitHubStripReasonValues returns every strip reason, in a fixed order that
+// Summary uses so its output is byte-stable.
+func GitHubStripReasonValues() []GitHubStripReason {
+ return []GitHubStripReason{
+ GitHubStripWebRequest,
+ GitHubStripWebResponse,
+ GitHubStripRunTaxonomies,
+ GitHubStripResultTaxa,
+ GitHubStripRuleRelationships,
+ GitHubStripDriverTaxa,
+ GitHubStripResultProvenance,
+ GitHubStripResultFixes,
+ GitHubStripAuditProperties,
+ GitHubStripRunProperties,
+ GitHubStripResultProperties,
+ GitHubStripLocationProperties,
+ GitHubStripExternalPropertyFileReferences,
+ GitHubStripPartialFingerprintKey,
+ GitHubStripRelatedLocationNotRepoRelative,
+ GitHubStripSecondaryLocationNotRepoRelative,
+ GitHubStripThreadFlowLocationNotRepoRelative,
+ GitHubStripCodeFlowEmptied,
+ GitHubStripLocationsOverCap,
+ GitHubStripThreadFlowLocationsOverCap,
+ GitHubStripUnreferencedRule,
+ GitHubStripDuplicateRule,
+ GitHubStripRunGUIDOnShard,
+ }
+}
+
+// Valid reports whether r is a member of the closed strip vocabulary.
+func (r GitHubStripReason) Valid() bool { return inEnum(r, GitHubStripReasonValues()) }
+
+// ---------------------------------------------------------------------------
+// The loss ledger
+// ---------------------------------------------------------------------------
+
+// GitHubDroppedResult identifies one result that never reached GitHub.
+//
+// It carries the record-local FindingID deliberately: without it, "we dropped
+// 412 results" is unactionable, and the anvil/* bag that would otherwise
+// answer "which ones" is exactly what the projection strips.
+type GitHubDroppedResult struct {
+ // SourceRunIndex and SourceResultIndex locate the result in the INPUT
+ // SARIFLog, so a consumer can go back to the record and read it.
+ SourceRunIndex int `json:"sourceRunIndex"`
+ SourceResultIndex int `json:"sourceResultIndex"`
+
+ FindingID string `json:"findingId"`
+ RuleID string `json:"ruleId,omitempty"`
+
+ // Half is the frozen anvil/half literal of the producing run.
+ Half Half `json:"half"`
+
+ // HalfStatus and AuditState are the two inputs to the read gate, recorded
+ // on every drop so a reader of the ledger can tell "this half never
+ // sealed" from "the audit expired holding a sealed half" without going
+ // back to the record. Both are frozen enum literals.
+ HalfStatus HalfStatus `json:"halfStatus,omitempty"`
+ AuditState State `json:"auditState,omitempty"`
+
+ Reason GitHubDropReason `json:"reason"`
+}
+
+// GitHubProjectionLoss is the complete, enumerable account of what the
+// projection discarded. One ledger describes one whole ProjectForGitHub call;
+// every returned file points at the SAME ledger object, so files[0].Loss is
+// the whole answer and the ledger is not sliced up per file.
+//
+// It is JSON-serialisable on purpose: the intended use is that the uploader
+// persists it next to the upload, so "GitHub shows 12 alerts and the audit
+// found 31" is answerable months later.
+type GitHubProjectionLoss struct {
+ // AuditID is anvil/auditId, copied from the record being projected.
+ AuditID string `json:"auditId"`
+
+ // SourceResultCount and ProjectedResultCount are the two numbers whose
+ // difference this ledger explains.
+ SourceResultCount int `json:"sourceResultCount"`
+ ProjectedResultCount int `json:"projectedResultCount"`
+
+ // DroppedResults is TOTAL — one entry per dropped result, never
+ // truncated or sampled. A ledger that summarised itself would reproduce
+ // the failure it exists to prevent.
+ DroppedResults []GitHubDroppedResult `json:"droppedResults"`
+
+ // DropCounts and StripCounts are the aggregate view. DropCounts is
+ // derivable from DroppedResults; it is materialised because the common
+ // question is a count, and recomputing it invites a second, disagreeing
+ // implementation of the aggregation.
+ DropCounts map[GitHubDropReason]int `json:"dropCounts"`
+ StripCounts map[GitHubStripReason]int `json:"stripCounts"`
+}
+
+func newGitHubProjectionLoss(auditID string) *GitHubProjectionLoss {
+ return &GitHubProjectionLoss{
+ AuditID: auditID,
+ DropCounts: map[GitHubDropReason]int{},
+ StripCounts: map[GitHubStripReason]int{},
+ }
+}
+
+func (l *GitHubProjectionLoss) dropResult(d GitHubDroppedResult) {
+ l.DroppedResults = append(l.DroppedResults, d)
+ l.DropCounts[d.Reason]++
+}
+
+func (l *GitHubProjectionLoss) strip(r GitHubStripReason, n int) {
+ if n <= 0 {
+ return
+ }
+ l.StripCounts[r] += n
+}
+
+// TotalDropped returns the number of results withheld from GitHub.
+func (l *GitHubProjectionLoss) TotalDropped() int { return len(l.DroppedResults) }
+
+// DroppedFor returns every result dropped for one reason, in input order.
+// This is the "ask what was dropped and why" entry point.
+func (l *GitHubProjectionLoss) DroppedFor(r GitHubDropReason) []GitHubDroppedResult {
+ var out []GitHubDroppedResult
+ for _, d := range l.DroppedResults {
+ if d.Reason == r {
+ out = append(out, d)
+ }
+ }
+ return out
+}
+
+// Summary renders the ledger as deterministic, loggable text: reasons in
+// GitHubDropReasonValues / GitHubStripReasonValues order, zero counts
+// omitted. Map iteration order never reaches the output.
+//
+// This is what satisfies "excluded with a LOGGED count rather than silently
+// dropped": the caller logs this string, and the counts in it are the ones a
+// human is owed when GitHub shows fewer alerts than the audit found.
+func (l *GitHubProjectionLoss) Summary() string {
+ var b strings.Builder
+ fmt.Fprintf(&b, "github projection loss for audit %q: %d source results, %d projected, %d dropped\n",
+ l.AuditID, l.SourceResultCount, l.ProjectedResultCount, l.TotalDropped())
+
+ b.WriteString(" dropped results (whole results GitHub never receives):\n")
+ anyDrop := false
+ for _, r := range GitHubDropReasonValues() {
+ if n := l.DropCounts[r]; n > 0 {
+ anyDrop = true
+ fmt.Fprintf(&b, " %-32s %6d — %s\n", r, n, r.Explain())
+ }
+ }
+ if !anyDrop {
+ b.WriteString(" (none)\n")
+ }
+
+ b.WriteString(" stripped fields (results GitHub receives, with less than the record holds):\n")
+ anyStrip := false
+ for _, r := range GitHubStripReasonValues() {
+ if n := l.StripCounts[r]; n > 0 {
+ anyStrip = true
+ fmt.Fprintf(&b, " %-32s %6d\n", r, n)
+ }
+ }
+ if !anyStrip {
+ b.WriteString(" (none)\n")
+ }
+ return b.String()
+}
+
+// GitHubLossOf returns the shared loss ledger carried by a projection's
+// files, or nil if there are none. Every file points at the same ledger, so
+// any file answers for all of them.
+func GitHubLossOf(files []GitHubSarifFile) *GitHubProjectionLoss {
+ if len(files) == 0 {
+ return nil
+ }
+ return files[0].Loss
+}
+
+// ---------------------------------------------------------------------------
+// The projected wire types
+//
+// These are a strict SUBSET of contract.go's SARIF types with every
+// `properties` member removed. They are separate types rather than reused
+// ones precisely so that no `anvil/*` bag can be emitted here by accident.
+// ---------------------------------------------------------------------------
+
+// GitHubSARIFLog is one uploadable SARIF 2.1.0 file. `$schema` and `version`
+// are pinned from the contract's constants: GitHub supports 2.1.0 only.
+type GitHubSARIFLog struct {
+ Schema string `json:"$schema"`
+ Version string `json:"version"`
+ Runs []GitHubRun `json:"runs"`
+}
+
+// GitHubRun is one projected run. It carries no `properties`, no
+// `taxonomies` and no `externalPropertyFileReferences`.
+type GitHubRun struct {
+ Tool Tool `json:"tool"`
+ AutomationDetails RunAutomationDetails `json:"automationDetails"`
+ OriginalURIBaseIDs map[string]ArtifactLocation `json:"originalUriBaseIds,omitempty"`
+ Results []GitHubResult `json:"results"`
+}
+
+// GitHubResult is one projected result: SARIF-native fields GitHub documents
+// support for, and nothing else.
+type GitHubResult struct {
+ RuleID string `json:"ruleId"`
+ RuleIndex *int `json:"ruleIndex,omitempty"`
+ Kind Kind `json:"kind,omitempty"`
+ Level Level `json:"level,omitempty"`
+ Rank *float64 `json:"rank,omitempty"`
+ GUID string `json:"guid,omitempty"`
+ CorrelationGUID string `json:"correlationGuid,omitempty"`
+
+ Message Message `json:"message"`
+ Locations []GitHubLocation `json:"locations"`
+ RelatedLocations []GitHubLocation `json:"relatedLocations,omitempty"`
+ CodeFlows []GitHubCodeFlow `json:"codeFlows,omitempty"`
+
+ // PartialFingerprints carries at most two keys: the one GitHub reads and
+ // the one that maps the alert back to a record finding.
+ PartialFingerprints map[string]string `json:"partialFingerprints"`
+
+ // srcResultIndex and srcFindingID are unexported and therefore never
+ // marshalled. They exist so that a result dropped LATE — during size
+ // bisection, long after the filter stage — still produces a ledger entry
+ // naming the record finding it came from. Without them the one drop
+ // reason that fires after projection would be the one drop reason a
+ // consumer could not trace, which is exactly the hole this projection is
+ // supposed to close.
+ srcResultIndex int
+ srcFindingID string
+}
+
+// GitHubLocation is SARIF §3.28 without the `anvil/*` location bag.
+type GitHubLocation struct {
+ ID *int `json:"id,omitempty"`
+ PhysicalLocation *PhysicalLocation `json:"physicalLocation,omitempty"`
+ LogicalLocations []LogicalLocation `json:"logicalLocations,omitempty"`
+ Message *Message `json:"message,omitempty"`
+}
+
+// GitHubCodeFlow is SARIF §3.36 with its steps projected.
+type GitHubCodeFlow struct {
+ Message *Message `json:"message,omitempty"`
+ ThreadFlows []GitHubThreadFlow `json:"threadFlows"`
+}
+
+// GitHubThreadFlow is SARIF §3.37.
+type GitHubThreadFlow struct {
+ Locations []GitHubThreadFlowLocation `json:"locations"`
+}
+
+// GitHubThreadFlowLocation is SARIF §3.38.
+type GitHubThreadFlowLocation struct {
+ Importance string `json:"importance,omitempty"`
+ Location GitHubLocation `json:"location"`
+}
+
+// ---------------------------------------------------------------------------
+// The output file
+// ---------------------------------------------------------------------------
+
+// GitHubSarifFile is one shard: a complete, uploadable SARIF file together
+// with the bytes that were actually measured against GitHub's caps.
+//
+// JSON and Gzip are both returned so that the caller uploads EXACTLY what was
+// measured. Returning only a byte count would leave the caller free to
+// re-compress at a different level and exceed a cap this package promised was
+// respected.
+type GitHubSarifFile struct {
+ // Name is a deterministic, filesystem-safe suggested filename. It is a
+ // suggestion: nothing here writes files.
+ Name string `json:"name"`
+
+ // Half is the frozen anvil/half literal of the source run this shard
+ // came from, so an uploader can label the analysis correctly.
+ Half Half `json:"half"`
+
+ // SourceRunIndex is the index of the source run in the input record.
+ // ShardIndex is 1-based within that run; ShardCount is the total number
+ // of shards that run produced.
+ SourceRunIndex int `json:"sourceRunIndex"`
+ ShardIndex int `json:"shardIndex"`
+ ShardCount int `json:"shardCount"`
+
+ // Log is the projected SARIF. JSON is its compact encoding and Gzip is
+ // the gzip of exactly those bytes.
+ Log GitHubSARIFLog `json:"-"`
+ JSON []byte `json:"-"`
+ Gzip []byte `json:"-"`
+
+ // ResultCount is len(Log.Runs[0].Results); GzipBytes is len(Gzip).
+ // Both are surfaced so a caller can report cap headroom without
+ // re-deriving it.
+ ResultCount int `json:"resultCount"`
+ GzipBytes int `json:"gzipBytes"`
+
+ // Loss is the WHOLE-PROJECTION ledger, shared by every file of one
+ // ProjectForGitHub call. It is not this file's private loss.
+ Loss *GitHubProjectionLoss `json:"-"`
+}
+
+// WithinCaps re-checks a built file against every documented GitHub cap.
+//
+// It is not a substitute for building the file correctly; it is the
+// independent check that the build was correct, and ProjectForGitHub runs it
+// on every file before returning. A cap violation is returned as an error
+// rather than logged, because an over-cap file is not a degraded upload — it
+// is a rejected one.
+func (f *GitHubSarifFile) WithinCaps() error {
+ if n := len(f.Log.Runs); n > GitHubMaxRunsPerFile {
+ return fmt.Errorf("github projection %q: %d runs exceeds GitHub's %d runs per file",
+ f.Name, n, GitHubMaxRunsPerFile)
+ }
+ if n := len(f.Log.Runs); n > GitHubRunsPerProjectedFile {
+ return fmt.Errorf("github projection %q: %d runs exceeds Anvil's own %d-run-per-file shard policy",
+ f.Name, n, GitHubRunsPerProjectedFile)
+ }
+ for i := range f.Log.Runs {
+ run := &f.Log.Runs[i]
+ if n := len(run.Results); n > GitHubMaxResultsPerRun {
+ return fmt.Errorf("github projection %q: run %d has %d results, exceeding GitHub's %d per run",
+ f.Name, i, n, GitHubMaxResultsPerRun)
+ }
+ if n := len(run.Tool.Driver.Rules); n > GitHubMaxRulesPerRun {
+ return fmt.Errorf("github projection %q: run %d has %d rules, exceeding GitHub's %d per run",
+ f.Name, i, n, GitHubMaxRulesPerRun)
+ }
+ for j := range run.Results {
+ res := &run.Results[j]
+ // locations + relatedLocations, together: the independent check
+ // has to ask the same question the builder answered, or it
+ // certifies a guarantee narrower than the one the file promises.
+ // See capLocationPair.
+ if n := len(res.Locations) + len(res.RelatedLocations); n > GitHubMaxLocationsPerResult {
+ return fmt.Errorf("github projection %q: run %d result %d has %d locations "+
+ "(%d locations + %d relatedLocations), exceeding GitHub's %d per result",
+ f.Name, i, j, n, len(res.Locations), len(res.RelatedLocations), GitHubMaxLocationsPerResult)
+ }
+ if n := countThreadFlowLocations(res.CodeFlows); n > GitHubMaxThreadFlowLocationsPerResult {
+ return fmt.Errorf("github projection %q: run %d result %d has %d thread-flow locations, exceeding GitHub's %d per result",
+ f.Name, i, j, n, GitHubMaxThreadFlowLocationsPerResult)
+ }
+ }
+ }
+ if n := len(f.Gzip); n > GitHubMaxGzipBytes {
+ return fmt.Errorf("github projection %q: %d gzip bytes exceeds GitHub's %d-byte file limit",
+ f.Name, n, GitHubMaxGzipBytes)
+ }
+ return nil
+}
+
+func countThreadFlowLocations(flows []GitHubCodeFlow) int {
+ n := 0
+ for _, cf := range flows {
+ for _, tf := range cf.ThreadFlows {
+ n += len(tf.Locations)
+ }
+ }
+ return n
+}
+
+// ---------------------------------------------------------------------------
+// ProjectForGitHub
+// ---------------------------------------------------------------------------
+
+// ProjectForGitHub reduces an Anvil audit record to one or more SARIF files
+// GitHub code scanning can accept.
+//
+// Every returned file satisfies WithinCaps. Every result in every returned
+// file has a repository-relative primary code location with a start line and
+// a populated PartialFingerprintPrimaryLocationLineHash. Everything the
+// projection refused to carry is enumerated in the shared
+// GitHubProjectionLoss ledger reachable from any returned file's Loss field
+// (or via GitHubLossOf).
+//
+// One file is emitted per source run even when that run contributed no
+// results. That is deliberate on two counts: a zero-result run is a
+// meaningful SARIF upload (it tells GitHub the previous scan's alerts for
+// that analysis are resolved), and it keeps the loss ledger reachable in the
+// exact case where the loss is total — a DAST-only half, where every result
+// is dropped and a "no files" return would take the explanation with it.
+// Callers that do not want to upload an empty analysis skip files whose
+// ResultCount is zero.
+//
+// It returns an error only when nothing legal can be emitted at all: a
+// result-free run whose tool metadata alone exceeds the gzip cap.
+func ProjectForGitHub(l *SARIFLog) ([]GitHubSarifFile, error) {
+ if l == nil {
+ return nil, fmt.Errorf("github projection: nil record")
+ }
+ // Masking is a PRECONDITION, exactly as it is on readpath.go's Reader.
+ // This projection strips every surface MaskRecord covers, so no leak could
+ // be demonstrated through it today (CRITIQUE-03 B1 records that as
+ // unverified harm) — which is the point: the safety currently rests on the
+ // strip list staying exhaustive, and the day a carried field is added,
+ // this post-condition is what catches it instead of a third party.
+ if err := AssertMasked(l); err != nil {
+ return nil, fmt.Errorf("github projection: refusing to project audit %q: %w "+
+ "(R.8's masker runs before any sink, and a third-party alert feed is a sink)",
+ l.Properties.AuditID, err)
+ }
+ p := &ghProjector{
+ loss: newGitHubProjectionLoss(l.Properties.AuditID),
+ auditSlug: fileSlug(l.Properties.AuditID),
+ shardSeq: map[int]int{},
+ }
+
+ // The audit-level anvil/* bag is never carried. Count it once: the bag
+ // is always present on a real record, and "the whole envelope was
+ // dropped" is part of the honest answer to "what did GitHub not get".
+ p.loss.strip(GitHubStripAuditProperties, 1)
+
+ var files []GitHubSarifFile
+ for i := range l.Runs {
+ src := &l.Runs[i]
+ p.loss.strip(GitHubStripRunProperties, 1)
+ if len(src.Taxonomies) > 0 {
+ p.loss.strip(GitHubStripRunTaxonomies, len(src.Taxonomies))
+ }
+ if len(src.Tool.Driver.Taxa) > 0 {
+ p.loss.strip(GitHubStripDriverTaxa, len(src.Tool.Driver.Taxa))
+ }
+ if src.ExternalPropertyFileReferences != nil {
+ p.loss.strip(GitHubStripExternalPropertyFileReferences, 1)
+ }
+
+ kept := p.projectResults(l, src, i)
+ runFiles, err := p.shardRun(src, i, kept)
+ if err != nil {
+ return nil, err
+ }
+ // The rule ledger is reconciled against the SHARD SET, once per source
+ // run, and never per shard. See tallyRuleLoss.
+ p.tallyRuleLoss(src, runFiles)
+ files = append(files, runFiles...)
+ }
+
+ // ShardCount is only knowable once a run is fully sharded.
+ counts := map[int]int{}
+ for _, f := range files {
+ counts[f.SourceRunIndex]++
+ }
+ for i := range files {
+ files[i].ShardCount = counts[files[i].SourceRunIndex]
+ files[i].Loss = p.loss
+ p.loss.ProjectedResultCount += files[i].ResultCount
+ if err := files[i].WithinCaps(); err != nil {
+ return nil, err
+ }
+ }
+ return files, nil
+}
+
+// ghProjector holds the mutable state of one ProjectForGitHub call. It exists
+// so the loss ledger is threaded through every decision by construction: a
+// drop that forgot to touch the ledger would have to go out of its way.
+type ghProjector struct {
+ loss *GitHubProjectionLoss
+ auditSlug string
+ // shardSeq maps a source run index to the number of shards emitted for
+ // it so far, so shard ids are assigned in emission order.
+ shardSeq map[int]int
+}
+
+// projectResults filters and projects one run's results, recording a reason
+// for every result it refuses.
+//
+// THE READ GATE RUNS FIRST, AND IT RUNS PER RUN. A run whose half has not
+// passed sealing.go's HalfReadGate contributes NO results, and every one of
+// them is ledgered under GitHubDropHalfNotReadable — the loss must be
+// countable here for the same reason every other refusal is, and CRITIQUE-03
+// B1's probe found the ledger recording ZERO drops in every unsealed case, so
+// the loss was not merely permitted but invisible.
+//
+// The gate is called, never re-derived. `src.Properties.Status` is right there
+// and `l.Properties.State` is one dereference further away; three of the four
+// bypasses sealing.go's header lists were made by reaching for the near one.
+func (p *ghProjector) projectResults(l *SARIFLog, src *Run, srcRunIdx int) []GitHubResult {
+ half := src.Properties.Half
+ seal := halfSealOfRun(l, src)
+ gateErr := HalfReadGate(l.Properties.AuditID, seal)
+
+ kept := make([]GitHubResult, 0, len(src.Results))
+ for j := range src.Results {
+ p.loss.SourceResultCount++
+ res := &src.Results[j]
+ entry := GitHubDroppedResult{
+ SourceRunIndex: srcRunIdx,
+ SourceResultIndex: j,
+ FindingID: res.Properties.FindingID,
+ RuleID: res.RuleID,
+ Half: half,
+ HalfStatus: seal.Status,
+ AuditState: seal.AuditState,
+ }
+ if gateErr != nil {
+ entry.Reason = GitHubDropHalfNotReadable
+ p.loss.dropResult(entry)
+ continue
+ }
+ gr, reason, ok := p.projectResult(res)
+ if !ok {
+ entry.Reason = reason
+ p.loss.dropResult(entry)
+ continue
+ }
+ gr.srcResultIndex = j
+ gr.srcFindingID = res.Properties.FindingID
+ kept = append(kept, gr)
+ }
+ return kept
+}
+
+// projectResult applies the drop rules in GitHubDropReasonValues order and,
+// for a surviving result, strips every unsupported field.
+//
+// THE PRIMARY LOCATION IS NEVER RE-CHOSEN. If locations[0] is not a
+// repository code location the result is dropped, even when a later entry in
+// locations[] is one. The reason is identity: primaryLocationLineHash was
+// computed against the record's PRIMARY location, so promoting locations[1]
+// would upload a fingerprint that describes a different line than the alert
+// it is attached to, and GitHub's de-duplication would key on the mismatch
+// forever.
+func (p *ghProjector) projectResult(r *Result) (GitHubResult, GitHubDropReason, bool) {
+ if len(r.Locations) == 0 {
+ return GitHubResult{}, GitHubDropNoLocations, false
+ }
+ primary := r.Locations[0]
+ if primary.PhysicalLocation == nil {
+ return GitHubResult{}, GitHubDropNoPhysicalLocation, false
+ }
+ if !isRepoRelativeURI(primary.PhysicalLocation.ArtifactLocation.URI) {
+ return GitHubResult{}, GitHubDropLocationNotRepoRelative, false
+ }
+ if primary.PhysicalLocation.Region == nil || primary.PhysicalLocation.Region.StartLine <= 0 {
+ return GitHubResult{}, GitHubDropNoStartLine, false
+ }
+ if r.PartialFingerprints[PartialFingerprintPrimaryLocationLineHash] == "" {
+ return GitHubResult{}, GitHubDropNoPrimaryLocationLineHash, false
+ }
+ if strings.TrimSpace(r.Message.Text) == "" {
+ return GitHubResult{}, GitHubDropNoMessageText, false
+ }
+
+ // From here the result is kept; everything below is field-level loss.
+ p.loss.strip(GitHubStripResultProperties, 1)
+ if r.WebRequest != nil {
+ p.loss.strip(GitHubStripWebRequest, 1)
+ }
+ if r.WebResponse != nil {
+ p.loss.strip(GitHubStripWebResponse, 1)
+ }
+ if len(r.Taxa) > 0 {
+ p.loss.strip(GitHubStripResultTaxa, len(r.Taxa))
+ }
+ if r.Provenance != nil {
+ p.loss.strip(GitHubStripResultProvenance, 1)
+ }
+ if len(r.Fixes) > 0 {
+ p.loss.strip(GitHubStripResultFixes, len(r.Fixes))
+ }
+
+ out := GitHubResult{
+ RuleID: r.RuleID,
+ Kind: r.Kind,
+ Level: r.Level,
+ Rank: r.Rank,
+ GUID: r.GUID,
+ CorrelationGUID: r.CorrelationGUID,
+ Message: r.Message,
+ PartialFingerprints: p.projectPartialFingerprints(r.PartialFingerprints),
+ }
+
+ // locations[0] is kept as-is; later entries survive only if they are
+ // repository files too.
+ out.Locations = append(out.Locations, p.projectLocation(primary))
+ for _, loc := range r.Locations[1:] {
+ if !isRepoCodeLocation(loc) {
+ p.loss.strip(GitHubStripSecondaryLocationNotRepoRelative, 1)
+ continue
+ }
+ out.Locations = append(out.Locations, p.projectLocation(loc))
+ }
+
+ for _, loc := range r.RelatedLocations {
+ if !isRepoCodeLocation(loc) {
+ p.loss.strip(GitHubStripRelatedLocationNotRepoRelative, 1)
+ continue
+ }
+ out.RelatedLocations = append(out.RelatedLocations, p.projectLocation(loc))
+ }
+
+ // THE CAP IS ON THE PAIR. See capLocationPair.
+ p.capLocationPair(&out)
+
+ out.CodeFlows = p.projectCodeFlows(r.CodeFlows)
+ return out, "", true
+}
+
+// capLocationPair enforces GitHubMaxLocationsPerResult across `locations` AND
+// `relatedLocations` together, filling from `locations` first.
+//
+// WHY THE PAIR AND NOT EACH ARRAY (CRITIQUE-03 M4). research/18 records the
+// limit as "1,000 locations per result (100 displayed)", sourced to [S2], and
+// nothing in the tree says whether GitHub counts `relatedLocations` toward
+// that figure — the critic could not source it and neither can this file. The
+// projection previously truncated `locations` at 1,000 and appended
+// `relatedLocations` without limit, so a fan-out finding shipped 4,000
+// locations under a cap the file's own header promises "no returned file
+// exceeds ... on any input".
+//
+// Of the two readings, only one is safe under both: capping the pair is
+// correct if `relatedLocations` DO count, and merely conservative if they do
+// not. An asymmetry that is wrong under one reading is not. If [S2] is ever
+// checked and says related locations are exempt, the fix is to relax this
+// function with the source quoted next to GitHubMaxLocationsPerResult, with
+// the same sourcing rigour every other number in that block has — not to
+// re-introduce an unexplained asymmetry.
+//
+// Both overflows are counted under GitHubStripLocationsOverCap: the strip
+// vocabulary answers "what kind of loss", and a truncated location is one kind
+// whichever array it sat in.
+func (p *ghProjector) capLocationPair(out *GitHubResult) {
+ if over := len(out.Locations) - GitHubMaxLocationsPerResult; over > 0 {
+ out.Locations = out.Locations[:GitHubMaxLocationsPerResult]
+ p.loss.strip(GitHubStripLocationsOverCap, over)
+ }
+ room := GitHubMaxLocationsPerResult - len(out.Locations)
+ if over := len(out.RelatedLocations) - room; over > 0 {
+ out.RelatedLocations = out.RelatedLocations[:room]
+ p.loss.strip(GitHubStripLocationsOverCap, over)
+ }
+ if len(out.RelatedLocations) == 0 {
+ out.RelatedLocations = nil
+ }
+}
+
+// projectPartialFingerprints keeps the key GitHub reads and the key that maps
+// an alert back to a record finding, and counts the rest.
+func (p *ghProjector) projectPartialFingerprints(in map[string]string) map[string]string {
+ out := make(map[string]string, 2)
+ for k, v := range in {
+ switch k {
+ case PartialFingerprintPrimaryLocationLineHash, PartialFingerprintAnvilFindingID:
+ if v != "" {
+ out[k] = v
+ }
+ default:
+ p.loss.strip(GitHubStripPartialFingerprintKey, 1)
+ }
+ }
+ return out
+}
+
+func (p *ghProjector) projectLocation(loc Location) GitHubLocation {
+ if len(loc.Properties) > 0 {
+ p.loss.strip(GitHubStripLocationProperties, 1)
+ }
+ out := GitHubLocation{
+ ID: loc.ID,
+ LogicalLocations: loc.LogicalLocations,
+ Message: loc.Message,
+ }
+ if loc.PhysicalLocation != nil {
+ pl := *loc.PhysicalLocation
+ out.PhysicalLocation = &pl
+ }
+ return out
+}
+
+// projectCodeFlows keeps GitHub-renderable taint paths, dropping steps that
+// are not repository files and flows that end up empty. The per-result
+// thread-flow cap is applied across the whole result, which is how GitHub
+// counts it.
+func (p *ghProjector) projectCodeFlows(flows []CodeFlow) []GitHubCodeFlow {
+ var out []GitHubCodeFlow
+ budget := GitHubMaxThreadFlowLocationsPerResult
+ for _, cf := range flows {
+ var gcf GitHubCodeFlow
+ gcf.Message = cf.Message
+ for _, tf := range cf.ThreadFlows {
+ var gtf GitHubThreadFlow
+ for _, tfl := range tf.Locations {
+ if !isRepoCodeLocation(tfl.Location) {
+ p.loss.strip(GitHubStripThreadFlowLocationNotRepoRelative, 1)
+ continue
+ }
+ if budget <= 0 {
+ p.loss.strip(GitHubStripThreadFlowLocationsOverCap, 1)
+ continue
+ }
+ budget--
+ gtf.Locations = append(gtf.Locations, GitHubThreadFlowLocation{
+ Importance: tfl.Importance,
+ Location: p.projectLocation(tfl.Location),
+ })
+ }
+ if len(gtf.Locations) == 0 {
+ continue
+ }
+ gcf.ThreadFlows = append(gcf.ThreadFlows, gtf)
+ }
+ if len(gcf.ThreadFlows) == 0 {
+ p.loss.strip(GitHubStripCodeFlowEmptied, 1)
+ continue
+ }
+ out = append(out, gcf)
+ }
+ return out
+}
+
+// ---------------------------------------------------------------------------
+// Location predicates
+// ---------------------------------------------------------------------------
+
+// isRepoCodeLocation reports whether a location is something GitHub can
+// render as a code-scanning alert: a repository-relative artifact with a
+// positive start line.
+func isRepoCodeLocation(loc Location) bool {
+ pl := loc.PhysicalLocation
+ if pl == nil || pl.Region == nil || pl.Region.StartLine <= 0 {
+ return false
+ }
+ return isRepoRelativeURI(pl.ArtifactLocation.URI)
+}
+
+// isRepoRelativeURI reports whether uri can resolve to a file inside the
+// repository GitHub is uploading against.
+//
+// It rejects, in order: the empty string; absolute POSIX and Windows-style
+// paths; anything carrying a URI scheme (`https:`, `file:`, and also `C:` —
+// a Windows drive letter is syntactically a scheme, and rejecting it is the
+// correct outcome either way); and any path with a `..` segment, which cannot
+// be guaranteed to stay inside the repository root.
+//
+// Note what it deliberately does NOT consult: `location.properties`
+// ["anvil/locationKind"]. That key has no frozen enum in R.1's contract, so
+// depending on its literals here would create a vocabulary this file
+// half-owns — the exact pattern §6 closed ten defects over. The URI shape is
+// self-contained and needs no shared vocabulary.
+func isRepoRelativeURI(uri string) bool {
+ if uri == "" {
+ return false
+ }
+ if strings.HasPrefix(uri, "/") || strings.HasPrefix(uri, `\`) {
+ return false
+ }
+ if hasURIScheme(uri) {
+ return false
+ }
+ for _, seg := range strings.Split(strings.ReplaceAll(uri, `\`, "/"), "/") {
+ if seg == ".." {
+ return false
+ }
+ }
+ return true
+}
+
+// hasURIScheme reports whether s begins with an RFC 3986 scheme followed by
+// ':'.
+func hasURIScheme(s string) bool {
+ for i := 0; i < len(s); i++ {
+ c := s[i]
+ if c == ':' {
+ return i > 0
+ }
+ alpha := (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
+ if i == 0 {
+ if !alpha {
+ return false
+ }
+ continue
+ }
+ digit := c >= '0' && c <= '9'
+ if !alpha && !digit && c != '+' && c != '-' && c != '.' {
+ return false
+ }
+ }
+ return false
+}
+
+// ---------------------------------------------------------------------------
+// Sharding
+// ---------------------------------------------------------------------------
+
+// shardRun splits one run's projected results into files that each satisfy
+// every cap, and guarantees at least one file per source run.
+func (p *ghProjector) shardRun(src *Run, srcRunIdx int, results []GitHubResult) ([]GitHubSarifFile, error) {
+ var out []GitHubSarifFile
+ if err := p.shard(src, srcRunIdx, results, &out); err != nil {
+ return nil, err
+ }
+ if len(out) == 0 {
+ // Every result was dropped for size. Emit the empty run anyway so
+ // the loss is still attached to something the caller receives.
+ f, fits, tally, err := p.tryBuild(src, srcRunIdx, nil)
+ if err != nil {
+ return nil, err
+ }
+ if !fits {
+ return nil, fmt.Errorf(
+ "github projection: run %d has no results and still exceeds GitHub's %d-byte gzip limit (%d bytes); "+
+ "the tool metadata alone cannot be uploaded", srcRunIdx, GitHubMaxGzipBytes, len(f.Gzip))
+ }
+ p.commit(tally)
+ p.shardSeq[srcRunIdx] = f.ShardIndex
+ out = append(out, f)
+ }
+ return out, nil
+}
+
+// shard emits files for results, splitting until every part fits.
+//
+// The count cap is applied first and greedily, so a run of 25,001 results
+// yields a full 25,000-result shard and a 1-result shard rather than two
+// half-full ones. The size cap is then applied by bisection, which
+// terminates: each step halves the slice, and a single result that still does
+// not fit is dropped with GitHubDropExceedsFileSizeCap.
+//
+// # The bisection re-marshals and re-gzips discarded candidates, deliberately
+//
+// CRITIQUE-03 m4: a run needing k size splits does O(n log n) bytes of
+// gzip.BestCompression work near the 10 MB boundary. That is real, and it is
+// KEPT, because the two obvious remedies both weaken the guarantee this file
+// exists to make:
+//
+// - estimating the split point from the uncompressed size makes the split
+// depend on a compression ratio nobody measured, so a mis-estimate turns
+// into an extra round of bisection anyway and buys nothing on the
+// pathological inputs;
+// - probing at a cheap gzip level and re-compressing the keeper at
+// BestCompression means the bytes that decided the split are not the bytes
+// that ship. GitHubSarifFile returns Gzip precisely so the caller uploads
+// exactly what was measured; measuring one artefact and shipping another
+// is the shape of cap guarantee that is false in the field and true in the
+// test.
+//
+// The count cap runs first and bounds every candidate at 25,000 results, so
+// the bisection only engages when 25,000 results still exceed 10 MB gzipped —
+// which needs roughly 400 bytes of incompressible payload per result. If that
+// ever becomes a routine shape rather than a pathological one, the fix is to
+// shard on a running uncompressed-size estimate BEFORE the first build, not to
+// make the measured bytes and the shipped bytes two different things.
+func (p *ghProjector) shard(src *Run, srcRunIdx int, results []GitHubResult, out *[]GitHubSarifFile) error {
+ if len(results) > GitHubMaxResultsPerRun {
+ if err := p.shard(src, srcRunIdx, results[:GitHubMaxResultsPerRun], out); err != nil {
+ return err
+ }
+ return p.shard(src, srcRunIdx, results[GitHubMaxResultsPerRun:], out)
+ }
+
+ f, fits, tally, err := p.tryBuild(src, srcRunIdx, results)
+ if err != nil {
+ return err
+ }
+ if fits {
+ p.commit(tally)
+ p.shardSeq[srcRunIdx] = f.ShardIndex
+ *out = append(*out, f)
+ return nil
+ }
+ if len(results) == 0 {
+ return fmt.Errorf(
+ "github projection: run %d has no results and still exceeds GitHub's %d-byte gzip limit (%d bytes)",
+ srcRunIdx, GitHubMaxGzipBytes, len(f.Gzip))
+ }
+ if len(results) == 1 {
+ p.loss.dropResult(GitHubDroppedResult{
+ SourceRunIndex: srcRunIdx,
+ SourceResultIndex: results[0].srcResultIndex,
+ FindingID: results[0].srcFindingID,
+ RuleID: results[0].RuleID,
+ Half: src.Properties.Half,
+ Reason: GitHubDropExceedsFileSizeCap,
+ })
+ return nil
+ }
+ mid := len(results) / 2
+ if err := p.shard(src, srcRunIdx, results[:mid], out); err != nil {
+ return err
+ }
+ return p.shard(src, srcRunIdx, results[mid:], out)
+}
+
+// ghStripTally accumulates the field-level loss of ONE candidate shard.
+//
+// It exists because tryBuild is speculative: a candidate that does not fit is
+// discarded and re-split, and a candidate's strips must not reach the ledger
+// unless that candidate is kept. Counting them directly would inflate every
+// strip count by the number of failed bisection attempts — a ledger that
+// over-reports loss is as untrustworthy as one that under-reports it.
+type ghStripTally map[GitHubStripReason]int
+
+func (t ghStripTally) add(r GitHubStripReason, n int) {
+ if n > 0 {
+ t[r] += n
+ }
+}
+
+// commit folds a KEPT candidate's strips into the projection ledger.
+func (p *ghProjector) commit(t ghStripTally) {
+ for r, n := range t {
+ p.loss.strip(r, n)
+ }
+}
+
+// tryBuild assembles, marshals and compresses one candidate file. It reports
+// whether the file is within the gzip cap; the caller either keeps it (and
+// commits the returned tally) or splits and discards it. The compressed bytes
+// are produced here and kept, so a file is never compressed twice and what
+// was measured is what is returned.
+func (p *ghProjector) tryBuild(src *Run, srcRunIdx int, results []GitHubResult) (GitHubSarifFile, bool, ghStripTally, error) {
+ shardIdx := p.shardSeq[srcRunIdx] + 1
+ tally := ghStripTally{}
+
+ rules, ruleIndex := projectRules(src.Tool.Driver.Rules, results)
+ // Re-point each result at its rule's index in THIS shard's rule array.
+ // The source index is meaningless after filtering, and a stale ruleIndex
+ // is worse than none: it names a different rule.
+ shardResults := make([]GitHubResult, len(results))
+ copy(shardResults, results)
+ for i := range shardResults {
+ if idx, ok := ruleIndex[shardResults[i].RuleID]; ok {
+ n := idx
+ shardResults[i].RuleIndex = &n
+ } else {
+ shardResults[i].RuleIndex = nil
+ }
+ }
+
+ driver := src.Tool.Driver
+ driver.Rules = rules
+ driver.Taxa = nil
+
+ auto := src.AutomationDetails
+ if shardIdx > 1 {
+ auto.ID = shardAutomationID(auto.ID, shardIdx)
+ if auto.GUID != "" {
+ auto.GUID = ""
+ tally.add(GitHubStripRunGUIDOnShard, 1)
+ }
+ }
+
+ run := GitHubRun{
+ Tool: Tool{Driver: driver},
+ AutomationDetails: auto,
+ OriginalURIBaseIDs: src.OriginalURIBaseIDs,
+ Results: shardResults,
+ }
+ if run.Results == nil {
+ run.Results = []GitHubResult{}
+ }
+
+ log := GitHubSARIFLog{
+ Schema: SARIFSchemaURI,
+ Version: SARIFVersion,
+ Runs: []GitHubRun{run},
+ }
+ raw, err := json.Marshal(&log)
+ if err != nil {
+ return GitHubSarifFile{}, false, nil, fmt.Errorf("github projection: marshal run %d shard %d: %w", srcRunIdx, shardIdx, err)
+ }
+ gz, err := gzipBytes(raw)
+ if err != nil {
+ return GitHubSarifFile{}, false, nil, fmt.Errorf("github projection: gzip run %d shard %d: %w", srcRunIdx, shardIdx, err)
+ }
+
+ half := src.Properties.Half
+ f := GitHubSarifFile{
+ Name: fmt.Sprintf("anvil-%s-%s-%03d.sarif", p.auditSlug, fileSlug(string(half)), shardIdx),
+ Half: half,
+ SourceRunIndex: srcRunIdx,
+ ShardIndex: shardIdx,
+ Log: log,
+ JSON: raw,
+ Gzip: gz,
+ ResultCount: len(shardResults),
+ GzipBytes: len(gz),
+ }
+ return f, len(gz) <= GitHubMaxGzipBytes, tally, nil
+}
+
+// projectRules returns the rule descriptors this shard's results actually
+// reference, in source order, with taxonomy relationships stripped, plus the
+// ruleId -> new index map.
+//
+// Emitting only referenced rules is what makes GitHubMaxRulesPerRun
+// unreachable: distinct referenced rules can never exceed the shard's result
+// count, which is already capped at GitHubMaxResultsPerRun.
+//
+// IT TALLIES NOTHING. Rule-level loss is a property of the SHARD SET, not of
+// one shard, and this function runs once per candidate shard — including the
+// candidates bisection throws away. tallyRuleLoss is where the counting
+// happens; see it for the whole argument.
+func projectRules(src []ReportingDescriptor, results []GitHubResult) ([]ReportingDescriptor, map[string]int) {
+ referenced := make(map[string]bool, len(results))
+ for i := range results {
+ referenced[results[i].RuleID] = true
+ }
+ out := make([]ReportingDescriptor, 0, len(referenced))
+ index := make(map[string]int, len(referenced))
+ for _, rule := range src {
+ if !referenced[rule.ID] {
+ continue
+ }
+ if _, dup := index[rule.ID]; dup {
+ continue
+ }
+ if len(rule.Relationships) > 0 {
+ rule.Relationships = nil
+ }
+ index[rule.ID] = len(out)
+ out = append(out, rule)
+ }
+ if len(out) == 0 {
+ return nil, index
+ }
+ return out, index
+}
+
+// tallyRuleLoss reconciles one SOURCE RUN's rule descriptors against the rules
+// its shards actually deliver, and is the only place rule-level loss reaches
+// the ledger.
+//
+// # Why this is not per shard (CRITIQUE-03 M2)
+//
+// projectRules used to count GitHubStripUnreferencedRule for every rule not
+// referenced by the shard being built. A rule referenced only by shard 2 was
+// therefore counted as "stripped" while shard 1 was assembled, even though it
+// IS delivered to GitHub in shard 2. The probe: 25,010 results across two
+// shards, four rules in the source run, two of them genuinely lost — and a
+// ledger reading 6. Six exceeds the four rules that exist, which is by itself
+// proof the number counted nothing real, and it scaled with shard count, so it
+// was worst on exactly the large audits the ledger exists for.
+//
+// That contradicts ghStripTally's own stated principle — "a ledger that
+// over-reports loss is as untrustworthy as one that under-reports it" — and
+// the ledger is the whole mechanism by which R.14 answers research/18 Risk #6.
+// A number a reader learns to discount is worse than no number.
+//
+// GitHubStripRuleRelationships moves here for the same reason: one source
+// descriptor's relationships are one loss however many shards carry the rule.
+//
+// GitHubStripDuplicateRule closes the file's one remaining silent drop: a
+// second descriptor for an id already emitted was skipped with no tally at
+// all.
+//
+// A per-shard rule COUNT, if anyone ever wants one, belongs on
+// GitHubSarifFile — not in the shared ledger, which describes the projection.
+func (p *ghProjector) tallyRuleLoss(src *Run, files []GitHubSarifFile) {
+ delivered := map[string]bool{}
+ for i := range files {
+ for j := range files[i].Log.Runs {
+ for _, rule := range files[i].Log.Runs[j].Tool.Driver.Rules {
+ delivered[rule.ID] = true
+ }
+ }
+ }
+
+ seen := map[string]bool{}
+ for _, rule := range src.Tool.Driver.Rules {
+ if seen[rule.ID] {
+ p.loss.strip(GitHubStripDuplicateRule, 1)
+ continue
+ }
+ seen[rule.ID] = true
+ if !delivered[rule.ID] {
+ p.loss.strip(GitHubStripUnreferencedRule, 1)
+ continue
+ }
+ if n := len(rule.Relationships); n > 0 {
+ p.loss.strip(GitHubStripRuleRelationships, n)
+ }
+ }
+}
+
+// shardAutomationID derives a distinct analysis id for shards after the
+// first.
+//
+// GitHub keys an analysis on runAutomationDetails.id, so uploading two shards
+// under one id makes the second REPLACE the first. GitHub's convention is
+// that the id is a category ending in '/', optionally followed by a run id,
+// so the suffix is appended as a further category segment and the trailing
+// '/' is preserved.
+func shardAutomationID(id string, shardIdx int) string {
+ suffix := fmt.Sprintf("shard-%03d/", shardIdx)
+ if id == "" {
+ return suffix
+ }
+ if strings.HasSuffix(id, "/") {
+ return id + suffix
+ }
+ return id + "/" + suffix
+}
+
+// fileSlug reduces a string to characters that are safe in a filename on
+// every platform Anvil builds for, so a suggested name never depends on what
+// an audit id happens to contain.
+func fileSlug(s string) string {
+ if s == "" {
+ return "unknown"
+ }
+ var b strings.Builder
+ for i := 0; i < len(s); i++ {
+ c := s[i]
+ switch {
+ case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == '-', c == '_', c == '.':
+ b.WriteByte(c)
+ default:
+ b.WriteByte('-')
+ }
+ }
+ return b.String()
+}
+
+// gzipBytes compresses raw at the maximum level. The level matters: these
+// exact bytes are what the caller uploads and what was measured against
+// GitHubMaxGzipBytes, so measuring at one level and uploading at another
+// would make the cap guarantee false.
+func gzipBytes(raw []byte) ([]byte, error) {
+ var buf bytes.Buffer
+ zw, err := gzip.NewWriterLevel(&buf, gzip.BestCompression)
+ if err != nil {
+ return nil, err
+ }
+ if _, err := zw.Write(raw); err != nil {
+ return nil, err
+ }
+ if err := zw.Close(); err != nil {
+ return nil, err
+ }
+ return buf.Bytes(), nil
+}
diff --git a/internal/record/sarif_github_test.go b/internal/record/sarif_github_test.go
new file mode 100644
index 0000000..6357ebc
--- /dev/null
+++ b/internal/record/sarif_github_test.go
@@ -0,0 +1,1321 @@
+package record
+
+// sarif_github_test.go — R.14's evidence.
+//
+// The packet's stop condition is two tests: the sharding test and the
+// DAST-exclusion-is-logged test. Both are here (TestGitHubShardsBeyondResults
+// PerRunCap, TestGitHubDastOnlyExclusionIsCountedNotSilent). The rest of the
+// file exists because the two named tests do not, on their own, establish the
+// property this projection is for: that the loss is TOTAL and ENUMERABLE
+// rather than merely small.
+//
+// In particular:
+//
+// - TestGitHubProjectionEmitsNoUnsupportedBytes searches the produced upload
+// bytes for the forbidden constructs. Asserting on the Go structs would
+// pass while the encoder emitted `"properties":{"anvil/findingId":""}`;
+// asserting on the bytes cannot.
+// - TestGitHubProjectedTypesHaveNoPropertiesMember proves the same thing
+// structurally, so a later edit that adds a properties member to a
+// projected type fails even if no fixture happens to populate it.
+// - TestGitHubDropReasonTableIsExhaustive fails when a new
+// GitHubDropReason is added without a test that produces it. A drop
+// reason nothing exercises is a claim, not a behaviour.
+// - TestGitHubSplitsOnGzipCapAndDropsUnshardableResult crosses the real
+// 10 MB boundary with real incompressible bytes rather than lowering the
+// cap for the test. Lowering it would test the arithmetic and not the
+// guarantee.
+
+import (
+ "bytes"
+ "compress/gzip"
+ "encoding/json"
+ "fmt"
+ "io"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+)
+
+// ---------------------------------------------------------------------------
+// Fixtures
+//
+// Names are prefixed `gh` throughout: this file shares a package with
+// contract_test.go, fingerprint_test.go, mask_test.go and sealing_test.go,
+// and a helper collision across test files is a compile error in a package
+// nobody owns end to end.
+// ---------------------------------------------------------------------------
+
+const (
+ ghAuditID = "3f0c8c1a-77e2-4b19-9c33-8a1d5f0e2b41"
+
+ // ghHash64 is a 64-character lowercase hex digest. Validate() requires
+ // exactly FingerprintDigestHexLen characters and never a truncation.
+ ghHash64 = "8c1e4b0f9a2d77c5e31048ab6f2c9d5e77b1a3c4d2e5f60718293a4b5c6d7e8f"
+
+ // ghLineHash is the shape research/18's annotated record shows for
+ // primaryLocationLineHash: a digest, a colon, and an ordinal.
+ ghLineHash = "4f2a9c71e3b85d60:1"
+
+ // ghEndpoint is an internal hostname. It must never appear in an upload:
+ // GitHub cannot render it as an alert, and publishing it leaks the
+ // staging topology to a third party.
+ ghEndpoint = "https://staging.payments.internal/api/login"
+)
+
+func ghTime() time.Time { return time.Date(2026, 8, 8, 12, 0, 0, 0, time.UTC) }
+
+// ghUntrusted is the trust assertion every fixture result carries. Default
+// untrusted is legal for every external string and is REQUIRED on any result
+// carrying a webResponse (contract.go, ValidateResultTrust).
+func ghUntrusted() TrustAssertion { return TrustAssertion{Default: TrustUntrusted} }
+
+// ghSastResult builds a SAST result GitHub can render: repo-relative path,
+// positive start line, both identity fingerprints.
+func ghSastResult(findingID, uri string, line int) Result {
+ return Result{
+ RuleID: "anvil.sqli.raw-concat",
+ Level: LevelError,
+ Message: Message{Text: "String-concatenated SQL reaches cur.execute()."},
+ Locations: []Location{{
+ PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: uri, URIBaseID: "REPOROOT"},
+ Region: &Region{StartLine: line, Snippet: &Snippet{Text: "query = \"SELECT * FROM users WHERE name = '\" + name"}},
+ },
+ }},
+ PartialFingerprints: map[string]string{
+ PartialFingerprintAnvilFindingID: ghHash64,
+ PartialFingerprintPrimaryLocationLineHash: ghLineHash,
+ },
+ Properties: ResultProperties{
+ FindingID: findingID,
+ Half: HalfSast,
+ Confidence: 0.9,
+ Verdict: VerdictTruePositive,
+ EvidenceClass: EvidenceClassSastReachable,
+ Detector: DetectorRef{Kind: DetectorKindSast, Model: "m", Revision: "r"},
+ Trust: ghUntrusted(),
+ },
+ }
+}
+
+// ghDastResult builds the DAST result research/18's annotated record shows:
+// an ENDPOINT location carrying the `startLine: 1` "placeholder so GitHub can
+// render it". The placeholder is exactly what this projection must refuse.
+func ghDastResult(findingID string) Result {
+ return Result{
+ RuleID: "anvil.dast.sqli",
+ Level: LevelError,
+ Message: Message{Text: "POST /api/login returned HTTP 500 after a quote was injected."},
+ Locations: []Location{{
+ PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: ghEndpoint},
+ Region: &Region{StartLine: 1, StartColumn: 1, EndLine: 1, EndColumn: 2},
+ },
+ Properties: map[string]any{PropLocationKind: "httpEndpoint"},
+ }},
+ WebRequest: &WebRequest{Method: "POST", Target: ghEndpoint},
+ WebResponse: &WebResponse{StatusCode: 500, Body: &ArtifactContent{Text: "sqlite3.OperationalError"}},
+ PartialFingerprints: map[string]string{
+ PartialFingerprintAnvilFindingID: ghHash64,
+ PartialFingerprintPrimaryLocationLineHash: ghLineHash,
+ },
+ Properties: ResultProperties{
+ FindingID: findingID,
+ Half: HalfDast,
+ Confidence: 0.8,
+ Verdict: VerdictTruePositive,
+ EvidenceClass: EvidenceClassDastConfirmed,
+ Detector: DetectorRef{Kind: DetectorKindDast, Model: "m", Revision: "r"},
+ Trust: ghUntrusted(),
+ },
+ }
+}
+
+// ghDastResultNoLocation is the other DAST shape: no file, no endpoint
+// placeholder, no locations at all.
+func ghDastResultNoLocation(findingID string) Result {
+ r := ghDastResult(findingID)
+ r.Locations = nil
+ return r
+}
+
+func ghDastCoverage() *DastCoverage {
+ return &DastCoverage{
+ ProbedCount: 1,
+ InventoryUnionCount: 2,
+ EndpointCoverage: 0.5,
+ InventoryProvenanceMix: map[InventoryProvenance]int{},
+ ConfirmedCount: 1,
+ CandidateCount: 1,
+ }
+}
+
+// ghValidAudit builds a record that passes SARIFLog.Validate(): both halves
+// sealed, deadline anchored to scan start, every frozen enum populated.
+//
+// Projecting a record that a producer could not legally emit would prove
+// nothing, so the tests that assert projection semantics run against this.
+func ghValidAudit(t *testing.T) *SARIFLog {
+ t.Helper()
+ created := ghTime()
+ sealed := created.Add(time.Hour)
+
+ sast := ghSastResult("sast:1", "app/db.py", 414)
+ // Cross-half pointer at the DAST endpoint (research/18's construct), a
+ // CWE taxon, regression provenance, a proposed fix, a third
+ // partialFingerprint key, and a code flow with one endpoint step: every
+ // one of these must be stripped.
+ sast.RelatedLocations = []Location{{
+ ID: ghPtrInt(1),
+ PhysicalLocation: &PhysicalLocation{ArtifactLocation: ArtifactLocation{URI: ghEndpoint}, Region: &Region{StartLine: 1}},
+ }}
+ sast.Taxa = []ReportingDescriptorReference{{ID: "89", Index: ghPtrInt(0)}}
+ sast.Provenance = &ResultProvenance{FirstDetectionRunGUID: "1c0b77aa-2d4e-4f11-9a3c-6e5d4c3b2a19"}
+ sast.Fixes = []Fix{{ArtifactChanges: []ArtifactChange{{ArtifactLocation: ArtifactLocation{URI: "app/db.py"}}}}}
+ sast.PartialFingerprints[PartialFingerprintRegionSHA256] = ghHash64
+ sast.CodeFlows = []CodeFlow{{ThreadFlows: []ThreadFlow{{Locations: []ThreadFlowLocation{
+ {Location: Location{PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: "app/routes.py"}, Region: &Region{StartLine: 88}}}},
+ {Location: Location{PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: ghEndpoint}, Region: &Region{StartLine: 1}}}},
+ }}}}}
+
+ return &SARIFLog{
+ Schema: SARIFSchemaURI,
+ Version: SARIFVersion,
+ Properties: AuditProperties{
+ SchemaVersion: SchemaVersion,
+ AuditID: ghAuditID,
+ State: StateBothSealed,
+ Version: 1,
+ CreatedAt: created,
+ Target: Target{
+ RepoURL: "https://example.invalid/repo.git",
+ Provenance: TargetProvenanceBootedClean,
+ Provisioning: TargetProvisioningEphemeralManifest,
+ },
+ Deadline: Deadline{
+ DeadlineAt: created.Add(time.Duration(DefaultClaimTimeoutSeconds) * time.Second),
+ ClaimTimeoutSeconds: DefaultClaimTimeoutSeconds,
+ },
+ Index: Index{ReadOrder: DefaultReadOrder()},
+ DastStatus: DastStatusCompletedFindings,
+ },
+ Runs: []Run{
+ {
+ Tool: Tool{Driver: ToolComponent{
+ Name: "anvil-sast",
+ Rules: []ReportingDescriptor{
+ {ID: "anvil.sqli.raw-concat",
+ ShortDescription: &Message{Text: "SQL injection"},
+ FullDescription: &Message{Text: "Concatenated SQL reaches a sink."},
+ Help: &Message{Text: "Parameterise the query."},
+ Relationships: []ReportingDescriptorRelationship{{Target: ReportingDescriptorReference{ID: "89"}}}},
+ {ID: "anvil.unused.rule", ShortDescription: &Message{Text: "unreferenced"}},
+ },
+ Taxa: []ReportingDescriptor{{ID: "89"}},
+ }},
+ AutomationDetails: RunAutomationDetails{
+ ID: "anvil-sast/", GUID: "aaaaaaaa-0000-4000-8000-000000000001", CorrelationGUID: ghAuditID,
+ },
+ Taxonomies: []ToolComponent{{Name: "CWE"}},
+ Results: []Result{sast},
+ Properties: RunProperties{
+ Half: HalfSast, Status: HalfStatusSealed, SealedAt: &sealed,
+ AdvisorySnapshot: &AdvisorySnapshot{SnapshotDigest: ghHash64, ScrapedAt: created},
+ },
+ },
+ {
+ Tool: Tool{Driver: ToolComponent{Name: "anvil-dast"}},
+ AutomationDetails: RunAutomationDetails{
+ ID: "anvil-dast/", GUID: "aaaaaaaa-0000-4000-8000-000000000002", CorrelationGUID: ghAuditID,
+ },
+ Results: []Result{ghDastResult("dast:1"), ghDastResultNoLocation("dast:2")},
+ Properties: RunProperties{
+ Half: HalfDast, Status: HalfStatusSealed, SealedAt: &sealed,
+ DastCoverage: ghDastCoverage(),
+ RuntimeTarget: &RuntimeTarget{BaseURL: "https://staging.payments.internal"},
+ },
+ },
+ },
+ }
+}
+
+func ghPtrInt(n int) *int { return &n }
+
+// ---------------------------------------------------------------------------
+// The packet's two named tests
+// ---------------------------------------------------------------------------
+
+// TestGitHubShardsBeyondResultsPerRunCap is the packet's sharding stop
+// condition: a fixture exceeding 25,000 results shards into multiple files,
+// none exceeding a cap, and nothing is truncated away.
+func TestGitHubShardsBeyondResultsPerRunCap(t *testing.T) {
+ const n = GitHubMaxResultsPerRun + 1
+
+ results := make([]Result, 0, n)
+ for i := 0; i < n; i++ {
+ results = append(results, ghSastResult(fmt.Sprintf("sast:%d", i), fmt.Sprintf("app/pkg%d/file%d.py", i%64, i), (i%900)+1))
+ }
+ log := ghOneRunLog(HalfSast, results)
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if len(files) < 2 {
+ t.Fatalf("a %d-result run must shard into multiple files, got %d", n, len(files))
+ }
+
+ total := 0
+ seenAutomationID := map[string]bool{}
+ for i, f := range files {
+ if err := f.WithinCaps(); err != nil {
+ t.Errorf("file %d: %v", i, err)
+ }
+ if got := len(f.Log.Runs); got != GitHubRunsPerProjectedFile {
+ t.Errorf("file %d: %d runs, want %d (one run per file is the shard policy)", i, got, GitHubRunsPerProjectedFile)
+ }
+ if f.ResultCount > GitHubMaxResultsPerRun {
+ t.Errorf("file %d: %d results exceeds the %d cap", i, f.ResultCount, GitHubMaxResultsPerRun)
+ }
+ if f.GzipBytes > GitHubMaxGzipBytes {
+ t.Errorf("file %d: %d gzip bytes exceeds the %d cap", i, f.GzipBytes, GitHubMaxGzipBytes)
+ }
+ if f.ShardCount != len(files) {
+ t.Errorf("file %d: ShardCount %d, want %d", i, f.ShardCount, len(files))
+ }
+ // A repeated automationDetails.id would make GitHub treat the second
+ // shard as REPLACING the first, losing half the alerts silently.
+ id := f.Log.Runs[0].AutomationDetails.ID
+ if seenAutomationID[id] {
+ t.Errorf("file %d: automationDetails.id %q repeats across shards; GitHub would replace, not append", i, id)
+ }
+ seenAutomationID[id] = true
+ total += f.ResultCount
+ }
+
+ if total != n {
+ t.Errorf("sharding lost results: %d across shards, want %d (shard by run, never truncate)", total, n)
+ }
+ loss := GitHubLossOf(files)
+ if loss == nil {
+ t.Fatal("no loss ledger reachable from the projection")
+ }
+ if loss.TotalDropped() != 0 {
+ t.Errorf("nothing should have been dropped, got %d:\n%s", loss.TotalDropped(), loss.Summary())
+ }
+ if loss.ProjectedResultCount != n || loss.SourceResultCount != n {
+ t.Errorf("ledger counts %d source / %d projected, want %d / %d",
+ loss.SourceResultCount, loss.ProjectedResultCount, n, n)
+ }
+
+ // The greedy count split must fill a shard before opening the next one,
+ // so a 25,001-result run is 25,000 + 1 and not two half-full files.
+ if files[0].ResultCount != GitHubMaxResultsPerRun {
+ t.Errorf("first shard holds %d results, want a full %d", files[0].ResultCount, GitHubMaxResultsPerRun)
+ }
+}
+
+// TestGitHubDastOnlyExclusionIsCountedNotSilent is the packet's second stop
+// condition. It covers BOTH DAST-only shapes: the one with no locations, and
+// research/18's endpoint location carrying the `startLine: 1` placeholder —
+// which passes contract.go's looser hasPhysicalCodeLocation and would sail
+// through a naive filter.
+func TestGitHubDastOnlyExclusionIsCountedNotSilent(t *testing.T) {
+ log := ghOneRunLog(HalfDast, []Result{
+ ghDastResult("dast:endpoint-placeholder"),
+ ghDastResultNoLocation("dast:no-location"),
+ ghDastResult("dast:endpoint-2"),
+ })
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+
+ // The loss here is TOTAL. A projection that returned no files at all
+ // would take the explanation with it, which is the silent-drop failure
+ // this packet exists to prevent.
+ if len(files) != 1 {
+ t.Fatalf("a fully-dropped run must still yield one file so the ledger is reachable, got %d files", len(files))
+ }
+ if files[0].ResultCount != 0 {
+ t.Fatalf("expected an empty projected run, got %d results", files[0].ResultCount)
+ }
+
+ loss := GitHubLossOf(files)
+ if loss == nil {
+ t.Fatal("no loss ledger reachable")
+ }
+ if loss.TotalDropped() != 3 {
+ t.Fatalf("want 3 dropped results, got %d:\n%s", loss.TotalDropped(), loss.Summary())
+ }
+ if got := loss.DropCounts[GitHubDropLocationNotRepoRelative]; got != 2 {
+ t.Errorf("endpoint-located DAST results dropped: %d, want 2", got)
+ }
+ if got := loss.DropCounts[GitHubDropNoLocations]; got != 1 {
+ t.Errorf("location-free DAST results dropped: %d, want 1", got)
+ }
+
+ // "Logged count rather than silently dropped": the finding ids are
+ // recoverable, not just a number.
+ wantIDs := map[string]bool{"dast:endpoint-placeholder": true, "dast:endpoint-2": true}
+ for _, d := range loss.DroppedFor(GitHubDropLocationNotRepoRelative) {
+ if !wantIDs[d.FindingID] {
+ t.Errorf("unexpected dropped finding id %q", d.FindingID)
+ }
+ delete(wantIDs, d.FindingID)
+ if d.Half != HalfDast {
+ t.Errorf("dropped %q recorded half %q, want %q", d.FindingID, d.Half, HalfDast)
+ }
+ if d.SourceRunIndex != 0 {
+ t.Errorf("dropped %q recorded source run %d, want 0", d.FindingID, d.SourceRunIndex)
+ }
+ }
+ if len(wantIDs) != 0 {
+ t.Errorf("these dropped findings were never recorded: %v", wantIDs)
+ }
+
+ summary := loss.Summary()
+ for _, want := range []string{
+ string(GitHubDropLocationNotRepoRelative),
+ string(GitHubDropNoLocations),
+ "3 dropped",
+ } {
+ if !strings.Contains(summary, want) {
+ t.Errorf("Summary() omits %q:\n%s", want, summary)
+ }
+ }
+}
+
+// ghOneRunLog builds a minimal single-run log. It is deliberately NOT
+// Validate()-clean for every half: several drop-reason cases describe records
+// a producer should never emit, and the projection must survive them anyway.
+func ghOneRunLog(half Half, results []Result) *SARIFLog {
+ sealed := ghTime().Add(time.Hour)
+ rp := RunProperties{Half: half, Status: HalfStatusSealed, SealedAt: &sealed}
+ if half == HalfDast {
+ rp.DastCoverage = ghDastCoverage()
+ }
+ return &SARIFLog{
+ Schema: SARIFSchemaURI,
+ Version: SARIFVersion,
+ Properties: AuditProperties{
+ SchemaVersion: SchemaVersion,
+ AuditID: ghAuditID,
+ State: StateBothSealed,
+ Version: 1,
+ CreatedAt: ghTime(),
+ DastStatus: DastStatusCompletedFindings,
+ },
+ Runs: []Run{{
+ Tool: Tool{Driver: ToolComponent{Name: "anvil-" + string(half), Rules: []ReportingDescriptor{
+ {ID: "anvil.sqli.raw-concat"}, {ID: "anvil.dast.sqli"},
+ }}},
+ AutomationDetails: RunAutomationDetails{ID: "anvil-" + string(half) + "/", CorrelationGUID: ghAuditID},
+ Results: results,
+ Properties: rp,
+ }},
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The loss is total and enumerable
+// ---------------------------------------------------------------------------
+
+// TestGitHubProjectionEmitsNoUnsupportedBytes asserts on the UPLOAD BYTES,
+// not on the Go structs. A zeroed ResultProperties would still marshal to a
+// bag of empty `anvil/*` keys, and only a byte search catches that.
+func TestGitHubProjectionEmitsNoUnsupportedBytes(t *testing.T) {
+ log := ghValidAudit(t)
+ if err := log.Validate(); err != nil {
+ t.Fatalf("the fixture must be a record a producer could legally emit: %v", err)
+ }
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+
+ forbidden := []struct{ needle, why string }{
+ {"anvil/", "an anvil/* property bag reached the upload"},
+ {"webRequest", "SARIF §3.27.14 DAST evidence reached the upload"},
+ {"webResponse", "SARIF §3.27.15 DAST evidence reached the upload"},
+ {"taxonomies", "taxonomies-as-relationships reached the upload"},
+ {"\"taxa\"", "a result taxa reference reached the upload with no taxonomies array to resolve it"},
+ {"relationships", "a rule's taxonomy relationship reached the upload"},
+ {"provenance", "SARIF §3.48 regression history reached the upload"},
+ {"\"fixes\"", "a proposed fix reached the upload (00-SPINE.md S7: propose only)"},
+ {"externalPropertyFileReferences", "a reference GitHub will never fetch reached the upload"},
+ {ghEndpoint, "an internal hostname was published to GitHub"},
+ {PartialFingerprintRegionSHA256, "an unread partial fingerprint reached the upload"},
+ }
+
+ for i, f := range files {
+ for _, fb := range forbidden {
+ if bytes.Contains(f.JSON, []byte(fb.needle)) {
+ t.Errorf("file %d (%s): %s — found %q", i, f.Name, fb.why, fb.needle)
+ }
+ }
+ // The gzip must be the gzip OF THE RETURNED JSON: a caller that
+ // uploads f.Gzip must be uploading exactly what was measured.
+ if got := ghGunzip(t, f.Gzip); !bytes.Equal(got, f.JSON) {
+ t.Errorf("file %d: Gzip is not the compression of JSON", i)
+ }
+ }
+
+ // What DID survive: exactly the one renderable SAST result.
+ loss := GitHubLossOf(files)
+ if loss.ProjectedResultCount != 1 {
+ t.Fatalf("want 1 projected result, got %d:\n%s", loss.ProjectedResultCount, loss.Summary())
+ }
+ var kept *GitHubResult
+ for i := range files {
+ if len(files[i].Log.Runs[0].Results) == 1 {
+ kept = &files[i].Log.Runs[0].Results[0]
+ }
+ }
+ if kept == nil {
+ t.Fatal("the renderable SAST result did not survive the projection")
+ }
+ if got := kept.PartialFingerprints[PartialFingerprintPrimaryLocationLineHash]; got != ghLineHash {
+ t.Errorf("primaryLocationLineHash is %q, want %q — GitHub reads only this key", got, ghLineHash)
+ }
+ if got := kept.PartialFingerprints[PartialFingerprintAnvilFindingID]; got != ghHash64 {
+ t.Errorf("the anvil finding id must survive so an alert traces back to a record finding, got %q", got)
+ }
+ if len(kept.PartialFingerprints) != 2 {
+ t.Errorf("partialFingerprints carries %d keys, want exactly the two identity keys", len(kept.PartialFingerprints))
+ }
+ if len(kept.RelatedLocations) != 0 {
+ t.Errorf("the endpoint relatedLocation must not survive, got %d", len(kept.RelatedLocations))
+ }
+ // The code flow keeps its repo step and loses its endpoint step.
+ if n := countThreadFlowLocations(kept.CodeFlows); n != 1 {
+ t.Errorf("code flow kept %d steps, want 1 (the repo step, not the endpoint step)", n)
+ }
+}
+
+// TestGitHubStripsAreCounted checks the second half of "enumerable": a field
+// removed from a result that DID reach GitHub is counted by kind.
+func TestGitHubStripsAreCounted(t *testing.T) {
+ files, err := ProjectForGitHub(ghValidAudit(t))
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ loss := GitHubLossOf(files)
+
+ want := map[GitHubStripReason]int{
+ GitHubStripAuditProperties: 1,
+ GitHubStripRunProperties: 2,
+ GitHubStripRunTaxonomies: 1,
+ GitHubStripDriverTaxa: 1,
+ GitHubStripResultTaxa: 1,
+ GitHubStripRuleRelationships: 1,
+ GitHubStripResultProvenance: 1,
+ GitHubStripResultFixes: 1,
+ GitHubStripResultProperties: 1,
+ GitHubStripPartialFingerprintKey: 1,
+ GitHubStripRelatedLocationNotRepoRelative: 1,
+ GitHubStripThreadFlowLocationNotRepoRelative: 1,
+ GitHubStripUnreferencedRule: 1,
+ }
+ for reason, n := range want {
+ if got := loss.StripCounts[reason]; got != n {
+ t.Errorf("StripCounts[%s] = %d, want %d\n%s", reason, got, n, loss.Summary())
+ }
+ }
+ // webRequest/webResponse belong to results that were DROPPED whole, so
+ // they are accounted as a drop and not double-counted as a strip.
+ if got := loss.StripCounts[GitHubStripWebResponse]; got != 0 {
+ t.Errorf("a dropped result's webResponse was counted twice: strip count %d", got)
+ }
+ for reason := range loss.StripCounts {
+ if !reason.Valid() {
+ t.Errorf("ledger carries a strip reason outside the closed vocabulary: %q", reason)
+ }
+ }
+}
+
+// TestGitHubDropReasonTableIsExhaustive drives one minimal record per drop
+// reason and then asserts the table covered the whole vocabulary. Adding a
+// GitHubDropReason without a case that produces it fails here.
+func TestGitHubDropReasonTableIsExhaustive(t *testing.T) {
+ base := func(mutate func(*Result)) *SARIFLog {
+ r := ghSastResult("sast:probe", "app/db.py", 12)
+ mutate(&r)
+ return ghOneRunLog(HalfSast, []Result{r})
+ }
+
+ // unreadable builds a run that is otherwise perfectly projectable and whose
+ // HALF the read gate refuses. It is first in the table because the gate is
+ // evaluated first: none of the per-result questions below is worth asking
+ // about results a consumer may not read at all.
+ unreadable := func(status HalfStatus, state State) *SARIFLog {
+ l := ghOneRunLog(HalfSast, []Result{ghSastResult("sast:probe", "app/db.py", 12)})
+ l.Runs[0].Properties.Status = status
+ if status != HalfStatusSealed {
+ l.Runs[0].Properties.SealedAt = nil
+ }
+ l.Properties.State = state
+ return l
+ }
+
+ cases := []struct {
+ name string
+ log *SARIFLog
+ want GitHubDropReason
+ }{
+ {"half still running", unreadable(HalfStatusRunning, StateCollecting), GitHubDropHalfNotReadable},
+ {"half failed", unreadable(HalfStatusFailed, StateCollecting), GitHubDropHalfNotReadable},
+ {"audit expired holding a sealed half", unreadable(HalfStatusSealed, StateExpired), GitHubDropHalfNotReadable},
+ {"no locations", base(func(r *Result) { r.Locations = nil }), GitHubDropNoLocations},
+ {"no physical location", base(func(r *Result) {
+ r.Locations = []Location{{LogicalLocations: []LogicalLocation{{FullyQualifiedName: "pkg.fn"}}}}
+ }), GitHubDropNoPhysicalLocation},
+ {"absolute http uri", base(func(r *Result) {
+ r.Locations[0].PhysicalLocation.ArtifactLocation.URI = ghEndpoint
+ }), GitHubDropLocationNotRepoRelative},
+ {"absolute host path", base(func(r *Result) {
+ r.Locations[0].PhysicalLocation.ArtifactLocation.URI = "/etc/nginx/nginx.conf"
+ }), GitHubDropLocationNotRepoRelative},
+ {"escapes repo root", base(func(r *Result) {
+ r.Locations[0].PhysicalLocation.ArtifactLocation.URI = "../outside/db.py"
+ }), GitHubDropLocationNotRepoRelative},
+ {"no start line", base(func(r *Result) {
+ r.Locations[0].PhysicalLocation.Region = nil
+ }), GitHubDropNoStartLine},
+ {"zero start line", base(func(r *Result) {
+ r.Locations[0].PhysicalLocation.Region = &Region{StartLine: 0}
+ }), GitHubDropNoStartLine},
+ {"missing line hash", base(func(r *Result) {
+ delete(r.PartialFingerprints, PartialFingerprintPrimaryLocationLineHash)
+ }), GitHubDropNoPrimaryLocationLineHash},
+ {"blank line hash", base(func(r *Result) {
+ r.PartialFingerprints[PartialFingerprintPrimaryLocationLineHash] = ""
+ }), GitHubDropNoPrimaryLocationLineHash},
+ {"blank message", base(func(r *Result) { r.Message = Message{Text: " "} }), GitHubDropNoMessageText},
+ }
+
+ covered := map[GitHubDropReason]bool{}
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ files, err := ProjectForGitHub(tc.log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ loss := GitHubLossOf(files)
+ if loss.TotalDropped() != 1 {
+ t.Fatalf("want exactly 1 drop, got %d:\n%s", loss.TotalDropped(), loss.Summary())
+ }
+ if got := loss.DroppedResults[0].Reason; got != tc.want {
+ t.Fatalf("drop reason %q, want %q", got, tc.want)
+ }
+ if !tc.want.Valid() {
+ t.Fatalf("%q is outside the closed drop vocabulary", tc.want)
+ }
+ if tc.want.Explain() == "unknown drop reason" {
+ t.Errorf("%q has no explanation; a consumer asking WHY gets nothing", tc.want)
+ }
+ })
+ covered[tc.want] = true
+ }
+
+ // GitHubDropExceedsFileSizeCap is produced by the gzip-cap test below;
+ // it needs a 10 MB fixture and does not belong in this table.
+ covered[GitHubDropExceedsFileSizeCap] = true
+ for _, r := range GitHubDropReasonValues() {
+ if !covered[r] {
+ t.Errorf("drop reason %q is declared but no test produces it", r)
+ }
+ }
+}
+
+// TestGitHubSplitsOnGzipCapAndDropsUnshardableResult crosses the real 10 MB
+// gzip boundary with real incompressible bytes.
+//
+// Two properties are proved at once. A run whose results do not fit is SPLIT
+// rather than truncated; and the one case that cannot be split — a single
+// result whose own file exceeds the cap — is dropped with a named reason
+// instead of being allowed to break the guarantee.
+func TestGitHubSplitsOnGzipCapAndDropsUnshardableResult(t *testing.T) {
+ // ~13.5 MiB of high-entropy text. gzip's floor on 64-symbol data is
+ // 6 bits per byte, so this cannot compress below ~10.1 MiB.
+ giant := ghSastResult("sast:giant", "app/giant.py", 1)
+ giant.Locations[0].PhysicalLocation.Region.Snippet = &Snippet{Text: ghHighEntropyText(14_200_000)}
+
+ results := []Result{
+ ghSastResult("sast:a", "app/a.py", 10),
+ giant,
+ ghSastResult("sast:b", "app/b.py", 20),
+ }
+ files, err := ProjectForGitHub(ghOneRunLog(HalfSast, results))
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+
+ loss := GitHubLossOf(files)
+ if got := loss.DropCounts[GitHubDropExceedsFileSizeCap]; got != 1 {
+ t.Fatalf("want the unshardable result dropped once, got %d:\n%s", got, loss.Summary())
+ }
+ // A drop that happens LATE, during size bisection, must still name the
+ // record finding it came from. A count without an identity is not an
+ // enumeration.
+ dropped := loss.DroppedFor(GitHubDropExceedsFileSizeCap)
+ if dropped[0].FindingID != "sast:giant" {
+ t.Errorf("size-dropped result recorded finding id %q, want %q", dropped[0].FindingID, "sast:giant")
+ }
+ if dropped[0].SourceResultIndex != 1 {
+ t.Errorf("size-dropped result recorded source index %d, want 1", dropped[0].SourceResultIndex)
+ }
+ total := 0
+ for i, f := range files {
+ if err := f.WithinCaps(); err != nil {
+ t.Errorf("file %d: %v", i, err)
+ }
+ if f.GzipBytes > GitHubMaxGzipBytes {
+ t.Errorf("file %d: %d gzip bytes exceeds the %d cap", i, f.GzipBytes, GitHubMaxGzipBytes)
+ }
+ total += f.ResultCount
+ }
+ if total != 2 {
+ t.Errorf("the two small results must survive, got %d", total)
+ }
+ if loss.ProjectedResultCount != 2 || loss.SourceResultCount != 3 {
+ t.Errorf("ledger says %d of %d projected, want 2 of 3", loss.ProjectedResultCount, loss.SourceResultCount)
+ }
+
+ // The harder case: the oversized result is the ONLY result, so the run
+ // bisects down to nothing. A projection that returned no files here
+ // would return no ledger either, and the loss would become invisible at
+ // exactly the moment it became total.
+ t.Run("only result is unshardable", func(t *testing.T) {
+ solo := ghSastResult("sast:solo-giant", "app/giant.py", 1)
+ solo.Locations[0].PhysicalLocation.Region.Snippet = giant.Locations[0].PhysicalLocation.Region.Snippet
+ files, err := ProjectForGitHub(ghOneRunLog(HalfSast, []Result{solo}))
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if len(files) != 1 || files[0].ResultCount != 0 {
+ t.Fatalf("want one empty file carrying the ledger, got %d files", len(files))
+ }
+ loss := GitHubLossOf(files)
+ if loss == nil || loss.DropCounts[GitHubDropExceedsFileSizeCap] != 1 {
+ t.Fatalf("the total loss was not recorded: %#v", loss)
+ }
+ if err := files[0].WithinCaps(); err != nil {
+ t.Errorf("the fallback empty file breaks a cap: %v", err)
+ }
+ })
+}
+
+// ghHighEntropyText returns n bytes drawn from a 64-symbol alphabet by a
+// deterministic xorshift. Deterministic because a test that sometimes crosses
+// a size boundary is not a test; 64 symbols because that is the JSON-safe
+// alphabet with the highest entropy per byte, and therefore the cheapest way
+// to defeat gzip.
+func ghHighEntropyText(n int) string {
+ const alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-"
+ b := make([]byte, n)
+ x := uint64(0x9E3779B97F4A7C15)
+ for i := range b {
+ x ^= x << 13
+ x ^= x >> 7
+ x ^= x << 17
+ b[i] = alphabet[x&63]
+ }
+ return string(b)
+}
+
+func ghGunzip(t *testing.T, gz []byte) []byte {
+ t.Helper()
+ zr, err := gzip.NewReader(bytes.NewReader(gz))
+ if err != nil {
+ t.Fatalf("gzip.NewReader: %v", err)
+ }
+ defer zr.Close()
+ out, err := io.ReadAll(zr)
+ if err != nil {
+ t.Fatalf("gunzip: %v", err)
+ }
+ return out
+}
+
+// ---------------------------------------------------------------------------
+// Structural guarantees
+// ---------------------------------------------------------------------------
+
+// TestGitHubProjectedTypesHaveNoPropertiesMember proves the anvil/* bag
+// cannot be emitted, independently of any fixture. The byte test above can
+// only catch a bag a fixture happened to populate; this catches the type.
+func TestGitHubProjectedTypesHaveNoPropertiesMember(t *testing.T) {
+ types := []reflect.Type{
+ reflect.TypeOf(GitHubSARIFLog{}),
+ reflect.TypeOf(GitHubRun{}),
+ reflect.TypeOf(GitHubResult{}),
+ reflect.TypeOf(GitHubLocation{}),
+ reflect.TypeOf(GitHubCodeFlow{}),
+ reflect.TypeOf(GitHubThreadFlow{}),
+ reflect.TypeOf(GitHubThreadFlowLocation{}),
+ }
+ for _, typ := range types {
+ for i := 0; i < typ.NumField(); i++ {
+ f := typ.Field(i)
+ tag := strings.Split(f.Tag.Get("json"), ",")[0]
+ if tag == "properties" || f.Name == "Properties" {
+ t.Errorf("%s.%s is a properties member; a projected type must have none, "+
+ "or an anvil/* bag can reach GitHub", typ.Name(), f.Name)
+ }
+ }
+ }
+}
+
+// TestGitHubCapsMatchTheDocumentedNumbers pins every cap to the value
+// research/18 quotes from GitHub's documentation [S2]. GitHub's limits "can
+// change without notice"; when they do, this test is the one place that has
+// to be edited, and it fails loudly rather than letting a silently-edited
+// constant ship.
+func TestGitHubCapsMatchTheDocumentedNumbers(t *testing.T) {
+ for _, c := range []struct {
+ name string
+ got int
+ want int
+ }{
+ {"results per run", GitHubMaxResultsPerRun, 25000},
+ {"displayed results per run", GitHubDisplayedResultsPerRun, 5000},
+ {"runs per file", GitHubMaxRunsPerFile, 20},
+ {"gzip bytes per file", GitHubMaxGzipBytes, 10 * 1024 * 1024},
+ {"rules per run", GitHubMaxRulesPerRun, 25000},
+ {"tool extensions per run", GitHubMaxToolExtensionsPerRun, 100},
+ {"locations per result", GitHubMaxLocationsPerResult, 1000},
+ {"thread-flow locations per result", GitHubMaxThreadFlowLocationsPerResult, 10000},
+ } {
+ if c.got != c.want {
+ t.Errorf("%s = %d, want GitHub's documented %d", c.name, c.got, c.want)
+ }
+ }
+ if GitHubRunsPerProjectedFile > GitHubMaxRunsPerFile {
+ t.Errorf("Anvil's own shard policy (%d runs/file) exceeds GitHub's limit (%d)",
+ GitHubRunsPerProjectedFile, GitHubMaxRunsPerFile)
+ }
+}
+
+// TestGitHubPinsSarifVersion: GitHub supports SARIF 2.1.0 only, and the
+// projection must not track the unratified 2.2 draft.
+func TestGitHubPinsSarifVersion(t *testing.T) {
+ files, err := ProjectForGitHub(ghValidAudit(t))
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ for i, f := range files {
+ if f.Log.Version != SARIFVersion {
+ t.Errorf("file %d: version %q, want %q", i, f.Log.Version, SARIFVersion)
+ }
+ if f.Log.Schema != SARIFSchemaURI {
+ t.Errorf("file %d: $schema %q, want %q", i, f.Log.Schema, SARIFSchemaURI)
+ }
+ }
+}
+
+// TestGitHubRuleIndexIsRemappedNotStale: filtering rules invalidates every
+// source ruleIndex. A stale index does not fail loudly — it names a DIFFERENT
+// rule, so GitHub renders the wrong description on the alert.
+func TestGitHubRuleIndexIsRemappedNotStale(t *testing.T) {
+ // Two rules, the referenced one second, so a copied index would point at
+ // the wrong rule and a dropped index would point at nothing.
+ log := ghOneRunLog(HalfSast, []Result{ghSastResult("sast:1", "app/db.py", 5)})
+ log.Runs[0].Tool.Driver.Rules = []ReportingDescriptor{
+ {ID: "anvil.unused.rule"},
+ {ID: "anvil.sqli.raw-concat"},
+ }
+ log.Runs[0].Results[0].RuleIndex = ghPtrInt(1)
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ run := files[0].Log.Runs[0]
+ if len(run.Tool.Driver.Rules) != 1 {
+ t.Fatalf("want only the referenced rule emitted, got %d", len(run.Tool.Driver.Rules))
+ }
+ res := run.Results[0]
+ if res.RuleIndex == nil {
+ t.Fatal("ruleIndex was dropped; the alert loses its rule metadata link")
+ }
+ if got := *res.RuleIndex; got != 0 {
+ t.Fatalf("ruleIndex = %d, want 0 after re-indexing", got)
+ }
+ if run.Tool.Driver.Rules[*res.RuleIndex].ID != res.RuleID {
+ t.Fatalf("ruleIndex %d names rule %q but the result's ruleId is %q",
+ *res.RuleIndex, run.Tool.Driver.Rules[*res.RuleIndex].ID, res.RuleID)
+ }
+}
+
+// TestGitHubProjectionIsDeterministic: two projections of one record must be
+// byte-identical. An upload that changes shape between runs makes GitHub's
+// de-duplication unreliable no matter what the fingerprints say.
+func TestGitHubProjectionIsDeterministic(t *testing.T) {
+ log := ghValidAudit(t)
+ a, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ b, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if len(a) != len(b) {
+ t.Fatalf("file counts differ: %d vs %d", len(a), len(b))
+ }
+ for i := range a {
+ if a[i].Name != b[i].Name {
+ t.Errorf("file %d name differs: %q vs %q", i, a[i].Name, b[i].Name)
+ }
+ if !bytes.Equal(a[i].JSON, b[i].JSON) {
+ t.Errorf("file %d JSON differs between identical projections", i)
+ }
+ }
+ if GitHubLossOf(a).Summary() != GitHubLossOf(b).Summary() {
+ t.Error("loss Summary() differs between identical projections; map order reached the output")
+ }
+ // The ledger must survive a round trip, since the intended use is to
+ // persist it next to the upload.
+ raw, err := json.Marshal(GitHubLossOf(a))
+ if err != nil {
+ t.Fatalf("marshal ledger: %v", err)
+ }
+ var back GitHubProjectionLoss
+ if err := json.Unmarshal(raw, &back); err != nil {
+ t.Fatalf("unmarshal ledger: %v", err)
+ }
+ if back.TotalDropped() != GitHubLossOf(a).TotalDropped() {
+ t.Errorf("ledger round trip lost drops: %d vs %d", back.TotalDropped(), GitHubLossOf(a).TotalDropped())
+ }
+}
+
+// TestGitHubLedgerReconciles: source == projected + dropped, on every fixture
+// in this file. This is the arithmetic that makes "the loss is total" a
+// checkable claim rather than a comment.
+func TestGitHubLedgerReconciles(t *testing.T) {
+ logs := map[string]*SARIFLog{
+ "valid audit": ghValidAudit(t),
+ "dast only": ghOneRunLog(HalfDast, []Result{ghDastResult("d1"), ghDastResultNoLocation("d2")}),
+ "sast only": ghOneRunLog(HalfSast, []Result{ghSastResult("s1", "a.py", 1), ghSastResult("s2", "b.py", 2)}),
+ "empty run": ghOneRunLog(HalfSast, nil),
+ }
+ for name, log := range logs {
+ t.Run(name, func(t *testing.T) {
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ loss := GitHubLossOf(files)
+ if got := loss.ProjectedResultCount + loss.TotalDropped(); got != loss.SourceResultCount {
+ t.Errorf("%d projected + %d dropped = %d, want %d source results\n%s",
+ loss.ProjectedResultCount, loss.TotalDropped(), got, loss.SourceResultCount, loss.Summary())
+ }
+ if loss.AuditID != ghAuditID {
+ t.Errorf("ledger audit id %q, want %q", loss.AuditID, ghAuditID)
+ }
+ // Every run contributes at least one file, so the ledger is
+ // always reachable.
+ if len(files) < len(log.Runs) {
+ t.Errorf("%d files for %d source runs; a run with no surviving results still owes a file",
+ len(files), len(log.Runs))
+ }
+ for i := range files {
+ if files[i].Loss != loss {
+ t.Errorf("file %d points at a different ledger; the ledger is whole-projection", i)
+ }
+ }
+ })
+ }
+}
+
+// TestIsRepoRelativeURI pins the predicate that decides what GitHub can
+// render. It is stricter than contract.go's hasPhysicalCodeLocation on
+// purpose — see the header of sarif_github.go.
+func TestIsRepoRelativeURI(t *testing.T) {
+ cases := []struct {
+ uri string
+ want bool
+ }{
+ {"app/db.py", true},
+ {"src/main/java/com/x/Y.java", true},
+ {"file.go", true},
+ {"a..b/c.go", true},
+ {"", false},
+ {"/etc/passwd", false},
+ {`\windows\system32\x.dll`, false},
+ {"https://staging.payments.internal/api/login", false},
+ {"http://x/y", false},
+ {"file:///tmp/x", false},
+ {"C:/Users/x/y.go", false},
+ {"../outside.py", false},
+ {"a/../../b.py", false},
+ }
+ for _, c := range cases {
+ if got := isRepoRelativeURI(c.uri); got != c.want {
+ t.Errorf("isRepoRelativeURI(%q) = %v, want %v", c.uri, got, c.want)
+ }
+ }
+}
+
+// TestGitHubShardAutomationIDsAreDistinct pins the id derivation, because the
+// failure it prevents is silent: GitHub keys an analysis on
+// automationDetails.id, and a duplicate makes the second upload replace the
+// first.
+func TestGitHubShardAutomationIDsAreDistinct(t *testing.T) {
+ cases := []struct {
+ id string
+ shard int
+ want string
+ }{
+ {"anvil/sast/", 2, "anvil/sast/shard-002/"},
+ {"anvil/sast", 3, "anvil/sast/shard-003/"},
+ {"", 2, "shard-002/"},
+ }
+ for _, c := range cases {
+ if got := shardAutomationID(c.id, c.shard); got != c.want {
+ t.Errorf("shardAutomationID(%q, %d) = %q, want %q", c.id, c.shard, got, c.want)
+ }
+ }
+}
+
+// TestGitHubNilRecordIsAnError: a nil record is a caller bug, not an empty
+// projection. Returning "no files, no error" would read as "nothing to
+// upload".
+func TestGitHubNilRecordIsAnError(t *testing.T) {
+ if _, err := ProjectForGitHub(nil); err == nil {
+ t.Fatal("want an error for a nil record")
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 B1 — the read gate reaches the most externally visible consumer
+// ---------------------------------------------------------------------------
+
+// TestGitHubNeverPublishesAnUnreadableHalf is the regression test for the
+// blocker: ProjectForGitHub consulted no read gate at all, so a half at
+// `running`, `failed`, `timed_out` or `skipped` — and a cleanly sealed half in
+// an EXPIRED audit — projected its results to GitHub code scanning
+// unconditionally, with the loss ledger recording zero drops in every case.
+//
+// The consequences are ordered in CRITIQUE-03 §6 B1 and the worst is not the
+// first: because GitHub keys an analysis on `runAutomationDetails.id`, a
+// premature upload of a `running` half is REPLACED by the real upload after
+// the seal, so the visible alert set silently depends on upload order.
+func TestGitHubNeverPublishesAnUnreadableHalf(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ status HalfStatus
+ state State
+ }{
+ {"running", HalfStatusRunning, StateCollecting},
+ {"failed", HalfStatusFailed, StateCollecting},
+ {"timed_out", HalfStatusTimedOut, StateCollecting},
+ {"skipped", HalfStatusSkipped, StateCollecting},
+ {"sealed but the audit expired", HalfStatusSealed, StateExpired},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ log := ghOneRunLog(HalfSast, []Result{
+ ghSastResult("sast:1", "app/db.py", 414),
+ ghSastResult("sast:2", "app/routes.py", 88),
+ })
+ log.Runs[0].Properties.Status = tc.status
+ if tc.status != HalfStatusSealed {
+ log.Runs[0].Properties.SealedAt = nil
+ }
+ log.Properties.State = tc.state
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+
+ // Nothing is published, and the projected BYTES are searched, not
+ // just the counts: a result that survived into Log.Runs would
+ // still reach GitHub even if ResultCount were reported as zero.
+ for i, f := range files {
+ if f.ResultCount != 0 {
+ t.Errorf("file %d carries %d results from a half whose status is %q and audit state %q",
+ i, f.ResultCount, tc.status, tc.state)
+ }
+ if bytes.Contains(f.JSON, []byte("app/db.py")) {
+ t.Errorf("file %d publishes a source path from an unreadable half", i)
+ }
+ }
+
+ // The loss is COUNTABLE. A withheld result that is not ledgered is
+ // a silent drop, and the ledger is the entire mechanism by which
+ // R.14 answers research/18 Risk #6.
+ loss := GitHubLossOf(files)
+ if loss == nil {
+ t.Fatal("a fully-withheld projection must still carry a reachable ledger")
+ }
+ if got := loss.DropCounts[GitHubDropHalfNotReadable]; got != 2 {
+ t.Errorf("DropCounts[%s] = %d, want 2\n%s", GitHubDropHalfNotReadable, got, loss.Summary())
+ }
+ for _, d := range loss.DroppedFor(GitHubDropHalfNotReadable) {
+ if d.FindingID == "" {
+ t.Error("a withheld result was ledgered with no finding id; a count without an identity is not an enumeration")
+ }
+ if d.HalfStatus != tc.status || d.AuditState != tc.state {
+ t.Errorf("ledger entry for %q records status %q / state %q, want %q / %q",
+ d.FindingID, d.HalfStatus, d.AuditState, tc.status, tc.state)
+ }
+ }
+ if !strings.Contains(loss.Summary(), string(GitHubDropHalfNotReadable)) {
+ t.Errorf("Summary() does not name the refusal:\n%s", loss.Summary())
+ }
+ })
+ }
+}
+
+// The projection and readpath.go must agree about which halves are readable.
+// Two gates would be two answers, which is the shape CRITIQUE-02 F6 recorded
+// and CRITIQUE-03 found twice more.
+func TestGitHubReadGateAgreesWithTheReadPath(t *testing.T) {
+ for _, status := range HalfStatusValues() {
+ for _, state := range StateValues() {
+ log := ghOneRunLog(HalfSast, []Result{ghSastResult("sast:1", "app/db.py", 414)})
+ log.Runs[0].Properties.Status = status
+ if status != HalfStatusSealed {
+ log.Runs[0].Properties.SealedAt = nil
+ }
+ log.Properties.State = state
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("status %q state %q: %v", status, state, err)
+ }
+ published := 0
+ for _, f := range files {
+ published += f.ResultCount
+ }
+ readable := halfSealOfRun(log, &log.Runs[0]).Readable()
+ if (published > 0) != readable {
+ t.Errorf("status %q state %q: the projection published %d results but the read gate says readable=%t",
+ status, state, published, readable)
+ }
+ }
+ }
+}
+
+// Masking is a precondition here for the same reason it is one on the Reader:
+// this projection strips every surface MaskRecord covers, so its safety rests
+// entirely on that strip list staying exhaustive.
+func TestGitHubRefusesAnUnmaskedRecord(t *testing.T) {
+ r := ghSastResult("sast:1", "app/db.py", 414)
+ r.WebRequest = &WebRequest{
+ Method: "POST", Target: ghEndpoint,
+ Headers: map[string]string{"Authorization": "Bearer sk-live-0123456789abcdefghij"},
+ }
+ log := ghOneRunLog(HalfSast, []Result{r})
+
+ if _, err := ProjectForGitHub(log); err == nil {
+ t.Fatal("an unmasked record must be refused, not projected")
+ }
+ if err := MaskRecord(log); err != nil {
+ t.Fatalf("MaskRecord: %v", err)
+ }
+ if _, err := ProjectForGitHub(log); err != nil {
+ t.Fatalf("the same record, masked, must project: %v", err)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 M2 — the rule ledger counts against the shard set
+// ---------------------------------------------------------------------------
+
+// A rule referenced only by shard 2 used to be tallied as "stripped" while
+// shard 1 was built, so the count scaled with shard count and could exceed the
+// number of rules that exist. The ledger's own stated principle is that
+// over-reporting loss is as untrustworthy as under-reporting it.
+func TestGitHubUnreferencedRuleCountIsAgainstTheShardSetNotPerShard(t *testing.T) {
+ const n = GitHubMaxResultsPerRun + 10
+
+ results := make([]Result, 0, n)
+ for i := 0; i < n; i++ {
+ r := ghSastResult(fmt.Sprintf("sast:%d", i), fmt.Sprintf("app/pkg%d/file%d.py", i%64, i), (i%900)+1)
+ // rule.0 is referenced only by the first (full) shard; rule.3 only by
+ // the overflow shard. rule.1 and rule.2 are referenced by nothing and
+ // are the only rules genuinely lost.
+ if i < GitHubMaxResultsPerRun {
+ r.RuleID = "rule.0"
+ } else {
+ r.RuleID = "rule.3"
+ }
+ results = append(results, r)
+ }
+ log := ghOneRunLog(HalfSast, results)
+ log.Runs[0].Tool.Driver.Rules = []ReportingDescriptor{
+ {ID: "rule.0"}, {ID: "rule.1"}, {ID: "rule.2"}, {ID: "rule.3"},
+ }
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if len(files) < 2 {
+ t.Fatalf("the fixture must shard to exercise the defect, got %d files", len(files))
+ }
+
+ delivered := map[string]bool{}
+ for _, f := range files {
+ for _, run := range f.Log.Runs {
+ for _, rule := range run.Tool.Driver.Rules {
+ delivered[rule.ID] = true
+ }
+ }
+ }
+ if !delivered["rule.0"] || !delivered["rule.3"] {
+ t.Fatalf("both referenced rules must reach GitHub across the shard set, delivered=%v", delivered)
+ }
+
+ loss := GitHubLossOf(files)
+ if got := loss.StripCounts[GitHubStripUnreferencedRule]; got != 2 {
+ t.Errorf("StripCounts[%s] = %d, want 2 (rule.1 and rule.2); "+
+ "a rule delivered in ANY shard is not lost, and a count that exceeds the "+
+ "%d rules in the source run cannot be a count of anything real\n%s",
+ GitHubStripUnreferencedRule, got, len(log.Runs[0].Tool.Driver.Rules), loss.Summary())
+ }
+ if got := loss.StripCounts[GitHubStripUnreferencedRule]; got > len(log.Runs[0].Tool.Driver.Rules) {
+ t.Errorf("the ledger reports more lost rules (%d) than the run has (%d)",
+ got, len(log.Runs[0].Tool.Driver.Rules))
+ }
+}
+
+// A relationship on one source descriptor is one loss however many shards
+// carry that rule — the same per-shard inflation, one field over.
+func TestGitHubRuleRelationshipsAreCountedOncePerSourceRule(t *testing.T) {
+ const n = GitHubMaxResultsPerRun + 5
+ results := make([]Result, 0, n)
+ for i := 0; i < n; i++ {
+ results = append(results, ghSastResult(
+ fmt.Sprintf("sast:%d", i), fmt.Sprintf("app/pkg%d/file%d.py", i%64, i), (i%900)+1))
+ }
+ log := ghOneRunLog(HalfSast, results)
+ log.Runs[0].Tool.Driver.Rules = []ReportingDescriptor{{
+ ID: "anvil.sqli.raw-concat",
+ Relationships: []ReportingDescriptorRelationship{{Target: ReportingDescriptorReference{ID: "89"}}},
+ }}
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if len(files) < 2 {
+ t.Fatalf("the fixture must shard to exercise the defect, got %d files", len(files))
+ }
+ if got := GitHubLossOf(files).StripCounts[GitHubStripRuleRelationships]; got != 1 {
+ t.Errorf("StripCounts[%s] = %d across %d shards, want 1: one source descriptor, one loss",
+ GitHubStripRuleRelationships, got, len(files))
+ }
+}
+
+// The one drop this file used to make silently: a second descriptor for a rule
+// id already emitted.
+func TestGitHubDuplicateRuleDescriptorIsCounted(t *testing.T) {
+ log := ghOneRunLog(HalfSast, []Result{ghSastResult("sast:1", "app/db.py", 414)})
+ log.Runs[0].Tool.Driver.Rules = []ReportingDescriptor{
+ {ID: "anvil.sqli.raw-concat", ShortDescription: &Message{Text: "first"}},
+ {ID: "anvil.sqli.raw-concat", ShortDescription: &Message{Text: "duplicate"}},
+ }
+
+ files, err := ProjectForGitHub(log)
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+ if got := GitHubLossOf(files).StripCounts[GitHubStripDuplicateRule]; got != 1 {
+ t.Errorf("StripCounts[%s] = %d, want 1; a duplicate descriptor cannot be carried "+
+ "and must not be dropped in silence", GitHubStripDuplicateRule, got)
+ }
+ for _, f := range files {
+ for _, run := range f.Log.Runs {
+ if n := len(run.Tool.Driver.Rules); n != 1 {
+ t.Errorf("a shard emitted %d descriptors for one rule id", n)
+ }
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 M4 — relatedLocations count against the locations cap
+// ---------------------------------------------------------------------------
+
+// `locations` was truncated at GitHubMaxLocationsPerResult and
+// `relatedLocations` was appended without limit, so a fan-out finding shipped
+// four times the cap. Neither reading of [S2] makes an asymmetry correct: if
+// related locations count, this was a cap violation; if they do not, the file
+// applied a documented cap to one of two arrays with nothing explaining why.
+func TestGitHubRelatedLocationsCountAgainstTheLocationsCap(t *testing.T) {
+ const fanOut = 3000
+
+ r := ghSastResult("sast:fanout", "app/db.py", 1)
+ for i := 0; i < fanOut; i++ {
+ loc := Location{PhysicalLocation: &PhysicalLocation{
+ ArtifactLocation: ArtifactLocation{URI: fmt.Sprintf("app/gen/f%04d.py", i)},
+ Region: &Region{StartLine: i + 1},
+ }}
+ r.Locations = append(r.Locations, loc)
+ r.RelatedLocations = append(r.RelatedLocations, loc)
+ }
+
+ files, err := ProjectForGitHub(ghOneRunLog(HalfSast, []Result{r}))
+ if err != nil {
+ t.Fatalf("ProjectForGitHub: %v", err)
+ }
+
+ var kept *GitHubResult
+ for i := range files {
+ for j := range files[i].Log.Runs[0].Results {
+ kept = &files[i].Log.Runs[0].Results[j]
+ }
+ }
+ if kept == nil {
+ t.Fatal("the fan-out result did not survive the projection")
+ }
+
+ total := len(kept.Locations) + len(kept.RelatedLocations)
+ t.Logf("locations=%d relatedLocations=%d total=%d cap=%d",
+ len(kept.Locations), len(kept.RelatedLocations), total, GitHubMaxLocationsPerResult)
+ if total > GitHubMaxLocationsPerResult {
+ t.Errorf("the result carries %d locations across both arrays, over the %d cap",
+ total, GitHubMaxLocationsPerResult)
+ }
+ // `locations` fills first: locations[0] is the primary and the whole
+ // projection's identity story rests on it.
+ if len(kept.Locations) != GitHubMaxLocationsPerResult {
+ t.Errorf("locations = %d, want the cap filled from locations first (%d)",
+ len(kept.Locations), GitHubMaxLocationsPerResult)
+ }
+ if len(kept.RelatedLocations) != 0 {
+ t.Errorf("relatedLocations = %d, want 0 once locations has taken the whole cap",
+ len(kept.RelatedLocations))
+ }
+
+ // The truncation is counted, not silent.
+ want := (1 + fanOut - GitHubMaxLocationsPerResult) + fanOut
+ if got := GitHubLossOf(files).StripCounts[GitHubStripLocationsOverCap]; got != want {
+ t.Errorf("StripCounts[%s] = %d, want %d (%d over the cap in locations, %d in relatedLocations)",
+ GitHubStripLocationsOverCap, got, want, 1+fanOut-GitHubMaxLocationsPerResult, fanOut)
+ }
+
+ // WithinCaps must ask the same question the builder answered, or the
+ // independent check certifies a narrower guarantee than the file promises.
+ over := GitHubSarifFile{
+ Name: "probe",
+ Log: GitHubSARIFLog{Runs: []GitHubRun{{Results: []GitHubResult{{
+ Locations: make([]GitHubLocation, GitHubMaxLocationsPerResult),
+ RelatedLocations: make([]GitHubLocation, 1),
+ }}}}},
+ }
+ if err := over.WithinCaps(); err == nil {
+ t.Errorf("WithinCaps accepted %d locations + %d relatedLocations against a %d cap",
+ GitHubMaxLocationsPerResult, 1, GitHubMaxLocationsPerResult)
+ }
+}
diff --git a/internal/record/sealing.go b/internal/record/sealing.go
index d86f7b5..db266ed 100644
--- a/internal/record/sealing.go
+++ b/internal/record/sealing.go
@@ -188,15 +188,142 @@ func TerminalHalfStatuses() []HalfStatus {
// sense — cleanly sealed, broken, out of clock, or never run.
func IsTerminalHalfStatus(s HalfStatus) bool { return inEnum(s, TerminalHalfStatuses()) }
-// IsReadableHalfStatus reports whether a consumer may read the half's
-// results. It is true for HalfStatusSealed and for nothing else.
+// IsReadableHalfStatus reports whether the half's STATUS arm of the read gate
+// is open. It is true for HalfStatusSealed and for nothing else.
//
// Written as a named predicate on purpose: `if status != HalfStatusRunning`
// and `if IsTerminalHalfStatus(status)` are both wrong here and both look
// plausible at a glance. A failed half is not a clean half; a skipped half
// has no results at all.
+//
+// IT IS HALF OF THE GATE, NOT THE GATE. Every caller that wants to know
+// whether a consumer may read a half's results must call HalfReadGate (or
+// HalfSeal.Readable, which is the same answer as a bool) — see the read-gate
+// section below for why calling this predicate alone is a bug that has now
+// been made four times.
func IsReadableHalfStatus(s HalfStatus) bool { return s == HalfStatusSealed }
+// ---------------------------------------------------------------------------
+// THE READ GATE — one predicate, one answer, and the reason there is only one
+// ---------------------------------------------------------------------------
+//
+// "May a consumer read this half's results?" is a TWO-ARM question:
+//
+// IsReadableHalfStatus(status) AND the audit has not expired
+//
+// Four independent authors have now derived that answer locally instead of
+// calling one gate, and each got a different arm wrong:
+//
+// CRITIQUE-02 M2 — ReadPacket checked neither arm.
+// CRITIQUE-02 M3 — Sealer.Inspect handed out HalfSeals with no state at all,
+// so Readable() said true on an audit ReadHalf refused.
+// CRITIQUE-03 B1 — the GitHub projection consulted neither arm and published
+// an unsealed half's results to a third party.
+// CRITIQUE-03 M1 — readpath.go's readOrder and ManifestFromLog checked the
+// status arm only, so an EXPIRED audit was fully readable and
+// handed a coding agent actionable task cards against a claim
+// window that had already closed.
+//
+// The pattern, not any one of those four, is the defect. So the question is
+// answered in exactly ONE function body — halfReadRefusal — and every other
+// spelling in this package is a thin wrapper over it:
+//
+// HalfReadGate the typed refusal, for a caller that must report WHY.
+// HalfSeal.Readable the same answer as a bool, for a caller that must branch.
+// Sealer.ReadHalf the in-memory consumer gate.
+// Sealer.ReadyForConsumption
+// the per-half form R.4's handoff rows key on.
+// readpath.go / taskcard.go / sarif_github.go
+// the record-side callers, via halfSealOfRun.
+//
+// THREE GUARDS WATCH THIS, and it took three rounds to get them to fail on
+// purpose. None of them is sufficient alone:
+//
+// TestEveryResultBearingEntryPointIsGated (readpath_test.go)
+// BEHAVIOUR. Runs every listed entry point against every state in which no
+// half may be read and fails if anything comes back. It is the only one
+// that executes the code, and it only covers entry points someone listed.
+//
+// TestResultReachingEntryPointsAreGated (readpath_test.go)
+// SOURCE REACHABILITY. Walks the package's own AST and fails when an
+// exported entry point can reach a half's results while nothing in its call
+// graph reaches this gate. It replaced a whitelist of return TYPES, which
+// did not name Result, []string or []byte and so let three leaks through.
+//
+// TestReadGateArmsAppearOnlyInsideTheGate (sealing_test.go)
+// ONE-ARM DETECTION. Fails when IsReadableHalfStatus is called, or a state
+// compared against StateExpired, or a status against HalfStatusSealed,
+// anywhere outside halfReadRefusal without an allowlisted reason. It
+// replaced a check that required BOTH arms in one body — which is why it
+// could not see CRITIQUE-03 M1, whose defect was one arm.
+//
+// The last two carry negative controls that re-introduce the historical
+// defects on every run, because a guard that has never been seen to fail has
+// not been tested. This package has now paid for that lesson three times.
+
+// halfReadRefusal returns the reason a consumer may NOT read this half's
+// results, or "" when the gate is open. It is THE definition of readability
+// and the only place the two arms are combined.
+//
+// The expiry arm is checked FIRST so that an expired audit holding a cleanly
+// sealed half reports the expiry — the fact the caller can act on — rather
+// than a status that is, on its own, fine.
+func halfReadRefusal(h HalfSeal) string {
+ if h.AuditState == StateExpired {
+ return "the claim timeout elapsed and the payload was dropped"
+ }
+ if !IsReadableHalfStatus(h.Status) {
+ return "this half has no readable results"
+ }
+ return ""
+}
+
+// HalfReadGate is the ONE read gate. It returns nil when a consumer may read
+// h's results, and a *ReadGateError naming the arm that refused when it may
+// not.
+//
+// Every refusal it returns satisfies errors.Is(err, ErrHalfNotSealed) and
+// carries the half, its status and the audit state, so a caller can report the
+// refusal instead of merely obeying it. auditID is used only to build that
+// message; pass "" when there is no audit context to name.
+//
+// Callers hold a HalfSeal because that is the value that already carries every
+// input the gate needs — status and audit state — and because reusing it means
+// there is no second vocabulary for "a half's readiness". A record-side caller
+// builds one with halfSealOfRun.
+func HalfReadGate(auditID string, h HalfSeal) error {
+ reason := halfReadRefusal(h)
+ if reason == "" {
+ return nil
+ }
+ return &ReadGateError{
+ AuditID: auditID, Half: h.Half, Status: h.Status, State: h.AuditState,
+ Reason: reason,
+ }
+}
+
+// halfSealOfRun projects one run of an ASSEMBLED RECORD onto the HalfSeal the
+// gate takes: the per-half seal from `run.properties` and the audit-level
+// lifecycle state from `sarifLog.properties`.
+//
+// It exists so that no reader of a record ever has to remember that the second
+// arm of the gate lives on a different object from the first. That is exactly
+// the mistake CRITIQUE-03 M1 records: `run.Properties.Status` is right there
+// and `l.Properties.State` is one dereference further away, so three of four
+// call sites reached for the near one and stopped.
+func halfSealOfRun(l *SARIFLog, run *Run) HalfSeal {
+ var state State
+ if l != nil {
+ state = l.Properties.State
+ }
+ return HalfSeal{
+ Half: run.Properties.Half,
+ Status: run.Properties.Status,
+ SealedAt: copyTime(run.Properties.SealedAt),
+ AuditState: state,
+ }
+}
+
// ---------------------------------------------------------------------------
// Value types
// ---------------------------------------------------------------------------
@@ -241,13 +368,11 @@ type HalfSeal struct {
// Readable reports whether a consumer may read this half's results.
//
-// It is the same predicate ReadHalf enforces, in both arms: the half's status
-// must be exactly HalfStatusSealed AND the audit must not have expired, whose
-// payload the reaper has dropped. TestInspectAgreesWithReadHalfOnEveryState
-// asserts the two never disagree for any (state, status) pair.
-func (h HalfSeal) Readable() bool {
- return IsReadableHalfStatus(h.Status) && h.AuditState != StateExpired
-}
+// It is HalfReadGate as a bool — literally the same function body — so a
+// caller that branches and a caller that reports a typed refusal can never
+// disagree. TestInspectAgreesWithReadHalfOnEveryState asserts the two never
+// disagree for any (state, status) pair.
+func (h HalfSeal) Readable() bool { return halfReadRefusal(h) == "" }
// DastOutcome is what the DAST half (or its absence) reports, and the sole
// input from which the audit-level DastStatus is derived.
@@ -795,10 +920,13 @@ func (s *Sealer) ReadyForConsumption(auditID string) (sastReady, dastReady bool)
defer s.mu.Unlock()
a, ok := s.audits[auditID]
- if !ok || a.state == StateExpired {
+ if !ok {
return false, false
}
- return IsReadableHalfStatus(a.sastStatus), IsReadableHalfStatus(a.dastStatus)
+ // The SAME gate ReadHalf enforces, asked twice. Asking
+ // IsReadableHalfStatus here and handling expiry separately is how the two
+ // answers drift apart; see the read-gate section above.
+ return a.halfSeal(HalfSast).Readable(), a.halfSeal(HalfDast).Readable()
}
// ReadHalf is the consumer's read gate. It returns the half's seal only when
@@ -831,17 +959,8 @@ func (s *Sealer) ReadHalf(auditID string, half Half) (HalfSeal, error) {
}
seal := a.halfSeal(half)
- if a.state == StateExpired {
- return HalfSeal{}, &ReadGateError{
- AuditID: auditID, Half: half, Status: seal.Status, State: a.state,
- Reason: "the claim timeout elapsed and the payload was dropped",
- }
- }
- if !IsReadableHalfStatus(seal.Status) {
- return HalfSeal{}, &ReadGateError{
- AuditID: auditID, Half: half, Status: seal.Status, State: a.state,
- Reason: "this half has no readable results",
- }
+ if err := HalfReadGate(auditID, seal); err != nil {
+ return HalfSeal{}, err
}
return seal, nil
}
diff --git a/internal/record/sealing_test.go b/internal/record/sealing_test.go
index d06102e..68aaf41 100644
--- a/internal/record/sealing_test.go
+++ b/internal/record/sealing_test.go
@@ -3,6 +3,13 @@ package record
import (
"errors"
"fmt"
+ "go/ast"
+ "go/parser"
+ "go/token"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
"sync"
"testing"
"time"
@@ -1272,3 +1279,418 @@ func TestInspectAgreesWithReadHalfOnEveryState(t *testing.T) {
}
}
}
+
+// ---------------------------------------------------------------------------
+// CRITIQUE-03 B1/M1 — one gate, every spelling, every (state, status) pair
+// ---------------------------------------------------------------------------
+
+// TestEverySpellingOfTheReadGateAgrees drives the whole cross product of
+// anvil/state and anvil/status through every exported way this package answers
+// "may a consumer read this half's results?" and asserts they never disagree.
+//
+// The four historical bypasses (sealing.go's read-gate section) were all one
+// shape: a second answer to a question that already had one. This test is what
+// makes a second answer visible immediately, without waiting for a critic to
+// find the consumer that acted on it.
+func TestEverySpellingOfTheReadGateAgrees(t *testing.T) {
+ for _, state := range StateValues() {
+ for _, status := range HalfStatusValues() {
+ seal := HalfSeal{Half: HalfSast, Status: status, AuditState: state}
+
+ // (1) The bool spelling, which is what a caller branches on.
+ readable := seal.Readable()
+
+ // (2) The typed spelling, which is what a caller reports.
+ err := HalfReadGate("audit-1", seal)
+ if (err == nil) != readable {
+ t.Errorf("state=%q status=%q: HalfReadGate says %v but Readable() says %t",
+ state, status, err, readable)
+ }
+ if err != nil {
+ if !errors.Is(err, ErrHalfNotSealed) {
+ t.Errorf("state=%q status=%q: refusal does not match ErrHalfNotSealed: %v",
+ state, status, err)
+ }
+ var rge *ReadGateError
+ if !errors.As(err, &rge) {
+ t.Errorf("state=%q status=%q: refusal is %T, want a *ReadGateError", state, status, err)
+ } else {
+ if rge.Status != status || rge.State != state || rge.Half != HalfSast {
+ t.Errorf("state=%q status=%q: refusal reports half=%q status=%q state=%q",
+ state, status, rge.Half, rge.Status, rge.State)
+ }
+ if rge.Reason == "" {
+ t.Errorf("state=%q status=%q: refusal carries no reason", state, status)
+ }
+ }
+ }
+
+ // (3) The record-side spelling, built from a run and its envelope.
+ // This is the projection readpath.go, taskcard.go and
+ // sarif_github.go all go through, and the one CRITIQUE-03 found
+ // two callers reaching around.
+ l := &SARIFLog{
+ Properties: AuditProperties{AuditID: "audit-1", State: state},
+ Runs: []Run{{Properties: RunProperties{Half: HalfSast, Status: status}}},
+ }
+ if got := halfSealOfRun(l, &l.Runs[0]).Readable(); got != readable {
+ t.Errorf("state=%q status=%q: halfSealOfRun(...).Readable() = %t, want %t",
+ state, status, got, readable)
+ }
+
+ // (4) The two arms are BOTH load-bearing, and neither alone is the
+ // gate. This is the assertion that fails if someone "simplifies"
+ // the gate back to one of its halves.
+ statusArm := IsReadableHalfStatus(status)
+ if statusArm && state == StateExpired && readable {
+ t.Errorf("state=%q status=%q: the status arm alone opened the gate; "+
+ "an expired audit's payload has been dropped", state, status)
+ }
+ if !statusArm && readable {
+ t.Errorf("state=%q status=%q: the gate opened on a half that is not sealed",
+ state, status)
+ }
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// THE HALF-GATE DETECTOR — what the previous version of this test could not see
+// ---------------------------------------------------------------------------
+//
+// The previous version counted function bodies that mentioned BOTH
+// `StateExpired` AND `IsReadableHalfStatus`, and required exactly one such
+// body (halfReadRefusal). It could not detect the defect it was written to
+// detect. CRITIQUE-03 M1's original bug, character for character, was
+//
+// if !IsReadableHalfStatus(run.Properties.Status) { continue }
+//
+// in readOrder — ONE arm. A body carrying one arm never matched the
+// two-literal test, so the count stayed at 1 and the suite stayed green while
+// an expired audit handed out nine task cards.
+//
+// The hazard is not a second COMPLETE gate. Nobody writes one of those; a
+// complete gate is correct. The hazard is a HALF-gate: a site that decides
+// readability from one arm and never learns about the other. So this test
+// looks for the arms THEMSELVES, anywhere outside the one function that is
+// allowed to combine them:
+//
+// IsReadableHalfStatus(...) the status arm, called
+// x == / != StateExpired the expiry arm, hand-rolled
+// x == / != HalfStatusSealed the status arm, hand-rolled (which is
+// what IsReadableHalfStatus itself is)
+//
+// Finding any of them outside halfReadRefusal is the signal. Not every hit is
+// a bug — the Sealer's lifecycle transitions legitimately ask "is this audit
+// expired?" before accepting a seal, and contract.go's validator legitimately
+// asks "is this half sealed?" before requiring a sealedAt — but every hit must
+// have been LOOKED AT, and the allowlist below is that record. A site is
+// keyed by file, function AND arm, so adding the missing arm to a function
+// that was cleared for one of them still trips.
+
+// gateArmSite is one place an arm of the read gate is spelled.
+type gateArmSite struct {
+ file string // base filename
+ fn string // "Func" or "Recv.Method"
+ arm string // the sentinel spelled there
+}
+
+func (s gateArmSite) key() string { return s.file + ":" + s.fn + ":" + s.arm }
+
+// gateArmSentinels are the three spellings of half a gate. IsReadableHalfStatus
+// is matched as a CALL; the other two as comparisons (`==`, `!=`, or a switch
+// case), never as a bare mention, because `string(HalfStatusSealed)` inside an
+// error message is prose about the gate and not a use of it.
+var gateArmSentinels = map[string]bool{
+ "IsReadableHalfStatus": true,
+ "StateExpired": true,
+ "HalfStatusSealed": true,
+}
+
+// gateArmAllowlist names the sites where an arm of the read gate is spelled
+// outside halfReadRefusal for a reason that is not a readability decision.
+//
+// Each entry is file:function:arm. Every one of them has been read; the reason
+// says what the site is deciding INSTEAD of readability. The test fails if an
+// entry stops matching a real site, so the list cannot outlive the code it
+// describes.
+func gateArmAllowlist() map[string]string {
+ return map[string]string{
+ // ---- lifecycle: may this audit still accept writes? --------------
+ "sealing.go:Sealer.RecordDastOutcome:StateExpired": "refuses an outcome update on a " +
+ "terminal audit. It decides whether the SEALER accepts a WRITE, not whether a " +
+ "consumer may read; the read direction is ReadHalf, which calls the gate.",
+ "sealing.go:Sealer.SealHalf:StateExpired": "refuses a seal on a terminal audit — the " +
+ "same write-side question as RecordDastOutcome.",
+ "sealing.go:Sealer.Consume:StateExpired": "refuses consumption of an expired audit. " +
+ "Consumption is a state transition on the audit, not a read of a half.",
+ "sealing.go:Sealer.ExpireIfDue:StateExpired": "the expiry transition itself: already " +
+ "expired means there is nothing to do. This is where StateExpired is PRODUCED.",
+
+ // ---- lifecycle: stamping and deriving, not gating ----------------
+ "sealing.go:Sealer.SealHalf:HalfStatusSealed": "stamps SealedAt only for a clean seal, " +
+ "per contract.go's rule that sealedAt is null unless the status is sealed. It is " +
+ "writing the field the gate later reads, not reading it.",
+ "sealing.go:DeriveDastStatus:HalfStatusSealed": "derives the audit-level DastStatus " +
+ "from the DAST half's outcome. It is a projection of the half's status onto a " +
+ "different enum; DastStatus is not a readability answer, and R.6 keeps " +
+ "'completed_clean' distinct from 'readable' on purpose.",
+
+ // ---- the status arm's own definition ------------------------------
+ "sealing.go:IsReadableHalfStatus:HalfStatusSealed": "IS the comparison, named. This is " +
+ "the one site allowed to spell the status arm as a literal, and sealing.go's own " +
+ "doc comment on it says in capitals that it is HALF OF THE GATE, NOT THE GATE. " +
+ "Every other site asks HalfReadGate, which asks halfReadRefusal, which asks this.",
+
+ // ---- the producer-side validator ---------------------------------
+ "contract.go:SARIFLog.validateStateAgainstHalves:HalfStatusSealed": "derives which " +
+ "anvil/state the halves imply, so the envelope and the runs cannot disagree. It " +
+ "runs on the PRODUCER side, on records no half of which may be readable yet.",
+ "contract.go:SARIFLog.validateStateAgainstHalves:StateExpired": "exempts the two " +
+ "terminal states from that derivation, because consumed and expired are not " +
+ "derivable from the halves. Same validator, same producer side.",
+ "contract.go:Run.validate:HalfStatusSealed": "enforces contract.go's sealedAt " +
+ "invariant — required when sealed, null otherwise — so that 'never cleanly " +
+ "sealed' and 'we forgot to write it' cannot be the same observation. It is a " +
+ "well-formedness check on one run, not a decision about a consumer.",
+ }
+}
+
+// TestReadGateArmsAppearOnlyInsideTheGate reads the package as data and reports
+// every site outside halfReadRefusal that spells an arm of the read gate. See
+// the section above for why it looks for ARMS and not for whole gates.
+//
+// It is a source assertion of the same kind as TestPatentRiskIsFlaggedInSource,
+// and it exists because the defect this package keeps re-acquiring is not a
+// wrong answer but a SECOND, PARTIAL answer. A behavioural test cannot see one
+// until some consumer calls it; this can.
+func TestReadGateArmsAppearOnlyInsideTheGate(t *testing.T) {
+ fset := token.NewFileSet()
+ pkgs, err := parser.ParseDir(fset, ".", func(fi os.FileInfo) bool {
+ return !strings.HasSuffix(fi.Name(), "_test.go")
+ }, 0)
+ if err != nil {
+ t.Fatalf("parsing the package source: %v", err)
+ }
+ if len(pkgs) == 0 {
+ t.Fatal("parsed no packages; this test asserts nothing unless it reads the source")
+ }
+
+ allow := gateArmAllowlist()
+ matched := map[string]bool{}
+ gateArms := map[string]bool{}
+ sites := 0
+
+ for _, pkg := range pkgs {
+ for path, file := range pkg.Files {
+ base := filepath.Base(path)
+ for _, decl := range file.Decls {
+ fn, ok := decl.(*ast.FuncDecl)
+ if !ok || fn.Body == nil {
+ continue
+ }
+ name := fn.Name.Name
+ if fn.Recv != nil && len(fn.Recv.List) == 1 {
+ if recv := gateBaseTypeName(fn.Recv.List[0].Type); recv != "" {
+ name = recv + "." + name
+ }
+ }
+ isTheGate := base == "sealing.go" && name == "halfReadRefusal"
+
+ for _, arm := range gateArmsIn(fn) {
+ site := gateArmSite{file: base, fn: name, arm: arm}
+ if isTheGate {
+ gateArms[arm] = true
+ continue
+ }
+ sites++
+ if reason, ok := allow[site.key()]; ok {
+ matched[site.key()] = true
+ if strings.TrimSpace(reason) == "" {
+ t.Errorf("%s is allowlisted with an empty reason", site.key())
+ }
+ continue
+ }
+ t.Errorf("%s spells %q outside the read gate.\n"+
+ " ONE ARM IS NOT THE GATE. Readability is\n"+
+ " IsReadableHalfStatus(status) AND the audit has not expired,\n"+
+ " and CRITIQUE-03 M1 was exactly this: readOrder asked the status arm\n"+
+ " alone, so an expired audit handed a coding agent nine task cards.\n"+
+ " Call HalfReadGate (or HalfSeal.Readable), or — if this site is\n"+
+ " deciding something OTHER than readability — add it to\n"+
+ " gateArmAllowlist with the reason.",
+ site.key(), arm)
+ }
+ }
+ }
+ }
+
+ // The gate must still BE the gate: both arms, in the one body.
+ for _, arm := range []string{"IsReadableHalfStatus", "StateExpired"} {
+ if !gateArms[arm] {
+ t.Errorf("halfReadRefusal no longer spells %q. It is the ONE place both arms are "+
+ "combined; if it has lost one, the gate is now half a gate and this test is "+
+ "reporting on a function that no longer decides anything.", arm)
+ }
+ }
+
+ // A stale exemption is how the next real one gets waved through.
+ for key := range allow {
+ if !matched[key] {
+ t.Errorf("gateArmAllowlist names %q, which is not a site in the current source. "+
+ "Delete the entry — an allowlist that outlives the code it describes is a "+
+ "standing exemption nobody has read.", key)
+ }
+ }
+ t.Logf("read-gate arms: %d sites outside halfReadRefusal, %d allowlisted", sites, len(matched))
+}
+
+// gateHalfGateProbeSource is the negative control for the detector above: two
+// half-gates and one innocent function, as source text.
+//
+// readOrderStatusArmOnly is CRITIQUE-03 M1's original defect, character for
+// character — the status arm alone, in readOrder, which the previous
+// two-literal test could not see because one arm never matched a check that
+// required both.
+//
+// expiryArmOnly is the mirror-image half-gate nobody has written yet.
+//
+// prosePloneNamesTheArms is the false-positive control: it TALKS about both
+// arms in comments and puts one in an error string, and must not fire. The
+// detector matching on the AST rather than on text is the whole reason it can
+// tell those apart, and sealing.go's header would trip a text search on every
+// run.
+const gateHalfGateProbeSource = `package record
+
+import "fmt"
+
+func readOrderStatusArmOnly(l *SARIFLog) int {
+ n := 0
+ for ri := range l.Runs {
+ if !IsReadableHalfStatus(l.Runs[ri].Properties.Status) {
+ continue
+ }
+ n += len(l.Runs[ri].Results)
+ }
+ return n
+}
+
+func expiryArmOnly(l *SARIFLog) bool {
+ if l.Properties.State == StateExpired {
+ return false
+ }
+ return true
+}
+
+// prosePloneNamesTheArms explains that readability is IsReadableHalfStatus AND
+// the audit is not StateExpired, and that HalfStatusSealed is the only readable
+// status. It decides nothing.
+func prosePloneNamesTheArms() error {
+ return fmt.Errorf("anvil/status must be %q", string(HalfStatusSealed))
+}
+`
+
+// TestTheHalfGateDetectorCatchesTheDefectItsPredecessorMissed is the negative
+// control for TestReadGateArmsAppearOnlyInsideTheGate.
+//
+// It runs gateArmsIn — the same function the detector calls — over three
+// synthetic functions and asserts the two half-gates are seen and the prose is
+// not. Without it, this test would be another guard that has never been
+// observed to fail, which is exactly how the previous one shipped: it counted
+// bodies carrying BOTH arms, so the one-arm defect it was written for went
+// through it twice.
+func TestTheHalfGateDetectorCatchesTheDefectItsPredecessorMissed(t *testing.T) {
+ fset := token.NewFileSet()
+ file, err := parser.ParseFile(fset, "zz_half_gate_probe.go", gateHalfGateProbeSource, parser.ParseComments)
+ if err != nil {
+ t.Fatalf("parsing the synthetic half-gate file: %v", err)
+ }
+
+ want := map[string][]string{
+ "readOrderStatusArmOnly": {"IsReadableHalfStatus"},
+ "expiryArmOnly": {"StateExpired"},
+ "prosePloneNamesTheArms": nil,
+ }
+ seen := map[string]bool{}
+ allow := gateArmAllowlist()
+
+ for _, decl := range file.Decls {
+ fn, ok := decl.(*ast.FuncDecl)
+ if !ok || fn.Body == nil {
+ continue
+ }
+ exp, ok := want[fn.Name.Name]
+ if !ok {
+ t.Fatalf("the probe file grew a function %q the control does not check", fn.Name.Name)
+ }
+ seen[fn.Name.Name] = true
+ got := gateArmsIn(fn)
+ if strings.Join(got, ",") != strings.Join(exp, ",") {
+ if len(exp) == 0 {
+ t.Errorf("%s spells no arm of the gate but the detector reported %v; a comment "+
+ "or an error message discussing the gate is not a decision about it",
+ fn.Name.Name, got)
+ } else {
+ t.Errorf("%s is a half-gate spelling %v, but the detector reported %v. This is "+
+ "the exact shape of CRITIQUE-03 M1, and a detector that cannot see it is "+
+ "the detector this one replaced.", fn.Name.Name, exp, got)
+ }
+ }
+ // And a hit must actually be REPORTED, not silently pre-cleared.
+ for _, arm := range got {
+ key := gateArmSite{file: "readpath.go", fn: fn.Name.Name, arm: arm}.key()
+ if _, ok := allow[key]; ok {
+ t.Errorf("%s is already in gateArmAllowlist, so the control proves nothing", key)
+ }
+ }
+ }
+ for name := range want {
+ if !seen[name] {
+ t.Errorf("the control never examined %q; the probe file did not parse as expected", name)
+ }
+ }
+}
+
+// gateArmsIn returns the arms of the read gate spelled in fn's body, sorted
+// and deduplicated.
+//
+// Matching is on the AST, not on text, so a comment naming an arm is never a
+// use of one — sealing.go's header discusses both arms at length, deliberately,
+// and prose is not a decision. `IsReadableHalfStatus` counts as a CALL; the two
+// enum literals count only in a comparison (`==`, `!=`) or a switch case, so
+// `string(HalfStatusSealed)` inside an error message does not fire.
+func gateArmsIn(fn *ast.FuncDecl) []string {
+ found := map[string]bool{}
+
+ note := func(e ast.Expr) {
+ if id, ok := e.(*ast.Ident); ok && gateArmSentinels[id.Name] {
+ found[id.Name] = true
+ }
+ }
+
+ ast.Inspect(fn.Body, func(n ast.Node) bool {
+ switch e := n.(type) {
+ case *ast.CallExpr:
+ if id, ok := e.Fun.(*ast.Ident); ok && id.Name == "IsReadableHalfStatus" {
+ found[id.Name] = true
+ }
+ case *ast.BinaryExpr:
+ if e.Op == token.EQL || e.Op == token.NEQ {
+ note(e.X)
+ note(e.Y)
+ }
+ case *ast.CaseClause:
+ for _, expr := range e.List {
+ note(expr)
+ }
+ }
+ return true
+ })
+
+ out := make([]string, 0, len(found))
+ for arm := range found {
+ out = append(out, arm)
+ }
+ sort.Strings(out)
+ return out
+}
diff --git a/internal/record/taskcard.go b/internal/record/taskcard.go
new file mode 100644
index 0000000..5f71644
--- /dev/null
+++ b/internal/record/taskcard.go
@@ -0,0 +1,1163 @@
+// Tier 1 of the read path: the task card the coding agent actually reads
+// (step R.13, with readpath.go).
+//
+// # A task card is DERIVED. The record is authoritative.
+//
+// This is the single most important sentence in this file. research/18:
+// "One self-contained JSON per finding, *derived* from the SARIF (the SARIF
+// stays authoritative)." A card is a projection built for one consumer's
+// context window. It is not a second copy of the truth, it is not written
+// back, and it has no independent lifetime:
+//
+// - Where a card and the record disagree, THE RECORD WINS and the card is a
+// stale projection to be rebuilt. TaskCard.CheckAgainstRecord reports the
+// disagreements rather than leaving a reader to find them.
+// - A card is never the input to another card. Rebuilding from the record is
+// always correct; rebuilding from a card is always a lossy copy of a lossy
+// copy.
+// - The agent's patch is written back to `result.fixes` in the RECORD
+// (SARIF §3.27.30) — TaskCard.WriteBackTo names the pointer — never to the
+// card.
+//
+// The one direction in which a card may legally differ from the record is
+// PERMISSIVENESS, and only downward: a card may withhold an action the record
+// would have allowed, and may never grant one the record does not. That is
+// what makes the three clamps below safe to implement here.
+//
+// # Three clamps, one rule
+//
+// Each is a gate the record's own validator already enforces, re-enforced here
+// because THE CARD IS WHAT THE AGENT RECEIVES and a malformed producer, a
+// hand-edited row or a future column default must not be able to reach past
+// the producer's gate into the agent's context.
+//
+// 1. THE HOST GATE — RemediableByAgent, below.
+// 2. THE VERIFIED GATE — cardCorrelation clamps `verified` against
+// CorrelationSignal.SufficientForVerified via correlation.go's own
+// verificationOf, never a second copy of the rule. CRITIQUE-03 m2: the
+// host gate was enforced three times and this one, the same class of S7
+// gate, was taken on trust.
+// 3. THE BORROWED LOCUS — cardActionable withholds the action from a cluster
+// member whose file and line came from its peer. See cardActionable for
+// the whole argument; the short form is that one defect must not become
+// two patch tasks, and that a locus the finding did not observe must not
+// be presented as one it did.
+//
+// All three are WITHHOLDINGS, so CheckAgainstRecord does not report them.
+// GroupID would be the mechanism that collapses a duplicated cluster task
+// downstream, and it is RESERVED for the consumption pipeline (contract.go),
+// so nothing collapses them here — which is why clamp 3 is a clamp and not a
+// note for a later step.
+//
+// # The host gate
+//
+// plan/00-SPINE.md S7 makes the host agent read-only — "no package manager in
+// a mutating mode, not behind a flag" — so `remediable_by_agent` is false for
+// every host finding. contract.go's Validate() enforces it on the record and
+// internal/store enforces it with a CHECK constraint.
+//
+// This file enforces it a THIRD time, on the card, because the card is what
+// the agent receives. The two upstream gates protect the record; if a record
+// reaches this package with a host finding marked remediable — a malformed
+// producer, a hand-edited row, a future column default — the card must still
+// not hand the agent a task it is forbidden to perform. RemediableByAgent is
+// clamped to false and the reason is recorded in ActionBlockers.
+//
+// # What a card must carry
+//
+// research/24-coding-agent-consumption.md's non-negotiable handoff fields,
+// "because there is no orchestrator to compute them later": `finding_id`,
+// `fingerprint.primary_location_line_hash`, `fingerprint.region_sha256`,
+// `evidence_class`, `dast.reproduction`, `risk.*`, `locus.*`,
+// `advisory_excerpt` (<=800 tokens) and `group_id`. Every one has a field
+// below, and TestCardCarriesTheNonNegotiableFields asserts it.
+package record
+
+import (
+ "fmt"
+ "sort"
+ "strconv"
+ "strings"
+)
+
+// ---------------------------------------------------------------------------
+// The card
+// ---------------------------------------------------------------------------
+
+// TaskCard is Tier 1: one self-contained JSON per finding, carrying
+// everything needed to write the patch with no further lookups, inside
+// MaxTier1CardTokens.
+type TaskCard struct {
+ CardVersion string `json:"cardVersion"`
+ AuditID string `json:"auditId"`
+ FindingID string `json:"findingId"`
+
+ // Bucket is which of DefaultReadOrder()'s buckets this card came from,
+ // and Position is its 0-based index in the whole read order. A consumer
+ // that receives cards out of band can verify the order it was handed.
+ Bucket string `json:"bucket"`
+ Position int `json:"position"`
+
+ Half Half `json:"half"`
+ ClusterID string `json:"clusterId,omitempty"`
+
+ // GroupID is research/24's `group_id`. It is RESERVED and assigned by the
+ // coding-agent consumption pipeline, not here (contract.go,
+ // ResultProperties.GroupID); the card carries whatever the record carries,
+ // which on a freshly assembled record is empty.
+ GroupID string `json:"groupId,omitempty"`
+
+ Rank float64 `json:"rank"`
+ EvidenceClass EvidenceClass `json:"evidenceClass"`
+ Verdict Verdict `json:"verdict"`
+ Confidence float64 `json:"confidence"`
+
+ // ConsumptionClass is DERIVED here; see deriveConsumptionClass. The
+ // authoritative value is `handoff.consumption_class` (R.4).
+ ConsumptionClass ConsumptionClass `json:"consumptionClass"`
+
+ // RemediableByAgent is the record's value CLAMPED: never true for a host
+ // finding. See this file's header.
+ RemediableByAgent bool `json:"remediableByAgent"`
+
+ // Actionable is the single gate a consumer should branch on: the agent may
+ // propose a patch for this finding. ActionBlockers is why not, when not —
+ // never empty when Actionable is false, so "not actionable" is never
+ // unexplained.
+ Actionable bool `json:"actionable"`
+ ActionBlockers []string `json:"actionBlockers,omitempty"`
+
+ Task string `json:"task"`
+ Rule *CardRule `json:"rule,omitempty"`
+
+ Fingerprint CardFingerprint `json:"fingerprint"`
+ Locus CardLocus `json:"locus"`
+
+ Static *CardStatic `json:"static,omitempty"`
+ Dynamic *CardDynamic `json:"dynamic,omitempty"`
+ Advisory *CardAdvisory `json:"advisory,omitempty"`
+ Risk *Risk `json:"risk,omitempty"`
+ Correlation *CardCorrelation `json:"correlation,omitempty"`
+ Constraints *PatchContext `json:"constraints,omitempty"`
+
+ // Trust classifies the card's own strings. See CardTrust.
+ Trust CardTrust `json:"trust"`
+
+ // Spills names everything this card moved to a Tier-2 blob, either
+ // because it exceeded an inline cap or to stay inside the token budget.
+ Spills []TierSpill `json:"spills,omitempty"`
+
+ // Override is non-nil only when the card exceeded its budget after every
+ // shrink step AND Reader.AllowOversizeTier1 explicitly authorised it.
+ Override *BudgetOverride `json:"budgetOverride,omitempty"`
+
+ // WriteBackTo is the RFC 6901 pointer, from the sarifLog root, of the
+ // result's `fixes` array. plan/00-SPINE.md S7: "Never auto-merge. Propose
+ // only." The proposal lands in the record, not in the card.
+ WriteBackTo string `json:"writeBackTo"`
+
+ Bytes int `json:"cardBytes"`
+ Tokens int `json:"cardTokens"`
+
+ // Blobs are the Tier-2 bytes this card spilled, keyed by reference. NOT
+ // serialised, for the same reason Manifest.Blobs is not.
+ Blobs map[string][]byte `json:"-"`
+}
+
+// CardRule is the rule that fired, as the agent needs to understand it.
+type CardRule struct {
+ ID string `json:"id"`
+ Name string `json:"name,omitempty"`
+ Description string `json:"description,omitempty"`
+ HelpURI string `json:"helpUri,omitempty"`
+}
+
+// CardFingerprint carries research/24's three non-negotiable identity fields.
+// The keys they live under are contract.go's, never spelled by hand.
+type CardFingerprint struct {
+ // AnvilFindingID is the full, never-truncated anvil-fp/v1 digest.
+ AnvilFindingID string `json:"anvilFindingId"`
+ // PrimaryLocationLineHash is the only partial fingerprint GitHub reads.
+ PrimaryLocationLineHash string `json:"primaryLocationLineHash,omitempty"`
+ // RegionSha256 is research/24's `fingerprint.region_sha256`. It is
+ // optional on the wire — CONTRACT.md deviation 2 leaves the decision to
+ // populate it open — so the card carries it when the record has it and
+ // says nothing when it does not, rather than inventing a value.
+ RegionSha256 string `json:"regionSha256,omitempty"`
+}
+
+// CardLocus is research/24's `locus.*`. Path, line range and enclosing symbol
+// are read out of the SARIF-native slots (Locus in the property bag carries
+// only ProximityClass, precisely so there is one source of truth for them).
+type CardLocus struct {
+ Path string `json:"path,omitempty"`
+ StartLine int `json:"startLine,omitempty"`
+ EndLine int `json:"endLine,omitempty"`
+ EnclosingSymbol string `json:"enclosingSymbol,omitempty"`
+ ProximityClass string `json:"proximityClass,omitempty"`
+
+ // BorrowedFrom names the cluster peer this locus came from, and is set
+ // exactly when this finding did not observe a file and line of its own.
+ //
+ // It is the difference between OBSERVED and INFERRED, and it is on the
+ // card because the card is what the agent reads: a DAST finding's locus is
+ // the correlation's conclusion about where the defect lives, not something
+ // the dynamic probe saw. Empty means the finding observed its own locus.
+ // A card with this set is never Actionable; see cardActionable.
+ BorrowedFrom string `json:"borrowedFrom,omitempty"`
+}
+
+// CardStatic is the static evidence: the code the agent edits.
+type CardStatic struct {
+ FindingID string `json:"findingId"`
+ File string `json:"file,omitempty"`
+ StartLine int `json:"startLine,omitempty"`
+ EndLine int `json:"endLine,omitempty"`
+ Symbol string `json:"symbol,omitempty"`
+
+ // Code is the defect region's snippet; Context is the surrounding region
+ // the agent needs to patch it (SARIF §3.29.4 / §3.29.5). Both are
+ // VERBATIM TARGET-REPO SOURCE and therefore untrusted — see CardTrust.
+ Code string `json:"code,omitempty"`
+ Context *CardContext `json:"context,omitempty"`
+
+ // TaintPath is the code flow flattened to one line per step.
+ TaintPath []string `json:"taintPath,omitempty"`
+
+ Confidence float64 `json:"confidence"`
+ Reasoning string `json:"reasoning,omitempty"`
+}
+
+// CardContext is the context region around the defect.
+type CardContext struct {
+ StartLine int `json:"startLine,omitempty"`
+ EndLine int `json:"endLine,omitempty"`
+ Text string `json:"text,omitempty"`
+}
+
+// CardDynamic is research/24's `dast.reproduction`: the replayable request and
+// what it produced, which doubles as the accept oracle for the fix.
+//
+// plan/00-SPINE.md S7: only a reproduction that now FAILS earns "verified
+// fixed", so Env is carried — a replay under a different sanitizer or ASLR
+// setting is not the same experiment.
+type CardDynamic struct {
+ FindingID string `json:"findingId"`
+
+ Method string `json:"method,omitempty"`
+ URL string `json:"url,omitempty"`
+
+ // RequestBody is capped at MaxInlineRequestBodyBytes and ResponseExcerpt
+ // at MaxInlineResponseBodyBytes — R.8's caps, restated here because the
+ // card is a second place the bytes could be inlined. The remainder is a
+ // Tier-2 blob named in TaskCard.Spills.
+ RequestBody string `json:"requestBody,omitempty"`
+ StatusCode int `json:"statusCode,omitempty"`
+ ResponseExcerpt string `json:"responseExcerpt,omitempty"`
+
+ // ResponseBodyRef is the record's own `sha256:` reference to the full
+ // masked response body, when it has one.
+ ResponseBodyRef string `json:"responseBodyRef,omitempty"`
+
+ BaselineStatusCode int `json:"baselineStatusCode,omitempty"`
+
+ Payload string `json:"payload,omitempty"`
+ PayloadEncoding string `json:"payloadEncoding,omitempty"`
+ InjectionPoint string `json:"injectionPoint,omitempty"`
+ ObservedSignal EvidenceSignal `json:"observedSignal,omitempty"`
+ // Observed is the regex-extracted evidence span, never a raw body
+ // (plan/00-SPINE.md S7).
+ Observed string `json:"observed,omitempty"`
+
+ Steps []string `json:"steps,omitempty"`
+ Curl string `json:"curl,omitempty"`
+ ExpectedAfterFix *ReproExpectation `json:"expectedAfterFix,omitempty"`
+ Env *ReproEnv `json:"env,omitempty"`
+ SideEffects string `json:"sideEffects,omitempty"`
+
+ Confidence float64 `json:"confidence"`
+}
+
+// CardAdvisory is the advisory context, with the excerpt capped at
+// research/24's <=800 tokens.
+type CardAdvisory struct {
+ Taxa []string `json:"taxa,omitempty"`
+ IDs []string `json:"ids,omitempty"`
+ CveIDs []string `json:"cveIds,omitempty"`
+ SourceFeed string `json:"sourceFeed,omitempty"`
+ LicenseSpdx string `json:"licenseSpdx,omitempty"`
+ AsOf string `json:"asOf,omitempty"`
+ StalenessSeconds int `json:"stalenessSeconds"`
+ // ParseDegraded means the feed parsed with loss. A consumer must
+ // down-weight, not silently trust, degraded context.
+ ParseDegraded bool `json:"parseDegraded"`
+ Excerpt string `json:"excerpt,omitempty"`
+}
+
+// CardCorrelation is the LINK, never a merge. Peers names the findings this
+// one is linked to; each of them has its own card.
+type CardCorrelation struct {
+ ClusterID string `json:"clusterId"`
+ Role Half `json:"role"`
+ Peers []string `json:"peers,omitempty"`
+ Signals []string `json:"signals,omitempty"`
+
+ // PeersUnreadable is the subset of Peers for which NO card exists, because
+ // the peer's half did not pass the read gate. It is a subset, not a
+ // removal: the link is a fact recorded on this result and deleting it
+ // would make "not linked" and "linked to something you cannot fetch yet"
+ // the same observation.
+ //
+ // CRITIQUE-03 m3: a card is documented as self-contained, and one that
+ // asserts `verified: true` against evidence the read gate has not opened
+ // is asserting something the consumer cannot check. Caveat says so in
+ // prose as well.
+ PeersUnreadable []string `json:"peersUnreadable,omitempty"`
+
+ Confidence float64 `json:"confidence"`
+ // Verified is true only when a stack-trace match or a re-run flip is
+ // present. Confidence alone never qualifies (plan/00-SPINE.md S7).
+ //
+ // It is CLAMPED, not copied: contract.go's Validate() rejects a record
+ // whose correlation claims verification with no sufficient signal, but the
+ // card is what the agent receives and a malformed producer, a hand-edited
+ // row or a future column default must not be able to hand it an unearned
+ // `verified`. Same reasoning as the host clamp; see cardCorrelation.
+ Verified bool `json:"verified"`
+ Caveat string `json:"caveat,omitempty"`
+
+ // Merged is always false and is emitted explicitly rather than omitted,
+ // because "the field is absent" and "we did not merge" are different
+ // statements to a reader.
+ Merged bool `json:"merged"`
+}
+
+// CardTrust classifies the card's own strings, by RFC 6901 pointer relative to
+// the card.
+//
+// The default is TrustUntrusted, unconditionally. A card exists to be pasted
+// into a repo-credentialed agent's context, and almost everything interesting
+// on it — the source snippet, the context region, the advisory text, the
+// response excerpt — originated outside Anvil. The record's TrustAssertion
+// cannot be copied across verbatim because its pointers address the RESULT
+// object and these address the CARD, so the classification is rebuilt here
+// with the conservative default and an explicit list of the few strings Anvil
+// itself wrote.
+type CardTrust struct {
+ Default Trust `json:"default"`
+ Fields map[string]Trust `json:"fields,omitempty"`
+}
+
+// ---------------------------------------------------------------------------
+// Building cards
+// ---------------------------------------------------------------------------
+
+// CardsFromLog builds every Tier-1 card from an already-loaded record, in the
+// deterministic read order: correlated clusters first, then SAST-only by rank,
+// then DAST-only by rank.
+func (rd *Reader) CardsFromLog(l *SARIFLog) ([]TaskCard, error) {
+ if l == nil {
+ return nil, fmt.Errorf("record: CardsFromLog got a nil *SARIFLog")
+ }
+ // readOrder is where the read gate is applied, so everything below this
+ // line is already past it: a half the gate refused contributes no
+ // orderedResult and therefore no card.
+ order := rd.readOrder(l)
+ clusters := clustersOf(order)
+ readable := readableFindingIDs(order)
+
+ cards := make([]TaskCard, 0, len(order))
+ for i, o := range order {
+ c, err := rd.buildCard(l, i, o, clusters[o.clusterID], readable)
+ if err != nil {
+ return nil, err
+ }
+ cards = append(cards, c)
+ }
+ return cards, nil
+}
+
+// clustersOf groups an already-ordered result set by cluster id. Both tiers
+// call it, so the manifest's view of a cluster and the card's view are the
+// same set — see cardActionable for why that matters.
+func clustersOf(order []orderedResult) map[string][]orderedResult {
+ clusters := map[string][]orderedResult{}
+ for _, o := range order {
+ if o.clusterID != "" {
+ clusters[o.clusterID] = append(clusters[o.clusterID], o)
+ }
+ }
+ return clusters
+}
+
+// readableFindingIDs is the set of findings that survived the read gate, i.e.
+// the exact set for which a card exists. cardCorrelation uses it to mark a
+// peer the consumer cannot fetch.
+func readableFindingIDs(order []orderedResult) map[string]bool {
+ out := make(map[string]bool, len(order))
+ for _, o := range order {
+ out[o.result.Properties.FindingID] = true
+ }
+ return out
+}
+
+// isActionable is the gate: may the coding agent propose a patch for r?
+//
+// Three independent conditions, each of which alone withholds the finding:
+//
+// 1. Not a host finding. plan/00-SPINE.md S7 — the host agent is read-only.
+// 2. remediable_by_agent is true in the record.
+// 3. The verdict is true_positive. The consumption pipeline drops
+// false_positive and demotes insufficient_context to report-only
+// (contract.go, Verdict); report-only is not actionable.
+func isActionable(r *Result) bool {
+ return !IsHostFinding(r) &&
+ r.Properties.RemediableByAgent &&
+ r.Properties.Verdict == VerdictTruePositive
+}
+
+// staticPeerFor returns the cluster peer whose static evidence r must BORROW
+// to have a file and a line at all, or nil when r has its own (or when the
+// cluster offers none).
+//
+// It is the single definition of "this card's locus is inferred, not
+// observed"; buildCard, cardActionable and the manifest all ask it, so the
+// three cannot disagree about which member of a cluster owns the patch site.
+func staticPeerFor(r *Result, cluster []orderedResult) *Result {
+ if hasStaticEvidence(r) {
+ return nil
+ }
+ for _, peer := range cluster {
+ if peer.result != r && hasStaticEvidence(peer.result) {
+ return peer.result
+ }
+ }
+ return nil
+}
+
+// cardActionable is THE read-path actionability gate: isActionable, plus the
+// borrowed-locus withholding below. Every tier asks this function — the card's
+// Actionable field and the manifest's CardRef.Actionable — so the read order
+// and the cards can never disagree about what the agent may act on.
+//
+// # Why a borrowed locus withholds the action (CRITIQUE-03 m1)
+//
+// A DAST-only member of a cluster has no file and no line of its own. It
+// borrows its SAST peer's path, line range, enclosing symbol and snippet so
+// that the card stays self-contained, which is what research/18's annotated
+// Tier-1 card shows and is genuinely what an agent needs to understand the
+// finding. But both members were then independently actionable and pointed at
+// the same line, so one defect produced TWO patch tasks writing proposals into
+// two different `result.fixes` arrays, and two `handoff` rows charged twice
+// against the budget R.11's reservation is dividing. `group_id` is the
+// mechanism that would collapse them and it is RESERVED for the consumption
+// pipeline, so nothing collapses them here.
+//
+// The borrowed side is withheld rather than the borrowing being removed,
+// because the locus is INFERRED for that finding — the DAST result did not
+// observe app/db.py:412, the correlation concluded it — and handing an agent
+// an inferred location as a patch site is presenting inference as observation.
+// The SAST peer, which observed the line and (by the same borrowing, in the
+// other direction) carries the DAST reproduction as its accept oracle, is
+// strictly the better task and stays actionable.
+//
+// This is the ONE legal direction of divergence between a card and the record,
+// named in this file's header: a card may withhold an action the record
+// allows, never grant one it does not. CheckAgainstRecord therefore does not
+// report it.
+func cardActionable(r *Result, cluster []orderedResult) (bool, []string) {
+ blockers := actionBlockers(r)
+ actionable := isActionable(r)
+ if peer := staticPeerFor(r, cluster); peer != nil {
+ if actionable {
+ var clusterID string
+ if co := r.Properties.Correlation; co != nil {
+ clusterID = co.ClusterID
+ }
+ blockers = append(blockers, fmt.Sprintf(
+ "corroborating half of cluster %q: this finding's locus is BORROWED from peer %q, "+
+ "which observed the file and line and carries the actionable card; "+
+ "patching from both members would write two proposals for one defect",
+ clusterID, peer.Properties.FindingID))
+ }
+ actionable = false
+ }
+ return actionable, blockers
+}
+
+// actionBlockers explains a false isActionable, in a fixed order so the text
+// is deterministic.
+func actionBlockers(r *Result) []string {
+ var out []string
+ if IsHostFinding(r) {
+ out = append(out, fmt.Sprintf(
+ "host finding: the host agent is read-only (00-SPINE.md S7), so %s is false for it and no patch may be proposed",
+ PropResultRemediableByAgent))
+ }
+ if !r.Properties.RemediableByAgent && !IsHostFinding(r) {
+ out = append(out, fmt.Sprintf("%s is false in the record", PropResultRemediableByAgent))
+ }
+ switch r.Properties.Verdict {
+ case VerdictFalsePositive:
+ out = append(out, fmt.Sprintf("%s is %q: dropped by the consumption pipeline",
+ PropResultVerdict, VerdictFalsePositive))
+ case VerdictInsufficientContext:
+ out = append(out, fmt.Sprintf("%s is %q: report-only, never silently dropped",
+ PropResultVerdict, VerdictInsufficientContext))
+ }
+ return out
+}
+
+// deriveConsumptionClass maps a finding onto the gate R.4 stores on the
+// handoff row.
+//
+// A finding carrying a reproduction is RequiresDynamicConfirmation: it has a
+// dynamic accept oracle, and plan/00-SPINE.md S7 says only that reproduction
+// failing afterwards earns "verified fixed" — a compile-and-test pass does
+// not. Everything else is StaticOnly, which is the class research/24's triage
+// gate operates on.
+//
+// This is DERIVED. `handoff.consumption_class` is authoritative; if the two
+// ever disagree, the stored row wins for the same reason the record wins over
+// the card.
+func deriveConsumptionClass(r *Result) ConsumptionClass {
+ if r.Properties.Repro != nil || r.Properties.EvidenceClass == EvidenceClassDastConfirmed {
+ return ConsumptionClassRequiresDynamicConfirmation
+ }
+ return ConsumptionClassStaticOnly
+}
+
+func (rd *Reader) buildCard(l *SARIFLog, position int, o orderedResult, cluster []orderedResult, readable map[string]bool) (TaskCard, error) {
+ r := o.result
+ p := &r.Properties
+ actionable, blockers := cardActionable(r, cluster)
+
+ c := TaskCard{
+ CardVersion: CardVersion,
+ AuditID: l.Properties.AuditID,
+ FindingID: p.FindingID,
+ Bucket: o.bucket,
+ Position: position,
+ Half: p.Half,
+ ClusterID: o.clusterID,
+ GroupID: p.GroupID,
+ Rank: o.rank,
+ EvidenceClass: p.EvidenceClass,
+ Verdict: p.Verdict,
+ Confidence: p.Confidence,
+ ConsumptionClass: deriveConsumptionClass(r),
+ // THE CLAMP. Never true for a host finding, whatever the record says.
+ RemediableByAgent: p.RemediableByAgent && !IsHostFinding(r),
+ Actionable: actionable,
+ ActionBlockers: blockers,
+ Task: r.Message.Text,
+ Fingerprint: CardFingerprint{
+ AnvilFindingID: r.PartialFingerprints[PartialFingerprintAnvilFindingID],
+ PrimaryLocationLineHash: r.PartialFingerprints[PartialFingerprintPrimaryLocationLineHash],
+ RegionSha256: r.PartialFingerprints[PartialFingerprintRegionSHA256],
+ },
+ Constraints: p.PatchContext,
+ Risk: p.Risk,
+ WriteBackTo: fmt.Sprintf("/runs/%d/results/%d/fixes", o.runIndex, o.resultIndex),
+ Blobs: map[string][]byte{},
+ }
+ c.Rule = cardRule(o.run, r)
+
+ // A cluster member's card carries its PEER's evidence as well as its own,
+ // so that one card is self-contained (research/18's annotated Tier-1 card
+ // has both halves in it). This is NOT a merge: both findings remain
+ // separate results in the record and each gets its own card. The card is a
+ // read-side convenience; the record's link is the fact.
+ //
+ // Borrowed evidence is LABELLED, never passed off as this finding's own
+ // observation — CardStatic.FindingID and CardLocus.BorrowedFrom both name
+ // the peer it came from — and a borrowed LOCUS withholds the action. See
+ // cardActionable.
+ staticFrom, dynamicFrom := r, r
+ if peer := staticPeerFor(r, cluster); peer != nil {
+ staticFrom = peer
+ }
+ for _, peer := range cluster {
+ if peer.result == r {
+ continue
+ }
+ if dynamicFrom == r && !hasDynamicEvidence(r) && hasDynamicEvidence(peer.result) {
+ dynamicFrom = peer.result
+ }
+ }
+
+ if hasStaticEvidence(staticFrom) {
+ c.Static = cardStatic(staticFrom)
+ }
+ if hasDynamicEvidence(dynamicFrom) {
+ c.Dynamic = cardDynamic(dynamicFrom)
+ }
+ c.Locus = cardLocus(staticFrom, r)
+ if staticFrom != r {
+ c.Locus.BorrowedFrom = staticFrom.Properties.FindingID
+ }
+ c.Advisory = cardAdvisory(r)
+ c.Correlation = cardCorrelation(r, cluster, readable)
+ c.Trust = cardTrust(&c)
+
+ if err := rd.fitCard(&c); err != nil {
+ return TaskCard{}, err
+ }
+ return c, nil
+}
+
+func hasStaticEvidence(r *Result) bool { return primaryPath(r) != "" }
+
+func hasDynamicEvidence(r *Result) bool {
+ return r.Properties.Repro != nil || r.WebRequest != nil || r.WebResponse != nil
+}
+
+func cardRule(run *Run, r *Result) *CardRule {
+ if r.RuleID == "" {
+ return nil
+ }
+ out := &CardRule{ID: r.RuleID}
+ for i := range run.Tool.Driver.Rules {
+ rule := &run.Tool.Driver.Rules[i]
+ if rule.ID != r.RuleID {
+ continue
+ }
+ out.Name = rule.Name
+ out.HelpURI = rule.HelpURI
+ if rule.ShortDescription != nil {
+ out.Description = rule.ShortDescription.Text
+ }
+ break
+ }
+ return out
+}
+
+func cardStatic(r *Result) *CardStatic {
+ s := &CardStatic{
+ FindingID: r.Properties.FindingID,
+ File: primaryPath(r),
+ Confidence: r.Properties.Confidence,
+ Reasoning: r.Properties.Reasoning,
+ }
+ if loc := primaryPhysical(r); loc != nil {
+ if loc.Region != nil {
+ s.StartLine, s.EndLine = loc.Region.StartLine, loc.Region.EndLine
+ if loc.Region.Snippet != nil {
+ s.Code = loc.Region.Snippet.Text
+ }
+ }
+ if loc.ContextRegion != nil {
+ ctx := &CardContext{StartLine: loc.ContextRegion.StartLine, EndLine: loc.ContextRegion.EndLine}
+ if loc.ContextRegion.Snippet != nil {
+ ctx.Text = loc.ContextRegion.Snippet.Text
+ }
+ s.Context = ctx
+ }
+ }
+ s.Symbol = enclosingSymbol(r)
+ s.TaintPath = taintPath(r)
+ return s
+}
+
+func cardDynamic(r *Result) *CardDynamic {
+ d := &CardDynamic{FindingID: r.Properties.FindingID, Confidence: r.Properties.Confidence}
+ if req := r.WebRequest; req != nil {
+ d.Method, d.URL = req.Method, req.Target
+ if req.Body != nil {
+ d.RequestBody = req.Body.Text
+ }
+ }
+ if resp := r.WebResponse; resp != nil {
+ d.StatusCode = resp.StatusCode
+ if resp.Body != nil {
+ d.ResponseExcerpt = resp.Body.Text
+ }
+ }
+ if rp := r.Properties.Repro; rp != nil {
+ d.Payload, d.PayloadEncoding = rp.Payload, rp.PayloadEncoding
+ d.InjectionPoint = string(rp.InjectionPoint.Kind)
+ if rp.InjectionPoint.Name != "" {
+ d.InjectionPoint += ":" + rp.InjectionPoint.Name
+ }
+ d.ObservedSignal = rp.ObservedSignal.Kind
+ if rp.ObservedSignal.Match != nil {
+ d.Observed = rp.ObservedSignal.Match.Text
+ }
+ d.ResponseBodyRef = rp.ObservedSignal.BodySha256
+ if rp.Baseline != nil {
+ d.BaselineStatusCode = rp.Baseline.StatusCode
+ }
+ d.Steps = rp.Steps
+ d.Curl = rp.Curl
+ d.ExpectedAfterFix = rp.ExpectedAfterFix
+ d.SideEffects = rp.SideEffects
+ env := rp.Env
+ d.Env = &env
+ }
+ return d
+}
+
+// cardLocus fills research/24's locus.*. Path, lines and enclosing symbol come
+// from the SARIF-native slots of the finding that HAS them (the SAST peer, for
+// a DAST-only card in a cluster); proximity_class comes from the card's own
+// finding, because it is that finding's grouping input.
+func cardLocus(staticFrom, own *Result) CardLocus {
+ lo := CardLocus{Path: primaryPath(staticFrom), EnclosingSymbol: enclosingSymbol(staticFrom)}
+ if loc := primaryPhysical(staticFrom); loc != nil && loc.Region != nil {
+ lo.StartLine, lo.EndLine = loc.Region.StartLine, loc.Region.EndLine
+ }
+ if own.Properties.Locus != nil {
+ lo.ProximityClass = own.Properties.Locus.ProximityClass
+ }
+ return lo
+}
+
+func cardAdvisory(r *Result) *CardAdvisory {
+ taxa := taxonIDs(r)
+ a := r.Properties.Advisory
+ if a == nil && len(taxa) == 0 {
+ return nil
+ }
+ out := &CardAdvisory{Taxa: taxa}
+ if a == nil {
+ return out
+ }
+ out.IDs, out.CveIDs = a.IDs, a.CveIDs
+ out.SourceFeed, out.LicenseSpdx = a.SourceFeed, a.LicenseSpdx
+ out.AsOf = formatTime(a.AsOf)
+ out.StalenessSeconds, out.ParseDegraded = a.StalenessSeconds, a.ParseDegraded
+ if a.Excerpt != nil {
+ out.Excerpt = a.Excerpt.Text
+ }
+ return out
+}
+
+func cardCorrelation(r *Result, cluster []orderedResult, readable map[string]bool) *CardCorrelation {
+ co := r.Properties.Correlation
+ if co == nil {
+ return nil
+ }
+ // THE VERIFIED CLAMP. verificationOf is correlation.go's own predicate,
+ // which asks CorrelationSignal.SufficientForVerified and nothing else —
+ // never a second copy of the rule. The record's bit can only be withheld
+ // here, never granted, which is the one legal direction of divergence.
+ earned, _ := verificationOf(co.Signals)
+ out := &CardCorrelation{
+ ClusterID: co.ClusterID,
+ Role: co.Role,
+ Confidence: co.Confidence,
+ Verified: co.Verified && earned,
+ Caveat: co.Caveat,
+ Merged: false,
+ }
+ for _, s := range co.Signals {
+ out.Signals = append(out.Signals, string(s.Name))
+ }
+ seen := map[string]bool{r.Properties.FindingID: true}
+ for _, id := range co.Peers {
+ if !seen[id] {
+ seen[id] = true
+ out.Peers = append(out.Peers, id)
+ }
+ }
+ // The cluster as actually assembled is the better peer list when the
+ // record's own Peers array is empty or stale; both are unioned, then
+ // sorted so the card is byte-stable.
+ for _, m := range cluster {
+ id := m.result.Properties.FindingID
+ if !seen[id] {
+ seen[id] = true
+ out.Peers = append(out.Peers, id)
+ }
+ }
+ sort.Strings(out.Peers)
+
+ // A peer named here that has no card is a peer the read gate refused. Say
+ // so, on the card, in both the machine-readable list and the caveat prose.
+ for _, id := range out.Peers {
+ if !readable[id] {
+ out.PeersUnreadable = append(out.PeersUnreadable, id)
+ }
+ }
+ if len(out.PeersUnreadable) > 0 {
+ note := fmt.Sprintf(
+ "peers %s are linked to this finding but their half has not passed the read gate, "+
+ "so no task card exists for them and this link cannot be checked against their evidence yet",
+ strings.Join(out.PeersUnreadable, ", "))
+ if out.Caveat == "" {
+ out.Caveat = note
+ } else {
+ out.Caveat += "; " + note
+ }
+ }
+ return out
+}
+
+// cardTrust classifies the card's strings: untrusted by default, with an
+// explicit list of the strings Anvil itself generated.
+func cardTrust(c *TaskCard) CardTrust {
+ t := CardTrust{Default: TrustUntrusted, Fields: map[string]Trust{}}
+ if c.Task != "" {
+ t.Fields["/task"] = TrustAnvilGenerated
+ }
+ if c.Static != nil && c.Static.Reasoning != "" {
+ t.Fields["/static/reasoning"] = TrustAnvilGenerated
+ }
+ if len(t.Fields) == 0 {
+ t.Fields = nil
+ }
+ return t
+}
+
+func primaryPhysical(r *Result) *PhysicalLocation {
+ for i := range r.Locations {
+ if r.Locations[i].PhysicalLocation != nil {
+ return r.Locations[i].PhysicalLocation
+ }
+ }
+ return nil
+}
+
+func enclosingSymbol(r *Result) string {
+ for i := range r.Locations {
+ for _, ll := range r.Locations[i].LogicalLocations {
+ if ll.FullyQualifiedName != "" {
+ return ll.FullyQualifiedName
+ }
+ if ll.Name != "" {
+ return ll.Name
+ }
+ }
+ }
+ return ""
+}
+
+// taintPath flattens the first code flow to one "path:line symbol-or-snippet"
+// string per step — research/18's `taintPath` array.
+func taintPath(r *Result) []string {
+ var out []string
+ for _, cf := range r.CodeFlows {
+ for _, tf := range cf.ThreadFlows {
+ for _, tfl := range tf.Locations {
+ pl := tfl.Location.PhysicalLocation
+ if pl == nil {
+ continue
+ }
+ step := pl.ArtifactLocation.URI
+ if pl.Region != nil && pl.Region.StartLine > 0 {
+ step += ":" + strconv.Itoa(pl.Region.StartLine)
+ }
+ switch {
+ case pl.Region != nil && pl.Region.Snippet != nil && pl.Region.Snippet.Text != "":
+ step += " " + strings.TrimSpace(pl.Region.Snippet.Text)
+ case tfl.Location.Message != nil && tfl.Location.Message.Text != "":
+ step += " " + tfl.Location.Message.Text
+ }
+ out = append(out, step)
+ }
+ }
+ }
+ return out
+}
+
+// ---------------------------------------------------------------------------
+// Fitting a card inside its budget
+// ---------------------------------------------------------------------------
+
+// fitCard enforces, in order:
+//
+// 1. The INLINE CAPS. The advisory excerpt at research/24's <=800 tokens; the
+// request body at MaxInlineRequestBodyBytes and the response excerpt at
+// MaxInlineResponseBodyBytes, which are R.8's ZAP-derived caps. These
+// apply whatever the card's total size is, because they are about what may
+// be inlined at all, not about what fits.
+//
+// 2. The TOKEN BUDGET, by spilling fields in a fixed order, least to most
+// load-bearing for writing the patch:
+//
+// reasoning — the detector's prose; the agent re-derives it from the code.
+// taintPath — navigational; the locus already names the sink.
+// context — the surrounding region; the snippet still names the defect.
+// excerpt — advisory prose.
+// response — the observed evidence span, which the repro can regenerate.
+// request — the request body, which `curl` already carries.
+// code — the defect snippet. LAST: without it the card cannot do its
+// one job, and a card that has spilled its code is a card
+// the agent must make a Tier-2 fetch to use.
+//
+// Every step writes the spilled text to a Tier-2 blob and records a TierSpill.
+// If the card is still over budget afterwards, it is an error unless
+// Reader.AllowOversizeTier1 carries an explicit reason.
+func (rd *Reader) fitCard(c *TaskCard) error {
+ if c.Advisory != nil {
+ if err := rd.capCardText(c, "/advisory/excerpt", &c.Advisory.Excerpt, MaxAdvisoryExcerptBytes); err != nil {
+ return err
+ }
+ }
+ if c.Dynamic != nil {
+ if err := rd.capCardText(c, "/dynamic/requestBody", &c.Dynamic.RequestBody, MaxInlineRequestBodyBytes); err != nil {
+ return err
+ }
+ if err := rd.capCardText(c, "/dynamic/responseExcerpt", &c.Dynamic.ResponseExcerpt, MaxInlineResponseBodyBytes); err != nil {
+ return err
+ }
+ }
+
+ steps := []struct {
+ field string
+ take func(*TaskCard) (string, bool)
+ }{
+ {"/static/reasoning", func(c *TaskCard) (string, bool) {
+ if c.Static == nil || c.Static.Reasoning == "" {
+ return "", false
+ }
+ v := c.Static.Reasoning
+ c.Static.Reasoning = ""
+ return v, true
+ }},
+ {"/static/taintPath", func(c *TaskCard) (string, bool) {
+ if c.Static == nil || len(c.Static.TaintPath) == 0 {
+ return "", false
+ }
+ v := strings.Join(c.Static.TaintPath, "\n")
+ c.Static.TaintPath = nil
+ return v, true
+ }},
+ {"/static/context/text", func(c *TaskCard) (string, bool) {
+ if c.Static == nil || c.Static.Context == nil || c.Static.Context.Text == "" {
+ return "", false
+ }
+ v := c.Static.Context.Text
+ c.Static.Context.Text = ""
+ return v, true
+ }},
+ {"/advisory/excerpt", func(c *TaskCard) (string, bool) {
+ if c.Advisory == nil || c.Advisory.Excerpt == "" {
+ return "", false
+ }
+ v := c.Advisory.Excerpt
+ c.Advisory.Excerpt = ""
+ return v, true
+ }},
+ {"/dynamic/responseExcerpt", func(c *TaskCard) (string, bool) {
+ if c.Dynamic == nil || c.Dynamic.ResponseExcerpt == "" {
+ return "", false
+ }
+ v := c.Dynamic.ResponseExcerpt
+ c.Dynamic.ResponseExcerpt = ""
+ return v, true
+ }},
+ {"/dynamic/requestBody", func(c *TaskCard) (string, bool) {
+ if c.Dynamic == nil || c.Dynamic.RequestBody == "" {
+ return "", false
+ }
+ v := c.Dynamic.RequestBody
+ c.Dynamic.RequestBody = ""
+ return v, true
+ }},
+ {"/static/code", func(c *TaskCard) (string, bool) {
+ if c.Static == nil || c.Static.Code == "" {
+ return "", false
+ }
+ v := c.Static.Code
+ c.Static.Code = ""
+ return v, true
+ }},
+ }
+
+ size, err := measureCard(c)
+ if err != nil {
+ return err
+ }
+ for _, st := range steps {
+ if size <= MaxTier1CardBytes {
+ break
+ }
+ text, ok := st.take(c)
+ if !ok {
+ continue
+ }
+ // A field the inline caps already spilled is NOT spilled twice: the
+ // full bytes are in Tier 2 under the reference already recorded, and a
+ // second blob holding the truncated prefix of the same field would
+ // make Spills name one field twice with two different contents.
+ if !hasSpill(c.Spills, st.field) {
+ if err := rd.spillBytes(&c.Spills, c.Blobs, st.field, []byte(text), 0); err != nil {
+ return err
+ }
+ }
+ if size, err = measureCard(c); err != nil {
+ return err
+ }
+ }
+
+ if size > MaxTier1CardBytes {
+ if rd.AllowOversizeTier1 == "" {
+ return &BudgetError{
+ Tier: "tier-1 task card", Subject: c.FindingID,
+ Bytes: size, Budget: MaxTier1CardBytes,
+ Tokens: ApproxTokens(size), MaxTok: MaxTier1CardTokens,
+ }
+ }
+ c.Override = &BudgetOverride{Reason: rd.AllowOversizeTier1, Bytes: size, Budget: MaxTier1CardBytes}
+ if _, err = measureCard(c); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// measureCard is measureManifest for a card: it stamps the measurement into
+// the card and returns the size the card actually has once stamped. See
+// measureManifest for why the stamp has to be inside the measurement.
+func measureCard(c *TaskCard) (int, error) {
+ size, err := measure(c)
+ if err != nil {
+ return 0, err
+ }
+ for i := 0; i < 8; i++ {
+ c.Bytes, c.Tokens = size, ApproxTokens(size)
+ next, err := measure(c)
+ if err != nil {
+ return 0, err
+ }
+ if next == size {
+ return size, nil
+ }
+ size = next
+ }
+ c.Bytes, c.Tokens = size, ApproxTokens(size)
+ return measure(c)
+}
+
+func hasSpill(spills []TierSpill, field string) bool {
+ for _, s := range spills {
+ if s.Field == field {
+ return true
+ }
+ }
+ return false
+}
+
+// capCardText enforces one inline cap: text longer than limit keeps a prefix
+// plus an in-band pointer, and the FULL text becomes a Tier-2 blob.
+//
+// The in-band notice is R.8's truncationNotice, byte for byte, so a reader
+// meets one truncation spelling across the record and the read path rather
+// than two.
+func (rd *Reader) capCardText(c *TaskCard, field string, dst *string, limit int) error {
+ if dst == nil || len(*dst) <= limit {
+ return nil
+ }
+ full := *dst
+ if err := rd.spillBytes(&c.Spills, c.Blobs, field, []byte(full), 0); err != nil {
+ return err
+ }
+ ref := c.Spills[len(c.Spills)-1].Ref
+ budget := limit - len(truncationNotice(limit, len(full), ref))
+ if budget < 0 {
+ budget = 0
+ }
+ inline := truncateToRuneBoundary(full, budget)
+ *dst = inline + truncationNotice(len(inline), len(full), ref)
+ return nil
+}
+
+// ---------------------------------------------------------------------------
+// The record wins
+// ---------------------------------------------------------------------------
+
+// ActionableTaskCards returns the cards the coding agent may act on.
+//
+// It exists so that "filter the cards" is one call with one definition rather
+// than a predicate each caller writes for itself — the second copy of a
+// predicate is how a host finding ends up in front of a read-only agent.
+func ActionableTaskCards(cards []TaskCard) []TaskCard {
+ out := make([]TaskCard, 0, len(cards))
+ for _, c := range cards {
+ if c.Actionable {
+ out = append(out, c)
+ }
+ }
+ return out
+}
+
+// CheckAgainstRecord reports every place this card contradicts the result it
+// was derived from.
+//
+// THE RECORD WINS. This function never repairs anything and never touches the
+// record; it names the disagreements so a caller can rebuild the card, which
+// is the only correct repair.
+//
+// One asymmetry is deliberate and is NOT reported: a card may be LESS
+// permissive than the record — RemediableByAgent false where the record says
+// true, Actionable false where the record would allow it. That is the host
+// clamp and the verdict demotion, and both are safe by construction. The
+// reverse — a card granting an action the record does not — is always an
+// error.
+func (c *TaskCard) CheckAgainstRecord(r *Result) error {
+ if r == nil {
+ return fmt.Errorf("record: CheckAgainstRecord got a nil *Result for card %q", c.FindingID)
+ }
+ var problems []string
+ note := func(format string, args ...any) {
+ problems = append(problems, fmt.Sprintf(format, args...))
+ }
+
+ if c.FindingID != r.Properties.FindingID {
+ note("card is for finding %q but was checked against %q", c.FindingID, r.Properties.FindingID)
+ }
+ if c.EvidenceClass != r.Properties.EvidenceClass {
+ note("evidenceClass: card %q, record %q", c.EvidenceClass, r.Properties.EvidenceClass)
+ }
+ if c.Verdict != r.Properties.Verdict {
+ note("verdict: card %q, record %q", c.Verdict, r.Properties.Verdict)
+ }
+ if c.Half != r.Properties.Half {
+ note("half: card %q, record %q", c.Half, r.Properties.Half)
+ }
+ if c.Confidence != r.Properties.Confidence {
+ note("confidence: card %v, record %v", c.Confidence, r.Properties.Confidence)
+ }
+ if want := r.PartialFingerprints[PartialFingerprintAnvilFindingID]; c.Fingerprint.AnvilFindingID != want {
+ note("fingerprint %s: card %q, record %q", PartialFingerprintAnvilFindingID,
+ c.Fingerprint.AnvilFindingID, want)
+ }
+ if c.RemediableByAgent && !r.Properties.RemediableByAgent {
+ note("%s is true on the card and false in the record; a card may withhold an action, never grant one",
+ PropResultRemediableByAgent)
+ }
+ if c.RemediableByAgent && IsHostFinding(r) {
+ note("%s is true on the card for a HOST finding; the host agent is read-only (00-SPINE.md S7)",
+ PropResultRemediableByAgent)
+ }
+ if c.Actionable && !isActionable(r) {
+ note("card is actionable but the record does not permit it (host=%t remediable=%t verdict=%q)",
+ IsHostFinding(r), r.Properties.RemediableByAgent, r.Properties.Verdict)
+ }
+ if c.Correlation != nil && c.Correlation.Merged {
+ note("correlation.merged is true; link, never merge")
+ }
+ // `verified` is an S7 gate of the same class as the host gate, so it gets
+ // the same treatment: the card may not assert a verification the SIGNALS
+ // do not earn, whatever bit the record carries. Checked against the
+ // signals rather than against the record's own `verified` flag, because a
+ // record whose flag is wrong is precisely the case this catches.
+ if c.Correlation != nil && c.Correlation.Verified {
+ co := r.Properties.Correlation
+ if co == nil {
+ note("correlation.verified is true on the card but the record carries no %s at all",
+ PropResultCorrelation)
+ } else if earned, _ := verificationOf(co.Signals); !earned {
+ note("correlation.verified is true on the card but no %q or %q signal is present in the record; "+
+ "confidence alone never qualifies (00-SPINE.md S7)",
+ CorrelationSignalResponseStackTrace, CorrelationSignalRerunFlip)
+ }
+ }
+
+ if len(problems) == 0 {
+ return nil
+ }
+ return fmt.Errorf("record: task card %q disagrees with the record, and the record wins: %s",
+ c.FindingID, strings.Join(problems, "; "))
+}
diff --git a/internal/store/queue.go b/internal/store/queue.go
new file mode 100644
index 0000000..909e5cb
--- /dev/null
+++ b/internal/store/queue.go
@@ -0,0 +1,821 @@
+// The queue re-cut rule (step R.11).
+//
+// plan/00-SPINE.md S6, in full, because the second half is the part that gets
+// lost: "re-cut the work queue on every version bump and **reserve a
+// configurable fraction (default 50%) of remaining budget for late
+// DAST-confirmed arrivals** — otherwise incremental publication silently
+// inverts the priority scheme, because nothing is DAST-confirmed when the
+// queue is first cut."
+//
+// WHY THE RESERVATION IS NOT REDUNDANT WITH RANKING. Within a single cut the
+// reservation buys nothing: `dast_confirmed` is the first component of
+// research/24 step 7's rank_key, so a cut that sorts by rank already puts the
+// proof-carrying findings first. The reservation exists because **a cut is a
+// commitment across time**, and DAST runs after SAST. At the first cut the DAST
+// half has not concluded, so `evidence_class = 'dast_confirmed'` matches zero
+// rows; ranking has nothing to rank. A cut with no reservation therefore
+// commits the entire budget to unconfirmed static findings, and when the DAST
+// half seals and its confirmed findings are enqueued at the next version, the
+// re-cut finds a budget that is already spent. The highest-value findings in
+// the audit — the only ones carrying a working reproduction — become
+// `skipped_budget`, i.e. *found, not fixed*, while 24%-precision static alarms
+// (research/24 Table 3, 105 TP / 433 alarms) get the whole window. That is the
+// inversion, and queue_test.go's TestRecutInvertsPriorityWithoutReservation
+// reproduces it end to end rather than asserting that a float equals 0.5.
+//
+// The reservation is a FLOOR FOR dast_confirmed, NOT A CEILING. A cut holds
+// back `fraction * remaining` and refuses to let any other evidence class draw
+// on it; `dast_confirmed` rows draw on the reserve first and on the open
+// budget after. Nothing is capped, and nothing is spent on a class that does
+// not exist yet.
+//
+// WHAT THIS FILE DOES NOT DO — the CRITIQUE-02 §7 open question, decided.
+// §7 records as unresolved "whether R.11's queue re-cut is intended to
+// Dispose(..., superseded) every stale row of a bumped audit". IT IS NOT.
+// A re-cut never touches a `leased` row, and never writes `superseded`.
+// Reasons, in the order they are load-bearing:
+//
+// 1. internal/handoff already makes a stale lease inert. checkRecordVersion
+// runs on RenewLease, ReleaseLease and the packet gate, so a Handle taken
+// at audit_version N cannot renew, cannot record any disposition, and
+// cannot read or write its packet once the version moves to N+1. It gets
+// ErrRecordVersionChanged. The lease therefore cannot be extended, expires
+// on its own clock, and ReclaimExpired — the ONE component authorised to
+// touch a lease it does not hold — returns the row to 'ready' or exhausts
+// it. The row rejoins the candidate set of the *next* cut with no action
+// from here. The guard is sufficient because it guards the writes, which
+// is where a stale version can do damage, rather than the row state, where
+// it cannot.
+// 2. Doing otherwise would expire a live claim. research/08 §4 point 2:
+// "Never expire a live claim. A finding at expires_at whose lease is still
+// alive must be allowed to finish. Expiring it would let a second agent
+// write a competing fix for the same defect." CRITIQUE-02 verdict (e)
+// ("reaper never drops a live claim") is a PASS that a re-cut yanking
+// leased rows would turn into a FAIL, and R.7's noSiblingLease invariant —
+// one live lease per (fingerprint, audit_version) — is defended by
+// refusing a SECOND grant, not by cancelling the first.
+// 3. It is not expressible without fighting the owner. handoff.Dispose
+// refuses a leased row on purpose ("only the lease holder may decide the
+// outcome of its own attempt"), and internal/store cannot call into
+// internal/handoff at all: handoff imports store, so the edge only runs
+// one way. A raw-SQL back door here would make internal/store a second
+// writer to the lease protocol, which is the defect class
+// plan/IMPLEMENTATION-PLAN.md §6 G9 and G10 exist to prevent.
+// 4. `superseded` is the wrong verb for this actor. A re-cut is a BUDGET
+// decision and its only disposition is `skipped_budget` (research/24 step
+// 9: "everything past the cut is marked SKIPPED_BUDGET and goes into the
+// report as *found, not fixed* — never silently dropped"). Deciding that a
+// version bump made a finding obsolete is a correlation judgement about
+// identity, which belongs to R.12, not to the component that divides
+// tokens.
+//
+// A row this file defers is 'ready' -> 'skipped_budget', which is a legal edge
+// of internal/handoff's state machine and is terminal there. That is deliberate
+// and it is why a re-cut can only ever narrow the admitted set: the machine has
+// no 'skipped_budget' -> 'ready' edge ("Terminal is terminal"), so a later cut
+// with a larger budget cannot re-admit a row it already deferred. The cut is
+// consequently monotone, which is also what makes it safe to repeat.
+
+package store
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "math"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/Susquehanna-Syntax/Anvil/internal/record"
+)
+
+// DefaultDastReserveFraction is the documented default for
+// RecutConfig.DastReserveFraction: plan/00-SPINE.md S6's "configurable
+// fraction (default 50%)".
+//
+// It is a DEFAULT, not the value. RecutConfig carries the configured fraction
+// and queue_test.go proves the arithmetic changes when the configuration
+// changes, because a constant that is read from exactly one place is
+// indistinguishable from a hardcoded one.
+const DefaultDastReserveFraction = 0.5
+
+// DefaultTokensPerCandidate is the default per-finding charge against the
+// budget, in prompt tokens.
+//
+// research/24-coding-agent-consumption.md Table 4 derives the 8-hour window's
+// throughput from a 10,000-token prompt per fix group, and step 12 of the same
+// document asserts a hard ceiling of 12,000 with SPLIT_REQUIRED above 16,000.
+// 10,000 is that document's own working figure and it is stated there to be
+// arithmetic rather than measurement, so it is a configurable default here.
+//
+// It is per CANDIDATE, not per fix group. Grouping (research/24 step 8:
+// same file, same enclosing symbol, capped at 5 findings) happens downstream
+// of this file and would only ever make the true cost LOWER, so charging per
+// finding is the conservative direction: a cut errs toward deferring work, not
+// toward overcommitting the window.
+const DefaultTokensPerCandidate = 10000
+
+var (
+ // ErrNoSuchAudit means the audit id did not resolve to an audit_record row.
+ ErrNoSuchAudit = errors.New("store: no such audit record")
+
+ // ErrInvalidReserveFraction means a configured reserve fraction is outside
+ // [0, 1] or is not a number. A fraction above 1 would reserve more than the
+ // whole remaining budget, which silently starves every class at once.
+ ErrInvalidReserveFraction = errors.New("store: DAST reserve fraction must be in [0, 1]")
+
+ // ErrInvalidBudget means a negative remaining budget was passed. Zero is
+ // legal and means "the window is spent"; negative is a caller bug and
+ // clamping it would hide the caller's arithmetic error inside a component
+ // whose whole job is arithmetic.
+ ErrInvalidBudget = errors.New("store: remaining budget must not be negative")
+)
+
+// ReserveFraction boxes a fraction for RecutConfig.DastReserveFraction.
+//
+// The field is a pointer precisely so that 0 is expressible. A plain float64
+// with "zero means default" would make "reserve nothing" unreachable through
+// configuration, and "reserve nothing" is the control arm that demonstrates the
+// inversion S6 describes — the one setting a test must be able to select.
+func ReserveFraction(f float64) *float64 { return &f }
+
+// RecutConfig is the queue re-cut's configuration. The zero value is usable
+// and means: default reserve fraction, default per-candidate cost, no severity
+// ordering, wall clock.
+type RecutConfig struct {
+ // DastReserveFraction is S6's "configurable fraction (default 50%) of
+ // remaining budget" held for late dast_confirmed arrivals. nil selects
+ // DefaultDastReserveFraction. Use ReserveFraction to set it.
+ DastReserveFraction *float64
+
+ // TokensPerCandidate is the flat charge per candidate when CostTokens is
+ // nil. Zero or negative selects DefaultTokensPerCandidate.
+ TokensPerCandidate int
+
+ // CostTokens overrides the flat charge per candidate. It exists so a later
+ // step can charge a measured prompt size (research/24 step 12's six
+ // sections) without editing this file. A non-positive return is treated as
+ // TokensPerCandidate; a cost-free candidate would let an unbounded number
+ // of rows through the cut, which is not a budget.
+ CostTokens func(Candidate) int
+
+ // SeverityRank orders findings within one evidence class, lower first.
+ // It is a map and not a Go enum ON PURPOSE: `finding.severity` is one of
+ // the vocabularies schema.sql deliberately leaves unconstrained because
+ // internal/record does not own it, and freezing it here would be area 40
+ // inventing another area's vocabulary — the exact defect
+ // plan/IMPLEMENTATION-PLAN.md §6 exists to stop. A severity absent from
+ // the map ranks last among its class; nil ranks every severity equally and
+ // the cut falls through to finding_id.
+ SeverityRank map[string]int
+
+ // Clock supplies handoff.updated_at. nil means time.Now.
+ Clock func() time.Time
+}
+
+func (c RecutConfig) reserveFraction() float64 {
+ if c.DastReserveFraction == nil {
+ return DefaultDastReserveFraction
+ }
+ return *c.DastReserveFraction
+}
+
+func (c RecutConfig) tokensPerCandidate() int {
+ if c.TokensPerCandidate <= 0 {
+ return DefaultTokensPerCandidate
+ }
+ return c.TokensPerCandidate
+}
+
+func (c RecutConfig) cost(cand Candidate) int {
+ if c.CostTokens != nil {
+ if n := c.CostTokens(cand); n > 0 {
+ return n
+ }
+ }
+ return c.tokensPerCandidate()
+}
+
+func (c RecutConfig) now() time.Time {
+ if c.Clock == nil {
+ return time.Now().UTC()
+ }
+ return c.Clock().UTC()
+}
+
+func (c RecutConfig) validate() error {
+ f := c.reserveFraction()
+ if math.IsNaN(f) || f < 0 || f > 1 {
+ return fmt.Errorf("%w, got %v", ErrInvalidReserveFraction, f)
+ }
+ return nil
+}
+
+// recutTimestampLayout is the timestamp format written into
+// handoff.updated_at.
+//
+// It must stay identical to internal/handoff/state_machine.go's timeLayout.
+// That constant is unexported and internal/store cannot import internal/handoff
+// (handoff imports store; the edge runs one way), so this is a deliberate
+// second copy of a FORMAT — not of a vocabulary. If the two ever diverge, the
+// handoff reaper's string comparisons over lease_expires_at and updated_at
+// start comparing differently-shaped strings; queue_test.go pins the exact
+// layout so a change here fails loudly rather than at 3am in a reaper.
+const recutTimestampLayout = "2006-01-02T15:04:05.000000000Z"
+
+func formatRecutTime(t time.Time) string { return t.UTC().Format(recutTimestampLayout) }
+
+// Candidate is one `handoff` row the cut can see, joined to the `finding` it
+// carries.
+type Candidate struct {
+ HandoffID int64
+ FindingID int64
+ Fingerprint string
+ State record.HandoffState
+ EvidenceClass record.EvidenceClass
+ Severity string
+
+ // CostTokens is what this row was charged, filled in by the cut.
+ CostTokens int
+}
+
+// DastConfirmed reports whether this candidate is in the class S6 reserves
+// budget for. It reads the frozen enum constant, never the literal.
+func (c Candidate) DastConfirmed() bool {
+ return c.EvidenceClass == record.EvidenceClassDastConfirmed
+}
+
+// Cut is one re-cut, reported in full so a caller — and a test — can see the
+// arithmetic instead of inferring it from the resulting row states.
+type Cut struct {
+ AuditRecordID int64
+ AuditVersion int64
+ DastStatus record.DastStatus
+
+ // Performed is false when the version guard declined to re-cut. S6 and the
+ // R.11 packet both say a re-cut is triggered by an audit_version bump and
+ // by nothing else; NotCutReason says which guard declined.
+ Performed bool
+ NotCutReason string
+
+ // RemainingBudgetTokens is what the caller passed in.
+ RemainingBudgetTokens int
+
+ // ReserveFraction is the fraction actually applied. It is 0 when
+ // LateDastArrivalsPossible is false, whatever the configuration says.
+ ReserveFraction float64
+ LateDastArrivalsPossible bool
+
+ // ReservedTokens is floor(ReserveFraction * RemainingBudgetTokens) — of
+ // REMAINING budget at re-cut time, never of the window's total. OpenTokens
+ // is the rest, and is the only pool a non-dast_confirmed row may draw on.
+ ReservedTokens int
+ OpenTokens int
+
+ // InFlightTokens is what currently-leased rows were charged before any
+ // candidate was considered. InFlightOverdraftTokens is the amount by which
+ // they exceeded the remaining budget, which is a real state (the caller's
+ // budget estimate shrank while leases were out) and is reported rather than
+ // hidden.
+ InFlightTokens int
+ InFlightOverdraftTokens int
+
+ // Admitted rows keep handoff.state = 'ready'. Deferred rows were moved to
+ // 'skipped_budget'. Contended rows were deferred by the arithmetic but had
+ // left 'ready' before the write landed — a claim won the race — and were
+ // therefore left alone.
+ Admitted []Candidate
+ Deferred []Candidate
+ Contended []Candidate
+}
+
+// AdmittedTokens is the total charged to admitted candidates.
+func (c Cut) AdmittedTokens() int { return sumCost(c.Admitted) }
+
+// AdmittedDastConfirmedTokens is the total charged to admitted candidates in
+// the class S6 reserves for. It is the number the reservation exists to keep
+// above ReservedTokens once such candidates exist.
+func (c Cut) AdmittedDastConfirmedTokens() int {
+ total := 0
+ for _, cand := range c.Admitted {
+ if cand.DastConfirmed() {
+ total += cand.CostTokens
+ }
+ }
+ return total
+}
+
+// DeferredDastConfirmed counts admitted-nothing in the highest-value class.
+// A cut where this is non-zero while a lower class was admitted is the
+// inversion S6 forbids.
+func (c Cut) DeferredDastConfirmed() int {
+ n := 0
+ for _, cand := range c.Deferred {
+ if cand.DastConfirmed() {
+ n++
+ }
+ }
+ return n
+}
+
+// InvertedPriority reports the failure mode S6 names: at least one
+// dast_confirmed candidate was pushed past the cut while at least one weaker
+// evidence class was admitted. It is a property of ONE cut; the inversion S6
+// describes is produced across cuts, and queue_test.go drives both.
+func (c Cut) InvertedPriority() bool {
+ if c.DeferredDastConfirmed() == 0 {
+ return false
+ }
+ for _, cand := range c.Admitted {
+ if !cand.DastConfirmed() {
+ return true
+ }
+ }
+ return false
+}
+
+func sumCost(cands []Candidate) int {
+ total := 0
+ for _, c := range cands {
+ total += c.CostTokens
+ }
+ return total
+}
+
+// Recutter re-cuts one store's work queues.
+//
+// It is safe for concurrent use: every re-cut holds the Recutter's mutex for
+// its whole duration, including its database work. A re-cut is a rare,
+// audit-scoped operation triggered by a version bump, so serialising them costs
+// nothing and removes the whole class of interleaving between the version
+// guard's read and the cut's writes.
+type Recutter struct {
+ db *sql.DB
+ cfg RecutConfig
+
+ mu sync.Mutex
+ // lastCut records the audit_version each audit was last cut at. See
+ // RecutQueueContext for why losing it across a restart is safe.
+ lastCut map[int64]int64
+}
+
+// NewRecutter returns a Recutter over an already-migrated store.
+func NewRecutter(db *sql.DB, cfg RecutConfig) (*Recutter, error) {
+ if db == nil {
+ return nil, errors.New("store: NewRecutter requires a non-nil *sql.DB")
+ }
+ if err := cfg.validate(); err != nil {
+ return nil, err
+ }
+ return &Recutter{db: db, cfg: cfg, lastCut: make(map[int64]int64)}, nil
+}
+
+// ReserveFraction reports the configured fraction, after defaulting.
+func (r *Recutter) ReserveFraction() float64 { return r.cfg.reserveFraction() }
+
+// RecutQueue is the R.11 packet's entry point: re-cut the work queue for one
+// audit against the budget remaining at this moment.
+//
+// It re-cuts only when the audit's `audit_version` has moved since the last cut
+// this Recutter performed, which is the S6 trigger and the only one — a write
+// to `handoff` is not a trigger, and the packet forbids making it one. Use
+// RecutQueueContext when the arithmetic matters to the caller.
+func (r *Recutter) RecutQueue(auditID string, remainingBudgetTokens int) error {
+ _, err := r.RecutQueueContext(context.Background(), auditID, remainingBudgetTokens)
+ return err
+}
+
+// RecutQueueContext is RecutQueue with a caller-supplied context, returning the
+// full arithmetic.
+//
+// THE VERSION GUARD IS AN OPTIMISATION, NOT AN INVARIANT. It lives in memory,
+// so a process restart re-cuts once at the current version. That is safe
+// because a cut is a pure function of (candidate set, remaining budget,
+// configuration) and its only write is 'ready' -> 'skipped_budget': repeating
+// it with the same inputs defers nothing further, and repeating it with a
+// smaller budget defers more, which is the correct response to a shrinking
+// window. Making the guard durable would need a column, and schema.sql is a
+// frozen interface this step may not extend.
+func (r *Recutter) RecutQueueContext(ctx context.Context, auditID string, remainingBudgetTokens int) (Cut, error) {
+ if remainingBudgetTokens < 0 {
+ return Cut{}, fmt.Errorf("%w, got %d", ErrInvalidBudget, remainingBudgetTokens)
+ }
+ if err := r.cfg.validate(); err != nil {
+ return Cut{}, err
+ }
+
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ auditRecordID, err := ResolveAuditRecordID(ctx, r.db, auditID)
+ if err != nil {
+ return Cut{}, err
+ }
+
+ var (
+ version int64
+ dastStatus string
+ )
+ err = r.db.QueryRowContext(ctx,
+ `SELECT audit_version, dast_status FROM audit_record WHERE audit_record_id = ?`,
+ auditRecordID).Scan(&version, &dastStatus)
+ if errors.Is(err, sql.ErrNoRows) {
+ return Cut{}, fmt.Errorf("store: audit_record %d: %w", auditRecordID, ErrNoSuchAudit)
+ }
+ if err != nil {
+ return Cut{}, fmt.Errorf("store: reading audit_record %d: %w", auditRecordID, err)
+ }
+ if err := record.ValidateDastStatus(dastStatus); err != nil {
+ return Cut{}, fmt.Errorf("store: audit_record %d: %w", auditRecordID, err)
+ }
+
+ cut := Cut{
+ AuditRecordID: auditRecordID,
+ AuditVersion: version,
+ DastStatus: record.DastStatus(dastStatus),
+ RemainingBudgetTokens: remainingBudgetTokens,
+ }
+
+ if previous, seen := r.lastCut[auditRecordID]; seen && previous == version {
+ cut.NotCutReason = fmt.Sprintf(
+ "audit_record %d is still at audit_version %d, already cut at that version; "+
+ "S6 re-cuts on a version bump, not on a handoff write",
+ auditRecordID, version)
+ return cut, nil
+ }
+
+ cut, err = r.performCut(ctx, cut)
+ if err != nil {
+ return cut, err
+ }
+ r.lastCut[auditRecordID] = version
+ return cut, nil
+}
+
+// performCut does the arithmetic and writes the deferrals. The caller holds
+// r.mu.
+func (r *Recutter) performCut(ctx context.Context, cut Cut) (Cut, error) {
+ inFlight, candidates, err := r.loadRows(ctx, cut.AuditRecordID)
+ if err != nil {
+ return cut, err
+ }
+
+ cut.Performed = true
+ cut.LateDastArrivalsPossible = LateDastArrivalsPossible(cut.DastStatus)
+ if cut.LateDastArrivalsPossible {
+ cut.ReserveFraction = r.cfg.reserveFraction()
+ }
+
+ // floor, not round: reserving a token more than the fraction allows would
+ // be the component's own arithmetic quietly outvoting the configuration.
+ cut.ReservedTokens = int(math.Floor(cut.ReserveFraction * float64(cut.RemainingBudgetTokens)))
+ if cut.ReservedTokens > cut.RemainingBudgetTokens {
+ cut.ReservedTokens = cut.RemainingBudgetTokens
+ }
+ cut.OpenTokens = cut.RemainingBudgetTokens - cut.ReservedTokens
+
+ p := &pools{reserve: cut.ReservedTokens, open: cut.OpenTokens}
+
+ // In-flight leases are charged first and cannot be refused: the work is
+ // already happening and the tokens are already being spent. They are the
+ // reason a re-cut must not simply divide the budget among candidates as if
+ // the queue were idle. See this file's header for why they are never
+ // deferred, superseded, or otherwise touched.
+ for i := range inFlight {
+ inFlight[i].CostTokens = r.cfg.cost(inFlight[i])
+ over := p.chargeInFlight(inFlight[i].DastConfirmed(), inFlight[i].CostTokens)
+ cut.InFlightTokens += inFlight[i].CostTokens
+ cut.InFlightOverdraftTokens += over
+ }
+
+ sortCandidates(candidates, r.cfg.SeverityRank)
+
+ // A cut, not a knapsack. research/24 step 9: "everything past the cut is
+ // marked SKIPPED_BUDGET". Once a pool cannot pay for the next candidate in
+ // rank order, that pool is closed and every later candidate drawing on it
+ // is deferred — including a cheaper one. Continuing to scan for something
+ // that fits is exactly how a budget re-orders itself behind the priority
+ // scheme's back.
+ dastClosed, openClosed := false, false
+ for i := range candidates {
+ cand := candidates[i]
+ cand.CostTokens = r.cfg.cost(cand)
+
+ closed := openClosed
+ if cand.DastConfirmed() {
+ closed = dastClosed
+ }
+ if closed || !p.charge(cand.DastConfirmed(), cand.CostTokens) {
+ if cand.DastConfirmed() {
+ // A dast_confirmed row draws on reserve + open. If the pair
+ // cannot pay for it, `open` alone cannot pay for anything at
+ // least as expensive either, so both pools close.
+ dastClosed, openClosed = true, true
+ } else {
+ openClosed = true
+ }
+ cut.Deferred = append(cut.Deferred, cand)
+ continue
+ }
+ cut.Admitted = append(cut.Admitted, cand)
+ }
+
+ deferred, contended, err := r.applyDeferrals(ctx, cut.Deferred)
+ if err != nil {
+ return cut, err
+ }
+ cut.Deferred = deferred
+ cut.Contended = contended
+ return cut, nil
+}
+
+// pools is the two-pool budget. The reserve is drawable only by
+// dast_confirmed; the open pool is drawable by everything.
+type pools struct {
+ reserve int
+ open int
+}
+
+// charge takes cost from the pools, or reports that it will not fit and takes
+// nothing. It never leaves a partial draw behind.
+func (p *pools) charge(dastConfirmed bool, cost int) bool {
+ if !dastConfirmed {
+ if p.open < cost {
+ return false
+ }
+ p.open -= cost
+ return true
+ }
+ if p.reserve+p.open < cost {
+ return false
+ }
+ fromReserve := cost
+ if fromReserve > p.reserve {
+ fromReserve = p.reserve
+ }
+ p.reserve -= fromReserve
+ p.open -= cost - fromReserve
+ return true
+}
+
+// chargeInFlight charges work that is already running and therefore cannot be
+// refused. It returns the overdraft, i.e. how much of the cost the pools could
+// not cover, and drains them rather than going negative.
+func (p *pools) chargeInFlight(dastConfirmed bool, cost int) int {
+ if p.charge(dastConfirmed, cost) {
+ return 0
+ }
+ available := p.open
+ if dastConfirmed {
+ available += p.reserve
+ p.reserve = 0
+ }
+ p.open = 0
+ return cost - available
+}
+
+// loadRows reads the audit's live handoff rows: the leased ones (in flight)
+// and the ready ones (candidates). Terminal rows are neither — they are
+// already paid for and a re-cut has nothing to say about them.
+func (r *Recutter) loadRows(ctx context.Context, auditRecordID int64) (inFlight, candidates []Candidate, err error) {
+ rows, err := r.db.QueryContext(ctx, `
+ SELECT h.handoff_id, h.finding_id, h.fingerprint, h.state, f.evidence_class, f.severity
+ FROM handoff h
+ JOIN finding f ON f.finding_id = h.finding_id
+ WHERE h.audit_record_id = ?
+ AND h.state IN (?, ?)
+ ORDER BY h.handoff_id`,
+ auditRecordID,
+ string(record.HandoffStateReady),
+ string(record.HandoffStateLeased))
+ if err != nil {
+ return nil, nil, fmt.Errorf("store: loading the queue for audit_record %d: %w", auditRecordID, err)
+ }
+ defer func() { _ = rows.Close() }()
+
+ for rows.Next() {
+ var (
+ c Candidate
+ state string
+ class string
+ )
+ if err := rows.Scan(&c.HandoffID, &c.FindingID, &c.Fingerprint, &state, &class, &c.Severity); err != nil {
+ return nil, nil, fmt.Errorf("store: scanning the queue for audit_record %d: %w", auditRecordID, err)
+ }
+ if err := record.ValidateHandoffState(state); err != nil {
+ return nil, nil, fmt.Errorf("store: handoff %d: %w", c.HandoffID, err)
+ }
+ if err := record.ValidateEvidenceClass(class); err != nil {
+ return nil, nil, fmt.Errorf("store: finding %d: %w", c.FindingID, err)
+ }
+ c.State = record.HandoffState(state)
+ c.EvidenceClass = record.EvidenceClass(class)
+
+ if c.State == record.HandoffStateLeased {
+ inFlight = append(inFlight, c)
+ continue
+ }
+ candidates = append(candidates, c)
+ }
+ if err := rows.Err(); err != nil {
+ return nil, nil, fmt.Errorf("store: reading the queue for audit_record %d: %w", auditRecordID, err)
+ }
+ return inFlight, candidates, nil
+}
+
+// applyDeferrals writes 'ready' -> 'skipped_budget' for everything past the
+// cut, in one transaction.
+//
+// The UPDATE is guarded on state = 'ready', so a row a consumer claimed between
+// the SELECT and this write is left exactly as the consumer left it: the claim
+// wins. Losing that race is not an error and does not lose the finding — the
+// row is leased, it will be worked or reclaimed, and the next cut sees it.
+func (r *Recutter) applyDeferrals(ctx context.Context, toDefer []Candidate) (deferred, contended []Candidate, err error) {
+ if len(toDefer) == 0 {
+ return nil, nil, nil
+ }
+ tx, err := r.db.BeginTx(ctx, nil)
+ if err != nil {
+ return nil, nil, fmt.Errorf("store: beginning the queue re-cut: %w", err)
+ }
+ defer func() { _ = tx.Rollback() }()
+
+ now := formatRecutTime(r.cfg.now())
+ for _, cand := range toDefer {
+ res, execErr := tx.ExecContext(ctx,
+ `UPDATE handoff SET state = ?, updated_at = ? WHERE handoff_id = ? AND state = ?`,
+ string(record.HandoffStateSkippedBudget), now, cand.HandoffID, string(record.HandoffStateReady))
+ if execErr != nil {
+ return nil, nil, fmt.Errorf("store: deferring handoff %d as %s: %w",
+ cand.HandoffID, record.HandoffStateSkippedBudget, execErr)
+ }
+ n, rowsErr := res.RowsAffected()
+ if rowsErr != nil {
+ return nil, nil, fmt.Errorf("store: deferring handoff %d as %s: %w",
+ cand.HandoffID, record.HandoffStateSkippedBudget, rowsErr)
+ }
+ if n == 1 {
+ cand.State = record.HandoffStateSkippedBudget
+ deferred = append(deferred, cand)
+ continue
+ }
+ contended = append(contended, cand)
+ }
+ if err := tx.Commit(); err != nil {
+ return nil, nil, fmt.Errorf("store: committing the queue re-cut: %w", err)
+ }
+ return deferred, contended, nil
+}
+
+// sortCandidates puts the queue in rank order: evidence class first (which is
+// research/24 step 7's first rank_key component and the order
+// record.EvidenceClassValues() is documented to return), then the configured
+// severity rank, then finding_id as a total, stable tiebreak.
+//
+// The rest of research/24's rank_key — KEV membership, KEV ransomware use,
+// EPSS, reachability, CVSS base, proximity — is not orderable here: no column
+// in schema.sql carries any of it, and that document is explicit that all six
+// weights come from configuration, never code. RecutConfig.SeverityRank and
+// the deterministic finding_id fallback are what this step can honestly
+// provide; a later step supplies the rest by ordering the candidate set before
+// it reaches the budget arithmetic.
+func sortCandidates(candidates []Candidate, severityRank map[string]int) {
+ sort.SliceStable(candidates, func(i, j int) bool {
+ a, b := candidates[i], candidates[j]
+ if ra, rb := EvidenceClassRank(a.EvidenceClass), EvidenceClassRank(b.EvidenceClass); ra != rb {
+ return ra < rb
+ }
+ if ra, rb := severityOrder(severityRank, a.Severity), severityOrder(severityRank, b.Severity); ra != rb {
+ return ra < rb
+ }
+ return a.FindingID < b.FindingID
+ })
+}
+
+func severityOrder(severityRank map[string]int, severity string) int {
+ if rank, ok := severityRank[severity]; ok {
+ return rank
+ }
+ return math.MaxInt
+}
+
+// evidenceClassRanks is derived from record.EvidenceClassValues(), which
+// internal/record documents as returning "every legal anvil/evidenceClass
+// literal, in descending evidence strength — which is also the default rank
+// order". Deriving it means this file holds no second copy of that ordering and
+// no bare evidence-class literal: adding a class to the frozen enum ranks it
+// here automatically, in the position R.1 put it.
+var evidenceClassRanks = func() map[record.EvidenceClass]int {
+ values := record.EvidenceClassValues()
+ ranks := make(map[record.EvidenceClass]int, len(values))
+ for i, v := range values {
+ ranks[v] = i
+ }
+ return ranks
+}()
+
+// EvidenceClassRank reports an evidence class's position in the frozen enum's
+// documented strength order, lower being stronger. An unknown class ranks last;
+// it cannot reach here through loadRows, which validates against the enum
+// first.
+func EvidenceClassRank(e record.EvidenceClass) int {
+ if rank, ok := evidenceClassRanks[e]; ok {
+ return rank
+ }
+ return math.MaxInt
+}
+
+// LateDastArrivalsPossible reports whether an audit whose DAST half is in this
+// state can still contribute `evidence_class = 'dast_confirmed'` rows to the
+// queue after this cut. It is the gate on the reservation: holding budget for a
+// class that provably cannot arrive would starve the classes that did.
+//
+// The switch is total over record.DastStatusValues() and queue_test.go asserts
+// that, so the eleventh dastStatus value cannot land as a silent default.
+//
+// IT IS NOT internal/handoff's HasDynamicEvidence AND MUST NOT BE ALIGNED WITH
+// IT. That predicate answers "can a reproduction exist NOW for the finding in
+// hand", which gates the `validated` disposition. This one answers "can more
+// proof-carrying findings still show up", which gates a budget reservation.
+// The two genuinely disagree on four values and each disagreement is correct:
+//
+// - running: no reproduction has been sealed yet (HasDynamicEvidence false),
+// but the half is still working and arrivals are exactly what is expected
+// (true here). This is S6's central case.
+// - completed_failed, timed_out: the half did not finish, so it has produced
+// no verdict about any particular finding (HasDynamicEvidence false) — but
+// findings it confirmed before it crashed or ran out of clock are real and
+// still get enqueued (true here). Reserving for them is right even though
+// they cannot reach `validated` on this audit: a proof-carrying finding is
+// still the highest-value patch Anvil can propose, and S7 withholds the
+// "verified fixed" verdict, not the fix.
+//
+// False for exactly the five states in which no dynamic evidence exists or can:
+// not_run (the DAST tier is not installed at all), skipped_no_manifest (it ran
+// and there was nothing to scan), completed_clean (it scanned and found
+// nothing), target_boot_failed and target_unreachable (there was never a live
+// target — the distinction plan/00-SPINE.md S6 exists to preserve).
+func LateDastArrivalsPossible(s record.DastStatus) bool {
+ switch s {
+ case record.DastStatusRunning,
+ record.DastStatusCompletedFindings,
+ record.DastStatusCompletedPartial,
+ record.DastStatusCompletedFailed,
+ record.DastStatusTimedOut:
+ return true
+ case record.DastStatusNotRun,
+ record.DastStatusSkippedNoManifest,
+ record.DastStatusCompletedClean,
+ record.DastStatusTargetBootFailed,
+ record.DastStatusTargetUnreachable:
+ return false
+ default:
+ // Unreachable through RecutQueueContext, which validates against the
+ // frozen enum first. Conservative if it is ever reached another way:
+ // reserving for an unknown state protects the class S6 protects.
+ return true
+ }
+}
+
+// ResolveAuditRecordID turns the R.11 packet's `auditID string` into the store's
+// audit_record primary key.
+//
+// THE PACKET NAMES A STRING AND THE SCHEMA HAS NO STRING KEY. `anvil/auditId`
+// is a required record field (plan/40-record-and-storage.md's Record Field
+// Contract) but schema.sql carries NO `audit_id` column — it is a frozen
+// interface and R.11 may not add one. So this resolver accepts the decimal
+// `audit_record_id` and says exactly that when it cannot.
+//
+// It is deliberately NOT the CRITIQUE-02 F7 mistake. F7 was about EXPORTING a
+// rowid as a portable identity — hashing it into a git trailer where it means
+// nothing outside one copy of one database file. This is the opposite
+// direction: a local lookup key, never emitted, never hashed, never handed to
+// another process. When a later step adds the `anvil/auditId` column, this
+// function is the single place that changes and every caller keeps its
+// signature.
+func ResolveAuditRecordID(ctx context.Context, db *sql.DB, auditID string) (int64, error) {
+ trimmed := strings.TrimSpace(auditID)
+ if trimmed == "" {
+ return 0, fmt.Errorf("store: empty audit id: %w", ErrNoSuchAudit)
+ }
+ id, err := strconv.ParseInt(trimmed, 10, 64)
+ if err != nil || id <= 0 {
+ return 0, fmt.Errorf(
+ "store: audit id %q is not a positive audit_record_id, and schema.sql has no anvil/auditId "+
+ "column to resolve it against: %w", auditID, ErrNoSuchAudit)
+ }
+ var found int64
+ err = db.QueryRowContext(ctx,
+ `SELECT audit_record_id FROM audit_record WHERE audit_record_id = ?`, id).Scan(&found)
+ if errors.Is(err, sql.ErrNoRows) {
+ return 0, fmt.Errorf("store: audit_record %d: %w", id, ErrNoSuchAudit)
+ }
+ if err != nil {
+ return 0, fmt.Errorf("store: resolving audit id %q: %w", auditID, err)
+ }
+ return found, nil
+}
diff --git a/internal/store/queue_test.go b/internal/store/queue_test.go
new file mode 100644
index 0000000..98cdd09
--- /dev/null
+++ b/internal/store/queue_test.go
@@ -0,0 +1,969 @@
+package store
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/Susquehanna-Syntax/Anvil/internal/record"
+
+ _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12
+)
+
+// ---------------------------------------------------------------------------
+// Fixture
+// ---------------------------------------------------------------------------
+
+const (
+ recutTargetID = 1
+ recutScanRunID = 1
+ recutAuditRecordID = 1
+ recutAuditID = "1"
+ recutTokens = 10000
+)
+
+// recutFixture is one audit with a queue behind it. Every row goes in through
+// schema.sql's own constraints, so a fixture that could not exist in production
+// fails here rather than proving something about a shape the store forbids.
+type recutFixture struct {
+ t *testing.T
+ db *sql.DB
+ nextFinding int64
+ nextHandoff int64
+}
+
+func newRecutFixture(t *testing.T, dastStatus record.DastStatus) *recutFixture {
+ t.Helper()
+ db := newDB(t)
+
+ mustExec(t, db,
+ `INSERT INTO target (target_id, kind, locator) VALUES (?, 'repo', 'https://example.invalid/r.git')`,
+ recutTargetID)
+ mustExec(t, db,
+ `INSERT INTO scan_run (scan_run_id, target_id, trigger_ref, commit_sha, started_at, status, ruleset_version)
+ VALUES (?, ?, 'v1.0.0', 'deadbeef', '2026-08-08T00:00:00Z', ?, 'rs/1')`,
+ recutScanRunID, recutTargetID, string(record.ScanRunStatusOK))
+ mustExec(t, db,
+ `INSERT INTO audit_record (audit_record_id, scan_run_id, schema_version, audit_version, state,
+ sast_status, sast_sealed_at, dast_status, target_provenance,
+ deadline_at, payload_sha256, created_at)
+ VALUES (?, ?, ?, 1, ?, ?, '2026-08-08T01:00:00Z', ?, ?, '2026-08-08T08:00:00Z', ?, '2026-08-08T00:00:00Z')`,
+ recutAuditRecordID, recutScanRunID, record.SchemaVersion,
+ string(record.StateSastSealed), string(record.HalfStatusSealed),
+ string(dastStatus), string(record.TargetProvenanceBootedClean),
+ "00112233445566778899001122334455667788990011223344556677889900aa")
+
+ return &recutFixture{t: t, db: db, nextFinding: 1, nextHandoff: 1}
+}
+
+// detectorFor maps an evidence class onto a legal `finding.detector`. It exists
+// so the fixture never writes a bare literal for either vocabulary.
+func detectorFor(t *testing.T, class record.EvidenceClass) record.DetectorKind {
+ t.Helper()
+ switch class {
+ case record.EvidenceClassDastConfirmed:
+ return record.DetectorKindDast
+ case record.EvidenceClassSastReachable, record.EvidenceClassSastStaticOnly:
+ return record.DetectorKindSast
+ case record.EvidenceClassSCA:
+ return record.DetectorKindSCA
+ case record.EvidenceClassHost:
+ return record.DetectorKindHost
+ default:
+ t.Fatalf("no detector mapping for evidence class %q", class)
+ return ""
+ }
+}
+
+func consumptionFor(class record.EvidenceClass) record.ConsumptionClass {
+ if class == record.EvidenceClassDastConfirmed {
+ return record.ConsumptionClassRequiresDynamicConfirmation
+ }
+ return record.ConsumptionClassStaticOnly
+}
+
+// enqueue inserts one finding and one ready handoff row for it, returning the
+// handoff_id.
+func (f *recutFixture) enqueue(class record.EvidenceClass, severity string) int64 {
+ f.t.Helper()
+
+ findingID := f.nextFinding
+ handoffID := f.nextHandoff
+ f.nextFinding++
+ f.nextHandoff++
+
+ detector := detectorFor(f.t, class)
+ remediable := 1
+ if detector == record.DetectorKindHost {
+ remediable = 0
+ }
+ fp := fmt.Sprintf("%064x", findingID)
+
+ mustExec(f.t, f.db,
+ `INSERT INTO finding (finding_id, target_id, fingerprint, detector, evidence_class, rule_id,
+ remediable_by_agent, severity, title, state, first_seen_scan, first_seen_at)
+ VALUES (?, ?, ?, ?, ?, 'anvil.py.sqli/v3', ?, ?, 'finding', ?, ?, '2026-08-08T00:30:00Z')`,
+ findingID, recutTargetID, fp, string(detector), string(class), remediable, severity,
+ string(record.FindingStateOpen), recutScanRunID)
+
+ mustExec(f.t, f.db,
+ `INSERT INTO handoff (handoff_id, finding_id, audit_record_id, fingerprint, state, consumption_class,
+ attempts, max_attempts, created_at, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, 0, 2, '2026-08-08T00:00:00Z', '2026-08-08T00:00:00Z')`,
+ handoffID, findingID, recutAuditRecordID, fp,
+ string(record.HandoffStateReady), string(consumptionFor(class)))
+
+ return handoffID
+}
+
+func (f *recutFixture) enqueueMany(n int, class record.EvidenceClass, severity string) []int64 {
+ f.t.Helper()
+ ids := make([]int64, 0, n)
+ for i := 0; i < n; i++ {
+ ids = append(ids, f.enqueue(class, severity))
+ }
+ return ids
+}
+
+// lease moves a ready row to 'leased' with a holder, satisfying
+// ck_handoff_lease_requires_holder.
+func (f *recutFixture) lease(handoffID int64, worker string) {
+ f.t.Helper()
+ mustExec(f.t, f.db,
+ `UPDATE handoff SET state = ?, claimed_by = ?, lease_expires_at = '2026-08-08T00:20:00Z',
+ attempts = attempts + 1, updated_at = '2026-08-08T00:05:00Z'
+ WHERE handoff_id = ? AND state = ?`,
+ string(record.HandoffStateLeased), worker, handoffID, string(record.HandoffStateReady))
+}
+
+// consume walks a row the long way — ready -> leased -> validated — so the
+// fixture never implies an edge internal/handoff's state machine forbids. It
+// is how the test spends budget between two cuts.
+func (f *recutFixture) consume(handoffIDs []int64) {
+ f.t.Helper()
+ for _, id := range handoffIDs {
+ f.lease(id, "worker-consumer")
+ mustExec(f.t, f.db,
+ `UPDATE handoff SET state = ?, claimed_by = NULL, lease_expires_at = NULL,
+ updated_at = '2026-08-08T00:10:00Z'
+ WHERE handoff_id = ? AND state = ?`,
+ string(record.HandoffStateValidated), id, string(record.HandoffStateLeased))
+ }
+}
+
+func (f *recutFixture) bumpVersion(to int64, dastStatus record.DastStatus) {
+ f.t.Helper()
+ mustExec(f.t, f.db,
+ `UPDATE audit_record SET audit_version = ?, dast_status = ?, state = ?
+ WHERE audit_record_id = ?`,
+ to, string(dastStatus), string(record.StateBothSealed), recutAuditRecordID)
+}
+
+func (f *recutFixture) state(handoffID int64) record.HandoffState {
+ f.t.Helper()
+ var s string
+ if err := f.db.QueryRow(`SELECT state FROM handoff WHERE handoff_id = ?`, handoffID).Scan(&s); err != nil {
+ f.t.Fatalf("reading handoff %d state: %v", handoffID, err)
+ }
+ if err := record.ValidateHandoffState(s); err != nil {
+ f.t.Fatalf("handoff %d holds an illegal state: %v", handoffID, err)
+ }
+ return record.HandoffState(s)
+}
+
+func (f *recutFixture) countState(s record.HandoffState) int {
+ f.t.Helper()
+ var n int
+ if err := f.db.QueryRow(
+ `SELECT COUNT(*) FROM handoff WHERE audit_record_id = ? AND state = ?`,
+ recutAuditRecordID, string(s)).Scan(&n); err != nil {
+ f.t.Fatalf("counting handoff rows in %s: %v", s, err)
+ }
+ return n
+}
+
+func (f *recutFixture) recutter(cfg RecutConfig) *Recutter {
+ f.t.Helper()
+ if cfg.TokensPerCandidate == 0 {
+ cfg.TokensPerCandidate = recutTokens
+ }
+ if cfg.Clock == nil {
+ cfg.Clock = func() time.Time { return time.Date(2026, 8, 8, 2, 0, 0, 0, time.UTC) }
+ }
+ r, err := NewRecutter(f.db, cfg)
+ if err != nil {
+ f.t.Fatalf("NewRecutter: %v", err)
+ }
+ return r
+}
+
+func mustExec(t *testing.T, db *sql.DB, query string, args ...any) {
+ t.Helper()
+ if _, err := db.Exec(query, args...); err != nil {
+ t.Fatalf("exec %s: %v", firstLine(query), err)
+ }
+}
+
+func firstLine(s string) string {
+ for i := 0; i < len(s); i++ {
+ if s[i] == '\n' {
+ return s[:i]
+ }
+ }
+ return s
+}
+
+func mustRecut(t *testing.T, r *Recutter, remaining int) Cut {
+ t.Helper()
+ cut, err := r.RecutQueueContext(context.Background(), recutAuditID, remaining)
+ if err != nil {
+ t.Fatalf("RecutQueueContext(%d): %v", remaining, err)
+ }
+ return cut
+}
+
+// ---------------------------------------------------------------------------
+// THE test: the inversion S6 describes, reproduced and then prevented.
+// ---------------------------------------------------------------------------
+
+// arrivalSequence is one run of plan/00-SPINE.md S6's scenario, driven by a
+// single knob: the reserve fraction. Everything else — the findings, their
+// arrival order, the window size, the per-candidate cost — is identical between
+// runs, so any difference in outcome is attributable to the reservation and to
+// nothing else.
+//
+// The sequence is the real one, in the real order:
+//
+// t0 version 1. The SAST half has sealed. 20 static findings are queued.
+// NOTHING IS dast_confirmed YET — that is the whole point; the DAST half
+// is still running, so ranking by evidence class has nothing to rank.
+// t1 the queue is cut against the full window.
+// t2 consumers work everything the cut admitted. Those tokens are gone.
+// t3 the DAST half seals with findings. anvil/version bumps to 2 and five
+// dast_confirmed findings — each carrying a runtime reproduction — are
+// enqueued.
+// t4 the queue is re-cut against what is LEFT of the window.
+type arrivalOutcome struct {
+ firstCut Cut
+ secondCut Cut
+ dastIDs []int64
+ sastIDs []int64
+ fixture *recutFixture
+}
+
+func runArrivalSequence(t *testing.T, fraction *float64, window, dastCount int) arrivalOutcome {
+ t.Helper()
+
+ // t0 — the DAST half is still running, so late arrivals are possible.
+ f := newRecutFixture(t, record.DastStatusRunning)
+ sastIDs := f.enqueueMany(20, record.EvidenceClassSastReachable, "high")
+
+ r := f.recutter(RecutConfig{DastReserveFraction: fraction})
+
+ // t1 — first cut, full window.
+ first := mustRecut(t, r, window)
+ if !first.Performed {
+ t.Fatalf("first cut did not run: %s", first.NotCutReason)
+ }
+ if got := first.AdmittedDastConfirmedTokens(); got != 0 {
+ t.Fatalf("first cut admitted %d dast_confirmed tokens; nothing is dast_confirmed when the queue is first cut", got)
+ }
+
+ // t2 — the admitted static work is done. Those tokens are spent for good.
+ admitted := make([]int64, 0, len(first.Admitted))
+ for _, c := range first.Admitted {
+ admitted = append(admitted, c.HandoffID)
+ }
+ f.consume(admitted)
+ remaining := window - first.AdmittedTokens()
+
+ // t3 — the DAST half seals with findings; anvil/version bumps; the
+ // proof-carrying findings arrive.
+ f.bumpVersion(2, record.DastStatusCompletedFindings)
+ dastIDs := f.enqueueMany(dastCount, record.EvidenceClassDastConfirmed, "high")
+
+ // t4 — re-cut against REMAINING budget.
+ second := mustRecut(t, r, remaining)
+ if !second.Performed {
+ t.Fatalf("the version bump did not re-cut the queue: %s", second.NotCutReason)
+ }
+ return arrivalOutcome{firstCut: first, secondCut: second, dastIDs: dastIDs, sastIDs: sastIDs, fixture: f}
+}
+
+// TestRecutInvertsPriorityWithoutReservationAndDoesNotWithIt is the test the
+// R.11 packet demands: it demonstrates the inversion happening without the
+// reservation and not happening with it. Asserting that a float equals 0.5
+// would prove nothing about either.
+func TestRecutInvertsPriorityWithoutReservationAndDoesNotWithIt(t *testing.T) {
+ const (
+ window = 200000 // 20 static candidates * 10,000 tok
+ dastCount = 5 // 50,000 tok of proof-carrying work, arriving late
+ dastDemand = dastCount * recutTokens
+ )
+
+ t.Run("no reservation inverts the priority scheme", func(t *testing.T) {
+ out := runArrivalSequence(t, ReserveFraction(0), window, dastCount)
+
+ // The first cut spent the entire window on findings that carry no
+ // proof, because at that moment there was nothing else to spend it on.
+ if got, want := out.firstCut.ReservedTokens, 0; got != want {
+ t.Fatalf("ReservedTokens = %d, want %d", got, want)
+ }
+ if got, want := len(out.firstCut.Admitted), 20; got != want {
+ t.Fatalf("first cut admitted %d static findings, want %d (the whole window)", got, want)
+ }
+ if got, want := out.firstCut.AdmittedTokens(), window; got != want {
+ t.Fatalf("first cut committed %d tok of a %d tok window, want the lot", got, want)
+ }
+
+ // THE INVERSION. Every finding with a working reproduction is now
+ // "found, not fixed", while 20 static alarms got the window.
+ if got, want := len(out.secondCut.Deferred), dastCount; got != want {
+ t.Fatalf("second cut deferred %d dast_confirmed findings, want %d", got, want)
+ }
+ if got := out.secondCut.AdmittedDastConfirmedTokens(); got != 0 {
+ t.Fatalf("second cut admitted %d dast_confirmed tokens; the window was already spent", got)
+ }
+ for _, id := range out.dastIDs {
+ if got := out.fixture.state(id); got != record.HandoffStateSkippedBudget {
+ t.Fatalf("dast_confirmed handoff %d is %s, want %s", id, got, record.HandoffStateSkippedBudget)
+ }
+ }
+ // ... and the inversion is exactly that a weaker class was admitted
+ // while the strongest was not.
+ if !crossCutInversion(out) {
+ t.Fatal("expected the cross-cut priority inversion S6 describes, and it did not occur")
+ }
+ })
+
+ t.Run("the default reservation prevents it", func(t *testing.T) {
+ out := runArrivalSequence(t, nil, window, dastCount) // nil selects DefaultDastReserveFraction
+
+ if got, want := out.firstCut.ReserveFraction, DefaultDastReserveFraction; got != want {
+ t.Fatalf("ReserveFraction = %v, want the documented default %v", got, want)
+ }
+ reserved := out.firstCut.ReservedTokens
+ if got, want := reserved, window/2; got != want {
+ t.Fatalf("ReservedTokens = %d, want %d", got, want)
+ }
+ if got, want := len(out.firstCut.Admitted), 10; got != want {
+ t.Fatalf("first cut admitted %d static findings, want %d (half the window)", got, want)
+ }
+
+ // Every proof-carrying finding arriving after the bump is admitted.
+ //
+ // A reservation guarantees AVAILABILITY, not consumption: here the late
+ // demand (50,000 tok) is below the 100,000 held back, so the right
+ // assertion is that the class took everything it asked for.
+ // TestRecutGivesDastConfirmedAtLeastTheReservation drives the other
+ // side, where demand exceeds the reserve.
+ if len(out.secondCut.Deferred) != 0 {
+ t.Fatalf("second cut deferred %d dast_confirmed findings, want none", len(out.secondCut.Deferred))
+ }
+ if want := min(reserved, dastDemand); out.secondCut.AdmittedDastConfirmedTokens() < want {
+ t.Fatalf("dast_confirmed findings received %d tok, want at least %d",
+ out.secondCut.AdmittedDastConfirmedTokens(), want)
+ }
+ for _, id := range out.dastIDs {
+ if got := out.fixture.state(id); got != record.HandoffStateReady {
+ t.Fatalf("dast_confirmed handoff %d is %s, want %s", id, got, record.HandoffStateReady)
+ }
+ }
+ if crossCutInversion(out) {
+ t.Fatal("the reservation was applied and the priority scheme inverted anyway")
+ }
+ })
+}
+
+// crossCutInversion is S6's failure mode stated as a predicate over the whole
+// arrival sequence rather than over one cut: a weaker evidence class was
+// admitted earlier, and a dast_confirmed finding arriving later found no budget.
+func crossCutInversion(out arrivalOutcome) bool {
+ weakerAdmitted := false
+ for _, c := range out.firstCut.Admitted {
+ if !c.DastConfirmed() {
+ weakerAdmitted = true
+ break
+ }
+ }
+ return weakerAdmitted && out.secondCut.DeferredDastConfirmed() > 0
+}
+
+// TestRecutReserveIsConfigDriven runs the same sequence at the default 50% and
+// at an overridden 25%, which is the R.11 packet's stop condition: the value
+// must be read from configuration, not compiled in. A change in the
+// configuration must change the arithmetic AND the row states, or the knob is
+// decorative.
+func TestRecutReserveIsConfigDriven(t *testing.T) {
+ const (
+ window = 200000
+ dastCount = 5
+ dastDemand = dastCount * recutTokens
+ )
+
+ cases := []struct {
+ name string
+ fraction *float64
+ wantFraction float64
+ wantReserved int
+ wantFirstAdmitted int
+ }{
+ {"default 50%", nil, 0.5, 100000, 10},
+ {"overridden 50%", ReserveFraction(0.5), 0.5, 100000, 10},
+ {"overridden 25%", ReserveFraction(0.25), 0.25, 50000, 15},
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ out := runArrivalSequence(t, tc.fraction, window, dastCount)
+
+ if got := out.firstCut.ReserveFraction; got != tc.wantFraction {
+ t.Fatalf("ReserveFraction = %v, want %v", got, tc.wantFraction)
+ }
+ if got := out.firstCut.ReservedTokens; got != tc.wantReserved {
+ t.Fatalf("ReservedTokens = %d, want %d", got, tc.wantReserved)
+ }
+ // The configuration changed the row states, not just a report
+ // field: a different fraction admits a different number of static
+ // findings at the first cut.
+ if got := len(out.firstCut.Admitted); got != tc.wantFirstAdmitted {
+ t.Fatalf("first cut admitted %d, want %d", got, tc.wantFirstAdmitted)
+ }
+
+ // The packet's own validation clause: dast_confirmed findings
+ // arriving after the bump receive at least the configured fraction
+ // of the budget that remained when the reservation was made — or
+ // all of what they asked for, when that is less.
+ if want := min(tc.wantReserved, dastDemand); out.secondCut.AdmittedDastConfirmedTokens() < want {
+ t.Fatalf("dast_confirmed findings received %d tok, want at least %d (%v of the %d remaining at the cut)",
+ out.secondCut.AdmittedDastConfirmedTokens(), want, tc.wantFraction, window)
+ }
+ if crossCutInversion(out) {
+ t.Fatalf("priority inverted at fraction %v", tc.wantFraction)
+ }
+ })
+ }
+}
+
+// TestRecutGivesDastConfirmedAtLeastTheReservation drives the side the
+// inversion test does not: late dynamic demand that EXCEEDS the reserve. The
+// claim under test is the packet's literal one — the class receives at least
+// the configured fraction of the budget remaining at the cut that reserved it —
+// and it is only meaningful when the class could have taken more.
+func TestRecutGivesDastConfirmedAtLeastTheReservation(t *testing.T) {
+ const (
+ window = 200000
+ dastCount = 12 // 120,000 tok of demand against a 100,000 tok reserve
+ )
+
+ cases := []struct {
+ name string
+ fraction *float64
+ wantReserved int
+ wantAdmitted int
+ }{
+ {"default 50%", nil, 100000, 10},
+ {"overridden 25%", ReserveFraction(0.25), 50000, 5},
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ out := runArrivalSequence(t, tc.fraction, window, dastCount)
+
+ if got := out.firstCut.ReservedTokens; got != tc.wantReserved {
+ t.Fatalf("ReservedTokens = %d, want %d", got, tc.wantReserved)
+ }
+ if got := len(out.secondCut.Admitted); got != tc.wantAdmitted {
+ t.Fatalf("second cut admitted %d dast_confirmed findings, want %d", got, tc.wantAdmitted)
+ }
+ if got := out.secondCut.AdmittedDastConfirmedTokens(); got < tc.wantReserved {
+ t.Fatalf("dast_confirmed findings received %d tok, want at least the %d reserved for them",
+ got, tc.wantReserved)
+ }
+ // The overflow is deferred, not dropped: research/24 step 9's
+ // "found, not fixed".
+ for _, c := range out.secondCut.Deferred {
+ if out.fixture.state(c.HandoffID) != record.HandoffStateSkippedBudget {
+ t.Fatalf("overflow handoff %d was not recorded as %s", c.HandoffID, record.HandoffStateSkippedBudget)
+ }
+ }
+ })
+ }
+
+ // Control: with no reservation the same demand gets nothing at all.
+ out := runArrivalSequence(t, ReserveFraction(0), window, dastCount)
+ if got := out.secondCut.AdmittedDastConfirmedTokens(); got != 0 {
+ t.Fatalf("unreserved control admitted %d dast_confirmed tokens, want 0", got)
+ }
+ if !crossCutInversion(out) {
+ t.Fatal("unreserved control did not invert the priority scheme")
+ }
+}
+
+// TestRecutReservesFromRemainingNotTotalBudget pins the R.11 packet's
+// Forbidden action: "Do not let the reservation apply to total budget rather
+// than *remaining* budget at re-cut time." The window is the same; only what is
+// left of it differs.
+func TestRecutReservesFromRemainingNotTotalBudget(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ f.enqueueMany(20, record.EvidenceClassSastReachable, "high")
+ r := f.recutter(RecutConfig{})
+
+ first := mustRecut(t, r, 200000)
+ if got, want := first.ReservedTokens, 100000; got != want {
+ t.Fatalf("first cut ReservedTokens = %d, want %d", got, want)
+ }
+
+ // Time passed and the window shrank. The reservation must shrink with it.
+ f.bumpVersion(2, record.DastStatusCompletedFindings)
+ second := mustRecut(t, r, 40000)
+
+ if got, want := second.RemainingBudgetTokens, 40000; got != want {
+ t.Fatalf("RemainingBudgetTokens = %d, want %d", got, want)
+ }
+ if got, want := second.ReservedTokens, 20000; got != want {
+ t.Fatalf("second cut ReservedTokens = %d, want %d (half of REMAINING, not of the 200000 total)", got, want)
+ }
+ if got, want := second.OpenTokens, 20000; got != want {
+ t.Fatalf("second cut OpenTokens = %d, want %d", got, want)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The trigger
+// ---------------------------------------------------------------------------
+
+// TestRecutTriggersOnVersionBumpAndNotOnHandoffWrites pins the other Forbidden
+// action: "Do not re-cut on every write to `handoff` — only on an
+// `audit_record.audit_version` bump."
+func TestRecutTriggersOnVersionBumpAndNotOnHandoffWrites(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ f.enqueueMany(4, record.EvidenceClassSastReachable, "high")
+ r := f.recutter(RecutConfig{})
+
+ first := mustRecut(t, r, 20000)
+ if !first.Performed {
+ t.Fatalf("first cut did not run: %s", first.NotCutReason)
+ }
+ if got, want := len(first.Admitted), 1; got != want {
+ t.Fatalf("admitted %d, want %d", got, want)
+ }
+
+ // A write to `handoff` — a new finding enqueued, no version bump.
+ late := f.enqueue(record.EvidenceClassSastReachable, "high")
+
+ second := mustRecut(t, r, 20000)
+ if second.Performed {
+ t.Fatal("a handoff write re-cut the queue; only an audit_version bump may")
+ }
+ if second.NotCutReason == "" {
+ t.Fatal("a declined cut must say which guard declined")
+ }
+ if got := f.state(late); got != record.HandoffStateReady {
+ t.Fatalf("the late row is %s after a declined cut, want it untouched at %s", got, record.HandoffStateReady)
+ }
+
+ // Bump the version and the same call now re-cuts.
+ f.bumpVersion(2, record.DastStatusCompletedFindings)
+ third := mustRecut(t, r, 20000)
+ if !third.Performed {
+ t.Fatalf("a version bump did not re-cut: %s", third.NotCutReason)
+ }
+ if got := f.state(late); got != record.HandoffStateSkippedBudget {
+ t.Fatalf("the late row is %s, want %s after a re-cut it did not fit", got, record.HandoffStateSkippedBudget)
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The CRITIQUE-02 §7 decision, asserted
+// ---------------------------------------------------------------------------
+
+// TestRecutLeavesLeasedRowsAloneAndNeverWritesSuperseded is the executable form
+// of this file's ruling on CRITIQUE-02 §7: a version bump does NOT dispose the
+// rows leased at the old version. internal/handoff's checkRecordVersion makes
+// such a lease unable to renew, release or touch its packet, and its reaper is
+// the only component authorised to take a lease its holder still has. A re-cut
+// that yanked it would be the "expire a live claim" that research/08 §4 point 2
+// forbids and that CRITIQUE-02 verdict (e) currently passes.
+func TestRecutLeavesLeasedRowsAloneAndNeverWritesSuperseded(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ ids := f.enqueueMany(6, record.EvidenceClassSastReachable, "high")
+ f.lease(ids[0], "worker-old-version")
+
+ r := f.recutter(RecutConfig{})
+
+ // Budget for two candidates' worth. The leased row is charged first, so
+ // only one further row can be admitted from the open half.
+ f.bumpVersion(2, record.DastStatusCompletedFindings)
+ cut := mustRecut(t, r, 40000)
+
+ if got, want := cut.InFlightTokens, recutTokens; got != want {
+ t.Fatalf("InFlightTokens = %d, want %d (the leased row is charged, not deferred)", got, want)
+ }
+ if got := f.state(ids[0]); got != record.HandoffStateLeased {
+ t.Fatalf("the leased row is %s after a version-bump re-cut, want %s", got, record.HandoffStateLeased)
+ }
+ var claimedBy string
+ if err := f.db.QueryRow(`SELECT claimed_by FROM handoff WHERE handoff_id = ?`, ids[0]).Scan(&claimedBy); err != nil {
+ t.Fatalf("reading claimed_by: %v", err)
+ }
+ if claimedBy != "worker-old-version" {
+ t.Fatalf("claimed_by = %q, want the original holder", claimedBy)
+ }
+
+ // No row anywhere in the audit was superseded or withdrawn: a budget
+ // decision's only disposition is skipped_budget.
+ for _, forbidden := range []record.HandoffState{
+ record.HandoffStateSuperseded,
+ record.HandoffStateWithdrawn,
+ record.HandoffStateExpired,
+ } {
+ if n := f.countState(forbidden); n != 0 {
+ t.Fatalf("the re-cut wrote %d rows as %s; a budget decision defers as %s and nothing else",
+ n, forbidden, record.HandoffStateSkippedBudget)
+ }
+ }
+ for _, c := range cut.Deferred {
+ if c.State != record.HandoffStateSkippedBudget {
+ t.Fatalf("deferred candidate %d reported state %s, want %s", c.HandoffID, c.State, record.HandoffStateSkippedBudget)
+ }
+ }
+}
+
+// TestRecutIsMonotoneAndCannotReadmit records the interaction with
+// internal/handoff's state machine, in which skipped_budget is terminal
+// ("Terminal is terminal"): a later cut with a bigger budget must not — and
+// cannot — pull a deferred row back to 'ready'.
+func TestRecutIsMonotoneAndCannotReadmit(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ ids := f.enqueueMany(4, record.EvidenceClassSastReachable, "high")
+ r := f.recutter(RecutConfig{})
+
+ first := mustRecut(t, r, 20000)
+ if got, want := len(first.Deferred), 3; got != want {
+ t.Fatalf("first cut deferred %d, want %d", got, want)
+ }
+
+ f.bumpVersion(2, record.DastStatusCompletedFindings)
+ second := mustRecut(t, r, 10000000)
+
+ if len(second.Deferred) != 0 {
+ t.Fatalf("second cut deferred %d with an enormous budget, want none", len(second.Deferred))
+ }
+ if got, want := len(second.Admitted), 1; got != want {
+ t.Fatalf("second cut saw %d candidates, want %d — deferred rows are terminal and out of the candidate set", got, want)
+ }
+ for _, id := range ids[1:] {
+ if got := f.state(id); got != record.HandoffStateSkippedBudget {
+ t.Fatalf("handoff %d is %s, want it to have stayed %s", id, got, record.HandoffStateSkippedBudget)
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// The reservation's gate
+// ---------------------------------------------------------------------------
+
+// TestLateDastArrivalsPossibleIsTotalOverTheFrozenEnum asserts the predicate
+// decides every one of the ten frozen anvil/dastStatus values explicitly, so a
+// future eleventh value cannot land as a silent default.
+func TestLateDastArrivalsPossibleIsTotalOverTheFrozenEnum(t *testing.T) {
+ want := map[record.DastStatus]bool{
+ record.DastStatusNotRun: false,
+ record.DastStatusSkippedNoManifest: false,
+ record.DastStatusRunning: true,
+ record.DastStatusCompletedClean: false,
+ record.DastStatusCompletedFindings: true,
+ record.DastStatusCompletedPartial: true,
+ record.DastStatusCompletedFailed: true,
+ record.DastStatusTargetBootFailed: false,
+ record.DastStatusTargetUnreachable: false,
+ record.DastStatusTimedOut: true,
+ }
+
+ values := record.DastStatusValues()
+ if got, want := len(values), 10; got != want {
+ t.Fatalf("record.DastStatusValues() has %d values, want %d; "+
+ "plan/IMPLEMENTATION-PLAN.md §6 freezes ten. Decide the new one here.", got, want)
+ }
+ for _, v := range values {
+ expected, ok := want[v]
+ if !ok {
+ t.Fatalf("dastStatus %q has no reservation ruling in this test or in LateDastArrivalsPossible", v)
+ }
+ if got := LateDastArrivalsPossible(v); got != expected {
+ t.Fatalf("LateDastArrivalsPossible(%q) = %v, want %v", v, got, expected)
+ }
+ }
+ if len(want) != len(values) {
+ t.Fatalf("the expectation table has %d entries and the enum has %d", len(want), len(values))
+ }
+}
+
+// TestRecutReleasesTheReserveWhenNoLateArrivalIsPossible: holding budget for a
+// class that provably cannot arrive would starve the classes that did.
+func TestRecutReleasesTheReserveWhenNoLateArrivalIsPossible(t *testing.T) {
+ for _, status := range []record.DastStatus{
+ record.DastStatusNotRun,
+ record.DastStatusSkippedNoManifest,
+ record.DastStatusCompletedClean,
+ record.DastStatusTargetBootFailed,
+ record.DastStatusTargetUnreachable,
+ } {
+ t.Run(string(status), func(t *testing.T) {
+ f := newRecutFixture(t, status)
+ f.enqueueMany(20, record.EvidenceClassSastReachable, "high")
+ r := f.recutter(RecutConfig{})
+
+ cut := mustRecut(t, r, 200000)
+ if cut.LateDastArrivalsPossible {
+ t.Fatalf("dast_status %q: late arrivals reported possible", status)
+ }
+ if cut.ReserveFraction != 0 || cut.ReservedTokens != 0 {
+ t.Fatalf("dast_status %q: reserved %d tok at fraction %v, want nothing held back",
+ status, cut.ReservedTokens, cut.ReserveFraction)
+ }
+ if got, want := cut.OpenTokens, 200000; got != want {
+ t.Fatalf("OpenTokens = %d, want the whole %d", got, want)
+ }
+ if got, want := len(cut.Admitted), 20; got != want {
+ t.Fatalf("admitted %d, want %d", got, want)
+ }
+ })
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Ordering, vocabulary, and the boring but load-bearing parts
+// ---------------------------------------------------------------------------
+
+// TestEvidenceClassRankFollowsTheFrozenEnumOrder proves the rank is derived
+// from record.EvidenceClassValues() and is not a second copy of that ordering.
+func TestEvidenceClassRankFollowsTheFrozenEnumOrder(t *testing.T) {
+ values := record.EvidenceClassValues()
+ if values[0] != record.EvidenceClassDastConfirmed {
+ t.Fatalf("the frozen enum no longer leads with %s; the reservation's premise changed",
+ record.EvidenceClassDastConfirmed)
+ }
+ for i, v := range values {
+ if got := EvidenceClassRank(v); got != i {
+ t.Fatalf("EvidenceClassRank(%q) = %d, want %d", v, got, i)
+ }
+ }
+ for i := 1; i < len(values); i++ {
+ if EvidenceClassRank(values[i-1]) >= EvidenceClassRank(values[i]) {
+ t.Fatalf("rank is not strictly increasing at %q -> %q", values[i-1], values[i])
+ }
+ }
+}
+
+// TestRecutRanksDastConfirmedAheadOfEveryWeakerClass drives a mixed queue too
+// small for all of it and checks who survives the cut.
+func TestRecutRanksDastConfirmedAheadOfEveryWeakerClass(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusCompletedFindings)
+ // Deliberately enqueued weakest-first, so an implementation that honoured
+ // insertion order rather than rank would fail here.
+ host := f.enqueue(record.EvidenceClassHost, "high")
+ sca := f.enqueue(record.EvidenceClassSCA, "high")
+ static := f.enqueue(record.EvidenceClassSastStaticOnly, "high")
+ reachable := f.enqueue(record.EvidenceClassSastReachable, "high")
+ confirmed := f.enqueue(record.EvidenceClassDastConfirmed, "high")
+
+ r := f.recutter(RecutConfig{})
+ // 20,000 tok: reserve 10,000 (one dast_confirmed row), open 10,000 (one
+ // weaker row).
+ cut := mustRecut(t, r, 20000)
+
+ if got := f.state(confirmed); got != record.HandoffStateReady {
+ t.Fatalf("the dast_confirmed row is %s, want %s", got, record.HandoffStateReady)
+ }
+ if got := f.state(reachable); got != record.HandoffStateReady {
+ t.Fatalf("the strongest static row is %s, want %s", got, record.HandoffStateReady)
+ }
+ for name, id := range map[string]int64{"sast_static_only": static, "sca": sca, "host": host} {
+ if got := f.state(id); got != record.HandoffStateSkippedBudget {
+ t.Fatalf("the %s row is %s, want %s", name, got, record.HandoffStateSkippedBudget)
+ }
+ }
+ if cut.InvertedPriority() {
+ t.Fatal("InvertedPriority reported an inversion in a correctly ordered cut")
+ }
+}
+
+// TestRecutSeverityRankOrdersWithinAClass checks the configured severity order
+// is honoured, and that it is configuration rather than a vocabulary this file
+// froze.
+func TestRecutSeverityRankOrdersWithinAClass(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusCompletedClean)
+ low := f.enqueue(record.EvidenceClassSastReachable, "low")
+ critical := f.enqueue(record.EvidenceClassSastReachable, "critical")
+ unknown := f.enqueue(record.EvidenceClassSastReachable, "spicy")
+
+ r := f.recutter(RecutConfig{
+ SeverityRank: map[string]int{"critical": 0, "high": 1, "medium": 2, "low": 3},
+ })
+ cut := mustRecut(t, r, 10000)
+
+ if got, want := len(cut.Admitted), 1; got != want {
+ t.Fatalf("admitted %d, want %d", got, want)
+ }
+ if cut.Admitted[0].HandoffID != critical {
+ t.Fatalf("admitted handoff %d, want the 'critical' row %d", cut.Admitted[0].HandoffID, critical)
+ }
+ // An unranked severity sorts last, not first: an unknown token must never
+ // outrank a configured one.
+ if f.state(low) != record.HandoffStateSkippedBudget || f.state(unknown) != record.HandoffStateSkippedBudget {
+ t.Fatal("the lower-ranked rows were not deferred")
+ }
+}
+
+// TestRecutDeferralUsesTheFrozenLiteralAndHandoffTimestampFormat pins both
+// halves of the write: the state token comes from the frozen enum, and
+// updated_at is in the exact layout internal/handoff parses.
+func TestRecutDeferralUsesTheFrozenLiteralAndHandoffTimestampFormat(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusCompletedClean)
+ ids := f.enqueueMany(2, record.EvidenceClassSastReachable, "high")
+ at := time.Date(2026, 8, 8, 3, 4, 5, 123456789, time.UTC)
+ r := f.recutter(RecutConfig{Clock: func() time.Time { return at }})
+
+ mustRecut(t, r, 10000)
+
+ var state, updatedAt string
+ if err := f.db.QueryRow(
+ `SELECT state, updated_at FROM handoff WHERE handoff_id = ?`, ids[1]).Scan(&state, &updatedAt); err != nil {
+ t.Fatalf("reading the deferred row: %v", err)
+ }
+ if state != string(record.HandoffStateSkippedBudget) {
+ t.Fatalf("state = %q, want %q", state, string(record.HandoffStateSkippedBudget))
+ }
+ if err := record.ValidateHandoffState(state); err != nil {
+ t.Fatalf("the written state is not a frozen handoff.state literal: %v", err)
+ }
+ // The literal layout, spelled out: it must stay identical to
+ // internal/handoff/state_machine.go's unexported timeLayout, which
+ // internal/store cannot import.
+ const handoffLayout = "2006-01-02T15:04:05.000000000Z"
+ if recutTimestampLayout != handoffLayout {
+ t.Fatalf("recutTimestampLayout = %q, want internal/handoff's %q", recutTimestampLayout, handoffLayout)
+ }
+ parsed, err := time.Parse(handoffLayout, updatedAt)
+ if err != nil {
+ t.Fatalf("updated_at %q does not parse with internal/handoff's layout: %v", updatedAt, err)
+ }
+ if !parsed.Equal(at) {
+ t.Fatalf("updated_at = %v, want %v", parsed, at)
+ }
+}
+
+// TestRecutRejectsBadConfigurationAndBudget: a fraction outside [0,1] or a
+// negative budget is a caller bug and is refused, not clamped. Clamping would
+// hide an arithmetic error inside the component whose entire job is arithmetic.
+func TestRecutRejectsBadConfigurationAndBudget(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ f.enqueue(record.EvidenceClassSastReachable, "high")
+
+ for _, bad := range []float64{-0.01, 1.01, 2} {
+ if _, err := NewRecutter(f.db, RecutConfig{DastReserveFraction: ReserveFraction(bad)}); !errors.Is(err, ErrInvalidReserveFraction) {
+ t.Fatalf("NewRecutter(fraction=%v) error = %v, want ErrInvalidReserveFraction", bad, err)
+ }
+ }
+ // 0 and 1 are both legal: 0 is the control arm and 1 hands the whole
+ // remaining window to late dynamic evidence.
+ for _, ok := range []float64{0, 1} {
+ if _, err := NewRecutter(f.db, RecutConfig{DastReserveFraction: ReserveFraction(ok)}); err != nil {
+ t.Fatalf("NewRecutter(fraction=%v): %v", ok, err)
+ }
+ }
+
+ r := f.recutter(RecutConfig{})
+ if err := r.RecutQueue(recutAuditID, -1); !errors.Is(err, ErrInvalidBudget) {
+ t.Fatalf("RecutQueue(-1) error = %v, want ErrInvalidBudget", err)
+ }
+ if _, err := NewRecutter(nil, RecutConfig{}); err == nil {
+ t.Fatal("NewRecutter(nil db) returned no error")
+ }
+}
+
+// TestResolveAuditRecordID covers the packet's `auditID string` against a
+// schema that has no string audit key.
+func TestResolveAuditRecordID(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusRunning)
+ ctx := context.Background()
+
+ got, err := ResolveAuditRecordID(ctx, f.db, " 1 ")
+ if err != nil {
+ t.Fatalf("ResolveAuditRecordID: %v", err)
+ }
+ if got != recutAuditRecordID {
+ t.Fatalf("ResolveAuditRecordID = %d, want %d", got, recutAuditRecordID)
+ }
+
+ for _, bad := range []string{"", " ", "0", "-3", "anvil-2026-08-08-abc", "1; DROP TABLE handoff"} {
+ if _, err := ResolveAuditRecordID(ctx, f.db, bad); !errors.Is(err, ErrNoSuchAudit) {
+ t.Fatalf("ResolveAuditRecordID(%q) error = %v, want ErrNoSuchAudit", bad, err)
+ }
+ }
+ if _, err := ResolveAuditRecordID(ctx, f.db, "99"); !errors.Is(err, ErrNoSuchAudit) {
+ t.Fatalf("ResolveAuditRecordID(unknown) error = %v, want ErrNoSuchAudit", err)
+ }
+ if err := f.recutter(RecutConfig{}).RecutQueue("99", 1000); !errors.Is(err, ErrNoSuchAudit) {
+ t.Fatalf("RecutQueue(unknown audit) error = %v, want ErrNoSuchAudit", err)
+ }
+}
+
+// TestRecutChargesInFlightLeasesBeforeAdmittingAnything: a queue whose leases
+// already exceed the remaining window admits nothing new, and says by how much
+// it is overdrawn rather than hiding it.
+func TestRecutChargesInFlightLeasesBeforeAdmittingAnything(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusCompletedClean)
+ ids := f.enqueueMany(4, record.EvidenceClassSastReachable, "high")
+ f.lease(ids[0], "w1")
+ f.lease(ids[1], "w2")
+
+ r := f.recutter(RecutConfig{})
+ cut := mustRecut(t, r, 15000)
+
+ if got, want := cut.InFlightTokens, 2*recutTokens; got != want {
+ t.Fatalf("InFlightTokens = %d, want %d", got, want)
+ }
+ if got, want := cut.InFlightOverdraftTokens, 5000; got != want {
+ t.Fatalf("InFlightOverdraftTokens = %d, want %d", got, want)
+ }
+ if len(cut.Admitted) != 0 {
+ t.Fatalf("admitted %d candidates while overdrawn, want none", len(cut.Admitted))
+ }
+ if got, want := len(cut.Deferred), 2; got != want {
+ t.Fatalf("deferred %d, want %d", got, want)
+ }
+ if got := f.state(ids[0]); got != record.HandoffStateLeased {
+ t.Fatalf("in-flight row is %s, want %s", got, record.HandoffStateLeased)
+ }
+}
+
+// TestRecutCostFuncIsConfigurable proves the per-candidate charge is a
+// configuration seam too, so a later step can charge a measured prompt size
+// without editing queue.go.
+func TestRecutCostFuncIsConfigurable(t *testing.T) {
+ f := newRecutFixture(t, record.DastStatusCompletedClean)
+ f.enqueueMany(4, record.EvidenceClassSastReachable, "high")
+
+ r := f.recutter(RecutConfig{
+ CostTokens: func(c Candidate) int { return 1000 * int(c.FindingID) },
+ })
+ cut := mustRecut(t, r, 6000)
+
+ // 1000 + 2000 + 3000 = 6000 fits; the fourth (4000) does not and closes
+ // the cut. A knapsack would have skipped the third to fit the fourth.
+ if got, want := len(cut.Admitted), 3; got != want {
+ t.Fatalf("admitted %d, want %d", got, want)
+ }
+ if got, want := cut.AdmittedTokens(), 6000; got != want {
+ t.Fatalf("AdmittedTokens = %d, want %d", got, want)
+ }
+ if got, want := len(cut.Deferred), 1; got != want {
+ t.Fatalf("deferred %d, want %d", got, want)
+ }
+}
diff --git a/scripts/compute_golden_fingerprints.py b/scripts/compute_golden_fingerprints.py
new file mode 100644
index 0000000..fdd5754
--- /dev/null
+++ b/scripts/compute_golden_fingerprints.py
@@ -0,0 +1,1060 @@
+#!/usr/bin/env python3
+# ruff: noqa: E501
+"""compute_golden_fingerprints.py — the INDEPENDENT oracle for anvil-fp/v1.
+
+===========================================================================
+WHY THIS FILE EXISTS
+===========================================================================
+
+plan/00-SPINE.md S6: "One fingerprint algorithm, defined once, in the record.
+Two branches specified different /v1 algorithms under the same name; two
+producers emitting different hashes means regression matching silently fails
+forever." *Silently* is the operative word. research/07-database-design.md
+and research/18-unified-audit-record.md really did ship two different
+algorithms under the one name `anvil-fp/v1`, and nothing in the tree surfaced
+it. A conformance test whose expected values were produced by the code under
+test would not have surfaced it either: it proves only that the Go equals
+itself.
+
+This script is therefore a SECOND, INDEPENDENT implementation of the same
+algorithm, in a different language, written from `internal/record/
+FINGERPRINT-SPEC.md` and from nothing else. It emits the `.golden` files that
+`internal/record/fingerprint_conformance_test.go` compares the Go
+implementation against. When the two agree, the algorithm has been reproduced
+from its written specification by an implementer who could not see the code —
+which is exactly the property a second producer will need, and the property
+S6 asks for.
+
+===========================================================================
+THE INDEPENDENCE CONTRACT — DO NOT WEAKEN IT
+===========================================================================
+
+This script MUST NOT, ever:
+
+ * read, parse, import, embed, transcribe or execute any Go source file;
+ * shell out to `go` (there is no `subprocess` import here, on purpose);
+ * copy a fixture's committed `expected_digest` into a `.golden` file.
+
+It reads exactly two kinds of input:
+
+ 1. `internal/record/FINGERPRINT-SPEC.md` — the normative algorithm. The
+ 193-entry reserved-word list (spec section 3.5) and the algorithm
+ constants (section 8) are PARSED OUT OF THE DOCUMENT at run time rather
+ than transcribed here, so this oracle is bound to the specification's
+ copy of them and cannot silently drift onto the code's copy.
+ 2. `testdata/fingerprint_corpus/**.json` — the fixed corpus.
+
+`expected_digest` IS read, but only to CROSS-CHECK: if this oracle's
+independently computed digest disagrees with the committed fixture, `--write`
+REFUSES to write and exits non-zero. A conformance harness that re-seals its
+own goldens is the exact failure this packet exists to prevent
+(FINGERPRINT-SPEC.md section 0: "Do not edit a golden digest to make a test
+pass").
+
+===========================================================================
+RESOLUTION OF FINGERPRINT-SPEC.md APPENDIX Z
+===========================================================================
+
+Appendix Z records six places where the prose admits more than one reading.
+This oracle takes one reading of each; every reading is now PINNED BY A
+FIXTURE, so a third implementation that guesses differently fails rather than
+diverging silently. See `testdata/fingerprint_corpus/derived/` and the
+`resolves` key each derived fixture carries.
+
+ Z1 section 3.2 rule 1 — "the Unicode space separators". Read as the
+ Unicode **White_Space** property: TAB, LF, VT, FF, CR, SPACE, U+0085,
+ U+00A0, and categories Zs, Zl (U+2028) and Zp (U+2029). Note this is a
+ reading the specification FORCES elsewhere rather than a free choice:
+ section 9 states that "a snippet containing NUL, BEL, ESC or a raw \\x1f
+ survives normalization and is then rejected by section 1.2", so
+ U+001C-U+001F must NOT be whitespace — which rules out the otherwise
+ obvious Python shortcut `str.isspace()`, whose class does include them.
+ PINNED BY: derived/ordinal-01, candidate `non-ascii-whitespace-separators`
+ (U+00A0 and U+2028 inside a snippet).
+
+ Z2 section 3.5 clauses (c) and (d) — "the next non-space input characters".
+ Read as the same class as Z1, so a newline between an identifier and
+ `::` still preserves the identifier. PINNED BY: derived/ordinal-01,
+ candidate `namespace-qualifier-across-a-newline`.
+
+ Z3 section 6.3 rule P — and/or precedence. Read as
+ `len >= 2 AND (starts { and ends } OR starts < and ends > OR
+ starts :)`, so a one-character segment `:` is NOT a placeholder.
+ PINNED BY: derived/route-01, cases `bare_colon_segment_is_not_a_placeholder`
+ and `two_character_placeholders_are`.
+
+ Z4 section 4 — ordinals were NOT EXERCISED BY THE CORPUS AT ALL: every SAST
+ fixture in the main corpus supplies a pre-computed `ordinal`, so an
+ implementation could get the grouping key wrong and still pass
+ everything. PINNED BY: derived/ordinal-01, a ten-candidate batch,
+ deliberately shuffled out of source order, that forces every component of
+ the grouping key (target_id, rule_id_versioned, CANONICALISED
+ repo_relpath, normalized_match) and every tier of the ordering rule
+ (line, then column, then original batch index) to be exercised
+ independently. It also pins the two documented NON-members of the key:
+ `enclosing_symbol_path` (FINGERPRINT-SPEC.md section 9 / CRITIQUE-01
+ finding 4 — still OPEN, pinned here as current behaviour, not endorsed)
+ and line/column.
+
+ Z5 section 3 generally — block comments, backtick raw strings, , and
+ most of the reserved-word list had no fixture. PARTIALLY RESOLVED:
+ derived/ordinal-01 candidate `block-comment-backtick-raw-string-and-number`
+ exercises section 3.2 rule 4, section 3.3's backtick no-escape rule,
+ section 3.4 hex ``, and one reserved word (`const`). The bulk of
+ the 193-word list REMAINS UNEXERCISED by any fixture; the list itself is
+ still locked by `fingerprint_spec_test.go`, which is a different
+ guarantee (the list is the same) from this one (the list is applied
+ correctly).
+
+ Z6 fixture schema — the corpus JSON says `evidence_signal` where the spec
+ says `evidence_class_detail`, and `repo_rel_path` / `manifest_rel_path`
+ where the spec says `repo_relpath` / `manifest_relpath`. RESOLVED by
+ writing the mapping down: see FIXTURE_KEY_MAP below, which is this
+ oracle's normative reading of the fixture interface.
+
+Resolving any of these is an `anvil-fp/v2` event IF IT CHANGES A DIGEST, and a
+v1 clarification if it does not. None of the readings above changes any
+committed digest: this script reproduces all eight committed
+`expected_digest` values and every committed mutation unchanged, which is
+asserted on every run.
+
+===========================================================================
+USAGE
+===========================================================================
+
+ python scripts/compute_golden_fingerprints.py # check (default)
+ python scripts/compute_golden_fingerprints.py --check
+ python scripts/compute_golden_fingerprints.py --write
+
+`--check` recomputes everything and compares against the committed `.golden`
+files and the committed `expected_digest` values; it writes nothing and exits
+non-zero on any disagreement. `--write` is the authoring path, used when a
+NEW fixture is added; it refuses to write if a digest would contradict a
+committed fixture.
+
+No third-party dependencies. Standard library only, Python 3.11+.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import sys
+import unicodedata
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+SPEC_PATH = REPO_ROOT / "internal" / "record" / "FINGERPRINT-SPEC.md"
+CORPUS_DIR = REPO_ROOT / "testdata" / "fingerprint_corpus"
+DERIVED_DIR = CORPUS_DIR / "derived"
+
+# ---------------------------------------------------------------------------
+# Constants, parsed out of FINGERPRINT-SPEC.md rather than transcribed
+# ---------------------------------------------------------------------------
+
+RESERVED_WORDS_BEGIN = ""
+RESERVED_WORDS_END = ""
+CONSTANTS_BEGIN = ""
+CONSTANTS_END = ""
+
+# Section 3.5: "Matching is exact and case-sensitive. [...] Whitespace-separated,
+# sorted in byte order (uppercase before lowercase), 193 entries".
+RESERVED_WORD_COUNT = 193
+
+
+class SpecError(Exception):
+ """The specification document could not be read as specified."""
+
+
+class FingerprintError(Exception):
+ """An input that anvil-fp/v1 refuses to fingerprint."""
+
+
+def _block(text: str, begin: str, end: str, what: str) -> str:
+ i = text.find(begin)
+ j = text.find(end)
+ if i < 0 or j < 0 or j < i:
+ raise SpecError(f"{SPEC_PATH}: {what} block markers not found")
+ body = text[i + len(begin) : j]
+ # Strip the fenced-code delimiters; the fence is presentation, not content.
+ return "\n".join(line for line in body.splitlines() if not line.strip().startswith("```"))
+
+
+def load_spec() -> tuple[frozenset[str], dict[str, str]]:
+ """Return (reserved words, constants) as the DOCUMENT states them."""
+ text = SPEC_PATH.read_text(encoding="utf-8")
+
+ words = _block(text, RESERVED_WORDS_BEGIN, RESERVED_WORDS_END, "reserved-word").split()
+ if len(words) != RESERVED_WORD_COUNT:
+ raise SpecError(
+ f"{SPEC_PATH}: reserved-word list has {len(words)} entries, "
+ f"the document says {RESERVED_WORD_COUNT}"
+ )
+ if len(set(words)) != len(words):
+ raise SpecError(f"{SPEC_PATH}: reserved-word list contains duplicates")
+ if words != sorted(words):
+ raise SpecError(f"{SPEC_PATH}: reserved-word list is not sorted in byte order")
+
+ consts: dict[str, str] = {}
+ for line in _block(text, CONSTANTS_BEGIN, CONSTANTS_END, "constants").splitlines():
+ if not line.strip():
+ continue
+ name, _, value = line.partition("=")
+ consts[name.strip()] = value.strip()
+
+ return frozenset(words), consts
+
+
+RESERVED_WORDS, SPEC_CONSTANTS = load_spec()
+
+
+def _const(name: str, want: str) -> str:
+ got = SPEC_CONSTANTS.get(name)
+ if got != want:
+ raise SpecError(
+ f"{SPEC_PATH}: constant {name} is {got!r}; this oracle implements {want!r}. "
+ "A changed constant is an anvil-fp/v2 event (FINGERPRINT-SPEC.md section 0)."
+ )
+ return want
+
+
+# Section 8's machine-checked block, asserted against this oracle's own reading.
+ALG_NAME = _const("FingerprintAlgV1", "anvil-fp/v1")
+_const("FingerprintFieldSeparator", "U+001F")
+_const("FingerprintDigestHexLen", "64")
+STR_TOKEN = _const("NormalizedStringToken", "")
+NUM_TOKEN = _const("NormalizedNumberToken", "")
+METAVAR_PREFIX = _const("NormalizedMetavarPrefix", "$")
+VAR_TOKEN = _const("NormalizedRouteSegmentToken", "")
+ROUTE_HEX_MIN_LEN = int(_const("routeHexSegmentMinLen", "16"))
+ROUTE_OPAQUE_MIN_LEN = int(_const("routeOpaqueSegmentMinLen", "20"))
+
+# Section 1.1: U+001F, the ASCII Unit Separator.
+SEP = "\x1f"
+DIGEST_HEX_LEN = 64
+
+# Section 2.1 / 2.3: the literal tier tokens hashed in field position 2. The
+# SAST token is "sast" for BOTH evidence classes.
+TIER_TOKEN_SAST = "sast"
+TIER_TOKEN_DAST = "dast"
+
+# Section 2.2: detector_kind.
+DETECTOR_KIND_SCA = "sca"
+DETECTOR_KIND_HOST = "host"
+
+# Section 2.3 fields 6 and 8: closed value sets, "any other value is rejected".
+INJECTION_POINTS = frozenset({"query", "body", "header", "cookie", "path"})
+EVIDENCE_CLASS_DETAILS = frozenset(
+ {
+ "responseStackTrace",
+ "statusCodeFlip",
+ "dbErrorString",
+ "timingSideChannel",
+ "reflectedPayload",
+ "other",
+ }
+)
+
+# Z6: the fixture JSON's key names, mapped onto the specification's field
+# names. Written down here because the specification never states the fixture
+# schema and Appendix Z records the mapping as "inferred by eye".
+FIXTURE_KEY_MAP = {
+ "sast": {
+ "target_id": "target_id",
+ "rule_id_versioned": "rule_id_versioned",
+ "repo_rel_path": "repo_relpath",
+ "enclosing_symbol_path": "enclosing_symbol_path",
+ "snippet": "(raw input to normalized_match)",
+ "ordinal": "ordinal",
+ },
+ "sca": {
+ "target_id": "target_id",
+ "advisory_id": "advisory_id",
+ "purl": "(raw input to purl_base)",
+ "manifest_rel_path": "manifest_relpath (the locator)",
+ },
+ "host": {
+ "target_id": "target_id",
+ "advisory_id": "advisory_id",
+ "purl": "(raw input to purl_base)",
+ "package_manager": "package_manager (locator, left half)",
+ "host_identifier": "host_identifier (locator, right half)",
+ },
+ "dast": {
+ "target_id": "target_id",
+ "rule_id_versioned": "rule_id_versioned",
+ "http_method": "http_method",
+ "route_template": "(raw input to CanonicalRouteTemplate)",
+ "injection_point": "injection_point",
+ "param_name": "param_name",
+ "evidence_signal": "evidence_class_detail",
+ },
+}
+
+
+# ---------------------------------------------------------------------------
+# Section 1 — primitives
+# ---------------------------------------------------------------------------
+
+
+def digest(fields: list[str]) -> str:
+ """Section 1.2 field guard + section 1.3 digest."""
+ if not fields:
+ raise FingerprintError("an empty field list is rejected (section 1.2)")
+ for i, f in enumerate(fields):
+ for ch in f:
+ if ch <= "\x1f" or ch == "\x7f":
+ what = "the U+001F field separator" if ch == SEP else f"control character U+{ord(ch):04X}"
+ raise FingerprintError(
+ f"field {i} contains {what}; a field boundary would move and two "
+ "distinct findings could collide (section 1.2)"
+ )
+ return hashlib.sha256(SEP.join(fields).encode("utf-8")).hexdigest()
+
+
+def validate_digest(s: str) -> None:
+ """Section 1.3: exactly 64 lowercase hex characters, never truncated."""
+ if len(s) != DIGEST_HEX_LEN:
+ raise FingerprintError(f"digest must be exactly {DIGEST_HEX_LEN} hex characters, got {len(s)}")
+ if any(c not in "0123456789abcdef" for c in s):
+ raise FingerprintError("digest must be lowercase hexadecimal (uppercase is rejected, not folded)")
+
+
+def require(field: str, value: str) -> str:
+ if value == "":
+ raise FingerprintError(f"{field} must not be empty")
+ return value
+
+
+# ---------------------------------------------------------------------------
+# Section 3 — normalized_match
+# ---------------------------------------------------------------------------
+#
+# Z1 lives here. Section 3.2 rule 1 names "\t \n \v \f \r, space, U+0085,
+# U+00A0, and the Unicode space separators". Read as the Unicode White_Space
+# property, i.e. the explicit list plus categories Zs, Zl and Zp. Python's
+# str.isspace() is deliberately NOT used: its class also contains U+001C-U+001F,
+# and section 9 requires a raw \x1f in a snippet to SURVIVE normalization so
+# that section 1.2 can reject it.
+
+
+# TAB, LF, VT, FF, CR, SPACE, U+0085 (NEL) and U+00A0 (NBSP), named
+# explicitly by section 3.2 rule 1. Spelled as escapes so no invisible
+# character can be lost to an editor or a copy-paste.
+SPEC_NAMED_WHITESPACE = frozenset("\t\n\v\f\r \u0085\u00a0")
+
+
+def is_space(ch: str) -> bool:
+ if ch in SPEC_NAMED_WHITESPACE:
+ return True
+ # "the Unicode space separators", read as Zs plus the line and paragraph
+ # separators Zl/Zp -- together exactly the Unicode White_Space property.
+ return unicodedata.category(ch) in ("Zs", "Zl", "Zp")
+
+
+def is_letter(ch: str) -> bool:
+ # Unicode general category L (section 3.2, "identifier-start").
+ return unicodedata.category(ch)[0] == "L"
+
+
+def is_digit_nd(ch: str) -> bool:
+ # Section 3.2: "any digit (IsDigit, i.e. Unicode category Nd - not only ASCII)".
+ return unicodedata.category(ch) == "Nd"
+
+
+def is_ident_start(ch: str) -> bool:
+ return ch in "_$" or is_letter(ch)
+
+
+def is_ident_part(ch: str) -> bool:
+ return ch in "_$" or is_letter(ch) or is_digit_nd(ch)
+
+
+def normalize_match(snippet: str) -> str:
+ """Section 3, one left-to-right pass over Unicode code points."""
+ # 3.1 preprocessing, in this order.
+ src = snippet.replace("\r\n", "\n").replace("\r", "\n")
+ n = len(src)
+
+ out: list[str] = []
+
+ def emit(s: str) -> None:
+ out.extend(s)
+
+ def emit_space() -> None:
+ # 3.1: "only if the output is non-empty and does not already end in a
+ # space". Never two consecutive spaces, never a leading space.
+ if out and out[-1] != " ":
+ out.append(" ")
+
+ def ends_with_selector() -> bool:
+ # 3.5 clause (b), asked of the OUTPUT, ignoring trailing spaces.
+ end = len(out)
+ while end > 0 and out[end - 1] == " ":
+ end -= 1
+ if end == 0:
+ return False
+ if out[end - 1] == ".":
+ return True
+ if end >= 2 and out[end - 2] == "-" and out[end - 1] == ">":
+ return True
+ if end >= 2 and out[end - 2] == ":" and out[end - 1] == ":":
+ return True
+ return False
+
+ def skip_spaces_from(k: int) -> int:
+ # Z2: "non-space" is read as the same class as rule 1's "whitespace".
+ while k < n and is_space(src[k]):
+ k += 1
+ return k
+
+ def next_non_space_is_scope(k: int) -> bool:
+ k = skip_spaces_from(k)
+ return k + 1 < n and src[k] == ":" and src[k + 1] == ":"
+
+ def next_non_space_is_call_open(k: int) -> bool:
+ k = skip_spaces_from(k)
+ return k < n and src[k] == "("
+
+ metavars: dict[str, str] = {}
+ next_metavar = 1
+
+ i = 0
+ while i < n:
+ c = src[i]
+
+ # Rule 1 — whitespace run.
+ if is_space(c):
+ while i < n and is_space(src[i]):
+ i += 1
+ emit_space()
+ continue
+
+ # Rule 2 — "//" line comment. Tested before rule 4, so "//*" is a line
+ # comment.
+ if c == "/" and i + 1 < n and src[i + 1] == "/":
+ while i < n and src[i] != "\n":
+ i += 1
+ emit_space()
+ continue
+
+ # Rule 3 — "#" line comment.
+ if c == "#":
+ while i < n and src[i] != "\n":
+ i += 1
+ emit_space()
+ continue
+
+ # Rule 4 — "/* ... */" block comment; unterminated means to end of input.
+ if c == "/" and i + 1 < n and src[i + 1] == "*":
+ i += 2
+ while i < n:
+ if src[i] == "*" and i + 1 < n and src[i + 1] == "/":
+ i += 2
+ break
+ i += 1
+ emit_space()
+ continue
+
+ # Rule 5 — string literal (section 3.3).
+ if c in "\"'`":
+ quote = c
+ i += 1
+ while i < n:
+ if src[i] == "\\" and quote != "`":
+ # Backslash escapes are honoured inside " and ', NOT inside `.
+ i += 2
+ continue
+ if src[i] == quote:
+ i += 1
+ break
+ i += 1
+ emit(STR_TOKEN)
+ continue
+
+ # Rule 6 — number token (section 3.4). ASCII digit only; a Unicode Nd
+ # digit that is not ASCII falls through to rule 7's identifier-part.
+ if "0" <= c <= "9":
+ while i < n:
+ r = src[i]
+ if is_letter(r) or is_digit_nd(r) or r in "_.":
+ i += 1
+ continue
+ if r in "+-" and i > 0 and src[i - 1] in "eE":
+ i += 1
+ continue
+ break
+ emit(NUM_TOKEN)
+ continue
+
+ # Rule 7 — identifier (section 3.5).
+ if is_ident_start(c):
+ j = i
+ while j < n and is_ident_part(src[j]):
+ j += 1
+ word = src[i:j]
+ i = j
+
+ if word in RESERVED_WORDS: # (a)
+ emit(word)
+ elif ends_with_selector(): # (b)
+ emit(word)
+ elif next_non_space_is_scope(i): # (c)
+ emit(word)
+ elif next_non_space_is_call_open(i): # (d)
+ emit(word)
+ else: # (e)
+ mv = metavars.get(word)
+ if mv is None:
+ mv = METAVAR_PREFIX + str(next_metavar)
+ next_metavar += 1
+ metavars[word] = mv
+ emit(mv)
+ continue
+
+ # Rule 8 — anything else, verbatim.
+ out.append(c)
+ i += 1
+
+ # 3.6 final trim. Only spaces can appear at the edges: emit_space() is the
+ # sole producer of whitespace and it emits U+0020 only.
+ return "".join(out).strip(" ")
+
+
+# ---------------------------------------------------------------------------
+# Section 7 — the remaining canonicalisations
+# ---------------------------------------------------------------------------
+
+
+def canonical_repo_relpath(p: str) -> str:
+ """Section 7.1, in order. No case folding, no '..' resolution."""
+ p = p.replace("\\", "/")
+ while "//" in p:
+ p = p.replace("//", "/")
+ while p.startswith("./"):
+ p = p[2:]
+ if p.startswith("/"):
+ p = p[1:]
+ if p.endswith("/"):
+ p = p[:-1]
+ return p
+
+
+def purl_base(purl: str) -> str:
+ """Section 7.2. The enforcement point for 'the version is never hashed'."""
+ p = purl.strip()
+ if p == "":
+ raise FingerprintError("purl_base: must not be empty")
+ if len(p) < 4 or p[:4].lower() != "pkg:":
+ raise FingerprintError(f"purl_base: must begin with 'pkg:', got {purl!r}")
+ rest = p[4:]
+ for delim in ("#", "?", "@"): # subpath, then qualifiers, then version
+ idx = rest.find(delim)
+ if idx >= 0:
+ rest = rest[:idx]
+ if rest.endswith("/"):
+ rest = rest[:-1]
+ if rest == "":
+ raise FingerprintError(f"purl_base: no type or name after 'pkg:': {purl!r}")
+ slash = rest.find("/")
+ if slash < 0:
+ raise FingerprintError(f"purl_base: a type but no name: {purl!r}")
+ # Only the type (up to the first '/') is lower-cased; namespace and name
+ # are left alone because their case-sensitivity is type-dependent.
+ rest = rest[:slash].lower() + rest[slash:]
+ return "pkg:" + rest
+
+
+def host_locator(package_manager: str, host_identifier: str) -> str:
+ """Section 7.3."""
+ if ":" in package_manager:
+ raise FingerprintError("host locator: package_manager must not contain ':' (its own delimiter)")
+ mgr = package_manager.strip()
+ ident = host_identifier.strip()
+ if mgr == "":
+ raise FingerprintError("host locator: package_manager must not be empty")
+ if ident == "":
+ raise FingerprintError("host locator: host_identifier must not be empty")
+ return mgr.lower() + ":" + ident
+
+
+def canonical_http_method(method: str) -> str:
+ """Section 7.4: trim, upper-case, reject empty or multi-token."""
+ if method.strip() == "":
+ raise FingerprintError("http_method: must not be empty")
+ m = method.strip().upper()
+ if any(is_space(ch) for ch in m):
+ raise FingerprintError("http_method: must be a single token")
+ return m
+
+
+# ---------------------------------------------------------------------------
+# Section 6 — route_template, a DERIVED value
+# ---------------------------------------------------------------------------
+
+
+def _is_all_ascii_digits(s: str) -> bool:
+ return s != "" and all("0" <= c <= "9" for c in s)
+
+
+def _is_ascii_hex(c: str) -> bool:
+ return ("0" <= c <= "9") or ("a" <= c <= "f") or ("A" <= c <= "F")
+
+
+def _is_route_placeholder(s: str) -> bool:
+ # Z3: read as len >= 2 AND (A or B or C). A one-character ":" is therefore
+ # NOT a placeholder.
+ if len(s) < 2:
+ return False
+ if s[0] == "{" and s[-1] == "}":
+ return True
+ if s[0] == "<" and s[-1] == ">":
+ return True
+ return s[0] == ":"
+
+
+def _is_uuid_segment(s: str) -> bool:
+ if len(s) != 36:
+ return False
+ for idx, ch in enumerate(s):
+ if idx in (8, 13, 18, 23):
+ if ch != "-":
+ return False
+ elif not _is_ascii_hex(ch):
+ return False
+ return True
+
+
+def _is_long_hex_segment(s: str) -> bool:
+ return len(s) >= ROUTE_HEX_MIN_LEN and all(_is_ascii_hex(c) for c in s)
+
+
+def _is_long_opaque_segment(s: str) -> bool:
+ if len(s) < ROUTE_OPAQUE_MIN_LEN:
+ return False
+ has_digit = has_letter = False
+ for c in s:
+ if "0" <= c <= "9":
+ has_digit = True
+ elif ("a" <= c <= "z") or ("A" <= c <= "Z"):
+ has_letter = True
+ else:
+ return False
+ return has_digit and has_letter
+
+
+def is_volatile_route_segment(s: str) -> bool:
+ """Section 6.3, rules P, N, U, H, O in that order. Empty is never volatile."""
+ if s == "":
+ return False
+ return (
+ _is_route_placeholder(s)
+ or _is_all_ascii_digits(s)
+ or _is_uuid_segment(s)
+ or _is_long_hex_segment(s)
+ or _is_long_opaque_segment(s)
+ )
+
+
+def canonical_route_template(route: str) -> str:
+ """Section 6.2, steps 1-8 in order."""
+ cut = min((i for i in (route.find("?"), route.find("#")) if i >= 0), default=-1)
+ if cut >= 0:
+ route = route[:cut]
+ route = route.replace("\\", "/")
+ while "//" in route:
+ route = route.replace("//", "/")
+ if route == "":
+ return ""
+ if not route.startswith("/"):
+ route = "/" + route
+ if len(route) > 1 and route.endswith("/"):
+ route = route[:-1]
+ if route == "/":
+ return "/"
+ segs = [VAR_TOKEN if is_volatile_route_segment(s) else s for s in route[1:].split("/")]
+ return "/" + "/".join(segs)
+
+
+# ---------------------------------------------------------------------------
+# Section 2 — the four tiers
+# ---------------------------------------------------------------------------
+
+
+def sast_fields(inp: dict) -> list[str]:
+ """Section 2.1: seven fields."""
+ target_id = require("target_id", inp["target_id"])
+ rule_id = require("rule_id_versioned", inp["rule_id_versioned"])
+ raw_path = require("repo_relpath", inp["repo_rel_path"])
+ relpath = canonical_repo_relpath(raw_path)
+ if relpath == "":
+ raise FingerprintError("repo_relpath canonicalises to the empty string")
+ snippet = require("normalized_match (snippet)", inp["snippet"])
+ normalized = normalize_match(snippet)
+ if normalized == "":
+ raise FingerprintError("snippet normalises to the empty string; it carries no identity")
+ ordinal = int(inp["ordinal"])
+ if ordinal < 0:
+ raise FingerprintError("ordinal must not be negative")
+ return [
+ target_id,
+ TIER_TOKEN_SAST,
+ rule_id,
+ relpath,
+ inp.get("enclosing_symbol_path", ""), # may be empty
+ normalized,
+ str(ordinal), # base 10, no padding, no sign
+ ]
+
+
+def sca_fields(inp: dict) -> list[str]:
+ """Section 2.2 with detector_kind = 'sca'."""
+ locator = canonical_repo_relpath(require("manifest_relpath", inp["manifest_rel_path"]))
+ if locator == "":
+ raise FingerprintError("manifest_relpath canonicalises to the empty string")
+ return [
+ require("target_id", inp["target_id"]),
+ DETECTOR_KIND_SCA,
+ require("advisory_id", inp["advisory_id"]), # verbatim, NOT case-folded
+ purl_base(require("purl", inp["purl"])),
+ locator,
+ ]
+
+
+def host_fields(inp: dict) -> list[str]:
+ """Section 2.2 with detector_kind = 'host'."""
+ return [
+ require("target_id", inp["target_id"]),
+ DETECTOR_KIND_HOST,
+ require("advisory_id", inp["advisory_id"]),
+ purl_base(require("purl", inp["purl"])),
+ host_locator(inp["package_manager"], inp["host_identifier"]),
+ ]
+
+
+def dast_fields(inp: dict) -> list[str]:
+ """Section 2.3: eight fields."""
+ target_id = require("target_id", inp["target_id"])
+ rule_id = require("rule_id_versioned", inp["rule_id_versioned"])
+ method = canonical_http_method(inp["http_method"])
+ route = canonical_route_template(require("route_template", inp["route_template"]))
+ if route == "":
+ raise FingerprintError("route_template canonicalises to the empty string")
+ injection = inp["injection_point"]
+ if injection not in INJECTION_POINTS:
+ raise FingerprintError(f"injection_point {injection!r} is not one of {sorted(INJECTION_POINTS)}")
+ # Z6: the fixture spells this `evidence_signal`; the specification calls the
+ # hashed field `evidence_class_detail`. Same field.
+ detail = inp["evidence_signal"]
+ if detail not in EVIDENCE_CLASS_DETAILS:
+ raise FingerprintError(
+ f"evidence_class_detail {detail!r} is not one of {sorted(EVIDENCE_CLASS_DETAILS)}"
+ )
+ return [
+ target_id,
+ TIER_TOKEN_DAST,
+ rule_id,
+ method,
+ route,
+ injection,
+ inp.get("param_name", ""), # may be empty
+ detail,
+ ]
+
+
+TIER_BUILDERS = {
+ "sast": sast_fields,
+ "sca": sca_fields,
+ "host": host_fields,
+ "dast": dast_fields,
+}
+
+
+def fields_for(tier: str, inp: dict) -> list[str]:
+ builder = TIER_BUILDERS.get(tier)
+ if builder is None:
+ raise FingerprintError(f"unknown tier {tier!r}")
+ unknown = set(inp) - set(FIXTURE_KEY_MAP[tier])
+ if unknown:
+ # Z6 again: an unmapped fixture key would otherwise default a hashed
+ # field to "" and lock in a wrong digest, silently.
+ raise FingerprintError(f"tier {tier}: fixture carries unmapped key(s) {sorted(unknown)}")
+ return builder(inp)
+
+
+# ---------------------------------------------------------------------------
+# Section 4 — ordinal and its grouping key
+# ---------------------------------------------------------------------------
+
+
+def ordinal_group_key(inp: dict) -> str:
+ """Section 4: target_id, rule_id_versioned, CanonicalRepoRelPath(repo_relpath),
+ normalized_match — joined with U+001F for COMPARISON only, never hashed.
+
+ Note what is NOT in this key and is documented as such: enclosing_symbol_path
+ (section 9 / CRITIQUE-01 finding 4, still open) and line/column.
+ """
+ return SEP.join(
+ [
+ inp["target_id"],
+ inp["rule_id_versioned"],
+ canonical_repo_relpath(inp["repo_rel_path"]),
+ normalize_match(inp["snippet"]),
+ ]
+ )
+
+
+def assign_ordinals(candidates: list[dict]) -> list[int]:
+ """Section 4. Ordering within a group is ascending by line, then column,
+ then the candidate's original index in the batch (a stable tiebreak).
+ """
+ groups: dict[str, list[int]] = {}
+ for idx, cand in enumerate(candidates):
+ groups.setdefault(ordinal_group_key(cand["input"]), []).append(idx)
+
+ ordinals = [-1] * len(candidates)
+ for members in groups.values():
+ members.sort(key=lambda i: (candidates[i]["line"], candidates[i]["column"], i))
+ for ordinal, idx in enumerate(members):
+ ordinals[idx] = ordinal
+ return ordinals
+
+
+# ---------------------------------------------------------------------------
+# Golden files
+# ---------------------------------------------------------------------------
+#
+# Format: TSV, three columns, "#" comment lines ignored.
+#
+# kind label value
+#
+# kind is "digest" (value is a 64-lowercase-hex digest) or "ordinal" (value is
+# a base-10 non-negative integer). Labels are "base", "mutation:",
+# "candidate:" and "case:". Both this script and
+# internal/record/fingerprint_conformance_test.go parse it; it is deliberately
+# trivial so neither parser can be the interesting part.
+
+GOLDEN_HEADER = """\
+# anvil-fp/v1 conformance golden — {fixture_id}
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+"""
+
+
+def render_golden(fixture_id: str, rows: list[tuple[str, str, str]]) -> str:
+ body = "".join(f"{kind}\t{label}\t{value}\n" for kind, label, value in rows)
+ return GOLDEN_HEADER.format(fixture_id=fixture_id) + body
+
+
+def parse_golden(text: str) -> list[tuple[str, str, str]]:
+ rows = []
+ for line in text.splitlines():
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ parts = line.split("\t")
+ if len(parts) != 3:
+ raise SpecError(f"malformed golden line: {line!r}")
+ rows.append((parts[0], parts[1], parts[2]))
+ return rows
+
+
+# ---------------------------------------------------------------------------
+# Corpus walking
+# ---------------------------------------------------------------------------
+
+
+class Mismatch(Exception):
+ """This oracle disagrees with something already committed."""
+
+
+def rows_for_main_fixture(fx: dict, path: Path) -> list[tuple[str, str, str]]:
+ """The eight tier fixtures: base + every mutation."""
+ tier = fx["tier"]
+ rows: list[tuple[str, str, str]] = []
+
+ fields = fields_for(tier, fx["input"])
+ if fields != fx["hashed_fields"]:
+ raise Mismatch(
+ f"{path.name}: hashed_fields disagree.\n"
+ f" this oracle: {fields!r}\n"
+ f" committed: {fx['hashed_fields']!r}"
+ )
+ base = digest(fields)
+ validate_digest(base)
+ committed = fx.get("expected_digest", "")
+ if committed and base != committed:
+ raise Mismatch(
+ f"{path.name}: base digest disagrees with the committed expected_digest.\n"
+ f" this oracle: {base}\n"
+ f" committed: {committed}\n"
+ "STOP. Do not re-seal. Either the specification and the implementation have "
+ "diverged, or this is an anvil-fp/v2 event."
+ )
+ rows.append(("digest", "base", base))
+
+ for m in fx.get("mutations", []):
+ d = digest(fields_for(tier, m["input"]))
+ validate_digest(d)
+ if d != base:
+ raise Mismatch(
+ f"{path.name}: mutation {m['name']!r} changed the digest.\n"
+ f" base: {base}\n"
+ f" mutation: {d}\n"
+ "Every mutation differs ONLY in fields the specification forbids hashing."
+ )
+ rows.append(("digest", f"mutation:{m['name']}", d))
+
+ return rows
+
+
+def rows_for_derived_fixture(fx: dict, path: Path) -> list[tuple[str, str, str]]:
+ """The derived corpus: values the fixture does NOT supply and an
+ implementation must compute (Appendix Z4 and friends)."""
+ kind = fx["kind"]
+ rows: list[tuple[str, str, str]] = []
+
+ if kind == "sast_ordinal_batch":
+ candidates = fx["candidates"]
+ ordinals = assign_ordinals(candidates)
+ for cand, ordinal in zip(candidates, ordinals, strict=True):
+ if "ordinal" in cand["input"]:
+ raise Mismatch(
+ f"{path.name}: candidate {cand['name']!r} supplies a pre-computed ordinal; "
+ "the whole point of this fixture is that the ordinal must be DERIVED."
+ )
+ if ordinal != cand["expected_ordinal"]:
+ raise Mismatch(
+ f"{path.name}: candidate {cand['name']!r} — derived ordinal {ordinal}, "
+ f"fixture says {cand['expected_ordinal']}"
+ )
+ inp = dict(cand["input"], ordinal=ordinal)
+ fields = fields_for("sast", inp)
+ if fields != cand["hashed_fields"]:
+ raise Mismatch(
+ f"{path.name}: candidate {cand['name']!r} hashed_fields disagree.\n"
+ f" this oracle: {fields!r}\n"
+ f" committed: {cand['hashed_fields']!r}"
+ )
+ d = digest(fields)
+ validate_digest(d)
+ committed = cand.get("expected_digest", "")
+ if committed and d != committed:
+ raise Mismatch(
+ f"{path.name}: candidate {cand['name']!r} digest disagrees with the fixture.\n"
+ f" this oracle: {d}\n committed: {committed}"
+ )
+ rows.append(("ordinal", f"candidate:{cand['name']}", str(ordinal)))
+ rows.append(("digest", f"candidate:{cand['name']}", d))
+ return rows
+
+ if kind == "dast_cases":
+ for case in fx["candidates"]:
+ # The fixture states the DERIVED route it expects, in the clear.
+ # Checking it separately from the digest turns "the digest moved"
+ # into "segment X templated when it should not have".
+ want_route = case["canonical_route"]
+ got_route = canonical_route_template(case["input"]["route_template"])
+ if got_route != want_route:
+ raise Mismatch(
+ f"{path.name}: case {case['name']!r} canonical_route disagrees.\n"
+ f" this oracle: {got_route!r}\n committed: {want_route!r}"
+ )
+ fields = fields_for("dast", case["input"])
+ if fields != case["hashed_fields"]:
+ raise Mismatch(
+ f"{path.name}: case {case['name']!r} hashed_fields disagree.\n"
+ f" this oracle: {fields!r}\n"
+ f" committed: {case['hashed_fields']!r}"
+ )
+ d = digest(fields)
+ validate_digest(d)
+ committed = case.get("expected_digest", "")
+ if committed and d != committed:
+ raise Mismatch(
+ f"{path.name}: case {case['name']!r} digest disagrees with the fixture.\n"
+ f" this oracle: {d}\n committed: {committed}"
+ )
+ rows.append(("digest", f"case:{case['name']}", d))
+ return rows
+
+ raise Mismatch(f"{path.name}: unknown derived fixture kind {kind!r}")
+
+
+def fixture_files() -> list[tuple[Path, bool]]:
+ """(path, is_derived) for every fixture, in a stable order."""
+ main = sorted(CORPUS_DIR.glob("*.json"))
+ derived = sorted(DERIVED_DIR.glob("*.json")) if DERIVED_DIR.is_dir() else []
+ if not main:
+ raise Mismatch(f"no fixtures in {CORPUS_DIR}; the fixed corpus is mandatory (00-SPINE.md S6)")
+ return [(p, False) for p in main] + [(p, True) for p in derived]
+
+
+def run(write: bool) -> int:
+ failures = 0
+ total_rows = 0
+
+ for path, is_derived in fixture_files():
+ fx = json.loads(path.read_text(encoding="utf-8"))
+ try:
+ rows = rows_for_derived_fixture(fx, path) if is_derived else rows_for_main_fixture(fx, path)
+ except (Mismatch, FingerprintError) as exc:
+ print(f"FAIL {path.name}\n {exc}", file=sys.stderr)
+ failures += 1
+ continue
+
+ total_rows += len(rows)
+ golden_path = path.with_suffix(".golden")
+ rendered = render_golden(fx["id"], rows)
+
+ if write:
+ golden_path.write_text(rendered, encoding="utf-8", newline="\n")
+ print(f"WROTE {golden_path.relative_to(REPO_ROOT)} ({len(rows)} rows)")
+ continue
+
+ if not golden_path.exists():
+ print(f"FAIL {golden_path.relative_to(REPO_ROOT)} is missing; run with --write", file=sys.stderr)
+ failures += 1
+ continue
+ committed = parse_golden(golden_path.read_text(encoding="utf-8"))
+ if committed != rows:
+ print(f"FAIL {golden_path.relative_to(REPO_ROOT)} disagrees with this oracle", file=sys.stderr)
+ for a, b in zip(committed, rows, strict=False):
+ if a != b:
+ print(f" committed {a}\n oracle {b}", file=sys.stderr)
+ if len(committed) != len(rows):
+ print(f" row count: committed {len(committed)}, oracle {len(rows)}", file=sys.stderr)
+ failures += 1
+ continue
+ print(f"OK {golden_path.relative_to(REPO_ROOT)} ({len(rows)} rows)")
+
+ if failures:
+ print(f"\n{failures} fixture(s) FAILED. Do NOT re-seal goldens to make this pass.", file=sys.stderr)
+ return 1
+ print(f"\n{ALG_NAME}: all fixtures reproduced from FINGERPRINT-SPEC.md ({total_rows} rows).")
+ return 0
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ g = ap.add_mutually_exclusive_group()
+ g.add_argument("--check", action="store_true", help="recompute and compare (default); writes nothing")
+ g.add_argument("--write", action="store_true", help="author mode: (re)write the .golden files")
+ args = ap.parse_args()
+ return run(write=args.write)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.golden b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.golden
new file mode 100644
index 0000000..2a31798
--- /dev/null
+++ b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.golden
@@ -0,0 +1,23 @@
+# anvil-fp/v1 conformance golden — dast-01-sqli-error-based-body
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:http_method_lowercased 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:http_method_padded 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_carrying_the_payload_in_a_query_string 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_carrying_a_fragment 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_with_a_trailing_slash 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_with_duplicate_slashes_and_no_leading_slash 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_carrying_the_concrete_numeric_id_this_scan_requested 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
+digest mutation:route_template_in_express_placeholder_syntax 199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90
diff --git a/testdata/fingerprint_corpus/dast-02-xss-reflected-query.golden b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.golden
new file mode 100644
index 0000000..caf06e2
--- /dev/null
+++ b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.golden
@@ -0,0 +1,18 @@
+# anvil-fp/v1 conformance golden — dast-02-xss-reflected-query
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489
+digest mutation:http_method_lowercased bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489
+digest mutation:route_template_carrying_the_injected_query_parameter bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489
+digest mutation:route_template_with_a_trailing_slash bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489
diff --git a/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.golden b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.golden
new file mode 100644
index 0000000..a3b57f1
--- /dev/null
+++ b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.golden
@@ -0,0 +1,20 @@
+# anvil-fp/v1 conformance golden — dast-03-path-traversal-no-param-name
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
+digest mutation:http_method_lowercased 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
+digest mutation:route_template_with_duplicate_leading_slashes 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
+digest mutation:route_template_carrying_the_traversal_payload_as_a_query_string 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
+digest mutation:route_template_in_flask_placeholder_syntax 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
+digest mutation:route_template_already_carrying_the_canonical_placeholder_token 5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550
diff --git a/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.golden b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.golden
new file mode 100644
index 0000000..3eb3235
--- /dev/null
+++ b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.golden
@@ -0,0 +1,27 @@
+# anvil-fp/v1 conformance golden — dast-04-idor-concrete-numeric-and-uuid-segments
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:different_numeric_id 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:different_uuid 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:both_ids_different_at_once 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:uuid_uppercased 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:uuid_without_dashes 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:sha256_hex_object_id 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:opaque_base64ish_order_token 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:openapi_curly_placeholders 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:express_colon_placeholders 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:flask_angle_placeholders_with_converters 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:already_carrying_the_canonical_placeholder_token 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
+digest mutation:concrete_ids_with_query_string_trailing_slash_and_duplicate_slashes 84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b
diff --git a/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.golden b/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.golden
new file mode 100644
index 0000000..18c53d8
--- /dev/null
+++ b/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.golden
@@ -0,0 +1,34 @@
+# anvil-fp/v1 conformance golden — ordinal-01-sast-batch-derived-ordinals
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+ordinal candidate:at-line-220-first-in-batch 1
+digest candidate:at-line-220-first-in-batch 93097339e68234a6f61dd65176518beb51aa03cf065438c52b103a95d542c162
+ordinal candidate:at-line-42-earliest-in-the-file 0
+digest candidate:at-line-42-earliest-in-the-file 7855ab98b4973b45e9c399ec241dcc237a5330fd6fa241cccbed04dc0e252d9d
+ordinal candidate:at-line-220-duplicate-position-second-in-batch 2
+digest candidate:at-line-220-duplicate-position-second-in-batch bf8c83929893191aae5da95e5de3a4c5bf19afbb5d3232a5dff9f16f6b0f37da
+ordinal candidate:different-path-same-normalized-match 0
+digest candidate:different-path-same-normalized-match 5f83f90740d4e3273d3f36362bd87aa684eea190dcebf9ca1858f1c5c1c87bd5
+ordinal candidate:different-rule-same-normalized-match 0
+digest candidate:different-rule-same-normalized-match bc56c39b9947d8614af0718ca28ec8b69614731467a6fca2be9f29b02b8cb802
+ordinal candidate:different-target-same-everything 0
+digest candidate:different-target-same-everything 724d19d66c7dda27eaee02e5fb2ce694c6f5dd4298ea18ae0950fe54527c8aa9
+ordinal candidate:at-line-900-via-a-windows-path 3
+digest candidate:at-line-900-via-a-windows-path fab727d4e4cd03d3d3457fa32078be96b02aebaeec6d987c6dfab3e0ace0b46b
+ordinal candidate:block-comment-backtick-raw-string-and-number 0
+digest candidate:block-comment-backtick-raw-string-and-number 82bf3a0b6d997db0b5815d60105b1be067cc3046c430048d6cc1f2a67bdc53c2
+ordinal candidate:namespace-qualifier-across-a-newline 0
+digest candidate:namespace-qualifier-across-a-newline f227218b4d61168c4673edd7664e9493e9dd7fed2767abb261367bb571d9b00f
+ordinal candidate:non-ascii-whitespace-separators 0
+digest candidate:non-ascii-whitespace-separators cdcba75f0e1dc47cd61dc3efcd9af4224b4c0716858f4d40cda9cb6fd063ff45
diff --git a/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.json b/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.json
new file mode 100644
index 0000000..f16db2f
--- /dev/null
+++ b/testdata/fingerprint_corpus/derived/ordinal-01-sast-batch-derived-ordinals.json
@@ -0,0 +1,248 @@
+{
+ "id": "ordinal-01-sast-batch-derived-ordinals",
+ "kind": "sast_ordinal_batch",
+ "resolves": ["Z1", "Z2", "Z4", "Z5"],
+ "description": "The fixture FINGERPRINT-SPEC.md Appendix Z4 says the corpus was missing. Every SAST fixture in the main corpus supplies a pre-computed `ordinal` in its input, so section 4's grouping key and ordering rule were NOT EXERCISED AT ALL: an independent implementation could get the grouping key wrong and still reproduce all eight committed digests. Here no candidate supplies an ordinal. The batch is presented deliberately OUT of source order, and each of the four components of the grouping key (target_id, rule_id_versioned, CANONICALISED repo_relpath, normalized_match) is varied one at a time so that getting any one of them wrong changes an ordinal and therefore a digest.",
+ "notes": [
+ "Section 4 grouping key: target_id U+001F rule_id_versioned U+001F CanonicalRepoRelPath(repo_relpath) U+001F normalized_match. Joined for COMPARISON only; the key itself is never hashed.",
+ "Section 4 ordering within a group: ascending by source line, then source column, then the candidate's original index in the batch. The sort must be stable. Line and column are used ONLY for this ordering and never reach the digest -- which is why they live beside `input` here rather than inside it.",
+ "Candidates `at-line-220-first-in-batch` and `at-line-220-duplicate-position-second-in-batch` are byte-identical in every hashed field EXCEPT the derived ordinal. That is the whole reason section 4 exists: without an ordinal they would collide on one fingerprint and the second finding would be LOST on upsert against UNIQUE (target_id, fingerprint). Losing a finding is worse than churning one.",
+ "`at-line-42-earliest-in-the-file` differs from the others by a pure local rename AND by a different enclosing_symbol_path, yet lands in the SAME group. Two separate facts are pinned there: locals abstract to $N so a rename does not fork identity, and enclosing_symbol_path is NOT part of the grouping key. The second is a KNOWN, OPEN defect -- FINGERPRINT-SPEC.md section 9, CRITIQUE-01 finding 4: deleting an unrelated function above a match can churn that match's ordinal and therefore its digest. It is pinned here as CURRENT BEHAVIOUR, not endorsed. If it is ever ruled on in the critic's favour that is an anvil-fp/v2 event and this fixture's ordinals move with it.",
+ "`at-line-900-via-a-windows-path` reports the same file as `.\\internal\\api\\repo.go`. It joins the same group only because the grouping key applies CanonicalRepoRelPath (section 7.1) before comparing. An implementation that grouped on the raw path would give it ordinal 0 instead of 3 and emit a different digest -- silently.",
+ "Appendix Z5: `block-comment-backtick-raw-string-and-number` is the first fixture anywhere to exercise section 3.2 rule 4 (block comments), section 3.3's rule that backslash escapes are NOT honoured inside a backtick raw string, section 3.4's token, and a reserved word (`const`, clause (a)). The bulk of the 193-word reserved list is still unexercised by any fixture; fingerprint_spec_test.go locks the list's CONTENT, which is a different guarantee from its APPLICATION.",
+ "Appendix Z2: `namespace-qualifier-across-a-newline` puts a newline between `Ns` and `::`. Clause (c) asks whether 'the next non-space input characters are ::'; reading 'non-space' as section 3.2 rule 1's full whitespace class preserves `Ns`, while a narrower reading (literal U+0020 only) would abstract it to $1. This fixture makes the two readings produce different digests instead of silently agreeing.",
+ "Appendix Z1: `non-ascii-whitespace-separators` separates its tokens with U+00A0 (NBSP) and U+2028 (LINE SEPARATOR, category Zl). Both must be treated as whitespace by rule 1. Note the counter-constraint that fixes the class from the other side: section 9 requires a raw U+001F in a snippet to SURVIVE normalization so section 1.2 can reject it, so the class is Unicode White_Space and NOT any broader class that swallows the C0 separators."
+ ],
+ "candidates": [
+ {
+ "name": "at-line-220-first-in-batch",
+ "description": "Group G1, second in source order. First of two candidates at the identical (line, column).",
+ "line": 220,
+ "column": 4,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 1,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Delete",
+ "$1.Exec( + $2)",
+ "1"
+ ]
+ },
+ {
+ "name": "at-line-42-earliest-in-the-file",
+ "description": "Group G1, first in source order despite appearing second in the batch. Locals are renamed and the enclosing symbol differs; neither affects the grouping key.",
+ "line": 42,
+ "column": 8,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Purge",
+ "snippet": "conn.Exec(\"DELETE FROM u WHERE key = \" + k)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Purge",
+ "$1.Exec( + $2)",
+ "0"
+ ]
+ },
+ {
+ "name": "at-line-220-duplicate-position-second-in-batch",
+ "description": "Group G1. Identical to `at-line-220-first-in-batch` in every hashed field and in (line, column); only section 4's third ordering component -- the original index in the batch -- separates them. A non-stable sort would swap these two identities on some runs and not others.",
+ "line": 220,
+ "column": 4,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 2,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Delete",
+ "$1.Exec( + $2)",
+ "2"
+ ]
+ },
+ {
+ "name": "different-path-same-normalized-match",
+ "description": "Group G2. Same target, same rule, same normalized match, DIFFERENT file: repo_relpath is in the grouping key, so this restarts at ordinal 0.",
+ "line": 7,
+ "column": 2,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/other.go",
+ "enclosing_symbol_path": "internal/api/other.go::Other.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/other.go",
+ "internal/api/other.go::Other.Delete",
+ "$1.Exec( + $2)",
+ "0"
+ ]
+ },
+ {
+ "name": "different-rule-same-normalized-match",
+ "description": "Group G3. Same target, same file, same line and column, same normalized match, DIFFERENT rule: rule_id_versioned is in the grouping key, so this restarts at ordinal 0 rather than becoming G1's fourth member.",
+ "line": 220,
+ "column": 4,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.dangerous-db-exec@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.dangerous-db-exec@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Delete",
+ "$1.Exec( + $2)",
+ "0"
+ ]
+ },
+ {
+ "name": "different-target-same-everything",
+ "description": "Group G4. Identical to `at-line-220-first-in-batch` except for target_id. Section 4 adds target_id to the specification's (rule, path, normalized) key precisely so that passing two targets' candidates in one batch cannot cross-index them; without it this candidate would take an ordinal from G1's run and both targets' findings would move.",
+ "line": 220,
+ "column": 4,
+ "input": {
+ "target_id": "t-9002",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9002",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Delete",
+ "$1.Exec( + $2)",
+ "0"
+ ]
+ },
+ {
+ "name": "at-line-900-via-a-windows-path",
+ "description": "Group G1, last in source order. The same file reported by a Windows producer as `.\\internal\\api\\repo.go`. It joins G1 only because the grouping key canonicalises the path first (section 7.1); an implementation that grouped on the raw string would emit ordinal 0 and a different digest.",
+ "line": 900,
+ "column": 1,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": ".\\internal\\api\\repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.Delete",
+ "snippet": "db.Exec(\"DELETE FROM t WHERE id = \" + id)"
+ },
+ "expected_ordinal": 3,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.Delete",
+ "$1.Exec( + $2)",
+ "3"
+ ]
+ },
+ {
+ "name": "block-comment-backtick-raw-string-and-number",
+ "description": "Group G5, alone. Appendix Z5: a leading /* block comment */ (rule 4), the reserved word `const` preserved verbatim by clause (a), a hex literal collapsing to (section 3.4), and a backtick raw string whose embedded backslash-n is ORDINARY CONTENT because section 3.3 does not honour escapes inside a backtick literal. An implementation that honoured the escape would consume past the closing backtick and normalise the tail differently.",
+ "line": 300,
+ "column": 5,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "internal/api/repo.go::Repo.List",
+ "snippet": "/* legacy */ const limit = 0xFF\nrows, err := db.Query(`SELECT * FROM t WHERE a = \\n`)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "internal/api/repo.go::Repo.List",
+ "const $1 = $2, $3 := $4.Query()",
+ "0"
+ ]
+ },
+ {
+ "name": "namespace-qualifier-across-a-newline",
+ "description": "Group G6, alone. Appendix Z2: a newline sits between `Ns` and `::`. Clause (c) preserves `Ns` only if 'the next non-space input characters' is read with rule 1's whitespace class. Note also that the newline still emits a space, so the output carries `Ns ::Helper(...)` -- clause (b) then fires for `Helper` because it ignores TRAILING spaces when inspecting the output. An empty enclosing_symbol_path is legal (section 2.1) and hashed as a zero-length field.",
+ "line": 310,
+ "column": 3,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "",
+ "snippet": "Ns\n::Helper(v)"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "",
+ "Ns ::Helper($1)",
+ "0"
+ ]
+ },
+ {
+ "name": "non-ascii-whitespace-separators",
+ "description": "Group G7, alone. Appendix Z1: the tokens are separated by U+00A0 (NBSP, named explicitly by rule 1) and U+2028 (LINE SEPARATOR, category Zl, reached only by reading 'the Unicode space separators' as the White_Space property). Both collapse to one U+0020, so the normalized match is plain ASCII and passes the section 1.2 field guard. An implementation that emitted U+2028 verbatim would produce a valid-looking but incompatible digest.",
+ "line": 320,
+ "column": 3,
+ "input": {
+ "target_id": "t-9001",
+ "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "repo_rel_path": "internal/api/repo.go",
+ "enclosing_symbol_path": "",
+ "snippet": "a\u00a0=\u2028b"
+ },
+ "expected_ordinal": 0,
+ "hashed_fields": [
+ "t-9001",
+ "sast",
+ "opengrep.go.lang.security.audit.sqli-exec-concat@2026.07.1",
+ "internal/api/repo.go",
+ "",
+ "$1 = $2",
+ "0"
+ ]
+ }
+ ]
+}
diff --git a/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.golden b/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.golden
new file mode 100644
index 0000000..fd6c637
--- /dev/null
+++ b/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.golden
@@ -0,0 +1,28 @@
+# anvil-fp/v1 conformance golden — route-01-dast-placeholder-and-threshold-boundaries
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest case:bare_colon_segment_is_not_a_placeholder 3b8e9c477e8c61b850f09cd4e9edf1dd0d3302c31ba8e7f6b11c9990cdf1d078
+digest case:bare_open_brace_is_not_a_placeholder e8bed342ee40ef83c1d80c4c9e5e6f4cb6fa40d09cd69713627b529ceec42126
+digest case:colon_placeholder_at_the_minimum_length 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:curly_placeholder_at_the_minimum_length 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:angle_placeholder_at_the_minimum_length 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:already_canonical_token_is_idempotent 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:hex_segment_at_the_threshold_is_volatile 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:hex_segment_one_below_the_threshold_is_preserved fcb68f3f235bb3ea1198041a5891110d3e23e8942a6eb2fa02c272fd9964b37b
+digest case:opaque_segment_at_the_threshold_is_volatile 1f831d799b66ef0a23f3d279200e7f745acdaa570adcf391e3aef2419571f08c
+digest case:opaque_segment_one_below_the_threshold_is_preserved e7e58828a73e63659f799745c0c81f3d4de8569aa9b8f1d090f157ef13b567f1
+digest case:twenty_letters_without_a_digit_is_preserved 6a51d974e1476428a520597a00ecf32dc6d046b03c2786101edde01e9830725e
+digest case:hyphenated_dated_slug_is_preserved 124c00ac4296f5330ee8f1193eca9a5748f0241a6947b4e99130025aa5728c66
+digest case:root_path_survives_step_seven b2a389327c48e3abdcd1ffddab37ae72cf47f30b7ac83d119cce8b94fcfa1e4a
+digest case:query_string_duplicate_slashes_and_missing_leading_slash 9181fdd3bc5c899893ebca4a91786f70310903f5e96a780899eaccfb5fed474d
diff --git a/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.json b/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.json
new file mode 100644
index 0000000..3f2a0cc
--- /dev/null
+++ b/testdata/fingerprint_corpus/derived/route-01-dast-placeholder-and-threshold-boundaries.json
@@ -0,0 +1,351 @@
+{
+ "id": "route-01-dast-placeholder-and-threshold-boundaries",
+ "kind": "dast_cases",
+ "resolves": ["Z3"],
+ "description": "FINGERPRINT-SPEC.md section 6.3's volatile-segment predicates, exercised at their boundaries. Appendix Z3 records that rule P's prose -- 'length >= 2 and (starts { and ends }) or (starts < and ends >) or starts :' -- is genuinely ambiguous about and/or precedence, and that the two parses differ for a one-character `:` segment. dast-04 in the main corpus proves templating HAPPENS; nothing proved where it STOPS. Over-templating is the unrecoverable direction (section 6.4): it merges two distinct routes onto one identity and loses a finding on upsert against UNIQUE (target_id, fingerprint), silently. These cases pin both sides of every threshold.",
+ "notes": [
+ "Every case varies ONLY route_template. target_id, rule_id_versioned, http_method, injection_point, param_name and evidence_class_detail are held constant, so any digest difference is attributable to CanonicalRouteTemplate alone.",
+ "Cases that canonicalise to the same route are EXPECTED to share a digest -- that is section 6's whole purpose (one defect, one identity, whatever syntax the producer used). The distinctness assertion belongs on the canonicalised routes, not on the case list.",
+ "Z3 resolution: read as `len >= 2 AND (A or B or C)`. `bare_colon_segment_is_not_a_placeholder` and `bare_open_brace_is_not_a_placeholder` are the two cases that separate the readings; under the alternative parse (`len>=2 AND A` or `B` or `C`) the bare `:` would template and the digest would move.",
+ "The threshold cases come in pairs, one on each side: 16 vs 15 hex characters for rule H, 20 vs 19 alphanumerics for rule O. A one-character change to either constant in section 8 changes a digest here, which is what makes routeHexSegmentMinLen and routeOpaqueSegmentMinLen anvil-fp/v2 material rather than tuning knobs.",
+ "`already_canonical_token_is_idempotent` is the property that lets a record be read out of the store and re-fingerprinted without losing its identity: is itself a rule-P segment, so CanonicalRouteTemplate is idempotent."
+ ],
+ "candidates": [
+ {
+ "name": "bare_colon_segment_is_not_a_placeholder",
+ "description": "Z3. A one-character `:` fails rule P's length gate under the chosen parse, so the segment survives verbatim.",
+ "canonical_route": "/a/:/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/:/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/:/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "bare_open_brace_is_not_a_placeholder",
+ "description": "Z3, the other half: a one-character `{` also fails the length gate and is not a well-formed placeholder in any syntax.",
+ "canonical_route": "/a/{/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/{/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/{/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "colon_placeholder_at_the_minimum_length",
+ "description": "Rule P at exactly length 2: Express/Rails/Sinatra syntax.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/:x/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "curly_placeholder_at_the_minimum_length",
+ "description": "Rule P at exactly length 2: an empty OpenAPI/ASP.NET placeholder. Rule P ignores the placeholder's NAME, so an empty name is still a placeholder.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/{}/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "angle_placeholder_at_the_minimum_length",
+ "description": "Rule P at exactly length 2: Flask/Werkzeug syntax with an empty name.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/<>/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "already_canonical_token_is_idempotent",
+ "description": " is itself a rule-P segment, so re-fingerprinting a record read back out of the store keeps its identity.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a//b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "hex_segment_at_the_threshold_is_volatile",
+ "description": "Rule H at exactly routeHexSegmentMinLen = 16.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/e3b0c44298fc1c14/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "hex_segment_one_below_the_threshold_is_preserved",
+ "description": "Rule H at 15 characters: under-templating is the recoverable direction, so the segment survives.",
+ "canonical_route": "/a/cafebabecafebab/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/cafebabecafebab/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/cafebabecafebab/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "opaque_segment_at_the_threshold_is_volatile",
+ "description": "Rule O at exactly routeOpaqueSegmentMinLen = 20, alphanumeric with both a digit and a letter.",
+ "canonical_route": "/a//b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/abcdefghij123456789k/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a//b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "opaque_segment_one_below_the_threshold_is_preserved",
+ "description": "Rule O at 19 characters.",
+ "canonical_route": "/a/abcdefghij123456789/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/abcdefghij123456789/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/abcdefghij123456789/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "twenty_letters_without_a_digit_is_preserved",
+ "description": "Rule O's digit requirement, which is what carries most of the safety at length 20: 'internationalization' is exactly 20 characters and is route structure, not an identifier.",
+ "canonical_route": "/a/internationalization/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/internationalization/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/internationalization/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "hyphenated_dated_slug_is_preserved",
+ "description": "Rule O's alphanumeric-only requirement. 'release-notes-2026-08' is 21 characters and carries digits; merging every dated release note onto one digest is exactly the over-templating failure section 6.4 warns about. The accepted cost is that a base64url token containing '-' or '_' is left un-templated.",
+ "canonical_route": "/a/release-notes-2026-08/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/a/release-notes-2026-08/b",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/release-notes-2026-08/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "root_path_survives_step_seven",
+ "description": "Section 6.2 step 7: a route of exactly '/' is the root path and is returned unchanged rather than being trimmed to the empty string, which the DAST tier would reject.",
+ "canonical_route": "/",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "/",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ },
+ {
+ "name": "query_string_duplicate_slashes_and_missing_leading_slash",
+ "description": "Section 6.2 steps 1, 3, 5 and 6 in one input: 'a//b/?x=1' loses its query string, collapses its duplicate slash, gains a leading slash and loses its trailing one.",
+ "canonical_route": "/a/b",
+ "input": {
+ "target_id": "t-9003",
+ "rule_id_versioned": "nuclei:route-canonicalisation-probe@2026.07.1",
+ "http_method": "GET",
+ "route_template": "a//b/?x=1",
+ "injection_point": "path",
+ "param_name": "",
+ "evidence_signal": "statusCodeFlip"
+ },
+ "hashed_fields": [
+ "t-9003",
+ "dast",
+ "nuclei:route-canonicalisation-probe@2026.07.1",
+ "GET",
+ "/a/b",
+ "path",
+ "",
+ "statusCodeFlip"
+ ]
+ }
+ ]
+}
diff --git a/testdata/fingerprint_corpus/host-01-openssl-debian.golden b/testdata/fingerprint_corpus/host-01-openssl-debian.golden
new file mode 100644
index 0000000..49787bf
--- /dev/null
+++ b/testdata/fingerprint_corpus/host-01-openssl-debian.golden
@@ -0,0 +1,20 @@
+# anvil-fp/v1 conformance golden — host-01-openssl-debian
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
+digest mutation:version_bumped c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
+digest mutation:purl_carrying_an_arch_qualifier c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
+digest mutation:package_manager_uppercased c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
+digest mutation:package_manager_and_identifier_padded c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
+digest mutation:purl_already_version_free c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953
diff --git a/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.golden b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.golden
new file mode 100644
index 0000000..f421e77
--- /dev/null
+++ b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.golden
@@ -0,0 +1,20 @@
+# anvil-fp/v1 conformance golden — sast-01-go-sql-string-concat
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
+digest mutation:shifted_down_three_lines_and_reindented 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
+digest mutation:crlf_endings_and_trailing_comment_naming_the_old_line 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
+digest mutation:leading_comment_line_added 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
+digest mutation:locals_renamed_and_literal_text_changed 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
+digest mutation:repo_rel_path_in_windows_form 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6
diff --git a/testdata/fingerprint_corpus/sast-02-python-shell-command.golden b/testdata/fingerprint_corpus/sast-02-python-shell-command.golden
new file mode 100644
index 0000000..5fe30cd
--- /dev/null
+++ b/testdata/fingerprint_corpus/sast-02-python-shell-command.golden
@@ -0,0 +1,19 @@
+# anvil-fp/v1 conformance golden — sast-02-python-shell-command
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242
+digest mutation:shifted_down_and_reindented_into_a_block d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242
+digest mutation:hash_line_comment_appended d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242
+digest mutation:local_renamed_and_literal_changed d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242
+digest mutation:repo_rel_path_with_dot_slash_prefix d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242
diff --git a/testdata/fingerprint_corpus/sca-01-log4shell-maven.golden b/testdata/fingerprint_corpus/sca-01-log4shell-maven.golden
new file mode 100644
index 0000000..bd69ae9
--- /dev/null
+++ b/testdata/fingerprint_corpus/sca-01-log4shell-maven.golden
@@ -0,0 +1,22 @@
+# anvil-fp/v1 conformance golden — sca-01-log4shell-maven
+#
+# PRODUCED BY: scripts/compute_golden_fingerprints.py, an implementation of
+# internal/record/FINGERPRINT-SPEC.md written WITHOUT reading
+# internal/record/fingerprint.go. These values were NOT produced by the code
+# they gate, and they are NOT a copy of the fixture's `expected_digest`.
+#
+# DO NOT regenerate this file to make a test pass. A changed digest means every
+# stored finding under it loses its identity, silently: `first_seen_at` resets,
+# every fingerprint-keyed suppression stops applying, and every handoff row is
+# orphaned. That is an anvil-fp/v2 event with a dual-write migration
+# (FINGERPRINT-SPEC.md section 0), never a v1 edit and never a re-seal.
+#
+# Format: TSV — kind label value.
+digest base c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:version_bumped_inside_the_vulnerable_range c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:version_bumped_out_of_the_vulnerable_range c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:purl_carrying_qualifiers c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:purl_carrying_qualifiers_and_a_subpath c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:purl_scheme_and_type_uppercased c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:purl_already_version_free c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8
+digest mutation:manifest_path_in_windows_form c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8