🧪 test(collect): make the load test assert something it can fail (PERF-FIX-04)

.github/workflows/e2e-nightly.yaml

@@ -187,28 +187,29 @@ jobs:
           retention-days: 14
           if-no-files-found: warn
 
-  load-test-10k:
-    name: load-test-10k
-    # Skipped by default: ubuntu-latest-8-cores runners unavailable (jobs never leave queue).
-    # Re-enable via workflow_dispatch with run_scale_jobs=true once org runners are provisioned.
-    if: github.event_name == 'workflow_dispatch' && inputs.run_scale_jobs == true
-    runs-on: ubuntu-latest-8-cores
-    timeout-minutes: 45
+  # PERF-FIX-04: this used to be `load-test-10k` on ubuntu-latest-8-cores, which read
+  # as cluster-scale evidence. It never was: it is a single-threaded, in-process
+  # extractor micro-benchmark with no API server, cluster, sinks or export path.
+  # Renamed, moved to a plain runner, and made to actually run every nightly.
+  # In-cluster scale evidence is the `scale-envtest-10k` job below.
+  extract-budget:
+    name: extract-budget
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
+      # No envtest cache on purpose: this job never starts an API server.
       - uses: ./.github/actions/go-cache
-        with:
-          envtest: true
 
       - uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
         with:
           version: 3.51.1
 
-      - name: Opt-in 10k load tier
-        run: task load-test:10k
+      - name: Extractor hot-path budget (micro-benchmark, not a scale tier)
+        run: task extract-budget
         env:
           CI: "true"
 

Taskfile.yml

@@ -224,15 +224,20 @@ tasks:
     cmds:
       - task: test:bench
 
-  load-test:
-    desc: Opt-in bounded load tests (requires KOLECT_LOAD_TEST=1, max 10000)
-    cmds:
-      - task: test:load-test
-
-  load-test:10k:
-    desc: Opt-in 10k load tier (KOLECT_LOAD_TEST_MAX=10000)
-    cmds:
-      - task: test:load-test:10k
+  extract-budget:
+    desc: >-
+      Extractor hot-path budget. ENFORCED: B/op and allocs/op at the recorded
+      baseline +25% — hardware-independent, so a >25% allocation regression
+      fails anywhere. NOT enforced: a >25% wall-clock gate; the ns/op ceiling is
+      a coarse catastrophic-regression net only (a CPU-only 3.5x slowdown at
+      unchanged allocations passes it) — pin KOLECT_EXTRACT_MAX_NS_PER_OP to
+      your own measured ns/op x 1.25 for a real latency floor. EXERCISES:
+      Extract over 128 varied objects, single-threaded, in-process. NOT
+      exercised: API server, cluster, informers, sinks, controller, concurrency,
+      export. Not cluster-scale evidence — see
+      docs/operator-manual/load-test-runbook.md for the scale tiers.
+    cmds:
+      - task: test:extract-budget
 
   perf-report:
     desc: Local perf snapshot via hack/perf-report.sh (CI writes artifacts/perf-snapshot.md)

docs/REQUIREMENTS.md

@@ -122,7 +122,7 @@ IDs are stable handles for discussion (`FR-<area>-<n>`).
 
 | ID | Target |
 | --- | --- |
-| NFR-PERF-1 | **Design target:** **100,000** collected rows/cluster (sharded exports; **unexecuted AR-02**). Default CI proves ≤500 synthetic; opt-in ≤2k synthetic ≠ in-cluster soak; Nightly **10k** CI **disabled / unverified** (`ubuntu-latest-8-cores` opt-in). Store **≤512 MiB @ 10k**, operator RSS **2–4 GiB @ 100k** — see [load-test runbook](operator-manual/load-test-runbook.md) / [performance](operator-manual/performance.md) |
+| NFR-PERF-1 | **Design target:** **100,000** collected rows/cluster (sharded exports; **unexecuted AR-02**). Default CI proves ≤500 synthetic; the in-process `task extract-budget` micro-benchmark is **not** a scale tier; Nightly **10k** envtest CI (`scale-envtest-10k`) **disabled / unverified** (`ubuntu-latest-8-cores` opt-in). Store **≤512 MiB @ 10k**, operator RSS **2–4 GiB @ 100k** — see [load-test runbook](operator-manual/load-test-runbook.md) / [performance](operator-manual/performance.md) |
 | NFR-PERF-2 | Giant cluster: 1000+ nodes — namespace-scoped informers + paginated list mandatory |
 | NFR-PERF-3 | Fleet: **100–500+** clusters via **shared sink** ([ADR-0501](adr/0501-multi-cluster-fleet.md)); no hub merge tier |
 | NFR-PERF-4 | One shared informer per GVK; memory scales with objects × GVKs, not with target count |

docs/adr/0603-performance-scalability.md

@@ -23,8 +23,8 @@ operator**.
 | Tier | Scope | Collected rows | Clusters | Test tier / evidence |
 | --- | --- | --- | --- | --- |
 | **CI / dev** | Synthetic envtest | ≤500 | 1 | `task test` — Active |
-| **Opt-in load** | Synthetic | ≤2,000 | 1 | `KOLECT_LOAD_TEST=1 task load-test` — synthetic ≠ in-cluster soak |
-| **Nightly load** | Synthetic | **10,000** | 1 | `task load-test:10k` on `ubuntu-latest-8-cores` — **disabled / opt-in** until runners exist ([load-test runbook](../operator-manual/load-test-runbook.md)) |
+| **Extractor budget** | In-process, no cluster | n/a (micro-benchmark) | 1 | `task extract-budget` — hot-path ns/op + B/op budget; **not** a scale tier, ≠ in-cluster soak |
+| **Nightly load** | Synthetic envtest | **10,000** | 1 | `scale-envtest-10k` on `ubuntu-latest-8-cores` — **disabled / opt-in** until runners exist ([load-test runbook](../operator-manual/load-test-runbook.md)) |
 | **Baseline production** | Single cluster | **10,000+** | 1 | Metrics + pprof — **unverified** until a named SHA / hardware evidence is published |
 | **Design target** | Single cluster | **100,000** | 1–2 cloud | Manual cloud soak — **unexecuted (AR-02)** |
 | **Fleet** | Shared Postgres/Git sink | 10k–100k × N operators | **many** | One ServiceMonitor per cluster release |

docs/adr/0706-testing-merge-gate-architecture.md

@@ -25,7 +25,7 @@ gates (e.g. Q16 RBAC audit, Q15 supply-chain attestations) without debate.
 | **L2 — Golden / contract** | OpenAPI fragments, sample YAML decode, extractor goldens | checked-in `test/` + `config/samples/` | `task test`; samples per [ADR-0301](0301-event-driven-informers.md) |
 | **L3 — Integration** | Real Postgres, Kafka, Git, S3, GCS, Redis, NATS via **testcontainers** | `-tags=integration` | `task test-integration` |
 | **L4 — E2E** | Kind cluster: Helm install, smoke, export asserts | `hack/kind/e2e/`, `hack/e2e/` | `task test:e2e`; nightly workflow |
-| **L5 — Load / perf (opt-in)** | Bounded synthetic scale (≤2000 objects) | `-tags=load`, `KOLECT_LOAD_TEST=1` | `task load-test`; `task bench`; `task perf-report` |
+| **L5 — Perf budget / benchmarks** | In-process extractor hot path (no API server, cluster, sinks or concurrency) | Default gate + nightly | `task extract-budget`; `task bench`; `task perf-report` |
 
 **Direction:** most tests live at L0–L2; every new sink backend must reach **L3** before merge
 ([NFR-EXT-3](../REQUIREMENTS.md)); L4 catches wiring regressions webhooks/RBAC/informers miss
@@ -80,7 +80,14 @@ matrix remains nightly + manual dispatch.
 ### Scale and load bounds
 
 - Default **`task test` / `task coverage`**: synthetic object caps **≤500** ([ADR-0603](0603-performance-scalability.md)).
-- **`task load-test`**: requires `KOLECT_LOAD_TEST=1`; hard cap **2000** objects — never in default CI.
+- **`task extract-budget`**: extractor hot-path budget. It **enforces >25%** on `B/op` and
+  `allocs/op` only — those are hardware-independent for a fixed toolchain, so an allocation
+  regression fails on any runner. It does **not** enforce a >25% wall-clock gate: the `ns/op`
+  ceiling is a coarse catastrophic-regression net, because a ceiling loose enough to be safe on a
+  shared CI runner is far too loose to catch 25%. A CPU-only regression at unchanged allocations
+  can pass it; pin `KOLECT_EXTRACT_MAX_NS_PER_OP` per machine for a real latency floor. In-process
+  only: no API server, cluster, sinks, controller or concurrency, so it is **not** cluster-scale
+  evidence (that is the opt-in `scale-envtest-10k` job). Cheap enough to run in the default gate.
 - **`task bench`**: micro-benchmarks on hot paths (CEL/JSONPath extract); safe in dev/CI excerpt via
   `task perf-report`.
 

docs/adr/0707-lab-harness.md

@@ -30,7 +30,7 @@ Merge-gate architecture ([ADR-0706](0706-testing-merge-gate-architecture.md)) al
 | Tier | Role |
 | --- | --- |
 | **L4 Kind e2e** | Wiring smoke — `hack/kind/e2e/`, `hack/e2e/`; Tier 0 blocks merge |
-| **L5 load** | Opt-in `task load-test` (≤2000), `task perf-report`; not merge-blocking |
+| **L5 perf** | `task extract-budget` (in-process extractor budget), `task perf-report`; not merge-blocking |
 | **100k cloud** | Separate runbook / `hack/loadtest/` — unexecuted claim gate |
 
 The lab harness must **not** become a Kind wrapper, a second CI pyramid, or a public claim that
@@ -72,7 +72,7 @@ The lab harness must **not** become a Kind wrapper, a second CI pyramid, or a pu
 | **C2 DR-\* only** | Matches proven protocols | Ubuntu A–G catalogue orphans |
 | **C3 Registry: schedule → primary IDs + aliases** | DR-\* primary for multi-node; LAB-\* aliases for Ubuntu mapping | Small registry file to maintain |
 
-### (d) Relationship to Kind L4 / loadtest L5
+### (d) Relationship to Kind L4 / perf-budget L5
 
 | Option | Pros | Cons |
 | --- | --- | --- |
@@ -138,7 +138,7 @@ flowchart TB
   subgraph ci ["CI / merge (ADR-0706)"]
     l0l3["L0–L3 unit / envtest / integration"]
     l4["L4 Kind e2e<br/>hack/kind + hack/e2e"]
-    l5["L5 load / perf-report<br/>opt-in"]
+    l5["L5 extract-budget / perf-report<br/>in-process, not scale"]
   end
 
   subgraph lab ["L4.5 Lab harness (this ADR)"]
@@ -209,7 +209,7 @@ flowchart TB
 ## Cross-links
 
 - [ADR-0706: Testing and merge-gate architecture](0706-testing-merge-gate-architecture.md) — L4 Kind /
-  L5 load ownership; this ADR sits beside them as maintainer **L4.5**.
+  L5 perf-budget ownership; this ADR sits beside them as maintainer **L4.5**.
 - [Local lab runbook](../operator-manual/local-lab-runbook.md) — LAB-DOC-01 adaptive schedules,
   `tier=auto`, isolation, and real `hack/lab/` flags.
 - [Lab evidence bundle](../operator-manual/lab-evidence-bundle.md) — LAB-DOC-02 publishable schema and

docs/development/guidelines.md

@@ -65,7 +65,9 @@ Operator test expectations. Pyramid tiers, coverage floors, and CI gates:
 - **Mocks** — mockery on small interfaces only.
 - **Metrics** — assert Prometheus counters/histograms in controller tests where behavior changes.
 - **Scale tests bounded** — default `task test` caps synthetic objects (500); load tests require
-  `KOLECT_LOAD_TEST=1` and `-tags=load` (max 2000). Never run 10k-object suites in default CI.
+  the opt-in envtest scale test (`TestEngine_ScaleEnvtestOptIn`, `KOLECT_SCALE_TEST_MAX`). Never
+  run 10k-object suites in default CI. `task extract-budget` is an in-process extractor
+  micro-benchmark budget — it is **not** cluster-scale evidence and must never be named as such.
 
 ## 5. Performance and scalability
 
@@ -79,7 +81,11 @@ Operator test expectations. Pyramid tiers, coverage floors, and CI gates:
 - **Rate limits and circuit breakers** — per-sink `gobreaker`; transient sink/API errors requeue with
   jitter; terminal config errors stop requeue ([ADR-0602](../adr/0602-error-taxonomy.md)).
 - **Profiling** — pprof on `:6060` behind feature gate (default off); document in [operator-manual/performance.md](../operator-manual/performance.md).
-- **Benchmarks** — `task bench` (`-short`, `-benchmem`); `BenchmarkExtract` for CEL/JSONPath hot path.
+- **Benchmarks** — `task bench` (`-short`, `-benchmem`); `BenchmarkExtract` for the CEL/JSONPath hot
+  path. `TestExtractHotPathBudget` drives the same workload and **fails on a >25% `B/op` or
+  `allocs/op` regression**; its `ns/op` ceiling is only a catastrophic-regression net, not a >25%
+  wall-clock gate. Re-record the baseline consts in
+  `internal/collect/extractor_budget_test.go` with evidence.
 
 ## 6. Definition of done (per change)
 

docs/development/setup.md

@@ -252,17 +252,35 @@ go test -short -bench=. -benchmem ./internal/collect/...
 Uses `-short` so long sub-benchmarks are skipped on laptops. Suitable for CI and quick regression
 checks. See [operator-manual/performance.md](../operator-manual/performance.md) and [ADR-0603](../adr/0603-performance-scalability.md).
 
-### Load tests (opt-in, bounded)
-
-**Not** part of default `task test`. Requires explicit opt-in and caps at **2000** synthetic objects:
+### Extractor hot-path budget (`task extract-budget`)
 
 ```sh
-KOLECT_LOAD_TEST=1 task load-test
+task extract-budget
 # equivalent:
-KOLECT_LOAD_TEST=1 go test -tags=load -count=1 -timeout=15m ./test/load/...
+go test -run TestExtractHotPathBudget -count=1 -v ./internal/collect/...
 ```
 
-Never run 10k-object load tests locally unless you have dedicated hardware and understand API-server
+`TestExtractHotPathBudget` drives the same workload as `BenchmarkExtract` — 128 varied
+Deployment-shaped objects through `collect.Extractor.Extract` — and checks ns/op, B/op and
+allocs/op against a recorded baseline. It also runs as part of `task test`.
+
+**Enforced at baseline +25%:** `B/op` and `allocs/op`. Hardware-independent for a fixed Go
+toolchain (verified across arm64/amd64, `-cover`, `-race`, and the go1.26.5 -> 1.26.6 bump), so an
+allocation regression fails on any runner.
+
+**Not enforced:** a >25% wall-clock gate. The `ns/op` ceiling is a coarse catastrophic-regression
+net only — any ceiling loose enough to be safe on a shared CI runner is far too loose to catch
+25%, and a CPU-only regression at unchanged allocations will pass it. For a real latency floor,
+measure your own hardware and pin `KOLECT_EXTRACT_MAX_NS_PER_OP` to that x 1.25 (also `_BYTES_` /
+`_ALLOCS_`).
+
+**What it exercises:** the extractor hot path, single-threaded, in-process.
+**What it does not:** API server, cluster, informers, sinks, controller, concurrency, export. It is
+a micro-benchmark budget, **not** cluster-scale evidence — the scale tiers live in the
+[load test runbook](../operator-manual/load-test-runbook.md).
+
+Cluster scale is the opt-in envtest test `TestEngine_ScaleEnvtestOptIn` (`KOLECT_SCALE_TEST_MAX`).
+Never run 10k-object suites locally unless you have dedicated hardware and understand API-server
 load. Default envtest suites cap synthetic objects at **500**.
 
 ### Performance report (`task perf-report`)

docs/development/testing.md

@@ -17,11 +17,13 @@ Kollect is **TDD-first**. Quality gates follow a six-tier test pyramid (L0–L5)
 | **L2 — Golden / contract** | OpenAPI fragments, sample YAML, extractor goldens | Yes | `task test` |
 | **L3 — Integration** | Real Postgres, Kafka, Git, S3, GCS, Redis, NATS (testcontainers) | Yes | `task test-integration` |
 | **L4 — E2E** | Kind cluster: Helm install, smoke, export asserts | **PR smoke (required)** + nightly / extended | `task test:e2e` |
-| **L5 — Load / perf** | Bounded synthetic scale (≤2000 objects), micro-benchmarks | Opt-in | `task load-test` · `task perf-report` |
+| **L5 — Perf budget / benchmarks** | In-process extractor hot path (no cluster, sinks or concurrency) | Default + nightly | `task extract-budget` · `task bench` · `task perf-report` |
 
 **Direction:** Most tests live at L0–L2. Every new sink backend must reach **L3** before merge
-([NFR-EXT-3](../REQUIREMENTS.md)). L4 catches wiring regressions that unit tests miss. L5 stays
-opt-in so default CI stays fast.
+([NFR-EXT-3](../REQUIREMENTS.md)). L4 catches wiring regressions that unit tests miss. L5 is an
+in-process budget, not a scale tier: it fails a >25% regression in the extractor's `B/op` or
+`allocs/op` (its `ns/op` ceiling is only a catastrophic-regression net), and says nothing about
+cluster scale. Cluster scale is the opt-in `scale-envtest-10k` job.
 
 ## Coverage target
 
@@ -124,7 +126,7 @@ For **local** runs the variable is optional: export `GIT_EXPORT_TEST_REPO` to a
 
 Kind L4 proves single-node wiring. Separately, published **v0.16.0** was exercised on a Talos lab
 with **1 control plane + 2 workers** (`quick+sinks`, **ready with conditions**). Maintainer
-multi-node / existing-cluster evidence sits as **L4.5** beside Kind L4 and load L5 —
+multi-node / existing-cluster evidence sits as **L4.5** beside Kind L4 and perf-budget L5 —
 [ADR-0707: Lab harness architecture](../adr/0707-lab-harness.md). Publishable shape, redaction,
 and an example matrix live in the
 [lab evidence bundle contract](../operator-manual/lab-evidence-bundle.md).
@@ -160,7 +162,7 @@ or the 100k cloud gate. Raw protocols stay local-only — see the
 | `task test-integration` | L3 sink/transport integration (Docker) |
 | `task test:e2e` | L4 kind smoke (setup → smoke → teardown) |
 | `task bench` | Micro-benchmarks on hot paths |
-| `KOLECT_LOAD_TEST=1 task load-test` | L5 bounded load (≤2000 objects, opt-in) |
+| `task extract-budget` | L5 extractor hot-path budget — >25% gate on B/op + allocs/op only (in-process; not cluster scale) |
 | `task perf-report` | Benchmark + unit pass summary (local only, gitignored output) |
 
 Full local setup: [development/setup.md](../development/setup.md).