fix(test): de-flake deploy-canary scaler test via env hermeticity - #4540
fix(test): de-flake deploy-canary scaler test via env hermeticity#4540rysweet wants to merge 3 commits into
Conversation
The `unit-test` deploy-gate canary reddened with exit 101 and stalled self-deploy. Root cause: `scaler_current_max_can_override_config` in tests/adaptive_scaling.rs built its OodaConfig with a `..OodaConfig::default()` spread. Under an inherited `SIMARD_SCALING=auto`, `OodaConfig::default()` (src/ooda_loop/types.rs:368) injects an `AdaptiveScaler` with ceiling 24, and `decide()` (src/ooda_loop/decide.rs:42) honors that scaler over `max_concurrent_actions`. With 5 over-quota priorities the test got 5 actions, the `<= 2` assertion panicked, `cargo test` exited 101, and the gate reddened. Fix (test-only, additive, no production/assertion change): - Pin an explicit `AdaptiveScaler::new(2, 2, 2)` (adjust() deterministically 2). - Fully enumerate all nine OodaConfig fields — drop the env-derived `..OodaConfig::default()` spread — and set `scaler: Some(pinned)` with `max_concurrent_actions: 8` so the pinned ceiling of 2 must win regardless of ambient env. This matches the established `scaler: None` hardening on the sibling `decide_respects_max_concurrent_actions` (issue #2732). - Add docs/testing/deploy-canary-scaler-env-hermeticity.md and link it from docs/index.md, specifying the scaler-sensitive-test hermeticity contract. Validated: original body reproduces exit-101 under SIMARD_SCALING=auto; fixed body passes deterministically across auto/fixed/unset (x3 each); full adaptive_scaling suite (22 tests) green under SIMARD_SCALING=auto; `cargo fmt --check` and `cargo clippy --all-targets -- -D warnings` clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a local `use std::sync::Arc;` + `Arc::new(...)` in scaler_current_max_can_override_config, matching the file's existing convention in the concurrency test below it, instead of the fully qualified `std::sync::Arc::new(...)`. Behavior-preserving readability tidy; no logic, assertion, or hermeticity-contract change. Validated: target test passes x3 under hostile SIMARD_SCALING=auto + SIMARD_MAX_CONCURRENT_ACTIONS=10; full adaptive_scaling suite (22) green; cargo fmt --check and clippy --test adaptive_scaling -D warnings clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rysweet
left a comment
There was a problem hiding this comment.
Step 17b: Comprehensive Code Review — PR #4540
Verdict: APPROVE (with non-blocking metadata findings). The code change is correct, minimal, well-tested, and thoroughly documented. Independently verified below. No blocking issues in the code, test, or docs.
Actual change set (3 files)
M docs/index.md (+1 link line)
A docs/testing/deploy-canary-scaler-env-hermeticity.md (+254, new reference)
M tests/adaptive_scaling.rs (test-only, +18/-7)
Test-only + docs. No production code, no signature changes, no algorithm change. Additive and non-breaking.
Independently verified ✅
| Check | Result |
|---|---|
OodaConfig field enumeration |
9/9 exact — struct in src/ooda_loop/types.rs (max_concurrent_actions, improvement_threshold, gym_suite_id, daily_budget_usd, weekly_budget_usd, distill_min_episodes, distill_interval_cycles, lesson_recurrence_threshold, scaler) fully matches the enumerated test config. Dropping ..OodaConfig::default() makes any drift a compile error, not a silent env-coupling — correct trade-off. |
| Scaler-override logic | src/ooda_loop/decide.rs:42 if let Some(ref scaler) = config.scaler { scaler.adjust() } confirms the scaler wins over max_concurrent_actions. Pinned AdaptiveScaler::new(2,2,2) → adjust()==2 < config's 8, so assert actions.len() <= 2 is deterministic. |
| Consistency with siblings | Matches the hardened pattern in decide_uses_scaler_adjusted_limit_when_scaler_is_present / decide_ignores_scaler_when_none / decide_respects_max_concurrent_actions (all pin scaler: explicitly, per #2732). |
| Hermeticity intent | The verdict is now a pure function of the test's own inputs; SIMARD_SCALING=auto no longer flips the assertion. Root cause (ambient SIMARD_SCALING=auto leaking through the ..default() spread) is correctly identified and eliminated. |
Review checklist
- Code quality and standards — idiomatic Rust;
Arcimport aligned with sibling; clear comments explaining the hermeticity contract - Test coverage adequate — the fix is a regression test; fails pre-fix (panic → exit 101) and passes post-fix under both ambient and clean env
- No TODOs, stubs, or swallowed exceptions
- No unimplemented functions
- Logic correctness — verified against
decide()andAdaptiveScaler - Edge-case handling — hostile ambient env (
SIMARD_SCALING=auto) is the explicit edge case covered
Non-blocking findings (PR hygiene — recommend fixing before merge, do NOT touch code)
-
Title/body do not match the actual diff. Title says "Update .github with 12 changed files (#980)" and the body lists 12 files (release.yml, verify.yml, scripts/*, meeting_backend tests, etc.) with a
330 insertions / 435 deletionsdiff stat. The actual PR is 3 files,+277/-6, and touches no.github/workflow orscripts/. The body's file list and diff stat are stale/templated. Recommend regenerating the title + body to describe the real change (deploy-canarySIMARD_SCALINGhermeticity test fix). This matters for auditability. -
Closes #980looks incorrect for this change. #980 is theNODE_OPTIONSsaved-preference item (branch name). The change actually addresses the deploy-canary exit-101 / self-deploy-stall lineage (docs reference #4420/#4440/#2732). Recommend pointingCloses/Refsat the correct canary issue and demoting #980 to aRefsif it's only the umbrella preference. -
Doc
status: implemented/ links toblob/main/...pre-merge (minor).docs/testing/deploy-canary-scaler-env-hermeticity.mddeclares status "implemented" and links the test atblob/main/tests/adaptive_scaling.rswhile still unmerged. Consistent with the repo's existing forward-reference doc convention, so acceptable — just be aware themainlinks 404 until this merges.
Correctly deferred (do NOT pull into this PR)
#4538(is_hijack_class_envdenylist hardening) — separate PR touchingsrc/self_relaunch/gates.rs.release.ymlduplicate-tag publish failure — explicitly out of scope; a greenunit-testgate does not turnreleasegreen. Track independently.
Recommendation: Merge-ready on code. Fix findings #1 and #2 (title/body/issue-ref metadata) before merge for a clean audit trail; no code changes required.
rysweet
left a comment
There was a problem hiding this comment.
Step 17c: Security Review — PR #4540
Verdict: No security concerns. APPROVE from a security standpoint.
Change set is test + docs only (3 files, +277/-6): tests/adaptive_scaling.rs, docs/testing/deploy-canary-scaler-env-hermeticity.md, docs/index.md. No production code, no dependencies, no CI/workflow, no config touched. Attack surface introduced: none.
Checklist
| Control | Result | Evidence |
|---|---|---|
| Sensitive-data handling | ✅ Pass | Secret scan (password/secret/token/api_key/private-key/bearer/aws_) over the full diff → no matches. Test values are literals (budgets 500.0/2500.0, scaler (2,2,2)) — not credentials. |
| Injection vulnerabilities | ✅ N/A | No new I/O, no string→command/SQL/path construction. Test builds an in-memory OodaConfig and calls decide() on static data; docs are static Markdown. Reproduction snippets use fixed literals, no interpolation of untrusted input. |
| AuthN / AuthZ | ✅ N/A | No auth surfaces touched. |
| New vulnerabilities | ✅ None | Removing ..OodaConfig::default() reduces ambient-env coupling (env→behavior fragility), a net hardening. Full field enumeration turns drift into a compile error, not a silent runtime change. |
| Env / secrets exfil surface | ✅ Pass (unchanged & sound) | The referenced gate scrub scrub_gate_env (src/self_relaunch/gates.rs) is deny-by-default; SIMARD_SCALING is not allow-listed, and a defense-in-depth test (hijack_class_name_in_canary_env_is_never_reinjected, e.g. GIT_SSH_COMMAND) confirms hijack-class names are never reinjected. This PR does not modify that allow-list — fail-closed posture preserved. |
Notes
- The test explicitly narrows env dependence (hermeticity), which is a security-positive: it eliminates a path where an inherited
SIMARD_SCALING=autoaltered execution outcome. No env var is newly read, written, or forwarded to a subprocess by this change. - Docs contain
blob/mainforward-reference links (hygiene, not security) — already noted in the Step 17b code review.
No blocking security findings. No sensitive-data exposure. Fail-closed env-scrub posture unchanged.
rysweet
left a comment
There was a problem hiding this comment.
Step 17d: Philosophy Guardian Review — PR #4540
Verdict: PASS. Independently verified against the diff (3 files, +277/−6, test + docs only; zero production code).
Compliance checklist
- [x] Ruthless simplicity achieved. The fix is the minimal correct move: delete the env-coupled
..OodaConfig::default()spread and pin an explicitAdaptiveScaler::new(2,2,2). No new abstraction, helper, or config surface introduced. It removes hidden behavior rather than adding machinery. - [x] Bricks & studs pattern followed. Touches only the test brick (
tests/adaptive_scaling.rs) and its doc; the productiondecide()/OodaConfigstuds are untouched. The scaler-override contract (scaler wins overmax_concurrent_actions) is documented at the boundary, not reimplemented. - [x] Zero-BS implementation. No stubs, no faked APIs, no swallowed exceptions. The assertion is real and load-bearing (
actions.len() <= 2with a descriptive panic message). All 9/9OodaConfigfields are enumerated againstsrc/ooda_loop/types.rs(max_concurrent_actions,improvement_threshold,gym_suite_id,daily_budget_usd,weekly_budget_usd,distill_min_episodes,distill_interval_cycles,lesson_recurrence_threshold,scaler) — a missing/renamed field is a compile error, never a silent drift. Verified independently. - [x] No over-engineering. The fix resists scope creep: it does not pull in #4538 denylist hardening or the
release.ymlduplicate-tag failure, both correctly marked out-of-scope. Dropping the spread in favor of explicit enumeration is the loud-and-explicit choice over implicit-and-env-coupled — the right trade for a hermeticity fix. - [x] Clean module boundaries. Test-only change respects the test↔production boundary. The doc explains the belt-and-suspenders relationship to
scrub_gate_envwithout duplicating or leaking production logic.
Note (non-blocking)
The 254-line reference doc is substantial for a one-test fix, but it documents a genuine, recurring failure mode (exit-101 canary stall) and the reusable S1/S2/S3 hermeticity contract for all scaler-sensitive tests — this is durable knowledge, not gold-plating. Acceptable and consistent with repo docs convention.
Philosophy compliance: PASS. No changes required.
…cs nav The new docs/testing/deploy-canary-scaler-env-hermeticity.md page was added and linked from docs/index.md but not registered in the mkdocs.yml nav, making it a nav orphan flagged by scripts/verify-docs.sh T4 (every docs/**/*.md must appear in nav). Its sibling testing docs (hermetic-tests, deflaking-known-flaky-tests) are all navved. Add the Testing nav entry so the page is discoverable and the PR does not add another orphan to the docs gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
De-flake the deploy-canary scaler test by removing ambient-environment coupling. The test now fully enumerates all 9
OodaConfigfields (dropping..OodaConfig::default()) and pins an explicitAdaptiveScaler::new(2,2,2), making the verdict a pure function of the test's own inputs rather than ofSIMARD_SCALING/SIMARD_*ambient env.Scope
Test + docs only. No production code, dependencies, CI, or config touched.
Changed files (3)
tests/adaptive_scaling.rs— enumerate all 9OodaConfigfields; pin scalerdocs/testing/deploy-canary-scaler-env-hermeticity.md— S1/S2/S3 hermeticity contract (new)docs/index.md— link the new docDiff stat
Issue
Related to the deploy-canary scaler-env hermeticity work (branch
feat/issue-980-...). The priorCloses #980auto-close ref was removed — it did not match this PR's deploy-canary lineage. Repoint to the correct tracking issue before merge if one exists.Behavior
Implemented through these branch commits:
a7194d81refactor(test): align Arc import with sibling in adaptive_scalingc1d60669fix(test): de-flake deploy-canary via SIMARD_SCALING hermeticityValidation
cargo test --test adaptive_scalingpasses deterministically, including under hostile ambient env (SIMARD_SCALING=auto SIMARD_MAX_CONCURRENT_ACTIONS=24 ...) across 3 consecutive serial runs (22 passed each). Regression fails pre-fix (exit 101), passes post-fix.Risk
Low. Test + docs only; net reduction in ambient-env coupling (hardening).
Checklist