Skip to content

fix(test): de-flake deploy-canary scaler test via env hermeticity - #4540

Draft
rysweet wants to merge 3 commits into
mainfrom
feat/issue-980-nodeoptions-max-old-space-size32768-saved-preferen
Draft

fix(test): de-flake deploy-canary scaler test via env hermeticity#4540
rysweet wants to merge 3 commits into
mainfrom
feat/issue-980-nodeoptions-max-old-space-size32768-saved-preferen

Conversation

@rysweet

@rysweet rysweet commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Summary

De-flake the deploy-canary scaler test by removing ambient-environment coupling. The test now fully enumerates all 9 OodaConfig fields (dropping ..OodaConfig::default()) and pins an explicit AdaptiveScaler::new(2,2,2), making the verdict a pure function of the test's own inputs rather than of SIMARD_SCALING/SIMARD_* ambient env.

Scope

Test + docs only. No production code, dependencies, CI, or config touched.

Changed files (3)

  • tests/adaptive_scaling.rs — enumerate all 9 OodaConfig fields; pin scaler
  • docs/testing/deploy-canary-scaler-env-hermeticity.md — S1/S2/S3 hermeticity contract (new)
  • docs/index.md — link the new doc

Diff stat

 docs/index.md                                      |   1 +
 docs/testing/deploy-canary-scaler-env-hermeticity.md | 254 +++++++++++++++++
 tests/adaptive_scaling.rs                          |  28 ++-
 3 files changed, 277 insertions(+), 6 deletions(-)

Note: an earlier revision of this PR body listed 12 .github/scripts/meeting_backend files. Those are not part of this diff. The list above reflects the actual changed files.

Issue

Related to the deploy-canary scaler-env hermeticity work (branch feat/issue-980-...). The prior Closes #980 auto-close ref was removed — it did not match this PR's deploy-canary lineage. Repoint to the correct tracking issue before merge if one exists.

Behavior

Implemented through these branch commits:

  • a7194d81 refactor(test): align Arc import with sibling in adaptive_scaling
  • c1d60669 fix(test): de-flake deploy-canary via SIMARD_SCALING hermeticity

Validation

cargo test --test adaptive_scaling passes deterministically, including under hostile ambient env (SIMARD_SCALING=auto SIMARD_MAX_CONCURRENT_ACTIONS=24 ...) across 3 consecutive serial runs (22 passed each). Regression fails pre-fix (exit 101), passes post-fix.

Risk

Low. Test + docs only; net reduction in ambient-env coupling (hardening).

Checklist

  • Branch 2 commits ahead of main
  • Code review completed (APPROVE — non-blocking metadata findings)
  • Security review completed (no concerns; net hardening)
  • Philosophy check passed (PASS)

rysweet and others added 2 commits July 24, 2026 01:42
The `unit-test` deploy-gate canary reddened with exit 101 and stalled
self-deploy. Root cause: `scaler_current_max_can_override_config` in
tests/adaptive_scaling.rs built its OodaConfig with a `..OodaConfig::default()`
spread. Under an inherited `SIMARD_SCALING=auto`, `OodaConfig::default()`
(src/ooda_loop/types.rs:368) injects an `AdaptiveScaler` with ceiling 24, and
`decide()` (src/ooda_loop/decide.rs:42) honors that scaler over
`max_concurrent_actions`. With 5 over-quota priorities the test got 5 actions,
the `<= 2` assertion panicked, `cargo test` exited 101, and the gate reddened.

Fix (test-only, additive, no production/assertion change):
- Pin an explicit `AdaptiveScaler::new(2, 2, 2)` (adjust() deterministically 2).
- Fully enumerate all nine OodaConfig fields — drop the env-derived
  `..OodaConfig::default()` spread — and set `scaler: Some(pinned)` with
  `max_concurrent_actions: 8` so the pinned ceiling of 2 must win regardless of
  ambient env. This matches the established `scaler: None` hardening on the
  sibling `decide_respects_max_concurrent_actions` (issue #2732).
- Add docs/testing/deploy-canary-scaler-env-hermeticity.md and link it from
  docs/index.md, specifying the scaler-sensitive-test hermeticity contract.

Validated: original body reproduces exit-101 under SIMARD_SCALING=auto; fixed
body passes deterministically across auto/fixed/unset (x3 each); full
adaptive_scaling suite (22 tests) green under SIMARD_SCALING=auto;
`cargo fmt --check` and `cargo clippy --all-targets -- -D warnings` clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a local `use std::sync::Arc;` + `Arc::new(...)` in
scaler_current_max_can_override_config, matching the file's existing
convention in the concurrency test below it, instead of the fully
qualified `std::sync::Arc::new(...)`. Behavior-preserving readability
tidy; no logic, assertion, or hermeticity-contract change.

Validated: target test passes x3 under hostile SIMARD_SCALING=auto +
SIMARD_MAX_CONCURRENT_ACTIONS=10; full adaptive_scaling suite (22) green;
cargo fmt --check and clippy --test adaptive_scaling -D warnings clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@rysweet rysweet left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 17b: Comprehensive Code Review — PR #4540

Verdict: APPROVE (with non-blocking metadata findings). The code change is correct, minimal, well-tested, and thoroughly documented. Independently verified below. No blocking issues in the code, test, or docs.

Actual change set (3 files)

M  docs/index.md                                             (+1 link line)
A  docs/testing/deploy-canary-scaler-env-hermeticity.md      (+254, new reference)
M  tests/adaptive_scaling.rs                                 (test-only, +18/-7)

Test-only + docs. No production code, no signature changes, no algorithm change. Additive and non-breaking.

Independently verified ✅

Check Result
OodaConfig field enumeration 9/9 exact — struct in src/ooda_loop/types.rs (max_concurrent_actions, improvement_threshold, gym_suite_id, daily_budget_usd, weekly_budget_usd, distill_min_episodes, distill_interval_cycles, lesson_recurrence_threshold, scaler) fully matches the enumerated test config. Dropping ..OodaConfig::default() makes any drift a compile error, not a silent env-coupling — correct trade-off.
Scaler-override logic src/ooda_loop/decide.rs:42 if let Some(ref scaler) = config.scaler { scaler.adjust() } confirms the scaler wins over max_concurrent_actions. Pinned AdaptiveScaler::new(2,2,2)adjust()==2 < config's 8, so assert actions.len() <= 2 is deterministic.
Consistency with siblings Matches the hardened pattern in decide_uses_scaler_adjusted_limit_when_scaler_is_present / decide_ignores_scaler_when_none / decide_respects_max_concurrent_actions (all pin scaler: explicitly, per #2732).
Hermeticity intent The verdict is now a pure function of the test's own inputs; SIMARD_SCALING=auto no longer flips the assertion. Root cause (ambient SIMARD_SCALING=auto leaking through the ..default() spread) is correctly identified and eliminated.

Review checklist

  • Code quality and standards — idiomatic Rust; Arc import aligned with sibling; clear comments explaining the hermeticity contract
  • Test coverage adequate — the fix is a regression test; fails pre-fix (panic → exit 101) and passes post-fix under both ambient and clean env
  • No TODOs, stubs, or swallowed exceptions
  • No unimplemented functions
  • Logic correctness — verified against decide() and AdaptiveScaler
  • Edge-case handling — hostile ambient env (SIMARD_SCALING=auto) is the explicit edge case covered

Non-blocking findings (PR hygiene — recommend fixing before merge, do NOT touch code)

  1. Title/body do not match the actual diff. Title says "Update .github with 12 changed files (#980)" and the body lists 12 files (release.yml, verify.yml, scripts/*, meeting_backend tests, etc.) with a 330 insertions / 435 deletions diff stat. The actual PR is 3 files, +277/-6, and touches no .github/ workflow or scripts/. The body's file list and diff stat are stale/templated. Recommend regenerating the title + body to describe the real change (deploy-canary SIMARD_SCALING hermeticity test fix). This matters for auditability.

  2. Closes #980 looks incorrect for this change. #980 is the NODE_OPTIONS saved-preference item (branch name). The change actually addresses the deploy-canary exit-101 / self-deploy-stall lineage (docs reference #4420/#4440/#2732). Recommend pointing Closes/Refs at the correct canary issue and demoting #980 to a Refs if it's only the umbrella preference.

  3. Doc status: implemented / links to blob/main/... pre-merge (minor). docs/testing/deploy-canary-scaler-env-hermeticity.md declares status "implemented" and links the test at blob/main/tests/adaptive_scaling.rs while still unmerged. Consistent with the repo's existing forward-reference doc convention, so acceptable — just be aware the main links 404 until this merges.

Correctly deferred (do NOT pull into this PR)

  • #4538 (is_hijack_class_env denylist hardening) — separate PR touching src/self_relaunch/gates.rs.
  • release.yml duplicate-tag publish failure — explicitly out of scope; a green unit-test gate does not turn release green. Track independently.

Recommendation: Merge-ready on code. Fix findings #1 and #2 (title/body/issue-ref metadata) before merge for a clean audit trail; no code changes required.

@rysweet rysweet left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 17c: Security Review — PR #4540

Verdict: No security concerns. APPROVE from a security standpoint.

Change set is test + docs only (3 files, +277/-6): tests/adaptive_scaling.rs, docs/testing/deploy-canary-scaler-env-hermeticity.md, docs/index.md. No production code, no dependencies, no CI/workflow, no config touched. Attack surface introduced: none.

Checklist

Control Result Evidence
Sensitive-data handling ✅ Pass Secret scan (password/secret/token/api_key/private-key/bearer/aws_) over the full diff → no matches. Test values are literals (budgets 500.0/2500.0, scaler (2,2,2)) — not credentials.
Injection vulnerabilities ✅ N/A No new I/O, no string→command/SQL/path construction. Test builds an in-memory OodaConfig and calls decide() on static data; docs are static Markdown. Reproduction snippets use fixed literals, no interpolation of untrusted input.
AuthN / AuthZ ✅ N/A No auth surfaces touched.
New vulnerabilities ✅ None Removing ..OodaConfig::default() reduces ambient-env coupling (env→behavior fragility), a net hardening. Full field enumeration turns drift into a compile error, not a silent runtime change.
Env / secrets exfil surface ✅ Pass (unchanged & sound) The referenced gate scrub scrub_gate_env (src/self_relaunch/gates.rs) is deny-by-default; SIMARD_SCALING is not allow-listed, and a defense-in-depth test (hijack_class_name_in_canary_env_is_never_reinjected, e.g. GIT_SSH_COMMAND) confirms hijack-class names are never reinjected. This PR does not modify that allow-list — fail-closed posture preserved.

Notes

  • The test explicitly narrows env dependence (hermeticity), which is a security-positive: it eliminates a path where an inherited SIMARD_SCALING=auto altered execution outcome. No env var is newly read, written, or forwarded to a subprocess by this change.
  • Docs contain blob/main forward-reference links (hygiene, not security) — already noted in the Step 17b code review.

No blocking security findings. No sensitive-data exposure. Fail-closed env-scrub posture unchanged.

@rysweet rysweet left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 17d: Philosophy Guardian Review — PR #4540

Verdict: PASS. Independently verified against the diff (3 files, +277/−6, test + docs only; zero production code).

Compliance checklist

  • [x] Ruthless simplicity achieved. The fix is the minimal correct move: delete the env-coupled ..OodaConfig::default() spread and pin an explicit AdaptiveScaler::new(2,2,2). No new abstraction, helper, or config surface introduced. It removes hidden behavior rather than adding machinery.
  • [x] Bricks & studs pattern followed. Touches only the test brick (tests/adaptive_scaling.rs) and its doc; the production decide()/OodaConfig studs are untouched. The scaler-override contract (scaler wins over max_concurrent_actions) is documented at the boundary, not reimplemented.
  • [x] Zero-BS implementation. No stubs, no faked APIs, no swallowed exceptions. The assertion is real and load-bearing (actions.len() <= 2 with a descriptive panic message). All 9/9 OodaConfig fields are enumerated against src/ooda_loop/types.rs (max_concurrent_actions, improvement_threshold, gym_suite_id, daily_budget_usd, weekly_budget_usd, distill_min_episodes, distill_interval_cycles, lesson_recurrence_threshold, scaler) — a missing/renamed field is a compile error, never a silent drift. Verified independently.
  • [x] No over-engineering. The fix resists scope creep: it does not pull in #4538 denylist hardening or the release.yml duplicate-tag failure, both correctly marked out-of-scope. Dropping the spread in favor of explicit enumeration is the loud-and-explicit choice over implicit-and-env-coupled — the right trade for a hermeticity fix.
  • [x] Clean module boundaries. Test-only change respects the test↔production boundary. The doc explains the belt-and-suspenders relationship to scrub_gate_env without duplicating or leaking production logic.

Note (non-blocking)

The 254-line reference doc is substantial for a one-test fix, but it documents a genuine, recurring failure mode (exit-101 canary stall) and the reusable S1/S2/S3 hermeticity contract for all scaler-sensitive tests — this is durable knowledge, not gold-plating. Acceptable and consistent with repo docs convention.

Philosophy compliance: PASS. No changes required.

@rysweet rysweet changed the title Update .github with 12 changed files (#980) fix(test): de-flake deploy-canary scaler test via env hermeticity Jul 24, 2026
…cs nav

The new docs/testing/deploy-canary-scaler-env-hermeticity.md page was added
and linked from docs/index.md but not registered in the mkdocs.yml nav, making
it a nav orphan flagged by scripts/verify-docs.sh T4 (every docs/**/*.md must
appear in nav). Its sibling testing docs (hermetic-tests, deflaking-known-flaky-tests)
are all navved. Add the Testing nav entry so the page is discoverable and the
PR does not add another orphan to the docs gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant