research(worldmodel): overnight forward-eval — no knob lifts control (nodes 7-13)#22
Open
aurascoper wants to merge 8 commits into
Open
research(worldmodel): overnight forward-eval — no knob lifts control (nodes 7-13)#22aurascoper wants to merge 8 commits into
aurascoper wants to merge 8 commits into
Conversation
…t a treatment Audited JEPATransitionDataset (WorldModel/eeg_jepa.py): the default normalize=True already z-scores per feature (mean/std from the train split, clamp at sigma 8), so the log_features=OFF baseline that node 33 calls raw z-scored IS standardization. Drop standardize as an A/B arm; it is the control. Next distinct buildable arm is symlog. Documentation only, no code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
Add symlog = sign(x)*log1p(|x|) as a mutually-exclusive-with-log_features transform at the JEPATransitionDataset seam (WorldModel/eeg_jepa.py), applied pre-normalization so stats are computed in symlog-space. Unlike log_features, log1p(0)=0 preserves zero/near-dead channels exactly instead of mapping them to log(1e-6) ~ -13.8, a large negative outlier the z-score amplifies. Thread symlog through evaluate() and all three dataset constructions (train, rollout, MPC-encode) so they share one input space; add --symlog CLI, config panel field, and a smoke assertion that the symlog path stays finite. Smoke test passes. A/B benchmark pending (node 9). WorldModel/ only, two files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
12-cell A/B {symlog off/on} x {signal,nuisance} x seeds{0,1,2}, n=384, 22 epochs.
Pooled OFF->ON: rollout 0.0120->0.0122 (delta +0.0002, well below the +/-0.0022
seed noise), mpc 0.342->0.358 (within noise), chi 0.9501->0.9545 (small consistent
gain). Confirms node 8: the epsilon floor was log_features' rollout harm (which was
+37%); symlog with log1p(0)=0 removes it, dropping the harm to ~0. But symlog does
not IMPROVE the forward metrics, so it stays default OFF. Three arms in (standardize
= baseline, log_features = reject, symlog = neutral), no transform beats raw z-score.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
Per the overnight user redirect: the transform-A/B line converged (no input-space transform beats raw z-score; nodes 7/33/9), so pivot to the weakest forward metric -- MPC success barely beats random. Expose the CEM planner knobs (mpc_cem_iters, mpc_n_samples, mpc_elite_frac) through evaluate() and the CLI; they were hardcoded at 3/64/0.2 in _mpc_success. Add them to the config panel. Single-seed sanity: base (3/64) MPC 0.300 -> tuned (8/192) MPC 0.400 vs the same 0.200 random baseline, same predictor -- more planning budget lifts success. Proper multi-seed sweep launched to background for node 11. Smoke test passes (defaults unchanged). WorldModel/forward_eval.py only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
12-cell A/B {default(3,64) vs tuned(8,192)} x {signal,nuisance} x seeds{0,1,2},
n=384, 22 epochs. mpc_success default=0.342+/-0.151 vs tuned=0.325+/-0.144 (delta
-0.017, within the +/-0.15 seed noise). Per (mode,seed) the tuned plans match
default almost exactly — more planning budget finds the same plans. The single-seed
0.30->0.40 sanity was noise. Reject the CEM-budget hypothesis; keep default 3/64.
The MPC ceiling (~0.34 vs 0.29 random) is the predictor's latent-dynamics accuracy,
not the planner. Next lever: predictor quality (epochs / latent_dim).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
12-cell A/B {default(22ep,32d) vs pushed(44ep,64d)} x {signal,nuisance} x seeds.
mpc_success 0.342 -> 0.292 (lands exactly on the 0.2917 random baseline: control
collapses to chance). rollout_mean 0.0120 -> 0.0478 (4x worse). chi 0.9501 -> 0.9744
(better static factor recovery). So a bigger/longer predictor polishes the
representation while destroying forward dynamics + control -- same dissociation as
node 33's log_features. The 4x rollout jump fingers latent_dim=64 over the epochs.
Reject the capacity lever; keep default. Accumulated: no knob (transform / planner /
predictor capacity) lifts control above ~0.34 vs 0.29 chance. The ceiling is the
JEPA objective (representation, not control) -- a design decision for the user, not
an overnight build. One clean confound-decoupling A/B (epochs-only vs latent-only)
remains before declaring the knob line converged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
…line converged
Decoupling A/B {epochs_only(44,32) vs latent_only(22,64)} vs node-12 default(22,32).
Corrects node 12's attribution: the 4x rollout blow-up was EPOCHS (rollout
0.0120->0.0494, mpc unchanged 0.333), not latent_dim. The MPC collapse was
LATENT_DIM (mpc 0.342->0.217, below the 0.292 random baseline, while rollout stayed
0.0105 and chi rose to 0.970). So rollout_error and mpc_success are driven by
different knobs and are not interchangeable proxies; and a bigger latent is a better
representation (chi up) yet a worse planning substrate (MPC below chance).
Keep default (22ep,32d). Across nodes 7-13 no knob lifts control above ~0.34 vs 0.29
chance; representation-improving changes are neutral-to-harmful for control. The knob
line is CONVERGED — the ceiling is the JEPA objective and the goal-latent planning
geometry, both design decisions for the user, not overnight builds. Parking here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
…symlog Two review findings (neither corrupts the delivered synthetic A/Bs, which both reviewers verified sound): - Finding A (provenance): add mpc_elite_frac to the forward_eval.py config panel; node 10 claimed all three CEM knobs were logged but elite_frac was omitted. The node-11 sweep held it at 0.2, so its numbers are unaffected. - Finding B (latent trap): add --symlog to the real-data trainer eeg_jepa.py and thread it into the train + val JEPATransitionDataset constructions; it had --log-features but not --symlog, so a future real-EEG symlog A/B would have silently trained the baseline. Mutual-exclusion guard confirmed on this path too. - Also refresh the evaluate() docstring to describe symlog (node-7) alongside log_features (node-33), and ledger the three findings incl. Finding C (symlog == log1p on nonnegative powers; the epsilon-floor distinction is the real one). Both smoke tests pass. WorldModel/ only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MprrYMDxCHVsP8UiHkwBBx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Overnight autonomous WorldModel forward-eval research (nodes 7–13), run under the /loop harness. Systematically tests whether any tunable knob improves control (goal-conditioned latent MPC) on the synthetic-1/f JEPA. Two small eval-harness features + a fully-ledgered chain of A/Bs.
WorldModel/only — noSources/, no app coupling.The result (converged)
No knob lifts control above chance, and every representation-improving change is neutral-to-harmful for it. The ceiling is the JEPA objective (it optimizes representation, not control) and the goal-latent planning geometry.
standardizeis the baseline, not a treatment (the dataset already z-scores at the seam)symlog(`sign(x)·log1p(symlog= neutral; confirmed the epsilon-floor (not log-compression) waslog_features' node-33 harmmpc_cem_iters/mpc_n_samples/mpc_elite_frac) throughevaluate()+ CLIlatent_dimcrashes MPC below chance (rollout-neutral); rollout and mpc are driven by different knobs and are not interchangeable proxiesFull narrative in
WorldModel/EXPERIMENT_LEDGER.md(nodes 7–13).Code changes (2 files)
WorldModel/eeg_jepa.py—symlogtransform onJEPATransitionDataset(mutually exclusive withlog_features;log1p(0)=0avoids the epsilon-floor outlier).WorldModel/forward_eval.py— threadsymlog+ expose CEM knobs throughevaluate(), the 3 dataset constructions, the config panel, and the CLI; smoke-test extended to exercise the symlog path.Tests
venv/bin/python WorldModel/forward_eval.py --smoke-testpasses (baseline +log_features+symlogarms all finite). A/Bs were run at n=384, 22 epochs, 3 seeds × 2 modes each; numbers in the ledger.Scope / caveats
WorldModel/README.md's decoupling).🤖 Generated with Claude Code