Skip to content

fix(plan): stop tweak/second pass regressing an in-progress force export - #4398

Closed
mbuhansen wants to merge 1 commit into
springfall2008:mainfrom
mbuhansen:fix/export-flapping-monotonic-tweak
Closed

fix(plan): stop tweak/second pass regressing an in-progress force export#4398
mbuhansen wants to merge 1 commit into
springfall2008:mainfrom
mbuhansen:fix/export-flapping-monotonic-tweak

Conversation

@mbuhansen

Copy link
Copy Markdown
Contributor

Summary

During a force export at the highest export price of the horizon, the 20:05 recompute cancelled the export and switched the inverter to Demand. The plan handed to tweak_plan had metric -16218.30 (cost -16670.26) and the plan it returned had metric -15846.94 (cost -16308.52) — 371.36 worse on metric, 361.74 worse on cost — with the running export moved from "start now at 0%" to "start at 20:40".

The whole-plan guard (should_replace_plan) only compares the new plan against the previous cycle's plan, so the value tweak_plan had just destroyed was invisible and the worse plan was adopted. The export then flapped Exporting -> Demand -> Exporting -> Demand over the next 16 minutes, idling the battery through the best-priced hour of the 48h horizon.

Two defects combine

1. tweak_plan and optimise_full_second_pass are non-monotonic. Both write the result of optimise_export / optimise_charge_limit straight back with no check that the whole plan improved. Those functions score candidates against the window being turned off, not against the setting the plan already holds, so they return the best of their own candidate set and can silently downgrade a better choice made by an earlier pass. tweak_plan also resets the window start back to start_orig first, re-opening every later start in the window.

2. The in-progress export commitment did not discriminate by start time. The metric bonus added by #4118 was applied to every candidate in the window, including ones starting after minutes_now. Giving "stop now, restart later" the same bonus as "keep exporting" cancels it out, so the commitment provided no stickiness at all — only the cost-gate relaxation (keep_export) was correctly scoped.

Changes

Adds plan_metric_current / plan_window_snapshot / plan_window_restore / keep_plan_change_if_improved, and uses them to make both passes monotonic: each window change is re-simulated against the whole plan and reverted when it does not improve on metric — the same objective should_replace_plan and the swap passes already use. The comparison is strict, so a change that merely ties is also reverted, which reduces plan churn.

Scopes the metric bonus to start <= minutes_now, matching the keep_export condition that already sits directly below it.

The sibling swap passes already guard themselves this way (optimise_swap_export, optimise_swap_charge); this brings tweak and the second pass in line rather than introducing a new principle.

tweak_plan loses its now-unused best_metric / metric_keep parameters, since it measures the plan it is handed instead of trusting the caller. optimise_full_second_pass keeps its signature.

Field verification

Two full days of production logs from the same system (Kostal, 44.79 kWh, calculate_second_pass off), comparing Swap export optimisation finished against Tweak optimisation finished on every cycle:

tweak passes regressed the plan
v8.46.8 (before) 104 25 (24%)
v8.46.10 + this change 122 0

tweak_plan remains useful — 86 improvements and 36 no-ops out of 122 — so the guard removes the losses without neutering the pass.

Behaviourally, the forced export then ran unbroken for two hours through the price peak: 25 consecutive cycles Exporting, with the window start pinned at 19:00 - 22:01 for the entire period and no set force export to False in between. Previously it flapped in and out of Demand. No exceptions or tracebacks across the full day.

Note the production run used the C++ prediction kernel, which the unit tests skip when the shared library is not built.

Performance

tweak_plan is capped at 8 windows, so it adds at most 8 extra run_prediction_metric calls. optimise_full_second_pass has no cap and adds one per window, which is the larger concern — measured on optimise_windows_kernel (the test that exercises second_pass=True), two runs each:

time
without the change 46.95s / 47.65s
with the change 42.69s / 43.04s

No regression. The pass appears to converge slightly faster, plausibly because later windows are no longer optimised on top of a degraded intermediate plan.

Tests

Extends tests/test_export_commitment.py with two cases:

  • an in-progress export must not be restarted later inside its own window
  • tweak_plan must revert a window change that makes the plan worse

The second was verified to fail without the guard, reproducing the exact symptom from the log (tweak_plan kept a delayed export start 780 instead of reverting to 690).

optimise_full_second_pass is covered by the existing optimise_windows_kernel test, which runs it with second_pass=True.

During a force export at the highest export price of the horizon, the 20:05
recompute cancelled the export and switched to Demand. The plan handed to
tweak_plan cost -16670.26 and the plan it returned cost -16308.52 - 361.74
worse - with the running export moved from "start now at 0%" to "start at
20:40". The whole-plan guard only compares against the previous cycle's plan,
so the value tweak had just destroyed was invisible and the worse plan was
adopted. The export then flapped Exporting -> Demand -> Exporting -> Demand
over the next 16 minutes.

Two defects combine:

1. tweak_plan and optimise_full_second_pass write the result of
   optimise_export/optimise_charge_limit straight back with no check that the
   whole plan improved. Those functions score candidates against the window
   being turned off, not against the setting the plan already holds, so they
   return the best of their own candidate set and can silently downgrade a
   better choice made by an earlier pass. tweak_plan also resets the window
   start back to start_orig first, re-opening every later start in the window.

2. The in-progress export commitment applied its metric bonus to every
   candidate in the window, including ones starting after minutes_now. Giving
   "stop now, restart later" the same bonus as "keep exporting" cancels it out,
   so the commitment provided no stickiness - only the cost-gate relaxation was
   correctly scoped.

Add plan_metric_current/plan_window_snapshot/plan_window_restore and
keep_plan_change_if_improved, and use them to make both passes monotonic: each
window change is re-simulated against the whole plan and reverted when it does
not improve. Scope the metric bonus to start <= minutes_now, matching the
existing keep_export condition.

The sibling swap passes already guard themselves this way (optimise_swap_export,
optimise_swap_charge); this brings tweak and the second pass in line.

Extends tests/test_export_commitment.py with two cases: an in-progress export
must not be restarted later inside its own window, and tweak_plan must revert a
window change that makes the plan worse (verified to fail without the guard).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@springfall2008

Copy link
Copy Markdown
Owner

Replace with a new implementation, mainly due to preference on the coding, thanks for the fix

springfall2008 added a commit that referenced this pull request Jul 31, 2026
* docs(plan): design spec for monotonic tweak and second pass

Records the root cause analysis for tweak_plan and optimise_full_second_pass
writing back window changes that make the whole plan worse, and the design for
guarding both passes.

Diagnosis, field logs and performance measurements originate from PR #4398 by
@mbuhansen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(plan): stop tweak and second pass regressing the plan

tweak_plan and optimise_full_second_pass call optimise_charge_limit /
optimise_export one window at a time and write the result straight back with
no check that the whole plan improved. Neither optimiser ranks candidates on
the metric the plan is finally judged by: both score against a fixed default
(charge to max, or export off) rather than the setting the plan already holds,
and both rank on a score carrying adjustments the plan metric does not.

The reported symptom was a force export at the best price of the horizon being
cancelled mid-flight, then flapping Exporting -> Demand for 16 minutes.
should_replace_plan could not catch it because it only compares against the
previous cycle's plan, not against the value tweak_plan had just destroyed.

Both passes now snapshot the single window they are about to change, and
revert it unless the whole-plan metric did not worsen. The metric comes from
the optimiser rather than a fresh simulation: it already simulates the entire
plan for every option it tries, so the unadjusted metric of the option it
selected describes the plan we now hold. Both optimisers therefore return that
unadjusted metric alongside the adjusted one they rank on, leaving
optimise_detailed_pass - which gates on the adjusted score - untouched.

Ties are kept, matching optimise_swap_export. Reverting them was tried first
and measured worse: a metric-neutral revert leaves a differently shaped plan
of equal value, and the passes after tweak amplify that downstream.

Also scopes the in-progress export commitment bonus to candidates that cover
the current minute. Applying it to later-starting candidates too cancels it
out, which is what let "stop now, restart later" win.

Removes the dead best_soc_margin field, which applied after selection and
would otherwise mean the SoC written back was not the one simulated. It is
assigned 0.0 in fetch.py and 0 in predbat.py and set nowhere else.

Tests: the shared harness built its Prediction before soc_kw/soc_max and the
rates were set, so it simulated an empty battery, every plan cost 0.0 and any
metric assertion was silently a no-op. Fixed, and guarded by a test. Adds
coverage for both passes reverting a regression, the in-progress export not
being restarted later in its window, and the reported metric matching a fresh
whole-plan simulation for both branches.

Random benchmark over 20 scenarios against main: 19 unchanged, 1 better, 0
worse. Diagnosis, field logs and the original fix are from PR #4398 by
@mbuhansen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants