From 39aae37b64c3734d575121b0ff7550942bd8baed Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 18:39:59 +0200 Subject: [PATCH 001/119] docs: design for the annual prediction tool Spec for a standalone tool that projects a year of household electricity costs using the real Predbat planning engine, reporting each month under three scenarios: no PV/battery, PV+battery without Predbat, and with Predbat. Prediction engine only; the web UI is separate later work. Co-Authored-By: Claude Opus 5 (1M context) --- ...026-07-25-annual-prediction-tool-design.md | 358 ++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md diff --git a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md new file mode 100644 index 000000000..ccb5ce9ba --- /dev/null +++ b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md @@ -0,0 +1,358 @@ +# Annual Prediction Tool — Design + +Date: 2026-07-25 +Status: Approved (design review complete) + +## Goal + +Add a standalone tool that projects a **year** of household electricity costs +using the real Predbat planning engine, so a prospective or existing user can +answer "what would a battery, or solar, or Predbat itself, actually save me +over a year?" without owning any of the hardware. + +For each month the tool reports three scenarios: + +1. **No PV, no battery** — the counterfactual bill. +2. **PV and battery, without Predbat** — a dumb battery charging on a static + cheap-rate timer. +3. **PV and battery, with Predbat** — the optimiser's plan. + +This spec covers the **prediction engine only**. The web UI that consumes its +output is a separate, later piece of work; the design's job here is to leave a +clean programmatic interface for it. + +## Scope decisions (agreed) + +- **Packaged as shipped modules, not a dev script** — new `annual_*.py` modules + in `apps/predbat/` plus a thin CLI, following the `compare.py` precedent, so + the web UI can import the engine directly later. +- **Drives the real `PredBat` object** — the tool injects state and calls + `calculate_plan()` / `run_prediction()`. It does not re-implement the + optimiser. Numbers are defensible because they come from the same code that + runs live. +- **Two sampled days per month, configurable** — chosen by irradiance + percentile, not by calendar position. +- **Real historical rates for the sampled date** — genuine Agile/Tracker/Flux + shape and seasonal price movement. +- **Load is either synthetic or Octopus-derived, never both** — providing an + Octopus API key together with `annual_kwh`/`car_charging_kwh` is a config + validation error. +- **Car charging is smart under Predbat, timer-driven in the baselines** — + Predbat is credited only for what it wins over an off-peak timer. + +## Architecture + +Six new flat modules in `apps/predbat/`. Flat `annual_` prefixing matches repo +convention (no subpackages exist besides `tests/` and `config/`). + +| Module | Responsibility | Depends on | +|---|---|---| +| `annual.py` | `AnnualPredictor` orchestrator, public API, config validation | all below, `PredBat` | +| `annual_weather.py` | Open-Meteo archive client → per-day PV kW series per array | shared GTI helper | +| `annual_load.py` | `LoadProfileSource`: synthetic and Octopus-consumption implementations | `annual_profiles` | +| `annual_profiles.py` | Embedded half-hourly domestic shape tables and monthly weights (data only) | nothing | +| `annual_tariff.py` | Per-date rate resolution: Octopus URL or basic rates | `basic_rates()` | +| `annual_cli.py` | Argument parsing, progress output, JSON and table writing | `annual` | + +The boundary that matters: **`annual.py` performs no HTTP, and +`annual_weather.py` / `annual_tariff.py` never touch `PredBat`.** That keeps the +web UI's future job to "build a config dict, call `AnnualPredictor.run()`, +render the returned JSON", and lets every module be tested against fixtures +rather than the network. + +HTTP responses are cached through the Storage component's `fetch_cached()` +(per CLAUDE.md, never direct file access): one cached blob per array-year for +weather, one per tariff-month for rates. + +### Public interface + +```python +class AnnualPredictor: + def __init__(self, config: dict, log=None, storage=None): ... + async def run(self, progress=None) -> dict: ... +``` + +`progress` is an optional callable receiving `(completed, total, message)` so +the future web UI can stream status. `run()` returns the results document +described under "Results" below. + +## Reused existing code + +The tool leans on four things that already exist, rather than reinventing them: + +- **`calculate_yesterday()`** (`apps/predbat/output.py`) already runs exactly + these three scenarios against a past day — zeroed-PV/`soc_max=0` for the + no-system case, a cheapest-window dumb charge for the baseline, and the + optimised plan. It is the template for the per-sample execution. +- **`Compare.apply_hardware_overrides()`** (`apps/predbat/compare.py`) already + maps battery kWh / charge rate / inverter limit onto a `PredBat` instance. +- **The battery-value correction** `metric_end − metric_start` via + `compute_metric()`, as `Compare.run_scenario()` applies it. +- **The GTI→kW model** in `apps/predbat/solcast.py` (`download_open_meteo_data`). + +`Compare` itself is *not* reused as a class: it is coupled to +`fetch_sensor_data()` and live inverter state, and plans "now → +48h" rather +than an arbitrary historical date. Bending it to serve a second purpose would +damage it. + +## Inputs + +A single YAML file, or the equivalent dict from the future web UI. Every +optional field has a stated default so a minimal config is short. + +```yaml +annual: + location: + postcode: "SW1A 1AA" # or latitude/longitude; resolved via postcodes.io + year: 2025 # defaults to the most recent complete calendar year + + solar: # list — multiple arrays supported; omit for battery-only + - kwp: 5.6 + declination: 35 # pitch in degrees. Default 35 + azimuth: 180 # 180 = south. Default 180 + efficiency: 0.95 # system loss. Default 0.95 + + battery: # omit entirely for a PV-only run + size_kwh: 9.5 + inverter_kw: 5.0 + export_limit_kw: 5.0 + hybrid: true # false = AC coupled + charge_rate_kw: 3.6 # defaults to inverter_kw + discharge_rate_kw: 3.6 # defaults to inverter_kw + + load: + annual_kwh: 3800 # mutually exclusive with load.octopus + shape: flat # night | day | flat + car_charging_kwh: 2500 # annual; 0 to disable + octopus: # mutually exclusive with annual_kwh/car_charging_kwh + api_key: !secret octopus_key + account_id: A-1234ABCD + + tariff: + import_octopus_url: "https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{dno_region}/standard-unit-rates/" + export_octopus_url: "..." + # or: rates_import: [{start: "00:30:00", end: "05:30:00", rate: 7.0}, ...] + standing_charge_p_per_day: 60.0 + + samples_per_month: 2 + pv10_derate: 0.7 # see "P10 estimate" below +``` + +### Input rules + +- **Battery and solar are each optional.** Omitting `battery:` produces a + two-scenario run (no-PV/no-battery versus PV-only); omitting `solar:` + produces a battery-only run. The tool must not force the user to configure + the very thing they are trying to evaluate. +- **`load.octopus` versus `load.annual_kwh` is exclusive**, and supplying both + is rejected at config-load time with a clear message. The Octopus consumption + series already contains any car charging, so accepting both would + double-count it. The consequence — a real-data user gets an accurate baseline + but no separately smart-planned EV — is documented in the CLI help and the + results document. +- **`{dno_region}` is resolved** through Predbat's existing region mechanism, so + a postcode selects the correct regional tariff. Silently defaulting to region + A would quote the wrong prices with no visible symptom. +- **Secrets are scrubbed** from the results document and any debug output, + matching the existing `_key` / `password` redaction in `create_debug_yaml()`. + +## Data flow + +### 1. Weather + +One Open-Meteo **archive** request per array, covering the whole year plus one +buffer day (the last sampled day needs a following day for its 48h plan): + +``` +https://archive-api.open-meteo.com/v1/archive?latitude=..&longitude=.. + &start_date=YYYY-01-01&end_date=YYYY+1-01-01 + &hourly=global_tilted_irradiance,temperature_2m,wind_speed_10m + &tilt=..&azimuth=..&wind_speed_unit=ms&timezone=UTC +``` + +The field names match the forecast endpoint already used, so the conversion is +identical. + +**The GTI→kW conversion is extracted, not copied.** The cell-temperature model, +the −0.4%/°C derate, the trapezoidal hourly integration, and `convert_azimuth` +move out of `SolarAPI.download_open_meteo_data()` into a shared helper that both +`solcast.py` and `annual_weather.py` call. Duplicating it would guarantee the +two drift; the azimuth convention in particular (Open-Meteo uses 0 = south) is +easy to get silently wrong and produces plausible-looking but incorrect output. + +#### P10 estimate + +The archive API exposes no ensemble members, so there is no true P10. The tool +applies a configurable flat derate (`pv10_derate`, default 0.7) — the same +fallback `solcast.py` already uses when ensemble data is unavailable. This is a +known approximation and is labelled as such in the results document. + +### 2. Load + +`annual_load.py` produces a **forward cumulative kWh series keyed by absolute +minute**, assigned to `load_forecast`, with `load_forecast_only = True`. + +This is the clean injection point: with `load_forecast_only` set, +`step_data_history()` ignores historical load entirely +(`apps/predbat/fetch.py`, in the `type_load and not forward` branch) and builds +the forward profile purely from `load_forecast`, which it reads via +`get_from_incrementing(..., backwards=False)`. No synthetic backwards history +needs to be fabricated. + +**Synthetic path.** `annual_kwh × month_weight[m] / days_in_month` gives the +day's kWh, multiplied by a 48-point half-hourly domestic shape. The night/day +shape setting tilts energy between the 00:00–07:00 band and the daytime band +while preserving the daily total exactly — an invariant with a dedicated test. +The tilt magnitude is a named constant in `annual_profiles.py` (a fixed +proportion of the day's energy moved between bands) so it can be tuned against +real data without touching the tilt logic. Monthly weights carry the +winter/summer split, which drives much of the annual answer. + +`annual_profiles.py` holds these tables as data only, with no logic, so they can +be revised against real measurements without touching behaviour. + +**Octopus path.** Account lookup resolves MPAN and meter serial, then the +half-hourly consumption endpoint supplies each sampled date directly. Gaps fall +back to the synthetic profile for that date and are logged. + +### 3. Tariff + +One fetch per tariff per month using `period_from` / `period_to`, sliced per +sampled day. Per-month rather than per-day fetching keeps the API call count +low; the existing pagination handling in `download_octopus_rates_func()` still +applies. Basic rates go through `basic_rates()` unchanged. + +### 4. Sample selection + +Within each month, every day's total PV kWh is summed across all arrays and the +days sorted ascending. For `N` samples, the day at percentile `(i + 0.5) / N` +is taken for `i` in `0..N-1` — so `N=2` picks the 25th and 75th percentile days, +each representing exactly half the month. Selection is fully deterministic; the +same config always yields the same days. + +Days without a valid following day are excluded from the candidate set, since +the 48h plan requires one. + +### 5. Per-sample execution + +State is reset, then `midnight_utc = D`, `minutes_now = 0`, +`forecast_minutes = 48 * 60`, `end_record = 24 * 60`. Three scenarios run +against the same day: + +1. **No PV, no battery** — `Prediction(..., pv_zero, pv_zero, load, load, + soc_kw=0, soc_max=0)` then `run_prediction([], [], [], [])`. +2. **Without Predbat** — dumb battery: the cheapest static charge window from + `compute_rate_low_for_yesterday()` + `rate_scan_window()`, capped at + `calculate_savings_max_charge_slots`, charging to 100%, with no export + optimisation. + +The car, where configured, charges in scenarios 1 and 2 on a fixed timer: the +same cheapest static window scenario 2 derives for the battery, extended as +needed to fit the day's car kWh at the configured charge rate. Scenario 1 uses +that identical window even though it has no battery, so the only difference +between the three scenarios is the system being evaluated. Only scenario 3 +plans the car smartly. +3. **With Predbat** — `calculate_plan(recompute=True, publish=False)` then + `run_prediction(charge_limit_best, charge_window_best, export_window_best, + export_limits_best)`. The car uses real smart slot planning. + +Each is billed over the **first 24 hours only**; the second day exists purely as +lookahead so the optimiser does not artificially drain the battery at the +horizon. Leftover or consumed charge is valued via the `compute_metric` +correction (`metric_end − metric_start`), so ending on a full battery is not +free. + +### 6. State isolation + +This is the principal risk of driving the real `PredBat` object. `PredBat` is a +large mutable object and derived state leaks between runs — the debug-case +harness already documents this for `dynamic_load_baseline` and +`battery_rate_max_export` (`apps/predbat/tests/test_single_debug.py`). + +Between samples the tool resets an explicit allow-list of mutable fields, and a +test asserts that a single month run in isolation produces output identical to +that same month within a full-year run. Without that test this class of bug is +invisible: results stay plausible while silently depending on run order. + +## Results + +Monthly cost for a scenario is `Σ over samples (sample_daily_cost × +days_in_month / N)`. Stratified percentile sampling gives each sample an equal +share of the month, so the weights are uniform. + +Per month, per scenario: + +| Field | Source | +|---|---| +| `cost_p` | `run_prediction` metric, battery-value corrected, excluding standing charge | +| `import_kwh` | `import_kwh_battery + import_kwh_house` | +| `export_kwh` | `export_kwh` | +| `export_credit_p` | `export_kwh` valued at the export rate | +| `pv_generated_kwh` | summed `pv_forecast_minute_step` over the billed 24h | +| `self_consumed_kwh` | `pv_generated_kwh − export_kwh`, clamped at 0 | +| `battery_throughput_kwh` | `battery_cycle` | +| `standing_charge_p` | `standing_charge_p_per_day × days_in_month` | + +`standing_charge_p` is identical across scenarios and reported separately, so +savings comparisons are not diluted by a fixed cost neither system affects. + +`self_consumed_kwh` is an approximation: when the battery exports grid-charged +energy, `export_kwh` exceeds the PV contribution and self-consumption is +understated. This is documented in the field description rather than papered +over with a decomposition the prediction does not actually track. + +Annual totals are the sum of the twelve months, plus two derived savings +figures: PV+battery versus no-PV/no-battery, and Predbat versus without-Predbat. + +Output is a JSON document — the future web UI's input — and a human-readable +table on stdout. `--debug` additionally retains the per-sample HTML plans; off +by default, since 24 retained plans make the payload large. + +## Failure handling + +**Failures are visible, never silent.** + +| Condition | Behaviour | +|---|---| +| Missing archive weather for a sampled day | Substitute the next-nearest day within the same percentile stratum; log it | +| Missing rate data for a month | Mark the month `"status": "unavailable"`, exclude it from annual totals, and state the exclusion in the printed output | +| Octopus consumption gap for a sampled date | Fall back to the synthetic profile for that date; log it | +| Octopus account lookup failure | Fail the run with a clear message rather than silently degrading to synthetic | + +A month must never quietly become zero. A zero month reads as "free +electricity" in a chart, and the user has no way to tell it apart from a real +result. + +## Testing + +All tests run offline against fixtures and are registered in `TEST_REGISTRY` in +`unit_test.py`, following the `tests/test_.py` convention. + +- **Load profile** — shape tilts preserve the daily total exactly; the twelve + monthly totals sum to `annual_kwh`. +- **Weather** — fixture GTI converts to known kW; the extracted shared helper + produces output identical to `solcast.py`'s current path for the same inputs. + This is what prevents the extraction being a silent regression. +- **Sample selection** — deterministic percentile picks for a known irradiance + series; correct handling of a month with missing days. +- **Config validation** — Octopus key plus manual load is rejected; a missing + `battery:` block yields a two-scenario run; `{dno_region}` resolves. +- **Scenario ordering** — on a synthetic day, + `predbat_cost ≤ without_predbat_cost ≤ no_pvbat_cost`. +- **State isolation** — a single month run in isolation matches that month + within a full-year run. +- **Rate fetch** — `period_from` / `period_to` slicing and pagination against a + fixture. + +## Performance + +Twelve months at two samples is 24 `calculate_plan()` runs, roughly one to three +minutes. `AnnualPredictor.run()` accepts a progress callback so the later web UI +can stream status rather than block on a silent request. + +## Out of scope + +- The web UI (separate, later work). +- Heat pump, iBoost, or gas modelling. +- Multi-year averaging or typical-meteorological-year construction. +- Tariff recommendation — the tool evaluates the tariff it is given. From afb4189cf74325c1ca864aad66fac2cbe6f1efdf Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 18:46:17 +0200 Subject: [PATCH 002/119] docs: ground the annual tool's P10 in measured forecast error Replace the flat 0.7 P10 derate with a value derived from real day-ahead forecast error: download both the Open-Meteo ERA5 actuals archive and the archived short-range forecast for the same dates, and take the 10th percentile of the per-month actual/forecast daily energy ratio. Predbat now also plans against the forecast series and is billed against actuals, rather than being handed perfect foresight. Because the plan is costed against a fixed actuals series, hedging can only add cost, so P10 is not a free parameter and must not come from climatological spread. Co-Authored-By: Claude Opus 5 (1M context) --- ...026-07-25-annual-prediction-tool-design.md | 109 +++++++++++++++--- 1 file changed, 95 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md index ccb5ce9ba..e9501c49b 100644 --- a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md +++ b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md @@ -39,6 +39,9 @@ clean programmatic interface for it. validation error. - **Car charging is smart under Predbat, timer-driven in the baselines** — Predbat is credited only for what it wins over an off-peak timer. +- **Predbat plans on the archived forecast and is billed on actuals** — it does + not get perfect foresight, and P10 comes from measured forecast error rather + than a constant. ## Architecture @@ -48,7 +51,7 @@ convention (no subpackages exist besides `tests/` and `config/`). | Module | Responsibility | Depends on | |---|---|---| | `annual.py` | `AnnualPredictor` orchestrator, public API, config validation | all below, `PredBat` | -| `annual_weather.py` | Open-Meteo archive client → per-day PV kW series per array | shared GTI helper | +| `annual_weather.py` | Open-Meteo actuals + forecast archive clients → per-day PV kW series per array, plus monthly P10 ratios | shared GTI helper | | `annual_load.py` | `LoadProfileSource`: synthetic and Octopus-consumption implementations | `annual_profiles` | | `annual_profiles.py` | Embedded half-hourly domestic shape tables and monthly weights (data only) | nothing | | `annual_tariff.py` | Per-date rate resolution: Octopus URL or basic rates | `basic_rates()` | @@ -61,8 +64,9 @@ render the returned JSON", and lets every module be tested against fixtures rather than the network. HTTP responses are cached through the Storage component's `fetch_cached()` -(per CLAUDE.md, never direct file access): one cached blob per array-year for -weather, one per tariff-month for rates. +(per CLAUDE.md, never direct file access): one cached blob per array-year per +source for weather — actuals and forecast are separate entries — and one per +tariff-month for rates. ### Public interface @@ -135,7 +139,7 @@ annual: standing_charge_p_per_day: 60.0 samples_per_month: 2 - pv10_derate: 0.7 # see "P10 estimate" below + pv10_derate_fallback: 0.7 # only used when the forecast archive lacks the year ``` ### Input rules @@ -155,13 +159,21 @@ annual: A would quote the wrong prices with no visible symptom. - **Secrets are scrubbed** from the results document and any debug output, matching the existing `_key` / `password` redaction in `create_debug_yaml()`. +- **`year` is accepted for any date the actuals archive covers**, but years + before the forecast archive begins (around 2021–2022) lose the forecast + grounding and fall back as described under "P10 estimate". The default — the + most recent complete calendar year — is always within coverage. ## Data flow ### 1. Weather -One Open-Meteo **archive** request per array, covering the whole year plus one -buffer day (the last sampled day needs a following day for its 48h plan): +**Two** Open-Meteo requests per array, each covering the whole year plus one +buffer day (the last sampled day needs a following day for its 48h plan). Both +endpoints serve `global_tilted_irradiance` with `tilt` / `azimuth` and both +accept `start_date` / `end_date`. + +**Actuals** — ERA5 reanalysis, what genuinely happened. Data back to 1940. ``` https://archive-api.open-meteo.com/v1/archive?latitude=..&longitude=.. @@ -170,8 +182,19 @@ https://archive-api.open-meteo.com/v1/archive?latitude=..&longitude=.. &tilt=..&azimuth=..&wind_speed_unit=ms&timezone=UTC ``` -The field names match the forecast endpoint already used, so the conversion is -identical. +**Forecast** — the archived short-range forecast for those same past dates, i.e. +what Predbat would actually have been looking at. Coverage starts around +2021–2022 depending on model. + +``` +https://historical-forecast-api.open-meteo.com/v1/forecast?latitude=..&longitude=.. + &start_date=YYYY-01-01&end_date=YYYY+1-01-01 + &hourly=global_tilted_irradiance,temperature_2m,wind_speed_10m + &tilt=..&azimuth=..&wind_speed_unit=ms&timezone=UTC +``` + +The field names match the live forecast endpoint already used, so one conversion +serves all three. **The GTI→kW conversion is extracted, not copied.** The cell-temperature model, the −0.4%/°C derate, the trapezoidal hourly integration, and `convert_azimuth` @@ -180,12 +203,56 @@ move out of `SolarAPI.download_open_meteo_data()` into a shared helper that both two drift; the azimuth convention in particular (Open-Meteo uses 0 = south) is easy to get silently wrong and produces plausible-looking but incorrect output. +Open-Meteo's Previous Runs API would give cleaner fixed-lead-time forecasts, but +it accepts only `past_days` rather than a date range, so it cannot reach an +arbitrary past year. It is not used. + +#### Plan on forecast, bill on actuals + +Predbat plans against the **forecast** series and is costed against the +**actuals** series. Without this the tool would grant Predbat perfect foresight +and overstate what it can really achieve. + +Mechanically: `calculate_plan()` runs with `pv_forecast_minute` set to the +forecast series; then, before the costing `run_prediction()`, `self.prediction` +is rebuilt from the actuals step data and the plan's best windows are replayed +against it. This is the same Prediction-swap that `calculate_yesterday()` and +`Compare.run_scenario()` already perform. + +This applies to scenario 3 only. Scenario 1 has no PV, and scenario 2's charge +window is derived from rates rather than from the PV forecast, so neither +depends on forecast quality — both run directly on actuals. `pv_generated_kwh` +in the results is always taken from the actuals series. + #### P10 estimate -The archive API exposes no ensemble members, so there is no true P10. The tool -applies a configurable flat derate (`pv10_derate`, default 0.7) — the same -fallback `solcast.py` already uses when ensemble data is unavailable. This is a -known approximation and is labelled as such in the results document. +The archive exposes no ensemble members, so P10 is derived from **measured +forecast error**. For each month, the daily energy ratio +`r = actual_kWh / forecast_kWh` is computed across every day of that month; the +10th percentile of `r` becomes that month's `p10_ratio`, and the planning P10 +series is the forecast series scaled by it (clamped to ≤ 1). Location- and +season-specific, and grounded in data rather than a constant. + +Why this matters more than it looks: because the plan is costed against a fixed +actuals series, hedging can only ever *add* cost — a pessimistic P10 makes +Predbat over-charge and understates its own savings, while `P10 = P50` yields a +perfect-foresight upper bound. P10 is therefore not a free parameter, and it +must not be derived from within-month climatological spread, which is far wider +than 24-hour-ahead forecast error and would bias the tool against itself. + +Two honest caveats, recorded in the results document: + +- The forecast-versus-ERA5 gap includes systematic model bias, not purely + forecast error, so measured uncertainty is slightly overstated. +- P90 is not used. The planner consumes only `pv_forecast_minute` and + `pv_forecast_minute10`; P90 reaches no decision, so computing it would be + decoration. + +**Fallback.** If the forecast archive does not cover the requested year, the +tool logs a clear warning, plans on actuals, and applies a flat +`pv10_derate_fallback` (default 0.7 — `solcast.py`'s existing no-ensemble +fallback). The results document records that the fallback was used, so a +degraded run is never mistaken for a grounded one. ### 2. Load @@ -224,8 +291,10 @@ applies. Basic rates go through `basic_rates()` unchanged. ### 4. Sample selection -Within each month, every day's total PV kWh is summed across all arrays and the -days sorted ascending. For `N` samples, the day at percentile `(i + 0.5) / N` +Within each month, every day's total PV kWh from the **actuals** series is +summed across all arrays and the days sorted ascending — ranking on the forecast +would select days by what was predicted rather than by what the month really +contained. For `N` samples, the day at percentile `(i + 0.5) / N` is taken for `i` in `0..N-1` — so `N=2` picks the 25th and 75th percentile days, each representing exactly half the month. Selection is fully deterministic; the same config always yields the same days. @@ -315,6 +384,8 @@ by default, since 24 retained plans make the payload large. | Condition | Behaviour | |---|---| | Missing archive weather for a sampled day | Substitute the next-nearest day within the same percentile stratum; log it | +| Forecast archive does not cover the requested year | Plan on actuals with `pv10_derate_fallback`; warn, and record the degradation in the results document | +| Forecast archive has gaps within a covered year | Exclude those days from the month's `p10_ratio` sample; if fewer than seven days remain, fall back for that month and record it | | Missing rate data for a month | Mark the month `"status": "unavailable"`, exclude it from annual totals, and state the exclusion in the printed output | | Octopus consumption gap for a sampled date | Fall back to the synthetic profile for that date; log it | | Octopus account lookup failure | Fail the run with a clear message rather than silently degrading to synthetic | @@ -333,6 +404,13 @@ All tests run offline against fixtures and are registered in `TEST_REGISTRY` in - **Weather** — fixture GTI converts to known kW; the extracted shared helper produces output identical to `solcast.py`'s current path for the same inputs. This is what prevents the extraction being a silent regression. +- **P10 derivation** — a fixture pair of forecast and actual series yields the + expected monthly `p10_ratio`; a year outside forecast-archive coverage falls + back to `pv10_derate_fallback` and flags the degradation. +- **Plan-on-forecast, bill-on-actuals** — with a deliberately inflated forecast + series, scenario 3's reported cost and `pv_generated_kwh` track the actuals, + not the forecast. Without this test the Prediction swap could silently be + skipped and every result would quietly assume perfect foresight. - **Sample selection** — deterministic percentile picks for a known irradiance series; correct handling of a month with missing days. - **Config validation** — Octopus key plus manual load is rejected; a missing @@ -355,4 +433,7 @@ can stream status rather than block on a silent request. - The web UI (separate, later work). - Heat pump, iBoost, or gas modelling. - Multi-year averaging or typical-meteorological-year construction. +- A P90-based confidence band on the monthly figures. The forecast-error + distribution would support one, but nothing consumes it and no decision + depends on it. - Tariff recommendation — the tool evaluates the tariff it is given. From ec2345a1cd7c9171e732265d61e3efc6608559bb Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 19:30:19 +0200 Subject: [PATCH 003/119] docs: add the annual prediction tool implementation plan Thirteen TDD tasks covering the shared solar model extraction, load profile sources, the Open-Meteo actuals and forecast archives, historical tariff resolution, the headless PredBat bootstrap, sample selection, the three-scenario day runner, orchestration, CLI and documentation. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-07-25-annual-prediction-tool.md | 4565 +++++++++++++++++ 1 file changed, 4565 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-25-annual-prediction-tool.md diff --git a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md new file mode 100644 index 000000000..84d043a9a --- /dev/null +++ b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md @@ -0,0 +1,4565 @@ +# Annual Prediction Tool Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a standalone tool that projects a year of household electricity costs using the real Predbat planning engine, reporting each month under three scenarios: no PV/battery, PV+battery without Predbat, and with Predbat. + +**Architecture:** Six new flat modules in `apps/predbat/` plus one extracted shared solar helper. `annual.py` drives a headless `PredBat` object per sampled day — it does not re-implement the optimiser. Weather and tariff modules perform all HTTP and never touch `PredBat`; `annual.py` performs no HTTP. Predbat plans against the archived Open-Meteo forecast and is costed against ERA5 actuals. + +**Tech Stack:** Python 3, `aiohttp` (async HTTP, already a dependency), `requests` (sync HTTP, already used by `octopus.py`), `pytz`, PyYAML, Predbat's own `Prediction` / `Plan` / `Fetch` mixins, `StorageLocalFiles` for caching. + +**Spec:** `docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md` + +## Global Constraints + +- **Line length:** 256 chars (Black), 250 chars (Flake8). +- **Docstrings:** 100% coverage required (`interrogate`) — every function *and* class needs one, including test functions. +- **Spelling:** British English (`en-gb`) via CSpell. New words go in `.cspell/custom-dictionary-workspace.txt`, which is auto-sorted on commit — re-stage after running pre-commit. `docs/superpowers/` is excluded from cspell; `docs/*.md` is **not**. +- **Variable naming:** `lower_case_with_underscores`. +- **String formatting:** this codebase uses `"...".format(...)`, not f-strings, in `apps/predbat/*.py`. Match it. +- **File header:** every new `apps/predbat/*.py` file starts with the five-line copyright block used by all existing modules. +- **Storage:** all caching goes through the Storage abstraction, never direct file access. +- **Tests:** every new module needs unit tests, registered in `TEST_REGISTRY` in `apps/predbat/unit_test.py`. Test signature is `def test_name(my_predbat):` returning a truthy value on failure. Tests must not perform network I/O. +- **Run tests from `coverage/`:** `cd coverage && source setup.csh` once, then `./run_all --test > /tmp/out.txt 2>&1` and grep the file. Never pipe test output straight to grep. +- **Pre-commit:** `./run_pre_commit` must pass before any commit is considered done. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `apps/predbat/solar_model.py` | **New.** Shared GTI→kW conversion extracted from `solcast.py`: SAPM cell temperature, azimuth conversion, trapezoidal hourly integration. | +| `apps/predbat/solcast.py` | **Modify.** `download_open_meteo_data()` delegates to `solar_model.py`. | +| `apps/predbat/annual_profiles.py` | **New.** Half-hourly domestic shape table, monthly weights, tilt constant. Data only, no logic. | +| `apps/predbat/annual_load.py` | **New.** `LoadProfileSource` interface; synthetic and Octopus-consumption implementations; cumulative `load_forecast` builder. | +| `apps/predbat/annual_weather.py` | **New.** Open-Meteo actuals + forecast archive clients, per-day PV series, monthly P10 ratios. | +| `apps/predbat/annual_tariff.py` | **New.** Per-date import/export rate resolution from an Octopus URL or basic rates. | +| `apps/predbat/annual.py` | **New.** `AnnualPredictor`: config validation, headless `PredBat` bootstrap, sample selection, three-scenario execution, aggregation. | +| `apps/predbat/annual_cli.py` | **New.** Argument parsing, progress output, JSON and table writing. | +| `apps/predbat/tests/test_annual_*.py` | **New.** One test module per new module above. | +| `docs/annual-prediction.md` | **New.** User documentation. | +| `mkdocs.yml` | **Modify.** Nav entry for the new doc page. | + +--- + +## Task 1: Extract the shared GTI→kW solar model + +`solcast.py` converts Open-Meteo GTI into PV kWh using a cell-temperature model and trapezoidal hourly integration. `annual_weather.py` needs the identical conversion. Copying it would guarantee drift, and the azimuth convention (Open-Meteo uses 0 = south, Predbat uses 180 = south) is easy to get silently wrong while still producing plausible output. + +**Files:** +- Create: `apps/predbat/solar_model.py` +- Modify: `apps/predbat/solcast.py` (lines ~20-40 for the constants, ~243-255 for `convert_azimuth`, ~356-408 for the two-pass conversion) +- Create: `apps/predbat/tests/test_solar_model.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: nothing (first task). +- Produces: + - `solar_model.pvwatts_cell_temperature(poa_global, temp_air, wind_speed) -> float` + - `solar_model.convert_azimuth(az) -> float` + - `solar_model.gti_hourly_to_period_kwh(times, gti_values, temp_values, wind_values, kwp, system_loss, shading_factors=None, p10_instant=None, p10_fallback=0.7) -> dict[datetime, dict]` where each value is `{"pv_estimate": float, "pv_estimate10": float}` keyed by a tz-aware UTC hour-start stamp, giving kWh generated during that hour for **one** array. + +- [ ] **Step 1: Capture a golden snapshot of current behaviour before touching anything** + +This is a refactor of live forecasting code. The parity fixture must be generated from the *current* implementation, before any edit, or it proves nothing. + +Create `apps/predbat/tests/golden_open_meteo.py` as a throwaway generator and run it from `coverage/`: + +```python +"""Throwaway generator for the Open-Meteo conversion golden fixture.""" +import json +import math + +_SAPM_A = -3.47 +_SAPM_B = -0.0594 +_SAPM_DELTA_T = 3.0 + +TIMES = ["2025-06-01T{:02d}:00".format(h) for h in range(24)] +GTI = [0, 0, 0, 0, 12, 88, 210, 355, 495, 610, 700, 755, 770, 742, 668, 560, 425, 275, 130, 30, 0, 0, 0, 0] +TEMP = [9, 9, 8, 8, 9, 11, 13, 15, 17, 19, 20, 21, 22, 22, 22, 21, 20, 18, 16, 14, 12, 11, 10, 10] +WIND = [1.5] * 24 + +def main(): + """Emit the golden fixture as JSON.""" + kwp = 5.6 + system_loss = 0.05 + out = [] + for idx, ts in enumerate(TIMES): + gti = GTI[idx] + t_cell = TEMP[idx] + gti * math.exp(_SAPM_A + _SAPM_B * WIND[idx]) + (gti / 1000.0) * _SAPM_DELTA_T + eta_temp = max(0.5, min(1.1, 1.0 - 0.004 * (t_cell - 25.0))) + pv50 = round((gti / 1000.0) * kwp * eta_temp * (1.0 - system_loss), 4) + out.append(pv50) + periods = [] + for i in range(len(out) - 1): + pv50 = round(0.5 * (out[i] + out[i + 1]), 4) + periods.append({"time": TIMES[i], "pv_estimate": pv50, "pv_estimate10": round(min(pv50 * 0.7, pv50), 4)}) + print(json.dumps(periods, indent=2)) + +if __name__ == "__main__": + main() +``` + +Run: `cd coverage && python3 ../apps/predbat/tests/golden_open_meteo.py > tests_golden.json` + +Copy the output to `apps/predbat/tests/fixtures/open_meteo_golden.json` (create the `fixtures` directory), then delete `golden_open_meteo.py` and `coverage/tests_golden.json`. + +This mirrors the exact arithmetic in `solcast.py:356-396` — instantaneous kW per sample, then trapezoidal integration across each hour pair, then the `pv50 * 0.7` P10 fallback used when no ensemble data exists. + +- [ ] **Step 2: Write the failing test** + +Create `apps/predbat/tests/test_solar_model.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the shared solar GTI to kW conversion model.""" + +import json +import os +from datetime import datetime + +import pytz + +from solar_model import convert_azimuth, gti_hourly_to_period_kwh, pvwatts_cell_temperature + +FIXTURE_DIR = os.path.join(os.path.dirname(__file__), "fixtures") + +TIMES = ["2025-06-01T{:02d}:00".format(h) for h in range(24)] +GTI = [0, 0, 0, 0, 12, 88, 210, 355, 495, 610, 700, 755, 770, 742, 668, 560, 425, 275, 130, 30, 0, 0, 0, 0] +TEMP = [9, 9, 8, 8, 9, 11, 13, 15, 17, 19, 20, 21, 22, 22, 22, 21, 20, 18, 16, 14, 12, 11, 10, 10] +WIND = [1.5] * 24 + + +def test_solar_model(my_predbat): + """Verify the extracted solar model matches the golden snapshot of the original solcast.py logic.""" + failed = False + print("**** Testing solar_model ****") + + print("Test: convert_azimuth round trips the Predbat/Open-Meteo conventions") + for predbat_az, expected in [(180, 0), (90, 90), (270, -90), (0, 180)]: + result = convert_azimuth(predbat_az) + if result != expected: + print(" ERROR: convert_azimuth({}) expected {}, got {}".format(predbat_az, expected, result)) + failed = True + + print("Test: pvwatts_cell_temperature raises cell temperature above ambient under irradiance") + cool = pvwatts_cell_temperature(0.0, 20.0, 1.5) + hot = pvwatts_cell_temperature(800.0, 20.0, 1.5) + if cool != 20.0: + print(" ERROR: zero irradiance should give ambient, got {}".format(cool)) + failed = True + if hot <= cool: + print(" ERROR: 800 W/m2 should raise cell temperature above ambient, got {}".format(hot)) + failed = True + + print("Test: gti_hourly_to_period_kwh matches the golden snapshot") + with open(os.path.join(FIXTURE_DIR, "open_meteo_golden.json"), "r") as handle: + golden = json.load(handle) + + result = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05) + + if len(result) != len(golden): + print(" ERROR: expected {} periods, got {}".format(len(golden), len(result))) + failed = True + + for entry in golden: + stamp = pytz.utc.localize(datetime.strptime(entry["time"], "%Y-%m-%dT%H:%M")) + got = result.get(stamp) + if got is None: + print(" ERROR: missing period {}".format(entry["time"])) + failed = True + continue + if abs(got["pv_estimate"] - entry["pv_estimate"]) > 0.0001: + print(" ERROR: {} pv_estimate expected {}, got {}".format(entry["time"], entry["pv_estimate"], got["pv_estimate"])) + failed = True + if abs(got["pv_estimate10"] - entry["pv_estimate10"]) > 0.0001: + print(" ERROR: {} pv_estimate10 expected {}, got {}".format(entry["time"], entry["pv_estimate10"], got["pv_estimate10"])) + failed = True + + print("Test: p10_fallback scales the P10 series") + scaled = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05, p10_fallback=0.5) + stamp = pytz.utc.localize(datetime.strptime("2025-06-01T12:00", "%Y-%m-%dT%H:%M")) + expected_p10 = round(scaled[stamp]["pv_estimate"] * 0.5, 4) + if abs(scaled[stamp]["pv_estimate10"] - expected_p10) > 0.0001: + print(" ERROR: p10_fallback 0.5 expected {}, got {}".format(expected_p10, scaled[stamp]["pv_estimate10"])) + failed = True + + print("Test: shading_factors apply the correct month") + shading = [0.5] * 12 + shaded = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05, shading_factors=shading) + unshaded = result[stamp]["pv_estimate"] + if abs(shaded[stamp]["pv_estimate"] - round(unshaded * 0.5, 4)) > 0.0001: + print(" ERROR: shading 0.5 expected {}, got {}".format(round(unshaded * 0.5, 4), shaded[stamp]["pv_estimate"])) + failed = True + + return failed +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test solar_model > /tmp/t1.txt 2>&1; grep -E "ERROR|FAILED|ModuleNotFound" /tmp/t1.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'solar_model'`. The test is not yet registered, so first add it (Step 5) if `--test solar_model` reports an unknown test name. + +- [ ] **Step 4: Create `apps/predbat/solar_model.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Shared photovoltaic conversion model. + +Converts Open-Meteo global tilted irradiance (GTI) into PV energy, applying a +SAPM/PVWatts cell-temperature derate and integrating each hourly sample pair. +Shared by the live Solcast/Open-Meteo forecast path and the annual prediction +tool so the two cannot drift apart. +""" + +import math +from datetime import datetime, timedelta + +import pytz + +from utils import dp4 + +# PVWatts / SAPM cell temperature model constants (glass/glass, open rack) +# Equivalent to pvlib.temperature.sapm_cell with open_rack_glass_glass parameters +_SAPM_A = -3.47 +_SAPM_B = -0.0594 +_SAPM_DELTA_T = 3.0 + +# c-Si temperature coefficient: -0.4%/degC relative to STC (25degC) +_TEMP_COEFF = 0.004 +_STC_TEMP_C = 25.0 + +# Defaults used when a sample has no measured value +_DEFAULT_TEMP_C = 25.0 +_DEFAULT_WIND_MS = 1.0 + +# Applied when no ensemble P10 data is available +_DEFAULT_P10_FALLBACK = 0.7 + + +def pvwatts_cell_temperature(poa_global, temp_air, wind_speed): + """Compute PV cell temperature using the SAPM (PVWatts) model. + + Parameters correspond to a glass/glass module on an open rack (the most + common residential case). Formula: T_cell = T_air + GTI*exp(a + b*wind) + (GTI/1000)*deltaT + """ + return temp_air + poa_global * math.exp(_SAPM_A + _SAPM_B * wind_speed) + (poa_global / 1000.0) * _SAPM_DELTA_T + + +def convert_azimuth(az): + """ + Convert azimuth from Predbat/Solcast convention to Forecast.solar/Open-Meteo convention. + Predbat/Solcast convention: 0 = North, -90 = East, 90 = West, 180 = South + Forecast.solar/Open-Meteo convention: 0 = South, -90 = East, 90 = West, +/-180 = North + """ + if az >= 0: + az = 180 - az + else: + az = -180 - az + + return az + + +def _temperature_efficiency(gti, temp, wind): + """Return the cell-temperature efficiency multiplier for one irradiance sample.""" + t_cell = pvwatts_cell_temperature(gti, temp, wind) + # No lower clamp on (t_cell - 25): cool cells genuinely produce more power. + # Cap at 1.1 (10% above STC) to prevent unrealistic gains at very cold temperatures. + return max(0.5, min(1.1, 1.0 - _TEMP_COEFF * (t_cell - _STC_TEMP_C))) + + +def gti_hourly_to_period_kwh(times, gti_values, temp_values, wind_values, kwp, system_loss, shading_factors=None, p10_instant=None, p10_fallback=_DEFAULT_P10_FALLBACK): + """Convert hourly GTI samples into per-hour PV energy for a single array. + + Open-Meteo returns point-in-time irradiance (W/m2) at the start of each hour, so the + samples are integrated trapezoidally across each adjacent pair rather than treated as + period energy. + + Args: + times: list of ISO timestamp strings, "%Y-%m-%dT%H:%M", assumed UTC + gti_values: list of global tilted irradiance values in W/m2, aligned to times + temp_values: list of air temperatures in degC, aligned to times + wind_values: list of wind speeds in m/s, aligned to times + kwp: array peak power in kW + system_loss: fractional system loss, e.g. 0.05 for 95% efficiency + shading_factors: optional list of 12 per-month multipliers + p10_instant: optional dict of timestamp string to raw P10 kW, before temperature derate + p10_fallback: multiplier applied to P50 when p10_instant has no entry + + Returns: + dict of tz-aware UTC hour-start datetime to {"pv_estimate": kWh, "pv_estimate10": kWh} + """ + instant_kw = {} + instant_stamps = [] + + for idx, ts in enumerate(times): + if idx >= len(gti_values): + break + gti = gti_values[idx] + if gti is None: + gti = 0.0 + temp = temp_values[idx] if idx < len(temp_values) and temp_values[idx] is not None else _DEFAULT_TEMP_C + wind = wind_values[idx] if idx < len(wind_values) and wind_values[idx] is not None else _DEFAULT_WIND_MS + eta_temp = _temperature_efficiency(gti, temp, wind) + pv50_inst = dp4((gti / 1000.0) * kwp * eta_temp * (1.0 - system_loss)) + raw_p10 = p10_instant.get(ts) if p10_instant else None + # p10_instant was computed without temperature derating; apply eta_temp now + pv10_inst = dp4(min(raw_p10 * eta_temp, pv50_inst) if raw_p10 is not None else pv50_inst * p10_fallback) + try: + stamp = datetime.strptime(ts, "%Y-%m-%dT%H:%M") + stamp = stamp.replace(tzinfo=pytz.utc) + except (ValueError, TypeError): + continue + instant_kw[stamp] = (pv50_inst, pv10_inst) + instant_stamps.append(stamp) + + period_data = {} + for i in range(len(instant_stamps) - 1): + stamp = instant_stamps[i] + next_stamp = instant_stamps[i + 1] + if (next_stamp - stamp) != timedelta(hours=1): + continue + pv50_start, pv10_start = instant_kw[stamp] + pv50_end, pv10_end = instant_kw[next_stamp] + pv50 = dp4(0.5 * (pv50_start + pv50_end)) + pv10 = dp4(0.5 * (pv10_start + pv10_end)) + + if shading_factors and len(shading_factors) == 12: + shading_month = shading_factors[stamp.month - 1] + pv50 = dp4(pv50 * shading_month) + pv10 = dp4(pv10 * shading_month) + + period_data[stamp] = {"pv_estimate": pv50, "pv_estimate10": pv10} + + return period_data +``` + +- [ ] **Step 5: Register the test** + +In `apps/predbat/unit_test.py`, add the import alongside the other `from tests.test_* import ...` lines: + +```python +from tests.test_solar_model import test_solar_model +``` + +and add to `TEST_REGISTRY` (near the other model entries): + +```python + ("solar_model", test_solar_model, "Shared solar GTI conversion model tests", False), +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test solar_model > /tmp/t1.txt 2>&1; grep -E "ERROR|FAILED|PASSED|Traceback" /tmp/t1.txt` + +Expected: no `ERROR` lines, test reports success. + +- [ ] **Step 7: Make `solcast.py` delegate to the new module** + +In `apps/predbat/solcast.py`: + +1. Delete the `_SAPM_A` / `_SAPM_B` / `_SAPM_DELTA_T` constants and the `pvwatts_cell_temperature` function (lines ~26-39). +2. Add to the import block: `from solar_model import convert_azimuth, gti_hourly_to_period_kwh`. +3. Delete the `convert_azimuth` method (lines ~243-255). Its call site becomes the module function: `az = convert_azimuth(az)`. +4. Replace the two-pass conversion block in `download_open_meteo_data()` (from `# Pass 1: compute instantaneous kW` through the `period_data[stamp] = data_item` else-branch) with: + +```python + array_periods = gti_hourly_to_period_kwh( + times, + gti_values, + temp_values, + wind_values, + kwp=kwp, + system_loss=system_loss, + shading_factors=shading_factors, + p10_instant=ensemble_p10, + ) + for stamp, values in array_periods.items(): + pv50 = values["pv_estimate"] + pv10 = values["pv_estimate10"] + if stamp in period_data: + period_data[stamp]["pv_estimate"] = dp4(period_data[stamp]["pv_estimate"] + pv50) + period_data[stamp]["pv_estimate10"] = dp4(period_data[stamp]["pv_estimate10"] + pv10) + else: + period_data[stamp] = {"period_start": stamp.strftime(TIME_FORMAT), "pv_estimate": pv50, "pv_estimate10": pv10} +``` + +Note `math` may now be unused in `solcast.py` — check before removing the import, as `download_open_meteo_ensemble_data()` still uses `math.ceil`. + +- [ ] **Step 8: Run the full existing solar test suites to confirm no regression** + +Run: `cd coverage && ./run_all -k solcast > /tmp/t1b.txt 2>&1; ./run_all -k pv_forecast >> /tmp/t1b.txt 2>&1; ./run_all --test open_meteo >> /tmp/t1b.txt 2>&1; grep -E "ERROR|FAILED|Traceback" /tmp/t1b.txt` + +Expected: no output from the grep. If `--test open_meteo` reports an unknown name, run `./run_all --list` and use the actual registered name for `tests/test_open_meteo.py`. + +- [ ] **Step 9: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/solar_model.py apps/predbat/solcast.py apps/predbat/unit_test.py apps/predbat/tests/test_solar_model.py apps/predbat/tests/fixtures/open_meteo_golden.json +git commit -m "refactor(solar): extract the shared GTI to kW conversion model" +``` + +--- + +## Task 2: Load profile data tables + +Pure data, no logic, so it can be revised against real measurements without touching behaviour. + +**Files:** +- Create: `apps/predbat/annual_profiles.py` +- Create: `apps/predbat/tests/test_annual_profiles.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `annual_profiles.HOURLY_SHAPE` — list of 24 unnormalised relative weights + - `annual_profiles.MONTH_WEIGHTS` — list of 12 relative daily-consumption multipliers, January first + - `annual_profiles.SHAPE_TILT_FRACTION` — float, proportion of the *source band's* energy moved to the destination band + - `annual_profiles.NIGHT_BAND_SLOTS` — list of half-hour indices for 00:00-07:00 + - `annual_profiles.DAY_BAND_SLOTS` — list of half-hour indices for 07:00-20:00 + - `annual_profiles.half_hour_shape() -> list[float]` — 48 values summing to exactly 1.0 + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_profiles.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction load profile data tables.""" + +from annual_profiles import DAY_BAND_SLOTS, HOURLY_SHAPE, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape + + +def test_annual_profiles(my_predbat): + """Verify the profile tables are well formed and normalise correctly.""" + failed = False + print("**** Testing annual_profiles ****") + + print("Test: HOURLY_SHAPE has 24 positive entries") + if len(HOURLY_SHAPE) != 24: + print(" ERROR: expected 24 hourly weights, got {}".format(len(HOURLY_SHAPE))) + failed = True + if any(value <= 0 for value in HOURLY_SHAPE): + print(" ERROR: all hourly weights must be positive") + failed = True + + print("Test: half_hour_shape returns 48 values summing to 1.0") + shape = half_hour_shape() + if len(shape) != 48: + print(" ERROR: expected 48 half-hourly values, got {}".format(len(shape))) + failed = True + total = sum(shape) + if abs(total - 1.0) > 1e-9: + print(" ERROR: half_hour_shape must sum to 1.0, got {}".format(total)) + failed = True + + print("Test: the evening peak exceeds the overnight trough") + evening = sum(shape[36:42]) + overnight = sum(shape[4:10]) + if evening <= overnight: + print(" ERROR: evening 18:00-21:00 share {} should exceed overnight 02:00-05:00 share {}".format(evening, overnight)) + failed = True + + print("Test: MONTH_WEIGHTS has 12 positive entries with winter above summer") + if len(MONTH_WEIGHTS) != 12: + print(" ERROR: expected 12 month weights, got {}".format(len(MONTH_WEIGHTS))) + failed = True + if any(value <= 0 for value in MONTH_WEIGHTS): + print(" ERROR: all month weights must be positive") + failed = True + if MONTH_WEIGHTS[0] <= MONTH_WEIGHTS[6]: + print(" ERROR: January weight {} should exceed July weight {}".format(MONTH_WEIGHTS[0], MONTH_WEIGHTS[6])) + failed = True + + print("Test: the night and day bands are disjoint and correctly sized") + if NIGHT_BAND_SLOTS != list(range(0, 14)): + print(" ERROR: NIGHT_BAND_SLOTS should cover 00:00-07:00, got {}".format(NIGHT_BAND_SLOTS)) + failed = True + if DAY_BAND_SLOTS != list(range(14, 40)): + print(" ERROR: DAY_BAND_SLOTS should cover 07:00-20:00, got {}".format(DAY_BAND_SLOTS)) + failed = True + if set(NIGHT_BAND_SLOTS) & set(DAY_BAND_SLOTS): + print(" ERROR: night and day bands must be disjoint") + failed = True + + print("Test: SHAPE_TILT_FRACTION is a sane proportion") + if not 0.0 < SHAPE_TILT_FRACTION < 0.5: + print(" ERROR: SHAPE_TILT_FRACTION should be between 0 and 0.5, got {}".format(SHAPE_TILT_FRACTION)) + failed = True + + return failed +``` + +- [ ] **Step 2: Run the test to verify it fails** + +First register it — add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_profiles import test_annual_profiles +``` + +```python + ("annual_profiles", test_annual_profiles, "Annual prediction load profile table tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_profiles > /tmp/t2.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t2.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_profiles'`. + +- [ ] **Step 3: Create `apps/predbat/annual_profiles.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Domestic load profile data tables for the annual prediction tool. + +Data only, no behaviour, so the shapes can be recalibrated against real +consumption data without touching the code that consumes them. +""" + +# Relative electricity consumption by hour of day for a typical UK domestic +# property, index 0 = 00:00. Unnormalised; half_hour_shape() normalises to 1.0. +# Shape: overnight trough, a modest morning peak, a midday plateau, and a +# pronounced evening peak from about 17:00 to 21:00. +HOURLY_SHAPE = [ + 2.6, # 00:00 + 2.4, # 01:00 + 2.3, # 02:00 + 2.2, # 03:00 + 2.2, # 04:00 + 2.4, # 05:00 + 3.0, # 06:00 + 3.8, # 07:00 + 4.2, # 08:00 + 4.1, # 09:00 + 3.9, # 10:00 + 3.8, # 11:00 + 3.9, # 12:00 + 3.8, # 13:00 + 3.7, # 14:00 + 3.9, # 15:00 + 4.6, # 16:00 + 5.8, # 17:00 + 6.5, # 18:00 + 6.3, # 19:00 + 5.6, # 20:00 + 5.0, # 21:00 + 4.3, # 22:00 + 3.4, # 23:00 +] + +# Relative daily consumption by month, index 0 = January. Captures the UK +# winter/summer split, which drives much of the annual answer. These are daily +# rates, so consumers must normalise by days-in-month to preserve the annual total. +MONTH_WEIGHTS = [ + 1.20, # January + 1.15, # February + 1.05, # March + 0.95, # April + 0.88, # May + 0.83, # June + 0.82, # July + 0.83, # August + 0.90, # September + 1.00, # October + 1.12, # November + 1.22, # December +] + +# Proportion of the SOURCE band's own energy moved to the destination band when +# the user selects a "night" or "day" biased profile. Expressed relative to the +# source band rather than to the whole day so the transfer can never exceed the +# energy available to move. Tunable against real data. +SHAPE_TILT_FRACTION = 0.30 + +# Half-hour slot indices, 0 = 00:00-00:30, 47 = 23:30-00:00. +NIGHT_BAND_SLOTS = list(range(0, 14)) # 00:00 - 07:00 +DAY_BAND_SLOTS = list(range(14, 40)) # 07:00 - 20:00 + + +def half_hour_shape(): + """Return the 48-slot half-hourly domestic shape, normalised to sum to exactly 1.0. + + Each hourly weight is split evenly across its two half-hour slots. The final + slot absorbs any floating-point residue so the total is exactly 1.0. + """ + total = float(sum(HOURLY_SHAPE)) + shape = [] + for weight in HOURLY_SHAPE: + half = (weight / total) / 2.0 + shape.append(half) + shape.append(half) + residue = 1.0 - sum(shape) + shape[-1] += residue + return shape +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_profiles > /tmp/t2.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t2.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_profiles.py apps/predbat/tests/test_annual_profiles.py apps/predbat/unit_test.py +git commit -m "feat(annual): add domestic load profile data tables" +``` + +--- + +## Task 3: Synthetic load profile source + +Turns an annual kWh figure plus a night/day/flat preference into the forward cumulative series Predbat consumes. The key insight from the spec: setting `load_forecast_only = True` makes `step_data_history()` ignore historical load entirely (`apps/predbat/fetch.py`, the `if type_load and not forward:` branch) and build the forward profile purely from `load_forecast`, read via `get_from_incrementing(load_forecast, minute, backwards=False)` which returns `data[m + 1] - data[m]`. So `load_forecast` must be a **cumulative** kWh series keyed by absolute minute. + +**Files:** +- Create: `apps/predbat/annual_load.py` +- Create: `apps/predbat/tests/test_annual_load.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual_profiles.half_hour_shape()`, `MONTH_WEIGHTS`, `SHAPE_TILT_FRACTION`, `NIGHT_BAND_SLOTS`, `DAY_BAND_SLOTS`. +- Produces: + - `annual_load.LoadProfileSource` — base class with `minute_profile(day)` and `daily_kwh(day)` + - `annual_load.SyntheticLoadProfile(annual_kwh, shape, year)` where `shape` is one of `"night"`, `"day"`, `"flat"` + - `annual_load.tilt_shape(shape_values, direction) -> list[float]` + - `annual_load.build_load_forecast(source, start_day, days) -> dict[int, float]` — cumulative kWh keyed by absolute minute, entries `0 .. days * 1440` inclusive + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_load.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction load profile sources.""" + +import calendar +from datetime import date + +from annual_load import SyntheticLoadProfile, build_load_forecast, tilt_shape +from annual_profiles import DAY_BAND_SLOTS, NIGHT_BAND_SLOTS, half_hour_shape + + +def test_annual_load(my_predbat): + """Verify the synthetic load profile preserves totals and tilts correctly.""" + failed = False + print("**** Testing annual_load ****") + + print("Test: tilt_shape preserves the daily total exactly") + base = half_hour_shape() + for direction in ["night", "day", "flat"]: + tilted = tilt_shape(base, direction) + total = sum(tilted) + if abs(total - 1.0) > 1e-9: + print(" ERROR: tilt '{}' changed the total to {}".format(direction, total)) + failed = True + if any(value < 0 for value in tilted): + print(" ERROR: tilt '{}' produced a negative slot".format(direction)) + failed = True + + print("Test: tilt 'night' moves energy into the night band") + night_tilted = tilt_shape(base, "night") + base_night = sum(base[slot] for slot in NIGHT_BAND_SLOTS) + tilted_night = sum(night_tilted[slot] for slot in NIGHT_BAND_SLOTS) + if tilted_night <= base_night: + print(" ERROR: night tilt should raise the night band from {} to more, got {}".format(base_night, tilted_night)) + failed = True + + print("Test: tilt 'day' moves energy into the day band") + day_tilted = tilt_shape(base, "day") + base_day = sum(base[slot] for slot in DAY_BAND_SLOTS) + tilted_day = sum(day_tilted[slot] for slot in DAY_BAND_SLOTS) + if tilted_day <= base_day: + print(" ERROR: day tilt should raise the day band from {} to more, got {}".format(base_day, tilted_day)) + failed = True + + print("Test: tilt 'flat' is a no-op") + flat_tilted = tilt_shape(base, "flat") + if flat_tilted != base: + print(" ERROR: flat tilt should leave the shape unchanged") + failed = True + + print("Test: the twelve monthly totals sum to annual_kwh") + annual_kwh = 3800.0 + source = SyntheticLoadProfile(annual_kwh=annual_kwh, shape="flat", year=2025) + year_total = 0.0 + for month in range(1, 13): + days_in_month = calendar.monthrange(2025, month)[1] + month_total = sum(source.daily_kwh(date(2025, month, day)) for day in range(1, days_in_month + 1)) + year_total += month_total + if abs(year_total - annual_kwh) > 1e-6: + print(" ERROR: twelve months summed to {}, expected {}".format(year_total, annual_kwh)) + failed = True + + print("Test: January daily consumption exceeds July") + january = source.daily_kwh(date(2025, 1, 15)) + july = source.daily_kwh(date(2025, 7, 15)) + if january <= july: + print(" ERROR: January daily {} should exceed July daily {}".format(january, july)) + failed = True + + print("Test: minute_profile has 1440 entries summing to the day's kWh") + day = date(2025, 3, 10) + profile = source.minute_profile(day) + if len(profile) != 1440: + print(" ERROR: expected 1440 minutes, got {}".format(len(profile))) + failed = True + if abs(sum(profile) - source.daily_kwh(day)) > 1e-9: + print(" ERROR: minute profile sums to {}, expected {}".format(sum(profile), source.daily_kwh(day))) + failed = True + + print("Test: build_load_forecast produces a cumulative series Predbat can difference") + forecast = build_load_forecast(source, date(2025, 3, 10), 2) + if forecast.get(0) != 0.0: + print(" ERROR: cumulative series must start at 0, got {}".format(forecast.get(0))) + failed = True + if 2 * 1440 not in forecast: + print(" ERROR: cumulative series must include the final boundary minute {}".format(2 * 1440)) + failed = True + for minute in range(1, 2 * 1440 + 1): + if forecast[minute] < forecast[minute - 1] - 1e-12: + print(" ERROR: cumulative series decreased at minute {}".format(minute)) + failed = True + break + expected_two_days = source.daily_kwh(date(2025, 3, 10)) + source.daily_kwh(date(2025, 3, 11)) + if abs(forecast[2 * 1440] - expected_two_days) > 1e-9: + print(" ERROR: two-day total {} expected {}".format(forecast[2 * 1440], expected_two_days)) + failed = True + + print("Test: differencing the cumulative series recovers the per-minute profile") + first_minute = forecast[1] - forecast[0] + if abs(first_minute - profile[0]) > 1e-12: + print(" ERROR: differenced minute 0 gave {}, expected {}".format(first_minute, profile[0])) + failed = True + + print("Test: a zero annual figure produces a zero profile rather than dividing by zero") + zero_source = SyntheticLoadProfile(annual_kwh=0.0, shape="flat", year=2025) + if sum(zero_source.minute_profile(day)) != 0.0: + print(" ERROR: zero annual kWh should give a zero profile") + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_load import test_annual_load +``` + +```python + ("annual_load", test_annual_load, "Annual prediction load profile tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_load > /tmp/t3.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t3.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_load'`. + +- [ ] **Step 3: Create `apps/predbat/annual_load.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Load profile sources for the annual prediction tool. + +Produces the forward cumulative kWh series that Predbat consumes as +``load_forecast`` when ``load_forecast_only`` is set, so no synthetic backwards +history has to be fabricated. +""" + +import calendar +from datetime import timedelta + +from annual_profiles import DAY_BAND_SLOTS, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape + +MINUTES_PER_DAY = 24 * 60 +MINUTES_PER_SLOT = 30 + + +def tilt_shape(shape_values, direction): + """Move energy between the night and day bands, preserving the total exactly. + + ``direction`` is one of "night" (move day energy into the night band), "day" + (the reverse), or "flat" (no change). The amount moved is + ``SHAPE_TILT_FRACTION`` of the source band's own energy, so the transfer can + never exceed what is available. Energy is taken from and added to individual + slots in proportion to their existing share of their band, which keeps the + within-band shape intact. + """ + if direction == "flat": + return list(shape_values) + + if direction == "night": + source_slots, dest_slots = DAY_BAND_SLOTS, NIGHT_BAND_SLOTS + elif direction == "day": + source_slots, dest_slots = NIGHT_BAND_SLOTS, DAY_BAND_SLOTS + else: + raise ValueError("Unknown load shape '{}', expected night, day or flat".format(direction)) + + tilted = list(shape_values) + source_total = sum(tilted[slot] for slot in source_slots) + dest_total = sum(tilted[slot] for slot in dest_slots) + if source_total <= 0 or dest_total <= 0: + return tilted + + moved = source_total * SHAPE_TILT_FRACTION + for slot in source_slots: + tilted[slot] -= moved * (tilted[slot] / source_total) + for slot in dest_slots: + tilted[slot] += moved * (shape_values[slot] / dest_total) + + # Push any floating-point residue into the largest slot so the total stays exact + residue = sum(shape_values) - sum(tilted) + largest = max(range(len(tilted)), key=lambda index: tilted[index]) + tilted[largest] += residue + return tilted + + +class LoadProfileSource: + """Base class for a source of daily household load profiles.""" + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + raise NotImplementedError + + def minute_profile(self, day): + """Return a list of 1440 per-minute kWh values for the given date, or None if unavailable.""" + raise NotImplementedError + + +class SyntheticLoadProfile(LoadProfileSource): + """Load profile synthesised from an annual kWh total and a shape preference. + + Monthly weights are normalised across the specific year's day counts so the + twelve monthly totals sum to exactly ``annual_kwh``. + """ + + def __init__(self, annual_kwh, shape, year): + """Build the synthetic profile for one calendar year.""" + self.annual_kwh = float(annual_kwh) + self.shape = shape + self.year = year + self.slot_shape = tilt_shape(half_hour_shape(), shape) + + weighted_days = 0.0 + for month in range(1, 13): + days_in_month = calendar.monthrange(year, month)[1] + weighted_days += MONTH_WEIGHTS[month - 1] * days_in_month + self.base_daily_kwh = (self.annual_kwh / weighted_days) if weighted_days > 0 else 0.0 + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + return self.base_daily_kwh * MONTH_WEIGHTS[day.month - 1] + + def minute_profile(self, day): + """Return a list of 1440 per-minute kWh values for the given date.""" + total = self.daily_kwh(day) + profile = [] + for slot_value in self.slot_shape: + per_minute = (total * slot_value) / MINUTES_PER_SLOT + profile.extend([per_minute] * MINUTES_PER_SLOT) + return profile + + +def build_load_forecast(source, start_day, days): + """Build the cumulative kWh series Predbat reads as ``load_forecast``. + + Keys are absolute minutes from midnight on ``start_day``. Predbat differences + consecutive entries via ``get_from_incrementing(..., backwards=False)``, so + the series must be cumulative and must include the final boundary minute + ``days * 1440`` for the last minute to be readable. + + Days for which the source has no data contribute zero and are skipped by the + caller, which is responsible for logging the gap. + """ + forecast = {0: 0.0} + running = 0.0 + for day_offset in range(days): + day = start_day + timedelta(days=day_offset) + profile = source.minute_profile(day) + if profile is None: + profile = [0.0] * MINUTES_PER_DAY + base = day_offset * MINUTES_PER_DAY + for index, value in enumerate(profile): + running += value + forecast[base + index + 1] = running + return forecast +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_load > /tmp/t3.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t3.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_load.py apps/predbat/tests/test_annual_load.py apps/predbat/unit_test.py +git commit -m "feat(annual): add synthetic load profile source" +``` + +--- + +## Task 4: Octopus consumption load profile source + +The real-data alternative to the synthetic profile. Mutually exclusive with it — that validation lives in Task 7, not here. + +**Files:** +- Modify: `apps/predbat/annual_load.py` +- Modify: `apps/predbat/tests/test_annual_load.py` + +**Interfaces:** +- Consumes: `annual_load.LoadProfileSource` from Task 3. +- Produces: + - `annual_load.OctopusConsumptionLoadProfile(api_key, account_id, log, storage=None, fallback=None)` + - `OctopusConsumptionLoadProfile.fetch(year)` — async, populates the internal per-day cache + - `OctopusConsumptionLoadProfile.missing_days` — set of `date` objects with no usable data + - `annual_load.parse_consumption_results(results) -> dict[date, list[float]]` — pure, testable without network; maps each date to 48 half-hourly kWh values + +- [ ] **Step 1: Write the failing test** + +Append to `apps/predbat/tests/test_annual_load.py`, and add `parse_consumption_results, OctopusConsumptionLoadProfile` to the `annual_load` import line: + +```python +def test_annual_load_octopus(my_predbat): + """Verify Octopus consumption parsing and its fallback behaviour.""" + failed = False + print("**** Testing annual_load Octopus source ****") + + print("Test: parse_consumption_results maps half-hourly readings onto dates") + results = [] + for slot in range(48): + hour = slot // 2 + minute = 30 * (slot % 2) + results.append( + { + "consumption": 0.25, + "interval_start": "2025-03-10T{:02d}:{:02d}:00Z".format(hour, minute), + "interval_end": "2025-03-10T{:02d}:{:02d}:00Z".format(hour, minute), + } + ) + parsed = parse_consumption_results(results) + target = date(2025, 3, 10) + if target not in parsed: + print(" ERROR: expected {} in parsed output, got {}".format(target, list(parsed.keys()))) + failed = True + elif len(parsed[target]) != 48: + print(" ERROR: expected 48 slots, got {}".format(len(parsed[target]))) + failed = True + elif abs(sum(parsed[target]) - 12.0) > 1e-9: + print(" ERROR: expected 12.0 kWh for the day, got {}".format(sum(parsed[target]))) + failed = True + + print("Test: a partial day is reported as missing rather than silently understated") + partial = parse_consumption_results(results[:20]) + if date(2025, 3, 10) in partial: + print(" ERROR: a day with only 20 of 48 slots must not be returned as complete") + failed = True + + print("Test: minute_profile falls back to the synthetic source for a missing day") + fallback = SyntheticLoadProfile(annual_kwh=3800.0, shape="flat", year=2025) + source = OctopusConsumptionLoadProfile(api_key="x", account_id="A-1", log=print, fallback=fallback) + source.consumption = parse_consumption_results(results) + + present = source.minute_profile(date(2025, 3, 10)) + if abs(sum(present) - 12.0) > 1e-9: + print(" ERROR: present day should use real data summing to 12.0, got {}".format(sum(present))) + failed = True + + absent = source.minute_profile(date(2025, 3, 11)) + if abs(sum(absent) - fallback.daily_kwh(date(2025, 3, 11))) > 1e-9: + print(" ERROR: missing day should fall back to synthetic, got {}".format(sum(absent))) + failed = True + if date(2025, 3, 11) not in source.missing_days: + print(" ERROR: a fallback day must be recorded in missing_days") + failed = True + + print("Test: no fallback and no data yields None so the caller can exclude the day") + bare = OctopusConsumptionLoadProfile(api_key="x", account_id="A-1", log=print) + if bare.minute_profile(date(2025, 3, 11)) is not None: + print(" ERROR: with no data and no fallback minute_profile must return None") + failed = True + + return failed +``` + +Register it in `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_load import test_annual_load, test_annual_load_octopus +``` + +```python + ("annual_load_octopus", test_annual_load_octopus, "Annual prediction Octopus consumption tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test annual_load_octopus > /tmp/t4.txt 2>&1; grep -E "ERROR|ImportError|cannot import" /tmp/t4.txt` + +Expected: FAIL with `cannot import name 'parse_consumption_results'`. + +- [ ] **Step 3: Add the Octopus source to `apps/predbat/annual_load.py`** + +Add these imports at the top of the file: + +```python +import base64 +from datetime import date, datetime, timedelta +``` + +(replacing the existing `from datetime import timedelta` line), plus: + +```python +import aiohttp +``` + +Then append: + +```python +OCTOPUS_API_BASE = "https://api.octopus.energy/v1" +SLOTS_PER_DAY = 48 + + +def parse_consumption_results(results): + """Turn raw Octopus consumption rows into complete per-day half-hourly kWh lists. + + Only days with all 48 slots present are returned. A partially reported day is + omitted entirely rather than returned short, because a half-populated day + looks like genuinely low consumption and would silently understate the bill. + """ + by_day = {} + for row in results or []: + start = row.get("interval_start") + consumption = row.get("consumption") + if start is None or consumption is None: + continue + try: + stamp = datetime.strptime(start[:16], "%Y-%m-%dT%H:%M") + except (ValueError, TypeError): + continue + slot = stamp.hour * 2 + (1 if stamp.minute >= 30 else 0) + day = stamp.date() + if day not in by_day: + by_day[day] = [None] * SLOTS_PER_DAY + by_day[day][slot] = float(consumption) + + complete = {} + for day, slots in by_day.items(): + if all(value is not None for value in slots): + complete[day] = slots + return complete + + +class OctopusConsumptionLoadProfile(LoadProfileSource): + """Load profile taken from the account's real half-hourly Octopus consumption. + + The meter series already includes any EV charging, which is why the config + layer rejects an Octopus key alongside a separate car charging figure. + """ + + def __init__(self, api_key, account_id, log, storage=None, fallback=None): + """Set up the Octopus consumption source, optionally backed by a fallback profile.""" + self.api_key = api_key + self.account_id = account_id + self.log = log + self.storage = storage + self.fallback = fallback + self.consumption = {} + self.missing_days = set() + self.mpan = None + self.serial = None + + def _auth_header(self): + """Return the HTTP Basic auth header Octopus expects, API key as username.""" + token = base64.b64encode("{}:".format(self.api_key).encode("utf-8")).decode("utf-8") + return {"Authorization": "Basic {}".format(token), "accept": "application/json", "user-agent": "predbat/1.0"} + + async def _get_json(self, session, url): + """Fetch and decode one JSON page, returning None on any failure.""" + try: + async with session.get(url, headers=self._auth_header(), timeout=aiohttp.ClientTimeout(total=30)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Octopus consumption request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Octopus consumption request to {} failed: {}".format(url, error)) + return None + + async def resolve_meter(self, session): + """Resolve the account's MPAN and meter serial. Returns True on success.""" + data = await self._get_json(session, "{}/accounts/{}/".format(OCTOPUS_API_BASE, self.account_id)) + if not data: + return False + for prop in data.get("properties", []) or []: + for point in prop.get("electricity_meter_points", []) or []: + if point.get("is_export"): + continue + meters = point.get("meters", []) or [] + if point.get("mpan") and meters: + self.mpan = point["mpan"] + self.serial = meters[-1].get("serial_number") + if self.serial: + self.log("Annual: Octopus resolved MPAN {} meter {}".format(self.mpan, self.serial)) + return True + self.log("Warn: Annual: Octopus account {} has no usable electricity import meter".format(self.account_id)) + return False + + async def fetch(self, year): + """Download a calendar year of half-hourly consumption. Returns True on success.""" + cache_key = "consumption_{}_{}".format(self.account_id, year) + if self.storage: + cached = await self.storage.load("annual", cache_key) + if isinstance(cached, dict) and cached: + self.consumption = {date.fromisoformat(key): value for key, value in cached.items()} + self.log("Annual: Octopus consumption for {} loaded from cache, {} days".format(year, len(self.consumption))) + return True + + async with aiohttp.ClientSession() as session: + if not await self.resolve_meter(session): + return False + + url = "{}/electricity-meter-points/{}/meters/{}/consumption/?period_from={}-01-01T00:00Z&period_to={}-01-01T00:00Z&page_size=25000&order_by=period".format( + OCTOPUS_API_BASE, self.mpan, self.serial, year, year + 1 + ) + rows = [] + pages = 0 + while url and pages < 40: + data = await self._get_json(session, url) + if not data or "results" not in data: + break + rows += data["results"] + url = data.get("next", None) + pages += 1 + + self.consumption = parse_consumption_results(rows) + if not self.consumption: + self.log("Warn: Annual: Octopus returned no complete days of consumption for {}".format(year)) + return False + + self.log("Annual: Octopus consumption for {} downloaded, {} complete days".format(year, len(self.consumption))) + if self.storage: + await self.storage.save("annual", cache_key, {day.isoformat(): slots for day, slots in self.consumption.items()}, format="json") + return True + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + slots = self.consumption.get(day) + if slots is not None: + return sum(slots) + if self.fallback: + return self.fallback.daily_kwh(day) + return 0.0 + + def minute_profile(self, day): + """Return 1440 per-minute kWh values, falling back or returning None when the day is missing.""" + slots = self.consumption.get(day) + if slots is None: + self.missing_days.add(day) + if self.fallback: + return self.fallback.minute_profile(day) + return None + profile = [] + for slot_value in slots: + profile.extend([slot_value / MINUTES_PER_SLOT] * MINUTES_PER_SLOT) + return profile +``` + +- [ ] **Step 4: Run both load tests to verify they pass** + +Run: `cd coverage && ./run_all -k annual_load > /tmp/t4.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t4.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_load.py apps/predbat/tests/test_annual_load.py apps/predbat/unit_test.py +git commit -m "feat(annual): add Octopus consumption load profile source" +``` + +--- + +## Task 5: Open-Meteo weather module + +Downloads two archives per array — ERA5 actuals and the archived short-range forecast — converts both through the Task 1 solar model, and derives each month's P10 ratio from the measured forecast error. + +The module takes an injectable `fetch_json` coroutine so tests never touch the network. + +**Files:** +- Create: `apps/predbat/annual_weather.py` +- Create: `apps/predbat/tests/test_annual_weather.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `solar_model.convert_azimuth`, `solar_model.gti_hourly_to_period_kwh`. +- Produces: + - `annual_weather.ARCHIVE_URL`, `annual_weather.FORECAST_ARCHIVE_URL` — base URL constants + - `annual_weather.percentile(values, fraction) -> float` + - `annual_weather.WeatherYear` with: + - `pv_minutes(series, midnight_utc, minutes) -> dict[int, float]` where `series` is `"actual"` or `"forecast"` + - `pv_minutes_p10(midnight_utc, minutes, month) -> dict[int, float]` + - `daily_actual_kwh(day) -> float` + - `has_actual(day) -> bool` + - `p10_ratio(month) -> float` + - `forecast_available` (bool), `fallback_months` (set of ints) + - `annual_weather.AnnualWeather(arrays, latitude, longitude, log, storage=None, p10_fallback=0.7, fetch_json=None)` with `async fetch(year) -> WeatherYear` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_weather.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction Open-Meteo weather module.""" + +import asyncio +from datetime import date, datetime, timedelta + +import pytz + +from annual_weather import AnnualWeather, percentile + +ARRAYS = [{"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}] + + +def build_hourly(start_day, days, peak_gti): + """Build a synthetic Open-Meteo hourly payload with a fixed daily irradiance curve.""" + times = [] + gti = [] + temp = [] + wind = [] + curve = [0, 0, 0, 0, 0, 0, 0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.3, 0.1, 0, 0, 0, 0, 0] + for offset in range(days): + day = start_day + timedelta(days=offset) + for hour in range(24): + times.append("{}T{:02d}:00".format(day.isoformat(), hour)) + gti.append(peak_gti * curve[hour]) + temp.append(15.0) + wind.append(1.5) + return {"hourly": {"time": times, "global_tilted_irradiance": gti, "temperature_2m": temp, "wind_speed_10m": wind}} + + +def test_annual_weather(my_predbat): + """Verify weather fetching, P10 derivation from forecast error, and the fallback path.""" + failed = False + print("**** Testing annual_weather ****") + + print("Test: percentile picks the expected order statistic") + values = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + got = percentile(values, 0.10) + if got != 1.0: + print(" ERROR: 10th percentile of 1..10 expected 1.0, got {}".format(got)) + failed = True + if percentile([], 0.10) != 0.0: + print(" ERROR: percentile of an empty list should be 0.0") + failed = True + + # Actuals peak at 800, forecast peaks at 1000, so every day's actual/forecast ratio is 0.8 + start = date(2025, 1, 1) + actual_payload = build_hourly(start, 40, 800.0) + forecast_payload = build_hourly(start, 40, 1000.0) + + async def fake_fetch(url): + """Return the archive or forecast payload depending on the host in the URL.""" + if "archive-api" in url: + return actual_payload + return forecast_payload + + weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=fake_fetch) + year = asyncio.get_event_loop().run_until_complete(weather.fetch(2025)) + + print("Test: the forecast archive is reported as available") + if not year.forecast_available: + print(" ERROR: forecast_available should be True when both payloads parse") + failed = True + + print("Test: January's P10 ratio reflects the measured 0.8 forecast error") + ratio = year.p10_ratio(1) + if abs(ratio - 0.8) > 0.01: + print(" ERROR: expected a P10 ratio near 0.8, got {}".format(ratio)) + failed = True + + print("Test: actual daily energy is below forecast daily energy") + day = date(2025, 1, 10) + if not year.has_actual(day): + print(" ERROR: expected actual data for {}".format(day)) + failed = True + midnight = pytz.utc.localize(datetime(2025, 1, 10, 0, 0)) + actual_minutes = year.pv_minutes("actual", midnight, 24 * 60) + forecast_minutes = year.pv_minutes("forecast", midnight, 24 * 60) + actual_total = sum(actual_minutes.values()) + forecast_total = sum(forecast_minutes.values()) + if actual_total <= 0: + print(" ERROR: actual PV for {} should be positive, got {}".format(day, actual_total)) + failed = True + if forecast_total <= actual_total: + print(" ERROR: forecast total {} should exceed actual total {}".format(forecast_total, actual_total)) + failed = True + + print("Test: pv_minutes covers a 48 hour window and is keyed by absolute minute") + two_day = year.pv_minutes("actual", midnight, 48 * 60) + if max(two_day.keys()) >= 48 * 60: + print(" ERROR: pv_minutes must not emit minutes at or beyond the window length") + failed = True + if abs(sum(two_day.values()) - (actual_total + year.daily_actual_kwh(date(2025, 1, 11)))) > 0.01: + print(" ERROR: the 48 hour window should equal two days of actual energy") + failed = True + + print("Test: pv_minutes_p10 scales the forecast series by the month ratio") + p10_minutes = year.pv_minutes_p10(midnight, 24 * 60, 1) + expected = forecast_total * year.p10_ratio(1) + if abs(sum(p10_minutes.values()) - expected) > 0.01: + print(" ERROR: P10 total {} expected {}".format(sum(p10_minutes.values()), expected)) + failed = True + + print("Test: a missing forecast archive falls back and records the degradation") + async def actuals_only_fetch(url): + """Serve actuals and fail every forecast request.""" + if "archive-api" in url: + return actual_payload + return None + + degraded_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=actuals_only_fetch, p10_fallback=0.7) + degraded = asyncio.get_event_loop().run_until_complete(degraded_weather.fetch(2025)) + if degraded.forecast_available: + print(" ERROR: forecast_available should be False when the forecast archive is empty") + failed = True + if abs(degraded.p10_ratio(1) - 0.7) > 1e-9: + print(" ERROR: expected the 0.7 fallback ratio, got {}".format(degraded.p10_ratio(1))) + failed = True + if 1 not in degraded.fallback_months: + print(" ERROR: January should be recorded in fallback_months") + failed = True + degraded_forecast = degraded.pv_minutes("forecast", midnight, 24 * 60) + degraded_actual = degraded.pv_minutes("actual", midnight, 24 * 60) + if abs(sum(degraded_forecast.values()) - sum(degraded_actual.values())) > 1e-9: + print(" ERROR: with no forecast archive the forecast series must fall back to actuals") + failed = True + + print("Test: a month with fewer than seven usable days falls back") + sparse_actual = build_hourly(date(2025, 1, 1), 4, 800.0) + sparse_forecast = build_hourly(date(2025, 1, 1), 4, 1000.0) + + async def sparse_fetch(url): + """Serve only four days of data.""" + return sparse_actual if "archive-api" in url else sparse_forecast + + sparse_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=sparse_fetch, p10_fallback=0.7) + sparse = asyncio.get_event_loop().run_until_complete(sparse_weather.fetch(2025)) + if abs(sparse.p10_ratio(1) - 0.7) > 1e-9: + print(" ERROR: a four-day month should fall back to 0.7, got {}".format(sparse.p10_ratio(1))) + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_weather import test_annual_weather +``` + +```python + ("annual_weather", test_annual_weather, "Annual prediction Open-Meteo weather tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_weather > /tmp/t5.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t5.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_weather'`. + +- [ ] **Step 3: Create `apps/predbat/annual_weather.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Open-Meteo historical weather for the annual prediction tool. + +Downloads two archives per PV array: ERA5 reanalysis actuals (what really +happened) and the archived short-range forecast for the same dates (what Predbat +would have been looking at). The gap between them is genuine day-ahead forecast +error, from which each month's P10 ratio is derived. +""" + +import math +from datetime import timedelta + +import aiohttp + +from solar_model import convert_azimuth, gti_hourly_to_period_kwh + +ARCHIVE_URL = "https://archive-api.open-meteo.com/v1/archive" +FORECAST_ARCHIVE_URL = "https://historical-forecast-api.open-meteo.com/v1/forecast" + +HOURLY_VARIABLES = "global_tilted_irradiance,temperature_2m,wind_speed_10m" + +# A month needs at least this many usable forecast/actual day pairs before its +# measured P10 ratio is trusted over the flat fallback. +MIN_DAYS_FOR_P10 = 7 + +# Fraction used for the P10 order statistic +P10_FRACTION = 0.10 + + +def percentile(values, fraction): + """Return the order statistic at ``fraction`` through a list of values. + + Uses the same convention as the Solcast ensemble P10 in ``solcast.py``: + sort ascending and take index ``ceil(n * fraction) - 1``, clamped to zero. + Returns 0.0 for an empty list. + """ + if not values: + return 0.0 + ordered = sorted(values) + index = max(0, math.ceil(len(ordered) * fraction) - 1) + return ordered[index] + + +class WeatherYear: + """A year of per-array-summed PV energy, for both actuals and forecast.""" + + def __init__(self, actual_periods, forecast_periods, p10_ratios, forecast_available, fallback_months): + """Hold the converted period data and the derived monthly P10 ratios.""" + self.actual_periods = actual_periods + self.forecast_periods = forecast_periods if forecast_available else actual_periods + self.p10_ratios = p10_ratios + self.forecast_available = forecast_available + self.fallback_months = fallback_months + self._daily_actual = self._daily_totals(self.actual_periods) + + @staticmethod + def _daily_totals(periods): + """Sum hourly period energy into per-date totals.""" + totals = {} + for stamp, kwh in periods.items(): + day = stamp.date() + totals[day] = totals.get(day, 0.0) + kwh + return totals + + def has_actual(self, day): + """Return True when actuals exist for the given date.""" + return day in self._daily_actual + + def daily_actual_kwh(self, day): + """Return the total actual PV kWh generated on the given date.""" + return self._daily_actual.get(day, 0.0) + + def p10_ratio(self, month): + """Return the P10 scaling ratio for the given month number, 1 = January.""" + return self.p10_ratios.get(month, 1.0) + + def pv_minutes(self, series, midnight_utc, minutes): + """Spread hourly period energy across per-minute kWh, keyed by absolute minute. + + ``series`` is "actual" or "forecast". Minutes outside [0, minutes) are + discarded, so the caller always receives a window it asked for. + """ + periods = self.actual_periods if series == "actual" else self.forecast_periods + result = {} + end_utc = midnight_utc + timedelta(minutes=minutes) + for stamp, kwh in periods.items(): + if stamp < midnight_utc or stamp >= end_utc: + continue + offset = int((stamp - midnight_utc).total_seconds() // 60) + per_minute = kwh / 60.0 + for minute in range(offset, offset + 60): + if 0 <= minute < minutes: + result[minute] = result.get(minute, 0.0) + per_minute + return result + + def pv_minutes_p10(self, midnight_utc, minutes, month): + """Return the P10 per-minute series: the forecast series scaled by the month's ratio.""" + ratio = self.p10_ratio(month) + return {minute: value * ratio for minute, value in self.pv_minutes("forecast", midnight_utc, minutes).items()} + + +class AnnualWeather: + """Fetches and converts a calendar year of Open-Meteo data for one site.""" + + def __init__(self, arrays, latitude, longitude, log, storage=None, p10_fallback=0.7, fetch_json=None): + """Configure the site's PV arrays and the JSON fetcher used for downloads.""" + self.arrays = arrays + self.latitude = latitude + self.longitude = longitude + self.log = log + self.storage = storage + self.p10_fallback = p10_fallback + self.fetch_json = fetch_json or self._default_fetch_json + + async def _default_fetch_json(self, url): + """Download and decode one JSON document, returning None on any failure.""" + try: + async with aiohttp.ClientSession() as session: + async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=120)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Open-Meteo request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Open-Meteo request to {} failed: {}".format(url, error)) + return None + + def _build_url(self, base, array, year): + """Build one Open-Meteo request URL for a single array and calendar year. + + The window runs to 1 January of the following year so the final sampled + day still has the following day its 48 hour plan needs. + """ + azimuth = array.get("azimuth", 180.0) + if not array.get("azimuth_zero_south", False): + azimuth = convert_azimuth(azimuth) + return "{}?latitude={}&longitude={}&start_date={}-01-01&end_date={}-01-01&hourly={}&tilt={}&azimuth={}&wind_speed_unit=ms&timezone=UTC".format( + base, self.latitude, self.longitude, year, year + 1, HOURLY_VARIABLES, array.get("declination", 35.0), azimuth + ) + + async def _fetch_series(self, base, year, cache_tag): + """Fetch one source for every array and return the summed hourly period energy.""" + totals = {} + any_data = False + for index, array in enumerate(self.arrays): + url = self._build_url(base, array, year) + data = None + cache_key = "weather_{}_{}_{}_{}_{}".format(cache_tag, year, index, self.latitude, self.longitude) + if self.storage: + data = await self.storage.load("annual", cache_key) + if not data: + data = await self.fetch_json(url) + if data and self.storage: + await self.storage.save("annual", cache_key, data, format="json") + if not data: + self.log("Warn: Annual: no {} data for array {}".format(cache_tag, index)) + continue + + hourly = data.get("hourly", {}) + times = hourly.get("time", []) + gti_values = hourly.get("global_tilted_irradiance", []) + if not times or not gti_values: + self.log("Warn: Annual: {} data for array {} has no hourly values".format(cache_tag, index)) + continue + + periods = gti_hourly_to_period_kwh( + times, + gti_values, + hourly.get("temperature_2m", []), + hourly.get("wind_speed_10m", []), + kwp=array.get("kwp", 3.0), + system_loss=1.0 - array.get("efficiency", 0.95), + shading_factors=array.get("shading_factors", None), + ) + for stamp, values in periods.items(): + totals[stamp] = totals.get(stamp, 0.0) + values["pv_estimate"] + any_data = True + + return totals if any_data else {} + + def _derive_p10_ratios(self, actual_periods, forecast_periods, forecast_available): + """Derive each month's P10 ratio from the measured actual/forecast daily energy error.""" + ratios = {} + fallback_months = set() + + actual_daily = WeatherYear._daily_totals(actual_periods) + forecast_daily = WeatherYear._daily_totals(forecast_periods) + + by_month = {} + if forecast_available: + for day, forecast_kwh in forecast_daily.items(): + if forecast_kwh <= 0: + continue + if day not in actual_daily: + continue + by_month.setdefault(day.month, []).append(actual_daily[day] / forecast_kwh) + + for month in range(1, 13): + samples = by_month.get(month, []) + if len(samples) >= MIN_DAYS_FOR_P10: + ratios[month] = min(1.0, percentile(samples, P10_FRACTION)) + else: + ratios[month] = self.p10_fallback + fallback_months.add(month) + + if fallback_months: + self.log("Warn: Annual: P10 fell back to the flat {} derate for months {}".format(self.p10_fallback, sorted(fallback_months))) + + return ratios, fallback_months + + async def fetch(self, year): + """Download and convert a calendar year, returning a populated WeatherYear.""" + actual_periods = await self._fetch_series(ARCHIVE_URL, year, "actual") + forecast_periods = await self._fetch_series(FORECAST_ARCHIVE_URL, year, "forecast") + forecast_available = bool(forecast_periods) + + if not forecast_available: + self.log("Warn: Annual: the Open-Meteo forecast archive returned nothing for {}; planning on actuals with the flat P10 derate".format(year)) + + ratios, fallback_months = self._derive_p10_ratios(actual_periods, forecast_periods, forecast_available) + return WeatherYear(actual_periods, forecast_periods, ratios, forecast_available, fallback_months) +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_weather > /tmp/t5.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t5.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_weather.py apps/predbat/tests/test_annual_weather.py apps/predbat/unit_test.py +git commit -m "feat(annual): add Open-Meteo actuals and forecast archive weather module" +``` + +--- + +## Task 6: Tariff module + +Resolves import and export rates for a specific historical date, either from an Octopus product URL with `period_from` / `period_to`, or from a basic rates structure. Fetches a month at a time and slices per day, which keeps the API call count low. + +**Files:** +- Create: `apps/predbat/annual_tariff.py` +- Create: `apps/predbat/tests/test_annual_tariff.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `utils.minute_data` (already used by `octopus.py:2338` for exactly this parsing), `PredBat.basic_rates`. +- Produces: + - `annual_tariff.build_period_url(base_url, start_utc, end_utc) -> str` + - `annual_tariff.AnnualTariff(config, log, predbat, storage=None, fetch_json=None)` with: + - `async fetch_month(year, month)` — populates the internal cache; returns True on success + - `rates_for(midnight_utc, minutes) -> (dict, dict)` — import and export rate dicts keyed by absolute minute + - `month_available(year, month) -> bool` + - `standing_charge_p_per_day` (float) + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_tariff.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction tariff module.""" + +import asyncio +from datetime import datetime, timedelta + +import pytz + +from annual_tariff import AnnualTariff, build_period_url + + +def build_agile_results(start_day, days, base_rate): + """Build a synthetic Octopus half-hourly rate payload with a repeating daily shape.""" + results = [] + stamp = pytz.utc.localize(datetime(start_day.year, start_day.month, start_day.day)) + for slot in range(days * 48): + valid_from = stamp + timedelta(minutes=30 * slot) + valid_to = valid_from + timedelta(minutes=30) + # Cheap overnight, expensive in the evening peak + hour = valid_from.hour + rate = base_rate * (0.3 if hour < 5 else (2.0 if 16 <= hour < 19 else 1.0)) + results.append( + { + "value_inc_vat": round(rate, 4), + "valid_from": valid_from.strftime("%Y-%m-%dT%H:%M:%SZ"), + "valid_to": valid_to.strftime("%Y-%m-%dT%H:%M:%SZ"), + } + ) + return results + + +def test_annual_tariff(my_predbat): + """Verify Octopus date-ranged rate fetching, pagination, slicing and basic rates.""" + failed = False + print("**** Testing annual_tariff ****") + from datetime import date + + print("Test: build_period_url appends the date range without losing existing query parameters") + start = pytz.utc.localize(datetime(2025, 3, 1)) + end = pytz.utc.localize(datetime(2025, 4, 1)) + plain = build_period_url("https://example.com/rates/", start, end) + if "?period_from=2025-03-01T00:00Z" not in plain or "period_to=2025-04-01T00:00Z" not in plain: + print(" ERROR: expected a period range in {}".format(plain)) + failed = True + existing = build_period_url("https://example.com/rates/?page_size=100", start, end) + if "&period_from=" not in existing or "page_size=100" not in existing: + print(" ERROR: existing query parameters must be preserved, got {}".format(existing)) + failed = True + + print("Test: an Octopus URL tariff resolves rates for a specific date, following pagination") + page_two = {"results": build_agile_results(date(2025, 3, 16), 16, 20.0), "next": None} + page_one = {"results": build_agile_results(date(2025, 3, 1), 15, 20.0), "next": "https://example.com/page2"} + + calls = [] + + async def fake_fetch(url): + """Serve two pages of rate data and record the URLs requested.""" + calls.append(url) + return page_two if "page2" in url else page_one + + config = {"import_octopus_url": "https://example.com/import/", "export_octopus_url": "https://example.com/export/", "standing_charge_p_per_day": 60.0} + tariff = AnnualTariff(config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + ok = asyncio.get_event_loop().run_until_complete(tariff.fetch_month(2025, 3)) + if not ok: + print(" ERROR: fetch_month should succeed with valid payloads") + failed = True + if not tariff.month_available(2025, 3): + print(" ERROR: March 2025 should be reported as available") + failed = True + if len(calls) != 4: + print(" ERROR: expected 4 requests (2 pages x import and export), got {}".format(len(calls))) + failed = True + + print("Test: rates_for returns a 48 hour window keyed by absolute minute") + midnight = pytz.utc.localize(datetime(2025, 3, 10)) + rate_import, rate_export = tariff.rates_for(midnight, 48 * 60) + if len(rate_import) < 48 * 60: + print(" ERROR: expected at least {} import rate minutes, got {}".format(48 * 60, len(rate_import))) + failed = True + if rate_import.get(0) is None: + print(" ERROR: minute 0 must have an import rate") + failed = True + # 02:00 is in the cheap overnight band, 17:00 is in the peak band + if not rate_import[120] < rate_import[17 * 60]: + print(" ERROR: overnight rate {} should be below peak rate {}".format(rate_import[120], rate_import[17 * 60])) + failed = True + if abs(rate_import[120] - 6.0) > 0.01: + print(" ERROR: overnight rate expected 6.0, got {}".format(rate_import[120])) + failed = True + + print("Test: the second day of the window carries the following day's rates") + if rate_import.get(24 * 60 + 120) is None: + print(" ERROR: the second day of the window must be populated") + failed = True + + print("Test: a failed download reports the month as unavailable rather than returning zeros") + async def failing_fetch(url): + """Simulate a download failure.""" + return None + + broken = AnnualTariff(config, log=print, predbat=my_predbat, fetch_json=failing_fetch) + if asyncio.get_event_loop().run_until_complete(broken.fetch_month(2025, 4)): + print(" ERROR: fetch_month should report failure when the download fails") + failed = True + if broken.month_available(2025, 4): + print(" ERROR: a failed month must not be reported as available") + failed = True + + print("Test: basic rates repeat a fixed daily pattern across the window") + basic_config = { + "rates_import": [{"start": "00:00:00", "end": "05:00:00", "rate": 7.0}, {"start": "05:00:00", "end": "00:00:00", "rate": 30.0}], + "rates_export": [{"rate": 15.0}], + "standing_charge_p_per_day": 45.0, + } + basic = AnnualTariff(basic_config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + if not asyncio.get_event_loop().run_until_complete(basic.fetch_month(2025, 3)): + print(" ERROR: basic rates should always be available") + failed = True + basic_import, basic_export = basic.rates_for(midnight, 48 * 60) + if abs(basic_import[120] - 7.0) > 0.001: + print(" ERROR: basic overnight import expected 7.0, got {}".format(basic_import[120])) + failed = True + if abs(basic_import[10 * 60] - 30.0) > 0.001: + print(" ERROR: basic daytime import expected 30.0, got {}".format(basic_import[10 * 60])) + failed = True + if abs(basic_import[24 * 60 + 120] - 7.0) > 0.001: + print(" ERROR: basic rates must repeat on day two, got {}".format(basic_import[24 * 60 + 120])) + failed = True + if abs(basic_export[10 * 60] - 15.0) > 0.001: + print(" ERROR: basic export expected 15.0, got {}".format(basic_export[10 * 60])) + failed = True + + print("Test: standing charge is carried through from config") + if abs(tariff.standing_charge_p_per_day - 60.0) > 0.001: + print(" ERROR: standing charge expected 60.0, got {}".format(tariff.standing_charge_p_per_day)) + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_tariff import test_annual_tariff +``` + +```python + ("annual_tariff", test_annual_tariff, "Annual prediction tariff tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_tariff > /tmp/t6.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t6.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_tariff'`. + +- [ ] **Step 3: Create `apps/predbat/annual_tariff.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Historical tariff resolution for the annual prediction tool. + +Resolves import and export rates for a specific past date, either from an +Octopus product URL using period_from/period_to, or from a static basic rates +structure. Fetching is done a month at a time and sliced per sampled day. +""" + +import calendar +from datetime import datetime, timedelta + +import aiohttp +import pytz + +from utils import minute_data + +MINUTES_PER_DAY = 24 * 60 + + +def build_period_url(base_url, start_utc, end_utc): + """Append period_from/period_to to an Octopus rates URL, preserving existing query parameters.""" + separator = "&" if "?" in base_url else "?" + return "{}{}period_from={}&period_to={}&page_size=1500".format(base_url, separator, start_utc.strftime("%Y-%m-%dT%H:%MZ"), end_utc.strftime("%Y-%m-%dT%H:%MZ")) + + +class AnnualTariff: + """Import and export rates for arbitrary historical dates.""" + + def __init__(self, config, log, predbat, storage=None, fetch_json=None): + """Configure the tariff from the annual config's ``tariff`` block. + + Octopus product codes are region-suffixed. ``resolve_arg`` substitutes + ``{dno_region}`` from ``predbat.args``, so the region is injected there + first. Without it a URL silently 404s and the month is reported + unavailable, which looks like an outage rather than a config mistake. + """ + self.config = config + self.log = log + self.predbat = predbat + self.storage = storage + self.fetch_json = fetch_json or self._default_fetch_json + if config.get("dno_region"): + predbat.args["dno_region"] = config["dno_region"] + self.import_url = self._resolve_url(config.get("import_octopus_url"), "import_octopus_url") + self.export_url = self._resolve_url(config.get("export_octopus_url"), "export_octopus_url") + self.basic_import = config.get("rates_import") + self.basic_export = config.get("rates_export") + self.standing_charge_p_per_day = float(config.get("standing_charge_p_per_day", 0.0)) + # Keyed by (year, month); each value is a dict of tz-aware UTC stamp to rate + self.import_rates = {} + self.export_rates = {} + self.available = set() + + def _resolve_url(self, url, name): + """Substitute templated arguments such as {dno_region} into a tariff URL.""" + if not url: + return None + return self.predbat.resolve_arg(name, url, indirect=False) + + async def _default_fetch_json(self, url): + """Download and decode one JSON document, returning None on any failure.""" + try: + async with aiohttp.ClientSession() as session: + async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=60)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Octopus rate request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Octopus rate request to {} failed: {}".format(url, error)) + return None + + async def _download_octopus(self, base_url, start_utc, end_utc, cache_key): + """Download every page of Octopus rates for a date range, returning raw result rows.""" + if self.storage: + cached = await self.storage.load("annual", cache_key) + if isinstance(cached, list) and cached: + return cached + + url = build_period_url(base_url, start_utc, end_utc) + rows = [] + pages = 0 + while url and pages < 10: + data = await self.fetch_json(url) + if not data or "results" not in data: + break + rows += data["results"] + url = data.get("next", None) + pages += 1 + + if rows and self.storage: + await self.storage.save("annual", cache_key, rows, format="json") + return rows + + @staticmethod + def _rows_to_stamped_rates(rows, start_utc, days): + """Convert Octopus rate rows into a dict of tz-aware UTC stamp to rate. + + Reuses ``minute_data`` exactly as ``octopus.py`` does, then re-keys the + minute offsets back onto absolute timestamps so a single monthly download + can be sliced for any day within it. + """ + parsed, _ = minute_data(rows, days + 1, start_utc, "value_inc_vat", "valid_from", backwards=False, to_key="valid_to") + return {start_utc + timedelta(minutes=minute): rate for minute, rate in parsed.items()} + + async def fetch_month(self, year, month): + """Fetch (or synthesise) the rates covering one calendar month plus a one day buffer. + + Returns True when usable rates exist for the month. The buffer day lets the + last sampled day of the month complete its 48 hour plan. + """ + key = (year, month) + if self.import_url or self.export_url: + days_in_month = calendar.monthrange(year, month)[1] + start_utc = pytz.utc.localize(datetime(year, month, 1)) + end_utc = start_utc + timedelta(days=days_in_month + 2) + days = days_in_month + 2 + + import_rates = {} + export_rates = {} + if self.import_url: + rows = await self._download_octopus(self.import_url, start_utc, end_utc, "rates_import_{}_{:02d}".format(year, month)) + if not rows: + self.log("Warn: Annual: no import rates available for {}-{:02d}".format(year, month)) + return False + import_rates = self._rows_to_stamped_rates(rows, start_utc, days) + if self.export_url: + rows = await self._download_octopus(self.export_url, start_utc, end_utc, "rates_export_{}_{:02d}".format(year, month)) + if rows: + export_rates = self._rows_to_stamped_rates(rows, start_utc, days) + else: + self.log("Warn: Annual: no export rates available for {}-{:02d}, treating export as unpaid".format(year, month)) + + if not import_rates: + return False + self.import_rates[key] = import_rates + self.export_rates[key] = export_rates + self.available.add(key) + return True + + # Basic rates repeat a fixed daily pattern, so nothing needs downloading + self.available.add(key) + return True + + def month_available(self, year, month): + """Return True when usable rates exist for the given month.""" + return (year, month) in self.available + + def _basic_window(self, info, name, minutes): + """Expand a basic rates structure across the requested window.""" + rates = self.predbat.basic_rates(info, name) + return {minute: rates.get(minute % MINUTES_PER_DAY, 0.0) for minute in range(minutes)} + + def rates_for(self, midnight_utc, minutes): + """Return (import, export) rate dicts keyed by absolute minute from ``midnight_utc``.""" + key = (midnight_utc.year, midnight_utc.month) + + if self.import_url or self.export_url: + import_stamped = self.import_rates.get(key, {}) + export_stamped = self.export_rates.get(key, {}) + # A 48 hour window starting late in a month spills into the next month's download + next_key = (midnight_utc.year, midnight_utc.month + 1) if midnight_utc.month < 12 else (midnight_utc.year + 1, 1) + import_stamped = dict(import_stamped) + import_stamped.update(self.import_rates.get(next_key, {})) + export_stamped = dict(export_stamped) + export_stamped.update(self.export_rates.get(next_key, {})) + + rate_import = {} + rate_export = {} + for minute in range(minutes): + stamp = midnight_utc + timedelta(minutes=minute) + if stamp in import_stamped: + rate_import[minute] = import_stamped[stamp] + if stamp in export_stamped: + rate_export[minute] = export_stamped[stamp] + return rate_import, rate_export + + rate_import = self._basic_window(self.basic_import or [], "rates_import", minutes) + rate_export = self._basic_window(self.basic_export or [], "rates_export", minutes) + return rate_import, rate_export +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_tariff > /tmp/t6.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t6.txt` + +Expected: no output. If `minute_data` returns fewer minutes than expected, check its `to_key` handling against `octopus.py:2338` — the call signature must match exactly. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_tariff.py apps/predbat/tests/test_annual_tariff.py apps/predbat/unit_test.py +git commit -m "feat(annual): add historical tariff resolution module" +``` + +--- + +## Task 7: Config validation + +The first piece of `annual.py`. Rejects contradictory input at load time rather than producing a plausible but wrong answer — in particular an Octopus API key alongside a manual load figure, because the meter series already contains any EV charging and accepting both would double-count it. + +**Files:** +- Create: `apps/predbat/annual.py` +- Create: `apps/predbat/tests/test_annual_config.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `annual.AnnualConfigError` — exception raised for invalid configuration + - `annual.validate_config(config, today=None) -> dict` — returns a fully defaulted, normalised config + - `annual.scrub_secrets(config) -> dict` — deep copy with secret-looking values replaced by `"xxx"` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_config.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for annual prediction config validation.""" + +from datetime import date + +from annual import AnnualConfigError, scrub_secrets, validate_config + + +def base_config(): + """Return a minimal valid annual config.""" + return { + "annual": { + "location": {"latitude": 51.5, "longitude": -0.1}, + "solar": [{"kwp": 5.6}], + "battery": {"size_kwh": 9.5, "inverter_kw": 5.0}, + "load": {"annual_kwh": 3800}, + "tariff": {"rates_import": [{"rate": 25.0}]}, + } + } + + +def expect_error(label, config, fragment, failed): + """Assert that validate_config rejects the config with a message containing fragment.""" + try: + validate_config(config) + except AnnualConfigError as error: + if fragment.lower() not in str(error).lower(): + print(" ERROR: {} raised '{}', expected it to mention '{}'".format(label, error, fragment)) + return True + return failed + print(" ERROR: {} should have raised AnnualConfigError".format(label)) + return True + + +def test_annual_config(my_predbat): + """Verify annual config defaulting, normalisation and rejection rules.""" + failed = False + print("**** Testing annual config validation ****") + + print("Test: a minimal config validates and gains defaults") + result = validate_config(base_config(), today=date(2026, 7, 25)) + if result["year"] != 2025: + print(" ERROR: year should default to the most recent complete calendar year, got {}".format(result["year"])) + failed = True + if result["samples_per_month"] != 2: + print(" ERROR: samples_per_month should default to 2, got {}".format(result["samples_per_month"])) + failed = True + if result["timezone"] != "Europe/London": + print(" ERROR: timezone should default to Europe/London, got {}".format(result["timezone"])) + failed = True + if abs(result["pv10_derate_fallback"] - 0.7) > 1e-9: + print(" ERROR: pv10_derate_fallback should default to 0.7, got {}".format(result["pv10_derate_fallback"])) + failed = True + if result["load"]["shape"] != "flat": + print(" ERROR: load shape should default to flat, got {}".format(result["load"]["shape"])) + failed = True + if result["solar"][0]["declination"] != 35 or result["solar"][0]["azimuth"] != 180: + print(" ERROR: solar defaults should be declination 35 azimuth 180, got {}".format(result["solar"][0])) + failed = True + if abs(result["solar"][0]["efficiency"] - 0.95) > 1e-9: + print(" ERROR: solar efficiency should default to 0.95, got {}".format(result["solar"][0]["efficiency"])) + failed = True + if result["battery"]["charge_rate_kw"] != 5.0 or result["battery"]["discharge_rate_kw"] != 5.0: + print(" ERROR: charge and discharge rates should default to inverter_kw, got {}".format(result["battery"])) + failed = True + if result["battery"]["export_limit_kw"] != 5.0: + print(" ERROR: export_limit_kw should default to inverter_kw, got {}".format(result["battery"])) + failed = True + if result["battery"]["hybrid"] is not True: + print(" ERROR: hybrid should default to True, got {}".format(result["battery"].get("hybrid"))) + failed = True + + print("Test: an unwrapped config without the 'annual' key is accepted") + unwrapped = base_config()["annual"] + result = validate_config(unwrapped, today=date(2026, 7, 25)) + if result["year"] != 2025: + print(" ERROR: an unwrapped config should validate the same way") + failed = True + + print("Test: Octopus load together with a manual figure is rejected") + config = base_config() + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + failed = expect_error("octopus plus annual_kwh", config, "mutually exclusive", failed) + + config = base_config() + del config["annual"]["load"]["annual_kwh"] + config["annual"]["load"]["car_charging_kwh"] = 2500 + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + failed = expect_error("octopus plus car_charging_kwh", config, "mutually exclusive", failed) + + print("Test: an Octopus-only load block validates") + config = base_config() + del config["annual"]["load"]["annual_kwh"] + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["load"].get("octopus", {}).get("account_id") != "A-1": + print(" ERROR: the Octopus load block should survive validation") + failed = True + + print("Test: a missing battery block yields a two-scenario run") + config = base_config() + del config["annual"]["battery"] + result = validate_config(config, today=date(2026, 7, 25)) + if result["battery"] is not None: + print(" ERROR: an omitted battery should normalise to None, got {}".format(result["battery"])) + failed = True + + print("Test: a missing solar block is allowed for a battery-only run") + config = base_config() + del config["annual"]["solar"] + result = validate_config(config, today=date(2026, 7, 25)) + if result["solar"] != []: + print(" ERROR: an omitted solar block should normalise to an empty list, got {}".format(result["solar"])) + failed = True + + print("Test: omitting both solar and battery is rejected as pointless") + config = base_config() + del config["annual"]["solar"] + del config["annual"]["battery"] + failed = expect_error("neither solar nor battery", config, "at least one", failed) + + print("Test: missing location is rejected") + config = base_config() + del config["annual"]["location"] + failed = expect_error("no location", config, "location", failed) + + print("Test: missing load is rejected") + config = base_config() + del config["annual"]["load"] + failed = expect_error("no load", config, "load", failed) + + print("Test: missing tariff is rejected") + config = base_config() + del config["annual"]["tariff"] + failed = expect_error("no tariff", config, "tariff", failed) + + print("Test: an unknown load shape is rejected") + config = base_config() + config["annual"]["load"]["shape"] = "sideways" + failed = expect_error("bad shape", config, "shape", failed) + + print("Test: a solar array without kwp is rejected") + config = base_config() + config["annual"]["solar"] = [{"declination": 30}] + failed = expect_error("array without kwp", config, "kwp", failed) + + print("Test: samples_per_month below 1 is rejected") + config = base_config() + config["annual"]["samples_per_month"] = 0 + failed = expect_error("zero samples", config, "samples_per_month", failed) + + print("Test: a postcode-only location validates") + config = base_config() + config["annual"]["location"] = {"postcode": "SW1A 1AA"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["location"].get("postcode") != "SW1A 1AA": + print(" ERROR: a postcode location should survive validation") + failed = True + + print("Test: a templated tariff URL without dno_region is rejected up front") + config = base_config() + config["annual"]["tariff"] = {"import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/"} + failed = expect_error("templated url without region", config, "dno_region", failed) + + print("Test: a templated tariff URL with dno_region validates and is carried through") + config = base_config() + config["annual"]["tariff"] = {"import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/", "dno_region": "A"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["tariff"].get("dno_region") != "A": + print(" ERROR: dno_region should survive validation, got {}".format(result["tariff"].get("dno_region"))) + failed = True + + print("Test: scrub_secrets removes API keys without mutating the original") + config = base_config() + config["annual"]["load"]["octopus"] = {"api_key": "sk_live_secret", "account_id": "A-1"} + scrubbed = scrub_secrets(config) + if scrubbed["annual"]["load"]["octopus"]["api_key"] != "xxx": + print(" ERROR: api_key should be scrubbed, got {}".format(scrubbed["annual"]["load"]["octopus"]["api_key"])) + failed = True + if config["annual"]["load"]["octopus"]["api_key"] != "sk_live_secret": + print(" ERROR: scrub_secrets must not mutate its input") + failed = True + if scrubbed["annual"]["load"]["octopus"]["account_id"] != "A-1": + print(" ERROR: non-secret values should survive scrubbing") + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_config import test_annual_config +``` + +```python + ("annual_config", test_annual_config, "Annual prediction config validation tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_config > /tmp/t7.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t7.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual'`. + +- [ ] **Step 3: Create `apps/predbat/annual.py` with the validation layer** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Annual prediction engine. + +Projects a year of household electricity costs using the real Predbat planning +engine, reporting each month under three scenarios: no PV or battery, PV and +battery without Predbat, and with Predbat. Performs no HTTP itself; the weather +and tariff modules own all network access. +""" + +import copy +from datetime import date + +VALID_SHAPES = ["night", "day", "flat"] + +DEFAULT_TIMEZONE = "Europe/London" +DEFAULT_SAMPLES_PER_MONTH = 2 +DEFAULT_PV10_DERATE_FALLBACK = 0.7 +DEFAULT_DECLINATION = 35 +DEFAULT_AZIMUTH = 180 +DEFAULT_EFFICIENCY = 0.95 +DEFAULT_HYBRID = True + +# Substrings that mark a config value as secret and therefore scrubbable +SECRET_MARKERS = ["_key", "password", "token", "secret"] + + +class AnnualConfigError(ValueError): + """Raised when the annual prediction config is invalid or self-contradictory.""" + + +def scrub_secrets(config): + """Return a deep copy of the config with secret-looking values replaced by "xxx". + + Mirrors the redaction ``create_debug_yaml()`` applies, so a results document or + debug dump can never carry an API key. + """ + if isinstance(config, dict): + scrubbed = {} + for key, value in config.items(): + if any(marker in str(key).lower() for marker in SECRET_MARKERS): + scrubbed[key] = "xxx" + else: + scrubbed[key] = scrub_secrets(value) + return scrubbed + if isinstance(config, list): + return [scrub_secrets(item) for item in config] + return config + + +def _validate_solar(raw): + """Normalise the solar array list, applying defaults and rejecting arrays without kwp.""" + if raw is None: + return [] + if isinstance(raw, dict): + raw = [raw] + if not isinstance(raw, list): + raise AnnualConfigError("annual.solar must be a list of arrays") + + arrays = [] + for index, array in enumerate(raw): + if not isinstance(array, dict): + raise AnnualConfigError("annual.solar[{}] must be a mapping".format(index)) + if "kwp" not in array: + raise AnnualConfigError("annual.solar[{}] is missing kwp, the array's peak power in kW".format(index)) + normalised = dict(array) + normalised["kwp"] = float(array["kwp"]) + normalised["declination"] = array.get("declination", DEFAULT_DECLINATION) + normalised["azimuth"] = array.get("azimuth", DEFAULT_AZIMUTH) + normalised["efficiency"] = float(array.get("efficiency", DEFAULT_EFFICIENCY)) + arrays.append(normalised) + return arrays + + +def _validate_battery(raw): + """Normalise the battery block, or return None for a run with no battery.""" + if raw is None: + return None + if not isinstance(raw, dict): + raise AnnualConfigError("annual.battery must be a mapping") + if "size_kwh" not in raw: + raise AnnualConfigError("annual.battery is missing size_kwh") + if "inverter_kw" not in raw: + raise AnnualConfigError("annual.battery is missing inverter_kw") + + inverter_kw = float(raw["inverter_kw"]) + return { + "size_kwh": float(raw["size_kwh"]), + "inverter_kw": inverter_kw, + "export_limit_kw": float(raw.get("export_limit_kw", inverter_kw)), + "hybrid": bool(raw.get("hybrid", DEFAULT_HYBRID)), + "charge_rate_kw": float(raw.get("charge_rate_kw", inverter_kw)), + "discharge_rate_kw": float(raw.get("discharge_rate_kw", inverter_kw)), + } + + +def _validate_load(raw): + """Normalise the load block and enforce the Octopus / manual exclusivity rule.""" + if not isinstance(raw, dict): + raise AnnualConfigError("annual.load is required and must be a mapping") + + octopus = raw.get("octopus") + has_manual = ("annual_kwh" in raw) or ("car_charging_kwh" in raw) + + if octopus and has_manual: + raise AnnualConfigError("annual.load.octopus and annual.load.annual_kwh/car_charging_kwh are mutually exclusive: the Octopus consumption series already includes any car charging, so supplying both would double-count it") + + if not octopus and "annual_kwh" not in raw: + raise AnnualConfigError("annual.load requires either annual_kwh or an octopus block") + + if octopus: + if not isinstance(octopus, dict) or not octopus.get("api_key") or not octopus.get("account_id"): + raise AnnualConfigError("annual.load.octopus requires both api_key and account_id") + return {"octopus": dict(octopus), "shape": raw.get("shape", "flat"), "car_charging_kwh": 0.0} + + shape = raw.get("shape", "flat") + if shape not in VALID_SHAPES: + raise AnnualConfigError("annual.load.shape must be one of {}, got '{}'".format(VALID_SHAPES, shape)) + + return {"annual_kwh": float(raw["annual_kwh"]), "shape": shape, "car_charging_kwh": float(raw.get("car_charging_kwh", 0.0))} + + +def _validate_tariff(raw): + """Normalise the tariff block, requiring at least one import rate source. + + A URL containing {dno_region} with no dno_region supplied is rejected here + rather than left to 404 at fetch time, where it would surface as an + unavailable month and read like an Octopus outage. + """ + if not isinstance(raw, dict): + raise AnnualConfigError("annual.tariff is required and must be a mapping") + if not raw.get("import_octopus_url") and not raw.get("rates_import"): + raise AnnualConfigError("annual.tariff requires either import_octopus_url or rates_import") + + templated = [name for name in ["import_octopus_url", "export_octopus_url"] if raw.get(name) and "{dno_region}" in raw[name]] + if templated and not raw.get("dno_region"): + raise AnnualConfigError("annual.tariff.{} uses {{dno_region}} but annual.tariff.dno_region is not set; supply your Octopus region letter, for example 'A' for Eastern England".format(templated[0])) + + tariff = dict(raw) + tariff["standing_charge_p_per_day"] = float(raw.get("standing_charge_p_per_day", 0.0)) + return tariff + + +def validate_config(config, today=None): + """Validate and normalise an annual prediction config, returning a fully defaulted copy. + + Accepts either the wrapped form ({"annual": {...}}) or the inner mapping directly. + Raises AnnualConfigError with an actionable message on any problem. + """ + if not isinstance(config, dict): + raise AnnualConfigError("The annual config must be a mapping") + + raw = config.get("annual", config) + if not isinstance(raw, dict): + raise AnnualConfigError("The annual config must be a mapping") + + location = raw.get("location") + if not isinstance(location, dict): + raise AnnualConfigError("annual.location is required, with either a postcode or latitude and longitude") + if not location.get("postcode") and not ("latitude" in location and "longitude" in location): + raise AnnualConfigError("annual.location needs either a postcode or both latitude and longitude") + + solar = _validate_solar(raw.get("solar")) + battery = _validate_battery(raw.get("battery")) + if not solar and battery is None: + raise AnnualConfigError("annual needs at least one of solar or battery: with neither there is nothing to evaluate") + + samples_per_month = int(raw.get("samples_per_month", DEFAULT_SAMPLES_PER_MONTH)) + if samples_per_month < 1: + raise AnnualConfigError("annual.samples_per_month must be at least 1, got {}".format(samples_per_month)) + + if today is None: + today = date.today() + year = int(raw.get("year", today.year - 1)) + + return { + "location": dict(location), + "year": year, + "solar": solar, + "battery": battery, + "load": _validate_load(raw.get("load")), + "tariff": _validate_tariff(raw.get("tariff")), + "samples_per_month": samples_per_month, + "timezone": raw.get("timezone", DEFAULT_TIMEZONE), + "pv10_derate_fallback": float(raw.get("pv10_derate_fallback", DEFAULT_PV10_DERATE_FALLBACK)), + "raw": scrub_secrets(copy.deepcopy(raw)), + } +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_config > /tmp/t7.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t7.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual.py apps/predbat/tests/test_annual_config.py apps/predbat/unit_test.py +git commit -m "feat(annual): add annual prediction config validation" +``` + +--- + +## Task 8: Headless PredBat bootstrap and per-sample state reset + +`PredBat()` inherits `hass.Hass`, whose `__init__` reads `apps.yaml` from the current directory (or `$PREDBAT_APPS_FILE`) and opens `predbat.log` for append in the current directory. The tool therefore writes its own minimal `apps.yaml` into a working directory and points `PREDBAT_APPS_FILE` at it. + +**State isolation is the principal risk of driving the real `PredBat` object.** Derived state leaks between runs — `apps/predbat/tests/test_single_debug.py` documents exactly this for `dynamic_load_baseline` and `battery_rate_max_export`. `reset_sample_state()` is the explicit allow-list that stops month N's result depending on month N-1 having run first. + +**Files:** +- Modify: `apps/predbat/annual.py` +- Create: `apps/predbat/tests/test_annual_bootstrap.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual.validate_config` from Task 7. +- Produces: + - `annual.AnnualNullHA` — a no-op Home Assistant interface + - `annual.MINIMAL_APPS_YAML` — template string + - `annual.write_minimal_apps_yaml(work_dir, timezone) -> str` — returns the written path + - `annual.create_headless_predbat(work_dir, timezone, log) -> PredBat` + - `annual.reset_sample_state(predbat)` — resets every field a previous sample could have left behind + - `annual.apply_hardware(predbat, battery, solar)` — maps the config's battery block onto the PredBat instance + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_bootstrap.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction headless bootstrap and state reset.""" + +import os +import tempfile + +import yaml + +from annual import AnnualNullHA, apply_hardware, reset_sample_state, write_minimal_apps_yaml +from const import MINUTE_WATT + + +def test_annual_bootstrap(my_predbat): + """Verify the minimal apps.yaml, the null HA interface, hardware mapping and state reset.""" + failed = False + print("**** Testing annual bootstrap ****") + + print("Test: write_minimal_apps_yaml produces a parseable pred_bat config") + with tempfile.TemporaryDirectory() as work_dir: + path = write_minimal_apps_yaml(work_dir, "Europe/London") + if not os.path.exists(path): + print(" ERROR: apps.yaml was not written to {}".format(path)) + failed = True + with open(path, "r") as handle: + parsed = yaml.safe_load(handle) + if "pred_bat" not in parsed: + print(" ERROR: the written apps.yaml has no pred_bat key") + failed = True + else: + section = parsed["pred_bat"] + for key in ["module", "class", "prefix", "timezone", "currency_symbols", "threads"]: + if key not in section: + print(" ERROR: the written apps.yaml is missing '{}'".format(key)) + failed = True + if section.get("threads") != 0: + print(" ERROR: threads must be 0 so plan runs are deterministic, got {}".format(section.get("threads"))) + failed = True + if section.get("timezone") != "Europe/London": + print(" ERROR: the timezone should be written through, got {}".format(section.get("timezone"))) + failed = True + + print("Test: AnnualNullHA satisfies the interface PredBat calls without a Home Assistant") + null_ha = AnnualNullHA() + if null_ha.get_state("sensor.anything", default=7) != 7: + print(" ERROR: get_state should return the supplied default") + failed = True + if null_ha.get_state(None) != {}: + print(" ERROR: get_state with no entity should return an empty mapping of all states") + failed = True + if null_ha.get_history("sensor.anything") is not None: + print(" ERROR: get_history should return None when no history exists") + failed = True + null_ha.set_state("sensor.written", "5", attributes={"unit": "kWh"}) + if null_ha.get_state("sensor.written") != "5": + print(" ERROR: set_state then get_state should round trip") + failed = True + if null_ha.call_service("some/service", value=1) is not None: + print(" ERROR: call_service should be a no-op returning None") + failed = True + + print("Test: apply_hardware maps the battery block onto PredBat's internal units") + battery = {"size_kwh": 9.5, "inverter_kw": 5.0, "export_limit_kw": 3.6, "hybrid": True, "charge_rate_kw": 3.7, "discharge_rate_kw": 4.2} + apply_hardware(my_predbat, battery, [{"kwp": 5.6}]) + if abs(my_predbat.soc_max - 9.5) > 1e-9: + print(" ERROR: soc_max expected 9.5, got {}".format(my_predbat.soc_max)) + failed = True + if abs(my_predbat.inverter_limit - (5.0 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: inverter_limit should be in kW per minute, got {}".format(my_predbat.inverter_limit)) + failed = True + if abs(my_predbat.export_limit - (3.6 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: export_limit should be in kW per minute, got {}".format(my_predbat.export_limit)) + failed = True + if abs(my_predbat.battery_rate_max_charge - (3.7 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: battery_rate_max_charge should be in kW per minute, got {}".format(my_predbat.battery_rate_max_charge)) + failed = True + if abs(my_predbat.battery_rate_max_discharge - (4.2 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: battery_rate_max_discharge should be in kW per minute, got {}".format(my_predbat.battery_rate_max_discharge)) + failed = True + if my_predbat.inverter_hybrid is not True: + print(" ERROR: inverter_hybrid should be True") + failed = True + + print("Test: apply_hardware with no battery produces a zero-capacity system") + apply_hardware(my_predbat, None, [{"kwp": 5.6}]) + if my_predbat.soc_max != 0.0 or my_predbat.soc_kw != 0.0: + print(" ERROR: a battery-less run should have soc_max and soc_kw of 0, got {} / {}".format(my_predbat.soc_max, my_predbat.soc_kw)) + failed = True + + print("Test: reset_sample_state clears every field a previous sample could have left behind") + my_predbat.dynamic_load_baseline = {5: 1.0} + my_predbat.battery_rate_max_export = 99.0 + my_predbat.manual_charge_times = [1, 2, 3] + my_predbat.manual_export_times = [4] + my_predbat.manual_all_times = [5] + my_predbat.cost_today_sofar = 123.0 + my_predbat.import_today_now = 4.0 + my_predbat.export_today_now = 5.0 + my_predbat.iboost_today = 6.0 + my_predbat.carbon_today_sofar = 7.0 + my_predbat.load_minutes_now = 8.0 + my_predbat.pv_today_now = 9.0 + my_predbat.charge_limit_best = [1.0] + my_predbat.charge_window_best = [{"start": 0, "end": 30}] + my_predbat.export_window_best = [{"start": 0, "end": 30}] + my_predbat.export_limits_best = [50.0] + my_predbat.plan_valid = True + + reset_sample_state(my_predbat) + + checks = [ + ("dynamic_load_baseline", {}), + ("battery_rate_max_export", 0.0333), + ("manual_charge_times", []), + ("manual_export_times", []), + ("manual_all_times", []), + ("cost_today_sofar", 0), + ("import_today_now", 0), + ("export_today_now", 0), + ("iboost_today", 0), + ("carbon_today_sofar", 0), + ("load_minutes_now", 0), + ("pv_today_now", 0), + ("charge_limit_best", []), + ("charge_window_best", []), + ("export_window_best", []), + ("export_limits_best", []), + ("plan_valid", False), + ] + for name, expected in checks: + actual = getattr(my_predbat, name) + if actual != expected: + print(" ERROR: reset_sample_state left {} as {}, expected {}".format(name, actual, expected)) + failed = True + + print("Test: reset_sample_state disables the live-system behaviours that make no sense offline") + if my_predbat.octopus_intelligent_charging is not False: + print(" ERROR: octopus_intelligent_charging should be disabled") + failed = True + if my_predbat.load_forecast_only is not True: + print(" ERROR: load_forecast_only must be True so the load profile is taken from load_forecast") + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_bootstrap import test_annual_bootstrap +``` + +```python + ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_bootstrap > /tmp/t8.txt 2>&1; grep -E "ERROR|ImportError|cannot import" /tmp/t8.txt` + +Expected: FAIL with `cannot import name 'AnnualNullHA'`. + +- [ ] **Step 3: Add the bootstrap to `apps/predbat/annual.py`** + +Add to the imports at the top of the file: + +```python +import os + +from const import MINUTE_WATT +``` + +Then append: + +```python +# Minimal apps.yaml for a headless run. PredBat's Hass base class reads this at +# construction time; nothing here talks to Home Assistant. +MINIMAL_APPS_YAML = """pred_bat: + module: predbat + class: PredBat + prefix: predbat + timezone: {timezone} + currency_symbols: + - '£' + - 'p' + threads: 0 + db_enable: false + db_mirror_ha: false + db_primary: false + web_enable: false + mcp_enable: false + notify_devices: [] + days_previous: + - 1 + days_previous_weight: + - 1 + forecast_hours: 48 +""" + +# The default discharge power cap. A leaked full-precision value from a previous +# sample can flip a plan at a decision boundary, so it is reset explicitly. +DEFAULT_BATTERY_RATE_MAX_EXPORT = 0.0333 + + +class AnnualNullHA: + """A no-op Home Assistant interface for headless annual runs. + + Provides the subset of the interface PredBat touches during ``auto_config()``, + ``load_user_config()`` and ``fetch_config_options()``. Nothing is published and + no history exists, which is correct: every input the annual tool needs is + injected directly. + """ + + def __init__(self): + """Create an empty in-memory state store.""" + self.history_enable = False + self.dummy_items = {} + self.service_store_enable = False + self.service_store = [] + self.db_primary = False + + def get_state(self, entity_id, default=None, attribute=None, refresh=False, raw=False): + """Return a stored state, the supplied default, or all states when no entity is given.""" + if not entity_id: + return {} + if entity_id in self.dummy_items: + result = self.dummy_items[entity_id] + if raw: + return result + if isinstance(result, dict): + return result.get(attribute, "") if attribute else result.get("state", default) + return default if attribute else result + return default + + def set_state(self, entity_id, state, attributes=None): + """Store a state locally so subsequent reads round trip.""" + self.dummy_items[entity_id] = state + return state + + def get_history(self, entity_id, now=None, days=30): + """Return None: a headless annual run has no Home Assistant history.""" + return None + + def call_service(self, service, **kwargs): + """Accept and discard a service call.""" + return None + + def get_service_store(self): + """Return and clear the recorded service calls.""" + stored = self.service_store + self.service_store = [] + return stored + + +def write_minimal_apps_yaml(work_dir, timezone): + """Write the headless apps.yaml into ``work_dir`` and return its path.""" + os.makedirs(work_dir, exist_ok=True) + path = os.path.join(work_dir, "apps.yaml") + with open(path, "w") as handle: + handle.write(MINIMAL_APPS_YAML.format(timezone=timezone)) + return path + + +def create_headless_predbat(work_dir, timezone, log): + """Construct a PredBat instance with no Home Assistant connection. + + PredBat's Hass base class reads apps.yaml from ``$PREDBAT_APPS_FILE`` at + construction time, so the environment variable is set before the import-time + construction happens. The predbat import is deliberately local to this + function so merely importing ``annual`` does not drag in the whole engine. + """ + path = write_minimal_apps_yaml(work_dir, timezone) + os.environ["PREDBAT_APPS_FILE"] = path + + import predbat + + instance = predbat.PredBat() + instance.states = {} + instance.reset() + instance.update_time() + instance.ha_interface = AnnualNullHA() + instance.auto_config() + instance.load_user_config() + instance.fetch_config_options() + instance.config_root = work_dir + instance.save_restore_dir = work_dir + instance.args["threads"] = 0 + instance.log = log + return instance + + +def apply_hardware(predbat, battery, solar): + """Map the config's battery block onto the PredBat instance. + + Rates are stored internally as kW per minute, matching + ``Compare.apply_hardware_overrides()``. With no battery block the system is + given zero capacity, which is how the no-battery scenario is expressed. + """ + if battery is None: + predbat.soc_max = 0.0 + predbat.soc_kw = 0.0 + predbat.battery_rate_max_charge = 0.0 + predbat.battery_rate_max_charge_dc = 0.0 + predbat.battery_rate_max_discharge = 0.0 + predbat.battery_rate_max_export = 0.0 + predbat.inverter_limit = (solar[0]["kwp"] if solar else 5.0) * 1000 / MINUTE_WATT + predbat.export_limit = predbat.inverter_limit + predbat.inverter_hybrid = False + return + + predbat.soc_max = battery["size_kwh"] + predbat.soc_kw = min(predbat.soc_kw, predbat.soc_max) + predbat.inverter_limit = battery["inverter_kw"] * 1000 / MINUTE_WATT + predbat.export_limit = battery["export_limit_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_charge = battery["charge_rate_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_charge_dc = predbat.battery_rate_max_charge + predbat.battery_rate_max_discharge = battery["discharge_rate_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_export = predbat.battery_rate_max_discharge + predbat.inverter_hybrid = battery["hybrid"] + + +def reset_sample_state(predbat): + """Reset every field a previous sample could have left behind. + + Without this, a month's result silently depends on what ran before it: the + numbers stay plausible while becoming order-dependent. The list covers the + accumulators, the previous plan, the manual overrides, and the two fields + ``tests/test_single_debug.py`` documents as leaking between debug cases. + """ + predbat.dynamic_load_baseline = {} + predbat.battery_rate_max_export = DEFAULT_BATTERY_RATE_MAX_EXPORT + + predbat.cost_today_sofar = 0 + predbat.carbon_today_sofar = 0 + predbat.iboost_today = 0 + predbat.import_today_now = 0 + predbat.export_today_now = 0 + predbat.load_minutes_now = 0 + predbat.pv_today_now = 0 + + predbat.manual_charge_times = [] + predbat.manual_export_times = [] + predbat.manual_freeze_charge_times = [] + predbat.manual_freeze_export_times = [] + predbat.manual_demand_times = [] + predbat.manual_all_times = [] + + predbat.charge_limit_best = [] + predbat.charge_window_best = [] + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.charge_limit = [] + predbat.charge_window = [] + predbat.export_window = [] + predbat.export_limits = [] + predbat.plan_valid = False + + predbat.octopus_intelligent_charging = False + predbat.load_forecast_only = True + predbat.load_scaling = 1.0 + predbat.load_scaling10 = 1.0 + predbat.load_inday_adjustment = 1.0 + predbat.load_scaling_dynamic = None + predbat.manual_load_adjust = {} + predbat.iboost_enable = False + predbat.carbon_enable = False + predbat.plan_debug = False + predbat.debug_enable = False + predbat.rate_import_replicated = {} + predbat.rate_export_replicated = {} + predbat.savings_last_updated = None +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_bootstrap > /tmp/t8.txt 2>&1; grep -E "ERROR|Traceback|AttributeError" /tmp/t8.txt` + +Expected: no output. If an `AttributeError` names a field that does not exist on `PredBat`, check it against `predbat.py`'s `reset()` and remove or rename it — do not silently `getattr` around it, since a misspelled field is exactly the leak this function exists to prevent. + +- [ ] **Step 5: Verify the headless bootstrap actually constructs** + +`create_headless_predbat` is not covered by the unit test (it constructs a second engine). Smoke-test it once by hand: + +```bash +cd coverage && python3 -c " +import sys +sys.path.insert(0, '../apps/predbat') +from annual import create_headless_predbat +pb = create_headless_predbat('/tmp/annual_work', 'Europe/London', print) +print('soc_max', pb.soc_max, 'forecast_minutes', pb.forecast_minutes) +" > /tmp/t8b.txt 2>&1; tail -20 /tmp/t8b.txt +``` + +Expected: prints a `soc_max` and `forecast_minutes` line with no traceback. If `apps.yaml` is missing a key that `load_user_config()` or `fetch_config_options()` requires, add that key to `MINIMAL_APPS_YAML` and re-run until it constructs cleanly. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual.py apps/predbat/tests/test_annual_bootstrap.py apps/predbat/unit_test.py +git commit -m "feat(annual): add headless PredBat bootstrap and per-sample state reset" +``` + +--- + +## Task 9: Sample selection + +Picks the days to plan. Stratified by irradiance percentile so the answer does not swing on whether the sampled days happened to be sunny, and fully deterministic so the same config always yields the same days. + +**Files:** +- Modify: `apps/predbat/annual.py` +- Create: `apps/predbat/tests/test_annual_sampling.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual_weather.WeatherYear.has_actual`, `daily_actual_kwh`. +- Produces: + - `annual.select_samples(weather, year, month, samples_per_month, has_solar=True) -> list[tuple[date, float]]` — list of `(day, weight_in_days)` pairs, ordered by date + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_sampling.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for annual prediction sample day selection.""" + +from datetime import date, timedelta + +from annual import select_samples + + +class FakeWeather: + """A stub WeatherYear exposing only what select_samples needs.""" + + def __init__(self, daily): + """Hold a mapping of date to daily actual PV kWh.""" + self.daily = daily + + def has_actual(self, day): + """Return True when the date has actual PV data.""" + return day in self.daily + + def daily_actual_kwh(self, day): + """Return the actual PV kWh for the date.""" + return self.daily.get(day, 0.0) + + +def build_january(kwh_by_day, extra_days=1): + """Build a FakeWeather covering January plus a buffer into February.""" + daily = {} + for day_number, kwh in kwh_by_day.items(): + daily[date(2025, 1, day_number)] = kwh + for offset in range(extra_days): + daily[date(2025, 2, 1) + timedelta(days=offset)] = 1.0 + return FakeWeather(daily) + + +def test_annual_sampling(my_predbat): + """Verify percentile sampling, weighting, determinism and degraded months.""" + failed = False + print("**** Testing annual sample selection ****") + + # January: day N generates N kWh, so the sorted order is simply day order + weather = build_january({day: float(day) for day in range(1, 32)}) + + print("Test: two samples land on the 25th and 75th percentile days") + samples = select_samples(weather, 2025, 1, 2) + if len(samples) != 2: + print(" ERROR: expected 2 samples, got {}".format(len(samples))) + failed = True + else: + days = [day.day for day, _ in samples] + # 31 candidates: indices int(31*0.25)=7 and int(31*0.75)=23 -> days 8 and 24 + if days != [8, 24]: + print(" ERROR: expected days [8, 24], got {}".format(days)) + failed = True + + print("Test: weights sum to the number of days in the month") + total_weight = sum(weight for _, weight in samples) + if abs(total_weight - 31.0) > 1e-9: + print(" ERROR: weights should sum to 31, got {}".format(total_weight)) + failed = True + + print("Test: selection is deterministic") + if select_samples(weather, 2025, 1, 2) != samples: + print(" ERROR: repeated selection returned different days") + failed = True + + print("Test: samples are returned in date order") + ordered = [day for day, _ in select_samples(weather, 2025, 1, 4)] + if ordered != sorted(ordered): + print(" ERROR: samples should be returned in date order, got {}".format(ordered)) + failed = True + + print("Test: samples are distinct") + four = select_samples(weather, 2025, 1, 4) + if len({day for day, _ in four}) != 4: + print(" ERROR: expected 4 distinct days, got {}".format([day for day, _ in four])) + failed = True + + print("Test: a day without a following day is excluded, since the 48 hour plan needs one") + truncated = build_january({day: float(day) for day in range(1, 32)}, extra_days=0) + truncated_days = [day for day, _ in select_samples(truncated, 2025, 1, 2)] + if date(2025, 1, 31) in truncated_days: + print(" ERROR: 31 January has no following day and must not be sampled") + failed = True + + print("Test: a month with fewer candidates than samples uses every candidate") + sparse = build_january({1: 5.0, 2: 9.0, 3: 1.0}) + sparse_samples = select_samples(sparse, 2025, 1, 4) + # Day 3 has no following day (4 January is absent), so only days 1 and 2 are usable + if len(sparse_samples) != 2: + print(" ERROR: expected 2 usable samples, got {}".format(len(sparse_samples))) + failed = True + if abs(sum(weight for _, weight in sparse_samples) - 31.0) > 1e-9: + print(" ERROR: weights must still sum to 31 when samples are scarce, got {}".format(sum(w for _, w in sparse_samples))) + failed = True + + print("Test: a month with no usable candidates returns nothing") + if select_samples(FakeWeather({}), 2025, 1, 2) != []: + print(" ERROR: a month with no weather data should return no samples") + failed = True + + print("Test: a battery-only run with no solar falls back to evenly spaced calendar days") + no_solar = select_samples(FakeWeather({}), 2025, 1, 2, has_solar=False) + if len(no_solar) != 2: + print(" ERROR: with no solar, expected 2 calendar samples, got {}".format(len(no_solar))) + failed = True + elif [day.day for day in [entry[0] for entry in no_solar]] != [8, 24]: + print(" ERROR: expected evenly spaced days [8, 24], got {}".format([entry[0].day for entry in no_solar])) + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_sampling import test_annual_sampling +``` + +```python + ("annual_sampling", test_annual_sampling, "Annual prediction sample selection tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_sampling > /tmp/t9.txt 2>&1; grep -E "ERROR|cannot import" /tmp/t9.txt` + +Expected: FAIL with `cannot import name 'select_samples'`. + +- [ ] **Step 3: Add sample selection to `apps/predbat/annual.py`** + +Add `import calendar` and `from datetime import date, timedelta` to the imports (replacing the existing `from datetime import date` line), then append: + +```python +def _percentile_indices(count, samples): + """Return ``samples`` distinct indices spread evenly through ``count`` sorted items. + + Index i sits at percentile (i + 0.5) / samples, so two samples land at the 25th + and 75th percentiles and each represents an equal share of the month. Collisions + are resolved by walking to the nearest unused index, which only matters when the + sample count approaches the number of candidate days. + """ + chosen = [] + used = set() + for index in range(samples): + target = min(count - 1, int(count * ((index + 0.5) / samples))) + while target in used and target < count - 1: + target += 1 + while target in used and target > 0: + target -= 1 + if target in used: + continue + used.add(target) + chosen.append(target) + return chosen + + +def select_samples(weather, year, month, samples_per_month, has_solar=True): + """Choose the days to plan for one month, with the weight in days each represents. + + Days are ranked by their *actual* PV energy and sampled at even percentiles, so an + unlucky sunny or dull draw cannot swing the month. Ranking uses actuals rather than + the forecast: the aim is to represent what the month really contained, not what was + predicted. Days without a following day are excluded because the 48 hour plan needs one. + + Weights always sum to the number of days in the month, so a month with fewer usable + candidates than requested is scaled up rather than silently under-counted. + """ + days_in_month = calendar.monthrange(year, month)[1] + all_days = [date(year, month, day) for day in range(1, days_in_month + 1)] + + if has_solar: + candidates = [day for day in all_days if weather.has_actual(day) and weather.has_actual(day + timedelta(days=1))] + candidates.sort(key=lambda day: (weather.daily_actual_kwh(day), day)) + else: + # With no PV there is nothing to rank by, so fall back to evenly spaced calendar days + candidates = all_days + + if not candidates: + return [] + + indices = _percentile_indices(len(candidates), samples_per_month) + chosen = sorted({candidates[index] for index in indices}) + weight = days_in_month / float(len(chosen)) + return [(day, weight) for day in chosen] +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_sampling > /tmp/t9.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t9.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual.py apps/predbat/tests/test_annual_sampling.py apps/predbat/unit_test.py +git commit -m "feat(annual): add irradiance-stratified sample day selection" +``` + +--- + +## Task 10: Three-scenario execution for one sampled day + +The heart of the tool. Mirrors `calculate_yesterday()` in `apps/predbat/output.py`, which already runs these exact three scenarios against a past day. + +Two things here are easy to get subtly wrong and both have dedicated tests: + +1. **Predbat plans on the forecast series but is costed on the actuals series.** Without the Prediction swap before the final `run_prediction()`, Predbat gets perfect foresight and every result overstates its savings. +2. **Only scenario 3 gets a smart car.** Scenarios 1 and 2 charge on the same fixed off-peak timer, so Predbat is credited only for what it wins over a timer. + +**Files:** +- Modify: `apps/predbat/annual.py` +- Create: `apps/predbat/tests/test_annual_scenarios.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual.reset_sample_state`, `annual.apply_hardware`, `annual_load.build_load_forecast`, `annual_weather.WeatherYear`, `annual_tariff.AnnualTariff`. +- Produces: + - `annual.DAY_MINUTES` (1440), `annual.PLAN_MINUTES` (2880), `annual.START_SOC_KWH` (0.0) + - `annual.build_step_data(predbat, pv_minute, pv_minute10) -> (load_step, pv_step, pv10_step)` + - `annual.timer_charge_window(rate_import, car_kwh, car_rate_kw) -> list[dict]` + - `annual.add_car_to_load(load_forecast, window, car_kwh) -> dict` — returns a new cumulative series with the car's energy inserted + - `annual.prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc)` — injects all per-day state + - `annual.run_day(predbat, config, weather, tariff, load_source, day, midnight_utc) -> dict` with keys `no_pvbat`, `without_predbat`, `with_predbat`, each a dict of `cost_p`, `import_kwh`, `export_kwh`, `pv_generated_kwh`, `battery_throughput_kwh` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_scenarios.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction three-scenario day runner.""" + +from annual import DAY_MINUTES, PLAN_MINUTES, add_car_to_load, timer_charge_window + + +def flat_rates(cheap_start, cheap_end, cheap_rate, peak_rate): + """Build a 48 hour import rate dict with one cheap overnight band per day.""" + rates = {} + for minute in range(PLAN_MINUTES): + in_day = minute % DAY_MINUTES + rates[minute] = cheap_rate if cheap_start <= in_day < cheap_end else peak_rate + return rates + + +def test_annual_scenarios(my_predbat): + """Verify the timer charge window and car load insertion helpers.""" + failed = False + print("**** Testing annual scenario helpers ****") + + print("Test: timer_charge_window finds the cheapest band and sizes it to the car's energy") + rates = flat_rates(cheap_start=30, cheap_end=330, cheap_rate=7.0, peak_rate=30.0) + window = timer_charge_window(rates, car_kwh=14.8, car_rate_kw=7.4) + if not window: + print(" ERROR: expected a charge window") + failed = True + else: + first = window[0] + if first["start"] != 30: + print(" ERROR: the window should start at the cheap band start 30, got {}".format(first["start"])) + failed = True + # 14.8 kWh at 7.4 kW is 2 hours + if first["end"] - first["start"] != 120: + print(" ERROR: expected a 120 minute window for 14.8 kWh at 7.4 kW, got {}".format(first["end"] - first["start"])) + failed = True + + print("Test: a car needing more than the cheap band gets a window extended beyond it") + long_window = timer_charge_window(rates, car_kwh=74.0, car_rate_kw=7.4) + if long_window[0]["end"] - long_window[0]["start"] < 300: + print(" ERROR: a 10 hour charge should extend past the 5 hour cheap band, got {} minutes".format(long_window[0]["end"] - long_window[0]["start"])) + failed = True + + print("Test: zero car energy produces no window") + if timer_charge_window(rates, car_kwh=0.0, car_rate_kw=7.4) != []: + print(" ERROR: no car energy should produce no window") + failed = True + + print("Test: add_car_to_load inserts exactly the car's energy and stays cumulative") + base = {minute: 0.01 * minute for minute in range(PLAN_MINUTES + 1)} + with_car = add_car_to_load(base, window, car_kwh=14.8) + added = with_car[PLAN_MINUTES] - base[PLAN_MINUTES] + if abs(added - 14.8) > 1e-6: + print(" ERROR: expected 14.8 kWh added over the window, got {}".format(added)) + failed = True + for minute in range(1, PLAN_MINUTES + 1): + if with_car[minute] < with_car[minute - 1] - 1e-12: + print(" ERROR: the series stopped being cumulative at minute {}".format(minute)) + failed = True + break + + print("Test: add_car_to_load leaves minutes before the window untouched") + if abs(with_car[10] - base[10]) > 1e-12: + print(" ERROR: minutes before the window must be unchanged") + failed = True + + print("Test: add_car_to_load does not mutate its input") + if abs(base[PLAN_MINUTES] - 0.01 * PLAN_MINUTES) > 1e-9: + print(" ERROR: add_car_to_load must not mutate the input series") + failed = True + + print("Test: the car repeats on the second day of the window") + second_day = [entry for entry in window if entry["start"] >= DAY_MINUTES] + if not second_day: + print(" ERROR: the timer should also charge on day two of the 48 hour window") + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_scenarios import test_annual_scenarios +``` + +```python + ("annual_scenarios", test_annual_scenarios, "Annual prediction scenario helper tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_scenarios > /tmp/t10.txt 2>&1; grep -E "ERROR|cannot import" /tmp/t10.txt` + +Expected: FAIL with `cannot import name 'DAY_MINUTES'`. + +- [ ] **Step 3: Add the scenario runner to `apps/predbat/annual.py`** + +Add to the imports: + +```python +from annual_load import build_load_forecast +from prediction import Prediction +``` + +Then append: + +```python +DAY_MINUTES = 24 * 60 +PLAN_MINUTES = 48 * 60 + +# Every sample starts from an empty battery. The compute_metric correction values +# whatever charge is left at the end, so the starting level does not bias the cost. +START_SOC_KWH = 0.0 + +# Cars are charged at this rate when the config gives no explicit figure +DEFAULT_CAR_RATE_KW = 7.4 + +# Maximum charge slots the dumb-battery baseline is allowed, matching the +# calculate_savings_max_charge_slots convention in calculate_yesterday() +BASELINE_MAX_CHARGE_SLOTS = 1 + + +def build_step_data(predbat, pv_minute, pv_minute10): + """Build the 5-minute step arrays the Prediction engine consumes. + + Mirrors the calls ``calculate_plan()`` makes in ``plan.py``. Because + ``load_forecast_only`` is set, the historical branch of ``step_data_history`` + contributes nothing and the whole load profile comes from ``load_forecast``. + """ + load_step = predbat.step_data_history( + predbat.load_minutes, + predbat.minutes_now, + forward=False, + scale_today=1.0, + scale_fixed=1.0, + type_load=True, + load_forecast=predbat.load_forecast, + load_scaling_dynamic=None, + cloud_factor=None, + load_adjust={}, + load_baseline={}, + ) + pv_step = predbat.step_data_history(pv_minute, predbat.minutes_now, forward=True, cloud_factor=None) + pv10_step = predbat.step_data_history(pv_minute10, predbat.minutes_now, forward=True, cloud_factor=None, flip=True) + return load_step, pv_step, pv10_step + + +def timer_charge_window(rate_import, car_kwh, car_rate_kw): + """Return the fixed off-peak timer windows a non-Predbat household would use. + + Finds the cheapest contiguous band of each day and starts the charge there, + extending past the band if the car needs longer than the cheap rate lasts. + Returns one window per day of the 48 hour plan so the second day matches the first. + """ + if car_kwh <= 0 or car_rate_kw <= 0: + return [] + + minutes_needed = int(round((car_kwh / car_rate_kw) * 60.0)) + if minutes_needed <= 0: + return [] + + windows = [] + for day_offset in range(2): + base = day_offset * DAY_MINUTES + day_rates = {minute: rate_import.get(base + minute, 0.0) for minute in range(DAY_MINUTES)} + if not day_rates: + continue + cheapest = min(day_rates.values()) + # The first minute of the longest run at the cheapest rate + start = None + best_start = 0 + best_length = 0 + for minute in range(DAY_MINUTES + 1): + at_cheapest = minute < DAY_MINUTES and day_rates[minute] <= cheapest + 1e-9 + if at_cheapest and start is None: + start = minute + elif not at_cheapest and start is not None: + if minute - start > best_length: + best_length = minute - start + best_start = start + start = None + windows.append({"start": base + best_start, "end": base + best_start + minutes_needed}) + return windows + + +def add_car_to_load(load_forecast, windows, car_kwh): + """Return a copy of the cumulative load series with the car's energy inserted. + + Used by the two baseline scenarios, where the car is simply extra load in a + fixed timer window rather than something Predbat schedules. + """ + if not windows or car_kwh <= 0: + return dict(load_forecast) + + per_window = car_kwh / float(len(windows)) + additions = {} + for window in windows: + length = max(1, window["end"] - window["start"]) + per_minute = per_window / length + for minute in range(window["start"], window["end"]): + additions[minute] = additions.get(minute, 0.0) + per_minute + + result = {} + running_extra = 0.0 + for minute in sorted(load_forecast.keys()): + result[minute] = load_forecast[minute] + running_extra + running_extra += additions.get(minute, 0.0) + return result + + +def _apply_rates(predbat, rate_import, rate_export): + """Install the day's rates and run the scans the planner depends on.""" + predbat.rate_import = rate_import + predbat.rate_export = rate_export + predbat.rate_low_threshold = 0 + predbat.rate_high_threshold = 0 + + if predbat.rate_import: + predbat.rate_scan(predbat.rate_import, print=False) + predbat.rate_import, predbat.rate_import_replicated = predbat.rate_replicate(predbat.rate_import, is_import=True) + predbat.rate_scan(predbat.rate_import, print=False) + if predbat.rate_export: + predbat.rate_scan_export(predbat.rate_export, print=False) + predbat.rate_export, predbat.rate_export_replicated = predbat.rate_replicate(predbat.rate_export, is_import=False) + predbat.rate_scan_export(predbat.rate_export, print=False) + + predbat.set_rate_thresholds() + + if predbat.rate_export: + predbat.high_export_rates, export_lowest, _ = predbat.rate_scan_window(predbat.rate_export, 5, predbat.rate_export_cost_threshold, True) + if predbat.rate_high_threshold == 0 and export_lowest <= predbat.rate_export_max: + predbat.rate_export_cost_threshold = export_lowest + else: + predbat.high_export_rates = [] + + if predbat.rate_import: + predbat.low_rates, _, highest = predbat.rate_scan_window(predbat.rate_import, 5, predbat.rate_import_cost_threshold, False) + if predbat.rate_low_threshold == 0 and highest >= predbat.rate_min: + predbat.rate_import_cost_threshold = highest + else: + predbat.low_rates = [] + + +def _baseline_charge_window(predbat): + """Return the dumb battery's charge windows: the cheapest static band, charged to full. + + Mirrors the baseline in ``calculate_yesterday()`` — a household without Predbat + that sets a timer for the cheapest rate and charges to 100%. + """ + if not predbat.rate_import or predbat.soc_max <= 0: + return [], [] + day_values = [value for minute, value in predbat.rate_import.items() if minute < DAY_MINUTES] + if not day_values or min(day_values) == max(day_values): + return [], [] + + combine = predbat.combine_charge_slots + predbat.combine_charge_slots = True + windows, _, _ = predbat.rate_scan_window(predbat.rate_import, 5, min(day_values), False, return_raw=True) + predbat.combine_charge_slots = combine + + windows = [window for window in windows if window["start"] < PLAN_MINUTES][:BASELINE_MAX_CHARGE_SLOTS] + return windows, [predbat.soc_max for _ in windows] + + +def _billed_result(predbat, end_record, pv_step): + """Run one scenario to completion and return its billed figures. + + The battery-value correction (metric_end minus metric_start) values whatever + charge is left at the end, so a scenario cannot look cheap simply by finishing + on an empty battery. This is the same correction ``Compare.run_scenario()`` applies. + """ + cost, import_kwh_battery, import_kwh_house, export_kwh, _, final_soc, _, battery_cycle, metric_keep, final_iboost, final_carbon_g = predbat.run_prediction( + predbat.charge_limit_best, predbat.charge_window_best, predbat.export_window_best, predbat.export_limits_best, False, end_record=end_record + ) + metric_start, _ = predbat.compute_metric(end_record, predbat.soc_kw, predbat.soc_kw, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) + metric_end, _ = predbat.compute_metric(end_record, final_soc, final_soc, cost, cost, final_iboost, final_iboost, battery_cycle, metric_keep, final_carbon_g, import_kwh_battery, import_kwh_house, export_kwh) + + pv_generated = sum(value for minute, value in pv_step.items() if minute < end_record) + return { + "cost_p": metric_end - metric_start, + "import_kwh": import_kwh_battery + import_kwh_house, + "export_kwh": export_kwh, + "pv_generated_kwh": pv_generated, + "battery_throughput_kwh": battery_cycle, + } + + +def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc): + """Inject every per-day input into the PredBat instance for one sampled day.""" + reset_sample_state(predbat) + + predbat.midnight_utc = midnight_utc + predbat.now_utc = midnight_utc + predbat.minutes_now = 0 + predbat.forecast_plan_hours = 48 + predbat.forecast_minutes = PLAN_MINUTES + predbat.forecast_days = 2 + predbat.end_record = PLAN_MINUTES + + predbat.load_minutes = {} + predbat.load_minutes_age = 0 + predbat.load_forecast = build_load_forecast(load_source, day, 2) + + rate_import, rate_export = tariff.rates_for(midnight_utc, PLAN_MINUTES) + _apply_rates(predbat, rate_import, rate_export) + + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + + +def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): + """Run all three scenarios against one sampled day and return their billed figures.""" + car_kwh = config["load"].get("car_charging_kwh", 0.0) / 365.0 + car_rate_kw = config["load"].get("car_rate_kw", DEFAULT_CAR_RATE_KW) + + prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc) + + actual_pv = weather.pv_minutes("actual", midnight_utc, PLAN_MINUTES) if config["solar"] else {} + forecast_pv = weather.pv_minutes("forecast", midnight_utc, PLAN_MINUTES) if config["solar"] else {} + p10_pv = weather.pv_minutes_p10(midnight_utc, PLAN_MINUTES, day.month) if config["solar"] else {} + + timer_windows = timer_charge_window(predbat.rate_import, car_kwh, car_rate_kw) + baseline_load = add_car_to_load(predbat.load_forecast, timer_windows, car_kwh) + + results = {} + + # Scenario 1: no PV, no battery. The car still charges on the same timer, so the + # only difference between the scenarios is the system being evaluated. + predbat.num_cars = 0 + predbat.load_forecast = baseline_load + load_step, actual_step, _ = build_step_data(predbat, actual_pv, actual_pv) + zero_step = {minute: 0.0 for minute in actual_step} + predbat.charge_limit_best = [] + predbat.charge_window_best = [] + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.prediction = Prediction(predbat, zero_step, zero_step, load_step, load_step, soc_kw=0, soc_max=0) + results["no_pvbat"] = _billed_result(predbat, DAY_MINUTES, zero_step) + + # Scenario 2: PV and battery on a dumb cheapest-rate timer, no export optimisation + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + charge_window, charge_limit = _baseline_charge_window(predbat) + predbat.charge_window_best = charge_window + predbat.charge_limit_best = charge_limit + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.prediction = Prediction(predbat, actual_step, actual_step, load_step, load_step, soc_kw=START_SOC_KWH) + results["without_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) + + # Scenario 3: Predbat plans on the FORECAST, then is costed against the ACTUALS. + # Skipping the Prediction swap below would hand Predbat perfect foresight. + predbat.load_forecast = build_load_forecast(load_source, day, 2) + if car_kwh > 0: + predbat.num_cars = 1 + predbat.car_charging_planned = [True] + predbat.car_charging_plan_smart = [True] + predbat.car_charging_battery_size = [max(car_kwh * 2, 50.0)] + predbat.car_charging_limit = [car_kwh] + predbat.car_charging_soc = [0.0] + predbat.car_charging_rate = [car_rate_kw] + predbat.car_charging_slots = [[] for _ in range(8)] + predbat.car_charging_from_battery = False + else: + predbat.num_cars = 0 + + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + predbat.pv_forecast_minute = forecast_pv + predbat.pv_forecast_minute10 = p10_pv + predbat.calculate_plan(recompute=True, debug_mode=False, publish=False) + + # Swap in the actuals before costing + forecast_load_step, _, _ = build_step_data(predbat, forecast_pv, p10_pv) + predbat.prediction = Prediction(predbat, actual_step, actual_step, forecast_load_step, forecast_load_step, soc_kw=START_SOC_KWH) + results["with_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) + + return results +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_scenarios > /tmp/t10.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t10.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual.py apps/predbat/tests/test_annual_scenarios.py apps/predbat/unit_test.py +git commit -m "feat(annual): add three-scenario day runner" +``` + +--- + +## Task 11: AnnualPredictor orchestration and results + +Ties everything together and produces the results document the CLI prints and the future web UI will consume. Also carries the three integration tests the spec calls for — scenario ordering, plan-on-forecast/bill-on-actuals, and state isolation — because they can only be checked once a full day can be run. + +**Files:** +- Modify: `apps/predbat/annual.py` +- Modify: `apps/predbat/annual_weather.py` (postcode resolution, so `annual.py` stays HTTP-free) +- Create: `apps/predbat/tests/test_annual_integration.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: everything from Tasks 3-10. +- Produces: + - `annual_weather.resolve_postcode(postcode, fetch_json, log) -> (lat, lon) or None` + - `annual.SCENARIO_KEYS` — `["no_pvbat", "without_predbat", "with_predbat"]` + - `annual.average_rate(rates, minutes) -> float` + - `annual.AnnualPredictor(config, log=None, storage=None, work_dir="./annual_work")` with `async run(progress=None) -> dict` + +- [ ] **Step 1: Add postcode resolution to `apps/predbat/annual_weather.py`** + +Append to that module: + +```python +POSTCODE_URL = "https://api.postcodes.io/postcodes/{}" + + +async def resolve_postcode(postcode, fetch_json, log): + """Resolve a UK postcode to (latitude, longitude), or None when it cannot be resolved.""" + data = await fetch_json(POSTCODE_URL.format(postcode)) + result = (data or {}).get("result", {}) if isinstance(data, dict) else {} + if "latitude" in result and "longitude" in result: + log("Annual: postcode {} resolved to latitude {} longitude {}".format(postcode, result["latitude"], result["longitude"])) + return result["latitude"], result["longitude"] + log("Warn: Annual: postcode {} could not be resolved".format(postcode)) + return None +``` + +- [ ] **Step 2: Write the failing integration test** + +Create `apps/predbat/tests/test_annual_integration.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Integration tests for the annual prediction day runner. + +These run real Predbat plans, so they are registered as slow. +""" + +from datetime import date, datetime, timedelta + +import pytz + +from annual import DAY_MINUTES, PLAN_MINUTES, run_day, validate_config +from annual_load import SyntheticLoadProfile +from tests.test_infra import reset_inverter + +SOLAR_CURVE = [0, 0, 0, 0, 0, 0, 0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.3, 0.1, 0, 0, 0, 0, 0] + + +class StubWeather: + """A WeatherYear stand-in with a fixed daily solar curve and a forecast multiplier.""" + + def __init__(self, peak_kw, forecast_multiplier=1.0, p10_ratio_value=0.8): + """Configure the actual peak power and how much the forecast overstates it.""" + self.peak_kw = peak_kw + self.forecast_multiplier = forecast_multiplier + self.p10_ratio_value = p10_ratio_value + + def _series(self, midnight_utc, minutes, scale): + """Build a per-minute kWh series from the fixed daily curve.""" + result = {} + for minute in range(minutes): + hour = (minute // 60) % 24 + result[minute] = (self.peak_kw * SOLAR_CURVE[hour] * scale) / 60.0 + return result + + def pv_minutes(self, series, midnight_utc, minutes): + """Return the actual or forecast per-minute series.""" + return self._series(midnight_utc, minutes, 1.0 if series == "actual" else self.forecast_multiplier) + + def pv_minutes_p10(self, midnight_utc, minutes, month): + """Return the P10 series, the forecast scaled by the month ratio.""" + return self._series(midnight_utc, minutes, self.forecast_multiplier * self.p10_ratio_value) + + def has_actual(self, day): + """Every day has data in this stub.""" + return True + + def daily_actual_kwh(self, day): + """Return the fixed daily total.""" + return sum(self._series(None, DAY_MINUTES, 1.0).values()) + + def p10_ratio(self, month): + """Return the fixed P10 ratio.""" + return self.p10_ratio_value + + +class StubTariff: + """A tariff stand-in with a cheap overnight band and an expensive evening peak.""" + + def __init__(self, cheap=7.0, normal=28.0, peak=45.0, export=15.0): + """Configure the four rate levels.""" + self.cheap = cheap + self.normal = normal + self.peak = peak + self.export = export + self.standing_charge_p_per_day = 50.0 + + def rates_for(self, midnight_utc, minutes): + """Return (import, export) rate dicts keyed by absolute minute.""" + rate_import = {} + rate_export = {} + for minute in range(minutes): + in_day = minute % DAY_MINUTES + if 30 <= in_day < 330: + rate_import[minute] = self.cheap + elif 16 * 60 <= in_day < 19 * 60: + rate_import[minute] = self.peak + else: + rate_import[minute] = self.normal + rate_export[minute] = self.export + return rate_import, rate_export + + def month_available(self, year, month): + """Always available.""" + return True + + +def make_config(with_car=False): + """Return a validated annual config for the integration tests.""" + raw = { + "location": {"latitude": 51.5, "longitude": -0.1}, + "solar": [{"kwp": 5.0}], + "battery": {"size_kwh": 10.0, "inverter_kw": 5.0}, + "load": {"annual_kwh": 3800, "shape": "flat"}, + "tariff": {"rates_import": [{"rate": 28.0}]}, + "year": 2025, + } + if with_car: + raw["load"]["car_charging_kwh"] = 2500 + return validate_config(raw, today=date(2026, 7, 25)) + + +def run_one(my_predbat, config, weather, day): + """Run all three scenarios for one day and return the results dict.""" + reset_inverter(my_predbat) + midnight = pytz.utc.localize(datetime(day.year, day.month, day.day)) + load_source = SyntheticLoadProfile(annual_kwh=config["load"]["annual_kwh"], shape=config["load"]["shape"], year=config["year"]) + return run_day(my_predbat, config, weather, StubTariff(), load_source, day, midnight) + + +def test_annual_integration(my_predbat): + """Verify scenario ordering, the forecast/actuals split, and state isolation.""" + failed = False + print("**** Testing annual integration ****") + + config = make_config() + weather = StubWeather(peak_kw=4.0) + day = date(2025, 5, 15) + + print("Test: the three scenarios run and produce the expected keys") + results = run_one(my_predbat, config, weather, day) + for key in ["no_pvbat", "without_predbat", "with_predbat"]: + if key not in results: + print(" ERROR: missing scenario '{}'".format(key)) + failed = True + continue + for field in ["cost_p", "import_kwh", "export_kwh", "pv_generated_kwh", "battery_throughput_kwh"]: + if field not in results[key]: + print(" ERROR: scenario '{}' is missing field '{}'".format(key, field)) + failed = True + + print("Test: predbat_cost <= without_predbat_cost <= no_pvbat_cost") + if not failed: + predbat_cost = results["with_predbat"]["cost_p"] + baseline_cost = results["without_predbat"]["cost_p"] + none_cost = results["no_pvbat"]["cost_p"] + if not predbat_cost <= baseline_cost + 1e-6: + print(" ERROR: Predbat cost {} should not exceed the dumb baseline {}".format(predbat_cost, baseline_cost)) + failed = True + if not baseline_cost <= none_cost + 1e-6: + print(" ERROR: the PV/battery baseline {} should not exceed the no-system cost {}".format(baseline_cost, none_cost)) + failed = True + + print("Test: the no-PV/no-battery scenario generates nothing and stores nothing") + if results["no_pvbat"]["pv_generated_kwh"] != 0.0: + print(" ERROR: the no-system scenario should generate no PV, got {}".format(results["no_pvbat"]["pv_generated_kwh"])) + failed = True + if results["no_pvbat"]["battery_throughput_kwh"] != 0.0: + print(" ERROR: the no-system scenario should cycle no battery, got {}".format(results["no_pvbat"]["battery_throughput_kwh"])) + failed = True + + print("Test: Predbat is billed on actuals, not on the forecast it planned against") + # The forecast claims three times the real generation. If the Prediction swap were + # skipped, pv_generated_kwh and the cost would follow the inflated forecast. + inflated = StubWeather(peak_kw=4.0, forecast_multiplier=3.0) + inflated_results = run_one(my_predbat, config, inflated, day) + honest_pv = results["with_predbat"]["pv_generated_kwh"] + inflated_pv = inflated_results["with_predbat"]["pv_generated_kwh"] + if abs(inflated_pv - honest_pv) > 0.01: + print(" ERROR: reported PV should track actuals ({}) regardless of the forecast, got {}".format(honest_pv, inflated_pv)) + failed = True + if inflated_results["with_predbat"]["cost_p"] < results["with_predbat"]["cost_p"] - 1e-6: + print(" ERROR: planning against an over-optimistic forecast must not make the billed cost cheaper") + failed = True + + print("Test: state isolation - a day run in isolation matches the same day run after another") + isolated = run_one(my_predbat, config, weather, day) + _ = run_one(my_predbat, config, StubWeather(peak_kw=1.0), date(2025, 11, 20)) + after_other = run_one(my_predbat, config, weather, day) + for scenario in ["no_pvbat", "without_predbat", "with_predbat"]: + for field in ["cost_p", "import_kwh", "export_kwh", "battery_throughput_kwh"]: + first = isolated[scenario][field] + second = after_other[scenario][field] + if abs(first - second) > 1e-6: + print(" ERROR: {}.{} changed from {} to {} depending on what ran before it".format(scenario, field, first, second)) + failed = True + + print("Test: a car charging config still produces an ordered result") + car_config = make_config(with_car=True) + car_results = run_one(my_predbat, car_config, weather, day) + if car_results["with_predbat"]["cost_p"] > car_results["without_predbat"]["cost_p"] + 1e-6: + print(" ERROR: with a car, Predbat cost {} should not exceed the timer baseline {}".format(car_results["with_predbat"]["cost_p"], car_results["without_predbat"]["cost_p"])) + failed = True + if car_results["no_pvbat"]["import_kwh"] <= results["no_pvbat"]["import_kwh"]: + print(" ERROR: adding a car should raise the no-system import, got {} vs {}".format(car_results["no_pvbat"]["import_kwh"], results["no_pvbat"]["import_kwh"])) + failed = True + + return failed +``` + +- [ ] **Step 3: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_integration import test_annual_integration +``` + +```python + ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), +``` + +Note the trailing `True` — this runs real plans, so it is marked slow and skipped by `--quick`. + +Run: `cd coverage && ./run_all --test annual_integration > /tmp/t11.txt 2>&1; grep -E "ERROR|Traceback|cannot import" /tmp/t11.txt` + +Expected: failures. `run_day` exists from Task 10, so this is a genuine behavioural test — expect to iterate on it. If the scenario-ordering assertion fails, do **not** relax the assertion: it is the property the whole tool exists to demonstrate. Debug the scenario setup instead, comparing against `calculate_yesterday()` in `apps/predbat/output.py`. + +- [ ] **Step 4: Fix whatever the integration test exposes** + +Likely areas, in order of probability: + +1. `_apply_rates()` — if `rate_scan_window` returns no low rates, the dumb baseline never charges and scenario 2 equals scenario 1. Check `predbat.rate_import_cost_threshold` after `set_rate_thresholds()`. +2. `build_step_data()` — if `load_step` is all zeros, confirm `predbat.load_forecast_only` is `True` and that `load_forecast` is cumulative and reaches minute `PLAN_MINUTES`. +3. `calculate_plan()` — if it raises, check `predbat.args["threads"] == 0` so no process pool is created, and that `predbat.end_record` is set. +4. State isolation — if results differ between runs, add the offending field to `reset_sample_state()`. + +- [ ] **Step 5: Add the orchestrator to `apps/predbat/annual.py`** + +Add to the imports. Note `datetime` joins the existing `from datetime import date, timedelta` line, and `pytz` moves to module scope rather than being imported inside `run()`: + +```python +import pytz + +from datetime import date, datetime, timedelta + +from annual_load import OctopusConsumptionLoadProfile, SyntheticLoadProfile +from annual_tariff import AnnualTariff +from annual_weather import AnnualWeather, resolve_postcode +``` + +and delete the `import pytz` line from inside `run()` shown below. + +Then append: + +```python +SCENARIO_KEYS = ["no_pvbat", "without_predbat", "with_predbat"] + +SCENARIO_FIELDS = ["cost_p", "import_kwh", "export_kwh", "pv_generated_kwh", "battery_throughput_kwh"] + + +def average_rate(rates, minutes): + """Return the mean rate across the first ``minutes`` of a rate dict.""" + values = [rates[minute] for minute in range(minutes) if minute in rates] + return (sum(values) / len(values)) if values else 0.0 + + +class AnnualPredictor: + """Projects a year of electricity costs under three scenarios using the Predbat engine.""" + + def __init__(self, config, log=None, storage=None, work_dir="./annual_work"): + """Validate the config and prepare the run.""" + self.log = log or print + self.config = validate_config(config) + self.storage = storage + self.work_dir = work_dir + self.predbat = None + self.weather = None + self.tariff = None + self.load_source = None + self.caveats = [] + + async def _resolve_location(self, weather_fetch): + """Return (latitude, longitude) from the config, resolving a postcode if needed.""" + location = self.config["location"] + if "latitude" in location and "longitude" in location: + return location["latitude"], location["longitude"] + resolved = await resolve_postcode(location["postcode"], weather_fetch, self.log) + if not resolved: + raise AnnualConfigError("annual.location.postcode '{}' could not be resolved; supply latitude and longitude instead".format(location["postcode"])) + return resolved + + async def _build_load_source(self): + """Build the load profile source, falling back to synthetic if Octopus data fails.""" + load_config = self.config["load"] + year = self.config["year"] + + if "octopus" not in load_config: + return SyntheticLoadProfile(annual_kwh=load_config["annual_kwh"], shape=load_config["shape"], year=year) + + # A synthetic profile at the UK average backs the real data so an isolated + # missing day does not silently become zero consumption + fallback = SyntheticLoadProfile(annual_kwh=2700.0, shape="flat", year=year) + source = OctopusConsumptionLoadProfile( + api_key=load_config["octopus"]["api_key"], + account_id=load_config["octopus"]["account_id"], + log=self.log, + storage=self.storage, + fallback=fallback, + ) + if not await source.fetch(year): + raise AnnualConfigError("Octopus consumption data could not be downloaded for {}; check the API key and account id".format(year)) + return source + + def _month_scenarios(self, samples, day_results): + """Weight each sample's daily figures into monthly totals per scenario.""" + totals = {key: {field: 0.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + for (_, weight), result in zip(samples, day_results): + for key in SCENARIO_KEYS: + for field in SCENARIO_FIELDS: + totals[key][field] += result[key][field] * weight + return totals + + async def run(self, progress=None): + """Run the full annual projection and return the results document.""" + year = self.config["year"] + samples_per_month = self.config["samples_per_month"] + has_solar = bool(self.config["solar"]) + + weather_client = AnnualWeather( + self.config["solar"], + latitude=0.0, + longitude=0.0, + log=self.log, + storage=self.storage, + p10_fallback=self.config["pv10_derate_fallback"], + ) + latitude, longitude = await self._resolve_location(weather_client.fetch_json) + weather_client.latitude = latitude + weather_client.longitude = longitude + + self.weather = await weather_client.fetch(year) if has_solar else None + if has_solar and not self.weather.forecast_available: + self.caveats.append("The Open-Meteo forecast archive did not cover {}, so Predbat planned against actuals and P10 used the flat {} derate. Savings are likely overstated.".format(year, self.config["pv10_derate_fallback"])) + elif has_solar and self.weather.fallback_months: + self.caveats.append("Months {} had too few forecast/actual day pairs, so their P10 used the flat {} derate.".format(sorted(self.weather.fallback_months), self.config["pv10_derate_fallback"])) + if has_solar: + self.caveats.append("The forecast-versus-ERA5 gap includes systematic model bias as well as forecast error, so measured solar uncertainty is slightly overstated.") + self.caveats.append("self_consumed_kwh is approximate: when the battery exports grid-charged energy it is understated.") + + self.predbat = create_headless_predbat(self.work_dir, self.config["timezone"], self.log) + self.load_source = await self._build_load_source() + self.tariff = AnnualTariff(self.config["tariff"], log=self.log, predbat=self.predbat, storage=self.storage) + + zone = pytz.timezone(self.config["timezone"]) + months = [] + total_units = 12 + completed = 0 + + for month in range(1, 13): + if progress: + progress(completed, total_units, "Month {:02d}/{}".format(month, year)) + + days_in_month = calendar.monthrange(year, month)[1] + standing_charge_p = self.tariff.standing_charge_p_per_day * days_in_month + + if not await self.tariff.fetch_month(year, month): + months.append({"month": month, "status": "unavailable", "reason": "no rate data available", "days": days_in_month, "standing_charge_p": standing_charge_p}) + completed += 1 + continue + # The 48 hour plan for the last sampled day can spill into the next month + next_year, next_month = (year, month + 1) if month < 12 else (year + 1, 1) + await self.tariff.fetch_month(next_year, next_month) + + samples = select_samples(self.weather, year, month, samples_per_month, has_solar=has_solar) if has_solar else select_samples(None, year, month, samples_per_month, has_solar=False) + if not samples: + months.append({"month": month, "status": "unavailable", "reason": "no usable weather days", "days": days_in_month, "standing_charge_p": standing_charge_p}) + completed += 1 + continue + + day_results = [] + for day, _ in samples: + midnight_utc = zone.localize(datetime(day.year, day.month, day.day)).astimezone(pytz.utc) + day_results.append(run_day(self.predbat, self.config, self.weather, self.tariff, self.load_source, day, midnight_utc)) + + totals = self._month_scenarios(samples, day_results) + first_midnight = zone.localize(datetime(samples[0][0].year, samples[0][0].month, samples[0][0].day)).astimezone(pytz.utc) + _, rate_export = self.tariff.rates_for(first_midnight, DAY_MINUTES) + export_rate = average_rate(rate_export, DAY_MINUTES) + + scenarios = {} + for key in SCENARIO_KEYS: + entry = {field: totals[key][field] for field in SCENARIO_FIELDS} + entry["export_credit_p"] = entry["export_kwh"] * export_rate + entry["self_consumed_kwh"] = max(0.0, entry["pv_generated_kwh"] - entry["export_kwh"]) + scenarios[key] = {name: round(value, 3) for name, value in entry.items()} + + months.append( + { + "month": month, + "status": "ok", + "days": days_in_month, + "sampled_days": [day.isoformat() for day, _ in samples], + "standing_charge_p": round(standing_charge_p, 3), + "scenarios": scenarios, + } + ) + completed += 1 + + if progress: + progress(total_units, total_units, "Complete") + + return self._build_results(months) + + def _build_results(self, months): + """Assemble the final results document from the per-month rows.""" + included = [entry for entry in months if entry["status"] == "ok"] + excluded = [entry["month"] for entry in months if entry["status"] != "ok"] + + annual_scenarios = {} + for key in SCENARIO_KEYS: + annual_scenarios[key] = {field: round(sum(entry["scenarios"][key][field] for entry in included), 3) for field in SCENARIO_FIELDS + ["export_credit_p", "self_consumed_kwh"]} + standing_total = round(sum(entry["standing_charge_p"] for entry in included), 3) + + savings = {} + if annual_scenarios: + savings["pv_battery_vs_none_p"] = round(annual_scenarios["no_pvbat"]["cost_p"] - annual_scenarios["without_predbat"]["cost_p"], 3) + savings["predbat_vs_baseline_p"] = round(annual_scenarios["without_predbat"]["cost_p"] - annual_scenarios["with_predbat"]["cost_p"], 3) + + return { + "year": self.config["year"], + "config": self.config["raw"], + "months": months, + "annual": { + "scenarios": annual_scenarios, + "standing_charge_p": standing_total, + "savings": savings, + "months_included": len(included), + "months_excluded": excluded, + }, + "caveats": self.caveats, + } +``` + +- [ ] **Step 6: Run the integration test and the whole annual suite** + +Run: `cd coverage && ./run_all -k annual > /tmp/t11.txt 2>&1; grep -E "ERROR|Traceback|FAILED" /tmp/t11.txt` + +Expected: no output. + +- [ ] **Step 7: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual.py apps/predbat/annual_weather.py apps/predbat/tests/test_annual_integration.py apps/predbat/unit_test.py +git commit -m "feat(annual): add AnnualPredictor orchestration and results document" +``` + +--- + +## Task 12: Command line interface + +**Files:** +- Create: `apps/predbat/annual_cli.py` +- Create: `apps/predbat/tests/test_annual_cli.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual.AnnualPredictor`. +- Produces: + - `annual_cli.format_table(results, currency="p") -> str` + - `annual_cli.main(argv=None) -> int` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_cli.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction command line output.""" + +from annual_cli import format_table + + +def sample_results(): + """Return a small results document covering an ok month and an unavailable one.""" + scenarios = { + "no_pvbat": {"cost_p": 12000.0, "import_kwh": 400.0, "export_kwh": 0.0, "pv_generated_kwh": 0.0, "battery_throughput_kwh": 0.0, "export_credit_p": 0.0, "self_consumed_kwh": 0.0}, + "without_predbat": {"cost_p": 8000.0, "import_kwh": 300.0, "export_kwh": 20.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 90.0, "export_credit_p": 300.0, "self_consumed_kwh": 100.0}, + "with_predbat": {"cost_p": 6000.0, "import_kwh": 280.0, "export_kwh": 45.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 140.0, "export_credit_p": 675.0, "self_consumed_kwh": 75.0}, + } + return { + "year": 2025, + "config": {}, + "months": [ + {"month": 1, "status": "ok", "days": 31, "sampled_days": ["2025-01-08", "2025-01-24"], "standing_charge_p": 1860.0, "scenarios": scenarios}, + {"month": 2, "status": "unavailable", "reason": "no rate data available", "days": 28, "standing_charge_p": 1680.0}, + ], + "annual": { + "scenarios": scenarios, + "standing_charge_p": 1860.0, + "savings": {"pv_battery_vs_none_p": 4000.0, "predbat_vs_baseline_p": 2000.0}, + "months_included": 1, + "months_excluded": [2], + }, + "caveats": ["An example caveat."], + } + + +def test_annual_cli(my_predbat): + """Verify the table output reports every month, including excluded ones.""" + failed = False + print("**** Testing annual CLI output ****") + + table = format_table(sample_results()) + + print("Test: the table names the year and every scenario") + for fragment in ["2025", "No PV/Battery", "Without Predbat", "With Predbat"]: + if fragment not in table: + print(" ERROR: the table should mention '{}'".format(fragment)) + failed = True + + print("Test: an unavailable month is shown as excluded, never as zero") + if "unavailable" not in table.lower(): + print(" ERROR: the table must state that February was unavailable") + failed = True + if "no rate data available" not in table: + print(" ERROR: the table should state why the month was excluded") + failed = True + + print("Test: annual savings appear") + if "Savings" not in table: + print(" ERROR: the table should include a savings section") + failed = True + + print("Test: caveats are printed rather than buried in the JSON") + if "An example caveat." not in table: + print(" ERROR: caveats must be shown to the user") + failed = True + + print("Test: the excluded-month count is stated alongside the annual totals") + if "1 of 12" not in table: + print(" ERROR: the table should state how many months are included, got:\n{}".format(table)) + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_cli import test_annual_cli +``` + +```python + ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), +``` + +Run: `cd coverage && ./run_all --test annual_cli > /tmp/t12.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/t12.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_cli'`. + +- [ ] **Step 3: Create `apps/predbat/annual_cli.py`** + +```python +#!/usr/bin/env python3 +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Command line entry point for the annual prediction tool. + +Usage: + python3 annual_cli.py --config annual.yaml --out results.json +""" + +import argparse +import asyncio +import calendar +import json +import os +import sys + +import yaml + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from annual import SCENARIO_KEYS, AnnualConfigError, AnnualPredictor # noqa: E402 +from storage import StorageLocalFiles # noqa: E402 + +SCENARIO_LABELS = {"no_pvbat": "No PV/Battery", "without_predbat": "Without Predbat", "with_predbat": "With Predbat"} + + +def format_table(results, currency="p"): + """Render the results document as a human-readable table.""" + lines = [] + lines.append("Annual prediction for {}".format(results["year"])) + lines.append("") + header = "{:<6}".format("Month") + "".join("{:>20}".format(SCENARIO_LABELS[key]) for key in SCENARIO_KEYS) + lines.append(header) + lines.append("-" * len(header)) + + for entry in results["months"]: + name = calendar.month_abbr[entry["month"]] + if entry["status"] != "ok": + lines.append("{:<6}{:>60}".format(name, "unavailable - {}".format(entry.get("reason", "unknown")))) + continue + row = "{:<6}".format(name) + for key in SCENARIO_KEYS: + row += "{:>20}".format("{:.2f}{}".format(entry["scenarios"][key]["cost_p"] / 100.0, currency.upper() if currency == "p" else currency)) + lines.append(row) + + annual = results["annual"] + lines.append("-" * len(header)) + total_row = "{:<6}".format("Year") + for key in SCENARIO_KEYS: + total_row += "{:>20}".format("{:.2f}".format(annual["scenarios"].get(key, {}).get("cost_p", 0.0) / 100.0)) + lines.append(total_row) + lines.append("") + lines.append("Based on {} of 12 months.".format(annual["months_included"])) + if annual["months_excluded"]: + lines.append("Excluded months: {}".format(", ".join(calendar.month_abbr[month] for month in annual["months_excluded"]))) + lines.append("") + lines.append("Savings") + lines.append(" PV and battery vs no system: {:.2f}".format(annual["savings"].get("pv_battery_vs_none_p", 0.0) / 100.0)) + lines.append(" Predbat vs without Predbat: {:.2f}".format(annual["savings"].get("predbat_vs_baseline_p", 0.0) / 100.0)) + lines.append(" Standing charge (all scenarios): {:.2f}".format(annual["standing_charge_p"] / 100.0)) + + if results.get("caveats"): + lines.append("") + lines.append("Caveats") + for caveat in results["caveats"]: + lines.append(" - {}".format(caveat)) + + return "\n".join(lines) + + +def make_progress(quiet): + """Return a progress callback that writes to stderr, or None when quiet.""" + if quiet: + return None + + def progress(completed, total, message): + """Report progress to stderr so stdout stays parseable.""" + sys.stderr.write("[{}/{}] {}\n".format(completed, total, message)) + sys.stderr.flush() + + return progress + + +def main(argv=None): + """Parse arguments, run the projection, and write the results. Returns an exit code.""" + parser = argparse.ArgumentParser(description="Project a year of electricity costs using the Predbat engine") + parser.add_argument("--config", required=True, help="Path to the annual prediction YAML config") + parser.add_argument("--out", default=None, help="Write the results JSON to this path") + parser.add_argument("--work-dir", default="./annual_work", help="Working directory for the headless Predbat instance and cache") + parser.add_argument("--quiet", action="store_true", help="Suppress progress output") + args = parser.parse_args(argv) + + try: + with open(args.config, "r") as handle: + config = yaml.safe_load(handle) + except (OSError, yaml.YAMLError) as error: + sys.stderr.write("Could not read config {}: {}\n".format(args.config, error)) + return 2 + + storage = StorageLocalFiles(args.work_dir, print) + + try: + predictor = AnnualPredictor(config, log=print if not args.quiet else lambda *a, **k: None, storage=storage, work_dir=args.work_dir) + results = asyncio.get_event_loop().run_until_complete(predictor.run(progress=make_progress(args.quiet))) + except AnnualConfigError as error: + sys.stderr.write("Config error: {}\n".format(error)) + return 2 + + if args.out: + with open(args.out, "w") as handle: + json.dump(results, handle, indent=2) + sys.stderr.write("Results written to {}\n".format(args.out)) + + print(format_table(results)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_cli > /tmp/t12.txt 2>&1; grep -E "ERROR|Traceback" /tmp/t12.txt` + +Expected: no output. + +- [ ] **Step 5: Verify the CLI rejects a bad config cleanly** + +```bash +cd coverage && echo "annual: {}" > /tmp/bad_annual.yaml && python3 ../apps/predbat/annual_cli.py --config /tmp/bad_annual.yaml; echo "exit=$?" +``` + +Expected: `Config error: annual.location is required...` and `exit=2`, with no traceback. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +./run_pre_commit +git add apps/predbat/annual_cli.py apps/predbat/tests/test_annual_cli.py apps/predbat/unit_test.py +git commit -m "feat(annual): add annual prediction command line interface" +``` + +--- + +## Task 13: Documentation + +**Files:** +- Create: `docs/annual-prediction.md` +- Modify: `mkdocs.yml` +- Modify: `.cspell/custom-dictionary-workspace.txt` + +- [ ] **Step 1: Write `docs/annual-prediction.md`** + +```markdown +# Annual prediction + +The annual prediction tool projects a year of household electricity costs using the +real Predbat planning engine. For each month it reports three scenarios: + +1. **No PV, no battery** — the counterfactual bill. +2. **PV and battery, without Predbat** — a battery charging on a static cheap-rate timer. +3. **PV and battery, with Predbat** — the optimiser's plan. + +It is a standalone command line tool; it does not need Home Assistant or any hardware. + +## How it works + +For each month the tool picks sample days by irradiance percentile, so the answer does +not swing on whether the sampled days happened to be sunny. Two samples per month is the +default; each represents half the month. + +Each sampled day gets a 48-hour plan starting at midnight, but only the first 24 hours +are billed — the second day exists so the optimiser does not artificially drain the +battery at the horizon. Whatever charge is left at the end is valued, so a scenario +cannot look cheap by finishing empty. + +Predbat plans against the **archived weather forecast** for that date and is costed +against **ERA5 actuals**. This matters: costing against the same series it planned from +would hand Predbat perfect foresight and overstate its savings. + +Solar uncertainty (P10) is derived from measured forecast error — for each month, the +10th percentile of the actual-over-forecast daily energy ratio. + +## Configuration + +```yaml +annual: + location: + postcode: "SW1A 1AA" # or latitude/longitude + year: 2025 # defaults to the most recent complete calendar year + + solar: # omit for a battery-only run + - kwp: 5.6 + declination: 35 # pitch in degrees, default 35 + azimuth: 180 # 180 = south, default 180 + efficiency: 0.95 # default 0.95 + + battery: # omit for a PV-only run + size_kwh: 9.5 + inverter_kw: 5.0 + export_limit_kw: 5.0 # defaults to inverter_kw + hybrid: true # false = AC coupled + charge_rate_kw: 3.6 # defaults to inverter_kw + discharge_rate_kw: 3.6 # defaults to inverter_kw + + load: + annual_kwh: 3800 + shape: flat # night | day | flat + car_charging_kwh: 2500 # annual, 0 to disable + + tariff: + import_octopus_url: "https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{dno_region}/standard-unit-rates/" + export_octopus_url: "..." + dno_region: "A" # required when a URL contains {dno_region} + standing_charge_p_per_day: 60.0 + + samples_per_month: 2 +``` + +Octopus product codes are region-suffixed. If your tariff URL contains `{dno_region}` +you must also set `dno_region` to your region letter (`A` for Eastern England, and so +on) — the tool rejects the config otherwise rather than letting the request 404 and +reporting the month as unavailable. + +Instead of `annual_kwh`, `shape` and `car_charging_kwh` you may supply real consumption: + +```yaml + load: + octopus: + api_key: !secret octopus_key + account_id: A-1234ABCD +``` + +These two forms are **mutually exclusive** and supplying both is rejected. The Octopus +consumption series already includes any EV charging, so accepting both would +double-count it. The trade-off is that a car baked into the meter data cannot be +smart-planned separately. + +Instead of an Octopus URL you may give a fixed rate structure: + +```yaml + tariff: + rates_import: + - start: "00:30:00" + end: "05:30:00" + rate: 7.0 + - start: "05:30:00" + end: "00:30:00" + rate: 28.0 + rates_export: + - rate: 15.0 +``` + +## Running it + +```bash +cd apps/predbat +python3 annual_cli.py --config annual.yaml --out results.json +``` + +A run takes roughly one to three minutes: 24 plan calculations plus the downloads, +which are cached between runs. + +## Limitations + +- The forecast archive only reaches back to about 2021. For earlier years the tool + plans on actuals and falls back to a flat P10 derate, which it states in its output. +- `self_consumed_kwh` is approximate. When the battery exports grid-charged energy it + is understated. +- The forecast-versus-ERA5 gap includes systematic model bias as well as genuine + forecast error, so measured solar uncertainty is slightly overstated. +- A month with no rate data is reported as `unavailable` and excluded from the annual + total, rather than counted as zero. +- Heat pump, iBoost and gas modelling are not included. +``` + +- [ ] **Step 2: Add the nav entry to `mkdocs.yml`** + +Under the `Viewing Predbat data:` section, after `compare.md`: + +```yaml + - annual-prediction.md +``` + +- [ ] **Step 3: Add new words to the CSpell dictionary** + +`docs/*.md` is spell-checked. Run pre-commit first to discover exactly which words it flags, then add only those: + +```bash +./run_pre_commit 2>&1 | grep -i "unknown word" | sort -u +``` + +Likely candidates: `ERA`, `kwp`, `dno`. Append each flagged word to `.cspell/custom-dictionary-workspace.txt`, then run `./run_pre_commit` again — the file is auto-sorted, so re-stage it afterwards. + +- [ ] **Step 4: Verify the docs build** + +```bash +mkdocs build --strict > /tmp/t13.txt 2>&1; grep -iE "error|warning" /tmp/t13.txt +``` + +Expected: no output. If `mkdocs` is not installed, `pip install mkdocs` inside `coverage/venv` first. + +- [ ] **Step 5: Run the full test suite** + +```bash +cd coverage && ./run_all > /tmp/full.txt 2>&1; grep -E "ERROR|FAILED|Traceback" /tmp/full.txt; tail -5 /tmp/full.txt +``` + +Expected: no `ERROR`/`FAILED` lines. This is the last gate — the `solcast.py` refactor in Task 1 touches live forecasting code, so the whole suite must pass, not just the annual tests. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +./run_pre_commit +git add docs/annual-prediction.md mkdocs.yml .cspell/custom-dictionary-workspace.txt +git commit -m "docs: add annual prediction tool documentation" +``` + +--- + +## Notes for the implementer + +**The three assertions that must not be weakened.** If any of these fail, the bug is in +the implementation, not the test: + +1. `predbat_cost <= without_predbat_cost <= no_pvbat_cost` — the property the tool exists + to demonstrate. +2. Reported PV tracks actuals regardless of what the forecast claimed — proves the + Prediction swap in scenario 3 is actually happening. +3. A month run in isolation matches that month within a longer run — proves + `reset_sample_state()` is complete. + +**Where to look when a scenario looks wrong.** `calculate_yesterday()` in +`apps/predbat/output.py` runs these same three scenarios against a real past day and is +the reference implementation. `Compare.run_scenario()` in `apps/predbat/compare.py` is +the reference for the battery-value correction. + +**Do not add a P90 output.** The planner consumes only `pv_forecast_minute` and +`pv_forecast_minute10`; P90 reaches no decision. It was considered and deliberately +excluded. From 4f30e8518c7d1b0bbac7da9572b0d87e51098cc4 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 19:44:50 +0200 Subject: [PATCH 004/119] docs: prove the solar model extraction with the existing open_meteo suite The golden-fixture generator re-implemented the arithmetic it was meant to verify, so a transcription error would have produced a green parity test that proved nothing. tests/test_open_meteo.py already exercises the real download_open_meteo_data() end to end; that is the parity guard, and the task now forbids editing it. The new unit test asserts hand-derived values instead of a snapshot. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-07-25-annual-prediction-tool.md | 195 +++++++++--------- 1 file changed, 102 insertions(+), 93 deletions(-) diff --git a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md index 84d043a9a..919099a6a 100644 --- a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md +++ b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md @@ -52,6 +52,7 @@ - Modify: `apps/predbat/solcast.py` (lines ~20-40 for the constants, ~243-255 for `convert_azimuth`, ~356-408 for the two-pass conversion) - Create: `apps/predbat/tests/test_solar_model.py` - Modify: `apps/predbat/unit_test.py` +- Do **not** modify: `apps/predbat/tests/test_open_meteo.py` — it is the parity guard for this refactor **Interfaces:** - Consumes: nothing (first task). @@ -60,56 +61,17 @@ - `solar_model.convert_azimuth(az) -> float` - `solar_model.gti_hourly_to_period_kwh(times, gti_values, temp_values, wind_values, kwp, system_loss, shading_factors=None, p10_instant=None, p10_fallback=0.7) -> dict[datetime, dict]` where each value is `{"pv_estimate": float, "pv_estimate10": float}` keyed by a tz-aware UTC hour-start stamp, giving kWh generated during that hour for **one** array. -- [ ] **Step 1: Capture a golden snapshot of current behaviour before touching anything** +- [ ] **Step 1: Record the current behaviour baseline** -This is a refactor of live forecasting code. The parity fixture must be generated from the *current* implementation, before any edit, or it proves nothing. +This refactor touches live forecasting code. Parity is proven by the *existing* suite in `apps/predbat/tests/test_open_meteo.py`, which calls the real `download_open_meteo_data()` against mocked responses and asserts on its output, including temperature derating and multi-array summing. Capture its result now, before any edit: -Create `apps/predbat/tests/golden_open_meteo.py` as a throwaway generator and run it from `coverage/`: +Run: `cd coverage && ./run_all -k open_meteo > /tmp/t1_before.txt 2>&1; tail -20 /tmp/t1_before.txt` -```python -"""Throwaway generator for the Open-Meteo conversion golden fixture.""" -import json -import math - -_SAPM_A = -3.47 -_SAPM_B = -0.0594 -_SAPM_DELTA_T = 3.0 - -TIMES = ["2025-06-01T{:02d}:00".format(h) for h in range(24)] -GTI = [0, 0, 0, 0, 12, 88, 210, 355, 495, 610, 700, 755, 770, 742, 668, 560, 425, 275, 130, 30, 0, 0, 0, 0] -TEMP = [9, 9, 8, 8, 9, 11, 13, 15, 17, 19, 20, 21, 22, 22, 22, 21, 20, 18, 16, 14, 12, 11, 10, 10] -WIND = [1.5] * 24 - -def main(): - """Emit the golden fixture as JSON.""" - kwp = 5.6 - system_loss = 0.05 - out = [] - for idx, ts in enumerate(TIMES): - gti = GTI[idx] - t_cell = TEMP[idx] + gti * math.exp(_SAPM_A + _SAPM_B * WIND[idx]) + (gti / 1000.0) * _SAPM_DELTA_T - eta_temp = max(0.5, min(1.1, 1.0 - 0.004 * (t_cell - 25.0))) - pv50 = round((gti / 1000.0) * kwp * eta_temp * (1.0 - system_loss), 4) - out.append(pv50) - periods = [] - for i in range(len(out) - 1): - pv50 = round(0.5 * (out[i] + out[i + 1]), 4) - periods.append({"time": TIMES[i], "pv_estimate": pv50, "pv_estimate10": round(min(pv50 * 0.7, pv50), 4)}) - print(json.dumps(periods, indent=2)) - -if __name__ == "__main__": - main() -``` - -Run: `cd coverage && python3 ../apps/predbat/tests/golden_open_meteo.py > tests_golden.json` - -Copy the output to `apps/predbat/tests/fixtures/open_meteo_golden.json` (create the `fixtures` directory), then delete `golden_open_meteo.py` and `coverage/tests_golden.json`. - -This mirrors the exact arithmetic in `solcast.py:356-396` — instantaneous kW per sample, then trapezoidal integration across each hour pair, then the `pv50 * 0.7` P10 fallback used when no ensemble data exists. +Expected: the suite passes. Record which tests ran — Step 8 re-runs exactly these and they must still pass, unchanged. Do **not** modify `test_open_meteo.py` at any point in this task; a refactor that requires editing the test that guards it is not a refactor. - [ ] **Step 2: Write the failing test** -Create `apps/predbat/tests/test_solar_model.py`: +Create `apps/predbat/tests/test_solar_model.py`. The expected values are hand-derived from the documented model rather than snapshotted, so a wrong constant fails the test instead of being baked into a fixture: ```python # ----------------------------------------------------------------------------- @@ -123,82 +85,129 @@ Create `apps/predbat/tests/test_solar_model.py`: """Tests for the shared solar GTI to kW conversion model.""" -import json -import os from datetime import datetime import pytz from solar_model import convert_azimuth, gti_hourly_to_period_kwh, pvwatts_cell_temperature -FIXTURE_DIR = os.path.join(os.path.dirname(__file__), "fixtures") +FLAT_TIMES = ["2025-06-01T{:02d}:00".format(hour) for hour in range(4)] -TIMES = ["2025-06-01T{:02d}:00".format(h) for h in range(24)] -GTI = [0, 0, 0, 0, 12, 88, 210, 355, 495, 610, 700, 755, 770, 742, 668, 560, 425, 275, 130, 30, 0, 0, 0, 0] -TEMP = [9, 9, 8, 8, 9, 11, 13, 15, 17, 19, 20, 21, 22, 22, 22, 21, 20, 18, 16, 14, 12, 11, 10, 10] -WIND = [1.5] * 24 + +def stamp_for(text): + """Return the tz-aware UTC datetime for an Open-Meteo timestamp string.""" + return pytz.utc.localize(datetime.strptime(text, "%Y-%m-%dT%H:%M")) def test_solar_model(my_predbat): - """Verify the extracted solar model matches the golden snapshot of the original solcast.py logic.""" + """Verify the shared solar model against hand-derived values.""" failed = False print("**** Testing solar_model ****") - print("Test: convert_azimuth round trips the Predbat/Open-Meteo conventions") + print("Test: convert_azimuth maps the Predbat convention onto the Open-Meteo one") for predbat_az, expected in [(180, 0), (90, 90), (270, -90), (0, 180)]: result = convert_azimuth(predbat_az) if result != expected: print(" ERROR: convert_azimuth({}) expected {}, got {}".format(predbat_az, expected, result)) failed = True - print("Test: pvwatts_cell_temperature raises cell temperature above ambient under irradiance") - cool = pvwatts_cell_temperature(0.0, 20.0, 1.5) - hot = pvwatts_cell_temperature(800.0, 20.0, 1.5) - if cool != 20.0: - print(" ERROR: zero irradiance should give ambient, got {}".format(cool)) + print("Test: pvwatts_cell_temperature matches the SAPM formula") + # T_cell = 25 + 1000*exp(-3.47 + -0.0594*0) + (1000/1000)*3.0 + # = 25 + 1000*0.031117 + 3 = 59.117 + hot = pvwatts_cell_temperature(1000.0, 25.0, 0.0) + if abs(hot - 59.117) > 0.001: + print(" ERROR: cell temperature expected 59.117, got {}".format(hot)) + failed = True + if pvwatts_cell_temperature(0.0, 20.0, 1.5) != 20.0: + print(" ERROR: zero irradiance should give ambient temperature") + failed = True + + print("Test: a constant-irradiance hour converts to the hand-derived energy") + # eta = 1 - 0.004*(59.117 - 25) = 0.863532; pv = (1000/1000) * 1 kWp * eta * 1.0 + # Both endpoints are equal so the trapezoid returns the same value. + flat_gti = [1000.0] * 4 + flat_temp = [25.0] * 4 + flat_wind = [0.0] * 4 + result = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0) + if len(result) != 3: + print(" ERROR: 4 samples should yield 3 integrated periods, got {}".format(len(result))) + failed = True + first = result.get(stamp_for(FLAT_TIMES[0])) + if first is None: + print(" ERROR: missing the first period") failed = True - if hot <= cool: - print(" ERROR: 800 W/m2 should raise cell temperature above ambient, got {}".format(hot)) + elif abs(first["pv_estimate"] - 0.8635) > 0.0001: + print(" ERROR: expected 0.8635 kWh, got {}".format(first["pv_estimate"])) failed = True - print("Test: gti_hourly_to_period_kwh matches the golden snapshot") - with open(os.path.join(FIXTURE_DIR, "open_meteo_golden.json"), "r") as handle: - golden = json.load(handle) + print("Test: cold panels are allowed to exceed their STC rating") + # T_cell = 0 + 200*exp(-3.47 - 0.0594) + 0.6 = 6.4645; eta = 1.074142 (above 1.0) + cold = gti_hourly_to_period_kwh(FLAT_TIMES, [200.0] * 4, [0.0] * 4, [1.0] * 4, kwp=1.0, system_loss=0.0) + cold_first = cold[stamp_for(FLAT_TIMES[0])] + if abs(cold_first["pv_estimate"] - 0.2148) > 0.0001: + print(" ERROR: expected 0.2148 kWh for cold panels, got {}".format(cold_first["pv_estimate"])) + failed = True - result = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05) + print("Test: the trapezoid integrates a rising ramp to the mean of its endpoints") + ramp = gti_hourly_to_period_kwh(FLAT_TIMES, [0.0, 1000.0, 1000.0, 0.0], [25.0] * 4, [0.0] * 4, kwp=1.0, system_loss=0.0) + # Endpoints 0.0 and 0.8635 average to 0.43175, rounded to 4 places + if abs(ramp[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - 0.4318) > 0.0001: + print(" ERROR: expected 0.4318 kWh across the sunrise hour, got {}".format(ramp[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True - if len(result) != len(golden): - print(" ERROR: expected {} periods, got {}".format(len(golden), len(result))) + print("Test: zero irradiance produces zero energy") + dark = gti_hourly_to_period_kwh(FLAT_TIMES, [0.0] * 4, [15.0] * 4, [1.0] * 4, kwp=5.0, system_loss=0.05) + if any(entry["pv_estimate"] != 0.0 for entry in dark.values()): + print(" ERROR: zero irradiance should give zero energy, got {}".format(dark)) failed = True - for entry in golden: - stamp = pytz.utc.localize(datetime.strptime(entry["time"], "%Y-%m-%dT%H:%M")) - got = result.get(stamp) - if got is None: - print(" ERROR: missing period {}".format(entry["time"])) - failed = True - continue - if abs(got["pv_estimate"] - entry["pv_estimate"]) > 0.0001: - print(" ERROR: {} pv_estimate expected {}, got {}".format(entry["time"], entry["pv_estimate"], got["pv_estimate"])) - failed = True - if abs(got["pv_estimate10"] - entry["pv_estimate10"]) > 0.0001: - print(" ERROR: {} pv_estimate10 expected {}, got {}".format(entry["time"], entry["pv_estimate10"], got["pv_estimate10"])) - failed = True + print("Test: system_loss and kwp scale the output linearly") + scaled = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=2.0, system_loss=0.5) + if abs(scaled[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - first["pv_estimate"]) > 0.0001: + print(" ERROR: doubling kwp and halving efficiency should cancel out, got {}".format(scaled[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True - print("Test: p10_fallback scales the P10 series") - scaled = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05, p10_fallback=0.5) - stamp = pytz.utc.localize(datetime.strptime("2025-06-01T12:00", "%Y-%m-%dT%H:%M")) - expected_p10 = round(scaled[stamp]["pv_estimate"] * 0.5, 4) - if abs(scaled[stamp]["pv_estimate10"] - expected_p10) > 0.0001: - print(" ERROR: p10_fallback 0.5 expected {}, got {}".format(expected_p10, scaled[stamp]["pv_estimate10"])) + print("Test: p10_fallback scales the P10 series and defaults to 0.7") + if abs(first["pv_estimate10"] - round(first["pv_estimate"] * 0.7, 4)) > 0.0001: + print(" ERROR: the default P10 fallback should be 0.7, got {}".format(first["pv_estimate10"])) + failed = True + half = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_fallback=0.5) + half_first = half[stamp_for(FLAT_TIMES[0])] + if abs(half_first["pv_estimate10"] - round(half_first["pv_estimate"] * 0.5, 4)) > 0.0001: + print(" ERROR: p10_fallback 0.5 not applied, got {}".format(half_first["pv_estimate10"])) + failed = True + + print("Test: p10_instant overrides the fallback and is capped at P50") + ensemble = {FLAT_TIMES[index]: 0.1 for index in range(4)} + with_ensemble = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_instant=ensemble) + ensemble_first = with_ensemble[stamp_for(FLAT_TIMES[0])] + if ensemble_first["pv_estimate10"] >= ensemble_first["pv_estimate"]: + print(" ERROR: an ensemble P10 below P50 should stay below it, got {}".format(ensemble_first["pv_estimate10"])) + failed = True + huge = {FLAT_TIMES[index]: 99.0 for index in range(4)} + capped = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_instant=huge) + capped_first = capped[stamp_for(FLAT_TIMES[0])] + if abs(capped_first["pv_estimate10"] - capped_first["pv_estimate"]) > 0.0001: + print(" ERROR: an ensemble P10 above P50 should be capped at P50, got {}".format(capped_first["pv_estimate10"])) failed = True print("Test: shading_factors apply the correct month") - shading = [0.5] * 12 - shaded = gti_hourly_to_period_kwh(TIMES, GTI, TEMP, WIND, kwp=5.6, system_loss=0.05, shading_factors=shading) - unshaded = result[stamp]["pv_estimate"] - if abs(shaded[stamp]["pv_estimate"] - round(unshaded * 0.5, 4)) > 0.0001: - print(" ERROR: shading 0.5 expected {}, got {}".format(round(unshaded * 0.5, 4), shaded[stamp]["pv_estimate"])) + shaded = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, shading_factors=[0.5] * 12) + if abs(shaded[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - round(first["pv_estimate"] * 0.5, 4)) > 0.0001: + print(" ERROR: a 0.5 shading factor was not applied, got {}".format(shaded[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True + + print("Test: a gap in the timestamps is not integrated across") + gapped_times = ["2025-06-01T00:00", "2025-06-01T01:00", "2025-06-01T05:00"] + gapped = gti_hourly_to_period_kwh(gapped_times, [1000.0] * 3, [25.0] * 3, [0.0] * 3, kwp=1.0, system_loss=0.0) + if len(gapped) != 1: + print(" ERROR: only the contiguous hour pair should integrate, got {} periods".format(len(gapped))) + failed = True + + print("Test: a None irradiance sample is treated as zero rather than raising") + with_none = gti_hourly_to_period_kwh(FLAT_TIMES, [None, 1000.0, 1000.0, None], [25.0] * 4, [0.0] * 4, kwp=1.0, system_loss=0.0) + if abs(with_none[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - 0.4318) > 0.0001: + print(" ERROR: a None sample should behave as zero, got {}".format(with_none[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) failed = True return failed @@ -403,15 +412,15 @@ Note `math` may now be unused in `solcast.py` — check before removing the impo - [ ] **Step 8: Run the full existing solar test suites to confirm no regression** -Run: `cd coverage && ./run_all -k solcast > /tmp/t1b.txt 2>&1; ./run_all -k pv_forecast >> /tmp/t1b.txt 2>&1; ./run_all --test open_meteo >> /tmp/t1b.txt 2>&1; grep -E "ERROR|FAILED|Traceback" /tmp/t1b.txt` +Run: `cd coverage && ./run_all -k open_meteo > /tmp/t1b.txt 2>&1; ./run_all -k solcast >> /tmp/t1b.txt 2>&1; ./run_all -k pv_forecast >> /tmp/t1b.txt 2>&1; grep -E "ERROR|FAILED|Traceback" /tmp/t1b.txt` -Expected: no output from the grep. If `--test open_meteo` reports an unknown name, run `./run_all --list` and use the actual registered name for `tests/test_open_meteo.py`. +Expected: no output from the grep, and the `open_meteo` tests must pass exactly as they did in Step 1 with `test_open_meteo.py` unedited. That suite calls the real `download_open_meteo_data()` end to end and is the parity guarantee for this whole refactor — if it fails, the extraction changed behaviour and the extraction is wrong, not the test. - [ ] **Step 9: Run pre-commit and commit** ```bash ./run_pre_commit -git add apps/predbat/solar_model.py apps/predbat/solcast.py apps/predbat/unit_test.py apps/predbat/tests/test_solar_model.py apps/predbat/tests/fixtures/open_meteo_golden.json +git add apps/predbat/solar_model.py apps/predbat/solcast.py apps/predbat/unit_test.py apps/predbat/tests/test_solar_model.py git commit -m "refactor(solar): extract the shared GTI to kW conversion model" ``` From 20abc1c3d82a1d4fc5bb75bfe7a706c09b0ad1d8 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 19:53:37 +0200 Subject: [PATCH 005/119] refactor(solar): extract the shared GTI to kW conversion model Moves the Open-Meteo cell-temperature derate and trapezoidal hourly integration out of solcast.py into solar_model.py so the upcoming annual prediction tool can reuse the identical conversion instead of duplicating it. Also updates the two call sites (test_solcast.py and the standalone open_meteo_live.py script) that referenced the removed SolarAPI.convert_azimuth instance method, and re-exports pvwatts_cell_temperature from solcast.py for test_open_meteo.py's existing import. --- apps/predbat/solar_model.py | 134 ++++++++++++++++++++++++ apps/predbat/solcast.py | 100 +++--------------- apps/predbat/tests/open_meteo_live.py | 4 +- apps/predbat/tests/test_solar_model.py | 137 +++++++++++++++++++++++++ apps/predbat/tests/test_solcast.py | 5 +- apps/predbat/unit_test.py | 2 + 6 files changed, 295 insertions(+), 87 deletions(-) create mode 100644 apps/predbat/solar_model.py create mode 100644 apps/predbat/tests/test_solar_model.py diff --git a/apps/predbat/solar_model.py b/apps/predbat/solar_model.py new file mode 100644 index 000000000..9b39508af --- /dev/null +++ b/apps/predbat/solar_model.py @@ -0,0 +1,134 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Shared photovoltaic conversion model. + +Converts Open-Meteo global tilted irradiance (GTI) into PV energy, applying a +SAPM/PVWatts cell-temperature derate and integrating each hourly sample pair. +Shared by the live Solcast/Open-Meteo forecast path and the annual prediction +tool so the two cannot drift apart. +""" + +import math +from datetime import datetime, timedelta + +import pytz + +from utils import dp4 + +# PVWatts / SAPM cell temperature model constants (glass/glass, open rack) +# Equivalent to pvlib.temperature.sapm_cell with open_rack_glass_glass parameters +_SAPM_A = -3.47 +_SAPM_B = -0.0594 +_SAPM_DELTA_T = 3.0 + +# c-Si temperature coefficient: -0.4%/degC relative to STC (25degC) +_TEMP_COEFF = 0.004 +_STC_TEMP_C = 25.0 + +# Defaults used when a sample has no measured value +_DEFAULT_TEMP_C = 25.0 +_DEFAULT_WIND_MS = 1.0 + +# Applied when no ensemble P10 data is available +_DEFAULT_P10_FALLBACK = 0.7 + + +def pvwatts_cell_temperature(poa_global, temp_air, wind_speed): + """Compute PV cell temperature using the SAPM (PVWatts) model. + + Parameters correspond to a glass/glass module on an open rack (the most + common residential case). Formula: T_cell = T_air + GTI*exp(a + b*wind) + (GTI/1000)*deltaT + """ + return temp_air + poa_global * math.exp(_SAPM_A + _SAPM_B * wind_speed) + (poa_global / 1000.0) * _SAPM_DELTA_T + + +def convert_azimuth(az): + """ + Convert azimuth from Predbat/Solcast convention to Forecast.solar/Open-Meteo convention. + Predbat/Solcast convention: 0 = North, -90 = East, 90 = West, 180 = South + Forecast.solar/Open-Meteo convention: 0 = South, -90 = East, 90 = West, +/-180 = North + """ + if az >= 0: + az = 180 - az + else: + az = -180 - az + + return az + + +def _temperature_efficiency(gti, temp, wind): + """Return the cell-temperature efficiency multiplier for one irradiance sample.""" + t_cell = pvwatts_cell_temperature(gti, temp, wind) + # No lower clamp on (t_cell - 25): cool cells genuinely produce more power. + # Cap at 1.1 (10% above STC) to prevent unrealistic gains at very cold temperatures. + return max(0.5, min(1.1, 1.0 - _TEMP_COEFF * (t_cell - _STC_TEMP_C))) + + +def gti_hourly_to_period_kwh(times, gti_values, temp_values, wind_values, kwp, system_loss, shading_factors=None, p10_instant=None, p10_fallback=_DEFAULT_P10_FALLBACK): + """Convert hourly GTI samples into per-hour PV energy for a single array. + + Open-Meteo returns point-in-time irradiance (W/m2) at the start of each hour, so the + samples are integrated trapezoidally across each adjacent pair rather than treated as + period energy. + + Args: + times: list of ISO timestamp strings, "%Y-%m-%dT%H:%M", assumed UTC + gti_values: list of global tilted irradiance values in W/m2, aligned to times + temp_values: list of air temperatures in degC, aligned to times + wind_values: list of wind speeds in m/s, aligned to times + kwp: array peak power in kW + system_loss: fractional system loss, e.g. 0.05 for 95% efficiency + shading_factors: optional list of 12 per-month multipliers + p10_instant: optional dict of timestamp string to raw P10 kW, before temperature derate + p10_fallback: multiplier applied to P50 when p10_instant has no entry + + Returns: + dict of tz-aware UTC hour-start datetime to {"pv_estimate": kWh, "pv_estimate10": kWh} + """ + instant_kw = {} + instant_stamps = [] + + for idx, ts in enumerate(times): + if idx >= len(gti_values): + break + gti = gti_values[idx] + if gti is None: + gti = 0.0 + temp = temp_values[idx] if idx < len(temp_values) and temp_values[idx] is not None else _DEFAULT_TEMP_C + wind = wind_values[idx] if idx < len(wind_values) and wind_values[idx] is not None else _DEFAULT_WIND_MS + eta_temp = _temperature_efficiency(gti, temp, wind) + pv50_inst = dp4((gti / 1000.0) * kwp * eta_temp * (1.0 - system_loss)) + raw_p10 = p10_instant.get(ts) if p10_instant else None + # p10_instant was computed without temperature derating; apply eta_temp now + pv10_inst = dp4(min(raw_p10 * eta_temp, pv50_inst) if raw_p10 is not None else pv50_inst * p10_fallback) + try: + stamp = datetime.strptime(ts, "%Y-%m-%dT%H:%M") + stamp = stamp.replace(tzinfo=pytz.utc) + except (ValueError, TypeError): + continue + instant_kw[stamp] = (pv50_inst, pv10_inst) + instant_stamps.append(stamp) + + period_data = {} + for i in range(len(instant_stamps) - 1): + stamp = instant_stamps[i] + next_stamp = instant_stamps[i + 1] + if (next_stamp - stamp) != timedelta(hours=1): + continue + pv50_start, pv10_start = instant_kw[stamp] + pv50_end, pv10_end = instant_kw[next_stamp] + pv50 = dp4(0.5 * (pv50_start + pv50_end)) + pv10 = dp4(0.5 * (pv10_start + pv10_end)) + + if shading_factors and len(shading_factors) == 12: + shading_month = shading_factors[stamp.month - 1] + pv50 = dp4(pv50 * shading_month) + pv10 = dp4(pv10 * shading_month) + + period_data[stamp] = {"pv_estimate": pv50, "pv_estimate10": pv10} + + return period_data diff --git a/apps/predbat/solcast.py b/apps/predbat/solcast.py index 8d8327976..cae9233f0 100644 --- a/apps/predbat/solcast.py +++ b/apps/predbat/solcast.py @@ -23,26 +23,11 @@ import pytz from datetime import datetime, timedelta, timezone -# PVWatts / SAPM cell temperature model constants (glass/glass, open rack) -# Equivalent to pvlib.temperature.sapm_cell with open_rack_glass_glass parameters -_SAPM_A = -3.47 -_SAPM_B = -0.0594 -_SAPM_DELTA_T = 3.0 - - -def pvwatts_cell_temperature(poa_global, temp_air, wind_speed): - """Compute PV cell temperature using the SAPM (PVWatts) model. - - Parameters correspond to a glass/glass module on an open rack (the most - common residential case). Formula: T_cell = T_air + GTI*exp(a + b*wind) + (GTI/1000)*deltaT - """ - return temp_air + poa_global * math.exp(_SAPM_A + _SAPM_B * wind_speed) + (poa_global / 1000.0) * _SAPM_DELTA_T - - from const import TIME_FORMAT, TIME_FORMAT_SOLCAST from utils import dp2, dp4, history_attribute_to_minute_data, minute_data, history_attribute, prune_today from predbat_metrics import record_api_call, metrics from component_base import ComponentBase +from solar_model import convert_azimuth, gti_hourly_to_period_kwh, pvwatts_cell_temperature # noqa: F401 - re-exported for tests/test_open_meteo.py parity checks """ Solcast class deals with fetching solar predictions, processing the data and publishing the results. @@ -240,19 +225,6 @@ async def cache_get_url(self, url, params, max_age=8 * 60): URL_PERSONAL = "https://api.forecast.solar/{api_key}/estimate/{lat}/{lon}/{dec}/{az}/{kwp}?time=utc" URL_PERSONAL_DUAL = "https://api.forecast.solar/{api_key}/estimate/{lat}/{lon}/{dec1}/{az1}/{kwp1}/{dec2}/{az2}/{kwp2}?time=utc" - def convert_azimuth(self, az): - """ - Convert azimuth from Predbat/Solcast convention to Forecast.solar/Open-Meteo convention. - Predbat/Solcast convention: 0 = North, -90 = East, 90 = West, 180 = South - Forecast.solar/Open-Meteo convention: 0 = South, -90 = East, 90 = West, ±180 = North - """ - if az >= 0: - az = 180 - az - else: - az = -180 - az - - return az - async def download_open_meteo_ensemble_data(self, lat, lon, tilt, az, kwp, system_loss): """ Download Open-Meteo ensemble data for P10 solar estimate. @@ -309,7 +281,7 @@ async def download_open_meteo_data(self, configs=None): tilt = config.get("declination", 35.0) az = config.get("azimuth", 180.0) if not config.get("azimuth_zero_south", False): - az = self.convert_azimuth(az) + az = convert_azimuth(az) kwp = config.get("kwp", 3.0) system_loss = 1.0 - config.get("efficiency", 0.95) shading_factors = config.get("shading_factors", None) @@ -350,62 +322,24 @@ async def download_open_meteo_data(self, configs=None): ensemble_p10 = await self.download_open_meteo_ensemble_data(lat, lon, tilt, az, kwp, system_loss) - # Pass 1: compute instantaneous kW at each UTC timestamp sample. - # Open-Meteo returns point-in-time irradiance (W/m²) at the start of each hour, - # so we must integrate over the period rather than treating the sample as the period energy. - instant_kw = {} # datetime stamp -> (pv50_kw, pv10_kw) - instant_stamps = [] - for idx, ts in enumerate(times): - if idx >= len(gti_values): - break - gti = gti_values[idx] - if gti is None: - gti = 0.0 - temp = temp_values[idx] if idx < len(temp_values) and temp_values[idx] is not None else 25.0 - wind = wind_values[idx] if idx < len(wind_values) and wind_values[idx] is not None else 1.0 - # Cell temperature via SAPM/PVWatts model: irradiance heats the cell above ambient - t_cell = pvwatts_cell_temperature(gti, temp, wind) - # c-Si temperature coefficient: -0.4%/°C relative to STC (25°C) - # No lower clamp on (t_cell - 25): cool cells genuinely produce more power. - # Cap at 1.1 (10% above STC) to prevent unrealistic gains at very cold temperatures. - eta_temp = max(0.5, min(1.1, 1.0 - 0.004 * (t_cell - 25.0))) - pv50_inst = dp4((gti / 1000.0) * kwp * eta_temp * (1.0 - system_loss)) - raw_p10 = ensemble_p10.get(ts) - # ensemble_p10 was computed without temperature derating; apply eta_temp now - pv10_inst = dp4(min(raw_p10 * eta_temp, pv50_inst) if raw_p10 is not None else pv50_inst * 0.7) - try: - stamp = datetime.strptime(ts, "%Y-%m-%dT%H:%M") - stamp = stamp.replace(tzinfo=pytz.utc) - except (ValueError, TypeError): - continue - instant_kw[stamp] = (pv50_inst, pv10_inst) - instant_stamps.append(stamp) - - # Pass 2: trapezoidal integration — energy over [T, T+1h] = 0.5*(kW_at_T + kW_at_T+1h). - # This correctly accounts for sunrise/sunset transitions where irradiance changes rapidly - # within the hour, e.g. the first post-sunrise hour contains only partial sunshine. - for i in range(len(instant_stamps) - 1): - stamp = instant_stamps[i] - next_stamp = instant_stamps[i + 1] - if (next_stamp - stamp) != timedelta(hours=1): - continue - pv50_start, pv10_start = instant_kw[stamp] - pv50_end, pv10_end = instant_kw[next_stamp] - pv50 = dp4(0.5 * (pv50_start + pv50_end)) - pv10 = dp4(0.5 * (pv10_start + pv10_end)) - - # Apply per-month site shading correction from Google Solar API if available - if shading_factors and len(shading_factors) == 12: - shading_month = shading_factors[stamp.month - 1] - pv50 = dp4(pv50 * shading_month) - pv10 = dp4(pv10 * shading_month) - - data_item = {"period_start": stamp.strftime(TIME_FORMAT), "pv_estimate": pv50, "pv_estimate10": pv10} + array_periods = gti_hourly_to_period_kwh( + times, + gti_values, + temp_values, + wind_values, + kwp=kwp, + system_loss=system_loss, + shading_factors=shading_factors, + p10_instant=ensemble_p10, + ) + for stamp, values in array_periods.items(): + pv50 = values["pv_estimate"] + pv10 = values["pv_estimate10"] if stamp in period_data: period_data[stamp]["pv_estimate"] = dp4(period_data[stamp]["pv_estimate"] + pv50) period_data[stamp]["pv_estimate10"] = dp4(period_data[stamp]["pv_estimate10"] + pv10) else: - period_data[stamp] = data_item + period_data[stamp] = {"period_start": stamp.strftime(TIME_FORMAT), "pv_estimate": pv50, "pv_estimate10": pv10} sorted_data = [] if period_data: @@ -452,7 +386,7 @@ async def download_forecast_solar_data(self): dec = config.get("declination", 35.0) az = config.get("azimuth", 180.0) if not config.get("azimuth_zero_south", False): - az = self.convert_azimuth(az) + az = convert_azimuth(az) kwp = config.get("kwp", 3.0) efficiency = config.get("efficiency", 0.95) api_key = config.get("api_key", None) diff --git a/apps/predbat/tests/open_meteo_live.py b/apps/predbat/tests/open_meteo_live.py index 3ac6ff669..16b598e4b 100644 --- a/apps/predbat/tests/open_meteo_live.py +++ b/apps/predbat/tests/open_meteo_live.py @@ -41,6 +41,7 @@ from const import TIME_FORMAT # noqa: E402 from solcast import SolarAPI # noqa: E402 +from solar_model import convert_azimuth # noqa: E402 from tests.test_solcast import MockBase # noqa: E402 # ── Solar array definitions ──────────────────────────────────────────────────── @@ -225,9 +226,8 @@ async def run(fs_api_key: str = None, solcast_api_key: str = None, solcast_host: print(f"Predbat PV forecast pipeline comparison — {today_str} UTC ({tz_name})") print() print(f"Arrays ({len(ARRAYS)} configured):") - _tmp = SolarAPI.__new__(SolarAPI) for i, a in enumerate(ARRAYS, 1): - az_api = SolarAPI.convert_azimuth(_tmp, a["azimuth"]) + az_api = convert_azimuth(a["azimuth"]) print(f" [{i}] postcode={a['postcode']} kwp={a['kwp']} declination={a['declination']}° azimuth={a['azimuth']}° (→API: {az_api:.0f}°) efficiency={a.get('efficiency', 1.0):.0%}") print(f" cache: {_CACHE_ROOT}/cache/") diff --git a/apps/predbat/tests/test_solar_model.py b/apps/predbat/tests/test_solar_model.py new file mode 100644 index 000000000..d597827af --- /dev/null +++ b/apps/predbat/tests/test_solar_model.py @@ -0,0 +1,137 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the shared solar GTI to kW conversion model.""" + +from datetime import datetime + +import pytz + +from solar_model import convert_azimuth, gti_hourly_to_period_kwh, pvwatts_cell_temperature + +FLAT_TIMES = ["2025-06-01T{:02d}:00".format(hour) for hour in range(4)] + + +def stamp_for(text): + """Return the tz-aware UTC datetime for an Open-Meteo timestamp string.""" + return pytz.utc.localize(datetime.strptime(text, "%Y-%m-%dT%H:%M")) + + +def test_solar_model(my_predbat): + """Verify the shared solar model against hand-derived values.""" + failed = False + print("**** Testing solar_model ****") + + print("Test: convert_azimuth maps the Predbat convention onto the Open-Meteo one") + for predbat_az, expected in [(180, 0), (90, 90), (270, -90), (0, 180)]: + result = convert_azimuth(predbat_az) + if result != expected: + print(" ERROR: convert_azimuth({}) expected {}, got {}".format(predbat_az, expected, result)) + failed = True + + print("Test: pvwatts_cell_temperature matches the SAPM formula") + # T_cell = 25 + 1000*exp(-3.47 + -0.0594*0) + (1000/1000)*3.0 + # = 25 + 1000*0.031117 + 3 = 59.117 + hot = pvwatts_cell_temperature(1000.0, 25.0, 0.0) + if abs(hot - 59.117) > 0.001: + print(" ERROR: cell temperature expected 59.117, got {}".format(hot)) + failed = True + if pvwatts_cell_temperature(0.0, 20.0, 1.5) != 20.0: + print(" ERROR: zero irradiance should give ambient temperature") + failed = True + + print("Test: a constant-irradiance hour converts to the hand-derived energy") + # eta = 1 - 0.004*(59.117 - 25) = 0.863532; pv = (1000/1000) * 1 kWp * eta * 1.0 + # Both endpoints are equal so the trapezoid returns the same value. + flat_gti = [1000.0] * 4 + flat_temp = [25.0] * 4 + flat_wind = [0.0] * 4 + result = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0) + if len(result) != 3: + print(" ERROR: 4 samples should yield 3 integrated periods, got {}".format(len(result))) + failed = True + first = result.get(stamp_for(FLAT_TIMES[0])) + if first is None: + print(" ERROR: missing the first period") + failed = True + elif abs(first["pv_estimate"] - 0.8635) > 0.0001: + print(" ERROR: expected 0.8635 kWh, got {}".format(first["pv_estimate"])) + failed = True + + print("Test: cold panels are allowed to exceed their STC rating") + # T_cell = 0 + 200*exp(-3.47 - 0.0594) + 0.6 = 6.4645; eta = 1.074142 (above 1.0) + cold = gti_hourly_to_period_kwh(FLAT_TIMES, [200.0] * 4, [0.0] * 4, [1.0] * 4, kwp=1.0, system_loss=0.0) + cold_first = cold[stamp_for(FLAT_TIMES[0])] + if abs(cold_first["pv_estimate"] - 0.2148) > 0.0001: + print(" ERROR: expected 0.2148 kWh for cold panels, got {}".format(cold_first["pv_estimate"])) + failed = True + + print("Test: the trapezoid integrates a rising ramp to the mean of its endpoints") + ramp = gti_hourly_to_period_kwh(FLAT_TIMES, [0.0, 1000.0, 1000.0, 0.0], [25.0] * 4, [0.0] * 4, kwp=1.0, system_loss=0.0) + # Endpoints 0.0 and 0.8635 average to 0.43175, rounded to 4 places + if abs(ramp[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - 0.4318) > 0.0001: + print(" ERROR: expected 0.4318 kWh across the sunrise hour, got {}".format(ramp[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True + + print("Test: zero irradiance produces zero energy") + dark = gti_hourly_to_period_kwh(FLAT_TIMES, [0.0] * 4, [15.0] * 4, [1.0] * 4, kwp=5.0, system_loss=0.05) + if any(entry["pv_estimate"] != 0.0 for entry in dark.values()): + print(" ERROR: zero irradiance should give zero energy, got {}".format(dark)) + failed = True + + print("Test: system_loss and kwp scale the output linearly") + scaled = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=2.0, system_loss=0.5) + if abs(scaled[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - first["pv_estimate"]) > 0.0001: + print(" ERROR: doubling kwp and halving efficiency should cancel out, got {}".format(scaled[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True + + print("Test: p10_fallback scales the P10 series and defaults to 0.7") + if abs(first["pv_estimate10"] - round(first["pv_estimate"] * 0.7, 4)) > 0.0001: + print(" ERROR: the default P10 fallback should be 0.7, got {}".format(first["pv_estimate10"])) + failed = True + half = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_fallback=0.5) + half_first = half[stamp_for(FLAT_TIMES[0])] + if abs(half_first["pv_estimate10"] - round(half_first["pv_estimate"] * 0.5, 4)) > 0.0001: + print(" ERROR: p10_fallback 0.5 not applied, got {}".format(half_first["pv_estimate10"])) + failed = True + + print("Test: p10_instant overrides the fallback and is capped at P50") + ensemble = {FLAT_TIMES[index]: 0.1 for index in range(4)} + with_ensemble = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_instant=ensemble) + ensemble_first = with_ensemble[stamp_for(FLAT_TIMES[0])] + if ensemble_first["pv_estimate10"] >= ensemble_first["pv_estimate"]: + print(" ERROR: an ensemble P10 below P50 should stay below it, got {}".format(ensemble_first["pv_estimate10"])) + failed = True + huge = {FLAT_TIMES[index]: 99.0 for index in range(4)} + capped = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, p10_instant=huge) + capped_first = capped[stamp_for(FLAT_TIMES[0])] + if abs(capped_first["pv_estimate10"] - capped_first["pv_estimate"]) > 0.0001: + print(" ERROR: an ensemble P10 above P50 should be capped at P50, got {}".format(capped_first["pv_estimate10"])) + failed = True + + print("Test: shading_factors apply the correct month") + shaded = gti_hourly_to_period_kwh(FLAT_TIMES, flat_gti, flat_temp, flat_wind, kwp=1.0, system_loss=0.0, shading_factors=[0.5] * 12) + if abs(shaded[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - round(first["pv_estimate"] * 0.5, 4)) > 0.0001: + print(" ERROR: a 0.5 shading factor was not applied, got {}".format(shaded[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True + + print("Test: a gap in the timestamps is not integrated across") + gapped_times = ["2025-06-01T00:00", "2025-06-01T01:00", "2025-06-01T05:00"] + gapped = gti_hourly_to_period_kwh(gapped_times, [1000.0] * 3, [25.0] * 3, [0.0] * 3, kwp=1.0, system_loss=0.0) + if len(gapped) != 1: + print(" ERROR: only the contiguous hour pair should integrate, got {} periods".format(len(gapped))) + failed = True + + print("Test: a None irradiance sample is treated as zero rather than raising") + with_none = gti_hourly_to_period_kwh(FLAT_TIMES, [None, 1000.0, 1000.0, None], [25.0] * 4, [0.0] * 4, kwp=1.0, system_loss=0.0) + if abs(with_none[stamp_for(FLAT_TIMES[0])]["pv_estimate"] - 0.4318) > 0.0001: + print(" ERROR: a None sample should behave as zero, got {}".format(with_none[stamp_for(FLAT_TIMES[0])]["pv_estimate"])) + failed = True + + return failed diff --git a/apps/predbat/tests/test_solcast.py b/apps/predbat/tests/test_solcast.py index 3fd294ad9..573dd8b04 100644 --- a/apps/predbat/tests/test_solcast.py +++ b/apps/predbat/tests/test_solcast.py @@ -19,6 +19,7 @@ import aiohttp from solcast import SolarAPI +from solar_model import convert_azimuth from storage import StorageLocalFiles from tests.test_infra import run_async, create_aiohttp_mock_response @@ -284,9 +285,9 @@ def test_convert_azimuth(my_predbat): ] for solcast_az, expected in test_cases: - result = test_api.solar.convert_azimuth(solcast_az) + result = convert_azimuth(solcast_az) if result != expected: - print(f"ERROR: convert_azimuth({solcast_az}) = {result}, expected {expected}") + print("ERROR: convert_azimuth({}) = {}, expected {}".format(solcast_az, result, expected)) failed = True finally: test_api.cleanup() diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 2463b19ea..5b6be5e78 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -124,6 +124,7 @@ from tests.test_enphase_api import run_enphase_api_tests from tests.test_solcast import run_solcast_tests from tests.test_open_meteo import run_open_meteo_tests +from tests.test_solar_model import test_solar_model from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -316,6 +317,7 @@ def main(): ("enphase_api", run_enphase_api_tests, "Enphase API tests", False), ("solcast", run_solcast_tests, "Solcast API tests", False), ("open_meteo", run_open_meteo_tests, "Open-Meteo solar forecast provider tests", False), + ("solar_model", test_solar_model, "Shared solar GTI conversion model tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From d6f649713f903424f983545c088c7b9809a64a12 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:02:49 +0200 Subject: [PATCH 006/119] feat(annual): add domestic load profile data tables --- apps/predbat/annual_profiles.py | 87 ++++++++++++++++++++++ apps/predbat/tests/test_annual_profiles.py | 72 ++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 161 insertions(+) create mode 100644 apps/predbat/annual_profiles.py create mode 100644 apps/predbat/tests/test_annual_profiles.py diff --git a/apps/predbat/annual_profiles.py b/apps/predbat/annual_profiles.py new file mode 100644 index 000000000..59a4ef88d --- /dev/null +++ b/apps/predbat/annual_profiles.py @@ -0,0 +1,87 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Domestic load profile data tables for the annual prediction tool. + +Data only, no behaviour, so the shapes can be recalibrated against real +consumption data without touching the code that consumes them. +""" + +# Relative electricity consumption by hour of day for a typical UK domestic +# property, index 0 = 00:00. Unnormalised; half_hour_shape() normalises to 1.0. +# Shape: overnight trough, a modest morning peak, a midday plateau, and a +# pronounced evening peak from about 17:00 to 21:00. +HOURLY_SHAPE = [ + 2.6, # 00:00 + 2.4, # 01:00 + 2.3, # 02:00 + 2.2, # 03:00 + 2.2, # 04:00 + 2.4, # 05:00 + 3.0, # 06:00 + 3.8, # 07:00 + 4.2, # 08:00 + 4.1, # 09:00 + 3.9, # 10:00 + 3.8, # 11:00 + 3.9, # 12:00 + 3.8, # 13:00 + 3.7, # 14:00 + 3.9, # 15:00 + 4.6, # 16:00 + 5.8, # 17:00 + 6.5, # 18:00 + 6.3, # 19:00 + 5.6, # 20:00 + 5.0, # 21:00 + 4.3, # 22:00 + 3.4, # 23:00 +] + +# Relative daily consumption by month, index 0 = January. Captures the UK +# winter/summer split, which drives much of the annual answer. These are daily +# rates, so consumers must normalise by days-in-month to preserve the annual total. +MONTH_WEIGHTS = [ + 1.20, # January + 1.15, # February + 1.05, # March + 0.95, # April + 0.88, # May + 0.83, # June + 0.82, # July + 0.83, # August + 0.90, # September + 1.00, # October + 1.12, # November + 1.22, # December +] + +# Proportion of the SOURCE band's own energy moved to the destination band when +# the user selects a "night" or "day" biased profile. Expressed relative to the +# source band rather than to the whole day so the transfer can never exceed the +# energy available to move. Tunable against real data. +SHAPE_TILT_FRACTION = 0.30 + +# Half-hour slot indices, 0 = 00:00-00:30, 47 = 23:30-00:00. +NIGHT_BAND_SLOTS = list(range(0, 14)) # 00:00 - 07:00 +DAY_BAND_SLOTS = list(range(14, 40)) # 07:00 - 20:00 + + +def half_hour_shape(): + """Return the 48-slot half-hourly domestic shape, normalised to sum to exactly 1.0. + + Each hourly weight is split evenly across its two half-hour slots. The final + slot absorbs any floating-point residue so the total is exactly 1.0. + """ + total = float(sum(HOURLY_SHAPE)) + shape = [] + for weight in HOURLY_SHAPE: + half = (weight / total) / 2.0 + shape.append(half) + shape.append(half) + residue = 1.0 - sum(shape) + shape[-1] += residue + return shape diff --git a/apps/predbat/tests/test_annual_profiles.py b/apps/predbat/tests/test_annual_profiles.py new file mode 100644 index 000000000..23bb61962 --- /dev/null +++ b/apps/predbat/tests/test_annual_profiles.py @@ -0,0 +1,72 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction load profile data tables.""" + +from annual_profiles import DAY_BAND_SLOTS, HOURLY_SHAPE, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape + + +def test_annual_profiles(my_predbat): + """Verify the profile tables are well formed and normalise correctly.""" + failed = False + print("**** Testing annual_profiles ****") + + print("Test: HOURLY_SHAPE has 24 positive entries") + if len(HOURLY_SHAPE) != 24: + print(" ERROR: expected 24 hourly weights, got {}".format(len(HOURLY_SHAPE))) + failed = True + if any(value <= 0 for value in HOURLY_SHAPE): + print(" ERROR: all hourly weights must be positive") + failed = True + + print("Test: half_hour_shape returns 48 values summing to 1.0") + shape = half_hour_shape() + if len(shape) != 48: + print(" ERROR: expected 48 half-hourly values, got {}".format(len(shape))) + failed = True + total = sum(shape) + if abs(total - 1.0) > 1e-9: + print(" ERROR: half_hour_shape must sum to 1.0, got {}".format(total)) + failed = True + + print("Test: the evening peak exceeds the overnight trough") + evening = sum(shape[36:42]) + overnight = sum(shape[4:10]) + if evening <= overnight: + print(" ERROR: evening 18:00-21:00 share {} should exceed overnight 02:00-05:00 share {}".format(evening, overnight)) + failed = True + + print("Test: MONTH_WEIGHTS has 12 positive entries with winter above summer") + if len(MONTH_WEIGHTS) != 12: + print(" ERROR: expected 12 month weights, got {}".format(len(MONTH_WEIGHTS))) + failed = True + if any(value <= 0 for value in MONTH_WEIGHTS): + print(" ERROR: all month weights must be positive") + failed = True + if MONTH_WEIGHTS[0] <= MONTH_WEIGHTS[6]: + print(" ERROR: January weight {} should exceed July weight {}".format(MONTH_WEIGHTS[0], MONTH_WEIGHTS[6])) + failed = True + + print("Test: the night and day bands are disjoint and correctly sized") + if NIGHT_BAND_SLOTS != list(range(0, 14)): + print(" ERROR: NIGHT_BAND_SLOTS should cover 00:00-07:00, got {}".format(NIGHT_BAND_SLOTS)) + failed = True + if DAY_BAND_SLOTS != list(range(14, 40)): + print(" ERROR: DAY_BAND_SLOTS should cover 07:00-20:00, got {}".format(DAY_BAND_SLOTS)) + failed = True + if set(NIGHT_BAND_SLOTS) & set(DAY_BAND_SLOTS): + print(" ERROR: night and day bands must be disjoint") + failed = True + + print("Test: SHAPE_TILT_FRACTION is a sane proportion") + if not 0.0 < SHAPE_TILT_FRACTION < 0.5: + print(" ERROR: SHAPE_TILT_FRACTION should be between 0 and 0.5, got {}".format(SHAPE_TILT_FRACTION)) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 5b6be5e78..5532591b0 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -125,6 +125,7 @@ from tests.test_solcast import run_solcast_tests from tests.test_open_meteo import run_open_meteo_tests from tests.test_solar_model import test_solar_model +from tests.test_annual_profiles import test_annual_profiles from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -318,6 +319,7 @@ def main(): ("solcast", run_solcast_tests, "Solcast API tests", False), ("open_meteo", run_open_meteo_tests, "Open-Meteo solar forecast provider tests", False), ("solar_model", test_solar_model, "Shared solar GTI conversion model tests", False), + ("annual_profiles", test_annual_profiles, "Annual prediction load profile table tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From 208acae5396bbe7c9815bb2fd5ae0858c9b0c60d Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:07:25 +0200 Subject: [PATCH 007/119] fix(annual): correct indentation and add cspell dictionary entry --- .cspell/custom-dictionary-workspace.txt | 1 + apps/predbat/annual_profiles.py | 98 ++++++++++---------- apps/predbat/tests/test_annual_profiles.py | 100 ++++++++++----------- 3 files changed, 100 insertions(+), 99 deletions(-) diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index 53cf8a682..0992994f7 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -472,6 +472,7 @@ twinx tzfile tzpath unconfigured +unnormalised unparseable unsmoothed unstaged diff --git a/apps/predbat/annual_profiles.py b/apps/predbat/annual_profiles.py index 59a4ef88d..9bd34a894 100644 --- a/apps/predbat/annual_profiles.py +++ b/apps/predbat/annual_profiles.py @@ -15,48 +15,48 @@ # Shape: overnight trough, a modest morning peak, a midday plateau, and a # pronounced evening peak from about 17:00 to 21:00. HOURLY_SHAPE = [ - 2.6, # 00:00 - 2.4, # 01:00 - 2.3, # 02:00 - 2.2, # 03:00 - 2.2, # 04:00 - 2.4, # 05:00 - 3.0, # 06:00 - 3.8, # 07:00 - 4.2, # 08:00 - 4.1, # 09:00 - 3.9, # 10:00 - 3.8, # 11:00 - 3.9, # 12:00 - 3.8, # 13:00 - 3.7, # 14:00 - 3.9, # 15:00 - 4.6, # 16:00 - 5.8, # 17:00 - 6.5, # 18:00 - 6.3, # 19:00 - 5.6, # 20:00 - 5.0, # 21:00 - 4.3, # 22:00 - 3.4, # 23:00 + 2.6, # 00:00 + 2.4, # 01:00 + 2.3, # 02:00 + 2.2, # 03:00 + 2.2, # 04:00 + 2.4, # 05:00 + 3.0, # 06:00 + 3.8, # 07:00 + 4.2, # 08:00 + 4.1, # 09:00 + 3.9, # 10:00 + 3.8, # 11:00 + 3.9, # 12:00 + 3.8, # 13:00 + 3.7, # 14:00 + 3.9, # 15:00 + 4.6, # 16:00 + 5.8, # 17:00 + 6.5, # 18:00 + 6.3, # 19:00 + 5.6, # 20:00 + 5.0, # 21:00 + 4.3, # 22:00 + 3.4, # 23:00 ] # Relative daily consumption by month, index 0 = January. Captures the UK # winter/summer split, which drives much of the annual answer. These are daily # rates, so consumers must normalise by days-in-month to preserve the annual total. MONTH_WEIGHTS = [ - 1.20, # January - 1.15, # February - 1.05, # March - 0.95, # April - 0.88, # May - 0.83, # June - 0.82, # July - 0.83, # August - 0.90, # September - 1.00, # October - 1.12, # November - 1.22, # December + 1.20, # January + 1.15, # February + 1.05, # March + 0.95, # April + 0.88, # May + 0.83, # June + 0.82, # July + 0.83, # August + 0.90, # September + 1.00, # October + 1.12, # November + 1.22, # December ] # Proportion of the SOURCE band's own energy moved to the destination band when @@ -71,17 +71,17 @@ def half_hour_shape(): - """Return the 48-slot half-hourly domestic shape, normalised to sum to exactly 1.0. + """Return the 48-slot half-hourly domestic shape, normalised to sum to exactly 1.0. - Each hourly weight is split evenly across its two half-hour slots. The final - slot absorbs any floating-point residue so the total is exactly 1.0. - """ - total = float(sum(HOURLY_SHAPE)) - shape = [] - for weight in HOURLY_SHAPE: - half = (weight / total) / 2.0 - shape.append(half) - shape.append(half) - residue = 1.0 - sum(shape) - shape[-1] += residue - return shape + Each hourly weight is split evenly across its two half-hour slots. The final + slot absorbs any floating-point residue so the total is exactly 1.0. + """ + total = float(sum(HOURLY_SHAPE)) + shape = [] + for weight in HOURLY_SHAPE: + half = (weight / total) / 2.0 + shape.append(half) + shape.append(half) + residue = 1.0 - sum(shape) + shape[-1] += residue + return shape diff --git a/apps/predbat/tests/test_annual_profiles.py b/apps/predbat/tests/test_annual_profiles.py index 23bb61962..161bb7c22 100644 --- a/apps/predbat/tests/test_annual_profiles.py +++ b/apps/predbat/tests/test_annual_profiles.py @@ -13,60 +13,60 @@ def test_annual_profiles(my_predbat): - """Verify the profile tables are well formed and normalise correctly.""" - failed = False - print("**** Testing annual_profiles ****") + """Verify the profile tables are well formed and normalise correctly.""" + failed = False + print("**** Testing annual_profiles ****") - print("Test: HOURLY_SHAPE has 24 positive entries") - if len(HOURLY_SHAPE) != 24: - print(" ERROR: expected 24 hourly weights, got {}".format(len(HOURLY_SHAPE))) - failed = True - if any(value <= 0 for value in HOURLY_SHAPE): - print(" ERROR: all hourly weights must be positive") - failed = True + print("Test: HOURLY_SHAPE has 24 positive entries") + if len(HOURLY_SHAPE) != 24: + print(" ERROR: expected 24 hourly weights, got {}".format(len(HOURLY_SHAPE))) + failed = True + if any(value <= 0 for value in HOURLY_SHAPE): + print(" ERROR: all hourly weights must be positive") + failed = True - print("Test: half_hour_shape returns 48 values summing to 1.0") - shape = half_hour_shape() - if len(shape) != 48: - print(" ERROR: expected 48 half-hourly values, got {}".format(len(shape))) - failed = True - total = sum(shape) - if abs(total - 1.0) > 1e-9: - print(" ERROR: half_hour_shape must sum to 1.0, got {}".format(total)) - failed = True + print("Test: half_hour_shape returns 48 values summing to 1.0") + shape = half_hour_shape() + if len(shape) != 48: + print(" ERROR: expected 48 half-hourly values, got {}".format(len(shape))) + failed = True + total = sum(shape) + if abs(total - 1.0) > 1e-9: + print(" ERROR: half_hour_shape must sum to 1.0, got {}".format(total)) + failed = True - print("Test: the evening peak exceeds the overnight trough") - evening = sum(shape[36:42]) - overnight = sum(shape[4:10]) - if evening <= overnight: - print(" ERROR: evening 18:00-21:00 share {} should exceed overnight 02:00-05:00 share {}".format(evening, overnight)) - failed = True + print("Test: the evening peak exceeds the overnight trough") + evening = sum(shape[36:42]) + overnight = sum(shape[4:10]) + if evening <= overnight: + print(" ERROR: evening 18:00-21:00 share {} should exceed overnight 02:00-05:00 share {}".format(evening, overnight)) + failed = True - print("Test: MONTH_WEIGHTS has 12 positive entries with winter above summer") - if len(MONTH_WEIGHTS) != 12: - print(" ERROR: expected 12 month weights, got {}".format(len(MONTH_WEIGHTS))) - failed = True - if any(value <= 0 for value in MONTH_WEIGHTS): - print(" ERROR: all month weights must be positive") - failed = True - if MONTH_WEIGHTS[0] <= MONTH_WEIGHTS[6]: - print(" ERROR: January weight {} should exceed July weight {}".format(MONTH_WEIGHTS[0], MONTH_WEIGHTS[6])) - failed = True + print("Test: MONTH_WEIGHTS has 12 positive entries with winter above summer") + if len(MONTH_WEIGHTS) != 12: + print(" ERROR: expected 12 month weights, got {}".format(len(MONTH_WEIGHTS))) + failed = True + if any(value <= 0 for value in MONTH_WEIGHTS): + print(" ERROR: all month weights must be positive") + failed = True + if MONTH_WEIGHTS[0] <= MONTH_WEIGHTS[6]: + print(" ERROR: January weight {} should exceed July weight {}".format(MONTH_WEIGHTS[0], MONTH_WEIGHTS[6])) + failed = True - print("Test: the night and day bands are disjoint and correctly sized") - if NIGHT_BAND_SLOTS != list(range(0, 14)): - print(" ERROR: NIGHT_BAND_SLOTS should cover 00:00-07:00, got {}".format(NIGHT_BAND_SLOTS)) - failed = True - if DAY_BAND_SLOTS != list(range(14, 40)): - print(" ERROR: DAY_BAND_SLOTS should cover 07:00-20:00, got {}".format(DAY_BAND_SLOTS)) - failed = True - if set(NIGHT_BAND_SLOTS) & set(DAY_BAND_SLOTS): - print(" ERROR: night and day bands must be disjoint") - failed = True + print("Test: the night and day bands are disjoint and correctly sized") + if NIGHT_BAND_SLOTS != list(range(0, 14)): + print(" ERROR: NIGHT_BAND_SLOTS should cover 00:00-07:00, got {}".format(NIGHT_BAND_SLOTS)) + failed = True + if DAY_BAND_SLOTS != list(range(14, 40)): + print(" ERROR: DAY_BAND_SLOTS should cover 07:00-20:00, got {}".format(DAY_BAND_SLOTS)) + failed = True + if set(NIGHT_BAND_SLOTS) & set(DAY_BAND_SLOTS): + print(" ERROR: night and day bands must be disjoint") + failed = True - print("Test: SHAPE_TILT_FRACTION is a sane proportion") - if not 0.0 < SHAPE_TILT_FRACTION < 0.5: - print(" ERROR: SHAPE_TILT_FRACTION should be between 0 and 0.5, got {}".format(SHAPE_TILT_FRACTION)) - failed = True + print("Test: SHAPE_TILT_FRACTION is a sane proportion") + if not 0.0 < SHAPE_TILT_FRACTION < 0.5: + print(" ERROR: SHAPE_TILT_FRACTION should be between 0 and 0.5, got {}".format(SHAPE_TILT_FRACTION)) + failed = True - return failed + return failed From f13607073c39a453f950e564a2a2c66247cc4646 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:13:18 +0200 Subject: [PATCH 008/119] feat(annual): add synthetic load profile source --- apps/predbat/annual_load.py | 130 +++++++++++++++++++++++++ apps/predbat/tests/test_annual_load.py | 117 ++++++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 249 insertions(+) create mode 100644 apps/predbat/annual_load.py create mode 100644 apps/predbat/tests/test_annual_load.py diff --git a/apps/predbat/annual_load.py b/apps/predbat/annual_load.py new file mode 100644 index 000000000..312f7ecfa --- /dev/null +++ b/apps/predbat/annual_load.py @@ -0,0 +1,130 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Load profile sources for the annual prediction tool. + +Produces the forward cumulative kWh series that Predbat consumes as +``load_forecast`` when ``load_forecast_only`` is set, so no synthetic backwards +history has to be fabricated. +""" + +import calendar +from datetime import timedelta + +from annual_profiles import DAY_BAND_SLOTS, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape + +MINUTES_PER_DAY = 24 * 60 +MINUTES_PER_SLOT = 30 + + +def tilt_shape(shape_values, direction): + """Move energy between the night and day bands, preserving the total exactly. + + ``direction`` is one of "night" (move day energy into the night band), "day" + (the reverse), or "flat" (no change). The amount moved is + ``SHAPE_TILT_FRACTION`` of the source band's own energy, so the transfer can + never exceed what is available. Energy is taken from and added to individual + slots in proportion to their existing share of their band, which keeps the + within-band shape intact. + """ + if direction == "flat": + return list(shape_values) + + if direction == "night": + source_slots, dest_slots = DAY_BAND_SLOTS, NIGHT_BAND_SLOTS + elif direction == "day": + source_slots, dest_slots = NIGHT_BAND_SLOTS, DAY_BAND_SLOTS + else: + raise ValueError("Unknown load shape '{}', expected night, day or flat".format(direction)) + + tilted = list(shape_values) + source_total = sum(tilted[slot] for slot in source_slots) + dest_total = sum(tilted[slot] for slot in dest_slots) + if source_total <= 0 or dest_total <= 0: + return tilted + + moved = source_total * SHAPE_TILT_FRACTION + for slot in source_slots: + tilted[slot] -= moved * (tilted[slot] / source_total) + for slot in dest_slots: + tilted[slot] += moved * (shape_values[slot] / dest_total) + + # Push any floating-point residue into the largest slot so the total stays exact + residue = sum(shape_values) - sum(tilted) + largest = max(range(len(tilted)), key=lambda index: tilted[index]) + tilted[largest] += residue + return tilted + + +class LoadProfileSource: + """Base class for a source of daily household load profiles.""" + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + raise NotImplementedError + + def minute_profile(self, day): + """Return a list of 1440 per-minute kWh values for the given date, or None if unavailable.""" + raise NotImplementedError + + +class SyntheticLoadProfile(LoadProfileSource): + """Load profile synthesised from an annual kWh total and a shape preference. + + Monthly weights are normalised across the specific year's day counts so the + twelve monthly totals sum to exactly ``annual_kwh``. + """ + + def __init__(self, annual_kwh, shape, year): + """Build the synthetic profile for one calendar year.""" + self.annual_kwh = float(annual_kwh) + self.shape = shape + self.year = year + self.slot_shape = tilt_shape(half_hour_shape(), shape) + + weighted_days = 0.0 + for month in range(1, 13): + days_in_month = calendar.monthrange(year, month)[1] + weighted_days += MONTH_WEIGHTS[month - 1] * days_in_month + self.base_daily_kwh = (self.annual_kwh / weighted_days) if weighted_days > 0 else 0.0 + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + return self.base_daily_kwh * MONTH_WEIGHTS[day.month - 1] + + def minute_profile(self, day): + """Return a list of 1440 per-minute kWh values for the given date.""" + total = self.daily_kwh(day) + profile = [] + for slot_value in self.slot_shape: + per_minute = (total * slot_value) / MINUTES_PER_SLOT + profile.extend([per_minute] * MINUTES_PER_SLOT) + return profile + + +def build_load_forecast(source, start_day, days): + """Build the cumulative kWh series Predbat reads as ``load_forecast``. + + Keys are absolute minutes from midnight on ``start_day``. Predbat differences + consecutive entries via ``get_from_incrementing(..., backwards=False)``, so + the series must be cumulative and must include the final boundary minute + ``days * 1440`` for the last minute to be readable. + + Days for which the source has no data contribute zero and are skipped by the + caller, which is responsible for logging the gap. + """ + forecast = {0: 0.0} + running = 0.0 + for day_offset in range(days): + day = start_day + timedelta(days=day_offset) + profile = source.minute_profile(day) + if profile is None: + profile = [0.0] * MINUTES_PER_DAY + base = day_offset * MINUTES_PER_DAY + for index, value in enumerate(profile): + running += value + forecast[base + index + 1] = running + return forecast diff --git a/apps/predbat/tests/test_annual_load.py b/apps/predbat/tests/test_annual_load.py new file mode 100644 index 000000000..99e60a64d --- /dev/null +++ b/apps/predbat/tests/test_annual_load.py @@ -0,0 +1,117 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction load profile sources.""" + +import calendar +from datetime import date + +from annual_load import SyntheticLoadProfile, build_load_forecast, tilt_shape +from annual_profiles import DAY_BAND_SLOTS, NIGHT_BAND_SLOTS, half_hour_shape + + +def test_annual_load(my_predbat): + """Verify the synthetic load profile preserves totals and tilts correctly.""" + failed = False + print("**** Testing annual_load ****") + + print("Test: tilt_shape preserves the daily total exactly") + base = half_hour_shape() + for direction in ["night", "day", "flat"]: + tilted = tilt_shape(base, direction) + total = sum(tilted) + if abs(total - 1.0) > 1e-9: + print(" ERROR: tilt '{}' changed the total to {}".format(direction, total)) + failed = True + if any(value < 0 for value in tilted): + print(" ERROR: tilt '{}' produced a negative slot".format(direction)) + failed = True + + print("Test: tilt 'night' moves energy into the night band") + night_tilted = tilt_shape(base, "night") + base_night = sum(base[slot] for slot in NIGHT_BAND_SLOTS) + tilted_night = sum(night_tilted[slot] for slot in NIGHT_BAND_SLOTS) + if tilted_night <= base_night: + print(" ERROR: night tilt should raise the night band from {} to more, got {}".format(base_night, tilted_night)) + failed = True + + print("Test: tilt 'day' moves energy into the day band") + day_tilted = tilt_shape(base, "day") + base_day = sum(base[slot] for slot in DAY_BAND_SLOTS) + tilted_day = sum(day_tilted[slot] for slot in DAY_BAND_SLOTS) + if tilted_day <= base_day: + print(" ERROR: day tilt should raise the day band from {} to more, got {}".format(base_day, tilted_day)) + failed = True + + print("Test: tilt 'flat' is a no-op") + flat_tilted = tilt_shape(base, "flat") + if flat_tilted != base: + print(" ERROR: flat tilt should leave the shape unchanged") + failed = True + + print("Test: the twelve monthly totals sum to annual_kwh") + annual_kwh = 3800.0 + source = SyntheticLoadProfile(annual_kwh=annual_kwh, shape="flat", year=2025) + year_total = 0.0 + for month in range(1, 13): + days_in_month = calendar.monthrange(2025, month)[1] + month_total = sum(source.daily_kwh(date(2025, month, day)) for day in range(1, days_in_month + 1)) + year_total += month_total + if abs(year_total - annual_kwh) > 1e-6: + print(" ERROR: twelve months summed to {}, expected {}".format(year_total, annual_kwh)) + failed = True + + print("Test: January daily consumption exceeds July") + january = source.daily_kwh(date(2025, 1, 15)) + july = source.daily_kwh(date(2025, 7, 15)) + if january <= july: + print(" ERROR: January daily {} should exceed July daily {}".format(january, july)) + failed = True + + print("Test: minute_profile has 1440 entries summing to the day's kWh") + day = date(2025, 3, 10) + profile = source.minute_profile(day) + if len(profile) != 1440: + print(" ERROR: expected 1440 minutes, got {}".format(len(profile))) + failed = True + if abs(sum(profile) - source.daily_kwh(day)) > 1e-9: + print(" ERROR: minute profile sums to {}, expected {}".format(sum(profile), source.daily_kwh(day))) + failed = True + + print("Test: build_load_forecast produces a cumulative series Predbat can difference") + forecast = build_load_forecast(source, date(2025, 3, 10), 2) + if forecast.get(0) != 0.0: + print(" ERROR: cumulative series must start at 0, got {}".format(forecast.get(0))) + failed = True + if 2 * 1440 not in forecast: + print(" ERROR: cumulative series must include the final boundary minute {}".format(2 * 1440)) + failed = True + for minute in range(1, 2 * 1440 + 1): + if forecast[minute] < forecast[minute - 1] - 1e-12: + print(" ERROR: cumulative series decreased at minute {}".format(minute)) + failed = True + break + expected_two_days = source.daily_kwh(date(2025, 3, 10)) + source.daily_kwh(date(2025, 3, 11)) + if abs(forecast[2 * 1440] - expected_two_days) > 1e-9: + print(" ERROR: two-day total {} expected {}".format(forecast[2 * 1440], expected_two_days)) + failed = True + + print("Test: differencing the cumulative series recovers the per-minute profile") + first_minute = forecast[1] - forecast[0] + if abs(first_minute - profile[0]) > 1e-12: + print(" ERROR: differenced minute 0 gave {}, expected {}".format(first_minute, profile[0])) + failed = True + + print("Test: a zero annual figure produces a zero profile rather than dividing by zero") + zero_source = SyntheticLoadProfile(annual_kwh=0.0, shape="flat", year=2025) + if sum(zero_source.minute_profile(day)) != 0.0: + print(" ERROR: zero annual kWh should give a zero profile") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 5532591b0..ed5a41341 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -126,6 +126,7 @@ from tests.test_open_meteo import run_open_meteo_tests from tests.test_solar_model import test_solar_model from tests.test_annual_profiles import test_annual_profiles +from tests.test_annual_load import test_annual_load from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -320,6 +321,7 @@ def main(): ("open_meteo", run_open_meteo_tests, "Open-Meteo solar forecast provider tests", False), ("solar_model", test_solar_model, "Shared solar GTI conversion model tests", False), ("annual_profiles", test_annual_profiles, "Annual prediction load profile table tests", False), + ("annual_load", test_annual_load, "Annual prediction load profile tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From b82f404bdfd78476b09900d0975d06b13bd3814a Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:15:04 +0200 Subject: [PATCH 009/119] docs: correct the pre-commit path in the annual prediction plan The script is at coverage/run_pre_commit, not the repo root, and a run that rewrites files via the black or file-contents-sorter hooks has not passed. Two tasks reported pre-commit clean when it was not. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-07-25-annual-prediction-tool.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md index 919099a6a..379515b42 100644 --- a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md +++ b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md @@ -21,7 +21,7 @@ - **Storage:** all caching goes through the Storage abstraction, never direct file access. - **Tests:** every new module needs unit tests, registered in `TEST_REGISTRY` in `apps/predbat/unit_test.py`. Test signature is `def test_name(my_predbat):` returning a truthy value on failure. Tests must not perform network I/O. - **Run tests from `coverage/`:** `cd coverage && source setup.csh` once, then `./run_all --test > /tmp/out.txt 2>&1` and grep the file. Never pipe test output straight to grep. -- **Pre-commit:** `./run_pre_commit` must pass before any commit is considered done. +- **Pre-commit:** the script lives at `coverage/run_pre_commit`, NOT the repo root. Run it (or `coverage/venv/bin/pre-commit run --files `) and confirm every hook reports Passed with **no files modified** — the `black` and `file-contents-sorter` hooks rewrite files in place, so a first run that "passes" while rewriting has not passed. Re-stage and re-run until clean. --- @@ -419,7 +419,7 @@ Expected: no output from the grep, and the `open_meteo` tests must pass exactly - [ ] **Step 9: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/solar_model.py apps/predbat/solcast.py apps/predbat/unit_test.py apps/predbat/tests/test_solar_model.py git commit -m "refactor(solar): extract the shared GTI to kW conversion model" ``` @@ -641,7 +641,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_profiles.py apps/predbat/tests/test_annual_profiles.py apps/predbat/unit_test.py git commit -m "feat(annual): add domestic load profile data tables" ``` @@ -949,7 +949,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_load.py apps/predbat/tests/test_annual_load.py apps/predbat/unit_test.py git commit -m "feat(annual): add synthetic load profile source" ``` @@ -1231,7 +1231,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_load.py apps/predbat/tests/test_annual_load.py apps/predbat/unit_test.py git commit -m "feat(annual): add Octopus consumption load profile source" ``` @@ -1676,7 +1676,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_weather.py apps/predbat/tests/test_annual_weather.py apps/predbat/unit_test.py git commit -m "feat(annual): add Open-Meteo actuals and forecast archive weather module" ``` @@ -2070,7 +2070,7 @@ Expected: no output. If `minute_data` returns fewer minutes than expected, check - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_tariff.py apps/predbat/tests/test_annual_tariff.py apps/predbat/unit_test.py git commit -m "feat(annual): add historical tariff resolution module" ``` @@ -2516,7 +2516,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual.py apps/predbat/tests/test_annual_config.py apps/predbat/unit_test.py git commit -m "feat(annual): add annual prediction config validation" ``` @@ -2954,7 +2954,7 @@ Expected: prints a `soc_max` and `forecast_minutes` line with no traceback. If ` - [ ] **Step 6: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual.py apps/predbat/tests/test_annual_bootstrap.py apps/predbat/unit_test.py git commit -m "feat(annual): add headless PredBat bootstrap and per-sample state reset" ``` @@ -3183,7 +3183,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual.py apps/predbat/tests/test_annual_sampling.py apps/predbat/unit_test.py git commit -m "feat(annual): add irradiance-stratified sample day selection" ``` @@ -3616,7 +3616,7 @@ Expected: no output. - [ ] **Step 5: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual.py apps/predbat/tests/test_annual_scenarios.py apps/predbat/unit_test.py git commit -m "feat(annual): add three-scenario day runner" ``` @@ -4105,7 +4105,7 @@ Expected: no output. - [ ] **Step 7: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual.py apps/predbat/annual_weather.py apps/predbat/tests/test_annual_integration.py apps/predbat/unit_test.py git commit -m "feat(annual): add AnnualPredictor orchestration and results document" ``` @@ -4369,7 +4369,7 @@ Expected: `Config error: annual.location is required...` and `exit=2`, with no t - [ ] **Step 6: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add apps/predbat/annual_cli.py apps/predbat/tests/test_annual_cli.py apps/predbat/unit_test.py git commit -m "feat(annual): add annual prediction command line interface" ``` @@ -4545,7 +4545,7 @@ Expected: no `ERROR`/`FAILED` lines. This is the last gate — the `solcast.py` - [ ] **Step 6: Run pre-commit and commit** ```bash -./run_pre_commit +coverage/run_pre_commit git add docs/annual-prediction.md mkdocs.yml .cspell/custom-dictionary-workspace.txt git commit -m "docs: add annual prediction tool documentation" ``` From 6ff828b477874d63615c9fc227475e0fffe0f736 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:15:47 +0200 Subject: [PATCH 010/119] fix(spell): add solar_model.py's technical terms to the cspell dictionary Task 1 follow-up: derate, COEFF, and trapezoidally were flagged by the cspell pre-commit hook on apps/predbat/solar_model.py. All three are legitimate technical terms from the PVWatts/SAPM model description, so they're added to the workspace dictionary rather than reworded away. --- .cspell/custom-dictionary-workspace.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index 0992994f7..852122512 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -61,6 +61,7 @@ Chrg citem cloudfront Codespaces +COEFF collapsable compareform configform @@ -93,6 +94,7 @@ dedup dedupe dend denorm +derate derating devcontainer devcontainers @@ -463,6 +465,7 @@ timezone tmrw tojson tottime +trapezoidally trapz Trefor treforsiphone From 50cbdc76a7751190e7de4c9b6d380c32c00568fb Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:17:47 +0200 Subject: [PATCH 011/119] docs: warn that pre-commit skips untracked files pre-commit --all-files enumerates via git ls-files, so a brand-new module is never checked and the run reports a false pass. This is what let two tasks ship black and cspell violations while reporting clean. Co-Authored-By: Claude Opus 5 (1M context) --- docs/superpowers/plans/2026-07-25-annual-prediction-tool.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md index 379515b42..17ccf4464 100644 --- a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md +++ b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md @@ -22,6 +22,8 @@ - **Tests:** every new module needs unit tests, registered in `TEST_REGISTRY` in `apps/predbat/unit_test.py`. Test signature is `def test_name(my_predbat):` returning a truthy value on failure. Tests must not perform network I/O. - **Run tests from `coverage/`:** `cd coverage && source setup.csh` once, then `./run_all --test > /tmp/out.txt 2>&1` and grep the file. Never pipe test output straight to grep. - **Pre-commit:** the script lives at `coverage/run_pre_commit`, NOT the repo root. Run it (or `coverage/venv/bin/pre-commit run --files `) and confirm every hook reports Passed with **no files modified** — the `black` and `file-contents-sorter` hooks rewrite files in place, so a first run that "passes" while rewriting has not passed. Re-stage and re-run until clean. +- **`git add` your new files BEFORE running pre-commit.** `pre-commit --all-files` enumerates via `git ls-files`, which skips untracked files, so a brand-new module is silently never checked and the run reports a false pass. Two tasks on this branch shipped `black` and `cspell` violations this way. Either `git add` first, or pass the paths explicitly with `--files`. +- **Report only what you ran.** Never state that a check passed unless you executed the command and read its output. These reports are the review's evidence base. --- From c2e09f7f988cf58e0324b38361bf831bc8bafbaa Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:22:35 +0200 Subject: [PATCH 012/119] feat(annual): add Octopus consumption load profile source Task 4 of the annual electricity-cost prediction tool: downloads a year of real half-hourly Octopus meter readings and serves them as daily load profiles via OctopusConsumptionLoadProfile, discarding any day that doesn't have all 48 half-hour slots (a partial day would look like genuinely low consumption). Falls back to a synthetic profile, or None, for missing days. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual_load.py | 152 ++++++++++++++++++++++++- apps/predbat/tests/test_annual_load.py | 64 ++++++++++- apps/predbat/unit_test.py | 3 +- 3 files changed, 216 insertions(+), 3 deletions(-) diff --git a/apps/predbat/annual_load.py b/apps/predbat/annual_load.py index 312f7ecfa..f1d54270d 100644 --- a/apps/predbat/annual_load.py +++ b/apps/predbat/annual_load.py @@ -11,8 +11,11 @@ history has to be fabricated. """ +import base64 import calendar -from datetime import timedelta +from datetime import date, datetime, timedelta + +import aiohttp from annual_profiles import DAY_BAND_SLOTS, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape @@ -128,3 +131,150 @@ def build_load_forecast(source, start_day, days): running += value forecast[base + index + 1] = running return forecast + + +OCTOPUS_API_BASE = "https://api.octopus.energy/v1" +SLOTS_PER_DAY = 48 + + +def parse_consumption_results(results): + """Turn raw Octopus consumption rows into complete per-day half-hourly kWh lists. + + Only days with all 48 slots present are returned. A partially reported day is + omitted entirely rather than returned short, because a half-populated day + looks like genuinely low consumption and would silently understate the bill. + """ + by_day = {} + for row in results or []: + start = row.get("interval_start") + consumption = row.get("consumption") + if start is None or consumption is None: + continue + try: + stamp = datetime.strptime(start[:16], "%Y-%m-%dT%H:%M") + except (ValueError, TypeError): + continue + slot = stamp.hour * 2 + (1 if stamp.minute >= 30 else 0) + day = stamp.date() + if day not in by_day: + by_day[day] = [None] * SLOTS_PER_DAY + by_day[day][slot] = float(consumption) + + complete = {} + for day, slots in by_day.items(): + if all(value is not None for value in slots): + complete[day] = slots + return complete + + +class OctopusConsumptionLoadProfile(LoadProfileSource): + """Load profile taken from the account's real half-hourly Octopus consumption. + + The meter series already includes any EV charging, which is why the config + layer rejects an Octopus key alongside a separate car charging figure. + """ + + def __init__(self, api_key, account_id, log, storage=None, fallback=None): + """Set up the Octopus consumption source, optionally backed by a fallback profile.""" + self.api_key = api_key + self.account_id = account_id + self.log = log + self.storage = storage + self.fallback = fallback + self.consumption = {} + self.missing_days = set() + self.mpan = None + self.serial = None + + def _auth_header(self): + """Return the HTTP Basic auth header Octopus expects, API key as username.""" + token = base64.b64encode("{}:".format(self.api_key).encode("utf-8")).decode("utf-8") + return {"Authorization": "Basic {}".format(token), "accept": "application/json", "user-agent": "predbat/1.0"} + + async def _get_json(self, session, url): + """Fetch and decode one JSON page, returning None on any failure.""" + try: + async with session.get(url, headers=self._auth_header(), timeout=aiohttp.ClientTimeout(total=30)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Octopus consumption request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Octopus consumption request to {} failed: {}".format(url, error)) + return None + + async def resolve_meter(self, session): + """Resolve the account's MPAN and meter serial. Returns True on success.""" + data = await self._get_json(session, "{}/accounts/{}/".format(OCTOPUS_API_BASE, self.account_id)) + if not data: + return False + for prop in data.get("properties", []) or []: + for point in prop.get("electricity_meter_points", []) or []: + if point.get("is_export"): + continue + meters = point.get("meters", []) or [] + if point.get("mpan") and meters: + self.mpan = point["mpan"] + self.serial = meters[-1].get("serial_number") + if self.serial: + self.log("Annual: Octopus resolved MPAN {} meter {}".format(self.mpan, self.serial)) + return True + self.log("Warn: Annual: Octopus account {} has no usable electricity import meter".format(self.account_id)) + return False + + async def fetch(self, year): + """Download a calendar year of half-hourly consumption. Returns True on success.""" + cache_key = "consumption_{}_{}".format(self.account_id, year) + if self.storage: + cached = await self.storage.load("annual", cache_key) + if isinstance(cached, dict) and cached: + self.consumption = {date.fromisoformat(key): value for key, value in cached.items()} + self.log("Annual: Octopus consumption for {} loaded from cache, {} days".format(year, len(self.consumption))) + return True + + async with aiohttp.ClientSession() as session: + if not await self.resolve_meter(session): + return False + + url = "{}/electricity-meter-points/{}/meters/{}/consumption/?period_from={}-01-01T00:00Z&period_to={}-01-01T00:00Z&page_size=25000&order_by=period".format(OCTOPUS_API_BASE, self.mpan, self.serial, year, year + 1) + rows = [] + pages = 0 + while url and pages < 40: + data = await self._get_json(session, url) + if not data or "results" not in data: + break + rows += data["results"] + url = data.get("next", None) + pages += 1 + + self.consumption = parse_consumption_results(rows) + if not self.consumption: + self.log("Warn: Annual: Octopus returned no complete days of consumption for {}".format(year)) + return False + + self.log("Annual: Octopus consumption for {} downloaded, {} complete days".format(year, len(self.consumption))) + if self.storage: + await self.storage.save("annual", cache_key, {day.isoformat(): slots for day, slots in self.consumption.items()}, format="json") + return True + + def daily_kwh(self, day): + """Return the total household kWh for the given date.""" + slots = self.consumption.get(day) + if slots is not None: + return sum(slots) + if self.fallback: + return self.fallback.daily_kwh(day) + return 0.0 + + def minute_profile(self, day): + """Return 1440 per-minute kWh values, falling back or returning None when the day is missing.""" + slots = self.consumption.get(day) + if slots is None: + self.missing_days.add(day) + if self.fallback: + return self.fallback.minute_profile(day) + return None + profile = [] + for slot_value in slots: + profile.extend([slot_value / MINUTES_PER_SLOT] * MINUTES_PER_SLOT) + return profile diff --git a/apps/predbat/tests/test_annual_load.py b/apps/predbat/tests/test_annual_load.py index 99e60a64d..ae6d6b9ef 100644 --- a/apps/predbat/tests/test_annual_load.py +++ b/apps/predbat/tests/test_annual_load.py @@ -12,7 +12,7 @@ import calendar from datetime import date -from annual_load import SyntheticLoadProfile, build_load_forecast, tilt_shape +from annual_load import OctopusConsumptionLoadProfile, SyntheticLoadProfile, build_load_forecast, parse_consumption_results, tilt_shape from annual_profiles import DAY_BAND_SLOTS, NIGHT_BAND_SLOTS, half_hour_shape @@ -115,3 +115,65 @@ def test_annual_load(my_predbat): failed = True return failed + + +def test_annual_load_octopus(my_predbat): + """Verify Octopus consumption parsing and its fallback behaviour.""" + failed = False + print("**** Testing annual_load Octopus source ****") + + print("Test: parse_consumption_results maps half-hourly readings onto dates") + results = [] + for slot in range(48): + hour = slot // 2 + minute = 30 * (slot % 2) + results.append( + { + "consumption": 0.25, + "interval_start": "2025-03-10T{:02d}:{:02d}:00Z".format(hour, minute), + "interval_end": "2025-03-10T{:02d}:{:02d}:00Z".format(hour, minute), + } + ) + parsed = parse_consumption_results(results) + target = date(2025, 3, 10) + if target not in parsed: + print(" ERROR: expected {} in parsed output, got {}".format(target, list(parsed.keys()))) + failed = True + elif len(parsed[target]) != 48: + print(" ERROR: expected 48 slots, got {}".format(len(parsed[target]))) + failed = True + elif abs(sum(parsed[target]) - 12.0) > 1e-9: + print(" ERROR: expected 12.0 kWh for the day, got {}".format(sum(parsed[target]))) + failed = True + + print("Test: a partial day is reported as missing rather than silently understated") + partial = parse_consumption_results(results[:20]) + if date(2025, 3, 10) in partial: + print(" ERROR: a day with only 20 of 48 slots must not be returned as complete") + failed = True + + print("Test: minute_profile falls back to the synthetic source for a missing day") + fallback = SyntheticLoadProfile(annual_kwh=3800.0, shape="flat", year=2025) + source = OctopusConsumptionLoadProfile(api_key="x", account_id="A-1", log=print, fallback=fallback) + source.consumption = parse_consumption_results(results) + + present = source.minute_profile(date(2025, 3, 10)) + if abs(sum(present) - 12.0) > 1e-9: + print(" ERROR: present day should use real data summing to 12.0, got {}".format(sum(present))) + failed = True + + absent = source.minute_profile(date(2025, 3, 11)) + if abs(sum(absent) - fallback.daily_kwh(date(2025, 3, 11))) > 1e-9: + print(" ERROR: missing day should fall back to synthetic, got {}".format(sum(absent))) + failed = True + if date(2025, 3, 11) not in source.missing_days: + print(" ERROR: a fallback day must be recorded in missing_days") + failed = True + + print("Test: no fallback and no data yields None so the caller can exclude the day") + bare = OctopusConsumptionLoadProfile(api_key="x", account_id="A-1", log=print) + if bare.minute_profile(date(2025, 3, 11)) is not None: + print(" ERROR: with no data and no fallback minute_profile must return None") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index ed5a41341..b626491c3 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -126,7 +126,7 @@ from tests.test_open_meteo import run_open_meteo_tests from tests.test_solar_model import test_solar_model from tests.test_annual_profiles import test_annual_profiles -from tests.test_annual_load import test_annual_load +from tests.test_annual_load import test_annual_load, test_annual_load_octopus from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -322,6 +322,7 @@ def main(): ("solar_model", test_solar_model, "Shared solar GTI conversion model tests", False), ("annual_profiles", test_annual_profiles, "Annual prediction load profile table tests", False), ("annual_load", test_annual_load, "Annual prediction load profile tests", False), + ("annual_load_octopus", test_annual_load_octopus, "Annual prediction Octopus consumption tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From ac92333190f370c98fe3b3ce73fe8e9f087915e4 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:29:40 +0200 Subject: [PATCH 013/119] docs: stop the annual plan caching truncated downloads Task 4's review found that a mid-pagination failure was cached as a complete year. The same break-then-cache pattern appears in Task 5's weather fetch and Task 6's rate fetch; fix both in the plan rather than rediscovering it in two more review cycles. A partial cache with no expiry is worse than no cache: it pins wrong data permanently and reads back as genuine. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-07-25-annual-prediction-tool.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md index 17ccf4464..c7a703853 100644 --- a/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md +++ b/docs/superpowers/plans/2026-07-25-annual-prediction-tool.md @@ -1598,7 +1598,9 @@ class AnnualWeather: data = await self.storage.load("annual", cache_key) if not data: data = await self.fetch_json(url) - if data and self.storage: + # Only cache a response that actually carries the hourly block we need, + # so a rate-limit or error page is never pinned as this array's weather + if data and data.get("hourly", {}).get("global_tilted_irradiance") and self.storage: await self.storage.save("annual", cache_key, data, format="json") if not data: self.log("Warn: Annual: no {} data for array {}".format(cache_tag, index)) @@ -1610,6 +1612,11 @@ class AnnualWeather: if not times or not gti_values: self.log("Warn: Annual: {} data for array {} has no hourly values".format(cache_tag, index)) continue + if len(gti_values) < len(times): + # A response truncated mid-year would otherwise be cached and read back + # as a genuinely dark second half of the year + self.log("Warn: Annual: {} data for array {} is truncated ({} stamps, {} values); discarding".format(cache_tag, index, len(times), len(gti_values))) + continue periods = gti_hourly_to_period_kwh( times, @@ -1963,14 +1970,21 @@ class AnnualTariff: url = build_period_url(base_url, start_utc, end_utc) rows = [] pages = 0 + truncated = False while url and pages < 10: data = await self.fetch_json(url) if not data or "results" not in data: + # A failed page means we do not know what we are missing. Caching a + # partial month would permanently pin wrong rates for that month. + self.log("Warn: Annual: rate download for {} stopped early at page {}; not caching a partial result".format(cache_key, pages)) + truncated = True break rows += data["results"] url = data.get("next", None) pages += 1 + if truncated: + return [] if rows and self.storage: await self.storage.save("annual", cache_key, rows, format="json") return rows From 457cf776cff84aed717f0b267a69017853a87d27 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:32:38 +0200 Subject: [PATCH 014/119] fix(annual): stop caching truncated Octopus downloads and undercounted DST days Three failure-visibility gaps in the Octopus consumption source, all in tension with the design goal that failures must be visible, never silent: - fetch() treated a mid-pagination request failure identically to reaching the end of the pages, so a transient network blip could permanently cache a partial year. It now tracks whether the loop ended by exhausting `next` links versus a failed request (or the page safety cap), and only caches on a genuine natural end. - daily_kwh() silently returned 0.0 for a missing day without recording it in missing_days, disagreeing with minute_profile() for the same case. Both now record the date consistently. - parse_consumption_results() let the autumn clock-change day's duplicate 01:00-02:00 half-hour readings silently overwrite each other, producing an undercounted "complete" day. A day with any slot written twice is now discarded like a partial day, logged when a logger is supplied. Adds one covering test per fix in test_annual_load_octopus. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual_load.py | 39 +++++++++++++++--- apps/predbat/tests/test_annual_load.py | 56 ++++++++++++++++++++++++++ 2 files changed, 89 insertions(+), 6 deletions(-) diff --git a/apps/predbat/annual_load.py b/apps/predbat/annual_load.py index f1d54270d..307092f47 100644 --- a/apps/predbat/annual_load.py +++ b/apps/predbat/annual_load.py @@ -137,14 +137,24 @@ def build_load_forecast(source, start_day, days): SLOTS_PER_DAY = 48 -def parse_consumption_results(results): +def parse_consumption_results(results, log=None): """Turn raw Octopus consumption rows into complete per-day half-hourly kWh lists. - Only days with all 48 slots present are returned. A partially reported day is - omitted entirely rather than returned short, because a half-populated day - looks like genuinely low consumption and would silently understate the bill. + Only days with all 48 slots present, each written exactly once, are returned. + A partially reported day is omitted entirely rather than returned short, + because a half-populated day looks like genuinely low consumption and would + silently understate the bill. + + The autumn clock-change day gets 50 real half-hourly readings (local + 01:00-02:00 happens twice), but the slot index only has 48 possible values, + so the second pair of readings would silently overwrite the first and leave + an undercounted-but-"complete" day. Any day where a slot is written more than + once is therefore discarded too, exactly like a partial day, so the autumn + transition is handled symmetrically with the spring one (which already comes + in short at 46/48 and is rejected as partial). """ by_day = {} + duplicate_slot_days = set() for row in results or []: start = row.get("interval_start") consumption = row.get("consumption") @@ -158,10 +168,17 @@ def parse_consumption_results(results): day = stamp.date() if day not in by_day: by_day[day] = [None] * SLOTS_PER_DAY + if by_day[day][slot] is not None: + duplicate_slot_days.add(day) + continue by_day[day][slot] = float(consumption) complete = {} for day, slots in by_day.items(): + if day in duplicate_slot_days: + if log: + log("Warn: Annual: Octopus discarded {} - a half-hourly slot was reported more than once, likely a clock-change day".format(day)) + continue if all(value is not None for value in slots): complete[day] = slots return complete @@ -239,15 +256,24 @@ async def fetch(self, year): url = "{}/electricity-meter-points/{}/meters/{}/consumption/?period_from={}-01-01T00:00Z&period_to={}-01-01T00:00Z&page_size=25000&order_by=period".format(OCTOPUS_API_BASE, self.mpan, self.serial, year, year + 1) rows = [] pages = 0 + fetch_failed = False while url and pages < 40: data = await self._get_json(session, url) if not data or "results" not in data: + fetch_failed = True break rows += data["results"] url = data.get("next", None) pages += 1 - self.consumption = parse_consumption_results(rows) + # A run only reaches its natural end when `url` runs out on its own. If a page request + # failed, or the safety cap was hit while pages remained, `rows` is a truncated + # download - it must not be parsed, used, or cached as if it were complete. + if fetch_failed or url: + self.log("Warn: Annual: Octopus consumption download for {} did not complete, discarding the partial download rather than caching it".format(year)) + return False + + self.consumption = parse_consumption_results(rows, log=self.log) if not self.consumption: self.log("Warn: Annual: Octopus returned no complete days of consumption for {}".format(year)) return False @@ -258,10 +284,11 @@ async def fetch(self, year): return True def daily_kwh(self, day): - """Return the total household kWh for the given date.""" + """Return the total household kWh for the given date, recording a missing day like minute_profile does.""" slots = self.consumption.get(day) if slots is not None: return sum(slots) + self.missing_days.add(day) if self.fallback: return self.fallback.daily_kwh(day) return 0.0 diff --git a/apps/predbat/tests/test_annual_load.py b/apps/predbat/tests/test_annual_load.py index ae6d6b9ef..98f1f40c1 100644 --- a/apps/predbat/tests/test_annual_load.py +++ b/apps/predbat/tests/test_annual_load.py @@ -14,6 +14,25 @@ from annual_load import OctopusConsumptionLoadProfile, SyntheticLoadProfile, build_load_forecast, parse_consumption_results, tilt_shape from annual_profiles import DAY_BAND_SLOTS, NIGHT_BAND_SLOTS, half_hour_shape +from tests.test_infra import run_async + + +class FakeAnnualStorage: + """Minimal in-memory storage stub with async save/load, used to prove nothing is cached on a truncated download.""" + + def __init__(self): + """Start with an empty in-memory store and no recorded saves.""" + self.store = {} + self.saved = False + + async def load(self, namespace, key): + """Return the previously saved value for a key, or None if nothing was saved.""" + return self.store.get((namespace, key)) + + async def save(self, namespace, key, value, format="json"): + """Record that a save happened and remember the value under the given key.""" + self.saved = True + self.store[(namespace, key)] = value def test_annual_load(my_predbat): @@ -176,4 +195,41 @@ def test_annual_load_octopus(my_predbat): print(" ERROR: with no data and no fallback minute_profile must return None") failed = True + print("Test: daily_kwh records a missing day in missing_days, consistent with minute_profile") + bare_daily = OctopusConsumptionLoadProfile(api_key="x", account_id="A-1", log=print) + bare_daily.daily_kwh(date(2025, 3, 12)) + if date(2025, 3, 12) not in bare_daily.missing_days: + print(" ERROR: daily_kwh must record a missing day in missing_days just like minute_profile does") + failed = True + + print("Test: a slot reported twice (e.g. the autumn clock-change day) discards that day entirely") + duplicate_slot_results = list(results) + [dict(results[0])] + duplicate_slot_parsed = parse_consumption_results(duplicate_slot_results) + if date(2025, 3, 10) in duplicate_slot_parsed: + print(" ERROR: a day with a slot reported twice must be discarded, not silently undercounted") + failed = True + + print("Test: a page request failing mid-download is not cached and fetch() reports failure") + calls = [] + + async def fake_get_json(session, url): + """Return a canned meter resolution, one good consumption page with a next link, then a failure.""" + calls.append(url) + if len(calls) == 1: + return {"properties": [{"electricity_meter_points": [{"mpan": "1200000000000", "is_export": False, "meters": [{"serial_number": "S1"}]}]}]} + if len(calls) == 2: + return {"results": results, "next": "https://api.octopus.energy/v1/electricity-meter-points/x/meters/y/consumption/?page=2"} + return None + + fake_storage = FakeAnnualStorage() + truncated_source = OctopusConsumptionLoadProfile(api_key="x", account_id="A-2", log=print, storage=fake_storage) + truncated_source._get_json = fake_get_json + fetch_result = run_async(truncated_source.fetch(2025)) + if fetch_result is not False: + print(" ERROR: a download that fails mid-pagination must return False, got {}".format(fetch_result)) + failed = True + if fake_storage.saved: + print(" ERROR: a truncated download must not be written to storage") + failed = True + return failed From 0003ee35dc4e9f57b6eb66dcb42767ca12703c6e Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:42:25 +0200 Subject: [PATCH 015/119] feat(annual): add Open-Meteo actuals and forecast archive weather module --- apps/predbat/annual_weather.py | 233 ++++++++++++++++++++++ apps/predbat/tests/test_annual_weather.py | 153 ++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 388 insertions(+) create mode 100644 apps/predbat/annual_weather.py create mode 100644 apps/predbat/tests/test_annual_weather.py diff --git a/apps/predbat/annual_weather.py b/apps/predbat/annual_weather.py new file mode 100644 index 000000000..5ecdbd418 --- /dev/null +++ b/apps/predbat/annual_weather.py @@ -0,0 +1,233 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Open-Meteo historical weather for the annual prediction tool. + +Downloads two archives per PV array: ERA5 reanalysis actuals (what really +happened) and the archived short-range forecast for the same dates (what Predbat +would have been looking at). The gap between them is genuine day-ahead forecast +error, from which each month's P10 ratio is derived. +""" + +import math +from datetime import timedelta + +import aiohttp + +from solar_model import convert_azimuth, gti_hourly_to_period_kwh + +ARCHIVE_URL = "https://archive-api.open-meteo.com/v1/archive" +FORECAST_ARCHIVE_URL = "https://historical-forecast-api.open-meteo.com/v1/forecast" + +HOURLY_VARIABLES = "global_tilted_irradiance,temperature_2m,wind_speed_10m" + +# A month needs at least this many usable forecast/actual day pairs before its +# measured P10 ratio is trusted over the flat fallback. +MIN_DAYS_FOR_P10 = 7 + +# Fraction used for the P10 order statistic +P10_FRACTION = 0.10 + + +def percentile(values, fraction): + """Return the order statistic at ``fraction`` through a list of values. + + Uses the same convention as the Solcast ensemble P10 in ``solcast.py``: + sort ascending and take index ``ceil(n * fraction) - 1``, clamped to zero. + Returns 0.0 for an empty list. + """ + if not values: + return 0.0 + ordered = sorted(values) + index = max(0, math.ceil(len(ordered) * fraction) - 1) + return ordered[index] + + +class WeatherYear: + """A year of per-array-summed PV energy, for both actuals and forecast.""" + + def __init__(self, actual_periods, forecast_periods, p10_ratios, forecast_available, fallback_months): + """Hold the converted period data and the derived monthly P10 ratios.""" + self.actual_periods = actual_periods + self.forecast_periods = forecast_periods if forecast_available else actual_periods + self.p10_ratios = p10_ratios + self.forecast_available = forecast_available + self.fallback_months = fallback_months + self._daily_actual = self._daily_totals(self.actual_periods) + + @staticmethod + def _daily_totals(periods): + """Sum hourly period energy into per-date totals.""" + totals = {} + for stamp, kwh in periods.items(): + day = stamp.date() + totals[day] = totals.get(day, 0.0) + kwh + return totals + + def has_actual(self, day): + """Return True when actuals exist for the given date.""" + return day in self._daily_actual + + def daily_actual_kwh(self, day): + """Return the total actual PV kWh generated on the given date.""" + return self._daily_actual.get(day, 0.0) + + def p10_ratio(self, month): + """Return the P10 scaling ratio for the given month number, 1 = January.""" + return self.p10_ratios.get(month, 1.0) + + def pv_minutes(self, series, midnight_utc, minutes): + """Spread hourly period energy across per-minute kWh, keyed by absolute minute. + + ``series`` is "actual" or "forecast". Minutes outside [0, minutes) are + discarded, so the caller always receives a window it asked for. + """ + periods = self.actual_periods if series == "actual" else self.forecast_periods + result = {} + end_utc = midnight_utc + timedelta(minutes=minutes) + for stamp, kwh in periods.items(): + if stamp < midnight_utc or stamp >= end_utc: + continue + offset = int((stamp - midnight_utc).total_seconds() // 60) + per_minute = kwh / 60.0 + for minute in range(offset, offset + 60): + if 0 <= minute < minutes: + result[minute] = result.get(minute, 0.0) + per_minute + return result + + def pv_minutes_p10(self, midnight_utc, minutes, month): + """Return the P10 per-minute series: the forecast series scaled by the month's ratio.""" + ratio = self.p10_ratio(month) + return {minute: value * ratio for minute, value in self.pv_minutes("forecast", midnight_utc, minutes).items()} + + +class AnnualWeather: + """Fetches and converts a calendar year of Open-Meteo data for one site.""" + + def __init__(self, arrays, latitude, longitude, log, storage=None, p10_fallback=0.7, fetch_json=None): + """Configure the site's PV arrays and the JSON fetcher used for downloads.""" + self.arrays = arrays + self.latitude = latitude + self.longitude = longitude + self.log = log + self.storage = storage + self.p10_fallback = p10_fallback + self.fetch_json = fetch_json or self._default_fetch_json + + async def _default_fetch_json(self, url): + """Download and decode one JSON document, returning None on any failure.""" + try: + async with aiohttp.ClientSession() as session: + async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=120)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Open-Meteo request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Open-Meteo request to {} failed: {}".format(url, error)) + return None + + def _build_url(self, base, array, year): + """Build one Open-Meteo request URL for a single array and calendar year. + + The window runs to 1 January of the following year so the final sampled + day still has the following day its 48 hour plan needs. + """ + azimuth = array.get("azimuth", 180.0) + if not array.get("azimuth_zero_south", False): + azimuth = convert_azimuth(azimuth) + return "{}?latitude={}&longitude={}&start_date={}-01-01&end_date={}-01-01&hourly={}&tilt={}&azimuth={}&wind_speed_unit=ms&timezone=UTC".format( + base, self.latitude, self.longitude, year, year + 1, HOURLY_VARIABLES, array.get("declination", 35.0), azimuth + ) + + async def _fetch_series(self, base, year, cache_tag): + """Fetch one source for every array and return the summed hourly period energy.""" + totals = {} + any_data = False + for index, array in enumerate(self.arrays): + url = self._build_url(base, array, year) + data = None + cache_key = "weather_{}_{}_{}_{}_{}".format(cache_tag, year, index, self.latitude, self.longitude) + if self.storage: + data = await self.storage.load("annual", cache_key) + if not data: + data = await self.fetch_json(url) + # Only cache a response that actually carries the hourly block we need, + # so a rate-limit or error page is never pinned as this array's weather + if data and data.get("hourly", {}).get("global_tilted_irradiance") and self.storage: + await self.storage.save("annual", cache_key, data, format="json") + if not data: + self.log("Warn: Annual: no {} data for array {}".format(cache_tag, index)) + continue + + hourly = data.get("hourly", {}) + times = hourly.get("time", []) + gti_values = hourly.get("global_tilted_irradiance", []) + if not times or not gti_values: + self.log("Warn: Annual: {} data for array {} has no hourly values".format(cache_tag, index)) + continue + if len(gti_values) < len(times): + # A response truncated mid-year would otherwise be cached and read back + # as a genuinely dark second half of the year + self.log("Warn: Annual: {} data for array {} is truncated ({} stamps, {} values); discarding".format(cache_tag, index, len(times), len(gti_values))) + continue + + periods = gti_hourly_to_period_kwh( + times, + gti_values, + hourly.get("temperature_2m", []), + hourly.get("wind_speed_10m", []), + kwp=array.get("kwp", 3.0), + system_loss=1.0 - array.get("efficiency", 0.95), + shading_factors=array.get("shading_factors", None), + ) + for stamp, values in periods.items(): + totals[stamp] = totals.get(stamp, 0.0) + values["pv_estimate"] + any_data = True + + return totals if any_data else {} + + def _derive_p10_ratios(self, actual_periods, forecast_periods, forecast_available): + """Derive each month's P10 ratio from the measured actual/forecast daily energy error.""" + ratios = {} + fallback_months = set() + + actual_daily = WeatherYear._daily_totals(actual_periods) + forecast_daily = WeatherYear._daily_totals(forecast_periods) + + by_month = {} + if forecast_available: + for day, forecast_kwh in forecast_daily.items(): + if forecast_kwh <= 0: + continue + if day not in actual_daily: + continue + by_month.setdefault(day.month, []).append(actual_daily[day] / forecast_kwh) + + for month in range(1, 13): + samples = by_month.get(month, []) + if len(samples) >= MIN_DAYS_FOR_P10: + ratios[month] = min(1.0, percentile(samples, P10_FRACTION)) + else: + ratios[month] = self.p10_fallback + fallback_months.add(month) + + if fallback_months: + self.log("Warn: Annual: P10 fell back to the flat {} derate for months {}".format(self.p10_fallback, sorted(fallback_months))) + + return ratios, fallback_months + + async def fetch(self, year): + """Download and convert a calendar year, returning a populated WeatherYear.""" + actual_periods = await self._fetch_series(ARCHIVE_URL, year, "actual") + forecast_periods = await self._fetch_series(FORECAST_ARCHIVE_URL, year, "forecast") + forecast_available = bool(forecast_periods) + + if not forecast_available: + self.log("Warn: Annual: the Open-Meteo forecast archive returned nothing for {}; planning on actuals with the flat P10 derate".format(year)) + + ratios, fallback_months = self._derive_p10_ratios(actual_periods, forecast_periods, forecast_available) + return WeatherYear(actual_periods, forecast_periods, ratios, forecast_available, fallback_months) diff --git a/apps/predbat/tests/test_annual_weather.py b/apps/predbat/tests/test_annual_weather.py new file mode 100644 index 000000000..c854f8281 --- /dev/null +++ b/apps/predbat/tests/test_annual_weather.py @@ -0,0 +1,153 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction Open-Meteo weather module.""" + +import asyncio +from datetime import date, datetime, timedelta + +import pytz + +from annual_weather import AnnualWeather, percentile + +ARRAYS = [{"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}] + + +def build_hourly(start_day, days, peak_gti): + """Build a synthetic Open-Meteo hourly payload with a fixed daily irradiance curve.""" + times = [] + gti = [] + temp = [] + wind = [] + curve = [0, 0, 0, 0, 0, 0, 0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.3, 0.1, 0, 0, 0, 0, 0] + for offset in range(days): + day = start_day + timedelta(days=offset) + for hour in range(24): + times.append("{}T{:02d}:00".format(day.isoformat(), hour)) + gti.append(peak_gti * curve[hour]) + temp.append(15.0) + wind.append(1.5) + return {"hourly": {"time": times, "global_tilted_irradiance": gti, "temperature_2m": temp, "wind_speed_10m": wind}} + + +def test_annual_weather(my_predbat): + """Verify weather fetching, P10 derivation from forecast error, and the fallback path.""" + failed = False + print("**** Testing annual_weather ****") + + print("Test: percentile picks the expected order statistic") + values = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0] + got = percentile(values, 0.10) + if got != 1.0: + print(" ERROR: 10th percentile of 1..10 expected 1.0, got {}".format(got)) + failed = True + if percentile([], 0.10) != 0.0: + print(" ERROR: percentile of an empty list should be 0.0") + failed = True + + # Actuals peak at 800, forecast peaks at 1000, so every day's actual/forecast ratio is 0.8 + start = date(2025, 1, 1) + actual_payload = build_hourly(start, 40, 800.0) + forecast_payload = build_hourly(start, 40, 1000.0) + + async def fake_fetch(url): + """Return the archive or forecast payload depending on the host in the URL.""" + if "archive-api" in url: + return actual_payload + return forecast_payload + + weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=fake_fetch) + year = asyncio.run(weather.fetch(2025)) + + print("Test: the forecast archive is reported as available") + if not year.forecast_available: + print(" ERROR: forecast_available should be True when both payloads parse") + failed = True + + print("Test: January's P10 ratio reflects the measured 0.8 forecast error") + ratio = year.p10_ratio(1) + # The shared solar model applies a GTI-dependent cell-temperature derate (Task 1), so the + # measured ratio sits a little above the raw 800/1000 GTI ratio; 0.02 comfortably covers that. + if abs(ratio - 0.8) > 0.02: + print(" ERROR: expected a P10 ratio near 0.8, got {}".format(ratio)) + failed = True + + print("Test: actual daily energy is below forecast daily energy") + day = date(2025, 1, 10) + if not year.has_actual(day): + print(" ERROR: expected actual data for {}".format(day)) + failed = True + midnight = pytz.utc.localize(datetime(2025, 1, 10, 0, 0)) + actual_minutes = year.pv_minutes("actual", midnight, 24 * 60) + forecast_minutes = year.pv_minutes("forecast", midnight, 24 * 60) + actual_total = sum(actual_minutes.values()) + forecast_total = sum(forecast_minutes.values()) + if actual_total <= 0: + print(" ERROR: actual PV for {} should be positive, got {}".format(day, actual_total)) + failed = True + if forecast_total <= actual_total: + print(" ERROR: forecast total {} should exceed actual total {}".format(forecast_total, actual_total)) + failed = True + + print("Test: pv_minutes covers a 48 hour window and is keyed by absolute minute") + two_day = year.pv_minutes("actual", midnight, 48 * 60) + if max(two_day.keys()) >= 48 * 60: + print(" ERROR: pv_minutes must not emit minutes at or beyond the window length") + failed = True + if abs(sum(two_day.values()) - (actual_total + year.daily_actual_kwh(date(2025, 1, 11)))) > 0.01: + print(" ERROR: the 48 hour window should equal two days of actual energy") + failed = True + + print("Test: pv_minutes_p10 scales the forecast series by the month ratio") + p10_minutes = year.pv_minutes_p10(midnight, 24 * 60, 1) + expected = forecast_total * year.p10_ratio(1) + if abs(sum(p10_minutes.values()) - expected) > 0.01: + print(" ERROR: P10 total {} expected {}".format(sum(p10_minutes.values()), expected)) + failed = True + + print("Test: a missing forecast archive falls back and records the degradation") + + async def actuals_only_fetch(url): + """Serve actuals and fail every forecast request.""" + if "archive-api" in url: + return actual_payload + return None + + degraded_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=actuals_only_fetch, p10_fallback=0.7) + degraded = asyncio.run(degraded_weather.fetch(2025)) + if degraded.forecast_available: + print(" ERROR: forecast_available should be False when the forecast archive is empty") + failed = True + if abs(degraded.p10_ratio(1) - 0.7) > 1e-9: + print(" ERROR: expected the 0.7 fallback ratio, got {}".format(degraded.p10_ratio(1))) + failed = True + if 1 not in degraded.fallback_months: + print(" ERROR: January should be recorded in fallback_months") + failed = True + degraded_forecast = degraded.pv_minutes("forecast", midnight, 24 * 60) + degraded_actual = degraded.pv_minutes("actual", midnight, 24 * 60) + if abs(sum(degraded_forecast.values()) - sum(degraded_actual.values())) > 1e-9: + print(" ERROR: with no forecast archive the forecast series must fall back to actuals") + failed = True + + print("Test: a month with fewer than seven usable days falls back") + sparse_actual = build_hourly(date(2025, 1, 1), 4, 800.0) + sparse_forecast = build_hourly(date(2025, 1, 1), 4, 1000.0) + + async def sparse_fetch(url): + """Serve only four days of data.""" + return sparse_actual if "archive-api" in url else sparse_forecast + + sparse_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=sparse_fetch, p10_fallback=0.7) + sparse = asyncio.run(sparse_weather.fetch(2025)) + if abs(sparse.p10_ratio(1) - 0.7) > 1e-9: + print(" ERROR: a four-day month should fall back to 0.7, got {}".format(sparse.p10_ratio(1))) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index b626491c3..18c4a983a 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -127,6 +127,7 @@ from tests.test_solar_model import test_solar_model from tests.test_annual_profiles import test_annual_profiles from tests.test_annual_load import test_annual_load, test_annual_load_octopus +from tests.test_annual_weather import test_annual_weather from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -323,6 +324,7 @@ def main(): ("annual_profiles", test_annual_profiles, "Annual prediction load profile table tests", False), ("annual_load", test_annual_load, "Annual prediction load profile tests", False), ("annual_load_octopus", test_annual_load_octopus, "Annual prediction Octopus consumption tests", False), + ("annual_weather", test_annual_weather, "Annual prediction Open-Meteo weather tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From fecdbd3934e03727c0aa34bc5e2d0175ec572849 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 20:53:33 +0200 Subject: [PATCH 016/119] fix(annual): validate weather payloads before caching and require every array to parse Fixes review findings on the Open-Meteo weather module: the truncation guard previously ran after the cache write, so a bad response was cached and then rejected forever with no retry. A shared _payload_problem() check now gates both the cache write and the cache read, so a poisoned cache entry is discarded and re-fetched rather than trusted permanently. Also, _fetch_series() previously counted a series as available if any one array parsed, so a failed array silently produced an overly optimistic actual/forecast ratio (capped at 1.0, i.e. no P10 derate at all). Every configured array must now parse for a series to count; a single failure abandons the whole series. Adds covering tests: the P10 order statistic on a month with differing daily ratios, the Storage cache path (write/hit/poisoned-entry recovery), the truncation guard, and the multi-array partial-failure path for both the forecast and actual series. Tightens the deterministic January P10 assertion to the exact measured value and replaces a 48-hour-window assertion that could never fail with one that exercises a non-hour-aligned cutoff. --- apps/predbat/annual_weather.py | 78 +++++---- apps/predbat/tests/test_annual_weather.py | 183 +++++++++++++++++++++- 2 files changed, 225 insertions(+), 36 deletions(-) diff --git a/apps/predbat/annual_weather.py b/apps/predbat/annual_weather.py index 5ecdbd418..9156ea4b8 100644 --- a/apps/predbat/annual_weather.py +++ b/apps/predbat/annual_weather.py @@ -143,41 +143,62 @@ def _build_url(self, base, array, year): base, self.latitude, self.longitude, year, year + 1, HOURLY_VARIABLES, array.get("declination", 35.0), azimuth ) + @staticmethod + def _payload_problem(data): + """Return why a payload cannot be trusted, or None when it is complete and usable. + + Shared by the cache-write gate, the cache-read gate and the fetch failure log, so a + missing, empty or mid-year-truncated response is described identically wherever it + is encountered, and a poisoned cache entry is described the same way a bad live + download would be. + """ + if not data: + return "no data" + hourly = data.get("hourly", {}) + times = hourly.get("time", []) + gti_values = hourly.get("global_tilted_irradiance", []) + if not times or not gti_values: + return "no hourly values" + if len(gti_values) < len(times): + return "truncated ({} stamps, {} values)".format(len(times), len(gti_values)) + return None + async def _fetch_series(self, base, year, cache_tag): - """Fetch one source for every array and return the summed hourly period energy.""" + """Fetch one source for every array and return the summed hourly period energy. + + Every configured array must produce a usable payload for the series to count as + available. If any array fails or returns an unusable payload, the whole series is + abandoned (an empty dict) rather than silently blending a partial result for one + array with complete results for the others, which would understate the true forecast + error and let the P10 derate collapse to an artificially optimistic value. + """ totals = {} - any_data = False for index, array in enumerate(self.arrays): url = self._build_url(base, array, year) - data = None cache_key = "weather_{}_{}_{}_{}_{}".format(cache_tag, year, index, self.latitude, self.longitude) + data = None if self.storage: - data = await self.storage.load("annual", cache_key) + cached = await self.storage.load("annual", cache_key) + if self._payload_problem(cached) is None: + data = cached + # A cache entry written before this guard existed, or poisoned by a past + # rate-limit/error page, is discarded here and re-fetched below rather than + # trusted forever. if not data: - data = await self.fetch_json(url) - # Only cache a response that actually carries the hourly block we need, - # so a rate-limit or error page is never pinned as this array's weather - if data and data.get("hourly", {}).get("global_tilted_irradiance") and self.storage: - await self.storage.save("annual", cache_key, data, format="json") - if not data: - self.log("Warn: Annual: no {} data for array {}".format(cache_tag, index)) - continue - - hourly = data.get("hourly", {}) - times = hourly.get("time", []) - gti_values = hourly.get("global_tilted_irradiance", []) - if not times or not gti_values: - self.log("Warn: Annual: {} data for array {} has no hourly values".format(cache_tag, index)) - continue - if len(gti_values) < len(times): - # A response truncated mid-year would otherwise be cached and read back - # as a genuinely dark second half of the year - self.log("Warn: Annual: {} data for array {} is truncated ({} stamps, {} values); discarding".format(cache_tag, index, len(times), len(gti_values))) - continue - + fetched = await self.fetch_json(url) + problem = self._payload_problem(fetched) + if problem is None: + data = fetched + if self.storage: + await self.storage.save("annual", cache_key, data, format="json") + else: + self.log("Warn: Annual: {} data for array {} is unusable ({}); abandoning the {} series for {}".format(cache_tag, index, problem, cache_tag, year)) + return {} + + hourly = data["hourly"] periods = gti_hourly_to_period_kwh( - times, - gti_values, + hourly["time"], + hourly["global_tilted_irradiance"], hourly.get("temperature_2m", []), hourly.get("wind_speed_10m", []), kwp=array.get("kwp", 3.0), @@ -186,9 +207,8 @@ async def _fetch_series(self, base, year, cache_tag): ) for stamp, values in periods.items(): totals[stamp] = totals.get(stamp, 0.0) + values["pv_estimate"] - any_data = True - return totals if any_data else {} + return totals def _derive_p10_ratios(self, actual_periods, forecast_periods, forecast_available): """Derive each month's P10 ratio from the measured actual/forecast daily energy error.""" diff --git a/apps/predbat/tests/test_annual_weather.py b/apps/predbat/tests/test_annual_weather.py index c854f8281..9315915c3 100644 --- a/apps/predbat/tests/test_annual_weather.py +++ b/apps/predbat/tests/test_annual_weather.py @@ -17,6 +17,28 @@ from annual_weather import AnnualWeather, percentile ARRAYS = [{"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}] +TWO_ARRAYS = [ + {"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}, + {"kwp": 3.0, "declination": 45, "azimuth": 180, "efficiency": 0.95}, +] + + +class FakeAnnualStorage: + """Minimal in-memory async Storage stand-in for exercising the cache path with no network I/O.""" + + def __init__(self): + """Start with an empty cache and no recorded saves.""" + self.cache = {} + self.saved_keys = [] + + async def load(self, module, filename): + """Return the cached payload for a module/filename pair, or None if never saved.""" + return self.cache.get((module, filename)) + + async def save(self, module, filename, data, format="json", expiry=None): + """Record a cache write, keyed by module and filename, ignoring format/expiry.""" + self.cache[(module, filename)] = data + self.saved_keys.append((module, filename)) def build_hourly(start_day, days, peak_gti): @@ -72,10 +94,13 @@ async def fake_fetch(url): print("Test: January's P10 ratio reflects the measured 0.8 forecast error") ratio = year.p10_ratio(1) - # The shared solar model applies a GTI-dependent cell-temperature derate (Task 1), so the - # measured ratio sits a little above the raw 800/1000 GTI ratio; 0.02 comfortably covers that. - if abs(ratio - 0.8) > 0.02: - print(" ERROR: expected a P10 ratio near 0.8, got {}".format(ratio)) + # Every day in the fixture uses the identical curve, so the day-to-day ratio - and hence + # the P10 order statistic over it - is fully deterministic. The shared solar model's + # GTI-dependent cell-temperature derate (Task 1) means this is not the raw 800/1000 GTI + # ratio: a naive implementation that skipped the temperature derate would land on exactly + # 0.8, so a tight tolerance around the true measured value also catches that regression. + if abs(ratio - 0.8162221180559525) > 1e-6: + print(" ERROR: expected the measured P10 ratio of 0.8162221180559525, got {}".format(ratio)) failed = True print("Test: actual daily energy is below forecast daily energy") @@ -97,13 +122,27 @@ async def fake_fetch(url): print("Test: pv_minutes covers a 48 hour window and is keyed by absolute minute") two_day = year.pv_minutes("actual", midnight, 48 * 60) - if max(two_day.keys()) >= 48 * 60: - print(" ERROR: pv_minutes must not emit minutes at or beyond the window length") - failed = True if abs(sum(two_day.values()) - (actual_total + year.daily_actual_kwh(date(2025, 1, 11)))) > 0.01: print(" ERROR: the 48 hour window should equal two days of actual energy") failed = True + print("Test: pv_minutes discards minutes at or beyond a window that ends mid-hour") + # 32.5 hours: deliberately not hour-aligned, and lands inside an hour with non-zero + # irradiance in the fixture curve (hour 8 of the day, curve value 0.5), so the cut is + # meaningful rather than falling in an always-zero night hour. + partial_window_minutes = 32 * 60 + 30 + partial_window = year.pv_minutes("actual", midnight, partial_window_minutes) + if max(partial_window.keys()) >= partial_window_minutes: + print(" ERROR: pv_minutes must not emit minutes at or beyond the window length") + failed = True + dropped_minute = partial_window_minutes + 10 + if dropped_minute not in two_day: + print(" ERROR: test fixture problem: minute {} should exist in the full 48 hour window".format(dropped_minute)) + failed = True + if dropped_minute in partial_window: + print(" ERROR: minute {} is beyond the requested {} minute window and must be dropped".format(dropped_minute, partial_window_minutes)) + failed = True + print("Test: pv_minutes_p10 scales the forecast series by the month ratio") p10_minutes = year.pv_minutes_p10(midnight, 24 * 60, 1) expected = forecast_total * year.p10_ratio(1) @@ -150,4 +189,134 @@ async def sparse_fetch(url): print(" ERROR: a four-day month should fall back to 0.7, got {}".format(sparse.p10_ratio(1))) failed = True + print("Test: the P10 order statistic is the 10th percentile day, not the mean or the minimum") + # Build February directly at the _derive_p10_ratios level, bypassing the GTI/solar-model + # conversion, so the per-day actual/forecast ratios are exact and independently chosen: + # 20 days, sorted ratios [0.5, 0.55, 0.9 x18]. With n=20 and fraction=0.10 the order + # statistic index is ceil(20*0.10)-1 = 1, i.e. the *second* smallest sample (0.55) - clearly + # distinct from the minimum (0.5), the mean (0.8625) and the median/mode (0.9). + february_ratios = [0.5, 0.55] + [0.9] * 18 + february_actual_periods = {} + february_forecast_periods = {} + for day_index, day_ratio in enumerate(february_ratios, start=1): + stamp = pytz.utc.localize(datetime(2025, 2, day_index, 12, 0)) + february_forecast_periods[stamp] = 10.0 + february_actual_periods[stamp] = 10.0 * day_ratio + february_ratio_map, february_fallback_months = weather._derive_p10_ratios(february_actual_periods, february_forecast_periods, True) + expected_february_p10 = sorted(february_ratios)[1] + if abs(february_ratio_map.get(2, -1.0) - expected_february_p10) > 1e-9: + print(" ERROR: expected February's P10 ratio to be the 10th percentile day {}, got {}".format(expected_february_p10, february_ratio_map.get(2))) + failed = True + if abs(february_ratio_map.get(2, -1.0) - min(february_ratios)) < 1e-9: + print(" ERROR: February's P10 ratio must not equal the minimum day's ratio") + failed = True + mean_february_ratio = sum(february_ratios) / len(february_ratios) + if abs(february_ratio_map.get(2, -1.0) - mean_february_ratio) < 1e-9: + print(" ERROR: February's P10 ratio must not equal the mean of the month's ratios") + failed = True + if 2 in february_fallback_months: + print(" ERROR: 20 usable day-pairs should not trigger the flat-derate fallback") + failed = True + + print("Test: a cached response is only written when the payload is usable, and a cache hit avoids re-fetching") + cache_storage = FakeAnnualStorage() + fetch_calls = {"n": 0} + + async def counting_fetch(url): + """Return the good fixture payloads while counting how many times it is called.""" + fetch_calls["n"] += 1 + return actual_payload if "archive-api" in url else forecast_payload + + cached_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, storage=cache_storage, fetch_json=counting_fetch) + cached_year_first = asyncio.run(cached_weather.fetch(2025)) + if len(cache_storage.saved_keys) != 2: + print(" ERROR: a usable actual payload and a usable forecast payload should both be cached, got {} saves".format(len(cache_storage.saved_keys))) + failed = True + calls_after_first_fetch = fetch_calls["n"] + cached_year_second = asyncio.run(cached_weather.fetch(2025)) + if fetch_calls["n"] != calls_after_first_fetch: + print(" ERROR: a cache hit should not call fetch_json again, but call count went from {} to {}".format(calls_after_first_fetch, fetch_calls["n"])) + failed = True + first_total = sum(cached_year_first.pv_minutes("actual", midnight, 24 * 60).values()) + second_total = sum(cached_year_second.pv_minutes("actual", midnight, 24 * 60).values()) + if abs(first_total - second_total) > 1e-9: + print(" ERROR: a cache hit should reproduce the same result as a fresh fetch, got {} vs {}".format(first_total, second_total)) + failed = True + + print("Test: a truncated payload is rejected and never cached") + truncated_storage = FakeAnnualStorage() + truncated_payload = build_hourly(start, 5, 800.0) + truncated_payload["hourly"]["global_tilted_irradiance"] = truncated_payload["hourly"]["global_tilted_irradiance"][:-5] + + async def truncated_fetch(url): + """Always return a payload whose irradiance list is shorter than its time list.""" + return truncated_payload + + truncated_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, storage=truncated_storage, fetch_json=truncated_fetch) + truncated_year = asyncio.run(truncated_weather.fetch(2025)) + if truncated_year.forecast_available: + print(" ERROR: a truncated payload must not count as an available forecast series") + failed = True + if truncated_year.actual_periods: + print(" ERROR: a truncated payload must not be used for the actual series either") + failed = True + if truncated_storage.saved_keys: + print(" ERROR: a truncated payload must never be cached, but {} saves were recorded".format(truncated_storage.saved_keys)) + failed = True + + print("Test: a cache entry poisoned before this guard existed is discarded and re-fetched, not trusted forever") + poisoned_storage = FakeAnnualStorage() + poisoned_storage.cache[("annual", "weather_actual_2025_0_51.5_-0.1")] = truncated_payload + recovering_calls = {"n": 0} + + async def recovering_fetch(url): + """Return good fixture payloads, counting calls to prove the poisoned cache was bypassed.""" + recovering_calls["n"] += 1 + return actual_payload if "archive-api" in url else forecast_payload + + recovering_weather = AnnualWeather(ARRAYS, latitude=51.5, longitude=-0.1, log=print, storage=poisoned_storage, fetch_json=recovering_fetch) + recovering_year = asyncio.run(recovering_weather.fetch(2025)) + if not recovering_year.has_actual(date(2025, 1, 10)): + print(" ERROR: a poisoned cache entry should be discarded and re-fetched rather than trusted forever") + failed = True + if recovering_calls["n"] == 0: + print(" ERROR: fetch_json should have been called since the cached actual payload was poisoned") + failed = True + + print("Test: one array failing to parse abandons the whole series rather than blending a partial result") + + async def forecast_array_fails_fetch(url): + """Serve good actuals for both arrays but fail the second array's forecast request.""" + if "archive-api" in url: + return actual_payload + if "tilt=45" in url: + return None + return forecast_payload + + partial_forecast_weather = AnnualWeather(TWO_ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=forecast_array_fails_fetch, p10_fallback=0.7) + partial_forecast_year = asyncio.run(partial_forecast_weather.fetch(2025)) + if partial_forecast_year.forecast_available: + print(" ERROR: forecast_available should be False when one of several arrays fails to parse") + failed = True + if 1 not in partial_forecast_year.fallback_months: + print(" ERROR: a forecast series abandoned by one failed array should still record the flat-derate fallback") + failed = True + if not partial_forecast_year.has_actual(date(2025, 1, 10)): + print(" ERROR: the actual series should still be populated when only the forecast fetch failed") + failed = True + + print("Test: the actual series is also abandoned entirely when one of several arrays fails on that side") + + async def actual_array_fails_fetch(url): + """Fail the second array's actual request while every forecast request succeeds.""" + if "archive-api" in url: + return None if "tilt=45" in url else actual_payload + return forecast_payload + + partial_actual_weather = AnnualWeather(TWO_ARRAYS, latitude=51.5, longitude=-0.1, log=print, fetch_json=actual_array_fails_fetch, p10_fallback=0.7) + partial_actual_year = asyncio.run(partial_actual_weather.fetch(2025)) + if partial_actual_year.has_actual(date(2025, 1, 10)): + print(" ERROR: the actual series should be empty when one of several arrays fails to parse") + failed = True + return failed From b7a66100828295dc05bf517ff49cb18604fade43 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 21:02:09 +0200 Subject: [PATCH 017/119] feat(annual): add historical tariff resolution module Resolves import/export rates for a specific historical date from an Octopus product URL (period_from/period_to, paginated, cached per month) or a static basic-rates structure, for the annual electricity cost prediction tool. --- apps/predbat/annual_tariff.py | 192 +++++++++++++++++++++++ apps/predbat/tests/test_annual_tariff.py | 147 +++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 341 insertions(+) create mode 100644 apps/predbat/annual_tariff.py create mode 100644 apps/predbat/tests/test_annual_tariff.py diff --git a/apps/predbat/annual_tariff.py b/apps/predbat/annual_tariff.py new file mode 100644 index 000000000..210828fef --- /dev/null +++ b/apps/predbat/annual_tariff.py @@ -0,0 +1,192 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Historical tariff resolution for the annual prediction tool. + +Resolves import and export rates for a specific past date, either from an +Octopus product URL using period_from/period_to, or from a static basic rates +structure. Fetching is done a month at a time and sliced per sampled day. +""" + +import calendar +from datetime import datetime, timedelta + +import aiohttp +import pytz + +from utils import minute_data + +MINUTES_PER_DAY = 24 * 60 + + +def build_period_url(base_url, start_utc, end_utc): + """Append period_from/period_to to an Octopus rates URL, preserving existing query parameters.""" + separator = "&" if "?" in base_url else "?" + return "{}{}period_from={}&period_to={}&page_size=1500".format(base_url, separator, start_utc.strftime("%Y-%m-%dT%H:%MZ"), end_utc.strftime("%Y-%m-%dT%H:%MZ")) + + +class AnnualTariff: + """Import and export rates for arbitrary historical dates.""" + + def __init__(self, config, log, predbat, storage=None, fetch_json=None): + """Configure the tariff from the annual config's ``tariff`` block. + + Octopus product codes are region-suffixed. ``resolve_arg`` substitutes + ``{dno_region}`` from ``predbat.args``, so the region is injected there + first. Without it a URL silently 404s and the month is reported + unavailable, which looks like an outage rather than a config mistake. + """ + self.config = config + self.log = log + self.predbat = predbat + self.storage = storage + self.fetch_json = fetch_json or self._default_fetch_json + if config.get("dno_region"): + predbat.args["dno_region"] = config["dno_region"] + self.import_url = self._resolve_url(config.get("import_octopus_url"), "import_octopus_url") + self.export_url = self._resolve_url(config.get("export_octopus_url"), "export_octopus_url") + self.basic_import = config.get("rates_import") + self.basic_export = config.get("rates_export") + self.standing_charge_p_per_day = float(config.get("standing_charge_p_per_day", 0.0)) + # Keyed by (year, month); each value is a dict of tz-aware UTC stamp to rate + self.import_rates = {} + self.export_rates = {} + self.available = set() + + def _resolve_url(self, url, name): + """Substitute templated arguments such as {dno_region} into a tariff URL.""" + if not url: + return None + return self.predbat.resolve_arg(name, url, indirect=False) + + async def _default_fetch_json(self, url): + """Download and decode one JSON document, returning None on any failure.""" + try: + async with aiohttp.ClientSession() as session: + async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=60)) as response: + if response.status not in [200, 201]: + self.log("Warn: Annual: Octopus rate request to {} returned {}".format(url, response.status)) + return None + return await response.json() + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + self.log("Warn: Annual: Octopus rate request to {} failed: {}".format(url, error)) + return None + + async def _download_octopus(self, base_url, start_utc, end_utc, cache_key): + """Download every page of Octopus rates for a date range, returning raw result rows.""" + if self.storage: + cached = await self.storage.load("annual", cache_key) + if isinstance(cached, list) and cached: + return cached + + url = build_period_url(base_url, start_utc, end_utc) + rows = [] + pages = 0 + truncated = False + while url and pages < 10: + data = await self.fetch_json(url) + if not data or "results" not in data: + # A failed page means we do not know what we are missing. Caching a + # partial month would permanently pin wrong rates for that month. + self.log("Warn: Annual: rate download for {} stopped early at page {}; not caching a partial result".format(cache_key, pages)) + truncated = True + break + rows += data["results"] + url = data.get("next", None) + pages += 1 + + if truncated: + return [] + if rows and self.storage: + await self.storage.save("annual", cache_key, rows, format="json") + return rows + + @staticmethod + def _rows_to_stamped_rates(rows, start_utc, days): + """Convert Octopus rate rows into a dict of tz-aware UTC stamp to rate. + + Reuses ``minute_data`` exactly as ``octopus.py`` does, then re-keys the + minute offsets back onto absolute timestamps so a single monthly download + can be sliced for any day within it. + """ + parsed, _ = minute_data(rows, days + 1, start_utc, "value_inc_vat", "valid_from", backwards=False, to_key="valid_to") + return {start_utc + timedelta(minutes=minute): rate for minute, rate in parsed.items()} + + async def fetch_month(self, year, month): + """Fetch (or synthesise) the rates covering one calendar month plus a one day buffer. + + Returns True when usable rates exist for the month. The buffer day lets the + last sampled day of the month complete its 48 hour plan. + """ + key = (year, month) + if self.import_url or self.export_url: + days_in_month = calendar.monthrange(year, month)[1] + start_utc = pytz.utc.localize(datetime(year, month, 1)) + end_utc = start_utc + timedelta(days=days_in_month + 2) + days = days_in_month + 2 + + import_rates = {} + export_rates = {} + if self.import_url: + rows = await self._download_octopus(self.import_url, start_utc, end_utc, "rates_import_{}_{:02d}".format(year, month)) + if not rows: + self.log("Warn: Annual: no import rates available for {}-{:02d}".format(year, month)) + return False + import_rates = self._rows_to_stamped_rates(rows, start_utc, days) + if self.export_url: + rows = await self._download_octopus(self.export_url, start_utc, end_utc, "rates_export_{}_{:02d}".format(year, month)) + if rows: + export_rates = self._rows_to_stamped_rates(rows, start_utc, days) + else: + self.log("Warn: Annual: no export rates available for {}-{:02d}, treating export as unpaid".format(year, month)) + + if not import_rates: + return False + self.import_rates[key] = import_rates + self.export_rates[key] = export_rates + self.available.add(key) + return True + + # Basic rates repeat a fixed daily pattern, so nothing needs downloading + self.available.add(key) + return True + + def month_available(self, year, month): + """Return True when usable rates exist for the given month.""" + return (year, month) in self.available + + def _basic_window(self, info, name, minutes): + """Expand a basic rates structure across the requested window.""" + rates = self.predbat.basic_rates(info, name) + return {minute: rates.get(minute % MINUTES_PER_DAY, 0.0) for minute in range(minutes)} + + def rates_for(self, midnight_utc, minutes): + """Return (import, export) rate dicts keyed by absolute minute from ``midnight_utc``.""" + key = (midnight_utc.year, midnight_utc.month) + + if self.import_url or self.export_url: + import_stamped = self.import_rates.get(key, {}) + export_stamped = self.export_rates.get(key, {}) + # A 48 hour window starting late in a month spills into the next month's download + next_key = (midnight_utc.year, midnight_utc.month + 1) if midnight_utc.month < 12 else (midnight_utc.year + 1, 1) + import_stamped = dict(import_stamped) + import_stamped.update(self.import_rates.get(next_key, {})) + export_stamped = dict(export_stamped) + export_stamped.update(self.export_rates.get(next_key, {})) + + rate_import = {} + rate_export = {} + for minute in range(minutes): + stamp = midnight_utc + timedelta(minutes=minute) + if stamp in import_stamped: + rate_import[minute] = import_stamped[stamp] + if stamp in export_stamped: + rate_export[minute] = export_stamped[stamp] + return rate_import, rate_export + + rate_import = self._basic_window(self.basic_import or [], "rates_import", minutes) + rate_export = self._basic_window(self.basic_export or [], "rates_export", minutes) + return rate_import, rate_export diff --git a/apps/predbat/tests/test_annual_tariff.py b/apps/predbat/tests/test_annual_tariff.py new file mode 100644 index 000000000..34b2685c4 --- /dev/null +++ b/apps/predbat/tests/test_annual_tariff.py @@ -0,0 +1,147 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction tariff module.""" + +import asyncio +from datetime import datetime, timedelta + +import pytz + +from annual_tariff import AnnualTariff, build_period_url + + +def build_agile_results(start_day, days, base_rate): + """Build a synthetic Octopus half-hourly rate payload with a repeating daily shape.""" + results = [] + stamp = pytz.utc.localize(datetime(start_day.year, start_day.month, start_day.day)) + for slot in range(days * 48): + valid_from = stamp + timedelta(minutes=30 * slot) + valid_to = valid_from + timedelta(minutes=30) + # Cheap overnight, expensive in the evening peak + hour = valid_from.hour + rate = base_rate * (0.3 if hour < 5 else (2.0 if 16 <= hour < 19 else 1.0)) + results.append( + { + "value_inc_vat": round(rate, 4), + "valid_from": valid_from.strftime("%Y-%m-%dT%H:%M:%SZ"), + "valid_to": valid_to.strftime("%Y-%m-%dT%H:%M:%SZ"), + } + ) + return results + + +def test_annual_tariff(my_predbat): + """Verify Octopus date-ranged rate fetching, pagination, slicing and basic rates.""" + failed = False + print("**** Testing annual_tariff ****") + from datetime import date + + print("Test: build_period_url appends the date range without losing existing query parameters") + start = pytz.utc.localize(datetime(2025, 3, 1)) + end = pytz.utc.localize(datetime(2025, 4, 1)) + plain = build_period_url("https://example.com/rates/", start, end) + if "?period_from=2025-03-01T00:00Z" not in plain or "period_to=2025-04-01T00:00Z" not in plain: + print(" ERROR: expected a period range in {}".format(plain)) + failed = True + existing = build_period_url("https://example.com/rates/?page_size=100", start, end) + if "&period_from=" not in existing or "page_size=100" not in existing: + print(" ERROR: existing query parameters must be preserved, got {}".format(existing)) + failed = True + + print("Test: an Octopus URL tariff resolves rates for a specific date, following pagination") + page_two = {"results": build_agile_results(date(2025, 3, 16), 16, 20.0), "next": None} + page_one = {"results": build_agile_results(date(2025, 3, 1), 15, 20.0), "next": "https://example.com/page2"} + + calls = [] + + async def fake_fetch(url): + """Serve two pages of rate data and record the URLs requested.""" + calls.append(url) + return page_two if "page2" in url else page_one + + config = {"import_octopus_url": "https://example.com/import/", "export_octopus_url": "https://example.com/export/", "standing_charge_p_per_day": 60.0} + tariff = AnnualTariff(config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + ok = asyncio.run(tariff.fetch_month(2025, 3)) + if not ok: + print(" ERROR: fetch_month should succeed with valid payloads") + failed = True + if not tariff.month_available(2025, 3): + print(" ERROR: March 2025 should be reported as available") + failed = True + if len(calls) != 4: + print(" ERROR: expected 4 requests (2 pages x import and export), got {}".format(len(calls))) + failed = True + + print("Test: rates_for returns a 48 hour window keyed by absolute minute") + midnight = pytz.utc.localize(datetime(2025, 3, 10)) + rate_import, rate_export = tariff.rates_for(midnight, 48 * 60) + if len(rate_import) < 48 * 60: + print(" ERROR: expected at least {} import rate minutes, got {}".format(48 * 60, len(rate_import))) + failed = True + if rate_import.get(0) is None: + print(" ERROR: minute 0 must have an import rate") + failed = True + # 02:00 is in the cheap overnight band, 17:00 is in the peak band + if not rate_import[120] < rate_import[17 * 60]: + print(" ERROR: overnight rate {} should be below peak rate {}".format(rate_import[120], rate_import[17 * 60])) + failed = True + if abs(rate_import[120] - 6.0) > 0.01: + print(" ERROR: overnight rate expected 6.0, got {}".format(rate_import[120])) + failed = True + + print("Test: the second day of the window carries the following day's rates") + if rate_import.get(24 * 60 + 120) is None: + print(" ERROR: the second day of the window must be populated") + failed = True + + print("Test: a failed download reports the month as unavailable rather than returning zeros") + + async def failing_fetch(url): + """Simulate a download failure.""" + return None + + broken = AnnualTariff(config, log=print, predbat=my_predbat, fetch_json=failing_fetch) + if asyncio.run(broken.fetch_month(2025, 4)): + print(" ERROR: fetch_month should report failure when the download fails") + failed = True + if broken.month_available(2025, 4): + print(" ERROR: a failed month must not be reported as available") + failed = True + + print("Test: basic rates repeat a fixed daily pattern across the window") + basic_config = { + "rates_import": [{"start": "00:00:00", "end": "05:00:00", "rate": 7.0}, {"start": "05:00:00", "end": "00:00:00", "rate": 30.0}], + "rates_export": [{"rate": 15.0}], + "standing_charge_p_per_day": 45.0, + } + basic = AnnualTariff(basic_config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + if not asyncio.run(basic.fetch_month(2025, 3)): + print(" ERROR: basic rates should always be available") + failed = True + basic_import, basic_export = basic.rates_for(midnight, 48 * 60) + if abs(basic_import[120] - 7.0) > 0.001: + print(" ERROR: basic overnight import expected 7.0, got {}".format(basic_import[120])) + failed = True + if abs(basic_import[10 * 60] - 30.0) > 0.001: + print(" ERROR: basic daytime import expected 30.0, got {}".format(basic_import[10 * 60])) + failed = True + if abs(basic_import[24 * 60 + 120] - 7.0) > 0.001: + print(" ERROR: basic rates must repeat on day two, got {}".format(basic_import[24 * 60 + 120])) + failed = True + if abs(basic_export[10 * 60] - 15.0) > 0.001: + print(" ERROR: basic export expected 15.0, got {}".format(basic_export[10 * 60])) + failed = True + + print("Test: standing charge is carried through from config") + if abs(tariff.standing_charge_p_per_day - 60.0) > 0.001: + print(" ERROR: standing charge expected 60.0, got {}".format(tariff.standing_charge_p_per_day)) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 18c4a983a..b849293e7 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -128,6 +128,7 @@ from tests.test_annual_profiles import test_annual_profiles from tests.test_annual_load import test_annual_load, test_annual_load_octopus from tests.test_annual_weather import test_annual_weather +from tests.test_annual_tariff import test_annual_tariff from tests.test_rate_add_io_slots import run_rate_add_io_slots_tests from tests.test_battery_curve_keys import run_battery_curve_keys_tests from tests.test_balance_inverters import run_balance_inverters_tests @@ -325,6 +326,7 @@ def main(): ("annual_load", test_annual_load, "Annual prediction load profile tests", False), ("annual_load_octopus", test_annual_load_octopus, "Annual prediction Octopus consumption tests", False), ("annual_weather", test_annual_weather, "Annual prediction Open-Meteo weather tests", False), + ("annual_tariff", test_annual_tariff, "Annual prediction tariff tests", False), ("solax", run_solax_tests, "SolaX API tests", False), ("sigenergy", run_sigenergy_tests, "Sigenergy Cloud API tests", False), ("iboost_smart", run_iboost_smart_tests, "iBoost smart tests", False), From b1ba8aacc71870ad642fcef9c12584aa0d473a40 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 21:15:36 +0200 Subject: [PATCH 018/119] fix(annual): close tariff download/pricing gaps found in review - Treat a hit page-cap the same as a failed page: stop caching a truncated Octopus download instead of silently persisting it. - Fix month_available disagreeing with rates_for for an export-only configuration (no import URL). - Refuse to report a month available when neither an Octopus URL nor rates_import is configured, instead of silently pricing at zero. - Only append page_size to the Octopus URL when the caller didn't already specify one, so an explicit page_size is no longer overridden. - Stop mutating predbat.args for {dno_region} substitution (use resolve_arg's extra_args instead), so a live Octopus component's own region is never clobbered. - Warn and ignore day_of_week/date basic-rate entries, which cannot be honoured during a historical replay (they anchor to today's date, not the sampled historical date). - Cache the computed basic rate table instead of recomputing and re-logging it on every rates_for call. Adds covering tests for all of the above, including a December to January month-boundary merge case. --- apps/predbat/annual_tariff.py | 101 ++++++++--- apps/predbat/tests/test_annual_tariff.py | 203 ++++++++++++++++++++++- 2 files changed, 278 insertions(+), 26 deletions(-) diff --git a/apps/predbat/annual_tariff.py b/apps/predbat/annual_tariff.py index 210828fef..d12934e75 100644 --- a/apps/predbat/annual_tariff.py +++ b/apps/predbat/annual_tariff.py @@ -23,9 +23,17 @@ def build_period_url(base_url, start_utc, end_utc): - """Append period_from/period_to to an Octopus rates URL, preserving existing query parameters.""" + """Append period_from/period_to to an Octopus rates URL, preserving existing query parameters. + + ``page_size`` is only appended when the caller's URL does not already specify + one, so an explicit ``page_size=100`` is not silently overridden by a second, + conflicting ``page_size=1500`` appended after it. + """ separator = "&" if "?" in base_url else "?" - return "{}{}period_from={}&period_to={}&page_size=1500".format(base_url, separator, start_utc.strftime("%Y-%m-%dT%H:%MZ"), end_utc.strftime("%Y-%m-%dT%H:%MZ")) + url = "{}{}period_from={}&period_to={}".format(base_url, separator, start_utc.strftime("%Y-%m-%dT%H:%MZ"), end_utc.strftime("%Y-%m-%dT%H:%MZ")) + if "page_size=" not in base_url: + url += "&page_size=1500" + return url class AnnualTariff: @@ -34,20 +42,21 @@ class AnnualTariff: def __init__(self, config, log, predbat, storage=None, fetch_json=None): """Configure the tariff from the annual config's ``tariff`` block. - Octopus product codes are region-suffixed. ``resolve_arg`` substitutes - ``{dno_region}`` from ``predbat.args``, so the region is injected there - first. Without it a URL silently 404s and the month is reported - unavailable, which looks like an outage rather than a config mistake. + Octopus product codes are region-suffixed. ``resolve_arg``'s ``extra_args`` + substitutes ``{dno_region}`` from the config directly, without writing it + into ``predbat.args`` first, so a live Octopus component configured for a + different region is never clobbered by this tariff's region. Without a + region a URL silently 404s and the month is reported unavailable, which + looks like an outage rather than a config mistake. """ self.config = config self.log = log self.predbat = predbat self.storage = storage self.fetch_json = fetch_json or self._default_fetch_json - if config.get("dno_region"): - predbat.args["dno_region"] = config["dno_region"] - self.import_url = self._resolve_url(config.get("import_octopus_url"), "import_octopus_url") - self.export_url = self._resolve_url(config.get("export_octopus_url"), "export_octopus_url") + dno_region = config.get("dno_region") + self.import_url = self._resolve_url(config.get("import_octopus_url"), "import_octopus_url", dno_region) + self.export_url = self._resolve_url(config.get("export_octopus_url"), "export_octopus_url", dno_region) self.basic_import = config.get("rates_import") self.basic_export = config.get("rates_export") self.standing_charge_p_per_day = float(config.get("standing_charge_p_per_day", 0.0)) @@ -55,12 +64,18 @@ def __init__(self, config, log, predbat, storage=None, fetch_json=None): self.import_rates = {} self.export_rates = {} self.available = set() - - def _resolve_url(self, url, name): - """Substitute templated arguments such as {dno_region} into a tariff URL.""" + # Lazily computed, cached 1440-minute basic rate tables (see _basic_table); + # computing these afresh on every rates_for call would re-run basic_rates's + # per-entry logging and load_scaling_dynamic mutation roughly 365 times a year. + self._basic_import_table = None + self._basic_export_table = None + + def _resolve_url(self, url, name, dno_region=None): + """Substitute templated arguments such as {dno_region} into a tariff URL, without mutating predbat.args.""" if not url: return None - return self.predbat.resolve_arg(name, url, indirect=False) + extra_args = {"dno_region": dno_region} if dno_region else None + return self.predbat.resolve_arg(name, url, indirect=False, extra_args=extra_args) async def _default_fetch_json(self, url): """Download and decode one JSON document, returning None on any failure.""" @@ -98,7 +113,12 @@ async def _download_octopus(self, base_url, start_utc, end_utc, cache_key): url = data.get("next", None) pages += 1 - if truncated: + # A run only reaches its natural end when `url` runs out on its own. If a page + # request failed, or the safety cap was hit while pages remained, `rows` is a + # truncated download - it must not be parsed, used, or cached as if complete. + if truncated or url: + if not truncated: + self.log("Warn: Annual: rate download for {} hit the {}-page cap with more pages remaining; not caching a partial result".format(cache_key, pages)) return [] if rows and self.storage: await self.storage.save("annual", cache_key, rows, format="json") @@ -143,14 +163,23 @@ async def fetch_month(self, year, month): else: self.log("Warn: Annual: no export rates available for {}-{:02d}, treating export as unpaid".format(year, month)) - if not import_rates: + # Only import failing makes the month unusable: rates_for's gate is + # `self.import_url or self.export_url`, so an export-only configuration + # (no import_url) must not be marked unavailable just because + # `import_rates` is the empty dict it was initialised to above. + if self.import_url and not import_rates: return False self.import_rates[key] = import_rates self.export_rates[key] = export_rates self.available.add(key) return True - # Basic rates repeat a fixed daily pattern, so nothing needs downloading + # Basic rates repeat a fixed daily pattern, so nothing needs downloading - + # but without an import rate source there is nothing to price import with, + # and reporting the month available would silently price a year at zero. + if not self.basic_import: + self.log("Warn: Annual: no rate source configured for {}-{:02d} (no Octopus import URL and no rates_import); refusing to report this month as available".format(year, month)) + return False self.available.add(key) return True @@ -158,9 +187,37 @@ def month_available(self, year, month): """Return True when usable rates exist for the given month.""" return (year, month) in self.available - def _basic_window(self, info, name, minutes): - """Expand a basic rates structure across the requested window.""" - rates = self.predbat.basic_rates(info, name) + def _basic_table(self, info, name, cache_attr): + """Compute and cache the 1440-minute basic rate table for one rate name. + + ``basic_rates`` logs per entry and mutates ``predbat.load_scaling_dynamic``, + so recomputing it on every ``rates_for`` call (roughly 365 times a year) + would repeat that work for an identical table. Entries keyed by + ``day_of_week``/``date`` are stripped before calling it: ``basic_rates`` + anchors those to ``predbat.midnight`` (the day the tool is run), and + ``rates_for`` then collapses everything to a single repeating day via + ``minute % MINUTES_PER_DAY`` - so honouring them would make a historical + replay depend on today's weekday rather than the sampled historical date. + """ + cached = getattr(self, cache_attr) + if cached is not None: + return cached + usable = [] + ignored = 0 + for entry in info or []: + if isinstance(entry, dict) and ("day_of_week" in entry or "date" in entry): + ignored += 1 + continue + usable.append(entry) + if ignored: + self.log("Warn: Annual: {} contains {} day_of_week/date entries, which are anchored to today's date rather than the sampled historical date and cannot be honoured during an annual replay; ignoring them".format(name, ignored)) + table = self.predbat.basic_rates(usable, name) + setattr(self, cache_attr, table) + return table + + def _basic_window(self, info, name, minutes, cache_attr): + """Expand a cached basic rates table across the requested window.""" + rates = self._basic_table(info, name, cache_attr) return {minute: rates.get(minute % MINUTES_PER_DAY, 0.0) for minute in range(minutes)} def rates_for(self, midnight_utc, minutes): @@ -187,6 +244,6 @@ def rates_for(self, midnight_utc, minutes): rate_export[minute] = export_stamped[stamp] return rate_import, rate_export - rate_import = self._basic_window(self.basic_import or [], "rates_import", minutes) - rate_export = self._basic_window(self.basic_export or [], "rates_export", minutes) + rate_import = self._basic_window(self.basic_import or [], "rates_import", minutes, "_basic_import_table") + rate_export = self._basic_window(self.basic_export or [], "rates_export", minutes, "_basic_export_table") return rate_import, rate_export diff --git a/apps/predbat/tests/test_annual_tariff.py b/apps/predbat/tests/test_annual_tariff.py index 34b2685c4..cf274e3e7 100644 --- a/apps/predbat/tests/test_annual_tariff.py +++ b/apps/predbat/tests/test_annual_tariff.py @@ -10,7 +10,7 @@ """Tests for the annual prediction tariff module.""" import asyncio -from datetime import datetime, timedelta +from datetime import date, datetime, timedelta import pytz @@ -37,11 +37,50 @@ def build_agile_results(start_day, days, base_rate): return results +def build_month_aware_fetch(rate_by_month): + """Build a fetch stub whose synthesised page's base rate depends on the requested month. + + Parses ``period_from``/``period_to`` out of the requested URL and serves a single, + non-paginated page covering exactly that range. Because the base rate depends on the + requested start month, a test can tell which month's download actually produced a + given stamped rate - this is what makes it possible to prove ``rates_for``'s + month-boundary merge picks up the *next* month's own download for dates that spill + into it, rather than silently reusing the requesting month's download for those dates. + """ + + async def fetch(url): + """Serve a single page of synthetic rates sized to the requested period_from/period_to.""" + start_str = url.split("period_from=")[1][:10] + end_str = url.split("period_to=")[1][:10] + start_day = date(*(int(part) for part in start_str.split("-"))) + end_day = date(*(int(part) for part in end_str.split("-"))) + days = (end_day - start_day).days + base_rate = rate_by_month.get(start_day.month, 20.0) + return {"results": build_agile_results(start_day, days, base_rate), "next": None} + + return fetch + + +class FakeAnnualStorage: + """Minimal in-memory storage stub with async save/load, used to prove a truncated download is never cached.""" + + def __init__(self): + """Start with an empty in-memory store.""" + self.store = {} + + async def load(self, namespace, key): + """Return the previously saved value for a key, or None if nothing was saved.""" + return self.store.get((namespace, key)) + + async def save(self, namespace, key, value, format="json"): + """Record the value under the given key as if it had been persisted to disk.""" + self.store[(namespace, key)] = value + + def test_annual_tariff(my_predbat): - """Verify Octopus date-ranged rate fetching, pagination, slicing and basic rates.""" + """Verify Octopus date-ranged rate fetching, pagination, slicing, caching guards and basic rates.""" failed = False print("**** Testing annual_tariff ****") - from datetime import date print("Test: build_period_url appends the date range without losing existing query parameters") start = pytz.utc.localize(datetime(2025, 3, 1)) @@ -54,6 +93,9 @@ def test_annual_tariff(my_predbat): if "&period_from=" not in existing or "page_size=100" not in existing: print(" ERROR: existing query parameters must be preserved, got {}".format(existing)) failed = True + if existing.count("page_size=100") != 1 or "page_size=1500" in existing: + print(" ERROR: an explicit page_size must be preserved as-is, not duplicated or overridden, got {}".format(existing)) + failed = True print("Test: an Octopus URL tariff resolves rates for a specific date, following pagination") page_two = {"results": build_agile_results(date(2025, 3, 16), 16, 20.0), "next": None} @@ -79,7 +121,7 @@ async def fake_fetch(url): print(" ERROR: expected 4 requests (2 pages x import and export), got {}".format(len(calls))) failed = True - print("Test: rates_for returns a 48 hour window keyed by absolute minute") + print("Test: rates_for returns a 48 hour window keyed by absolute minute, for both import and export") midnight = pytz.utc.localize(datetime(2025, 3, 10)) rate_import, rate_export = tariff.rates_for(midnight, 48 * 60) if len(rate_import) < 48 * 60: @@ -95,12 +137,58 @@ async def fake_fetch(url): if abs(rate_import[120] - 6.0) > 0.01: print(" ERROR: overnight rate expected 6.0, got {}".format(rate_import[120])) failed = True + # The export URL is fed the same synthetic payload shape as import in this fixture, + # so its overnight/peak values must match the same 20.0 base rate formula. Asserting + # on actual values (not just presence) proves the export branch is truly exercised - + # deleting it outright would previously still pass this test. + if len(rate_export) < 48 * 60: + print(" ERROR: expected at least {} export rate minutes, got {}".format(48 * 60, len(rate_export))) + failed = True + if abs(rate_export[120] - 6.0) > 0.01: + print(" ERROR: export overnight rate expected 6.0, got {}".format(rate_export[120])) + failed = True + if abs(rate_export[17 * 60] - 40.0) > 0.01: + print(" ERROR: export peak rate expected 40.0, got {}".format(rate_export[17 * 60])) + failed = True print("Test: the second day of the window carries the following day's rates") if rate_import.get(24 * 60 + 120) is None: print(" ERROR: the second day of the window must be populated") failed = True + print("Test: an export-only tariff (no import URL) is reported available when its export download succeeds") + export_only_config = {"export_octopus_url": "https://example.com/export/", "standing_charge_p_per_day": 0.0} + export_only_tariff = AnnualTariff(export_only_config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + if not asyncio.run(export_only_tariff.fetch_month(2025, 3)): + print(" ERROR: an export-only tariff should be available once its export download succeeds") + failed = True + if not export_only_tariff.month_available(2025, 3): + print(" ERROR: March 2025 should be reported as available for an export-only tariff") + failed = True + + print("Test: a 48 hour window starting on the last day of a month carries the following month's own download, across a December to January year boundary") + dec_jan_config = {"import_octopus_url": "https://example.com/import/", "standing_charge_p_per_day": 0.0} + dec_jan_tariff = AnnualTariff(dec_jan_config, log=print, predbat=my_predbat, fetch_json=build_month_aware_fetch({12: 50.0, 1: 90.0})) + if not asyncio.run(dec_jan_tariff.fetch_month(2025, 12)): + print(" ERROR: December 2025 should fetch successfully") + failed = True + if not asyncio.run(dec_jan_tariff.fetch_month(2026, 1)): + print(" ERROR: January 2026 should fetch successfully") + failed = True + boundary_midnight = pytz.utc.localize(datetime(2025, 12, 31)) + boundary_import, _ = dec_jan_tariff.rates_for(boundary_midnight, 48 * 60) + # Day one (31 Dec) overnight reflects December's own download: 50.0 * 0.3 = 15.0 + if abs(boundary_import[120] - 15.0) > 0.01: + print(" ERROR: December overnight rate expected 15.0, got {}".format(boundary_import[120])) + failed = True + # Day two (1 Jan) overnight must reflect January's own download (90.0 * 0.3 = 27.0), + # not December's download merely extending into the buffer day at the old rate (15.0) - + # dropping the next-month merge entirely would leave this at 15.0 and still pass every + # other assertion above, since December's own buffer days also contain stamps for 1 Jan. + if abs(boundary_import[24 * 60 + 120] - 27.0) > 0.01: + print(" ERROR: January overnight rate expected 27.0 (carried from the following month's own fetch), got {}".format(boundary_import[24 * 60 + 120])) + failed = True + print("Test: a failed download reports the month as unavailable rather than returning zeros") async def failing_fetch(url): @@ -115,6 +203,56 @@ async def failing_fetch(url): print(" ERROR: a failed month must not be reported as available") failed = True + print("Test: a failed page and a page-cap hit both refuse to cache a partial download") + + async def failing_page_fetch(url): + """Simulate the very first page request failing outright.""" + return None + + fail_storage = FakeAnnualStorage() + fail_tariff = AnnualTariff({"import_octopus_url": "https://example.com/import/", "standing_charge_p_per_day": 0.0}, log=print, predbat=my_predbat, storage=fail_storage, fetch_json=failing_page_fetch) + if asyncio.run(fail_tariff.fetch_month(2025, 5)): + print(" ERROR: fetch_month should fail when the first page request fails") + failed = True + if fail_storage.store: + print(" ERROR: a failed page must not be cached, got {}".format(fail_storage.store)) + failed = True + + async def never_ending_fetch(url): + """Always report another page is available, to exercise the pagination safety cap.""" + return {"results": build_agile_results(date(2025, 6, 1), 1, 20.0), "next": url + "&more"} + + cap_storage = FakeAnnualStorage() + cap_tariff = AnnualTariff({"import_octopus_url": "https://example.com/import/", "standing_charge_p_per_day": 0.0}, log=print, predbat=my_predbat, storage=cap_storage, fetch_json=never_ending_fetch) + if asyncio.run(cap_tariff.fetch_month(2025, 6)): + print(" ERROR: fetch_month should fail when the page cap is hit with pages still remaining") + failed = True + if cap_storage.store: + print(" ERROR: a page-cap truncation must not be cached, got {}".format(cap_storage.store)) + failed = True + + print("Test: {dno_region} in a tariff URL is substituted from config without mutating predbat.args") + region_calls = [] + + async def region_fetch(url): + """Record the URL requested and serve a minimal single page of results.""" + region_calls.append(url) + return {"results": build_agile_results(date(2025, 7, 1), 1, 20.0), "next": None} + + dno_region_before = my_predbat.args.get("dno_region") + region_config = {"import_octopus_url": "https://example.com/rates-{dno_region}/", "dno_region": "A", "standing_charge_p_per_day": 0.0} + region_tariff = AnnualTariff(region_config, log=print, predbat=my_predbat, fetch_json=region_fetch) + if region_tariff.import_url != "https://example.com/rates-A/": + print(" ERROR: expected the {{dno_region}} template resolved to A, got {}".format(region_tariff.import_url)) + failed = True + asyncio.run(region_tariff.fetch_month(2025, 7)) + if not region_calls or "rates-A/" not in region_calls[0]: + print(" ERROR: expected the requested URL to contain the resolved region, got {}".format(region_calls)) + failed = True + if my_predbat.args.get("dno_region") != dno_region_before: + print(" ERROR: dno_region must not be written into predbat.args (would clobber a live Octopus component's own region)") + failed = True + print("Test: basic rates repeat a fixed daily pattern across the window") basic_config = { "rates_import": [{"start": "00:00:00", "end": "05:00:00", "rate": 7.0}, {"start": "05:00:00", "end": "00:00:00", "rate": 30.0}], @@ -139,6 +277,63 @@ async def failing_fetch(url): print(" ERROR: basic export expected 15.0, got {}".format(basic_export[10 * 60])) failed = True + print("Test: an unconfigured tariff (no URLs and no rates_import) is reported unavailable rather than pricing at an implicit zero") + empty_config = {"standing_charge_p_per_day": 0.0} + empty_tariff = AnnualTariff(empty_config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + if asyncio.run(empty_tariff.fetch_month(2025, 3)): + print(" ERROR: an unconfigured tariff (no import URL and no rates_import) must not be reported as available") + failed = True + if empty_tariff.month_available(2025, 3): + print(" ERROR: an unconfigured tariff must not be reported as available") + failed = True + + print("Test: day_of_week/date basic rate entries are ignored (not applied using today's real weekday) and a warning is logged") + logged = [] + + def capturing_log(message): + """Capture log messages for assertion, while still printing them for visibility.""" + logged.append(message) + print(message) + + weekday_config = { + "rates_import": [{"start": "00:00:00", "end": "00:00:00", "rate": 12.0}, {"start": "00:00:00", "end": "00:00:00", "rate": 99.0, "day_of_week": "1,2,3,4,5"}], + "rates_export": [{"rate": 5.0}], + "standing_charge_p_per_day": 0.0, + } + weekday_tariff = AnnualTariff(weekday_config, log=capturing_log, predbat=my_predbat, fetch_json=fake_fetch) + if not asyncio.run(weekday_tariff.fetch_month(2025, 3)): + print(" ERROR: basic rates should always be available") + failed = True + weekday_import, _ = weekday_tariff.rates_for(midnight, 24 * 60) + if abs(weekday_import[0] - 12.0) > 0.001: + print(" ERROR: the day_of_week entry must be ignored during a historical replay, expected the base rate 12.0, got {}".format(weekday_import[0])) + failed = True + if not any(("day_of_week" in message) and ("ignoring" in message.lower()) for message in logged): + print(" ERROR: expected a warning that day_of_week/date entries cannot be honoured and are being ignored, got {}".format(logged)) + failed = True + + print("Test: basic rates are computed once and cached, not recomputed on every rates_for call") + basic_rates_calls = [] + original_basic_rates = my_predbat.basic_rates + + def counting_basic_rates(info, rtype, prev=None, rate_replicate=None): + """Wrap basic_rates to count how many times it is actually invoked, then delegate to the real implementation.""" + basic_rates_calls.append(rtype) + return original_basic_rates(info, rtype, prev=prev, rate_replicate=rate_replicate) + + my_predbat.basic_rates = counting_basic_rates + try: + caching_tariff = AnnualTariff(basic_config, log=print, predbat=my_predbat, fetch_json=fake_fetch) + asyncio.run(caching_tariff.fetch_month(2025, 3)) + caching_tariff.rates_for(midnight, 48 * 60) + caching_tariff.rates_for(midnight, 48 * 60) + caching_tariff.rates_for(pytz.utc.localize(datetime(2025, 3, 11)), 48 * 60) + finally: + my_predbat.basic_rates = original_basic_rates + if basic_rates_calls.count("rates_import") != 1 or basic_rates_calls.count("rates_export") != 1: + print(" ERROR: expected basic_rates called exactly once per rate type across three rates_for calls, got {}".format(basic_rates_calls)) + failed = True + print("Test: standing charge is carried through from config") if abs(tariff.standing_charge_p_per_day - 60.0) > 0.001: print(" ERROR: standing charge expected 60.0, got {}".format(tariff.standing_charge_p_per_day)) From d83744ec2e1f6877aa04345dd0b567826443afe0 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 21:24:41 +0200 Subject: [PATCH 019/119] feat(annual): add annual prediction config validation --- apps/predbat/annual.py | 191 ++++++++++++++++++++++ apps/predbat/tests/test_annual_config.py | 195 +++++++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 388 insertions(+) create mode 100644 apps/predbat/annual.py create mode 100644 apps/predbat/tests/test_annual_config.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py new file mode 100644 index 000000000..cab7e9c4b --- /dev/null +++ b/apps/predbat/annual.py @@ -0,0 +1,191 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Annual prediction engine. + +Projects a year of household electricity costs using the real Predbat planning +engine, reporting each month under three scenarios: no PV or battery, PV and +battery without Predbat, and with Predbat. Performs no HTTP itself; the weather +and tariff modules own all network access. +""" + +import copy +from datetime import date + +VALID_SHAPES = ["night", "day", "flat"] + +DEFAULT_TIMEZONE = "Europe/London" +DEFAULT_SAMPLES_PER_MONTH = 2 +DEFAULT_PV10_DERATE_FALLBACK = 0.7 +DEFAULT_DECLINATION = 35 +DEFAULT_AZIMUTH = 180 +DEFAULT_EFFICIENCY = 0.95 +DEFAULT_HYBRID = True + +# Substrings that mark a config value as secret and therefore scrubbable +SECRET_MARKERS = ["_key", "password", "token", "secret"] + + +class AnnualConfigError(ValueError): + """Raised when the annual prediction config is invalid or self-contradictory.""" + + +def scrub_secrets(config): + """Return a deep copy of the config with secret-looking values replaced by "xxx". + + Mirrors the redaction ``create_debug_yaml()`` applies, so a results document or + debug dump can never carry an API key. + """ + if isinstance(config, dict): + scrubbed = {} + for key, value in config.items(): + if any(marker in str(key).lower() for marker in SECRET_MARKERS): + scrubbed[key] = "xxx" + else: + scrubbed[key] = scrub_secrets(value) + return scrubbed + if isinstance(config, list): + return [scrub_secrets(item) for item in config] + return config + + +def _validate_solar(raw): + """Normalise the solar array list, applying defaults and rejecting arrays without kwp.""" + if raw is None: + return [] + if isinstance(raw, dict): + raw = [raw] + if not isinstance(raw, list): + raise AnnualConfigError("annual.solar must be a list of arrays") + + arrays = [] + for index, array in enumerate(raw): + if not isinstance(array, dict): + raise AnnualConfigError("annual.solar[{}] must be a mapping".format(index)) + if "kwp" not in array: + raise AnnualConfigError("annual.solar[{}] is missing kwp, the array's peak power in kW".format(index)) + normalised = dict(array) + normalised["kwp"] = float(array["kwp"]) + normalised["declination"] = array.get("declination", DEFAULT_DECLINATION) + normalised["azimuth"] = array.get("azimuth", DEFAULT_AZIMUTH) + normalised["efficiency"] = float(array.get("efficiency", DEFAULT_EFFICIENCY)) + arrays.append(normalised) + return arrays + + +def _validate_battery(raw): + """Normalise the battery block, or return None for a run with no battery.""" + if raw is None: + return None + if not isinstance(raw, dict): + raise AnnualConfigError("annual.battery must be a mapping") + if "size_kwh" not in raw: + raise AnnualConfigError("annual.battery is missing size_kwh") + if "inverter_kw" not in raw: + raise AnnualConfigError("annual.battery is missing inverter_kw") + + inverter_kw = float(raw["inverter_kw"]) + return { + "size_kwh": float(raw["size_kwh"]), + "inverter_kw": inverter_kw, + "export_limit_kw": float(raw.get("export_limit_kw", inverter_kw)), + "hybrid": bool(raw.get("hybrid", DEFAULT_HYBRID)), + "charge_rate_kw": float(raw.get("charge_rate_kw", inverter_kw)), + "discharge_rate_kw": float(raw.get("discharge_rate_kw", inverter_kw)), + } + + +def _validate_load(raw): + """Normalise the load block and enforce the Octopus / manual exclusivity rule.""" + if not isinstance(raw, dict): + raise AnnualConfigError("annual.load is required and must be a mapping") + + octopus = raw.get("octopus") + has_manual = ("annual_kwh" in raw) or ("car_charging_kwh" in raw) + + if octopus and has_manual: + raise AnnualConfigError("annual.load.octopus and annual.load.annual_kwh/car_charging_kwh are mutually exclusive: the Octopus consumption series already includes any car charging, so supplying both would double-count it") + + if not octopus and "annual_kwh" not in raw: + raise AnnualConfigError("annual.load requires either annual_kwh or an octopus block") + + if octopus: + if not isinstance(octopus, dict) or not octopus.get("api_key") or not octopus.get("account_id"): + raise AnnualConfigError("annual.load.octopus requires both api_key and account_id") + return {"octopus": dict(octopus), "shape": raw.get("shape", "flat"), "car_charging_kwh": 0.0} + + shape = raw.get("shape", "flat") + if shape not in VALID_SHAPES: + raise AnnualConfigError("annual.load.shape must be one of {}, got '{}'".format(VALID_SHAPES, shape)) + + return {"annual_kwh": float(raw["annual_kwh"]), "shape": shape, "car_charging_kwh": float(raw.get("car_charging_kwh", 0.0))} + + +def _validate_tariff(raw): + """Normalise the tariff block, requiring at least one import rate source. + + A URL containing {dno_region} with no dno_region supplied is rejected here + rather than left to 404 at fetch time, where it would surface as an + unavailable month and read like an Octopus outage. + """ + if not isinstance(raw, dict): + raise AnnualConfigError("annual.tariff is required and must be a mapping") + if not raw.get("import_octopus_url") and not raw.get("rates_import"): + raise AnnualConfigError("annual.tariff requires either import_octopus_url or rates_import") + + templated = [name for name in ["import_octopus_url", "export_octopus_url"] if raw.get(name) and "{dno_region}" in raw[name]] + if templated and not raw.get("dno_region"): + raise AnnualConfigError("annual.tariff.{} uses {{dno_region}} but annual.tariff.dno_region is not set; supply your Octopus region letter, for example 'A' for Eastern England".format(templated[0])) + + tariff = dict(raw) + tariff["standing_charge_p_per_day"] = float(raw.get("standing_charge_p_per_day", 0.0)) + return tariff + + +def validate_config(config, today=None): + """Validate and normalise an annual prediction config, returning a fully defaulted copy. + + Accepts either the wrapped form ({"annual": {...}}) or the inner mapping directly. + Raises AnnualConfigError with an actionable message on any problem. + """ + if not isinstance(config, dict): + raise AnnualConfigError("The annual config must be a mapping") + + raw = config.get("annual", config) + if not isinstance(raw, dict): + raise AnnualConfigError("The annual config must be a mapping") + + location = raw.get("location") + if not isinstance(location, dict): + raise AnnualConfigError("annual.location is required, with either a postcode or latitude and longitude") + if not location.get("postcode") and not ("latitude" in location and "longitude" in location): + raise AnnualConfigError("annual.location needs either a postcode or both latitude and longitude") + + solar = _validate_solar(raw.get("solar")) + battery = _validate_battery(raw.get("battery")) + if not solar and battery is None: + raise AnnualConfigError("annual needs at least one of solar or battery: with neither there is nothing to evaluate") + + samples_per_month = int(raw.get("samples_per_month", DEFAULT_SAMPLES_PER_MONTH)) + if samples_per_month < 1: + raise AnnualConfigError("annual.samples_per_month must be at least 1, got {}".format(samples_per_month)) + + if today is None: + today = date.today() + year = int(raw.get("year", today.year - 1)) + + return { + "location": dict(location), + "year": year, + "solar": solar, + "battery": battery, + "load": _validate_load(raw.get("load")), + "tariff": _validate_tariff(raw.get("tariff")), + "samples_per_month": samples_per_month, + "timezone": raw.get("timezone", DEFAULT_TIMEZONE), + "pv10_derate_fallback": float(raw.get("pv10_derate_fallback", DEFAULT_PV10_DERATE_FALLBACK)), + "raw": scrub_secrets(copy.deepcopy(raw)), + } diff --git a/apps/predbat/tests/test_annual_config.py b/apps/predbat/tests/test_annual_config.py new file mode 100644 index 000000000..340ce5616 --- /dev/null +++ b/apps/predbat/tests/test_annual_config.py @@ -0,0 +1,195 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for annual prediction config validation.""" + +from datetime import date + +from annual import AnnualConfigError, scrub_secrets, validate_config + + +def base_config(): + """Return a minimal valid annual config.""" + return { + "annual": { + "location": {"latitude": 51.5, "longitude": -0.1}, + "solar": [{"kwp": 5.6}], + "battery": {"size_kwh": 9.5, "inverter_kw": 5.0}, + "load": {"annual_kwh": 3800}, + "tariff": {"rates_import": [{"rate": 25.0}]}, + } + } + + +def expect_error(label, config, fragment, failed): + """Assert that validate_config rejects the config with a message containing fragment.""" + try: + validate_config(config) + except AnnualConfigError as error: + if fragment.lower() not in str(error).lower(): + print(" ERROR: {} raised '{}', expected it to mention '{}'".format(label, error, fragment)) + return True + return failed + print(" ERROR: {} should have raised AnnualConfigError".format(label)) + return True + + +def test_annual_config(my_predbat): + """Verify annual config defaulting, normalisation and rejection rules.""" + failed = False + print("**** Testing annual config validation ****") + + print("Test: a minimal config validates and gains defaults") + result = validate_config(base_config(), today=date(2026, 7, 25)) + if result["year"] != 2025: + print(" ERROR: year should default to the most recent complete calendar year, got {}".format(result["year"])) + failed = True + if result["samples_per_month"] != 2: + print(" ERROR: samples_per_month should default to 2, got {}".format(result["samples_per_month"])) + failed = True + if result["timezone"] != "Europe/London": + print(" ERROR: timezone should default to Europe/London, got {}".format(result["timezone"])) + failed = True + if abs(result["pv10_derate_fallback"] - 0.7) > 1e-9: + print(" ERROR: pv10_derate_fallback should default to 0.7, got {}".format(result["pv10_derate_fallback"])) + failed = True + if result["load"]["shape"] != "flat": + print(" ERROR: load shape should default to flat, got {}".format(result["load"]["shape"])) + failed = True + if result["solar"][0]["declination"] != 35 or result["solar"][0]["azimuth"] != 180: + print(" ERROR: solar defaults should be declination 35 azimuth 180, got {}".format(result["solar"][0])) + failed = True + if abs(result["solar"][0]["efficiency"] - 0.95) > 1e-9: + print(" ERROR: solar efficiency should default to 0.95, got {}".format(result["solar"][0]["efficiency"])) + failed = True + if result["battery"]["charge_rate_kw"] != 5.0 or result["battery"]["discharge_rate_kw"] != 5.0: + print(" ERROR: charge and discharge rates should default to inverter_kw, got {}".format(result["battery"])) + failed = True + if result["battery"]["export_limit_kw"] != 5.0: + print(" ERROR: export_limit_kw should default to inverter_kw, got {}".format(result["battery"])) + failed = True + if result["battery"]["hybrid"] is not True: + print(" ERROR: hybrid should default to True, got {}".format(result["battery"].get("hybrid"))) + failed = True + + print("Test: an unwrapped config without the 'annual' key is accepted") + unwrapped = base_config()["annual"] + result = validate_config(unwrapped, today=date(2026, 7, 25)) + if result["year"] != 2025: + print(" ERROR: an unwrapped config should validate the same way") + failed = True + + print("Test: Octopus load together with a manual figure is rejected") + config = base_config() + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + failed = expect_error("octopus plus annual_kwh", config, "mutually exclusive", failed) + + config = base_config() + del config["annual"]["load"]["annual_kwh"] + config["annual"]["load"]["car_charging_kwh"] = 2500 + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + failed = expect_error("octopus plus car_charging_kwh", config, "mutually exclusive", failed) + + print("Test: an Octopus-only load block validates") + config = base_config() + del config["annual"]["load"]["annual_kwh"] + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["load"].get("octopus", {}).get("account_id") != "A-1": + print(" ERROR: the Octopus load block should survive validation") + failed = True + + print("Test: a missing battery block yields a two-scenario run") + config = base_config() + del config["annual"]["battery"] + result = validate_config(config, today=date(2026, 7, 25)) + if result["battery"] is not None: + print(" ERROR: an omitted battery should normalise to None, got {}".format(result["battery"])) + failed = True + + print("Test: a missing solar block is allowed for a battery-only run") + config = base_config() + del config["annual"]["solar"] + result = validate_config(config, today=date(2026, 7, 25)) + if result["solar"] != []: + print(" ERROR: an omitted solar block should normalise to an empty list, got {}".format(result["solar"])) + failed = True + + print("Test: omitting both solar and battery is rejected as pointless") + config = base_config() + del config["annual"]["solar"] + del config["annual"]["battery"] + failed = expect_error("neither solar nor battery", config, "at least one", failed) + + print("Test: missing location is rejected") + config = base_config() + del config["annual"]["location"] + failed = expect_error("no location", config, "location", failed) + + print("Test: missing load is rejected") + config = base_config() + del config["annual"]["load"] + failed = expect_error("no load", config, "load", failed) + + print("Test: missing tariff is rejected") + config = base_config() + del config["annual"]["tariff"] + failed = expect_error("no tariff", config, "tariff", failed) + + print("Test: an unknown load shape is rejected") + config = base_config() + config["annual"]["load"]["shape"] = "sideways" + failed = expect_error("bad shape", config, "shape", failed) + + print("Test: a solar array without kwp is rejected") + config = base_config() + config["annual"]["solar"] = [{"declination": 30}] + failed = expect_error("array without kwp", config, "kwp", failed) + + print("Test: samples_per_month below 1 is rejected") + config = base_config() + config["annual"]["samples_per_month"] = 0 + failed = expect_error("zero samples", config, "samples_per_month", failed) + + print("Test: a postcode-only location validates") + config = base_config() + config["annual"]["location"] = {"postcode": "SW1A 1AA"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["location"].get("postcode") != "SW1A 1AA": + print(" ERROR: a postcode location should survive validation") + failed = True + + print("Test: a templated tariff URL without dno_region is rejected up front") + config = base_config() + config["annual"]["tariff"] = {"import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/"} + failed = expect_error("templated url without region", config, "dno_region", failed) + + print("Test: a templated tariff URL with dno_region validates and is carried through") + config = base_config() + config["annual"]["tariff"] = {"import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/", "dno_region": "A"} + result = validate_config(config, today=date(2026, 7, 25)) + if result["tariff"].get("dno_region") != "A": + print(" ERROR: dno_region should survive validation, got {}".format(result["tariff"].get("dno_region"))) + failed = True + + print("Test: scrub_secrets removes API keys without mutating the original") + config = base_config() + config["annual"]["load"]["octopus"] = {"api_key": "sk_live_secret", "account_id": "A-1"} + scrubbed = scrub_secrets(config) + if scrubbed["annual"]["load"]["octopus"]["api_key"] != "xxx": + print(" ERROR: api_key should be scrubbed, got {}".format(scrubbed["annual"]["load"]["octopus"]["api_key"])) + failed = True + if config["annual"]["load"]["octopus"]["api_key"] != "sk_live_secret": + print(" ERROR: scrub_secrets must not mutate its input") + failed = True + if scrubbed["annual"]["load"]["octopus"]["account_id"] != "A-1": + print(" ERROR: non-secret values should survive scrubbing") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index b849293e7..72ab7040b 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -163,6 +163,7 @@ from tests.test_savings_stability import test_savings_stability from tests.test_calculate_yesterday import test_calculate_yesterday from tests.test_load_today_comparison import test_load_today_comparison +from tests.test_annual_config import test_annual_config # Mock the components and plugin system @@ -404,6 +405,7 @@ def main(): ("optimise_solar", run_optimise_solar_tests, "Optimise export more solar tests", False), ("optimise_swap_charge", run_optimise_swap_charge_tests, "Optimise pairwise charge-window swap tests", False), ("debug_cases", run_debug_cases, "Debug case file tests", True), + ("annual_config", test_annual_config, "Annual prediction config validation tests", False), ] # Parse command line arguments From 2e872782736edd0fe93eb2c6a397c321b7d39cf4 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 21:32:10 +0200 Subject: [PATCH 020/119] fix(annual): guard numeric conversions and tighten range checks in config validation Address code review findings on the config validator: route every float()/int() conversion through a _require_number() helper that catches TypeError/ValueError and re-raises AnnualConfigError, add range checks (size_kwh/kwp/inverter_kw > 0, efficiency and pv10_derate_fallback in (0, 1], year within [1940, current year], etc.), close the octopus:{} loophole in the load exclusivity check, name every templated tariff field missing dno_region rather than just the first, and drop the redundant deepcopy ahead of scrub_secrets. Add covering tests for each. --- apps/predbat/annual.py | 70 +++++++++++----- apps/predbat/tests/test_annual_config.py | 101 +++++++++++++++++++++-- 2 files changed, 143 insertions(+), 28 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index cab7e9c4b..e494b4d8d 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -12,7 +12,6 @@ and tariff modules own all network access. """ -import copy from datetime import date VALID_SHAPES = ["night", "day", "flat"] @@ -25,6 +24,9 @@ DEFAULT_EFFICIENCY = 0.95 DEFAULT_HYBRID = True +# The Open-Meteo ERA5 archive, which the weather module draws on, starts in 1940. +MINIMUM_YEAR = 1940 + # Substrings that mark a config value as secret and therefore scrubbable SECRET_MARKERS = ["_key", "password", "token", "secret"] @@ -52,6 +54,31 @@ def scrub_secrets(config): return config +def _require_number(value, field, minimum=None, maximum=None, integer=False, exclusive_minimum=False): + """Coerce a config value to a number, raising AnnualConfigError with an actionable message. + + Rejects booleans explicitly (``True`` silently becoming ``1.0`` would be a confusing + outcome), converts with ``int()``/``float()`` inside a try/except so a malformed value + never escapes as a bare ``ValueError``/``TypeError``, and enforces the optional bounds. + ``minimum`` is inclusive unless ``exclusive_minimum`` is set, in which case the value + must be strictly greater than it. ``maximum`` is always inclusive. + """ + if isinstance(value, bool): + raise AnnualConfigError("{} must be a number, not a boolean (got {})".format(field, value)) + try: + number = int(value) if integer else float(value) + except (TypeError, ValueError): + raise AnnualConfigError("{} must be a number, got {!r}".format(field, value)) + if minimum is not None: + if exclusive_minimum and number <= minimum: + raise AnnualConfigError("{} must be greater than {}, got {}".format(field, minimum, number)) + if not exclusive_minimum and number < minimum: + raise AnnualConfigError("{} must be at least {}, got {}".format(field, minimum, number)) + if maximum is not None and number > maximum: + raise AnnualConfigError("{} must be at most {}, got {}".format(field, maximum, number)) + return number + + def _validate_solar(raw): """Normalise the solar array list, applying defaults and rejecting arrays without kwp.""" if raw is None: @@ -68,10 +95,10 @@ def _validate_solar(raw): if "kwp" not in array: raise AnnualConfigError("annual.solar[{}] is missing kwp, the array's peak power in kW".format(index)) normalised = dict(array) - normalised["kwp"] = float(array["kwp"]) + normalised["kwp"] = _require_number(array["kwp"], "annual.solar[{}].kwp".format(index), minimum=0, exclusive_minimum=True) normalised["declination"] = array.get("declination", DEFAULT_DECLINATION) normalised["azimuth"] = array.get("azimuth", DEFAULT_AZIMUTH) - normalised["efficiency"] = float(array.get("efficiency", DEFAULT_EFFICIENCY)) + normalised["efficiency"] = _require_number(array.get("efficiency", DEFAULT_EFFICIENCY), "annual.solar[{}].efficiency".format(index), minimum=0, exclusive_minimum=True, maximum=1) arrays.append(normalised) return arrays @@ -87,14 +114,14 @@ def _validate_battery(raw): if "inverter_kw" not in raw: raise AnnualConfigError("annual.battery is missing inverter_kw") - inverter_kw = float(raw["inverter_kw"]) + inverter_kw = _require_number(raw["inverter_kw"], "annual.battery.inverter_kw", minimum=0, exclusive_minimum=True) return { - "size_kwh": float(raw["size_kwh"]), + "size_kwh": _require_number(raw["size_kwh"], "annual.battery.size_kwh", minimum=0, exclusive_minimum=True), "inverter_kw": inverter_kw, - "export_limit_kw": float(raw.get("export_limit_kw", inverter_kw)), + "export_limit_kw": _require_number(raw.get("export_limit_kw", inverter_kw), "annual.battery.export_limit_kw", minimum=0), "hybrid": bool(raw.get("hybrid", DEFAULT_HYBRID)), - "charge_rate_kw": float(raw.get("charge_rate_kw", inverter_kw)), - "discharge_rate_kw": float(raw.get("discharge_rate_kw", inverter_kw)), + "charge_rate_kw": _require_number(raw.get("charge_rate_kw", inverter_kw), "annual.battery.charge_rate_kw", minimum=0, exclusive_minimum=True), + "discharge_rate_kw": _require_number(raw.get("discharge_rate_kw", inverter_kw), "annual.battery.discharge_rate_kw", minimum=0, exclusive_minimum=True), } @@ -103,16 +130,17 @@ def _validate_load(raw): if not isinstance(raw, dict): raise AnnualConfigError("annual.load is required and must be a mapping") + has_octopus = "octopus" in raw octopus = raw.get("octopus") has_manual = ("annual_kwh" in raw) or ("car_charging_kwh" in raw) - if octopus and has_manual: + if has_octopus and has_manual: raise AnnualConfigError("annual.load.octopus and annual.load.annual_kwh/car_charging_kwh are mutually exclusive: the Octopus consumption series already includes any car charging, so supplying both would double-count it") - if not octopus and "annual_kwh" not in raw: + if not has_octopus and "annual_kwh" not in raw: raise AnnualConfigError("annual.load requires either annual_kwh or an octopus block") - if octopus: + if has_octopus: if not isinstance(octopus, dict) or not octopus.get("api_key") or not octopus.get("account_id"): raise AnnualConfigError("annual.load.octopus requires both api_key and account_id") return {"octopus": dict(octopus), "shape": raw.get("shape", "flat"), "car_charging_kwh": 0.0} @@ -121,7 +149,11 @@ def _validate_load(raw): if shape not in VALID_SHAPES: raise AnnualConfigError("annual.load.shape must be one of {}, got '{}'".format(VALID_SHAPES, shape)) - return {"annual_kwh": float(raw["annual_kwh"]), "shape": shape, "car_charging_kwh": float(raw.get("car_charging_kwh", 0.0))} + return { + "annual_kwh": _require_number(raw["annual_kwh"], "annual.load.annual_kwh", minimum=0), + "shape": shape, + "car_charging_kwh": _require_number(raw.get("car_charging_kwh", 0.0), "annual.load.car_charging_kwh", minimum=0), + } def _validate_tariff(raw): @@ -138,10 +170,10 @@ def _validate_tariff(raw): templated = [name for name in ["import_octopus_url", "export_octopus_url"] if raw.get(name) and "{dno_region}" in raw[name]] if templated and not raw.get("dno_region"): - raise AnnualConfigError("annual.tariff.{} uses {{dno_region}} but annual.tariff.dno_region is not set; supply your Octopus region letter, for example 'A' for Eastern England".format(templated[0])) + raise AnnualConfigError("annual.tariff.{} uses {{dno_region}} but annual.tariff.dno_region is not set; supply your Octopus region letter, for example 'A' for Eastern England".format(", ".join(templated))) tariff = dict(raw) - tariff["standing_charge_p_per_day"] = float(raw.get("standing_charge_p_per_day", 0.0)) + tariff["standing_charge_p_per_day"] = _require_number(raw.get("standing_charge_p_per_day", 0.0), "annual.tariff.standing_charge_p_per_day", minimum=0) return tariff @@ -169,13 +201,11 @@ def validate_config(config, today=None): if not solar and battery is None: raise AnnualConfigError("annual needs at least one of solar or battery: with neither there is nothing to evaluate") - samples_per_month = int(raw.get("samples_per_month", DEFAULT_SAMPLES_PER_MONTH)) - if samples_per_month < 1: - raise AnnualConfigError("annual.samples_per_month must be at least 1, got {}".format(samples_per_month)) + samples_per_month = _require_number(raw.get("samples_per_month", DEFAULT_SAMPLES_PER_MONTH), "annual.samples_per_month", minimum=1, integer=True) if today is None: today = date.today() - year = int(raw.get("year", today.year - 1)) + year = _require_number(raw.get("year", today.year - 1), "annual.year", minimum=MINIMUM_YEAR, maximum=today.year, integer=True) return { "location": dict(location), @@ -186,6 +216,6 @@ def validate_config(config, today=None): "tariff": _validate_tariff(raw.get("tariff")), "samples_per_month": samples_per_month, "timezone": raw.get("timezone", DEFAULT_TIMEZONE), - "pv10_derate_fallback": float(raw.get("pv10_derate_fallback", DEFAULT_PV10_DERATE_FALLBACK)), - "raw": scrub_secrets(copy.deepcopy(raw)), + "pv10_derate_fallback": _require_number(raw.get("pv10_derate_fallback", DEFAULT_PV10_DERATE_FALLBACK), "annual.pv10_derate_fallback", minimum=0, exclusive_minimum=True, maximum=1), + "raw": scrub_secrets(raw), } diff --git a/apps/predbat/tests/test_annual_config.py b/apps/predbat/tests/test_annual_config.py index 340ce5616..357888b6e 100644 --- a/apps/predbat/tests/test_annual_config.py +++ b/apps/predbat/tests/test_annual_config.py @@ -40,6 +40,23 @@ def expect_error(label, config, fragment, failed): return True +def expect_config_error_type(label, config, failed): + """Assert that validate_config raises AnnualConfigError specifically, not a bare ValueError/TypeError. + + A malformed numeric field must be caught and re-raised as AnnualConfigError so that a + later CLI layer, which only catches that type, never sees a raw traceback. + """ + try: + validate_config(config) + except AnnualConfigError: + return failed + except (ValueError, TypeError) as error: + print(" ERROR: {} raised a bare {} ('{}') instead of AnnualConfigError".format(label, type(error).__name__, error)) + return True + print(" ERROR: {} should have raised AnnualConfigError".format(label)) + return True + + def test_annual_config(my_predbat): """Verify annual config defaulting, normalisation and rejection rules.""" failed = False @@ -96,6 +113,11 @@ def test_annual_config(my_predbat): config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} failed = expect_error("octopus plus car_charging_kwh", config, "mutually exclusive", failed) + print("Test: an empty octopus block alongside a manual figure is still rejected as mutually exclusive") + config = base_config() + config["annual"]["load"]["octopus"] = {} + failed = expect_error("empty octopus plus annual_kwh", config, "mutually exclusive", failed) + print("Test: an Octopus-only load block validates") config = base_config() del config["annual"]["load"]["annual_kwh"] @@ -125,37 +147,37 @@ def test_annual_config(my_predbat): config = base_config() del config["annual"]["solar"] del config["annual"]["battery"] - failed = expect_error("neither solar nor battery", config, "at least one", failed) + failed = expect_error("neither solar nor battery", config, "at least one of solar or battery", failed) print("Test: missing location is rejected") config = base_config() del config["annual"]["location"] - failed = expect_error("no location", config, "location", failed) + failed = expect_error("no location", config, "annual.location is required", failed) print("Test: missing load is rejected") config = base_config() del config["annual"]["load"] - failed = expect_error("no load", config, "load", failed) + failed = expect_error("no load", config, "annual.load is required", failed) print("Test: missing tariff is rejected") config = base_config() del config["annual"]["tariff"] - failed = expect_error("no tariff", config, "tariff", failed) + failed = expect_error("no tariff", config, "annual.tariff is required", failed) print("Test: an unknown load shape is rejected") config = base_config() config["annual"]["load"]["shape"] = "sideways" - failed = expect_error("bad shape", config, "shape", failed) + failed = expect_error("bad shape", config, "annual.load.shape must be one of", failed) print("Test: a solar array without kwp is rejected") config = base_config() config["annual"]["solar"] = [{"declination": 30}] - failed = expect_error("array without kwp", config, "kwp", failed) + failed = expect_error("array without kwp", config, "is missing kwp", failed) print("Test: samples_per_month below 1 is rejected") config = base_config() config["annual"]["samples_per_month"] = 0 - failed = expect_error("zero samples", config, "samples_per_month", failed) + failed = expect_error("zero samples", config, "annual.samples_per_month must be at least", failed) print("Test: a postcode-only location validates") config = base_config() @@ -168,7 +190,7 @@ def test_annual_config(my_predbat): print("Test: a templated tariff URL without dno_region is rejected up front") config = base_config() config["annual"]["tariff"] = {"import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/"} - failed = expect_error("templated url without region", config, "dno_region", failed) + failed = expect_error("templated url without region", config, "dno_region is not set", failed) print("Test: a templated tariff URL with dno_region validates and is carried through") config = base_config() @@ -178,6 +200,69 @@ def test_annual_config(my_predbat): print(" ERROR: dno_region should survive validation, got {}".format(result["tariff"].get("dno_region"))) failed = True + print("Test: both import and export templated URLs without dno_region are both named in the error") + config = base_config() + config["annual"]["tariff"] = { + "import_octopus_url": "https://api.octopus.energy/v1/products/AGILE/electricity-tariffs/E-1R-AGILE-{dno_region}/standard-unit-rates/", + "export_octopus_url": "https://api.octopus.energy/v1/products/AGILE-OUTGOING/electricity-tariffs/E-1R-AGILE-OUTGOING-{dno_region}/standard-unit-rates/", + } + try: + validate_config(config, today=date(2026, 7, 25)) + print(" ERROR: both templated URLs without dno_region should have raised AnnualConfigError") + failed = True + except AnnualConfigError as error: + message = str(error) + if "import_octopus_url" not in message or "export_octopus_url" not in message: + print(" ERROR: the dno_region error should name both offending fields, got '{}'".format(message)) + failed = True + + print("Test: a non-numeric kwp raises AnnualConfigError, not a bare ValueError") + config = base_config() + config["annual"]["solar"] = [{"kwp": "not-a-number"}] + failed = expect_config_error_type("non-numeric kwp", config, failed) + + print("Test: a None kwp raises AnnualConfigError, not a bare TypeError") + config = base_config() + config["annual"]["solar"] = [{"kwp": None}] + failed = expect_config_error_type("None kwp", config, failed) + + print("Test: a non-numeric size_kwh raises AnnualConfigError, not a bare ValueError") + config = base_config() + config["annual"]["battery"]["size_kwh"] = "lots" + failed = expect_config_error_type("non-numeric size_kwh", config, failed) + + print("Test: a None size_kwh raises AnnualConfigError, not a bare TypeError") + config = base_config() + config["annual"]["battery"]["size_kwh"] = None + failed = expect_config_error_type("None size_kwh", config, failed) + + print("Test: a negative size_kwh is rejected") + config = base_config() + config["annual"]["battery"]["size_kwh"] = -5 + failed = expect_error("negative size_kwh", config, "size_kwh", failed) + + print("Test: a zero kwp is rejected") + config = base_config() + config["annual"]["solar"] = [{"kwp": 0}] + failed = expect_error("zero kwp", config, "kwp", failed) + + print("Test: an efficiency above 1 is rejected") + config = base_config() + config["annual"]["solar"] = [{"kwp": 5.6, "efficiency": 1.5}] + failed = expect_error("efficiency 1.5", config, "efficiency", failed) + + print("Test: a year in the future is rejected") + config = base_config() + config["annual"]["year"] = 2027 + try: + validate_config(config, today=date(2026, 7, 25)) + print(" ERROR: future year should have raised AnnualConfigError") + failed = True + except AnnualConfigError as error: + if "annual.year must be at most" not in str(error): + print(" ERROR: future year raised '{}', expected it to mention 'annual.year must be at most'".format(error)) + failed = True + print("Test: scrub_secrets removes API keys without mutating the original") config = base_config() config["annual"]["load"]["octopus"] = {"api_key": "sk_live_secret", "account_id": "A-1"} From 5b80f15a32448357c8f98e4a88d63da1ea548265 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 21:40:27 +0200 Subject: [PATCH 021/119] feat(annual): add headless PredBat bootstrap and per-sample state reset --- apps/predbat/annual.py | 201 ++++++++++++++++++++ apps/predbat/tests/test_annual_bootstrap.py | 151 +++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 354 insertions(+) create mode 100644 apps/predbat/tests/test_annual_bootstrap.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index e494b4d8d..274f1621e 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -12,8 +12,11 @@ and tariff modules own all network access. """ +import os from datetime import date +from const import MINUTE_WATT + VALID_SHAPES = ["night", "day", "flat"] DEFAULT_TIMEZONE = "Europe/London" @@ -219,3 +222,201 @@ def validate_config(config, today=None): "pv10_derate_fallback": _require_number(raw.get("pv10_derate_fallback", DEFAULT_PV10_DERATE_FALLBACK), "annual.pv10_derate_fallback", minimum=0, exclusive_minimum=True, maximum=1), "raw": scrub_secrets(raw), } + + +# Minimal apps.yaml for a headless run. PredBat's Hass base class reads this at +# construction time; nothing here talks to Home Assistant. +MINIMAL_APPS_YAML = """pred_bat: + module: predbat + class: PredBat + prefix: predbat + timezone: {timezone} + currency_symbols: + - '£' + - 'p' + threads: 0 + db_enable: false + db_mirror_ha: false + db_primary: false + web_enable: false + mcp_enable: false + notify_devices: [] + days_previous: + - 1 + days_previous_weight: + - 1 + forecast_hours: 48 +""" + +# The default discharge power cap. A leaked full-precision value from a previous +# sample can flip a plan at a decision boundary, so it is reset explicitly. +DEFAULT_BATTERY_RATE_MAX_EXPORT = 0.0333 + + +class AnnualNullHA: + """A no-op Home Assistant interface for headless annual runs. + + Provides the subset of the interface PredBat touches during ``auto_config()``, + ``load_user_config()`` and ``fetch_config_options()``. Nothing is published and + no history exists, which is correct: every input the annual tool needs is + injected directly. + """ + + def __init__(self): + """Create an empty in-memory state store.""" + self.history_enable = False + self.dummy_items = {} + self.service_store_enable = False + self.service_store = [] + self.db_primary = False + + def get_state(self, entity_id, default=None, attribute=None, refresh=False, raw=False): + """Return a stored state, the supplied default, or all states when no entity is given.""" + if not entity_id: + return {} + if entity_id in self.dummy_items: + result = self.dummy_items[entity_id] + if raw: + return result + if isinstance(result, dict): + return result.get(attribute, "") if attribute else result.get("state", default) + return default if attribute else result + return default + + def set_state(self, entity_id, state, attributes=None): + """Store a state locally so subsequent reads round trip.""" + self.dummy_items[entity_id] = state + return state + + def get_history(self, entity_id, now=None, days=30): + """Return None: a headless annual run has no Home Assistant history.""" + return None + + def call_service(self, service, **kwargs): + """Accept and discard a service call.""" + return None + + def get_service_store(self): + """Return and clear the recorded service calls.""" + stored = self.service_store + self.service_store = [] + return stored + + +def write_minimal_apps_yaml(work_dir, timezone): + """Write the headless apps.yaml into ``work_dir`` and return its path.""" + os.makedirs(work_dir, exist_ok=True) + path = os.path.join(work_dir, "apps.yaml") + with open(path, "w") as handle: + handle.write(MINIMAL_APPS_YAML.format(timezone=timezone)) + return path + + +def create_headless_predbat(work_dir, timezone, log): + """Construct a PredBat instance with no Home Assistant connection. + + PredBat's Hass base class reads apps.yaml from ``$PREDBAT_APPS_FILE`` at + construction time, so the environment variable is set before the import-time + construction happens. The predbat import is deliberately local to this + function so merely importing ``annual`` does not drag in the whole engine. + """ + path = write_minimal_apps_yaml(work_dir, timezone) + os.environ["PREDBAT_APPS_FILE"] = path + + import predbat + + instance = predbat.PredBat() + instance.states = {} + instance.reset() + instance.update_time() + instance.ha_interface = AnnualNullHA() + instance.auto_config() + instance.load_user_config() + instance.fetch_config_options() + instance.config_root = work_dir + instance.save_restore_dir = work_dir + instance.args["threads"] = 0 + instance.log = log + return instance + + +def apply_hardware(predbat, battery, solar): + """Map the config's battery block onto the PredBat instance. + + Rates are stored internally as kW per minute, matching + ``Compare.apply_hardware_overrides()``. With no battery block the system is + given zero capacity, which is how the no-battery scenario is expressed. + """ + if battery is None: + predbat.soc_max = 0.0 + predbat.soc_kw = 0.0 + predbat.battery_rate_max_charge = 0.0 + predbat.battery_rate_max_charge_dc = 0.0 + predbat.battery_rate_max_discharge = 0.0 + predbat.battery_rate_max_export = 0.0 + predbat.inverter_limit = (solar[0]["kwp"] if solar else 5.0) * 1000 / MINUTE_WATT + predbat.export_limit = predbat.inverter_limit + predbat.inverter_hybrid = False + return + + predbat.soc_max = battery["size_kwh"] + predbat.soc_kw = min(predbat.soc_kw, predbat.soc_max) + predbat.inverter_limit = battery["inverter_kw"] * 1000 / MINUTE_WATT + predbat.export_limit = battery["export_limit_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_charge = battery["charge_rate_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_charge_dc = predbat.battery_rate_max_charge + predbat.battery_rate_max_discharge = battery["discharge_rate_kw"] * 1000 / MINUTE_WATT + predbat.battery_rate_max_export = predbat.battery_rate_max_discharge + predbat.inverter_hybrid = battery["hybrid"] + + +def reset_sample_state(predbat): + """Reset every field a previous sample could have left behind. + + Without this, a month's result silently depends on what ran before it: the + numbers stay plausible while becoming order-dependent. The list covers the + accumulators, the previous plan, the manual overrides, and the two fields + ``tests/test_single_debug.py`` documents as leaking between debug cases. + """ + predbat.dynamic_load_baseline = {} + predbat.battery_rate_max_export = DEFAULT_BATTERY_RATE_MAX_EXPORT + + predbat.cost_today_sofar = 0 + predbat.carbon_today_sofar = 0 + predbat.iboost_today = 0 + predbat.import_today_now = 0 + predbat.export_today_now = 0 + predbat.load_minutes_now = 0 + predbat.pv_today_now = 0 + + predbat.manual_charge_times = [] + predbat.manual_export_times = [] + predbat.manual_freeze_charge_times = [] + predbat.manual_freeze_export_times = [] + predbat.manual_demand_times = [] + predbat.manual_all_times = [] + + predbat.charge_limit_best = [] + predbat.charge_window_best = [] + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.charge_limit = [] + predbat.charge_window = [] + predbat.export_window = [] + predbat.export_limits = [] + predbat.plan_valid = False + + predbat.octopus_intelligent_charging = False + predbat.load_forecast_only = True + predbat.load_scaling = 1.0 + predbat.load_scaling10 = 1.0 + predbat.load_inday_adjustment = 1.0 + predbat.load_scaling_dynamic = None + predbat.manual_load_adjust = {} + predbat.iboost_enable = False + predbat.carbon_enable = False + predbat.plan_debug = False + predbat.debug_enable = False + predbat.rate_import_replicated = {} + predbat.rate_export_replicated = {} + predbat.savings_last_updated = None diff --git a/apps/predbat/tests/test_annual_bootstrap.py b/apps/predbat/tests/test_annual_bootstrap.py new file mode 100644 index 000000000..8160732c4 --- /dev/null +++ b/apps/predbat/tests/test_annual_bootstrap.py @@ -0,0 +1,151 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction headless bootstrap and state reset.""" + +import os +import tempfile + +import yaml + +from annual import AnnualNullHA, apply_hardware, reset_sample_state, write_minimal_apps_yaml +from const import MINUTE_WATT + + +def test_annual_bootstrap(my_predbat): + """Verify the minimal apps.yaml, the null HA interface, hardware mapping and state reset.""" + failed = False + print("**** Testing annual bootstrap ****") + + print("Test: write_minimal_apps_yaml produces a parseable pred_bat config") + with tempfile.TemporaryDirectory() as work_dir: + path = write_minimal_apps_yaml(work_dir, "Europe/London") + if not os.path.exists(path): + print(" ERROR: apps.yaml was not written to {}".format(path)) + failed = True + with open(path, "r") as handle: + parsed = yaml.safe_load(handle) + if "pred_bat" not in parsed: + print(" ERROR: the written apps.yaml has no pred_bat key") + failed = True + else: + section = parsed["pred_bat"] + for key in ["module", "class", "prefix", "timezone", "currency_symbols", "threads"]: + if key not in section: + print(" ERROR: the written apps.yaml is missing '{}'".format(key)) + failed = True + if section.get("threads") != 0: + print(" ERROR: threads must be 0 so plan runs are deterministic, got {}".format(section.get("threads"))) + failed = True + if section.get("timezone") != "Europe/London": + print(" ERROR: the timezone should be written through, got {}".format(section.get("timezone"))) + failed = True + + print("Test: AnnualNullHA satisfies the interface PredBat calls without a Home Assistant") + null_ha = AnnualNullHA() + if null_ha.get_state("sensor.anything", default=7) != 7: + print(" ERROR: get_state should return the supplied default") + failed = True + if null_ha.get_state(None) != {}: + print(" ERROR: get_state with no entity should return an empty mapping of all states") + failed = True + if null_ha.get_history("sensor.anything") is not None: + print(" ERROR: get_history should return None when no history exists") + failed = True + null_ha.set_state("sensor.written", "5", attributes={"unit": "kWh"}) + if null_ha.get_state("sensor.written") != "5": + print(" ERROR: set_state then get_state should round trip") + failed = True + if null_ha.call_service("some/service", value=1) is not None: + print(" ERROR: call_service should be a no-op returning None") + failed = True + + print("Test: apply_hardware maps the battery block onto PredBat's internal units") + battery = {"size_kwh": 9.5, "inverter_kw": 5.0, "export_limit_kw": 3.6, "hybrid": True, "charge_rate_kw": 3.7, "discharge_rate_kw": 4.2} + apply_hardware(my_predbat, battery, [{"kwp": 5.6}]) + if abs(my_predbat.soc_max - 9.5) > 1e-9: + print(" ERROR: soc_max expected 9.5, got {}".format(my_predbat.soc_max)) + failed = True + if abs(my_predbat.inverter_limit - (5.0 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: inverter_limit should be in kW per minute, got {}".format(my_predbat.inverter_limit)) + failed = True + if abs(my_predbat.export_limit - (3.6 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: export_limit should be in kW per minute, got {}".format(my_predbat.export_limit)) + failed = True + if abs(my_predbat.battery_rate_max_charge - (3.7 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: battery_rate_max_charge should be in kW per minute, got {}".format(my_predbat.battery_rate_max_charge)) + failed = True + if abs(my_predbat.battery_rate_max_discharge - (4.2 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: battery_rate_max_discharge should be in kW per minute, got {}".format(my_predbat.battery_rate_max_discharge)) + failed = True + if my_predbat.inverter_hybrid is not True: + print(" ERROR: inverter_hybrid should be True") + failed = True + + print("Test: apply_hardware with no battery produces a zero-capacity system") + apply_hardware(my_predbat, None, [{"kwp": 5.6}]) + if my_predbat.soc_max != 0.0 or my_predbat.soc_kw != 0.0: + print(" ERROR: a battery-less run should have soc_max and soc_kw of 0, got {} / {}".format(my_predbat.soc_max, my_predbat.soc_kw)) + failed = True + + print("Test: reset_sample_state clears every field a previous sample could have left behind") + my_predbat.dynamic_load_baseline = {5: 1.0} + my_predbat.battery_rate_max_export = 99.0 + my_predbat.manual_charge_times = [1, 2, 3] + my_predbat.manual_export_times = [4] + my_predbat.manual_all_times = [5] + my_predbat.cost_today_sofar = 123.0 + my_predbat.import_today_now = 4.0 + my_predbat.export_today_now = 5.0 + my_predbat.iboost_today = 6.0 + my_predbat.carbon_today_sofar = 7.0 + my_predbat.load_minutes_now = 8.0 + my_predbat.pv_today_now = 9.0 + my_predbat.charge_limit_best = [1.0] + my_predbat.charge_window_best = [{"start": 0, "end": 30}] + my_predbat.export_window_best = [{"start": 0, "end": 30}] + my_predbat.export_limits_best = [50.0] + my_predbat.plan_valid = True + + reset_sample_state(my_predbat) + + checks = [ + ("dynamic_load_baseline", {}), + ("battery_rate_max_export", 0.0333), + ("manual_charge_times", []), + ("manual_export_times", []), + ("manual_all_times", []), + ("cost_today_sofar", 0), + ("import_today_now", 0), + ("export_today_now", 0), + ("iboost_today", 0), + ("carbon_today_sofar", 0), + ("load_minutes_now", 0), + ("pv_today_now", 0), + ("charge_limit_best", []), + ("charge_window_best", []), + ("export_window_best", []), + ("export_limits_best", []), + ("plan_valid", False), + ] + for name, expected in checks: + actual = getattr(my_predbat, name) + if actual != expected: + print(" ERROR: reset_sample_state left {} as {}, expected {}".format(name, actual, expected)) + failed = True + + print("Test: reset_sample_state disables the live-system behaviours that make no sense offline") + if my_predbat.octopus_intelligent_charging is not False: + print(" ERROR: octopus_intelligent_charging should be disabled") + failed = True + if my_predbat.load_forecast_only is not True: + print(" ERROR: load_forecast_only must be True so the load profile is taken from load_forecast") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 72ab7040b..eb3b2db74 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -164,6 +164,7 @@ from tests.test_calculate_yesterday import test_calculate_yesterday from tests.test_load_today_comparison import test_load_today_comparison from tests.test_annual_config import test_annual_config +from tests.test_annual_bootstrap import test_annual_bootstrap # Mock the components and plugin system @@ -406,6 +407,7 @@ def main(): ("optimise_swap_charge", run_optimise_swap_charge_tests, "Optimise pairwise charge-window swap tests", False), ("debug_cases", run_debug_cases, "Debug case file tests", True), ("annual_config", test_annual_config, "Annual prediction config validation tests", False), + ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), ] # Parse command line arguments From ac4cb2417a2825e893689aacd72af277d46ad9bd Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 22:06:52 +0200 Subject: [PATCH 022/119] fix(annual): correct state-reset ownership, soc_kw determinism and env-var leak in headless bootstrap Fixes 6 defects found in review of Task 8's headless bootstrap: - battery_rate_max_export is now owned solely by apply_hardware(), not clobbered back to a hardcoded default by reset_sample_state() - soc_kw is reset per sample and set deterministically by apply_hardware() instead of being clamped against inherited state - the full rate-derived family (low_rates, high_export_rates, rate_import, rate_export, thresholds, min/max/average) is now cleared alongside the replicated rates that fed them - reset_sample_state() now only clears per-sample leaks; the deliberate offline-mode choices moved to a new configure_offline_mode(), called once from create_headless_predbat() - PREDBAT_APPS_FILE is saved and restored around PredBat() construction so it cannot leak into a later PredBat() in the same process - the no-battery branch sums kwp across all solar arrays (not just the first) and zeroes battery_rate_min, so PV-only and battery scenarios are compared against the same export cap --- apps/predbat/annual.py | 120 +++++++++++++----- apps/predbat/tests/test_annual_bootstrap.py | 133 ++++++++++++++++++-- 2 files changed, 210 insertions(+), 43 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 274f1621e..bf9be8c97 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -225,12 +225,14 @@ def validate_config(config, today=None): # Minimal apps.yaml for a headless run. PredBat's Hass base class reads this at -# construction time; nothing here talks to Home Assistant. +# construction time; nothing here talks to Home Assistant. timezone is quoted since it +# is interpolated raw into YAML and a value containing ':' or '#' would otherwise be +# invalid or reparsed as something other than a plain string. MINIMAL_APPS_YAML = """pred_bat: module: predbat class: PredBat prefix: predbat - timezone: {timezone} + timezone: "{timezone}" currency_symbols: - '£' - 'p' @@ -248,10 +250,6 @@ def validate_config(config, today=None): forecast_hours: 48 """ -# The default discharge power cap. A leaked full-precision value from a previous -# sample can flip a plan at a decision boundary, so it is reset explicitly. -DEFAULT_BATTERY_RATE_MAX_EXPORT = 0.0333 - class AnnualNullHA: """A no-op Home Assistant interface for headless annual runs. @@ -266,8 +264,6 @@ def __init__(self): """Create an empty in-memory state store.""" self.history_enable = False self.dummy_items = {} - self.service_store_enable = False - self.service_store = [] self.db_primary = False def get_state(self, entity_id, default=None, attribute=None, refresh=False, raw=False): @@ -296,18 +292,12 @@ def call_service(self, service, **kwargs): """Accept and discard a service call.""" return None - def get_service_store(self): - """Return and clear the recorded service calls.""" - stored = self.service_store - self.service_store = [] - return stored - def write_minimal_apps_yaml(work_dir, timezone): """Write the headless apps.yaml into ``work_dir`` and return its path.""" os.makedirs(work_dir, exist_ok=True) path = os.path.join(work_dir, "apps.yaml") - with open(path, "w") as handle: + with open(path, "w", encoding="utf-8") as handle: handle.write(MINIMAL_APPS_YAML.format(timezone=timezone)) return path @@ -315,28 +305,39 @@ def write_minimal_apps_yaml(work_dir, timezone): def create_headless_predbat(work_dir, timezone, log): """Construct a PredBat instance with no Home Assistant connection. - PredBat's Hass base class reads apps.yaml from ``$PREDBAT_APPS_FILE`` at - construction time, so the environment variable is set before the import-time - construction happens. The predbat import is deliberately local to this - function so merely importing ``annual`` does not drag in the whole engine. + ``Hass.__init__`` is the only place that reads ``$PREDBAT_APPS_FILE``, and it does so + synchronously while ``PredBat()`` is constructed below, so the environment variable only + needs to be set for the duration of that one call. It is restored to whatever it held + before (unset if it was unset) in a ``finally`` block, so a later ``PredBat()`` in the + same process — including ``unit_test.py``'s own ``create_predbat()`` — never silently + picks up this work directory's apps.yaml. The predbat import is deliberately local to + this function so merely importing ``annual`` does not drag in the whole engine. """ path = write_minimal_apps_yaml(work_dir, timezone) + previous_apps_file = os.environ.get("PREDBAT_APPS_FILE") os.environ["PREDBAT_APPS_FILE"] = path + try: + import predbat - import predbat + instance = predbat.PredBat() + finally: + if previous_apps_file is None: + os.environ.pop("PREDBAT_APPS_FILE", None) + else: + os.environ["PREDBAT_APPS_FILE"] = previous_apps_file - instance = predbat.PredBat() instance.states = {} + instance.log = log instance.reset() instance.update_time() instance.ha_interface = AnnualNullHA() instance.auto_config() instance.load_user_config() instance.fetch_config_options() + configure_offline_mode(instance) instance.config_root = work_dir instance.save_restore_dir = work_dir instance.args["threads"] = 0 - instance.log = log return instance @@ -345,25 +346,44 @@ def apply_hardware(predbat, battery, solar): Rates are stored internally as kW per minute, matching ``Compare.apply_hardware_overrides()``. With no battery block the system is - given zero capacity, which is how the no-battery scenario is expressed. + given zero capacity, which is how the no-battery scenario is expressed. This + function is the sole owner of ``battery_rate_max_export``: unlike the other + accumulators, it is hardware-derived rather than per-sample state, so + ``reset_sample_state()`` must never overwrite it. ``soc_kw``, the prediction's + starting SOC, is set deterministically here rather than clamped against + whatever it happened to hold before, since clamping only makes sense when a + caller has deliberately set a starting SOC first — a later task does that + explicitly, after calling this function. """ if battery is None: predbat.soc_max = 0.0 predbat.soc_kw = 0.0 predbat.battery_rate_max_charge = 0.0 + # Synthetic configs have no separate DC figure to scale proportionally (unlike + # compare.py's apply_hardware_overrides(), which scales an inherited DC/AC ratio), + # so the DC rate is simply set equal to the AC rate. predbat.battery_rate_max_charge_dc = 0.0 predbat.battery_rate_max_discharge = 0.0 predbat.battery_rate_max_export = 0.0 - predbat.inverter_limit = (solar[0]["kwp"] if solar else 5.0) * 1000 / MINUTE_WATT + # A zero-capacity battery has no meaningful minimum reserve either. + predbat.battery_rate_min = 0.0 + # Sum kWp across every array, not just the first: a PV-only run's export limit + # must match what the battery run would see from the same solar array(s), or the + # difference between scenarios - this tool's whole output - is computed against + # two different caps. + predbat.inverter_limit = (sum(array["kwp"] for array in solar) if solar else 5.0) * 1000 / MINUTE_WATT predbat.export_limit = predbat.inverter_limit predbat.inverter_hybrid = False return predbat.soc_max = battery["size_kwh"] - predbat.soc_kw = min(predbat.soc_kw, predbat.soc_max) + predbat.soc_kw = predbat.soc_max predbat.inverter_limit = battery["inverter_kw"] * 1000 / MINUTE_WATT predbat.export_limit = battery["export_limit_kw"] * 1000 / MINUTE_WATT predbat.battery_rate_max_charge = battery["charge_rate_kw"] * 1000 / MINUTE_WATT + # Synthetic configs have no separate DC figure to scale proportionally (unlike + # compare.py's apply_hardware_overrides(), which scales an inherited DC/AC ratio), so + # the DC rate is simply set equal to the AC rate. predbat.battery_rate_max_charge_dc = predbat.battery_rate_max_charge predbat.battery_rate_max_discharge = battery["discharge_rate_kw"] * 1000 / MINUTE_WATT predbat.battery_rate_max_export = predbat.battery_rate_max_discharge @@ -375,11 +395,20 @@ def reset_sample_state(predbat): Without this, a month's result silently depends on what ran before it: the numbers stay plausible while becoming order-dependent. The list covers the - accumulators, the previous plan, the manual overrides, and the two fields - ``tests/test_single_debug.py`` documents as leaking between debug cases. + accumulators, the previous plan, the manual overrides, the starting SOC, the + full rate-derived family that ``calculate_plan`` seeds its best windows from, + and the field ``tests/test_single_debug.py`` documents as leaking between + debug cases (``dynamic_load_baseline``). + + Deliberately excluded: ``battery_rate_max_export`` is hardware-derived and + owned solely by ``apply_hardware()``, not reset here; the offline-mode + choices (Octopus intelligent charging disabled, load taken from + load_forecast, iBoost/carbon disabled, debug off) are one-shot configuration + handled by ``configure_offline_mode()``, not per-sample leaks. """ predbat.dynamic_load_baseline = {} - predbat.battery_rate_max_export = DEFAULT_BATTERY_RATE_MAX_EXPORT + + predbat.soc_kw = 0.0 predbat.cost_today_sofar = 0 predbat.carbon_today_sofar = 0 @@ -406,17 +435,40 @@ def reset_sample_state(predbat): predbat.export_limits = [] predbat.plan_valid = False + # The rate-derived family calculate_plan() reads to seed the best charge/export + # windows (plan.py). Clearing rate_import_replicated/rate_export_replicated but + # leaving these downstream products in place would look handled while still leaking. + predbat.low_rates = [] + predbat.high_export_rates = [] + predbat.rate_import = {} + predbat.rate_export = {} + predbat.rate_import_replicated = {} + predbat.rate_export_replicated = {} + predbat.rate_import_cost_threshold = 99 + predbat.rate_export_cost_threshold = 99 + predbat.rate_min = 0 + predbat.rate_max = 0 + predbat.rate_average = 0 + + predbat.load_inday_adjustment = 1.0 + predbat.load_scaling_dynamic = None + predbat.manual_load_adjust = {} + predbat.savings_last_updated = None + + +def configure_offline_mode(predbat): + """Apply the one-shot configuration choices that make sense only for an offline run. + + These are deliberate choices, not per-sample leaks: they are set once, after + ``fetch_config_options()`` has populated its own defaults, and are not part of + ``reset_sample_state()`` because re-applying them every sample would silently + override whatever ``fetch_config_options()`` or a scenario override set. + """ predbat.octopus_intelligent_charging = False predbat.load_forecast_only = True predbat.load_scaling = 1.0 predbat.load_scaling10 = 1.0 - predbat.load_inday_adjustment = 1.0 - predbat.load_scaling_dynamic = None - predbat.manual_load_adjust = {} predbat.iboost_enable = False predbat.carbon_enable = False predbat.plan_debug = False predbat.debug_enable = False - predbat.rate_import_replicated = {} - predbat.rate_export_replicated = {} - predbat.savings_last_updated = None diff --git a/apps/predbat/tests/test_annual_bootstrap.py b/apps/predbat/tests/test_annual_bootstrap.py index 8160732c4..375890671 100644 --- a/apps/predbat/tests/test_annual_bootstrap.py +++ b/apps/predbat/tests/test_annual_bootstrap.py @@ -14,7 +14,7 @@ import yaml -from annual import AnnualNullHA, apply_hardware, reset_sample_state, write_minimal_apps_yaml +from annual import AnnualNullHA, apply_hardware, configure_offline_mode, create_headless_predbat, reset_sample_state, write_minimal_apps_yaml from const import MINUTE_WATT @@ -72,6 +72,9 @@ def test_annual_bootstrap(my_predbat): if abs(my_predbat.soc_max - 9.5) > 1e-9: print(" ERROR: soc_max expected 9.5, got {}".format(my_predbat.soc_max)) failed = True + if abs(my_predbat.soc_kw - 9.5) > 1e-9: + print(" ERROR: soc_kw should be set deterministically to soc_max, got {}".format(my_predbat.soc_kw)) + failed = True if abs(my_predbat.inverter_limit - (5.0 * 1000 / MINUTE_WATT)) > 1e-9: print(" ERROR: inverter_limit should be in kW per minute, got {}".format(my_predbat.inverter_limit)) failed = True @@ -84,6 +87,9 @@ def test_annual_bootstrap(my_predbat): if abs(my_predbat.battery_rate_max_discharge - (4.2 * 1000 / MINUTE_WATT)) > 1e-9: print(" ERROR: battery_rate_max_discharge should be in kW per minute, got {}".format(my_predbat.battery_rate_max_discharge)) failed = True + if abs(my_predbat.battery_rate_max_export - (4.2 * 1000 / MINUTE_WATT)) > 1e-9: + print(" ERROR: battery_rate_max_export should equal the discharge rate, got {}".format(my_predbat.battery_rate_max_export)) + failed = True if my_predbat.inverter_hybrid is not True: print(" ERROR: inverter_hybrid should be True") failed = True @@ -93,10 +99,36 @@ def test_annual_bootstrap(my_predbat): if my_predbat.soc_max != 0.0 or my_predbat.soc_kw != 0.0: print(" ERROR: a battery-less run should have soc_max and soc_kw of 0, got {} / {}".format(my_predbat.soc_max, my_predbat.soc_kw)) failed = True + if my_predbat.battery_rate_max_charge != 0.0 or my_predbat.battery_rate_max_charge_dc != 0.0 or my_predbat.battery_rate_max_discharge != 0.0 or my_predbat.battery_rate_max_export != 0.0: + print(" ERROR: a battery-less run should zero every rate field") + failed = True + if my_predbat.battery_rate_min != 0.0: + print(" ERROR: a zero-capacity battery should have battery_rate_min of 0, got {}".format(my_predbat.battery_rate_min)) + failed = True + if my_predbat.export_limit != my_predbat.inverter_limit: + print(" ERROR: export_limit should match inverter_limit when there is no battery") + failed = True + if my_predbat.inverter_hybrid is not False: + print(" ERROR: a battery-less run should not be a hybrid inverter") + failed = True + + print("Test: apply_hardware with no battery sums kwp across every solar array, not just the first") + apply_hardware(my_predbat, None, [{"kwp": 5.6}, {"kwp": 4.0}]) + expected_limit = (5.6 + 4.0) * 1000 / MINUTE_WATT + if abs(my_predbat.inverter_limit - expected_limit) > 1e-9: + print(" ERROR: inverter_limit should sum kwp across all arrays, expected {}, got {}".format(expected_limit, my_predbat.inverter_limit)) + failed = True + if abs(my_predbat.export_limit - expected_limit) > 1e-9: + print(" ERROR: export_limit should track the summed inverter_limit, got {}".format(my_predbat.export_limit)) + failed = True + + print("Test: reset_sample_state clears the leaked fields but leaves apply_hardware's output alone") + apply_hardware(my_predbat, battery, [{"kwp": 5.6}]) + configured_battery_rate_max_export = my_predbat.battery_rate_max_export + configured_inverter_limit = my_predbat.inverter_limit - print("Test: reset_sample_state clears every field a previous sample could have left behind") my_predbat.dynamic_load_baseline = {5: 1.0} - my_predbat.battery_rate_max_export = 99.0 + my_predbat.soc_kw = 3.3 my_predbat.manual_charge_times = [1, 2, 3] my_predbat.manual_export_times = [4] my_predbat.manual_all_times = [5] @@ -112,12 +144,23 @@ def test_annual_bootstrap(my_predbat): my_predbat.export_window_best = [{"start": 0, "end": 30}] my_predbat.export_limits_best = [50.0] my_predbat.plan_valid = True + my_predbat.low_rates = [{"start": 0, "end": 30}] + my_predbat.high_export_rates = [{"start": 0, "end": 30}] + my_predbat.rate_import = {0: 25.0} + my_predbat.rate_export = {0: 15.0} + my_predbat.rate_import_replicated = {0: 25.0} + my_predbat.rate_export_replicated = {0: 15.0} + my_predbat.rate_import_cost_threshold = 1.0 + my_predbat.rate_export_cost_threshold = 2.0 + my_predbat.rate_min = 5.0 + my_predbat.rate_max = 6.0 + my_predbat.rate_average = 7.0 reset_sample_state(my_predbat) checks = [ ("dynamic_load_baseline", {}), - ("battery_rate_max_export", 0.0333), + ("soc_kw", 0.0), ("manual_charge_times", []), ("manual_export_times", []), ("manual_all_times", []), @@ -133,6 +176,17 @@ def test_annual_bootstrap(my_predbat): ("export_window_best", []), ("export_limits_best", []), ("plan_valid", False), + ("low_rates", []), + ("high_export_rates", []), + ("rate_import", {}), + ("rate_export", {}), + ("rate_import_replicated", {}), + ("rate_export_replicated", {}), + ("rate_import_cost_threshold", 99), + ("rate_export_cost_threshold", 99), + ("rate_min", 0), + ("rate_max", 0), + ("rate_average", 0), ] for name, expected in checks: actual = getattr(my_predbat, name) @@ -140,12 +194,73 @@ def test_annual_bootstrap(my_predbat): print(" ERROR: reset_sample_state left {} as {}, expected {}".format(name, actual, expected)) failed = True - print("Test: reset_sample_state disables the live-system behaviours that make no sense offline") - if my_predbat.octopus_intelligent_charging is not False: - print(" ERROR: octopus_intelligent_charging should be disabled") + print("Test: reset_sample_state must not clobber the hardware-derived fields apply_hardware owns") + if my_predbat.battery_rate_max_export != configured_battery_rate_max_export: + print(" ERROR: reset_sample_state changed battery_rate_max_export from {} to {}; it must be apply_hardware's alone".format(configured_battery_rate_max_export, my_predbat.battery_rate_max_export)) failed = True - if my_predbat.load_forecast_only is not True: - print(" ERROR: load_forecast_only must be True so the load profile is taken from load_forecast") + if my_predbat.inverter_limit != configured_inverter_limit: + print(" ERROR: reset_sample_state changed inverter_limit from {} to {}; it must be apply_hardware's alone".format(configured_inverter_limit, my_predbat.inverter_limit)) failed = True + print("Test: configure_offline_mode applies the deliberate offline choices, separately from reset_sample_state") + my_predbat.octopus_intelligent_charging = True + my_predbat.load_forecast_only = False + my_predbat.load_scaling = 0.5 + my_predbat.load_scaling10 = 0.5 + my_predbat.iboost_enable = True + my_predbat.carbon_enable = True + my_predbat.plan_debug = True + my_predbat.debug_enable = True + + configure_offline_mode(my_predbat) + + offline_checks = [ + ("octopus_intelligent_charging", False), + ("load_forecast_only", True), + ("load_scaling", 1.0), + ("load_scaling10", 1.0), + ("iboost_enable", False), + ("carbon_enable", False), + ("plan_debug", False), + ("debug_enable", False), + ] + for name, expected in offline_checks: + actual = getattr(my_predbat, name) + if actual != expected: + print(" ERROR: configure_offline_mode left {} as {}, expected {}".format(name, actual, expected)) + failed = True + + print("Test: reset_sample_state must not re-apply configure_offline_mode's choices") + my_predbat.octopus_intelligent_charging = True + my_predbat.debug_enable = True + reset_sample_state(my_predbat) + if my_predbat.octopus_intelligent_charging is not True: + print(" ERROR: reset_sample_state should not touch octopus_intelligent_charging any more") + failed = True + if my_predbat.debug_enable is not True: + print(" ERROR: reset_sample_state should not touch debug_enable any more") + failed = True + + print("Test: create_headless_predbat restores PREDBAT_APPS_FILE afterwards, even if it was previously unset") + saved_env = os.environ.get("PREDBAT_APPS_FILE") + try: + os.environ["PREDBAT_APPS_FILE"] = "/tmp/should-not-leak-annual-bootstrap-test.yaml" + with tempfile.TemporaryDirectory() as work_dir: + create_headless_predbat(work_dir, "Europe/London", print) + if os.environ.get("PREDBAT_APPS_FILE") != "/tmp/should-not-leak-annual-bootstrap-test.yaml": + print(" ERROR: PREDBAT_APPS_FILE was not restored to its prior value, got {}".format(os.environ.get("PREDBAT_APPS_FILE"))) + failed = True + + os.environ.pop("PREDBAT_APPS_FILE", None) + with tempfile.TemporaryDirectory() as work_dir: + create_headless_predbat(work_dir, "Europe/London", print) + if "PREDBAT_APPS_FILE" in os.environ: + print(" ERROR: PREDBAT_APPS_FILE should be unset again, got {}".format(os.environ.get("PREDBAT_APPS_FILE"))) + failed = True + finally: + if saved_env is None: + os.environ.pop("PREDBAT_APPS_FILE", None) + else: + os.environ["PREDBAT_APPS_FILE"] = saved_env + return failed From 26da4b8a586ea057f08b92ff6602fe0878d146c6 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 22:13:43 +0200 Subject: [PATCH 023/119] feat(annual): add irradiance-stratified sample day selection --- apps/predbat/annual.py | 56 +++++++++- apps/predbat/tests/test_annual_sampling.py | 118 +++++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 175 insertions(+), 1 deletion(-) create mode 100644 apps/predbat/tests/test_annual_sampling.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index bf9be8c97..d2ef91275 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -12,8 +12,9 @@ and tariff modules own all network access. """ +import calendar import os -from datetime import date +from datetime import date, timedelta from const import MINUTE_WATT @@ -472,3 +473,56 @@ def configure_offline_mode(predbat): predbat.carbon_enable = False predbat.plan_debug = False predbat.debug_enable = False + + +def _percentile_indices(count, samples): + """Return ``samples`` distinct indices spread evenly through ``count`` sorted items. + + Index i sits at percentile (i + 0.5) / samples, so two samples land at the 25th + and 75th percentiles and each represents an equal share of the month. Collisions + are resolved by walking to the nearest unused index, which only matters when the + sample count approaches the number of candidate days. + """ + chosen = [] + used = set() + for index in range(samples): + target = min(count - 1, int(count * ((index + 0.5) / samples))) + while target in used and target < count - 1: + target += 1 + while target in used and target > 0: + target -= 1 + if target in used: + continue + used.add(target) + chosen.append(target) + return chosen + + +def select_samples(weather, year, month, samples_per_month, has_solar=True): + """Choose the days to plan for one month, with the weight in days each represents. + + Days are ranked by their *actual* PV energy and sampled at even percentiles, so an + unlucky sunny or dull draw cannot swing the month. Ranking uses actuals rather than + the forecast: the aim is to represent what the month really contained, not what was + predicted. Days without a following day are excluded because the 48 hour plan needs one. + + Weights always sum to the number of days in the month, so a month with fewer usable + candidates than requested is scaled up rather than silently under-counted. + """ + days_in_month = calendar.monthrange(year, month)[1] + all_days = [date(year, month, day) for day in range(1, days_in_month + 1)] + + if has_solar: + candidates = [day for day in all_days if weather.has_actual(day) and weather.has_actual(day + timedelta(days=1))] + candidates.sort(key=lambda day: (weather.daily_actual_kwh(day), day)) + else: + # With no PV there is nothing to rank by, so fall back to evenly spaced calendar days + candidates = all_days + + if not candidates: + return [] + + indices = _percentile_indices(len(candidates), samples_per_month) + chosen = sorted({candidates[index] for index in indices}) + weight = days_in_month / float(len(chosen)) + return [(day, weight) for day in chosen] diff --git a/apps/predbat/tests/test_annual_sampling.py b/apps/predbat/tests/test_annual_sampling.py new file mode 100644 index 000000000..91a9ee1de --- /dev/null +++ b/apps/predbat/tests/test_annual_sampling.py @@ -0,0 +1,118 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for annual prediction sample day selection.""" + +from datetime import date, timedelta + +from annual import select_samples + + +class FakeWeather: + """A stub WeatherYear exposing only what select_samples needs.""" + + def __init__(self, daily): + """Hold a mapping of date to daily actual PV kWh.""" + self.daily = daily + + def has_actual(self, day): + """Return True when the date has actual PV data.""" + return day in self.daily + + def daily_actual_kwh(self, day): + """Return the actual PV kWh for the date.""" + return self.daily.get(day, 0.0) + + +def build_january(kwh_by_day, extra_days=1): + """Build a FakeWeather covering January plus a buffer into February.""" + daily = {} + for day_number, kwh in kwh_by_day.items(): + daily[date(2025, 1, day_number)] = kwh + for offset in range(extra_days): + daily[date(2025, 2, 1) + timedelta(days=offset)] = 1.0 + return FakeWeather(daily) + + +def test_annual_sampling(my_predbat): + """Verify percentile sampling, weighting, determinism and degraded months.""" + failed = False + print("**** Testing annual sample selection ****") + + # January: day N generates N kWh, so the sorted order is simply day order + weather = build_january({day: float(day) for day in range(1, 32)}) + + print("Test: two samples land on the 25th and 75th percentile days") + samples = select_samples(weather, 2025, 1, 2) + if len(samples) != 2: + print(" ERROR: expected 2 samples, got {}".format(len(samples))) + failed = True + else: + days = [day.day for day, _ in samples] + # 31 candidates: indices int(31*0.25)=7 and int(31*0.75)=23 -> days 8 and 24 + if days != [8, 24]: + print(" ERROR: expected days [8, 24], got {}".format(days)) + failed = True + + print("Test: weights sum to the number of days in the month") + total_weight = sum(weight for _, weight in samples) + if abs(total_weight - 31.0) > 1e-9: + print(" ERROR: weights should sum to 31, got {}".format(total_weight)) + failed = True + + print("Test: selection is deterministic") + if select_samples(weather, 2025, 1, 2) != samples: + print(" ERROR: repeated selection returned different days") + failed = True + + print("Test: samples are returned in date order") + ordered = [day for day, _ in select_samples(weather, 2025, 1, 4)] + if ordered != sorted(ordered): + print(" ERROR: samples should be returned in date order, got {}".format(ordered)) + failed = True + + print("Test: samples are distinct") + four = select_samples(weather, 2025, 1, 4) + if len({day for day, _ in four}) != 4: + print(" ERROR: expected 4 distinct days, got {}".format([day for day, _ in four])) + failed = True + + print("Test: a day without a following day is excluded, since the 48 hour plan needs one") + truncated = build_january({day: float(day) for day in range(1, 32)}, extra_days=0) + truncated_days = [day for day, _ in select_samples(truncated, 2025, 1, 2)] + if date(2025, 1, 31) in truncated_days: + print(" ERROR: 31 January has no following day and must not be sampled") + failed = True + + print("Test: a month with fewer candidates than samples uses every candidate") + sparse = build_january({1: 5.0, 2: 9.0, 3: 1.0}) + sparse_samples = select_samples(sparse, 2025, 1, 4) + # Day 3 has no following day (4 January is absent), so only days 1 and 2 are usable + if len(sparse_samples) != 2: + print(" ERROR: expected 2 usable samples, got {}".format(len(sparse_samples))) + failed = True + if abs(sum(weight for _, weight in sparse_samples) - 31.0) > 1e-9: + print(" ERROR: weights must still sum to 31 when samples are scarce, got {}".format(sum(w for _, w in sparse_samples))) + failed = True + + print("Test: a month with no usable candidates returns nothing") + if select_samples(FakeWeather({}), 2025, 1, 2) != []: + print(" ERROR: a month with no weather data should return no samples") + failed = True + + print("Test: a battery-only run with no solar falls back to evenly spaced calendar days") + no_solar = select_samples(FakeWeather({}), 2025, 1, 2, has_solar=False) + if len(no_solar) != 2: + print(" ERROR: with no solar, expected 2 calendar samples, got {}".format(len(no_solar))) + failed = True + elif [day.day for day in [entry[0] for entry in no_solar]] != [8, 24]: + print(" ERROR: expected evenly spaced days [8, 24], got {}".format([entry[0].day for entry in no_solar])) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index eb3b2db74..7acdece03 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -165,6 +165,7 @@ from tests.test_load_today_comparison import test_load_today_comparison from tests.test_annual_config import test_annual_config from tests.test_annual_bootstrap import test_annual_bootstrap +from tests.test_annual_sampling import test_annual_sampling # Mock the components and plugin system @@ -408,6 +409,7 @@ def main(): ("debug_cases", run_debug_cases, "Debug case file tests", True), ("annual_config", test_annual_config, "Annual prediction config validation tests", False), ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), + ("annual_sampling", test_annual_sampling, "Annual prediction sample selection tests", False), ] # Parse command line arguments From 676f27dfbbb9e15293278bea63030cd88c8c4b70 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 22:21:50 +0200 Subject: [PATCH 024/119] fix(annual): pin sample-selection edge cases from review - _percentile_indices(0, samples) now returns [] instead of a meaningless [-1]; add direct tests for count 0/1, samples > count and samples == count. - Document why the has_solar=False branch deliberately skips the following-day guard, and cover that a battery-only run still samples the last day of the month. - Add a December -> following-January boundary test through select_samples() to pin the timedelta roll-over. - Drop the redundant set() dedup in select_samples() (indices are already distinct) and soften the weight-sum docstring wording to acknowledge ordinary floating-point rounding. --- apps/predbat/annual.py | 25 +++++-- apps/predbat/tests/test_annual_sampling.py | 83 +++++++++++++++++++++- 2 files changed, 100 insertions(+), 8 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index d2ef91275..1e0656056 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -480,9 +480,14 @@ def _percentile_indices(count, samples): Index i sits at percentile (i + 0.5) / samples, so two samples land at the 25th and 75th percentiles and each represents an equal share of the month. Collisions - are resolved by walking to the nearest unused index, which only matters when the - sample count approaches the number of candidate days. + are resolved by scanning forward toward the end of the array and, failing that, + backward from the target, which only matters when the sample count approaches + the number of candidate days. With no candidates at all there is nothing to + index, so this returns an empty list rather than the meaningless index -1. """ + if count <= 0: + return [] + chosen = [] used = set() for index in range(samples): @@ -506,8 +511,9 @@ def select_samples(weather, year, month, samples_per_month, has_solar=True): the forecast: the aim is to represent what the month really contained, not what was predicted. Days without a following day are excluded because the 48 hour plan needs one. - Weights always sum to the number of days in the month, so a month with fewer usable - candidates than requested is scaled up rather than silently under-counted. + Weights sum to the number of days in the month, up to ordinary floating-point + rounding, so a month with fewer usable candidates than requested is scaled up + rather than silently under-counted. """ days_in_month = calendar.monthrange(year, month)[1] all_days = [date(year, month, day) for day in range(1, days_in_month + 1)] @@ -516,13 +522,20 @@ def select_samples(weather, year, month, samples_per_month, has_solar=True): candidates = [day for day in all_days if weather.has_actual(day) and weather.has_actual(day + timedelta(days=1))] candidates.sort(key=lambda day: (weather.daily_actual_kwh(day), day)) else: - # With no PV there is nothing to rank by, so fall back to evenly spaced calendar days + # With no PV there is nothing to rank by, so fall back to evenly spaced calendar days. + # Deliberately no following-day filter here: that guard exists solely to keep the + # sample within the weather series, and a battery-only run has no weather series to + # run past the end of. Day 2's rates come from the tariff module (which fetches the + # month plus a 2-day buffer, and the orchestrator additionally fetches the next month) + # and its load is synthetic and unbounded, so the last day of the month is a + # legitimate sample and must not be filtered out. candidates = all_days if not candidates: return [] + # _percentile_indices() already guarantees distinct indices, so no set() is needed here. indices = _percentile_indices(len(candidates), samples_per_month) - chosen = sorted({candidates[index] for index in indices}) + chosen = sorted(candidates[index] for index in indices) weight = days_in_month / float(len(chosen)) return [(day, weight) for day in chosen] diff --git a/apps/predbat/tests/test_annual_sampling.py b/apps/predbat/tests/test_annual_sampling.py index 91a9ee1de..812f5f7a6 100644 --- a/apps/predbat/tests/test_annual_sampling.py +++ b/apps/predbat/tests/test_annual_sampling.py @@ -11,7 +11,7 @@ from datetime import date, timedelta -from annual import select_samples +from annual import _percentile_indices, select_samples class FakeWeather: @@ -40,11 +40,60 @@ def build_january(kwh_by_day, extra_days=1): return FakeWeather(daily) +def build_december(kwh_by_day, extra_days=1): + """Build a FakeWeather covering December plus a buffer into January of the following year.""" + daily = {} + for day_number, kwh in kwh_by_day.items(): + daily[date(2025, 12, day_number)] = kwh + for offset in range(extra_days): + daily[date(2026, 1, 1) + timedelta(days=offset)] = 1.0 + return FakeWeather(daily) + + +def _check_indices(indices, count, samples, label, failed): + """Assert that indices from _percentile_indices() are distinct and within range. + + Shared by the degenerate-case checks below so each one only has to state its + inputs and let this helper verify the two invariants every case must satisfy. + Returns the (possibly updated) ``failed`` flag. + """ + if len(set(indices)) != len(indices): + print(" ERROR: {} produced duplicate indices, got {}".format(label, indices)) + failed = True + if any(index < 0 or index >= count for index in indices): + print(" ERROR: {} produced an out-of-range index, got {} for count {}".format(label, indices, count)) + failed = True + return failed + + def test_annual_sampling(my_predbat): - """Verify percentile sampling, weighting, determinism and degraded months.""" + """Verify percentile sampling, weighting, determinism, degraded months and year boundaries.""" failed = False print("**** Testing annual sample selection ****") + print("Test: _percentile_indices with no candidates returns nothing") + if _percentile_indices(0, 2) != []: + print(" ERROR: expected no indices for a candidate count of 0, got {}".format(_percentile_indices(0, 2))) + failed = True + + print("Test: _percentile_indices with a single candidate always picks index 0") + for requested in (1, 2, 5): + single = _percentile_indices(1, requested) + if single != [0]: + print(" ERROR: expected [0] for count=1, samples={}, got {}".format(requested, single)) + failed = True + + print("Test: _percentile_indices with more samples requested than candidates stays in range") + over_requested = _percentile_indices(3, 7) + failed = _check_indices(over_requested, 3, 7, "samples > count", failed) + + print("Test: _percentile_indices with samples equal to count selects every index") + exact = _percentile_indices(5, 5) + if sorted(exact) != list(range(5)): + print(" ERROR: expected every index 0-4 when samples == count, got {}".format(sorted(exact))) + failed = True + failed = _check_indices(exact, 5, 5, "samples == count", failed) + # January: day N generates N kWh, so the sorted order is simply day order weather = build_january({day: float(day) for day in range(1, 32)}) @@ -115,4 +164,34 @@ def test_annual_sampling(my_predbat): print(" ERROR: expected evenly spaced days [8, 24], got {}".format([entry[0].day for entry in no_solar])) failed = True + print("Test: a battery-only run still includes the last day of the month when requested") + # samples_per_month == days_in_month so every calendar day, including the last, is chosen; + # weather has no data at all, confirming the no-solar branch never needs a following-day check + no_solar_full_month = select_samples(FakeWeather({}), 2025, 12, 31, has_solar=False) + if len(no_solar_full_month) != 31: + print(" ERROR: expected all 31 December calendar days, got {}".format(len(no_solar_full_month))) + failed = True + else: + last_day, last_weight = no_solar_full_month[-1] + if last_day != date(2025, 12, 31): + print(" ERROR: expected the last sample to be 31 December, got {}".format(last_day)) + failed = True + if abs(last_weight - 1.0) > 1e-9: + print(" ERROR: expected a weight of 1.0 for 31 December, got {}".format(last_weight)) + failed = True + + print("Test: a December selection with solar consults 1 January of the following year") + december = build_december({day: float(day) for day in range(1, 32)}) + december_days = [day for day, _ in select_samples(december, 2025, 12, 31)] + if date(2025, 12, 31) not in december_days: + print(" ERROR: 31 December should be included when 1 January of the following year has data") + failed = True + + print("Test: a December selection with solar excludes 31 December when the new year has no data") + december_truncated = build_december({day: float(day) for day in range(1, 32)}, extra_days=0) + december_truncated_days = [day for day, _ in select_samples(december_truncated, 2025, 12, 31)] + if date(2025, 12, 31) in december_truncated_days: + print(" ERROR: 31 December has no following day in the next year and must not be sampled") + failed = True + return failed From 93e3d15f6d7c5dbebcc9987a2a486a4698aaab76 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sat, 25 Jul 2026 22:37:22 +0200 Subject: [PATCH 025/119] feat(annual): add three-scenario day runner Runs no-PV/battery, PV+battery-on-a-timer, and Predbat-planned scenarios for one sampled day through the real Predbat engine and returns their billed figures. Predbat plans on the forecast PV series and is costed against the actuals series so it is never given perfect foresight; only the Predbat scenario gets a smart car, the other two use a fixed off-peak timer. Fixes a gap in the plan: calculate_plan() never calls plan_car_charging() itself (that only happens in the live fetch cycle or Compare's helper, both bypassed here), so scenario 3 now calls plan_car_charging() directly to populate car_charging_slots -- otherwise the smart car would silently never charge and would look free in the Predbat scenario. --- apps/predbat/annual.py | 304 ++++++++++++++++++++ apps/predbat/tests/test_annual_scenarios.py | 85 ++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 391 insertions(+) create mode 100644 apps/predbat/tests/test_annual_scenarios.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 1e0656056..032b8fe47 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -16,7 +16,9 @@ import os from datetime import date, timedelta +from annual_load import build_load_forecast from const import MINUTE_WATT +from prediction import Prediction VALID_SHAPES = ["night", "day", "flat"] @@ -539,3 +541,305 @@ def select_samples(weather, year, month, samples_per_month, has_solar=True): chosen = sorted(candidates[index] for index in indices) weight = days_in_month / float(len(chosen)) return [(day, weight) for day in chosen] + + +DAY_MINUTES = 24 * 60 +PLAN_MINUTES = 48 * 60 + +# Every sample starts from an empty battery. The compute_metric correction values +# whatever charge is left at the end, so the starting level does not bias the cost. +START_SOC_KWH = 0.0 + +# Cars are charged at this rate when the config gives no explicit figure +DEFAULT_CAR_RATE_KW = 7.4 + +# Maximum charge slots the dumb-battery baseline is allowed, matching the +# calculate_savings_max_charge_slots convention in calculate_yesterday() +BASELINE_MAX_CHARGE_SLOTS = 1 + +# The smart car in scenario 3 must be ready by this time, matching the +# "car_charging_plan_time" default in config.py. Only the first day of the 48 +# hour plan is billed (see run_day()'s end_record), so a single morning ready +# time is enough to give Predbat a fair one-shot charging decision to make. +DEFAULT_CAR_READY_TIME = "07:00:00" + + +def build_step_data(predbat, pv_minute, pv_minute10): + """Build the 5-minute step arrays the Prediction engine consumes. + + Mirrors the calls ``calculate_plan()`` makes in ``plan.py``. Because + ``load_forecast_only`` is set, the historical branch of ``step_data_history`` + contributes nothing and the whole load profile comes from ``load_forecast``. + """ + load_step = predbat.step_data_history( + predbat.load_minutes, + predbat.minutes_now, + forward=False, + scale_today=1.0, + scale_fixed=1.0, + type_load=True, + load_forecast=predbat.load_forecast, + load_scaling_dynamic=None, + cloud_factor=None, + load_adjust={}, + load_baseline={}, + ) + pv_step = predbat.step_data_history(pv_minute, predbat.minutes_now, forward=True, cloud_factor=None) + pv10_step = predbat.step_data_history(pv_minute10, predbat.minutes_now, forward=True, cloud_factor=None, flip=True) + return load_step, pv_step, pv10_step + + +def timer_charge_window(rate_import, car_kwh, car_rate_kw): + """Return the fixed off-peak timer windows a non-Predbat household would use. + + Finds the cheapest contiguous band of each day and starts the charge there, + extending past the band if the car needs longer than the cheap rate lasts. + Returns one window per day of the 48 hour plan so the second day matches the first. + """ + if car_kwh <= 0 or car_rate_kw <= 0: + return [] + + minutes_needed = int(round((car_kwh / car_rate_kw) * 60.0)) + if minutes_needed <= 0: + return [] + + windows = [] + for day_offset in range(2): + base = day_offset * DAY_MINUTES + day_rates = {minute: rate_import.get(base + minute, 0.0) for minute in range(DAY_MINUTES)} + if not day_rates: + continue + cheapest = min(day_rates.values()) + # The first minute of the longest run at the cheapest rate + start = None + best_start = 0 + best_length = 0 + for minute in range(DAY_MINUTES + 1): + at_cheapest = minute < DAY_MINUTES and day_rates[minute] <= cheapest + 1e-9 + if at_cheapest and start is None: + start = minute + elif not at_cheapest and start is not None: + if minute - start > best_length: + best_length = minute - start + best_start = start + start = None + windows.append({"start": base + best_start, "end": base + best_start + minutes_needed}) + return windows + + +def add_car_to_load(load_forecast, windows, car_kwh): + """Return a copy of the cumulative load series with the car's energy inserted. + + Used by the two baseline scenarios, where the car is simply extra load in a + fixed timer window rather than something Predbat schedules. + """ + if not windows or car_kwh <= 0: + return dict(load_forecast) + + per_window = car_kwh / float(len(windows)) + additions = {} + for window in windows: + length = max(1, window["end"] - window["start"]) + per_minute = per_window / length + for minute in range(window["start"], window["end"]): + additions[minute] = additions.get(minute, 0.0) + per_minute + + result = {} + running_extra = 0.0 + for minute in sorted(load_forecast.keys()): + result[minute] = load_forecast[minute] + running_extra + running_extra += additions.get(minute, 0.0) + return result + + +def _apply_rates(predbat, rate_import, rate_export): + """Install the day's rates and run the scans the planner depends on.""" + predbat.rate_import = rate_import + predbat.rate_export = rate_export + predbat.rate_low_threshold = 0 + predbat.rate_high_threshold = 0 + + if predbat.rate_import: + predbat.rate_scan(predbat.rate_import, print=False) + predbat.rate_import, predbat.rate_import_replicated = predbat.rate_replicate(predbat.rate_import, is_import=True) + predbat.rate_scan(predbat.rate_import, print=False) + if predbat.rate_export: + predbat.rate_scan_export(predbat.rate_export, print=False) + predbat.rate_export, predbat.rate_export_replicated = predbat.rate_replicate(predbat.rate_export, is_import=False) + predbat.rate_scan_export(predbat.rate_export, print=False) + + predbat.set_rate_thresholds() + + if predbat.rate_export: + predbat.high_export_rates, export_lowest, _ = predbat.rate_scan_window(predbat.rate_export, 5, predbat.rate_export_cost_threshold, True) + if predbat.rate_high_threshold == 0 and export_lowest <= predbat.rate_export_max: + predbat.rate_export_cost_threshold = export_lowest + else: + predbat.high_export_rates = [] + + if predbat.rate_import: + predbat.low_rates, _, highest = predbat.rate_scan_window(predbat.rate_import, 5, predbat.rate_import_cost_threshold, False) + if predbat.rate_low_threshold == 0 and highest >= predbat.rate_min: + predbat.rate_import_cost_threshold = highest + else: + predbat.low_rates = [] + + +def _baseline_charge_window(predbat): + """Return the dumb battery's charge windows: the cheapest static band, charged to full. + + Mirrors the baseline in ``calculate_yesterday()`` — a household without Predbat + that sets a timer for the cheapest rate and charges to 100%. + """ + if not predbat.rate_import or predbat.soc_max <= 0: + return [], [] + day_values = [value for minute, value in predbat.rate_import.items() if minute < DAY_MINUTES] + if not day_values or min(day_values) == max(day_values): + return [], [] + + combine = predbat.combine_charge_slots + predbat.combine_charge_slots = True + windows, _, _ = predbat.rate_scan_window(predbat.rate_import, 5, min(day_values), False, return_raw=True) + predbat.combine_charge_slots = combine + + windows = [window for window in windows if window["start"] < PLAN_MINUTES][:BASELINE_MAX_CHARGE_SLOTS] + return windows, [predbat.soc_max for _ in windows] + + +def _billed_result(predbat, end_record, pv_step): + """Run one scenario to completion and return its billed figures. + + The battery-value correction (metric_end minus metric_start) values whatever + charge is left at the end, so a scenario cannot look cheap simply by finishing + on an empty battery. This is the same correction ``Compare.run_scenario()`` applies. + """ + cost, import_kwh_battery, import_kwh_house, export_kwh, _, final_soc, _, battery_cycle, metric_keep, final_iboost, final_carbon_g = predbat.run_prediction( + predbat.charge_limit_best, predbat.charge_window_best, predbat.export_window_best, predbat.export_limits_best, False, end_record=end_record + ) + metric_start, _ = predbat.compute_metric(end_record, predbat.soc_kw, predbat.soc_kw, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) + metric_end, _ = predbat.compute_metric(end_record, final_soc, final_soc, cost, cost, final_iboost, final_iboost, battery_cycle, metric_keep, final_carbon_g, import_kwh_battery, import_kwh_house, export_kwh) + + pv_generated = sum(value for minute, value in pv_step.items() if minute < end_record) + return { + "cost_p": metric_end - metric_start, + "import_kwh": import_kwh_battery + import_kwh_house, + "export_kwh": export_kwh, + "pv_generated_kwh": pv_generated, + "battery_throughput_kwh": battery_cycle, + } + + +def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc): + """Inject every per-day input into the PredBat instance for one sampled day.""" + reset_sample_state(predbat) + + predbat.midnight_utc = midnight_utc + predbat.now_utc = midnight_utc + predbat.minutes_now = 0 + predbat.forecast_plan_hours = 48 + predbat.forecast_minutes = PLAN_MINUTES + predbat.forecast_days = 2 + predbat.end_record = PLAN_MINUTES + + predbat.load_minutes = {} + predbat.load_minutes_age = 0 + predbat.load_forecast = build_load_forecast(load_source, day, 2) + + rate_import, rate_export = tariff.rates_for(midnight_utc, PLAN_MINUTES) + _apply_rates(predbat, rate_import, rate_export) + + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + + +def _plan_smart_car(predbat): + """Configure a single smart-charged car and return its planned charging slots. + + ``calculate_plan()`` never calls ``plan_car_charging()`` itself — that only + happens in the real fetch cycle (``fetch_sensor_data_car_planning()`` in + ``fetch.py``) or in ``Compare.recompute_car_charging()``, both of which this + headless tool bypasses. Without calling it here, ``car_charging_slots`` would + stay at whatever empty placeholder scenario setup left it as, ``in_car_slot()`` + would report zero load for every minute, and the car would silently cost + nothing at all in the "with Predbat" scenario — a much bigger and more + misleading error than merely losing Predbat's optimisation credit. The + ready-by time and max price mirror ``config.py``'s defaults for a car with no + explicit user override. + """ + predbat.car_charging_now = [False] + predbat.car_charging_plan_time = [DEFAULT_CAR_READY_TIME] + predbat.car_charging_plan_max_price = [0] + return predbat.plan_car_charging(0, predbat.low_rates) + + +def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): + """Run all three scenarios against one sampled day and return their billed figures.""" + car_kwh = config["load"].get("car_charging_kwh", 0.0) / 365.0 + car_rate_kw = config["load"].get("car_rate_kw", DEFAULT_CAR_RATE_KW) + + prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc) + + actual_pv = weather.pv_minutes("actual", midnight_utc, PLAN_MINUTES) if config["solar"] else {} + forecast_pv = weather.pv_minutes("forecast", midnight_utc, PLAN_MINUTES) if config["solar"] else {} + p10_pv = weather.pv_minutes_p10(midnight_utc, PLAN_MINUTES, day.month) if config["solar"] else {} + + timer_windows = timer_charge_window(predbat.rate_import, car_kwh, car_rate_kw) + baseline_load = add_car_to_load(predbat.load_forecast, timer_windows, car_kwh) + + results = {} + + # Scenario 1: no PV, no battery. The car still charges on the same timer, so the + # only difference between the scenarios is the system being evaluated. + predbat.num_cars = 0 + predbat.load_forecast = baseline_load + load_step, actual_step, _ = build_step_data(predbat, actual_pv, actual_pv) + zero_step = {minute: 0.0 for minute in actual_step} + predbat.charge_limit_best = [] + predbat.charge_window_best = [] + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.prediction = Prediction(predbat, zero_step, zero_step, load_step, load_step, soc_kw=0, soc_max=0) + results["no_pvbat"] = _billed_result(predbat, DAY_MINUTES, zero_step) + + # Scenario 2: PV and battery on a dumb cheapest-rate timer, no export optimisation + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + charge_window, charge_limit = _baseline_charge_window(predbat) + predbat.charge_window_best = charge_window + predbat.charge_limit_best = charge_limit + predbat.export_window_best = [] + predbat.export_limits_best = [] + predbat.prediction = Prediction(predbat, actual_step, actual_step, load_step, load_step, soc_kw=START_SOC_KWH) + results["without_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) + + # Scenario 3: Predbat plans on the FORECAST, then is costed against the ACTUALS. + # Skipping the Prediction swap below would hand Predbat perfect foresight. + predbat.load_forecast = build_load_forecast(load_source, day, 2) + if car_kwh > 0: + predbat.num_cars = 1 + predbat.car_charging_planned = [True] + predbat.car_charging_plan_smart = [True] + predbat.car_charging_battery_size = [max(car_kwh * 2, 50.0)] + predbat.car_charging_limit = [car_kwh] + predbat.car_charging_soc = [0.0] + predbat.car_charging_rate = [car_rate_kw] + predbat.car_charging_from_battery = False + # low_rates was computed from this day's rates in prepare_sample() and untouched + # since, so it is safe to use for planning the car here. + predbat.car_charging_slots = [_plan_smart_car(predbat)] + else: + predbat.num_cars = 0 + predbat.car_charging_slots = [] + + apply_hardware(predbat, config["battery"], config["solar"]) + predbat.soc_kw = START_SOC_KWH + predbat.pv_forecast_minute = forecast_pv + predbat.pv_forecast_minute10 = p10_pv + predbat.calculate_plan(recompute=True, debug_mode=False, publish=False) + + # Swap in the actuals before costing + forecast_load_step, _, _ = build_step_data(predbat, forecast_pv, p10_pv) + predbat.prediction = Prediction(predbat, actual_step, actual_step, forecast_load_step, forecast_load_step, soc_kw=START_SOC_KWH) + results["with_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) + + return results diff --git a/apps/predbat/tests/test_annual_scenarios.py b/apps/predbat/tests/test_annual_scenarios.py new file mode 100644 index 000000000..ced333275 --- /dev/null +++ b/apps/predbat/tests/test_annual_scenarios.py @@ -0,0 +1,85 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction three-scenario day runner.""" + +from annual import DAY_MINUTES, PLAN_MINUTES, add_car_to_load, timer_charge_window + + +def flat_rates(cheap_start, cheap_end, cheap_rate, peak_rate): + """Build a 48 hour import rate dict with one cheap overnight band per day.""" + rates = {} + for minute in range(PLAN_MINUTES): + in_day = minute % DAY_MINUTES + rates[minute] = cheap_rate if cheap_start <= in_day < cheap_end else peak_rate + return rates + + +def test_annual_scenarios(my_predbat): + """Verify the timer charge window and car load insertion helpers.""" + failed = False + print("**** Testing annual scenario helpers ****") + + print("Test: timer_charge_window finds the cheapest band and sizes it to the car's energy") + rates = flat_rates(cheap_start=30, cheap_end=330, cheap_rate=7.0, peak_rate=30.0) + window = timer_charge_window(rates, car_kwh=14.8, car_rate_kw=7.4) + if not window: + print(" ERROR: expected a charge window") + failed = True + else: + first = window[0] + if first["start"] != 30: + print(" ERROR: the window should start at the cheap band start 30, got {}".format(first["start"])) + failed = True + # 14.8 kWh at 7.4 kW is 2 hours + if first["end"] - first["start"] != 120: + print(" ERROR: expected a 120 minute window for 14.8 kWh at 7.4 kW, got {}".format(first["end"] - first["start"])) + failed = True + + print("Test: a car needing more than the cheap band gets a window extended beyond it") + long_window = timer_charge_window(rates, car_kwh=74.0, car_rate_kw=7.4) + if long_window[0]["end"] - long_window[0]["start"] < 300: + print(" ERROR: a 10 hour charge should extend past the 5 hour cheap band, got {} minutes".format(long_window[0]["end"] - long_window[0]["start"])) + failed = True + + print("Test: zero car energy produces no window") + if timer_charge_window(rates, car_kwh=0.0, car_rate_kw=7.4) != []: + print(" ERROR: no car energy should produce no window") + failed = True + + print("Test: add_car_to_load inserts exactly the car's energy and stays cumulative") + base = {minute: 0.01 * minute for minute in range(PLAN_MINUTES + 1)} + with_car = add_car_to_load(base, window, car_kwh=14.8) + added = with_car[PLAN_MINUTES] - base[PLAN_MINUTES] + if abs(added - 14.8) > 1e-6: + print(" ERROR: expected 14.8 kWh added over the window, got {}".format(added)) + failed = True + for minute in range(1, PLAN_MINUTES + 1): + if with_car[minute] < with_car[minute - 1] - 1e-12: + print(" ERROR: the series stopped being cumulative at minute {}".format(minute)) + failed = True + break + + print("Test: add_car_to_load leaves minutes before the window untouched") + if abs(with_car[10] - base[10]) > 1e-12: + print(" ERROR: minutes before the window must be unchanged") + failed = True + + print("Test: add_car_to_load does not mutate its input") + if abs(base[PLAN_MINUTES] - 0.01 * PLAN_MINUTES) > 1e-9: + print(" ERROR: add_car_to_load must not mutate the input series") + failed = True + + print("Test: the car repeats on the second day of the window") + second_day = [entry for entry in window if entry["start"] >= DAY_MINUTES] + if not second_day: + print(" ERROR: the timer should also charge on day two of the 48 hour window") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 7acdece03..059439442 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -166,6 +166,7 @@ from tests.test_annual_config import test_annual_config from tests.test_annual_bootstrap import test_annual_bootstrap from tests.test_annual_sampling import test_annual_sampling +from tests.test_annual_scenarios import test_annual_scenarios # Mock the components and plugin system @@ -410,6 +411,7 @@ def main(): ("annual_config", test_annual_config, "Annual prediction config validation tests", False), ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), ("annual_sampling", test_annual_sampling, "Annual prediction sample selection tests", False), + ("annual_scenarios", test_annual_scenarios, "Annual prediction scenario helper tests", False), ] # Parse command line arguments From a1ddb663fed455e684b141cc93241173cc880614 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 07:45:17 +0200 Subject: [PATCH 026/119] fix(annual): correct planner-off, car-billing and metric leakage bugs in Task 10 Coordinator review of the three-scenario day runner found several bugs that biased the comparison, most critically that the headless bootstrap's Monitor-mode default left calculate_best_charge/calculate_best_export/ set_charge_window/set_export_window all False, so calculate_plan() never produced a charge or export window and "with Predbat" scored no better than demand-only. - Enable the four planner flags in configure_offline_mode(); assert them both on the mock fixture and on a real create_headless_predbat() instance. - Fix add_car_to_load() billing half the car's daily energy per day instead of the full amount (it divided by len(windows) when car_kwh was already a per-day figure). - Set car_charging_from_battery = True in scenario 3 so the battery is free to serve the car exactly as it implicitly is in scenarios 1/2. - Warn when plan_car_charging() cannot fit the car's full daily energy before its ready time, so a shortfall is visible rather than silently under-billed. - Set num_cars from config before _apply_rates() runs in prepare_sample(), removing an order-dependent leak into set_rate_thresholds() from the previous sample's scenario 3. - Match Compare.run_scenario()'s zeroed end-of-period compute_metric() call exactly, so the optimiser's internal metric_keep heuristic cannot leak into the reported billed cost. - Round the car's timer window to a 5 minute grid so step_data_history()'s sampling cannot quantise its billed duration. - Rename forecast_load_step to predbat_load_step (there is no forecast/actual split for load, only PV). Verified end-to-end with the planner enabled: with_predbat now costs less than without_predbat, which costs less than no_pvbat, on a synthetic day with a cheap overnight band. --- apps/predbat/annual.py | 98 ++++++++++++++++++--- apps/predbat/tests/test_annual_bootstrap.py | 26 ++++++ apps/predbat/tests/test_annual_scenarios.py | 25 +++++- 3 files changed, 131 insertions(+), 18 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 032b8fe47..da59e9f4e 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -17,7 +17,7 @@ from datetime import date, timedelta from annual_load import build_load_forecast -from const import MINUTE_WATT +from const import MINUTE_WATT, PREDICT_STEP from prediction import Prediction VALID_SHAPES = ["night", "day", "flat"] @@ -466,6 +466,18 @@ def configure_offline_mode(predbat): ``fetch_config_options()`` has populated its own defaults, and are not part of ``reset_sample_state()`` because re-applying them every sample would silently override whatever ``fetch_config_options()`` or a scenario override set. + + ``fetch_config_options()`` derives ``calculate_best_charge``, ``calculate_best_export``, + ``set_charge_window`` and ``set_export_window`` from ``predbat_mode`` (``fetch.py``), and + the minimal headless ``apps.yaml`` has no ``mode`` key, so ``get_arg("mode")`` returns the + "Monitor" default — which sets all four False. With all four False, ``calculate_plan()``'s + charge- and export-window branches (``plan.py``, gated on + ``self.low_rates and self.calculate_best_charge and self.set_charge_window`` and the export + equivalent) never fire, ``charge_window_best``/``export_window_best`` fall back to the + (empty) live ``charge_window``/``export_window``, and the "with Predbat" scenario would plan + no charging or exporting at all — a silent demand-only system indistinguishable from + scenario 1. These four are therefore forced on here, matching what + ``PREDBAT_MODE_CONTROL_CHARGEDISCHARGE`` sets in a live install with full control enabled. """ predbat.octopus_intelligent_charging = False predbat.load_forecast_only = True @@ -475,6 +487,10 @@ def configure_offline_mode(predbat): predbat.carbon_enable = False predbat.plan_debug = False predbat.debug_enable = False + predbat.calculate_best_charge = True + predbat.calculate_best_export = True + predbat.set_charge_window = True + predbat.set_export_window = True def _percentile_indices(count, samples): @@ -603,6 +619,13 @@ def timer_charge_window(rate_import, car_kwh, car_rate_kw): if minutes_needed <= 0: return [] + # fetch.py's step_data_history() samples load_forecast once every PREDICT_STEP (5) minutes, + # so a window whose length or start is not on that grid is billed at a quantised length + # rather than its true one (e.g. an 81 minute need would be billed as 85 minutes, +5%). + # Round the duration UP so the car is never under-delivered, and align the start DOWN to + # the grid. + minutes_needed = -(-minutes_needed // PREDICT_STEP) * PREDICT_STEP + windows = [] for day_offset in range(2): base = day_offset * DAY_MINUTES @@ -623,7 +646,8 @@ def timer_charge_window(rate_import, car_kwh, car_rate_kw): best_length = minute - start best_start = start start = None - windows.append({"start": base + best_start, "end": base + best_start + minutes_needed}) + aligned_start = base + (best_start // PREDICT_STEP) * PREDICT_STEP + windows.append({"start": aligned_start, "end": aligned_start + minutes_needed}) return windows @@ -631,12 +655,18 @@ def add_car_to_load(load_forecast, windows, car_kwh): """Return a copy of the cumulative load series with the car's energy inserted. Used by the two baseline scenarios, where the car is simply extra load in a - fixed timer window rather than something Predbat schedules. + fixed timer window rather than something Predbat schedules. ``car_kwh`` is + already the day's energy (``run_day`` divides the annual figure by 365) and + ``windows`` holds one window per day of the plan, each independently sized by + ``timer_charge_window`` to hold the full ``car_kwh`` — so every window gets + the full daily amount, not a share of it. Dividing by ``len(windows)`` here + would silently bill only half the car's energy on the one day that matters + (day 1 is the only one ``_billed_result`` costs). """ if not windows or car_kwh <= 0: return dict(load_forecast) - per_window = car_kwh / float(len(windows)) + per_window = car_kwh additions = {} for window in windows: length = max(1, window["end"] - window["start"]) @@ -711,13 +741,24 @@ def _billed_result(predbat, end_record, pv_step): The battery-value correction (metric_end minus metric_start) values whatever charge is left at the end, so a scenario cannot look cheap simply by finishing - on an empty battery. This is the same correction ``Compare.run_scenario()`` applies. + on an empty battery. This is exactly the correction ``Compare.run_scenario()`` + applies (``compare.py``), including passing zero for ``battery_cycle``, + ``metric_keep``, ``final_carbon_g``, ``import_kwh_battery``, ``import_kwh_house`` + and ``export_kwh`` in the end-of-period ``compute_metric()`` call, matching what + the start-of-period call already zeroes. Passing the real values there instead + (an earlier version of this function did) would leak the optimiser's internal + planning heuristics into a reported billed cost: ``metric_keep`` in particular + is added into ``compute_metric()``'s "metric" unconditionally, not gated by a + zero-default weight the way the carbon/self-sufficiency/cycle terms are, so it + would inflate ``cost_p`` for any scenario whose plan happens to accrue it — + typically the battery scenarios, never the no-battery baseline — for a reason + that has nothing to do with money actually billed. """ - cost, import_kwh_battery, import_kwh_house, export_kwh, _, final_soc, _, battery_cycle, metric_keep, final_iboost, final_carbon_g = predbat.run_prediction( + cost, import_kwh_battery, import_kwh_house, export_kwh, _, final_soc, _, battery_cycle, _, final_iboost, _ = predbat.run_prediction( predbat.charge_limit_best, predbat.charge_window_best, predbat.export_window_best, predbat.export_limits_best, False, end_record=end_record ) metric_start, _ = predbat.compute_metric(end_record, predbat.soc_kw, predbat.soc_kw, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) - metric_end, _ = predbat.compute_metric(end_record, final_soc, final_soc, cost, cost, final_iboost, final_iboost, battery_cycle, metric_keep, final_carbon_g, import_kwh_battery, import_kwh_house, export_kwh) + metric_end, _ = predbat.compute_metric(end_record, final_soc, final_soc, cost, cost, final_iboost, final_iboost, 0, 0, 0, 0, 0, 0) pv_generated = sum(value for minute, value in pv_step.items() if minute < end_record) return { @@ -745,6 +786,14 @@ def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_ predbat.load_minutes_age = 0 predbat.load_forecast = build_load_forecast(load_source, day, 2) + # set_rate_thresholds() (called from _apply_rates() below) reads num_cars to decide + # whether to widen rate_import_cost_threshold for car charging (fetch.py). num_cars is + # not part of reset_sample_state()'s leaked-state list — it is a value this day's config + # determines, not a scenario override to merely clear — so it is set here, from config, + # before _apply_rates() runs rather than left at whatever the previous sample's scenario 3 + # happened to leave it as (or the headless bootstrap's own default of 1). + predbat.num_cars = 1 if config["load"].get("car_charging_kwh", 0.0) > 0 else 0 + rate_import, rate_export = tariff.rates_for(midnight_utc, PLAN_MINUTES) _apply_rates(predbat, rate_import, rate_export) @@ -752,7 +801,7 @@ def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_ predbat.soc_kw = START_SOC_KWH -def _plan_smart_car(predbat): +def _plan_smart_car(predbat, day, car_kwh): """Configure a single smart-charged car and return its planned charging slots. ``calculate_plan()`` never calls ``plan_car_charging()`` itself — that only @@ -765,11 +814,25 @@ def _plan_smart_car(predbat): misleading error than merely losing Predbat's optimisation credit. The ready-by time and max price mirror ``config.py``'s defaults for a car with no explicit user override. + + Unlike ``timer_charge_window()`` (which always extends past the cheap band + until the car's full energy fits), ``plan_car_charging()`` stops at the + ready time and can therefore return a plan short of ``car_kwh`` if the + cheap-rate windows before then cannot hold it all. A shortfall here would + make scenario 3 look artificially cheap for delivering less car energy than + the other two scenarios are billed for — inflating Predbat's apparent + saving, the mirror image of ``add_car_to_load()``'s bug where the baseline + scenarios were billed for too little and Predbat's saving looked smaller + than it was — so a shortfall is logged rather than left to pass unnoticed. """ predbat.car_charging_now = [False] predbat.car_charging_plan_time = [DEFAULT_CAR_READY_TIME] predbat.car_charging_plan_max_price = [0] - return predbat.plan_car_charging(0, predbat.low_rates) + slots = predbat.plan_car_charging(0, predbat.low_rates) + planned_kwh = sum(slot.get("kwh", 0.0) for slot in slots) + if planned_kwh < car_kwh - 1e-6: + predbat.log("Warn: Annual: {} smart car plan only fitted {:.2f} of {:.2f} kWh into the cheap-rate windows before the {} ready time".format(day, planned_kwh, car_kwh, DEFAULT_CAR_READY_TIME)) + return slots def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): @@ -823,10 +886,15 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): predbat.car_charging_limit = [car_kwh] predbat.car_charging_soc = [0.0] predbat.car_charging_rate = [car_rate_kw] - predbat.car_charging_from_battery = False + # In scenarios 1 and 2 the car's energy is baked into ordinary house load + # (add_car_to_load()), which the battery may serve freely. Leaving this False would + # pin discharge_rate_now to battery_rate_min for every car-charging minute + # (prediction.py), forcing scenario 3's battery idle exactly when the other two + # scenarios let it help — the same physics must apply to both sides of the comparison. + predbat.car_charging_from_battery = True # low_rates was computed from this day's rates in prepare_sample() and untouched # since, so it is safe to use for planning the car here. - predbat.car_charging_slots = [_plan_smart_car(predbat)] + predbat.car_charging_slots = [_plan_smart_car(predbat, day, car_kwh)] else: predbat.num_cars = 0 predbat.car_charging_slots = [] @@ -837,9 +905,11 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): predbat.pv_forecast_minute10 = p10_pv predbat.calculate_plan(recompute=True, debug_mode=False, publish=False) - # Swap in the actuals before costing - forecast_load_step, _, _ = build_step_data(predbat, forecast_pv, p10_pv) - predbat.prediction = Prediction(predbat, actual_step, actual_step, forecast_load_step, forecast_load_step, soc_kw=START_SOC_KWH) + # Swap in the actuals before costing. There is no forecast/actual split for load (only PV + # has one) — predbat_load_step is just the household load re-sampled onto the PREDICT_STEP + # grid, identical regardless of which PV series build_step_data() was called with. + predbat_load_step, _, _ = build_step_data(predbat, forecast_pv, p10_pv) + predbat.prediction = Prediction(predbat, actual_step, actual_step, predbat_load_step, predbat_load_step, soc_kw=START_SOC_KWH) results["with_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) return results diff --git a/apps/predbat/tests/test_annual_bootstrap.py b/apps/predbat/tests/test_annual_bootstrap.py index 375890671..d3946abde 100644 --- a/apps/predbat/tests/test_annual_bootstrap.py +++ b/apps/predbat/tests/test_annual_bootstrap.py @@ -211,6 +211,10 @@ def test_annual_bootstrap(my_predbat): my_predbat.carbon_enable = True my_predbat.plan_debug = True my_predbat.debug_enable = True + my_predbat.calculate_best_charge = False + my_predbat.calculate_best_export = False + my_predbat.set_charge_window = False + my_predbat.set_export_window = False configure_offline_mode(my_predbat) @@ -223,6 +227,13 @@ def test_annual_bootstrap(my_predbat): ("carbon_enable", False), ("plan_debug", False), ("debug_enable", False), + # Without these, calculate_plan()'s charge/export-window branches never fire (they are + # gated on calculate_best_charge/set_charge_window and the export equivalent), and the + # "with Predbat" scenario would silently plan no charging or exporting at all. + ("calculate_best_charge", True), + ("calculate_best_export", True), + ("set_charge_window", True), + ("set_export_window", True), ] for name, expected in offline_checks: actual = getattr(my_predbat, name) @@ -241,6 +252,21 @@ def test_annual_bootstrap(my_predbat): print(" ERROR: reset_sample_state should not touch debug_enable any more") failed = True + print("Test: create_headless_predbat leaves the planner switched on, not the Monitor-mode default") + with tempfile.TemporaryDirectory() as work_dir: + headless = create_headless_predbat(work_dir, "Europe/London", print) + planner_checks = [ + ("calculate_best_charge", True), + ("calculate_best_export", True), + ("set_charge_window", True), + ("set_export_window", True), + ] + for name, expected in planner_checks: + actual = getattr(headless, name) + if actual != expected: + print(" ERROR: create_headless_predbat left {} as {}, expected {} (the headless apps.yaml has no 'mode' key, so fetch_config_options() defaults to Monitor mode, which sets all four False)".format(name, actual, expected)) + failed = True + print("Test: create_headless_predbat restores PREDBAT_APPS_FILE afterwards, even if it was previously unset") saved_env = os.environ.get("PREDBAT_APPS_FILE") try: diff --git a/apps/predbat/tests/test_annual_scenarios.py b/apps/predbat/tests/test_annual_scenarios.py index ced333275..343c8be67 100644 --- a/apps/predbat/tests/test_annual_scenarios.py +++ b/apps/predbat/tests/test_annual_scenarios.py @@ -53,12 +53,29 @@ def test_annual_scenarios(my_predbat): print(" ERROR: no car energy should produce no window") failed = True - print("Test: add_car_to_load inserts exactly the car's energy and stays cumulative") + print("Test: timer_charge_window rounds the duration up to a 5 minute grid, never under-delivering") + # 81 minutes of raw need at 60kW/hr rate: 81/60*60 = 81.0 kWh, not a multiple of 5 minutes. + # step_data_history() only samples load_forecast every 5 minutes, so an unaligned window + # would be billed at a quantised (and here shorter) length than the car actually needs. + quantised_window = timer_charge_window(rates, car_kwh=81.0, car_rate_kw=60.0) + quantised_length = quantised_window[0]["end"] - quantised_window[0]["start"] + if quantised_length != 85: + print(" ERROR: an 81 minute need should round up to 85 minutes (next multiple of 5), got {}".format(quantised_length)) + failed = True + if quantised_window[0]["start"] % 5 != 0: + print(" ERROR: the window start should be aligned to a 5 minute boundary, got {}".format(quantised_window[0]["start"])) + failed = True + + print("Test: add_car_to_load inserts the car's full energy on EACH day, not split across the plan, and stays cumulative") base = {minute: 0.01 * minute for minute in range(PLAN_MINUTES + 1)} with_car = add_car_to_load(base, window, car_kwh=14.8) - added = with_car[PLAN_MINUTES] - base[PLAN_MINUTES] - if abs(added - 14.8) > 1e-6: - print(" ERROR: expected 14.8 kWh added over the window, got {}".format(added)) + added_total = with_car[PLAN_MINUTES] - base[PLAN_MINUTES] + if abs(added_total - 14.8 * 2) > 1e-6: + print(" ERROR: expected 14.8 kWh added on each of the two days ({} total), got {}".format(14.8 * 2, added_total)) + failed = True + added_day_one = with_car[DAY_MINUTES] - base[DAY_MINUTES] + if abs(added_day_one - 14.8) > 1e-6: + print(" ERROR: the billed first day (minutes 0..{}) should receive the full 14.8 kWh, not a share of it, got {}".format(DAY_MINUTES - 1, added_day_one)) failed = True for minute in range(1, PLAN_MINUTES + 1): if with_car[minute] < with_car[minute - 1] - 1e-12: From b0876cfa44200ac1f4a2128487e823bb1f420bf9 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 12:36:10 +0200 Subject: [PATCH 027/119] feat(annual): add AnnualPredictor orchestration and results document --- apps/predbat/annual.py | 213 +++++++++++++++++- apps/predbat/annual_weather.py | 14 ++ apps/predbat/tests/test_annual_integration.py | 195 ++++++++++++++++ apps/predbat/unit_test.py | 2 + 4 files changed, 421 insertions(+), 3 deletions(-) create mode 100644 apps/predbat/tests/test_annual_integration.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index da59e9f4e..ff538496e 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -14,9 +14,13 @@ import calendar import os -from datetime import date, timedelta +from datetime import date, datetime, timedelta -from annual_load import build_load_forecast +import pytz + +from annual_load import build_load_forecast, OctopusConsumptionLoadProfile, SyntheticLoadProfile +from annual_tariff import AnnualTariff +from annual_weather import AnnualWeather, resolve_postcode from const import MINUTE_WATT, PREDICT_STEP from prediction import Prediction @@ -774,6 +778,16 @@ def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_ """Inject every per-day input into the PredBat instance for one sampled day.""" reset_sample_state(predbat) + # calculate_plan() spins up a multiprocessing Pool sized from args["threads"] (plan.py) + # unless it is exactly 0. The annual tool plans hundreds of individual days per run, each a + # small, fast calculation, so per-day pool creation is both wasted overhead and, on a + # spawn-based multiprocessing start method, unsafe: pool workers read the module-level + # PRED_GLOBAL dict in prediction.py, which is only populated in the parent process. This + # matches create_headless_predbat()'s own choice for a fully offline run; it is forced here + # too so run_day() behaves the same way against a caller-supplied PredBat instance (such as + # the standard unit test fixture) that has not gone through that bootstrap. + predbat.args["threads"] = 0 + predbat.midnight_utc = midnight_utc predbat.now_utc = midnight_utc predbat.minutes_now = 0 @@ -897,7 +911,12 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): predbat.car_charging_slots = [_plan_smart_car(predbat, day, car_kwh)] else: predbat.num_cars = 0 - predbat.car_charging_slots = [] + # A single empty slot list, not an empty outer list: fetch_config_options() always + # sizes car_charging_slots for at least the configured car count (fetch.py), and + # other code (including the standard test fixture's reset_inverter()) indexes + # car_charging_slots[0] unconditionally. num_cars=0 already keeps every car-planning + # loop from iterating it, so this is a shape placeholder rather than a live car. + predbat.car_charging_slots = [[]] apply_hardware(predbat, config["battery"], config["solar"]) predbat.soc_kw = START_SOC_KWH @@ -913,3 +932,191 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): results["with_predbat"] = _billed_result(predbat, DAY_MINUTES, actual_step) return results + + +SCENARIO_KEYS = ["no_pvbat", "without_predbat", "with_predbat"] + +SCENARIO_FIELDS = ["cost_p", "import_kwh", "export_kwh", "pv_generated_kwh", "battery_throughput_kwh"] + + +def average_rate(rates, minutes): + """Return the mean rate across the first ``minutes`` of a rate dict.""" + values = [rates[minute] for minute in range(minutes) if minute in rates] + return (sum(values) / len(values)) if values else 0.0 + + +class AnnualPredictor: + """Projects a year of electricity costs under three scenarios using the Predbat engine.""" + + def __init__(self, config, log=None, storage=None, work_dir="./annual_work"): + """Validate the config and prepare the run.""" + self.log = log or print + self.config = validate_config(config) + self.storage = storage + self.work_dir = work_dir + self.predbat = None + self.weather = None + self.tariff = None + self.load_source = None + self.caveats = [] + + async def _resolve_location(self, weather_fetch): + """Return (latitude, longitude) from the config, resolving a postcode if needed.""" + location = self.config["location"] + if "latitude" in location and "longitude" in location: + return location["latitude"], location["longitude"] + resolved = await resolve_postcode(location["postcode"], weather_fetch, self.log) + if not resolved: + raise AnnualConfigError("annual.location.postcode '{}' could not be resolved; supply latitude and longitude instead".format(location["postcode"])) + return resolved + + async def _build_load_source(self): + """Build the load profile source, falling back to synthetic if Octopus data fails.""" + load_config = self.config["load"] + year = self.config["year"] + + if "octopus" not in load_config: + return SyntheticLoadProfile(annual_kwh=load_config["annual_kwh"], shape=load_config["shape"], year=year) + + # A synthetic profile at the UK average backs the real data so an isolated + # missing day does not silently become zero consumption + fallback = SyntheticLoadProfile(annual_kwh=2700.0, shape="flat", year=year) + source = OctopusConsumptionLoadProfile( + api_key=load_config["octopus"]["api_key"], + account_id=load_config["octopus"]["account_id"], + log=self.log, + storage=self.storage, + fallback=fallback, + ) + if not await source.fetch(year): + raise AnnualConfigError("Octopus consumption data could not be downloaded for {}; check the API key and account id".format(year)) + return source + + def _month_scenarios(self, samples, day_results): + """Weight each sample's daily figures into monthly totals per scenario.""" + totals = {key: {field: 0.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + for (_, weight), result in zip(samples, day_results): + for key in SCENARIO_KEYS: + for field in SCENARIO_FIELDS: + totals[key][field] += result[key][field] * weight + return totals + + async def run(self, progress=None): + """Run the full annual projection and return the results document.""" + year = self.config["year"] + samples_per_month = self.config["samples_per_month"] + has_solar = bool(self.config["solar"]) + + weather_client = AnnualWeather( + self.config["solar"], + latitude=0.0, + longitude=0.0, + log=self.log, + storage=self.storage, + p10_fallback=self.config["pv10_derate_fallback"], + ) + latitude, longitude = await self._resolve_location(weather_client.fetch_json) + weather_client.latitude = latitude + weather_client.longitude = longitude + + self.weather = await weather_client.fetch(year) if has_solar else None + if has_solar and not self.weather.forecast_available: + self.caveats.append("The Open-Meteo forecast archive did not cover {}, so Predbat planned against actuals and P10 used the flat {} derate. Savings are likely overstated.".format(year, self.config["pv10_derate_fallback"])) + elif has_solar and self.weather.fallback_months: + self.caveats.append("Months {} had too few forecast/actual day pairs, so their P10 used the flat {} derate.".format(sorted(self.weather.fallback_months), self.config["pv10_derate_fallback"])) + if has_solar: + self.caveats.append("The forecast-versus-ERA5 gap includes systematic model bias as well as forecast error, so measured solar uncertainty is slightly overstated.") + self.caveats.append("self_consumed_kwh is approximate: when the battery exports grid-charged energy it is understated.") + + self.predbat = create_headless_predbat(self.work_dir, self.config["timezone"], self.log) + self.load_source = await self._build_load_source() + self.tariff = AnnualTariff(self.config["tariff"], log=self.log, predbat=self.predbat, storage=self.storage) + + zone = pytz.timezone(self.config["timezone"]) + months = [] + total_units = 12 + completed = 0 + + for month in range(1, 13): + if progress: + progress(completed, total_units, "Month {:02d}/{}".format(month, year)) + + days_in_month = calendar.monthrange(year, month)[1] + standing_charge_p = self.tariff.standing_charge_p_per_day * days_in_month + + if not await self.tariff.fetch_month(year, month): + months.append({"month": month, "status": "unavailable", "reason": "no rate data available", "days": days_in_month, "standing_charge_p": standing_charge_p}) + completed += 1 + continue + # The 48 hour plan for the last sampled day can spill into the next month + next_year, next_month = (year, month + 1) if month < 12 else (year + 1, 1) + await self.tariff.fetch_month(next_year, next_month) + + samples = select_samples(self.weather, year, month, samples_per_month, has_solar=has_solar) if has_solar else select_samples(None, year, month, samples_per_month, has_solar=False) + if not samples: + months.append({"month": month, "status": "unavailable", "reason": "no usable weather days", "days": days_in_month, "standing_charge_p": standing_charge_p}) + completed += 1 + continue + + day_results = [] + for day, _ in samples: + midnight_utc = zone.localize(datetime(day.year, day.month, day.day)).astimezone(pytz.utc) + day_results.append(run_day(self.predbat, self.config, self.weather, self.tariff, self.load_source, day, midnight_utc)) + + totals = self._month_scenarios(samples, day_results) + first_midnight = zone.localize(datetime(samples[0][0].year, samples[0][0].month, samples[0][0].day)).astimezone(pytz.utc) + _, rate_export = self.tariff.rates_for(first_midnight, DAY_MINUTES) + export_rate = average_rate(rate_export, DAY_MINUTES) + + scenarios = {} + for key in SCENARIO_KEYS: + entry = {field: totals[key][field] for field in SCENARIO_FIELDS} + entry["export_credit_p"] = entry["export_kwh"] * export_rate + entry["self_consumed_kwh"] = max(0.0, entry["pv_generated_kwh"] - entry["export_kwh"]) + scenarios[key] = {name: round(value, 3) for name, value in entry.items()} + + months.append( + { + "month": month, + "status": "ok", + "days": days_in_month, + "sampled_days": [day.isoformat() for day, _ in samples], + "standing_charge_p": round(standing_charge_p, 3), + "scenarios": scenarios, + } + ) + completed += 1 + + if progress: + progress(total_units, total_units, "Complete") + + return self._build_results(months) + + def _build_results(self, months): + """Assemble the final results document from the per-month rows.""" + included = [entry for entry in months if entry["status"] == "ok"] + excluded = [entry["month"] for entry in months if entry["status"] != "ok"] + + annual_scenarios = {} + for key in SCENARIO_KEYS: + annual_scenarios[key] = {field: round(sum(entry["scenarios"][key][field] for entry in included), 3) for field in SCENARIO_FIELDS + ["export_credit_p", "self_consumed_kwh"]} + standing_total = round(sum(entry["standing_charge_p"] for entry in included), 3) + + savings = {} + if annual_scenarios: + savings["pv_battery_vs_none_p"] = round(annual_scenarios["no_pvbat"]["cost_p"] - annual_scenarios["without_predbat"]["cost_p"], 3) + savings["predbat_vs_baseline_p"] = round(annual_scenarios["without_predbat"]["cost_p"] - annual_scenarios["with_predbat"]["cost_p"], 3) + + return { + "year": self.config["year"], + "config": self.config["raw"], + "months": months, + "annual": { + "scenarios": annual_scenarios, + "standing_charge_p": standing_total, + "savings": savings, + "months_included": len(included), + "months_excluded": excluded, + }, + "caveats": self.caveats, + } diff --git a/apps/predbat/annual_weather.py b/apps/predbat/annual_weather.py index 9156ea4b8..1ccaabf37 100644 --- a/apps/predbat/annual_weather.py +++ b/apps/predbat/annual_weather.py @@ -251,3 +251,17 @@ async def fetch(self, year): ratios, fallback_months = self._derive_p10_ratios(actual_periods, forecast_periods, forecast_available) return WeatherYear(actual_periods, forecast_periods, ratios, forecast_available, fallback_months) + + +POSTCODE_URL = "https://api.postcodes.io/postcodes/{}" + + +async def resolve_postcode(postcode, fetch_json, log): + """Resolve a UK postcode to (latitude, longitude), or None when it cannot be resolved.""" + data = await fetch_json(POSTCODE_URL.format(postcode)) + result = (data or {}).get("result", {}) if isinstance(data, dict) else {} + if "latitude" in result and "longitude" in result: + log("Annual: postcode {} resolved to latitude {} longitude {}".format(postcode, result["latitude"], result["longitude"])) + return result["latitude"], result["longitude"] + log("Warn: Annual: postcode {} could not be resolved".format(postcode)) + return None diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py new file mode 100644 index 000000000..919b8c956 --- /dev/null +++ b/apps/predbat/tests/test_annual_integration.py @@ -0,0 +1,195 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Integration tests for the annual prediction day runner. + +These run real Predbat plans, so they are registered as slow. +""" + +from datetime import date, datetime + +import pytz + +from annual import DAY_MINUTES, run_day, validate_config +from annual_load import SyntheticLoadProfile +from tests.test_infra import reset_inverter + +SOLAR_CURVE = [0, 0, 0, 0, 0, 0, 0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.3, 0.1, 0, 0, 0, 0, 0] + + +class StubWeather: + """A WeatherYear stand-in with a fixed daily solar curve and a forecast multiplier.""" + + def __init__(self, peak_kw, forecast_multiplier=1.0, p10_ratio_value=0.8): + """Configure the actual peak power and how much the forecast overstates it.""" + self.peak_kw = peak_kw + self.forecast_multiplier = forecast_multiplier + self.p10_ratio_value = p10_ratio_value + + def _series(self, midnight_utc, minutes, scale): + """Build a per-minute kWh series from the fixed daily curve.""" + result = {} + for minute in range(minutes): + hour = (minute // 60) % 24 + result[minute] = (self.peak_kw * SOLAR_CURVE[hour] * scale) / 60.0 + return result + + def pv_minutes(self, series, midnight_utc, minutes): + """Return the actual or forecast per-minute series.""" + return self._series(midnight_utc, minutes, 1.0 if series == "actual" else self.forecast_multiplier) + + def pv_minutes_p10(self, midnight_utc, minutes, month): + """Return the P10 series, the forecast scaled by the month ratio.""" + return self._series(midnight_utc, minutes, self.forecast_multiplier * self.p10_ratio_value) + + def has_actual(self, day): + """Every day has data in this stub.""" + return True + + def daily_actual_kwh(self, day): + """Return the fixed daily total.""" + return sum(self._series(None, DAY_MINUTES, 1.0).values()) + + def p10_ratio(self, month): + """Return the fixed P10 ratio.""" + return self.p10_ratio_value + + +class StubTariff: + """A tariff stand-in with a cheap overnight band and an expensive evening peak.""" + + def __init__(self, cheap=7.0, normal=28.0, peak=45.0, export=15.0): + """Configure the four rate levels.""" + self.cheap = cheap + self.normal = normal + self.peak = peak + self.export = export + self.standing_charge_p_per_day = 50.0 + + def rates_for(self, midnight_utc, minutes): + """Return (import, export) rate dicts keyed by absolute minute.""" + rate_import = {} + rate_export = {} + for minute in range(minutes): + in_day = minute % DAY_MINUTES + if 30 <= in_day < 330: + rate_import[minute] = self.cheap + elif 16 * 60 <= in_day < 19 * 60: + rate_import[minute] = self.peak + else: + rate_import[minute] = self.normal + rate_export[minute] = self.export + return rate_import, rate_export + + def month_available(self, year, month): + """Always available.""" + return True + + +def make_config(with_car=False): + """Return a validated annual config for the integration tests.""" + raw = { + "location": {"latitude": 51.5, "longitude": -0.1}, + "solar": [{"kwp": 5.0}], + "battery": {"size_kwh": 10.0, "inverter_kw": 5.0}, + "load": {"annual_kwh": 3800, "shape": "flat"}, + "tariff": {"rates_import": [{"rate": 28.0}]}, + "year": 2025, + } + if with_car: + raw["load"]["car_charging_kwh"] = 2500 + return validate_config(raw, today=date(2026, 7, 25)) + + +def run_one(my_predbat, config, weather, day): + """Run all three scenarios for one day and return the results dict.""" + reset_inverter(my_predbat) + midnight = pytz.utc.localize(datetime(day.year, day.month, day.day)) + load_source = SyntheticLoadProfile(annual_kwh=config["load"]["annual_kwh"], shape=config["load"]["shape"], year=config["year"]) + return run_day(my_predbat, config, weather, StubTariff(), load_source, day, midnight) + + +def test_annual_integration(my_predbat): + """Verify scenario ordering, the forecast/actuals split, and state isolation.""" + failed = False + print("**** Testing annual integration ****") + + config = make_config() + weather = StubWeather(peak_kw=4.0) + day = date(2025, 5, 15) + + print("Test: the three scenarios run and produce the expected keys") + results = run_one(my_predbat, config, weather, day) + for key in ["no_pvbat", "without_predbat", "with_predbat"]: + if key not in results: + print(" ERROR: missing scenario '{}'".format(key)) + failed = True + continue + for field in ["cost_p", "import_kwh", "export_kwh", "pv_generated_kwh", "battery_throughput_kwh"]: + if field not in results[key]: + print(" ERROR: scenario '{}' is missing field '{}'".format(key, field)) + failed = True + + print("Test: predbat_cost <= without_predbat_cost <= no_pvbat_cost") + if not failed: + predbat_cost = results["with_predbat"]["cost_p"] + baseline_cost = results["without_predbat"]["cost_p"] + none_cost = results["no_pvbat"]["cost_p"] + if not predbat_cost <= baseline_cost + 1e-6: + print(" ERROR: Predbat cost {} should not exceed the dumb baseline {}".format(predbat_cost, baseline_cost)) + failed = True + if not baseline_cost <= none_cost + 1e-6: + print(" ERROR: the PV/battery baseline {} should not exceed the no-system cost {}".format(baseline_cost, none_cost)) + failed = True + + print("Test: the no-PV/no-battery scenario generates nothing and stores nothing") + if results["no_pvbat"]["pv_generated_kwh"] != 0.0: + print(" ERROR: the no-system scenario should generate no PV, got {}".format(results["no_pvbat"]["pv_generated_kwh"])) + failed = True + if results["no_pvbat"]["battery_throughput_kwh"] != 0.0: + print(" ERROR: the no-system scenario should cycle no battery, got {}".format(results["no_pvbat"]["battery_throughput_kwh"])) + failed = True + + print("Test: Predbat is billed on actuals, not on the forecast it planned against") + # The forecast claims three times the real generation. If the Prediction swap were + # skipped, pv_generated_kwh and the cost would follow the inflated forecast. + inflated = StubWeather(peak_kw=4.0, forecast_multiplier=3.0) + inflated_results = run_one(my_predbat, config, inflated, day) + honest_pv = results["with_predbat"]["pv_generated_kwh"] + inflated_pv = inflated_results["with_predbat"]["pv_generated_kwh"] + if abs(inflated_pv - honest_pv) > 0.01: + print(" ERROR: reported PV should track actuals ({}) regardless of the forecast, got {}".format(honest_pv, inflated_pv)) + failed = True + if inflated_results["with_predbat"]["cost_p"] < results["with_predbat"]["cost_p"] - 1e-6: + print(" ERROR: planning against an over-optimistic forecast must not make the billed cost cheaper") + failed = True + + print("Test: state isolation - a day run in isolation matches the same day run after another") + isolated = run_one(my_predbat, config, weather, day) + _ = run_one(my_predbat, config, StubWeather(peak_kw=1.0), date(2025, 11, 20)) + after_other = run_one(my_predbat, config, weather, day) + for scenario in ["no_pvbat", "without_predbat", "with_predbat"]: + for field in ["cost_p", "import_kwh", "export_kwh", "battery_throughput_kwh"]: + first = isolated[scenario][field] + second = after_other[scenario][field] + if abs(first - second) > 1e-6: + print(" ERROR: {}.{} changed from {} to {} depending on what ran before it".format(scenario, field, first, second)) + failed = True + + print("Test: a car charging config still produces an ordered result") + car_config = make_config(with_car=True) + car_results = run_one(my_predbat, car_config, weather, day) + if car_results["with_predbat"]["cost_p"] > car_results["without_predbat"]["cost_p"] + 1e-6: + print(" ERROR: with a car, Predbat cost {} should not exceed the timer baseline {}".format(car_results["with_predbat"]["cost_p"], car_results["without_predbat"]["cost_p"])) + failed = True + if car_results["no_pvbat"]["import_kwh"] <= results["no_pvbat"]["import_kwh"]: + print(" ERROR: adding a car should raise the no-system import, got {} vs {}".format(car_results["no_pvbat"]["import_kwh"], results["no_pvbat"]["import_kwh"])) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 059439442..29740533e 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -167,6 +167,7 @@ from tests.test_annual_bootstrap import test_annual_bootstrap from tests.test_annual_sampling import test_annual_sampling from tests.test_annual_scenarios import test_annual_scenarios +from tests.test_annual_integration import test_annual_integration # Mock the components and plugin system @@ -412,6 +413,7 @@ def main(): ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), ("annual_sampling", test_annual_sampling, "Annual prediction sample selection tests", False), ("annual_scenarios", test_annual_scenarios, "Annual prediction scenario helper tests", False), + ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), ] # Parse command line arguments From 45eeb1b8fbdd5b541d23c6fb3e3db97f645ec9af Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 12:59:13 +0200 Subject: [PATCH 028/119] fix(annual): correct results zeroing, add results tests, and 6 review fixes Fixes 9 issues found in review of the AnnualPredictor orchestrator: - Critical: an all-unavailable year fabricated a zero-cost, zero-saving result instead of reporting no result (dead truthiness guard). - Critical: run()/_build_results()/_month_scenarios()/average_rate() had no tests; added a fast, non-slow test_annual_results covering all four. - One failing sample no longer aborts the whole run (logged, sample dropped, month marked degraded or unavailable as appropriate). - The forecast/actuals integration test now asserts the with-Predbat cost is materially worse under an inflated forecast, not merely "not cheaper"; its misleading comment about pv_generated_kwh is corrected. - export_credit_p renamed to export_credit_p_estimate (not additive with cost_p); self_consumed_kwh's clamp now carries a self_consumed_kwh_meaningful flag plus a caveat when it fires. - A failed next-month tariff fetch is now logged and caveated instead of discarded; a docstring corrected to match its code; a dead ternary collapsed. --- apps/predbat/annual.py | 96 +++++++--- apps/predbat/tests/test_annual_integration.py | 20 ++- apps/predbat/tests/test_annual_results.py | 168 ++++++++++++++++++ apps/predbat/unit_test.py | 2 + 4 files changed, 261 insertions(+), 25 deletions(-) create mode 100644 apps/predbat/tests/test_annual_results.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index ff538496e..2eb30c56b 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -959,6 +959,9 @@ def __init__(self, config, log=None, storage=None, work_dir="./annual_work"): self.tariff = None self.load_source = None self.caveats = [] + # month -> [scenario keys] where export exceeded generation, so self_consumed_kwh was + # clamped to zero rather than being genuinely negative (see run()'s post-loop caveat) + self.grid_arbitrage_scenarios = {} async def _resolve_location(self, weather_fetch): """Return (latitude, longitude) from the config, resolving a postcode if needed.""" @@ -971,7 +974,12 @@ async def _resolve_location(self, weather_fetch): return resolved async def _build_load_source(self): - """Build the load profile source, falling back to synthetic if Octopus data fails.""" + """Build the load profile source: synthetic, or Octopus consumption data. + + The synthetic fallback is only used to backfill an isolated missing day within + an otherwise successful Octopus download; a download that fails outright raises + ``AnnualConfigError`` rather than silently substituting the synthetic profile. + """ load_config = self.config["load"] year = self.config["year"] @@ -1050,62 +1058,106 @@ async def run(self, progress=None): continue # The 48 hour plan for the last sampled day can spill into the next month next_year, next_month = (year, month + 1) if month < 12 else (year + 1, 1) - await self.tariff.fetch_month(next_year, next_month) + if not await self.tariff.fetch_month(next_year, next_month): + spill_message = "Rate data for {}-{:02d} could not be downloaded, so any plan hours for month {} spilling into it may be costed as free.".format(next_year, next_month, month) + self.log("Warn: Annual: {}".format(spill_message)) + if spill_message not in self.caveats: + self.caveats.append(spill_message) - samples = select_samples(self.weather, year, month, samples_per_month, has_solar=has_solar) if has_solar else select_samples(None, year, month, samples_per_month, has_solar=False) + samples = select_samples(self.weather, year, month, samples_per_month, has_solar=has_solar) if not samples: months.append({"month": month, "status": "unavailable", "reason": "no usable weather days", "days": days_in_month, "standing_charge_p": standing_charge_p}) completed += 1 continue + surviving_samples = [] day_results = [] - for day, _ in samples: + failed_days = [] + for day, weight in samples: midnight_utc = zone.localize(datetime(day.year, day.month, day.day)).astimezone(pytz.utc) - day_results.append(run_day(self.predbat, self.config, self.weather, self.tariff, self.load_source, day, midnight_utc)) + try: + result = run_day(self.predbat, self.config, self.weather, self.tariff, self.load_source, day, midnight_utc) + except Exception as exc: # noqa: BLE001 - one bad sample must not abort the whole year + self.log("Warn: Annual: {} in month {} failed to plan/cost ({}: {}); excluding it from this month's total".format(day.isoformat(), month, type(exc).__name__, exc)) + failed_days.append(day.isoformat()) + continue + surviving_samples.append((day, weight)) + day_results.append(result) + + if not day_results: + months.append({"month": month, "status": "unavailable", "reason": "every sampled day failed to plan", "days": days_in_month, "standing_charge_p": standing_charge_p, "failed_days": failed_days}) + completed += 1 + continue - totals = self._month_scenarios(samples, day_results) - first_midnight = zone.localize(datetime(samples[0][0].year, samples[0][0].month, samples[0][0].day)).astimezone(pytz.utc) + totals = self._month_scenarios(surviving_samples, day_results) + first_midnight = zone.localize(datetime(surviving_samples[0][0].year, surviving_samples[0][0].month, surviving_samples[0][0].day)).astimezone(pytz.utc) _, rate_export = self.tariff.rates_for(first_midnight, DAY_MINUTES) export_rate = average_rate(rate_export, DAY_MINUTES) scenarios = {} for key in SCENARIO_KEYS: entry = {field: totals[key][field] for field in SCENARIO_FIELDS} - entry["export_credit_p"] = entry["export_kwh"] * export_rate - entry["self_consumed_kwh"] = max(0.0, entry["pv_generated_kwh"] - entry["export_kwh"]) - scenarios[key] = {name: round(value, 3) for name, value in entry.items()} + entry["export_credit_p_estimate"] = entry["export_kwh"] * export_rate + self_consumed = entry["pv_generated_kwh"] - entry["export_kwh"] + entry["self_consumed_kwh"] = max(0.0, self_consumed) + meaningful = self_consumed >= 0.0 + rounded = {name: round(value, 3) for name, value in entry.items()} + rounded["self_consumed_kwh_meaningful"] = meaningful + scenarios[key] = rounded + if not meaningful: + self.grid_arbitrage_scenarios.setdefault(month, []).append(key) months.append( { "month": month, - "status": "ok", + "status": "ok" if not failed_days else "degraded", "days": days_in_month, - "sampled_days": [day.isoformat() for day, _ in samples], + "sampled_days": [day.isoformat() for day, _ in surviving_samples], + "failed_days": failed_days, "standing_charge_p": round(standing_charge_p, 3), "scenarios": scenarios, } ) completed += 1 + if self.grid_arbitrage_scenarios: + arbitrage_count = sum(len(keys) for keys in self.grid_arbitrage_scenarios.values()) + first_month = sorted(self.grid_arbitrage_scenarios)[0] + message = "self_consumed_kwh was clamped to zero rather than negative for {} scenario/month row(s) (from month {}): export exceeded generation there. See self_consumed_kwh_meaningful.".format(arbitrage_count, first_month) + self.caveats.append(message) + if progress: progress(total_units, total_units, "Complete") return self._build_results(months) def _build_results(self, months): - """Assemble the final results document from the per-month rows.""" - included = [entry for entry in months if entry["status"] == "ok"] - excluded = [entry["month"] for entry in months if entry["status"] != "ok"] - - annual_scenarios = {} - for key in SCENARIO_KEYS: - annual_scenarios[key] = {field: round(sum(entry["scenarios"][key][field] for entry in included), 3) for field in SCENARIO_FIELDS + ["export_credit_p", "self_consumed_kwh"]} - standing_total = round(sum(entry["standing_charge_p"] for entry in included), 3) - + """Assemble the final results document from the per-month rows. + + A month that is entirely ``"unavailable"`` (no rate data, no usable weather days, or + every sampled day failed) contributes nothing. An ``"ok"`` or ``"degraded"`` month + (some, but not all, of its sampled days failed - see ``run()``) still carries real + figures and is included. When no month is included at all, ``annual.scenarios`` and + ``annual.standing_charge_p`` are ``None`` and ``annual.savings`` is empty rather than + reporting a fabricated zero-cost, zero-saving year. + """ + included = [entry for entry in months if entry["status"] in ("ok", "degraded")] + excluded = [entry["month"] for entry in months if entry["status"] not in ("ok", "degraded")] + + annual_scenarios = None + standing_total = None savings = {} - if annual_scenarios: + if included: + annual_scenarios = {} + for key in SCENARIO_KEYS: + annual_scenarios[key] = {field: round(sum(entry["scenarios"][key][field] for entry in included), 3) for field in SCENARIO_FIELDS + ["export_credit_p_estimate", "self_consumed_kwh"]} + standing_total = round(sum(entry["standing_charge_p"] for entry in included), 3) savings["pv_battery_vs_none_p"] = round(annual_scenarios["no_pvbat"]["cost_p"] - annual_scenarios["without_predbat"]["cost_p"], 3) savings["predbat_vs_baseline_p"] = round(annual_scenarios["without_predbat"]["cost_p"] - annual_scenarios["with_predbat"]["cost_p"], 3) + else: + no_result_message = "No month produced a usable result, so no annual totals or savings could be calculated." + if no_result_message not in self.caveats: + self.caveats.append(no_result_message) return { "year": self.config["year"], diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py index 919b8c956..72b40d7ab 100644 --- a/apps/predbat/tests/test_annual_integration.py +++ b/apps/predbat/tests/test_annual_integration.py @@ -157,8 +157,16 @@ def test_annual_integration(my_predbat): failed = True print("Test: Predbat is billed on actuals, not on the forecast it planned against") - # The forecast claims three times the real generation. If the Prediction swap were - # skipped, pv_generated_kwh and the cost would follow the inflated forecast. + # pv_generated_kwh is derived from the ACTUAL pv_step regardless of scenario + # (_billed_result's pv_step argument in annual.py always comes from actual_step), so it is + # identical between the honest and inflated runs by construction and does NOT itself + # exercise the Prediction swap - the assertion below on it is a sanity check, not proof. + # What does exercise the swap is cost: with the swap in place, calculate_plan() commits to + # the inflated forecast's phantom solar, then run_prediction() bills against the actuals + # that never delivered it, so the inflated run's billed cost should be MATERIALLY worse, + # not merely "not cheaper". Observed on this fixture: honest with_predbat cost_p is + # -471.6792p, 3x-inflated-forecast cost_p is -461.2486p, a ~10.43p degradation; + # material_degradation_p is set below that with margin so the assertion is not a knife edge. inflated = StubWeather(peak_kw=4.0, forecast_multiplier=3.0) inflated_results = run_one(my_predbat, config, inflated, day) honest_pv = results["with_predbat"]["pv_generated_kwh"] @@ -166,9 +174,15 @@ def test_annual_integration(my_predbat): if abs(inflated_pv - honest_pv) > 0.01: print(" ERROR: reported PV should track actuals ({}) regardless of the forecast, got {}".format(honest_pv, inflated_pv)) failed = True - if inflated_results["with_predbat"]["cost_p"] < results["with_predbat"]["cost_p"] - 1e-6: + honest_cost = results["with_predbat"]["cost_p"] + inflated_cost = inflated_results["with_predbat"]["cost_p"] + material_degradation_p = 3.0 + if inflated_cost < honest_cost - 1e-6: print(" ERROR: planning against an over-optimistic forecast must not make the billed cost cheaper") failed = True + if inflated_cost < honest_cost + material_degradation_p: + print(" ERROR: an over-optimistic forecast should cost at least {}p more (over-commits to solar that never arrives); got honest {} vs inflated {}".format(material_degradation_p, honest_cost, inflated_cost)) + failed = True print("Test: state isolation - a day run in isolation matches the same day run after another") isolated = run_one(my_predbat, config, weather, day) diff --git a/apps/predbat/tests/test_annual_results.py b/apps/predbat/tests/test_annual_results.py new file mode 100644 index 000000000..a94991ae6 --- /dev/null +++ b/apps/predbat/tests/test_annual_results.py @@ -0,0 +1,168 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Unit tests for assembling the annual prediction results document. + +Covers ``AnnualPredictor._build_results``, ``AnnualPredictor._month_scenarios`` and +``average_rate`` against hand-built month rows and samples. None of this touches the +network, a Predbat instance or a real plan run, so it is fast and registered as a +non-slow test - unlike ``test_annual_integration``, which needs a real day run to +exercise this same code from the top. +""" + +from datetime import date + +from annual import SCENARIO_FIELDS, SCENARIO_KEYS, AnnualPredictor, average_rate + + +def make_predictor(): + """Return an ``AnnualPredictor`` built from a minimal valid config. + + ``AnnualPredictor.__init__`` validates the config itself and touches no network or + Predbat state, so this is safe to call directly in a unit test. + """ + raw = { + "location": {"latitude": 51.5, "longitude": -0.1}, + "solar": [{"kwp": 5.0}], + "battery": {"size_kwh": 10.0, "inverter_kw": 5.0}, + "load": {"annual_kwh": 3800, "shape": "flat"}, + "tariff": {"rates_import": [{"rate": 28.0}]}, + "year": 2025, + } + return AnnualPredictor(raw) + + +def make_month_row(month, costs, status="ok"): + """Build a month row with the given status and per-scenario cost_p values. + + Every other scenario field is fixed at 1.0 (or 0.5 for the two derived fields) so + the totals in each test are easy to predict by hand. + """ + scenarios = {} + for key in SCENARIO_KEYS: + entry = {field: 1.0 for field in SCENARIO_FIELDS} + entry["cost_p"] = costs[key] + entry["export_credit_p_estimate"] = 0.5 + entry["self_consumed_kwh"] = 0.5 + entry["self_consumed_kwh_meaningful"] = True + scenarios[key] = entry + row = {"month": month, "status": status, "days": 30, "standing_charge_p": 100.0, "scenarios": scenarios} + if status == "degraded": + row["failed_days"] = ["2025-{:02d}-15".format(month)] + return row + + +def make_unavailable_row(month, reason="no rate data available"): + """Build an 'unavailable' month row, as run() emits when a month has no usable result.""" + return {"month": month, "status": "unavailable", "reason": reason, "days": 30, "standing_charge_p": 15.0} + + +def test_annual_results(my_predbat): + """Verify _build_results, _month_scenarios and average_rate against hand-built inputs.""" + failed = False + print("**** Testing annual results assembly ****") + + print("Test: _build_results with every month ok sums totals across all months") + predictor = make_predictor() + months = [ + make_month_row(1, {"no_pvbat": 100.0, "without_predbat": 50.0, "with_predbat": 20.0}), + make_month_row(2, {"no_pvbat": 200.0, "without_predbat": 80.0, "with_predbat": 30.0}), + ] + result = predictor._build_results(months) + if result["annual"]["scenarios"]["no_pvbat"]["cost_p"] != 300.0: + print(" ERROR: expected no_pvbat annual cost_p 300.0, got {}".format(result["annual"]["scenarios"]["no_pvbat"]["cost_p"])) + failed = True + if result["annual"]["scenarios"]["with_predbat"]["cost_p"] != 50.0: + print(" ERROR: expected with_predbat annual cost_p 50.0, got {}".format(result["annual"]["scenarios"]["with_predbat"]["cost_p"])) + failed = True + if result["annual"]["months_included"] != 2 or result["annual"]["months_excluded"] != []: + print(" ERROR: expected 2 months included and none excluded, got {} / {}".format(result["annual"]["months_included"], result["annual"]["months_excluded"])) + failed = True + expected_saving = 300.0 - 130.0 + if result["annual"]["savings"]["pv_battery_vs_none_p"] != expected_saving: + print(" ERROR: expected pv_battery_vs_none_p {}, got {}".format(expected_saving, result["annual"]["savings"]["pv_battery_vs_none_p"])) + failed = True + + print("Test: _build_results with a mix of ok and unavailable months excludes the unavailable one") + predictor = make_predictor() + months = [ + make_month_row(1, {"no_pvbat": 100.0, "without_predbat": 50.0, "with_predbat": 20.0}), + make_unavailable_row(2), + make_month_row(3, {"no_pvbat": 40.0, "without_predbat": 10.0, "with_predbat": 5.0}), + ] + result = predictor._build_results(months) + if result["annual"]["scenarios"]["no_pvbat"]["cost_p"] != 140.0: + print(" ERROR: expected no_pvbat annual cost_p 140.0 (month 2 excluded), got {}".format(result["annual"]["scenarios"]["no_pvbat"]["cost_p"])) + failed = True + if result["annual"]["months_included"] != 2 or result["annual"]["months_excluded"] != [2]: + print(" ERROR: expected 2 months included and month 2 excluded, got {} / {}".format(result["annual"]["months_included"], result["annual"]["months_excluded"])) + failed = True + + print("Test: _build_results includes a 'degraded' month (partial-sample) in totals, not excluded") + predictor = make_predictor() + months = [make_month_row(4, {"no_pvbat": 60.0, "without_predbat": 25.0, "with_predbat": 10.0}, status="degraded")] + result = predictor._build_results(months) + if result["annual"]["months_included"] != 1 or result["annual"]["months_excluded"] != []: + print(" ERROR: expected the degraded month to be included, not excluded, got included={} excluded={}".format(result["annual"]["months_included"], result["annual"]["months_excluded"])) + failed = True + if result["annual"]["scenarios"]["no_pvbat"]["cost_p"] != 60.0: + print(" ERROR: expected the degraded month's totals to be counted, got {}".format(result["annual"]["scenarios"]["no_pvbat"]["cost_p"])) + failed = True + + print("Test: _build_results with every month unavailable does not fabricate a zero-cost year") + predictor = make_predictor() + months = [make_unavailable_row(1), make_unavailable_row(2, reason="no usable weather days")] + result = predictor._build_results(months) + if result["annual"]["scenarios"] is not None: + print(" ERROR: expected annual scenarios to be None when nothing is included, got {}".format(result["annual"]["scenarios"])) + failed = True + if result["annual"]["standing_charge_p"] is not None: + print(" ERROR: expected annual standing_charge_p to be None when nothing is included, got {}".format(result["annual"]["standing_charge_p"])) + failed = True + if result["annual"]["savings"] != {}: + print(" ERROR: expected empty savings when nothing is included, got {}".format(result["annual"]["savings"])) + failed = True + if result["annual"]["months_included"] != 0 or result["annual"]["months_excluded"] != [1, 2]: + print(" ERROR: expected 0 months included and both excluded, got {} / {}".format(result["annual"]["months_included"], result["annual"]["months_excluded"])) + failed = True + if not any("no month produced a usable result" in caveat.lower() for caveat in result["caveats"]): + print(" ERROR: expected a caveat explaining that no month produced a usable result, got {}".format(result["caveats"])) + failed = True + + print("Test: _month_scenarios weights uneven samples correctly") + predictor = make_predictor() + samples = [(date(2025, 1, 5), 3), (date(2025, 1, 20), 5)] + day_results = [ + {key: {field: (10.0 if key == "no_pvbat" else 2.0) for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS}, + {key: {field: (2.0 if key == "no_pvbat" else 1.0) for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS}, + ] + totals = predictor._month_scenarios(samples, day_results) + expected_no_pvbat = 10.0 * 3 + 2.0 * 5 + if totals["no_pvbat"]["cost_p"] != expected_no_pvbat: + print(" ERROR: expected weighted no_pvbat cost_p {}, got {}".format(expected_no_pvbat, totals["no_pvbat"]["cost_p"])) + failed = True + expected_with_predbat = 2.0 * 3 + 1.0 * 5 + if totals["with_predbat"]["import_kwh"] != expected_with_predbat: + print(" ERROR: expected weighted with_predbat import_kwh {}, got {}".format(expected_with_predbat, totals["with_predbat"]["import_kwh"])) + failed = True + + print("Test: average_rate") + full_rates = {0: 10.0, 1: 20.0, 2: 30.0} + if average_rate(full_rates, 3) != 20.0: + print(" ERROR: expected average_rate of a full dict to be 20.0, got {}".format(average_rate(full_rates, 3))) + failed = True + partial_rates = {0: 10.0} + if average_rate(partial_rates, 3) != 10.0: + print(" ERROR: expected average_rate to ignore minutes missing from the dict, got {}".format(average_rate(partial_rates, 3))) + failed = True + if average_rate({}, 10) != 0.0: + print(" ERROR: expected average_rate of an empty dict to be 0.0 rather than raising, got {}".format(average_rate({}, 10))) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 29740533e..c9c387159 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -167,6 +167,7 @@ from tests.test_annual_bootstrap import test_annual_bootstrap from tests.test_annual_sampling import test_annual_sampling from tests.test_annual_scenarios import test_annual_scenarios +from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration # Mock the components and plugin system @@ -413,6 +414,7 @@ def main(): ("annual_bootstrap", test_annual_bootstrap, "Annual prediction bootstrap and state reset tests", False), ("annual_sampling", test_annual_sampling, "Annual prediction sample selection tests", False), ("annual_scenarios", test_annual_scenarios, "Annual prediction scenario helper tests", False), + ("annual_results", test_annual_results, "Annual prediction results assembly tests", False), ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), ] From 136bd06fa10a05c86a3ff8c609447bfe0e755307 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 16:30:55 +0200 Subject: [PATCH 029/119] fix(annual): reweight degraded-month samples, document export estimate, fix flaky swap test - Reweight surviving samples in a degraded month (days_in_month / len(survivors)) so a partially-failed month still represents a full month in the annual total, instead of silently under-counting by the dropped samples' share. - Document export_credit_p_estimate at its assignment site and in a results-document caveat as an approximation already reflected inside cost_p, so a consumer does not double-count it. - Replace the annual_integration forecast/actuals cost-threshold assertion (which measured plan-search noise and could flip sign depending on suite order) with a structural check that predbat.prediction.pv_forecast_minute_step holds the actuals, not the inflated forecast, after run_day() returns. - Carry forward the test_annual_bootstrap fix that stopped debug_enable leaking across the shared suite fixture and forcing every later prediction off the C++ kernel. --- .cspell/custom-dictionary-workspace.txt | 2 + apps/predbat/annual.py | 28 +++++++++++ apps/predbat/tests/test_annual_bootstrap.py | 30 ++++++++---- apps/predbat/tests/test_annual_integration.py | 48 +++++++++++-------- apps/predbat/tests/test_annual_results.py | 18 +++++++ 5 files changed, 97 insertions(+), 29 deletions(-) diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index 852122512..ad8ddbf49 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -377,6 +377,8 @@ remotecontrol resetmidnight resultid resultmid +reweight +reweighted rname Roboto rowspan diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 2eb30c56b..dc5e58f34 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -1000,6 +1000,24 @@ async def _build_load_source(self): raise AnnualConfigError("Octopus consumption data could not be downloaded for {}; check the API key and account id".format(year)) return source + def _reweight_survivors(self, surviving_samples, days_in_month): + """Rescale surviving samples so they still represent a full month between them. + + ``select_samples()`` gives every chosen sample an equal weight of + ``days_in_month / len(chosen)``, on the assumption all of them get planned + successfully. When one or more are dropped after a ``run_day()`` failure, keeping + their original weight would under-represent the month by the dropped days' share - + a month that lost half its samples would silently contribute about half its true + cost to the annual total, even though ``standing_charge_p`` and ``days`` still + reflect the full month. Recomputing the weight from the surviving count alone + keeps the total weight equal to ``days_in_month``; the caller still records + ``"degraded"``/``failed_days`` so the reduced sample count remains visible. + """ + if not surviving_samples: + return surviving_samples + reweighted = days_in_month / float(len(surviving_samples)) + return [(day, reweighted) for day, _ in surviving_samples] + def _month_scenarios(self, samples, day_results): """Weight each sample's daily figures into monthly totals per scenario.""" totals = {key: {field: 0.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} @@ -1035,6 +1053,7 @@ async def run(self, progress=None): if has_solar: self.caveats.append("The forecast-versus-ERA5 gap includes systematic model bias as well as forecast error, so measured solar uncertainty is slightly overstated.") self.caveats.append("self_consumed_kwh is approximate: when the battery exports grid-charged energy it is understated.") + self.caveats.append("export_credit_p_estimate is money ALREADY included inside cost_p (which prices every export minute at its real rate); it is informational only - adding it to cost_p double-counts export income.") self.predbat = create_headless_predbat(self.work_dir, self.config["timezone"], self.log) self.load_source = await self._build_load_source() @@ -1089,6 +1108,9 @@ async def run(self, progress=None): completed += 1 continue + if failed_days: + surviving_samples = self._reweight_survivors(surviving_samples, days_in_month) + totals = self._month_scenarios(surviving_samples, day_results) first_midnight = zone.localize(datetime(surviving_samples[0][0].year, surviving_samples[0][0].month, surviving_samples[0][0].day)).astimezone(pytz.utc) _, rate_export = self.tariff.rates_for(first_midnight, DAY_MINUTES) @@ -1097,6 +1119,12 @@ async def run(self, progress=None): scenarios = {} for key in SCENARIO_KEYS: entry = {field: totals[key][field] for field in SCENARIO_FIELDS} + # An approximation, not a second income stream: cost_p already prices export at + # the real per-minute export rate for every minute it happened, so the export + # credit is already inside it. This is a cruder second estimate of the same + # money (a single day's flat average export rate), kept only for a human- + # readable "how much of that came from export" figure. Adding it to cost_p + # double-counts the export income - see the results-document caveat below. entry["export_credit_p_estimate"] = entry["export_kwh"] * export_rate self_consumed = entry["pv_generated_kwh"] - entry["export_kwh"] entry["self_consumed_kwh"] = max(0.0, self_consumed) diff --git a/apps/predbat/tests/test_annual_bootstrap.py b/apps/predbat/tests/test_annual_bootstrap.py index d3946abde..a3521f14d 100644 --- a/apps/predbat/tests/test_annual_bootstrap.py +++ b/apps/predbat/tests/test_annual_bootstrap.py @@ -242,15 +242,27 @@ def test_annual_bootstrap(my_predbat): failed = True print("Test: reset_sample_state must not re-apply configure_offline_mode's choices") - my_predbat.octopus_intelligent_charging = True - my_predbat.debug_enable = True - reset_sample_state(my_predbat) - if my_predbat.octopus_intelligent_charging is not True: - print(" ERROR: reset_sample_state should not touch octopus_intelligent_charging any more") - failed = True - if my_predbat.debug_enable is not True: - print(" ERROR: reset_sample_state should not touch debug_enable any more") - failed = True + # my_predbat is the shared suite fixture, so anything set here outlives this test. + # debug_enable in particular must be put back: kernel_supported() in + # prediction_kernel.py requires `not pred.debug_enable`, so leaving it True makes every + # later prediction in the whole suite bypass the C++ kernel and fall back to the Python + # engine - roughly 8x slower per plan, which looks like a hang in the slow tests rather + # than a leak from here. + previous_octopus_intelligent = my_predbat.octopus_intelligent_charging + previous_debug_enable = my_predbat.debug_enable + try: + my_predbat.octopus_intelligent_charging = True + my_predbat.debug_enable = True + reset_sample_state(my_predbat) + if my_predbat.octopus_intelligent_charging is not True: + print(" ERROR: reset_sample_state should not touch octopus_intelligent_charging any more") + failed = True + if my_predbat.debug_enable is not True: + print(" ERROR: reset_sample_state should not touch debug_enable any more") + failed = True + finally: + my_predbat.octopus_intelligent_charging = previous_octopus_intelligent + my_predbat.debug_enable = previous_debug_enable print("Test: create_headless_predbat leaves the planner switched on, not the Monitor-mode default") with tempfile.TemporaryDirectory() as work_dir: diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py index 72b40d7ab..8c7d5ba06 100644 --- a/apps/predbat/tests/test_annual_integration.py +++ b/apps/predbat/tests/test_annual_integration.py @@ -16,7 +16,7 @@ import pytz -from annual import DAY_MINUTES, run_day, validate_config +from annual import DAY_MINUTES, PLAN_MINUTES, run_day, validate_config from annual_load import SyntheticLoadProfile from tests.test_infra import reset_inverter @@ -157,32 +157,40 @@ def test_annual_integration(my_predbat): failed = True print("Test: Predbat is billed on actuals, not on the forecast it planned against") - # pv_generated_kwh is derived from the ACTUAL pv_step regardless of scenario - # (_billed_result's pv_step argument in annual.py always comes from actual_step), so it is - # identical between the honest and inflated runs by construction and does NOT itself - # exercise the Prediction swap - the assertion below on it is a sanity check, not proof. - # What does exercise the swap is cost: with the swap in place, calculate_plan() commits to - # the inflated forecast's phantom solar, then run_prediction() bills against the actuals - # that never delivered it, so the inflated run's billed cost should be MATERIALLY worse, - # not merely "not cheaper". Observed on this fixture: honest with_predbat cost_p is - # -471.6792p, 3x-inflated-forecast cost_p is -461.2486p, a ~10.43p degradation; - # material_degradation_p is set below that with margin so the assertion is not a knife edge. + # pv_generated_kwh is NOT proof the swap ran: _billed_result() reads it from the pv_step + # ARGUMENT run_day() passes in, which is always actual_step regardless of the swap, so it + # would still read correctly even if the swap back to actuals were deleted entirely - the + # check on it below is a sanity check only. The decisive, non-noise-sensitive check is + # structural: after run_day() returns, predbat.prediction is the exact object annual.py + # builds from actual_step immediately before costing scenario 3 (the "swap"), and + # run_prediction() (called from _billed_result) reads pv data from THIS object, not from + # whatever calculate_plan() searched against. So predbat.prediction.pv_forecast_minute_step + # must total the ACTUAL pv energy, not the (here, 3x inflated) forecast calculate_plan() + # was given - a plan-search-noise-proof check, since actual and 3x-inflated-forecast + # totals differ by a factor of three, not a fraction of a percent. + midnight = pytz.utc.localize(datetime(day.year, day.month, day.day)) + actual_pv_total = sum(weather.pv_minutes("actual", midnight, PLAN_MINUTES).values()) + honest_prediction_pv_total = sum(my_predbat.prediction.pv_forecast_minute_step.values()) + if abs(honest_prediction_pv_total - actual_pv_total) > 0.5: + print(" ERROR: after an honest-forecast run, predbat.prediction.pv_forecast_minute_step should total the actual PV energy ({}), got {}".format(actual_pv_total, honest_prediction_pv_total)) + failed = True + inflated = StubWeather(peak_kw=4.0, forecast_multiplier=3.0) + inflated_forecast_total = sum(inflated.pv_minutes("forecast", midnight, PLAN_MINUTES).values()) inflated_results = run_one(my_predbat, config, inflated, day) + inflated_prediction_pv_total = sum(my_predbat.prediction.pv_forecast_minute_step.values()) + if abs(inflated_prediction_pv_total - actual_pv_total) > 0.5: + print(" ERROR: with a 3x inflated forecast, pv_forecast_minute_step should still total the actual PV energy ({}), got {} (forecast alone totals {})".format(actual_pv_total, inflated_prediction_pv_total, inflated_forecast_total)) + failed = True + if abs(inflated_prediction_pv_total - inflated_forecast_total) < 1.0: + print(" ERROR: predbat.prediction.pv_forecast_minute_step matches the inflated FORECAST total ({}) rather than actuals ({}) - the swap back to actuals did not happen".format(inflated_forecast_total, actual_pv_total)) + failed = True + honest_pv = results["with_predbat"]["pv_generated_kwh"] inflated_pv = inflated_results["with_predbat"]["pv_generated_kwh"] if abs(inflated_pv - honest_pv) > 0.01: print(" ERROR: reported PV should track actuals ({}) regardless of the forecast, got {}".format(honest_pv, inflated_pv)) failed = True - honest_cost = results["with_predbat"]["cost_p"] - inflated_cost = inflated_results["with_predbat"]["cost_p"] - material_degradation_p = 3.0 - if inflated_cost < honest_cost - 1e-6: - print(" ERROR: planning against an over-optimistic forecast must not make the billed cost cheaper") - failed = True - if inflated_cost < honest_cost + material_degradation_p: - print(" ERROR: an over-optimistic forecast should cost at least {}p more (over-commits to solar that never arrives); got honest {} vs inflated {}".format(material_degradation_p, honest_cost, inflated_cost)) - failed = True print("Test: state isolation - a day run in isolation matches the same day run after another") isolated = run_one(my_predbat, config, weather, day) diff --git a/apps/predbat/tests/test_annual_results.py b/apps/predbat/tests/test_annual_results.py index a94991ae6..dd045391e 100644 --- a/apps/predbat/tests/test_annual_results.py +++ b/apps/predbat/tests/test_annual_results.py @@ -152,6 +152,24 @@ def test_annual_results(my_predbat): print(" ERROR: expected weighted with_predbat import_kwh {}, got {}".format(expected_with_predbat, totals["with_predbat"]["import_kwh"])) failed = True + print("Test: a degraded month's survivor is reweighted to represent the full month, not its original share") + # select_samples() would have given each of an original two samples in a 31 day month a + # weight of 31/2 = 15.5. One of them failed run_day() and was dropped. Naively summing the + # survivor at its original 15.5 weight would silently under-count the month by half; the + # fix must recompute the survivor's weight from the surviving count (31/1 = 31) so the + # month's total still represents all 31 days, using the ONE plan that actually ran. + predictor = make_predictor() + original_samples = [(date(2025, 3, 10), 15.5), (date(2025, 3, 24), 15.5)] + surviving_samples = original_samples[:1] + day_result = {key: {field: 2.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + reweighted_samples = predictor._reweight_survivors(surviving_samples, days_in_month=31) + totals = predictor._month_scenarios(reweighted_samples, [day_result]) + expected_reweighted = 2.0 * 31 + wrong_if_unweighted = 2.0 * 15.5 + if totals["no_pvbat"]["cost_p"] != expected_reweighted: + print(" ERROR: expected the degraded month's total to be the survivor's figure times the full 31 days ({}), got {} (original-weight total would have been {})".format(expected_reweighted, totals["no_pvbat"]["cost_p"], wrong_if_unweighted)) + failed = True + print("Test: average_rate") full_rates = {0: 10.0, 1: 20.0, 2: 30.0} if average_rate(full_rates, 3) != 20.0: From 8e523b160e025bc7bf1c12d5ab5b51ddd52f6e3a Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 16:42:28 +0200 Subject: [PATCH 030/119] feat(annual): add annual prediction command line interface --- apps/predbat/annual_cli.py | 157 ++++++++++++++++++++++++++ apps/predbat/tests/test_annual_cli.py | 113 ++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 272 insertions(+) create mode 100644 apps/predbat/annual_cli.py create mode 100644 apps/predbat/tests/test_annual_cli.py diff --git a/apps/predbat/annual_cli.py b/apps/predbat/annual_cli.py new file mode 100644 index 000000000..7431dba83 --- /dev/null +++ b/apps/predbat/annual_cli.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Command line entry point for the annual prediction tool. + +Usage: + python3 annual_cli.py --config annual.yaml --out results.json +""" + +import argparse +import asyncio +import calendar +import json +import os +import sys + +import yaml + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from annual import SCENARIO_KEYS, AnnualConfigError, AnnualPredictor # noqa: E402 +from storage import StorageLocalFiles # noqa: E402 + +SCENARIO_LABELS = {"no_pvbat": "No PV/Battery", "without_predbat": "Without Predbat", "with_predbat": "With Predbat"} + + +def _format_pence(pence, currency): + """Format a pence amount as a pounds figure, or append an explicit currency label. + + ``cost_p`` and friends are stored in pence throughout the results document, + so the default ``currency="p"`` simply renders the pounds equivalent with no + suffix (the values are conventionally reported in £ once divided). Any other + ``currency`` value is appended so a caller adapting this for a non-GBP config + still gets a labelled figure rather than a bare, ambiguous number. + """ + amount = pence / 100.0 + if currency == "p": + return "{:.2f}".format(amount) + return "{:.2f} {}".format(amount, currency) + + +def format_table(results, currency="p"): + """Render the results document as a human-readable table. + + Handles two cases the original design missed: a "degraded" month (some, but not + all, of its sampled days failed) is still costed and included in the annual total, + so it is rendered with its figures rather than being folded into the "unavailable" + branch; and when no month produced a usable result at all, ``annual["scenarios"]`` + and ``annual["standing_charge_p"]`` are ``None`` and ``annual["savings"]`` is empty, + so the totals/savings section is skipped entirely instead of dividing ``None`` or + printing a fabricated zero-cost year. + """ + lines = [] + lines.append("Annual prediction for {}".format(results["year"])) + lines.append("") + header = "{:<6}".format("Month") + "".join("{:>20}".format(SCENARIO_LABELS[key]) for key in SCENARIO_KEYS) + lines.append(header) + lines.append("-" * len(header)) + + for entry in results["months"]: + name = calendar.month_abbr[entry["month"]] + if entry["status"] not in ("ok", "degraded"): + lines.append("{:<6}{:>60}".format(name, "unavailable - {}".format(entry.get("reason", "unknown")))) + continue + row = "{:<6}".format(name) + for key in SCENARIO_KEYS: + row += "{:>20}".format(_format_pence(entry["scenarios"][key]["cost_p"], currency)) + if entry["status"] == "degraded": + row += " (degraded - some sampled days failed to plan)" + lines.append(row) + + annual = results["annual"] + lines.append("-" * len(header)) + + if annual["scenarios"] is not None: + total_row = "{:<6}".format("Year") + for key in SCENARIO_KEYS: + total_row += "{:>20}".format(_format_pence(annual["scenarios"][key]["cost_p"], currency)) + lines.append(total_row) + else: + lines.append("No annual total available: no month produced a usable result.") + + lines.append("") + lines.append("Based on {} of 12 months.".format(annual["months_included"])) + if annual["months_excluded"]: + lines.append("Excluded months: {}".format(", ".join(calendar.month_abbr[month] for month in annual["months_excluded"]))) + + if annual["scenarios"] is not None: + lines.append("") + lines.append("Savings") + lines.append(" PV and battery vs no system: {}".format(_format_pence(annual["savings"].get("pv_battery_vs_none_p", 0.0), currency))) + lines.append(" Predbat vs without Predbat: {}".format(_format_pence(annual["savings"].get("predbat_vs_baseline_p", 0.0), currency))) + lines.append(" Standing charge (all scenarios): {}".format(_format_pence(annual["standing_charge_p"], currency))) + lines.append(" Export credit (with Predbat, estimate - already included in cost above): {}".format(_format_pence(annual["scenarios"]["with_predbat"]["export_credit_p_estimate"], currency))) + + if results.get("caveats"): + lines.append("") + lines.append("Caveats") + for caveat in results["caveats"]: + lines.append(" - {}".format(caveat)) + + return "\n".join(lines) + + +def make_progress(quiet): + """Return a progress callback that writes to stderr, or None when quiet.""" + if quiet: + return None + + def progress(completed, total, message): + """Report progress to stderr so stdout stays parseable.""" + sys.stderr.write("[{}/{}] {}\n".format(completed, total, message)) + sys.stderr.flush() + + return progress + + +def main(argv=None): + """Parse arguments, run the projection, and write the results. Returns an exit code.""" + parser = argparse.ArgumentParser(description="Project a year of electricity costs using the Predbat engine") + parser.add_argument("--config", required=True, help="Path to the annual prediction YAML config") + parser.add_argument("--out", default=None, help="Write the results JSON to this path") + parser.add_argument("--work-dir", default="./annual_work", help="Working directory for the headless Predbat instance and cache") + parser.add_argument("--quiet", action="store_true", help="Suppress progress output") + args = parser.parse_args(argv) + + try: + with open(args.config, "r") as handle: + config = yaml.safe_load(handle) + except (OSError, yaml.YAMLError) as error: + sys.stderr.write("Could not read config {}: {}\n".format(args.config, error)) + return 2 + + storage = StorageLocalFiles(args.work_dir, print) + + try: + predictor = AnnualPredictor(config, log=print if not args.quiet else lambda *a, **k: None, storage=storage, work_dir=args.work_dir) + results = asyncio.run(predictor.run(progress=make_progress(args.quiet))) + except AnnualConfigError as error: + sys.stderr.write("Config error: {}\n".format(error)) + return 2 + + if args.out: + with open(args.out, "w") as handle: + json.dump(results, handle, indent=2) + sys.stderr.write("Results written to {}\n".format(args.out)) + + print(format_table(results)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/apps/predbat/tests/test_annual_cli.py b/apps/predbat/tests/test_annual_cli.py new file mode 100644 index 000000000..27f9e6f7a --- /dev/null +++ b/apps/predbat/tests/test_annual_cli.py @@ -0,0 +1,113 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the annual prediction command line output.""" + +from annual_cli import format_table + + +def sample_results(): + """Return a small results document covering an ok month and an unavailable one.""" + scenarios = { + "no_pvbat": {"cost_p": 12000.0, "import_kwh": 400.0, "export_kwh": 0.0, "pv_generated_kwh": 0.0, "battery_throughput_kwh": 0.0, "export_credit_p_estimate": 0.0, "self_consumed_kwh": 0.0, "self_consumed_kwh_meaningful": True}, + "without_predbat": {"cost_p": 8000.0, "import_kwh": 300.0, "export_kwh": 20.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 90.0, "export_credit_p_estimate": 300.0, "self_consumed_kwh": 100.0, "self_consumed_kwh_meaningful": True}, + "with_predbat": {"cost_p": 6000.0, "import_kwh": 280.0, "export_kwh": 45.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 140.0, "export_credit_p_estimate": 675.0, "self_consumed_kwh": 75.0, "self_consumed_kwh_meaningful": True}, + } + return { + "year": 2025, + "config": {}, + "months": [ + {"month": 1, "status": "ok", "days": 31, "sampled_days": ["2025-01-08", "2025-01-24"], "standing_charge_p": 1860.0, "scenarios": scenarios}, + {"month": 2, "status": "unavailable", "reason": "no rate data available", "days": 28, "standing_charge_p": 1680.0}, + ], + "annual": { + "scenarios": scenarios, + "standing_charge_p": 1860.0, + "savings": {"pv_battery_vs_none_p": 4000.0, "predbat_vs_baseline_p": 2000.0}, + "months_included": 1, + "months_excluded": [2], + }, + "caveats": ["An example caveat."], + } + + +def sample_results_no_usable_month(): + """Return a results document where every month is unavailable and no annual total exists. + + Mirrors what ``AnnualPredictor._build_results`` returns when nothing is included: + ``annual.scenarios`` and ``annual.standing_charge_p`` are ``None`` and ``savings`` is + empty. ``format_table`` must not divide those ``None`` values or print them as zeroes. + """ + return { + "year": 2025, + "config": {}, + "months": [ + {"month": 1, "status": "unavailable", "reason": "no rate data available", "days": 31, "standing_charge_p": 1860.0}, + {"month": 2, "status": "unavailable", "reason": "no usable weather days", "days": 28, "standing_charge_p": 1680.0}, + ], + "annual": { + "scenarios": None, + "standing_charge_p": None, + "savings": {}, + "months_included": 0, + "months_excluded": [1, 2], + }, + "caveats": ["No month produced a usable result, so no annual totals or savings could be calculated."], + } + + +def test_annual_cli(my_predbat): + """Verify the table output reports every month, including excluded ones.""" + failed = False + print("**** Testing annual CLI output ****") + + table = format_table(sample_results()) + + print("Test: the table names the year and every scenario") + for fragment in ["2025", "No PV/Battery", "Without Predbat", "With Predbat"]: + if fragment not in table: + print(" ERROR: the table should mention '{}'".format(fragment)) + failed = True + + print("Test: an unavailable month is shown as excluded, never as zero") + if "unavailable" not in table.lower(): + print(" ERROR: the table must state that February was unavailable") + failed = True + if "no rate data available" not in table: + print(" ERROR: the table should state why the month was excluded") + failed = True + + print("Test: annual savings appear") + if "Savings" not in table: + print(" ERROR: the table should include a savings section") + failed = True + + print("Test: caveats are printed rather than buried in the JSON") + if "An example caveat." not in table: + print(" ERROR: caveats must be shown to the user") + failed = True + + print("Test: the excluded-month count is stated alongside the annual totals") + if "1 of 12" not in table: + print(" ERROR: the table should state how many months are included, got:\n{}".format(table)) + failed = True + + print("Test: when no month is usable, the table does not fabricate a zero-cost year") + empty_table = format_table(sample_results_no_usable_month()) + if "0 of 12" not in empty_table: + print(" ERROR: the table should state 0 of 12 months were used, got:\n{}".format(empty_table)) + failed = True + if "Savings" in empty_table: + print(" ERROR: the table must not print a savings section when there is no annual total") + failed = True + if "0.00" in empty_table: + print(" ERROR: the table must not render the missing annual total as a zero cost, got:\n{}".format(empty_table)) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index c9c387159..4c3440be2 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -169,6 +169,7 @@ from tests.test_annual_scenarios import test_annual_scenarios from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration +from tests.test_annual_cli import test_annual_cli # Mock the components and plugin system @@ -416,6 +417,7 @@ def main(): ("annual_scenarios", test_annual_scenarios, "Annual prediction scenario helper tests", False), ("annual_results", test_annual_results, "Annual prediction results assembly tests", False), ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), + ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), ] # Parse command line arguments From 0d0fc1e3bba81a2eff203356653007d9c36cd4c5 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 16:49:55 +0200 Subject: [PATCH 031/119] fix(annual): don't lose --out failures, label the currency, and cover degraded months MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - main() no longer lets a failed --out write kill the process before the table is printed: the write is wrapped in try/except, the table still prints to stdout, and a non-zero exit code reports the failure. - format_table's pence values now render with an explicit £ prefix instead of a bare, ambiguous number. - Added fixtures/assertions covering a "degraded" month end-to-end (costed, included, not "unavailable") and the export-credit-already-included line, so reverting either behaviour now fails the test. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual_cli.py | 34 ++++++++++------ apps/predbat/tests/test_annual_cli.py | 56 +++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 12 deletions(-) diff --git a/apps/predbat/annual_cli.py b/apps/predbat/annual_cli.py index 7431dba83..999e8c74a 100644 --- a/apps/predbat/annual_cli.py +++ b/apps/predbat/annual_cli.py @@ -29,17 +29,18 @@ def _format_pence(pence, currency): - """Format a pence amount as a pounds figure, or append an explicit currency label. - - ``cost_p`` and friends are stored in pence throughout the results document, - so the default ``currency="p"`` simply renders the pounds equivalent with no - suffix (the values are conventionally reported in £ once divided). Any other - ``currency`` value is appended so a caller adapting this for a non-GBP config - still gets a labelled figure rather than a bare, ambiguous number. + """Format a pence amount as an explicitly-labelled pounds figure. + + ``cost_p`` and friends are stored in pence throughout the results document. + A bare "12.34" tells the reader nothing about the unit, so the default + ``currency="p"`` renders the pounds equivalent with a leading "£" (e.g. + "£12.34") rather than an unlabelled number. Any other ``currency`` value is + appended as a label instead, so a caller adapting this for a non-GBP config + still gets an unambiguous figure. """ amount = pence / 100.0 if currency == "p": - return "{:.2f}".format(amount) + return "£{:.2f}".format(amount) return "{:.2f} {}".format(amount, currency) @@ -144,13 +145,22 @@ def main(argv=None): sys.stderr.write("Config error: {}\n".format(error)) return 2 + exit_code = 0 if args.out: - with open(args.out, "w") as handle: - json.dump(results, handle, indent=2) - sys.stderr.write("Results written to {}\n".format(args.out)) + try: + with open(args.out, "w") as handle: + json.dump(results, handle, indent=2) + sys.stderr.write("Results written to {}\n".format(args.out)) + except (OSError, TypeError, ValueError) as error: + # The projection just took several minutes to compute; a failed write to + # --out must not throw the results away. The table below is still printed + # and the failure is only reported through a non-zero exit code, so a + # caller relying on --out to have succeeded is not fooled into thinking it did. + sys.stderr.write("Could not write results to {}: {}\n".format(args.out, error)) + exit_code = 1 print(format_table(results)) - return 0 + return exit_code if __name__ == "__main__": diff --git a/apps/predbat/tests/test_annual_cli.py b/apps/predbat/tests/test_annual_cli.py index 27f9e6f7a..d930b3439 100644 --- a/apps/predbat/tests/test_annual_cli.py +++ b/apps/predbat/tests/test_annual_cli.py @@ -62,6 +62,36 @@ def sample_results_no_usable_month(): } +def sample_results_with_degraded_month(): + """Return a results document containing one 'degraded' month, built from fewer samples. + + Mirrors what ``AnnualPredictor.run()`` emits when some, but not all, of a month's + sampled days failed to plan: the month still carries real ``scenarios`` figures and + a non-empty ``failed_days`` list, and - unlike an "unavailable" month - it IS counted + in ``annual.months_included`` and is absent from ``annual.months_excluded``. + """ + scenarios = { + "no_pvbat": {"cost_p": 9000.0, "import_kwh": 300.0, "export_kwh": 0.0, "pv_generated_kwh": 0.0, "battery_throughput_kwh": 0.0, "export_credit_p_estimate": 0.0, "self_consumed_kwh": 0.0, "self_consumed_kwh_meaningful": True}, + "without_predbat": {"cost_p": 7000.0, "import_kwh": 250.0, "export_kwh": 15.0, "pv_generated_kwh": 100.0, "battery_throughput_kwh": 80.0, "export_credit_p_estimate": 200.0, "self_consumed_kwh": 85.0, "self_consumed_kwh_meaningful": True}, + "with_predbat": {"cost_p": 5000.0, "import_kwh": 230.0, "export_kwh": 30.0, "pv_generated_kwh": 100.0, "battery_throughput_kwh": 120.0, "export_credit_p_estimate": 450.0, "self_consumed_kwh": 70.0, "self_consumed_kwh_meaningful": True}, + } + return { + "year": 2025, + "config": {}, + "months": [ + {"month": 3, "status": "degraded", "days": 31, "sampled_days": ["2025-03-10"], "failed_days": ["2025-03-24"], "standing_charge_p": 1860.0, "scenarios": scenarios}, + ], + "annual": { + "scenarios": scenarios, + "standing_charge_p": 1860.0, + "savings": {"pv_battery_vs_none_p": 2000.0, "predbat_vs_baseline_p": 2000.0}, + "months_included": 1, + "months_excluded": [], + }, + "caveats": [], + } + + def test_annual_cli(my_predbat): """Verify the table output reports every month, including excluded ones.""" failed = False @@ -98,6 +128,32 @@ def test_annual_cli(my_predbat): print(" ERROR: the table should state how many months are included, got:\n{}".format(table)) failed = True + print("Test: the export credit line warns it is already included in cost, not additional income") + if "export credit" not in table.lower(): + print(" ERROR: the table should show an export credit line, got:\n{}".format(table)) + failed = True + if "already included" not in table.lower(): + print(" ERROR: the export credit line must warn it is already counted inside cost, to stop it being double-counted, got:\n{}".format(table)) + failed = True + + print("Test: a degraded month (some sampled days failed) is costed and included, not treated as unavailable") + degraded_table = format_table(sample_results_with_degraded_month()) + if "unavailable" in degraded_table.lower(): + print(" ERROR: a degraded month must not be rendered as unavailable, got:\n{}".format(degraded_table)) + failed = True + if "£90.00" not in degraded_table: + print(" ERROR: the degraded month's no_pvbat cost (9000p = £90.00) should still be rendered, got:\n{}".format(degraded_table)) + failed = True + if "Excluded months" in degraded_table: + print(" ERROR: a degraded month must not appear as excluded, got:\n{}".format(degraded_table)) + failed = True + if "1 of 12" not in degraded_table: + print(" ERROR: the degraded month should still count towards months_included, got:\n{}".format(degraded_table)) + failed = True + if "degraded" not in degraded_table.lower(): + print(" ERROR: the table should signal that the month came from fewer samples than planned, got:\n{}".format(degraded_table)) + failed = True + print("Test: when no month is usable, the table does not fabricate a zero-cost year") empty_table = format_table(sample_results_no_usable_month()) if "0 of 12" not in empty_table: From efa85d31f995e579ba8d9744522e27d9f479b79a Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 17:12:28 +0200 Subject: [PATCH 032/119] docs: add annual prediction tool documentation Documents the annual electricity-cost prediction tool (annual.py/annual_cli.py): the three scenarios, configuration schema with validated ranges, CLI usage, and the results document fields (status, export_credit_p_estimate, self_consumed_kwh) and their caveats. Adds the page to mkdocs.yml's nav. --- docs/annual-prediction.md | 185 ++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 186 insertions(+) create mode 100644 docs/annual-prediction.md diff --git a/docs/annual-prediction.md b/docs/annual-prediction.md new file mode 100644 index 000000000..cea8c14d8 --- /dev/null +++ b/docs/annual-prediction.md @@ -0,0 +1,185 @@ +# Annual prediction + +The annual prediction tool projects a year of household electricity costs using the +real Predbat planning engine. For each month it reports three scenarios: + +1. **No PV, no battery** — the counterfactual bill. +2. **PV and battery, without Predbat** — a battery charging on a static cheap-rate timer. +3. **PV and battery, with Predbat** — the optimiser's plan. + +It is a standalone command line tool; it does not need Home Assistant or any hardware, +and it makes no changes to a running Predbat installation. + +## How it works + +For each month the tool picks sample days by irradiance percentile (ranked by actual PV +energy, not what was forecast), so the answer does not swing on whether the sampled days +happened to be sunny. Two samples per month is the default; each represents half the +month. On a battery-only run (no solar arrays configured) there is nothing to rank by, so +days are instead spread evenly across the calendar. + +Each sampled day gets a 48-hour plan starting at midnight, but only the first 24 hours +are billed — the second day exists so the optimiser does not artificially drain the +battery at the horizon. Whatever charge is left at the end is valued, so a scenario +cannot look cheap by finishing empty. + +Predbat plans against the **archived weather forecast** for that date and is costed +against **ERA5 actuals**. This matters: costing against the same series it planned from +would hand Predbat perfect foresight and overstate its savings. + +Solar uncertainty (P10) is derived from measured forecast error — for each month, the +10th percentile of the actual-over-forecast daily energy ratio, taken across every day in +that month with both a usable actual and forecast reading. A month needs at least seven +such day pairs before its measured P10 ratio is trusted; below that it falls back to a +flat derate (0.7 by default, `pv10_derate_fallback`), and the run's caveats say so. + +## Configuration + +```yaml +annual: + location: + postcode: "SW1A 1AA" # or latitude/longitude + year: 2025 # defaults to the most recent complete calendar year + + solar: # omit for a battery-only run + - kwp: 5.6 + declination: 35 # pitch in degrees, default 35 + azimuth: 180 # 180 = south, default 180 + efficiency: 0.95 # default 0.95, must be greater than 0 and at most 1 + + battery: # omit for a PV-only run + size_kwh: 9.5 + inverter_kw: 5.0 + export_limit_kw: 5.0 # defaults to inverter_kw + hybrid: true # false = AC coupled + charge_rate_kw: 3.6 # defaults to inverter_kw + discharge_rate_kw: 3.6 # defaults to inverter_kw + + load: + annual_kwh: 3800 + shape: flat # night | day | flat + car_charging_kwh: 2500 # annual, 0 to disable + + tariff: + import_octopus_url: "https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{dno_region}/standard-unit-rates/" + export_octopus_url: "..." + dno_region: "A" # required when a URL contains {dno_region} + standing_charge_p_per_day: 60.0 + + samples_per_month: 2 +``` + +At least one of `solar` or `battery` must be given — with neither there is nothing to +evaluate. `annual.location` needs either `postcode` or both `latitude` and `longitude`; +if both are given, latitude/longitude wins. + +Octopus product codes are region-suffixed. If your tariff URL contains `{dno_region}` +you must also set `dno_region` to your region letter (`A` for Eastern England, and so +on) — the config is rejected up front with an error naming the offending field, rather +than left to 404 at fetch time and reported as an unavailable month, which would look +like an Octopus outage rather than a config mistake. + +Numeric fields are range-checked and a bad value is rejected with an explanatory message +rather than silently producing a nonsense result: `kwp`, `size_kwh`, `inverter_kw`, +`charge_rate_kw` and `discharge_rate_kw` must all be greater than zero; `efficiency` and +`pv10_derate_fallback` must be greater than zero and at most one; `samples_per_month` +must be a whole number of at least one; and `year` must be between 1940 (the start of +the Open-Meteo ERA5 archive) and the current year. + +Instead of `annual_kwh`, `shape` and `car_charging_kwh` you may supply real consumption: + +```yaml + load: + octopus: + api_key: !secret octopus_key + account_id: A-1234ABCD +``` + +These two forms are **mutually exclusive** and supplying both is rejected. The Octopus +consumption series already includes any EV charging, so accepting both would +double-count it. The trade-off is that a car baked into the meter data cannot be +smart-planned separately. A day missing from the downloaded consumption falls back to a +synthetic UK-average profile rather than being billed as zero. + +Instead of an Octopus URL you may give a fixed rate structure: + +```yaml + tariff: + rates_import: + - start: "00:30:00" + end: "05:30:00" + rate: 7.0 + - start: "05:30:00" + end: "00:30:00" + rate: 28.0 + rates_export: + - rate: 15.0 +``` + +## Running it + +```bash +cd apps/predbat +python3 annual_cli.py --config annual.yaml --out results.json +``` + +Other options: `--work-dir` (default `./annual_work`) sets where the headless Predbat +instance and the download cache live, and `--quiet` suppresses the per-month progress +lines written to stderr. A human-readable table is always printed to stdout; `--out` +additionally writes the full results document as JSON. + +A run takes roughly one to three minutes with the default two samples per month: 24 plan +calculations (12 months × 2 samples) plus the weather, tariff and (if configured) +Octopus consumption downloads, which are cached between runs in `--work-dir`. + +## Reading the results + +Each month in the results JSON has a `status`: + +- `ok` — every sampled day planned successfully. +- `degraded` — some sampled days failed to plan or cost; the survivors are reweighted + so the month still represents a full month, and the failed dates are listed under + `failed_days`. It is still included in the annual totals. +- `unavailable` — no rate data, no usable weather days, or every sampled day failed. + Excluded from the annual totals entirely, rather than counted as zero. + +Within each included month, every scenario reports `cost_p`, `import_kwh`, `export_kwh`, +`pv_generated_kwh`, `battery_throughput_kwh`, `export_credit_p_estimate` and +`self_consumed_kwh` (plus `self_consumed_kwh_meaningful`, see below). + +`export_credit_p_estimate` is **not** extra money on top of `cost_p` — `cost_p` already +prices every export minute at its real per-minute rate, so the export credit is already +inside it. `export_credit_p_estimate` is a cruder second estimate of that same income +(the day's flat average export rate applied to total export), kept only as a +human-readable "how much of that came from export" figure. Adding it to `cost_p` +double-counts export income. + +`self_consumed_kwh` is `pv_generated_kwh` minus `export_kwh` and is approximate: when the +battery exports grid-charged energy (rather than genuine excess solar) it is understated. +If export exceeds generation for a scenario, `self_consumed_kwh` is clamped to zero and +`self_consumed_kwh_meaningful` is `False` for that scenario/month, flagging that the +figure should not be trusted there. + +The top-level `annual` block sums the included months' scenarios and reports two savings +figures: `pv_battery_vs_none_p` (PV and battery vs. no system) and +`predbat_vs_baseline_p` (Predbat vs. the dumb timer baseline). If no month produced a +usable result, `annual.scenarios`, `annual.standing_charge_p` and `annual.savings` are +empty rather than a fabricated zero-cost year. + +The `caveats` list in the results document records anything that could affect how much +to trust the numbers — a P10 fallback, a missing month's rate data, and the +`export_credit_p_estimate`/`self_consumed_kwh` notes above among them — and is worth +reading before quoting the totals. + +## Limitations + +- The Open-Meteo forecast archive only reaches back to about 2021. For earlier years the + tool plans against actuals instead and P10 uses the flat fallback derate, which it + states in the results' caveats — savings are likely overstated for those years. +- `self_consumed_kwh` is approximate, as described above. +- The forecast-versus-ERA5 gap includes systematic model bias as well as genuine + forecast error, so measured solar uncertainty is slightly overstated. +- A month with no rate data, no usable weather days, or where every sampled day failed to + plan is reported as `unavailable` and excluded from the annual total, rather than + counted as zero. +- Heat pump, iBoost and gas modelling are not included. diff --git a/mkdocs.yml b/mkdocs.yml index 5fb8786bc..99e0db2ca 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -26,6 +26,7 @@ nav: - predbat-plan-card.md - creating-charts.md - compare.md + - annual-prediction.md - 'Predbat development': - todo-list.md - developing.md From 2d7a892ad8d4aedfdf97b402ebedc9f5b4655a34 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 17:16:34 +0200 Subject: [PATCH 033/119] fix(annual): stop silently discarding car_rate_kw during config validation _validate_load() rebuilt the manual load dict from scratch and never copied car_rate_kw through, so run_day() always fell back to the 7.4 kW default regardless of what a user configured, while validation raised no error. Now validated with _require_number (>0) and defaulted to DEFAULT_CAR_RATE_KW when omitted. Left unvalidated/ignored on the Octopus load branch, where car_charging_kwh is forced to zero and there is no separate car energy for a rate to apply to. Adds config-validation tests (default, survives validation, zero/negative rejected, ignored under Octopus) and a scenario test proving a slower car_rate_kw roughly doubles timer_charge_window's length for the same energy. Documents the field in docs/annual-prediction.md. --- apps/predbat/annual.py | 1 + apps/predbat/tests/test_annual_config.py | 35 +++++++++++++++++++++ apps/predbat/tests/test_annual_scenarios.py | 12 +++++++ docs/annual-prediction.md | 11 ++++++- 4 files changed, 58 insertions(+), 1 deletion(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index dc5e58f34..a83362a45 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -163,6 +163,7 @@ def _validate_load(raw): "annual_kwh": _require_number(raw["annual_kwh"], "annual.load.annual_kwh", minimum=0), "shape": shape, "car_charging_kwh": _require_number(raw.get("car_charging_kwh", 0.0), "annual.load.car_charging_kwh", minimum=0), + "car_rate_kw": _require_number(raw.get("car_rate_kw", DEFAULT_CAR_RATE_KW), "annual.load.car_rate_kw", minimum=0, exclusive_minimum=True), } diff --git a/apps/predbat/tests/test_annual_config.py b/apps/predbat/tests/test_annual_config.py index 357888b6e..1907eb47c 100644 --- a/apps/predbat/tests/test_annual_config.py +++ b/apps/predbat/tests/test_annual_config.py @@ -127,6 +127,16 @@ def test_annual_config(my_predbat): print(" ERROR: the Octopus load block should survive validation") failed = True + print("Test: car_rate_kw is meaningless on an Octopus load (no separate car energy to rate) and is simply ignored") + config = base_config() + del config["annual"]["load"]["annual_kwh"] + config["annual"]["load"]["octopus"] = {"api_key": "sk_x", "account_id": "A-1"} + config["annual"]["load"]["car_rate_kw"] = 3.7 + result = validate_config(config, today=date(2026, 7, 25)) + if "car_rate_kw" in result["load"]: + print(" ERROR: car_rate_kw should not appear in an Octopus load block, got {}".format(result["load"])) + failed = True + print("Test: a missing battery block yields a two-scenario run") config = base_config() del config["annual"]["battery"] @@ -164,6 +174,31 @@ def test_annual_config(my_predbat): del config["annual"]["tariff"] failed = expect_error("no tariff", config, "annual.tariff is required", failed) + print("Test: car_rate_kw defaults to 7.4 kW when omitted") + config = base_config() + result = validate_config(config, today=date(2026, 7, 25)) + if result["load"]["car_rate_kw"] != 7.4: + print(" ERROR: car_rate_kw should default to 7.4, got {}".format(result["load"]["car_rate_kw"])) + failed = True + + print("Test: an explicit car_rate_kw survives validation") + config = base_config() + config["annual"]["load"]["car_rate_kw"] = 3.7 + result = validate_config(config, today=date(2026, 7, 25)) + if result["load"]["car_rate_kw"] != 3.7: + print(" ERROR: car_rate_kw should survive validation as 3.7, got {}".format(result["load"]["car_rate_kw"])) + failed = True + + print("Test: a zero car_rate_kw is rejected") + config = base_config() + config["annual"]["load"]["car_rate_kw"] = 0 + failed = expect_error("zero car_rate_kw", config, "car_rate_kw", failed) + + print("Test: a negative car_rate_kw is rejected") + config = base_config() + config["annual"]["load"]["car_rate_kw"] = -3.7 + failed = expect_error("negative car_rate_kw", config, "car_rate_kw", failed) + print("Test: an unknown load shape is rejected") config = base_config() config["annual"]["load"]["shape"] = "sideways" diff --git a/apps/predbat/tests/test_annual_scenarios.py b/apps/predbat/tests/test_annual_scenarios.py index 343c8be67..479d7240c 100644 --- a/apps/predbat/tests/test_annual_scenarios.py +++ b/apps/predbat/tests/test_annual_scenarios.py @@ -48,6 +48,18 @@ def test_annual_scenarios(my_predbat): print(" ERROR: a 10 hour charge should extend past the 5 hour cheap band, got {} minutes".format(long_window[0]["end"] - long_window[0]["start"])) failed = True + print("Test: a slower car_rate_kw produces a roughly proportionally longer window for the same energy") + # Guards the annual.py config-validation fix that lets a configured car_rate_kw actually + # reach timer_charge_window (previously validate_config() silently dropped it and every + # run used the 7.4 kW default regardless of what was configured). + fast_window = timer_charge_window(rates, car_kwh=14.8, car_rate_kw=7.4) + slow_window = timer_charge_window(rates, car_kwh=14.8, car_rate_kw=3.7) + fast_length = fast_window[0]["end"] - fast_window[0]["start"] + slow_length = slow_window[0]["end"] - slow_window[0]["start"] + if abs(slow_length - 2 * fast_length) > 5: + print(" ERROR: halving car_rate_kw from 7.4 to 3.7 should roughly double the window length ({} minutes), got {} minutes for the fast case and {} minutes for the slow case".format(2 * fast_length, fast_length, slow_length)) + failed = True + print("Test: zero car energy produces no window") if timer_charge_window(rates, car_kwh=0.0, car_rate_kw=7.4) != []: print(" ERROR: no car energy should produce no window") diff --git a/docs/annual-prediction.md b/docs/annual-prediction.md index cea8c14d8..49fb30a24 100644 --- a/docs/annual-prediction.md +++ b/docs/annual-prediction.md @@ -59,6 +59,7 @@ annual: annual_kwh: 3800 shape: flat # night | day | flat car_charging_kwh: 2500 # annual, 0 to disable + car_rate_kw: 7.4 # charger power, default 7.4, must be greater than 0 tariff: import_octopus_url: "https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{dno_region}/standard-unit-rates/" @@ -99,7 +100,15 @@ These two forms are **mutually exclusive** and supplying both is rejected. The O consumption series already includes any EV charging, so accepting both would double-count it. The trade-off is that a car baked into the meter data cannot be smart-planned separately. A day missing from the downloaded consumption falls back to a -synthetic UK-average profile rather than being billed as zero. +synthetic UK-average profile rather than being billed as zero. `car_rate_kw` has no +effect on an Octopus load and is ignored there — there is no separately-tracked car +energy to apply a charging rate to. + +`car_rate_kw` is the charger's power, used to size both the dumb timer's charge window +(scenario 2) and the smart plan's charging rate (scenario 3): a smaller number (say 3.0 +for a granny charger) spreads the same annual `car_charging_kwh` over a longer window +each day, while a larger one (up to a three-phase charger's 22 kW) charges it faster. It +must be greater than zero. Instead of an Octopus URL you may give a fixed rate structure: From c018f54ff9bbd3174b5a284fcfadb5f366805dd0 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 17:19:51 +0200 Subject: [PATCH 034/119] docs: model car charging as episodic sessions, not a daily trickle Smearing the annual car figure across 365 days makes every top-up short enough to fit the cheapest window, so the dumb-timer baseline prices as well as Predbat does and the EV contributes nothing to the measured saving. Real sessions overflow short cheap bands, which is exactly where smart charging pays. One session a week, split further if it would exceed six hours at the configured rate. Each sampled day is planned with and without a session and blended by frequency, which preserves the irradiance stratification. Co-Authored-By: Claude Opus 5 (1M context) --- ...026-07-25-annual-prediction-tool-design.md | 30 ++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md index e9501c49b..74f39622b 100644 --- a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md +++ b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md @@ -317,10 +317,38 @@ against the same day: The car, where configured, charges in scenarios 1 and 2 on a fixed timer: the same cheapest static window scenario 2 derives for the battery, extended as -needed to fit the day's car kWh at the configured charge rate. Scenario 1 uses +needed to fit the session's kWh at the configured charge rate. Scenario 1 uses that identical window even though it has no battery, so the only difference between the three scenarios is the system being evaluated. Only scenario 3 plans the car smartly. + +#### Charging is episodic, not a daily trickle + +Spreading the annual car figure evenly across 365 days is wrong in a way that +matters. A 2,500 kWh/year smear is 6.85 kWh/day — under an hour at 7.4 kW — +which fits trivially inside any cheap overnight window, so the dumb timer gets +the cheap rate too and the gap between it and Predbat collapses to nothing. Real +owners charge in sessions of 20-40 kWh that can overflow a short cheap band +(Flux, Cosy), forcing part onto expensive rates under a timer while Predbat +splits the load across the cheapest half-hours. That overflow is where smart +charging earns its money, and smearing deletes it. + +The schedule is derived, not configured: + +- One session per week by default, carrying the whole week's energy. +- If that session would exceed **six hours** at the configured charge rate, split + into as many sessions per week as needed to bring each under six hours, + capped at seven (daily). + +Each sampled day is then planned **twice** — once carrying a full session, once +with no car — and the two results are blended by how often charging actually +happens: `cost = f × with_car + (1 − f) × without_car`, where +`f = sessions_per_week / 7`. Blending per sampled day rather than dedicating +separate sample days keeps the irradiance stratification intact; every cost and +energy field blends linearly, and `pv_generated_kwh` is identical in both legs. + +This doubles the plan count for configs with a car (24 → 48 runs/year). Configs +without one are unaffected. 3. **With Predbat** — `calculate_plan(recompute=True, publish=False)` then `run_prediction(charge_limit_best, charge_window_best, export_window_best, export_limits_best)`. The car uses real smart slot planning. From e5a0422120324dd064e1cdd288b4d6c4d0b2546f Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 18:07:10 +0200 Subject: [PATCH 035/119] feat(annual): model car charging as weekly sessions, not a daily trickle Dividing the annual car figure by 365 made every top-up short enough to fit the cheapest overnight window, so the dumb-timer baseline priced as well as Predbat and the EV contributed almost nothing to the measured saving. Real sessions overflow short cheap bands, and that overflow is where smart charging pays. car_charging_schedule() derives one session a week from the annual energy and charger power, splitting further if a session would exceed six hours. run_day() plans each sampled day twice, with and without a session, and blends by frequency - which keeps the irradiance stratification intact instead of tying charging state to how sunny the sampled day was. Configs without a car still run a single leg. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual.py | 133 ++++++++++++++++-- apps/predbat/tests/test_annual_integration.py | 47 ++++++- apps/predbat/tests/test_annual_scenarios.py | 73 +++++++++- docs/annual-prediction.md | 33 ++++- 4 files changed, 265 insertions(+), 21 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index a83362a45..245b8e1a7 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -13,6 +13,7 @@ """ import calendar +import math import os from datetime import date, datetime, timedelta @@ -584,6 +585,53 @@ def select_samples(weather, year, month, samples_per_month, has_solar=True): # time is enough to give Predbat a fair one-shot charging decision to make. DEFAULT_CAR_READY_TIME = "07:00:00" +# A charging session longer than this will not reliably fit inside a typical cheap +# overnight band (Flux, Cosy), so car_charging_schedule() splits into more, shorter +# sessions per week once a single weekly session would exceed it. +CAR_SESSION_MAX_HOURS = 6.0 + +# However many sessions a week would be needed to keep every session under +# CAR_SESSION_MAX_HOURS, never plan more than one a day. +MAX_SESSIONS_PER_WEEK = 7 + + +def car_charging_schedule(annual_kwh, car_rate_kw): + """Derive how often, and how much, a car charges per week from its annual energy. + + Smearing an annual car figure evenly across 365 days is wrong in a way that + matters: a 2,500 kWh/year smear is 6.85 kWh/day, under an hour at 7.4 kW, which + fits trivially inside any cheap overnight window. A dumb timer would then get the + cheap rate just as easily as Predbat, and the EV would contribute almost nothing + to the measured saving. Real owners charge in sessions of tens of kWh that can + overflow a short cheap band, forcing part of the session onto an expensive rate + under a timer while Predbat spreads it across the cheapest half-hours instead - + that overflow is where smart charging earns its keep, and smearing deletes it. + + The schedule is derived from the annual energy and the charger's power, not + configured directly: one session a week carries the whole week's energy unless + that session would run longer than ``CAR_SESSION_MAX_HOURS`` at the given rate, in + which case the week's energy is split across as many sessions as needed to bring + each under the cap, capped at ``MAX_SESSIONS_PER_WEEK`` (one a day). When even a + daily session cannot get under the cap (a very low charge rate against a very high + annual figure), seven sessions are still returned, but each will run long - the + caller is responsible for logging that the overflow this model is meant to capture + is then understated. + + Returns (sessions_per_week, session_kwh). ``sessions_per_week * session_kwh`` + always equals ``annual_kwh / 52.0`` exactly (division and its inverse), so summing + the week's sessions recovers the annual total. + """ + weekly_kwh = annual_kwh / 52.0 + if weekly_kwh <= 0 or car_rate_kw <= 0: + return 0, 0.0 + + session_hours = weekly_kwh / car_rate_kw + if session_hours <= CAR_SESSION_MAX_HOURS: + return 1, weekly_kwh + + sessions_per_week = min(MAX_SESSIONS_PER_WEEK, math.ceil(session_hours / CAR_SESSION_MAX_HOURS)) + return sessions_per_week, weekly_kwh / sessions_per_week + def build_step_data(predbat, pv_minute, pv_minute10): """Build the 5-minute step arrays the Prediction engine consumes. @@ -661,10 +709,11 @@ def add_car_to_load(load_forecast, windows, car_kwh): Used by the two baseline scenarios, where the car is simply extra load in a fixed timer window rather than something Predbat schedules. ``car_kwh`` is - already the day's energy (``run_day`` divides the annual figure by 365) and - ``windows`` holds one window per day of the plan, each independently sized by + already the energy this leg is charging - a full weekly session, or zero on the + without-car leg ``run_day()`` blends against (see ``car_charging_schedule()``) - + and ``windows`` holds one window per day of the plan, each independently sized by ``timer_charge_window`` to hold the full ``car_kwh`` — so every window gets - the full daily amount, not a share of it. Dividing by ``len(windows)`` here + the full session amount, not a share of it. Dividing by ``len(windows)`` here would silently bill only half the car's energy on the one day that matters (day 1 is the only one ``_billed_result`` costs). """ @@ -775,8 +824,13 @@ def _billed_result(predbat, end_record, pv_step): } -def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc): - """Inject every per-day input into the PredBat instance for one sampled day.""" +def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh): + """Inject every per-day input into the PredBat instance for one sampled day. + + ``car_kwh`` is the energy this specific leg is charging (a full weekly session, or + zero for the without-car leg run_day() blends against - see run_day()), not the raw + annual config figure, so num_cars below reflects what this leg actually models. + """ reset_sample_state(predbat) # calculate_plan() spins up a multiprocessing Pool sized from args["threads"] (plan.py) @@ -803,11 +857,11 @@ def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_ # set_rate_thresholds() (called from _apply_rates() below) reads num_cars to decide # whether to widen rate_import_cost_threshold for car charging (fetch.py). num_cars is - # not part of reset_sample_state()'s leaked-state list — it is a value this day's config - # determines, not a scenario override to merely clear — so it is set here, from config, - # before _apply_rates() runs rather than left at whatever the previous sample's scenario 3 - # happened to leave it as (or the headless bootstrap's own default of 1). - predbat.num_cars = 1 if config["load"].get("car_charging_kwh", 0.0) > 0 else 0 + # not part of reset_sample_state()'s leaked-state list — it is a value this leg's car_kwh + # determines, not a scenario override to merely clear — so it is set here, from the leg's + # own car_kwh, before _apply_rates() runs rather than left at whatever the previous + # sample's scenario 3 happened to leave it as (or the headless bootstrap's own default of 1). + predbat.num_cars = 1 if car_kwh > 0 else 0 rate_import, rate_export = tariff.rates_for(midnight_utc, PLAN_MINUTES) _apply_rates(predbat, rate_import, rate_export) @@ -850,12 +904,15 @@ def _plan_smart_car(predbat, day, car_kwh): return slots -def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): - """Run all three scenarios against one sampled day and return their billed figures.""" - car_kwh = config["load"].get("car_charging_kwh", 0.0) / 365.0 - car_rate_kw = config["load"].get("car_rate_kw", DEFAULT_CAR_RATE_KW) +def _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh, car_rate_kw): + """Run all three scenarios against one sampled day at a fixed car charging energy. - prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc) + ``car_kwh`` is the actual energy this leg charges - either a full weekly charging + session or zero, never the smeared daily average an earlier version of this tool + used (see ``car_charging_schedule()``). ``run_day()`` calls this once when no car is + configured, or twice (once per leg) and blends the two sets of results when one is. + """ + prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh) actual_pv = weather.pv_minutes("actual", midnight_utc, PLAN_MINUTES) if config["solar"] else {} forecast_pv = weather.pv_minutes("forecast", midnight_utc, PLAN_MINUTES) if config["solar"] else {} @@ -940,6 +997,52 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): SCENARIO_FIELDS = ["cost_p", "import_kwh", "export_kwh", "pv_generated_kwh", "battery_throughput_kwh"] +def _blend_results(with_car, without_car, fraction): + """Blend two full three-scenario result dicts field by field. + + ``fraction`` is the share of the week charging actually happens + (``sessions_per_week / 7`` - see ``car_charging_schedule()``), so every field of + every scenario blends linearly: ``fraction * with_car + (1 - fraction) * + without_car``. ``pv_generated_kwh`` is identical between the two legs (the car has + no effect on solar generation), so its blend is a no-op - a useful self-check that + the two legs really are the same sampled day underneath. + """ + return {key: {field: fraction * with_car[key][field] + (1 - fraction) * without_car[key][field] for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + + +def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): + """Run all three scenarios against one sampled day and return their billed figures. + + A configured car charges in weekly sessions, not a daily smear (see + ``car_charging_schedule()``), so the sampled day is planned TWICE when a car is + configured: once carrying a full session, once with no car at all, and the two are + blended by how often a session actually happens that week. Blending per sampled day, + rather than dedicating separate sample days to "car" and "no car", keeps the + irradiance-percentile stratification of the sampled days intact instead of + confounding solar percentile with charging state. A config with no car runs a + single leg, exactly as before this blending was introduced. + """ + car_charging_kwh = config["load"].get("car_charging_kwh", 0.0) + car_rate_kw = config["load"].get("car_rate_kw", DEFAULT_CAR_RATE_KW) + + if car_charging_kwh <= 0: + return _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=0.0, car_rate_kw=car_rate_kw) + + sessions_per_week, session_kwh = car_charging_schedule(car_charging_kwh, car_rate_kw) + if sessions_per_week >= MAX_SESSIONS_PER_WEEK and session_kwh > car_rate_kw * CAR_SESSION_MAX_HOURS + 1e-6: + predbat.log( + "Warn: Annual: {} car charging needs {:.1f} kWh/week at {:.1f} kW, which cannot fit into {} sessions of {:.0f} hours or less; sessions are running long ({:.1f} hours), so the timer/Predbat overflow this model is meant to capture is understated".format( + day, sessions_per_week * session_kwh, car_rate_kw, MAX_SESSIONS_PER_WEEK, CAR_SESSION_MAX_HOURS, session_kwh / car_rate_kw + ) + ) + + with_car = _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=session_kwh, car_rate_kw=car_rate_kw) + without_car = _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=0.0, car_rate_kw=car_rate_kw) + + fraction = sessions_per_week / float(MAX_SESSIONS_PER_WEEK) + return _blend_results(with_car, without_car, fraction) + + def average_rate(rates, minutes): """Return the mean rate across the first ``minutes`` of a rate dict.""" values = [rates[minute] for minute in range(minutes) if minute in rates] diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py index 8c7d5ba06..fdbb81536 100644 --- a/apps/predbat/tests/test_annual_integration.py +++ b/apps/predbat/tests/test_annual_integration.py @@ -115,6 +115,31 @@ def run_one(my_predbat, config, weather, day): return run_day(my_predbat, config, weather, StubTariff(), load_source, day, midnight) +def _count_calculate_plan_calls(my_predbat, action): + """Run ``action`` and return how many times it called ``calculate_plan``. + + run_day() plans a sampled day once when no car is configured and twice - a with-car + leg and a without-car leg it blends against - when one is. Counting the calls is the + only externally visible way to tell those two paths apart, since the blended result + of two identical no-car legs would be indistinguishable from a single leg's. + """ + calls = {"n": 0} + original = my_predbat.calculate_plan + + def counting_calculate_plan(*args, **kwargs): + """Count the call, then delegate to the real planner.""" + calls["n"] += 1 + return original(*args, **kwargs) + + my_predbat.calculate_plan = counting_calculate_plan + try: + action() + finally: + # my_predbat is the shared suite fixture, so the patch must not outlive this call + del my_predbat.calculate_plan + return calls["n"] + + def test_annual_integration(my_predbat): """Verify scenario ordering, the forecast/actuals split, and state isolation.""" failed = False @@ -204,9 +229,27 @@ def test_annual_integration(my_predbat): print(" ERROR: {}.{} changed from {} to {} depending on what ran before it".format(scenario, field, first, second)) failed = True - print("Test: a car charging config still produces an ordered result") + print("Test: a config with no car plans a single leg (one calculate_plan call), not two") + # Guards run_day()'s dispatch: a config with no car must take the cheap, single-leg path + # rather than always running both the with-car and without-car legs. + plan_calls = _count_calculate_plan_calls(my_predbat, lambda: run_one(my_predbat, config, weather, day)) + if plan_calls != 1: + print(" ERROR: a config with no car should call calculate_plan exactly once, got {}".format(plan_calls)) + failed = True + + print("Test: a car charging config plans TWICE (a with-car leg and a without-car leg) and blends the two") car_config = make_config(with_car=True) - car_results = run_one(my_predbat, car_config, weather, day) + car_results = {} + + def _run_car_config(): + car_results.update(run_one(my_predbat, car_config, weather, day)) + + plan_calls = _count_calculate_plan_calls(my_predbat, _run_car_config) + if plan_calls != 2: + print(" ERROR: a config with a car should call calculate_plan exactly twice (with-car and without-car legs), got {}".format(plan_calls)) + failed = True + + print("Test: a car charging config still produces an ordered result") if car_results["with_predbat"]["cost_p"] > car_results["without_predbat"]["cost_p"] + 1e-6: print(" ERROR: with a car, Predbat cost {} should not exceed the timer baseline {}".format(car_results["with_predbat"]["cost_p"], car_results["without_predbat"]["cost_p"])) failed = True diff --git a/apps/predbat/tests/test_annual_scenarios.py b/apps/predbat/tests/test_annual_scenarios.py index 479d7240c..45e511de4 100644 --- a/apps/predbat/tests/test_annual_scenarios.py +++ b/apps/predbat/tests/test_annual_scenarios.py @@ -9,7 +9,7 @@ """Tests for the annual prediction three-scenario day runner.""" -from annual import DAY_MINUTES, PLAN_MINUTES, add_car_to_load, timer_charge_window +from annual import DAY_MINUTES, PLAN_MINUTES, SCENARIO_FIELDS, SCENARIO_KEYS, _blend_results, add_car_to_load, car_charging_schedule, timer_charge_window def flat_rates(cheap_start, cheap_end, cheap_rate, peak_rate): @@ -111,4 +111,75 @@ def test_annual_scenarios(my_predbat): print(" ERROR: the timer should also charge on day two of the 48 hour window") failed = True + print("Test: car_charging_schedule gives a modest annual figure a single weekly session") + # 1500 kWh/year at 7.4 kW is a 28.8 kWh/week session, under the 6 hour cap (3.9 hours). + sessions, session_kwh = car_charging_schedule(annual_kwh=1500, car_rate_kw=7.4) + if sessions != 1: + print(" ERROR: expected 1 session/week for a modest annual figure, got {}".format(sessions)) + failed = True + if abs(session_kwh - 1500 / 52.0) > 1e-9: + print(" ERROR: a single weekly session should carry the whole week's energy ({}), got {}".format(1500 / 52.0, session_kwh)) + failed = True + + print("Test: a session that would exceed 6 hours splits into enough sessions that each is under 6 hours") + # 2500 kWh/year at 7.4 kW is a 48.08 kWh/week need - 6.50 hours as one session, so it must split. + sessions, session_kwh = car_charging_schedule(annual_kwh=2500, car_rate_kw=7.4) + if sessions <= 1: + print(" ERROR: a session over 6 hours should have split into more than 1 session/week, got {}".format(sessions)) + failed = True + session_hours = session_kwh / 7.4 + if session_hours > 6.0 + 1e-9: + print(" ERROR: every split session should be under 6 hours, got {:.3f} hours across {} sessions".format(session_hours, sessions)) + failed = True + + print("Test: the session count caps at 7 even when that still exceeds 6 hours per session") + # 8000 kWh/year at 3.0 kW is a 153.8 kWh/week need - 51.3 hours as one session. Even 7 + # sessions (7.3 hours each) cannot get under the 6 hour cap, so the cap must still hold at 7 + # rather than climbing past a session a day. + sessions, session_kwh = car_charging_schedule(annual_kwh=8000, car_rate_kw=3.0) + if sessions != 7: + print(" ERROR: expected the session count to cap at 7/week, got {}".format(sessions)) + failed = True + if session_kwh / 3.0 <= 6.0: + print(" ERROR: expected this case to still exceed 6 hours per session even at the cap, got {:.3f} hours".format(session_kwh / 3.0)) + failed = True + + print("Test: sessions_per_week * session_kwh * 52 recovers the annual total") + for annual_kwh, car_rate_kw in [(1500, 7.4), (2500, 7.4), (8000, 7.4), (1500, 3.0), (2500, 3.0), (8000, 3.0)]: + sessions, session_kwh = car_charging_schedule(annual_kwh, car_rate_kw) + recovered = sessions * session_kwh * 52.0 + if abs(recovered - annual_kwh) > 1e-6: + print(" ERROR: {} kWh/year at {} kW should recover to {} annual kWh, got {} ({} sessions of {:.3f} kWh)".format(annual_kwh, car_rate_kw, annual_kwh, recovered, sessions, session_kwh)) + failed = True + + print("Test: no annual energy produces no sessions") + sessions, session_kwh = car_charging_schedule(annual_kwh=0, car_rate_kw=7.4) + if sessions != 0 or session_kwh != 0.0: + print(" ERROR: zero annual energy should produce 0 sessions of 0 kWh, got {} sessions of {} kWh".format(sessions, session_kwh)) + failed = True + + print("Test: _blend_results blends every field of every scenario as fraction * with + (1 - fraction) * without") + with_car_results = {key: {field: 10.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + without_car_results = {key: {field: 2.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + with_car_results["with_predbat"]["cost_p"] = 100.0 + without_car_results["with_predbat"]["cost_p"] = 20.0 + fraction = 3.0 / 7.0 + blended = _blend_results(with_car_results, without_car_results, fraction) + expected_cost = fraction * 100.0 + (1 - fraction) * 20.0 + if abs(blended["with_predbat"]["cost_p"] - expected_cost) > 1e-9: + print(" ERROR: expected blended with_predbat cost_p {}, got {}".format(expected_cost, blended["with_predbat"]["cost_p"])) + failed = True + expected_other = fraction * 10.0 + (1 - fraction) * 2.0 + if abs(blended["no_pvbat"]["import_kwh"] - expected_other) > 1e-9: + print(" ERROR: expected blended no_pvbat import_kwh {}, got {}".format(expected_other, blended["no_pvbat"]["import_kwh"])) + failed = True + + print("Test: _blend_results is a no-op for a field identical in both legs (pv_generated_kwh)") + identical = {key: {field: 5.0 for field in SCENARIO_FIELDS} for key in SCENARIO_KEYS} + blended = _blend_results(identical, identical, fraction=0.4) + for key in SCENARIO_KEYS: + if blended[key]["pv_generated_kwh"] != 5.0: + print(" ERROR: blending identical legs should be a no-op, got {} for {}".format(blended[key]["pv_generated_kwh"], key)) + failed = True + return failed diff --git a/docs/annual-prediction.md b/docs/annual-prediction.md index 49fb30a24..24180efa2 100644 --- a/docs/annual-prediction.md +++ b/docs/annual-prediction.md @@ -106,9 +106,36 @@ energy to apply a charging rate to. `car_rate_kw` is the charger's power, used to size both the dumb timer's charge window (scenario 2) and the smart plan's charging rate (scenario 3): a smaller number (say 3.0 -for a granny charger) spreads the same annual `car_charging_kwh` over a longer window -each day, while a larger one (up to a three-phase charger's 22 kW) charges it faster. It -must be greater than zero. +for a granny charger) spreads the same energy over a longer window, while a larger one +(up to a three-phase charger's 22 kW) charges it faster. It must be greater than zero. + +### How car charging is spread across the year + +The car does **not** charge a little every day. Spreading an annual figure evenly across +365 days would make every top-up short enough to fit inside the cheapest overnight +window, so a dumb timer would price just as well as Predbat and the car would contribute +almost nothing to the measured saving. Real sessions are large enough to overflow a short +cheap band, and that overflow is where smart charging earns its money. + +The schedule is derived from `car_charging_kwh` and `car_rate_kw` rather than configured +directly: + +- One session a week, carrying the whole week's energy. +- If that session would run longer than **six hours** at the configured rate, the week's + energy is split across as many sessions as needed to bring each under six hours, up to + a maximum of one a day. + +Each sampled day is then planned **twice** — once carrying a full session, once with no +car — and the two results are blended by how often charging actually happens that week. +Blending each sampled day, rather than setting aside separate "car" and "no car" sample +days, keeps the irradiance stratification intact instead of tying charging state to how +sunny the sampled day happened to be. + +Two consequences worth knowing. A configuration with a car doubles the number of plan +runs, so it takes roughly twice as long. And if even one session a day cannot get under +six hours (a very low charge rate against very high mileage), the tool still uses seven +sessions but logs a warning: the sessions run long, so the overflow effect — and +therefore Predbat's advantage — is understated. Instead of an Octopus URL you may give a fixed rate structure: From c8f8d3981295843af3219fb86bc82d69c72d257c Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 18:25:04 +0200 Subject: [PATCH 036/119] fix(annual): reset scenario-3 car overrides, fix doc heading order, add leg-independence test reset_sample_state() now clears the car_charging_* fields the with-car leg sets, instead of relying on every consumer being gated on num_cars. Fixes docs/annual-prediction.md's heading nesting so the tariff example no longer reads as part of the car-charging section, and adds an integration assertion proving the blended result equals f * with_car_leg + (1 - f) * standalone_no_car_leg field by field. Also hoists the car-session overflow warning to run once per run instead of once per sampled day, softens an inexact floating-point docstring claim, and corrects two other doc inaccuracies. --- apps/predbat/annual.py | 82 ++++++++++++++++--- apps/predbat/tests/test_annual_bootstrap.py | 19 ++++- apps/predbat/tests/test_annual_integration.py | 32 +++++++- docs/annual-prediction.md | 41 +++++----- 4 files changed, 142 insertions(+), 32 deletions(-) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 245b8e1a7..d0c76252d 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -406,7 +406,8 @@ def reset_sample_state(predbat): numbers stay plausible while becoming order-dependent. The list covers the accumulators, the previous plan, the manual overrides, the starting SOC, the full rate-derived family that ``calculate_plan`` seeds its best windows from, - and the field ``tests/test_single_debug.py`` documents as leaking between + the scenario-3 smart-car overrides ``_run_scenarios()`` sets on the with-car + leg, and the field ``tests/test_single_debug.py`` documents as leaking between debug cases (``dynamic_load_baseline``). Deliberately excluded: ``battery_rate_max_export`` is hardware-derived and @@ -464,6 +465,19 @@ def reset_sample_state(predbat): predbat.manual_load_adjust = {} predbat.savings_last_updated = None + # Scenario 3's smart-car overrides (_run_scenarios()). Reset here rather than relying on + # every consumer being gated on num_cars (prediction.py, plan.py) - that guarantee is + # fragile in a file that has already had several state-leak bugs. Safe to reset even + # though _run_scenarios() sets these on the with-car leg: prepare_sample() calls this + # function BEFORE that leg sets them, never after. + predbat.car_charging_planned = [False] + predbat.car_charging_limit = [0.0] + predbat.car_charging_soc = [0.0] + predbat.car_charging_rate = [DEFAULT_CAR_RATE_KW] + predbat.car_charging_battery_size = [50.0] + predbat.car_charging_plan_smart = [False] + predbat.car_charging_from_battery = False + def configure_offline_mode(predbat): """Apply the one-shot configuration choices that make sense only for an offline run. @@ -617,12 +631,29 @@ def car_charging_schedule(annual_kwh, car_rate_kw): caller is responsible for logging that the overflow this model is meant to capture is then understated. - Returns (sessions_per_week, session_kwh). ``sessions_per_week * session_kwh`` - always equals ``annual_kwh / 52.0`` exactly (division and its inverse), so summing - the week's sessions recovers the annual total. + Returns (sessions_per_week, session_kwh). ``sessions_per_week * session_kwh`` equals + ``annual_kwh / 52.0`` to within floating-point rounding (division and its inverse), + so summing the week's sessions recovers the annual total. + + Note a deliberate 52-vs-7 mismatch: a session is sized here as ``annual_kwh / 52.0`` + per week, but ``run_day()`` blends it back in at ``sessions_per_week / 7`` of each + sampled day - and 52 weeks * 7 days = 364, one short of a real (365 or 366 day) year. + So the car energy actually modelled over a year is ``annual_kwh * 365 / 364``, about + +0.27% high. Harmless at that size, but it is a real error hiding behind two numbers + that look interchangeable: "fixing" only one of the 52 here or the 7 in ``run_day()`` + (e.g. switching this to weeks-per-365-days) without the matching change to the other + would turn a harmless 0.27% into a much larger one. Keep the two divisors paired if + either is ever revisited. """ weekly_kwh = annual_kwh / 52.0 if weekly_kwh <= 0 or car_rate_kw <= 0: + # A configured car (annual_kwh > 0) with an unusable rate is silently dropped here: + # (0, 0.0) reads to a caller exactly like "no car configured" at all. Not reachable + # through _validate_load() today (car_rate_kw is validated greater than zero there), + # but a caller reaching this function some other way gets no signal that a real car + # was discarded. Not raised, since this guard is defensive rather than a real + # validation boundary - see car_charging_overflow_warning() below for the one + # warning this module does emit about the schedule it derives. return 0, 0.0 session_hours = weekly_kwh / car_rate_kw @@ -633,6 +664,27 @@ def car_charging_schedule(annual_kwh, car_rate_kw): return sessions_per_week, weekly_kwh / sessions_per_week +def car_charging_overflow_warning(car_charging_kwh, car_rate_kw): + """Return a warning message if the car's sessions cannot fit under the six-hour cap. + + This is a static property of ``car_charging_kwh``/``car_rate_kw`` alone, not of any + particular sampled day, so a caller should log it once per run (``AnnualPredictor.run()`` + does) rather than once per sampled day - the same condition would otherwise repeat + identically for every one of the roughly two dozen days a run samples. Returns None + when there is nothing to warn about, including when no car is configured at all. + """ + if car_charging_kwh <= 0: + return None + sessions_per_week, session_kwh = car_charging_schedule(car_charging_kwh, car_rate_kw) + if sessions_per_week >= MAX_SESSIONS_PER_WEEK and session_kwh > car_rate_kw * CAR_SESSION_MAX_HOURS + 1e-6: + return ( + "Annual: car charging needs {:.1f} kWh/week at {:.1f} kW, which cannot fit into {} sessions of {:.0f} hours or less; sessions are running long ({:.1f} hours), so the timer/Predbat overflow this model is meant to capture is understated".format( + sessions_per_week * session_kwh, car_rate_kw, MAX_SESSIONS_PER_WEEK, CAR_SESSION_MAX_HOURS, session_kwh / car_rate_kw + ) + ) + return None + + def build_step_data(predbat, pv_minute, pv_minute10): """Build the 5-minute step arrays the Prediction engine consumes. @@ -1028,17 +1080,19 @@ def run_day(predbat, config, weather, tariff, load_source, day, midnight_utc): if car_charging_kwh <= 0: return _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=0.0, car_rate_kw=car_rate_kw) + # The "sessions running long" overflow warning is a static property of the config + # (car_charging_kwh, car_rate_kw), not of this particular day, so it is emitted once + # per run by AnnualPredictor.run() (via car_charging_overflow_warning()) rather than + # here, where it would otherwise repeat once per sampled day. sessions_per_week, session_kwh = car_charging_schedule(car_charging_kwh, car_rate_kw) - if sessions_per_week >= MAX_SESSIONS_PER_WEEK and session_kwh > car_rate_kw * CAR_SESSION_MAX_HOURS + 1e-6: - predbat.log( - "Warn: Annual: {} car charging needs {:.1f} kWh/week at {:.1f} kW, which cannot fit into {} sessions of {:.0f} hours or less; sessions are running long ({:.1f} hours), so the timer/Predbat overflow this model is meant to capture is understated".format( - day, sessions_per_week * session_kwh, car_rate_kw, MAX_SESSIONS_PER_WEEK, CAR_SESSION_MAX_HOURS, session_kwh / car_rate_kw - ) - ) with_car = _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=session_kwh, car_rate_kw=car_rate_kw) without_car = _run_scenarios(predbat, config, weather, tariff, load_source, day, midnight_utc, car_kwh=0.0, car_rate_kw=car_rate_kw) + # sessions_per_week / 7, not / 52: this is a fraction of a WEEK (how many of its 7 days + # actually carry a session), independent of car_charging_schedule()'s own 52-weeks-per-year + # sizing above. See car_charging_schedule()'s docstring for the harmless ~+0.27% this + # 52-vs-7 pairing produces, and why the two must be changed together if either is. fraction = sessions_per_week / float(MAX_SESSIONS_PER_WEEK) return _blend_results(with_car, without_car, fraction) @@ -1159,6 +1213,14 @@ async def run(self, progress=None): self.caveats.append("self_consumed_kwh is approximate: when the battery exports grid-charged energy it is understated.") self.caveats.append("export_credit_p_estimate is money ALREADY included inside cost_p (which prices every export minute at its real rate); it is informational only - adding it to cost_p double-counts export income.") + # A static property of the config, not of any one sampled day, so this is checked and + # logged exactly once here rather than inside run_day(), which runs once per sampled + # day (roughly two dozen times per year at the default samples_per_month). + car_overflow_warning = car_charging_overflow_warning(self.config["load"].get("car_charging_kwh", 0.0), self.config["load"].get("car_rate_kw", DEFAULT_CAR_RATE_KW)) + if car_overflow_warning: + self.log("Warn: {}".format(car_overflow_warning)) + self.caveats.append(car_overflow_warning) + self.predbat = create_headless_predbat(self.work_dir, self.config["timezone"], self.log) self.load_source = await self._build_load_source() self.tariff = AnnualTariff(self.config["tariff"], log=self.log, predbat=self.predbat, storage=self.storage) diff --git a/apps/predbat/tests/test_annual_bootstrap.py b/apps/predbat/tests/test_annual_bootstrap.py index a3521f14d..2ae22aacd 100644 --- a/apps/predbat/tests/test_annual_bootstrap.py +++ b/apps/predbat/tests/test_annual_bootstrap.py @@ -14,7 +14,7 @@ import yaml -from annual import AnnualNullHA, apply_hardware, configure_offline_mode, create_headless_predbat, reset_sample_state, write_minimal_apps_yaml +from annual import DEFAULT_CAR_RATE_KW, AnnualNullHA, apply_hardware, configure_offline_mode, create_headless_predbat, reset_sample_state, write_minimal_apps_yaml from const import MINUTE_WATT @@ -156,6 +156,16 @@ def test_annual_bootstrap(my_predbat): my_predbat.rate_max = 6.0 my_predbat.rate_average = 7.0 + # Scenario 3's smart-car overrides (_run_scenarios() in annual.py), leaked here the way a + # previous sample's with-car leg would leave them. + my_predbat.car_charging_planned = [True] + my_predbat.car_charging_limit = [20.0] + my_predbat.car_charging_soc = [10.0] + my_predbat.car_charging_rate = [22.0] + my_predbat.car_charging_battery_size = [100.0] + my_predbat.car_charging_plan_smart = [True] + my_predbat.car_charging_from_battery = True + reset_sample_state(my_predbat) checks = [ @@ -187,6 +197,13 @@ def test_annual_bootstrap(my_predbat): ("rate_min", 0), ("rate_max", 0), ("rate_average", 0), + ("car_charging_planned", [False]), + ("car_charging_limit", [0.0]), + ("car_charging_soc", [0.0]), + ("car_charging_rate", [DEFAULT_CAR_RATE_KW]), + ("car_charging_battery_size", [50.0]), + ("car_charging_plan_smart", [False]), + ("car_charging_from_battery", False), ] for name, expected in checks: actual = getattr(my_predbat, name) diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py index fdbb81536..c3662812d 100644 --- a/apps/predbat/tests/test_annual_integration.py +++ b/apps/predbat/tests/test_annual_integration.py @@ -16,7 +16,7 @@ import pytz -from annual import DAY_MINUTES, PLAN_MINUTES, run_day, validate_config +from annual import DAY_MINUTES, MAX_SESSIONS_PER_WEEK, PLAN_MINUTES, SCENARIO_FIELDS, SCENARIO_KEYS, _blend_results, _run_scenarios, car_charging_schedule, run_day, validate_config from annual_load import SyntheticLoadProfile from tests.test_infra import reset_inverter @@ -242,6 +242,7 @@ def test_annual_integration(my_predbat): car_results = {} def _run_car_config(): + """Run the car-charging config for one day and record its blended result.""" car_results.update(run_one(my_predbat, car_config, weather, day)) plan_calls = _count_calculate_plan_calls(my_predbat, _run_car_config) @@ -257,4 +258,33 @@ def _run_car_config(): print(" ERROR: adding a car should raise the no-system import, got {} vs {}".format(car_results["no_pvbat"]["import_kwh"], results["no_pvbat"]["import_kwh"])) failed = True + print("Test: the blended result equals f * with-car leg + (1 - f) * standalone no-car leg") + # Proves two things at once: the blend arithmetic itself, and that the with-car leg + # does not contaminate the without-car leg it runs alongside (run_day() runs the + # with-car leg first, then the without-car leg, on the SAME predbat instance with no + # reset_inverter() between them - only reset_sample_state() separates them, exactly as + # reproduced below). If reset_sample_state() ever stopped clearing the scenario-3 car + # overrides (car_charging_planned, car_charging_limit, etc.), the without-car leg run + # here would silently inherit them and this comparison would drift outside tolerance. + car_annual_kwh = car_config["load"]["car_charging_kwh"] + car_rate_kw = car_config["load"]["car_rate_kw"] + sessions_per_week, session_kwh = car_charging_schedule(car_annual_kwh, car_rate_kw) + fraction = sessions_per_week / float(MAX_SESSIONS_PER_WEEK) + + reset_inverter(my_predbat) + midnight = pytz.utc.localize(datetime(day.year, day.month, day.day)) + car_load_source = SyntheticLoadProfile(annual_kwh=car_config["load"]["annual_kwh"], shape=car_config["load"]["shape"], year=car_config["year"]) + with_car_leg = _run_scenarios(my_predbat, car_config, weather, StubTariff(), car_load_source, day, midnight, car_kwh=session_kwh, car_rate_kw=car_rate_kw) + standalone_no_car_leg = _run_scenarios(my_predbat, car_config, weather, StubTariff(), car_load_source, day, midnight, car_kwh=0.0, car_rate_kw=car_rate_kw) + + expected_blend = _blend_results(with_car_leg, standalone_no_car_leg, fraction) + run_day_blend = run_one(my_predbat, car_config, weather, day) + for key in SCENARIO_KEYS: + for field in SCENARIO_FIELDS: + expected_value = expected_blend[key][field] + actual_value = run_day_blend[key][field] + if abs(expected_value - actual_value) > 1e-6: + print(" ERROR: blended {}.{} = {}, expected f * with_car_leg + (1 - f) * standalone_no_car_leg = {}".format(key, field, actual_value, expected_value)) + failed = True + return failed diff --git a/docs/annual-prediction.md b/docs/annual-prediction.md index 24180efa2..347a75779 100644 --- a/docs/annual-prediction.md +++ b/docs/annual-prediction.md @@ -105,9 +105,25 @@ effect on an Octopus load and is ignored there — there is no separately-tracke energy to apply a charging rate to. `car_rate_kw` is the charger's power, used to size both the dumb timer's charge window -(scenario 2) and the smart plan's charging rate (scenario 3): a smaller number (say 3.0 -for a granny charger) spreads the same energy over a longer window, while a larger one -(up to a three-phase charger's 22 kW) charges it faster. It must be greater than zero. +(scenarios 1 and 2, which share the same fixed timer) and the smart plan's charging rate +(scenario 3): a smaller number (say 3.0 for a granny charger) spreads the same energy over +a longer window, while a larger one (up to a three-phase charger's 22 kW) charges it +faster. It must be greater than zero. + +Instead of an Octopus URL you may give a fixed rate structure: + +```yaml + tariff: + rates_import: + - start: "00:30:00" + end: "05:30:00" + rate: 7.0 + - start: "05:30:00" + end: "00:30:00" + rate: 28.0 + rates_export: + - rate: 15.0 +``` ### How car charging is spread across the year @@ -137,21 +153,6 @@ six hours (a very low charge rate against very high mileage), the tool still use sessions but logs a warning: the sessions run long, so the overflow effect — and therefore Predbat's advantage — is understated. -Instead of an Octopus URL you may give a fixed rate structure: - -```yaml - tariff: - rates_import: - - start: "00:30:00" - end: "05:30:00" - rate: 7.0 - - start: "05:30:00" - end: "00:30:00" - rate: 28.0 - rates_export: - - rate: 15.0 -``` - ## Running it ```bash @@ -199,8 +200,8 @@ figure should not be trusted there. The top-level `annual` block sums the included months' scenarios and reports two savings figures: `pv_battery_vs_none_p` (PV and battery vs. no system) and `predbat_vs_baseline_p` (Predbat vs. the dumb timer baseline). If no month produced a -usable result, `annual.scenarios`, `annual.standing_charge_p` and `annual.savings` are -empty rather than a fabricated zero-cost year. +usable result, `annual.scenarios` and `annual.standing_charge_p` are `null` and +`annual.savings` is an empty object (`{}`), rather than a fabricated zero-cost year. The `caveats` list in the results document records anything that could affect how much to trust the numbers — a P10 fallback, a missing month's rate data, and the From 7bae6b41e850f82b4a135f621c2a575d4897146d Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 18:37:41 +0200 Subject: [PATCH 037/119] test(annual): compute the expected blend independently of _blend_results The assertion built its expected value by calling the same function it was testing, so a bug inside _blend_results would have cancelled out across both sides and passed. Use explicit arithmetic instead, and correct the comment: this covers run_day's wiring, not leg contamination, which the manual replay cannot detect because it runs the same path in the same order. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/tests/test_annual_integration.py | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/apps/predbat/tests/test_annual_integration.py b/apps/predbat/tests/test_annual_integration.py index c3662812d..dd5117ee0 100644 --- a/apps/predbat/tests/test_annual_integration.py +++ b/apps/predbat/tests/test_annual_integration.py @@ -16,7 +16,7 @@ import pytz -from annual import DAY_MINUTES, MAX_SESSIONS_PER_WEEK, PLAN_MINUTES, SCENARIO_FIELDS, SCENARIO_KEYS, _blend_results, _run_scenarios, car_charging_schedule, run_day, validate_config +from annual import DAY_MINUTES, MAX_SESSIONS_PER_WEEK, PLAN_MINUTES, SCENARIO_FIELDS, SCENARIO_KEYS, _run_scenarios, car_charging_schedule, run_day, validate_config from annual_load import SyntheticLoadProfile from tests.test_infra import reset_inverter @@ -259,13 +259,18 @@ def _run_car_config(): failed = True print("Test: the blended result equals f * with-car leg + (1 - f) * standalone no-car leg") - # Proves two things at once: the blend arithmetic itself, and that the with-car leg - # does not contaminate the without-car leg it runs alongside (run_day() runs the - # with-car leg first, then the without-car leg, on the SAME predbat instance with no - # reset_inverter() between them - only reset_sample_state() separates them, exactly as - # reproduced below). If reset_sample_state() ever stopped clearing the scenario-3 car - # overrides (car_charging_planned, car_charging_limit, etc.), the without-car leg run - # here would silently inherit them and this comparison would drift outside tolerance. + # What this covers: run_day()'s WIRING - that it feeds the two legs into the blend in + # the right order, with the right fraction, and mutates nothing else along the way. + # The expected value below is computed with explicit arithmetic rather than by calling + # _blend_results(), so a bug inside _blend_results() cannot cancel itself out across + # both sides of the comparison. + # + # What it does NOT cover, despite the tempting reading: leg contamination. The manual + # replay below runs the same two legs through the same code path in the same order as + # run_day() does, so a reset_sample_state() regression would corrupt both sides + # identically and stay inside tolerance. Regression cover for the car-field reset is + # the state-based assertion in test_annual_bootstrap.py; blend arithmetic on fabricated + # inputs is covered in test_annual_scenarios.py. car_annual_kwh = car_config["load"]["car_charging_kwh"] car_rate_kw = car_config["load"]["car_rate_kw"] sessions_per_week, session_kwh = car_charging_schedule(car_annual_kwh, car_rate_kw) @@ -277,11 +282,10 @@ def _run_car_config(): with_car_leg = _run_scenarios(my_predbat, car_config, weather, StubTariff(), car_load_source, day, midnight, car_kwh=session_kwh, car_rate_kw=car_rate_kw) standalone_no_car_leg = _run_scenarios(my_predbat, car_config, weather, StubTariff(), car_load_source, day, midnight, car_kwh=0.0, car_rate_kw=car_rate_kw) - expected_blend = _blend_results(with_car_leg, standalone_no_car_leg, fraction) run_day_blend = run_one(my_predbat, car_config, weather, day) for key in SCENARIO_KEYS: for field in SCENARIO_FIELDS: - expected_value = expected_blend[key][field] + expected_value = fraction * with_car_leg[key][field] + (1.0 - fraction) * standalone_no_car_leg[key][field] actual_value = run_day_blend[key][field] if abs(expected_value - actual_value) > 1e-6: print(" ERROR: blended {}.{} = {}, expected f * with_car_leg + (1 - f) * standalone_no_car_leg = {}".format(key, field, actual_value, expected_value)) From 12e7a57ff8a7b004268a64b874b08b8d396e9799 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 19:33:08 +0200 Subject: [PATCH 038/119] fix(annual): clamp baseline car window to its billed day, tighten year bound, keep --quiet warnings visible, dedupe HTTP fetchers Final pre-merge review fixes for the annual prediction tool: - timer_charge_window() could place a baseline car session partly outside DAY_MINUTES when the cheapest band fell late in the day, so _billed_result() billed the baselines for only part of the session while scenario 3 always bills the full amount - phantom baseline saving that could invert a month's predbat_vs_baseline_p. Now clamps the window to end within its own day whenever the session fits in a day at all. - annual.year accepted the current, still-in-progress year. Open-Meteo answers a mid-year request with short-but-consistent data that looks complete, and it was cached with no expiry - permanently pinning the remaining months as unavailable. Tightened the bound to the most recently completed year, per the tool's own documented contract. - --quiet silenced AnnualPredictor's warnings (P10 fallback, missing rates, failed days, car shortfalls) along with progress output, breaking the "failures are visible, never silent" contract. Now only progress output is suppressed. - Extracted the three near-identical default JSON fetchers in annual_weather.py, annual_tariff.py and annual_load.py into a shared annual_http.fetch_json(), preserving each caller's timeout/headers/log wording and OctopusConsumptionLoadProfile's shared-session behaviour. Also documents that the without-Predbat baseline is a more pessimistic comparator on half-hourly tariffs (Agile) than on banded ones (Economy 7, Cosy, Flux), and reconciles the design spec's PredBat-touching claim with annual_tariff.py's legitimate use of resolve_arg()/basic_rates(). Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual.py | 21 +++++++- apps/predbat/annual_cli.py | 6 ++- apps/predbat/annual_http.py | 48 +++++++++++++++++++ apps/predbat/annual_load.py | 11 +---- apps/predbat/annual_tariff.py | 13 +---- apps/predbat/annual_weather.py | 14 +----- apps/predbat/tests/test_annual_cli.py | 48 +++++++++++++++++++ apps/predbat/tests/test_annual_scenarios.py | 24 ++++++++++ docs/annual-prediction.md | 26 ++++++++-- ...026-07-25-annual-prediction-tool-design.md | 13 +++-- 10 files changed, 183 insertions(+), 41 deletions(-) create mode 100644 apps/predbat/annual_http.py diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index d0c76252d..de426430a 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -217,7 +217,12 @@ def validate_config(config, today=None): if today is None: today = date.today() - year = _require_number(raw.get("year", today.year - 1), "annual.year", minimum=MINIMUM_YEAR, maximum=today.year, integer=True) + # Capped at the most recent COMPLETE calendar year, not the current (in-progress) one: + # Open-Meteo answers a mid-year request with short but internally-consistent arrays, so + # _payload_problem() cannot tell a truncated current-year download from a genuinely + # complete one, and it gets cached with no expiry - permanently pinning the remaining + # months as "unavailable" until the work dir is deleted by hand. See annual_weather.py. + year = _require_number(raw.get("year", today.year - 1), "annual.year", minimum=MINIMUM_YEAR, maximum=today.year - 1, integer=True) return { "location": dict(location), @@ -752,6 +757,20 @@ def timer_charge_window(rate_import, car_kwh, car_rate_kw): best_start = start start = None aligned_start = base + (best_start // PREDICT_STEP) * PREDICT_STEP + + # _billed_result only costs minutes < DAY_MINUTES of the billed day (day_offset 0), so a + # window that runs past its own day's boundary gets only partially billed there: the + # baselines (scenarios 1/2) would then be charged for less car energy than scenario 3, + # which always bills the full session, making Predbat's saving look bigger than it is (or, + # on a day whose cheap band happens to run low, even negative). Pull the start back so the + # window still ends inside its own day whenever the session fits in a day at all. When + # minutes_needed itself exceeds a whole day, day_end - minutes_needed is before the day + # even starts, so clamp to the day's start instead of producing a negative offset - the + # window still overflows into the next day, which is unavoidable for a session that long. + day_end = base + DAY_MINUTES + if aligned_start + minutes_needed > day_end: + aligned_start = max(base, day_end - minutes_needed) + windows.append({"start": aligned_start, "end": aligned_start + minutes_needed}) return windows diff --git a/apps/predbat/annual_cli.py b/apps/predbat/annual_cli.py index 999e8c74a..8e5064c4d 100644 --- a/apps/predbat/annual_cli.py +++ b/apps/predbat/annual_cli.py @@ -139,7 +139,11 @@ def main(argv=None): storage = StorageLocalFiles(args.work_dir, print) try: - predictor = AnnualPredictor(config, log=print if not args.quiet else lambda *a, **k: None, storage=storage, work_dir=args.work_dir) + # --quiet suppresses only the per-month progress lines (make_progress() below), never + # the warnings AnnualPredictor.log emits: P10 fallbacks, missing rate data, failed + # sample days and car-charging shortfalls must stay visible even in a quiet run, per + # the "failures are visible, never silent" contract. + predictor = AnnualPredictor(config, log=print, storage=storage, work_dir=args.work_dir) results = asyncio.run(predictor.run(progress=make_progress(args.quiet))) except AnnualConfigError as error: sys.stderr.write("Config error: {}\n".format(error)) diff --git a/apps/predbat/annual_http.py b/apps/predbat/annual_http.py new file mode 100644 index 000000000..3314ed733 --- /dev/null +++ b/apps/predbat/annual_http.py @@ -0,0 +1,48 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Shared JSON-over-HTTP helper for the annual prediction tool's default fetchers. + +``annual_weather.py``, ``annual_tariff.py`` and ``annual_load.py`` each had their own +default JSON fetcher, differing only in timeout, headers and the module-specific prefix +on the warning log line. A third near-copy (this module collapses the first two plus +``OctopusConsumptionLoadProfile``'s) was the trigger to extract one shared implementation +instead of a fourth review flagging the same duplication again. +""" + +import aiohttp + + +async def fetch_json(url, log, log_prefix, headers, timeout_seconds, session=None): + """Download and decode one JSON document, returning None on any failure. + + ``log_prefix`` names the caller in the warning line (e.g. "Open-Meteo request", + "Octopus rate request", "Octopus consumption request"), so the three callers keep + their own distinguishable log wording despite sharing this implementation. + + ``session`` is optional: ``OctopusConsumptionLoadProfile`` passes one in so a single + TCP connection is reused across its meter-resolution call and its paginated + consumption downloads, exactly as it did before this helper was extracted. When + omitted, a fresh session is opened and closed around this one request, matching what + the weather and tariff fetchers did on their own. + """ + + async def _request(active_session): + """Issue the GET on the given session and return its decoded JSON, or None on a bad status.""" + async with active_session.get(url, headers=headers, timeout=aiohttp.ClientTimeout(total=timeout_seconds)) as response: + if response.status not in [200, 201]: + log("Warn: Annual: {} to {} returned {}".format(log_prefix, url, response.status)) + return None + return await response.json() + + try: + if session is not None: + return await _request(session) + async with aiohttp.ClientSession() as new_session: + return await _request(new_session) + except (aiohttp.ClientError, ValueError, TimeoutError) as error: + log("Warn: Annual: {} to {} failed: {}".format(log_prefix, url, error)) + return None diff --git a/apps/predbat/annual_load.py b/apps/predbat/annual_load.py index 307092f47..9bec53985 100644 --- a/apps/predbat/annual_load.py +++ b/apps/predbat/annual_load.py @@ -17,6 +17,7 @@ import aiohttp +from annual_http import fetch_json from annual_profiles import DAY_BAND_SLOTS, MONTH_WEIGHTS, NIGHT_BAND_SLOTS, SHAPE_TILT_FRACTION, half_hour_shape MINUTES_PER_DAY = 24 * 60 @@ -210,15 +211,7 @@ def _auth_header(self): async def _get_json(self, session, url): """Fetch and decode one JSON page, returning None on any failure.""" - try: - async with session.get(url, headers=self._auth_header(), timeout=aiohttp.ClientTimeout(total=30)) as response: - if response.status not in [200, 201]: - self.log("Warn: Annual: Octopus consumption request to {} returned {}".format(url, response.status)) - return None - return await response.json() - except (aiohttp.ClientError, ValueError, TimeoutError) as error: - self.log("Warn: Annual: Octopus consumption request to {} failed: {}".format(url, error)) - return None + return await fetch_json(url, self.log, "Octopus consumption request", self._auth_header(), 30, session=session) async def resolve_meter(self, session): """Resolve the account's MPAN and meter serial. Returns True on success.""" diff --git a/apps/predbat/annual_tariff.py b/apps/predbat/annual_tariff.py index d12934e75..d61619662 100644 --- a/apps/predbat/annual_tariff.py +++ b/apps/predbat/annual_tariff.py @@ -14,9 +14,9 @@ import calendar from datetime import datetime, timedelta -import aiohttp import pytz +from annual_http import fetch_json from utils import minute_data MINUTES_PER_DAY = 24 * 60 @@ -79,16 +79,7 @@ def _resolve_url(self, url, name, dno_region=None): async def _default_fetch_json(self, url): """Download and decode one JSON document, returning None on any failure.""" - try: - async with aiohttp.ClientSession() as session: - async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=60)) as response: - if response.status not in [200, 201]: - self.log("Warn: Annual: Octopus rate request to {} returned {}".format(url, response.status)) - return None - return await response.json() - except (aiohttp.ClientError, ValueError, TimeoutError) as error: - self.log("Warn: Annual: Octopus rate request to {} failed: {}".format(url, error)) - return None + return await fetch_json(url, self.log, "Octopus rate request", {"accept": "application/json", "user-agent": "predbat/1.0"}, 60) async def _download_octopus(self, base_url, start_utc, end_utc, cache_key): """Download every page of Octopus rates for a date range, returning raw result rows.""" diff --git a/apps/predbat/annual_weather.py b/apps/predbat/annual_weather.py index 1ccaabf37..87feca6b5 100644 --- a/apps/predbat/annual_weather.py +++ b/apps/predbat/annual_weather.py @@ -15,8 +15,7 @@ import math from datetime import timedelta -import aiohttp - +from annual_http import fetch_json from solar_model import convert_azimuth, gti_hourly_to_period_kwh ARCHIVE_URL = "https://archive-api.open-meteo.com/v1/archive" @@ -119,16 +118,7 @@ def __init__(self, arrays, latitude, longitude, log, storage=None, p10_fallback= async def _default_fetch_json(self, url): """Download and decode one JSON document, returning None on any failure.""" - try: - async with aiohttp.ClientSession() as session: - async with session.get(url, headers={"accept": "application/json", "user-agent": "predbat/1.0"}, timeout=aiohttp.ClientTimeout(total=120)) as response: - if response.status not in [200, 201]: - self.log("Warn: Annual: Open-Meteo request to {} returned {}".format(url, response.status)) - return None - return await response.json() - except (aiohttp.ClientError, ValueError, TimeoutError) as error: - self.log("Warn: Annual: Open-Meteo request to {} failed: {}".format(url, error)) - return None + return await fetch_json(url, self.log, "Open-Meteo request", {"accept": "application/json", "user-agent": "predbat/1.0"}, 120) def _build_url(self, base, array, year): """Build one Open-Meteo request URL for a single array and calendar year. diff --git a/apps/predbat/tests/test_annual_cli.py b/apps/predbat/tests/test_annual_cli.py index d930b3439..fdcb2d2b9 100644 --- a/apps/predbat/tests/test_annual_cli.py +++ b/apps/predbat/tests/test_annual_cli.py @@ -9,9 +9,36 @@ """Tests for the annual prediction command line output.""" +import io +import os +import tempfile +from contextlib import redirect_stderr, redirect_stdout + +import annual_cli from annual_cli import format_table +class _StubPredictor: + """Stands in for AnnualPredictor so main()'s argument wiring can be tested in isolation. + + Records the ``log`` callable it was constructed with rather than doing any real work, + so a test can assert what main() actually passes through under --quiet without needing + a real config, the network, or a headless PredBat instance. + """ + + captured_log = None + + def __init__(self, config, log=None, storage=None, work_dir=None): + """Record the log callable and discard everything else.""" + _StubPredictor.captured_log = log + + async def run(self, progress=None): + """Report one fake progress step (if asked) and return canned results.""" + if progress: + progress(0, 1, "stub") + return sample_results() + + def sample_results(): """Return a small results document covering an ok month and an unavailable one.""" scenarios = { @@ -166,4 +193,25 @@ def test_annual_cli(my_predbat): print(" ERROR: the table must not render the missing annual total as a zero cost, got:\n{}".format(empty_table)) failed = True + print("Test: --quiet suppresses only progress output, never AnnualPredictor's warnings") + # Regression guard: --quiet used to pass log=lambda *a, **k: None, silencing the + # P10-fallback/missing-rate/failed-day/car-shortfall warnings along with progress, which + # broke the "failures are visible, never silent" contract. main() must still pass + # log=print through under --quiet, suppressing only make_progress()'s per-month lines. + original_predictor = annual_cli.AnnualPredictor + annual_cli.AnnualPredictor = _StubPredictor + with tempfile.TemporaryDirectory() as work_dir: + config_path = os.path.join(work_dir, "annual.yaml") + with open(config_path, "w") as handle: + handle.write("annual: {}\n") + try: + with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): + annual_cli.main(["--config", config_path, "--work-dir", os.path.join(work_dir, "work"), "--quiet"]) + finally: + annual_cli.AnnualPredictor = original_predictor + + if _StubPredictor.captured_log is not print: + print(" ERROR: --quiet should still construct AnnualPredictor with log=print, got {}".format(_StubPredictor.captured_log)) + failed = True + return failed diff --git a/apps/predbat/tests/test_annual_scenarios.py b/apps/predbat/tests/test_annual_scenarios.py index 45e511de4..013781671 100644 --- a/apps/predbat/tests/test_annual_scenarios.py +++ b/apps/predbat/tests/test_annual_scenarios.py @@ -78,6 +78,30 @@ def test_annual_scenarios(my_predbat): print(" ERROR: the window start should be aligned to a 5 minute boundary, got {}".format(quantised_window[0]["start"])) failed = True + print("Test: a cheapest band late in the day is pulled back so the billed day still gets the FULL car session") + # Regression guard for the bug where a day-0 window could run past DAY_MINUTES: the + # cheapest half-hour here is at 23:00 (minute 1380), and a 34 kWh session at 7.4 kW needs + # about 4.6 hours - far more than the 60 minutes left in the day from 23:00. Without + # clamping, the window would end at 1380 + ~280 = ~1660, so _billed_result (which only + # costs minutes < DAY_MINUTES) would bill the baselines for only the ~60 minutes that fall + # before midnight while scenario 3 (with_predbat) is always billed for the full session - + # phantom baseline saving that can invert a month's headline predbat_vs_baseline_p. + late_rates = flat_rates(cheap_start=1380, cheap_end=1410, cheap_rate=7.0, peak_rate=30.0) + late_car_kwh = 34.0 + late_window = timer_charge_window(late_rates, car_kwh=late_car_kwh, car_rate_kw=7.4) + day_zero_window = late_window[0] + if day_zero_window["start"] < 0: + print(" ERROR: the day-0 window start must never go negative, got {}".format(day_zero_window["start"])) + failed = True + if day_zero_window["end"] > DAY_MINUTES: + print(" ERROR: the day-0 window must end within the billed day (< {}), got end={} (start={})".format(DAY_MINUTES, day_zero_window["end"], day_zero_window["start"])) + failed = True + late_baseline_load = add_car_to_load({minute: 0.0 for minute in range(DAY_MINUTES + 1)}, late_window, car_kwh=late_car_kwh) + billed_day_energy = late_baseline_load[DAY_MINUTES] + if abs(billed_day_energy - late_car_kwh) > 1e-6: + print(" ERROR: the billed day should receive the full {} kWh session (matching what scenario 3 is always billed for), got {} kWh".format(late_car_kwh, billed_day_energy)) + failed = True + print("Test: add_car_to_load inserts the car's full energy on EACH day, not split across the plan, and stays cumulative") base = {minute: 0.01 * minute for minute in range(PLAN_MINUTES + 1)} with_car = add_car_to_load(base, window, car_kwh=14.8) diff --git a/docs/annual-prediction.md b/docs/annual-prediction.md index 347a75779..cb7bb00fa 100644 --- a/docs/annual-prediction.md +++ b/docs/annual-prediction.md @@ -85,7 +85,10 @@ rather than silently producing a nonsense result: `kwp`, `size_kwh`, `inverter_k `charge_rate_kw` and `discharge_rate_kw` must all be greater than zero; `efficiency` and `pv10_derate_fallback` must be greater than zero and at most one; `samples_per_month` must be a whole number of at least one; and `year` must be between 1940 (the start of -the Open-Meteo ERA5 archive) and the current year. +the Open-Meteo ERA5 archive) and the most recently completed calendar year - the current, +still-in-progress year is rejected, since Open-Meteo answers a mid-year request with +short but internally-consistent data that looks complete, and it would then be cached +permanently as if it were. Instead of `annual_kwh`, `shape` and `car_charging_kwh` you may supply real consumption: @@ -161,9 +164,11 @@ python3 annual_cli.py --config annual.yaml --out results.json ``` Other options: `--work-dir` (default `./annual_work`) sets where the headless Predbat -instance and the download cache live, and `--quiet` suppresses the per-month progress -lines written to stderr. A human-readable table is always printed to stdout; `--out` -additionally writes the full results document as JSON. +instance and the download cache live, and `--quiet` suppresses only the per-month +progress lines written to stderr. Warnings - a P10 fallback, missing rate data, a failed +sample day, a car-charging shortfall - are never suppressed, `--quiet` or not: failures +stay visible. A human-readable table is always printed to stdout; `--out` additionally +writes the full results document as JSON. A run takes roughly one to three minutes with the default two samples per month: 24 plan calculations (12 months × 2 samples) plus the weather, tariff and (if configured) @@ -203,6 +208,19 @@ figures: `pv_battery_vs_none_p` (PV and battery vs. no system) and usable result, `annual.scenarios` and `annual.standing_charge_p` are `null` and `annual.savings` is an empty object (`{}`), rather than a fabricated zero-cost year. +**`predbat_vs_baseline_p` is not equally hard to beat on every tariff.** The "without +Predbat" baseline charges in the cheapest contiguous band of the day (mirroring +Predbat's own `calculate_yesterday()` savings baseline, deliberately kept consistent with +it rather than made tariff-aware here). On a banded tariff such as Economy 7, Cosy or +Flux, where the cheap rate holds for several contiguous hours, that band is almost the +whole cheap period, so the baseline is a strong comparator and Predbat's measured saving +mostly reflects genuine optimisation. On a half-hourly tariff such as Agile, where the +day's single cheapest rate is often just one 30 minute slot, the same rule yields a much +narrower baseline window - a comparator that is easier to beat - so `predbat_vs_baseline_p` +reads more flattering on Agile than the equivalent system would on a banded tariff. This +is a property of the comparator, not of Predbat performing differently; treat cross-tariff +comparisons of this figure with that in mind. + The `caveats` list in the results document records anything that could affect how much to trust the numbers — a P10 fallback, a missing month's rate data, and the `export_credit_p_estimate`/`self_consumed_kwh` notes above among them — and is worth diff --git a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md index 74f39622b..ef9f65351 100644 --- a/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md +++ b/docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md @@ -57,12 +57,19 @@ convention (no subpackages exist besides `tests/` and `config/`). | `annual_tariff.py` | Per-date rate resolution: Octopus URL or basic rates | `basic_rates()` | | `annual_cli.py` | Argument parsing, progress output, JSON and table writing | `annual` | -The boundary that matters: **`annual.py` performs no HTTP, and -`annual_weather.py` / `annual_tariff.py` never touch `PredBat`.** That keeps the -web UI's future job to "build a config dict, call `AnnualPredictor.run()`, +The boundary that matters: **`annual.py` performs no HTTP.** All network access +lives in `annual_weather.py`, `annual_tariff.py` and `annual_load.py`. That keeps +the web UI's future job to "build a config dict, call `AnnualPredictor.run()`, render the returned JSON", and lets every module be tested against fixtures rather than the network. +`annual_weather.py` never touches `PredBat`. `annual_tariff.py` does hold a +`PredBat` reference, legitimately: it calls `resolve_arg()` to expand a +templated Octopus URL and `basic_rates()` to expand a static rate structure - +both listed as its dependency above - rather than reimplementing either. That +reference is used only for those two calls, never to drive a plan or touch +live state, so it does not compromise the no-HTTP boundary above. + HTTP responses are cached through the Storage component's `fetch_cached()` (per CLAUDE.md, never direct file access): one cached blob per array-year per source for weather — actuals and forecast are separate entries — and one per From 9b183fc0731856e91fe00668901f77c35081df73 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 19:40:39 +0200 Subject: [PATCH 039/119] fix(annual): surface the baseline-comparator caveat in the results JSON The docs explained that the without-Predbat baseline charges in the cheapest contiguous band, making it a harsher comparator on Agile than on a banded tariff, but the caveats list the results document carries did not. That list is what a JSON consumer reads, which is the whole reason it exists. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index de426430a..73af66413 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -1231,6 +1231,9 @@ async def run(self, progress=None): self.caveats.append("The forecast-versus-ERA5 gap includes systematic model bias as well as forecast error, so measured solar uncertainty is slightly overstated.") self.caveats.append("self_consumed_kwh is approximate: when the battery exports grid-charged energy it is understated.") self.caveats.append("export_credit_p_estimate is money ALREADY included inside cost_p (which prices every export minute at its real rate); it is informational only - adding it to cost_p double-counts export income.") + self.caveats.append( + "The without_predbat baseline charges in the single cheapest contiguous band of each day, mirroring Predbat's own savings baseline. On a half-hourly tariff such as Agile the cheapest band is often one 30 minute slot, so the baseline is a more pessimistic comparator there than on a banded tariff (Economy 7, Cosy, Flux) where it covers the whole cheap period. Compare predbat_vs_baseline_p across tariffs with that in mind." + ) # A static property of the config, not of any one sampled day, so this is checked and # logged exactly once here rather than inside run_day(), which runs once per sampled From 576891c41e317c68da951062ac7d23a9e66a4ed2 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 20:04:46 +0200 Subject: [PATCH 040/119] docs: design for the Annual prediction web UI Adds an Annual tab to the Predbat web interface: a form that prefills from the live instance where it can, a tariff dropdown built from the Compare template merged with the user's own compare_list, a subprocess run with a real progress bar, and results as annual totals plus a grouped monthly bar chart. The run goes in a subprocess because the planning inside AnnualPredictor is minutes of synchronous CPU work that would otherwise freeze the web interface and the five-minute optimiser loop, and because create_headless_predbat mutates process-global state that would race. The tab must work against a completely unconfigured Predbat - that is the prospective-buyer path, and eventually the unregistered Predbat.com path - so prefill is best-effort per field and the defaults describe a typical UK system. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-07-26-annual-web-ui-design.md | 268 ++++++++++++++++++ 1 file changed, 268 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-26-annual-web-ui-design.md diff --git a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md new file mode 100644 index 000000000..576be1252 --- /dev/null +++ b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md @@ -0,0 +1,268 @@ +# Annual Prediction Web UI — Design + +Date: 2026-07-26 +Status: Approved (design review complete) + +## Goal + +Add an **Annual** tab to the Predbat web interface that lets a user fill in +their home's details, run the annual prediction engine, watch its progress, and +read the result as annual totals plus a month-by-month chart. + +This is the UI half of the annual prediction tool. The engine — `annual.py`, +`annual_weather.py`, `annual_load.py`, `annual_profiles.py`, `annual_tariff.py`, +`annual_cli.py` — already exists and is specified in +`docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md`. This spec +adds no modelling; it exposes what is there. + +## Scope decisions (agreed) + +- **The run happens in a subprocess**, not in the web server's event loop. +- **Form values prefill from the live Predbat instance where possible**, and are + saved to a config file the CLI consumes directly. +- **Common fields up front, advanced collapsed.** +- **Monthly results as grouped bars**, three per month. +- **Self-hosted only for now** — no job queue, no tenancy, no quotas. +- **It must work with Predbat entirely unconfigured**, and defaults to a typical + UK system so a visitor can run it immediately. +- **A tariff dropdown**, built from a curated catalogue merged with the user's + own `compare_list` if they have one. + +## Why a subprocess + +`AnnualPredictor.run()` is `async`, but the planning inside it is one to three +minutes of *synchronous* CPU work — two to six with a car, since a car config +plans each sampled day twice. Awaiting that from aiohttp would freeze the whole +Predbat web interface, and the five-minute optimiser loop shares that event +loop. + +A subprocess also sidesteps a concrete hazard: `create_headless_predbat()` +mutates `os.environ["PREDBAT_APPS_FILE"]` and writes an `apps.yaml` into its +work directory. Both are process-global, so an in-process run would race with +the live instance and with any second run. + +The boundary pays a second dividend. Because the child builds its own headless +`PredBat` from a minimal `apps.yaml`, the **engine needs nothing from the user's +configuration**. Only the form's prefill touches live state, and that is +best-effort — which is what makes the unconfigured case work at all. + +## Architecture + +| File | Responsibility | +|---|---| +| `apps/predbat/tariff_catalogue.py` | **New.** The curated tariff list as data, plus the mapping from Compare's key names to the annual engine's. No logic beyond lookup and merge. | +| `apps/predbat/web_annual.py` | **New.** The tab: form rendering, config load/save, results rendering. | +| `apps/predbat/annual_job.py` | **New.** Subprocess lifecycle only — spawn, parse progress, track state, cancel, reap. No HTML. | +| `apps/predbat/web.py` | **Modify.** Five route registrations delegating to `web_annual.py`. | +| `apps/predbat/web_helper.py` | **Modify.** One nav link, alongside the others at the `Compare` block. | +| `apps/predbat/annual_cli.py` | **Modify.** Add `--progress-json` for machine-readable progress. | + +`web.py` is already 5,730 lines and `web_helper.py` 8,903, so the tab gets its +own module rather than growing either — following the `web_metrics_dashboard.py` +precedent. + +**The split that matters:** `annual_job.py` knows nothing about HTML and +`web_annual.py` knows nothing about process handling. The job control can then +be tested by driving a stub process with no aiohttp request in sight, which is +where the bugs will be. + +### Routes + +| Route | Purpose | +|---|---| +| `GET /annual` | The form, plus the last run's results if any | +| `POST /annual` | Save the config without running | +| `POST /annual_run` | Validate, save, spawn | +| `GET /annual_status` | JSON, polled once a second | +| `POST /annual_cancel` | Terminate a running job | + +### Persistence + +Two files in `config_root`, alongside `comparisons.yaml`: + +- `annual.yaml` — the config the form edits, handed to the subprocess as + `--config`. Hand-editable, survives restarts. +- `annual_results.json` — the last completed run, so revisiting the tab shows + results with a timestamp rather than an empty page. + +### Progress protocol + +`annual_cli.py` currently writes progress to stderr as `[3/12] Month 03/2025`. +That is parseable, but parsing prose couples the parent to wording that will be +reworded. `--progress-json` emits one JSON object per line instead: + +```json +{"completed": 3, "total": 12, "message": "Month 03/2025"} +``` + +The human-readable form stays the default so the CLI remains pleasant to use by +hand. + +## The form + +Six groups: **Location**, **Solar**, **Battery**, **Load**, **Tariff**, and a +collapsed **Advanced** holding `year`, `samples_per_month`, +`pv10_derate_fallback`, per-array `efficiency`, and the fixed-rate-band editor. + +### Prefill is best-effort, field by field + +Each value is read from the live instance where available and falls back to a +typical-UK default otherwise, **independently**. A half-configured Predbat gets +its real battery size alongside example solar, rather than all-or-nothing. + +When the live instance supplied neither a battery nor a solar array — the two +that signal a configured system — a banner says so: + +> Predbat isn't configured yet — these are example values, edit them to match +> your home. + +Without that line the defaults could be mistaken for a reading of the visitor's +actual system, which would make the result look authoritative when it is +illustrative. + +### The unconfigured case is a first-class requirement + +The tab must render, validate and run against a Predbat with nothing set up — +no inverter, no tariff, no meaningful `apps.yaml`. This is not a graceful +degradation afterthought: it is the path a prospective buyer takes, and +eventually the path an unregistered Predbat.com visitor takes. + +Defaults for that visitor describe a plausible system they might buy — around +5 kWp of solar, a 9.5 kWh battery on a 5 kW hybrid inverter, 3,800 kWh/year of +flat load, no car, a price-cap tariff — so Run works immediately and tweaking +follows. + +### Load + +A radio pair mirroring the engine's own exclusivity rule: + +- *Enter my usage* — annual kWh, day/night/flat shape, car kWh, charger kW +- *Import from Octopus* — API key, account ID + +Choosing one greys the other. The engine already rejects both being set, because +the Octopus consumption series contains any car charging; the UI expresses a +constraint that exists rather than inventing one. + +### Tariff + +A dropdown built from `tariff_catalogue.py` merged with the user's `compare_list` +if configured. Selecting an entry **fills in** the import and export URL fields, +which stay visible and editable — so it is obvious what was chosen and the +custom path still works. A "Custom…" entry leaves them blank. + +`dno_region` sits beside it and is required whenever the chosen URL contains +`{dno_region}`. The engine rejects that combination up front; the form should +catch it before spending minutes to fail. + +### The catalogue + +The ~14 tariffs currently living as a commented-out `compare_list` template in +`apps/predbat/config/apps.yaml` — Agile, Go, Intelligent Go, Flux, Cosy, Snug, +Intelligent Flux, Eon Next Drive, price cap with SEG, and their export pairings. + +Those entries use Compare's key names, so the catalogue owns the mapping: + +| Compare | Annual engine | +|---|---| +| `rates_import_octopus_url` | `import_octopus_url` | +| `rates_export_octopus_url` | `export_octopus_url` | +| `rates_import` / `rates_export` | unchanged | + +Extracting this as its own module means Compare could later read the same source +instead of every user hand-copying a commented block into `apps.yaml`. **Compare +is not changed by this work** — the extraction just stops a second copy being +created. + +## Running + +`POST /annual_run` saves `annual.yaml`, validates it through the engine's own +`validate_config`, and only then spawns: + +``` +python3 annual_cli.py --config /annual.yaml \ + --out /annual_results.json \ + --progress-json +``` + +`annual_job.py` holds a single `AnnualJob`: state (`idle` / `running` / +`complete` / `failed` / `cancelled`), progress counters, start time, last error, +and the process handle. One run at a time — a second Run is refused rather than +spawning a competitor for the same CPU. + +The page polls `GET /annual_status` once a second for +`{state, completed, total, message, elapsed}`. Because the job is server-side, +navigating away and back shows a run still in progress. Cancel sends `SIGTERM`, +then `SIGKILL` if the child does not exit. + +### Two limitations, stated rather than hidden + +- **A Predbat restart orphans a running child.** Job state resets to idle and + the tab reports that the last run did not complete, rather than showing a + progress bar stuck forever. +- **Validation happens twice.** The browser checks for immediate feedback; + `validate_config` inside the subprocess is the authority. The form's copy is a + convenience and never the gate. + +## Results + +Three things, top to bottom. + +**Annual totals** — the three scenario costs, then the two figures people came +for: what PV+battery saves over nothing, and what Predbat adds over a dumb +battery. Standing charge is shown separately, since it is identical across +scenarios and folding it in would dilute the comparison. + +**The monthly chart** — twelve months, three grouped bars each, in ApexCharts to +match the existing charts and their dark-mode styling. + +**A monthly table** — per scenario: import, export, PV generated, self-consumed, +battery throughput, plus which days were sampled. + +Three properties carry through from the engine and must survive into the UI: + +- **Caveats are displayed, not buried.** The results document carries them: P10 + fallback, `export_credit_p_estimate` not being additive with `cost_p`, the + Agile-versus-banded baseline asymmetry. +- **Unavailable and degraded months are marked**, never drawn as a zero-height + bar. A zero bar reads as free electricity. +- **`self_consumed_kwh` is greyed with its reason** when + `self_consumed_kwh_meaningful` is false, rather than showing a bare 0. + +Plus a "last run" timestamp and a link to download the raw JSON. + +## Failure handling + +Failures stay visible, matching the engine's own contract: + +| Condition | Behaviour | +|---|---| +| Config invalid | Rendered inline, form still populated with what was entered | +| Subprocess exits non-zero | Exit code and last stderr lines shown, not a blank page | +| Results file missing or unreadable | Says so, rather than rendering an empty chart | +| Run already in progress | Second Run refused with a clear message | +| Predbat restarted mid-run | Reported as "did not complete", state reset to idle | + +## Testing + +All offline. No network, and **no test spawns the real engine** — `annual_job` +is driven with a stub script. + +- **`tariff_catalogue`** — the Compare-to-annual key mapping is right; user + `compare_list` entries merge without clobbering built-ins; a malformed entry is + skipped rather than breaking the dropdown. +- **`annual_job`** — progress lines parse; malformed lines do not crash the + parser; cancel terminates; a non-zero exit is reported as failed; a second Run + while running is refused. +- **`web_annual`** — **the form renders and validates against a completely + unconfigured PredBat.** That is the acceptance criterion for the unconfigured + requirement, and the one most likely to regress silently. Plus: prefill falls + back per field; the load radio pair enforces exclusivity; a `{dno_region}` URL + with no region is rejected before spawning. + +## Out of scope + +- Predbat.com multi-user support — no queue, tenancy or quotas. Revisit when + that becomes the target. +- Changes to the Compare tab, including making it read `tariff_catalogue.py`. +- Run history. The tab keeps the last result only. +- Any change to the prediction model itself. From 6d7bba13e4b157275892dd984ba095b4d01f0b36 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 20:27:23 +0200 Subject: [PATCH 041/119] docs: keep five annual runs in storage, prefill from the args dict Three corrections from review: Prefill reads self.base.args via get_arg(), never apps.yaml from disk - the file may not exist at all in some deployments, which is exactly where the unconfigured case matters most. The spec now names the actual arg keys per field, and notes battery capacity is discovered from the inverter rather than living in args. Results go to Storage as a five-run ring with a newest-first index, and the tab gains a selector so a user can flip between "5 kWh battery" and "10 kWh battery" without re-running either. Stable ids and generated labels leave room for a compare-runs feature without migrating stored results. The UI ships on the same branch as the engine, as one PR. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-07-26-annual-web-ui-design.md | 73 +++++++++++++++++-- 1 file changed, 65 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md index 576be1252..7fba3edd9 100644 --- a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md +++ b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md @@ -27,6 +27,10 @@ adds no modelling; it exposes what is there. UK system so a visitor can run it immediately. - **A tariff dropdown**, built from a curated catalogue merged with the user's own `compare_list` if they have one. +- **The last five runs are kept**, in Storage, with a selector to switch between + them. +- **Same branch, one PR** — this builds on `feat/annual-prediction-tool`, since + the UI cannot work without the engine. ## Why a subprocess @@ -78,12 +82,35 @@ where the bugs will be. ### Persistence -Two files in `config_root`, alongside `comparisons.yaml`: +**The config** is `annual.yaml` in `config_root`, alongside `comparisons.yaml` — +handed to the subprocess as `--config`, hand-editable, and surviving restarts. -- `annual.yaml` — the config the form edits, handed to the subprocess as - `--config`. Hand-editable, survives restarts. -- `annual_results.json` — the last completed run, so revisiting the tab shows - results with a timestamp rather than an empty page. +**The results go to Storage**, not a bare file, per CLAUDE.md. The Storage +component is reached the way the rest of the codebase reaches it: +`self.base.components.get_component("storage")`. + +The **last five runs** are kept: + +| Storage key | Contents | +|---|---| +| `annual` / `runs_index` | Newest-first list of `{id, timestamp, label, months_included, status}` | +| `annual` / `run_` | That run's full results document | + +Saving appends to the index and deletes the blob of anything falling off the +end, so the ring never leaks entries. + +`id` is timestamp-derived. `label` is generated from the config that produced the +run — for example *"9.5 kWh battery · 5.6 kWp · Agile"* — because a selector +listing five bare timestamps tells the user nothing about which run was which. + +**The subprocess still writes a plain file.** `annual_cli.py` keeps its `--out` +path, writing to a temporary file; the parent reads that on completion and saves +it into Storage. The CLI stays usable standalone and knows nothing about +Storage or run history. + +**Forward compatibility.** A stable `id` and a human label in the index are +exactly what a later *compare runs* feature needs, so it can be added without +migrating stored results. That feature is out of scope here. ### Progress protocol @@ -108,7 +135,26 @@ collapsed **Advanced** holding `year`, `samples_per_month`, Each value is read from the live instance where available and falls back to a typical-UK default otherwise, **independently**. A half-configured Predbat gets -its real battery size alongside example solar, rather than all-or-nothing. +its real inverter limits alongside example solar, rather than all-or-nothing. + +**Read from the in-memory args dictionary, never from `apps.yaml` on disk.** +`self.base.args` is already the parsed configuration; the file itself may not +exist at all in some deployments, so re-reading it would fail exactly where the +unconfigured case matters most. Use `get_arg()` so indirection and defaults +behave as they do everywhere else in Predbat. + +| Form field | Source | +|---|---| +| Solar arrays | `open_meteo_forecast`, else `forecast_solar` — both are already lists of `{kwp, declination, azimuth, efficiency}`, a direct shape match | +| Location | `latitude`/`longitude` or `postcode` from the same solar entries | +| Inverter and export limits | `inverter_limit`, `export_limit` | +| Hybrid or AC coupled | `inverter_type` | +| Tariff URLs and region | `rates_import_octopus_url`, `rates_export_octopus_url`, `dno_region` | +| Tariff dropdown extras | `compare_list` | +| Battery capacity | Not reliably in args — it is discovered from the inverter at runtime. Use the live `soc_max` when the instance has one, otherwise the default | + +Anything absent falls back to the typical-UK value. No field is required to be +present for the form to render. When the live instance supplied neither a battery nor a solar array — the two that signal a configured system — a banner says so: @@ -228,7 +274,13 @@ Three properties carry through from the engine and must survive into the UI: - **`self_consumed_kwh` is greyed with its reason** when `self_consumed_kwh_meaningful` is false, rather than showing a bare 0. -Plus a "last run" timestamp and a link to download the raw JSON. +Above all three sits a **run selector** listing the stored runs newest-first by +their generated label and timestamp, so a user can flip between "with a 5 kWh +battery" and "with a 10 kWh battery" without re-running either. Selecting a run +re-renders the totals, chart and table from that run's stored document. + +Plus a "last run" timestamp and a link to download the raw JSON of the selected +run. ## Failure handling @@ -253,6 +305,9 @@ is driven with a stub script. - **`annual_job`** — progress lines parse; malformed lines do not crash the parser; cancel terminates; a non-zero exit is reported as failed; a second Run while running is refused. +- **Run history** — saving a sixth run evicts the oldest and deletes its blob; + the index stays newest-first; a corrupt or missing run blob is reported rather + than rendering an empty chart; labels are generated from the config. - **`web_annual`** — **the form renders and validates against a completely unconfigured PredBat.** That is the acceptance criterion for the unconfigured requirement, and the one most likely to regress silently. Plus: prefill falls @@ -264,5 +319,7 @@ is driven with a stub script. - Predbat.com multi-user support — no queue, tenancy or quotas. Revisit when that becomes the target. - Changes to the Compare tab, including making it read `tariff_catalogue.py`. -- Run history. The tab keeps the last result only. +- Comparing two stored runs side by side. The selector switches between them; + it does not overlay them. The index is designed so this can be added later + without migrating stored results. - Any change to the prediction model itself. From a1d63b587931fe2f35877dbf6ae372f1e4207bab Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 20:31:09 +0200 Subject: [PATCH 042/119] docs: route annual results through storage, never a bare file write There may be no filesystem, so the simulation must not write results with open(). Both paths go through Storage instead: the web tab passes the live Storage component down, and annual_cli.py imports Storage and constructs a backend itself. AnnualPredictor already takes storage=; it now uses it rather than handing a dict back to be json.dump'd. A new annual_store.py owns the five-run ring for both callers, so the CLI builds history too rather than the web tab being the only thing that remembers. soc_max is available via get_arg after all - a zero or absent value means unset, so it falls back to the default for the user to adjust. Recorded the limitation this exposes: a subprocess needs reach into the same Storage as its parent, which is free for the local-files backend and unsolved for anything else. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-07-26-annual-web-ui-design.md | 36 +++++++++++++++---- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md index 7fba3edd9..af0a4eabd 100644 --- a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md +++ b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md @@ -57,9 +57,11 @@ best-effort — which is what makes the unconfigured case work at all. | `apps/predbat/tariff_catalogue.py` | **New.** The curated tariff list as data, plus the mapping from Compare's key names to the annual engine's. No logic beyond lookup and merge. | | `apps/predbat/web_annual.py` | **New.** The tab: form rendering, config load/save, results rendering. | | `apps/predbat/annual_job.py` | **New.** Subprocess lifecycle only — spawn, parse progress, track state, cancel, reap. No HTML. | +| `apps/predbat/annual_store.py` | **New.** The run ring: save, list, load, evict, and label generation. Storage-backed, shared by the CLI and the web tab so there is one implementation. | | `apps/predbat/web.py` | **Modify.** Five route registrations delegating to `web_annual.py`. | | `apps/predbat/web_helper.py` | **Modify.** One nav link, alongside the others at the `Compare` block. | -| `apps/predbat/annual_cli.py` | **Modify.** Add `--progress-json` for machine-readable progress. | +| `apps/predbat/annual_cli.py` | **Modify.** Add `--progress-json`; construct a Storage backend and stop writing results with a bare `open()`. | +| `apps/predbat/annual.py` | **Modify.** Write the results document through Storage rather than returning it for a caller to dump. | `web.py` is already 5,730 lines and `web_helper.py` 8,903, so the tab gets its own module rather than growing either — following the `web_metrics_dashboard.py` @@ -103,10 +105,25 @@ end, so the ring never leaks entries. run — for example *"9.5 kWh battery · 5.6 kWp · Agile"* — because a selector listing five bare timestamps tells the user nothing about which run was which. -**The subprocess still writes a plain file.** `annual_cli.py` keeps its `--out` -path, writing to a temporary file; the parent reads that on completion and saves -it into Storage. The CLI stays usable standalone and knows nothing about -Storage or run history. +**Nothing writes results to the filesystem directly.** There may be no +filesystem at all, so results go through Storage on both paths: + +- **Integrated:** the web tab passes the live Storage component down. +- **Command line:** `annual_cli.py` imports Storage and constructs a backend + itself. + +`AnnualPredictor` already accepts a `storage=` argument; it now uses it to write +the results, instead of handing a dict back for the caller to `json.dump`. +`annual_cli.py`'s current `open(args.out, "w")` goes away as the mechanism — +`--out` survives only as an explicit opt-in for a human who wants a JSON file on +disk. + +Because parent and child share the same Storage, the web tab reads a completed +run straight out of Storage. No temporary file changes hands, and the progress +stream over stdout stays the only thing crossing the process boundary. + +`annual_store.py` owns the ring for both callers, so the CLI builds run history +too rather than the web tab being the only thing that remembers. **Forward compatibility.** A stable `id` and a human label in the index are exactly what a later *compare runs* feature needs, so it can be added without @@ -151,7 +168,7 @@ behave as they do everywhere else in Predbat. | Hybrid or AC coupled | `inverter_type` | | Tariff URLs and region | `rates_import_octopus_url`, `rates_export_octopus_url`, `dno_region` | | Tariff dropdown extras | `compare_list` | -| Battery capacity | Not reliably in args — it is discovered from the inverter at runtime. Use the live `soc_max` when the instance has one, otherwise the default | +| Battery capacity | `soc_max` — a zero or absent value means it is not set, so fall back to the default and let the user adjust | Anything absent falls back to the typical-UK value. No field is required to be present for the form to render. @@ -240,8 +257,13 @@ The page polls `GET /annual_status` once a second for navigating away and back shows a run still in progress. Cancel sends `SIGTERM`, then `SIGKILL` if the child does not exit. -### Two limitations, stated rather than hidden +### Three limitations, stated rather than hidden +- **A subprocess needs reach into the same Storage as its parent.** With the + local-files backend both sides share `config_root` and this is free. A + non-filesystem backend would have to pass its connection details to the child, + which is unsolved here and is a real consideration whenever Predbat.com + becomes the target. - **A Predbat restart orphans a running child.** Job state resets to idle and the tab reports that the last run did not complete, rather than showing a progress bar stuck forever. From 93a3c64e9c509bbff4b22c14b0ea88933ad8707d Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 20:33:27 +0200 Subject: [PATCH 043/119] docs: the simulation returns results, the wrapper persists them Better separation than routing results through Storage from inside the engine: AnnualPredictor.run() keeps returning the document and writes nothing. Its storage= argument stays, but only for caching weather and rate downloads, which is an input concern. The subprocess hands results back over stdout and progress over stderr, so each wrapper decides where they go - the web tab into Storage via annual_store.py, a human via the table or --out. This removes the limitation the previous revision had to record: the child never needs reach into the parent's Storage, so nothing assumes a filesystem or shared backend on either side. One flag, --machine, switches both streams together, since the results JSON cannot share stdout with the human table. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-07-26-annual-web-ui-design.md | 66 ++++++++++--------- 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md index af0a4eabd..ca53a9ac1 100644 --- a/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md +++ b/docs/superpowers/specs/2026-07-26-annual-web-ui-design.md @@ -57,11 +57,10 @@ best-effort — which is what makes the unconfigured case work at all. | `apps/predbat/tariff_catalogue.py` | **New.** The curated tariff list as data, plus the mapping from Compare's key names to the annual engine's. No logic beyond lookup and merge. | | `apps/predbat/web_annual.py` | **New.** The tab: form rendering, config load/save, results rendering. | | `apps/predbat/annual_job.py` | **New.** Subprocess lifecycle only — spawn, parse progress, track state, cancel, reap. No HTML. | -| `apps/predbat/annual_store.py` | **New.** The run ring: save, list, load, evict, and label generation. Storage-backed, shared by the CLI and the web tab so there is one implementation. | +| `apps/predbat/annual_store.py` | **New.** The run ring: save, list, load, evict, and label generation. Takes a Storage object; used by the web tab. | | `apps/predbat/web.py` | **Modify.** Five route registrations delegating to `web_annual.py`. | | `apps/predbat/web_helper.py` | **Modify.** One nav link, alongside the others at the `Compare` block. | -| `apps/predbat/annual_cli.py` | **Modify.** Add `--progress-json`; construct a Storage backend and stop writing results with a bare `open()`. | -| `apps/predbat/annual.py` | **Modify.** Write the results document through Storage rather than returning it for a caller to dump. | +| `apps/predbat/annual_cli.py` | **Modify.** Add a `--machine` mode: results JSON on stdout, progress JSON on stderr, no human table. | `web.py` is already 5,730 lines and `web_helper.py` 8,903, so the tab gets its own module rather than growing either — following the `web_metrics_dashboard.py` @@ -105,25 +104,28 @@ end, so the ring never leaks entries. run — for example *"9.5 kWh battery · 5.6 kWp · Agile"* — because a selector listing five bare timestamps tells the user nothing about which run was which. -**Nothing writes results to the filesystem directly.** There may be no -filesystem at all, so results go through Storage on both paths: +**The simulation returns its results; persisting them is the wrapper's job.** +`AnnualPredictor.run()` keeps returning the results document, as it does today. +It does not write them anywhere. Its `storage=` argument stays, but only for +what it was always for — caching weather and rate downloads, which is an input +concern. -- **Integrated:** the web tab passes the live Storage component down. -- **Command line:** `annual_cli.py` imports Storage and constructs a backend - itself. +The subprocess hands the results back over the pipe: -`AnnualPredictor` already accepts a `storage=` argument; it now uses it to write -the results, instead of handing a dict back for the caller to `json.dump`. -`annual_cli.py`'s current `open(args.out, "w")` goes away as the mechanism — -`--out` survives only as an explicit opt-in for a human who wants a JSON file on -disk. +| Stream | Carries | +|---|---| +| stdout | The results document, one JSON object, on completion | +| stderr | Progress, one JSON object per line, as it runs | -Because parent and child share the same Storage, the web tab reads a completed -run straight out of Storage. No temporary file changes hands, and the progress -stream over stdout stays the only thing crossing the process boundary. +Each wrapper then decides where results go. The web tab parses stdout and saves +through `annual_store.py` into the live Storage component. A human running +`annual_cli.py` gets the table on stdout as before, or `--out` to write a file. -`annual_store.py` owns the ring for both callers, so the CLI builds run history -too rather than the web tab being the only thing that remembers. +**This is why the child never needs reach into the parent's Storage** — the +process boundary carries everything, so no filesystem or shared backend is +assumed on either side. A results document without `--debug` is tens of +kilobytes, which is unremarkable over a pipe; `--debug` retains per-sample HTML +plans and is deliberately not used by the web path. **Forward compatibility.** A stable `id` and a human label in the index are exactly what a later *compare runs* feature needs, so it can be added without @@ -131,16 +133,21 @@ migrating stored results. That feature is out of scope here. ### Progress protocol -`annual_cli.py` currently writes progress to stderr as `[3/12] Month 03/2025`. -That is parseable, but parsing prose couples the parent to wording that will be -reworded. `--progress-json` emits one JSON object per line instead: +`annual_cli.py` currently writes progress to stderr as `[3/12] Month 03/2025` +and a human-readable table to stdout. Parsing prose would couple the parent to +wording that will get reworded, and the results cannot simply join the table on +stdout. -```json -{"completed": 3, "total": 12, "message": "Month 03/2025"} -``` +A single `--machine` flag switches both streams at once, rather than two flags +that interact: + +| Stream | Default | Under `--machine` | +|---|---|---| +| stdout | Human-readable table | The results document as one JSON object | +| stderr | `[3/12] Month 03/2025` | `{"completed": 3, "total": 12, "message": "..."}` per line | -The human-readable form stays the default so the CLI remains pleasant to use by -hand. +The human-readable behaviour stays the default so the CLI remains pleasant by +hand, and the web tab always passes `--machine`. ## The form @@ -257,13 +264,8 @@ The page polls `GET /annual_status` once a second for navigating away and back shows a run still in progress. Cancel sends `SIGTERM`, then `SIGKILL` if the child does not exit. -### Three limitations, stated rather than hidden +### Two limitations, stated rather than hidden -- **A subprocess needs reach into the same Storage as its parent.** With the - local-files backend both sides share `config_root` and this is free. A - non-filesystem backend would have to pass its connection details to the child, - which is unsolved here and is a real consideration whenever Predbat.com - becomes the target. - **A Predbat restart orphans a running child.** Job state resets to idle and the tab reports that the last run did not complete, rather than showing a progress bar stuck forever. From ca26116d19d38d6e795303d77cb1ccc73e917e9d Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 20:45:31 +0200 Subject: [PATCH 044/119] docs: add the Annual web UI implementation plan Nine TDD tasks: tariff catalogue, CLI machine mode, subprocess job control, the five-run store, prefill, the form, routes and navigation, the results view with the monthly chart, and documentation. The chart palette is measured, not chosen. Predbat's house trio (#2196F3/#FF9800/#4CAF50) fails the dataviz validator - green against orange is deltaE 3.6 under protanopia, so roughly one man in twelve could not distinguish "Without Predbat" from "With Predbat", the exact comparison the tool exists to make. The plan pins the Okabe-Ito trio, which passes every check in both light and dark mode. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-07-26-annual-web-ui.md | 2767 +++++++++++++++++ 1 file changed, 2767 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-26-annual-web-ui.md diff --git a/docs/superpowers/plans/2026-07-26-annual-web-ui.md b/docs/superpowers/plans/2026-07-26-annual-web-ui.md new file mode 100644 index 000000000..ed021cf87 --- /dev/null +++ b/docs/superpowers/plans/2026-07-26-annual-web-ui.md @@ -0,0 +1,2767 @@ +# Annual Prediction Web UI Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add an **Annual** tab to the Predbat web interface — a form that prefills from the live instance, a subprocess run with a real progress bar, and results as annual totals plus a grouped monthly bar chart, with the last five runs kept in Storage. + +**Architecture:** The tab spawns `annual_cli.py --machine` as a child process; results come back as JSON on stdout and progress as JSON-per-line on stderr, so the child needs no filesystem and no reach into the parent's Storage. `annual_job.py` owns the process lifecycle and knows no HTML; `web_annual.py` owns the HTML and knows no process handling; `annual_store.py` owns the five-run ring. + +**Tech Stack:** Python 3, aiohttp (already the web server), `asyncio.create_subprocess_exec`, ApexCharts (already loaded from CDN by the existing chart pages), Predbat's Storage component. + +**Spec:** `docs/superpowers/specs/2026-07-26-annual-web-ui-design.md` +**Engine spec (already built):** `docs/superpowers/specs/2026-07-25-annual-prediction-tool-design.md` + +## Global Constraints + +- **Line length:** 256 chars (Black), 250 chars (Flake8). +- **Docstrings:** 100% coverage — every function *and* class, including test functions and nested helpers. +- **Spelling:** British English (`en-gb`) via CSpell. New words go in `.cspell/custom-dictionary-workspace.txt`, which a pre-commit hook auto-sorts — **re-stage it** after running pre-commit. `docs/*.md` is spell-checked; `docs/superpowers/` is not. +- **String formatting:** `"...".format(...)`, **not** f-strings, in `apps/predbat/*.py`. +- **Indent:** 4 spaces, never tabs. +- **File header:** every new `apps/predbat/*.py` starts with the five-line copyright block used by all existing modules. +- **Storage:** all persistence goes through the Storage component, never direct file access. Reached via `self.base.components.get_component("storage")`. +- **Never re-read `apps/predbat/apps.yaml` from disk.** Read configuration from the in-memory args dictionary via `get_arg()`. The file may not exist in some deployments. +- **Tests:** registered in `TEST_REGISTRY` in `apps/predbat/unit_test.py`; signature `def test_name(my_predbat):` returning truthy on failure. No network I/O. **No test may spawn the real engine** — `annual_job` is driven by a stub script. +- **`git add` new files BEFORE running pre-commit.** `pre-commit --all-files` enumerates via `git ls-files` and silently skips untracked files, producing a false pass. This has bitten this branch repeatedly. +- **Pre-commit:** the script is `coverage/run_pre_commit`, NOT the repo root. Prefer `coverage/venv/bin/pre-commit run --files `. A run reporting "files were modified by this hook" has **not** passed — re-stage and re-run until clean. +- **Report only what you ran.** Never state a check passed unless you executed it and read the output. +- **Tests run from `coverage/`:** `cd coverage && ./run_all --test > /tmp/out.txt 2>&1`, then grep the FILE. Never pipe test output straight to grep. +- **The full annual suite takes ~95 seconds.** If it runs for many minutes, stop — that symptom previously meant a test leaked `debug_enable` onto the shared fixture, disabling the C++ prediction kernel. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `apps/predbat/tariff_catalogue.py` | **New.** Curated tariff list as data, the Compare→annual key mapping, and the merge with a user's `compare_list`. | +| `apps/predbat/annual_job.py` | **New.** Subprocess lifecycle only: spawn, parse progress, track state, cancel, reap. No HTML, no Storage. | +| `apps/predbat/annual_store.py` | **New.** The five-run ring over a Storage object: save, list, load, evict, label generation. | +| `apps/predbat/web_annual.py` | **New.** The tab: prefill, config load/save, form HTML, results HTML, and the five route handlers. | +| `apps/predbat/annual_cli.py` | **Modify.** Add `--machine`: results JSON on stdout, progress JSON on stderr, no human table. | +| `apps/predbat/web.py` | **Modify.** Import, instantiate `AnnualPage`, register six routes. | +| `apps/predbat/web_helper.py` | **Modify.** One nav link beside `Compare`. | +| `apps/predbat/tests/test_annual_*.py` | **New.** One test module per new module. | +| `docs/annual-prediction.md` | **Modify.** Document the tab. | + +### The validated chart palette + +The three scenario colours are **not** free choice. Predbat's house chart trio +(`#2196F3`, `#FF9800`, `#4CAF50`) was measured with the `dataviz` skill's +validator and **fails**: green↔orange is ΔE 3.6 under protanopia, far below the +floor of 8 — meaning roughly 1 in 12 men cannot distinguish "Without Predbat" +from "With Predbat", the exact comparison this tool exists to make. + +Use the Okabe-Ito trio below. It passes all five checks — lightness band, chroma +floor, CVD separation across all pairs, normal-vision floor, and contrast — in +**both** light and dark mode: + +| Scenario | Colour | +|---|---| +| No PV/Battery | `#0072B2` (blue) | +| Without Predbat | `#D55E00` (vermillion) | +| With Predbat | `#009E73` (bluish green) | + +Worst all-pairs CVD separation is ΔE 11.0 (deutan). Do not substitute +"Predbat-looking" colours without re-running +`node scripts/validate_palette.js "" --mode light --pairs all` and +the same for `--mode dark`. + +--- + +## Task 1: Tariff catalogue + +Pure data plus two small functions. No dependencies, so it goes first and later tasks can rely on it. + +**Files:** +- Create: `apps/predbat/tariff_catalogue.py` +- Create: `apps/predbat/tests/test_tariff_catalogue.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `tariff_catalogue.BUILTIN_TARIFFS` — list of `{"id", "name", "import_octopus_url", "export_octopus_url"}` + - `tariff_catalogue.CUSTOM_ID` — the string `"custom"` + - `tariff_catalogue.convert_compare_entry(entry) -> dict | None` + - `tariff_catalogue.merged_catalogue(compare_list=None) -> list[dict]` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_tariff_catalogue.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the tariff catalogue used by the Annual tab's dropdown.""" + +from tariff_catalogue import BUILTIN_TARIFFS, CUSTOM_ID, convert_compare_entry, merged_catalogue + + +def test_tariff_catalogue(my_predbat): + """Verify the built-in catalogue, the Compare key mapping, and the merge.""" + failed = False + print("**** Testing tariff_catalogue ****") + + print("Test: every built-in entry has an id, a name and at least an import URL") + if not BUILTIN_TARIFFS: + print(" ERROR: the built-in catalogue is empty") + failed = True + seen_ids = set() + for entry in BUILTIN_TARIFFS: + for key in ["id", "name"]: + if not entry.get(key): + print(" ERROR: entry {} is missing '{}'".format(entry, key)) + failed = True + if not entry.get("import_octopus_url") and not entry.get("rates_import"): + print(" ERROR: entry {} has neither an import URL nor fixed rates".format(entry.get("id"))) + failed = True + if entry.get("id") in seen_ids: + print(" ERROR: duplicate id {}".format(entry.get("id"))) + failed = True + seen_ids.add(entry.get("id")) + + print("Test: no built-in entry uses Compare's key names") + for entry in BUILTIN_TARIFFS: + for stale in ["rates_import_octopus_url", "rates_export_octopus_url"]: + if stale in entry: + print(" ERROR: entry {} still uses Compare's key '{}'".format(entry.get("id"), stale)) + failed = True + + print("Test: convert_compare_entry maps Compare's URL keys onto the engine's") + converted = convert_compare_entry( + { + "id": "agile_agile", + "name": "Agile import/Agile export", + "rates_import_octopus_url": "https://example.com/import/", + "rates_export_octopus_url": "https://example.com/export/", + } + ) + if converted is None: + print(" ERROR: a valid Compare entry should convert") + failed = True + else: + if converted.get("import_octopus_url") != "https://example.com/import/": + print(" ERROR: import URL not mapped, got {}".format(converted)) + failed = True + if converted.get("export_octopus_url") != "https://example.com/export/": + print(" ERROR: export URL not mapped, got {}".format(converted)) + failed = True + if "rates_import_octopus_url" in converted: + print(" ERROR: the Compare key should not survive conversion") + failed = True + + print("Test: fixed rate structures pass through unchanged") + converted = convert_compare_entry({"id": "cap", "name": "Price cap", "rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}]}) + if converted is None or converted.get("rates_import") != [{"rate": 24.86}]: + print(" ERROR: fixed rates should pass through, got {}".format(converted)) + failed = True + + print("Test: an entry with no usable rate source is rejected rather than shown") + if convert_compare_entry({"id": "current", "name": "Current Tariff"}) is not None: + print(" ERROR: an entry with no rates should be rejected") + failed = True + if convert_compare_entry({"name": "No id"}) is not None: + print(" ERROR: an entry with no id should be rejected") + failed = True + if convert_compare_entry("not a dict") is not None: + print(" ERROR: a non-dict should be rejected rather than raising") + failed = True + + print("Test: merged_catalogue with no user list returns the built-ins plus Custom") + merged = merged_catalogue(None) + if len(merged) != len(BUILTIN_TARIFFS) + 1: + print(" ERROR: expected {} entries, got {}".format(len(BUILTIN_TARIFFS) + 1, len(merged))) + failed = True + if merged[-1]["id"] != CUSTOM_ID: + print(" ERROR: Custom should be the last entry, got {}".format(merged[-1])) + failed = True + + print("Test: a user's compare_list is merged in and does not clobber a built-in id") + builtin_id = BUILTIN_TARIFFS[0]["id"] + merged = merged_catalogue( + [ + {"id": builtin_id, "name": "My override", "rates_import_octopus_url": "https://example.com/mine/"}, + {"id": "my_tariff", "name": "My tariff", "rates_import": [{"rate": 20.0}]}, + ] + ) + ids = [entry["id"] for entry in merged] + if ids.count(builtin_id) != 1: + print(" ERROR: a user entry sharing a built-in id should not duplicate it, ids were {}".format(ids)) + failed = True + if "my_tariff" not in ids: + print(" ERROR: a new user entry should appear, ids were {}".format(ids)) + failed = True + + print("Test: a malformed user entry is skipped rather than breaking the dropdown") + merged = merged_catalogue([{"junk": True}, None, "string", {"id": "ok", "name": "Ok", "rates_import": [{"rate": 5.0}]}]) + ids = [entry["id"] for entry in merged] + if "ok" not in ids: + print(" ERROR: the valid entry should survive alongside malformed ones, ids were {}".format(ids)) + failed = True + + return failed +``` + +- [ ] **Step 2: Register the test and run it to verify it fails** + +Add to `apps/predbat/unit_test.py`, alongside the other `from tests.test_* import ...` lines: + +```python +from tests.test_tariff_catalogue import test_tariff_catalogue +``` + +and to the `TEST_REGISTRY` list inside `main()`: + +```python + ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), +``` + +Run: `cd coverage && ./run_all --test tariff_catalogue > /tmp/w1.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/w1.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'tariff_catalogue'`. + +- [ ] **Step 3: Create `apps/predbat/tariff_catalogue.py`** + +The entries are transcribed from the commented-out `compare_list` template in `apps/predbat/config/apps.yaml` (around lines 531-592), with Compare's key names mapped to the engine's. + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Curated tariff catalogue for the Annual prediction tab's dropdown. + +The entries mirror the commented-out ``compare_list`` template in +``config/apps.yaml`` so a user does not have to hand-copy a YAML block to get a +realistic tariff. Compare's key names differ from the annual engine's, so the +mapping lives here and nowhere else. +""" + +# Compare writes rates_import_octopus_url; the annual engine reads import_octopus_url +COMPARE_KEY_MAP = { + "rates_import_octopus_url": "import_octopus_url", + "rates_export_octopus_url": "export_octopus_url", +} + +# Keys that mean the same thing in both and need no translation +PASSTHROUGH_KEYS = ["rates_import", "rates_export"] + +# The dropdown's escape hatch: leaves the URL fields blank for a hand-entered tariff +CUSTOM_ID = "custom" + +_OCTOPUS = "https://api.octopus.energy/v1/products" + +BUILTIN_TARIFFS = [ + {"id": "cap_seg", "name": "Price cap import / SEG export", "rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}]}, + { + "id": "eon_next_drive", + "name": "Eon Next Drive import / Fixed export", + "rates_import": [{"rate": 6.7, "start": "00:00:00", "end": "07:00:00"}, {"rate": 24.86, "start": "07:00:00", "end": "00:00:00"}], + "rates_export": [{"rate": 16.5}], + }, + { + "id": "igo_fixed", + "name": "Intelligent GO import / Fixed export", + "import_octopus_url": "{}/INTELLI-VAR-24-10-29/electricity-tariffs/E-1R-INTELLI-VAR-24-10-29-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "igo_agile", + "name": "Intelligent GO import / Agile export", + "import_octopus_url": "{}/INTELLI-VAR-24-10-29/electricity-tariffs/E-1R-INTELLI-VAR-24-10-29-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "go_fixed", + "name": "GO import / Fixed export", + "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "agile_fixed", + "name": "Agile import / Fixed export", + "import_octopus_url": "{}/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "agile_agile", + "name": "Agile import / Agile export", + "import_octopus_url": "{}/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "flux", + "name": "Flux import / Flux export", + "import_octopus_url": "{}/FLUX-IMPORT-23-02-14/electricity-tariffs/E-1R-FLUX-IMPORT-23-02-14-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/FLUX-EXPORT-23-02-14/electricity-tariffs/E-1R-FLUX-EXPORT-23-02-14-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + }, + { + "id": "cosy_fixed", + "name": "Cosy import / Fixed export", + "import_octopus_url": "{}/COSY-22-12-08/electricity-tariffs/E-1R-COSY-22-12-08-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "cosy_agile", + "name": "Cosy import / Agile export", + "import_octopus_url": "{}/COSY-22-12-08/electricity-tariffs/E-1R-COSY-22-12-08-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "snug_fixed", + "name": "Snug import / Fixed export", + "import_octopus_url": "{}/SNUG-24-11-07/electricity-tariffs/E-1R-SNUG-24-11-07-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "iflux", + "name": "Intelligent Flux import / export", + "import_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/INTELLI-FLUX-EXPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-EXPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, +] + + +def convert_compare_entry(entry): + """Convert one Compare ``compare_list`` entry into the annual engine's shape. + + Returns None when the entry cannot be used - it is not a mapping, it has no + id, or it carries no rate source at all. Compare allows an entry with neither + (the 'current' pseudo-tariff, which means "whatever is configured"), and that + has no meaning here, so it is dropped rather than offered as a broken choice. + """ + if not isinstance(entry, dict): + return None + if not entry.get("id") or not entry.get("name"): + return None + + converted = {"id": entry["id"], "name": entry["name"]} + for source_key, target_key in COMPARE_KEY_MAP.items(): + if entry.get(source_key): + converted[target_key] = entry[source_key] + for key in PASSTHROUGH_KEYS: + if entry.get(key): + converted[key] = entry[key] + + if not converted.get("import_octopus_url") and not converted.get("rates_import"): + return None + return converted + + +def merged_catalogue(compare_list=None): + """Return the dropdown's entries: built-ins, then the user's own, then Custom. + + A user entry sharing a built-in id replaces it rather than appearing twice - + the user's own definition is the more specific one. Malformed entries are + skipped so one bad line in apps.yaml cannot empty the dropdown. + """ + catalogue = [dict(entry) for entry in BUILTIN_TARIFFS] + by_id = {entry["id"]: index for index, entry in enumerate(catalogue)} + + for entry in compare_list or []: + converted = convert_compare_entry(entry) + if converted is None: + continue + if converted["id"] in by_id: + catalogue[by_id[converted["id"]]] = converted + else: + by_id[converted["id"]] = len(catalogue) + catalogue.append(converted) + + catalogue.append({"id": CUSTOM_ID, "name": "Custom - enter URLs below"}) + return catalogue +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test tariff_catalogue > /tmp/w1.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w1.txt` + +Expected: no output. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +git add apps/predbat/tariff_catalogue.py apps/predbat/tests/test_tariff_catalogue.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/tariff_catalogue.py apps/predbat/tests/test_tariff_catalogue.py apps/predbat/unit_test.py +git commit -m "feat(annual): add the tariff catalogue for the Annual tab dropdown" +``` + +--- + +## Task 2: Machine mode for the CLI + +The child process must hand results back over the pipe. `annual_cli.py` currently prints a human table to stdout and prose progress to stderr, so results cannot simply join stdout. One flag switches both streams together. + +**Files:** +- Modify: `apps/predbat/annual_cli.py` +- Modify: `apps/predbat/tests/test_annual_cli.py` + +**Interfaces:** +- Consumes: `annual.AnnualPredictor` (existing). +- Produces: + - `annual_cli.make_progress(quiet, machine=False)` — returns a callback writing either `[3/12] msg` or `{"completed": 3, "total": 12, "message": "msg"}` to stderr + - `--machine` flag: stdout carries the results document as one JSON object; the human table is suppressed + +- [ ] **Step 1: Write the failing test** + +Append to `apps/predbat/tests/test_annual_cli.py` (and add `make_progress` to its `annual_cli` import line): + +```python +def test_annual_cli_machine(my_predbat): + """Verify machine mode emits JSON progress on stderr and nothing human on stdout.""" + import io + import json + import sys + + failed = False + print("**** Testing annual CLI machine mode ****") + + print("Test: machine progress writes one JSON object per line to stderr") + captured = io.StringIO() + original_stderr = sys.stderr + sys.stderr = captured + try: + progress = make_progress(quiet=False, machine=True) + progress(3, 12, "Month 03/2025") + finally: + sys.stderr = original_stderr + + line = captured.getvalue().strip() + try: + parsed = json.loads(line) + except ValueError: + print(" ERROR: machine progress should be JSON, got {!r}".format(line)) + parsed = {} + failed = True + if parsed.get("completed") != 3 or parsed.get("total") != 12 or parsed.get("message") != "Month 03/2025": + print(" ERROR: unexpected progress payload {}".format(parsed)) + failed = True + + print("Test: human progress is unchanged when machine mode is off") + captured = io.StringIO() + sys.stderr = captured + try: + progress = make_progress(quiet=False, machine=False) + progress(3, 12, "Month 03/2025") + finally: + sys.stderr = original_stderr + if "[3/12]" not in captured.getvalue(): + print(" ERROR: expected the human form, got {!r}".format(captured.getvalue())) + failed = True + + print("Test: quiet still suppresses progress in both modes") + if make_progress(quiet=True, machine=False) is not None: + print(" ERROR: quiet should give no progress callback") + failed = True + if make_progress(quiet=True, machine=True) is not None: + print(" ERROR: quiet should give no progress callback in machine mode either") + failed = True + + return failed +``` + +Register it in `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine +``` + +```python + ("annual_cli_machine", test_annual_cli_machine, "Annual CLI machine mode tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test annual_cli_machine > /tmp/w2.txt 2>&1; grep -E "ERROR|TypeError|cannot import" /tmp/w2.txt` + +Expected: FAIL — `make_progress()` does not yet accept `machine`. + +- [ ] **Step 3: Add machine mode to `apps/predbat/annual_cli.py`** + +Replace `make_progress` with: + +```python +def make_progress(quiet, machine=False): + """Return a progress callback writing to stderr, or None when quiet. + + Machine mode emits one JSON object per line so the parent process never has + to parse prose - the human wording is free to change without breaking a + caller. Progress always goes to stderr so stdout carries only the result, + whichever mode is in use. + """ + if quiet: + return None + + if machine: + + def progress(completed, total, message): + """Emit one JSON progress record to stderr.""" + sys.stderr.write(json.dumps({"completed": completed, "total": total, "message": message}) + "\n") + sys.stderr.flush() + + return progress + + def progress(completed, total, message): + """Report progress to stderr so stdout stays parseable.""" + sys.stderr.write("[{}/{}] {}\n".format(completed, total, message)) + sys.stderr.flush() + + return progress +``` + +Add the flag in `main()`, beside the existing arguments: + +```python + parser.add_argument("--machine", action="store_true", help="Emit results as JSON on stdout and progress as JSON on stderr, for a calling process") +``` + +Pass it through where the progress callback is built: + +```python + results = asyncio.run(predictor.run(progress=make_progress(args.quiet, machine=args.machine))) +``` + +Then replace the output section at the end of `main()` so machine mode emits JSON and suppresses the table: + +```python + if args.out: + try: + with open(args.out, "w", encoding="utf-8") as handle: + json.dump(results, handle, indent=2) + except (OSError, TypeError, ValueError) as error: + sys.stderr.write("Could not write results to {}: {}\n".format(args.out, error)) + exit_code = 1 + + if args.machine: + # The parent reads exactly one JSON object from stdout; the human table would + # corrupt it, so it is suppressed rather than merely reordered. + json.dump(results, sys.stdout) + sys.stdout.write("\n") + else: + print(format_table(results)) + + return exit_code +``` + +Ensure `import json` and `import sys` are present at the top of the file. + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd coverage && ./run_all -k annual_cli > /tmp/w2.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w2.txt` + +Expected: no output; both `annual_cli` and `annual_cli_machine` pass. + +- [ ] **Step 5: Verify machine mode by hand** + +```bash +cd coverage && echo "annual: {}" > /tmp/bad.yaml && ./venv/bin/python3 ../apps/predbat/annual_cli.py --config /tmp/bad.yaml --machine; echo "exit=$?" +``` + +Expected: a readable config error on stderr and `exit=2`, with **nothing** on stdout — a parent parsing stdout must not receive half a document on failure. Put the actual output in your report. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +git add apps/predbat/annual_cli.py apps/predbat/tests/test_annual_cli.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/annual_cli.py apps/predbat/tests/test_annual_cli.py apps/predbat/unit_test.py +git commit -m "feat(annual): add machine mode so a parent process can drive the CLI" +``` + +--- + +## Task 3: Subprocess job control + +Owns the child process and nothing else — no HTML, no Storage, no knowledge of what the results mean. That isolation is what lets it be tested against a stub script rather than the real three-minute engine. + +**Files:** +- Create: `apps/predbat/annual_job.py` +- Create: `apps/predbat/tests/test_annual_job.py` +- Create: `apps/predbat/tests/annual_stub.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `annual_job.AnnualJob(log)` with: + - `async start(command) -> bool` — False if a run is already in progress + - `async cancel() -> bool` + - `status() -> dict` of `{state, completed, total, message, elapsed, error}` + - `results` — the parsed results document once state is `complete`, else None + - `state` — one of `idle`, `running`, `complete`, `failed`, `cancelled` + +- [ ] **Step 1: Write the stub child process** + +Create `apps/predbat/tests/annual_stub.py`. This stands in for `annual_cli.py` so no test ever spawns the real engine: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""A stand-in for annual_cli.py --machine, used to drive AnnualJob in tests. + +Behaviour is chosen by argv[1] so one script covers every case the job control +has to survive, without ever running the real three-minute engine. +""" + +import json +import sys +import time + + +def main(): + """Emit the behaviour named by argv[1] and exit with a matching code.""" + mode = sys.argv[1] if len(sys.argv) > 1 else "ok" + + if mode == "ok": + for step in range(1, 4): + sys.stderr.write(json.dumps({"completed": step, "total": 3, "message": "step {}".format(step)}) + "\n") + sys.stderr.flush() + json.dump({"year": 2025, "months": [], "annual": {"months_included": 0}}, sys.stdout) + return 0 + + if mode == "garbage_progress": + sys.stderr.write("not json at all\n") + sys.stderr.write(json.dumps({"completed": 1, "total": 1, "message": "recovered"}) + "\n") + sys.stderr.flush() + json.dump({"year": 2025, "months": [], "annual": {"months_included": 0}}, sys.stdout) + return 0 + + if mode == "fail": + sys.stderr.write("something went wrong\n") + return 3 + + if mode == "bad_output": + sys.stdout.write("this is not json") + return 0 + + if mode == "hang": + while True: + time.sleep(0.1) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 2: Write the failing test** + +Create `apps/predbat/tests/test_annual_job.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's subprocess job control.""" + +import asyncio +import os +import sys + +from annual_job import AnnualJob + +STUB = os.path.join(os.path.dirname(__file__), "annual_stub.py") + + +def stub_command(mode): + """Return the argv for the stub child in the given mode.""" + return [sys.executable, STUB, mode] + + +async def run_to_completion(job, mode, timeout=20): + """Start the stub in the given mode and wait for the job to leave 'running'.""" + started = await job.start(stub_command(mode)) + waited = 0.0 + while job.state == "running" and waited < timeout: + await asyncio.sleep(0.1) + waited += 0.1 + return started + + +def test_annual_job(my_predbat): + """Verify progress parsing, completion, failure, cancellation and refusal to double-run.""" + failed = False + print("**** Testing annual_job ****") + messages = [] + + print("Test: a successful run parses progress and returns the results document") + job = AnnualJob(log=messages.append) + started = asyncio.run(run_to_completion(job, "ok")) + if not started: + print(" ERROR: start() should return True for a fresh job") + failed = True + if job.state != "complete": + print(" ERROR: expected state 'complete', got {} ({})".format(job.state, job.status().get("error"))) + failed = True + if job.status().get("completed") != 3 or job.status().get("total") != 3: + print(" ERROR: final progress should be 3/3, got {}".format(job.status())) + failed = True + if (job.results or {}).get("year") != 2025: + print(" ERROR: the results document should be parsed from stdout, got {}".format(job.results)) + failed = True + + print("Test: a malformed progress line does not crash the parser") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "garbage_progress")) + if job.state != "complete": + print(" ERROR: a garbage progress line should not fail the run, got {}".format(job.state)) + failed = True + if job.status().get("message") != "recovered": + print(" ERROR: parsing should recover after a bad line, got {}".format(job.status())) + failed = True + + print("Test: a non-zero exit is reported as failed, with the child's stderr kept") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "fail")) + if job.state != "failed": + print(" ERROR: expected state 'failed', got {}".format(job.state)) + failed = True + error_text = job.status().get("error") or "" + if "something went wrong" not in error_text: + print(" ERROR: the child's stderr should be reported, got {!r}".format(error_text)) + failed = True + if "3" not in error_text: + print(" ERROR: the exit code should be reported, got {!r}".format(error_text)) + failed = True + + print("Test: unparseable stdout is reported as failed rather than a silent empty result") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "bad_output")) + if job.state != "failed": + print(" ERROR: unparseable output should fail the run, got {}".format(job.state)) + failed = True + if job.results is not None: + print(" ERROR: no results should be exposed after a parse failure, got {}".format(job.results)) + failed = True + + print("Test: a second start while running is refused, and cancel stops the child") + + async def double_start_then_cancel(): + """Start a hanging child, try to start another, then cancel.""" + job = AnnualJob(log=messages.append) + first = await job.start(stub_command("hang")) + await asyncio.sleep(0.5) + second = await job.start(stub_command("hang")) + cancelled = await job.cancel() + waited = 0.0 + while job.state == "running" and waited < 10: + await asyncio.sleep(0.1) + waited += 0.1 + return first, second, cancelled, job + + first, second, cancelled, job = asyncio.run(double_start_then_cancel()) + if not first: + print(" ERROR: the first start should succeed") + failed = True + if second: + print(" ERROR: a second start while running must be refused") + failed = True + if not cancelled: + print(" ERROR: cancel should report that it acted") + failed = True + if job.state != "cancelled": + print(" ERROR: expected state 'cancelled', got {}".format(job.state)) + failed = True + + print("Test: a fresh job reports idle with no results") + job = AnnualJob(log=messages.append) + if job.state != "idle" or job.results is not None: + print(" ERROR: a fresh job should be idle with no results, got {} / {}".format(job.state, job.results)) + failed = True + if job.status().get("elapsed") != 0: + print(" ERROR: an idle job should report zero elapsed, got {}".format(job.status())) + failed = True + + return failed +``` + +Register it in `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_job import test_annual_job +``` + +```python + ("annual_job", test_annual_job, "Annual subprocess job control tests", False), +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test annual_job > /tmp/w3.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/w3.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_job'`. + +- [ ] **Step 4: Create `apps/predbat/annual_job.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Subprocess control for the Annual prediction run. + +The annual engine is one to three minutes of synchronous CPU work - two to six +with a car - so running it inside the web server's event loop would freeze the +whole Predbat interface and the five minute optimiser loop that shares it. It +runs as a child process instead, handing progress back on stderr and the results +document on stdout. + +This module owns the process and nothing else: no HTML, no Storage, no opinion +about what the results mean. That is what lets it be tested against a stub child +rather than the real engine. +""" + +import asyncio +import json +import time + +# How much of the child's stderr to keep for the failure message. Enough to carry +# a traceback, bounded so a chatty failure cannot grow without limit. +MAX_ERROR_LINES = 20 + +# Grace period between asking the child to stop and killing it outright +CANCEL_GRACE_SECONDS = 5.0 + + +class AnnualJob: + """Runs one annual prediction child process at a time and tracks its progress.""" + + def __init__(self, log): + """Create an idle job that logs through the supplied callable.""" + self.log = log + self.state = "idle" + self.completed = 0 + self.total = 0 + self.message = "" + self.error = None + self.results = None + self.started_at = None + self._process = None + self._stderr_tail = [] + + def status(self): + """Return a JSON-serialisable snapshot for the polling endpoint.""" + elapsed = 0 + if self.started_at is not None: + end = self.started_at if self.state == "idle" else time.time() + elapsed = int(end - self.started_at) + return { + "state": self.state, + "completed": self.completed, + "total": self.total, + "message": self.message, + "elapsed": elapsed, + "error": self.error, + } + + async def start(self, command): + """Spawn the child. Returns False when a run is already in progress. + + Refusing rather than queueing is deliberate: two annual runs on the same + machine would compete for the same CPU and both would take twice as long. + """ + if self.state == "running": + self.log("Warn: Annual: a run is already in progress, refusing to start another") + return False + + self.state = "running" + self.completed = 0 + self.total = 0 + self.message = "Starting" + self.error = None + self.results = None + self.started_at = time.time() + self._stderr_tail = [] + + try: + self._process = await asyncio.create_subprocess_exec(*command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) + except (OSError, ValueError) as exception: + self.state = "failed" + self.error = "Could not start the annual run: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) + return False + + asyncio.ensure_future(self._supervise()) + return True + + async def cancel(self): + """Ask the child to stop, killing it if it does not. Returns False if nothing was running.""" + if self.state != "running" or self._process is None: + return False + self.state = "cancelled" + self.message = "Cancelled" + try: + self._process.terminate() + except ProcessLookupError: + return True + try: + await asyncio.wait_for(self._process.wait(), timeout=CANCEL_GRACE_SECONDS) + except asyncio.TimeoutError: + self.log("Warn: Annual: the run did not stop when asked, killing it") + try: + self._process.kill() + except ProcessLookupError: + pass + return True + + async def _read_progress(self, stream): + """Consume the child's stderr, updating progress and keeping a tail for errors.""" + while True: + line = await stream.readline() + if not line: + break + text = line.decode("utf-8", errors="replace").strip() + if not text: + continue + self._stderr_tail.append(text) + if len(self._stderr_tail) > MAX_ERROR_LINES: + self._stderr_tail.pop(0) + try: + record = json.loads(text) + except ValueError: + # Not a progress record - the child is allowed to write plain + # warnings to stderr, and one bad line must not stop the parse. + continue + if isinstance(record, dict) and "completed" in record: + self.completed = record.get("completed", self.completed) + self.total = record.get("total", self.total) + self.message = record.get("message", self.message) + + async def _supervise(self): + """Read both streams to completion, then settle the final state.""" + process = self._process + try: + stdout_data, _ = await asyncio.gather(process.stdout.read(), self._read_progress(process.stderr)) + await process.wait() + except (OSError, ValueError) as exception: + self.state = "failed" + self.error = "The annual run could not be read: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) + return + + if self.state == "cancelled": + return + + tail = "\n".join(self._stderr_tail) + + if process.returncode != 0: + self.state = "failed" + self.error = "The annual run exited with code {}.\n{}".format(process.returncode, tail) + self.log("Warn: Annual: {}".format(self.error)) + return + + try: + self.results = json.loads(stdout_data.decode("utf-8", errors="replace")) + except ValueError as exception: + # A zero exit with unreadable output is worse than a crash: it would + # otherwise render as an empty result that looks like a real answer. + self.state = "failed" + self.results = None + self.error = "The annual run finished but its output could not be read: {}\n{}".format(exception, tail) + self.log("Warn: Annual: {}".format(self.error)) + return + + self.state = "complete" + self.message = "Complete" + if self.total: + self.completed = self.total +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_job > /tmp/w3.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w3.txt` + +Expected: no output. If the cancel case hangs, check that `terminate()` is reached before `wait()` and that the stub's `hang` mode is actually being spawned. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +git add apps/predbat/annual_job.py apps/predbat/tests/test_annual_job.py apps/predbat/tests/annual_stub.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/annual_job.py apps/predbat/tests/test_annual_job.py apps/predbat/tests/annual_stub.py apps/predbat/unit_test.py +git commit -m "feat(annual): add subprocess job control for the Annual tab" +``` + +--- + +## Task 4: The run store + +A five-run ring over a Storage object. Keeping the ring here rather than in the web layer means the eviction rule has one implementation and one test. + +**Files:** +- Create: `apps/predbat/annual_store.py` +- Create: `apps/predbat/tests/test_annual_store.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: a Storage object exposing `async save(module, filename, data, format=...)`, `async load(module, filename)`. +- Produces: + - `annual_store.MAX_RUNS` (5), `annual_store.STORAGE_MODULE` (`"annual"`), `annual_store.INDEX_NAME` (`"runs_index"`) + - `annual_store.build_label(config) -> str` + - `annual_store.save_run(storage, results, config, run_id) -> str` (async) + - `annual_store.list_runs(storage) -> list[dict]` (async) + - `annual_store.load_run(storage, run_id) -> dict | None` (async) + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_annual_store.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's five-run store.""" + +import asyncio + +from annual_store import INDEX_NAME, MAX_RUNS, STORAGE_MODULE, build_label, list_runs, load_run, save_run + + +class FakeStorage: + """An in-memory stand-in for the Storage component.""" + + def __init__(self): + """Start with nothing stored.""" + self.store = {} + self.deleted = [] + + async def save(self, module, filename, data, format="yaml", expiry=None): + """Record a saved value.""" + self.store[(module, filename)] = data + + async def load(self, module, filename): + """Return a stored value, or None.""" + return self.store.get((module, filename)) + + async def delete(self, module, filename): + """Remove a stored value and record that it happened.""" + self.deleted.append(filename) + self.store.pop((module, filename), None) + + +def sample_results(cost): + """Return a minimal results document with a distinguishing cost.""" + return {"year": 2025, "annual": {"scenarios": {"with_predbat": {"cost_p": cost}}, "months_included": 12}, "months": []} + + +def sample_config(size_kwh=9.5): + """Return a minimal validated-shape config.""" + return {"battery": {"size_kwh": size_kwh}, "solar": [{"kwp": 5.6}], "tariff": {"import_octopus_url": "https://example.com/AGILE-24-10-01/x"}} + + +def test_annual_store(my_predbat): + """Verify saving, listing, loading, eviction and label generation.""" + failed = False + print("**** Testing annual_store ****") + + print("Test: a saved run appears in the index and can be loaded back") + storage = FakeStorage() + run_id = asyncio.run(save_run(storage, sample_results(100), sample_config(), "run-1")) + if run_id != "run-1": + print(" ERROR: save_run should return the id it was given, got {}".format(run_id)) + failed = True + index = asyncio.run(list_runs(storage)) + if len(index) != 1 or index[0]["id"] != "run-1": + print(" ERROR: expected one indexed run, got {}".format(index)) + failed = True + loaded = asyncio.run(load_run(storage, "run-1")) + if (loaded or {}).get("annual", {}).get("scenarios", {}).get("with_predbat", {}).get("cost_p") != 100: + print(" ERROR: the loaded run should match what was saved, got {}".format(loaded)) + failed = True + + print("Test: the index is newest-first") + asyncio.run(save_run(storage, sample_results(200), sample_config(), "run-2")) + index = asyncio.run(list_runs(storage)) + if [entry["id"] for entry in index] != ["run-2", "run-1"]: + print(" ERROR: expected newest first, got {}".format([entry["id"] for entry in index])) + failed = True + + print("Test: a sixth run evicts the oldest AND deletes its stored document") + storage = FakeStorage() + for number in range(1, MAX_RUNS + 2): + asyncio.run(save_run(storage, sample_results(number), sample_config(), "run-{}".format(number))) + index = asyncio.run(list_runs(storage)) + if len(index) != MAX_RUNS: + print(" ERROR: the ring should hold {} runs, got {}".format(MAX_RUNS, len(index))) + failed = True + if "run-1" in [entry["id"] for entry in index]: + print(" ERROR: the oldest run should have been evicted from the index") + failed = True + if "run_run-1" not in storage.deleted: + print(" ERROR: the evicted run's document should be deleted, deletions were {}".format(storage.deleted)) + failed = True + if asyncio.run(load_run(storage, "run-1")) is not None: + print(" ERROR: an evicted run should no longer load") + failed = True + + print("Test: loading an unknown or missing run returns None rather than raising") + if asyncio.run(load_run(storage, "does-not-exist")) is not None: + print(" ERROR: an unknown run id should give None") + failed = True + + print("Test: an index entry whose document is missing is reported, not rendered empty") + storage = FakeStorage() + asyncio.run(save_run(storage, sample_results(1), sample_config(), "orphan")) + storage.store.pop((STORAGE_MODULE, "run_orphan")) + if asyncio.run(load_run(storage, "orphan")) is not None: + print(" ERROR: a missing document should give None so the caller can say so") + failed = True + + print("Test: an empty store lists nothing rather than raising") + if asyncio.run(list_runs(FakeStorage())) != []: + print(" ERROR: an empty store should list no runs") + failed = True + + print("Test: the label describes the configuration, not just a timestamp") + label = build_label(sample_config(size_kwh=9.5)) + if "9.5" not in label or "5.6" not in label: + print(" ERROR: the label should name the battery and array size, got {!r}".format(label)) + failed = True + if "Agile" not in label: + print(" ERROR: the label should name the tariff, got {!r}".format(label)) + failed = True + + print("Test: a label is still produced for a config with no battery or solar") + label = build_label({"tariff": {"rates_import": [{"rate": 25.0}]}}) + if not label: + print(" ERROR: a sparse config should still produce a label") + failed = True + + print("Test: the index survives a corrupt stored value") + storage = FakeStorage() + storage.store[(STORAGE_MODULE, INDEX_NAME)] = "not a list" + if asyncio.run(list_runs(storage)) != []: + print(" ERROR: a corrupt index should read as empty rather than raising") + failed = True + + return failed +``` + +Register it in `apps/predbat/unit_test.py`: + +```python +from tests.test_annual_store import test_annual_store +``` + +```python + ("annual_store", test_annual_store, "Annual run store tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test annual_store > /tmp/w4.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/w4.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'annual_store'`. + +- [ ] **Step 3: Create `apps/predbat/annual_store.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Storage-backed history of annual prediction runs. + +Keeps the most recent runs so a user can flip between "with a 5 kWh battery" and +"with a 10 kWh battery" without re-running either. Everything goes through the +Storage abstraction rather than the filesystem, because there may not be one. +""" + +MAX_RUNS = 5 +STORAGE_MODULE = "annual" +INDEX_NAME = "runs_index" + + +def _run_key(run_id): + """Return the storage filename holding one run's results document.""" + return "run_{}".format(run_id) + + +def _describe_tariff(tariff): + """Return a short human name for the tariff a run used.""" + if not isinstance(tariff, dict): + return "tariff" + url = tariff.get("import_octopus_url") or "" + for name in ["AGILE", "INTELLI-FLUX", "INTELLI", "FLUX", "COSY", "SNUG", "GO"]: + if name in url.upper(): + return name.title().replace("Intelli-Flux", "Intelligent Flux").replace("Intelli", "Intelligent Go") + if tariff.get("rates_import"): + return "fixed rates" + return "tariff" + + +def build_label(config): + """Return a short human label describing the configuration a run used. + + A selector listing five bare timestamps tells the user nothing about which + run was which, which defeats the point of keeping more than one. + """ + parts = [] + battery = config.get("battery") if isinstance(config, dict) else None + if isinstance(battery, dict) and battery.get("size_kwh"): + parts.append("{}kWh battery".format(battery["size_kwh"])) + else: + parts.append("no battery") + + solar = config.get("solar") if isinstance(config, dict) else None + if solar: + total_kwp = sum(array.get("kwp", 0) for array in solar if isinstance(array, dict)) + if total_kwp: + parts.append("{}kWp".format(round(total_kwp, 2))) + else: + parts.append("no solar") + + parts.append(_describe_tariff((config or {}).get("tariff"))) + return " · ".join(parts) + + +async def list_runs(storage): + """Return the stored runs newest-first, or an empty list when there are none. + + A corrupt or unexpected index reads as empty rather than raising: the tab + must still render so the user can start a fresh run. + """ + if not storage: + return [] + index = await storage.load(STORAGE_MODULE, INDEX_NAME) + if not isinstance(index, list): + return [] + return [entry for entry in index if isinstance(entry, dict) and entry.get("id")] + + +async def load_run(storage, run_id): + """Return one run's results document, or None when it is unknown or missing.""" + if not storage or not run_id: + return None + return await storage.load(STORAGE_MODULE, _run_key(run_id)) + + +async def save_run(storage, results, config, run_id): + """Save a completed run and prune the ring to MAX_RUNS. Returns the run id. + + The evicted run's document is deleted as well as its index entry, so the ring + cannot leak documents that nothing references. + """ + if not storage: + return run_id + + await storage.save(STORAGE_MODULE, _run_key(run_id), results, format="json") + + annual = results.get("annual", {}) if isinstance(results, dict) else {} + entry = { + "id": run_id, + "timestamp": run_id, + "label": build_label(config), + "months_included": annual.get("months_included", 0), + "status": "ok" if annual.get("months_included") else "empty", + } + + index = await list_runs(storage) + index = [existing for existing in index if existing.get("id") != run_id] + index.insert(0, entry) + + for dropped in index[MAX_RUNS:]: + if hasattr(storage, "delete"): + await storage.delete(STORAGE_MODULE, _run_key(dropped["id"])) + index = index[:MAX_RUNS] + + await storage.save(STORAGE_MODULE, INDEX_NAME, index, format="json") + return run_id +``` + +- [ ] **Step 4: Check the Storage component actually has `delete`** + +The fake in the test provides `delete`, and `save_run` guards with `hasattr`. Confirm what the real component offers: + +Run: `grep -n "async def delete\|def delete" apps/predbat/storage.py` + +If there is no `delete`, say so in your report and instead overwrite the evicted key with `None` (still through `storage.save`) so the ring does not leave a live document behind. Do not invent a filesystem call. + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test annual_store > /tmp/w4.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w4.txt` + +Expected: no output. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +git add apps/predbat/annual_store.py apps/predbat/tests/test_annual_store.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/annual_store.py apps/predbat/tests/test_annual_store.py apps/predbat/unit_test.py +git commit -m "feat(annual): add the five-run store for the Annual tab" +``` + +--- + +## Task 5: Prefill and config persistence + +Reads what the live instance knows and fills the rest from a typical UK system. This is the task that makes the unconfigured case work, so its test is the acceptance criterion for that requirement. + +**Files:** +- Create: `apps/predbat/web_annual.py` +- Create: `apps/predbat/tests/test_web_annual.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `tariff_catalogue.merged_catalogue`. +- Produces: + - `web_annual.DEFAULT_CONFIG` — the typical-UK example as a dict + - `web_annual.AnnualPage(web_interface)` with: + - `prefill_config() -> dict` + - `is_configured() -> bool` + - `load_config() -> dict` + - `save_config(config)` + - `catalogue() -> list[dict]` + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_web_annual.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's prefill and configuration handling.""" + +from annual import validate_config +from web import WebInterface +from web_annual import DEFAULT_CONFIG, AnnualPage + + +def make_page(my_predbat): + """Return an AnnualPage backed by a WebInterface over the test fixture.""" + return AnnualPage(WebInterface(my_predbat, web_port=5054)) + + +def test_web_annual(my_predbat): + """Verify prefill against a configured and an unconfigured instance.""" + failed = False + print("**** Testing web_annual prefill ****") + + saved_args = dict(my_predbat.args) + try: + print("Test: an unconfigured instance still produces a complete, valid config") + # This is the acceptance criterion for "must work with Predbat unconfigured": + # a prospective buyer, and eventually an unregistered Predbat.com visitor, + # arrives with none of this set. + for key in ["soc_max", "inverter_limit", "export_limit", "open_meteo_forecast", "forecast_solar", "compare_list", "dno_region"]: + my_predbat.args.pop(key, None) + page = make_page(my_predbat) + config = page.prefill_config() + try: + validate_config(config) + except Exception as error: + print(" ERROR: an unconfigured prefill must still validate, got {}".format(error)) + failed = True + if page.is_configured(): + print(" ERROR: with no battery and no solar the page should report unconfigured") + failed = True + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: battery should fall back to the default, got {}".format(config["battery"])) + failed = True + if not config["solar"]: + print(" ERROR: solar should fall back to the default array") + failed = True + + print("Test: a zero soc_max counts as unset and falls back to the default") + my_predbat.args["soc_max"] = 0 + config = make_page(my_predbat).prefill_config() + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: a zero soc_max should fall back, got {}".format(config["battery"]["size_kwh"])) + failed = True + + print("Test: configured values are read from args and used") + my_predbat.args["soc_max"] = 12.5 + my_predbat.args["open_meteo_forecast"] = [{"kwp": 7.2, "declination": 30, "azimuth": 170, "efficiency": 0.9}] + config = make_page(my_predbat).prefill_config() + if config["battery"]["size_kwh"] != 12.5: + print(" ERROR: soc_max from args should be used, got {}".format(config["battery"]["size_kwh"])) + failed = True + if config["solar"][0]["kwp"] != 7.2 or config["solar"][0]["azimuth"] != 170: + print(" ERROR: the solar array should come from args, got {}".format(config["solar"])) + failed = True + + print("Test: prefill is per-field, not all-or-nothing") + # Solar configured but no battery: the real array must survive alongside the + # default battery rather than the whole prefill collapsing to defaults. + my_predbat.args.pop("soc_max", None) + config = make_page(my_predbat).prefill_config() + if config["solar"][0]["kwp"] != 7.2: + print(" ERROR: configured solar should survive an absent battery, got {}".format(config["solar"])) + failed = True + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: the battery should still fall back, got {}".format(config["battery"])) + failed = True + if not make_page(my_predbat).is_configured(): + print(" ERROR: a configured solar array alone should count as configured") + failed = True + + print("Test: the catalogue merges the user's compare_list") + my_predbat.args["compare_list"] = [{"id": "mine", "name": "My tariff", "rates_import_octopus_url": "https://example.com/x"}] + ids = [entry["id"] for entry in make_page(my_predbat).catalogue()] + if "mine" not in ids: + print(" ERROR: a user compare_list entry should appear in the catalogue, got {}".format(ids)) + failed = True + if "agile_agile" not in ids: + print(" ERROR: built-in entries should still be present, got {}".format(ids)) + failed = True + + print("Test: the default config validates on its own") + try: + validate_config(DEFAULT_CONFIG) + except Exception as error: + print(" ERROR: DEFAULT_CONFIG must be valid, got {}".format(error)) + failed = True + + finally: + my_predbat.args.clear() + my_predbat.args.update(saved_args) + + return failed +``` + +Register it in `apps/predbat/unit_test.py`: + +```python +from tests.test_web_annual import test_web_annual +``` + +```python + ("web_annual", test_web_annual, "Annual web tab prefill tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test web_annual > /tmp/w5.txt 2>&1; grep -E "ERROR|ModuleNotFound" /tmp/w5.txt` + +Expected: FAIL with `ModuleNotFoundError: No module named 'web_annual'`. + +- [ ] **Step 3: Create `apps/predbat/web_annual.py` with prefill and config handling** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""The Annual prediction tab. + +Renders the configuration form, drives the subprocess that runs the annual +prediction engine, and presents the results. Prefills from whatever the live +Predbat instance knows and falls back to a typical UK system for the rest, so +the tab is usable by someone who has not configured Predbat at all - which is +the prospective-buyer path the tool exists to serve. +""" + +import copy +import os + +import yaml + +from tariff_catalogue import merged_catalogue + +# A plausible UK home, used for any field the live instance cannot supply. These +# are an EXAMPLE, not a recommendation - the form says so, because a visitor +# could otherwise mistake them for a reading of their own system. +DEFAULT_CONFIG = { + "location": {"postcode": "SW1A 1AA"}, + "solar": [{"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}], + "battery": {"size_kwh": 9.5, "inverter_kw": 5.0, "export_limit_kw": 5.0, "hybrid": True}, + "load": {"annual_kwh": 3800, "shape": "flat", "car_charging_kwh": 0, "car_rate_kw": 7.4}, + "tariff": {"rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}], "standing_charge_p_per_day": 60.0}, + "samples_per_month": 2, +} + +CONFIG_FILENAME = "annual.yaml" + + +class AnnualPage: + """Renders and drives the Annual prediction tab.""" + + def __init__(self, web_interface): + """Attach to the running web interface so args and Storage are reachable.""" + self.web = web_interface + self.base = web_interface.base + self.log = web_interface.log + + def _arg(self, name, default=None): + """Read one configuration value from the in-memory args dictionary. + + Never reads apps.yaml from disk: the file may not exist at all in some + deployments, which is exactly where the unconfigured case matters most. + """ + try: + return self.base.get_arg(name, default) + except Exception: + return default + + def _solar_from_args(self): + """Return the configured solar arrays, or an empty list. + + open_meteo_forecast and forecast_solar are already lists of + {kwp, declination, azimuth, efficiency}, which is the annual engine's own + shape, so no translation is needed. + """ + for name in ["open_meteo_forecast", "forecast_solar"]: + configured = self._arg(name, None) + if isinstance(configured, dict): + configured = [configured] + if isinstance(configured, list) and configured: + arrays = [] + for entry in configured: + if not isinstance(entry, dict) or not entry.get("kwp"): + continue + arrays.append( + { + "kwp": entry.get("kwp"), + "declination": entry.get("declination", DEFAULT_CONFIG["solar"][0]["declination"]), + "azimuth": entry.get("azimuth", DEFAULT_CONFIG["solar"][0]["azimuth"]), + "efficiency": entry.get("efficiency", DEFAULT_CONFIG["solar"][0]["efficiency"]), + } + ) + if arrays: + return arrays + return [] + + def _location_from_args(self): + """Return the configured location, taken from the solar entries if present.""" + for name in ["open_meteo_forecast", "forecast_solar"]: + configured = self._arg(name, None) + if isinstance(configured, dict): + configured = [configured] + for entry in configured or []: + if not isinstance(entry, dict): + continue + if entry.get("postcode"): + return {"postcode": entry["postcode"]} + if entry.get("latitude") is not None and entry.get("longitude") is not None: + return {"latitude": entry["latitude"], "longitude": entry["longitude"]} + return None + + def is_configured(self): + """Return True when the live instance has a battery or a solar array. + + Those two are what signal a configured system; with neither, the form + shows a banner saying the values on screen are examples. + """ + battery_kwh = self._arg("soc_max", 0) or 0 + try: + battery_kwh = float(battery_kwh) + except (TypeError, ValueError): + battery_kwh = 0 + return battery_kwh > 0 or bool(self._solar_from_args()) + + def prefill_config(self): + """Build a complete config from the live instance, filling gaps with the example. + + Every field falls back independently, so a half-configured Predbat gets its + real values alongside example ones rather than all-or-nothing. + """ + config = copy.deepcopy(DEFAULT_CONFIG) + + location = self._location_from_args() + if location: + config["location"] = location + + arrays = self._solar_from_args() + if arrays: + config["solar"] = arrays + + battery_kwh = self._arg("soc_max", 0) or 0 + try: + battery_kwh = float(battery_kwh) + except (TypeError, ValueError): + battery_kwh = 0 + # A zero or absent soc_max means it is not set - fall back so the user can adjust + if battery_kwh > 0: + config["battery"]["size_kwh"] = battery_kwh + + for arg_name, field, divisor in [("inverter_limit", "inverter_kw", 1000.0), ("export_limit", "export_limit_kw", 1000.0)]: + watts = self._arg(arg_name, 0) or 0 + try: + watts = float(watts) + except (TypeError, ValueError): + watts = 0 + if watts > 0: + config["battery"][field] = round(watts / divisor, 2) + + inverter_type = self._arg("inverter_type", None) + if inverter_type: + config["battery"]["hybrid"] = True + + import_url = self._arg("rates_import_octopus_url", None) + export_url = self._arg("rates_export_octopus_url", None) + if import_url: + config["tariff"] = {"import_octopus_url": import_url, "standing_charge_p_per_day": DEFAULT_CONFIG["tariff"]["standing_charge_p_per_day"]} + if export_url: + config["tariff"]["export_octopus_url"] = export_url + + dno_region = self._arg("dno_region", None) + if dno_region: + config["tariff"]["dno_region"] = dno_region + + return config + + def catalogue(self): + """Return the tariff dropdown entries: built-ins merged with the user's own.""" + return merged_catalogue(self._arg("compare_list", None)) + + def _config_path(self): + """Return the path of the saved annual configuration.""" + return os.path.join(self.base.config_root, CONFIG_FILENAME) + + def load_config(self): + """Return the saved configuration, or a fresh prefill when none exists.""" + path = self._config_path() + try: + if os.path.exists(path): + with open(path, "r", encoding="utf-8") as handle: + saved = yaml.safe_load(handle) + if isinstance(saved, dict) and saved: + return saved.get("annual", saved) + except (OSError, yaml.YAMLError) as error: + self.log("Warn: Annual: could not read {}: {}".format(path, error)) + return self.prefill_config() + + def save_config(self, config): + """Write the configuration so the CLI subprocess can consume it directly.""" + path = self._config_path() + try: + with open(path, "w", encoding="utf-8") as handle: + yaml.safe_dump({"annual": config}, handle, default_flow_style=False, allow_unicode=True) + except OSError as error: + self.log("Warn: Annual: could not write {}: {}".format(path, error)) + raise +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all --test web_annual > /tmp/w5.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w5.txt` + +Expected: no output. If `WebInterface(my_predbat, web_port=5054)` raises, check how `tests/test_web_functions.py` constructs it and match that. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +git add apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +git commit -m "feat(annual): add Annual tab prefill and config persistence" +``` + +--- + +## Task 6: The form + +**Files:** +- Modify: `apps/predbat/web_annual.py` +- Modify: `apps/predbat/tests/test_web_annual.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `AnnualPage.prefill_config`, `load_config`, `catalogue`, `is_configured` from Task 5; `tariff_catalogue.CUSTOM_ID`. +- Produces: + - `AnnualPage.render_form(config, errors=None) -> str` + - `AnnualPage.config_from_post(postdata) -> dict` + +- [ ] **Step 1: Write the failing test** + +Append to `apps/predbat/tests/test_web_annual.py` (add `CUSTOM_ID` to the imports from `tariff_catalogue`): + +```python +def test_web_annual_form(my_predbat): + """Verify the form renders every group, reflects config, and round-trips a post.""" + failed = False + print("**** Testing web_annual form ****") + + saved_args = dict(my_predbat.args) + try: + for key in ["soc_max", "open_meteo_forecast", "forecast_solar"]: + my_predbat.args.pop(key, None) + page = make_page(my_predbat) + config = page.prefill_config() + html = page.render_form(config) + + print("Test: every configuration group is present") + for heading in ["Location", "Solar", "Battery", "Load", "Tariff", "Advanced"]: + if heading not in html: + print(" ERROR: the form is missing the '{}' group".format(heading)) + failed = True + + print("Test: an unconfigured instance gets the example-values banner") + if "example values" not in html.lower(): + print(" ERROR: an unconfigured instance should be told these are examples") + failed = True + + print("Test: a configured instance does NOT get the banner") + my_predbat.args["soc_max"] = 10.0 + configured_html = make_page(my_predbat).render_form(make_page(my_predbat).prefill_config()) + if "example values" in configured_html.lower(): + print(" ERROR: a configured instance should not be told its values are examples") + failed = True + my_predbat.args.pop("soc_max", None) + + print("Test: the tariff dropdown lists the catalogue and a Custom entry") + if CUSTOM_ID not in html: + print(" ERROR: the dropdown should offer a Custom entry") + failed = True + if "Agile import / Agile export" not in html: + print(" ERROR: the dropdown should list the built-in tariffs") + failed = True + + print("Test: the load source is a radio pair, not two independent sections") + if html.count('type="radio"') < 2: + print(" ERROR: expected a radio pair for the load source") + failed = True + if "octopus" not in html.lower(): + print(" ERROR: the Octopus load option should be offered") + failed = True + + print("Test: current values are rendered into the inputs") + if 'value="3800"' not in html.replace("'", '"'): + print(" ERROR: the annual kWh value should appear in the form") + failed = True + + print("Test: validation errors are shown with the form still populated") + html_with_error = page.render_form(config, errors="annual.solar[0] is missing kwp") + if "annual.solar[0] is missing kwp" not in html_with_error: + print(" ERROR: the error message should be displayed") + failed = True + if 'value="3800"' not in html_with_error.replace("'", '"'): + print(" ERROR: the form should stay populated when an error is shown") + failed = True + + print("Test: config_from_post rebuilds a config the engine accepts") + postdata = { + "postcode": "SW1A 1AA", + "solar_kwp_0": "5.6", + "solar_declination_0": "35", + "solar_azimuth_0": "180", + "solar_efficiency_0": "0.95", + "battery_size_kwh": "9.5", + "battery_inverter_kw": "5.0", + "battery_export_limit_kw": "5.0", + "battery_hybrid": "on", + "load_source": "manual", + "load_annual_kwh": "3800", + "load_shape": "flat", + "load_car_charging_kwh": "2500", + "load_car_rate_kw": "7.4", + "tariff_id": CUSTOM_ID, + "tariff_import_url": "https://example.com/import/", + "tariff_export_url": "https://example.com/export/", + "tariff_standing_charge": "60.0", + "samples_per_month": "2", + } + rebuilt = page.config_from_post(postdata) + try: + validate_config(rebuilt) + except Exception as error: + print(" ERROR: a posted form should rebuild into a valid config, got {}".format(error)) + failed = True + if rebuilt["load"]["car_charging_kwh"] != 2500: + print(" ERROR: car charging should survive the round trip, got {}".format(rebuilt["load"])) + failed = True + + print("Test: choosing the Octopus load source drops the manual figures") + postdata["load_source"] = "octopus" + postdata["load_octopus_api_key"] = "sk_test" + postdata["load_octopus_account_id"] = "A-1234ABCD" + rebuilt = page.config_from_post(postdata) + if "annual_kwh" in rebuilt["load"] or "car_charging_kwh" in rebuilt["load"]: + print(" ERROR: the manual figures must not be sent alongside Octopus, got {}".format(rebuilt["load"])) + failed = True + try: + validate_config(rebuilt) + except Exception as error: + print(" ERROR: the Octopus form should rebuild into a valid config, got {}".format(error)) + failed = True + + finally: + my_predbat.args.clear() + my_predbat.args.update(saved_args) + + return failed +``` + +Register it: + +```python +from tests.test_web_annual import test_web_annual, test_web_annual_form +``` + +```python + ("web_annual_form", test_web_annual_form, "Annual web tab form tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test web_annual_form > /tmp/w6.txt 2>&1; grep -E "ERROR|AttributeError" /tmp/w6.txt` + +Expected: FAIL — `AnnualPage` has no `render_form`. + +- [ ] **Step 3: Add the form to `apps/predbat/web_annual.py`** + +Add `from tariff_catalogue import CUSTOM_ID, merged_catalogue` to the imports, then append these methods to `AnnualPage`: + +```python + def _number_field(self, name, label, value, step="any", suffix=""): + """Return one labelled numeric input row.""" + return '
{suffix}
\n'.format( + name=name, label=label, step=step, value=value if value is not None else "", suffix=" {}".format(suffix) if suffix else "" + ) + + def _text_field(self, name, label, value): + """Return one labelled text input row.""" + return '
\n'.format( + name=name, label=label, value=value if value is not None else "" + ) + + def render_form(self, config, errors=None): + """Return the configuration form as HTML, populated from ``config``. + + ``errors`` is displayed above the form with every field left as the user + entered it - losing their input on a validation failure would be worse + than the failure. + """ + solar = config.get("solar") or [{}] + battery = config.get("battery") or {} + load = config.get("load") or {} + tariff = config.get("tariff") or {} + location = config.get("location") or {} + + text = '
\n' + + if errors: + text += '
Could not run: {}
\n'.format(errors) + + if not self.is_configured(): + text += '
Predbat isn\'t configured yet — these are example values, edit them to match your home.
\n' + + text += '
\n' + + text += '
Location\n' + text += self._text_field("postcode", "Postcode", location.get("postcode", "")) + text += self._number_field("latitude", "Latitude (instead of postcode)", location.get("latitude")) + text += self._number_field("longitude", "Longitude", location.get("longitude")) + text += "
\n" + + text += '
Solar\n' + for index, array in enumerate(solar): + text += '
Array {}\n'.format(index + 1) + text += self._number_field("solar_kwp_{}".format(index), "Peak power", array.get("kwp"), suffix="kWp") + text += self._number_field("solar_declination_{}".format(index), "Pitch", array.get("declination", 35), suffix="degrees") + text += self._number_field("solar_azimuth_{}".format(index), "Azimuth (180 = south)", array.get("azimuth", 180), suffix="degrees") + text += "
\n" + text += "
\n" + + text += '
Battery\n' + text += self._number_field("battery_size_kwh", "Usable capacity", battery.get("size_kwh"), suffix="kWh") + text += self._number_field("battery_inverter_kw", "Inverter size", battery.get("inverter_kw"), suffix="kW") + text += self._number_field("battery_export_limit_kw", "Export limit", battery.get("export_limit_kw"), suffix="kW") + text += '
\n'.format("checked" if battery.get("hybrid", True) else "") + text += "
\n" + + using_octopus = "octopus" in load + text += '
Load\n' + text += '
\n'.format("" if using_octopus else "checked") + text += '
\n' + text += self._number_field("load_annual_kwh", "Annual consumption", load.get("annual_kwh", DEFAULT_CONFIG["load"]["annual_kwh"]), suffix="kWh") + shape = load.get("shape", "flat") + text += '
\n" + text += self._number_field("load_car_charging_kwh", "Car charging per year (0 for none)", load.get("car_charging_kwh", 0), suffix="kWh") + text += self._number_field("load_car_rate_kw", "Charger power", load.get("car_rate_kw", 7.4), suffix="kW") + text += "
\n" + text += '
\n'.format("checked" if using_octopus else "") + text += '
\n' + text += self._text_field("load_octopus_api_key", "Octopus API key", (load.get("octopus") or {}).get("api_key", "")) + text += self._text_field("load_octopus_account_id", "Account ID", (load.get("octopus") or {}).get("account_id", "")) + text += '

Your meter readings already include any car charging, so the figures above are not used with this option.

\n' + text += "
\n" + text += "
\n" + + text += '
Tariff\n' + text += '
\n" + text += self._text_field("tariff_import_url", "Import rates URL", tariff.get("import_octopus_url", "")) + text += self._text_field("tariff_export_url", "Export rates URL", tariff.get("export_octopus_url", "")) + text += self._text_field("tariff_dno_region", "Octopus region letter", tariff.get("dno_region", "")) + text += self._number_field("tariff_standing_charge", "Standing charge", tariff.get("standing_charge_p_per_day", 60.0), suffix="p/day") + text += "
\n" + + text += '
Advanced\n' + text += self._number_field("year", "Year to model (blank for the most recent complete year)", config.get("year")) + text += self._number_field("samples_per_month", "Days sampled per month", config.get("samples_per_month", 2), step="1") + text += self._number_field("pv10_derate_fallback", "P10 fallback derate", config.get("pv10_derate_fallback", 0.7)) + for index, array in enumerate(solar): + text += self._number_field("solar_efficiency_{}".format(index), "Array {} efficiency".format(index + 1), array.get("efficiency", 0.95)) + text += "
\n" + + text += '\n' + text += "
\n
\n" + return text + + def config_from_post(self, postdata): + """Rebuild a config dict from submitted form fields. + + Values are left as the strings the browser sent; validate_config() in the + engine does the coercion and range checking, so there is exactly one place + that decides what a valid number is. + """ + + def value(name, default=None): + """Return one posted field, or the default when absent or blank.""" + raw = postdata.get(name) + if raw is None or str(raw).strip() == "": + return default + return str(raw).strip() + + config = {} + + location = {} + if value("postcode"): + location["postcode"] = value("postcode") + if value("latitude") is not None and value("longitude") is not None: + location["latitude"] = value("latitude") + location["longitude"] = value("longitude") + config["location"] = location + + arrays = [] + index = 0 + while value("solar_kwp_{}".format(index)) is not None: + arrays.append( + { + "kwp": value("solar_kwp_{}".format(index)), + "declination": value("solar_declination_{}".format(index), 35), + "azimuth": value("solar_azimuth_{}".format(index), 180), + "efficiency": value("solar_efficiency_{}".format(index), 0.95), + } + ) + index += 1 + if arrays: + config["solar"] = arrays + + if value("battery_size_kwh") is not None: + config["battery"] = { + "size_kwh": value("battery_size_kwh"), + "inverter_kw": value("battery_inverter_kw", 5.0), + "export_limit_kw": value("battery_export_limit_kw", 5.0), + "hybrid": bool(postdata.get("battery_hybrid")), + } + + # The engine rejects an Octopus block alongside manual figures, because the + # meter series already contains any car charging. Send one or the other. + if value("load_source", "manual") == "octopus": + config["load"] = {"octopus": {"api_key": value("load_octopus_api_key", ""), "account_id": value("load_octopus_account_id", "")}} + else: + config["load"] = { + "annual_kwh": value("load_annual_kwh", 3800), + "shape": value("load_shape", "flat"), + "car_charging_kwh": value("load_car_charging_kwh", 0), + "car_rate_kw": value("load_car_rate_kw", 7.4), + } + + tariff = {"standing_charge_p_per_day": value("tariff_standing_charge", 0)} + if value("tariff_import_url"): + tariff["import_octopus_url"] = value("tariff_import_url") + if value("tariff_export_url"): + tariff["export_octopus_url"] = value("tariff_export_url") + if value("tariff_dno_region"): + tariff["dno_region"] = value("tariff_dno_region") + if not tariff.get("import_octopus_url"): + tariff["rates_import"] = DEFAULT_CONFIG["tariff"]["rates_import"] + tariff["rates_export"] = DEFAULT_CONFIG["tariff"]["rates_export"] + config["tariff"] = tariff + + if value("year"): + config["year"] = value("year") + config["samples_per_month"] = value("samples_per_month", 2) + if value("pv10_derate_fallback"): + config["pv10_derate_fallback"] = value("pv10_derate_fallback") + + return config +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd coverage && ./run_all -k web_annual > /tmp/w6.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w6.txt` + +Expected: no output; both `web_annual` and `web_annual_form` pass. + +- [ ] **Step 5: Run pre-commit and commit** + +```bash +git add apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +git commit -m "feat(annual): render the Annual tab configuration form" +``` + +--- + +## Task 7: Routes and wiring + +**Files:** +- Modify: `apps/predbat/web_annual.py` +- Modify: `apps/predbat/web.py` +- Modify: `apps/predbat/web_helper.py` +- Modify: `apps/predbat/tests/test_web_annual.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual_job.AnnualJob`, `annual_store.save_run/list_runs/load_run`, `annual.validate_config`, `AnnualPage.render_form/config_from_post/save_config/load_config`. +- Produces: + - `AnnualPage.html_annual(request)`, `html_annual_post`, `html_annual_run`, `html_annual_status`, `html_annual_cancel` — all async aiohttp handlers + - `AnnualPage.cli_command(config_path) -> list[str]` + +- [ ] **Step 1: Write the failing test** + +Append to `apps/predbat/tests/test_web_annual.py`: + +```python +def test_web_annual_routes(my_predbat): + """Verify the run command, validation gating and the status payload.""" + import asyncio + + failed = False + print("**** Testing web_annual routes ****") + + page = make_page(my_predbat) + + print("Test: the CLI command targets annual_cli.py in machine mode") + command = page.cli_command("/tmp/annual.yaml") + if "--machine" not in command: + print(" ERROR: the child must be run in machine mode, got {}".format(command)) + failed = True + if not any("annual_cli.py" in part for part in command): + print(" ERROR: the command should invoke annual_cli.py, got {}".format(command)) + failed = True + if "--config" not in command or "/tmp/annual.yaml" not in command: + print(" ERROR: the config path should be passed, got {}".format(command)) + failed = True + + print("Test: the status payload is JSON-serialisable and names its state") + status = asyncio.run(page.status_payload()) + for key in ["state", "completed", "total", "message", "elapsed"]: + if key not in status: + print(" ERROR: status is missing '{}', got {}".format(key, status)) + failed = True + if status.get("state") != "idle": + print(" ERROR: a fresh page should report idle, got {}".format(status.get("state"))) + failed = True + + print("Test: an invalid config is rejected before anything is spawned") + bad = {"location": {}, "load": {"annual_kwh": 1}, "tariff": {"rates_import": [{"rate": 5}]}} + error = page.validation_error(bad) + if not error: + print(" ERROR: an invalid config should produce an error message") + failed = True + if page.job.state != "idle": + print(" ERROR: validation must not start a job, state was {}".format(page.job.state)) + failed = True + + print("Test: a valid config produces no error") + if page.validation_error(page.prefill_config()): + print(" ERROR: the prefill config should validate, got {}".format(page.validation_error(page.prefill_config()))) + failed = True + + return failed +``` + +Register it: + +```python +from tests.test_web_annual import test_web_annual, test_web_annual_form, test_web_annual_routes +``` + +```python + ("web_annual_routes", test_web_annual_routes, "Annual web tab route tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test web_annual_routes > /tmp/w7.txt 2>&1; grep -E "ERROR|AttributeError" /tmp/w7.txt` + +Expected: FAIL — `AnnualPage` has no `cli_command`. + +- [ ] **Step 3: Add the handlers to `apps/predbat/web_annual.py`** + +Add to the imports: + +```python +import datetime +import sys + +from aiohttp import web + +from annual import AnnualConfigError, validate_config +from annual_job import AnnualJob +from annual_store import list_runs, load_run, save_run +``` + +Add to `AnnualPage.__init__`, after `self.log`: + +```python + self.job = AnnualJob(log=self.log) + self.last_error = None +``` + +Then append: + +```python + def cli_command(self, config_path): + """Return the argv for the child process that performs the run.""" + script = os.path.join(os.path.dirname(os.path.abspath(__file__)), "annual_cli.py") + work_dir = os.path.join(self.base.config_root, "annual_work") + return [sys.executable, script, "--config", config_path, "--work-dir", work_dir, "--machine"] + + def validation_error(self, config): + """Return a message when the config is invalid, or None when it is fine. + + Validation runs here for immediate feedback, but the same validate_config + runs again inside the child and remains the authority. + """ + try: + validate_config(config) + except AnnualConfigError as error: + return str(error) + return None + + def _storage(self): + """Return the Storage component, or None when it is unavailable.""" + components = getattr(self.base, "components", None) + return components.get_component("storage") if components else None + + async def status_payload(self): + """Return the polling payload: job state plus the stored run list.""" + payload = self.job.status() + payload["runs"] = await list_runs(self._storage()) + return payload + + async def _store_completed_run(self, config): + """Save a finished run into the ring, once.""" + if self.job.state != "complete" or self.job.results is None: + return + run_id = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + await save_run(self._storage(), self.job.results, config, run_id) + self.job.results = None + + async def html_annual(self, request): + """Render the Annual tab: the form, then the selected run's results.""" + self.web.default_page = "./annual" + config = self.load_config() + + text = self.web.get_header("Predbat Annual") + text += "\n" + text += self.render_css() + text += self.render_form(config, errors=self.last_error) + text += self.render_progress() + + storage = self._storage() + runs = await list_runs(storage) + selected = request.query.get("run") or (runs[0]["id"] if runs else None) + results = await load_run(storage, selected) if selected else None + text += self.render_results(results, runs, selected) + text += self.render_script() + text += "\n" + return web.Response(content_type="text/html", text=text) + + async def html_annual_post(self, request): + """Save the configuration without running.""" + postdata = await request.post() + config = self.config_from_post(postdata) + self.last_error = self.validation_error(config) + if not self.last_error: + self.save_config(config) + return await self.html_annual(request) + + async def html_annual_run(self, request): + """Validate, save, and spawn the run.""" + postdata = await request.post() + config = self.config_from_post(postdata) + self.last_error = self.validation_error(config) + if self.last_error: + return await self.html_annual(request) + + self.save_config(config) + self._running_config = config + started = await self.job.start(self.cli_command(self._config_path())) + if not started and self.job.state != "running": + self.last_error = self.job.status().get("error") or "The run could not be started" + return await self.html_annual(request) + + async def html_annual_status(self, request): + """Return the job status as JSON for the page to poll.""" + if self.job.state == "complete" and self.job.results is not None: + await self._store_completed_run(getattr(self, "_running_config", self.load_config())) + return web.json_response(await self.status_payload()) + + async def html_annual_cancel(self, request): + """Cancel a running job.""" + await self.job.cancel() + return web.json_response(self.job.status()) + + async def html_annual_download(self, request): + """Return one stored run's raw results document as a JSON download.""" + run_id = request.query.get("run") + results = await load_run(self._storage(), run_id) + if results is None: + return web.json_response({"error": "No stored run with id {}".format(run_id)}, status=404) + return web.json_response(results, headers={"Content-Disposition": 'attachment; filename="annual-{}.json"'.format(run_id)}) +``` + +- [ ] **Step 4: Add the progress, CSS and script fragments** + +Note the `render_results` placeholder below: `html_annual` calls it, so the module +would not render without one. The results task replaces it wholesale — replace it, +do not add a second definition. + +Append to `AnnualPage`: + +```python + def render_results(self, results, runs, selected_id): + """Placeholder replaced in full by the results task; keeps the page renderable.""" + return "

No results yet — fill in the form above and press Run.

\n" + + def render_css(self): + """Return the scoped styles for the tab.""" + return """ +""" + + def render_progress(self): + """Return the progress area, hidden until a run starts.""" + return """ +""" + + def render_script(self): + """Return the polling and tariff-picker script.""" + return """ +""" +``` + +- [ ] **Step 5: Wire it into `apps/predbat/web.py`** + +Add the import beside the other web module imports (near `from web_metrics_dashboard import ...`): + +```python +from web_annual import AnnualPage +``` + +In `WebInterface.__init__`, after the other attribute setup, create the page: + +```python + self.annual_page = AnnualPage(self) +``` + +Register the routes beside the existing `app.router.add_get("/compare", ...)` lines: + +```python + app.router.add_get("/annual", self.annual_page.html_annual) + app.router.add_post("/annual", self.annual_page.html_annual_post) + app.router.add_post("/annual_run", self.annual_page.html_annual_run) + app.router.add_get("/annual_status", self.annual_page.html_annual_status) + app.router.add_post("/annual_cancel", self.annual_page.html_annual_cancel) + app.router.add_get("/annual_download", self.annual_page.html_annual_download) +``` + +- [ ] **Step 6: Add the nav link in `apps/predbat/web_helper.py`** + +Find `Compare` and add immediately after it: + +```html +Annual +``` + +- [ ] **Step 7: Run the tests** + +Run: `cd coverage && ./run_all -k web_annual > /tmp/w7.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w7.txt` + +Then the existing web tests, since `web.py` changed: + +Run: `./run_all -k web_ > /tmp/w7b.txt 2>&1; grep -E "ERROR|FAILED|Traceback" /tmp/w7b.txt` + +Expected: no output from either. + +- [ ] **Step 8: Run pre-commit and commit** + +```bash +git add apps/predbat/web_annual.py apps/predbat/web.py apps/predbat/web_helper.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/web_annual.py apps/predbat/web.py apps/predbat/web_helper.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +git commit -m "feat(annual): wire the Annual tab routes and navigation" +``` + +--- + +## Task 8: Results, chart and run selector + +**Files:** +- Modify: `apps/predbat/web_annual.py` +- Modify: `apps/predbat/tests/test_web_annual.py` +- Modify: `apps/predbat/unit_test.py` + +**Interfaces:** +- Consumes: `annual_store.list_runs`, `load_run`. +- Produces: `AnnualPage.render_results(results, runs, selected_id) -> str` + +**The palette is fixed and validated. Do not substitute.** Predbat's house chart trio was measured and fails: green↔orange is ΔE 3.6 under protanopia, so roughly 1 in 12 men could not tell "Without Predbat" from "With Predbat". These three pass every check in both light and dark mode: + +| Series | Colour | +|---|---| +| No PV/Battery | `#0072B2` | +| Without Predbat | `#D55E00` | +| With Predbat | `#009E73` | + +- [ ] **Step 1: Write the failing test** + +Append to `apps/predbat/tests/test_web_annual.py`: + +```python +def sample_run_results(): + """Return a results document covering an ok, a degraded and an unavailable month.""" + scenarios = { + "no_pvbat": {"cost_p": 18000.0, "import_kwh": 400.0, "export_kwh": 0.0, "pv_generated_kwh": 0.0, "battery_throughput_kwh": 0.0, "export_credit_p_estimate": 0.0, "self_consumed_kwh": 0.0, "self_consumed_kwh_meaningful": True}, + "without_predbat": {"cost_p": 9000.0, "import_kwh": 300.0, "export_kwh": 20.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 90.0, "export_credit_p_estimate": 300.0, "self_consumed_kwh": 100.0, "self_consumed_kwh_meaningful": True}, + "with_predbat": {"cost_p": 6600.0, "import_kwh": 280.0, "export_kwh": 145.0, "pv_generated_kwh": 120.0, "battery_throughput_kwh": 140.0, "export_credit_p_estimate": 675.0, "self_consumed_kwh": 0.0, "self_consumed_kwh_meaningful": False}, + } + return { + "year": 2025, + "months": [ + {"month": 1, "status": "ok", "days": 31, "sampled_days": ["2025-01-08", "2025-01-24"], "standing_charge_p": 1860.0, "scenarios": scenarios}, + {"month": 2, "status": "degraded", "days": 28, "failed_days": ["2025-02-14"], "standing_charge_p": 1680.0, "scenarios": scenarios}, + {"month": 3, "status": "unavailable", "reason": "no rate data available", "days": 31, "standing_charge_p": 1860.0}, + ], + "annual": {"scenarios": scenarios, "standing_charge_p": 3540.0, "savings": {"pv_battery_vs_none_p": 9000.0, "predbat_vs_baseline_p": 2400.0}, "months_included": 2, "months_excluded": [3]}, + "caveats": ["An example caveat about the P10 fallback."], + } + + +def test_web_annual_results(my_predbat): + """Verify the results view: totals, chart series, month statuses, caveats, selector.""" + failed = False + print("**** Testing web_annual results ****") + + page = make_page(my_predbat) + runs = [{"id": "20260726-101500", "label": "9.5kWh battery · 5.6kWp · Agile", "months_included": 12}, {"id": "20260725-090000", "label": "no battery · 5.6kWp · Agile", "months_included": 12}] + html = page.render_results(sample_run_results(), runs, "20260726-101500") + + print("Test: the annual savings figures are shown") + if "90.00" not in html: + print(" ERROR: the PV/battery saving (9000p = £90.00) should be shown") + failed = True + if "24.00" not in html: + print(" ERROR: the Predbat saving (2400p = £24.00) should be shown") + failed = True + + print("Test: the validated colourblind-safe palette is used, not the house trio") + for colour in ["#0072B2", "#D55E00", "#009E73"]: + if colour not in html: + print(" ERROR: expected the validated colour {} in the chart".format(colour)) + failed = True + for banned in ["#4CAF50", "#FF9800", "#2196F3"]: + if banned in html: + print(" ERROR: {} fails CVD separation for this chart and must not be used".format(banned)) + failed = True + + print("Test: an unavailable month is marked, never drawn as zero") + if "unavailable" not in html.lower(): + print(" ERROR: the unavailable month should be marked as such") + failed = True + if "no rate data available" not in html: + print(" ERROR: the reason for exclusion should be shown") + failed = True + + print("Test: a degraded month is shown with its cost and flagged as partial") + if "degraded" not in html.lower(): + print(" ERROR: the degraded month should be flagged") + failed = True + + print("Test: months_included is stated so the annual figure's coverage is clear") + if "2 of 12" not in html: + print(" ERROR: the annual figure should say how many months it covers") + failed = True + + print("Test: caveats are displayed, not buried in the JSON") + if "An example caveat about the P10 fallback." not in html: + print(" ERROR: caveats must be shown to the user") + failed = True + + print("Test: self_consumed_kwh is qualified when it is not meaningful") + if "not meaningful" not in html.lower(): + print(" ERROR: a non-meaningful self-consumption figure should be qualified, not shown bare") + failed = True + + print("Test: the run selector lists every stored run and marks the selected one") + for run in runs: + if run["label"] not in html: + print(" ERROR: run {} should appear in the selector".format(run["id"])) + failed = True + if "selected" not in html: + print(" ERROR: the selected run should be marked in the dropdown") + failed = True + + print("Test: a download link is offered for the selected run") + if "annual_download?run=20260726-101500" not in html: + print(" ERROR: the selected run should be downloadable as JSON") + failed = True + + print("Test: with no runs at all the view says so rather than rendering an empty chart") + empty = page.render_results(None, [], None) + if "apexcharts" in empty.lower() and "series" in empty.lower(): + print(" ERROR: no chart should be drawn when there are no results") + failed = True + if "no results" not in empty.lower(): + print(" ERROR: the empty state should say there are no results yet") + failed = True + + return failed +``` + +Register it: + +```python +from tests.test_web_annual import test_web_annual, test_web_annual_form, test_web_annual_results, test_web_annual_routes +``` + +```python + ("web_annual_results", test_web_annual_results, "Annual web tab results tests", False), +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd coverage && ./run_all --test web_annual_results > /tmp/w8.txt 2>&1; grep -E "ERROR|AttributeError" /tmp/w8.txt` + +Expected: FAIL — `AnnualPage` has no `render_results`. + +- [ ] **Step 3: Add the results view to `apps/predbat/web_annual.py`** + +Add near the top of the module: + +```python +import calendar +import json + +# Validated with the dataviz palette checker in BOTH light and dark mode, all pairs. +# Predbat's house chart trio (#2196F3/#FF9800/#4CAF50) FAILS here: green vs orange is +# only deltaE 3.6 under protanopia, so roughly one man in twelve could not tell +# "Without Predbat" from "With Predbat" - the exact comparison this chart exists to +# make. Do not substitute without re-running the validator. +SCENARIO_COLOURS = {"no_pvbat": "#0072B2", "without_predbat": "#D55E00", "with_predbat": "#009E73"} +SCENARIO_LABELS = {"no_pvbat": "No PV/Battery", "without_predbat": "Without Predbat", "with_predbat": "With Predbat"} +SCENARIO_ORDER = ["no_pvbat", "without_predbat", "with_predbat"] +``` + +Then append to `AnnualPage`: + +```python + @staticmethod + def _pounds(pence): + """Return a pence value formatted as pounds with an explicit unit.""" + try: + return "£{:.2f}".format(float(pence) / 100.0) + except (TypeError, ValueError): + return "n/a" + + def render_results(self, results, runs, selected_id): + """Return the results view: selector, totals, chart, monthly table, caveats.""" + text = '
\n' + text += self._render_selector(runs, selected_id) + + if not results: + text += "

No results yet — fill in the form above and press Run.

\n
\n" + return text + + annual = results.get("annual", {}) or {} + scenarios = annual.get("scenarios") + included = annual.get("months_included", 0) + + text += "

Annual totals for {}

\n".format(results.get("year", "")) + if not scenarios: + text += "

No month produced a usable result, so there is no annual figure.

\n" + else: + text += "\n" + for key in SCENARIO_ORDER: + entry = scenarios.get(key, {}) + text += "\n".format( + SCENARIO_LABELS[key], self._pounds(entry.get("cost_p")), round(entry.get("import_kwh", 0), 1), round(entry.get("export_kwh", 0), 1) + ) + text += "
ScenarioCostImportExport
{}{}{} kWh{} kWh
\n" + savings = annual.get("savings", {}) or {} + text += "

PV and battery save {} against no system.

\n".format(self._pounds(savings.get("pv_battery_vs_none_p", 0))) + text += "

Predbat saves a further {} against a timer-controlled battery.

\n".format(self._pounds(savings.get("predbat_vs_baseline_p", 0))) + text += "

Standing charge (identical in every scenario): {}

\n".format(self._pounds(annual.get("standing_charge_p", 0))) + + text += "

Based on {} of 12 months.".format(included) + excluded = annual.get("months_excluded") or [] + if excluded: + text += " Excluded: {}.".format(", ".join(calendar.month_abbr[month] for month in excluded)) + text += "

\n" + + text += self._render_chart(results) + text += self._render_month_table(results) + text += self._render_caveats(results) + if selected_id: + text += '

Download this run as JSON

\n'.format(selected_id) + text += "\n" + return text + + def _render_selector(self, runs, selected_id): + """Return the run selector, or nothing when there are no stored runs.""" + if not runs: + return "" + text = '
\n" + return text + + def _render_chart(self, results): + """Return the grouped monthly bar chart. + + Only months with a usable result contribute a bar. An unavailable month is + left out of the series entirely rather than plotted as zero - a zero-height + bar reads as free electricity, which is the opposite of what happened. + """ + categories = [] + series = {key: [] for key in SCENARIO_ORDER} + for entry in results.get("months", []): + if entry.get("status") not in ("ok", "degraded"): + continue + categories.append(calendar.month_abbr[entry["month"]]) + for key in SCENARIO_ORDER: + series[key].append(round(entry.get("scenarios", {}).get(key, {}).get("cost_p", 0) / 100.0, 2)) + + if not categories: + return "

No month produced a usable result, so there is nothing to chart.

\n" + + payload = { + "chart": {"type": "bar", "height": 400, "toolbar": {"show": False}}, + "series": [{"name": SCENARIO_LABELS[key], "data": series[key]} for key in SCENARIO_ORDER], + "colors": [SCENARIO_COLOURS[key] for key in SCENARIO_ORDER], + "xaxis": {"categories": categories}, + "yaxis": {"title": {"text": "Cost (£)"}}, + "plotOptions": {"bar": {"columnWidth": "70%", "borderRadius": 4, "borderRadiusApplication": "end"}}, + "stroke": {"show": True, "width": 2, "colors": ["transparent"]}, + "dataLabels": {"enabled": False}, + "legend": {"position": "top"}, + "tooltip": {"y": {"formatter": None}}, + } + text = '
\n' + text += '\n' + text += "\n" + return text + + def _render_month_table(self, results): + """Return the per-month energy breakdown, marking degraded and unavailable months.""" + text = "

By month

\n\n" + text += "\n" + for entry in results.get("months", []): + name = calendar.month_abbr[entry["month"]] + if entry.get("status") not in ("ok", "degraded"): + text += "\n".format(name, entry.get("reason", "no result")) + continue + suffix = " (degraded — {} sampled day(s) failed)".format(len(entry.get("failed_days", []))) if entry.get("status") == "degraded" else "" + for key in SCENARIO_ORDER: + scenario = entry.get("scenarios", {}).get(key, {}) + if scenario.get("self_consumed_kwh_meaningful", True): + self_consumed = "{} kWh".format(round(scenario.get("self_consumed_kwh", 0), 1)) + else: + self_consumed = "not meaningful" + text += "\n".format( + name if key == SCENARIO_ORDER[0] else "", + suffix if key == SCENARIO_ORDER[0] else "", + SCENARIO_LABELS[key], + self._pounds(scenario.get("cost_p")), + round(scenario.get("import_kwh", 0), 1), + round(scenario.get("export_kwh", 0), 1), + round(scenario.get("pv_generated_kwh", 0), 1), + self_consumed, + round(scenario.get("battery_throughput_kwh", 0), 1), + ) + text += "
MonthScenarioCostImportExportPVSelf-consumedBattery
{}unavailable — {}
{}{}{}{}{} kWh{} kWh{} kWh{}{} kWh
\n" + return text + + def _render_caveats(self, results): + """Return the caveats the engine attached to this run.""" + caveats = results.get("caveats") or [] + if not caveats: + return "" + text = "

Caveats

\n
    \n" + for caveat in caveats: + text += "
  • {}
    • \n".format(caveat) + text += "
\n" + return text +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd coverage && ./run_all -k web_annual > /tmp/w8.txt 2>&1; grep -E "ERROR|Traceback" /tmp/w8.txt` + +Expected: no output. + +- [ ] **Step 5: Look at the rendered page** + +The validator checks colour, not layout. Start Predbat's web interface (or render `render_results(sample_run_results(), runs, id)` to a file and open it) and check the chart for label collisions, bar overflow and legend placement in **both** light and dark mode. Report what you saw — "it rendered" is not an observation. + +- [ ] **Step 6: Run pre-commit and commit** + +```bash +git add apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +coverage/venv/bin/pre-commit run --files apps/predbat/web_annual.py apps/predbat/tests/test_web_annual.py apps/predbat/unit_test.py +git commit -m "feat(annual): add the Annual tab results view and monthly chart" +``` + +--- + +## Task 9: Documentation + +**Files:** +- Modify: `docs/annual-prediction.md` +- Modify: `.cspell/custom-dictionary-workspace.txt` (only if pre-commit flags a word) + +- [ ] **Step 1: Add a "Using the web interface" section to `docs/annual-prediction.md`** + +Insert before the existing "Running it" section, which covers the CLI. Cover: + +- Where the tab is (the **Annual** entry in the navigation) and that it needs no Home Assistant and no configured Predbat. +- That the form prefills from your existing setup where it can, and shows example values otherwise — and that the banner tells you which you are looking at. +- The tariff dropdown, including that your own `compare_list` entries appear in it if you have any, and that picking one fills in the URL fields which stay editable. +- That a run takes one to three minutes, or two to six with a car configured, and shows a progress bar; that it keeps running if you navigate away; and that Cancel stops it. +- That the last five runs are kept and can be switched between with the selector, so you can compare a 5 kWh battery against a 10 kWh one without re-running either. +- That the chart omits unavailable months rather than drawing them as zero. + +Keep the existing CLI documentation — both interfaces are supported. + +- [ ] **Step 2: Verify the docs build** + +```bash +coverage/venv/bin/pre-commit run --files docs/annual-prediction.md +cd coverage && ./venv/bin/mkdocs build --strict -f ../mkdocs.yml > /tmp/w9.txt 2>&1; grep -iE "error|warning" /tmp/w9.txt +``` + +Expected: no errors for `annual-prediction.md`. If cspell flags a word, add it to `.cspell/custom-dictionary-workspace.txt`, re-run pre-commit, and **re-stage** that file — a hook sorts it. + +- [ ] **Step 3: Run the full test suite** + +The tab touches `web.py`, which the whole web test suite depends on. + +```bash +cd coverage && ./run_all > /tmp/w9full.txt 2>&1; grep -E "FAILED|All tests passed|tests failed" /tmp/w9full.txt | tail -5 +``` + +Expected: "All tests passed", around 250-300 seconds. Read the file. + +- [ ] **Step 4: Commit** + +```bash +git add docs/annual-prediction.md .cspell/custom-dictionary-workspace.txt +coverage/venv/bin/pre-commit run --files docs/annual-prediction.md .cspell/custom-dictionary-workspace.txt +git commit -m "docs: document the Annual prediction web tab" +``` + +--- + +## Notes for the implementer + +**The chart palette is not a style preference.** It was measured. If you find yourself wanting Predbat's usual green/orange/blue because it looks more consistent, re-read Task 8's opening: that trio is ΔE 3.6 apart under protanopia for the two series the whole tool exists to compare. Consistency that hides the answer from one reader in twelve is not consistency worth having. + +**Three properties must survive from the engine into the UI.** They were hard-won during the engine work and are easy to undo here: +1. An unavailable month is never a zero-height bar or a `£0.00` cell. +2. `self_consumed_kwh` is qualified when `self_consumed_kwh_meaningful` is false. +3. Caveats are on screen, not only in the JSON. + +**The unconfigured case is the acceptance criterion**, not a nicety. `test_web_annual`'s first assertion is the one that proves it. If it starts failing, something has made the tab depend on a configured Predbat, which breaks the prospective-buyer path the tool exists for. + +**Do not read `apps.yaml` from disk anywhere.** Everything comes from `get_arg()`. From 244136ae457a2585b9e46532838aa6cea03ee14a Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:00:12 +0200 Subject: [PATCH 045/119] feat(annual): add the tariff catalogue for the Annual tab dropdown Adds a curated built-in tariff list and the Compare-key-to-annual-engine mapping that will populate the new Annual tab's tariff dropdown, plus the merge helper that layers a user's compare_list on top of the built-ins. --- apps/predbat/tariff_catalogue.py | 147 ++++++++++++++++++++ apps/predbat/tests/test_tariff_catalogue.py | 117 ++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 266 insertions(+) create mode 100644 apps/predbat/tariff_catalogue.py create mode 100644 apps/predbat/tests/test_tariff_catalogue.py diff --git a/apps/predbat/tariff_catalogue.py b/apps/predbat/tariff_catalogue.py new file mode 100644 index 000000000..f578f4f8d --- /dev/null +++ b/apps/predbat/tariff_catalogue.py @@ -0,0 +1,147 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Curated tariff catalogue for the Annual prediction tab's dropdown. + +The entries mirror the commented-out ``compare_list`` template in +``config/apps.yaml`` so a user does not have to hand-copy a YAML block to get a +realistic tariff. Compare's key names differ from the annual engine's, so the +mapping lives here and nowhere else. +""" + +# Compare writes rates_import_octopus_url; the annual engine reads import_octopus_url +COMPARE_KEY_MAP = { + "rates_import_octopus_url": "import_octopus_url", + "rates_export_octopus_url": "export_octopus_url", +} + +# Keys that mean the same thing in both and need no translation +PASSTHROUGH_KEYS = ["rates_import", "rates_export"] + +# The dropdown's escape hatch: leaves the URL fields blank for a hand-entered tariff +CUSTOM_ID = "custom" + +_OCTOPUS = "https://api.octopus.energy/v1/products" + +BUILTIN_TARIFFS = [ + {"id": "cap_seg", "name": "Price cap import / SEG export", "rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}]}, + { + "id": "eon_next_drive", + "name": "Eon Next Drive import / Fixed export", + "rates_import": [{"rate": 6.7, "start": "00:00:00", "end": "07:00:00"}, {"rate": 24.86, "start": "07:00:00", "end": "00:00:00"}], + "rates_export": [{"rate": 16.5}], + }, + { + "id": "igo_fixed", + "name": "Intelligent GO import / Fixed export", + "import_octopus_url": "{}/INTELLI-VAR-24-10-29/electricity-tariffs/E-1R-INTELLI-VAR-24-10-29-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "igo_agile", + "name": "Intelligent GO import / Agile export", + "import_octopus_url": "{}/INTELLI-VAR-24-10-29/electricity-tariffs/E-1R-INTELLI-VAR-24-10-29-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "go_fixed", + "name": "GO import / Fixed export", + "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "agile_fixed", + "name": "Agile import / Fixed export", + "import_octopus_url": "{}/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "agile_agile", + "name": "Agile import / Agile export", + "import_octopus_url": "{}/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "flux", + "name": "Flux import / Flux export", + "import_octopus_url": "{}/FLUX-IMPORT-23-02-14/electricity-tariffs/E-1R-FLUX-IMPORT-23-02-14-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/FLUX-EXPORT-23-02-14/electricity-tariffs/E-1R-FLUX-EXPORT-23-02-14-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + }, + { + "id": "cosy_fixed", + "name": "Cosy import / Fixed export", + "import_octopus_url": "{}/COSY-22-12-08/electricity-tariffs/E-1R-COSY-22-12-08-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "cosy_agile", + "name": "Cosy import / Agile export", + "import_octopus_url": "{}/COSY-22-12-08/electricity-tariffs/E-1R-COSY-22-12-08-{{dno_region}}/standard-unit-rates".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "snug_fixed", + "name": "Snug import / Fixed export", + "import_octopus_url": "{}/SNUG-24-11-07/electricity-tariffs/E-1R-SNUG-24-11-07-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, + { + "id": "iflux", + "name": "Intelligent Flux import / export", + "import_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/INTELLI-FLUX-EXPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-EXPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, +] + + +def convert_compare_entry(entry): + """Convert one Compare ``compare_list`` entry into the annual engine's shape. + + Returns None when the entry cannot be used - it is not a mapping, it has no + id, or it carries no rate source at all. Compare allows an entry with neither + (the 'current' pseudo-tariff, which means "whatever is configured"), and that + has no meaning here, so it is dropped rather than offered as a broken choice. + """ + if not isinstance(entry, dict): + return None + if not entry.get("id") or not entry.get("name"): + return None + + converted = {"id": entry["id"], "name": entry["name"]} + for source_key, target_key in COMPARE_KEY_MAP.items(): + if entry.get(source_key): + converted[target_key] = entry[source_key] + for key in PASSTHROUGH_KEYS: + if entry.get(key): + converted[key] = entry[key] + + if not converted.get("import_octopus_url") and not converted.get("rates_import"): + return None + return converted + + +def merged_catalogue(compare_list=None): + """Return the dropdown's entries: built-ins, then the user's own, then Custom. + + A user entry sharing a built-in id replaces it rather than appearing twice - + the user's own definition is the more specific one. Malformed entries are + skipped so one bad line in apps.yaml cannot empty the dropdown. + """ + catalogue = [dict(entry) for entry in BUILTIN_TARIFFS] + by_id = {entry["id"]: index for index, entry in enumerate(catalogue)} + + for entry in compare_list or []: + converted = convert_compare_entry(entry) + if converted is None: + continue + if converted["id"] in by_id: + catalogue[by_id[converted["id"]]] = converted + else: + by_id[converted["id"]] = len(catalogue) + catalogue.append(converted) + + catalogue.append({"id": CUSTOM_ID, "name": "Custom - enter URLs below"}) + return catalogue diff --git a/apps/predbat/tests/test_tariff_catalogue.py b/apps/predbat/tests/test_tariff_catalogue.py new file mode 100644 index 000000000..2591eb2c5 --- /dev/null +++ b/apps/predbat/tests/test_tariff_catalogue.py @@ -0,0 +1,117 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the tariff catalogue used by the Annual tab's dropdown.""" + +from tariff_catalogue import BUILTIN_TARIFFS, CUSTOM_ID, convert_compare_entry, merged_catalogue + + +def test_tariff_catalogue(my_predbat): + """Verify the built-in catalogue, the Compare key mapping, and the merge.""" + failed = False + print("**** Testing tariff_catalogue ****") + + print("Test: every built-in entry has an id, a name and at least an import URL") + if not BUILTIN_TARIFFS: + print(" ERROR: the built-in catalogue is empty") + failed = True + seen_ids = set() + for entry in BUILTIN_TARIFFS: + for key in ["id", "name"]: + if not entry.get(key): + print(" ERROR: entry {} is missing '{}'".format(entry, key)) + failed = True + if not entry.get("import_octopus_url") and not entry.get("rates_import"): + print(" ERROR: entry {} has neither an import URL nor fixed rates".format(entry.get("id"))) + failed = True + if entry.get("id") in seen_ids: + print(" ERROR: duplicate id {}".format(entry.get("id"))) + failed = True + seen_ids.add(entry.get("id")) + + print("Test: no built-in entry uses Compare's key names") + for entry in BUILTIN_TARIFFS: + for stale in ["rates_import_octopus_url", "rates_export_octopus_url"]: + if stale in entry: + print(" ERROR: entry {} still uses Compare's key '{}'".format(entry.get("id"), stale)) + failed = True + + print("Test: convert_compare_entry maps Compare's URL keys onto the engine's") + converted = convert_compare_entry( + { + "id": "agile_agile", + "name": "Agile import/Agile export", + "rates_import_octopus_url": "https://example.com/import/", + "rates_export_octopus_url": "https://example.com/export/", + } + ) + if converted is None: + print(" ERROR: a valid Compare entry should convert") + failed = True + else: + if converted.get("import_octopus_url") != "https://example.com/import/": + print(" ERROR: import URL not mapped, got {}".format(converted)) + failed = True + if converted.get("export_octopus_url") != "https://example.com/export/": + print(" ERROR: export URL not mapped, got {}".format(converted)) + failed = True + if "rates_import_octopus_url" in converted: + print(" ERROR: the Compare key should not survive conversion") + failed = True + + print("Test: fixed rate structures pass through unchanged") + converted = convert_compare_entry({"id": "cap", "name": "Price cap", "rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}]}) + if converted is None or converted.get("rates_import") != [{"rate": 24.86}]: + print(" ERROR: fixed rates should pass through, got {}".format(converted)) + failed = True + + print("Test: an entry with no usable rate source is rejected rather than shown") + if convert_compare_entry({"id": "current", "name": "Current Tariff"}) is not None: + print(" ERROR: an entry with no rates should be rejected") + failed = True + if convert_compare_entry({"name": "No id"}) is not None: + print(" ERROR: an entry with no id should be rejected") + failed = True + if convert_compare_entry("not a dict") is not None: + print(" ERROR: a non-dict should be rejected rather than raising") + failed = True + + print("Test: merged_catalogue with no user list returns the built-ins plus Custom") + merged = merged_catalogue(None) + if len(merged) != len(BUILTIN_TARIFFS) + 1: + print(" ERROR: expected {} entries, got {}".format(len(BUILTIN_TARIFFS) + 1, len(merged))) + failed = True + if merged[-1]["id"] != CUSTOM_ID: + print(" ERROR: Custom should be the last entry, got {}".format(merged[-1])) + failed = True + + print("Test: a user's compare_list is merged in and does not clobber a built-in id") + builtin_id = BUILTIN_TARIFFS[0]["id"] + merged = merged_catalogue( + [ + {"id": builtin_id, "name": "My override", "rates_import_octopus_url": "https://example.com/mine/"}, + {"id": "my_tariff", "name": "My tariff", "rates_import": [{"rate": 20.0}]}, + ] + ) + ids = [entry["id"] for entry in merged] + if ids.count(builtin_id) != 1: + print(" ERROR: a user entry sharing a built-in id should not duplicate it, ids were {}".format(ids)) + failed = True + if "my_tariff" not in ids: + print(" ERROR: a new user entry should appear, ids were {}".format(ids)) + failed = True + + print("Test: a malformed user entry is skipped rather than breaking the dropdown") + merged = merged_catalogue([{"junk": True}, None, "string", {"id": "ok", "name": "Ok", "rates_import": [{"rate": 5.0}]}]) + ids = [entry["id"] for entry in merged] + if "ok" not in ids: + print(" ERROR: the valid entry should survive alongside malformed ones, ids were {}".format(ids)) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 4c3440be2..e95bb3912 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -170,6 +170,7 @@ from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration from tests.test_annual_cli import test_annual_cli +from tests.test_tariff_catalogue import test_tariff_catalogue # Mock the components and plugin system @@ -418,6 +419,7 @@ def main(): ("annual_results", test_annual_results, "Annual prediction results assembly tests", False), ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), + ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), ] # Parse command line arguments From 46449c5737071176b991d0eb93a961e548d11a9f Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:01:28 +0200 Subject: [PATCH 046/119] docs: correct two tariff catalogue errors in the web UI plan The plan invented INTELLI-FLUX-EXPORT-23-07-14 by analogy with the other tariffs. The Octopus products API lists only INTELLI-FLUX-IMPORT-23-07-14 and the export rates live under that code, so apps.yaml was right and the plan's "correction" would have 404'd as an unavailable month. go_agile was also omitted, despite every other tariff in the template carrying both its Fixed and Agile export pairing. Both found by the Task 1 implementer questioning the brief. Co-Authored-By: Claude Opus 5 (1M context) --- docs/superpowers/plans/2026-07-26-annual-web-ui.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-26-annual-web-ui.md b/docs/superpowers/plans/2026-07-26-annual-web-ui.md index ed021cf87..db213a20d 100644 --- a/docs/superpowers/plans/2026-07-26-annual-web-ui.md +++ b/docs/superpowers/plans/2026-07-26-annual-web-ui.md @@ -287,6 +287,12 @@ BUILTIN_TARIFFS = [ "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), }, + { + "id": "go_agile", + "name": "GO import / Agile export", + "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, { "id": "agile_fixed", "name": "Agile import / Fixed export", @@ -327,7 +333,10 @@ BUILTIN_TARIFFS = [ "id": "iflux", "name": "Intelligent Flux import / export", "import_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), - "export_octopus_url": "{}/INTELLI-FLUX-EXPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-EXPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + # Not a copy-paste slip: the Octopus products API lists only + # INTELLI-FLUX-IMPORT-23-07-14, and the export rates live under that same + # product code. There is no INTELLI-FLUX-EXPORT product to point at. + "export_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), }, ] From 1689dab933abd579372f6cc738b6dd13d09540e5 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:03:50 +0200 Subject: [PATCH 047/119] fix(annual): correct iflux export URL and restore missing go_agile tariff The iflux export URL invented a non-existent Octopus product code (INTELLI-FLUX-EXPORT-23-07-14); Octopus publishes both the import and export rates for Intelligent Flux under INTELLI-FLUX-IMPORT-23-07-14, as apps.yaml's template already had. Also restores the go_agile (GO import / Agile export) entry that apps.yaml's template has but the catalogue was missing. Adds an explicit ids assertion to the test so a future accidental deletion of a built-in tariff is caught rather than silently shrinking the dropdown. --- apps/predbat/tariff_catalogue.py | 11 ++++++++++- apps/predbat/tests/test_tariff_catalogue.py | 21 +++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/apps/predbat/tariff_catalogue.py b/apps/predbat/tariff_catalogue.py index f578f4f8d..ec3798e70 100644 --- a/apps/predbat/tariff_catalogue.py +++ b/apps/predbat/tariff_catalogue.py @@ -52,6 +52,12 @@ "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), "export_octopus_url": "{}/OUTGOING-VAR-24-10-26/electricity-tariffs/E-1R-OUTGOING-VAR-24-10-26-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), }, + { + "id": "go_agile", + "name": "GO import / Agile export", + "import_octopus_url": "{}/GO-VAR-22-10-14/electricity-tariffs/E-1R-GO-VAR-22-10-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + }, { "id": "agile_fixed", "name": "Agile import / Fixed export", @@ -91,8 +97,11 @@ { "id": "iflux", "name": "Intelligent Flux import / export", + # There is no INTELLI-FLUX-EXPORT product - Octopus publishes both the import and + # export rates for Intelligent Flux under the import product code below. Do not + # "fix" this to a distinct export code; that product does not exist and 404s. "import_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), - "export_octopus_url": "{}/INTELLI-FLUX-EXPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-EXPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), + "export_octopus_url": "{}/INTELLI-FLUX-IMPORT-23-07-14/electricity-tariffs/E-1R-INTELLI-FLUX-IMPORT-23-07-14-{{dno_region}}/standard-unit-rates/".format(_OCTOPUS), }, ] diff --git a/apps/predbat/tests/test_tariff_catalogue.py b/apps/predbat/tests/test_tariff_catalogue.py index 2591eb2c5..56f580732 100644 --- a/apps/predbat/tests/test_tariff_catalogue.py +++ b/apps/predbat/tests/test_tariff_catalogue.py @@ -35,6 +35,27 @@ def test_tariff_catalogue(my_predbat): failed = True seen_ids.add(entry.get("id")) + print("Test: the built-in catalogue contains exactly the expected ids") + expected_ids = [ + "cap_seg", + "eon_next_drive", + "igo_fixed", + "igo_agile", + "go_fixed", + "go_agile", + "agile_fixed", + "agile_agile", + "flux", + "cosy_fixed", + "cosy_agile", + "snug_fixed", + "iflux", + ] + actual_ids = [entry["id"] for entry in BUILTIN_TARIFFS] + if actual_ids != expected_ids: + print(" ERROR: built-in ids changed, expected {} got {}".format(expected_ids, actual_ids)) + failed = True + print("Test: no built-in entry uses Compare's key names") for entry in BUILTIN_TARIFFS: for stale in ["rates_import_octopus_url", "rates_export_octopus_url"]: From 8fcdeb3efb46a443a8b6291ea4373fc4cd32af40 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:09:26 +0200 Subject: [PATCH 048/119] feat(annual): add machine mode so a parent process can drive the CLI The web tab runs the annual engine as a subprocess and needs the results back over a pipe. --machine makes annual_cli.py emit the results document as one JSON object on stdout and progress as JSON-per-line on stderr, suppressing the human table so it cannot corrupt the parent's parse. --- apps/predbat/annual_cli.py | 41 +++++++++++++++----- apps/predbat/tests/test_annual_cli.py | 55 ++++++++++++++++++++++++++- apps/predbat/unit_test.py | 3 +- 3 files changed, 88 insertions(+), 11 deletions(-) diff --git a/apps/predbat/annual_cli.py b/apps/predbat/annual_cli.py index 8e5064c4d..889595cf2 100644 --- a/apps/predbat/annual_cli.py +++ b/apps/predbat/annual_cli.py @@ -107,11 +107,26 @@ def format_table(results, currency="p"): return "\n".join(lines) -def make_progress(quiet): - """Return a progress callback that writes to stderr, or None when quiet.""" +def make_progress(quiet, machine=False): + """Return a progress callback writing to stderr, or None when quiet. + + Machine mode emits one JSON object per line so the parent process never has + to parse prose - the human wording is free to change without breaking a + caller. Progress always goes to stderr so stdout carries only the result, + whichever mode is in use. + """ if quiet: return None + if machine: + + def progress(completed, total, message): + """Emit one JSON progress record to stderr.""" + sys.stderr.write(json.dumps({"completed": completed, "total": total, "message": message}) + "\n") + sys.stderr.flush() + + return progress + def progress(completed, total, message): """Report progress to stderr so stdout stays parseable.""" sys.stderr.write("[{}/{}] {}\n".format(completed, total, message)) @@ -127,6 +142,7 @@ def main(argv=None): parser.add_argument("--out", default=None, help="Write the results JSON to this path") parser.add_argument("--work-dir", default="./annual_work", help="Working directory for the headless Predbat instance and cache") parser.add_argument("--quiet", action="store_true", help="Suppress progress output") + parser.add_argument("--machine", action="store_true", help="Emit results as JSON on stdout and progress as JSON on stderr, for a calling process") args = parser.parse_args(argv) try: @@ -144,7 +160,7 @@ def main(argv=None): # sample days and car-charging shortfalls must stay visible even in a quiet run, per # the "failures are visible, never silent" contract. predictor = AnnualPredictor(config, log=print, storage=storage, work_dir=args.work_dir) - results = asyncio.run(predictor.run(progress=make_progress(args.quiet))) + results = asyncio.run(predictor.run(progress=make_progress(args.quiet, machine=args.machine))) except AnnualConfigError as error: sys.stderr.write("Config error: {}\n".format(error)) return 2 @@ -152,18 +168,25 @@ def main(argv=None): exit_code = 0 if args.out: try: - with open(args.out, "w") as handle: + with open(args.out, "w", encoding="utf-8") as handle: json.dump(results, handle, indent=2) - sys.stderr.write("Results written to {}\n".format(args.out)) except (OSError, TypeError, ValueError) as error: # The projection just took several minutes to compute; a failed write to - # --out must not throw the results away. The table below is still printed - # and the failure is only reported through a non-zero exit code, so a - # caller relying on --out to have succeeded is not fooled into thinking it did. + # --out must not throw the results away. The table (or JSON, in machine mode) + # below is still emitted and the failure is only reported through a non-zero + # exit code, so a caller relying on --out to have succeeded is not fooled into + # thinking it did. sys.stderr.write("Could not write results to {}: {}\n".format(args.out, error)) exit_code = 1 - print(format_table(results)) + if args.machine: + # The parent reads exactly one JSON object from stdout; the human table would + # corrupt it, so it is suppressed rather than merely reordered. + json.dump(results, sys.stdout) + sys.stdout.write("\n") + else: + print(format_table(results)) + return exit_code diff --git a/apps/predbat/tests/test_annual_cli.py b/apps/predbat/tests/test_annual_cli.py index fdcb2d2b9..6fe45b14b 100644 --- a/apps/predbat/tests/test_annual_cli.py +++ b/apps/predbat/tests/test_annual_cli.py @@ -15,7 +15,7 @@ from contextlib import redirect_stderr, redirect_stdout import annual_cli -from annual_cli import format_table +from annual_cli import format_table, make_progress class _StubPredictor: @@ -215,3 +215,56 @@ def test_annual_cli(my_predbat): failed = True return failed + + +def test_annual_cli_machine(my_predbat): + """Verify machine mode emits JSON progress on stderr and nothing human on stdout.""" + import io + import json + import sys + + failed = False + print("**** Testing annual CLI machine mode ****") + + print("Test: machine progress writes one JSON object per line to stderr") + captured = io.StringIO() + original_stderr = sys.stderr + sys.stderr = captured + try: + progress = make_progress(quiet=False, machine=True) + progress(3, 12, "Month 03/2025") + finally: + sys.stderr = original_stderr + + line = captured.getvalue().strip() + try: + parsed = json.loads(line) + except ValueError: + print(" ERROR: machine progress should be JSON, got {!r}".format(line)) + parsed = {} + failed = True + if parsed.get("completed") != 3 or parsed.get("total") != 12 or parsed.get("message") != "Month 03/2025": + print(" ERROR: unexpected progress payload {}".format(parsed)) + failed = True + + print("Test: human progress is unchanged when machine mode is off") + captured = io.StringIO() + sys.stderr = captured + try: + progress = make_progress(quiet=False, machine=False) + progress(3, 12, "Month 03/2025") + finally: + sys.stderr = original_stderr + if "[3/12]" not in captured.getvalue(): + print(" ERROR: expected the human form, got {!r}".format(captured.getvalue())) + failed = True + + print("Test: quiet still suppresses progress in both modes") + if make_progress(quiet=True, machine=False) is not None: + print(" ERROR: quiet should give no progress callback") + failed = True + if make_progress(quiet=True, machine=True) is not None: + print(" ERROR: quiet should give no progress callback in machine mode either") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index e95bb3912..e80f2f0ff 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -169,7 +169,7 @@ from tests.test_annual_scenarios import test_annual_scenarios from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration -from tests.test_annual_cli import test_annual_cli +from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine from tests.test_tariff_catalogue import test_tariff_catalogue # Mock the components and plugin system @@ -419,6 +419,7 @@ def main(): ("annual_results", test_annual_results, "Annual prediction results assembly tests", False), ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), + ("annual_cli_machine", test_annual_cli_machine, "Annual CLI machine mode tests", False), ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), ] From 6bb50e39617d77b87d6f3cf7522178213b35c918 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:23:56 +0200 Subject: [PATCH 049/119] fix(annual): keep --machine's stdout pure JSON when the engine logs or self-updates Review found two gaps in the machine-mode contract: a successful --out write had silently lost its stderr confirmation, and predictor.run()'s log=print plus predbat.py's module-level self-update check both write straight to stdout, which would corrupt a parent's single-JSON-object read. Route the engine's log to stderr under --machine, redirect stdout to stderr for the duration of predictor construction/run so any stray print() (including download.py's check_install()) cannot leak onto stdout, and restore the --out confirmation on stderr in both modes. Added a real subprocess check and an in-process warning-during-run() check to test_annual_cli.py. --- apps/predbat/annual_cli.py | 53 +++++++++++-- apps/predbat/tests/test_annual_cli.py | 104 ++++++++++++++++++++++++++ apps/predbat/unit_test.py | 3 +- 3 files changed, 151 insertions(+), 9 deletions(-) diff --git a/apps/predbat/annual_cli.py b/apps/predbat/annual_cli.py index 889595cf2..41f03ebfc 100644 --- a/apps/predbat/annual_cli.py +++ b/apps/predbat/annual_cli.py @@ -14,6 +14,7 @@ import argparse import asyncio import calendar +import contextlib import json import os import sys @@ -107,6 +108,20 @@ def format_table(results, currency="p"): return "\n".join(lines) +def _stderr_log(message): + """Write a log message to stderr, so machine mode's stdout stays pure JSON. + + ``StorageLocalFiles`` and ``AnnualPredictor`` are normally given ``log=print``, which + writes to stdout. Under ``--machine`` that would interleave plain-text warnings - P10 + fallbacks, missing rate data, failed sample days, car-charging shortfalls, postcode + resolution notices - ahead of the final JSON document, so a parent reading stdout would + see garbage-then-JSON and fail to parse it. Routing those same warnings to stderr instead + keeps them visible without ever touching stdout. + """ + sys.stderr.write("{}\n".format(message)) + sys.stderr.flush() + + def make_progress(quiet, machine=False): """Return a progress callback writing to stderr, or None when quiet. @@ -152,15 +167,33 @@ def main(argv=None): sys.stderr.write("Could not read config {}: {}\n".format(args.config, error)) return 2 - storage = StorageLocalFiles(args.work_dir, print) - + # Under --machine, the engine's log must go to stderr rather than the default print() + # (stdout): predictor.run() can log warnings - P10 fallbacks, missing rate data, failed + # sample days, car-charging shortfalls, postcode resolution - and any of those landing on + # stdout ahead of the final json.dump() would corrupt the one-JSON-object contract a parent + # process depends on. The default (non-machine) path keeps log=print unchanged. + log = _stderr_log if args.machine else print + + storage = StorageLocalFiles(args.work_dir, log) + + # predictor.run() lazily imports the full Predbat engine (predbat.py) on its first call + # to create_headless_predbat(); that module's top-level self-update check + # (download.check_install()) writes plain text straight to real stdout via a bare + # print(), bypassing the log callable entirely. Routing our own log calls to stderr + # (above) cannot catch that, so in machine mode stdout itself is redirected to stderr for + # the duration of construction and run() - any stray print(), from this code or anything + # it pulls in, lands on stderr instead of corrupting the one-JSON-object stdout contract. + # The messages stay visible, just on the correct stream; only json.dump() below writes to + # the real stdout. + stdout_guard = contextlib.redirect_stdout(sys.stderr) if args.machine else contextlib.nullcontext() try: - # --quiet suppresses only the per-month progress lines (make_progress() below), never - # the warnings AnnualPredictor.log emits: P10 fallbacks, missing rate data, failed - # sample days and car-charging shortfalls must stay visible even in a quiet run, per - # the "failures are visible, never silent" contract. - predictor = AnnualPredictor(config, log=print, storage=storage, work_dir=args.work_dir) - results = asyncio.run(predictor.run(progress=make_progress(args.quiet, machine=args.machine))) + with stdout_guard: + # --quiet suppresses only the per-month progress lines (make_progress() below), never + # the warnings AnnualPredictor.log emits: P10 fallbacks, missing rate data, failed + # sample days and car-charging shortfalls must stay visible even in a quiet run, per + # the "failures are visible, never silent" contract. + predictor = AnnualPredictor(config, log=log, storage=storage, work_dir=args.work_dir) + results = asyncio.run(predictor.run(progress=make_progress(args.quiet, machine=args.machine))) except AnnualConfigError as error: sys.stderr.write("Config error: {}\n".format(error)) return 2 @@ -170,6 +203,10 @@ def main(argv=None): try: with open(args.out, "w", encoding="utf-8") as handle: json.dump(results, handle, indent=2) + # stderr, not stdout, so this confirmation is safe in both modes: --machine's + # stdout purity only concerns stdout, and stderr already carries plain text + # (config errors, progress) in machine mode. + sys.stderr.write("Results written to {}\n".format(args.out)) except (OSError, TypeError, ValueError) as error: # The projection just took several minutes to compute; a failed write to # --out must not throw the results away. The table (or JSON, in machine mode) diff --git a/apps/predbat/tests/test_annual_cli.py b/apps/predbat/tests/test_annual_cli.py index 6fe45b14b..ac3ebe987 100644 --- a/apps/predbat/tests/test_annual_cli.py +++ b/apps/predbat/tests/test_annual_cli.py @@ -39,6 +39,27 @@ async def run(self, progress=None): return sample_results() +class _WarningPredictor: + """Stub predictor whose run() logs a warning before returning results. + + Mirrors a real mid-run warning (a P10 fallback, missing rate data, a failed sample day) so a + test can check where that warning lands: on stderr, never on stdout, in machine mode. The + regression this guards against is a stray warning-turned-print inside ``main()``'s machine + branch, which ``test_annual_cli_machine``'s unit-level checks of ``make_progress`` alone + cannot see, since that warning is emitted by the engine, not by the progress callback. + """ + + def __init__(self, config, log=None, storage=None, work_dir=None): + """Record the log callable so run() can use it.""" + self._log = log + + async def run(self, progress=None): + """Emit one warning through the recorded log callable, then return canned results.""" + if self._log: + self._log("Warn: stub P10 fallback warning") + return sample_results() + + def sample_results(): """Return a small results document covering an ok month and an unavailable one.""" scenarios = { @@ -268,3 +289,86 @@ def test_annual_cli_machine(my_predbat): failed = True return failed + + +def test_annual_cli_machine_end_to_end(my_predbat): + """Verify main() itself, not just make_progress(), keeps stdout pure JSON under --machine. + + Two checks that unit-testing make_progress() alone cannot make: + + 1. A real subprocess invocation with a config that fails validation before + predictor.run() is ever reached - confirming exit code, empty stdout and a readable + stderr message hold end-to-end, not just when main() is called in-process. + 2. An in-process call to main() with AnnualPredictor stubbed to log a warning from inside + run() (mirroring a P10 fallback or similar engine warning) - confirming that warning + lands on stderr and never on stdout, and that stdout still parses as exactly one JSON + object. This is the actual regression risk: a stray print() reachable only once + predictor.run() executes, which a bad-config-only check cannot exercise. + """ + import io + import json + import subprocess + import sys + + failed = False + print("**** Testing annual CLI machine mode end-to-end (main()) ****") + + print("Test: a real subprocess run with --machine and a bad config emits nothing on stdout") + with tempfile.TemporaryDirectory() as work_dir: + config_path = os.path.join(work_dir, "bad.yaml") + with open(config_path, "w") as handle: + handle.write("annual: {}\n") + completed = subprocess.run( + [sys.executable, annual_cli.__file__, "--config", config_path, "--work-dir", os.path.join(work_dir, "work"), "--machine"], + capture_output=True, + text=True, + timeout=60, + ) + if completed.returncode != 2: + print(" ERROR: expected exit code 2 for a config error, got {}".format(completed.returncode)) + failed = True + if completed.stdout != "": + print(" ERROR: stdout must be empty on a config error, got {!r}".format(completed.stdout)) + failed = True + if "annual.location" not in completed.stderr: + print(" ERROR: expected a readable config error on stderr, got {!r}".format(completed.stderr)) + failed = True + + print("Test: a warning logged mid-run() lands on stderr, never on stdout, alongside clean JSON") + original_predictor = annual_cli.AnnualPredictor + annual_cli.AnnualPredictor = _WarningPredictor + stdout_capture = io.StringIO() + stderr_capture = io.StringIO() + try: + with tempfile.TemporaryDirectory() as work_dir: + config_path = os.path.join(work_dir, "annual.yaml") + with open(config_path, "w") as handle: + handle.write("annual: {}\n") + with redirect_stdout(stdout_capture), redirect_stderr(stderr_capture): + exit_code = annual_cli.main(["--config", config_path, "--work-dir", os.path.join(work_dir, "work"), "--machine"]) + finally: + annual_cli.AnnualPredictor = original_predictor + + if exit_code != 0: + print(" ERROR: a successful run should exit 0, got {}".format(exit_code)) + failed = True + + stdout_text = stdout_capture.getvalue() + if "stub P10 fallback warning" in stdout_text: + print(" ERROR: the engine's warning leaked onto stdout, got {!r}".format(stdout_text)) + failed = True + try: + parsed = json.loads(stdout_text) + except ValueError: + print(" ERROR: stdout should be exactly one JSON object even when the engine logs mid-run, got {!r}".format(stdout_text)) + parsed = None + failed = True + if parsed is not None and parsed != sample_results(): + print(" ERROR: stdout's JSON should match the results document exactly, got {}".format(parsed)) + failed = True + + if "stub P10 fallback warning" not in stderr_capture.getvalue(): + print(" ERROR: the engine's warning should still be visible, on stderr, got {!r}".format(stderr_capture.getvalue())) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index e80f2f0ff..e5b6ac648 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -169,7 +169,7 @@ from tests.test_annual_scenarios import test_annual_scenarios from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration -from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine +from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine, test_annual_cli_machine_end_to_end from tests.test_tariff_catalogue import test_tariff_catalogue # Mock the components and plugin system @@ -420,6 +420,7 @@ def main(): ("annual_integration", test_annual_integration, "Annual prediction integration tests", True), ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), ("annual_cli_machine", test_annual_cli_machine, "Annual CLI machine mode tests", False), + ("annual_cli_machine_end_to_end", test_annual_cli_machine_end_to_end, "Annual CLI machine mode end-to-end tests", False), ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), ] From 0cf8bdaae01a62c912682698688df54b029ad4f9 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:32:43 +0200 Subject: [PATCH 050/119] feat(annual): add subprocess job control for the Annual tab --- apps/predbat/annual_job.py | 176 ++++++++++++++++++++++++++ apps/predbat/tests/annual_stub.py | 52 ++++++++ apps/predbat/tests/test_annual_job.py | 130 +++++++++++++++++++ apps/predbat/unit_test.py | 2 + 4 files changed, 360 insertions(+) create mode 100644 apps/predbat/annual_job.py create mode 100644 apps/predbat/tests/annual_stub.py create mode 100644 apps/predbat/tests/test_annual_job.py diff --git a/apps/predbat/annual_job.py b/apps/predbat/annual_job.py new file mode 100644 index 000000000..68cbed08e --- /dev/null +++ b/apps/predbat/annual_job.py @@ -0,0 +1,176 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Subprocess control for the Annual prediction run. + +The annual engine is one to three minutes of synchronous CPU work - two to six +with a car - so running it inside the web server's event loop would freeze the +whole Predbat interface and the five minute optimiser loop that shares it. It +runs as a child process instead, handing progress back on stderr and the results +document on stdout. + +This module owns the process and nothing else: no HTML, no Storage, no opinion +about what the results mean. That is what lets it be tested against a stub child +rather than the real engine. +""" + +import asyncio +import json +import time + +# How much of the child's stderr to keep for the failure message. Enough to carry +# a traceback, bounded so a chatty failure cannot grow without limit. +MAX_ERROR_LINES = 20 + +# Grace period between asking the child to stop and killing it outright +CANCEL_GRACE_SECONDS = 5.0 + + +class AnnualJob: + """Runs one annual prediction child process at a time and tracks its progress.""" + + def __init__(self, log): + """Create an idle job that logs through the supplied callable.""" + self.log = log + self.state = "idle" + self.completed = 0 + self.total = 0 + self.message = "" + self.error = None + self.results = None + self.started_at = None + self._process = None + self._stderr_tail = [] + + def status(self): + """Return a JSON-serialisable snapshot for the polling endpoint.""" + elapsed = 0 + if self.started_at is not None: + end = self.started_at if self.state == "idle" else time.time() + elapsed = int(end - self.started_at) + return { + "state": self.state, + "completed": self.completed, + "total": self.total, + "message": self.message, + "elapsed": elapsed, + "error": self.error, + } + + async def start(self, command): + """Spawn the child. Returns False when a run is already in progress. + + Refusing rather than queueing is deliberate: two annual runs on the same + machine would compete for the same CPU and both would take twice as long. + """ + if self.state == "running": + self.log("Warn: Annual: a run is already in progress, refusing to start another") + return False + + self.state = "running" + self.completed = 0 + self.total = 0 + self.message = "Starting" + self.error = None + self.results = None + self.started_at = time.time() + self._stderr_tail = [] + + try: + self._process = await asyncio.create_subprocess_exec(*command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) + except (OSError, ValueError) as exception: + self.state = "failed" + self.error = "Could not start the annual run: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) + return False + + asyncio.ensure_future(self._supervise()) + return True + + async def cancel(self): + """Ask the child to stop, killing it if it does not. Returns False if nothing was running.""" + if self.state != "running" or self._process is None: + return False + self.state = "cancelled" + self.message = "Cancelled" + try: + self._process.terminate() + except ProcessLookupError: + return True + try: + await asyncio.wait_for(self._process.wait(), timeout=CANCEL_GRACE_SECONDS) + except asyncio.TimeoutError: + self.log("Warn: Annual: the run did not stop when asked, killing it") + try: + self._process.kill() + except ProcessLookupError: + pass + return True + + async def _read_progress(self, stream): + """Consume the child's stderr, updating progress and keeping a tail for errors.""" + while True: + line = await stream.readline() + if not line: + break + text = line.decode("utf-8", errors="replace").strip() + if not text: + continue + self._stderr_tail.append(text) + if len(self._stderr_tail) > MAX_ERROR_LINES: + self._stderr_tail.pop(0) + try: + record = json.loads(text) + except ValueError: + # Not a progress record - the child is allowed to write plain + # warnings to stderr, and one bad line must not stop the parse. + continue + if isinstance(record, dict) and "completed" in record: + self.completed = record.get("completed", self.completed) + self.total = record.get("total", self.total) + self.message = record.get("message", self.message) + + async def _supervise(self): + """Read both streams to completion, then settle the final state.""" + process = self._process + try: + stdout_data, _ = await asyncio.gather(process.stdout.read(), self._read_progress(process.stderr)) + await process.wait() + except (OSError, ValueError) as exception: + self.state = "failed" + self.error = "The annual run could not be read: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) + return + + if self.state == "cancelled": + return + + tail = "\n".join(self._stderr_tail) + + if process.returncode != 0: + self.state = "failed" + self.error = "The annual run exited with code {}.\n{}".format(process.returncode, tail) + self.log("Warn: Annual: {}".format(self.error)) + return + + try: + self.results = json.loads(stdout_data.decode("utf-8", errors="replace")) + except ValueError as exception: + # A zero exit with unreadable output is worse than a crash: it would + # otherwise render as an empty result that looks like a real answer. + self.state = "failed" + self.results = None + self.error = "The annual run finished but its output could not be read: {}\n{}".format(exception, tail) + self.log("Warn: Annual: {}".format(self.error)) + return + + self.state = "complete" + if not self.message or self.message == "Starting": + # Only fall back to a generic message when the child never reported + # one of its own - the last progress message is more informative. + self.message = "Complete" + if self.total: + self.completed = self.total diff --git a/apps/predbat/tests/annual_stub.py b/apps/predbat/tests/annual_stub.py new file mode 100644 index 000000000..dce9d8563 --- /dev/null +++ b/apps/predbat/tests/annual_stub.py @@ -0,0 +1,52 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""A stand-in for annual_cli.py --machine, used to drive AnnualJob in tests. + +Behaviour is chosen by argv[1] so one script covers every case the job control +has to survive, without ever running the real three-minute engine. +""" + +import json +import sys +import time + + +def main(): + """Emit the behaviour named by argv[1] and exit with a matching code.""" + mode = sys.argv[1] if len(sys.argv) > 1 else "ok" + + if mode == "ok": + for step in range(1, 4): + sys.stderr.write(json.dumps({"completed": step, "total": 3, "message": "step {}".format(step)}) + "\n") + sys.stderr.flush() + json.dump({"year": 2025, "months": [], "annual": {"months_included": 0}}, sys.stdout) + return 0 + + if mode == "garbage_progress": + sys.stderr.write("not json at all\n") + sys.stderr.write(json.dumps({"completed": 1, "total": 1, "message": "recovered"}) + "\n") + sys.stderr.flush() + json.dump({"year": 2025, "months": [], "annual": {"months_included": 0}}, sys.stdout) + return 0 + + if mode == "fail": + sys.stderr.write("something went wrong\n") + return 3 + + if mode == "bad_output": + sys.stdout.write("this is not json") + return 0 + + if mode == "hang": + while True: + time.sleep(0.1) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/apps/predbat/tests/test_annual_job.py b/apps/predbat/tests/test_annual_job.py new file mode 100644 index 000000000..ef7cc6136 --- /dev/null +++ b/apps/predbat/tests/test_annual_job.py @@ -0,0 +1,130 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's subprocess job control.""" + +import asyncio +import os +import sys + +from annual_job import AnnualJob + +STUB = os.path.join(os.path.dirname(__file__), "annual_stub.py") + + +def stub_command(mode): + """Return the argv for the stub child in the given mode.""" + return [sys.executable, STUB, mode] + + +async def run_to_completion(job, mode, timeout=20): + """Start the stub in the given mode and wait for the job to leave 'running'.""" + started = await job.start(stub_command(mode)) + waited = 0.0 + while job.state == "running" and waited < timeout: + await asyncio.sleep(0.1) + waited += 0.1 + return started + + +def test_annual_job(my_predbat): + """Verify progress parsing, completion, failure, cancellation and refusal to double-run.""" + failed = False + print("**** Testing annual_job ****") + messages = [] + + print("Test: a successful run parses progress and returns the results document") + job = AnnualJob(log=messages.append) + started = asyncio.run(run_to_completion(job, "ok")) + if not started: + print(" ERROR: start() should return True for a fresh job") + failed = True + if job.state != "complete": + print(" ERROR: expected state 'complete', got {} ({})".format(job.state, job.status().get("error"))) + failed = True + if job.status().get("completed") != 3 or job.status().get("total") != 3: + print(" ERROR: final progress should be 3/3, got {}".format(job.status())) + failed = True + if (job.results or {}).get("year") != 2025: + print(" ERROR: the results document should be parsed from stdout, got {}".format(job.results)) + failed = True + + print("Test: a malformed progress line does not crash the parser") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "garbage_progress")) + if job.state != "complete": + print(" ERROR: a garbage progress line should not fail the run, got {}".format(job.state)) + failed = True + if job.status().get("message") != "recovered": + print(" ERROR: parsing should recover after a bad line, got {}".format(job.status())) + failed = True + + print("Test: a non-zero exit is reported as failed, with the child's stderr kept") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "fail")) + if job.state != "failed": + print(" ERROR: expected state 'failed', got {}".format(job.state)) + failed = True + error_text = job.status().get("error") or "" + if "something went wrong" not in error_text: + print(" ERROR: the child's stderr should be reported, got {!r}".format(error_text)) + failed = True + if "3" not in error_text: + print(" ERROR: the exit code should be reported, got {!r}".format(error_text)) + failed = True + + print("Test: unparseable stdout is reported as failed rather than a silent empty result") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "bad_output")) + if job.state != "failed": + print(" ERROR: unparseable output should fail the run, got {}".format(job.state)) + failed = True + if job.results is not None: + print(" ERROR: no results should be exposed after a parse failure, got {}".format(job.results)) + failed = True + + print("Test: a second start while running is refused, and cancel stops the child") + + async def double_start_then_cancel(): + """Start a hanging child, try to start another, then cancel.""" + job = AnnualJob(log=messages.append) + first = await job.start(stub_command("hang")) + await asyncio.sleep(0.5) + second = await job.start(stub_command("hang")) + cancelled = await job.cancel() + waited = 0.0 + while job.state == "running" and waited < 10: + await asyncio.sleep(0.1) + waited += 0.1 + return first, second, cancelled, job + + first, second, cancelled, job = asyncio.run(double_start_then_cancel()) + if not first: + print(" ERROR: the first start should succeed") + failed = True + if second: + print(" ERROR: a second start while running must be refused") + failed = True + if not cancelled: + print(" ERROR: cancel should report that it acted") + failed = True + if job.state != "cancelled": + print(" ERROR: expected state 'cancelled', got {}".format(job.state)) + failed = True + + print("Test: a fresh job reports idle with no results") + job = AnnualJob(log=messages.append) + if job.state != "idle" or job.results is not None: + print(" ERROR: a fresh job should be idle with no results, got {} / {}".format(job.state, job.results)) + failed = True + if job.status().get("elapsed") != 0: + print(" ERROR: an idle job should report zero elapsed, got {}".format(job.status())) + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index e5b6ac648..c098f912d 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -170,6 +170,7 @@ from tests.test_annual_results import test_annual_results from tests.test_annual_integration import test_annual_integration from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine, test_annual_cli_machine_end_to_end +from tests.test_annual_job import test_annual_job from tests.test_tariff_catalogue import test_tariff_catalogue # Mock the components and plugin system @@ -421,6 +422,7 @@ def main(): ("annual_cli", test_annual_cli, "Annual prediction CLI output tests", False), ("annual_cli_machine", test_annual_cli_machine, "Annual CLI machine mode tests", False), ("annual_cli_machine_end_to_end", test_annual_cli_machine_end_to_end, "Annual CLI machine mode end-to-end tests", False), + ("annual_job", test_annual_job, "Annual subprocess job control tests", False), ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), ] From 4b166146b89f28ae47b6c64d9cc455e4c1835655 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:46:19 +0200 Subject: [PATCH 051/119] fix(annual): close a stale-supervisor race and other job control gaps Code review on the Annual tab's subprocess job control found a critical race: a cancelled run's supervisor task could resume inside the window where start() had already flipped state to "running" for a new run but had not yet reassigned self._process (subprocess creation is awaited), letting the old supervisor's identity check pass and stamp its own (stale) outcome over the new run. Fixed with a generation counter bumped synchronously as the first thing start() does, which has no such window. Also: reference the supervisor task and guard it with a broad except plus a done-callback so it can never vanish leaving state stuck at "running"; freeze elapsed once a run reaches a terminal state instead of counting up forever; reap the child after kill() in cancel() so returncode is set before cancel() returns; treat valid-but-non-dict stdout as a parse failure, not an empty success; restore the unconditional terminal "Complete" message (the test was wrong to check message instead of parser recovery); and add coverage for the race, a start failure, restart-after-complete, double cancel, cancel-when-idle, and concurrent large stdout/stderr streams. Co-Authored-By: Claude Opus 5 (1M context) --- apps/predbat/annual_job.py | 121 ++++++++++++++++++++----- apps/predbat/tests/annual_stub.py | 26 ++++++ apps/predbat/tests/test_annual_job.py | 124 ++++++++++++++++++++++++-- 3 files changed, 242 insertions(+), 29 deletions(-) diff --git a/apps/predbat/annual_job.py b/apps/predbat/annual_job.py index 68cbed08e..3e6ebda9c 100644 --- a/apps/predbat/annual_job.py +++ b/apps/predbat/annual_job.py @@ -42,14 +42,22 @@ def __init__(self, log): self.error = None self.results = None self.started_at = None + self.finished_at = None self._process = None + self._task = None self._stderr_tail = [] + # Bumped synchronously - with no `await` in between - the instant a new + # run is accepted. A supervisor captures the value current at its own + # spawn and must not touch shared state once it no longer matches: see + # the comment in _supervise for why comparing `self._process` alone is + # not enough. + self._generation = 0 def status(self): """Return a JSON-serialisable snapshot for the polling endpoint.""" elapsed = 0 if self.started_at is not None: - end = self.started_at if self.state == "idle" else time.time() + end = self.finished_at if self.finished_at is not None else time.time() elapsed = int(end - self.started_at) return { "state": self.state, @@ -70,6 +78,15 @@ async def start(self, command): self.log("Warn: Annual: a run is already in progress, refusing to start another") return False + # Bumped first, synchronously, before anything else: this is what a + # stale supervisor checks to know it has been superseded. Setting it + # here - rather than after the subprocess exists - closes the window + # where `self.state` already says "running" for the new run but + # `self._process` has not been reassigned yet, which would otherwise + # let an old supervisor's identity check pass by accident. + self._generation += 1 + generation = self._generation + self.state = "running" self.completed = 0 self.total = 0 @@ -77,45 +94,80 @@ async def start(self, command): self.error = None self.results = None self.started_at = time.time() + self.finished_at = None self._stderr_tail = [] try: - self._process = await asyncio.create_subprocess_exec(*command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) + process = await asyncio.create_subprocess_exec(*command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) except (OSError, ValueError) as exception: - self.state = "failed" - self.error = "Could not start the annual run: {}".format(exception) - self.log("Warn: Annual: {}".format(self.error)) + if self._generation == generation: + self.state = "failed" + self.finished_at = time.time() + self.error = "Could not start the annual run: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) return False - asyncio.ensure_future(self._supervise()) + self._process = process + self._task = asyncio.create_task(self._supervise(generation, process)) + self._task.add_done_callback(self._on_task_done) return True async def cancel(self): """Ask the child to stop, killing it if it does not. Returns False if nothing was running.""" if self.state != "running" or self._process is None: return False + process = self._process self.state = "cancelled" + self.finished_at = time.time() self.message = "Cancelled" try: - self._process.terminate() + process.terminate() except ProcessLookupError: return True try: - await asyncio.wait_for(self._process.wait(), timeout=CANCEL_GRACE_SECONDS) + await asyncio.wait_for(process.wait(), timeout=CANCEL_GRACE_SECONDS) except asyncio.TimeoutError: self.log("Warn: Annual: the run did not stop when asked, killing it") try: - self._process.kill() + process.kill() except ProcessLookupError: pass + else: + # Reap it - without this, cancel() could return before the + # child is actually gone, leaving returncode unset. + await process.wait() return True - async def _read_progress(self, stream): - """Consume the child's stderr, updating progress and keeping a tail for errors.""" + def _on_task_done(self, task): + """Log if the supervisor task ended by raising rather than settling a terminal state itself. + + Every expected outcome inside `_supervise` is caught and turned into a + terminal state there. This callback is the fallback for anything that + still escapes - otherwise the job would be stuck reporting 'running' + forever, refusing every further start(). + """ + if task.cancelled(): + return + exception = task.exception() + if exception is not None: + self.log("Warn: Annual: the supervisor task ended unexpectedly: {}".format(exception)) + + async def _read_progress(self, generation, stream): + """Consume the child's stderr, updating progress and keeping a tail for errors. + + `generation` is the run this call was started for, captured by the + caller at spawn time. If a later run has since been accepted - because + cancel() returned before this stream had drained, and a fresh start() + followed straight after - the line is still consumed so the pipe keeps + draining, but it must not be allowed to overwrite the newer run's + progress or error tail. + """ while True: line = await stream.readline() if not line: break + if self._generation != generation: + continue text = line.decode("utf-8", errors="replace").strip() if not text: continue @@ -133,16 +185,34 @@ async def _read_progress(self, stream): self.total = record.get("total", self.total) self.message = record.get("message", self.message) - async def _supervise(self): - """Read both streams to completion, then settle the final state.""" - process = self._process + async def _supervise(self, generation, process): + """Read both streams to completion for this child, then settle the final state. + + `generation` is captured by the caller at the instant this run was + accepted, before the child even existed. Comparing `self._process` to + this call's `process` is not sufficient on its own: `start()` sets + `self.state = "running"` before it awaits the subprocess's creation, + so there is a window where a new run is already "running" but + `self._process` still points at the previous child. A stale supervisor + resuming inside that window would see its own `process` still matching + `self._process`, and `self.state` no longer "cancelled", and wrongly + conclude it was safe to report its own child's outcome. The generation + counter has no such window - it is bumped synchronously as the very + first thing `start()` does - so it is the only reliable guard. + """ try: - stdout_data, _ = await asyncio.gather(process.stdout.read(), self._read_progress(process.stderr)) + stdout_data, _ = await asyncio.gather(process.stdout.read(), self._read_progress(generation, process.stderr)) await process.wait() - except (OSError, ValueError) as exception: - self.state = "failed" - self.error = "The annual run could not be read: {}".format(exception) - self.log("Warn: Annual: {}".format(self.error)) + except Exception as exception: # noqa: BLE001 - a supervisor must never die silently, see _on_task_done + if self._generation == generation: + self.state = "failed" + self.finished_at = time.time() + self.error = "The annual run could not be read: {}".format(exception) + self.log("Warn: Annual: {}".format(self.error)) + return + + if self._generation != generation: + # Superseded by a later run - its own supervisor owns state now. return if self.state == "cancelled": @@ -152,25 +222,28 @@ async def _supervise(self): if process.returncode != 0: self.state = "failed" + self.finished_at = time.time() self.error = "The annual run exited with code {}.\n{}".format(process.returncode, tail) self.log("Warn: Annual: {}".format(self.error)) return try: - self.results = json.loads(stdout_data.decode("utf-8", errors="replace")) + results = json.loads(stdout_data.decode("utf-8", errors="replace")) + if not isinstance(results, dict): + raise ValueError("the results document was not a JSON object, got {}".format(type(results).__name__)) except ValueError as exception: # A zero exit with unreadable output is worse than a crash: it would # otherwise render as an empty result that looks like a real answer. self.state = "failed" + self.finished_at = time.time() self.results = None self.error = "The annual run finished but its output could not be read: {}\n{}".format(exception, tail) self.log("Warn: Annual: {}".format(self.error)) return + self.results = results self.state = "complete" - if not self.message or self.message == "Starting": - # Only fall back to a generic message when the child never reported - # one of its own - the last progress message is more informative. - self.message = "Complete" + self.finished_at = time.time() + self.message = "Complete" if self.total: self.completed = self.total diff --git a/apps/predbat/tests/annual_stub.py b/apps/predbat/tests/annual_stub.py index dce9d8563..bc5edbad4 100644 --- a/apps/predbat/tests/annual_stub.py +++ b/apps/predbat/tests/annual_stub.py @@ -41,6 +41,32 @@ def main(): sys.stdout.write("this is not json") return 0 + if mode == "null_output": + # Valid JSON, but not the object AnnualJob's results document must be. + json.dump(None, sys.stdout) + return 0 + + if mode == "big_streams": + # Push well over the default OS pipe buffer (typically 64 KiB) through + # both stdout and stderr, so a job control that reads one stream to + # completion before draining the other would deadlock: this child + # would block writing to the second pipe while nothing reads it. + # Stderr is spread across many moderate lines rather than one huge + # one - a single line over the StreamReader's own line-length limit + # would raise an unrelated error and prove nothing about deadlocking. + line_filler = "x" * 500 + written = 0 + step = 0 + while written < 200000: + step += 1 + payload = json.dumps({"completed": step, "total": step, "message": line_filler}) + sys.stderr.write(payload + "\n") + written += len(payload) + 1 + sys.stderr.flush() + filler = "x" * 200000 + json.dump({"year": 2025, "months": [], "annual": {"months_included": 0}, "filler": filler}, sys.stdout) + return 0 + if mode == "hang": while True: time.sleep(0.1) diff --git a/apps/predbat/tests/test_annual_job.py b/apps/predbat/tests/test_annual_job.py index ef7cc6136..37fb1f6f6 100644 --- a/apps/predbat/tests/test_annual_job.py +++ b/apps/predbat/tests/test_annual_job.py @@ -3,7 +3,7 @@ # Copyright Trefor Southwell 2026 - All Rights Reserved # This application maybe used for personal use only and not for commercial use # ----------------------------------------------------------------------------- -# fmt off +# fmt: off # pylint: disable=consider-using-f-string # pylint: disable=line-too-long @@ -12,6 +12,7 @@ import asyncio import os import sys +import time from annual_job import AnnualJob @@ -54,6 +55,12 @@ def test_annual_job(my_predbat): if (job.results or {}).get("year") != 2025: print(" ERROR: the results document should be parsed from stdout, got {}".format(job.results)) failed = True + elapsed_first = job.status().get("elapsed") + time.sleep(0.2) + elapsed_second = job.status().get("elapsed") + if elapsed_first != elapsed_second: + print(" ERROR: elapsed should freeze once a run is complete, got {} then {}".format(elapsed_first, elapsed_second)) + failed = True print("Test: a malformed progress line does not crash the parser") job = AnnualJob(log=messages.append) @@ -61,7 +68,12 @@ def test_annual_job(my_predbat): if job.state != "complete": print(" ERROR: a garbage progress line should not fail the run, got {}".format(job.state)) failed = True - if job.status().get("message") != "recovered": + # The proof that the parser recovered is the line *after* the garbage one + # being applied - completed/total went from 0/0 to 1/1. Asserting on the + # terminal message instead would test message policy, not the parser, and + # a correct implementation is free to replace the last progress message + # with a generic "Complete" once the run finishes. + if job.status().get("completed") != 1 or job.status().get("total") != 1: print(" ERROR: parsing should recover after a bad line, got {}".format(job.status())) failed = True @@ -75,8 +87,8 @@ def test_annual_job(my_predbat): if "something went wrong" not in error_text: print(" ERROR: the child's stderr should be reported, got {!r}".format(error_text)) failed = True - if "3" not in error_text: - print(" ERROR: the exit code should be reported, got {!r}".format(error_text)) + if "exited with code 3" not in error_text: + print(" ERROR: the exit code should be reported precisely, got {!r}".format(error_text)) failed = True print("Test: unparseable stdout is reported as failed rather than a silent empty result") @@ -89,7 +101,27 @@ def test_annual_job(my_predbat): print(" ERROR: no results should be exposed after a parse failure, got {}".format(job.results)) failed = True - print("Test: a second start while running is refused, and cancel stops the child") + print("Test: valid JSON that is not an object is reported as failed, not a silent empty result") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "null_output")) + if job.state != "failed": + print(" ERROR: a non-object results document should fail the run, got {}".format(job.state)) + failed = True + if job.results is not None: + print(" ERROR: no results should be exposed after a non-object parse, got {}".format(job.results)) + failed = True + + print("Test: streams bigger than a pipe buffer on both stdout and stderr do not deadlock") + job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(job, "big_streams", timeout=30)) + if job.state != "complete": + print(" ERROR: a large-output run should still complete, got {} ({})".format(job.state, job.status().get("error"))) + failed = True + if (job.results or {}).get("year") != 2025: + print(" ERROR: the large results document should still be parsed, got keys {}".format(list((job.results or {}).keys()))) + failed = True + + print("Test: a second start while running is refused, and cancel stops and reaps the child") async def double_start_then_cancel(): """Start a hanging child, try to start another, then cancel.""" @@ -117,6 +149,74 @@ async def double_start_then_cancel(): if job.state != "cancelled": print(" ERROR: expected state 'cancelled', got {}".format(job.state)) failed = True + # Proof that the child was actually reaped, not just that the state string + # was set: `state` is assigned synchronously before terminate() is even + # called, so checking `state` alone would pass even if cancel() never + # touched the process. + if job._process is None or job._process.returncode is None: + print(" ERROR: cancel should have reaped the child, but returncode is {}".format(job._process and job._process.returncode)) + failed = True + + print("Test: a run started immediately after cancelling is not clobbered by the old supervisor") + + async def cancel_then_restart_immediately(): + """Cancel a hanging child, then start a fresh run before the old supervisor has drained its pipes.""" + job = AnnualJob(log=messages.append) + await job.start(stub_command("hang")) + await asyncio.sleep(0.2) + await job.cancel() + # No delay here: the old supervisor task is still scheduled to run and + # may not have observed the cancellation yet - that overlap is exactly + # what let a stale supervisor stamp its state over a new run. + started = await job.start(stub_command("ok")) + waited = 0.0 + while job.state == "running" and waited < 20: + await asyncio.sleep(0.1) + waited += 0.1 + return started, job + + restart_started, restart_job = asyncio.run(cancel_then_restart_immediately()) + if not restart_started: + print(" ERROR: starting again immediately after cancel should succeed") + failed = True + if restart_job.state != "complete": + print(" ERROR: the new run should complete cleanly, got {} ({})".format(restart_job.state, restart_job.status().get("error"))) + failed = True + if (restart_job.results or {}).get("year") != 2025: + print(" ERROR: the new run's results must not be clobbered by the superseded supervisor, got {}".format(restart_job.results)) + failed = True + + print("Test: cancelling an already-cancelled job is a no-op that reports nothing to act on") + second_cancel = asyncio.run(job.cancel()) + if second_cancel: + print(" ERROR: cancelling a job that is not running should return False, got {}".format(second_cancel)) + failed = True + + print("Test: cancelling an idle job reports nothing to act on") + idle_job = AnnualJob(log=messages.append) + idle_cancel = asyncio.run(idle_job.cancel()) + if idle_cancel: + print(" ERROR: cancelling an idle job should return False, got {}".format(idle_cancel)) + failed = True + + print("Test: a bad command is a start failure, not a silent no-op") + + async def failed_start(): + """Try to start a command that cannot be executed at all.""" + job = AnnualJob(log=messages.append) + started = await job.start(["/nonexistent/predbat-annual-stub-does-not-exist"]) + return job, started + + bad_job, bad_started = asyncio.run(failed_start()) + if bad_started: + print(" ERROR: starting a non-existent command should return False") + failed = True + if bad_job.state != "failed": + print(" ERROR: a start failure should leave the job in state 'failed', got {}".format(bad_job.state)) + failed = True + if not any("Could not start the annual run" in message for message in messages): + print(" ERROR: a start failure should be logged, got {}".format(messages)) + failed = True print("Test: a fresh job reports idle with no results") job = AnnualJob(log=messages.append) @@ -127,4 +227,18 @@ async def double_start_then_cancel(): print(" ERROR: an idle job should report zero elapsed, got {}".format(job.status())) failed = True + print("Test: a job that has already completed can be started again") + reused_job = AnnualJob(log=messages.append) + asyncio.run(run_to_completion(reused_job, "ok")) + if reused_job.state != "complete": + print(" ERROR: the first run on the reused job should complete, got {}".format(reused_job.state)) + failed = True + second_started = asyncio.run(run_to_completion(reused_job, "ok")) + if not second_started: + print(" ERROR: starting again after completion should succeed") + failed = True + if reused_job.state != "complete": + print(" ERROR: the second run should also complete, got {}".format(reused_job.state)) + failed = True + return failed From 6c2c7ebe8b7a6a3418ace373790ccadd02518fdf Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 21:57:37 +0200 Subject: [PATCH 052/119] feat(annual): add the five-run store for the Annual tab --- apps/predbat/annual_store.py | 129 ++++++++++++++++ apps/predbat/tests/test_annual_store.py | 188 ++++++++++++++++++++++++ apps/predbat/unit_test.py | 2 + 3 files changed, 319 insertions(+) create mode 100644 apps/predbat/annual_store.py create mode 100644 apps/predbat/tests/test_annual_store.py diff --git a/apps/predbat/annual_store.py b/apps/predbat/annual_store.py new file mode 100644 index 000000000..8ba674938 --- /dev/null +++ b/apps/predbat/annual_store.py @@ -0,0 +1,129 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""Storage-backed history of annual prediction runs. + +Keeps the most recent runs so a user can flip between "with a 5 kWh battery" and +"with a 10 kWh battery" without re-running either. Everything goes through the +Storage abstraction rather than the filesystem, because there may not be one. +""" + +import datetime + +MAX_RUNS = 5 +STORAGE_MODULE = "annual" +INDEX_NAME = "runs_index" + + +def _run_key(run_id): + """Return the storage filename holding one run's results document.""" + return "run_{}".format(run_id) + + +def _describe_tariff(tariff): + """Return a short human name for the tariff a run used.""" + if not isinstance(tariff, dict): + return "tariff" + url = tariff.get("import_octopus_url") or "" + for name in ["AGILE", "INTELLI-FLUX", "INTELLI", "FLUX", "COSY", "SNUG", "GO"]: + if name in url.upper(): + return name.title().replace("Intelli-Flux", "Intelligent Flux").replace("Intelli", "Intelligent Go") + if tariff.get("rates_import"): + return "fixed rates" + return "tariff" + + +def build_label(config): + """Return a short human label describing the configuration a run used. + + A selector listing five bare timestamps tells the user nothing about which + run was which, which defeats the point of keeping more than one. + """ + parts = [] + battery = config.get("battery") if isinstance(config, dict) else None + if isinstance(battery, dict) and battery.get("size_kwh"): + parts.append("{}kWh battery".format(battery["size_kwh"])) + else: + parts.append("no battery") + + solar = config.get("solar") if isinstance(config, dict) else None + if solar: + total_kwp = sum(array.get("kwp", 0) for array in solar if isinstance(array, dict)) + if total_kwp: + parts.append("{}kWp".format(round(total_kwp, 2))) + else: + parts.append("no solar") + + parts.append(_describe_tariff((config or {}).get("tariff"))) + return " · ".join(parts) + + +async def list_runs(storage): + """Return the stored runs newest-first, or an empty list when there are none. + + A corrupt or unexpected index reads as empty rather than raising: the tab + must still render so the user can start a fresh run. + """ + if not storage: + return [] + index = await storage.load(STORAGE_MODULE, INDEX_NAME) + if not isinstance(index, list): + return [] + return [entry for entry in index if isinstance(entry, dict) and entry.get("id")] + + +async def load_run(storage, run_id): + """Return one run's results document, or None when it is unknown or missing.""" + if not storage or not run_id: + return None + return await storage.load(STORAGE_MODULE, _run_key(run_id)) + + +async def _discard_run(storage, run_id): + """Remove an evicted run's stored document so the ring does not leak documents. + + The real Storage component has no ``delete`` method, so the primary path is + to overwrite the document with ``None`` and set an expiry in the past, which + lets ``storage.cleanup()`` reclaim it later. A ``delete`` method is still + preferred when a backend (or the test's fake) provides one. + """ + if hasattr(storage, "delete"): + await storage.delete(STORAGE_MODULE, _run_key(run_id)) + return + expired = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=1) + await storage.save(STORAGE_MODULE, _run_key(run_id), None, format="json", expiry=expired) + + +async def save_run(storage, results, config, run_id): + """Save a completed run and prune the ring to MAX_RUNS. Returns the run id. + + The evicted run's document is discarded as well as its index entry, so the + ring cannot leak documents that nothing references. + """ + if not storage: + return run_id + + await storage.save(STORAGE_MODULE, _run_key(run_id), results, format="json") + + annual = results.get("annual", {}) if isinstance(results, dict) else {} + entry = { + "id": run_id, + "timestamp": run_id, + "label": build_label(config), + "months_included": annual.get("months_included", 0), + "status": "ok" if annual.get("months_included") else "empty", + } + + index = await list_runs(storage) + index = [existing for existing in index if existing.get("id") != run_id] + index.insert(0, entry) + + for dropped in index[MAX_RUNS:]: + await _discard_run(storage, dropped["id"]) + index = index[:MAX_RUNS] + + await storage.save(STORAGE_MODULE, INDEX_NAME, index, format="json") + return run_id diff --git a/apps/predbat/tests/test_annual_store.py b/apps/predbat/tests/test_annual_store.py new file mode 100644 index 000000000..2ef14e9a7 --- /dev/null +++ b/apps/predbat/tests/test_annual_store.py @@ -0,0 +1,188 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's five-run store.""" + +import asyncio +import datetime + +from annual_store import INDEX_NAME, MAX_RUNS, STORAGE_MODULE, build_label, list_runs, load_run, save_run + + +class FakeStorage: + """An in-memory stand-in for the Storage component that supports ``delete``. + + Exercises the primary eviction path in ``annual_store``, where a genuine + ``delete`` method is available and preferred over the fallback. + """ + + def __init__(self): + """Start with nothing stored.""" + self.store = {} + self.deleted = [] + + async def save(self, module, filename, data, format="yaml", expiry=None): + """Record a saved value.""" + self.store[(module, filename)] = data + + async def load(self, module, filename): + """Return a stored value, or None.""" + return self.store.get((module, filename)) + + async def delete(self, module, filename): + """Remove a stored value and record that it happened.""" + self.deleted.append(filename) + self.store.pop((module, filename), None) + + +class FakeStorageNoDelete: + """An in-memory stand-in for the Storage component with no ``delete`` method. + + This matches the real Storage component's actual surface (``save``, ``load``, + ``age``, ``cleanup``, ``fetch_cached`` — no ``delete``), so it exercises the + fallback eviction path: overwriting the evicted document with ``None`` and a + past expiry, via ``save``, rather than calling a method that does not exist. + """ + + def __init__(self): + """Start with nothing stored.""" + self.store = {} + self.expiries = {} + + async def save(self, module, filename, data, format="yaml", expiry=None): + """Record a saved value and the expiry it was saved with, if any.""" + self.store[(module, filename)] = data + self.expiries[(module, filename)] = expiry + + async def load(self, module, filename): + """Return a stored value, or None.""" + return self.store.get((module, filename)) + + +def sample_results(cost): + """Return a minimal results document with a distinguishing cost.""" + return {"year": 2025, "annual": {"scenarios": {"with_predbat": {"cost_p": cost}}, "months_included": 12}, "months": []} + + +def sample_config(size_kwh=9.5): + """Return a minimal validated-shape config.""" + return {"battery": {"size_kwh": size_kwh}, "solar": [{"kwp": 5.6}], "tariff": {"import_octopus_url": "https://example.com/AGILE-24-10-01/x"}} + + +def test_annual_store(my_predbat): + """Verify saving, listing, loading, eviction (both code paths) and label generation.""" + failed = False + print("**** Testing annual_store ****") + + print("Test: a saved run appears in the index and can be loaded back") + storage = FakeStorage() + run_id = asyncio.run(save_run(storage, sample_results(100), sample_config(), "run-1")) + if run_id != "run-1": + print(" ERROR: save_run should return the id it was given, got {}".format(run_id)) + failed = True + index = asyncio.run(list_runs(storage)) + if len(index) != 1 or index[0]["id"] != "run-1": + print(" ERROR: expected one indexed run, got {}".format(index)) + failed = True + loaded = asyncio.run(load_run(storage, "run-1")) + if (loaded or {}).get("annual", {}).get("scenarios", {}).get("with_predbat", {}).get("cost_p") != 100: + print(" ERROR: the loaded run should match what was saved, got {}".format(loaded)) + failed = True + + print("Test: the index is newest-first") + asyncio.run(save_run(storage, sample_results(200), sample_config(), "run-2")) + index = asyncio.run(list_runs(storage)) + if [entry["id"] for entry in index] != ["run-2", "run-1"]: + print(" ERROR: expected newest first, got {}".format([entry["id"] for entry in index])) + failed = True + + print("Test: a sixth run evicts the oldest AND deletes its stored document (delete-capable backend)") + storage = FakeStorage() + for number in range(1, MAX_RUNS + 2): + asyncio.run(save_run(storage, sample_results(number), sample_config(), "run-{}".format(number))) + index = asyncio.run(list_runs(storage)) + if len(index) != MAX_RUNS: + print(" ERROR: the ring should hold {} runs, got {}".format(MAX_RUNS, len(index))) + failed = True + if "run-1" in [entry["id"] for entry in index]: + print(" ERROR: the oldest run should have been evicted from the index") + failed = True + if "run_run-1" not in storage.deleted: + print(" ERROR: the evicted run's document should be deleted, deletions were {}".format(storage.deleted)) + failed = True + if asyncio.run(load_run(storage, "run-1")) is not None: + print(" ERROR: an evicted run should no longer load") + failed = True + + print("Test: a sixth run evicts the oldest via the fallback path when delete is unavailable") + no_delete_storage = FakeStorageNoDelete() + for number in range(1, MAX_RUNS + 2): + asyncio.run(save_run(no_delete_storage, sample_results(number), sample_config(), "run-{}".format(number))) + index = asyncio.run(list_runs(no_delete_storage)) + if len(index) != MAX_RUNS: + print(" ERROR: the ring should hold {} runs without delete, got {}".format(MAX_RUNS, len(index))) + failed = True + if "run-1" in [entry["id"] for entry in index]: + print(" ERROR: the oldest run should have been evicted from the index without delete") + failed = True + if asyncio.run(load_run(no_delete_storage, "run-1")) is not None: + print(" ERROR: an evicted run should read back as None when the fallback blanking is used") + failed = True + evicted_expiry = no_delete_storage.expiries.get((STORAGE_MODULE, "run_run-1")) + if evicted_expiry is None: + print(" ERROR: the fallback should set an expiry on the blanked document so cleanup() can reclaim it") + failed = True + elif evicted_expiry.tzinfo is None: + print(" ERROR: the fallback expiry should be timezone-aware, got {!r}".format(evicted_expiry)) + failed = True + elif evicted_expiry >= datetime.datetime.now(datetime.timezone.utc): + print(" ERROR: the fallback expiry should be in the past, got {!r}".format(evicted_expiry)) + failed = True + + print("Test: loading an unknown or missing run returns None rather than raising") + if asyncio.run(load_run(storage, "does-not-exist")) is not None: + print(" ERROR: an unknown run id should give None") + failed = True + + print("Test: an index entry whose document is missing is reported, not rendered empty") + storage = FakeStorage() + asyncio.run(save_run(storage, sample_results(1), sample_config(), "orphan")) + storage.store.pop((STORAGE_MODULE, "run_orphan")) + if asyncio.run(load_run(storage, "orphan")) is not None: + print(" ERROR: a missing document should give None so the caller can say so") + failed = True + + print("Test: an empty store lists nothing rather than raising") + if asyncio.run(list_runs(FakeStorage())) != []: + print(" ERROR: an empty store should list no runs") + failed = True + + print("Test: the label describes the configuration, not just a timestamp") + label = build_label(sample_config(size_kwh=9.5)) + if "9.5" not in label or "5.6" not in label: + print(" ERROR: the label should name the battery and array size, got {!r}".format(label)) + failed = True + if "Agile" not in label: + print(" ERROR: the label should name the tariff, got {!r}".format(label)) + failed = True + + print("Test: a label is still produced for a config with no battery or solar") + label = build_label({"tariff": {"rates_import": [{"rate": 25.0}]}}) + if not label: + print(" ERROR: a sparse config should still produce a label") + failed = True + + print("Test: the index survives a corrupt stored value") + storage = FakeStorage() + storage.store[(STORAGE_MODULE, INDEX_NAME)] = "not a list" + if asyncio.run(list_runs(storage)) != []: + print(" ERROR: a corrupt index should read as empty rather than raising") + failed = True + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index c098f912d..00af33e1a 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -172,6 +172,7 @@ from tests.test_annual_cli import test_annual_cli, test_annual_cli_machine, test_annual_cli_machine_end_to_end from tests.test_annual_job import test_annual_job from tests.test_tariff_catalogue import test_tariff_catalogue +from tests.test_annual_store import test_annual_store # Mock the components and plugin system @@ -423,6 +424,7 @@ def main(): ("annual_cli_machine", test_annual_cli_machine, "Annual CLI machine mode tests", False), ("annual_cli_machine_end_to_end", test_annual_cli_machine_end_to_end, "Annual CLI machine mode end-to-end tests", False), ("annual_job", test_annual_job, "Annual subprocess job control tests", False), + ("annual_store", test_annual_store, "Annual run store tests", False), ("tariff_catalogue", test_tariff_catalogue, "Tariff catalogue tests", False), ] From de1f680be931f78951b77b2b1ed71c6d92762b06 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:04:42 +0200 Subject: [PATCH 053/119] docs: fix build_label raising on a truthy non-dict config The plan's build_label used `(config or {}).get(...)`, which short-circuits past the guard for any truthy non-dict - a string, an int, a non-empty list, i.e. a corrupted stored document - and then raises. A label failure would have taken out the whole Annual tab, which is the page the user needs in order to start a fresh run. Co-Authored-By: Claude Opus 5 (1M context) --- docs/superpowers/plans/2026-07-26-annual-web-ui.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-annual-web-ui.md b/docs/superpowers/plans/2026-07-26-annual-web-ui.md index db213a20d..c2306399e 100644 --- a/docs/superpowers/plans/2026-07-26-annual-web-ui.md +++ b/docs/superpowers/plans/2026-07-26-annual-web-ui.md @@ -1234,14 +1234,20 @@ def build_label(config): A selector listing five bare timestamps tells the user nothing about which run was which, which defeats the point of keeping more than one. """ + # Normalise once: `config or {}` is NOT enough, because any truthy non-dict + # (a string, an int, a non-empty list - i.e. a corrupted stored document) + # short-circuits past it and then raises on .get(). A label failure must never + # take out the page the user needs in order to start a fresh run. + config = config if isinstance(config, dict) else {} + parts = [] - battery = config.get("battery") if isinstance(config, dict) else None + battery = config.get("battery") if isinstance(battery, dict) and battery.get("size_kwh"): parts.append("{}kWh battery".format(battery["size_kwh"])) else: parts.append("no battery") - solar = config.get("solar") if isinstance(config, dict) else None + solar = config.get("solar") if solar: total_kwp = sum(array.get("kwp", 0) for array in solar if isinstance(array, dict)) if total_kwp: @@ -1249,7 +1255,7 @@ def build_label(config): else: parts.append("no solar") - parts.append(_describe_tariff((config or {}).get("tariff"))) + parts.append(_describe_tariff(config.get("tariff"))) return " · ".join(parts) From 504694ba40101f64665e934ef71bf4d926b4251f Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:06:00 +0200 Subject: [PATCH 054/119] fix(annual): stop build_label raising on a non-dict config A corrupted stored config (a string, int, or non-empty list) made `config or {}` short-circuit to the truthy non-dict value itself, so the tariff lookup raised AttributeError instead of degrading to a label. Normalise config to a dict once at the top of build_label and use it throughout instead of guarding each lookup individually. --- apps/predbat/annual_store.py | 7 ++++--- apps/predbat/tests/test_annual_store.py | 7 +++++++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/apps/predbat/annual_store.py b/apps/predbat/annual_store.py index 8ba674938..1d8f0faea 100644 --- a/apps/predbat/annual_store.py +++ b/apps/predbat/annual_store.py @@ -42,14 +42,15 @@ def build_label(config): A selector listing five bare timestamps tells the user nothing about which run was which, which defeats the point of keeping more than one. """ + config = config if isinstance(config, dict) else {} parts = [] - battery = config.get("battery") if isinstance(config, dict) else None + battery = config.get("battery") if isinstance(battery, dict) and battery.get("size_kwh"): parts.append("{}kWh battery".format(battery["size_kwh"])) else: parts.append("no battery") - solar = config.get("solar") if isinstance(config, dict) else None + solar = config.get("solar") if solar: total_kwp = sum(array.get("kwp", 0) for array in solar if isinstance(array, dict)) if total_kwp: @@ -57,7 +58,7 @@ def build_label(config): else: parts.append("no solar") - parts.append(_describe_tariff((config or {}).get("tariff"))) + parts.append(_describe_tariff(config.get("tariff"))) return " · ".join(parts) diff --git a/apps/predbat/tests/test_annual_store.py b/apps/predbat/tests/test_annual_store.py index 2ef14e9a7..60a0ae4fd 100644 --- a/apps/predbat/tests/test_annual_store.py +++ b/apps/predbat/tests/test_annual_store.py @@ -178,6 +178,13 @@ def test_annual_store(my_predbat): print(" ERROR: a sparse config should still produce a label") failed = True + print("Test: build_label tolerates a config that is not a dict at all") + for not_a_config in [None, "not-a-dict", 42, [], [1, 2, 3], {"tariff": "also-not-a-dict"}]: + label = build_label(not_a_config) + if not label: + print(" ERROR: build_label should still return a non-empty string for {!r}, got {!r}".format(not_a_config, label)) + failed = True + print("Test: the index survives a corrupt stored value") storage = FakeStorage() storage.store[(STORAGE_MODULE, INDEX_NAME)] = "not a list" From 9e8728ac4ff6bd4ea9e17cbdc7f1196c0a5d5e9a Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:13:45 +0200 Subject: [PATCH 055/119] feat(annual): add Annual tab prefill and config persistence Reads what the live Predbat instance knows via get_arg() (never apps.yaml from disk) and fills every remaining field independently from a typical UK system, so the tab produces a complete, valid config even with nothing configured - the prospective-buyer path this tool exists to serve. Fixed a truncation bug versus the plan's literal code: get_arg() type- directs its return value from the default argument's type, so passing an int default of 0 for soc_max/inverter_limit/export_limit (all declared as sensor_type: float in APPS_SCHEMA) silently truncated real values (12.5 -> 12). Switched those defaults to 0.0 so the float branch is taken instead. --- apps/predbat/tests/test_web_annual.py | 106 ++++++++++++++ apps/predbat/unit_test.py | 2 + apps/predbat/web_annual.py | 194 ++++++++++++++++++++++++++ 3 files changed, 302 insertions(+) create mode 100644 apps/predbat/tests/test_web_annual.py create mode 100644 apps/predbat/web_annual.py diff --git a/apps/predbat/tests/test_web_annual.py b/apps/predbat/tests/test_web_annual.py new file mode 100644 index 000000000..9baabf821 --- /dev/null +++ b/apps/predbat/tests/test_web_annual.py @@ -0,0 +1,106 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the Annual tab's prefill and configuration handling.""" + +from annual import validate_config +from web import WebInterface +from web_annual import DEFAULT_CONFIG, AnnualPage + + +def make_page(my_predbat): + """Return an AnnualPage backed by a WebInterface over the test fixture.""" + return AnnualPage(WebInterface(my_predbat, web_port=5054)) + + +def test_web_annual(my_predbat): + """Verify prefill against a configured and an unconfigured instance.""" + failed = False + print("**** Testing web_annual prefill ****") + + saved_args = dict(my_predbat.args) + try: + print("Test: an unconfigured instance still produces a complete, valid config") + # This is the acceptance criterion for "must work with Predbat unconfigured": + # a prospective buyer, and eventually an unregistered Predbat.com visitor, + # arrives with none of this set. + for key in ["soc_max", "inverter_limit", "export_limit", "open_meteo_forecast", "forecast_solar", "compare_list", "dno_region"]: + my_predbat.args.pop(key, None) + page = make_page(my_predbat) + config = page.prefill_config() + try: + validate_config(config) + except Exception as error: + print(" ERROR: an unconfigured prefill must still validate, got {}".format(error)) + failed = True + if page.is_configured(): + print(" ERROR: with no battery and no solar the page should report unconfigured") + failed = True + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: battery should fall back to the default, got {}".format(config["battery"])) + failed = True + if not config["solar"]: + print(" ERROR: solar should fall back to the default array") + failed = True + + print("Test: a zero soc_max counts as unset and falls back to the default") + my_predbat.args["soc_max"] = 0 + config = make_page(my_predbat).prefill_config() + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: a zero soc_max should fall back, got {}".format(config["battery"]["size_kwh"])) + failed = True + + print("Test: configured values are read from args and used") + my_predbat.args["soc_max"] = 12.5 + my_predbat.args["open_meteo_forecast"] = [{"kwp": 7.2, "declination": 30, "azimuth": 170, "efficiency": 0.9}] + config = make_page(my_predbat).prefill_config() + if config["battery"]["size_kwh"] != 12.5: + print(" ERROR: soc_max from args should be used, got {}".format(config["battery"]["size_kwh"])) + failed = True + if config["solar"][0]["kwp"] != 7.2 or config["solar"][0]["azimuth"] != 170: + print(" ERROR: the solar array should come from args, got {}".format(config["solar"])) + failed = True + + print("Test: prefill is per-field, not all-or-nothing") + # Solar configured but no battery: the real array must survive alongside the + # default battery rather than the whole prefill collapsing to defaults. + my_predbat.args.pop("soc_max", None) + config = make_page(my_predbat).prefill_config() + if config["solar"][0]["kwp"] != 7.2: + print(" ERROR: configured solar should survive an absent battery, got {}".format(config["solar"])) + failed = True + if config["battery"]["size_kwh"] != DEFAULT_CONFIG["battery"]["size_kwh"]: + print(" ERROR: the battery should still fall back, got {}".format(config["battery"])) + failed = True + if not make_page(my_predbat).is_configured(): + print(" ERROR: a configured solar array alone should count as configured") + failed = True + + print("Test: the catalogue merges the user's compare_list") + my_predbat.args["compare_list"] = [{"id": "mine", "name": "My tariff", "rates_import_octopus_url": "https://example.com/x"}] + ids = [entry["id"] for entry in make_page(my_predbat).catalogue()] + if "mine" not in ids: + print(" ERROR: a user compare_list entry should appear in the catalogue, got {}".format(ids)) + failed = True + if "agile_agile" not in ids: + print(" ERROR: built-in entries should still be present, got {}".format(ids)) + failed = True + + print("Test: the default config validates on its own") + try: + validate_config(DEFAULT_CONFIG) + except Exception as error: + print(" ERROR: DEFAULT_CONFIG must be valid, got {}".format(error)) + failed = True + + finally: + my_predbat.args.clear() + my_predbat.args.update(saved_args) + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 00af33e1a..505503734 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -70,6 +70,7 @@ from tests.test_web_charts import run_web_charts_tests from tests.test_web_chart_grouping import run_web_chart_grouping_tests from tests.test_web_entity_unit_resolution import run_web_entity_unit_resolution_tests +from tests.test_web_annual import test_web_annual from tests.test_window import run_window_sort_tests, run_intersect_window_tests from tests.test_find_charge_rate import test_find_charge_rate, test_find_charge_rate_string_temperature, test_find_charge_rate_string_charge_curve from tests.test_manual_api import run_test_manual_api @@ -297,6 +298,7 @@ def main(): ("manual_select", run_test_manual_select, "Manual select tests", False), ("web_if", run_test_web_if, "Web interface tests", False), ("web_functions", run_web_functions_tests, "Web function unit tests", False), + ("web_annual", test_web_annual, "Annual web tab prefill tests", False), ("web_history_table", run_web_history_table_tests, "Web /entity history table bucketing tests", False), ("web_charts", run_web_charts_tests, "Web chart rendering tests (percent/special-character units)", False), ("web_chart_grouping", run_web_chart_grouping_tests, "Web /entity chart numeric vs timeline grouping tests", False), diff --git a/apps/predbat/web_annual.py b/apps/predbat/web_annual.py new file mode 100644 index 000000000..944100ae7 --- /dev/null +++ b/apps/predbat/web_annual.py @@ -0,0 +1,194 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- + +"""The Annual prediction tab. + +Renders the configuration form, drives the subprocess that runs the annual +prediction engine, and presents the results. Prefills from whatever the live +Predbat instance knows and falls back to a typical UK system for the rest, so +the tab is usable by someone who has not configured Predbat at all - which is +the prospective-buyer path the tool exists to serve. +""" + +import copy +import os + +import yaml + +from tariff_catalogue import merged_catalogue + +# A plausible UK home, used for any field the live instance cannot supply. These +# are an EXAMPLE, not a recommendation - the form says so, because a visitor +# could otherwise mistake them for a reading of their own system. +DEFAULT_CONFIG = { + "location": {"postcode": "SW1A 1AA"}, + "solar": [{"kwp": 5.0, "declination": 35, "azimuth": 180, "efficiency": 0.95}], + "battery": {"size_kwh": 9.5, "inverter_kw": 5.0, "export_limit_kw": 5.0, "hybrid": True}, + "load": {"annual_kwh": 3800, "shape": "flat", "car_charging_kwh": 0, "car_rate_kw": 7.4}, + "tariff": {"rates_import": [{"rate": 24.86}], "rates_export": [{"rate": 4.1}], "standing_charge_p_per_day": 60.0}, + "samples_per_month": 2, +} + +CONFIG_FILENAME = "annual.yaml" + + +class AnnualPage: + """Renders and drives the Annual prediction tab.""" + + def __init__(self, web_interface): + """Attach to the running web interface so args and Storage are reachable.""" + self.web = web_interface + self.base = web_interface.base + self.log = web_interface.log + + def _arg(self, name, default=None): + """Read one configuration value from the in-memory args dictionary. + + Never reads apps.yaml from disk: the file may not exist at all in some + deployments, which is exactly where the unconfigured case matters most. + """ + try: + return self.base.get_arg(name, default) + except Exception: + return default + + def _solar_from_args(self): + """Return the configured solar arrays, or an empty list. + + open_meteo_forecast and forecast_solar are already lists of + {kwp, declination, azimuth, efficiency}, which is the annual engine's own + shape, so no translation is needed. + """ + for name in ["open_meteo_forecast", "forecast_solar"]: + configured = self._arg(name, None) + if isinstance(configured, dict): + configured = [configured] + if isinstance(configured, list) and configured: + arrays = [] + for entry in configured: + if not isinstance(entry, dict) or not entry.get("kwp"): + continue + arrays.append( + { + "kwp": entry.get("kwp"), + "declination": entry.get("declination", DEFAULT_CONFIG["solar"][0]["declination"]), + "azimuth": entry.get("azimuth", DEFAULT_CONFIG["solar"][0]["azimuth"]), + "efficiency": entry.get("efficiency", DEFAULT_CONFIG["solar"][0]["efficiency"]), + } + ) + if arrays: + return arrays + return [] + + def _location_from_args(self): + """Return the configured location, taken from the solar entries if present.""" + for name in ["open_meteo_forecast", "forecast_solar"]: + configured = self._arg(name, None) + if isinstance(configured, dict): + configured = [configured] + for entry in configured or []: + if not isinstance(entry, dict): + continue + if entry.get("postcode"): + return {"postcode": entry["postcode"]} + if entry.get("latitude") is not None and entry.get("longitude") is not None: + return {"latitude": entry["latitude"], "longitude": entry["longitude"]} + return None + + def is_configured(self): + """Return True when the live instance has a battery or a solar array. + + Those two are what signal a configured system; with neither, the form + shows a banner saying the values on screen are examples. + """ + battery_kwh = self._arg("soc_max", 0.0) or 0 + try: + battery_kwh = float(battery_kwh) + except (TypeError, ValueError): + battery_kwh = 0 + return battery_kwh > 0 or bool(self._solar_from_args()) + + def prefill_config(self): + """Build a complete config from the live instance, filling gaps with the example. + + Every field falls back independently, so a half-configured Predbat gets its + real values alongside example ones rather than all-or-nothing. + """ + config = copy.deepcopy(DEFAULT_CONFIG) + + location = self._location_from_args() + if location: + config["location"] = location + + arrays = self._solar_from_args() + if arrays: + config["solar"] = arrays + + battery_kwh = self._arg("soc_max", 0.0) or 0 + try: + battery_kwh = float(battery_kwh) + except (TypeError, ValueError): + battery_kwh = 0 + # A zero or absent soc_max means it is not set - fall back so the user can adjust + if battery_kwh > 0: + config["battery"]["size_kwh"] = battery_kwh + + for arg_name, field, divisor in [("inverter_limit", "inverter_kw", 1000.0), ("export_limit", "export_limit_kw", 1000.0)]: + watts = self._arg(arg_name, 0.0) or 0 + try: + watts = float(watts) + except (TypeError, ValueError): + watts = 0 + if watts > 0: + config["battery"][field] = round(watts / divisor, 2) + + inverter_type = self._arg("inverter_type", None) + if inverter_type: + config["battery"]["hybrid"] = True + + import_url = self._arg("rates_import_octopus_url", None) + export_url = self._arg("rates_export_octopus_url", None) + if import_url: + config["tariff"] = {"import_octopus_url": import_url, "standing_charge_p_per_day": DEFAULT_CONFIG["tariff"]["standing_charge_p_per_day"]} + if export_url: + config["tariff"]["export_octopus_url"] = export_url + + dno_region = self._arg("dno_region", None) + if dno_region: + config["tariff"]["dno_region"] = dno_region + + return config + + def catalogue(self): + """Return the tariff dropdown entries: built-ins merged with the user's own.""" + return merged_catalogue(self._arg("compare_list", None)) + + def _config_path(self): + """Return the path of the saved annual configuration.""" + return os.path.join(self.base.config_root, CONFIG_FILENAME) + + def load_config(self): + """Return the saved configuration, or a fresh prefill when none exists.""" + path = self._config_path() + try: + if os.path.exists(path): + with open(path, "r", encoding="utf-8") as handle: + saved = yaml.safe_load(handle) + if isinstance(saved, dict) and saved: + return saved.get("annual", saved) + except (OSError, yaml.YAMLError) as error: + self.log("Warn: Annual: could not read {}: {}".format(path, error)) + return self.prefill_config() + + def save_config(self, config): + """Write the configuration so the CLI subprocess can consume it directly.""" + path = self._config_path() + try: + with open(path, "w", encoding="utf-8") as handle: + yaml.safe_dump({"annual": config}, handle, default_flow_style=False, allow_unicode=True) + except OSError as error: + self.log("Warn: Annual: could not write {}: {}".format(path, error)) + raise From cfcc5143eb9cffbd2f69895eadf86120379d8704 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:14:18 +0200 Subject: [PATCH 056/119] docs: use float defaults with get_arg in the prefill plan get_arg() type-directs its coercion off the default's type, so the plan's int defaults silently truncated real values - a 12.5 kWh soc_max came back as 12. Found by the Task 5 implementer when the brief's own test caught the truncation. Co-Authored-By: Claude Opus 5 (1M context) --- docs/superpowers/plans/2026-07-26-annual-web-ui.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-annual-web-ui.md b/docs/superpowers/plans/2026-07-26-annual-web-ui.md index c2306399e..84ddb9661 100644 --- a/docs/superpowers/plans/2026-07-26-annual-web-ui.md +++ b/docs/superpowers/plans/2026-07-26-annual-web-ui.md @@ -1538,6 +1538,10 @@ class AnnualPage: def _arg(self, name, default=None): """Read one configuration value from the in-memory args dictionary. + Pass a FLOAT default for any numeric field. get_arg() type-directs its + coercion off the default's type, so an int default silently truncates a + real float - soc_max 12.5 would come back as 12. + Never reads apps.yaml from disk: the file may not exist at all in some deployments, which is exactly where the unconfigured case matters most. """ @@ -1595,7 +1599,7 @@ class AnnualPage: Those two are what signal a configured system; with neither, the form shows a banner saying the values on screen are examples. """ - battery_kwh = self._arg("soc_max", 0) or 0 + battery_kwh = self._arg("soc_max", 0.0) or 0.0 try: battery_kwh = float(battery_kwh) except (TypeError, ValueError): @@ -1618,7 +1622,7 @@ class AnnualPage: if arrays: config["solar"] = arrays - battery_kwh = self._arg("soc_max", 0) or 0 + battery_kwh = self._arg("soc_max", 0.0) or 0.0 try: battery_kwh = float(battery_kwh) except (TypeError, ValueError): @@ -1628,7 +1632,7 @@ class AnnualPage: config["battery"]["size_kwh"] = battery_kwh for arg_name, field, divisor in [("inverter_limit", "inverter_kw", 1000.0), ("export_limit", "export_limit_kw", 1000.0)]: - watts = self._arg(arg_name, 0) or 0 + watts = self._arg(arg_name, 0.0) or 0.0 try: watts = float(watts) except (TypeError, ValueError): From a1bac4a70bbe09169baf340490111b1de5a813a1 Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:22:39 +0200 Subject: [PATCH 057/119] docs: pass indirect=False when prefilling tariff URLs resolve_arg treats any dotted string as a Home Assistant entity id, and a URL is full of dots, so the plan's generic _arg helper resolved every Octopus tariff URL to None. A configured user's tariff would never have prefilled, silently, and the form would have shown example price-cap rates instead. Every other call site passes indirect=False for these args; the plan now says why so it does not get simplified back. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-07-26-annual-web-ui.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-annual-web-ui.md b/docs/superpowers/plans/2026-07-26-annual-web-ui.md index 84ddb9661..b83997d7d 100644 --- a/docs/superpowers/plans/2026-07-26-annual-web-ui.md +++ b/docs/superpowers/plans/2026-07-26-annual-web-ui.md @@ -1535,9 +1535,13 @@ class AnnualPage: self.base = web_interface.base self.log = web_interface.log - def _arg(self, name, default=None): + def _arg(self, name, default=None, indirect=True): """Read one configuration value from the in-memory args dictionary. + Pass indirect=False for any value that may contain a dot but is NOT an entity + id - a URL, most obviously. resolve_arg would otherwise try to resolve it as a + Home Assistant entity and hand back None. + Pass a FLOAT default for any numeric field. get_arg() type-directs its coercion off the default's type, so an int default silently truncates a real float - soc_max 12.5 would come back as 12. @@ -1546,7 +1550,7 @@ class AnnualPage: deployments, which is exactly where the unconfigured case matters most. """ try: - return self.base.get_arg(name, default) + return self.base.get_arg(name, default, indirect=indirect) except Exception: return default @@ -1644,8 +1648,13 @@ class AnnualPage: if inverter_type: config["battery"]["hybrid"] = True - import_url = self._arg("rates_import_octopus_url", None) - export_url = self._arg("rates_export_octopus_url", None) + # indirect=False is REQUIRED here. resolve_arg (userinterface.py:149) treats any + # dotted string as a Home Assistant entity id and looks it up, and a URL is full + # of dots - so with the default indirect=True the URL resolves to None and the + # prefill silently does nothing. Every other call site in the codebase passes + # indirect=False for these two args; see compare.py:75 and fetch.py:848. + import_url = self._arg("rates_import_octopus_url", None, indirect=False) + export_url = self._arg("rates_export_octopus_url", None, indirect=False) if import_url: config["tariff"] = {"import_octopus_url": import_url, "standing_charge_p_per_day": DEFAULT_CONFIG["tariff"]["standing_charge_p_per_day"]} if export_url: From 66b555cec48d493db942649aaf6fd3df59319c4b Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Sun, 26 Jul 2026 22:27:15 +0200 Subject: [PATCH 058/119] fix(annual): read multi-inverter soc_max and Octopus tariff URLs correctly Two prefill bugs found in review: - get_arg()'s default indirect=True treats any dotted string as a Home Assistant entity id to resolve. An Octopus tariff URL is nothing but dots, so it silently resolved to None and a configured user's real tariff never survived prefill - the form showed the example price-cap rates instead. Fixed by passing indirect=False, matching every other Octopus URL read in the codebase (compare.py, fetch.py). - soc_max is a sensor_list in APPS_SCHEMA; a multi-inverter system holds a list there. Reading it with a float default made get_arg() raise internally on the list and silently fall back to 0.0, so a multi-inverter user's battery read as absent. Fixed with combine=True, which sums the per-inverter capacities before get_arg()'s own type coercion runs - verified directly against a live PredBat instance for list, scalar, absent and zero soc_max. Added tests for both paths, confirmed to fail against the pre-fix code via git stash, plus a test that prefill_config() never opens apps.yaml. --- apps/predbat/tests/test_web_annual.py | 45 +++++++++++++++++++++++++++ apps/predbat/web_annual.py | 32 +++++++++++++++---- 2 files changed, 71 insertions(+), 6 deletions(-) diff --git a/apps/predbat/tests/test_web_annual.py b/apps/predbat/tests/test_web_annual.py index 9baabf821..944b93c38 100644 --- a/apps/predbat/tests/test_web_annual.py +++ b/apps/predbat/tests/test_web_annual.py @@ -9,6 +9,9 @@ """Tests for the Annual tab's prefill and configuration handling.""" +import builtins +from unittest.mock import patch + from annual import validate_config from web import WebInterface from web_annual import DEFAULT_CONFIG, AnnualPage @@ -49,6 +52,17 @@ def test_web_annual(my_predbat): print(" ERROR: solar should fall back to the default array") failed = True + print("Test: prefill_config() never reads apps.yaml (or anything else) from disk") + # The unconfigured case above is exactly where this matters: apps.yaml may not + # exist at all. Tracks every open() call made during prefill_config() and asserts + # none of them targets apps.yaml, rather than relying on inspection alone. + with patch("builtins.open", wraps=builtins.open) as mock_open: + make_page(my_predbat).prefill_config() + opened_paths = [call.args[0] for call in mock_open.call_args_list] + if any(str(path).endswith("apps.yaml") for path in opened_paths): + print(" ERROR: prefill_config() must never read apps.yaml from disk, opened {}".format(opened_paths)) + failed = True + print("Test: a zero soc_max counts as unset and falls back to the default") my_predbat.args["soc_max"] = 0 config = make_page(my_predbat).prefill_config() @@ -56,6 +70,19 @@ def test_web_annual(my_predbat): print(" ERROR: a zero soc_max should fall back, got {}".format(config["battery"]["size_kwh"])) failed = True + print("Test: a multi-inverter soc_max list is summed rather than treated as absent") + # soc_max is a sensor_list in APPS_SCHEMA - a real multi-inverter system holds a + # list here, not a scalar, and the annual model wants one total usable capacity. + my_predbat.args["soc_max"] = [6.0, 6.5] + page = make_page(my_predbat) + config = page.prefill_config() + if config["battery"]["size_kwh"] != 12.5: + print(" ERROR: a multi-inverter soc_max should be summed to a total capacity, got {}".format(config["battery"]["size_kwh"])) + failed = True + if not page.is_configured(): + print(" ERROR: a multi-inverter battery should count as configured") + failed = True + print("Test: configured values are read from args and used") my_predbat.args["soc_max"] = 12.5 my_predbat.args["open_meteo_forecast"] = [{"kwp": 7.2, "declination": 30, "azimuth": 170, "efficiency": 0.9}] @@ -82,6 +109,24 @@ def test_web_annual(my_predbat): print(" ERROR: a configured solar array alone should count as configured") failed = True + print("Test: an Octopus tariff URL survives prefill (a dotted URL must not be read as an entity id)") + # get_arg()'s default indirect=True treats any dotted string as a Home Assistant + # entity id to resolve; a URL is full of dots, so this only passes if the URL + # fields are read with indirect=False. + import_url = "https://api.octopus.energy/v1/products/AGILE-24-10-01/electricity-tariffs/E-1R-AGILE-24-10-01-A/standard-unit-rates/" + export_url = "https://api.octopus.energy/v1/products/AGILE-OUTGOING-19-05-13/electricity-tariffs/E-1R-AGILE-OUTGOING-19-05-13-A/standard-unit-rates/" + my_predbat.args["rates_import_octopus_url"] = import_url + my_predbat.args["rates_export_octopus_url"] = export_url + config = make_page(my_predbat).prefill_config() + if config["tariff"].get("import_octopus_url") != import_url: + print(" ERROR: the Octopus import URL should survive into tariff.import_octopus_url, got {}".format(config["tariff"])) + failed = True + if config["tariff"].get("export_octopus_url") != export_url: + print(" ERROR: the Octopus export URL should survive into tariff.export_octopus_url, got {}".format(config["tariff"])) + failed = True + my_predbat.args.pop("rates_import_octopus_url", None) + my_predbat.args.pop("rates_export_octopus_url", None) + print("Test: the catalogue merges the user's compare_list") my_predbat.args["compare_list"] = [{"id": "mine", "name": "My tariff", "rates_import_octopus_url": "https://example.com/x"}] ids = [entry["id"] for entry in make_page(my_predbat).catalogue()] diff --git a/apps/predbat/web_annual.py b/apps/predbat/web_annual.py index 944100ae7..9c28c4f8f 100644 --- a/apps/predbat/web_annual.py +++ b/apps/predbat/web_annual.py @@ -44,14 +44,21 @@ def __init__(self, web_interface): self.base = web_interface.base self.log = web_interface.log - def _arg(self, name, default=None): + def _arg(self, name, default=None, indirect=True, combine=False): """Read one configuration value from the in-memory args dictionary. Never reads apps.yaml from disk: the file may not exist at all in some deployments, which is exactly where the unconfigured case matters most. + + ``indirect`` defaults to True to match ``get_arg()``'s own default, but a + caller reading a value that can itself contain a literal dot - a URL, most + obviously - must pass False. With indirect left True, ``resolve_arg()`` + (``userinterface.py``) treats any dotted string as a Home Assistant entity + id, fails to find one, and silently returns the default instead of the + real value - see the call sites below for the field this bit precisely. """ try: - return self.base.get_arg(name, default) + return self.base.get_arg(name, default, indirect=indirect, combine=combine) except Exception: return default @@ -104,7 +111,11 @@ def is_configured(self): Those two are what signal a configured system; with neither, the form shows a banner saying the values on screen are examples. """ - battery_kwh = self._arg("soc_max", 0.0) or 0 + # combine=True sums a multi-inverter soc_max (a sensor_list in APPS_SCHEMA) into one + # total usable capacity, which is what the annual model wants. Without it, get_arg()'s + # float-default coercion raises on a list, is caught internally, and silently returns + # the default - a multi-inverter system would otherwise read as unconfigured. + battery_kwh = self._arg("soc_max", 0.0, combine=True) or 0 try: battery_kwh = float(battery_kwh) except (TypeError, ValueError): @@ -127,7 +138,11 @@ def prefill_config(self): if arrays: config["solar"] = arrays - battery_kwh = self._arg("soc_max", 0.0) or 0 + # combine=True sums a multi-inverter soc_max (a sensor_list in APPS_SCHEMA) into one + # total usable capacity, which is what the annual model wants. Without it, get_arg()'s + # float-default coercion raises on a list, is caught internally, and silently returns + # the default - a multi-inverter system would otherwise read as unconfigured. + battery_kwh = self._arg("soc_max", 0.0, combine=True) or 0 try: battery_kwh = float(battery_kwh) except (TypeError, ValueError): @@ -149,8 +164,13 @@ def prefill_config(self): if inverter_type: config["battery"]["hybrid"] = True - import_url = self._arg("rates_import_octopus_url", None) - export_url = self._arg("rates_export_octopus_url", None) + # indirect=False: these values are URLs, full of literal dots, which get_arg()'s + # default indirect=True would otherwise treat as a Home Assistant entity id to look + # up - failing to find one and silently returning None instead of the real URL. See + # compare.py's and fetch.py's own Octopus URL reads for the same requirement. Do not + # "simplify" this back to the _arg() default. + import_url = self._arg("rates_import_octopus_url", None, indirect=False) + export_url = self._arg("rates_export_octopus_url", None, indirect=False) if import_url: config["tariff"] = {"import_octopus_url": import_url, "standing_charge_p_per_day": DEFAULT_CONFIG["tariff"]["standing_charge_p_per_day"]} if export_url: From 37f91173a459d657635ade52037838b42da6b41f Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Mon, 27 Jul 2026 08:49:01 +0200 Subject: [PATCH 059/119] feat(annual): render the Annual tab configuration form --- .cspell/custom-dictionary-workspace.txt | 1 + apps/predbat/tests/test_web_annual.py | 125 +++++++++++++++++ apps/predbat/unit_test.py | 3 +- apps/predbat/web_annual.py | 179 ++++++++++++++++++++++++ 4 files changed, 307 insertions(+), 1 deletion(-) diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index ad8ddbf49..d1d48692b 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -12,6 +12,7 @@ aiomqtt aios Alertfeed allclose +annualform Anson apexcharts apikey diff --git a/apps/predbat/tests/test_web_annual.py b/apps/predbat/tests/test_web_annual.py index 944b93c38..6c6f57004 100644 --- a/apps/predbat/tests/test_web_annual.py +++ b/apps/predbat/tests/test_web_annual.py @@ -13,6 +13,7 @@ from unittest.mock import patch from annual import validate_config +from tariff_catalogue import CUSTOM_ID from web import WebInterface from web_annual import DEFAULT_CONFIG, AnnualPage @@ -149,3 +150,127 @@ def test_web_annual(my_predbat): my_predbat.args.update(saved_args) return failed + + +def test_web_annual_form(my_predbat): + """Verify the form renders every group, reflects config, and round-trips a post.""" + failed = False + print("**** Testing web_annual form ****") + + saved_args = dict(my_predbat.args) + try: + # compare_list is also cleared: the coverage/apps.yaml test fixture ships an + # active demo compare_list whose entries share ids with several built-ins (see + # test_web_annual's own "catalogue merges the user's compare_list" case below) + # and, by the documented "a user entry replaces a built-in of the same id" rule, + # would otherwise silently substitute their names for the ones this test checks. + for key in ["soc_max", "open_meteo_forecast", "forecast_solar", "compare_list"]: + my_predbat.args.pop(key, None) + page = make_page(my_predbat) + config = page.prefill_config() + html = page.render_form(config) + + print("Test: every configuration group is present") + for heading in ["Location", "Solar", "Battery", "Load", "Tariff", "Advanced"]: + if heading not in html: + print(" ERROR: the form is missing the '{}' group".format(heading)) + failed = True + + print("Test: an unconfigured instance gets the example-values banner") + if "example values" not in html.lower(): + print(" ERROR: an unconfigured instance should be told these are examples") + failed = True + + print("Test: a configured instance does NOT get the banner") + my_predbat.args["soc_max"] = 10.0 + configured_html = make_page(my_predbat).render_form(make_page(my_predbat).prefill_config()) + if "example values" in configured_html.lower(): + print(" ERROR: a configured instance should not be told its values are examples") + failed = True + my_predbat.args.pop("soc_max", None) + + print("Test: the tariff dropdown lists the catalogue and a Custom entry") + if CUSTOM_ID not in html: + print(" ERROR: the dropdown should offer a Custom entry") + failed = True + if "Agile import / Agile export" not in html: + print(" ERROR: the dropdown should list the built-in tariffs") + failed = True + + print("Test: the load source is a radio pair, not two independent sections") + if html.count('type="radio"') < 2: + print(" ERROR: expected a radio pair for the load source") + failed = True + if "octopus" not in html.lower(): + print(" ERROR: the Octopus load option should be offered") + failed = True + + print("Test: current values are rendered into the inputs") + if 'value="3800"' not in html.replace("'", '"'): + print(" ERROR: the annual kWh value should appear in the form") + failed = True + + print("Test: validation errors are shown with the form still populated") + html_with_error = page.render_form(config, errors="annual.solar[0] is missing kwp") + if "annual.solar[0] is missing kwp" not in html_with_error: + print(" ERROR: the error message should be displayed") + failed = True + if 'value="3800"' not in html_with_error.replace("'", '"'): + print(" ERROR: the form should stay populated when an error is shown") + failed = True + + print("Test: config_from_post rebuilds a config the engine accepts") + postdata = { + "postcode": "SW1A 1AA", + "solar_kwp_0": "5.6", + "solar_declination_0": "35", + "solar_azimuth_0": "180", + "solar_efficiency_0": "0.95", + "battery_size_kwh": "9.5", + "battery_inverter_kw": "5.0", + "battery_export_limit_kw": "5.0", + "battery_hybrid": "on", + "load_source": "manual", + "load_annual_kwh": "3800", + "load_shape": "flat", + "load_car_charging_kwh": "2500", + "load_car_rate_kw": "7.4", + "tariff_id": CUSTOM_ID, + "tariff_import_url": "https://example.com/import/", + "tariff_export_url": "https://example.com/export/", + "tariff_standing_charge": "60.0", + "samples_per_month": "2", + } + rebuilt = page.config_from_post(postdata) + try: + # config_from_post deliberately leaves posted values as the strings the + # browser sent - validate_config() is the one place that coerces and range + # checks them - so the round trip is only meaningful once validated. + validated = validate_config(rebuilt) + except Exception as error: + print(" ERROR: a posted form should rebuild into a valid config, got {}".format(error)) + failed = True + validated = None + if validated is not None and validated.get("load", {}).get("car_charging_kwh") != 2500: + print(" ERROR: car charging should survive the round trip, got {}".format(rebuilt["load"])) + failed = True + + print("Test: choosing the Octopus load source drops the manual figures") + postdata["load_source"] = "octopus" + postdata["load_octopus_api_key"] = "sk_test" + postdata["load_octopus_account_id"] = "A-1234ABCD" + rebuilt = page.config_from_post(postdata) + if "annual_kwh" in rebuilt["load"] or "car_charging_kwh" in rebuilt["load"]: + print(" ERROR: the manual figures must not be sent alongside Octopus, got {}".format(rebuilt["load"])) + failed = True + try: + validate_config(rebuilt) + except Exception as error: + print(" ERROR: the Octopus form should rebuild into a valid config, got {}".format(error)) + failed = True + + finally: + my_predbat.args.clear() + my_predbat.args.update(saved_args) + + return failed diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 505503734..4c7a01526 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -70,7 +70,7 @@ from tests.test_web_charts import run_web_charts_tests from tests.test_web_chart_grouping import run_web_chart_grouping_tests from tests.test_web_entity_unit_resolution import run_web_entity_unit_resolution_tests -from tests.test_web_annual import test_web_annual +from tests.test_web_annual import test_web_annual, test_web_annual_form from tests.test_window import run_window_sort_tests, run_intersect_window_tests from tests.test_find_charge_rate import test_find_charge_rate, test_find_charge_rate_string_temperature, test_find_charge_rate_string_charge_curve from tests.test_manual_api import run_test_manual_api @@ -299,6 +299,7 @@ def main(): ("web_if", run_test_web_if, "Web interface tests", False), ("web_functions", run_web_functions_tests, "Web function unit tests", False), ("web_annual", test_web_annual, "Annual web tab prefill tests", False), + ("web_annual_form", test_web_annual_form, "Annual web tab form tests", False), ("web_history_table", run_web_history_table_tests, "Web /entity history table bucketing tests", False), ("web_charts", run_web_charts_tests, "Web chart rendering tests (percent/special-character units)", False), ("web_chart_grouping", run_web_chart_grouping_tests, "Web /entity chart numeric vs timeline grouping tests", False), diff --git a/apps/predbat/web_annual.py b/apps/predbat/web_annual.py index 9c28c4f8f..be4efe18d 100644 --- a/apps/predbat/web_annual.py +++ b/apps/predbat/web_annual.py @@ -212,3 +212,182 @@ def save_config(self, config): except OSError as error: self.log("Warn: Annual: could not write {}: {}".format(path, error)) raise + + def _number_field(self, name, label, value, step="any", suffix=""): + """Return one labelled numeric input row.""" + return '
{suffix}
\n'.format( + name=name, label=label, step=step, value=value if value is not None else "", suffix=" {}".format(suffix) if suffix else "" + ) + + def _text_field(self, name, label, value): + """Return one labelled text input row.""" + return '
\n'.format(name=name, label=label, value=value if value is not None else "") + + def render_form(self, config, errors=None): + """Return the configuration form as HTML, populated from ``config``. + + ``errors`` is displayed above the form with every field left as the user + entered it - losing their input on a validation failure would be worse + than the failure. + """ + solar = config.get("solar") or [{}] + battery = config.get("battery") or {} + load = config.get("load") or {} + tariff = config.get("tariff") or {} + location = config.get("location") or {} + + text = '
\n' + + if errors: + text += '
Could not run: {}
\n'.format(errors) + + if not self.is_configured(): + text += '
Predbat isn\'t configured yet — these are example values, edit them to match your home.
\n' + + text += '
\n' + + text += "
Location\n" + text += self._text_field("postcode", "Postcode", location.get("postcode", "")) + text += self._number_field("latitude", "Latitude (instead of postcode)", location.get("latitude")) + text += self._number_field("longitude", "Longitude", location.get("longitude")) + text += "
\n" + + text += "
Solar\n" + for index, array in enumerate(solar): + text += '
Array {}\n'.format(index + 1) + text += self._number_field("solar_kwp_{}".format(index), "Peak power", array.get("kwp"), suffix="kWp") + text += self._number_field("solar_declination_{}".format(index), "Pitch", array.get("declination", 35), suffix="degrees") + text += self._number_field("solar_azimuth_{}".format(index), "Azimuth (180 = south)", array.get("azimuth", 180), suffix="degrees") + text += "
\n" + text += "
\n" + + text += "
Battery\n" + text += self._number_field("battery_size_kwh", "Usable capacity", battery.get("size_kwh"), suffix="kWh") + text += self._number_field("battery_inverter_kw", "Inverter size", battery.get("inverter_kw"), suffix="kW") + text += self._number_field("battery_export_limit_kw", "Export limit", battery.get("export_limit_kw"), suffix="kW") + text += '
\n'.format("checked" if battery.get("hybrid", True) else "") + text += "
\n" + + using_octopus = "octopus" in load + text += "
Load\n" + text += '
\n'.format("" if using_octopus else "checked") + text += '
\n' + text += self._number_field("load_annual_kwh", "Annual consumption", load.get("annual_kwh", DEFAULT_CONFIG["load"]["annual_kwh"]), suffix="kWh") + shape = load.get("shape", "flat") + text += '
\n" + text += self._number_field("load_car_charging_kwh", "Car charging per year (0 for none)", load.get("car_charging_kwh", 0), suffix="kWh") + text += self._number_field("load_car_rate_kw", "Charger power", load.get("car_rate_kw", 7.4), suffix="kW") + text += "
\n" + text += '
\n'.format("checked" if using_octopus else "") + text += '
\n' + text += self._text_field("load_octopus_api_key", "Octopus API key", (load.get("octopus") or {}).get("api_key", "")) + text += self._text_field("load_octopus_account_id", "Account ID", (load.get("octopus") or {}).get("account_id", "")) + text += '

Your meter readings already include any car charging, so the figures above are not used with this option.

\n' + text += "
\n" + text += "
\n" + + text += "
Tariff\n" + text += '
\n" + text += self._text_field("tariff_import_url", "Import rates URL", tariff.get("import_octopus_url", "")) + text += self._text_field("tariff_export_url", "Export rates URL", tariff.get("export_octopus_url", "")) + text += self._text_field("tariff_dno_region", "Octopus region letter", tariff.get("dno_region", "")) + text += self._number_field("tariff_standing_charge", "Standing charge", tariff.get("standing_charge_p_per_day", 60.0), suffix="p/day") + text += "
\n" + + text += "
Advanced\n" + text += self._number_field("year", "Year to model (blank for the most recent complete year)", config.get("year")) + text += self._number_field("samples_per_month", "Days sampled per month", config.get("samples_per_month", 2), step="1") + text += self._number_field("pv10_derate_fallback", "P10 fallback derate", config.get("pv10_derate_fallback", 0.7)) + for index, array in enumerate(solar): + text += self._number_field("solar_efficiency_{}".format(index), "Array {} efficiency".format(index + 1), array.get("efficiency", 0.95)) + text += "
\n" + + text += '\n' + text += "
\n
\n" + return text + + def config_from_post(self, postdata): + """Rebuild a config dict from submitted form fields. + + Values are left as the strings the browser sent; validate_config() in the + engine does the coercion and range checking, so there is exactly one place + that decides what a valid number is. + """ + + def value(name, default=None): + """Return one posted field, or the default when absent or blank.""" + raw = postdata.get(name) + if raw is None or str(raw).strip() == "": + return default + return str(raw).strip() + + config = {} + + location = {} + if value("postcode"): + location["postcode"] = value("postcode") + if value("latitude") is not None and value("longitude") is not None: + location["latitude"] = value("latitude") + location["longitude"] = value("longitude") + config["location"] = location + + arrays = [] + index = 0 + while value("solar_kwp_{}".format(index)) is not None: + arrays.append( + { + "kwp": value("solar_kwp_{}".format(index)), + "declination": value("solar_declination_{}".format(index), 35), + "azimuth": value("solar_azimuth_{}".format(index), 180), + "efficiency": value("solar_efficiency_{}".format(index), 0.95), + } + ) + index += 1 + if arrays: + config["solar"] = arrays + + if value("battery_size_kwh") is not None: + config["battery"] = { + "size_kwh": value("battery_size_kwh"), + "inverter_kw": value("battery_inverter_kw", 5.0), + "export_limit_kw": value("battery_export_limit_kw", 5.0), + "hybrid": bool(postdata.get("battery_hybrid")), + } + + # The engine rejects an Octopus block alongside manual figures, because the + # meter series already contains any car charging. Send one or the other. + if value("load_source", "manual") == "octopus": + config["load"] = {"octopus": {"api_key": value("load_octopus_api_key", ""), "account_id": value("load_octopus_account_id", "")}} + else: + config["load"] = { + "annual_kwh": value("load_annual_kwh", 3800), + "shape": value("load_shape", "flat"), + "car_charging_kwh": value("load_car_charging_kwh", 0), + "car_rate_kw": value("load_car_rate_kw", 7.4), + } + + tariff = {"standing_charge_p_per_day": value("tariff_standing_charge", 0)} + if value("tariff_import_url"): + tariff["import_octopus_url"] = value("tariff_import_url") + if value("tariff_export_url"): + tariff["export_octopus_url"] = value("tariff_export_url") + if value("tariff_dno_region"): + tariff["dno_region"] = value("tariff_dno_region") + if not tariff.get("import_octopus_url"): + tariff["rates_import"] = DEFAULT_CONFIG["tariff"]["rates_import"] + tariff["rates_export"] = DEFAULT_CONFIG["tariff"]["rates_export"] + config["tariff"] = tariff + + if value("year"): + config["year"] = value("year") + config["samples_per_month"] = value("samples_per_month", 2) + if value("pv10_derate_fallback"): + config["pv10_derate_fallback"] = value("pv10_derate_fallback") + + return config From 96c38a6f41d4d3db844014a790efbe3be96d1bdd Mon Sep 17 00:00:00 2001 From: Trefor Southwell Date: Mon, 27 Jul 2026 08:56:52 +0200 Subject: [PATCH 060/119] fix(annual): HTML-escape user-controlled form values and preserve tariff selection Postcode, Octopus credentials, tariff URLs/DNO region, and catalogue entries (a user's own compare_list supplies id/name too) were interpolated into HTML attributes unescaped, so a stray quote or angle bracket could break out of value="..." and mangle the form. Also mark the tariff