fix(weather): era-complete GOES coverage — filter, clamps, fallback, scope keys, size cap#101
Merged
Merged
Conversation
Observed live (station backfill 28734982038): KNYC 2026 completed
ALL-EMPTY because the fleet requests goes16 only, and GOES-16 stopped
being GOES-East in 2025 (goes19 took over). The West side has the mirror
hole: goes18 only reaches back to 2022-07, so West stations were missing
the 2018-2022 goes17 era. And the roster defaults ("goes16","goes18")
carried a latent fleet-cost pathology: _enumerate_slices was a plain
cartesian product, so every East station would fetch the ENTIRE West
archive (and vice versa) — whole-month downloads whose extraction fails
out-of-grid only AFTER the bytes are fetched.
- _sources: add goes_satellites_for_longitude() — East {goes16,goes19} /
West {goes17,goes18}, keyed on the SAME _GOES_EAST_WEST_SPLIT_LON_DEG
as the live router (single source of truth; boundary test pins the
agreement).
- _backfill._enumerate_slices: GOES hemisphere visibility filter — a GOES
satellite enumerates for a station only on its own side of the split.
Skips are logged loudly (once per satellite), never silent. Non-GOES
sources are untouched (they own their coverage).
- __main__._DEFAULT_ROSTER_SATELLITES -> (goes16, goes17, goes18, goes19):
era-complete; the filter keeps each station on its side pair and the
available_since clamp skips pre-first-light slices with no I/O.
- run-weather-backfill.yml station+pilot modes pass the full four
(NOTE: requires an image built after this commit — an old image would
cartesian-product them).
Transport already supported all four platforms (buckets + available_since
for goes17/goes19 were registered); only the request layer was era-blind.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P2 on the era-coverage branch: goes_satellites_for_longitude returns every platform that EVER served a side, and the only downstream clamp was _AVAILABLE_SINCE — so retired platforms (goes17 post-2023 storage, goes16 post-2025 handover) kept enumerating post-retirement months and burning empty per-hour listing calls, growing every future fleet year. - _goes_s3._AVAILABLE_UNTIL: the symmetric counterpart of _AVAILABLE_SINCE, RETIRED platforms only (goes17 -> 2024-01-01, goes16 -> 2026-01-01). SAFETY over savings: ~1 yr margin past the public retirement dates (a too-early date silently skips real settlement-adjacent data; a too-late one only wastes cheap listings), and the goes16 date is observation-backed (2026 Jan-Jun measured ALL-EMPTY live). goes18/goes19 deliberately absent. - _enumerate_slices: month-granular clamp at ENUMERATION — clamped slices are never submitted, write NO progress marker (raising/removing a date later backfills them normally), and are logged loudly per satellite. - Tests: 6-case TestGoesOperationalUntilClamp (per-era keep/drop, operational platforms never clamped, month-granular partial window); hemisphere tests re-anchored to 2022 (in-era for all four platforms); the two future-slice resume tests switched goes16->goes19 so the 2099 window still genuinely exercises resume semantics instead of passing vacuously under the clamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable concurrent multi-station measurement runs (KNYC + Asia + Europe): - Progress marker objects are now keyed on the RESOLVED (stations, year-window) scope instead of the Batch shard index. Shard-index keys COLLIDED across concurrent jobs: two station-mode jobs both wrote progress/shard-00000/progress.json — a concurrent read-modify-write clobber on the same GCS object. The scope key is stable across Spot retries (the shard->(station,year) mapping is pure), disjoint across array tasks AND across concurrent different-station jobs, and resume-stable across re-submissions. Whole-roster scopes (>3 stations, e.g. the incremental Cloud Run Job) use a bounded stable digest. Old shard-NNNNN objects are orphaned (one idempotent re-fetch; slice keys inside the files are full-identity so no wrong-skip either way). - run-weather-backfill.yml: new `satellites` input for pilot/station modes (default: the era-complete GOES set). Enables Asia (himawari8,himawari9 + AHI-L2-FLDK-Clouds) and Europe (viirs-* + VIIRS-JRR-CloudMask) station measurements; Meteosat backfill stays unwired (GRIB reader seam) per the _sources registry. Tests: scope-key disjointness across stations/year-shards, multi-station digest bounding, and the C4 wiring test updated to the scope scheme. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P1 on the hemisphere filter: treating the East/West routing split as a coverage boundary silently lost the West side's 2017-2018 era. The sector geometry is ASYMMETRIC: GOES-West PACUS truly cannot see East stations, but GOES-East CONUS/full-disk genuinely covers the whole Americas (oblique at the far West) — and for 2017 -> 2018-08-27 (goes17 first light) goes16 was the ONLY ABI platform in existence, so a pure hemisphere filter left KSEA-class stations with zero 2017-2018 coverage that the previous defaults DID fetch. _enumerate_slices now applies direction-correct rules: - West platforms x East stations: still excluded (true out-of-grid). - East platforms x West stations: month-granular era FALLBACK — kept ONLY for months no REQUESTED native West platform covers (per the same since/until month semantics as the child-side clamp). August 2018 counts as goes17-covered (last day >= first light), so the fallback is Jan-Jul 2018 + all of 2017. - Explicit oblique requests (e.g. --satellites goes16 --stations KSEA, no native platform requested) are honored IN FULL — never silently narrowed. All three skip classes (out-of-grid exclusion, retirement clamp, fallback clip) log loudly. Tests: 6-case TestGoesEastForWestFallback (2017 kept, 2019 dropped, month-granular 2018 boundary, explicit-request honored, East exclusion unchanged, 2024 native-gap handling). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…te scope keys Adversarial exit panel (18 agents, findings verified by EXECUTING the enumeration code) caught 4 blocking defects in the fallback/scope work: - P1: _covers() counted a month as native-covered from a SINGLE day of data. goes17's Aug 27-31 2018 first-light tail clipped the goes16 fallback for all of August while the child-side clamp fetched only the tail -> Aug 1-26 2018 fetched by NO platform for all 8 West roster stations, on the DEFAULT request (and Jul 1-27 2022 under the legacy goes16,goes18 pair). The old boundary test enshrined the hole. Positive coverage now requires the FULL month (first day >= available_since); partial first-light months keep the East fallback (~5 duplicate days in separate per-satellite partitions — the safe direction). - P2: _covers() consumed the MARGINED _AVAILABLE_UNTIL as a positive coverage claim, inverting the margin's "too-late only wastes" safety contract (goes17 "covered" Feb-Dec 2023 with zero real data -> 11-month hole for goes16,goes17 requests; margin widening would silently widen the suppression window). New _GOES_ACTUAL_DATA_END map (un-margined real last-data dates, verified against the live archive) is used for claims; the margined map stays fetch-side only. - P2: the scope-keyed progress object omitted the products/satellites axes -> concurrent ACMC vs DSRF jobs for the same station+window still clobbered one GCS object (full-map PUTs, last-write-wins). The scope now appends a SORTED satellites#products digest (order-insensitive). - P2: the stale-image danger (all-four satellites x pre-filter image = cartesian wrong-hemisphere downloads) was documented only in shell comments invisible at dispatch; it is now in the image_tag and satellites INPUT descriptions the operator actually sees. Regressions added for every scenario the panel reproduced: Aug-2018 partial-month keep, Jul-2022 legacy-pair keep, 2023 margined-until non-claim, default-2023 goes18 clip, and scope disjointness across products/satellites with order-insensitivity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live-fleet finding (2026-07-05, KNYC station run, tasks 2022/2023): every modern ABI-L2-ACMC file was rejected "before read (P2-d resource limit)" — listed sizes 4.3-4.6 MB vs the 2 MiB default cap. The cap was calibrated on 2017-era 2-variable files (~0.3-1.5 MB); NOAA later added ACM + Cloud_Probabilities to the product, roughly tripling file sizes, so the guard zeroed ACMC coverage for the modern era (2021+) fleet-wide. Fail-safe held (per-file errors block completion markers — no wrong "done" state), but the affected year-tasks burned full listing time producing zero ACMC rows. Third instance of the same failure family today: a guard hardcoded from old-era data (DSRF grid, goes16-only satellites, now file sizes). Default cap -> 16 MiB (>3x headroom over observed modern sizes, still bounds the in-memory BytesIO well below DSRF's 64 MiB). Regression test pins: modern observed size admitted with margin, cap still < DSRF cap, DSRF cap unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
📝 Docs-required check: REMINDER API-surface change without docs and no opt-out — surfacing reminder. API-surface files changed: Docs files changed: Docs surfaces to consider
How to silence this reminder
This check is advisory — it never blocks the merge. |
|
Parity ticket gate: PASSED See |
…h sides Codex P2 (final stamp): the West-for-East exclusion was PRODUCT-blind. PACUS/CONUS-sector products truly cannot see East stations, but FULL-DISK products (ABI-L2-DSRF, 5424x5424 registry grid) see most of the Americas from either GOES slot — an explicit goes18+DSRF+KNYC request was silently dropped to zero slices, with the exclusion log falsely claiming "the PACUS sector cannot see them". _enumerate_slices West-for-East branch is now product-aware: - sector product -> always excluded (unchanged; true out-of-grid); - full-disk + a requested native covering the station -> skipped as redundant (default cost control), logged distinctly; - full-disk WITHOUT any native in the request (explicit cross-side study) -> honored in full. Full-disk detection reads grid_shape_expected == (5424, 5424) from the PRODUCTS registry (no hardcoded product list). Tests: explicit cross-side DSRF honored, sector product still excluded, default all-four still skips cross-side DSRF, mixed-products explicit request keeps only the full-disk slices. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k rule Codex P2 (re-stamp): the cross-side full-disk redundancy check was REQUEST-level (any native present in the request suppressed the cross-side platform outright), while the mirror-image East-for-West fallback was already month-granular. goes16,goes18 + ABI-L2-DSRF + KNYC in 2026: goes16 is era-clamped (no 2026 data), yet its mere presence in the request dropped goes18 — zero valid slices remained. Cross-side full-disk platforms now fall through to the SAME month-granular fallback machinery as East-for-West (_native_fully_covers against the requested natives): kept exactly for months no requested native fully covers, honored in full when no native is requested. Sector products remain always-excluded cross-side (true out-of-grid). One unified skip bucket + generalized log wording (both directions). Tests: era-clamped-native 2026 scenario keeps goes18 (12 months) and drops goes16; 2023 control still clips goes18 (goes16 fully covers). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex P2 (round 8): the progress scope key was built from RAW CLI station tokens, but bulk_backfill resolves NWS aliases to canonical ICAOs (NYC -> KNYC) via _resolve_station_infos — so an alias submission and a canonical submission of the SAME work got different progress objects: resume never rehydrated markers across the pair, and concurrent alias/canonical dispatches double-processed partitions. The scope now resolves stations to canonical ICAOs, dedupes, and sorts (order-insensitive) before keying. Also collapses the SIM102 nested-if in the cross-side branch (lint slipped through the previous chained run; behavior identical, verified by the full filter-family suite). Test: alias (NYC) and canonical (KNYC) submissions share one scope URI; multi-station order-insensitivity pinned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The KNYC measurement run exposed that the fleet's satellite request layer was era-blind, and stress-testing the fix surfaced a family of related defects. Observed live: KNYC 2026 backfilled ALL-EMPTY (
--satellites goes16; GOES-19 became GOES-East in 2025), and modern ACMC files were 100% rejected by a stale size cap.What (6 commits)
goes16,goes17,goes18,goes19;_enumerate_slicesgains a GOES coverage filter so the cartesian product no longer fetches archives that can't serve a station._AVAILABLE_UNTILretirement clamp (goes17→2024-01-01, goes16→2026-01-01; ~1yr margins, month-granular, no progress markers written for clamped slices — reversible).{stations}_{years}_{satsproducts-digest}instead of Batch shard index) — concurrent station-mode jobs (multi-station measurement) no longer clobber each other's GCS markers. Plus asatellitesworkflow input enabling Asia (Himawari) / Europe (VIIRS) station runs._GOES_ACTUAL_DATA_ENDmap (archive-verified), keeping margins fetch-side only.Review record (loop iterations that CAUGHT things)
Operator notes
shard-NNNNNprogress objects are orphaned (one idempotent re-fetch; slice keys are full-identity so no wrong-skips either way).Tests
29 new/updated tests: era-set helper + boundary agreement with the live router; hemisphere/coverage filter (East + West + mixed); until-clamp (6 cases incl. month-granular partial windows); East-for-West fallback (8 cases incl. Aug-2018/Jul-2022 partial-month keeps, margined-until non-claim, explicit-request honored); scope-key disjointness across stations/years/products/satellites + order-insensitivity + multi-station digest; modern-ACMC size admission. Full satellite suite green.
TS Parity
python_only: true — internal satellite backfill enumeration/ops; no
mostlyright.*public API change; the TS SDK has no satellite ingest.🤖 Generated with Claude Code