feat(ilamb): use obs4REF reference data for standard diagnostics#779
Open
lewisjared wants to merge 6 commits into
Open
feat(ilamb): use obs4REF reference data for standard diagnostics#779lewisjared wants to merge 6 commits into
lewisjared wants to merge 6 commits into
Conversation
Point ten ILAMB standard diagnostics at their obs4REF reference files instead of the legacy ilamb/iomb registry paths: gpp-WECANN, gpp-FLUXNET2015, mrro-LORA, cSoil-HWSD2, nbp-Hoffman, snc-ESACCI and burntFractionAll-GFED on land, plus thetao/so-WOA2023-surface and amoc-RAPID in the ocean suite. burntFractionAll-GFED now reads the CMIP burntFractionAll variable directly rather than the legacy burntArea alternate, and amoc-RAPID derives the AMOC index from the RAPID-2023-1a msftmz field through the existing msftmz_to_rapid transform. Bumps ILAMBStandard.version to 3: the reference change alters diagnostic outputs, so the ILAMB regression baselines must be re-minted.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…er diagnostic Replace the bare obs4REF registry paths with the ingested obs4MIPs data requirement form, pinning each reference to a single file by source_id + variable_id + grid_label + version. This routes the reference through the obs4MIPs catalogue as a first-class dataset rather than a raw registry key. Loosen the obs4REF key parser to accept monthly-climatology (monC) frequencies and single-year fixed-field (fx) time ranges so WOA-23 and HWSD-2-0 files resolve. Give each rewired diagnostic its own version (3) via a per-diagnostic ``version`` config key, so changing one diagnostic's reference data no longer forces every ILAMB baseline to be re-minted; the untouched diagnostics stay at version 2. The esgf-catalog test fixture and solve-regression snapshots are regenerated to include the newly ingested obs4REF reference datasets.
obs4REF is the REF-specific observational product and follows the same metadata conventions as obs4MIPs, but its files carry ``activity_id = "obs4REF"``. The obs4MIPs adapter rejected them, so obs4REF references could not be ingested and any diagnostic that requests an obs4REF dataset as an obs4MIPs source solved to nothing. Accept ``activity_id`` of either ``obs4MIPs`` or ``obs4REF`` so these references ingest as first-class datasets.
…urce_id The FLUXNET2015 obs4REF file spells its source_id three different ways across the registry path (FLUXNET2015-1-0), the filename (Fluxnet-2015-1-0) and the internal attribute (Fluxnet-2015), so the ingested facet form cannot match both the fetch and solve stages. Reference it by exact registry key until the upstream metadata is reconciled.
Add the nine ingested obs4REF reference datasets to the obs4MIPs test catalogue and regenerate the ILAMB solve-regression snapshots so the rewired diagnostics resolve their reference through the obs4MIPs requirement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Points ten ILAMB standard diagnostics at their newly published obs4REF reference files instead of the legacy
ilamb/iombregistry paths. Stacked on #778, which registers and uploads the underlying obs4REF data — merge that first.Diagnostics rewired
Land (
ilamb.yaml):gpp-WECANN(WECANN-1-0 v20250902),gpp-FLUXNET2015(Fluxnet-2015-1-0),mrro-LORA(LORA-1-0),cSoil-HWSD2(HWSD-2-0),nbp-Hoffman(Hoffman-1-0),snc-ESACCI(CCI-CryoClim-FSC-1),burntFractionAll-GFED(GFED-5-0). Ocean (iomb.yaml):thetao-WOA2023-surfaceandso-WOA2023-surface(WOA-23),amoc-RAPID(RAPID-2023-1a).The bare-path form is used throughout because the obs4REF key parser (
_parse_obs4ref_key) cannot yet parse themonCfrequency (WOA-23 climatologies) or the single-yearfxtime range (HWSDcSoil), so the ingestedobs_source: obs4refdict form is not an option for those. Bare paths resolve directly against the concatenated obs4REF registry at execute time, matching the existing GPCP reference inemp-GLEAMGPCP2.3.Two edits that are more than a path swap
burntFractionAll-GFEDnow reads the CMIPburntFractionAllvariable directly and drops theburntAreaalternate, because the obs4REF file carriesburntFractionAll.amoc-RAPIDswitches its reference to the raw RAPID-2023-1amsftmzfield and relies on the existingmsftmz_to_rapidtransform to derive the AMOC index, replacing the pre-derived legacyamocfile.Versioning and baselines
ILAMBStandard.versionis bumped 2 -> 3. This attribute is shared by every ILAMB diagnostic, so all ILAMB regression baselines (not only the ten rewired here) must be re-minted before the coupling gate passes.Validation done
All obs4REF source keys resolve against the registry, all diagnostics construct, the ILAMB unit suite passes (28 passed), and ruff/mypy are clean. Not yet validated: an end-to-end run against the real reference data.
amoc-RAPIDin particular is unverified — it carries both the transform-semantics question above and the known upstream ilamb3trim_timecrash, and needs a real run to confirm. WOA-23 is a 2005-2022 monthly climatology replacing a 2005-2014 monthly field, which interacts with the surface-variable time-coverage handling inexecute().