Add VPU-delineated metadata support for multi-geopackage runs#90
Open
pratikshac15 wants to merge 2 commits into
Open
Add VPU-delineated metadata support for multi-geopackage runs#90pratikshac15 wants to merge 2 commits into
pratikshac15 wants to merge 2 commits into
Conversation
Contributor
Author
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds VPU-delineated metadata outputs to better support multi-GeoPackage / multi-weight forcing runs (Issue #9), while preserving the existing global metadata outputs.
Changes:
- Adds VPU inference and VPU-level metadata writing (
metadata_by_vpu.csvplus VPU-tagged catchment stats). - Improves handling of local NWM file size tracking and makes the grid-orientation branch more robust via dimension checks.
- Standardizes VPU naming derived from filenames in the weights pipeline.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/forcingprocessor/weights_hf2ds.py | Adds infer_vpu_id() and uses it to standardize jcatchment_dict keys for multi-file weight extraction. |
| src/forcingprocessor/processor.py | Adds VPU-aware metadata helpers, config-driven/inferred VPU ids, and writes new VPU-delineated metadata artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| elif type(vpu_ids) is not list: | ||
| vpu_ids = [vpu_ids] | ||
|
|
||
| vpu_ids = [str(x) for x in vpu_ids] |
Comment on lines
+968
to
+979
| for i, (vpu_id, gpkg) in enumerate(zip(vpu_ids, gpkg_files)): | ||
| if str(gpkg).endswith(".parquet"): | ||
| # Prefer exact VPU key if the weights pipeline preserved it. | ||
| if vpu_id in jcatchment_dict: | ||
| vpu_catchment_map[vpu_id] = list(jcatchment_dict[vpu_id]) | ||
| # Otherwise fall back to positional mapping. | ||
| elif i < len(jcatchment_values): | ||
| vpu_catchment_map[vpu_id] = list(jcatchment_values[i]) | ||
| else: | ||
| vpu_catchment_map[vpu_id] = [] | ||
| else: | ||
| vpu_catchment_map[vpu_id] = read_gpkg_catchments(gpkg) |
Comment on lines
+1357
to
+1360
| # Issue 9: write VPU-delineated metadata. | ||
| # This keeps the original global metadata.csv, and adds metadata_by_vpu.csv. | ||
| if data_source == "forcings": | ||
| vpu_metadata_rows = [] |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
pratikshac15
force-pushed
the
vpu-metadata-support
branch
from
July 15, 2026 23:54
e0c9663 to
a42b42f
Compare
Contributor
Author
|
Fixed the Copilot-suggested changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds VPU-delineated metadata support for multi-geopackage and multi-weight-file forcing runs.
Additions
processor.pymetadata_by_vpu.csvoutputweights_hf2ds.pyasVPU_XXChanges
metadata.csvjcatchment_dictkeys and processor metadata keysTesting
Tested locally using two small GeoPackages created from
NWM_v3_hydrofabric.gdb:nextgen_VPU_03W.gpkgnextgen_VPU_10L.gpkgVerified successful generation of:
VPU_03W_forcings.ncVPU_10L_forcings.ncmetadata_by_vpu.csvNotes
This addresses Issue #9 by providing VPU-delineated stats when processing multiple geopackages/weight files.