Datastructure for FE conditioning - #2706
Conversation
|
@evenmn could you have a look? |
github-actions
Bot
added
data
evenmn
left a comment
evenmn
left a comment
There was a problem hiding this comment.
Looks good to me, just a few comments. The splitting of streams_datasets could perhaps be moved to keep the changes in _init_stream_datasets minimal, but I don't think it's a big issue. I'm looking forward to testing this!
| self.conditioning_type = stream_info.get("conditioning_type", "time_based") | ||
| value_type = stream_info.get("value_type", "hour") | ||
| if isinstance(value_type, list): | ||
| self.value_types = value_type |
There was a problem hiding this comment.
When will value_type be a list? Can a stream have multiple value types?
| self.mean = np.array([0.0], dtype=np.float32) | ||
| self.stdev = np.array([1.0], dtype=np.float32) | ||
| self.mean_geoinfo = np.zeros(0, dtype=np.float32) | ||
| self.stdev_geoinfo = np.ones(0, dtype=np.float32) |
There was a problem hiding this comment.
For consistency we could use either
self.mean = np.array([0.0], dtype=np.float32)
self.stdev = np.array([1.0], dtype=np.float32)
self.mean_geoinfo = np.array([], dtype=np.float32)
self.stdev_geoinfo = np.array([], dtype=np.float32)
or
self.mean = np.zeros(1, dtype=np.float32)
self.stdev = np.ones(1, dtype=np.float32)
self.mean_geoinfo = np.zeros(0, dtype=np.float32)
self.stdev_geoinfo = np.ones(0, dtype=np.float32)
| values = [] | ||
| for vt in self.value_types: | ||
| if self.conditioning_type == "time_based": | ||
| if vt == "hour": |
There was a problem hiding this comment.
Can we add a comment here stressing that value here is the fraction of the day (e.g. 0.75 at 6 PM)? And same for vt == "day"?
| filenames_cfg = stream_info.get("filenames", [pathlib.Path()]) | ||
| conditioning_cfg = stream_info.get("conditioning", False) | ||
|
|
||
| if conditioning_cfg: |
There was a problem hiding this comment.
Can this logic be moved to DataReaderTimeConditioning, too avoid logics in multi_stream_data_sampler.py that is specific for a data reader?
| conditioning_cfg = stream_info.get("conditioning", False) | ||
|
|
||
| if conditioning_cfg: | ||
| if filenames_cfg is None or len(filenames_cfg) == 0: |
There was a problem hiding this comment.
This handles filenames when it is explicitly set to null/None. However, when the entry is omitted, it defaults to [pathlib.Path()], so this if-statement is not invoked.
Since the default is needed by the processing below, should this condition handle both cases?
if filenames_cfg is None or filenames_cfg == [pathlib.Path()]:2 participants
Description
This PR closes the first subtasks of having forcing in WeatherGen. It included scalar and field data reading and loading.
Run with:
--options streams_directory="./config/streams/era5_1deg_forecasting_conditioning/"Issue Number
Closes #2536 #2537
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60