Fix/greedy time impute refactoring - #39
Conversation
| # A diverse set of statuses to diminish oversimplification during gap filling. | ||
| OPERATING = "operating" | ||
| RETIRED = "retired" | ||
| HISTORICAL = {OPERATING, RETIRED} | ||
| PLANNED = {"construction", "pre-construction", "announced"} | ||
| SCENARIO_MAP = { | ||
| "historical": HISTORICAL, | ||
| "construction": HISTORICAL | {"construction"}, | ||
| "pre_construction": HISTORICAL | {"construction", "pre-construction"}, | ||
| "announced": HISTORICAL | PLANNED, | ||
| } | ||
| # Status categorisation shown to users (and accepted in user imputed files). | ||
| IMPUTED_STATUS = {"planned", "operating", "retired"} |
There was a problem hiding this comment.
Moved this here to allow the schema to use it, for higher standardisation.
| @gpa.dataframe_check | ||
| def end_after_start(cls, plants: pd.DataFrame): | ||
| """Require ordered dates wherever both years are known.""" | ||
| known_dates = plants[["start_year", "end_year"]].notna().all(axis="columns") | ||
| return ~known_dates | plants["end_year"].gt(plants["start_year"]) | ||
|
|
There was a problem hiding this comment.
We now validate that start_year < end_year.
This should've been there from the start, frankly 😬
| known_dates = filtered[["start_year", "end_year"]].notna().all(axis="columns") | ||
| zero_duration = known_dates & filtered["start_year"].eq(filtered["end_year"]) | ||
| return filtered.loc[~zero_duration].copy() | ||
|
|
There was a problem hiding this comment.
All the 'prepare' scripts should use this function to remove powerplants installed and then removed on the same year. They do not contribute to statistics under our convention:
start_year <= operation < end_year (a plant starts at day 1 of start year, and is decommissioned on day 1 of end year).
| def push(year: int) -> None: | ||
| """Add a year to the allocation queue with its current priority.""" | ||
| year_tie = -year if prefer_later_years else year | ||
| # heapq uses a min-heap algorithm | ||
| # negating values prioritize the largest deficit and then the largest target. | ||
| heapq.heappush(heap, (-remaining.loc[year], -target.loc[year], year_tie, year)) | ||
|
|
There was a problem hiding this comment.
This replaces the re-computation per powerplant in the previous script.
https://docs.python.org/3/library/heapq.html
Priority is given in this order: biggest deficit, biggest target powerplant, earlier year (reversible via perfer_later_years).
We negate because the algorithm minimises.
| def impute_time( | ||
| plants: pd.DataFrame, reference_capacity_df: pd.DataFrame, imputation: Mapping | ||
| ) -> tuple[pd.DataFrame, pd.DataFrame]: | ||
| """Impute missing dates and return plants plus normalized profile diagnostics.""" |
There was a problem hiding this comment.
This lays down the architecture for this stage of the module.
Standard regardless of the selected method:
- Adjust statuses to
DATASET_YEAR. - Filter to the requested scenario
- Run lifetime imputation for plants with known years
- Run commissioning windows for start years of future projects with unknown years
- Re-run lifetime imputation to add the end year of those future projects
Then, the remaining 'unknown' plants are imputed using a selectable method.
For now, only capacity_profiles is available. In the future we might want to add a heavier optimisation method.
Lastly, the status is simplified for users.
1 participant
Fixes #40
Summary of changes in this pull request
impute_ages.pyfor improved maintainability by standardising the order of operations and making assumptions / priorities clearer for future development.capacity_profilesalgorithm.Examples of solved retirement profile edge case
Before, Mexico was pretending to be a wind power pioneer.
This happened because the start-year of powerplants imputed through the retirement profile were derived using the lifetime.
After, Mexico is less of a pioneer (although the super old facility is legitimate!)
The new method 'pre-detects' unknown retired plants, and adjusts their individual end-date limit so it can always be decommissioned in time. Then, their start year is solved using the commissioning profile just as any other plant.
Disadvantages
The trade-off is that the powerplant might be introduced and immediately retired the year after when the most empty years are near the end of the commissioning profile.
It is a bit hard to see, but this happens for solar in Mexico due to lack of data.
I think it's generally fine, as at least this shields us from getting ahistorical profiles (such as lots of wind / PV in the 1990's).
Reviewer checklist
pipdependencies in the module's environment files (workflow/envs/).pathvars(e.g.,<results>) in their inputs and outputs.pre-commit.citests pass.INTERFACE.yamlmentions all relevantpathvarsandwildcards.README.mddescribes how to use the module and has the necessary citations.