75 notebooks by dalonsoa · Pull Request #84 · EnergySystemsModellingLab/muse2_data_analysis

dalonsoa · 2026-07-16T06:48:03Z

Description

Update the capacity notebook to account for the new output file structure - which is far from ideal from the processing perspective. Performance-wise, there are vectorised options that are faster than apply, but this one was more readable.

The output plot is:

Close #75

Type of change

Documentation (non-breaking change that adds or improves the documentation)
New feature (non-breaking change which adds functionality)
Optimization (non-breaking, back-end change that speeds up the code)
Bug fix (non-breaking change which fixes an issue)
Breaking change (whatever its nature)

Key checklist

All tests pass (eg. python -m pytest)
Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

Code is commented, particularly in hard-to-understand areas
Tests added or an issue has been opened to tackle that in the future.
(Indicate issue here: # (issue))

dalonsoa · 2026-07-16T06:52:41Z

The latest MUSE2 release uses the old file structure, so those tests are failing - but it is for a known reason.

dc2917

Seems reasonable to me!

is far from ideal from the processing perspective. Performance-wise, there are vectorised options that are faster than apply, but this one was more readable.

I think for a notebook this sounds like a good approach.

dalonsoa · 2026-07-16T07:48:07Z

This is the vectorised version, by the way:

# Try merging on asset_id first, then fall back to group_id for unmatched rows
merged = asset_capacities.merge(
    assets[["asset_id", "agent_id", "process_id", "commission_year"]],
    on="asset_id",
    how="left"
)

# Fill in rows where asset_id was None using group_id
mask = merged["agent_id"].isna()
fallback = asset_capacities[mask].merge(
    assets[["group_id", "agent_id", "process_id", "commission_year"]],
    on="group_id",
    how="left"
)

merged.loc[mask, ["agent_id", "process_id", "commission_year"]] = fallback[["agent_id", "process_id", "commission_year"]].values
asset_capacities = merged

dc2917 · 2026-07-16T07:57:00Z

This is the vectorised version, by the way:

Yeah, it's not that it's unclear, but it does make the process look more cumbersome, even if it's less work for the computer to do. I guess it depends on the purpose of the notebook?

tsmbland

This isn't quite right I'm afraid. asset_capacities.csv has a row for each asset in each year that it has capacity, so it's just a matter of merging this table with assets.csv to add metadata, then grouping by (milestone_year, agent_id, process_id) and summing. With some pandas trickery this can be relatively concise:

assets = pd.read_csv(OUTPUT_DIR / "assets.csv")
asset_capacities = pd.read_csv(OUTPUT_DIR / "asset_capacities.csv")

merged = asset_capacities.merge(assets, on=["asset_id", "group_id"])
agg_capacities = merged.groupby(["milestone_year", "agent_id", "process_id"])[
    "capacity"
].sum()
agg_capacities_wide = agg_capacities.unstack(["agent_id", "process_id"], fill_value=0)

agents = agg_capacities_wide.columns.get_level_values("agent_id").unique()
fig, axes = plt.subplots(1, len(agents), figsize=(4 * len(agents), 4))
for ax, agent in zip(axes, agents):
    agg_capacities_wide[agent].plot(
        kind="bar", stacked=True, ax=ax, title=agent, xlabel="Year", ylabel="Capacity"
    )

plt.tight_layout()

I think stacked bars look better as well

If we wanted to use something like seaborn I think this could be even more concise (e.g. no need to loop over agents)

tsmbland · 2026-07-16T14:08:19Z

We now copy the input data to the output folder (EnergySystemsModellingLab/MUSE2#1282), so we can drop this.

I don't think we need to read model.toml anyway

tsmbland · 2026-07-16T14:09:10Z

This is apparently no longer true

Should remove this comment as it isn't true

tsmbland · 2026-07-16T14:09:56Z

This block needs updating

dalonsoa · 2026-07-17T08:10:04Z

New plot:

Copilot

Pull request overview

Updates the example analysis notebooks to align with the new MUSE2 output file structure (splitting time-varying capacities into asset_capacities.csv and keeping metadata in assets.csv), and improves plot layout/legibility.

Changes:

Refactors the capacity notebook to compute invested capacity from asset_capacities.csv merged with metadata from assets.csv.
Adjusts plot layouts (legend placement in prices.ipynb, stacked bar plots in capacity.ipynb).
Updates notebook metadata (kernel display name / Python version).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
notebooks/prices.ipynb	Improves legend placement for the price bar plot and updates notebook metadata.
notebooks/capacity.ipynb	Reworks capacity aggregation to use `asset_capacities.csv` + `assets.csv`, and updates plotting logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tsmbland · 2026-07-17T08:44:17Z

+    "merged = asset_capacities.merge(assets, on=[\"asset_id\", \"group_id\"])\n",
+    "agg_capacities = merged.groupby([\"milestone_year\", \"agent_id\", \"process_id\"])[\n",
+    "    \"capacity\"\n",
+    "].sum()\n",
+    "agg_capacities_wide = agg_capacities.unstack([\"agent_id\", \"process_id\"], fill_value=0)\n",


It does work though 🤷

It might work in this case because there's no group_id in any of the rows. Maybe if there were, things would fail.

I tried it with the simple_divisible model (which has rows with group_id populated) and it worked, so I think it's fine

tsmbland · 2026-07-17T08:44:59Z

+    "agents = agg_capacities_wide.columns.get_level_values(\"agent_id\").unique()\n",
+    "fig, axes = plt.subplots(1, len(agents), figsize=(4 * len(agents), 4))\n",
    "for ax, agent in zip(axes, agents):\n",
-    "    capacity[capacity[\"agent_id\"] == agent].pivot(\n",
-    "        index=\"year\", columns=\"process_id\", values=\"capacity\"\n",
-    "    ).plot(kind=\"bar\", ax=ax)\n",
-    "    ax.set_title(agent)\n",
-    "    ax.set_xlabel(\"Year\")\n",
-    "    ax.set_ylabel(\"Capacity\")\n",
-    "    ax.legend(title=\"Process\")"
+    "    agg_capacities_wide[agent].plot(\n",
+    "        kind=\"bar\", stacked=True, ax=ax, title=agent, xlabel=\"Year\", ylabel=\"Capacity\"\n",
+    "    )\n",


Not too concerned about this. If there was only one agent then you'd plot in a different way anyway

tsmbland

Almost there!

tsmbland · 2026-07-17T08:41:00Z

-    "We begin by loading the `model.toml` input file to get the list of milestone years.\n",
-    "\n",
-    "[output-format]: https://energysystemsmodellinglab.github.io/MUSE2/file_formats/output_files.html"
+    "Output files are mostly in CSV format. The format of output files is documented [here][output-format]."


You need to add the link back

tsmbland · 2026-07-17T08:42:00Z

Should remove this comment as it isn't true

tsmbland · 2026-07-17T08:44:17Z

+    "merged = asset_capacities.merge(assets, on=[\"asset_id\", \"group_id\"])\n",
+    "agg_capacities = merged.groupby([\"milestone_year\", \"agent_id\", \"process_id\"])[\n",
+    "    \"capacity\"\n",
+    "].sum()\n",
+    "agg_capacities_wide = agg_capacities.unstack([\"agent_id\", \"process_id\"], fill_value=0)\n",


It does work though 🤷

tsmbland · 2026-07-17T08:44:59Z

+    "agents = agg_capacities_wide.columns.get_level_values(\"agent_id\").unique()\n",
+    "fig, axes = plt.subplots(1, len(agents), figsize=(4 * len(agents), 4))\n",
    "for ax, agent in zip(axes, agents):\n",
-    "    capacity[capacity[\"agent_id\"] == agent].pivot(\n",
-    "        index=\"year\", columns=\"process_id\", values=\"capacity\"\n",
-    "    ).plot(kind=\"bar\", ax=ax)\n",
-    "    ax.set_title(agent)\n",
-    "    ax.set_xlabel(\"Year\")\n",
-    "    ax.set_ylabel(\"Capacity\")\n",
-    "    ax.legend(title=\"Process\")"
+    "    agg_capacities_wide[agent].plot(\n",
+    "        kind=\"bar\", stacked=True, ax=ax, title=agent, xlabel=\"Year\", ylabel=\"Capacity\"\n",
+    "    )\n",


Not too concerned about this. If there was only one agent then you'd plot in a different way anyway

for more information, see https://pre-commit.ci

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

+    "    agg_capacities_wide[agent].plot(\n",
+    "        kind=\"bar\", stacked=True, ax=ax, title=agent, xlabel=\"Year\", ylabel=\"Capacity\"\n",
+    "    )\n",


dalonsoa added 2 commits July 16, 2026 07:44

update capacity plot

71d441e

Move legend in prices plot

6cf4d68

Copilot AI review requested due to automatic review settings July 16, 2026 06:48

Copilot started reviewing on behalf of dalonsoa July 16, 2026 06:48 View session

dalonsoa requested review from AdrianDAlessandro, dc2917 and tsmbland and removed request for Copilot July 16, 2026 06:50

dc2917 approved these changes Jul 16, 2026

View reviewed changes

tsmbland requested changes Jul 16, 2026

View reviewed changes

Simplify code following review

e524b00

dalonsoa requested review from Copilot, dc2917 and tsmbland July 17, 2026 08:08

Copilot started reviewing on behalf of dalonsoa July 17, 2026 08:08 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

tsmbland requested changes Jul 17, 2026

View reviewed changes

dalonsoa added 2 commits July 17, 2026 12:22

Update capacity notebook

3d5ff46

Update pre-commit-hooks to remove empy cells

008bd04

Copilot AI review requested due to automatic review settings July 17, 2026 11:24

[pre-commit.ci] auto fixes from pre-commit.com hooks

b230c86

for more information, see https://pre-commit.ci

dalonsoa requested a review from tsmbland July 17, 2026 11:24

Copilot started reviewing on behalf of dalonsoa July 17, 2026 11:24 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread notebooks/capacity.ipynb

Comment on lines +89 to +91

" agg_capacities_wide[agent].plot(\n",

" kind=\"bar\", stacked=True, ax=ax, title=agent, xlabel=\"Year\", ylabel=\"Capacity\"\n",

" )\n",

tsmbland approved these changes Jul 17, 2026

View reviewed changes

Uh oh!

Conversation

dalonsoa commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Key checklist

Further checks

Uh oh!

dalonsoa commented Jul 16, 2026

Uh oh!

dc2917 left a comment

Choose a reason for hiding this comment

Uh oh!

dalonsoa commented Jul 16, 2026

Uh oh!

dc2917 commented Jul 16, 2026

Uh oh!

tsmbland left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dalonsoa commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsmbland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dalonsoa commented Jul 16, 2026 •

edited

Loading

tsmbland left a comment •

edited

Loading