National scale example model pathway optimisation¶

In this tutorial, we use the Calliope national scale example model to solve a pathway optimisation problem. This model can be loaded in its non-pathway format within Calliope as calliope.examples.national_scale().

In [1]:

import calliope
import calliope_pathways
import plotly.express as px

calliope.set_log_verbosity("INFO", include_solver_output=False)

import calliope import calliope_pathways import plotly.express as px calliope.set_log_verbosity("INFO", include_solver_output=False)

Model input¶

In [2]:

# Initialise with the National Scale example model and the "pathways" scenario
model = calliope_pathways.models.national_scale()

# Initialise with the National Scale example model and the "pathways" scenario model = calliope_pathways.models.national_scale()

[2024-07-19 11:27:02] INFO     Model: initialising

[2024-07-19 11:27:02] INFO     Model: preprocessing stage 1 (model_run)

[2024-07-19 11:27:06] INFO     Model: preprocessing stage 2 (model_data)

[2024-07-19 11:27:06] INFO     Model: preprocessing complete

Assessing the input data¶

Pathway analysis requires to us have new timeseries dimensions to track investment decisions across years. In this example, we define two dimensions: investsteps and vintagesteps.

At each investstep, investment in new capacity is possible and some previously invested capacity is decommissioned as it has reached the end of its lifetime. Investing in new capacity is likely to be necessary because of this end-of-life decommissioning but also because of phase-out decommissioning and operational constraints - demand might have increased or emissions caps might be more strict requiring more zero-emissions technologies. To verify that capacity in a given investstep is sufficient to meet that steps's operational constraints, technology dispatch is optimised in each step.

For example, if you have the investment steps [2030, 2040, 2050] you will have three points at which to change technology capacities and three sets of annual dispatch decisions (e.g., hourly dispatch over a whole year).

Deployed capacity in every investment year will be a combination of capacities deployed in previous years. Each will have a different age and may have different characteristics (maintenance costs, efficiency, etc.). We refer to each of these previous capacities as "vintages" and track them using vintagesteps. The available capacity in any investstep is the sum of all historical vintages. For example, In 2040, the vintages from 2030 and 2040 are available.

We use vintagesteps to track two things:

1. when a technology is due for decommissioning (if we invested in 100kW of PV in 2030 and it has a 10-year lifetime, it will not be available in 2040).
2. the characteristics of older models of a given technology. We don't use this at the moment, except to track the investment costs of vintages - these costs are applied in every investstep that the given vintage is still available.

Because we use the steps suffix for these dimensions, Calliope will read them in as timestamps.

In [3]:

model.inputs.investsteps

model.inputs.investsteps

Out[3]:

<xarray.DataArray 'investsteps' (investsteps: 4)> Size: 32B
array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
       '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * investsteps  (investsteps) datetime64[ns] 32B 2020-01-01 ... 2050-01-01

xarray.DataArray

'investsteps'

• investsteps: 4

• 2020-01-01 2030-01-01 2040-01-01 2050-01-01

  array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
         '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
        dtype='datetime64[ns]')

• Coordinates: (1)
  • investsteps
    (investsteps)
    datetime64[ns]
    2020-01-01 ... 2050-01-01

    array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
           '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • investsteps
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-01', '2030-01-01', '2040-01-01', '2050-01-01'], dtype='datetime64[ns]', name='investsteps', freq=None))

• Attributes: (0)

In [4]:

model.inputs.vintagesteps

model.inputs.vintagesteps

Out[4]:

<xarray.DataArray 'vintagesteps' (vintagesteps: 4)> Size: 32B
array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
       '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * vintagesteps  (vintagesteps) datetime64[ns] 32B 2020-01-01 ... 2050-01-01

xarray.DataArray

'vintagesteps'

• vintagesteps: 4

• 2020-01-01 2030-01-01 2040-01-01 2050-01-01

  array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
         '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
        dtype='datetime64[ns]')

• Coordinates: (1)
  • vintagesteps
    (vintagesteps)
    datetime64[ns]
    2020-01-01 ... 2050-01-01

    array(['2020-01-01T00:00:00.000000000', '2030-01-01T00:00:00.000000000',
           '2040-01-01T00:00:00.000000000', '2050-01-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • vintagesteps
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-01', '2030-01-01', '2040-01-01', '2050-01-01'], dtype='datetime64[ns]', name='vintagesteps', freq=None))

• Attributes: (0)

We can see some of our input data is indexed over these dimensions:

In [5]:

# Decreasing costs of investing in technologies
model.inputs.cost_flow_cap.to_series().dropna()

# Decreasing costs of investing in technologies model.inputs.cost_flow_cap.to_series().dropna()

Out[5]:

costs     techs               vintagesteps
monetary  ccgt                2020-01-01       750.0
                              2030-01-01       720.0
                              2040-01-01       700.0
                              2050-01-01       650.0
          csp                 2020-01-01      1000.0
                              2030-01-01       833.0
                              2040-01-01       769.0
                              2050-01-01       714.0
          region1_to_region2  2020-01-01       200.0
                              2030-01-01       180.0
                              2040-01-01       170.0
                              2050-01-01       160.0
Name: cost_flow_cap, dtype: float64

In [6]:

# Forced phase-out of combined-cycle gas turbines
model.inputs.flow_cap_max_systemwide.to_series().dropna()

# Forced phase-out of combined-cycle gas turbines model.inputs.flow_cap_max_systemwide.to_series().dropna()

Out[6]:

techs  investsteps
ccgt   2020-01-01     50000.0
       2030-01-01     30000.0
       2040-01-01     20000.0
       2050-01-01         0.0
Name: flow_cap_max_systemwide, dtype: float64

Initial capacity¶

Unlike greenfield optimisation ("plan" mode in Calliope), pathway optimisation should be initialised with a certain amount of existing technology capacity:

In [7]:

model.inputs.flow_cap_initial.to_series().dropna()

model.inputs.flow_cap_initial.to_series().dropna()

Out[7]:

techs               nodes    
battery             region2          0.0
ccgt                region1      10000.0
csp                 region1_1     1000.0
                    region1_2     2000.0
                    region1_3     3000.0
region1_to_region2  region1       5000.0
                    region2       5000.0
Name: flow_cap_initial, dtype: float64

This technology capacity can then be phased out as we step through to the end of our time horizon:

In [8]:

# This is a fraction of the initial capacity that remains available in investment steps
model.inputs.available_initial_cap.to_series().dropna()

# This is a fraction of the initial capacity that remains available in investment steps model.inputs.available_initial_cap.to_series().dropna()

Out[8]:

techs  investsteps
ccgt   2020-01-01     1.0
       2030-01-01     0.6
       2040-01-01     0.0
       2050-01-01     0.0
csp    2020-01-01     1.0
       2030-01-01     0.9
       2040-01-01     0.3
       2050-01-01     0.0
Name: available_initial_cap, dtype: float64

Vintages¶

Technology vintages have specific characteristics, such as the cost to invest in the model. Recently commercialised technologies may see their deployment costs decrease substantially in the first decade or two of mass deployment. Even established technologies can reduce in cost over time as the manufacturing facilities and logistical pipelines are constantly optimised.

In [9]:

model.inputs.cost_flow_cap.to_series().dropna()

model.inputs.cost_flow_cap.to_series().dropna()

Out[9]:

costs     techs               vintagesteps
monetary  ccgt                2020-01-01       750.0
                              2030-01-01       720.0
                              2040-01-01       700.0
                              2050-01-01       650.0
          csp                 2020-01-01      1000.0
                              2030-01-01       833.0
                              2040-01-01       769.0
                              2050-01-01       714.0
          region1_to_region2  2020-01-01       200.0
                              2030-01-01       180.0
                              2040-01-01       170.0
                              2050-01-01       160.0
Name: cost_flow_cap, dtype: float64

In [10]:

model.inputs.cost_storage_cap.to_series().dropna()

model.inputs.cost_storage_cap.to_series().dropna()

Out[10]:

costs     techs    vintagesteps
monetary  battery  2020-01-01      200.0
                   2030-01-01      150.0
                   2040-01-01      100.0
                   2050-01-01       80.0
          csp      2020-01-01       50.0
                   2030-01-01       42.0
                   2040-01-01       38.0
                   2050-01-01       36.0
Name: cost_storage_cap, dtype: float64

End-of-life decommissioning is tracked with a matrix similar to initial capacities. Fractional availability accounts for technologies whose lifetimes fall in-between two investment steps.

In [11]:

# Note how vintages are never available in investsteps that are in their _future_.
model.inputs.available_vintages.to_series().dropna().unstack("investsteps")

# Note how vintages are never available in investsteps that are in their _future_. model.inputs.available_vintages.to_series().dropna().unstack("investsteps")

Out[11]:

	investsteps	2020-01-01	2030-01-01	2040-01-01	2050-01-01
techs	vintagesteps
battery	2020-01-01	1.0	0.0	0.0	0.0
	2030-01-01	NaN	1.0	0.0	0.0
	2040-01-01	NaN	NaN	1.0	0.0
	2050-01-01	NaN	NaN	NaN	1.0
ccgt	2020-01-01	1.0	0.6	0.4	0.0
	2030-01-01	NaN	1.0	0.6	0.4
	2040-01-01	NaN	NaN	1.0	0.6
	2050-01-01	NaN	NaN	NaN	1.0
csp	2020-01-01	1.0	1.0	0.4	0.6
	2030-01-01	NaN	1.0	1.0	0.4
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0
demand_power	2020-01-01	1.0	1.0	1.0	1.0
	2030-01-01	NaN	1.0	1.0	1.0
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0
region1_to_region1_1	2020-01-01	1.0	1.0	1.0	1.0
	2030-01-01	NaN	1.0	1.0	1.0
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0
region1_to_region1_2	2020-01-01	1.0	1.0	1.0	1.0
	2030-01-01	NaN	1.0	1.0	1.0
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0
region1_to_region1_3	2020-01-01	1.0	1.0	1.0	1.0
	2030-01-01	NaN	1.0	1.0	1.0
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0
region1_to_region2	2020-01-01	1.0	1.0	0.5	0.5
	2030-01-01	NaN	1.0	1.0	0.5
	2040-01-01	NaN	NaN	1.0	1.0
	2050-01-01	NaN	NaN	NaN	1.0

Building a pathways optimisation problem¶

We have created a math YAML file with updates to all the pre-defined math to handle the existence of investsteps and vintagesteps. Tracking new capacity in each investment period and linking it to technology vintages requires new variables and constraints

Variables¶

In [12]:

# Note the "investsteps" dimension added to this pre-defined variable
model.math["variables"]["flow_cap"]

# Note the "investsteps" dimension added to this pre-defined variable model.math["variables"]["flow_cap"]

Out[12]:

{'description': "A technology's flow capacity, also known as its nominal or nameplate capacity.",
 'default': 0,
 'unit': 'power',
 'foreach': ['nodes', 'techs', 'carriers', 'investsteps'],
 'bounds': {'min': 'flow_cap_min', 'max': 'flow_cap_max'}}

In [13]:

# This new variable tracks the amount of each technology vintage that exists
model.math["variables"]["flow_cap_new"]

# This new variable tracks the amount of each technology vintage that exists model.math["variables"]["flow_cap_new"]

Out[13]:

{'bounds': {'max': 'flow_cap_new_max', 'min': 0},
 'description': 'Additional flow capacity commissioned in an investstep.',
 'foreach': ['nodes', 'techs', 'carriers', 'vintagesteps'],
 'unit': 'power',
 'where': 'flow_cap'}

Constraints¶

In [14]:

# All existing constraints have the "investsteps" dimension added.
# This allows the dispatch decisions to be optimised individually for each investment step.
model.math["constraints"]["system_balance"]

# All existing constraints have the "investsteps" dimension added. # This allows the dispatch decisions to be optimised individually for each investment step. model.math["constraints"]["system_balance"]

Out[14]:

{'description': 'Set the global carrier balance of the optimisation problem by fixing the total production of a given carrier to equal the total consumption of that carrier at every node in every timestep.',
 'foreach': ['nodes', 'carriers', 'timesteps', 'investsteps'],
 'equations': [{'expression': 'sum(flow_out, over=techs) - sum(flow_in, over=techs) - $flow_export + $unmet_demand_and_unused_supply == 0'}],
 'sub_expressions': {'flow_export': [{'where': 'any(carrier_export, over=techs)',
    'expression': 'sum(flow_export, over=techs)'},
   {'where': 'NOT any(carrier_export, over=techs)', 'expression': '0'}],
  'unmet_demand_and_unused_supply': [{'where': 'config.ensure_feasibility=True',
    'expression': 'unmet_demand + unused_supply'},
   {'where': 'NOT config.ensure_feasibility=True', 'expression': '0'}]}}

In [15]:

# In each investment period, capacities are a combination of all available vintages.
model.math["constraints"]["flow_cap_bounding"]

# In each investment period, capacities are a combination of all available vintages. model.math["constraints"]["flow_cap_bounding"]

Out[15]:

{'description': 'Flow capacity in a given investment period is the capacity in the previous period plus the new capacity installed minus old capacity that has been decommissioned between the previous period and this one.',
 'equations': [{'expression': 'flow_cap == sum(flow_cap_new * available_vintages, over=vintagesteps) + flow_cap_initial * available_initial_cap'}],
 'foreach': ['nodes', 'techs', 'carriers', 'investsteps'],
 'where': 'flow_cap'}

Building¶

In [16]:

model.inputs.flow_cap_max.to_series().dropna()

model.inputs.flow_cap_max.to_series().dropna()

Out[16]:

nodes      techs             
region1    ccgt                  30000.0
           region1_to_region2    10000.0
region1_1  csp                   20000.0
region1_2  csp                   20000.0
region1_3  csp                   20000.0
region2    region1_to_region2    10000.0
Name: flow_cap_max, dtype: float64

In [17]:

model.build()

model.build()

[2024-07-19 11:27:07] INFO     Optimisation Model | parameters | Generated.

[2024-07-19 11:27:08] INFO     Optimisation Model | variables | Generated.

[2024-07-19 11:27:09] INFO     Optimisation Model | global_expressions | Generated.

[2024-07-19 11:27:14] INFO     Optimisation Model | constraints | Generated.

[2024-07-19 11:27:14] INFO     Optimisation Model | objectives | Generated.

Analyse results¶

In [18]:

model.solve()

model.solve()

[2024-07-19 11:27:14] INFO     Optimisation model | starting model in plan mode.

[2024-07-19 11:27:22] INFO     Backend: solver finished running. Time since start of solving optimisation problem: 0:00:08.596177

[2024-07-19 11:27:22] INFO     Postprocessing: started

[2024-07-19 11:27:23] INFO     Postprocessing: All values < 1e-10 set to 0 in flow_out, flow_in, storage, flow_out_inc_eff, flow_in_inc_eff, cost_var, capacity_factor

[2024-07-19 11:27:23] INFO     Postprocessing: ended. Time since start of solving optimisation problem: 0:00:08.764552

[2024-07-19 11:27:23] INFO     Model: loaded model_data

In [19]:

df_capacity = (
    model.results.flow_cap.where(model.results.techs != "demand_power")
    .sel(carriers="power")
    .sum("nodes")
    .to_series()
    .where(lambda x: x != 0)
    .dropna()
    .to_frame("Flow capacity (kW)")
    .reset_index()
)

print(df_capacity.head())

fig = px.bar(
    df_capacity,
    x="investsteps",
    y="Flow capacity (kW)",
    color="techs",
    color_discrete_map=model.inputs.color.to_series().to_dict(),
)
fig.show()

df_capacity = ( model.results.flow_cap.where(model.results.techs != "demand_power") .sel(carriers="power") .sum("nodes") .to_series() .where(lambda x: x != 0) .dropna() .to_frame("Flow capacity (kW)") .reset_index() ) print(df_capacity.head()) fig = px.bar( df_capacity, x="investsteps", y="Flow capacity (kW)", color="techs", color_discrete_map=model.inputs.color.to_series().to_dict(), ) fig.show()

     techs investsteps  Flow capacity (kW)
0  battery  2020-01-01           2000.0000
1  battery  2030-01-01           2695.3370
2  battery  2040-01-01           2983.4921
3  battery  2050-01-01          10000.0000
4     ccgt  2020-01-01          22007.1460

In [20]:

df_capacity = (
    model.results.storage_cap.sum("nodes")
    .to_series()
    .where(lambda x: x != 0)
    .dropna()
    .to_frame("Storage capacity (kWh)")
    .reset_index()
)

print(df_capacity.head())

fig = px.bar(
    df_capacity,
    x="investsteps",
    y="Storage capacity (kWh)",
    color="techs",
    color_discrete_map=model.inputs.color.to_series().to_dict(),
)
fig.show()

df_capacity = ( model.results.storage_cap.sum("nodes") .to_series() .where(lambda x: x != 0) .dropna() .to_frame("Storage capacity (kWh)") .reset_index() ) print(df_capacity.head()) fig = px.bar( df_capacity, x="investsteps", y="Storage capacity (kWh)", color="techs", color_discrete_map=model.inputs.color.to_series().to_dict(), ) fig.show()

     techs investsteps  Storage capacity (kWh)
0  battery  2020-01-01               61453.684
1  battery  2030-01-01               61453.684
2  battery  2040-01-01               74097.895
3  battery  2050-01-01              252631.580
4      csp  2020-01-01              600000.000

In [21]:

df_capacity = (
    model.results.flow_cap_new.where(model.results.techs != "demand_power")
    .sel(carriers="power")
    .sum("nodes")
    .to_series()
    .where(lambda x: x != 0)
    .dropna()
    .to_frame("New flow capacity (kW)")
    .reset_index()
)

print(df_capacity.head())

fig = px.bar(
    df_capacity,
    x="vintagesteps",
    y="New flow capacity (kW)",
    color="techs",
    color_discrete_map=model.inputs.color.to_series().to_dict(),
)
fig.show()

df_capacity = ( model.results.flow_cap_new.where(model.results.techs != "demand_power") .sel(carriers="power") .sum("nodes") .to_series() .where(lambda x: x != 0) .dropna() .to_frame("New flow capacity (kW)") .reset_index() ) print(df_capacity.head()) fig = px.bar( df_capacity, x="vintagesteps", y="New flow capacity (kW)", color="techs", color_discrete_map=model.inputs.color.to_series().to_dict(), ) fig.show()

     techs vintagesteps  New flow capacity (kW)
0  battery   2020-01-01               2000.0000
1  battery   2030-01-01               2695.3370
2  battery   2040-01-01               2983.4921
3  battery   2050-01-01              10000.0000
4     ccgt   2020-01-01              12007.1460

In [22]:

df_outflow = (
    (model.results.flow_out.fillna(0) - model.results.flow_in.fillna(0))
    .sel(carriers="power")
    .sum(["nodes", "timesteps"], min_count=1)
    .to_series()
    .where(lambda x: x > 1)
    .dropna()
    .to_frame("Annual outflow (kWh)")
    .reset_index()
)

print(df_capacity.head())

fig = px.bar(
    df_outflow,
    x="investsteps",
    y="Annual outflow (kWh)",
    color="techs",
    color_discrete_map=model.inputs.color.to_series().to_dict(),
)
df_demand = (
    model.results.flow_in.sel(techs="demand_power", carriers="power")
    .sum(["nodes", "timesteps"])
    .to_series()
    .reset_index()
)
fig.add_scatter(
    x=df_demand.investsteps, y=df_demand.flow_in, line={"color": "black"}, name="Demand"
)
fig.show()

df_outflow = ( (model.results.flow_out.fillna(0) - model.results.flow_in.fillna(0)) .sel(carriers="power") .sum(["nodes", "timesteps"], min_count=1) .to_series() .where(lambda x: x > 1) .dropna() .to_frame("Annual outflow (kWh)") .reset_index() ) print(df_capacity.head()) fig = px.bar( df_outflow, x="investsteps", y="Annual outflow (kWh)", color="techs", color_discrete_map=model.inputs.color.to_series().to_dict(), ) df_demand = ( model.results.flow_in.sel(techs="demand_power", carriers="power") .sum(["nodes", "timesteps"]) .to_series() .reset_index() ) fig.add_scatter( x=df_demand.investsteps, y=df_demand.flow_in, line={"color": "black"}, name="Demand" ) fig.show()

     techs vintagesteps  New flow capacity (kW)
0  battery   2020-01-01               2000.0000
1  battery   2030-01-01               2695.3370
2  battery   2040-01-01               2983.4921
3  battery   2050-01-01              10000.0000
4     ccgt   2020-01-01              12007.1460

In [23]:

df_electricity = (
    (model.results.flow_out.fillna(0) - model.results.flow_in.fillna(0))
    .sel(carriers="power")
    .sum("nodes")
    .to_series()
    .where(lambda x: x != 0)
    .dropna()
    .to_frame("Flow in/out (kWh)")
    .reset_index()
)
df_electricity_demand = df_electricity[df_electricity.techs == "demand_power"]
df_electricity_other = df_electricity[df_electricity.techs != "demand_power"]

invest_order = sorted(df_electricity.investsteps.unique())

fig = px.bar(
    df_electricity_other,
    x="timesteps",
    y="Flow in/out (kWh)",
    facet_row="investsteps",
    color="techs",
    height=1000,
    category_orders={"investsteps": invest_order},
    color_discrete_map=model.inputs.color.to_series().to_dict(),
)

showlegend = True
# we reverse the investment year order (`[::-1]`) because the rows are numbered from bottom to top.
for row, year in enumerate(invest_order[::-1]):
    demand_ = df_electricity_demand.loc[(df_electricity_demand.investsteps == year)]
    fig.add_scatter(
        x=demand_["timesteps"],
        y=-1 * demand_["Flow in/out (kWh)"],
        row=row + 1,
        col="all",
        marker_color="black",
        name="Demand",
        legendgroup="demand",
        showlegend=showlegend,
    )
    showlegend = False
fig.update_yaxes(matches=None)
fig.show()

df_electricity = ( (model.results.flow_out.fillna(0) - model.results.flow_in.fillna(0)) .sel(carriers="power") .sum("nodes") .to_series() .where(lambda x: x != 0) .dropna() .to_frame("Flow in/out (kWh)") .reset_index() ) df_electricity_demand = df_electricity[df_electricity.techs == "demand_power"] df_electricity_other = df_electricity[df_electricity.techs != "demand_power"] invest_order = sorted(df_electricity.investsteps.unique()) fig = px.bar( df_electricity_other, x="timesteps", y="Flow in/out (kWh)", facet_row="investsteps", color="techs", height=1000, category_orders={"investsteps": invest_order}, color_discrete_map=model.inputs.color.to_series().to_dict(), ) showlegend = True # we reverse the investment year order (`[::-1]`) because the rows are numbered from bottom to top. for row, year in enumerate(invest_order[::-1]): demand_ = df_electricity_demand.loc[(df_electricity_demand.investsteps == year)] fig.add_scatter( x=demand_["timesteps"], y=-1 * demand_["Flow in/out (kWh)"], row=row + 1, col="all", marker_color="black", name="Demand", legendgroup="demand", showlegend=showlegend, ) showlegend = False fig.update_yaxes(matches=None) fig.show()