Behind the paper: ‘human-in-the-loop’ energy system design

We talk to Francesco Lombardi and Stefan Pfenninger, authors of the recent PLOS Climate publication “Human-in-the-loop MGA to generate energy system design options matching stakeholder needs”

What led you to decide on this research question?

The energy transition is urgent, but progress is not as fast as the goals agreed upon internationally would require. Among the reasons for the slow progress is the difficulty of making decisions that may appeal to the many heterogeneous stakeholders involved. With this work, we wanted to contribute to facilitating such decisions and thereby accelerate the energy transition.

Energy planning decisions typically rely on the support of ‘energy system optimisation models’. These models can find investment decisions on new energy infrastructure that enable the achievement of a set goal, such as the energy transition, at the lowest possible cost for society. In theory, this sounds perfect. In practice, countless alternative investment decisions can achieve the same goal for roughly the same cost and remain hidden from view by this insistence on cost minimisation. While having roughly the same cost, these alternative decisions may be preferred by stakeholders for reasons that cannot be modelled, such as social acceptability, environmental sustainability, ease of implementation or other practical matters. Relying overly on a single, least-cost solution may thus mislead into thinking that these alternatives do not exist and thereby prevent the identification of a consensus solution that, while being affordable, matches other critical stakeholder preferences.

Mathematical methods to systematically explore these alternative energy planning decision options exist and are known as ‘Modelling to Generate Alternatives’ or MGA methods. They have gained momentum in the last couple of years thanks to many original advancements specific to energy planning applications, to which we contributed. However, with finite computational power, MGA can only explore a sample of the countless alternatives that may matter to real-world stakeholders. To ensure that MGA is helpful, we must capture all the trade-offs real-world stakeholders need to discuss and reach a consensus. This what led us to our specific research question: how can we directly leverage stakeholder preferences to guide the MGA search for meaningful decision alternatives?

How did you go about designing your study?

We hypothesised that, first, we could set up an automated procedure to elicit stakeholder preferences based on their interaction with a first sample of MGA-generated energy planning decision options. Second, we could translate those into parameters that could re-tune the algorithm underlying the MGA search for planning options so that it would generate more options in line with stakeholder preferences and thus facilitate consensus formation. We call this a ‘human-in-the-loop’ version of MGA since human inputs are directly and automatically fed to the MGA method to enhance it.

After careful reflection, we purposefully decided to perform an experiment under synthetic conditions rather than immediately testing the hypothesis with real-world stakeholders. In other words, for the case of the Portuguese energy system, we simulated synthetic stakeholders with different preferences about energy planning and energy system features. For instance, we assumed that some stakeholders prefer to rely as little as possible on energy imports from abroad, while others prefer to avoid high concentrations of wind farms in specific regions.

The method is agnostic to these high-level preferences and does not require any knowledge about them, making it easily applicable to any multifaceted real-world preference. All it needs is a list of the ‘most liked’ energy planning decision options among those initially provided, from which it automatically decodes the underlying system features to be further prioritised by the MGA search. However, having precise knowledge of the initial high-level preferences allows us to more precisely assess the relative merits of our new ‘human-in-the-loop’ method compared to a real-world experiment. We can measure with exactness whether the decision options generated via the ‘human-in-the-loop’ iteration of MGA perform better with regard to the initial preferences.

Did you encounter any challenges in collecting or interpreting your data?

Well, yes, to a certain extent. The synthetic conditions of our experiments facilitate measuring how many of the energy planning decision options generated before and after the ‘human-in-the-loop’ iteration of MGA match any given synthetic stakeholder preference. But how do we measure whether this increases the overall likelihood of consensus formation? In a real-world experiment, you could ask the participating stakeholders whether the human-in-the-loop iteration brings them any closer to finding a solution that allows them to strike a balance between their respective preferences. In our synthetic experiment, we had the challenge of defining a metric that could approximate the likelihood of consensus formation well enough.

To this end, we applied a multi-criteria decision analysis to the assumed high-level stakeholder preferences. The degree to which a given energy planning strategy matches one of the high-level preferences is summed up to the degree to which it matches each of the other preferences. The more an energy planning strategy performs well across all the five assumed high-level preferences, the higher its aggregate multi-criteria score and its potential to represent a compromise between conflicting preferences. Thanks to this score, we could approximate the increased likelihood of consensus formation in the energy planning options generated via human-in-the-loop MGA.

What struck you most about your results? What are the key messages and who do you hope might benefit from these new insights?

We were confident that the human-in-the-loop approach to MGA would increase the likelihood of consensus formation, but we were struck by how substantial the benefits are. For the same number of generated alternative energy planning strategies, conventional MGA has only about 1% of the generated strategies showing a high consensus potential – based on the assumed synthetic preferences – as opposed to 18% for the case of human-in-the-loop MGA. This underscores how well the procedure we designed to automatically elicit and translate hidden stakeholder preferences starting from a list of ‘most preferred’ strategies works – even though, of course, the study also highlighted the potential for further improvements.

Overall, our study has two key messages, one for fellow energy system modellers and one for stakeholders involved in energy transition planning and relying on modelled insights.

For energy system modellers, the main message is that it is technically possible to integrate the preferences of human stakeholders directly into an MGA-based energy planning model to generate meaningful planning options with a higher chance of consensus formation. Moreover, this integration does not require any knowledge or understanding of such preferences, thanks to the automated decoding procedure that we showcase in this study. While we use a particular energy system modelling framework (Calliope) and version of MGA (our in-house SPORES method), our human-in-the-loop MGA approach is broadly applicable to any model or MGA method. This is particularly exciting because MGA was always meant for use in an interactive setting with stakeholders, but it wasn’t clear how to make this work in practice. The insights from our study should thus hopefully spur a new wave of real-world applications.

For stakeholders, our analysis further corroborates that modelling analyses that only provide a single, cost-minimising energy planning strategy for a handful of scenarios generated with no integration of stakeholder inputs should be taken with substantial caution. In fact, many alternative planning strategies exist that may support a more meaningful decision-making process, and stakeholder preferences and knowledge of the problem are the key to ensuring that the alternatives that matter the most for such a process are systematically explored.

What further research would you like to see in this area?

MGA application to energy planning has experienced huge advancements in the last couple of years only. Other research groups globally are contributing brilliant original developments, and the number of publications on the topic is growing steeply. Still, the more advancements we make, the more we open up possibilities for doing more, and the ceiling for applying MGA to energy planning questions is very high.

We advocate for more research on combining MGA methods with methods to deal with so-called ‘parametric uncertainty’, namely the uncertainty in model parameters such as weather or demand conditions or cost projections. The potential is to generate energy planning alternatives that also include information on the degree of ‘resilience’ they would ensure across broad ranges of the above uncertain conditions. There is already interesting work in this area, but there is still a large untapped potential. For instance, artificial intelligence methods, such as machine learning and meta-heuristics, may facilitate the exploration of a very broad range of uncertain conditions.

We are very interested in collaborations to drive this topic forward, especially internationally, and across disciplines. One of us (Francesco) is setting up an interdisciplinary initiative for this.

We are also further developing the modelling framework we used, Calliope, with particular attention to making it easier than ever for users to customise every aspect of the model maths – a pre-requisite for this kind of method experimentation and innovation. We are really keen to get more feedback and contributions on this approach, as well as on our effort as part of the Calliope project: Clio, a modular, transparent and reproducible system to make input data for energy system models, which is another step towards making experiments such as ours easier to run.

What made you choose PLOS Climate as a venue for your article?

We really appreciate PLOS’s editorial philosophy, which is grounded in transparent peer review and high-quality standards. PLOS Climate, in particular, felt like the ideal journal for this article for a couple of reasons. First, due to its thematic affinity with our overarching goal of contributing to the acceleration of planning decisions towards the energy transition. Secondly, because the journal had recently advocated for more transdisciplinary research on this topic and, particularly, for ‘fair and unbiased algorithms’ that could support socially viable decisions. With our paper, we believe we are answering precisely this call.

Ready to submit your own work to PLOS Climate? Follow our step-by-step guide to the submission process!