How Do Energy-Economy Models Compare? A Survey of Model Developers and Users in Canada

: Governments at all levels rely on energy-economy models to design climate policy portfo-lios. Models vary in their purposes and methodologies, yet there is limited research comparing model characteristics and identifying models suitable for speciﬁc policy questions. We conduct a web-based survey of energy-economy model users and developers ( n = 14) in Canada’s public, private, and non-proﬁt sectors, to systematically compare seventeen models against the following characteristics: Technology representations, microeconomic and macroeconomic details, policy representations, treatment of uncertainty, high-resolution spatial and temporal representations, and data transparency. We ﬁnd that for the most part, models represent technology, micro-, and macroeconomic characteristics according to the typology of bottom-up, top-down, and hybrid models. However, several modelling evolutions have emerged. To varying extents, top-down models can explicitly represent technologies and some bottom-up models incorporate microeconomic (non-ﬁnancial) characteristics. We ﬁnd that models differ in the types of policies they can simulate, sometimes underrepresenting performance regulations, government procurement, and research and development programs. All models use at least one method to explore uncertainty, rarely incorporate spatial and temporal representations, and most models lack publicly available methodological documentation. We discuss the implications of our comparative model analysis for climate policy projections and future research.


Introduction
In a pledge to limit the rise of average global temperatures beyond two degrees Celsius, 189 nations signed the Paris Agreement in 2015 [1,2]. As part of this pledge, countries developed nationally determined contributions (NDCs) that outline their strategies to reduce greenhouse gas (GHG) emissions [3]. While some NDCs are supported by climate policy, the current efforts are projected to miss the global commitment, requiring additional and more stringent national climate policy [4]. This is especially important in the developed countries, such as Canada, that have the financial capacity and responsibility for GHG mitigation [4].
Governments often rely on energy-economy models to design and assess the effects of climate policy proposals on GHG emissions and economic outcomes at the national, subnational, and municipal levels [5,6]. When choosing among policy options, governments often seek different types of modelling information that is important to their specific policy questions and/or to a broader range of government objectives [7,8]. This information may include different pathways for reducing emissions and their associated abatement costs, exploratory research approach to gather information on the key strengths and gaps in existing energy-economy models to help guide researchers and policy-makers in their model choices and interpretations of modelling results. Technology characteristics refer to the level of detail about technology-specific parameters and the representation of technological change in an energy-economy model [15,21]. A high level of technological detail aids policy-makers in making market-specific predictions of consumer and firm responses to a particular policy [6]. Technologically explicit models can consider commercial (e.g., hydroelectricity, nuclear, hybrid electric vehicles) and near-commercial (e.g., direct air capture, carbon capture and storage, hydrogen-fuel cell vehicles, first-and second-generation biofuels) technologies. These technologies can be characterized by their capital and operating costs and performance attributes (e.g., energy efficiency) [6]. When technologies are represented in adequate detail, the evidence base for developing policies is generally strengthened [34]. The representation of technological change dynamics also impacts climate policy projections [35]. Exogenous technological change is often modelled as a function of time, and is therefore represented independent from, and thus unresponsive to, policy measures [35][36][37]. It can be represented via an autonomous energy-efficiency improvement (AEEI) parameter in more aggregated models, the addition of new energy-efficient technology in more disaggregated models, or through backstop technologies (i.e., undefined processes or technologies used to limit abatement costs) [35]. In contrast, endogenous technological change is influenced by investments in research, expected prices, and policies in addition to the passage of time [35,37]. The three main methods of incorporating endogenous technological change are direct price-induced, investments in research and development, and learning by doing [35].
The second model characteristic is microeconomic representations that enables the model to account for people's non-financial (behavioral) choices that impact the effectiveness of policies [20,38]. Addressing the microeconomic characteristics in energy-economy modelling can be accomplished through two main methods: Market heterogeneity and nonfinancial decision factors used in decision-making [15]. Market heterogeneity reflects the reality that variation exists amongst consumers and producers. This variation can include factors such as income stratification or other socio-economic and behavioral parameters, that can influence technology choice [39]. The inclusion of market heterogeneity in energyeconomy models is more likely to portray a realistic outcome of technology adoption and the distributive impacts of policies [38]. Non-financial (sometimes called "intangible") costs are additional costs incurred by consumers and firms when making technology-purchasing decisions. Typically, models represent firms and consumers as 'rational agents' in their decision-making when choosing between technology alternatives [20,40]. Rational agents aim to maximize utility, have fixed and known preferences, as well as perfect information. However, real-world decision-making is much more complex and agents do not always choose the rational option that is the most cost-minimizing or utility maximizing [41]. The risk of adopting new technologies, quality variations between technologies, and the lack of information are all important non-financial considerations that, if ignored, can lead to underestimated costs of GHG reductions [14,15,20,37].
Third, macroeconomic characteristics represent a region's economy through the relationships of energy supply-demand to the structure of the economy [15,20,38]. Climate policies have the ability to cause economic benefits and costs, which can often occur at the same time [42]. These macroeconomic feedbacks have the ability to change both the structure and the growth rate of the economy, and its outputs [21]. Understanding the influence of climate policies on the economy is therefore important to policy-makers when they are considering decarbonization pathways. The two common approaches for incorporating macroeconomic feedbacks in an energy-economy model are via full or partial equilibrium methods [15]. Full equilibrium methods link the whole economic output to energy supply and demand in the macroeconomy. The effects of policy instruments, price fluctuations, and resources on every sector are examined [43]. In contrast, partial equilibrium methods only consider a part of the economy, with the focus on the energy sector. Generally, both full and partial equilibrium methods link the energy-consuming sectors to the energy-producing sectors of the economy to represent the supply and demand of energy in the economy [15]. Through price and quantity adjustments, equilibrium is achieved between the supply and demand of all energy commodities. For example, due to the variable nature of renewable energy sources, the representation of the electric grid is crucial to understanding the balance of supply and demand of electricity between economic sectors [44]. This can also be done for non-energy commodities, though it is not incorporated as often as it is for energy commodities [15].
Another important macroeconomic characteristic is the incorporation of trade and financial feedbacks [20,21]. Models can range from not including trade effects all together to assuming domestic goods are preferred over imports or that goods are traded easily and are homogenous globally [37]. Further, models can differ in how they represent the financial and monetary sectors [5]. Given that financial investments in clean energy are required to meet climate goals, knowing the origins of investments and how they affect the economy is crucial in climate policy projections [39].
Fourth, models differ in their representation of different types of climate policies. Policies can be considered individually or in combination with each other [15]. Their modeled impacts can range to include GHG reductions, abatement costs, and/or distributional impacts [7]. The ability for a model to represent multiple climate policies allows policy-makers to choose the best combination of policies for a given goal, if policy interactions are in fact carefully considered to avoid the double-counting of emission reductions [45,46]. The common climate policy instruments that are of interest in modelling include government investments and subsidies, emission pricing, performance standards, and prescriptive regulations [7]. Government investment and subsidies are measures that provide financial aid or support to lower the barrier to accessing low-carbon technologies. Research and development investments are a common example of government investments, while rebates for energy efficiency are a typical form of subsidy. Emission pricing can take the form of a direct tax on emissions, or a tradeable allowance system also known as a cap-and-trade, or a hybrid combination of the two. Both methods require economic agents to either decrease their emissions or pay a carbon price [7]. Performance standards set regulatory requirements without prescribing specific technologies and sometimes allowing for credit trading as a compliance option [7]. Low-carbon fuel standards and zero-emission vehicle mandates are examples of performance standards. Prescriptive regulations do not give firms the flexibility to decide how to meet government standards and instead require them to adopt specific technologies, with non-compliance leading to penalties [47].
Fifth, the incorporation of uncertainty is important to provide a range of possible real-life scenarios given the complexity of modelling methodologies [43]. However, policy reports and academic literature usually do not consistently summarize the treatment of uncertainty across different models. Energy-economy models can consider parametric and structural uncertainty, where the former represents the uncertainty in input parameters and the latter refers to the uncertainty in the structure of the model describing energyeconomic systems [48,49]. A sensitivity analysis is a common method to address parametric uncertainty by assessing the relative influence of each input parameter on modelling results [48,50]. A Monte Carlo analysis is another method to determine the uncertainty range of a variable in addition to determining which parameters are important to the results [43]. The effects of uncertainty in economic growth rates, discount rates, energy prices, and technology details, can be explored to provide a range of possible GHG and economic outputs under climate policy [51].
Sixth, high temporal modelling resolutions can help account for the intermittency of renewable energy supply via real-time data and time slices [27,32]. High spatial resolutions, on the other hand, are crucial in community-scale energy management where the effects of policies are not spatially uniform (e.g., urban transportation and zoning bylaws) [11].
Finally, the transparency of the methods, data, and assumptions used in an energyeconomy model is important for the results to be reproduced and trusted by relevant stakeholders [52,53]. Transparency can be increased by making the model's source code, input data, and documentation publicly available as well as using free software tools to allow for wider model accessibility [53]. Greater transparency can help third parties to reproduce and validate modelling results, and to increase the general understanding and trust in models among researchers and policy-makers.

Data Collection
We implemented a web-based survey of energy-economy model 'experts' (n = 14) in the public, private, and non-profit sectors in Canada, to collect primary data about their models as per the seven assessment characteristics in Section 3. We chose a purposive convenience sampling methodology to recruit the 'experts' [54] including model developers and model users because they tend to simulate the effects of climate policies and use the results to inform policy decisions. Purposive convenience sampling requires making judgements regarding including or excluding certain individuals to achieve specific research objectives [55]. Given that the energy modelling field includes a variety of models (i.e., energy-economy, integrated assessment, and energy systems) and our focus is on energy-economy models only, the purposive convenience sampling allowed to select the most appropriate individuals. Our population included experts from a comprehensive scoping review of energy-economy models by Rhodes et al. [15] that identified 33 model users and developers of energy-economy models in Canada. These 33 individuals represented 30 organizations consisting of 20 public organizations, 6 private companies, and 4 non-profit organizations. We identified email addresses of the 33 individuals through their organizations' open-access websites and sent electronic invitations. In our invites, we stated that different individuals from each organization could fill out the survey for different models, and different individuals who develop and/or use the same model(s) could complete the survey together, in one response, given the complex nature of some models and the small size of the energy-economy modelling field in Canada.
We administered the survey in September 2020 using the University of Victoria's SurveyMonkey platform. We employed tailored survey design methods to ensure high quality of responses while minimizing the overall survey error [56]. We pre-tested the survey questions with a select group of energy experts and economists in public (i.e., academic and government) institutions to reduce survey error. The average time for a respondent to complete the survey was 1 h and 30 min but it is skewed by six respondents who completed the survey for more than one model.
To encourage participation and establish trust, we sent out personalized survey invitations explaining the purpose of the study and its benefits to the potential participant. Before beginning the survey, all respondents were presented with consent information outlining the terms of participation, including the risks and benefits to participating as well as how their data would be used, analyzed, and stored. To begin the survey, all respondents were required to agree to these terms.
We received complete survey responses from 14 individuals, 10 model owners and 4 model users, resulting in a 42% response rate. However, these individuals reported on 19 distinct models out of the total of 21 models identified in the exhaustive scoping literature review of all Canadian energy-economy models in Rhodes et al. [15], making the survey representative of the energy-economy models in Canada. Receiving data from 14 out of 33 invited respondents was a result of (a) allowing respondents who use/develop the same model(s) to work together to submit a joint response, and (b) several model users/developers reporting on more than one model (i.e., eight individuals responded for one model, five responded for two models, and one responded for five models). Collecting data representative of models, rather than model users and developers, was important for the primary goal of the study to compare and contrast energy-economy models, rather than opinions of model users and developers. Further, the 14 respondents represent many of the top energy-economy modelers in Canada, which is a small field comprised of specialized knowledge holders. As 71% of the respondents are model owners/developers (the rest are model users), they represent a population with the most up to date and in-depth knowledge about their model designs. The individuals represented 13 organizations: Five public organizations, six private companies, and two non-profit organizations.
The survey contained a mix of closed-ended and open-ended questions in each of the sections. The questions were based on the seven assessment characteristics from Section 3 and aimed to collect objective information about model features rather than opinions or judgements of model developers and users, to minimize the social desirability bias. Specifically, the survey consisted of eight sections: (1) Information about the respondent; (2) general model information; In the first section, respondents were asked general questions about their identity, organizational affiliation, and the number of models they use or run in their line of work. The subsequent sections and questions were repeated for each model based on the indicated number of models that are run or used by the respondent. The second section asked general questions about the model including the model name and owner/operator, model description to identify its type (i.e., top-down, bottom-up, or hybrid), and other general information such as the jurisdictional application and simulation period.
In the third section on technology characteristics, respondents were asked questions about the level and dynamics of technology representation in their model. A definition for technology characteristics was provided at the beginning of the section. Respondents were asked questions about the number of represented technologies, types of included and excluded near-commercial and backstop technologies (i.e., defined in the survey as "an undefined processes used to limit abatement costs"), how technological change is represented, and how often technology parameters are updated.
The fourth section on microeconomic characteristics asked respondents questions on the model's ability to realistically represent agent behavior within the energy-economy. A definition of microeconomic characteristics was provided at the beginning of the section and other terms were defined throughout the questions. Respondents were asked about their model's ability to capture perceptions of upfront costs, lack of information, quality of technology service, risks of new technology failure, and how often microeconomic parameters are updated.
The fifth section started with a definition of macroeconomic characteristics and asked questions about the model's representation of equilibrium feedbacks, balances of energy and non-energy commodities, representation of the electric grid (due to electricity typically portrayed as its own sector), trade, the monetary and finance sectors, and how often macroeconomic parameters are updated.
The sixth section on policy representation asked questions about the model's ability to accurately represent different types of climate policies and policy mechanisms, including government investments, subsidies, a carbon tax, cap-and-trade, hybrid carbon pricing (combining carbon tax with cap-and-trade), carbon revenue recycling, performance standards, and prescriptive regulations. The final question asked how often policy parameters are updated.
The seventh section asked questions about the uncertainty method(s) used by the modelers and the parameter(s) most often explored through uncertainty analysis. Respondents were also asked to answer questions about high-resolution spatial and temporal representations as well as the transparency of the model and data. The final section asked a single open-ended question to share any other model details.
It is important to note that the survey responses represent a static form of self-reported knowledge of the contacted 14 experts. Other stakeholders using these models may frequently update them and perceive model characteristics differently. Therefore, the responses may not always capture the most up to date information and heterogeneity in perceptions of model capabilities, but rather provide a snapshot of the energy-economy modelling landscape to guide policy-makers and researchers in their model choice and interpretation of modelling results.

Data Analysis
The responses for four models (i.e., CIMS, GCAM, gTech, LEAP) with multiple survey participants were merged into one synthesized response per model through the use of the following methods: (1) For "I don't know" answers from one respondent, any alternative response from another respondent would replace "I don't know;" (2) questions that allowed multiple answers (e.g., the timeframe in which parameters are updated) were merged in the combined response; and (3) if both the developer and a user of a model submitted contradictory responses, the developer's answers were used instead of the model user's. We contacted four respondents by email to confirm information where the variation in responses could not be resolved using the aforementioned methods.
We categorized models into the three main model types discussed in Section 2bottom-up, top-down, and hybrid, in order to examine the general trends in how they incorporate each of the seven characteristics. We used model description answers from the first part of the survey to assign models to either of the three categories. Specifically, macroeconomic top-down models included models described as "input-output" and "computable general equilibrium;" technological bottom-up models were described as "simulation," "optimization," "linear programming," or "technology adoption;" and hybrid models included "hybrid" and "system dynamics" descriptors. We confirmed this model classification with past literature into these models [15,57,58], though we acknowledge that some models may not be completely attributed to just one category. The integrated assessment model EC-IAM was included in the top-down category due to its focus on the explicit representation of macroeconomic characteristics.
We received one response for a macroeconomic model The Infometrica Model (TIM), but we did not analyze it as a stand-alone model because it does not meet the definition of an energy-economy model in Section 1. TIM represents as a sub-component of the E3MC model, composed of TIM and ENERGY 2020; therefore, we analyzed TIM as part of the entire E3MC model. Similarly, the Integrated Electricity Supply and Demand (IESD) model was not analyzed separately-it is used in conjunction with gTech to provide increased detail in the electricity sector so we considered its characteristics in the analysis of gTech [9]. As a result of merging the models, we analyzed the survey data for 17 models, not 19 as identified by respondents.
We analyzed responses using descriptive statistics to calculate frequencies in multiplechoice questions. The open-ended responses were manually scanned and analyzed to identify common themes to support and further explain multiple-choice responses. When respondents provided answers that did not align with the definition of a question, we reassigned the response to a more suitable section of the analysis. Where a model was described by only one respondent, we treated the "I don't know" responses as missing values. We used matrix tables (see Appendix B) to summarize our assessments for each model against the seven characteristics from Section 3. Some of the details from the openended responses were also included in the assessment matrices to enhance the comparative analysis of models.

Results
The surveyed energy-economy models are primarily owned by public organizations (8 models), followed by private companies (7 models), and non-for-profit groups (2 models) in Canada (Table 1). These models use a diverse set of analytical approaches. A hybrid approach was the most common approach, employed by eight models: CIMS, CIMS-Urban, E3MC, ENERGY 2020, Energy Policy Simulator, GCAM, gTech, and NATEM-TIMES. A bottom-up approach was the next most common approach, employed by CanESS, CityInSight, LEAP, MEDEE, MESSAGE, and REPAC. A top-down approach was found only in three models: EC-PRO, EC-MSMR, and EC-IAM. Models' simulation periods range from a sub-annual time period to every 10 years, with a select group of models having the ability to simulate multiple time periods (i.e., GCAM, LEAP, MESSAGE, NATEM-TIMES). The most common simulation period is every year or every 5 years. Fewer models use a simulation period of every 10 years (two models) or more often than at an annual basis (three models). Models vary in their simulation timeframes with almost three-quarters of models running to 2050. A smaller portion of models run to 2030 and 2100. The respondents for six models-CIMS, CIMS-Urban, CityInSight, EC-MSMR, gTech, and LEAP, indicated they can be run to multiple dates in the future.
There is a wide range of jurisdictional applications of models, from cities all the way up to an international level, depending on models' specific objectives. Three-quarters of models can be used in multiple jurisdictions with the provincial/territorial level being the most common application (observed in twelve models), followed by a national application (nine models). Fewer models can be applied to regional and municipal scales as well as the broad international jurisdiction (five to six models).
About half of the models represent multiple economic sectors at once ranging from buildings, to waste, transport, industry, electricity, and land use. The least represented economic sector is land use with six models excluding this sector. A few models include additional sectors, such as agriculture (i.e., CIMS, MEDEE) and the forest sector (i.e., NATEM-TIMES). The transportation sector is the only sector to be included in all surveyed models.
The following sub-sections describe the surveyed models against the seven characteristics from Section 3.

Treatment of Technologies and Technological Change
All of the surveyed models explicitly represent technologies (Table A1 in Appendix B). Seven models explicitly represent both backstop and near-commercial technologies: GCAM, CIMS, EC-IAM, EC-PRO, EC-MSMR, Energy Policy Simulator, and MESSAGE. Over half of the models explicitly represent technologies in certain sectors, with the rest explicitly representing technologies in all sectors. The number of represented technologies ranges from five (i.e., REPAC) to thousands (e.g., CIMS, NATEM-TIMES). Backstop technologies exist in only seven models and include carbon capture and storage (i.e., CIMS, Energy Policy Simulator, GCAM), direct air capture (i.e., CIMS, Energy Policy Simulator), and biomass/bioliquids (i.e., GCAM).
With the exception of MEDEE, all of the models represent at least one near-commercial technology. Near-commercial technologies are technologies that are used in a limited way and require some further development to achieve widespread adoption. Of the nearcommercial technologies surveyed, hydrogen fuel cell vehicles are the most common, being represented in fifteen models. Other near-commercial technologies that were represented in the surveyed models include carbon capture and storage, electrolysis-based hydrogen production, and direct air capture. Finally, the majority of models (i.e., 14 models) represent first-and/or second-generation biofuels, with 10 models representing both categories of biofuels.
Almost all models include representations of technological change, declining capital costs, and annual operating costs. Models most commonly represent technological change using both exogenous and endogenous methods depending on technology types ( Figure 1). All three model types-bottom-up (i.e., LEAP, MEDEE), top-down (i.e., EC-PRO, EC-MSMR), and hybrid (i.e., E3MC, ENERGY 2020, Energy Policy Simulator, NATEM-TIMES), use this method. The models that represent technological change exogenously consist of mostly bottom-up (i.e., CanESS, CityInSight) and top-down (i.e., EC-IAM) approaches, while models that represented it endogenously mostly use a hybrid approach (i.e., CIMS, CIMS-Urban, gTech). The large majority of models (i.e., 16 models) include declining capital costs in their representation of technologies. Only one model, MEDEE, does not include declining capital cost functions. Finally, fuel and maintenance costs are included as annual operating costs in almost all surveyed models.

Representation of Microeconomic Characteristics
With the exception of CanESS and CityInSight, all models include some level of microeconomic characteristics (Table A2 in Appendix B). Only a few models represent a full range of microeconomic characteristics including upfront costs, lack of information, varying quality of technology service, risk of new technology failure, and other non-financial characteristics-these tend to be hybrid models including CIMS, CIMS-Urban, E3MC, and gTech ( Figure 2). Models address market heterogeneity through several methods including choice methods (i.e., E3MC, ENERGY 2020, Energy Policy Simulator, REPAC) and behavioral parameters (i.e., CIMS, CIMS-Urban). The models that do not address market heterogeneity are bottom-up models. The same models that address market heterogeneity also include at least one nonfinancial cost parameter. The most common parameters are upfront costs of technologies and associated discount rates, with fourteen models including these parameters. Of those fourteen models, the majority represent this parameter by disaggregating technologies and representing the non-financial upfront costs of each technology.
Characteristics that represent the lack of technology information, varying quality of technology service, and risk of technology failure are included less frequently than nonfinancial upfront costs of technologies. The models that include these characteristics are often of a hybrid nature (e.g., CIMS, E3MC, gTech). Just over half of models acknowledge that firms and consumers do not have complete information about all technologies with the vast majority of those models representing this characteristic explicitly. The quality of technology service is addressed in eight models and the risk of new technology failure in six. Almost half of the models contain additional non-financial decision-making characteristics including technology availability (i.e., REPAC) and externality values of pollution (i.e., LEAP).

Representation of Macroeconomic Characteristics
The majority of surveyed models (i.e., 12 models) incorporate macroeconomic characteristics to some degree to represent the structural systematic relationships of a region's economy (Table A3 in Appendix B). These models are almost all hybrid or top-down models due to their parameterization through historical macroeconomic data ( Figure 3). The models EC-IAM, EC-PRO, EC-MSMR, gTech, and NATEM-TIMES include all the surveyed macroeconomic characteristics of full and/or partial equilibrium methods, supply-demand balance both energy and non-energy commodities and represent the electric grid. Five bottom-up models that do not represent the macroeconomy include CanESS, CIMS-Urban, CityInSight, MEDEE, and REPAC. Of the twelve models that incorporate macroeconomic characteristics, seven models use partial equilibrium methods and five full equilibrium methods. Full-equilibrium models are typically more of a top-down (i.e., EC-IAM, EC-PRO, EC-MSMR) or hybrid (i.e., gTech) nature. Most models represent the supply-demand balance of energy commodities through price-quantity adjustments, with fewer models balancing non-energy commodities. Of these models, six (i.e., EC-IAM, EC-PRO, EC-MSMR, GCAM, gTech, NATEM-TIMES) balance both energy and non-energy commodities and two (i.e., Energy Policy Simulator, MESSAGE) partially balance both energy and non-energy commodities. The six models that balance both energy and non-energy commodities are top-down (i.e., EC-PRO, EC-IAM, EC-MSMR) or hybrid models (i.e., GCAM, gTech, NATEM-TIMES). E3MC is the only model that balances energy commodities, but not non-energy commodities. Two-thirds of the surveyed models include a representation of regional electric grids (i.e., E3MC, EC-IAM, EC-PRO, EC-MSMR, ENERGY 2020, gTech, LEAP, NATEM-TIMES). These are mostly full-equilibrium top-down or hybrid models.
The incorporation of trade effects is found in almost all the models that represent macroeconomic characteristics, while the representation of the monetary and financial sectors was rare. gTech is the only model found to represent inter-regional and international trade as well as the monetary and finance sectors. Of the nine models that incorporate trade effects, inter-regional trade is represented endogenously in eight models and exogenously in one model, LEAP. Similarly, international trade is represented by more models endogenously than exogenously (i.e., LEAP, NATEM-TIMES). The Energy Policy Simulator is the only model, among those that represent the macroeconomy, to not include trade effects. Overall, the monetary and finance sectors have little representation in the models surveyed, with only two hybrid models, Energy Policy Simulator and gTech, incorporating these sectors.

Representation of Policies and Policy Interactions
All models except for CanESS and CityInSight are able to represent at least one policy type, with almost all hybrid or top-down models having the ability to represent all tested policy types (i.e., EC-IAM, EC-PRO, EC-MSMR, Energy Policy Simulator, gTech, LEAP, and NATEM-TIMES) ( Table A4 in Appendix B). The most represented policy types across all models are the carbon tax and prescriptive regulations, found in 15 models. This is followed by subsidies (14 models), performance standards (13 models), cap-and-trade as well as hybrid carbon pricing (13 models), recycling carbon revenue (12 models), government procurement (10 models), and research and development (7 models) ( Figure 4). Most models that explicitly represent carbon pricing (i.e., carbon tax, cap-and-trade and a combination of thereof) are hybrid or top-down. Of the models that represent a carbon tax, all but one model (MEDEE) simulates this policy explicitly. These models can also simulate the recycling of carbon revenue. Cap-and-trade is represented by fewer models than a carbon tax, with most models simulating it explicitly and two models implicitly (i.e., Energy Policy Simulator, MEDEE). The same models can simulate hybrid carbon pricing due to their ability to represent cap-and-trade policy mechanisms.
The representation of prescriptive regulations mirrors the results for the carbon tax with most models representing this policy type and MEDEE being the only model representing prescriptive regulations implicitly. The majority of models represent subsidies and do so explicitly. Performance standard policies are represented 13 models, with 12 of those models representing this policy explicitly. The models that do not represent either prescriptive regulations or performance standards are bottom-up models. Models vary in their representation of government procurement and investment with half of them representing it explicitly and one model implicitly (i.e., Energy Policy Simulator). Investment in research and development is the least-represented policy, with only seven models having the ability to simulate it. Five of these models simulate it explicitly (i.e., EC-IAM, EC-PRO, EC-MSMR, LEAP, NATEM-TIMES) and fall under the top-down or hybrid model category-these models are mostly used by Canada's federal government.
Regardless of model type, the majority of models (i.e., 14 models) explicitly consider the interactions between multiple climate policies. However, not all models that consider these interactions also avoid double-counting emissions from multiple climate policies. The model E3MC implicitly represents both of these sub-characteristics.

Other Characteristics and Data Management
Uncertainty is explored in all models with sensitivity analysis being used in all surveyed models, a Monte Carlo analysis in seven models, and other methods being employed in four models (Table A5 in Appendix B). Just over half of model users employ two or more methods to explore uncertainty. The use of a Monte Carlo analysis and/or other methods to explore uncertainty is most often found in hybrid models. Economic growth and energy prices are the most common characteristics explored through uncertainty analyses. The model NATEM-TIMES is the only model found to not explore either of these characteristics through uncertainty analysis. In addition, many models explored uncertainty in other parameters including technology-related parameters (i.e., CanESS, CityInSight, E3MC, gTech, NATEM-TIMES, REPAC) and intangible costs (i.e., CIMS, CIMS-Urban).
High-resolution spatial and/or temporal representations are included in only eight models, with more models including high-resolution temporal representations compared to high-resolution spatial representations. The four models that include high-resolution spatial characteristics represent explicit geographic blocks (i.e., CIMS-Urban, CityInSight, LEAP) and water-related infrastructure (i.e., MESSAGE). Most of the models that include high-resolution spatial or temporal representations are bottom-up.
In terms of data management, most models (i.e., 11 models) are not freely available for public use and do not have open-source code-these models are most often run by governments or private organizations ( Figure 5). However, models are more likely to be transparent in their use of open-source data inputs and have at least some of their modelling equations and assumptions publicly accessible. Of the six models that are freely available for public use two are from academic institutions (i.e., CIMS, CIMS-Urban) and four from non-profit organizations (i.e., Energy Policy Simulator, GCAM, LEAP, MESSAGE). Only five models (i.e., CIMS, CIMS-Urban, GCAM, LEAP, MESSAGE) are both freely available and use open-source code. In contrast, 13 models include at least some open-source data with certain inputs to the model being from publicly available sources such as Statistics Canada, Environment and Climate Change Canada, and Natural Resources Canada. Most models include a mixture of data from publicly available sources and confidential ones. More than half of the models have at least some of their modelling equations and assumptions documented in a publicly accessible manner. While some models do not have the equations and/or assumptions publicly available currently, several respondents indicated they are in the process of or plan to open source this information (i.e., CityInSight, NATEM-TIMES). Related to data management, we asked all respondents to indicate how often each of the core model characteristics is updated (Table A6 in Appendix B). For each characteristic, the most common update timeframe is every year, followed by every 2-5 years for all characteristics except for macroeconomic details. Some models such as GCAM and gTech have more than one update timeframe due to different model end-users (e.g., clients and/or policy-makers) choosing to update at different times.

Conclusions
Energy-economy models are important tools that aid governments at all levels in developing climate policies by assessing their effects on GHG emissions and economic outcomes [12]. Energy-economy models provide policy-makers with different types of information relevant to their specific policy questions ranging from the most effective pathways to reduce emissions to their associated abatement costs, distributional impacts, changes to jobs and economic activity levels [9][10][11]. This study represents the first attempt in a Canadian context to compare energy-economy models against seven assessment characteristics to help policy-makers answer these important questions.
Our web-based survey of energy-economy model developers and users in Canada identifies 17 distinct models, representative of the energy-economy models used across public, private, and non-profit sectors [15]. Most of these models can be described as hybrid (i.e., CIMS, CIMS-Urban, E3MC, ENERGY 2020, Energy Policy Simulator, GCAM, gTech, and NATEM-TIMES), followed by bottom-up (i.e., CanESS, CityInSight, LEAP, MEDEE, MESSAGE, REPAC), and top-down models (i.e., EC-PRO, EC-MSMR, EC-IAM). While the study focuses on models that are used in Canada, most energy-economy modelling methodologies are universal and some of the reviewed models (i.e., Energy Policy Simulator, GCAM, LEAP, MESSAGE, and NATEM) are in fact developed and also used outside of Canada, making the study's results applicable to other jurisdictions [12,20,22]. We compare all models against seven assessment characteristics generally considered important in literature for projecting climate policy effects on GHG emissions and economic outcomes.
These characteristics include technology representations, microeconomic and macroeconomic details, policy representations, treatment of uncertainty, high-resolution spatial and temporal representations, and data transparency.
For the most part, models represent technology, micro-, and macroeconomic characteristics according to the typology of top-down, bottom-up, and hybrid models, validating past modelling reviews in Canada [6,21] and worldwide [12,20,22]. In line with past literature, the surveyed top-down (e.g., EC-PRO, EC-MSMR) and hybrid (e.g., NATEM-TIMES, gTech) models include microeconomic (behavioral) and macroeconomic characteristics [6,20]. Bottom-up models (e.g., REPAC, MEDEE) explicitly represent technological characteristics, while excluding or poorly representing macroeconomic details [21,24]. However, the survey data suggest that models have evolved in several ways due to a growing variety and complexity of different policy tools used in climate policy mixes [59]. Some top-down models have evolved to include explicit representations of technologies in order to model technology-specific policies, while some bottom-up models have started to incorporate market heterogeneity and behavioral preferences to produce more realistic simulations.
Our study expands the three-characteristic based model typology [20] to include additional four characteristics of policy representations, treatment of uncertainty, highresolution spatial and temporal representations, and data transparency. We find that while bottom-up models can simulate a carbon tax and prescriptive regulations, they do not generally represent macroeconomic policy mechanisms, such as the recycling of carbon revenues, adequately. Model users do address uncertainty, though often only through a sensitivity analysis. Bottom-up models are more likely than other models to include high-resolution spatial and/or temporal representations due to their explicit technological characteristics. In contrast, most hybrid and all top-down models can simulate the tested policy types due to a combination of explicit technology representations (to varying degrees in top-down models) and the incorporation of macroeconomic feedbacks [23]. Similar to bottom-up models, top-down models generally address uncertainty through a sensitivity analysis, while hybrid models almost always use other methods in combination with the sensitivity analysis, including Monte Carlo analysis. Top-down models lack the inclusion of high-resolution spatial and temporal representations due to their aggregated approach, while a small number of hybrid models include these representations. Differences in data transparency could not be attributed to model type, but rather to the organization type that uses and/or develops the model.

Limitations and Contributions
There are several limitations to this analysis. First, there are potential biases that might have impacted survey responses. Because many respondents are model developers, they have a vested interest in promoting their model(s) and answering the survey questions in a way that reflects positively on their model and its assumptions. All respondents might have also been influenced by a social desirability bias whereby the capacity of the model or degree that characteristics are represented may have been overemphasized (e.g., several answers included "yes" and "explicitly" but failed to explain how exactly a characteristic was represented). In addition, the findings might have been affected by the varying levels of knowledge between model users and developers who completed the survey. For example, model users were more likely to choose the answer "I don't know" or not answer an open-ended portion of the question, and sometimes provided a conflicting answer about the same model that was described by a model developer. Therefore, model assessments should be interpreted as a static form of self-reported knowledge from model users and developers in Canada, consistent with the goal of the study to collect and review the most recent and at times publicly unavailable information about energy-economy models. Future research could conduct external validations of models against each assessment characteristic.
Second, this study used a convenience sampling method to recruit energy-economy model 'experts' in Canada as identified by Rhodes et al. [15] in their scoping literature review. This methodology might have limited the sample size potentially affecting the representation of the full model landscape in Canada. Finally, we chose the seven assessment characteristics using past literature on their general importance for modelling economic and GHG emission impacts of climate policy [15,20,32,44]. We did not test a specific conceptual framework and therefore did not conduct inferential analyses to suggest that some of these characteristics are more or less significant in influencing the quality of climate policy projections. Future research can employ a standard set of assumptions and climate policy scenarios to run different models and compare differences in results, in order to identify the relative importance of the seven assessment characteristics.
Despite these limitations, this study offers important contributions to the existing body of modelling literature and climate policy-making. The comprehensive model assessment matrices in Appendix B help update past modelling reviews in and outside of Canada [12,[20][21][22]] and provide novel model information that is not otherwise publicly available, enabling more systematic comparisons of model strengths and gaps. Researchers and policy-makers can refer to the matrix tables when choosing a suitable model for their specific research or policy question. No model is ideal for every policy question, but rather certain models or model types are better suited to answer certain questions than others. All surveyed models seem to explicitly represent some technologies making them suitable to answer technology-specific policy questions. The high-resolution temporal representations in many bottom-up models (e.g., CanESS, LEAP) can further represent the fluctuations in renewable energy technologies caused by changing weather conditions. The evolution of explicit technology representations in all model types could reflect the fact that technologyspecific policies such as subsidies and regulations are often preferred by policy-makers due to their higher political acceptability [14]. Almost all models are able to simulate carbon pricing; however, hybrid or top-down models (e.g., gTech or EC-PRO) would be more suited to represent this policy type due to their incorporation of macroeconomic feedbacks and the ability to represent carbon revenue recycling. All top-down and hybrid models (e.g., EC-IAM, GCAM, NATEM-TIMES) can simulate a variety of prescriptive regulations, performance standards, and subsidies, due to their incorporation of strengths of bottom-up and top-down methodologies. When developing climate policies at the municipal scale, using models that incorporate high-resolution spatial representation (e.g., CIMS-Urban, CityInSight) can help account for the non-spatial uniformity of land-use policies. Other implications of each model characteristic for climate policy projections are discussed in the section below.

Implications for Climate Policy Projections
Our study has several practical implications for researchers and policy-makers. First, all models explicitly represent technologies and technological change. This explicit representation strengthens the evidence base of policies [34] and allows them to make better predictions of future energy demands [60]. While all models represent technological change, the diverse methods of modelling technological change can produce different results. Models that use endogenous technological change can respond to socio-economic factors in addition to the passage of time. Therefore, the projected cost of abatement in these models can be considerably lower than projections from models that use exogenous technological change [24].
Second, most models represent at least one microeconomic characteristic such as market heterogeneity and non-financial costs, which helps produce more behaviorally realistic projections [6]. However, the most frequent and often the only microeconomic characteristic is upfront costs of technologies. The lack of other characteristics (e.g., imperfect information, quality of technology service, and risk of technology failure) in modelling methodologies implies that many real-life behaviors are likely to be ignored in climate policy projections resulting in underestimated mitigation costs and overly optimistic GHG reductions [14,34].
Third, while some top-down and hybrid models may not represent microeconomic characteristics, their inclusion of macroeconomic characteristics allows policy-makers to understand the economic costs and benefits of a policy, including changes to energy prices, economic activity, gross domestic product, job composition, consumption, and investments [42]. These are important impacts to consider given that all governments face multiple and sometimes conflicting socio-economic and environmental objectives [61]. While most models with macroeconomic feedbacks represent trade effects, they lack representation of the financial and monetary sectors potentially ignoring differences in costs of capital for low-carbon technology in different regions [5].
Fourth, some of the most popular policies used by governments, such as performance standards, government procurement, and research and development, are not represented in all the models, decreasing their usefulness for policy-making. These underrepresented policies tend to receive high political support due to implicit abatement costs and associated higher chances of long-term implementation [62,63]. Future modelling improvements should aim to incorporate most policy types as jurisdictions strengthen their climate policy portfolios.
While all models consider the interactions between policies, they do not all avoid double-counting of emission reductions caused by multiple climate policies, leading to potentially overstated GHG reductions. Future research could explore the extent to which different models double-count GHG reductions and assess methodological solutions to overcome the issue in different model types.
Fifth, the incorporation of uncertainty analysis methods in all models allows policymakers compare a range of modelling projections contributing to more credible and politically acceptable policy decisions [33]. Uncertainty analyses can be particularly useful to policy-makers when estimating the world's transition out of the COVID-19 pandemic.
Sixth, high-resolution spatial and/or temporal representations are rarely incorporated in the surveyed models. This lack of spatial representations may impact the effectiveness of local-scale climate policy development regarding land-use issues, while the lack of temporal representations can affect projections concerned with electricity and renewable energy demand.
Finally, the observed lack of transparency in model data and assumptions/equations is one of the biggest concerns deserving academic and policy-making attention. Nontransparent models can raise credibility questions, especially if their results are used to inform public policy decisions. Without transparent and open access data, model results cannot be effectively reproduced and the implications of a policy scenario may not be fully understood and trusted [27]. The movement towards more transparent and open access data can advance the accuracy of modelling results and lead to more informed and effective climate policy decisions [53]. One example of such movement in Canada is the Energy Modelling Initiative, which aims to provide open access tools and bridge the gap between model developers and users, similar to an Energy Modelling Forum in the United States [8,64]. Future research could explore governance mechanisms to amplify and sustain modelling transparency efforts.

Data Availability Statement:
The data presented in this study are openly available in MendeleyData at doi:10.17632/mxctpp43n9.1 [65].

Acknowledgments:
We gratefully acknowledge the support of our SSHRC grant collaborator, Glasha Obrekht (Environment and Climate Change Canada, Government of Canada), who contributed to the development of the research idea and survey design.

Conflicts of Interest:
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

1.
Your Information What is the type of organization(s) you are associated with?
Other. Please specify____________________ 1.3. How many energy-economy models (i.e., a model that examines the linkages between all energy sectors and the economy of a region) do you use/run in your line of work? If you use more than one model you will be asked to fill out the survey for each model.

Microeconomic Characteristics
Microeconomic characteristics refers to the ability of a model to realistically represent agent behavior within the energy-economy, including the heterogeneity of consumer preferences, and non-financial decision factors. 4.1. Is market heterogeneity (i.e., differences in how different consumers and producers make choices between technologies) addressed in the model?

Macroeconomic Characteristics
Macroeconomic characteristics refers to the ability of a model to represent the structural systematic relationships of a region's economy. This includes feedbacks such as trade, financing, and links between energy supply-demand and the economy's structure and output.
5.1. Does the model incorporate macroeconomic characteristics (i.e., represents the structural systematic relationships of a region's economy)?
• Yes • No (If selected the survey will skip to question 6.1) • I don't know/I prefer not to say If you answered yes, please explain how macroeconomic characteristics is incorporated in the model. ______________________ 5.2. Does the model use general equilibrium methods to link economic feedbacks in a full equilibrium framework? A full equilibrium framework estimates aggregate relationships between the relative costs and markets shares of energy and other inputs to the economy, and links these estimates to sectoral and economic output.
I don't know/I prefer not to say If you answered yes, please explain how the model uses full equilibrium methods to link economic feedbacks in a full equilibrium framework. ______________________ 5.3. Does the model use partial equilibrium methods to partially link major equilibrium feedbacks? Partial equilibrium methods do not simulate the entire economy, but instead only considers a specific part of the market or sector where the economic equilibrium is determined independently from the prices, supply and demand from other markets.
I don't know/I prefer not to say • Not applicable (model uses full equilibrium methods) If you answered yes, please explain how the model uses partial equilibrium methods to partially link major equilibrium feedbacks. ______________________ 5.4. Are energy commodities supply-demand balanced through price-quantity adjustments? Examples of energy commodities include electricity, refined petroleum products, and/or natural gas.
• Yes • Partially, via own-price elasticities • No • I don't know/I prefer not to say 5.5. Are non-energy commodities supply-demand balanced through price-quantity adjustments? Examples of non-energy commodities include agriculture, metal, and/or livestock.
• Yes • Partially, via own-price elasticities • No • I don't know/I prefer not to say 5.6. Is the electric grid represented in the model (e.g., hourly supply and demand and/or voltage and frequency of the electricity transmission and distribution system by province or other region)

know/I prefer not to say
If you answered yes, please explain how the electric grid is represented in the model. ______________________ 5.7. Is trade (i.e., the flow of goods and services between regions) represented in the model?
No (If selected the survey will skip to question 5.10) • I don't know/I prefer not to say 5.8. How is inter-regional trade treated within the model bounds?
Inter-regional trade is not represented • I don't know/I prefer not to say If you answered endogenously, exogenously, or other, please explain how interregional trade is treated within the model. ______________________ 5.9. How is international trade treated within the model bounds?