A Decision Support System for Public Funding of Experimental Development in Energy Research

New energy technologies may fail to make the transition to the market once research funding has ended due to a lack of private engagement to conclude their development. Extending public funding to cover such experimental developments could be one way to improve this transition. However, identifying promising research and development (R&D) proposals for this purpose is a difficult task for the following reasons: Close-to-market implementations regularly require substantial resources while public budgets are limited; the allocation of public funds needs to be fair, open, and documented; the evaluation is complex and subject to public sector regulations for public engagement in R&D funding. This calls for a rigorous evaluation process. This paper proposes an operational three-staged decision support system (DSS) to assist decision-makers in public funding institutions in the ex-ante evaluation of R&D proposals for large-scale close-to-market projects in energy research. The system was developed based on a review of literature and related approaches from practice combined with a series of workshops with practitioners from German public funding institutions. The results confirm that the decision-making process is a complex one that is not limited to simply scoring R&D proposals. Decision-makers also have to deal with various additional issues such as determining the state of technological development, verifying market failures or considering existing funding portfolios. The DSS that is suggested in this paper is unique in the sense that it goes beyond mere multi-criteria aggregation procedures and addresses these issues as well to help guide decision-makers in public institutions through the evaluation process.


Introduction
In view of the 1.5 degree Paris Agreement [1] and the European target to reduce greenhouse gas emissions by at least 80% by 2050 when compared to 1990 levels [2], the successful and the widespread utilization of efficient and climate friendly new energy technologies is becoming more important.Industrialized countries are striving to facilitate and accelerate the transition to higher technology readiness levels (TRL, e.g., [3]) by using public funds to spend on research activities with a defined scope, budget, and duration (e.g., [4]).The member governments of the International Energy Agency (IEA) alone, for example, spent USD 16.6 billion on energy research, development, and demonstration in 2016 [5].The theory is that, apart from private benefits, the publicly funded research and development (R&D) activities of firms also yield positive spillovers for the economy as a whole.Without the additional funding, individual firms would not make the level of investment on their own, which is considered optimal from a societal perspective (e.g., [6,7]).In this sense, state intervention here is to overcome market failures due to the public good character of knowledge and to reduce systemic failures that are negatively affecting the diffusion of knowledge and/or technological learning [8].
Traditionally, public funding for energy research projects tends to focus on TRL 1 to 6 or 7.The lowest level, TRL 1, indicates that the basic principles of a technology have been observed, while the highest level, TRL 9, underlines the availability of proven systems on the market.Though not without ambiguity [9], technologies beyond TRL 6 can be attributed to the group of 'experimental development', as defined in European legislation relating to R&D [10].Such developments run the risk of being stuck in the 'valley of death' [11,12] after traditional research funding has ended.More specifically, they fail to be adopted by the market or this process simply takes long (Figure 1).
Energies 2018, 11, x FOR PEER REVIEW 2 of 18 on their own, which is considered optimal from a societal perspective (e.g., [6,7]).In this sense, state intervention here is to overcome market failures due to the public good character of knowledge and to reduce systemic failures that are negatively affecting the diffusion of knowledge and/or technological learning [8].
Traditionally, public funding for energy research projects tends to focus on TRL 1 to 6 or 7.The lowest level, TRL 1, indicates that the basic principles of a technology have been observed, while the highest level, TRL 9, underlines the availability of proven systems on the market.Though not without ambiguity [9], technologies beyond TRL 6 can be attributed to the group of 'experimental development', as defined in European legislation relating to R&D [10].Such developments run the risk of being stuck in the 'valley of death' [11,12] after traditional research funding has ended.More specifically, they fail to be adopted by the market or this process simply takes long (Figure 1).Extending public research funding on energy projects beyond its traditional focus could be a means to bridge the valley of death in energy research.Identifying promising proposals for public R&D funding is a very important part of the funding process [13], but this can also be very challenging for decision-makers in public institutions:


Full industry-scale implementations require considerable resources for scale-ups and real-world adaptations and involve major risks (e.g., [14,15]).Given that public budgets are limited, funds can only be allocated to the most promising projects and these need to be identified among numerous funding opportunities. Funding agencies have some leeway in how they spend their money (e.g., [16]).The use of public resources is governed by political factors and is subject to the influence of stakeholders in R&D, e.g., from academia, industry, policy making, or the general population.Yet, the allocation of funds is expected to be fair and open [17].An increasing pressure for accountability has been observed, which requires public institutions to present the results of their programs and to justify their existence (e.g., [13]).There is therefore a need to document the coherency of funding decisions transparently and comprehensively.


It has been pointed out that public sector financers have a different perspective on R&D than private investors [11].This concerns the strategic and long-term character of public funding, which intends to encourage private investments in potentially attractive areas.As a consequence, traditional financial justification techniques are seen as probably unsuitable [17].Furthermore, research project evaluation itself is considered complex since it usually involves the consideration of multiple criteria and the views of multiple experts [18].This makes it necessary Extending public research funding on energy projects beyond its traditional focus could be a means to bridge the valley of death in energy research.Identifying promising proposals for public R&D funding is a very important part of the funding process [13], but this can also be very challenging for decision-makers in public institutions:

•
Full industry-scale implementations require considerable resources for scale-ups and real-world adaptations and involve major risks (e.g., [14,15]).Given that public budgets are limited, funds can only be allocated to the most promising projects and these need to be identified among numerous funding opportunities.

•
Funding agencies have some leeway in how they spend their money (e.g., [16]).The use of public resources is governed by political factors and is subject to the influence of stakeholders in R&D, e.g., from academia, industry, policy making, or the general population.Yet, the allocation of funds is expected to be fair and open [17].An increasing pressure for accountability has been observed, which requires public institutions to present the results of their programs and to justify their existence (e.g., [13]).There is therefore a need to document the coherency of funding decisions transparently and comprehensively.

•
It has been pointed out that public sector financers have a different perspective on R&D than private investors [11].This concerns the strategic and long-term character of public funding, which intends to encourage private investments in potentially attractive areas.As a consequence, traditional financial justification techniques are seen as probably unsuitable [17].Furthermore, research project evaluation itself is considered complex since it usually involves the consideration of multiple criteria and the views of multiple experts [18].This makes it necessary to judge proposals from different angles.However, obtaining the required information can also be difficult.

•
Public funding is subject to legal frameworks, which also define the limits for state interventions.
It is therefore necessary to consider these.
To help overcome these challenges and to facilitate the evaluation process, this paper proposes an operational decision support system (DSS) for evaluating the proposals of large-scale projects in experimental energy research as public funding opportunities.To the best of our knowledge, no such DSS is described in the literature at present.
The remainder of this paper presents a brief description of the background and the methodological approach to develop the DSS (Section 2).This is followed by a review of the relevant literature and the practical applications of public R&D funding (Section 3), and a discussion of the requirements and methods for the DSS (Section 4).The proposed DSS is then described (Section 5), followed by a discussion (Section 6) and the final conclusions (Section 7).

Proposal Evaluation Process in Germany's Applied Energy Research Funding
The DSS put forward in this paper was elaborated within a research project on future energy technologies funded by the German Federal Ministry for Economic Affairs and Energy (BMWi).BMWi is the major funding institution for applied energy research in Germany.Its activities are governed by the 'Sixth Energy Research Programme of the Federal Government' of 2011 [19].A forthcoming seventh program is being prepared [20].As with any national research funding activities in the European Union (EU), funding is subject to the European rules for public aid [10].
The current energy research program has three objectives: to help achieve the German government's targets relating to the energy sector and climate policy; to foster the leadership of German companies in modern energy technologies; and, to secure and enhance the availability of technological options.To this end, funding is granted as both institutional and project funding [19].
Figure 2 provides an overview of the generic evaluation process for selecting project proposals for energy research funding in Germany.Several groups are involved in this process: applicants submitting proposals for projects, funding agencies/bodies, the Ministry, and, where applicable, additional internal or external evaluators.to judge proposals from different angles.However, obtaining the required information can also be difficult.


Public funding is subject to legal frameworks, which also define the limits for state interventions.
It is therefore necessary to consider these.
To help overcome these challenges and to facilitate the evaluation process, this paper proposes an operational decision support system (DSS) for evaluating the proposals of large-scale projects in experimental energy research as public funding opportunities.To the best of our knowledge, no such DSS is described in the literature at present.
The remainder of this paper presents a brief description of the background and the methodological approach to develop the DSS (Section 2).This is followed by a review of the relevant literature and the practical applications of public R&D funding (Section 3), and a discussion of the requirements and methods for the DSS (Section 4).The proposed DSS is then described (Section 5), followed by a discussion (Section 6) and the final conclusions (Section 7).

Proposal Evaluation Process in Germany's Applied Energy Research Funding
The DSS put forward in this paper was elaborated within a research project on future energy technologies funded by the German Federal Ministry for Economic Affairs and Energy (BMWi).BMWi is the major funding institution for applied energy research in Germany.Its activities are governed by the 'Sixth Energy Research Programme of the Federal Government' of 2011 [19].A forthcoming seventh program is being prepared [20].As with any national research funding activities in the European Union (EU), funding is subject to the European rules for public aid [10].
The current energy research program has three objectives: to help achieve the German government's targets relating to the energy sector and climate policy; to foster the leadership of German companies in modern energy technologies; and, to secure and enhance the availability of technological options.To this end, funding is granted as both institutional and project funding [19].
Figure 2 provides an overview of the generic evaluation process for selecting project proposals for energy research funding in Germany.Several groups are involved in this process: applicants submitting proposals for projects, funding agencies/bodies, the Ministry, and, where applicable, additional internal or external evaluators.

Figure 2.
Administrative process for a successful proposal in Germany's public energy research funding with amendments based on [19].
German energy research funding traditionally focuses on basic and applied research.In order to support the German Energiewende (e.g., [21]) and to contribute to the national long-term energy and climate goals [22], the suggestion has been made to allocate funds to close-to-market experimental research as a potential additional option.German energy research funding traditionally focuses on basic and applied research.In order to support the German Energiewende (e.g., [21]) and to contribute to the national long-term energy and climate goals [22], the suggestion has been made to allocate funds to close-to-market experimental research as a potential additional option.

Methodological Approach to Developing the DSS
The DSS that is described in this paper was developed in four phases with regard to the potential coverage of close-to-market research.The impetus for the first problem definition phase was an early exchange with BMWi to pinpoint the challenges that are underlining the need for a DSS.Based on this exchange, a draft outline was prepared, and a formalized development process started.
The second phase of information collection followed, which concentrated on reviewing existing methods of R&D proposal evaluation that is related to energy and climate research.For this purpose, selected programs from different contexts and countries were analyzed with a focus on their evaluation criteria and mechanisms.Furthermore, the legal framework of public funding in energy research in the EU was reviewed, particularly with regard to the definitions and prerequisites for funding.
The third phase of system design consisted of three successive workshops addressing problem framing, conceptual development, and system validation.These workshops lasting 3 to 6 h were carried out with representatives of public funding institutions in Germany.Alongside a core team consisting of five members from one organization, who attended all of the workshops, additional participants from other funding bodies were involved in the problem-framing phase.The first workshop was designed to arrive at a common understanding of the problem and to define the requirements for the DSS.The second workshop discussed an early draft of the DSS, as well as the evaluation criteria and their importance.The third application and verification workshop served to critically review the final draft.
The fourth and final phase, a project-internal system review, started in parallel to the preparation of the third workshop to critically review the draft and to challenge the assumptions made.Following this review phase, the DSS was applied to two sample projects in order to verify its validity.

Literature on Public R&D Funding
Findings from the literature review suggest that a considerable body of literature exists on choosing R&D projects, e.g., in the context of R&D portfolios or concerning individual projects (e.g., [23][24][25][26]).A broad range of evaluation approaches has also been proposed for such purposes.These span both qualitative and quantitative approaches that are attributed to various methodological families, including mathematical methods (e.g., mathematical programming), economic methods (e.g., net present value, internal rate of return), decision analysis (e.g., Multi-Attribute Utility Theory), scoring methods (e.g., scoring algorithms), interactive methods (e.g., Delphi), and strategic models (e.g., Growth-Share Matrix) [25,27].Processes for evaluating research projects can be clustered into three groups: Ex-ante evaluations for choosing projects before their start, monitoring processes that address ongoing projects, and ex-post evaluations after conclusion of a project [18].It has been pointed out that most studies in the literature focus on ex-post evaluations of projects and programs.The subject matter of this paper, the ex-ante evaluation of R&D projects for public funding, has received less attention [7,13].
Peer or expert review is considered the most common mechanism for R&D evaluation in major funding agencies [7].Some works address the aggregation of different views in the evaluation process.A multi-criteria approach to integrate different opinions into an Analytical Hierarchy Process (AHP) combined with the utilization of fuzzy numbers has been introduced [17].To overcome the limitations of simple additive methods, it has been suggested to use evidential reasoning rules, which is another multi-criteria approach [18].The problem of a 'conservative' bias in R&D evaluation, i.e., that really innovative R&D proposals run a higher risk of rejection, has been addressed by using a randomization mechanism [7].
Other studies strive to learn about ex-ante processes from ex-post R&D data: They concentrate, for example, on exploiting the knowledge-base from ex-post evaluations in order to derive a framework for the ex-ante evaluation of publicly funded R&D projects with a focus on their regional benefits [6].Or they seek to check the performance of peer reviewers by comparing their results with those of a Data Envelopment Analysis [28].
Other studies examine the role and processes of innovation and funding agencies.This includes, for example, an analysis of the structures, norms, and interests within funding agencies and their impact on the cognitive development of science [16].An investigation of the ex-ante evaluation of public funding for R&D projects within the Danish energy program underlines the need for clarity for the experts that are involved in the evaluation process [13].The selection process across several European funding agencies has also been dealt with [29,30].Major findings include that selection procedures can be quite heterogeneous even within individual organizations, that agencies often rely on both internal and external experts, that there are differences in how the evaluation process is designed and who is responsible for financing decisions.Further investigations [31,32] underline that there is no specific best practice for proposal evaluation, but rather a set of good practices that have to be tailored to the specific situation.
Although these publications deal with the ex-ante evaluation of R&D projects for public funding, there are only a few suggestions concerning the operational implementation of such evaluations, especially when it comes to energy research and when the process goes beyond simply aggregating the scores for different criteria or various individuals.

Selected Practical Approaches to Proposal Selection
The theoretical perspective can be complemented by reviewing practical approaches to proposal evaluation in the domain of public energy research funding.Though publicly available information seems rather scarce, call documents and submission guidelines yield some insights into the evaluation mechanisms.It has to be noted that calls and guidelines are often tailored to specific situations, and, consequently, there is a large number of such documents.Therefore, only selected approaches are analyzed here.They are from different contexts, relate to proposal evaluations similar to experimental research in general, and they mostly concern energy and climate research (Table 1).
One approach is the New Entrants Reserve 300 (NER 300) (e.g., [14]) program for demonstration projects for CCS (Carbon Capture and Storage) and innovative renewable energies within the European Emissions Trading System (EU ETS).While project selection is mainly based on economic performance, specific requirements for knowledge transfer are a prerequisite for funding as well [33].Another example dealing with demonstration projects is a call for proposals from 2012 [34] within the Danish Energy Technology Development and Demonstration Program (EUDP).Four main evaluation criteria ('energy-related and technological perspective', 'economic perspective', 'own contribution and incentivizing effect', and 'organization, collaboration and knowledge transfer') with additional sub-items, i.e., explanatory additions, are described for evaluating the proposals.Another approach is that of the US-American Advanced Research Agency-Energy (ARPA-E) (e.g., [35]), which addresses energy technologies with high potentials and high impacts that are not yet attractive for private investment.In a specific call on energy-saving facsimile appearances [36], full proposals had to be evaluated using four weighted criteria ('impact of the proposed technology', 'overall scientific and technical merit', 'qualifications, experience and capabilities of the proposed project team', and 'soundness of management plan').Within the EU's Innovation and Research Program Horizon 2020 (H2020), award criteria for research grants within the funding areas 'Secure, clean and efficient energy' are by default unweighted for full proposals and include 'excellence', 'impact', and the 'quality and efficiency of the implementation'.Each criterion is scored on a 0 to 5 range and thresholds are used, i.e., a minimum of three points per criterion and of 10 for the overall evaluation are required for funding [37].Another more detailed, yet non-energy focused approach can be found in a guideline by the Austrian Research Promotion Agency (FFG) [38].Here, four main criteria ('quality of the proposal', 'suitability of applicants', 'impact and utilization', and 'relevance for the call') are used with minimum thresholds and additional sub-criteria.The range of criteria that are used in these funding programs seems to underline that evaluation in practice requires a multi-dimensional approach.Weights and thresholds are used in some cases with more formalized evaluation mechanisms, but most aggregation methods are quite simple and they usually rely on straightforward weighted sum models.

Requirements for the DSS for Public Funding Decisions in Germany
A set of characteristics for a practice-oriented DSS was derived from the review of the documents and approaches combined with the results from the first workshop.Accordingly, the DSS should be:

•
Universal: In view of multiple energy technologies and varying implementations of specific funding lines, the DSS should be broadly applicable and not be tailored to specific energy technologies or limited to either draft or full proposals.Furthermore, it should allow both the scoring of individual proposals, e.g., in open-ended submission formats, and their comparison, e.g., in the case of fixed submission deadlines.

•
Simple: While advanced methods for decision-support are attractive from a methodological perspective, there seem to be empirical limits to their adoption (e.g., [27]).As implied by Figure 2, different and varying groups are involved in proposal evaluations for energy research.They are not necessarily proficient day-to-day users of a DSS (e.g., experts on dedicated topics or personnel only occasionally involved in evaluations).The DSS should therefore be an intuitive non-expert system.

•
Supportive: The workshops highlighted that proposal evaluation is a process that is requiring experience.The DSS should therefore help in the evaluation process, but should not be a mechanical scoring mechanism or algorithm that seeks to replace skills and knowledge.Otherwise, acceptance might be low and a 'mechanistic' view of proposals might affect the quality of the evaluation.

•
Documenting: Due to the increasing demand for transparent records of public funding decisions, the DSS should also allow for documenting decisions.

•
Specific: The need for clear criteria definitions has been pointed out as a strategy in [29] to harmonize results across several evaluators.Correspondingly, the DSS should provide specific guidance in case of potentially ambiguous terms and concepts (e.g., 'excellence', 'innovation').
One observation from the workshop was that there is a general understanding of terms like 'demonstration project' or 'model project', which were selected as default project types for the DSS.Yet, there is no formalized operational definition of these terms.The DSS should provide specific guidance for identifying such project types.

•
Linked to TRL: The use of TRL is established in the context of energy research, but experience from practice shows that linking specific TRL levels to the content of proposals is challenging.
The DSS should facilitate that link.

•
Non-compensatory: Proposals may vary in their quality.The DSS should ensure that good scores in some areas do not compensate for bad scores in other areas.This means that the DSS should allow for minimum aspiration levels to be set.

•
Related to funding ceilings: Discussions during the workshops pointed out that funding must be a compromise between minimizing public spending and giving sufficient incentives to ensure private engagement.Finding this balance was perceived as difficult by the participants, but they thought it would be helpful to the DSS to include information on the maximum admissible ceilings for public funding without a formalized notification process, according to EU legislation.

Selection of a Multi-Criteria Decision Aid (MCDA) Method for Proposal Evaluation
To respond to the multi-criteria nature of decisions on proposals in energy research, there is a need for a multi-criteria evaluation approach within the DSS.A large number of methodological approaches have been developed since the 1960s for formalized explicit models to support decision-makers in evaluating and selecting alternatives (such as R&D proposals) [39].For such models, it is necessary to operationalize the relevant target criteria, in order to determine the benefits of an alternative course of action, and to resolve any existing conflicts of targets [40].
Many Multi-Criteria Decision Aid (MCDA) methods have been developed to support decision-makers to analyze several goals concurrently and transparently.The selection of a specific method is usually based on the problem to be solved [41].Within MCDA, two main families of methods can be distinguished: Methods attributed to Multi-Objective Decision-Making (MODM) allow for the determination of optimal alternatives, which are implicitly described by mathematical objective functions and constraints.The second family, methods of Multi-Attribute Decision-Making (MADM), allow for the evaluation of a set of explicitly given alternatives, as is the case with proposals.
Several aspects determine the selection of the MCDA method for the DSS.Employing a morphological box helps to list them in a structured manner (Figure 3):

•
User group: Methods of varying complexity are suitable for different user groups and purposes.
In line with the general requirements, the DSS should be suitable to non-MCDA experts.

•
Perspective: The DSS should document decisions transparently, but as it is primarily intended as an operational tool for funding bodies, it should consider proposals from their perspective.

•
Decision-makers: Though decisions on proposals are rarely taken by individuals, the use of a method for individuals or a group treated as an individual ensures a simpler, straight-forward DSS.

•
Alternatives: Proposals are given as is and thus belong to the group of explicitly defined alternatives in MCDA terminology.

•
Problem statement: The DSS may address several different types of problems that are distinguished in MCDA theory (e.g., [42]).The R&D proposal evaluation could concern the description problem (i.e., documentation of performance parameters without aggregation), the ranking problem (i.e., ranking the best to the worst proposal), or the sorting problem (i.e., sorting into classes such as excellent, adequate, and insufficient proposals).

•
Type of information: Some of the information in the proposal evaluation could be quantitative (e.g., [18]) and the analysis of existing methods shows that some use weights to differentiate criteria.This suggests that the DSS should be able to process cardinal information, i.e., numerical values.

•
Uncertainties: Real-world decision problems are fraught with uncertainty due to non-quantifiable, incomplete, unavailable information, or ignorance (e.g., [43]).Despite numerous methods to deal with uncertain information, the need for simplicity suggests that the DSS should avoid an explicit procedural consideration of uncertainty.

•
Weights: An important component of a multi-criteria procedure are the weights assigned to decision criteria.A flexible DSS should allow for individual weights to be selected.


Type of information: Some of the information in the proposal evaluation could be quantitative (e.g., [18]) and the analysis of existing methods shows that some use weights to differentiate criteria.This suggests that the DSS should be able to process cardinal information, i.e., numerical values.


Uncertainties: Real-world decision problems are fraught with uncertainty due to nonquantifiable, incomplete, unavailable information, or ignorance (e.g., [43]).Despite numerous methods to deal with uncertain information, the need for simplicity suggests that the DSS should avoid an explicit procedural consideration of uncertainty. Weights: An important component of a multi-criteria procedure are the weights assigned to decision criteria.A flexible DSS should allow for individual weights to be selected.These requirements are used to test the multitude of available MCDA methods.As alternatives are explicitly defined, only the MADM methods apply.These can be further structured by the type of information about the alternatives (i.e., what information is available about the alternatives) and then by the quality of information (i.e., which type of information is used to reflect the decisionmaker's preference).With regard to the former, different cases can be distinguished: those with no information, those with information about alternatives as a whole, or those with information about the alternatives' individual attributes (Figure 4).

Pairwise preferences
Order of pairwise proximity

Cardinal information
Marginal rate of substitution These requirements are used to test the multitude of available MCDA methods.As alternatives are explicitly defined, only the MADM methods apply.These can be further structured by the type of information about the alternatives (i.e., what information is available about the alternatives) and then by the quality of information (i.e., which type of information is used to reflect the decision-maker's preference).With regard to the former, different cases can be distinguished: those with no information, those with information about alternatives as a whole, or those with information about the alternatives' individual attributes (Figure 4).

Information on attributes
In the case of proposals, information about individual attributes is available.The relevant methods are further disaggregated by the use of nominal (disjunctive vs. conjunctive procedure), ordinal (e.g., Lexicographical method, Permutation method), or cardinal information (e.g., Analytical hierarchy process, ELECTRE, PROMETHEE, Cost-benefit analysis), or whether rates of substitution apply (e.g., MAUT, Hierarchical trade-offs) [44,45].For the latter two types, weights can be attributed, and thus specific importance can be assigned to the attributes of the alternatives.As compensation, i.e., substitution, is excluded by the requirements, a method from the group of cardinal approaches seems well suited.Despite its methodological limitations, the weighted sum method is a good candidate for a practical DSS due to its simplicity and its usage in other practice-oriented R&D evaluation activities.
are explicitly defined, only the MADM methods apply.These can be further structured by the type of information about the alternatives (i.e., what information is available about the alternatives) and then by the quality of information (i.e., which type of information is used to reflect the decisionmaker's preference).With regard to the former, different cases can be distinguished: those with no information, those with information about alternatives as a whole, or those with information about the alternatives' individual attributes (Figure 4).[44,45]).

Pairwise preferences
Order of pairwise proximity

Cardinal information
Marginal rate of substitution

Description of the DSS
Based on these requirements, the DSS was implemented as an operational guideline for R&D proposal evaluation.It consists of a compilation of supporting questions and tables for verification and review combined with the multi-criteria approach.It spans three stages (Figure 5): A first filtering stage serves to determine whether a proposal actually meets the basic requirements for funding.A second evaluation stage is an in-depth analysis of the proposal using the multi-criteria approach.The final third review stage considers the degree of public engagement.Each stage consists of steps and additional sub-steps in some cases.Proposals that are deemed unsuitable for further funding can be eliminated successively to avoid overburdening the later steps with an excessive number of proposals.
Energies 2018, 11, x FOR PEER REVIEW 9 of 18 In the case of proposals, information about individual attributes is available.The relevant methods are further disaggregated by the use of nominal (disjunctive vs. conjunctive procedure), ordinal (e.g., Lexicographical method, Permutation method), or cardinal information (e.g., Analytical hierarchy process, ELECTRE, PROMETHEE, Cost-benefit analysis), or whether rates of substitution apply (e.g., MAUT, Hierarchical trade-offs) [44,45].For the latter two types, weights can be attributed, and thus specific importance can be assigned to the attributes of the alternatives.As compensation, i.e., substitution, is excluded by the requirements, a method from the group of cardinal approaches seems well suited.Despite its methodological limitations, the weighted sum method is a good candidate for a practical DSS due to its simplicity and its usage in other practice-oriented R&D evaluation activities.

Description of the DSS
Based on these requirements, the DSS was implemented as an operational guideline for R&D proposal evaluation.It consists of a compilation of supporting questions and tables for verification and review combined with the multi-criteria approach.It spans three stages (Figure 5): A first filtering stage serves to determine whether a proposal actually meets the basic requirements for funding.A second evaluation stage is an in-depth analysis of the proposal using the multi-criteria approach.The final third review stage considers the degree of public engagement.Each stage consists of steps and additional sub-steps in some cases.Proposals that are deemed unsuitable for further funding can be eliminated successively to avoid overburdening the later steps with an excessive number of proposals.

Figure 5.
Overview of the decision support system's general structure.

Filtering Stage
The filtering stage of the DSS regularly uses dichotomous supporting questions with additional explanatory sub-items for each of the outlined steps.This design keeps questions general, while also providing evaluators with specific guidance on what to look for when judging certain aspects of a proposal.An example of a dichotomous question is given in Table 2.One step in the DSS usually

Filtering Stage
The filtering stage of the DSS regularly uses dichotomous supporting questions with additional explanatory sub-items for each of the outlined steps.This design keeps questions general, while also providing evaluators with specific guidance on what to look for when judging certain aspects of a proposal.An example of a dichotomous question is given in Table 2.One step in the DSS usually consists of answering several of these questions.In the case that some or all (depends on the context) questions are answered positively, the proposal proceeds to the next step.Otherwise, the decision-maker should critically review it and justify whether it is still eligible for funding, should be revised or entirely rejected.
Table 2. Example for a dichotomous supporting question used to verify the project type.

Does the Project Foresee Broad Transfer Activities Regarding Its Innovative Parts?
Yes No e.g.,

•
Will a transfer concept be developed and implemented?

•
Are there planned regular events to accompany/present the project?• Does the proposal contain reporting activities to the public or relevant market participants that considerably exceed the transfer activities of traditional applied research activities (e.g., a structured press campaign, participations in trade fairs, social-media presence, regular report in applied journals, patents, other publications?)

•
Is it possible to visit the demonstrator?• Does the measure promise to have a broad impact beyond the specific case at hand?

•
Is there a realistic possibility that the findings can be transferred to other companies or users?

Review of Fulfilling the Formal Criteria
The aim of this first step is to ensure the submitted documents comply with the formal requirements of the call (e.g., submission of complete set of documents, consortium has expected composition, etc.).

Review of Innovative Parts
The second step serves to identify the innovative parts of the proposal and the related surplus costs that are eligible for funding.Splitting the project into smaller parts was put forward during the workshops as a way to deal with the inherent complexity of proposals.The DDS therefore suggests systematically subdividing the whole project 'content-wise', e.g., (a) along the work plan, (b) by work packages, (c) by sub-parts of the real-world installation, (d) by technical process steps, or (e) by delivered services.Thereafter, those 'contents' that are relevant for the research activities need to be identified by answering a set of questions for every 'content'.Due to the close-to-market nature of experimental research, these questions focus only high TRL beyond 6.'Content' is not considered eligible for funding if it already belongs to TRL 9, unless it is indispensable for the implementation of other innovative parts.Determining the surplus costs follows the same approach: The individual 'contents' are reviewed and questions are asked about their relevance for realizing the innovative parts of the project.

Verification of the Proposal Type
Despite a general understanding, the workshop results indicated a lack of specific definitions of project types, which makes it difficult to identify experimental research.Therefore, the third step is to verify whether the submitted proposal meets the key characteristics of experimental research.As the European legal framework on R&D funding defines this term, a list of supporting questions closely follows the legal texts to exclude proposals that do not qualify as experimental research.
Furthermore, a funding program could focus on either 'demonstration projects' or 'model projects', depending on its specific setup.Thus, there is a need to determine whether a proposal belongs to either category.An additional set of supporting questions is provided for this purpose.Table 3 summarizes the key characteristics of each type to be verified in the DSS.

Review of Existing Market Failures
A set of questions is also used to identify market failures.Market failures exist if implementing the proposal's innovative contents is expected to be so unattractive from an economic point of view that they would not be realized without funding.Alternatively, economic risks may also prevent investments in innovative approaches, e.g., if they endanger the existence of the beneficiary in the event of a warranty claim.

Review of the Relevance for the Funding Program
A final important step concerns how a proposal fits into the funding scheme in general.Aspects to check include, for example, whether the proposal makes an adequate contribution to the scheme, whether it closes gaps in the existing funding portfolio, and whether it does not endanger the balance of the funding portfolio (e.g., by requiring too much funding).

Evaluation Stage
If a proposal successfully passes the filtering stage, a detailed assessment takes place in the evaluation stage.Here, the weighted sum model is used as a multi-criteria method to cover six main criteria.Three of them ('contribution to the fulfilment of energy and climate policy requirements', 'strengthening and securing the competitiveness of German companies', and 'securing and expanding technological development') reflect the objectives of the German energy research program.The three remaining criteria cover the inherent characteristics of the suggested project ('suitability of applicants', 'quality of the project', and 'risk of the project').Each main criterion consists of further sub-criteria to be evaluated individually, resulting in a total of 19 sub-criteria (Figure 6).
Each sub-criterion deals with specific aspects of the proposal.The sub-criterion 'quality and effectiveness of the work plan', for example, falls under the main criterion 'quality of the project'.A list of supporting questions is provided for each criterion to facilitate the evaluation.With regard to the 'quality and efficiency of the working plan', for example, it is discussed whether the plan follows a logical and goal-oriented structure, and whether its timing is plausible.An excerpt from the guideline for this sub-criterion is given in Table 4. Based on the answers to the individual supporting questions, an aggregated score is awarded to each criterion on a scale from 0 (criterion not fulfilled at all) to 5 (criterion fulfilled to a very high extent).This score should be complemented by a statement on the quality of the information in order to track the areas with uncertain or incomplete information.Each sub-criterion deals with specific aspects of the proposal.The sub-criterion 'quality and effectiveness of the work plan', for example, falls under the main criterion 'quality of the project'.A list of supporting questions is provided for each criterion to facilitate the evaluation.With regard to the 'quality and efficiency of the working plan', for example, it is discussed whether the plan follows a logical and goal-oriented structure, and whether its timing is plausible.An excerpt from the guideline for this sub-criterion is given in Table 4. Based on the answers to the individual supporting questions, an aggregated score is awarded to each criterion on a scale from 0 (criterion not fulfilled at all) to 5 (criterion fulfilled to a very high extent).This score should be complemented by a statement on the quality of the information in order to track the areas with uncertain or incomplete information.
Each main and sub-criterion can be attributed a weight to reflect its relevance.The weighted criteria are then aggregated in order to obtain an overall score.Threshold values can also be selected on both the level of individual sub-criteria and the level of the main criteria.If these thresholds are not met, the proposal will not be considered further.Table 4. Sample excerpt (specific guidance omitted) from the guideline for the criterion 'quality and efficiency of the working plan'.

Definition
Transparency and quality of the procedure for achieving the objectives of the overall project and its parts

Premises
The higher the quality and efficiency of the working plan, the more attractive the project

Main criterion
Quality of the project Perspective Perspective of the funding body Supporting Questions  Does the work plan follow a logical and purposeful structure that makes the project feasible? Does the work plan have a plausible time plan? Are all the work plan phases really relevant to the project? Is the methodology for the different phases of the work plan plausible and does it contribute to achieving the targets? Have suitable measures been taken to monitor the progress of the project?Verification How reliable, justified and transparent is the judgement of this criterion?
Figure 7 shows an excerpt from a simple Excel tool used to illustrate the scoring model in the evaluation stage.Its upper part shows the set of criteria, their weights, and minimum thresholds.The Table 4. Sample excerpt (specific guidance omitted) from the guideline for the criterion 'quality and efficiency of the working plan'.

Definition
Transparency and quality of the procedure for achieving the objectives of the overall project and its parts

Premises
The higher the quality and efficiency of the working plan, the more attractive the project Have suitable measures been taken to monitor the progress of the project?

Verification
How reliable, justified and transparent is the judgement of this criterion?
Each main and sub-criterion can be attributed a weight to reflect its relevance.The weighted criteria are then aggregated in order to obtain an overall score.Threshold values can also be selected on both the level of individual sub-criteria and the level of the main criteria.If these thresholds are not met, the proposal will not be considered further.
Figure 7 shows an excerpt from a simple Excel tool used to illustrate the scoring model in the evaluation stage.Its upper part shows the set of criteria, their weights, and minimum thresholds.The lower part gives a list of projects with sample scores.To facilitate understanding, the tool offers different default options, e.g., a set of draft weights computed from workshop results using a simple criteria-ranking approach or a set of threshold values.
Energies 2018, 11, x FOR PEER REVIEW 13 of 18 lower part gives a list of projects with sample scores.To facilitate understanding, the tool offers different default options, e.g., a set of draft weights computed from workshop results using a simple criteria-ranking approach or a set of threshold values.

Review Stage
If a proposal has successfully passed all of the previous stages, it is then subject to the review stage with a final assessment of the need for public funding.When compared to the previous stages, this stage is less formalized, but it is also based on supporting questions.

Review Stage
If a proposal has successfully passed all of the previous stages, it is then subject to the review stage with a final assessment of the need for public funding.When compared to the previous stages, this stage is less formalized, but it is also based on supporting questions.

Minimizing Deadweight Losses
Here, potential deadweight losses (i.e., beneficiaries who would also implement the project without funding) are considered in further detail.It is planned to check whether public funding is really required or whether potential beneficiaries would be willing to implement the project on their own.Current market trends can be an indication here, but (forthcoming) national and international policies should also be considered.It should be noted that the general issue of deadweight losses is also addressed indirectly during the review of market failures in the filtering stage.There, it serves to eliminate obvious cases of non-eligible proposals early on in the process, while, here, the assessment of the remaining proposals is expected to be more thorough.

Level and Extent of Funding
With regard to the level and extent of funding, legislation requires the effective use of public funds.As a basic rule for research projects, aid intensity for a beneficiary has a ceiling of 25% for experimental research under EU regulations.Depending on the specific situation, other ceilings and additional absolute thresholds may apply; these are illustrated by the DSS in flow diagrams.Additional national provisions for public funding may apply, as well as specific regulations for individual funding schemes.Thus, it is difficult to make generic provisions for the amount of funding.While the DSS gives a general guideline, appropriate funding levels have to be determined for each specific project proposal in this step, while taking into account the respective context.The DSS suggests which aspects to consider in this context.

Discussion Concerning the DSS and the Funding Process
If public funding of experimental research proposals becomes more important, there will be an increasing need to focus the limited public budgets on the most promising proposals and to respond to stricter requirements for transparency and documentation.The DDS has been suggested as a means to facilitate the evaluation process because it (a) provides a structured approach that bundles important aspects of the evaluation into a process; (b) helps to conduct a uniform evaluation of proposals that is largely independent of the individual evaluator; (c) enables less experienced evaluators to conduct an evaluation; and, (d) is a documented and transparent process.
In line with the requirement for a universal system, the DSS has been designed to be neutral with regard to technologies, draft or full proposals, and the organizational context of a funding line.However, it should be noted that it focuses on demonstration and model projects that can be perceived as a particular subcategory of experimental development.Furthermore, the evaluation stage with its main criteria relates to the German energy research program.While this is a limitation, it can also be expected that energy research activities in other countries generally follow similar programmatic criteria.To ensure that the DSS is simple, it relies on the same multi-criteria approach as already used in practice.Despite the resulting methodological limitations (e.g., [18]), the weighted sum model is one of the most simple and intuitive MCDA methods.Furthermore, with regard to the supportive character of the DSS, discussions in the workshops underlined that evaluation practitioners appreciate some liberty for interpretation when applying a DSS.This liberty is needed to avoid the risk of a prescriptive DSS that is too rigid to consider context-specific information from the expert's point of view.Obviously, excessive liberty can come at the detriment of obtaining evaluation results that are largely independent of the evaluator.However, the documenting function of the DSS can also help to ensure transparency, as it requires its users to record how they evaluated particular aspects of a proposal.Concerning specific guidance, the DSS was perceived by some of the workshop participants as a helpful formalization of existing knowledge bundled together in one place.Adding specific guidance and interpretations of definitions and questions was considered to enhance the quality of the evaluations.The link to TRL to identify content that is eligible for funding within the two specific types of experimental development was also perceived as helpful.The non-compensatory character was implemented using minimum thresholds, as in [37] or [38].Unfortunately, due to time constraints, this feature was not discussed in detail during the workshops.The consideration of funding ceilings was also only implemented as a simple flowsheet as workshop participants underlined the difficulties in bringing the multitude of context-dependent constraints together in a generalized concept.
eceee Industrial Summer Study (Hirzel, S.; Hettesheimer, T.; Viebahn, P.; Fischedick, M. 'Bridging the valley of death: A multi-staged multi-criteria decision support system for evaluating proposals for large-scale energy demonstration projects as public funding opportunities'.Accepted for the Proceedings of the 2018 eceee Industry Summer Study, 11-13 June 2018, Berlin).

Figure 1 .
Figure 1.Conceptual illustration of public funding overcoming the valley of death after traditional project funding has ended.

Figure 1 .
Figure 1.Conceptual illustration of public funding overcoming the valley of death after traditional project funding has ended.

Figure 2 .
Figure 2.Administrative process for a successful proposal in Germany's public energy research funding with amendments based on[19].

Figure 3 .
Figure 3. Morphological box to describe the setup for the decision support system (relevant aspects in grey).

Figure 3 .
Figure 3. Morphological box to describe the setup for the decision support system (relevant aspects in grey).

IFigure 5 .
Figure 5. Overview of the decision support system's general structure.

Figure 6 .
Figure 6.Overview of the six main and nineteen sub-criteria of the evaluation stage.

Figure 6 .
Figure 6.Overview of the six main and nineteen sub-criteria of the evaluation stage.

Figure 7 .
Figure 7. Tool to illustrate the evaluation stage, including (sample) weights for the main and subcriteria, (sample) threshold values, and dummy proposals to illustrate elimination due to insufficient performance (bold red numbers: sub criterion below threshold; colored backfill: sum of sub-criteria in main criterion below threshold).

Figure 7 .
Figure 7. Tool to illustrate the evaluation stage, including (sample) weights for the main and sub-criteria, (sample) threshold values, and dummy proposals to illustrate elimination due to insufficient performance (bold red numbers: sub criterion below threshold; colored backfill: sum of sub-criteria in main criterion below threshold).

Table 1 .
Overview of selected approaches to evaluating research and development proposals within public research funding.
NER 300: New Entrants Reserve 300; EUDP: Danish Energy Technology Development and Demonstration Program; ARPA-E: US-American Advanced Research Agency-Energy; H2020: EU's Innovation and Research Program Horizon 2020; FFG: Austrian Research Promotion Agency; CCS: Carbon Capture and Storage.

Table 3 .
Summary of key commonalities and differences between demonstration and model proposals used in the decision support system (DSS) to identify the proposal type.