Next Article in Journal
A Novel Method for Lithium-Ion Battery Online Parameter Identification Based on Variable Forgetting Factor Recursive Least Squares
Previous Article in Journal
On the Optimal Pole Width for Direct Drive Linear Wave Power Generators Using Ferrite Magnets
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Decision Support System for Public Funding of Experimental Development in Energy Research

1
Fraunhofer Institute for Systems and Innovation Research ISI, Breslauer Str. 48, 76139 Karlsruhe, Germany
2
Wuppertal Institut für Klima, Umwelt, Energie, Döppersberg 19, 42103 Wuppertal, Germany
*
Author to whom correspondence should be addressed.
Energies 2018, 11(6), 1357; https://doi.org/10.3390/en11061357
Submission received: 8 May 2018 / Revised: 23 May 2018 / Accepted: 24 May 2018 / Published: 26 May 2018
(This article belongs to the Section A: Sustainable Energy)

Abstract

:
New energy technologies may fail to make the transition to the market once research funding has ended due to a lack of private engagement to conclude their development. Extending public funding to cover such experimental developments could be one way to improve this transition. However, identifying promising research and development (R&D) proposals for this purpose is a difficult task for the following reasons: Close-to-market implementations regularly require substantial resources while public budgets are limited; the allocation of public funds needs to be fair, open, and documented; the evaluation is complex and subject to public sector regulations for public engagement in R&D funding. This calls for a rigorous evaluation process. This paper proposes an operational three-staged decision support system (DSS) to assist decision-makers in public funding institutions in the ex-ante evaluation of R&D proposals for large-scale close-to-market projects in energy research. The system was developed based on a review of literature and related approaches from practice combined with a series of workshops with practitioners from German public funding institutions. The results confirm that the decision-making process is a complex one that is not limited to simply scoring R&D proposals. Decision-makers also have to deal with various additional issues such as determining the state of technological development, verifying market failures or considering existing funding portfolios. The DSS that is suggested in this paper is unique in the sense that it goes beyond mere multi-criteria aggregation procedures and addresses these issues as well to help guide decision-makers in public institutions through the evaluation process.

1. Introduction

In view of the 1.5 degree Paris Agreement [1] and the European target to reduce greenhouse gas emissions by at least 80% by 2050 when compared to 1990 levels [2], the successful and the widespread utilization of efficient and climate friendly new energy technologies is becoming more important. Industrialized countries are striving to facilitate and accelerate the transition to higher technology readiness levels (TRL, e.g., [3]) by using public funds to spend on research activities with a defined scope, budget, and duration (e.g., [4]). The member governments of the International Energy Agency (IEA) alone, for example, spent USD 16.6 billion on energy research, development, and demonstration in 2016 [5]. The theory is that, apart from private benefits, the publicly funded research and development (R&D) activities of firms also yield positive spillovers for the economy as a whole. Without the additional funding, individual firms would not make the level of investment on their own, which is considered optimal from a societal perspective (e.g., [6,7]). In this sense, state intervention here is to overcome market failures due to the public good character of knowledge and to reduce systemic failures that are negatively affecting the diffusion of knowledge and/or technological learning [8].
Traditionally, public funding for energy research projects tends to focus on TRL 1 to 6 or 7. The lowest level, TRL 1, indicates that the basic principles of a technology have been observed, while the highest level, TRL 9, underlines the availability of proven systems on the market. Though not without ambiguity [9], technologies beyond TRL 6 can be attributed to the group of ‘experimental development’, as defined in European legislation relating to R&D [10]. Such developments run the risk of being stuck in the ‘valley of death’ [11,12] after traditional research funding has ended. More specifically, they fail to be adopted by the market or this process simply takes long (Figure 1).
Extending public research funding on energy projects beyond its traditional focus could be a means to bridge the valley of death in energy research. Identifying promising proposals for public R&D funding is a very important part of the funding process [13], but this can also be very challenging for decision-makers in public institutions:
  • Full industry-scale implementations require considerable resources for scale-ups and real-world adaptations and involve major risks (e.g., [14,15]). Given that public budgets are limited, funds can only be allocated to the most promising projects and these need to be identified among numerous funding opportunities.
  • Funding agencies have some leeway in how they spend their money (e.g., [16]). The use of public resources is governed by political factors and is subject to the influence of stakeholders in R&D, e.g., from academia, industry, policy making, or the general population. Yet, the allocation of funds is expected to be fair and open [17]. An increasing pressure for accountability has been observed, which requires public institutions to present the results of their programs and to justify their existence (e.g., [13]). There is therefore a need to document the coherency of funding decisions transparently and comprehensively.
  • It has been pointed out that public sector financers have a different perspective on R&D than private investors [11]. This concerns the strategic and long-term character of public funding, which intends to encourage private investments in potentially attractive areas. As a consequence, traditional financial justification techniques are seen as probably unsuitable [17]. Furthermore, research project evaluation itself is considered complex since it usually involves the consideration of multiple criteria and the views of multiple experts [18]. This makes it necessary to judge proposals from different angles. However, obtaining the required information can also be difficult.
  • Public funding is subject to legal frameworks, which also define the limits for state interventions. It is therefore necessary to consider these.
To help overcome these challenges and to facilitate the evaluation process, this paper proposes an operational decision support system (DSS) for evaluating the proposals of large-scale projects in experimental energy research as public funding opportunities. To the best of our knowledge, no such DSS is described in the literature at present.
The remainder of this paper presents a brief description of the background and the methodological approach to develop the DSS (Section 2). This is followed by a review of the relevant literature and the practical applications of public R&D funding (Section 3), and a discussion of the requirements and methods for the DSS (Section 4). The proposed DSS is then described (Section 5), followed by a discussion (Section 6) and the final conclusions (Section 7).

2. Background and Methodological Approach to Developing the DSS

2.1. Proposal Evaluation Process in Germany’s Applied Energy Research Funding

The DSS put forward in this paper was elaborated within a research project on future energy technologies funded by the German Federal Ministry for Economic Affairs and Energy (BMWi). BMWi is the major funding institution for applied energy research in Germany. Its activities are governed by the ‘Sixth Energy Research Programme of the Federal Government’ of 2011 [19]. A forthcoming seventh program is being prepared [20]. As with any national research funding activities in the European Union (EU), funding is subject to the European rules for public aid [10].
The current energy research program has three objectives: to help achieve the German government’s targets relating to the energy sector and climate policy; to foster the leadership of German companies in modern energy technologies; and, to secure and enhance the availability of technological options. To this end, funding is granted as both institutional and project funding [19].
Figure 2 provides an overview of the generic evaluation process for selecting project proposals for energy research funding in Germany. Several groups are involved in this process: applicants submitting proposals for projects, funding agencies/bodies, the Ministry, and, where applicable, additional internal or external evaluators.
German energy research funding traditionally focuses on basic and applied research. In order to support the German Energiewende (e.g., [21]) and to contribute to the national long-term energy and climate goals [22], the suggestion has been made to allocate funds to close-to-market experimental research as a potential additional option.

2.2. Methodological Approach to Developing the DSS

The DSS that is described in this paper was developed in four phases with regard to the potential coverage of close-to-market research. The impetus for the first problem definition phase was an early exchange with BMWi to pinpoint the challenges that are underlining the need for a DSS. Based on this exchange, a draft outline was prepared, and a formalized development process started.
The second phase of information collection followed, which concentrated on reviewing existing methods of R&D proposal evaluation that is related to energy and climate research. For this purpose, selected programs from different contexts and countries were analyzed with a focus on their evaluation criteria and mechanisms. Furthermore, the legal framework of public funding in energy research in the EU was reviewed, particularly with regard to the definitions and prerequisites for funding.
The third phase of system design consisted of three successive workshops addressing problem framing, conceptual development, and system validation. These workshops lasting 3 to 6 h were carried out with representatives of public funding institutions in Germany. Alongside a core team consisting of five members from one organization, who attended all of the workshops, additional participants from other funding bodies were involved in the problem-framing phase. The first workshop was designed to arrive at a common understanding of the problem and to define the requirements for the DSS. The second workshop discussed an early draft of the DSS, as well as the evaluation criteria and their importance. The third application and verification workshop served to critically review the final draft.
The fourth and final phase, a project-internal system review, started in parallel to the preparation of the third workshop to critically review the draft and to challenge the assumptions made. Following this review phase, the DSS was applied to two sample projects in order to verify its validity.

3. Literature and Selected Practical Applications for Public R&D Funding

3.1. Literature on Public R&D Funding

Findings from the literature review suggest that a considerable body of literature exists on choosing R&D projects, e.g., in the context of R&D portfolios or concerning individual projects (e.g., [23,24,25,26]). A broad range of evaluation approaches has also been proposed for such purposes. These span both qualitative and quantitative approaches that are attributed to various methodological families, including mathematical methods (e.g., mathematical programming), economic methods (e.g., net present value, internal rate of return), decision analysis (e.g., Multi-Attribute Utility Theory), scoring methods (e.g., scoring algorithms), interactive methods (e.g., Delphi), and strategic models (e.g., Growth-Share Matrix) [25,27]. Processes for evaluating research projects can be clustered into three groups: Ex-ante evaluations for choosing projects before their start, monitoring processes that address ongoing projects, and ex-post evaluations after conclusion of a project [18]. It has been pointed out that most studies in the literature focus on ex-post evaluations of projects and programs. The subject matter of this paper, the ex-ante evaluation of R&D projects for public funding, has received less attention [7,13].
Peer or expert review is considered the most common mechanism for R&D evaluation in major funding agencies [7]. Some works address the aggregation of different views in the evaluation process. A multi-criteria approach to integrate different opinions into an Analytical Hierarchy Process (AHP) combined with the utilization of fuzzy numbers has been introduced [17]. To overcome the limitations of simple additive methods, it has been suggested to use evidential reasoning rules, which is another multi-criteria approach [18]. The problem of a ‘conservative’ bias in R&D evaluation, i.e., that really innovative R&D proposals run a higher risk of rejection, has been addressed by using a randomization mechanism [7].
Other studies strive to learn about ex-ante processes from ex-post R&D data: They concentrate, for example, on exploiting the knowledge-base from ex-post evaluations in order to derive a framework for the ex-ante evaluation of publicly funded R&D projects with a focus on their regional benefits [6]. Or they seek to check the performance of peer reviewers by comparing their results with those of a Data Envelopment Analysis [28].
Other studies examine the role and processes of innovation and funding agencies. This includes, for example, an analysis of the structures, norms, and interests within funding agencies and their impact on the cognitive development of science [16]. An investigation of the ex-ante evaluation of public funding for R&D projects within the Danish energy program underlines the need for clarity for the experts that are involved in the evaluation process [13]. The selection process across several European funding agencies has also been dealt with [29,30]. Major findings include that selection procedures can be quite heterogeneous even within individual organizations, that agencies often rely on both internal and external experts, that there are differences in how the evaluation process is designed and who is responsible for financing decisions. Further investigations [31,32] underline that there is no specific best practice for proposal evaluation, but rather a set of good practices that have to be tailored to the specific situation.
Although these publications deal with the ex-ante evaluation of R&D projects for public funding, there are only a few suggestions concerning the operational implementation of such evaluations, especially when it comes to energy research and when the process goes beyond simply aggregating the scores for different criteria or various individuals.

3.2. Selected Practical Approaches to Proposal Selection

The theoretical perspective can be complemented by reviewing practical approaches to proposal evaluation in the domain of public energy research funding. Though publicly available information seems rather scarce, call documents and submission guidelines yield some insights into the evaluation mechanisms. It has to be noted that calls and guidelines are often tailored to specific situations, and, consequently, there is a large number of such documents. Therefore, only selected approaches are analyzed here. They are from different contexts, relate to proposal evaluations similar to experimental research in general, and they mostly concern energy and climate research (Table 1).
One approach is the New Entrants Reserve 300 (NER 300) (e.g., [14]) program for demonstration projects for CCS (Carbon Capture and Storage) and innovative renewable energies within the European Emissions Trading System (EU ETS). While project selection is mainly based on economic performance, specific requirements for knowledge transfer are a prerequisite for funding as well [33]. Another example dealing with demonstration projects is a call for proposals from 2012 [34] within the Danish Energy Technology Development and Demonstration Program (EUDP). Four main evaluation criteria (‘energy-related and technological perspective’, ‘economic perspective’, ‘own contribution and incentivizing effect’, and ‘organization, collaboration and knowledge transfer’) with additional sub-items, i.e., explanatory additions, are described for evaluating the proposals. Another approach is that of the US-American Advanced Research Agency-Energy (ARPA-E) (e.g., [35]), which addresses energy technologies with high potentials and high impacts that are not yet attractive for private investment. In a specific call on energy-saving facsimile appearances [36], full proposals had to be evaluated using four weighted criteria (‘impact of the proposed technology’, ‘overall scientific and technical merit’, ‘qualifications, experience and capabilities of the proposed project team’, and ‘soundness of management plan’). Within the EU’s Innovation and Research Program Horizon 2020 (H2020), award criteria for research grants within the funding areas ‘Secure, clean and efficient energy’ are by default unweighted for full proposals and include ‘excellence’, ‘impact’, and the ‘quality and efficiency of the implementation’. Each criterion is scored on a 0 to 5 range and thresholds are used, i.e., a minimum of three points per criterion and of 10 for the overall evaluation are required for funding [37]. Another more detailed, yet non-energy focused approach can be found in a guideline by the Austrian Research Promotion Agency (FFG) [38]. Here, four main criteria (‘quality of the proposal’, ‘suitability of applicants’, ‘impact and utilization’, and ‘relevance for the call’) are used with minimum thresholds and additional sub-criteria.
The range of criteria that are used in these funding programs seems to underline that evaluation in practice requires a multi-dimensional approach. Weights and thresholds are used in some cases with more formalized evaluation mechanisms, but most aggregation methods are quite simple and they usually rely on straightforward weighted sum models.

4. Requirements and Methods for the DSS

4.1. Requirements for the DSS for Public Funding Decisions in Germany

A set of characteristics for a practice-oriented DSS was derived from the review of the documents and approaches combined with the results from the first workshop. Accordingly, the DSS should be:
  • Universal: In view of multiple energy technologies and varying implementations of specific funding lines, the DSS should be broadly applicable and not be tailored to specific energy technologies or limited to either draft or full proposals. Furthermore, it should allow both the scoring of individual proposals, e.g., in open-ended submission formats, and their comparison, e.g., in the case of fixed submission deadlines.
  • Simple: While advanced methods for decision-support are attractive from a methodological perspective, there seem to be empirical limits to their adoption (e.g., [27]). As implied by Figure 2, different and varying groups are involved in proposal evaluations for energy research. They are not necessarily proficient day-to-day users of a DSS (e.g., experts on dedicated topics or personnel only occasionally involved in evaluations). The DSS should therefore be an intuitive non-expert system.
  • Supportive: The workshops highlighted that proposal evaluation is a process that is requiring experience. The DSS should therefore help in the evaluation process, but should not be a mechanical scoring mechanism or algorithm that seeks to replace skills and knowledge. Otherwise, acceptance might be low and a ‘mechanistic’ view of proposals might affect the quality of the evaluation.
  • Documenting: Due to the increasing demand for transparent records of public funding decisions, the DSS should also allow for documenting decisions.
  • Specific: The need for clear criteria definitions has been pointed out as a strategy in [29] to harmonize results across several evaluators. Correspondingly, the DSS should provide specific guidance in case of potentially ambiguous terms and concepts (e.g., ‘excellence’, ‘innovation’). One observation from the workshop was that there is a general understanding of terms like ‘demonstration project’ or ‘model project’, which were selected as default project types for the DSS. Yet, there is no formalized operational definition of these terms. The DSS should provide specific guidance for identifying such project types.
  • Linked to TRL: The use of TRL is established in the context of energy research, but experience from practice shows that linking specific TRL levels to the content of proposals is challenging. The DSS should facilitate that link.
  • Non-compensatory: Proposals may vary in their quality. The DSS should ensure that good scores in some areas do not compensate for bad scores in other areas. This means that the DSS should allow for minimum aspiration levels to be set.
  • Related to funding ceilings: Discussions during the workshops pointed out that funding must be a compromise between minimizing public spending and giving sufficient incentives to ensure private engagement. Finding this balance was perceived as difficult by the participants, but they thought it would be helpful to the DSS to include information on the maximum admissible ceilings for public funding without a formalized notification process, according to EU legislation.

4.2. Selection of a Multi-Criteria Decision Aid (MCDA) Method for Proposal Evaluation

To respond to the multi-criteria nature of decisions on proposals in energy research, there is a need for a multi-criteria evaluation approach within the DSS. A large number of methodological approaches have been developed since the 1960s for formalized explicit models to support decision-makers in evaluating and selecting alternatives (such as R&D proposals) [39]. For such models, it is necessary to operationalize the relevant target criteria, in order to determine the benefits of an alternative course of action, and to resolve any existing conflicts of targets [40].
Many Multi-Criteria Decision Aid (MCDA) methods have been developed to support decision-makers to analyze several goals concurrently and transparently. The selection of a specific method is usually based on the problem to be solved [41]. Within MCDA, two main families of methods can be distinguished: Methods attributed to Multi-Objective Decision-Making (MODM) allow for the determination of optimal alternatives, which are implicitly described by mathematical objective functions and constraints. The second family, methods of Multi-Attribute Decision-Making (MADM), allow for the evaluation of a set of explicitly given alternatives, as is the case with proposals.
Several aspects determine the selection of the MCDA method for the DSS. Employing a morphological box helps to list them in a structured manner (Figure 3):
  • User group: Methods of varying complexity are suitable for different user groups and purposes. In line with the general requirements, the DSS should be suitable to non-MCDA experts.
  • Perspective: The DSS should document decisions transparently, but as it is primarily intended as an operational tool for funding bodies, it should consider proposals from their perspective.
  • Decision-makers: Though decisions on proposals are rarely taken by individuals, the use of a method for individuals or a group treated as an individual ensures a simpler, straight-forward DSS.
  • Alternatives: Proposals are given as is and thus belong to the group of explicitly defined alternatives in MCDA terminology.
  • Problem statement: The DSS may address several different types of problems that are distinguished in MCDA theory (e.g., [42]). The R&D proposal evaluation could concern the description problem (i.e., documentation of performance parameters without aggregation), the ranking problem (i.e., ranking the best to the worst proposal), or the sorting problem (i.e., sorting into classes such as excellent, adequate, and insufficient proposals).
  • Type of information: Some of the information in the proposal evaluation could be quantitative (e.g., [18]) and the analysis of existing methods shows that some use weights to differentiate criteria. This suggests that the DSS should be able to process cardinal information, i.e., numerical values.
  • Uncertainties: Real-world decision problems are fraught with uncertainty due to non-quantifiable, incomplete, unavailable information, or ignorance (e.g., [43]). Despite numerous methods to deal with uncertain information, the need for simplicity suggests that the DSS should avoid an explicit procedural consideration of uncertainty.
  • Weights: An important component of a multi-criteria procedure are the weights assigned to decision criteria. A flexible DSS should allow for individual weights to be selected.
These requirements are used to test the multitude of available MCDA methods. As alternatives are explicitly defined, only the MADM methods apply. These can be further structured by the type of information about the alternatives (i.e., what information is available about the alternatives) and then by the quality of information (i.e., which type of information is used to reflect the decision-maker’s preference). With regard to the former, different cases can be distinguished: those with no information, those with information about alternatives as a whole, or those with information about the alternatives’ individual attributes (Figure 4).
In the case of proposals, information about individual attributes is available. The relevant methods are further disaggregated by the use of nominal (disjunctive vs. conjunctive procedure), ordinal (e.g., Lexicographical method, Permutation method), or cardinal information (e.g., Analytical hierarchy process, ELECTRE, PROMETHEE, Cost-benefit analysis), or whether rates of substitution apply (e.g., MAUT, Hierarchical trade-offs) [44,45]. For the latter two types, weights can be attributed, and thus specific importance can be assigned to the attributes of the alternatives. As compensation, i.e., substitution, is excluded by the requirements, a method from the group of cardinal approaches seems well suited. Despite its methodological limitations, the weighted sum method is a good candidate for a practical DSS due to its simplicity and its usage in other practice-oriented R&D evaluation activities.

5. Description of the DSS

Based on these requirements, the DSS was implemented as an operational guideline for R&D proposal evaluation. It consists of a compilation of supporting questions and tables for verification and review combined with the multi-criteria approach. It spans three stages (Figure 5): A first filtering stage serves to determine whether a proposal actually meets the basic requirements for funding. A second evaluation stage is an in-depth analysis of the proposal using the multi-criteria approach. The final third review stage considers the degree of public engagement. Each stage consists of steps and additional sub-steps in some cases. Proposals that are deemed unsuitable for further funding can be eliminated successively to avoid overburdening the later steps with an excessive number of proposals.

5.1. Filtering Stage

The filtering stage of the DSS regularly uses dichotomous supporting questions with additional explanatory sub-items for each of the outlined steps. This design keeps questions general, while also providing evaluators with specific guidance on what to look for when judging certain aspects of a proposal. An example of a dichotomous question is given in Table 2. One step in the DSS usually consists of answering several of these questions. In the case that some or all (depends on the context) questions are answered positively, the proposal proceeds to the next step. Otherwise, the decision-maker should critically review it and justify whether it is still eligible for funding, should be revised or entirely rejected.

5.1.1. Review of Fulfilling the Formal Criteria

The aim of this first step is to ensure the submitted documents comply with the formal requirements of the call (e.g., submission of complete set of documents, consortium has expected composition, etc.).

5.1.2. Review of Innovative Parts

The second step serves to identify the innovative parts of the proposal and the related surplus costs that are eligible for funding. Splitting the project into smaller parts was put forward during the workshops as a way to deal with the inherent complexity of proposals. The DDS therefore suggests systematically subdividing the whole project ‘content-wise’, e.g., (a) along the work plan, (b) by work packages, (c) by sub-parts of the real-world installation, (d) by technical process steps, or (e) by delivered services. Thereafter, those ‘contents’ that are relevant for the research activities need to be identified by answering a set of questions for every ‘content’. Due to the close-to-market nature of experimental research, these questions focus only high TRL beyond 6. ‘Content’ is not considered eligible for funding if it already belongs to TRL 9, unless it is indispensable for the implementation of other innovative parts. Determining the surplus costs follows the same approach: The individual ‘contents’ are reviewed and questions are asked about their relevance for realizing the innovative parts of the project.

5.1.3. Verification of the Proposal Type

Despite a general understanding, the workshop results indicated a lack of specific definitions of project types, which makes it difficult to identify experimental research. Therefore, the third step is to verify whether the submitted proposal meets the key characteristics of experimental research. As the European legal framework on R&D funding defines this term, a list of supporting questions closely follows the legal texts to exclude proposals that do not qualify as experimental research.
Furthermore, a funding program could focus on either ‘demonstration projects’ or ‘model projects’, depending on its specific setup. Thus, there is a need to determine whether a proposal belongs to either category. An additional set of supporting questions is provided for this purpose. Table 3 summarizes the key characteristics of each type to be verified in the DSS.

5.1.4. Review of Existing Market Failures

A set of questions is also used to identify market failures. Market failures exist if implementing the proposal’s innovative contents is expected to be so unattractive from an economic point of view that they would not be realized without funding. Alternatively, economic risks may also prevent investments in innovative approaches, e.g., if they endanger the existence of the beneficiary in the event of a warranty claim.

5.1.5. Review of the Relevance for the Funding Program

A final important step concerns how a proposal fits into the funding scheme in general. Aspects to check include, for example, whether the proposal makes an adequate contribution to the scheme, whether it closes gaps in the existing funding portfolio, and whether it does not endanger the balance of the funding portfolio (e.g., by requiring too much funding).

5.2. Evaluation Stage

If a proposal successfully passes the filtering stage, a detailed assessment takes place in the evaluation stage. Here, the weighted sum model is used as a multi-criteria method to cover six main criteria. Three of them (‘contribution to the fulfilment of energy and climate policy requirements’, ‘strengthening and securing the competitiveness of German companies’, and ‘securing and expanding technological development’) reflect the objectives of the German energy research program. The three remaining criteria cover the inherent characteristics of the suggested project (‘suitability of applicants’, ‘quality of the project’, and ‘risk of the project’). Each main criterion consists of further sub-criteria to be evaluated individually, resulting in a total of 19 sub-criteria (Figure 6).
Each sub-criterion deals with specific aspects of the proposal. The sub-criterion ‘quality and effectiveness of the work plan’, for example, falls under the main criterion ‘quality of the project’. A list of supporting questions is provided for each criterion to facilitate the evaluation. With regard to the ‘quality and efficiency of the working plan’, for example, it is discussed whether the plan follows a logical and goal-oriented structure, and whether its timing is plausible. An excerpt from the guideline for this sub-criterion is given in Table 4. Based on the answers to the individual supporting questions, an aggregated score is awarded to each criterion on a scale from 0 (criterion not fulfilled at all) to 5 (criterion fulfilled to a very high extent). This score should be complemented by a statement on the quality of the information in order to track the areas with uncertain or incomplete information.
Each main and sub-criterion can be attributed a weight to reflect its relevance. The weighted criteria are then aggregated in order to obtain an overall score. Threshold values can also be selected on both the level of individual sub-criteria and the level of the main criteria. If these thresholds are not met, the proposal will not be considered further.
Figure 7 shows an excerpt from a simple Excel tool used to illustrate the scoring model in the evaluation stage. Its upper part shows the set of criteria, their weights, and minimum thresholds. The lower part gives a list of projects with sample scores. To facilitate understanding, the tool offers different default options, e.g., a set of draft weights computed from workshop results using a simple criteria-ranking approach or a set of threshold values.

5.3. Review Stage

If a proposal has successfully passed all of the previous stages, it is then subject to the review stage with a final assessment of the need for public funding. When compared to the previous stages, this stage is less formalized, but it is also based on supporting questions.

5.3.1. Minimizing Deadweight Losses

Here, potential deadweight losses (i.e., beneficiaries who would also implement the project without funding) are considered in further detail. It is planned to check whether public funding is really required or whether potential beneficiaries would be willing to implement the project on their own. Current market trends can be an indication here, but (forthcoming) national and international policies should also be considered. It should be noted that the general issue of deadweight losses is also addressed indirectly during the review of market failures in the filtering stage. There, it serves to eliminate obvious cases of non-eligible proposals early on in the process, while, here, the assessment of the remaining proposals is expected to be more thorough.

5.3.2. Level and Extent of Funding

With regard to the level and extent of funding, legislation requires the effective use of public funds. As a basic rule for research projects, aid intensity for a beneficiary has a ceiling of 25% for experimental research under EU regulations. Depending on the specific situation, other ceilings and additional absolute thresholds may apply; these are illustrated by the DSS in flow diagrams. Additional national provisions for public funding may apply, as well as specific regulations for individual funding schemes. Thus, it is difficult to make generic provisions for the amount of funding. While the DSS gives a general guideline, appropriate funding levels have to be determined for each specific project proposal in this step, while taking into account the respective context. The DSS suggests which aspects to consider in this context.

6. Discussion Concerning the DSS and the Funding Process

If public funding of experimental research proposals becomes more important, there will be an increasing need to focus the limited public budgets on the most promising proposals and to respond to stricter requirements for transparency and documentation. The DDS has been suggested as a means to facilitate the evaluation process because it (a) provides a structured approach that bundles important aspects of the evaluation into a process; (b) helps to conduct a uniform evaluation of proposals that is largely independent of the individual evaluator; (c) enables less experienced evaluators to conduct an evaluation; and, (d) is a documented and transparent process.
In line with the requirement for a universal system, the DSS has been designed to be neutral with regard to technologies, draft or full proposals, and the organizational context of a funding line. However, it should be noted that it focuses on demonstration and model projects that can be perceived as a particular subcategory of experimental development. Furthermore, the evaluation stage with its main criteria relates to the German energy research program. While this is a limitation, it can also be expected that energy research activities in other countries generally follow similar programmatic criteria. To ensure that the DSS is simple, it relies on the same multi-criteria approach as already used in practice. Despite the resulting methodological limitations (e.g., [18]), the weighted sum model is one of the most simple and intuitive MCDA methods. Furthermore, with regard to the supportive character of the DSS, discussions in the workshops underlined that evaluation practitioners appreciate some liberty for interpretation when applying a DSS. This liberty is needed to avoid the risk of a prescriptive DSS that is too rigid to consider context-specific information from the expert’s point of view. Obviously, excessive liberty can come at the detriment of obtaining evaluation results that are largely independent of the evaluator. However, the documenting function of the DSS can also help to ensure transparency, as it requires its users to record how they evaluated particular aspects of a proposal. Concerning specific guidance, the DSS was perceived by some of the workshop participants as a helpful formalization of existing knowledge bundled together in one place. Adding specific guidance and interpretations of definitions and questions was considered to enhance the quality of the evaluations. The link to TRL to identify content that is eligible for funding within the two specific types of experimental development was also perceived as helpful. The non-compensatory character was implemented using minimum thresholds, as in [37] or [38]. Unfortunately, due to time constraints, this feature was not discussed in detail during the workshops. The consideration of funding ceilings was also only implemented as a simple flowsheet as workshop participants underlined the difficulties in bringing the multitude of context-dependent constraints together in a generalized concept.
Apart from the issues that are directly related to the requirements, it should be noted that, while this paper was being prepared, there was no funding program in place that allowed for us to verify the use of the DSS in practice. The results from the workshops as well as from an internal review with sample applications based on two modified real-world proposals seem to underline its practical usability, but adaptations to specific funding programs might be necessary.
Furthermore, it should be noted that the evaluation process here is based on the assumption of a ‘traditional’ peer review. The DSS could be extended by adding interactive elements, as mentioned in [35], or involving the public. In addition, the selection of experts and their interaction was not considered here. Finally, the funding process here focused mainly on energy research activities, but other funding mechanisms outside the R&D domain or completely different approaches to fostering market transitions could be considered as well.

7. Conclusions and Outlook

If energy policy increasingly is to support large-scale, close-to-market energy projects, decision support systems could be helpful to support funding bodies in selecting promising proposals. The aim of this contribution was to suggest a DSS for evaluating proposals for large-scale energy demonstration projects as public funding opportunities. The DSS that is presented in this paper is a multi-stage system spanning the evaluation process. It makes it possible to consider various evaluation criteria concurrently while using a multi-criteria approach.
The structure of the DSS underlines that the operational public decision-making process is not limited to a multi-criteria assessment of R&D proposals alone. Other considerations are also relevant for decision-making in practice. Among others, these include identifying the type of proposal, determining the state of development, the existence of market failures, or the existing funding portfolio. Furthermore, they underline that a practice-oriented DSS should complement existing knowledge without imposing a mechanistic tool. In that sense, the DSS that is suggested in this paper seems unique and is able to help structure the decision-making process.
The DSS suggested here can be characterized as a validated draft system in the sense that it has been discussed during workshops with practitioners and that it has been applied to sample applications in an internal system review. The discussions during the workshops and the findings from the system review suggest that it is useful when dealing with the complexity of the evaluation process. However, the DSS has not yet been reviewed in daily operation within a corresponding funding program. There is also the need to explore what adaptations might be required to a specific program and to what extent it could be used to help those submitting proposals to submit ‘better’ proposals.
Future research could focus on the weights and elimination criteria, for which various default sets were defined during the workshop, but were not investigated in further detail. Furthermore, although the DSS provides a structure for an exchange with external experts, the processes concerning their involvement could be further investigated. Finally, when it comes to choosing large energy projects for funding, further consideration should be given to further involving stakeholders in funding decisions (e.g., [46]) and the role of DSS in this context.

Author Contributions

M.F. contributed to the conceptualization of the DSS; P.V. contributed to the literature review, to the workshops and to the internal review of the paper; T.H. and S.H. elaborated and implemented the DSS and S.H. prepared the manuscript supported by T.H.

Acknowledgments

The authors gratefully acknowledge the support of the German Federal Ministry for Economic Affairs and Energy BMWi under grant agreement number 03ET4036A-C (August 2016 to April 2018). The responsibility for this publication lies with the authors. They express their gratitude to all the participants at the workshops, to their project partners at IZES gGmbH Saarbrücken, particularly Patrick Hoffmann and Horst Juri for their support and review of the DSS, to two anonymous reviewers for their helpful comments on the initial manuscript, and to the Fraunhofer Gesellschaft zur Förderung der angewandten Forschung e.V. for covering the costs of this open access publication.

Conflicts of Interest

The project was formally funded by German BMWi via PTJ who also participated in the workshops. The authors confirm that this did not affect the design of the DSS and declare no conflict of interest. This paper is based on a completely revised and extended contribution by the authors to the 2018 eceee Industrial Summer Study (Hirzel, S.; Hettesheimer, T.; Viebahn, P.; Fischedick, M. ‘Bridging the valley of death: A multi-staged multi-criteria decision support system for evaluating proposals for large-scale energy demonstration projects as public funding opportunities’. Accepted for the Proceedings of the 2018 eceee Industry Summer Study, 11–13 June 2018, Berlin).

Acronyms

AHPAnalytical Hierarchy Process
ARPA-EUS-American Advanced Research Agency-Energy
BMWiGerman Federal Ministry for Economic Affairs and Energy
CCSCarbon Capture and Storage
DSSDecision Support System
ELECTREElimination and Choice Expressing the Reality
EUEuropean Union
EUDPDanish Energy Technology Development and Demonstration Program
ETSEmissions Trading System
FFGAustrian Research Promotion Agency
H2020Horizon 2020
IEAInternational Energy Agency
LINMAPLinear Programming Technique for Multidimensional Analysis of Preference
MCDAMulti-Criteria Decision Aid
MADMMulti-Attribute Decision-Making
MAUTMulti-Attribute Utility Theory
MDSMultidimensional Scaling
MODMMulti-Objective Decision-Making
NER 300New Entrants Reserve 300
ORESTEOrganization, Ranking and Synthesis of Relational Data
PROMETHEEPreference Ranking Organization Method for Enrichment Evaluations
R&DResearch & Development
SAWSimple Additive Weighting
TOPSISTechnique for Order Performance by Similarity to Ideal Solution
TRLTechnology Readiness Level
USDUS-Dollar

References and Notes

  1. United Nations. Paris Agreement. 2015. Available online: http://unfccc.int/files/essential_background/convention/application/pdf/english_paris_agreement.pdf (accessed on 17 April 2018).
  2. European Commission. A Roadmap for Moving to a Competitive Low Carbon Economy in 2050. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. COM (2011) 112 Final. 2011. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2011:0112:FIN (accessed on 17 April 2018).
  3. Mankins, J.C. Technology readiness assessments: A retrospective. Acta Astronaut. 2009, 65, 1216–1223. [Google Scholar] [CrossRef]
  4. Lepori, B.; van den Besselaar, P.; Dinges, M.; van der Meulen, B.; Potì, B.; Reale, E.; Slipersaeter, S.; Theves, J. Indicators for comparative analysis of public project funding: Concepts, implementation and evaluation. Res. Eval. 2007, 16, 243–255. [Google Scholar] [CrossRef]
  5. OECD/IEA. Energy Technology RD&D Budgets. Overview: Paris, 2017. Available online: http://wds.iea.org/WDS/tableviewer/document.aspx?FileId=1578 (accessed on 17 April 2018).
  6. Roper, S.; Hewitt-Dundas, N.; Love, J.H. An ex ante evaluation framework for the regional benefits of publicly supported R&D projects. Res. Policy 2004, 33, 487–509. [Google Scholar] [CrossRef]
  7. Brezis, E.S. Focal randomisation: An optimal mechanism for the evaluation of R&D projects. Sci. Public Policy 2007, 34, 691–698. [Google Scholar] [CrossRef]
  8. Bointner, R. Innovation in the energy sector: Lessons learnt from R&D expenditures and patents in selected IEA countries. Energy Policy 2014, 73, 733–747. [Google Scholar] [CrossRef]
  9. OECD. Frascati Manual 2015. Guidelines for Collecting and Reporting Data on Research and Experimental Development; The Measurement of Scientific, Technological and Innovation Activities; OECD Publishing: Paris, France, 2015. [Google Scholar]
  10. European Commission. Commission Regulation (EU) No 651/2014 of 17 June 2014 declaring certain categories of aid compatible with the internal market in application of Articles 107 and 108 of the Treaty. Official Journal of the European Union, 2014; L187/1-78.
  11. Murphy, L.M.; Edwards, P.L. Bridging the Valley of Death: Transitioning from Public to Private Sector Financing. NREL/MP-720-34036. Available online: https://www.nrel.gov/docs/gen/fy03/34036.pdf (accessed on 29 March 2018).
  12. Markham, S.K.; Ward, S.J.; Aiman-Smith, L.; Kingon, A.I. The Valley of Death as Context for Role Theory in Product Innovation. J. Product Innov. Manag. 2010, 27, 402–417. [Google Scholar] [CrossRef]
  13. Bulathsinhala, N.A. Ex-ante evaluation of publicly funded R&D projects: Searching for exploration. Sci. Public Policy 2015, 42, 162–175. [Google Scholar] [CrossRef]
  14. Åhman, M.; Skjærseth, J.B.; Eikeland, P.O. Demonstrating climate mitigation technologies: An early assessment of the NER 300 programme. Energy Policy 2018, 117, 100–107. [Google Scholar] [CrossRef]
  15. European Commission. A European Strategy for Key Enabling Technologies—A Bridge to Growth and Jobs. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. COM (2012) 341 Final. 2012. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2012:0341:FIN (accessed on 17 April 2018).
  16. Braun, D. The role of funding agencies in the cognitive development of science. Res. Policy 1998, 27, 807–821. [Google Scholar] [CrossRef]
  17. Hsu, Y.-G.; Tzeng, G.-H.; Shyu, J.Z. Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Manag. 2003, 33, 539–551. [Google Scholar] [CrossRef]
  18. Zhu, W.-D.; Liu, F.; Chen, Y.-W.; Yang, J.-B.; Xu, D.-L.; Wang, D.-P. Research project evaluation and selection: An evidential reasoning rule-based method for aggregating peer review information with reliabilities. Scientometrics 2015, 105, 1469–1490. [Google Scholar] [CrossRef]
  19. Federal Ministry of Economics and Technology. Research for an Environmentally Sound, Reliable and Affordable Energy Supply. 6th Energy Research Programme of the Federal Government: Berlin, Germany. 2011. Available online: https://www.bmwi.de/Redaktion/EN/Publikationen/research-for-an-environmentally-sound-reliable-and-affordable-energy-supply.pdf?__blob=publicationFile&v=3 (accessed on 17 April 2018).
  20. Projektträger Jülich; Forschungszentrum Jülich. Konsultationsprozess für das 7. Energieforschungsprogramm. Available online: https://www.energieforschung.de/ (accessed on 17 April 2018).
  21. Hake, J.-F.; Fischer, W.; Venghaus, S.; Weckenbrock, C. The German Energiewende—History and status quo. Energy 2015, 92, 532–546. [Google Scholar] [CrossRef]
  22. Bundesministerium für Wirtschaft und Technologie; Bundesministerium für Umwelt, Naturschutz und Reaktorsicherheit. Energiekonzept für Eine Umweltschonende, Zuverlässige und Bezahlbare Energieversorgung. 28 September 2010: Berlin, Germany, 2010. Available online: https://www.bmwi.de/Redaktion/DE/Downloads/E/energiekonzept2010.pdf?__blob=publicationFile&v=3; see also: https://www.bmwi.de/Redaktion/EN/Dossier/energy-transition.html (accessed on 17 April 2018).
  23. Linton, J.D.; Walsh, S.T.; Morabito, J. Analysis, ranking and selection of R&D projects in a portfolio. R&D Manag. 2002, 32, 139–148. [Google Scholar] [CrossRef]
  24. Solak, S.; Clarke, J.-P.B.; Johnson, E.L.; Barnes, E.R. Optimization of R&D project portfolios under endogenous uncertainty. Eur. J. Oper. Res. 2010, 207, 420–433. [Google Scholar] [CrossRef]
  25. Verbano, C.; Nosella, A. Addressing R&D investment decisions: A cross analysis of R&D project selection methods. Eur. J. Oper. Res. 2010, 13, 355–379. [Google Scholar] [CrossRef]
  26. Vandaele, N.J.; Decouttere, C.J. Sustainable R&D portfolio assessment. Decis. Support Syst. 2013, 54, 1521–1532. [Google Scholar] [CrossRef]
  27. Henriksen, A.D.; Traynor, A.J. A Practical R&D Project-Selection Scoring Tool. IEEE Trans. Eng. Manag. 1999, 46, 158–170. [Google Scholar] [CrossRef]
  28. Duch-Brown, N.; Garcia-Quevedo, J.; Montolio, D. Assessing the Assignation of Public Subsidies: Do the Experts Choose the Most Efficient R&D Projects? Res. Appl. Econ. Netw. 2008, 12. [Google Scholar] [CrossRef]
  29. Biegelbauer, P.; Palfinger, T. Verschiedene Verfahren der Auswahl von Forschungsprojekten: Ein Vergleich von neun angewandten Forschungsförderungsorganisationen. Forsch. Polit. Strateg. Manag. Fo 2014, 7, 45–51. [Google Scholar]
  30. Biegelbauer, P.; Palfinger, T. Auswahlverfahren von Forschungsförderungsorganisationen im internationalen Vergleich. Fteval J. Res. Technol. Policy Eval. 2016, 41, 63–68. [Google Scholar]
  31. Biegelbauer, P.; Mayer, S.; Palfinger, T. Task Force Select. Final Report. 12 May 2016. Available online: http://www.taftie.org/sites/default/files/Taskforce_SELECT_final_report_0.pdf (accessed on 17 April 2018).
  32. Biegelbauer, P.; Palfinger, T.; Mayer, S. How Do Innovation Agencies Evaluate and Select Projects? A Comparison of 12 European Agencies. Fteval J. Res. Technol. Policy Eval. 2017, 43, 21–23. [Google Scholar]
  33. European Commission. Commission Decision of 3 November 2010 laying down criteria and measures for the financing of commercial demonstration projects that aim at the environmentally safe capture and geological storage of CO2 as well as demonstration projects of innovative renewable energy technologies und the scheme for greenhouse gas emission allowance trading within the Community established by Directive 2003/87/EC of the European Parliament and of the Council. Official Journal of the European Union, 2010; L290/39-48.
  34. ENS. Indkaldelse af Ansøgninger til Energiteknologisk Udviklings- og Demonstrationsprogram (EUDP) 2012-I. Available online: https://ens.dk/sites/ens.dk/files/info/nyheder/nyhedsarkiv/indkaldes-ansoegninger-eudp-2012/Indkaldelse_Januar_2012_DK.pdf (accessed on 19 April 2018).
  35. Bonvillian, W.B.; Atta, R. ARPA-E and DARPA: Applying the DARPA model to energy innovation. J. Technol. Transf. 2011, 36, 469–513. [Google Scholar] [CrossRef]
  36. ARPA-E. Facsimile Appearance to Create Energy Savings (FACES): Financial Assistance Funding Opportunity Announcement. Advanced Research Projects Agency—Energy (ARPA-E) U.S. Department of Energy. Available online: https://arpa-e-foa.energy.gov/FileContent.aspx?FileID=74a4a1bc-8f23-495e-8225-899144a24605 (accessed on 17 May 2017).
  37. European Commission. Horizon 2020. Work Programme 2016–2017. 20. General Annexes. European Commission Decision C (2017) 2468 of 24 April 2017. Available online: http://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2016-2017/annexes/h2020-wp1617-annex-ga_en.pdf (accessed on 17 April 2018).
  38. Österreichische Forschungsförderungsgesellschaft. Leitfaden für Leitprojekte. Version 2.3. Gültig ab 1 September 2017. Available online: https://www.ffg.at/sites/default/files/dok/il_leitprojekte_v23.pdf (accessed on 17 April 2018).
  39. Zhang, K.; Kluck, C.; Achari, G. A Comparative Approach for Ranking Contaminated Sites Based on the Risk Assessment Paradigm Using Fuzzy PROMETHEE. Environ. Manag. 2009, 44, 952–967. [Google Scholar] [CrossRef] [PubMed]
  40. Klein, R.; Scholl, A. Planung und Entscheidung. Konzepte, Modelle und Methoden Einer Modernen Betriebswirtschaftlichen Entscheidungsanalyse, 2nd ed.; Vahlen: München, Germany, 2011. [Google Scholar]
  41. Zimmermann, H.-J.; Gutsche, L. Multi-Criteria Analyse. Einführung in die Theorie der Entscheidungen bei Mehrfachzielsetzungen; Springer: Berlin, Germany, 1991. [Google Scholar]
  42. Belton, V.; Stewart, T.J. Multiple Criteria Decision Analysis. An Integrated Approach; Kluwer: Boston, MA, USA, 2002. [Google Scholar]
  43. Chen, S.-J.; Hwang, C.-L. Fuzzy Multiple Attribute Decision Making. Methods and Applications; Springer: Berlin Germany, 1992. [Google Scholar]
  44. Hwang, C.-L.; Yoon, K. Multiple Attribute Decision Making. Methods and Applications A State-of-the-Art Survey; Springer: Berlin/Heidelberg, Germany, 1981. [Google Scholar]
  45. Götze, U. Investitionsrechnung. Modelle und Analysen zur Beurteilung von Investitionsvorhaben, 6th ed.; Springer: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
  46. Jellema, J.; Mulder, H.A.J. Public Engagement in Energy Research. Energies 2016, 9. [Google Scholar] [CrossRef]
Figure 1. Conceptual illustration of public funding overcoming the valley of death after traditional project funding has ended.
Figure 1. Conceptual illustration of public funding overcoming the valley of death after traditional project funding has ended.
Energies 11 01357 g001
Figure 2. Administrative process for a successful proposal in Germany’s public energy research funding with amendments based on [19].
Figure 2. Administrative process for a successful proposal in Germany’s public energy research funding with amendments based on [19].
Energies 11 01357 g002
Figure 3. Morphological box to describe the setup for the decision support system (relevant aspects in grey).
Figure 3. Morphological box to describe the setup for the decision support system (relevant aspects in grey).
Energies 11 01357 g003
Figure 4. Overview of methods for Multi-Attribute Decision Making (with modifications based on [44,45]).
Figure 4. Overview of methods for Multi-Attribute Decision Making (with modifications based on [44,45]).
Energies 11 01357 g004
Figure 5. Overview of the decision support system’s general structure.
Figure 5. Overview of the decision support system’s general structure.
Energies 11 01357 g005
Figure 6. Overview of the six main and nineteen sub-criteria of the evaluation stage.
Figure 6. Overview of the six main and nineteen sub-criteria of the evaluation stage.
Energies 11 01357 g006
Figure 7. Tool to illustrate the evaluation stage, including (sample) weights for the main and sub-criteria, (sample) threshold values, and dummy proposals to illustrate elimination due to insufficient performance (bold red numbers: sub criterion below threshold; colored backfill: sum of sub-criteria in main criterion below threshold).
Figure 7. Tool to illustrate the evaluation stage, including (sample) weights for the main and sub-criteria, (sample) threshold values, and dummy proposals to illustrate elimination due to insufficient performance (bold red numbers: sub criterion below threshold; colored backfill: sum of sub-criteria in main criterion below threshold).
Energies 11 01357 g007
Table 1. Overview of selected approaches to evaluating research and development proposals within public research funding.
Table 1. Overview of selected approaches to evaluating research and development proposals within public research funding.
ProgramNER 300EUDPARPA-EH2020FFG
TargetDemonstration projects for renewable energies/CCSDemonstration and development projects for new energy technologiesEarly high-potential, high-impact energy technologiesInnovation, Research and Innovation, Coordination and Support Action in energy domainLead projects
RegionEuropeDenmarkUSAEuropeAustria
Criteriaeconomic performance4 criteria supported by sub-items4 criteria supported by sub-items3 criteria supported by sub-items4 criteria with 3 to 4 sub-criteria supported by sub-items
Weights--by criterionequal (default) by criterionby sub-criterion
Threshold---per criterion and for entire proposalper criterion
Aggregation--weighted sumweighted sumweighted sum
Remarksrequirements for knowledge sharing applyrefers to specific call from 2012refers to full applications in specific call ‘FACES’, additional criteria may applyadditional criteria apply in case of identical scoring-
NER 300: New Entrants Reserve 300; EUDP: Danish Energy Technology Development and Demonstration Program; ARPA-E: US-American Advanced Research Agency-Energy; H2020: EU’s Innovation and Research Program Horizon 2020; FFG: Austrian Research Promotion Agency; CCS: Carbon Capture and Storage.
Table 2. Example for a dichotomous supporting question used to verify the project type.
Table 2. Example for a dichotomous supporting question used to verify the project type.
Does the Project Foresee Broad Transfer Activities Regarding Its Innovative Parts? ☐ Yes
☐ No
e.g.,
  • Will a transfer concept be developed and implemented?
  • Are there planned regular events to accompany/present the project?
  • Does the proposal contain reporting activities to the public or relevant market participants that considerably exceed the transfer activities of traditional applied research activities (e.g., a structured press campaign, participations in trade fairs, social-media presence, regular report in applied journals, patents, other publications?)
  • Is it possible to visit the demonstrator?
  • Does the measure promise to have a broad impact beyond the specific case at hand?
  • Is there a realistic possibility that the findings can be transferred to other companies or users?
Table 3. Summary of key commonalities and differences between demonstration and model proposals used in the decision support system (DSS) to identify the proposal type.
Table 3. Summary of key commonalities and differences between demonstration and model proposals used in the decision support system (DSS) to identify the proposal type.
Demonstration Project…Model Project…
Specificcharacteristics… contributes to enhancing norms, standards or approval processes
… contributes to research on or improvement of acceptance
… is based on single-location implementation or complementary parts in several locations
… has realistic remaining lead for development of less than five years
… largely follows existing regulatory framework
… contributes to improving public acceptance
… is based on multiple implementations of similar concepts and technologies
… provides realistic suggestion for broad utilization after conclusion
Commoncharacteristics… foresees broad set of transfer activities for its innovative parts
… is carried out by a team with proficient competence on knowledge transfer
… is first-of-its-kind activity
… can yield economically viable solutions after project conclusion
Table 4. Sample excerpt (specific guidance omitted) from the guideline for the criterion ‘quality and efficiency of the working plan’.
Table 4. Sample excerpt (specific guidance omitted) from the guideline for the criterion ‘quality and efficiency of the working plan’.
DefinitionTransparency and quality of the procedure for achieving the objectives of the overall project and its parts
PremisesThe higher the quality and efficiency of the working plan, the more attractive the project
Main criterionQuality of the project
PerspectivePerspective of the funding body
Supporting Questions
  • Does the work plan follow a logical and purposeful structure that makes the project feasible?
  • Does the work plan have a plausible time plan?
  • Are all the work plan phases really relevant to the project?
  • Is the methodology for the different phases of the work plan plausible and does it contribute to achieving the targets?
  • Have suitable measures been taken to monitor the progress of the project?
VerificationHow reliable, justified and transparent is the judgement of this criterion?

Share and Cite

MDPI and ACS Style

Hirzel, S.; Hettesheimer, T.; Viebahn, P.; Fischedick, M. A Decision Support System for Public Funding of Experimental Development in Energy Research. Energies 2018, 11, 1357. https://doi.org/10.3390/en11061357

AMA Style

Hirzel S, Hettesheimer T, Viebahn P, Fischedick M. A Decision Support System for Public Funding of Experimental Development in Energy Research. Energies. 2018; 11(6):1357. https://doi.org/10.3390/en11061357

Chicago/Turabian Style

Hirzel, Simon, Tim Hettesheimer, Peter Viebahn, and Manfred Fischedick. 2018. "A Decision Support System for Public Funding of Experimental Development in Energy Research" Energies 11, no. 6: 1357. https://doi.org/10.3390/en11061357

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop