Research on Analyzing the Efficiency of R&D Projects for Climate Change Response Using DEA–Malmquist

Abstract

In responding to climate change, the world is focusing on technology development. Korea also continues to invest in R&D to reduce greenhouse gas emissions and adapt to climate change. However, compared to the government’s continuous investment in R&D, there is a lack of systematic analysis of R&D investment performance. Rather than simply reducing and increasing the investment in R&D to respond to climate change in terms of high and low efficiency, we aim to improve the efficiency of national R&D projects by analyzing the causes of low efficiency and deriving improvement directions. In this study, data envelopment analysis (DEA) was used to analyze the efficiency of climate change response technology development projects conducted by the Ministry of Science and ICT in Korea. The efficiency of 1500 projects conducted during the 2014–2020 period was analyzed from a static and dynamic perspective, focusing on project information. Through static efficiency analysis, total efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE) were measured, and the causes of inefficiency were identified. In addition, the results of the dynamic efficiency analysis using the Malmquist analysis were presented, and alternatives for each field were suggested by presenting the static and dynamic results as an integrated model.

1. Introduction

To address climate change issues, the international community began discussions by signing a climate change agreement. The Kyoto Protocol of 1997 and the Paris Agreement of 2015 were produced as a result of the process in which 198 developing and developed countries gathered to find solutions to climate change. In particular, as shown in Figure 1, it was agreed in the Paris Agreement that finance, technology development and transfer, and capacity building should be determined as the means to achieve the goals of reducing greenhouse gas emissions and adapting to climate change [1]. In other words, new technologies that can replace current fossil fuels that emit greenhouse gases should be developed and applied; developing countries need to avoid using low-priced fossil fuels in the process of growing; more advanced countries should transfer their emission-free technologies to less developed countries and help educate them so that those countries can upgrade the technologies themselves.

Since joining the United Nations Framework Convention on Climate Change in 1993, South Korea has made various efforts to address climate change through its green growth branding, including the establishment of the Global Green Growth Institute (GGGI) and the hosting of the Green Climate Fund (GCF) in South Korea [3]. In particular, South Korea considers technology development as a key means of addressing climate change and continues to invest in developing innovative technologies to replace coal and oil, which have high levels of carbon emissions [3].

The Ministry of Science and ICT is the central administrative agency in charge of establishing national science and technology policy, research and development, broadcasting and communication policy, and the information and communication industry, and it has been carrying out the climate change response technology development project since 2009. This project was created to strengthen basic research to respond to climate change and drive low-carbon green growth, and about Korean won 550 billion has been invested up to 2021 [4]. Above all, the implementation of the Climate Change Response Technology Development Promotion Act [5] in 2021, which clarified the basis for the climate change response technology development project, further strengthened the need for technology development in the climate crisis. The importance of climate R&D investment has been justified by the urgency of the climate crisis and the importance of international technology development, but the evaluation of its efficiency has not been emphasized. In addition, it is even more difficult to discuss the efficiency of climate technologies that are not yet economically viable for the private sector to invest in and develop on their own. Evaluating the efficiency of climate R&D investment is expected to be a useful criterion for selecting and focusing on key technologies to respond to climate change and to prioritize investments.

R&D efficiency is highly important, especially given that the government’s R&D budget is limited. R&D efficiency can be defined as utilizing the given resources as effectively as possible to achieve technological innovation, which in turn plays a major role in improving the competitiveness of a company or country [6]. Whether or not a R&D project has achieved effective performance in relation to its investment can be determined by analyzing the output for the input, and the current study aims to analyze the investment efficiency of climate change response technology development projects using the data envelopment analysis (DEA) model. The present study is organized as follows: Section 2 reviews DEA theory and related studies; Section 3 describes the research flow and data; Section 4 presents the efficiency analysis results of climate change response R&D projects; Section 5 draws conclusions and implications from the analysis results.

2. DEA Model and Previous Research

2.1. Data Envelopment Analysis (DEA) Model

Parametric or non-parametric methodologies are used to measure the efficiency of policies or R&D. Parametric methodologies make it difficult to derive an accurate cost function, so it is difficult to objectively prove the correlation between inputs and outputs, and it is also difficult to analyze efficiency by considering multiple inputs and outputs simultaneously. DEA is a non-parametric model that has emerged to overcome these limitations and effectively evaluate productivity [7]. The DEA method for analyzing the relative efficiency of organizations originated from the concept of efficiency proposed by Farrell (1957) [8]. Farrell defined the efficiency of an organization as the ratio of outputs to inputs, which Charnes et al. (1978) [9] developed into a linear programming method and developed the CCR model from, the first DEA model.

The advantages of DEA over parametric methodologies are, first, that it can take multiple inputs and outputs and aggregate them into a single efficiency value. Second, efficiency comparisons can be made between decision-making units (DMUs). Efficient and inefficient DMUs can be identified along with their causes. Third, there is no need to define a specific type of function, so no prerequisite such as a normal distribution is required [10,11]. Lastly, the DEA model estimates the weights of inputs and outputs, so there is no need to subjectively determine the weights of factors with varying units [12].

DEA integrates various inputs and outputs to derive efficiency, while providing information on the contribution of each indicator to efficiency and giving insights into the influencing factors of high- and low-efficiency groups. The DEA model is largely represented by logics that reflect the CCR model (Charnes–Cooper–Rhodes), which assumes constants returns to scale (CRS), and the BCC model (Banker–Charnes–Cooper), assuming variable returns to scale (VRS). Assuming that there are returns to scale, constant returns to scale (CRS) efficiency can be divided into variable returns to scale (VRS) efficiency and scale efficiency (SE) as shown in Figure 2. VRS efficiency measures the optimal combination of resources with which to optimize production, while SE indicates whether or not the minimum cost is used during production. The efficiency of the CCR model is represented by total efficiency (TE), TE = AB/AD, and the pure technical efficiency (PTE) of the BCC model is PTE = AC/AD, as shown in Figure 2. Using the efficiency values of both models, scale efficiency (hereinafter SE) can be calculated, which represents the optimality of scale. The formula for SE is TE/PTE = AB/AC. By comparing SE and PTE, it is possible to determine whether or not DMU inefficiency is due to technical or scale factors (Figure 2) [7]. The DEA model can be input-oriented or output-oriented, and the input-oriented model aims to fix the output factors and minimize the input factors. The output-oriented model, on the other hand, aims to maximize the output while keeping the inputs fixed [7].

2.2. Malmquist Productivity Growth Index

Malmquist analysis was developed to analyze efficiency changes due to technological change using the concept of productivity by applying the DEA technique to panel data [13]. After being proposed by Malmquist (1953) [14], Cave et al. (1982) [15] first started using the Malmquist productivity index (MPI) to analyze data obtained in the form of panel data with the DEA method. Fare et al. (1994) [16] later began to use it as a useful measure with which to analyze productivity changes by decomposing the productivity index.

Productivity is defined as output divided by input, and the change in productivity is expressed as the ratio of productivity at the current time point to the change in productivity between two time points. The MPI utilizes the concept of distance function to define the change in production, which can be easily calculated by modifying the DEA model because the distance function can be considered the reciprocal of the efficiency indicator derived from the CCR model.

The input-oriented MPI represents the productivity change index in periods t and (T + 1). M^t is defined as the distance to the observation at time t + 1 in relation to the distance to the observation at time t relative to the efficiency boundary at time t. M^t+1 is defined as the distance to the observation at time t + 1 relative to the distance to the observation at time t based on the efficiency boundary at time t + 1. In Equation (1), D denotes the efficiency boundary generated by the DMU at time t, and D^t+1 denotes the efficiency boundary constructed by the DMU at time t + 1. D(x^t, y^t) means the efficiency of DMU at time t based on the production boundary created by DMU at time t, and D(x^t+1, y^t+1) means the efficiency between DMUs at time t + 1 based on the efficiency boundary created by DMU at time t. Here, if the result is MPI > 1, it refers to productivity increase, MPI < 1 indicates productivity decrease, and MPI = 1 means there is no change [17]. The equation is as follows.

$$M_I^{t,t+1}={\left[\frac{D_I^t\left(x^{t+1},y^{t+1}\right)}{D_I^t\left(x^t,y^t\right)}\times \frac{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}{D_I^{t+1}\left(x^t,y^t\right)}\right]}^{1/2}$$

(1)

The Malmquist index, MPI, as shown in Equation (2), can be decomposed into the technological change index (TCI) and technical efficiency change index (TECI), and is expressed through Equation (3).

$$M_I^{t,t+1}=TECI\times TCI,$$

(2)

$$M_I^{t,t+1}=\frac{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}{D_I^t\left(x^t,y^t\right)}\times {\left[\frac{D_I^t\left(x^{t+1},y^{t+1}\right)}{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}\times \frac{D_I^t\left(x^t,y^t\right)}{D_I^{t+1}\left(x^t,y^t\right)}\right]}^{1/2}$$

(3)

Technology Efficiency Change Index (TECI) is decomposed into Scale Efficiency Change Index (SECI) and Pure Efficiency Change Index (PECI) as in Equation (4), and is explained in Equation (5).

$$M_I^{t,t+1}=PECI\times SECI\times TCI,$$

(4)

$$M_I^{t,t+1}=\begin{array}{c}\frac{V_I^{t+1}\left(x^{t+1},y^{t+1}\right)}{V_I^t\left(x^t,y^t\right)}\\ \times \left[\frac{V_I^t\left(x^t,y^t\right)}{D_I^t\left(x^t,y^t\right)}\times \frac{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}\right]\times {\left[\frac{D_I^t\left(x^{t+1},y^{t+1}\right)}{D_I^{t+1}\left(x^{t+1},y^{t+1}\right)}\times \frac{D_I^t\left(x^t,y^t\right)}{D_I^{t+1}\left(x^t,y^t\right)}\right]}^{1/2}\end{array}$$

(5)

Technical efficiency, TECI, indicates that the closer to the production boundary a technology is, the more efficient it is, and a TECI > 1 at time t + 1 indicates an increase in efficiency. A TCI greater than 1 indicates technological progress, while a TCI less than 1 indicates a technological decline. SECI is the ratio of the input distance function of VRS technology to that of CRS technology, and it is the geometric mean of the evaluations made at time t and t + 1 [18].

2.3. Previous Research

Early research using DEA was heavily utilized to evaluate the efficiency of public organizations such as healthcare, education, and government. Since then, it has been applied to a wide range of industries, including finance and banking, energy, and transportation. With the increasing budgetary investment in national R&D projects and the increasing importance of their performance, efficiency analysis of national R&D projects is being conducted using DEA [6]. CCR and BCC models have been used as representative DEA analysis techniques, and DEA/AR [19] and DEA/PS [20] models have been used to overcome the problem of weights assigned to input and output factors, which is a limitation of the basic model [21].

As national R&D projects cover a wide range of fields, efficiency analysis was conducted by focusing on specific fields rather than the whole range of them. As shown in

Table 1. Previous research on analyzing the efficiency of national R&D projects.

Author(s)	Research Area	Input Variables	Output Variables	Research Methodology
Lee, H. et al. (2016) [6]	Core defense technology R&D projects	Research funding, number of researchers, research period	Journal papers, patents, amount of commercialization	DEA
Woo, C. (2019) [18]	New and renewable energy technologies	Government and private research funding	Journal papers, patents, employment creation	DEA
Nam, I. et al. (2008) [19]	National R&D entities	Number of researchers, research funding, equipment	R&D performance, technology transfer performance, etc.	DEA/AR
Kim, K. (2016) [20]	Industry core technology development	Number of researchers, research funding, research period	Patents, journal papers, revenue	DEA/PS
Shin (2018) [21]	Renewable energy projects	Research funding	Journal papers, patents, technology fees, amount of commercialization	DEA
Baek, C. (2014) [22]	Government’s R&D investment in renewable energy	Research funding	Journal papers, patents	DEA
Park, S. et al. (2015) [23]	Defense technologies	Research funding, number of researchers	Patents, amount of commercialization	DEA
Kim, T. et al. (2009) [24]	Nuclear R&D projects	Research funding, number of researchers	Journal papers, conferences, patents, workforce development, technology diffusion	DEA
Ga, S. (2020) [25]	Environmental R&D projects	Government research funding, number of researchers	Journal papers, patents	DEA
Nam, K. H. (2021) [26]	National environmental R&D programs	Research funding, number of researchers, research period	Journal papers, patents, sales	DEA–MPI
Hur, S. et al. (2013) [27]	National horticultural and herbal R&D	Research funding	Patents, technology transfer, policy suggestions, agricultural use	DEA–MPI
Nam, K. et al. (2021) [28]	National environmental R&D programs	Research funding, number of researchers	Journal papers, patents, amount of revenues	DEA–MPI
Choi, K. et al. (2014) [29]	Energy R&D program	Research funding	Journal papers, patents, technology fees	DEA
Lim, Y. (2020) [30]	Defense Basic project	Research funding, number of researchers, R&D period	Journal papers, patents	DEA
Soh, A. et al. (2015) [31]	National convergence R&D projects	Research funding, number of researchers	Journal papers, patents, technology fees	DEA–MPI
Park, C. et al. (2018) [32]	6T National R&D	Government research funding, number of projects	Journal papers, patents, technology fees, amount of commercialization	DEA
Yang, E. et al. (2019) [33]	National R&D in fundamental sciences	Research funding, number of researchers	Journal papers, patents	DEA–MPI
Park, H. (2014) [34]	BT, NT Technologies	Research funding, number of researchers, research period, number of journal publications by principal investigator	Journal papers, patents, conferences, workforce development	DEA
Park, S. (2015) [35]	Defense technologies	Research funding, number of researchers	Patents, amount of commercialization	DEA

below, a number of studies were conducted on the basis of projects or tasks targeting specific fields such as renewable energy [18,21,22], defense technology [6,23], nuclear energy [24], the environmental field [25,26] and the agricultural field [27], and the DMUs in the analysis were analyzed based on detailed technology or project types (performing entity and research stage) within the field. The key input variables included research funds, number of researchers, and input period, and the key output variables were number of journal papers, patents, commercialization of technology fees, and sales.

According to the results of previous studies in the field of climate change response, the DEA efficiency index is often used to analyze the efficiency of national R&D projects, and with sufficient time series data, analysis is carried out in conjunction with the MPI. Nam et al. (2021) [28] analyzed the investment appropriateness of individual projects and the causes of inefficiency by linking DEA and the MPI for projects in the environmental field. In the field of renewable energy, Woo (2019) [18] classified DMUs for each technology and used the BCC model to compare the efficiency of each technology and analyze the causes of inefficiency. Choi et al. (2014) [29] analyzed the appropriateness of the energy R&D investment scale and sources of inefficiency using the BCC model for energy technology development projects, while Shin (2018) [21] presented the efficiency of renewable energy technology development projects. Lim (2020) [30] analyzed the CCR and BCC models for technologies in the defense field and suggested causes of inefficiency and improvement measures, but did not conduct a dynamic analysis. Soh et al. (2015) [31] presented the investment direction of convergence research projects by conducting efficiency and Malmquist analysis on 77 projects related to convergence technology development.

3. Research Methods

3.1. Research Framework

The present study aims to measure the efficiency of the climate change response technology development project conducted by the Ministry of Science and ICT of South Korea by selecting a field as a DMU (decision making unit) according to the climate technology classification system. The overall process of the study is shown in the figure below. First, the input and output variables of the projects were preprocessed to match the research purpose, and correlation analysis was conducted for each variable. In addition, static (DEA) and dynamic efficiency (Malmquist) analyses were conducted, and based on the results, results are presented by classifying the types of efficiency according to the integrated management model (Figure 3).

The static efficiency analysis was carried out utilizing the Charmes–Cooper–Rhodes (CCR), Banker–Charnes–Cooper (BCC), and scale efficiency models. For dynamic efficiency, the Malmquist DEA model was used. The government projects were analyzed based on the output model because there are limitations in active changes regarding input variables.

3.2. Data Collection

The data for this study were collected from the National Science & Technology Information Service (NTIS) and included information on 1502 projects and performance data of the climate change response technology development project for the period of 2014–2020. It would appear meaningful to organize 1502 sub-projects into DMUs and find the ratio of efficient projects. The aim of the current study, however, was to evaluate efficiencies by technology unit so that the results of the efficiency analysis could be utilized as a basis for R&D planning. Therefore, in the present study, each sub-project was classified according to the types of technology and the resultant nine technical categories were analyzed by each year (2014–2020), giving a total of 64 DMUs. For the DEA analysis, R&D projects were classified into nine technologies according to the climate technology classification system and DMUs were organized for each year. This study was conducted with the common goal of developing technologies to respond to climate change and satisfies the homogeneity criterion of DMUs proposed by Golany and Roll (1989) [36] by providing the same items of data. The input and output factors utilized in the DEA analysis were derived based on the previous studies.

3.3. Index Selection

The input and output variables of the DEA model were selected through reviewing previous studies that measured the efficiency of national R&D projects and the input and output variables are summarized in

Table 2. Inputs and outputs utilized in DEA.

Index	Variables	Description
Input	Government R&D investment (billion KRW)	Total research funding invested in 2014–2020
	Number of researchers	Number of researchers deployed in 2014–2020
Output	Number of journal papers	Number of project-related journal papers published in 2014–2020 (domestic and international)
	Number of patents	Number of project-related patent applications and registrations in 2014–2020 (domestic and international)

.

In previous studies, the number of researchers and amount of research funding were used as indices of labor and capital, whereas numbers of journal papers and patents were used as key output variables. In the present study, consequently, research funding and number of researchers were selected as input variables, and number of papers and patents were selected as output variables. In general, research expenses and number of researchers are representative inputs for estimating R&D efficiency. In addition, journal papers representing academic achievements, intellectual property resulting from technology development, and patents as R&D outputs to introduce products to the market can be seen as representative indices.

Research on the appropriate number of reliability DMUs varies, as shown in

Table 3. The case for the right number of Decision Making Units (DMUs).

Author(s)	Appropriate Number of DMUs
Banker et al. (1984) [37]	No. of DMUs > 3 × (no. of input variables + no. of output variables)
Golany and Roll (1989) [36]	No. of DMUs > 2 × (no. of input variables + no. of output variables)
Boussofian et al. (1991) [38]	No. of DMUs > (no. of input variables + no. of output variables)

. Banker et al. (1984) [37] suggested that the number of DMUs to be evaluated should be three times larger than the sum of the number of inputs and outputs, Golany and Roll (1989) [36] validated that the number of DMUs should be at least twice as large as the sum of the number of inputs and outputs to be discriminative, whereas Boussofiane et al. (1991) [38] argued that the number of DMUs should be the product of the minimum number of inputs and outputs to solve the problem of decreasing discrimination among DMUs.

In the current study, DMUs were constructed by classifying project information in 2014–2020 into nine technologies; the values were organized for each year, and a total of 64 DMUs were analyzed separately, hence all of the conditions proposed for DMUs by the previous studies were met.

The descriptive statistics of the DMU inputs and outputs used in the analysis are shown in

Table 4. Basic statistics for inputs and outputs.

Index	Variable	Obs	Mean	Std. Dev.	Min	Max
Input	R&D investment (billion KRW)	63	8028.3	7687	80	27,278.0
	Number of researchers	63	166.6	142	1	506.0
Output	Number of journal papers	63	40.1	35	1.5	133.6
	Number of patents	52	1143.4	1410	2	4920.0

. In the case of patents, there is a zero value, but Golany et al. (1989) [36] and Bowlin (1988) [39] suggest that even if a variable has a value of ‘0’, it is still possible to include it in the analysis if there is a positive value in at least one of the various input and output variables of the DMU. They suggested replacing a very small positive number with an absolute constant as an alternative method. Thus, ‘10⁻³’ was used to replace the zero value in the current analysis.

In statistical analyses, it is a common procedure to examine the causal relationship between input and output variables with a correlation analysis [26]. However, since DEA analysis is a non-parametric method, results of correlation analysis in this case should only be used as a reference value. According to the person correlation analysis, the correlation between the variables used in the present study is statistically significant at 0.01 level, indicating significant correlations among the variables (

Table 5. Correlate input and output variables.

Variable	R&D Investment (Billion KRW)	Number of Researchers	Number of Journal Papers	Number of Patents
R&D investment (billion KRW)	Number of researchers	-	-	-
Number of researchers	Number of journal papers	1	-	-
Number of journal papers	Number of patents	0.9187 ***	1	-
Number of patents	0.6429 ***	0.5214 ***	0.5432 ***	1

Significance level: *** 0.01.

).

4. Results

In the present study, output-oriented CCR and BCC models were used to evaluate the efficiency of R&D projects. For efficiency improvement and policy implications, the output-oriented model was used to increase output rather than decrease input factors. In addition, productivity changes over the seven-year period from 2014 to 2020 were analyzed through Malmquist’s productivity change index. Finally, the integrated efficiency model proposed by Nam and Park (2021) [37] was utilized to identify static and dynamic efficiency. The results of the analysis for each of the 64 DMUs are presented in Appendix A, and the present section focuses on the average values for interpretation.

4.1. Status of R&D Projects

The investment in climate change response technology development projects started with 43 billion KRW and increased to 99 billion KRW as of 2020. The number of projects continued to expand from 169 to 247. The number of journal papers also grew dramatically from 353 to 481, and the number of patents grew from 84 to 19,817. However, the increase in the number of research staff was lower compared to the other variables (

Table 6. Trends in inputs and outputs of climate change response R&D projects.

Year	R&D Investment	Number of Projects	Number of Researchers	Number of Journal Papers	Number of Patents
2014	43,221	169	1502	353	84
2015	45,385	155	1386	314	117
2016	52,286	138	761	271	106
2017	78,738	246	1546	337	12,983
2018	86,582	264	1612	365	11,883
2019	99,598	283	1878	405	14,465
2020	99,973	247	1811	481	19,817
Sum	505,783	1502	10,496	2526	59,455

).

In terms of number of achievements per unit of R&D investment, climate change prediction and monitoring had the highest number of journal papers (2.94) per billion KRW, followed by other technologies (2.74) and energy demand (2.46). Patent outputs per billion KRW were highest in the water sector with 0.19 patents, followed by other technologies (

Table 7. Number of journal papers and patents per unit of R&D investment.

Technology	No. of Journal Papers per Billion KRW	No. of Patents per Billion KRW
01. Renewable Energy	1.77	0.08
02. New Energy	2.11	0.08
03. Energy Storage	1.17	0.07
04. Energy Demand	2.46	0.04
05. GHG Fixation	2.30	0.09
06. Water	1.13	0.19
07. Climate Change Prediction and Monitoring	2.94
08. Multi-Sector Overlap	1.77	0.06
09. Other Technologies	2.74	0.15
Average	2.00	0.09

).

4.2. Qualitative Analysis

In the current study, technical efficiency (TE) under the CCR model, pure technical efficiency (PTE) under the BCC model, and scale efficiency (SE = TE/PTE) are presented. If PTE is smaller than SE, the inefficiency is mainly due to technical aspects, and if SE is smaller than PTE, the inefficiency is mainly due to scale aspects [29]. Returns to scale can be categorized into three different types depending on how the output changes when all factors of production are increased simultaneously: constant returns to scale (CRS) occur when output increases proportionally; declining returns to scale (DRS) happen when output decreases to a larger extent compared to the input; increasing returns to scale (IRS) occur when output increases to a greater degree than the input [17,29].

Based on the efficiency measurement results of 64 DMUs for 1502 projects carried out in the 2014–2020 period of the climate change response technology development project (Figure 4), the overall average of technology efficiency (TE) is low at 41%. The pure technology efficiency (PTE) was 64% and the scale efficiency (SE) was 66%, indicating that more than half of the DMUs were efficient. The overall trend of efficiency during the analysis period shows that both TE and PTE exhibit fluctuations, whereas SE (scale efficiency) shows a declining pattern.

Looking at the average values of efficiency measures by technology, as shown in

Table 8. Average DEA analysis results by technology.

Technology	Average: TE	Average: PTE	Average: SE	No. of Causes of Inefficiency
				Scale Factors	Technical Factors	Total
01. Renewable Energy	0.36	0.77	0.47	6	1	7
02. New Energy	0.40	0.60	0.62	2	4	6
03. Energy Storage	0.56	0.86	0.65	7	-	7
04. Energy Demand	0.56	0.65	0.85	2	4	6
05. GHG Fixation	0.34	0.86	0.39	7	-	7
06. Water	0.50	0.52	0.95	-	6	6
07. Climate Change Prediction and Monitoring	0.26	0.28	0.85	-	7	7
08. Multi-Sector Overlap	0.52	0.75	0.67	4	2	6
09. Other Technologies	0.22	0.48	0.49	3	4	7
Average	0.41	0.64	0.66	31	28	59

, energy storage and energy demand showed the highest TE value, with 56% efficiency, while multi-sector overlap appeared to have 52% efficiency. Regarding PTE, energy storage and GHG fixation were 86% efficient and renewable energy showed 77% efficiency. Scale efficiency was high for water (95%), climate change prediction and monitoring (85%), and energy demand (85%). For RTS, 85% (55 out of 64) of the DMUs are in IRS (increasing returns to scale), indicating that performance is expected to improve with mid- to long-term investments.

In addition, the causes of inefficiency can be analyzed using SE and PTE. PTE (pure technical efficiency) is calculated as the product of technical efficiency (TE) and scale efficiency (SE); if PTE > SE, the technical efficiency is higher than the scale efficiency, and it can be interpreted that the size of the scale has not affected the technical efficiency. Therefore, the scale factor is the cause of the inefficiency, and if PTE < SE, the scale efficiency is higher than the technical efficiency, and it can be interpreted that the cause of the inefficiency is a technical factor.

The results suggest that inefficiency occurs primarily due to scale factors in renewable energy, energy storage, GHG fixation, and multi-sector overlap. In addition, water and climate change prediction and monitoring technologies are interpreted as areas that require efficiency improvements with regard to technical factors rather than scale factors such as investment.

Figure 5 shows comparisons among average values of TE, PTE, and SE by technology. energy storage and multi-sector overlap showed high values in both TE and PTE, indicating that these two categories have high efficiencies when technology alone is taken into account and also when it is considered together with scale effects. renewable energy and GHG fixation have the highest discrepancies between TE and PTE, indicating that both technologies have low scale efficiencies. Water and climate change prediction and monitoring are sectors with the same efficiency based on TE and PTE, and their SE values are close to 1, meaning that the current production scale is almost adequate.

As shown in

Table 9. Reference numbers of efficient DMUs.

DMU	No. of References to Technologies with TE Value of 1	No. of References to Technologies with PTE Value of 1
03. Energy Storage 2016	-	33
03. Energy Storage 2015	-	24
05. GHG Fixation 2020	-	21
01. Renewable Energy 2020	-	20
08. Multi-Sector Overlap 2019	45	19
02. New Energy 2014	46	13
05. GHG Fixation 2016	-	10
06. Water 2016	56	9
03. Energy Storage 2014	-	8
04. Energy Demand 2014	5	3
04. Energy Demand 2016	-	2

below, when looking at efficient projects according to the DMU, there are four projects with a TE value of 1 based on CCR (new energy 2014, energy demand 2014, water 2016, and multi-sector overlap 2019) and 11 projects with a PTE of 1 based on BCC (renewable energy 2020, new energy 2014, energy storage 2014, 2015, 2016, energy demand 2014, 2016, GHG Fixation 2016, 2020, water 2016, and multi-sector overlap 2019). When looking at the number of references to technologies with an efficiency of 1, water 2016, new energy 2014, and energy demand 2014 were the top three according to TE. Based on PTE, energy storage 2016 showed the highest number of references, 33, followed by energy storage 2015 and GHG fixation 2020.

4.3. Dynamic Analysis

The following table shows the average of the time series productivity change trends for nine technology fields using continuous data for the period 2014–2020 (

Table 10. Average values of productivity over time periods.

Period	Average: MPI	Average: TC	Average: TEC	Average: PTE	Average: SEC
2014–2015	1.04	1.03	0.96	0.98	1.01
2015–2016	1.74	0.75	2.39	0.84	0.86
2016–2017	6.40	2.30	2.89	1.48	1.51
2017–2018	0.90	1.21	0.74	1.11	1.08
2018–2019	0.98	0.88	1.10	0.94	0.95
2019–2020	1.76	1.18	1.31	0.98	1.21
Average	2.14	1.23	1.57	1.05	1.10

). The productivity change rate of the climate change response technology development projects is high, increasing by 214% on average. When examining the TC and TEC that influenced the rate of productivity change, it was found that the technical efficiency change index (TECI, 157%) caused a larger change than the technical change rate (TC, 123%) did. In other words, the increase in productivity was driven by external changes in conditions rather than changes made within technology itself. The change rate of PTE was 5%, and that of SEC was 10%, but the fluctuations over the period were low.

As shown in Figure 6 below, most of the technologies are highly productive in the 2016–2017 period. While the average of all MPIs is 2.14, the MPI for the 2016–2017 period is 6.50, indicating that productivity growth was actually driven by internal TC at a level of 2.3, but the external change of circumstances, TEC, was the highest over the entire period at 2.89.

Based on the MPI index, as summarized in

Table 11. DMU rank by MPI.

Rank	Technology and Time Period	MPI	Rank	Technology and Time Period	MPI
1	02. New Energy 2016–2017	18.17	28	04. Energy Demand 2018–2019	1.16
2	01. Renewable Energy 2016–2017	13.83	29	02. New Energy 2017–2018	1.15
3	05. GHG Fixation 2016–2017	8.95	30	05. GHG Fixation 2018–2019	1.12
4	06. Water 2019–2020	7.01	31	03. Energy Storage 2018–2019	1.10
5	03. Energy Storage 2016–2017	5.03	32	01. Renewable Energy 2014–2015	1.06
6	08. Multi-Sector Overlap 2016–2017	4.29	33	03. Energy Storage 2014–2015	1.02
7	04. Energy Demand 2015–2016	3.54	34	04. Energy Demand 2019–2020	1.01
8	06. Water 2015–2016	3.05	35	01. Renewable Energy 2018–2019	0.96
9	06. Water 2016–2017	2.56	36	05. GHG Fixation 2017–2018	0.96
10	04. Energy Demand 2016–2017	2.37	37	04. Energy Demand 2017–2018	0.96
11	03. Energy Storage 2015–2016	2.19	38	08. Multi-Sector Overlap 2017–2018	0.92
12	08. Multi-Sector Overlap 2014–2015	2.05	39	01. Renewable Energy 2017–2018	0.89
13	02. New Energy 2019–2020	1.89	40	07. Climate Change Prediction and Monitoring 2016–2017	0.87
14	05. GHG Fixation 2014–2015	1.85	41	06. Water 2014–2015	0.80
15	08. Multi-Sector Overlap 2018–2019	1.76	42	02. New Energy 2018–2019	0.72
16	07. Climate Change Prediction and Monitoring 2015–2016	1.74	43	06. Water 2017–2018	0.71
17	09. Other Technologies 2016–2017	1.55	44	01. Renewable Energy 2015–2016	0.66
18	08. Multi-Sector Overlap 2015–2016	1.45	45	08. Multi-Sector Overlap 2019–2020	0.63
19	09. Other Technologies 2017–2018	1.44	46	03. Energy Storage 2017–2018	0.62
20	09. Other Technologies 2014–2015	1.42	47	07. Climate Change Prediction and Monitoring 2014–2015	0.54
21	03. Energy Storage 2019–2020	1.31	48	07. Climate Change Prediction and Monitoring 2017–2018	0.50
22	05. GHG Fixation 2019–2020	1.30	49	02. New Energy 2015–2016	0.44
23	09. Other Technologies 2015–2016	1.29	50	06. Water 2018–2019	0.42
24	05. GHG Fixation 2015–2016	1.28	51	07. Climate Change Prediction and Monitoring 2019–2020	0.37
25	09. Other Technologies 2018–2019	1.25	52	02. New Energy 2014–2015	0.34
26	01. Renewable Energy 2019–2020	1.18	53	07. Climate Change Prediction and Monitoring 2018–2019	0.30
27	09. Other Technologies 2019–2020	1.17	54	04. Energy Demand 2014–2015	0.25

, the efficient DMUs were new energy in 2016–2017, renewable energy in 2016–2017, GHG fixation in 2016–2017, water in 2019–2020, and energy storage in 2016–2017. (Table 11) On the other hand, there were 20 inefficient DMUs with a productivity change rate of less than 1 in the 2014–2015 period, including energy demand, new energy, climate change prediction and monitoring, and water.

New energy, renewable energy, and GHG fixation showed a significant increase in productivity, while climate change prediction and monitoring showed a decrease in productivity (

Table 12. Productivity and percentage change in productivity components by technology.

Technology	Average: MPI	Average: TC	Average: TEC	Factors of Change	Average: PTEC	Average: SEC
01. Renewable Energy	3.10	1.38	1.63	External	1.04	1.20
02. New Energy	3.78	1.79	1.43	Internal	1.24	1.12
03. Energy Storage	1.88	1.00	1.87	External	1.00	1.00
04. Energy Demand	1.55	1.08	1.32	External	1.00	1.07
05. GHG Fixation	2.58	1.29	1.90	External	1.00	1.29
06. Water	2.42	1.21	1.74	External	1.00	1.20
07. Climate Change Prediction and Monitoring	0.72	0.97	0.99	External	0.99	1.00
08. Multi-Sector Overlap	1.85	1.27	1.59	External	1.17	1.07
09. Other Technologies	1.35	1.05	1.63	External	1.05	0.98
Average	2.14	1.23	1.57	-	1.05	1.10

). An increase in productivity due to the rate of change of its own technology was found in new energy, while the rest of the productivity increase was due to external factors, reflected in higher average TEC values compared to TC. In particular, energy storage, GHG fixation, and water showed a significant difference in efficiency between TECs and TCs, indicating that productivity increase is largely driven by external contextual changes for these technologies. When examining productivity changes in terms of PTEC and SEC, new energy and multi-sector overlap were found to have greater PTEC than SEC did, while renewable energy, energy demand, GHG fixation, and water were found to have greater SEC compared to PTEC.

4.4. Integrated Analysis

Based on the results of the static and dynamic analyses above, an integrated model of the two analyses [28] is introduced for a comprehensive evaluation of efficiency. The results of analyses made for the period of 2014–2020 are organized in four quadrants based on average values of efficiency (Figure 7).

Quadrant I (top left) includes Energy storage, multi-sector overlap, and energy demand technologies with above-average efficiency and below-average productivity, while quadrant II (bottom left) includes other technologies with below-average efficiency and below-average productivity, and climate change prediction and monitoring technologies. In quadrant III (bottom right), new energy and water are indicated to have average efficiency and above-average productivity. Quadrant IV (top right) is occupied by GHG fixation and renewable energy, which are above average in efficiency and above average in productivity.

The technologies in the fourth quadrant can be benchmarked for other technologies in other fields, while other technologies and climate change prediction and monitoring in the second quadrant can be judged as urgent areas for improvement. Accordingly, the targets and reference values for technologies in the fourth quadrants based on CCR analysis are as follows (

Table 13. Means to improve inefficient climate change response technology development projects.

Project Name	TE	Actual No. of Journal Papers	Actual No. of Patents	Target No. of Journal Papers	Target No. of Patents	Reference Value 1 (λ)	Reference Value 2 (λ)
07. Climate Change Prediction and Monitoring 2014	0.5	15	0.001	30.14	2.57	02. New Energy 2014 (102.8)	04. Energy Demand 2014 (61.31)
07. Climate Change Prediction and Monitoring 2015	0.27	7.33	0.001	27.37	2.41	02. New Energy 2014 (96.5)	04. Energy Demand 2014 (20.61)
07. Climate Change Prediction and Monitoring 2016	0.37	3.33	0.001	9.05	2.50	06. Water 2016 (100)
07. Climate Change Prediction and Monitoring 2017	0.41	7.67	0.001	18.77	4.13	02. New Energy 2014 (20.28)	06. Water 2016 (145.08)
07. Climate Change Prediction and Monitoring 2018	0.18	3.83	0.001	21.72	5.17	02. New Energy 2014 (15.95)	06. Water 2016 (190.93)
07. Climate Change Prediction and Monitoring 2019	0.05	3.37	0.001	61.43	8.97	02. New Energy 2014 (154.31)	06. Water 2016 (204.29)
07. Climate Change Prediction and Monitoring 2020	0.02	2.17	0.001	102.61	24.99	02. New Energy 2014 (64.61)	06. Water 2016 (935.11)

).

First, climate change prediction and monitoring was analyzed to be the least efficient technology due to a lack of patent performance. With reference to the year of 2020, the number of journal papers should have increased from 2.17 to 102, and the number of patents should have increased to 24.99. The lambda (λ) value refers to the degree of influence an efficient DMU has on an inefficient DMU. An inefficient DMU may prioritize its benchmarking subjects based on the lambda (λ) value. The reference organizations that the inefficient DMU should benchmark against are the DMUs that constitute efficiency change with an efficiency score of 100, so that they are relatively close to the efficiency change of the DMU being evaluated. In other words, the DMUs in the reference organizations are relatively efficient DMUs with the most similar characteristics to the DMU being evaluated. The lambda (λ) value of an inefficient DMU can be used to adjust the amount of inputs and outputs to become an efficient project [40]. Larger lambda values indicate that the reference group is more influential; thus, it is suggested that climate change prediction and monitoring 2020 could produce higher performance by referring to the water 2016 DMU.

5. Conclusions

The climate issue is a global challenge that the world is responding to as a whole, and investments are being made for the necessity of mitigating it rather than for economic purposes. For this reason, it is difficult to decide whether to expand or reduce investments based on immediate performance alone. Categorizing efficient and inefficient sectors based on their effectiveness and productivity can yield meaningful achievements by providing means to discover and support factors that can further enhance efficiency and productivity in efficient sectors and also suggest ways to improve inefficient sectors.

The current study analyzes the efficiency and productivity of the climate change response technology development projects conducted by the Ministry of Science and ICT in South Korea using DEA and Malmquist analysis, and suggests ways to improve efficiency by placing technology-specific DMUs in the quadrants by analyzing them from an integrated perspective.

First, the static analysis shows that the investment efficiency of national R&D projects for climate change response needs to be improved. Of the 64 DMUs, only 4 were analyzed as efficient using CCR and 11 were analyzed as efficient using BCC. Most of the DMUs have an IRS status, suggesting that increasing investment will increase performance. In terms of technology, the TE results considering scale and technology were high for energy storage, energy demand, and multi-sector overlap, while the PTE results considering only technology were high for energy storage and renewable energy. When examining the causes of inefficiency based on SE and PTE, scale factors appeared to be the causes of inefficiency in highly efficient technologies including renewable energy, energy storage, and multi-sector overlap, while technology factors were found to be responsible for inefficiency in climate change prediction and monitoring and water.

Dynamic analysis showed that productivity is very high. When looking at the TE and TEC that influenced the change in productivity, the TEC value is large, indicating that productivity increased due to changes in external conditions rather than internal causes. In terms of different time periods, the most productive period was between 2016 and 2017, with a TEC of 289%, which is thought to have influenced R&D projects since the Paris Agreement, a period of intensive investment and interest in technology development on both domestic and international levels.

Finally, using a model that integrates static and dynamic analyses, the highest productivity and efficiency were found in the GHG fixation and renewable energy sectors. On the other hand, climate change prediction and monitoring, with below-average efficiency and productivity, was judged to be an area in urgent need of improvement. Consequently, target values and suitable reference areas for each of the DMUs were chosen based on the CCR. As a result, new energy, water, and energy demand were selected as the benchmarking subjects. It is worth noting that despite the fact that climate change prediction and monitoring has the highest number of journal papers per billion KRW (2.74) in all of the 11 technical areas, it was revealed as an area in urgent need of improvement in the present study.

The limitations of the present study are as follows. Firstly, the present analysis assumes radical efficiency in its definition of efficiency, meaning that it gives results of varying inputs and outputs at a constant rate while holding either of them fixed. The assumption of radical efficiency, however, is not always absolute and does not account for cases in which both outputs and inputs change simultaneously. Secondly, the weight of each input and output variable was not considered. It would be meaningful to compare the importance of each variable using the AHP method or the weights used in existing studies. Thirdly, in order to identify the causes of inefficiency and suggest improvement measures, various measures were proposed through the analysis of reference values, review of scale profitability, etc. There are, however, limitations to finding specific improvement measures since each technology has different goals and characteristics. Thus, it appears necessary to present case studies with experts for each technology in future research. Fourthly, the current study did not analyze the factors affecting efficiency based on the results of the DEA model. Future research may identify the characteristics of efficient climate change response technology development projects using the Tobit model and analyze the determinants to be considered in project planning. Finally, applying the meta-frontier analysis methodology in future studies would allow efficiency comparisons between DMUs for each climate technology with varying levels of technological development. Finding the meta-efficiency and group efficiency, and defining the technological gap through the discrepancy between them, would allow more extensive analysis of the specificities between different groups.

Mitigating climate change is a global issue that goes beyond individuals and countries, and must be addressed on a global scale. Although the present research analyzed only governmental R&D projects, the results are adequate to indicate that investments in R&D need to be sustained in order to develop and realize technologies that can respond to the climate crisis. The recently released IPCC Sixth Assessment Report suggests that all of the climate crisis indicators currently point to extremely high-risk impacts. However, it is estimated that GHG emissions could be halved by utilizing and applying currently developed climate response technologies (solar, wind, methane reduction, BECCS, local and hydropower). Research and development to combat climate change will require more aggressive investment than ever before, as well as efforts to eliminate inefficiencies where they exist.