Identification of Influencing Factors for Sustainable Development: Evaluation and Management of Regional Innovation Performance in Heilongjiang, China

Abstract

Although the innovation factor endowment of Heilongjiang province is relatively rich, many innovation resources cannot effectively promote the sustainable development of innovation-driven performance. The combined drift degree evaluation method and grey-DEMATEL model were used to study regional innovation performance in Heilongjiang province and the key factors influencing it. The results of this study show that the overall trend of regional innovation performance is slowly rising. Compared with other regions, the level of science and technology output is in the forefront, and the level of economic output and social effect is at the end. The innovation performance level of enterprises and research institutions is at the end position, showing a trend of slow rise and then slow decline. The innovation performance level of universities is in the middle position, showing a trend of rising first, then falling and then rising. The degree of marketization, the financial environment, the intellectual property system, the transformation of scientific and technological achievements and the level of human capital are the core factors that affect the regional innovation performance. The dominant position and leading role of regional enterprises and the cooperation between industry, university and research institutes should be strengthened through government policies.

1. Introduction

With the progress of science and technology, human beings are rapidly evolving from an industrial economic society to a knowledge economic society. Science and technology, as the source of the construction of an innovation-oriented country, has gradually become a new economic growth point and plays an increasingly important role in promoting the sustainable development of national and regional economy [1]. At the same time, energy shortage and environmental pollution make the traditional economic development model unsustainable. Innovation has gradually become a powerful booster of national and regional economic sustainable development. The implementation of the innovation project provides high-quality intellectual services and gradually becomes the decision-making guarantee for the sustainable development of regional economy [2].

At present, economic growth slowdown has become a global phenomenon, and China’s economic structural slowdown is increasingly prominent [3]. Demand-side to pull the role of the economy gradually weakened, and the reform of the supply-side is imperative. In 2015, China proposed supply-side structural reform, one of the main purposes of which is to adjust the economic structure through innovation-driven development strategy and promote the smooth implementation of Made in China 2025 strategy [4]. China’s regional economic development has not balanced for a long time, and the structural problems between regions and within regions are also very prominent. Traditional advantageous industries in some zones are no longer beautiful and emerging industries develop slowly. The driving force of export, investment and consumption has weakened, and the contradiction between supply and demand has become increasingly prominent [5]. These problems are often the reflection of regional innovation ability and regional innovation performance. The root cause is that regional innovation factors cannot be effectively transformed into real productivity, thus affecting regional innovation performance and regional sustainable development [6]. Therefore, effectively adjusting the allocation of regional innovation resources plays an important role in accelerating the formation of a regional economic system mainly supported by innovation, and promoting the sustainable development of regional economy.

Regional innovation system is an integral part of a national innovation system. Regional innovation performance is of great significance to regional economic sustainable development and national overall economic development [7]. Heilongjiang province, an old industrial base in northeast China, ranked 22nd in China in 2017 with a GDP of 1.590268 billion yuan, up 6.4% year-on-year to 41,916 yuan, lower than the national average. In 2018, the GDP reached 1,636,162 billion yuan, ranking 23rd in China, with a year-on-year growth rate of 4.7%, lower than the national average and better than expected [8]. The main reason lies in the lack of innovation drive in the case of the troika dynamic weakening. According to the evaluation report on China’s regional innovation capacity in 2018, the comprehensive innovation level index of Heilongjiang province dropped two places to 16th in the country. From the perspective of the scientific and technological innovation environment, Heilongjiang province has significantly improved its innovation environment, rising from 17th to 12th place [9]. In 2017, the full-time equivalent of R&D personnel in Heilongjiang province was 47,406 person-years, which was in the middle of the whole country. However, the full-time equivalent of researchers is 29,660 person-years, which is in the middle and upper reaches of the country and has certain advantages in quantity. The total number of the three kinds of patents authorized is 18,221, which is in the middle and lower reaches of the country.

With the rapid development of China’s economy, the number of large and medium-sized state-owned enterprises from the original advantages gradually become disadvantages. Heilongjiang province has gradually lost the original advantages. From 2007 to 2017, the contribution of industry to Heilongjiang’s economic growth remained above 50%. Among them, the proportion of the added value of energy industry in the proportion of industries above the scale is highest at 72.9% and the lowest at 53.8%, and the growth rate is highest at 13.1% and the lowest at 6.3% [8,10]. The growth potential of a number of industries has not yet been fully unleashed, and the vitality of economic growth is insufficient. In the last 10 years, the innovation vitality of Heilongjiang regional enterprises is insufficient, and the quality of innovation output is average. The number of high-tech enterprises is obviously low and needs attention and improvement. The theoretical innovation and quantity of innovation in universities need to be further strengthened. The number of government research institutes in Heilongjiang decreased a lot, while the number of enterprise research institutes increased a little. The total quantity presents the decreasing trend, and Heilongjiang province still has certain development potential [6,9,10].

There are many researches on regional innovation performance, but there is still a lack of analysis on the actual situation of Heilongjiang province. The setting of the existing index system is not reasonable, and the innovation performance of specific regions cannot be analyzed effectively. Most of the methods focus on the evaluation of static methods and lack of dynamic analysis methods for empirical research. Although scholars have found that there are many factors influencing regional innovation performance, there is a lack of systematic sorting and the study of influencing factors is scattered. Many innovation resources cannot be effectively converted into real productive forces, and their contribution to economic sustainable development is limited [11].

The purpose of this study includes the following three aspects. (1) Based on relevant studies, the regional innovation performance dynamic evaluation index system and regional innovation subject dynamic evaluation index system of Heilongjiang province are constructed based on the current status of regional innovation. (2) Based on the current status of regional innovation, the science and technology output, economic output and social effect of enterprises, universities and research institutions in Heilongjiang province are effectively analyzed. (3) Through comparative analysis, the key factors affecting the regional innovation performance in Heilongjiang province are selected. The theoretical purpose of this study is not only to expand the research perspective and method of regional innovation performance evaluation, but also to enrich the regional innovation system theory and provide important theoretical support for the construction of a regional innovation system. The practical purpose of this study is not only to provide reference for the development strategies of regional innovation subjects, but also to contribute a little to the new path of comprehensive sustainable development of Heilongjiang province.

The following structure of this paper is as follows. The regional innovation review is in Section 2. Section 3 shows the research method and index system. Empirical results are listed in Section 4. Section 5 presents on the discussion analysis. Section 6 summarizes the research conclusion and enlightenment of this paper, and points out the shortcomings of this paper.

2. Literature Review

2.1. Theoretical Research on Regional Innovation Performance

(1) Research on concepts related to regional innovation. Riddle pointed out that regional innovation ability refers to the potential of relevant subjects in innovation activities to use relevant resources to create new products in a specific region [12]. Tura et al. pointed out that regional innovation capacity is an ability to allocate economic, cultural and social resources effectively [13]. Li et al. believed that regional technology innovation capability is an important foundation for economic sustainable development [14]. Huang et al. believed that regional innovation ability refers to the regional ability to transform new knowledge into new products [15]. Jin et al. regarded innovation activities as mainly determined by independent R&D input [16]. Yang et al. believed that regional innovation performance is the output and benefits are brought by regional innovation activities [17]. Hu et al. believed that the performance of scientific and technological innovation reflects the allocation of scientific and technological R&D innovation resources in each region [18].

(2) Research on factors influencing regional innovation performance. Söderlund et al. believed that the improvement of regional financial service efficiency can help enterprises make better use of external information [19]. Barra believed that regional knowledge innovation is conducive to improving regional innovation capacity and promoting regional economic development [20]. Hauser et al. pointed out that the development of finance contributes to the technological innovation of middle and small-sized enterprises [21]. Li et al. studied the direct and indirect effects of research institution behavior on regional innovation performance [22]. Based on the innovation value chain theory, Wang et al. divided the regional innovation process into two sub-stages: innovation R&D and achievement transformation [23]. Xiao et al. studied the impact of scientific and technological human resource input on regional innovation performance when the coupling coordination degree of industry-university-research was taken as the threshold variable [24].

As for the research on regional innovation performance in Heilongjiang province, Tan et al. analyzed the distribution pattern and development process of innovation capacity of Heilongjiang province, and pointed out that the absolute difference of innovation capacity was very obvious [25]. Li believed that a sound market system can make Heilongjiang province take the lead in system construction [26]. Han et al. believed that the main driving force for the growth of innovative total factor productivity (TFP) in the three northeastern provinces is technological progress, and the driving force of TFP innovation in Liaoning and Heilongjiang provinces is mainly technological progress [27].

2.2. The Index System of Regional Innovation Performance Evaluation

Many scholars studied the index system of regional innovation performance evaluation [28,29,30,31,32,33]. Li et al. investigated regional innovation performance from two stages: intermediate output and final output [28]. Li et al. measured regional innovation performance from four aspects: innovation system strength, patent application authorization, new products and high-tech industry development [29]. Deng believed that the sales revenue of new products can better express the marketization ability of innovation output than the patent index [30].

As for the research on the evaluation index system of innovation performance in Heilongjiang province, Liang believed that the evaluation index system of regional innovation ability in Heilongjiang province includes innovation resource input ability, innovation subject allocation ability, innovation output ability and innovation environment support ability [31]. Wu et al. selected investment indicators, including the proportion of R&D personnel in an enterprise’s employees, as well as the fund guarantee for independent R&D. Output indicators are the proportion of new product sales revenue to main business revenue, and the ratio of the number of invention patents applied to the number of patents applied [32]. Hong combined the specific contents of the input and output projects of science and technology innovation in Heilongjiang province, and selected evaluation indexes for the input projects from two aspects: human input and financial input. For the output project, three evaluation indexes are selected from the two aspects of technical output and high-tech output, with a total of seven evaluation indexes [33].

2.3. The Evaluation Methods of Regional Innovation Performance

Many scholars studied the evaluation methods of regional innovation performance, such as data envelopment analysis (DEA) model, entropy weight method and coupling coordination degree model. Christoph et al. believed that the efficiency of achievement transformation and the efficiency of comprehensive innovation in each region are used to describe the regional innovation performance [34]. Corrado et al. studied the relationship between the innovation of intangible assets and economic growth in some EU countries, and found that the importance of knowledge capital in the economic growth of developed countries has become increasingly prominent [35]. Vitola et al. used the institutional potential index method to study the impact of technological innovation on economic development and found that a good innovation environment has a positive impact on regional development [36]. There are many evaluation methods, such as principal component analysis [37,38] and weighted comprehensive evaluation method [39,40]. Li et al. studied the comprehensive development capability of regional manufacturing industry by using the comprehensive evaluation model [41]. Zhao et al. studied the sustainable development ability of regional manufacturing industry by using analytic hierarchy process [42]. Li et al. evaluated the green competitiveness of regional manufacturing industry by using the projection pursuit model of genetic algorithm [43]. Yu et al. used the three-stage DEA model to evaluate the quality competitiveness of regional manufacturing industry [44]. Cao et al. used collinearity and coefficient of variation method to screen indicators for evaluation of provincial green innovation capacity, and used Delphi method and coefficient of variation method to calculate the weight of each indicator [45]. Feng used DEA-SBM method to measure the green innovation efficiency of industrial enterprises [46]. Li et al. used the combined weighting method based on the fuzzy analytic hierarchy process and the maximum deviation method to calculate the index weight [47].

3. Research Methods and Indicator System

3.1. Research Methods

3.1.1. Drift Degree Combination Evaluation Method

The evaluation methods of regional innovation mainly include subjective evaluation method and objective evaluation method. Subjective evaluation method mainly includes analytic hierarchy process, Delphi method, and fuzzy synthesis method. Objective evaluation methods mainly include parametric method and non-parametric method. The parameter method is represented by the random frontier method, but its disadvantage is that it needs to set the production function in advance. While the non-parameter method is represented by the data envelope method, its disadvantage is that it is very sensitive to outliers. China’s regional innovation ability report uses the expert prior scoring method to give weight. China innovation index research adopts the grade-by-grade equal-weight method for weight distribution. The European innovation scoreboard uses steps such as weighting to measure indicators. The Global Innovation Index (GII) is evaluated using methods such as multiple models.

Most methods in the literature are single evaluation methods [48]. In fact, the operating mechanism and application objects of different single evaluation methods are different. When different evaluation methods are applied to evaluate the same problem, there exists the problem of non-consistency of the evaluation results. Therefore, scholars put forward the idea of combined evaluation. At present, the research on combination evaluation mainly focuses on the combination of evaluation weights and the combination of evaluation conclusions. Compared with the method of combination weighting, the combination of evaluation conclusions is more direct and avoids the deviation caused by the result of weight combination. Therefore, this study draws on the multi-model evaluation method in the GII to ensure the reliability of the results without systematic bias. Based on the combination of subjective and objective evaluation models, the comprehensive dynamic evaluation method of regional innovation performance is adopted.

In this study, experts in regional innovation and related personnel in science and technology innovation management of government departments were consulted. This paper selected the objective evaluation methods of entropy weight method and maximum deviation method, and the subjective evaluation methods of TOPSIS method and grey relational degree method. On the basis of single evaluation, drift evaluation method is applied. Drift degree is the difference between the evaluation conclusion and the authenticity conclusion of each single evaluation method. Drift evaluation method is a combined evaluation method based on the evaluation conclusion of a single evaluation method. Based on the consistency of the evaluation conclusions of multiple single evaluation methods, the drift degree model measures the drift degree according to the rank correlation coefficient between the evaluation conclusion and the real value. The evaluation method with large drift degree is given a smaller weight, while the evaluation method with small drift degree is given a larger weight. This method effectively solves the problems such as the lack of consistency of the evaluation results and the difference between the evaluation conclusions and the objective reality.

(1) Establishment of the evaluation model. (a) Standardized processing of the original data of evaluation indicators. (b) Entropy weight method, TOPSIS method, deviation maximization method and grey relational degree method were used for single evaluation. (c) Spearman rank correlation coefficient is used to test the consistency of the evaluation results. If the consistency is strong, the combination evaluation method of drift degree is used for the combination evaluation. The evaluation model is shown in Figure 1.

The application of regional innovation performance evaluation should first determine the efficacy function. Let $v_{ij}$ represent the order parameter, and ${\alpha }_{ij}$ and ${\beta }_{ij}$ are the upper and lower limits of the order parameters of the critical point. $p_{ij}$ represents the contribution of the variable $v_{ij}$ to the system, and $p_{ij}\in [0,1]$. The efficacy coefficients are as follows [49]:

$$p_{ij}=\{\begin{array}{l}(v_{ij}-{\beta }_{ij})/({\alpha }_{ij}-{\beta }_{ij}),\mathrm{Positive}\mathrm{effect}\\ ({\alpha }_{ij}-v_{ij})/({\alpha }_{ij}-{\beta }_{ij}),\mathrm{Negative}\mathrm{effect}\end{array}$$

(1)

(2) Subjective and objective evaluation methods. The evaluation methods are shown in Appendix A.

(3) Combined drift evaluation method. Compatibility: The regional innovation performance evaluation is combined with the evaluation results of four single evaluation methods: Entropy weight method, maximum deviation method, TOPSIS method and grey relational degree method. The scientific rationality of the combined evaluation results depends on the rationality of the results of the four single evaluation methods mentioned above. The applicability of different single evaluation methods to different evaluation problems is different. A single evaluation method for a specific problem to be evaluated, if applicable, is called a compatible method, and constitutes a single evaluation method set for the problem [50]. Therefore, it is necessary to evaluate the compatibility test of the above four single evaluation methods. Combined evaluation results are reasonable and effective only when the selected single evaluation methods are compatible. The implementation process of the combined drift evaluation method is as follows.

① Construct evaluation method set $M_0$. The regional innovation performance index system is selected and the evaluation objects are evaluated by using each single evaluation method.

② Determine the compatibility evaluation method set $M_1$. Spearman rank correlation coefficient was used to test the compatibility of the evaluation results. The formula of Spearman rank correlation coefficient is as follows:

$$r_{ij}=1-\frac{6{\displaystyle \sum _{s=1}^n{d}_s^2}}{n\left(n^2-1\right)}$$

(2)

where $i=1,2,\cdots ,r;j=1,2,\cdots ,r$. $d_s$ is the grade difference between the combination evaluation methods in $i$ and $j$.

③ The average value of each single evaluation method is used as the reference frame for drift measurement. The correlation coefficient $r_j\left(t_k\right)$ of the evaluation value of each single evaluation method $u_{ij}\left(t_k\right)$ and the reference frame $\overline{u}\left(t_k\right)$ at time $t_k$ can be obtained. The drift degree of each single evaluation method used for combined evaluation is as follows:

$$p_j\left(t_k\right)=1-r_j\left(t_k\right),j=1,2,\cdots ,b$$

(3)

④ On the basis of driftability measure, according to the evaluation method with large driftness, it is given a small weight; otherwise, it is the opposite principle. Then the weight of method $j$ at time $t_k$ is as follows:

$$w_j\left(t_k\right)=\frac{\underset{1\le j\le b}{\min }[p_j\left(t_k\right)]+\underset{1\le j\le b}{\max }[p_j\left(t_k\right)]-p_j\left(t_k\right)}{{\displaystyle \sum _{j=1}^b\left\{\underset{1\le j\le b}{\min }[p_j\left(t_k\right)]+\underset{1\le j\le b}{\max }[p_j\left(t_k\right)]-p_j\left(t_k\right)\right\}}}$$

(4)

where $j=1,2,\cdots ,b;k=1,2,\cdots ,N$.

⑤ Assuming the evaluation conclusion $u_{ij}\left(t_k\right)$ of the method $j$ of the object $i$ to be evaluated at time $t_k$, the combined evaluation result of drift degree is as follows:

$$P_i\left(t_k\right)={\displaystyle \sum _{j=1}^b{u}_{ij}\left(t_k\right)}{w}_j\left(t_k\right),i=1,2,\cdots ,m$$

(5)

3.1.2. Grey-DEMATEL Model

Grey system theory is the uncertain system of small samples and poor data with partial information known and partial information unknown. By generating and developing some known information and extracting valuable information, the theory can accurately describe and monitor the system’s operation behavior and evolution law. In the case of less sample data, the uncertainty of the system and the imperfection of sample data are overcome, and a relatively accurate prediction model is established [51]. Assuming that any random process is the grey quantity that varies within a certain range, the random process is called the grey process.

DEMATEL method, proposed by American scholar Battelle in 1971, is mainly based on graph theory and matrix to establish a structural model to analyze the causal relationship among complex factors [52]. Collecting information is the first step of system analysis. When collecting information, the scientific nature, rigor, and rationality of the collection method must be ensured. The collected information should try to include all the influencing factors.

(1) Key factor selection method based on Grey-DEMATEL model. Grey model and DEMATEL model are shown in Appendix B.

(2) Key factor selection process based on Grey-DEMATEL model.

Step 1: Establish the index system of influencing factors. Six experts in the field (2 heads of government departments, 2 executives of enterprises and 2 researchers from universities) are invited to compare the factor relations in the pairwise relation matrix, and then the evaluation and scoring are conducted to construct the initial matrix.

Step 2: Establish direct impact matrix based on grey theory.

① Establish the grey number matrix. The classification results of expert evaluation semantic variables are shown in

Table 1. Experts evaluate semantic variables.

Semantic Variable	Grey Number	Evaluation Value
No (N)	[0.00, 0.00]	1
Very Low (VL)	(0.00, 0.25)	2
Low (L)	[0.25, 0.50)	3
High (H)	[0.50, 0.75)	4
Very High (VH)	[0.75, 1.00]	5

, which are divided into five grades. According to this rule, the initial matrix is transformed into grey number matrix.

②The normalized results of upper and lower bounds of grey numbers are as follows:

$$\{\begin{array}{l}\underset{\_}{\otimes }{X}_{ij}^{~k}=\left(\underset{\_}{\otimes }{X}_{ij}^k-\min \underset{\_}{\otimes }{X}_{ij}^k\right)/{\Delta }_{\min }^{\max }\\ \overline{\otimes }{X}_{ij}^{~k}=\left(\overline{\otimes }{X}_{ij}^k-\min \overline{\otimes }{X}_{ij}^k\right)/{\Delta }_{\min }^{\max }\\ {\Delta }_{\min }^{\max }=\max \overline{\otimes }{X}_{ij}^k-\min \underset{\_}{\otimes }{X}_{ij}^k\end{array}$$

(6)

where $\underset{\_}{\otimes }{X}_{ij}^k$ and $\overline{\otimes }{X}_{ij}^k$ are the lower and upper bounds of expert evaluation, and ${\Delta }_{\min }^{\max }$ is the maximum and minimum difference.

③ The clear definition value of grey number matrix is as follows:

$$Y_{ij}^k=\frac{\left\{\underset{\_}{\otimes }{X}_{ij}^{~k}\left(1-\underset{\_}{\otimes }{X}_{ij}^{~k}\right)+\left(\overline{\otimes }{X}_{ij}^{~k}\times \overline{\otimes }{X}_{ij}^{~k}\right)\right\}}{1-\underset{\_}{\otimes }{X}_{ij}^{~k}+\overline{\otimes }{X}_{ij}^{~k}}$$

(7)

④ The direct impact matrix is constructed as follows:

$$Z_{ij}^k=\min \overline{\otimes }{X}_{ij}^k+Y_{ij}^k{\Delta }_{\min }^{\max }$$

(8)

Step 3: Experts set weights. According to the degree of each expert’s mastery of the research field, experts assign different weights, which are fuzzy. The weighted semantic variables are shown in

Table 2. The weighted semantic variables.

Semantic Variable	Grey Number
Not Important (NI)	[0.0, 0.3]
Little Important (LI)	[0.3, 0.5]
Important (I)	[0.4, 0.7]
Much Important (MI)	[0.5, 0.9]
Very Important (VI)	[0.7, 1.0]

. The grey number weights of the six experts are shown in

Table 3. The grey number weights of the six experts.

Experts	Weighted Grey Numeral
University researcher 1	[0.7, 1.0]
University researcher 2	[0.3, 0.5]
Corporate executive 1	[0.7, 1.0]
Corporate executive 2	[0.4, 0.7]
Head of government department 1	[0.5, 0.9]
Head of government department 2	[0.3, 0.5]

.

Step 4: The weight value of grey number is standardized and clarified, and the calculation formula of the final direct impact matrix is as follows:

$$\{\begin{array}{l}{Z}_{ij}={\omega }_1{Z}_{ij}^1+{\omega }_2{Z}_{ij}^2+\dots +{\omega }_p{Z}_{ij}^p,{\displaystyle \sum _{i=1}^p{\omega }_i=1}\\ Z=\left[\begin{array}{cccc}0& Z_{12}& \dots & Z_{1j}\\ Z_{21}& 0& \cdots & Z_{2j}\\ ⋮& ⋮& ⋮& ⋮\\ Z_{i1}& Z_{i2}& \cdots & 0\end{array}\right]\end{array}$$

(9)

Step 5: The calculation formula of the standardized matrix is as follows:

$$S=\frac{1}{\underset{1\le i\le n}{\max }{\displaystyle {\sum }_{j=1}^n{z}_{ij}}},i,j=1,2,\dots ,n$$

(10)

$$N=S\times Z$$

(11)

Step 6: The calculation formula of the comprehensive influence matrix is as follows:

$$T=N{(1-N)}^{-1}$$

(12)

Step 7: According to the comprehensive influence matrix, the calculation formulas of the centrality and causation of each factor are as follows:

$$\begin{array}{l}\{\begin{array}{l}{P}_i=\left\{R_i+C_j|i=j\right\}\\ E_i=\left\{R_i-C_j|i=j\right\}\end{array}\\ R_i={\displaystyle \sum _{j=1}^n{t}_{ij}},C_j={\displaystyle \sum _{i=1}^n{t}_{ij}},i\in N^{+},j\in N^{+}\end{array}$$

(13)

Step 8: According to the numerical results of degree of centrality and degree of causation, the cartesian coordinate system is constructed and the influencing factors are marked in the coordinate system.

(3) The framework of key factor selection process based on Grey-DEMATEL model.

Based on the above analysis and Steps, DEMATEL method is a systematic factor classification method used to study the correlation and importance of factors. The method can effectively determine the relationship between criteria or elements, as well as the degree of interaction between factors. However, DEMATEL method does not take into account the weight between evaluation indicators and the evaluation fuzziness. In this paper, interval grey number method is introduced to construct Grey-DEMATEL model which is practical and more flexible. The main advantage of the Grey-DEMATEL model is to construct a flexible decision model and make the decision result closer to the reality by using grey interval number. The framework of key factor selection process based on Grey-DEMATEL model is shown in Figure 2.

Figure 2 presents the processes used in the key factor selection process based on Grey-DEMATEL model. The main steps for the model to be implemented are as follows. (a) Four index systems of influencing factors are established based on expert group in the field. (b) The evaluation and scoring are conducted to construct the initial matrix. (c) The initial matrix is transformed into grey number matrix. (d) The upper and lower bounds of grey numbers are normalized. (e) The clear definition value of grey number matrix is obtained. (f) The direct impact matrix is constructed. (g) The weighted semantic variables are given by experts. (h) The weight value of grey number is standardized and clarified. (i) The matrix of the weight value is standardized. (j) The comprehensive influence matrix is obtained. (k) The centrality and causation of each factor are obtained. (l) The cartesian coordinate system is constructed and the influencing factors are marked.

3.2. Construction of Index System

An important theoretical basis for evaluating innovation performance is innovation value chain theory. Hansen and Birkinshaw (2007) argued that the innovation value chain is a stream of activity that transforms ideas from generation to commodity, which contributes to the improvement of innovation performance [53]. The theory of regional innovation system also holds that the innovation unit realizes the innovation function through the interaction of the innovation structure itself and its environment, and influences the regional society, economy and ecology. The innovation performance can be divided into the ability of each innovation subject to make use of scientific and technological innovation resources for R&D. The ability to commercialize and market new products, processes or technologies and the benefits to the entire region. In this study, experts in regional innovation and related personnel in science and technology innovation management of government departments were consulted to measure regional innovation performance from three aspects: Science and technology output, economic output and social effect.

3.2.1. Dynamic Evaluation Index System of Regional Overall Innovation Performance

The research on regional innovation should grasp the motivation and characteristics of regional innovation comprehensively from the perspective of the innovation performance of micro-subject [54]. Based on the functional considerations of regional innovation subjects, regional innovation subjects mainly include enterprises, universities, research institutions and intermediary institutions. Although the intermediary institutions are the auxiliary institutions of regional innovation resource connection and allocation, they are not enough to play the main role of leading the regional innovation development. In China, innovation subjects not only refer to enterprises, as universities and research institutions are still important recipients of innovation resources and producers of innovation achievements [55]. Therefore, this study defines the subject of regional innovation as enterprises, universities and research institutions. The following is the construction of the regional overall innovation performance dynamic evaluation index system and three innovation subjects.

Aiming at the dynamic evaluation index system of regional overall innovation performance, this study evaluates science and technology output, economic output and social effect. Science and technology output includes indicators such as patents, papers and trademarks. Economic output includes product sales, technology transfer and other related indicators. Social effect includes economic, social and environmental effects brought by innovation. The dynamic evaluation index system constructed is shown in

Table 4. Comprehensive dynamic evaluation index system of regional overall innovation performance.

Main Criteria	Subcriteria	Criteria Unit	Criteria Attribute
Science and technology output	Number of invention patents authorized per 10,000 R&D personnel	A/ten thousand	+
	Number of patents authorized per 10,000 R&D personnel	A/ten thousand	+
	The proportion of the number of invention patent applications in the number of patent applications	%	+
	The number of scientific and technological papers included in the main retrieval tools abroad per 10,000 R&D personnel	Article/ten thousand	+
	Trademark ownership per hundred enterprises	A/hundred	+
Economic output	New product sales revenue in the main business income proportion	%	+
	Technology market turnover accounted for the proportion of gross domestic product	%	+
	Income from patent ownership transfer and licensing per 10,000 R&D personnel	Ten thousand yuan/ten thousand yuan	+
	Technical achievement turnover per 10,000 R&D personnel	Hundred million yuan/ten thousand people	+
	High-tech exports accounted for the proportion of exports	%	+
Social effect	Energy consumption per unit GDP	Ten thousand tons of standard coal/hundred million yuan	−
	Regional GDP per capita	Hundred million yuan/ten thousand people	+
	Comprehensive utilization rate of industrial solid waste	%	+
	The unemployment rate	%	+
	Labor productivity	Hundred million yuan/ten thousand people	+

.

3.2.2. Dynamic Evaluation Index System of Regional Enterprise Innovation Performance

Enterprise innovation is an important part of regional innovation, which involves not only the R&D of invention, utility and appearance patented technology, but also the commercialization and industrialization of technological achievements and the economic and social impact [56]. There are many factors influencing the innovation performance of enterprises with complicated structure. Only by constructing the enterprise innovation performance evaluation index system from multiple perspectives and levels can the regional enterprise innovation performance be fully reflected. The science and technology output of regional enterprises reflects the input and output of patents and trademarks of regional enterprises through the innovation of human resources, financial resources and material resources. The economic output of regional enterprise innovation reflects the direct income brought by regional enterprise innovation. The social effect of regional enterprise innovation reflects the positive influence of regional enterprise innovation on regional economy, society and environment. The dynamic evaluation index system constructed is shown in

Table 5. Dynamic evaluation index system of regional enterprise innovation performance.

Main Criteria	Subcriteria	Criteria Unit	Criteria Attribute
Science and technology output	Number of valid invention patents per 10,000 R&D personnel	A/ten thousand	+
	Number of patent applications per 10,000 R&D personnel	A/ten thousand	+
	The proportion of the number of invention patent applications in the number of patent applications	%	+
	Trademark ownership per hundred enterprises	A/hundred	+
Economic output	New product sales revenue in the main business income proportion	%	+
	Patent ownership transfer and licensing revenue per hundred enterprises	RMB 10,000/100	+
	High-tech exports accounted for the proportion of exports	%	+
	Number of new product development projects per 100 enterprises	A/hundred	+
Social effect	Comprehensive utilization rate of industrial solid waste	%	+
	Labor productivity in high-tech industries	RMB ten thousand/person	+
	Energy consumption per unit GDP	Ten thousand tons of standard coal/hundred million yuan	−
	The number of national or industry standards formed for each hundred enterprises	Item/hundred	+

.

3.2.3. Dynamic Evaluation Index System of Regional Universities’ Innovation Performance

Universities are the subject of regional knowledge innovation, whose innovation activities are mainly basic research and applied research [57]. It is the source of regional innovation, the subject of research and the subject of talent training. Generally, the innovation achievements are not directly converted into economic profits, and are mainly reflected in the form of patents and papers. The science and technology output of regional universities reflects the input and output of patents, scientific and technological papers, and works of regional universities through the innovation of manpower, financial resources and material resources. The economic output of regional universities’ innovation reflects the direct income brought by regional universities’ innovation. The social effect of regional universities’ innovation reflects the positive influence of regional universities’ innovation on regional talent training and improvement of scientific and technological levels. The dynamic evaluation index system constructed is shown in

Table 6. Dynamic evaluation index system of regional universities’ innovation performance.

Main Criteria	Subcriteria	Criteria Unit	Criteria Attribute
Science and technology output	The number of scientific and technological papers per 10,000 R&D personnel	Article/ten thousand	+
	Number of scientific and technological publications per 10,000 R&D personnel	A/ten thousand	+
	Number of effective invention patents per 10,000 R&D personnel	A/ten thousand	+
	The proportion of the number of invention patent applications in the number of patent applications	%	+
Economic output	The contract amount of technology transfer per 10,000 R&D personnel	Thousand yuan/ten thousand people	+
	Income from patent ownership transfer and licensing per 10,000 R&D personnel	Ten thousand yuan/ten thousand yuan	+
	The proportion of the transfer of patent ownership and license income in the contract value of technology transfer	%	+
	The number of projects of R&D achievement application and technology service in each university	Item/home	+
Social effect	Scientific and technological progress award for each department of the state council of each university	Item/home	+
	Each university attends international conferences and exchanges	M/home	+
	The number of graduate students per 10,000 R&D personnel	Ten thousand	+

.

3.2.4. Dynamic Evaluation Index System of Regional Research Institutions’ Innovation Performance

Research institutions play an important role as creators of core technologies and disseminators of technical knowledge in regional innovation systems. Compared with the innovation function of enterprises, regional research institutions focus more on basic research, experimental development, design and tests, and other original innovation functions in the regional innovation system [58]. Generally, the regional research institution adopts three development modes: Independent innovation based on its own technological resources, imitation innovation based on technology introduction and joint participation of various parties. Similar to the role of universities, regional research institutions tend to focus on basic technical research, and are now gradually promoting the model of basic research, applied research, experimental development, trial production and sales of scientific discovery. Regional research institutions are the subject of regional knowledge innovation. Most of their innovative achievements are not directly converted into economic profits, but mainly in the form of patents and scientific papers. The dynamic evaluation index system is shown in

Table 7. Dynamic evaluation index system of regional research institutions’ innovation performance.

Main Criteria	Subcriteria	Criteria Unit	Criteria Attribute
Science and technology output	The number of scientific and technological papers per 10,000 R&D personnel	Article/ten thousand	+
	Number of scientific and technological publications per 10,000 R&D personnel	A/ten thousand	+
	Number of effective invention patents per 10,000 R&D personnel	A/ten thousand	+
	The proportion of the number of invention patent applications in the number of patent applications	%	+
Economic output	Income from patent ownership transfer and licensing per 10,000 R&D personnel	Ten thousand yuan/ten thousand yuan	+
Social effect	Each research institute forms a national or industry standard number	Item/home	+

.

4. Empirical Results

4.1. Data Source

In order to comprehensively analyze the regional innovation performance in Heilongjiang province, the regional innovation performance in Heilongjiang province is dynamically evaluated from the following two aspects: Regional innovation performance and each innovation subject in Heilongjiang province. Research data from 2010–2018 “China Statistical Yearbook of Science and Technology, China’s Environmental Statistical Yearbook, China Energy Statistical Yearbook and China Statistical Yearbook, Heilongjiang Statistical Yearbook, Heilongjiang Science and Technology Statistical Manual, China Regional Innovation Capacity Report, China Regional Innovation Index Report, 2018 National Large and Medium Sized Cities Update for the Prevention and Control of Environmental Pollution by Solid Waste and the Regional Environment Bulletin Data Documents and Materials”.

This study draws on relevant scholars’ research results and consults relevant experts’ suggestions. In this study, 16 regions in China, namely six growth triangles, were selected as the research objects. Growth triangle refers to the economic region of the inter-provincial domain cooperation, covering southeast China’s coastal, central, western and northeastern regions, which mainly includes the Beijing-Tianjin-Hebei metropolitan zone (Beijing, Tianjin and Hebei), the Yangtze River Delta economic zone, (Shanghai, Jiangsu and Zhejiang), the triangle economic zone (Hubei, Hunan and Jiangxi), the Pearl River Delta economic zone (due to Hong Kong and Macau University innovation system differences with China, here, we only include Guangdong province), west triangle economic zone (Sichuan, Chongqing and Shaanxi), and the northeast economic zone (Heilongjiang, Jilin and Liaoning), totaling six districts, including 16 provinces and municipalities directly under the central government. The six growth triangles are concentrated in enterprises, universities and research institutions (especially industrial enterprises and key universities above the scale). These zones are home to the country’s important scientific, educational and intellectual resources, with a large number of inputs and outputs in scientific research. As the relevant data on innovation of Jiangxi provincial research institutions cannot be obtained, the innovation performance of Jiangxi provincial research institutions has not been evaluated.

In this study, the combined evaluation method of drift degree based on entropy weight method, maximum deviation method, TOPSIS method and grey relational degree method was used to evaluate the innovation performance in Heilongjiang province as a whole and each innovation subject from 2009 to 2017. The evaluation results are shown in Appendix C. In order to compare and analyze the innovation performance in Heilongjiang province and each subject, this paper evaluates the science and technology output, economic output, social effect and innovation performance of the six major triangle regions and each innovation subject, and draws Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7 for visual display.

4.2. Dynamic Evaluation of Regional Overall Innovation Performance

Figure 3 shows regional overall science and technology output, regional economic output and regional social effect.

As shown in Figure 3a, from a static perspective, Heilongjiang ranked 3 among 16 provinces in the six big triangles in 2017. The level of science and technology output was 0.4421, about 0.1 lower than that of Shanghai, and slightly higher than that of Shaanxi province, about four times that of Hebei province. Dynamically, from 2009 to 2017, Heilongjiang province first dropped and then rose in the ranking of 16 provinces in the six big triangles. The overall level of science and technology output showed an upward trend, reaching a peak of 0.4421 in 2017, higher than other regions except Beijing and Shanghai.

As shown in Figure 3b, from a static perspective, Heilongjiang province ranked 14th among 16 provinces in the six big triangles in 2017. The level of economic output was 0.1035, much lower than that of Beijing, Shanghai and Guangdong. Heilongjiang is about one-sixth the level of Beijing’s economic output, only slightly higher than Jiangxi and Hebei, and at the end of the range. Dynamically, from 2009 to 2017, Heilongjiang province showed a slow rise in the ranking of 16 provinces in the six big triangles. However, the trend is not obvious. Heilongjiang province hovered between 14th and 16th, and the overall level of economic output presented a slow downward trend with a relatively low decline rate.

As shown in Figure 3c, from a static perspective, Heilongjiang province ranked 15th among the 16 provinces in the six big triangles in 2017. The social effect level of Heilongjiang province is lower than that of other regions, about 1/4 of the economic effect level of Beijing, which is at the end of the social effect level. Dynamically, from 2009 to 2017, Heilongjiang province’s ranking among the 16 provinces in the six big triangles showed a declining trend, with a relatively large decline. The overall level of social effect increased first and then decreased, reaching a peak of 0.3403 in 2012. In 2017, the level of social effect decreased to 0.2039, with a relatively significant decline.

As shown in Figure 3d, from a static perspective, Heilongjiang ranked 12th among 16 provinces in the six big triangles in 2017. The innovation performance level was 0.2692, higher than that of Hebei, Jilin, Jiangxi and Hunan, and about 1/3 of that of Beijing, ranking in the middle and lower level. Dynamically, from 2009 to 2017, Heilongjiang province fluctuated between 12 and 15 in the ranking of 16 provinces in the big six triangles. The overall trend shows a slow fluctuation rising trend, but the trend is not obvious. The highest level of innovation performance in 2017 was 0.2692.

4.3. Dynamic Evaluation of Innovation Performance of Regional Innovation Subjects

4.3.1. Dynamic Evaluation of Science and Technology Output Level

Figure 4 shows science and technology output of each innovation subject.

As shown in Figure 4a, from a static point of view, in 2017, the science and technology output of enterprises in Heilongjiang province ranked 12th among the 16 provinces in the six big triangles. The level of science and technology output was 0.2800, higher than that of Hebei, Jilin, Zhejiang and Chongqing, and about 1/3 of that of Beijing. Dynamically, from 2009 to 2017, Heilongjiang province rose from 15th to 12th among the 16 provinces in the big six triangles, showing a slow development trend. The level of science and technology output showed a slow upward trend, and the upward trend was obvious compared with other provinces.

As shown in Figure 4b, from a static point of view, the subject of universities in Heilongjiang province ranked 6th among 16 provinces in the six big triangles in 2017. The science and technology output level of universities is 0.5440, which is higher than that of Tianjin, Hebei, Liaoning, Zhejiang, Jiangxi, Hunan, Guangdong, Chongqing and Sichuan, and about 3/4 of that of Beijing and Shanghai. Dynamically, from 2009 to 2017, Heilongjiang province showed a slight fluctuation in the ranking of 16 provinces in the big six triangles. The overall ranking showed a slow upward trend, while the level of science and technology output showed a rapid upward trend, which was not changed much compared with other regions.

As shown in Figure 4c, from a static point of view, the subject of research institutions in Heilongjiang province was ranked 15th among the 16 provinces in the six big triangles in 2017. The level of science and technology output of research institutions is 0.1069, about 1/10 of that of Beijing and at the end of the range. Dynamically, from 2009 to 2017, Heilongjiang province was ranked in the top 15 of 16 provinces in the big six triangles, that is, the bottom position. The overall level of science and technology output showed a downward trend, which was more obvious compared with other regions.

4.3.2. Dynamic Evaluation of Economic Output Level

Figure 5 shows economic output of each innovation subject.

As shown in Figure 5a, from a static point of view, Heilongjiang province ranked 16th among 16 provinces in the six big triangles in 2017. The economic output level is 0.0720, which is about 1/7 of the economic output level of Tianjin and at the end of the economic output level. Dynamically, from 2009 to 2017, Heilongjiang province ranked 16th among the 16 provinces in the six big triangles. The level of economic output showed a trend of fluctuation and decline, and the decline was more obvious than that in most zones.

As shown in Figure 5b, from a static point of view, the subject of universities in Heilongjiang province ranked 13th among 16 provinces in the six big triangles in 2017. The economic output level of universities is 0.0842, higher than that of Hebei, Jilin and Jiangxi, and about 1/7 of the economic output level of Beijing, which was at the end of the economic output level. Dynamically, from 2009 to 2017, Heilongjiang province ranked 12-16 among the 16 provinces in the six big triangles. The overall ranking showed a slow upward trend, but the trend was not obvious. The overall level of economic output showed a fluctuating trend. Compared with other regions, the degree of fluctuation is relatively large, with significant differences.

As shown in Figure 5c, from a static point of view, the subject of research institutions in Heilongjiang province was ranked 10th among the 16 provinces in the six big triangles in 2017. The economic output level of research institutions is 0.0264, which was about 1/20 of Beijing’s economic output level and at the end of the economic output level. Dynamically, from 2009 to 2017, Heilongjiang province showed a slow rise in the ranking of 16 provinces in the six big triangles. The overall level of economic output showed a slow upward trend. Compared with other regions, the upward trend was slow and less volatile.

4.3.3. Dynamic Evaluation of Social Effect Level

Figure 6 shows social effect of each innovation subject.

As shown in Figure 6a, from a static perspective, Heilongjiang province ranked 15th among the 16 provinces in the six big triangles in 2017. The level of social effect was 0.2501, which was only slightly higher than that of Hebei province, and about 1/3 of that of Beijing, which was at the end of the social effect level. Dynamically, from 2009 to 2017, Heilongjiang province showed a slow decline and fluctuation in the ranking of 16 provinces in the six big triangles. The overall social effect level presented a slow decline trend. Compared with other regions, the decline was larger.

As shown in Figure 6b, from a static point of view, the subject of universities in Heilongjiang province ranked 7th among 16 provinces in the six big triangles in 2017. The social effect level of the subject of universities was 0.3016, which was about 1/3 of the innovation performance level of Beijing, and was in the middle position of the social effect level. Dynamically, from 2009 to 2017, Heilongjiang province rose from 10th to 7th among the 16 provinces in the six big triangles. The level of social effect fluctuates. Compared with other regions, the degree of fluctuation was small and there was room for improvement.

As shown in Figure 6c, from a static point of view, the subject of research institutions in Heilongjiang province was ranked 9th among the 16 provinces in the six big triangles in 2017. The social effect level of research institutions was 0.1145, which was about 1/10 of the social effect level of Beijing, and was in the middle of the social effect level. Dynamically, from 2009 to 2017, Heilongjiang province rose from 15th to 9th among the 16 provinces in the six big triangles. The overall social effect level presented a slow rising trend. Compared with other regions, the upward trend was slow and the degree of fluctuation was slow.

4.3.4. Dynamic Evaluation of Innovation Performance Level of Each Subject

Figure 7 shows innovation performance of each innovation subject.

As shown in Figure 7a, from a static perspective, Heilongjiang province ranked 15th among 16 provinces in the six big triangles in 2017. The innovation performance level was 0.1612, which was only slightly higher than that of Hebei province and about 1/4 of that of Beijing, which was at the end of the innovation performance level. Dynamically, Heilongjiang province showed 15 or 16 fluctuations in the ranking of 16 provinces in the six big triangles from 2009 to 2017. The overall trend of innovation performance was fluctuating. The innovation performance level showed a trend of slow rise and then slow decline, and the highest innovation performance level in 2013 was 0.2230.

As shown in Figure 7b, from a static point of view, the subject of universities in Heilongjiang province ranked the 8th among 16 provinces in the six big triangles in 2017. The innovation performance level of universities was 0.2847, which was about 1/4 of that of Beijing. Dynamically, Heilongjiang province fluctuated in the rankings of 16 provinces in the six big triangles from 2009 to 2017. The overall ranking showed a slow upward trend, but the trend was not obvious. The innovation performance level showed a trend of rising first, then falling and then rising. The highest level of innovation performance in 2011 was 0.2896, which was similar to that of 2017.

As shown in Figure 7c, from a static point of view, the research institutions in Heilongjiang province ranked 15th among the 16 provinces in the six big triangles in 2017. The innovation performance level of the research institutions subject was 0.0511, about 1/10 of the innovation performance level of Beijing, and it was at the end of the innovation performance level. Dynamically, Heilongjiang province ranked 15th out of 16 provinces in the six big triangles from 2009 to 2017, which was the end position. The overall innovation performance level showed an upward trend and then a downward trend, and the highest innovation performance level in 2012 was 0.1290.

4.4. Key Factors Influencing Regional Innovation Performance

This study selected key factors of regional innovation performance in Heilongjiang province. On the basis of using objective data to evaluate the innovation performance of each innovation subject, this study used subjective data to select the key factors influencing the regional innovation performance and the innovation performance of each innovation subject.

4.4.1. Key Factors of Regional Overall Innovation Performance

This study selects the existing regional innovation performance evaluation system and the factors outside the system. The comprehensive influence was calculated according to steps 1 to 7, as shown in

Table 8. The overall comprehensive influence of the region.

Symbol	Row Sum	Column Sum	Centrality	Cause Degree	Factors
C1	6.325	6.362	12.687	−0.037	Number of invention patents authorized per 10,000 R&D personnel
C2	5.694	6.843	12.537	−1.149	Number of patents authorized per 10,000 R&D personnel
C3	7.354	7.418	14.772	−0.064	The proportion of the number of invention patent applications in the number of patent applications
C4	6.984	7.065	14.049	−0.081	The number of scientific and technological papers in foreign countries is mainly included for each 10,000 R&D personnel
C5	7.339	7.381	14.720	−0.042	Trademark ownership per hundred enterprises
C6	7.795	7.947	15.742	−0.152	New product sales revenue in the main business income proportion
C7	7.849	7.353	15.202	0.496	Technology market turnover accounted for the proportion of gross domestic product
C8	6.864	7.534	14.398	−0.670	Income from patent ownership transfer and licensing per 10,000 R&D personnel
C9	7.868	7.318	15.186	0.550	Technical achievement turnover per 10,000 R&D personnel
C10	7.794	7.113	14.907	0.681	High-tech exports accounted for the proportion of exports
C11	7.191	7.362	14.553	−0.171	Energy consumption per unit GDP
C12	7.832	7.235	15.067	0.597	Regional GDP per capita
C13	7.649	7.056	14.705	0.593	Comprehensive utilization rate of industrial solid waste
C14	7.141	7.553	14.694	−0.412	The unemployment rate
C15	6.157	6.968	13.125	−0.811	Labor productivity
C16	7.854	5.981	13.835	1.873	Degree of marketization
C17	7.365	5.673	13.038	1.692	The financial environment
C18	7.137	6.086	13.223	1.051	Intellectual property system
C19	7.734	6.694	14.428	1.040	Transformation of scientific and technological achievements
C20	7.565	6.492	14.057	1.073	Human capital level

Note: Data from 2010–2018 yearbook of science and technology statistics.

. According to step 8, the cause-center distribution diagram was constructed, as shown in Figure 8.

Table 8 and Figure 8 show the key factors influencing regional overall innovation performance. The reason factor is the core factor that affects the regional innovation performance. The factor value represents the influence degree of other factors. The reasons are listed as follows: C16 > C17 > C20 > C18 > C19 > C10 > C12 > C13 > C9 > C7. The result factor is the direct factor that affects the regional innovation performance and is greatly influenced by other factors. The ranking of the results is C1 > C5 > C3 > C4 > C6 > C11 > C14 > C8 > C15 > C2. The degree of centrality represents the relationship between this factor and other factors. The larger the value, the greater the influence of this factor on the decision. The order of centrality is C6 > C7 > C9 > C12 > C10 > C3 > C5 > C13 > C14 > C11 > C19 > C8 > C20 > C4 > C16 > C18 > C15 > C17 > C1 > C2.

4.4.2. Key Factors of Enterprise Innovation Performance

This study selects the existing regional enterprise innovation performance evaluation system and the factors outside the system. The comprehensive influence was calculated according to steps 1 to 7, as shown in

Table 9. Comprehensive influence of regional enterprises.

Symbol	Row Sum	Column Sum	Centrality	Cause Degree	Factors
C1	6.164	6.286	12.450	−0.122	Number of valid invention patents per 10,000 R&D personnel
C2	6.864	6.594	13.458	0.270	Number of patent applications per 10,000 R&D personnel
C3	6.531	6.672	13.203	−0.141	The proportion of the number of invention patent applications in the number of patent applications
C4	6.927	7.036	13.963	−0.109	Trademark ownership per hundred enterprises
C5	7.484	7.823	15.307	−0.339	New product sales revenue in the main business income proportion
C6	7.642	7.054	14.696	0.588	Patent ownership transfer and licensing revenue per hundred enterprises
C7	6.036	7.022	13.058	−0.986	High-tech exports accounted for the proportion of exports
C8	7.835	7.136	14.971	0.699	Number of new product development projects per 100 enterprises
C9	6.868	7.252	14.120	−0.384	Comprehensive utilization rate of industrial solid waste
C10	7.894	7.545	15.439	0.349	Labor productivity in high-tech industries
C11	6.038	7.032	13.070	−0.994	Energy consumption per unit GDP
C12	6.731	7.331	14.062	−0.600	The number of national or industry standards formed for each hundred enterprises
C13	7.984	6.256	14.240	1.728	Enterprise research investment
C14	8.023	6.843	14.866	1.180	Enterprise human resources input
C15	7.684	6.518	14.202	1.166	Enterprise management system
C16	7.051	6.116	13.167	0.935	Organizational structure
C17	7.113	6.254	13.367	0.859	Enterprise R&D equipment

Note: Data from 2010–2018 yearbook of science and technology statistics.

. According to step 8, the cause-center distribution diagram was constructed, as shown in Figure 9.

Table 9 and Figure 9 show the key factors for enterprise innovation performance. The reason factor is the core factor that affects the innovation performance of regional enterprises, and the factor value represents the influence degree of other factors. The ranking of causes is C13 > C14 > C15 > C16 > C17 > C8 > C6 > C10 > C2. The result factor is the direct factor that affects the innovation performance of the subject of the regional enterprises. The ranking of the results is C4 > C1 > C3 > C5 > C9 > C12 > C7 > C11. The degree of centrality represents the relationship between this factor and other factors. The larger the value, the greater the influence of this factor on the decision. The order of degree of centrality is C10 > C5 > C8 > C14 > C6 > C13 > C15 > C9 > C12 > C4 > C2 > C17 > C3 > C16 > C11 > C7 > C1.

4.4.3. Key Factors of University Innovation Performance

This study selects the existing regional universities innovation performance evaluation system and the factors outside the system. The comprehensive influence was calculated according to steps 1 to 7, as shown in

Table 10. Comprehensive influence of regional universities.

Symbol	Row Sum	Column Sum	Centrality	Cause Degree	Factors
C1	7.248	7.328	14.576	−0.080	The number of scientific and technological papers per 10,000 R&D personnel
C2	6.238	6.359	12.597	−0.121	Number of scientific and technological publications per 10,000 R&D personnel
C3	7.496	7.054	14.550	0.442	Number of effective invention patents per 10,000 R&D personnel
C4	6.264	6.284	12.548	−0.020	The proportion of the number of invention patent applications in the number of patent applications
C5	6.584	7.684	14.268	−1.100	The contract amount of technology transfer per 10,000 R&D personnel
C6	7.325	6.648	13.973	0.677	Income from patent ownership transfer and licensing per 10,000 R&D personnel
C7	6.023	7.012	13.035	−0.989	The proportion of the transfer of patent ownership and license income in the amount of technology transfer
C8	6.951	7.634	14.585	−0.683	The number of projects of R&D achievement application and technology service in each university
C9	7.206	7.513	14.719	−0.307	Scientific and technological progress award for each department of the state council of each university
C10	7.305	6.026	13.331	1.279	Each university attends international conferences and exchanges
C11	7.764	7.564	15.328	0.200	The number of graduate students per 10,000 R&D personnel
C12	7.298	6.165	13.463	1.133	Including scientists and engineers in R&D full-time staff
C13	7.535	6.248	13.783	1.287	The number of senior titles in teaching and research personnel
C14	7.035	6.034	13.069	1.001	Total internal expenditure in the expenditure of science and technology funds

Note: Data from 2010–2018 yearbook of science and technology statistics.

. According to step 8, the distribution diagram of cause-center degree was constructed, as shown in Figure 10.

Table 10 and Figure 10 show the key factors for the universities’ innovation performance. The reason factor is the core factor that affects the innovation performance of regional universities. The factor value represents the influence degree of other factors. The ranking of causes is C13 > C10 > C12 > C14 > C6 > C3 > C11. The result factor is the direct factor that affects the innovation performance of the subject in regional universities, and is greatly influenced by other factors. The ranking of the results is C4 > C1 > C2 > C9 > C8 > C7 > C5. The degree of centrality represents the relationship between this factor and other factors. The larger the value, the greater the influence of this factor on the decision. The order of centrality is C11 > C9 > C8 > C1 > C3 > C5 > C6 > C13 > C12 > C10 > C14 > C7 > C2 > C4.

4.4.4. Key Factors of Research Institutions’ Innovation Performance

This study selects the existing regional research institutions’ innovation performance evaluation system and the factors outside the system. According to steps 1 to 7, the comprehensive influence was calculated, as shown in

Table 11. Comprehensive influence of regional research institutions.

Symbol	Row Sum	Column Sum	Centrality	Cause Degree	Factors
C1	6.023	6.234	12.257	−0.211	The number of scientific and technological papers per 10,000 R&D personnel
C2	7.316	7.335	14.651	−0.019	Number of scientific and technological publications per 10,000 R&D personnel
C3	7.201	6.984	14.185	0.217	Number of effective invention patents per 10,000 R&D personnel
C4	7.246	7.467	14.713	−0.221	The proportion of the number of invention patent applications in the number of patent applications
C5	6.916	7.904	14.820	−0.988	Income from patent ownership transfer and licensing per 10,000 R&D personnel
C6	6.923	8.017	14.940	−1.094	Each research institute forms a national or industry standard number
C7	7.812	6.767	14.579	1.045	Total number of personnel engaged in R&D activities (R&D)
C8	7.234	6.054	13.288	1.180	Scientific research instruments and equipment above 100,000 yuan
C9	7.066	6.143	13.209	0.923	Number of national or departmental recognized qualifications and institutions

Note: Data from 2010–2018 yearbook of science and technology statistics.

. According to step 8, the cause-center distribution diagram was constructed, as shown in Figure 11.

Table 11 and Figure 11 show the key factors for the research institutions’ innovation performance. The reason factor is the core factor that affects the subject innovation performance of regional research institutions. The factor value represents the influence degree of other factors. The reasons are ranked C8 > C7 > C9 > C3. The result factor is the direct factor that affects the innovation performance of the subject of the regional research institutions. The ranking of the results is C2 > C1 > C4 > C5 > C6. The degree of centrality represents the relationship between this factor and other factors. The larger the value, the greater the influence of this factor on the decision. The order of center degree is C6 > C5 > C4 > C2 > C7 > C3 > C8 > C9 > C1.

5. Discussion

5.1. Comparative Analysis of Regional Overall Innovation Performance

The science and technology output level in northeast China is relatively high, showing a weak trend and a rising trend in the ranking. Heilongjiang province is in the leading position in northeast China, and its science and technology output level and ranking were on the rise. Compared with other regions, Heilongjiang’s science and technology output was in the forefront. Both the level of science and technology output and the ranking were on the rise, higher than other regions except Beijing and Shanghai. The economic output in northeast China is relatively low, showing a steady decline and ranking at the bottom. Heilongjiang province ranks ranked the 3rd in northeast China, and its economic output level and ranking were in steady decline. Compared with other regions, Heilongjiang’s level of economic output was at the bottom, only slightly higher than that of Jiangxi and Hebei. The level of economic output was slowly declining and the ranking was slowly rising, but the trend was not obvious. The social effect in northeast China is relatively low, showing a steady decline and ranking at the bottom. Heilongjiang province ranked the 3rd in northeast China, and the whole province was at the bottom of the ranking. The level of social effect first rose and then declined. Compared with other regions, the social effect level of Heilongjiang province was in the terminal position, and the overall social effect level showed a trend of first rising and then declining, with a relatively obvious decline.

In general, the innovation performance level of northeast China was 0.2710, slightly higher than that of the Middle Triangle region (0.2589) and lower than that of other regions, showing a trend of slow growth. Heilongjiang province ranked second in northeast China, and its innovation performance level and ranking were rising steadily. Compared with other regions, the innovation performance level of Heilongjiang province was in the middle and lower level, only slightly higher than that of Hebei, Jilin, Jiangxi and Hunan. The overall trend showed a slow fluctuation rising trend, but the trend was not obvious, and the ranking was in a fluctuation trend.

5.2. Comparative Analysis of Innovation Performance of Regional Innovation Subjects

5.2.1. Comparative Analysis of Science and Technology Output of Each Innovation Subject

The science and technology output level of enterprises in northeast China was lower than that of other regions, showing a slow rising trend and a slow development trend. The science and technology output level of the subject of enterprises was in the middle of the overall level, and the overall science and technology output level presented a slow rising trend. Compared with other regions, the level of science and technology output of Heilongjiang province was at the middle and end of the level of science and technology output.

The science and technology output level of universities in northeast China was certain advantages. Compared with other regions, the upward trend was more obvious. Heilongjiang province ranked first in northeast China, and its overall level was in an advantageous position. The overall level of science and technology output showed a rapid rising trend. Compared with other regions, it had obvious advantages in science and technology output. Compared with other regions, it was in an advantageous position at the level of science and technology output. The overall ranking showed a slow upward trend, and the level of science and technology output showed a rapid upward trend.

The science and technology output level of research institutions in northeast China was relatively low, slightly higher than that in the Pearl River Delta region, but lower than that in other regions. The overall trend showed a slow downward trend of fluctuation. Compared with other regions, the downward trend was obvious. Heilongjiang province ranked the third among the northeast regions, and the overall level of science and technology output showed a declining trend. Compared with other regions, the downward trend was obvious. Compared with other regions, the subject of research institutions in Heilongjiang province was at the end of the science and technology output level. The overall level of science and technology output showed a downward trend, which was more obvious compared with other regions.

5.2.2. Comparative Analysis of Economic Output of Each Innovation Subject

The economic output level of enterprises in northeast China was relatively low, far lower than that of other regions. Compared with other regions, the change range was larger, showing a slow downward trend. The economic output level of enterprises in Heilongjiang province was at the end position, and the ranking was basically at the end position. The overall trend showed a downward fluctuation, and the decline was more obvious. Compared with other regions, the level of economic output showed a downward trend of fluctuation. Contrary to the change in most zones, the decline was more obvious.

The economic output level of universities in northeast China was slightly higher than that of the Pearl River Delta, and lower than that of the Beijing-Tianjin-Hebei region, the Yangtze River Delta, the Western Triangle and the Middle Triangle. The economic output level of universities in Heilongjiang province was at the middle and lower level, and the overall economic output level showed a fluctuating trend. Compared with other regions, the fluctuation was more obvious and had certain development potential. Compared with other regions, the main economy of universities was in the terminal position, showing 12-16 fluctuations. The overall ranking showed a slow upward trend, but the trend was not obvious. Compared with other regions, there were significant differences. The degree of fluctuation was relatively large.

The economic output level of research institutions in northeast China was 0.0781, which was higher than that of the Middle Triangle, the Pearl River Delta and the Western Triangle. Compared with other regions, the fluctuation range was small. The overall level of Heilongjiang province was in the terminal position, and the overall level of economic output presented a slow rising trend. However, compared with other regions, the upward trend was not obvious. Compared with other regions, the subject of research institutions in Heilongjiang province was at the end of the economic output level, showing a slow rise in the ranking. The overall level showed a slow upward trend. Compared with other regions, the degree of volatility was small and the upward trend was slow.

5.2.3. Comparative Analysis of Social Effect of Each Innovation Subject

The social effect level of the subject of enterprises in northeast China was lower than that in other regions, showing a fluctuation trend of 0.20-0.25, and the development trend was slow. The social effect level of the subject of enterprises was in the terminal position, and the overall social effect level presented a slow decline trend. Compared with other regions, the downward trend was more obvious and the decline range was larger. The ranking of Heilongjiang province showed a slow decline and fluctuation, and the overall social effect level showed a slow decline. Compared with other regions, the decline was larger.

The social effect level of universities in northeast China was 0.1662, lower than that in the Middle Triangle, the Pearl River Delta, the Beijing-Tianjin-Hebei region, the Yangtze River Delta and the Western Triangle, showing a slow decline. Compared with other regions, the level of social effect was decreasing significantly. The overall level of Heilongjiang province was in a medium dominant position, and the overall level of social effect showed a fluctuating trend. Compared with other regions, the fluctuation range was larger. Heilongjiang province was in the middle position of social effect level, rising from 10th to 7th in the ranking, and the level of social effect showed a fluctuating trend. Compared with other regions, the degree of volatility is small.

The social effect level of research institutions in northeast China was relatively low. The overall level of social effect showed a slow fluctuation trend. Compared with other regions, the fluctuation trend was not obvious, and had increased significantly in the past three years. Heilongjiang province’s overall level was in an advantageous position, ranking from the third to the first place. The overall social effect level presented a slow rising trend. Compared with other regions, the upward trend was obvious. The social effect level of research institutions in Heilongjiang province was in the middle position, rising from 15th to 9th in the ranking. The overall social effect level presented a slow rising trend. The upward trend had been slow compared with other regions.

5.2.4. Comparative Analysis of Innovation Performance of Each Innovation Subject

The innovation performance level of enterprises in northeast China was lower than that in other regions, with a fluctuation trend of 0.20–0.25 and a slow development trend. The innovation performance level of enterprises was at the end position, and the ranking was basically at the end position. The overall level showed a slow rise and then a downward trend, basically changing between 0.15 and 0.23. Compared with other regions, the innovation performance level of Heilongjiang province was at the end position, only slightly higher than Hebei province. The overall trend was that of a slow rise followed by a slow decline.

The innovation performance level of universities in northeast China was 0.2687, higher than that in the Middle Triangle and the Pearl River Delta, showing an upward trend and then a downward trend. Heilongjiang province ranked the second in the northeast region, and the overall level was in the middle position. The overall level of innovation performance showed a trend of decline and then a slow rise, which had a certain development potential. Compared with other regions, the innovation performance level of Heilongjiang province was in the middle position of the innovation performance level. The overall ranking showed a slow upward trend.

The innovation performance level of research institutions in northeast China was relatively low, which was slightly higher than that in the Middle Triangle region and lower than that in other regions. Heilongjiang province ranked third in the northeast region and was at the end of the scale. The overall innovation performance level showed a trend of rising first and then declining. Compared with other regions, the innovation performance of research institutions in Heilongjiang province was at the end of the scale. The overall innovation performance level showed an upward trend and then a downward trend, with a maximum decrease of about 0.08 levels.

5.3. Selection of Key Factors Influencing Innovation Performance

5.3.1. Selection of Key Factors Influencing Regional Overall Innovation Performance

(1) Cause factor analysis. The market environment changes the business performance of enterprises, and then acts on the flow degree and concentration direction of regional innovation activities. Both the initial innovation input and the subsequent innovation achievements into the actual production process need a lot of capital support. The improvement and convenience of the financial environment affect the acquisition of funds and the operation cost of innovative production activities. The intellectual property protection system affect the transaction cost of innovation activities, the ownership of innovative products, and thus affect the allocation status and efficiency of various resources, especially intellectual resources. This is conducive to transforming the innovation output into actual products and improving the actual productivity of the region. However, at the same time, it should also be noted that the higher level of intellectual property protection may also inhibit the diffusion and dissemination of knowledge to a greater extent. The key is to generate real economic value from the resulting patented products. This includes trading in technology markets that produce direct economic benefits, as well as turning patented products into commodities that generate economies of scale by satisfying consumer needs and spreading them widely. As a special factor of production, human capital can directly affect the actual productivity level of a region by participating in the production process. Human capital not only promotes technology transfer and diffusion from original country to imitation country, but also improves the probability of technology potential being transformed into innovation output.

(2) Result factor analysis. The result factors include the number of invention patent authorizations per 10,000 R&D personnel, the proportion of the number of invention patent applications in the number of patent applications, the number of scientific and technological papers included in the main foreign retrieval tools per 10,000 R&D personnel, and the ownership of trademarks per hundred enterprises. The number of patent authorizations is an important form of intermediate output of innovation activities, and the gold content of invention patents is the highest among the three kinds of patents. In addition to patents, scientific and technological papers are important achievements produced by regional universities and research institutes, and trademarks are also important achievements produced by regional enterprises.

(3) Center analysis. The top four factors, which are the proportion of new product sales revenue in main business revenue, the proportion of technology market turnover in regional GDP, the turnover of technological achievements per 10,000 R&D personnel, and the per capita GDP of the region, are the four factors with the greatest influence in regional innovation performance.

5.3.2. Selection of Key Factors Influencing Enterprise Innovation Performance

(1) Cause factor analysis. Enterprises are the subject of the market, the micro-foundation of the economy, and the actual carrier of innovation activities and economic output. Enterprises, research institutions and universities are the main implementers of R&D activities. Among them, enterprises prefer some applied research that can be quickly converted into productivity. Therefore, the investment of enterprises in R&D activities is an important backbone of a regional innovation system. The internal system of enterprises plays a guiding and managing role in the innovation activities of enterprises. Moreover, it directly affects the cycle and scale of innovation output into actual products. The institutional arrangement within enterprises, the investment in scientific research, and the interactive connection between enterprises all have a significant impact on the regional innovation performance. Different types of organizations have different goals, structures, and operating mechanisms, so their efficiency in engaging in innovative activities is varied. Creative talents have knowledge effect, knowledge reproducibility, no depletion, and diffusion. Its value condenses in the product, and affects the later innovation investment and the economic production. Therefore, it changes the regional innovation performance.

(2) Result factor analysis. The result factors include the number of valid invention patents owned per 10,000 R&D personnel, the proportion of invention patent applications to patent applications, and the ownership of trademarks per 100 enterprises. The number of patent authorizations is an important form of intermediate output of innovation activities. The gold content of invention patents is the highest among the three kinds of patents. In addition to patents, trademarks are an important achievement produced by regional enterprises.

(3) Center analysis. The proportion of new product sales revenue in the main business revenue, the number of new product development projects per hundred enterprises, and the labor productivity of high-tech industries ranked in the top three, are the three factors with the greatest influence on the innovation performance of regional enterprises.

5.3.3. Selection of Key Factors Influencing Universities’ Innovation Performance

(1) Cause factor analysis. The factors include the number of scientists and engineers working full-time in R&D, the number of senior titles in teaching and research personnel, and the total internal expenditure in the expenditure of science and technology funds. Based on the process of knowledge innovation in universities, it can be seen that the human resources with knowledge innovation ability in universities are generally knowledgeable professors and experienced scientists and engineers. Therefore, the human input that affects the innovation performance of regional universities are mainly the scientists with teaching and scientific research tasks, and the teaching and scientific research personnel with senior titles. University expenditure on science and technology includes the expenses of researchers, business expenses and the purchase of fixed assets. In the process of knowledge innovation, universities should fully reflect the innovation performance of regional universities. These factors have important influence on the innovation performance of regional universities.

(2) Result factor analysis. The result factors include the number of scientific and technological papers per 10,000 R&D personnel, the number of scientific and technological works published per 10,000 R&D personnel, and the proportion of the number of invention patent applications in the number of patent applications. Scientific and technological papers are one of the most direct and important results of regional university innovation. In addition, patents are also an important outcome of innovation. The number of patents granted is an important form of intermediate output of innovation activities, and invention patents have the highest gold content among the three types of patents.

(3) Center analysis. The number of R&D achievements’ application and technology service projects of each university ranked in the top three items, the science and technology progress award of each department of the state council of each university, and the number of graduate students cultivated by each 10,000 R&D personnel are the four most influential factors in the subject innovation performance of regional universities.

5.3.4. Selection of Key Factors Influencing Research Institutions’ Innovation Performance

(1) Cause factor analysis. The factors include the total number of personnel engaged in R&D activities, scientific research instruments and equipment above 100,000 yuan, and the number of national or departmental recognized qualifications and institutions. Compared with the innovation function of enterprises, research institutions focus more on basic research and experimental development functions in the system. Generally, it adopts three development modes: Independent innovation, imitation innovation and joint participation of various parties. Talents engaged in R&D activities have reproducibility, non-dissipation and diffusion of knowledge, thus changing the innovation performance of regional research institutions. Due to research institutions having to carry out basic research or applied basic research innovation activities, the equipment requirements are very high. The quality and use of laboratory equipment affect the experimental results, the practice of creative ideas, and the innovation performance of research institutions. While doing basic research, research institutions fully consider the transformation of basic research results. If there is national or departmental recognition, it can effectively improve their reputation. It has an important impact on obtaining the benefits of scientific and technological innovation.

(2) Result factor analysis. The result factors include the number of scientific and technological papers per 10,000 R&D personnel, the number of scientific and technological works published per 10,000 R&D personnel, and the proportion of the number of invention patent applications in the number of patent applications. Scientific and technological papers are one of the most direct and important results produced by regional research institutions. In addition, patents are also an important outcome of innovation. The number of patents granted is an important form of intermediate output of innovation activities, and invention patents have the highest gold content among the three types of patents.

(3) Center analysis. The order of center degree is C6 > C5 > C4 > C2 > C7 > C3 > C8 > C9 > C1. The transfer of patent ownership and license income per 10,000 R&D personnel ranked in the top two items, and the number of national or industry standards formed by each research institute are the two factors with the greatest influence on the innovation performance of regional research institutions.

6. Conclusions and Future Prospects

The coordinated development of regional innovation in Heilongjiang province plays an important role in stimulating the regional innovation and entrepreneurship vitality. This can not only accelerate the formation of a regional economic system with innovation as the main guide and support, but also play an important role in promoting the healthy and sustainable development of Heilongjiang’s regional economy. Although, at present, Heilongjiang province is rich in innovation factor endowment and has great potential for innovation-driven development, many innovation resources cannot be effectively converted into real productive forces and their contribution to economic development is limited. The allocation of regional innovation resources in Heilongjiang province should be adjusted scientifically and effectively to alleviate the shortage of regional innovation-driven development.

In this study, the combined evaluation method of drift degree based on entropy weight method, maximum deviation method, TOPSIS method and grey relational degree method was used to carry out the dynamic evaluation of regional innovation performance in Heilongjiang province. The grey-DEMATEL model was further applied to select the key factors influencing the regional innovation performance in Heilongjiang province. The purpose of this study is to summarize the innovation experience of each innovation subject and select the key factors that influence the regional innovation performance in Heilongjiang province. On this basis, this study puts forward countermeasures to improve regional innovation performance in Heilongjiang province based on regional economic development. The results of this study are as follows.

(1) In the last 10 years, the innovation vitality of enterprises was insufficient, and the quality of innovation output was average in Heilongjiang province. The number of high-tech enterprises was obviously low. The theoretical innovation and quantity of innovation in universities should be further strengthened. The number of government research institutes in Heilongjiang decreased a lot, while the number of enterprise research institutes increased a little. The total quantity presented the decreasing trend, while Heilongjiang province still had certain development potential.

(2) The innovation performance level of Heilongjiang province ranked second in northeast China, and its innovation performance level and ranking rose steadily. Compared with other regions, the innovation performance level of Heilongjiang province was in the middle and lower level, which was only slightly higher than that of Hebei, Jilin, Jiangxi and Hunan. However, the trend was not obvious, and the ranking was in a fluctuating trend.

(3) The innovation performance level of enterprises in Heilongjiang province was at the end, which was only slightly higher than that in Hebei province. The overall trend showed a slow rise first and then a slow decline. The innovation performance level of universities in Heilongjiang province was in the middle position, showing the trend of first rising, then falling and then rising. The overall ranking showed a slow upward trend. The innovation performance level of research institutions in Heilongjiang province was in the terminal position. The overall innovation performance level showed a trend of first rising and then declining, with the largest decline of about 0.08 level.

(4) At present, the degree of marketization, financial environment, intellectual property system, transformation of scientific and technological achievements and human capital level are the core factors influencing the regional innovation performance in Heilongjiang province. R&D investment, human resources investment, enterprise management system and enterprise organization structure are the core factors that affect the innovation performance of enterprises in Heilongjiang province. The number of scientists and engineers in universities, the number of senior titles in teaching and research personnel, and the total internal expenditure in science and technology expenditure are the core factors that affect the innovation performance of universities in Heilongjiang province. The total number of personnel engaged in research activities, the number of R&D instruments and equipment above 100,000 yuan, the qualification recognized by the state or departments and the number of institutions are the core factors that affect the innovation performance of regional research institutions in Heilongjiang province.

This study lays a theoretical foundation for the formulation and implementation of the regional innovation-driven development strategy of Heilongjiang province, and provides practical guidance for the regional supply-side structural reform and the promotion of regional economic development in Heilongjiang province. In order to promote the regional sustainable innovation, this study puts forward the following suggestions. (1) Regional enterprises in Heilongjiang province should fully rely on universities and research institutes to establish strategic alliances of industry, university and research institutes within the region. When regional enterprises carry out innovation activities, they should pay attention to the guiding role of the market and seek inspiration for innovation from the market. At the same time, the monographs, papers, intellectual property and other innovative achievements in Heilongjiang province are transformed into products to meet people’s needs. (2) The government should strengthen the cooperation between industry, university and research institutes in Heilongjiang province. The government should also fundamentally promote the comprehensive cooperation between universities and enterprises in Heilongjiang province and realize the industrialization of scientific and technological achievements. (3) Research institutes in Heilongjiang province should issue corresponding incentive policies. These policies should improve the enthusiasm and initiative of scientific and technological personnel so as to produce more high-level scientific and technological achievements. In particular, the authorization of patents can improve the conversion rate of scientific and technological achievements.

Although these important results were obtained in this study, there are some shortcomings, which are also an important direction for future research. First, the development proposal has very good applicability to promote the regional development of Heilongjiang province, and further study is needed for different regions. Second, the evaluation index system in this study can be used as the basic index system for the innovation performance evaluation of regions and various innovation subjects. Future study can be further expanded and improved for different regions.