Evaluating the Performance of Systemic Innovation Problems of the IoT in Manufacturing Industries by Novel MCDM Methods

Abstract

The Internet of Things (IoT) is an important technological innovation that can enhance industrial competitiveness and sustainability. Thus, governments need to carefully construct an innovation portfolio that promotes sustainable IoT development. To help define an accurate innovation policy and promote development of the IoT industries, potential problems in terms of systemic perspectives should be examined. Such problems, so-called “systemic innovation problems”, influence and block sustainable development of IoT technology as well as the IoT industry. However, past studies that explored systemic innovation problems in IoT-related industries are limited. Thus, this research aims to explore systemic innovation problems related to configuring an IoT innovation policy portfolio. A hybrid Bayesian rough based evaluation model was used to derive the most feasible policy instruments. The modified Delphi, Bayesian Rough Decision-Making Trial and Evaluation Laboratory Based Network Procedures (BR-DNP), and the modified Bayesian rough Vlse Kriterijumska Optimizacija I Kompromisno Resenje (MBR-VIKOR) were introduced. Gaps in performance corresponding to each systemic innovation problem can thus be assessed based on the features of technological innovation systems. The applicability of the proposed model for promoting industrial sustainability of IoT in the Taiwanese smart manufacturing industry (based on the opinions provided by Taiwanese experts) was verified by an empirical study. Eleven systemic innovation problems that influence the development of the IoT for the smart manufacturing industry were compared and ranked. Based on the results of the empirical study, the performance-gap ratio of “low level of interdisciplinary collaboration” problem is the lowest, as compared to other systemic innovation problems. In addition, the systemic functions of entrepreneurial activities and knowledge development are relatively more important than other systemic functions. The empirical results can serve as a basis for planning an IoT innovation policy portfolio definition and roadmap. Moreover, suggestions for enhancing current systemic innovation problems are provided for policy makers and industrial researchers, according to the results of the evaluation.

1. Introduction

The objective of innovation policy definition is to influence and help the development of technological innovation by public actions, such as determination of research budgets, direction of education, and infrastructure establishment [1]. Comparing innovation policy to conventional public policy, the former is based on the concept of multi-faceted factors to be formulated. In other words, innovation policy is more flexible and can effectively address actual problems by considering a variety of factors. To strengthen the competitiveness of an economy, innovation policy is often viewed as an essential instrument adopted by governments. However, it is not easy to define innovation policy, since such definitions always involve many complex factors. Hence, it is necessary for government sectors to systematically consider how innovation policies for supporting technological innovations should be developed.

Recently, the policy analysis of systemic innovation has received considerable attentions [2,3,4,5]. The concept of systemic innovation analysis regards the process of innovation policy development, as a problem of systemic improvement. In this sense, when government sectors intend to formulate a related innovation policy to help develop a specific technology, the initial task for government sectors is to structurally find out and understand what kind of challenges and difficulties face the specific industry. These challenges, called systemic innovation problems, influence and block technological innovation development. Next, policy makers should perform a systemic assessment and define corresponding solutions. In other words, once systemic innovation problems are identified, subsequent policy intervention can be planned and conducted to improve these systemic problems and help realize the eventual goal of technological innovation and thus, industrial sustainability. Industrial sustainability is defined as the end state of a transformation process where an industry is one part of, and actively contributing to, a socially, environmentally and economically sustainable planet [6]. For simplicity, industrial sustainability can be regarded as the process of the sustainable development in specific industries. Industrial sustainability has become a research domain in recent years. For technological innovations, understanding how an industrial ecosystem can be well constructed and developed can contribute to the sustainability of an industry. Related issues are very important for national governments.

In recent years, the application of systemic approaches on analyzing and improving the performance of technological innovation has achieved immense successes [4,7,8]. These achievements can be attributed to two major factors. First, scholars have demonstrated the influential effects of the technological innovation system (TIS) on the development and diffusion of new technologies [2,8,9]. Therefore, using the theory of TIS to analyze systemic innovation problems associated with definitions of innovation policy is appropriate. Second, the feasibility and effectiveness of applications of TIS functions on structural problem analysis have been demonstrated [10,11,12,13]. Such theoretical models can be utilized to analyze the challenges from current technology industries and, based on analytical results, provide solutions for industrial development and technological innovations. Furthermore, the systemic analysis for factors influencing and blocking technological innovation and industrial sustainability can inform policy makers of critical information essential for decision making.

The Internet of Things (IoT), a novel technology that recently emerged, can connect robots, artificial intelligence (AI), and big data analytics. As such, IoT will become the dominant technology in the next decade. A typical example of the IoT is the network-capable computer numerical control (CNC) machines that can enhance the performance of production lines. Such a networking of equipment enables engineers to adjust the operating mode by uploading the new or revised CNC programs (e.g., [14,15,16]) from a remote site. Engineers do not need to install these programs on-site by themselves [17]. Furthermore, many machining monitoring systems based on the AI process models have been successfully developed in the past for optimizing, predicting and/or controlling machining processes [18]. The AI technology can also be applied to network-capable CNC machines to save time by enabling self-diagnosis functions (e.g., [18,19,20]).

Thus far, advanced economies, such as Germany and Japan, have begun to promote the application of IoT techniques and the development of IoT industries by defining innovation policies. However, the facilitation of IoT development in emerging or quickly catching-up economies still lags far behind the progress made in leading countries due to various barriers and factors. Without interventions by innovation policy, the development of such technologies in quickly catching-up economies (e.g., China, South Korea, and Taiwan) will remain far behind efforts in advanced countries (e.g., Germany and Japan). Therefore, examining potential problems from a systemic perspective is essential. Considering this issue, numerous scholars have broadly discussed how to design an effective analytic framework to configure or re-configure innovation policy portfolios. Such an analytic framework can help policy makers identify weaknesses, figure out the current situation of IoT development, and design corresponding innovation policy portfolios. However, very few researchers have evaluated systemic innovation problems for the sustainability of industrial IoT in emerging economies [3,9,11,21]. In order to fill this gap, this article aims to explore systemic innovation problems that are hindering IoT industrial sustainability and further improve these systemic innovation problems to promote IoT industrial sustainability.

The assessment of various systemic innovation problems affecting industrial sustainability and technological development can extensively be viewed as a multiple criteria decision making (MCDM) problem. Such an evaluation process, that is based on MCDM methods, is often associated with experts’ reasoning and personal experiences [22,23]. Based on prior studies, human judgments involved with linguistic variable transformation associated with MCDM problems always generate imprecision information such as bias and vagueness. Therefore, the fuzzy and grey theories have been proposed by academic scholars to address the aforementioned inaccurate information problems. In addition to these two theories, the rough interval number that is based on the Rough Set Theory for addressing imprecision information issues has also been widely adopted recently [2,24,25]. Furthermore, the assessment process that was defined in the analytic work by most MCDM research often neglected prior and posterior conditions of event-occurrence, which can also lead to an imprecise evaluation [2]. To solve this problem, Bayesian theory is an effective approach since it is mainly built by both prior and posterior conditions of the event occurrence. Consequently, since Bayesian theories consider previous conditions of an analytic objective, the evaluation process will be rational and suitable.

Given the above reasons, a novel hybrid Bayesian Rough framework will be proposed to solve these problems [2]. The analytic framework will first explore the systemic innovation problems that can influence the application of the IoT techniques based on literature review results and interviews with experts. Second, the modified Delphi method is used to confirm problems derived from the literature review and interviews of experts. Third, the influential weights corresponding to the criteria and systemic functions will be derived by the Bayesian Rough DEMATEL-based ANP (BR-DNP) approach. Finally, performance of the optimal systemic innovation problem and enhancement goals for IoT industrial sustainability are identified and ranked by leveraging the modified rough VlseKriterijum-ska Optimizacija I Kompromisno Resenje (VIKOR) method.

An empirical study that explores and evaluates systemic innovation problems associated with IoT-sustainability in the Taiwanese manufacturing industry is introduced for verifying the proposed analytic framework based on opinions of experts. In practice, this hybrid systemic model that is based on performance-gap analysis informs policy makers of the insights from which to derive crucial criteria and priorities associated with all systemic innovation problems. The analytic framework can also derive factors with superior performance and enhance the weakness associated with the systemic innovation problems. From the aspect of research methods, the proposed model can effectively address inaccurate information collected from experts’ assessment.

The remainder of this paper is organized as follows. In Section 2, past work regarding systemic innovation problems within innovation policy definitions as well as the adoption of MCDM methods for performance evaluation are reviewed and discussed. Additionally, research gaps are addressed. In Section 3, a novel hybrid Bayesian Rough model is developed for exploring and assessing systemic innovation problems in IoT industrial sustainability. Section 4 demonstrates the effectiveness and feasibility of the proposed analytic framework based on an empirical case study of smart manufacturing industries in Taiwan. Discussions and implications are presented in Section 5. Finally, Section 6 concludes the entire work.

2. Literature Review

This section introduces configuring or reconfiguring an innovation policy portfolio for systemic innovation. Based on results of the literature review, a holistic understanding can be constructed. The evaluation of systemic innovation problems can be considered to be a type of performance assessment issues. Hence, MCDM methods are suitable for such studies. A review of past work on the applications of MCDM methods in performance assessment is constructive for further introducing these methods for evaluating the performance of innovation policy.

2.1. Configuring or Reconfiguring an Innovation Policy Portfolio for Systemic Innovation

During the past several years, both structural and functional analyses have been extensively applied to development of innovation policies [13]. Based on the definition of theories related to innovation systems, structural analyses associated with the components, which include actors, networks, and institutions, were often used to analyze national and general innovation systems [26,27]. In structural analyses, actors stand for the stakeholders of an innovation system, which include private and public actors as well as the research staff of technology adopters; these stakeholders can contribute to the innovation(s) of some specific technology [13]. The networks in structural analyses represents the cooperation between stakeholders [13] while institutions represent the rules that regulate action and interaction of actors in the innovation system [28].

Although structural analyses have been widely used to uncover the development of specific technologies and industries, application of such theoretical frameworks to the exploration of technological innovations are rare. Thus, based on the theoretical groundwork of structural analysis, assessment analysis, which incorporates several different aspects and is called function analysis, was proposed to further explain the systemic innovation process in which the systemic functions are confirmed. Thus, functional analysis can further be used to evaluate the performance of an innovation system [13].

According to Kieft, et al. [29], structure and functions are interdependent and both should be used together to identify systemic problems in innovation systems. The combination of structural and functional analyses offers an analytic framework that can assist policy makers to identify systemic innovation problems and help define systemic policy instruments that can cross the barriers confirmed by industrial experts, government officers, and scholars. In short, by integrating all the features of structural and functional analyses, a systemic analytic process can be established. In such an analytic process, systemic problems that hinder industrial development and technological innovation can be defined by related actors from industry, government sectors, and universities. Then, systemic functions are used to explore potential reasons for the problems in systemic innovation. Finally, a set of feasible systemic innovation policy instruments can be defined to solve systemic problems, which accelerates technological and industrial development [8,9,30].

To date, such systemic analytic models have been used in empirical validations of several industries. Results of these empirical studies contain many implications for the improvement of innovation systems, as well as formulation and configuration of innovation policy portfolios. Bergek, Jacobsson, Carlsson, Lindmark and Rickne [26] defined a systemic innovation framework for integrating systemic functions based on the technology innovation system (TIS) for analyzing systemic problems and also provide policies as solutions. The framework has been verified by using three empirical cases, which include the TISs for home-care technology, mobile data, and bio-composites.

The analytic framework proposed by Bergek, Jacobsson, Carlsson, Lindmark and Rickne [26] has been demonstrated to be feasible and effective. Thus, this framework can be used in other studies of policy formulations. Further, Purkus, et al. [31] used a systemic innovation perspective for exploring the issue of renewable energy sustainability that increased the demand for bio-based resources in the German wood industry. Based on the research by Purkus, Hagemann, Bedtke and Gawel [31], functions of a TIS model were adopted as evaluation criteria for analyzing systemic problems that block adoption of renewable resources. These problems include lack of a credible long-term transition policy mix, shortage of an effective selection environment, and lack of a broad advocacy coalition [31]. Sixt, Klerkx and Griffin [7] used the TIS model to identify and analyze systemic problems that hamper water conservation in the agriculture industry.

In light of the above literature review, this research identifies systemic innovation problems in terms of structural components. Next, systemic functions of the TIS model, proposed by Wieczorek and Hekkert [13], are adopted to assess systemic innovation problems. Based on the analysis that adopted the TIS framework, three main purposes can be achieved: (1) identification of systemic innovation problems that hinder technological innovation and industrial sustainability; (2) determination of the performance of systemic innovation problems that is based on evaluation of systemic functions versus all systemic innovation problems. Meanwhile, the strength of any systemic criterion can be derived; and (3) policy suggestions for enhancing systemic innovation problems can be obtained.

2.2. Systemic Functions of the TIS

The applicability of the TIS has been demonstrated for analyzing the diffusion of technologies [3,8,32], technological innovation development, and industrial sustainability in a variety of technological domains and industries [2,31]. The TIS is based on systemic views of technological and industrial evolution and provides policy makers with an extensive theoretical structure to investigate factors that can affect technological innovations. Further, solutions for these problems can be derived. In a technological innovation system, the factors influencing the definition of innovation policy, industrial sustainability, and technological innovations are termed systemic functions [13]. These systemic functions can be classified into seven functions according to the work by Wieczorek and Hekkert [13], including entrepreneurial activities, knowledge development, knowledge diffusion through network, guidance of the search, market formation, resource mobilization, and creation of legitimacy.

Table 1. Description of crucial systemic functions of technological innovation system (TIS).

Functions	Symbol	Features/Criteria
Entrepreneurial activities (F1)	e1	Experimenting new applications of IoT
	e3	Entry of firms to IoT markets
	e4	System for innovation and incubation
Knowledge development (F2)	k1	Conducting feasibility studies
	k3	Developing complementary technologies
Knowledge diffusion through networks (F3)	kd1	Training of professionals
	kd3	Organizing conference/workshops/seminars/meetings
	kd4	Demonstrations and exhibitions
Guidance of the search (F4)	g1	Setting collective goals for IoT development
	g2	Design of favorable rules and regulations
	g3	Publicizing expectations
	g4	Providing direction of development
Market formation (F5)	m1	Providing subsidies
	m2	Government procurement programs
	m3	Regulatory reform
	m4	Standardizations
Resource mobilization (F6)	r1	Providing R&D budgets
	r3	Launching IoT related education programs
	r4	Mobilizing human resources
	r5	Funding scale up on IoT projects
Creation of legitimacy (F7)	c1	Strength of lobby actions
	c2	Rise and growth of interest groups
	c3	Social acceptability

shows the collective systemic functions from TIS theory.

Entrepreneurial activities can transform the potential of new ideas, markets, and networks into certain actions to commercialize a novel technological innovation and explore new business opportunities [32]. To help commercial activities, experimentation with technological applications, launching of pilot projects, entry of firms into markets, and establishment of incubation systems can serve as the usable evaluation indicators for entrepreneurial activities.

Knowledge development often plays a key role in the process of early development of a specific technology [26]. In the experimental stage of a new technology, knowledge is likely to be transformed into successful technologies. Knowledge development is composed of several drivers, which include performing feasibility studies, researching technology markets, development of complementary networks of technology, and research cooperation [33].

Knowledge diffusion represents the promotion and spread of knowledge through networks in which stakeholders are involved and provide information and knowledge to each other [2]. Thus, the training of professionals, conducting promotional campaigns, and organizing conferences and workshops are important.

Guidance of search represents the selection process that is necessary to facilitate a convergence of technological and industrial development involving the direction and goal of innovation policy definitions [34]. In general, guidance of the search is composed of several indicators, comprised of setting collective goals for technology development, designing favorable rules and regulations, publicizing expectations, and providing direction of development [2,30].

Market formation is an important driver that can stimulate the emergence of markets for new products. To support market formation, subsidy provisions, government procurements, regulatory changes, and establishment of technical standards will be the key policy instruments that can be taken into consideration [13,30,35].

Resource mobilization refers to financial and human resources and infrastructure that are mobilized for development and diffusion of a new technology [28]. Hence, financial capital such as financial grants and government investment in R&D as well as human capital such as talent and educational programs are essential [13,30,35].

Creation of legitimacy, which indicates whether development of technological innovation can be accomplished, relies on the relationship between the advocacy coalition and policy makers [13]. Therefore, the function of creation of legitimacy gives rise to several criteria that are used for assessment of technological innovation and industrial development. These criteria include the strength of lobbying actions, growth of interest groups, and social acceptability [2,4,34].

2.3. Performance Evaluation and Innovation Systems

Evaluation and analysis of technological developments and industrial growth have been researched empirically in previous works linked to innovation and technology policy assessment. According to the definition of innovation system by Freeman [36], an innovation system is the “the network of institutions in the public and private sector whose activities and interactions initiate, import, and diffuse new technologies”. Hence, the development and process of industrial sustainability and technological innovation can be regarded as an innovation system.

Many researchers and policy analysts have tried to understand the structure, dynamics, and performance of innovation systems. Based on systemic analysis, policy makers and researchers can understand the performance of policy goals, policy problems, and systemic functions that can influence technological innovation and industrial development. In the following sections, previous work that described the performance of innovation systems is reviewed and summarized.

Hermans, Geerling-Eiff, Potters and Klerkx [34] adopted the systemic functions of TIS to evaluate various technologies, including energy production effects, propagation materials, and green genomics, all at different development stages. These systemic functions form various combinations in terms of different technologies. In this sense, specific technologies can be enhanced by analyzing the performance of systemic functions. Corresponding policies can be defined further. Arbolino, et al. [37] provided industrial sustainability indicators for evaluating the performance of industrial and innovation policies; these indicators were verified to be feasible based on an empirical case that was based on the Italian ecology industry. Chen, et al. [38] defined a mathematical model for evaluating R&D and commercialization efficiencies of innovation systems in Chinese high-tech industries.

Based on the published literature described above, innovation systems for industrial sustainability and technological innovation development can be evaluated and analyzed. The performance of innovation systems can also be improved accordingly. This paper evaluates the innovation system that is used to enhance the IoT-sustainability in the manufacturing industry; an evaluation framework for systemic innovation is also proposed and used to assess a set of systemic innovation problems that may hinder the sustainability of IoT industries. A performance-gap evaluation of each systemic innovation problems versus systemic functions is provided. Based on the analytic results, policy suggestions for improving systemic innovation problems can be derived as the basis for policy makers.

2.4. MCDM in Performance Evaluation

In any decision-making problem, decision makers always make decisions in which various factors need to be considered simultaneously. In this sense, MCDM models can be a suitable approach for addressing decision making problems. According to previous studies of performance evaluation, MCDM methods have broadly been applied and developed. For example, Lin, et al. [39] assessed the performance of different digital music providers and compared these providers using hybrid MCDM methods. Based on the work by Lin et al. (2016), advantageous and disadvantageous factors corresponding to each provider can be derived. Strategic suggestions were thus provided to improve current disadvantages. Lu, et al. [40] employed a DANP-VIKOR to explore adoption of RFID in the healthcare industry and evaluate the performances of two alternatives consisting of patient tracking management performance and asset tracking management performance. Liu, et al. [41] proposed a comprehensive MCDM model to assess factors affecting the implementation of complex national tourism policies. Gao, et al. [42] systematically evaluated performance levels among four options for nuclear fuel transition scenarios by drawing on the MCDM framework and dynamic system modeling. Additionally, their research analyzed the influence of actors of this system on the overall ranking utilizing different weighted modeling techniques. Büyüközkan and Güleryüz [43] took the aspects of economic, social, and the environment into account for evaluating the performance of an energy project with Turkey. The ranking results of their paper showed that the wind project is the best choice, as compared to other projects including biogas, hydro, and natural gas projects. By above mentioned literature, MCDM techniques for performance evaluation problems have gained significant attention and have been validated in empirical cases across different domains. Among many of the MCDM techniques, the VIKOR approach, which uses the concept of compromise solution in terms of a maximum group utility as well as minimum individual regret of the opponent, is considered to be a valid method for addressing performance evaluation issues [44].

On the basis of performance evaluation, there is an important problem with studies found in the literature. The imprecise information being generated by subjective judgments of experts can create considerable problems, which will contribute to bias in the evaluation results. Due to the complexity of assessment features, it is not often observed that the scores of evaluation features are presented as crisp numbers. Thus, approaches such as fuzzy, grey, and rough theories are incorporated into the evaluation models in order to address such problems. For example, Hsu, et al. [45] employed a fuzzy AHP method to explore the formation of new biotechnology firms in terms of the notions of policy tools. Wu, et al. [46] established a decision model for a large commercial rooftop system site selection by using the fuzzy ANP-VIKOR approach. In order to dispose of the imprecise information on such issues, the triangular intuitionistic fuzzy numbers were adopted. Qin, et al. [47] developed a model by hybridizing VIKOR and type-2 fuzzy methods to assess the risk of high-tech project investments. Validation comparative analysis was conducted between their proposed method and other interval type-2 fuzzy techniques. Parkouhi and Ghadikolaei [48] evaluated the performance of suppliers and further determined the best supplier corresponded with resilient capabilities of the company’s supply chain by using the fuzzy-ANP-VIKOR based on grey theory. Tiwari, et al. [49] used the rough interval number to reduce the imprecise content within the customer evaluation process and, thus, the product design was improved. In light of their analysis, a VIKOR-based rough interval number was used to evaluate the performance of product design concepts and identify the best concept in terms of collective judgments. Pamučar, et al. [50] proposed a novel MCDM framework for integrating the rough set theory to address vagueness in decision making and used such framework to assess the performance of alternatives. In order to verify the robustness of the proposed model, they introduced 36 scenarios to test the stability of ranking. Based on the aforementioned literature regarding the application of MCDM methods, it can be noticed that the subjective judgment of decision makers under vagueness conditions can be properly tackled with fuzzy, grey, and rough theories. Although such theories have advantages, it still has a problem where conventional methods such as triangular fuzzy numbers are built on the assumption of determining the linguistic scale. In contrast, the rough theory can treat imprecise information without this assumption and hence it can be regarded as an effective tool for converting the exact values into interval scores.

2.5. Research Gaps

In summary, several research gaps were found based on the above literature review results. First, previous empirical research on TIS theory has achieved huge success. However, performance-gap evaluations between systemic functions and systemic innovation problems were scarce [4,13,21,51]. Conventional TIS studies focused on performance analyses of innovations using systemic functions. How the performance of innovation can be enhanced to achieve the aspired goal based on systemic functions has been seldomly addressed. On the basis of innovation system enhancement, an understanding of the performance-gaps between the ideal goal and the current systemic problem is important. Therefore, a performance-gap evaluation approach will be indispensable to cross the research gap. Moreover, policy suggestions and improvement directions for systemic innovation problems can be provided accordingly.

Second, the evaluation frameworks for uncovering and improving systemic problems that can influence industrial sustainability and technological innovations were rare [4,8,13,32]. In order to fill these research gaps, the authors propose a TIS based systemic performance evaluation framework with a corresponding analytic process. Innovation policy portfolio can thus be defined accordingly for solving systemic innovation problems that hinder industrial sustainability and technological innovations.

Third, previous studies of empirical applications of TIS model primarily relied on qualitative analyses. Although conventional qualitative methods have been widely adopted, these methods can be subjective and misleading [2,52]. Hence, an evaluation framework based on the systemic functions with quantitative approaches that emphasize the influence relationships can cross the research gap.

Finally, empirical studies on analyzing the development and innovation of IoT in manufacturing industries are still rare. For most catching-up economies like Taiwan, the commercialization of IoT applications and innovations will be essential for accelerating industrial and digital transformation. Thus, deriving and solving potential systemic problems that are influencing sustainable development of the IoT in manufacturing industries is important. This systemic analytic model based on the TIS theory can analyze the performance-gap by focusing on different systemic innovation problems. The analytic results will be very helpful for enhancing the sustainability of IoT in manufacturing industries.

3. Proposed Evaluation Framework for Systemic Innovation Problems

The analysis by Bayesian Rough MCDM framework for evaluating and exploring the systemic innovation problems in IoT innovation policy definition is divided into several steps: (1) the modified Delphi method is employed to assess the appropriateness of systemic innovation problems, (2) applies the BR-DNP method to derive and analyze the influential weights, and (3) evaluates the performance-gaps of systemic innovation problems by drawing on modified Bayesian Rough VIKOR (MBR-VIKOR) based on influential weights by BR-DNP method. The analytic procedure is illustrated in Figure 1.

3.1. Modified Delphi Method

Compared to the traditional Delphi method, the objective of modified Delphi approach is to simplify the procedure of survey and summarize the survey results [53]. Such method accelerates the survey process including the results of anonymous decision makers that communicate in written and discussion on a certain issue [54]. By doing so, collective opinion consensus will be more easily achieved. This method can be divided into several stages: (1) the expert members will be determined and invited to participate the first round survey; (2) the second round survey is conducted after the finish of first round survey; (3) conduct the third round of a questionnaire survey; and (4) combine the experts’ opinions and further analyze the results for checking if the collective opinions have reached the consensus [55]. In general, the last two rounds of survey can be neglected when first round survey has obtained collective consensus over 80% on a particular issue. On the basis of such survey process, the subsequent empirical analysis can be more quickly implemented. Modified Delphi method has been used frequently in various domains and obtained the huge successes [53,54,55,56]. Because the exploration of systemic innovation problems influencing the definition of IoT innovation policy is a special situation in which the available data is very limited and unexplored. Hence, in this present work, the modified Delphi method was chosen to policy makers for determining the suitable alternative problems in innovation system.

3.2. Bayesian Rough DEMATEL Based ANP Approach (BR-DNP)

DEMATEL based ANP method (DNP) has been extensively applied into explore various issues across several fields with empirical validation. The main characteristics of this method are to solve non-linear and complex relations and derived matrices can further be leveraged to generate the influential weights by integrating the concepts of ANP method. Given these advantages, this method enables the decision makers to effectively solve the current problems in practical. For most of real situations, practical problem evaluation often involves in multi-dimensional discussion. In many of empirical studies, treating current research problems often omits the prior possibilities associated with present problems. This will reflect the imprecise outcomes. To avoid this result, Bayesian theory can effectively deal with above matters. The Bayesian theory is based on the probabilistic theorem to deal with the problems in uncertainty environment. Through the probability of current conditions and prior probability, the posterior probability can be obtained and modified which will be closer to the real circumstance. This method has been verified in a wide variety of domains, including operation management [57], sustainable development for smart city [58], building simulation analysis [59], and software task effort evaluation [60]. Although Bayesian methods have received huge successes in various studies, for multiple criteria decision-making issues, there is a main problem. The values of conditional probability being obtained are mostly based on the opinions of expert groups. Under real situation, due to the complicated network relations among the criteria, using the crisp values for problem evaluation may not a suggest way. On the contrary, adopting the interval numbers for assessing the practical problem will be a relatively reliable method. Thus, introducing the concept of interval numbers, such as fuzzy theory or rough set theory [61], into the Bayesian method may be one of the ways to treat subjective judgment and imprecision problems. In this research, a novel model by hybridizing Bayesian theory, rough interval number, and DNP approach will be introduced for deriving the influential weights of criteria and functions. Based on the previous works [2,23,52], the procedures for manipulating the BR-DEMATEL are demonstrated below:

Step 1: Construct the initial group direct matrix, $M_{}^{\alpha }$, which contains the direct influences from one criterion to other criterion; these influences are summarized based on opinions being provided by experts.

For the construction of $M_{}^{\alpha }$ matrices, including the prior matrix of $M_{}^p$ and the conditional matrix of $M_{}^c$, the level of influence from function $i$ to function $j$ is assessed based on an integer scale from 0 to 4 (0—no influence, 1—low influence, 2—medium influence, 3—high influence, and 4—very high influence) by $q$ experts. The non-negative $n\times n$ matrix, $D{M}_k^{\alpha }$, is constructed accordingly, as depicted below:

$$D{M}_k^{\alpha }=\left[\begin{array}{cccc}0& m_{12}^{{\alpha }_k}& \cdots & m_{1n}^{{\alpha }_k}\\ m_{21}^{{\alpha }_k}& 0& \cdots & m_{2n}^{{\alpha }_k}\\ ⋮& ⋮& 0& ⋮\\ m_{n1}^{{\alpha }_k}& m_{n2}^{{\alpha }_k}& \cdots & 0\end{array}\right],k=1,\cdots ,q;\alpha =p,c.$$

(1)

Step 2: Determination of the rough group direct-influence matrix, ${\tilde{Z}}^{\alpha }$.

Based on the rough set concept comprising of lower approximation and the upper approximation, the lower limits $a_{ij(l)}^{{\alpha }_k}$ and upper limits $a_{ij(u)}^{{\alpha }_k}$ can be defined as follows:

$$\mathrm{Lower}\mathrm{limits}:\underset{\_}{\mathrm{Lim}}{a}_{ij(l)}^{{\alpha }_k}=\frac{1}{n_{ij(l)}^{\alpha }}{\displaystyle \sum _{m=1}^{n_{ij(l)}^{}}{\eta }_{ij}^{\alpha }}$$

(2)

$$\mathrm{Upper}\mathrm{limits}:\overline{\mathrm{Lim}}{a}_{ij(u)}^{{\alpha }_k}=\frac{1}{n_{ij(u)}^{\alpha }}{\displaystyle \sum _{m=1}^{n_{ij(u)}^{}}{y}_{ij}^{\alpha }}$$

(3)

When $\alpha =p$, the approximation matrix can be seen as the prior matrix, $M_{}^p$. When $\alpha =c$, the approximation matrix can serve as the conditional matrix, $M_{}^c$. ${\eta }_{ij}^{\alpha }$ is the lower approximation for $a_{ij(l)}^{{\alpha }_k}$ and $y_{ij}^{\alpha }$ is the upper approximation for $a_{ij(u)}^{{\alpha }_k}$. $n_{ij(l)}^{\alpha }$ and $n_{ij(u)}^{\alpha }$ stand for the number of objects which are included in the lower approximation and upper approximation, respectively. Next, the values in $D{M}_k^{\alpha }$ matrices will be converted as a rough interval number using Equations (2) and (3), as follows:

$$RN(a_{ij}^{{\alpha }_k})=[\underset{\_}{\mathrm{Lim}}(a_{ij(l)}^{{\alpha }_k}),\overline{\mathrm{Lim}}(a_{ij(u)}^{{\alpha }_k})]=[a_{ij(l)}^{{\alpha }_k},a_{ij(u)}^{{\alpha }_k}],$$

(4)

where the lower and upper limits are represented as $a_{ij(l)}^{{\alpha }_{kL}}$ and $a_{ij(u)}^{{\alpha }_{kU}}$, respectively, in $RN(a_{ij}^{{\alpha }_k})$. Moreover, these two limits depict the level of vagueness. The mean rough interval numbers of the prior and conditional rough sequences are defined as $\overline{RN({\tilde{a}}_{ij}^p)}$ and $\overline{RN({\tilde{a}}_{ij}^c)}$, respectively. The two means are generated using the rough derivation equations defined in Equation (5), as follows:

$$\overline{RN}({\tilde{a}}_{ij}^{\alpha })=\left[\frac{{\displaystyle \sum _{k=1}^m{\tilde{a}}_{ij(l)}^{{\alpha }_k}}}{m},\frac{{\displaystyle \sum _{k=1}^m{\tilde{a}}_{ij(u)}^{{\alpha }_k}}}{m}\right],$$

(5)

Based on the above calculations, the rough group direct influence matrices, ${\tilde{Z}}^{\alpha }$, can be defined as follows:

$${\tilde{Z}}^{\alpha }={\left[\overline{RN({\tilde{a}}_{ij}^{\alpha })}\right]}_{n\times n}=\begin{array}{c}\\ \left[\begin{array}{cccc}[0,0]& [a_{12(l)}^{\alpha },a_{12(u)}^{\alpha }]& \cdots & [a_{1n(l)}^{\alpha },a_{1n(u)}^{\alpha }]\\ [a_{21(l)}^{\alpha },a_{21(u)}^{\alpha }]& [0,0]& \cdots & [a_{2n(l)}^{\alpha },a_{2n(u)}^{\alpha }]\\ ⋮& ⋮& \ddots & ⋮\\ [a_{n1(l)}^{\alpha },a_{n1(u)}^{\alpha }]& [a_{n2(l)}^{\alpha },a_{n2(u)}^{\alpha }]& \cdots & [0,0]\end{array}\right]\end{array}$$

(6)

Step 3: Calculation of the normalized ${\tilde{D}}^{\alpha }$ matrix.

Matrix ${\tilde{D}}^{\alpha }$ is derived from the normalized group direct influence matrix, ${\tilde{Z}}^{\alpha }$. The ${\tilde{D}}^{\alpha }$ matrix, including ${\tilde{D}}^p$ and ${\tilde{D}}^c$, is acquired when each element, $\overline{RN({\tilde{a}}_{ij}^{\alpha })}$, of matrix ${\tilde{Z}}^{\alpha }$ is divided by the maximum value of every rough interval number, as depicted in Equation (7):

$${\tilde{D}}^{\alpha }={\left[\overline{RN({\tilde{d}}_{ij}^{\alpha })}\right]}_{n\times n}=\begin{array}{c}\left[\begin{array}{cccc}[0,0]& [d_{12(l)}^{\alpha },d_{12(u)}^{\alpha }]& \cdots & [d_{1n(l)}^{\alpha },d_{1n(u)}^{\alpha }]\\ [d_{21(l)}^{\alpha },d_{21(u)}^{\alpha }]& [0,0]& \cdots & [d_{2n(l)}^{\alpha },d_{2n(u)}^{\alpha }]\\ ⋮& ⋮& \ddots & ⋮\\ [d_{n1(l)}^{\alpha },d_{n1(u)}^{\alpha }]& [d_{n2(l)}^{\alpha },d_{n2(u)}^{\alpha }]& \cdots & [0,0]\end{array}\right]\\ \end{array}$$

(7)

Step 4: Establishing the total influence matrix, $\tilde{T}$.

Based on Equation (23), the total influence matrix, ${\tilde{T}}^{\alpha }$, where $I$ denotes the identity matrix of the $n\times n$ rank. The total influence matrix based on the prior situation can be denoted as ${\tilde{T}}_{}^{\alpha }=\left[T_{(l)}^{\alpha },T_{(u)}^{\alpha }\right]$.

$$\begin{array}{l}{T}_{(l)}^{\alpha }=(D_{(l)}^{\alpha })+{(D_{(l)}^{\alpha })}^2+\cdots +{(D_{(l)}^{\alpha })}^{\theta }=(D_{(l)}^{\alpha }){(I-D_{(l)}^{\alpha })}^{-1},\\ T_{(u)}^{\alpha }=(D_{(u)}^{\alpha })+{(D_{(u)}^{\alpha })}^2+\cdots +{(D_{(u)}^{\alpha })}^{\theta }=(D_{(u)}^{\alpha }){(I-D_{(u)}^{\alpha })}^{-1},\mathrm{when}\theta \to \infty .\end{array}$$

(8)

$${\tilde{T}}^{\alpha }={\left[\overline{RN({\tilde{t}}_{ij}^{\alpha })}\right]}_{n\times n}=\begin{array}{c}\left[\begin{array}{cccc}[0,0]& [t_{12(l)}^{\alpha },t_{12(u)}^{\alpha }]& \cdots & [t_{1n(l)}^{\alpha },t_{1n(u)}^{\alpha }]\\ [t_{21(l)}^{\alpha },d_{21(u)}^{\alpha }]& [0,0]& \cdots & [t_{2n(l)}^{\alpha },t_{2n(u)}^{\alpha }]\\ ⋮& ⋮& \ddots & ⋮\\ [t_{n1(l)}^{\alpha },d_{n1(u)}^{\alpha }]& [t_{n2(l)}^{\alpha },t_{n2(u)}^{\alpha }]& \cdots & [0,0]\end{array}\right]\\ \end{array}$$

(9)

After the calculation of the total influence matrix, by utilizing Equations (9) and (10), the posterior matrix can be obtained, as illustrated below:

$$\begin{array}{ll}\tilde{T}& =\tilde{\beta }\times P({\tilde{T}}^p|{\tilde{T}}^c)=\tilde{\beta }\times P({\tilde{T}}^p\cap {\tilde{T}}^c)/P({\tilde{T}}^c)\\ & =\tilde{\beta }\times \left(P({\tilde{T}}^c|{\tilde{T}}^p)P({\tilde{T}}^p)\right)/{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n{\tilde{t}}_{ij}^{c’}\times }}{\tilde{t}}_{ij}^{p’}\end{array}$$

(10)

where $P({\tilde{T}}^p)={\left[{\tilde{t}}_{ij(l)}^{p’},{\tilde{t}}_{ij(u)}^{p’}\right]}_{n\times n}$ and $P({\tilde{T}}^c)={\left[{\tilde{t}}_{ij(l)}^{c’},{\tilde{t}}_{ij(u)}^{c’}\right]}_{n\times n}$ stand for the probability matrices that are being normalized from the total influence matrices, and $\tilde{\beta }$ is a parameter acquired through the sum of posterior probability matrices. $\tilde{\beta }$ is used for transforming the posterior probability matrices into the posterior total influence matrix. The posterior rough total influence matrix of $\tilde{T}$ is shown below:

$$\tilde{T}={\left[\overline{RN({\tilde{t}}_{ij}^{})}\right]}_{n\times n}=\left[\begin{array}{cccc}[0,0]& [t_{12(l)}^{},t_{12(u)}^{}]& \cdots & [t_{1n(l)}^{},t_{1n(u)}^{}]\\ [t_{21(l)}^{},t_{21(u)}^{}]& [0,0]& \cdots & [t_{2n(l)}^{},t_{2n(u)}^{}]\\ ⋮& ⋮& \ddots & ⋮\\ [t_{n1(l)}^{},t_{n1(u)}^{}]& [t_{n2(l)}^{},t_{n2(u)}^{}]& \cdots & [0,0]\end{array}\right]$$

(11)

To examine the reliability of the collected data, the inconsistency rate is calculated using Equation (12):

$$\mathrm{Inconsistency}\mathrm{ratio}=\frac{1}{n(n-1)}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^n\left|\frac{t_{ij}^n-t_{ij}^{n-1}}{t_{ij}^n}\right|}}\times 100\%,$$

(12)

where $t_{ij}=(t_{ij(l)}+t_{ij(u)})/2,$ $t_{ij}^n$ is the average influence of factor $i$ on $j$. $n$ denotes the number of samples. An inconsistency rate that is less than 5% represents the great reliability of the collected samples.

Using Equations (13) and (14), row sums and column sums are respectively defined as $h_i=[h_{i(l)},h_{i(u)}]$ ($h_{i(l)}$ and $h_{i(u)}$ are the lower limit and upper limit) and $v_i=[v_{i(l)},v_{i(u)}]$ ($v_{i(l)}$ and $v_{i(u)}$ are the lower limit and upper limit) within the posterior rough total influence matrix, $\tilde{T}$:

$$h_i=[h_{i(l)},h_{i(u)}]=[{\displaystyle \sum _{j=1}^n{t}_{ij(l)}},{\displaystyle \sum _{j=1}^n{t}_{ij(u)}]}$$

(13)

$$v_j=[v_{j(l)},v_{j(u)}]=[{\displaystyle \sum _{i=1}^n{t}_{ij(l)}},{\displaystyle \sum _{i=1}^n{t}_{ij(u)}]}$$

(14)

Then, in order to obtain the crisp values, an effective de-roughness method proposed by Opricovic and Tzeng [62] is adopted. The de-roughness result, including $r_i$ and $c_i$ ($r_i$ and $c_j$ are the final transformed crisp values and $i=j$), is presented as follows:

$$r_i=\underset{i}{\min }{h}_i^L+\frac{{\tilde{h}}_i^L\times (1-{\tilde{h}}_i^L)+{\tilde{h}}_i^U\times {\tilde{h}}_i^U}{1-{\tilde{h}}_i^L+{\tilde{h}}_i^U}\underset{i}{(\max }{h}_i^U-\max h_i^L)$$

(15)

$$c_j=\underset{j}{\min }{v}_j^L+\frac{{\tilde{v}}_j^L\times (1-{\tilde{v}}_j^L)+{\tilde{v}}_j^U\times {\tilde{v}}_j^U}{1-{\tilde{v}}_j^L+{\tilde{v}}_j^U}\underset{j}{(\max }{v}_j^U-\max v_j^L)$$

(16)

where

$${\tilde{h}}_i^L=(h_i^L-\underset{i}{\min }{h}_i^L)/\underset{i}{(\max }{h}_i^U-\max h_i^L)$$

(17)

$${\tilde{h}}_i^U=(h_i^U-\underset{i}{\min }{h}_i^L)/\underset{i}{(\max }{h}_i^U-\max h_i^L)$$

(18)

$${\tilde{v}}_j^L=(v_j^L-\underset{i}{\min }{v}_j^L)/(\underset{i}{\max }{v}_i^U-\max v_i^L)$$

(19)

$${\tilde{v}}_j^U=(v_j^U-\underset{i}{\min }{v}_j^L)/(\underset{i}{\max }{v}_i^U-\max v_i^L),$$

(20)

By the obtains crisp values, the coordinate graph can be illustrated by mapping the ordered pairs of $(r_i+c_i{,}_i{r}_i-c_i)$. Subsequently, the cause-effect diagram can be derived accordingly. Furthermore, the causal network map is suitable for use by decision makers to examine the difference between cause and effect factors to determine useful strategies and policies $W=({\tilde{T}}_C^e{)}^{\prime }$.

Step 5: Computing the unweighted supermatrix $W=({\tilde{T}}_C^{\alpha }{)}^{\prime }$. Based on BR-DEMATEL method, the total influence matrix $\tilde{T}$ can be denoted as ${\tilde{T}}_C^{}$ by assuming that there were $p$ functions and $n$ criteria in $\tilde{T}$, as indicated in Equation (21) where ${\tilde{T}}_C^{ij}$ as a $p_i\times p_j$ submatrix.

$$\begin{array}{l}\begin{array}{cccccccccccc}& & & & & & F_1& \cdots & F_i& \cdots & \cdots & F_p\end{array}\\ {\tilde{T}}_C=\begin{array}{c}{F}_1\\ ⋮\\ F_i\\ ⋮\\ F_p\end{array}\left[\begin{array}{ccccc}{\tilde{T}}_C^{11}& \cdots & {\tilde{T}}_C^{1j}& \cdots & {\tilde{T}}_C^{1p}\\ ⋮& & ⋮& & ⋮\\ {\tilde{T}}_C^{i1}& \cdots & {\tilde{T}}_C^{ij}& \cdots & {\tilde{T}}_C^{ip}\\ ⋮& & ⋮& & ⋮\\ {\tilde{T}}_C^{p1}& \cdots & {\tilde{T}}_C^{pj}& \cdots & {\tilde{T}}_C^{pp}\end{array}\right]\end{array}$$

(21)

Through the normalization principle, the normalized matrix ${\tilde{T}}_C^{\alpha }$ can be obtained, as denoted in Equation (22).

$$\begin{array}{l}\begin{array}{ccccccccccccccc}& & & & & & & F_1& \cdots & & F_1& \cdots & & F_1& \end{array}\\ {\tilde{T}}_C^{\alpha }=\begin{array}{c}{F}_1\\ ⋮\\ F_i\\ ⋮\\ F_p\end{array}\left[\begin{array}{ccccc}{\tilde{T}}_C^{\alpha 11}& \cdots & {\tilde{T}}_C^{\alpha 1j}& \cdots & {\tilde{T}}_C^{\alpha 1p}\\ ⋮& & ⋮& & ⋮\\ {\tilde{T}}_C^{\alpha i1}& \cdots & {\tilde{T}}_C^{\alpha ij}& \cdots & {\tilde{T}}_C^{\alpha ip}\\ ⋮& & ⋮& & ⋮\\ {\tilde{T}}_C^{\alpha p1}& \cdots & {\tilde{T}}_C^{\alpha pj}& \cdots & {\tilde{T}}_C^{\alpha pp}\end{array}\right]\end{array}$$

(22)

where the submatrix ${\tilde{T}}_C^{\alpha 11}$ can be acquired by using Equations (23) and (24); Likewise, all of other submatrices ${\tilde{T}}_C^{\alpha nn}$ can also be obtained.

$$\begin{array}{l}\begin{array}{cccccccccccccc}& & & & & & & & & c_{11}& \cdots & c_{1j}& \cdots & c_{1p_1}\end{array}\\ {\tilde{T}}_C^{11}=\begin{array}{c}{c}_{11}\\ ⋮\\ c_{1i}\\ ⋮\\ c_{1p_1}\end{array}\left[\begin{array}{ccccc}{\tilde{t}}_{C11}^{11}& \cdots & {\tilde{t}}_{C1j}^{11}& \cdots & {\tilde{t}}_{C1p_1}^{11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{Ci1}^{11}& \cdots & {\tilde{t}}_{Cij}^{11}& \cdots & {\tilde{t}}_{Ci{p}_i}^{11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{C{p}_{1}1}^{11}& \cdots & {\tilde{t}}_{C{p}_{i}j}^{11}& \cdots & {\tilde{t}}_{C{p}_1{p}_1}^{11}\end{array}\right]\begin{array}{c}\to {\tilde{d}}_1^{11}={\displaystyle {\sum }_{j=1}^{p_1}{\tilde{t}}_{C1j}^{11}}\\ ⋮\\ \to {\tilde{d}}_i^{11}={\displaystyle {\sum }_{j=1}^{p_1}{\tilde{t}}_{Cij}^{11}}\\ ⋮\\ \to {\tilde{d}}_{p_1}^{11}={\displaystyle {\sum }_{j=1}^{p_1}{\tilde{t}}_{C{p}_{i}j}^{11}}\end{array}\end{array}$$

(23)

where $\begin{array}{cc}{d}_i^{11}={\displaystyle {\sum }_{j=1}^{p_1}{\tilde{t}}_{Cij}^{11}},& i=1,2,\dots ,p_1.\end{array}$

$$\begin{array}{l}\begin{array}{cccccccccccccccc}& & & & & & & & & & c_{11}& & \cdots & & c_{1j}& \begin{array}{cccc}\cdots & & c_{1p_1}& \end{array}\end{array}\\ {\tilde{T}}_C^{11}=\begin{array}{c}{c}_{11}\\ ⋮\\ c_{1i}\\ ⋮\\ c_{1p_1}\end{array}\left[\begin{array}{ccccc}{\tilde{t}}_{C11}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{C1j}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{C1p_1}^{11}/{\tilde{d}}_1^{11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{Ci1}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{Cij}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{Ci{p}_i}^{11}/{\tilde{d}}_1^{11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{C{p}_{1}1}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{C{p}_{i}j}^{11}/{\tilde{d}}_1^{11}& \cdots & {\tilde{t}}_{C{p}_1{p}_1}^{11}/{\tilde{d}}_1^{11}\end{array}\right]\\ \begin{array}{cccccccccccccc}& & & & & & & & c_{11}& \cdots & c_{1j}& \cdots & c_{1p_1}& \end{array}\\ \begin{array}{cc}& =\begin{array}{c}{c}_{11}\\ ⋮\\ c_{1i}\\ ⋮\\ c_{1p_1}\end{array}\left[\begin{array}{ccccc}{\tilde{t}}_{C11}^{\alpha 11}& \cdots & {\tilde{t}}_{C1j}^{\alpha 11}& \cdots & {\tilde{t}}_{C1p_1}^{\alpha 11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{Ci1}^{\alpha 11}& \cdots & {\tilde{t}}_{Cij}^{\alpha 11}& \cdots & {\tilde{t}}_{Ci{p}_i}^{\alpha 11}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{C{p}_{1}1}^{\alpha 11}& \cdots & {\tilde{t}}_{C{p}_{i}j}^{\alpha 11}& \cdots & {\tilde{t}}_{C{p}_1{p}_1}^{\alpha 11}\end{array}\right]\end{array}\end{array}$$

(24)

Then, the unweighted supermatrix $\tilde{W}={\left({\tilde{T}}_C^{\alpha }\right)}^{\prime }$ can be obtained in terms of transpose form, as shown in Equation (25).

$$\begin{array}{l}\begin{array}{cccccccccccccccccccc}& & & & & & & & & & & & F_1& \cdots & & F_j& & \cdots & F_p& \end{array}\\ \begin{array}{cc}\tilde{W}={\left({\tilde{T}}_C^{\alpha }\right)}^{\prime }& =\begin{array}{c}{F}_1\\ ⋮\\ F_j\\ ⋮\\ F_p\end{array}\left[\begin{array}{ccccc}{\tilde{W}}^{11}& \cdots & {\tilde{W}}^{i1}& \cdots & {\tilde{W}}^{p1}\\ ⋮& & ⋮& & ⋮\\ {\tilde{W}}^{1j}& \cdots & {\tilde{W}}^{ij}& \cdots & {\tilde{W}}^{pj}\\ ⋮& & ⋮& & ⋮\\ {\tilde{W}}^{1p}& \cdots & {\tilde{W}}^{ip}& \cdots & {\tilde{W}}^{pp}\end{array}\right]\end{array}\end{array}$$

(25)

where submatrix ${\tilde{W}}^{11}$ denotes values of criteria influences using transpose principle within F₁ aspect.

Step 6: Computing the weighted supermatrix ${\tilde{W}}^{\mathsf{\alpha }}{=\tilde{T}}_D^{\mathsf{\alpha }}\tilde{W}$. The weighted supermatrix ${\tilde{W}}^{\alpha }$ can be gained by multiplying normalized total influence matrix ${\tilde{T}}_D^{\alpha }$ and unweighted supermatrix ${\tilde{W}}^{}$. The normalized total influence matrix ${\tilde{T}}_D^{\alpha }$ is stated in Equation (26).

$${\tilde{T}}_D^{\mathsf{\alpha }}=\begin{array}{c}{c}_{11}\\ ⋮\\ c_{1i}\\ ⋮\\ c_{1p_1}\end{array}\left[\begin{array}{ccccc}{\tilde{t}}_{11}^{D_{11}}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{1j}^{D_{1j}}}& \cdots & {\tilde{t}}_{C1j}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{1j}^{D_{1j}}}& \cdots & {\tilde{t}}_{C1p_1}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{1j}^{D_{1j}}}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{i1}^{D_{i1}}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{ij}^{D_{ij}}}& \cdots & {\tilde{t}}_{Cij}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{ij}^{D_{ij}}}& \cdots & {\tilde{t}}_{Ci{p}_i}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{ij}^{D_{ij}}}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{p1}^{D_{p1}}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{pj}^{D_{pj}}}& \cdots & {\tilde{t}}_{C{p}_{i}j}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{pj}^{D_{pj}}}& \cdots & {\tilde{t}}_{C{p}_1{p}_1}^{11}/{\displaystyle {\sum }_{j=1}^p{\tilde{t}}_{pj}^{D_{pj}}}\end{array}\right]=\left[\begin{array}{ccccc}{\tilde{t}}_{11}^{{\alpha }_{11}}& \cdots & {\tilde{t}}_{i1}^{{\alpha }_{i1}}& \cdots & {\tilde{t}}_{p1}^{{\alpha }_{p1}}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{1j}^{{\alpha }_{1j}}& \cdots & {\tilde{t}}_{ij}^{{\alpha }_{ij}}& \cdots & {\tilde{t}}_{pj}^{{\alpha }_{pj}}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{1p}^{{\alpha }_{1p}}& \cdots & {\tilde{t}}_{ip}^{{\alpha }_{ip}}& \cdots & {\tilde{t}}_{pp}^{{\alpha }_{pp}}\end{array}\right]$$

(26)

Once the ${\tilde{T}}_D^{\alpha }$ matrix is derived, the weighted supermatrix ${\tilde{W}}^{\alpha }$ will be calculated, as indicated in Equation (27).

$${\tilde{W}}^{\alpha }={\tilde{T}}_D^{\mathsf{\alpha }}\tilde{W}=\left[\begin{array}{ccccc}{\tilde{t}}_{11}^{{\alpha }_{11}}{\tilde{W}}^{11}& \cdots & {\tilde{t}}_{i1}^{{\alpha }_{i1}}{\tilde{W}}^{i1}& \cdots & {\tilde{t}}_{p1}^{{\alpha }_{p1}}{\tilde{W}}^{p1}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{1j}^{{\alpha }_{1j}}{\tilde{W}}^{1j}& \cdots & {\tilde{t}}_{ij}^{{\alpha }_{ij}}{\tilde{W}}^{ij}& \cdots & {\tilde{t}}_{pj}^{{\alpha }_{pj}}{\tilde{W}}^{pj}\\ ⋮& & ⋮& & ⋮\\ {\tilde{t}}_{1p}^{{\alpha }_{1p}}{\tilde{W}}^{1p}& \cdots & {\tilde{t}}_{ip}^{{\alpha }_{ip}}{\tilde{W}}^{ip}& \cdots & {\tilde{t}}_{pp}^{{\alpha }_{pp}}{\tilde{W}}^{pp}\end{array}\right]$$

(27)

Step 7: Calculating the influential crisp weights with the limiting process approach. The weighted supermatrix will be raised to limiting powers until convergence. It means that such matrix will become a long-term stable supermatrix where the global influence weights can be thereby derived, such as $\underset{x\to \infty }{\lim }{\left({\tilde{W}}^{\alpha }\right)}^x$, ${\tilde{W}}^{\alpha }$ denotes limited supermatrix while $x$ stands for the any number. The final crisp weights can be derived through arithmetic average method, such as ${\tilde{W}}_{global}^{\alpha }=({\tilde{W}}_{(l)}^{\alpha }+{\tilde{W}}_{(u)}^{\alpha })/2$. The obtained influential weights from BR-DNP method will reveal the signals for policy makers in considering how to make better decisions. Subsequently, these will be utilized with modified Bayesian Rough VIKOR method for weighted gap analysis and eventually to make improving proposal of systemic innovation problems.

3.3. Modified Bayesian Rough VIKOR (MBR-VIKOR)

In general, conventional VIKOR aims to evaluate the alternatives in multivariate condition with a measure of proximity to the ideal solution. The concept of relative good for evaluating MCDM problems may not effectively reflect the real-world situation. Thus, Opricovic and Tzeng [44] further proposed the extended VIKOR for replacing traditional one by using the concept of aspiration level. In this research, in order to explore the systemic innovation problems under vague circumstance, the rough interval numbers and Bayesian theory will simultaneously be introduced. The VIKOR method will be modified accordingly. The modified VIKOR will be named as Modified Bayesian Rough VIKOR (MBR-VIKOR). Besides, the weights being derived by the BR-DNP will also be introduced into the MBR-VIKOR approach for evaluating the systemic innovation problems. The computing process is illustrated as follows [44,63,64,65]:

Step 8: Construct a rough decision matrix $V{D}_R^{\alpha }$ and define an aggregation function for the compromise ranking. The rough decision matrix $V{D}_R^{\alpha }$ is first established in by using the concept of Equations (1)–(5), where alternatives are denoted as ${\overline{Al}}_1^{},{\overline{Al}}_2^{},\dots ,{\overline{Al}}_k^{},\dots ,{\overline{Al}}_m^{}$ and $V{D}_R^{\alpha }=[V{D}_1^{\alpha },V{D}_2^{\alpha },\dots ,V{D}_n^{\alpha }]$ represents the rough decision matrices derived by each of experts’ opinions. The performance score of the jth criterion is denoted by ${\tilde{g}}_{jk(l)}^{\alpha }=\left[g_{jk(l)}^{\alpha },g_{jk(u)}^{\alpha }\right]$ for the $k_{th}$ alternative ${\overline{Al}}_k^{}$. Here, the $V{D}_R^{\alpha }$ comprises of the prior decision matrix $V{D}_R^{\alpha }$ and the conditional decision matrix $V{D}_R^{\alpha }$.

$$\begin{array}{l}\begin{array}{cccccccccccccc}& & & & & & & \overline{A{l}_1}& \begin{array}{ccc}& & \end{array}& \overline{A{l}_2}& & \cdots & \begin{array}{cc}& \end{array}& \overline{A{l}_m}\end{array}\\ V{D}_R^{\alpha }=\begin{array}{c}\begin{array}{c}\left[\begin{array}{cccc}\left[g_{11(l)}^{\alpha },g_{11(u)}^{\alpha }\right]& \left[g_{12(l)}^{\alpha },g_{12(u)}^{\alpha }\right]& \cdots & \left[g_{1m(l)}^{\alpha },g_{1m(u)}^{\alpha }\right]\\ \left[g_{21(l)}^{\alpha },g_{21(u)}^{\alpha }\right]& \left[g_{22(l)}^{\alpha },g_{22(u)}^{\alpha }\right]& \cdots & \left[g_{2m(l)}^{\alpha },g_{2m(u)}^{\alpha }\right]\\ ⋮& ⋮& \ddots & ⋮\\ \left[g_{n1(l)}^{\alpha },g_{n1(u)}^{\alpha }\right]& \left[g_{n2(l)}^{\alpha },g_{n2(u)}^{\alpha }\right]& \cdots & \left[g_{nm(l)}^{\alpha },g_{nm(u)}^{\alpha }\right]\end{array}\right]\\ \end{array}\end{array},R=1,2,\dots ,n\end{array}$$

(28)

After the obtain of the $V{D}_R^{\alpha }$, by utilizing Equation (29), the posterior matrix can be acquired, as depicted below:

$${\tilde{\mathsf{\Omega }}}_{}^{}=\tilde{\sigma }\times \frac{\left(P(V{D}_k^c)|P(V{D}_k^p)P(V{D}_k^p)\right)}{{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{i=1}^m{g}_{ij}^{c’}\times g_{ij}^{p’}}}},$$

(29)

where $P(V{D}_k^p)={\left[g_{ij(l)}^{p’},g_{ij(u)}^{p’}\right]}_{n\times m}$ and $P(V{D}_k^c)={\left[g_{ij(l)}^{c’},g_{ij(u)}^{c’}\right]}_{n\times m}$ stand for the probability matrices that are being normalized from the aggregation matrices, and $\tilde{\sigma }$ is a parameter gained by the sum of posterior probability matrices. $\tilde{\sigma }$ is utilized for converting the posterior probability matrices into the posterior total influence matrix. The posterior aggregation matrix of ${\tilde{\mathsf{\Omega }}}_{}^{}$ is shown below:

$${\tilde{\mathsf{\Omega }}}_{}^{}={\left[\overline{RN({\tilde{\rho }}_{ij}^{})}\right]}_{n\times m}=\left[\begin{array}{cccc}[{\rho }_{11(l)}^{},{\rho }_{11(u)}^{}]& [{\rho }_{12(l)}^{},{\rho }_{12(u)}^{}]& \cdots & [{\rho }_{1m(l)}^{},{\rho }_{1m(u)}^{}]\\ [{\rho }_{21(l)}^{},{\rho }_{21(u)}^{}]& [{\rho }_{22(l)}^{},{\rho }_{22(u)}^{}]& \cdots & [{\rho }_{2m(l)}^{},{\rho }_{2m(u)}^{}]\\ ⋮& ⋮& \ddots & ⋮\\ [{\rho }_{n1(l)}^{},{\rho }_{n1(u)}^{}]& [{\rho }_{n2(l)}^{},{\rho }_{n2(u)}^{}]& \cdots & [{\rho }_{nm(l)}^{},{\rho }_{nm(u)}^{}]\end{array}\right]$$

(30)

Then, an aggregation function, L_p metric, can be derived by Equation (31) according to Yu [66].

$$L_k^a={\left\{{\displaystyle \sum _{j=1}^n{\left[{\tilde{w}}_j\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_{kj}^{}\right|/\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_{kj}^{-}\right|\right]}^a}\right\}}^{1/a},\begin{array}{cc}1\le a\le \infty ;& k=1,2,\dots ,m\end{array}$$

(31)

Step 9: Calculate the ranking measures of ${\tilde{S}}_k^{}=\left[S_{k(l)}^{},S_{k(u)}^{}\right]$ and ${\tilde{Q}}_k^{}=\left[Q_{k(l)}^{},Q_{k(u)}^{}\right]$. Based on the L_p metric, the ranking measures of ${\tilde{S}}_k^{}$ and ${\tilde{Q}}_k^{}$ can further be derived as ${\tilde{L}}_k^{a=1}$ and ${\tilde{L}}_k^{a=\infty }$, respectively.

$${\tilde{S}}_k^{}={\tilde{L}}_k^{a=1}={\displaystyle \sum _{j=1}^n\left[{\tilde{w}}_j\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_{kj}^{}\right|/\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_j^{-}\right|\right]};$$

(32)

$${\tilde{Q}}_k^{}={\tilde{L}}_k^{\infty =1}=\underset{j=1,2,\dots ,n}{\max }\left({\displaystyle \sum _{j=1}^n\left[{\tilde{w}}_j\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_{kj}^{}\right|/\left|{\tilde{\rho }}_j^{*}-{\tilde{\rho }}_j^{-}\right|\right]}\right)$$

(33)

According to Equations (32) and (33), the best values is denoted by ${\tilde{\rho }}_j^{*}$, which means the aspiration level of the jth criterion. The worst value is denoted by ${\tilde{\rho }}_j^{-}$, while means the tolerable value of the jth criterion. Then, the compromise ranking measure, ${\tilde{R}}_k^{}=\left[R_{k(l)}^{},R_{k(u)}^{}\right]$, can be derived based on the ${\tilde{S}}_k^{}$, ${\tilde{Q}}_k^{}$, and the weighted group utility (i.e., weight = $v$) as well as the individual regret (i.e., weight = $1-v$) as follows:

$${\tilde{R}}_k^{}=v\times \frac{\left({\tilde{S}}_k^{}-{\tilde{S}}_{}^{*}\right)}{\left({\tilde{S}}_{}^{-}-{\tilde{S}}_{}^{*}\right)}+\left(1-v\right)\times \frac{\left({\tilde{Q}}_k^{}-{\tilde{Q}}_{}^{*}\right)}{\left({\tilde{Q}}_{}^{-}-{\tilde{Q}}_{}^{*}\right)}$$

(34)

where ${\tilde{S}}_{}^{*}={\min }_k{\tilde{S}}_k^{{}^{}}$, ${\tilde{S}}_k^{-}={\max }_k{\tilde{S}}_k^{{}^{}}$, ${\tilde{Q}}_k^{*}={\min }_k{\tilde{Q}}_k^{}$, and ${\tilde{Q}}_k^{-}={\max }_k{\tilde{Q}}_k^{}$.

Based on the concept of the aspiration level, here the best value will be set as 0 and the worst value will be set as 1. To easily analyze the performance-gap and rank the alternatives, the rough interval values of ${\tilde{R}}_k^{}$ matrix will be converted into crisp scores by using the arithmetic average method.

Through the analytic framework hybridizing the BR-DNP and the MBR-VIKOR, the compromise solution can be determined. In addition, the gap analysis of systemic innovation problems in vague environment can also be proposed. With the results of gap analysis, the improvement directions of systemic innovation problems will be clearly illustrated.

4. Empirical Study

In this section, an empirical case study based on exploring problems of systemic innovation for defining innovation policy for the manufacturing industry in Taiwan is proposed. The feasibility of the proposed framework for evaluating the performance of systemic innovation is demonstrated.

4.1. Background and Problem Description

Based on the perspective of systemic innovation, understanding potential problems in systemic innovation prior to the definition of innovation policy is indispensable. According to previous studies of the TIS models [4,13,21,51,67], proper definition of innovation policies can greatly enhance industrial competitiveness and therefore growth of a specific economy.

In some advanced countries, IoT development and sustainability have been focused by the definitions of supportive innovation policies. For instance, German and Finland launched a large project designed to facilitate smart factories by helping with industrial transformation, as well as establishing production and manufacturing platforms for SMEs. However, for emerging economies, such as China, Taiwan, and South Korea, the technology life cycle of IoT applications is still in the initial stages. To accelerate industrial development of IoT, national governments need to put more effort into promoting industrial IoT applications by defining innovation policy tools. In this sense, government sectors and policy makers should identify problems of systemic innovations that are blocking sustainable development of IoT. Meawhile, after identifying systemic innovation problems, how to resolve these problems should also be discussed. Therefore, this research proposes a performance evaluation framework that can serve as the basis for defining innovation policies by considering the interactions between systemic innovation problems and systemic functions in terms of the TIS theoretic model.

The analytical process can be divided into three stages. (1) Derivation of the systemic innovation problem: the derived systemic innovation problems focused on IoT-sustainability in the manufacturing industry of Taiwan will be proposed. Then, the appropriateness of these problems will be further evaluated based on the current situation of IoT development in the manufacturing industry of Taiwan. (2) Derivation of the structure and criterion weights of the decision-making problem: the BR-DNP approach will be introduced and used to derive the influential relationships between the systemic innovation problems. Then, the influential weight versus each criterion within the structure of the decision problem will be derived. (3) Finally, performance-gaps of systemic innovation problems derived by the proposed evaluation model will be analyzed and policy recommendations for improving systemic innovation problems can be offered to policy makers.

4.2. Data Collection

Experts’ opinions were collected by questionnaires in the period from February to October 2018. The questionnaire was designed based on feasible systemic innovation problems elicited from previous studies [4,7,8,10,12,13,26,52] and on the current status of Taiwan. To find the proper systemic innovation problems, each term of systemic innovation problems in the designed questionnaire can be suggested and thereby revised by experts.

First, the developed questionnaires concerning systemic innovation problems in terms of past literature were sent by email to 15 experts who have strong work experiences, including marketing or engineering management in IoT-related industries. In the meantime, in order to derive influential weights for later usage, the BR-DNP method was introduced. According to Section 3, the total influence matrix was introduced as the input to the BR-DNP. The influence weights can be derived accordingly. After identifying and determining the systemic innovation problems (SPs), the analytic process of the MBR-VIKOR was complete. In the final stage, experts were invited to assess the relations between the SPs and refined systemic innovation criteria that are being derived.

Table 2. The crucial systemic innovation problems (SPs).

Systemic Innovation Problems	Symbols
Lack of uniform technical standards	SP1
The innovation intensity is insufficient	SP2
Regulatory constrains for IoT development	SP3
Lack of sufficient infrastructure for IoT development	SP4
Low level of interdisciplinary collaboration	SP5
Lack of advanced sensor technology for IoT application	SP6
Lack of effective innovation application services	SP7
Low capability of system and platform integration	SP8
Lack of professionals	SP9
Low level of industrial upgrading for SMEs	SP10
Weak advocacy coalition	SP11

summarizes the eleven SPs that can influence the configuration of IoT innovation policy. The detailed analytical processes are presented in the following sections.

4.3. The Derivation of Systemic Innovation Problems by Modified Delphi Method

Based on results of the literature review [4,7,8,10,12,13,26,52], the potential systemic innovation problems are summarized in Table 2. The modified Delphi method (Section 3.1) was introduced to derive the applicable systemic innovation problems based on opinions provided by the 15 experts. Based on the modified Delphi approach, 75% was recognized as a minimum percentage of agreement for each alternative being evaluated.

Table 3. The evaluative results of SPs based on the modified Delphi method.

No.	Gender	Experiences	SP1	SP2	SP3	SP4	SP5	SP6	SP7	SP8	SP9	SP10	SP11
1	Male	15	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
2	Male	15	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
3	Male	20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
4	Male	20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No
5	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
6	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
7	Male	>20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No
8	Male	15	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
9	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
10	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
11	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
12	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
13	Male	15~20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
14	Male	>20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
15	Male	>20	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No
Agree (Yes)			15	15	15	15	15	15	15	15	15	15	12
Disagree (No)			0	0	0	0	0	0	0	0	0	0	3
Agree %			100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	80%
Disagree %			0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	20%

Note: Experiences means the years of work experiences.

shows the percentage of consensus of eleven systemic innovation problems being reached by experts. All alternative systemic innovation problems exceeding the minimum rate of 75% are appropriate for use in this study.

4.4. Derivations of the Influential Weights by the BR-DNP Method

For the evaluation and exploration of problems associated with the systemic innovation of IoT applications in smart manufacturing, the BR-DNP method based on aggregation of individual judgments can derive the influential weights versus each systemic criterion.

Based on the normalization, the scores of criteria associated with each systemic function within the ${\tilde{T}}_D^{\alpha }$ were converted into values between 0 and 1. The unweighted supermatrix $\tilde{W}=\left({\tilde{T}}_C^{\alpha }\right)$ can be derived by Equation (25). Then, the weighted supermatrix ${\tilde{W}}^{\alpha }$ can be derived by Equation (27). The final influential weights corresponding to each systemic function (

Table 4. Influential weights on systemic functions and criteria.

Functions	Local Weight	Ranking	Criteria	Local Weight	Ranking	Global Weight	Ranking
F1	0.172	1	e1	0.375	1	0.065	3
			e3	0.358	2	0.062	4
			e4	0.267	3	0.046	7
F2	0.168	2	k1	0.493	2	0.083	2
			k3	0.507	1	0.085	1
F3	0.143	4	kd1	0.364	1	0.052	5
			kd3	0.312	2	0.045	8
			kd4	0.324	1	0.046	6
F4	0.160	3	g1	0.259	1	0.042	9
			g2	0.259	2	0.041	10
			g3	0.238	4	0.038	14
			g4	0.243	3	0.039	11
F5	0.142	5	m1	0.240	3	0.034	16
			m2	0.271	1	0.038	13
			m3	0.230	4	0.033	18
			m4	0.259	2	0.037	15
F6	0.119	6	r1	0.227	3	0.027	22
			r3	0.326	1	0.039	12
			r4	0.200	4	0.024	23
			r5	0.247	2	0.029	21
F7	0.096	7	c1	0.327	2	0.031	19
			c2	0.355	1	0.034	17
			c3	0.318	3	0.030	20

) can be derived by raising the power of the weighted supermatrix to approach infinity. This method allows us to derive the local weights of the systemic criteria at their respective hierarchical levels as well as the global weights. These global weights can help determine the importance assigned to each individual criterion involved in the policy making process. Based on Table 4, the weights associated with each systemic innovation function as well as criteria are demonstrated. According to the analytical results, developing complementary technologies (k₃), conducting feasibility studies (k₁), experimenting with new applications of IoT (e₁), entry of firms to IoT markets (e₄), and training of professionals (kd₁), were ranked as criteria with the highest weights.

4.5. Assessing the Gap between the Current Status and the Aspired Level by the MBR-VIKOR

In order to analyze the performance-gaps in the systemic innovation problems for helping define the IoT innovation policy, the MBR-VIKOR based BR-DNP weights was introduced to treat such problems. Based on the experts’ opinions (refer

Table A1. Evaluation data for systemic innovation problems.

VD1
Functions/Criteria		SP1	SP2	…	SP9	SP10	SP11
F1	e1	(4;4)	(3;2)	…	(4;3)	(3;2)	(2;1)
	e3	(4;4)	(3;2)	…	(2;1)	(2;2)	(2;1)
	e4	(3;3)	(3;3)	…	(3;3)	(1;1)	(0;1)
F2	k1	(2;2)	(2;2)	…	(2;0)	(1;1)	(0;3)
	k3	(2;2)	(1;1)	…	(1;1)	(1;0)	(0;3)
F3	kd1	(0;0)	(0;0)	…	(2;2)	(3;3)	(0;4)
	kd3	(0;0)	(0;0)	…	(3;3)	(1;0)	(2;2)
	kd4	(0;0)	(1;1)	…	(3;3)	(1;0)	(2;2)
F4	g1	(0;3)	(0;0)	…	(3;3)	(1;0)	(0;2)
	g2	(3;1)	(2;2)	…	(4;4)	(1;1)	(2;2)
	g3	(1;1)	(2;2)	…	(3;3)	(1;1)	(0;2)
	g4	(1;0)	(3;3)	…	(2;2)	(4;1)	(0;0)
F5	m1	(0;0)	(0;0)	…	(2;2)	(1;1)	(0;3)
	m2	(0;0)	(1;1)	…	(2;2)	(1;1)	(2;2)
	m3	(0;3)	(1;1)	…	(3;3)	(2;1)	(0;0)
	m4	(3;0)	(2;2)	…	(3;3)	(2;1)	(4;0)
F6	r1	(0;2)	(2;2)	…	(0;1)	(0;1)	(0;3)
	r3	(2;0)	(3;3)	…	(3;3)	(1;1)	(3;2)
	r4	(0;0)	(0;0)	…	(3;3)	(4;1)	(0;2)
	r5	(0;0)	(0;0)	…	(3;3)	(2;2)	(0;2)
F7	c1	(2;2)	(2;2)	…	(2;2)	(1;1)	(2;3)
	c2	(1;1)	(1;1)	…	(2;2)	(2;2)	(2;2)
	c3	(0;0)	(0;0)	…	(3;3)	(2;2)	(2;1)
⁝
VD15
Functions/criteria		SP1	SP2	…	SP9	SP10	SP11
F1	e1	(3;4)	(3;4)	…	(3;4)	(2;4)	(2;4)
	e3	(3;4)	(3;4)	…	(3;4)	(2;4)	(2;4)
	e4	(4;4)	(4;4)	…	(4;4)	(4;4)	(2;2)
F2	k1	(4;4)	(4;4)	…	(4;4)	(4;4)	(2;2)
	k3	(4;4)	(4;4)	…	(4;4)	(4;4)	(2;2)
F3	kd1	(4;4)	(4;4)	…	(4;4)	(4;4)	(3;3)
	kd3	(4;4)	(4;4)	…	(2;2)	(4;4)	(2;2)
	kd4	(4;4)	(3;3)	…	(4;4)	(4;4)	(2;2)
F4	g1	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	g2	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	g3	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	g4	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
F5	m1	(3;3)	(4;4)	…	(3;3)	(3;3)	(3;3)
	m2	(4;4)	(1;1)	…	(1;1)	(3;3)	(3;3)
	m3	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	m4	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
F6	r1	(3;3)	(4;4)	…	(4;4)	(4;4)	(4;4)
	r3	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	r4	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	r5	(2;2)	(2;2)	…	(2;2)	(2;2)	(2;2)
F7	c1	(4;4)	(4;4)	…	(4;4)	(4;4)	(4;4)
	c2	(3;3)	(3;3)	…	(3;3)	(3;3)	(3;3)
	c3	(4;4)	(2;2)	…	(3;3)	(3;3)	(3;3)

in Appendix A), the evaluation data are further transformed into the rough decision matrix by using the Equations (1)–(5). The results are demonstrated in

Table A2. Rough decision matrix of systemic innovation problems, $V{D}_{}^p$.

Criteria	SP1	SP2	…	SP9	SP10	SP11
e1	[3.751, 3.982]	[2.863, 3.788]	…	[2.361, 3.681]	[2.207, 2.875]	[1.692, 2.727]
e3	[3.360, 3.840]	[2.704, 3.554]	…	[1.748, 2.119]	[2.403, 3.469]	[1.748, 2.119]
e4	[1.738, 3.169]	[3.360, 3.840]	…	[2.738, 3.124]	[1.903, 3.521]	[1.225, 2.038]
k1	[2.329, 3.721]	[2.947, 3.834]	…	[1.725, 2.881]	[1.439, 2.476]	[1.049, 2.520]
k3	[2.678, 3.707]	[2.329, 3.727]	…	[1.853, 3.347]	[1.814, 3.518]	[1.049, 2.520]
kd1	[1.091, 3.049]	[1.813, 3.656]	…	[2.569, 3.564]	[3.284, 3.782]	[0.893, 2.400]
⁝	⁝	⁝	⁝	⁝	⁝	⁝
r5	[0.394, 1.914]	[0.417, 2.393]	…	[2.538, 2.929]	[2.099, 1.249]	[0.934, 2.496]
c1	[2.349, 3.130]	[0.579, 1.862]	…	[2.065, 2.628]	[1.261, 3.840]	[1.964, 3.491]
c2	[2.042, 3.002]	[1.168, 2.726]	…	[2.071, 2.462]	[2.108, 0.000]	[1.879, 3.320]
c3	[0.693, 2.347]	[1.574, 2.881]	…	[3.000, 3.000]	[2.108, 3.674]	[1.791, 3.152]

and

Table A3. Rough decision matrix of systemic innovation problems, $V{D}_{}^c$.

Criteria	SP1	SP2	…	SP9	SP10	SP11
e1	[3.284, 3.782]	[3.125, 3.793]	…	[2.679, 3.786]	[2.274, 3.078]	[1.404, 2.803]
e3	[3.218, 3.716]	[3.007, 3.651]	…	[1.530, 2.379]	[2.250, 3.239]	[1.561, 2.757]
e4	[1.578, 2.947]	[3.284, 3.782]	…	[2.738, 3.124]	[1.370, 2.922]	[1.180, 2.282]
k1	[2.153, 3.692]	[3.125, 3.793]	…	[0.916, 2.494]	[1.214, 2.321]	[1.130, 2.927]
k3	[2.863, 3.778]	[2.966, 3.789]	…	[1.333, 2.667]	[0.885, 2.812]	[1.130, 2.927]
kd1	[1.651, 3.069]	[2.806, 3.784]	…	[2.222, 3.111]	[3.111, 3.556]	[1.315, 3.058]
⁝	⁝	⁝	⁝	⁝	⁝	⁝
r5	[0.394, 1.914]	[0.417, 2.393]	…	[2.538, 3.929]	[2.099, 2.859]	[1.136, 2.604]
c1	[2.349, 3.130]	[0.579, 1.862]	…	[2.065, 2.628]	[1.261, 2.609]	[2.020, 3.543]
c2	[2.042, 3.002]	[1.168, 2.726]	…	[2.071, 2.462]	[2.108, 2.858]	[1.870, 3.320]
c3	[0.693, 2.347]	[1.574, 2.881]	…	[3.000, 3.000]	[2.108, 2.858]	[1.680, 3.130]

(refer Appendix A). After that, Equations (29)–(35) were introduced to derive the performance-gaps. The priorities verse each function and criteria were ranked accordingly. The findings being derived from the hybrid Bayesian Rough framework can serve the basis of innovation policy reconfigurations by policy makers.

The empirical results of performance-gap (see

Table 5. Gap evaluation of systemic innovation problems by MBR-VIKOR approach.

Criteria	Local Weights	Global Weights	The Performance Gap of Systemic Innovation Problems
			SP1	SP2	SP3	SP4	SP5	SP6	SP7	SP8	SP9	SP10	SP11
F1	0.172	–	0.007	0.004	0.011	0.014	0.005	0.024	0.010	0.004	0.021	0.018	0.037
e1	0.375	0.065	0.001	0.005	0.017	0.014	0.003	0.014	0.003	0.006	0.011	0.022	0.040
e3	0.358	0.062	0.002	0.007	0.007	0.012	0.007	0.035	0.015	0.002	0.045	0.015	0.039
e4	0.267	0.046	0.017	0.001	0.010	0.015	0.004	0.022	0.013	0.005	0.007	0.015	0.032
F2	0.168	–	0.017	0.010	0.064	0.031	0.023	0.038	0.025	0.030	0.065	0.074	0.085
k1	0.493	0.083	0.021	0.006	0.050	0.026	0.031	0.050	0.026	0.037	0.072	0.085	0.080
k3	0.507	0.085	0.013	0.015	0.079	0.037	0.015	0.026	0.024	0.024	0.058	0.063	0.090
F3	0.143	–	0.040	0.010	0.044	0.043	0.009	0.076	0.018	0.018	0.005	0.024	0.033
kd1	0.364	0.052	0.025	0.008	0.012	0.008	0.003	0.077	0.009	0.002	0.010	0.002	0.033
kd3	0.312	0.045	0.048	0.014	0.057	0.060	0.015	0.075	0.026	0.027	0.003	0.044	0.032
kd4	0.324	0.046	0.048	0.007	0.062	0.060	0.011	0.076	0.019	0.024	0.002	0.026	0.034
F4	0.160	–	0.022	0.008	0.018	0.042	0.006	0.050	0.015	0.011	0.004	0.012	0.026
g1	0.259	0.042	0.044	0.007	0.019	0.046	0.005	0.056	0.013	0.017	0.004	0.019	0.025
g2	0.259	0.041	0.004	0.023	0.014	0.042	0.011	0.055	0.019	0.014	0.001	0.016	0.015
g3	0.238	0.038	0.020	0.002	0.035	0.039	0.003	0.042	0.016	0.012	0.004	0.013	0.023
g4	0.243	0.039	0.020	0.002	0.005	0.040	0.003	0.048	0.012	0.003	0.009	0.002	0.042
F5	0.142	–	0.015	0.011	0.010	0.023	0.004	0.021	0.009	0.010	0.010	0.009	0.014
m1	0.240	0.034	0.014	0.012	0.021	0.033	0.006	0.015	0.009	0.008	0.013	0.008	0.013
m2	0.271	0.038	0.038	0.019	0.012	0.025	0.003	0.030	0.014	0.011	0.018	0.011	0.019
m3	0.230	0.033	0.008	0.012	0.003	0.019	0.004	0.020	0.008	0.012	0.004	0.007	0.013
m4	0.259	0.037	0.001	0.002	0.003	0.014	0.002	0.018	0.007	0.009	0.005	0.009	0.011
F6	0.119	–	0.018	0.007	0.014	0.013	0.001	0.031	0.008	0.007	0.007	0.009	0.010
r1	0.227	0.027	0.019	0.001	0.007	0.006	0.001	0.020	0.008	0.005	0.019	0.012	0.009
r3	0.326	0.039	0.022	0.002	0.022	0.019	0.002	0.055	0.011	0.008	0.006	0.016	0.013
r4	0.200	0.024	0.012	0.010	0.010	0.005	0.001	0.018	0.005	0.004	0.001	0.001	0.007
r5	0.247	0.029	0.019	0.016	0.015	0.022	0.001	0.030	0.008	0.009	0.004	0.006	0.012
F7	0.096	–	0.006	0.013	0.010	0.016	0.012	0.037	0.016	0.010	0.007	0.008	0.005
c1	0.327	0.031	0.004	0.021	0.010	0.015	0.015	0.043	0.023	0.013	0.007	0.012	0.005
c2	0.355	0.034	0.007	0.014	0.017	0.024	0.017	0.044	0.020	0.009	0.010	0.008	0.007
c3	0.318	0.030	0.007	0.003	0.003	0.007	0.003	0.023	0.005	0.007	0.003	0.003	0.002
Gaps			0.231	0.116	0.284	0.324	0.099	0.486	0.170	0.151	0.193	0.250	0.343
Ranking			6	2	8	9	1	11	4	3	5	7	10

) analysis have pinpointed that which systemic criteria need to be improved preferentially. This practical and flexible instrument, on the one hand, enables policy makers to make decisions efficiently. On the other hand, this analytical tool can derive the priorities in terms of all the systemic innovation problems to concentrate on the improvements of pain points in order to enhance the firm competitiveness within the related IoT industries. More specifically, based on the analytic results of MBR-VIKOR, as show in Figure A1 and Figure A2, policy makers can realize the advantages and weaknesses in each of systemic problem and understand the gaps to the aspiration levels. Additionally, the findings can further be used as a reference for innovation policy formulation. Based on the obtained results, the proposed systemic evaluation framework being verified the effectiveness is appropriate for policy makers to make decision.

5. Discussion and Implications

Defining a set of innovation policies to enhance current technology development and increase industrial competitiveness is necessary. Therefore, policy makers should first define the main difficulties that influence technological innovations and industrial development. Once the challenging problems of systemic innovation have been confirmed, policy makers can define an innovation policy portfolio to cope with the identified problems. Thus, this work proposes an integrated framework for evaluating the systemic innovation problems. This framework hybridizes the BR-DNP and the MBR-VIKOR to evaluate systemic innovation problems. In this Section, gaps for each systemic innovation problems will be discussed at first. Then, the advances in methodologies and innovation policy analyses will be discussed.

5.1. Gaps for Each Systemic Innovation Problems

In order to confirm the influence of performance evaluation in systemic innovation problems, sensitivity analysis was performed by changing 10% of the parameters at a time. The results of sensitivity analysis are shown in

Table 6. Sensitivity analysis for R_k score and ranking.

SP	v = 0	Ranking	v = 0.1	Ranking	v = 0.2	Ranking	v = 0.3	Ranking	v = 0.4	Ranking	v = 0.5	Ranking	v = 0.6	Ranking	v = 0.7	Ranking	v = 0.8	Ranking	v = 0.9	Ranking	v = 1.0	Ranking
SP1	0.048	5	0.085	5	0.121	6	0.158	6	0.194	6	0.231	6	0.268	6	0.304	6	0.341	6	0.378	6	0.414	6
SP2	0.023	1	0.041	1	0.060	2	0.079	2	0.097	2	0.116	2	0.135	2	0.153	2	0.172	2	0.191	2	0.210	2
SP3	0.079	9	0.120	9	0.161	8	0.202	8	0.243	8	0.284	8	0.326	8	0.367	8	0.408	8	0.449	8	0.490	8
SP4	0.060	6	0.113	7	0.166	9	0.218	9	0.271	9	0.324	9	0.377	9	0.429	9	0.482	9	0.535	9	0.588	9
SP5	0.031	3	0.044	2	0.058	1	0.071	1	0.085	1	0.099	1	0.112	1	0.126	1	0.140	1	0.153	1	0.167	1
SP6	0.077	8	0.159	11	0.241	11	0.323	11	0.404	11	0.486	11	0.568	11	0.650	11	0.732	11	0.814	11	0.896	11
SP7	0.026	2	0.055	3	0.084	4	0.112	4	0.141	4	0.170	4	0.198	4	0.227	4	0.256	4	0.285	4	0.313	4
SP8	0.037	4	0.060	4	0.083	3	0.106	3	0.129	3	0.151	3	0.174	3	0.197	3	0.220	3	0.243	3	0.266	3
SP9	0.072	7	0.096	6	0.120	5	0.145	5	0.169	5	0.193	5	0.218	5	0.242	5	0.266	5	0.291	5	0.315	5
SP10	0.085	10	0.118	8	0.151	7	0.184	7	0.217	7	0.250	7	0.283	7	0.316	7	0.349	7	0.382	7	0.415	7
SP11	0.090	11	0.140	10	0.191	10	0.241	10	0.292	10	0.343	10	0.393	10	0.444	10	0.494	10	0.545	10	0.595	10

. It can be noticed that a low level of interdisciplinary collaboration (SP₅) possesses the lowest scores of R_k (Table 6) implying that it has the maximum priority in most situations. More specifically, when v = 0.0 and 0.1, the lowest R_k is the “the innovation intensity is insufficient” (SP₂) with the values of 0.023 and 0.041. The largest R_k scores are “weak advocacy coalition” (SP₁₁) and the “lack of advanced sensor technology for IoT application” (SP₆) with, scores of 0.090 and 0.159, respectively. Based on the overall performance-gap, the ranking result shows that SP₆ < SP₁₁ < SP₄ < SP₃ < SP₁₀ < SP₁ < SP₉ < SP₇ < SP₈ < SP₂ < SP₅. In addition to the result of ranking under v = 0.0 and 0.1, all ranking results demonstrate the robustness and stability of MBR-VIKOR. The visualization result from sensitivity analysis is presented in Figure A3 (refer Appendix B). By the above analyses, the policy makers can use this result as a basis for improving current potential systemic problems. In the following section, detailed discussions for gap evaluation in each systemic innovation problem and implications are further illustrated. In the following sub-sections, detailed discussions for gap evaluation in each systemic innovation problem and implications are further illustrated.

5.1.1. Lack of Uniform Technical Standards (SP₁)

Based on the results of performance-gap analysis (Table 5) of systemic innovation problems, for the first systemic problem, i.e., lack of uniform technical standards (SP₁), both entrepreneurial activities (F₁; gap ratio 0.007) and creation of legitimacy (F₇; gap ratio 0.006) are the most advantageous systemic functions. In contrast, knowledge diffusion through networks (F₃; gap ratio 0.040), guidance of the search (F₄; gap ratio 0.022), and resource mobilization (F₆; gap ratio 0.018) are the most disadvantageous systemic functions. From the aspect of systemic criteria, experimenting new applications of IoT (e₁; gap ratio 0.001), entry of firms to IoT markets (e₃; gap ratio 0.002), design of favorable rules and regulations (g₂; gap ratio 0.004), and standardizations (m₄; gap ratio 0.001) have the best performance; in contrast, the worst criteria consist of organizing conference/workshops/seminars/meetings (kd₃; gap ratio 0.048), demonstrations and exhibitions (kd₄; gap ratio 0.048), setting collective goals for IoT development (g₁; gap ratio 0.044) and government procurement programs (m₂; gap ratio 0.038). In light of this result, policy makers should focus on improving the most disadvantageous systemic functions for formulating technical standards.

5.1.2. Insufficient Innovation Intensity (SP₂)

From the aspect of insufficient innovation intensity (SP₂), entrepreneurial activities (F₁; gap ratio 0.004) and resource mobilization (F₆; gap ratio 0.007) are the most advantageous functions. Conversely, knowledge development (F₂; gap ratio 0.010), market formation (F₅; gap ratio 0.011), and creation of legitimacy (F₇; gap ratio 0.013) are the most disadvantageous systemic functions. From the aspect of systemic criteria, systems for innovation and incubation (e₄; gap ratio 0.001), providing directions of development (g₄; gap ratio 0.002), and providing R&D budgets (r₁; gap ratio 0.001) are the most advantageous criteria. On the other hand, the most disadvantageous systemic criteria include design of favorable rules and regulations (g₂; gap ratio 0.023), government procurement programs (m₂; gap ratio 0.020), and strength of lobby actions (c₁; gap ratio 0.021). These criteria need to be improved from the aspect of insufficient innovation intensity (SP₂). As such, in developing IoT applications and technologies, the government should relax regulations for IoT products and should subsidize an IoT program. Interest groups such as industrial organizations should emphasize IoT-related development by interacting with and maintaining relationships with public sectors and proposing useful suggestions.

5.1.3. Regulatory Constrains for IoT Development (SP₃)

From the aspect of regulatory constrains for IoT development (SP₃), n market formation (F₅; gap ratio 0.010) and creation of legitimacy (F₇; gap ratio 0.010) are the most advantageous systemic functions. Knowledge development (F₂; gap ratio 0.064) and knowledge diffusion through networks (F₃; gap ratio 0.044) are the most advantageous systemic function that should be improved. From the aspect of systemic criteria, the most advantageous group includes regulatory reform (m₃; gap ratio 0.003), standardizations (m₄; gap ratio 0.003), and social acceptability (c₃; gap ratio 0.003). Conversely, the most disadvantageous groups are conducting feasibility studies (k₁; gap ratio 0.050), developing complementary technologies (k₃; gap ratio 0.079), organizing conference/workshops/seminars/meetings (kd₃; gap ratio 0.057), and demonstrations and exhibitions (kd₄; gap ratio 0.062). The promotion of IoT applications can be useful for resolving the third systemic problem (SP₃), because the measure can appeal and encourage additional IoT development. Supporting the development of complementary technologies and organization of IoT related activities with other firms and research institutes will also be helpful for relaxing regulations. Additionally, open laboratories for testing and experiments of IoT applications will help resolve regulatory constraints related to the third systemic problem (SP₃).

5.1.4. Insufficient Infrastructure for IoT Development (SP₄)

For the results of performance-gap analysis related to SP₄, resource mobilization (F₆; gap ratio 0.013) is more advantageous. However, knowledge development (F₂; gap ratio 0.031), knowledge diffusion through networks (F₃; gap ratio 0.043), and guidance of the search (F₄; gap ratio 0.042) are the most disadvantageous functions that need to be enhanced. From the aspect of systemic criteria, setting collective goals for IoT development (g₁; gap ratio 0.031), design of favorable rules and regulations (g₂; gap ratio 0.031), publicizing expectations (g₃; gap ratio 0.039), and providing directions of development (g₄; gap ratio 0.040) are prioritized as the first group of criteria to be enhanced. In order to solve the systemic innovation problem regarding insufficient infrastructure for IoT development (SP₄), policy makers should consider the most disadvantageous systemic functions and criteria. For example, since open laboratories for testing of IoT applications cannot fulfill the needs of firms, the government should help establish such laboratories. Furthermore, the government should encourage smart manufacturing firms to join the cooperation projects.

5.1.5. Insufficient Interdisciplinary Collaboration (SP₅)

For the problem of insufficient interdisciplinary collaboration (SP₅), entrepreneurial activities (F₁; gap ratio 0.005), market formation (F₅; gap ratio 0.001), and resource mobilization (F₆; gap ratio 0.001) functions contained the least number of performance gaps. In contrast, knowledge development (F₂; gap ratio 0.023) and creation of legitimacy (F₇; gap ratio 0.012) are the most disadvantageous functions and need to be enhanced. From the perspective of systemic criteria, R&D budget provisions (r₁; gap ratio 0.001), launches IoT related education programs (r₃; gap ratio 0.002), mobilization of human resources (r₄; gap ratio 0.001), and governmental grants for IoT projects (r₅; gap ratio 0.001) are the most advantageous. In contrast, the conduction of feasibility studies (k₁; gap ratio 0.031) and the rise and growth of interest groups (c₂; gap ratio 0.017) are the most disadvantageous criteria. To strengthen interdisciplinary collaboration for IoT development, conducting feasibility studies will be necessary. Typical strategies include motivating research institutes to develop new technology applications with universities and firms. For industries, interest groups can be formed based on strategies that include discussion forums on social network sites that are related to IoT development.

5.1.6. Lack of Advanced Sensor Technology for IoT Applications (SP₆)

For the performance gap analysis results for the advanced sensor technology problem, functions of entrepreneurial activities (F₁; gap ratio 0.024) and market formation (F₅; gap ratio 0.021) as well as the criterion of experimenting new applications of IoT (e₁; gap ratio 0.014) are more advantageous. In contrast, the knowledge diffusion through networks function (F₃; gap ratio 0.076), the criteria for organizing conference/workshops/seminars/meetings (kd₃; gap ratio 0.075), training of professionals (kd₁; gap ratio 0.077), as well as demonstrations and exhibitions (kd₄; gap ratio 0.076) are most disadvantageous. In order to resolve the shortage of less-developed sensor technologies for IoT applications, policy makers should promote the knowledge diffusion through networks function. More specifically, policy makers should hold more conferences/workshops of IoT research on sensors as well as associated applications. Additionally, the government should help train experts in sensor design and development. Further, the government can help with construction of demonstration and exhibition areas for validating sensor innovations.

5.1.7. Shortage of Effective Services for Innovation Applications (SP₇)

For the shortage of effective services for innovation applications, the market formation (F₅; gap ratio 0.009) and resource mobilization (F₆; gap ratio 0.008) functions as well as the experimenting new applications of IoT (e₁; gap ratio 0.003) criteria outperformed the others. In contrast, the most disadvantageous function is knowledge development (F₂; gap ratio 0.025), which must be improved. Within the F₂ function, two criteria, conducting feasibility studies (k₁; gap ratio 0.026), developing complementary technologies (k₃; gap ratio 0.024), and organizing conference/workshops/seminars/meetings (kd₃; gap ratio 0.026) should be emphasized and further addressed. In enhancing the services for innovation applications of IoT, supporting the promotion of applicable studies and complementary technology development are prioritized. Based on this result, policy makers should formulate certain helpful innovation policies such as support of financial funds and various incentives to solve this systemic problem.

5.1.8. Insufficient Capabilities of System and Platform Integration (SP₈)

For insufficient capabilities of system and platform integration, entrepreneurial activities (F₁; gap ratio 0.004) and resource mobilization (F₆; gap ratio 0.007) are more efficient than the others for this problem of systemic innovation. In contrast, knowledge development (F₂; gap ratio 0.030) performed poorly and therefore should be enhanced immediately. Further, criteria including conducting feasibility studies (k₁; gap ratio 0.037) and developing complementary technologies (k₃; gap ratio 0.024) are the most disadvantageous ones. Therefore, improving these two features, including k₁ and k₃, will benefit the system and platform integration for IoTs. For example, although investment in IoT development is gradually increasing during the previous years, over fifty-percent of IoT programs are still in the proof of concept (POC) stage. An effective way to solve this problem is to establish a center for system and platform integration that enable various IoT devices to connect to this center. Using this platform, on the one hand, users can choose and design the most suitable services based on their needs. On the other hand, based on the platform, various manufacturing products can be managed and communicated. To realize this idea, complementary technology integration will be indispensable. Moreover, such system and platform integration toward various feasibility studies will be essential.

5.1.9. Lack of Professionals (SP₉)

For the shortage problem of professionals (SP₉), guidance of the search (F₄; gap ratio 0.004) is the most advantageous function while knowledge development (F₂; gap ratio 0.065) is the most disadvantageous function. From the aspect of criteria, mobilization of human resources (r₄; gap ratio 0.001) and design of favorable rules and regulations (g₂; gap ratio 0.001) are the most advantageous features while the conduction of feasibility studies (k₁; gap ratio 0.072) and developing complementary technologies (k₃; gap ratio 0.058) are the most disadvantageous features. Based on these analytic results, government sectors should aggressively support improvements of these criteria. Thus, global talent can be attracted to Taiwan for realization of IoT applications as well as development of new technologies. To date, professionals with related experience in IoT innovation cannot fulfill the demand of the IoT market. To accelerate the development of IoT talent, training of professionals and recruitment of talent are very critical. Therefore, policy makers should formulate related innovation polices for helping to improve the problem of lack of talented people.

5.1.10. Low Level of Industrial Upgrading for SMEs (SP₁₀)

In the systemic innovation problem associated with industrial upgrading of SMEs, resource mobilization (F₆; gap ratio 0.009) and creation of legitimacy (F₇; gap ratio 0.008) are the most advantageous functions while knowledge development (F₂; gap ratio 0.074) is the most disadvantageous function. From the aspect of criteria, mobilization of human resources (r₄; gap ratio 0.001), training of professionals (kd₁; gap ratio 0.002), and social acceptability (c₃; gap ratio 0.003) belong to the most advantageous group, while conducting feasibility studies (k₁; gap ratio 0.085) and developing complementary technologies (k₃; gap ratio 0.063) are classified into the advantageous group. In order to solve the problem of low levels of industrial upgrading of SMEs, the Taiwanese government should prioritize efforts toward promoting knowledge development. For instance, the high costs of industrial upgrading of IoT are unaffordable to SMEs, the government should subsidize these firms. In addition, the government should encourage feasibility studies of IoT applications for smart manufacturing. Further, the government can motivate the development of complementary technology that can facilitate the integration of technology and applications. Thus, industrial upgrading of SMEs can be realized and accelerated.

5.1.11. Weak Advocacy Coalition (SP₁₁)

In the systemic problem of weak advocacy coalition, creation of legitimacy (F₇; gap ratio 0.005) and social acceptability (c₃; gap ratio 0.002) are the most advantageous functions. In contrast, knowledge development (F₂; gap ratio 0.085), conducting feasibility studies (k₁; gap ratio 0.080), and developing complementary technologies (k₃; gap ratio 0.090) are the most disadvantageous criteria. Thus, the government should concentrate on enhancing the performance of feasibility studies as well as developing complementary technologies. In order to enhance the weak advocacy coalition, promotion of various feasibility analyses for IoT applications and complementary technologies will facilitate the development of IoT. This measure will also motivate more interest groups. Further, mutual collaborations between industries can be established and developed.

5.2. The Implications in Industrial Sustainabillity for IoT Industries

Based on the analytic results, this research offers several insights regarding industrial sustainability in manufacturing industries. First, in order to strengthen interdisciplinary collaboration for IoT industrial sustainability, programs of feasibility analysis (k₁) will be essential. The feasibility analyses include technological experiments in IoT applications, resource integration from different sectors, and IoT market analysis and investigation. Besides, mutual collaborations between industries (SP₅) can also enhance industrial sustainability.

Second, enhancing the innovation intensity can strengthen industrial ecosystems and thus, enhance the industrial sustainability for IoT industries. As such, the government sectors should relax regulations (g₂) for IoT products and subsidize IoT programs (m₁). The advocacy of coalitions (SP₁₁) is also an important policy that can facilitate the industrial sustainability. The interest groups (c₂) can generate significant influence on the formulation of regulations and launch of supportive policies. Interest groups such as industrial organizations should emphasize IoT-related development by interacting with and maintaining relationships with public sectors and proposing useful suggestions.

Finally, a lack of talented people (SP₉) is always a factor influencing industrial sustainability. Based on these analytic results, government sectors should aggressively provide various incentives to appeal more professionals. To date, professionals with related experience in IoT innovations cannot fulfill the demand of the IoT market. To accelerate the development of IoT talent, trainings of professionals and recruiting of talents are very critical. Therefore, policy makers should formulate related innovation polices for helping to improve the problem of lack of talented people.

5.3. Advances in Methodologies and Innovation Policy Analyses

In this research, the authors leveraged the hybrid model by incorporating modified Delphi method, rough interval number, Bayesian theory, DNP and modified VIKOR to explore and analyze the systemic innovation problems hindering industrial sustainability of IoT. With the practical case of manufacturing industries in Taiwan, the proposed methodologies and analytic processes are validated the feasibility. The several advantageous and contributions of proposed model and performance evaluation in systemic innovation problems are presented as below.

First of all, conventional systemic innovation policy analyses are based on content analysis in specific policy related documents and experts’ interview. By doing so, potential systemic problems can be defined and, innovation policies can thus be derived for dealing with such problems. However, this research pays more attentions in the evaluation of systemic innovation problems that are hampering IoT industrial sustainability in Taiwan. More specifically, in order to evaluate these systemic innovation problems, several research steps are adopted. First step is to collect feasible problems influencing industrial sustainability from the previous literature and experts’ interview. In the second step, these systemic problems are confirmed in terms of experts’ opinions. The final step is to evaluate the gap-performances among these identified systemic problems. The gap-performance analysis can show how to improve current systemic problems for achieving aspiring level. In other words, this gap-performance analysis for systemic innovation problems can offer policy makers the basis to determine feasible policies to solve current systemic problems.

Second, most of past studies focusing on innovation policy analysis and industrial sustainability adopted qualitative methods as the foundation for research problems. Despite of popular, traditional qualitative methods could be subjective and misleading [2]. Instead, the proposed model based quantitative method used in this research can overcome this problem. Moreover, the comprehensive frameworks based on quantitative methods in performance evaluation of systemic innovation problems are pretty scarce. Hence, this research proposes an analytic framework that can be seen as the one of options for policy makers and analysts to explore the systemic problems and exploit the reasons for understanding how to effectively improve these systemic problems using systemic steps.

Third, past literature would always use fuzzy theory, grey theory, and rough interval number to solve inaccurate human judgment views. In recent, Bayesian theory is also used in treating human judgments [2]. The primary superiority of Bayesian theory is that it simultaneously takes into account the prior possibility and conditional possibility of an event for deriving the eventual result. Decision systems can also be based on deep and machine learning models [68]. Given this reason, the proposed methods here adopted Bayesian theory and rough interval number to deal with bias and imprecision resulted from human judgments instead. In this paper, combination of Bayesian theory and rough interval number is validated the feasibility in performance evaluation. Consequently, the MCDM models by hybridizing Bayesian theory and rough interval number will be effective in decision making problems.

5.4. Limitations and Future Research Possibilities

The advantage of the proposed framework is its practical applicability under conditions of limited quantitative information; however, this research also has some limitations. First, this systemic evaluation model that is composed of systemic features and innovation problems will change based on external conditions, such as different industrial environments. Hence, such a model should be adjusted depending on other scenarios in future research. This research adopted the extensively used theoretic model—namely, the systemic innovation policy model—as an underpinning basis for the study. In the future, certain advanced theories and aspects should be considered and incorporated into the analytical framework as an extended empirical model. Second, although the rough interval numbers can effectively treat the subjective collective judgments, objective information and data should also be simultaneously taken into consideration, as this measure may be a better way to eliminate subjective judgments as much as possible. Third, the proposed methodology achieved success with empirical validation. Nevertheless, it would be feasible to apply other approaches such as MAIRCA and fuzzy integral and compare their results. Therefore, future research can use the methods from this research as a reference for developing more robust techniques and can generalize it to other academic and practical areas.

6. Conclusions

This research integrated the BR-DNP and modified Bayesian rough VIKOR methods to propose a Bayesian rough framework for exploring and analyzing the systemic innovation problems. Based upon the above analysis, two major conclusions can be drawn:

First, the BR-DNP analysis demonstrated that the entrepreneurial activities (F₁; weight 0.172) function had the largest influential weight, followed by knowledge development (F₂; weight 0.168). These results showed that these two systemic functions are relatively more important than other functions. In the gap analysis of the MBR-VIKOR approach, the ranking priority, which has demonstrated robustness under different scenarios, showed that SP₆ < SP₁₁ < SP₄ < SP₃ < SP₁₀ < SP₁ < SP₉ < SP₇ < SP₈ < SP₂ < SP₅. These results reveal that the gap in the systemic problem of “low level of interdisciplinary collaboration” (SP₅) to the aspiration level is less than other systemic problems. In contrast, systemic problems, including resource mobilization (F₆), weak advocacy coalition (SP₁₁), guidance of the search (F₄), and regulatory constrains for IoT development (SP₃), had relatively poor performance.

Second, through the gap analysis, the improving directions, belonging to each systemic problem, can also be identified in terms of systemic features. This study defined an effective model for exploring systemic innovation problems. In many complex interdependence research issues, human evaluation always involves various uncertainties that will lead to imprecise outcomes. Although the proposed model relies heavily on experts’ judgment, the combination of Bayesian theory and rough interval values provide a solution that can address these subjective judgments without assumptions. In conclusion, in this work, the proposed hybrid systemic evaluation model not only efficiently addresses practical issues, but also can serve as a basis for formulation of future innovation policy and can be generalized to other fields.