Evaluating a Business Ecosystem of Open Data Services Using the Fuzzy DEMATEL-AHP Approach

Abstract

The government has formulated and implemented an open data policy to promote administrative transparency and economic development in recent years. Therefore, most previous studies on open data have focused on e-government. Consequently, an open data service industry committed to providing innovative value-added data application services has emerged in Taiwan, with small- and medium-sized enterprises being the driving force. However, in a complex industrial environment, enterprises need to promote efficient data services development by developing a cross-disciplinary business ecosystem cooperatively. Nevertheless, few studies have discussed the open data service industry from the perspective of business ecosystems, making it impossible for enterprises to evaluate the business ecosystem of open data services built by them. In this study, we used the fuzzy analytic hierarchy process and fuzzy decision-making and trial evaluation laboratory methods to construct five evaluation dimensions and thirty-one evaluation criteria. We organized them into an evaluation scale to measure the business ecosystem’s performance of open data services built by enterprises. Then, using the case analysis method, we tested the applicability of the evaluation scale. This study examined the assessment scale of ecosystem construction in the open data industry, from the perspective of a business ecosystem, and analyzed the importance of each key criterion.

1. Introduction

Many countries worldwide have implemented open data strategies to promote administrative transparency and citizen participation and encourage enterprises to develop innovative value-added open data applications, thus fostering innovative industry and socio-economic development [1]. Taiwan has been implementing an open data policy for several years and ranked first globally twice (in 2015 and 2017), according to the Global Open Data Index created by the Open Knowledge Foundation, thus promoting the development of the data-centered data service industry in the country.

Open data refers to the digitalized and structured data derived from raw governmental data and made available to both the public and private sectors through online government platforms. As governments play an essential role in releasing open data, its applicability should be considered in policy-making and implementation [2]. The application of open data enables governments and enterprises to improve the value they add and create new business patterns or markets in diverse fields through public–private cooperation [3]. According to a survey of the Industrial Technology Research Institute, the output of Taiwan’s data services industry reached NTD 93.4 billion in 2019, with a market size of NTD 120.9 billion, and is estimated to exceed NTD 126.2 billion by 2022.

The “business ecosystem” concept was introduced by Moore in 1993 [4]. A business ecosystem is a value network developed by a group that can never be operated by any enterprise alone [5]. In a business ecosystem, enterprises should enhance synergy through co-specialization while complementarity among enterprises creates different value for resources, products, and services [6].

Kandiah and Gossain [7] discussed the critical role of the Internet in today’s information economy. Power and Jerjian [8] defined a business ecosystem as a worldwide network system interacting with real and virtual worlds. Peltoniemi and Vuori [9] argued that a business ecosystem must be able to emerge on its own, self-organize, co-evolve, and adapt to an industrial environment. There are both cooperative and competitive relationships in a business ecosystem, with the system composition changing in the direction of more adaptability and new insights constantly emerging. Iansiti and Levien [10] defined four types of players in a business ecosystem (i.e., keystones, physical dominators, hub landlords, and niche players), depending on whether they possess the core technology or dominate the operational order, to discuss their strategy choice in the system environment.

Dawes et al. [11] stated that the combination of open and external data can generate various innovative applications. However, existing international open data assessment frameworks do not incorporate user perspectives. Therefore, it is not easy to specify the relationship and benefits between open data providers, users, and other stakeholders.

Immonen et al. [12] reported that the integration of open and private data, followed by analytical processing, can generate information different from that of competitors and provide extra added value. However, current companies face some challenges in using data because of a lack of data processing technology and high costs. Thus, it is impossible to provide customized services to the customers effectively. In addition, sizeable international information companies are in a leading position, but there is a small demand for open data in Taiwan. This has led to a lack of clarity in the professional division of labor in the value chain industry. Therefore, enterprises must clarify the supply and demand relationship between various stakeholders to build a precise data flow ecosystem.

Although previous studies have laid a solid foundation for industry and academia, the following gaps and application limitations remain:

• Certain evaluation models do not consider the indices related to service demand.
  In Taiwan’s data service industry, a specialized division of labor is not yet apparent, and international brands integrated with Taiwanese systems play a dominant role. Thus, new data service providers have emerged to satisfy specific data application needs. However, most new data services focus on customized data analysis and application rather than “data market”, “data processing”, “tool development”, and “data application consulting”, implying a gap in the industrial value chain.
• Few studies have discussed the operations of a business ecosystem.
  Both governments and enterprises need an evaluation framework to evaluate themselves and further examine the operations and status of the business ecosystems of open data. However, most existing open data evaluation models focus on data supply and quality, making it difficult to evaluate the operations of business ecosystems constantly [13]. Existing studies have not yet comprehensively evaluated the open data service industry from a business ecosystem perspective. Nevertheless, it is imperative to evaluate the health of a business ecosystem of open data services under the existing corporate operation mode.
• Models for open data have to address the decision-making problem in an uncertain environment.

In real-world situations, most qualitative evaluations involve ambiguity and uncertainty during the decision-making process because of unknown and ambiguous information. Moreover, ambiguity and subjective judgments can significantly influence the decision-making process in an uncertain environment. Therefore, this study provides an effective evaluation tool (soft calculation method) that can be combined with the judgments of experts or decision-makers to offer managerial insights applicable in real-world situations.

This study introduced the concept of linguistic variables from fuzzy logic as one of the methodological tools used for research. The main reason for this is that the uncertainty of expert judgment and preference makes it difficult to measure precise values, which is an inherent ambiguity of human languages. In recent years, scholars in different fields have applied the fuzzy decision-making and trial evaluation laboratory (Fuzzy DEMATEL) method in different research areas, such as consumer behavior [14], the green supply chain [15], and tourism [16]. The open data industry is a complex, multi-faceted, and uncertain concept, as shown by the data format disorder and lack of data science personnel. Citizens, businesses, and the government need to work together to overcome these difficulties and develop feasible solutions. However, the rapid changes in the current market trends make the traditional industrial value chain unable to cope with the variability and complexity of the industry. Therefore, this study incorporates the semantic variables of Fuzzy DEMATEL, which will enable expert preferences and judgments to be measured more accurately.

As open data services can produce innovative service patterns and economic value, it is necessary to evaluate the effectiveness of such services. However, most existing studies evaluate open data platforms regarding data quality, accessibility, granularity, security, and application interfaces. Existing open data evaluation frameworks only adopt the perspective of data supply, neglecting the perspective of data users, making it difficult to evaluate a business ecosystem of open data services comprehensively. This study presents a systematic multiple-criteria decision-making model to overcome the above problem.

Therefore, the main contributions of this study are as follows:

• We discussed and developed an evaluation framework for the open data service industry from the perspective of business ecosystems.
• We developed 5 evaluation dimensions (i.e., data governance, productivity, robustness, niche creation, and co-creation) and 31 evaluation criteria.
• Based on the evaluation framework, we developed an evaluation scale for the business ecosystems of open data services that can evaluate the performance of open data service providers and assess the overall effectiveness and maturity of the business ecosystem.

Information technology allows industries to involve an increased number of stakeholders [13]. As a result, much time is spent on data collection, which may lead to a gap in information and cause inaccuracies in the integration processes. In addition, many companies use historical data as the basis for service design and planning. Hence, the immediacy of information in service delivery is being lost owing to time and location constraints [6]. Using open data, industries can not only solve the shortcomings mentioned above but also optimize services and improve business models with value-added applications.

Specifically, we constructed the evaluation scale preliminarily based on a literature review and pre-test expert interviews. Afterward, we analyzed the evaluation dimensions and criteria weights using the fuzzy analytic hierarchy process (FAHP) method. We determined the causality between different evaluation dimensions and criteria using the fuzzy DEMATEL method. Finally, we tested the evaluation scale through case analysis. In summary, Taiwan’s open data industry is bound to encounter many difficulties in building an ecosystem. Therefore, this study examined the assessment scale of ecosystem construction in the open data industry, from the perspective of a business ecosystem, and analyzed the importance of each key criterion. The purpose was to provide a reference for building an ecosystem for the open data industry in Taiwan for managerial implications and subsequent research.

This study is organized into six sections. The introduction is presented in the first section. The second section introduces the pen data service industry and evaluation criteria for the business ecosystem. The methodology of this study, FAHP and fuzzy DEMATEL methods, is described in Section 3. Section 4 illustrates the procedure described in Section 3 using a real case. The Fuzzy DEMATEL analysis results and related discussions are presented in Section 5. Finally, the conclusion of this research is presented in Section 6.

2. Literature Review

2.1. Open Data Service Industry

Open knowledge gained from open data can create enormous economic value for the open data service industry [17]. Additionally, open data can solve social problems, increase economic value, and stimulate more enterprises to develop novel business models and innovative services. Based on business needs, enterprises in the open data service industry can be classified into suppliers, aggregators, developers, enrichers, and enablers [18].

As innovative services are rooted in the needs of people, governments, as the providers of open data, must consider the needs of users and create metadata to coordinate the supply and demand of open data [19]. Open data involves a variety of communities and stakeholders. The flow of data is bidirectional rather than unidirectional. The feedback loop between data supply and demand should be organized into a business ecosystem to promote the use of open data [20]. Hence, a business ecosystem of open data services can be evaluated on four criteria: data cyclicity; data sustainability [21]; demand-oriented supply; and correlation between data suppliers, intermediaries, and users [1].

Based on the existing data governance patterns, this study investigated nine significant factors affecting the development of the open data service industry: data cyclicity, data supply sustainability, demand matching, mutual correlation of value delivery, creation of open data websites, data integration, data quality management, link to descriptive data, and governmental policies.

2.2. Evaluation Criteria for the Business Ecosystem

The health of a business ecosystem of the Internet of Things can be evaluated in terms of seven indices under three dimensions (i.e., robustness, productivity, and diversity) [22]. Members of a business ecosystem must pay attention to the connectivity of stakeholders (i.e., community) in the business ecosystem, improve existing technological platforms through communication and collaboration, and attract and retain external developers [23].

2.2.1. Productivity

Productivity can be defined as the effectiveness in transforming innovations into cost advantages, new products, and new functions [6]. Specifically, business ecosystem members, with their collaborative and competitive relationships and the multifaceted development of new technologies to stimulate innovation, can not only increase the competitiveness of the overall business ecosystem but also effectively improve productivity [24].

2.2.2. Robustness

In an increasingly complex and fast-changing industrial environment, enterprises’ research and development input, operational strategy, risk prediction, and partner diversity affect their viability in a business ecosystem. They must constantly make endeavors to build a solid and robust business ecosystem [25]. To build a sound business ecosystem, it is necessary to emphasize the planning mechanism and value consensus among system members, resource allocation capacity, and performance and consider its adaptability and responsiveness to external factors [24]. Additionally, member viability, structural sustainability, predictability, limited elimination, and continuity are five key factors that can improve the robustness of a business ecosystem. The cornerstones for maintaining the innovation of a business ecosystem include knowledge transfer, innovation specificity, and network stability [26]. A business ecosystem must have four capabilities—sustainability, adaptability, innovation, and renewal [27]—and identify and reduce various risks through crowdsourcing [28].

2.2.3. Niche Creation

Enterprise leaders in a business ecosystem must demonstrate visionary leadership, actively participate in its activities, and interact with other members to discover innovative ideas [29]. Innovation is driven by the integration, exchange, and application of resources by consolidating knowledge, skills, experience, and provision of services [30]. Product or service diversity in a business ecosystem depends on openness and knowledge exchange among stakeholders. The synergy in a business ecosystem is a mechanism that facilitates the acquisition of tangible or intangible resources by stakeholders [31].

2.2.4. Co-Creation

Based on synergy and shared goals, the members of a business ecosystem can learn from each other to promote the sharing, integration, and innovation of resources, thus benefiting from the results of co-creation [32]. In a complex, loose, and fiercely competitive ecosystem, enterprises must have consistent motivations and goals to accurately understand the relationships between partners and their roles [28]. Moreover, a new-type business ecosystem should tap the potential of its members through co-creation, which is measured by five factors: resource richness, knowledge spillover, direct externality, indirect externality, and resource derivation [33]. In addition, an open information platform can provide stakeholders with diverse and more flexible ways of resource utilization [6].

Moreover, the application of new technologies and the cooperation and co-creation of members can improve productivity and competitiveness [26]. Furthermore, risks in a business ecosystem can be reduced through crowdsourcing to maximize its robustness [24]. Thus, sustainability, adaptability, innovativeness, and stability are key to a healthy business ecosystem. In particular, innovation is driven by integrating, exchanging, and applying resources across multiple members [20].

Based on a literature review, this study constructed a preliminary evaluation framework for a business ecosystem of open data services, which comprises 33 criteria categorized under 5 dimensions (as described in

Table 1. Preliminary evaluation framework for the business ecosystem of open data services.

Dimension	Criteria	Literatures
Data governance	Data cyclicity	[6,34]
	Data supply sustainability	[23,24]
	Demand matching	[6,23]
	Mutual correlation of value delivery	[6,34]
	Creation of open data websites	[6,23]
	Data integration	[6,34]
	Data quality management	[6,23,24]
	Links to explanatory data	[6,23,24]
	Governmental policy	[23,24]
Productivity	Factor productivity	[23,24]
	Ability to deliver innovation	[23,24]
	Financial productivity	[6,34]
	Value productivity	[6,23,24]
	Simplicity of the interactions among system members	[6,34]
Robustness	Member viability	[24,25]
	Sustainability	[26,27,28]
	Predictability	[24,25]
	Limited elimination	[26,27,28]
	Network stability	[24,28]
Niche creation	Visionary leadership	[29,30,31]
	Brainstorming	[30,31]
	Enterprise participation willingness	[29,30]
	Product or service diversity	[29,31]
	Intangible resources	[30,31]
	Knowledge and experience	[29,30,31]
	Partnership with a third party	[29,30,31]
Co-creation	Resource richness	[20,28]
	Knowledge spillover	[24,26,28]
	Direct externality	[26,28,33]
	Indirect externality	[24,26,28]
	Resource derivation	[24,32,33]
	Platform openness	[20,24,26]
	Goal-value congruence	[26,33]

).

3. Methods

Figure 1 shows the procedures of this study. An evaluation scale for a business ecosystem of open data services was developed using the FAHP and fuzzy DEMATEL methods. The applicability of the evaluation scale was tested through case analysis.

FAHP can simplify complex evaluation criteria and determine their weights through a clear hierarchical structure. However, it only assumes a direct relationship between different dimensions or criteria (i.e., dimensions or criteria are all conditionally independent), ignoring the possible indirect relationship between them. This study adopted fuzzy DEMATEL to offset this deficiency and determine the causality and degree of correlation between different dimensions or criteria. However, fuzzy DEMATEL lacks a consistency-check step in the calculation process. It cannot determine whether the deviation is within an acceptable range. Therefore, we combined FAHP with fuzzy DEMATEL to determine the evaluation criteria for a business ecosystem of open data services.

3.1. FAHP

Saaty [35] introduced AHP to analyze the decisions regarding complex and multiple evaluation criteria under uncertainty and ensure consistency in chaotic decision-making processes. AHP can determine each criterion’s weight and execution priority by creating a pairwise comparison matrix and comparing each criterion with its lower-layer criteria sequentially (specifically, it compares and identifies the degree of influence between different layers) [13]. However, traditional AHP suffers from fuzziness. Therefore, Buckley [36] proposed the FAHP method, which combines AHP with the fuzzy theory.

Subsequently, FAHP has been employed in numerous studies. Using AHP combined with the fuzzy theory, Chen, Wu, Chen, and Huang [13] argued that the maintenance personnel of factories could develop an optimal maintenance strategy for each component of specific equipment. Using fuzzy Delphi combined with FAHP, Chauhan, et al. [37] examined technology-driven enablers of supply chain responsiveness in an uncertain and complex business environment.

3.2. Fuzzy DEMATEL

The fuzzy DEMATEL method has been applied in many fields [38]. For example, Mahmoudi et al. [39] used it to help enterprises identify the right supply chain partners to improve organizational performance, and identified key influencing factors. To define the criteria for technological human resources, Kazancoglu and Ozkan-Ozen [40] used the FAHP method to analyze the weight of each criterion. They used fuzzy DEMATEL to analyze the interrelationships between different criteria.

The execution procedures of FAHP combined with fuzzy DEMATEL (shown in Figure 2) are as follows.

Step 1: Form a project team and establish and amend criteria.

Identify related criteria through a literature review, form an expert team, adopt the experts’ industrial practice experience and suggestions, and adjust the dimensions and criteria accordingly, to amend the formal questionnaire for the study.

Step 2: Create a fuzzy linguistic scale for FAHP factors.

Create the pairwise comparison matrix A and use each layer as a benchmark to calculate the weight values of its lower-layer evaluation criteria (including $A_1,A_2,\cdots ,A_n$) (note: The relative importance between different criteria at each layer can be expressed as $a_{ij}\left(i,j=1,2,\cdots ,n\right)$). Then, the results of relative comparisons among evaluation criteria are placed at the top right of the main diagonal of matrix A, and their reciprocal value are placed at the bottom left of the main diagonal. The main diagonal is self-compared ($i=j$). Hence, the criterion values are all set to 1, and the pairwise comparison matrix is expressed as:

$$\mathrm{A}=\left[a_{ij}\right]=\left[\begin{array}{cccc}1& a_{12}& \cdots & a_{1n}\\ a_{21}& 1& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & 1\end{array}\right],i,j=1,2\dots ,\mathrm{n},$$

(1)

where $a_{ij}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{ji}$}\right.=\left[\begin{array}{cccc}1& a_{12}& \cdots & a_{1n}\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{12}$}\right.& 1& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{1n}$}\right.& \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{2n}$}\right.& \cdots & 1\end{array}\right]$.

Fuzziness exists when humans make decisions. By defining fuzzy linguistic variables, experts can make pairwise comparisons and score different criteria, thus reflecting the implied meaning of linguistic variables and measuring the degrees of influence among the criteria.

Table 2. FAHP-based fuzzy linguistic scale.

Fuzzy Number Evaluation Scale	Linguistic Variable	Triangular Fuzzy Number (l,m,u)
9	Absolute importance	(8,9,10)
8	Between absolute importance and demonstrated importance	(7,8,9)
7	Demonstrated importance	(6,7,8)
6	Between demonstrated importance and essential importance	(5,6,7)
5	Essential importance	(4,5,6)
4	Between essential importance and weak importance	(3,4,5)
3	Weak importance	(2,3,4)
2	Between weak importance and equal importance	(1,2,3)
1	Equal importance	(1,1,1)

describes the triangular fuzzy number of the FAHP-based linguistic variable scale [13].

Step 3: Calculate the eigenvector and maximum eigenvalue of the matrix.

After a pairwise comparison matrix is created, use the eigenvalue solving method to calculate the eigenvector $w_i$ or priority vector. As most matrices are inconsistent matrices, to ensure high calculation accuracy of the eigenvector, the eigenvector is calculated using the standardization method of the row vector mean (as expressed in Equation (2)). Based on the eigenvector, calculate the maximized eigenvalue ${\lambda }_{max}$, as expressed in Equation (3):

$$w_i=\frac{1}{n}{\displaystyle \sum }_{j=1}^n\frac{a_{ij}}{{{\displaystyle \sum }}_{i=1}^n{a}_{ij}}$$

(2)

$${\lambda }_{max}={\displaystyle \sum }_1^n\frac{{\left(AW\right)}_i}{n{W}_i}$$

(3)

Step 4: Conduct a consistency test.

Values in a pairwise comparison matrix are scores given by experts based on their subjective judgments. However, experts may make inconsistent judgments about criteria when the number of criteria or layers is large. Hence, it is necessary to conduct a consistency test to check whether the errors of expert scores are within a reasonable range, that is, the consistency of weights are tested in terms of the consistency index (C.I) and consistency ratio (C.R) in this study.

• C.I: When $C.I=0$, experts’ early and subsequent judgments are completely consistent. When $C.I>0.1$, experts’ early and subsequent judgments are completely inconsistent. When $C.I\le 0.1$, experts’ judgment errors are within an acceptable range [41]. C.I is expressed as follows:

  $$C.I=\frac{{\lambda }_{max}-n}{n-1}.$$

  (4)
• C.R: The C.R value (i.e., n value) also varies with the order [41]. The random index (R.I) values in

  Table 3. Random indices.

  Order	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
  R.I	0.00	0.00	0.58	0.90	1.12	1.24	1.32	1.41	1.45	1.49	1.51	1.48	1.56	1.57	1.58

  show that C.R values can be used to judge the consistency of a matrix with the same n value (as expressed in Equation (5)). When C.R ≤ 0.1, the consistency reaches an acceptable level. Otherwise, it is necessary to re-examine the correlations among different layers or criteria:

  $$C.R=\frac{C.I}{R.I}$$

  (5)

Step 5: Calculate the overall triangular fuzzy number of each layer.

To determine the relative fuzzy weight of each evaluation criterion, it is necessary to construct triangular fuzzy numbers and calculate their values using the minimum ($L_i$), median ($M_i$), and maximum values ($R_i$) of each criterion in the questionnaire [42]. Specifically, h denotes h experts, i denotes i evaluation criteria, and n denotes the total number of experts. The three values are calculated using Equations (6)–(8), respectively:

$$L_i=mi{n}_h\left\{L_i^h,h=1,2,\cdots ,n\right\}$$

(6)

$$M_i={[{{\displaystyle \prod }}_{h=1}^m\left\{M_i^h,h=1,2,\cdots ,n\right\}]}^{\frac{1}{n}}$$

(7)

$$R_i=ma{x}_h=\left\{R_i^h,h=1,2,\cdots ,n\right\}$$

(8)

Step 6: Normalize the triangular fuzzy number of each layer.

To increase the accuracy of the calculation results, the triangular fuzzy numbers determined in Step 5 need to be normalized (as expressed in Equations (9)–(11)). $n{L}_i, n{M}_i, \mathrm{and} n{R}_i$ denote the normalized triangular fuzzy numbers, as represented in the following equations:

$$n{L}_i=\frac{L_j}{{\left\{\left[{{\displaystyle \sum }}_i^k{R}_i\right]\ast \left[{{\displaystyle \sum }}_i^k{L}_i\right]\right\}}^{0.5}}$$

(9)

$$n{M}_i=\frac{M_i}{{{\displaystyle \sum }}_i^k{M}_i}$$

(10)

$$n{R}_i=\frac{R_i}{{\left\{\left[{{\displaystyle \sum }}_i^k{R}_i\right]\ast \left[{{\displaystyle \sum }}_i^k{L}_i\right]\right\}}^{0.5}}$$

(11)

Step 7: Defuzzification and normalization.

Triangular fuzzy numbers are not specific numerical values. Normalized triangular fuzzy numbers need to be defuzzified to facilitate the subsequent ranking of weights. The center of area (COA) method can be used for defuzzification regardless of expert preferences, thus determining the best non-fuzzy performance (BNP) value [43]. Each element is converted into a specific weight (expressed in Equation (12)). To ensure that the sum of defuzzified weights ($BN{P}_i$) of all layers is equal to 1, it is necessary to perform normalization again to determine the final weights ($N{W}_i$) of all dimensions and criteria (as expressed in Equation (13)):

$$BN{P}_i=\frac{\left\{\left(n{R}_i-n{L}_i\right)+\left(n{M}_i-n{L}_i\right)\right\}}{3}+n{L}_i,\forall i$$

(12)

$$N{W}_i=\frac{BN{P}_i}{{{\displaystyle \sum }}_{i=1}^{k}BN{P}_i}$$

(13)

Step 8: Interrelate and rank the weights of dimensions and criteria across different layers.

After the final weights of criteria at different layers are determined in Step 7, it is necessary to interrelate the weights of different dimensions and criteria across different layers, thereby calculating the relative total weights of criteria selected by experts (expressed in Equation (14)). $N{W}_j$ denotes the weight of the j-th criterion at the third layer below the first layer (target layer). $N{W}_i$ denotes the weight of the i-th dimension at the second layer below the first layer. $N{W}_{ij}$ denotes the weight of the j-th sub-criterion at the third layer at the second layer below the i-th criterion:

$$N{W}_j=N{W}_i\ast N{W}_{ij}$$

(14)

Through the above calculation across different layers, we can determine the absolute weight of each sub-criterion in the hierarchical evaluation framework and further rank the selected criteria in order of importance.

Step 9: Create a fuzzy linguistic scale for fuzzy DEMATEL factors.

Lin and Wu [44] incorporated the fuzzy theory with the DEMATEL method and created fuzzy linguistic variables, enabling experts to determine and score the causality and importance between different criteria in a fuzzy and uncertain environment.

Table 4. Fuzzy DEMATEL-based fuzzy linguistic scale.

Evaluation Scale	Linguistic Variable	Fuzzy Number (l,m,u)
0	No influence	(0,0,0.25)
1	Low degree of influence	(0,0.25,0.5)
2	Moderate degree of influence	(0.25,0.5,0.75)
3	High degree of influence	(0.5,0.75,1)
4	Extremely high degree of influence	(0.75,1,1)

describes the triangular fuzzy numbers associated with the fuzzy DEMATEL-based linguistic variables.

Step 10: Create a direction relation matrix.

Based on experts’ opinions from the questionnaire survey, pairwise comparison is performed according to the relationships and degrees of influence among criteria, to generate an $n\times n$ direct-relation matrix ${\tilde{Z}}_{ij}^k$. ${\tilde{Z}}_{ij}^k=\left(L_{ij}^k,M_{ij}^k,U_{ij}^k\right)$ is used to evaluate the triangular fuzzy number, indicating the opinion of the expert k regarding the degree of the influence of criterion i on criterion j. The H experts’ fuzzy direct-relation matrix can be calculated as ${\tilde{Z}}^k=\left[\begin{array}{cccc}0& {\tilde{Z}}_{12}^k& \cdots & {\tilde{Z}}_{1n}^k\\ {\tilde{Z}}_{21}^k& 0& \cdots & {\tilde{Z}}_{2n}^k\\ ⋮& ⋮& \ddots & ⋮\\ {\tilde{Z}}_{n1}^k& {\tilde{Z}}_{n2}^k& \cdots & 0\end{array}\right]$, k = 1, 2…., H. An arithmetic mean can be used to calculate the H experts’ triangular fuzzy number, to estimate each criterion $a_{ij}$:

$${\tilde{a}}_{ij}=\left(L_{ij},M_{ij},U_{ij}\right)=\frac{1}{H}{\displaystyle \sum }_{k=1}^H{\tilde{Z}}_{ij}^k=\frac{1}{H}{\displaystyle \sum }_{k=1}^H\left(L_{ij}^k,M_{ij}^k,U_{ij}^k\right)$$

(15)

Once the experts’ evaluation scores are converted into fuzzy numbers, they need to be defuzzified to facilitate the subsequent calculation. In this study, defuzzification is also performed using the COA method, and the BNP value is calculated to determine the specific weight of each criterion as:

$$BN{P}_i=\frac{\left\{\left(U_i-L_i\right)+\left(M_i-L_i\right)\right\}}{3}+L_i,\forall i.$$

(16)

Then, the fuzzy matrix of expert consensus is integrated, that is, the geometric means of triangular fuzzy numbers determined by the experts are calculated, and the BNP value using the COA method is computed to create the direct-relation matrix Z.

Step 11: Convert the direct-relation matrix into a normalized direct-relation matrix.

A direct-relation matrix can be normalized using two methods: (1) using the maximum column vector sum as the normalization benchmark; or (2) using the maximum column or row vector sum as the normalization benchmark [44]. This study adopted the second method (expressed in Equations (17) and (18)). Specifically, the direct-relation matrix Z was divided by the $\mathsf{\lambda }$ value to generate the normalized direct-relation matrix X:

$$\mathsf{\lambda }=\max \left(\underset{1\le i\le n}{\max }{\displaystyle \sum }_{j=1}^n{z}_{ij}, \underset{1\le j\le n}{\max }{\displaystyle \sum }_{i=1}^n{z}_{ij}\right)$$

(17)

$$\mathrm{X}=\frac{z}{\lambda }$$

(18)

Step 12: Create a total-relation matrix.

The dimensions and criteria involved in the decision-making process are not merely influenced by a single factor. They are directly or indirectly influenced by themselves and other criteria, and the influence decreases with the increase in the number of times. When the k-th power of the normalized direct-relation matrix X is infinite, the matrix reaches a steady state (i.e., the value of influence is zero), thus generating the total-relation matrix T, in which I is an $\mathrm{n}\times \mathrm{n}$ unit matrix, which is expressed as:

$$\mathrm{T}=\underset{k\to \infty }{\lim }\left(X+X^2+\cdots +X^k\right)=X{\left(I-X\right)}^{-1}, (\underset{k\to \infty }{\lim }{X}^k={\left[0\right]}_{n\ast n})$$

(19)

Step 13: Draw a causality diagram.

Assume that $t_{ij}$ is the criterion feature in the total-relation matrix T. Summate each column and each row in the total-relation matrix T. $D_i$ denotes the sum of the i-th column, and $R_j$ denotes the sum of the j-th row (expressed in Equations (20) and (21), respectively). The $D_i$ and $R_j$ values both represent the direct and indirect relationship:

$$D_i={{\displaystyle \sum }}_{j=1}^n{t}_{ij},i=1,2,\dots ,n$$

(20)

$$R_j={\displaystyle \sum }_{j=1}^n{t}_{ij},j=1,2,\dots ,n$$

(21)

Figure 2. The flowchart of the suggested algorithm.

Figure 2. The flowchart of the suggested algorithm.

4. Illustration of a Real Case

This section illustrates the procedure described in Section 3 using a real case.

4.1. Problem Description

In this study, we tested the applicability of the proposed evaluation scale for a business ecosystem of open data services using a real case. Company T was used as an example to illustrate the applicability and analysis results of the evaluation scale. Founded in 2007, Company T has evolved from a graphics company to an intelligent transportation system information services provider. With its advantage in software development and expertise in transportation, Company T has committed to developing value-added transportation applications based on open governmental data combined with private data since 2013. Consequently, Company T has transformed itself into a transportation data service provider and data market operator by re-processing, standardizing, and analyzing transportation data per transportation data standards.

We designed a pre-test questionnaire using the evaluation dimensions and criteria mentioned above to make the evaluation scale more suitable for industrial needs and reduce the gap between academic research and practice. Then, we interviewed three experts in related fields to clarify and revise the related definitions. Accordingly, we constructed 5 evaluation dimensions (i.e., data governance, productivity, robustness, niche creation, and co-creation) and 31 evaluation criteria under (

Table 5. Description of the evaluation criteria for a business ecosystem of open data services.

Dimension	Criteria	Description
Productivity	Factor productivity	It reflects the ability of business ecosystem members to transform production factors into products (e.g., the sale of data or tools and growth of business revenue).
	Adoption of new technology	It indicates whether new technologies can be quickly and effectively adopted by business ecosystem members, and stimulate and produce innovations.
	Financial productivity	It is defined as the overall financial capacity of a business ecosystem.
	Value productivity	The comprehensive efficiency from the complementarity, competition, or cooperation between members can produce other meaningful values.
Robustness	Member viability	It reflects whether the responsiveness of business ecosystem members and the operating mode can make them adapt to and survive rapid environmental changes.
	Structural sustainability	It indicates whether the overall structure of a business ecosystem, which is built on the relationship between different types of members, can effectively respond to internal and external environmental changes.
	Predictability	It reflects the ability to collect information on future trends in different fields to predict or control future environmental changes and, accordingly, formulate appropriate operation strategies.
	Timely elimination	Enterprises not able to adapt to environmental change should be timely eliminated from the business ecosystem on a small scale.
	Network stability	The stability of a business ecosystem can be strengthened by organizing the diverse relationships between its members into a rigid structure.
Niche creation	Visionary leadership	Visionary leadership can improve the innovation process by determining and developing common goals.
	Brainstorming	Ideas or expertise from stakeholders in various fields can be adopted to improve the research and development capabilities.
	Enterprise participation willingness	Active participation of enterprises serves to enhance competitiveness and innovation.
	Product or service diversity	Product or service updates serve to improve the innovation level.
	Intangible resources	Based on intangible resources (e.g., brands, image, and culture), business ecosystem members can learn from and communicate with each other to accelerate innovations.
	Knowledge and experience	The exchange of information (e.g., knowledge, experience, data, and ideas) serves to strengthen the update of information and innovations.
	Partnership with third parties	The business ecosystem can provide various third-party organizations with new technologies to increase the diversity of outputs.
Co-creation	Resource richness	Business ecosystem members can improve the quality and availability of resources through the exchange and feedback of information.
	Knowledge spillover	New knowledge generated by resource integration can be disseminated through various channels, thus improving the quality of products or services.
	Direct externality	The utility of products or services depends on the quantity of users.
	Indirect externality	Complementary products or after-sale services can affect customer benefits.
	Resource value-added derivation	Enterprises can add value to, optimize, or innovate resources through third-party guidance or assistance.
	Platform openness	Platform openness serves to improve the visibility of information and efficiency of information access.
	Goal-value congruence	Enterprises need to set common goals and pursue common values during value activities.

).

4.2. Calculating Weights by FAHP

Based on the evaluation framework established by experts’ interviews, nine industrial experts were invited to fill in the pre-test questionnaire. These industrial experts were company leaders or senior managers, who possessed professional knowledge of open data and business ecosystems. The FAHP method determined the weight values and assigned the weights of evaluation dimensions and criteria on the evaluation scale. As described in

Table 6. Results of the evaluation dimensions and criteria for the business ecosystem of open data services.

Dimension	Normalized Weight	Rank	Evaluation Criteria	Normalized Weight	Rank of Intra-Group Weights	Weight across Different Layers	Overall Rank
Data governance	0.19236	4	Data cyclicity	0.11286	4	0.0217	24
			Data applicability	0.09684	7	0.0186	30
			Demand matching	0.11173	5	0.0215	25
			Mutual correlation of value delivery	0.09872	6	0.0190	29
			Creation of open data websites	0.09243	8	0.0178	31
			Data quality	0.19348	1	0.0372	8
			Link to explanatory data	0.13684	3	0.0263	17
			Governmental policy	0.15709	2	0.0302	16
Productivity	0.22437	1	Factor productivity	0.23931	2	0.0537	3
			Adoption of new technology	0.22187	3	0.0498	5
			Financial productivity	0.20443	4	0.0459	6
			Value productivity	0.33439	1	0.0750	1
Robustness	0.22065	2	Member viability	0.15376	4	0.0339	10
			Structural sustainability	0.24148	2	0.0533	4
			Predictability	0.28355	1	0.0626	2
			Timely elimination	0.14979	5	0.0331	11
			Network stability	0.17142	3	0.0378	7
Niche creation	0.19430	3	Visionary leadership	0.16016	2	0.0311	13
			Brainstorming	0.16186	1	0.0315	12
			Enterprise participation willingness	0.15955	3	0.0310	14
			Product or service diversity	0.15642	4	0.0304	15
			Intangible resources	0.12067	6	0.0234	20
			Knowledge and experience	0.12673	5	0.0246	19
			Partnership with third parties	0.11461	7	0.0223	23
Co-creation	0.16831	5	Resource richness	0.13476	4	0.0227	22
			Knowledge spillover	0.13766	3	0.0232	21
			Direct externality	0.20384	1	0.0343	9
			Indirect externality	0.15593	2	0.0262	18
			Resource value-added derivation	0.12496	5	0.0210	26
			Platform openness	0.12349	6	0.0208	27
			Goal-value congruence	0.11937	7	0.0201	28

, the enterprises have reached a very high degree of consensus on the dimension of “productivity”, indicating that they attach great importance to the advantage of transforming open data innovation into profits through synergy. Specifically, value productivity indicates that synergy can help develop innovative services by promoting data flow. However, open data is mainly used to optimize the internal process and not as a profitable product. In the business ecosystem of open data services, financial productivity is scarcely highlighted.

The open data services industry is yet to mature. Specifically, enterprises are committed to developing innovative business models with open data. They have not yet acquired sufficient capacity or experience to mobilize the resources of more parties, failing to achieve “co-creation”. Moreover, as the open data stakeholders seek different benefits, the overall business ecosystem lacks a goal-value congruence. However, open data enterprises have a strong affirmation of “direct externality”, namely, open data products and services developed through two-way feedback with users depend on user satisfaction and loyalty.

Under the existing data governance model, enterprises agree that “data quality” can improve the benefits on the application side (e.g., direct access, ease of understanding, and ease of processing). In addition, the variety and quantity of datasets in the existing open data platform are already sufficient. Therefore, the “creation of open data websites” is not a priority for enterprises. Instead, enterprises prefer to improve data quality continuously through feedback from all stakeholders.

Regarding the dimension of robustness, “predictability” has led to a high degree of consensus on the trends and risk management in the open data services industry. However, most enterprises in this industry are small and medium-sized, and business model innovation remains at an exploratory stage, thus lacking sufficient experience in selecting key partners. Therefore, “timely elimination” is an immature skill in the current business ecosystem of open data services.

Moreover, the enterprises agree that “brainstorming” can help enterprises develop innovative services that meet diverse user needs by combining existing niches with open data. However, the current open data services market focuses its operations on two-way feedback with end-users. Therefore, “partnership with third parties”, as a catalyst to stimulate or optimize innovative services, is currently not a concern for the open data service industry.

4.3. Results of the Case Analysis Using Fuzzy DEMATEL

This subsection describes how to use the evaluation scale, which can help enterprises check whether they have achieved their goals in each evaluation item for the business ecosystem by comparing their current business performance with their previous business performance. The weighted score of each criterion can be determined by multiplying the FAHP assigned weight value and the enterprises’ evaluation score. Then, the weighted score of each dimension can be determined by summating the weighted scores of all criteria under the dimension. Finally, the weighted scores of the five dimensions are summated to determine the evaluation score of the maturity of the business ecosystem of open data services. Hence, enterprises can analyze the degree of influence between criteria according to the fuzzy DEMATEL-based causality diagram to improve their performance.

The maturity evaluation score of the business ecosystem of open data services built by Company T was 7.083 (

Table 7. Evaluation results of Company T’s business ecosystem of open data services.

Dimension	Evaluation Criteria	Weight across Different Layers	Score	Rank	Score	Criterion Weight	Rank
Data governance	Data cyclicity	0.022	6	7	0.130	1.445	3
	Data applicability	0.019	8	5	0.149
	Demand matching	0.021	8	4	0.172
	Mutual correlation of value delivery	0.019	5	8	0.095
	Creation of open data websites	0.018	8	6	0.142
	Data quality	0.037	9	1	0.335
	Link to explanatory data	0.026	8	3	0.211
	Governmental policy	0.030	7	2	0.212
Productivity	Factor productivity	0.054	6	3	0.322	1.350	4
	Adoption of new technology	0.050	7	2	0.348
	Financial productivity	0.046	5	4	0.229
	Value productivity	0.075	6	1	0.450
Robustness	Member viability	0.034	8	3	0.271	1.459	2
	Structural sustainability	0.053	6	2	0.320
	Predictability	0.063	7	1	0.438
	Timely elimination	0.033	5	5	0.165
	Network stability	0.038	7	4	0.265
Niche creation	Visionary leadership	0.031	8	3	0.249	1.532	1
	Brainstorming	0.031	8	2	0.252
	Enterprise participation willingness	0.031	9	1	0.279
	Product or service diversity	0.030	7	4	0.213
	Intangible resources	0.023	7	7	0.164
	Knowledge and experience	0.025	8	5	0.197
	Partnership with third parties	0.022	8	6	0.178
Co-creation	Resource richness	0.023	8	4	0.181	1.297	5
	Knowledge spillover	0.023	7	5	0.162
	Direct externality	0.034	8	1	0.274
	Indirect externality	0.026	7	3	0.184
	Resource value-added derivation	0.021	7	7	0.147
	Platform openness	0.021	9	2	0.187
	Goal-value congruence	0.020	8	6	0.161

). The business ecosystem can be classified as one with “high maturity”, as defined in this study. It integrates Taiwanese and international transportation graphic data and the data needs of automobile manufacturers through data exchange markets, and helps transportation data service providers standardize data. Through platform services, the business ecosystem promotes the circulation of data and cross-organization and cross-discipline collaboration in the transportation field, thus increasing the applicability of open data combined with private data.

This study evaluated Company T’s business ecosystem of open data services using the fuzzy DEMATEL-based causality diagram. The analysis results of all dimensions in Table 7 show that Company T has a comparative advantage in the dimension of “niche creation”. Based on the diversity and collaboration of business ecosystem members, it actively transforms the existing niche advantage combined with open data into new innovative services. However, the fuzzy DEMATEL analysis shows that “co-creation” is a significant core dimension. Company T is relatively disadvantaged in this dimension, mainly because public works dominate the transportation sector, international automobile manufacturers, and international graphics companies (e.g., Google and HERE). However, as Company T is relatively small-sized and inexperienced in resource integration in this dimension, niche creation should be prioritized for improvement.

Regarding the dimension of data governance, “data quality” is highly important for Company T because the quality of open data affects the extent and depth of various value-added applications. Company T provides integrated data services and does not optimize open data through other stakeholders. Thus, it ignores the development of “mutual correlation of value delivery”. According to the fuzzy DEMATEL analysis, the overall performance of data governance needs to be enhanced by improving other criteria, and data governance should be prioritized for improvement.

Regarding the dimension of productivity, Company T focuses on value productivity, namely, strengthening the mastery of new technologies through complementary, competitive, or division of labor. As open data are considered a development tool, the “financial productivity” performance in the business ecosystem is yet to be manifested. However, according to the fuzzy DEMATEL analysis, it is necessary to improve the financial and value productivity performance.

Regarding the dimension of robustness, Company T performs well in terms of “predictability” (i.e., effectively formulating operational strategies suited to the development trends of the open data market) but lacks the ability of “timely elimination”. However, the fuzzy DEMATEL analysis shows that “predictability” and “timely elimination” are sufficient to affect the stability of the overall robustness and member viability in the business ecosystem. Hence, both criteria need to be improved.

Regarding the dimension of niche creation, Company T has a very high degree of participation in open data-related services. “Enterprise participation willingness” can effectively help enterprises improve competitiveness and innovation levels. The fuzzy DEMATEL analysis shows that the “intangible resource” significantly affects niche creation. However, Company T merely develops open data services without establishing its independent brand. Thus, its development strategy should establish a brand image in the open data service industry.

Regarding the dimension of co-creation, Company T places great importance on “direct externality” but ignores “value-added resource derivation”. As consultancy services are scarce in Taiwan, Company T fails to acquire effective third-party services. However, value-added resource derivation relies on the improved direct externality to provide more knowledge about open data services to existing third-party players. Therefore, to develop innovative value-added products and services, Company T needs to gain deep insights into the needs of business ecosystem members.

5. Results and Discussion

The proposed evaluation framework will improve the past business ecosystems used for evaluating companies, which were constructed based on qualitative approaches. Specifically, systematic and structured evaluations are achieved through the scoring assessment method. The criterion weights obtained by FAHP were compared. The results indicated that productivity (0.0750), predictability (0.0626), factor productivity (0.0537), structural sustainability (0.0533), and adoption of new technology (0.0498) were the top five criteria with the most significant weights, indicating their importance for company evaluations. When assessing business performances related to open data implementation by the open data services industry, priority should be given to reviewing these five criteria.

The bottom three criteria were mutual correlation of value delivery (0.0190), data applicability (0.0186), and the creation of open data websites (0.0178). Currently, they are not criteria of high priority when implementing open data businesses. Results showed that existing open data services industries should strengthen their inherent abilities at executing businesses to complete systematic data connections with the public sector. Moreover, public–private interactions will promote open data utilization and improve data quality through feedback among ecosystem members.

The ranking of the importance of the various criteria was viewed from another perspective. It revealed that the top-ranking criteria were highly concentrated in two dimensions: productivity and robustness. The productivity dimension indicates that the operational focus of companies should be establishing good interactive relationships with other ecosystem members. Companies should improve their abilities to master new industrial technologies through such relationships, thereby developing meaningful business values for the entire ecosystem.

The evaluation framework of the robustness dimension suggests that existing open data industries must possess excellent abilities to predict industrial changes. Through acquiring industry information and observing industry trends, companies can effectively plan operating strategies in advance to cope with risks arising from future environmental changes. This is also one of the core elements affecting the development of the data industry, and it can improve the understanding of the future needs of companies.

The degree of compliance of Company T for the evaluation aspects of the open data ecosystem was, in descending order, as follows: niche creation (1.532), robustness (1.459), data governance (1.445), productivity (1.350), and co-creation (1.297). The ranking shows that Company T values the diversity of the ecosystem members and the synergy between them. In this way, technology, knowledge, and resources from various parties can be acquired. The ability to run an open data business can be enhanced; external market needs (including government, business, and the public) can be met; and overall market competitiveness can be enhanced. However, Company T is weak in terms of productivity and co-creation. Company T still has room for improvement in converting innovation into profit, resource integration, and interfacing with partners.

Firstly, in terms of co-creation, Company T is weak in knowledge spillover, alignment of goals and values, and derivation of value-added resources. Moreover, the market lacks clear and consistent goals, and as a result, Company T may lack the crucial resources to optimize its products and services. The alignment of goals and values is not susceptible to other criteria. In addition to satisfying various needs, stakeholders still need to innovate value-added applications through data complementation. The range of open data applications needs to be expanded based on collaborative partnerships and platform openness.

Secondly, the ranking results showed that financial productivity has the lowest degree of compliance in terms of co-creation. The ability to create value for business on a sustainable basis is a result of having excellent and competitive financial capabilities. Therefore, if Company T ignores the financial capabilities of the ecosystem as a whole in the future, it will seriously affect the activities of other ecosystem members. Specifically, it will limit the ability of members to adopt new technologies and the revenue performance generated by the sale of data and tools.

With the increasing type and quantity of data and diversification of user needs, many innovative and value-added data applications have been developed, indicating that the extraction of new knowledge from data and their applications is already a significant market trend. This study focused on the role of service providers in the open data industry, that is, data service providers whose primary business is to clean and integrate raw data and provide customized services according to users’ needs. Therefore, enterprises should abandon the past thought of fighting alone and work together to build a business ecosystem for synergistic operations to develop fair competition and cooperation strategies.

6. Conclusions

The open data service industry has built a business ecosystem to meet the rapid industrial changes and strengthen the specialized division of labor. The industry must develop a comprehensive evaluation framework to assess the operation and maturity of the business ecosystem and accordingly help enterprises formulate appropriate operational strategies. This study aimed to construct an evaluation scale for the business ecosystems of open data services. Through a preliminary literature review and expert questionnaire survey, we developed 5 dimensions and 31 criteria for evaluating the business ecosystems of open data services. Using the FAHP and fuzzy DEMATEL methods, we determined the weights of the dimensions and criteria, and identified the causality between them. Finally, through case analysis, we tested the applicability of the evaluation scale to increase the depth of this study.

Overall, enterprises can improve their performance in the short term by improving the high-weight criteria under the dimension of productivity. In the long run, profitability is not sufficient to help a business ecosystem meet the external impact of the industrial environment. Thus, enterprises need to improve productivity and co-creation to obtain long-term benefits. Likewise, enterprises can obtain short-term benefits by improving the high-weight criteria under different dimensions, including (1) data quality, governmental policy, and link to descriptive data (under data governance); (2) value productivity (under productivity); (3) predictability and structural sustainability (under robustness); (4) brain-storming, visionary leadership, and enterprise participation willingness (under niche creation); and (5) direct and indirect externality (under co-creation). As enterprises usually emphasize sustainable long-term operations, it is necessary to improve the evaluation criteria such as demand matching, creation of open data websites, financial productivity, timely elimination, and intangible resources.

Finally, the proposed evaluation scale may produce different evaluation results and determine different priorities for improvement, depending on the business type and the model of the enterprise. However, quantitative information enables enterprises to understand their advantages. Furthermore, collaboration with all stakeholders in the business ecosystem should be implemented to achieve goals such as resource integration, complementarity, and optimization.

Despite its contributions, this study has certain limitations. First, the evaluation scale provides “data application tools” for assessing the business ecosystems of open data services. The scale is merely oriented toward the current status of the open data service industry. Moreover, its evaluation items are expected to vary with industrial changes. Therefore, future studies should consider development factors (e.g., open APIs or the 5G environment) to offer more multifaceted suggestions. Second, evaluating the business ecosystems of open data services involves many aspects. However, this study only discussed the criteria for the maturity of a business ecosystem of open data services. Future studies could examine the evaluation criteria for different vertical application fields (e.g., retail, e-commerce, and transportation). In the future, researchers can expand our research using different tools (e.g., analytic network process or the best-worst method) to determine the criterion’s weights and compare the difference and applicability with the current model. In addition, the group multiple criteria decision-making approach can be used to aggregate the opinions of experts from various backgrounds.

This study has certain limitations despite its significant contribution to determining each key criterion’s importance for the open data services business ecosystem. The proposed method focuses on determining criterion weights, with the importance of each criterion being assessed by experts through pairwise-criteria comparisons. The disparity between the evaluated company and the benchmark cannot be determined because such a benchmark does not exist. In the future, other methods can be used to establish benchmarks for analysis. They include grey relational analysis (GRA), the technique for order preference by similarity to ideal solution (TOPSIS) technique, and the Visekriterijumska Optimizacija I Kompromisno Resenje (VIKOR) method, also known as the multicriteria optimization and compromise solution.