A Rough Hybrid Multicriteria Decision-Making Model for Improving the Quality of a Research Information System

Abstract

Improving the quality of research information systems is an important goal in the process of improving the performance of research management in Chinese universities. Since the evaluation of information system (IS) quality is a multicriteria decision problem, it is critical to identify the interrelationships among the dimensions and criteria, and decide on the important criteria for proposed improvement strategies. This paper suggests a hybrid multicriteria decision-making (MCDM) model for improving the quality of a research information system. First, a rough method combined with the decision-making trial and evaluation laboratory and analytical network process (rough DANP) model is used to improve the objectivity of expert judgements. Additionally, the rough DANP can be used to construct an influential network relationship map (INRM) between research information system components to derive the criterion weights. The complex proportional assessment of alternatives with rough numbers (COPRAS-R) is applied to evaluate the performance of the research information system. A Chinese university research information system is chosen to illustrate the usefulness of the proposed model. The results show that efficiency, effectiveness, and user frequency have the highest priorities for improvement. Selected management implications based on the actual case study are supplied.

1. Introduction

Currently, the increasing speed of scientific and technological achievements is crucial to improvements in innovation ability. The university research platform is a major focus of a country’s innovation-driven development. As an important component of the national scientific research system, research in colleges and universities plays a highly important role in the development of science and technology and economic construction. Enterprises, universities, scientific research institutions, and other departments have established online scientific research information systems to offer a service platform for technology research and development. For example, Slovenia has established a network of scientific research cooperation since 1960, and scientists are looking for partners through the network [1]. Through an analysis of scientists’ output data from information systems in Slovenia over the past 40 years, Perc found that researchers’ output is consistent with Zipf’s law and log-normal distributions [2]. Lužar et al. (2014) studied the interdisciplinary nature of the research community in the scientific network, and they found that the number of cooperation between different professional disciplines is growing slowly and needs further stimulation [3]. A well-designed information system needs to address issues such as system stability and user feedback [4]. However, colleges and universities are slow to develop their uses of these scientific research systems in China. An excellent research information system could act as an effective method that aids university administrators in managing the achievements of researchers and teachers. Researchers and teachers can obtain and share important information using the research system, and arrange scientific research activities in an orderly manner. Therefore, the establishment of a high-performance university research information system is essential for the overall improvement of university research.

Information on research tasks and projects must be available in less time and with greater reliability, and the content in the system must be accurate and useful. Therefore, the quality of the research information system decides the satisfaction degree of the users. The evaluation of the quality of the information system is a multidimensional problem. Many researchers have offered ideas and advice for assessing the quality of an information system. One of the classic information system models is the information systems success model developed by Delone and Mclean [5,6], which is also known as the information system (IS) success model or D&M model. This model identified and explained six of the most critical dimensions on which information systems are commonly evaluated, namely, information quality, system quality, service quality, use intention, user satisfaction, and net system benefit.

Based on the D&M IS success model, certain researchers updated and improved this model in different types of information systems. Pitt et al. [7] argued that service quality is a highly important component in the success of a D&M model, and added service quality as a new variable to reflect the quality of the IS model. Seddon et al. [8] thought that net benefit could replace the function of individual impact and organizational impact. Yeo [9] updated the D&M model by changing “Use” to “Intention to use | Use”. In addition to updating the model, academia also applied the D&M model to different industry information systems. Zheng et al. [10] argued that information quality and system quality have a strong influence on individual benefits and user satisfaction in a virtual community. Azeroual et al. [11] suggested that the success of a research information system is largely related to the quality of the available data, and they improved the quality of the research information system via data profiling. Dwivedi et al. [12] summarized the factors that determine the success and failures of the information systems, and found that the type of information systems, such as enterprise resource planning (ERP) e-government, and the degree of implementation by employees and management are the main factors. Grudzień and Hammrol [13] suggested that the quality of the information is an important component of information system quality management. These statistical models contributed to exploration of the interrelationships among variables. However, due to the limitations of statistical methods, these studies encountered difficulty in helping decision makers understand the gaps between the current performance and the aspirational level.

To solve this problem, certain researchers used multiple criteria decision-making (MCDM) models to evaluate the information system. For example, Lin [14] applied fuzzy Analytics Hierarchy Process (AHP) to evaluate course website system quality. Hsu and Lee [15] used the decision-making trial and evaluation laboratory (DEMATEL) method to explore the critical factors that influence the quality of blog interfaces. Tsai et al. [16] developed an effectiveness evaluation model for web-based marketing using DEMATEL, analytic network process (ANP), and VlseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR). Su et al. [17] developed a hybrid fuzzy MCDM model for improving e-learning innovation performance in a fuzzy environment. These MCDM models solved the evaluation or improvement problems for the service quality of an information system, but they did not consider the group summary of expert opinions, and the D&M model was not considered in their studies. Furthermore, none of the literature discusses the relationship of the factors that influence the success of an information system, and researchers have seldom proposed methods for exploring the importance of those factors.

This paper attempts to fill this gap by proposing a causal relationship framework for the development of research information system assessment capability. First, based on the D&M IS success model, this study proposes a hybrid rough MCDM model that combines rough concepts and the DEMATEL-based ANP model (rough DANP, or RDANP) to construct interdependent connections among the assessment dimensions and criteria, and obtain the weights of each dimension and criterion. Second, based on the results of the use of the RDANP model, a complex proportional assessment of alternatives with the rough numbers method (COPRAS-R) is applied to evaluate the performance of a research information system in China as a case study. The D&M model is easy to understand by confirming the relationship on each category and criteria. The derived influential network relationship map (INRM) can help decision makers understand the complex relationships in the research information system. The weights of the information system assessment factors can be calculated using the rough DANP method. The strength of this hybrid model can be used to show the cause-and-effect relationships and obtain the influence weights of factors within the research information system. Consequently, differing from existing studies that focused on finding the influence factors, this paper contributes to the literature by attempting to construct a cause-and-effect method for a research information system that could help to identify the key factors for the research information system and supply directions for improvement.

2. Literature Review

Based on the previous decision-making research on the service quality of information systems, it is common to apply statistical methods to verify the application of the D&M models in various fields. Section 2.2 introduces the MCDM method to evaluate the performance of information systems, followed by an explanation of research gaps in Section 2.3.

2.1. Statistical Methods for the D&M Model

DeLone and McLean [5] proposed a famous information system success model known as the D&M model that has the ability to measure system standards, and this model has been theoretically and empirically developed and improved in several studies [18,19]. For example, Bernroider [20] applied the D&M model to measure the effectiveness of adopting ERP in different sizes of companies from the perspective of information technology. Correlation analysis and principal component analysis were used in this paper, and the researchers found that the D&M success model was well suited to assessment of the performance of application of the ERP system in small- and medium-sized companies. Chen et al. [21] evaluated the quality of a web-based learning system for nurses in a hospital and found that system quality, information quality, and service quality are the three dimensions used to measure the quality of a learning system. Two-factor analysis and SEM methods were applied to validate the model practicality, and this research developed an instrument used to assess the quality of a web-based learning system based on the D&M model. By applying the updated D&M model, Bossen et al. [22] satisfactorily evaluated the implementation of comprehensive electronic health records (EHR) in a hospital in Denmark. Bossen found that medical secretaries and physicians encounter certain difficulties in using the patient administration method and establishing professionally relevant data. Kyoung et al. [23] also evaluated the performance of a public hospital information system using the D&M model, and the relationships among IS success factors were assessed using the structural equation model. The results indicated that the intention to use the system had no influence on the net benefit. Chiu et al. [24] evaluated mobile e-books in a cloud bookcase using an information system success model and applied partial least squares structural equation modeling (PLS-SEM) to verify the model. The result showed that system quality and service quality had a positive influence on intention to use the cloud e-bookcase and user satisfaction. Ayyash [18] developed four dimensions (accuracy, timeliness, completeness, and relevancy) used to reflect information quality, and discussed the relationship between these four dimensions and customer satisfaction in the e-banking industry. Ayyash used multiple regression to analyze the data, and the results indicated that all of the dimensions had positive effects on customer satisfaction.

Selected researchers applied the D&M model in online learning systems such as Massive Open Online Courses (MOOCs), mobile learning systems, and blog-based learning systems [25,26]. Based on the D&M theory, Wu and Chen [27] applied the structure equation model and multiple group analysis method to a Facebook system, and the result showed that social influence and information quality are critical and direct determinants that affect users’ continuous intention to use Facebook in learning. E-commerce and e-government system success could also be measured by the D&M model from a statistical perspective [28,29,30].

The above-mentioned literature used statistical techniques such as multiple regression or factor analysis to analyze the application effects of D&M models in various industries. Certain researchers attempted to find causal relationships among the variables. The results showed that the conclusions were more or less different when the D&M model was applied in different contexts. Most results indicated that system quality, service quality, and information quality had a positive influence on user satisfaction and intention to use the system. However, the most significant factors affecting the user satisfaction and the interrelationship among all dimensions and criteria are seldom discussed.

2.2. MCDM Models

The information system has become a common practice in various industries and organizations, and has helped to improve operational efficiency through the application of an information system. Many organizations have currently established their own information systems by outsourcing to third-party companies, purchasing existing systems, or self-development. The question of how to evaluate the effectiveness and quality of an information system is a problem faced by users. In practice, the purchaser of an information system organizes experts to appraise the system quality. However, assessing the quality of the information systems requires multiple perspectives, and is a multicriteria decision-making management problem. The MCDM method is a practical and academic methodology that offers a systematic modeling framework and methodology under multiple measurement indicators. This method is used to simultaneously consider multiple criteria to supply a decision-making optimization and decision-making model for decision makers [31].

Thus far, a lack of research persists on the goal of integrating the interdependencies to improve information system performance. To identify the interrelationships among indicators of information systems, the DEMATEL approach is used to construct a network structure map with interdependent relationships [32,33,34,35]. A hybrid MCDM model that combines DEMATEL and the ANP methods can show the cause-and-effect relationships and also obtain the influence factors in systems. Therefore, this method was used to construct interdependent connections among indicators [36,37,38,39]. However, these two methods often ignore the subjective defects of decision makers. As a result, updated models were proposed by certain researchers. Bai and Sarkis [40] introduced a gray-based DEMATEL model that can be used to solve uncertainty problems for evaluating critical success factors in business process management. The fuzzy-based DEMATEL method was proposed to evaluate green supply chain management (SCM) [41,42]. Compared with gray and fuzzy technology, rough technology does not need to assume a semantic scale. Rough DEMATEL establishes scales based on the values filled out by experts. Therefore, the semantic scale of each question is not fixed. Song and Cao [43] proposed a rough DEMATEL-based approach for assessing the internal relationship between requirements of the product–service system. This study adopts a flexible rough interval, which better reflects the subjective judgment and fuzzy judgment of decision makers and successfully adapts to DEMATEL in an uncertain environment. Pamučar et al. [44] defined a hybrid DEMATEL–ANP–Multi-Attributive Ideal-Real Comparative Analysis (MAIRCA) model based on interval rough numbers to solve uncertainty in the multicriteria decision-making process, and this model showed high stability with respect to changes in the nature and characteristics of the criteria.

2.3. Research Gaps

As a well-known model in the field of information systems, D&M is widely used in both theoretical and practical fields. Most studies have focused on verifying the relevance and causality of the D&M model dimensions in different domains from a statistical perspective. However, the question of how to improve the quality of information systems has not been well resolved in theory and practice. As a powerful tool for decision-making in the field of decision management, MCDM can supply a scientific and rational strategy for improving the quality of information systems based on the D&M model. This paper offers ideas for studying the scientific nature of D&M from the perspective of management science. As discussed in the introduction, although selected researchers evaluated information by applying the AHP, ANP, and DAMATEL methods, they seldom considered the ambiguity of the opinions from expert groups, and the D&M model was also rarely used in their papers. Based on the above analysis, this study applies the rough DEEMATEL and the rough DANP methods to explore the network relationship of the criteria and the weights of the criteria in the research information system. Based on the result of rough DANP, the COPRAS-R is used to assess the performance of a research information system in China as a case study.

3. Methodology

In this section, we introduce the RMADM model that combines RDANP with COPRAS-R to establish the interdependent structure, get the weights of dimensions and criteria, and assess the aspiration gap of alternative criteria. Schools can figure out the complex relationship between information system and the cause–effect within the service criteria through the derived rough influence-network relation map (RINRM). The obtained influential weights of the criteria and the RINRM can help schools and the information system set the improved priorities for bettering the school information system in schools. The detailed procedures are illustrated as follows.

3.1. The Rough Number

The rough set theory was developed by Pawlak in 1982 [45] for solving a group of subjective and imprecise expert perceptions. Zhai et al. [46] used rough numbers, a set of interval values that are clearly transformed from a crisp group, to show subjectivity and imprecise human thinking in a multiple-attribute decision-making (MADM) environment. Since the application of rough number theory to address decision problems does not require any auxiliary information, the theory is better able to capture the perception of experts [47,48,49]. The detailed steps are as follows:

• Step 1: Conform lower and upper approximations of rough number for each crisp scale.

U is a set of the universe that contains all the respondents with multi-attributes, and Θ is a set of z class under respondents. $R=\left\{z_1,z_2,\dots ,z_{\mathsf{\Theta }}\right\}$ are ordered in the manner of $z_1$ < $z_2$ < … < $z_{\mathsf{\Theta }}$ and Y is an arbitrary respondent of U. The lower approximation and upper approximation of $z_{\mathsf{\Theta }}$ can be defined as [46]:

Lower approximation:

$$\underset{¯}{Apr}\left(z_{\mathsf{\Theta }}\right)=\cup \left\{Y\in U/R\left(Y\right)\le z_{\mathsf{\Theta }}\right\}$$

(1)

Upper approximation:

$$\overline{Apr}\left(z_{\mathsf{\Theta }}\right)=\cup \left\{Y\in U/R\left(Y\right)\ge z_{\mathsf{\Theta }}\right\}$$

(2)

• Step 2: Compute the interval value of the rough number.

A group of expert opinions can be expressed by $\left[\underset{¯}{Lim}\left(z_{\mathsf{\Theta }}\right),\overline{Lim}\left(z_{\mathsf{\Theta }}\right)\right]$, which are derived by:

$$\underset{¯}{Lim}\left(z_{\mathsf{\Theta }}\right)=\frac{{\displaystyle \sum {}_{i=1}^{N_L}}{x}_i}{N_L},\overline{Lim}\left(z_{\mathsf{\Theta }}\right)=\frac{{\displaystyle \sum {}_{i=1}^{N_U}}{y}_i}{N_U}$$

(3)

where x_i and y_i are denoted the elements in the lower approximation or upper approximation of z_Θ, respectively. In addition, N_L and N_U are represented the total number of respondents involved in the lower and upper approximations of z_Θ, respectively.

• Step 3: Drive operations for two rough numbers.

Through Equations (1)–(3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ(z_Θ), as shown in Equation (4):

$$\mathsf{\Upsilon }\left(z_{\mathsf{\Theta }}\right)=\left[z_{\mathsf{\Theta }}^{\nabla },z_{\mathsf{\Theta }}^{\mathsf{\Delta }}\right]=\left[\underset{¯}{Lim}\left(z_{\mathsf{\Theta }}\right),\overline{Lim}\left(z_{\mathsf{\Theta }}\right)\right]$$

(4)

Assuming two rough numbers $\mathsf{\Upsilon }\left(\alpha \right)$ and $\mathsf{\Upsilon }\left(\beta \right)$ and $\mu$ as a nonzero constant, the arithmetic operations of rough number can be followed by:

$$\mathsf{\Upsilon }\left(\alpha \right)\times \mu =\left[\underset{¯}{{\alpha }_{\mathsf{\Theta }}^{\nabla }\times \mu ,{\alpha }_{\mathsf{\Theta }}^{\mathsf{\Delta }}\times \mu }\right]$$

(5)

$$\mathsf{\Upsilon }\left(\alpha \right)+\mathsf{\Upsilon }\left(\beta \right)=\left[{\alpha }_{\mathsf{\Theta }}^{\nabla }+{\beta }_{\mathsf{\Theta }}^{\nabla },{\alpha }_{\mathsf{\Theta }}^{\mathsf{\Delta }}+{\beta }_{\mathsf{\Theta }}^{\mathsf{\Delta }}\right]$$

(6)

$$\mathsf{\Upsilon }\left(\alpha \right)\times \mathsf{\Upsilon }\left(\beta \right)=\left[{\alpha }_{\mathsf{\Theta }}^{\nabla }\times {\beta }_{\mathsf{\Theta }}^{\nabla },{\alpha }_{\mathsf{\Theta }}^{\mathsf{\Delta }}\times {\beta }_{\mathsf{\Theta }}^{\mathsf{\Delta }}\right]$$

(7)

• Step 4: Transfer into crisp value from rough interval value. When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

$$z_{\mathsf{\Theta }}^{deRou}=\left[z_{\mathsf{\Theta }}^{\nabla }+z_{\mathsf{\Theta }}^{\mathsf{\Delta }}\right]/2$$

(8)

3.2. The RDANP Method

The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM. The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained. The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51]. In addition, we also consider the ambiguity of the human domain knowledge in the real environment. The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria. The detailed steps are as follows [52]:

• Step 1: Build a rough original influence relationship matrix $\overline{\mathit{M}}$ on a measuring scale of 0–4 ranging from “no influence (0)” to “very high influential (4)”.

Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by m_ij. We checked the consistency of the raw data, using the formula as follows:

The average gap-ratio in consensus (%) $=\frac{1}{m\left(m-1\right)}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^m\left(\raisebox{1ex}{$\left|d_{ij}^k-d_{ij}^{k-1}\right|$}\!\left/ \!\raisebox{-1ex}{$d_{ij}^s$}\right.\right)}}\ast 100\%$, where m is the number of the criteria, k is the number of the expert, $d_{ij}^k$ is the average direct influence value of criterion i to j. If the value of the average gap ratio in consensus is less than 0.05, we believe that the expert’s rating is consistent [48]. Then, the rough direct relation average matrix $\overline{\mathit{M}}$ is acquired through Equations (1)–(8) in the k matrices for the respondents. Finally, we can obtain a rough original influence relationship matrix $\overline{\mathit{M}}$ = [${\overline{m}}_{ij}$]_nxn = ${\left[m_{ij}^{\nabla },m_{ij}^{\mathsf{\Delta }}\right]}_{n\times n}$, where is n the number of criteria.

• Step 2: Obtain the rough initial influence relationship matrix $\overline{\mathit{P}}$ = [${\overline{p}}_{ij}$]_nxn, which is the multiplication of $\overline{\mathit{M}}$ and v.

  $$\overline{\mathit{P}}=v\times \overline{\mathit{M}}$$

  (9)

  $$v=\min \left[\begin{array}{cc}\frac{1}{\underset{i}{\max }{\displaystyle {\sum }_j^n\left|m_{ij}^u\right|}},& \frac{1}{\underset{j}{\max }{\displaystyle {\sum }_i^n\left|m_{ij}^u\right|}}\end{array}\right]$$

  (10)

• Step 3: Calculate the rough total influence relationship matrix $\overline{\mathit{T}}$ with Equation (11). The element ${\overline{t}}_{ij}$ indicates the rough interdependent effects that criteria i has on criteria j, where I is an identity matrix.

  $$\overline{\mathit{T}}=\overline{\mathit{P}}+{\overline{\mathit{P}}}^2+\cdots +{\overline{\mathit{P}}}^{\mathsf{\Omega }}=\overline{\mathit{P}}{(\mathit{I}-\overline{\mathit{P}})}^{-1}\mathrm{when}\mathsf{\Omega }\to \infty$$

  (11)

• Step 4: Derive each column sum (${\overline{c}}_i$) and row sum (${\overline{r}}_i$) from the rough total influence relationship matrix $\overline{\mathit{T}}$ as follows:

  $${\overline{c}}_j={\left({\overline{c}}_j\right)}_{1\times n}={\left({\overline{c}}_j\right)}_{n\times 1}^{\prime }={\left[{\displaystyle \sum _{i=1}^n{\overline{t}}_{ij}}\right]}^{\prime }$$

  (12)

  $${\overline{r}}_i={\left({\overline{r}}_i\right)}_{n\times 1}=\left[{\displaystyle \sum _{j=1}^n{\overline{t}}_{ij}}\right]$$

  (13)

The element ${\overline{c}}_j$ denotes the rough total effects by criterion j, from the other criteria. Similarly, ${\overline{r}}_i$ represents the rough direct and indirect effects of criterion i on the other criteria.

• Step 5: Get the RINRM for whole evaluation model.

Thus, ${\overline{r}}_i+{\overline{c}}_j$ reflects the strength of the influences given and received on criterion i, while ${\overline{r}}_i-{\overline{c}}_j$ shows the net effect of criterion i on the other criteria. Clearly, if ${\overline{r}}_i-{\overline{c}}_j$ is positive, criterion i is a causal component, and if ${\overline{r}}_i-{\overline{c}}_j$ is negative, then criterion i is an affected component. As a result, the rough influence network relationship map (RINRM) can be finished by mapping the data set (${\overline{r}}_i+{\overline{c}}_j$,${\overline{r}}_i-{\overline{c}}_j$).

• Step 6: Derive rough total influence relationship matrix ${\overline{\mathit{T}}}_C$ based on the criteria and ${\overline{\mathit{T}}}_D$ based on the dimensions.

The rough matrix $\overline{\mathit{T}}$ could be differentiated into ${\overline{\mathit{T}}}_C$ based on the criterion and ${\overline{\mathit{T}}}_D$ based on the dimensions. The matrix ${\overline{\mathit{T}}}_C$ modularizes the matrix $\overline{\mathit{T}}$ according to the dimensions, which are essentially the same matrix. The rough matrix ${\overline{\mathit{T}}}_D$ is found by averaging the rough degree of criterion influence in each dimension. As shown in Equation (14), D_i denotes the ith dimension; c_ij denotes the jth criteria in the ith dimension. For example, we get a crisp value by averaging ${\overline{\mathit{T}}}_c^{mn}$, where ${\overline{\mathit{T}}}_c^{mn}$ means the extent to which the criteria in the mth dimension affect the criteria in the nth dimension. Then, we get the ${\overline{\mathit{T}}}_D$ by averaging the ${\overline{\mathit{T}}}_c^{mn}$ in the ${\overline{\mathit{T}}}_C$.

$${\overline{\mathit{T}}}_C=\begin{array}{ll}& \begin{array}{cccc}{D}_1& D_2& \cdots & D_n\\ C_{11}\cdots C_{1m_1}& C_{21}\cdots C_{2m_2}& \cdots & C_{n1}\cdots C_{n{m}_n}\end{array}\\ \begin{array}{cc}{D}_1& \begin{array}{}{C}_{11}\\ C_{12}\\ ⋮\\ C_{1m_1}\end{array}\\ \begin{array}{}{D}_2& ⋮\end{array}& \begin{array}{}{C}_{21}\\ C_{22}\\ ⋮\\ C_{2m_2}\\ ⋮\end{array}\\ D_n& \begin{array}{}{C}_{n1}\\ C_{n2}\\ ⋮\\ C_{n{m}_n}\end{array}\end{array}& \left[\begin{array}{cccc}{\overline{\mathit{T}}}_c^{11}& {\overline{\mathit{T}}}_c^{12}& \cdots & {\overline{\mathit{T}}}_c^{1n}\\ {\overline{\mathit{T}}}_c^{21}& {\overline{\mathit{T}}}_c^{22}& \cdots & {\overline{\mathit{T}}}_c^{2n}\\ ⋮& ⋮& \cdots & ⋮\\ {\overline{\mathit{T}}}_c^{n1}& {\overline{\mathit{T}}}_c^{n2}& \cdots & {\overline{\mathit{T}}}_{\mathrm{c}}^{nn}\end{array}\right]\end{array}=\left[{\mathit{T}}_C^{\nabla },{\mathit{T}}_C^{\mathsf{\Delta }}\right]$$

(14)

• Step 7: Obtain the rough unweighted supermatrix.

We get a rough matrix ${\overline{\mathit{T}}}_C^{\delta }$ by normalizing ${\overline{\mathit{T}}}_C$.

$${\overline{\mathit{T}}}_C^{\delta }=\begin{array}{ll}& \begin{array}{cccc}{D}_1& D_2& \cdots & D_n\\ C_{11}\cdots C_{1m_1}& C_{21}\cdots C_{2m_2}& \cdots & C_{n1}\cdots C_{n{m}_n}\end{array}\\ \begin{array}{cc}{D}_1& \begin{array}{}{C}_{11}\\ C_{12}\\ ⋮\\ C_{1m_1}\end{array}\\ \begin{array}{}{D}_2& ⋮\end{array}& \begin{array}{}{C}_{21}\\ C_{22}\\ ⋮\\ C_{2m_2}\\ ⋮\end{array}\\ D_n& \begin{array}{}{C}_{n1}\\ C_{n2}\\ ⋮\\ C_{n{m}_n}\end{array}\end{array}& \left[\begin{array}{cccc}{\overline{\mathit{T}}}_c^{\delta 11}& {\overline{\mathit{T}}}_c^{\delta 12}& \cdots & {\overline{\mathit{T}}}_c^{\delta 1n}\\ {\overline{\mathit{T}}}_c^{\delta 21}& {\overline{\mathit{T}}}_c^{\delta 22}& \cdots & {\overline{\mathit{T}}}_c^{\delta 2n}\\ ⋮& ⋮& \cdots & ⋮\\ {\overline{\mathit{T}}}_c^{\delta n1}& {\overline{\mathit{T}}}_c^{\delta n2}& \cdots & {\overline{\mathit{T}}}_{\mathrm{c}}^{\delta nn}\end{array}\right]\end{array}=\left[{\mathit{T}}_C^{\delta \nabla },{\mathit{T}}_C^{\delta \mathsf{\Delta }}\right]$$

(15)

For example, ${\mathit{T}}_C^{\delta \mathsf{\Delta }}$, which is a rough submatrix of ${\overline{\mathit{T}}}_C^{\delta }$, can be normalized to ${\mathit{T}}_C^{\delta \mathsf{\Delta }pq}$, as follows:

$$\begin{array}{l}\begin{array}{cccc} & \hspace{0.17em}{c}_{q1}\hspace{1em}& \hspace{1em}\cdots & \begin{array}{cccc}{c}_{qj}& \hspace{1em}\cdots & c_{q{m}_q}& \end{array}\end{array}\\ {\mathit{T}}_c^{\mathsf{\Delta }pq}=\begin{array}{c}{c}_{p1}\\ ⋮\\ c_{pi}\\ \begin{array}{c}⋮\\ c_{p{m}_p}\end{array}\end{array}\left[\begin{array}{c}{t}_{11}^{\mathsf{\Delta }pq} \cdots t_{1j}^{\mathsf{\Delta }pq} \cdots t_{1m_q}^{\mathsf{\Delta }pq}\\ ⋮ \hspace{1em}⋮ ⋮\hfill \\ \boxed{t_{i1}^{\mathsf{\Delta }pq} \cdots t_{ij}^{\mathsf{\Delta }pq} \cdots t_{i{m}_q}^{\mathsf{\Delta }pq}}\\ ⋮ \hspace{1em}⋮ ⋮\hfill \\ t_{m_{p}1}^{\mathsf{\Delta }pq} \cdots t_{m_{p}j}^{\mathsf{\Delta }pq} \cdots t_{m_p{m}_q}^{\mathsf{\Delta }pq}\end{array}\right]\begin{array}{cc}& \\ & \\ \to & t_i^{\mathsf{\Delta }pq}={\displaystyle \sum _{j=1}^{m_q}{t}_{ij}^{\mathsf{\Delta }pq}}\\ & \\ & \end{array}\end{array}$$

(16)

$$\begin{array}{l}\begin{array}{cccc}& & & \end{array}\begin{array}{cccc}& c_{q1}\begin{array}{cc}& \end{array}& \hspace{1em}\hspace{0.17em}\cdots & \begin{array}{cccc}\hspace{1em}{c}_{qj}& \begin{array}{cc}& \end{array}\hspace{1em}\cdots & \hspace{0.17em}{c}_{q{m}_q}& \begin{array}{cc}& \end{array}\end{array}\end{array}\\ {\mathit{T}}_C^{\delta \mathsf{\Delta }pq}=\begin{array}{c}{c}_{p1}\\ ⋮\\ c_{pi}\\ \begin{array}{c}⋮\\ c_{p{m}_p}\end{array}\end{array}\left[\begin{array}{cccc}\raisebox{1ex}{$t_{11}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{\mathsf{\Delta }pq}$}\right.& \cdots & \raisebox{1ex}{$t_{1j}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{\mathsf{\Delta }pq}$}\right.& \begin{array}{cc}\cdots & \raisebox{1ex}{$t_{1m_q}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{\mathsf{\Delta }pq}$}\right.\end{array}\\ ⋮& & ⋮& \begin{array}{cc}& ⋮\end{array}\\ \raisebox{1ex}{$t_{i1}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{\mathsf{\Delta }pq}$}\right.& \cdots & \raisebox{1ex}{$t_{ij}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{\mathsf{\Delta }pq}$}\right.& \begin{array}{cc}\cdots & \raisebox{1ex}{$t_{i{m}_q}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{\mathsf{\Delta }pq}$}\right.\end{array}\\ \begin{array}{c}⋮\\ \raisebox{1ex}{$t_{m_{p}1}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{\mathsf{\Delta }pq}$}\right.\end{array}& \begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ \raisebox{1ex}{$t_{m_{p}j}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{\mathsf{\Delta }pq}$}\right.\end{array}& \begin{array}{cc}\begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ \raisebox{1ex}{$t_{m_p{m}_q}^{\mathsf{\Delta }pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{\mathsf{\Delta }pq}$}\right.\end{array}\end{array}\end{array}\right]=\left[\begin{array}{cccc}{t}_{11}^{\delta \mathsf{\Delta }pq}& \cdots & t_{1j}^{\delta \mathsf{\Delta }pq}& \begin{array}{cc}\cdots & t_{1m_q}^{\delta \mathsf{\Delta }pq}\end{array}\\ ⋮& & ⋮& \begin{array}{cc}& ⋮\end{array}\\ t_{i1}^{\delta \mathsf{\Delta }pq}& \cdots & t_{ij}^{\delta \mathsf{\Delta }pq}& \begin{array}{cc}\cdots & t_{i{m}_q}^{\delta \mathsf{\Delta }pq}\end{array}\\ \begin{array}{c}⋮\\ t_{m_{p}1}^{\delta \mathsf{\Delta }pq}\end{array}& \begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ t_{m_{p}j}^{\delta \mathsf{\Delta }pq}\end{array}& \begin{array}{cc}\begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ t_{m_p{m}_q}^{\delta \mathsf{\Delta }pq}\end{array}\end{array}\end{array}\right]\end{array}$$

(17)

The rough unweighted supermatrix $\overline{\mathit{W}}$ is transposed from rough matrix ${\overline{\mathit{T}}}_C^{\delta }$, as shown in Equation (18).

$${\overline{\mathit{W}}}_C=\begin{array}{ll}& \begin{array}{cccc}{D}_1& D_2& \cdots & D_n\\ C_{11}\cdots C_{1m_1}& C_{21}\cdots C_{2m_2}& \cdots & C_{n1}\cdots C_{n{m}_n}\end{array}\\ \begin{array}{cc}{D}_1& \begin{array}{}{C}_{11}\\ C_{12}\\ ⋮\\ C_{1m_1}\end{array}\\ \begin{array}{}{D}_2& ⋮\end{array}& \begin{array}{}{C}_{21}\\ C_{22}\\ ⋮\\ C_{2m_2}\\ ⋮\end{array}\\ D_n& \begin{array}{}{C}_{n1}\\ C_{n2}\\ ⋮\\ C_{n{m}_n}\end{array}\end{array}& \left[\begin{array}{cccc}{\overline{\mathit{W}}}_{11}& {\overline{\mathit{W}}}_{12}& \cdots & {\overline{\mathit{W}}}_{1n}\\ {\overline{\mathit{W}}}_{21}& {\overline{\mathit{W}}}_{22}& \cdots & {\overline{W}}_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ {\overline{\mathit{W}}}_{n1}& {\overline{\mathit{W}}}_{n2}& \cdots & {\overline{\mathit{W}}}_{nn}\end{array}\right]\end{array}={\left({\overline{\mathit{T}}}_C^{\delta }\right)}^{\prime }$$

(18)

• Step 8: Derive the rough weighted supermatrix.

Referring to step 5, we can get rough matrix ${\overline{\mathit{T}}}_{\mathit{D}}=\left[{\mathit{T}}_D^{\nabla },{\mathit{T}}_D^{\mathsf{\Delta }}\right]$ by averaging the degree of the criterion influence in each dimension, such as ${\mathit{T}}_D^{\mathsf{\Delta }}=[t_D^{\mathsf{\Delta }nm}]$, which is derived by Equations (19)–(21).

$$\raisebox{1ex}{$t_D^{\mathsf{\Delta }nm}={\displaystyle \sum _{i=1}^{i_n}{\displaystyle \sum _{j=1}^{j_m}{t}_{dij}^{\mathsf{\Delta }}}}$}\!\left/ \!\raisebox{-1ex}{$i_n\times j_m$}\right.$$

(19)

where i_n is the number of criteria in dimension n; and j_m is the number of criteria in the dimension m.

$${\mathit{T}}_D^{\mathsf{\Delta }}=\left[\begin{array}{c}{t}_D^{\mathsf{\Delta }11} t_D^{\mathsf{\Delta }12}\cdots t_D^{\mathsf{\Delta }1n}\\ t_D^{\mathsf{\Delta }21} t_D^{\mathsf{\Delta }22}\cdots t_D^{\mathsf{\Delta }2n}\\ ⋮ ⋮ ⋮\hfill \\ t_D^{\mathsf{\Delta }n1} { }_D^{\mathsf{\Delta }n2}\cdots t_D^{\mathsf{\Delta }nn}\end{array}\right]\begin{array}{cc}& \\ \to & {\delta }_2^{\mathsf{\Delta }}={\displaystyle \sum _{j=1}^n{t}_D^{\mathsf{\Delta }2j}}\\ & \\ & \end{array}$$

(20)

$${\mathit{W}}_D^{\mathsf{\Delta }}={\left({\mathit{T}}_D^{\delta \mathsf{\Delta }}\right)}^{\prime }=\left[\begin{array}{cccc}\raisebox{1ex}{$t_D^{\mathsf{\Delta }11}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1^{\mathsf{\Delta }}$}\right.& \raisebox{1ex}{$t_D^{\mathsf{\Delta }12}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1^{\mathsf{\Delta }}$}\right.& \cdots & \raisebox{1ex}{$t_D^{\mathsf{\Delta }1n}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1^{\mathsf{\Delta }}$}\right.\\ \raisebox{1ex}{$t_D^{\mathsf{\Delta }21}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2^{\mathsf{\Delta }}$}\right.& \raisebox{1ex}{$t_D^{\mathsf{\Delta }22}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2^{\mathsf{\Delta }}$}\right.& \cdots & \raisebox{1ex}{$t_D^{\mathsf{\Delta }2n}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2^{\mathsf{\Delta }}$}\right.\\ ⋮& ⋮& ⋮& ⋮\\ \raisebox{1ex}{$t_D^{\mathsf{\Delta }n1}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n^{\mathsf{\Delta }}$}\right.& \raisebox{1ex}{$t_D^{\mathsf{\Delta }n2}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n^{\mathsf{\Delta }}$}\right.& \cdots & \raisebox{1ex}{$t_D^{\mathsf{\Delta }nn}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n^{\mathsf{\Delta }}$}\right.\end{array}\right]=\left[\begin{array}{cccc}{t}_D^{\delta \mathsf{\Delta }11}& t_D^{\delta \mathsf{\Delta }12}& \cdots & t_D^{\delta \mathsf{\Delta }1n}\\ t_D^{\delta \mathsf{\Delta }21}& t_D^{\delta \mathsf{\Delta }22}& \cdots & t_D^{\delta \mathsf{\Delta }2n}\\ ⋮& ⋮& ⋮& ⋮\\ t_D^{\delta \mathsf{\Delta }n1}& t_D^{\delta \mathsf{\Delta }n2}& \cdots & t_D^{\delta \mathsf{\Delta }nn}\end{array}\right]$$

(21)

Then, we can obtain the rough weighted supermatrix ${\overline{\mathit{W}}}_{\delta }=\left[{\mathit{W}}_{\delta }^{\nabla },{\mathit{W}}_{\delta }^{\mathsf{\Delta }}\right]$ by multiplying the unweighted supermatrix ${\overline{\mathit{W}}}_C=\left[{\mathit{W}}_C^{\nabla },{\mathit{W}}_C^{\mathsf{\Delta }}\right]$ with ${\overline{\mathit{W}}}_D^{}=\left[{\mathit{W}}_D^{\nabla },{\mathit{W}}_D^{\mathsf{\Delta }}\right]$, such as ${\mathit{W}}_{\delta }^{\mathsf{\Delta }}$, which is shown in Equation (22).

$${\mathit{W}}_{\delta }^{\mathsf{\Delta }}={\mathit{W}}_D^{\mathsf{\Delta }}\times {\mathit{W}}_C^{\mathsf{\Delta }}=\left[\begin{array}{cccc}{w}_D^{\mathsf{\Delta }11}\times {\mathit{W}}_C^{\mathsf{\Delta }11}& w_D^{\mathsf{\Delta }12}\times {\mathit{W}}_C^{\mathsf{\Delta }12}& \cdots & w_D^{\mathsf{\Delta }1n}\times {\mathit{W}}_C^{\mathsf{\Delta }1n}\\ w_D^{\mathsf{\Delta }21}\times {\mathit{W}}_C^{\mathsf{\Delta }21}& w_D^{\mathsf{\Delta }22}\times {\mathit{W}}_C^{\mathsf{\Delta }22}& \cdots & w_D^{\mathsf{\Delta }2n}\times {\mathit{W}}_C^{\mathsf{\Delta }2n}\\ ⋮& ⋮& & ⋮\\ w_D^{\mathsf{\Delta }n1}\times {\mathit{W}}_C^{\mathsf{\Delta }n1}& w_D^{\mathsf{\Delta }n2}\times {\mathit{W}}_C^{\mathsf{\Delta }n2}& w_D^{\mathsf{\Delta }n3}\times {\mathit{W}}_C^{\mathsf{\Delta }n3}& w_D^{\mathsf{\Delta }nn}\times {\mathit{W}}_C^{\mathsf{\Delta }nn}\end{array}\right]$$

(22)

• Step 9: Obtain the rough influential weights.

Limit the rough weighted super matrix ${\overline{\mathit{W}}}_{\delta }$ by Equation (23) until the super matrix has converged and become stable.

$${\overline{\mathit{W}}}_{\ast }=\underset{\lambda \to \infty }{\lim }{({\overline{W}}_{\delta })}^{\lambda }$$

(23)

3.3. The COPRAS-R Method with Aspiration Level

Zavadskas et al. [53] proposed the COPRAS model in 1994, and this approach was used to solve the most appropriate alternative evaluation and selection problems [54,55]. To solve the uncertainty of information and characteristics in the performance data, a hybrid model combined the COPRAS with the gray system, and the model known as COPRAS-G was applied to evaluate the performance of alternatives [54,56]. However, the COPRAS-G adopted the “max–min” concept to set the “best–worst” for each attribute. That approach cannot replay the evaluation situation of the attributes in the alternatives, especially the best solution among all alternatives. Thus, Liou et al. [57] combined the concept of aspiration level to address this defect in the original method. However, the uncertainty degree of the gray number is defined based on the assumption [58]. To avoid this assumption, this study integrates the rough set theory and COPRAS method to construct the COPRAS-R model, which is used to establish the rough relative gap between the current levels and aspirational levels. The detailed steps of are as follows:

• Step 1: Build a rough decision matrix.

Experts give a score on each criterion of alternatives, ranging from 0 (very dissatisfied) to 100 (very satisfied) by the questionnaire. The rough decision matrix $\overline{\mathit{Q}}={\left[{\overline{q}}_{sj}\right]}_{o\times n}={\left[q_{sj}^{\nabla },q_{sj}^{\mathsf{\Delta }}\right]}_{o\times n}$ is obtained from the scores of alternatives by using Equations (1)–(8), in which $q_{sj}^{\nabla }$ and $q_{sj}^{\mathsf{\Delta }}$ respectively are the lower and upper limits for the sth alternative respective to the jth criterion.

• Step 2: Obtain an aspirated rough decision matrix.

Over the whole evaluation process, we clearly understand the limits of a scale i.e., “very dissatisfied (0)” to “very satisfied (100)”. For helping each alternative to catch the real gaps on each criterion, in this step, we combined an aspiration-level concept into the COPRAS-R method. That is, the positive and negative points are setting the 100 and 0, respectively. The normalized level is shown in Equation (24):

$$\begin{array}{l}\overline{\mathit{B}}=\left[{\overline{b}}_{sj}\right]=\left[b_{sj}^{\nabla },b_{sj}^{\mathsf{\Delta }}\right],\begin{array}{c}\end{array}s=1,2,\dots ,o;\hspace{0.17em}j=1,2,\dots ,n\\ b_{sj}^{\nabla }=\frac{q_{sj}^{\nabla }}{q_j^{\mathsf{\Delta }aspire}-q_j^{\mathsf{\Delta }worst}},\begin{array}{c}\end{array}{b}_{sj}^{\mathsf{\Delta }}=\frac{q_{sj}^{\mathsf{\Delta }}}{q_j^{\mathsf{\Delta }aspire}-q_j^{\mathsf{\Delta }worst}},\hspace{0.17em}{q}_j^{\mathsf{\Delta }aspire}=100;q_j^{\mathsf{\Delta }worst}=0\end{array}$$

(24)

• Step 3: Calculate the rough proximity degree of gray relation.

In the step, we can use Equations (25) and (26) to compute the rough proximity degree between the current level and aspiration level for alternatives.

$$\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\nabla })=\frac{\underset{s}{\min }\underset{j}{\min }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\nabla }|+\zeta \underset{s}{\max }\underset{j}{\max }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\nabla }|}{\hspace{0.17em}\hspace{1em}\hspace{1em}\hspace{1em}|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\nabla }|+\zeta \underset{s}{\max }\underset{j}{\max }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\nabla }|}$$

(25)

$$\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\mathsf{\Delta }})=\frac{\underset{s}{\min }\underset{j}{\min }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\mathsf{\Delta }}|+\zeta \underset{s}{\max }\underset{j}{\max }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\mathsf{\Delta }}|}{\hspace{0.17em}\hspace{1em}\hspace{1em}\hspace{1em}|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\mathsf{\Delta }}|+\zeta \underset{s}{\max }\underset{j}{\max }|b_j^{\mathsf{\Delta }aspire}-b_{sj}^{\mathsf{\Delta }}|}$$

(26)

where $\zeta$ is the adjusted coefficient (generally, the setting is 0.5) and the rough proximity degree of gray relation $\gamma ((b_j^{\mathsf{\Delta }aspire},b_{sj}^{\nabla });(b_j^{\mathsf{\Delta }aspire},b_{sj}^{\mathsf{\Delta }}))={\displaystyle \sum _{j=1}^n{w}_j}(\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\nabla });\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\mathsf{\Delta }}))$ can be divided $\gamma (b_{}^{\mathsf{\Delta }aspire},b_s^{\mathsf{\Delta }})$ $={\displaystyle \sum _{j=1}^n{w}_j}\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\mathsf{\Delta }})$ and $\gamma (b_{}^{\mathsf{\Delta }aspire},b_s^{\mathsf{\Delta }})={\displaystyle \sum _{j=1}^n{w}_j}\gamma (b_j^{\mathsf{\Delta }aspire},b_{sj}^{\mathsf{\Delta }})$, and aspiration levels of $b_{}^{\mathsf{\Delta }aspire}$ and $b_j^{\mathsf{\Delta }aspire}$ are all equal to 1 [59]. Also, the w_j is obtained from Equation (23).

• Step 4: Integrate the aspirated proximity index.

The aspirated proximity index H_s represents the de-roughness degree of satisfaction on each criterion for alternative s. The relative proximity H_s of the criteria are calculated as shown in Equation (27):

$$H_s^{}=\frac{1}{2}(\gamma (b_{}^{\mathsf{\Delta }aspire},b_s^{\nabla })+\gamma (b_{}^{\mathsf{\Delta }aspire},b_s^{\mathsf{\Delta }}))$$

(27)

• Step 5: Calculate the utility ratio for each alternative.

For each alternative, we can use Equation (28) to obtain the utility degree between $A_s^{aspired}$ relative significance and aspiration level, where $B_{}^{\mathsf{\Delta }aspire}$ is the aspired alternative. The computation is shown in Equation (28):

$$A_s^{aspired}=\frac{H_s^{}}{B_{}^{\mathsf{\Delta }aspire}}\times 100\%$$

(28)

4. Case Study

In this section, we apply the proposed hybrid model that combines RDANP with COPRAS-R to assess the performance of a research information system in a university in China, namely, Xiamen University of Technology (XMUT). We take XMUT as a case study, because this school represents a medium level among approximately 2600 universities in China, and can reflect the status and problems of scientific research management in most universities in China.

4.1. Identification Dimensions and Criteria for Evaluation of a Research Information System

As in prior discussions, we use the dimensions and indicators in the D&M model as elements to evaluate the scientific information system in this paper. We construct a scientific information evaluation system with four dimensions and 15 criteria that are extracted based on the D&M model. The results are displayed in

Table 1. Evaluation indicators for the scientific research information system. IS: information system.

Dimension	Criteria	Description
System quality (C1)	Ease of use (C11)	Does not require excessive professional guidance
	Integration (C12)	System function integration level
	Reliability (C13)	System robustness, few system crashes
	Response time (C14)	The reaction time after users make a request to the system
Information quality (C2)	Accuracy (C21)	Accuracy of the information delivered by the system
	Completeness (C22)	Integrity of the information supplied by the system
	Timelines (C23)	System information update speed
	Usefulness (C24)	Value of the information produced by the system
Service quality (C3)	Assurance (C31)	Frequency and effect of enterprise maintenance system
	IS training (C32)	Effect of training scientific research personnel
	Organization design (C33)	Service awareness and management improvement for system design
Intention to user (C4)	User frequency (C41)	Number of times the user uses the system
	Navigation patterns (C42)	How users access the system (computer or mobile phone)
	Effectiveness (C43)	Does accessing the system help to improve job performance?
	Efficiency (C44)	Does productivity increase after accessing the system?

. The system quality includes ease of use, integration, reliability, and response time, while information quality contains accuracy, completeness, timeliness, and usefulness. User frequency, navigation patterns, effectiveness, and efficiency are the four criteria in the dimension of intention to user. Table 1 also describes in detail the specific meaning of these 15 criteria.

4.2. Measuring the Relationship between Dimensions and Criteria by RDANP Method

To measure the initial direct influence matrix, we designed a questionnaire with the aim of obtaining the influential degrees between any two criteria according to Table 1. Ten staff members with abundant experience in this university were invited to judge the influential degrees between different criteria in terms of crisp scores 4, 3, 2, 1 and 0 (Extremely high impact = 4, High impact = 3, Medium impact = 2, Low impact = 1, and No impact = 0). The group was comprised of five professors from colleges, three managers from research departments, and two members from the information technology center.

During the survey, experts were asked to respond to a question by making pairwise comparisons of the degrees of influence between the criteria (the results are shown in

Table A1. The criteria (C₁₁) to other criteria of raw influence relation data from ten experts.

Criterion	No. 1	No. 2	No. 3	No. 4	No. 5	No. 6	No. 7	No. 8	No. 9	No. 10
C11–C11	0	0	0	0	0	0	0	0	0	0
C11–C12	3	3	3	3	3	3	4	3	3	3
C11–C13	2	3	3	3	2	3	3	2	3	1
C11–C14	3	3	4	4	4	3	4	4	4	3
C11–C21	1	2	0	0	3	3	2	1	3	1
C11–C22	4	1	1	1	2	3	3	1	3	1
C11–C23	2	1	4	1	3	3	1	2	2	2
C11–C24	3	0	1	1	0	3	3	1	2	1
C11–C31	4	3	4	4	3	3	4	3	3	1
C11–C32	4	3	4	3	3	3	4	2	2	1
C11–C33	4	3	2	3	4	3	4	2	1	0
C11–C41	0	1	2	3	3	3	1	3	3	2
C11–C42	2	2	0	0	1	2	1	0	0	0
C11–C43	0	2	3	1	3	1	1	2	2	1
C11–C44	1	1	2	2	3	1	1	2	2	1

). After integrating the questionnaire responses of the 10 experts, a 15 × 15 rough initial direct influence matrix was obtained according to step 1 of DANP. For example, an influential degree set of C₁₁ on C₁₂ is denoted by $C_{11}-C_{12}=\left\{3,3,3,3,3,3,4,3,3,3\right\}$, which can be converted into a rough number through Equations (1)–(8) as follows:

$$\underset{¯}{Lim}\left(3\right)=\frac{3+3+3+3+3+3+3+3+3}{9}=3,\begin{array}{l}\hfill \end{array}\overline{Lim}\left(3\right)=\frac{3+3+3+3+3+3+3+3+3+4}{10}=3.1$$

$$\underset{¯}{Lim}\left(4\right)=\frac{3+3+3+3+3+3+3+3+3+4}{10}=3.1,\begin{array}{l}\hfill \end{array}\overline{Lim}\left(4\right)=\frac{4}{1}=4$$

Then, $C_{11}-C_{12}=\left\{3,3,3,3,3,3,4,3,3,3\right\}$ can be transferred into a rough influential set as C₁₂–C₁₁ = {[3, 3.1], [3, 3.1], [3, 3.1], [3, 3.1], [3, 3.1], [3, 3.1], [3.1, 4], [3, 3.1], [3, 3.1], [3, 3.1]}.

The rough value of C₁₂–C₁₁ (i.e., m₁₂) is as follows:

$$\begin{array}{l}\mathsf{\Upsilon }\left(m_{12}\right)=\left[3.01,3.19\right]\\ =\left[\begin{array}{l}\frac{3+3+3+3+3+3+3.1+3+3+3}{10},\\ \frac{3.1+3.1+3.1+3.1+3.1+3.1+4+3.1+3.1+3.1}{10}\end{array}\right]\end{array}$$

The other rough values of elements in the rough original influential relationship matrix $\overline{\mathit{M}}=\left[{\overline{m}}_{ij}\right]$ are obtained in a similar way. The average rough initial direct influence matrix is shown in

Table 2. Rough original influence relationship matrix.

	C11	C12	C13	C14	C21	…	C44
C11	[0.00, 0.00]	[2.40, 3.17]	[2.71, 3.64]	[2.94, 3.65]	[2.12, 3.27]		[2.81, 3.78]
C12	[3.01, 3.19]	[0.00, 0.00]	[2.35, 3.06]	[2.48, 3.13]	[2.12, 3.27]		[3.25, 3.75]
C13	[2.13, 2.85]	[2.33, 3.62]	[0.00, 0.00]	[2.83, 3.37]	[0.74, 2.13]		[3.25, 3.75]
C14	[3.36, 3.84]	[2.47, 3.52]	[2.28, 3.48]	[0.00, 0.00]	[0.24, 0.98]		[2.75, 3.64]
C21	[0.86, 2.34]	[1.07, 2.71]	[1.33, 2.60]	[0.57, 1.62]	[0.00, 0.00]		[3.02, 3.76]
C22	[1.35, 2.68]	[1.59, 2.60]	[1.09, 2.46]	[0.38, 1.67]	[1.68, 3.42]		[2.87, 3.52]
C23	[1.52, 2.73]	[1.35, 2.06]	[1.65, 2.60]	[1.41, 2.35]	[1.95, 3.42]		[3.13, 3.85]
C24	[0.80, 2.23]	[2.12, 3.27]	[1.69, 2.51]	[0.78, 2.20]	[2.83, 3.75]		[3.16, 3.64]
C31	[2.71, 3.64]	[1.31, 2.87]	[2.61, 3.18]	[1.81, 2.99]	[1.54, 3.23]		[1.66, 3.14]
C32	[2.28, 3.48]	[1.12, 2.27]	[1.49, 2.31]	[0.52, 1.73]	[1.24, 2.32]		[2.36, 3.25]
C33	[1.68, 3.42]	[1.75, 3.55]	[1.90, 3.42]	[1.68, 3.42]	[2.63, 3.36]		[2.00, 3.00]
C41	[1.42, 2.72]	[1.49, 2.64]	[0.38, 1.67]	[0.75, 2.08]	[0.45, 2.25]		[2.24, 2.98]
C42	[0.30, 1.32]	[0.25, 1.43]	[0.16, 1.11]	[0.58, 2.21]	[0.16, 1.32]		[1.28, 2.48]
C43	[1.02, 2.19]	[1.40, 2.17]	[0.38, 1.67]	[0.54, 1.86]	[0.65, 1.94]		[2.75, 3.64]
C44	[1.24, 1.98]	[1.49, 2.31]	[0.66, 2.14]	[0.86, 2.34]	[0.75, 2.08]		[0.00, 0.00]

Note: The average gap ratio in consensus (%) $=\frac{1}{m\left(m-1\right)}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^m\left(\raisebox{1ex}{$\left|d_{ij}^s-d_{ij}^{s-1}\right|$}\!\left/ \!\raisebox{-1ex}{$d_{ij}^s$}\right.\right)}}\ast 100\%$ = 3.90% <5%, where m is the number of criteria (m = 15), and s is the sample of 10 experts (s = 10) whose practical experience and significant confidence reach 96.10% (more than 95%).

. In addition, the consensus rate of significant confidence was 96.10%, which is greater than 95% (i.e., gap error rate = 3.90%; less than 5%), as shown in the footnote in Table 2. The result shows a good consistency with acceptable reliability.

The total rough influence relationship matrix (

Table 3. Rough total influence relationship matrix.

	C11	C12	C13	C14	C21	…	C44
C11	[0.04, 0.24]	[0.08, 0.30]	[0.09, 0.29]	[0.09, 0.28]	[0.07, 0.29]		[0.11, 0.37]
C12	[0.10, 0.29]	[0.04, 0.23]	[0.08, 0.27]	[0.08, 0.26]	[0.07, 0.28]		[0.12, 0.35]
C13	[0.07, 0.27]	[0.08, 0.28]	[0.03, 0.20]	[0.08, 0.26]	[0.04, 0.24]		[0.11, 0.33]
C14	[0.10, 0.27]	[0.08, 0.26]	[0.07, 0.25]	[0.03, 0.18]	[0.03, 0.21]		[0.10, 0.31]
C21	[0.04, 0.25]	[0.05, 0.26]	[0.05, 0.24]	[0.03, 0.22]	[0.02, 0.20]		[0.10, 0.33]
C22	[0.05, 0.26]	[0.06, 0.25]	[0.04, 0.24]	[0.03, 0.22]	[0.06, 0.26]		[0.10, 0.32]
C23	[0.06, 0.27]	[0.06, 0.25]	[0.06, 0.25]	[0.05, 0.24]	[0.06, 0.27]		[0.11, 0.34]
C24	[0.04, 0.24]	[0.07, 0.26]	[0.06, 0.23]	[0.04, 0.22]	[0.08, 0.26]	…	[0.11, 0.32]
C31	[0.08, 0.27]	[0.05, 0.26]	[0.08, 0.25]	[0.06, 0.24]	[0.05, 0.26]		[0.08, 0.31]
C32	[0.07, 0.24]	[0.04, 0.22]	[0.05, 0.21]	[0.03, 0.19]	[0.04, 0.22]		[0.08, 0.28]
C33	[0.06, 0.29]	[0.06, 0.29]	[0.06, 0.28]	[0.06, 0.27]	[0.08, 0.28]		[0.09, 0.34]
C41	[0.05, 0.22]	[0.05, 0.22]	[0.02, 0.19]	[0.03, 0.19]	[0.02, 0.21]		[0.07, 0.27]
C42	[0.01, 0.15]	[0.01, 0.15]	[0.01, 0.13]	[0.02, 0.15]	[0.01, 0.14]		[0.04, 0.20]
C43	[0.04, 0.20]	[0.05, 0.20]	[0.02, 0.18]	[0.03, 0.18]	[0.03, 0.19]		[0.08, 0.27]
C44	[0.04, 0.21]	[0.05, 0.22]	[0.03, 0.20]	[0.03, 0.20]	[0.03, 0.21]		[0.03, 0.21]

) is calculated using Equation (11), and the values in the matrix indicate the total extent to which a criterion affects other criteria and other criteria affect it.

Table 4. The sum of rough influences given and received on dimensions and criteria.

	${\overline{\mathit{r}}}_{\mathit{i}}$	${\overline{\mathit{c}}}_{\mathit{i}}$	${\overline{\mathit{r}}}_{\mathit{i}}+{\overline{\mathit{c}}}_{\mathit{i}}$	${\overline{\mathit{r}}}_{\mathit{i}}-{\overline{\mathit{c}}}_{\mathit{i}}$		${\overline{\mathit{r}}}_{\mathit{i}}$	${\overline{\mathit{c}}}_{\mathit{i}}$	${\overline{\mathit{r}}}_{\mathit{i}}+{\overline{\mathit{c}}}_{\mathit{i}}$	${\overline{\mathit{r}}}_{\mathit{i}}-{\overline{\mathit{c}}}_{\mathit{i}}$
C1	[0.27, 1.07]	[0.21, 0.94]	[0.47, 2.01]	[−0.67, 0.86]	C11	[1.10, 4.36]	[0.87, 3.67]	[1.97, 8.03]	[−2.57, 3.50]
					C12	[1.15, 4.21]	[0.83, 3.65]	[1.98, 7.85]	[−2.50, 3.38]
					C13	[0.89, 3.92]	[0.74, 3.42]	[1.64, 7.33]	[−2.52, 3.17]
					C14	[0.90, 3.59]	[0.67, 3.30]	[1.57, 6.89]	[−2.40, 2.92]
C2	[0.22, 1.01]	[0.18, 0.95]	[0.41, 1.96]	[−0.73, 0.82]	C21	[0.79, 3.77]	[0.69, 3.52]	[1.48, 7.29]	[−2.73, 3.08]
					C22	[0.83, 3.79]	[0.69, 3.58]	[1.52, 7.37]	[−2.74, 3.10]
					C23	[0.95, 3.94]	[0.64, 3.41]	[1.60, 7.36]	[−2.46, 3.30]
					C24	[0.84, 3.68]	[0.70, 3.71]	[1.54, 7.40]	[−2.87, 2.98]
C3	[0.21, 0.99]	[0.15, 0.88]	[0.36, 1.87]	[−0.67, 0.83]	C31	[0.82, 3.75]	[0.63, 3.39]	[1.45, 7.15]	[−2.57, 3.12]
					C32	[0.69, 3.28]	[0.58, 3.32]	[1.27, 6.61]	[−2.63, 2.71]
					C33	[0.92, 4.15]	[0.51, 3.22]	[1.43, 7.38]	[−2.30, 3.64]
C4	[0.13, 0.77]	[0.28, 1.05]	[0.41, 1.82]	[−0.93, 0.49]	C41	[0.54, 3.15]	[1.22, 4.26]	[1.76, 7.41]	[−3.72, 1.93]
					C42	[0.24, 2.24]	[0.41, 2.52]	[0.66, 4.76]	[−2.28, 1.82]
					C43	[0.56, 2.97]	[1.27, 4.46]	[1.83, 7.43]	[−3.90, 1.70]
					C44	[0.56, 3.17]	[1.34, 4.55]	[1.89, 7.72]	[−4.00, 1.83]

shows the sum of the influences received and given among the dimensions and criteria, and the results are derived by implementing Equations (12) and (13) of steps 4 and 5 in the RDANP method. The values in the matrix shown in Table 4 represent the total and net impact values of the dimensions and criteria. The de-roughness matrix can be calculated by Equation (8) based on Table 4, which can be found in

Table 5. The de-roughness influences given and received on dimensions and criteria.

	${\mathit{r}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	${\mathit{r}}_{\mathit{i}}+{\mathit{c}}_{\mathit{i}}$	${\mathit{r}}_{\mathit{i}}-{\mathit{c}}_{\mathit{i}}$		${\mathit{r}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	${\mathit{r}}_{\mathit{i}}+{\mathit{c}}_{\mathit{i}}$	${\mathit{r}}_{\mathit{i}}-{\mathit{c}}_{\mathit{i}}$
C1	0.67	0.57	1.24	0.09	C11	2.73	2.27	5.00	0.47
					C12	2.68	2.24	4.91	0.44
					C13	2.41	2.08	4.48	0.33
					C14	2.25	1.98	4.23	0.26
C2	0.62	0.57	1.18	0.05	C21	2.28	2.10	4.38	0.17
					C22	2.31	2.13	4.44	0.18
					C23	2.45	2.03	4.48	0.42
					C24	2.26	2.21	4.47	0.06
C3	0.60	0.52	1.12	0.08	C31	2.29	2.01	4.30	0.28
					C32	1.99	1.95	3.94	0.04
					C33	2.54	1.87	4.41	0.67
C4	0.45	0.67	1.11	−0.22	C41	1.85	2.74	4.58	−0.89
					C42	1.24	1.47	2.71	−0.23
					C43	1.77	2.87	4.63	−1.10
					C44	1.86	2.94	4.80	−1.08

. The crisp value in Table 5 can be used to compare the impact of criteria among dimensions and under each dimension. Both of the largest (r_i − c_i) and (r_i + c_i) values are related to system quality, meaning that system quality not only has the greatest total impact of the four dimensions, but also has the most profound impact on the other three dimensions. Therefore, system quality affects information quality, service quality, and intention to use; furthermore, it is also affected by information quality and service quality, and is the key to the quality of the scientific research information system. Intention to use (C₄) has the smallest (r_i + c_i) value (1.11), and its (r_i − c_i) is negative (−0.22), meaning that it is greatly influenced by other factors and is the resulting element in the evaluation system. The influential network relationship map (INRM) (Figure 1) of the four dimensions and their respective subsystems can be drawn according to Table 3, Table 4 and Table 5. As shown in Figure 1, the arrow source represents the cause element, and is pointed to the result destination. System quality (C₁), information quality (C₂), and service quality (C₃) are the three main factors that affect use by users. From Figure 1, we can also see the most important criteria for each dimension. For example, organization design (C₃₃) affects IS training (C₃₁) and assurance (C₃₂); this shows that it is the most critical criterion in the dimension of service quality (C₃). Therefore, to maintain the enthusiasm of users in using the system, we need to improve the quality of systems, services, and information. Additionally, service quality and system quality have an important impact on information quality, and service quality and system quality show an interactive relationship.

4.3. Obtaining the Weights of Each Dimension and Criterion

The weights of the four dimensions and 15 criteria were calculated according to Equations (14)–(18).

Table 6. Rough un-weighted supermatrix.

	C11	C12	C13	C14	C21	…	C44
C11	[0.128, 0.213]	[0.333, 0.275]	[0.289, 0.267]	[0.356, 0.281]	[0.255, 0.260]		[0.286, 0.255]
C12	[0.285, 0.268]	[0.129, 0.215]	[0.298, 0.280]	[0.289, 0.274]	[0.279, 0.266]		[0.314, 0.261]
C13	[0.290, 0.262]	[0.269, 0.258]	[0.099, 0.197]	[0.261, 0.260]	[0.283, 0.250]		[0.192, 0.243]
C14	[0.297, 0.256]	[0.269, 0.252]	[0.314, 0.256]	[0.094, 0.184]	[0.183, 0.224]		[0.208, 0.241]
C21	[0.252, 0.248]	[0.247, 0.246]	[0.249, 0.242]	[0.166, 0.229]	[0.123, 0.202]		[0.234, 0.248]
C22	[0.235, 0.243]	[0.278, 0.255]	[0.247, 0.260]	[0.207, 0.256]	[0.272, 0.265]		[0.229, 0.248]
C23	[0.263, 0.247]	[0.231, 0.237]	[0.240, 0.235]	[0.343, 0.249]	[0.230, 0.250]		[0.260, 0.242]
C24	[0.250, 0.262]	[0.244, 0.262]	[0.265, 0.263]	[0.284, 0.266]	[0.375, 0.283]	…	[0.278, 0.262]
C31	[0.366, 0.340]	[0.315, 0.333]	[0.399, 0.346]	[0.379, 0.344]	[0.394, 0.347]		[0.361, 0.335]
C32	[0.304, 0.329]	[0.337, 0.332]	[0.292, 0.334]	[0.326, 0.335]	[0.360, 0.335]		[0.351, 0.335]
C33	[0.330, 0.331]	[0.348, 0.335]	[0.309, 0.320]	[0.295, 0.320]	[0.246, 0.319]		[0.288, 0.330]
C41	[0.292, 0.269]	[0.282, 0.270]	[0.276, 0.272]	[0.291, 0.266]	[0.284, 0.272]		[0.329, 0.295]
C42	[0.111, 0.168]	[0.095, 0.158]	[0.098, 0.158]	[0.118, 0.165]	[0.089, 0.155]		[0.094, 0.162]
C43	[0.288, 0.277]	[0.308, 0.284]	[0.312, 0.283]	[0.288, 0.278]	[0.323, 0.286]		[0.417, 0.307]
C44	[0.309, 0.286]	[0.315, 0.288]	[0.313, 0.286]	[0.302, 0.291]	[0.303, 0.288]		[0.161, 0.236]

indicates the unweighted rough supermatrix according to Equation (18), and the weighted rough supermatrix can be calculated according to Equations (19)–(23), which can be found in

Table 7. Rough weighted supermatrix.

	C11	C12	C13	C14	C21	…	C44
C11	[0.034, 0.051]	[0.088, 0.066]	[0.076, 0.064]	[0.094, 0.068]	[0.056, 0.063]		[0.070, 0.063]
C12	[0.075, 0.065]	[0.034, 0.052]	[0.078, 0.067]	[0.076, 0.066]	[0.061, 0.064]		[0.076, 0.064]
C13	[0.076, 0.063]	[0.071, 0.062]	[0.026, 0.047]	[0.069, 0.063]	[0.062, 0.060]		[0.047, 0.060]
C14	[0.078, 0.062]	[0.071, 0.061]	[0.083, 0.062]	[0.025, 0.044]	[0.040, 0.054]		[0.051, 0.059]
C21	[0.053, 0.061]	[0.052, 0.061]	[0.053, 0.060]	[0.035, 0.057]	[0.027, 0.049]		[0.048, 0.061]
C22	[0.050, 0.060]	[0.059, 0.063]	[0.052, 0.064]	[0.044, 0.063]	[0.061, 0.065]		[0.046, 0.061]
C23	[0.056, 0.061]	[0.049, 0.059]	[0.051, 0.058]	[0.072, 0.062]	[0.051, 0.061]		[0.053, 0.060]
C24	[0.053, 0.065]	[0.051, 0.065]	[0.056, 0.065]	[0.060, 0.066]	[0.083, 0.069]	…	[0.057, 0.065]
C31	[0.067, 0.080]	[0.058, 0.078]	[0.074, 0.081]	[0.070, 0.081]	[0.070, 0.080]		[0.082, 0.080]
C32	[0.056, 0.077]	[0.062, 0.078]	[0.054, 0.078]	[0.060, 0.079]	[0.064, 0.078]		[0.080, 0.080]
C33	[0.061, 0.078]	[0.064, 0.079]	[0.057, 0.075]	[0.054, 0.075]	[0.044, 0.074]		[0.065, 0.079]
C41	[0.099, 0.075]	[0.096, 0.075]	[0.094, 0.076]	[0.099, 0.074]	[0.108, 0.077]		[0.107, 0.079]
C42	[0.038, 0.047]	[0.032, 0.044]	[0.034, 0.044]	[0.040, 0.046]	[0.034, 0.044]		[0.031, 0.043]
C43	[0.098, 0.077]	[0.105, 0.079]	[0.106, 0.079]	[0.098, 0.077]	[0.123, 0.081]		[0.136, 0.082]
C44	[0.105, 0.079]	[0.107, 0.080]	[0.107, 0.080]	[0.103, 0.081]	[0.115, 0.081]		[0.052, 0.063]

. Subsequently, the de-rough influential weights of the dimensions and criteria can be found according to Equation (8). After obtaining the rough weight of the values of the dimensions and criteria, we get the crisp value by de-roughing the interval value, which can be found in

Table 8. The rough influential weights on dimensions and criteria.

	Local Weight	De-Roughness		Local Weight	De-Roughness	Global Weight	De-Roughness
C1	[0.054, 0.245]	0.150 (2)	C11	[0.283, 0.262]	0.273 (1)	[0.015, 0.064]	0.040 (7)
			C12	[0.275, 0.260]	0.267 (2)	[0.015, 0.064]	0.039 (8)
			C13	[0.228, 0.243]	0.235 (3)	[0.012, 0.060]	0.036 (12)
			C14	[0.214, 0.235]	0.224 (4)	[0.012, 0.058]	0.035 (14)
C2	[0.047, 0.249]	0.143 (3)	C21	[0.247, 0.247]	0.247 (3)	[0.012, 0.062]	0.037 (11)
			C22	[0.250, 0.251]	0.250 (2)	[0.012, 0.062]	0.037 (10)
			C23	[0.245, 0.241]	0.243 (4)	[0.012,0.060]	0.036 (13)
			C24	[0.258, 0.261]	0.259 (1)	[0.012, 0.065]	0.038 (9)
C3	[0.042, 0.231]	0.137 (4)	C31	[0.366, 0.342]	0.354 (1)	[0.015, 0.079]	0.047 (4)
			C32	[0.338, 0.334]	0.336 (2)	[0.014, 0.077]	0.046 (5)
			C33	[0.296, 0.324]	0.310 (3)	[0.012, 0.075]	0.044 (6)
C4	[0.073, 0.275]	0.174 (1)	C41	[0.283, 0.270]	0.277 (3)	[0.021, 0.074]	0.047 (3)
			C42	[0.099, 0.160]	0.129 (4)	[0.007, 0.044]	0.026 (15)
			C43	[0.300, 0.282]	0.291 (2)	[0.022, 0.078]	0.050 (2)
			C44	[0.318, 0.288]	0.303 (1)	[0.023,0.079]	0.051 (1)

Note: ( ) is ranking.

. We note that in Table 8, intention to use (C₄, 0.174) is the most important dimension, followed by system quality (C₁, 0.150), information quality (C₂, 0.143), and service quality (C₃, 0.137). Among the 15 criteria, efficiency (C₄₄, 0.051), effectiveness (C₄₃, 0.050), and frequency of use (C₄₁) are the three most important criteria.

4.4. Evaluating and Improving School Information System Using the COPRAS-R Method

We assessed the quality of a university scientific research information system using the COPRAS-R method, which is outlined in Section 3.3. Xiamen University of Technology’s research information system is chosen as a case study in this paper. Ten experienced university staff, who are referred to in Section 4.2, filled out the questionnaire. Satisfaction values ranged from 0 to 100, and experts scored the research information system based on the 15 criteria. The original evaluation scores of the research information system delivered by the 10 experts are indicated in

Table 9. The original evaluation scores of alternatives on 10 experts.

	No. 1	No. 2	No. 3	No. 4	No. 5	No. 6	No. 7	No. 8	No. 9	No. 10
C11	75	71	90	80	75	75	80	90	80	85
C12	70	75	85	85	75	75	60	80	90	80
C13	70	84	80	80	75	85	80	80	90	90
C14	60	77	76	75	70	80	50	90	95	95
C21	65	71	71	85	83	85	80	90	85	85
C22	70	88	62	90	85	85	71	80	85	85
C23	70	72	77	75	70	60	70	80	90	80
C24	70	76	80	80	70	85	70	80	85	95
C31	75	66	87	70	70	75	60	80	80	80
C32	60	62	88	70	81	60	50	70	80	80
C33	70	69	77	70	70	60	60	70	85	88
C41	70	80	70	80	70	70	30	70	70	80
C42	70	61	64	80	80	70	30	80	85	85
C43	70	70	77	85	80	70	70	80	90	90
C44	75	80	80	80	70	70	60	80	90	90

. The rough evaluation scores of the research information system with aspiration level can be calculated according to Equations (24)–(28), as shown in

Table 10. The rough evaluation scores of research information systems with aspiration level.

	Global Weight	Aspiration-Level	Rough Evaluation Scores	Relative Significance
C11	[0.015, 0.064]	[100, 100]	[76.054, 84.427]	0.030 (10)
C12	[0.015, 0.064]	[100, 100]	[71.487, 83.015]	0.029 (7)
C13	[0.012, 0.060]	[100, 100]	[77.298, 85.415]	0.027 (4)
C14	[0.012, 0.058]	[100, 100]	[65.851, 86.906]	0.026 (3)
C21	[0.012, 0.062]	[100, 100]	[74.431, 84.755]	0.027 (5)
C22	[0.012, 0.062]	[100, 100]	[73.702, 85.560]	0.028 (6)
C23	[0.012,0.060]	[100, 100]	[69.070, 80.022]	0.025 (2)
C24	[0.012, 0.065]	[100, 100]	[74.070, 84.447]	0.029 (8)
C31	[0.015, 0.079]	[100, 100]	[68.763 79.569]	0.033 (13)
C32	[0.014, 0.077]	[100, 100]	[61.654, 78.396]	0.031 (12)
C33	[0.012, 0.075]	[100, 100]	[66.126, 77.937]	0.030 (9)
C41	[0.021, 0.074]	[100, 100]	[62.271, 74.900]	0.031 (11)
C42	[0.007, 0.044]	[100, 100]	[58.679, 79.539]	0.018 (1)
C43	[0.022, 0.078]	[100, 100]	[73.076, 83.480]	0.036 (14)
C44	[0.023,0.079]	[100, 100]	[71.458, 83.187]	0.037 (15)

Note: ( ) is priority ranking for improvement.

. The value of relative significance indicates the priority for improvement. Therefore, the smaller the value of relative importance among the 15 criteria, the higher the ranking, and the greater the need for performance to be improved. According to Table 10, navigation patterns (C₄₂, 0.018), timeliness (C₂₄, 0.025), and response time (C₁₄, 0.026) are the three criteria that have smaller relative significance value than the others.

4.5. Discussion

We used the R-DEMATEL method to examine the causal relationship between indicators, as shown in Table 5. The INRM, which can show the causal relationships between any two dimensions and between any two criteria of each dimension, was plotted as shown in Figure 1. The INRM shows that system quality (C₁) has the largest (r_i + c_i) value and (r_i − c_i) value, and only intention to use (C₄) has a negative (r_i − c_i) value. The results indicate that system quality is the most important reason for influencing intention to use, and information quality (C₂) and service quality (C₃) also have a significant impact on intention to use (C₄). This result means that intention to use (C₄) is the result variable for the entire model, which reflects the usage of the information system, and it is the value of user satisfaction with system performance. The ease of use (C₁₁), timeliness (C₂₃), and organization design (C₃₃) are the three main elements of the respective subsystems, according to Figure 1. Therefore, if we want to improve the frequency of use (C₄₁), effectiveness (C₄₃), and efficiency (C₄₄) of the information system, improving system usability, reducing system response time, and optimizing system design are objectives worth considering.

The R-DEMATEL-based ANP approach was applied to find the weights of the dimensions and criteria. From Table 8, intention to use (C₄) has the greatest weight, followed by system quality (C₁), information quality (C₂), and service quality (C₃). These results are consistent with the results of RDEMATEL analysis, and system quality (C₁) and intention to use (C₄) are the two priorities in the evaluation information system. Considering the criteria, efficiency (C₄₄, 0.051), effectiveness (C₄₃, 0.050), and frequency of use (C₄₁, 0.047) are the three most important criteria, followed by assurance (C₃₁, 0.047), IS training (C₃₂, 0.046), and organization design (C₃₃, 0.044). From their point of view, users are highly concerned with the effectiveness and efficiency of the system, which play a major role in user satisfaction with the system. In addition, service quality is becoming increasingly important for information system products. Users care whether the system design can fully meet their needs and pay attention to the developer’s service commitment and careful training.

As shown in Table 10, efficiency (C₄₄) and effectiveness (C₄₃) ranked in the top two in terms of relative significance value. This result reflects that the research information system of XMUT can improve the efficiency of teachers and researchers. Most teachers are quite satisfied with the effectiveness and efficiency of the system. As the informatization of Chinese universities deepens, the use of information systems has become a trend. University staff must change the way they work using the internet to improve work efficiency and effectiveness. The navigation pattern (C₄₂) is the ranking with a relative significance value. As with the development of mobile internet technology, users want to access the system through mobile terminals. However, the speed of development of mobile APPs in Chinese universities is slow, and cannot meet the needs of teachers and researchers. Timelines (C₂₃) and response time (C₁₄) are ranked lower in the performance evaluation of this system. As we know, users are quite disgusted with the slow response of the system and slow updating of content. If users find that the system is not responding, they are likely to give up on using the system. Therefore, improving the response speed of the system itself and speeding up the content updating process are important prerequisites for satisfying users. Assurance (C₃₁), IS training (C₃₂), and organization design (C₃₃) are urgently needed to improve performance, according to Table 10. The service quality of the system has long been a problem in the construction of information technology in Chinese universities. Many system developers focus on improving system performance while ignoring user guidance and training. Therefore, the poor quality of service is also an important reason for user dissatisfaction with the system.

5. Conclusions

Different from previous papers that used statistical perspectives to verify the D&M model, this paper applied a hybrid MCDM approach that combined RDANP with the COPRAS-R method to evaluate the performance of specific information systems based on the D&M model. We selected four dimensions and 15 criteria. These dimensions include system quality, information quality, service quality, and intension to use. We designed two surveys, one to measure the influential degrees between dimensions and the criteria for each dimension, and the other to calculate the user satisfaction for each criterion on a specific system. We chose a research information system in a Chinese university as a case. Ten experienced university teachers or research managers were invited to interview and fill out two questionnaires.

The results demonstrate that intention to use (C₄) and system quality (C₁) are the two most significant indicators for the evaluation of the quality of research information systems for universities in China, making up 60% of the total weight, and system quality (C₁) is the main factor affecting the intension to use (C₄). Therefore, ease of use (C₁₁), integration (C₁₂), reliability (C₁₃), and response time (C₁₄) have an important influence on the frequency of use (C₄₁), effectiveness (C₄₂), and efficiency (C₄₃) of the system. This study offers several contributions to the literature. First, this paper attempts to apply the D&M model to the university research platform using the hybrid MCDM method, and fills the gaps in the current research using confirmatory statistical analysis without analysis from expert decision management. Second, we applied the R-DANP approach, which analyzes the causal relationship between indicators and obscures the subjective limitations of experts to obtain the weights of the dimensions and criteria. Taking the scientific research system of Chinese universities as a case, we proposed corresponding strategies to improve the performance of scientific research systems.

Certain limitations exist in using the R-DANP and COPRAS-R models. First, we interviewed 10 experienced university employees from three different sections in depth. We cannot claim that the opinions of these 10 experts are absolutely authoritative, although the average gap ratio in consistency is smaller than 5% (3.9%). Second, experts are more or less biased when they analyze problems. We used rough numbers to remedy this issue, and we cannot infer that this approach is better than fuzzy or gray relational technologies. Finally, the empirical data are limited to the research information system of a university in China, and these research findings might vary for other areas and countries. In future research, the proposed model could also be used in similar decision-making problems in other fields.