A Rough Hybrid Multicriteria Decision-Making Model for Improving the Quality of a Research Information System

Improving the quality of research information systems is an important goal in the process of improving the performance of research management in Chinese universities. Since the evaluation of information system (IS) quality is a multicriteria decision problem, it is critical to identify the interrelationships among the dimensions and criteria, and decide on the important criteria for proposed improvement strategies. This paper suggests a hybrid multicriteria decision-making (MCDM) model for improving the quality of a research information system. First, a rough method combined with the decision-making trial and evaluation laboratory and analytical network process (rough DANP) model is used to improve the objectivity of expert judgements. Additionally, the rough DANP can be used to construct an influential network relationship map (INRM) between research information system components to derive the criterion weights. The complex proportional assessment of alternatives with rough numbers (COPRAS-R) is applied to evaluate the performance of the research information system. A Chinese university research information system is chosen to illustrate the usefulness of the proposed model. The results show that efficiency, effectiveness, and user frequency have the highest priorities for improvement. Selected management implications based on the actual case study are supplied.


Introduction
Currently, the increasing speed of scientific and technological achievements is crucial to improvements in innovation ability.The university research platform is a major focus of a country's innovation-driven development.As an important component of the national scientific research system, research in colleges and universities plays a highly important role in the development of science and technology and economic construction.Enterprises, universities, scientific research institutions, and other departments have established online scientific research information systems to offer a service platform for technology research and development.For example, Slovenia has established a network of scientific research cooperation since 1960, and scientists are looking for partners through the network [1].Through an analysis of scientists' output data from information systems in Slovenia over the past 40 years, Perc found that researchers' output is consistent with Zipf's law and log-normal distributions [2].Lužar et al. (2014) studied the interdisciplinary nature of the research community in the scientific network, and they found that the number of cooperation between different professional disciplines is growing slowly and needs further stimulation [3].A well-designed information system needs to address issues such as system stability and user feedback [4].However, colleges and universities are slow to develop their uses of these scientific research systems in China.An excellent research information system could act as an effective method that aids university administrators in managing the achievements of researchers and teachers.Researchers and teachers can obtain and share important information using the research system, and arrange scientific research activities in an orderly manner.Therefore, the establishment of a high-performance university research information system is essential for the overall improvement of university research.
Information on research tasks and projects must be available in less time and with greater reliability, and the content in the system must be accurate and useful.Therefore, the quality of the research information system decides the satisfaction degree of the users.The evaluation of the quality of the information system is a multidimensional problem.Many researchers have offered ideas and advice for assessing the quality of an information system.One of the classic information system models is the information systems success model developed by Delone and Mclean [5,6], which is also known as the information system (IS) success model or D&M model.This model identified and explained six of the most critical dimensions on which information systems are commonly evaluated, namely, information quality, system quality, service quality, use intention, user satisfaction, and net system benefit.
Based on the D&M IS success model, certain researchers updated and improved this model in different types of information systems.Pitt et al. [7] argued that service quality is a highly important component in the success of a D&M model, and added service quality as a new variable to reflect the quality of the IS model.Seddon et al. [8] thought that net benefit could replace the function of individual impact and organizational impact.Yeo [9] updated the D&M model by changing "Use" to "Intention to use | Use".In addition to updating the model, academia also applied the D&M model to different industry information systems.Zheng et al. [10] argued that information quality and system quality have a strong influence on individual benefits and user satisfaction in a virtual community.Azeroual et al. [11] suggested that the success of a research information system is largely related to the quality of the available data, and they improved the quality of the research information system via data profiling.Dwivedi et al. [12] summarized the factors that determine the success and failures of the information systems, and found that the type of information systems, such as enterprise resource planning (ERP) e-government, and the degree of implementation by employees and management are the main factors.Grudzie ń and Hammrol [13] suggested that the quality of the information is an important component of information system quality management.These statistical models contributed to exploration of the interrelationships among variables.However, due to the limitations of statistical methods, these studies encountered difficulty in helping decision makers understand the gaps between the current performance and the aspirational level.
To solve this problem, certain researchers used multiple criteria decision-making (MCDM) models to evaluate the information system.For example, Lin [14] applied fuzzy Analytics Hierarchy Process (AHP) to evaluate course website system quality.Hsu and Lee [15] used the decision-making trial and evaluation laboratory (DEMATEL) method to explore the critical factors that influence the quality of blog interfaces.Tsai et al. [16] developed an effectiveness evaluation model for web-based marketing using DEMATEL, analytic network process (ANP), and VlseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR).Su et al. [17] developed a hybrid fuzzy MCDM model for improving e-learning innovation performance in a fuzzy environment.These MCDM models solved the evaluation or improvement problems for the service quality of an information system, but they did not consider the group summary of expert opinions, and the D&M model was not considered in their studies.Furthermore, none of the literature discusses the relationship of the factors that influence the success of an information system, and researchers have seldom proposed methods for exploring the importance of those factors.This paper attempts to fill this gap by proposing a causal relationship framework for the development of research information system assessment capability.First, based on the D&M IS success model, this study proposes a hybrid rough MCDM model that combines rough concepts and the DEMATEL-based ANP model (rough DANP, or RDANP) to construct interdependent connections among the assessment dimensions and criteria, and obtain the weights of each dimension and criterion.Second, based on the results of the use of the RDANP model, a complex proportional assessment of alternatives with the rough numbers method (COPRAS-R) is applied to evaluate the performance of a research information system in China as a case study.The D&M model is easy to understand by confirming the relationship on each category and criteria.The derived influential network relationship map (INRM) can help decision makers understand the complex relationships in the research information system.The weights of the information system assessment factors can be calculated using the rough DANP method.The strength of this hybrid model can be used to show the cause-and-effect relationships and obtain the influence weights of factors within the research information system.Consequently, differing from existing studies that focused on finding the influence factors, this paper contributes to the literature by attempting to construct a cause-and-effect method for a research information system that could help to identify the key factors for the research information system and supply directions for improvement.

Literature Review
Based on the previous decision-making research on the service quality of information systems, it is common to apply statistical methods to verify the application of the D&M models in various fields.Section 2.2 introduces the MCDM method to evaluate the performance of information systems, followed by an explanation of research gaps in Section 2.3.[5] proposed a famous information system success model known as the D&M model that has the ability to measure system standards, and this model has been theoretically and empirically developed and improved in several studies [18,19].For example, Bernroider [20] applied the D&M model to measure the effectiveness of adopting ERP in different sizes of companies from the perspective of information technology.Correlation analysis and principal component analysis were used in this paper, and the researchers found that the D&M success model was well suited to assessment of the performance of application of the ERP system in small-and medium-sized companies.Chen et al. [21] evaluated the quality of a web-based learning system for nurses in a hospital and found that system quality, information quality, and service quality are the three dimensions used to measure the quality of a learning system.Two-factor analysis and SEM methods were applied to validate the model practicality, and this research developed an instrument used to assess the quality of a web-based learning system based on the D&M model.By applying the updated D&M model, Bossen et al. [22] satisfactorily evaluated the implementation of comprehensive electronic health records (EHR) in a hospital in Denmark.Bossen found that medical secretaries and physicians encounter certain difficulties in using the patient administration method and establishing professionally relevant data.Kyoung et al. [23] also evaluated the performance of a public hospital information system using the D&M model, and the relationships among IS success factors were assessed using the structural equation model.The results indicated that the intention to use the system had no influence on the net benefit.Chiu et al. [24] evaluated mobile e-books in a cloud bookcase using an information system success model and applied partial least squares structural equation modeling (PLS-SEM) to verify the model.The result showed that system quality and service quality had a positive influence on intention to use the cloud e-bookcase and user satisfaction.Ayyash [18] developed four dimensions (accuracy, timeliness, completeness, and relevancy) used to reflect information quality, and discussed the relationship between these four dimensions and customer satisfaction in the e-banking industry.

DeLone and McLean
Ayyash used multiple regression to analyze the data, and the results indicated that all of the dimensions had positive effects on customer satisfaction.
Selected researchers applied the D&M model in online learning systems such as Massive Open Online Courses (MOOCs), mobile learning systems, and blog-based learning systems [25,26].Based on the D&M theory, Wu and Chen [27] applied the structure equation model and multiple group analysis method to a Facebook system, and the result showed that social influence and information quality are critical and direct determinants that affect users' continuous intention to use Facebook in learning.E-commerce and e-government system success could also be measured by the D&M model from a statistical perspective [28][29][30].
The above-mentioned literature used statistical techniques such as multiple regression or factor analysis to analyze the application effects of D&M models in various industries.Certain researchers attempted to find causal relationships among the variables.The results showed that the conclusions were more or less different when the D&M model was applied in different contexts.Most results indicated that system quality, service quality, and information quality had a positive influence on user satisfaction and intention to use the system.However, the most significant factors affecting the user satisfaction and the interrelationship among all dimensions and criteria are seldom discussed.

MCDM Models
The information system has become a common practice in various industries and organizations, and has helped to improve operational efficiency through the application of an information system.Many organizations have currently established their own information systems by outsourcing to third-party companies, purchasing existing systems, or self-development.The question of how to evaluate the effectiveness and quality of an information system is a problem faced by users.In practice, the purchaser of an information system organizes experts to appraise the system quality.However, assessing the quality of the information systems requires multiple perspectives, and is a multicriteria decision-making management problem.The MCDM method is a practical and academic methodology that offers a systematic modeling framework and methodology under multiple measurement indicators.This method is used to simultaneously consider multiple criteria to supply a decision-making optimization and decision-making model for decision makers [31].
Thus far, a lack of research persists on the goal of integrating the interdependencies to improve information system performance.To identify the interrelationships among indicators of information systems, the DEMATEL approach is used to construct a network structure map with interdependent relationships [32][33][34][35].A hybrid MCDM model that combines DEMATEL and the ANP methods can show the cause-and-effect relationships and also obtain the influence factors in systems.Therefore, this method was used to construct interdependent connections among indicators [36][37][38][39].However, these two methods often ignore the subjective defects of decision makers.As a result, updated models were proposed by certain researchers.Bai and Sarkis [40] introduced a gray-based DEMATEL model that can be used to solve uncertainty problems for evaluating critical success factors in business process management.The fuzzy-based DEMATEL method was proposed to evaluate green supply chain management (SCM) [41,42].Compared with gray and fuzzy technology, rough technology does not need to assume a semantic scale.Rough DEMATEL establishes scales based on the values filled out by experts.Therefore, the semantic scale of each question is not fixed.Song and Cao [43] proposed a rough DEMATEL-based approach for assessing the internal relationship between requirements of the product-service system.This study adopts a flexible rough interval, which better reflects the subjective judgment and fuzzy judgment of decision makers and successfully adapts to DEMATEL in an uncertain environment.Pamučar et al. [44] defined a hybrid DEMATEL-ANP-Multi-Attributive Ideal-Real Comparative Analysis (MAIRCA) model based on interval rough numbers to solve uncertainty in the multicriteria decision-making process, and this model showed high stability with respect to changes in the nature and characteristics of the criteria.

Research Gaps
As a well-known model in the field of information systems, D&M is widely used in both theoretical and practical fields.Most studies have focused on verifying the relevance and causality of the D&M model dimensions in different domains from a statistical perspective.However, the question of how to improve the quality of information systems has not been well resolved in theory and practice.As a powerful tool for decision-making in the field of decision management, MCDM can supply a scientific and rational strategy for improving the quality of information systems based on the D&M model.This paper offers ideas for studying the scientific nature of D&M from the perspective of management science.As discussed in the introduction, although selected researchers evaluated information by applying the AHP, ANP, and DAMATEL methods, they seldom considered the ambiguity of the opinions from expert groups, and the D&M model was also rarely used in their papers.Based on the above analysis, this study applies the rough DEEMATEL and the rough DANP methods to explore the network relationship of the criteria and the weights of the criteria in the research information system.Based on the result of rough DANP, the COPRAS-R is used to assess the performance of a research information system in China as a case study.

Methodology
In this section, we introduce the RMADM model that combines RDANP with COPRAS-R to establish the interdependent structure, get the weights of dimensions and criteria, and assess the aspiration gap of alternative criteria.Schools can figure out the complex relationship between information system and the cause-effect within the service criteria through the derived rough influence-network relation map (RINRM).The obtained influential weights of the criteria and the RINRM can help schools and the information system set the improved priorities for bettering the school information system in schools.The detailed procedures are illustrated as follows.

The Rough Number
The rough set theory was developed by Pawlak in 1982 [45] for solving a group of subjective and imprecise expert perceptions.Zhai et al. [46] used rough numbers, a set of interval values that are clearly transformed from a crisp group, to show subjectivity and imprecise human thinking in a multiple-attribute decision-making (MADM) environment.Since the application of rough number theory to address decision problems does not require any auxiliary information, the theory is better able to capture the perception of experts [47][48][49].The detailed steps are as follows: Step 1: Conform lower and upper approximations of rough number for each crisp scale.
U is a set of the universe that contains all the respondents with multi-attributes, and Θ is a set of z class under respondents.R = {z 1 , z 2 , . . ., z Θ } are ordered in the manner of z 1 < z 2 < . . .< z Θ and Y is an arbitrary respondent of U. The lower approximation and upper approximation of z Θ can be defined as [46]: Lower approximation: Upper approximation: Step 2: Compute the interval value of the rough number.
A group of expert opinions can be expressed by Lim(z Θ ), Lim(z Θ ) , which are derived by: Symmetry 2019, 11, 1248 6 of 21 where x i and y i are denoted the elements in the lower approximation or upper approximation of z Θ , respectively.In addition, N L and N U are represented the total number of respondents involved in the lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ(z Θ ), as shown in Equation ( 4 where xi and yi are denoted the elements in the lower approximation or upper approximation of z Θ , respectively.In addition, NL and NU are represented the total number of respondents involved in the lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ (z Θ ), as shown in Equation ( 4): Assuming two rough numbers ( ) α ϒ and ( ) and μ as a nonzero constant, the arithmetic operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM.The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained.The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge in the real environment.The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "no influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by mij.We checked the consistency of the raw data, using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is the number of the criteria, k is the number of the expert, k ij d is the average direct influence value of criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that the expert's rating is consistent [48].Then, the rough direct relation average matrix M is acquired through Equations ( 1)-( 8) in the k matrices for the respondents.Finally, we can obtain a rough  , where is n the number of criteria.
Step 2: Obtain the rough initial influence relationship matrix P = [ ij p ]nxn, which is the multiplication of M and v.
Assuming two rough numbers Symmetry 2019, 11, x FOR PEER REVIEW 6 where xi and yi are denoted the elements in the lower approximation or upper approximation of respectively.In addition, NL and NU are represented the total number of respondents involved in lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a se rough numbers, Υ (z Θ ), as shown in Equation ( 4): and μ as a nonzero constant, the arithm operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteri alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between fac and construct an INRM.The rough total influence relationship matrix is taken as the input of rough DANP method, and the rough influence weights of the criteria are obtained.The DA approach solves the unreasonable problem of assuming that each cluster has equal weight in the A method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledg the real environment.The rough degrees of influence of each dimension can be obtained by the rou DEMATEL, and the rough DANP process can be applied to normalize the unweighted super ma and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough di influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw d using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is number of the criteria, k is the number of the expert, k ij d is the average direct influence valu criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that expert's rating is consistent [48].Then, the rough direct relation average matrix M is acqui through Equations ( 1)-( 8) in the k matrices for the respondents.Finally, we can obtain a rou  , where is n the number of criter Step 2: Obtain the rough initial influence relationship matrix P ]nxn, which is the multiplication of and v. where xi and yi are denoted the elements in the lower approximation or upper approx respectively.In addition, NL and NU are represented the total number of respondents i lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converte rough numbers, Υ (z Θ ), as shown in Equation ( 4): and μ as a nonzero constant, operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analys alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship be and construct an INRM.The rough total influence relationship matrix is taken as th rough DANP method, and the rough influence weights of the criteria are obtaine approach solves the unreasonable problem of assuming that each cluster has equal weig method [38,50,51].In addition, we also consider the ambiguity of the human domain the real environment.The rough degrees of influence of each dimension can be obtaine DEMATEL, and the rough DANP process can be applied to normalize the unweighted and obtain the rough influential weights of criteria.The detailed steps are as follows [5 Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ra influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of th influence between two pairwise criteria, denoted by mij.
We checked the consistency o using the formula as follows: The average gap-ratio in consensus (%) ( ) number of the criteria, k is the number of the expert, k ij d is the average direct influ criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we b expert's rating is consistent [48].Then, the rough direct relation average matrix M through Equations ( 1)-( 8) in the k matrices for the respondents.Finally, we can o  , where is n the numb Step 2: Obtain the rough initial influence relationship matrix P ]nxn, which is the multi and v.
(β) and µ as a nonzero constant, the arithmetic operations of rough number can be followed by: where xi and yi are denoted the elements in the lower approximation or upper approximation of z respectively.In addition, NL and NU are represented the total number of respondents involved in th lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set rough numbers, Υ (z Θ ), as shown in Equation ( 4): Assuming two rough numbers and μ as a nonzero constant, the arithmet operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between facto and construct an INRM.The rough total influence relationship matrix is taken as the input of th rough DANP method, and the rough influence weights of the criteria are obtained.The DAN approach solves the unreasonable problem of assuming that each cluster has equal weight in the AN method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge the real environment.The rough degrees of influence of each dimension can be obtained by the roug DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matr and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "n influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough dire influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw dat using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is th number of the criteria, k is the number of the expert, k ij d is the average direct influence value criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that th expert's rating is consistent [48].Then, the rough direct relation average matrix M is acquire through Equations ( 1)-( 8) in the k matrices for the respondents.Finally, we can obtain a roug  , where is n the number of criteria where xi and yi are denoted the elements in the lower approximation or upper approximation of z Θ , respectively.In addition, NL and NU are represented the total number of respondents involved in the lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ (z Θ ), as shown in Equation ( 4): Assuming two rough numbers and μ as a nonzero constant, the arithmetic operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM.The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained.The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge in the real environment.The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "no influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw data, using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is the number of the criteria, k is the number of the expert, k ij d is the average direct influence value of criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that the expert's rating is consistent [48].Then, the rough direct relation average matrix M is acquired where xi and yi are denoted the elements in the lower approximation or upper approximation respectively.In addition, NL and NU are represented the total number of respondents involved lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a rough numbers, Υ (z Θ ), as shown in Equation ( 4): and μ as a nonzero constant, the arith operations of rough number can be followed by: Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for crit alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between f and construct an INRM.The rough total influence relationship matrix is taken as the input rough DANP method, and the rough influence weights of the criteria are obtained.The D approach solves the unreasonable problem of assuming that each cluster has equal weight in the method [38,50,51].In addition, we also consider the ambiguity of the human domain knowle the real environment.The rough degrees of influence of each dimension can be obtained by the DEMATEL, and the rough DANP process can be applied to normalize the unweighted super m and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging fro influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m number of the criteria, k is the number of the expert, k ij d is the average direct influence va criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe th expert's rating is consistent [48].Then, the rough direct relation average matrix M is acq Symmetry 2019, 11, x FOR PEER REVIEW 6 of 21 where xi and yi are denoted the elements in the lower approximation or upper approximation of z Θ , respectively.In addition, NL and NU are represented the total number of respondents involved in the lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ (z Θ ), as shown in Equation ( 4): Assuming two rough numbers and μ as a nonzero constant, the arithmetic operations of rough number can be followed by: ( ) ( ) ( ) ( ) ( ) Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM.The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained.The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge in the real environment.The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "no influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw data, using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is the number of the criteria, k is the number of the expert, k ij d is the average direct influence value of where xi and yi are denoted the elements in the lower approximation or upper approximation respectively.In addition, NL and NU are represented the total number of respondents involved lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a rough numbers, Υ (z Θ ), as shown in Equation ( 4): and μ as a nonzero constant, the arith operations of rough number can be followed by: ( ) Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for crit alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between f and construct an INRM.The rough total influence relationship matrix is taken as the input rough DANP method, and the rough influence weights of the criteria are obtained.The D approach solves the unreasonable problem of assuming that each cluster has equal weight in th method [38,50,51].In addition, we also consider the ambiguity of the human domain knowle the real environment.The rough degrees of influence of each dimension can be obtained by the DEMATEL, and the rough DANP process can be applied to normalize the unweighted super and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging fro influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough influence between two pairwise criteria, denoted by mij.
We checked the consistency of the raw using the formula as follows: The average gap-ratio in consensus (%) ( ) , where m number of the criteria, k is the number of the expert, k ij d is the average direct influence va Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM.The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained.The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge in the real environment.The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "no influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by m ij .We checked the consistency of the raw data, using the formula as follows: The average gap-ratio in consensus (%) = , where m is the number of the criteria, k is the number of the expert, d k ij is the average direct influence value of criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that the expert's rating is consistent [48].Then, the rough direct relation average matrix M is acquired through Equations ( 1)- (8) in the k matrices for the respondents.Finally, we can obtain a rough original influence relationship , where is n the number of criteria.
Step 2: Obtain the rough initial influence relationship matrix P = [p ij ] nxn , which is the multiplication of M and v.
Step 3: Calculate the rough total influence relationship matrix T with Equation (11).The element t ij indicates the rough interdependent effects that criteria i has on criteria j, where I is an identity matrix.
Step 4: Derive each column sum (c i ) and row sum (r i ) from the rough total influence relationship matrix T as follows: The element c j denotes the rough total effects by criterion j, from the other criteria.Similarly, r i represents the rough direct and indirect effects of criterion i on the other criteria.
Step 5: Get the RINRM for whole evaluation model.Thus, r i + c j reflects the strength of the influences given and received on criterion i, while r i − c j shows the net effect of criterion i on the other criteria.Clearly, if r i − c j is positive, criterion i is a causal component, and if r i − c j is negative, then criterion i is an affected component.As a result, the rough influence network relationship map (RINRM) can be finished by mapping the data set (r i + c j ,r i − c j ).
Step 6: Derive rough total influence relationship matrix T C based on the criteria and T D based on the dimensions.
The rough matrix T could be differentiated into T C based on the criterion and T D based on the dimensions.The matrix T C modularizes the matrix T according to the dimensions, which are essentially the same matrix.The rough matrix T D is found by averaging the rough degree of criterion influence in each dimension.As shown in Equation ( 14), D i denotes the ith dimension; c ij denotes the jth criteria in the ith dimension.For example, we get a crisp value by averaging T mn c , where T mn c means the extent to which the criteria in the mth dimension affect the criteria in the nth dimension.Then, we get the T D by averaging the T mn c in the Step 3: Calculate the rough total influence relationship matrix T with Equation (11).The element ij t indicates the rough interdependent effects that criteria i has on criteria j, where I is an identity matrix.Step 4: Derive each column sum ( i c ) and row sum ( i r ) from the rough total influence relationship matrix T as follows: ( ) ( ) The element j c denotes the rough total effects by criterion j, from the other criteria.Similarly, i r represents the rough direct and indirect effects of criterion i on the other criteria.
Step 5: Get the RINRM for whole evaluation model.
Thus, i j r c + reflects the strength of the influences given and received on criterion i, while  ,

T T T T T T T TT T T T (14)
Step 7: Obtain the rough unweighted supermatrix.
We get a rough matrix C δ T by normalizing C T . ( Step 7: Obtain the rough unweighted supermatrix., , as follows: The rough unweighted supermatrix W is transposed from rough matrix C δ T , as shown in Equation (18).
Referring to step 5, we can get rough matrix , by averaging the degree of the criterion influence in each dimension, such as , which is derived by Equations ( 19)-( 21).
where in is the number of criteria in dimension n; and jm is the number of criteria in the dimension m. , as follows: The rough unweighted supermatrix W is transposed from rough matrix T δ C , as shown in Equation (18)., , as follows: The rough unweighted supermatrix W is transposed from rough matrix C δ T , as shown in Equation (18).
Referring to step 5, we can get rough matrix , by averaging the degree of the criterion influence in each dimension, such as , which is derived by Equations ( 19)-( 21).
where in is the number of criteria in dimension n; and jm is the number of criteria in the dimension m. ( Step 8: Derive the rough weighted supermatrix.
Referring to step 5, we can get rough matrix T D = T ∇ D , T ∆ D by averaging the degree of the criterion influence in each dimension, such as , which is derived by Equations ( 19)- (21).
Symmetry 2019, 11, 1248 9 of 21 where i n is the number of criteria in dimension n; and j m is the number of criteria in the dimension m.
Then, we can obtain the rough weighted supermatrix , which is shown in Equation (22).
Step 9: Obtain the rough influential weights.
Limit the rough weighted super matrix W δ by Equation ( 23) until the super matrix has converged and become stable.

The COPRAS-R Method with Aspiration Level
Zavadskas et al. [53] proposed the COPRAS model in 1994, and this approach was used to solve the most appropriate alternative evaluation and selection problems [54,55].To solve the uncertainty of information and characteristics in the performance data, a hybrid model combined the COPRAS with the gray system, and the model known as COPRAS-G was applied to evaluate the performance of alternatives [54,56].However, the COPRAS-G adopted the "max-min" concept to set the "best-worst" for each attribute.That approach cannot replay the evaluation situation of the attributes in the alternatives, especially the best solution among all alternatives.Thus, Liou et al. [57] combined the concept of aspiration level to address this defect in the original method.However, the uncertainty degree of the gray number is defined based on the assumption [58].To avoid this assumption, this study integrates the rough set theory and COPRAS method to construct the COPRAS-R model, which is used to establish the rough relative gap between the current levels and aspirational levels.The detailed steps of are as follows: Step 1: Build a rough decision matrix.
Experts give a score on each criterion of alternatives, ranging from 0 (very dissatisfied) to 100 (very satisfied) by the questionnaire.The rough decision matrix Q = q sj o×n = q ∇ sj , q ∆ sj o×n is obtained from the scores of alternatives by using Equations ( 1)-( 8), in which q ∇ sj and q ∆ sj respectively are the lower and upper limits for the sth alternative respective to the jth criterion.Step 2: Obtain an aspirated rough decision matrix.
Over the whole evaluation process, we clearly understand the limits of a scale i.e., "very dissatisfied (0)" to "very satisfied (100)".For helping each alternative to catch the real gaps on each criterion, in this step, we combined an aspiration-level concept into the COPRAS-R method.That is, the positive and negative points are setting the 100 and 0, respectively.The normalized level is shown in Equation ( 24): Step 3: Calculate the rough proximity degree of gray relation.
In the step, we can use Equations ( 25) and ( 26) to compute the rough proximity degree between the current level and aspiration level for alternatives.
where ζ is the adjusted coefficient (generally, the setting is 0.5) and the rough proximity degree are all equal to 1 [59].Also, the w j is obtained from Equation (23).
Step 4: Integrate the aspirated proximity index.
The aspirated proximity index H s represents the de-roughness degree of satisfaction on each criterion for alternative s.The relative proximity H s of the criteria are calculated as shown in Equation ( 27): Step 5: Calculate the utility ratio for each alternative.
For each alternative, we can use Equation ( 28) to obtain the utility degree between A aspired s relative significance and aspiration level, where B ∆aspire is the aspired alternative.The computation is shown in Equation ( 28):

Case Study
In this section, we apply the proposed hybrid model that combines RDANP with COPRAS-R to assess the performance of a research information system in a university in China, namely, Xiamen Then, C 11 − C 12 = {3, 3, 3, 3, 3, 3, 4, 3, 3, 3} can be transferred into a rough influential set as The rough value of C 12 -C 11 (i.e., m 12 ) is as follows: Symmetry 2019, 11, x FOR PEER REVIEW 6 of 21 where xi and yi are denoted the elements in the lower approximation or upper approximation of z Θ , respectively.In addition, NL and NU are represented the total number of respondents involved in the lower and upper approximations of z Θ , respectively.
Step 3: Drive operations for two rough numbers.
Through Equations ( 1)-( 3), all the domain knowledge of experts can be converted into a set of rough numbers, Υ (z Θ ), as shown in Equation ( 4): Assuming two rough numbers ( ) and μ as a nonzero constant, the arithmetic operations of rough number can be followed by: ( ) Step 4: Transfer into crisp value from rough interval value.When needing to compare analysis for criteria or alternatives ranking, the de-roughness of the rough number into a crisp value can be used by:

The RDANP Method
The rough DEMATEL method is used to establish the rough interrelationship between factors and construct an INRM.The rough total influence relationship matrix is taken as the input of the rough DANP method, and the rough influence weights of the criteria are obtained.The DANP approach solves the unreasonable problem of assuming that each cluster has equal weight in the ANP method [38,50,51].In addition, we also consider the ambiguity of the human domain knowledge in the real environment.The rough degrees of influence of each dimension can be obtained by the rough DEMATEL, and the rough DANP process can be applied to normalize the unweighted super matrix and obtain the rough influential weights of criteria.The detailed steps are as follows [52]: Step 1: Build a rough original influence relationship matrix M on a measuring scale of 0-4 ranging from "no influence (0)" to "very high influential (4)".
Using the aforementioned scale, k respondents are asked to judge the extent of the rough direct influence between two pairwise criteria, denoted by mij.We checked the consistency of the raw data, using the formula as follows: The average gap-ratio in consensus (%) ( )


, where m is the number of the criteria, k is the number of the expert, k ij d is the average direct influence value of criterion i to j.If the value of the average gap ratio in consensus is less than 0.05, we believe that the expert's rating is consistent [48].Then, the rough direct relation average matrix M is acquired through Equations ( 1)-( 8) in the k matrices for the respondents.Finally, we can obtain a rough  , where is n the number of criteria.
Step 2: Obtain the rough initial influence relationship matrix P = [ ij p ]nxn, which is the multiplication of M and v.The other rough values of elements in the rough original influential relationship matrix M = m i j are obtained in a similar way.The average rough initial direct influence matrix is shown in Table 2.In addition, the consensus rate of significant confidence was 96.10%, which is greater than 95% (i.e., gap error rate = 3.90%; less than 5%), as shown in the footnote in Table 2.The result shows a good consistency with acceptable reliability.
, where m is the number of criteria (m = 15), and s is the sample of 10 experts (s = 10) whose practical experience and significant confidence reach 96.10% (more than 95%).
The total rough influence relationship matrix (Table 3) is calculated using Equation (11), and the values in the matrix indicate the total extent to which a criterion affects other criteria and other criteria affect it.Table 4 shows the sum of the influences received and given among the dimensions and criteria, and the results are derived by implementing Equations ( 12) and ( 13) of steps 4 and 5 in the RDANP method.The values in the matrix shown in Table 4 represent the total and net impact values of the dimensions and criteria.The de-roughness matrix can be calculated by Equation ( 8) based on Table 4, which can be found in Table 5.The crisp value in Table 5 can be used to compare the impact of criteria among dimensions and under each dimension.Both of the largest (r i − c i ) and (r i + c i ) values are related to system quality, meaning that system quality not only has the greatest total impact of the four dimensions, but also has the most profound impact on the other three dimensions.Therefore, system quality affects information quality, service quality, and intention to use; furthermore, it is also affected by information quality and service quality, and is the key to the quality of the scientific research information system.Intention to use (C 4 ) has the smallest (r i + c i ) value (1.11), and its (r i − c i ) is negative (−0.22), meaning that it is greatly influenced by other factors and is the resulting element in the evaluation system.The influential network relationship map (INRM) (Figure 1) of the four dimensions and their respective subsystems can be drawn according to Tables 3-5.As shown in Figure 1, the arrow source represents the cause element, and is pointed to the result destination.
System quality (C 1 ), information quality (C 2 ), and service quality (C 3 ) are the three main factors that affect use by users.From Figure 1, we can also see the most important criteria for each dimension.For example, organization design (C 33 ) affects IS training (C 31 ) and assurance (C 32 ); this shows that it is the most critical criterion in the dimension of service quality (C 3 ).Therefore, to maintain the enthusiasm of users in using the system, we need to improve the quality of systems, services, and information.Additionally, service quality and system quality have an important impact on information quality, and service quality and system quality show an interactive relationship.

Obtaining the Weights of Each Dimension and Criterion
The weights of the four dimensions and 15 criteria were calculated according to Equations ( 14)- (18).Table 6 indicates the unweighted rough supermatrix according to Equation (18), and the weighted rough supermatrix can be calculated according to Equations ( 19)-( 23), which can be found

Obtaining the Weights of Each Dimension and Criterion
The weights of the four dimensions and 15 criteria were calculated according to Equations ( 14)-( 18).Table 6 indicates the unweighted rough supermatrix according to Equation (18), and the weighted rough supermatrix can be calculated according to Equations ( 19)-( 23), which can be found in Table 7. Subsequently, the de-rough influential weights of the dimensions and criteria can be found according to Equation (8).After obtaining the rough weight of the values of the dimensions and criteria, we get the crisp value by de-roughing the interval value, which can be found in Table 8.We note that in Table 8, intention to use (C 4 , 0.174) is the most important dimension, followed by system quality (C 1 , 0.150), information quality (C 2 , 0.143), and service quality (C 3 , 0.137).Among the 15 criteria, efficiency (C 44 , 0.051), effectiveness (C 43 , 0.050), and frequency of use (C 41 ) are the three most important criteria.

Evaluating and Improving School Information System Using the COPRAS-R Method
We assessed the quality of a university scientific research information system using the COPRAS-R method, which is outlined in Section 3.3.Xiamen University of Technology's research information system is chosen as a case study in this paper.Ten experienced university staff, who are referred to in Section 4.2, filled out the questionnaire.Satisfaction values ranged from 0 to 100, and experts scored the research information system based on the 15 criteria.The original evaluation scores of the research Symmetry 2019, 11, 1248 16 of 21 information system delivered by the 10 experts are indicated in Table 9.The rough evaluation scores of the research information system with aspiration level can be calculated according to Equations ( 24)-( 28), as shown in Table 10.The value of relative significance indicates the priority for improvement.Therefore, the smaller the value of relative importance among the 15 criteria, the higher the ranking, and the greater the need for performance to be improved.According to Table 10, navigation patterns (C 42 , 0.018), timeliness (C 24 , 0.025), and response time (C 14 , 0.026) are the three criteria that have smaller relative significance value than the others.Note: ( ) is priority ranking for improvement.

Discussion
We used the R-DEMATEL method to examine the causal relationship between indicators, as shown in Table 5.The INRM, which can show the causal relationships between any two dimensions and between any two criteria of each dimension, was plotted as shown in Figure 1.The INRM shows that system quality (C 1 ) has the largest (r i + c i ) value and (r i − c i ) value, and only intention to use (C 4 ) has a negative (r i − c i ) value.The results indicate that system quality is the most important reason for influencing intention to use, and information quality (C 2 ) and service quality (C 3 ) also have a significant impact on intention to use (C 4 ).This result means that intention to use (C 4 ) is the result variable for the entire model, which reflects the usage of the information system, and it is the value of user satisfaction with system performance.The ease of use (C 11 ), timeliness (C 23 ), and organization design (C 33 ) are the three main elements of the respective subsystems, according to Figure 1.Therefore, if we want to improve the frequency of use (C 41 ), effectiveness (C 43 ), and efficiency (C 44 ) of the information system, improving system usability, reducing system response time, and optimizing system design are objectives worth considering.
The R-DEMATEL-based ANP approach was applied to find the weights of the dimensions and criteria.From Table 8, intention to use (C 4 ) has the greatest weight, followed by system quality (C 1 ), information quality (C 2 ), and service quality (C 3 ).These results are consistent with the results of RDEMATEL analysis, and system quality (C 1 ) and intention to use (C 4 ) are the two priorities in the evaluation information system.Considering the criteria, efficiency (C 44 , 0.051), effectiveness (C 43 , 0.050), and frequency of use (C 41 , 0.047) are the three most important criteria, followed by assurance (C 31 , 0.047), IS training (C 32 , 0.046), and organization design (C 33 , 0.044).From their point of view, users are highly concerned with the effectiveness and efficiency of the system, which play a major role in user satisfaction with the system.In addition, service quality is becoming increasingly important for information system products.Users care whether the system design can fully meet their needs and pay attention to the developer's service commitment and careful training.
As shown in Table 10, efficiency (C 44 ) and effectiveness (C 43 ) ranked in the top two in terms of relative significance value.This result reflects that the research information system of XMUT can improve the efficiency of teachers and researchers.Most teachers are quite satisfied with the effectiveness and efficiency of the system.As the informatization of Chinese universities deepens, the use of information systems has become a trend.University staff must change the way they work using the internet to improve work efficiency and effectiveness.The navigation pattern (C 42 ) is the ranking with a relative significance value.As with the development of mobile internet technology, users want to access the system through mobile terminals.However, the speed of development of mobile APPs in Chinese universities is slow, and cannot meet the needs of teachers and researchers.Timelines (C 23 ) and response time (C 14 ) are ranked lower in the performance evaluation of this system.As we know, users are quite disgusted with the slow response of the system and slow updating of content.If users find that the system is not responding, they are likely to give up on using the system.Therefore, improving the response speed of the system itself and speeding up the content updating process are important prerequisites for satisfying users.Assurance (C 31 ), IS training (C 32 ), and organization design (C 33 ) are urgently needed to improve performance, according to Table 10.The service quality of the system has long been a problem in the construction of information technology in Chinese universities.Many system developers focus on improving system performance while ignoring user guidance and training.Therefore, the poor quality of service is also an important reason for user dissatisfaction with the system.

Conclusions
Different from previous papers that used statistical perspectives to verify the D&M model, this paper applied a hybrid MCDM approach that combined RDANP with the COPRAS-R method to evaluate the performance of specific information systems based on the D&M model.We selected four dimensions and 15 criteria.These dimensions include system quality, information quality, service quality, and intension to use.We designed two surveys, one to measure the influential degrees between dimensions and the criteria for each dimension, and the other to calculate the user satisfaction for each criterion on a specific system.We chose a research information system in a Chinese university as a case.Ten experienced university teachers or research managers were invited to interview and fill out two questionnaires.
The results demonstrate that intention to use (C 4 ) and system quality (C 1 ) are the two most significant indicators for the evaluation of the quality of research information systems for universities in China, making up 60% of the total weight, and system quality (C 1 ) is the main factor affecting the (α) andSymmetry 2019, 11, x FOR PEER REVIEW

Step 6 :T
effect of criterion i on the other criteria.Clearly, if i j r c − is positive, criterion i is a causal component, and if i j r c − is negative, then criterion i is an affected component.As a result, the rough influence network relationship map (RINRM) can be finished by mapping the data set ( Derive rough total influence relationship matrix C T based on the criteria and D T based on the dimensions.The rough matrix T could be differentiated into C T based on the criterion and D T based on the dimensions.The matrix C T modularizes the matrix T according to the dimensions, which are essentially the same matrix.The rough matrix D T is found by averaging the rough degree of criterion influence in each dimension.As shown in Equation (14), Di denotes the ith dimension; cij denotes the jth criteria in the ith dimension.For example, we get a crisp value by averaging means the extent to which the criteria in the mth dimension affect the criteria in the nth dimension.Then, we get the D T by averaging the mn c T in the C T .

( 15 )
For example, T δ∆ C , which is a rough submatrix of T δ C , can be normalized to T δ∆pq C

T
, which is a rough submatrix of C δ T , can be normalized to pq C δΔ T sj ) and γ(b ∆aspire , b ∆ s sj ), and aspiration levels of b ∆aspire and b ∆aspire j

Table 2 .
Rough original influence relationship matrix.
Note: The average gap ratio in consensus

Table 3 .
Rough total influence relationship matrix.

Table 4 .
The sum of rough influences given and received on dimensions and criteria.

Table 5 .
The de-roughness influences given and received on dimensions and criteria.

Table 5 .
The de-roughness influences given and received on dimensions and criteria.

Table 8 .
The rough influential weights on dimensions and criteria.

Table 9 .
The original evaluation scores of alternatives on 10 experts.

Table 10 .
The rough evaluation scores of research information systems with aspiration level.