A Novel Integration Method for D Numbers Based on Horizontal Comparison

: D numbers theory is an extension of Dempster–Shafer evidence theory. It eliminates the constraints of mutual exclusion and completeness under the frame of discernment of Dempster– Shafer evidence theory, so it has been widely used to deal with uncertainty modelling, but if it cannot effectively deal with the problem of missing information, sometimes unreasonable conclusions will be drawn. This paper proposes a new type of integration representation of D numbers, which compares the data of multiple evaluation items horizontally, and can reasonably ﬁll in missing information. We apply this method to the user experience evaluation problem of online live course platform to verify the effectiveness of this method.


Introduction
The theory of multiple criteria decision making (MCDM) originated from the Pareto optimal concept, and was introduced into the field of decision science as a standardized decision-making method in the 1960s. In 1981, Hwang and Yoon [1] divided MCDM into two types: multiple objective decision making (MODM) and multiple attribute decision making (MADM) based on different decision spaces. The decision space of MODM is continuous, and the goal of decision-making is to plan and design the optimal plan; MADM corresponds to the discrete decision space, and the goal of decision-making is to use the evaluation information of multiple attributes, and through certain calculation or processing, several attribute values are converted into the comprehensive evaluation value of the scheme, so as to reasonably rank the limited alternatives. In the actual decisionmaking process of MADM problem, due to the complexity of the actual decision-making problem, the evaluation information of the project attributes often needs to be obtained with the cognition of domain experts, and the cognition of the experts participating in the evaluation is often difficult to accurately describe the problem, which brings great uncertainty. On the one hand, this uncertainty comes from the vagueness of the expert's own cognition, and on the other hand, it comes from the inherent uncertainty of realistic decision-making problems. Moreover, for complex decision-making problems, a single expert has insufficient ability to deal with it. Therefore, uncertain multi-attribute group decision-making has become a research hotspot in decision theory in recent years. The main research methods include: grey correlation analysis [2], Dempster-Shafer evidence theory (D-S evidence theory) [3][4][5], fuzzy set theory [6][7][8], and rough sets theory [9][10][11], etc., among them, D-S evidence theory can well describe the uncertainty and incompleteness of evaluation information without prior probability. At the same time, the Dempster combination rule can effectively integrate data from multiple information sources, which makes it widely used in fields such as information fusion [12], fault diagnosis [13], reliability evaluation [14], human reliability analysis [15], etc. 2 of 14 An important extension of the D-S evidence theory is the D numbers theory [16]. The D number theory is mainly used for the representation and processing of uncertain information. It can be regarded as a special form of random set. It relaxes the assumptions of mutual independence between elements in the frame of discernment of D-S evidence theory, and relaxes the constraints of the completeness of the BPA function. Compared with fuzzy theories such as intuitionistic fuzzy sets and hesitation fuzzy sets and their extended forms, D numbers theory can directly express uncertainty by assigning probabilities to multiple subsets, which greatly improves the ability to represent and process uncertain information. However, the D numbers theory also has its own limitations. It is essentially a method of gathering multi-attribute information, which does not solve the shortcomings of the D-S evidence theory [16]. Therefore, scholars continue to improve the D numbers theory. For example, for multi-level and multi-attribute evaluation data, Deng et al. [17,18] combined the D numbers theory with analytic hierarchy process (AHP) and proposed the D-AHP method. Aiming at the problem that the combination rule cannot satisfy the associative law, Wang et al. [19] developed an improved multiple D numbers combination method, and Guan X et al. [20] proposed a new combination rule. In view of the problem that both qualitative and quantitative data exist at the same time, through the conversion model, all quantitative and qualitative data are transformed into the same qualitative description level, forming a D numbers decision matrix of the evaluation criteria, and using the D numbers integration method to integrate information under different criteria [21]. When the elements in the frame of discernment are not mutually exclusive, the distance between two D numbers is defined according to the definition form of the distance between two BPA in D-S evidence theory, so that the distance between two D numbers is smaller than the distance between D-S evidence theory under the same identification framework [22]. For data with incomplete information, Wang et al. [23] defined an improved D numbers integrated representation method to solve the MADM problem and so on. The D numbers approach is widely applied in different fields, including human reliability analysis [24], green supply chain management [25], and emergency management [26], university scientific research evaluation [27], failure model and effects analysis [28,29], renewable energy resources evaluation [30], and health-care waste treatment technology assessment [31]. Despite there are the above advantages for D numbers theory in dealing with uncertain information issues, some disadvantages still exist when transforming D numbers into the real value of D numbers' integration rules. In this paper, improvements on the disadvantages of D numbers' integration rules in incomplete information will be made, etc.
In the previous integrated representation of the D numbers, if the information is incomplete, the missing part of the assessment is directly ignored, and the result may violate common sense. In order to overcome this shortcoming, Wang et al. [23] proposed an improved D numbers integrated representation method, which proportionally distributes the missing information to the decision according to the original value of D numbers. However, this method does not consider the evaluation object globally, and the results may be counterintuitive. For instance, ten experts make an assessment of program A and program B. For program A, seven experts give eight points, and the other three experts make no assessment as they are unfamiliar with A, and this part is regarded as missing information, and the result can be expressed by D A = {(8, 0.7)}. Based on the integrated method from the literature [16], it can be transformed into the real number I(D A ) = 8 × 0.7 = 5.6, that is to say, when D numbers are integrated, 0 is given as the three experts do not make an assessment. According to the literature [23], when the D numbers are integrated, the missing part of information should be allocated to the given results of assessment in a certain proportion. There is only eight points for the given results, so the missing part should be assigned eight points, and the integration result is expressed by I (D A ) = 8 × 0.7 + 8 × 0.3 = 8. The different ways the two integration results treat missing information leads to vastly different integration results. When it comes to the integration of D A , we should also consider the assessment on program B made by the experts, and the integration results obtained can be more comparable. Based on [16,23], Axioms 2021, 10, 312 3 of 14 this paper proposes a new type of D numbers integrated representation, which can deal with the lack of information more reasonably. Then the proposed method is applied to the user experience evaluation problem of the online live course platform to verify the effectiveness of the method.
The rest of this paper is organized as follows. Section 2 first briefly introduces the basic concepts. Section 3 introduces our D numbers integration method (stepwise comparison and replacement integration (SCRI) method). Section 4 studies the implementation of our proposed method on the user experience evaluation problem of the online live course platform. Section 5 discusses and summarizes.

Dempster-Shafer Theory
Aiming at the uncertainty and incompleteness of information, how to use expert group experience and collected relevant data as decision-making information to reduce the uncertainty of decision-making and improve the credibility of the results is the key content in the research of multi-attribute group decision-making methods. D-S evidence theory [3][4][5] provides a general framework for constructing uncertain reasoning models, which can satisfy an axiom system that is weaker than probability, can describe the uncertainty and incompleteness of evaluation information well. It also has a synthesis rule for synthesizing the opinions of multiple experts, so it is an important and effective group decision-making method.
Definition 2 [3,4]. (Dempster's combination rule) Suppose m 1 and m 2 are two independent BPAs from different sources, the Dempster combination rule, denoted by m = m 1 ⊕ m 2 ,is defined as follows: where k is a normalization constant, called conflict coefficient of two BPAs. When k = 1, it means that m 1 and m 2 completely conflict, and they cannot be combined by Dempster's combination rule.

D Numbers Theory
D numbers theory [16] is an extension of Dempster-Shafer evidence theory. It eliminates the constraints of mutual exclusion and completeness under the frame of discernment. The general definition and combination rules of D numbers are given below. Definition 3 [16]. Let Ω be a finite nonempty set, D number is a mapping D : where φ is an empty set and B is a subset of Ω. Note that different from the concept of frame of discernment in Dempster-Shafer theory, in set Ω the elements don't require mutually exclusive.
There is a special situation of D numbers.
Definition 4 [16]. For a discrete set Definition 5 [16]. (D numbers theory combination rule) Suppose D 1 and D 2 are two D numbers, denoted by The combination of D 1 and D 2 , indicated by D = D 1 ⊕ D 2 , is defined by where We can treat D numbers as "special" discrete random variables. Taking is the value for it, and v 1 i (i = 1, 2, · · · , n) stands for the corresponding probability. It is a "special" discrete random variable as it requires the sum of probabilities to be less than or equal to 1, that is, regarded as a regulation constant, and the value of C is determined by (10). In this case, it is guaranteed that the sum of all probabilities in new D numbers is still less than or equal to 1.
where I(D) is real number.

Limitations
According to Deng [16], the integration representation of D numbers is defined as the missing part of evaluation is directly excluded. Wang et al. [23] argued that this may lead to unreasonable outcomes of decision-making.
Example 1 (Wang et al. [23]). According to Wang et al. [23], it is assumed that 10 experts are invited to evaluate the automobile performance of A and B, and the evaluation system is set Ω = {1, 2, 3, 4, 5, 6, 7, 8, 9}, in which "9" means "very good" and "1" means "very bad". When the 10 experts give an evaluation on the performance of car A, six of them choose "7", two choose In terms of the integrated results, car B gets a better score than car A, as two experts are uncertain about the score when evaluating A and do not give an evaluation, and incomplete information is abandoned in integration of D numbers. That is to say, if the experts give no evaluation, 0 is applied to deal with it, thus making the final result of car A relatively low.
Wang et al. [23] put forward a method to improve the integration of D numbers. They argued that for experts with incomplete cognition, they could refer to the evaluation results given by experts familiar with the evaluation results, and the incomplete part of incomplete information should be assigned to the evaluation results given in line with certain rules. For the given numbers, stands for the percentage of credibility of evaluation level b i . Wang et al. [23] believed that the incomplete part of incomplete information should be assigned to the known evaluation level b i (i = 1, 2, · · · , n) according to the percentage of credibility.

of 14
Obviously, if the information is complete, that is, when n ∑ i=1 v i = 1, the proposed method is degenerated as the traditional D numbers' integration.
The improved integration representation of D numbers is adopted to reevaluate the performance of car A and car B in Example 1. For car B, the information is complete, thus: For car A, D A = {(7, 0.6), (8, 0.2)} is incomplete, and the incomplete part of 0.2 should be assigned to the known evaluation level based on the percentage of credibility, thereby: Obviously, when the improved integration representation of D numbers is adopted, car A gets a higher total score than car B, which is contrary to the traditional integrated results of D numbers.
Nevertheless, we also see that the improved integration representation of D numbers can also bring about some counterintuitive results, as shown in the following example: For car B, the information is complete, thus: Obviously, the evaluation result obtained through this method is that car A is in the ascendant, but from the original evaluation result, the number of experts who give a score of "8" to car B is more than that of experts who give score of "8" to car A, and complete information is obtained for the evaluation result of car B. The reason for this lies in the fact that in certain information of D A , the score of "8" accounts for 100%, and when the improved integration representation of D numbers is adopted, the incomplete information remained, namely opinions from the three experts who do not give and evaluation, are replaced with "8". If the above operation is conducted when the evaluation results of car B are not fully considered, the final score of car A might be made too high. In this case, a more effective integration representation of D numbers needs to be proposed.

Proposed Method (Stepwise Comparison and Replacement Integration (SCRI) Method)
In this section, a new method of D numbers integration is proposed, which not only considers the assessment results of a single program, but also horizontally compares with the assessment results of other programs. It can fill in the missing information in a more reasonable way. Compared with existing methods, it is more effective.
It assumes that there are N objects O 1 , O 2 , · · · , O N ,to be evaluated, and M experts in total who participate in the evaluation. The stepwise representation of the proposed method is given as follows. The flowchart of the proposed method is shown in Figure 1.
In this section, a new method of D numbers integration is proposed, which not only considers the assessment results of a single program, but also horizontally compares with the assessment results of other programs. It can fill in the missing information in a more reasonable way. Compared with existing methods, it is more effective.
It assumes that there are N objects , , ⋯ , ,to be evaluated, and experts in total who participate in the evaluation. The stepwise representation of the proposed method is given as follows. The flowchart of the proposed method is shown in Figure 1.

Convert evaluation data into D numbers
Step1: Convert D numbers into original evaluation matrix R Step2: Consider the weight of experts, get initial integrated value matrix Step5:Reassign other 0 elements in the matrix Step6:Add the numbers in each row to get the integrated value of the D number Step4:Reassign a column whose elements are all zero

Yes No
Step3:observe whether there is a column in which elements are all 0 Step 1 for the object , the evaluation grade from each expert is arranged in descending order, and if there is no evaluation, the temporary record is 0. That is to say, the evaluation result from experts on the object arranged in descending order is viewed as a row vector, thus forming the original evaluation matrix of × . Step 1 for the object O i , the evaluation grade from each expert is arranged in descending order, and if there is no evaluation, the temporary record is 0. That is to say, the evaluation result from experts on the object O i arranged in descending order is viewed as a row vector, thus forming the original evaluation matrix R of N × M.
Step 2 there is a total of M experts participating in the evaluation, and the corresponding credibility from each expert is considered as 1/M. Each element in the original evaluation matrix R is multiplied by 1/M, and it is concluded that the initial integrated value matrix is R Int .
Step 3 observe whether there is a column in which elements are all 0. If so, enter step 4; otherwise, go directly to step 5.
Step 4 check each column from the first column to see whether there exists element 0. It assumes that 0 appears in the s + 1 column for the first time, that is to say, the corresponding elements for the first s columns are initial integrated value obtained based on the definite evaluation and calculation from experts. Add up each element of each row in the first s columns and then take the average of their sum, that is to divide the sum of each row by s, which is called reference value for basic evaluation. As for the columns where all the column elements are 0, replace the element 0 in this column with the reference value for basic evaluation of the corresponding row, that is to assign this part of incomplete information to the evaluation results clearly given in the previous s columns in proportion. Then it comes to the matrix R Int−r .
Step 5 check whether there is element 0 in each column. If so, make a comparison of the non-zero elements in this column, and substitute the minimum value for the element 0. That is to say, the incomplete information is replaced with the minimum evaluation value corresponding to the evaluation object through horizontal comparison, until all elements in the matrix are nonzero, which is called the modified integrated value matrix R * Int .
Step 6 add up elements of each row in the matrix, and then it comes to the value, which is the integrated value of D numbers corresponding to the evaluation objects Let us use this method to operate Example 2.
Step 2 multiply each element by 1/M , here M = 10 is the number of experts, and then it comes to the initial integrated value matrix. The weights of experts are the same 1/10. Step 3 no column with all 0 elements, then go to step 5.

 
Step 3 observe whether there exists a column in which all column elements are 0. Herein, the elements are all 0 in the last column, then go to step 4.
Step 4 0 appears in the 8th column for the first time, that is, the elements corresponding to the first seven columns are interval integrated values obtained based on the definite evaluation given by the experts. The elements in each row of the first seven columns are added up and then averaged, and 59/70,59/70,60/70 are obtained respectively. Replace these values with the elements in the last column of the initial integrated value matrix, and then it comes to the matrix R Int−r .

Applications in MADM
In this section, we will put forward the method of D numbers integration that is adopted to make an assessment of user experience of the online live course platform, and account for the rationality of the proposed method by data collection based on the evaluation index system in the literature [32].
During the rapid transmission of the COVID-19 pandemic, online learning has become an effective way to implement "suspended class, ongoing learning", and carried out all over the world. Since there are reasons such as technology and network bandwidth, etc., some telecommuting and conferencing software has undertaken the role of online live course platforms. However, due to the original design purpose and the characteristics of the software, the software cannot fully meet the needs of schools in online live courses. In this case, it is prominently significant to optimize the user experience of the online live course platforms [32].
In Wang et al. [32], qualitative and quantitative methods were combined to establish evaluation indicators by integrating data obtained through the literature research, interviews and network research from the perspective of user experience. Delphi method and analytic hierarchy process were utilized to establish the weight of each indicator and set the scoring grade. T-test method was adopted to establish the comparative analysis method of model application, and the standard procedure of model application was designed to build a complete evaluation system. However, there is an important issue that has not been well resolved in [32]: when evaluating different platforms, some experts may not give definite scores as they are not familiar with the scoring items. In this case, the scores made reluctantly may greatly affect the final evaluation results. The SCRI method proposed by us can solve this in a good way, and it can deal with possible uncertainty, ignorance and missing information in a simple and effective way.
In Wang et al. [32], the main factors affecting the user experience of the online live course platforms are divided into four primary indicators, which are respectively Ease of Operation, Functional Completeness, Interface Rationality, Technical Reliability. Each primary indicator is divided into several secondary indicators. The hierarchical structure and corresponding weights of user experience evaluation of the online live course platforms are shown in the Figure 2. below. evaluating different platforms, some experts may not give definite scores as they are no familiar with the scoring items. In this case, the scores made reluctantly may greatly affec the final evaluation results. The SCRI method proposed by us can solve this in a good way, and it can deal with possible uncertainty, ignorance and missing information in a simple and effective way.
In Wang et al. [32], the main factors affecting the user experience of the online live course platforms are divided into four primary indicators, which are respectively Ease o Operation, Functional Completeness, Interface Rationality, Technical Reliability. Each pri mary indicator is divided into several secondary indicators. The hierarchical structure and corresponding weights of user experience evaluation of the online live course platforms are shown in the Figure 2. below. In the hierarchical structure, a total of 16 secondary indicators are included. Experts are invited to evaluate all the influencing factors. To keep consistent with the require ments of the scale, the evaluation results are divided into 9 grades, = {1,2,3,4,5,6,7,8,9} in which "9" means "very good" and "1" means "very bad". If the experts do not have a good understanding of some factors, they do not need to score, and finally we obtain eval uation data with missing information.
All the results are expressed in the form of D numbers herein. For instance, 10 experts are invited to evaluate the user experience of the platform. When evaluating a certain in fluencing factor, 4 experts give 8 scores, 3 experts give 7 scores, 2 experts give 5 scores and the other one expert does not score as he or she is unfamiliar with the factor. Hence the evaluation result can be expressed as {(8, 0.4), (7,0.3), (5,0.2)}, which is not a complete D number. Ding Talk, Tencent Classroom and Tencent QQ are adopted in this paper, and the evaluation results are shown in the following  In the hierarchical structure, a total of 16 secondary indicators are included. Experts are invited to evaluate all the influencing factors. To keep consistent with the requirements of the scale, the evaluation results are divided into 9 grades, Ω = {1, 2, 3, 4, 5, 6, 7, 8, 9}, in which "9" means "very good" and "1" means "very bad". If the experts do not have a good understanding of some factors, they do not need to score, and finally we obtain evaluation data with missing information.
All the results are expressed in the form of D numbers herein. For instance, 10 experts are invited to evaluate the user experience of the platform. When evaluating a certain influencing factor, 4 experts give 8 scores, 3 experts give 7 scores, 2 experts give 5 scores, and the other one expert does not score as he or she is unfamiliar with the factor. Hence, the evaluation result can be expressed as {(8, 0.4), (7, 0.3), (5, 0.2)}, which is not a complete D number. Ding Talk, Tencent Classroom and Tencent QQ are adopted in this paper, and the evaluation results are shown in the following Table 1: Then we adopted the SCRI method to transform the evaluation results from all experts into a clear form of real numbers. For instance, in the Table 2, the evaluation results on Y2 made by the ten experts are as follows:  Step 2, multiply each element by 1/10 to obtain the initial integrated value matrix R Int =   9/10 9/10 8/10 8/10 8/10 8/10 8/10 6/10 6/10 0 8/10 8/10 7/10 7/10 6/10 6/10 5/10 5/10 0 0 8/10 8/10 7/10 7/10 7/10 6/10 5/10 0 0 0   Step 3, elements in the last column are all 0, then go to step 4; Step 4, 0 appears in the 8th column for the first time, that is, the elements corresponding to the first 7 columns are initial integrated values obtained based on the definite evaluation and calculation given by the experts. Add up the elements in each row of the first 7 columns and calculate the arithmetic mean, and 58/70,47/70,48/70 are obtained respectively. Replace these with the elements of the last column in the matrix R Int , and it is concluded that: Step 5, check whether element 0 appears in each column. If so, make a comparison of the non-zero elements in this column and replace element 0 with the minimum value. As for min 6 10 , 5 10 = 5 10 in the 8th column, replace 0 with 5/10, and in the 9th column, replace 0 with 6/10 to obtain the modified integrated value matrix R * Int : R * Int =   9/10 9/10 8/10 8/10 8/10 8/10 8/10 6/10 6/10 58/70 8/10 8/10 7/10 7/10 6/10 6/10 5/10 5/10 6/10 47/70 8/10 8/10 7/10 7/10 7/10 6/10 5/10 5/10 6/10 48/70   Step 6, add up the elements of each row in the matrix and convert the evaluation results from all experts into a clear form of real numbers, as follows: 7.8286, 6.4714 and 6.5857.
The evaluation results of all secondary indicators are converted into real numbers as shown in Table 3: Table 3. Integration of the assessment results.

Basic Attribute Ding Talk Tencent Classroom Tencent QQ
Finally, multiply these numbers by the corresponding weights and total them up to obtain the comprehensive evaluation value of user experience from the three platforms, as well as the scores of each primary indicator in Table 4:

Conclusions and Future Work
As an extension of D-S evidence theory, D numbers theory has recently attracted the attention from scholars, but it is not quite satisfactory while dealing with incomplete information. In this paper, a new integration method SCRI is proposed, of which the advantages are as follows: reasonably filling the missing information by comparing the data of several assessment programs in a horizontal way, to a certain extent, makes up for the decision-making mistakes which may be led by missing information for the D numbers theory in integrated data. The possible limitation of this method is that the information filled in is only the inference of the missing part, which is not guaranteed to be completely accurate. This method can be combined with other MADM classical methods to deal with decision-making issues in uncertain environments such as fuzziness and incompleteness, which is also the direction of subsequent efforts.