Towards Better Concordance among Contextualized Evaluations in FAST-GDM Problems

Abstract

A flexible attribute-set group decision-making (FAST-GDM) problem consists in finding the most suitable option(s) out of the options under consideration, with a general agreement among a heterogeneous group of experts who can focus on different attributes to evaluate those options. An open challenge in FAST-GDM problems is to design consensus reaching processes (CRPs) by which the participants can perform evaluations with a high level of consensus. To address this challenge, a novel algorithm for reaching consensus is proposed in this paper. By means of the algorithm, called FAST-CR-XMIS, a participant can reconsider his/her evaluations after studying the most influential samples that have been shared by others through contextualized evaluations. Since exchanging those samples may make participants’ understandings more like each other, an increase of the level of consensus is expected. A simulation of a CRP where contextualized evaluations of newswire stories are characterized as augmented intuitionistic fuzzy sets (AIFS) shows how FAST-CR-XMIS can increase the level of consensus among the participants during the CRP.

1. Introduction

Group decision-making (GDM) is concerned with choosing the most adequate option among several potential options. While a straightforward strategy for solving a GDM problem is to reach a decision without the agreement of the participants, a more inclusive one consists in making a decision with a general agreement among them [1]. Since a unanimous agreement might be difficult to reach, a partial agreement may be preferable to make a decision [2,3,4]. The level of agreement is expected to be higher in cooperative environments. Scientific or medical decisions where participants are willing to share their knowledge are examples of such environments [5,6,7]. In contrast, lower levels of agreement are expected in non-cooperative environments where participants are reluctant to share their knowledge (e.g., political or economical decisions) [5,8,9]. In situations where some participants have more expertise than others, the agreement on the decision might be highly influenced by the expertise of such participants [10,11,12].

Bearing in mind that participants of a GDM problem might have access to multiple sources of information [1,13,14,15], models that manage homogeneous [16,17,18] and heterogeneous information [13,19,20,21] in GDM can be found in the literature. Models handling homogeneous information often represent the attribute values characterizing the options through a single domain (e.g., numerical, interval-valued or linguistic domain). In contrast, models handling heterogeneous information usually transform different domains into one to handle those values [19]. Nevertheless, a single domain does not guarantee an agreement on which aspects or attributes of the options should be considered in their evaluation. This is the case when a heterogeneous group of participants, i.e., participants having different levels of knowledge, areas of expertise and personal backgrounds, differ in their opinions [22]. For instance, consider an editorial team, consisting of an agronomist, an economist and an editor, trying to reach an agreement on the articles (potential options) to be published in a special issue on ecological and safe transport of agricultural products. The agronomist might focus on aspects (attributes) like agricultural crop categories (‘soil’, ‘wheat’, ‘corn’, etc.) during the evaluation of the articles. The economist might pay attention to aspects like ‘costs’ and ‘delivery time’. Finally, the editor might focus on aspects like ‘originality’, ‘relevance’ and ‘linguistic quality’.

A modular approach to handle heterogeneous information has been proposed in [23]. In that approach, the participants may carry out their own selection of attributes to perform their evaluations. However, this approach does not consider situations where the participants share their attributes to solve a GDM problem. Such a situation has been studied in [24]. In that work, the authors define a flexible attribute-set group decision-making (FAST-GDM) problem in which the participants may be suggested to refocus their attention on a shared collection of attributes that were initially observed by some persons, but unobserved by others.

In FAST-GDM two processes can be identified: a consensus reaching process (CRP) and a selection process (SP). During the CRP, the participants try to agree on the most suitable option(s) with a satisfactory level of consensus [1,25,26]. If a satisfactory level of consensus is reached, a SP starts by selecting the option(s) according to the preferences of the participants [14,16]. To quantify the level of consensus in a CRP, a moderator can be supported by indices, e.g., a concordance index between the evaluations given by the participants [27]. In the case of FAST-GDM, the usability of several theoretical concordance indices has been studied in [28].

Designing CRPs for the participants to perform evaluations with a high level of consensus in FAST-GDM is a key challenge. To address this challenge, a novel variant of the CRP proposed in [24] is described in this paper. The variant, called flexible attribute-set consensus reaching by exchange of the most influential samples (FAST-CR-XMIS), aims at increasing the level of consensus by additionally exchanging the most influential samples identified by the participants (experts or non-experts) during the evaluation process. Such samples are well-known by the participants according to their individual experiences and are regarded as relevant cases to put the evaluated options in context. For instance, if the above-mentioned agronomist might recall an old article included in a previous special issue on ecological and safe transport of agricultural products, which is intrinsically connected to a new article. The agronomist might then use the old article to contextualize the evaluation of the new article. In this regard, the idea behind FAST-CR-XMIS is that, after exchanging the most influential samples, the participants’ understandings about the problem will become better attuned to each other and, thus, the collective level of consensus will be increased.

To model and handle the previous idea, a mathematical framework based on augmented intuitionistic fuzzy sets (AIFSs) [29] is used within FAST-CR-XMIS. As will be shown in the next section, the FAST-GDM problem is mathematically modeled using this framework. Thus, FAST-CR-XMIS makes use of this framework to, e.g., quantify the level of consensus among the evaluations performed during a CRP.

In addition to increasing the level of consensus, a key advantage of FAST-CR-XMIS is that it may be used to perform recurrent CRPs, where a particular group (or panel) of participants is established to carry out periodic evaluations in a given GDM problem. In this case, since the participants’ understanding about the problem will become better attuned to each other after the first CRP, forthcoming CRPs are expected to be even more efficient.

To show how FAST-CR-XMIS works, a computerized simulation of a CRP in which a given number of participants try to reach an agreement on the category of newswire stories is presented in Section 4. Before, in Section 2 the definitions and formal notations that are used throughout the paper are introduced. Next, a comprehensive explanation of the novel FAST-CR-XMIS algorithm is presented in Section 3. The results of the simulation are presented in Section 5 and a discussion about these is presented in Section 6. Finally, the paper concludes in Section 7 with some future research directions.

2. Preliminaries

As has been mentioned above, group decision-making is usually understood as a process by which a group of experts (Since participants in GDM are considered to have some expertise on the subject under discussion, hereafter the term ‘experts’ is used for referring to them.) try to reach a collective decision about potential solutions for a particular problem. During that process, each expert evaluates the potential solutions, called options, according to his/her knowledge or experience. Mathematically, such evaluations can be described as follows:

Consider a discrete collection $X=\{x_1,\cdots ,x_n\}$ consisting of the potential solutions for a given problem, as well as a collection $A\subseteq X$ consisting of the suitable options for this problem. Consider also a collection $E=\{E_1,\cdots ,E_m\}$ representing a group of experts who have been asked to evaluate the level to which each option $x_i\in X$ satisfies a proposition p having the canonical form ‘$x_i$ IS A’ meaning $x_i$ is member of A and hence is considered a suitable option.

In the framework of fuzzy set theory [30], the evaluation of the level to which $x_i$ satisfies p performed by an expert $E_j\in E$ can be characterized by a membership grade ${\mu }_{A@E_j}\left(x_i\right)$, which is a number in the unit interval $[0,1]$ where 0 and 1 respectively represent the lowest and the highest membership level. Hence, the evaluations of the options performed by $E_j$ can be denoted by a fuzzy set of suitable options, say $A_{@E_j}$, such that

$$\begin{array}{cc}\hfill A_{@E_j}=& \left\{\langle x_i,{\mu }_{A@E_j}\left(x_i\right)\rangle |\left(x_i\in X\right)\wedge \left(0<{\mu }_{A@E_j}\left(x_i\right)\le 1\right)\right\}.\hfill \end{array}$$

(1)

Notice that, in this framework the evaluation of p is considered as being a matter of degree, i.e., the evaluation of p is not limited to the lowest and the highest membership levels, but all the values in between.

In circumstances where $E_j$ hesitates about the level to which $x_i$ satisfies p, such an evaluation can be better described in the framework of intuitionistic fuzzy sets (IFSs) [31,32]. In this framework, the evaluation can be characterized by an IFS element $\langle x_i,{\mu }_{A@E_j}\left(x_i\right),{\nu }_{A@E_j}\left(x_i\right)\rangle$, in which the components ${\mu }_{A@E_j}\left(x_i\right)$ and ${\nu }_{A@E_j}\left(x_i\right)$ respectively represent the levels of membership and nonmembership of $x_i$ to the IFS $A_{@E_j}$. Thus, the evaluations performed by $E_j$ can be denoted by an IFS, say $A_{@E_j}$, such that

$$\begin{array}{cc}\hfill A_{@E_j}=& \left\{\langle x_i,{\mu }_{A@E_j}\left(x_i\right),{\nu }_{A@E_j}\left(x_i\right)\rangle |\left(x_i\in X\right)\wedge \left(0\le {\mu }_{A@E_j}\left(x_i\right)+{\nu }_{A@E_j}\left(x_i\right)\le 1\right)\right\},\hfill \end{array}$$

(2)

where $0\le {\mu }_{A@E_j}\left(x_i\right)+{\nu }_{A@E_j}\left(x_i\right)\le 1$ represents the consistency condition. A hesitation margin defined by $h_{A@E_j}\left(x_i\right)=1-({\mu }_{A@E_j}\left(x_i\right)+{\nu }_{A@E_j}\left(x_i\right))$ has been proposed to represent the hesitation of $E_j$ during the evaluation of the membership and nonmembership levels [31,32].

In situations where a heterogeneous group of experts try to find a collective decision, experts might like to express not only the level to which $x_i$ satisfies p, but also the reasons justifying that level. That is, experts might like to perform contextualized evaluations of p. Such contextualized evaluations can be described in the augmented framework proposed in [29]. In this framework, a contextualized evaluation of the level to which $x_i$ satisfies p carried out by an expert $E_j$ can be characterized by an augmented appraisal degree (AAD). An AAD, say ${\widehat{\mu }}_{A@E_j}\left(x_i\right)$, is a pair $\langle {\mu }_{A@E_j}\left(x_i\right),F_{{\mu }_{A@E_j}}\left(x_i\right)\rangle$, whose components denote the level ${\mu }_{A@E_j}\left(x_i\right)$ to which $x_i$ satisfies p, as well as the particular collection (More specifically this collection might be represented by a list, a set, a multi-set, among others.) $F_{{\mu }_{A@E_j}}\left(x_i\right)$ of the $x_i$’s features that have been relevant to the evaluation according to the knowledge about A possessed by $E_j$, further denoted by $K_{A@E_j}$.

The augmentation of IFS elements by means of AADs has also been proposed in [29]. An augmented IFS element $\langle x_i,{\widehat{\mu }}_{A@E_j}\left(x_i\right),{\widehat{\nu }}_{A@E_j}\left(x_i\right)\rangle$ consists of both a membership AAD ${\widehat{\mu }}_{A@E_j}\left(x_i\right)$ and a nonmembership AAD ${\widehat{\nu }}_{A@E_j}\left(x_i\right)$. While the meaning of ${\widehat{\mu }}_{A@E_j}\left(x_i\right)$ is the same as described above, ${\widehat{\nu }}_{A@E_j}\left(x_i\right)$ is a pair $\langle {\nu }_{A@E_j}\left(x_i\right),F_{{\nu }_{A@E_j}}\left(x_i\right)\rangle$ whose components denote the level ${\nu }_{A@E_j}\left(x_i\right)$ to which $x_i$ dissatisfies p and the collection $F_{{\nu }_{A@E_j}}\left(x_i\right)$ of the $x_i$’s features considered by $E_j$ for quantifying the nonmembership level. Hence, the contextualized evaluations performed by $E_j$ can be denoted by an augmented IFS (AIFS) (The terms Atanassov insuitionistic fuzzy set (AIFS) and augmented Atanassov intuitionistic fuzzy set (AAIFS) are also found in the literature. ), say ${\widehat{A}}_{@E_j}$, such that

$$\begin{array}{cc}\hfill {\widehat{A}}_{@E_j}=& \left\{\langle x_i,{\widehat{\mu }}_{A@E_j}\left(x_i\right),{\widehat{\nu }}_{A@E_j}\left(x_i\right)\rangle |\left(x_i\in X\right)\wedge \left(0\le {\mu }_{A@E_j}\left(x_i\right)+{\nu }_{A@E_j}\left(x_i\right)\le 1\right)\right\}.\hfill \end{array}$$

(3)

As can be noticed, the condition $0\le {\mu }_{A@E_j}\left(x_i\right)+{\nu }_{A@E_j}\left(x_i\right)\le 1$ has been inherited from the original definition of an IFS. A depiction of the contextualized evaluations performed by $E_j$ characterized as an AIFS is shown in Figure 1.

In [24], the authors make use of the above-mentioned characterization to define a FAST-GDM problem as follows:

Let ${\widehat{A}}_{@E_j}$ be an AIFS representing the contextualized evaluations given by an expert $E_j\in E$, and let

$$\begin{array}{cc}\hfill \widehat{A}=& \left\{\langle x_i,{\widehat{\mu }}_A\left(x_i\right),{\widehat{\nu }}_A\left(x_i\right)\rangle |\left(x_i\in X\right)\wedge \left(0\le {\mu }_A\left(x_i\right)+{\nu }_A\left(x_i\right)\le 1\right)\right\}\hfill \end{array}$$

(4)

be an AIFS representing the computed overall collective evaluation of the group of experts. Let also $cix(·,·)$ be a function, named concordance index, that is used for computing the level of concordance between ${\widehat{A}}_{@E_j}$ and $\widehat{A}$ such that it obtains a maximum value when the concordance between them is the highest. Under these considerations, a FAST-GDM problem runs into finding the most suitable option(s) with a general agreement among the experts. That is, finding the most suitable option(s) in such a way that the aggregation of the concordance indices (e.g., the average $\frac{1}{m}{\sum }_{E_j\in E}cix({\widehat{A}}_{@E_j},\widehat{A})$ where m denotes the number of experts) is maximized.

A way to compute a concordance index between the individual and collective evaluations is by means of a function $S(·,·)$ that computes the similarity between the AIFSs that represent those evaluations [28], i.e., the concordance index between ${\widehat{A}}_{@E_j}$ and $\widehat{A}$ can be computed through the expression $cix({\widehat{A}}_{@E_j},\widehat{A})=S({\widehat{A}}_{@E_j},\widehat{A})$.

Functions that compute the similarity between two IFSs, say J and A, have been proposed in [28] to compute the concordance index between the individual and collective evaluations. Among those functions, one can find the following proposed in [33]:

$$S_{SK1}(J,A)=1-f\left(l(J,A),l(J,A^c)\right),$$

(5)

$$S_{SK2}(J,A)=\frac{1-f\left(l(J,A),l(J,A^c)\right)}{1+f\left(l(J,A),l(J,A^c)\right)},$$

(6)

$$S_{SK3}(J,A)=\frac{{\left(1-f\left(l(J,A),l(J,A^c)\right)\right)}^2}{{\left(1+f\left(l(J,A),l(J,A^c)\right)\right)}^2}$$

(7)

and

$$S_{SK4}(J,A)=\frac{e^{-f\left(l(J,A),l(J,A^c)\right)}-e^{-1}}{1-e^{-1}},$$

(8)

where $A^c$ is the complement of A, i.e.,

$$A^c=\left\{\langle x_i,{\nu }_A\left(x_i\right),{\mu }_A\left(x_i\right)\rangle \right|(x_i\in X)\wedge (0\le {\mu }_A\left(x_i\right)+{\nu }_A\left(x_i\right)\le 1)\},$$

(9)

$l(J,A)$ represents the Hamming distance between A and J, i.e.,

$$l(J,A)=\frac{1}{2n}\sum _{i=1}^n\left(\phantom{X_2^2}|{\mu }_A\left(x_i\right)-{\mu }_J\left(x_i\right)|+|{\nu }_A\left(x_i\right)-{\nu }_J\left(x_i\right)|+|h_A\left(x_i\right)-h_J\left(x_i\right)|\phantom{X_2^2}\right),$$

(10)

and

$$f\left(l(J,A),l(J,A^c)\right)=\frac{l(J,A)}{l(J,A)+l(J,A^c)}.$$

(11)

It is worth mentioning that the flat operator, $⫣·⊩$, which turns an AIFS into an IFS by excluding the feature collections contained in each of its elements [28], can be used for converting ${\widehat{A}}_{@E_j}$ and $\widehat{A}$ into IFSs J and A respectively.

The above-mentioned concepts are used in the next section to describe a novel variant of the method for reaching consensus in FAST-GDM problems proposed in [24].

3. Increasing the Concordance by Exchanging the Most Influential Samples

As indicated in Section 1, a CRP and a SP are commonly used for solving a FAST-GDM problem [24]. During the CRP, each expert is first asked to evaluate the options. Then, the collective evaluations and the level of consensus are computed. If the computed level is not enough and asking the experts to perform a new round of evaluations is possible, the experts are given feedback on their evaluations and the CRP starts all over again. If the computed level is enough, the selection of the best suitable option(s) based on the computed collective evaluations is performed during the SP. Otherwise, the experts are notified that no consensus has been reached. The novel FAST-CR-XMIS, which aims at increasing the level of consensus in a CRP, is described in this section.

3.1. Idea behind FAST-CR-XMIS

During the evaluation of an option, an expert can recall one or more samples that show what he/she understands as suitable (or unsuitable) options for a given problem. Since such samples have an influence on his/her evaluation, the expert can use them to put the evaluation in context –cf. [34,35] where similar ideas have been used to handle subjective evaluations carried out by persons with different background. For instance, Figure 2 depicts a case in which an expert, say $E_j$, considers $s_{\mu @E_j}\left(x_i\right)$ as a good sample to put the evaluation of $x_i$ satisfying the proposition ‘$x_i$ IS A’ in context, which is reflected in the AAD ${\widehat{\mu }}_{A@E_j}\left(x_i\right)$. In this case, $E_j$ also considers that $s_{\nu @E_j}\left(x_i\right)$ is a good sample to contextualize the evaluation of the level to which $x_i$ dissatisfies ‘$x_i$ IS A’, which is reflected in the AAD ${\widehat{\nu }}_{A@E_j}\left(x_i\right)$. Notice that $s_{\mu @E_j}\left(x_i\right)$ and $s_{\nu @E_j}\left(x_i\right)$ are part of the training collection $X_{0@E_j}$ used by $E_j$ to acquire the knowledge $K_{A@E_j}$ about the (collection A of) suitable options. This knowledge is then used during the evaluation of the potential options included in a collection $X=\{x_1,\cdots ,x_n\}$.

The samples $s_{\mu @E_j}\left(x_i\right)$ and $s_{\nu @E_j}\left(x_i\right)$ detected by $E_j$ are included into two collections, $M_{{\mu }_A}$ and $M_{{\nu }_A}$, along with the samples detected by other experts. These collections of influential samples can be shared among the experts in such a way that the experts can study those samples and choose some of them to update their knowledge models.

In that regard, the idea behind FAST-CRP-XMIS is for experts to use their updated knowledge models to perform a new round of evaluations. Since the updated knowledge models of all the experts might be more aligned to each other after the exchange of the influential samples, an increment of the level of consensus among the new evaluations is expected. This idea is depicted in Figure 3. Notice that, after a round of contextualized evaluations, each expert $E_j$ can use the collections $M_{{\mu }_A}^{*}$ and $M_{{\nu }_A}^{*}$, which are subsets of $M_{{\mu }_A}$ and $M_{{\nu }_A}$ respectively, along with the training collection $X_{0@E_j}$ to update his/her knowledge $K_{A@E_j}$ –here, the use of $M_{{\mu }_A}^{*}$ and $M_{{\nu }_A}^{*}$ reflect the fact that $E_j$ might put his/her attention only on some of the samples included in $M_{{\mu }_A}$ and $M_{{\nu }_A}$ for updating his/her knowledge. After that, $E_j$ can use the updated knowledge to perform a new evaluation of the level to which $x_i$ satisfies ‘$x_i$ IS A’.

3.2. FAST-CR-XMIS Algorithm

The above-mentioned idea is implemented in Algorithm 1. The algorithm takes the same inputs used by the algorithm FAST-CR proposed in [24], i.e., a collection of experts (E), a collection of potential solutions (X), a consensus threshold ($\tau$) and the maximum number of iterations ($\eta$) that is allowed while trying to reach consensus. Like FAST-CR, FAST-CR-XMIS tries to obtain (a collection of) contextualized evaluations ($\widehat{A}$) so that the computed level of consensus is greater than or equal to the required consensus threshold ($\tau$). FAST-CR-XMIS returns a collection of collective evaluations and a flag that indicates whether a consensus has been reached or not.

Like in FAST-CR, four logical phases are identified in FAST-CR-XMIS: characterization, aggregation, quantification and feedback. In the characterization phase the evaluations performed by the experts (see Line 6) are characterized as AIFSs (see Lines 7–8). Such evaluations are aggregated and, then, included into the collection of collective evaluations during the aggregation phase (see Lines 10–15). The collective level of consensus is computed during the quantification phase (see Line 16). The experts are given feedback on their evaluations through the feedback phase (see Lines 26–30).

In addition to those four phases, a fifth assembling phase is considered in FAST-CR-XMIS (see Lines 18–25). During this phase, the most influential samples $s_{\mu @E_j}\left(x_i\right)$ and $s_{\nu @E_j}\left(x_i\right)$ detected by each expert are included into the collections $M_{{\mu }_A}$ and $M_{{\nu }_A}$ respectively.

Even though the main difference between FAST-CR-XMIS and FAST-CR is the assembling phase, another difference exists in the feedback phase. In FAST-CR, each expert is notified with a suggestion on how to modify the evaluation of $x_i$ taking into account the collection of $x_i$’s attributes $F_{{\mu }_A}\left(x_i\right)$ and $F_{{\nu }_A}\left(x_i\right)$, which are respectively part of the AADs ${\widehat{\mu }}_A\left(x_i\right)$ and ${\widehat{\nu }}_A\left(x_i\right)$ of the AIFS element $\langle x_i,{\widehat{\mu }}_A\left(x_i\right),{\widehat{\nu }}_A\left(x_i\right)\rangle$. In contrast, in FAST-CR-XMIS each expert is additionally notified with a suggestion on how to modify the evaluation of $x_i$ considering the most influential samples detected for $x_i$: while $M_{{\mu }_A}\left(x_i\right)$ is offered in the case of the level to which $x_i$ is a suitable option (see Line 29), $M_{{\nu }_A}\left(x_i\right)$ is offered in the case of the level to which $x_i$ is an unsuitable option (see Line 30). It is worth mentioning that the suggestions to modify the evaluations of $x_i$ in FAST-CR-XMIS are based on the most influential samples included in $M_{{\mu }_A}\left(x_i\right)$ and $M_{{\nu }_A}\left(x_i\right)$, which complement to the aggregated collections of attributes included in $F_{{\mu }_A}\left(x_i\right)$ and $F_{{\nu }_A}\left(x_i\right)$. Thus, the experts can choose between selecting the samples to update their understandings or using the values of the attributes to modify a specific evaluation.

Regarding the interpretation of the notification, while ${\mu }_A\left(x_i\right)-{\mu }_{A@E_j}\left(x_i\right)>0$ suggests that $E_j$ should increase ${\mu }_{A@E_j}\left(x_i\right)$ to an extent $|{\mu }_A\left(x_i\right)-{\mu }_{A@E_j}\left(x_i\right)|$, the expression ${\mu }_A\left(x_i\right)-{\mu }_{A@E_j}\left(x_i\right)<0$ suggests that $E_j$ should decrease ${\mu }_{A@E_j}\left(x_i\right)$ to the same extent. Likewise, while ${\nu }_A\left(x_i\right)-{\nu }_{A@E_j}\left(x_i\right)>0$ suggests that $E_j$ should increase ${\nu }_{A@E_j}\left(x_i\right)$ to an extent $|{\nu }_A\left(x_i\right)-{\nu }_{A@E_j}\left(x_i\right)|$, the expression ${\nu }_A\left(x_i\right)-{\nu }_{A@E_j}\left(x_i\right)<0$ indicates that $E_j$ should decrease ${\nu }_{A@E_j}\left(x_i\right)$ to the same extent.

Even though it is not explicitly mentioned in Algorithm 1, the study of the most influential samples and the update of the knowledge models carried out by each expert are expected to happen before a new round of evaluations. As previously stated, each expert might use the samples in $M_{{\mu }_A}^{*}$ and $M_{{\nu }_A}^{*}$, which are subsets of of $M_{{\mu }_A}$ and $M_{{\nu }_A}$ respectively, for updating the knowledge models. In this regard, since different knowledge models exist, each of them having specific update mechanisms, handling them all is outside the scope of this paper. However, an example that illustrates how those knowledge models work and can be updated is provided in the simulation presented in the next section.

Algorithm 1:FAST-CR-XMIS.

4. Simulation

In this section, a computerized simulation of a CRP in which a configurable number of experts try to reach consensus on the category of newswire stories is described. This simulation has been created to show how the novel FAST-CR-XMIS can help to increase the level of consensus among the participants in FAST-GDM problems.

As mentioned in Section 2, an AIFS ${\widehat{A}}_{@E_j}$ can be used for denoting the contextualized evaluations of a collection $X=\{x_1,\cdots ,x_n\}$ of potential options (newswire stories) satisfying the proposition ‘$x_i$ belongs to category A’, which are performed by an expert $E_j$ according to the knowledge $K_{A@E_j}$ that the expert has on how a typical story in category A looks like (see Figure 1). Since such an AIFS is used inside Algorithm 1, a learning process and an evaluation process are needed to obtain $K_{A@E_j}$ and ${\widehat{A}}_{@E_j}$ respectively. For the sake of illustration, the learning process and the augmented evaluation process applied in explainable support vector machine classification (XSVMC) [36] have been used for this simulation–other techniques like those proposed in [37] can also be applied.

To develop the simulation, a collection consisting of 21578 newswire stories provided by Reuters, Ltd., named Reuters-21578 [38], has been used. Among those newswire stories, 5108 stories related to one or more categories in $C= \{$acq, corn, earn, grain, ship, wheat} were distributed among a configurable number of m experts ($m\ge 2$) to build a training collection $X_{0@E_j}$ for each expert $E_j$ where $j\le m$. For instance,

Table 1. Example of the distribution of newswire stories among experts $E_1$, $E_2$ and $E_3$.

Category	${\mathit{E}}_1$	${\mathit{E}}_2$	${\mathit{E}}_3$
acq	551	551	386
corn	61	61	37
earn	960	960	789
grain	145	145	104
ship	66	66	59
wheat	71	71	56

shows the distribution of newswire stories among experts $E_1$, $E_2$ and $E_3$ (i.e., $m=3$). Notice that the number of stories assigned to $E_3$ differs from $E_1$ and $E_2$ to imitate by some means the heterogeneity of this group.

To obtain a knowledge model $K_{A@E_j}$, the XSVMC learning process requires each of the stories in $X_{0@E_j}$ being associated with a label that indicates whether the story belongs to the category A. Thus, to obtain, e.g., $K_{corn@E_j}$, which represents the knowledge about the category corn possessed by $E_j$, the articles in $X_{0@E_j}$ were labeled following an ‘one-versus-the-rest’ strategy, i.e., the stories belonging to corn were labeled as positive examples, while the stories that do not belong to this category were labeled as negative examples.

To obtain a collection of contextualized evaluations ${\widehat{A}}_{@E_j}$, the XSVMC evaluation process requires a knowledge model $K_{A@E_j}$ and a collection X consisting of the stories subject to evaluation. Hence, to simulate the evaluations of the level to which the stories in X belong to the category corn performed by $E_j$, $K_{corn@E_j}$ along with X were used as input in the XSVMC evaluation process. The main advantage of using XSVMC for the simulation is that the XSVMC evaluation process makes use of the most influential support vectors to contextualize the evaluations and, thus, it makes the obtention of the most influential samples (i.e., newswire stories) easier (see Lines 21–25 in Algorithm 1).

To compute the concordance index between the collection ${\widehat{A}}_{@E_j}$ consisting of the evaluations performed by $E_j$ and the collection $\widehat{A}$ consisting of the collective evaluations, Equations (5)–(8) have been used in the simulation–the interested reader is referred to [39] for an open-source implementation of these concordance indices in FAST-GDM problems. If the computed collective concordance index is less than the required level of consensus in a particular round (see Line 16 in Algorithm 1), the most influential samples are incorporated into the training collections and a new XSVMC learning process is performed for each expert before the next round of evaluations is initiated (see Figure 3).

To measure the effect of the updated (knowledge) models on the level of consensus, the collective concordance indices ${\tau }_{first}^{*}$ and ${\tau }_{last}^{*}$, corresponding to the first and last rounds respectively, were computed in 420 simulated FAST-CR-XMIS processes. Each category $A\in C$, each number of experts $m\in \{2,\cdots ,8\}$ and 10 different test collections, say $X_1,\cdots ,X_{10}$, each containing between 15 and 19 newswire stories, were used as input of these FAST-CR-XMIS processes. The results are presented in the next section.

5. Experimental Results

The averages ${\overline{\tau }}_{first}^{*}$ and ${\overline{\tau }}_{last}^{*}$ of the computed collective concordance indices ${\tau }_{first}^{*}$ and ${\tau }_{last}^{*}$ per category corresponding to the FAST-CR-XMIS processes simulated with $m=3$ and $m=8$ experts are shown in

Table 2. Average of the computed collective concordance indices per category ($m=3$).

Category	SK1			SK2			SK3			SK4
	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var
acq	0.85	0.94	11	0.75	0.89	20	0.57	0.81	41	0.78	0.91	16
corn	0.87	0.95	10	0.77	0.91	18	0.60	0.83	38	0.81	0.92	15
earn	0.85	0.94	11	0.74	0.89	20	0.57	0.80	42	0.78	0.91	17
grain	0.87	0.95	10	0.77	0.91	18	0.60	0.83	38	0.80	0.92	15
ship	0.87	0.95	10	0.77	0.91	18	0.60	0.83	38	0.81	0.93	15
wheat	0.87	0.95	10	0.77	0.91	18	0.60	0.83	38	0.80	0.92	15

and

Table 3. Average of the computed collective concordance indices per category ($m=8$).

Category	SK1			SK2			SK3			SK4
	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var	${\overline{\mathit{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathit{\tau }}}_{\mathit{last}}^{*}$	%var
acq	0.83	0.94	13	0.72	0.90	24	0.53	0.81	51	0.76	0.91	20
corn	0.85	0.95	12	0.74	0.91	22	0.56	0.83	47	0.78	0.93	19
earn	0.83	0.94	14	0.72	0.89	25	0.53	0.80	52	0.76	0.91	21
grain	0.84	0.95	12	0.74	0.91	23	0.56	0.82	48	0.78	0.92	19
ship	0.85	0.95	12	0.74	0.91	22	0.56	0.83	47	0.78	0.93	19
wheat	0.85	0.95	12	0.74	0.91	22	0.56	0.82	47	0.78	0.92	19

respectively–the tables corresponding to the FAST-CR-XMIS processes simulated with 2, 4, 5, 6 and 7 experts are shown in Appendix A. In these tables, the collective concordance computed with SK1 (cf. Equation (5)), SK2 (cf. Equation (6)), SK3 (cf. Equation (7)), and SK4 (cf. Equation (8)) are listed. For example, the average of the collective concordance indices computed with SK1 after completing the first and last rounds of the FAST-CR-XMIS process simulated to reach consensus on the category corn with $m=3$ experts are ${\overline{\tau }}_{first}^{*}=0.87$ and ${\overline{\tau }}_{last}^{*}=0.95$ respectively (see Table 2). In this case, the percent variance is computed by $({\overline{\tau }}_{last}^{*}-{\overline{\tau }}_{first}^{*})/{\overline{\tau }}_{first}^{*}=10\%$. Notice that, independently of the function used to compute the concordance indices, the percent variance is positive for each category.

Such positive increments of the concordance indices are also depicted in Figure 4. Notice that the increments of the concordance indices in FAST-CR-XMIS processes simulated with 8 experts are greater than the increments of the concordance indices in FAST-CR-XMIS processes simulated with 3 experts. Bear in mind that the higher the concordance the higher the consensus.

Figure 5, Figure 6, Figure 7 and Figure 8 show, in that order, the variation of the concordance indices ${\tau }_{first}^{*}$ and ${\tau }_{last}^{*}$ computed with SK1, SK2, SK3 and SK4 in FAST-CR-XMIS processes about the category corn simulated with different numbers of experts ($2\le m\le 8$). Notice that, regardless the function used to compute the concordance indices, in general the variation of ${\tau }_{last}^{*}$ is less than the variation of ${\tau }_{first}^{*}$.

Table 4. Average of the computed collective concordance indices per category ($m=3$).

	SK1		SK2		SK3		SK4
	${\overline{\mathbf{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{last}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{last}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{last}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{first}}^{*}$	${\overline{\mathbf{\tau }}}_{\mathit{last}}^{*}$
Mean	0.847	0.951	0.742	0.908	0.561	0.828	0.780	0.925
Variance	0.004	0.001	0.007	0.002	0.013	0.007	0.006	0.002
Observations	420	420	420	420	420	420	420	420
df	419		419		419		419
t-value	−51.165		−61.712		−72.570		−57.207
p-value	0.0		0.0		0.0		0.0

shows the results of the t-test for the null hypothesis “the average of the collective concordance indices is the same after performing a simulated FAST-CR-XMIS process.” Notice that the t-values are statistically significant ($p<0.05$). This indicates that the concordance indices ${\tau }_{first}^{*}$ and ${\tau }_{last}^{*}$ are significantly different from each other after performing the simulated FAST-CR-XMIS processes.

6. Discussion

The results suggest that the collective concordance indices increase significantly after performing simulated FAST-CR-XMIS processes. Such increments are independent of the function that is used for computing the concordance indices, as well as the number of experts that participate in a FAST-CR-XMIS process. This means that exchanging the most influential samples during such simulated CRPs can increase the level of consensus.

Nevertheless, the results should be interpreted with caution since all the samples deemed to be the most influential were used for updating the experts’ knowledge models during the simulated CRPs, i.e., during the simulation, the collections $M_{{\mu }_A}^{*}$ and $M_{{\nu }_A}^{*}$ have been deemed to be equal to the collections $M_{{\mu }_A}$ and $M_{{\nu }_A}$ respectively (see Figure 3). In addition, each contextualized evaluation has been associated with the most influential sample in the simulations. In a real scenario, the experts might partially share the samples that influence their evaluations. Also, the experts might only consider a few of the shared samples to update their understandings of the suitable options for a given problem.

Another note of caution is the assumption of a cooperative environment where all the experts are willing to share their samples. In this regard, situations where participants might be reluctant to share their experiences are subject to further study.

7. Conclusions

A novel algorithm for reaching consensus in FAST-GDM problems has been proposed in this paper. The algorithm, named FAST-CR-XMIS, aims at increasing the level of consensus in CRPs where participants are open to reconsider their evaluations after studying the most influential samples that have been identified and shared by other participants.

In FAST-CR-XMIS, participants can perform contextualized evaluations of the potential options to solve a FAST-GDM problem. By means of this kind of evaluations, participants can express not only the level to which a potential option is deemed to be suitable, but also the reasons that justify that level. Since such contextualized evaluations are mathematically represented by AIFSs, the participants can express not only positive but also negative aspects during a CRP.

The results of simulated CRPs suggest that FAST-CR-XMIS can increase the level of consensus among the participants. However, these findings may be somewhat limited by the assumption of a cooperative scenario where participants are willing to share their experiences and update their understandings. Further research should be undertaken to confirm the applicability of FAST-CR-XMIS to scenarios where participants are reluctant to share their experiences.

The applicability of FAST-CR-XMIS to recurrent CRPs in which a given group of experts is organized for carrying out periodical evaluations is also considered and suggested as future work.