Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology

: Multi-criteria decision analyses (MCDA) for prioritizations may be performed applying a variety of available software, e.g., methods such as Analytic Network Process (ANP) and Elimination Et Choice Translating Reality (ELECTRE III) as recently suggested by Kalifa et al. In addition to a data matrix, usually based on indicators and designed for describing the parts of the framework intended for the MCDA, these methods require input of a variety of other parameters that are not necessarily immediately obtainable. Often the indicators are simply combined by a weighted sum to obtain a ranking score, which is supposed to reﬂect the opinion of a multitude of stakeholders. A single ranking score facilitates the decision as a unique ordering is obtained; however, such a ranking score masks potential conﬂicts that are expressed by the values of the single indicators. Beyond hiding the inherent conﬂicts, the problem arises that the weights, needed for summing up the indicator values are difﬁcult to obtain or are even controversially discussed. Here we show a procedure, which takes care of potential different weighting schemes but nevertheless does not mask any inherent conﬂicts. Two examples are given, one with a small (trafﬁc) system and one with a pretty large data matrix (food sustainability). The results show how decisions can be facilitated even taking a multitude of stakeholder opinions into account although conﬂicts are not necessarily completely eliminated as demonstrated in the second case.


Ranking
Ranking is a means to support decisions. The advantage of a ranking approach is that even if a best solution is found but not realizable, ranking provides other acceptable solutions. PROMETHEE [1][2][3], AHP [4], ANP [5], ELECTRE-family [6], and others are methods to obtain a ranking based on a data matrix and additional supporting information, such as weights, associated with indicators (ELECTRE, PROMETHEE). On one side, these methods are highly sophisticated; however, for a public understanding, they are rarely understandable. On the other side, partial order can explore the data matrix without complicated algebraic/arithmetic transformations but will not in all cases allow a unique decision, because the values of the single indicators lead to many conflicts which may be important for their own right, but clearly hampering a decision.
In the literature on application of partial ordering in decision making, many attempts can be found where-while keeping the framework of partial order-the degree of comparability (in a ranking the degree of comparability gets its maximum) is enhanced. Here, we present a procedure called General Linear Aggregation (GLA) [7] which not only enhances the degree of comparability, but also simultaneously covers the fact that different stakeholders may have different weightings in mind. The general idea with GLA is not to let the

Semantics
The advantages of the partial order-based approach may lose some value when the number of incomparabilities is increasing. To be clear: partial order may or even will disclose incomparabilities resulting from a multi-indicator system, which indicates the presence of conflicts within the set of indicators (details below).
It is appropriate to clarify the semantics of "importance of indicators" and "weighting schemes". Within partial ordering, the concept "importance" has unfortunately often been associated with exchangeability of indicators expressing their low mutual importance. In connection with "weighting schemes" concerning the indicators importance is simply to stress the contextual importance of the single indicators.

Partial Ordering-Basics
The basics of partial ordering is the relation between objects (in our paper, either the four single transport systems or the 78 countries) to be ordered. The set of objects is called X. A priori, the data are analyzed without any pretreatments such as, e.g., aggregation of the single indicators. The only mathematical term applied in this context is "≤" (cf. e.g., [10,11]). By this, a relational instead of a numerical point of view is taken (cf. discussion). Two objects, x and y, are connected with each other if and only if the relation x ≤ y holds (see below, Equation (1)).

Main Equation, Compensation, Binary Relations
Object, x, characterized by the a set of indicators rs(x), s = 1, ..., m, can be compared with object y, characterized by the same set of indicators rs(y), if rs(x) ≤ rs(y) for all s = 1, . . . , m Application of Equation (1) needs a convention about the orientation of the single indicators, i.e., the larger the value of an indicator, the better a non-measurable quantity (a "latent construct") by indicators being mutually co-monotone. Equation (1) is the basis for a comparison of objects. As the indicator values are not numerically combined, the problem of compensation (a "good" value of an indicator may compensate a "bad" one of another indicator) is avoided, which is one of the main advantages of partial ordering [12,13]. By several steps, the binary relations due to Equation (1) are prepared for a representation by a graph, the Hasse diagram [10,14]. Independent of a graphical representation, the result of the application of Equation (1) is a partially order set (poset) of the objects.

Hasse Diagram, Incomparabilities, Chains
In the Hasse diagram, comparable objects are connected by a sequence of lines [10,14]. If Equation (1) is not fulfilled for some objects x, y, then x is incomparable with y, denoted by x y. Generally, incomparability expresses that the data lead to conflicts between the objects. A subset X' of X where for every x, y a ≤ -relation can be found is a chain, the number of objects constituting a chain is denoted its length. A subset X" of X, where for every x, y ∈ X" x y, is called an antichain. The number of incomparabilities, U, is an important characterizing quantity.
If for X a chain emerges, then U = 0 and the typical ranking (a linear order) is obtained.

Extension/Enrichment
An extension/enrichment of a partial order is transforming incomparabilities of the original poset into comparabilities (within the new poset) by maintaining the already given comparabilities.

Framework of Enrichments
As mentioned in Section 1, there are many attempts to enhance the degree of comparability without leaving the principles of partial order. As a detailed description is by far outside the scope of the paper, a table may be sufficient at this stage (Table 1). At the top is just the poset approach directly applied to data.
In the middle, there are more involved techniques to enrich the posets, and finally at the bottom, there are linear orders or construction of composite indicators, which condense the complexity inherent in data matrices to unidimensional representation. In that context, it is worth citing Arcagni et al. [15]: "Admittedly, however compressing the input posets into a simple linear order can be somewhat artificial and misleading . . . ". Table 1. Attempts to enhance the degree of comparability.

Method U (Number of Incomparisons) Remark References
Application of Equation (1) May be very large. Data matrix analyzed without any pretreatment by partial order leading to an "input poset".
No external information needed beyond the data matrix [14,16] Weights not as a sharp number but elements of certain intervals U will be reduced Stakeholders have to find intervals for the weights [17] Different weighting systems U will be reduced GLA (more details below) [18] Matrix of mutual ranking probability (MRP) U will be reduced Dominance structure of posets [15,19,20] Bucket order U will be reduced A systematic procedure to reduce U until the value 0 [15,[21][22][23] POSAC (Partial Order Scalogram Analysis by coordinates) U will be reduced A bidimensional representation is searched keeping as much as possible the original comparabilities [24]

Method U (Number of Incomparisons) Remark References
Ranking due to mean of different heights U = 0 Any partial order can be equivalently described by a set of linear orders. The vertical position of an object within a linear order is called its height.

Enrichment of the MIS-Generalized Linear Aggregation
Although the procedure is explained in detail in [7], we give a brief explanation for the sake of convenience of the reader.

Need of Normalization/Data Pretreatment
Whereas the original partial order method does not need a column wise normalization, the intended numerical combination of weights and indicator values (cf. Equation (4)) makes it favorable to ensure that both the indicator values and the weights are in the same order of amount. Hence, when indicators of different dimensionality are to be combined by weights, they both must be normalized to a common [0, 1] scale. Still worse is when indicators are on an ordinal scale, then the step of normalization is a critical and important step and there will be a trade-off: Either try to analyze all the incomparabilities, based on the original ordinal indicators, or perform a transformation which-in the last consequence-is a perhaps an acceptable data manipulation.

Orientation
In the first example, the 15 criteria values are oriented in both directions. In order to obtain equal orientation of the MIS (co-monotony of the single indicators) the six criteria c2, c9, c11, and c13-c15 (cf. Table 1 below) were all multiplied with −1 to make sure that for all criteria the higher values the better) A normalization of the indicators is performed as follows: r s = (r s (x) − min(r s (. . .)))/(max(r s (. . .)) − min(r s (. . .)) i f r s (x) > 0 −1 · (abs(r s (x)) − min(abs(r s (. . .))))/(max(abs(r s (. . .))) − min(abs(r s (. . .))) i f r s (x) < 0 One may consider the application of Equation (2) as an introduction of preference functions within the framework of partial order.
Therein, min or max of r s ( . . . ) and abs(r s ( . . . )), respectively, is to be taken over the considered objects.

Aggregation Process
When the MIS (denoted as "MIS(old)" in order to emphasize the role of the aggregation process) is written in the form, where r r are the indicators and e i the studied elements then the aggregation to a single scalar, Cf (composite indicator), which serves as a ranking index can be formulated by means of the weights for one single stakeholder, i.e., one single weight scheme, (Equation (4)) Cf· · · =· · · (g 1 · · · g 2 · · · . . . · · · g m ) · · · MIS(old), i.e., Cf = ∑ g i ·r i old (4) where the selection of weights is responsible for the composite indicator Cf but is based on a system of indicators riold, where riold refers to the members of MIS(old).
Thus, performing the matrix-multiplication Equation (4), where a row of m entries is acting on each column of matrix MIS, the traditional weighted sum as expression for the aggregation process is obviously obtained. The difficulty in Equation (4) is not its mathematics, but the way how the weights can be found.
The weights bear important information concerning the roles played by the single indicators of an original MIS. There is no need for any equalizing of indicators as pointed out in [29]. The aggregation to a set of single scalar can be formulated as: Equation (5) describes the calculation of a new MIS by a conventional matrix multiplication of a weight matrix, called G (having m columns, corresponding to the m indicators of the original MIS and as many rows as alternative weighting models that are/can be constructed) and the matrix MIS(old). Each row of matrix G is denoted as a weighting-scheme. Equation (5) can be more formally written as shown in in Equation (6), where the role of G as an operatorĜ is stressed.Ĝ *MIS(old) = MIS(new) As mentioned above, in practical application of Equation (5), it is more convenient to accept any number for the weights, and to normalize them before Equation (5) is applied. It should be noted that preference functions are needed in other MCDA too and most MCDA further need weights, ignoring the inherent uncertainty in weight findings.
The advantages of the procedure, formulated by Equation (6) are that (a) the system of weight-regimes can be checked; for example, the matrix G can be analyzed by correlation measures, or even by posetic tools. (b) Equation (5) can also be written as a mapping, performed by an operatorĜ.Ĝ can be applied to set of indicators of the MIS(old) leading to a set of new indicators, MIS(new) (Equation (7)).Ĝ results, wherein Y stands for a subgroup of stakeholders, e.g., A, or B or C, respectively. The ≥-sign indicates that the combination of subsets itself can generate new incomparabilities. The number of incomparabilities in a new MIS cannot be larger than that of the old MIS and will typically be lower.
Equation (8) allows to interpret the role of the sub-operators and hence of the different groups of stakeholders. When for instance selectedĜ(Y) (Y is associated with any subgroups of stakeholders) in that manner that it is approximately just an m*m-unit matrix, then the corresponding MIS(new,Y) is just the MIS(old). Then, with respect to the incomparabilities, the role of other subsets no longer plays any role.
As mentioned above, one advantage of the procedure presented (Equations (5)- (8)) is that the multitude of stakeholders' opinions can be mapped onto a matrix, that subsequently can be evaluated in a holistic manner. Here, the correlation measures characterizing G, i.e., the mutual correlations between the weight schemes, may be checked. Hence, if, e.g., the correlation between two regimes is very high, then one of the two rows (of matrix G) apparently is redundant and could be but must not be excluded. A perfect correlation (i.e., value 1 between two weight regimes) indicates that one of the two weight regimes could be ignored.

Software
All partial order analyses were carried out using the PyHasse software [30]. PyHasse is programmed using the interpreter language Python (version 2.6) [30]. Today, the software package contains more than 100 specialized modules and is available upon request from the developer, Dr. R. Bruggemann (brg_home@web.de).

Evaluation Transportation Variants in Kampala
The original MIS is adopted from Kalifa et al. [8] (Table 2) and the eventual normalized MIS is given in Table 3. It is emphasized that for the criteria c2, c9, c11, c13, c14, and c15, the lower the values the better, whereas for the remaining 9 criteria, the higher the better is valid.

Application of Two Different G Matrices
In the present study, we applied two different G-matrices, comprising 4 and 6 weightregimes, respectively. It should be noted that the weight-regimes applied here are partly artificially generated for illustrative purposed. In real life cases, the different weightregimes are typically offered by the participants of the decision process. Thus, the number of weight-regimes is practically determined by stakeholders' opinions.
The first G matrix includes in addition to the regime, where all weights have the value 1, the weights reported [8] (or) and subjectively chosen weights by the authors of the present paper (br and ca). The weight matrix G1 is shown in Table 4. Base on the Pearson correlations between the single regimes it is concluded that they are all mutually only to a minor extent correlated and as such that all should be taken into account. Table 4. Weight-regimes of indifferent-weights ("cw1"), Kalifa et al. ("or"), Bruggemann ("br") and Carlsen ("ca"). The correlation (unsquared) is relatively large for "or" and "br", thus the two weight schemes will not describe too different ideas as to how the coefficients in the weighted sum are to be selected (cf. Table 5). The other two correlations are positive too, but remarkably lower. Hence, one could expect that the weight scheme "ca" infers another conceptual idea. As mentioned above, the possible presence of strongly correlated weight-regimes as such do not pose a problem. Thus, in such cases it will be enough to include only one, i.e., reduced the G matrix, as including highly correlated weight-regimes will yield no new information.

Regimes
Based on the G1 matrix (Table 4) and the normalized original MIS (Table 3) a new MIS is generated ( Table 6), remembering that the criteria c2, c9, c11, c13, c14, and c15 are multiplied by −1 to secure identical orientation. The same ranking is obtained as was shown in [8]. Although four different weight schemes are applied, with correlation coefficients varying from 0.1378 to 0.7589, the result is an invariant, namely a chain BB < TT < Kam < Coa. Its visualization as Hasse diagram is shown in Figure 1. Based on the G1 matrix (Table 4) and the normalized original MIS (Tabl MIS is generated ( Table 6), remembering that the criteria c2, c9, c11, c13, c14, a multiplied by −1 to secure identical orientation. The same ranking is obtained as was shown in [8]. Although four differ schemes are applied, with correlation coefficients varying from 0.1378 to 0.7589 is an invariant, namely a chain BB < TT < Kam < Coa. Its visualization as Hass is shown in Figure 1. The second G matrix was generated based on 6 randomly generated weigh rd1-rd6. The random selection was [0, 25], respectively. The resulting G matrix Table 7. A Pearson correlation showed that the 6 random weights display onl correlation (in absolute terms) ( Table 8).   Table 4).
The second G matrix was generated based on 6 randomly generated weight-regimes, rd1-rd6. The random selection was [0, 25], respectively. The resulting G matrix is given in Table 7. A Pearson correlation showed that the 6 random weights display only very low correlation (in absolute terms) ( Table 8).
As can be seen, in absolute terms the maximum correlation coefficient is found for the weight regimes rd1 and rd6 (−0.4426).

Evaluation of Food Sustainability within 78 Countries
The data for this part of the study were adopted from the Food Sustainability Index (FSI) 2021 [9] and comprises the fraction of the FSI that deals with diet composition ([9], sect. 8.1). This sub study includes 4 indicators (Table 10).
"All indicator scores are normalized to a 0 to 100 scale, where 100 indicates the highest sustainability and greatest progress towards meeting environmental, social and economic key performance indicators (KPI) and 0 represents the lowest" ( [9], cf. Excel Workbook: Methodology). Hence, for all four indicators, the higher the indicator value, the better.
A fifth indicator (consumption of fruit and vegetables was left out due to the missing information concerning the weights of this indicators. The study comprises four possible weights, i.e., experts, uniform, outcome, and politics (vide infra). In total, 78 countries were included in the study [9]. In Table 11, the indicator values for the 78 countries are given.   Based on the above data (Table 11), a Hasse diagram was constructed ( Figure 2). Based on the above data (Table 11), a Hasse diagram was constructed ( Figure 2). Immediately, the diagram has a 'broad' structure and is highly complex and do nated by a high number of incomparabilities (1955) and a relative low number of com rabilities (1048). Although the number of incomparabilities is very high (around 66% p centage of all possible binary relations), there are nevertheless chains of length 8. B special program of the software package PyHasse, details about chains can be fou Here, however, a more detailed discussion is out of the scope of the paper. The main po is that the Hasse diagram is complex; however, there are tools to unfold the jungle of lin In the analog study from 2017 [31], four different weighting schemes were discus corresponding to the relative weighting of the four indicators by four different stakeho ers (Expert, Political, Outcome, and Uniform). The four weight schemes are given in Ta 12.  Immediately, the diagram has a 'broad' structure and is highly complex and dominated by a high number of incomparabilities (1955) and a relative low number of comparabilities (1048). Although the number of incomparabilities is very high (around 66% percentage of all possible binary relations), there are nevertheless chains of length 8. By a special program of the software package PyHasse, details about chains can be found. Here, however, a more detailed discussion is out of the scope of the paper. The main point is that the Hasse diagram is complex; however, there are tools to unfold the jungle of lines.
In the analog study from 2017 [31], four different weighting schemes were discussed corresponding to the relative weighting of the four indicators by four different stakeholders (Expert, Political, Outcome, and Uniform). The four weight schemes are given in Table 12. Applying the GLA methodology using the Tables 11 and 12 gives rise to a new data matrix incorporation the original data as well as the four stakeholder opinions/weight schemes, the new indicators being denoted r1GLA, r2GLA, r3GLA, and r4GLA, respectively (Table 13). The corresponding Hasse diagram is shown in Figure 3.  Table 12 and the data obtained by GLA and shown in Table 13.
In Figure 3, we see a significantly enriched and much 'slimmer' Hasse diagram with only 285 incomparabilities and 2718 comparabilities.  Table 12 and the data obtained by GLA and shown in Table 13.
In Figure 3, we see a significantly enriched and much 'slimmer' Hasse diagram with only 285 incomparabilities and 2718 comparabilities.
Despite the obvious enrichment of the diagram, there are now chains of the length 30, we are not in this system achieving a strict linear order as was the above example studying various transportation forms. However, here, the opinions of all four stakeholders are simultaneously taken into account. Furthermore, the remaining incomparabilities, surviving the effect of the G*-operator motivate to further investigations, which is causing the conflicts. The relatively high number of comparabilities (compared to the original poset) indicates that the stakeholder opinions do not oppose each other but give gradually some more importance to the single indicators of the 4-indicator set. Hence, a possible subsequently decision process is remarkably facilitated.
It is here worth mentioning that simply the shape of Hasse diagrams can be a valuable tool in the analysis of complex data structures [32]. The input poset (i.e., the poset obtained by Equation (1)) of the food sustainability is broad and rather flat, whereas the Hasse diagram after GLA is rather slim and has a remarkable vertical range (cf. chain of lengths of 30).
A subsequent calculation of the average rank [26,33,34] makes much more sense than based on the diagram in Figure 2 although here the essential role of conflicts is no more visualized. In Table 14, the top 10 and bottom 10 countries based on an average ranking are shown. Looking at the data shown in Table 14, it is immediately clear that USA, AUS, and ARG are virtually non-sustainable based on meat consumption (i2). Moreover, in the indicators i1 and i3, these three countries display non-sustainability, whereas in the case of salt consumption (i4), the three countries display values around 50.

Discussion
There are some points which should be stated in a clear manner: (a) Partial order takes a relational point of view, even if numerical algorithms, as indicated by Equation (5) are applied. Hence, the MIS(new) will once again analyzed in terms of a graph, indicating comparabilities and incomparabilities. Consequently, the data are only used to decide whether a ≤ -relation can be established. This is seen as some zooming out; however, clearly numerical details must be a posteriori analyzed. (b) In Equation (5), needs weights are combined with indicator values and then summed up. In a strict mathematical reasoning, this can only be done when the scaling level is metric. If this is not the case, or when the indicator values have very different ranges, which may depend on the used unit of measurements, then a normalization is needed. A normalization in turn requires metric values; when MIS(old) contains ordinal indicators, then a normalization is a crucial step which needs a carefully justification. (c) The characterization of the weight scheme (matrix G) can be performed in many ways, as, e.g., different correlation measures can be applied. Further, G itself can be investigated by partial order methods to disclose whether some weight regimes dominate some others.

Conclusions and Outlook
As mentioned above, there are constellations where partial ordering is insufficient and delivers only antichains or has an uncomfortable high degree of incomparabilities. Then, the generalized aggregation (Equation (5)) is still simple and remains within the theoretical framework of partial order methodology, since based on the normalized MIS (Equation (2)) a new MIS is generated, which is important on its own right, because stakeholder opinions and (measured) data are included. The new MIS still may have conflicts that require a deeper contextual discussion. In the most general case, Equation (5) cannot be applied without introducing preference functions, which in the framework of partial order are simple [0, 1] linear transformations (whereby it is not excluded that sometimes other preference models could be considered). Thus, in the results section, the effect whereby normalization the ranges of weight and of indicator values is made comparable is discussed. In the example of the Kampala transportation variants, a linear order is obtained, which is the most comfortable case for decision making. That a linear order is not necessarily the result when the matrix G has more than one row, shows the second example of food. Certainly, on the one side, a decision is not that easy (because both, the data of the original indicators (MIS(old)) and the weight regimes must be checked), but on the other side, the incomparabilities indicate that there are still conflicts, which must be discussed in a fair decision process.
However, accepting Equation (5) which frees the decision makers from the process of finding weights in a crisp manner (i.e., as a number with possibly some decimals) seems to be simple enough to be of help in public decision makings.

Limitations and Future Work
Obviously, there is a need of future work. Thus, the interpretation of ordinal data as metric ones is a big and crucial step that needs attention. However, not in every case such a step can be justified. In that case, GLA breaks down. Therefore, there is a need to find an enrichment procedure, where stakeholder opinions can be collected and can influence the poset. It seems as if the methodological way should start with the set of linear orders, representing the input poset. Hereto, a good basic material is given by Patil and Joshi [35]. The attempts presented therein aim at a final linear order, whereas our intention is an enrichment procedure, which keeps the most important conflicts. Another starting point could be the paper of Arcagni et al. [15], where especially the bucket poset approach seems to be attractive.

The Novelty of the Here-Presented Approach
Modern MCDA methods such as, e.g., PROMETHEE and members of the ELECTRE family, are typically based on an intricate combination of stakeholders opinions and arithmetic operations according to the methodology used. However, these close interactions make it very difficult to judge the actual effect of the stakeholders and effects that can be assigned to the pure arithmetic procedure.
The here-presented method separates completely the role of stakeholders from that of the procedure. Thus, the outcome is that statistical measures, made as Pearson correlations are available as a further tool to understand the stakeholders' opinions as a whole. This task is facilitated by the-admitted-simplicity of the partial order methodology itself. The new here-described method is based on conventional matrix multiplication combined with the previously well-described partial order methodology cf., e.g., [7].
Author Contributions: Both authors have equally participated in all phases of the work leading to this paper. All authors have read and agreed to the published version of the manuscript.