4.2. Results of the Framework Application
When analyzing the defined set of indicators presented in
Table 1, it was decided that the 3rd and 6th ones would be evaluated in a quantitative way, due to the existence of data in the plant, and also the possibility of establishing objective values (goals) for them. The remaining ones were evaluated in a linguistic way. The minimal thresholds to meet in terms of the consensus and consistency indexes were the linguistic term (label) “high” and the value of 0.8, respectively. The results obtained for the indicators evaluated in a qualitative way are the first to be presented next.
Each expert began by assigning each indicator the linguistic label that, in accordance with his/her opinion, represented its current state within the company. The results are shown in
Table 2, where the indicators appear numbered in the first column in the exact order they were referred to in
Table 1, while the eight selected experts (E1, …, E8) appear in the first row. On the other hand,
Table 3 shows the result of the main parameters that determine the consensus level (
) achieved in the evaluation with respect to each indicator
, by using the linguistic quantifier “much”, with parameters
a and
b having values of 0.3 and 0.8, respectively.
For the sake of comprehension and demonstrative purposes, an exemplification of the computation of the parameters that determine the consensus level is presented next for the case of the indicator 1. The
values are as follows:
Given that, in this case, according to the used scale, the linguistic terms are consecutive ones, by considering Equation (4) we have that:
Similarly, in order to obtain , Equation (8) was implemented using the linguistic quantifier “much”, whose a and b values were 0.3 and 0.8, respectively.
Before using Equation (10) to obtain the value of
, it was necessary to first determine the vector
M, which is integrated by those linguistic labels that accomplish or meet the condition defined through Equation (9). In this sense, it was also necessary to determine the membership of the following value with respect to the linguistic terms appearing in
Figure 2.
In this regard, by using Equation (1), it is possible to see that this value presents a membership grade of 0.12 to the linguistic term “medium" (m), 0.88 to the linguistic term “high” (h), and 0 to the remaining terms of the scale. This way, we can conclude that the vector M consists of only one linguistic term, in this case “high”, which coincides with the value and, at the same time, with the consensus level value reached by the experts when evaluating the indicator 1.
As appreciated in
Table 3, in all of the cases the obtained consensus level was equal to or higher than the linguistic term “high”. This leads to the conclusion that the demanded quality level for the qualitative attribute evaluation process was met. In the execution of the step 8 of the methodology for obtaining the Global Consensus Level (
GLC), we first determined the
through the
values, see
Table 3. In this case, we obtained the following value:
By applying the linguistic quantifier to this value, in accordance with Equation (12), then we have that the GCL is “very high”, i.e., = vh.
It is important to emphasize the fact that our proposal for determining the consensus relation (
), unlike others, for instance the one presented in [
20], has an inclusive character, that is, it considers the opinions of all involved experts and, even more important, it is also sensitive to the proximity among the linguistic terms emitted in evaluation process, as is appreciated when comparing the
results and the corresponding
value in the case of the indicators 5 and 9.
In the case of these indicators, it can be seen that there is the same balance in terms of the proportion of experts who provide one or another linguistic label as a measure of the evaluation of both indicators, regardless of whether the linguistic labels used to evaluate are not the same for the indicator. In the case of indicator 5, 87.5% of the experts consider that it is at a “high” level and the 12.5% consider that it is in another state, “perfect”. Similarly, in the case of indicator 9, 87.55% of the experts consider that said indicator is in one state and 12.5% of the members consider that it is presented in another evaluation state. However, since our proposal takes into account the proximity among the linguistic terms (labels), and the labels used in evaluating indicator 9 are closer than those used for indicator 5, then , and consequently , are also lower than and respectively.
For both of these indicators the cardinality of the vector
is the same, even the
values for the emitted labels in each case are equal, however, given that the proximity among the labels through which indicator 5 was evaluated is inferior, then the
value is lower than the corresponding one to
. This fact would not have been detected nor considered if we had simply applied the methodology proposed in [
20], in which case the
value would have been equal to 0.875 for both indicators. This is, again, another of the added values of the presented research.
Another important aspect also lies in the same selection of the proportional linguistic quantifier. Here, it is necessary to find an adequate trade-off between accuracy and application cost, given that the results of might be overvalued if the quantifier “At least half” were to be used, or otherwise undervalued if the linguistic quantifier “All” were to be used instead.
The final evaluation of each indicator was obtained using the LOWA operator, as indicated in step 9 of the proposed framework. For the sake of comprehension and demonstrative purposes, an exemplification of the computing or determination is presented next for the case of indicator 1. In
Table 3 it is possible to see that the evaluation vector of indicator 1 includes the terms
p and
vh, with cardinalities of 2 and 6, respectively, which relate to and constitute the values
S6 and
S5 of the used scale’s term set. According to Equation (17), the weights’ vector (
w) can be calculated as follows:
For the case of two linguistic terms to be aggregated, the adjusted weight vector
coincides with the vector
W, which is calculated by means of Equation (17). In accordance with Equation (14), the LOWA can be represented as follows:
At the same time, and as mentioned before, this operator requires the non-increasing ordering of the linguistic terms, in accordance with their semantic, and therefore
constitutes the weight associated to the term
(the term with the higher semantic value), while 1 −
represents the weight associated to
(the term with the lowest semantic value). By substituting this into Equation (14),
k can be determined, and with it, the linguistic term (
) resulting from the aggregation.
The linguistic values defining the evaluation of the remaining qualitative indicators were also determined in a similar way. These results can be appreciated in the last column of
Table 2.
Next, the framework continues with the evaluation of the quantitative indicators, in this case the 3rd and 6th ones. In the case of the 3rd indicator, at the moment of the realization of this study in the company, it was well known that it behaved and reached values of around 90%; however, the defined reference value for it was 100%. As for the 6th indicator, it was also known that 25% of the production resources could be used in equipment repairs, however, the experts defined 30% as the reference value in this case. By implementing Equation (13), the following normalized values were obtained:
On the other hand, the linguistic quantifier “All” was used for the homogenization process with a and b values equal to 0.5 and 1, respectively. Similarly, by making use of Equations (11) and (10) for the values and , it was possible to determine, in the case of indicator 3, that the M vector included only one element (vh), for that reason, this is the linguistic term that defines its evaluation. In the case of indicator 6, the M vector only included the term h, which also corresponded to its evaluation.
All these results were presented to the experts, and it was concluded that there was an adequate correspondence among the obtained evaluations for each indicator and its current state in the production plant. Subsequently, the application of the framework proceeded with module II. The paired comparisons between indicators were performed by the experts in a joint way, by using the approach described in steps 11 to 13 (see
Section 3.3.1,
Section 3.3.2 and
Section 3.3.3). The results are shown in
Figure 3.
As the paired comparisons were performed by the group of experts in a joint way, the weighting process quality was analyzed only by the consistency index. The number of triplets with total inconsistency was five, for instance, the triplet 2-4-8 to just cite one. The number of triplets with partial inconsistency was 21, for instance, the triplet 1-5-7 to just cite another one.
Similarly, making the necessary substitutions in Equation (21), it was also possible to determine the consistency index achieved whose value was 0.87. This value was higher than the minimal established value of 0.8. In
Table 4, is also possible to appreciate the weight of each indicator, obtained by the Simple Ordering method.
Having, at this point, the evaluation and weight of each indicator, it was time to proceed with the aggregation process, which is detailed in the third module of the framework and leads to the obtaining of the PMII index. Before doing this, it was first necessary to use the negation operator over indicators 9 and 10, given that these were inversely proportional to the PMII index itself.
Table 5 summarizes the final evaluation of each indicator, the linguistic term to be aggregated, as well as their weight values.
The n fixed-weight values to aggregate using the LOWA operator () required the definition of an ordered vector B, where its components are the n linguistic terms ordered in a non-increasing fashion according to their weights. The aggregation process implied the realization of n − 1 iterations, where, in each iteration i, two terms were aggregated, one of these being the result of the aggregation at iteration i − 1 and the other one the (n-i)th term of the ordered vector B. The operator was applied over the following B and W vectors, which are:
According to vector
W’
s components, one subset of indicators of equal weight was identified. In this case, this subset is formed by indicators 5 and 9. The results of the aggregation of both indicators is shown in
Table 6. For these two indicators of equal weight, the components of the vector
are equal to 0.5, the same applies to the values
and 1 −
.
If we now use the linguistic term “high” (
h) as the evaluation value for the same indicators 5 and 9, the new vector
B can be expressed as follows:
On the other hand, by keeping invariable the weight vector
W and using Equation (20), it was possible to generate the corresponding adjusted weight vector
. Using this vector, it was then also possible to generate the weight components
and 1 −
which were to be used in each aggregating iteration.
Table 7 shows the result of the aggregation in each iteration. The last iteration provides the PMII index result, i.e., PMII = “very high” (
vh). This result was contrasted with the opinions of the group of experts, and their agreement with the result was also verified.
A Partial Comparative Analysis in the Calculation of the PMII Index
To partially and further demonstrate the feasibility of our proposal, we proceed to determine the same PMII index by applying the LOWA operator in its original version (
), as it appears in [
20,
23,
28]. In this case, the weights of the terms to be aggregated are calculated from the concept of the relative quantifier (
).
As explained in
Section 3.2.8, the application of the LOWA operator requires a non-decreasing ordering of the vector of linguistic terms
B of to be aggregated; this ordering will be performed according to its semantics. Thus, considering the evaluation of each indicator that appears in
Table 5, we have:
In this case, the weight vector will be calculated from the application of the proportional fuzzy quantifier
, expressed in a numerical domain
[0, 1], as shown in [
20]. Specifically,
indicates the degree to which a portion
r of objects satisfies the concept expressed by the quantifier
. This degree of satisfaction is calculated using the following equation:
With a and b the minimum and maximum values that define the semantics of the quantifier (, , [0, 1]).
In terms of the original LOWA version, the weight
of each indicator
i if is calculated using Equation (25):
where
j is the position occupied by indicator
i within the ordered vector
B.Table 8 shows the evaluations and weights of the set of ordered indicators of the vector
B calculated by Equation (25) according to this alternative comparison method, this using the LOWA operator in its original form. The majority proportional quantifier was used as defined in [
20] with
= 0.3 and b = 0.8.
Table 9 shows the iterations of the aggregation of the indicators of the ordered vector
B (from right to left as it is performed in the LOWA operator). As can be seen, indicators 9 and 10 are not considered in the aggregation process since both have a weight equal to zero. This is one of the limitations of the original approach and something we have solved with our modifications.
As a result of the application of this LOWA operator in its original formulation, the PMII index is evaluated as “high”. This result differs from the value obtained from our proposal since the original approach does not take into account the real weight of the indicators, but instead the weight is associated to and depends on its the evaluation of the indicator itself. This is something that in practice may be often far from reality, and thus we have proposed its modification.
To sum this partial comparison up, it is important to once more highlight the impact of the aggregation method proposed in this paper over the final result of the PMII index. In this sense, the idea of identifying subsets of indicators with the same weight and performing partial aggregations of these subsets, as was done in the case of indicators 5 and 9 (see
Table 6), in order to later substitute the result into a global aggregation which considers all indicators, produces a different result to the one that would have been obtained in the case of simply establishing an arbitrary ordering of the vector
B, by placing the linguistic term that defines the evaluation of indicator 9 (in this case
m) before the term that defines the evaluation of the indicator 5 (term
h), and considering both had equal weights. If this had been performed this way, the result of the PMII index after the global aggregation would have been “high” (
h) instead.
In addition, given the non-linear character and non-decreasing monotonic characteristics of the aggregation operator used, it can be also seen how it produces a result that reflects, to a better extent, the state of the indicators of higher relative importance, which is usually a very favorable element from the practical point of view. In this case study, the indicator 1, evaluated as “very high”, concentrates around 20% of the weight.
Finally, based on the case study and the research presented here, it is also possible for the authors of this paper to conclude that the use of different well-known evaluation methods, as for instance, the Likert numeric scales or numeric operators such as the Weighted Average, would have produced a different result, a result influenced by a lack of flexibility of the numeric evaluation methods and the linear character of the mentioned operator. However, as indicated in the framework’s last step, it is still and always up to the experts involved in the study to evaluate the effectiveness of the results achieved.