Efﬁciency of Methods for Determining the Relevance of Criteria in Sustainable Transport Problems: A Comparative Case Study

: Problems related to sustainable urban transport have gained in importance with the rapid growth of urban agglomerations. There is, therefore, a need to support decision-making processes in this area, a trend that is visible in the literature. Many methods have already been presented as a useful decision-making tool in this ﬁeld. However, it is still a signiﬁcant challenge to properly determine the relevance of the criteria because it is one of the most critical points of many presented techniques to solve decision problems. In this work, we propose two new approaches to determining the relevance of particular decision criteria effectively in sustainable transport problems. For this purpose, we examine a study case for the evaluation of electric bikes evaluated against eight criteria, which have been taken from earlier work. We calculate the relevance of each criterion using four different approaches and then evaluate their effectiveness using a reference ranking and popular multi-criteria decision analysis methods. The results are compared with each other by using similarity coefﬁcients. Finally, we summarize the results obtained and set out further methods of development.


Introduction
The massive desire to move to larger cities allows humanity to develop at faster rates. Nearly half of today's world lives in urban areas, and by 2045 the quantity of citizens will rise by 150%. Scientists estimate that the population of cities can reach 6 billion [1]. The number of motorized vehicles is increasing every day, and this number has already reached 1 billion. Growing quality of life provokes the production of millions of new units in the transport sector [2,3]. Urbanization is taking the transport system to the next level. The fundamental factor in human development is the ability to save time. The availability of affordable transportation at our fingertips allows us to save several hours a day. Critical attitudes towards Sustainable Development Goals reaffirm the importance of this issue [4]. However, an equally important aspect is the impact of so many vehicles on the environment. All this leads to increased air pollution, which in turn has dire consequences: worsening of the immune system, respiratory diseases, and premature death [5]. Scientists estimate the number of deaths caused by pollution to be 9 million in 2015, three times more fatalities than from AIDS, tuberculosis, and malaria united and 15 times more than from all kinds of violence [6].
The importance of urban transport is becoming more and more evident. Growing influence provokes the growth of city-mover complexity in terms of sustainability [6]. Scientists search hybrid cars. In such context, developing the methodological foundations for the rating of sustainable transport becomes more and more necessary. Natural loss of selected data as malfunction reports data, and loss of value of newly introduced variants over a few-year time span causes a specific problem that as a result forces the requirement to improve the methodological guidelines in the background of incomplete information in the model.
In modeling sustainable transport decision-making problems, a very big challenge remains to determine the relevance of the decision criteria. In the literature, there are methods to obtain the values of the criteria weights. However, these methods are not sufficiently investigated. In this article, we research effectiveness for determining the relevance of criteria in sustainable transport problems. Figure 1 presents the plan for the proposed research. We propose two new weighting methods which are based on ranking similarity coefficients. For these purposes, we present a comparative case study on the evaluation of electric bicycles. This is a continuation of previous work [54], where a modern Characteristic Object METhod (COMET) method was used, and a reference bicycle ranking was obtained. We compare the results for two proposed and two commonly used weighting method. Based on four different approaches for determining the relevance of criteria, we re-analyze the decision-making process using different standardization methods and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and VIKOR (in Serbian: VlseKriterijumska Optimizacija I Kompromisno Resenje) methods. The resulting rankings are then compared with a reference ranking that will help to determine the effectiveness of the investigated weighting methods. The main contribution of our research is, therefore, to propose two new approaches to the analysis of the relevance of decision-making criteria, and additionally to examine their effectiveness. The differences between the existing and proposed methods are significant and encourage further work in this direction. The rest of the paper is organized as follows. Section 2 contains a brief introduction to fuzzy set theory, MCDA methods, correlation coefficients and normalization methods. The investigated study case is described in Section 3, which was divided into two parts, describing the data used for research (Section 3.1) and the research methodology (Section 3.2). Section 4 is devoted to the presentation of results and their discussion, which takes place in Sections 4.1 and 4.2 for TOPSIS and VIKOR methods respectively. Finally, the conclusions are formulated in Section 5.

Fuzzy Set Theory
The idea of Fuzzy Set Theory was introduced by Lofti Zadeh in [55]. Fuzzy Set Theory is used in many scientific fields and could be especially useful for solving MCDA problems [56][57][58]. Here we present some definitions and basic concepts of the Fuzzy Set Theory which are necessary to understand COMET method [59][60][61]. Definition 1. The fuzzy set A in a certain non-empty space of solutions X is defined by (1). where is a membership function of the fuzzy set A. This function indicates the degree of the membership of the element in the set A. µ A (x) = 1 means full membership, 0 < µ A (x) < 1 means partial membership and µ A (x) = 0 means no membership at all.

Definition 2.
The triangular fuzzy number A(a, m, b) is a fuzzy set which membership function is defined as followed (3): and fulfill characteristics (4) and (5): Definition 3. The support of a TFN-subset of the A set in which all elements have a non-zero membership value in the A set (6).
Definition 4. The core of a TFN is a one-element fuzzy set (singleton) with membership value 1 (7).
Definition 5. The fuzzy rule-single fuzzy rule can be based on the Modus Ponens tautology. The reasoning process uses the IF − THEN, OR and AND logical connectives.

Definition 6.
The rule base-the rule base contains logical rules determining the relationships existing in the system between the input and output sets.

COMET
The Characteristic Objects METhod (COMET) is based on fuzzy logic and triangular fuzzy sets. The accuracy of the COMET method was verified in previous works [20,30,62]. The formal notation of the COMET must be recalled based on [63][64][65], and Figure 2 presents the flowchart of the COMET method as summarizing.
Step 0: Initiate the process Step 1: Modeling structure of the problem Step 2: Expert evaluation of the Characteristic Objects Step 5: Evaluation of the set of alternatives  Step 1. Definition of the space of the problem-the expert determines the dimensionality of the problem by selecting r criteria, C 1 , C 2 , . . . , C r . Then, a set of fuzzy numbers is selected for each criterion C i , e.g., {C i1 ,C i2 , . . . ,C ic i } (16): C 1 = C 11 ,C 12 , . . . ,C 1c 1 where C 1 , C 2 , . . . , C r are the ordinals of the fuzzy numbers for all criteria.
Step 2. Generation of the characteristic objects-the characteristic objects (CO) are obtained with the usage of the Cartesian product of the fuzzy numbers' cores of all the criteria (17): As a result, an ordered set of all CO is obtained (18): where t is the count of COs and is equal to (19): Step 3. Evaluation of the characteristic objects-the expert determines the Matrix of Expert Judgment (MEJ) by comparing the COs pairwise. The matrix is presented below (20): where α ij is the result of comparing CO i and CO j by the expert. The function f exp denotes the mental judgment function of the expert. It depends solely on the knowledge of the expert. The expert's preferences can be presented as (21): After the MEJ matrix is prepared, a vertical vector of the Summed Judgments (SJ) is obtained as follows (22): Eventually, the values of preference are approximated for each characteristic object. As a result, a vertical vector P is obtained, where the i-th row contains the approximate value of preference for CO i .
Step 4. The rule base-each characteristic object and its value of preference is converted to a fuzzy rule as (23): IF C C 1i AND C C 2i AND . . . THEN P i (23) In this way, a complete fuzzy rule base is obtained.
Step 5. Inference and the final ranking-each alternative is presented as a set of crisp numbers, e.g., A i = {α i1 , α i2 , α ri }. This set corresponds to the criteria C 1 , C 2 , . . . , C r . Mamdani's fuzzy inference method is used to compute the preference of the i-th alternative. The rule base guarantees that the obtained results are unequivocal. The bijection makes the COMET completely rank reversal free.

TOPSIS
The TOPSIS method is a simple MCDA technique used in many practical problems. Thanks to its simplicity of use it is widely used in solving multi-criteria problems. Below we present its algorithm [66]. We assume that we have decision matrix with m alternatives and n criteria is represented as X = (x ij ) m×n .
Step 1. Calculate the normalized decision matrix. The normalized values r ij calculated according to Equation (39) for profit criteria and (40) for cost criteria. We use this normalization method, because [67] shows that it performs better that classical vector normalization.
Step 2. Calculate the weighted normalized decision matrix v ij according to Equation (26).
v ij = w i r ij (26) Step 3. Calculate Positive Ideal Solution (PIS) and Negative Ideal Solution (NIS) vectors. PIS is defined as maximum values for each criterion (27) and NIS as minimum values (28). We do not need to split criteria into profit and cost here, because in step 1 we use normalization which turns cost criteria into profit criteria. v Step 4. Calculate distance from PIS and NIS for each alternative. As shows Equations (29) and (30).
Step 5. Calculate each alternative's score according to Equation (31). This value is always between 0 and 1, and the alternatives which got values closer to 1 are better.

VIKOR
The VIKOR method, similarly to the TOPSIS method, is based on distance measurements. In this approach a compromise solution is sought. The description of the method will be quoted according to [68,69]. Let us say that we have decision matrix with m alternatives and n criteria is represented as Step 1. Determine the best f * i and the worth f − i values for each criteria functions. Use (32) for profit criteria and (33) for cost criteria.
Step 2. Calculate the S i and R i values by Equations (34) and (35).
Step 3. Compute the Q i values using Equation (36). where and v is introduced as a weigh for the strategy "majority of criteria". We use v = 0.5 here.
Step 4. Rank alternatives, sorting by the values S, R, Q in ascending order. Result is three ranking lists.
Step 5. Normally, we should use S, R, Q ranking lists to propose the compromise solution or set of compromise solutions, as showed in [69,70]. However, in this paper would be used only Q ranking list.

Correlation Coefficients
Correlation coefficients make it possible to compare obtained results and determine how similar they are. In this paper, we would use the weighted Spearman correlation coefficient (37) and the rank similarity coefficient (38) to determine how similar obtained with COMET rankings to reference rankings.

Weighted Spearman's Rank Correlation Coefficient
For a sample of size N, the rank values x i and y i is defined as (37). In this approach, the positions at the top of both rankings are more important. The weight of significance is calculated for each comparison. It is the element that determines the main difference to the Spearman's rank correlation coefficient, which examines whether the differences appeared and not where they appeared.

Rank Similarity Coefficient
For a sample of size N, the rank values x i and y i is defined as (38) [71]. It is an asymmetric measure. The weight of a given comparison is determined based on the significance of the position in the first ranking, which is used as a reference ranking during the calculation.

Normalization Methods
In the literature, there is no clear assignment to which decision-makers' methods of data normalization are used. This situation poses a problem, as it is necessary to consider the influence of particular normalization on the result. The most common normalization methods in MCDA methods can be divided into two groups [72], i.e., methods designed to profit (39), (41), (43) and (45) and cost criteria (40), (42), (44) and (46).
The minimum-maximum method-in this approach, the greatest and the least values in the considered set are used. The formulas are described as follows (39) and (40): The maximum method-in this technique, only the greatest value in the considered set is used. The formulas are described as follows (41) and (42): The sum method-in this method, the sum of all values in the considered set is used. The formulas are described as follows (43) and (44): The vector method-in this method, the square root of the sum of all values. The formulas are described as follows (45) and (46):

Entropy Method
According to [73], entropy method is based on a measure of uncertainty in the information. It is calculated using Equations (47)-(49) below.

Standard Deviation Method
This method is similar to entropy at some point, and assigns small weights to an attribute which has similar values across alternatives. The SD method defined with Equations (50) and (51), where w j is weight of criteria and σ j is standard deviation [73].

Materials
The material used for this case study will be the data from the article [54], which concerns the assessment of preferences for electric city bikes. For the analysis, eight criteria were used, which are presented in detail in Table 1. It should be noted that each criterion has a different numerical domain.
This will be of particular importance when determining the relevance weighting of criteria. In Table 1, there are also characteristic values determined for particular criteria, which is one of the most important steps in the COMET method. Visualization of linguistic values, i.e., low, medium and high, is shown in Figure 3. All the presented fuzzy numbers are asymmetrical because they are matched to the data distributions identified by the expert. The identified structural model was transformed into a monolithic model, which was only auxiliary. The values of the preference, and thus the rankings of the identified decision-making sets, remained unchanged. In our work, we focus only on those alternatives that were described with complete information. Therefore, 55 alternatives that are presented in detail in Table 2 will be analyzed. Table 2 shows the full decision matrix, which is contained in columns from C 1 to C 8 . There is also presented information about the value of preferences and the position in the ranking of each of the examined bicycles. For example, the most attractive bicycle turned out to be Blix Sol, i.e., alternative A 5 , which received a rating of 0.6870. California Bicycle S, which received a rating of 0.3669 and the last place in the ranking, was indicated as the least attractive bicycle. It is worth noting the differences between some alternatives are relatively low, where for example, the second position in the ranking has a rating worse by only 0.0021.
It is worth recalling that the obtained results come from a method that does not require a model of knowledge of the significance of decision-making criteria to be determined. The expert identifies the whole model by comparing pairs of characteristic objects. In this way, an indirect identification takes place, and the models obtained in this way are well suited to complex non-linear problems.
Thanks to this, and based on Table 2, we will propose two approaches to identify the relevance of criteria in Section 3.2. Table 2. Alternatives description and preference.

A i
Name

Methods
To determine the relevance of the criteria, a reference ranking was used. It was created using the model identified by the COMET method. In the next step, we set eight rankings which are created by successive exclusions of individual component criteria from the primary model. Each such ranking has exactly seven active criteria and one excluded criterion. Table 3 presents the main ranking without exclusions and eight derivative rankings with their ten initial alternatives. The most important information contained in this table is the value of the similarity coefficient between the main ranking and individual derivative rankings. For this purpose, two measures of r w and WS were used. The highest value of r w was recorded for the ranking that excluded the C 7 criterion from the main model (value 0.8429). The smallest value was obtained for the ranking that excluded criterion C 6 (value 0.7176). This means that the C 6 criterion had a more significant impact on the ranking change than the C 7 criterion, so it was more significant than the r w criterion. The lowest value is obtained for criterion C 2 , i.e., 0.8377, so based on the WS coefficient, the most important is criterion C 2 and the least important is criterion C 3 . It should be reminded here that both coefficients have a different paradigm of determining the similarity of the two rankings. Therefore, both these coefficients will be used for further investigation.  The r w coefficient indicated that the most important criterion is the range and the least important criterion is the weight of the bike. It is common sense that this is in line with the expectations of many consumers. The most important criterion is the charging time expressed in hours, and the least important is the number of gears. It should be noted that in both cases, the battery capacity is a less important parameter in this case than the range of the bicycle. It can be explained by the fact that a battery with a specific capacity does not always guarantee the same distance and this attribute can be less important than range. Figure 4 presents the similarity of the reference and derivative rankings.  Due to the nature of both coefficients, we propose the following formula (52) to calculate the significance weights of the individual criteria: where N means the number of all criteria, w i is a i-th weights of criterion C i , and x i is i-th coefficient (WS or r w ). This gives us two different approaches to the importance of weights, which will be compared with entropy (49) and standard deviation (51) methods. Table 4 provides detailed results of the relevance weights obtained using two new and two traditional methods for determining the relevance of criteria. To better compare the obtained criteria weights, they are shown in Figure 5. The significance weights determined by means of the r w coefficient are characterized by quite equal distribution of significance. The most significant criterion is the range with weight 0.1687 and the least significant is the weight with weight 0.0939. Greater differences between the individual weights occur with the approach using the WS coefficient. In this case charging time (0.2914) is the most important and the number of gears is the least important (0.0262). Using the entropy of information approach, we get as the most important criterion the number of gears (0.3854) and the least important criterion is the maximum speed (0.0109). The most asymmetrical in its assessment is the method based on standard deviation where the price has a significance equal to 0.9185. According to this approach, the least important parameter is the charging time. The results obtained using information entropy and standard deviation are less effective than our proposed approaches, which will be proven in the next section.

TOPSIS
To check the effectiveness of the four criteria weighting methods, the first TOPSIS method will be used, where 55 electric bicycles will be evaluated using different normalization methods with each of the four methods to identify the significance of the weights. Figures 6-9 show a comparison of reference ranking and rankings determined by TOPSIS method, selected normalizations and ways of determining criterion significance. In case of perfect compatibility, the black dots should completely cover the blue line. Figure 6 shows visualizations of the results using minimax normalization. Visually, the best representation was achieved using the scales determined by the r w approach. The ranking circle was also well reflected by using the WS approach. At the same time, the entropy and standard deviation method reflected best the tail of the ranking. This is confirmed by the numerical values presented in Table 5. the method based on the r w coefficient undoubtedly did best in this aspect.  Then, Figure 7 shows a comparison of results obtained using normalization max type. The r w method again proved to be the most similar to the reference ranking results obtained with the COMET method. This is confirmed by the numbers presented in Table 5. In this case, both coefficients also inform us about the advantage of the WS method over the entropy and standard deviation methods. Another normalization based on the sum does not bring such unambiguous results as in the two previous cases. In all four methods presented in Figure 8, the dispersion is higher than in the previous cases. However, based on the data from Table 5, it can be concluded that the r w significance test method received relatively good results. It should be reminded here once again that the ideal situation is when all the dots are exactly on the blue line. This means that the two rankings are identical, which was well visible in Figure 7 when using the r w approach weights was the closest to this situation. At that time, the rankings' similarity coefficients were 0.9283 and 0.9897, respectively.  The last examined case is vector normalization, presented in Figure 9, which again gives the best results in the r w ratio approach, where similarity measures are 0.7311 and 0.9675, respectively. Again, the least effective is the standard deviation approach. When comparing rankings using r w ratio, entropy approach is more effective than WS approach. However, using WS coefficient, the situation is the opposite. This is due to better matching of the ranking front in the WS method than in the case of entropy. It is characteristic for the WS ratio because it is the top of the ranking that is most important there, and the ratio itself is an asymmetric measure. The complete list of results using the similarity of rankings is presented in Table 5. The normalization methods that best managed to reconstruct the initial ranking were minmax and max, and the worst managed to be the sum-based method.
To summarize the effectiveness of the proposed methods and standard approaches, we present Figure 10. This is a visualization of the results obtained with a r w ratio. Its value can range from −1 to 1. As we can see, for all types of standardization, the method based on the standard deviation is the least effective. The r w ratio method proposed in our article works most effectively as an alternative approach. The entropy method and our second proposition are poorly matched when using the r w similarity measure. However, the effectiveness advantage of the r w approach is undeniable. When using the second-ranking matching indicator WS, one should pay attention to its different nature. In this case, the values may vary from zero to one. Figure 11 shows a comparison of the proposed approaches. Again, the most effective approach seems to be the r w weighting approach. In contrast to the previous list, the WS weighting method came second. Interestingly, the method based on standard deviation was also used in three out of four cases and obtained better results than those obtained using entropy. In the analyzed case study, both comparative approaches indicate the highest efficiency using the r w ratio approach. In the next section, we will examine the efficiency of the set significance criteria using the example of the VIKOR method.

VIKOR
The next stage of checking the effectiveness of the proposed approaches will be based on the use of the VIKOR method without normalization and with four types of it. Although the method does not require the use of normalization, quite often one can meet the opinion that it is convicted to obtain better results. For readability of the paper, the drawings containing comparisons of the results from the reference ranking and the ranking obtained by using VIKOR are shown in the Appendix A. In Figure A1, all methods of determining the significance of the weights indicate high dispersion. This is the first case when entropy proves to be more effective and moreover with both parameters of ranking similarity measurement. In this case, the weakest match was obtained with the WS coefficient. However, it should be emphasized that all obtained results indicate low effectiveness of the methods used.
Much lower dispersion is achieved when standardizing the decision matrix with the minmax method. The VIKOR method then has a much better matching of results than without normalization. This is visible in Figure A2, and the detailed measures of the matching are presented in Table 6. The WS-based method is slightly more effective when testing the matching by WS coefficient than the r w -based approach. The weakest results were obtained using a method based on standard deviation. Identical results are obtained using max and vector normalizations. Figures A3 and A5 return precisely the same results despite using a different normalization method. Thus, both cases are entirely equivalent.
The last tested normalization is a sum, where the results are presented in Figure A4. In this case, we obtain results similar to those of minmax, max and vector normalizations. The difference is that the effectiveness of the approach based on the WS is deteriorated when using both meters. Detailed results of the analysis are presented in Table 6. Summary of the analyses based on the r w coefficient is presented in Figure 12. Using any standardization, the determination of the significance of criteria with r w proves to be the most effective. The weakest approach was the one based on the standard deviation when we use any transformation of the decision matrix. Using VIKOR without normalization, the best result was obtained using the method based on entropy. However, it should be remembered that all the obtained results were much weaker. As many as three cases returned the same value. However, the highest effectiveness again falls to the approach based on r w coefficient.
When analyzing the data in Figure 13 we get very similar results. Again, without the use of techniques transforming the decision matrix, entropy turns out to be the best in the other two cases, the two most effective methods are those based on ranking similarity coefficients. The weakest approach is again, a standard deviation.

Conclusions
In this work, we present an entirely new approach to determining the relevance of criteria in sustainable transport issues. Based on a comparative study, we analyze the similarity of the approach to determining materiality using r w and WS as well as standard methods such as entropy and standard deviation. The obtained materiality levels differ significantly. When analyzing which criteria were indicated as the most relevant in the four approaches, it should be pointed out that the most logical ones seem to be those related to the r w and WS. However, in order to empirically verify, an additional study was carried out to check the effectiveness of the calculated criteria weights when using two popular MCDA methods, i.e., TOPSIS and VIKOR. We also examined what the final rankings look like when using different contact normalizations in two cases. We received precise results which indicate that the most effective approach is based on the r w ratio. At the same time, it is essential to note that the WS ratio approach also has its advantages. Only in one of the nine cases did the entropy method turn out to be the best way to calculate the weighting values. However, it was a case where all the examined methods had poor results. Therefore, based on the conducted research, it can be indicated that the effectiveness of the proposed approaches is higher than the previously used approaches related to entropy and standard deviation.
The limitations of our research are that it concerns one research field, i.e., sustainable transport. Additionally, it should be noted that this is preliminary research which should be extended with extended simulation. Further improvement of approaches related to ranking similarity coefficients should be indicated as the main directions for future work. Further empirical tests related to the use of other methods of determining criterion weighting and other cases of use should be conducted.

Acknowledgments:
The authors would like to thank the editor and the anonymous reviewers, whose insightful comments and constructive suggestions helped us to significantly improve the quality of this paper.

Conflicts of Interest:
The authors declare no conflict of interest.

Abbreviations
The following abbreviations are used in this manuscript:

MCDA
Multi     Figure A5. Comparison of the results from the reference ranking and the ranking obtained by using VIKOR-vector.