Exploring Symmetry of Binary Classification Performance Metrics

Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications.


Introduction
Symmetry has played and continues playing, a highly significant role in the way of how humans perceive the world [1].In the scientific fields, symmetry plays a key role as it can be discovered in nature [2,3], society [4] and mathematics [5].Moreover, symmetry also provides an intuitive way to attain faster and deeper insights into scientific problems.
In recent years, an increasing interest has arisen in detecting and taking advantage of symmetry in various aspects of theoretical and applied computing [6].Several studies involving symmetry have been published in network technology [7], human interfaces [8], image processing [9], data hiding [10] and many other applications [11].
On the other hand, pattern recognition and machine learning procedures are becoming key aspects of modern science [12] and the hottest topics in the scientific literature on computing [13].Furthermore, in this field, symmetry is playing an interesting role either as a subject of study, in the form of machine learning algorithms to discover symmetries [14] or as a means to improve the results obtained by automatic recognition systems [15].Let us emphasize this point: not only can knowing the symmetry of a certain computer algorithm be intrinsically rewarding since it sheds light on the behaviour of the algorithm but it can also be very useful for its interpretation, its optimization or as a criterion for the selection among various competing algorithms.As an example, in recent research, we have employed a symmetric criterion to select the best feature-extraction procedures (Discrete Cosine Transform versus Discrete Fourier Transform) [16] in an application of the classification of sounds [17,18] effectively deployed in a Wireless Sensor Network as shown in Figure 1.Another examples of industrial applications using classification of sounds can be found in Refs.[19,20].
Symmetry 2018, 10, x FOR PEER REVIEW 2 of 31 in recent research, we have employed a symmetric criterion to select the best feature-extraction procedures (Discrete Cosine Transform versus Discrete Fourier Transform) [16] in an application of the classification of sounds [17,18] effectively deployed in a Wireless Sensor Network as shown in Figure 1.Another examples of industrial applications using classification of sounds can be found in Refs.[19,20].In the broad field of machine learning, the study of how to measure the performance of various classifiers has attracted continued attention [21][22][23].Classification performance metrics play a key role in the assessment of the overall classification process in the test phase, in the selection from among various competing classifiers in the validation phase and are even sometimes used as the loss function to be optimized in the process of model construction during the classification training phase.
However, to the best of our knowledge, no systematic study into the symmetry of these metrics has yet been undertaken.By discovering their symmetries, we would reach a better understanding of their meaning, we could obtain useful insights into when their use would be more appropriate and we would also gain additional and meaningful indicators for the selection of the best performance metric.
Although several dozen performance metrics can be found in the literature, we will focus on those which are probably the most commonly used: the metrics based on the confusion matrix [24].Accuracy, precision and recall (sensitivity) are undoubtedly some of the most popular metrics.On the other hand, our research will be focused on the cases where there are only two classes (binary classifiers).Although this is certainly a limitation, it does provide a solid ground base for further research.Moreover, multiclass performance metrics are usually obtained by decomposing the multiclass problem into several binary classification sub-problems [25].

Definitions
Let us first consider an original (baseline) experiment  , defined by the duple  =  ,  ) composed of a set of  classifiers,  =  and a set of their corresponding  datasets,  =  ,  = 1, ⋯ ,  .The elements in every dataset belong to either of two classes,  and  , which are called Positive () and Negative () classes, respectively.The -th classifier  operates on the corresponding  dataset, thereby obtaining a resulting classification which can be defined by its binary confusion matrix  and hence   .The set of confusion matrices are denominated In the broad field of machine learning, the study of how to measure the performance of various classifiers has attracted continued attention [21][22][23].Classification performance metrics play a key role in the assessment of the overall classification process in the test phase, in the selection from among various competing classifiers in the validation phase and are even sometimes used as the loss function to be optimized in the process of model construction during the classification training phase.
However, to the best of our knowledge, no systematic study into the symmetry of these metrics has yet been undertaken.By discovering their symmetries, we would reach a better understanding of their meaning, we could obtain useful insights into when their use would be more appropriate and we would also gain additional and meaningful indicators for the selection of the best performance metric.
Although several dozen performance metrics can be found in the literature, we will focus on those which are probably the most commonly used: the metrics based on the confusion matrix [24].Accuracy, precision and recall (sensitivity) are undoubtedly some of the most popular metrics.On the other hand, our research will be focused on the cases where there are only two classes (binary classifiers).Although this is certainly a limitation, it does provide a solid ground base for further research.Moreover, multiclass performance metrics are usually obtained by decomposing the multiclass problem into several binary classification sub-problems [25].

Definitions
Let us first consider an original (baseline) experiment CM B = cm B i .The baseline experiment can therefore be defined as the set of classifiers operating on the set of datasets to obtain a set of confusion matrices, E B : D B C B → CM B .This paper will explore the behaviour of binary classification performance metrics when the original experiment is subject to n E different types of transformations.Let us define the k-th transformed experiment E k = C k , D k composed of a set of n k classifiers, C k = c k i and a set of their corresponding n k datasets, D k = D k i , whose result is a set of confusion matrices CM k = cm k i .
Hence, E k : D k C k → CM k , where k = B, 1, 2, • • • , n E , indicates the type of transformation.In the k-th experiment, when the i-th classifier c k i operates on its corresponding D k i dataset, the result is summarized in the binary confusion matrix defined as where • a k i is the number of positive elements in D k i correctly classified as positive; • b k i is the number of negative elements in D k i correctly classified as negative; • f k i is the number of positive elements in D k i incorrectly classified as negative; and • g k i is the number of negative elements in D k i incorrectly classified as positive.
Let us call P k i , N k i and M k i the positive, negative and total number of elements in D k i .Therefore The confusion matrix can then be described as Let us now define α k i as the ratio of positive elements in D k i correctly classified as positive; and β k i as the ratio of negative elements in D k i correctly classified as negative.That is, The confusion matrix can therefore be rewritten as On the other hand, a dataset D k i is called imbalanced if it has a different number of positive and negative elements, that is, P k i = N k i .Classification on the presence of imbalanced datasets is a challenging task requiring specific considerations [26].To quantify the imbalance, several indicators have been proposed, such as the dominance [27,28], the proportion between positive and negative instances (formalized as 1 : X) [29] and the imbalance ratio (IR) defined as P k i /N k i [30], which is also called skew [31].This value lies within the [0, ∞] range and has a value IR = 1 in the balanced case.We prefer to use an indicator showing a value 0 in the balanced case, a value +1 when all the elements in the dataset are positive and −1 if all the elements are negative.We define the imbalance coefficient δ k i , which is an indicator that has these characteristics, as The imbalance coefficient is graphically shown in Figure 2 (solid blue cline) as a function of the proportion of positive elements in the dataset.For the sake of comparison, that figure also shows the IR imbalance ratio (dashed green line).Based on the imbalance coefficient, the number of positive and negative elements in the dataset can be rewritten as By substituting these expressions into Equation (4), the confusion matrix becomes where  is the unitary confusion matrix defined as It can be seen that  is a function of 3 variables: the ratio of positive  and negative  correctly classified elements and the imbalance coefficient  , that is,  =   ,  ,  .
In order to measure the performance of the classification process,  metrics are used.In this paper we focus on metrics that are based on the unitary confusion matrix and, for the sake of much easier comparison, all these metrics are converted within the range [−1,1].Let us define  as the -th of such metrics for the  classifier operating on the  dataset, where  = 1, … , .Since it is based on the unitary confusion matrix,  =   =   ,  ,  .
Let us now define  as the set of the -th metric values corresponding to the -th experiment  =  ,  ) , that is,  ≡  ,  = 1,2, ⋯ ,  .Additionally, the sets  ≡  ,  ≡    ≡  are also defined.

Representation of Metrics.
With these definitions, it is clear that the metric  =   ,  ,  ) and hence it is a 4-dimensional function since  (one dimension) depends on  ,    (three independent dimensions).To depict their values, a first approach could involve a 3D representation space where each  ,  ,  point is color-coded according to the value   ,  ,  .
To show the different types of representations, let us define an arbitrary metric function Based on the imbalance coefficient, the number of positive and negative elements in the dataset can be rewritten as By substituting these expressions into Equation (4), the confusion matrix becomes where λ k i is the unitary confusion matrix defined as It can be seen that λ k i is a function of 3 variables: the ratio of positive α k i and negative β k i correctly classified elements and the imbalance coefficient In order to measure the performance of the classification process, m metrics are used.In this paper we focus on metrics that are based on the unitary confusion matrix and, for the sake of much easier comparison, all these metrics are converted within the range [−1, 1].Let us define j γ k i as the j-th of such metrics for the c k i classifier operating on the D k i dataset, where j = 1, . . ., m.Since it is based on the unitary confusion matrix, Let us now define µ k j as the set of the j-th metric values corresponding to the k-th experiment are also defined.

Representation of Metrics
With these definitions, it is clear that the metric µ k j = µ k j α k , β k , δ k and hence it is a 4-dimensional function since µ k j (one dimension) depends on α k , β k and δ k (three independent dimensions).To depict their values, a first approach could involve a 3D representation space where each α k i , To show the different types of representations, let us define an arbitrary metric function This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results.Figure 3 depicts the 3D representation for said example function.
The n E = 1000 pairs of classifiers and datasets used in the experiment E k = C k , D k are selected in such a way that the space α k , β k , δ k is covered with equally spaced points.The above figure may cause confusion, mainly when the number of points (n E ) increases.An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient.Figure 4a depicts such a slice in the 3D graphic for an arbitrary value δ = 0.75 and Figure 4b shows the slice on a 2D plane.
This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results.Figure 3 depicts the 3D representation for said example function.The  = 1000 pairs of classifiers and datasets used in the experiment  =  ,  ) are selected in such a way that the space  ,  ,  ) is covered with equally spaced points.The above figure may cause confusion, mainly when the number of points ( ) increases.An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient.Figure 4a depicts such a slice in the 3D graphic for an arbitrary value  = 0.75 and Figure 4b shows the slice on a 2D plane.In the previous figure, the slice contains 100 values of the metric.However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended.For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets  ,  ) in such a way that the plane  ,  ) is fully covered.The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient ( = 0.75 in the example).
This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results.Figure 3 depicts the 3D representation for said example function.The  = 1000 pairs of classifiers and datasets used in the experiment  =  ,  ) are selected in such a way that the space  ,  ,  ) is covered with equally spaced points.The above figure may cause confusion, mainly when the number of points ( ) increases.An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient.Figure 4a depicts such a slice in the 3D graphic for an arbitrary value  = 0.75 and Figure 4b shows the slice on a 2D plane.In the previous figure, the slice contains 100 values of the metric.However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended.For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets  ,  ) in such a way that the plane  ,  ) is fully covered.The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient ( = 0.75 in the example).In the previous figure, the slice contains 100 values of the metric.However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended.For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets C k , D k in such a way that the plane α k , β k is fully covered.The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient (δ = 0.75 in the example).In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6.

Transformations
The original baseline experiment  =  ,  ) is subject to various types of transformations.As a result of the -th transformation, the metrics related to the baseline experiment   ,  ,  ) are transformed into   ,  ,  ) , which can be written either as   ,  ,  ) =    ,  ,  ) or as   ,  ,  )   ,  ,  ).(11) It is said that the metric  is symmetric under the transformation  if  =  .Conversely,  is called antisymmetric under  (or symmetric under the complementary transformation  ) if  = − .
Analogously, it is said that the metrics  and  are cross-symmetric under the transformation  if  =  .Conversely,  and  are called anti-cross-symmetric under  (or cross-symmetric under the complementary transformation  ) if  = − .In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6.In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6.

Transformations
The original baseline experiment  =  ,  ) is subject to various types of transformations.As a result of the -th transformation, the metrics related to the baseline experiment   ,  ,  ) are transformed into   ,  ,  ) , which can be written either as   ,  ,  ) =    ,  ,  ) or as   ,  ,  )   ,  ,  ).(11) It is said that the metric  is symmetric under the transformation  if  =  .Conversely,  is called antisymmetric under  (or symmetric under the complementary transformation  ) if  = − .
Analogously, it is said that the metrics  and  are cross-symmetric under the transformation  if  =  .Conversely,  and  are called anti-cross-symmetric under  (or cross-symmetric under the complementary transformation  ) if  = − .

Transformations
The original baseline experiment E B = C B , D B is subject to various types of transformations.As a result of the k-th transformation, the metrics related to the baseline experiment µ B j α B , β B , δ B are transformed into µ k j α k , β k , δ k , which can be written either as It is said that the metric µ j is symmetric under the transformation T k if µ k j = µ B j .Conversely, µ j is called antisymmetric under T k (or symmetric under the complementary transformation T k ) if µ k j = −µ B j .Analogously, it is said that the metrics µ u and µ v are cross-symmetric under the transformation Conversely, µ u and µ v are called anti-cross-symmetric under T k (or cross-symmetric under the complementary transformation T

One-Dimensional Transformations
One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric.Type α transformation implies that the i-th transformed classifier c α i shows a ratio of correctly classified positive elements α α i which has the symmetric value of the ratio α B i obtained by the baseline classifier c B i .Since the values of such ratios lie within the range [0, 1], the symmetry exists with respect to the hyperplane α = 0.5 and can be stated as α α i = 1 − α B i .An example of this transformation is depicted in Figure 7.
One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric.Type  transformation implies that the -th transformed classifier  ) shows a ratio of correctly classified positive elements  ) which has the symmetric value of the ratio  ) obtained by the baseline classifier  ).Since the values of such ratios lie within the range [0,1], the symmetry exists with respect to the hyperplane  = 0.5 and can be stated as  = 1 −  .An example of this transformation is depicted in Figure 7. Analogously, type  transformation implies that the -th transformed classifier  shows a ratio of correctly classified negative elements  , which has the symmetric value of the ratio  ) obtained by the baseline classifier  ).Since the value of such ratios also lie within the range [0,1], the symmetry exists with respect to the hyperplane  = 0.5 and can be stated as  = 1 −  .An example of this transformation is depicted in Figure 8. Conversely, type  transformation, which, instead of operating on classifiers, operates on datasets, implies that the -th transformed dataset  has an imbalance ratio  , which has the symmetric value of the imbalanced ratio  ) in the baseline corresponding to dataset  ).Since the value of such imbalance ratios lie within the range [−1,1], the symmetry exists with respect to the hyperplane  = 0 and can be stated as  = − .An example of this transformation is depicted in Figure 9. i .Since the value of such ratios also lie within the range [0, 1], the symmetry exists with respect to the hyperplane β = 0.5 and can be stated as i .An example of this transformation is depicted in Figure 8. One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric.Type  transformation implies that the -th transformed classifier  ) shows a ratio of correctly classified positive elements  ) which has the symmetric value of the ratio  ) obtained by the baseline classifier  ).Since the values of such ratios lie within the range [0,1], the symmetry exists with respect to the hyperplane  = 0.5 and can be stated as  = 1 −  .An example of this transformation is depicted in Figure 7. Analogously, type  transformation implies that the -th transformed classifier  shows a ratio of correctly classified negative elements  , which has the symmetric value of the ratio  ) obtained by the baseline classifier  ).Since the value of such ratios also lie within the range [0,1], the symmetry exists with respect to the hyperplane  = 0.5 and can be stated as  = 1 −  .An example of this transformation is depicted in Figure 8. Conversely, type  transformation, which, instead of operating on classifiers, operates on datasets, implies that the -th transformed dataset  has an imbalance ratio  , which has the symmetric value of the imbalanced ratio  ) in the baseline corresponding to dataset  ).Since the value of such imbalance ratios lie within the range [−1,1], the symmetry exists with respect to the hyperplane  = 0 and can be stated as  = − .An example of this transformation is depicted in Figure 9. Conversely, type δ transformation, which, instead of operating on classifiers, operates on datasets, implies that the i-th transformed dataset D δ i has an imbalance ratio δ δ i , which has the symmetric value of the imbalanced ratio δ B i in the baseline corresponding to dataset δ B i .Since the value of such imbalance ratios lie within the range [−1, 1], the symmetry exists with respect to the hyperplane δ = 0 and can be stated as δ δ i = −δ B i .An example of this transformation is depicted in Figure 9.

Multidimensional Transformations.
Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric.Firstly, let us define type  transformation as that which exchanges  and  dimensions.This implies that the  -th transformed classifier/dataset pair  ,  ) shows a ratio of correctly classified positive elements  ) which has the same value as the ratio of correctly classified negative elements  ) obtained by the baseline classifier/dataset pair  ,  ).This exchange can be seen as the symmetry with respect to the hyperplane  =  (main diagonal of the ,  plane) and can be stated as  =  ;  =  .An example of this transformation is depicted in Figure 11.Finally, type µ transformation jointly operates on classifiers and datasets in such a way that the j-th of performance metrics j γ µ i for the c µ i classifier operating on the D µ i dataset has the symmetric value of the performance metric in the baseline experiment ( j γ B i ).Since the value of such metrics lie within the range [−1, 1], the symmetry exists with respect to the hyperplane µ = 0 and can be stated as j γ µ i = − j γ B i .An example of this transformation is depicted in Figure 10 where it should be noted that the µ dimension is shown by the colour code of each point.Therefore, an inversion in µ is shown as a colour inversion.

Multidimensional Transformations.
Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric.Firstly, let us define type  transformation as that which exchanges  and  dimensions.This implies that the  -th transformed classifier/dataset pair  ,  ) shows a ratio of correctly classified positive elements  ) which has the same value as the ratio of correctly classified negative elements  ) obtained by the baseline classifier/dataset pair  ,  ).This exchange can be seen as the symmetry with respect to the hyperplane  =  (main diagonal of the ,  plane) and can be stated as  =  ;  =  .An example of this transformation is depicted in Figure 11.

Multidimensional Transformations
Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric.Firstly, let us define type σ transformation as that which exchanges α and β dimensions.This implies that the i-th transformed classifier/dataset pair c σ i , D σ i shows a ratio of correctly classified positive elements α σ i which has the same value as the ratio of correctly classified negative elements β B i obtained by the baseline classifier/dataset pair c B i , D B i .This exchange can be seen as the symmetry with respect to the hyperplane α = β (main diagonal of the α, β plane) and can be stated as An example of this transformation is depicted in Figure 11.

Multidimensional Transformations.
Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric.Firstly, let us define type  transformation as that which exchanges  and  dimensions.This implies that the  -th transformed classifier/dataset pair  ,  ) shows a ratio of correctly classified positive elements  ) which has the same value as the ratio of correctly classified negative elements  ) obtained by the baseline classifier/dataset pair  ,  ).This exchange can be seen as the symmetry with respect to the hyperplane  =  (main diagonal of the ,  plane) and can be stated as  =  ;  =  .An example of this transformation is depicted in Figure 11.Although the four axes in these plots remain dimensionless, not all of them have the same meaning.So, α and β are both ratios of correctly classified elements.It would be nonsensical, for instance, to rescale α without also rescaling β.However, δ has a completely different meaning Symmetry 2019, 11, 47 9 of 31 and its scale can and in fact does, differ from α and β.The same reasons can be applied to the axes µ.Therefore, all the exchanges of multidimensional axes are meaningless, except the interchange of α and β.All the other remaining exchanges are dismissed in our study.
The one-and two-dimensional transformations described above are called basic transformations and are summarized in Table 1.
Table 1.Summary of basic transformations.
More complex transformations can be obtained by concatenating basic transformations.For instance, applying basic transformation α (T α ) and then basic transformation As each of the one-dimensional transformations operates on an independent axis, they have the commutative and associative properties, that is, given 3 one-dimensional transformations, T U , T V and T W , it is true that However, bi-dimensional type σ transformation T σ operates on the same axis as T α and T β .In this case, the order of transformation matters, as they do not have the commutative property.For instance, Having 5 basic transformations and not initially considering their order, any combined transformation can be binary coded in terms of the presence/absence of each basic component.Therefore 2 5 = 32 combinations are possible; only 31 if the identity transformation (coded 00000) is dismissed.In order to code a combined transformation, the order µ, σ, δ, β, α is used where transformation µ indicates the Most Significant Bit (MSB) and the transformation α specifies the Least Significant Bit (LSB).An example of this code is shown in Table 2.With this selection, codes greater than 15 contain a transformation type µ, that is, they are useful in exploring antisymmetric behaviour.In the cases where the order of transformations matters, σ = 1 and (α = 1 or β = 1), then their corresponding codes refer to various different combined transformations.A first example of combined transformations is that of the inverse labelling of classes.As stated above, the elements in every dataset belong to either of two classes, G 1 and G 2 , which are called Positive (P) and Negative (N) classes, respectively.The inverse labelling transformation (T L ) explores the classification metric behaviour when the labelling of the classes is inverted, that is, when G 2 is called the Positive class and G 1 the Negative class.Let us consider the i-th classifier c L i operating on its corresponding D L i dataset.In the baseline experiment, the ratio of correctly classified positive elements (α B i ) refers to class G 1 and conversely (β B i ) refers to class G 2 .In the T L transformed experiment, the ratio of correctly classified positive elements (α L i ) refers to class G 2 and conversely (β L i ) refers to That is, the first step of this transformation implies interchanging the axes α and β, which is equivalent to reflection symmetry with respect to the main diagonal, formerly defined as the basic transformation of type σ (Figure 12b).
Additionally, in the baseline experiment, the number of positive elements (P B i ) refers to class G 1 , while in the T L transformed experiment, the number of positive elements (P L i ) refers to class G 2 , which means that P L i = N B i and N L i = P B i , while the total number of elements remains unaltered: M L i = M B i .Therefore, by recalling Equation ( 5), Hence, the second step of this transformation also implies reflection symmetry with respect to the hyperplane δ = 0, previously defined as the basic transformation of type δ (Figure 12c).Finally, the complementary transformation T L involves a third and final step of inverting the sign of the metric, which is equivalent to reflection symmetry with respect to the hyperplane µ = 0, formerly defined as the basic transformation of type µ (Figure 12d).( ) refers to class  , which means that  =  and  =  .That is, the first step of this transformation implies interchanging the axes  and , which is equivalent to reflection symmetry with respect to the main diagonal, formerly defined as the basic transformation of type  (Figure 12b).Additionally, in the baseline experiment, the number of positive elements ( ) refers to class  , while in the  transformed experiment, the number of positive elements ( ) refers to class  , which means that  =  and  =  , while the total number of elements remains unaltered:  =  .Therefore, by recalling Equation (5), Hence, the second step of this transformation also implies reflection symmetry with respect to the hyperplane  = 0, previously defined as the basic transformation of type  (Figure 12c).Finally, the complementary transformation  involves a third and final step of inverting the sign of the metric, which is equivalent to reflection symmetry with respect to the hyperplane  = 0, formerly defined as the basic transformation of type  (Figure 12d).
A second example of combined transformations is given by the inverse-scoring transformation ( ) which explores classification metric behaviour when the scoring of the classification results are inverted.In the baseline experiment, let us consider the  -th classifier  operating on its corresponding  dataset, thereby obtaining a ratio  of correctly classified positive elements and a ratio  in the negative case.The -th metric assigns a score of   ,  ,  ) to this result .High values of the score  usually correspond to high ratios  ,  .In the inverted score transformation ( ), the -th classifier  operating on its corresponding  dataset obtains a ratio  of correctly classified positive elements which is equal to the ratio of positive elements incorrectly classified in the baseline experiment, that is,  = 1 −  , which implies a type  transformation.Analogously, for the negative class,  = 1 −  , which implies a type  transformation.If  ,  have high values, then  ,  will have low values and, to be consistent, the result should be marked with a low score.For that reason, the inverse scoring transform also implies a transformation type , that is, it uses the symmetric value of the metric  = −  .Therefore, the inverse labelling transformation can be defined as  =  =  •  •  where The results are depicted in Figure 13.Therefore, the inverse labelling transformation can be defined as T L = T σδ = T σ •T δ and its complementary as where A second example of combined transformations is given by the inverse-scoring transformation (T S ) which explores classification metric behaviour when the scoring of the classification results are inverted.In the baseline experiment, let us consider the i-th classifier c B i operating on its corresponding D B i dataset, thereby obtaining a ratio α B i of correctly classified positive elements and a ratio β B i in the negative case.The j-th metric assigns a score of j γ B i α B i , β B i , δ B i to this result .High values of the score j γ B i usually correspond to high ratios α B i , β B i .In the inverted score transformation (T S ), the i-th classifier c S i operating on its corresponding D S i dataset obtains a ratio α S i of correctly classified positive elements which is equal to the ratio of positive elements incorrectly classified in the baseline experiment, that is, α S i = 1 − α B i , which implies a type α transformation.Analogously, for the negative class, will have low values and, to be consistent, the result should be marked with a low score.For that reason, the inverse scoring transform also implies a transformation type µ, that is, it uses the symmetric value of the metric j γ S i = − j γ B i .Therefore, the inverse labelling transformation can be defined as The results are depicted in Figure 13.A third example is that of the full inversion ( ), which explores the classification metric behaviour when both the labelling ( ) and the scores ( ) are inverted.This transformation can be featured by the concatenation of their two components, which can be written as The results are depicted in Figure 14.
that is,  =  .Analogously, it can be shown that  =  .

Performance Metrics
Based on the binary confusion matrix, numerous performance metrics have been proposed [32][33][34][35][36].For our study, the focus is placed on 10 of these metrics, which are summarized in Table 3.The terms used in that table are taken from the elements of a generic confusion matrix which can be stated as  A third example is that of the full inversion (T F ), which explores the classification metric behaviour when both the labelling (T L ) and the scores (T S ) are inverted.This transformation can be featured by the concatenation of their two components, which can be written as The results are depicted in Figure 14.A third example is that of the full inversion ( ), which explores the classification metric behaviour when both the labelling ( ) and the scores ( ) are inverted.This transformation can be featured by the concatenation of their two components, which can be written as The results are depicted in Figure 14.
that is,  =  .Analogously, it can be shown that  =  .

Performance Metrics
Based on the binary confusion matrix, numerous performance metrics have been proposed [32][33][34][35][36].For our study, the focus is placed on 10 of these metrics, which are summarized in Table 3.The terms used in that table are taken from the elements of a generic confusion matrix which can be stated as  Finally let us consider the T ασβ transformation that is, T ασβ = T σ .Analogously, it can be shown that T βσα = T σ .

Performance Metrics
Based on the binary confusion matrix, numerous performance metrics have been proposed [32][33][34][35][36].For our study, the focus is placed on 10 of these metrics, which are summarized in Table 3.The terms used in that table are taken from the elements of a generic confusion matrix which can be stated as The last three metrics (MCC, BM and MK) take values within the [−1, 1] range, while the ranges for the first seven lie within the [0, 1] interval.For comparison purposes, these metrics are used herein in their normalized version ([−1, 1] interval).By naming a metric defined within the [0, 1] interval as µ, it can be normalized within the [−1, 1] range by the expression It can easily be shown that all these metrics can be expressed as a function µ = µ(α, β, δ).
Although only performance metrics based on the confusion matrix are considered, a marginal approach to Receiver Operating Characteristics (ROC) analysis [37] can also be carried out.In this analysis, the Area Under Curve (AUC) is commonly used as a performance metric.However, for classifiers offering only a label (and not a set of scores for each label) or when a single threshold is used on scores, the value of AUCn and BM are the same [38].Therefore, in the forthcoming sections, whenever BM is mentioned it could also be understood as AUCn.

Exploring Symmetries
In order to determine the existence of any symmetric or cross-symmetric behaviour on the 10 classification performance metrics described in the previous section, we should explore whether, for each metric (or pair of metrics), its baseline and any of the 31 combinations of transformations obtain the same result as that of the baseline of the same metric (symmetry) or any other metric (cross-symmetry).Moreover, many of these combined transformations must take the order into account.Therefore, several thousands of different analyses have to be undertaken.Although performing this task using analytical derivations is not an impossible assignment (preferably using some kind of symbolic computation), it is certainly arduous.
An alternative approach is to identify the distance of two metrics.More formally, for the U-th transformation, let us consider the i-th combination of classifier c U i operating on the D U i dataset.The classification result is measured using the r-th metric, r γ U i .Similarly, for the V-th transform and the i-th combination of classifier c V i operating on the D V i dataset, let us measure its performance using the s-th metric, s γ V i .The distance between these measures is defined as and the s-th metric µ V s = s γ V i can then be defined as Therefore, symmetric or cross-symmetric behaviour can be identified by a distance equal to zero.It should be noted that if the r-th metric is symmetric under the U-th transformation, that is, r , it will also be symmetric under the concatenation of the two transformations.In effect, Conversely, this is not true for cross-symmetries.If the r-th and s-th metric are cross-symmetric under the U-th transformation, that is, µ U r = T U µ B r = µ B s and also under the V-th transformations, µ V r = T V µ B r = µ B s , they are not necessarily cross-symmetric under the concatenation of the two transformations.In effect,

Statistical Symmetries
The symmetries of the performance metrics can also be explored from a statistical point of view.Let us recall that D k i is the i-th dataset in the k-th experiment with an imbalance described by its imbalance coefficient δ k i .The elements in D k i are processed by the c k i classifier in order to obtain a ratio of correctly classified positive α k i and negative β k i elements.The j-th metric j γ k i is based on these values and hence Let us also recall that the set of all these values for i Let us now suppose that the elements c k i , D k i in the experiments are randomly selected in such a way that α k , β k and δ k are uniformly distributed within their respective ranges.Therefore, µ k j becomes a random variable, which can be statistically described.
First of all, the probability density function (pdf) of µ k j : pd f (µ k j ) is obtained and its symmetry (or lack thereof) is ascertained.A more precise assessment of the statistical symmetry can be obtained by computing the skewness, which is defined as where µ k j is the mean of µ k j and var µ k j is its variance.

Identifying Symmetries
The symmetric behaviour of the 10 metrics is first determined by means of computing the distance between the baseline and each of the 31 possible transformations, in accordance with Equation (20).The results are depicted in Figure 15.Each row shows the symmetries of a metric.In the columns are the 31 different transformations.Any given metric-transformation pair (small rectangles in the graphic) is shown in yellow if it has zero-distance with the metric baseline.The right-hand-side of the plot (whose code is greater than or equal to 16) corresponds to a combined transformation where the µ axis has been inverted, that is, where the transformation type µ is present.This is therefore the area for antisymmetric behaviour.Let us first analyse each metric in terms of the accuracy (), the Matthews correlation coefficient () and the markedness ().These three metrics present a symmetric behaviour for the combined transformations shown in Table 4.For instance, the first row indicates that the three metrics are symmetric for a combination of the transformations  and  taken in any order ( or ), which corresponds to the code 12 (01100) for a coding scheme (, , , , ) where  represents the Most Significant Bit and  represents the Least Significant Bit.
The results are depicted in Figure 16.Let us first analyse each metric in terms of the accuracy (ACCn), the Matthews correlation coefficient (MCC) and the markedness (MK).These three metrics present a symmetric behaviour for the combined transformations shown in Table 4.For instance, the first row indicates that the three metrics are symmetric for a combination of the transformations δ and σ taken in any order (δσ or σδ), which corresponds to the code 12 (01100) for a coding scheme (µ, σ, δ, β, α) where µ represents the Most Significant Bit and α represents the Least Significant Bit.

Code µ σ δ β α
Specific Order Any Order The first case (code 12) corresponds to the transformation T σδ , or, in other words, to the inverse labelling transformation T L = T σδ which can be formulated for accuracy as The results are depicted in Figure 16.Let us first analyse each metric in terms of the accuracy (), the Matthews correlation coefficient () and the markedness ().These three metrics present a symmetric behaviour for the combined transformations shown in Table 4.For instance, the first row indicates that the three metrics are symmetric for a combination of the transformations  and  taken in any order ( or ), which corresponds to the code 12 (01100) for a coding scheme (, , , , ) where  represents the Most Significant Bit and  represents the Least Significant Bit.
The results are depicted in Figure 16.The second case (code 15) corresponds to 4 transformations ordered in two different ways.In the first ordering, we have T ασβδ = T ασβ •T δ .Recalling Equation ( 17), T ασβ = T σ .It can therefore be written that T ασβδ = T σ •T δ = T σδ = T L , that is, it is equivalent to the inverse labelling transformation.The same result is obtained for T βσαδ .Hence, code 15 is the same case as code 12.
The third case (code 19) corresponds to the transformation T αβµ , or, in other words, to the inverse scoring transformation T S = T αβµ , which can be formulated for accuracy as The results are depicted in Figure 17.
The results are depicted in Figure 17.
The results are depicted in Figure 18.Let us now focus on precision () and the negative prediction value ().These two metrics present a symmetric behaviour for the combined transformations shown in Table 5.Finally, code 31 corresponds to 5 transformations ordered in 4 different ways.In the first ordering we have T αβσδµ but, by considering that the order of T δ and T µ are not relevant, it can also be written as that is, it is equivalent to the full transformation.The same result is obtained for the 3 remaining orderings which can be formulated for accuracy as The results are depicted in Figure 18.
The results are depicted in Figure 17.
The results are depicted in Figure 18.Let us now focus on precision () and the negative prediction value ().These two metrics present a symmetric behaviour for the combined transformations shown in Table 5.Let us now focus on precision (PRCn) and the negative prediction value (NPVn).These two metrics present a symmetric behaviour for the combined transformations shown in Table 5.These two metrics present symmetric behaviour for only the combined transformations code 31 (11111) which, in any of its ordering, is equivalent to the full inversion T F = T L •T S = T αβδσµ and can be formulated for precision as In other words, precision is symmetric with respect to the concatenation of inverse labelling and the inverse scoring transformations.The results are depicted in Figure 19.
In other words, precision is symmetric with respect to the concatenation of inverse labelling and the inverse scoring transformations.The results are depicted in Figure 19.Let us now analyse the geometric mean (), which presents symmetric behaviour for the combined transformations shown in Table 6.Let us now analyse the geometric mean (GMn), which presents symmetric behaviour for the combined transformations shown in Table 6.Table 6.Symmetric transformations of GMn.

Code µ σ δ β α
Specific Order Any Order 4 0 0 1 0 0 δ 8 0 1 0 0 0 σ In first place, code 4 corresponds to T δ .In fact, this metric is not only symmetric with respect to δ but also independent of δ, as it can be seen in Table 3.Secondly, combined transformations coded as 8 and 11 are equivalent to the T σ transformation, that is, GMn is symmetric with respect to the diagonal in the α, β plane.This can be formulated as Finally, codes 12 and 15 imply concatenating T δ to T σ but as the metric is independent of δ, it is again equivalent to T σ , that is, T σδ = T σ •T δ = T σ .These results are depicted in Figure 20.Code      Specific Order Any Order 4 0 0 1 0 0  8 0 1 0 0 0 In first place, code 4 corresponds to  .In fact, this metric is not only symmetric with respect to  but also independent of , as it can be seen in Table 3.Secondly, combined transformations coded as 8 and 11 are equivalent to the  transformation, that is,  is symmetric with respect to the diagonal in the ,  plane.This can be formulated as Finally, codes 12 and 15 imply concatenating  to  but as the metric is independent of , it is again equivalent to  , that is,  =  •  =  .These results are depicted in Figure 20.In the case of bookmaker informedness (), the symmetric behaviour is obtained for the combined transformations shown in Table 7.In the case of bookmaker informedness (BM), the symmetric behaviour is obtained for the combined transformations shown in Table 7.
Table 7. Symmetric transformations of BM.

Code µ σ δ β α
Specific Order Any Order Again code 4 corresponds to T δ as a consequence that this metric is independent of δ (see Table 3).Secondly, combined transformations coded as 8 and 11 are equivalent to the T σ transformation, that is, BM is symmetric with respect to the diagonal in the α, β plane.This can be formulated as Additionally, codes 12 and 15 imply concatenating T δ to T σ but since the metric is independent of δ, it is again equivalent to T σ , that is, T σδ = T σ •T δ = T σ .These results are depicted in Figure 21.
Symmetry 2018, 10, x FOR PEER REVIEW 18 of 31 Again code 4 corresponds to  as a consequence that this metric is independent of  (see Table 3).Secondly, combined transformations coded as 8 and 11 are equivalent to the  transformation, that is,  is symmetric with respect to the diagonal in the ,  plane.This can be formulated as Additionally, codes 12 and 15 imply concatenating  to  but since the metric is independent of , it is again equivalent to  , that is,  =  •  =  .These results are depicted in Figure 21.Code 19 and also code 23 since the metric does not depend on  , correspond to the transformation  or, in other words, to the inverse scoring transformation  =  , which can be formulated for bookmaker informedness as The results are depicted in Figure 22.Code 19 and also code 23 since the metric does not depend on δ, correspond to the transformation T αβµ or, in other words, to the inverse scoring transformation T S = T αβµ , which can be formulated for bookmaker informedness as The results are depicted in Figure 22.
transformation  or, in other words, to the inverse scoring transformation  =  , which can be formulated for bookmaker informedness as The results are depicted in Figure 22.In other words, the bookmaker informedness is symmetric with respect to the inverse labelling and to the inverse scoring transformations.This implies that it is also symmetric with respect to the concatenations of these two transforms, which occurs in codes 27 and 31 (recall that the latter is independent of  ) corresponding to the full inversion  =  +  =  , which can be formulated as The results are depicted in Figure 23.In other words, the bookmaker informedness is symmetric with respect to the inverse labelling and to the inverse scoring transformations.This implies that it is also symmetric with respect to the concatenations of these two transforms, which occurs in codes 27 and 31 (recall that the latter is independent of δ) corresponding to the full inversion T F = T L + T S = T αβδσµ , which can be formulated as The results are depicted in Figure 23.In the case of sensitivity (  ), the symmetric behaviour is found for the combined transformations shown in Table 8.Code      Specific Order Any Order 2 0 0 0 1 0 Codes 2 and 4 correspond to  and  as a consequence of this metric being independent of  and  (see Table 3).Code 19 (and also codes 17, 21 and 23 since the metric does not depend on  nor  ) corresponds to the transformation  , or, in other words, to the inverse scoring transformation  =  , which can be formulated as This result is depicted in Figure 24.In the case of sensitivity (SNSn), the symmetric behaviour is found for the combined transformations shown in Table 8.

Code µ σ δ β α Specific Order Any Order
Codes 2 and 4 correspond to T β and T δ as a consequence of this metric being independent of β and δ (see Table 3 δ) corresponds to the transformation T αβµ , or, in other words, to the inverse scoring transformation T S = T αβµ , which can be formulated as This result is depicted in Figure 24.17 1 0 0 0 1 Codes 2 and 4 correspond to  and  as a consequence of this metric being independent of  and  (see Table 3).Code 19 (and also codes 17, 21 and 23 since the metric does not depend on  nor  ) corresponds to the transformation  , or, in other words, to the inverse scoring transformation  =  , which can be formulated as This result is depicted in Figure 24.On considering the specificity (), its symmetric behaviour is shown in Table 9.On considering the specificity (SPCn), its symmetric behaviour is shown in Table Table 9. Symmetric transformations of specificity.Codes 1 and 4 corresponds to T α and T δ as a consequence of this metric being independent of α and δ (see Table 3).Code 19 (and also codes 18, 22 and 23 as the metric depends neither on α nor on δ) corresponds to the transformation T αβµ , that is, to the inverse scoring transformation T S = T αβµ , which can be formulated as

Code µ σ δ β α Specific Order Any Order
This result is depicted in Figure 25.Code      Specific Order Any Order 1 0 0 0 0 1 Codes 1 and 4 corresponds to  and  as a consequence of this metric being independent of  and  (see Table 3).Code 19 (and also codes 18, 22 and 23 as the metric depends neither on  nor on ) corresponds to the transformation  , that is, to the inverse scoring transformation  =  , which can be formulated as This result is depicted in Figure 25.Finally, it can be observed that the   score metric is not symmetric under any transformation.The results for each metric are summarized in Table 10.Finally, it can be observed that the F 1 n score metric is not symmetric under any transformation.The results for each metric are summarized in Table 10.

Identifying Cross-Symmetries
In order to explore whether any cross-symmetry can be identified among the 10 metrics, we have computed the distance (using Equation ( 20)) of the baseline of each metric (and its 31 possible transformations), to the remaining baseline metrics.The results are depicted in Figure 26.Each row corresponds to the baseline of a metric and each column to the baseline and its 31 transformations of the other metric.Any given metric-metric pair (small squares in the graphic) is shown in yellow if it has zero-distance for any possible transformation.The diagonal presents a summary of the results explored in the previous section, that is, every metric, except for the   score, presents some kind of symmetry under some transformation.The cases of cross-symmetries appear in the elements off diagonal.Two cross-symmetries arise: the  −  and the  − .In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric.The results are depicted in Figure 27.
Each row shows the cross-symmetries of a pair of metrics.In the columns are the 32 different transformations (including the baseline) of the second metric in the pair.Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline.As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the  axis has been inverted, that is, where the transformation type  is present.This is therefore the area for antisymmetric behaviour.The diagonal presents a summary of the results explored in the previous section, that is, every metric, except for the F 1 n score, presents some kind of symmetry under some transformation.The cases of cross-symmetries appear in the elements off diagonal.Two cross-symmetries arise: the SNSn − SPCn and the PRCn − NPVn.
In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric.The results are depicted in Figure 27.Each row shows the cross-symmetries of a pair of metrics.In the columns are the 32 different transformations (including the baseline) of the second metric in the pair.Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline.As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the µ axis has been inverted, that is, where the transformation type µ is present.This is therefore the area for antisymmetric behaviour.In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric.The results are depicted in Figure 27.
Each row shows the cross-symmetries of a pair of metrics.In the columns are the 32 different transformations (including the baseline) of the second metric in the pair.Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline.As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the  axis has been inverted, that is, where the transformation type  is present.This is therefore the area for antisymmetric behaviour.Let us first analyse each pair of metrics in terms of the  −  or  −  pair, which present cross-symmetric behaviour for the combined transformations shown in Table 11.Let us first analyse each pair of metrics in terms of the PRCn − NPVn or PRCn − NPVn pair, which present cross-symmetric behaviour for the combined transformations shown in Table 11.

Code µ σ δ β α
Specific Order Any Order Codes 12 and 15 correspond to the transformation T σδ or, in other words, to the inverse labelling transformation T L = T σδ , which can be formulated as The results are depicted in Figure 28.
The results are depicted in Figure 28.
The results are depicted in Figure 29.Code 19 corresponds to the transformation T αβµ or, in other words, to the inverse scoring transformation T S = T αβµ , which can be formulated as The results are depicted in Figure 29.
The results are depicted in Figure 29.Although the  −  pair is cross-symmetric with respect to the inverse labelling and to the inverse scoring transformations, this does not imply that it is also cross-symmetric with respect to the concatenations of these two transforms (see equation 22).This is the reason why code 31 (corresponding to the full inversion  =  +  =  is not present in Table 11.The results for the pair  −  are exactly the same.Therefore, Let us now consider the pair of metrics  −  and its cross-symmetric behaviour, which is found for the combined transformations shown in Table 12.Although the PRCn − NPVn pair is cross-symmetric with respect to the inverse labelling and to the inverse scoring transformations, this does not imply that it is also cross-symmetric with respect to the concatenations of these two transforms (see Equation ( 22)).This is the reason why code 31 (corresponding to the full inversion T F = T L + T S = T αβδσµ is not present in Table 11.
The results for the pair NPVn − PRCn are exactly the same.Therefore, Let us now consider the pair of metrics SNSn − SPCn and its cross-symmetric behaviour, which is found for the combined transformations shown in Table 12.

Code µ σ δ β α
Specific Order Any Order Since specificity remains independent from δ (see Table 3), codes 25, 11, 12 and 15 correspond to T σδ , that is, to the inverse labelling which can be formulated as µ SNSn (α, β) = µ SPCn (β, α). (37) Additionally, since specificity is also independent of α, then codes 9 (T ασ ) and 13 (T ασδ ) are equivalent to T σδ .Moreover, after a T σ transformation, the resulting metric has no dependence on β (due to the axis inversion) and hence codes 10 (T σβ ) and 14 (T σβδ ) are also equivalent to T σδ .These results are depicted in Figure 30.3), codes 25, 11, 12 and 15 correspond to  , that is, to the inverse labelling which can be formulated as  , ) =  , ).
Additionally, since specificity is also independent of , then codes 9 ( ) and 13 ( ) are equivalent to  .Moreover, after a  transformation, the resulting metric has no dependence on  (due to the axis inversion) and hence codes 10 ( ) and 14 ( ) are also equivalent to  .These results are depicted in Figure 30.
It can be shown that the remaining codes (25, 26, 27, 29 and 30) are also equivalent to  .Moreover, after a  transformation, the resulting metric does not depend on  (due to the axis On the other hand, code 31 corresponds to full inversion transformation T F = T σδαβµ , which can be formulated as It can be shown that the remaining codes (25, 26, 27, 29 and 30) are also equivalent to T F .Moreover, after a T σ transformation, the resulting metric does not depend on β (due to the axis inversion) and hence codes 10 (T σβ ) and 14 (T σβδ ) are also equivalent to T σδ .These results are depicted in Figure 31.The results for the pair  −  are exactly the same, so The results for every pair of cross-symmetric metrics are summarized in Table 13.The results for the pair SPCn − SNSn are exactly the same, so The results for every pair of cross-symmetric metrics are summarized in Table 13.In order to explore the symmetric behaviour of the statistical descriptions of the metrics, let us recall that, for the baseline experiment, µ B j = µ B j α B , β B , δ B can be considered a statistical variable.First of all, let us select a subset of the µ B j corresponding to a certain value δ 0 of the imbalance coefficient, that is, µ B j α B , β B , δ 0 and obtain its probability density function (pdf) which will be called local pdf (since it is obtained solely for a value of δ B ).The results pd f (µ k j , δ 0 ) for every metric with δ B = 0.5 are shown in Figure 32.
The results for every pair of cross-symmetric metrics are summarized in Table 13.

Skewness of the Statistical Descriptions of the Metrics
In order to explore the symmetric behaviour of the statistical descriptions of the metrics, let us recall that, for the baseline experiment,  =   ,  ,  ) can be considered a statistical variable.First of all, let us select a subset of the  corresponding to a certain value  of the imbalance coefficient, that is,   ,  ,  ) and obtain its probability density function (pdf) which will be called local pdf (since it is obtained solely for a value of  ).The results   ,  for every metric with  = 0.5 are shown in Figure 32.This result can be generalized for various values of the imbalance coefficient δ B by obtaining the pd f (µ k j , δ B ) depicted in Figure 33 as a set of heatmap plots.In every plot, the horizontal axis represents the imbalance coefficient while the value of the metric is drawn in the vertical axis.The value of the pd f is colour-coded.This result can be generalized for various values of the imbalance coefficient  by obtaining the   ,  depicted in Figure 33 as a set of heatmap plots.In every plot, the horizontal axis represents the imbalance coefficient while the value of the metric is drawn in the vertical axis.The value of the  is colour-coded.In Figure 32 and Figure 33, the symmetry of the statistical descriptions of the metrics can easily be observed.However, in order to achieve a more precise insight, the local skewness  of every  is obtained in accordance with equation 23 and its value   ) is shown in Figure 34 for every In Figures 32 and 33, the symmetry of the statistical descriptions of the metrics can easily be observed.However, in order to achieve a more precise insight, the local skewness ξ B j of every pd f is obtained in accordance with Equation ( 23) and its value ξ B j δ B is shown in Figure 34 for every metric.It can be observed that 6 metrics (SNSn, SPCn, ACCn, MCC, BM and MK) have a symmetric pd f ; one metric (GMn) has a pd f slightly asymmetric but its asymmetry does not depend on δ B ; 2 metrics (PRCn and NPVn) have a clearly asymmetric pd f but their skewness is symmetric with respect to the origin; and finally, the F 1 n metric has a pd f and a skewness that are both asymmetric.In Figure 32 and Figure 33, the symmetry of the statistical descriptions of the metrics can easily be observed.However, in order to achieve a more precise insight, the local skewness  of every  is obtained in accordance with equation 23 and its value   ) is shown in Figure 34 for every metric.It can be observed that 6 metrics (, , , ,   ) have a symmetric ; one metric () has a  slightly asymmetric but its asymmetry does not depend on  ; 2 metrics ( and ) have a clearly asymmetric  but their skewness is symmetric with respect to the origin; and finally, the   metric has a  and a skewness that are both asymmetric.Let us now examine the  for all the values of the imbalance coefficient  , that is,   ,  ,  ) and obtain its probability density function (pdf) which will be called global pdf (as it is obtained for every  ).The resulting   is shown in Figure 35 for every metric.Let us now examine the µ B j for all the values of the imbalance coefficient δ B , that is, µ B j α B , β B , δ B and obtain its probability density function (pdf) which will be called global pdf (as it is obtained for every δ B ).The resulting pd f (µ k j ) is shown in Figure 35 for every metric.It can be observed that all the metrics show a symmetric  except for  and  .The global pdf for  maintains the slight asymmetry of local pdf (global skewness of 0.18) since  does not depend on .In the cases of  and , the symmetry of the local skewness compensates for their values and hence they show a symmetric global pdf.Finally, the positive values of   local skewness partially compensate for its negative values (see Figure 34), which results in an almost uniform global pdf except for their extreme values (global skewness of 0.14).These results are summarized in Table 14.It can be observed that all the metrics show a symmetric pd f except for GMn and F 1 n.The global pdf for GMn maintains the slight asymmetry of local pdf (global skewness of 0.18) since GMn does not depend on δ.In the cases of PRCn and NPVn, the symmetry of the local skewness compensates for their values and hence they show a symmetric global pdf.Finally, the positive values of F 1 n local skewness partially compensate for its negative values (see Figure 34), which results in an almost uniform global pdf except for their extreme values (global skewness of 0.14).These results are summarized in Table 14.

Discussion
From the previous results, summarized in Tables 10, 13 and 14, it can be seen that although several thousands of combined transformations have been tested, the performance metrics only present three types of symmetries: under labelling inversion; under scoring inversion; and under full inversion (the sequence of labelling and scoring inversion).
For a certain performance metric to be symmetric under labelling inversion means that it pays attention to or focuses on, positive and negative classes with the same intensity and therefore classes can be exchanged without affecting the value of the metric.These metrics should be used in applications where the cost of misclassification is the same for each class.This is the case for 5 out of the 10 metrics tested: ACCn, MCC, BM, MK and GMn.
Other metrics, however, are more focused on the classification results obtained for the positive class.This is the case of 3 metrics: SNSn, which only depends on α; PRCn, which measures the ratio of success on the elements classified as positive; and the F 1 score, which is a combination of SNSn and PRCn.These metrics found their main applications when the cost of misclassifying the positive class is higher than the cost of misclassifying the negative class, for instance, in the case of disease detection in medical diagnostics.Finally, other metrics are more focused on the classification results obtained for the negative class.This is the case of 2 metrics: SPCn, which only depends on β; and NPVn, which measures the ratio of success on the elements classified as negative.These 2 metrics are mainly applied if the most important issue is the misclassification of negative classes, for instance, in the case of identification of non-reliable clients in granting loans.
On the other hand, if a metric shows symmetric behaviour under scoring inversion it means that the good classifiers are positively scored to the same extent as bad classifiers are negatively scored.For instance, let us consider a first classifier which correctly classifies 80% of positive elements and also 70% of negative elements.Additionally, a second classifier obtains a ratio of 20% for positive and 30% for negative elements.A scoring-inversion symmetric-performance metric would have a value of, for example, +0.5 for the first classifier and a value of −0.5 for the second classifier.Therefore, the scoring symmetry indicates the relative importance assigned by the metric to the good and bad classifiers.This is the case for 6 out of the 10 metrics tested: ACCn, MCC, BM, MK, GMn, SNSn and SPCn.Conversely, GMn is more demanding as regards scoring good results than scoring bad results.This feature can be useful if the objective of the classification is focused on obtaining excellent results (and not just good results).Finally, on 3 of the metrics tested (PRCn, NPVn and F 1 n), awarding good results differs from scoring bad results in that it depends on the relative values of the parameters (α, β and δ).
Additionally, it can be seen that metrics showing both labelling and scoring symmetries also show symmetry for the full inversion (concatenation of the two symmetries).This is the case for 4 out of the 10 metrics tested: ACCn, MCC, BM and MK.An interesting result is that for PRCn and NPVn, although they have no labelling nor scoring symmetry, they do have full inversion symmetry.This fact means that swapping the positive and negative class labels also inverts how the good and bad classifiers are scored.An example of all these symmetries can be found in Table 15.A particular degenerate case of symmetry arises when a metric depends on none of the variables.For example, from the results obtained in this research, several metrics have shown themselves to be independent of the imbalance coefficient δ.This is the case for 4 out of the 10 metrics tested: SNSn, SPCn, GMn and BM.This is a particularly interesting result, since these metrics have no kind of bias if the classes are imbalanced.Conversely, the interpretation of classification metrics which do depend on δ should be carefully considered since they can be misleading as to what a good classifier is.
Additionally, some other metrics appear to be independent from the classification success ratios: SNSn, which only depends on α; and SPCn, which only depends on β.This can be interpreted as a sort of one-dimensionality of these metrics, that is, SNSn is only focused on the positive class, while SPCn is only concerned about the negative class.
On the other hand, the two pairs of cross-symmetries found can be straightforwardly interpreted: when the labelling of classes are inverted, SNSn becomes SPCn and PRCn becomes NPVn.Moreover, by exchanging the scoring procedure of good and bad classifiers, PRCn becomes NPVn.
Let us now focus on the interpretation of the results of statistical symmetries.Statistical local symmetry means that, for a certain dataset, that is, for a certain value of the imbalance coefficient, the probability that a random classifier obtains a good score is the same as the probability that it obtains a bad score.This is the case for 6 out of the 10 metrics tested: ACCn, MCC, BM, MK, GMn, SNSn and SPCn.They coincide with the metrics in that they have scoring symmetry, which shows that both concepts are closely related.Conversely, GMn has a greater probability of having a bad result than a good result, which is consistent with the fact that it is more demanding on obtaining excellent results (and not just good results).Additionally, PRCn obtains good results with a higher probability (lower probability in the case of NPVn) if the positive class is the majority class and vice versa if it is the minority class.Awarding good results differs from scoring bad ones in a way that depends on the relative values of the parameters (α, β and δ).Finally, in the case of balanced classes, the probability of obtaining good F 1 n scores is greater than obtaining bad scores for, which shows some sort of indulgent judgment.However, the detailed behaviour of F 1 n scores for different values of δ is more complex.
On the other hand, statistical global symmetry means that the probability that a random classifier operating on a random dataset obtains a good score is the same as obtaining a bad score.This is the case for 8 out of the 10 metrics tested: ACCn, MCC, BM, MK, GMn, SNSn, SPCn, PRCn and NPVn.Conversely, GMn and F 1 n are more likely to have a bad result than a good result, which can be interpreted as meaning that they are slightly tough judges.
On considering all these results and their meanings, the ten metrics can be organized into 5 clusters that show the features described in Table 16.In Table 16, the identification of clusters has been carried out by means of informal reasoning.To formalize these analyses, every metric has been described with a set of features corresponding to the columns in Table 16.Most of the columns are binary valued (yes or no), while others admit several values.For instance, labelling symmetry value can be yes, no, SNSn − SPCn cross-symmetry or PRCn − NPVn cross-symmetry.In these cases, a one-hot coding mechanism (also called 1-of-K scheme) is employed [39].The result is that each metric is defined using a set of 14 features.Although regular or advanced clustering techniques can be used [40][41][42][43], the reduced number of elements in the dataset (10 performance metrics) invites to address the problem using more intuitive methods.Using Principal Component Analysis (PCA) [44], the problem can be reduced to a bi-dimensional plane and its result is depicted in Figure 36.The 5 clusters mentioned in this section clearly appear therein.Another way to represent how performance metrics are grouped according to their symmetries is by drawing a dendrogram [45].To this end, the 14 features are employed to characterize each performance metric.The distances between the metrics are then computed in the space of the ℝ features.These distances are employed to gauge how much the metrics are separated, as shown in Figure 37. Once again, this result is consistent with the 5 previously identified clusters.Another way to represent how performance metrics are grouped according to their symmetries is by drawing a dendrogram [45].To this end, the 14 features are employed to characterize each performance metric.The distances between the metrics are then computed in the space of the R 14 features.These distances are employed to gauge how much the metrics are separated, as shown in Figure 37. Once again, this result is consistent with the 5 previously identified clusters.
Another way to represent how performance metrics are grouped according to their symmetries is by drawing a dendrogram [45].To this end, the 14 features are employed to characterize each performance metric.The distances between the metrics are then computed in the space of the ℝ features.These distances are employed to gauge how much the metrics are separated, as shown in Figure 37. Once again, this result is consistent with the 5 previously identified clusters.

Conclusions
Based on the results obtained in our analysis, it can be stated that the majority of the most commonly used classification performance metrics present some type of symmetry.We have identified 3 and only 3 types of symmetric behaviour: labelling inversion, scoring inversion and the combination of the two inversions.Additionally, several metrics have been revealed as being robust under imbalanced datasets, while others do not show this important feature.Finally two metrics has been identified as one-dimensional, in that they focus exclusively on the positive (sensitivity) or on the negative class (specificity).The metrics have been grouped into 5 clusters according to their symmetries.
Selecting one performance metric or another is mainly a matter of its application, depending on issues such as whether the dataset is balanced, misclassification has the same cost in either class and whether good scores should only be reserved for very good classification ratios.None of the studied metrics can be universally applied.However, according to their symmetries, two of these metrics appear especially worthy in general-purpose applications: the Bookmaker Informedness () and the Geometric Mean ().Both of these metrics are robust under imbalanced datasets and treat both classes in the same way (labelling symmetry).The former metric () also has scoring symmetry while the latter () is slightly more demanding in terms of scoring good results over bad results.

Conclusions
Based on the results obtained in our analysis, it can be stated that the majority of the most commonly used classification performance metrics present some type of symmetry.We have identified 3 and only 3 types of symmetric behaviour: labelling inversion, scoring inversion and the combination of the two inversions.Additionally, several metrics have been revealed as being robust under imbalanced datasets, while others do not show this important feature.Finally two metrics has been identified as one-dimensional, in that they focus exclusively on the positive (sensitivity) or on the negative class (specificity).The metrics have been grouped into 5 clusters according to their symmetries.
Selecting one performance metric or another is mainly a matter of its application, depending on issues such as whether the dataset is balanced, misclassification has the same cost in either class and whether good scores should only be reserved for very good classification ratios.None of the studied metrics can be universally applied.However, according to their symmetries, two of these metrics appear especially worthy in general-purpose applications: the Bookmaker Informedness (BM) and the Geometric Mean (GM).Both of these metrics are robust under imbalanced datasets and treat both classes in the same way (labelling symmetry).The former metric (BM) also has scoring symmetry while the latter (GM) is slightly more demanding in terms of scoring good results over bad results.
In future research, the methodology for the analysis of symmetry developed in this paper can be extended to other classification performance metrics, such as those derived from multiclass confusion matrix or some ranking metrics (i.e.Receiver Operating Characteristic curve).

Figure 1 .
Figure 1.Node of the Wireless Sensor Network where the symmetry of classification performance metrics has been primarily applied.

Figure 1 .
Figure 1.Node of the Wireless Sensor Network where the symmetry of classification performance metrics has been primarily applied.
E B , defined by the duple E B = C B , D B composed of a set of n B classifiers, C B = c B i and a set of their corresponding n B datasets, D B = D B i , i = 1, • • • , n B .The elements in every dataset belong to either of two classes, G 1 and G 2 , which are called Positive (P) and Negative (N) classes, respectively.The i-th classifier c B i operates on the corresponding D B i dataset, thereby obtaining a resulting classification which can be defined by its binary confusion matrix cm B i and hence D B i c B i → cm B i .The set of confusion matrices are denominated Symmetry 2019, 11, 47 3 of 31

Figure 2 .
Figure 2. Imbalance coefficient (solid blue line) and imbalance ratio (dashed green line) vs. the proportion of positive elements in the dataset.

Figure 2 .
Figure 2. Imbalance coefficient (solid blue line) and imbalance ratio (dashed green line) vs. the proportion of positive elements in the dataset.

Figure 3 .Figure 4 .
Figure 3. 3D representation of a 4-dimension metric value   ,  ,  ).The value of the metric  is colour-coded for every point in the  ,  ,  ) 3D space.

Figure 3 .
Figure 3. 3D representation of a 4-dimension metric value µ k j α k , β k , δ k .The value of the metric µ k j is colour-coded for every point in the α k , β k , δ k 3D space.

Figure 3 .
Figure 3. 3D representation of a 4-dimension metric value   ,  ,  ).The value of the metric  is colour-coded for every point in the  ,  ,  ) 3D space.

Figure 4 .
Figure 4. Representation of a metric value   ,  ) for  = 0.75.(a) Slice of the 3D graphic by a plane corresponding to  = 0.75; (b) 2D representation of the slice.

Figure 4 .
Figure 4. Representation of a metric value µ k j α k , β k for δ = 0.75.(a) Slice of the 3D graphic by a plane corresponding to δ = 0.75; (b) 2D representation of the slice.

Figure 6 .
Figure 6.Panel of heat maps representing the metric µ k j α k , β k , δ k .

Figure 7 .
Figure 7. Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.5.

Figure 8 .
Figure 8. Transformation type  of a metric.(a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane  = 0.5.

Figure 7 .
Figure 7. Transformation type α of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane α = 0.5.Analogously, type β transformation implies that the i-th transformed classifier (c β i ) shows a ratio of correctly classified negative elements (β β i ), which has the symmetric value of the ratio β B i obtained by the baseline classifier c Bi .Since the value of such ratios also lie within the range [0, 1], the symmetry exists with respect to the hyperplane β = 0.5 and can be stated as β

Figure 7 .
Figure 7. Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.5.

Figure 8 .
Figure 8. Transformation type  of a metric.(a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane  = 0.5.

Figure 9 .
Figure 9. Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0. Finally, type  transformation jointly operates on classifiers and datasets in such a way that the  -th of performance metrics  for the  classifier operating on the  dataset has the symmetric value of the performance metric in the baseline experiment  .Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane  = 0 and can be stated as  = −  .An example of this transformation is depicted in Figure10where it should be noted that the  dimension is shown by the colour code of each point.Therefore, an inversion in  is shown as a colour inversion.

Figure 10 .
Figure 10.Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.

Figure 11 .
Figure 11.Transformation type  of a metric.(a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane  = .

Symmetry 2018 , 31 Figure 9 .
Figure 9. Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0. Finally, type  transformation jointly operates on classifiers and datasets in such a way that the  -th of performance metrics  for the  classifier operating on the  dataset has the symmetric value of the performance metric in the baseline experiment  .Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane  = 0 and can be stated as  = −  .An example of this transformation is depicted in Figure 10 where it should be noted that the  dimension is shown by the colour code of each point.Therefore, an inversion in  is shown as a colour inversion.

Figure 10 .
Figure 10.Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.

Figure 11 .
Figure 11.Transformation type  of a metric.(a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane  = .

Symmetry 2018 , 31 Figure 9 .
Figure 9. Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.Finally, type  transformation jointly operates on classifiers and datasets in such a way that the  -th of performance metrics  for the  classifier operating on the  dataset has the symmetric value of the performance metric in the baseline experiment  .Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane  = 0 and can be stated as  = −  .An example of this transformation is depicted in Figure 10 where it should be noted that the  dimension is shown by the colour code of each point.Therefore, an inversion in  is shown as a colour inversion.

Figure 10 .
Figure 10.Transformation type  of a metric.(a) Baseline metric.(b) Reflection symmetry with respect to the hyperplane  = 0.

Figure 11 .
Figure 11.Transformation type  of a metric.(a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane  = .

Figure 15 .
Figure 15.Symmetric behaviour of performance metrics for any combined transformation.

Code
Specific Order Any Order 12 case (code 12) corresponds to the transformation  , or, in other words, to the inverse labelling transformation  =  which can be formulated for accuracy as  , , ) =  , , −).

Figure 16 .
Figure 16.Symmetry of accuracy with respect to inverse labelling ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ).

Figure 15 .
Figure 15.Symmetric behaviour of performance metrics for any combined transformation.

Symmetry 2018 , 31 Figure 15 .
Figure 15.Symmetric behaviour of performance metrics for any combined transformation.
case (code 12) corresponds to the transformation  , or, in other words, to the inverse labelling transformation  =  which can be formulated for accuracy as  , , ) =  , , −).

Figure 16 .
Figure 16.Symmetry of accuracy with respect to inverse labelling ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ).

Figure 16 .
Figure 16.Symmetry of accuracy with respect to inverse labelling (T L ).(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (T σ ); (c) Reflection symmetry with respect to the plane δ = 0 (T δ ).

Symmetry 2018 ,
10, x FOR PEER REVIEW 15 of 31 The second case (code 15) corresponds to 4 transformations ordered in two different ways.In the first ordering, we have  =  •  .Recalling equation 17,  =  .It can therefore be written that  =  •  =  =  , that is, it is equivalent to the inverse labelling transformation.The same result is obtained for  .Hence, code 15 is the same case as code 12.The third case (code 19) corresponds to the transformation  , or, in other words, to the inverse scoring transformation  =  , which can be formulated for accuracy as  , , ) = − 1 − , 1 − , ).

Figure 17 .
Figure 17.Symmetry of accuracy with respect to the inverse scoring ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (d) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Finally, code 31
corresponds to 5 transformations ordered in 4 different ways.In the first ordering we have  but, by considering that the order of  and  are not relevant, it can also be written as  =  =  •  =  •  =  , that is, it is equivalent to the full transformation.The same result is obtained for the 3 remaining orderings which can be formulated for accuracy as  , , ) = − 1 − , 1 − , −).

Figure 18 .
Figure 18.Symmetry of accuracy with respect to the full inversion ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (d) Reflection symmetry with respect to the main diagonal ( ); (e) Reflection symmetry with respect to the plane  = 0 ( ); (f) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 17 .
Figure 17.Symmetry of accuracy with respect to the inverse scoring (T S ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane α = 0 (T α ); (c) Reflection symmetry with respect to the plane β = 0 (T β ); (d) Reflection symmetry with respect to the plane µ = 0 (colour inversion, T µ ).

Symmetry 2018 ,
10, x FOR PEER REVIEW 15 of 31 The second case (code 15) corresponds to 4 transformations ordered in two different ways.In the first ordering, we have  =  •  .Recalling equation 17,  =  .It can therefore be written that  =  •  =  =  , that is, it is equivalent to the inverse labelling transformation.The same result is obtained for  .Hence, code 15 is the same case as code 12.The third case (code 19) corresponds to the transformation  , or, in other words, to the inverse scoring transformation  =  , which can be formulated for accuracy as  , , ) = − 1 − , 1 − , ).

Figure 17 .
Figure 17.Symmetry of accuracy with respect to the inverse scoring ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (d) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Finally, code 31
corresponds to 5 transformations ordered in 4 different ways.In the first ordering we have  but, by considering that the order of  and  are not relevant, it can also be written as  =  =  •  =  •  =  , that is, it is equivalent to the full transformation.The same result is obtained for the 3 remaining orderings which can be formulated for accuracy as  , , ) = − 1 − , 1 − , −).

Figure 18 .
Figure 18.Symmetry of accuracy with respect to the full inversion ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (d) Reflection symmetry with respect to the main diagonal ( ); (e) Reflection symmetry with respect to the plane  = 0 ( ); (f) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 18 .
Figure 18.Symmetry of accuracy with respect to the full inversion (T F ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane α = 0 (T α ); (c) Reflection symmetry with respect to the plane β = 0 (T β ); (d) Reflection symmetry with respect to the main diagonal (T σ ); (e) Reflection symmetry with respect to the plane δ = 0 (T δ ); (f) Reflection symmetry with respect to the plane µ = 0 (colour inversion, T µ ).

Symmetry 2018 ,
10, x FOR PEER REVIEW 16 of 31   These two metrics present symmetric behaviour for only the combined transformations code 31 (11111) which, in any of its ordering, is equivalent to the full inversion  =  •  =  and can be formulated for precision as  , , ) = − 1 − , 1 − , −).

Figure 19 .
Figure 19.Symmetry of precision with respect to the full inversion ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (d) Reflection symmetry with respect to the main diagonal ( ); (e) Reflection symmetry with respect to the plane  = 0 ( ); (f) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 19 .
Figure 19.Symmetry of precision with respect to the full inversion (T F ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane α = 0 (T α ); (c) Reflection symmetry with respect to the plane β = 0 (T β ); (d) Reflection symmetry with respect to the main diagonal (T σ ); (e) Reflection symmetry with respect to the plane δ = 0 (T δ ); (f) Reflection symmetry with respect to the plane µ = 0 (colour inversion, T µ ).

Figure 20 .
Figure 20.Symmetry of geometric mean with respect to  .(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal ( ).

Figure 20 .
Figure 20.Symmetry of geometric mean with respect to T σ .(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (T σ ).

Figure 21 .
Figure 21.Symmetry of bookmaker informedness with respect to  .(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal ( ).

Figure 21 .
Figure 21.Symmetry of bookmaker informedness with respect to T σ .(a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (T σ ).

Symmetry 2018 , 31 Figure 23 .
Figure 23.Symmetry of bookmaker informedness with respect to the full inversion ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the main diagonal ( ); (d) Reflection symmetry with respect to the plane  = 0 ( ); (e) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 23 .
Figure 23.Symmetry of bookmaker informedness with respect to the full inversion (T F ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane α = 0 (T α ); (c) Reflection symmetry with respect to the plane β = 0 (T β ); (c) Reflection symmetry with respect to the main diagonal (T σ ); (d) Reflection symmetry with respect to the plane δ = 0 (T δ ); (e) Reflection symmetry with respect to the plane µ = 0 (colour inversion, T µ ).

Figure 24 .
Figure 24.Symmetry of sensitivity with respect to the combined transformation ( ).(a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 25 .
Figure 25.Symmetry of specificity with respect to the combined transformation ( ) (a) Baseline metric; (b) Reflection symmetry with respect to the plane  = 0 ( ); (c) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 25 .
Figure 25.Symmetry of specificity with respect to the combined transformation (T βµ ) (a) Baseline metric; (b) Reflection symmetry with respect to the plane β = 0 (T β ); (c) Reflection symmetry with respect to the plane µ = 0 (colour inversion, T µ ).
Symmetry 2018, 10, x FOR PEER REVIEW 21 of 31 transformations), to the remaining baseline metrics.The results are depicted in Figure 26.Each row corresponds to the baseline of a metric and each column to the baseline and its 31 transformations of the other metric.Any given metric-metric pair (small squares in the graphic) is shown in yellow if it has zero-distance for any possible transformation.

Figure 26 .
Figure 26.Cross-symmetric behaviour of performance metrics for any combined transformation.

Figure 26 .
Figure 26.Cross-symmetric behaviour of performance metrics for any combined transformation.

Figure 26 .
Figure 26.Cross-symmetric behaviour of performance metrics for any combined transformation.

Figure 27 .
Figure 27.Cross-symmetric behaviour for any combined transformation.

Figure 27 .
Figure 27.Cross-symmetric behaviour for any combined transformation.

Figure 28 .
Figure 28.Cross-symmetry of the  −  pair with respect to the inverse labelling ( ).(a) Baseline  metric; (b) Baseline  metric; (c) Reflection symmetry of  with respect to the main diagonal ( ); (d) Reflection symmetry of  with respect to the plane  = 0 ( ).Code 19 corresponds to the transformation  or, in other words, to the inverse scoring transformation  =  , which can be formulated as  , , ) = − 1 − , 1 − , ).(35)

Figure 28 .
Figure 28.Cross-symmetry of the PRCn − NPVn pair with respect to the inverse labelling (T L ).(a) Baseline PRCn metric; (b) Baseline NPVn metric; (c) Reflection symmetry of NPVn with respect to the main diagonal (T σ ); (d) Reflection symmetry of NPVn with respect to the plane δ = 0 (T δ ).
Code 19 corresponds to the transformation  or, in other words, to the inverse scoring transformation  =  , which can be formulated as  , , ) = − 1 − , 1 − , ).

Figure 29 .
Figure 29.Cross-symmetry of the  −  pair with respect to the inverse scoring ( ).(a) Baseline  metric; (b) Baseline  metric.(c) Reflection symmetry of  with respect to the plane  = 0 ( ); (d) Reflection symmetry of  with respect to the plane  = 0 ( ); (e) Reflection symmetry of  with respect to the plane  = 0 (colour inversion,  ).
independent from  (see Table

Figure 30 .
Figure 30.Cross-symmetry of the  −  pair with respect to the inverse labelling ( ).(a) Baseline  metric; (b) Baseline  metric; (c) Reflection symmetry of  with respect to the main diagonal ( ); (d) Reflection symmetry of  with respect to the plane  = 0 ( ).On the other hand, code 31 corresponds to full inversion transformation  =  , which can be formulated as

Figure 30 .
Figure 30.Cross-symmetry of the SNSn − SPCn pair with respect to the inverse labelling (T L ).(a) Baseline SNSn metric; (b) Baseline SPCn metric; (c) Reflection symmetry of SPCn with respect to the main diagonal (T σ ); (d) Reflection symmetry of SPCn with respect to the plane δ = 0 (T δ ).

Symmetry 2018 ,
10, x FOR PEER REVIEW 24 of 31 inversion) and hence codes 10 ( ) and 14 ( ) are also equivalent to  .These results are depicted in Figure31.

Figure 31 .
Figure 31.Cross-symmetry of the  −  pair with respect to the full inversion ( ).(a) Baseline  metric.(b) Baseline  metric.(c) Reflection symmetry of  with respect to the main diagonal ( ).(d) Reflection symmetry of  with respect to the plane  = 0 ( ).(e) Reflection symmetry with respect to the plane  = 0 ( ).(f) Reflection symmetry with respect to the plane  = 0 ( ).(g) Reflection symmetry with respect to the plane  = 0 (colour inversion,  ).

Figure 32 .
Figure 32.Local probability density function of every metric and  = 0.

Figure 32 .
Figure 32.Local probability density function of every metric and δ = 0.

Figure 33 .
Figure 33.Local probability density function of every metric as a function of .The value of pdf is colour coded.

Figure 33 .
Figure 33.Local probability density function of every metric as a function of δ.The value of pdf is colour coded.

Figure 33 .
Figure 33.Local probability density function of every metric as a function of .The value of pdf is colour coded.

Figure 34 .
Figure 34.Skewness of the statistical description for every metric as a function of .

Figure 34 .
Figure 34.Skewness of the statistical description for every metric as a function of δ.

Figure 35 .
Figure 35.Global probability density function of every metric and  = 0.

Figure 35 .
Figure 35.Global probability density function of every metric and δ = 0.

Symmetry 2018 ,
10, x FOR PEER REVIEW 29 of 31a bi-dimensional plane and its result is depicted in Figure36.The 5 clusters mentioned in this section clearly appear therein.

Figure 36 .
Figure 36.Bi-dimensional representation of performance metrics according to their symmetries.

Figure 36 .
Figure 36.Bi-dimensional representation of performance metrics according to their symmetries.

Figure 37 .
Figure 37. Dendrogram of performance metrics according to their symmetries.

Figure 37 .
Figure 37. Dendrogram of performance metrics according to their symmetries.

Table 2 .
Example of the coding of combined transformations.

Table 3 .
Definition of classification performance metrics.

Table 3 .
Definition of classification performance metrics.

Table 3 .
Definition of classification performance metrics.

Table 4 .
Symmetric transformations of ,  and .

Table 4 .
Symmetric transformations of ACCn, MCC and MK.

Table 4 .
Symmetric transformations of ,  and .

Table 5 .
Symmetric transformations of  and .

Table 5 .
Symmetric transformations of  and .

Table 8 .
Symmetric transformations of sensitivity.

Table 8 .
Symmetric transformations of sensitivity.

Table 11 .
Cross-symmetric transformations of the  −  pair.

Table 14 .
Summary of statistical symmetry.

Table 14 .
Summary of statistical symmetry.

Table 15 .
Examples of symmetric behaviour of metrics under several transformations (for balanced classes).Numbers in bold represent cases of asymmetric behaviour.

Table 16 .
Summary of symmetric behaviour.