Linear Dimensionality Reduction: What Is Better?

Abstract

This research paper focuses on dimensionality reduction, which is a major subproblem in any data processing operation. Dimensionality reduction based on principal components is the most used methodology. Our paper examines three heuristics, namely Kaiser’s rule, the broken stick, and the conditional number rule, for selecting informative principal components when using principal component analysis to reduce high-dimensional data to lower dimensions. This study uses 22 classification datasets and three classifiers, namely Fisher’s discriminant classifier, logistic regression, and K nearest neighbors, to test the effectiveness of the three heuristics. The results show that there is no universal answer to the best intrinsic dimension, but the conditional number heuristic performs better, on average. This means that the conditional number heuristic is the best candidate for automatic data pre-processing.

1. Introduction

There are several reasons to use dimensionality reduction (DR) in most data-based projects. The first reason is data-hungry problems [1]. This type of problem is very common in the modern world. In reality, this means that the number of observations is less than the number of attributes [2] or even less than the logarithm of the number of attributes [3,4]. The second reason for DR is the difference between the number of attributes and the intrinsic dimension (ID) of data: there are no reasons to spend resources to work with a large number of attributes when the ID of data is essentially less. Differences between the number of attributes and the ID can depend on many factors. In [5], Mirkes et al. presented table (Table 2 in [5]) with five ID values for 26 datasets. For some datasets, some of the ID values are the same as the number of attributes (for example, the cryotherapy and immunotherapy conditional number dimension equals the number of attributes: 6 and 7, respectively), but for some datasets, the ID is several times less than the number of attributes (for example, for MiniBooNE particle identification, the number of attributes is 50, but the maximum of the tested IDs is 4).

There is no unique, exact definition of ID because of the existence of many IDs. Bac et al. in [6] presented software to calculate 19 IDs and presented a list of IDs that is not exhaustive. In [6], Bac et al. wrote, “The well-known curse of dimensionality, which states that many problems become exponentially difficult in high dimensions, does not depend on the number of features, but on the dataset’s ID [7]. More precisely, the effects of the dimensionality curse are expected to be manifested when $ID\gg ln\left(M\right)$, where M is the number of data points [8,9]”.

According to [10] there are two main DR approaches: filters and wrappers. Filtering methods do not required feedback on predictor/classifier performance. These methods can be considered universal methods to estimate IDs. In contrast, wrapping methods provide feature selection based on predictor/classifier feedback. The most well known filtering method based on principle components (PCs) is LASSO [11]. The set of selected features depends on the used predictor/classifier. This means that the ID can differ depending on the method. This method works for linear regression, generalised linear models, and some other models. There is no k nearest neighbours (KNN) or decision tree version of LASSO. In this study, we decided to consider only linear filtering DR methods.

There are two main classes of ID: projective and non-projective. For projective ID, there is a procedure of projection of data points into lower dimensional space. The most well known projective DR methods are methods based on PCs. There are many heuristics to identify a number of informative (useful, meaningful, etc.) PCs, some of which are considered below. There are many non-projective IDs: the correlation (fractal) dimension [12], manifold-adaptive fractal dimension [13], the method of moments [14], maximum likelihood [15,16,17], and estimators based on the concentration of measure [18,19,20,21,22,23,24].

There are also semi-projective methods (e.g., minimum spanning trees [25]). Such methods allow for the projection of points into low-dimensional space but require the full recalculation of embedding if any points are added to/removed from the dataset.

Many definitions of ID are based on different properties of “dimensionality”. In this study, we consider only projective linear DR. Reviews of different IDs and their usage can be found in [6,26,27]. Macocco et al. considered IDs for discrete spaces [28].

Almost all linear methods of dimensionality reduction are based on PCs. The standard version of PCs was proposed by Karl Pearson in 1901 [29]. There are many generalisations of PCs for different problems, for example, to search for PCs that do not explain the general variance of the dataset but select directions useful for solving the problem under consideration. Usually, such generalisations are called supervised principal components (SPCs). Some SPCs can be found in [30,31,32,33,34]. Recently, a version of PCs was proposed for the domain adaptation problem [35]. There are also sparse PCs [36] specially developed for genomics problems, where the number of attributes is essentially (tens or even hundreds of times) greater than the number of observations. There are also nonlinear generalisations of PCs. Some such generalisations are presented in [37].

An Alternative to the PC linear DR method is DR based on linear discriminant analysis (LDA) directions [38]. The main preference of this method is a search for direction, which is most useful for classification. The main drawback of this method is the ability to define only $C-1$ dimensions, where C is the number of classes. Since all benchmarks investigated in this paper are binary classification problems, the maximal LDA dimension for them is one, and this dimension is not appropriate for most of the considered benchmarks.

In this paper, we consider only projective linear DR methods based on PCs. We consider standard-version PCs and the following rules for defining the number of informative PCs:

• The Kaiser rule [39,40];
• The broken stick rule [41];
• The Fukunaga–Olsen rule or the condition number of the covariance matrix rule [23,42].

A detailed description of these rules is presented in Section 3. All these rules are widely used, and the question of which of these rules is preferable is open. In this study, we tried to answer this question.

We estimated the quality of DR based on the quality of the solution to the classification problem. From our point of view, it is essentially more informative and reasonable than a comparison of IDs.

Our goal is to identify heuristics to identify the number of PCs that allow us to solve the classification problem successfully.

The general result was expected: there is no universally best method of DR, but, on average, the Fukunaga–Olsen rule is better than the Kaiser and broken stick rules.

In [43], Ayesha et al. presented a nice review of DR. The authors compared many different DR techniques but for a specific area of usage and general properties:

• DR features: Supervised/unsupervised, linear/nonlinear, ability to work with data with non-Gaussian distribution, etc. (see Table 2 in [43]);
• Robustness to outliers, sensitivity to noise, robustness to singularity of data (see Table 3 in [43]);
• Areas of data sources (see Table 4 in [43]);
• Types of original data: text, image, signals, etc. (see Table 5 in [43]).

The authors did not consider the classification problem and did not compare DR methods according to the quality of the final classifier.

In [44], Tang et al. compared several DR methods for clustering with the purity-induced measure microaverage of classification accuracy. Their work was similar to our study but considered different DR methods and clustering instead of classification.

In a paper titled “Dimensionality Reduction: A Comparative Review” [10], Van der Maaten et al. presented the results of a similar study. They studied 13 DR methods, one of which was PCA. They used generalisation errors, trustworthiness ((28) in [10]), and continuity ((29) in [10]) as quality measure for a nearest neighbour classifier. According to the authors, PCA does not have parameters, and the authors defined “the target dimensionality in the experiments by means of the maximum likelihood intrinsic dimensionality estimator [16]”. As we can see, the authors defined number of used components by estimating the ID, and the choice of this ID was not motivated. In our study, we compared three different estimations of ID.

In [45], Konstorum et al. compared four DR methods, including PCA for mass cytometry datasets “with respect to computation time, neighborhood proportion error, residual variance, and ability to cluster known cell types and track differentiation trajectories”. As we can see, the classification problem is outside the scope of consideration. Unfortunately, the authors simply chose to require a dimension of 2 without any explanation.

Most reviews of DR methods do not consider methods of estimation of the number of informative PCs (see, for example, [43,46]). Other reviews and comparative studied simply used the stated method (“means of the maximum likelihood intrinsic dimension” in [10] or the elbow rule or fixing of the threshold (${\lambda }_0$) using only PCs with eigenvalues greater than ${\lambda }_0$ in [47]). Deegalla et al. [48] estimated the number of PCs by exhaustively testing for the selected classifier. This may be the best approach, but it can be very expensive for classifiers other than KNN because of the long time required for classifier parameter fitting.

Another set of studies compared different heuristics to chose the number of PCAs. For example, in [41], Jackson described nine heuristics, include the Kaiser rule and the broken stick rule. The Fukunaga–Olsen rule—or the condition number of the covariance matrix rule—was published in 1971 but was not included in Jackson’s review. Jackson applied PCA for data with a known ID and simply compared the estimated number with the known ID. This is a robust measure, but it has no relation to the classification problem. The author found that the broken stick method was best among the compared methods, but in our work, we found that for the classification problem, the broken stick method was the worst.

In [49], Ledesma considered the number of factors to retain and consider four heuristics. The only included PCA heuristic was the Kaiser rule.

In [50], Cangelosi et al. described several heuristics for application to cDNA microarray data. The only conclusion was that several heuristics provided consistent IDs for some databases.

Our goal is to recommend one a heuristic for automatic data pre-processing or for non-experienced researchers. The literature review above shows that, to the best of our best knowledge, there has been no comparison of different heuristics for the estimation of the number of PCs necessary for the classification problem.

The remainder of this paper is organised as follows. Section 2 presents a list of benchmarks used in this study. Section 3 presents descriptions of the used IDs. Section 4 describes the classifiers used in this study. Section 5 presents the testing protocol and the methods and results of comparison. Section 6 presents the results of testing. Section 7 contains our conclusions.

2. Data

In this study, we used 22 benchmarks taken from the UCI machine learning data repository [51]. To select the datasets, we applied the following criteria:

• Dataset does not contain any missing values.
• Data are not time series.
• Dataset is formed for the binary classification problem.
• The number of attributes is less than the number of observations and is greater than 2.
• All predictors are binary or numeric.

The used benchmarks are presented in Table

Table 1. Datasets selected for analysis.

Name	Source	#Attr.	Cases	PCA-K	PCA-BS	PCA-CN
Banknote authentication	[52]	4	1372	2	2	3
Blood	[53]	4	748	2	2	3
Breast Cancer	[54]	30	569	6	3	5
Climate Model Simulation Crashes	[55]	18	540	10	1 *	18
Cryotherapy	[56,57,58]	6	90	3	1 *	6
Diabetic Retinopathy Debrecen	[59,60]	19	1151	5	3	8
EEG Eye State	[61]	14	14,980	4	4	5
HTRU2	[62,63,64]	8	17,898	2	2	4
Immunotherapy	[56,57,65]	7	90	3	1 *	7
ILPD (Indian Liver Patient Dataset)	[66]	10	579	4	1 *	7
Madelon	[67,68]	500	2600	224	1*	362
MiniBooNE particle identification	[69]	50	130,064	4	1	1
Musk 1	[70]	166	476	23	9	7
Musk 2	[70]	166	6598	25	13	6
Planning Relax	[71]	10	182	4	1 *	6
QSAR biodegradation	[72,73]	41	1055	11	6	15
Skin Segmentation	[74]	3	245,057	1	1	2
Connectionist Bench (Sonar)	[75]	60	208	13	6	11
SPECT Heart	[76]	22	267	7	3	12
SPECTF Heart	[76]	44	267	10	3	6
MAGIC Gamma Telescope	[77]	10	19,020	3	1	6
Vertebral Column	[78]	6	310	2	1	5

* The maximal eigenvalue is less than the first broken stick value: the PCA-BS dimension is zero.

. We can observe wide variation in the number of attributes (from 3 to 500) and the number of records (from 90 to 245,057).

Since some datasets have very big differences in values of different attributes (for example, for the Musk 2 dataset, the variance of 76 attributes is 14, and that of 64 attribute is 131), we decided to standardise all attributes of all datasets: we subtracted the mean value and divided by the standard deviation. As a result, each attribute of each pre-processed dataset has a zero mean and unit variance.

3. Used Intrinsic Dimensions

In this study, we used only projective DR methods because we need not only estimate the dimension but also project data to lower dimensional space and solve the classification problem. Since we want to use cross validation to estimate the quality of classifiers (see Section 5), we cannot use semi-projective methods because these methods require complete recalculation of projections into low-dimensional space if data are changed. Calculation of PCs followed by projection on the basis of all data (training and test sets or for all folds) compromises data because the test set is used in data pre-processing. As a result, we calculate PCs for the training set only. This means that we have 100 sets of PCs for each dataset (one collection of PCs for each fold).

We cannot use LDA-based DR methods because this means using projections into one-dimensional space, which is too radical for most datasets.

As a result, we can use only PC-based DR methods, and the question is, ’What is the best rule to define the number of PCs to use?’. We consider three of the most popular formal methods of estimation of the number of informative PCs:

• The Kaiser rule [39,40];
• The broken stick rule [41];
• The Fukunaga–Olsen or the condition number of the covariance matrix rule [23,42].

There are several informal methods, like the elbow rule [41,79], but these methods are not appropriate for automatic work. There are also methods based on the explanation of the desired part of the total data variance (usually 95%). We do not consider such methods because it is very difficult to explain clearly why the specified value (e.g., 95%) is an appropriate choice for the problem under consideration.

In the next paragraph and in the following three subsections, we follow [5].

Let us consider a dataset (X) with n records ($X=x_1,\dots ,x_n$) and d real-valued attributes ($x_i=(x_{i1},\dots ,x_{id})$). The empirical covariance matrix ($\Sigma \left(X\right)$) is symmetric and non-negative definite. The eigenvalues of the $\Sigma \left(X\right)$ matrix are non-negative real numbers. These values are denoted as ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_d$. PCs are defined using the eigenvectors of matrix $\Sigma \left(X\right)$. If the ith eigenvector ($w_i$) is defined, then the ith principal coordinate of the data vector (x) is the inner product $(x,w_i)$. The Fraction of Variance Explained (FVE) by the ith principal component for the dataset (X) is

$$f_i={\displaystyle \frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}}.$$

3.1. The Kaiser Rule

The Kaiser rule [39,40] states that all principal components with an FVE greater than or equal to the average FVE are informative. The average FVE is $1/d$. Thus, components with $f_i\ge 1/d$ are considered informative and should be retained, and components with $f_i<1/d$ should not. Another popular version uses thresholds that are twice as low ($0.5/d$) and retains more components.

Furthermore, we use PCA-K as the name of the DR method, as it is defined by the Kaiser rule and the number of PCs.

3.2. The Broken Stick Rule

The broken stick rule [41] compares set $f_i$ with the distribution of random intervals that appear if we break the stick at $d-1$ points randomly and independently sampled from the uniform distribution. Consider a unit interval (stick) randomly broken into d fragments. Let us numerate these fragments in descending order of their length: $s_1\ge s_2\ge \cdots \ge s_d$. The expected length of fragment i is [41]

$$b_i={\displaystyle \frac{1}{d}}\sum _{j=i}^d{\displaystyle \frac{1}{j}}.$$

(1)

The broken stick rule states that the first k principal components are informative, where k is the maximum number such that $f_i\ge b_i,\forall i\le k$.

Furthermore, we use PCA-BS as the name of this DR method, which is defined by the broken stick rule and the number of PCs.

3.3. The Fukunaga–Olsen or Condition Number Rule

In many problems, the empirical covariance matrix degenerates or almost degenerates. That means that the smallest eigenvalues are much smaller than the largest one. Consider the projection of data on the first k principal components: $\widehat{X}=XV$, where the columns of the matrix (V) are the first k eigenvectors of matrix $\Sigma \left(X\right)$. Eigenvalues of the empirical covariance matrix of the reduced data ($\Sigma \left(\widehat{X}\right)$) are ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$. After DR, the condition number (the ratio of the lowest eigenvalue to the greatest eigenvalue) [80] of the reduced covariance matrix should not be too high in order to avoid the multicollinearity problem. The relevant definition [23,42] of intrinsic dimensionality refers directly to the condition number of the matrix ($\Sigma \left(\widehat{X}\right)$): k is the number of informative principal components if it is the smallest number such that

$${\displaystyle \frac{{\lambda }_{k+1}}{{\lambda }_1}}<{\displaystyle \frac{1}{C}},$$

(2)

where C is the specified condition number, for example, $C=10$. This approach is hereafter referred to as PCA-CN. The PCA-CN intrinsic dimensionality is defined as the number of eigenvalues of the covariance matrix exceeding a specified fraction of its largest eigenvalue [42].

3.4. General Comparison of Three IDs

Before a study question about which of the three listed IDs is better, it is necessary to investigate the following question: Are these IDs statistically significantly different? Upon first looking at Table 1, it seems that the broken stick rule is too radical: for six datasets, the FVE of the first PC is less than the first broken stick value ($b_1$). This means that according to the broken stick rule, the ID of the data is zero. In such cases, we increase the dimension to 1. From the other side, there are datasets that have PCA-BS values greater than the PCA-K or PCA-CN value. It is interesting, but for all considered datasets, PCA-BS was never preferred.

To test the significance of differences in dimensions that can be used, Student’s t-test for dependent samples (t-test) [81] can be used if we want to compare mean values, the Wilcoxon signed-rank test (WSR test) [82] if we want to compare medians of two samples; or the Kolmogorov–Smirnov test (KS test) [83] can be used if we want to compare two empirical distributions. We exactly understand that the KS test was developed for two independent samples, but we could find any statistical test for paired samples to test differences between two distributions. Since we have only 22 datasets, we decided to use the 95% confidence level (the critical value is 5%).

We used all three listed tests for dimensions, as presented in Table 1. Since we have only three IDs, we can ignore the multiple testing problem [84]. The p-values for three tests with three pairs of original dimensions and one of IDs and for three pairs of IDs are presented in

Table 2. p-values of Student’s t-test (T-test) for dependent samples, Wilcoxon’s signed-rank test (WSR test), and the Kolmogorov–Smirnov test (KS test) for pair comparison of the original dimension (Attr) and three IDs: PCA with the Kaiser rule (PCA-K), the broken stick rule (PCA-BS), and the Fukunaga–Olsen rule (PCA-CN).

Rule 1	Rule 2	T-Test	WSR Test	KS Test
Attr	PCA-K	0.01465	<0.00001	0.02005
Attr	PCA-BS	0.03789	<0.00001	<0.00001
Attr	PCA-CN	0.00935	0.00013	0.02005
PCA-K	PCA-BS	0.18430	0.00028	0.04935
PCA-K	PCA-CN	0.34277	0.12068	0.21837
PCA-BS	PCA-CN	0.23360	0.00126	0.00067

. For comparison of the original dimension with IDs, we can see that, formally, all IDs are significantly different from the number of attributes. However the t-test has a relatively large p-value for PCA-BS. In the ID comparisons, the t-test shows that we do not have enough evidence to reject the hypothesis that all three samples of distances were taken from populations with the same mean value. From the KS test, we can conclude that we do not have enough evidence to reject the hypothesis that distributions of IDs for PCA-K and for PCA-CN are identical, but both these distributions are statistically significantly different from the distribution of IDs for PCA-BS with a confidence level of 95%. From the WSR test, we can conclude that the median of PCA-BS is statistically significantly less than the median of PCA-K and the median of PCA-CN with a significance level of 99%. This finding confirms the well known fact that the Kaiser rule is too conservative and considers too many PCs as informative [79] compared to the broken stick rule. From the other side, the Fukunaga–Olsen rule is even more conservative than Kaiser’s rule.

Let us compare the three listed rules as possible IDs. What properties do we expect from any dimension? This value must be, at least, uniquely defined. This means that if our data have a dimension of d, then the data have a dimension of d in any space that includes these data. This also means that if we originally had a dataset ($DB1$) and used the DR method ($DR$), then we have $dim\left(DR\right(DB1\left)\right)=d$, where $dim\left(S\right)$ is the dimension of space S. Moreover, we can expect that after the second DR, we will have the same dimensions: $d=dim\left(DR\right(DB1\left)\right)=dim\left(DR\right(DR\left(DB1\right)\left)\right)$.

In [5], Mirkes et al. presented proof that the Kaiser rule and the broken stick rule almost never satisfy this property (see pages 13–15). From the other side, PCA-CN IDs satisfy the described property.

4. Used Classifiers

We decided to use three robust classifiers with a minimal number of hyperparameters: KNN [85], Logistic Regression (LR) [86] and Fisher’s linear discriminant analysis (LDA) [87]. All these methods are robust with respect to small changes in the training set and are reproducible (each run produces the same result). KNN has one hyperparameter: the number of neighbours (k). LR and LDA also have one hyperparameter: the threshold to split two classes. We should also emphasise that we used Fisher’s linear discriminant analysis (the first model in [87]), which does not assume normality of distribution and equality of covariance matrices for two classes (these two assumptions are usual for linear discriminant analysis in most software packages). A description of hyperparameters selection is presented below.

4.1. Performance Estimation

There are many measures of the quality of the classifier, from usual accuracy to the F₁ score ([88] lists more than 20 such quality measures). Since some datasets have class imbalances (a very different fraction of cases for two classes), we decided to use balanced accuracy (BA ) as a quality measure:

$$\mathrm{BA}={\displaystyle \frac{TPR+TNR}{2}},$$

(3)

where $TPR$ is the fraction of correctly recognised cases of the positive class and $TNR$ is the fraction of correctly recognised cases of the negative class.

One of the very important elements of performance measures is the testing protocol. We decided to use 10 times 10-fold cross validation to calculate the BA. This means that we independently applied 10-fold cross validation 10 times. For each fold (there are 100 folds in total), we used the selected fold as a test set and all other data as a training set. For the test set, we calculated the BA. As a result, for each dataset and for each classifier, we have 100 BAs.

To ensure the comparability of results, we generated folds for each dataset, then used the same set of folds for all classifiers and for all datasets, whether original or with reduced dimensionality.

4.2. k Selection in KNN

There is no unique approach to defining an optimal k for KNN. There are some heuristics for k selection (see, for example, [89]), but the most useful and widely used method is estimation of k through empirical testing. The optimal value of k for each dataset was selected by calculating the classifier performance (BA) using the leave-one-out cross validation (LOOCV) technique for the original dataset. We tested all k values from 1 to $min\{Number\_of\_cases,500\}$. This is computationally expensive but results in a reliable and unbiased estimation of classifier performance. Selected values of k are presented in

Table 3. Unbiased estimation of k.

Dataset Name	Cases	Best k
Banknote authentication	1372	17
Blood	748	23
Breast Cancer	569	5
Climate Model Simulation Crashes	540	2
Cryotherapy	90	7
Diabetic Retinopathy Debrecen	1.151	143
EEG Eye State	14,980	3
HTRU2	17,898	3
Immunotherapy	90	2
ILPD (Indian Liver Patient Dataset)	579	2
Madelon	2600	161
MiniBooNE particle identification	130,064	15
Musk 1	476	5
Musk 2	6598	3
Planning Relax	182	1
QSAR biodegradation	1055	34
Skin Segmentation	245,257	5
Connectionist Bench (Sonar)	208	1
SPECT Heart	267	4
SPECTF Heart	267	2
MAGIC Gamma Telescope	19,020	4
Vertebral Column	310	66

. For each dataset, we selected k only once because optimal value of k is a property of the underlying (unknown) distribution of data and must not change with small changes in the data. As we can see from Table 3, there is no correlation between the optimal k and the number of cases (n) (Pearson’s correlation coefficient is −0.1). Figure 1 presents the BA as a function of k for the Banknote authentication and Maledon datasets. Graphs for all other datasets can be found in the ‘figures’ folder in [90]. As we can see, for some datasets, the optimal k is very small and the decay of the BA is observed with increasing k values (for example, in the Banknote authentication dataset), but for other datasets, the optimal k is large enough and the BA behaviour is different.

4.3. Decision Threshold Selection in LDA and LR

LR and LDA are both linear classifiers. Both these classifiers are defined by a linear direction for class discrimination. Then, all data points are projected in this direction, and the value of the projection is the score. For logistic regression, the calculated score is processed by the logit function to define the final score. There are special methods to define thresholds between classes in this one-dimensional problem: for LDA, this threshold can be calculated through the assumption of the multivariate normality of data distributions, and for LR, on the basis of the assumption that the final score is a probability. These methods are not applicable in the general case because data are not normally distributed, and output of LR is not a probability if not all LR assumptions are held. To avoid problems with testing assumptions, we applied an empirical search of the optimal threshold for LR and LDA. For this, we calculated a set of m unique scores ($s_i,i=1,\dots ,m$), then tested all thresholds in the middle of pairs of identified scores: $t_i=(s_i+s_{i+1})/2$. Now, we can define the optimal threshold as

$$tr=\arg \underset{i=1,\dots ,m}{max}acc\left(t_i\right),$$

(4)

where $acc\left(t_i\right)$ is the selected function of classifier quality, for example, accuracy, sensitivity, specificity, balanced accuracy, etc. In our study, we used balanced accuracy to define the optimal threshold.

More exactly, for each fold, we fit the model using the training set and calculated the scores of LR or LDA for the training set. Then, we searched for threshold using Equation (4) for the BA (3). Then, we used the calculated threshold to estimate the BA for the test set.

Figure 2 illustrates the selection of the best threshold for the Banknote authentication and Maledon datasets. Graphs for all other datasets can be found in the ‘figures’ folder in [90]. For illustration, we used LDA for a full dataset. We should emphasise that Figure 2 illustrates the process of threshold selection but does not present optimal threshold values because, in our experiments, for each fold, the individual threshold was searched for using the training set only. Figure 2 shows that for some datasets, we can expect good accuracy because of the minimal overlapping of classes (for example, in the Banknote authentication dataset), but for other datasets, we cannot expect good results because of massive class overlapping (for example, in the Maledon dataset).

5. ID Comparison

As stated in Section 4.1, for each ID and for each dataset, we have 100 BAs. There are many ways to compare such data. The simplest is a comparison of mean values. These results are presented in

Table A1. Average BAs and results of average ranking for KNN.

Dataset	Average Balanced Accuracy				Average Rank
	All attr.	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	0.99849	0.85792	0.85792	0.95848	2.5	2.5	1
Blood	0.63163	0.58495	0.58495	0.63046	2.5	2.5	1
Breast Cancer	0.95981	0.95491	0.92729	0.94956	1	3	2
Climate Model Simulation Crashes	0.67376	0.61583	0.51278	0.67376	2	3	1
Cryotherapy	0.91025	0.80575	0.78675	0.91025	2	3	1
Diabetic Retinopathy Debrecen	0.65401	0.62264	0.63371	0.64629	3	2	1
EEG Eye State	0.83697	0.65245	0.65245	0.72798	2.5	2.5	1
HTRU2	0.91506	0.88699	0.88699	0.90465	2.5	2.5	1
Immunotherapy	0.64482	0.58063	0.53571	0.64482	2	3	1
ILPD (Indian Liver Patient Dataset)	0.61713	0.63630	0.57547	0.61332	1	3	2
Madelon	0.60312	0.60942	0.63589	0.60423	2	1	3
MiniBooNE particle identification	0.87217	0.76305	0.57920	0.57920	1	2.5	2.5
Musk 1	0.89606	0.87110	0.85144	0.82472	1	2	3
Musk 2	0.92549	0.91910	0.91200	0.88173	1	2	3
Planning Relax	0.55860	0.50050	0.52582	0.56630	3	2	1
QSAR biodegradation	0.85499	0.84119	0.82214	0.84616	2	3	1
Skin Segmentation	0.99965	0.89500	0.89500	0.99005	2.5	2.5	1
Connectionist Bench (Sonar)	0.86777	0.85531	0.83795	0.86116	2	3	1
SPECT Heart	0.53831	0.58770	0.56106	0.58773	2	3	1
SPECTF Heart	0.68828	0.70349	0.66088	0.72737	2	3	1
MAGIC Gamma Telescope	0.81087	0.68549	0.54955	0.77014	2	3	1
Vertebral Column	0.80417	0.73002	0.66879	0.80417	2	3	1
Mean of rank					1.98	2.59	1.43

,

Table A2. Average BAs and results of average ranking for LR.

Dataset	Average Balanced Accuracy				Average Rank
	All attr.	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	0.99070	0.77577	0.77577	0.91934	2.5	2.5	1
Blood	0.69630	0.68823	0.68823	0.69630	2.5	2.5	1
Breast Cancer	0.95275	0.96683	0.94342	0.96802	2	3	1
Climate Model Simulation Crashes	0.88419	0.74890	0.52699	0.88409	2	3	1
Cryotherapy	0.88625	0.81500	0.78825	0.88625	2	3	1
Diabetic Retinopathy Debrecen	0.74235	0.60971	0.61088	0.67891	3	2	1
EEG Eye State	0.62448	0.54818	0.54818	0.57282	2.5	2.5	1
HTRU2	0.94256	0.90952	0.90952	0.92837	2.5	2.5	1
Immunotherapy	0.67598	0.62446	0.56482	0.67598	2	3	1
ILPD (Indian Liver Patient Dataset)	0.68838	0.69217	0.64976	0.68910	1	3	2
Madelon	0.55762	0.57112	0.61377	0.56131	2	1	3
MiniBooNE particle identification	0.88054	0.76747	0.58211	0.58211	1	2.5	2.5
Musk 1	0.83407	0.79060	0.72277	0.70543	1	2	3
Musk 2	0.91264	0.83786	0.79275	0.67075	1	2	3
Planning Relax	0.43547	0.45121	0.44633	0.43049	1	2	3
QSAR biodegradation	0.84809	0.82462	0.82130	0.83721	2	3	1
Skin Segmentation	0.95009	0.64193	0.64193	0.95109	2.5	2.5	1
Connectionist Bench (Sonar)	0.75921	0.77880	0.73740	0.78446	2	3	1
SPECT Heart	0.74587	0.76329	0.76345	0.73099	2	1	3
SPECTF Heart	0.70703	0.75859	0.77300	0.77517	3	2	1
MAGIC Gamma Telescope	0.76887	0.71919	0.57313	0.76557	2	3	1
Vertebral Column	0.83057	0.73129	0.73321	0.83010	3	2	1
Mean of rank					2.02	2.41	1.57

and

Table A3. Average BAs and results of average ranking for LDA.

Dataset	Average Balanced Accuracy				Average Rank
	All attr.	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	0.84293	0.75323	0.75323	0.81406	2.5	2.5	1
Blood	0.66590	0.65882	0.65882	0.66590	2.5	2.5	1
Breast Cancer	0.93096	0.92943	0.91489	0.93146	2	3	1
Climate Model Simulation Crashes	0.87415	0.74411	0.51142	0.87415	2	3	1
Cryotherapy	0.87825	0.82475	0.78825	0.87825	2	3	1
Diabetic Retinopathy Debrecen	0.58636	0.58359	0.58311	0.57856	1	2	3
EEG Eye State	0.56719	0.54081	0.54081	0.55229	2.5	2.5	1
HTRU2	0.91210	0.90952	0.90952	0.91084	2.5	2.5	1
Immunotherapy	0.67545	0.63366	0.44875	0.67545	2	3	1
ILPD (Indian Liver Patient Dataset)	0.66663	0.65460	0.50179	0.66736	2	3	1
Madelon	0.60154	0.60912	0.52196	0.60573	1	3	2
MiniBooNE particle identification	0.77407	0.72854	0.58211	0.58211	1	2.5	2.5
Musk 1	0.68760	0.68412	0.66491	0.62216	1	2	3
Musk 2	0.70488	0.70708	0.68934	0.65670	1	2	3
Planning Relax	0.41376	0.43394	0.48717	0.42159	2	1	3
QSAR biodegradation	0.80041	0.79526	0.78380	0.79890	2	3	1
Skin Segmentation	0.85056	0.50974	0.50974	0.84985	2.5	2.5	1
Connectionist Bench (Sonar)	0.71214	0.70907	0.68904	0.70473	1	3	2
SPECT Heart	0.77615	0.78049	0.77791	0.77905	1	3	2
SPECTF Heart	0.75318	0.75102	0.74335	0.74549	1	3	2
MAGIC Gamma Telescope	0.72460	0.68250	0.57313	0.72325	2	3	1
Vertebral Column	0.75579	0.73379	0.49586	0.75579	2	3	1
Mean of rank					1.75	2.64	1.61

. Our goal is a comparison of IDs and average values; this is not a good method of comparison, but we can apply a ranking of average values.

5.1. Average Ranking

Average ranking is ranking according to values of the average BA: 1 is the minimal value, and 3 is the maximal value. In cases in which several values coincided (ties), we applied the average method (rank is the average of ranks of all elements with the same value) [91]. Results of the average ranking are presented in Table A1, Table A2 and Table A3.

5.2. T-Test-Based Ranking

For one dataset, we can compare average BAs for two IDs: is the observed difference statistically significant? For this goal, we used a paired t-test [92]. In this case, we created the ranking in the following way: if two elements have statistically insignificantly different values, then they have the same rank. The basic assumptions of a paired t-test are listed as follows:

• The value should be continuous in nature. BAs are continuous.
• The mean values of samples should follow the normal distribution. This can be checked, but it is well known (see the Central Limit Theorem [92]) that, for a large sample from any distribution, the sample mean follows a normal distribution. We can conclude that for 100 observations, we have acceptable accuracy.
• Samples should be large and randomly selected from the population. In our case, we have a sample size of 100, which is not too large but not small. BA values are calculated using independently randomly generated splits.

One more question concerns the selection of the significance level. For 100 observations, we consider 95% to an acceptable significance. T-test-based ranks are presented in

Table A4. T-test and WSR test ranking for KNN.

Dataset	t-Test Based Rank			WSR Test Based Rank
	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	2.5	2.5	1	2.5	2.5	1
Blood	2.5	2.5	1	2.5	2.5	1
Breast Cancer	1	3	2	1	3	2
Climate Model Simulation Crashes	2	3	1	2	3	1
Cryotherapy	2.5	2.5	1	2.5	2.5	1
Diabetic Retinopathy Debrecen	3	2	1	3	2	1
EEG Eye State	2.5	2.5	1	2.5	2.5	1
HTRU2	2.5	2.5	1	2.5	2.5	1
Immunotherapy	2.5	2.5	1	2.5	2.5	1
ILPD (Indian Liver Patient Dataset)	1	3	2	1	3	2
Madelon	2	1	3	2	1	3
MiniBooNE particle identification	1	2.5	2.5	1	2.5	2.5
Musk 1	1	2	3	1	2	3
Musk 2	1	2	3	1	2	3
Planning Relax	2.5	2.5	1	3	2	1
QSAR biodegradation	2	3	1	2	3	1
Skin Segmentation	2.5	2.5	1	2.5	2.5	1
Connectionist Bench (Sonar)	1.5	3	1.5	1.5	3	1.5
SPECT Heart	1.5	3	1.5	1.5	3	1.5
SPECTF Heart	2	3	1	2	3	1
MAGIC Gamma Telescope	2	3	1	2	3	1
Vertebral Column	2	3	1	2	3	1
Mean of rank	1.95	2.57	1.48	1.98	2.55	1.48

,

Table A5. T-test and WSR test ranking for LR.

Dataset	t-Test Based Rank			WSR Test Based Rank
	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	2.5	2.5	1	2.5	2.5	1
Blood	2.5	2.5	1	2.5	2.5	1
Breast Cancer	1.5	3	1.5	1.5	3	1.5
Climate Model Simulation Crashes	2	3	1	2	3	1
Cryotherapy	2	3	1	2	3	1
Diabetic Retinopathy Debrecen	2.5	2.5	1	2.5	2.5	1
EEG Eye State	2.5	2.5	1	2.5	2.5	1
HTRU2	2.5	2.5	1	2.5	2.5	1
Immunotherapy	2	3	1	2	3	1
ILPD (Indian Liver Patient Dataset)	1.5	3	1.5	1.5	3	1.5
Madelon	2	1	3	2	1	3
MiniBooNE particle identification	1	2.5	2.5	1	2.5	2.5
Musk 1	1	2	3	1	2	3
Musk 2	1	2	3	1	2	3
Planning Relax	2	2	2	2	2	2
QSAR biodegradation	2.5	2.5	1	2.5	2.5	1
Skin Segmentation	2.5	2.5	1	2.5	2.5	1
Connectionist Bench (Sonar)	1.5	3	1.5	1.5	3	1.5
SPECT Heart	1.5	1.5	3	1.5	1.5	3
SPECTF Heart	3	1.5	1.5	3	1.5	1.5
MAGIC Gamma Telescope	2	3	1	2	3	1
Vertebral Column	2.5	2.5	1	2.5	2.5	1
Mean of rank	2.00	2.43	1.57	2.00	2.43	1.57

and

Table A6. T-test and WSR test ranking for LDA.

Dataset	t-Test Based Rank			WSR Test Based Rank
	PCA-K	PCA-BS	PCA-CN	PCA-K	PCA-BS	PCA-CN
Banknote authentication	2.5	2.5	1	2.5	2.5	1
Blood	2.5	2.5	1	2.5	2.5	1
Breast Cancer	1.5	3	1.5	1.5	3	1.5
Climate Model Simulation Crashes	2	3	1	2	3	1
Cryotherapy	2	3	1	2	3	1
Diabetic Retinopathy Debrecen	1.5	1.5	3	1.5	1.5	3
EEG Eye State	2.5	2.5	1	2.5	2.5	1
HTRU2	2.5	2.5	1	2.5	2.5	1
Immunotherapy	2	3	1	2	3	1
ILPD (Indian Liver Patient Dataset)	2	3	1	2	3	1
Madelon	1	3	2	1	3	2
MiniBooNE particle identification	1	2.5	2.5	1	2.5	2.5
Musk 1	1	2	3	1	2	3
Musk 2	1	2	3	1	2	3
Planning Relax	2.5	1	2.5	2.5	1	2.5
QSAR biodegradation	2	3	1	2	3	1
Skin Segmentation	2.5	2.5	1	2.5	2.5	1
Connectionist Bench (Sonar)	1.5	3	1.5	1.5	3	1.5
SPECT Heart	2	2	2	2	2	2
SPECTF Heart	2	2	2	2	2	2
MAGIC Gamma Telescope	2	3	1	2	3	1
Vertebral Column	2	3	1	2	3	1
Mean of rank	1.89	2.52	1.59	1.89	2.52	1.59

.

5.3. Wilcoxon Signed-Rank Test Ranking

The Wilcoxon signed-rank test is a non-parametric analogue of the t-test. We used a paired version of this test with a two-tailed alternative hypothesis [92]. In this case, we created the rank in the same way as in t-test-based ranking: if two elements have statistically insignificantly different values, then they have the same rank. We used the same significance level of 95%. WSR test-based ranks are presented in Table A4, Table A5 and Table A6.

5.4. Final ID Comparison

Table A1, Table A2, Table A3, Table A4, Table A5 and Table A6 present three ranks of IDs for 22 datasets and three classifiers. We need to compare IDs “in general”. There are several methods for such a comparison. The simplest is the calculation of the mean value of ranks for each dimension and each classifier. We can also apply a t-test to test the significance of differences in the mean of ranks. In this case, we use a 95% significance level because we have only 22 observations.

6. Results

During the described testing protocol, it was found that for some datasets, the formation of models faced the multicollinearity problem [93] for logistic regression (QSAR biodegradation, MiniBooNE particle identification, Musk 2, and SPECTF Heart datasets). In our version of LDA (see the fisherDir function in forPaper.py in [90]), we used Tikhonov regularisation [94] to avoid this problem. We did not use any approach to resolve this problem that could influence to result, as our goal was to compare IDs. The multicollinearity problem was observed only for the original dataset and supports the idea that DR is necessary for some data.

The average balanced accuracies for all datasets and for all IDs are presented in Table A1, Table A2 and Table A3. Average rankings are presented in Table A1, Table A2 and Table A3. T-test ranking and Wilcoxon signed-rank test ranking are presented in Table A4, Table A5 and Table A6. Average ranks for all three rankings are presented in the bottom rows of Table A1, Table A2, Table A3, Table A4, Table A5 and Table A6.

In Table A1, Table A2, Table A3, Table A4, Table A5 and Table A6, we can see that average, t-test-based, and WSR-based rankings are consistent: for all classifiers, we have almost the same mean value of IDs (see

Table 4. Values of mean ranks for all classifiers.

Classifier	Ranking	Mean of Ranks			p-Value of t-Test for Differences
		PCA-K	PCA-BS	PCA-CN	PCA-K/PCA-BS	PCA-BS/PCA-CN	PCA-K/PCA-CN
KNN	Average	1.98	2.59	1.43	0.003402	0.000113	0.060596
	T-test	1.95	2.57	1.48	0.003006	0.000134	0.098642
	WSR test	1.98	2.55	1.48	0.008428	0.000163	0.089742
LR	Average	2.02	2.41	1.57	0.063491	0.008455	0.150906
	T-test	2.00	2.43	1.57	0.022280	0.003817	0.124516
	WSR test	2.00	2.43	1.57	0.022280	0.003817	0.124516
LDA	Average	1.75	2.64	1.61	0.000030	0.000762	0.639755
	T-test	1.89	2.52	1.59	0.001097	0.001975	0.257972
	WSR test	1.89	2.52	1.59	0.001097	0.001975	0.257972

). As we can see, exact values of mean ranks are different, but for all methods of ranking and for all classifiers, the best ID (minimal mean rank) is associated with PCA-CN, and the worst ID (maximal mean rank) is associated with PCA-BS.

The results of the test of significance of differences in mean ranking are presented in Table 4. We can see that for KNN and LDA for all three rankings and for LR for the t-test and WSR test rankings, PCA-K is statistically significantly better than PCA-BS. For LR and the average ranking, this difference is statistically insignificant.

For all three rankings and for all three classifiers, PCA-CN is statistically significantly better than PCA-BS.

Differences between PCA-CN and PCA-K are statistically insignificant for all rankings and all classifiers, but the mean rank of PCA-CN is less than the mean rank of PCA-K for all rankings and all classifiers.

These observations correlate with findings reported in Section 3.4: PCA-BS ID is statistically significantly different from PCA-K and PCA-CN.

7. Conclusions

After extensive testing of the balanced accuracy of three classifiers (KNN, LR, and LDA), for three PCA-based IDs, we found that there is no universal answer with respect to whether one of the IDs is better for all data (see

Table 5. The number of datasets for which the specified ID provides the best BA.

Classifier	PCA-K	PCA-BS	PCA-CN
KNN	5	1	16
LR	5	2	15
LDA	8	1	13

). For each classifier, PCA-BS was the best ID for one or two datasets.

However, on average, PCA-CN and PCA-K are statistically significantly better than PCA-BS (see Table 4). This is interesting, but in all cases, the mean rank of all three of the considered rankings for PCA-K is greater than for PCA-CN, although in all cases, this difference is statistically insignificant.

As a result, we can recommend not using the broken stick rule to define the ID of data. To choose the preferable ID among PCA-K and PCA-CN, we should take into account one more property of these dimensions. As we described in Section 3.4, PCA-K does not have recurrent stability. If we applied PCA-K twice, then after the second DR, we would have a dimension off less than that after the first DR. From the other side, PCA-CN is recurrently stable; if you applied PCA-CN, then after an arbitrary number of further applications, the dimension would not change (see

Table 6. Dimensions after recurrent usage of three IDs for the Musk 2 dataset.

Iteration	PCA-K	PCA-BS	PCA-CN
0	166	166	166
1	25	13	6
2	4	2	6
3	2	0	6
4	1	0	6

). As we can see, the only linear DR method that corresponds to the intuitive meaning of the intrinsic dimension is PCA-CN. This method is the best according to mean ranks, even if this is not statistically significant.

We can recommend the usage of PCA-CN for automatic data pre-processing for classification problems because this method is the only recurrently stable linear method and, on average, is the best one.