Training Data Augmentation with Data Distilled by Principal Component Analysis

Abstract

This work develops a new method for vector data augmentation. The proposed method applies principal component analysis (PCA), determines the eigenvectors of a set of training vectors for a machine learning (ML) method and uses them to generate the distilled vectors. The training and PCA-distilled vectors have the same dimension. The user chooses the number of vectors to be distilled and augmented to the set of training vectors. A statistical approach determines the lowest number of vectors to be distilled such that when augmented to the original vectors, the extended set trains an ML classifier to achieve a required accuracy. Hence, the novelty of this study is the distillation of vectors with the PCA method and their use to augment the original set of vectors. The advantage that comes from the novelty is that it increases the statistics of ML classifiers. To validate the advantage, we conducted experiments with four public databases and applied four classifiers: a neural network, logistic regression and support vector machine with linear and polynomial kernels. For the purpose of augmentation, we conducted several distillations, including nested distillation (double distillation). The latter notion means that new vectors were distilled from already distilled vectors. We trained the classifiers with three sets of vectors: the original vectors, original vectors augmented with vectors distilled by PCA and original vectors augmented with distilled PCA vectors and double distilled by PCA vectors. The experimental results are presented in the paper, and they confirm the advantage of the PCA-distilled vectors increasing the classification statistics of ML methods if the distilled vectors augment the original training vectors.

1. Introduction

One fundamental problem in the field of data classification with supervised machine learning (ML) is a shortage of or imbalanced training data [1,2,3]. For example, the training set of the public image database ISIC2020 [4] contains only 584 malignant images and approximately 33,126 benign images. Such a large class imbalance leads to overfitting [5], which impacts the prediction accuracy of a classifier. This holds because the model is biased toward the class with the larger set of samples. In the example given above, the ML classifier may produce a large number of false negatives (type I error), which will indicate poor performance of the model.

A natural way to tackle the problem of imbalanced classes and data shortage is to fill up the gap with additional data, but in many practical cases, the process of obtaining additional real data is time-consuming and expensive. Hence, other sources of and approaches to data generation should be explored and used. One approach is to artificially produce data which will have the same characteristics as the original data, and after augmenting the training data with the artificial data, the entire set will increase the statistical outcomes of the ML method. An approach that produces artificial data is to apply geometric transformations on a certain part of the training images and augment the original training set with the transformed set of images. Typical geometric image transformations consist of rotation, translation, flipping [6] or cropping [7]. Another set of techniques that generate images for the purpose of training data augmentation includes Gaussian blur, image sharpening, smoothing [8], and noise injection [6].

In recent years, a method was developed to embed a vector field (VF) into an image [9]. This method augments the set of image features with VF features like singular points and trajectories. Augmenting the set of features in every image of an image database increases the learning capabilities of the ML methods and leads to an increase in its classification abilities. One may observe that the method listed above produce new data on the basis of existing data. Also, the VF embedding methods [9] augment the set of image features with VF features, but none of them change the main theme of (keeping the subjects that belong to) the images in the case of image data.

On the other hand, data distilation methods based on NNs [10] and infinite NNs [11] generate new data or images which do not resemble the original sets at all but are still useful for training data augmentation. Another useful way to generate new images on the basis of existence, for the purpose of training data augmentation, is to use a generative adversarial network (GAN). A review of this type of NN is given in [12]. In [13], the authors used the GAN architecture to generate synthetic images and resolve the problem of limited data availability for three different classes. For the purpose of augmentation, the authors of [14] developed and implemented a CNN to determine the likelihood of an object category being present inside a box that encompasses a given neighborhood. Further, the method found suitable locations in images to place new objects.

A comprehensive survey on the recent methods of data distillation, including the application of NNs, is given in [15]. The authors provide detailed descriptions of multiple data distillation techniques based on the concepts of “meta-model”, “gradient”, “distribution”, “trajectory” and “factorization” matching. Additional papers reviewing state-of-the-art augmentation techniques are available [16,17,18].

In [19], the principal component analysis (PCA) [20] method was used to distill the photo response non-uniformity reference by removing the interference noise. In the present paper, we propose a novel idea to extend the well-known PCA [21] method for vector data distillation from existing training data. Our scientific conjecture, regarding an extension of PCA that may lead to enhancement of the classification statistics of the ML methods, is that the PCA method lies in the eigenvectors of the data feature correlation matrix. Also, this matrix maps every eigenvector to a vector on the same straight line. This implies that we may receive (distill) different vectors which possess feature correlations similar to the original vectors. To validate that extended PCA distills vectors whose augmentation, relative to the original training vectors, increases the ML classification statistics, we conducted a set of experiments with four publicly available databases. We applied on them three classifiers: an NN, a support vector machine (SVM) and logistic regression. The experimental results are shown in this paper, and they confirm the advantage that augmenting the training set with PCA-distilled vectors leads to an increase in the classification statistics.

The rest of this paper is organized as follows. In its first subsection, Section 2 describes the new method for data distillation with PCA, while the second subsection introduces a statistical method [22] that determines the lower bound of the number of distilled vectors necessary to augment the set of training vectors in order to provide certain accuracy of classification. Section 3 presents the four classifiers implemented to validate the new distillation method, and the next section describes the original datasets along with the augmented vector datasets. Finally, Section 5 and Section 6 describe our results and discuss the novelties and advantages of the new vector distillation method.

2. Vector Distilation with Principal Component Analysis (PCA)

PCA [20,21] is a commonly employed technique for decreasing the dimensionality of given data. This is achieved through mapping a high-dimensional dataset into a lower dimensional space while preserving the important features of the dataset. PCA has numerous applications to data visualization, feature extraction and data compression. In the present paper, we propose a new application of PCA and employ the method to generate new data ordered from top to bottom according to their importance. To augment the original data, we select the most representative vectors from the top. We apply a statistical method to estimate the minimum number of vectors to be selected.

2.1. Principal Components and the Projection Matrix

Assume we are given a set of training vectors ${({\mathbf{v}}_{\mathbf{1}},{\mathbf{v}}_{\mathbf{2}},\dots ,{\mathbf{v}}_{\mathbf{n}})}^T={\mathbf{D}}_{\mathbf{t}}$, where ${\mathbf{v}}_{\mathbf{i}}\in {\mathbf{R}}^{\mathbf{k}}$. Hence, every training vector ${\mathbf{v}}_{\mathbf{i}}$ consists of k variables, and we will also call features $x_{ij}\in R$ for $j=1,\dots ,k$ and $i=1,\dots ,n$. Therefore, the matrix of the database ${\mathbf{D}}_{\mathbf{t}}\in R^{n\times k}$. Then, we denote the vector of the jth feature with ${\mathbf{f}}_{\mathbf{j}}\in {\mathbf{R}}^{\mathbf{n}}$ and introduce it as ${\mathbf{f}}_{\mathbf{j}}={(x_{1j},\dots ,x_{nj})}^T$. Furthermore, we denote the mean of the jth feature (variable) vector ${\mathbf{f}}_{\mathbf{j}}$, which is a column of ${\mathbf{D}}_{\mathbf{t}}$, with ${\mu }_j$ and the rank of ${\mathbf{D}}_{\mathbf{t}}$ with $r\le min\{n,k\}$:

1.

Data normalization is the first step of the calculation, and we subtract the mean of each feature vector from the corresponding vector entries. This is performed to ensure that the data are centered around zero. This means that the mean of every feature vector is zero. The equation for data normalization, with the help of the feature vector mean, is

$$x_{ij}\leftarrow x_{ij}-{\mu }_j,$$

(1)

where $x_{ij}$ is the i-th feature value, in which $i=1,\dots ,n$ for the j-th feature (variable) vector and $j=1,\dots ,k$, while ${\mu }_j$ is the mean of this feature vector.

2.

The covariance matrixmeasures the statistical relationship between the features in the entire dataset [21]. For this purpose, we developed the equation below and calculated the $F_{pj}$ entry of the features’ covariance matrix of the training data matrix ${\mathbf{D}}_{\mathbf{t}}$:

$$F_{pj}=\frac{1}{n-1}\sum _{i=1}^n(x_{ij}-{\mu }_j){(x_{ip}-{\mu }_p)}^T.$$

(2)

It follows from Equation (2) that the covariance matrix of the ${\mathbf{D}}_{\mathbf{t}}$ features is of the form

$$\mathbf{F}=\left\{F_{pj}\right\},p=1,\dots ,k,j=1,\dots ,k.$$

(3)

Now, we calculate the eigenvectors ${\mathbf{u}}_{\mathbf{j}}$ and the eigenvalues ${\lambda }_i$ of the covariance matrix $\mathbf{F}$ of the database features as follows:

$$\mathbf{F}{\mathbf{u}}_{\mathbf{j}}={\lambda }_j{\mathbf{u}}_{\mathbf{j}},|\mathbf{F}-\lambda \mathbf{I}|=0,(\mathbf{F}-\lambda \mathbf{I}){\mathbf{u}}_{\mathbf{j}}=\mathbf{0},$$

(4)

where we denote the $k\times k$ identity with $\mathbf{I}$.

3.

For projection, in the present step, we construct the matrix $\mathbf{U}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{r}}$, whose columns are the normalized eigenvectors of the covariance matrix $\mathbf{F}$ of the dataset ${\mathbf{D}}_{\mathbf{t}}$ features:

$$\mathbf{U}=[{\mathbf{u}}_{\mathbf{1}},{\mathbf{u}}_{\mathbf{2}},\dots ,{\mathbf{u}}_{\mathbf{r}}],$$

(5)

where every eigenvector ${\mathbf{u}}_{\mathbf{j}}\in {\mathbf{R}}^{\mathbf{k}}$ and r is the rank of ${\mathbf{D}}_{\mathbf{t}}$. Also, let us arrange the vector columns ${\mathbf{u}}_{\mathbf{j}}$ from left to right according to the decreasing eigenvalues of the corresponding eigenvectors. We consider the largest one of them at the leftmost position if we think of the eigenvalues as a decreasing sequence of consecutive numbers. The matrix $\mathbf{U}$ is applied to project the training dataset ${\mathbf{D}}_{\mathbf{t}}$ onto the new subspace spanned by the eigenvectors $u_i,i=1,\dots ,r$. The equation that conducts the projection is [21]

$${\mathbf{y}}_{\mathbf{i}}={\left({\mathbf{U}}^T{\mathbf{x}}_{\mathbf{i}}\right)}^T={\mathbf{x}}_{\mathbf{i}}^T\mathbf{U},$$

(6)

where ${\mathbf{y}}_{\mathbf{i}}\in {\mathbf{R}}^{\mathbf{r}}$ is the new low-dimensional representation of the ith training sample ${\mathbf{x}}_{\mathbf{i}}\in {\mathbf{R}}^{\mathbf{k}}$, in which ${\mathbf{x}}_{\mathbf{i}}$ is a vector row in ${\mathbf{D}}_{\mathbf{t}}$ whose rank $r\le \mathbf{min}\{k,n\}$.

We consider that an eigenvector is stronger than another one if the former has a greater eigenvalue than the latter. Hence, we selected from ${\mathbf{U}}^{\mathbf{T}}$ the $m<r$ strongest components (eigenvectors) ${\mathbf{u}}_{\mathbf{j}}$, which appeared as the top m rows of the matrix. Using the strongest components as vector columns, we constructed the matrix $\mathbf{T}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{m}}$ which transformed (mapped) the original dataset ${\mathbf{D}}_{\mathbf{t}}$ into ${\mathcal{D}}_{\mathbf{t}}\in {\mathbf{R}}^{\mathbf{n}\times \mathbf{m}}$, which we call the transformed data:

$${\mathbf{D}}_{\mathbf{t}}\mathbf{T}={\mathcal{D}}_{\mathbf{t}}.$$

(7)

2.2. Finding the Distilled Vectors

We conducted the inverse projection with the matrix ${\mathbf{T}}^{\mathbf{T}}\in {\mathbf{R}}^{\mathbf{m}\times \mathbf{k}}$, which mapped the transformed data ${\mathcal{D}}_{\mathbf{t}}$, where

$${\mathcal{D}}_{\mathbf{t}}{\mathbf{T}}^{\mathbf{T}}={\Delta }_{\mathbf{t}},$$

(8)

into the modified original dataset ${\mathbf{\Delta }}_{\mathbf{t}}\in {\mathbf{R}}^{\mathbf{n}\times \mathbf{k}}$ such that ${\mathbf{\Delta }}_{\mathbf{t}}\ne D_t$. Note that the modified original data belong to the original $kD$ feature space and have the same cardinality n as the initial original data. Now, among the modified training data set ${\mathbf{\Delta }}_{\mathbf{t}}$, we determine the images of the m strongest components (eigenvectors) we selected from ${\mathbf{U}}^{\mathbf{T}}$. In order to accomplish this, we used the consecutive numbers of their positions in Equation (4). We call the set of m modified vectors in ${\mathbf{\Delta }}_{\mathbf{t}}$ the set of distillated vectors from the original training vectors ${\mathbf{D}}_{\mathbf{t}}$. We augmented the set of initial original training vectors with the set of distilled vectors and used the union as a training set of an ML classifier. One immediate advantage of this kind of augmentation is enlarging a class in case of a massive disproportion between two classes.

2.3. Minimum Number of Training Samples (MNTS)

The present subsection provides an algorithm to determine the minimum number of training samples [22] m that need to be distilled out of the total number of training samples n such that the original samples augmented with the distilled samples train a classifier to provide an accuracy a:

1.

Assume we are looking for an accuracy $a\in (0,1]$. It follows that $\alpha =1-a$ and the confidence level $C=1-\alpha /2$;

2.

Using the confidence level C, we determine $Z_{\alpha /2}$ from the standard normal distribution table [23];

3.

We select the success and failure rates, denote them as p and q, respectively, and calculate m from

$$\frac{\alpha }{\sqrt{\frac{pq}{m}}}\ge Z_{\frac{\alpha }{2}}$$

(9)

4.

We consider a training set with two classes (the number of benign samples $\left|B\right|$ and the number of malignant samples $\left|M\right|$), where $\left|B\right|+\left|M\right|=\left|S\right|$, which is the total number of samples;

5.

We calculate $\frac{\left|M\right|}{\left|S\right|}=c_1$, $\frac{\left|B\right|}{\left|S\right|}=c_2$;

6.

Here, $m_M$ and $m_B$ are the number of malignant and number of benign distilled vectors, respectively, while $m_M=n·c_1$, $m_B=n·c_2$ such that $m_M+m_B=m$.

3. Classifiers Applied for Experimental Validation

In the present section, we describe three classifiers we use to validate our method for distillation and augmentation and the advantage that comes from them. The novelty is that by using the training vectors of an ML algorithm, the PCA method generates (distills) vectors which increase the classification statistics of the ML method if the original training set is augmented with the distilled set.

3.1. Logistic Regression (LR)

Logistic regression is a statistical technique employed to model the relation between dependent variables. In the case of binary classification, it models the probability of an observation belonging to a particular class. Given a dataset with n observations and k independent variables where ${\mathbf{f}}_{\mathbf{j}}\in R^n$, the logistic regression model can be represented as follows:

$$logit(P(Y=1|{\mathbf{D}}_{\mathbf{t}})={\beta }_0+{\beta }_1{\mathbf{f}}_{\mathbf{1}}+{\beta }_2{\mathbf{f}}_{\mathbf{2}}+\dots +{\beta }_k{\mathbf{f}}_{\mathbf{k}},$$

(10)

where Y is the binary dependent variable with outcomes “0” or “1”, ${\mathbf{D}}_{\mathbf{t}}$ is the training data matrix with n rows (samples of training vectors) and k columns (variables), ${\beta }_0,{\beta }_1,\dots ,{\beta }_k$ are the coefficients or parameters estimated by the model and $logit(P(Y=1|{\mathbf{D}}_{\mathbf{t}})$ is the natural logarithm of the odds ratio of the probability of $Y=1$ to the probability of $Y=0$. The logistic function or sigmoid function is used to convert the linear combination of independent variables ${\mathbf{f}}_{\mathbf{j}}$ and coefficients into a probability between 0 and 1:

$$P(Y=1|X)=\frac{1}{1+e^{-{\beta }_0-{\beta }_1{\mathbf{f}}_{\mathbf{1}}-{\beta }_2{\mathbf{f}}_{\mathbf{2}}-\dots -{\beta }_k}}.$$

(11)

To evaluate the coefficients of the model, the maximum likelihood estimation algorithm [24] was used. The goal was to maximize the likelihood of observing the dataset if the model was given. This could be accomplished by minimizing the negative log-likelihood:

$$L({\beta }_0,{\beta }_1,\dots ,{\beta }_p)=-\sum _{i=1}^n[y_{i}log\left(p_i\right)+(1-y_i)log(1-p_i)].$$

(12)

where $y_i$ is the ground truth label for the training vector ${\mathbf{v}}_{\mathbf{i}}$, while $p_i$ is the predicted probability of $Y=1$ for this vector. The parameters ${\beta }_j$ for $j=1,\dots ,k$ can be estimated using optimization algorithms such as gradient descent [25] or Newton’s method.

3.2. Support Vector Machine (SVM)

The SVM is a useful ML tool for vector classification [26,27]. The method was developed and gained popularity during the 1990s in the 20th century [26] and is applied mainly for binary classification, but multi-class SVMs could be built as well. Unlike NNs, which provide accurate classification if thousands or hundreds of thousands of training samples are used, SVMs are effective classifiers if hundreds of training samples are available. In the present study, we employ a binary SVM to classify medical vector data. For the purpose of SVM training, we use two kinds of data: original data and original data augmented with PCA-distilled data.

Following the concepts from [26,27], we define the binary SVM as the function given below:

$$f\left(\mathbf{s}\right)=\mathrm{sign}\left(\sum _i{\alpha }_i{l}_{i}K({\mathbf{s}}_{\mathbf{i}},\mathbf{s})+\mathbf{b}\right).$$

(13)

In Equation (13), the symbol $l_i$ denotes the label of the ith training sample ${\mathbf{s}}_{\mathbf{i}}$, $\mathbf{s}$ is the unknown sample subject to classification and ${\alpha }_i$ is the Lagrange multiplier associated with the ith support vector such that ${\sum }_i{\alpha }_i{l}_i=0$, while $\mathbf{b}={\sum }_i{\alpha }_i$ is a bias. Furthermore, with $K({\mathbf{s}}_{\mathbf{i}},\mathbf{s})$, we denote the kernel of the SVM. Examples of the most used kernels are the linear, polynomial, radial basis function (RBF), Gaussian and sigmoid kernels. For the purpose of our experimental validation of the advantage of this study, we will use the linear and the polynomial kernels. The latter is defined below:

$$K({\mathbf{s}}_{\mathbf{i}},\mathbf{s})={(1+{\mathbf{s}}_{\mathbf{i}}^T\mathbf{s})}^p,$$

(14)

where $p\in Z$. Hence, we difine an SVM with a linear (SVML) kernel for $p=1$ and an SVM with a polynomial (SVMP) kernel for $p=2$.

3.3. Neural Network (NN)

In recent years, the NN became the most used classifier because of its reliability and repeatability as well as the opportunity to greatly increase performance statistics. Note that the image is a 2D signal and can be presented in the form of a 1D signal or as a sequence. Despite this, we could roughly divide NNs into those which are designed to work with images [6,8] and those designed to work with signals and sequences of numbers [28,29]. A well-established representative of the first kind is the convolutional NN (CNN) [6,8], while examples of the second kind include long short-term memory (LSTM) NNs [28,29]. The main hyperparameters of an NN consist of the architecture, input layers, hidden layers, output layers, activation functions, loss function, optimization method and training process. The architecture of an NN typically consists of multiple interconnected layers, each containing a set of neurons or nodes. The arrangement and number of layers, as well as the number of neurons in each layer, vary depending on the specific problem and desired model complexity [8,12,28,29]. The NN model we use consists of two hidden layers and an output layer. Each hidden layer has 512 and 256 neurons and uses the ReLU activation function. The output layer has one neuron with the sigmoid activation function. The equations used in the two hidden layers are described as follows:

$${\mathbf{Z}}_{\mathbf{1}}={\mathbf{W}}_{\mathbf{1}}{\mathbf{\Delta }}_{\mathbf{t}}+{\mathbf{B}}_{\mathbf{1}};{\mathbf{A}}_{\mathbf{1}}=max(\mathbf{0},{\mathbf{Z}}_{\mathbf{1}}),$$

(15)

$${\mathbf{Z}}_{\mathbf{2}}={\mathbf{W}}_{\mathbf{2}}{\mathbf{A}}_{\mathbf{1}}+{\mathbf{B}}_{\mathbf{2}};{\mathbf{A}}_{\mathbf{2}}=max(\mathbf{0},{\mathbf{Z}}_{\mathbf{2}}).$$

(16)

Here, $W_1$ and $W_2$ represent the matrices of weights for the first and second hidden layer, respectively, and ${\mathbf{\Delta }}_{\mathbf{t}}$ denotes the matrix of the input data. The terms ${\mathbf{B}}_{\mathbf{1}}$ and ${\mathbf{B}}_{\mathbf{2}}$ represent matrices of the bias terms for the first and second hidden layers, respectively, while ${\mathbf{Z}}_{\mathbf{1}}$ and ${\mathbf{Z}}_{\mathbf{2}}$ denote the pre-activation calculations of the two hidden layers. Furthermore, $A_1$ and $A_2$ represent the ReLU-calculated outputs of the first and the second layers, respectively.

Concerning the output layer that calculates the vector of predicted values, we implement the following equations:

$${\mathbf{z}}_{\mathbf{3}}={{\mathbf{w}}_{\mathbf{3}}}^{\mathbf{T}}{\mathbf{A}}_{\mathbf{2}}+{\mathbf{b}}_{\mathbf{3}};{\mathbf{A}}_{\mathbf{3}}=\sigma ({\mathbf{z}}_{\mathbf{3}}.)$$

(17)

In Equation (17), ${\mathbf{w}}_{\mathbf{3}}$ represents the vector of weights for the output layer, ${\mathbf{b}}_{\mathbf{3}}$ denotes the vector of biases, ${\mathbf{z}}_{\mathbf{3}}$ represents the pre-activation output of the output layer and ${\mathbf{A}}_{\mathbf{3}}$ represents the vector of the predicted output calculated by the sigmoid activation function $\sigma$.

During the training process, we apply the Adam optimizer to update the matrix and vector parameters ${\mathbf{W}}_{\mathbf{1}}$, ${\mathbf{W}}_{\mathbf{2}}$, ${\mathbf{w}}_{\mathbf{3}}$, ${\mathbf{B}}_{\mathbf{1}}$, ${\mathbf{B}}_{\mathbf{2}}$ and ${\mathbf{b}}_{\mathbf{3}}$. The loss function used is binary cross-entropy, and the model’s accuracy is measured using the accuracy metric.

4. Validation Datasets and Training of the Models

4.1. Original Datasets

In the present section, we describe the four datasets of feature vectors we used to validate that augmenting the training dataset with PCA-distilled vectors improves the statistical outcomes of an ML classifier:

• The first one we call the skin lesion (SL) dataset, which comprises 162 observations (samples S) of 5D feature vectors ($B,C,D,A_B^M,A_C^M$) extracted by an active contour in [30] from skin lesion images with a ground truth [31]. The SL dataset contains 100 benign samples (skin lesion feature vectors labeled B) and 62 malignant observations (labeled M).
• The diabetes (D) dataset [32] consists of 768 feature vectors with a dimension of 8 ($8D$). The vectors are distributed in two sets labeled P if the corresponding vector indicates diabetes. Otherwise, the vector is labeled N, which means that the vector indicates no diabetes. Class P, which suggests diabetes, consists of 268 vectors, while class N contains 500 vectors with a dimension of 8.
• The heart disease (HD) dataset [33] consists of 1025 feature vectors with a dimension of 13. The data samples are distributed in two classes: $0=negative$ (N) and $1=positive$ (P). The N samples in the HD datasets total 499, while the P samples total 526.
• The breast cancer (BC) dataset [34] contains total of 569 feature vectors (observations) with a dimension of 30. Out of them, 347 are labeled as benign B, while the remaining 212 were determined by medical experts to be malignant M.

A read of the above databases’ information shows that all databases are binary. In three of the cases—$SL$, D and $BC$—the two classes are unbalanced. The benign (negative class) contains almost twice as many feature vectors as the malignant (positive) class.

4.2. Augmentation of the Original Training Data

For the validating experiments, we used the following split of the original data: 90% for training and 10% for testing. Then, we distilled the vectors from the selected training set and augmented this set with the distilled vectors. Hence, we constructed three kinds of augmentations. For the purpose of distillation, we applied the method from Section 2.3 to determine the minimum number of training samples (MNTS) to be distilled and used this to augment the original training samples such that they trained a classifier to provide an accuracy of $90\%$.

Table 1. Comparison of accuracy in classification of the SL data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML
Original	79%	71%	88%
Augmented Data 1	75%	76%	76%
Augmented Data 2	78%	82%	82%

presents the experimental results of classifying the 10% testing data with the three models introduced above and trained by the augmented data, as well as the same models trained by the original data.

Augmented Dataset 1 was created from the SL dataset using a testing/training split of 10%/90%, where we kept the proportion $M/B=0.62$ the same as it was in the original SL dataset. Therefore, we had 90 B and 56 M vectors for training as well as 10 B and 6 M for testing. Next, using PCA, we distilled the 26 vectors from the selected 90% (90 B and 56 M) training original vectors such that the proportion $M/B=0.62$ was preserved. The cardinality of the distilled vectors was 26 and was determined by the MNST approach, consisting of 16 B and 10 M. Next, we added the distilled vectors to the original 90 B and 56 M training samples. Then, we trained our classifiers to learn from the augmented training set and tested the models on the remaining 10% B and 10% M original vectors.

Augmented Dataset 2 was for the SL dataset as well. In this augmentation, we randomly selected 90% of the entire original data for training and used the remaining 10% of original data for testing. This means that in the selected training vectors, the proportion $M/B$ did not need to be $M/B=0.62$ as in the entire original SL dataset. Furthermore, in the distilled set of 26 vectors, the ratio $M/B$ equals the ratio of the randomly selected training vectors.

Augmented Dataset 3 analogous to the case of the SL dataset, used 90% randomly selected samples for training from each class, and we distilled from the training samples, 101, 151 and 201 new vectors for each of the datasets D, HD and BC, respectively. Note that in the distilled sets, the proportion M/B or P/N was the same as in the original dataset. Then, for each of these datasets, we created augmented training sets of the form Original + 101-PCA-d, Original + 151-PCA-d and Original + 201-PCA-d:. With Original, we denoted the randomly selected 90% original vector samples, which consisted of 90% of the M (P) and 90% of the B (N) samples. The sets of 101, 151 and 201 were distilled from the 90% original vectors with the help of the PCA method described in Section 2. Therefore, for each of the original datasets D, HD and BC, we developed three augmented datasets for training. Then, we trained the models and tested them on the remaining 10% of the original data.

Augmented Dataset 4 involved distilling 51, 76 and 101 samples from the 101, 151 and 201 samples, respectively, which were distilled earlier with the help of the same PCA method described in Section 2. In the double distilled three sets of vectors, the proportion M/B or P/N was kept the same as in the initially distilled dataset. With the help of these new and nested distillations, we created three augmented training sets of the form Original + (101 + 51)-PCA-d, Original + (151 + 76)-PCA-d and Original + (201 + 101)-PCA-d for each of the original datasets D, HD, and BC. Therefore, we developed nine training augmented datasets and tested them on the remaining 10% of the original data.

4.3. Model Training

Since the SL data were quite small, we designed a slightly different NN than the one described in Section 3.3. The modified NN was designed with the Keras Sequential API and contained two fully connected layers. The first layer comprised 10 neurons and implemented the ReLU activation function, while the second layer consisted of 1 neuron with the sigmoid activation function. The model used the Adam optimizer for 100 epochs with a batch size of 32 in order to be trained with the rather small sample sets of 100 and 106 samples.

As for the classification of the other three datasets, we applied the neural network presented in Section 3.3. The model also included two dropout layers with a rate of 0.5 to reduce overfitting. The input dimension was set to eight. The model was compiled with the Adam optimizer and used the binary cross-entropy loss function.

Recall that the advantage of this study is that augmenting the training data with PCA-distilled vectors increases the classification statistics. The next model we used to validate this advantage was the logistic regression model described in Section 3.1. We defined this classifier again with the scikit-learn library as a logistic regression object. The model used a maximum of 1000 iterations to ensure convergence.

The last model we implemented to validate our approach was the support vector machine (SVM). We designed this classifier with the help of the scikit-learn library and made the SVM works with two kernels: linear and polynomial degree 2. Furthermore, we set the machine’s regularization strength C = 1.

For the NN model, the accuracy score was calculated using the evaluate() method with the testing data. For the logistic regression and SVM models, the accuracy score was calculated using the score() method with the testing data. The confusion matrix was also calculated for each model using the confusion_matrix() function from the scikit-learn library.

4.4. Cross-Validation

To fairly validate the classification statistics of the three classifiers, we implemented the Monte Carlo cross-validation approach. It randomly split the entire set of samples into training and test samples and repeated the splitting m times. For each split, a sample appeared in exactly one of the sets of training or testing. The mean of the m experiments was calculated for each evaluation statistic. In our experiments, we set m = 10 and show the results in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11, Table 12 and Table 13 for the SL and HD datasets. The advantages of applying Monte Carlo cross-validation come from the fact that the approach decreases the variance of the split sample error estimate, and the proportion of the training-test random splits do not depend on the selected number m.

4.5. Classification Metrics

Accuracy, sensitivity and specificity are commonly used metrics for evaluating the effectiveness of a classification model. Below is a description of each metric along with their equations. The notations used are TP = true positive, TN = true negative, FP = false positive and FN = false negative:

• Accuracy measures the overall correctness of the model’s predictions by calculating the ratio of correctly predicted instances to the total number of instances:

  $$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$
• Sensitivity, also known as recall or the true positive rate, measures the model’s ability to correctly identify positive instances out of all the actual positive instances:

  $$\mathrm{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$
• Specificity (true negative rate) calculates the model’s ability to correctly identify negative instances out of all the actual negative instances:

  $$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

5. Experimental Results

To validate that augmenting a training set of vectors with vectors distilled from them by the PCA method increases the classification statistics, we conducted experiments applying NN, SVM and LR classifiers on the four databases of feature vectors: skin lesion SL [30,31], diabetes D [32], heart disease HD [33] and breast cancer BC [34]. For this purpose, we designed different set-ups for the training datasets as described in Section 4.2. The classification results obtained are presented in various tables throughout the present section. In the reporting tables, we show two types of accuracy for each experiment. The first percentage represents the model’s accuracy for one iteration, while the second percentage indicates the mean accuracy of the Monte Carlo cross-validation approach for 10 experiments. The bold percentages in all the tables indicate the best results. We present our results in three separate tables for the classification metrics mentioned in Section 4.5 for each dataset.

In Figure 1 are shown the curves of the loss function for the NN model on the SL data. One may observe that the curves of the loss functions for the training processes with the original dataset and Augmented Dataset 1 resemble each other. On the other hand, the curves of the loss functions for the two validation processes are quite different. The curve for the validation process when the NN was trained with the original vectors is convex from below. This indicates possible divergence after the 100 epochs, while the curve for validation when the NN was trained with Augmented Dataset 1 steadily decreases, which suggests a classification improvement if more than 100 epochs are conducted.

In Table 1, we show a comparison of the classification accuracies of different classifiers for the SL dataset and its augmentations mentioned in Section 4.2. The total number of samples in the original dataset was 162, and we split it into 90% vs. 10% sets for training and testing, respectively. By studying Table 1, one can tell that the mean accuracy of the LR significantly increased from 71% to 82% for Augmented Dataset 2.Failure in the remaining classifiers may have occurred due to the fact that the total sample size was too small. Hence, we continued the experimental validation with relatively larger feature vector databases: D, which contained 768 feature vectors of 8D with a P/N ratio of 268/500 [32]; heart disease HD, which contained 1025 feature vectors of 13D and a P/N ratio of 499/526 [33], and breast cancer BC, which contained 569 feature vectors of 30D with a P/N ratio of 212/347 [34].

The results from D data classification with the four classifiers are presented in Table 2, Table 3 and Table 4. We applied the NN, LR and SVM classifiers with linear and polynomial degree 2 kernels, denoted as SVML and SVMP, respectively. The classifiers were trained with the Original, Original + $N_1$-PCA-d and Original + $N_1+N_2$-PCA-d datasets, which are described in Section 4.2. One may notice that the training with the augmented data improved the classification accuracy if compared with the training by the original data. For the NN and SVMP classifiers, the highest increase came for the 10-fold experiments with the single augmentation by 101 distilled vectors. For the LR classifier, the highest results came from the double augmentation of $101+51$ and $151+76$ distilled vectors, while for the SVML classifier, the highest outcome was obtained with $101+51$ distilled vectors. On the other hand, the outcomes for sensitivity and specificity showed high results for the former statistic and results twice as small for the latter. This tells us that the augmentations did not balance the classification models.

Table 2. Comparison of classification accuracies of four classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of 10-fold experiments. Highest values are shown in bold.

Training Data	NN	LR	SVML	SVMP
Original	75%	70%	70%	70%
Original + 101-PCA-d	82%	77%	74%	82%
Original + (101 + 51)-PCA-d	79%	78%	78%	78%
Original + 151-PCA-d	77%	73%	74%	80%
Original + (151 + 76)-PCA-d	79%	78%	64%	67%
Original + 201-PCA-d	75%	67%	73%	73%
Original + (201 + 101)-PCA-d	75%	70%	70%	73%

Table 2. Comparison of classification accuracies of four classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of 10-fold experiments. Highest values are shown in bold.

Training Data	NN	LR	SVML	SVMP
Original	75%	70%	70%	70%
Original + 101-PCA-d	82%	77%	74%	82%
Original + (101 + 51)-PCA-d	79%	78%	78%	78%
Original + 151-PCA-d	77%	73%	74%	80%
Original + (151 + 76)-PCA-d	79%	78%	64%	67%
Original + 201-PCA-d	75%	67%	73%	73%
Original + (201 + 101)-PCA-d	75%	70%	70%	73%

Table 3. Comparison of classification sensitivities of different classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of the 10-fold experiments. In bold is given the highest value of a column.

Training Data	NN	LR	SVML	SVMP
Original	52%	56%	68%	67%
Original + 101-PCA-d	77%	48%	30%	37%
Original + (101 + 51)-PCA-d	81%	56%	26%	56%
Original + 151-PCA-d	52%	41%	30%	56%
Original + (151 + 76)-PCA-d	74%	52%	19%	22%
Original + 201-PCA-d	44%	22%	33%	33%
Original + (201 + 101)-PCA-d	52%	41%	41%	22%

Table 3. Comparison of classification sensitivities of different classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of the 10-fold experiments. In bold is given the highest value of a column.

Training Data	NN	LR	SVML	SVMP
Original	52%	56%	68%	67%
Original + 101-PCA-d	77%	48%	30%	37%
Original + (101 + 51)-PCA-d	81%	56%	26%	56%
Original + 151-PCA-d	52%	41%	30%	56%
Original + (151 + 76)-PCA-d	74%	52%	19%	22%
Original + 201-PCA-d	44%	22%	33%	33%
Original + (201 + 101)-PCA-d	52%	41%	41%	22%

Table 4. Comparison of classification specificities of different classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original data	93%	83%	72%	72%
Original + 101-PCA-d	88%	92%	94%	94%
Original + (101 + 51)-PCA-d	86%	90%	88%	90%
Original + 151-PCA-d	94%	90%	98%	94%
Original + (151 + 76)-PCA-d	86%	92%	88%	92%
Original + 201-PCA-d	96%	94%	94%	94%
Original + (201 + 101)-PCA-d	90%	86%	98%	90%

Table 4. Comparison of classification specificities of different classifiers trained on 90% of the original D data and various augmentations, tested on 10% of the original D data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original data	93%	83%	72%	72%
Original + 101-PCA-d	88%	92%	94%	94%
Original + (101 + 51)-PCA-d	86%	90%	88%	90%
Original + 151-PCA-d	94%	90%	98%	94%
Original + (151 + 76)-PCA-d	86%	92%	88%	92%
Original + 201-PCA-d	96%	94%	94%	94%
Original + (201 + 101)-PCA-d	90%	86%	98%	90%

As mentioned above, we conducted distillation-augmentation experiments using the heart disease HD [33] dataset as well. Recall that it contained 1025 feature vectors with a dimension of 13. Out of them, there were 499 positive (P) vectors and 526 negative (N) vectors. The former vectors indicate heart disease, while the latter indicate heathy samples. The four classifiers were trained again with the following types of sets: Original, Original + $N_1$-PCA-d and Original + $N_1+N_2$-PCA-d. The number $N_1$ denotes the number of vectors distilled from the original training vectors, while $N_2$ denotes the number of vectors distilled from the already distilled $N_1$ vectors. The experimental results, with 90% P and 90% N randomly selected samples for training, are shown in Table 5, Table 6 and Table 7. One may tell from there that the average accuracy increased for all classifiers when they were trained with the augmented set Original + $101+51$-PCA-d, which contained double distillation. The sensitivity increased only for the SVMP classifier because it was already at the maximum for the original training data for the other classifiers. The specificity increased (significantly for the LR, SVML and SVMP classifiers) for all classifiers if trained with augmented sets that contained single and double distillated data. Another important achievement obtained with the HD dataset and the used distillation augmentations is that all classifiers were balanced according to the sensitivity/specificity ratio.

Table 5. Comparison of classification accuracies of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	95%	86%	83%	83%
Original + 101-PCA-d	96%	87%	88%	86%
Original + (101 + 51)-PCA-d	96%	90%	91%	91%
Original + 151-PCA-d	95%	86%	86%	84%
Original + (151 + 76)-PCA-d	95%	83%	83%	83%
Original + 201-PCA-d	89%	86%	85%	83%
Original + (201 + 101)-PCA-d	90%	83%	83%	83%

Table 5. Comparison of classification accuracies of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	95%	86%	83%	83%
Original + 101-PCA-d	96%	87%	88%	86%
Original + (101 + 51)-PCA-d	96%	90%	91%	91%
Original + 151-PCA-d	95%	86%	86%	84%
Original + (151 + 76)-PCA-d	95%	83%	83%	83%
Original + 201-PCA-d	89%	86%	85%	83%
Original + (201 + 101)-PCA-d	90%	83%	83%	83%

Table 6. Comparison of classification sensitivities of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	100%	96%	98%	69%
Original + 101-PCA-d	100%	94%	94%	94%
Original + (101 + 51)-PCA-d	96%	94%	96%	85%
Original + 151-PCA-d	100%	85%	89%	87%
Original + (151 + 76)-PCA-d	98%	85%	83%	87%
Original + 201-PCA-d	100%	96%	96%	98%
Original + (201 + 101)-PCA-d	91%	87%	92%	89%

Table 6. Comparison of classification sensitivities of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	100%	96%	98%	69%
Original + 101-PCA-d	100%	94%	94%	94%
Original + (101 + 51)-PCA-d	96%	94%	96%	85%
Original + 151-PCA-d	100%	85%	89%	87%
Original + (151 + 76)-PCA-d	98%	85%	83%	87%
Original + 201-PCA-d	100%	96%	96%	98%
Original + (201 + 101)-PCA-d	91%	87%	92%	89%

Table 7. Comparison of classification specificities of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	96%	79%	71%	64%
Original + 101-PCA-d	94%	80%	82%	78%
Original + (101 + 51)-PCA-d	98%	86%	86%	88%
Original + 151-PCA-d	94%	88%	84%	78%
Original + (151 + 76)-PCA-d	98%	80%	84%	90%
Original + 201-PCA-d	86%	76%	74%	68%
Original + (201 + 101)-PCA-d	92%	76%	74%	78%

Table 7. Comparison of classification specificities of different classifiers trained on 90% of the original HD data and various augmentations, tested on 10% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	96%	79%	71%	64%
Original + 101-PCA-d	94%	80%	82%	78%
Original + (101 + 51)-PCA-d	98%	86%	86%	88%
Original + 151-PCA-d	94%	88%	84%	78%
Original + (151 + 76)-PCA-d	98%	80%	84%	90%
Original + 201-PCA-d	86%	76%	74%	68%
Original + (201 + 101)-PCA-d	92%	76%	74%	78%

With the help of the heart disease (HD) [33] dataset, we conducted a second set of experiments, decreasing the number of randomly selected original vectors for training to 70%. Then, the testing vectors were the remaining 30% of the feature vectors. All other activities such as distilations and augmentations were the same as in the case of the 90% selected original feature vectors for training. The obtained results for classification with the four chosen classifiers are shown in Table 8, Table 9 and Table 10.

Table 8. Comparison of classification accuracies of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	92%	84%	81%	81%
Original + 101-PCA-d	86%	82%	82%	82%
Original + (101 + 51)-PCA-d	91%	80%	80%	86%
Original + 151-PCA-d	93%	85%	85%	85%
Original + (151 + 76)-PCA-d	88%	84%	83%	82%
Original + 201-PCA-d	91%	83%	83%	87%
Original + (201 + 101)-PCA-d	90%	83%	84%	82%

Table 8. Comparison of classification accuracies of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	92%	84%	81%	81%
Original + 101-PCA-d	86%	82%	82%	82%
Original + (101 + 51)-PCA-d	91%	80%	80%	86%
Original + 151-PCA-d	93%	85%	85%	85%
Original + (151 + 76)-PCA-d	88%	84%	83%	82%
Original + 201-PCA-d	91%	83%	83%	87%
Original + (201 + 101)-PCA-d	90%	83%	84%	82%

Table 9. Comparison of classification sensitivities of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	92%	93%	56%	76%
Original + 101-PCA-d	93%	89%	91%	89%
Original + (101 + 51)-PCA-d	96%	88%	90%	90%
Original + 151-PCA-d	97%	94%	95%	96%
Original + (151 + 76)-PCA-d	91%	92%	93%	93%
Original + 201-PCA-d	93%	87%	86%	91%
Original + (201 + 101)-PCA-d	94%	92%	94%	89%

Table 9. Comparison of classification sensitivities of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	92%	93%	56%	76%
Original + 101-PCA-d	93%	89%	91%	89%
Original + (101 + 51)-PCA-d	96%	88%	90%	90%
Original + 151-PCA-d	97%	94%	95%	96%
Original + (151 + 76)-PCA-d	91%	92%	93%	93%
Original + 201-PCA-d	93%	87%	86%	91%
Original + (201 + 101)-PCA-d	94%	92%	94%	89%

Table 10. Comparison of classification specificities of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	96%	82%	36%	63%
Original + 101-PCA-d	85%	74%	73%	75%
Original + (101 + 51)-PCA-d	88%	71%	81%	81%
Original + 151-PCA-d	91%	77%	73%	74%
Original + (151 + 76)-PCA-d	88%	76%	72%	71%
Original + 201-PCA-d	90%	79%	81%	83%
Original + (201 + 101)-PCA-d	91%	73%	74%	75%

Table 10. Comparison of classification specificities of different classifiers trained on 70% of the original HD data and various augmentations, tested on 30% of the original HD data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	96%	82%	36%	63%
Original + 101-PCA-d	85%	74%	73%	75%
Original + (101 + 51)-PCA-d	88%	71%	81%	81%
Original + 151-PCA-d	91%	77%	73%	74%
Original + (151 + 76)-PCA-d	88%	76%	72%	71%
Original + 201-PCA-d	90%	79%	81%	83%
Original + (201 + 101)-PCA-d	91%	73%	74%	75%

A study of the results in the above tables shows that selecting 70% of the original HD feature vectors for training, distillation and augmentation kept the same trends of increasing the classification statistics as in the case of 90% selection. Moreover, one may observe that with the 70/30% split, the specificities for all classifiers increased when trained with the Original+151-PCA-d dataset. In summary, the augmentation of the original training set with distilled vectors from this set increased the classification statistics and made the classifiers balanced. Only the classification statistics in the 70% split were smaller than the classification statistics in the 90% case.

We conducted the final set of experiments with the BC dataset, which contained 569 feature vectors of 30D with a P/N ratio of 212/347 [34]. For training, we randomly selected 90% of the P and 90% of the negative samples 10 times. Every time, from every selection of training samples, we distilled 101, 151 and 201 vectors with same dimensions as the original training vectors. From every distilled set, we distilled 51, 76, and 101 vectors, respectively, as a second distillation. The experimental results with the original set and the augmented Original + $N_1$-PCA-d and Original + $N_1+N_2$-PCA-d training sets are reported in Table 11, Table 12 and Table 13. Recall that with $N_1$, we denote a set of vectors distilled during the first distillation, while $N_2$ denotes a set of vectors from the second distillation.

A study of the results shows a significant increase in the classification accuracy of the NN classifier for the augmented set Original+101-PCA-d and for the SVML classifier with the augmented set Original + 151-PCA-d. Concerning the LR and SVMP classifiers, their highest results with the latter augmented set were the same as the accuracy of classification with the original training data set. The reason for this is that the accuracies of the latter data set were high enough, and there was no room for further increases. The same observation and conclusion hold for the specificity of the NN and the sensitivities of the LR and SVMP classifiers. The last two statistics exhibited a significant increase for the remaining classifiers when trained with augmented datasets. Moreover, the sensitivity/specificity ratio was balanced well, leading to the highest balanced accuracy = (sensitivity + specificity)/2.

Table 11. Comparison of classification accuracies of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	87%	95%	65%	98%
Original + 101-PCA-d	97%	84%	88%	86%
Original + (101 + 51)-PCA-d	90%	84%	86%	88%
Original + 151-PCA-d	81%	95%	98%	98%
Original + (151 + 76)-PCA-d	95%	81%	86%	90%
Original + 201-PCA-d	94%	95%	97%	97%
Original + (201 + 101)-PCA-d	84%	93%	97%	93%

Table 11. Comparison of classification accuracies of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of 10-fold experiments. In bold are the highest values.

Training Data	NN	LR	SVML	SVMP
Original	87%	95%	65%	98%
Original + 101-PCA-d	97%	84%	88%	86%
Original + (101 + 51)-PCA-d	90%	84%	86%	88%
Original + 151-PCA-d	81%	95%	98%	98%
Original + (151 + 76)-PCA-d	95%	81%	86%	90%
Original + 201-PCA-d	94%	95%	97%	97%
Original + (201 + 101)-PCA-d	84%	93%	97%	93%

Table 12. Comparison of classification sensitivities of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	100%	79%	47%	94%
Original + 101-PCA-d	100%	100%	100%	100%
Original + (101 + 51)-PCA-d	100%	95%	95%	95%
Original + 151-PCA-d	100%	100%	100%	100%
Original + (151 + 76)-PCA-d	100%	100%	100%	100%
Original + 201-PCA-d	100%	95%	95%	95%
Original + (201 + 101)-PCA-d	100%	100%	100%	100%

Table 12. Comparison of classification sensitivities of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	100%	79%	47%	94%
Original + 101-PCA-d	100%	100%	100%	100%
Original + (101 + 51)-PCA-d	100%	95%	95%	95%
Original + 151-PCA-d	100%	100%	100%	100%
Original + (151 + 76)-PCA-d	100%	100%	100%	100%
Original + 201-PCA-d	100%	95%	95%	95%
Original + (201 + 101)-PCA-d	100%	100%	100%	100%

Table 13. Comparison of classification specificities of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	83%	100%	73%	100%
Original + 101-PCA-d	100%	75%	81%	78%
Original + (101 + 51)-PCA-d	89%	77%	80%	83%
Original + 151-PCA-d	83%	92%	97%	97%
Original + (151 + 76)-PCA-d	94%	69%	78%	83%
Original + 201-PCA-d	94%	94%	97%	97%
Original + (201 + 101)-PCA-d	81%	89%	94%	89%

Table 13. Comparison of classification specificities of different classifiers trained on 90% of the original BC data and various augmentations, tested on 10% of the original BC data. Each column shows the average of the 10-fold experiments. In bold is given the highest value in a column.

Training Data	NN	LR	SVML	SVMP
Original	83%	100%	73%	100%
Original + 101-PCA-d	100%	75%	81%	78%
Original + (101 + 51)-PCA-d	89%	77%	80%	83%
Original + 151-PCA-d	83%	92%	97%	97%
Original + (151 + 76)-PCA-d	94%	69%	78%	83%
Original + 201-PCA-d	94%	94%	97%	97%
Original + (201 + 101)-PCA-d	81%	89%	94%	89%

6. Discussion

The present paper developed a method for vector data augmentation through distillation. The method is based on principal component analysis (PCA) [20,21]. Its difference with the extension we developed for vector data distillation is that the former method maps the set of original vectors to a set of vectors with a smaller dimension and the same cardinality. On the other hand, the extended PCA method maps the original set of vectors to a set of vectors with the same dimension as the original set but with a smaller cardinality.

The extended PCA method for distillation is unproductive if the matrix $\mathbf{T}$ in Equation (7) is 1-orthogonal. Note that by definition, the matrix $\mathbf{T}$ is 1-orthogonal if and only if $\mathbf{T}{\mathbf{T}}^T=\mathbf{I}$, where $\mathbf{I}$ is the identity matrix. Therefore, if $\mathbf{T}$ is 1-orthogonal, then Equations (7) and (8) will map the original training dataset ${\mathbf{D}}_{\mathbf{t}}$ onto itself, and distilled vectors will not be generated. To remedy the problem, we selected $(m+1)<r$ strongest vectors instead of m, as is shown above in Equation (7).

The novelty of the present study is the development and use of the extended PCA method for distillation of vectors from given set of vectors such that the distilled vectors have the same dimension as the original vectors.

The advantage that comes from the proposed novelty is that by adding the PCA-distilled vectors to the original vectors, from which the former ones were distilled, we received a new training set. This new set trains a classifier such that its model has statistics higher than the statistics of the model trained only with the original dataset.

We validated the advantage by applying four classifiers—NN, LR, SVML, SVMP—on four different datasets of vectors: SL, D, HD, and BC. Every experimental result was produced by Monte-Carlo 10-fold cross-validation, where we randomly selected 10% of the samples for testing 10 times while the remaining 90% were used for training. Then, the average was taken. For every selection, the augmented vectors were distilled from the set of selected training vectors. We conducted a second (nested) distillation from every set of vectors generated during the first distillation. The experimental results are shown in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 11, Table 12 and Table 13, where we compared the outcomes after training with the original vectors only and the outcomes after training with augmented datasets. It is evident that the latter outcomes were higher and better balanced if compared with the former.

In Table 1, one may observe that the NN accuracy was lower than the LR accuracy. It is known that NNs are intrinsically more prone to overfitting than LR. While being a universal approximator, it is expected that the NNs will outperform LRs in some instances. However, these models will not always be superior, especially when a rather small amount of training data is available. Recall that we used 90% of the SL data for training. This gave 145 training vectors, which is a quite insufficient amount for the efficient training of an NN and is the main reason for the lower NN results if compared with LR.

In [19], the PCA method was used to distill a photo response non-uniformity reference by removing the interference noise, which is another example of applying this method to the field of data enhancement. However, a direct comparison is not possible because our method distills vectors, while the method in [19] distills images. Our tests on a laptop with an Intel (13th Gen) i5-1335U processor at 1.30 GHz and 16.0 GB of RAM show that on average (averaged over 10 runs), the time required for distillation with PCA and augmentation of the dataset was approximately 0.1 milliseconds per vector.

GANs [12] and VAEs are typically used to generate or augment image data, and they achieve extraordinary results in this domain. On the contrary, the extended PCA method distills vector data. However, 1D data generation and augmentation are more difficult to achieve, and the results of generative or variational networks are much more modest. Aside from that, these networks require a significant amount of training data in the first place to produce good results. We are tackling the problem that no more data are available for training.

Our future work continues with enlarging classes which have small cardinalities. In certain datasets, a big difference exists between the number of samples in the different classes. Hence, we will conduct distillations from the samples of the class with the smallest cardinality and will augment this class to increase its cardinality. For this purpose, we will investigate the number of consecutive distillations which provide meaningful and useful sample vectors.