Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data

Machine learning has been proven to be effective in various application areas, such as object and speech recognition on mobile systems. Since a critical key to machine learning success is the availability of large training data, many datasets are being disclosed and published online. From a data consumer or manager point of view, measuring data quality is an important first step in the learning process. We need to determine which datasets to use, update, and maintain. However, not many practical ways to measure data quality are available today, especially when it comes to large-scale high-dimensional data, such as images and videos. This paper proposes two data quality measures that can compute class separability and in-class variability, the two important aspects of data quality, for a given dataset. Classical data quality measures tend to focus only on class separability; however, we suggest that in-class variability is another important data quality factor. We provide efficient algorithms to compute our quality measures based on random projections and bootstrapping with statistical benefits on large-scale high-dimensional data. In experiments, we show that our measures are compatible with classical measures on small-scale data and can be computed much more efficiently on large-scale high-dimensional datasets.


Introduction
We are witnessing the success of machine learning in various research and application areas, such as vision inspection, energy consumption estimation, and autonomous driving, just to name a few. One major contributor to the success is the fact that the datasets are continuously accumulated and openly published in several domains. Low data quality is very likely to cause inferior prediction performance of machine learning models, and therefore measuring data quality is an indispensable step in a machine learning process. Especially in real-time and mission-critical cyber-physical-system, defining appropriate data quality measures is critical since the low generalization performance of a deployed model can result in system malfunction and possibly catastrophic damages to the physical world. Despite the importance, there exist only a few works for measuring data quality where most of them are hard to evaluate on large-scale high-dimensional data due to computation complexity.
A popular early work on data quality measures includes Ho and Basu [1], proposing 12 types of quality measures which are simple but powerful enough to address different aspects of data quality. These measures have limitations, however, in that it is difficult to compute them for large-scale high-dimensional and multi-class datasets. Baumgartner and Somorjai [2] proposed a quality measure designed for high-dimensional biomedical datasets; however, it does not work efficiently on large-scale data. Recently, Branchaud-Charron et al. [3] proposed a quality measure for high-dimensional data using spectral clustering. Although this measure is adequate for large-scale high-dimensional data, it requires an embedding network which involves a large amount of computation time for training.
In this paper, we propose three new quality measures called M sep , M var , and M var i , and their computation methods that overcome the above limitations. Our approach is inspired by Fisher's linear discriminant analysis (LDA) [4], which is mathematically well-defined for finding a feature subspace that maximizes class separability. Our computation method makes use of the techniques from statistics, random projection [5] and bootstrapping [6], to compute the measure for large-scale high-dimensional data efficiently.
The contributions of our paper are summarized as follows: • We propose three new data quality measures that can be compuated directly from a given dataset and can be compared across datasets with different numbers of classes, examples, and features. Although our approach takes ideas from LDA (linear discriminant analysis) and PCA (principal component analysis), the techniques by themselves do not produce single numbers that are comparable across different datasets. • We provide efficient algorithms to approximate the suggested data quality measures, making them available for large-scale high-dimensional data. • The proposed class separability measure M sep is strongly correlated with the actual classification performance of linear and non-linear classifiers in our experiments. • The proposed in-class variability measures M var and M var i quantify the diversity of data within each class and can be used to analyze redundancy or outlier issues.

Related Work
In general, the quality of data can be measured by Kolmogorov complexity which is also known as the descriptive complexity for algorithmic entropy [7]. However, the Kolmogorov complexity is not computationally feasible; instead, an approximation is used in practice. To our knowledge, there are three main approaches for approximating the Kolmogorov complexity: descriptor-based, classifier-based, and graph-based approaches. We describe these three main categories below.

Descriptor-Based Approaches
Ho and Basu [1] proposed simple but powerful quality measures based on descriptors. They proposed 12 quality measures, namely F1, F2, F3, L1, L2, L3, N1, N2, N3, N4, T1, and T2. The F measures represent the amount of feature overlap. In particular, F1 measures the maximum Fisher's ratio which represents the maximum discriminant power of between features. F2 represents the volume of overlap region in the two-class conditional distributions. F3 captures the ratio of overlapping features using the maximum and minimum value. The L measures are for the linear separability of classes. L1 is a minimum error of linear programming (LP), L2 is an error rate of a linear classifier by LP, and L3 is an error rate of linear classifier after feature interpolation. The N measures represent mixture identifiability, the distinguishability of the data points belonging to two different classes. N1 represents the ratio of nodes connected to the different classes using the minimum spanning tree of all data points. N2 is the ratio of the average intra-class distance and average inter-class distance. N3 is the leave-one-out error rate of the nearest neighbor (1NN). N4 is an error rate of 1NN after feature interpolation. The T measure represents the topological characteristic of a dataset. T1 represents the number of hyperspheres adjacent to other class features. T2 is the average number of data points per dimension. These quality measures can capture various aspects of data quality; however, they are fixed for binary classification and not applicable for multi-class problems. Furthermore, quality measures, such as N1, N2, and N3, require a large amount of computation time on large-scale high-dimensional data.
Baumgartner and Somorjai [2] proposed a quality measure for high-dimensional but small biomedical datasets. They used singular value decomposition (SVD) with time complexity O(min(m 2 n, mn 2 )), where m is the number of data points, and n is the number of features. Thus, it is computationally demanding to calculate their measures for the datasets with large m and n, such as recent image datasets.
There are other descriptor-based approaches, for example for meta learning [8,9], for classifier recommendation [10], and for synthetic data generation [11]. However, only a small number of data points in a low dimensional space have been considered in these works.

Graph-Based Approaches
Branchaud-Charron et al. [3] proposed a graph-based quality measure using spectral clustering. First, they compute a probabilistic divergence-based K × K class similarity matrix S, where K is the number of classes. Then, an adjacency matrix W is computed from the S matrix. The quality measure is defined as a cumulative sum of the eigenvalues gap which is called as cumulative spectral gradient (CSG), which represents the minimum cutting cost of the S. The authors also used a convolutional neural network-based auto-encoder and t-SNE [12,13] to find an embedding that can represent data points (images in their case) well. Although the method is designed for high-dimensional data, it requires to train a good embedding network to reach quality performance.
Duin and Pękalska [14] proposed a quality measure based on a dissimilarity matrix of data points. Since calculating the dissimilarity matrix is a time-consuming process, the method is not adequate for large-scale high-dimensional data.

Classifier-Based Approaches
Li et al. [15] proposed a classifier-based quality measure called an intrinsic dimension, which is the minimum number of solutions for certain problems. For example, in a neural network, the intrinsic dimension is the minimum number of parameters to reach the desired prediction performance.
The method has a benefit that it can be applied to many different types of data as long as one has trainable classifiers for the data; however, it often incurs high computation cost since it needs to change the number of classifier parameters iteratively during data quality evaluation.
Overall, the existing data quality measures are mostly designed for binary classification in low dimension spaces with a small number of data points. Due to their computation complexity, they tend to consume large amount of time when applied to large-scale highdimensional data. In addition, the existing measures tend to focus only on the inter-class aspects of data quality. In this paper, we propose two new data quality measures suitable for large-scale high-dimensional data resolving the above mentioned issues.

Methods
In this section, we formally describe our data quality measures. We focus on multi-class classification tasks where each data point is associated with a class label out of c categories (c ≥ 2). Our measures are created by adapting ideas from Fisher's LDA [4]. Fisher's LDA is a dimensionality reduction technique, finding a projection matrix that maximizes the betweenclass variance and minimizes the within-class variance at the same time. Motivated by the idea, we propose two types of data quality measures, class separability M sep and in-class variability M var and M var i . For efficient handling of large-scale high-dimensional data, we also propose techniques to reduce both computation and memory requirements taking advantage of statistical methods, bootstrapping [6] and random projection [5].

Fisher's LDA
The objective of Fisher's LDA [4] is to find the feature subspace which maximizes the linear separability of a given dataset. Fisher's LDA achieves the objective by minimizing the within-class variance and maximizing the between-class variance simultaneously.
To describe the Fisher's LDA formally, let us consider an input matrix X ∈ R m×n where m is the number of data points, n is the input dimension, and x i,j ∈ R n is a j-th data point in the i-th class. The within-class scatter matrix S w ∈ R n×n is defined as follows: Here, c is the number of classes, m i is the number of data points in the i-th class, and x i ∈ R n is the mean of data points in the i-th class. This formulation can be interpreted as the sum of class-wise scatter matrices. A small determinant of S w indicates that data points in the same class exist densely in a narrow area, which may lead to high class separability.
Next, the between-class scatter matrix S b ∈ R n×n is defined as follows: where x is the mean of entire data points in the given dataset. A large determinant of S b indicates that the mean vector x i of each class is far from the x, another condition hinting for high class separability. Using these two matrices, we can describe the objective of Fisher's LDA as follows: Here, Φ lda ∈ R n×d is the projection matrix where d is the dimension of feature subspace (in general, we choose d n). The column vectors of projection matrix Φ lda are the axes of feature subspace, which maximize the class separability. The term in the objective function is also known as the Fisher's criterion. By projecting X onto these axes, we obtain a ddimensional projection of the original data X ∈ R m×d : In general, if S w is an invertible matrix, we can calculate the projection matrix which maximizes the objective of the Fisher's LDA by eigenvalue decomposition.

Proposed Data Quality Measures
Motivated by the ideas in Fisher's LDA, we propose two types of new data quality measures: M sep (class separability), M var and M var i (in-class variability).

Class Separability
Our first data quality measure tries to capture the class separability of a dataset by combining the within-class variance and between-class variance, similarly to Fisher's LDA (1) but more efficiently for large-scale and high-dimensional data and comparable with other datasets.
We start from creating the normalized versions of the matrices S w and S b in Fisher's LDA (1) so that they will not be affected Considering the determinants of these n × n matrices as in the original Fisher's LDA will be too costly for a high-dimensional data where n is large, since the time complexity to compute the determinants will be proportional nearly to n 3 . Instead, we consider the direction of maximum linear separation v ∈ R n that maximizes the ratio of between-class variance to the within-class variance being projected onto the vector. Using the vector, we define our first data quality measure M sep for class separability as follows: This formulation is almost the same as (1) in Fisher's LDA except that (1) finds the projection matrix φ lda which maximizes the Fisher's criterion, while, in (3), we will focus on finding the maximum value of Fisher's criterion itself. Unlike Fisher's criterion, our measure M sep is comparable across datasets with different numbers of classes and examples due to normalization, to check the relative difficulty of linear classification.
Solving (3) directly will be preventive for a large n as in the original LDA. IfŜ w is invertible, we can calculate the vector which maximizes M sep as follows using simple linear algebra. To find the vector v which maximizes the equation in (3), first differentiate it with respect to v to get: ( This leads us to the following generalized eigenvalue problem in the form of: where λ = v TŜ b v v TŜ w v can be thought as an eigenvalue of the matrixŜ −1 wŜb . The maximizer v is the eigenvector corresponding to the largest eigenvalue ofŜ −1 wŜb which can be found rather efficiently by the Lanczos algorithm [16]. However, the overall time complexity for computation can be up to O(n 3 ), which makes it difficult to calculate the optimal vector for high-dimensional data, such as images. In Section 3.3, we provide an efficient algorithm to compute M sep using random projection.

In-Class Variability
Our second data quality measure gauges the in-class variability. Figure 1 shows one of the motivating examples to consider in-class variability for data quality. In the figure, we have two photos of the Bongeunsa temple in Seoul, Korea, taken by the same photographer. The photographer had been asked to take photos of Korean objects from several different angles, and it turned out that quite a few of the photos were taken in only marginal angle differences. Since the data creation was a government-funded project providing data for developing object recognition systems in academia and industry, low data variability was definitely an issue. Here, we define two types of in-class variability measure, the overall in-class variability of a given dataset M var and the in-class variability of the i-th class, M var i . First, the overall in-class variability M var tries to capture the minimum variance of data points being projected onto any direction, based on the matrixŜ w defined in (2): where c is the number of class and n is the dimension of data. Unlike class separability, we added additional normalization factors c and n, since the valueŜ w is affected by the number of class and the data dimension. Second, the class-wise in-class variability M var i is based on the sample covariance matrix of each class:Ŝ where m i is the number of data points in the i-th class. The class-wise in-class variability measure M var i is defined as follows: The normalization factors c and n are required for the same reason as in M var . The measure M var i represents the smallest variance of the data points in the same class after being projected onto any direction.
As a matter of fact, M var and M var i are the same as the smallest eigenvalue of 1 c·nŜ w and 1 c·nŜ w i which can be computed for instance using the Lanczos algorithm [16] on the inverse of them with O(n 3 ) computation, which will be preventive for large data dimensions n. We discuss a more efficient way to estimate the value in the next section, which can be computed alongside with our first data quality measure without significant extra cost.
Using the M var and M var i , we can analyze the variety or redundancy in a given dataset. For instance, a very small M var and M var i would indicate that we may have a small diversity issue where data points in the invested class are mostly alike. On the other hand, the overly large M var and M var i may indicate a noise issue in the class, possibly including incorrect labeling. The difference between M var and M var i is that the M var aggregates the information of diversity for each class, and the M var i represents the information of diversity for a specific class. Since the M var aggregates the information of variability of data points in each class, we can use this for comparing the in-class variability between datasets. On the other hand, we can use M var i for the datasets analysis, i.e., data points of a specific class with less M var i than other classes may cause low generalization performance. We will discuss more details in Section 4.

Methods for Efficient Computation
One of the key properties required for data quality measures is that they should be computable in a reasonable amount of time and computation resources since the amount and the dimension of data are keep increasing as new advanced sensing technologies become available. In this section, we describe how we avoid a large amount of time and memory complexity to compute our suggested data quality measures.

Random Projection
Random projection [5] is a dimension reduction technique that can transform an ndimensional vector into a k-dimensional vector (k n), while preserving the critical information of the original vector. The idea behind of random projection is the Johnson-Lindenstrauss lemma [17]. That is, for any vectors {x, x } ∈ X from a set of m vectors in X ⊂ R n and for ∈ (0, 1), there exists a linear mapping f : R n → R k such that the pairwise distances of vectors are almost preserved after projection in the sense that: where k > 8 ln(m)/ 2 . It is known that when the original dimension n is large, a random projection matrix P ∈ R k×n can serve as the feature mapping f in the lemma, since random vectors in R n tend to be orthogonal to each other as n increases [18]. Motivated by the above phenomenon, we use random projection to find a vector that satisfies (3) instead of calculating the eigenvalue decomposition to solve (4). The idea is that if the number of random vectors is sufficiently large, the maximum value of the Fisher's criterion calculated by random projection can approximate the behavior of a true solution.
Furthermore, random projection makes it unnecessary to explicitly storeŜ w andŜ b since we can simply compute the denominator and numerator of (3) as follows: where w is a random unit vector drawn from N (0, 1). This technique is critical for dealing with high-dimensional data, such as images, in a memory-efficient way. In our experiments, ten random projection vectors were sufficient in most cases to accurately estimate our quality measures.

Bootstrapping
Bootstrapping [6] is a sampling-based technique that estimates the statistic of the population with little data using sampling with replacement. For instance, bootstrapping can be used to estimate the mean and the variance of a statistic from an unknown population. Let s i is a statistic of interest that is calculated from a randomly drawn sample of an unknown population. The mean and variance of the statistic can be estimated as follows: where B is the number of bootstrap samples,μ is a mean estimate of the statistic, andσ 2 is the variability of the estimate. By using a small B, we can reduce the number of data points to be considered at once. We found that B = 100 and making each bootstrap sample to be 25% of a given dataset in size worked well overall our experiments. We summarized the above proce- Calculate the number of class c and data dimension n from the dataset Sampling with replacement using stratified sampling as much as R ratio from the dataset Standardize the sampled dataset while j ≤ nv do j ← − j + 1 w ← − a unit vector drawn from N (0, 1) Compute w TŜ w j,c w w TŜ w j w ← − Sum({w TŜ w j,1 w, . . . , w TŜ w j,c w}) c·n min(w TŜ w 1,1 w, . . . , w TŜ w nv,1 w), . . . , 1 c·n min(w TŜ w 1,c w, . . . , w TŜ w nv,c w)}) A var .insert

Experiment Results
In this section, we show that our method can evaluate the data quality of the large-scale high-dimensional dataset efficiently.
To verify the representative performance of M sep for class separability, we calculated the correlation between the accuracy of chosen classifiers and M sep . Classifiers used in our experiments are as follows: a perceptron, a multi-layer perceptron with one hidden layer and LeakyReLU (denoted by MLP-1), and a multi-layer perceptron with two hidden layers and LeakyReLU (denoted by MLP-2). To simplify the experiments, we trained the models with the following settings: 30 epochs, a batch size of 100, a learning rate of 0.002, the Adam optimizer, and the cross-entropy loss function. Additionally, we fixed the hyperparameters of Algorithm 1 as B = 100, R = 0.25, and nv = 10 since there was no big difference in performance when larger hyperparameter values were used.
For comparison with other quality measures, we chose F1, N1, and N3 from Ho and Basu [1] and CSG from Branchaud-Charron et al. [3]. Here, N1, N3, and CSG are known to be highly correlated with test accuracy of classifiers Branchaud-Charron et al. [3]. F1 is similar to our M sep in its basic idea. Other quality measures suggested in Ho and Basu [1] showed very similar characteristics to F1, N1, N3, and CSG and are therefore not included in the results.

Datasets
To evaluate the representative performance of M sep for class separability, we used various image datasets that are high-dimensional and popular in mobile applications. We chose ten benchmark image datasets for our experiments: MNIST, notMNIST, CIFAR10, Linnaeus, STL10, SVHN, ImageNet-1, ImageNet-2, ImageNet-3, and ImageNet-4. MNIST [19] consists of ten handwritten digits from 0 to 9. The dataset contains 60,000 training and 10,000 test data points. We sampled 10,000 data from the training data for a model training and measuring the quality, and we sampled 2500 data from the test data for assessing the model accuracy. The notMNIST [20] dataset is quite similar to MNIST, containing English letters from A to J in various fonts. It has 13,106 training and 5618 test samples. We sampled the data in the same way as MNIST. Linnaeus [21] consists of five classes: berry, bird, dog, flower, and others. Although the dataset is available in various image sizes, we chose 32 × 32 to reduce the computation time of N1, N3, and CSG. CIFAR10 [22] is for object recognition with ten general object classes. It consists of 50,000 training data and 10,000 test data points. We sampled the CIFAR10 dataset in the same way as MNIST. STL10 [23] is also for object recognition with ten classes, and it has 92 × 92 images: we resized the images into 32 × 32 to reduce the computation time for the N1, N3, and CSG. The dataset consists of 5000 training and 8000 test data points. We combined these two sets into a single dataset, and then sampled 10,000 data points from the combined set for model training and measuring quality. We also sampled 2500 data points from the combined set for assessing prediction model accuracy if necessary. SVHN [24] consists of street view house number images. The dataset contains 73,200 data points. We sampled 10,000 training data for a model training and measuring the quality, and we sampled 2500 data for assessing the model accuracy. ImageNet-1, ImageNet-2, ImageNet-3 and ImageNet-4 are subsets of Tiny ImageNet dataset [25]. The Tiny ImageNet dataset contains 200 classes, and each class has 500 images. They are consist of randomly selected ten classes of the Tiny ImageNet dataset (total 5000 data points). We used 4500 data points for model training and measuring the quality and 500 data points for assessing the model accuracy, respectively.
We summarized the details of datasets in Table 1. The accuracy values in the Table 1 are calculated from the MLP-2 model since it showed good overall performance compared to the perceptron and the MLP-1 models.

Representation Performance of the Class Separability Measure M Sep
Here, we show in experiments that how well our first quality measure M sep represents class separability, compared to simple but popular classifiers and the existing data quality measures.

Correlation with Classifier Accuracy
To demonstrate how well M sep represents the class separability of given datasets, we compared the absolute value of Pearson correlation and Spearman rank correlation between quality measures M sep , N1, N3, F1, and CSG to the prediction accuracy of three classification models: perceptron, MLP-1, and MLP-2. Table 2 summarizes the results.
In the case of the perceptron, M sep has a similar Pearson correlation with the shortest computation time to the N1 and N3 which have the highest correlation with the accuracy of classifiers. Furthermore, M sep and F1 have the highest Spearman rank correlation. This is because M sep and F1 measure linear separability that is essentially the information captured by the linear classifier, the perceptron in our case. In the case of MLP-1 and MLP-2, M sep also showed a sufficiently high correlation with classification accuracy although it is slightly lower in Pearson correlation compared to the case of the perceptron. On the other hand, CSG does not seem to have noticeable benefits considering its computation time. This is because CSG is affected by an embedding network which requires a large amount of training time.
In summary, the result shows that our measure M sep can capture separability of data as good as the existing data quality measures, while reducing computation time significantly. Table 2. The absolute Pearson and Spearman rank correlation between the quality measures and the accuracy of three classifiers on the ten image datasets (MNIST, CIFAR10, notMNIST, Linnaeus, STL10, SVHN, ImageNet-1, ImageNet-2, ImageNet-3, and ImageNet-4). The computation time of our method M sep is the fastest.

Classifier
Quality In order to check if our suggested data quality measure M sep is compatible with the existing ones in quality, and therefore ours can be a faster alternative to the existing data quality measures, we computed the Pearson correlation between F1, N1, N3, CSG, and M sep . The results are summarized in Table 3. Our measure M sep showed a high correlation with all four existing measures F1, N1, N3, and CSG, indicating that M sep is able to capture the data quality information represented by F1, N1, N3, and CSG.

Computation Time
As mentioned above, our quality measure M sep represents the class separability well, but much faster in computation than F1, N1, N3, and CSG. Here, we show how the computation time changes according to data dimension and sample sizes, in order to show that our suggested data quality measure can be used for many big-data situations.
The computation time according to the data dimension is shown in Figure 2 and Table 4. In all dimensions, our measure M sep was on average 3.8 times faster than F1, 13.1 times faster than N1, 25.9 times faster than N3, and 17.7 times faster than CSG. Since the N1, N3, and CSG have to calculate the MST and to train a 1NN classifier and embedding networks, respectively, it is inevitable that they would take a large amount of computation time (see more details in Sections 2.1 and 2.2). On the other hand, since M sep utilizes random projection and bootstrapping to avoid eigenvalue decomposition problem and to deal with the big-data situations, the computation time of M sep is shortest in all cases.   Figure 3 and Table 5 show how computation time changes for various sample sizes. Our measure M sep was on average 2.8 times faster than F1, 47.0 times faster than N1, 94.5 times faster than N3, and 41.6 times faster than CSG. N1 and N3 show extremely increasing computation time with respect to the sample size, which is not suitable for large-scale highdimensional datasets.
All the above results show that our measure M sep is suitable for the big-data situations and compatible with other well-accepted data quality measures.

Comparison to Exact Computation
In Section 3.2, we proposed to use random projections and bootstrapping for fast approximation of the solution of (4), which can be computed exactly as an eigenvalue. Here, we compare the values of M sep using the proposed approximate computation (denoted by "Approx") and the exact computation (denoted by "Exact") due to an eigensolver in the Python scipy package. One thing is that, since we use only Gaussian random vectors for projection, it is likely that they may not match the true eigenvectors; therefore, the approximated quantity would differ from the exact value. However, we found that the approximate quantities match well the exact values in their correlation, as indicated in Table 6, and, therefore, can be used for fast comparison of data quality of high-dimensional large-scale datasets. In fact, many of the existing data quality measures are designed to measure the difficulty of classification for a given dataset. However, we believe that the in-class variability of data must be considered as another important factor of data quality. One example to show the importance and usefulness of our in-class variability measure M var i is the generalization performance of a classifier.
The generalization performance of the learning model is an important consideration especially in mission-critical AI-augmented systems. There are many possible reasons causing low generalization, and overfitting is one of the troublemakers. Although we have techniques to alleviate overfitting, e.g., checking the learning curve, regularization [26,27], and ensemble learning [28], it is critical to check if there is an issue in data to begin with which may lead to any inductive bias. For example, a very small value of M var i in a class compared to the others would indicate a lack of variability in the class, which can lead to low generalization due to, e.g., unchecked input noise, background signal, object occlusion, and angle/brightness/contrast differences during training. On the other hand, the overly large M var i may indicate outliers or even mislabeled data points likely to incur unwanted inductive bias in training.
To show the importance and usefulness of in-class variability, we created a degraded version of CIFAR10 (denoted by degraded-CIFAR10) by reducing the variability of a specific class. The degraded-CIFAR10 is created by the following procedure. First, we chose an image of the deer class in the training data, then selected the nine mostly similar images in the angular distance to the chosen one. Figure 4 shows the total ten images selected by the above procedure that have similar backgrounds and shapes. Next, we created 1000 images by sampling with replacement from the ten images, while adding random Gaussian noise with zero mean and unit variance, and we replaced the original deer class data with sampled degraded deer class data.  Table 7 shows that the value of M var i is significantly small on the degraded deer class compared to the other classes. That is, it can capture small in-class variability. In contrast, Table 8 shows that the existing quality measures F1, N1, N3, and CSG may not be enough to signify the degradation of the dataset. As we can see, all quality measures indicate that class separability increased in degraded-CIFAR10 compared to the original CIFAR10; however, the test accuracy from MLP-2 decreased. This is because the reduction in in-class variability is very likely to decrease the generalization performance. Therefore, class separability measures can deliver incorrect information regarding data quality in terms of in-class variability, which can be a critical problem for generating a trustworthy dataset or training a trustworthy model. As we showed above, the small value of M var i of a specific class represents that similar images do exist in the invested class, which can lead to low generalization performance of classifiers. Suppose we have generated a dataset for an autonomous driving object classification task. The dataset has been revealed that it has a high class separability through various quality measures. Moreover, the training accuracy was also high. Therefore, one may expect high generalization performance. Unfortunately, the exact opposite can happen. If the variability in the specific class is small as in the degraded-CIFAR10 example above, high generalization performance cannot be expected. For instance, if a car with new colors and new shapes that have never been trained is given as an input to the model, the probability of properly classifying the car will be low. This example indicates that in-class variability plays an important role in data quality evaluation.

Quality Ranking Using M Sep and M Var
As we mentioned before, quality measures M sep and M var can be compared among different datasets. The class separability M sep represents the relative difficulty of linear classification, and the overall in-class variability M var represents the average variability of data points in classes. Figure 5 shows a data quality comparison plot of datasets in our experiments. The direction towards the lower-left corner indicates lower class separability and lower in-class variability, and the upper-right direction is for higher class separability and higher in-class variability. According to the plot, the MNIST and the notMNIST dataset show very high linear separability compared to other datasets, indicating that their classification might be easier than the other datasets. The SVHN dataset is at the lower-left corner, indicating low linear separability and possible redundancy issues (this could be just the reflection of the fact that many SVHN images contain changing digits but the same backgrounds). The four ImageNet datasets, Linnaeus, CIFAR10 and STL10 have similar class separability and inclass variability values. This appears to be understandable considering their similar data construction designed for object recognition.

Conclusions
In this paper, we proposed data quality measures M sep , M var and M var i , which can be applied efficiently on large-scale high-dimensional datasets. Our measures are estimated using random projection and bootstrapping and therefore can be applied efficiently on largescale high-dimensional data. We showed that M sep can be used as a good alternative to the existing data quality measures capturing class separability, while reducing their computational overhead significantly. In addition, M var and M var i measures in-class variability, which is another important factor to avoid unwanted inductive bias in trained models.