Lightweight and Elegant Data Reduction Strategies for Training Acceleration of Convolutional Neural Networks

Abstract

Due to industrial demands to handle increasing amounts of training data, lower the cost of computing one model at a time, and lessen the ecological effects of intensive computing resource consumption, the job of speeding the training of deep neural networks becomes exceedingly challenging. Adaptive Online Importance Sampling and IDS are two brand-new methods for accelerating training that are presented in this research. On the one hand, Adaptive Online Importance Sampling accelerates neural network training by lowering the number of forward and backward steps depending on how poorly a model can identify a given data sample. On the other hand, Intellectual Data Selection accelerates training by removing semantic redundancies from the training dataset and subsequently lowering the number of training steps. The study reports average 1.9x training acceleration for ResNet50, ResNet18, MobileNet v2 and YOLO v5 on a variety of datasets: CIFAR-100, CIFAR-10, ImageNet 2012 and MS COCO 2017, where training data are reduced by up to five times. Application of Adaptive Online Importance Sampling to ResNet50 training on ImageNet 2012 results in 2.37 times quicker convergence to 71.7% top-1 accuracy, which is within 5% of the baseline. Total training time for the same number of epochs as the baseline is reduced by 1.82 times, with an accuracy drop of 2.45 p.p. The amount of time required to apply Intellectual Data Selection to ResNet50 training on ImageNet 2012 is decreased by 1.27 times with a corresponding decline in accuracy of 1.12 p.p. Applying both methods to ResNet50 training on ImageNet 2012 results in 2.31 speedup with an accuracy drop of 3.5 p.p.

1. Introduction

In recent years, Deep Learning (DL) has gained significant traction as a preferred technology in diverse application domains. Examples of these domains include computer vision (CV) [1], signal processing [2], natural language processing [3], robotics [4], autonomous driving [5,6], and reinforcement learning [7].

The ubiquitous adoption of DL in various industrial applications can be attributed to the steady advancement of computational power in modern hardware and the overwhelming availability of data for analysis. According to a corresponding study [8] on digitization, it is posited that the volume of data is projected to reach 175 zettabytes by 2025 [8]. Simultaneously, it is evident that contemporary datasets intended for the purpose of training deep neural networks (DNN) aimed at CV have grown considerably. Specifically, the number of images used for training such networks has increased from 14.2 million in the ImageNet challenge in 2015 [9] to 300 million in JFT-300M in 2017 [10], and to 3 billion images in JFT-3B 2022 [11].

DNN excel in the task of selecting salient features from input data and subsequently making decisions based on the processed data. Therefore, it can be reasonably postulated that by augmenting the size of DNN, one can possibly enhance their cognitive capacities and enable them to handle voluminous data with greater proficiency. The present discourse highlights the remarkable surge in model parameters observed in the field of machine learning over the past decade. Specifically, the number of parameters in the models intensified from a mere 60 million in AlexNet in 2012 [12] to an astounding figure of 175 billion in GPT-3 that came into existence in 2020 [13]. Furthermore, it is projected that GPT-4 [14] possesses a parameter count of approximately 100 trillion [15].

The training of DNN is frequently depicted as an iterative endeavor, wherein the number of iterations is directly proportional to the quantity of data samples present in the training dataset. As indicated by the previously mentioned collection of datasets and the necessity of numerous epochs in the training of extensive models, it can be inferred that a scientist would be expected to undertake a significant quantity of iterations, potentially reaching hundreds of thousands [16]. One pertinent illustration is that the Transformer-large model necessitated a duration of 3.5 days to attain the essential level of accuracy [3]. One of the primary challenges in this particular field is to decrease the expenses associated with training, while also ensuring that the precision of the generated model is not significantly impacted [17]. Various endeavors have been devoted to the development of novel approaches and methodologies aimed at mitigating the expenses of training, chiefly the temporal outlays related to training. This paper offers a primary focus on Convolutional Neural Networks (CNN) in light of their substantial uptake in industry and a pressing need for novel advancements in this domain [18].

The present study warrants the following pivotal contributions. This research paper offers a comprehensive evaluation of current methods for training acceleration, emphasizing the use of lightweight algorithms and methods that facilitate seamless integration into current training protocols. This research presents the design of two innovative methods that aim to accelerate the training of CNN. These methods can be used independently, as well as in conjunction, to achieve higher levels of training acceleration as demonstrated in Figure 1. The proposed methods allow a reduction in the amount of data required to train a model of quality close to the baseline (full training). In addition, they are designed to scale to other CV and NLP tasks and to be combined with other methods, for example, fine-tuning [19]. The present study conducts a comparative analysis with contemporary, high-performance methodologies and establishes the superiority of the suggested methods over existing approaches, namely Selective Backpropagation (SB) [20] and Self-Supervised Prototypes (SSP) [21]. For example, our findings indicate that the proposed ADONIS methodology enables accelerated training, demonstrating speedups of 1.15–1.38× in comparison to SB.

The structure of the paper is as follows. Section 2 provides a comprehensive synopsis of current approaches to enhancing training speed. In Section 3, novel methods for expediting the process of training are presented. Section 4 provides an elaborate assessment of the proposed methods. Section 5 elucidates the findings of the study and underscores salient avenues for future inquiry.

2. Related Works

2.1. Problem Definition

To explicate diverse strategies for CNN training acceleration, we need to first introduce certain terms that will be used throughout the paper. Let $w\in {\mathbb{R}}^m$ represent the parameters of a model. Those weights are optimized to fit $f_w$ on a dataset $D=\{(x_i,y_i),i=1,\dots ,N\}$, where $x_i\in {\mathbb{R}}^d$ is a data sample and $y_i$ is a ground truth label. The model prediction is denoted as $f_w\left(x_i\right)$ and means the output of a model on a given data sample $x_i$. The objective of a DNN training is to minimize the loss $l(f_w\left(x_i\right),y_i)$ between predictions given by a model and ground truth values in a whole training dataset (1), where $F\left(x\right)$ is an empirical risk.

$$\underset{w}{min}F\left(w\right)=\frac{1}{N}\sum _{i=1}^{N}l(f_w\left(x_i\right),y_i)$$

(1)

Within DL field, gradient descent methods are widely adopted for training CNN in an iterative manner. After neural network parameters are initialized as $w^{\left(0\right)}$, they are updated with respect to the gradient of the loss function (2).

$$\begin{gathered} w^{(k+1)}=w^{\left(k\right)} - {\eta }_k{g}_k \\ {g}_k=\frac{1}{N}\sum _{i=1}^N{\nabla }_{w^{(k)}}l(f_w(x_i),y_i), \end{gathered}$$

(2)

where $\eta$ is the learning rate, k is the training iteration and $g_t$ is the gradient of loss. This update process happens unless either number of iterations exceeds the specified threshold or the model is converged so that $∥\begin{array}{c}{\nabla }_{w^{(k+1)}}l(f_w\left(x_i\right),y_i)-{\nabla }_{w^{\left(k\right)}}l(f_w\left(x_i\right),y_i)\end{array}∥<ϵ$, where $ϵ$ is a small constant.

An empirical approach is typically employed by authors of training pipelines to define the number of epochs, denoted by E, in which the training process is segregated for practical purposes. During each epoch, the data from the dataset D is segregated into smaller units of data called mini-batches, each comprising a quantity of n samples. During each epoch, the model parameters are updated a total of $\left[\frac{D}{n}\right]$ times. In summary, the majority of proposals related to training acceleration pertain to diminishing the overall training duration, which is necessary for achieving the convergence of a model or for completing a given number of epochs. This objective may be attained through the minimization of the duration of a single epoch or by ensuring an expedited convergence of the model. Simultaneously, the utilization of a novel training algorithm frequently leads to a reduction in accuracy in contrast to the level of accuracy achieved with a full baseline. In addition to the temporal resources necessary for training, an auxiliary metric for assessing novel training algorithms exists, which is accuracy reduction. It is maximum acceptable value is defined by project demands. For instance, it could be set at a maximum of 5% drop from the full training or even less.

2.2. Key Research Directions in the Field

There exist several methods that are targeted toward decreasing the time taken for training and accelerating the speed at which a model converges. However, it can be observed that the aforementioned methods approach this objective through varying vantage points. Therefore, they are often not listed together in dedicated surveys, for example, dataset distillation methods overview [22,23] does not consider distributed methods overview [24,25,26] and vice versa. In this section, we provide a succinct overview of the predominant research avenues in this domain.

2.2.1. Quantization and Pruning Methods

The practice of quantization is regarded as a prominent approach in expediting the training and inference of CNN. The method of quantization can be utilized to condense the model as a whole by lowering the accuracy of its parameters [27,28]. Alternatively, quantization can be applied to the gradients required for updating the parameters [29,30]. Moreover, the research on accelerating training encompasses model pruning as one of its focal points, wherein the elimination of model weights is determined by their impact on the accuracy of the resultant model [31,32].

2.2.2. Distributed Training

The second strategy involves the application of distributed computing concepts to the process of training DNN [24,25,26]. On one hand, a proliferation of novel DL chips and circuits, such as the Google Edge TPU [33], NVIDIA GPU [34], and Huawei Ascend [35], have emerged from various vendors. Conversely, contemporary methodologies capitalize on the innate parallelism exhibited by DNN. The training of DNN can be carried out through pipeline-parallel, model-parallel, or data-parallel methods [24,26]. However, the employment of distributed training is often associated with mounting training expenditures and potential security susceptibilities. Simultaneously, in computational settings comprising a group of low-power edge devices, commonly referred to as the Internet of Things (IoT) or decentralized systems, DL assumes paramount significance and, thus, merits extensive consideration [36,37].

2.2.3. Dataset Reduction

One of the cardinal aspects pertains to data selection, as the third direction in the study. As contemporary training datasets grow exponentially in size, the task of managing and processing them, as well as training CNN on such data, becomes increasingly arduous. Hence, one of the preeminent avenues of research pertains to the reduction of the dataset size. It can be posited that not all samples from a given dataset possess equal importance, thereby enabling their exclusion from training without any discernible effect on accuracy. Conversely, certain samples may necessitate multiple iterations to be correctly identified by a DNN [38,39].

The present study enumerates three principal approaches utilized in the field of data science, namely dataset condensation or distillation [22,40,41], core-set selection [17,42,43], and importance sampling [20,44,45,46]. The principal disparity pertains to the definition of what constitutes a reduced version of the training dataset. Data distillation methods propose producing instances as synthesized entities based on certain statistics derived from the initial dataset. Core-set selection methods enable the automated identification of a subset from the initial dataset, such that the resultant model achieves a level of accuracy comparable to the baseline, within a specified threshold, typically 5%. The utilization of core-set selection and dataset distillation methods is primarily intended for active learning scenarios rather than the acceleration of training. The primary objective of active learning is to identify and designate pertinent data samples for the purpose of training or refining DNN to optimize their precision. The selection of data is constrained by an upper limit, often referred to as a budget). Moreover, the quality of the data chosen is directly proportional to the quality of the fine-tuned model. The data selection direction encompasses a third approach that involves the utilization of importance sampling. This approach aims to select samples that are more likely to bring about significant updates to the weights during (online) training and thereby impart greater informational value to the process of CNN training [44,45]. Selection is executed based on knowledge obtained from the presently trained model via additional forward propagation on novel data, thereby gathering data on said samples. Nevertheless, these methods introduce supplementary costs to the training procedure, which may involve conducting an additional forward pass for all instances within a given dataset [44], or deploying a separate network in parallel [45]. Two of the most notable methods that rely on this concept include DL with Importance Sampling [46], and Selective Backpropagation (SB) [20].

2.2.4. New Training Strategies

The fourth aspect of training acceleration pertains to expediting the training process by means of novel training methodologies. In the realm of this particular field, numerous novel concepts and auspicious outcomes have been observed. An illustrative instance pertains to the substitution of backpropagation with layer-wise updates for the purpose of deep supervised hashing [47]. An alternative method involves utilizing the Koopman operator theory to simulate the dynamics of weight updates in DNN, as opposed to relying on gradient descent methods [48]. Considerable attention has been directed toward the application of the Hebbian learning rule and its adaptations for training CNN [49,50]. To this end, mixed strategies (SGD + Hebbian learning) have been introduced and have demonstrated training speedup of up to 1.5× [51]. Additionally, implementation optimizations have been proposed in the literature [52,53,54]. Hebbian winner-take-all models are deemed feasible for neuromorphic computing in light of their characteristic of local and unsupervised weight update rule [55,56]. In contrast to previously mentioned approaches, the Hebbian learning method has demonstrated reasonable levels of convergence when used for training on large datasets such as ImageNet [57,58], while also exhibiting training time comparable to backpropagation [59]. Despite the progress made in the field, opportunities remain for improvement. One such area pertains to the lack of comprehensive assessments concerning the generability and efficacy of mixed strategies in the context of CNN training.

In conclusion, the field of training acceleration presents a multitude of potential avenues for exploration and advancement. This study presents novel methodologies that facilitate the expeditious training of CNN, which are hardware agnostic, implying that they can be implemented across a diverse range of hardware types and quantities. Both methods have been thoroughly described in Section 3.

3. Proposed Methods for Training Acceleration

3.1. Adaptive Online Importance Sampling

One of the important ideas is to reduce overall training time by decreasing the number of training steps through the implementation of specific rules. The current status of advancement in this domain has been accomplished through the utilization of a Selective Backpropagation (SB) approach [20]. This method exhibits superior performance over alternative methods [46]. This is due to its ability to operate independently of the loss distribution within a batch and its insensitivity to variable starting conditions. The present paper proposes a novel methodology, referred to as Adaptive Online Importance Sampling, which draws on various concepts and principles from SB. Our research findings, as elucidated in Section 4, demonstrate the superior performance of our methodology in comparison to SB.

$$choose(p,\theta )=\left\{\begin{array}{c}1,p>\theta \\ 0,p\le \theta \end{array}\right.$$

(3)

The proposed design of Adaptive Online Importance Sampling is elucidated in Algorithm 1. The present approach is consonant with existing online importance sampling methodologies and endeavors to curtail the frequency of backward passes. The aforementioned accomplishment is carried out through the selection of samples for these passes from the pool of obtainable training data, and the creation of fresh training batches that exclusively comprise the chosen elements. The act of executing a model on a designated training subset is commonly referred to as a forward pass, while a backward pass entails updating the neural network based on gradients obtained for the applicable training subset.

In the context of training a model, forward passes are executed on a set of input data that represents a specific subset of the training dataset with a defined size denoted as $\rho$, referred to as the original batch. Backward passes are executed on batches of data specifically designated for training purposes. In the absence of importance sampling or selection rules applied to an initial batch, the original batch and the subsequent training batch will represent identical sets of data samples.

Algorithm 1 Adaptive Online Importance Sampling training algorithm

1: $H\leftarrow deque\left(\gamma \right)$: history of losses as double-ended queue of size $\gamma$ 2: $batc{h}_{train}\leftarrow \left[\right]$: batch for CNN training, $|batc{h}_{train}|=\rho$ 3: $\eta ,\eta \in \mathbb{N},\eta \ge 1$: number of epochs when loss information is considered actual 4: $losses\leftarrow \left[\right]$: loss history 5: E: number of epochs to train 6: $\omega$: number of epoch to start ADONIS 7: $last\_update\leftarrow 0$: number of last epoch when loss history was updated 8: $k\leftarrow 1$: CNN training step 9: while epoch $i<E$ do 10:   for batch $b_j$ from D do 11:    if $i\le \omega$ then 12:     $batc{h}_{train}\leftarrow b_j$ 13:     $f_{w^{\left(k\right)}}\leftarrow trainBatch(b_{train},f_{w^{(k-1)}})$ 14:     $k\leftarrow k+1$ 15:     $batc{h}_{train}\leftarrow \left[\right]$ 16:     continue 17:    end if 18:    if $i-last\_update<\eta$ then 19:     $losses\leftarrow f_{w^{\left(k\right)}}\left(j\right)$ 20:    end if 21:    for example e from $b_j$, loss l from $losses$ do 22:     $H\leftarrow H\cup l$ 23:     $\Phi \leftarrow buildHistogram\left(H\right)$: losses histogram representing losses 24:     $p\leftarrow PMF(\Phi ,l)$: give preference to the most frequent losses 25:     $is\_chosen\leftarrow choose\left(p\right)$: defined in (3) 26:     if $is\_chosen=1$ then 27:      $batc{h}_{train}\leftarrow batc{h}_{train}\cup e$ 28:     end if 29:     if $|batc{h}_{train}|=\rho$ then 30:      $f_{w^{\left(k\right)}}\leftarrow trainBatch(b_{train},f_{w^{(k-1)}})$ 31:      $batc{h}_{train}\leftarrow \left[\right]$ 32:     end if 33:    end for 34:   end for 35: end while

The process of selecting samples for backward passes is probabilistic in nature. Each sample is accompanied by a loss value, and its selection probability $p_i=P\left(l(f_w\left(x_i\right),y_i)\right)$ is determined based on this metric before being ultimately selected. To calculate the losses for every sample present in the initial batch, a forward pass is executed, during which $l(f_w\left(x_i\right),y_i)$ is computed. To generate each training batch of a predetermined size, denoted as $\rho$, for use in the backward process, it is necessary to perform forward passes on multiple training batches of size $\rho$. After the accumulation of the complete training batch, a singular backpropagation iteration is executed.

The probability of single sample selection denoted by $p_i$ is derived from the Probability Mass Function (PMF) for losses incurred for every training sample. As the precise distribution of losses cannot be predetermined, an effective approach is to construct an approximation using a histogram. The initial phase involves the construction of a histogram. The number of bins is defined in accordance with the Sturges rule [60] (4). It is postulated that all bins have an equal width that is equivalent to h.

$$k=⌈{log}_{2}n⌉+1,$$

(4)

where k is the number of bins in a histogram. Let sample $x_i$ results in loss $l(f_w\left(x_i\right),y_i)$. Let this loss be placed into j-th bin with a width of h and the corresponding bin density is $d_j$. Then a probability value that corresponds to this sample will be equal to $⌈\frac{d_j}{L}⌉\times h$.

Pertaining to the sample selection strategy, the mechanics thereof entail the evaluation of each original batch employing a contemporaneous state of a model that has undergone an update using gradients derived from the antecedent training batch. The aforementioned statement implies that the probabilities of samples lose their relevance as a result of weight adjustments in the model. Consequently, the model is capable of discerning distinctions between the current and prior states of a sample. Consequently, the aforementioned task necessitates the computation of selection probabilities for the entirety of the training data. With respect to the extensive amount of training data, this process entails a substantial computational burden that in turn mitigates the potential advantages associated with the reduction of backpropagation passes. As per the guidelines put forth in SB [20], it is preferable to adopt an approach wherein the re-calculation of PMF after each update (5) is replaced with an approximation achieved by maintaining a running tally of the losses of last R samples (6).

$$P_{theoretical}\left(l(f_w\left(x_i\right),y_i)\right)=PMF\left(l(f_w\left(x_i\right),y_i)\right)$$

(5)

$$\begin{array}{cc}\hfill P_{approximate}\left(l(f_w\left(x_i\right),y_i)\right)& =PM{F}_R\left(l(f_w\left(x_i\right),y_i)\right)=\frac{fre{q}_j}{R},\hfill \\ \hfill \mathrm{s}.\mathrm{t}.j& \leftarrow \Phi \left(l(f_w\left(x_i\right),y_i)\right),\hfill \end{array}$$

(6)

where $\Phi$ is a histogram function that maps the given loss value to the bin number that corresponds to it.

For a better understanding of ADONIS dynamics, it is important to inspect selected samples through the training process. This is further demonstrated by comparing selected samples during ResNet18 training on CIFAR-100 at 15th and 190th epochs. To begin with, the loss history is updated with every forward batch (Figure 2).

After that, the new state of the loss history is analyzed from the perspective of loss distribution and clipped due to a certain amount of standard distribution $\sigma$, as depicted in Figure 3.

During the next step, a histogram is built from the current loss history and only bins that exceed the given probability threshold $\theta$ are accepted for the selection (Figure 4).

Finally, as illustrated in Figure 5, only samples that correspond to accepted bins are selected from a batch and are put into the training batch that is later used for backward step.

There are three distinct variances between the revised algorithm and its original counterpart. Initially, it is suggested to employ the novel $\omega$ parameter prior to transitioning to the training phase utilizing ADONIS (Algorithm 1). The second concept pertains to the computation of probabilities via the PMF in contrast to the initial proposition of utilizing the Cumulative Distribution Function (CDF). The proposed approach centers on prioritizing the most commonly occurring samples over the anomalous ones that incur the greatest losses because they might not only represent “hard” samples, but also wrongly annotated samples and outliers. The third approach involves the utilization of a deterministic probability threshold in the selection phase (3). This method stands in contrast to the probabilistic mode employed by the original SB. Given that ADONIS represents a notable advancement beyond the SB algorithm, it is of considerable import to elucidate the pertinent contrasts vis-à-vis extant implementations. There exist two open-source implementations: the first is the source code of the original paper as per reference [61], whereas the second is a constituent of the Composer library offering as per references [62,63]. The implementation of the Composer algorithm is not in line with the SB algorithm. Specifically, it utilizes a fixed number of chosen samples per batch, the selection is solely executed on the highest layer of a given batch, and it disregards loss history. Moreover, the computations for selection probabilities are not contingent on the actual loss values, while backpropagation is executed for each batch despite the fact that it may not be complete, resulting in costly additional backpropagation passes from a computational standpoint.

3.2. Intellectual Data Selection

A conceptual notion for expediting the period of model training involves minimizing the number of samples before training actually starts. The act of diminishing the number of samples present in the training dataset results in a reduction of the overall epoch time and not a decrease in the step time. Such a reduction in sample quantity can potentially lead to optimization processes achieving better computational efficiency. The rationale behind the presence of redundant information in datasets has been extensively explored and documented in scholarly literature [20,21,64]. The identification of a core set for a particular dataset presents various obstacles, such as clearly defined selection criteria, restricted time allocations for the selection process, and the requirement for proof of convergence [17].

The proposed approach for the filtration of training datasets involves the curation of a diverse set of samples from every class represented in a dataset. To assess diversity among samples, it becomes indispensable to establish a distance metric denoted as $d(·,·)$. Notably, the utilization of Euclidean distance in the process of image comparison has been shown to yield inaccurate outcomes [65] if it is applied straightforwardly to images. In other words, when the distance of two dissimilar images is measured with Euclidean metric between corresponding tensors they might be considered to be close. For a comprehensive list of distance metrics applicable to grayscale images, readers are advised to consult the work of [66]. Despite the widespread use of metrics such as Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity Index (SSIM) for image similarity measurement, these metrics have been reported to be inadequate for accurately reflecting human assessments of the similarity between images [67]. This is why in this paper we cultivate the idea of first projecting images to the latent space and calculating the distance between them as the distance between images with the help of Euclidean distance as a metric.

On the contrary, empirical evidence has shown that the image comparison that utilizes DL features derived from pre-trained models exhibits superior performance when viewed with regard to perceptual similarity, surpassing other metrics [67]. It is hereby recommended to commence the projection of dataset samples into a space of Z dimensions by utilizing the attributes of a pre-trained feature extractor, which will serve as the embedding function denoted as $\varphi :D\to {\mathbb{R}}^Z$. The present study has conducted experimentation on ResNet18 to investigate its efficacy. Upon projection into the latent space (Figure 6a), image similarity can be assessed by determining the Euclidean distance separating their respective embeddings as depicted in Equation (7).

$$d^2(x,y)=\sum _{k=1}^Z{(x^{\left(k\right)}-y^{\left(k\right)})}^{2}x\in {\mathbb{R}}^Z,y\in {\mathbb{R}}^Z$$

(7)

Given the inherent structure of classification datasets, we posit the presence of duplicative information within each class, which may be removed without substantial impact on the ultimate precision of the analysis. As shown in [38], a conservative estimate suggests that a minimum of 10% of sample instances in both CIFAR-10 and ImageNet 2012 datasets possess such qualities. In one paper [38], it was proposed to utilize Hierarchical Clustering as a means to identify and group similar samples, ultimately enabling the removal of all but one representative from each grouping.

Accordingly, the identification of the prototype of the group is a prerequisite for the elimination of any individual cases. Considering that every sample is depicted in a latent space, there might exist diverse strategies to discover the aforementioned prototype. It is recommended to employ the K-Means clustering algorithm for the purpose of discerning class centroids. One primary motivation for employing K-Means clustering as opposed to the hierarchical approach stems from its potential applicability in the context of active learning. In particular, when novel samples emerge over time, it becomes crucial to assess their potential value for engaging in supplementary training processes [18].

After centroids are detected for each class (Figure 6b), we identify samples that should be selected according to the selection ratio $\alpha \in (0,1]$. In case $\alpha =1$, training is run on the original dataset. Let all distributed representations of samples in the i-th class $D^i=\{{\xi }_1^i,{\xi }_2^i,\dots ,{\xi }_L^i\}$, where L denotes number of samples in given class, be sorted in descending order according to the Euclidean distance to the centroid. Then, we select the first $\alpha L$ samples from $D^i$. In addition, we insist on adding the last sample to this list, as it plays the role of the prototype for all samples removed from a class. As a result, we obtain a subset of the original dataset within a given selection ratio. The formal description of the proposed method is demonstrated in Algorithm 2 and Figure 6.

Algorithm 2 Intellectual Data Selection (IDS) algorithm

1: $Mode$: mode of selection {EASY, HARD} 2: $\alpha$: selection ratio 3: $\varphi$: embedding function 4: $Q=\left\{\right\}$: final dataset 5: for each class $D^i$ in D do 6:   $L^i=\left|D^i\right|$: number of samples in class 7:   ${\Xi }^i=\left\{\mathbf{inf}\right\},\left|{\Xi }^i\right|=L^i$: distributed representations of samples in $D^i$ 8:   for each sample j in $D^i$ do 9:    ${\xi }_j^i\leftarrow \varphi \left(j\right)$: obtain distributed sample representation 10:   end for 11:   ${\lambda }^i\leftarrow KMeans\left({\Xi }^i\right)$: find centroid of a cluster, i.e., class 12:   ${\Theta }^i=\left\{\mathbf{inf}\right\},\left|{\Xi }^i\right|=L^i$: distances from samples to centroid ${\lambda }^i$ 13:   for each sample j in $D^i$ do 14:    ${\theta }_j^i\leftarrow d(j,{\lambda }^i)$: calculate distance to centroid 15:   end for 16:   $O^i\leftarrow sort(\Theta ,Mode)$: sort samples w.r.t to their cluster centroid in ascending order if mode is EASY, descending otherwise 17:   $Q\leftarrow Q\cup \{O_j^i:j=1,2,\dots ,\lceil \alpha L^i\rceil \}\cup \left\{O_{{\mathbf{L}}^{\mathbf{i}}}\right\}$: add selected samples to final dataset (including cluster prototype) 18: end for

The key dissimilarity between IDS and SSP is performing clustering within individual classes as opposed to enumerating a predetermined number of clusters and conducting the operation on the entire training dataset. This method facilitates the prevention of imbalanced class selection scenarios and eliminates the requirement for the number of clusters as an algorithmic hyperparameter. The present proposal advocates the utilization of the Euclidean metric in lieu of the cosine similarity method as originally posited in the SSP paradigm.

4. Results

In the context of experimental evaluation, we employed four CV datasets: CIFAR-100 [68], CIFAR-10, ImageNet 2012 and MS COCO 2017 which are frequently utilized for the purpose of benchmarking the efficacy of training acceleration algorithms [20,22,51]. CIFAR-100 and CIFAR-10 datasets are partitioned such that there are 50 K images available for training purposes, while 10 K images are allotted specifically for testing. For CIFAR-100 there are 100 classes each containing 600 distinct 32 × 32 RGB examples, for CIFAR-10—10 classes with 6000 distinct 32 × 32 RGB examples. ImageNet 2012 consists of RGB images from 1000 different classes: 1.28 M for training and 50 K for validation. MS COCO 2017 is a dataset and consists of 118 K of training and 5K of validation RGB samples for object detection task, where the number of classes is equal to 80. When training on datasets the following common settings were used. For training on ImageNet 2012: mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225] were used for samples normalization. For CIFAR-100: mean = [0.5070751592371323, 0.48654887331495095, 0.4409178433670343], std = [0.2673342858792401, 0.2564384629170883, 0.27615047132568404], for CIFAR-10: mean = [0.49139968, 0.48215827, 0.44653124], std = [0.24703233, 0.24348505, 0.26158768] for. The number of training epochs on ImageNet 2012 is 90, on CIFAR-100 and CIFAR-10—200, on MS COCO 2017—100.

With regards to the models that are currently under evaluation, our focus is 4 CV models: ResNet18 [69], ResNet50, MobileNet v2 [70], YOLO v5. The implementation of ResNet18, ResNet50 and MobileNet v2 and training pipelines for CIFAR-10 and CIFAR-100 datasets were taken from the official repository (https://github.com/weiaicunzai/pytorch-cifar100 (accessed on 15 June 2023)). Implementation of ResNet18, ResNet50 and MobileNet v2 for ImageNet 2012 was taken from torchvision library [71] v0.14.0. Training pipeline for YOLO v5 (yolov5m) training on MS COCO 2017 was used from ultralytics (https://github.com/ultralytics/yolov5 (accessed on 15 June 2023)) GitHub repository. For the needs of IDS, the pre-trained ResNet18 neural network implemented in the torchvision library is utilized as the embedding function. This involves the extraction of the 512-dimensional output from the penultimate layer. The weights for this model were obtained as a result of ResNet18 training on the ImageNet 2012 dataset [72] and have been made publicly available in the official repository.

The training protocol for ResNet18, ResNet50 and MobileNet v2 on CIFAR-100 and CIFAR-10 is as follows: the selected loss function is Cross Entropy, the optimizer is Stochastic Gradient Descent with a learning rate equal to 0.1, momentum is 0.9, and weight decay is 5 × 10${}^{-4}$. The learning rate is dynamically reduced by a gamma equal to 0.2 during training at specified milestones: 60th, 120th, and 160th epochs. The data are loaded with 32 workers with a batch size equal to 128, randomly shuffled at each epoch. Each sample is processed with the following pipeline: random crop to size equal to 32 with padding equal to 4, then random horizontal flip, random rotation by angle, and sample normalization.

The training protocol for ResNet18 and ResNet50 on ImageNet 2012 is as follows: the selected loss function is Cross Entropy, the optimizer is Stochastic Gradient Descent with a learning rate equal to 0.1, momentum is 0.9, and weight decay is 1 × 10${}^{-4}$. The learning rate is dynamically reduced by a gamma equal to 0.1 at each 30th epoch. The data are loaded with 32 workers with a batch size equal to 256, randomly shuffled at each epoch. Each sample is processed with the following pipeline: random crop to size equal to 224 with bilinear interpolation, then random horizontal flip and sample normalization.

The training protocol for MobileNet v2 on ImageNet 2012 is as follows: the selected loss function is Cross Entropy, the optimizer is Stochastic Gradient Descent with a learning rate equal to 0.045, momentum is 0.9, and weight decay is 4 × 10${}^{-5}$. The learning rate is dynamically reduced by a gamma equal to 0.98 at each training epoch. The data are loaded with 32 workers with a batch size equal to 256, randomly shuffled at each epoch. Each sample is processed with the following pipeline: random crop to size equal to 224 with bilinear interpolation, then random horizontal flip and sample normalization.

The training protocol for YOLO v5 on MS COCO 2017 is as follows: the selected loss is the cumulative sum of bounding box regression, classification, and confidence losses, the optimizer is Stochastic Gradient Descent with a learning rate equal to 0.01, momentum is 0.937, and weight decay is 4 × 10${}^{-5}$. The learning rate is dynamically reduced by a gamma equal to 0.98 at each training epoch. The data are loaded with eight workers with a batch size equal to 16, randomly shuffled at each epoch. All input samples during YOLO v5 training are resized to 640 pixels.

The overall dynamics of training is demonstrated as an example for ResNet18 training on CIFAR-100 in Figure 7. The present study employed various methodologies and their corresponding configurations to conduct a comparative analysis. The system under investigation was configured to include the SSP, Random Per Class (RPC), Random Per Dataset (RPD), ADONIS, and IDS. RPC means offline a reduction in the dataset by keeping only $\alpha$ samples from each class. RPD means an offline reduction in the dataset by keeping only $\alpha$ samples from the whole dataset regardless of the class it belongs to. The SB was initialized with the standard parameters of $\beta =2$, $\gamma =1024$, $staleness=2$.

All training experiments were performed on a single GPU device in mixed precision (full and half-precision floating point format) [73]. Certain hardware optimizations were enabled: pinning memory for data loaders (pin_memory = True) and benchmarking mode (cudnn.benchmark = True). For training evaluation experiments, the following hardware was used: CPU is Intel(R) Xeon(R) Gold 6151 CPU @ 3.00 GHz with 32 cores (frequency not fixed), OS: Ubuntu 20.04 LTS, kernel version: 5.4.0-136-generic; GPU is 1x NVIDIA Tesla V100 PCIe 16GB, driver version: 470.129.06, CUDA: 11.7; training framework: PyTorch 1.13.0, programming language: Python 3.9. K-Means algorithm implementation is taken from scikit-learn library v1.1.2. Additional libraries: imageio library v2.26.0, numpy library v1.21.5, opencv-python library v4.7.0.72, Pillow library v9.3.0, pycocotools library v2.0.6.

The present study showcases a comprehensive evaluation of the efficacy of training acceleration methods, as illustrated in

Table A1. Evaluation of training acceleration for selected computer vision models on ImageNet 2012, CIFAR-100, CIFAR-10 and MS COCO 2017. All results are grouped in sections, each section stands for a unique combination of a model and a dataset. Bold represents a result that is the best across given range of methods within a section.

Dataset	Model	Method	Acc. Drop (Full Train), p.p.	Train Boost (Full Train), x	Boost to 5 p.p. Drop, x
CIFAR-10	ResNet18	IDS ($\gamma =180$)	0.72	1.21	3.91
		ADONIS ($\eta =1$)	0.76	1.28	4.0
		ADONIS ($\eta =2$)	0.77	1.41	4.05
		ADONIS ($\eta =3$)	0.55	1.46	4.35
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	1.27	1.58	4.08
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	1.05	1.75	4.87
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	1.29	1.81	4.99
		SSP	0.65	1.24	3.97
		RPC	0.74	1.23	3.95
		RPD	0.53	1.23	3.93
CIFAR-10	ResNet50	IDS ($\gamma =180$)	0.45	1.22	4.01
		ADONIS ($\eta =1$)	0.97	1.35	3.77
		ADONIS ($\eta =2$)	0.99	1.52	4.07
		ADONIS ($\eta =3$)	0.98	1.69	4.49
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	1.62	1.59	2.62
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	1.69	1.76	2.79
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	1.65	1.93	3.05
		SSP	0.78	1.18	3.79
		RPC	0.38	1.18	3.87
		RPD	0.58	1.19	3.95
CIFAR-10	MobileNet v2	IDS ($\gamma =180$)	−0.24	1.27	1.72
		ADONIS ($\eta =1$)	0.44	1.39	1.86
		ADONIS ($\eta =2$)	0.19	1.59	2.08
		ADONIS ($\eta =3$)	0.4	1.68	2.18
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	1.04	1.75	2.31
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	1.17	1.96	2.56
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	0.96	2.04	2.66
		SSP	0.55	1.28	1.73
		RPC	0.23	1.19	1.62
		RPD	0.29	1.13	1.53
CIFAR-100	ResNet18	IDS ($\gamma =120$)	2.12	1.15	1.9
		ADONIS ($\eta =1$)	2.76	1.16	1.83
		ADONIS ($\eta =2$)	2.5	1.23	1.97
		ADONIS ($\eta =3$)	2.66	1.27	2.03
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	3.55	1.58	2.5
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	3.26	1.71	2.72
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	3.26	1.75	2.76
		Vanilla SB	2.43	0.92	2.84
		Vanilla Stale-SB	2.81	1.1	1.8
		SSP	2.32	1.14	3.63
		RPC	2.49	1.29	2.12
		RPD	2.43	1.16	1.91
CIFAR-100	ResNet50	IDS ($\gamma =120$)	1.88	1.22	1.85
		ADONIS ($\eta =1$)	2.2	1.32	1.96
		ADONIS ($\eta =2$)	2.36	1.56	2.28
		ADONIS ($\eta =3$)	2.19	1.63	2.37
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	4.24	1.61	2.21
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	4.65	1.89	2.1
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	4.92	1.9	2.08
		SSP	2.74	1.17	1.76
		RPC	2.77	1.18	1.79
		RPD	2.13	1.18	1.78
CIFAR-100	MobileNet v2	IDS ($\gamma =120$)	1.82	1.3	1.73
		ADONIS ($\eta =1$)	1.9	1.54	1.94
		ADONIS ($\eta =2$)	1.87	1.7	2.14
		ADONIS ($\eta =3$)	1.37	1.77	2.23
		IDS ($\gamma =120$) + ADONIS ($\eta =1$)	2.77	1.78	1.96
		IDS ($\gamma =120$) + ADONIS ($\eta =2$)	3.41	1.99	2.05
		IDS ($\gamma =120$) + ADONIS ($\eta =3$)	3.7	2.08	2.06
		SSP	0.96	1.27	1.69
		RPC	1.59	1.21	1.6
		RPD	1.63	1.21	1.61
ImageNet 2012	ResNet18	IDS ($\gamma =14$)	1.2	1.22	1.72
		ADONIS ($\eta =1$)	2.55	1.1	1.53
		ADONIS ($\eta =2$)	2.39	1.11	1.54
		ADONIS ($\eta =3$)	2.51	1.11	1.54
		IDS ($\gamma =14$) + ADONIS ($\eta =1$)	3.64	1.63	2.23
		IDS ($\gamma =14$) + ADONIS ($\eta =2$)	3.84	1.66	2.25
		IDS ($\gamma =14$) + ADONIS ($\eta =3$)	3.74	1.74	2.35
		SSP	4.41	1.22	1.52
		RPC	4.64	1.22	1.51
		RPD	4.52	1.22	1.52
ImageNet 2012	ResNet50	IDS ($\gamma =14$)	1.12	1.27	1.81
		ADONIS ($\eta =1$)	2.18	1.48	2.02
		ADONIS ($\eta =2$)	2.4	1.71	2.27
		ADONIS ($\eta =3$)	2.45	1.82	2.37
		IDS ($\gamma =14$) + ADONIS ($\eta =1$)	3.48	1.77	2.37
		IDS ($\gamma =14$) + ADONIS ($\eta =2$)	3.43	2.17	2.88
		IDS ($\gamma =14$) + ADONIS ($\eta =3$)	3.5	2.31	3.05
		SSP	4.03	1.23	1.67
		RPC	4.22	1.25	1.62
		RPD	4.17	1.28	1.66
ImageNet 2012	MobileNet v2	IDS ($\gamma =14$)	0.61	1.25	2.61
		ADONIS ($\eta =1$)	2.25	1.6	2.46
		ADONIS ($\eta =2$)	1.74	1.72	2.66
		ADONIS ($\eta =3$)	1.91	1.87	2.99
		IDS ($\gamma =14$) + ADONIS ($\eta =1$)	3.36	1.95	2.62
		IDS ($\gamma =14$) + ADONIS ($\eta =2$)	3.36	2.19	2.82
		IDS ($\gamma =14$) + ADONIS ($\eta =3$)	3.38	2.3	2.95
		SSP	0.33	1.24	2.85
		RPC	0.73	1.25	2.45
		RPD	0.83	1.25	2.69
MS COCO 2017	YOLO v5	IDS ($\gamma =60$)	0.76	1.19	2.41
		RPD	0.86	1.23	2.46

for ImageNet 2012, CIFAR-100, CIFAR-10, employing the aforementioned configurations. The quantification of the decrease in accuracy, commonly referred to as Accuracy drop, is determined in absolute percentage points for each method in relation to its corresponding baseline. The present study entails a comparative analysis of the efficiency of several algorithms, namely IDS, SSP, Random, and ADONIS, against the default baseline. Additionally, the performance of the Vanilla SB algorithm is assessed against its eponymous baselines. These findings are presented in

Table 1. Evaluation of baseline training.

Model	Dataset	Acc.	Train Time, s
ResNet18	CIFAR-10	95.09	2518
	CIFAR-100	76.43	2540
	ImageNet 2012	70.02	55,943
ResNet50	CIFAR-10	95.1	5913
	CIFAR-100	79.06	5910
	ImageNet 2012	75.6	153,409
MobileNet v2	CIFAR-10	90.89	4074
	CIFAR-100	68.26	4173
	ImageNet 2012	68.04	94,470
YOLO v5	MS COCO 2017	42.55	110,581

. Boost is determined by taking the ratio of the duration of training with the standard baseline approach to the duration of training achieved when utilizing an acceleration algorithm during training of ResNet18.

The best results for training research models on CIFAR-10, CIFAR-100, ImageNet 2012 and MS COCO 2017 are shown in Table A1. Application of Adaptive Online Importance Sampling to ResNet50 training on ImageNet 2012 results in 2.37 times quicker convergence to 71.7% top-1 accuracy, which is within 5% from baseline. Total training time for the same number of epochs as in baseline is reduced by 1.82 times, with an accuracy drop of 2.45 p.p. The amount of time required to apply Intellectual Data Selection to ResNet50 training on ImageNet 2012 is decreased by 1.27 times with a corresponding decline in accuracy of 1.12 p.p. Applying both methods to ResNet50 training on ImageNet 2012 results in 2.31 speedup with an accuracy drop of 3.5 p.p.

It is imperative to acknowledge that approaches geared toward reducing datasets, such as SSP and IDS, necessitate preliminary preprocessing of the dataset. The processing time required by the SSP algorithm for handling CIFAR-100 amounts to 66.10 s, while it takes 28.82 s to process with IDS. For processing the CIFAR-10 dataset the SSP requires 31.90 s, while IDS will cope in 20.31 s. The SSP preprocessing for the ImageNet 2012 dataset will take 15,201.03 s, while IDS requires 10,019.81 s. Unlike SSP, IDS is able to process MS COCO 2017 in 1983.51 s. Evidently, the aforementioned duration is not encompassed within the documented period of training.

Furthermore, we present empirical evidence of enhanced model convergence through an evaluation of the rate at which a model trained using the acceleration method attains particular accuracy thresholds when compared to the ultimate baseline accuracy. As an illustration, ADONIS with a parameter value of $\eta =2$ exhibits an acceleration ratio of 2.37 when subjected to a 5% reduction in accuracy. The results indicate that the utilization of the ADONIS algorithm during training of ResNet18 results in an achieved accuracy of 71.53%, which is found to deviate by no more than 5% from the final accuracy obtained via a default baseline. Furthermore, this approach yields a notable speedup of 2.37 compared to models that were trained without the incorporation of the ADONIS algorithm. The per-sample analysis is demonstrated in Figure 8.

Beyond that, the results of measurements of IDS of object detection are presented in Table A1 and Figure 9. To apply IDS approach to the MS COCO 2017 dataset, where each sample could contain objects from different classes, we proposed to define one prototype in the entire dataset results show the possibility of reducing 20% of the MS COCO 2017 dataset with a mAP reduction of 0.76 p.p.

5. Discussion

5.1. Summary

In the present study, we introduce two novel methods: Adaptive Online Importance Sampling (ADONIS), and Intellectual Data Selection (IDS). The acceleration of training using the IDS approach is accomplished through the elimination of semantic redundancies within the training dataset. This results in a reduction in the number of training samples utilized for the study. Additionally, this approach involves the coordination of training procedures through the rigorous categorization of sample sets into distinct groups, namely those of easy and hard difficulty levels. ADONIS—a novel algorithmic approach—offers acceleration capabilities by means of minimizing the frequency of forward and backpropagation iterations observed during the training phase of DNN. The extent to which a given model is unable to appropriately identify certain data samples serves as the determining factor for the optimization measures imposed by ADONIS.

The proposed methods allow reducing the amount of data required to train a model of quality close to the baseline (full training), might be easily scaled to other CV and NLP tasks, and can be combined with other methods, for example during model fine-tuning [19].

5.2. Limitations and Outlook

One of the limitations of the current study pertains to the absence of extensive experimentation to establish the viability of the proposed methods on other CV tasks such as semantic segmentation. Addressing this limitation would pave the way for future research to leverage the proposed methods for scaling up image processing algorithms. It is of utmost significance to appropriately tailor the suggested methodologies to the sphere of NLP, given its emergence as one of the most arduous subdomains of DL. The application of distinct pre-trained models for feature extraction and various distance metrics for the training dataset pruning process represents a promising approach to enhance the performance of the IDS. Distance metric selection constitutes a pivotal area of investigation within our research endeavor.

There are several potential areas for improvement in ADONIS. One of the major concepts is to eliminate the extra step required to calculate selection probabilities and choose samples. By increasing the batch size in the selection process, this supplement might be lessened. The second strategy to enhance ADONIS involves computing sample probability using a weighted histogram. The overall idea is to allow a bigger impact on the most recent losses as they are obtained with a more up-to-date model. Third, it is possible to eliminate extra computations of losses that are duplicated for each sample during selection and backward passes. Reusing calculated loss might significantly increase ADONIS speed as additional forward passes and losses computations make up the majority of the selection process.

We define the following development horizons from the perspective of IDS improvement. The first direction is linked to the change from looking for one prototype per class to looking for several prototypes. This makes it possible to identify unused samples more precisely and prevent accuracy loss. Analyzing the capacity to alter the dataset more than once using a particular set of $\gamma$ parameters is another topic of inquiry. We can decrease the forgetting component of the model, which can boost the resulting accuracy, by repeatedly swapping the dataset between “easy” and “hard” examples. Finding prototypes without using the clustering approach is one of the intriguing areas for the development of the IDS.

In conclusion, the presented methodologies exhibit significant promise in enhancing the efficiency of CNN training through the reduction of training data prior to and during the training phase while surpassing current solutions in this domain.