PARS: Proxy-Based Automatic Rank Selection for Neural Network Compression via Low-Rank Weight Approximation

Abstract

Low-rank matrix/tensor decompositions are promising methods for reducing the inference time, computation, and memory consumption of deep neural networks (DNNs). This group of methods decomposes the pre-trained neural network weights through low-rank matrix/tensor decomposition and replaces the original layers with lightweight factorized layers. A main drawback of the technique is that it demands a great amount of time and effort to select the best ranks of tensor decomposition for each layer in a DNN. This paper proposes a Proxy-based Automatic tensor Rank Selection method (PARS) that utilizes a Bayesian optimization approach to find the best combination of ranks for neural network (NN) compression. We observe that the decomposition of weight tensors adversely influences the feature distribution inside the neural network and impairs the predictability of the post-compression DNN performance. Based on this finding, a novel proxy metric is proposed to deal with the abovementioned issue and to increase the quality of the rank search procedure. Experimental results show that PARS improves the results of existing decomposition methods on several representative NNs, including ResNet-18, ResNet-56, VGG-16, and AlexNet. We obtain a $3\times$ FLOP reduction with almost no loss of accuracy for ILSVRC-2012ResNet-18 and a $5.5\times$ FLOP reduction with an accuracy improvement for ILSVRC-2012 VGG-16.

1. Introduction

Over the last decade, deep neural networks (DNNs) have shown superior performance in various ML tasks. However, their deployment in real-world practical applications is constrained by their high latency, high memory requirements, and energy consumption. Compressing a heavy pre-trained model is a straightforward approach to overcoming these constraints. One of the most effective approaches is a low-rank matrix/tensor approximation of the kernel weights in neural network (NN) layers by exploiting the fact that they lie in a low-rank linear space [1].

Despite the efficiency of low-rank tensor decomposition methods, their application to NN compression requires a great effort to find optimal ranks that balance a trade-off between model performance and resource consumption. Tensor rank determination problems are, in general, NP-hard [2]. The problem is complicated because influence of layers to accuracy of the compressed NNs is different and they should be processed individually. An improper decomposition rank for one layer can even lead to a degradation of the model, which can make it inapplicable.

This paper proposes a new method, the Proxy-based Automatic tensor Rank Selection method (PARS), for automatic rank selection given the model complexity level. More specifically, PARS considers the whole NN when choosing the suitable rank configuration for each layer. It models the problem as a constrained discrete optimization of a black-box function and solves it using Bayesian optimization. We also investigate the effect of low-rank weight decomposition on the compressed model feature distribution. This paper proposes an efficient proxy metric that highly correlates with the post-fine-tuning model metric.

To demonstrate the efficiency and applicability of the proposed method, PARS was tested with various tensor decomposition methods (CPD-EPC [3], TKD-2 decomposition [4], and Spatial-SVD [5]), several DNN architectures (AlexNet [6], VGG-16 [7], ResNet-18, and ResNet-56 [8]), and two datasets (the CIFAR-10 [9] and ILSVRC-2012 [10] datasets). For ResNet-18, PARS in combination with the recently proposed CPD-EPC decomposition showed a state-of-the-art accuracy–compression ratio trade-off, outperforming the results reported in the original paper [3] (Figure 1).

The contributions of this paper are summarized as follows:

• A novel universal automatic rank selection algorithm (PARS) that selects the best sets of ranks for DNN compression with a given model compression ratio and can work with different matrix/tensor decompositions;
• A novel proxy metric for an efficient automatic rank search that improves the predictability of post-compression DNN performance and increases the quality of the rank search procedure;
• Investigation of the effect of layer decomposition on the NN feature distribution. This paper discovers that the decomposition of weight tensors adversely influences the feature distribution inside a neural network and impairs the predictability of post-compression DNN performance.

The rest of the paper is organized as follows. Section 2 presents an overview of neural network compression and decomposition rank selection methods. Section 3 provides additional clarifications on how decomposition-based neural network compression is performed. Section 4 describes PARS and the proposed proxy metric. Section 5 analyzes the effect of low-rank weight decomposition on the compressed network feature distribution and justifies the selection of a proxy metric. Section 6 evaluates the performance of PARS in combination with different matrix/tensor decompositions. Section 7 summarizes and interprets the key findings of the paper. Section 8 concludes the paper.

2. Related Works

2.1. Neural Network Compression and Acceleration

Extensive research has been conducted on neural network compression and acceleration. Such methods modify neural networks by reducing the redundancy in their weights. The scope of neural network compression approaches can be divided into four subgroups: pruning (structured/unstructured), low-rank approximation of weights, quantization, and knowledge distillation. Weight sparsification or unstructured pruning compresses neural networks by removing individual weights with low importance [11]. As a result, the number of parameters in the network can be significantly reduced [12]. Structural pruning [13] is an extension of the pruning approach that removes entire structural elements of the network (e.g., filters, channels). This approach allows for simultaneous speedup and compression of NNs without specialized software. A low-rank approximation-based method exploits the fact that neural networks are over-parameterized and that their weights lie in a low-rank linear space [1]. Therefore, it is possible to approximate NN layer weights with a compact low-rank matrix or tensor decomposition, such as canonical polyadic decomposition (CPD) [14,15], Spatial-SVD [5,16,17], or Tucker decomposition [18,19]. As a result, the initial layers can be replaced with lightweight layers in a factorized format, thereby making the model faster and smaller (we refer the reader to Section 3 for a more detailed explanation). Quantization reduces redundancy in the weights’ and activations’ representations by approximating them with numbers with a lower precision [20]. This approach allows one to reduce the model’s latency and memory footprint. The knowledge distillation approach leverages knowledge from cumbersome pre-trained teacher models to facilitate the training of a small student model [21]. The knowledge can be outputs [21], intermediate features [22,23], or relationships between features [24,25]. This approach reduces the performance gap between compact/fast and large/slow models.

2.2. Rank Selection or Tensor Decomposition Model Selection

This paper focuses on low-rank approximation-based NN compression methods. Matrix and tensor decompositions are parameterized by the decomposition rank, which defines the trade-off between the approximation error and number of parameters [26]. DNN compression requires a choice of the decomposition rank for each compressed layer in a neural network, the number of which usually exceeds dozens [5]. Poor rank selection may lead to significant DNN performance degradation (in the case of too small a number of ranks) or an insufficient DNN compression level (in the case of too high a number of ranks).

Rank search methods for NN compression usually use heuristic approaches to find the best decomposition rank for each model layer. The authors of [18] used the global analytic solution of variational Bayesian matrix factorization (VBMF) [27] to determine the ranks for TKD-2 for convolutional layers and the ranks for SVD for fully connected ones. An iterative-search-based approach was adopted in [3,16,17] to obtain the suitable rank of CPD. The authors of [16] used a greedy layer-wise strategy to determine the ranks of all layers satisfying the specific target complexity. This strategy assumed that the accuracy- network energy relation is linear and can be estimated as the leading principal component analysis (PCA). The authors defined an objective function to maximize the accumulated energy with time complexity constraint. In [3], the authors applied a simple binary search to find the lowest possible rank for each layer that did not exceed a pre-selected accuracy drop threshold. The authors of [17] proposed a model-wise rank search that selected ranks for all layers simultaneously, and it showed a better performance compared to that of the layer-wise search used in [16]. A non-iterative approach that utilized energy-based and measurement-based PCA approaches was proposed in [5]. Unlike other authors, the authors of [5,17] conducted a model-wise rank search, which was more optimal than a layer-wise search.

The literature shows that existing rank search methods have three major drawbacks. Firstly, there is a lack of a general rank search algorithm for a general tensor decomposition. Secondly, the authors of a recent paper [28] found out that the pre-fine-tuning accuracy, which is used in most methods to accelerate the iteration of the rank search procedure, is a flawed proxy metric. Moreover, it does not correlate with the post-fine-tuning accuracy of the model. Thirdly, existing methods do not allow the explicit control of the compression ratio. Each model and each tensor decomposition method may have its own unique accuracy–compression ratio trade-off curve [28]. For practical applications, NN compression should be able to control the model size during the compression process. This paper aims to develop a rank selection method that alleviates the above drawbacks.

3. Low-Rank Approximation of Neural Network Weight

Most weights in NNs are matrices (e.g., fully connected layers) and tensors (e.g., convolutional layers). Denil et al. [1] showed that parameters in NNs are extremely redundant and lie in a low-rank space. This observation gave rise to many methods for compressing NNs by using low-rank approximations of weights. These methods replace weights in the layer with their low-rank approximations that are obtained by minimizing $\left|\right|W_{init}-W_{factorized}\left|\right|$, where $W_{factorized}$ is a low-rank model (for more details, refer to [26]). As a result, the initial linear operation is replaced by a sequence of linear operations defined by a low-rank decomposition $Y=W_{factorized}\ast X$, where X and Y are the input and output of the layer, * is a linear operation performed by the layer (e.g., matrix multiplication in the case of fully connected layers or convolution in the case of a convolutional layer).

In the case of a convolutional layer, the original layer is replaced by the sequence of convolutions, as shown in Figure 2. The structure of the factorized convolutions depends on the decomposition model.

CPD-3 [3] replaces the initial layer with three layers (Figure 2b): a $1\times 1$ convolution that projects $I_{ch}$ input channels to R channels, a $D\times D$ convolution with R groups that perform spatial convolution, and a $1\times 1$ convolution that projects R input channels to $O_{ch}$ channels. The parameter and computation reduction rate is shown in Formula (1). CPD-EPC is CPD-3 with a sensitivity minimization constraint. For more details, see Appendix C.

$$C{R}_{CPD-3}=\frac{(I_{ch}+O_{ch}+D^2)R}{I_{ch}{O}_{ch}{D}^2}.$$

(1)

Spatial-SVD [5] replaces the initial layer with two layers (Figure 2c): a $D\times 1$ convolution layer that performs convolution in the vertical direction projects $I_{ch}$ input channels to R channels and a $1\times D$ convolution that performs convolution in the horizontal direction projects R input channels to $O_{ch}$ channels. The parameter and computation reduction rate is

$$C{R}_{Spatial-SVD}=\frac{(I_{ch}D+O_{ch}D)R}{I_{ch}{O}_{ch}{D}^2}.$$

(2)

TKD-2 [18] replaces the initial layer with layers (Figure 2d): a $1\times 1$ convolution that projects $I_{ch}$ input channels to $R_1$ channels, a $D\times D$ ordinary convolution that performs spatial convolution and projects $R_1$ input channels to $R_2$ output channels, and a $1\times 1$ convolution that projects $R_2$ input channels to $O_{ch}$ channels. The parameter and computation reduction rate is  

$$C{R}_{TKD-2}=\frac{I_{ch}{R}_1+R_1{R}_2{D}^2+O_{ch}{R}_2}{I_{ch}{O}_{ch}{D}^2}.$$

(3)

4. Method

Consider a DNN of L layers. The goal is to find a rank set $R=\{r_1,...,r_L\}$ that maximizes the post-fine-tuning performance $F\left(R\right)$ of the model for a given compression ratio $\alpha$, e.g., accuracy in the case of a classification problem. The rank search procedure is defined as the following optimization problem:

Problem 1

(Optimal Rank Search).

$$\begin{array}{cccc}\hfill & \underset{R=(r_1,...,r_L)}{\mathit{maximize}}\hfill & \hfill & F\left(R\right)\hfill \\ \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & C\left(R\right)=\alpha C_{orig},\hfill \end{array}$$

(4)

where $C\left(R\right)$ and $C_{orig}$ define the complexity, FLOPs, and number of parameters of the compressed and original models. In practice, applying a fine-tuning procedure in order to evaluate each set of ranks requires a significant amount of computational resources and is highly time-consuming.

4.1. Proxy Metric

Using the target metric of the compressed model after fine-tuning is impractical, since fine-tuning usually takes 10–20 epochs. Replacing expensive target metrics with an easy-to-compute proxy metric may resolve this issue and significantly reduce the optimization time. In the case of the classification task, a straightforward approach is to use the accuracy of the model after compression [29]. However, the authors of [28] showed that for decomposition-based compression, this metric was not coherent with the accuracy of the model after fine-tuning. In this work, this important observation is confirmed in Figure 3a. To this end, a novel proxy metric for the optimization problem is proposed. It was found that calibrating the BatchNorm (BN) statistics in the decomposed model significantly increased the correlation between the model accuracies before and after fine-tuning (see Figure 3b). Thus, the accuracy of the calibrated model $\tilde{F}\left(R\right)$ was used as a proxy metric for the accuracy of the model after fine-tuning $F\left(R\right)$ to accelerate the optimization process, since it was much cheaper in terms of time and computation. Section 5 explains how the calibration of the BN statistics affected the accuracy of the model.

4.2. Search Space

The search space of our algorithm was defined by all possible combinations of rank values in layers to be compressed. These combinations represent a vector space $\mathcal{R}$, which can be defined by the Cartesian product:

$$\mathcal{R}=\prod _{l=1}^L{R}_l=\{\{r_1,...,r_L\}=R:r_l\in R_l\},$$

(5)

where $r_l$ is a decomposition rank of the l-th layer, $1\le r_l\le r_l^{upper}$, and $r^{upper}$ is the rank of the layer when the compression ratio is equal to 1. This vector space grows exponentially with the number of layers in the NN and may have a huge number of elements in the case of NNs with a great depth.

To facilitate the rank search process, additional constraints are introduced into Problem 1:

$$c_l^{min}\le c_l\left(r_l\right)\le c_l^{max},l=1,\dots ,L,$$

(6)

where $c_l$, $c_l^{max}$, and $c_l^{min}$ define the current, maximum, and minimum complexity of layer l, respectively. In addition, the model complexity constraint can be rewritten as:

$$(\alpha -\Delta \alpha )C_{orig}\le C\left(R\right)\le (\alpha +\Delta \alpha )C_{orig}.$$

(7)

By combining them together, these constraints limit the search space.

4.2.1. Single-Rank Case

In special cases in which the layer decomposition is defined by a single rank, e.g., CPD or SVD, the layer complexity is linear with respect to the rank value $c_l\left(r_l\right)=k_l{r}_l$. Thus, constraints (6) and (7) can be transformed into the following linear constraint:

$$r_l^{min}\le r_l\le r_l^{max},l=1,\dots ,L,$$

(8)

$$(\alpha -\Delta \alpha )C_{orig}\le \sum _{l=1}^L{k}_l{r}_l+C^{add}\le (\alpha +\Delta \alpha )C_{orig},$$

(9)

where $r_l^{max}=c_l^{max}/k_l$, $r_l^{min}=c_l^{min}/k_l$, and $C^{add}$ is the complexity of the uncompressed layers in the decomposed model, e.g., fully connected layers or $1\times 1$ convolutions in the shortcut connections of residual blocks.

4.2.2. Multi-Rank Case

In cases in which several rank values determine the layer decomposition (e.g., Tucker decomposition, block-term tensor decomposition, tensor train, tensor ring), the expression for the layer complexity can take a more complex form. To avoid complications, one can search not in the space of ranks, but in the space of layer complexities. In this case, PARS finds the best complexity for a given layer, and the next auxiliary function $r_l=g\left(c_l\right)$ finds a set of ranks $r_l$ for layer l that has an optimal approximation error for the given compression ratio $c_l$.

4.3. Search Algorithm

Taking into account the proposed proxy metric $\tilde{F}\left(R\right)$ and adding constraint (6) and relaxation (7) to Problem 1, the optimization problem can be reformulated as follows:

Problem 2

(Optimal Rank Search using the Proxy Metric).

$$\begin{array}{cccc}\hfill & \underset{R=(r_1,...,r_L)}{\mathit{maximize}}\hfill & \hfill & \tilde{F}\left(R\right)\hfill \\ \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & (\alpha -\Delta \alpha )C_{orig}\le C\left(R\right)\le (\alpha +\Delta \alpha )C_{orig}\hfill \\ \hfill & \hfill & & {\alpha }^{min}{c}_l^{orig}\le c_l\left(r_l\right)\le {\alpha }^{max}{c}_i^{orig},l=1,\dots ,L.\hfill \end{array}$$

(10)

The function $\tilde{F}\left(R\right)$ may not have an explicit form, and the problem is formulated as a discrete black-box optimization problem.In addition, compression of the model with the given ranks is still very time-consuming, since tensor approximation methods are computationally expensive and typically require up to several hours per layer.

The complexity of the problem described above means that a black-box optimization method is needed to find a reasonably good set of ranks in a relatively small number of iterations. To this end, a Bayesian optimization procedure [30] is used to solve the constrained optimization (Problem 2) and to find the best set of ranks. It builds a surrogate function for the objective and iteratively samples an appropriate point based on the acquisition function defined from this surrogate.

Algorithm 1 and Figure 4 describe the proposed rank search procedure in the single-rank case. In order to provide initial information about the objective function to our surrogate, K quasi-random points are firstly sampled in the search space with a Sobol sequence [31] and then evaluated. After the initial evaluation of the points, the Bayesian optimization procedure is conducted by using Gaussian processes (GPs) as a surrogate model and the expected improvement (EI) as the acquisition function:

$$EI\left(R\right)=\mathbb{E}\left[max(\tilde{F}\left(R\right)-{\tilde{F}}^{*},0)\right],$$

(11)

where ${\tilde{F}}^{*}$ is the best observed outcome. A Matern 5/2 kernel is used for the kernel function. For more details, see Appendix A.

Algorithm 1 PARS

Input: training set, validation set, pretrained neural network, $\alpha ,{\alpha }^{max},{\alpha }^{min}$, init budget K, search budget N Space exploration stage: $\mathcal{R}\leftarrow R_{1:K}$, $Y\leftarrow \tilde{F}\left(R_{1:K}\right)$                          Initialize with Sobol seq. Bayesian optimization stage: for n = (1, …, N) do    $UpdateSurrogateModel(\mathcal{R},Y)$                          update GP prior    $R_n\leftarrow argma{x}_{R}E{I}_n\left(R\right)$                            find new candidate    $y_n\leftarrow \tilde{F}\left(R_n\right)$                                  evaluate candidate    $\mathcal{R}\left(n\right)=R_n,Y\left(n\right)=y_n$ end for Return: $R^{*}=argma{x}_{\mathcal{R}}\left(\tilde{F}\left(R\right)\right)$

5. Feature Distribution Analysis

NN compression based on low-rank tensor decomposition replaces the original layer with a sequence of new lightweight layers with factor matrices or tensors as weights. This approach assumes that $W_{init}\cong W_{factorized}$; the outputs of these layers are also close $W_{init}\ast X\cong W_{factorized}\ast X$, and their outcome error $E\ast X=(W_{init}-W_{factorized})\ast X$ is relatively small.

However, in practice, when the decomposition rank is relatively small, $E\ast X$ is not negligible. As observed in Figure 5a, the distribution of the output features of the decomposed layer is significantly affected. The errors have different distributions in all channels with non-zero means. The problem is amplified when multiple layers are simultaneously decomposed. Shifts in the output distributions of the decomposed layers are aggregated, and the compressed model’s performance becomes unpredictable.

This problem can be partially resolved by calibrating the statistics of the BN layers after the decomposed layers. The BN layer produces the following operation along each channel of input features:

$$y=\gamma \frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}+\beta ,$$

(12)

where $\gamma$ and $\beta$ are trainable parameters, i.e., the scaling and shifting factors, and $ϵ$ is a constant. During training, $\mu$ and ${\sigma }^2$ are calculated with a moving mean and variance:

$${\mu }_t=m{\mu }_{t-1}+(1-m){\mu }_{\mathcal{B}},{\sigma }_t^2=m{\sigma }_{t-1}^2+(1-m){\sigma }_{\mathcal{B}}^2,$$

(13)

where m is the momentum coefficient, subscript t denotes the number of training iterations, and ${\mu }_{\mathcal{B}}$ and ${\sigma }_{\mathcal{B}}^2$ are unbiased estimations of the features’ mean and variance in the training batch. The BN layer uses statistics in the training regime, as shown in the formula above. During the testing time, the parameters and accumulated statistics are frozen, and the model uses aggregated statistics, ${\mu }_{T_0}$ and ${\sigma }_{T_0}^2$, where $T_0$ is the total number of iterations passed.

The decomposition of the convolutional layer affects the feature distribution. As a result, ${\mu }_{T_0}$ and ${\sigma }_{T_0}^2$ are no longer the valid mean and variance of the input distribution of the BN layer input. This problem demands calibration of the BN statistics. For this purpose, the BN layers are switched to a training mode and updated for several iterations (in our experiments, 100–200 iterations were sufficient) without updating the training parameters of the model. After the calibration of the BN statistics, the new aggregated statistics, ${\mu }_T$ and ${\sigma }_T^2$, represent more accurate feature distributions. This procedure corrects the feature distribution in the decomposed model and provides the difference between the output features of the compressed and original models in order to be zero-mean (Figure 5b). Moreover, it partially recovers the model degradation and makes the behavior of the model more predictable (Figure 3b), making the accuracy of the decomposed and calibrated model a good proxy metric for the prediction of the post-fine-tuning model behavior.

Single- and Multi-Layer Decomposition

The positive effect of BN calibration can be seen more clearly in the case of simultaneous multi-layer decomposition. For this purpose, three layers of ResNet-18 are chosen for demonstration: layer1.1.conv1, layer2.0.conv2, and layer3.1.conv2. Their kernel weights are decomposed using CPD-EPC with different CP ranks. Next, the outputs of BN layers after original and decomposed layers are collected in two settings: Single, where the compressed model has only one decomposed layer, and Joint, where three layers are simultaneously decomposed.

The output of the BN layer is a 3-way array with modes: channels, width, and height. For the spatial dimension, width, and height, the $L_1$-norm of differences between outputs of original and decomposed layers is close to some constant level (see an example in Figure A1 in Appendix B). Thus, the values of the $L_1$-norm can be averaged over the spatial dimensions without substantial loss of information, resulting in a mean absolute error (MAE). Further, to compare the shifts in outputs of the decomposed layers for different CP ranks, the maximum of the MAE over the channels (mMAE) is taken (results for the mean of the MAE over the channels are presented in Figure A2a in Appendix B).

The results shown in Figure 6 indicate that the mMAE before calibration is higher than after the calibration for both settings. In addition, for the Joint setting, the mMAE for subsequent decomposed layers is higher than that for the Single setting for both cases, before and after BN calibration. However, this difference decreases while the CP rank increases. Thus, the shift in the feature distribution is most notable when the decomposition rank is relatively low.

The experiments in Appendix B for the cosine distance instead of the $L_1$-norm (see Figure A2b,c) and for the bias (difference in the means of the outputs, see Figure A2d,e) show similar behaviors.

6. Experimental Results

In this section, we present an experimental evaluation of the proposed method. Firstly, we analyze the effect of each component in PARS—the rank search and proxy metric. Secondly, we compare our method with state-of-the-art NN compression methods by compressing ResNet-18 [8] and VGG-16 [7] after training on the ILSVRC-2012 [10] dataset. Thirdly, we compare our results with those of other representative rank search methods in compressing ResNet-56 by Spatial-SVD on the CIFAR-10 dataset. Lastly, we evaluate our rank search method in a multi-rank tensor decomposition scenario by compressing AlexNet [6] after training on the ILSVRC-2012 [10] dataset.

Implementation Details: The experiments were conducted with the efficient NN framework Pytorch [32] on a GPU server with one NVIDIA A-100 GPU, AMD EPYC 7452 32-Core CPU, and 80 GB RAM. We used a pre-trained ILSVRC-2012 model shipped with Torchvision: ResNet-18 had top1 and top5 accuracies of 69.76 and 89.08, VGG-16 with BN had top1 and top5 accuracies of 73.36 and 91.52, and AlexNet had top1 and top5 accuracies 56.52 and 79.07; for CIFAR-10, the pre-trained ResNet-56 model has had a top1 accuracy of $93.3\%$. The compressed models were fine-tuned with an SGD optimizer and the StepLR learning rate schedule; the details of the fine-tuning are provided in

Table 1. Details of the fine-tuning of the hyperparameters.

Model	Batch Size	Initial lr	Epochs	lr Decay	Step Size	Weight Decay
AlexNet	256	${10}^{-3}$	16	0.5	2	0
VGG-16			25	0.1	10	${10}^{-4}$
ResNet-18			25	0.1	10	${10}^{-4}$
ResNet-56			51	0.1	20	${10}^{-4}$

.

Selection of the PARS hyperparameters:

• ${\alpha }^{max}$ and ${\alpha }^{min}$ define the maximum and minimum compression ratios of the DNN’s layers during the rank search procedure. We used the following heuristic setting: ${\alpha }^{max}=min(1.5\alpha ,1)$ and ${\alpha }^{min}=0.15\alpha$, where $\alpha$ is the target compression ratio. This allowed us to avoid evaluating unreasonably low or high compression ratios, and it reduced the search space.
• $\Delta \alpha$ defines the acceptable deviation from the target compression level of the model. We set $\Delta \alpha =0.0005$. If $\alpha =0.5$, then the output compressed models will have compression ratios in the range: $[0.4995,0.5005]$.
• K and N denote the number of space exploration steps and the number of Bayesian optimization steps. We set $K=0.2N^{total}$ and $N=0.8N^{total}$, where $N^{total}$ is the total number of evaluations, which depends on the depth of the DNN; the higher the number of layers in the model, the more evaluation iterations it requires. We set $N^{total}=100$ for AlexNet, VGG-16, and ResNet-18 and $N^{total}=200$ for ResNet-56.
• To avoid over-fitting to a validation set during the rank search process, we used two subsets of the training set to calibrate the BN statistics and evaluate the compressed model; these subsets consisted of 25,600 and 48,000 randomly sampled images, respectively.

6.1. Algorithm Analysis

6.1.1. Rank Search Process

Figure 7 shows the convergence behavior of the rank search process for ResNet-18 with a compression ratio of 3.5. After the random search stage in the first 20 iterations, the validation accuracy of our model significantly increased. This result indicates that we obtained better ranks than those obtained by using random sampling. During the Bayesian optimization process, the compressed model accuracy gradually increased with the number of iterations despite the metric drop at iteration 90. This drop meant that at iteration 90, the GP’s surrogate function put the maximum EI value at a worse point than its predecessors. This behavior is normal for the Bayesian search process, as each iteration of the search process updates and improves the accuracy of the surrogate function. Note that we evaluated our model on a part of the training set, so the validation accuracy values were higher than those on the test set. The maximal validation accuracy was achieved at iteration 64, which was then taken as a final prediction for this experiment. However, after this, the model found many other points with validation accuracies that were very close to the optimal one.

6.1.2. Rank Analysis

Figure 8 shows the relative ranks found by PARS for different layers of ResNet-18 with different compression ratios. The relative rank is denoted by the ratio $R_l/R_l^{bound}$, where $R_l$ is a decomposition rank found for layer l and $R_l^{bound}$ is the rank for layer l when the compression ratio is 1. Overall, PARS found comparable levels of relative ranks for different compression ratios, except for the first and the last layers. Conv1 had higher relative ranks. This observation could be explained by the fact that this was the first layer in the network, and it did not have any residual connections; thus, the overcompression in this layer could not be compensated by the other layers. Layer4.1.conv2 had a notably lower relative rank compared to those of the other layers. The last layer before the average pooling operation squeezed all spatial information in the feature map. This fact might explain the relatively low importance of this layer. Moreover, this might also be connected to a weight banding phenomenon [33]. Note that our rank search algorithm did not have explicit information about the network structure and its weights; despite this, PARS found patterns that could later be explained by external knowledge.

6.1.3. Proxy Metric Evaluation

To evaluate the effectiveness of the proposed proxy metric, we applied PARS for CPD-EPC for $4\times$ compression of ResNet-18 on the ILSVRC-2012 dataset with and without calibration of the BN statistics. After compression, both models were fine-tuned with the same schedule.

Table 2. Comparison of the performance of PARS with and without the calibration of the BN statistics. The ↓ FLOPs column denotes the compression of the computations; the $\Delta$ top-1 and $\Delta$ top-5 columns denote the top-1 and top-5 accuracy losses—the higher, the better.

BN Calibration	↓ FLOPs	$\mathbf{\Delta }$ Top-1, %	$\mathbf{\Delta }$ Top-5, %
✗	4	−1.64	−0.78
✓		−0.90	−0.44

shows that using BN calibration led to a noticeably smaller accuracy drop for the same compression ratio. Figure 9 shows that using proxy metrics without the calibration of the BN statistics resulted in a poor balance of computational cost between layers. For example, layers layer2.1.conv1 and layer3.1.conv2 seemed to be overcompressed, whereas the adjacent layers had 2–3 times lower compression ratios. In contrast, using BN calibration resulted in a better compression ratio balance between layers. We also refer the reader to Appendix E for batch size effect evaluation.

6.2. Model Compression Results

We compared our results from compression of ResNet-18 and VGG-16 on the ILSVRC-2012 dataset with those of other recently proposed methods (See

Table 3. Comparison of different model compression methods on the ILSVRC-12 validation dataset. The ↓ FLOPs column denotes the compression of the computations; the $\Delta$ top-1 and $\Delta$ top-5 columns denote the differences in top-1 and top-5 accuracy from the original model—the higher, the better.

Model	Method	Type	↓ FLOPs	$\mathbf{\Delta }$ Top-1, %	$\mathbf{\Delta }$ Top-5, %
VGG-16	TKD+VBMF [18]	LR	4.93	-	−0.50
	Asym. [16]	LR	5.00	-	−1.00
	CP-3C [34]	LR	5.00	-	−0.30
	ENC-Inf [5]	LR	5.00	-	+0.00
	CPD-EPC [3]	LR	5.26	−0.92	−0.34
	GFA [35]	LR	5.36	−1.06	−0.27
	CPD-EPC+PARS (Ours)	LR	5.49	+0.33	+0.11
ResNet-18	DACP [36]	CP	1.89	−2.29	−1.38
	Channel Gating NN [37]	CP	1.93	−0.40	-
	FBS [38]	CP	1.98	−2.54	−1.46
	CPD-EPC+PARS (Ours)	LR	2.00	+0.26	+0.40
	ManiDP [39]	CP	2.04	0.88	0.32
	GBIP [40]	CP	2.07	0.82	0.63
	MUSCO [19]	LR	2.42	−0.47	−0.30
	TT [41]	LR	2.47	-	+0.00
	CPD-EPC+PARS (Ours)	LR	3.03	−0.14	+0.03
	CPD-EPC [3]	LR	3.09	−0.69	−0.15
	HODEC [42]	LR	3.13	−0.61	−0.09
	BATUDE [43]	LR	3.22	-	−0.04
	CPD-EPC+PARS (Ours)	LR	3.22	−0.30	−0.04
	TT [41]	LR	4.62	-	−1.61
	CPD-EPC+PARS (Ours)	LR	4.99	−1.81	−0.83

and Spatial-SVD results in Appendix D). Our method was compared with other methods for decomposition-based neural network compression (LR Type), as well as with channel-pruning DNN compression methods (CP Type). Our model-wise rank search algorithm showed considerably better results when in combination with CPD-EPC than those of the layer-wise binary search reported in the original paper [3]. Moreover, to the best of our knowledge, our method provides new state-of-the-art results for the accuracy–compression ratio trade-off for ResNet-18 and VGG-16 compression on the ILSVRC-2012 dataset.

6.3. Comparison with Other Rank Selection Methods

We compared PARS with a representative group of rank search methods, ENC [5]. For this purpose, we compressed ResNet-56 when pre-trained on the CIFAR-10 dataset by using the Spatial-SVD decomposition method (Figure 2c); the pre-trained model had a $93.3\%$ top-1 accuracy.

Table 4. Comparison of different rank search methods for the Spatial-SVD compression of ResNet-56 on the CIFAR-10 dataset.

Method	↓ FLOPs	$\mathbf{\Delta }$ Top-1, %, w/o f-t	$\mathbf{\Delta }$ Top-1, %, w f-t	${\mathit{N}}^{\mathbf{total}}$
Uniform [5]	2	−12.7	−0.2	20
ENC-Inf [5]		−2.9	−0.1	20
ENC-Model [5]		−3.5	−0.1	20
ENC-MAP [5]		−3.3	−0.1	20
PARS (Ours)		−2.1	−0.1	200

shows that PARS resulted in a smaller accuracy drop than that of the family of ENC [5] methods and that a uniform rank reduction was applied with the same compression ratio, but it required more evaluation iterations $N^{total}$. It should be mentioned that ENC rank search methods are designed specifically for the Spatial-SVD layer compression method. In contrast, PARS is a universal method that works with an arbitrary weight decomposition type.

6.4. Multi-Rank Compression

To evaluate our rank search method in a multi-rank decomposition scenario, we compressed AlexNet [6] with TKD-2, which had two rank parameters per layer. As a function $g\left(c_l\right)$, we used a simple heuristic that kept the ratio $R_1/R_2$ equivalent to $C_{in}/C_{out}$. We compressed all convolutional and fully connected layers of the network except for the first convolutional layer, whereas the fully connected layers were compressed using truncated SVD.

The results were compared to those of [18], who applied TKD-2 and used VBMF [27] for rank selection (

Table 5. Evaluation of PARS in the multi-rank case of AlexNet on the ILSVRC-2012 dataset.

Rank Search	↓ FLOPs	$\mathbf{\Delta }$ TOP-1, %	$\mathbf{\Delta }$ Top-5, %
VBMF	2.67	-	−1.70
PARS (Ours)	2.76	−0.94	−0.37

). Compared to VBMF, PARS found ranks that resulted in a smaller accuracy drop with a slightly higher compression ratio.

6.5. Search Time

We provided a search time for the Spatial-SVD compression of ResNet-56 when pre-trained on the CIFAR-10 dataset from Section 6.3. The rank search procedure consisted of 40 initialization iterations with the Sobol sequence [31] and 160 iterations of Bayesian optimization. Model decomposition, BN calibration, and accuracy evaluation were performed during the candidate evaluation at each iteration.

Table 6. Search time summary for the Spatial-SVD compression of ResNet-56 on the CIFAR-10 dataset.

Operation	Time Spent
Search	35 min
Model decomposition	28 min
BN calibration	9 min
Accuracy evaluation	4 min
Total	1 h 16 min

shows the details of the search time. Note that model decomposition time heavily depended on the type of decomposition used.

7. Discussion

This paper demonstrates that the rank search algorithm for decomposition-based neural network compression can be efficiently solved by using constrained black-box optimization, and a method called PARS is proposed. PARS eliminates the problems that its predecessors had: suboptimal layer-wise rank search, dependence on the decomposition algorithm, a flawed target proxy metric, and the absence of a compression ratio control.

We observed that layer factorization resulted in a shift in the distribution of the layer outcomes (Figure 5a), which affected the predictability of the neural network performance after fine-tuning (Figure 3a). This issue was resolved by calibrating the statistics in the BatchNorm (BN) layer (Figure 3b). The proposed proxy metric showed a considerable improvement in the rank search method (Table 5, Figure 9).

A similar effect of BN calibration was observed in the models compressed by using channel pruning [44]. Despite the similarities in the use of BN calibration, our method had a different motivation. Figure 3 shows that matrix/tensor weight decomposition affected the feature distribution after the compressed layer. Most importantly, the difference between the original and factorized layer outputs was not zero-mean. These feature distribution shifts affected the performance of the compressed model. The calibration of the BN statistics aligned the compressed model’s feature distribution and made the difference of the feature distribution from that of the original model zero-centered. This effect could also be achieved through bias correction. EagleEye [44] applied BN calibration (adaptive BN) to models compressed by using channel pruning. We cannot speak about feature distribution differences in channel pruning, since the compressed layer has fewer channels in the output feature map.

In PARS, we formulated the rank search procedure as a constrained optimization problem and solved it with Bayesian search (Figure 7). The optimization constraint defines the compression ratio of the model and, therefore, allows control over the model’s whole performance–compression ratio curve (Figure 1). The experimental results show that PARS may significantly improve the performance of well-known tensor-based neural network compression methods (Table 3 and Table 5).

Generally, PARS selects a compression ratio for each model layer. This allows PARS to work with each type of weight decomposition model. Moreover, one can combine it with other neural network compression methods, such as structured or unstructured pruning.

Despite the results shown here, PARS has several areas for further improvement. Firstly, the proposed algorithm works with only one type of decomposition at a time. Thus, the selection of a layer decomposition scheme is required. Secondly, the current compression constraints should have analytical formulas (such as FLOPs/MACs or the number of parameters). Working with non-analytical constraints (such as inference time on a particular device) might be more beneficial in practical scenarios.

8. Conclusions

This paper proposes PARS, a novel rank search algorithm that can automatically find ranks for decomposition-based neural network compression. PARS models the task as a constrained black-box optimization problem and solves it through Bayesian optimization. This study discovered that a low-rank weight decomposition of DNN weights adversely affects its feature distribution and makes its post-fine-tuning performance unpredictable. Based on this finding, we propose the calibration of the statistics in BatchNorm after compression. This approach stabilizes the feature distribution and results in a novel proxy metric for the rank search algorithm. As a result, PARS improves the performance of existing decomposition-based rank selection by providing a more efficient set of ranks for the selected compression ratio.