KT-NAS: Knowledge Transfer for Efficient Neural Architecture Search

Abstract

Pre-trained models have played important roles in many tasks, such as domain adaptation and out-of-distribution generalization, by transferring matured knowledge. In this paper, we study Neural Architecture Search (NAS) in the feature space level and observe that low-level features of NAS-based networks (generated networks from a NAS space) become stable in the earlier stage of training. In addition, these low-level features are similar to those from hand-crafted networks such as VGG, ResNet, and DenseNet. This phenomenon is consistent over different search spaces and datasets. Motivated by these observations, we propose a new architectural method for NAS, called Knowledge-Transfer NAS, which utilizes the features from a pre-trained hand-crafted network. Specifically, we replace the first few cells of NAS-based networks with pre-trained manually designed blocks and freeze them, and then only train the remaining cells. We perform extensive experiments using various NAS algorithms and search spaces, and show that Knowledge-Transfer NAS achieves higher/comparable performance while requiring less memory footprint and search time, offering a new perspective on the applicability of pre-trained models for improved NAS algorithms.

1. Introduction

Neural Architecture Search (NAS) has become an important domain in the machine learning field due to its superior performance and democratization feasibility for deep learning. The major advantage of NAS is that it automatically discovers top performing architectures from a large search space on a target dataset. Since the solution can be found without human involvement, NAS has a wide range of applications such as image classification [1], object detection [2,3], and pruning [4].

Many NAS algorithms have been developed based on reinforcement learning [5,6], evolution [7], gradient-based optimization [8], and so on. Although these works show impressive performance, they usually require a huge amount of computational costs and training time, which prevent them from being deployed in practice.

To solve this problem, researchers have made many attempts to reduce the computational costs, especially in the searching phase of NAS [9,10]. Query-based NAS such as Regularized Evolution (REA) [11] trains and evaluates thousands of small (proxy) models before restoring the best model into the original size where the network’s depth and the number of channels are enlarged. Despite the reduced model size, these approaches still require training all layers of each candidate network during the search phase.

Recently, gradient-based NAS algorithms [10,12] have been constructed, where a very large network, called a supernet, is constructed and trained from scratch. The target network is obtained by eliminating unimportant operations in the supernet once converged. Although effective in amortizing training cost across architectures, the supernet is also fully optimized end-to-end during the search process.

From another perspective of advancement in machine learning, it has been revealed that pre-trained models are very useful for data-efficient and/or compute-efficient learning in domain adaptation [13,14], natural language processing [15], and meta-learning tasks [16]. These rely on the representation power of the features in the pre-trained models in terms of the generalization ability. Motivated by this, we consider whether part of a pre-trained network can be reused as a fixed feature extractor in NAS, such that only the remaining layers are trained during the search phase.

Figure 1 motivates our knowledge transfer-based NAS (KT-NAS) idea, where we visualize the feature maps generated by (i) a randomly selected network from NAS-Bench-201 [17] during training in the first four rows and (ii) different networks when converged in the last four rows. As shown in Figure 1, low-level features reach stability quicker than high-level features along the epochs during training. Additionally, both NAS-based networks and manually designed networks learn similar low-level representations. Thus, repeatedly training the whole networks formed by different cells may lead to unnecessary efforts [6,11]. A natural question arises: Can we perform the search step in NAS by training only a few cells rather than the whole network? Technically, this paradigm shortens the training time and reduces the memory footprint, because the required memory for calculating the gradients becomes smaller than that required in conventional approaches.

In this paper, we show that it is possible to perform efficient searching in NAS by replacing several initial cells of a child network with pre-trained layers and let NAS find the remaining cells. We first investigate the stability of the feature maps during training generated by networks from NAS-Bench-101 [18] and NAS-Bench-201 [17] in Section 3.1. Second, we analyze the similarity of feature maps between the networks sampled from NAS-Bench-201 and hand-crafted ones, namely VGG [19], ResNet [20], and DenseNet [21] in Section 3.2. Motivated from our comprehensive analysis of the aforementioned two perspectives of NAS-based networks, we propose to transfer the knowledge from a pre-trained network to NAS-based networks for first few cells in the search phase. To this end, the contributions of our paper are summarized as follows:

• We propose a new KT-NAS that leverages the features generated by a pre-trained network to improve the efficiency of NAS. Specifically, we replace several initial layers of NAS-based networks with pre-trained layers and freeze them, while leaving the other layers as trainable. Hence, the proposed method further accelerates current NAS methods.
• We demonstrate the effectiveness of our method by incorporating the proposed method into three search algorithms: evolutionary-based REA [11], gradient-based DARTS+PT [22], and one-shot NAS, namely Single Path One-Shot (SPOS) [7]. On NAS-Bench-201, we save up to $2\times$ memory footprint and the search runs $1.4$–$1.87\times$ faster than the conventional method, while achieving a higher test accuracy. On the DARTS search space, DARTS+PT using our proposed method can find the best cell in $0.53$ GPU day and allocate $1.6\times$ less memory, while maintaining competitiveness compared to the conventional method. On ImageNet [23], KT-SPOS achieves the same performance as SPOS while consuming less memory footprint.

2. Related Work

Transfer learning. Many studies have shown that pre-trained models with large data such as ImageNet [23] can provide features which are fairly general for various tasks [13,14,15,24]. This characteristic of pre-trained models has been effectively used for enhancing the performance of a target task by remedying overfitting [25], where the target task usually contains small training datasets. In general, knowledge from the pre-trained model with a total of l layers can be transferred by freezing the first i layers of the pre-trained models, and the remaining $l-i$ layers are trained with the target training datasets. The choice of the parameter i depends on the the size of the target dataset and the model capacity and type [26]. In this paper, relying on the generalization power of the features extracted from the pre-trained models, we establish an efficient NAS algorithm.

Neural Architecture Search. The goal of NAS is to automatically discover high-performance networks on a specific task. Reinforcement learning (RL)-based, evolutionary-based, and gradient-based search algorithms are widely used for NAS [5,6,9,10,11]. In the work of [6], the authors use RL and train a controller to generate the network’s configurations (e.g., topology and operations). Evolutionary-based NAS such as [11] outperforms RL-based NAS in terms of accuracy and efficiency as it reaches higher accuracy given the same amount of time during searching. Although these methods show high potential of NAS by outperforming hand-crafted networks, the major drawback of RL-based and evolutionary-based algorithms are the repeated processes of training and evaluation of candidate networks, which demands a huge amount of GPU hours.

On the other hand, gradient-based NAS [10,12,27] utilizes the back-propagation process to find the optimal network, where the optimal model parameters and operations are found during training in an alternative manner. Differentiable architecture search (DARTS) [10] introduces architectural weights $\alpha$ beside the network’s weights w, forming a supernet consisting of large candidate networks. In order to find the optimal $\alpha$ and w, the process requires bi-level optimization [28,29], which is computationally expensive. DARTS gets rid of this issue by approximating the architecture gradient by using only a single training step, which optimizes $\alpha$ and w alternately. In DARTS+PT [22], perturbation-based architecture selection is added on top of DARTS. The authors of DARTS+PT show that the architectural weights $\alpha$ do not represent the strength of the operations. Thus, they introduce an alternative way to derive the final architecture, which relies on the contribution of the operations to the supernet’s accuracy. Specifically, after the supernet converged, the operation which has less impact on the supernet’s accuracy is removed from the supernet. Then, the supernet is tuned for some epochs and the process is repeated until a stopping criterion is met (e.g., becoming a network with a single path).

One-shot NAS [7,30] trains a supernet by randomly activating some paths or only one path, and uses the weights from the one-shot model to rank the candidates. The best candidate is chosen by running a random or evolutionary search. Another line of work on NAS focuses on the search space [31,32,33].

Pre-trained Backbones in NAS. Some recent NAS methods leverage pre-trained backbones to reduce search cost or improve performance. For example, BM-NAS selects and fuses features extracted from pre-trained unimodal backbones and focuses on searching feature selection and fusion strategies, while the backbone networks themselves are fixed and not part of the search space [34]. In a different direction, BO-based NAS methods pretrain surrogate models or predictors to guide the search more efficiently, while each candidate architecture is still fully trained during evaluation [35]. In contrast, KT-NAS directly replaces and freezes shallow blocks of the candidate architectures during the NAS, thereby eliminating their optimization cost rather than accelerating it through surrogate guidance or feature fusion.

3. Analysis on Stability and Similarity of Representations

In order to take advantage of knowledge transfer in NAS, it is crucial to understand (i) how features evolve during training and (ii) how similar the features generated by different networks are. While previous works [36,37] mainly focus on manually designed networks such as ResNet, we made the first step to study these properties for NAS-based networks from two different perspectives, i.e., the fidelity of output features in RMSE and the similarity of neural network representations in CKA.

3.1. Stability of Representations During Training

A simple way to measure the stability of the feature maps during training is to calculate the distance between the consequent features along the epoch. We use RMSE to track the change in low-level, mid-level, and high-level feature maps (in this paper, we consider the features generated at the first, second, and third stage of a neural network as low-level, mid-level, and high-level features) along the training epochs. It is noted that we do apply additional processing to the feature maps. Measuring the RMSE over the entire dataset requires lots of memory to store the feature maps. Thus, we calculate the RMSE over a mini-batch and take the average of all RMSEs over all batches.

Let $f^e$ and $f^{e+1}$ be the feature maps generated by a neural network at current epoch e and next epoch $e+1$. The change in feature maps at layer l is defined as:

$${\mathrm{RMSE}}_l={\displaystyle \frac{1}{b}}\sum _{k=1}^b\sqrt{{\displaystyle \frac{{\sum }_{m=1}^n{(f_{km}^{e+1}-f_{km}^e)}^2}{n}}}$$

(1)

where $f_{km}^e$ is the feature maps with n elements of $kth$ minibatch, and b is the total minibatch.

Setup. We now measure the change in the feature maps in the training phase using Equation (1). We randomly select 200 networks from NAS-Bench-101 (NB101) and NAS-Bench-201 (NB201) on CIFAR-10 (C10), CIFAR-100 (C100), and ImageNet-16-120 (ImageNet-16), and train them for 4, 12, 36 epochs for NB101 and 12, 30, 60 for NB201. The hyper-parameters are kept identical as used in NB101 and NB201. In addition, we set the initial convolution channels to 32 instead of 128 for NB101 due to memory issue. We average the RMSE over 200 networks and plot them in Figure 2.

Observation. As shown in Figure 2, low-level features always obtain a lower RMSE value than mid-level and high-level features in the training phase. Mid-level and high-level features require longer training epochs to reach the same RMSE value as low-level features. This phenomenon is consistent over different search spaces and datasets. This suggest that one can freeze cells that generate these low-level features after training the network for some epochs to reduce the training time.

3.2. Similarity of Representations over Architectures

Recent advances have highlighted the increasing interest in analyzing the similarity of neural network representations [36,37,38,39,40]. In particular, Kornblith et al. [36] introduced Centered Kernel Alignment (CKA), a powerful metric for quantifying the similarity between the representations learned by two neural networks. We refer the reader to the original work for full methodological details. In this subsection, we take an initial step toward understanding the CKA relationship between hand-crafted architectures and NAS-generated architectures.

Hand-crafted architectures. We consider three manually designed architectures: VGG, ResNet, and DenseNet. As shown in

Table 1. Macro-level skeleton of hand-crafted architectures.

Name	Operation	Repeat
Init Conv	3 × 3 conv	1
Stage 1	BLOCK	n
Reduce	RBLOCK	1
Stage 2	BLOCK	n
Reduce	RBLOCK	1
Stage 3	BLOCK	n
Output	FC	1

, all models follow a unified macro-level skeleton consisting of an initial convolution, three stages of repeated blocks (BLOCK), two reduction blocks (RBLOCK), and a final fully connected classifier:

For VGG, BLOCK is BN–3 × 3 conv and RBLOCK is max pooling. For ResNet, both BLOCK and RBLOCK follow a two-layer residual unit: BN–ReLU–3 × 3 conv–BN–ReLU–3 × 3 conv, where RBLOCK uses stride 2 for downsampling. For DenseNet, BLOCK is BN–ReLU–3 × 3 conv followed by feature concatenation, and RBLOCK is BN–1 × 1 conv followed by a 2 × 2 average pooling layer. We set $n=3$ for NB101/DARTS and $n=5$ for NB201, using an initial channel width of 16 for NB101/NB201 and 64 for DARTS.

NAS-based architectures. We randomly sample 200 architectures from NAS-Bench-201. Weights for these networks are retrieved from the official repository whenever available; if not available, we train the models ourselves using the same hyper-parameters as in NAS-Bench-201.

Training protocol. We evaluate representation similarity under three training regimes with a total budget of 12, 30, and 60 epochs, respectively. For each training protocol, we compute CKA between each hand-crafted model and each of the 200 NAS-based networks. The final similarity score is obtained by averaging the 200 pairwise CKA values.

Observation. As demonstrated in Figure 3, it can be observed that the some of the first few layers of the manually designed and NAS-generated networks tend to have a high CKA (over $0.8$). This phenomenon is consistent over different proxy training setups (e.g., total epochs of 12, 30, 60), hand-crafted networks, and datasets. In general, this suggests that shallower layers learn generic representations over architectures.

4. Proposed Method

Figure 4 shows the overviews of the proposed method and the conventional NAS-based network, where the main difference comes from the first few layers. That is, contrary to conventional NAS networks, the proposed method exploits the features from the pre-trained layers for searching.

Let $f_{\theta }=f_l\circ ···\circ f_1$ be the network function with parameters $\theta$ and $l\in \mathbb{N}$ layers, which maps an input vector $x_i$ to output vector $y_i$. The proposed network $h_{\theta }$ function is given by:

$$h_{\theta }=h_l\circ ··\circ h_{l-i-1}\circ f_i\circ ··\circ f_1$$

(2)

The network function of the proposed method is composed of two different network functions. One can train the above network function and evaluate it on a test set. We note that the pre-trained functions $f_1,\dots ,f_i$ are kept fixed during the search process and are not adapted or optimized. Therefore, the proposed method does not involve any meta-objective, task-level adaptation or episodic training, and is fundamentally different from meta-learning-based feature reuse approaches.

Based on our empirical evidence that shallower layers tend to reach stability faster than others and learn similar representations to hand-crafted networks, we propose to leverage the pre-trained features to improve the efficiency of NAS. The choice of i controls how many feature maps that $h_{\theta }$ obtains from $f_{\theta }$. The proposed method can be generally used for various NAS algorithms in a plug-in manner. In this paper we perform experiments by plugging our method into three representative NAS algorithms, i.e., (i) query-based NAS; (ii) gradient-based NAS; and (iii) one-shot NAS. Here, we formally describe how to use the proposed method for the conventional NAS algorithms in the search phase.

Query-based NAS. In many query-based NAS algorithms [6,11], a large number of child networks are sampled and trained for limited epochs. Instead of training these networks, which requires computing the gradient for all cells, we replace them with the proposed paradigm and keep other settings as default. Let $\mathcal{A}$ be the search space and $p_{\theta }^{*}=p_l\circ ···\circ p_1$ be an architecture. Instead of training $p_{\theta }^{*}\in \mathcal{A}$, we train $p_{\theta }=p_l\circ ··\circ p_{l-i-1}\circ f_i\circ ··\circ f_1$, where $f_i$ is the feature maps obtained from a pre-trained architecture $f_{\theta }$ and is shared for all networks in $\mathcal{A}$. The performance of $p_{\theta }$ on the validation set is used for updating the controller in [6] or selecting the parent for mutation in [11].

Gradient-based NAS. Differentiable architecture search [10,12] trains a supernet S from which subnetworks are generated. The conventional supernet is formed by stacking cells. We propose to utilize pre-trained features extracted from a hand-crafted network for the first few cells while training others. Once the supernet is converged, the best cell is obtained by keeping the top-k strongest operation.

One-shot NAS. Different from gradient-based NAS, one-shot NAS [7,30] trains a supernet S where a portion of available paths [30] are activated or only one path is enabled [7]. Once the search phase finishes, we run a random search [30] or evolutionary search [7] to find the best network, where the weights of candidate networks are obtained from the trained one-shot model. To reduce the search cost, we propose to leverage pre-trained features for shallower layers and only search for deeper layers. In the evaluation phase, for layer-wise searches such as SPOS [7], we use the same layer (same operation and topology) as used for the supernet for the first few layers. For the cell-based type, we stack the best cell × N times assuming that the best cell here is mostly enough for extracting low-level features.

5. Experiments

In this section, we demonstrate the effectiveness of our proposed method when incorporating the proposed method into various NAS algorithms. Unless stated otherwise, the training time and memory footprint are measured on a single Nvidia Geforce 1080 Ti with a PyTorch v1.13.0 deep learning framework.

5.1. Result for Query-Based NAS Algorithm

5.1.1. Setup

As demonstrated in Figure 4, our proposed method uses several features from pre-trained networks. There are two major questions for our proposed method. First, which network is suitable for extracting features? Second, how many cells can be replaced? To answer these questions, we perform the experiments using different hand-crafted networks and numbers of replaced blocks.

Dataset. We use NAS-Bench-101 (NB101) [18] and NAS-Bench-201 (NB201) [17]. NB101 has 423k networks trained on C10. There are three operations in the search space: 3 × 3 convolution, 1 × 1 convolution, and 3 × 3 max-pool. NB201 consists of 15.6k networks, where five operations exist in the search space, namely none, 3 × 3 avg-pool, 1 × 1 convolution, 3 × 3 convolution, and skip-connect. These networks are trained on C10, C100, and ImageNet-16. The networks in NB101 and NB201 are cell-based structures. The cell is stacked 9 times for NB101 and 15 times for NB201 (in total). The baseline chosen for comparison is the conventional one, which trains for 12 epochs.

Pre-trained architectures. We consider three hand-crafted networks, namely VGG [19], ResNet [20], and DenseNet [21]. We choose these architectures because these networks achieved SOTA performance on ImageNet at the time introduced. In addition, they are different in their building blocks. For training these networks, we use the same training procedure, which is used in the training phase, from NB101 and NB201 for a fair comparison.

Search algorithm. We compare the performance of Regularized Evolution REA [11] using the conventional network and the proposed method on C10, C100, and ImageNet-16. The initial population is 10 with a sample size of 10, as it is in NB201. We perform the search phase on the target dataset and terminate the search step when the number of evaluated networks exceeds 100. For the proposed method, we measure the performance of REA with different levels of pre-trained features. We consider four cases, where i in Equation (2) is set to 1, 3, 5, 7 on NB101 and 1, 5, 7, 11 on NB201 for Case 1, 2, 3 and 4, respectively. For example, setting i to 5 means we replace the first five NAS-based cells with the first five hand-crafted pre-trained blocks. We train the remaining cells using the same hyper-parameter as used in NB101/NB201 with six additional warmup epochs for stabilizing learning. Once the search phase is finished, we query the performance of the best cell on NB101/201. We repeated the experiments 30 times on C10/C100, and 10 times on ImageNet-16 with different random seeds. We show the average test accuracy in Figure 5.

5.1.2. Results and Analysis

Performance comparison. As shown in Figure 5, REA achieves higher performance in test accuracy compared to the baseline when we replace some of the first cells with pre-trained blocks. In addition, we observed that, on NB201-C10, REA obtains the highest test accuracy when we replace the first cell with the first pre-trained VGG. When we increase the number of pre-trained blocks, the performance starts decreasing. If we use too many pre-trained blocks (e.g., 11 pre-trained blocks), the performance drops significantly and is much lower than the conventional method. Also, we observe similar trends for other pre-trained models such as ResNet and DenseNet. Likewise, on C100 and ImageNet-16, the aforementioned behavior was consistent wiith C10 for all pre-trained models. Therefore, it can be concluded that low-level features extracted from pre-trained networks (e.g., VGG, ResNet, DenseNet) can help to improve the performance of the NAS-based networks regarding test accuracy. For further analysis, we plot the search trajectory of the average test accuracy, where we use the features obtained from the 5-th layer of a pre-trained model, Figure 6. One can see that the proposed method is on par with the conventional method.

Memory comparison. Since we utilize pre-trained features from some networks, we do not need to allocate the memory footprint for calculating the gradients for these pre-trained blocks. Thus, the proposed method has an advantage in saving memory footprint during the search phase. To illustrate this point, we report the allocated memory (average over the entire networks from the search space) on NB201 and show the results in Figure 7. As shown in Figure 7, the proposed method requires less memory footprint than the conventional approach. The memory footprint is reduced at the cost of a degradation in test accuracy. However, one can save roughly 50% memory footprint without accuracy loss (when five pre-trained blocks are used).

Training time comparison. The training time of a random network sampled from NAS-Bench-201 is displayed in Figure 8. We can see that the proposed method requires less time to finish one epoch. As a result, we can increase the number of epochs for training the proposed method as long as it does not exceed the training time of the conventional method for a fair comparison. In addition, we observe a similar trend for other datasets (e.g., CIFAR-10 and ImageNet-16-120) and networks.

We show the training time for one epoch when a different number of pre-trained blocks are used in Figure 9. We can see that training fewer cells does reduce the training time. However, using too many pre-trained blocks degrades the performance of REA because the subnetwork tends to be dissimilar compared to its original structure and thus cannot reflect its true performance. These results show that replacing only the first stage (i.e., the first five blocks) achieves the best trade-off between accuracy and search cost.

5.2. Results for Gradient-Based NAS Algorithm

We shift our evaluation to another type of NAS algorithm, which uses the gradient to guide the search. For this, we incorporate the proposed method into DARTS+PT [22].

Supernet. The original supernet is a cell-based paradigm, which repeatedly stacks cells. Based on our experimental result in Section 5.1.1, we perform the experiment with DARTS+PT when i is set to 5 on the NB201 search space and 2 on the DARTS search space. Specifically, our KT-Supernet is formed by replacing the first stage of the original supernet with those from a pre-trained hand-crafted network.

Training setup. We use the baseline hyper-parameters to ensure a fair comparison. Specifically, we train KT-Supernet and original supernet for 50 epochs. For pre-trained architectures, we train VGG, ResNet, and DenseNet for 50 epochs. We warmup our supernet for 6 epochs at the beginning. Once the training of the supernet is finished, we perform architecture selection followed by DARTS+PT. After the operation is removed from the supernet, we tune the supernet for 10 and 5 epochs for the NB201 and DARTS search spaces, respectively. The search phase is finished when all edges are processed. Additionally, we enable Cutout [41] during training and tuning of the proposed supernet and the original one.

Results. On NB201, we run the experiments 25 times on C10/C100 and 10 times on ImageNet-16 with different random seeds and report the mean test accuracy in

Table 2. Test accuracy using DARTS+PT on NAS-Bench-201.

Dataset	Original	Pre-Trained Architecture
		VGG	ResNet	DenseNet
C10	$93.04$	$93.28$	$93.86$	$93.60$
C100	$69.88$	$70.50$	$69.43$	$70.82$
ImageNet-16	$43.87$	$45.68$	$44.69$	$44.39$

. We can see that the proposed method achieves a higher test accuracy than the original method, except for ResNet on C100. We plot the trajectory of test accuracy during searching and report the allocated memory footprint in Figure 10. Using the pre-trained features, we can save roughly $2\times$ memory footprint and finish the search phase faster (around $1.87\times$ on C10/C100 and $1.4\times$ on ImageNet-16) than the conventional method while achieving a higher test accuracy.

On the DARTS search space, we run the experiment four times with different random seeds and report the average (best) test error in

Table 3. Performance comparison on DARTS search space using pre-trained ResNet.

Method	Test Error	Params	Search	GPU
	(%)	(M)	Cost ⋆	Mem *
DARTS+PT	$2.78$ ($2.68$)	$2.93$	$0.82$	$8.12$
DARTS+PT+Cutout	$2.73$ ($2.61$)	$3.68$	$0.85$	$8.12$
DARTS+PT+KT	$2.94$ ($2.70$)	$2.48$	$0.50$	$4.64$
DARTS+PT+KT+Cutout	$2.68$ ($2.60$)	$3.07$	$0.53$	$4.64$

* in GB. ^⋆ GPU day.

. The result of DARTS+PT is obtained by running the official code provided by the authors. From Table 3, we can see that the proposed method works well on a larger search space. The conventional DARTS takes $0.4$ days to finish the search phase and $0.85$ days when applying perturbation-based architecture selection. Using the proposed method, DARTS+PT finishes the search phase in $0.53$ days and allocates $1.6\times$ less memory while maintaining performance.

5.3. Result for One-Shot NAS Algorithm

We verify the effectiveness of KT-NAS on the one-shot NAS algorithm. We select the Single Path One-Shot algorithm [7] (SPOS) as the baseline. SPOS is a layer-wise search algorithm that finds the optimal choice for every block. There are four choices in each block: a Shuffle block with a kernel size of 3, 5, or 7, and an Xception block. Thus, after searching, the best candidate network may have different choice blocks for different layers.

Setup. As suggested by our findings that earlier layers tend to learn similar representations, and to demonstrate the effectiveness of knowledge transfer, we select the Shuffle block with a kernel size of 3 for the first stage (which has four blocks) and search for the optimal choice in other stages. The total block is 20. During training the supernet, the weight for the first stage comes from a pre-trained ShuffleNet-v2 (kernel size of 3). Once the search phase is finished, we train the best network from scratch (not using pre-trained features).

Result. We follow the same search/evaluation procedure and use the same hyper-parameters as SPOS and report the result on the ImageNet dataset (average over 2 runs) in

Table 4. Top1-Acc (%) on ImageNet (2nd col), mAP (%) on COCO dataset (3rd col), and supernet training memory (in GB) (4th col). All networks have 20 blocks and their FLOPs are under 330 M. DetNAS searches for best backbone while others are transferred. The training mem of supernet is measured on one GPU with batch 128.

Method	ImageNet	COCO-FPN	GPU Mem
ShuffleNet-v2	$73.4$	$34.77$	-
SPOS †	$74.3$	-	$7.87$
ClsNASNet †	$74.3$	$35.1$	-
DetNAS †	73.9–74.1	$36.6$	-
KT-SPOS (Ours)	$74.3$	$36.2$	$5.31$

^† Obtain from the original paper.

, 2nd col. KT-SPOS achieves a similar performance as SPOS while requiring less memory footprint. SPOS finds the optimal choice for each block; however, SPOS does not outperform KT-SPOS. This suggests that from the feature space perspective, any choice (among the four choices) is sufficient for the first stage since these layers develop similar representations.

We further validate the transferability of the searched network on the object detection task. We select the COCO dataset and FPN detector. We use the same training setups as used in DetNAS [42]. As shown in the 3rd col of Table 4, our searched network has a high mAP compared to DetNAS. This confirms that our searched model found by KT-SPOS has good transferability.

5.4. Sanity Test for Knowledge Transfer

Since the proposed method leverages several features from a pre-trained network, we perform two sanity tests to verify how much the knowledge from the pre-trained model helps to improve the performance. In the first test, we perform REA with two different scenarios. In the first scenario, REA is performed in the original manner, where the only difference is that the first few layers in the NAS-based networks are frozen during training. In the second scenario, REA is performed with our KT-NAS. The number of pre-trained (or untrained) blocks is set to 5. We use the NB201 dataset. In the search phase, we train the networks for 12 epochs (without warmup). Other hyper-parameters are kept identical to those in NB201. As shown in

Table 5. Effectiveness of the knowledge from the pre-trained models of KT-NAS in test accuracy.

Dataset	Untrained	Pre-Trained Architecture
		VGG	ResNet	DenseNet
C10	$93.5$	$93.82$	$93.85$	$93.88$
C100	$71.17$	$71.92$	$72.14$	$71.81$
ImageNet-16	$38.74$	$45.14$	$46.00$	$45.97$

, REA in the second scenario outperforms the REA in the first scenario. The results are consistent over different pre-trained networks and datasets. One can point out that as the dataset becomes more complicated (from C10 to ImageNet), the effectiveness of the knowledge transferred from the pre-trained model in NAS increases.

For the second test, we verify whether REA still performs well when using the pre-trained model trained on a different task from the target task. For example, we train ResNet on C10 and use this model for extracting features on C100 and ImageNet-16. We also test on the Street View House Numbers (SVHN) [43] dataset, which only contains digits. We warmup the NAS-based networks for 6 epochs and train for 12 epochs. As shown in

Table 6. Performance comparison when pre-training ResNet on different datasets.

Dataset	Pre-Trained Dataset
	C10	C100	ImageNet-16	SVHN
C10	$94.04$	$93.99$	$93.92$	$93.97$
C100	$72.10$	$72.09$	$72.25$	$72.01$
ImageNet-16	$45.82$	$46.24$	$46.02$	$44.78$

, although the hand-crafted networks are trained on a different task from the target task, the performance of REA remains competitive. This suggests that shallower layers can have the ability to generate generic representations across different datasets.

6. Visualization of Searched Network

6.1. On KT-DARTS

We show the searched cell found by DARTS+PT+KT in

Table 7. Searched cell on DARTS search space using C10 dataset.

DARTS+PT+KT
Normal Cell	Reduction Cell

. By leveraging the knowledge from pre-trained features during supernet training, the best cells found by the proposed method achieve similar performance to the conventional approach while reducing the search cost (memory footprint and search time). It is noted that we do not use pre-trained features in the evaluation phase. The final network is formed by stacking the best cell × N times and training from scratch (on the DARTS search space) or obtaining it from NAS-Bench (on the NAS-Bench-101/201 search space).

6.2. On KT-SPOS

We report the two architectures found by KT-SPOS in Figure 11. Although we use the Shuffle block with a kernel size of 3 for the first four blocks and search the optimal choice for other blocks (deeper layer), the top-1 accuracy of these models on ImageNet is similar (i.e., $74.3$%) to that of those which perform the search for every block (the conventional method). Thus, one can leverage knowledge transfer to reduce the memory footprint during the search phase.

7. Limitations and Future Work

While the proposed method demonstrates effective acceleration of NAS by integrating pre-trained blocks with untrained cells, several limitations remain. First, the current design relies on empirical observations to select the replacement depth; developing principled similarity or stability metrics to automatically determine which and how many layers should be replaced is an important direction for future research. Second, the effectiveness of KT-NAS depends on the quality and generality of the pre-trained model, highlighting the need for robust and transferable pre-trained baselines that can serve as reliable starting points across diverse NAS settings and related tasks. Third, gaining deeper insights into supernetwork behavior under pre-trained layer replacement—particularly through loss landscape analysis—remains an important direction for future work. Finally, while the current study focuses on widely adopted backbone architectures, evaluating the generalization to more recent designs such as ConvNeXt and EfficientNet is a promising avenue for future research.

Building upon these observations, the present work focuses on image classification; extending KT-NAS to other domains, including object detection, image segmentation, and natural language processing, constitutes the natural next step. Moreover, we plan to investigate its applicability to transformer-based search spaces and to integrate KT-NAS with hardware-aware NAS frameworks by incorporating deployment-related objectives, such as latency, into the architecture search process.

8. Conclusions

We propose knowledge transfer-based NAS, where we extend the applicability of pre-trained models to accelerate NAS algorithms. The proposed method is established based on our extensive preliminary experiments. Consequently, the proposed method has two advantages of saving memory footprint and reducing training time. Extensive experiments on different search spaces, NAS algorithms, and datasets confirm the effectiveness of the proposed method.