Improving Heterogeneous Network Knowledge Transfer Based on the Principle of Generative Adversarial

Abstract

Deep learning requires a large amount of datasets to train deep neural network models for specific tasks, and thus training of a new model is a very costly task. Research on transfer networks used to reduce training costs will be the next turning point in deep learning research. The use of source task models to help reduce the training costs of the target task models, especially heterogeneous systems, is a problem we are studying. In order to quickly obtain an excellent target task model driven by the source task model, we propose a novel transfer learning approach. The model linearly transforms the feature mapping of the target domain and increases the weight value for feature matching to realize the knowledge transfer between heterogeneous networks and add a domain discriminator based on the principle of generative adversarial to speed up feature mapping and learning. Most importantly, this paper proposes a new objective function optimization scheme to complete the model training. It successfully combines the generative adversarial network with the weight feature matching method to ensure that the target model learns the most beneficial features from the source domain for its task. Compared with the previous transfer algorithm, our training results are excellent under the same benchmark for image recognition tasks.

1. Introduction

The objective of deep learning models is to optimize the function of large datasets. Models can learn an optimal function using the general process described above. It is foreseeable that traditional machine learning processes will be stretched when the number of tasks is very large or the learning process is very slow. How to maximize the use of previously learned tasks to help the learning of new tasks? How to achieve a better learning effect on the target task on a small network? The initial method uses a learned source task to apply it directly to the target task, and the model achieves the learning of the target task through fine-tuning on the target task [1]. This has been proven to be an effective way. The task of transfer learning is to reduce the amount of data and improve the generalization of the model, so that it can quickly converge and transfer other tasks with a small amount of training data. At present, a lot of work has made a good performance for the homogeneous transfer networks, from the early method of fine-tuning [1] to the fixed network feature extraction layer, and where the learning distance is increased in the classification layer [2,3] through domain adversarial [4,5,6,7,8] idea of learning implicitly distributed distances; therefore, deep transfer learning methods have quickly become an active research field. If the network architectures of the source task and the target task are quite different, there exists no direct method of fine-tuning. A general algorithm is needed to enable the heterogeneous network to complete the transfer. Nowadays, there are several earlier works [9,10,11,12,13], which can be applied to the challenging scenario of knowledge transfer between heterogeneous models and tasks: Attention transfer [12] and Jacobian matching [13] use attention maps generated from feature maps or Jacobians for transferring the source knowledge. L2W-FF [9] further implements the matching rules of knowledge transfer in an automatic way instead of manually adjusting the transfer configuration considering the differences in architecture and tasks between the source and the target domain. Our motivation is that these heterogeneous network transfers are currently mainly driven by feature matching traditional algorithms. We believe that we can use generative adversarial ideas and use network-driven networks combined with weight feature matching to more effectively perform knowledge transfer. Deep neural networks are more powerful for learning general features and transferable features, but some experiments [1] prove that deep features must eventually transition from general features to specific features along the network. As the domain difference increases, feature transfer ability drops significantly in higher layers. Our research topic is to realize the transfer from source task to target task around heterogeneous network. According to previous theories [1], we divide the network layer characteristics into two parts, namely general feature layer and specific feature layer. In our experiments, the general feature layer refers specifically to the low-level convolutional layer and the specific feature layer refers to the high-level fully connected layer. Our new method is to use a combination of generative adversarial network, and feature matching for different network feature maps to improve transmission capacity. The experimental results on benchmarks are excellent. Our contributions are as follows.

We use generative adversarial thinking methods to achieve transfer to heterogeneous networks. As a result of the heterogeneity of the network, we apply the point-to-convolution layer to the target domain network to complete the linear transformation of the target domain feature map and realize the domain discriminant network drive that incorporates the principle of generative transfer.

We successfully combine the generative transfer network with the weight feature matching method and propose a new objective function optimization scheme to complete the model training, ensuring that the target model learns the low-level features that are most beneficial to its own task from the source domain.

2. Related Work

We review two dominant research directions for transfer learning.

2.1. Feature Matching

Feature matching can be understood as a linear transformation, which is a good method in transfer learning. There are also many related researchers in the field of transfer learning exploring feature matching. At the beginning, it was only manual layer-to-layer matching of heterogeneous networks [10,11,12]; however, there are unavoidable disadvantages with this method. One reason is that too many extra operations will be added. Another reason is that the transferred knowledge is not necessarily available for the target task, which may cause the knowledge that is not conducive to the target task to be transferred into it, thereby reducing the transfer effect. Later, some scholars [9] proposed feature matching with additional weights and realized a method of automatically matching features based on weights. This method mainly updates the weight of the feature matching layer by continuously measuring the distance between the target domain and the source domain, thereby increasing the transfer of useful knowledge and weakening the transfer of knowledge with little relevance. Our model is based on predecessors, using weight feature matching and generative adversarial network mechanisms to improve the effect of knowledge transfer.

2.2. Generative Adversarial Net

The generative adversarial network is composed of a generative network and a discriminant network. The generator is used to generate fake samples, and the discriminator is used to distinguish between true and false simples. The two game each other until the system reaches a Nash equilibrium. In transfer learning, there is a source domain and a target domain. The target domain can be directly assumed as the sample generated by the generator. The original generator is responsible for extracting features and continuously learning the knowledge of the source domain data, making the discriminator unable to distinguish between the two-domain data. From [10] and others, the idea of adversarial was first introduced into the field of transfer learning, mainly used for the adaptive problem of an important branch of transfer learning. Domain adaptation focuses on the same feature space. Given a labeled source domain $D_s$ and an unlabeled target domain $D_t$, it is assumed that their feature space and category space are same, but the edge distribution of the domain is different. The labeled source domain data is used to predict the target domain label. Subsequently, there are many different applications and productions in the field of transfer learning, such as image attribute transfer [14] and super-resolution image reconstruction [15]. These are all domain adaptation issues, since domain adaptation is to transfer knowledge with the same feature space, category space and the homogeneous network. Its core function is to adapt the feature distribution of the target domain to the source domain feature distribution, thereby completing the domain feature in-variance. In this paper, we propose a new heterogeneous transfer network, which combines the idea of generative adversarial to perform transfer learning on the heterogeneous network and linear transformation of target domain features. The domain discriminator is used to drive the common layer characteristics of the source domain and the target domain. This can make the target domain more effective and accurate in order to learn the common layer characteristics of the source domain.

The rest of the paper consists of the following parts. In Section 3, we describe our heterogeneous transfer network structure principle and training method. In Section 4, we show the experimental results and evaluations under different configurations. Section 5 explains the conclusion.

3. Our Approach

3.1. Motivation

According to the experiment of previous research [1] in the neural networks, as the domain difference increases, the features learned by the network are gradually proprietary, which means that the transfer ability will significantly decrease as the number of network layers deepens. We divide the network layer features into two parts, general feature layer and specific feature layer. In our experiments, the general feature layer refers specifically to the low-level convolutional layer and the specific feature layer refers to the high-level fully connected layer. We propose a novel transfer network for transfer training. Our goal is to use a combination of generative adversarial nets (GAN), distribution adaptation (this part mainly pays attention to the sample of the same feature and category spaces, same conditional probability distribution and the different edge distributions), and feature matching for different network feature layers to improve the transfer effect. And the generality of the model is proved through testing and evaluation on different general data sets. Our novel method is shown in Figure 1. Deep learning has strongly developed in the two major areas of natural language processing and computer vision [16,17,18,19,20]. Here we mainly use the convolutional neural network commonly used in the field of computer vision as the experimental model of transfer to verify the migration. This method is suitable for convolutional neural networks, but it is not limited to this.

Figure 1. (a) Representation of Heterogeneous transfer network model. (b) Schematic representation of weight feature matching: Linear transformation ${\phi }_{\theta }$ the target domain feature map. $w_c^{m,n}$ denote the matching weight of the c-th channel between the feature map of the m-th layer of the source network and the feature map of the n-th layer of the target network after linearly transform ${\phi }_{\theta }$.

In Section 3.2, we describe the transfer process based on generative adversarial networks (the main function is the domain offset of the middle layer of the network), and Section 3.3 focuses on weight feature matching. In Section 3.4, the domain adaptation method of the high-level network is described. In the last Section 3.5, we specifically describe the experimental training process of our model.

3.2. Generative Adversarial Nets

The goal of traditional GAN is to generate training samples. Since there is a source domain and a target domain naturally in transfer learning, we can avoid the process of generating samples and directly treat the data of the target domain as the generated samples. At this time, the function of the generator changes and does not generate new samples; it plays the function of feature extraction: learning the characteristics of the domain data continuously, making the discriminator unable to distinguish between the two domains. In this way, the original generator can also be called a feature extractor.

A discriminate mechanism is added to the training of the neural network. The goal is to make the discriminator unable to distinguish the difference between the two fields, continuously promote the knowledge transfer of the target domain network and accelerate the driving of the target network to learn the common characteristics of the source domain and the target domain.

Traditional transfer problems generally use fixed feature representations, but the adversarial transfer network in this paper focuses on how to select transferable features between different domains and make tidy target networks learn knowledge through source network more accurately and quickly. In other words, a good transferable feature should meet two conditions: firstly, in the face of these features, it is impossible to distinguish whether they come from the target domain or the source domain; secondly, using these features to complete the classification task better. Therefore, the network loss consists of two parts: training loss (label predictor loss) and domain discriminate loss [21].

We further define a discriminator network $D_{{\theta }_d}$, a source domain network $S_{{\theta }_s},$ and a target domain network $T_{{\theta }_t}$. ${\theta }_d$, ${\theta }_s,$ and ${\theta }_t$ represent the parameters of the $D_{{\theta }_d}$, $S_{{\theta }_s},$ and $T_{{\theta }_t}$, respectively. Our ultimate goal is to be able to predict labels $y^t$ given the input $I^t$ for the target distribution. We assume that the model works with input samples $I^t \in T$, where T is input space and certain output of $y^t$ from the label space $Y^t$. For sample $I_n^t$, n = 1, ..., N, we describe the sample by a real-valued tensor of size W × H × C. We assume that there are two distributions $P_S \left(I^s, y^s\right)$ and $P_t \left(I^t, y^t\right)$ on $X \otimes Y$, which will be referred to as the source distribution and the target distribution. We denote with $d_i$ ($d_i \in \left\{0, 1\right\}$) the binary variable (discriminator output) for the i-th example. If $d_i=1$, it proves that the sample comes from the source distribution $I_i^s~P_s\left(I^s\right)$. If $d_i=0$, it proves that the sample comes from the source distribution $I_i^t~P_t \left(I^t\right)$.

$$mi{n}_{{\theta }_t}ma{x}_{{\theta }_d}V \left(T, D\right)=E_{I^s~P_s\left(I^s\right) }\left[\log D_{{\theta }_d}\left(I^s\right)\right]+E_{I^t~P_G\left(I^t\right)}\left[\log (1-D_{{\theta }_d }(T_{{\theta }_t }\left(I^t\right))\right]$$

(1)

$$mi{n}_{{\theta }_t}ma{x}_{{\theta }_d}V \left(T, D\right)=E_{I^s~P_s\left(I^s\right)} \left[\log D_{{\theta }_d }\left(I^s\right)\right]+E_{I^t~P_G\left(I^t\right)}\left[\log (-D_{{\theta }_d} (T_{{\theta }_t} \left(I^t\right))\right]$$

(2)

3.3. Weight Feature Matching

If the convolutional neural network is well trained for the task, its intermediate feature space should have useful knowledge for the task [9,10,11,12,13]. Many predecessors have studied neural network feature matching, some researchers have manually matched the features [10,11,12,13], and some have discerned automatic matching of features [9]. Intermediate feature mapping of the m-th layer of the pre-trained source network is used to mean $S_{{\theta }_s}^m \left(I^s\right)$ and feature mapping of the n-th layer of the target network is used to mean $S_{{\theta }_t}^n \left(I^t\right)$. We minimize the following $l^2$ objective, similar to that used in FitNet [10] and L2t-ww [9] to transfer the knowledge from $S_{{\theta }_s}^m\left(I^s\right)$ to $S_{{\theta }_t}^n\left(I^t\right)$.

$${\Vert {\phi }_{\theta }\left(T_{{\theta }_t}^n \left(I^t\right)\right)-S_{{\theta }_s}^m\left(I^s\right))\Vert }_2^2$$

(3)

This is Equation (3) of the equation: we used pointwise convolution to linearly transform ${\phi }_{\theta }$ the target domain feature map $T_{{\theta }_t}^n\left(I^t\right)$. This process produces parameter $\theta$. We set weights for feature matching between channels to focus on the more closely related channels. We use $w_c^{m,n}$ to denote the matching weight of the c-th channel between the feature map of the m-th layer of the source network $S_{{\theta }_s}^m\left(I^s\right)$ and the feature map of the n-th layer of the target network ${\phi }_{\theta }\left(T_{{\theta }_t}^n\left(I^t\right)\right)$ after linearly transforming ${\phi }_{\theta }$. ${\mathcal{L}}_{fm}^{m,n} \left(\left.\theta \right|I^t,w^{m,n}\right)$ is used to represent the loss function of weight feature matching.

$${\mathcal{L}}_{fm}^{m,n} \left(\left.{\theta }_t\right|I^t,w^{m,n}\right)=\frac{1}{HW}{\displaystyle \sum }_c{w}_c^{m,n}{\displaystyle \sum }_{i,j} {\left({\phi }_{\theta } \left(T_{{\theta }_t}^n\left(I^t\right)\right)-S_{{\theta }_s}^m \left(I^s\right)\right)}^2$$

(4)

3.4. Maximum Mean Discrepancy

This part is mainly for the same feature space of the source domain and target domain, $S=T$, and their category spaces are also the same where $Y^s=Y^t$, and the conditional probability distribution is also the same $P_s \left(Y^s|I^s\right)=P_T (Y^t|I^t)$. But the edge distributions of these two domains are different, $P_S \left(I^s\right) \ne P_T \left(I^t\right)$. It also can be seen as a domain adaptation problem. Domain adaptation is also an important part of the field of transfer learning, and it is commonly used in many unsupervised and less-supervised tasks. When we encounter other types of transfer data, we can manually set the hyperparameter $\gamma$ to 0. For high-level networks, features are the most exclusive, so domain adaptation of high-level networks is inevitable. Most of the documents [2,3] have carried out various transfer experiments on the transfer of high-level features. Here, we use the most widely used MMD measurement criteria to transfer the upper layer such as FC layer. Use the same method as DDC model [2]. This model adds a distance loss to the final classification layer to reduce the use of a kernel function method, the maximum mean discrepancy (MMD), which measures the two distributions of the source domain and the target domain in the regeneration Hilbert. The distance of space is a nuclear learning method. $\varphi \left(·\right)$ is a mapping used to map the original variable to the reproducing kernel Hilbert space (RKHS) [22]. The Hilbert space is complete for the inner product of the function, the reproducing nuclear Hilbert space is a Hilbert space with reproducibility $⟨K\left(x,·\right),K{\left(y,·\right)⟩}_H=K\left(x,y\right)$. After expanding the square, the inner product in the RKHS space is converted into a kernel function, so MMD can be directly calculated by the kernel function.

$${\mathcal{L}}_{MMD} \left(\left.{\theta }_t\right|I^t,\left.{\theta }_s\right|I^s\right)={\Vert \frac{1}{\left|I^s\right|}{\displaystyle \sum }_{I^s\in S}\varphi \left(I^s\right)-\frac{1}{\left|I^t\right|}{\displaystyle \sum }_{I^t\in T}\varphi \left(I^t\right)\Vert }^2$$

(5)

3.5. Model Holistic Training

Our final loss ${\mathcal{L}}_s$ to train a target model is given as follows. In particular, when we train data, the same feature space of the source domain and target domain, $S=T$, and their category spaces are also the same where $Y^s=Y^t$, and the conditional probability distribution is also the same $P_s \left(Y^s|I^s\right)=P_T (Y^t|I^t)$. But the edge distributions of these two domains are different, $P_S\left(I^s\right)\ne P_T\left(I^t\right)$, we should make $\gamma \ne 0$.${\mathcal{L}}_{org} \left(\left.{\theta }_t\right|I^t,y^t\right)$ is the original loss (e.g., cross entropy) in the target model, ${\mathcal{L}}_g\left(\left.{\theta }_t\right|I^t,\left.{\theta }_s\right|I^s\right)$ is the loss of generative adversarial net and $\lambda$, $\beta$ > 0,$\gamma \left(\gamma =0\right)$ is a hyper-parameter:

$${\mathcal{L}}_s={\mathcal{L}}_{org}\left(\left.{\theta }_t\right|I^t,y^t\right)+\lambda {\mathcal{L}}_g\left(\left.{\theta }_t\right|I^t,\left.{\theta }_s\right|I^s\right)+\beta {\mathcal{L}}_{fm}^{m,n}\left(\left.{\theta }_t\right|I^t,w^{m,n}\right)+\gamma {\mathcal{L}}_{MMD}\left(\left.{\theta }_t\right|I^t,\left.{\theta }_s\right|I^s\right)$$

(6)

Firstly, the resulting parameter ${\theta }_t^T$ is learned only using the knowledge of the source model, thus we updated the target model for T times via gradient-based algorithms for minimizing ${\mathcal{L}}_{fm}^{m,n}$ and ${\mathcal{L}}_g$. We designed a new type of training scheme to update the network parameters of feature matching and generative discriminate network emphatically by setting a hyper parameter value (T). The purpose of this process is obvious and important; it enhances the influence of the regularization term ${\mathcal{L}}_{fm}^{m,n}$ and ${\mathcal{L}}_g$. on the target model parameters, and because the source domain data is not used, the target features are completely provided by the source domain. Secondly, we used ${\theta }_t^{T+1}$ from ${\theta }_t^T$ to update and minimize ${\mathcal{L}}_{org} \left(\left.{\theta }_t\right|I^t,y^t\right)$ once. Thirdly, we measured ${\mathcal{L}}_{org} \left(\left.{\theta }_t\right|I^t, y^t\right)$ and updated $w_c^{m,n}$,${\theta }_d$ to minimize ${\mathcal{L}}_{fm}^{m,n}$ and ${\mathcal{L}}_g$. To train the target model, we alternatively updated the target model parameters ${\theta }_t$ and parameters $w_c^{m,n}$ to make the ability of our model. The purpose is to increase the influence of the source domain network on the target domain network training and help the target domain network training quickly. The proposed training scheme is formally outlined in Algorithm 1.

Algorithm 1 Minibatch stochastic gradient descent training of model. Learning of ${\theta }_t$,$w^{m,n}$,${\theta }_d$

Input: Dataset ${\mathrm{D}}_{\mathrm{train}}=\left\{\left(I_i^t,y_i^t\right)\right\},\mathrm{learning}\mathrm{rate}\mathsf{\alpha }$

Repeat Sample a batch ${\rm B}\subset D_{train}with \left|B\right|=B$

Update ${\theta }_t$ to minimize $\frac{1}{B}{\displaystyle \sum }_{\left(I_i^t,y_i^t\right)\in B}{\mathcal{L}}_s$ for t$=$0 to T $-$ 1 do  $${\theta }_t^{T+1}\leftarrow {\theta }_t^T-\alpha {\nabla }_{\theta }\frac{1}{B}{\displaystyle \sum }_{\left(I_i^t,y_i^t\right)\in B}={\mathcal{L}}_{fm}^{m,n}\left(\left.{\theta }_t\right|I^t,w^{m,n}\right)$$ Update the discriminator by ascending its stochastic gradient: $${\nabla }_{{\theta }_d}\frac{1}{B}{\displaystyle \sum }_{\left(I_i^t,y_i^t\right)\in B}\left[\log D_{{\theta }_d}\left(I^s\right)\right.+\left.\log (-D_{{\theta }_d}(T_{{\theta }_t}\left(I^t\right))\right]$$ Update the generator by descending its stochastic gradient: $${\nabla }_{{\theta }_t}\frac{1}{B}{\displaystyle \sum }_{\left(I_i^t,y_i^t\right)\in B}\log (-D_{{\theta }_d}(T_{{\theta }_t}\left(I^t\right))$$ end for ${\theta }_t^{T+1}\leftarrow {\theta }_t^T-\alpha {\nabla }_{{\theta }_t}\frac{1}{B}{\displaystyle \sum }_{\left(I_i^t,y_i^t\right)\in B}{\mathcal{L}}_{org}\left(\left.{\theta }_t\right|I^t,y^t\right)$

Update $w_c^{m,n}$ using $\alpha {\nabla }_{\varphi }\frac{1}{B}{\displaystyle \sum }_{\left(,I_i^t,y_i^t\right)\in B}{\mathcal{L}}_{fm}^{m,n}\left(\left.{\theta }_t\right|I^t,w^{m,n}\right)$

until done

4. Experiments

Our experiments are mainly divided into two parts. The first part is the transfer experiment on the public benchmark [23,24,25,26,27], and compared with the experimental results of the heterogeneous transfer network of the predecessors. The second part obtains the transfer effect of different network layers by using the different parameter regularization methods adopted by our model, and discusses the transfer characteristics and transfer methods of each layer of the network.

4.1. Setup

In order to evaluate our model and other models more easily, we chose classical and universal dataset tests and a backbone with superior performance as our source and target domain heterogeneous network. We performed experiments on 32 × 32 image classification tasks, using the Tiny ImageNet [27] dataset as a source task, and CIFAR-10 CIFAR-100 [28] and STL-10 [24] datasets as target tasks. Tiny ImageNet has 200 classes. Each class has 500 training images, 50 validation images, and 50 test images. The sample size used in this experiment is 32 × 32 × 3. CIFAR-10 datasets have 10 classes. There are 5000 training images. CIFAR-100 datasets have 100 classes. There are 500 training images and 100 testing images per class. The 100 classes in the CIFAR-100 are grouped into 20 super classes. Each image comes with a “fine” label (the class to which it belongs) and a “coarse” label (the superclass to which it belongs). STL-10 [24] datasets have 10 classes. There are 500 training images, with 800 test images per class. The same is the case with L2T-ww [9] we resize them into 32 × 32 when training and testing. We trained 32-layer ResNet [29] on the source tasks and 9-layer VGG [30] on the target tasks. At the same time, we conducted experiments on a deeper target network, training 34-layer ResNet [29] on the source tasks and 19-layer VGG [30] on the target tasks. We performed experiments on 224 × 224 × 3 image classification tasks, and used the ImageNet [23] dataset as a source task, PASCAL VOC2007 [31] and CUB200 [25] datasets as target tasks. In order to reflect the key role played by every part in transfer learning intuitively, we used the MNIST dataset [32] as a source task, and MNIST-M dataset as a target task.

In terms of optimizer settings for network training, all target networks are trained by stochastic gradient descent (SGD) with a momentum of 0.9. We used an initial learning rate 0.1 and 200 epochs for all experiments.

4.2. Results on Different Experiments

We compared our methods with the following prior methods: learning without forgetting (LwF) [11], and attention transfer (AT) [12]. In our experimental setup, every method came from scratch for baselines. Here, these models include the model [9] that can perform automatic feature matching. Attention transfer [12] and Jacobian matching [13] use attention maps generated from feature maps or Jacobians for transferring the source knowledge. L2W-FF [9] further implements the matching rules of knowledge transfer in an automatic way, taking into account the differences in architecture and tasks between the source and the target, without the need to manually adjust the transfer configuration. For small network experiments, the inputted sample size was 32 × 32 × 3. In order to verify the versatility of the model, two different migration tasks were performed: Tiny ImageNet$\to$CIFAR-100 and Tiny ImageNet$\to$STL-10 (see Table 1). For big network experiments, the sample size we inputted was 224 × 224 × 3. We used a pre-trained 34-layer ResNet on ImageNet. In order to verify the versatility of the model, two different transfer tasks were performed: ImageNet$\to$ Pascal VOC2007 and ImageNet$\to$ CUB-200 (see Table 2 and Figure 2).

Table 1. Classification accuracy (%) of small network.

Source Task	Tiny ImageNet
Target task	CIFAR-100	STL-10
Scratch	67.69 ± 0.22	67.69 ± 0.22
AT	69.23 ± 0.09	69.23 ± 0.09
LwF	69.97 ± 0.24	69.97 ± 0.24
L2T-ww	70.96 ± 0.61	70.96 ± 0.61
ours	72.85 ± 0.25	70.99 ± 0.88

Table 2. Classification accuracy (%) of big network.

Source Task	ImageNet
Target task	Pascal VOC	CUB 200
AT	79.22 ± 0.59	44.52 ± 0.09
LwF	79.55 ± 0.64	44.56 ± 0.24
L2T-ww	80.96 ± 0.61	46.96 ± 0.67
ours	83.33 ± 0.64	47.11 ± 0.69

Figure 2. The network detail architecture of MNIST$\to$MNIST-M.

5. Discussion

In order to reflect the key role played by every part in transfer learning intuitively, our task was MNIST$\to$MNIST-M (see Figure 2). This is the same feature space of the source domain and target domain, $S=T$, and their category spaces are also the same where $Y^s=Y^t$, and the conditional probability distribution is also the same where $P_s \left(Y^s|I^s\right)=P_T (Y^t|I^t)$. However, the edge distributions of these two domains are different, where $P_S \left(I^s\right) \ne P_T \left(I^t\right)$. We used the popular MNIST [32] dataset as the source domain, and MNIST-M was created by using each MNIST digit as a binary mask and inverting with it the colors of a background image. The background images are random crops uniformly sampled from the Berkeley Segmentation Data Set (BSDS500) [31].

In order to reflect the effect of knowledge transfer in the source network, we divided the training data and conducted a comparative experiment to control the number of datasets. We used Tiny ImageNet as a source task, and used CIFAR-10 datasets as target tasks. We divided the target domain dataset into five levels. We used $N\in \left\{50, 100, 250, 500, 1000\right\}$ training samples for each class and compared with previous models. The training of each class was done under the same hyperparameters. It can be seen from Figure 3a that our model has a higher accuracy rate with a smaller number of samples, thus has a greater advantage. There are two main reasons. First, compared with other previous methods, we do not use a one-step training gradient update in the training iteration scheme. We designed a new type of training scheme to update the network parameters of feature matching and generative discriminate network emphatically by setting a hyper parameter value ($T$). This increases the influence of the source network on the learning procedure of the target model, since the target features are solely trained without target labels. Secondly, our discriminator mechanism can drive the training efficiency of the target domain network faster, and is more conducive to training with fewer samples. This fully illustrates that our new heterogeneous transfer network is driven by generative adversarial network and weight feature matching is more obvious in transfer knowledge.

Figure 3. (a) Classification accuracy (%) of Tiny ImageNet$\to$CIFAR-10 with varying numbers of training samples per class in CIFAR-10. (b) Classification accuracy (%) of MNIST$\to$MNIST-M. A is the reference.

To research the effectiveness of knowledge transfer between different layers and different transfer algorithms, we adopted the method of controlled variables and designed four comparative experiments. The training of each class is done under the same hyper-parameters. Experiment A is trained under the complete training model system designed by our novel method. The B experimental model cancel the Generative Adversarial Nets, which are used to transfer the general feature layer and middle feature layer. The C experimental model cancel the weight feature matching that has a transfer effect on the middle feature layer. The D experimental model cancel the distribution adaptation that has a transfer effect on the proprietary feature layer. To visualize the difference between the three experiments, we have drawn the experimental results into a line chart (see Figure 3b). We evaluate the results of the experiment. Experiment D has the greatest impact on the model, followed by experiments B and C. This results also verifies the conclusions of some network feature studies [1]. In neural networks, as the domain difference increases, the features learned by the network are proprietary gradually, which means that the portability will decrease significantly as the number of network layers deepens.

6. Conclusions

This paper proposes a new, more optimized, heterogeneous transfer network model, which mainly uses the principle of generative adversarial, as well as the network-driven network to combine weight feature matching for more effective knowledge transfer on middle-level feature maps. We used the knowledge transfer of complex source networks to train simple target domain networks effectively and use less target domain data to complete the training of heterogeneous target networks on the basis of pre-trained complex networks. Progress has been made for improvement in the research of heterogeneous networks and our findings on the field of transfer learning are pivotal.