Frequency-Auxiliary One-Shot Domain Adaptation of Generative Adversarial Networks

Abstract

Generative domain adaptation in a one-shot scenario involves transferring a pretrained generator from one domain to another using only a single reference image. To address the issue of extremely scarce data, existing methods resort to complex parameter constraints and leverage additional semantic knowledge from CLIP models to mitigate it. However, these methods still suffer from overfitting and underfitting issues due to the lack of prior knowledge about the domain adaptation task. In this paper, we firstly introduce the perspective of the frequency domain into the generative domain adaptation task to support the model in understanding the adaptation goals in a one-shot scenario and propose a method called frequency-auxiliary GAN (FAGAN). The FAGAN contains two core modules: a low-frequency fusion module (LFF-Module) and high-frequency guide module (HFG-Module). Specifically, the LFF-Module aims to inherit the domain-sharing information of the source module by fusing the low-frequency features of the source model. In addition, the HFG-Module is designed to select the domain-specific information of the reference image and guide the model to fit them by utilizing high-frequency guidance. These two modules are dedicated to alleviating overfitting and underfitting issues, thereby enchancing the diversity and fidelity of generated images. Extensive experimental results showed that our method leads to better quantitative and qualitative results than the existing methods under a wide range of task settings.

1. Introduction

Generative adversarial networks have undergone tremendous development in the past few years [1,2,3,4] and occupy an important position in the field of image generation. Until now, according to different task settings, GANs have been widely used in image editing [5,6,7], image super-resolution [8,9], image inpainting [10,11], and image translation [12,13].

However, behind the widespread application of GANs, large-scale datasets are usually required to support them, such as 70 K FFHQ [3], which will greatly limit the usage scenarios of GANs. Insufficient data can easily lead to overfitting in GANs, so training GANs on a small number of samples is a challenging task. Although there are some studies on data-limited methods [14,15,16,17] that efficiently utilize training samples and can alleviate the problem of insufficient datasets to some extent, they still require at least a few hundred training images.

In order to address the situation of scarce datasets, a domain adaptation method [18,19,20,21,22] has been proposed. The main idea of this task is to use a model pretrained on a large-scale source domain dataset as the model’s initialization parameter and fine-tune it on few-shot target domain images. This type of method can effectively generate diverse and high-quality images with only a few target domain images (e.g., 10 images) by inheriting reusable domain-sharing knowledge from the source domain model and learning the domain-specific knowledge from the target domain images. However, for one-shot settings, the target domain is represented by only one reference image, and the above methods still suffer from obvious overfiiting problems due to the lack of sufficient target domain representation. A series of methods [23,24,25,26,27] additionally utilizing the rich semantic representation ability of Contrastive Language–Image Pretraining (CLIP) [28] to guide the direction of fine-tuning have improved the performance of the adaptation process in one-shot scenarios.

However, in some common adaptation tasks, existing methods still exhibit overfitting phenomena, which we do not want to see. In addition, when the domain-specific features of the reference image are complex, the model may also encounter underfitting problems. As shown in Figure 1, when the source domain is real faces and the reference image is a cartoon girl, all the images generated by the fine-tuned model show the same expression of laughter, which is a kind of signal of overfitting and can be harmful to the generation diversity. And when the source domain is cats and the reference image is a tiger, the generated image cannot sufficiently capture the complex stripes of the tiger, which can be considered as an underfitting issue and can impair the fidelity of the generated images. We argue that the previous methods lack explicitly modeling domain-sharing and domain-specific features, resulting in the model having difficulty knowing which information needs to be inherited and which information needs to be learned. Due to the lack of prior knowledge about the domain adaptation task, the model cannot achieve a trade-off between diversity and fidelity through a single reference image.

In this work, we firstly introduce the perspective of the frequency domain into the generative domain adaptation task to precisely assist model fine-tuning. We find that the low-frequency part of the image is highly correlated with domain-sharing information, and the high-frequency part of the image is highly correlated with domain-specific information. As shown in Figure 2, the image can be deconstructed into a low-frequency part (e.g., posture, expression) and high-frequency part (e.g., style, texture) by discrete wavelet transform (DWT). Follwing the above observation, we propose our one-shot domain adaptation method (FAGAN), which utilizes additional frequency-auxiliary to guide the model’s fine-tuning. The two core components of FAGAN are the low-frequency fusion module (LFF-Module) and high-frequency guide module (HFG-Module). The LFF-Module regards the low-frequency part of the source model’s feature maps as domain-sharing knowledge and preserves it to be unchanged during the fine-tuning process. This operation will be implemented at the shallow layer of the model, which is considered to control the coarse attributes of generated images. In this way, the domain-sharing knowledge of the source model will be explicitly inherited, and the diversity of the generated images will be improved. The HFG-Module aims to find the local domain-specific information of the reference image and drive the model to pay more attention to learning them. Specifically, we first decompose the reference image using discrete wavelet transform to extract high-frequency components. Then, we calculate the high-frequency content of each patch and select the patches with a higher high-frequency content, which are considered to contain more domain-specific information. Finally, we calculate the loss between the generated image patches and the selected patches using a similarity matrix. By utilizing high-frequency guidance, the model can learn domain-specific information in a fine-grained and targeted manner, thereby enchancing the fidelity of the generated images. The main contributions of this work can be summarized as outlined below:

• We analyze the limitations of previous methods, which lack explicit modeling for the domain adaptation task, and introduce the perspective of the frequency domain into the generative domain adaptation task.
• We propose a novel method called Frequency-Auxiliary GAN (FAGAN), which mainly contains two submodules: the low-frequency fusion module (LFF) and high-frequency guide module (HFG), which are designed to improve the diversity and fidelity of the generated images, respectively.
• Extensive experiments demonstrate that our method outperforms the state-of-the-art methods on several benchmark datasets in terms of qualitative and quantitative comparison.

2. Related Work

2.1. Domain Adaptation of GANs

The aim of the few-shot domain adaptation of GANs is to learn an accurate and diverse distribution of the target domain, which is represented by only a few images. The core idea of this approach is to use a generator pretrained on the source domain and fine-tune it to a target domain by an adaptation process. The domain adaptation methods of GANs are usually categorized into two directions: fine-tuning-based and regularization-based methods.

The fine-tuning-based methods reduce the number of trainable parameters by only fine-tuning a part of the generative model [30,31], adding additional trainable parameters/modules while fixing the main backbone model [19,32], or decomposing the parameters using singular value decomposition (SVD) [19]. These methods aim to reuse the transferable parameters in the source model and inherit the diversity of the source domain. The LFP-Module used in our method can be seen as an implementation of fine-tuning-based methods which keep the low-frequency information frozen during the adapting process.

The regularization-based methods usually introduce additional regularization terms to constrain or guide the parameters/features changes in the adaptation process. It can show obvious correspondences between images in the source domain and target domain. CDC [18] maintained feature similarity among source/target samples via extra consistency loss. RSSA [21] introduced spatial consistency losses to keep structural identity between the source images and corresponding target images. DCL [22] used contrastive losses in the feature space of both the generator and discriminator. DWSC [33] designed both perceptual and contextual loss, respectively, in a local patch level. These methods employ the concept of multi-task learning, introducing additional loss terms to provide more meaningful guidance for the model adapting process. The HFG-Module used in our method is a kind of regularization-based method, and it can guide the model to learn more representative local domain-specific information through the high-frequency information of the reference image.

For a one-shot scenario, the above methods usually exhibit obvious overfitting problems because of the deteriorating adversarial loss. With the success of CLIP [28], recent works [23,24,25,26,27] build the difference between the source and target domains in the CLIP-embedding space to guide the adaptation process. The abundant semantic knowledge of the pretrained CLIP model can compensate for the extremely scarce samples in one-shot scenarios. Among these clip-based methods, MTG [23] used GAN inversion methods [29,34] to calculate the global domain-gap direction between the source and target domains, which is more reasonable than the mean value used in StyleGAN-NADA [24]. MTG also adopted the style mixing trick during the inference time to better capture domain-specific styles. Our method mainly builds upon this baseline method.

2.2. Frequency Information Used in GANs

In the field of computer vision, the spatial information of images is commonly utilized for image processing. As two-dimensional signals, images can also utilize frequency transformation methods commonly employed in the field of signal processing to address image-related issues. Discrete wavelet transformation (DWT) [35], which decomposes images into different frequency components, has made great success in various generative tasks such as image editing [36], image reconstruction [37], image inpainting [38], and style transfer [39]. For image synthesis, FreGAN [40] and WaveGAN [41] improved GAN’s capabilities of understanding and generating high-frequency signals by adjusting the network structure, thereby enchancing the quality of image generation under limited data. Unlike the above methods, we explicity model domain-sharing and domain-specific information in different frequency signals of images, thus alleviating the learning pressure of the one-shot domain adaptation tasks.

3. Our Method

To alleviate the overfitting and underfitting issues of previous methods in one-shot scenarios, we propose a frequency-auxiliary domain adaptation method, FAGAN, which uses different frequency domain components of images as tasks prior and mainly contains two auxiliary modules: a low-frequency fusion module (LFF-Module) and high-frequency guide module (HFG-Module). The overall framework of FAGAN is shown in Figure 3. There are two generator networks in our framework: a pretrained source domain generator $G_{source}$ whose parameters are fixed and a target domain generator $G_{target}$ guided by only a single reference image $I_{reference}$. The goal of the generative domain adaptation task is to enable the fine-tuned model $G_{target}$ to generate diverse and faithful target domain images using the sampled latent vector z.

3.1. Low-Frequency Fusion Module

To alleviate the overfitting issues that occurred in previous methods, we propose a low-frequency fusion module (LFF-Module), which aims to reuse the low-frequency components of $G_{source}$ to inherit the diversity of the source domain. As mentioned in Figure 3b, the low-frequency part of the images is highly correlated with domain-sharing information, which is embedded within the source model $G_{source}$. The LFF-Module integrates the low-frequency components of $G_{source}$ with $G_{target}$ at the feature map level, thereby inheriting the domain-sharing attributes of the source domain. To disentangle the features into multiple frequency components, we adopt a simple yet effective discrete wavelet transformation, i.e., Haar wavelet [35]. Haar wavelet consists of two operations: discrete wavelet transformation (DWT) and inverse discrete wavelet transformation (IDWT). The former converts images into frequency components, and the latter inversely reconstructs frequency components into the spatial domain, which can be used to fuse images [42]. There are four kernels in Haar wavelet, namely $L{L}^T$, $L{H}^T$, $H{L}^T$, and $H{H}^T$.

$$L^T=\frac{1}{\sqrt{2}}\left[11\right],H^T=\frac{1}{\sqrt{2}}[-11],$$

(1)

where L and H denote the low and high-pass filters, respectively. The low-pass filter focuses on low-frequency signals containing the outline and structural information, which is highly correlated to domain-sharing information. The high-pass filter focuses on high-frequency signals that capture fine-grained details like texture and style, which is highly correlated to domain-specific information. After discrete wavelet transformation, the images would be decomposed into a low-frequency component $LL$ and high-frequency components $LH$, $HL$, $HH$ (as shown in Figure 2). As for inverse discrete wavelet transformation, this operation can reconstruct these frequency components back to the original images. Specifically, it can be implemented by transposed convolution on each frequency component, which is analyzed in WaveGAN [41].

In the LFF-Module, as shown in Figure 3b, we use discrete wavelet transformation on feature maps of both $G_{source}$ and $G_{target}$ to obtain two groups of frequency components. And then, we employ discrete inverse wavelet transformation using $LL$ of $G_{source}$ and $LH,HL,HH$ of $G_{target}$ and eventually obtain the fused feature maps $F_{fused}$.

$$F_{fused}=IDWT(L{L}_{source},L{H}_{target},H{L}_{target},H{H}_{target}),$$

(2)

where $L{L}_{source}$ denotes the freezed low-frequency features of $G_{source}$, and $L{H}_{target},H{L}_{target},$ and $H{H}_{target}$ denote the trainable high-frequency features of $G_{target}$. In addition, influenced by the interpretation of models as mentioned in StyleGAN [3], we employ the LFF-Module only on the shallow layers (e.g., 6 layers) of the synthesis network, which mainly control the coarse-grained attributes of the generates images.

3.2. High-Frequency Guide Module

To alleviate the underfitting issue in which the complex domain-specific attributes cannot be learned well, we propose a high-frequency guide module (HFG-Module). Motivated by the methods [25,27] which use patch-based consistency loss to fit the local level features of the target domain, we also employ a consistency loss on the patch level. Based on previous frequency domain observations, we believe that patches with high levels of high-frequency signals possess richer domain-sharing information and should be emphasized in the model adaptation process. So, we use the high-frequency signals of the reference image to guide the model, whose local attributes should be learned carefully.

Motivated by LoFGAN [43], the HFG-Module mainly has two phases: local selection and local matching. As shown in Figure 3c, in the phase of local selection, we crop the reference image into N patches which are correlated with the intermediate tokens in the CLIP image encoder. And then, we employ wavelet transformation on every patch and calculate $LH+HL+HH$ as their high-frequency strength. To obtain the patches earning more domain-specific information, we select the top M patches $P_{reference}$ with the highest high-frequency content from them. After that, the corresponding M intermediate tokens of reference image $F_{reference}$ can be selected. In the phase of local matching, we want to find the patches in generated images $I_{target}$ which are semantically matching the selected patches $P_{reference}$. And these patches should be guided to faithfully fit the reference image, thereby learning domain-specific information more accurately. Similarly, we embed the generated images in intermediate feature maps of the CLIP image encoder and obtain the same N tokens of local representations $F_{target}$. For each token $f_{reference}$ in $F_{reference}$, we calculate the similarity between every two tokens in $f_{target}$ and $f_{reference}$ to build a similarity map M as shown below,

$$M^{(i,j)}=g(f_{target}^{\left(i\right)},f_{reference}^{\left(j\right)}),$$

(3)

where $i\in \{1,...,n\}$, $j\in \{1,...,m\}$ and g is a similarity metric. According to the similarity map $M^{(i,j)}$, we can find the most corresponding local token $f_{target}^i$ for each token $f_{reference}^j$, which has the minimum cosine similarity loss. After finding the corresponding tokens in generated images, we eventually sum up the loss between every two matched tokens.

$$L_{HFG}=\sum _{i=0}^{n}mi{n}_j(1-\frac{f_{target}^i·f_{reference}^j}{\left|f_{target}^i\right|·\left|f_{reference}^j\right|}),$$

(4)

through this loss, the generated images from the target domain can fully capture the domain-specific information of the reference image in local details, thereby alleviating the underfitting problem. For more details, please refer to Algorithm 1.

Algorithm 1 Construction of HFG

Input: The sampled generated images of target model $I_{target}$; the reference image $I_{reference}$ Output: The proposed loss $L_{patch}$, which is used to update the parameters of $G_{target}$ stage1: Local Selection 1: Crop the reference image $I_{reference}$ into N patches 2: Calculate the strength of high-frequency signal $LH+HL+HH$ in every patch using DWT 3: Build a 0,1 mask according to the high-frequency strength of every patch and select the top M patches $P_{reference}$, which have more high-frequency signals. 4: Get the embedding of each selected patch $F_{reference}$ by CLIP image encoder stage2: Local Matching 1: Crop the sampled generated image $I_{target}$ into N patches $P_{target}$ 2: Get the embedding of each patch $F_{target}$ by CLIP image encoder 3: Match every token i in $F_{target}$ to the most related token j in $F_{reference}$ 4: Sum the distance between every matched token as the proposed loss term $L_{patch}$ 5: $L_{patch}={\sum }_{i=1}^{N}mi{n}_j(1-\frac{f_{target}^i\ast f_{reference}^j}{|f_{target}^i|\ast |f_{reference}^j|})$

3.3. Overall Training Loss

Two frequency-auxiliary modules are mainly based on the MTG baseline model [23]. Specifically, it utilized GAN inversion methods [29,34] to find the source domain image $I_{reference}^{\prime }$ corresponding to the target domain reference image $I_{reference}$. And then, it computed the CLIP loss $L_{CLIP}$ between $I_{reference}^{\prime }$ and $I_{reference}$ in clip semantic space, which was accompanied by LPIPS loss $L_{LPIPS}$ and MSE loss $L_{MSE}$ in perceptual and pixel space.

$$L_{overall}={\lambda }_1{L}_{CLIP}+{\lambda }_2{L}_{LPIPS}+{\lambda }_3{L}_{HFG},$$

(5)

during the adaptation process, by utilizing the $L_{overall}$ combined with our LFF module, the fine-tuned model $G_{target}$ can alleviate overfitting and underfitting issues in most one-shot scenarios.

4. Experiments

In this section, we first introduce the experimental settings of our FAGAN, including implementation details, datasets, and evaluation metrics (Section 4.1). The qualitative and quantitative results across various datasets are shown in Section 4.2 and Section 4.3, respectively, and they demonstrate the effectiveness of our FAGAN to alleviate the issues of overfitting and underfitting. Finally, ablation studies are considered to evaluate the hyper-parameters used by our two modules and validate their effectiveness on the overall model (Section 4.4).

4.1. Experimental Settings

4.1.1. Implementation Details

In our experiments, we use the pretrained Stylegan2 [4] as a generator for domain adaptation like other methods. To verify the efficiency of our frequency-auxiliary modules, we conducte experiments using different source domain datasets: FFHQ [3] and AFHQ-Cat [44]. The pretrained models $G_{source}$ of FFHQ and AFHQ-Cat can generate images at resolutions of 1024 × 1024 and 512 × 512, respectively. Following MTG [23], we use the Adam optimizer [45] with coefficients $\beta 1=0.0$ and $\beta 2=0.99$ to train our model. And we set the learning rate to 0.002 and the batch size to 2. For the clip model we use in our model, we utilize the ‘ViT-B/16’ and ‘ViT-B/32’ CLIP image encoders [46]. For the $L_{patch}$ calculated in the HFG-Module, we first resize all the images into 224 × 224 resolution and use a patch size of 32 to divide each image into 7 × 7 patches, which corresponds to the number of intermediate tokens of the ‘ViT-B/32’ image encoder. For the weight of losses in $L_{overall}$, we set ${\lambda }_1=1$ and ${\lambda }_2={\lambda }_3=10$ for all settings. Similar to previous works [23,24,25,27], we freeze ‘toRGB’ layers and the mapping network in the adaptation process. In the training stage, we fine-tune the pretrained generator for 200 ∼ 400 iterations, and the total training time took about 25 min using a single RTX 3090 GPU.

At the inference time, we also apply the style mixing trick of MTG [23] to better reflect the global color attribute of the reference image. Specifically, due to the different resolutions of the generator (1024 × 1024, 512 × 512), their representations in W+ space [34] have different numbers of latent vectors (18, 16). In our experiment, during the inference process on the FFHQ dataset which has 18 latent vectors, we replace the last 11 latent vectors of the generated images with those of the reference image. In the AFHQ-Cat inference process, which has 16 latent vectors, we substitute the last nine vectors of the sampled vectors.

4.1.2. Datasets

When FFHQ serves as the source domain for adaptation tasks, we use the reference image mainly collected from three datasets which are commonly used in other related works: ArtstationArtistic-face-HQ (AAHQ) [47], MetFaces [14], and face paintings by Amedeo Modigliani, Fernand Leger and Raphael [48]. Each of them represents a different style of face image, including cartoons, oil paintings, sketches, etc. And each target domain consists of 10 images; we only selected one to train the model. For the adaptation of the AFHQ-Cat source model, we collect target images from the AFHQ-Wild validation dataset and divide them into Tiger, Fox and Lion datasets, which include 103, 49, and 136 images, respectively. In particular, Amedeo Modigliani, Fernand Leger, Raphael, Tiger, Fox and Lion are used to conduct the quantitative comparsion in Section 4.3.

4.1.3. Evaluation Metrics

Following DiFa [27], we use the Fréchet Inception Distance (FID) [49] and Kernel Inception Distance (KID) [50] to quantitatively evaluate our FAGAN. Both metrics can measure the quality and diversity of the generated images, while KID is more suitable for the few-shot setting when there are only a few images in the validation set. In all our experiments, FID and KID are calculated between 1000 synthesized images and each validation set.

4.2. Qualitative Results

Figure 4 shows the image result generated by the target generator adapted from the source generator pretrained with FFHQ. As shown in the Figure 4, the baseline model MTG [23] shows a certain degree of overfitting to the local part of the reference image. For example, in the 2nd and 3rd rows, the expression of the generated images overfit the big smiles and twisted mouths, which lost the domain-sharing information hidden in the source model. In contrast, our method can better preserve the original expressions of the source images with the help of our low-frequency fusion module, thus enhancing the diversity of the generated images. And from the 5 ∼ 6th rows, we can observe that our model can capture more domain-specific information (e.g., the bright red mouth of the clown and the artistic style of paintings) than other methods, which demonstrates the effectiveness of our high-frequency guide module.

Additionally, we also illustrate the qualitative results adapted from AFHQ-Cat in Figure 5. As this is a domain adaptation task that is inherently more challenging compared to the FFHQ dataset, existing methods often exhibit significant underfitting issues. For example, because of the huge difference between the source domain images and target domain images, both MTG and DynaGAN struggle with fitting the complex domain-specific information, like the circular stripes of tigers, dense manes of lions, and so on. However, thanks to our high-frequency guide module, our model can identify which local patterns need to be adjusted to learn and can effectively handle conflicts between source patterns and target patterns. In other words, our model can address the trade-off between diversity and fidelity in one-shot scenarios by utilizing frequency assistance.

4.3. Quantitative Results

For the evaluation of our proposed model, we conducted quantitative experiments in this section. Specifically, we compare our FAGAN with other competing methods [18,23,24,26,27] under six settings, i.e., FFHQ → {Amedeo Modigliani, Fernand Leger, Raphael} and AFHQ-Cat → {Tiger, Fox, Lion}. For each setting, we randomly sample an image from the validation set as the reference image and then perform adaptation, recording both the Fréchet Inception Distance (FID) [49] and Kernel Inception Distance (KID) [50] for each method. Additionally, to mitigate any random sampling error, we conduct five repetitions and calculate the mean value as the final score on the 1000 generated images. The results are listed in

Table 1. FID (↓) comparisons between different one-shot domain adaptation methods. Each result is averaged over 5 training shots.

Methods	FFHQ			AFHQ-Cat
	Amedeo	Fernand	Raphael	Fox	Lion	Tiger
FSA [18]	173.66 ± 19.43	227.19 ± 20.33	177.43 ± 29.29	-	-	-
NADA [24]	187.42 ± 21.33	258.35 ± 19.17	185.37 ± 30.14	84.28 ± 14.33	60.55 ± 17.44	16.74 ± 2.57
MTG [23]	207.17 ± 33.18	306.03 ± 51.29	184.35 ± 27.41	67.64 ± 17.77	63.28 ± 18.05	21.25 ± 8.32
DynaGAN [26]	229.19 ± 27.32	317.97 ± 47.98	186.03 ± 14.20	93.04 ± 20.01	86.35 ± 17.33	25.14 ± 9.79
DiFa [27]	178.72 ± 18.95	255.17 ± 16.61	170.76 ± 9.15	69.92 ± 15.55	42.01 ± 11.28	16.26 ± 2.88
FAGAN (Ours)	170.68 ± 17.99	281.74 ± 30.43	166.36 ± 14.70	65.39 ± 10.66	35.20 ± 10.98	21.41 ± 4.33

and

Table 2. KID (↓) comparisons between different one-shot domain adaptation methods. Each result is averaged over 5 training shots.

Methods	FFHQ			AFHQ-Cat
	Amedeo	Fernand	Raphael	Fox	Lion	Tiger
FSA [18]	178.04 ± 4.97	190.26 ± 13.88	164.64 ± 60.19	-	-	-
NADA [24]	131.77 ± 23.57	177.47 ± 30.48	148.32 ± 45.89	77.31 ± 35.24	54.37 ± 18.02	14.22 ± 2.44
MTG [23]	138.55 ± 30.12	205.01 ± 28.11	128.51 ± 37.26	69.97 ± 25.99	69.55 ± 21.74	21.50 ± 7.78
DynaGAN [26]	159.31 ± 31.77	219.81 ± 23.89	112.08 ± 27.39	82.73 ± 37.88	77.14 ± 22.19	22.81 ± 8.10
DiFa [27]	120.78 ± 23.91	167.22 ± 33.15	110.13 ± 17.77	51.01 ± 31.24	36.73 ± 20.45	13.29 ± 1.97
FAGAN (Ours)	116.79 ± 21.58	189.67 ± 25.82	103.33 ± 23.15	47.91 ± 24.43	34.62 ± 16.80	20.90 ± 9.31

. As shown in the tables, our FAGAN achieves competitive FID and KID results on most settings against other competing methods, which is consistent with the outstanding performance of our methods observed in qualitative experiments. The experimental results also demonstrate the contribution of our method to both the diversity and fidelity of the generated images.

4.4. Ablation Study

In this section, we conduct several ablation experiments to further analyze the impact of different hyper-parameters of the proposed frequency-auxiliary modules as well as the influence of each module/operation on the overall generated images.

4.4.1. The Number of Layers Employing LFF-Module

As mentioned in Section 3.1, we only employ the LFF-Module on the shallow layers of the synthesis network. To verify the impact of the number of layers using the LFF-Module, we conduct ablation experiments on a cartoon character dataset. As shown in Figure 6, we use (2, 4, 6, 8, 10, 12) layers, respectively. And one can easily observe that the model tends to overfitting when the number of fusion layers is too low (such as 2 or 4 layers), and the underfitting issue will appear when the number of fusion layers is too large (such as 8 to 12 layers). We explain that too few fusion layers can result in the model’s inability to fully inherit the diversity of the source domain, while too many fusion layers lead to the model having a limited learning capacity, thereby preventing it from faithfully fitting the reference image. So, in our experiment, we set the fusion layers to 6 in the FFHQ dataset and 4 in the AFHQ-Cat dataset to achieve balanced generation results.

4.4.2. The Number of Tokens Selected by HFG-Module

To verify the impact of the amount of selected tokens for calculating losses in the HFG-Module, we conduct ablation experiments on the Tiger dataset. Specifically, we use (1, 9, 25, 49) tokens of high-frequency patches to calculate $L_{patch}$ in the HFG-Module. As shown in Figure 7, using too few tokens (such as 1 or 9) can lead to underfitting of the model, resulting in prominent cat-like features and insufficient tiger stripes. Using too many high-frequency tokens (such as 49) can cause the model to struggle to understand the learning priorities, thus increasing the learning difficulty. This can result in disruptions to the symmetry of the eyes and the appearance of monotonous local patterns of the generated images. In our experiment, we set the number of selected tokens to 25 for both the FFHQ and AFHQ-Cat datasets.

4.4.3. Effect of Two Frequency Modules

Eventually, we conduct ablation studies on the effect of our proposed two critical modules of our FAGAN, i.e., the low-frequency fusion module (LFF-Module) and high-frequency guide module (HFG-Module). As shown in Figure 8, the images can obtain global color attributes through the style mixing trick and generate preliminary clown images as the baseline method. However, the expression and teeth of the generated images are all similar, and the features of the clown are not emphasized. By using the LFF-Module, the generated images exhibit stronger consistency with the original images in terms of expressions, indicating that more domain-sharing information is inherited. After incorporating the HFG-Module, which is our whole-method FAGAN, the generated images clearly capture the significant features of the clown, indicating that the domain-specific attributes of the reference image are adequately learned, e.g., bright red mouth and dark eye circles.

5. Conclusions

In this paper, we have proposed frequency-auxiliary GAN (FAGAN) for the one-shot generative domain adaptation task, which introduces the perspective of frequency into the adapting process and helps alleviate the overfitting and underfitting issues of previous methods. Our FAGAN contains two core modules, the low-frequency fusion module (LFF-Module) and the high-frequency guide module (HFG-Module), which are combined with the baseline clip-based method. The LFF-Module focuses on preserving the domain-sharing information of the source domain by fusing the low-frequency features of the source model. The HFG-Module is dedicated to selecting the domain-specific information of the reference image and guiding the model to focus on fitting them by utilizing high-frequency guidance. Extensive experimental results showed that our method leads to better quantitative and qualitative results than the existing methods under a wide range of task settings.