Few-Shot Face Recognition: Leveraging GAN for Effective Data Augmentation

Abstract

Face recognition technology is a prominent research area in the digital age, with significant applications in commerce and security. This technology relies on high-quality training data, which poses a challenge in practical engineering applications owing to the substantial investment required and the stringent privacy protection regulations that aggravate the problem of sample scarcity, resulting in few-shot learning conditions. The collection of face data under specific conditions, such as extreme lighting and poses significant challenges. Furthermore, the imbalance in sample distribution severely impacts the model’s ability to generalize and achieve accurate recognition. This paper addresses this issue by leveraging Generative Adversarial Networks (GANs) for effective data augmentation. We propose using the architectures of SR-StarGAN and FPNSA-AttGAN to generate diverse virtual face images in different feature domains, constructing a large-scale, widely distributed dataset to support face recognition under various attributes and complex conditions, enabling effective few-shot face recognition. We detail the core algorithms and network frameworks of SR-StarGAN and FPNSA-AttGAN, and demonstrate the training process of IdentiFace using the synthetic samples. The results demonstrate a significant enhancement in face recognition accuracy, from 83.59% to 96.64%, providing a viable approach to address data scarcity, achieving enhanced generalization capability under data-constrained few-shot learning scenarios and offering valuable insights for future studies in generative-based face recognition.

1. Introduction

With the rapid advancement of data-driven technologies, face recognition has emerged as a significant driver of social development in the intelligence age. Face recognition networks play an indispensable role in personal privacy protection, social security governance, and user authentication of smart terminals. However, the training of face recognition models is encountering previously unheard-of difficulties due to the ongoing growth of application situations. The capacity of these models to apply their learning to new situations hinges on the availability of a broad and varied dataset. However, there are numerous obstacles that arise during the collection of such data, primarily including the requirements regarding privacy maintenance, security regulation compliance, and efficient resource allocation. Consequently, there is a pressing need to develop a model capable of achieving high levels of performance through training on a limited set of examples, a concept known as few-shot learning [1,2].

Currently, meta-learning [3], metric learning [4], and data augmentation [5] are the most popular techniques for few-shot learning. The most well-known optimization-based meta-learning techniques comprise MAML [4] and its variations, including Reptile [6], Meta-Transfer Learning [7], iMAL [8], and FAML [9]. These techniques only have 1–20 categories in the target task, whereas face recognition tasks typically involve thousands of categories; hence, they must cope with greater generalization challenges and higher application requirements. As a metric model, Partial-FC [10] has a solidified feature space after training, which makes it hard to adjust to the feature distribution of new categories. Moreover, in situations where only a small number of samples are available, these models struggle to adjust their feature representation to accommodate new data effectively. Although data augmentation can expand the dataset by generating varied additional data through various transformations of existing data, this technique cannot resolve the underlying problem of limited data availability. In contrast, Generative Adversarial Networks (GANs [11,12]) can generate poses, expressions, makeup, and other features that do not appear in the dataset through learning, while data augmentation focuses on affine transformations (such as translation, scaling, rotation, and shearing), color transformations (such as brightness and saturation adjustments), and perspective transformations. These transformations are based on a relatively small range of local changes in the original image, and they are insufficient to provide diverse data for the models.

Given the above considerations, this study aims to construct a data-constrained but high-accuracy face recognition network with comparable performance to those trained on large-scale datasets by image processing techniques. This essentially represents few-shot face recognition.

More specifically, cutting-edge methods like GANs are employed in this study to provide high-quality data to overcome the limitations of few-shot training paradigms. In addition to generating a widely dispersed face recognition dataset with enough samples, this method can significantly lessen the reliance on actual data collection, thereby satisfying the needs of data richness and variety for model training while maintaining user privacy. To remove the technological barrier of multi-attribute joint optimization in identity networks, this study aims to investigate more effective attribute migration techniques. There are some inherent drawbacks in the early attempts regarding GANs. For instance, dedicated generators for each attribute are necessary for attribute-specific modeling techniques (e.g., Dual Residual Learning Strategy and GeneGAN [13]). Despite maintaining an acceptable level of attribute control, this discretized processing method may result in an exponential expansion of the model’s complexity as the number of attributes increases, thus inducing significant scalability issues. Attribute-agnostic constraint techniques like VAE [14], IcGAN [15], and Fader Networks [16] try to accomplish multi-attribute control using unified frameworks. Although these methods can reduce computational complexity, they still frequently present less-than-ideal generation quality in real-world applications, which manifests as technological bottlenecks like unstable identity retention, blurry images, and inaccurate attribute control. These two methods demonstrate a trade-off relationship between computational efficiency and generation quality. This fundamental contradiction has driven researchers to explore more efficient multi-attribute joint optimization solutions. It has been corroborated that simply relying on naive network stacking or global constraints cannot simultaneously meet the requirements for attribute editing precision and generation quality, necessitating systematic innovations at both the architectural and training mechanism levels.

Inspired by StarGAN [17], we propose SR-StarGAN based on domain transformation for the first time. This method preserves both the original identity information and target attributes when generating images by introducing cyclic consistency constraints and efficient mapping of target attribute labels. Compared with earlier methods, SR-StarGAN achieves multi-attribute joint optimization through a single model, which significantly improves the training efficiency and practicality and can also effectively process overly smooth or distorted images.

Meanwhile, a latent space-based FPNSA-AttGAN developed from AttGAN [18] is also introduced in this study. It incorporates an attention mechanism to enhance the interplay among various features, thereby boosting the diversity and fidelity of the images it produces. Compared with previous techniques, FPNSA-AttGAN demonstrates a precise manipulation of desired attributes while maintaining the integrity of the original identity data. This contributes to more resilient performance of synthesized images in real-world scenarios and markedly enhances the visual richness and variety of output images.

2. Related Work

2.1. Data Augmentation

Data augmentation techniques utilized in this context can be mainly divided into two categories: geometric transformations and photometric transformations. Both of them are extensively applied across various computer vision assignments.

Geometric transformations manipulate the spatial layout of an image by reassigning pixel values to alternate coordinates. Photometric transformations modify the RGB channel by changing the pixel color values. Hsu et al. [19] and Taylor et al. [20], respectively, demonstrated that geometric and photometric transformations could effectively prevent model overfitting and enhance image classification performance compared with baseline methods. These findings revealed that the integration of data augmentation consistently enhanced the classification capabilities of the network across all scenarios, with the most substantial improvements observed in flipping and cropping. With the advancement in relevant research, more innovative approaches have been proposed in increasing studies to extend the scope and effectiveness of data augmentation [21]. For example, AutoAugment [22] and Fast AutoAugment [23] utilize a reinforcement learning approach to determine the optimal combinations of data augmentation operations; Mixup [24], CutMix [25], and GridMask [26] generate novel image data through the application of techniques, such as noise addition, image blending, and masking, thereby enriching the variety and complexity of datasets. Khaled [27] proposed the Random Local Rotation method to overcome boundary distortion caused by conventional global rotation by randomly selecting circular regions in the image for rotation. Concurrently, data augmentation approaches rooted in generative models, including GANs and VAEs, have been introduced to bolster the volume and enhance the caliber of training data through the synthesis of new imagery.

2.2. Generative Adversarial Networks

A typical GAN framework is composed of two components: a generator and a discriminator. The generator is tasked with producing counterfeit samples that closely mimic authentic ones, while the discriminator is trained to distinguish between genuine and fabricated samples. Within the realm of computer vision, GANs are extensively utilized for a variety of tasks, including image style transfer, image-to-image translation, and image generation, with the synthesis of facial images standing out as a particularly significant application.

The remarkable diversity and technology development potential of GAN-based face attribute synthesis have been demonstrated in many studies. As an important advancement, conditional GANs [28] (cGANs) introduce conditional constraints on top of conventional GANs, which significantly enhances the ability of generating face images with specific attributes, thus extending it to the application of image style migration. This approach offers a viable solution to address the limited variability in facial recognition datasets. Building upon this concept, CycleGAN [29] introduces an unsupervised image transformation framework that utilizes cycle-consistency constraints, thereby removing the dependency on paired training samples. This method significantly enhances the diversity of available training data for facial analysis tasks. Luo et al. [30] proposed EA2F-GAN, which utilized eye information as a condition to dynamically modify the attribute vocabulary, so as to generate a more realistic face image. The goal of an image translation GAN is to facilitate the transformation of images across different domains, which is commonly accomplished by model training on aligned pairs of input and resultant images. Pix2pix [31] is the first conditional GAN-based image-to-image translation model trained with paired images. To address the challenge of acquiring paired datasets, Wang et al. [32] proposed a CP-EB model, which utilized audio signals to pair with images to generate faces capable of speaking. Meanwhile, Xie et al. [33] proposed a BPFRe model, which removed facial blemishes and modified them by a two-phase framework that efficiently improved the training effect under limited paired data. Based on that, StarGAN and AttGAN were introduced nearly simultaneously to address the issue of a single model corresponding to a single feature. Although these two networks trained great generators, there is still room for improvement.

Rombach et al. [34] proposed applying diffusion and denoising models in the latent space. StableSR [35] and DiffBIR [36] expanded the model’s scale to accomplish super-resolution image restoration by employing the principles of stable diffusion. Full Convolutional Networks [37] (FCNs) proposed the use of convolution and upsampling instead of conventional deconvolution after PSPNet [38], HRNet [39], LinkNet [40], respectively, and the structure in the decoder was used to fuse multi-scale images. In this study, an enhanced super-resolution component integrated with the adversarial framework of StarGAN is developed, and it collectively elevates the perceptual quality and structural coherence of synthesized face images and information, thus enriching feature details. The attention mechanism [41,42] in GANs is introduced in SAGAN [43], BigGAN [44], and Attention-GAN [45] to augment the discriminator’s ability to extract features. In this study, FPN [46] and FCN are integrated in the generator of AttGAN to facilitate the generation of end-to-end pixel-level images. Simultaneously, the attention mechanism is embedded within the discriminator, which enables the model to grasp the global structure of images, substantially enhancing the output quality of generators.

As a GAN for multi-domain image-to-image translation, StarGAN consists of a generator (G) and a discriminator (D). The generator transforms the input image x into the output y based on a target domain label c $G(x,c)\to y$; while the discriminator distinguishes real images from generated ones and classifies domain labels $D:x\to \left\{Dsrc\right(x),Dcls(x\left)\right\}$. StarGAN uses the adversarial loss $Ladv$ to make generated images indistinguishable from real ones, and it also employs the domain classification loss $Lcls$ to ensure correct domain labeling. StarGAN also incorporates a reconstruction loss based on cycle consistency to maintain image similarity after domain transformation. To process partially known label information in multi-dataset training, StarGAN uses the mask vector m to focus on known labels and enable joint training. A series of improvements has been made based on StarGAN in previous studies. Zhang et al. developed MU-GAN [47], which featured a symmetric U-Net with additive attention connections and self-attention mechanisms to improve attribute editing while preserving other facial details. Ko et al. constructed SuperstarGAN [48], which significantly enhanced cross-domain image translation capabilities by incorporating an independent classifier and data augmentation techniques. The OMGD-StarGAN [49] developed by Gu integrates PatchGAN discriminator, dynamic training strategy, and modulated ResNet generator are integrated to achieve lightweight multi-domain image editing. However, these improved GANs still have fundamental limitations, such as insufficient cross-domain generalization ability and discontinuous dynamic attribute control.

As a GAN for facial attribute editing, AttGAN utilizes a triple-collaborative mechanism: attribute classification, reconstruction learning, and adversarial learning. Different from conventional methods, it directly applies an attribute classification constraint to ensure that target attributes are modified while preserving non-target details. An auxiliary classifier is employed to ensure correct attribute modification, and reconstruction learning is adopted to maintain identity and illumination. Adversarial learning based on WGAN-GP enhances visual authenticity. The encoder–decoder architecture of AttGAN allows multi-attribute editing with improved efficiency and supports attribute intensity control and style manipulation. Some researchers have made relatively few improvements to AttGAN. For example, Lin’s MAGAN [50] innovatively combined the GRU structure with the AGU. The application of a discriminative attention mechanism contributed to the accurate localization of key facial regions.

3. Proposed Method

3.1. Network Architecture

SR-StarGAN enhances the generator of StarGAN by adding a super-resolution module, which improves the clarity of the generated images. In contrast, FPNSA-AttGAN incorporates a feature pyramid and self-attention structure into AttGAN. It also replaces deconvolution with a combination of the nearest neighbor interpolation, upsampling, and convolution to reduce artifacts in the generated images. Figure 1 shows the overall network architecture.

3.2. SR-StarGAN Face Generation Network

The SR-StarGAN face generation network consists of StarGAN and super-resolution. StarGAN provides an efficient means of data augmentation by using a single generator to learn mappings between multiple feature domains and directionally change face feature attributes. Nonetheless, the images generated by StarGAN are still prone to insufficient resolution and lack of clarity of faces; hence, it is still not an excellent training set for face recognition networks. In this study, a super-resolution module is incorporated into the generator framework of StarGAN, with the aim of improving the sharpness and overall quality of the images produced. This augmented network is termed SR-StarGAN, and its architecture is depicted in Figure 2.

The architecture of SR-StarGAN is characterized by a conditional generator designed to generate various attributes in tandem with a discriminator. This generator processes two inputs simultaneously: an original image and a merged product that integrates conditional labels. The original image is normalized before being passed to the generator, but its spatial structure remains unchanged. The image label, which is a binary vector that delineates the desired domain attributes, undergoes a spatial expansion to align with the image’s dimensions. After that, it is input into the generator. By employing this input method, the generator can effectively utilize the spatial configuration of the original image while incorporating the information on the target domain attributes furnished by the conditional label. This enables the generator to perform deep feature extraction and transformation on the input data. The conditional generator is constructed using fundamental components, including convolution layers, residual blocks, upsampling units, and super-resolution mechanisms. The convolutional layers methodically extract low-level features, such as textures and edges, before advancing to discern high-level, semantically relevant features, thereby extracting all the important aspects of the input image. Incorporating residual blocks along with instance normalization layers enhances the stability of the model during training, expedites the convergence process, and bolsters the model’s capacity to accommodate features across diverse scales and distributions. This process guarantees that the generated image adheres closely to the target specifications. Subsequently, the data emerging from the upsampling layer are channeled into the super-resolution module, which refines the image to yield a high-definition facial representation. The super-resolution module consists of a Surface Encoding Denoising Module (SEDM), Feature Space Affine Module (FSAM), Latent Prompt Parameter Module (LPPM), and Pretrained Denoising U-Net Module (PDUM).

The Surface Encoding Denoising Module comprises an encoder as well as a Residual Block. The encoder is responsible for downscaling the image resolution, allowing it to isolate and retain the essential characteristics of the image. Then, the encoded image, F0, is combined with random noise, Zt, allowing the model to learn the behavior patterns of features under various noise levels. Subsequently, the output is fed into the ResBlock, which improves the model’s ability to handle noise. Additionally, it ensures the preservation of image features, enabling the model to refine these features while maintaining the essential information.

The Feature Space Affine Module receives the output feature F1 of SEDM and the temporal embedding obtained by the MLP encoding of the time step t as the inputs. The diffusion model leverages a U-Net framework, which is augmented with Transformer capabilities, for extracting features across multiple scales. The Transformer’s multi-head self-attention mechanism takes the input features F1 and transforms them into a series of distinct, reduced-dimensionality representations. Within each subspace, attention coefficients are independently calculated and subsequently aggregated through weighted summation, thus enabling the concurrent processing of diverse input sequence segments across multiple representation spaces. This parallelized attention mechanism significantly enriches feature encoding. Subsequently, a feed-forward network applies nonlinear transformations to the attention outputs, facilitating more discriminative feature mapping while progressively refining the hierarchical semantic representations. Temporal embedding is incorporated into various aspects of SEDM, which provides the model with important information about the current denoising stage and realizes the dynamic adjustment of the model to more accurately capture the feature changes in the image at different denoising stages.

The Latent Prompt Parameter Module first encodes the discrete time step t into a continuous time embedding through the MLP to capture the stage characteristics in the diffusion process. Then, the cross-attention mechanism is adopted to realize the feature fusion between the time embedding and the trainable parameters. Finally, the time-aware prompts are generated through the residual linkage with the MLP, which, according to the characteristics of different times, guides the network to more accurately predict and remove the noise in the image. Based on that, the module can more adeptly conform to the varied prerequisites of image reconstruction endeavors, consequently bolstering the model’s proficiency in generalization over assorted contexts.

The Pretrained Denoising U-Net Module receives the multi-scale feature Fn output from the MFSM and the time-aware cues generated by the LPPM as the inputs. In addition, a pretrained U-Net structure is employed to perform feature extraction and dimensional mapping of Fn through a series of convolutions and downsampling. Simultaneously, it leverages the supplementary semantic guidance offered by temporal cues to refine and enhance the noise prediction for further tuning and optimization.

Meanwhile, to guarantee the preservation of inherent identity attributes in the synthesized image, the generator is also tasked with reverting the image from the target domain to its original domain, thereby establishing a cyclic consistency check. This ensures the fidelity and veracity of the transformation. This constraint mechanism safeguards the retention of identity characteristics throughout the domain translation process.

The cyclic consistency loss can be expressed as the following equation:

$${\mathcal{L}}_{\mathrm{r}\mathrm{e}\mathrm{c}}={\mathbb{E}}_{\mathrm{X},\mathrm{c},{\mathrm{c}}^{\prime }}\left[\right|\left|\mathrm{x}-\mathrm{G}\left(\mathrm{x},{\mathrm{c}}^{\prime },{\mathrm{c}}^{\prime }\right)\right|{|}_1]$$

(1)

The discriminator not only needs to judge the realism of the image, but also needs to classify the attribute properties of the image. Hence, it can offer guiding signals that steer the generator towards generating an image aligned with the desired attributes. This multi-task discriminator ensures that the generated image is realistic and has the expected attributes, thus improving the image conversion quality and multi-attribute control accuracy. The adversarial training architecture is conducive to a dynamic balance between the generative and discriminative models. Through iterative refinement, the generator continuously enhances the visual verisimilitude and the proficiency of attribute modification within its artificial creations. At the same time, the discriminator develops increasingly sophisticated discrimination boundaries to differentiate authentic samples from generated ones, thereby driving mutual improvement through competitive interaction. Ultimately, it realizes efficient image conversion and enhancement between multiple attribute domains, which makes image editing more flexible and precise. To enable the discriminator to effectively differentiate between authentic and fabricated images, the adversarial loss for both the generator and the discriminator can be expressed as the following equation:

$${\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}}={\mathbb{E}}_{\mathrm{x}}\left[\mathrm{l}\mathrm{o}\mathrm{g}{\mathrm{D}}_{\mathrm{s}\mathrm{r}\mathrm{c}}\left(\mathrm{x}\right)\right]+{\mathbb{E}}_{\mathrm{x},\mathrm{c}}\left[\log \left(1-{\mathrm{D}}_{\mathrm{s}\mathrm{r}\mathrm{c}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{c}\right)\right)\right)\right]$$

(2)

The loss function used for the domain classification of real images can be expressed as the following equation:

$${\mathcal{L}}_{cls}^r={\mathbb{E}}_{x,c^{\prime }}\left[-log{D}_{cls}\left(c^{\prime }|x\right)\right]$$

(3)

The loss function for the domain classification of generated images can be expressed as the following equation:

$${\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}^{\mathrm{f}}={\mathbb{E}}_{\mathrm{x},\mathrm{c}}\left[-\log {\mathrm{D}}_{\mathrm{c}\mathrm{l}\mathrm{s}}\left(\mathrm{c}|\mathrm{G}\left(\mathrm{x},\mathrm{c}\right)\right)\right]$$

(4)

The aggregated overall loss can be expressed as the following equation:

$$\begin{array}{l}{\mathcal{L}}_{\mathrm{D}}=-{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}}+{\mathsf{\lambda }}_{\mathrm{c}\mathrm{l}\mathrm{s}}{\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}^{\mathrm{r}},\\ {\mathcal{L}}_{\mathrm{G}}={\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}}+{\mathsf{\lambda }}_{\mathrm{c}\mathrm{l}\mathrm{s}}{\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}^{\mathrm{f}}+{\mathsf{\lambda }}_{\mathrm{r}\mathrm{e}\mathrm{c}}{\mathcal{L}}_{\mathrm{r}\mathrm{e}\mathrm{c}}\end{array}$$

(5)

The final images generated by the SR-StarGAN face generation network have achieved a remarkable breakthrough in their quality. This is manifested as not only a high degree of detail, clarity, and realism but also effective enhancement in the expression of facial features. By accurately capturing and reconstructing the subtle differences in the face, the network dramatically enriches the diversity of face images while enhancing visual realism, which in turn expands the coverage of the dataset.

3.3. FPNSA-AttGAN Face Generation Network

FPNSA-AttGAN is constructed based on the basic structure of AttGAN, while the generator of AttGAN adopts a U-Net-like encoding–decoding structure, which transforms the facial features into the potential space for attribute manipulation to generate diversified images. Despite the high flexibility, the details of the images generated by the generator of AttGAN are not realistic enough, and are accompanied by some artifacts. In this study, based on AttGAN, a feature pyramid network (FPN) is added to the encoder and the combination of the nearest neighbor interpolation upsampling and convolution is used in the decoder instead of deconvolution, and the self-attention mechanism is incorporated into the discriminator to form the FPNSA-AttGAN face-generating network, which generates more diversified images with higher quality. Figure 3 shows the overall network architecture of FPNSA-AttGAN.

The original encoder utilizes a multilayer convolutional and normalization structure to map the input image to a latent space. This process involves the downsampling of the image and the progressive extraction of its features until it reaches a predefined maximum dimension, eventually resulting in a compact latent representation. This representation serves as the foundation for attribute manipulation and reconstruction. In this study, an FPN is incorporated into the encoder to effectively merge feature information across different scales through bottom–up paths, top–down paths, and lateral connections. In the bottom–up path, the encoder retains its multilayer convolution and normalization structure, sequentially downsampling the input image through these layers to generate feature maps at various scales. The top–down path, conversely, begins with the highest layer of the encoder’s feature map and progressive upsampling to increase the size of the feature map. The lateral connections combine the feature maps from both paths, preserving the fine details of the lower-level features while incorporating the semantic information of the higher-level features. By introducing FPN, the encoder’s capacity to perceive attributes at multiple scales is significantly improved, thus providing more detailed information for subsequent attribute reconstruction.

In the conventional decoder architecture, a series of deconvolutional layers with attribute vector concatenation are employed to perform feature upsampling, ultimately producing attribute-modified images via tanh activation. However, this approach may be restrained by parameterized deconvolution operations that may induce inconsistent weight updates across spatial positions in the feature maps, resulting in localized texture discontinuity and perceptually unnatural artifacts. To address these limitations, the method proposed in this study implements a hybrid upsampling strategy, which integrates nearest-neighbor interpolation with convolutional refinement. This alternative approach enables more efficient image enlargement while preserving spatial coherence, substantially reducing visual artifacts and enhancing the overall realism of synthesized outputs.

At the same time, the decoder performs image reconstruction based on potential vectors and original attribute labels. Based on the conditional generation strategy, the decoder can precisely control the specific attributes of the generated images while preserving the latent spatial information. This framework achieves dual optimization of output diversity and attribute-aware generation, where the model can precisely modulate synthesized images according to target attribute conditions. This capability of the decoder significantly strengthens both the expressive power and attribute fidelity of the generated results, while maintaining rich variation across outputs.

The original discriminator shares its convolution structure with the attribute classifier. After the downsampling is completed, the feature map is warped and flattened, branching through two independent fully connected layers and then completing the output process. In this study, the self-attention mechanism is introduced to enhance its ability to model global features, thus improving the accuracy of image discrimination. The self-attention operation is initiated by projecting the input feature representation into three distinct subspaces through linear transformations, yielding the query, key, and value matrices. Subsequently, the pairwise similarity computation between query and key vectors produces attention scores that can be employed to quantify inter-positional feature dependency. These scores are normalized through softmax activation to generate attention weights, which are then used to perform weighted aggregation of value vectors, ultimately producing a refined feature representation with enhanced contextual relationships. The feature map thus formed contains abundant global information, and the discriminator and attribute classifier can make decisions based on more comprehensive information when branching the output through two independent fully connected layers, thus augmenting the accuracy of image discrimination and attribute classification. The attribute classifier C is designed to ensure that the generated image $x^{\widehat{b}}$ reflects the target attribute $b$, namely, $C\left(x^{\widehat{b}}\right)\to b$, indicating that the classifier guides the output image to match the intended attribute.

The equation can be expressed as follows:

$$\begin{array}{c}\mathrm{m}\mathrm{i}\mathrm{n}\\ {\mathrm{G}}_{\mathrm{e}\mathrm{n}\mathrm{c}},{\mathrm{G}}_{\mathrm{d}\mathrm{e}\mathrm{c}}\end{array}{\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}^{\mathrm{g}}={\mathrm{E}}_{{\mathrm{x}}^{\mathrm{a}}~{\mathrm{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}},\mathrm{b}~{\mathrm{p}}_{\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{r}}}\left[{\mathrm{l}}_{\mathrm{g}}\right({\mathrm{x}}^{\mathrm{a}},\mathrm{b}\left)\right]$$

(6)

where

$$l_g\left(x^a,b\right)=\sum _{i=1}^n-b_{i}log{C}_i\left(x^{\widehat{b}}\right)-\left(1-b_i\right)log\left(1-C_i\left(x^{\widehat{b}}\right)\right)$$

(7)

$p_{data}$ and $p_{attr}$ represent the distributions of the real image and the attributes, respectively; $C_i\left(x^{\widehat{b}}\right)$ represents the predicted value of the ith attribute; $l_g(x^a,b)$ represents the sum of the binary cross-entropy losses of all attributes.

The attribute classifier C is trained using the input image and its original attributes, with the following training objective:

$$\underset{C}{min}{L}_{cls}^c=E_{x^a\sim p_{data}}\left[{\mathcal{l}}_r\left(x^a,a\right)\right]$$

(8)

where

$${\mathcal{l}}_r\left(x^a,a\right)=\sum _{i=1}^n-a_i\mathrm{l}\mathrm{o}\mathrm{g}{C}_i\left(x^a\right)-\left(1-a_i\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-C_i\left(x^a\right)\right)$$

(9)

To enhance the preservation of non-target attributes during reconstruction learning, the decoder architecture is designed to faithfully reconstruct the original input image $x^a$ when processing the latent representation $z$ under the strict constraint of maintaining all source attributes $a$ unchanged. This constraint ensures that attribute-independent features remain intact throughout the encoding–decoding process while allowing targeted modifications only to specified attributes. The learning objective can be expressed as follows:

$$\underset{G_{enc},G_{dec}}{min}{L}_{rec}=E_{x^a\sim p_{data}}\left[\parallel x^a-x^{\widehat{a}}{\parallel }_1\right]$$

(10)

where L1 loss instead of L2 loss is used to attenuate the ambiguity.

Adversarial learning is employed between the generator and discriminator to ensure that the generated image $x^{\widehat{b}}$ appears visually realistic. Building upon WGAN, the adversarial loss functions for the discriminator and generator can be defined as follows:

$$\underset{\parallel D{\parallel }_L\le 1}{min}{L}_{adv}^d=-E_{x^a\sim p_{data}}\left[D\left(x^a\right)\right]+E_{x^a\sim p_{data},b\sim p_{attr}}\left[D\left(x^{\widehat{b}}\right)\right]$$

(11)

$$\underset{G_{enc},G_{dec}}{min}{L}_{adv}^g=-E_{x^a\sim p_{data},b\sim p_{attr}}\left[D\left(x^{\widehat{b}}\right)\right]$$

(12)

where $D$ represents the discriminator that satisfies the $1-\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ condition.

By integrating attribute classification constraints, reconstruction loss, and adversarial loss, a unified AttGAN framework is constructed. This framework effectively edits the specified attributes while maintaining the details of non-target attributes. The overall goal of the encoder and decoder can be expressed as follows:

$$\underset{G_{enc},G_{dec}}{min}{L}_{enc,dec}={\lambda }_1{L}_{rec}+{\lambda }_2{L}_{cls}^g+L_{adv}^g$$

(13)

The overall objectives of the discriminator and attribute classifiers can be expressed as follows:

$$\underset{D,C}{min}{L}_{dis,cls}={\lambda }_3{L}_{cls}^c+L_{adv}^d$$

(14)

where the discriminator and attribute classifiers share most of the layers; ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ represent hyperparameters used to balance the loss.

3.4. IdentiFace Face Recognition Network

To validate whether the face images produced by GANs can serve as a dataset for face recognition networks, this study employs the generated images to train the IdentiFace face recognition network, which is based on FaceNet [51], thus resolving the problems related to limited sample sizes and broad distributions.

As a deep learning-based face recognition model, IdentiFace effectively represents face features as high-dimensional vectors by learning the ‘embedding space’ and then realizes face recognition and authentication. IdentiFace is developed based on the basic structure of FaceNet, including Inception ResNet v1 and Inception ResNet v2. The network receives pre-processed face images as the input layer, which is the starting point for feature extraction. Convolutional layers serve as the core of the network, with several layers progressively extracting features (such as edges, textures, and contours) from the image through weight sharing. The Inception module processes the input data simultaneously by applying convolutional kernels of different sizes in parallel, thus capturing multi-scale feature information. Meanwhile, a jump connection mechanism is introduced to the ResNet module, which effectively mitigates the gradient vanishing problem during deep network training and facilitates the flow of information between network layers. The integration of the Inception and ResNet modules allows the network to capture feature representations in a more profound and effective manner. In the convolutional layer, the maximum pooling is utilized in the pooling operation to reduce the size of the feature map while preserving essential feature details. After the convolution and pooling stages, the feature maps are transformed into one-dimensional vectors, which are then processed further by a fully connected layer to generate a unified embedding vector. To guarantee that the distances between various face embedding vectors correctly represent the similarity of distinct identities, the output feature vectors undergo L2 regularization, which ensures that the vectors are mapped to a unit sphere, indicating that their magnitudes are normalized to one.

4. Experiments

4.1. Dataset

The base dataset is sourced from the CelebA dataset, which includes more than 202,599 high-resolution facial images of 10,177 different celebrities. Each image is labeled with multiple binary facial attributes, such as hair color, gender, glasses, and smile. Additionally, CelebA provides location annotations for five key facial feature points, including the eyes, nose, and mouth.

4.2. Face Generation Network

In this study, the experiments are conducted using ordinary data augmentation, classical GAN models, and improved GAN models as the face generation network, respectively, to expand the CelebA dataset, thus generating the training set for the face recognition network. In each of the experiments, the dataset was split into training, validation, and test sets to ensure proper evaluation and generalization of the models. Specifically, 80% of the images (161,759 images) were used for training, 10% (20,260 images) were used for validation, and 10% (20,260 images) were used for testing. The training set was employed for model training, the validation set for hyperparameter tuning, and the test set for evaluating the model’s final performance.

4.2.1. Image Data Augmentation

The original image is flipped horizontally to simulate left–right symmetry; the image is slightly deformed by pixel translation, scaling, rotation, and clipping; color variations are added by brightness fine-tuning and saturation perturbation; variations from different perspectives are simulated by perspective transformations. Eventually, 28 different augmented versions of each image are generated as the first dataset for the face recognition network.

4.2.2. Typical GAN

(1) The generative network of StarGAN is composed of two convolutional layers with a downsampling stride of 2, interspersed with six residual blocks, and capped by two transposed convolutional layers with an upsampling stride of 2. Meanwhile, instance normalization is also integrated in the generator’s design. The discriminator’s configuration is adopted from PatchGANs and is built entirely out of convolutional layers.

Each model undergoes training utilizing the Adam optimizer, which preserves a learning rate of 0.0001 across the initial 10 epochs and then linearly reduces the rate to zero for the subsequent duration of the training regimen. Before feeding the images into the network, they are resized to a fixed resolution of 128 × 128 pixels, and their pixel values are normalized to the range of [−1, 1] by dividing each pixel by 127.5 and subtracting 1. Furthermore, the generator executes five updates for every single update of the discriminator. The additional detailed parameter configurations are presented in

Table 1. StarGAN parameter settings.

Parameterization	StarGAN
lambda_cls	1
lambda_rec	10
lambda_gp	10
g_lr/d_lr	0.0001
n_critic	5
beta1	0.5
beta2	0.999
num_iters	200,000
Num_iter_decay	100,000
batch_size	16

. The training duration is approximately 10 h on a solitary NVIDIA RTX 2080Ti GPU. The output images are first rescaled back to the range of [0, 1] by adding 1 to the pixel values and dividing by 2, followed by scaling them to the range of [0, 255] for visualization. Moreover, no further post-processing is applied to the generated adversarial images. Operations such as smoothing, sharpening, or any other enhancement techniques are avoided to ensure the authenticity and consistency of the training data.

(2) The generator network of AttGAN is an amalgamation of an encoder and a decoder. Specifically, the encoder comprises five convolutional layers, and the decoder is built with five transposed convolutional layers, which are linked by skip connections akin to the U-Net architecture. The discriminator network is constructed with five convolutional layers, followed by a sequence of dense layers. The classifier within the network utilizes the same convolutional layers as the discriminator, feeding the outputs into the subsequent fully connected layers.

The models are trained using the Adam optimizer with a learning rate of 0.0002 for the first 30 epochs, followed by a linear decay to zero for the remainder of the training period. Three loss coefficients are established to ensure the loss values remain within the same order of magnitude. The data are resized to 143 × 143 pixels and cropped to 128 × 128 pixels before being fed into the network, with pixel values normalized to the range of [−1, 1]. The generator performs five updates for every single update of the discriminator. The generated images are post-processed by rescaling the pixel values back to the range of [0, 255] for visualization. As with StarGAN, no additional post-processing is applied. Further detailed parameter configurations are provided in

Table 2. AttGAN parameter design.

Parameterization	AttGAN
n_epochs	60
epoch_start_decay	30
batch_size	32
learning_rate	0.0002
beta_1	0.5
n_d	5
d_gradient_penalty_weight	10
d_attribute_loss_weight	1
g_attribute_loss_weight	10
g_reconstruction_loss_weight	100

. The entire training process spans approximately 18 h on a single NVIDIA RTX 2080Ti GPU.

(3) StarGAN and AttGAN are selected as the base GANs, and excellent generators are trained to expand the image samples. The experiments show that StarGAN performs well on features such as hair color and makeup, and AttGAN achieves better results for features such as the five senses, silhouettes, and bone structure. Eventually, a total of 27 facial features are selected to form a new dataset with the original images, so that every person has 28 face images with different morphological features.

4.2.3. Improving the GAN

While training SR-StarGAN and FPNSA-AttGAN, to ensure the rigor of the experiments, no modifications are made to the original parameter settings, data preprocessing methods, or training procedures of StarGAN and AttGAN. This principle is implemented to maintain consistency and comparability with the baseline models. The experimental results demonstrate that the parameters chosen for this study are both reasonable and effective, contributing to the improved performance of the proposed models.

(1) The face images generated by the StarGAN-trained generator exhibit some facial features but lack clarity and have blurred details. This study improves network performance by integrating a super-resolution module, consisting of SEDM, MSAM, LPPM, and PDUM, into the StarGAN framework. Specifically, SEDM removes shallow degradation from the input image to maintain quality; MSAM adaptively extracts multi-scale features to capture key information at different resolutions; LPPM generates semantic guidance at different time steps to enhance the recovery process; and PDUM combines multi-scale features and cue information for refined image recovery through step-by-step denoising. The results of the loss visualization during training are shown below:

The training performance of StarGAN and SR-StarGAN is evaluated based on four key metrics: generator loss on false images (G/loss_fake), reconstruction loss (G/loss_rec), discriminator loss on false images (D/loss_fake), and discriminator classification loss (D/loss_cls). As shown in Figure 4, the generator loss on false images of both models declines rapidly, then rises, and stabilizes near 0. However, the discriminator’s loss on false images shows the opposite trend, rising quickly, then declining, and stabilizing near 0. However, the loss of StarGAN fluctuates more significantly compared with that of SR-StarGAN. The reconstruction loss and classification loss of both models gradually decrease and approach 0, indicating that the generator produces more realistic images. This makes it harder for the discriminator to distinguish between real and generated images. The reconstruction loss of SR-StarGAN is smaller than that of StarGAN. These results indicate that SR-StarGAN is more stable than StarGAN.

In this study, the generators of StarGAN and SR-StarGAN are used to generate face datasets with different facial features. Compared with StarGAN, SR-StarGAN significantly improves the clarity of the generated images by incorporating a super-resolution module, making facial details sharper and greatly enhancing recognition performance. This highlights the critical role that image clarity plays in the effectiveness of face recognition models. Figure 5 shows some of the face images generated by StarGAN and SR-StarGAN, clearly illustrating the superior clarity and detail in the images generated by SR-StarGAN.

(2) In the complex image detail recovery process of AttGAN, the back-convolution operation leads to the blurring of image edges or the generation of unnatural textures and image artefacts. In addition, the feature map fusion is relatively simple and lacks effective integration of image information at different scales. In addition, the performance is unfavorable with regard to the processing of image details. The FPN added in this study can generate multi-scale feature maps, which allows the model to recover images at multiple scales after layer-by-layer fusion, thus enhancing the recovery of image details. The combination of the nearest neighbor upsampling and convolution instead of inverse convolution provides a simple and efficient way to recover image resolution, avoiding artifacts and unnatural textures. The self-attention mechanism module is placed in the deep feature extraction stage, which effectively mitigates the problem of missing contextual information due to the sensory field limitation of the traditional convolution operation. Based on that, the model can adaptively focus on the important regions in the image, thus enhancing feature representation.

The loss visualization results during training are shown below:

The performance of AttGAN and FPNSA-AttGAN is evaluated based on four key metrics: the gradient penalty of the discriminator (D/gp), adversarial loss (D/loss_gan), reconstruction loss of the generator (G/xa_loss_rec), and adversarial loss (G/xb_loss_gan). As shown in Figure 6, as iterations increase, the gradient penalty of both models decreases and stabilizes at smaller values, with FPNSA-AttGAN achieving a smaller stable value than AttGAN. The adversarial loss of the discriminator quickly drops and then stabilizes, with FPNSA-AttGAN showing a smaller and stable value. The generator’s reconstruction loss decreases, with FPNSA-AttGAN’s value being closer to 0. The generator’s adversarial loss increases, then decreases, and finally stabilizes around 0, with FPNSA-AttGAN’s value also closer to 0. These results indicate that FPNSA-AttGAN is more stable than AttGAN.

In this study, the generators of AttGAN and FPNSA-AttGAN are used to generate face datasets. Compared with AttGAN, FPNSA-AttGAN significantly reduces artifacts in the generated images by incorporating an FPN and a self-attention mechanism, resulting in more natural facial details. Specifically, AttGAN often generates images with noticeable artifacts or unnatural textures, which can affect the realism of certain facial features. In contrast, the improved generator structure of FPNSA-AttGAN can effectively mitigate these issues, resulting in more natural-looking images with abundant details. Figure 7 shows some of the face images generated by AttGAN and FPNSA-AttGAN, clearly illustrating the significant reduction of artifacts and better detail representation in the images generated by FPNSA-AttGAN.

After the two face generation networks are improved, the generated face images achieve significant improvement in terms of clarity and detail performance. Specifically, the facial contours are clearer and more accurate; the details of the five senses, such as the texture of the eyes and the shape of the lips, become more distinct; and the skin texture is more realistic, which is more in line with the requirements of the face recognition system for input images.

4.3. IdentiFace Face Recognition Network

In this study, training is conducted based on five datasets, and the models are evaluated using the Labelled Faces in the Wild (LFWs) dataset, a standard dataset used for face recognition. The evaluation involves the calculation of the embedding vectors for images, comparison of pairs to identify whether they belong to the same person, and the application of metrics such as accuracy, validation at a specific false acceptance rate (FAR), area under the ROC curve (AUC), and equal error rate (EER). The accuracy can be used to reflect the overall performance of a model; the validation can be employed to measure its ability at a specific FAR; the AUC can be utilized to assess classification performance, and the EER can be leveraged to capture the model’s performance when the FAR and false rejection rate (FRR) are equal.

The comparative analysis of results shows that the IdentiFace face recognition network trained on the dataset generated by the face generation network consisting of SR-StarGAN and FPNSA-AttGAN performs well in all the evaluation metrics on the LFW test set. As shown in

Table 3. Comparison of assessment indicators.

Methodologies	Accuracy (%)	Validation (%)	AUC (%)	EER (%)
Traditional data augmentation	83.59	21.34	90.5	17.6
StarGAN	86.72	43.28	92.4	15.2
AttGAN	87.33	47.75	93.9	13.8
StarGAN+AttGAN	90.61	53.87	96.9	9.5
SR-StarGAN+FPNSA-AttGAN	96.64	82.48	99.3	3.5

, the accuracy of the model is 96.64%, indicating its efficiency in the image pair matching task. The AUC of the model is 0.035, proving its excellent performance in various matching tasks. The validation rate and equal error rate are 82.48% and 99.3%, respectively, further demonstrating the robustness and balance of the model in face recognition tasks.

5. Summary

This paper proposes a data augmentation method to address the challenges of data scarcity, uneven distribution, and the difficulty of acquiring real-world samples for face recognition tasks. By leveraging SR-StarGAN and FPNSA-AttGAN to generate face samples with different attributes from the CelebA dataset and training the IdentiFace face recognition model using these generated virtual samples, the experimental results demonstrate that the model trained on the augmented dataset is significantly superior to conventional recognition algorithms in terms of accuracy and adaptability.

However, the proposed method still has some notable limitations. For example, the insufficient diversity of the generated samples remains a significant challenge, as the model can only modify one feature at a time. This limitation prevents the model from fully capturing the diversity of facial features in real-world scenarios. Despite this, 27 well-performing attributes are selected from 40 available attributes in the CelebA dataset. These attributes maintain high quality and stability during generation and achieve favorable performance in face recognition tasks. However, the selection of these attributes does not resolve the model’s issues under different lighting conditions. Under varying lighting conditions, the generated images may still suffer from artifacts, loss of detail, or unclear facial features, which pose a potential hazard to the model’s applicability in complex environments.

Additionally, the computational resources required to train such complex generative models cannot be overlooked. The trade-off between high accuracy and the computational resources needed for training and deployment may limit the practical feasibility of this method in certain applications. For example, both SR-StarGAN and FPNSA-AttGAN require substantial computational resources when generating high-resolution images, which may represent a bottleneck in resource-constrained environments. Therefore, future research should focus on optimizing the generative network architecture to improve generation efficiency, thus enabling the simultaneous editing of multiple features to enhance sample diversity.