Face-Inception-Net for Recognition

Abstract

Face recognition in general scenarios has been saturated in recent years, but there is still room to enhance model performance in extreme scenarios and fairness situations. Inspired by the successful application of Transformer and ConvNet in computer vision, we propose a FIN-Block, which gives a more flexible composition paradigm for building a novel pure convolution model and provides a foundation for constructing a new framework for general face recognition in both extreme scenarios and fairness situations. FIN-Block-A uses a combination of stacked large-size convolution kernels and parallel convolution branches to ensure a large spatial receptive field while improving the module’s deep feature embedding and extraction capabilities. FIN-Block-B takes advantage of stacked orthogonal convolution kernels and parallel branches to balance model size and performance. By applying FIN-Block with an adapted convolution kernel size in different stages, we built a reasonable and novel framework Face-Inception-Net, and the performance of the model is highly competitive with ConvNeXt and InceptionNeXt. The models were trained on CASIA-WebFace and MS-wo-RFW databases and evaluated on 14 mainstream benchmarks, including LFW, extreme scene, and fairness test sets. The proposed Face-Inception-Net achieved the highest average [email protected] of 95.9% in all used benchmarks, fully demonstrating effectiveness and generality in various scenarios.

1. Introduction

In recent years, deep learning has made significant progress in the field of face recognition [1,2,3,4,5,6], with high-precision face recognition models trained on public face datasets now widely used in industrial fields, such as security, surveillance, and mobile applications. The tremendous success of face recognition is mainly attributed to three aspects: (1) large-scale training datasets, including CASIA Webface, VGGFace2, and MS-wo-RFW [5,6,7], have greatly improved the training accuracy of deep models [8]; (2) the network structure, such as GoogleNet [9] and ResNet [10] models, which form the backbone of face recognition models, help improve the performance of deep face recognition; (3) various loss functions are used as control signals during model training, introduced based on Euclidean distance [3] or angular/cosine margin [1,11] to enhance the distinctiveness of face features. These loss functions effectively guide the network to learn more discriminative features by emphasizing inter-class differences and intra-class consistency. However, the performance of DNNs in general scenarios, extreme conditions, and fairness still needs to be improved, making the design of new universal face recognition models a challenge. In IMAN [7], the impact of race-biased face datasets on model performance is demonstrated. In SFace [8], experiments show that the recognition rate of face recognition models in extreme conditions needs to be improved. Therefore, integrating existing achievements in face recognition with new technologies to design innovative universal face recognition models has significant scientific importance and potential application value.

Face recognition is a highly versatile task, facing challenges such as varying poses, lighting, obstructions, facial expressions, and age-related changes [12]. While current research often concentrates on specific aspects of face recognition, there is a pressing need for a universally applicable solution. Recent studies have made notable advancements. Zhang et al. [13] improved the Inception module, significantly enhancing performance on datasets featuring masked faces. LIAAD [14] developed a novel, lightweight attention-based method that, through knowledge distillation, improves accuracy and robustness against age variations in face recognition. Zhang et al. delved into face recognition and its applications, including expression recognition [15], generation [15], and face reconstruction [16,17]. The evolution of hardware and new technologies have spurred the development of more advanced model architectures. Taking inspiration from the successful incorporation of GELU, AdamW [18,19], and Dropout layers in Transformers, ConvNet led to ConvNext [20], a new convolutional model architecture that leverages the ResNet framework. ConvNext has surpassed Transformer models in performance, renewing interest in convolutional models in the computer vision field. InceptionNeXt [21] was introduced; it employs parallel branches, avoiding large convolutional kernels, which leads to superior performance and faster inference, offering new avenues for refining face recognition models. Despite these advancements, the quest for a universally applicable face recognition model has seen limited progress in recent years. The unique demands of face recognition tasks pose significant challenges in making precise adjustments to model structures to enhance versatility across various scenarios.

On the LFW dataset [22], face recognition technologies based on ResNet and Trans-former [23] have reached a performance saturation point. To thoroughly evaluate the effectiveness of our proposed improvements, we introduced verification datasets from multiple scenarios [7,22,24,25,26,27,28,29,30,31] and measured the models’ universality by calculating their average precision across these datasets. Much of the current research in face recognition focuses on Inception and its ResNet variants, demonstrating the possibility of achieving exceptional performance at lower computational costs by optimizing local network topology. Additionally, multilayer perceptrons (MLPs) [32] have been proven to achieve results comparable to complex attention mechanisms, providing theoretical support for enhancing the performance of pure convolutional network structures. Inspired by the parallel branches of InceptionNeXt [21] and the stacked convolutional kernels of Inception, we introduce two types of modules, FIN-Block-A/B. FIN-Block-A combines the design of stacked large convolutional kernels with parallel branches, aimed at expanding the model’s spatial receptive field and enhancing the efficiency of deep feature embedding and extraction. FIN-Block-B, on the other hand, optimizes the balance between model size and performance through stacking orthogonal convolutional kernels and parallel branches.

Based on these innovations, we constructed a novel Face-Inception-Net model, following the InceptionNeXt [21] stage structure design and adjusting the size of convolutional kernels at different stages to form a framework that is both rational and innovative. Compared to existing ConvNet [20] and InceptionNeXt [21] models, our model demonstrates strong competitive performance. The novelty of this study lies in the meticulous stage-wise improvements to the model structure: using large pure convolutional kernels for feature extraction in Stage 1; employing stacked large convolutional kernels for deeper feature mining in Stage 2; and using orthogonal square kernels of varying sizes in Stages 3/4 to optimize the balance between model performance and computational efficiency. We used the average [email protected] across different scenarios as a metric for model universality. Our experimental results not only proved that increasing model width is more effective than depth in large-scale face recognition tasks but also demonstrated the wide applicability of our method through evaluations on the extreme scene and fairness test sets [7].

The main contributions of our work are summarized as follows.

• We present an innovative FIN block that utilizes a hybrid of state-of-the-art convolution and Inception technologies. This architecture is designed to offer an expansive spatial receptive field alongside advanced capabilities for the extraction of deep feature embeddings.
• We use convolutional branches that combine different stages and dimensions to build the novel Face-Inception-Net. Evaluated on the extreme scene and fairness test set, experiments show that our model outperforms the compared models.
• By analyzing the experimental data, we found that our model can better balance the gap between the optimal solution and the average solution of the model. This means that our model is more powerful and stable.

Our model helps to accelerate the research of face recognition model architecture in complex cases.

2. Related Work

In this section, we commence with an overview of deep learning-based algorithms specifically tailored for deep face recognition. Subsequently, we delve into methodologies for integrating transformations with convolutions, exploring their synergistic potential. In conclusion, we highlight recent innovations in face recognition algorithms, tailored for a variety of scenarios, thereby presenting a comprehensive update on the state-of-the-art in the field.

2.1. Deep Face Recognition

Deep face recognition systems, underpinned by convolutional neural networks (CNNs) [33], have exhibited notable performance. These algorithms excel in extracting discriminative features from facial images, subsequently encoding them into an n-dimensional vector space. This space is distinguished by minimized intra-class distances and maximized inter-class distances, enabling precise identification and differentiation of individual faces.

During the training process of deep face recognition models, two key elements play a crucial role: (1) network architecture and (2) loss functions. For the network architecture, researchers have employed both traditional and novel network structures to design these models [6]. Well-known classical network structures, such as AlexNet [34], VGGNet [2], GoogleNet [9], ResNet [10], and SENet [35], are frequently utilized as backbone networks for face recognition models. For instance, Taigman et al. [36] pioneered the use of a nine-layer CNN with multiple locally connected layers for face recognition. In addition, novel architectures specifically designed for face recognition have been proposed to improve model performance [6]. For the latter role, researchers have proposed novel loss functions or modified existing ones to improve the discriminative capabilities of convolutional networks for distinguishing individuals. In early deep face recognition algorithms [37,38], cross-entropy loss is used for feature learning. Schroff et al. [39] introduced triplet loss as a means to directly optimize the embedded features, further improving the performance of the model. Recent research [1,3,11,40] has predominantly adopted angle/cosine margin loss, integrating an angle margin penalty to amplify the discriminative capabilities of features. By leveraging true labels as supervisory signals, these loss functions facilitate feature separability, fostering a pronounced differentiation among various categories.

Face recognition has realized remarkable achievements through years of development and technological advancement. Building on these accomplishments, we have undertaken our research. The optimization of the model and the application of loss functions are complex tasks, each requiring meticulous attention to detail and a deep understanding of underlying principles.

2.2. The Fusion of Transformer and Convolution

Transformer [23] is a widely used architecture in natural language processing (NLP) based on the self-attention architecture. The Transformer model has achieved top results in computer classification and segmentation tasks, but the model execution efficiency is far behind convolution. How to effectively combine Transformer and convolution has become a hot issue.

The application of Transformer [23] to various computer vision tasks has become a prominent trend. So far, research on models has branched out in several directions. (1) The fusion of two models has led to the development of hybrid architectures, such as Conformer (by the National University of Science and Technology) [41], Next-ViT [42], and CoTNet [43]. (2) The combination of the CNN module and the Transformer layer can be used to construct novel models, such as the module-parallel design scheme proposed by Mobile-Former [39] and CMT [38]. (3) Researchers have explored specialized approaches, including using the convolutional model as a guide to training tiny Transformers [44] and using the Transformer to improve the ConvNet convolutional training technique [20]. The Transformer model has gained prominence in computer vision, but CNNs have also made significant progress. More specifically, pure convolutional networks, such as ConvNeXt [20] and InceptionNeXt [21], exploit the inherent strengths of convolution in computer vision. By integrating “network modernization” techniques [41] with training techniques from Transformer [45,46] and other technologies, these models have achieved superior performance on Imagenet, surpassing that of the Transformer model.

The performance of convolutional network models in unconstrained face recognition has reached a plateau of accuracy. Nonetheless, with the relentless advancement of hardware and the continuous emergence of novel technologies, there is an ongoing trend toward designing new model structures that boast enhanced performance.

2.3. Face Recognition in Special Scenes

Conventional face recognition technology has matured to the point of fulfilling commercial deployment criteria, indicating a saturation in its development for standard applications. However, when it comes to extreme scenarios and fairness assessments, the model’s performance reveals room for improvement.

Face Transformer [47] proves that Transformer can be used for face recognition. Considering that the original Transformer may ignore the inter-patch information, Face Transformer modifies the patch generation process and makes the tokens with sliding patches that overlap each other. Inspired by the successful application of self-attention in computer vision, authors of [48] proposed a convolutional visual self-attention network (CVSAN), which uses self-attention to improve convolutional operators. A study conducted by [7] revealed the existence of racial bias in face recognition algorithms, resulting in significant performance degradation in real-world face recognition systems. To address racial bias, a novel IMAN method was proposed to bridge the domain gap, and its potential and effectiveness were validated through experiments. Authors of [14] proposed a novel lightweight attention angle distillation (LIAAD) method, which introduces a learning paradigm that can effectively distill the age-invariant attention and angle knowledge of these teachers into a lightweight student network, enabling it to be more robust with higher accuracy and robustness to the age factor.

The assessment of a model’s performance is fundamentally linked to the test set employed. In light of the saturation encountered in routine face recognition tasks, we incorporated test sets designed for fairness [7] and extreme scenarios [8]. This approach allowed for a thorough evaluation of our model’s capabilities, with the objective of showcasing its adaptability and versatility across a broad range of conditions.

3. Method

3.1. FIN-Block

Formulation of ConvNeXt Block. ConvNeXt [20] is a contemporary CNN model known for its streamlined architecture. In each ConvNeXt block, the input X undergoes an initial processing step with a depth convolution to propagate information across spatial dimensions. As shown in Figure 1, the token mixer is effectively established by using a depthwise convolution in ConvNeXt,

$$X^{\prime }=TokenMixer\left(X\right)={DWConv}_{k\times k}^{C\to C}\left(X\right)$$

(1)

where ${DWConv}_{k\times k}^{C\to C}$ denotes depthwise convolution with a kernel size of k × k. In ConvNeXt, k is set as 7 by default.

Formulation of Inception depthwise convolution. As shown in Figure 1, conventional depthwise convolution with a large kernel size significantly hampers model speed. Therefore, InceptionNeXt [21] leaves partial channels unchanged and denotes them as a branch of identity mapping. For the processing channels, InceptionNeXt decomposes the depthwise operations with Inception style [9,49,51], using 3 × 3, 1 × ${\mathrm{k}}_w$, and ${\mathrm{k}}_h$ × 1 kernels as branches. Specifically, for input X, we split it into four groups along the channel dimension,

$$\begin{array}{ll}{X}_{\mathrm{hw}},X_{\mathrm{w}},X_{\mathrm{h}},X_{\mathrm{id}}& =Split\left(X\right)\\ & =X_{:,:g},X_{:g:2g},X_{:2g:3g},X_{:3g:},\end{array}$$

(2)

where g is the channel numbers of convolution branches. Set ratio $r_g$ to determine the branch channel numbers by g = $r_{g}C$. The splitting inputs are fed into different parallel branches,

$$\begin{array}{ll}{X}_{\mathrm{hw}}^{\prime }\hfill & ={\mathrm{DWConv}}_{k_{\mathrm{s}}\times k_{\mathrm{s}}}^{g\to g}g\left(X_{\mathrm{hw}}\right),\\ X_{\mathrm{w}}^{\prime }\hfill & ={\mathrm{DWConv}}_{1\times k_b}^{g\to g}g\left(X_{\mathrm{w}}\right),\\ X_{\mathrm{h}}^{\prime }\hfill & ={\mathrm{DWConv}}_{k_b\times 1}^{g\to g}g\left(X_{\mathrm{h}}\right),\\ X_{\mathrm{id}}^{\prime }\hfill & =X_{\mathrm{id}},\end{array}$$

(3)

where $k_s$ denotes the small square kernel size set as 3 by default and $k_b$ represents the band kernel size set as 11 by default. Finally, the outputs from each branch are concatenated:

$$X^{\prime }=Concat(X_{\mathrm{h}{\mathrm{w}}^{\prime }},X_{{\mathrm{w}}^{\prime }},X_{{\mathrm{h}}^{\prime }},X_{\mathrm{i}{\mathrm{d}}^{\prime }}).$$

(4)

Formulation of FIN-Block. As shown in Figure 1, Inception [9] shows excellent performance with relatively low computational cost. MLPs [32,49,50] have also been shown to achieve comparable performance to attention mechanisms. Inception has shown that 1 × 1 convolutions are used to compute reductions before the expensive 3 × 3 and 5 × 5 convolutions. Besides being used as reductions, they include the use of rectified linear activation, which makes them dual-purpose. The ubiquitous use of dimension reduction allows for shielding a large number of input filters from the last stage to the next layer, reducing their dimension before convolving over them with a large patch size. Another practical and useful aspect of this design is that it aligns with the intuition that visual information should be processed at various scales and then aggregated so that the next stage can abstract features from different scales simultaneously. The design of the FIN-Block is inspired by Inception, which uses deep convolutional stacking and orthogonal bands kernel stacking to improve the performance of the model. The modified formula is shown below.

$$\begin{array}{ll}{X}^{\prime }\hfill & ={\mathrm{Conv}}_{1\times 1}^{C\to C/3}\left(X\right),\\ X_v^{\prime }\hfill & ={\mathrm{DWConv}}_{k_{\mathrm{s}}\times k_{\mathrm{s}}}^{g\to g}g\left(X^{\prime }\right),\\ X_{w1}^{\prime }\hfill & ={\mathrm{DWConv}}_{k_b\times k_b}^{g\to g}g\left(X^{\prime }\right),\\ X_{w2}^{\prime }\hfill & ={\mathrm{DWConv}}_{k_b\times k_b}^{g\to g}g\left(X_{w1}^{\prime }\right),\\ {X_o}^{\prime }\hfill & =Concat\left(X^{\prime },X_v^{\prime },X_{w2}^{\prime }\right).\end{array}$$

(5)

$$\begin{array}{ll}{X}^{\prime }\hfill & ={\mathrm{Conv}}_{1\times 1}^{C\to C/2}\left(X\right),\\ X_{w1}^{\prime }\hfill & ={\mathrm{DWConv}}_{1\times k_b}^{g\to g}g\left(X^{\prime }\right),\\ X_{w2}^{\prime }\hfill & ={DWConv}_{k_b\times 1}^{g\to g}g\left(X_{w1}^{\prime }\right),\\ {X^{\prime }}_o\hfill & =Concat(X^{\prime },X_{w2}^{\prime }),\end{array}$$

(6)

where $k_s$ and $k_b$ denote the small square kernel size set as 11 by default. Finally, the outputs from each are concatenated. The equation represents FIN-Block-A on the up and FIN-Block-B on the down. FIN-Block-A comprises a 1 × 1 small square kernel, large square kernel, and stack of two large square kernel. FIN-Block-B comprises a 1 × 1 small square kernel and stack of two orthogonal bands kernel.

3.2. Face-Inception-Net

Based on FIN-Block-A and FIN-Block-B, we developed a set of models called Face-Inception-Net. Considering ConvNeXt [20] and InceptionNeXt as our primary baselines for comparison, we mainly followed their approach to construct models of different sizes. Similar to ResNet [10] and ConvNeXt, we adopted a 4-stage framework. For tiny models, the stage numbers are (3, 3, 9, 3), and for the base model, the stage numbers are (3, 3, 27, 3). Additionally, we experimented with stage numbers (5, 5, 10, 5) and dimensions (96, 192, 384, 786) for TV2 and stage numbers (5, 5, 10, 5) and dimensions (192, 384, 576, 786) for SV2. Our model uses BatchNorm instead of LayerNorm in the stem to prioritize speed. Detailed model configurations are shown in

Table 1. Face-Inception-Net model configuration. It has a similar model configuration to Swin [45] and ConvNeXt [20].

Stage	# of Tokens	Layer Specification		Face-Inception-Net
				T	S	B	TV2	SV2
1 Stem	$\frac{H}{4}\times \frac{H}{4}$	Down- sampling	Kernel size	4 × 4, stride 4
			Embed. dim.	96		128	96	192
		ConvNeXt block	DWConv kernel size	7 × 7
			MLP ratio	4
			# of blocks	3			5
2 Stage	$\frac{H}{8}\times \frac{H}{8}$	Down- sampling	Kernel size	2 × 2, stride 2
			Embed. dim.	192		256	192	384
		FIN-Block-A	DWConv kernel size	$(11\times 11),\left(\begin{array}{c}11\times 11\hfill \\ 11\times 11\hfill \end{array}\right)$
			Conv. group ratio	1/8
			MLP ratio	4
			# of blocks	3			5
3 Stage	$\frac{H}{16}\times \frac{H}{16}$	Down- sampling	Kernel size	2 × 2, stride 2
			Embed. dim.	384		512	384	576
		FIN-Block-B	DWConv kernel size	$\left(\begin{array}{c}11\times 1\hfill \\ 1\times 11\hfill \end{array}\right)$
			Conv. group ratio	1/8
			MLP ratio	4
			# of blocks	9	27		10
4 Stage	$\frac{H}{32}\times \frac{H}{32}$	Down- sampling	Kernel Size	2 × 2, stride 2
			Embed. dim.	786		1024	786
		FIN-Block-B	DWConv kernel size	$\left(\begin{array}{c}7\times 1\hfill \\ 1\times 7\hfill \end{array}\right)$
			Conv. group ratio	1/8
			MLP ratio	3
			# of blocks	3			5
BatchNorm + Dropout + Flatten + Dense (512) + BatchNorm

and Figure 2. In terms of model structure, we mainly experimented with two popular stage-block combinations and explored the impact of block kernel size, model depth, and width on performance to find the optimal model that meets the requirements of face recognition.

3.3. Loss Function

In our experiments, after evaluating various loss functions [1,11], we selected ArcFace [1] as our definitive choice. ArcFace introduces a unified formula that blends the angular margin principles of both ArcFace and CosFace [11] with traditional softmax loss, providing a versatile framework for facial recognition. This adaptable formula, incorporating angular margin penalties from SphereFace [4], ArcFace [1], and CosFace [11], enhances the discriminative power of embeddings in face recognition models. The integration of these three penalty methods is succinctly captured in the following formula:

$$(\cos (m_1\theta +m_2)-m_3),$$

(7)

where $m_1$ represents the multiplicative angular margin used in SphereFace, $m_2$ denotes the additive angular margin found in ArcFace, and $m_3$ signifies the additive cosine margin from CosFace. The generalized loss expression is as follows:

$$L=-\frac{1}{N}\sum _{i=1}^{N}log\frac{e^{s\left(cos\left(m_1{\theta }_{y_i}+m_2\right)-m_3\right)}}{e^{s\left(cos\left(m_1{\theta }_{y_i}+m_2\right)-m_3\right)}+{\displaystyle \sum _{j=1,j\ne y_i}^n}{e}^{s cos{\theta }_j}},$$

(8)

where N is the number of samples; s denotes the scaling factor to control the logits’ magnitude; $m_1$, $m_2$ and $m_3$ are parameters that adjust the margin effect, allowing the formula to adapt between Softmax $(m_1=1,m_2=0,m_3=0)$, SphereFace $(m_1\ne 1,m_2=0,m_3=0)$, CosFace $(m_1=1,m_2=0,m_3=m)$, and ArcFace $(m_1=1,m_2=m,m_3=0)$; m denotes the margin added to the angle ${\theta }_{y_i}$ between the feature vector of the i-th sample and the weight vector of its corresponding true class $y_i$, which is central to the angular margin concept; and n represents the total number of classes.

4. Experiment

Following the Open Face Recognition pipeline [4], we trained Face-Inception-Net on a face database comprising images (X) and corresponding labels (y) in a supervised manner. We utilized a carefully designed loss function [1,3,11], which enhanced the discriminative capacity of the face image embedding. The LFW dataset served as the benchmark for evaluating face recognition performance. We introduced the dataset, including 2 training datasets and 14 test datasets, and assessed Face-Inception-Net’s effectiveness through various experiments. To ensure a fair comparison with established baselines, we maintained consistent hyperparameters during training. The experimental outcomes, achieved using learning rate decay and without model fine-tuning, are detailed further in Appendix A 

Table A1. Hyperparameters of Face-Inception-Net on CASIA-WebFace and MS-wo-RFW face recognition.

	Face-Inception-Net
	CASIA-WebFace	MS-wo-RFW
Input resolution	112 × 112	112 × 112
Epochs	120	120
Batch size	256	512
Optimizer	Yogi	Yogi
Learning rate	0.002	0.0025
Learning rate decay	Cosine 0.1	Cosine 0.1
Warmup epochs	0.1 × epochs	0.1 × epochs
Weight decay	None	None
Dropout	0.6	0.6
LayerScale initialization	1.00 × 10−6	1.00 × 10−6
EMA decay rate	0.99	0.99

.

4.1. Dataset and Evaluate

For Training. We utilized CASIA-WebFace and MS-wo-RFW [5,6,7] as our training datasets. CASIA-WebFace comprises approximately 500,000 images of 1000 celebrities, featuring a wide variety of poses, lighting conditions, expressions, ages, and races. This diversity makes it an ideal dataset for developing robust face recognition systems. MS-wo-RFW, containing 3.5 million images from 7500 celebrities, includes specific test sets for African, Asian, Caucasian, and Indian demographics, facilitating region-specific evaluations. The CASIA-WebFace dataset is made available by InsightFace, while MS-wo-RFW is provided by IMAN [7].

For Testing. Given privacy concerns in face recognition, benchmarks typically encompass 5000 to 8000 face-matching tasks. We utilized 14 benchmarks for accuracy assessment. These include LFW [22], which tests face recognition algorithms under real-world conditions, AgeDB-30 [24] for age invariance, CFP-FF and CFP-FP [25] for pose variations, and VGG2-FP [26] for frontal to profile comparisons. For extreme scenarios, we used CALFW [27] to assess real-world variations, CPLFW [28] for cross-pose recognition, MLFW [29] for ethnic diversity, SLLFW [30] for low-light conditions, and TALFW [31] for temporal aging variations. Additionally, RFW [7] provided fairness metrics by evaluating racial fairness across diverse racial groups (Africa-test, Asian-test, Caucasian-test, and Indian-test). AgeDB-30, VGG2-FP, and CALFW were also pivotal in evaluating cross-age recognition performance.

For Evaluation. To thoroughly assess model performance, we utilized 14 test sets, categorizing them for detailed analysis. Our evaluation metrics include mean accuracy, ROC curve plots, and performance analysis across common, extreme, and fairness scenarios. For training, we adopted the ConvNeXt method. Unlike traditional approaches that focus solely on the optimal value, disregarding its correlation with fluctuating epoch results (where the optimum across different benchmarks rarely coincides in a single epoch), we averaged the accuracy across benchmarks for a more comprehensive performance and generalization assessment. Furthermore, we compared the optimal value and average [email protected] differences between our model and established models [34], like ConvNeXt and InceptionNeXt. Our model demonstrates exceptional performance in extreme scenarios and on racially balanced face datasets.

4.2. Implementation and Training Techniques

The effectiveness of ConvNeXt and InceptionNeXt shows that the final performance of the network is influenced by the training process as well as the architecture design. Transformers introduce different model designs for vision along with different training methods (such as the AdamW [18,19] optimizer) to the computer vision (CV) domain. Optimization strategies and related hyperparameter settings can help the model find better saddle points, but more hyperparameters also increase the difficulty of model training. Because of the variety of training tasks and loss functions, we experimented with different optimizers and learning rate decays. In our study, we first trained a baseline model using ResNet50 and Inception-ResNet-V1. After many rounds of pre-training, the number of rounds was set to approximately 120 epochs in the loop dataset. The optimizer adopts the Yogi optimizer proposed by Google [18,19,46,52]. By controlling the increase of the effective learning rate, better performance could be obtained under the similar convergence theory guarantee. Then, the EMA wrapper optimizer [37] was used to replace the training parameters with the moving average of the deep network training parameters. The learning rate decay uses learning rate warmup and cosine learning rate decay. The full set of hyperparameters we used can be found in Appendix A. In terms of the learning rate itself, this training method led to a 0.43% improvement in accuracy.

In addition, the improvement in training speed of technical teams, such as Mixed Precision [53], XLA [54], and TF32, is huge, reducing the time cost by more than half. This shows that training skills can affect performance in face recognition. We used a fixed training method with the same hyperparameters throughout the comparative training process, and each reported [email protected] is the average of all benchmarks obtained from training. The experiments were implemented in Tensorflow and performed on a Linux operating system using two Nvidia A100-40G GPUs.

4.3. Result

Training on CASIA-WebFace and MS-wo-RFW. Our initial learning rate was lr = 0.002, and the epoch was 120. There was a 1/10 × epoch linear warmup and then a cosine decay schedule; the learning rate decay multiple was 1/10. Batchsize = 256 for the CASIA-WebFace dataset, and Batchsize = 512 for the MS-wo-RFW dataset. We did not use data augmentation. For regularization, we used label smoothing [9], stochastic depth [55], and weight decay. Like ConvNeXt, we also used LayerScale [56] in the stem, a technique that helps train deep models. We used the exponential moving average (EMA) [57] because we find that it facilitates the overfitting of larger models. Our code is based on the TensorFlow [58,59,60] and TensorFlow Addons [60] libraries.

Figure 3 and Figure 4 below show our first experiments where the values in the MLFW, SLLFW, and TALFW test sets were not used. We compare the improved Face-Inception-Net model with the state-of-the-art ConvNeXt and InceptionNeXt models. As shown in Figure 3 and Figure 4, our proposed model shows strong competitiveness on both the CASIA-WebFace and MS-wo-RFW datasets. With similar model sizes and flops, Face-Inception-Net consistently outperforms ConvNeXt and InceptionNeXt in terms of accuracy. And, comparable experimental results were obtained on two different scale datasets, CASIA-WebFace and MS-wo-RFW.

The training results of the CASIA-WebFace dataset are shown in Figure 3 and Figure 5. The accuracy Acc-mean([email protected]) obtained by the ResNet and Inception-ResNet models are 0.8263, 0.8137, 0.8157, and 0.8167, respectively. This is the result of using the old model training. Then, we reproduced ConvNeXt-Tiny and InceptionNeXt-Tiny, and the obtained accuracy was 0.8807 and 0.8833, respectively. So far, we conclude that using Inception to transform ConvNeXt is effective in the field of face recognition. We designed five models of Face-Inception-Net (FIN-T, FIN-S, FIN-B, FIN-TV2, and FIN-SV2), among which FIN-TV2 obtained the highest Acc-mean([email protected]) value of 0.9106. When we experimented with deeper networks, the performance of the model decreased, such as the small and basic models of ConvNeXt and InceptionNeXt. The FIN-S, FIN-B, and FIN-SV2 models we developed also showed a decrease in model performance. In this regard, we suggest that the size of the CASIA-WebFace dataset is too small and the complexity of the model is too high, resulting in overfitting. Furthermore, the performance of our model on the TALFW (Transferable Adversarial Labeled Faces in the Wild) dataset is worse, which is a bad sign that our model is not well-defended and vulnerable to attacks.

The results of training on the MS-wo-RFW dataset are shown in Figure 4 and Figure 6. The accuracy Acc-mean([email protected]) obtained by the ResNet and Inception-ResNet models are 0.9040, 0.8878, 0.9059, and 0.8933, respectively. Training on the CASIA-WebFace dataset, we found that the use of a deep model resulted in performance degradation. The result of training with ConvNeXt is that the performance degradation of the model is suppressed but not completely eliminated. The results of the InceptionNeXt model improved compared to ConvNeXt, but the results were unstable. We propose that the model achieved an improvement in accuracy when the stage was (3,3,9,3) and (5,5,10,5). And, under the deeper small and base models, the accuracy of our model improved, which solved the problem of model performance degradation. In comparison, under the same parameters of our model, the accuracy of the FIN-T model is 0.9533, which is higher than the 0.9477 and 0.9494 of ConvNeXt-Tiny and InceptionNeXt-Tiny. The highest performance of FIN-SV2 is 0.9599, which is the best result in our experiments. Furthermore, the performance of our model on the TALFW (Transferable Adversarial Labeled Faces in the Wild) dataset leads to the same conclusion as on CASIA-WebFace.

On the fairness and extreme scene test sets. Our findings exceed those achieved by IMAN, based on comparative data sourced from their paper. The analysis shows our model, FIN-SV2, significantly surpasses previous models in accuracy while reducing racial bias. This improvement is particularly notable in test sets for African, Asian, and Indian demographics, where our model achieved accuracies over 0.94. These results, detailed in

Table 2. Face fairness test set performance. The first three sets of data in the table come from the paper. The rest of the data are obtained from our experiments.

Models	Caucasian-Test	Indian-Test	Asian-Test	African-Test
Casia [7]	92.15	88.00	83.98	84.93
MS-wo-RFW [7]	93.92	92.98	90.60	90.98
Iman [7]	94.78	94.15	91.15	91.42
Inception-RV1 [9]	91.92	86.87	83.35	84.32
ResNet50 [10]	92.70	87.40	83.78	84.38
ConvNeXt-T-v1 [20]	97.68	93.95	92.10	92.53
InceptionNeXt-T [21]	97.77	93.95	92.28	92.57
FIN-Tiny	98.23	94.22	93.02	93.22
FIN-Small	98.08	94.75	93.78	93.10
FIN-Base	98.22	95.02	93.72	93.80
FIN-Tiny-V2	98.53	94.85	93.52	93.65
FIN-Small-V2	98.73	95.33	94.33	94.27

, suggest that using large datasets can mitigate racial bias in model performance effectively. Additionally, we assessed the model’s accuracy in various special scenarios, with results shown as ROC curves in Figure 5 and Figure 6. The experiments confirm that our improvements enhance facial recognition performance in complex scenarios. An analysis of the difference between optimal and average values further demonstrates the versatility and robustness of our approach.

Optimal and mean analysis. In our experiments, we used several benchmarks and averaged all [email protected] results for consistent accuracy evaluation. The analysis showed a difference between the optimal and average values for models trained on various datasets, as shown in

Table 3. The gap between the optimal and average performance reflects the model’s generality. The difference between the model’s optimal and average performance is presented in the table for the Casia and RFW datasets, indicating the disparity between the best and average values. This metric reflects the model’s generalizability; a smaller value signifies stronger generalizability.

Model	#Param. (M)	FLOPs (G)	Value Casia(%)	Value RFW (%)
Inception-RV1 [9]	33	1.23	0.58	0.39
Inception-RV2 [9]	62.9	2.4	0.72	0.36
ResNet-50 [10]	49	1.27	1.17	0.35
ResNet-101 [10]	70.6	2.42	0.54	0.31
ConvNEXT-T-v1 [20]	39.5	2.41	0.51	0.32
ConvNeXt-S-v1 [20]	61.2	4.53	0.54	0.25
ConvNeXt-B-v1 [20]	101.4	7.97	0.61	0.19
InceptionNeXt-T [21]	32.9	1.02	0.42	0.16
InceptionNeXt-S [21]	54.3	2.06	0.58	0.15
InceptionNeXt-B [21]	90.6	3.66	0.58	0.14
FIN-Tiny	34.6	1.09	0.32	0.21
FIN-Small	57.3	2.21	0.31	0.20
FIN-Base	98.4	4.34	0.36	0.19
FIN-Tiny-V2	44.4	1.48	0.28	0.19
FIN-Small-V2	67.4	4.15	0.37	0.19

. Using a larger training set reduces this gap. Our FIN-Tiny and FIN-Tiny-V2 models outperformed others on the CASIA-WebFace dataset, whereas the InceptionNeXt series was more effective with the MS-wo-RFW dataset, highlighting our model’s improved performance. Further examination of the model structure revealed that a deeper stacked convolution structure boosts feature extraction, thus enhancing model performance. The gap between the optimal and average values reflects the model’s generalizability, with a smaller gap indicating stronger stability and generalizability across different datasets. However, this balance is not as optimal as with InceptionNeXt, indicating room for improvement.

Ablation studies. 

Table 4. Ablation for Face-Inception-Net on CASIA-WebFace face recognition benchmark. FIN-Tiny was utilized as the baseline for the ablation study. Acc-mean([email protected]) accuracy on the validation set is reported.

Ablation	Variant	#Param. (M)	FLOPs. (G)	Acc-Mean ([email protected])
Baseline	FIN-Tiny	34.64	1.09	90.48
Branch	Without FIN-Block-A	34.56	1.08	90.23
	Without FIN-Block-B-11	34.10	1.07	89.96
	Without FIN-Block-B-7	33.85	1.09	90.28
Stage	stage (3,3,9,3)–(5,5,10,5)	44.41	1.48	91.06
	dim (96,192,384,786)–(192,384,576,786)	67.40	4.15	90.49
Training Skill	Without linear warmup and a cosine decaying schedule afterward.	34.64	1.09	90.05

highlights the three main improvements in our model training: the design of FIN-Block-A and FIN-Block-B, model structure enhancements, and training techniques. We conducted Face-Inception-Net ablation studies using FIN-Tiny as a baseline, focusing on the utilization of different FIN-Blocks at various stages. Specifically, ConvNeXt-proposed blocks are employed in the stem, while in the subsequent three stages, we systematically remove each block to assess our module’s effectiveness. Notably, removing the FIN-Block-B-11 model in stage 3 results in the most significant performance drop, from 90.48 to 89.96. These experiments underscore the importance of these blocks, as large convolution kernels increase the model’s receptive field, enabling deeper feature extraction and achieving an optimal balance between model parameters and performance.

As shown in Table 4, the enhancement of our model training primarily encompasses three aspects. The first is changing the Inception stage configuration from (3,3,9,3) to (5,5,10,5), leading to the proposal of FIN-Tiny-V2. Furthermore, we adjusted the dimensions from (96,192,384,786) to (192,384,576,786), introducing FIN-Small-V2. This experiment validated our improvement direction, showing increased parameters and FLOPs for FIN-Tiny-V2 alongside enhanced model performance. Specifically, in the CASIA-WebFace database, FIN-Tiny-V2 outperformed both FIN-Small and FIN-Base, demonstrating the superiority of the (5,5,10,5) configuration for face recognition. FIN-Small-V2 also delivered remarkable results on the MS-wo-RFW dataset, with fewer parameters yet significant performance gains compared to FIN-Base. This confirms that, in face recognition, widening the model is more efficacious than deepening it, aligning with conclusions from the inception paper. Additionally, the adoption of a learning rate warmup and decay strategy, as opposed to a fixed learning rate, further improved our model accuracy by 0.43%.

5. Conclusions and Future Work

This study introduces Face-Inception-Net, utilizing advanced convolution techniques for improved performance. Unlike other architectures, FIN-Block-A and FIN-Block-B offer versatile composition, enabling swift adjustments in model width and depth for various tasks. FIN-Block-B’s extensive convolutional kernel stack provides a wide spatial receptive field and strong deep feature embedding, achieving an ideal balance between performance and model size. Our experiments showed that expanding the model and adjusting stage sizes significantly enhances face recognition performance. Moreover, results from the extreme scene and fairness test sets significantly demonstrate our method’s versatility and effectiveness.

Our findings indicate that incorporating fine-grained blocks at various stages enhances performance. These results were achieved in a single training session without subsequent fine-tuning, indicating the potential for further optimization and mean analysis improvement relative to InceptionNeXt. Future efforts will focus on simplifying the model structure and integrating pre-training with fine-tuning to maximize model performance.