CGFTNet: Content-Guided Frequency Domain Transform Network for Face Super-Resolution

Abstract

Recent advancements in face super resolution (FSR) have been propelled by deep learning techniques using convolutional neural networks (CNN). However, existing methods still struggle with effectively capturing global facial structure information, leading to reduced fidelity in reconstructed images, and often require additional manual data annotation. To overcome these challenges, we introduce a content-guided frequency domain transform network (CGFTNet) for face super-resolution tasks. The network features a channel attention-linked encoder-decoder architecture with two key components: the Frequency Domain and Reparameterized Focus Convolution Feature Enhancement module (FDRFEM) and the Content-Guided Channel Attention Fusion (CGCAF) module. FDRFEM enhances feature representation through transformation domain techniques and reparameterized focus convolution (RefConv), capturing detailed facial features and improving image quality. CGCAF dynamically adjusts feature fusion based on image content, enhancing detail restoration. Extensive evaluations across multiple datasets demonstrate that the proposed CGFTNet consistently outperforms other state-of-the-art methods.

1. Introduction

Facial super resolution (FSR), commonly referred to as facial hallucination, is a pivotal technology designed to enhance low-resolution (LR) images into high-resolution (HR) counterparts. This enhancement is particularly important in various applications, notably in identity recognition, video surveillance, and facial analysis. In practical settings, the quality of captured facial images often suffers due to variations in hardware configurations, positioning, and camera angles. Such degraded image quality significantly hinders essential tasks like facial recognition and analysis.

FSR tasks are inherently complex, as the nuanced structures and textural details of facial images, which occupy only a minor fraction of the overall face, are vital for distinguishing individual faces. These details are frequently obscured in LR images, presenting a daunting challenge in accurately restoring these critical features.

Historically, FSR methods such as bicubic interpolation, local feature analysis, and sparse representation have been employed. These techniques are adept at improving resolution where image content is straightforward, with clear edges and repetitive patterns. However, they falter with the intricate and varied structures typical of facial images. Convolutional neural networks (CNNs), known for their robust local feature extraction capabilities, have enhanced the prediction of subtle facial features, leading to the development of numerous CNN-based FSR frameworks [1,2,3]. Despite their success, CNNs generally lack the ability to model spatial correlations over extended distances due to their static filters, resulting in suboptimal recovery of facial structures and often blurred reconstructions.

To address the limitations of CNN in capturing global facial information, researchers are exploring architectures with enhanced global sensing capabilities. Some approaches have integrated auxiliary priors [4,5], such as facial parsing maps, landmark heatmaps, and 3D models, to aid reconstruction. Although these methods provide some improvement, they rely heavily on labor-intensive manual annotations, and inaccuracies in these annotations can degrade the reconstruction quality. The introduction of transformer models, which have revolutionized natural language processing, into the visual domain offers promising new directions for FSR tasks. Transformers excel in managing long-distance dependencies, thereby improving the reconstruction of the global facial structure and spawning a new generation of transformer-based FSR models [6,7,8,9].

Nevertheless, despite their proficiency in global information processing, transformers are computationally intensive and may lack sensitivity to localized details. Thus, it is crucial to balance the enhancement of local information processing with maintaining an extensive receptive field and minimizing computational demands. To this end, we propose a novel content-guided frequency domain transform network (CGFTNet) for FSR. Our network incorporates the Frequency Domain Reparameterized Focus Enhancement module (FDRFEM) and the Content-Guided Multi-Scale Fusion (CGCAF) module. FDRFEM enhances feature representation in the frequency domain to capture detailed facial structures, while CGCAF leverages a content-guided attention mechanism to optimally merge features from the encoder and decoder, ensuring effective feature preservation and enhancing super-resolution performance. Consistent with the architectural paradigms of the majority of preceding FSR models [10,11,12], we also utilize an encoder-decoder framework.

In summary, our contributions are threefold:

• We introduce the FDRFEM, which includes a frequency domain transformation branch and reparameterized focus convolution. This module enhances feature detail while preserving global structures.
• We present a novel module CGCAF that dynamically integrates rich features from both the encoder and decoder, facilitating the reconstruction of high-quality images.
• We develop the CGFTNet, a synergistic CNN-transformer network that achieves competitive performance on various FSR metrics.

2. Related Work

This section offers a thorough review of the advancements in image super-resolution technology, placing special emphasis on the role of deep learning in enhancing the resolution of natural and facial images. We also delve into the super-resolution techniques that are particularly effective under challenging conditions, such as those encountered in low-light scenarios. Furthermore, the discussion extends to the cutting-edge applications of transformer models within this domain and examines strategies for leveraging the synergistic strengths of CNN and transformers to achieve superior quality in super-resolution reconstruction. The following discourse elaborates on these technologies and their implications for the field of image super-resolution.

2.1. Super Resolution

Deep convolutional neural networks have achieved transformative advancements in enhancing the resolution of natural images, leveraging their superior feature extraction capabilities [13,14,15]. Li et al. [16] developed a novel multi-scale residual network (MSRN), which optimizes image feature utilization for super-resolution reconstruction using multi-scale residual blocks (MSRBs). Wang et al. [17] developed a lightweight remote-sensing image super-resolution network based on attention-driven multilevel feature fusion. Guo et al. [18] introduced an innovative dual regression network approach, incorporating additional constraints on low-resolution data to significantly reduce the potential mapping function space, thereby enhancing the performance of single-image super-resolution models. Gao et al. [19] designed a lightweight, efficient feature distillation interaction weighted network (FDIWN) for single-image super resolution (SISR), which, through its specially designed modules, achieves enhanced super-resolution performance with reduced computational overhead.

2.2. Face Super Resolution

CNN-based super-resolution technology has also significantly propelled the development of FSR. For example, Zhang et al. [20] proposed the super-identity convolutional neural network (SICNN), which improves identity information recovery in low-resolution facial images by employing super-identity loss and domain ensemble training methods, significantly boosting the recognizability of ultra-low-resolution faces. Additionally, some researchers have tailored FSR models to exploit facial inherent features, such as facial contour maps and feature points. Chen et al. [21] presented FSRNet, an end-to-end trainable network that exploits facial geometric priors, like facial landmark heatmaps and parsing maps, to reconstruct extremely low-resolution facial images without the need for precise alignment. Jin et al. [22] used a stepwise training approach, facial attention loss functions, and a streamlined facial alignment network (FAN) to produce highly realistic 8× super-resolution images that retain intricate facial details. Chan et al. [23] developed a hybrid covariance attention and cross-layer fusion transformer network. Although these models demonstrate promising results, they depend on extensive dataset annotations, and the accuracy of prior information significantly influences the reconstruction outcomes.

2.3. Face Super Resolution Under Specific Conditions

In addition to natural-state facial reconstruction, researchers have also explored face super-resolution techniques under specific conditions. Deepak Rai et al. [24] proposed a novel neuro-fuzzy inference-based local constraint representation (NFILcR) method for robust facial image super-resolution under low-light conditions, which substantially enhances the super-resolution performance of low-resolution facial images captured in such environments by integrating an adaptive neuro-fuzzy inference system (ANFIS) with local constraint representation methods. Yin et al. [25] developed MetaF2N, a blind image super-resolution method that fine-tunes model parameters through a meta-learning framework to target specific degradations in natural images, incorporating MaskNet to predict variable loss weights across different positions, thereby minimizing the disparity between recovered and actual facial images, which significantly impacts super-resolution performance.

2.4. Transformer

Recent attention has been drawn to transformer models, noted for their exemplary performance in multiple domains, including face super-resolution reconstruction tasks, thus advancing the field considerably. Bao et al. [6] unveiled a spatial attention guided CNN-transformer aggregation network (SCTANet), which synergistically blends CNN and transformer strengths to enhance the reconstruction quality and efficiency of facial details through a hybrid attention aggregation (HAA) module and sub-pixel MLP upsampling (SMU) module. Qi et al. [7] introduced an efficient latent style guided transformer-CNN framework (ELSFace), utilizing latent style encoding to direct the reconstruction of basic facial features, enhanced by high-frequency enhancement blocks (HFEB) and a sharp loss optimization algorithm to improve the quality and perceptual clarity of reconstructed facial images. Zhang et al. [26] developed a multi-stage auxiliary learning network. Gao et al. [8] developed a CNN-transformer cooperative network (CTCNet) with a multi-scale connected encoder-decoder backbone, featuring the Local-Global Feature Cooperation module (LGCM) and a multi-scale feature fusion unit (MFFU) that effectively concentrate on both local facial details and global facial structures, significantly improving face super-resolution reconstruction quality. Shi et al. [9] proposed a novel dual-branch network architecture that integrates the benefits of transformers and CNN through multi-scale parallel self-attention mechanisms and local variant convolutions, capturing both local and non-local dependencies in facial images to substantially enhance facial detail reconstruction quality. Although transformer architectures excel in capturing global image features, an over-reliance on global self-attention mechanisms may neglect important subtle local features. Thus, effectively integrating both macrostructures and microdetails remains crucial for achieving high-quality image reconstruction, which is the central aim of this research.

In summary, the domain of image super-resolution has experienced significant advancements, largely attributable to the emergence of deep learning methodologies. The convergence of CNN and transformers has paved the way for innovative strategies aimed at resolution enhancement, particularly under challenging circumstances such as low-light scenarios. The cutting-edge techniques reviewed herein not only underscore the efficacy of these architectures in the reconstruction of high-fidelity images but also accentuate the necessity of harmonizing global and local features to achieve superior outcomes. As ongoing research endeavors to expand the frontiers of the feasible, the synergistic integration of CNN, and transformers is poised to transform our approach to image super-resolution.

3. Architectural Details

In this section, we begin with a comprehensive overview of the overall architecture of the proposed CGFTNet. Subsequently, we explore in detail the critical components that constitute the network.

3.1. Overview of CGFTNet

As depicted in Figure 1, our proposed CGFTNet, which is derived from the baseline model CTCNet [8], features a tripartite U-shaped architecture, consisting of encoding, bottleneck, and decoding stages. In the encoding stage, the network is tailored to capture multi-scale local and global features; in contrast, the decoding stage focuses on the integration of these features and the reconstruction of the image. Moreover, to facilitate effective integration of features, connections along with FDRFEM are implemented between the encoding and decoding stages. For a clearer exposition of our model’s architecture, we introduce several key terminologies: $I_{LR}$ denotes the input low-resolution image, $I_{SR}$ refers to the super-resolution image produced by the model, and $I_{HR}$ represents the actual high-resolution image.

(1)

Encoding Stage: As outlined earlier, our objective in the feature extraction phase is to capture the inherent structure of the image. Beginning with a low-resolution input image $I_{LR}$, we initially employ a $3\times 3$ convolutional kernel for basic feature extraction. These features are then processed through three consecutive stages of feature extraction. Each stage incorporates a Local-Global Coherence module (LGCM), which consists of a facial structure awareness unit (FSAU) and a transformer unit. For more details about LGCM, FSAU, and FRM, please refer to [8]. Following this, the features are subjected to a downsampling process, involving a $3\times 3$ convolutional layer with a stride of 2 designed to reduce the dimensions of the feature maps while retaining essential information. This is coupled with a LeakyReLU activation function and another $3\times 3$ convolutional layer. As a result, each feature extraction stage halves the size of the output feature maps and doubles the channel count. For example, given an initial feature map $I_{LR}\in {\mathbb{R}}^{C\times H\times W}$, then at the $\mathit{i}\mathrm{-th}$ feature extraction stage, the encoder will produce a feature map: $I_{en}^i\in {\mathbb{R}}^{2^{i}C\times {\displaystyle \frac{H}{2^i}}\times {\displaystyle \frac{W}{2^i}}}$.

(2)

Bottleneck: Situated between the feature extraction and reconstruction phases is a critical bottleneck stage. In this stage, all features accumulated during the encoding process are consolidated. To ensure these features are effectively leveraged in the subsequent reconstruction phase, a Feature Refinement module (FRM) is employed. The FRM is designed to optimize and amplify the encoded features, enhancing the delineation of facial structure details. The intervention of the FRM allows for a more focused processing of facial features, thereby enhancing information expression across different facial regions.

(3)

Decoding Stage: The objective of the decoding stage is to restore high-quality facial images. We introduce an innovative module, CGCAF. The decoder receives deep features from the low-resolution image and integrates these features progressively via the CGCAF to construct a super-resolution image. As illustrated in Figure 1, the decoder comprises an upsampling module, the CGCAF, and the Local-Global Coherence module (LGCM). The upsampling module includes a $6\times 6$ transposed convolution layer with a stride of 2, followed by a LeakyReLU activation function and a $3\times 3$ convolution layer. This transposed convolution layer expands the feature map dimensions and facilitates information extraction. Consequently, the decoder reduces the number of output feature channels while doubling the feature map dimensions with each step. The CGCAF fuses features extracted during the encoding phase, ensuring comprehensive utilization of both local and global features from the encoding and decoding stages to produce high-quality facial images. The decoding phase culminates with a $3\times 3$ convolution layer that maps the integrated features to the final super-resolution output $I_{Out}$.

Ultimately, an enhanced facial image is produced by merging the low-resolution image $I_{LR}$ with the super-resolution output $I_{Out}$. Throughout the training phase, provided with a dataset ${\left\{I_{LR}^i,I_{HR}^i\right\}}_{i=1}^N$, we refine the parameters of our CGFTNet model by minimizing a pixel-level loss function:

$$\mathcal{L}\left(\mathsf{\Theta }\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|\right|F_{CGFTNet}(I_{LR}^i,\mathsf{\Theta })-I_{HR}^i{\left|\right|}_1,$$

(1)

where N denotes the total number of training images. The terms $I_L^{i}R$ and $I_H^{i}R$ correspond to the low-resolution (LR) image and the associated true high-resolution (HR) image of the i-th image, respectively. Concurrently, $F_{CGFTNet}(·)$ and $\mathsf{\Theta }$ refer to the CGFTNet network and its respective network parameters. The notation $\left|\right|·{\left|\right|}_1$ represents the L1 norm.

3.2. Content-Guided Channel Attention Fusion

To augment the capability for feature representation and to enhance the integration of features between the encoder and decoder, we have incorporated a content-guided channel attention fusion strategy in the decoding phase. This approach strengthens the convolutional neural network’s ability to represent distinct regions within the features, particularly those regions that are crucial for facial restoration tasks.

Specifically, our principal goal is to explore and exploit the features derived during the encoding stage in the decoding process. Traditional convolution layers, known as vanilla convolution, lack constraints in exploring solution spaces, such as spatial and channel attentions, which are typically computed independently, without interchanging information between channel and spatial dimensions. This results [17] in suboptimal utilization of facial priors such as structural, color, and texture consistency and does not adequately account for the unique significance of each channel within the feature space or the contextual information across spatial dimensions. To address this challenge, we designed the CGCAF module, detailed in Figure 2. As outlined, CGCAF first computes channel and spatial attentions using global average pooling (GAP) and global max pooling (GMP), along with $1\times 1$ and $7\times 7$ convolution layers. It then merges channel and spatial attentions through a straightforward addition process to create a preliminary spatial importance map (PSIM). Following this, the content of the input features guides the production of the final channel-specific spatial importance map (SIM). This is achieved by alternating the arrangement of each channel of feature X through channel shuffle operations, followed by group convolution and activation via a sigmoid function. To effectively merge features from the encoding and decoding processes at corresponding levels, they are weighted and summed based on the SIM. This approach acknowledges that different layers have varying receptive fields, and simple additive or concatenative operations are insufficient to address mismatches in receptive fields before fusion. The specific formula is provided below:

$$X=F_{en}+F_{de},$$

(2)

$$F_C=C_{1\times 1}\left(max(0,C_{1\times 1}\left(X_{GAP}^C\right))\right),$$

(3)

$$F_S=C_{7\times 7}\left([X_{GAP}^S,X_{GMP}^S]\right),$$

(4)

$$F_{COS}=F_C+F_S,$$

(5)

$$F=\sigma \left(G{C}_{7\times 7}\left(GS\left([X,F_{COS}]\right)\right)\right),$$

(6)

where $max(0,x)$ represents the activation function ReLU, $C_{k\times k}(·)$ denotes a convolution with a kernel size of $k\times k$, and $[·]$ signifies a channel-wise concatenation operation. $F_C$, $F_S$, $X_{GAP}^C$, $X_{GAP}^S$ and $X_{GMP}^S$ respectively refer to features processed by channel attention, features processed by spatial attention, global average pooling across the channel dimension, global average pooling across the spatial dimension, and global max pooling across the spatial dimension. $\sigma$ represents the activation function Sigmoid, $CS(·)$ indicates channel shuffle operations, and $G{C}_{k\times k}(·)$ denotes a grouped convolution layer with a kernel size of $k\times k$. By incorporating skip connections, the input features are directly added to the fusion process, aiding in mitigating the problem of vanishing gradients and simplifying the learning trajectory. A $1\times 1$ convolution then aligns the channels with the subsequent network layers. For example, the feature maps acquired during the encoding phase have channel counts of 32, 64, and 128. According to our overall network design requirements, the 128-channel feature maps from the encoding stage must also emerge as 128 channels after processing through our CGCAF module. The purpose of the post-concatenation $1\times 1$ convolution is precisely this, where as the pre-concatenation $1\times 1$ convolution is intended to reduce computational demands. It is evident that the two feature maps sourced from the encoding and decoding phases, along with the feature map derived through weighted summation, ultimately undergo a concatenation operation. This aims to minimize the loss of prior knowledge during the complex computational process.

Due to the extensive concatenation of information, some channel importance information is lost; we resolve this issue using a channel attention (CA) strategy. As illustrated in Figure 3, the specific architecture of CA comprises adaptive average pooling, two $1\times 1$ convolutions, dual activation functions, and a residual connection. The complete mathematical formulation of the CGCAF module is as follows:

$$F_{out}=CA\left(C_{1\times 1}\left(Concat(F_{en},F_{de},C_{1\times 1}\left(F_p\right))\right)\right),$$

(7)

$$F_p=F_{en}+F_{de}+F_{en}\times F+F_{de}\times (1-F),$$

(8)

where $CA$ represents the channel attention and F refers to the spatial importance map (SIM) mentioned earlier.

3.3. Focus Convolution Feature Enhancement Module

As a critical module in CGFTNet, the FDRFEM is designed to facilitate global spatial interactions. As illustrated in Figure 4, FDRFEM is composed of a frequency-domain branch and a reparameterized focused convolution, each responsible for global and local feature extraction, respectively.

(1)

Frequency-Domain Branch: In the context of FSR, a major challenge lies in effectively extracting key facial features (such as eyes, eyebrows, and mouth) while simultaneously guiding the network to focus on these features. To address this, we leverage Fourier transform to enable the model to extract critical information for improved detail recovery. As depicted in Figure 4, the frequency-domain branch first transforms the input features into the frequency domain via a 2D real-valued fast Fourier transform (FFT). In the frequency domain, the feature maps are modulated by a set of learnable complex filters, which selectively adjust the frequency components of the feature maps, enhancing or suppressing specific frequencies. This process allows the model to capture the global frequency characteristics of the input data. Subsequently, an inverse fast Fourier transform (iFFT) is applied to convert the features back into the spatial domain, allowing the model to exploit these frequency-enhanced feature maps for improved performance.

(2)

Reparameterized Focused Convolution: As mentioned earlier, the frequency-domain branch simulates a convolution operation in the spatial domain, utilizing kernels of global size and circular padding, thereby capturing spatial information across the entire feature map and facilitating the modeling of long-range dependencies. The reparameterized focused convolution is designed to capture local information and enhance the model’s representational capacity through reparameterization without incurring additional inference costs. For a pre-trained model, RefConv applies a learnable refocusing transformation to the base convolution kernels inherited from the pre-trained model, establishing parameter correlations. For instance, in depth-wise convolution, RefConv allows the convolution kernel parameters of one channel to be linked with those of other channels, enabling a broader refocus beyond the input features alone. In the case of depth-wise convolution, RefConv utilizes the pre-trained convolution kernel as a base weight and applies the refocusing transformation to generate new convolution kernels. This transformation incorporates trainable refocusing weights that are combined with the base weights to produce transformed weights for processing the input features. The objective of the refocusing transformation is to learn incremental adjustments to the base weights, analogous to residual learning, where residual blocks capture incremental changes to the base feature maps. In doing so, RefConv enables each convolution channel to establish connections with other channels, facilitating the learning of novel feature representations.

By integrating the frequency-domain branch and reparameterized focused convolution, FDRFEM effectively captures both local facial features and global relationships, thereby enabling high-quality image reconstruction.

4. Experiment

4.1. Datasets

In our experiments, we employed the CelebA [27] dataset for training and evaluated the model’s performance on the Helen [28] dataset. Since the face images in CelebA exhibit varying heights and widths, we first cropped the images based on their central points and resized them to a fixed resolution of $128\times 128$ pixels to serve as high-resolution (HR) images. Subsequently, we downsampled these high-resolution (HR) images to $16\times 16$ pixels, and then upsampled them back to $128\times 128$ pixels using bicubic interpolation, treating them as the low-resolution (LR) inputs. For all subsequent experiments that required interpolation, we used bicubic interpolation, and all conclusions are based on this premise. The model was trained using 18,000 samples from the CelebA, with an additional 1000 samples reserved for testing. Furthermore, we evaluated the model’s generalization ability by directly testing it on the Helen dataset using the model pre-trained on CelebA.

4.2. Implementation Details

We implemented our model using the PyTorch framework and optimized it with the Adam optimizer, setting ${\beta }_1$ to 0.9 and ${\beta }_2$ to 0.99. The initial learning rate was configured to $2\times {10}^{-4}$. To assess the quality of the super-resolution (SR) results, we utilized four objective image quality evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [29], Learned Perceptual Image Patch Similarity (LPIPS) [30], and Visual Information Fidelity (VIF) [31].

4.3. Ablation Studies

In this section, we present a series of ablation studies to validate the effectiveness of our proposed model. All ablation studies were conducted on the CelebA test set.

(1)

Effectiveness of FDRFEM: The FDRFEM is a key component of CGFTNet, designed to achieve global spatial interactions. To evaluate the effectiveness of FDRFEM and the feasibility of this integration, we conducted a set of ablation studies. FDRFEM comprises a frequency-domain branch and a reparameterized focused convolution. Thus, we designed three modified models for comparison. The first model removes all FDRFEM from the encoder and decoder, labeled as “w/o FDRFEM”. The second model removes all frequency-domain branches while retaining the reparameterized focused convolutions, labeled as “FDRFEM w/o Fourier”. The third model removes all reparameterized focused convolutions while keeping the frequency-domain branches in FDRFEM, labeled as “FDRFEM w/o Refconv”. The results of these modified networks are shown in

Table 1. Verify the effectiveness of FDRFEM in Figure 4.

Methods	PSNR	SSIM	VIF	LPIPS
w/o FDRFEM	28.062	0.8049	0.5364	0.1514
FDRFEM w/o Fourier	28.065	0.8067	0.5363	0.1487
FDRFEM w/o Refconv	28.120	0.8071	0.5369	0.1513
FDRFEM	28.125	0.8059	0.5374	0.1481

. Based on these results, we can draw the following observations:

(a)

By comparing the first and last rows of Table 1, we observe that the introduction of FDRFEM significantly improves model performance, confirming its effectiveness.

(b)

Comparing the first three rows, it becomes clear that either the frequency-domain branch or the reparameterized focused convolution independently improves model performance, as both global relationships and local features contribute to image reconstruction.

(c)

The comparison of the last three rows further demonstrates that both the frequency-domain branch and reparameterized focused convolution serve distinct roles in the FSR task. The frequency-domain branch helps the network capture long-range dependencies, while the reparameterized focused convolution captures local details, providing complementary information for the final SR image reconstruction. Relying on only one of these components leads to suboptimal performance, further validating the effectiveness of FDRFEM and the viability of combining CNN and transformer architectures.

(2)

Effectiveness of CGCAF: The CGCAF module is specifically designed for feature fusion between the encoding and decoding stages. In this section, we conduct a series of experiments to demonstrate the efficacy of this connection. The first two experiments verify the necessity of content-guided attention (CGA), which refers to the part of the CGCAF module that precedes the acquisition of the $F_p$ feature map, as shown in the Figure 2. The third and fourth experiments retain CGA but apply only concatenation or addition operations for feature fusion. The fifth and sixth experiments explore the necessity of the final channel attention mechanism. The last two experiments, where neither CGA nor channel attention is used, serve as baselines. Each set of experiments includes a comparison between the Concat and Add operations to explain our final choice of Concat. From the results in

Table 2. Verify the effectiveness of CGCAF in Figure 2.

CGA	Concat	Add	CA	PSNR	SSIM
×	✓	×	✓	28.125	0.8059
×	×	✓	✓	27.991	0.8039
✓	✓	×	✓	28.161	0.8075
✓	×	✓	✓	28.159	0.8074
✓	✓	×	×	28.153	0.8069
✓	×	✓	×	28.047	0.8047
×	✓	×	×	28.121	0.8050
×	×	✓	×	27.977	0.8028

, we can observe the following:

(a)

Employing feature fusion strategies between the encoding and decoding stages can significantly impact model performance, which underscores the importance of investigating strategies that leverage encoding-stage features during decoding for image reconstruction.

(b)

By comparing the third row with the fifth, we can find that the channel attention (CA) mechanism also positively impacts model performance.

(c)

By comparing the third row with the others, we can observe that the combination of the Concat operation and CA yields the best results.

(d)

By comparing each pair of experiments that includes CGA with those that do not, such as the first row with the third row, the second row with the fourth row, we can observe that models incorporating CGA demonstrate superior performance metrics compared to those without.

(3)

Effectiveness of two key modules: Based on the results presented in the

Table 3. Verify the effectiveness of CGFTNet in Figure 1.

Baseline	Threelines	FDRFEM	CGCAF	PSNR	SSIM
✓	×	×	×	27.990	0.8036
✓	✓	×	×	28.062	0.8049
✓	✓	✓	×	28.125	0.8059
✓	✓	×	✓	28.034	0.8039
✓	✓	✓	✓	28.161	0.8075

, several observations can be made. We adopt CTCNet [8] as our baseline model. The term “Threelines” refers to a pruned variant of CTCNet, wherein the number of feature lines interconnecting the encoder and decoder has been reduced, retaining only three lines. Comparing the first and second rows, it is evident that the pruned model performs better than the baseline. The comparison between the second and third rows indicates that introducing the FDRFEM module alone improves the model’s performance. Analyzing the last four rows shows that simultaneously incorporating both FDRFEM and CGCAF yields higher performance than introducing each module individually.

4.4. Comparison with Other Methods

In this section, we compare our CGFTNet with several methods, including classical FSR techniques such as SPARNet [32] and SISN [33], as well as more recent approaches like ELSFace [7], SCGAN [34], SCTANet [6], SFMNet [35], and the pioneering transformer-based image restoration model SwinIR [36]. For a fair comparison, all models were trained on the same CelebA. (See Figure 5).

(1)

Comparison on the CelebA: Quantitative comparisons on the CelebA test set with other SOTA methods are presented in

Table 4. Quantitative comparisons for x8 SR on the CelebA and Helen test sets.

Methods	CelebA				Helen
	PSNR	SSIM	VIF	LPIPS	PSNR	SSIM	VIF	LPIPS
Bicubic	24.05	0.6449	0.4019	0.5697	23.79	0.6739	0.4353	0.5436
SwinIR	27.88	0.7967	0.4590	0.2001	26.53	0.7856	0.4398	0.2644
SISN	27.91	0.7971	0.4785	0.2005	26.64	0.7908	0.4623	0.2571
ELSFace	28.02	0.8018	0.5232	0.1526	26.90	0.8013	0.4931	0.1874
SCGAN	28.03	0.8013	0.5236	0.1511	27.04	0.8115	0.5056	0.1809
SCTANet	28.03	0.8017	0.5240	0.1520	26.85	0.8074	0.5041	0.1911
SFMNet	28.04	0.8024	0.5245	0.1496	27.09	0.8094	0.4989	0.1788
SPARNet	27.73	0.7949	0.4505	0.1995	26.43	0.7839	0.4262	0.2674
CTCNet	27.99	0.8036	0.5348	0.1559	26.96	0.8048	0.5245	0.1797
Ours	28.16	0.8075	0.5369	0.1455	27.18	0.8129	0.5294	0.1748

. The results clearly demonstrate that CGFTNet outperforms competing methods in terms of PSNR, VIP, LPIPS, and SSIM. These findings highlight the superior performance and effectiveness of CGFTNet. Additionally, as shown in the visual comparisons in Figure 6, most previous methods struggle to accurately restore facial features, such as the eyes and nose, whereas our CGFTNet reconstructs facial structures with greater precision, producing results that more closely resemble the original high-resolution (HR) images. This further substantiates the advantages of CGFTNet in facial restoration tasks.

(2)

Comparison on the Helen dataset: To assess the generalization capability of CGFTNet, we directly tested the model, trained on the CelebA, on the Helen test set. The quantitative results for ×8 SR experiments on the Helen test set are provided in Table 4. As shown in Table 4, CGFTNet consistently achieves the best performance on the Helen dataset as well. Furthermore, the visual comparisons in Figure 7 reveal that the performance of many competing methods significantly degrades on this dataset, failing to accurately restore facial details. In contrast, CGFTNet maintains its ability to recover realistic facial contours and details, further affirming its robustness and generalizability.

(3)

Comparison of real-world facial images: Recovering facial images in real-world scenarios remains a formidable challenge. While the previously mentioned experiments were conducted in simulated environments, these settings do not fully capture the complexity of real-world conditions. To further validate the effectiveness of CGFTNet, we performed experiments on low-quality facial images from the WiderFace dataset [37], which were captured in natural scenes, representing real-world diversity and complexity. These images are inherently low-resolution, requiring no additional downsampling. In this context, we aim to recover facial images with finer texture details and well-preserved facial structures. Figure 7 illustrates a visual comparison of the reconstruction performance on real-world images. Thanks to the collaborative CNN-Transformer mechanism and the specifically designed modules of CGFTNet, our method effectively restores cleaner facial details and more accurate facial structures. Furthermore, we evaluated the performance of CGFTNet in downstream tasks, such as facial matching. For this experiment, high-resolution frontal facial images of the candidates were used as source samples, while the corresponding low-resolution facial images from real-world conditions served as target samples. To ensure the robustness of our results, we conducted 8 separate trials, randomly selecting five pairs of candidate samples in each trial and calculating the average similarity. The quantitative results, presented in

Table 5. Comparison results for average similarity of face images super-resolved by different methods.

Methods	Average Similarity
	Case1	Case2	Case3	Case4	Case5	Case6	Case7	Case8
SwinIR	0.8942	0.8980	0.9031	0.8958	0.8958	0.9096	0.9019	0.8827
SISN	0.8930	0.9125	0.8960	0.9085	0.8888	0.9041	0.9075	0.8731
ELSFace	0.9111	0.9196	0.8963	0.9056	0.9084	0.9047	0.9139	0.8940
SCGAN	0.8984	0.8836	0.9038	0.8979	0.8917	0.9043	0.8911	0.8964
SCTANet	0.8855	0.8786	0.8699	0.8777	0.8760	0.8775	0.8840	0.8741
SFMNet	0.9032	0.8996	0.9055	0.9037	0.9051	0.8956	0.8966	0.8887
SPARNet	0.9076	0.9186	0.9013	0.9007	0.9054	0.9004	0.9080	0.9043
CTCNet	0.9138	0.9191	0.9108	0.9026	0.9066	0.9018	0.9145	0.9069
Ours	0.9278	0.9237	0.9109	0.9184	0.9242	0.9216	0.9202	0.9138

, demonstrate that our method consistently achieves higher similarity scores across all trials. This further demonstrates that CGFTNet can generate more realistic high-resolution facial images in real-world surveillance applications, showcasing its high practicality and adaptability.

4.5. Model Complexity Analysis

The results demonstrate that our model surpasses most competing methods in both quantitative and qualitative evaluations. Furthermore, model size and execution time are crucial metrics for assessing model efficiency. In Figure 8, we present a comparison of parameter count, performance, and execution time across models. Notably, our CGFTNet achieves the best quantitative performance while maintaining comparable execution time and parameter count. Overall, CGFTNet achieves an optimal balance between model size, performance, and execution time.

5. Conclusions

In this study, we introduce a deep learning framework termed CGFTNet, meticulously engineered for the purpose of FSR. Our methodology is anchored in an encoder-decoder architecture, augmented with two pioneering modules: the FDRFEM and the CGCAF. These modules are pivotal in capturing both the minutiae of local facial features and the overarching structures of the face, which are essential for achieving high-fidelity image reconstruction.

The FDRFEM module incorporates a frequency-domain branch that harnesses the fast Fourier transform (FFT) to distill global features, complemented by a reparameterized focused convolution that refines the representation of local features. This synergistic approach enables our model to adeptly capture and synthesize both global and local facial attributes, thereby enhancing super-resolution performance.

The CGCAF module is crafted to dynamically amalgamate features from the encoder and decoder phases, ensuring that the reconstructed image is enriched with a comprehensive array of features that encapsulate the intricacies of facial morphology. This content-guided attention mechanism optimizes the feature fusion process, culminating in superior detail restoration and elevated image quality.

Through rigorous experimentation on both synthetic datasets, such as CelebA, and real-world datasets, we have demonstrated that CGFTNet surpasses several state-of-the-art methodologies across a spectrum of quantitative metrics, including peak signal-to-noise ratio (PSNR), Structural Similarity Index (SSIM), learned perceptual image patch similarity (LPIPS), and visual information fidelity (VIF). Qualitative assessments further reveal that our model adeptly reconstructs facial images with a higher degree of accuracy and detail, particularly in the depiction of intricate features such as eyes, noses, and mouths, which often elude other methods.

Additionally, we have conducted experiments to appraise the generalization prowess of CGFTNet by subjecting it to the Helen dataset and real-world facial imagery from the WiderFace dataset. The findings indicate that our model not only excels on standardized datasets but also demonstrates robustness and efficacy in real-world contexts, underscoring its practical applicability.

Regarding model complexity, CGFTNet achieves a judicious equilibrium between performance, parameter count, and execution time. Our analysis reveals that, while attaining superior quantitative performance, CGFTNet maintains a comparable level of efficiency, rendering it a feasible solution for real-world applications where computational resources may be constrained.

In conclusion, the CGFTNet signifies a substantial leap forward in the realm of facial super resolution. Its proficiency in restoring high-quality facial images from low-resolution inputs has far-reaching implications for applications such as identity recognition, video surveillance, and facial analysis. The model’s proficiency in downstream tasks, including facial matching, further accentuates its practical utility. Future endeavors will concentrate on expanding the model’s capabilities to address more complex and diverse real-world conditions, as well as exploring additional domains where the reconstruction of high-quality facial images is paramount.