Feature Intensification Using Perception-Guided Regional Classification for Remote Sensing Image Super-Resolution

Abstract

In recent years, super-resolution technology has gained widespread attention in the field of remote sensing. Despite advancements, current methods often employ uniform reconstruction techniques across entire remote sensing images, neglecting the inherent variability in spatial frequency distributions, particularly the distinction between high-frequency texture regions and smoother areas, leading to computational inefficiency, which introduces redundant computations and fails to optimize the reconstruction process for regions of higher complexity. To address these issues, we propose the Perception-guided Classification Feature Intensification (PCFI) network. PCFI integrates two key components: a compressed sensing classifier that optimizes speed and performance, and a deep texture interaction fusion module that enhances content interaction and detail extraction. This network mitigates the tendency of Transformers to favor global information over local details, achieving improved image information integration through residual connections across windows. Furthermore, a classifier is employed to segment sub-image blocks prior to super-resolution, enabling efficient large-scale processing. The experimental results on the AID dataset indicate that PCFI achieves state-of-the-art performance, with a PSNR of 30.87 dB and an SSIM of 0.8131, while also delivering a 4.33% improvement in processing speed compared to the second-best method.

1. Introduction

Remote sensing images, acquired using remote sensing technology, contain a wealth of geographical information and depict the surface or atmosphere of the Earth. These images are extensively utilized in several industries, including catastrophe monitoring, agricultural and forestry surveys, resource exploration, and even military surveillance. However, the visual clarity of remote sensing images acquired from satellites is frequently constrained by the capabilities of the imaging devices, optical sensors, and long-distance transmission, as well as environmental interference, resulting in noise that diminishes the quality of the image [1]. This leads to a decrease in the amount of useful information, which impairs the ability to further analyze and process the images. Due to the expensive nature of hardware upgrades, research has turned towards software-based methods for processing remote sensing images at a more affordable price. As an extensively utilized method for enhancing image resolution, image super-resolution (SR) technology can greatly improve the quality of remote sensing images.

Image super-resolution technology is a crucial technique for enhancing image resolution and quality. The objective of super-resolution is to generate high-resolution (HR) images from low-resolution (LR) images while preserving essential image details [2,3,4]. Traditional super-resolution techniques, such as the iterative back-projection method developed by Irani et al. [5], have achieved significant success in restoring image information and laid the foundation for this field. However, these traditional techniques are limited when dealing with complex scenes or lower-quality images, resulting in suboptimal reconstruction quality characterized by artifacts such as jagged edges and blurriness, which negatively impact the overall reconstruction quality.

As an important part of deep learning, convolutional neural networks (CNNs) [6] have attracted a lot of interest from academics looking to improve super-resolution (SR) technology. Due to their superior feature learning and representation capabilities, CNNs have demonstrated excellent performance in super-resolution reconstruction [7,8]. The initial CNN-based SR reconstruction network, SRCNN [9], shows substantial improvements over traditional techniques. Lim et al. [10] make significant progress in this field by developing EDSR, which is built upon SRResNet and eliminates batch normalization in each module. However, because CNN models typically employ fixed-size convolutional kernels, they struggle to effectively capture long-range dependencies and allocate computational resources efficiently. This results in redundancy in smooth regions and insufficient attention to edges and texture regions, failing to adequately focus on high-frequency features. With the advent of the Transformer [11], this novel deep learning architecture has also been applied to SR tasks. The Transformer uses a self-attention mechanism as the core component of its encoder–decoder [12] structure, which helps the model efficiently capture comprehensive information from input sequences. This allows for more organized and effective feature extraction. Dosovitskiy et al. [13] propose the Vision Transformer (ViT) model as a solution for image classification. This model demonstrates superior capability in capturing holistic information from images, outperforming earlier CNN-based SR models.

However, even with the application of Transformers to super-resolution tasks, existing methods still exhibit significant shortcomings. Firstly, while the multi-scale self-attention mechanism employed by Transformers greatly enhances the capture of global information dependencies, it neglects local edge details. This oversight poses a challenge in effectively reconstructing precise features in the generated images, particularly when dealing with remote sensing images in complex terrains or dense urban environments, which may lead to inaccuracies in the restored information. Secondly, many models seek to improve performance by increasing depth; however, the complexity of self-attention grows quadratically with the size of the image blocks. Although this complexity allows for parallel computation, it also incurs higher computational overhead, thereby reducing the processing speed. This drawback becomes particularly pronounced in large-scale super-resolution tasks for satellite-acquired remote sensing images, further intensifying the demand for computational hardware. Consequently, balancing model depth with computational efficiency emerges as a key challenge in optimizing super-resolution models. To accelerate large-scale image SR tasks, Kong et al. [14] propose ClassSR, which first classifies [15] and then performs SR tasks. RL-Restore [16] and Path-Restore [17] methods decompose the image into sub-images and then use reinforcement learning to estimate and select suitable processing paths. However, these methods still suffer from limited receptive fields, resulting in inflexible partitioning, numerous partition errors, and overall lower model performance.

In order to tackle the previously mentioned concerns, we suggest implementing a novel and innovative technique for super-resolution known as Perception-guided Classification Feature Intensification Network (PCFI). PCFI consists of two key modules: Integrated compressive sensing (CS)-based perception classifier module, abbreviated as ICPC, and depth–texture interaction fusion module, abbreviated as DTIF. ICPC utilizes a more concise and focused approach to classifying features. This module employs the CS technique to extract data from the image. We denote the mapping domain, where the measurements derived from compressed sensing are utilized for analyzing image features, as the perceptual domain. The sampling rate is defined, and the resulting sensing matrix has a limited quantity of sampled data. Subsequently, this sensing matrix is fed into a pre-trained model designed to extract perceptual domain features for compilation. By learning the perceptual features of the image for classification, ICPC can better capture semantic information and visual features within the image.

Since perceptual domain features are less sensitive to noise and redundant information in the image [18], ICPC can reduce the influence of these factors on classification results, thereby improving the stability and robustness of classification. In addition, ICPC offers more precise previous information for subsequent image super-resolution operations, hence enhancing the quality of image reconstruction. By integrating perceptual domain feature classification, ICPC not only boosts the accuracy of image classification but also enhances the quality of reconstructing huge remote sensing images while speeding up SR activities. Consequently, it improves the overall effectiveness and efficacy of image processing. The other primary module, DTIF, is based on the Swin Transformer framework and employs window interactions to extract deep texture information, thereby accurately restoring the detailed features of complex textured regions.

To address the inclination of Transformer to prioritize overall information while neglecting specific details, we incorporate an N–Gram language model into the window sliding component. Window-based Self-Attention (WSA) enables pixels to interact with each other across windows, resulting in more precise capture of detailed information. Additionally, within DTIF, we incorporate the Cross-Window Importance Aggregation (CWIA) block that spans both W-Trans and SW-Trans, residually connecting with the output of the self-attention mechanism. This block implements a technique called two-dimensional channel-wise local significance pooling to improve the representation of edge blur features. The well-designed PCFI network addresses the limitations of previous methods caused by the self-attention mechanism and their poor performance on large images. Such a manner exhibits substantial benefits in handling extensive remote sensing images, consequently enhancing the overall quality and efficiency of image super-resolution operations.

In summary, this paper presents the following contributions:

• Proposed Perception-guided Classification Feature Intensification Network. Unlike traditional super-resolution methods, PCFI significantly enhances processing speed and improves image reconstruction quality by simultaneously extracting detailed features for super-resolution tasks.
• Constructed Integrated Compressive Sensing-based Perception Classifier Module. ICPC leverages perceptual domain features to classify image blocks, substantially improving classification accuracy and effectively accelerating large-scale image reconstruction tasks.
• Designed Depth–Texture Interaction Fusion Module. DTIF integrates attention mechanisms and texture interactions, enhancing information exchange across windows and spatial dimensions, thereby strengthening the representation of local details in complex textures or edge areas. This approach achieves more precise restoration of degraded image details.

This paper contains a total of five chapters, and the four subsequent chapters successively present the related work of the article, the methodology, the experiments, and finally summarize the methodology of this paper.

2. Related Work

2.1. Super-Resolution in Natural Images

In recent years, advancements in super-resolution reconstruction technology have significantly propelled the development of low-resolution image processing. Most models have been improved and innovated based on natural image datasets such as ImageNet and DF2K. Concurrently, the construction of diverse datasets and enhancements in computer performance have led to the emergence of numerous models based on classic networks like CNNs and GANs. Dong et al. [9] introduce SRCNN, the inaugural CNN network model for single-image super-resolution reconstruction. This model is capable of immediately acquiring the end-to-end mapping from low-resolution to high-resolution images. This model features a simple network structure and high pixel accuracy, but its numerous parameters make training challenging. VDSR [19], a classic residual model proposed by Kim et al., uses fewer parameters and increases network depth to extract more feature maps. This approach demonstrates that deeper networks facilitate image reconstruction. The model, although easier to train and able to improve the reconstruction accuracy more than SRCNN, still suffers from high training difficulty and low speed. Lai et al. [20] propose LapSRN, which incorporates the traditional Laplacian Pyramid, representing a further enhancement over previous residual models. This model enlarges feature maps progressively through stepwise upsampling, allowing for residual prediction at each level, thus improving training speed. SRGAN [21], based on the GAN network, employs a deep residual network as the generative function and uses perceptual loss as the optimization target. By training both the generator and discriminator simultaneously, it produces more natural images. However, the high complexity of the network results in significant training difficulty and lower pixel accuracy of the reconstructed images. ESRGAN [22] builds on SRGAN by integrating multi-level residual networks and dense connections into dense residual blocks. This model is relatively more stable and produces reconstructed images with richer texture details.

2.2. Super-Resolution in Remote Sensing

As super-resolution technology in remote sensing imagery advances, researchers are increasingly concentrating on techniques to improve the resolution of these images. Remote sensing photographs exhibit notable dissimilarities from natural images due to the inherent interdependence of objects and surroundings in remote sensing images, necessitating the inclusion of environmental data. Liebel et al. [9] are the first to propose the Sentinel-2 remote sensing image dataset to train SRCNN for single-image super-resolution reconstruction in remote sensing. However, due to the multi-scale nature of remote sensing images, the reconstruction results were still suboptimal. To address multi-scale feature extraction, Lei et al. [23] introduce LGCNet based on CNN for remote sensing images. This network cascades the results from different layers, employing a “multi-branch” structure to learn multi-level representations of remote sensing images. Compared to SRCNN, the quality of the reconstructed images is better, but the large number of parameters results in slower processing speeds. Xu et al. [24] propose DMCN, a symmetric hourglass-shaped CNN architecture. Through the design of multiple skip connections, the network complexity is significantly reduced, and the processing time is shortened. However, the reconstruction results do not show a substantial improvement and still exhibited artifacts. Jiang et al. [25] introduce EEGAN, a method that employs an adversarial learning approach to enhance remote sensing images by reducing noise sensitivity and improving the restoration of high-frequency edge details. These methods provide various solutions that develop the resolution of remote sensing images by utilizing distinct network topologies and learning algorithms. Therefore, they contribute to the advancement of remote sensing image processing.

2.3. Transformer in Super-Resolution

Transformer-based super-resolution reconstruction is an innovative approach in recent years. The Transformer model relies entirely on the self-attention mechanism, which replaces the conventional RNN sequence processing with a parallelized self-attention mechanism to address the issue of long-range dependencies. Transformer-based super-resolution reconstruction methods are mainly classified into two main categories, one is the Transformer that combines the attention mechanism and CNN network, and the other is the Transformer that consists of the attention mechanism only. Liang et al. [26] propose SwinIR, which integrates Swin Transformer with SR tasks for the first time. This model divides the image into blocks and applies self-attention mechanisms at the block level to learn global dependencies between pixels. RCAN [27] updates the weights according to the importance of different channels, strengthens the useful channels while suppressing the useless ones, and improves the utilization of computational resources. CRAN [28], proposed by Zhang et al., which utilizes contextual reasoning based on the global context, introduces channel and spatial interaction to generate attention masks to enhance the reconstruction performance of the network. To improve the traditional disadvantage of Transformer regarding ignoring local detail localization due to its global focus, TraFuse [29] employs a sequential stacked coding structure of CNN and Transformer and uses simple progressive upsampling in the encoders of the branches to increase the reconstruction performance of the network.

2.4. N–Gram Language Model

N-gram is a statistical language model (LM) [30] based on the assumption that the occurrence of the n-th word is related to the previous $n-1$ words and unrelated to any other words. The probability of an entire sentence is the product of the probabilities of each word, which can be calculated through statistical analysis of the corpus. The first characteristic of the N-gram model is that the occurrence of a word depends on several preceding words. The second characteristic is that the more information we have, the more accurate the prediction becomes.

Assume a sentence S consists of a sequence of words $w_1,w_2,w_3,\dots ,w_n$. The N-gram language model [31] can be expressed with the following formula (each word $w_i$ depends on the influence of the words from the first word $w_1$ to the preceding word $w_{i-1}$):

$$\begin{array}{cc}\hfill P\left(S\right)& =P(w_1,w_2,w_3,\dots ,w_n)\hfill \\ \hfill & =P\left(w_1\right)P\left(w_2\right|w_1)P\left(w_3\right|w_1,w_2)\dots P\left(w_n\right|w_1,w_2,w_3,\dots ,w_{n-1})\hfill \end{array}$$

(1)

Initially, N-gram language models were only used for text analysis tasks such as spell checking, grammar checking, and text generation. However, due to the similarity of the N-gram concept in images to that in text, in recent years, some researchers have begun to utilize N-gram language models to various computer vision tasks. Choi et al. [32] proposed the NGSwin network, which was the first to combine N-gram with super-resolution reconstruction tasks. They defined Uni–Gram as a local window within Swin Transformer, where pixels interact through self-attention, overcoming the limitations of the Swin Transformer that arise from its emphasis on global information.

3. Method

Our proposed approach primarily involves compressive sensing classification of input remote sensing images, followed by deep texture extraction on the classified images to enhance the representation capability of the super-resolved images. In Section 3.1, we present a comprehensive description of our proposed PCFI model structure. In Section 3.2, we provide a detailed explanation of ICPC, which involves classifying sub-image blocks according to predetermined ranges. Smooth blocks primarily contain background and contour information, while edge blocks are more complex, containing detailed information but may suffer from information blur and partial feature loss due to edge blurring. Therefore, in Section 3.3, we introduce the deep texture extraction module, which enhances the representation capability of blurred features caused by edge blur, thereby improving the ability of SR model to express details when dealing with complex remote sensing images.

3.1. Overall Structure

Figure 1 provides a visual representation of the overall network architecture of our proposed PCFI network. Comprising two main modules, namely the ICPC and the DTIF, our PCFI architecture is designed to process single HR images. Upon input, the HR image undergoes cropping to generate several equally sized sub-images. These sub-images are then individually fed into ICPC, where compressive sensing technique [33] is applied, followed by feature classification using compressive sensing: smooth blocks, margin blocks, and texture blocks. The output consists of three classes based on features, detailed in Section 3.2. Subsequently, the classified sub-images are directed into the DTIF for deep content extraction. This module, a hybrid of the U-Net [34] architecture and Swin Transformer architecture, adheres to the fundamental structure of U-Net. The specific composition of the DTIF will be elucidated in Section 3.3.

3.2. Integrated Compressive Sensing-Based Perception Classifier Module

Our integrated compressive sensing-based perception classifier (ICPC) module utilizes compressed sensing for feature classification, which effectively reduces computational complexity compared to the commonly employed deep learning methods, such as CNNs or self-attention mechanisms. Although CNNs demonstrate excellent performance in feature extraction, they often incur substantial computational and memory overhead. For instance, remote sensing images are typically large-scale; when processed using self-attention mechanisms, the model must account for dependencies between every pixel and region, significantly increasing the computational complexity. In contrast, CNNs rely on deep networks for layer-by-layer processing, requiring numerous convolutional kernels and network layers to capture features of varying complexities. This often leads to the processing of considerable redundant information to achieve comprehensive feature classification, which greatly diminishes processing speed and contradicts the ICPC module’s goal of accelerating super-resolution tasks, thereby increasing the computational burden.

Moreover, remote sensing images frequently contain a substantial amount of redundant information, with critical data often concentrated in a few specific regions or frequency bands. Many areas within remote sensing images, such as oceans and deserts, are usually uniform or exhibit extensive smooth textures. The pixel value variations in these regions are minimal and can be represented with fewer non-zero or significant values. By leveraging the spatial sparsity of such areas, the compressed sensing technique can save storage space and reduce processing complexity. When addressing complex textures, these regions typically exhibit sparse representations in certain transformation domains (e.g., wavelet transforms, discrete cosine transforms). CS effectively captures these sparse features, facilitating the identification of different texture categories in classification tasks.

In this section, we start by cropping the image into equally sized blocks and employ the CS theory [35] discussed in this section to map pixel values to perceptual values. Subsequently, the pixel values of each block are multiplied by the measurement matrix to obtain measurement values. These measurement values are then processed to derive features within the measurement value range, allowing for the classification of image blocks based on features within the perceptual domain [36]. This module categorizes image blocks into three classes: smooth, margin, and texture blocks. The specific structure is illustrated in Figure 2.

3.2.1. Definition of Correlation Between Signals in Two Domains

Since the mutual covariance measures the similarity between two signals, we utilize mutual covariance to assess the correlation of frequency domain signals. Specifically, the correlation between two frequency domain image signals is defined as

$${\mathrm{\Gamma }}_F=\mathrm{cov}(\alpha ,\theta )$$

(2)

where ${\mathrm{\Gamma }}_F$ denotes the frequency domain correlation of the signals, cov represents the mutual covariance computation symbol, and $\alpha$ and $\theta$ are the video signals in the DCT domain.

When the value of ${\mathrm{\Gamma }}_F$ is positive, it indicates that the two signals are positively correlated; conversely, a negative value indicates a negative correlation. The larger the absolute value of ${\mathrm{\Gamma }}_F$, the higher the similarity between the two signals, meaning the correlation is stronger. Conversely, a smaller absolute value indicates lower similarity and weaker correlation. In image analysis, we typically start from the pixel domain. However, directly analyzing in the pixel domain can be challenging due to the large volume of data and the lack of intuitive insights, making it difficult to effectively extract useful information. Therefore, we project the image data into the frequency domain for analysis, where frequency domain signals can more clearly reflect the pixel features of the image. According to the CS theory [37] presented in this section, there exists a linear relationship between the perceptual domain and the frequency domain. This means that the frequency domain can be regarded as an effective representation of the pixel domain, while the perceptual domain can reflect the detail level and complexity of the pixel domain images. Thus, by analyzing the signals in the perceptual domain, we can achieve a certain degree of extraction and analysis of image pixel features.

3.2.2. Correlation of Linear Relationships Between Perceptual Domain and Frequency Domain

For each block $S_{z1}$, $S_{z2}$, $S_{z3}$, $S_{z4}$, corresponding to frequency domain signals ${\alpha }_1$, ${\alpha }_2$, ${\alpha }_3$, ${\alpha }_4$, the independent compressed sensing measurements [38] are conducted utilizing an identical sensing matrix. This process yields perception values $z_1$, $z_2$, $z_3$, $z_4$, which are directly concatenated as row vectors to construct the perceptual domain $z={[z_1,z_2,z_3,z_4]}^T$. Given $z_i={[z_{i1},z_{i2},\dots ,z_{in}]}^T$ for $i=1,2,\dots ,n$, the representation of the feature image in the perceptual domain is denoted by the following complex formula:

$$z=\left[\begin{array}{cccc}{z}_{11}& z_{12}& \cdots & z_{1n}\\ z_{21}& z_{22}& \cdots & z_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ z_{41}& z_{42}& \cdots & z_{4n}\end{array}\right]$$

(3)

In this formulation, each $z_i$ is represented as a column vector, where $z_{ij}$ signifies the j-th perception value of the i-th block.

Covariance Matrix Vectorization. If $X_1,X_2,\dots ,X_n$ constitute a collection of random variables forming a random vector $X={[X_1,X_2,\dots ,X_n]}^T$, and each random variable has m samples, then there exists a sample matrix, as follows:

$$O={[{\gamma }_1,{\gamma }_2,\dots ,{\gamma }_n]}^T=[{\zeta }_1,{\zeta }_2,\dots ,{\zeta }_m]$$

(4)

where ${\gamma }_i$ (for $i=1,\dots ,n$) corresponds to the first i vectors of the sample values of the first random variable, while ${\zeta }_j$ (for $j=1,\dots ,m$) represents each random vector K within the sample vector. Consequently, the expression for the mutual covariance between random variables $X_i$ and $X_j$ is given by

$$M_{ij}=E\left[X_i{X}_j\right]-E\left(X_i\right)E\left(X_j\right)$$

(5)

Covariance estimates can be derived from sample values, as follows:

$$M_{ij}={\displaystyle \frac{1}{m}}{\gamma }_i^T{\gamma }_j-{\displaystyle \frac{1}{m^2}}\sum _{a=1}^m{O}_{ia}\sum _{b=1}^m{O}_{jb}$$

(6)

Perceptual Domain Covariance Matrix. We conduct measurements on each image block using the same randomly generated measurement matrix. The process follows a precise measurement procedure:

$${\mathbf{z}}_i=\sigma {\mathbf{x}}_i(i=1,2,\cdots ,N)$$

(7)

where $\mathbf{x}$ represents the original signal of size $n\times 1$ and $\mathbf{z}$ denotes the corresponding signal in the perceptual domain of size $m\times 1$. This procedure operates under the assumption that the random measurement matrix $\sigma$ adheres to a Gaussian distribution with a mean of zero and a variance of $1/m$. Each signal in the perceptual domain $\mathbf{z}$ is associated with a signal in the frequency domain $\mathit{\alpha }$, as represented by

$$\mathbf{x}=\mathbf{D}\mathit{\alpha },\mathbf{z}=\sigma \mathbf{D}\mathit{\alpha }=\mathrm{\Gamma }\mathit{\alpha }$$

(8)

where $\mathrm{\Gamma }=\sigma \mathbf{D}$ represents the projection matrix and $\mathbf{D}$ signifies the sparse basis Discrete Cosine Transform (DCT) matrix. The elements of the projection matrix $\mathrm{\Gamma }$ are derived as follows:

$${\mathrm{\Gamma }}_{ij}=\sum {\sigma }_{ik}{\mathbf{D}}_{kj}=1,2,\dots ,m;j=1,2,\dots ,n)$$

(9)

The variance of the elements of the projection matrix $\mathrm{\Gamma }$ are determined as

$$\begin{array}{cc}\hfill & \mathbb{D}\left[{\mathrm{\Gamma }}_{ij}\right]=\mathbb{D}[\sum {\sigma }_{i}k{\mathbf{D}}_{kj}]=\left(\sum {\mathbf{D}}_{kj}^T\right)\left(\sum {\mathbf{D}}_{21}\right)=1/m\hfill \end{array}$$

(10)

The frequency domain signals corresponding to N image blocks form the frequency domain sample matrix:

$${\mathit{\alpha }}_{N\times n}={[{\mathit{\alpha }}_1,{\mathit{\alpha }}_2,\dots ,{\mathit{\alpha }}_N]}^T$$

(11)

Assuming each signal in the perceptual domain is a random variable, the sample matrix over the perceptual domain is obtained as follows:

$$\mathbf{z}=[{\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_N]=[{\mathbf{Z}}_1,{\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_m]$$

(12)

where ${\mathbf{z}}_i$ represents the sample vector of the i-th random variable, and ${\mathbf{Z}}_j$$(j=1,\dots ,m)$ represents the sample vector of each random vector.

Let ${\mathrm{\Gamma }}_i$$(i=1,2,\dots ,m)$ denote the row vector of the projection matrix $\mathrm{\Gamma }$. Utilizing the vector form of the covariance matrix, we have

$${\mathbf{M}}_z={\displaystyle \frac{1}{m}}\left([{\mathbf{Z}}_1,{\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_m]{[{\mathbf{Z}}_1,{\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_m]}^T-{\mathbf{Z}}_o{\mathbf{Z}}_o^T\right)$$

(13)

$${\mathbf{M}}_z\approx {\displaystyle \frac{1}{m}}{\left(\mathrm{\Gamma }\mathit{\alpha }\right)}^T\left(\mathrm{\Gamma }\mathit{\alpha }\right)$$

(14)

Finally, the covariance matrix in the perceptual domain can be approximated as

$${\mathbf{M}}_z\approx {\displaystyle \frac{1}{m}}\left(\mathit{\alpha }{\mathrm{\Gamma }}^T\right)\left(\mathrm{\Gamma }{\mathit{\alpha }}^T\right)\approx {\displaystyle \frac{1}{m}}\mathit{\alpha }{\mathit{\alpha }}^T$$

(15)

Frequency Domain Covariance Matrix. The frequency domain sample matrix is represented as

$${\mathit{\sigma }}_{N\times n}={[{\mathit{\sigma }}_1,{\mathit{\sigma }}_2,\dots ,{\mathit{\sigma }}_N]}^T=[{\mathit{\xi }}_1,{\mathit{\xi }}_2,\dots ,{\mathit{\xi }}_n]$$

(16)

where ${\mathit{\sigma }}_i$ ($i=1,2,\dots ,N$) denotes the vector containing all sample values for the ith random variable, and ${\mathit{\xi }}_j$ ($j=1,2,\dots ,m$) corresponds to the sample vector of each random vector. Then, the i-th row of the covariance matrix and j-th column elements depict the mutual covariance between signals ${\mathit{\sigma }}_i$ and ${\mathit{\sigma }}_j$:

$${\mathbf{M}}_{\mathit{\sigma }}(i,j)=\mathbb{E}\left[{\mathit{\sigma }}_i{\mathit{\sigma }}_j\right]-\mathbb{E}\left[{\mathit{\sigma }}_i\right]\mathbb{E}\left[{\mathit{\sigma }}_j\right]$$

(17)

All Discrete Cosine Transform (DCT) coefficients within an image block adhere to a Gaussian distribution with a mean of zero; thus,

$$\mathbb{E}\left[{\mathit{\sigma }}_i\right]=\mathbb{E}\left[{\mathit{\sigma }}_j\right]=0$$

(18)

$${\mathbf{M}}_z\approx {\displaystyle \frac{n}{m}}{\mathbf{M}}_{\mathit{\sigma }}$$

(19)

The correlation between the signals in the perception and frequency domains is seen in the equation above. This model shows that the cross-covariance matrix between perception and frequency signals is essentially linear, corresponding to the correlation between the two domains. Clearly, in compression-based image processing, correlation evaluation of frequency domain signals may be directly performed using perceptual domain signals. Therefore, based on the content of this section, in compression-based image processing, the correlation analysis of frequency domain signals can be directly performed using signals from the perceptual domain.

3.2.3. Image Patch Classification Based on Perceptual Domain Features

The elements of the perceptual domain covariance matrix are correspondingly related to the variances of the respective frequency domain signals, emphasizing the close relationship between frequency domain signals and pixel domain signals. Specifically, smooth image patches correspond to frequency domain signals with lower sparsity, indicating a prevalence of uniform information across frequencies. Conversely, edge and texture patches are associated with frequency domain signals that exhibit higher sparsity, reflecting the presence of distinct high-frequency components that characterize sharp transitions and intricate textures in the image. This relationship underscores the importance of understanding the interplay between spatial characteristics and their spectral representations in image processing tasks. Consequently, the variance of the perceptual domain covariance matrix $\mathrm{var}\left({\mathbf{M}}_z\right)$ reflects the characteristics of image patches and can be utilized for their classification. The criteria for classification are defined as follows:

$$\mathrm{if}\left\{\begin{array}{cc}\mathrm{var}\left({\mathbf{M}}_z\right)<T_1,\hfill & \mathrm{Smooth}\mathrm{Block}\hfill \\ T_1\le \mathrm{var}\left({\mathbf{M}}_z\right)\le T_2,\hfill & \mathrm{Margin}\mathrm{Block}\hfill \\ \mathrm{var}\left({\mathbf{M}}_z\right)>T_2,\hfill & \mathrm{Texture}\mathrm{Block}\hfill \end{array}\right.$$

(20)

When judging the categories of image blocks, we combine multi-directional block classification as follows:

(1) Let ${\overline{z}}_i=\mathsf{\Phi }\{x_i+\lambda F_{i+1}^h\}$ ($i=1,2,\dots ,N$), where $F_{i+1}^h$ refers to the feature vector of the block adjacent to the i-th block in the horizontal direction. Based on the covariance matrix of ${\overline{z}}_i$, determine its variance $var\left(M_z\right)$ to classify it into one of the image block types and assign a parameter $CL{S}_h$.

(2) Let ${\overline{z}}_j=\mathsf{\Phi }\{x_j+\lambda F_{j+1}^v\}$ ($j=1,2,\dots ,N$), where $F_{j+1}^v$ refers to the feature vector of the block adjacent to the j-th block in the vertical direction. Based on the covariance matrix of ${\overline{z}}_j$, determine its variance $var\left(M_z\right)$ to classify it into one of the image block types and assign a parameter $CL{S}_v$.

(3) For the same image sub-block, consider the category parameters $CL{S}_h$ and $CL{S}_v$ from both horizontal and vertical directions. The priority order, from high to low, is edge block, texture block, and smooth block; choose the category with the highest priority as the final category of the image block.

The algorithm simultaneously takes into account the global statistical characteristics and local distribution properties of image segments. By employing the 2$\sigma$ principle, it effectively computes the threshold. Smooth blocks primarily contain background and contour information, exhibiting relatively uniform features. In contrast, edge blocks are more complex, encompassing detailed information, such as boundaries corresponding to high-frequency components in the image. Consequently, edge blocks demonstrate higher sparsity in the frequency domain compared to smooth blocks.

3.3. Depth–Texture Interaction Fusion Module

The depth–texture interaction fusion module (DTIF) module employs a traditional U-Net encoder-decoder architecture, which effectively extracts and integrates contextual information while flexibly handling features of varying scales, demonstrating strong adaptability. Before inputting the image blocks into DTIF, they have already undergone feature classification via the ICPC module. This preprocessing allows us to dynamically adjust the number of DTIF modules based on the prior classification results. Specifically, for image blocks with different feature types, we can allocate varying numbers of DTIF modules. For example, simple and smooth regions may be assigned three DTIF modules; fast regions with pronounced edge features may utilize six modules; and complex texture regions may be assigned nine modules. This adjustment mechanism ensures that the model operates efficiently under different circumstances and optimizes feature extraction. In the DTIF, the primary task of the encoder is to extract detailed features of the image, while the decoder combines the encoder’s features with its own through skip connections, ensuring the retention of detail information. This design not only enhances the quality of reconstruction but also effectively reduces information loss, resulting in a clearer and more realistic final output image. By combining the advantages of both the encoder and decoder, the DTIF module is better equipped to adapt to various image features, thereby improving the performance of super-resolution reconstruction. The basic architecture of DTIF consists of a layered encoder, pooling layers, a compression layer, and a decoder, along with skip connection layers linking the encoder and decoder. This section will begin by presenting a comprehensive outline of the overall structure of DTIF. It will then proceed to offer in-depth explanations of the DTIF, CWIA, and MSFE.

Encoder. For a given LR image $X\in {\mathbb{R}}^{H\times W\times 3}$, the extracted $Y_i\in {\mathbb{R}}^{H\times W\times C}$ is obtained through pixel shuffle, where H, W, and C represent the height, width, and channels, respectively. To enrich the transferred feature details and enhance the expression of content features, we design a three-layer encoder. Specifically, the extracted $Y_i$ first undergoes two stages composed of two DTIT (DTI Transformer) blocks and soft pooling [39] with a kernel size of 2 × 2, followed by processing through a third DTIT block. The details of the DTIT will be discussed in Section 3.3.1.

Pooling Layer. In contrast to non-deep learning methods such as average pooling [40] and max pooling [41], we utilize soft pooling [39] in this process, as shown in Figure 3. The calculation formula for soft pooling is as follows:

$$SoftPooling\left(\gamma \right)=\sum _{i=1}^N{\omega }_i{\gamma }_i$$

(21)

where x is the input data, N is the number of input data(${\gamma }_i$), and ${\omega }_i$ is the weight corresponding to the input data ${\gamma }_i$, usually calculated using the softmax function:

$${\omega }_i={\displaystyle \frac{e^{{\gamma }_i}}{{\sum }_{j=1}^N{e}^{{\gamma }_j}}}$$

(22)

The objective of this pooling approach is to minimize the information loss that occurs during the pooling process, while yet preserving the functionality of the pooling layer. By retaining information content features effectively, soft pooling facilitates deeper extraction of blurred information, thus significantly enhancing the performance of traditional SR methods in handling images with varying levels of clarity.

Compression Layer. The design of a three-layer encoder and the choice of pooling layers enable the extraction of more image details, but they also convey a significant amount of image information. To address this, the introduction of the compression layer module (MSFE) effectively compresses input image information while extracting features. This means that, while compressing redundant information from simple image blocks, the compression layer also aids in capturing detailed features [42] from complex, blurred image blocks. The design of the compression layer not only reduces computational complexity and alleviates the computational burden on the subsequent decoder but also enhances the representation capability of similar features, effectively minimizing information loss during the feature transfer process to the decoder, thereby improving the quality of image reconstruction. The specific details of this module will be elaborated in Section 3.3.4.

Decoder. Similar to most U-Net network architectures, the decoder adopts the same modular structure as the encoder. Here, we also use a combination of DTIT and soft pooling block. Additionally, The final result of the first DTIT is designed to bypass the completely connected layer and instead enter the compression layer module. It is then combined with the output of the three-layer encoder for residual connection [43], before being inputted into the decoder. This skip-connection [44] design aids in optimizing the model by reducing computational complexity while leveraging locality and dependency to perform multiscale feature extraction on the input.

3.3.1. Depth–Texture Interaction Transformer

Due to the fixed-size window mechanism employed by the Swin Transformer for self-attention operations, features near the window boundaries are prone to blurriness or incoherence, particularly in areas with complex texture details, making it difficult to effectively recover degraded edge pixels. This local window division hinders seamless information connectivity between different windows, resulting in suboptimal performance in texture blocks within remote sensing images. Furthermore, although the Swin Transformer facilitates information interaction through its window mechanism, its alternating local operations still impose limitations on information propagation, particularly in terms of long-range dependencies across windows. While the Swin Transformer employs a multi-layer shifted window approach for inter-window information interaction, its ability to capture global information remains constrained. Unlike global self-attention mechanisms, the Swin Transformer cannot capture global information at each layer; instead, it gradually builds global perception through multi-layer accumulation. Consequently, when addressing tasks requiring traversal across multiple local regions, it may not sufficiently model long-range dependencies.

To effectively overcome these challenges, we introduce an N–Gram window interaction module prior to self-attention operations within the module. This N–Gram window interaction module establishes tight feature relationships between multiple adjacent windows, breaking the limitations of the traditional Swin Transformer’s local window operations. It enables features from different windows to interact during the computation process, facilitating more comprehensive information propagation and fusion. Leveraging the concept of N–Gram from natural language processing, the N–Gram window interaction module fosters tighter feature relationships among adjacent windows, achieving efficient inter-window information dissemination and integration. This enhancement not only boosts the model’s information interaction capabilities within local windows but also effectively alleviates issues of edge blurriness and disjointed local features by capturing contextual information across windows. In the depth–texture interaction fusion (DTIT) module, our designed Cross-Window Importance Aggregation (CWIA) block spans the combined areas of W-Trans and SW-Trans following the N–Gram interaction module, connecting with the outputs of the self-attention mechanism through residual connections. This module allows for simultaneous consideration of feature importance in both directions, employing different pooling strategies to extract finer texture information based on directional significance. Through this integrative feature extraction approach, the model can more accurately identify and restore critical structures and details within images, enhancing overall image quality and recognition accuracy. The DTIT module is adept at capturing complex textures that are challenging to process using traditional convolution or window mechanisms, thereby transcending the local limitations of the Swin Transformer and ultimately improving overall image quality.

3.3.2. N–Gram Window Interaction Block

In the Swin Transformer, the original sliding window self-attention (SA) and cross-window self-attention (WSA) are computed as follows:

$$\mathrm{SA}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(23)

where Q, K, and V denote the matrices representing the query, key, and value, respectively. Additionally, $d_k$ refers to the dimension of the key.

WSA cross-window self-attention calculation formula:

$$\mathrm{WSA}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q(K^T+A)}{\sqrt{d_k}}}\right)V$$

(24)

where A is the window shift matrix, which is used to address the limited receptive field issue caused by window shifting.

To address the window shifting issue in Swin-V1, we adopt the concept proposed by Choi et al. [32], which utilizes a Uni–Gram non-overlapping local window based on the N–Gram language model, as shown in Figure 4. The accurate formula is as follows:

$$\mathrm{Uni}-\mathrm{Gram}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q(K^T\odot M)}{\sqrt{d_k}}}\right)V$$

(25)

where ⊙ denotes element-wise multiplication, and M is the Uni–Gram window mask matrix used to define the window range.

Each adjacent Uni–Gram window can be combined into a larger N–Gram window by concatenating the query, key, and value matrices of multiple Uni–Gram windows. In the N–Gram language model, consecutive forward, backward, or bidirectional words are considered as target words. Using N–Gram to define the window for WSA allows pixels within the window to influence each other through WSA, thereby enlarging the receptive field. This expansion enhances the capability of the model by increasing the accuracy of extracting details through a broader contextual understanding.

3.3.3. Cross-Window Importance Aggregation

Given that the Swin Transformer relies heavily on window alternation for information interaction, it tends to perform poorly when dealing with edge-blurred textures. This makes it challenging to precisely restore damaged pixels through content interaction. In order to resolve this problem, we have developed the CWIA block. This module extends across the window-blocked area in both W-Trans and SW-Trans and is linked to the result of self-attention by residual connections. The CWIA block utilizes two-dimensional Local Importance Pooling (LIP) [45] in both the horizontal as well as vertical orientations, using the following calculation:

Let X denotes the input feature map of size $H\times W\times C$, where H is the height, W is the width, and C is the number of channels. For each position $(i,j)$ in the input feature map X, the LIP operation computes the importance score $I_{ij}$ as the local sum of absolute differences (LSAD) [46] between the pixel values within a neighborhood window centered at $(i,j)$.

The importance score $I_{ij}$ for position $(i,j)$ is calculated as

$$I_{ij}=\sum _{p=-k}^k\sum _{q=-k}^k|X_{(i+p)(j+q)}-X_{ij}|$$

(26)

where k is the radius of the neighborhood window, and $X_{(i+p)(j+q)}$ represents the pixel value at position $(i+p,j+q)$ in the input feature map X.

The two outputs after pooling are multiplied and subjected to self-attention. The result is then added to the output after passing through W-Trans and SW-Trans directly via an adder. The inclusion of this module enhances the representational capacity of details, allowing for deep extraction of texture block details to improve the accuracy of pixel recovery.

3.3.4. Multi-Scale Feature Enhancer

The Multi-Scale Feature Enhancer (MSFE) block, serving as the compression layer of the network, consists of three steps: transpose interpolation, concatenation, and grouped convolution. First, the output $y_s$ from the third layer of the encoder is fed into the MSFE block, where it undergoes upsampling through the transpose interpolation layer, resulting in the expanded feature map $y_s^{\prime }$ with an increased resolution of $I^{\prime }$. Unlike traditional interpolation methods, transpose interpolation better integrates the feature representations within convolutional neural networks, demonstrating significant advantages in preserving spatial information and detail features. Additionally, transpose interpolation smooths the feature map during expansion, avoiding artifacts such as the checkerboard effect, while requiring no additional training weights. This process allows for simple parameter control over the output image resolution, effectively alleviating computational burdens and enhancing the overall quality and efficiency of the upsampling. Subsequently, the MSFE block concatenates the results across channels in the channel domain to amplify and reinforce identical features. The combined features are then processed through the grouped convolution layer. Grouped convolution operates by partitioning channels into groups, performing $3\times 3$ convolution operations within each group. This mechanism enables the network to learn different feature subspaces, resulting in richer feature representations upon final aggregation, particularly enhancing edge and complex texture block representations when processing intricate remote sensing images. Compared to standard convolution, grouped convolution significantly reduces the number of parameters and computational load, thereby further lowering the demands on computational resources. Through these steps, the MSFE block efficiently extracts multi-scale features while ensuring the capability to handle complex textures. The final output $Y_S$ is then fed into the decoder to complete the subsequent reconstruction tasks.

4. Experiment

4.1. Experiment Setup

Implementation details. PCFI is constructed using the PyTorch 1.10.0 framework, specifically leveraging a single NVIDIA GTX 3090Ti GPU. The network training strategy utilizes the Adaptive Moment Estimation (Adam) [47] algorithm as the approach for optimizing parameters. The Adam hyperparameters are set at ${\beta }_1=0.9$ and ${\beta }_2=0.999$. In addition, we implement cosine annealing as the approach for decaying the learning rate. We initialize the learning rate to $2\times {10}^{-4}$ and perform training for 250,000 iterations. During the training procedure, we utilize a stochastic selection technique to randomly choose 16 LR-HR image pairings from the training set. Subsequently, we perform random cropping to get LR image patches with dimensions of $64\times 64$. In addition, to increase the diversity of the dataset, we enrich each input image by applying data augmentation techniques like flipping the images vertically and horizontally. The training loss function employs the L1 norm, minimizing the standard L1 loss between the predicted image $I_{SR}$ and the high-resolution image $I_{HR}$: $L = \parallel I_{HR}-I_{SR}{\parallel }_1$.

Evaluation metrics. In conducting quantitative comparisons, we adopted Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Feature Similarity Index (FSIM), and Spectral Angle Mapper (SAM) as evaluation metrics to assess our model’s performance in the SR task. PSNR, measured in decibels (dB), indicates that a higher value signifies greater similarity between the reconstructed image and the original image. The SSIM index evaluates the structural similarity of images, with higher values also indicating increased similarity between images. Increases in PSNR and SSIM values represent superior performance. The detailed calculation formulas are as follows:

$$\mathrm{PSNR}=10{log}_{10}\left({\displaystyle \frac{R^2}{\mathrm{MSE}}}\right)$$

(27)

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(x_i-y_i)}^2$$

(28)

$$\mathrm{SSIM}(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}}$$

(29)

We employed LPIPS to better assess the quality of the reconstructed image from a perceptual perspective, where a lower score indicates better visual quality of the reconstructed image. The detailed calculation formula is as follows:

$$\mathrm{LPIPS}(x,y)=\sum _i{\displaystyle \frac{\left|\right|{\varphi }_i\left(x\right)-{\varphi }_i\left(y\right){\left|\right|}^2}{N}}$$

(30)

FSIM focuses on low-level features of images, such as the perceived quality of edges and textures, by evaluating key visual information through phase congruency and gradient magnitude. A higher score indicates better recovery of edge and texture details in the image. The calculation formula is as follows:

$$\mathrm{FSIM}(x,y)=\mathrm{PC}(x,y)·\mathrm{GM}(x,y)$$

(31)

SAM is used to evaluate the similarity of spectral information by comparing the spectral angles of each pixel in hyperspectral bands, effectively detecting whether spectral information is lost during reconstruction. A lower SAM score indicates a smaller difference between the reconstructed image and the original image in the spectral dimension, reflecting higher fidelity of spectral information. The detailed calculation formula is as follows:

$$\mathrm{SAM}(x,y)=arccos\left({\displaystyle \frac{x·y}{\left|\right|x\left|\right|·\left|\right|y\left|\right|}}\right)$$

(32)

4.2. Datasets

To systematically evaluate the performance of our model, we selected multiple datasets. First, we utilized the AID dataset, a widely used large-scale benchmark dataset for remote sensing image scene classification, comprising 10,000 remote sensing images of fixed size 600 × 600 pixels [48]. The AID dataset is primarily obtained through aerial imaging techniques and contains 30 different scene categories, such as airports, bare land, and baseball fields (as shown in Figure 5). We used the AID dataset as both the training and testing set to assess the model’s performance on standard remote sensing images. To further explore the adaptability of the model under different imaging modalities, we conducted experiments on SAR images and hyperspectral images. For SAR images, we used the SEN1-2 dataset [49], which contains data from the Sentinel-1 and Sentinel-2 satellites. The SAR images provided by Sentinel-1 are single-channel 8-bit images of size 256 × 256 pixels, while the optical images from Sentinel-2 are three-channel 8-bit images of the same size. We selected the “Summer” dataset, which includes registered pairs of SAR and optical images, as the training and testing sets. For hyperspectral images, we employed the AVIRIS dataset [50] developed by NASA. This dataset is specifically designed for capturing hyperspectral images of the Earth’s surface, with each image typically being 256 × 256 pixels, where pixel values represent reflectance with a range of [0, 1], reflecting the surface’s radiation reflection characteristics. The AVIRIS dataset was also used as the training and testing set to validate the model’s performance in processing hyperspectral images. Finally, to comprehensively demonstrate the model’s performance, we conducted tests on natural images. We selected five commonly used standard benchmark natural image datasets—Set5 [51], Set14 [52], BSD100 [53], Urban100 [54], and Manga109 [55]—as the testing set to evaluate the model’s performance on natural images.

4.3. Performance on Task

Quantitative evaluations. To better evaluate the performance of the PCFI network, we compared it with current efficient super-resolution (SR) models, such as Bicubic, EDSR [10], NLSN [56], and HAT-L [57], across 30 different scene categories in the AID dataset. As shown in

Table 1. Comparison on AID datasets with Scale 3. We tested different models on the 30 different scenes in the AID dataset and marked the best-performing ones in red and the second-best ones in blue.

Classes	Bicucbic		EDSR [10]		NLSN [56]		HAT-L [57]		PCFI(Ours)
	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Airport	27.83	0.7554	29.93	0.8282	30.16	0.8322	30.15	0.8319	30.18	0.8325
Bare Land	35.60	0.8564	36.94	0.8837	37.00	0.8845	36.88	0.8841	37.12	0.8849
Baseball Field	31.00	0.8305	33.05	0.8765	33.24	0.8787	33.25	0.8789	33.28	0.8792
Beach	32.90	0.8446	34.18	0.8727	34.31	0.8749	34.24	0.8756	34.43	0.8758
Bridge	30.22	0.8283	32.93	0.8800	33.12	0.8818	33.04	0.8809	33.16	0.8823
Center	26.51	0.6944	28.77	0.7921	28.95	0.7971	28.92	0.7956	28.98	0.7974
Church	24.29	0.6333	26.30	0.7469	26.51	0.7528	26.56	0.7532	26.63	0.7536
Commercial	27.33	0.7174	29.01	0.7940	29.21	0.7996	29.21	0.8007	29.24	0.8011
Dense Residential	22.93	0.5671	24.38	0.6839	24.60	0.6936	24.67	0.6936	24.69	0.6939
Desert	39.26	0.9100	40.20	0.9268	40.27	0.9278	40.37	0.9278	40.42	0.9280
Farmland	33.10	0.8226	35.00	0.8683	35.10	0.8699	35.03	0.8691	35.15	0.8702
Forest	28.79	0.6605	29.85	0.7315	29.98	0.7369	30.10	0.7363	30.13	0.7371
Industrial	26.77	0.6952	28.88	0.7931	29.04	0.7982	29.04	0.7980	29.07	0.7986
Meadow	33.86	0.7483	34.63	0.7804	34.69	0.7821	34.70	0.7815	34.73	0.7825
Medium Residential	26.36	0.6335	28.34	0.7365	28.49	0.7418	28.46	0.7408	28.52	0.7422
Mountain	29.51	0.7349	30.63	0.7885	30.74	0.7916	30.78	0.7923	30.81	0.7925
Park	29.06	0.7530	30.54	0.8130	30.71	0.8177	30.71	0.8189	30.74	0.8191
Parking	24.24	0.7060	27.25	0.8317	27.57	0.8408	27.56	0.8405	27.59	0.8412
Playground	32.64	0.8450	35.37	0.8943	35.58	0.8967	35.49	0.8959	35.60	0.8969
Pond	30.70	0.8167	32.11	0.8542	32.22	0.8559	32.18	0.8555	32.25	0.8563
Port	26.67	0.7986	28.50	0.8596	28.71	0.8631	28.81	0.8638	28.84	0.8642
Railway Station	26.78	0.6793	28.72	0.7738	28.89	0.7783	28.88	0.7780	28.91	0.7786
Resort	26.79	0.7029	28.52	0.7799	28.68	0.7845	28.71	0.7849	28.73	0.7853
River	30.37	0.7402	31.55	0.7891	31.64	0.7914	31.63	0.7909	31.68	0.7917
School	27.41	0.7237	29.36	0.8044	29.55	0.8097	29.54	0.8104	29.59	0.8108
Sparse Residential	26.66	0.6006	27.71	0.6728	27.84	0.6767	27.88	0.6759	27.91	0.6769
Square	28.55	0.7391	30.84	0.8200	31.04	0.8244	31.00	0.8251	31.06	0.8257
Stadium	27.16	0.7547	29.63	0.8387	29.79	0.8422	29.77	0.8422	29.83	0.8424
Storage Tanks	25.65	0.6793	27.44	0.7664	27.61	0.7709	27.60	0.7698	27.65	0.7714
Viaduct	26.97	0.6755	28.99	0.7757	29.17	0.7813	29.11	0.7794	29.21	0.7818
Average	28.86	0.7382	30.65	0.8086	30.81	0.8126	30.81	0.8124	30.87	0.8131

, PCFI significantly outperforms other models in terms of SSIM and PSNR scores, demonstrating exceptional performance. However, since SSIM and PSNR primarily assess image quality from a data perspective, they do not comprehensively reflect human visual perception. Therefore, we introduced LPIPS and FSIM as additional metrics for the quality assessment of standard remote sensing images, to better illustrate the model’s capabilities in visual effectiveness. As presented in

Table 2. LPIPS and FSIM comparison of different networks on AID dataset. We mark the best-performing ones in red and the second-best ones in blue.

	Bicubic	EDSR [10]	NLSN [56]	HAT-L [57]	TransENet [58]	HAN [59]	PCFI
LPIPS	0.4802	0.3069	0.3041	0.3076	0.3133	0.3077	0.3038
FSIM	0.7572	0.8108	0.8232	0.8214	0.8279	0.8267	0.8285

, PCFI outperformed other advanced models, including Bicubic, EDSR [10], NLSN [56], HAT-L [57], TransENet [58], and HAN [59], in both LPIPS and FSIM scores. This indicates that PCFI excels in the recovery of edge and texture details, especially when dealing with edge regions and complex textures. Through the PCFI approach, we effectively enhanced the local modeling capabilities of the Transformer model, facilitating efficient interactions between windows and improving performance in super-resolution reconstruction tasks, particularly in restoring complex features such as object edges and textures in standard remote sensing images.

To assess the model’s complexity, we conducted a computation time comparison on 100 random images at a scaling factor of 4. As shown in Figure 6, the scatter plot compares the number of parameters and FLOPs of six models, including PCFI. The bar chart displays the PSNR and SSIM comparisons for each model at a 2× upscaling factor on the AID dataset. The experimental results indicate that our model achieves the highest PSNR and SSIM scores at a 3× upscaling factor while maintaining a balance between the number of parameters and FLOPs. As shown in

Table 3. Running time comparison of different networks.

Networks	EDSR [10]	NLSN [56]	HAT-L [57]	TransENet [58]	GRL [60]	PCFI
Running Time (ms)	132.62	186.18	284.16	152.67	213.43	178.12

, our model outperforms other models in terms of image reconstruction quality while also demonstrating excellent processing speed. Compared to other equally outstanding models, such as EDSR [10], NLSN [56], HAT-L [57] and TransENet [58], as well as GRL [60], our model exhibits faster processing speeds. Specifically, relative to the second-best performing model on the AID dataset, our computation time is reduced by approximately 4.33%. This is particularly important when processing large-scale datasets such as remote sensing images, as it significantly decreases computational overhead and enhances processing efficiency.

In addition, we compared PCFI with EDSR [10], RRDB [61], SNGAN [62], SRGAN [21], and HSENet [63] on the SAR images from the SEN1-2 dataset on ×2 and ×4 scale. As shown in

Table 4. PSNR, SSIM, and FSIM comparison of different networks on SEN1-2 dataset. We mark the best-performing ones in red and the second-best ones in blue.

Scale		EDSR [10]	RRDB [61]	SNGAN [62]	SRGAN [21]	HSENet [63]	PCFI
	PSNR	41.19	41.27	41.86	41.74	42.23	42.91
×2	SSIM	0.981	0.986	0.987	0.986	0.993	0.996
	FSIM	0.9989	0.9996	0.9994	0.9991	0.9989	0.9993
	PSNR	31.15	31.39	27.81	27.62	24.34	31.85
×4	SSIM	0.838	0.841	0.771	0.763	0.622	0.839
	FSIM	0.9848	0.9836	0.9653	0.9657	0.9718	0.9831

, while PCFI still maintains an advantage in PSNR scores, it performed slightly worse in SSIM and FSIM. This discrepancy is mainly due to the impact of typical speckle noise (multiplicative noise) present in SAR images, which cannot be easily removed through conventional filtering methods, significantly reducing image contrast and detail representation, thus making it challenging for the model to extract effective features. Additionally, the electromagnetic reflection characteristics of SAR images pose difficulties for methods designed for optical images. Finally, we performed a quantitative comparison of PCFI with Bicubic, RDN [64], ESPCN [65], TransENet, and HSENet on the AVIRIS hyperspectral dataset, using PSNR, SSIM, and SAM as evaluation metrics. As shown in

Table 5. PSNR, SSIM, and SAM comparison of different networks on the AVIRIS dataset. We mark the best-performing ones in red and the second-best ones in blue.

Model	×2			×4			×8
	PSNR	SSIM	SAM	PSNR	SSIM	SAM	PSNR	SSIM	SAM
Bicubic	43.72	0.9733	0.1376	39.19	0.9369	0.9683	36.19	0.9029	2.8346
RDN	45.69	0.9835	0.4558	40.61	0.9532	0.9436	37.27	0.9194	2.2317
ESPCN	41.93	0.9681	1.9268	38.41	0.9246	1.8725	33.73	0.8324	3.0341
TransENet	45.91	0.9843	0.6671	41.43	0.9621	1.0423	38.09	0.9281	2.1462
HSENet	45.84	0.9839	0.4603	40.75	0.9546	0.9873	37.15	0.9175	2.2937
PCFI(Ours)	45.97	0.9851	0.4037	41.65	0.9623	0.9752	38.12	0.9228	2.2731

, although PCFI continues to lead in PSNR and SSIM scores at ×2 and ×4, its performance on SAM scores is relatively poor. Hyperspectral images contain a large number of spectral channels, far exceeding the three channels of standard RGB remote sensing images, which poses challenges for PCFI, a model adept at processing low-dimensional optical images, in capturing hyperspectral information. Additionally, the signal-to-noise ratios of spectral edge channels (such as infrared and ultraviolet bands) are comparatively low, further impacting the overall performance of the model and leading to less accurate recovery of spectral information. Nevertheless, PCFI performs well in PSNR and SSIM scores, indicating its advantages in handling spatial details and pixel value differences, excelling in both spatial dimensions and visual effects.

To further assess the performance of this model, we also compared PCFI with IMDN [66], EDSR [10], HNCT [67], LatticeNet [68], SwinIR [26], ESRT [69], HAT [57], SRFormer [70], and GRL [60]. As shown in

Table 6. Quantitative comparison of super resolution. We train the result of PCFI and other different methods with 5 standard datasets under a scale factor of 2/3/4 and mark the best-performing ones in red and the second-best ones in blue.

Method	Scale	Set5 [51]		Set14 [52]		BSD100 [53]		Urban100 [54]		Manga109 [55]
		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
IMDN [66]		38.00	0.9605	33.63	0.9177	32.19	0.8996	32.17	0.9283	38.88	0.9774
EDSR [10]		38.11	0.9602	33.92	0.9195	32.32	0.9013	32.93	0.9351	39.10	0.9773
HNCT [67]		38.08	0.9608	33.65	0.9182	32.22	0.9001	32.22	0.9294	38.87	0.9774
LatticeNet [68]		38.06	0.9607	33.70	0.9187	32.20	0.8999	32.25	0.9288	38.94	0.9774
SwinIR [26]	×2	38.14	0.9611	33.86	0.9206	32.31	0.9012	32.76	0.9340	39.12	0.9783
ESRT [69]		-	-	-	-	-	-	-	-	-	-
HAT [57]		38.63	0.9630	34.86	0.9274	32.62	0.9053	34.45	0.9466	40.26	0.9809
SRFormer [70]		38.51	0.9627	34.44	0.9253	32.57	0.9046	34.09	0.9449	40.07	0.9802
GRL [60]		38.67	0.9647	35.08	0.9303	32.68	0.9087	35.06	0.9505	40.67	0.9818
PCFI (Ours)		38.69	0.9653	35.14	0.9311	32.72	0.9092	35.12	0.9509	40.71	0.9824
IMDN [66]		34.36	0.9270	30.32	0.8417	29.09	0.8046	28.17	0.8519	33.61	0.9445
EDSR [10]		34.65	0.9280	30.52	0.8462	29.25	0.8093	28.80	0.8653	34.17	0.9476
HNCT [67]		34.47	0.9275	30.44	0.8439	29.15	0.8067	28.28	0.8557	33.81	0.9459
LatticeNet [68]		34.40	0.9272	30.32	0.8416	29.10	0.8049	28.19	0.8513	33.63	0.9442
SwinIR [26]	×3	34.62	0.9289	30.54	0.8463	29.20	0.8082	28.66	0.8624	34.78	0.9478
ESRT [69]		34.42	0.9268	30.43	0.8433	29.15	0.8063	28.46	0.8574	33.95	0.9455
HAT [57]		35.07	0.9329	31.08	0.8555	29.54	0.8167	30.23	0.8896	35.53	0.9552
SRFormer [70]		35.02	0.9323	30.94	0.8540	29.48	0.8156	30.04	0.8865	35.26	0.9543
GRL [60]		-	-	-	-	-	-	-	-	-	-
PCFI (Ours)		35.11	0.9336	31.13	0.8560	29.58	0.8169	30.29	0.8991	35.58	0.9557
IMDN [66]		32.21	0.8948	28.58	0.7811	27.56	0.7353	26.04	0.7838	30.45	0.9075
EDSR [10]		32.46	0.8968	28.80	0.7876	27.71	0.7420	26.64	0.8033	30.02	0.9148
HNCT [67]		32.31	0.8957	28.71	0.7834	27.63	0.7381	26.20	0.7896	30.70	0.9112
LatticeNet [68]		32.30	0.8943	28.61	0.7812	27.57	0.7355	26.14	0.7844	30.54	0.9075
SwinIR [26]	×4	32.92	0.9044	29.09	0.7950	27.92	0.7489	27.45	0.8254	32.03	0.9260
ESRT [69]		32.19	0.8947	28.69	0.7833	27.69	0.7379	26.39	0.7962	30.75	0.9100
HAT [57]		33.04	0.9056	29.23	0.7973	28.00	0.7517	27.97	0.8368	32.48	0.9292
SRFormer [70]		32.93	0.9041	29.08	0.7953	27.94	0.7502	27.68	0.8311	32.21	0.9271
GRL [60]		33.10	0.9094	29.37	0.8058	28.01	0.7611	28.53	0.8504	32.77	0.9325
PCFI (Ours)		33.19	0.9098	29.41	0.8069	28.06	0.7628	28.68	0.8508	32.83	0.9337

, we quantitatively evaluated multiple SR models at scaling factors of 2×, 3×, and 4× across five benchmark datasets. Our network outperformed other models in both SSIM and PSNR scores, demonstrating exceptional performance. This indicates that our network not only excels in enhancing the resolution of remote sensing images but also shows significant advantages in improving the quality of natural images.

Visual comparison. As shown in Figure 7, we selected representative images from five different scenes in the AID dataset: airport, park, sparse residential area, and viaduct. To evaluate the effects of super-resolution reconstruction, we compared these images with the results from traditional methods such as Bicubic, EDSR [10], NLSN [56], and HAT-L [57], while also referencing high-resolution (HR) images. The comparison clearly indicates that images reconstructed using the PCFI network significantly outperform other models in detail preservation, particularly in complex edge and texture regions, which often exhibit blurring or artifacts. Furthermore, as illustrated in Figure 8, we selected two typical “Summer” images from the SAR image dataset SEN1-2 and performed super-resolution processing using EDSR [10], RRDB [61], SNGAN [62], SRGAN [21], HSENet [63], and our model, followed by a detailed comparison of the results. Additionally, we selected two images from the hyperspectral dataset AVIRIS and compared the outputs from Bicubic, RDN [64], ESPCN [65], TransENet [58], HSENet [63], and our model. These comparative analyses demonstrate that, despite the differing imaging modalities of remote sensing images, our model still excels in human visual perception, effectively preserving details and resulting in more realistic images. Finally, to validate the effectiveness of our model in processing natural images, we selected a representative image from each of the five benchmark datasets: Manga109 [55], BSD100 [53], Set5 [51], Set14 [52], and Urban100 [54]. We obtained outputs using Bicubic, SwinIR [26], HAT [57], and GRL [60], and we then compared these outputs with the images processed by PCFI. As shown in Figure 9, our PCFI outperforms other state-of-the-art methods in natural images effectively and significantly. This result verifies that the introduction of DTIF substantially enhances the effectiveness of the super-resolution model, greatly improving its reconstruction capabilities.

4.4. Ablation Studies

The PCFI model comprises two main modules, the Integrated Compressed Perception Classifier (ICPC) and the Deep Texture Interaction Module (DTIF), which includes two key blocks: CWIA and NWIB. In this section, we provide a detailed description of the ablation study conducted to validate the effectiveness and necessity of each module within the PCFI network. We selected the AID dataset as the experimental basis, as it contains a large number of high-resolution natural images that can adequately assess the performance of our model. We established a baseline model, denoted as ${\mathrm{Model}}_L$, which excludes the ICPC, CWIA, and NWIB modules, while the complete model is referred to as ${\mathrm{Model}}_O$. From ${\mathrm{Model}}_O$, we sequentially removed the ICPC module, CWIA block, and NWIB block, resulting in the following additional models: ${\mathrm{Model}}_A$, ${\mathrm{Model}}_B$, and ${\mathrm{Model}}_C$. We conducted experiments on these models to measure their output PSNR and SSIM after processing 100 random images, using the comparison of PSNR and SSIM scores to evaluate the quality of the reconstructed images. Furthermore, to demonstrate the impact of the ICPC module, we also present the processing times for each model.

Integrated Compressive Sensing-based Perception Classifier Module. The ICPC module performs compressed sensing sampling on sub-images cropped based on perceptual domains, followed by feature classification of these sub-images. In our ablation study, we first removed the ICPC module, resulting in a new model ${\mathrm{Model}}_A$, to assess its impact on the performance of the PCFI network. As shown in

Table 7. Comparison of PSNR, SSIM scores, and running times of PCFI with removed modules on the AID dataset at a ×3 scale. We rename the PCFI after removing certain modules. (×: Indicates that the module is not used. ✓: Indicates that the module is used.)

Method	Running Time (ms)	ICPC	DTIF		AID
			CWIA	NWIB	PSNR	SSIM
ModelL	129.18	×	×	×	30.28	0.8078
ModelA	195.53	×	✓	✓	30.71	0.8119
ModelB	135.58	✓	×	✓	30.41	0.8092
ModelC	153.24	✓	✓	×	30.62	0.8104
ModelO	178.12	✓	✓	✓	30.83	0.8135

, compared to the full model, ${\mathrm{Model}}_A$ exhibited a significant performance drop, with PSNR decreasing by 0.12 dB and SSIM decreasing by 0.0016. This result clearly indicates that the removal of the ICPC module adversely affects the quality of the reconstructed images and validates that processing sub-image blocks through feature classification before reconstruction can more accurately recover local detail textures, thereby improving the reconstruction accuracy. To further validate the acceleration effect of the ICPC module within PCFI, we also compared the computational speeds of various models investigated in this study. Notably, the runtime of ${\mathrm{Model}}_A$, with the ICPC module removed, significantly increased. We conclude that the inclusion of the ICPC module improves the model’s efficiency by approximately 8.91%. This improvement is attributed to the ability of the model, after compressed sensing classification, to allocate varying numbers of DTIF modules to tasks involving edge, smooth, and texture feature types, as opposed to assigning the same number of modules to all three feature types. This clearly demonstrates that the ICPC module’s use of compressed sensing for feature extraction is highly beneficial for accelerating super-resolution tasks.

Depth–Texture Interaction Module. The N–Gram Window Interaction Block (NWIB) and the CWIA block are essential components of the DTIT, crucial for enhancing the accuracy of the reconstructed images. First, we formed a new model ${\mathrm{Model}}_B$ by removing the CWIA block from ${\mathrm{Model}}_O$ to validate the role of CWIA. As shown in Table 7, the PSNR and SSIM of ${\mathrm{Model}}_O$ improved by approximately 1.38% and 0.53% compared to ${\mathrm{Model}}_B$, respectively, indicating better reconstruction quality. This improvement is attributed to the CWIA block’s capability to capture local features in both vertical and horizontal directions. This bidirectional extraction method enables the model to effectively identify complex textures and edges within the image, thereby enhancing the accuracy and quality of pixel reconstruction. Therefore, the inclusion of CWIA had a significant positive impact on model performance, further consolidating the advantages of DTIT in super-resolution tasks. Next, we formed a new model ${\mathrm{Model}}_C$ by removing the NWIB block from ${\mathrm{Model}}_O$ to assess the impact of NWIB on model performance. As shown in Table 7, compared to ${\mathrm{Model}}_O$, the PSNR and SSIM of ${\mathrm{Model}}_C$ decreased by 0.21 dB and 0.0031 dB, respectively. This result clearly indicates that NWIB plays a significant role in enhancing the similarity between the reconstructed images and the original images. This improvement is primarily due to the introduction of the N–Gram window interaction before the self-attention mechanism. This interaction effectively captures correlations within local regions, allowing the model to better understand the relationships between adjacent elements when processing images. This precise capture of local features not only aids in the restoration of complex textures but also significantly improves detail retention during the reconstruction process, reducing the occurrence of blurriness and artifacts. This further highlights the indispensable role of the NWIB block within the model, providing stronger support for image reconstruction and ensuring that the final results are closer to the original images.

5. Conclusions

In this study, we propose a novel network for remote sensing image super-resolution, called Perception-guided Classification Feature Intensification (PCFI). This network consists of two main modules: integrated compressive sensing-based perception classifier module (ICPC) and depth–texture interaction fusion module (DTIF). The ICPC module employs a perception-guided feature classification strategy, using perception mechanisms to guide targeted feature classification of different regions in the input image, such as edges, textures, and smooth areas. This approach enhances the capture of local feature details, contributing to the acceleration of the super-resolution (SR) task. The DTIF module combines Transformer architectures with a window texture interaction mechanism to extract complex textures from two dimensions, enabling it to capture texture variations and edge information in the image with precision, particularly in high-frequency textures and fine edges, while avoiding information loss or blurriness. Comprehensive experimental results demonstrate that our PCFI model outperforms current state-of-the-art super-resolution reconstruction methods on standard remote sensing image datasets. It not only significantly improves image reconstruction quality, especially in edge regions and complex texture restoration, but also enhances the computational efficiency. This dual improvement in performance and efficiency allows the PCFI to exhibit stronger advantages when processing large-scale datasets such as remote sensing images.