Transform Domain Based GAN with Deep Multi-Scale Features Fusion for Medical Image Super-Resolution

Abstract

High-resolution (HR) medical images provide clearer anatomical details and facilitate early disease diagnosis, yet acquiring HR scans is often limited by imaging conditions, device capabilities, and patient factors. We propose a transform domain deep multiscale feature fusion generative adversarial network (MSFF-GAN) for medical image super-resolution (SR). Considering the advantages of generative adversarial networks (GANs) and convolutional neural networks (CNNs), MSFF-GAN integrates a deep multi-scale convolution network into the GAN generator, which is composed primarily of a series of cascaded multi-scale feature extraction blocks in a coarse-to-fine manner to restore the medical images. Two tailored blocks are designed: a multiscale information distillation (MSID) block that adaptively captures long- and short-path features across scales, and a granular multiscale (GMS) block that expands receptive fields at fine granularity to strengthen multiscale feature extraction with reduced computational cost. Unlike conventional methods that predict HR images directly in the spatial domain, which often yield excessively smoothed outputs with missing textures, we formulate SR as the prediction of coefficients in the non-subsampled shearlet transform (NSST) domain. This transform domain modeling enables better preservation of global anatomical structure and local texture details. The predicted coefficients are inverted to reconstruct HR images, and the transform domain subbands are also fed to the discriminator to enhance its discrimination ability and improve perceptual fidelity. Extensive experiments on medical image datasets demonstrate that MSFF-GAN outperforms state-of-the-art approaches in structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR), while more effectively preserving global anatomy and fine textures. These results validate the effectiveness of combining multiscale feature fusion with transform domain prediction for high-quality medical image super-resolution.

1. Introduction

CT and magnetic resonance imaging (MRI) technologies make non-invasive medical treatment convenient and help doctors make diagnoses. High-resolution (HR) medical imaging reveals finer lesion structures and therefore substantially improves diagnostic accuracy and prognostic assessment. Acquiring HR medical images is challenging because of various factors. In addition to potential technical limitations, clinical constraints related to patient health and limited scan time often impede image acquisition. For instance, patient movement from fatigue and physiological motions like cardiac and respiratory cycles further degrade image quality and lower the signal-to-noise ratio (SNR). Low-resolution images with constrained fields of view and elevated noise can obscure critical pathological features and undermine diagnostic reliability. Consequently, super-resolution (SR) reconstruction methods, which recover fine structural details and enhance clinical utility, have become a major focus of recent research [1,2,3,4].

Super-resolution (SR) is an effective approach for improving image quality and has attracted substantial interest in computer vision. Learning-based SR methods typically perform better when trained with external example pairs. Traditional SR approaches are commonly divided into reconstruction-based methods [5,6] and shallow learning techniques [7,8,9,10,11]. Reconstruction-based SR methods usually require explicit prior information to constrain the reconstruction results [12], leading to a substantial amount of calculation, difficulty in solving, and significant time consumption. Shallow learning SR techniques capture representations that relate Low-Resolution (LR) images to their HR counterparts to guide reconstruction, yet the feature extraction and representational power of these learned models remain constrained. Numerous works report that deep learning (DL) achieves superior results compared with conventional machine learning on problems like image classification [13] and object detection [14]. With advances in computing resources and the availability of massive training data, DL-based SR methods [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34] have markedly enhanced the fidelity of reconstructed high-resolution images. These methods develop a series of network models with excellent reconstruction effects according to the specific SR needs based on CNNs, ResNet [35], GANs [36], or other network structures. A fresh concept for image generation is offered by the GAN model [36], which also offers a model foundation for high-resolution image generation. The first study to use the GAN model for SR reconstruction is SRGAN [30], which has produced images with improved visual quality and more accurate high-frequency (HF) features. However, the relatively straightforward design of the SRGAN generation network results in the extracted features being frequently insufficient, which has an impact on the reconstruction quality. To boost super-resolution performance across scales, researchers have proposed various GAN-based and deep convolutional network techniques [31,32,33,34,35,36,37,38].

Multi-scale features have been widely used in deep learning [24,39,40,41,42], yielding excellent performance. Their inherent ability to extract information at multiple scales provides effective representations for a variety of visual tasks. All of the aforementioned methods also accomplish the image SR task in the spatial domain, but the output is typically too smooth and lacks texture detail. Images in the transform domain can retain the context and texture information of the image at different levels, by contrast, resulting in better SR effects. For medical images, they have more texture and greater correlation than natural images, so higher resolution is required to achieve more accurate matching, detection, and segmentation tasks. For medical image SR, we present a deep multi-scale generative adversarial network (MSFF-GAN) in the transform domain in this paper. The following are the main contributions:

• We designed two types of multi-scale feature extraction blocks (the MSID block and the GMS block). The MSID block can exploit the potential features from medical images by adaptively detecting the long- and short-path features at different scales, and the GMS block can achieve greater multi-scale feature extraction capabilities with less computing load at the granular level by expanding the range of receptive fields for each network layer.
• We conducted experiments on our medical image dataset, which demonstrates that our method can achieve higher PSNR/SSIM values and preserve global topological structure and local texture detail more effectively than existing state-of-the-art methods.

The rest of this paper is structured as follows. In Section 2, we introduce related work, Section 3 details our method, Section 4 presents the experiments, and conclusions are provided in Section 5.

2. Related Work

In this section, we will state the research developments of CNN-based SR, GAN-based SR, and medical image SR methods, respectively.

2.1. CNN-Based SR

Since the initial SRCNN [17] through the most recent feature extraction module-based SR networks, CNN-based models have greatly improved the SR quality [20,21,22,23], and then to the latest attention-based SR learning [29], which is noticeably superior to traditional SR approaches. The performance of SRCNN was hindered by its shallow structure. Kim et al. proposed the deeper VDSR model [19], arguing that increasing network depth leads to improved performance. Later, some extremely complex models were put forth, such as RCAN [21], which performs admirably on the SR problem. Models that incorporate dense connections, such as MemNet [20], have been introduced to further boost SR performance. More efficient CNN-based SR methods, including IDN [23] and MSRN [24], build the network by chaining similar feature-extraction modules, which highlights the capability of each block. Marco et al. [29] propose an AMRSR network, which improves performance by using multiple reference images and attention-based sampling. In order to efficiently finish the image SR, attention-based SR network models have recently been proposed. One example is Du et al. [27], who present a MARP network for image SR, which better bridges the resolution gap between LR and reference images and enhances texture details by using attention and residual pooling. Xie et al. [28] introduce a multi-range attention transformer architecture for image SR that adaptively varies attention ranges to model local details and sparse global context, enriching the diversity and representational power of extracted features.

Multi-scale architectures [24,41,42] consistently deliver strong performance on tasks including semantic segmentation, object detection, and image classification. Li et al. [24] propose a multi-scale residual network for image super-resolution that adaptively captures image features across scales. Wang et al. [41] develop a deep multi-scale network for medical image SR that improves representation of overall topology and local texture in HR medical scans. In order to successfully recover the low-resolution image, Liu et al. [42] develop an integrated position coding to improve multi-scale implicit learning in image SR. In order to obtain abundant attention maps at different granularity levels, Wang et al. [43] propose a multi-scale attention network that corrects large kernel attention with multi-scale and gate schemes. By jointly aggregating global and local information, this network enhances the performance of convolutional SR networks. However, the tremendous computational load brought on by the numerous parameters is a weakness shared by these multi-scale networks. Gao et al. [44] proposed Res2Net to address this issue, which outperforms baseline approaches that are currently on the cutting edge by creating hierarchical residual-like connections within individual residual blocks. In this study, the better Res2Net block will be used, and it will be plugged into our network for SR medical images.

The techniques described above carry out image SR in the spatial domain, but they frequently produce extremely smooth output that omits textural features. On the other hand, transform-domain SR is able to preserve contextual and textural information at multiple levels, leading to improved reconstruction. In light of this, Guo et al. [45] create a deep wavelet super-resolution (DWSR) network that reconstructs HR details by predicting missing wavelet coefficients from LR inputs. Li et al. [46] further introduce a wavelet-based feature enhancement network to ensure accurate capture of both local and global textures and to mitigate distortion of high-frequency components.

2.2. GAN-Based SR

In addition to models built on CNNs, GAN-based models exhibit exceptional performance over a wide range of vision fields. They significantly enhance image visual quality for super-resolution tasks; the HF features of the images are more realistic than those built on CNNs. The first study to apply the GAN model to SR reconstruction is SRGAN [30], which has produced images with improved visual quality and more accurate HF features. Wang et al. [31] propose an improved SR generative adversarial network (ESRGAN). Dou et al. [32] propose PCA-SRGAN, which introduces incremental orthogonal projection discrimination in a PCA subspace for face super-resolution and improves perceptual fidelity. Zhang et al. [33] design RankSRGAN, which uses non-differentiable perception indicators to optimize the generator in terms of performance. Shunta et al. [34] describe a method for training an unpaired SR network with produced pseudo-data pairs using a GAN, which would increase the generalizability of the model. By creating a more intricate yet useful degradation model, Zhang et al. [36] are able to successfully super-resolve both synthetic and real images with a variety of degradations. Angarano et al. [47] present an efficient GAN model for real-time SR that uses a tailored SRGAN architecture and model quantization that is fast and lightweight while maintaining a fairly satisfactory image quality. Xiao et al. [48] address the slow sampling speed and small step size denoising issues in diffusion model-based SR by integrating denoising diffusion models with GANs. This method models each denoising step with a GAN to achieve large-step denoising, enhancing sampling efficiency while maintaining image quality. Aloisi et al. [49] further introduce a Wavelet Diffusion-GAN for image super-resolution, performing denoising in the wavelet domain and coupling conditional diffusion with adversarial training to better preserve high-frequency details while remaining efficient. Park et al. [50] introduce a single-image SR framework aimed at perceptually faithful HR reconstruction, in which an objective estimator determines the best mix of objectives for each image patch and a generator synthesizes SR results that correspond to those targets. Duan et al. [51] propose a local implicit wavelet transformer (LIWT) to improve the reconstruction of high-frequency textures. Shrish et al. [52] present NVS-GAN, which adopts a lightweight GAN design for novel view synthesis to reduce trainable parameters while maintaining synthesis quality.

2.3. Medical Image SR

In the medical field, substantial medical image SR reconstruction techniques and their applications are developed, especially the DL-based ones proposed in recent years. To solve the issue of cross-modal medical image synthesis, Huang et al. [53] suggest an innovative dual convolutional filter learning approach. It builds a closed-loop joint filter learning technique to make better use of the data while using less training data. Chaudhari et al. [1] propose a CNN-based SR method to produce thin-slice knee MR images from thicker input slices. Umehara et al. [2] propose to apply SRCNN to enhance the resolution of chest CT images and achieve higher image restoration quality. To produce a more stable and effective training and enhance the perceived quality of the super-resolved findings, Zhu et al. [3] present a multi-scale GAN taking into account several magnification parameters. Huang et al. [54] address SR and cross-modality medical image synthesis jointly by introducing a weakly supervised joint convolutional sparse coding method. The method achieves significant improvement in semantic interpretability scores. Iwamoto et al. [55] propose a novel unsupervised multimodal priors guided SR for the resolution augmentation of MRI images on the basis of external learning. Pan et al. [56] design a Light-ESRGAN, which integrates a stochastic degradation modeling process and a GAN architecture to significantly enhance cell SR reconstruction. Nimitha et al. [57] propose a multi-slice MRI SR that combines GAN with a pre-trained slice interpolation network to achieve higher quality MRI slices. Liu et al. [58] propose a multi-scale attention-guided progressive aggregation network (MAPANet), aiming to utilize multi-scale and appropriate non-local information to promote SR reconstruction. For multimodal medical image SR, Dharejo et al. [4] suggest a GAN with deep multi-attention modules and learned HF information via WT to produce promising results. Wei et al. [59] introduce a misalignment-robust deep unfolding network (MAR-DUN) to enhance multi-modal MRI SR reconstruction. Ji et al. [60] propose Deform-Mamba, an architecture that integrates local and global features across multiple scales to better capture image information and enhance MRI resolution.

Actually, medical images have more texture and greater correlation than natural images, and thus more accurate feature extraction is required. We will use a new GAN-based network with multi-scale feature extraction in the transform domain for medical image SR in this research.

3. Proposed Method

In this section, Section 3.1 overviews the network architecture, Section 3.2 details the MSFF-GAN structure, Section 3.3 describes the loss functions, and Section 3.4 introduces non-subsampled shearlet transform (NSST) prediction in the generator. Unlike conventional multi-scale or multi-modal SR frameworks that simply integrate existing feature-extraction modules, MSFF-GAN adopts a targeted design that combines two complementary multi-scale feature-extraction blocks, MSID and GMS, with transform-domain reconstruction via NSST. MSID captures both long- and short-path dependencies across scales, while GMS efficiently enlarges the receptive field. By operating in the transform domain, the proposed method mitigates over-smoothing and texture loss, a benefit supported by multi-scale signal representation theory and frequency-domain analysis, and leads to improved preservation of fine structural details.

3.1. Overview

We propose a new GAN-based medical image SR approach with transform prediction, which fundamentally differs from native HR imaging by reconstructing enhanced details from LR inputs rather than capturing them directly from imaging devices. Our method designs a series of cascaded deep multi-scale feature extraction (MSFE) blocks in the GAN generator, which is a mechanism for extracting features from images at multiple scales to capture both global and local information, forming MSFF-GAN, to exploit abundant and potential features from medical images. Figure 1 depicts the pipeline of our network architecture. We first apply the NSST, which is a multiscale and multidirectional image decomposition method combining the advantages of non-subsampled pyramid decomposition for multiscale analysis and shearlet transform for multidirectional representation, on the LR image, obtaining one LF sub-band and a series of HF sub-bands. The LF sub-band preserves the global topology information, while the HF sub-bands collect the structure and texture information. These sub-bands are sent together to the MSFF-GAN generator to predict the transform coefficients of the generated HR image that can be obtained via an inverse transform. After that, the sub-bands of the HR image and the generated sub-bands are together sent to the MSFF-GAN discriminator to distinguish between true and false. The sub-bands as the input of the discriminator can better promote the discrimination ability than the spatial domain image. Our model’s total loss function $L_{Total}^{SR}$ is defined as the weighted sum of all the individual loss functions:

$$L_{Total}^{SR}={\alpha }_1{L}_{Con}^{SR}+{\alpha }_2{L}_{Gen}^{SR}+{\alpha }_3{L}_{TV}^{SR},$$

(1)

where ${\alpha }_i$, i = 1,2,3, is the weighting parameter. $L_{Con}^{SR}$ stands for content loss, which is the most popular optimization target for image SR and the foundation for many cutting-edge methods [18,20,24]. $L_{Gen}^{SR}$ stands for the adversarial loss [30,36] of the generative network, which aims to trick the discriminator network. $L_{TV}^{SR}$ denotes the total variation (TV) loss to restrain the noise and encourage spatially coherent solutions [6,30,61].

3.2. Multi-Scale Features Fusion Generative Adversarial Network (MSFF-GAN) Structure

For the medical image SR task, our aim is to reconstruct the SR image $I^{SR}$ from the input LR image $I^{LR}$. The $I^{LR}$ is the LR version of its HR counterpart $I^{HR}$. Only while training are the HR images accessible. A bicubic operation is used to obtain $I_{LR}$. We describe the LR medical image $I^{LR}$ with a tensor of size W × H and denote the $I^{HR}$,$I^{SR}$ with $rW\times rH$, where $r$ represents the upscaling factor.

Our final objective is to train a generating function G that predicts its corresponding HR counterpart for a given input LR image. In order to do this, we train a generator network as the feed-forward CNN ${\mathrm{G}}_{{\theta }_G}$ parameterized by ${\theta }_G$. Here ${\theta }_G=\left\{w^1,w^2,\dots ,w^p,b^1,b^2,\dots ,b^p\right\}$ stands for the weights and bias of the p convolutional layers and is obtained by optimizing the loss function $L_{Total}^{SR}$ of our model. For the given N training images $I_i^{HR}$, $i=1,\dots ,N$ with corresponding $I_i^{LR}$, we solve:

$${\widehat{\theta }}_G=\arg \underset{{\theta }_G}{\min }{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{L}_{Total}^{SR}\left(G_{{\theta }_G}\left(I_i^{LR}\right),I_i^{HR}\right)}.$$

(2)

which utilizes the ADAM optimizer with an initial learning rate of 0.0001 (halved every 50 epochs) until convergence, determined when the validation set PSNR plateaus with less than 0.1 dB improvement over 20 epochs.

In our study, we explicitly design a perceptual loss $L_{Total}^{SR}$ as a weighted combination of several loss components that model various desirable characteristics of the recovered SR image. The loss function $L_{Total}^{SR}$ is used to reduce the difference between the $I^{LR}$ and $I^{HR}$. More details on each loss function are described in Section 3.3.

In accordance with SRGAN [30], we define the discriminator network $D_{{\theta }_D}$, in which we alternately optimize with $G_{{\theta }_G}$ in order to tackle the adversarial min-max problem:

$$L_1=E_{I^{HR}~P_{data}\left(I^{HR}\right)}\left[\log D_{{\theta }_D}\left(I^{HR}\right)\right],$$

(3)

$$L_2=E_{I^{LR}~PG\left(I^{LR}\right)}\left[\log \left(1-D_{{\theta }_D}\left(G_{{\theta }_G}\left(I^{LR}\right)\right)\right)\right],$$

(4)

$$L=\underset{{\theta }_G}{\min }\underset{{\theta }_D}{\max }(L_1+L_2),$$

(5)

where $P_{\mathrm{data}}(I^{HR})$ denotes the real HR sample distribution, and $\mathrm{P}G(I^{LR})$ denotes the generator distribution [62]. The core idea is to train a generative model G to deceive a differentiable discriminator D, which is optimized to distinguish real images from super-resolved ones. This adversarial setup encourages G to generate outputs that closely resemble real images and are difficult for D to classify.

The core of our generator network G consists of two parts, as illustrated in Figure 1: the shallow feature extraction (SFE) module and the deep feature extraction (DFE) module. Specifically, we use two convolution layers to extract the shallow feature $M_0$ in a coarse manner from the NSST sub-bands $S_{LR}$ of the input medical image $I^{LR}$. Therefore, we can have:

$$M_0=H_{SFE1}\left(H_{SFE2}\left(S_{LR}\right)\right),$$

(6)

where M is the number of HF directional sub-bands from NSST (e.g., M = 4 for scale 1 and M = 8 for scale 2). The shallow feature $S_{LR}\in R^{W\times H\times 64}$ is extracted via two 3 × 3 convolutions with 64 filters each. $H_{SFE1}$ and $H_{SFE2}$ stand for the two layers’ respective convolution operations in the SFE module. After the SFE module, the shallow feature $M_0$ is used for the DFE module that contains a set of cascaded MSFE blocks, i.e., MSID blocks or GMS blocks that will be elaborated in Section 3.2.1 and Section 3.2.2, respectively. Each MSFE block can extract multi-scale features in a fine manner. The output information is then adaptively controlled using a 1 × 1 convolutional layer. We refer to this process as feature fusion, expressed as:

$$M_{GF}=H_{GFF}\left(\left[M_1,M_2,\dots ,M_C\right]\right),$$

(7)

where $\left[M_1,M_2,\dots ,M_C\right]$ is the result of concatenating the feature maps created by MSFE blocks 1, 2, …, D. $H_{GFF}$ is the composite function of 1 × 1 convolutional layer. After feature fusion, global residual learning is used to obtain the feature maps $S_{SR}$, which can be expressed as:

$$S_{SR}=M_{GF}+M_0.$$

(8)

To produce the reconstructed medical image $I^{SR}$ through the super-resolution reconstruction workflow, we use inverse NSST on the sub-bands $S_{SR}$. In the proposed MSFF-GAN generator network, all convolutional layers have 64 filters and are followed by ReLU activation [63].

We train a discriminator network to distinguish real HR images from generated SR samples. Notice that we concatenate the HR sub-bands and the distribution of generated data (i.e., generated sub-bands) and input them to the MSFF-GAN discriminator to distinguish between true and false, which can better promote the discrimination ability than the spatial domain image. Here we use ReLU activation and avoid max-pooling throughout the network. As in the VGG network [63], it has eight convolutional layers and a rising number of filter kernels that increases by a factor of 2 from 64 to 512 kernels. The 512 feature maps, resulting from two fully connected layers and a ReLU activation function, are used to calculate the probability for sample classification.

3.2.1. MSID Block for MSFE

Unlike conventional multi-scale convolutional modules with fixed receptive fields, the proposed MSID block employs an adaptive long short-path fusion to capture dependencies across scales, selectively retaining informative features while suppressing noise. This approach aligns with multi-scale signal decomposition theory, balancing contextual aggregation and texture preservation. As shown in Figure 2, each MSID block comprises two parts exploiting short- and long-path properties. Unlike the IDN model [23], each part contains three bypasses with distinct convolutional kernels, enabling multi-scale detection of both short- and long-path features.

Assuming $M_{d-1}$ and $O_{p1}$ are the first part’s input and output, we have:

$$k_1=\sigma \left(Y_{3\times 3}^2\left(M_{d-1}\right)\right),$$

(9)

$$k_2=\sigma \left(Y_{5\times 5}^3\left(M_{d-1}\right)\right),$$

(10)

$$k_3=\sigma \left(Y_{7\times 7}^4\left(M_{{}_{d-1}}\right)\right),$$

(11)

$$O_{P1}=\sigma \left(Y_{1\times 1}^1[k_1+k_2+k_3]\right),$$

(12)

where $Y_{1\times 1}^1$, $Y_{3\times 3}^2$, $Y_{5\times 5}^3$ and $Y_{7\times 7}^4$, respectively, refer to the 1 × 1, 3 × 3, 5 × 5, and 7 × 7 convolutional layers’ functions in the first part. $\left[\cdot \right]$ denotes the concatenation of different convolutional kernels’ feature maps, and the ReLU function [64] is denoted by $\sigma$. The 64-dimensional feature maps of $O_{P1}$ and the input $M_{d-1}$ are then concatenated in the channel dimension,

$$R=C\left(S\left(O_{P1},64\right),M_{d-1}\right),$$

(13)

where $C$ and $S$ stand for the operations of concatenation and slicing, respectively. For example, for a 4 × 4 × 64 input feature $M_{d-1}$ and 4 × 4 × 64 multi-scale features $O_{P1}$, concatenation yields a 4 × 4 × 128 feature map, from which the slicing operation $S$ extracts the last 64 channels (4 × 4 × 64) and $S$ is used to acquire the 64-dimensional features. The goal is to merge the previous information with the current multi-scale information. It can be thought of as stored short-path knowledge. We then use the remaining 64-dimensional feature maps as the input for the second part, which mostly further extracts long-path information. Let $q_i$ be the intermediate feature maps from Equation (14), which are processed by the second part’s convolutional layers of sizes 1 × 1, 3 × 3, 5 × 5, and 7 × 7,

$$q_1=\sigma \left(Y_{3\times 3}^6\left(O_{P1},64\right)\right),$$

(14)

$$q_2=\sigma \left(Y_{5\times 5}^7\left(O_{P1},64\right)\right),$$

(15)

$$q_3=\sigma \left(Y_{7\times 7}^8\left(O_{P1},64\right)\right),$$

(16)

$$O_{P2}=\sigma \left(Y_{1\times 1}^5\left(\left[q_1+q_2+q_3\right]\right)\right),$$

(17)

where $Y_{1\times 1}^5$, $Y_{3\times 3}^6$, $Y_{5\times 5}^7$, $Y_{7\times 7}^8$, respectively, refer to the 1 × 1, 3 × 3, 5 × 5, and 7 × 7 convolutional layers’ functions in the second part. The final step is to aggregate the input information, short-path information, and long-path information, which can be expressed as follows:

$$M_d=R+O_{P2},$$

(18)

where $M_d$ indicates the output of the MSID block.

3.2.2. GMS Block for MSFE

In this subsection, we present the GMS block [44], a variant of the MSFE block. To capture finer-scale features and enlarge the receptive field of each network layer, it is integrated into the generator as a sequence of cascaded MSFE blocks. The block has improved MSFE performance while retaining a light computational load. In particular, it connects various filter groups in a hierarchical residual-like manner using a group of 3 $\times$ 3 filters with smaller groups of filters, which can be collectively referred to as a Res2Net block.

The GMS block is depicted in Figure 3. The feature maps are uniformly separated into s feature map subsets, indicated by $X_i$ following the 11 convolutions, where $i\in \left\{1,2,\dots ,s\right\}$. Compared to the input feature map, each feature subset ${\mathrm{X}}_i$ has the same spatial extent but 1/s number of channels. Except for $X_1$ each $X_i$ has a matching 3 $\times$ 3 convolution indicated by $K_i(\cdot )$,and the output of $K_i(\cdot )$ is denoted by $F_i$. The feature subset $X_i$ is added with the output of $K_{i-1}(\cdot )$ and then fed into $K_i(\cdot )$. The 3 × 3 convolution for $X_1$ might be skipped to decrease parameters while increasing $s$. So $F_i$ can be expressed as:

$$F_i=\left\{\begin{array}{c}{\mathrm{X}}_i\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{1em}\hspace{1em}\hspace{1em}i=1;\\ {\mathrm{K}}_i\left({\mathrm{X}}_i\right)\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{1em}\hspace{1em}\hspace{1em}i=2;\\ {\mathrm{K}}_i\left({\mathrm{X}}_i+{\mathrm{Y}}_{i-1}\right)\hspace{1em}\hspace{1em}2<i\le s.\end{array}\right.$$

(19)

Note that each 3 $\times$ 3 convolutional operator $K_i(\cdot )$ might be able to obtain feature information from every feature split $\left\{{\mathrm{x}}_j,j\le i\right\}$. Therefore, the output result of a 3 × 3 convolutional operator may have a bigger receptive field than the input feature split $X_j$ each time. The output of the Res2Net block has a varying number and different combinations of receptive field sizes as a result of the combinatorial explosion effect.

Splits are handled on multiple scales in the GMS module, making it possible to extract both global and local information. All splits are concatenated and then passed through a 1 × 1 convolution to better combine information at various scales. Convolutions can be forced to process features more successfully using the split and concatenation approach, and s may also be utilized as a scale dimension control parameter. A larger s may make it possible to learn features with wider receptive field sizes while incurring minimal computational overhead from concatenation.

3.3. Loss Function

Our loss function $L_{Total}^{SR}$, as shown in Formula (1), is essential for the generator network’s effectiveness and, thus, the SR algorithm. We define $L_{Total}^{SR}={\alpha }_1{L}_{Con}^{SR}+{\alpha }_2{L}_{Gen}^{SR}+{\alpha }_3{L}_{TV}^{SR},$ as the weighted sum of the individual loss functions given the weighting factors ${\alpha }_i$, i = 1,2,3, and we specifically identify the following loss components: content loss, adversarial loss, and regularization loss.

The mean square error (MSE) function is the most widely used objective optimization function in image SR [20,22,65]. But Lim et al. [66] have empirically shown that using MSE loss in training is not a wise decision. We employ the mean absolute error (MAE) function loss as a superior substitute for the content loss $L_{Con}^{SR}$ in order to prevent the introduction of pointless training tactics and minimize computations, which can be expressed below:

$$L_{Con}^{SR}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\Vert G_{{\theta }_G}\left(I_i^{LR}\right)-I_i^{HR}\Vert }_1},$$

(20)

We incorporate the generative component into our loss in addition to the content loss. By attempting to trick the discriminator network, this encourages our network to favor solutions that are found on the variety of medical images with relevance. Based on the probabilities of the discriminator $D_{{\theta }_D}\left(G_{{\theta }_G}\left(I_i^{LR}\right)\right)$ across all training samples, the generative loss $L_{Gen}^{SR}$ is defined as follows:

$$L_{Gen}^{SR}={\displaystyle \sum _{i=1}^N-\log D_{{\theta }_D}\left(G_{{\theta }_G}\left(I_i^{LR}\right)\right)},$$

(21)

where $D_{{\theta }_D}\left(G_{{\theta }_G}\left(I_i^{LR}\right)\right)$ is the estimated probability that the reconstructed image $G_{{\theta }_G}\left(I_i^{LR}\right)$ is a medical HR image. Note that we minimize $-\log D_{{\theta }_D}\left(G_{{\theta }_G}\left(I_i^{LR}\right)\right)$ instead of $\log \left[1-D_{{\theta }_D}\left(G_{{\theta }_G}\left(I_i^{LR}\right)\right)\right]$ for better gradient behavior [62].

We further employ a regularizer based on the TV to encourage spatially coherent solutions [6,67]. The regularization loss $L_{TV}^{SR}$, is calculated as:

$$L_{TV}^{SR}={\displaystyle \frac{1}{r^{2}WH}}{\displaystyle \sum _{m=1}^{rW}{\displaystyle \sum _{n=1}^{rH}\Vert \nabla G_{{\theta }_G}{\left(I_{}^{LR}\right)}_{m,n}\Vert }},$$

(22)

where $\nabla$ denotes the gradient operator, a vector operator that describes the rate of change and direction of a function at a given point.

3.4. Non-Subsampled Shearlet Transform (NSST) Prediction

Rather than performing SR reconstruction directly in the spatial domain, where GAN-based methods often lead to over-smoothing and texture loss, we formulate the task in the transform domain using NSST. NSST provides a mathematically grounded framework to decompose images into multi-scale, multi-directional components, effectively separating high-frequency directional details from low-frequency global structures. As a result, the generator can better preserve textures and edges. By combining NSST with adversarial learning, our approach explicitly targets both global topology and fine-grained texture, achieving a balance that is difficult to obtain with purely spatial-domain methods.

In the transform domain, images can be well perceived with richer global topological structure and local texture details than in the spatial domain. The most common transform domain methods include WT, curvelet transform, contourlet transform, shearlet transform, and NSST with superior performance. Wavelets can sparsely represent one-dimensional signals by smoothing out point discontinuities in digital signals. However, for two-dimensional image signals, the commonly used two-dimensional wavelets have only limited directions and cannot make full use of the geometric regularity of the image itself. As a result, with straight lines and curves, wavelet analysis [67] cannot “optimally” represent visual functions.

In contrast, wavelets and curvelets [68] capture the anisotropic regularity of surfaces along edges but lack a true multi-resolution geometric representation. Contourlets, composed of a Laplacian pyramid (LP) and a directional filter bank (DFB) in two filter-bank stages, offer limited directional diversity. The contourlet transform (CT) [69] samples at both LP and DFB stages, making it shift-variant—an undesirable property for many multimedia applications. To address this, Cunha et al. [70] proposed the non-subsampled contourlet transform (NSCT), a translation-invariant form of CT that removes all sub-sampling at the cost of increased redundancy. Labate et al. [71] later introduced the shearlet transform (ST), providing multi-scale, multi-resolution geometric representation via general multi-resolution analysis. Building on ST, NSST [72] was developed to overcome its limitations while preserving its advantages.

In this paper, NSST decomposition consists of two parts: multi-scale decomposition via a non-subsampled Laplacian pyramid filter (NSLPF) and multi-directional decomposition via modified shearing filters. The LF sub-band is recursively decomposed by NSLPF to capture singularities, producing k + 1 HF sub-bands and one LF sub-band. Shearing filters are mapped from the pseudo-polar grid to the Cartesian coordinate system, with all operations performed directly in the transform domain. This design avoids the negative effects of sampling operations and provides the desirable properties of multi-scale, multi-directional, and translation-invariant representation, making it particularly suited for medical image SR. As noted in the introduction, spatial-domain HR image prediction often leads to over-smoothing and texture loss. Transform-domain SR can better preserve details. In this work, we integrate NSST with MSFF-GAN to construct a medical image SR network, formulating SR as the prediction of NSST coefficients (Figure 1), enabling richer structural detail preservation compared to spatial-domain methods. In Figure 4 and Figure 5, the high-frequency coefficients of NSST and WT are compared on the lung image and the head image, respectively. Figure 4a and Figure 5a indicate the original images, Figure 4b and Figure 5b indicate the low-frequency image of NSST, and Figure 4c,d and Figure 5c,d indicate the high-frequency image of NSST at different scales and orientations.

Specifically, in our implementation, the four directions at scale 1 correspond to orientation channels centered approximately at 0°, 45°, 90°, and 135°, while the eight directions at scale 2 correspond to orientation channels centered approximately at 0°, 22.5°, 45°, 67.5°, 90°, 112.5°, 135°, and 157.5°, as determined by the directional filter banks in the NSST. Figure 4e and Figure 5e indicate the fused image of the high-frequency coefficients of NSST, and Figure 4f and Figure 5f indicate the fused image of the high-frequency coefficients of WT. By comparing Figure 4a–e and Figure 5a–f, it can be clearly observed that NSST can represent texture curvature and details more accurately.

Specifically, in order to fully mine LF structure and HF detail information of CT images from LR images, we obtain an LF sub-band and HF sub-bands by NSST transformation of LR images, which are, respectively, input to the generation network, in which the LF generation network is used to learn the global structure information of LR to HR images. The HF generation network is used to extract HF details such as edges and textures through several cascade multi-scale feature extractions from different levels of features, the production of LF and HF networks before output using a matching layer gathered in front of all the feature extractions, and then using the pixels on the convolution layer for sampling, and obtaining the corresponding HR images of LF and HF sub-bands through NSST inverse transformation to obtain the reconstruction image. Similarly, the discriminant network input is transformed from the traditional spatial synthesis data to the frequency domain sub-band, and the discriminant network input is the NSST sub-band generated by the network reconstruction and the NSST sub-band of the original HR image. NSST can be implemented in various SR networks, and it is a straightforward and effective method for improving the performance. Further experiments will be conducted in Section 4.5.3 to reveal the role of NSST. See Ref. [73] for more implementation details of NSST.

4. Experimental Results and Analysis

In the experiments, the effectiveness of the suggested method is assessed in terms of both qualitative and quantitative factors. We begin by outlining the training and testing datasets. The implementation process and evaluation protocol are then covered. Finally, we perform a number of ablation investigations and compare our approach to a number of cutting-edge SR approaches.

4.1. Medical Image Datasets

To build a medical image SR dataset, we collected images from four anatomical regions: brain, lung, bone, and abdomen, totaling 8000 samples (2000 per region). Brain and lung images were obtained from the Cancer Imaging Archive (TCIA) [29], while bone and abdominal images were sourced from the radiology department of a people’s hospital in Liaoning Province, China. The dataset was split into 7000 training images (1750 per region) and 1000 test images.

4.2. Implementation Details

For the 700 training images described in Section 4.1, data augmentation is performed. We evaluate training images that have been rotated by 90, 180, and 270 degrees and then have been flipped horizontally, as inspired by [7,8]. This process generated seven augmented variants in addition to each original image. Our network contains either 10 MSID blocks or 10 GMS blocks. Training used the ADAM optimizer with an initial learning rate of 0.0001, reduced by half every 50 epochs, and converged after 200 epochs. The entire training process took about 9.5 h on a single Tesla P40 GPU.

In the proposed MSFF-GAN, the selection of hyper-parameters such as convolution kernel size, number of feature channels, receptive field scale parameters (s and D), learning rate, and the number of multi-scale feature-extraction blocks is guided by both prior literature and empirical tuning. Initial values are set according to commonly adopted configurations in GAN-based super-resolution networks, while adjustments are made based on validation performance and computational efficiency. For example, a kernel size of 3 × 3 is used as a baseline because it balances receptive-field coverage and computational cost; the number of feature channels is fixed at 64 to provide sufficient representation capacity without excessive memory use; and the receptive-field scale parameters are set to s = 4 and D = 10 after comparative trials show that this combination yields optimal PSNR/SSIM gains within acceptable training time. During training, we observe a gradual decrease in content loss and adversarial loss, indicating stable convergence. The total loss typically reaches a plateau at approximately 150 epochs, and validation metrics (PSNR and SSIM) consistently improve until convergence, without obvious overfitting. These observations confirm the robustness and stability of the proposed training strategy.

4.3. Evaluation Protocols

We first employ two common evaluation measures, PSNR and SSIM, for the evaluation process [74]. These two metrics are important criteria currently used to evaluate advanced methods for SR. Better performance is indicated by higher values for both criteria.

The ratio of the maximum signal to background noise, or PSNR, is an objective measurement. The PSNR between the predicted image of size M × N and the true value is calculated as:

$$PSNR,\widehat{y},\mathrm{y},=10{\log }_{10}{\displaystyle \frac{{\displaystyle \sum _{i=1}^M}{\displaystyle \sum _{j=1}^{N}MA{X}_R^2}}{{\displaystyle \sum _{\mathrm{i}=1}^M}{\displaystyle \sum _{j=1}^N{(\widehat{y}(i,j)-y(i,j))}^2}}},$$

(23)

where $MA{X}_R$ is the maximum signal value.

The similarity between two images is gauged by the SSIM. Compared with PSNR, SSIM can better match human perception. The SSIM of the two images can be found as follows:

$$SSIM(X,Y)={\displaystyle \frac{(2{\mu }_X{\mu }_Y+c_1)(2{\sigma }_{XY}+c_2)}{({\mu }_X^2+{\mu }_y^2+c_1)({\sigma }_X^2+{\sigma }_Y^2+c_2)}},$$

(24)

where ${\mu }_X$ and ${\mu }_Y$ are averages, ${\sigma }_X$ and ${\sigma }_Y$ are variances, and ${\sigma }_{XY}$ is the covariance of X and Y, and $c_1$ and $c_2$ are constants that are used to maintain stability. The structural similarity is between −1 and 1. The result of SSIM equals 1 when there are no differences between two images.

In addition, the test results of this article are evaluated by a senior radiologist with over 10 years of imaging diagnosis experience, using the mean opinion score (MOS) [3] as the subjective evaluation criterion.

4.4. Comparison with State-of-the-Art Methods

The training process of MSFF-GAN exhibits stable convergence: loss curves decline smoothly, and evaluation metrics improve steadily until they reach a plateau. Such training dynamics indicate that the proposed architecture reliably captures multi-scale and transform-domain features, without exhibiting instability or mode collapse commonly seen in adversarial training. We evaluate the method on four medical datasets (brain, lung, bone, and abdomen) using PSNR, SSIM, and MOS as evaluation metrics. To guarantee a fair comparison, all models are trained following the same protocol on identical training data, and we adopt publicly available implementations of baseline methods. Quantitative results presented in

Table 1. Comparison of PSNR for different methods on the medical dataset. Bold indicates the best; underline indicates the second best.

Datasets	Scale	Bicubic	DWSR [45]	IDN [23]	MSRN [24]	RCAN [21]	DMSN [41]	USRNET [73]	TSAN [75]	Diff-GAN [48]	MapaNet [58]	MAT [28]	Ours (with MSID)	Ours (with GMS)
Abdomen	4	27.902	29.934	30.341	30.407	30.673	30.523	30.696	30.913	30.927	31.051	31.338	31.654	32.186
	8	24.694	26.095	26.199	26.325	26.900	26.591	26.964	27.217	27.436	27.514	27.911	28.134	28.625
Bone	4	26.414	28.451	28.555	28.631	28.822	28.685	28.863	28.995	29.073	29.231	29.352	29.877	30.175
	8	24.478	25.759	25.790	25.956	26.165	26.031	26.204	26.297	26.220	26.319	26.431	26.962	27.224
Brain	4	26.517	28.268	28.454	28.841	29.374	28.964	29.443	29.570	29.773	29.816	30.029	30.582	30.838
	8	22.338	23.926	24.184	24.317	24.952	24.733	24.957	25.016	24.958	25.326	25.634	25.851	26.413
Lung	4	27.338	29.196	29.323	29.455	30.356	29.958	30.506	30.693	30.718	30.936	30.942	31.314	31.793
	8	23.795	25.027	25.231	25.402	25.744	25.685	25.831	25.948	25.943	26.164	26.232	26.696	27.088

and

Table 2. Comparison of SSIM for different methods on the medical datasets. Bold indicates the best; underline indicates the second best.

Datasets	Scale	Bicubic	DWSR [45]	IDN [23]	MSRN [24]	RCAN [21]	DMSN [41]	USRNET [73]	TSAN [75]	Diff-GAN [48]	MapaNet [58]	MAT [28]	Ours (with MSID)	Ours (with GMS)
Abdomen	4	0.796	0.852	0.856	0.857	0.860	0.858	0.861	0.863	0.863	0.865	0.867	0.870	0.875
	8	0.673	0.717	0.722	0.728	0.740	0.730	0.741	0.743	0.745	0.746	0.749	0.751	0.757
Bone	4	0.427	0.644	0.649	0.662	0.666	0.661	0.669	0.670	0.671	0.673	0.674	0.678	0.681
	8	0.342	0.368	0.372	0.377	0.381	0.380	0.382	0.384	0.383	0.384	0.385	0.389	0.392
Brain	4	0.831	0.865	0.868	0.872	0.880	0.875	0.883	0.884	0.886	0.886	0.889	0.894	0.896
	8	0.704	0.755	0.759	0.763	0.776	0.764	0.780	0.785	0.784	0.788	0.790	0.792	0.798
Lung	4	0.895	0.869	0.871	0.874	0.881	0.879	0.884	0.888	0.889	0.891	0.891	0.895	0.899
	8	0.739	0.779	0.783	0.786	0.795	0.791	0.798	0.802	0.802	0.804	0.805	0.810	0.814

demonstrate that our method achieves superior performance across metrics and datasets, and qualitative comparisons confirm improved preservation of fine textures and anatomical boundaries in the reconstructed images.

For the MOS evaluation, 120 images were randomly sampled from the 400-image test set, each paired with its HR ground truth. A senior radiologist from a tertiary hospital scored the SR images based on image quality factors, including excessive smoothing, artifacts, texture fidelity, and signal-to-noise ratio. A four-point scale was used, where 1 indicates poor, 2 fair, 3 good, and 4 very good. The MOS was computed as the mean and standard deviation of all scores. As reported in

Table 3. Comparison (8×) of MOS for different methods. Bold indicates the best.

Methods	Number of Evaluation Images (120 in Total)				MOS (Mean and Standard Deviation)
	1 (Poor)	2 (Fair)	3 (Good)	4 (Very Good)
Bicubic	94	26	0	0	1.22 ± 0.5932
DWSR [45]	16	42	58	4	2.42 ± 0.8457
MSRN [24]	9	34	70	7	2.63 ± 0.8514
RCAN [21]	9	31	73	7	2.65 ± 0.8433
TSAN [75]	7	30	75	8	3.70 ± 0.8526
MAT [28]	4	20	88	8	2.83 ± 0.7656
Ours (with MSID)	1	14	93	12	2.97 ± 0.7430
Ours (with GMS)	1	4	101	14	3.07 ± 0.7146

, our method achieved the highest MOS at the ×8 scale. Visual comparisons in Figure 6 further confirm that the proposed method preserves abundant structural and textural details, producing reconstructions that are visually close to the ground truth, thereby demonstrating its strong capability for medical image SR.

4.5. Ablation Study

4.5.1. Effectiveness of MSID Block

In this subsection, we investigate the impact of MSID block and single-scale feature extraction. As shown in

Table 4. Average PSNR values (8×) under different numbers of MSID blocks.

Number of MSID Blocks	2	4	6	8	10	12
Average PSNR value	26.983	27.136	27.351	27.465	27.568	27.583

, we found consistent performance improvements using the MSID block, suggesting that MSID works better than single scaling. To achieve an optimal trade-off between computational complexity and reconstruction accuracy, we choose 10 MSID blocks as the standard configuration of the MSID network.

4.5.2. Effectiveness of GMS Block

In this subsection, we examine the effects of the quantity of feature map subsets (designated as s) and the quantity of Res2Net blocks (designated as D) in our GMS. As seen in Figure 7, we found that increasing either s or D consistently results in performance gains, demonstrating that deeper is better. Similarly, we chose the s = 4 and D = 10 combination, taking the trade-off between accuracy and speed into consideration.

4.5.3. Effectiveness of NSST

To evaluate the effectiveness of NSST, we incorporate MSFF-GAN in the spatial domain and combine it with wavelet transform [51], curvelet transform [70], and the proposed NSST. The experimental results across multiple datasets are shown in Figure 8. It can be observed that integrating MSFF-GAN with NSST significantly outperforms both the spatial-domain approach and the other two transform-domain methods, with this advantage remaining consistent across different network configurations and benchmark datasets. Furthermore, as shown in

Table 5. PSNR values for different high-frequency levels of decomposition based on our method with the MSID block (Scale: 8×, T = 10).

High-Frequency Level of NSST Decomposition	Body Part
	Abdomen	Bone	Brain	Lung
(The number of decomposition directions is 2 and 4, respectively) 2 levels	27.284	26.751	25.419	26.273
(The number of decomposition directions is 4 and 8, respectively) 3 levels	27.634	26.762	25.435	26.296
(The number of decomposition directions are 2, 4, and 8, respectively) 4 levels	27.288	26.764	25.438	26.297

, when combined with the MSID block, it achieves the highest scores across all decomposition levels, and the performance improves consistently as the number of high-frequency decomposition levels increases, further validating the effectiveness of NSST in multi-scale high-frequency information modeling.

5. Conclusions

In this work, we presented MSFF-GAN, a novel medical image SR framework that embeds a deep multi-scale convolutional architecture within a GAN generator to reconstruct images through cascaded multi-scale feature extraction blocks in a coarse-to-fine manner. The SR task is reformulated as the prediction of transform-domain coefficients, where incorporating NSST into the network enables more accurate retention of structural information from LR medical images than spatial-domain approaches, thereby further boosting reconstruction quality. Experimental results, both qualitative and quantitative, confirm that the proposed approach outperforms existing state-of-the-art methods and significantly improves the fidelity of restored medical images.