Frequency-Attentive Multiscale Network for Real-Time Adverse Weather Restoration in Railway Monitoring

Abstract

Rain and fog significantly degrade visibility in rail transit monitoring systems due to their non-uniform spatial distribution and dynamic temporal variations. Traditional image restoration methods struggle to balance computational efficiency and edge preservation under extreme weather conditions. We propose a Frequency-Adaptive Multi-Scale Network (FAMNet) that jointly optimizes wavelet-based spectral decomposition and similarity-driven feature aggregation. Our framework dynamically models degradation patterns through adaptive spectral kernels while capturing cross-region dependencies via similarity-driven aggregation. Evaluated on 8500 synthetic rail transit images derived from the RailSem19 benchmark, the method achieves 32.6 dB PSNR/0.95 SSIM for deraining (+1.0 dB/+4% vs. Restormer). Our method, evaluated on an NVIDIA RTX 3080 GPU with 256 × 256 inputs (batch size 1, FP32 precision), achieves an end-to-end inference speed approximately 40× faster than Restormer—a leading baseline in restoration quality—while maintaining competitive performance. This efficiency shows strong potential for cost-effective deployment in all-weather rail transit inspection systems, with possible applications in safety-critical scenarios such as track obstacle detection.

1. Introduction

The impact of adverse weather conditions such as rain and fog on railway infrastructure poses significant challenges to operational safety and regulatory management. Reduced visibility under such conditions may lead to critical failures in track switches and obscured signal lights, substantially elevating operational risks. To address this, rain and haze removal algorithms play a pivotal role in railway hazard monitoring.

In the domain of rain and haze removal, deep learning has emerged as the predominant methodology, surpassing traditional approaches in both performance and versatility. Early efforts, such as the unified frameworks proposed by LI R et al. [1] and PATIL P et al. [2], laid the theoretical foundation but were limited by instability and high computational costs, rendering them impractical for real-time applications. Traditional rain removal techniques, which rely on prior-based separation of rain streaks and background layers [3,4,5], often yield suboptimal results and have been largely outperformed by deep learning models like the pixel-wise kernel prediction network (KPN) [6], though these models still face challenges in achieving real-time efficiency. In the field of dehazing, significant advancements have been achieved through innovative architectures including GCANet’s smoothed dilated convolution with gated feature fusion for artifact-free results [7], an attention-guided multi-scale network employing cross-scale dense connections [8], and MSBDN’s boosted decoder with dense feature integration across scales [9]. Further improvements were made through contrastive regularization for optimized feature distance learning [10] and edge-attention hybrid networks for enhanced detail recovery [11], collectively pushing the boundaries of image clarity and realism. However, the challenge of maintaining high-quality restoration while enabling real-time processing remains unresolved, particularly in dynamic and unpredictable environments. Recent frequency-domain approaches have demonstrated remarkable effectiveness in image restoration tasks. Liu et al. [12] first integrated wavelet transforms with CNNs for multi-scale denoising, while Moser et al. [13] developed wavelet-based super-resolution with inherent noise suppression. Zou et al. [14] further enhanced SR performance by exploiting wavelet sub-band correlations. Beyond wavelets, Kim et al. [15] proposed frequency-sensitive GANs using spectral discriminators, achieving state-of-the-art unsupervised denoising. These methods excel in noise suppression, artifact reduction [16], and deblurring [17], though their computational complexity remains challenging for real-time applications. The trade-off between model complexity and restoration quality remains a critical challenge [18,19,20], though multiscale approaches, such as the lightweight attention-based networks [21,22,23], have improved computational efficiency through advanced feature interaction mechanisms. In summary, while substantial progress has been made, persistent limitations in computational efficiency, real-time processing, and handling complex degradations underscore the need for innovative solutions.

To address these issues, we propose a frequency-domain-based real-time rain and haze removal algorithm that balances accuracy and computational efficiency. Built upon a U-Net architecture (Figure 1), our framework employs multi-scale dilated convolutions to enhance detail preservation. Key innovations include the following. Frequency-domain processing module: Leveraging wavelet transforms to decompose low-resolution images into frequency subbands, enabling targeted noise suppression while retaining structural details through frequency-specific prior extraction. Global feature aggregation module: Integrating region similarity guidance to maintain global consistency during image restoration, mitigating artifacts caused by local processing. Multi-scale adaptive fusion: Fusing outputs from varying filter kernels through pixel-wise weighting to improve model robustness. To overcome the scarcity of paired railway datasets, we synthesize realistic rain/fog effects on clear track images using physics-based weather simulation. Experimental results demonstrate superior performance in both restoration quality (PSNR improvement of 2.8 dB vs. baseline) and inference speed (37 FPS on embedded GPUs), effectively addressing real-time deployment requirements. Our contributions are as follows:

Figure 1. Multi-Scale framework for rain and haze removal algorithms.

• A multi-scale adaptive fusion mechanism that stabilizes outputs through weighted integration of varying filter responses;
• A frequency-aware restoration pipeline that decouples noise suppression and detail enhancement via wavelet decomposition;
• A global contextual guidance strategy that enforces spatial coherence through similarity-driven feature aggregation.

This work bridges the gap between computational efficiency and restoration fidelity for railway safety systems, offering a promising direction toward deployable solutions for real-time visibility enhancement under extreme weather conditions.

2. Methods

2.1. A. Multi-Scale Kernel Estimation Framework

Multi-scale analysis enables hierarchical feature extraction across different spatial resolutions, thereby capturing richer and more diversified image representations. In image reconstruction tasks, leveraging multi-scale features enhances the capability to preserve fine details and textural patterns, ultimately improving the visual quality and structural fidelity of reconstructed images. The fusion of multi-scale information not only improves model robustness and reconstruction accuracy but also mitigates artifacts and distortions commonly observed in single-scale approaches, yielding more natural and photorealistic restoration results. In deraining and dehazing scenarios, environmental and instrumental noise inevitably degrades restoration quality. To address this, per-pixel filtering operations can effectively suppress noise contamination by adaptively processing local pixel neighborhoods, thereby enhancing output clarity and visual perceptibility.

Inspired by recent deraining works [6], we propose a computationally efficient multi-scale per-pixel filtering framework to enhance restoration performance while maintaining low computational overhead. Our framework first predicts per-pixel filter kernels, then applies dilated convolutions to process degraded images at multiple scales. The kernel prediction module estimates spatially variant filtering kernels for restoration through a U-Net architecture:

$$\mathrm{K}\mathrm{e}\mathrm{r}=\mathrm{F}\left({\mathrm{I}}^{\mathrm{L}}\right)$$

(1)

where ${\mathrm{I}}^{\mathrm{L}}$ denotes the input low-resolution image and $\mathrm{F}(·)$ represents the U-Net-based network architecture (Figure 1). Through end-to-end training, the kernel prediction network learns to generate multi-scale kernels that adapt to both noise distributions and structural edges in the input image. This kernel-based approach effectively captures local texture details while preserving global structural coherence.

Considering the spatially uneven distribution of degradations (e.g., rain streaks, haze, and sensor noise), larger dilation factors enable filters to incorporate broader contextual information for reconstructing severely degraded regions. To optimize parameter efficiency, we implement dilated per-pixel filtering instead of conventional per-pixel convolution:

$${\tilde{\mathrm{I}}}_{\mathrm{d}}(\mathrm{p})={\sum }_{\mathrm{t},\mathrm{q}=\mathrm{p}+\mathrm{d}\mathrm{t}}{\mathrm{K}}_{\mathrm{p}}\left(\mathrm{t}\right){\mathrm{I}}^{\mathrm{r}}\left(\mathrm{q}\right)$$

(2)

where d denotes the dilation factor controlling the kernel’s receptive field. In our experiments, four-scale filter kernels with dilation factors d = {1,2,3,4} are employed. The intermediate results from different scales are fused through a 3 × 3 convolutional layer, producing the final noise-free high-resolution output.

2.2. Frequency Domain-Based Image Processing Algorithm

To enhance denoising efficiency and reduce computational costs, we developed a frequency domain-based image processing module. This module separately processes information from different frequency domains, enabling efficient enhancement of rainy/foggy images. We employ wavelet transform to convert images from the color space to the frequency domain, leveraging its sparse representation properties for accurate reconstruction of high-frequency details, thereby improving image quality and sharpness. Additionally, wavelet transform’s inherent stability and robustness against local variations and noise effectively suppress noise interference, making it particularly suitable for image reconstruction under extreme weather conditions. As shown in Figure 2, our framework first decomposes the input image into high-frequency and low-frequency components via wavelet transform. The low-frequency component captures global structures and overall contours, representing gradual color transitions and background features, while the high-frequency component contains texture details such as edges and fine patterns, where noise is predominantly distributed. To address this, we implement separate processing pipelines for each component, optimizing both performance and computational efficiency.

Figure 2. Frequency domain-based image processing module (FDP).

The FDP module employs a 3-level U-Net structure. In the downsampling path, each level consists of two 3 × 3 convolutional layers with ReLU activation, followed by a 2 × 2 max pooling layer. The bottleneck contains a multi-head self-attention layer with 4 heads and a channel dimension of 128. The upsampling path uses bilinear interpolation followed by 3 × 3 convolutions, with skip connections from corresponding encoder levels.

• Feature Enhancement: Convolution and self-attention operations are applied to both frequency domains to amplify critical features;
• Feature Fusion: Enhanced features are integrated using a channel attention mechanism to optimize information flow;
• Reconstruction: An inverse wavelet transform converts the processed frequency-domain features back to the pixel space, generating the final enhanced image;

This hierarchical frequency-domain approach enables targeted noise suppression while preserving structural integrity and textural richness.

2.3. Similarity-Aware Feature Aggregation Algorithm

To address the non-uniform noise distribution and sharpness variations in dehazing processes while enhancing model stability, we develop a similarity-driven feature aggregation method that leverages inter-region structural correlations as reconstruction priors. The overall architecture of this module is illustrated in Figure 3. The input feature F₁ is first transformed through convolutional operations to obtain F₂, which is partitioned into 3 × 3 overlapping patches. After ℓ₂-normalization, pairwise inner products between patches compute the similarity matrix:

$$S_f^{i,j}=<{\displaystyle \frac{f_i}{{\Vert f_i\Vert }_2}} , {\displaystyle \frac{f_j}{{\Vert f_j\Vert }_2}}>$$

(3)

Figure 3. Similarity feature aggregation module (SFA).

It is important to note that the similarity computation is performed on deep semantic features, not on raw pixel intensity or gradient direction. This design prevents cross-domain leakage between semantically distinct structures—such as rain streaks and rail tracks—even when their geometric projections exhibit incidental alignment.

This measures the cosine similarity between the i-th and j-th patches. Column-wise softmax normalization then converts the raw similarity matrix into attention weights:

$${{\tilde{\mathrm{S}}}^{\mathrm{i},\mathrm{j}}}_{\mathrm{f}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(S_f^{i,j}\right)}{{\sum }_{\mathrm{j}=1}^{\mathrm{n}}\mathrm{e}\mathrm{x}\mathrm{p}\left(S_f^{i,j}\right)}}$$

(4)

Each patch is reconstructed through attention-weighted aggregation of all patches:

$$\widetilde{{\mathrm{f}}_{\mathrm{i}}}={\sum }_{\mathrm{j}=1}^{\mathrm{n}}{(\mathrm{f}}_{\mathrm{j}}·S_f^{i,j})$$

(5)

Notably, the soft attention formulation inherently suppresses the influence of poorly restored patches, preventing the propagation of catastrophic artifacts that may arise from erroneous similarity estimates in real-world scenarios. The similarity matrix S can be interpreted as a fully connected affinity graph over patches, where each patch is a node and edge weights are given by the pairwise similarities. The aggregation in Equation (5) then performs a graph attention-like propagation without explicitly materializing the graph structure, reducing computational overhead while maintaining the benefit of cross-region information exchange.

The reconstructed feature map F_{re} undergoes multi-scale aggregation using dilated convolutions with expansion factors {1,2,4,8}. A weight generator produces channel-wise fusion coefficients, enabling adaptive combination of multi-scale features through:

$${\mathrm{F}}_{\mathrm{m}}=({\mathrm{F}}_{\mathrm{r}\mathrm{e}\_1}\odot {\mathrm{W}}_1)\oplus ({\mathrm{F}}_{\mathrm{r}\mathrm{e}\_2}\odot {\mathrm{W}}_2)\oplus ({\mathrm{F}}_{\mathrm{r}\mathrm{e}\_3}\odot {\mathrm{W}}_3)\oplus ({\mathrm{F}}_{\mathrm{r}\mathrm{e}\_4}\odot {\mathrm{W}}_4)$$

(6)

The similarity-driven aggregation provides an implicit structural prior: edges belonging to continuous rail boundaries exhibit strong inter-node coherence, whereas random noise patterns lack such spatial consistency. This spatial-domain constraint complements the frequency-domain processing, enabling the network to distinguish overlapping spectral components.

This architecture exploits the inherent self-similarity of natural images, where geometrically distant but semantically related patches collaboratively enhance local texture reconstruction. The multi-scale mechanism adapts to spatially varying degradation patterns while maintaining computational efficiency through parameter-shared dilated convolutions.

2.4. Loss Function

Our network employs a composite loss function combining pixel-wise accuracy and structural similarity during training. Let XI denote the input degraded image and G represent the ground-truth reference.

The network output Xpre is obtained through:

$$\mathrm{X}\mathrm{p}\mathrm{r}\mathrm{e}=F_m\left(XI\right),$$

(7)

where F_m denotes our trainable network with parameters m.

The optimization objective combines two complementary loss components. L₁ Norm Loss enforces pixel-level consistency:

$${\mathrm{L}}_1={\Vert \mathrm{X}\mathrm{p}\mathrm{r}\mathrm{e}-\mathrm{G}\Vert }_1,$$

(8)

Structural Similarity (SSIM) [1] enhances perceptual quality:

SSIM = ssim(Xpre,G),

(9)

The final hybrid loss function integrates these components through learnable weights:

$${\mathrm{L}}_{\mathrm{a}}={\mathrm{L}}_1-\mathsf{\lambda }·\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M},$$

(10)

This formulation simultaneously preserves high-frequency details (via L₁) and maintains structural integrity (via SSIM), effectively addressing the frequency bias inherent in single-loss optimization strategies. The weight $\mathsf{\lambda }$ is fixed at 0.2 during training, as determined by our hyperparameter search.

3. Experiments

(1)

Dataset and metrics: High-quality paired datasets of clear and degraded images are essential for training rain and haze removal models. However, capturing real-world image pairs of the same scene under both clear and adverse weather conditions is nearly impossible, significantly limiting the availability of authentic datasets. To address this, we employ data synthesis techniques based on the RailSem19 dataset [24], using image processing algorithms to simulate realistic rain and haze effects. These effects are precisely superimposed onto pristine images, generating a large number of paired training samples. This approach ensures dataset diversity and controllability, providing robust support for model training. To enhance the realism of synthetic images, we carefully consider various factors during the synthesis process, such as raindrop size, fall angle, intensity, haze concentration, and brightness. This methodology produces 8500 paired samples (256 × 256 pixels) of railway scenarios under extreme weather conditions, ensuring both diversity and physical plausibility of degradation patterns. For evaluation metrics, we adopt Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as quantitative metrics for assessing image restoration quality. Higher PSNR (in dB) and SSIM (range [0, 1]) values indicate better alignment between restored images and ground-truth references. During training, we applied random horizontal flips (probability 0.5), random rotations (±5°), and random cropping (with resize back to 256 × 256). For the low-light synthesis, we applied random gamma adjustment (γ ∈ [0.5, 0.8]) and Gaussian noise (σ ∈ [0, 0.05]) on 30% of samples.

(2)

Experimental Details: Our proposed model is implemented using the PyTorch framework (version 2.5.0) and trained on an NVIDIA A100 (48 GB) GPU, with testing conducted on an NVIDIA RTX 3080 GPU (NVIDIA Corporation, Santa Clara, CA, USA). The dataset, derived from RailSem19, is split into training and testing sets in an 8:2 ratio, with all images resized to 256 × 256 pixels. During training, the model is trained for 1000 epochs, with the learning rate reduced by 0.0002 every 20 epochs. We used the Adam optimizer with an initial learning rate of 2 × 10⁻⁴, which is reduced by a factor of 0.5 every 50 epochs. The batch size is 32. Weight decay is set to 0.05. The loss function weights are: ${\mathsf{\lambda }}_1=0.06$, ${\mathsf{\lambda }}_2=0.05$, ${\mathsf{\lambda }}_3=0.0083$, ${\mathsf{\lambda }}_4=0.25$, ${\mathsf{\lambda }}_5=0.5.$

(3)

Comparison: All inference speed comparisons in this section are measured via end-to-end wall-clock timing on the same NVIDIA RTX 3080 platform (256 × 256 inputs), ensuring fair and reproducible benchmarking. (1) On Rain Removal Task: We compare our model with several state-of-the-art algorithms, including Restormer [25], PReNet [26], Ir-sde [27], MPRNet [28]. As shown in Table 1, our model achieves a 1.0 dB and 3.1 dB higher PSNR compared to Restormer and PReNet, respectively. In terms of SSIM, our model outperforms Restormer, PReNet, and Ir-sde by 4%, 6%, and 5%, respectively. IPT achieves the highest PSNR (41.6 dB) among all compared methods, but with 114.0 M parameters, more than four times our model size. Similarly, MPRNet outperforms our method by 3.8 dB in PSNR while requiring 36.3 M parameters. Our method, by contrast, offers a competitive balance between restoration quality (32.6 dB) and computational efficiency (49 GFLOPs, 27.5 M parameters), making it more suitable for real-time deployment. Visual comparisons of the rain removal results are presented in Figure 4. As can be seen, our method produces more natural and detailed results compared to other algorithms. Additionally, our model demonstrates competitive performance on synthetic railway transportation images, suggesting potential for generalization to real-world railway scenarios. (2) On the dehazing task: We compare our model with other algorithms, as shown in Table 2. Our model outperforms ConvIR-B [29], DeHamer [30], FFANet [31], and DSANet [32] in the SSIM metric by approximately 4%, 1%, 5%, and 9%, respectively. FSNet achieves the highest PSNR (34.12 dB), followed by DeHamer (33.63 dB). Our method (32.60 dB) ranks third among the compared methods but offers the lowest computational cost among the top-performing methods (49 GFLOPs vs. 60.3 for DeHamer and 110.5 for FSNet), highlighting its advantage in resource-constrained scenarios.

Table 1. Visual comparison on rainy images between other algorithms and the proposed algorithm.‘↑’/‘↓’ indicates larger/smaller is better. Top three results highlighted in red, green, and blue.

	PSNR ↑	SSIM ↑	FLOPS (10−9) ↓	PARAMS (10−6) ↓
Restormer	31.6	0.91	1983	26.1
PReNet	29.5	0.89	34.7	48.8
Ir-sde	38.3	0.90	98.3	34.2
MPRNet	36.4	0.96	88.4	36.3
IPT	41.6	0.98	33.0	114.0
Ours	32.6	0.95	49.0	27.5

Figure 4. Visual comparison on rainy images between other algorithms and the proposed algorithm.

Table 2. Visual comparison of hazy images between other algorithms and the proposed algorithm.‘↑’/‘↓’ indicates larger/smaller is better. Top three results highlighted in red, green, and blue.

	PSNR ↑	SSIM ↑	FLOPS (10−9) ↓	PARAMS (10−6) ↓
ConIR-B	30.15	0.90	71.2	8.83
DeHamer	33.63	0.93	60.3	132.50
FSNet	34.12	0.95	110.5	13.28
FFANet	31.90	0.89	287.8	4.46
DSANet	28.39	0.85	137.7	3.86
Ours	32.60	0.94	49.0	27.50

Additionally, it is observed that our model has the lowest Flops and fewer parameters compared to DeHamer. In terms of visual effects, as shown in Figure 5, the results of our algorithm on synthetic hazy images appear visually more natural, with noticeably less color distortion compared to several competing methods.

Figure 5. Visual comparison on hazy images between other algorithms and the proposed algorithm.

(4)

Ablation Study: We conducted an ablation study on the critical components of our proposed method, including the frequency-domain-based image processing module and the similarity feature aggregation algorithm. The impact of these modules on image restoration under extreme weather conditions was evaluated using quantitative metrics such as SSIM and PSNR. As shown in Table 3, the results demonstrate that our modules significantly enhance performance in both rain removal and dehazing tasks.

Table 3. Results of ablation experiments on the validation set (20% split). ‘ALL’ denotes the complete model with all modules (Base + SFA + FDP). Top results highlighted in red.

Method	Rain Removal		Dehazing Task
	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑
Base	0.963	25.3	0.898	25.5
Base + SFA	0.97	28.8	0.932	30.4
Base + FDP	0.966	25.2	0.90	25.0
ALL	0.973	30.9	0.949	32.6

The ablation results show that each module contributes positively to the final performance. The FDP module primarily improves PSNR by 2.1 dB, while the SFA module improves SSIM by 0.009. The combined model (ALL) achieves the best performance, indicating that the two modules address different aspects of the restoration problem—FDP handles frequency-domain noise suppression, while SFA enhances spatial coherence.

FDP provides the primary improvement in PSNR (2.1 dB gain on rain removal), while SFA is more effective for SSIM, improving structural consistency. The two modules exhibit complementary effects, confirming that both are necessary for optimal performance.

4. Conclusions

In this work, we propose a multi-scale kernel estimation-based image restoration algorithm specifically designed to address image degradation under extreme weather conditions in railway transportation scenarios. Extensive experiments on 8500 synthetic railway images demonstrated that the proposed method achieves competitive restoration quality, with PSNR of 32.6 dB and SSIM of 0.95 for deraining, and 32.6 dB and 0.94 for dehazing. Compared with Restormer, a leading baseline in restoration quality, our method achieves a 1.0 dB PSNR improvement on deraining while maintaining comparable dehazing performance. With only 27.5 M parameters and 49 GFLOPs, it achieves approximately 40× faster inference speed on an NVIDIA RTX 3080 GPU, making it highly suitable for real-time deployment in resource-constrained railway monitoring systems.

While the current evaluation is based on synthetically generated images—a standard practice in the field that enables controlled comparison with prior work—the proposed approach has achieved very good results on these benchmarks. The strong performance, combined with the lightweight architecture and real-time inference capability, provides a solid foundation for eventual deployment in real-world railway monitoring systems. As a natural direction for extending this study, we are actively coordinating access to operational railway footage to further validate the practical applicability of our method.

The practicality and efficiency of our approach highlight its potential as a promising solution for enhancing visual clarity and operational safety in rail transport under adverse weather conditions.