Semi-Supervised Degradation-Aware Learning for All-in-One Weather-Degraded Image Restoration

Abstract

All-in-one weather-degraded image restoration aims to restore clean images from diverse weather-degraded observations (such as rain, haze, and snow) using a unified model. However, this topic remains challenging due to its ill-posed nature and the scarcity of large-scale paired training data. This article develops a novel semi-supervised learning framework, termed Semi-Supervised Degradation-Aware Learning (S²DAL), to adjust the feature space to align with the unified parameter space for all-in-one adverse weather removal. Specifically, the proposed S²DAL consists of two backbone networks: a Degradation-guided Histogram Transformer (DHformer) for weather-degraded image restoration and a Degradation-guided Convolutional Neural Network (DCNN) for degradation generation. A key component, the Degradation-guided Histogram Transformer (DHT) block, is designed to effectively capture intrinsic image features while suppressing diverse degradation interference through channel shuffling modulation, dynamic-range histogram self-attention, and dual-scale gated feed forward. Furthermore, a Monte Carlo-based Expectation-Maximization (EM) algorithm is introduced to jointly optimize latent variables and network parameters under both labeled and unlabeled data. Extensive quantitative and qualitative results on synthetic and real-world datasets consistently demonstrate that the proposed S²DAL achieves superior restoration performance compared to multiple state-of-the-art fully supervised and semi-supervised approaches.

1. Introduction

Image restoration under adverse weather conditions (e.g., rain, fog, and snow) has attracted significant attention in the computer vision community. This task constitutes a highly ill-posed inverse problem, as multiple plausible solutions may correspond to the restoration. Previous studies have primarily focused on single weather-degraded image restoration, such as de-raining [1,2,3], de-hazing [4,5,6], and de-snowing [7,8,9]. Although these approaches demonstrate strong performance under specific weather conditions, their single-task nature substantially limits their practicality in real-world applications, such as autonomous driving [10], surveillance systems [11], and more. Considering this, recent efforts have shifted toward all-in-one weather-degraded image restoration, which seeks to recover clean images from diverse weather-degraded inputs using a unified model with consistent parameters [12].

In recent years, there has been a growing interest in all-in-one weather-degraded image restoration tasks, and numerous advanced approaches [13,14,15,16,17,18,19,20,21,22,23,24,25,26] have been developed to achieve state-of-the-art restoration performance across various weather conditions. Despite the notable progress, most of them are designed to fit joint regression models over degraded/clean image pairs in a fully supervised manner, which necessitates the manual collection of large-scale paired samples for model training. Consequently, their generalization ability on real-world weather-degraded images is limited. More recently, some researchers have adopted semi-supervised schemes to improve the generalization ability of restoration networks. For example, Long et al. [27] proposed a semi-supervised framework for all-in-one adverse weather removal, which is built upon a teacher–student architecture that adopts a Wavelet diffusion model [28] as the backbone. Subsequently, Xu et al. [29] leveraged the vision–language model to construct a semi-supervised learning framework, aiming to enhance restoration performance across multiple adverse weather conditions in real-world settings. Although these semi-supervised learning frameworks mentioned above have achieved promising results, the quality of pseudo-labels generated by the teacher network plays a critical role in determining the final restoration performance because inaccurate or biased pseudo-labels may propagate errors and limit the effectiveness of the student model. Therefore, all-in-one weather-degraded image restoration remains a challenging and largely unresolved problem, which leaves substantial room for further improvement.

Given the limitations in previous semi-supervised adverse weather removal methods, this article presents a new semi-supervised learning framework, called Semi-Supervised Degradation-Aware Learning (S²DAL), to adjust the feature space to align with the unified parameter space for all-in-one adverse weather removal. Specifically, the proposed S²DAL is composed of two backbone networks: a Degradation-guided Histogram Transformer (DHformer) and a Degradation-guided Convolutional Neural Network (DCNN). Among them, DHformer is built upon multiple Degradation-guided Histogram Transformer (DHT) blocks, which incorporate the guidance of degradation-type prompts, and can fully mine the inherent features of the clean image while reducing the interference of different weather-induced degradation patterns. In order to form a semi-supervised learning framework under the unlabeled data condition, DCNN is designed to mimic the generation process of degradation layers, which is also guided by the degradation-type prompts to effectively disentangle the generation of degradation layers. Beyond that, a Monte Carlo-based Expectation-Maximization (EM) algorithm is introduced to optimize the parameters and latent variables in our proposed S²DAL. The main contributions of this article are:

• We propose a novel semi-supervised learning framework for all-in-one weather-degraded image restoration. The proposed method, called S²DAL, gets rid of the large-scale labeled data for fully supervised training.
• We design a DHT block to build the DHformer for restoring the clean image from its corresponding weather-degraded observation. The DHT block can adaptively regulate the feature space through channel shuffling modulation to better align with the shared parameter and network structure, thereby enhancing the all-in-one restoration performance.
• We introduce the Monte Carlo-based EM algorithm to jointly optimize the network parameters and latent variables in the proposed S²DAL, and extensive experimental results on both synthetic and real-world weather-degraded images demonstrate the effectiveness and superiority of our method quantitatively and qualitatively.

The remainder of this paper is organized as follows. Section 2 surveys the related work on image restoration under adverse weather conditions, including single weather-degraded image restoration and all-in-one weather-degraded image restoration. Section 3 details the proposed S²DAL framework. Section 4 introduces the experimental setup and provides a comprehensive performance analysis. Finally, Section 5 concludes the paper.

2. Related Work

2.1. Single Weather-Degraded Image Restoration

Previous studies have predominantly focused on single weather-degraded image restoration tasks (e.g., de-raining [2,3,30], de-hazing [5,6,31], and de-snowing [8,9,32]), which have achieved substantial advancements.

Conventional model-based methods [33,34,35,36,37,38] formulate the single weather-degraded image restoration as a signal separation problem and attempt to exploit various effective priors to impose additional constraints to characterize the background or degradation layers, and then separate them by solving an objective function using a proper optimization algorithm. The representative works include but are not limited to dark channel [33], discriminative sparse coding [37], color attenuation [38], non-local means filter [36], dictionary learning [35], Gaussian mixture model [34], and more. With the rise of deep learning, numerous approaches [5,6,8,9,31,39] based on Convolutional Neural Networks (CNNs) have been developed. Besides the CNN-based approaches, some works [32,40,41] seek to employ the adversarial learning strategy to train Generative Adversarial Networks (GANs [42]) to better restore clean images from real-world weather-degraded ones. However, both CNNs and GANs rely on convolution operations, which are spatially invariant and have a limited receptive field, thereby hindering their ability to model spatially variant properties and global structures of clean images [43]. Considering these limitations, recent researchers [44,45] introduced Transformers [46] to solve the single weather-degraded image restoration problem and yielded significant advancements, as they were able to effectively model the non-local information to achieve better image restoration. While single weather-degraded image restoration methods have received significant interest, most of them are difficult to generalize effectively to different weather-degraded image restoration tasks.

2.2. All-in-One Weather-Degraded Image Restoration

In contrast to single weather-degraded image restoration, all-in-one weather-degraded image restoration aims to address multiple different weather-degraded image restoration tasks using a unified model. And the key challenge lies in how to alleviate the mutual interference among different degradation patterns while faithfully preserving the inherent features of the clean image [47].

One line of research addresses this issue by adapting the network’s parameter space to accommodate different degradation types, and some works have explored this line of research to remove diverse weather degradations in the images. For example, Li et al. [14] made the first attempt to handle the all-in-one weather-degraded image restoration problem. They employed a neural architecture search technique to find the proper network architectures for multiple weather-specific encoders. However, their all-in-one framework incurs a large number of parameters due to the usage of separate encoders for each individual weather removal task. Considering this, Valanarasu et al. [15] developed a transformer-based all-in-one weather removal framework by incorporating intra-patch transformer blocks and learnable weather-type embeddings. Zhu et al. [48] introduced a unified weather removal model that utilizes two separate training stages to learn the weather-general and weather-specific features automatically. Özdenizci et al. [16] designed a patch-based weather-degraded image restoration framework by performing smoothed noise estimation across overlapping patches. Chen et al. [49] explored a two-stage knowledge learning mechanism for all-in-one adverse weather removal, where a student network collaboratively distills knowledge from multiple weather-specific teacher networks and is further refined using a multi-contrastive knowledge regularization loss. Sun et al. [18] proposed a histogram transformer to achieve all-in-one weather-degraded image restoration by introducing histogram self-attention to capture long-range spatial dependencies under adverse weather conditions.

Another line of research is to introduce image or degradation-type prompts to modulate feature representations so as to allow the restoration model to have a unified parameter space. For example, Tan et al. [17] designed a text-based classification loss that computes the cosine distance between textual and visual features, producing a similarity tensor that reflects the probabilistic association between each image and specific weather category. This design enables the network to adaptively accommodate diverse weather conditions. Wen et al. [25] proposed an adaptive degradation-aware self-prompting framework, which introduces the contrastive language–image pre-training model (CLIP [50]) to learn latent caption prompts to guide the reverse sampling process of a diffusion model for all-in-one weather-degrade image restoration. Liao et al. [51] developed a cyclic prompt learning approach, which integrates weather-specific knowledge, textual context, and reliable textures to enhance the performance for universal adverse weather removal. Shao et al. [52] proposed a unified memory-enhanced visual–language recovery method that combines a lightweight encoder–decoder backbone with a visual–language model and an implicit memory bank to restore images under diverse and mixed adverse weather conditions. Yang et al. [21] introduced a language-driven restoration framework, which exploits pre-trained vision–language models to infer the degradation occurrence, type, and severity, and to generate description-based degradation priors for all-in-one weather-degraded image restoration.

Even though the abovementioned all-in-one weather-degraded image restoration approaches have achieved impressive performance, they heavily rely on a large amount of manually labeled data, i.e., degraded/clean image pairs. Due to the inherent limitations in collecting paired degraded and clean images, existing fully supervised methods for adverse weather removal may struggle to generalize to real-world weather-degraded cases. Recently, self-learning methods have achieved significant progress in many vision-related tasks due to their ability to leverage large-scale unlabeled data without requiring expensive manually labeled data, thereby improving model generalization capability [53]. Despite that, self-learning may face substantial challenges in all-in-one adverse weather image restoration, as multiple degradation types in an all-in-one setting exhibit substantially different physical properties and visual characteristics. Without reliable ground-truth supervision from labeled data, purely self-learning methods, where supervisory signals are automatically generated from the data itself rather than relying on manually annotated labels, may introduce error accumulation and unstable optimization when handling diverse weather degradations. In contrast, the semi-supervised paradigm allows the model to benefit from a limited amount of reliable labeled data while simultaneously exploiting abundant unlabeled degraded images, thereby achieving a better trade-off between restoration fidelity and generalization capability [27,29]. In light of this, several approaches have employed semi-supervised learning strategies to enhance the generalization ability of restoration networks when they are applied to real-world weather-degraded images. For example, Long et al. [27] developed a semi-supervised all-in-one adverse weather removal framework built upon the teacher-student network that uses a Wavelet diffusion model [28] as the backbone. Subsequently, Xu et al. [29] introduced the vision–language model to construct a semi-supervised learning framework, aiming to enhance restoration performance across multiple adverse weather conditions in real-world settings. However, both methods require the teacher network to generate high-quality pseudo-labels to impose the supervised training on the student network, while inaccurate or biased pseudo-labels may propagate errors and thereby impair the restoration performance of the student model.

3. Proposed Semi-Supervised Degradation-Aware Learning Framework

3.1. Model Formulation

Let $\mathbf{y}$ denote a weather-degraded image, and ${\mathit{P}}_d$ is its corresponding degradation-type prompt. This paper is devoted to building a Semi-Supervised Degradation-Aware Learning (S²DAL) framework to restore the clean image from its corresponding weather-degraded input, with the guidance of degradation-type prompts. Similar to [54,55], this work decomposes the weather-degraded image $\mathbf{y}$ into three components as follows:

$$\begin{array}{c}\hfill \mathbf{y}=\mathbf{x}+\mathcal{D}+\mathcal{R},{\mathcal{R}}_{ij}\sim \mathcal{N}(0,{\sigma }^2),\end{array}$$

(1)

where $\mathbf{x}$, $\mathcal{D}$, and $\mathcal{R}$ represent the latent clean background, degradation layer, and residual term, respectively. The residual term $\mathcal{R}$ is assumed to follow a zero-mean Gaussian distribution with variance ${\sigma }^2$, and ${\mathcal{R}}_{ij}$ is the element of $\mathcal{R}$ at spatial location $(i,j)$. The mapping from the observed degraded image $\mathbf{y}$ to its underlying clean background $\mathbf{x}$ is parameterized by a deep neural network, which we refer to as image restoration model $f(·;\mathit{W})$, where $\mathit{W}$ represents the network parameters. In the following, we separately model the clean background $\mathbf{x}$ and the degradation layer $\mathcal{D}$.

3.1.1. Modeling of Background Layer

For degraded images caused by various adverse weather conditions, the corresponding clean background typically exhibits strong local correlations among neighboring pixels. To capture this prior knowledge, we introduce a Markov Random Field (MRF) prior for unlabeled degraded images $\mathbf{y}$, formulated as

$$\begin{array}{c}\hfill p\left(\mathit{W}\right)\propto \exp \left(-\rho \sum _{i,j}\mathit{\upsilon }\mathit{\gamma }\right),\end{array}$$

(2)

where

$$\mathit{\upsilon }=\left[\begin{array}{c}|f_{i+1,j}-f_{i,j}|\hfill \\ |f_{i,j+1}-f_{i,j}|\hfill \end{array}\right],\gamma =\left[\begin{array}{c}{\gamma }_1\hfill \\ {\gamma }_2\hfill \end{array}\right],$$

(3)

where $f_{i,j}$ represents the value of the restored image $f(\mathbf{y},{\mathit{P}}_d;\mathit{W})$ at spatial location $(i,j)$. $\rho$ and $\gamma$ are manually specified hyper-parameters.

Furthermore, for labeled degraded images $\mathbf{y}$, the corresponding ground-truth clean background $\mathbf{x}$ provides an additional strong prior, such as Gaussian assumption and linear correlation prior, which can be incorporated into Equation (2), i.e.,

$$\begin{array}{cc}\hfill & p\left(\mathit{W}\right)\propto \hfill \\ & \exp \left(-\rho \sum _{i,j}{\mathit{\upsilon }}^T\gamma -{\displaystyle \frac{{\parallel f(\mathbf{y},{\mathit{P}}_d;\mathit{W})-\mathbf{x}\parallel }_2}{{\epsilon }_0}}-\left(1-{\displaystyle \frac{{\sum }_{n=1}^{3HW}\left(f{(\mathbf{y},{\mathit{P}}_d;\mathit{W})}_n-\overline{f}(\mathbf{y},{\mathit{P}}_d;\mathit{W})\right)({\mathbf{x}}_n-\overline{\mathbf{x}})}{3HW\sigma \left(f(\mathbf{y},{\mathit{P}}_d;\mathit{W})\right)\sigma \left(\mathbf{x}\right)}}\right)\right),\hfill \end{array}$$

(4)

where ${\epsilon }_0$ is a very small hyper-parameter close to zero. H and W denote the height and width of images. $f{(\mathbf{y},{\mathit{P}}_d;\mathit{W})}_n$ denotes the n-th pixel of the restored image, whereas $\overline{f}(\mathbf{y},{\mathit{P}}_d;\mathit{W})$ and $\sigma \left(f(\mathbf{y},{\mathit{P}}_d;\mathit{W})\right)$ represent the mean and standard deviation of the restored image, respectively. Similarly, ${\mathbf{x}}_n$ denotes the n-th pixel of the ground truth clean image, whereas $\overline{\mathbf{x}}$ and $\sigma \left(\mathbf{x}\right)$ are the mean and standard deviation of the ground truth clean image, respectively. The introduction of linear correlation prior can regulate the linear relationship between images. To realize the image restoration model $f(·;\mathit{W})$, we designed a novel Degradation-guided Histogram Transformer (DHformer) architecture, and its detailed network architecture is described in Section 3.3.

3.1.2. Modeling of Degradation Layer

To simulate the diverse degradation patterns induced by various adverse weather conditions, a degradation generation model is constructed to model the degradation layer $\mathcal{D}$, defined as

$$\begin{array}{c}\hfill \mathcal{D}=G(\mathit{z},{\mathit{P}}_d;\mathit{\theta }),\end{array}$$

(5)

where $\mathit{z}\in \mathcal{N}(0,\mathit{I})$ denotes a noise vector (i.e., latent variable), and $G(·;\mathit{\theta })$ represents a degradation generator parameterized by a deep neural network with parameters $\mathit{\theta }$. Specifically, we designed a Degradation-guided Convolutional Neural Network (DCNN) which takes the latent variable $\mathit{z}$ as input and adaptively predicts the weather-related degradation layers, with the guidance of weather degradation prompts. Similarly, its detailed network architecture is stated in Section 3.3.

3.2. Learning Guideline

3.2.1. Maximum A Posteriori Estimation

To optimize the aforementioned image restoration model $f(·;\mathit{W})$ and the degradation generation model $G(·;\mathit{\theta })$, we adopt Maximum A Posteriori (MAP) estimation as the learning guideline to estimate the model parameters $\mathit{W}$ and $\mathit{\theta }$, i.e.,

$$\begin{array}{cc}\hfill & logp(\mathit{W},\mathit{\theta }|\mathbf{y},{\mathit{P}}_d)=logp(\mathbf{y},{\mathit{P}}_d|\mathit{W},\mathit{\theta })+logp\left(\mathit{W}\right)+\mathrm{const}\hfill \\ \hfill & \triangleq \mathrm{\Omega }(\mathbf{y},{\mathit{P}}_d;\mathit{W},\mathit{\theta }),\hfill \end{array}$$

(6)

where $p(\mathbf{y},{\mathit{P}}_d|\mathit{W},\mathit{\theta })$ denotes the joint likelihood function of the observed degradation image $\mathbf{y}$ and its corresponding degradation-type prompt ${\mathit{P}}_d$, whose integral form can be expressed as

$$\begin{array}{cc}\hfill & p(\mathbf{y},{\mathit{P}}_d|\mathit{W},\mathit{\theta })=\int p(\mathbf{y},{\mathit{P}}_d|\mathit{W},\mathit{\theta },\mathit{z})p\left(\mathit{z}\right)d_{\mathit{z}}\hfill \\ \hfill & =\int \mathcal{N}\left(f(\mathbf{y},{\mathit{P}}_d;\mathit{W})+G(\mathit{z};\mathit{\theta }),\sigma \mathit{I}\right)p\left(\mathit{z}\right)d_{\mathit{z}}.\hfill \end{array}$$

(7)

Correspondingly, Equation (7) can be also reformulated into an expectation form as

$$\begin{array}{c}\hfill p(\mathbf{y},{\mathit{P}}_d|\mathit{W},\mathit{\theta })={\mathbb{E}}_{p\left(\mathit{z}\right)}\left[p(\mathbf{y},{\mathit{P}}_d,\mathit{W},\mathit{\theta },\mathit{z})\right].\end{array}$$

(8)

To achieve adverse weather removal through a semi-supervised learning framework, we seek to optimize the following objective function under both labeled and unlabeled data settings:

$$\begin{array}{c}\hfill \underset{\mathit{W},\mathit{\theta }}{max}logp(\mathit{W},\mathit{\theta }|\mathbf{y},{\mathit{P}}_d)=\underset{\mathit{W},\mathit{\theta }}{max}\sum _{\mathbf{y}\in \{\varphi ,\phi \}}\mathrm{\Omega }(\mathbf{y},{\mathit{P}}_d;\mathit{W},\mathit{\theta }),\end{array}$$

(9)

where $\varphi ={\{{\mathbf{y}}_k,{\mathbf{x}}_k\}}_{k=1}^{N_{\varphi }}$ and $\phi ={\left\{{\mathbf{y}}_k\right\}}_{k=1}^{N_{\phi }}$ denote a labeled dataset and an unlabeled dataset, respectively. ${\mathbf{y}}_k$ represents the k-th weather-degraded image, whereas ${\mathbf{x}}_k$ is corresponding ground-truth clean image.

3.2.2. Optimization Algorithm

In order to optimize the model parameters $\mathit{W}$ and $\mathit{\theta }$ in Equation (9), an alternative back-propagation technology, named Monte Carlo-based Expectation-Maximization (EM [56]) algorithm, is introduced to maximize $\mathrm{\Omega }(\mathit{W},\mathit{\theta })$. Specifically, in the Expectation step (E-step), the latent variable $\mathit{z}$ is sampled from the posterior distribution $p\left(\mathit{z}\right|\mathbf{y},{\mathit{P}}_d)$. With the inferred latent variable $\mathit{z}$, the model parameters $\mathit{W}$ and $\mathit{\theta }$ are subsequently updated in the Maximization step (M-step).

E-step: Let $\{{\mathit{W}}^{old},{\mathit{\theta }}^{old}\}$ denote the current model parameters and $p_{old}\left(\mathit{z}\right|\mathbf{y},{\mathit{P}}_d)$ is the posterior distribution under them. Then, the E-step samples the latent variable $\mathit{z}$ from $p_{old}\left(\mathit{z}\right|\mathbf{y},{\mathit{P}}_d)$ according to Langevin dynamics, which can be formulated as

$$\begin{array}{cc}\hfill & {\mathit{z}}^{(t+1)}={\mathit{z}}^{\left(t\right)}+{\displaystyle \frac{{\alpha }^2}{2}}\left[{\displaystyle \frac{\partial }{\partial \mathit{z}}}log{p}_{old}\left(\mathit{z}\right|\mathbf{y},{\mathit{P}}_d)\right]{|}_{\mathit{z}={\mathit{z}}^{\left(t\right)}}+\alpha {\mathit{\xi }}^{\left(t\right)}\hfill \\ \hfill & ={\mathit{z}}^{\left(t\right)}-{\displaystyle \frac{{\alpha }^2}{2}}\left[{\displaystyle \frac{\partial }{\partial \mathit{z}}}J\left(\mathit{z}\right)\right]{|}_{\mathit{z}={\mathit{z}}^{\left(t\right)}}+\alpha {\mathit{\xi }}^{\left(t\right)}\hfill \end{array}$$

(10)

where t indexes the time step in Langevin dynamics, whereas $\alpha$ is the step size. ${\mathit{\xi }}^{\left(t\right)}$ represents the Gaussian white noise, which is introduced to prevent falling into the local modes, and $J\left(\mathit{z}\right)$ is defined as

$$\begin{array}{c}\hfill J\left(\mathit{z}\right)={\displaystyle \frac{1}{2{\alpha }^2}}{∥\mathbf{y}-f(\mathbf{y},{\mathit{P}}_d;{\mathit{W}}^{old})-G(\mathit{z};{\mathit{\theta }}^{old})∥}_2+{\displaystyle \frac{1}{2}}{\parallel \mathit{z}\parallel }_2.\end{array}$$

(11)

In practice, to reduce the computational cost of Markov Chain Monte Carlo (MCMC) sampling, the update process in Equation (10) at each learning iteration is initialized from the previous sampling result, rather than starting from a new random state.

M-step: For convenience, the latent variable sampled in the E-step is denoted as $\widehat{\mathit{z}}$. The M-step aims to maximize the approximate upper bound with respect to the parameters $\mathit{W}$ and $\mathit{\theta }$, as follows:

$$\begin{array}{cc}\hfill & \underset{\mathit{W},\mathit{\theta }}{max}\mathrm{\Omega }(\mathbf{y},{\mathit{P}}_d;\mathit{W},\mathit{\theta })=\underset{\mathit{W},\mathit{\theta }}{max}\left\{\int p_{old}\left(\mathit{z}\right|\mathbf{y},{\mathit{P}}_d)logp(\mathbf{y},{\mathit{P}}_d,\mathit{z}|\mathit{W},\mathit{\theta })d_{\mathit{z}}+logp\left(\mathit{W}\right)\right\}\hfill \\ \hfill & \approx \underset{\mathit{W},\mathit{\theta }}{max}\left\{logp(\mathbf{y},{\mathit{P}}_d,\widehat{\mathit{z}}|\mathit{W},\mathit{\theta })+logp\left(\mathit{W}\right)\right\}.\hfill \end{array}$$

(12)

Since Maximum A Posteriori (MAP) estimation is equivalent to empirical risk minimization with regularization terms, Equation (12) can be further reformulated as the following minimization problem:

$$\begin{array}{cc}\hfill & \underset{\mathit{W},\mathit{\theta }}{min}\mathcal{L}(\mathit{W},\mathit{\theta })=\underset{\mathit{W},\mathit{\theta }}{min}\{{\displaystyle \frac{1}{2{\beta }^2}}{\parallel \mathbf{y}-f(\mathbf{y},{\mathit{P}}_d;\mathit{W})-G(\widehat{\mathit{z}};\mathit{\theta })\parallel }_2+\rho \sum _{i,j}{\mathit{\upsilon }}^T\gamma +1_{[\mathbf{y}\in \varphi ]}·{\displaystyle \frac{{\parallel f(\mathbf{y},{\mathit{P}}_d;\mathit{W})-\mathbf{x}\parallel }_2}{{\epsilon }_0}}\hfill \\ \hfill & +1_{[\mathbf{y}\in \varphi ]}·{\displaystyle \frac{1}{2}}\left(1-{\displaystyle \frac{{\sum }_{n=1}^{3HW}\left(f{(\mathbf{y},{\mathit{P}}_d;\mathit{W})}_n-\overline{f}(\mathbf{y},{\mathit{P}}_d;\mathit{W})\right)({\mathbf{x}}_n-\overline{\mathbf{x}})}{3HW\sigma \left(f(\mathbf{y},{\mathit{P}}_d;\mathit{W})\right)\sigma \left(\mathbf{x}\right)}}\right)\},\hfill \end{array}$$

(13)

To solve the optimization objective in Equation (13), the gradient descent based on back-propagation (BP) is employed to update the model parameters $\mathit{W}$ and $\mathit{\theta }$ as follows:

$$\begin{array}{c}\hfill \Theta \leftarrow \Theta -\eta {\displaystyle \frac{\partial }{\partial \Theta }}\mathcal{L}(\mathit{W},\mathit{\theta }),\Theta \in \{\mathit{W},\mathit{\theta }\},\end{array}$$

(14)

where $\eta$ is the step size. The optimization procedure of our proposed S²DAL framework is summarized in Algorithm 1.

Algorithm 1 Optimization procedure for S2DAL

Require:  Training data $\varphi ={\{{\mathbf{y}}_k,{\mathbf{x}}_k\}}_{k=1}^{N_{\varphi }}$ and $\phi ={\left\{{\mathbf{y}}_k\right\}}_{k=1}^{N_{\phi }}$, as well as the corresponding degradation-type prompts ${\mathit{P}}_d$. T iterations correspond to one epoch. Number of Langevin steps l. 1: Randomly initialize $\mathit{W}$ and $\mathit{\theta }$ 2: while not converged do 3:    for $j=1,2,\dots ,T$ do 4:       Sample a mini-batch data from $\varphi$ or $\phi$ 5:       E-step: For each data example ${\mathbf{y}}_k$, run l steps of Langevin dynamics to sample $\mathit{z}$ using Equation (10). 6:       M-step: Initialize $\mathit{W}$ and $\mathit{\theta }$ with those updated from the previous epoch, and then are further optimized using Equation (14). 7:    end for 8: end while 9: return the optimized weights $\{{\mathit{W}}^{*},{\mathit{\theta }}^{*}\}$

3.3. Network Architecture

The overall architecture of the proposed Semi-Supervised Degradation-Aware Learning (S²DAL) framework is illustrated in Figure 1, which is composed of the following two backbone networks.

3.3.1. Degradation-Guided Histogram Transformer

Since diverse weather-induced degradation usually exhibits different patterns, we develop a novel Degradation-guided Histogram Transformer (DHformer) to adaptively remove these weather-induced degradation patterns, according to the guidance of corresponding degradation-type prompts.

As shown in Figure 1, the proposed DHformer first extracts shallow features ${\mathit{F}}_0$ via a $3\times 3$ convolutional layer. Next, these features ${\mathit{F}}_0$ are fed into a 4-level encoder–decoder network to extract intricate features which are finally converted to the restored image $f(\mathbf{y},{\mathit{P}}_d;\mathit{W})$ via another $3\times 3$ convolutional layer. Within both the encoder and decoder, multiple well-designed Degradation-guided Histogram Transformer (DHT) blocks are designed to mine the inherent features of the clean image while reducing the interference of different weather-induced degradations. At each encoding stage, the proposed DHformer introduces a coarse skip connection to supplement the original input features. This branch is implemented through a sequence of operations, including average pooling, point-wise convolution, and depth-wise convolution. This setup enables the encoders to focus more on the weather-induced degradation factors. Furthermore, at each corresponding encoding–decoding stage, the encoder and decoder are linked via skip connections which enable the transfer of intermediate features across matching levels, thereby enhancing the training stability.

Degradation-guided Histogram Transformer Block. In order to fully mine the inherent features of clean images while reducing the interference of different weather-induced degradation patterns, we propose a Degradation-guided Histogram Transformer (DHT) block to build the DHformer. DHT block can adaptively regulate the feature space through channel shuffling modulation to better align with the shared parameter and network structure, thereby enhancing the all-in-one restoration performance. As shown in Figure 2, as the key component of DHformer, DHT block is composed of three pivotal modules, namely the Degradation-guided Channel Shuffling Module (DCSM), Dynamic-range Histogram Self-Attention (DHSA), and Dual-scale Gated Feed Forward (DGFF). These modules are organized to interact in conjunction with layer normalization, and the overall process can be formulated as follows:

$$\begin{array}{c}\hfill {\mathit{F}}_j=\mathrm{DCSM}\left({\mathit{F}}_{j-1}\right),\end{array}$$

(15)

$$\begin{array}{c}\hfill {\mathit{F}}_j={\mathit{F}}_j+\mathrm{DHSA}\left(\mathrm{LN}\left({\mathit{F}}_j\right)\right),\end{array}$$

(16)

$$\begin{array}{c}\hfill {\mathit{F}}_j={\mathit{F}}_j+\mathrm{DGFF}\left(\mathrm{LN}\left({\mathit{F}}_j\right)\right),\end{array}$$

(17)

where $\mathrm{LN}(·)$ denotes a layer normalization operation and ${\mathit{F}}_j$ represents the features at the j-th stage.

Regarding DCSM, its objective is to reorder the feature channels through a shuffling mechanism, with the guidance of weather degradation-type prompts. However, directly shuffling the channels on the features ${\mathit{F}}_{j-1}\in {\mathbb{R}}^{C\times H\times W}$ may make the model difficult to converge due to excessive perturbation [47], which finally impacts the restoration performance. To address this issue, we follow the same practice in [47], expanding the channel number of the image features ${\mathit{F}}_{j-1}\in {\mathbb{R}}^{C\times H\times W}$ to ${\widehat{\mathit{F}}}_{j-1}\in {\mathbb{R}}^{2C\times H\times W}$; this enables the feature channels to be shuffled in a higher-dimensional channel space through the Degradation Guidance Module (DGM). In DGM, the weather degradation-type prompts (i.e., ${\mathit{P}}_d\in {\mathbb{R}}^{C\times 1}$) are encoded to the corresponding index vector, which is used to indicate the order of feature channels in the shuffling process. To make the dimensionality of ${\mathit{P}}_d$ align with the channel dimensionality of ${\widehat{\mathit{F}}}_{j-1}$, a Multi-Layer Perceptron (MLP), which contains two linear layers, is used to convert the input prompts ${\mathit{P}}_d\in {\mathbb{R}}^{C\times 1}$ to ${\widehat{\mathit{P}}}_d\in {\mathbb{R}}^{2C\times 1}$ so as to achieve dimension matching with the number of channels in ${\widehat{\mathit{F}}}_{j-1}\in {\mathbb{R}}^{2C\times H\times W}$. In the channel shuffling stage, the index values in ${\widehat{\mathit{P}}}_d$ are utilized to reorder the feature channels of ${\widehat{\mathit{F}}}_{j-1}$. After shuffling, the number of channels in ${\widehat{\mathit{F}}}_{j-1}$ is reduced by half, aiming to maintain channel consistency before and after the transformation. The overall process of DCSM is formulated as follows:

$$\begin{array}{c}\hfill {\mathit{F}}_j={\mathrm{Conv}}_{1\times 1}^{\mathrm{d}}\left({\mathrm{CS}}_{topK}\left({\mathrm{Conv}}_{1\times 1}^{\mathrm{h}}\left({\mathit{F}}_{j-1}\right)\mid \mathrm{DGM}\left({\mathit{P}}_d\right)\right)\right),\end{array}$$

(18)

where ${\mathrm{Conv}}_{1\times 1}^{\mathrm{d}}(·)$ and ${\mathrm{Conv}}_{1\times 1}^{\mathrm{h}}(·)$ respectively denote a $1\times 1$ point-wise convolution to double and halve the feature channels, whereas ${\mathrm{CS}}_{topK}(·)$ is a top-K channel shuffling operation which is guided by the index values resulting from $\mathrm{DGM}\left({\mathit{P}}_d\right)$. By reordering the feature channels that are guided by degradation-type prompts, the adaptive shuffling strategy can effectively mitigate inter-channel interference while preserving the intrinsic image characteristics because there is no feature information loss during the channel shuffling process. Note that, to obtain degradation-type prompts, we introduce a pre-trained CLIP [50] model to encode the textual degradation type descriptions.

Regarding DHSA, its objective is to capture dynamically distributed weather-induced degradation. For that, this module comprises a dynamic-range convolution that re-arranges the spatial distribution of partial features, together with a dual-path histogram self-attention mechanism [18] that combines global and local dynamic feature aggregation. For the dynamic-range convolution, it divides the input features ${\mathit{F}}_j$ into two branches along the channel dimension, i.e., ${\mathit{F}}_j^1$ and ${\mathit{F}}_j^2$. For the first feature branch, sorting operations are conducted along both the horizontal and vertical directions, after which the sorted features are concatenated with those from the second branch. The resulting re-combined features are subsequently passed through a separable convolution layer. Accordingly, the entire calculation process of dynamic-range convolution can be formulated as follows:

$$\begin{array}{c}\hfill {\mathit{F}}_j^1,{\mathit{F}}_j^2=\mathrm{Split}\left({\mathit{F}}_j\right),{\mathit{F}}_j^1={\mathrm{Sort}}_{\mathrm{v}}\left({\mathrm{Sort}}_{\mathrm{h}}\left({\mathit{F}}_j^1\right)\right),\end{array}$$

(19)

$$\begin{array}{c}\hfill {\mathit{F}}_j={\mathrm{Conv}}_{3\times 3}^d\left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{Concat}({\mathit{F}}_j^1,{\mathit{F}}_j^2)\right)\right),\end{array}$$

(20)

where $\mathrm{Split}(·)$ represents an operation of splitting features along the channel dimension, and ${\mathrm{Sort}}_{i\in \mathrm{v},\mathrm{h}}(·)$ denotes an operation of sorting horizontally or vertically. ${\mathrm{Conv}}_{1\times 1}(·)$ is a $1\times 1$ point-wise convolution, and ${\mathrm{Conv}}_{3\times 3}^d(·)$ represents a $3\times 3$ depth-wise convolution. By arranging pixels with high and low intensities into structured patterns at the diagonal corners of the matrices, dynamic-range convolution enables convolutional operations to capture interactions across different dynamic ranges.

On the other hand, pixels corresponding to background content or weather degradations with varying intensities should be assigned different extents of attention. Considering this, a dual-path histogram self-attention is introduced to partition spatial elements into discrete bins and then adaptively assign varying attention within and across bins. Given the output of dynamic-range convolution, i.e., ${\mathit{F}}_j\in {\mathbb{R}}^{C\times H\times W}$, it is first decomposed into value features $\mathit{V}$ and two pairs of query–key, i.e., ${\mathit{F}}_j^{QK,1}$ and ${\mathit{F}}_j^{QK,2}$, which are subsequently fed into two branches. The sequence of $\mathit{V}$ is sorted, after which the corresponding query–key pairs are re-arranged according to the obtained indices as follows:

$$\begin{array}{c}\hfill \mathit{V},t=\mathrm{Sort}\left({\mathrm{R}}_{C\times H\times W}^{C\times HW}\left({\mathit{F}}_j\right)\right),\end{array}$$

(21)

$$\begin{array}{c}\hfill {\mathit{Q}}^1,{\mathit{K}}^1=\mathrm{Split}\left(\mathrm{Gather}({\mathrm{R}}_{C\times H\times W}^{C\times HW}\left({\mathit{F}}_j^{QK,1}\right),t)\right),\end{array}$$

(22)

$$\begin{array}{c}\hfill {\mathit{Q}}^2,{\mathit{K}}^2=\mathrm{Split}\left(\mathrm{Gather}({\mathrm{R}}_{C\times H\times W}^{C\times HW}\left({\mathit{F}}_j^{QK,2}\right),t)\right),\end{array}$$

(23)

where ${\mathrm{R}}_{C\times H\times W}^{C\times HW}(·)$ denotes an operation of reshaping features from ${\mathbb{R}}^{C\times H\times W}$ to ${\mathbb{R}}^{C\times HW}$, t is an index of sorted value, and $\mathrm{Gather}(·)$ represents an operation of retrieving elements. Furthermore, to capture both global and local information, two types of reshaping operations are defined, namely Bin-wise Histogram Reshaping (BHR) and Frequency-wise Histogram Reshaping (FHR). The number of bins in the first one is set to N, and each bin contains $HW/N$ elements. The frequency of each bin in the second one is equal to B, and the number of bins is $HW/N$. In this manner, BHR can capture large-scale contextual information, as each bin aggregates a substantial number of dynamically distributed pixels, whereas FHR facilitates the extraction of fine-grained details by grouping a small number of pixels with similar intensity values within each bin. The two query–key feature pairs are separately processed by two types of reshaping and subsequent self-attention operations, and their outputs are then multiplied element-wise to produce the final outcome. This procedure can be formulated as follows:

$$\begin{array}{c}\hfill {\mathit{M}}_B=\mathrm{softmax}\left({\displaystyle \frac{{\mathrm{R}}_B\left({\mathit{Q}}^1\right){\mathrm{R}}_B{\left({\mathit{K}}^1\right)}^T}{\sqrt{k}}}\right){\mathrm{R}}_B\left(\mathit{V}\right),\end{array}$$

(24)

$$\begin{array}{c}\hfill {\mathit{M}}_F=\mathrm{softmax}\left({\displaystyle \frac{{\mathrm{R}}_F\left({\mathit{Q}}^2\right){\mathrm{R}}_F{\left({\mathit{K}}^2\right)}^T}{\sqrt{k}}}\right){\mathrm{R}}_F\left(\mathit{V}\right),\end{array}$$

(25)

$$\begin{array}{c}\hfill \mathit{M}={\mathit{M}}_B\odot {\mathit{M}}_F,\end{array}$$

(26)

where k denotes the number of heads, ${\mathrm{R}}_B(·)$ and ${\mathrm{R}}_F(·)$ represent the reshaping operation of BHR and FHR, respectively, whereas ${\mathit{M}}_B$ and ${\mathit{M}}_F$ are the corresponding obtained attention maps.

Regarding DGFF, its objective is to pay more attention to the correlations among dynamically distributed weather-induced degradation. To achieve this, DGFF incorporates two complementary transmission pathways composed of multi-range and multi-scale depth-wise convolutions. Given the input of DGFF, i.e., ${\mathit{F}}_j\in {\mathbb{R}}^{C\times H\times W}$, a $1\times 1$ point-wise convolution is first applied to expand the channel dimension. And then, the resulting augmentation features are fed into two parallel branches for subsequent processing. During feature transformation, $5\times 5$ and dilated $3\times 3$ depth-wise convolutions are employed to strengthen the extraction of multi-range and multi-scale representations. Following the gating mechanism in [57], the output of the second branch serves as a gating map for the first branch. Accordingly, the overall feature fusion process within the DGFF module can be formulated as follows:

$$\begin{array}{c}\hfill {\mathit{F}}_{j,1},{\mathit{F}}_{j,2}=\mathrm{Split}\left(\mathrm{Shuffle}\left({\mathrm{Conv}}_{1\times 1}\left({\mathit{F}}_j\right)\right)\right),\end{array}$$

(27)

$$\begin{array}{c}\hfill {\mathit{F}}_{j,1}={\mathrm{Conv}}_{5\times 5}^{\mathrm{d}}\left({\mathit{F}}_{j,1}\right),{\mathit{F}}_{j,2}={\mathrm{Conv}}_{3\times 3}^{\mathrm{d},\mathrm{dilated}}\left({\mathit{F}}_{j,2}\right),\end{array}$$

(28)

$$\begin{array}{c}\hfill {\mathit{F}}_j={\mathrm{Conv}}_{1\times 1}\left(\mathrm{Unshuffle}(\mathrm{Mish}\left({\mathit{F}}_{j,2}\right)\odot {\mathit{F}}_{j,1})\right),\end{array}$$

(29)

where ${\mathrm{Conv}}_{5\times 5}^{\mathrm{d}}(·)$ denotes a $5\times 5$ depth-wise convolution, and ${\mathrm{Conv}}_{3\times 3}^{\mathrm{d},\mathrm{dilated}}(·)$ is a $3\times 3$ dilated depth-wise convolution. $\mathrm{Shuffle}(·)$ and $\mathrm{Unshuffle}(·)$ denote an operation of pixel-shuffling and unshuffling, respectively. $\mathrm{Mish}(·)$ represents a Mish activation [58].

3.3.2. Degradation-Guided Convolutional Neural Network

The DCNN plays a crucial role in our proposed Semi-Supervised Degradation-Aware Learning (S²DAL) framework, as it directly impacts the adverse weather removal performance. As shown in Figure 1, given a latent variable $\mathit{z}\in \mathcal{N}(0,\mathit{I})$, its dimensionality is first elevated by a fully connected layer with ReLU activation, i.e., from ${\mathbb{R}}^{1\times L}$ to ${\mathbb{R}}^{1\times 16L}$, where $L=256$. Next, a reshaping operation is conducted to reshape the output of fully connected layer from ${\mathbb{R}}^{1\times 16L}$ to ${\mathbb{R}}^{1\times [{\displaystyle \frac{H}{r}}]\times [{\displaystyle \frac{W}{r}}]}$, where $r=1$, and $H=W=64$, obtaining the corresponding feature tensors $\mathit{T}$. The process can be formulated as the following expression:

$$\begin{array}{c}\hfill \mathit{T}={\mathrm{R}}_T\left(\mathrm{ReLU}\left(\mathrm{FC}\left(\mathit{z}\right)\right)\right),\end{array}$$

(30)

where $\mathrm{FC}(·)$ is a fully connected layer, $\mathrm{ReLU}(·)$ denotes a Rectified Linear Unit, and ${\mathrm{R}}_T(·)$ represents a reshaping operation.

Afterwards, the obtained feature tensors $\mathit{T}$ undergo a shallow convolutional network, which is mainly composed of five $3\times 3$ convolutional layers and two Degradation-guided Shuffling Modules (DCSMs). To facilitate the recovery process, the encoding features from the first and second convolutions are concatenated with the decoding features of the third and fourth convolutions via skip connections, and the DCSM is also inserted between the encoder and decoder, and the encoding feature channels are also shuffled under the guidance of weather degradation-type prompts. By incorporating DCSM, the generation of weather-induced degradation layers can be more effectively disentangled, thereby facilitating the generation of more realistic degradation layers. More details of the network configuration for DCNN can be found in

Table 1. The network configuration of the DCNN.

Types	Filter Sizes	Neuron/Filter Number	Stride	Padding	Output Sizes
Input ($\mathit{z}$)	-	-	-	-	$1\times 256$
FC	-	4096	-	-	$1\times 4096$
ReLU	-	-	-	-	$1\times 4096$
Reshape	-	-	-	-	$1\times 64\times 64$
Convolution	$3\times 3$	128	1	1	$128\times 64\times 64$
ReLU	-	-	-	-	$128\times 64\times 64$
Convolution	$3\times 3$	256	1	1	$256\times 64\times 64$
ReLU	-	-	-	-	$256\times 64\times 64$
DCSM	-	-	-	-	$256\times 64\times 64$
Convolution	3 × 3	128	1	1	$128\times 64\times 64$
ReLU	-	-	-	-	$128\times 64\times 64$
DCSM	-	-	-	-	$128\times 64\times 64$
Convolution	3 × 3	512	1	1	$512\times 64\times 64$
ReLU	-	-	-	-	$512\times 64\times 64$
Upsampling	-	-	-	-	$512\times 128\times 128$
Convolution	3 × 3	3	1	1	$3\times 128\times 128$
ReLU	-	-	-	-	$3\times 128\times 128$

.

4. Experiments

4.1. Datasets

To evaluate the performance, we follow prior works [15,18] and employ the curated AllWeather dataset, which is constructed by combining subsets of training images from Snow100K [59], Outdoor-Rain [60], and RainDrop [61]. The Allweather dataset provides a balanced training set across three adverse weather conditions, including snow, heavy rain with haze, and raindrops. Specifically, the training set comprises 9000 images from Snow100K [59], 818 images from RainDrop [61], and 8250 images from Outdoor-Rain [60]. Snow100K consists of synthetic images degraded by snow, whereas RainDrop contains images affected by raindrops. Outdoor-Rain provides synthetic images simultaneously corrupted by rain streaks and haze. For performance evaluation, we utilized the Test1 dataset [60] with 750 rain-haze images, the RainDrop [61] test set with 58 raindrop images, and the Snow100K-L and Snow100-S test sets [59] with 16801 large snow images and 16611 small snow images, respectively. The sample quantity as well as the corresponding degradation-type prompts for these three datasets are shown in

Table 2. The sample quantity as well as the degradation-type prompts for Snow100K [59], Outdoor-Rain [60], and RainDrop [61], respectively.

Dataset	Training Set	Test Set	Degradation-Type Prompts
Snow100K	9000	33,412	Snow degradation by normal snowflakes
Outdoor-Rain	8250	750	Rain degradation by rain streaks and normal haze
RainDrop	818	58	Rain degradation by normal raindrops
Total	18,068	34,220	-

.

Quantitative evaluations between ground-truth and restored images were conducted using the conventional Peak Signal-to-Noise Ratio (PSNR [62]) and Structural SIMilarity (SSIM [63]), where higher values generally denote superior performance. In addition, two no-reference metrics were adopted to assess real-world restoration quality, namely the Natural Image Quality Evaluator (NIQE [64]) and the feature-enriched completely blind image quality evaluator (IL-NIQE [65]). Lower NIQE and IL-NIQE scores indicate better perceptual quality of the restored images.

4.2. Implementation Details

To train the proposed Semi-Supervised Degradation-Aware Learning S²DAL framework, we randomly split the training samples from Snow100K [59], RainDrop [60], and Outdoor-Rain [61] into labeled and unlabeled data at a ratio of 1:1. Specifically, regrading the training set of Snow100K [59], 4500 snowy/clean image pairs were randomly selected as the labeled data and the remaining 4500 snowy images as the unlabeled data. Likewise, for the training sets of RainDrop [60] and Outdoor-Rain [61], half of the paired degraded/clean images from their corresponding training sets were selected as the labeled data, and the remaining degraded ones as the unlabeled data. The implementation of our S²DAL uses the PyTorch (v1.12.1) deep learning framework on a single GPU of NVIDIA RTX A6000. The proposed S²DAL was trained in an end-to-end manner, where the Adam [66] solver is applied to optimize the model parameters in the M-step of the Monte Carlo-based EM algorithm [56]. The initial learning rates for the image restoration model, i.e., DHformer, and the degradation generative model, i.e., DCNN, were set to $1\times {10}^{-4}$ and $1\times {10}^{-3}$, respectively, and decayed by a factor of 10 after 20 epochs. The mini-batch was set to 10, and each image was randomly cropped into $128\times 128$ pixels. Note that to make the training more stable, we only optimized the parameter $\mathit{W}$ of the image restoration model for the first 2 epochs and then jointly optimized all network parameters, including the parameters $\mathit{\theta }$ in DCNN. As for the hyper-parameters, we set ${\epsilon }_0=1\times {10}^{-6}$, $\rho =0.1$, and $\gamma ={[1,1]}^T$.

Note that, due to limited model capacity, it is difficult to accurately fit the degradation layers across the entire dataset using only a single generator model $G(·;\mathit{\theta })$ defined in Equation (5). To address this issue, we followed the same practice in [55], employing a considerable number of degradation generators with the identical network architecture, where each generator $G(·;\mathit{\theta })$ is assigned to a specific mini-batch for degradation layer generation. Under this configuration, each degradation generator is only responsible for modeling the degradation layers of its corresponding mini-batch data, thereby avoiding the difficulty of modeling all degradation layers across the entire dataset. During the training procedure, when a mini-batch is revisited in a new epoch, the corresponding degradation generator $G(·;\mathit{\theta })$ is initialized using the parameters updated from the previous epoch, and then is further optimized based on Equation (14). Therefore, the parameter evolution of each generator $G(·;\mathit{\theta })$ remains continuous throughout the iterative optimization process, which is consistent with the iterative refinement mechanism of the EM algorithm. Moreover, the storage overhead of $G(·;\mathit{\theta })$ for each mini-batch during the training procedure is $13.55$ MB.

4.3. Comparisons with the State of the Art

To demonstrate the effectiveness and superiority of our method, the proposed S²DAL framework is compared with multiple state-of-the-art semi-supervised approaches. The representative semi-supervised approaches used for comparison include SnowMaster [67], SAT-UIR [68], DNDM [69], SSID-KD [70], Muss [71], and SemiDDM-weather [27]. Furthermore, we also perform a performance comparison with two existing fully supervised all-in-one methods for adverse weather removal, including All-in-One network [14] and TransWeather [15]. Note that, to make a fair comparison, all these competing methods were trained to handle raindrops, snow, as well as heavy rain and haze removal using a single model.

Table 3. Quantitative comparisons on three adverse weather removal tasks are reported in terms of PSNR and SSIM. The top halves of the tables show the results of fully supervised all-in-one image restoration models, while the bottom halves of the tables report the results of semi-supervised methods. The best and second-best results are marked in bold and underlined, respectively.

(a) RainDrop removal
Method	RainDrop
	PSNR ↑	SSIM ↑
All-in-One [14]	31.12	0.927
TransWeather [15]	30.17	0.916
SemiDDM-weather [27]	20.65	0.596
Muss [71]	23.19	0.787
SSID-KD [70]	21.12	0.813
DNDM [69]	24.07	0.840
SAT-UIR [68]	26.86	0.873
SnowMaster [67]	28.53	0.908
S2DAL (Ours)	31.35	0.935
(b) De-raining & De-hazing
Method	Outdoor-Rain
	PSNR ↑	SSIM ↑
All-in-One [14]	24.71	0.898
TransWeather [15]	28.83	0.900
SemiDDM-weather [27]	22.33	0.786
Muss [71]	17.98	0.603
SSID-KD [70]	18.83	0.724
DNDM [69]	17.09	0.705
SAT-UIR [68]	24.19	0.852
SnowMaster [67]	24.08	0.860
S2DAL (Ours)	28.13	0.916
(c) De-snowing
Method	Snow100K-S		Snow100K-L
	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑
All-in-One [14]	-	-	28.33	0.882
TransWeather [15]	32.51	0.934	29.31	0.888
SemiDDM-weather [27]	23.84	0.780	23.84	0.780
Muss [71]	24.54	0.775	21.27	0.667
SSID-KD [70]	28.68	0.872	24.30	0.782
DNDM [69]	29.79	0.880	24.13	0.782
SAT-UIR [68]	30.35	0.906	25.63	0.827
SnowMaster [67]	32.74	0.948	27.72	0.901
S2DAL (Ours)	34.90	0.951	29.45	0.905

presents the quantitative evaluation of different methods on the test sets of RainDrop [61], Outdoor-Rain [61], and Snow100K [59]. It can be clearly observed that the proposed S²DAL presents superior restoration performance compared to all competing methods on three adverse weather removal tasks. Particularly, for raindrop removal, the proposed S²DAL is superior to SnowMaster [67] (semi-supervised method) and TransWeather [15] (fully supervised method) by $2.82$ dB$/0.027$ and $1.18$ dB$/0.016$ in terms of PSNR and SSIM. For snow removal as well as heavy rain and haze removal tasks, the proposed S²DAL still yields the best results (i.e., 28.13 dB/0.916 on Outdoor-Rain, $34.90$ dB$/0.951$ on Snow100K-S, and $29.29$ dB$/0.905$ on Snow100K-L) compared to the semi-supervised methods. Although the proposed S²DAL obtains slightly lower PSNR than that of TransWeather [15], it achieves higher SSIM and still competitive PSNR on the Outdoor-Rain [60] test set. More importantly, compared with fully supervised methods including TransWeather [15], our proposed S²DAL can significantly reduce the dependence on large-scale paired labeled data, demonstrating its effectiveness and practical applicability in labeled data-scarce scenarios.

Besides the superiority in quantitative evaluation, Figure 3 depicts some visualizations of raindrop removal by different methods. As can be seen, although those competing methods, i.e., SemiDDM-weather [27], SnowMaster [67], and TransWeather [15], are capable of effectively removing raindrops, they leave behind artifacts in the corresponding removal regions (see the enlarged areas), resulting in undesirable visual effects. In contrast, the proposed S²DAL not only removes raindrops more thoroughly but also avoids leaving behind artifacts caused by the removal process, thereby achieving a superior restoration of the background information. Figure 4 depicts the visual results of different methods for rain and haze removal. As illustrated, TransWeather [15] achieves better visual quality compared to SemiDDM-weather [27] and SnowMaster [67], which appear to have under-de-raining issues, leaving noticeable rain streaks in their restored images. Despite that, the image restored by our method still presents the superior visual effect (see the enlarged areas), compared to that of TransWeather [15]. Figure 5 shows the visualizations of snow removal by different methods. We can see that the competing methods invariably leave behind undesirable artifacts (see the enlarged blue-sky regions) after snow removal, whereas the de-snowed image delivered by our S²DAL is much closer to the Ground Truth, yielding a visually more natural result.

4.4. Comparison on Real-World Images

To demonstrate the practicality, we conduct real-world adverse weather removal on Snow100K-Real [59] and RainDS [72] test sets. Among them, the RainDS [72] test set includes 97 real raindrop test images, whereas the Snow100K-Real [59] test set contains 1329 real-world snowy images. We compare our S²DAL with the recent SemiDDM-weather [27], as both methods are specifically designed for all-in-one weather-degraded image restoration via a semi-supervised learning framework. Figure 6 presents their qualitative results for raindrop removal from real images. Note that these two compared methods were first trained on the AllWeather dataset, and then were applied to restore vision from real-world weather-degraded images. One can see that the SemiDDM-weather [27] method fails to adequately remove real raindrops, resulting in noticeable artifacts within raindrop-contaminated regions. In contrast, the proposed method can more thoroughly eliminate raindrops in the line of sight, thereby producing a more natural visual result. Additionally, Figure 7 presents their qualitative comparisons for snow removal of selected images with heavy snow from Snow100K-Real [59]. As can be seen, although the SemiDDM-weather [27] can remove snowflakes from real images to a certain extent, its restoration effect is still inferior to that of the proposed S²DAL method. Considering that mixed-weather conditions, such as sleet accompanied by haze, may also occur in real-world scenarios, we thus conducted the fixed-weather removal by applying SemiDDM-weather [27] and the proposed S²DAL on an image of sleet accompanied by haze selected from the Snow100K-Real [59] test set. The corresponding visual results are shown in Figure 8. As can be seen, compared with SemiDDM-weather [27], the proposed S²DAL can more effectively remove the sleet and haze in the image. This demonstrates that the proposed S²DAL possesses stronger all-in-one removal capability under complex mixed-weather scenarios.

Finally, the quantitative results in terms of NIQE and IL-NIQE scores between SemiDDM-weather [27] and the proposed S²DAL are reported in

Table 4. Quantitative NIQE and IL-NIQE comparisons between SemiDDM-weather [27] and S²DAL on Snow100K-Real [59] and RainDS [72]. Lower is better.

Method	Snow100K-Real		RainDS
	NIQE ↓	IL-NIQE ↓	NIQE ↓	IL-NIQE ↓
SemiDDM-weather [27]	3.021	21.939	4.286	23.708
S2DAL (Ours)	2.925	21.656	3.621	21.261

. The quantitative results show that our S²DAL yields better perceptual image quality scores on both Snow100K-Real [59] and RainDS [72] test sets, and significantly outperforms the state-of-the-art semi-supervised all-in-one adverse weather removal method, i.e., SemiDDM-weather [27].

4.5. Ablation Studies

We conducted several ablation experiments on the Outdoor-Rain [60] dataset to further validate the effectiveness of our proposed Semi-Supervised Degradation-Aware Learning (S²DAL) method.

A key component in the proposed S²DAL framework is the Degradation-guided Histogram Transformer (DHT) block, which consists of three modules: Degradation-guided Channel Shuffling Module (DCSM), Dynamic-range Histogram Self-Attention (DHSA), and Dual-scale Gated Feed Forward (DGFF). To analyze the impact of these three modules for adverse weather removal,

Table 5. Ablation results on the Outdoor-Rain [60] test set after removing different components from the proposed S²DAL. The best and second-best results are marked in bold and underlined, respectively.

Method	DCSM †	DCSM ‡	DHSA	DGFF	PSNR ↑	SSIM ↑
(a)	×	×	✓	✓	23.68	0.852
(b)	×	✓	✓	✓	26.49	0.876
(c)	✓	×	✓	✓	25.78	0.886
(d)	✓	✓	×	✓	24.13	0.858
(e)	✓	✓	✓	×	27.46	0.904
S2DAL	✓	✓	✓	✓	28.13	0.916

^† DCSM within the DHT block. ^‡ DCSM within the DCNN network.

illustrates their quantitative results on the test set of Outdoor-Rain [60]. As can be seen, simultaneously removing the DCSM module from both the DHT block and the DCNN network (Method (a)) leads to the lowest PSNR and SSIM scores. This result validates the critical role of the DCSM module in the proposed S²DAL framework, which can adaptively modulate feature representations according to different degradation-type prompts. Additionally, by removing the DHSA module (Method (d)), the performance of S²DAL also reduces significantly, which is because removing the DHSA module may weaken the ability of S²DAL to extract long-range spatial features, leading to poor restoration performance. Finally, removing the DGFF module (Method (e)), the performance in terms of PSNR and SSIM also drops to a certain extent. This could be due to the fact that, after removing the DGFF module, our proposed S²DAL fails to focus on the correlation among dynamically distributed weather-induced degradation, leading to insufficient feature interaction and sub-optimal image restoration.

On the other hand, in order to demonstrate the contribution of the shuffling mechanism,

Table 6. Comparison of PSNR and SSIM results on the Outdoor-Rain [60] test set, with and without (w/o) using a top-K channel shuffling operation ${\mathrm{CS}}_{topK}(·)$ in DCSM. The best and second-best results are marked in bold and underlined, respectively.

Method	PSNR ↑	SSIM ↑
${\mathrm{CS}}_{topK}(·)$(w/o)	24.73	0.862
${\mathrm{CS}}_{topK}(·)$ + RSI	25.41	0.895
${\mathrm{CS}}_{topK}(·)$ + DSI	28.13	0.916

and Figure 9 respectively show the quantitative and qualitative results with and without (w/o) using a top-K channel shuffling operation ${\mathrm{CS}}_{topK}(·)$ in the DCSM. Note that not using the shuffling mechanism means the top-K channel shuffling operation ${\mathrm{CS}}_{topK}(·)$ is disabled in Equation (18) which formulates the overall process of DCSM. In addition, Table 6 also compares the performance between ${\mathrm{CS}}_{topK}(·)$ using the Randomly Shuffling Index (RSI) and ${\mathrm{CS}}_{topK}(·)$ using Degradation-guided Shuffling Index (DSI). From Table 6 and Figure 9, we can observe that the performance of ${\mathrm{CS}}_{topK}(·)$ + RSI and ${\mathrm{CS}}_{topK}(·)$ + DSI consistently outperforms that of ${\mathrm{CS}}_{topK}(·)$ (w/o), which shows that the channel shuffling operation contributes to improving the restoration performance, as channel shuffling could alleviate inter-channel interference while preserving the intrinsic image characteristics. Moreover, the performance of ${\mathrm{CS}}_{topK}(·)$ + DSI also exceeds that of ${\mathrm{CS}}_{topK}(·)$ + RSI; this implies that channel shuffling guided by the degradation-guided shuffling index can make the deep features more discriminative and representative for recovering cleaner images, compared with channel shuffling using the randomly shuffling index.

4.6. Discussion

Although our proposed semi-supervised framework S²DAL can achieve impressive results for all-in-one weather-degraded image restoration, there still exist some limitations. First, as stated previously, the proposed S²DAL requires a considerable number of degradation generators with the identical network architecture to model the degradation layers for different mini-batch data; this will inevitably increase storage requirements during the training process. Second, when weather-induced degradations are extremely dense, complex, or highly coupled, some fine structural details and texture information may be irreversibly corrupted in the input images, especially for the real-world degraded ones shown in Figure 6, Figure 7 and Figure 8. As can be seen, although the proposed S²DAL helps alleviate this challenge and achieves better visual effects compared to SemiDDM-weather [27], accurately recovering those regions severely corrupted by severe weather remains difficult. Third, the proposed S²DAL belongs to a semi-supervised learning method, which implies that a limited amount of manually labeled data is still required to impose ground-truth supervision for image restoration. Self-learning methods (e.g., self-supervised learning) have recently achieved significant progress in many vision-related tasks [73,74] due to their ability to leverage large-scale unlabeled data without requiring expensive manually labeled data. To address these limitations, future efforts will focus on lightweight model design, fully self-supervised restoration for weather-degraded images, and performance optimization for mixed and complex degradation scenarios.

5. Conclusions

In this paper, we proposed a novel semi-supervised learning framework, termed Semi-Supervised Degradation-Aware Learning (S²DAL), for all-in-one adverse weather image restoration under limited paired data. Our proposed S²DAL is a dual-backbone architecture, consisting of a Degradation-guided Histogram Transformer (DHformer) for weather-degraded image restoration and a Degradation-guided Histogram Transformer (DCNN) for degradation layer generation. In particular, we designed a Degradation-guided Histogram Transformer (DHT) block to construct the DHformer, which can adaptively regulate the feature space through channel shuffling modulation to better align with the shared parameter and network structure. Furthermore, we also introduced a Monte Carlo-based EM optimization algorithm to allow for the joint estimation of latent variables and network parameters under both labeled and unlabeled data. Extensive experimental results on both synthetic and real-world datasets demonstrate that S²DAL consistently outperforms multiple existing state-of-the-art methods.