SGM-EMA: Speech Enhancement Method Score-Based Diffusion Model and EMA Mechanism

Abstract

The score-based diffusion model has made significant progress in the field of computer vision, surpassing the performance of generative models, such as variational autoencoders, and has been extended to applications such as speech enhancement and recognition. This paper proposes a U-Net architecture using a score-based diffusion model and an efficient multi-scale attention mechanism (EMA) for the speech enhancement task. The model leverages the symmetric structure of U-Net to extract speech features and captures contextual information and local details across different scales using the EMA mechanism, improving speech quality in noisy environments. We evaluate the method on the VoiceBank-DEMAND (VB-DMD) dataset and the DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus–TUT Sound Events 2017 (TIMIT-TUT) dataset. The experimental results show that the proposed model performed well in terms of speech quality perception (PESQ), extended short-time objective intelligibility (ESTOI), and scale-invariant signal-to-distortion ratio (SI-SDR). Especially when processing out-of-dataset noisy speech, the proposed method achieved excellent speech enhancement results compared to other methods, demonstrating the model’s strong generalization capability. We also conducted an ablation study on the SDE solver and the EMA mechanism, and the results show that the reverse diffusion method outperformed the Euler–Maruyama method, and the EMA strategy could improve the model performance. The results demonstrate the effectiveness of these two techniques in our system. Nevertheless, since the model is specifically designed for Gaussian noise, its performance under non-Gaussian or complex noise conditions may be limited.

1. Introduction

The core goal of speech enhancement is to suppress background noise and improve the quality and intelligibility of noisy speech signals. Traditional speech enhancement methods are typically divided into time–frequency (T-F) domain filtering methods and statistical model-based methods. Wiener filtering and the spectral minimum mean square error (MMSE) estimator proposed by Ephraim and Malah [1] are pioneering works in the field of statistical modeling, laying the foundation for this area. Recent advancements in generalized kernel methods, including the works of Soares et al. [2] and May et al. [3], have also brought new progress to the field, further expanding the toolkit for speech enhancement and recognition tasks. T-F domain filtering methods include spectral subtraction [4,5,6,7], subspace methods [8,9], Wiener filtering [10,11,12,13,14], and minimum mean square error estimation [15,16]. These methods usually assume that the noise is stationary and achieve noise reduction by attenuating the noise spectrum. However, real-world noise lacks structured patterns, unlike synthetically generated non-stationary noise, which may still exhibit regular behavior. Another category of methods is based on statistical models, such as hidden Markov models [17] and Gaussian mixture models [18], which rely on specific acoustic assumptions and probabilistic model structures. However, in real scenarios, noise often exhibits variability, making acoustic assumptions about noise difficult and thereby limiting the denoising performance.

Unlike traditional methods, deep learning [19,20,21,22] techniques can automatically learn complex patterns from large-scale datasets to achieve better speech enhancement performance. Deep learning models are generally categorized into discriminant models (D) and generative models (G). The core idea of the discriminant model is to learn a mapping from noisy speech to clean speech, typically using time domain or frequency domain methods [23,24,25,26,27], complex spectrum mapping [28], or directly operating in the time domain [29,30,31]. In contrast, generative models learn the underlying statistical properties of clean speech data, enabling them to perform well even when the training and test datasets differ. Typical generative models include approaches such as generative adversarial networks [32,33,34] and variational autoencoders [35,36].

Many areas have been studied regarding diffusion models. Zhang et al. [37] proposed a method for restoring degraded speech using an improved diffusion model, modifying the DiffWave architecture to recover the original speech signal better. Joan Serrà et al. [38] proposed a generative model that combines score-based diffusion with a multi-resolution conditioning network enhanced by a mixture density network, capable of handling 55 different types of distortion simultaneously. Julius Richter et al. [39] proposed an audiovisual speech enhancement system that leverages a score-based diffusion model informed by visual cues and uses audiovisual embeddings derived from a self-supervised learning model fine-tuned for lip reading. Their system improves speech quality and reduces generation artifacts, such as speech confusion. Yang et al. [40] proposed a unified speech enhancement and editing model using conditional diffusion models to handle various tasks in a generative manner. Based on these developments, this paper adopts a score-based diffusion model, which consists of a forward process that gradually corrupts clean speech with noise and a reverse process that starts from the noisy input to iteratively estimate the original signal. However, diffusion-based methods may encounter challenges in generalizing to unfamiliar conditions. To address this problem, we adopt a score-based diffusion model integrated with the EMA mechanism. This approach effectively captures multi-level information and utilizes a U-Net structure for speech restoration.

The main contributions of this study are as follows:

• We pay special attention to the score-based diffusion model and the multi-resolution U-Net model and make corresponding modifications to the U-Net structure to better handle speech enhancement tasks.
• We introduce an EMA mechanism that captures both multi-scale contextual information and local details, thereby enhancing the performance of speech enhancement.
• To verify the effectiveness of our proposed method, we conducted experiments on the VB-DMD and TIMIT-TUT datasets. The results show that our SGM-EMA method performed well on the speech enhancement task.
• We also conducted ablation studies to evaluate the contribution of the method in the predictor of the PC sampler and the EMA mechanism to the overall model performance.

2. Methodology

2.1. Diffusion Process Based on SDE

In speech enhancement tasks, clean speech is defined as the original speech signal, which includes only the speaker’s voice and excludes any external interference sources. Noisy speech refers to a speech signal that is mixed with other sound sources in addition to the speaker’s voice. The diffusion process consists of two stages: the forward process and the reverse process, as shown in Figure 1.

(1)

Forward process: The forward process is implemented by gradually adding Gaussian noise to clean speech. However, as the model is specifically designed for Gaussian noise, it may not generalize well to non-Gaussian or more complex real-world noise conditions. Following the approach of Song et al. [41], we designed a random diffusion process ${\left\{{\mathbf{x}}_t\right\}}_{t=0}^{\mathrm{T}}$, which is the solution to the following linear stochastic differential equation (SDE):

$$\mathrm{d}{\mathbf{x}}_t=\mathbf{f}\left({\mathbf{x}}_t,\mathbf{y}\right)\mathrm{d}\mathrm{t}+g\left(t\right)\mathrm{d}\mathbf{w},$$

(1)

where ${\mathbf{x}}_t$ represents the current speech state, $t\in \left[0,\mathrm{T}\right]$ is a continuous time step variable describing the process progress, $\mathbf{y}$ is the noisy or reverberant speech, $\mathbf{w}$ is the standard Wiener process, ${\mathbf{x}}_0$ represents the clean speech, and ${\mathbf{x}}_{\mathrm{T}}$ represents the Gaussian distribution centered on the noisy speech. We incorporated the noisy speech $\mathbf{y}$ into the SDE by modifying the drift coefficient to $\mathbf{f}\left({\mathbf{x}}_t,\mathbf{y}\right)≔\mathsf{\gamma }\left(\mathbf{y}-{\mathbf{x}}_t\right)$, where $\mathsf{\gamma }$ is a constant, called stiffness, which controls the transition from ${\mathbf{x}}_0$ to $\mathbf{y}$. The diffusion coefficient $g\left(\mathrm{t}\right)$ controls the amount of Gaussian white noise injected at each time step and is defined as follows:

$$g\left(t\right)≔{\sigma }_{min}{\left({\sigma }_{max}/{\sigma }_{min}\right)}^t\sqrt{2log\left({\sigma }_{max}/{\sigma }_{min}\right),}$$

(2)

where ${\sigma }_{min}$ and ${\sigma }_{max}$ are parameters that define the noise schedule of the Wiener process.

(2)

Reverse process: The reverse process gradually removes noise from the noisy signal and estimates a signal close to the original clean speech. According to Song et al. [41] and Anderson et al. [42], the SDE in Equation (1) has a related reverse SDE. This is achieved by solving the following differential equation:

$$d{\mathbf{x}}_t=\left[g{\left(t\right)}^2{\nabla }_{{\mathbf{x}}_t}\log p_t\left({\mathbf{x}}_t|\mathbf{y}\right)-\mathbf{f}\left({\mathbf{x}}_t,\mathbf{y}\right)\right]dt+g\left(t\right)d\overline{\mathit{w}},$$

(3)

where $\overline{\mathit{w}}$ is the standard Wiener process evolving backward in time, and the score function ${\nabla }_{{\mathbf{x}}_t}{\log }_{\mathit{pt}}\left({\mathbf{x}}_t|\mathbf{y}\right)$ is a term approximated by the DNN, called the score model. We denote the score model as $S_{\theta }\left({\mathbf{x}}_t,\mathbf{y},t\right)$, and by substituting the score model into the reverse SDE in Equation (3), we obtain the so-called plug-in reverse SDE:

$$d{\mathbf{x}}_t=\left[g{\left(t\right)}^2{S}_{\theta }\left({\mathbf{x}}_t,\mathbf{y},t\right)-\mathbf{f}\left({\mathbf{x}}_t,\mathbf{y}\right)\right]dt+g\left(t\right)d\overline{\mathit{w}},$$

(4)

Sampling is then performed as follows:

$${\mathbf{x}}_T\sim {\mathcal{N}}_{\mathbb{C}}\left({\mathbf{x}}_T;\mathbf{y},\mathsf{\sigma }{\left(T\right)}^2\mathbf{I}\right),$$

(5)

where ${\mathbf{x}}_T$ is the severely corrupted data distribution of the noisy speech $\mathbf{y}$, $\mathbf{I}$ is the identity matrix, and ${\mathcal{N}}_{\mathbb{C}}$ is the circularly symmetric complex normal distribution. Once the score model has been trained, the reverse SDE defined in Equation (4) can be used to iteratively estimate clean speech using the predictor–corrector sampling algorithm proposed by Song et al. [41].

(3)

Training objective: The objective function for training the score model $S_{\theta }\left({\mathbf{x}}_t,\mathbf{y},t\right)$ is derived as follows. Based on Equation (1), the forward diffusion process defines a Gaussian process, allowing the mean and variance of the process state ${\mathbf{x}}_t$ to be determined when the initial conditions are known [43]. Consequently, this allows for direct sampling of ${\mathbf{x}}_t$ at any time step $t$ given ${\mathbf{x}}_0$ and $\mathbf{y}$ by using the perturbation kernel:

$$p_{0t}\left({\mathbf{x}}_t|{\mathbf{x}}_0,\mathbf{y}\right)={\mathcal{N}}_{\mathbb{C}}\left({\mathbf{x}}_t;\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right),\mathsf{\sigma }{\left(t\right)}^2\mathbf{I}\right),$$

(6)

where ${\mathcal{N}}_{\mathbb{C}}$ is a circularly symmetric complex normal distribution and $\mathbf{I}$ is the identity matrix. We use Equations (5.50, 5.53) in Särkkä and Solin [43] to determine the mean:

$$\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)=e^{-\mathsf{\gamma }t}{\mathbf{x}}_0+\left(1-e^{-\mathsf{\gamma }t}\right)\mathbf{y},$$

(7)

and the variance of the closed-form solution.

$$\mathsf{\sigma }{\left(t\right)}^2={\displaystyle \frac{{\mathsf{\sigma }}_{\min }^2\left({\left(\frac{{\mathsf{\sigma }}_{\max }}{{\mathsf{\sigma }}_{\min }}\right)}^{2t}-e^{-2\mathsf{\gamma }t}\right)\log \left(\frac{{\mathsf{\sigma }}_{\max }}{{\mathsf{\sigma }}_{\min }}\right)}{\mathsf{\gamma }+\log \left(\frac{{\mathsf{\sigma }}_{\max }}{{\mathsf{\sigma }}_{\min }}\right)}}$$

(8)

Vincent [44] showed that the score model $S_{\theta }$ fitted to the perturbation kernel ${\nabla }_{{\mathbf{x}}_t}{\log }_{p0t}\left({\mathbf{x}}_t\right|{\mathbf{x}}_0,\mathbf{y})$ is equivalent to implicit and explicit score matching under certain regularity conditions [45]. This is essentially an estimation result.

Accordingly, ${\mathbf{x}}_t$ can be efficiently calculated as

$${\mathbf{x}}_t=\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)+\mathsf{\sigma }\left(t\right)\mathbf{z},$$

(9)

where $\mathbf{z}\sim {\mathcal{N}}_{\mathbb{C}}\left(\mathbf{z};0,\mathbf{I}\right)$.

Using the score-matching principle [44], we simplify the perturbation kernel ${\nabla }_{{\mathbf{x}}_{\mathit{t}}}{\log }_{p0t}\left({\mathbf{x}}_t\right|{\mathbf{x}}_0,\mathbf{y})$ to

$${\nabla }_{{\mathbf{x}}_t}\log p_{0t}\left({\mathbf{x}}_t|{\mathbf{x}}_0,\mathbf{y}\right)={\nabla }_{{\mathbf{x}}_t}\log \left[{\left|2\pi \sigma \mathbf{I}\right|}^{-{\displaystyle \frac{1}{2}}}{e}^{-{\displaystyle \frac{‖{\mathbf{x}}_t-\mathit{\mu }‖\genfrac{}{}{0pt}{}{2}{2}}{2{\mathsf{\sigma }}^2}}}\right]$$

(10)

$$={\nabla }_{{\mathbf{x}}_t}{\log \left|2\mathsf{\pi }\mathsf{\sigma }\left(t\right)\right|}^{-{\displaystyle \frac{1}{2}}}-{\nabla }_{{\mathbf{x}}_t}{\displaystyle \frac{‖{\mathbf{x}}_t-\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)‖\genfrac{}{}{0pt}{}{2}{2}}{2\mathsf{\sigma }{\left(t\right)}^2}}$$

(11)

$$=-{\displaystyle \frac{{\mathbf{x}}_t-\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)}{\mathsf{\sigma }{\left(t\right)}^2}}$$

(12)

Substituting Equation (9) into Equation (12), we obtain

$${\mathbf{x}}_t-\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)=\left(\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)+\mathsf{\sigma }\left(t\right)\mathbf{z}\right)-\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)=\mathsf{\sigma }\left(t\right)\mathbf{z}$$

(13)

Therefore, Equation (12) can be rewritten as

$${\nabla }_{{\mathbf{x}}_t}\log p_{0t}\left({\mathbf{x}}_t|{\mathbf{x}}_0,\mathbf{y}\right)=-{\displaystyle \frac{{\mathbf{x}}_t-\mathit{\mu }\left({\mathbf{x}}_0,\mathbf{y},t\right)}{\mathsf{\sigma }{\left(t\right)}^2\hspace{1em}}}={\displaystyle \frac{\mathsf{\sigma }\left(t\right)\mathbf{z}}{\mathsf{\sigma }{\left(t\right)}^2}}$$

(14)

Simplifying the expression yields

$${\nabla }_{{\mathbf{x}}_t}\log p_{0t}\left({\mathbf{x}}_t|{\mathbf{x}}_0,\mathbf{y}\right)={\displaystyle \frac{\mathbf{z}}{\mathsf{\sigma }\left(t\right)}}$$

(15)

After $\left({\mathbf{x}}_t,\mathbf{y},t\right)$ is input into the score model (see Equation (4)), the final loss is the unweighted $L_2$ loss between the model output (${\mathit{s}}_{\theta }\left({\mathbf{x}}_t,\mathbf{y},t\right)$) and the score of the perturbation kernel (derived from Equation (15)). The overall training objective is given by Equation (16):

$$\arg \underset{\mathsf{\theta }}{\min }{\mathbb{E}}_{t,\left({\mathbf{x}}_0,\mathbf{y}\right),\mathbf{z},{\mathbf{x}}_t|\left({\mathbf{x}}_0,\mathbf{y}\right)}\left[‖{\mathit{s}}_{\theta }\left({\mathbf{x}}_t,\mathbf{y},t\right)+{\displaystyle \frac{\mathbf{z}}{\sigma \left(t\right)}}‖{\displaystyle \genfrac{}{}{0pt}{}{2}{2}}\right],$$

(16)

2.2. Numerical SDE Solver

In the speech enhancement task, the accuracy and stability of the numerical SDE solver directly affect the performance of the model. This section elaborates on the solution method for numerical SDEs.

When dealing with the numerical solution of stochastic differential equations, researchers have developed a variety of approximate methods based on discrete time steps. In general, the time interval $[0,\mathrm{T}]$ is divided into $\mathrm{N}$ equal parts, with each subinterval having a length of $\mathsf{\Delta }t=\mathrm{T}/\mathrm{N}$. Based on this, the original continuous formula is transformed into a discrete sequence {${\mathrm{x}}_t$, ${\mathrm{x}}_{T-\mathsf{\Delta }\mathrm{t}}$, …, ${\mathrm{x}}_0$} to be solved numerically. Among the many single-step methods, the Euler–Maruyama method is widely used. In each iteration, it first checks the state of the previous moment and then combines the characteristics of drift and Brownian motion to infer the state value of the current moment. This method is relatively intuitive in calculation, but it is also necessary to pay attention to the impact of the choice of step size on the result.

In this study, we used the predictor–corrector (PC) samplers proposed by Song et al. [41]. The core of this sampler is to cleverly combine the solution strategy of the reverse SDE with numerical optimization methods, such as annealing Langevin dynamics [46]. Algorithm 1 [41] outlines the principles of the PC sampler. The PC sampler mainly consists of two main components: the predictor and the corrector. The predictor can use various single-step calculation methods, and its primary function is to approximate the reverse SDE through iterative computation. After each iteration of the predictor, the current state of the process will be further optimized and improved by the corrector. The corrector is essentially a stochastic gradient ascent optimizer based on Markov chain Monte Carlo sampling. In each iterative step, a small adjustment is made along the gradient direction of the estimated score function, and a small amount of noise interference is introduced to ensure that the sampling points converge near the target distribution, improving the generation quality and stability. The solver is implemented in Python 3.12.3.

In this study, we explore two different numerical methods for the predictor: the Euler–Maruyama method, a classic single-step method for approximating SDEs, and the reverse diffusion method, which offers higher accuracy in solving the reverse SDE. Through iterative updates, the predictor guides the sampling trajectory toward cleaner speech representations. Our ablation studies demonstrate that the choice of method in the predictor impacts speech enhancement quality.

Algorithm 1: Predictor–Corrector (PC) Sampling

Require:   N: Number of discretization steps for the reverse-time SDE   M: Number of Corrector steps $1: \mathrm{Initialize} {\mathrm{x}}_N\backsim {\mathrm{x}}_T$ $2: \mathrm{for} i=N-1 \mathrm{t}\mathrm{o}$0$\mathrm{d}\mathrm{o}$ $3: {\mathrm{x}}_i\leftarrow \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}\left({\mathrm{x}}_{i+1}\right)$ $4: \mathrm{for} j=1 \mathrm{t}\mathrm{o} M \mathrm{d}\mathrm{o}$ $5: \hspace{1em} {\mathrm{x}}_i\leftarrow Corrector\left({\mathrm{x}}_i\right)$ $6: \mathrm{return} {\mathrm{x}}_0$

2.3. EMA Mechanism

The basic principle of the traditional attention mechanism is to assign weights to each element in the input sequence and perform a weighted summation to extract the key features. It has been shown [47,48] that capturing multi-level contextual information is crucial to improving the performance of speech enhancement models based on deep neural networks. The EMA mechanism [49] adopts a novel approach by dynamically generating attention weights through multi-scale parallel sub-networks and cross-space learning instead of the traditional exponential moving average. To update the attention weights, EMA incorporates three parallel pathways, with two in the 1 × 1 branch and one in the 3 × 3 branch, as shown in Figure 2 [49].

In the EMA mechanism, the global spatial information from the 1 × 1 convolution branch is processed by applying Softmax activation to generate channel-wise attention weights, while the 3 × 3 convolution branch output is reshaped to match the dimensions of the 1 × 1 branch. The features from these two branches are then combined through matrix multiplication, leading to the formation of the first spatial attention map. At the same time, a 2D average pooling operation is performed on the 3 × 3 branch, and the resulting features are passed through a Softmax function to produce another set of attention weights. Similarly, the 1 × 1 branch output is reshaped, and matrix multiplication between the two branches yields the second spatial attention map. To refine the spatial attention distribution, the EMA module aggregates the feature outputs guided by the attention maps from both branches. This design allows the model to exploit complementary spatial features learned in parallel, enhancing its ability to capture richer contextual information. In this way, when the model performs attention calculations, it not only considers the current input features but also effectively integrates contextual information from previous layers.

It is particularly worth mentioning that the EMA mechanism adopts a parallel substructure design, which effectively avoids excessive sequential processing and eliminates the need for increased network depth. This reduces redundant sequence processing steps and decreases instability during training, thereby improving the overall performance of the model. In addition, the parallelization of convolution operations further strengthens the expressive capacity of the structure. By combining the parallel computing methods of 3 × 3 convolution and 1 × 1 convolution, the model can simultaneously capture local short-term dependencies and global long-term dependencies, thereby merging more contextual information into the intermediate feature map. The EMA module will be further evaluated through ablation studies in Section 4.5.

2.4. Multi-Resolution U-Net Network

In recent years, significant progress has been made in enhancing acoustic quality through deep learning methods, with the U-Net architecture emerging as a particularly effective model [50,51]. As illustrated in Figure 3, the U-Net consists of three main components: an encoder, a decoder, and skip connections. Specifically, the encoder progressively downsamples the input audio to extract high-level acoustic representations, while the decoder performs upsampling to reconstruct enhanced audio with dimensions matching the original input. The skip connections directly transfer feature maps from the encoder to the corresponding decoder layers, enabling the integration of low-level and high-level information to facilitate more accurate detail restoration.

Some U-Net-based speech enhancement models rely on ordinary convolution operations in the encoder and decoder, which may overlook speech contextual information and result in the loss of detailed features. To address these problems, we incorporated the EMA mechanism into the U-Net-based network. This approach focuses on the detailed speech features, extracts local features at different scales, and captures more contextual information, thereby enabling more effective speech enhancement.

3. Network Model Structure

The SGM-EMA model is built on the Noise Conditional Score Network++ (NCSN++) architecture [41] and uses a multi-resolution U-Net-based structure. Previous studies [52] have shown that this type of structure has strong capabilities in tasks such as generation and segmentation. In Figure 4, we show feature maps at different resolutions, annotate their spatial sizes and number of channels, and use arrows to illustrate the transformations between feature maps.

The main body of the network consists of symmetrical downsampling and upsampling paths, along with skip connections. The input and output layers use Conv2D layers with 3 × 3 convolution kernels and a stride of 1. The residual block structure derived from the BigGAN architecture [53] is embedded in both the downsampling and upsampling blocks. Each residual block consists of the same Conv2D layer as above, group normalization [54], LeakyReLU activation function, and either upsampling or downsampling layers based on finite impulse response (FIR) filters [55]. Each downsampling path contains two residual blocks, and each upsampling path contains three residual blocks [56]. A 1 × 1 Conv2D layer is employed to facilitate the progressive transformation of feature dimensionality [57]. The network performs symmetrical dimensionality reduction and expansion through feature maps of specific resolutions. The output dimensions of the downsampling paths are (256, 256, 128), (128, 128, 128), (64, 64, 256), (32, 32, 256), (16, 16, 256), (8, 8, 256), and (4, 4, 256). To enhance feature fusion, an EMA mechanism is added to the bottleneck layer and the 16 × 16 and 64 × 64 resolution layers. The overall architecture supports efficient gradient propagation and multi-scale feature aggregation through the combination of residual connections and attention mechanisms, showing significant performance advantages in speech enhancement tasks.

To enhance the model’s temporal perception, we used Fourier embeddings [58] to map the time scalar $t$ into an m-dimensional vector $t_{emb}$. Fourier transform was then applied to encode temporal information into a representation suitable for neural network processing. This vector was embedded into each residual block, as shown in Figure 4, so that the model could simultaneously consider the interaction between temporal dynamics and spatial features. This temporal encoding mechanism not only allowed the network to model both the current state and overall temporal trends but also improved speech quality and enhanced generalization.

4. Experimental Setup

4.1. Dataset

We used two datasets, the VB-DMD dataset and the TIMIT-TUT dataset, as detailed in

Table 1. Datasets used in the experiment.

Dataset Type	SNR	Quantity	Number of Speakers	Sampling Rate
VB-DMD Training Set	15 dB, 10 dB, 5 dB, 0 dB	11,572	28	16 kHz
VB-DMD Test Set	17.5 dB, 12.5 dB, 7.5 dB, 2.5 dB	824	2	16 kHz
TIMIT-TUT Test Set	0–20 dB uniform sampling	1344	168	16 kHz

. The performance of the proposed model was evaluated by dual dataset verification. Specifically, one dataset was used for training, while the other dataset was used as the test set. This experimental design effectively assessed the model’s adaptability to data with different distribution characteristics, thereby more realistically reflecting its generalization performance.

(a)

VB-DMD: The experiments used the publicly available VB-DMD dataset [59], which is widely used in speech enhancement and denoising research (it is available at https://doi.org/10.7488/ds/2117, accessed on 10 September 2024). The clean speech signal came from the VoiceBank dataset, with a sampling rate of 48 kHz, and the noisy signal came from the DEMAND dataset, with a sampling rate of 48 kHz. The dataset contains a variety of real-world background noises, such as cafes, streets, and offices. The training set (11,572 utterances) consisted of 28 speakers with signal-to-noise ratios (SNRs) of 15, 10, 5, and 0 dB. The test set (824 utterances) consisted of 2 speakers with SNRs of 17.5, 12.5, 7.5, and 2.5 dB. Both the training and test sets were resampled to 16 kHz.

(b)

TIMIT-TUT: The TIMIT-TUT dataset was created by selecting the complete test set of the acoustic–phonetic continuous speech corpus jointly constructed by Texas Instruments (TI), Massachusetts Institute of Technology (MIT), and Stanford Research Institute (SRI), and the noise signal from TUT. The TUT noise dataset covers a variety of sound events in daily environments, such as traffic noise, human voices, animal calls, etc. (it is available at https://paperswithcode.com/dataset/tut-sound-events-2017, accessed on 13 September 2024). The TIMIT [60] complete test set of clean speech consists of 1344 utterances spoken by 168 speakers with a sampling rate of 16 kHz (it is available at https://goo.gl/l0sPwz, accessed on 14 September 2024). The TIMIT-TUT mixed dataset was a random segment from a noise file randomly selected from the noise signal and added to the clean speech signal, generating a total of 1344 mixed speech samples for testing. All mixed datasets were resampled uniformly to 16 kHz, with a signal-to-noise ratio between 0 and 20 dB.

4.2. Hyperparameter Configuration

The hyperparameters in Equations (2) and (7) were set to ${\sigma }_{min}=0.05$, ${\sigma }_{max}=0.5,$ and $\gamma =1.5$. The Adam optimizer was used for training, the learning rate was set to ${10}^{-4}$, and the batch size was 8. The exponential moving average decay rate of the model weight was 0.999, which was used for sampling [61], and the sampler used the PC sampler. The number of backward steps in the SDE was set to N = 30 [56], and the step size r of the annealing Langevin dynamics in the corrector was set to 0.5.

The U-Net network had 14 layers (7 layers for the encoder and 7 layers for the decoder). The resolution and number of channels of the encoder and decoder were symmetrical. We show the number of channels and resolution of the encoder in

Table 2. U-Net encoder layer parameter table.

Number of Encoder Layers	Resolution	Channels	Normalization
1	256 × 256	128	Group normalization
2	128 × 128	128
3	64 × 64	256
4	32 × 32	256
5	16 × 16	256
6	8 × 8	256
7	4 × 4	256

.

The configurations used in our experiments are shown in

Table 3. Experimental configuration table.

Name	Specific Configuration
Operating System	Linux-5.15.0-86-generic-x86_64-with-glibc2.35
Processor	NVIDIA vGPU-32 GB
Memory	32 GB
OS Bit	64-bit
Programming Language	CPython 3.12.3
Dataset	VB-DMD and TIMIT-TUT
Deep Learning Framework	PyTorch Lightning 2.1.4

.

4.3. Baseline

We compared the proposed method with three discriminative baselines (MetricGAN+ [62], SERGAN [63], and CMGAN [64]) and two generative baselines (RVAE [65] and CDiffuse [66]). All methods utilized the pre-trained models provided in the publicly available code from the corresponding papers to evaluate the datasets. Except for the SERGAN method, we retrained the models from scratch on the VB-DMD dataset and then evaluated them on both the VB-DMD and TIMIT-TUT datasets.

4.4. Evaluation Metrics

To evaluate the performance of the proposed method, three objective metrics were adopted: PESQ [67], ESTOI [68], and SI-SDR [69].

(a)

PESQ: This metric assesses perceptual speech quality and is commonly used in voice communication and speech enhancement. It produces scores ranging from −0.5 to 4.5, where a higher score indicates better speech quality and a lower score indicates poor speech quality.

(b)

ESTOI: This is an objective indicator for evaluating speech intelligibility in speech enhancement systems. ESTOI provides scores between 0 and 1, where values closer to 1 indicate better speech intelligibility, and values closer to 0 indicate poorer speech intelligibility.

(c)

SI-SDR: This metric evaluates signal reconstruction accuracy in tasks such as speech enhancement, denoising, and source separation. Values below 10 dB suggest significant distortion and poor reconstruction, scores between 10 and 20 dB indicate acceptable quality with moderate artifacts, and values above 20 dB imply high-fidelity reconstruction with minimal distortion or noise.

4.5. Ablation Study

To evaluate the effectiveness of the Euler–Maruyama method and the reverse diffusion method in the predictor component of the PC sampler, as well as the contribution of the EMA mechanism to the performance of the speech enhancement model, we conducted a series of ablation studies. By selectively disabling or replacing key components of the predictor and the EMA module, we compared multiple model configurations on the VB-DMD and TIMIT-TUT datasets. The results, presented in

Table 4. Ablation study results on the VB-DMD dataset. (Model A: Euler–Maruyama method; Model B: reverse diffusion method; Model C: EMA mechanism).

Experiment Number	Model A	Model B	Model C	PESQ (↑)	ESTOI (↑)	SI-SDR (↑)
1	×	×	×	2.12 ± 0.49	0.76 ± 0.09	12.8 ± 4.1
2	×	×	√	2.24 ± 0.51	0.80 ± 0.09	14.1 ± 4.0
3	√	×	×	2.57 ± 0.65	0.78 ± 0.10	16.3 ± 3.7
4	√	×	√	2.69 ± 0.63	0.86 ± 0.10	17.3 ± 3.5
5	×	√	×	2.72 ± 0.63	0.85 ± 0.10	16.6 ± 3.1
6	×	√	√	2.89 ± 0.80	0.91 ± 0.07	19.6 ± 3.5

↑: indicates that a higher value corresponds to better performance.

and

Table 5. Ablation study results on the TIMIT-TUT dataset. (Model A: Euler–Maruyama method; Model B: reverse diffusion method; Model C: EMA mechanism).

Experiment Number	Model A	Model B	Model C	PESQ (↑)	ESTOI (↑)	SI-SDR (↑)
1	×	×	×	1.74 ± 0.40	0.72 ± 0.11	11.2 ± 4.1
2	×	×	√	1.81 ± 0.44	0.75 ± 0.12	12.1 ± 4.4
3	√	×	×	2.40 ± 0.56	0.71 ± 0.10	15.1 ± 4.2
4	√	×	√	2.58 ± 0.53	0.76 ± 0.09	16.3 ± 3.5
5	×	√	×	2.63 ± 0.53	0.80 ± 0.11	15.8 ± 4.1
6	×	√	√	2.79 ± 0.66	0.86 ± 0.10	17.1 ± 3.5

↑: indicates that a higher value corresponds to better performance.

as the means ± standard deviations, demonstrate that the reverse diffusion method was more effective than the Euler–Maruyama method, and that the inclusion of the EMA mechanism further improved the overall model performance. Specifically, removing the reverse diffusion method led to decreases in the SI-SNR, ESTOI, and PESQ scores, highlighting the importance of accurately solving the reverse SDE to generate high-quality enhanced speech. Among these, the drop in ESTOI reflects a reduction in speech intelligibility, while decreases in the PESQ and SI-SNR indicate losses in perceptual quality and signal fidelity, respectively. Similarly, removing the EMA mechanism also caused performance degradation, confirming its critical role in ensuring model stability and convergence. In addition, the ESTOI scores consistently increased with the inclusion of each component, particularly the reverse diffusion method and EMA. When both were applied together, the highest ESTOI score of 0.91 was achieved, indicating a substantial improvement in intelligibility. These findings confirm the necessity of incorporating both techniques into the proposed framework, as they not only improved signal quality and perceptual clarity but also ensured that the enhanced speech remained intelligible, an essential factor for real-world applications, such as voice communication and assistive technologies.

5. Experimental Results and Discussion

5.1. Experimental Results

We used matched training and testing datasets—specifically, we trained the model on the VB-DMD training set and evaluated it on the corresponding test set; the experimental results are presented in

Table 6. Under matched evaluation conditions, the results in this table are categorized by model types D and G. Except for SGM-EMA and SERGAN, all methods used pre-trained models published in their original papers and were evaluated on the same test set (VB-DMD) with five independent runs. The data in the table are presented as the means ± standard deviations (95% confidence interval).

Method	Type	Training Set	Test Set	PESQ (↑)	ESTOI (↑)	SI-SDR (↑)
MetricGAN+ [62]	D	VB-DMD	VB-DMD	3.13 ± 0.55 (±0.68)	0.83 ± 0.11 (±0.14)	8.5 ± 3.6 (±4.5)
SERGAN [63]	D	VB-DMD	VB-DMD	2.62 ± 0.63 (±0.78)	0.85 ± 0.06 (±0.08)	17.2 ± 3.2 (±4.0)
CMGAN [64]	D	VB-DMD	VB-DMD	3.41 ± 0.68 (±0.84)	0.88 ± 0.10 (±0.12)	18.4 ± 4.2 (±5.2)
RVAE [65]	G	VB	VB-DMD	2.48 ± 0.55 (±0.68)	0.81 ± 0.11 (±0.14)	17.1 ± 5.0 (±6.2)
CDiffuse [66]	G	VB-DMD	VB-DMD	2.46 ± 0.51 (±0.63)	0.79 ± 0.11 (±0.14)	12.6 ± 5.0 (±6.2)
SGM-EMA [ours]	G	VB-DMD	VB-DMD	2.89 ± 0.80 (±0.99)	0.91 ± 0.07 (±0.09)	19.6 ± 3.5 (±4.3)

↑: indicates that a higher value corresponds to better performance.

. The average values of all metrics in Table 6 are visualized in Figure 5a. RVAE is an unsupervised speech enhancement method trained solely on clean speech (VB). The results show that although our SGM-EMA model did not outperform CMGAN in terms of the PESQ metric, this was largely due to CMGAN’s specifically designed PESQ optimization module in its discriminator, as well as the use of a multi-domain joint loss function. These design choices effectively suppress distortion, allowing CMGAN to achieve higher PESQ scores. Nevertheless, SGM-EMA still outperformed other models and surpassed the discriminative model SERGAN, demonstrating strong speech restoration capabilities. Compared to other baseline methods, such as RVAE and CDiffuse, SGM-EMA achieved significant improvements in both the PESQ and SI-SDR metrics.

To further validate the statistical reliability of the results, we report the 95% confidence intervals for each evaluation metric (values in parentheses in Table 6). Although the confidence intervals of SGM-EMA for PESQ and ESTOI metrics partially overlapped with those of baseline models, like CMGAN and SERGAN, our model demonstrated systematic superiority: it achieved higher mean values across all metrics while maintaining narrower confidence intervals, highlighting its exceptional speech recovery capability. Particularly for the SI-SDR metric, SGM-EMA not only sustained superior average performance but also exhibited lower variance, demonstrating remarkable stability. These results not only verify the outstanding performance of SGM-EMA in speech enhancement tasks but also statistically confirm its reliability, establishing a solid foundation for further advancements in speech enhancement technology.

When the training and testing datasets were mismatched—that is, when the models were trained on the VB-DMD dataset and evaluated on the TIMIT-TUT dataset—we evaluated model performance under these mismatch conditions. We tested the TIMIT-TUT data using pretrained models provided in the official codebases of the respective papers. The experimental results are summarized in

Table 7. For the speech enhancement results under mismatch conditions, the table is sorted according to model types D and G. All methods except SGM-EMA and SERGAN used the pre-trained models published in the original paper and performed 5 independent runs on the same test set (TIMIT-TUT), reporting their means and standard deviations (confidence intervals).

Method	Type	Training Set	Test Set	PESQ (↑)	ESTOI (↑)	SI-SDR (↑)
MetricGAN+ [62]	D	VB-DMD	TIMIT-TUT	1.54 ± 0.53 (±0.66)	0.62 ± 0.12 (±0.15)	4.1 ± 3.1 (±3.9)
SERGAN [63]	D	VB-DMD	TIMIT-TUT	2.17 ± 0.58 (±0.72)	0.75 ± 0.07 (±0.09)	10.0 ± 2.8 (±3.5)
CMGAN [64]	D	VB-DMD	TIMIT-TUT	2.65 ± 0.60 (±0.75)	0.75 ± 0.09 (±0.11)	16.2 ± 3.6 (±4.5)
RVAE [65]	G	VB	TIMIT-TUT	2.24 ± 0.49 (±0.61)	0.72 ± 0.11 (±0.14)	13.8 ± 4.8 (±6.0)
CDiffuse [66]	G	VB-DMD	TIMIT-TUT	1.71 ± 0.42 (±0.52)	0.69 ± 0.11 (±0.14)	10.6 ± 3.2 (±4.0)
SGM-EMA [ours]	G	VB-DMD	TIMIT-TUT	2.79 ± 0.66 (±0.82)	0.86 ± 0.10 (±0.12)	17.1 ± 3.5 (±4.4)

↑: indicates that a higher value corresponds to better performance.

. To provide a more intuitive comparison, the average values of each metric in Table 7 are visualized in Figure 5b. As expected, the overall performance declined compared to the matched condition results in Table 4. This drop is primarily attributed to the fact that the specific characteristics of the test set were not exposed during training. The signal characteristic differences between the VB-DMD training data and the TIMIT-TUT test set were significant enough to cause a degradation in the evaluation scores.

Nevertheless, under the mismatched conditions, our SGM-EMA model outperformed all other baseline methods across every evaluation metric, demonstrating exceptional robustness to variations in signal characteristics. Although the scores on the TIMIT-TUT test set were lower than those on the VB-DMD test set, SGM-EMA still outperformed the competing methods. This enabled SGM-EMA to achieve the best overall scores on the TIMIT-TUT dataset, highlighting its robustness to unseen noise characteristics and indicating its strong generalization ability.

To further validate the statistical reliability of these results, we also report the 95% confidence intervals for each evaluation metric, as shown in the parentheses in Table 7. SGM-EMA consistently achieved the best average scores across all evaluation metrics under the mismatched conditions. While some confidence intervals partially overlap, the performance trend suggests an improvement compared to the baseline methods, especially in the SI-SDR and ESTOI.

5.2. Trade-Offs in Model Design

Our model utilizes a deep U-Net with multiple resolution levels, a progressive growth path, and attention modules. While this complexity enables high-quality reconstruction and better modeling of global dependencies, it also leads to increased computational costs and slower inference times. We employed a score-based generative objective to encourage the generation of natural, manifold-aligned speech. This approach typically generalizes better to unseen noise types, as demonstrated by the results under mismatched conditions in Table 7. However, this comes at the cost of waveform fidelity, occasionally leading to the generation of artifacts under extreme noise conditions. The introduction of attention modules at specific resolutions improves global dependency modeling but increases the number of model parameters and inference complexity. The choice of group norm over alternatives like batch norm is due to its robustness to small batch sizes, which is critical for high-resolution spectrogram modeling. However, compared to the more lightweight batch norm, it introduces a small amount of additional computational overhead.

6. Conclusions

We propose a novel speech enhancement method that integrates an EMA mechanism with a U-Net architecture, trained and optimized as a score-based diffusion model. The experimental results show that under the conditions of matching training and testing, our method achieved better performance than other generative models and outperformed other comparative methods in both the ESTOI and SI-SDR evaluation indicators, demonstrating its excellent speech enhancement capability. Under mismatched conditions, while exhibiting some performance degradation compared to the matched scenario, our approach still outperformed all competing methods across all evaluation metrics, demonstrating remarkable generalization ability. Ablation studies further confirmed the importance of the reverse diffusion predictor and the EMA mechanism, revealing their critical roles in improving both enhancement quality and model stability.

For future research, we plan to explore multimodal information fusion to further enhance speech enhancement performance. For instance, we will investigate incorporating visual information (such as speaker lip movement features) to combine the complementary advantages of audio and visual modalities, enabling the model to capture more comprehensive speech cues and thereby achieving better enhancement results.