Simultaneous Speech Denoising and Super-Resolution Using mGLFB-Based U-Net, Fine-Tuned via Perceptual Loss ^†

Abstract

This paper presents an efficient U-Net architecture featuring a modified Global Local Former Block (mGLFB) for simultaneous speech denoising and resolution reconstruction. Optimized for computational efficiency in the discrete cosine transform domain, the proposed architecture reduces model size by 13.5% compared to a standard GLFB-based U-Net, while maintaining comparable performance across multiple quality metrics. In addition to the mGLFB redesign, we introduce a perceptual loss that better captures high-frequency magnitude spectra, yielding notable gains in high-resolution recovery, especially in unvoiced speech segments. However, the mGLFB-based U-Net still shows limitations in retrieving spectral details with substantial energy in 4–6 kHz frequencies.

1. Introduction

Speech is the primary mode of human communication, but its intelligibility and quality are often degraded by factors such as background noise, reverberation, and bandwidth limitations. To address these issues, speech denoising and super-resolution have emerged as key tasks in speech enhancement [1,2], focusing on improving clarity and restoring missing high-frequency components. Speech denoising aims to remove additive noise and preserve intelligibility, while super-resolution seeks to reconstruct high-frequency details often lost during recording or transmission.

In recent years, deep neural networks (DNNs) have significantly advanced both denoising and super-resolution techniques [3,4]. Traditional signal processing approaches—such as spectral subtraction, Wiener filtering, and minimum mean squared error (MMSE)-based estimation—have gradually been surpassed by more sophisticated neural network-based methods. Prominent models include convolutional neural networks (CNNs) [4,5,6,7,8], recurrent neural networks (RNNs) [9,10], and transformer-based models [11,12,13], all of which have demonstrated impressive results in speech enhancement.

In the realm of speech super-resolution, predictive DNNs have been developed to map low-resolution speech to high-resolution outputs [14,15,16]. While the developed DNNs appeared effective in experimentally settled environments, these models often produce overly smoothed results and struggle with generalization due to their deterministic mapping design [17]. Hence, a few researchers explored generative approaches, including generative adversarial networks (GANs) [18,19] and diffusion models [20,21], which treat super-resolution as a conditional generation problem. While more robust to variability, the generative methods generally fell behind predictive models in terms of reconstruction accuracy. Hybrid models that combine the advantages of prediction and generation have become the latest evolution [22].

Over the past decade, deep learning-based speech enhancement has become a major focus in both academia and industry. The IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) has introduced the Deep Noise Suppression (DNS) Challenge [2,23], emphasizing the improvement of speech quality and intelligibility in noisy environments. Since noise suppression is increasingly vital for real-world applications, such as live video and smart devices, developing efficient and deployable solutions is critical. While much of the literature has centered on enhancing denoising capability, the real-time performance of super-resolution remains underexplored. Nevertheless, for speech enhancement to genuinely benefit end-users, solutions capable of real-time operation are imperative.

In our previous study [12], we demonstrated the potential of a six-level U-Net architecture, enhanced with Global Local Former Blocks (GLFBs) [11], for achieving efficient and effective speech enhancement in the short-time discrete cosine transform (STDCT) domain. Given the U-Net’s compatibility with resolution restoration, extending this framework to include super-resolution is a logical next step. More recently, the conformer-based CMGAN model [13] has shown promise for speech enhancement in the time-frequency (T-F) domain, handling tasks such as denoising, dereverberation, and super-resolution. However, a unified design that effectively integrates these tasks for robust real-world deployment remains an open challenge.

To address the above issues, this paper proposes reorganizing the U-Net architecture using modified GLFBs (termed mGLFBs) to enable simultaneous, real-time speech denoising and super-resolution. We further refine the loss function to emphasize perceptual quality, specifically targeting improvements in spectral feature restoration. By combining these enhancements, our approach unifies denoising and super-resolution within a real-time, practical U-Net framework, offering additional gains from the perceptually motivated loss function.

The remainder of this paper is organized as follows: Section 2 describes the proposed model architecture, including the modified GLFBs, U-Net framework, and perceptual loss function. Section 3 outlines the experimental setup, including datasets, evaluation metrics, and training procedures. Section 4 presents the results and discussions. Finally, Section 5 concludes the paper and suggests potential directions for future research.

2. DNN for Speech Enhancement

2.1. Processing Framework

A noisy speech signal $y\left[n\right]$ is generally modeled as a sum of clean speech $x\left[n\right]$ and additive noise $z\left[n\right]$:

$$y[n]=x[n]+z[n].$$

(1)

Our processing framework for speech denoising follows a frame-based overlap-and-add (OLA) method. Specifically, the noisy speech signal is partitioned into overlapping frames of fixed length, and each frame is processed through a denoising DNN to generate the desired clean speech. Previous research has shown that performing speech denoising in the short-time discrete cosine transform (STDCT) domain offers distinct advantages over traditional time-domain approaches [12], thanks to the energy-compacting properties of the DCT. Therefore, we transform the speech signal into the STDCT domain before further processing.

To model temporal context in speech signals, we feed the STDCT sequences into the DNN using a sliding buffer that holds a certain number of past frames for causal modeling. After the denoising process, the frames are converted back to the time domain using the inverse DCT and then reassembled into a whole speech signal through the OLA method.

Figure 1 illustrates the overall framework for speech denoising. Given an 8 kHz sampling rate, we select a frame length $L_f$ of 256 samples, with a shift distance $L_s$ of 64 samples, resulting in a 32 ms window with an 8 ms stride. These settings introduce a 40 ms delay in real-time processing.

2.2. U-Net Architecture

The U-Net architecture has become widely adopted for speech enhancement tasks due to its effectiveness in recovering clean signals from noisy inputs. Originally developed for biomedical image segmentation, U-Net [24] has proven highly effective across various signal processing applications, including speech denoising [4,5,6,7,8] and super-resolution [14,15,25]. A U-Net is characterized by its U-shaped structure, consisting of an encoding path that extracts essential features and a decoding path that reconstructs the features to match the original input dimensions. The encoding and decoding stages are connected via skip connections, enabling the network to retain high-resolution details while learning hierarchical features at multiple scales.

In our previous study [12], we employed GLFBs to construct a six-level U-Net for speech denoising. GLFB, inspired by the transformer architecture [26], refers to a network module combining global and local modeling to better capture dependencies in the data. The GLFB blocks provide transformer-like functionality without the heavy computational cost typically associated with transformer networks. Compared to U-Nets built with conventional convolutional blocks, the GLFB-based U-Net achieves a 39% reduction in model size, decreasing from 612 K to 238.6 K. With fewer parameters and a more efficient design, the GLFB-based U-Net delivers superior performance while consuming less memory, making it an essential feature for real-time applications.

Building on our previous work, we adopt a modified GLFB (termed mGLFB) [27] to rebuild a U-Net capable of performing both speech denoising and super-resolution. Figure 2 illustrates the configuration of the adopted U-Net, where mGLFBs form the core building submodules of both the encoder and decoder. At the bottleneck, located on the right side of the figure, is a two-layer dense block that enhances parameter efficiency in the latent space.

Table 1. Specifications of network layers in the six-level U-Net. Each submodule in the encoder comprises an mGLFB (in the form of two 2D convolutional operations in tandem) and a downsampler, while a decoder submodule is a combination of an upsampler and an mGLFB. The labels in the table are interpreted as follows: (1) $F_i\left[D_i\right]:2D Kernel\left[Dilation Factor\right]$ used in the $i^{th}$ filter; (2) $C:Channel Number;$ (3) $DS:\left[2D Downsampling Factor\right];$ (4) $US:\left[2D Upsampling Factor\right];$ (5) $S:\left[2D Stride\right];$ (6) $RS:[2D Scaling Factor in a Resampler]$; (7) →: Additive Connection; (8) ⇉: Dense Connection.

Layer Index	6-Level U-Net
	Encoder Side		Decoder Side
Input/Output Projection	F: 3 $\times$ 3, C: 16; S: [1 1]		F: 5 × 4, C: 1; S: [1 4]
–			Resampler: C: 8; RS: [ρ 1/2]; F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[2];
1	F1[D1]: 5[1]$\times$3[1]; F2[D2]: 3[3]$\times$3[2]; C:16; DS:[2 1]	→	C:16; US: [1 1]; F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[2];
2	F1[D1]: 5[1]$\times$3[1]; F2[D2]: 3[3]$\times$3[2]; C:16; DS:[2 2]	→	C:16; US: [2 2]; F1[D1]: 5[1]$\times$3[1]; F2[D2]: 3[3]$\times$3[2];
3	F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[1]; C:32; DS: [2 1]	→	C:16; US: [2 1]; F1[D1]: 5[1]$\times$3[1]; F2[D2]: 3[3]$\times$3[2];
4	F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[1]; C:32; DS: [2 2]	→	C:32; US: [2 2]; F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[1];
5	F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$1[1]; C:64; DS: [2 1]	→	C:32; US: [2 1]; F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$3[1];
6	F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$1[1]; C:64; DS: [2 2]	→	C:64; US: [2 2]; F1[D1]: 3[1]$\times$3[1]; F2[D2]: 3[2]$\times$1[1];
Latent space (dense block)	C: 64; F[D]: 3[1]$\times$1[1]; S: [1 1]	⇉	C: 64; F[D]: 3[2]$\times$1[2]; S: [1 1]

summarizes the relevant parameters of each submodule in Figure 2 for better understanding.

Evolving from the GLFB, the mGLFB adopts a structure similar to a transformer. Figure 3 illustrates the transition from GLFB to mGLFB, with additional details provided in the following subsection. Within the U-Net architecture, mGLFBs in both the encoder and decoder are paired with downsamplers and upsamplers to dynamically adjust the dimensions of intermediate feature maps as needed, as depicted in Figure 4. This multi-level design enables hierarchical feature extraction, while skip connections between the encoder and decoder help preserve detailed information, ultimately enhancing reconstruction quality.

2.3. Inner Composition of mGLFB

As shown in Figure 3, mGLFB inherits the basic architecture of GLFB, which comprises global and local modeling. The global modeling relies on the cooperation of the depthwise separable convolution, the gating mechanism, and the channel attention mechanism to capture global features. Local modeling, primarily involving pointwise convolutions and gating, focuses on local interactions within the input features. One special feature of the GLFB is that the gating operation replaces the role of standard activation. This alteration helps to reduce complexity and prevent overfitting by controlling information flow.

In our previous implementation of the GLFB-based U-Net, we did not consider the impact of kernel sizes in convolution operations. In general, large kernels increase computational cost while expanding the receptive field and introduce a probable risk of overfitting. This presented an opportunity for further improvement in the structural design of GLFB. Our modification lies in the involved depthwise separable convolutions, combined with gating in global modeling. As depicted in Figure 3b, we replace a single large kernel in the depthwise convolution with a cascade of two small-kernel convolutions, while keeping the number of channels intact (i.e., without doubling the channels). This cascade arrangement allows the receptive field to attain the desired level, as the second convolution layer builds upon the output of the first. To further extend the receptive field, we introduce dilation in the second convolution to help increase the coverage of this filter kernel.

In addition to expanding the receptive field, the cascade arrangement in depthwise convolution can save the need to double the number of channels required for gating. In the original GLFB, the use of gating functions to replace the activations requires doubling the number of channels in advance. Half of the channels process the information, while the other half regulates the flow of information through the gating mechanism. In our modification, we apply the gating mechanism to the intermediate outputs of the two convolutions individually, as shown in the dashed box in Figure 3. This results in two convolved outputs, each with half the number of channels. These two outputs are then concatenated along the channel dimension, eventually restoring the original number of channels. Though the cascading of two depthwise separable convolutions splits the receptive field into diverse patterns, the overall coverage of the receptive field in the 6-level U-Net remains comparable.

2.4. Super-Resolution Reconstruction

As in the case of GLFB, each mGLFB requires cooperation with either an upsampler or downsampler whenever feature maps need to be reshaped or the number of channels needs to be adjusted. The implementations of the upsamplers and downsamplers follow the specifications outlined in [11]. In addition to integrating up-samplers and down-samplers with the mGLFBs, we introduce a resampler to enable super-resolution, which aims at predicting high-frequency spectral components from their low-frequency counterparts. Since the height of the feature map corresponds to the frequency dimension, we utilize the pixel shuffling technique [28] to upscale resolution without introducing interpolation artifacts.

The intended resampler uses a $3 \times 3$ kernel with a stride of $[1, 2]$ and is equipped with a required number of channels to perform dimensional shuffling. As demonstrated in Figure 2, the input and output of the 6-level U-Net contain 16 channels of feature maps, each with a size of $256 \times 8$. The resampler increases the height while reducing the width of the feature maps, ensuring enough channels to facilitate pixel shuffling. For instance, if the target output of the resampler consists of 8 channels with feature maps of size $512 \times 4$, the resampler thus executes a 2-D convolution with 16 filters, each with a stride of [1, 2] in the vertical and horizontal directions, respectively. After convolution, the data (of size $256 \times 4\times (16 channels)$) is shuffled along the channel dimension to form new 2-D feature maps of size $512 \times 4 \times (8 channels)$.

2.5. Perceptual Loss Function

In [12], we employed a composite loss function to guide the Adam optimizer (short for Adaptive Moment Estimation [29]) in adjusting the model’s weights. The loss function comprises two sources: the mean squared error (MSE) loss derived from the magnitude values and the MSE loss computed from the original polar values.

$${\mathit{ℒ}}_{comp}\left({\widehat{Y}}_{DCT}[k],X_{DCT}[k]\right)=\alpha {\mathit{ℒ}}_{MSE\_Mag}\left(|{\widehat{Y}}_{DCT,\beta }[k]|,|X_{DCT,\beta }[k]|\right)+(1-\alpha ){\mathit{ℒ}}_{MSE\_Polar}\left({\widehat{Y}}_{DCT,\beta }[k],X_{DCT,\beta }[k]\right),$$

(2)

where ${\widehat{Y}}_{DCT}[k]$ represents the DCT sequence obtained from the U-Net’s output, and $X_{DCT}[k]$ is the DCT sequence of the clean speech. ${\widehat{Y}}_{DCT,\beta }[k]$ and $X_{DCT,\beta }[k]$ are the two derivatives defined as follows:

$$\begin{array}{l}{\widehat{Y}}_{DFT,\beta }(k)=\mathrm{sgn}\left({\widehat{Y}}_{DCT}[k]\right)\cdot {\left|{\widehat{Y}}_{DCT}[k]\right|}^{\beta };\\ X_{DFT,\beta }(k)=\mathrm{sgn}\left(X_{DCT}[k]\right)\cdot {\left|X_{DCT}[k]\right|}^{\beta },\end{array}$$

where $\mathrm{sgn}(\cdot )$ denotes the sign function. The parameter setting (α, β) = (0,1) in Equation (2) corresponds to the pure MSE loss between the DCT sequences of the recovered and clean speech, while $(\alpha ,\beta )=(0.5,0.5)$ activates both magnitude compression and phase recovery concurrently. While the formula with $(\alpha ,\beta )=(0.5,0.5)$ has yielded satisfactory results, we believe that other auditory features could further enhance performance.

In this study, we propose the use of a perceptual loss function that better reflects the characteristics of the human auditory system. The formulation of this perceptual loss is inspired by the concept of critical bands in psychoacoustics [30,31], where a critical band represents a frequency range within which the ear processes and perceives sounds as a single auditory unit. Two key properties of critical bands guide our design: (1) spectral components within a band should be processed collectively, and (2) bands become wider at higher frequencies. To incorporate these two properties, we employ a $(4l + 1)$-point FIR filter with a symmetric triangular shape to estimate the energy of ${\widehat{Y}}_{DCT}[k]$ within a given band, spanning $(2l + 1)$ points.

$${\left.E_{\widehat{Y}}(j; i)\right|}_{i=j(4l+1)}={\displaystyle \frac{1}{\left(2l+1\right)}}\left[(2l+1)C_{(l)}^2[i]+{\displaystyle \sum _{k=1}^{2l}\left(2l+1-k\right)\left(C_{(l)}^2[i-k]+C_{(l)}^2[i+k]\right)}\right],$$

(3)

where $E_{\widehat{Y}}(j; i)$ denotes the estimated energy for the $j^{th}$ band with the center at position $i=j(2l+1)$. $C_{(l)}\left[i\right]$ signifies the $l^{th}$-level coefficient sequence defined as follows:

$$C_{(l)}[i]={\widehat{Y}}_{DCT}\left[{\displaystyle \frac{L_f}{4}}\left(2^l-\delta [l]\right)+i\right];i=\left\{\begin{array}{l}0,1,2,\dots ,L_f/2-1 \mathrm{for} l=0;\\ 0,1,2,\dots ,2^{l-2}{L}_f-1 \mathrm{for} l>0. \end{array}\right.$$

(4)

Here, $C_{(l)}\left[i\right]$ is an intermediate variable for convenient explanation, and $\delta [l]$ stands for the Dirac delta function. $C_{(0)}\left[i\right]$ corresponds to the STDCT sequence in the 0–4 kHz frequency range, $C_{(1)}\left[i\right]$ spans 4–8 kHz, and $C_{(2)}\left[i\right]$ covers the 8 kHz to 16 kHz range.

Figure 5 illustrates the triangular FIR filter used to estimate the energy level of a frequency band. The perceptual distance ${\mathit{𝒟}}_{CB}(j)$ between the energy estimates (termed $E_{\widehat{Y}}(j; i)$ and $E_X(j; i)$, respectively) acquired from the super-resolved and original STDCT sequences in a frequency band can be defined as follows:

$${\mathit{𝒟}}_{CB}(j)={\displaystyle \frac{1}{\left(2l+1\right)}}{\displaystyle \sum _{i=j(2l+1)-l}^{j(2l+1)+l}{\left|{\left(E_{\widehat{Y}}(j; i)\right)}^{\beta /2}-{\left(E_X(j; i)\right)}^{\beta /2}\right|}^2}.$$

(5)

In the above equation, the exponent $\beta /2$ signifies the application of magnitude compression. Since $E_{\widehat{Y}}(j; i)$ and $E_X(j; i)$ vary with position $i$, we take the average over all possible indices within the $j^{th}$ band, as in Equation (5). Extending this concept to all STDCT coefficients at the same frequency level, we reformulate the perceptual loss function into three main steps: (1) smoothing the squared STDCT coefficients using the triangular filter, (2) applying magnitude compression, and (3) computing the MSE between the compared energy estimates.

To better simulate the auditory system’s processing, we apply a variable-tap FIR filter to the squared STDCT coefficients. The smoothing range, manifested as $(4l + 1)$, is narrower for lower frequency bands and gradually widens at higher frequencies to align with the critical bands in human hearing [32], as follows:

$${\tilde{C}}_{(l)}^2(i)={\displaystyle \frac{1}{{\left(2l+1\right)}^2}}\left[(2l+1)C_{(l)}^2(i)+{\displaystyle \sum _{k=1}^{2l}\left(2l+1-k\right)\left(C_{(l)}^2(i-k)+C_{(l)}^2(i+k)\right)}\right].$$

(6)

The smoothed STDCT magnitude at the $l^{th}$ frequency level is then calculated as:

$$\left|{\tilde{Y}}_{DCT,\beta }^{(l)}\left[{\displaystyle \frac{L_f}{4}}\left(2^l-\delta [l]\right)+i\right]\right|={\left({\tilde{C}}_{(l)}^2[i]\right)}^{\beta /2}={\left|{\tilde{C}}_{(l)}[i]\right|}^{\beta }; i=\left\{\begin{array}{l}0,1,2,\dots ,L_f/2-1 \mathrm{for} l=0;\\ 0,1,2,\dots ,2^{l-2}{L}_f-1 \mathrm{for} l>0, \end{array}\right.$$

(7)

with

$$\left|{\tilde{C}}_{(l)}[i]\right|={\left[{\displaystyle \frac{1}{2l+1}}{C}_{(l)}^2(i)+{\displaystyle \sum _{k=1}^{2l}\frac{2l+1-k}{{\left(2l+1\right)}^2}\left(C_{(l)}^2(i-k)+C_{(l)}^2(i+k)\right)}\right]}^{{\displaystyle \frac{1}{2}}}.$$

(8)

In Equation (7), the superscript $\left(l\right)$ in ${\tilde{Y}}_{DCT,\beta }^{(l)}$ simply reflects the corresponding level. The above spectral smoothing process (i.e., Equations (6)–(8)) allows the loss function to treat neighboring STDCT coefficients as groups rather than as individual elements, thereby enhancing perceptual relevance.

In addition to spectral smoothing, we adapt the weight factor (i.e., parameter α) for each frequency band. The weight assigned to the smoothed magnitude spectrum increases with frequency, thereby encouraging the mGLFB-based U-Net to place greater emphasis on the spectral magnitudes for high-frequency components. This concept is mathematically expressed as follows:

$${\mathit{ℒ}}_{comp}\left({\widehat{Y}}_{DCT}[k],X_{DCT}[k]\right)={\displaystyle \sum _{l=1}^2\alpha [l]{\mathit{ℒ}}_{MSE\_Mag}\left(|{\tilde{Y}}_{DCT,\beta }^{(l)}[k]|,|{\tilde{X}}_{DCT,\beta }^{(l)}[k]|\right)}+{\displaystyle \sum _{l=1}^2\left(1-\alpha [l]\right){\mathit{ℒ}}_{MSE\_Polar}\left({\widehat{Y}}_{DCT,\beta }^{(l)}[k],X_{DCT,\beta }^{(l)}[k]\right)}$$

(9)

where $\alpha \left[l\right]$ is set to 0.5, 0.55, and 0.75 for $l=\mathrm{0,1},2$, respectively. Note that in the first term on the right-hand side of Equation (9), $|{\tilde{Y}}_{DCT,\beta }^{(l)}[k]|$ and $|{\tilde{X}}_{DCT,\beta }^{(l)}[k]|$ have replaced the original $|{\widehat{Y}}_{DCT,\beta }^{(l)}[k]|$ and $X_{DCT,\beta }^{(l)}[k]|$, respectively, for spectral magnitude difference estimation, indicating the incorporation of the perceptual loss into the formulation.

3. Experiment and Assessment Metrics

The speech samples used in our experiments were sourced from the Centre for Speech Technology Voice Cloning ToolKit (CSTR VCTK) Corpus [33], which contains utterances from 110 individuals. We utilized recordings from 58 individuals (29 males and 29 females), each contributing approximately 400 sentences for training, while the recordings from another 20 speakers (10 males and 10 females) were selected for testing. These recordings were originally sampled at 48 kHz but were down-sampled to 8 kHz for our experiments. Noise data was sourced from the Diverse Environments Multi-channel Acoustic Noise Database (DEMAND) [34], which includes six categories of ambient noise, each containing three distinct recordings. During training, noise was randomly mixed with speech at signal-to-noise ratios (SNR) of −5, 0, 5, 10, and 15 dB.

For real-time speech denoising, we employed the OLA method with a frame length $L_f=256$ and a frame shift $L_s=64$, resulting in a frame update rate of 125 times per second. This setup ensures real-time processing, as long as each frame of the input speech signal can be processed within 8 milliseconds.

The mGLFB-based U-Nets were trained on a desktop computer with an Intel^® i5-14500 CPU (Intel Corporation, Santa Clara, CA, USA) and 128 GB RAM, coupled with an NVIDIA^® RTX 4080 Super graphics card (NVIDIA Corporation, Santa Clara, CA, USA). We employed the Adam optimizer [29] and adjusted the model’s weights with the maximum batch size allowed by the hardware.

To assess the quality of the denoised speech, we employed the speech enhancement metrics proposed by Hu and Loizou [35], specifically CSIG, CBAK, and COVL, along with two quality metrics: the Perceptual Evaluation of Speech Quality (PESQ) [36] and Virtual Speech Quality Objective Listener (ViSQOL) [37]. CSIG and CBAK correspond to predicted speech and background distortion ratings, respectively, while COVL summarizes the overall quality prediction. These three metrics and ViSQOL are all based on the mean opinion score (MOS) scale, where values range from 1 (bad) to 5 (excellent). By contrast, PESQ provides a score between −0.5 and 4.5, with higher scores indicating better speech quality.

The metric for measuring speech intelligibility is the short-time objective intelligibility (STOI) [38], which ranges from 0 to 1 and is often presented as a percentage. Higher STOI scores correspond to better speech intelligibility.

For general objective evaluation, we used signal-to-noise ratio (SNR) and log-spectral distance (LSD) [39] to measure the temporal and spectral differences in the speech signals, respectively. As defined in Equation (10), SNR measures the ratio of the signal power to the noise power, providing insight into the level of noise in the signal:

$${\mathrm{SNR}}_{\left[\mathrm{dB}\right]}=10{\log }_{10}\left[{\displaystyle \frac{{\displaystyle \sum _{l=0}^{L-1}{s}^2[l]}}{{\displaystyle \sum _{l=0}^{L-1}{\left(s[l]-\widehat{s}[l]\right)}^2}}}\right],$$

(10)

where $s[l]$ and $\widehat{s}[l]$ are the original and processed speech signals with a length of $L$. While SNR is useful for understanding the noise level in a signal, it is not an ideal metric for evaluating speech quality as it does not account for spectral differences. To address this limitation, we utilized LSD to gauge the distance between the spectrograms of the original and processed speech signals in the T-F domain:

$$LSD={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=0}^{N-1}{\left\{\frac{1}{L_f}{\displaystyle \sum _{k=0}^{L_f-1}{\left[\log P_{(n)}[k]-\log {\widehat{P}}_{(n)}[k]\right]}^2}\right\}}^{1/2}},$$

(11)

where $P_{(n)}[k]$ and ${\widehat{P}}_{(n)}[k]$ represent the $k^{th}$ power spectral coefficients derived from the original and processed speech signals, respectively, at the $n^{th}$ frame, and $N$ denotes the total number of frames.

4. Performance Evaluation

Our first concern is the efficiency gained by replacing GLFBs with mGLFBs for speech denoising. With this modification, the number of learnable parameters required for constructing the 6-level U-Net decreases from 238.5 K to 206.2 K, resulting in a 13.5% reduction. We evaluated the performance of both U-Nets (built with GLFBs and mGLFBs) using the metrics discussed in the previous section.

Table 2. Performance comparison for the U-Nets built with GLFBs and mGLFBs.

Building Module	$\mathbf{Loss} \mathbf{Function} (\mathit{\alpha },\mathit{\beta })$	Initial SNR	Final SNR (dB)	LSD	CSIG	CBAK	COVL	PESQ	ViSQOL	STOI (%)
Original GLFB	(0, 1)	−2.5 dB	13.64	2.765	3.249	3.069	3.066	2.980	3.705	86.30
		2.5 dB	16.13	2.428	3.775	3.342	3.478	3.238	4.031	90.62
		7.5 dB	18.07	2.169	4.198	3.554	3.803	3.440	4.264	90.08
		12.5 dB	20.04	1.929	4.515	3.768	4.063	3.621	4.444	89.72
		Average	16.97	2.323	3.934	3.433	3.603	3.320	4.111	89.18
	(0.5, 0.5)	−2.5 dB	13.73	2.513	3.857	3.169	3.421	3.044	3.874	88.80
		2.5 dB	16.37	2.238	4.275	3.439	3.792	3.338	4.161	92.72
		7.5 dB	18.39	2.004	4.598	3.666	4.083	3.578	4.373	93.96
		12.5 dB	20.20	1.806	4.840	3.877	4.308	3.769	4.525	94.74
		Average	17.17	2.140	4.393	3.538	3.901	3.432	4.233	92.56
Modified GLFB	(0, 1)	−2.5 dB	13.40	2.769	3.328	3.043	3.094	2.962	3.711	85.14
		2.5 dB	16.03	2.424	3.832	3.316	3.493	3.218	4.029	89.56
		7.5 dB	18.04	2.137	4.246	3.544	3.822	3.433	4.280	90.28
		12.5 dB	19.97	1.895	4.555	3.763	4.081	3.619	4.468	90.51
		Average	16.86	2.306	3.990	3.416	3.622	3.308	4.122	88.87
	(0.5, 0.5)	−2.5 dB	13.65	2.530	3.872	3.168	3.428	3.039	3.855	88.55
		2.5 dB	16.20	2.248	4.286	3.443	3.798	3.336	4.144	92.93
		7.5 dB	18.27	2.019	4.603	3.669	4.084	3.571	4.358	93.99
		12.5 dB	20.15	1.818	4.856	3.885	4.320	3.772	4.518	94.23
		Average	17.07	2.153	4.404	3.541	3.907	3.430	4.219	92.42

presents the experimental results when these two U-Nets were trained using the loss function described in Equation (2) with two settings: $(\alpha ,\beta )=\left(\mathrm{0,1}\right)$ and $(\alpha ,\beta )=\left(\mathrm{0.5,0.5}\right)$.

With the exception of STOI for the case of $(\alpha ,\beta ) = \left(\mathrm{0,1}\right)$, the results for SNR, LSD, CSIG, CBAK, COVL, PESQ, and ViSQOL are comparable between the two U-Nets. These results demonstrate that the mGLFB-based U-Net performs similarly to the GLFB-based U-Net, despite using fewer learnable parameters. The lower STOI for the mGLFB-based U-Net is conceivably due to the heavy use of dilated convolutions, which can reduce the network’s ability to model local continuity or fine details.

The second concern focuses on integrating speech denoising with resolution reconstruction. Assuming the input is an 8 kHz narrowband noisy speech signal and the output is a 16 kHz denoised and super-resolved speech signal, we implemented two approaches for this enhancement task. The first approach involved denoising the speech using the mGLFB-based U-Net at 8 kHz, followed by a resolution increase to 16 kHz using spline interpolation. The second approach performed speech denoising and super-resolution using a U-Net equipped with a resampler using an upscaling factor $\rho$ of 2.

Table 3. Evaluation for 16 kHz denoised and super-resolved speech signals obtained by (1) denoising at 8 kHz followed by upsampling to 16 kHz using spline interpolation, and (2) simultaneously performing denoising and super-resolution using the mGLFB-based U-Net at 16 kHz.

Approach	Initial SNR	Final SNR (dB)	LSD	CSIG	CBAK	COVL	PESQ	ViSQOL	STOI (%)
(1) mGLFB-based U-Net (8 kHz) + spline interpolation (16 kHz)	−2.5 dB	11.40	3.741	−1.861	3.072	0.534	2.961	3.403	88.12
	2.5 dB	13.07	3.545	−1.636	3.330	0.815	3.269	3.644	92.62
	7.5 dB	14.24	3.359	−1.442	3.550	1.056	3.534	3.831	93.82
	12.5 dB	14.99	3.224	−1.273	3.740	1.255	3.743	3.965	94.35
	Average	13.42	3.467	−1.553	3.423	0.915	3.377	3.711	92.22
(2) mGLFB-based U-Net (16 kHz)	−2.5 dB	11.37	2.807	3.384	3.017	3.103	2.896	3.510	87.35
	2.5 dB	13.23	2.583	3.795	3.275	3.470	3.188	3.802	92.24
	7.5 dB	14.69	2.421	4.124	3.501	3.774	3.443	4.025	94.43
	12.5 dB	15.66	2.313	4.359	3.694	4.002	3.645	4.174	95.15
	Average	13.74	2.531	3.916	3.372	3.587	3.293	3.878	92.29

presents the measured scores for the metrics used to assess speech quality. Compared to the results in Table 2, all metrics show a declining tendency, primarily due to the increase in frequency range. After the bandwidth expansion from 8 to 16 kHz, overall speech quality decreased by approximately 0.137 in PESQ (from 3.430 to 3.293) and 0.341 in ViSQOL (from 4.219 to 3.878). Regarding the advantages and disadvantages of the two approaches, the results in Table 2 reveal a notable divergence. While the commonly used LSD metric favors the second approach, PESQ indicates a preference for the first approach. Nevertheless, ViSQOL aligns with LSD, indicating better perceived quality for the second approach.

Given the importance of real-time processing for practical applications, we conducted a comprehensive performance comparison of the real-time factor (RTF) across two CPUs (Intel^® i5-14500 and i9-11900: Intel Corporation, Santa Clara, CA, USA) and two GPUs (NVIDIA^® RTX-3080Ti and RTX-4080S: NVIDIA Corporation, Santa Clara, CA, USA) using the MATLAB^® environment (MathWorks, Natick, MA, USA). For the first approach, we specifically compare two denoising U-Nets built on GLFBs and mGLFBs. As shown in

Table 4. Comparison of real-time performance.

Approach	Model	Parameters	Real-Time Factor
			Intel® i5-14500	Intel® i9-11900	NVIDIA® RTX-3080Ti	NVIDIA® RTX-4080S
Approach (1)	GLFB-based U-Net (8 kHz) + spline interpolation (16 kHz)	238.5 K	0.338	0.381	0.053	0.046
	mGLFB-based U-Net (8 kHz) + spline interpolation (16 kHz)	206.2 K	0.450	0.447	0.100	0.071
Approach (2)	mGLFB-based U-Net (16 kHz)	207.3 K	0.482	0.476	0.107	0.070

, the highest observed RTF was 0.482 when the second approach was executed on the i5-14500 processor. Since all RTF values are less than 1, these results confirm that every tested processor can handle speech data in real time.

Table 4 contains two variants of Approach 1, where U-Nets constructed using GLFBs and mGLFBs are studied. In theory, mGLFB should require less computation because it uses smaller kernels and fewer channels in the depthwise convolution. However, the first two rows of Table 4 show the opposite trend: the GLFB-based U-Net achieves a lower RTF than the mGLFB-based U-Net. We attribute this discrepancy to our implementation, where the program code adopts a highly modular, layer-wise design. As illustrated in Figure 3, each mGLFB block includes an additional convolutional layer and a gating layer compared with GLFB. In our implementation, these are invoked as separate functions, which introduces extra function-call and data-passing overhead that offsets the theoretical savings and slows the overall runtime. To fully realize mGLFB’s efficiency, the computations within the depthwise convolution should be fused into a single unit in future revisions.

To further investigate the differences in quality metrics reported in Table 3, we analyzed the spectrograms produced by different methods. As illustrated in Figure 6, incorporating the modified loss function from Equation (9) enables the U-Net to more effectively recover high-frequency components, especially in unvoiced segments. However, in the second approach, the U-Net is less successful at reconstructing the spectral range between 4 and 6 kHz. In contrast, applying spline interpolation after denoising at an 8 kHz sampling rate tends to produce stronger spectral intensity in the 4–7 kHz range, regardless of speech type. As a result, PESQ considers the intensified spectra in the 4–7 kHz region to be more perceptually acceptable, even though the overall LSD may indicate otherwise.

Finally, in order to assess the contribution of the perceptual loss function, we also conducted an ablation study focusing on the impact of the weight factor and spectral smoothing.

Table 5. Effects of the weight factor and spectral smoothing in the perceptual loss function.

Approach (2) mGLFB-Based U-Net (16 kHz)	Initial SNR	Final SNR (dB)	LSD	CSIG	CBAK	COVL	PESQ	ViSQOL	STOI (%)
$\mathit{\alpha }=\left[0.5 0.5 0.5\right];$  Spectral Smoothing in high frequencies: OFF	−2.5 dB	10.99	3.086	3.221	2.952	2.985	2.835	3.316	85.72
	2.5 dB	12.97	2.945	3.518	3.224	3.301	3.134	3.603	90.89
	7.5 dB	14.39	2.838	3.737	3.453	3.553	3.393	3.812	92.82
	12.5 dB	15.35	2.755	3.908	3.655	3.758	3.610	3.957	93.83
	Average	13.43	2.906	3.596	3.321	3.399	3.243	3.672	90.81
$\mathit{\alpha }=\left[0.5 0.55 0.75\right];$  Spectral Smoothing in high frequencies: OFF	−2.5 dB	11.11	2.864	3.330	2.973	3.051	2.851	3.423	85.99
	2.5 dB	13.06	2.658	3.702	3.239	3.404	3.152	3.713	90.47
	7.5 dB	14.49	2.513	3.987	3.463	3.685	3.405	3.925	92.03
	12.5 dB	15.46	2.410	4.209	3.663	3.912	3.618	4.078	92.89
	Average	13.53	2.611	3.807	3.335	3.513	3.256	3.784	90.34
$\mathit{\alpha }=\left[0.5 0.5 0.5\right];$  Spectral Smoothing in high frequencies: ON	−2.5 dB	11.02	2.883	3.294	2.957	3.027	2.845	3.417	85.51
	2.5 dB	13.00	2.663	3.693	3.229	3.393	3.144	3.722	90.64
	7.5 dB	14.41	2.508	3.997	3.456	3.685	3.399	3.942	92.52
	12.5 dB	15.36	2.395	4.226	3.653	3.915	3.608	4.095	93.18
	Average	13.45	2.612	3.803	3.324	3.505	3.249	3.794	90.46

summarizes the metrics obtained under three experimental conditions. The top section shows the results using $\mathit{\alpha }=\left[0.5 0.5 0.5\right]$ with spectral smoothing disabled, which is equivalent to utilizing the original loss formulation (Equation (2)). As indicated in the middle and lower sections of Table 5, adjusting $\mathit{\alpha }$ to [0.5, 0.55, 0.75] reduces LSD while increasing CSIG, COVL, and ViSQOL, suggesting that prioritizing differences in high-frequency spectral magnitude benefits overall speech quality. Enabling spectral smoothing alone in the perceptual loss function can also yield similar improvements. The combined effect of these two enhancements, as previously shown in the bottom section of Table 3, results in further increases across all performance metrics, including STOI.

5. Conclusions

In this study, we utilized the mGLFB as the core building block to efficiently construct a 6-level U-Net for real-time speech enhancement. Additionally, we developed a frequency-adaptive perceptual loss function designed to improve spectral recovery in high-frequency regions. By integrating a resampler at the network output, the mGLFB-based U-Net is capable of performing both denoising and super-resolution simultaneously. Experimental results demonstrate that the proposed model achieves speech denoising with a more compact architecture, without compromising quality performance. However, the current U-Net shows limitations in capturing high-frequency content, particularly in the high-energy spectra within the 4–6 kHz range.

For future work, we plan to refine the bandwidth extension branch of the mGLFB-based U-Net to further enhance the predictability of high-frequency components. Additionally, in order to improve the restoration of high-frequency details, we intend to incorporate PESQ and ViSQOL scores into adversarial loss functions within a GAN training framework.