A Multi-Resolution Approach to GAN-Based Speech Enhancement

1. Introduction

Speech enhancement is essential for various speech applications such as robust speech recognition, hearing aids, and mobile communications [1,2,3,4]. The main objective of speech enhancement is to improve the quality and intelligibility of the noisy speech by suppressing the background noise or interferences.

In the early studies on speech enhancement, the minimum mean-square error (MMSE)-based spectral amplitude estimator algorithms [5,6] were popular producing enhanced signal with low residual noise. However, the MMSE-based methods have been reported ineffective in non-stationary noise environments due to their stationarity assumption on speech and noise. An effective way to deal with the non-stationary noise is to utilize a priori information extracted from a speech or noise database (DB), called the template-based speech enhancement techniques. One of the most well-known template-based schemes is the non-negative matrix factorization (NMF)-based speech enhancement technique [7,8]. NMF is a latent factor analysis technique to discover the underlying part-based non-negative representations of the given data. Since there is no strict assumption on the speech and noise distributions, the NMF-based speech enhancement technique shows robustness to non-stationary noise environments. Since, however, the NMF-based algorithm assumes that all data is described as a linear combination of finite bases, it is known to suffer from speech distortion not covered by this representational form.

In the past few years, deep neural network (DNN)-based speech enhancement has received tremendous attention due to its ability to model complex mappings [9,10,11,12]. These methods map the noisy spectrogram to the clean spectrogram via the neural networks such as the convolutional neural network (CNN) [11] or recurrent neural network (RNN) [12]. Although the DNN-based speech enhancement techniques have shown promising performance, most of the techniques typically focus on modifying the magnitude spectra. This could cause a phase mismatch between the clean and enhanced speech since the DNN-based speech enhancement methods usually reuse the noisy phase for waveform reconstruction. For this reason, there has been growing interest in phase-aware speech enhancement [13,14,15] that exploits the phase information during the training and reconstruction. To circumvent the difficulty of the phase estimation, end-to-end (E2E) speech enhancement technique which directly enhances noisy speech waveform in the time domain has been developed [16,17,18]. Since the E2E speech enhancement techniques are performed in a waveform-to-waveform manner without any consideration of the spectra, their performance is not dependant on the accuracy of the phase estimation.

The E2E approaches, however, rely on a distance-based loss functions between the time-domain waveforms. Since these distance-based costs do not take human perception into account, the E2E approaches are not guaranteed to achieve good human-perception-related metrics, e.g., the perceptual evaluation of speech quality (PESQ) [19], short-time objective intelligibility (STOI) [20], and etc. Recently, generative adversarial network (GAN) [21]-based speech enhancement techniques have been developed to overcome the limitation of the distance-based costs [22,23,24,25,26]. Adversarial losses of GAN provide an alternative objective function to reflect the human auditory property, which can make the distribution of the enhanced speech close to that of the clean speech. To our knowledge, SEGAN [22] was the first attempt to apply GAN to the speech enhancement task, which used the noisy speech as a conditional information for a conditional GAN (cGAN) [27]. In [26], an approach to replace a vanilla GAN with advanced GAN, such as Wasserstein GAN (WGAN) [28] or relativistic standard GAN (RSGAN) [29] was proposed based on the SEGAN framework.

Even though the GAN-based speech enhancement techniques have been found successful, there still remain two important issues: (1) training instability and (2) a lack in considering various speech characteristics. Since GAN aims at finding the Nash equilibrium to solve a mini-max problem, it has been known that the training is usually unstable. A number of efforts have been devoted to stabilize the GAN training in image processing, by modifying the loss function [28] or the generator and discriminator structures [30,31]. However, in speech processing, this problem has not been extensively studied yet. Moreover, since most of the GAN-based speech enhancement techniques directly employ the models used in image generation, it is necessary to modify them to suit the inherent nature of speech. For instance, the GAN-based speech enhancement techniques [22,24,26] commonly used U-Net generator originated from image processing tasks. Since the U-net generator consisted of multiple CNN layers, it was insufficient to reflect the temporal information of speech signal. In regression-based speech enhancement, the modified U-net structure adding RNN layers to capture the temporal information of speech showed prominent performances [32]. In [33] for the speech synthesis, an alternative loss function depended on multiple sizes of window length and fast Fourier transform (FFT) was proposed and generated a good quality of speech, which also considered speech characteristics in frequency domain.

In this paper, we propose novel generator and discriminator structures for the GAN-based speech enhancement which reflect the speech characteristics while ensuring stable training. The conventional generator is trained to find a mapping function from the noisy speech to the clean speech by using sequential convolution layers, which is considered an ineffective approach especially for speech data. In contrast, the proposed generator progressively estimates the wide frequency range of the clean speech via a novel up-sampling layer.

In the early stage of GAN training, it is too easy for the conventional discriminator to differentiate real samples from fake samples for high-dimensional data. This often lets GAN fail to reach the equilibrium point due to vanishing gradient [30]. To address this issue, we propose a multi-scale discriminator that is composed of multiple sub-discriminators processing speech samples at different sampling rates. Even if the training is in the early stage, the sub-discriminators at low-sampling rates can not easily differentiate the real samples from the fake, which contributes to stabilize the training. Empirical results showed that the proposed generator and discriminator were successful in stabilizing GAN training and outperformed the conventional GAN-based speech enhancement techniques. The main contributions of this paper are summarized as follows:

• We propose a progressive generator to reflect the multi-resolution characteristics of speech.
• We propose a multi-scale discriminator to stabilize the GAN training without additional complex training techniques.
• The experimental results showed that the multi-scale structure is an effective solution for both deterministic and GAN-based models, outperforming the conventional GAN-based speech enhancement techniques.

The rest of the paper is organized as follows: In Section 2, we introduce GAN-based speech enhancement. In Section 3, we present the progressive generator and multi-scale discriminator. Section 4 describes the experimental settings and performance measurements. In Section 5, we analyze the experimental results. We draw conclusions in Section 6.

2. GAN-Based Speech Enhancement

An adversarial network models the complex distribution of the real data via a two-player mini-max game between a generator and a discriminator. Specifically, the generator takes a randomly sampled noise vector z as input and produces a fake sample $G\left(z\right)$ to fool the discriminator. On the other hand, the discriminator is a binary classifier that decides whether an input sample is real or fake. In order to generate a realistic sample, the generator is trained to deceive the discriminator, while the discriminator is trained to distinguish between the real sample and $G\left(z\right)$. In an adversarial training process, the generator and the discriminator are alternatively trained to minimize their respective loss functions. The loss functions for the standard GAN can be defined as follows:

$$L_G={\mathbb{E}}_{z\sim {\mathbb{P}}_z\left(z\right)}\left[log(1-D\left(G\left(z\right)\right))\right],$$

(1)

$$L_D=-{\mathbb{E}}_{x\sim {\mathbb{P}}_{clean}\left(x\right)}\left[log\left(D\left(x\right)\right)\right]-{\mathbb{E}}_{z\sim {\mathbb{P}}_z\left(z\right)}\left[log(1-D\left(G\left(z\right)\right))\right]$$

(2)

where z is a randomly sampled vector from ${\mathbb{P}}_z\left(z\right)$ which is usually a normal distribution, and ${\mathbb{P}}_{clean}\left(x\right)$ is the distribution of the clean speech in the training dataset.

Since GAN was initially proposed for unconditional image generation that has no exact target, it is inadequate to directly apply GAN to speech enhancement which is a regression task to estimate the clean target corresponding to the noisy input. For this reason, GAN-based speech enhancement employs a conditional generation framework [27] where both the generator and discriminator are conditioned on the noisy waveform c that has the clean waveform x as the target. By concatenating the noisy waveform c with the randomly sampled vector z, the generator G can produce a sample that is closer to the clean waveform x. The training process of the cGAN-based speech enhancement is shown in Figure 1a, and the loss functions of the cGAN-based speech enhancement are

$$L_G={\mathbb{E}}_{z\sim {\mathbb{P}}_z\left(z\right),c\sim {\mathbb{P}}_{noisy}\left(c\right)}\left[log(1-D(G(z,c),c))\right],$$

(3)

$$L_D=-{\mathbb{E}}_{x\sim {\mathbb{P}}_{clean}\left(x\right),c\sim {\mathbb{P}}_{noisy}\left(c\right)}\left[logD(x,c)\right]-{\mathbb{E}}_{z\sim {\mathbb{P}}_z\left(z\right),c\sim {\mathbb{P}}_{noisy}\left(c\right)}\left[log(1-D(G(z,c),c))\right]$$

(4)

where ${\mathbb{P}}_{clean}\left(x\right)$ and ${\mathbb{P}}_{noisy}\left(c\right)$ are respectively the distributions of the clean and noisy speech in the training dataset.

In the conventional training of the cGAN, both the probabilities that a sample is from the real data $D(x,c)$ and generated data $D\left(G\right(z,c),c)$ should reach the ideal equilibrium point $0.5$. However, unlike the expected ideal equilibrium point, they both have a tendency to become 1 because the generator can not influence the probability of the real sample $D(x,c)$. In order to alleviate this problem, RSGAN [29] proposed a discriminator to estimate the probability that the real sample is more realistic than the generated sample. The proposed discriminator makes the probability of the generated sample $D\left(G\right(z,c),c)$ increase when that of the real sample $D(x,c)$ decreases so that both the probabilities could stably reach the Nash equilibrium state. In [26], the experimental results showed that, compared to other conventional GAN-based speech enhancements, the RSGAN-based speech enhancement technique improved the stability of training and enhanced the speech quality. The training process of the RSGAN-based speech enhancement is given in Figure 1b, and the loss functions of RSGAN-based speech enhancement can be written as:

$$L_G=-{\mathbb{E}}_{(x_r,x_f)\sim ({\mathbb{P}}_r,{\mathbb{P}}_f)}\left[log\left(\sigma (C\left(x_f\right)-C\left(x_r\right))\right)\right],$$

(5)

$$L_D=-{\mathbb{E}}_{(x_r,x_f)\sim ({\mathbb{P}}_r,{\mathbb{P}}_f)}\left[log\left(\sigma (C\left(x_r\right)-C\left(x_f\right))\right)\right]$$

(6)

where the real and fake data-pairs are defined as $x_r\triangleq (x,c)\sim {\mathbb{P}}_r$ and $x_f\triangleq (G(z,c),c)\sim {\mathbb{P}}_f$, and $C\left(x\right)$ is the output of the last layer in discriminator before the sigmoid activation function $\sigma (·)$, i.e., $D\left(x\right)=\sigma \left(C\right(x\left)\right)$.

In order to stabilize GAN training, there are two penalties commonly used: A gradient penalty for discriminator [28] and $L_1$ loss penalty for generator [24]. First, the gradient penalty regularization for discriminator is used to prevent exploding or vanishing gradients. This regularization penalizes the model if the $L_2$ norm of the discriminator gradient moves away from 1 to satisfy the Lipschitz constraint. The modified discriminator loss functions with the gradient penalty are as follows:

$$L_{GP}\left(D\right)={\mathbb{E}}_{\tilde{x},c\sim \tilde{\mathbb{P}}}\left[\left(\right||{▽}_{\tilde{x},c}C(\tilde{x},c){\left)\right||}_2{-1)}^2\right],$$

(7)

$$L_{D-GP}\left(D\right)=-{\mathbb{E}}_{(x_r,x_f)\sim ({\mathbb{P}}_r,{\mathbb{P}}_f)}\left[log\left(\sigma (C\left(x_r\right)-C\left(x_f\right))\right)\right]+{\lambda }_{GP}{L}_{GP}\left(D\right)$$

(8)

where $\tilde{\mathbb{P}}$ is the joint distribution of c and $\tilde{x}=ϵx+(1-ϵ)\widehat{x}$, $ϵ$ is sampled from a uniform distribution in $[0,1]$, and $\widehat{x}$ is the sample from $G(z,c)$. ${\lambda }_{GP}$ is the hyper-parameter that controls the gradient penalty loss and the adversarial loss of the discriminator.

Second, several prior studies [22,23,24] found that it is effective to use an additional loss term that minimizes the $L_1$ loss between the clean speech x and the generated speech $G(z,c)$ for the generator training. The modified generator loss with the $L_1$ loss is defined as

$$L_1\left(G\right)={\parallel G(z,c)-x\parallel }_1,$$

(9)

$$L_{G-L_1}\left(G\right)=-{\mathbb{E}}_{(x_r,x_f)\sim ({\mathbb{P}}_r,{\mathbb{P}}_f)}\left[log\left(\sigma (C\left(x_f\right)-C\left(x_r\right))\right)\right]+{\lambda }_{L_1}{L}_1\left(G\right)$$

(10)

where ${∥·∥}_1$ is $L_1$ norm, and ${\lambda }_{L_1}$ is a hyper-parameter for balancing the $L_1$ loss and the adversarial loss of the generator.

3. Multi-Resolution Approach for Speech Enhancement

In this section, we propose a novel GAN-based speech enhancement model which consists of a progressive generator and a multi-scale discriminator. The overall architecture of the proposed model is shown in Figure 2, and the details of the progressive generator and the multi-scale discriminator are given in Figure 3.

3.1. Progressive Generator

Conventionally, GAN-based speech enhancement systems adopt U-Net generator [22] which is composed of two components: An encoder $G_{enc}$ and a decoder $G_{dec}$. The encoder $G_{enc}$ consists of repeated convolutional layers to produce compressed latent vectors from a noisy speech, and the decoder $G_{dec}$ contains multiple transposed convolutional layers to restore the clean speech from the compressed latent vectors. These transposed convolutional layers in $G_{dec}$ are known to be able to generate low-resolution data from the compressed latent vectors, however, the capability to generate a high-resolution data is severely limited [30]. Especially in the case of speech data, it is difficult for the transposed convolutional layers to generate the speech with a high-sampling rate because it should cover a wide frequency range.

Motivated from the progressive GAN, which starts with generating low-resolution images and then progressively increases the resolution [30,31], we propose a novel generator that can incrementally widen the frequency band of the speech by applying an up-sampling block to the decoder $G_{dec}$. As shown in Figure 3, the proposed up-sampling block consists of 1D-convolution layers, element-wise addition, and liner interpolation layers. The up-sampling block yields the intermediate enhanced speech $G_n(z,c)$ at each layer through the 1D convolution layer and element-wise addition so that the wide frequency band of the clean speech is progressively estimated. Since a sampling rate is increased through the linear interpolation layer, it is possible to generate the intermediate enhanced speech at the higher layer while maintaining estimated frequency components at the lower layer. This incremental process is repeated until the sampling rate reaches the target sampling rate which is $16\mathrm{kHz}$ in our experiment. Finally, we exploit the down-sampled clean speech $x_n$ processed by low-pass filtering and decimation as the target for each layer to provide multi-resolution loss functions. We define the real and fake data-pairs at different sampling rates as $x_{r_n}\triangleq (x_n,c_n)\sim {\mathbb{P}}_{r_n}$ and $x_{f_n}\triangleq (G_n(z,c),c_n)\sim {\mathbb{P}}_{f_n}$, and the proposed multi-resolution loss functions with $L_1$ loss are given as follows:

$$\begin{array}{cc}\hfill L_G\left(p\right)& =\sum _{\begin{array}{c}n\ge p\\ n\in N_G\end{array}}{L}_{G_n}+{\lambda }_{L_1}{L}_1\left(G_n\right),N_G\in \left\{1k,2k,4k,8k,16k\right\},\hfill \\ \hfill & =\sum _{\begin{array}{c}n\ge p\\ n\in N_G\end{array}}-{\mathbb{E}}_{(x_{r_n},x_{f_n})\sim ({\mathbb{P}}_{r_n},{\mathbb{P}}_{f_n})}\left[log\left(\sigma (C_n\left(x_{f_n}\right)-C_n\left(x_{r_n}\right))\right)\right]+{\lambda }_{L_1}{\parallel G_n(z,c)-x_n\parallel }_1\hfill \end{array}$$

(11)

where $N_G$ is the possible set of n for the proposed generator, and p is the sampling rate at which the intermediate enhanced speech is firstly obtained.

3.2. Multi-Scale Discriminator

When generating high-resolution image and speech data in the early stage of training, it is hard for the generator to produce a realistic sample due to the insufficient model capacity. Therefore, the discriminator can easily differentiate the generated samples from the real samples, which means that the real and fake data distributions do not have substantial overlap. This problem often causes training instability and even mode collapses [30]. For the stabilization of the training, we propose a multi-scale discriminator that consists of multiple sub-discriminators treating speech samples at different sampling rates.

As presented in Figure 3, the intermediate enhanced speech $G_n(z,c)$ at each layer restores the down-sampled clean speech $x_n$. Based on this, we can utilize the intermediate enhanced speech and down-sampled clean speech as the input to each sub-discriminator $D_n$. Since each sub-discriminator can only access limited frequency information depending on the sampling rate, we can make each sub-discriminator solve different levels of discrimination tasks. For example, discriminating the real from the generated speech is more difficult at the lower sampling rate than at the higher rate. The sub-discriminator at a lower sampling rate plays an important role in stabilizing the early stage of the training. As the training progresses, the role shifts upwards to the sub-discriminators at higher sampling rates. Finally, the proposed multi-scale loss for discriminator with gradient penalty is given by

$$\begin{array}{cc}\hfill L_D\left(q\right)& =\sum _{\begin{array}{c}n\ge q\\ n\in N_D\end{array}}{L}_{D_n}+{\lambda }_{GP}{L}_{GP}\left(D_n\right),N_D\in \left\{1k,2k,4k,8k,16k\right\},\hfill \\ \hfill & =\sum _{\begin{array}{c}n\ge q\\ n\in N_D\end{array}}-{\mathbb{E}}_{(x_{r_n},x_{f_n})\sim ({\mathbb{P}}_{r_n},{\mathbb{P}}_{f_n})}\left[log\left(\sigma (C_n\left(x_{r_n}\right)-C\left(x_{f_n}\right))\right)\right]+{\lambda }_{GP}{\mathbb{E}}_{\widetilde{x_n},c_n\sim \widetilde{{\mathbb{P}}_n}}\left[\right(\left|\right|{▽}_{\widetilde{x_n},c_n}C(\widetilde{x_n},c_n){\left)\right||}_2-1{)}^2]\hfill \end{array}$$

(12)

where $\widetilde{{\mathbb{P}}_n}$ is the joint distribution of the down-sampled noisy speech $c_n$ and $\widetilde{x_n}=ϵx_n+(1-ϵ)\widehat{x_n}$, $ϵ$ is sampled from a uniform distribution in $[0,1]$, $x_n$ is the down-sampled clean speech, and $\widehat{x_n}$ is the sample from $G_n(z,c)$. $N_D$ is the possible set of n for the proposed discriminator, and q is the minimum sampling rate at which the intermediate enhanced output was utilized as the input to a sub-discriminator for the first time. The adversarial losses $L_{D_n}$ are equally weighted.

4. Experimental Settings

4.1. Dataset

We used a publicly available dataset in [34] for evaluating the performance of the proposed speech enhancement technique. The dataset consists of 30 speakers from the Voice Bank corpus [35], and used 28 speakers (14 male and 14 female) for the training set (11572 utterances) and 2 speakers (one male and one female) for the test set (824 utterances). The training set simulated a total of 40 noisy conditions with 10 different noise sources (2 artificial and 8 from the DEMAND database [36]) at signal-to-noise ratios (SNRs) of 0, 5, 10, and 15 dB. The test set was created using 5 noise sources (living room, office, bus, cafeteria, and public square noise from the DEMAND database), which were different from the training noises, added at SNRs 2.5, 7.5, 12.5, and 17.5 dB. The training and test sets were down-sampled from $48\mathrm{kHz}$ to $16\mathrm{kHz}$.

4.2. Network Structure

The configuration of the proposed generator is described in

Table 1. Architecture of the proposed generator. Output shape represented temporal dimension and feature maps.

Block	Operation		Output Shape
Input			$16,384\times 1$
Encoder	Conv1D (filterlength = 31, stide = 2)		$8192\times 16$
			$4096\times 32$
			$2048\times 32$
			$1024\times 64$
			$512\times 64$
			$256\times 128$
			$128\times 128$
			$64\times 256$
			$32\times 256$
			$16\times 512$
Latent vector			$8\times 1024$
Decoder	Trconv (filterlength = 31, stide = 2)		$16\times 1024$
			$32\times 512$
			$64\times 512$
			$128\times 256$
			$256\times 256$
			$512\times 128$
	Trconv (filterlength = 31, stide = 2)	Conv1D (filterlength = 17, stide = 1) Element-wise addition Linear interpolation layer	$1024\times 128$	$1024\times 1$
			$2048\times 64$	$2048\times 1$
			$4096\times 64$	$4096\times 1$
			$8192\times 32$	$8192\times 1$
			$\mathrm{16,384}\times 1$

. We used the U-Net structure with 11 convolutional layers for the encoder $G_{enc}$ and the decoder $G_{dec}$ as in [22,26]. Output shapes at each layer were represented by the number of temporal dimensions and feature maps. Conv1D in the encoder denotes a one-dimensional convolutional layer, and TrConv in the decoder means a transposed convolutional layer. We used approximately 1 s of speech (16,384 samples) as the input to the encoder. The last output of the encoder was concatenated with a noise which had the shape of $8\times 1024$ randomly sampled from the standard normal distribution $N(0,1)$. In [27], it was reported that the generator usually learns to ignore the noise prior z in the CGAN, and we also observed a similar tendency in our experiments. For this reason, we removed the noise from the input, and the shape of the latent vector became $8\times 1024$. The architecture of $G_{dec}$ was a mirroring of $G_{enc}$ with the same number and width of the filters per layer. However, skip connections from $G_{enc}$ made the number of feature maps in every layer to be doubled. The proposed up-sampling block $G_{up}$ consisted of 1D convolution layers, element-wise addition operations, and linear interpolation layers.

In this experiment, the proposed discriminator had the same serial convolutional layers as $G_{enc}$. The input to the discriminator had two channels of 16,384 samples, which were the clean speech and enhanced speech. The rest of the temporal dimension and feature-maps were the same as those of $G_{enc}$. In addition, we used LeakyReLU activation function without a normalization technique. After the last convolutional layers, there were a $1\times 1$ convolution, and its output was fed to a fully-connected layer. To construct the proposed multi-scale discriminator, we used 5 different sub-discriminators, which were $D_{16k},D_{8k},D_{4k},D_{2k},and{D}_{1k}$ trained according to in Equation (12). Each sub-discriminator had a different input dimension depending on the sampling rate.

The model was trained using the Adam optimizer [37] for 80 epochs with 0.0002 learning rate for both the generator and discriminator. The batch size was 50 with 1-s audio signals that were sliced using windows of length 16,384 with 8192 overlaps. We also applied a pre-emphasis filter with impulse response $[-0.95,1]$ to all training samples. For inference, the enhanced signals were reconstructed through overlap-add. The hyper-parameters to balance the penalty terms were set as ${\lambda }_{L_1}$ = 200 and ${\lambda }_{GP}$ = 10 such that they could match the dynamic range of magnitude with respect to the generator and discriminator losses. Note that we gave the same weight to the adversarial losses, $L_{G_n}$ and $L_{D_n}$, for all $n\in \left\{1k,2k,4k,8k,16k\right\}$. We implemented all the networks using Keras with Tensorflow [38] back-end using the public code (The SERGAN framework is available at https://github.com/deepakbaby/se_relativisticgan). All training was performed on single Titan RTX 24 GB GPU, and it took around 2 days.

4.3. Evaluation Methods

4.3.1. Objective Evaluation

The quality of the enhanced speech was evaluated using the following objective metrics:

• PESQ: Perceptual evaluation of speech quality defined in the ITU-T P.862 standard [19] (from −0.5 to 4.5),
• STOI: Short-time objective intelligibility [20] (from 0 to 1),
• CSIG: Mean opinion score (MOS) prediction of the signal distortion attending only to the speech signal [39] (from 1 to 5),
• CBAK: MOS prediction of the intrusiveness of background noise [39] (from 1 to 5),
• COVL: MOS prediction of the overall effect [39] (from 1 to 5).

4.3.2. Subjective Evaluation

To compare the subjective quality of the enhanced speech by baseline and proposed methods, we conducted two pairs of AB preference tests: AECNN versus the progressive generator and SERGAN versus the progressive generator with the multi-scale discriminator. Two speech in each pair were given in arbitrary order. For each listening test, 14 listeners participated, and 50 pairs of the speech were randomly selected. Listeners could listen to the speech pairs as many times as they wanted and were instructed to choose the speech with better perceptual quality. If the quality of the two samples was indistinguishable, listeners could select no preference.

5. Experiments and Results

In order to investigate the individual effect of the proposed generator and discriminator, we experimented on the progressive generator with and without the multi-scale discriminator. Furthermore, we plotted $L_1$ losses at each layer $L_1\left(G_n\right)$ to show that the proposed model makes training fast and stable. Finally, the performance of the proposed model is compared with that of the other GAN-based speech enhancement techniques.

5.1. Performance of Progressive Generator

5.1.1. Objective Results

The purpose of these experiments is to show the effectiveness of the progressive generator.

Table 2. Comparison of results between different architectures of the progressive generator. The best model is shown in bold type.

Model	${\sum }_{\mathit{n}\ge \mathit{p}}{\mathit{L}}_1\left({\mathit{G}}_{\mathit{n}}\right)$	PESQ
AECNN [26]	$p=16k$	2.5873
Proposed	$p=8k$	2.6257
	$p=4k$	2.6335
	$p=2k$	2.6407
	$\mathit{p}=\mathbf{1}\mathit{k}$	2.6516

presents the performance of the proposed generator when we minimized only the $L_1\left(G_n\right)$ in Equation (11). In order to better understand the influence of the progressive structure on the PESQ score, we conducted an ablation study with different p in ${\sum }_{n\ge p}{L}_1\left(G_n\right)$. As illustrated in Table 2, compared to the auto-encoder CNN (AECNN) [26] that is the conventional U-net generator minimizing the $L_1$ loss only, the PESQ score of the progressive generator improved from $2.5873$ to $2.6516$. Furthermore, for the smaller p, we got a better PESQ score, and the best PESQ score was achieved when p was the lowest, i.e., $1k$. For enhancing high-resolution speech, we verified that it is very useful to progressively generate intermediate enhanced speech while maintaining the estimated information obtained at lower sampling rate. We used the best generator $p=1k$ in Table 2 for the subsequent experiments.

5.1.2. Subjective Results

The preference score of AECNN and the progressive generator was shown in Figure 4a. The progressive generator was preferred to AECNN in 43.08% of the cases, while the opposite preference was 25.38% (no preference in 31.54% of the cases). From the results, we verified that the proposed generator could produce the speech with not only higher objective measurements but also better perceptual quality.

5.2. Performance of Multi-Scale Discriminator

5.2.1. Objective Results

The goal of these experiments is to show the efficiency of the multi-scale discriminator compared to the conventional single discriminator. As shown in

Table 3. Comparison of results between different architectures of the multi-scale discriminator. Except for the SERGAN, the generator of all architectures used the best model in Table 2. The best model is shown in bold type.

Model	Generator	Discriminator	${\mathit{L}}_{\mathit{D}}\left(\mathit{q}\right)$	PESQ	RTF
SERGAN [26]	U-net	Single	$q=16k$	2.5898	0.008
Proposed	Progressive	Single	$q=16k$	2.6514	0.010
	Progressive	Multi-scale	$q=8k$	2.6541
			$\mathit{q}=\mathbf{4}\mathit{k}$	2.7077
			$q=2k$	2.6664
			$q=1k$	2.6700

, we evaluated the performance of the multi-scale discriminator while varying q of the multi-scale loss $L_D\left(q\right)$ in Equation (12), which means varying the number of sub-discriminators. Compared to the baseline proposed in [26], the progressive generator with the single discriminator showed an improved PESQ score from $2.5898$ to $2.6514$. The multi-scale discriminators outperformed the single discriminators, and the best PESQ score of $2.7077$ was obtained when $q=4k$. Interestingly, we could observe that the performance was degraded when the q became below $4k$. One possible explanation for this phenomenon would be that since the progressive generator faithfully generated the speech below the 4 kHz sampling rate, it was difficult for the discriminator to differentiate the fake from the real speech. This let the additional sub-discriminators a little bit useless for performance improvement.

5.2.2. Subjective Results

The preference scores of SERGAN and the progressive generator with multi-scale discriminator were shown in Figure 4b. The proposed method was preferred over SERGAN in 42.00% of the cases, while SERGAN was preferred in 26.31% of the cases (no preference in 31.69% of the cases). These results showed that the proposed method could enhance the speech with better objective metrics and subjective perceptual scores.

5.2.3. Real-Time Feasibility

SERGAN and the proposed method were evaluated in terms of the real-time factor(RTF) to verify the real-time feasibility, which is defined as the ratio of the time taken to enhance the speech to the duration of the speech (small factors indicate faster processing). CPU and graphic card used for the experiment were Intel Xeon Silver 4214 CPU 2.20 GHz and single Nvidia Titan RTX 24 GB. Since the generator of AECNN and SERGAN is the same, their RTF has the same value. Therefore, we only compared the RTF of SERGAN and the proposed method in Table 3. As the input window length was about 1 s of speech (16,384 samples), and the overlap was 0.5 s of speech (8192 samples), the total processing delay of all models can be computed by the sum of the 0.5 s and the actual processing time of the algorithm. In Table 3, we observed that the RTF of SERGAN and the proposed model was small enough for the semi-real-time applications. The similar value of the RTF for SEGAN and the proposed model also verified that adding the up-sampling network did not significantly increase the computational complexity.

5.3. Analysis and Comparison of Spectorgrams

An example of the spectrograms of clean speech, noisy speech, and the enhanced speech by different models are shown in Figure 5. First, we focused on the black box to verify the effectiveness of the progressive generator. Before 0.6 s, a non-speech period, we could observe that the noise containing wide-band frequencies was considerably reduced since the progressive generator incrementally estimated the wide frequency range of the clean speech. Second, when we compared spectrograms of the multi-scale discriminator and that of the single discriminator, the different pattern was presented in the red box. The multi-scale discriminator was able to suppress more noise than the single discriminator in the non-speech period. We could confirm that the multi-scale discriminator selectively reduced high-frequency noise in a speech period as the sub-discriminators in multi-scale discriminator differentiate the real and fake speech at the different sampling rates.

5.4. Fast and Stable Training of Proposed Model

To analyze the learning behavior of the proposed model in more depth, we plotted $L_1\left(G_n\right)$ in Equation (11) obtained from the best model in Table 3 and SERGAN [26] during the whole training periods. As the clean speech was progressively estimated by the intermediate enhanced speech, the stable convergence behavior of $L_1\left(G_n\right)$ was shown in Figure 6. With the help of $L_1\left(G_n\right)$ at low layers ($n=1,2,4,8$), $L_1\left(G_{16k}\right)$ for the proposed model decreased faster and more stable than that of SERGAN. From the results, we can convince that the proposed model accelerates and stabilizes the GAN training.

5.5. Comparison with Conventional GAN-Based Speech Enhancement Techniques

Table 4. Comparison of results between different GAN-based speech enhancement Techniques. The best result is highlighted in bold type.

Model	PESQ	CSIG	CBAK	COVL	STOI
Noisy	1.97	3.35	2.44	2.63	0.91
SEGAN [24]	2.16	3.48	2.68	2.67	0.93
AECNN [26]	2.59	3.82	3.30	3.20	0.94
SERGAN [26]	2.59	3.82	3.28	3.20	0.94
CP-GAN [38]	2.64	3.93	3.29	3.28	0.94
The progressive generator without adversarial training	2.65	3.90	3.30	3.27	0.94
The progressive generator with the multi-scale discriminator	2.71	3.97	3.26	3.33	0.94

shows the comparison with other GAN-based speech enhancement methods that have the E2E structure. The GAN-based enhancement techniques which were evaluated in this experiment are as follows: SEGAN [22] has the U-net structure with conditional GAN. Similar to the structure of SEGAN, AECNN [26] is trained to only minimize $L_1$ loss, and SERGAN [26] is based on relativistic GAN. CP-GAN [40] has modified the generator and discriminator of SERGAN to utilize contextual information of the speech. The progressive generator without adversarial training even showed better results than CP-GAN on PESQ and CBAK. Finally, the progressive generator with the multi-scale discriminator outperformed the other GAN-based speech enhancement methods for three metrics.

6. Conclusions

In this paper, we proposed a novel GAN-based speech enhancement technique utilizing the progressive generator and multi-scale discriminator. In order to reflect the speech characteristic, we introduced a progressive generator which can progressively estimate the wide frequency range of the speech by incorporating an up-sampling layer. Furthermore, for accelerating and stabilizing the training, we proposed a multi-scale discriminator which consists of a number of sub-discriminators operating at different sampling rates.

For performance evaluation of the proposed methods, we conducted a set of speech enhancement experiments using the VoiceBank-DEMAND dataset. From the results, it was shown that the proposed technique provides a more stable GAN training while showing consistent performance improvement on objective and subjective measures for speech enhancement. We also checked the semi-real-time feasibility by observing a small increment of RTF between the baseline generator and the progressive generator.

As the proposed network mainly focused on the multi-resolution attribute of speech in the time domain, one possible future study is to expand the proposed network to utilize the multi-scale attribute of speech in the frequency domain. Since the progressive generator and multi-scale discriminator can also be applied to the GAN-based speech reconstruction models such as neural vocoder for speech synthesis and codec, we will study the effects of the proposed methods.