A Dual-Channel End-to-End Speech Enhancement Method Using Complex Operations in the Time Domain

Abstract

This study investigates the utilization of complex operations to perform multichannel speech enhancement in the time domain using a neural network. Previous studies have demonstrated the advantages of incorporating complex operations when designing neural networks; however, they have solely focused on frequency-domain enhancement techniques. In contrast, our research study presents an end-to-end approach to perform speech enhancement in the time domain. We used the Hilbert transform to intelligently generate complex time-domain waveforms as inputs to the network. This allowed us to create an end-to-end approach that explores spatial information. To handle the complexity of the inputs, we developed a complex neural adaptive beamformer (CNAB). We utilized complex shared long short-term memory (LSTM), split-LSTM, and complex convolutions to generate the beamforming output. Following this, we developed a complex full convolutional network (CFCN) to enhance the beamforming output. We leveraged complex dilated convolutions to model the long-term temporal dependencies of speech. By cascading the CNAB and CFCN, we created the final end-to-end time-domain enhancement network, named CNABCFCN. We trained and tested CNABCFCN using the deep noise suppression (DNS) challenge dataset. Our results demonstrate the advantages of using complex operations over the baseline model. Furthermore, the proposed CNABCFCN performed better in terms of both objective and subjective measures compared with other networks.

1. Introduction

Speech enhancement refers to a set of signal processing techniques aimed at improving the quality and intelligibility of speech signals by suppressing interference signals, such as background noise and reverberation. External noise and environmental reverberation can negatively impact speech signals received by microphones, resulting in a low signal-to-noise ratio (SNR) that impairs human hearing and the accuracy of systems used in human–computer interactions, such as speech recognition and speech wake-up applications [1]. Therefore, the study of speech enhancement techniques is critical to enhancing speech signals and improving overall system performance.

Traditional single-channel speech enhancement algorithms rely on signal processing techniques such as time-domain, frequency-domain, spatial-domain, and high-order statistics. However, it is challenging to adapt these methods to changing acoustic scenarios [2]. Over the past decade, deep learning-based algorithms have achieved significant success in speech enhancement. In particular, several methods that exploit time–frequency-domain feature mapping have made marked progress in noise reduction, as demonstrated in [3,4,5]. Nevertheless, these approaches typically ignore the modeling of speech signal phase information. Unlike time–frequency-domain-based methods that only focus on magnitude regression using the short-time Fourier transform (STFT), some recent methods [6,7] reconstruct the speech signal by directly processing the time-domain waveform. Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (DCCRN) [8] innovatively proposes complex number-domain features as input features to neural networks and demonstrates great potential in noise reduction. The DCCRN design simulates complex value operations using a complex number network. In DCCRN, the input of the network is both the real and imaginary (RI) information of the complex STFT. By concurrently considering the amplitude and phase of the STFT, DCCRN significantly improves the performance of speech enhancement models.

In addition, multichannel speech enhancement generally outperforms single-channel speech enhancement due to additional spatial information provided by the number of available microphones [9]. There are two main types of multichannel approaches: time-domain and frequency-domain approaches. Classic signal processing-based techniques such as beamforming have been used, including minimum variance distortionless response (MVDR) [10] and generalized sidelobe canceller (GSC) formulation [11], where the energy of the signal at the output is minimized, while the energy in the target direction is maintained. A beamformer requires a target steering vector [12], and in most cases, the method for estimating the steering vector is based on direction of arrival (DOA) estimation [13]. However, the estimation is not error-free and can significantly impact the performance of the beamforming technique.

Recently, time-domain convolutional denoising autoencoders (TCDAEs) [9] have been designed; they exploit the delay between multichannel signals to directly extract the characteristics of time-domain signals using an encoder–decoder network. Although TCDAEs exhibit better performance than single-channel DAEs [14], they use limited noise types, which results in suboptimal performance. Wang et al., proposed a complex spectral mapping approach combined with minimum variance distortionless response (MVDR) beamforming, which produces beamforming filter coefficients based on the covariance matrix of the signal and noise [15]. Alternatively, Li et al. [16] proposed neural network adaptive beamforming (NAB), which learns beamforming filters without estimating the direction of arrival (DOA) or computing the covariance matrix. NAB outperforms conventional beamforming methods but is applied to an adaptive neural network in the time–frequency domains. The filter-and-sum network (FaSNet) by Luo et al., performs time-domain filter-and-sum beamforming at the frame level and directly estimates beamforming filters with a neural network [17]. The beamforming filters are then applied to the time-domain input signal of each frame, similar to the beamforming method.

In this study, we propose a dual-channel speech enhancement technique based on complex time-domain mapping inspired by adaptive neural network beamforming and complex time-domain operations. To fully utilize the information from the two speech channels in the time domain, we developed a complex neural network adaptive beamformer (CNAB) approach to predict complex time-domain beamforming filter coefficients. During the training process, the coefficients are updated according to changes in the noisy dataset, which differs from the fixed-filter approach of previous studies [18,19]. Additionally, the real and imaginary (RI) parts of the beamforming filter coefficients estimated by the CNAB perform complex convolution and summation operations on the input to each channel. The resulting complex time-domain beamforming output is then used as input to the local complex full convolutional network (CFCN) to predict the complex time-domain information of the enhanced speech. The primary contributions of this study are as follows:

• To enable time-domain complex operations and prepare the input of the network, we utilized the Hilbert transform to construct a complex signal.
• We developed a complex neural network adaptive beamforming technique that utilizes complex inputs to perform speech enhancement.
• We developed a complex full convolutional network that takes the output of the CNAB as input to enhance speech.
• By combining all of these components, we designed and tested a dual-channel complex enhancement network that performs speech denoising in the time domain.

The rest of the paper is structured as follows: Section 2 presents the proposed algorithm, which covers the preparation of complex inputs, the loss function, the structure of the CNAB, and the structure of the CFCN. In Section 3, we describe the experimental settings, including dataset creation, different scenarios, and network parameters. Section 4 presents the experimental results and associated discussions on the performance of the proposed network compared with the baseline and different models in various scenarios. Finally, we draw conclusions and suggest topics for future research in Section 5.

2. Algorithm Description

In this section, we introduce a dual-channel, end-to-end, time-domain speech enhancement method that employs complex operations, as illustrated in Figure 1. This approach comprises two stages: complex neural network adaptive beamformer (CNAB) and complex fully convolutional network (CFCN). In the first stage, the CNAB performs preliminary speech enhancement on the dual-channel noisy speech. In the second stage, the CFCN produces the final high-quality speech.

2.1. Analysis of the Signal Characteristics at the Input

In this section, we provide the details to show how to generate the complex inputs in time domain.

The received signals by each microphone in time domain are

$$x_0\left(k\right)\left[t\right]=s_0\left(k\right)\left[t\right]+n_0\left(k\right)\left[t\right],$$

(1)

$$x_1\left(k\right)\left[t\right]=s_1\left(k\right)\left[t\right]+n_1\left(k\right)\left[t\right],$$

(2)

where $t\in \left\{0,1,\dots ,N\right\}$ denotes the sample index in each frame and $k\in \left\{0,1,\dots ,M\right\}$ denotes the frame index. The clean speech captured by the two microphones, denoted by $s_0\left(k\right)\left[t\right]$ and $s_1\left(k\right)\left[t\right]$, respectively, is given by

$$s_0\left(k\right)\left[t\right]=s\left(k\right)\left[t\right]\ast h_0\left(k\right)\left[n\right],$$

(3)

$$s_1\left(k\right)\left[t\right]=s\left(k\right)\left[t\right]\ast h_1\left(k\right)\left[n\right],$$

(4)

where $h_0\left(k\right)\left[n\right]\in {\mathbb{R}}^{K\times 1}$ and $h_1\left(k\right)\left[n\right]\in {\mathbb{R}}^{K\times 1}$ indicate the room impulse responses (RIRs) corresponding to the two microphones; $n\in \left\{0,1,\dots ,K-1\right\}$ denotes the sampling point of the filters; and $s\left(k\right)\left[t\right]\in {\mathbb{R}}^{N\times 1}$ is the sound source. In this study, $s_0\left(k\right)\left[t\right]$ corresponds to the direct path signal in the reference microphone and is utilized as the target [15]. Since the distance between the two microphones is small and they are positioned on the same horizontal line, we select $c=0$ as the reference microphone. While noise is also affected by the RIR, it must be eliminated in speech noise reduction tasks, regardless of its form. Therefore, the effect of RIR on noise is generally not explicitly expressed.

In frequency-domain processing, the RI parts are naturally present in spectrograms. However, in the time domain, only real-time speech is available, which poses the first challenge of generating its corresponding imaginary part. In this study, we used the Hilbert transform to construct the analytic signal. The Hilbert transform constructs the analytic function $x_c^a\left(k\right)\left[t\right]\in {\mathbb{C}}^{N\times 1}$ of a discrete signal, where $\mathbb{C}$ is the complex number field. For convenience and because the network operates in a frame-by-frame manner, we omit the frame number (k) and denote $x_c^a\left(k\right)\left[t\right]$ as $x_c^a\left[t\right]$ hereafter. For the negative frequency part of the signal spectrum, we apply the following correction:

$$\begin{array}{cc}\hfill X_c^a\left(f\right)& =\left\{\begin{array}{cc}2X_c\left(f\right),\hfill & f>0\hfill \\ X_c\left(f\right),\hfill & f=0\hfill \\ 0,\hfill & f<0\hfill \end{array}\right.\hfill \\ \hfill & =X_c\left(f\right)+sgn\left(f\right)X_c\left(f\right),\hfill \end{array}$$

(5)

where $X_c^a\left(f\right)\in {\mathbb{R}}^{L\times 1}$ and $X_c\left(f\right)\in {\mathbb{R}}^{L\times 1}$ are the spectra of $x_c^a\left[t\right]$ and $x_c\left[t\right]$, respectively, and subscript c represents the microphone index. The function $sgn(·)$ represents the sign function. Thus, the definition of the analytic function $x_c^a\left[t\right]$ is

$$\begin{array}{cc}\hfill x_c^a\left[t\right]& ={\mathcal{F}}^{-1}[X_c\left(f\right)+sgn\left(f\right)X_c\left(f\right)]\hfill \\ \hfill & ={\mathcal{F}}^{-1}\left[X_c\left(f\right)\right)+{\mathcal{F}}^{-1}\left(sgn\left(f\right)\right)\ast {\mathcal{F}}^{-1}\left(X_c\left(f\right)\right)]\hfill \\ \hfill & =x_c\left[t\right]+j(\frac{1}{\pi t}\ast x_c\left[t\right])\hfill \\ \hfill & =x_c\left[t\right]+j{\widehat{x}}_c\left[t\right],\hfill \end{array}$$

(6)

where ${\mathcal{F}}^{-1}$ is the inverse Fourier transform. Equation (6) shows that the imaginary part ${\widehat{x}}_c\left[t\right]\in {\mathbb{R}}^{N\times 1}$, which can be obtained with the inverse Fourier transform of the positive half of the frequency spectrum, $X_c\left(f\right)$. The one-dimensional real signal $x_c\left[t\right]\in {\mathbb{R}}^{N\times 1}$ is convolved with the impulse response of $\frac{1}{\pi t}$ to produce its corresponding imaginary part, and a two-dimensional complex signal $x_c^a\left[t\right]\in {\mathbb{C}}^{N\times 1}$ finally serves as input to the proposed network.

2.2. Loss Function

Unlike traditional end-to-end speech enhancement models that only estimate the real part information in the time domain, our proposed method trains CNABCFCN to recover both the RI parts of clean speech. The loss function used to train the proposed model is

$$\begin{array}{cc}\hfill \mathrm{SI-SDR}& =(1-\lambda )\times 10{log}_{10}\frac{\parallel {\beta }_r\times Re\left(s^a\left[t\right]\right){\parallel }^2}{\parallel {\beta }_r\times Re\left(s^a\left[t\right]\right)-\widehat{R}{\left[t\right]\parallel }^2}\hfill \\ \hfill & +\lambda \times 10{log}_{10}\frac{\parallel {\beta }_i\times Im\left(s^a\left[t\right]\right){\parallel }^2}{\parallel {\beta }_i\times Im\left(s^a\left[t\right]\right)-\widehat{I}{\left[t\right]\parallel }^2},\hfill \end{array}$$

(7)

$${\beta }_r=\frac{{\widehat{R}}_t^{T}Re\left(s^a\left(t\right)\right)}{\parallel s^a{\left[t\right]\parallel }^2}=\underset{{\beta }_r}{\arg \min }{\parallel {\beta }_r\times Re\left(s^a\left(t\right)\right)-\widehat{R}\left[t\right]\parallel }^2,$$

(8)

$${\beta }_i=\frac{\widehat{I}{\left[t\right]}^{T}Im\left(s^a\left(t\right)\right)}{\parallel s^a{\left[t\right]\parallel }^2}=\underset{{\beta }_i}{\arg \min }{\parallel {\beta }_i\times Im\left(s^a\left(t\right)\right)-\widehat{I}\left[t\right]\parallel }^2,$$

(9)

where $Re(·)$ and $Im(·)$ denote the RI part operations, respectively. Because the features we consider are essentially time-domain information, we adopt the scale-invariant signal-to-distortion ratio (SI-SDR) loss function to train the proposed model, which results in better speech enhancement performance [20]. The RI information of each frame estimated by the CNABCFCN model is denoted by $\widehat{R}\left[t\right]\in {\mathbb{R}}^{N\times 1}$ and $\widehat{I}\left[t\right]\in {\mathbb{R}}^{N\times 1}$, respectively. And $s^a\left[t\right]$ is the analytic function of the reference channel that received clean speech $s_0\left[t\right]$. In (7), $\lambda$ is a constant within $[0,1]$ that weighs RI information. When $\lambda =0$, the network prefers to use real information to train the network parameters, whereas $\lambda =1$ indicates a preference for using imaginary information to update the network. From the experiments, a good empirical hyperparameter $\lambda =0.5$ was chosen.

2.3. CNAB Architecture

The proposed CNABCFCN framework utilizes complex operations in all CNAB structures. As shown in the first part of Figure 1, the first layer in the CNAB structure consists of a complex LSTM layer, which we call complex shared-LSTM. The Hilbert transform ($x_c^a\left[t\right]$) of speech in each channel passes through this layer. Figure 2 demonstrates the rules of complex LSTM used in complex shared-LSTM. The complex operations inside complex LSTM are, explicitly,

$$L_r=LST{M}_r\left(Re\left(x_c^a\left[t\right]\right)\right)-LST{M}_r\left(Im\left(x_c^a\left[t\right]\right)\right),$$

$$L_i=LST{M}_i\left(Re\left(x_c^a\left[t\right]\right)\right)+LST{M}_i\left(Re\left(x_c^a\left[t\right]\right)\right),$$

$$L_{out}=L_r+j{L}_i.$$

(10)

The complex shared-LSTM in the CNAB includes two ordinary LSTM networks, $LST{M}_r$ and $LST{M}_i$, which process the real and imaginary parts of speech, respectively. The feature mappings of the real and imaginary parts are denoted by $L_r\in {\mathbb{R}}^{N\times 1}$ and $L_i\in {\mathbb{R}}^{N\times 1}$, respectively.

The output of complex LSTM is a complex feature mapping, $L_{out}\in {\mathbb{C}}^{N\times 1}$. After passing through the shared-LSTM layer, the output is fed into two complex LSTM layers, which correspond to the features of each channel. This process is called complex split-LSTM, and the complex operations are the same as in (10). The outputs of the split-LSTM layers are activated by complex linear functions and serve as the estimated beamforming filters, denoted by $h_c^a\left[n\right]$, as shown in Figure 1. Finally, the original inputs are convolved with the estimated coefficients $h_c^a\left[n\right]$ and summed together to produce the output of the CNAB. A detailed explanation of the CNAB will follow.

2.4. Complex Adaptive Spatial Filtering

Traditional beamforming requires the alignment of the received speech signal with the reference microphone due to differences in the time it takes for sound to reach different microphones. The accurate estimation of steering delay ${\tau }_c$ plays a crucial role in speech enhancement performance. The proposed CNAB model represents a novel approach to estimating filter coefficients by minimizing a loss function that measures the difference between clean speech and the enhanced one in the time–frequency domains. As a result, the estimation of steering delay ${\tau }_c$ is implicit in the estimated filter coefficients. For the k-th frame, the CNAB model’s output is denoted by $y^a\left(k\right)\left[t\right]$, expressed in a frame-wise manner as

$$\begin{array}{cc}\hfill y^a\left(k\right)\left[t\right]=& (y_{rr}\left(k\right)\left[t\right]-y_{ii}\left(k\right)\left[t\right])\hfill \\ \hfill & +j(y_{ri}\left(k\right)\left[t\right]+y_{ir}\left(k\right)\left[t\right]),\hfill \end{array}$$

(11)

where $y_{rr}\left(k\right)\left[t\right]\in {\mathbb{R}}^{N\times 1}$ is the output result of the real feature, expressed as

$$y_{rr}\left(k\right)\left[t\right]=\sum _{c=0}^{C-1}\sum _{n=0}^{N-1}Re\left(h_c^a\left(k\right)\left[n\right]\right)Re\left(x_c^a\left(k\right)[t-n]\right),$$

(12)

where $h_c^a\left(k\right)\left[n\right]\in {\mathbb{C}}^{K\times 1}$ is the estimated beamforming coefficient for channel c. To estimate $h_c^a\left[n\right]$, we train the entire CNABCFCN network in a joint manner to produce the filter coefficients of each channel. In Equation (11), the features $y_{ii}\left(k\right)\left(t\right)$, $y_{ri}\left(k\right)\left(t\right)$, and $y_{ir}\left(k\right)\left(t\right)$ are defined as in Equation (12), and their specific representations are omitted for simplicity.

2.5. CFCN Architecture

The second part of CNABCFCN consists of a complex fully convolutional neural network, which is illustrated in Figure 3. As defined above, the output of the CNAB is denoted by $y^a\left(k\right)\left[t\right]\in {\mathbb{C}}^{N\times 1}$, where $\mathbb{C}$ is the set of complex numbers, $k\in \left\{0,1,\dots ,M-1\right\}$ is the index of each frame, and M is the total number of indices. At this point, $y^a\left(k\right)\left[t\right]$ is no longer a dual-channel feature but a single-channel complex feature, and it is passed through a 1D convolutional layer. The formulation of the 1D convolution is given by

$$D_r\left(k\right)=f_{dconv}\left(Re\left(y^a\left(k\right)\right)\right),$$

$$D_i\left(k\right)=f_{dconv}\left(Im\left(y^a\left(k\right)\right)\right),$$

(13)

where $f_{dconv}$ denotes the feature mapping function of 1D Conv, and $D_r\left(k\right)\in {\mathbb{R}}^{P\times 1}$ and $D_i\left(k\right)\in {\mathbb{R}}^{P\times 1}$ denote the real and imaginary outputs of 1D Conv, respectively.

To facilitate the CFCN network in learning the characteristics of time-domain data, the real and imaginary outputs $(D_r,D_i)\in {\mathbb{R}}^{S\times 2}$ must be normalized as follows:

$$(N_r,N_i)=f_{norm}\left((D_r,D_i)\right),$$

(14)

where $f_{norm}$ is the feature mapping function of the layernorm, BatchNorm [21] is used as the layernorm, and $(N_r,N_i)\in {\mathbb{R}}^{S\times 2}$ are the normalized feature mappings.

The CFCN network uses a complex $1\times 1$ convolutional layer to perform complex convolutions on $N_r$ and $N_i$, as follows:

$$\begin{array}{cc}\hfill C_{out}=& (con{v}_r\left(N_r\right)-con{v}_i\left(N_i\right))\hfill \\ \hfill & +j(con{v}_r\left(N_i\right)+con{v}_i\left(N_r\right)).\hfill \end{array}$$

(15)

Equation (15) shows that the complex $1\times 1$ convolutional layer is constructed from two one-dimensional convolutional layers, $con{v}_r$ and $con{v}_i$, representing the feature mapping functions for the RI parts, respectively. The final output of the complex $1\times 1$ convolutional layer is $C_{out}\in {\mathbb{C}}^{T\times 1}$.

Following the complex convolutional layer, the CNABCFCN network uses 1D dilated convolutional blocks [22]. To maximize the utilization of the temporal context window of the speech signal, the dilation factors of the 1D convolutional blocks in each layer are exponentially increased. The proposed CNABCFCN system repeats X convolutional blocks with dilation factors $1,2,4,\dots ,2^{X-1}$R times. Although we attempted to replace all 1D dilated convolutional blocks with complex convolutional blocks, this approach did not improve the performance of the CNABCFCN model and instead consumed significant computational memory. The experimental section of the paper describes the effect of the number of complex convolutional layers on the system’s performance. In the final designed CNABCFCN network, the final 1D convolutional blocks are replaced with a complex convolutional block, which is highlighted in different colors in Figure 3. The network employs the parametric rectified linear unit (PReLU) as its nonlinear activation function, defined by

$$PReLU\left(x\right)=\left\{\begin{array}{c}x,x>=0\hfill \\ ax,x<0,\hfill \end{array}\right.$$

(16)

where $a\in \mathbb{R}$ is a trainable scalar used to control the negative slope of the rectifier. After the complex $1\times 1$ convolutional layer, the network adds a sigmoid activation function. The output of the sigmoid function is then multiplied by beamforming feature mapping, resulting in the final enhanced speech signal.

3. Experimental Settings

This section presents the dataset used in the experiments and outlines the procedure used to create the two-channel dataset. We also describe the hyperparameter settings of each neural network layer in the CNABFCN model and select the appropriate network parameters using experimental data. In the experiments, we compared the SI-SDR performance of the model using different hyperparameters to determine the optimal model parameters.

3.1. Generation of the Two-Channel Dataset

In the experiment, we trained and evaluated the proposed speech enhancement model using the deep noise suppression (DNS) challenge dataset [23]. The clean audio signals were derived from LibriVox, a public English audiobook dataset, and the noises were selected from Audioset and Freesound. All audio signals were sampled at 16 kHz, with clean clips and noise clips having lengths of 31 s and 10 s, respectively. We used the script provided in the DNS challenge to generate 75 h of audio clips, with each clip having a size of 6 s. In total, we generated 36,400 clips (approximately 60 h) for the training set and 4550 clips (approximately 8 h) for the validation set, and the remaining 4050 clips (approximately 7 h) were used as the test set.

To produce dual-channel datasets, we used the image method [24] and studied the effects of single-direction and multiple-direction target sources on the performance of speech enhancement. We tested two different scenarios.

Scenario I: In the first scenario, the target source was located in a single direction. As shown in Figure 4, the target source was close to the reference microphone, and it was on the same horizontal line as the microphones. The target position was marked with the red box, while the incoming noise was uniformly sampled from 0 and 90 degrees at 15-degree intervals.

Scenario II: In the second scenario, shown in Figure 5, the azimuth angle of the target source was indicated by the red dotted lines and ranged between −45 and 45${}^{\circ }$. The azimuth angle of the noise was between −90 and 90${}^{\circ }$ and evenly distributed at 15-degree intervals. This scenario created a so-called acoustic fence, where the speech inside the region of interest (ROI) could be passed with minimal distortion.

The clean and noisy signals were located 1 m away from the microphones, while the distance between the two microphones was 3 cm. In the simulated reverberation room, which had a size of 10 m × 7 m × 3 m (width × depth × height), the sound source signal was convolved with two different room impulse responses (RIRs) before being received by the two microphones. The signal-to-noise ratios (SNRs) were randomly mixed from −5 to 10 dB at 1 dB intervals. In the test set, the SNRs were set to −5, 0, 5, 10, and 20 dB, and the clean speech signal captured by the reference microphone was used as the label for the CNABCFCN model.

3.2. Configurations of CNAB

In this study, the 6 s simulated speech signal with a sampling frequency of 16 kHz was divided into 1 s segments, each consisting of 16,000 samples. The Hilbert transform was applied to each input speech segment to extract the real and imaginary parts, which were used as inputs to the complex shared-LSTM. Since both real information and imaginary information were in the time domain, LSTM was used to extract features directly, as it is better suited for modeling temporal information.

Table 1. CNAB model, where B denotes the batch size and the format of hyperparameters is the size of the input and output channels of the LSTM.

Layer Name	Input Size	Hyperparameters
complex input	(B, 2, 16,000), (B, 2, 16,000)	-
complex shared-LSTM	(B, 2, 100, 160)	(160, 512)
complex split-LSTM	(B, 2, 512)	(512, 256)
complex split-LSTM	(B, 2, 512)	(512, 256)
complex linear	(B, 2, 256)	(256, 25)

summarizes the parameters of the CNAB model.

According to Table 1, the input segment dimension was (B, 2, 16,000), with B representing the batch size, and the input had real and imaginary information, with the middle dimension being set to 2. Each segment consisted of 16,000 sampling points, which were divided into 100 temporal sequences, with each sequence having a duration of 10 ms, creating 160 sampling points. The input format of the complex shared-LSTM was $batch\times 2\times timesteps\times in putsize$. A complex shared-LSTM layer was composed of two 512-cell LSTM layers, denoted by $LST{M}_r$ and $LST{M}_i$. The last step of the timesteps was used as input to the two complex split-LSTM layers. Finally, the complex linear activation layer was employed to estimate the beamforming filter coefficients of 25 sampling points.

3.3. CFCN Configurations

The CFCN model is responsible for processing single-channel features after beamforming and estimating the final clean speech. The CFCN configuration is presented in

Table 2. CFCN configuration, where B is the batch size and L is the length of the feature map. The format of hyperparameters for the convolutional network is the dimensions of the input and output, the size of the convolutional kernel, and the step size of stride.

Layer Name	Input Size	Hyperparameters
complex input	(B, 2, 16,000)	---
1D Conv	(B, 16,000) × 2	(1, 256), 40, 20
LayerNorm	(B, 2, 256, L)	BatchNorm2d
complex $1\times 1$ Conv	(B, 2, 256, L)	(1, 1), (5, 2), (2, 1)
partial complex 1D Conv	…	…
$1\times 1$ Conv	(B, 256, L)	(256, 512), 1, 1

. The model consists of eight 1D dilated Convs forming a block, and this process is repeated three times; the detailed structure of 1D Conv is provided in Figure 6. Dilated convolution is known for modeling long-term temporal dependencies through large receptive fields. The use of complex operations can convert all Convs into complex ones, but this process does not substantially improve noise reduction performance and may increase computational complexity. A suitable number of complex Convs for balancing performance and complexity is, therefore, crucial.

In

Table 3. Number of complex 1D Conv in CFCN and its corresponding SI-SDRs.

Numbers	0	3	6	9	12
SI-SDR	−17.823	−18.296	−8.825	−8.724	−8.834

, we provide SI-SDR results for different numbers of complex Convs in the CFCN, demonstrating that increasing the use of complex Convs cannot always enhance performance. We modified the last 1D dilated Conv of each repetition into a complex one, marked by the gray rectangle in Figure 3, and this configuration was adopted for the CFCN.

3.4. Baseline NABFCN System

We compared our network with a baseline system, named NABFCN, to demonstrate the potential of the complex network. NABFCN is an end-to-end system, whose structure is similar to that of CNABCFCN, except that it accepts real inputs. All network structures of NABFCN consist of common LSTM and convolutions, and the frame structure of NABFCN is illustrated in Figure 7. The input noisy speech is still L = 16,000 long, like the CNAB’s. The NAB model is also composed of three LSTM layers and two linear layers, whose input and output parameters are consistent with the CNAB and which are summarized in

Table 4. NAB model, where B represents the batch size and the format of hyperparameters is the size of the input and output channels of LSTM.

Layer Name	Input Size	Hyperparameters
input	(B, 16,000), (B, 16,000)
shared-LSTM	(B, 100, 160)	(160, 512)
split-LSTM	(B, 512)	(512, 256)
split-LSTM	(B, 512)	(512, 256)
linear	(B, 256)	(256, 26)

. To be consistent with the length of the filter, the size of the linear layer was chosen as $N=26$.

The linear layer of NAB produces beamforming filter coefficients that are convolved with the 160 time-domain sampling points to generate the NAB output. The FCN model used in NABFCN has a structure similar to that of the CFCN, but it lacks complex 1D Convs. As the output of NABFCN is a real signal, its loss function only utilizes real information to calculate the loss value as follows:

$$\mathrm{SI-SDR}=10{log}_{10}\frac{{\parallel \beta s\parallel }^2}{\parallel \beta s-\widehat{s}{\parallel }^2},$$

$$\beta =\frac{{\widehat{s}}^{T}s}{{\parallel s\parallel }^2}=\underset{\beta }{argmin}{\parallel \beta s-\widehat{s}\parallel }^2,$$

(17)

where $s\in {\mathbb{R}}^{1\times T}$ and $\widehat{s}\in {\mathbb{R}}^{1\times T}$ denote clean speech and enhanced speech, respectively.

To evaluate the benefits of the complex time-domain features and complex neural networks developed in this work, we kept the network parameters of NABFCN and CNABCFCN consistent.

4. Evaluation Results

4.1. Comparisons with the NABFCN Model

In the experiments, the proposed CNABCFCN model and the baseline NABFCN model were objectively evaluated using short-time objective intelligibility (STOI) [25] and perceptual evaluation of speech quality (PESQ) [26]. The evaluation was conducted on the dataset produced using the configurations in Figure 4 and Figure 5. The results of the sound source at 0 degrees are shown in

Table 5. PESQ and STOI of CNABCFCN and NABFCN (Scenario I).

Model	Metrics	SNR
		−5 dB	0 dB	5 dB	10 dB	20 dB
NABFCN	PESQ	2.65	2.97	3.21	3.42	3.74
	STOI	87.44	92.14	94.98	96.77	98.62
CNABCFCN	PESQ	2.75	3.07	3.32	3.52	3.81
	STOI	89.22	93.29	95.74	97.27	98.81

, while the results of the sound source in the range of −45 to 45${}^{\circ }$ are shown in

Table 6. PESQ and STOI of CNABCFCN and NABFCN (Scenario II).

Model	Metrics	SNR
		−5 dB	0 dB	5 dB	10 dB	20 dB
NABFCN	PESQ	2.52	2.85	3.16	3.34	3.64
	STOI	85.94	89.95	93.27	95.68	97.52
CNABCFCN	PESQ	2.66	2.97	3.24	3.45	3.74
	STOI	87.54	92.40	95.24	96.99	98.70

.

Table 5 reveals that, with a stationary sound source at 0${}^{\circ }$, the CANBCFCN network exhibited significant enhancement over NABFCN at all SNRs. This finding illustrates the potential benefits of complex operations. Table 6 shows a similar conclusion, that is, CNABCFCN outperformed NABFCN. Nonetheless, the ability of the models to suppress noise was somewhat reduced in complex acoustic scenes, where there were multiple target directions to consider. This result was expected given the complex task.

Figure 8 presents enhanced spectrograms to highlight the differences between the NABFCN and CNABCFCN models. A visual comparison reveals that the output of NABFCN retained residual noise at middle and high frequencies, whereas CNABCFCN significantly suppressed noise, resulting in minimal speech distortion.

4.2. Comparisons with Other Models

To enable a comprehensive comparison with other networks, we applied Conv-TasNet (C-TasNet) [22], DCCRN [8], DPCRN [27], MNTFA [28], and CRN [29] on the proposed dataset:

• Conv-TasNet shares structural similarities with the second part of the NABFCN network. In this experiment, we employed one channel of the proposed noisy dataset for its training and testing.
• Similarly to C-TasNet, DCCRN is a single-channel model [8] and was the winner of the Interspeech2020 DNS challenge, where complex rules were implemented based on CNNs and RNNs. It was also trained and tested on one channel of the proposed noisy datasets.
• Unlike C-TasNet and DCCRN, CRN is a dual-channel network [29]. For CRN-i, the parameters were selected based on [29], while for CRN-ii, we fine-tuned the parameters based on the dataset, with reference to [4].
• The dual-path convolutional recurrent network (DPCRN) [27] combines DPRNN and CRN, making it possible to obtain a well-behaved model.
• The multi-loss convolutional network with time-frequency attention (MNTFA) [28] for speech enhancement is an attention-based speech enhancement model where axial self-attention (ASA) is developed to model long-time dependency.

Table 7. PESQ and STOI of different networks on the DNS dataset (Scenario I).

Model	Mics	Para (M)	PESQ			STOI
			−5 dB	0 dB	5 dB	−5 dB	0 dB	5 dB
noisy	-	-	1.37	1.67	1.98	66.91	75.57	83.26
C-TasNet	1	5.1	2.39	2.75	3.02	81.38	88.95	93.17
DCCRN	1	5.3	2.23	2.63	2.95	80.79	89.02	92.58
DPCRN	1	0.8	2.36	2.67	2.91	75.08	81.60	86.28
MNTFA	1	0.23	2.42	2.72	3.02	75.64	81.97	86.78
CRN-i	2	0.08	1.56	1.89	2.28	69.51	78.68	86.27
CRN-ii	2	17.6	1.61	1.96	2.33	71.06	79.85	87.15
CNABCFCN	2	9.2	2.75	3.07	3.32	89.22	93.29	95.74

presents the experimental results, where the best outcomes are highlighted in bold. Notably, the proposed method achieved a significant improvement over the single-channel models of C-TasNet, DCCRN, DPCRN, and MNTFA, showing the benefits over the usage of a single channel. Furthermore, compared with the dual-channel CRN-ii model, the proposed method achieved superior performance, regardless of the size of the CRN–ii model.

4.3. Comparisons in Complex Acoustic Scenes

Although Scenarios I and II provide a degree of realism, they may not fully reflect real-world situations. To create a more challenging dataset that aligns with real-world conditions, we developed Scenario III, which is based on the U-Net-GCN model in [30]. The dataset comprised 65 h of audio with a sampling rate of 16 kHz. The training set included three rooms with dimensions of 3 × 3 × 2, 5 × 4 × 6, and 8 × 9 × 10 m, respectively; the development set comprised two rooms with dimensions of 5 × 8 × 3 and 4 × 7 × 8 m, respectively; and the test set included two rooms with dimensions of 4 × 5 × 3 and 6 × 8 × 5 m, respectively. All rooms had a reverberation time of 0.5 s (RT60 = 0.5 s). The sound source was located anywhere in the room, and RIRs were generated using the image method. Reverberant speech and noise were mixed at SNRs of −7.5, −5, 0, 5, and 7.5 dB. The training, validation, and test sets comprised 60 h, 4 h, and 1 h of audio, respectively.

We report the in STOI, PESQ, and SDR results of both U-Net-GCN and CNABCFCN at different SNRs in Figure 9. The metrics for both networks improved as the SNR increased. However, the proposed model outperformed U-Net-GCN, as demonstrated by, for example, the gains of 0.12, 0.56, and 3.74 in STOI, PESQ, and SDR, respectively, achieved by CNABCFCN over U-Net-GCN when SNR = 0 dB (as seen in

Table 8. STOI, PESQ, and SDR on the DNS dataset (Scenario III).

Model	−7.5 dB			−5 dB			0 dB			5 dB			7.5 dB
	STOI	PESQ	SDR	STOI	PESQ	SDR	STOI	PESQ	SDR	STOI	PESQ	SDR	STOI	PESQ	SDR
noisy	0.55	1.46	−7.51	0.60	1.54	−5.06	0.71	1.87	1.57	0.80	2.23	4.81	0.84	2.41	7.22
U-Net-GCN	0.60	1.67	3.84	0.66	1.98	5.74	0.72	2.11	6.84	0.77	2.24	8.86	0.81	2.36	9.94
CNABCFCN	0.71	2.14	3.70	0.76	2.33	5.58	0.84	2.67	10.58	0.89	2.99	12.29	0.92	3.13	13.88

).

We compared the proposed CNABFCN model with the following dual-channel models in Scenario III to demonstrate its superior performance:

• DNN [31,32], a vector-to-vector regression method that employs fully connected layers to perform speech enhancement.
• TNN [33], a tensor-to-vector regression method used for multichannel speech enhancement.
• BC-SE [34], a dual-microphone speech enhancement algorithm that employs a bone conduction sensor to map noisy speech at low frequencies.

Table 9. PESQ of different dual-channel networks (Scenario III).

Model	Mics	Para (M)	PESQ
DNN	2	33	3.00
DNN-SVD	2	5	2.93
TNN	2	0.6	2.75
TNN	2	5	2.96
BC-SE	2	--	2.24
CNABCFCN	2	9.2	3.20

presents the PESQ results of different networks. The PESQ results of DNN and TNN were averaged at input SNRs of 10 dB to 20 dB, while the results of BC-SE and CNABCFCN were computed at input SNRs of 5 dB to 15 dB. Despite the lower SNRs, the proposed network outperformed both DNN and TNN. Additionally, the proposed model outperformed BC-SE by 0.96, as shown in the table.

5. Conclusions

In this study, we propose a novel complex time-domain process and design a complex neural network structure called CNABCFCN for dual-channel end-to-end speech enhancement. We leveraged the Hilbert transform to generate complex time-domain waveforms and incorporated complex operation rules to develop a network consisting of a CNAB and a CFCN. The CNAB takes dual-channel complex inputs to produce a complex single-channel output, while the CFCN is cascaded to obtain the final enhancement. Furthermore, we used a new loss function called SI-SDR, which considers both the real and imaginary parts of the speech waveform to balance speech quality. We evaluated the proposed network under different scenarios and demonstrated its versatility and effectiveness. Our results show that adding imaginary information improves denoising performance, and the proposed network outperformed existing methods in all scenarios based on both objective and subjective evaluations. Future research should explore extensions to more than two channels.