Self-Supervised Voice Denoising Network for Multi-Scenario Human–Robot Interaction

Abstract

Human–robot interaction (HRI) via voice command has significantly advanced in recent years, with large Vision–Language–Action (VLA) models demonstrating particular promise in human–robot voice interaction. However, these systems still struggle with environmental noise contamination during voice interaction and lack a specialized denoising network for multi-speaker command isolation in an overlapping speech scenario. To overcome these challenges, we introduce a method to enhance voice command-based HRI in noisy environments, leveraging synthetic data and a self-supervised denoising network to enhance its real-world applicability. Our approach focuses on improving self-supervised network performance in denoising mixed-noise audio through training data scaling. Extensive experiments show our method outperforms existing approaches in simulation and achieves 7.5% higher accuracy than the state-of-the-art method in noisy real-world environments, enhancing voice-guided robot control.

1. Introduction

In recent years, with the development of Vision–Language–Action (VLA) models [1,2,3,4] facilitating significant advancements in robotic manipulation, there has been growing interest in integrating action prompts and voice interaction. However, one of the most critical and unresolved issues is ensuring effective command recognition in human–robot interaction (HRI). Current VLA models often rely on voice encoders (e.g., LLama [5], T5 [6]) to understand task prompts during HRI in real-world applications, but may miss critical instructions due to environmental noise [7]. This leads to the first research question: (1) How can audio commands be stably transmitted in real-world interactive environments?

Regarding the first question, supervised training strategies [8,9] have emerged as reliable tools for voice denoising during HRI. Nevertheless, this method requires collecting multiple high-quality audio samples for training, a process that demands a carefully controlled recording environment, resulting in both increased complexity and higher cost. Thanks to previous works [10,11,12], researchers have employed self-supervised learning to train models for generating voice sequences. Noise variation can also be estimated from noisy audio datasets [13] through latent audio features [14], while Large-Language Model (LLM) [15] enables audio-to-text conversion followed by semantic reasoning to interpret distorted speech. However, many of these systems still struggle in real-world robotic applications, particularly in multi-scenario or multi-speaker environments. First, existing voice datasets fail to adequately simulate ambient sound, particularly in cases of overlapping human voice, limiting practical performance. Second, interactive audio exhibits discrete properties, limiting the effectiveness of time-domain denoising for frequency feature reconstruction. Finally, LLMs are prone to hallucinations, generating inaccurate or unsafe responses that may produce erroneous robotic commands with potentially harmful consequences. This leads to two further research questions: (2) How to scale training data for enhanced real-world denoising capability? (3) How to effectively denoise audio for real-world HRI applications?

Through answering questions (1) to (3), a self-supervised voice denoising approach is proposed to enhance robotic audio command transmission quality in real-world applications, as illustrated in Figure 1. In contrast to previous work, our model employs realistic synthetic data (i.e., multi-environment noise or overlapping voices) in self-supervised training and integrates the convolution and attention network to extract key features from the audio magnitude and phase information. The intuition behind our design is the integration of scaling data and self-supervised denoising capabilities, where both training inputs and targets are directly generated from noisy signals. The specialized denoising architecture with regularized loss functions delivers results matching supervised training, providing reliable audio for VLAs. A comprehensive comparison of the evaluation results shows that our model outperforms state-of-the-art methods in simulation benchmarks and achieves a 7.5% improvement in real-world manipulation tasks.

The contributions of this work are summarized below:

• We propose training the denoising model with synthetic data, using data scaling to enhance robustness in model application.
• We present a self-supervised method using a convolutional network to denoise both magnitude and phase components in the frequency domain.
• We demonstrate proof-of-concept of the proposed approach in both simulation and hardware. For the benefit of the community, we will open-source part of the scaled dataset at https://tinyurl.com/2ybfz3jt (accessed on 25 August 2025).

2. Related Work

Synthetic Data Application. Synthetic data can rapidly generate large-scale training sets, enhancing the practicality of datasets [16,17,18]. Recent advances in voice synthesis include generating various pulses, tones, and altered timbres [19,20,21]. In contrast, HRI applications require interactive voice in diverse environments. Prior work using public audio datasets for denoising model training often shows poor generalization in complex real-world scenarios. In addition, collecting pure datasets requires significant expense. Therefore, we aim to use audio synthesis to generate diverse datasets for model training, covering voices in multiple scenarios.

Voice Denoising Network. Noise–Clean Training (NCT) [11,22,23,24], a common audio denoising method, uses clean audio signals as training targets. However, it struggles to handle variable environmental noise in real-world applications. Thus, noise-only training has recently gained wide attention [25,26]; the network can be trained with multiple independent noisy voice signals using self-supervised methods. Xu [14] leverages silent intervals in speech signal to isolate noise-only periods for time-varying noise estimation, and studies like [10,11] demonstrate that voice denoising models can be built from datasets containing diverse noise. Inspired by Noisy-to-Noisy, Wu [12] employs a complex-valued network architecture for multi-dimensional audio denoising and [13] constructs mixed-audio datasets combining robot ego-voice, fan noise, and concurrent human speech to improve low-frequency noise suppression. However, these methods fail to process overlapping audio in multi-speaker environments, making it difficult to extract target signals in interaction dialogues. In our work, we improve the model’s denoising capability for unseen environments by synthesizing audio from diverse scenarios and expanding the training dataset.

Voice Control in HRI. Voice can be used in HRI to deliver action instructions and guide robot manipulation [27,28,29,30,31]. The development of LLMs [5,6,15] has led to significant advancements in voice reasoning, particularly in environments requiring high adaptability and generation. Lai [28] proposed a natural multimodal fusion-based HRI framework (NMM-HRI) that combines voice commands and deictic postures, leveraging LLMs to generate robot action sequences. In [32], the PlanCollabNL framework combines LLMs with task planning to generate robot action subgoals from abstract natural language goals. Building on the success of VLA models [2,3,4], voice interaction is gaining significant attention for its ability to provide operational conditions and task requirements for robotic actions through voice prompts. Most VLAs [2,3,4,33,34], treat the frozen-weight LLM as a policy to generate action commands for robot operation. However, none of the works address that audio distortion in noisy environments further reduces the LLM model’s instruction recognition accuracy. In this work, we aim to use a self-supervised denoising algorithm to optimize audio quality, enabling stable transmission of audio prompts to VLAs.

3. Preliminaries

3.1. Human–Robot Voice Interaction System

In the human–robot voice interaction scenario, we consider there are two equidistant microphone arrays (i.e., simulating human binaural distribution) to capture the interactive sound in the environment, as shown in Figure 2. This ensures recorded voice signals vary randomly and nonlinearly over time. The interaction architecture uses the Z-transform to convert signals to the time–frequency domain, with the collected signal $M\left(z\right)$ composed of the source sound $T\left(z\right)$ and the mixing matrix $A\left(z\right)$. The following equation holds:

$$M\left(z\right)=T\left(z\right)·A\left(z\right),$$

(1)

$$A\left(z\right)=\left[\begin{array}{cc}1& a_0{z}^{-d_0}\\ a_1{z}^{-d_1}& 1\end{array}\right],$$

(2)

where $d_i$ and $a_i$ denote the fractional delay in samples and the attenuation factor along the indirect signal path, respectively, and $i=0,1$ represents the two recording microphones. However, one challenge in voice denoising is that $M\left(z\right)$ contains complex environmental interference, and direct application of audio enhancement can easily distort the source sound. It is valuable to separately denoise frequency-domain information (audio magnitude or phase) to enable effective extraction of key semantic information from discontinuous mixed-audio signals.

3.2. Self-Supervised Denoising Strategy Foundation

A clean voice dataset for training models demands a huge amount of data; Noise2Noise (N2N) [35] introduces a self-supervised optimization method to overcome the lack of clean data. It uses pairs of independent noisy data m and n (which share the same ground-truth s) for learning. Kashyap [11] proposed that symmetric noisy voice signals in the same environment can also serve as training pairs. Equation (3) indicates the loss function in denoising network $f_{\theta }$ with respect to parameter $\theta$.

$$\underset{\theta }{\arg \min }\mathbb{E}{\parallel f_{\theta }\left(m\right)-n\parallel }_2^2.$$

(3)

As reported in [36], denoising performance comparable to supervised training with ground truth can be achieved by minimizing Equation (4) as follows:

$$\underset{\theta }{\arg \min }\mathbb{E}{\parallel f_{\theta }\left(m\right)-s\parallel }_2^2.$$

(4)

Wu [12] proposed that an audio sub-sampler P can be used to transform the noisy audio signal x into a training pair $(p_1,p_2)$, i.e., $\epsilon =\mathbb{E}\left(p_2\left(n\right)\right)-\mathbb{E}\left(p_1\left(n\right)\right)$, which can be directly utilized to train the denoising network:

$$\underset{\theta }{\arg \min }\mathbb{E}{\parallel f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)\parallel }_2^2.$$

(5)

Given an ideal denoiser $f_{\theta }^{\ast }$ trained on clean data and defined by Equation (5), it guarantees that the output of $f_{\theta }^{\ast }$ for noisy input x is the clean voice y, i.e., $y=f_{\theta }^{\ast }\left(x\right)$. Therefore, the optimal network is as follows:    

$$\begin{array}{cc}\hfill & \mathbb{E}\left\{f_0^{\ast }\left(p_1\left(x\right)\right)-p_2\left(x\right)-\left(p_1\left(f_0^{\ast }\left(x\right)\right)-p_2\left(f_0^{\ast }\left(x\right)\right)\right)\right\}\hfill \\ \hfill & =p_1\left(y\right)-\mathbb{E}\left\{p_2\left(x\right)\right\}-\left(p_1\left(y\right)-p_2\left(y\right)\right)\hfill \\ \hfill & =p_2\left(y\right)-\mathbb{E}\left\{p_2\left(x\right)\right\}=0.\hfill \end{array}$$

(6)

Neighbor2Neighbor (Ner2Ner) [37] indicates that a regularization term $\mu$ can address the non-zero difference between $p_1\left(x\right)$ and $p_2\left(x\right)$ in $\theta$. Inspired by [37], we add spectral constraints to minimize the gap between the output and the target audio, while [38] demonstrates that spectral structure optimization enhances the ability of the convolutional network in voice enhancement. Hence, instead of directly optimizing Equation (5), a constrained optimization problem incorporating Equation (6) and the spectral optimization loss is considered:

$$\begin{array}{cc}\hfill & \mathbb{E}{∥f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)-\left(p_1\left(f_{\theta }\left(x\right)\right)-p_2\left(f_{\theta }\left(x\right)\right)\right)∥}_2^2\hfill \\ \hfill & +\underset{\theta }{argmin}\mathbb{E}{∥f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)∥}_2^2\hfill \\ \hfill & +\delta \mathbb{E}{∥log\left(S\left(f_{\theta }\left(p_1\left(x\right)\right)\right)\right)-log\left(S\left(s_2\left(x\right)\right)\right)∥}_2^2,\hfill \end{array}$$

(7)

where $S\left(f_{\theta }\left(p_1\left(x\right)\right)\right)$ represents the optimized audio spectrum, and $S\left(s_2\left(x\right)\right)$ represents the target audio spectrum.

4. Methodology

Project Overview. The proposed framework comprises three main components: (1) synthetic voice generation for training data expansion (Section 4.1); (2) a self-supervised denoising framework for environmental noise suppression (Section 4.2, Section 4.3, Section 4.4 and Section 4.5); (3) real-world deployment of denoising model in human–robot interaction scenarios (Section 4.6). The overall workflow of the approach is depicted in Figure 3.

Problem Formulation. The voice denoising model can endow a robot with the ability to recognize commands in noisy and audio-overlapping environments. This enables the robot to perform various tasks by leveraging a specific level of environmental perception. Formally, our goal is to enhance the ability of the denoising model to extract and understand conversational information in complex environments.

Model Architecture. As illustrated in Figure 4, a noisy voice signal x is passed through a sub-sampler to generate a noisy training data pair $(p_1\left(x\right),p_2\left(x\right))$, sharing the same ground truth. We transform $p_1\left(x\right)$ into STFT representations as input to the training network, and design the encoding and decoding stages as a hierarchical symmetric U-shaped network. The real and imaginary domains of the audio are denoised separately, with the loss between $f_{\theta }\left(p_1\left(x\right)\right)$ and $p_2\left(x\right)$ being calculated to update network weights.

4.1. Voice Dataset Synthesis

To enhance adaptability to complex environments with noisy human voices, as shown in Figure 3b, two training datasets are generated via audio synthesis, covering multiple scenarios and overlapping voices:

• Multi-scenario dataset: To synthesize audio in a realistic voice dataset across various environments, we employ PyDub (https://pydub.com/, (accessed on 25 August 2025)) to overlap diverse UrbanSound8K (US8K) [39] noise onto clean audio signals from the Voice Bank dataset [40], which includes 28 speakers for training and 2 for testing. The complete noisy speech sample is generated by truncating or repeating the noise to cover the entire voice segment.
• Multi-speaker dataset: Using the same methodology as for generating the multi-scenario dataset, we combine the TIMIT dataset [41], which contains recordings of 630 speaker utterances, with the Voice Bank dataset [40] to generate an audio dataset of overlapping speaker voices. However, direct training on multi-speaker audio fails to achieve effective noise reduction performance. Hence, we employ Kullback–Leibler Divergence [42] to post-process the synthesized audio, thereby generating training data that can be effectively utilized in real-world HRI scenarios (more details in Appendix C).

4.2. Model Encoding and Decoding Stages

Voice Encoder. The encoding stage consists of four consecutive encoders for multi-scale voice spectral feature extraction. Each encoder’s downsampling block contains a complex convolution layer with a complex batch normalization layer (CBN) and a multilayer perceptron (MLP). It employs a complex filter $W=(A+iB)$. For a complex vector $h=c+id$, it performs complex convolution via two independent real convolutions formulated as $W\ast h=(A\ast c-B\ast d)+i(B\ast c+A\ast d)$.

Voice Decoder. The decoder and encoder are symmetric architectures. Each decoder’s upsampling block contains a complex transposed convolution layer, followed by CBN and nonlinear MLP [3]. The decoding stage focuses on reconstructing the spectrogram locally and globally using voice features.

4.3. Self-Supervised Denoising Strategy

We illustrate the optimization function of the self-supervised strategy in Section 3.2, which incorporates the audio spectrum constraint. To prove the feasibility of the optimization problem, the function rationalization and solution existence of Equation (7) are analyzed.

Non-negativity of the Objective Function. For any $\theta$, the squared Euclidean norm is non-negative; since expectation is the integral of the objective function, each term of the objective function is non-negative. We set

$$\left\{\begin{array}{c}{L}_1\left(\theta \right)=\mathbb{E}{\parallel f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)\parallel }_2^2,\hfill \\ L_2\left(\theta \right)=\mathbb{E}{\parallel f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)-p_1\left(f_{\theta }\left(x\right)\right)+p_2\left(f_{\theta }\left(x\right)\right)\parallel }_2^2,\hfill \\ L_3\left(\theta \right)=\mathbb{E}{∥log\left(S\left(f_{\theta }\left(p_1\left(x\right)\right)\right)\right)-log\left(S\left(s_2\left(x\right)\right)\right)∥}_2^2.\hfill \end{array}\right.$$

(8)

Then, the function expectation is expressed as follows:

$$\left\{\begin{array}{c}{\int }_{x,y}{\parallel L_1\left(\theta \right)\parallel }_2^2{f}_{X,Y}(x,y)dxdy\ge 0,\hfill \\ {\int }_{x,y}{\parallel L_2\left(\theta \right)\parallel }_2^2{f}_{X,Y}(x,y)dxdy\ge 0,\hfill \\ {\int }_{x,y}{\parallel L_3\left(\theta \right)\parallel }_2^2{f}_{X,Y}(x,y)dxdy\ge 0,\hfill \end{array}\right.$$

(9)

where $f_{X,Y}(x,y)$ represents the joint probability density function of two-dimensional random variables x and y. The objective function $L\left(\theta \right)=L_1\left(\theta \right)+\mu L_2\left(\theta \right)+\delta L_3\left(\theta \right)\ge 0$, where $\delta \ge 0$.

Existence of Solutions within the Parameter Space. The particular parameter space $\Omega$ is assumed to be a compact set (bounded and closed) in ${\mathbb{R}}^n$. Since $f_{\theta }$ is continuous with respect to $\theta$ (a differentiable function is continuous) and $p_1$ and $p_2$ are measurable functions, by the property of the composition of continuous functions. There exist

$$\left\{\begin{array}{c}{g}_1(\theta ,x)={\parallel f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)\parallel }_2^2,\hfill \\ g_2(\theta ,x)={\parallel f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)-p_1\left(f_{\theta }\left(x\right)\right)+p_2\left(f_{\theta }\left(x\right)\right)\parallel }_2^2,\hfill \\ g_3(\theta ,x)={∥log\left(S\left(f_{\theta }\left(p_1\left(x\right)\right)\right)\right)-log\left(S\left(s_2\left(x\right)\right)\right)∥}_2^2.\hfill \end{array}\right.$$

(10)

Suppose that $f_{\theta }$ is bounded, i.e., there exist constants $C_1$ and $C_2$ such that $|f_{\theta }\left(p_1\left(x\right)\right)| \le C_1$ and $|p_1\left(f_{\theta }\left(x\right)\right)| \le C_2$ for all $\theta \in \Omega$.

It can be shown that there exist integrable functions $M_1\left(x\right)$ and $M_2\left(x\right)$ such that $|g_1(\theta ,x)|\le M_1\left(x\right)$ and $|g_2(\theta ,x)|\le M_2\left(x\right)$. Since logarithmic operations exhibit continuity within their respective domains, $g_3(\theta ,x)$ is continuous in $\theta$, and there exists an integrable function $|g_3(\theta ,x)|\le M_3\left(x\right)$ for all $\theta \in \Omega$.

Hence, the expectations $L_1\left(\theta \right)$, $L_2\left(\theta \right)$, and $L_3\left(\theta \right)$ are continuous in $\theta$. Therefore, $L\left(\theta \right)=L_1\left(\theta \right)+L_2\left(\theta \right)+\delta L_3\left(\theta \right)$ is continuous in $\theta$. According to the Lebesgue dominated convergence theorem [43], on the compact parameter space $\Omega$, there always exists ${\theta }^{\ast }\in \Omega$ such that $L\left({\theta }^{\ast }\right)=minL\left(\theta \right)$.

4.4. Denoising Network

Conventional 2D convolution kernels with fixed geometries usually struggle to capture detailed features in complex spectral data, particularly in sub-sampled voice signals. As formulated in Equations (1) and (2), directly separating the target voice from the noisy signal $M\left(z\right)$ in the noisy environment is challenging.

To address this limitation, separating the real and imaginary components of the audio spectrum enables the denoising model to optimize magnitude and phase independently. Audio processing without component separation fails to establish a clear phase spectrogram structure [44], which is essential for perceptual quality in voice denoising. Hence, we aim to estimate the real and imaginary spectrograms of the target voice from the noisy signal via complex spectral mapping, simultaneously capturing signal structure and preserving phase information.

Therefore, our goal is to separately denoise the information in different dimensions (real and imaginary) to enable more effective extraction of audio features. The denoising network model $\pi$ is composed of a convolutional multi-head attention architecture followed by a ResNet module (shown in Figure 5). Taking the real part $X_r$ and imaginary part $X_i$ of the voice spectral data as input, the model produces a complex-valued output Y defined as

$$Y_r=\pi \left(X_r\right)-\pi \left(X_i\right),$$

(11)

$$Y_i=\pi \left(X_r\right)+\pi \left(X_i\right),$$

(12)

where $\pi \left(X_r\right)$ and $\pi \left(X_i\right)$ represent the real and imaginary parts of the denoising model, and $Y=Y_r+j{Y}_i$ denotes the model’s complex value output. This approach enables precise processing and reconstruction of spectrogram amplitude and phase data, while preserving voice contextual details. It improves the ability of robots to accurately understand key instructions during interactions (more details in Section 4.6).

4.5. Training Loss

Our primary objective is to satisfy the constraints of sub-sampled audio pairs by solving the optimization function in Equation (7) for voice denoising. Hence, the loss function consists of $L_{\mathrm{basic}}$ and $L_{\mathrm{reg}}$.

Basic Loss. We define the basic loss $L_{\mathrm{basic}}$ [45] as a combination of the waveform loss and an STFT loss to fit the target audio waveform:

$$\begin{array}{cc}\hfill L_{\mathrm{basic}}& ={\displaystyle \frac{1}{TF}}\sum _{t=0}^{T-1}\sum _{f=0}^{F-1}\left(|S_r(t,f)| + |S_i(t,f)|\right.\hfill \\ \hfill & \left.-|{\widehat{S}}_r(t,f)| - |{\widehat{S}}_i(t,f)|\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{N}}\sum _{i=0}^{N-1}{(n_i-{\widehat{n}}_i)}^2,\hfill \end{array}$$

(13)

where $n_i$ and ${\widehat{n}}_i$ denote the i-th sub-sampled audio pair and its denoised version, respectively, with N being the total number of speech samples. Let S and $\widehat{S}$ be the spectrograms of the sub-sampled and its denoised version, respectively, where T represents the number of frames and F represents the number of frequency bins.

Regularization Loss. We also need to consider the regularization loss discussed in Section 3.2. The regularization loss $L_{\mathrm{reg}}$, formulated as $L_{\mathrm{reg}}=L_{\mathrm{reg}1}+L_{\mathrm{reg}2}$, is defined as follows:

$$L_{\mathrm{reg}1}={∥f_{\theta }\left(p_1\left(x\right)\right)-p_2\left(x\right)-\left(p_1\left(f_{\theta }\left(x\right)\right)-p_2\left(f_{\theta }\left(x\right)\right)\right)∥}_2^2,$$

(14)

$$L_{\mathrm{reg}2}={∥logS\left(f_{\theta }\left(p_1\left(x\right)\right)\right)-logS\left(s_2\left(x\right)\right)∥}_2^2.$$

(15)

The total loss L is the sum of the basic loss and the regularization loss, as $L=L_{\mathrm{basic}}+L_{\mathrm{reg}1}+\delta L_{\mathrm{reg}2}$. The performance of the model can be enhanced by adjusting the value of the parameters $\delta$ (for more details see Section 5.3).

4.6. Human–Robot Voice Interaction System

In this work, voice recognition is established as a precondition for robot task execution during HRI. As described in Section 4.4, the denoising model takes noisy speech x as input and outputs audio y, which contains the task instructions for guiding robot action. It enables the robot to more effectively identify key semantics in interaction by reducing background noise and overlapping voices. Based on the action information (i.e.,  pick, grasp, pull), the system executes various actions to interact with the environment. We develop an interaction system that integrates voice recognition and action execution. The VLA model is employed to generate robot action, with denoising voice serving as both the supporting condition for language prompts and the prerequisite for action execution.

An example is shown in Algorithm 1. When the interaction command is “Grasp the steel pipe and place it on the table behind”, the robot identifies the key semantics of the task instruction in the voice:

• Grasp: target task action;
• Steel Pipe: target object;
• Table Behind: placing position.

This approach provides sufficient task instructions and action information. Additionally, it offers an efficient approach to verify whether the audio has been effectively denoised. If the key semantic information within the denoised voice remains difficult to discern, the robot will be unable to execute the task action. Meanwhile, to enhance edge-case handling, we design action termination prompts (i.e.,  stop, no, wait). They can instantaneously terminate inference actions during HRI, prompting the robot to await subsequent task instructions (see Section 5.5 for a detailed discussion of our test).

Algorithm 1 Human–robot voice interaction

Input: noisy voice     Output: robot action     RobotTask(noisy_voice)          interaction_audio ← Denoise(noisy_voice)          instruction ← RecognizeVoice(interaction_audio)          action ← VLAModel(instruction)          return RobotExecute(action)

5. Experiment

In Section 5.2, we compare the denoising ability and generalization results of our method with previous self-supervised methods in simulation environments. The effectiveness of each component is validated in Section 5.3. In Section 5.4, we present application results of our method in real-world scenarios, including single-arm, dual-arm, and humanoid robot tasks. The effectiveness of termination prompts is tested in Section 5.5, demonstrating they can achieve closed-loop voice interaction control.

5.1. Experiment Setup

Dataset. Noisy signals from three distinct environments are used to evaluate algorithm robustness under varied conditions. In addition to the synthetic dataset introduced in Section 4.1, clean speech is augmented with white Gaussian noise to generate a synthetic white-noise training dataset. The training set includes 6000 white-noise voices as Env1, randomly selects 10,000 voices from the multi-scenario dataset as Env2, and takes 8000 voices from the multi-speaker dataset as Env3.

Training Details. In the denoising model, most convolutional layers adopt a 3 × 3 kernel and a 2 × 2 stride, while the middle two layers in the encoder and decoder utilize a 2 × 1 stride. We employ an Adam optimizer with a learning rate of 0.001, and the complex-valued voice spectrograms for model input are obtained via STFT with a 64 ms Hamming window and 16 ms hop size. All experiments were conducted on a workstation equipped with an Intel Core i9-14900K CPU and an NVIDIA RTX 4090 GPU for both data training and evaluation.

5.2. Evaluation in Simulation

Metric. To evaluate the denoising effect quality, we use five metrics in the simulation experiment [46,47,48]: signal-to-noise ratio (SNR), segmental signal-to-noise ratio (SSNR), wide-band perceptual evaluation of speech quality (PESQ-WB), narrow-band perceptual evaluation of speech quality (PESQ-NB), and short-term objective intelligibility (STOI).

Baselines. To comprehensively evaluate our model, comparisons are made with state-of-the-art baselines in voice denoising and self-supervised learning, including NCT [11], NNT [11], NerNT [10], and ONT [12]. These methods use noisy voice for model training and testing in the Env1-Env3 datasets. For more details on the differences in the training processes of the methods, see Appendix D.

Result. To ensure fair experimental evaluation, all baseline models are pre-trained on the same synthetic datasets.

Table 1. Voice denoising evaluation with different methods.

Dataset	Network	SNR	SSNR	PESQ-NB	PESQ-WB	STOI
Env1	NCT	$17.323\pm 3.488$	$4.047\pm 4.738$	$2.655\pm 0.428$	$1.891\pm 0.359$	$0.655\pm 0.18$
	NNT	$16.937\pm 3.973$	$3.752\pm 4.918$	$2.597\pm 0.462$	$1.943\pm 0.375$	$0.645\pm 0.018$
	NerNT	−	−	−	−	−
	ONT	$17.563\pm 2.596$	$8.389\pm 2.961$	$2.690\pm 0.347$	$1.878\pm 0.293$	$0.833\pm 0.066$
	ONT + rTSTM	$18.137\pm 2.122$	$9.077\pm 2.437$	$2.643\pm 0.317$	$2.003\pm 0.282$	$0.839\pm 0.067$
	ONT + cTSTM	$18.209\pm 2.095$	$9.088\pm 2.222$	$2.811\pm 0.288$	$1.997\pm 0.276$	$0.847\pm 0.09$
	Ours	$\mathbf{18}.\mathbf{671}\pm \mathbf{3}.\mathbf{093}$	$\mathbf{9}.\mathbf{628}\pm \mathbf{3}.\mathbf{002}$	$\mathbf{2}.\mathbf{961}\pm \mathbf{0}.\mathbf{508}$	$\mathbf{2}.\mathbf{166}\pm \mathbf{0}.\mathbf{326}$	$\mathbf{0}.\mathbf{864}\pm \mathbf{0}.\mathbf{073}$
Env2	NCT	$3.44\pm 3.457$	$-0.684\pm 3.767$	$1.773\pm 0.326$	$1.326\pm 0.190$	$0.52\pm 0.188$
	NNT	$3.99\pm 5.451$	$-0.402\pm 5.084$	$2.147\pm 0.535$	$1.55\pm 0.372$	$0.593\pm 0.221$
	NerNT	$3.537\pm 3.465$	$-1.336\pm 3.105$	$1.787\pm 0.26$	$1.249\pm 0.126$	$0.569\pm 0.177$
	ONT	$5.64\pm 2.18$	$-0.28\pm 0.69$	$2.15\pm 0.189$	$1.76\pm 0.39$	$0.54\pm 0.56$
	ONT + rTSTM	$5.71\pm 2.62$	$-0.23\pm 1.36$	$2.23\pm 0.27$	$1.83\pm 0.35$	$0.68\pm 0.27$
	ONT + cTSTM	$5.86\pm 3.44$	$-0.15\pm 3.6$	$2.41\pm 0.304$	$1.97\pm 0.21$	$0.72\pm 0.019$
	Ours	$\mathbf{6}.\mathbf{029}\pm \mathbf{3}.\mathbf{431}$	$-\mathbf{0}.\mathbf{074}\pm \mathbf{0}.\mathbf{993}$	$\mathbf{2}.\mathbf{818}\pm \mathbf{0}.\mathbf{556}$	$\mathbf{2}.\mathbf{123}\pm \mathbf{0}.\mathbf{26}$	$\mathbf{0}.\mathbf{87}\pm \mathbf{0}.\mathbf{23}$
Env3	NCT	$3.75\pm 3.899$	$-1.915\pm 3.664$	$1.924\pm 0.313$	$1.37\pm 0.208$	$0.562\pm 0.201$
	NNT	$4.428\pm 3.166$	$-1.824\pm 4.558$	$2.445\pm 0.481$	$1.77\pm 0.41$	$0.634\pm 0.199$
	NerNT	$4.464\pm 3.858$	$-1.837\pm 3.714$	$2.321\pm 0.351$	$1.484\pm 0.256$	$0.651\pm 0.198$
	ONT	$4.89\pm 1.62$	$-1.52\pm 2.45$	$2.58\pm 0.317$	$1.58\pm 0.273$	$0.662\pm 0.16$
	ONT + rTSTM	$5.14\pm 2.63$	$-1.45\pm 2.36$	$2.61\pm 0.17$	$1.69\pm 0.31$	$0.674\pm 0.18$
	ONT + cTSTM	$5.327\pm 3.309$	$-1.391\pm 4.22$	$2.65\pm 0.49$	$1.84\pm 0.324$	$0.69\pm 0.21$
	Ours	$\mathbf{5}.\mathbf{416}\pm \mathbf{1}.\mathbf{232}$	$-\mathbf{1}.\mathbf{223}\pm \mathbf{2}.\mathbf{458}$	$\mathbf{2}.\mathbf{727}\pm \mathbf{0}.\mathbf{437}$	$\mathbf{1}.\mathbf{991}\pm \mathbf{0}.\mathbf{189}$	$\mathbf{0}.\mathbf{712}\pm \mathbf{0}.\mathbf{16}$

shows that our model has achieved notable improvements in different environments and outperformed other baselines. Moreover, it has the ability to denoise in multi-speaker overlapping scenarios. This is because (1) the scaling dataset enhances self-supervised denoising performance across diverse scenarios and speaker backgrounds, and (2) the convolutional multi-head attention network improves feature extraction from discrete time–frequency representations (e.g., real and imaginary components).

We validate the model’s effectiveness by comparing its denoising performance with ONT [12] using a real-world utterance. Figure 6 presents the spectral changes before and after model processing. As shown in Figure 6a, the original interactive voice is heavily corrupted by noise, making it difficult to directly discern speech features (annotated in red). From Figure 6b,c, it is evident that the proposed method achieves significantly better performance in audio denoising. Figure 6c retains more command audio harmonics with less distortion than Figure 6b, demonstrating our method’s superior performance in a real-world scenario.

Generalization Evaluation. We evaluate the model’s generalization ability in unseen scenarios and real-world multi-speaker voices.

Table 2. Generalization evaluation of unseen scenarios.

Training	Testing	Network	SNR	PESQ-NB
Factory	Street	NCT	$3.645\pm 3.676$	$1.8\pm 0.46$
		NNT	$3.948\pm 3.285$	$2.121\pm 0.467$
		NerNT	$3.799\pm 3.293$	$2.114\pm 0.459$
		ONT	$4.38\pm 2.31$	$2.33\pm 0.39$
		ONT + rTSTM	$4.65\pm 2.35$	$2.58\pm 0.74$
		ONT + cTSTM	$4.91\pm 2.46$	$2.73\pm 0.4$
		Ours	$\mathbf{5}.\mathbf{11}\pm \mathbf{4}.\mathbf{825}$	$\mathbf{2}.\mathbf{92}\pm \mathbf{0}.\mathbf{435}$

presents the denoising results of PESQ-NB and SNR under different scenario conditions, using training and testing datasets sourced from Env2. The model is trained using interactive voices under industrial noise (i.e., factory environment) and tested using interactive voices with street music (i.e., street environment). The results indicate that the model can successfully adapt to an unseen scenario.

Table 3. Generalization evaluation of multi-speaker voices.

Network	k = 0	k = 1	k = 2
NCT	$2.172\pm 0.367$	$1.924\pm 0.313$	$1.78\pm 0.269$
NNT	$2.583\pm 0.392$	$2.445\pm 0.481$	$2.17\pm 0.635$
NerNT	$2.56\pm 0.26$	$2.321\pm 0.351$	$2.13\pm 0.29$
ONT	$2.63\pm 0.16$	$2.58\pm 0.317$	$2.24\pm 0.439$
ONT + rTSTM	$2.68\pm 0.21$	$2.61\pm 0.17$	$2.29\pm 0.524$
ONT + cTSTM	$2.71\pm 0.23$	$2.65\pm 0.49$	$2.38\pm 0.15$
Ours	$\mathbf{2.802}\pm \mathbf{0.376}$	$\mathbf{2.727}\pm \mathbf{0.437}$	$\mathbf{2.41}\pm \mathbf{0.29}$

presents the the PESQ-NB evaluation results for multi-speaker voices from Env3, where k denotes the number of speakers. When $k=0$ (signifying no overlapping noise), we select three values (0, 1, 2) for the experiment. The presented method maintains effective noise reduction as k increases, achieving the highest voice quality compared to other methods.

5.3. Ablation Study

Analysis of Regularization Loss. Here we analyze the sensitivity of the regularization loss of the model optimization function in Equation (7). The SNR evaluation results are shown in

Table 4. Ablation of regularization loss weights.

Noise	δ = 0	δ = 1	δ = 2	δ = 4	δ = 10
Env1	2.832	2.961	2.879	2.793	2.524
Env2	2.783	2.818	2.837	2.774	2.328
Env3	2.673	2.727	2.711	2.668	2.192

, with five values (0, 1, 2, 4, 10) selected for experiments. The results demonstrate that model performance degrades when $\delta$ exceeds 2. This is because excessive constraints induce critical information loss, thereby leading to a poorer denoising effect. However, when $\delta$ is zero, the model fails to bridge the gap between the audio sub-sampler and ground truth, causing training instability.

Analysis of Network Module. Next, we analyze the sensitivity of different modules: (a) spectrogram training without separating real and imaginary components (denoted as Net-1), (b) feature extraction using convolutions without the attention module (denoted as Net-2), and (c) feature extraction using only the attention module (denoted as Net-3).

Table 5. Ablation of different network modules.

Dataset	Network	SNR	SSNR	PESQ-NB	PESQ-WB	STOI
Env1	Net-1	16.831	3.996	2.655	1.728	0.598
	Net-2	18.274	7.811	2.864	1.921	0.816
	Net-3	17.348	5.921	2.715	1.814	0.726
	Ours	18.671	9.268	2.96	2.166	0.864

presents the denoising results, with the full model demonstrating the best performance. Two key reasons are identified: (1) Decomposing voice into real and imaginary parts can denoise time–frequency and phase simultaneously, enhancing information extraction from discontinuous audio. (2) The convolutional attention module can effectively suppress overlapping noise and extract critical voice features.

5.4. Evaluation in Real World

Hardware Setup. We perform real-world experiments using various robots: the Elite robot arm (7 DOFs), the dual-arm UR robot (12 DOFs), and the Realman humanoid robot (14 DOFs). All robotic arms are equipped with 1 DOF grippers, and external microphones are used for voice interaction, as shown in Figure 7.

Tasks and Metric. As shown in Figure 7 (right), we designed different tasks to evaluate model performance in real-world noisy scenarios, testing basic actions (e.g., pull, grasp, wipe) through voice interaction. The robots would recognize critical information in noisy interactive voice and execute instructed actions. The action success rate, calculated across multiple experiments, served as the primary metric.

Implementation Details. Figure 7 (left) shows the interactive environment. We designed two experimental conditions to evaluate real-world model performance: (1) computer-simulated urban music environments (https://pixabay.com/sound-effects/search/city/, (accessed on 25 August 2025)), and (2) overlapping speech with interfering speakers during voice interaction. The microphone collects interactive voice, and the noise meter measures environmental noise. We consider 50–70 dB as normal noise and above 70 dB as high-frequency noise (https://www.epa.gov/archive/epa/aboutepa/epa-identifies-noise-levels-affecting-health-and-welfare.html, (accessed on 25 August 2025)), and select RRnoise [49], DeepFilterNet [50], and ONT [12] as baselines with frozen weights to provide the denoised audio containing action information to the robot.

Result.

Table 6. Voice denoising evaluation in real word.

Method	Urban Music (dB)		Multi-Speaker (dB)		Avg Time (s)
	50–70	70–80	50–70	70–80
RRnoise	0.91	0.87	0.78	0.67	4.2
DeepFilterNet	0.94	0.91	0.82	0.72	1.9
ONT	0.92	0.90	0.84	0.79	3.1
Llama-Omni	0.98	0.97	0.86	0.82	3.7
Ours	0.96	0.94	0.89	0.85	1.2

presents the instruction recognition success rates of these strategies, indicating that our model exhibits excellent generalization capabilities across various scenarios and overlapping interactive voices with shorter response times. It achieves notable improvements in noisy and overlapping environments, particularly under 70–80 dB mixed-audio conditions, outperforming DeepFilterNet [50] and ONT [12] by 15% and 7.5% in success rate, respectively. Moreover, it can recognize action information in a discrete noisy voice, as shown in Figure 8. The enhanced performance is attributed to the use of training data augmented with overlapping voices, improving denoising capability in multi-speaker scenarios through accurate modeling of mixed-speech discontinuity.

The proposed model is also compared with state-of-the-art LLMs. The result indicates that Llama-Omni [15] handles ambient noise more effectively in a musical environment, demonstrating stronger robustness. However, its performance declines significantly in a multi-speaker scenario, whereas the proposed method achieves higher recognition success rates under the tested conditions. The performance degradation may occur because the multi-speaker conversational audio in its training data is out of distribution (OOD), making it difficult for the reasoning chain to infer target instructions. Notably, our model contains eight hundred thousand more parameters, which is relatively minor when compared to an 8B-parameter LLM backbone [5], while achieving shorter inference time for audio recognition.

5.5. Terminate Action Test

To verify the feasibility of interruptible interaction, we tested task-action changes through termination prompts during the interaction, as shown in Figure 9. The robot first executes the instruction “Pick the orange beside the apple”. At $t=7$ s, the “Wait” interrupt stops the action, and the robot returns to the initial pose. When $t=10$ s, a new instruction “Grasp the yellow cup handle and move to the white mat” initiates a new task. Termination words and instructions allow us to adjust action information, achieving a closed-loop HRI. Our mind is focused on improving the safety of the execution of actions through interrupt words, allowing the model to handle execution actions in unseen environments in a controlled manner.

6. Conclusions

In this paper, we introduce a self-supervised strategy for voice denoising, enabling effective HRI in unseen noisy real-world environments. Specifically, our method generates multi-scenario noisy voice datasets through data synthesis, constructs an effective complex-valued denoising network, and recognizes critical action information from discrete mixed audio. With the rapid advancement of HRI, this study focuses on enhancing recognition efficiency for action commands in noisy environments.

Limitations. Due to hardware limitations, real-word tasks in a laboratory cannot fully replicate outdoor environmental sounds, posing challenges for practical algorithm application. Moreover, convolutional attention networks cannot effectively separate and transfer speech attributes (e.g., content, pitch, timbre) [19], limiting style-preserving denoising. In future research, we will adapt flow matching for discrete speech modeling and deploy it on the kid-sized humanoid robot for outdoor validation.