Multi-Task Flow Matching Speech Enhancement Method with Dual Guidance of Semantic Understanding and Speaker Perception

Abstract

Generative speech enhancement models often suffer from the “generative hallucination” issue, where artifacts are introduced or original speech characteristics are distorted during reconstruction due to insufficient prior constraints, severely limiting their practicality. To address this, we propose a multi-task flow-matching speech enhancement network with dual guidance from semantic understanding and speaker perception. The method employs a flow-matching model as the generative backbone, leveraging both semantic and speaker embeddings as auxiliary conditions to guide the denoising process. The semantic embedding, extracted by a pretrained audio foundation model, provides high-level content priors to ensure the intelligibility and clarity of the enhanced speech. The speaker embedding, derived from a speaker verification model, captures timbre representations, which are further regularized by a contrastive InfoNCE loss to enhance intra-class compactness and inter-class separability of speaker characteristics. During optimization, the primary denoising reconstruction task and the auxiliary speaker discrimination task are jointly optimized. The two guidance streams, together with acoustic features, are collaboratively fed into the generation network, effectively suppressing hallucinations from both content correctness and timbre consistency perspectives. Experimental results demonstrate that the proposed multi-task framework substantially mitigates hallucination-related errors in generative scenarios, achieving comprehensive improvements in objective metrics, including both reference-based measures and non-intrusive perceptual metrics, thus validating the effectiveness of the dual-guidance strategy for generative speech enhancement.

1. Introduction

In speech signal processing, discriminative methods perform effectively under stationary noise, yet their deterministic mapping tends to cause over-smoothing in low-SNR or non-stationary conditions, often at the cost of fine speech details. Generative approaches, by contrast, learn the underlying distribution of clean speech as a prior, enabling high-quality and highly natural reconstructions even under severe interference. Early generative enhancement models, such as variational autoencoders (VAEs) [1] and generative adversarial networks (GANs) [2], suffered from posterior estimation errors or training instability, which limited their fidelity and naturalness. More recently, diffusion models have offered a robust learning paradigm for high-fidelity speech generation, thanks to their well-defined forward diffusion and reverse denoising processes [3]. The step-wise refinement inherent in diffusion models aligns naturally with the time-frequency structure of speech, allowing progressive noise suppression while preserving essential acoustic details. Nevertheless, the iterative sampling required by diffusion models incurs high inference latency, which hinders real-time deployment. To overcome this bottleneck, FlowSE [4] adopts the flow matching framework combined with Diffusion Transformer (DiT) modules. By learning a continuous probability flow ordinary differential equation that maps noise to clean speech, it attains high-fidelity reconstruction with substantially fewer sampling steps than conventional diffusion models, thereby greatly accelerating inference.

Notably, none of the above methods explicitly or sufficiently exploits high-level semantic content or speaker identity cues from the input speech. When the input is heavily distorted, acoustic features alone provide limited guidance in distinguishing “meaningful speech” from erroneous reconstructions. This ambiguity can cause the generative model to introduce spurious components that deviate from the true source distribution, or to warp original voice characteristics—a phenomenon known as generative hallucination. In this work, we define generative hallucination as any content or speaker attribute in the enhanced speech that is not supported by the clean reference. It manifests in two forms: linguistic hallucination, where the model inserts or substitutes words absent from the clean speech, and acoustic hallucination, where the linguistic content is correct but the speaker identity or timbre drifts from the reference. Hallucinations not only degrade intelligibility and naturalness, but can also distort semantic meaning and alter speaker identity, critically undermining the reliability of generative enhancement in practice.

Recent work has explored semantic guidance for flow-matching speech enhancement. However, existing approaches still suffer from certain limitations. For instance, FlowSE [4] relies on external text transcripts as guidance, while allowing this condition to be omitted during inference. Such a loose dependency undermines the stability of guidance and restricts its applicability to scenarios where text annotations are unavailable. SenSE [5], on the other hand, adopts a two-stage training scheme: it first extracts semantic tokens via a language model, and then uses these tokens to guide the flow matching process. The discretization step inevitably incurs a loss of acoustic information, and the optimization objectives of the two stages may become misaligned, potentially introducing bias into the final reconstruction.

From a complementary perspective, G-MaP-SE [6] tackles the problem from a speaker information perspective. It constructs a prior distribution over clean-speech speaker embeddings using a Gaussian mixture model (GMM), and then matches the noisy speaker embeddings towards this prior, thereby providing stable speaker identity guidance for the enhancement network. Nevertheless, the reliance on a fixed GMM prior fundamentally limits its generalization ability. In real-world scenarios, the distribution of speaker embeddings is highly complex and constantly evolving; a pre-fitted static prior can hardly cover unseen speakers, new accents, or novel acoustic conditions. When test data deviates from the training prior, the matching process may fail or even introduce additional bias, thus compromising the robustness and practical reliability of the method. To address the above limitations, we propose a multi-task flow matching speech enhancement method with dual guidance from semantic understanding and speaker perception. The main contributions of this work are threefold:

1.

Building a unified multi-task learning framework in which semantic understanding and speaker perception serve as consistently active auxiliary tasks. The semantic branch provides a stable content prior, while the speaker branch supplies timbre identity guidance; together, they jointly constrain the generation process.

2.

Leveraging a frozen pretrained large audio model (Qwen2-Audio-7B) to extract continuous high-level semantic embeddings, thus circumventing discrete quantization of semantic information. This yields richer and more direct content priors for the generation process.

3.

Devising a speaker branch to explicitly model speaker identity. Instead of the absolute GMM matching used in G-MaP-SE [6], we impose a relative discriminative constraint via InfoNCE loss on the embedding space, which preserves greater representational freedom for unseen speakers and effectively avoids timbre distortion.

The two streams of guidance, together with acoustic features, are fed into the generation network, suppressing linguistic and acoustic hallucinations from content correctness and timbre consistency, respectively. Experiments verify that the proposed multi-task framework significantly mitigates hallucination phenomena in generative scenarios and effectively improves the overall performance of generative speech enhancement.

We name our method SS-FMSE (Semantic and Speaker-aware Flow Matching for Speech Enhancement). The rest of this paper is organized as follows: Section 2 reviews related work; Section 3 details the proposed SS-FMSE; Section 4 presents the experimental setup and results; and Section 5 concludes the paper.

2. Related Work

2.1. Flow Matching-Based Generative Speech Enhancement Methods

Flow matching [7] is a generative modeling framework that learns a continuous-time probability flow to smoothly transform a simple noise distribution into a complex target data distribution. It directly learns a deterministic, time-dependent velocity field $v_t\left(x_t\right)$, where t denotes the time step along the flow matching trajectory and $x_t$ is the data distribution at that position. This velocity field describes how a noise point $x_0$ is smoothly pushed to a data point $x_1$ over the interval $t\in [0,1]$. The underlying assumption is that for any pair of samples $x_0\sim p_0$ (from a simple distribution, e.g., Gaussian noise) and $x_1\sim p_1$ (from the target data distribution), there exists a smooth path connecting them. Taking linear interpolation as an example,

$$x_t=(1-t)x_0+t{x}_1$$

(1)

the true velocity is given by

$${\dot{x}}_t=x_1-x_0.$$

(2)

The neural network is trained to approximate this velocity field by minimizing the mean-square error (MSE) between predicted and true velocities. Generation then only requires solving an ordinary differential equation (ODE), which can be completed in a few time steps, bypassing the redundant iterative denoising steps.

Flow matching has shown promising results in speech enhancement and separation. FlowSep [8] combines flow matching with a VAE and introduces language conditioning to achieve semantically guided audio source separation. FlowAVSE [9] adopts a U-Net backbone for flow matching, conditioned on both audio and visual features, and achieves competitive performance. FlowSE [4] integrates flow matching with the Transformer-based diffusion framework DiT, laying a foundation for applying flow matching to speech enhancement. However, FlowSE relies on optional external text transcripts rather than directly perceiving semantic content from the speech waveform, which renders semantic guidance unstable and restricts its applicability in real-world scenarios where text annotations are often unavailable.

2.2. Semantic-Guided Generative Methods

Leveraging high-level semantic information to guide the generation process has become an important trend in modern generative models, and its effectiveness has been well demonstrated in image generation. Zhang et al. [10] exploit the reasoning capabilities of multimodal large language models to guide the image generation process of diffusion models, using a semantic-to-detail connector that aligns fine-grained visual features with semantic space in advance, thereby enabling joint guidance from both global semantics and local details. Chang et al. [11] directly optimize semantic embeddings to guide text-to-image diffusion models for editing tasks, without relying on text prompts or model training. The core idea behind these works is highly consistent: they extract high-level semantic embeddings using pretrained models and inject them as conditions into the diffusion model to constrain the semantic direction of the generation process.

Similar approaches have also been widely adopted in audio and speech generation. Ren et al. [12] propose STA-V2A, a video-to-audio generation model that extracts global semantic and local temporal features from video, and combines them with text as cross-modal guidance, producing audio that closely matches the video content. Li et al. [13] present PL-TTS, which integrates prompt embeddings extracted by a large language model with a diffusion-based TTS system, effectively improving both synthesis quality and style controllability. These studies demonstrate that using semantic embeddings from pretrained models as conditional guidance for diffusion or flow matching models has become a well-established and effective technical approach in the generative AI community.

In the domain of speech enhancement, researchers have also explored the introduction of semantic information. SenSE [5] adopts a two-stage generation framework. It first extracts semantic tokens from distorted speech using a semantic-aware speech language model to capture semantic priors, and then feeds these tokens together with the mel-spectrogram of the distorted speech as conditions to guide the flow matching-based enhancement process. Experimental results confirm that the introduction of semantic information effectively alleviates semantic ambiguity in generation. Nevertheless, the two-stage decoupled design of SenSE has inherent limitations. In the first stage, the abstraction of continuous speech content into discrete tokens inevitably leads to the loss of some fine-grained acoustic information.

Moreover, the optimization objectives of the two stages are independent, which may cause misalignment between them and restrict the model’s ability to fully exploit rich contextual information. More importantly, none of the existing methods simultaneously leverages semantic content and speaker identity as conditional guidance for speech enhancement (see Table 1 for a systematic comparison). Relying solely on content-level prior constraints is insufficient to suppress acoustic hallucination, i.e., identity feature drift during the generation process.

Table 1. Comparison of representative guidance-based speech enhancement methods. “Semantic content” and “Speaker identity” indicate whether the corresponding information is explicitly used as a condition. “Training scheme” indicates whether the method is trained in one stage or multiple stages. “Prior type” indicates the form of the guidance prior.

Method	Semantic Content	Speaker Identity	Training Scheme	Prior Type
SenSE [5]	✓	×	Two-stage	Discrete semantic tokens
FlowSE [4]	✓	×	One-stage	Text embedding
G-MaP-SE [6]	×	✓	One-stage	Static GMM prior

3. Proposed Method

3.1. Model Overview

We propose a multi-task flow matching speech enhancement method with dual guidance from semantic understanding and speaker perception. The model adopts flow matching as the generative backbone, and its overall architecture is illustrated in Figure 1. To address the generative hallucination problem caused by the lack of high-level priors, the model leverages both semantic content and speaker identity as conditional information to guide the entire flow-field prediction process. Specifically, the semantic branch provides content-correctness constraints to suppress linguistic hallucination, while the speaker branch imposes identity-consistency constraints to suppress acoustic hallucination.

Figure 1. Overview of the SS-FMSE architecture. Three conditioning streams—distorted-speech mel-spectrogram $M^N$, semantic embeddings $E^{Se}$, and speaker embeddings $E^{Sp}$—are concatenated with $M^T$ and fed into the DiT backbone. The training path jointly optimizes the flow loss and the InfoNCE contrastive loss on the speaker branch, while the inference path integrates the flow field via an ODE solver and reconstructs the waveform with a vocoder.

The model treats clean-speech mel-spectrograms as samples from the target data distribution, which are mixed with randomly sampled standard Gaussian noise along a probability path. Let $M^T$ denote the intermediate state at time step T, which serves as the core input to the DiT network. The mel-spectrogram of the distorted speech, denoted as $M^N$, is incorporated as a conditional embedding. For guidance, the semantic branch employs a pretrained audio large language model to extract semantic content embeddings $E^{Se}$; the speaker branch adopts a frozen CAM++ model to extract speaker embeddings $E^{Sp}$, which are then fine-tuned via a lightweight linear adaptation layer, with a discriminative constraint imposed on the mapped embedding space through the InfoNCE contrastive loss. The three streams of conditional information—namely, the distorted speech mel-spectrogram $M^N$, semantic embeddings $E^{Se}$, and speaker embeddings $E^{Sp}$—are concatenated with $M^T$ along the channel dimension and fed into the DiT network, which stacks N transformer modules to predict the flow field. The flow loss serves as the primary optimization objective for joint multi-task training. During inference, the model integrates the predicted flow field using an ODE solver, iteratively sampling from pure noise to clean data, and the resulting mel-spectrogram is reconstructed into a time-domain enhanced waveform via a vocoder.

3.2. Semantic and Speaker Embedding

For semantic guidance, we adopt Qwen2-Audio-7B, a pretrained audio large language model, as the semantic feature extractor. Benefiting from pretraining on massive and diverse audio data, this model has demonstrated a strong ability to abstract speech content under the evaluated conditions. When the input waveform is distorted by environmental noise or the local spectral envelope is altered, it can still typically disentangle interference components from the time-frequency aliased acoustic signal and recover useful semantic information. Traditional acoustic features, such as MFCCs or STFT magnitude spectra, tend to degrade rapidly as noise power increases, struggling to retain valid phonetic content cues. In contrast, the high-level semantic embeddings produced by Qwen2-Audio-7B tend to be more anchored to the linguistic content of speech, providing relatively stable content-level priors for the subsequent backbone generation network. By injecting these embeddings as conditional information into the DiT network, the flow matching model can focus its limited generative capacity on frequency regions that are semantically plausible when solving the probability transport path, thereby reducing erroneous speech reconstruction caused by noise-induced ambiguity.

For speaker identity representation, we employ a pretrained speaker recognition model, CAM++, to extract embeddings related to individual attributes such as timbre and prosody. Although CAM++ has demonstrated a certain degree of generalization and noise robustness in public benchmarks, the noise types and reverberation conditions in our experimental environment still exhibit non-negligible domain shifts from the pretraining corpus. If the model parameters are directly frozen, the output speaker features cannot be perfectly aligned with the true identity distribution under the current noisy conditions. To address this, we cascade a lightweight trainable adaptation module, consisting of a single linear layer, after the CAM++ output, and impose a discriminative constraint on the mapped embedding space via the InfoNCE loss. This constraint forces the projected vectors of utterances from the same speaker to cluster tightly while pushing those from different speakers apart (as illustrated in Figure 2). This design leverages the small parameter size and rapid convergence of the linear layer, helping the speaker features to better align with the current experimental environment. With only a minimal increase in computational overhead, this ensures that the speaker guidance can accurately inject the target speaker’s identity attributes into the flow-field prediction process of the DiT network.

Figure 2. Contrastive loss on speaker embeddings.

Building on the above motivations, we detail the implementation as follows: For the semantic branch, we use the hidden states from the last transformer layer of Qwen2-Audio-7B, which are 4096-dimensional. Since semantic content evolves over time, we preserve the temporal dimension and apply linear interpolation along the time axis to align the hidden states with the mel-spectrogram frame rate; the interpolated features are then projected from 4096 to 512 dimensions, yielding $E^{Se}\in {\mathbb{R}}^{B\times T\times 512}$. For the speaker branch, CAM++ produces a 192-dimensional utterance-level embedding, which is projected to 256 dimensions by a single trainable linear layer. Because speaker identity is stable within an utterance, we broadcast the projected embedding along the time axis to obtain $E^{Sp}\in {\mathbb{R}}^{B\times T\times 256}$. Finally, $M^T$, $M^N$, $E^{Se}$, and $E^{Sp}$ are concatenated along the channel dimension, resulting in a tensor of size $B\times T\times (2F+512+256)$, which is projected to the DiT hidden dimension before entering the backbone.

3.3. Loss Function

The optimization objective of our method consists of two components: the primary flow matching loss and the auxiliary contrastive learning loss. The total loss is defined as

$$\mathcal{L}=\lambda {\mathcal{L}}_{\mathrm{flow}}+(1-\lambda ){\mathcal{L}}_{\mathrm{InfoNCE}},$$

(3)

where $\lambda$ is set to 0.9 to balance the optimization weights between the primary and auxiliary tasks.

The flow matching loss ${\mathcal{L}}_{\mathrm{flow}}$ serves as the main optimization objective for the generative backbone. We adopt a conditional straight-line probability path. Let $x_0\sim p_0=\mathcal{N}(0,\mathbf{I})$ denote a Gaussian noise sample and $x_1\sim p_1=p_{\mathrm{data}}$ denote a clean mel-spectrogram sample. Let $\mathbf{c}=(M^N,E^{Se},E^{Sp})$ denote the conditioning variable, where $M^N$ is the distorted-speech mel-spectrogram, and $E^{Se}$ and $E^{Sp}$ are the semantic and speaker embeddings, respectively. For $t\in [0,1]$, the intermediate state $x_t$ is defined by linear interpolation between $x_0$ and $x_1$, as defined in Equation (1). The target velocity field along this path is

$$u_t(x_t\mid \mathbf{c})={\displaystyle \frac{d{x}_t}{dt}}=x_1-x_0.$$

(4)

The DiT network predicts the velocity field $v_t(x_t,\mathbf{c})$ to approximate $u_t(x_t\mid \mathbf{c})$. The conditional flow matching objective is then defined as

$${\mathcal{L}}_{\mathrm{flow}}={\mathbb{E}}_{x_0\sim p_0,x_1\sim p_1}\left[{∥v_t(x_t,\mathbf{c})-(x_1-x_0)∥}_2^2\right].$$

(5)

During inference, we solve the ordinary differential equation

$${\displaystyle \frac{d{x}_t}{dt}}=v_t(x_t,\mathbf{c}),$$

(6)

starting from $x_0\sim \mathcal{N}(0,\mathbf{I})$ at $t=0$ and integrating to $t=1$ with the Euler method over 32 steps to obtain the clean estimate $x_1$, which is finally reconstructed into a waveform by the vocoder.

The contrastive loss ${\mathcal{L}}_{\mathrm{InfoNCE}}$ serves as a discriminative constraint for the speaker branch, applied to the feature space mapped by the lightweight adapter after the CAM++ output. Given an anchor sample ${\mathbf{z}}_a$, its positive sample set is denoted as ${\mathbf{z}}_p\in \mathcal{P}$ (different distorted speaker embeddings synthesized from the same utterance), and its negative sample set is denoted as ${\mathbf{z}}_n\in \mathcal{N}$ (random clean speaker embeddings from different speakers). Let $\mathrm{sim}(·,·)$ denote the cosine similarity function and $\tau$ the temperature coefficient. The InfoNCE loss is defined as

$${\mathcal{L}}_{\mathrm{InfoNCE}}=-\mathbb{E}\left[\log {\displaystyle \frac{{\sum }_{{\mathbf{z}}_p\in \mathcal{P}}\exp \left(\mathrm{sim}({\mathbf{z}}_a,{\mathbf{z}}_p)/\tau \right)}{{\sum }_{{\mathbf{z}}_p\in \mathcal{P}}\exp \left(\mathrm{sim}({\mathbf{z}}_a,{\mathbf{z}}_p)/\tau \right)+{\sum }_{{\mathbf{z}}_n\in \mathcal{N}}\exp \left(\mathrm{sim}({\mathbf{z}}_a,{\mathbf{z}}_n)/\tau \right)}}\right].$$

(7)

This loss maximizes the mutual information lower bound between the anchor and positive samples while minimizing the similarity with negative samples, forcing mapped embeddings of utterances from the same speaker to cluster tightly in the feature space and those from different speakers to be pushed apart. This provides a discriminative identity prior for the flow matching process, as illustrated in Figure 2.

In the total loss, the flow matching loss dominates, ensuring that generation quality remains the primary priority. The InfoNCE loss acts as an auxiliary constraint, injecting explicit timbre consistency guidance into the model without interfering with the optimization of the primary task. Through joint optimization of the two losses, the model not only pursues high-fidelity reconstruction but also effectively preserves the target speaker’s identity characteristics, jointly suppressing hallucinations from both the generation path and the training objective.

4. Experimental Setup and Results

4.1. Datasets

To construct a diverse training dataset, we integrate multiple public data sources. For speech data, we adopt the VoiceBank corpus and the DNS Challenge 2021 dataset, utilizing both the English and Chinese portions for multilingual training. To simulate complex noise environments, we incorporate environmental noise from the DEMAND dataset and background sounds from the WHAM! dataset, enhancing model robustness under noisy conditions. In addition, we employ open-source room impulse response (RIR) data (OpenSLR26 and OpenSLR28) to generate reverberant speech samples.

In our experiments, we adopt two data construction strategies: (1) directly using the provided pairs of distorted and clean speech in the datasets, and (2) programmatically adding noise and reverberation to clean speech by controlling the signal-to-noise ratio (SNR) and reverberation time to generate distorted samples. Combining these two strategies effectively improves the diversity of the training set.

For evaluation, we use the English and Chinese subsets of the VoiceBank and DNS Challenge 2021 test sets. Since dedicated Chinese and long-utterance (duration > 30 s) test sets are scarce, we construct an additional held-out test set (hereafter, the supplementary test set). To evaluate generalization to unseen speakers and utterances, we perform the data split at the source-data level before synthesis, ensuring that no speaker or utterance in the supplementary test set appears in the training set. The supplementary test set is synthesized using the same noise and reverberation pipeline as the training set; therefore, the training and test data share the same noise and RIR distributions. This supplementary test set accounts for approximately 50% of the total test set and is never used for training or model selection. Table 2 summarizes the dataset configurations.

Table 2. Summary of dataset configurations. Ratio denotes the proportion of each subset within the total training or test duration.

Purpose	Dataset	Noise Source	Duration (Hours)	SNR (dB)	Ratio
Training	VoiceBank + DEMAND Training set	Provided by the official source	23.03	0 to 15	3%
Training	DNS Challenge 2021 Training set	DEMAND + WHAM! + RIRs	725.13	−5 to 20	97%
Test	VoiceBank + DEMAND Test set	Provided by the official source	0.41	2.5 to 17.5	10%
Test	DNS Challenge 2021 Test set	Provided by the official source	1.5	−5 to 20	40%
Test	Supplementary	DEMAND + WHAM! + RIRs	1.93	−5 to 20	50%

4.2. Evaluation Metrics

We employ a comprehensive set of metrics to evaluate the proposed model, covering perceptual quality, semantic similarity, signal-level fidelity, and generation efficiency.

DNSMOS (Deep Noise Suppression Mean Opinion Score) is a non-intrusive metric proposed by Microsoft for the DNS Challenge. It predicts the mean opinion score without requiring a clean reference, outputting three scores: SIG (signal quality), BAK (background noise suppression), and OVRL (overall quality). Each score ranges from 1 to 5, with higher values indicating better performance.

PESQ (Perceptual Evaluation of Speech Quality) is a standardized intrusive metric established by ITU-T (P.862). It compares the enhanced speech against a clean reference to predict the subjective MOS, with scores ranging from −0.5 to 4.5, where higher scores indicate better perceptual quality.

STOI (Short-Time Objective Intelligibility) is an objective intelligibility measure ranging from 0 to 1, with higher values indicating better intelligibility. It is derived by computing the correlation between the enhanced and clean speech in the short-time Fourier transform domain.

SECS (Speaker Embedding Cosine Similarity) serves as a direct metric for acoustic hallucination, defined as speaker-identity or timbre drift in the enhanced speech relative to the clean reference. It computes the cosine similarity between speaker embeddings extracted from the enhanced and clean speech using a pretrained ECAPA-TDNN model. The score ranges from $-1$ to 1, with higher values indicating less acoustic hallucination and better speaker preservation.

WER (Word Error Rate) is a metric for evaluating the preservation of semantic information. It employs the Whisper large-v2 ASR model to compare the transcriptions of enhanced speech against those of the clean reference. Lower WER values indicate better semantic retention. We further decompose WER into substitution (S), deletion (D), and insertion (I) rates. Since linguistic hallucination can manifest as both content insertion and content substitution, we use the combined S+I rate as a proxy for linguistic hallucination event frequency, denoted as the Hallucination Event Rate (HER). HER is an ASR-output-based operational proxy for hallucination, not a human-annotated hallucination rate; its results should therefore be interpreted as relative indicators of hallucination-like errors rather than actual hallucination rates.

Since WER additionally includes deletions, which are omissions rather than hallucinations, WER serves as a broader content-fidelity metric, while HER provides a hallucination-focused refinement of it. We therefore report WER in the main comparison and ablation study, and report HER only on the supplementary synthetic test set, which provides clean references and controllable SNRs. The official VoiceBank + DEMAND test set has a fixed SNR range, and the DNS Challenge 2021 test set is reference-free; therefore, HER is not reported on these datasets. For the supplementary test set, we report HER at SNRs of $-5$, 0, and ≥5 dB to provide an SNR-stratified analysis of hallucination behavior. This stratification is motivated by the observation that generative models are more prone to hallucination under low-SNR conditions, where degraded inputs provide weaker semantic cues; thus, the $-5$ dB condition is particularly diagnostic, while 0 and ≥5 dB serve as intermediate and relatively clean references.

We quantify computational efficiency using four metrics: FLOPs, total parameters (Params), peak GPU memory (GPU Mem), and real-time factor (RTF). FLOPs are measured for one complete end-to-end inference pass under identical input conditions for all compared methods. Params denote the total number of model parameters. GPU memory is the peak inference memory measured on an NVIDIA A100 GPU under BF16 precision. RTF is defined as inference time divided by audio duration; lower values indicate better efficiency, and RTF < 1 means faster than real time.

4.3. Experimental Configuration

We use 100-dimensional log mel-filterbank features extracted from 16 kHz audio as the input representation, with a window length of 1024, an FFT size of 1024, and a frame shift of 256. Let B and T denote the batch size and the number of mel frames, and let $F=100$ denote the mel dimension. The notation follows Section 3: $M^T$ and $M^N$ denote the intermediate state and the noisy mel-spectrogram, $E^{Se}$ and $E^{Sp}$ denote the semantic and speaker embeddings, and $x_1$ and $x_0$ denote the clean target and the Gaussian noise, respectively.

Semantic branch. We adopt Qwen2-Audio-7B as the semantic feature extractor, which is frozen during training. The hidden states from its last transformer layer have a dimension of 4096. Since the semantic content evolves over time, we preserve the temporal dimension and apply linear interpolation along the time axis to align the hidden states with the mel frame rate. The interpolated features are then projected from 4096 to 512 dimensions via a fully connected layer, yielding $E^{Se}\in {\mathbb{R}}^{B\times T\times 512}$.

Speaker branch. We employ the CAM++ model to extract speaker embeddings, which is also kept frozen. CAM++ produces a 192-dimensional utterance-level embedding for each utterance. A lightweight adaptation module consisting of a single linear layer projects it to 256 dimensions. Because speaker identity is stable within an utterance, the projected embedding is broadcast along the time axis to obtain $E^{Sp}\in {\mathbb{R}}^{B\times T\times 256}$. The adaptation module is trainable and is optimized by an InfoNCE loss with temperature $\tau =0.1$. For each anchor (the projected embedding of a noisy training utterance), we construct the positive set from projected embeddings of different distorted versions of the same utterance, and the negative set from projected embeddings randomly drawn from clean utterances of different speakers.

Conditioning and fusion. The four features $M^T\in {\mathbb{R}}^{B\times T\times F}$, $M^N\in {\mathbb{R}}^{B\times T\times F}$, $E^{Se}$, and $E^{Sp}$ are concatenated along the channel dimension, yielding a tensor of size $B\times T\times (2F+512+256)$. A linear layer projects it to the DiT hidden dimension, followed by a convolutional positional embedding with a residual connection.

DiT backbone. The above features are fed into the DiT network to estimate the velocity field in the flow matching model. The network comprises 16 encoder layers, each containing 16 attention heads, with a hidden dimension of 1024 and a feed-forward network dimension of 2048. A dropout rate of 0.1 is applied to both the attention layers and the feed-forward layers.

Optimization. We train all models with AdamW using a peak learning rate of $7.5\times {10}^{-5}$, a linear warm-up over the first 40,000 steps, and a subsequent linear decay. Training is conducted for 20 epochs with a batch size of 2048 on a single NVIDIA A100 GPU. To ensure statistical reliability, each model is trained with 5 different random seeds, and all baselines are trained independently under the same seed set. The test set is fixed across all runs, and we report all main results as mean ± standard deviation over the 5 runs. This protocol ensures a fair comparison across all evaluated methods.

Inference. During inference, we adopt the Euler method as the ordinary differential equation solver, performing 32 integration steps to sample from the noise distribution to the clean data distribution.

The overall training and inference procedure of SS-FMSE is summarized in Algorithm 1.

Algorithm 1: Training and inference of SS-FMSE

4.4. Baseline Methods

To facilitate the presentation of the experimental results, we briefly introduce several mainstream speech enhancement models included in our comparison.

MP-SEnet [14] is a time-frequency domain single-channel speech enhancement model whose core innovation lies in the parallel explicit estimation of magnitude and phase spectra. The model adopts a Transformer-based encoder–decoder architecture, achieving state-of-the-art performance on various speech enhancement tasks.

G-MaP-SE [6] is a guided enhancement framework that constructs a speaker embedding prior from clean speech using a Gaussian mixture model (GMM), and refines distorted conditioning embeddings by matching them to this prior. The matched embeddings are then injected into the enhancement backbone via a lightweight gated fusion module without requiring enrollment audio at inference time.

AnyEnhance [15] is a general-purpose generative speech enhancement model based on masked generative models. It introduces a prompt guidance mechanism for in-context learning with reference timbre cues and a self-critique mechanism that iteratively refines the generated output, achieving excellent performance on both objective metrics and subjective listening tests.

FlowSE [4] is a generative speech enhancement method based on flow matching. By learning a continuous transformation between the distributions of distorted and clean speech, it achieves high-fidelity reconstruction while significantly reducing inference latency.

SenSE [5] is a two-stage universal generative speech enhancement framework that leverages a semantic-aware speech language model to extract semantic tokens from distorted speech, and then integrates these semantic priors into a flow-matching-based enhancement process via a semantic guidance mechanism to effectively mitigate semantic ambiguity.

These methods represent state-of-the-art approaches across different speech enhancement paradigms and provide comprehensive baselines for our study.

4.5. Experimental Results

4.5.1. Overall Comparison and Qualitative Analysis

To comprehensively evaluate the performance of SS-FMSE, we compare our method against several mainstream speech enhancement models on a unified test set. To ensure comparability with public benchmarks, we conduct experiments separately on the public test set and the synthetic test set, as shown in Table 3 and Table 4. For fair comparison, all baseline methods are retrained using their official code, with the same training data as our method, and evaluated under consistent input conditions and experimental settings. The evaluation metrics include the no-reference speech quality metric DNSMOS (SIG, BAK, OVRL), the reference-based quality metrics STOI and PESQ, the downstream word error rate (WER), and the speaker embedding cosine similarity (SECS). Since the DNS Challenge 2021 test set provides only distorted speech without clean references, reference-based metrics cannot be computed; therefore, we report only no-reference metrics for this test set. Unless otherwise stated, all main results are reported as mean ± standard deviation across 5 random seeds.

Table 3. Performance comparison of different models on the public test sets.

Model/Method	DNS Challenge 2021			VoiceBank + DEMAND
	SIG	BAK	OVRL	SIG	BAK	OVRL	STOI	PESQ	WER	SECS
Noisy	2.45	2.235	1.968	3.244	3.125	2.695	0.687	1.877	0.095	0.972
MP-SEnet [14]	3.443±0.018	3.923±0.016	2.969±0.021	3.343±0.019	3.832±0.017	3.179±0.023	0.824±0.007	2.581±0.031	0.074±0.003	0.915±0.009
G-MaP-SE [6]	3.424±0.016	3.979±0.014	3.028±0.019	3.419±0.017	4.042±0.015	3.228±0.020	0.795±0.006	2.329±0.028	0.072±0.003	0.949±0.008
AnyEnhance [15]	3.542±0.015	3.934±0.017	3.121±0.018	3.426±0.016	4.114±0.014	3.261±0.019	0.768±0.008	1.997±0.026	0.062±0.003	0.931±0.009
FlowSE [4]	3.486±0.014	4.028±0.013	3.212±0.016	3.401±0.017	4.132±0.013	3.229±0.020	0.744±0.007	2.033±0.025	0.072±0.003	0.925±0.008
SenSE [5]	3.523±0.012	4.086±0.011	3.283±0.014	3.496±0.014	4.144±0.011	3.312±0.014	0.743±0.006	2.112±0.019	0.054±0.002	0.933±0.007
SS-FMSE	3.594±0.008	4.118±0.007	3.317±0.010	3.534±0.009	4.185±0.007	3.345±0.011	0.763±0.004	2.144±0.015	0.049±0.001	0.962±0.005

Note: Bold values indicate the best result in each column.

Table 4. Comparison of different models on the synthetic test set.

Model/Method	SIG	BAK	OVRL	STOI	PESQ	WER	SECS
Noisy	2.231	1.868	1.787	0.637	1.432	0.426	0.921
MP-SEnet [14]	3.341±0.018	3.437±0.016	2.743±0.021	0.714±0.007	2.182±0.031	0.291±0.008	0.789±0.009
G-MaP-SE [6]	3.384±0.016	3.646±0.014	2.943±0.019	0.718±0.006	2.192±0.028	0.251±0.007	0.819±0.008
AnyEnhance [15]	3.327±0.015	3.843±0.017	3.081±0.018	0.688±0.008	1.732±0.026	0.263±0.006	0.811±0.009
FlowSE [4]	3.356±0.014	3.834±0.013	3.062±0.016	0.637±0.007	1.643±0.025	0.241±0.008	0.803±0.008
SenSE [5]	3.441±0.012	3.932±0.011	3.143±0.014	0.702±0.006	1.882±0.022	0.213±0.005	0.814±0.007
SS-FMSE	3.483±0.008	4.095±0.007	3.221±0.010	0.703±0.004	1.963±0.016	0.174±0.003	0.868±0.005

Note: Bold values indicate the best result in each column.

Among the baseline methods, MP-SEnet [14], with its PESQ-optimized loss function and explicit phase modeling, achieves the best performance on the objective metrics PESQ and STOI. Nevertheless, its improvement on the BAK (background noise suppression) dimension is relatively limited, indicating insufficient capability in suppressing background noise. G-MaP-SE [6], built upon MP-SEnet, introduces a GMM-based speaker embedding prior matching mechanism. By refining distorted speaker embeddings towards a clean prior, it achieves consistent improvements over MP-SEnet across most metrics, particularly in BAK and OVRL, while also obtaining a higher SECS, demonstrating the effectiveness of speaker identity guidance in enhancing perceptual quality and preserving speaker consistency.

In contrast, generative methods generally suffer from phase mismatch issues, which degrade the alignment between enhanced outputs and reference signals, resulting in a notable gap from MP-SEnet on PESQ and STOI. However, the DNSMOS metrics (SIG, BAK, OVRL) do not strictly depend on phase alignment and better reflect human auditory perception. On these three dimensions, generative methods consistently outperform MP-SEnet. AnyEnhance [15], through its prompt learning and self-critique mechanisms, achieves excellent performance on OVRL and WER. FlowSE [4] adopts flow matching framework as an alternative to iterative diffusion sampling, striking a balance between inference efficiency and generation fidelity. Its performance on most metrics is comparable to that of AnyEnhance, validating the advantages of flow matching in perceptual quality. SenSE [5], building upon FlowSE, further incorporates semantic tokens extracted by a semantic-aware speech language model as guidance. This semantic-aware design effectively mitigates semantic ambiguity and acoustic discontinuities, achieving improvements over FlowSE on both DNSMOS and WER.

Compared with the above methods, our proposed SS-FMSE achieves the best results on SIG, BAK, OVRL, WER, and SECS, validating the effectiveness of the dual guidance mechanism—semantic understanding and speaker perception—in improving perceptual quality and content intelligibility. The SECS results further confirm that the speaker branch effectively preserves the target timbre, complementing the semantic branch. Notably, SS-FMSE outperforms FlowSE and SenSE on all metrics, demonstrating that the joint guidance from semantic embeddings and speaker identity information mitigates generative hallucinations, significantly enhancing both semantic fidelity and timbre consistency while maintaining generation quality.

Figure 3 presents example 100-dimensional mel-spectrograms of different models on the synthetic test set. Panel (a) shows the clean spectrogram, panel (b) shows the noisy spectrogram, and panels (c)–(f) display the outputs of different baseline models and our SS-FMSE. Although MP-SEnet achieves relatively good performance on STOI and PESQ, its spectrograms still contain considerable residual noise, resulting in less satisfactory subjective listening quality compared to other methods. AnyEnhance removes noise effectively, yet suffers from energy loss in certain frequency bands. This is evident in panel (d), where the region marked by the blue box corresponds to the same time-frequency region marked by the blue box in the clean reference panel (a). FlowSE also recovers the audio well, but introduces spurious energy in some frequency regions. This can be seen in panel (e), where the region marked by the green box corresponds to the same time-frequency region marked by the green box in the clean reference panel (a). In contrast, the spectrograms reconstructed by our proposed SS-FMSE most closely resemble the clean reference: background noise is effectively suppressed, the harmonic structure across the full frequency band is restored with greater completeness and continuity, and no obvious spectral artifacts or spurious harmonics are observed.

Figure 3. Comparison of mel-spectrograms reconstructed by different generative models on the synthetic test set. (a) Clean, (b) Distorted, (c) MP-SEnet, (d) AnyEnhance, (e) FlowSE, and (f) SS-FMSE (Proposed). Colored boxes mark the same time-frequency regions across panels; no magnification is applied.

4.5.2. SNR-Stratified Hallucination Analysis

To examine how generative hallucination behaves under different noise conditions, we stratify the supplementary synthetic test set into three SNR groups: $-5$, 0, and ≥5 dB. This stratification is motivated by the fact that lower SNRs provide weaker semantic cues and thus make generative models more prone to hallucination, manifesting as content insertion or substitution. As shown in Table 5, all models improve as SNR increases, but the gap between SS-FMSE and the baselines is most pronounced at the lowest SNR. In the $-5$ dB condition, SS-FMSE achieves the lowest HER and WER, reducing the HER to 16.2%. This suggests that dual guidance is particularly beneficial for reducing hallucination-related errors under severe degradation. At higher SNRs, the performance gap decreases, yet SS-FMSE still achieves the best results.

Table 5. SNR-stratified performance comparison on the supplementary synthetic test set.

Model/Method	$-5$ dB			0 dB			≥5 dB
	SECS	WER	HER	SECS	WER	HER	SECS	WER	HER
Noisy	0.823	0.566	0.368	0.908	0.352	0.231	0.917	0.145	0.085
MP-SEnet [14]	0.701±0.012	0.352±0.012	0.254±0.011	0.792±0.011	0.268±0.011	0.192±0.010	0.831±0.010	0.113±0.010	0.076±0.009
G-MaP-SE [6]	0.744±0.010	0.315±0.010	0.231±0.009	0.835±0.009	0.224±0.009	0.164±0.008	0.874±0.009	0.089±0.008	0.062±0.007
AnyEnhance [15]	0.726±0.011	0.328±0.011	0.242±0.010	0.823±0.010	0.235±0.010	0.172±0.009	0.871±0.009	0.095±0.009	0.066±0.008
FlowSE [4]	0.712±0.009	0.302±0.009	0.225±0.008	0.801±0.008	0.216±0.008	0.159±0.007	0.872±0.008	0.084±0.007	0.059±0.006
SenSE [5]	0.741±0.008	0.276±0.008	0.208±0.007	0.832±0.007	0.189±0.007	0.141±0.006	0.886±0.007	0.072±0.006	0.051±0.005
SS-FMSE	0.794±0.005	0.238±0.005	0.162±0.004	0.881±0.004	0.152±0.004	0.104±0.003	0.909±0.004	0.058±0.003	0.038±0.002

Note: Bold values indicate the best result in each column.

In terms of SECS, SS-FMSE consistently outperforms all enhancement baselines across the three SNR groups, confirming that the speaker branch suppresses acoustic hallucination, i.e., timbre drift, even under strong noise. The SECS gap is also largest at $-5$ dB, indicating that speaker guidance is particularly effective when the input is severely degraded. Note that the distorted input itself yields a high SECS, since the degradation process mainly affects the acoustic environment rather than the speaker identity; SECS therefore specifically reflects whether the enhancement process introduces speaker-level distortion, i.e., acoustic hallucination.

Overall, the SNR-stratified results show that SS-FMSE maintains consistent improvements across noise levels and specifically reduces hallucination-related errors in low-SNR scenarios.

4.5.3. Ablation Study

To investigate the impact of each module on various speech quality evaluation metrics and speaker consistency, we conduct ablation experiments on the synthetic test set. As shown in Table 6, Method A (flow matching only) performs the weakest across all metrics, with a particularly high WER of 24.1%, indicating that generative models without high-level guidance suffer from significant deficiencies in content intelligibility.

Table 6. Ablation study results. “Full Speaker Branch” denotes the complete speaker branch with adapter and InfoNCE, and “Full Model” denotes the full dual-guidance model combining the semantic branch and the complete speaker branch.

Model/Method	Embedding Info	InfoNCE	SIG	BAK	OVRL	STOI	PESQ	WER	SECS
A. Flow Matching	/	/	3.356±0.014	3.834±0.013	3.062±0.016	0.637±0.007	1.643±0.025	0.241±0.008	0.803±0.008
B1. + Speaker Branch (only Adapter)	Speaker	/	3.372±0.018	3.856±0.016	3.094±0.021	0.652±0.007	1.663±0.028	0.233±0.005	0.829±0.009
B2. + Full Speaker Branch	Speaker	✓	3.441±0.015	3.937±0.013	3.192±0.018	0.694±0.006	1.742±0.025	0.221±0.004	0.849±0.008
C. + Semantic Branch	Semantic	/	3.397±0.017	3.943±0.015	3.143±0.020	0.696±0.006	1.835±0.027	0.185±0.004	0.813±0.009
D. Full Model	Semantic, Speaker	✓	3.483±0.009	4.095±0.007	3.221±0.011	0.703±0.004	1.963±0.016	0.174±0.002	0.868±0.005

Note: Bold values indicate the best result in each column.

For the speaker branch, we compare two configurations: B1 adds a speaker embedding with a lightweight adapter but without InfoNCE, while B2 further applies the InfoNCE regularization to form the full speaker branch. As shown in Table 6, B1 improves over A on all DNSMOS dimensions and SECS, suggesting that speaker identity constraints guide the model toward more consistent target timbre. However, its WER reduction is limited (from 24.1% to 23.3%), implying that timbre constraints alone are insufficient to fully address linguistic hallucinations. In contrast, B2 achieves consistent improvements over B1 across all metrics, especially on SECS (from 0.829 to 0.849) and WER (from 23.3% to 22.1%). This confirms that InfoNCE enhances intra-class compactness and inter-class separability of speaker characteristics and contributes independently to hallucination suppression. Note that the adapter is required for dimension alignment before feature concatenation and is therefore included in both B1 and B2; the comparison between them isolates the effect of InfoNCE.

Method C introduces semantic embeddings and substantially reduces WER to 18.5%, confirming the core role of the semantic branch in enhancing content intelligibility. Steady improvements are also observed across all DNSMOS dimensions. Nevertheless, its SECS is only 0.813, lower than that of B2, indicating that semantic priors alone cannot maintain timbre consistency. Method D, the full model with dual guidance, achieves the best results across all metrics, attaining the highest PESQ and STOI scores, leading all methods on the three DNSMOS metrics, and further reducing WER to 17.4% while achieving the highest SECS of 0.868. These results demonstrate the synergistic effect of dual semantic and speaker guidance: the semantic branch ensures the plausibility of generated content, while the complete speaker branch with InfoNCE maintains timbre consistency. The two branches complement each other, jointly suppressing generative hallucinations from both content and identity perspectives.

4.5.4. Sensitivity Analysis of InfoNCE Loss Weight

The InfoNCE loss weight ${\lambda }_{\mathrm{InfoNCE}}$ controls the strength of the contrastive regularization on the speaker branch. To examine its impact, we vary ${\lambda }_{\mathrm{InfoNCE}}$ over $\{0.01,0.05,0.1,0.2,0.5,0.9\}$ while keeping all other settings identical to the full model. We report SECS and WER to measure speaker consistency and semantic fidelity, respectively, and OVRL as an overall perceptual quality indicator. The results are shown in Table 7.

Table 7. Sensitivity analysis of the InfoNCE loss weight ${\lambda }_{\mathrm{InfoNCE}}$.

${\mathit{\lambda }}_{\mathbf{InfoNCE}}$	0.01	0.05	0.1	0.2	0.5	0.9
OVRL	3.135±0.021	3.172±0.018	3.221±0.010	3.122±0.015	3.079±0.023	2.934±0.028
WER	0.189±0.006	0.182±0.005	0.174±0.003	0.189±0.004	0.295±0.009	0.418±0.013
SECS	0.816±0.012	0.849±0.009	0.868±0.005	0.876±0.004	0.861±0.006	0.826±0.011

Note: Bold values indicate the best result in each row.

A very small weight (e.g., $0.01$) barely activates the contrastive regularization, yielding the lowest SECS and relatively high WER. As ${\lambda }_{\mathrm{InfoNCE}}$ increases to $0.1$, SECS improves steadily and both WER and OVRL reach their best values, indicating that a moderate contrastive strength effectively compacts the speaker embedding space and improves overall training stability, which also benefits content preservation. Interestingly, SECS reaches its maximum at ${\lambda }_{\mathrm{InfoNCE}}=0.2$, slightly higher than at $0.1$, but this comes at the cost of a higher WER and a lower OVRL. This suggests a mild trade-off between speaker consistency and semantic fidelity: once the contrastive objective becomes dominant, the model tends to over-emphasize timbre consistency at the expense of content preservation.

When ${\lambda }_{\mathrm{InfoNCE}}$ is further increased to $0.5$ and $0.9$, the performance degrades sharply, with WER rising to 0.418 and OVRL dropping to 2.934 at $\lambda =0.9$. This suggests that the speaker branch should serve as an auxiliary constraint rather than dominate the training objective, since an overly strong contrastive signal interferes with the optimization of the primary denoising objective, ultimately degrading generation fidelity. We therefore adopt ${\lambda }_{\mathrm{InfoNCE}}=0.1$ in all experiments, as it achieves the best balance among speaker consistency, semantic fidelity, and overall perceptual quality.

4.5.5. Computational Efficiency and Quality Trade-Off

Table 8 reports the quality and computational efficiency of SS-FMSE and the baselines. All efficiency metrics are measured end-to-end on the complete model under identical input and hardware conditions. SS-FMSE provides a controllable quality–efficiency trade-off through the number of ODE steps: more steps consistently improve enhancement quality, while FLOPs and RTF also increase. The total parameter count and peak GPU memory remain unchanged across step counts. The inference cost consists of a fixed part from the frozen condition encoders, i.e., the Qwen2-Audio-7B semantic encoder and the CAM++ speaker encoder together with their projection layers, which run once per utterance and are independent of the ODE step count, and a variable part from repeated DiT evaluations. The total parameter count of about 7.4B is dominated by the frozen Qwen2-Audio-7B encoder, while the trainable parameters (DiT backbone, projection layers, and speaker adapter) amount to approximately 400 M. As the ODE steps increase, RTF rises from 0.54 to 1.17. Compared with FlowSE, SS-FMSE achieves better quality at sufficient steps at the cost of a larger model. Compared with SenSE, SS-FMSE achieves better quality at 32 steps with a slightly higher RTF, but at 16 steps it already attains comparable quality while being faster, and the additional overhead mainly stems from the frozen encoder that does not scale with the number of ODE steps. Overall, SS-FMSE is not limited to a single operating point and can be adapted to different latency and quality requirements.

Table 8. Quality and computational efficiency comparison.

Model/Method	OVRL	WER	SECS	FLOPs (G)	Params (M)	GPU Mem (GB)	RTF
MP-SEnet [14]	2.743±0.021	0.291±0.008	0.749±0.009	49.98	4.26	6.3	0.14
G-MaP-SE [6]	2.943±0.021	0.251±0.007	0.809±0.008	59.43	5.02	6.6	0.16
AnyEnhance [15]	3.081±0.021	0.263±0.006	0.811±0.009	328.29	122.65	10.4	0.27
FlowSE [4]	3.062±0.016	0.241±0.008	0.803±0.008	384.91	334.44	12.1	0.51
SenSE [5]	3.143±0.014	0.213±0.005	0.814±0.007	814.92	1763.37	18.4	1.03
SS-FMSE (ODE = 4)	2.847±0.013	0.252±0.003	0.811±0.007	1078.34	7423.12	25.5	0.54
SS-FMSE (ODE = 8)	3.043±0.012	0.221±0.002	0.835±0.004	1122.47	7423.12	25.5	0.64
SS-FMSE (ODE = 16)	3.113±0.013	0.209±0.003	0.849±0.003	1241.23	7423.12	25.5	0.82
SS-FMSE (ODE = 32)	3.221±0.010	0.174±0.003	0.868±0.005	1480.31	7423.12	25.5	1.17

Note: Bold values indicate the best result in each column.

5. Conclusions

To address the problem of generative hallucinations in speech enhancement caused by the lack of high-level priors, this paper proposes SS-FMSE, a multi-task flow matching speech enhancement method with dual guidance from semantic understanding and speaker perception. The method adopts flow matching as the generative backbone and introduces two auxiliary guidance branches—semantic and speaker—on top of acoustic features.

Experimental results demonstrate that SS-FMSE achieves the highest DNSMOS and SECS among all enhancement methods and the lowest WER on both test sets. Within the generative category, it also achieves the best PESQ and the best or comparable STOI. The SNR-stratified analysis using the HER proxy further shows that, at $-5$ dB, SS-FMSE achieves both the lowest HER (16.2%) and the highest SECS among all enhancement methods. This suggests that SS-FMSE reduces linguistic and acoustic hallucination-related errors under severe noise.

Although SS-FMSE achieves a favorable balance between generation quality and identity fidelity, there remains room for further improvement in computational efficiency, guidance richness, acoustic generalization, and data efficiency. In particular, although separation at both the speaker and utterance levels was enforced between the training and test sets, the noise recordings and RIRs were not split disjointly, so generalization to unseen acoustic conditions has not been evaluated. Future work will address this limitation via disjoint noise/RIR splits and cross-dataset evaluation. In addition, more lightweight semantic extraction schemes, such as distillation or lightweight audio encoders, could be explored to reduce computational overhead; richer guidance dimensions, such as emotion and prosody, could be incorporated to improve controllability and expressiveness; and self-supervised or unsupervised adaptation could be integrated to reduce the reliance on paired training data.