Enhanced Conformer-Based Speech Recognition via Model Fusion and Adaptive Decoding with Dynamic Rescoring

Abstract

Speech recognition is widely applied in fields like security, education, and healthcare. While its development drives global information infrastructure and AI strategies, current models still face challenges such as overfitting, local optima, and inefficiencies in decoding accuracy and computational cost. These issues cause instability and long response times, hindering AI’s competitiveness. Therefore, addressing these technical bottlenecks is critical for advancing national scientific progress and global information infrastructure. In this paper, we propose improvements to the model structure fusion and decoding algorithms. First, based on the Conformer network and its variants, we introduce a weighted fusion method using training loss as an indicator, adjusting the weights, thresholds, and other related parameters of the fused models to balance the contributions of different model structures, thereby creating a more robust and generalized model that alleviates overfitting and local optima. Second, for the decoding phase, we design a dynamic adaptive decoding method that combines traditional decoding algorithms such as connectionist temporal classification and attention-based models. This ensemble approach enables the system to adapt to different acoustic environments, improving its robustness and overall performance. Additionally, to further optimize the decoding process, we introduce a penalty function mechanism as a regularization technique to reduce the model’s dependence on a single decoding approach. The penalty function limits the weights of decoding strategies to prevent over-reliance on any single decoder, thus enhancing the model’s generalization. Finally, we validate our model on the Librispeech dataset, a large-scale English speech corpus containing approximately 1000 h of audio data. Experimental results demonstrate that the proposed method achieves word error rates (WERs) of 3.92% and 4.07% on the development and test sets, respectively, significantly improving over single-model and traditional decoding methods. Notably, the method reduces WER by approximately 0.4% on complex datasets compared to several advanced mainstream models, underscoring its superior robustness and adaptability in challenging acoustic environments. The effectiveness of the proposed method in addressing overfitting and improving accuracy and efficiency during the decoding phase was validated, highlighting its significance in advancing speech recognition technology.

1. Introduction

Speech recognition is a technology that converts human speech signals into text, enabling computers to understand and interpret human language. This technology plays a critical role in human–computer interaction and is widely applied in various fields such as security, education, healthcare, and the military. It significantly improves people’s quality of life and work efficiency, demonstrating tremendous application potential and market prospects across multiple industries. As this technology rapidly develops globally, the market size reached USD 11.4 billion in 2023 and is projected to grow to USD 44.7 billion by 2032 [1]. The advancement of speech recognition not only drives the development of intelligent devices but also has a substantial impact on global information technology infrastructure and artificial intelligence strategies.

Speech recognition technology has undergone three major stages of development. Initially, methods based on Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) were used [2], relying on manual feature extraction and statistical modeling for speech recognition. While these methods performed reasonably well in environments with low noise, they struggled with complex speech inputs. The introduction of deep learning later brought about the application of convolutional neural networks (CNNs) and deep neural networks (DNNs), significantly improving the accuracy of speech recognition, particularly in noisy environments [3]. In recent years, end-to-end architectures have become mainstream, with models such as Transformer. The self-attention mechanism in Transformer models addresses the limitations of traditional models in parallel processing and long-range dependencies, further enhancing recognition accuracy and training efficiency [4]. With these advancements, speech recognition systems have seen substantial improvements in accuracy, robustness, and adaptability to dynamic environments, particularly in handling multiple languages, dialects, and complex background conditions.

However, despite the significant progress made in speech recognition technology, several challenges and pain points remain. One notable issue is the lack of robustness. Speech recognition systems experience a significant drop in accuracy when faced with noisy environments and varying background noise [5]. This limitation hinders the performance of speech recognition technology in complex and diverse real-world applications. Another challenge lies in the shortcomings of current decoding algorithms. These algorithms have limitations in handling the complexity and diversity of natural language, which affects the accuracy of speech recognition systems and constrains their response speed and efficiency in real-time applications. This is particularly evident when dealing with continuous speech and complex commands, where the efficiency and accuracy of existing algorithms need improvement [6].

To address these issues, this paper proposes the following innovative methods: First, during the model training phase, to alleviate overfitting and local optima, we introduce an innovative network architecture and propose a loss-weighted model fusion approach. This method combines multiple model structures by adjusting their weights during the convergence phase, enhancing the model’s generalization ability and robustness. Second, to address the balance between accuracy and efficiency in decoding algorithms, we design a dynamic adaptive decoding algorithm. This algorithm integrates CTC and attention mechanisms using nonlinear functions, enabling more precise capture of the complex relationships and interactions in speech data. Furthermore, to improve decoding efficiency, we introduce a dynamically adjusted penalty mechanism that limits the model’s reliance on a single decoder, thereby enhancing the robustness of the decoding process and accelerating the decoding speed.

The structure of this paper is divided into six sections, arranged as follows: Section 2 presents the related work, reviewing the development history of speech recognition models, discussing challenges in model performance, decoding algorithms, and the scarcity of datasets, and outlining the solutions adopted in this paper. Section 3 introduces the methodology, detailing the design concepts behind model fusion and the innovations in the decoding algorithm. Section 4 describes the experimental setup, including the model parameter configuration, dataset sources, and performance evaluation metrics. Section 5 presents the experimental results, exploring the effects of model fusion and decoding algorithm optimization, conducting ablation experiments, and comparing the performance of the proposed method with mainstream network models. Section 6 concludes the paper based on the experimental results. Section 7 discusses future work, providing a critical examination of the limitations inherent in the proposed methods and suggesting strategies to address these challenges, while identifying avenues for further exploration.

2. Related Work

With the emergence and continuous development of deep learning neural networks, the mainstream methods used in speech recognition today predominantly adopt end-to-end model structures and leverage deep learning algorithms for training. In recent years, many scholars have focused their research on this popular area. However, there are still limitations and aspects for improvement in the model architecture, performance, generalization, and decoding search algorithms.

2.1. Research Process

In the early stages of speech recognition, traditional models typically involved a complex sequence of processes, including capturing speech signals, feature processing, training acoustic and language models, performing statistical calculations, and decoding the output [7]. This intricate structure, along with the cumbersome process and extensive data handling, significantly impacted the accuracy and real-time performance of speech recognition systems.

With the continuous development of machine learning, integrating deep learning with traditional Hidden Markov Models (HMMs) has become a popular trend. Seltzer et al. focused on the application of deep neural networks (DNNs) in noisy and robust environments, finding that DNNs outperformed traditional Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) systems in these settings [8]. Ghaffarzadegan et al. [9] improved the learning process of DNN-HMM by generating a large number of pseudo-samples to address the issue of insufficient data. Their experiments demonstrated that the DNN-HMMs trained using this method performed excellently in whisper speech recognition. Chauhan et al. [10] used Mel-Frequency Cepstral Coefficients (MFCC) as features combined with DNNs to design a speech recognition and separation model, which achieved promising results in multi-speaker speech recognition. Wang et al. [11] employed the DNN-HMM method, using the lower layers of the network to filter out noise and the upper layers to extract speech features, ultimately achieving good performance in smart home speech recognition. Vetráb et al. [12] utilized a hybrid HMM/DNN model for computational paralinguistic tasks, proposing a new embedding extraction method that performed exceptionally well in handling variable-length speech segments and small-scale corpora. While the integration of deep learning in the aforementioned methods has led to improvements in model performance and accuracy, these systems still retain the original architecture, which includes multiple complex components such as acoustic models, language models, and pronunciation dictionaries. As a result, the training process remains cumbersome, with significant computational resource consumption. This does not fundamentally resolve the issues of structural complexity and computational burden, which continue to affect the accuracy and efficiency of speech recognition.

With the emergence of end-to-end model architectures, speech recognition has advanced further. Unlike traditional structures, end-to-end methods directly convert raw audio into text output, with intermediate structures only designed as encoders and decoders. This approach eliminates several cumbersome processing steps, such as the construction of pronunciation dictionaries and the separation of acoustic and language models, thereby simplifying the model architecture. In recent years, end-to-end speech recognition, particularly research based on Transformer networks, has become a popular trend. Zhi et al. [13] employed the Espnet end-to-end architecture to recognize Mongolian speech and achieved satisfactory results. Liu et al. [14] explored the enhancement of emotion recognition performance in their study on pre-trained models, utilizing Wav2Vec 2.0 as the foundational framework. They investigated the optimization of low-level speech features and introduced improvements to input feature distribution and model architecture. The proposed approach demonstrated significant performance gains, particularly on small-scale or low-quality datasets. Karita et al. [15] conducted a comparative study on the performance of Transformer and traditional RNN models in speech applications. Their research demonstrated that Transformers outperformed RNNs in speech recognition tasks, especially when handling long sentences and noisy environments. Miao et al. [16] proposed an online CTC/attention-based end-to-end speech recognition architecture using Transformer, which performed exceptionally well in multiple benchmark tests, particularly with long sequence inputs. Tanaka et al. [17] introduced a sequence-to-sequence voice conversion model (ATTS2s-VC) based on attention mechanisms and context retention mechanisms. This model, utilizing the Transformer architecture, achieved high-quality voice conversion, particularly excelling in cross-linguistic speech conversion tasks. Wang et al. [18] proposed a method for dynamically inducing attention probability sparsity by incorporating a weak attention mechanism into a Transformer-based speech recognition model. This method improved the model’s recognition performance in noisy environments by suppressing irrelevant information. Gulati et al. introduced the Conformer model, a hybrid architecture that integrates convolutional neural networks (CNNs) and Transformers, aimed at effectively modeling both local and global dependencies in audio sequences [19]. Experimental results showed that it significantly outperformed traditional RNN and pure Transformer models on the LibriSpeech dataset, achieving state-of-the-art performance at the time. Liu et al. [20] proposed an improved Conformer-based model for high-precision speech feature extraction and learning. The experimental results indicated that the improved model significantly enhanced feature extraction and recognition accuracy.

Speech recognition technology has evolved from traditional acoustic statistical models to neural networks, and then to end-to-end deep learning models. While early systems improved accuracy, they were often complex in structure and computationally expensive. With the emergence of end-to-end architectures, speech recognition has become more streamlined, significantly enhancing both model accuracy and efficiency. Currently, end-to-end methods based on Transformer architectures have become mainstream and have achieved outstanding results across various speech recognition tasks.

2.2. Research Difficulties

End-to-end models have significantly improved the accuracy of speech recognition due to their simplified architecture and powerful deep learning algorithms. In recent years, much research has focused on this promising approach. However, there remain certain limitations and areas for improvement, particularly in optimizing model architecture performance and decoding search algorithms.

Zhong et al. [21], in the OpenASR21 Challenge, collected external text models to train a text-to-speech model that generates data matching speech-to-text. They trained and fused various models (CNN, DNN, ResNet, etc.), ultimately improving speech recognition performance under low-resource conditions. However, since they used a traditional hybrid DNN-HMM framework, the complex structure significantly reduced efficiency in both training and recognition compared to end-to-end models. Chen et al. [22] applied a DNN-HMM model architecture to recognize remote multi-microphone Mandarin audiovisual data. They conducted comprehensive ablation studies on all audio and video data in the purely audio, purely video, and audiovisual systems. The results revealed that the video modality supplemented the acoustics degraded by noise, reducing deletion errors and providing discriminative information in overlapping speech, thereby reducing substitution errors. While they explored end-to-end model structures toward the end of the paper, the results were not satisfactory, indicating the need for further research. Li et al. proposed a network structure that randomly perturbs acoustic features to obtain speaker representation vectors that are insensitive to speech order through joint learning. They first segmented audio features into various length scales, randomly perturbed the audio feature segments at each scale, and then concatenated them back into audio features consistent with the original audio duration. The original and perturbed audio features were then fed into two similar neural networks. Finally, they added a Kullback–Leibler (KL) divergence loss function to the original target loss function to drive model training [23]. This approach yielded good experimental results on the VoxCeleb corpus. However, using a residual neural network (ResNet), which is similar in structure to a convolutional neural network (CNN), presents limitations. Compared to end-to-end structures, ResNet is less effective in capturing long-range dependencies and global information than attention-based models. Nakatani [24] proposed an improved Transformer-based end-to-end speech recognition model by integrating connectionist temporal classification (CTC) and a language model (LM) to further enhance recognition accuracy. Although this approach effectively improved recognition accuracy, a single model structure still faces the challenges of overfitting and local optima. Additionally, while integrating CTC and LM increases model robustness, it also adds to the computational complexity and training time, affecting the model’s efficiency in real-time applications. Wu [25] introduced a deep sparse Conformer model to improve long-sequence representation capabilities. Experiments demonstrated that the deep sparse Conformer performed well in handling long audio sequences. Kim and Lee [26] proposed an RNN-Transducer model using sparse self-attention layers to process out-of-domain audio, with experiments showing significantly improved recognition performance across different audio domains. Despite the deep sparse Conformer’s superior performance in long-sequence processing, its implementation and training process are complex, requiring substantial computational resources. Additionally, its performance in short-sequence and real-time applications needs further exploration. Burchi and Timofte [27] developed a Conformer model for efficient audio and visual processing, enhancing noise robustness. Experiments showed that the model significantly improved speech recognition performance in noisy environments. However, while the model excels in noisy conditions, its reliance on multimodal data increases the complexity and data requirements for implementation. Further validation is needed to assess its performance in scenarios lacking visual information. Shi et al. [28] proposed a Conformer variant that combines convolutional neural networks and Transformers, demonstrating performance improvements on the AISHELL-1 and LibriSpeech datasets. The results indicated significant gains in recognition accuracy and training speed. However, the complex structure of this model increases training difficulty and hardware resource requirements. Moreover, while the multi-level design enhances the model’s expressive ability, it also introduces more hyperparameters, complicating the tuning process. Wang et al. [29] developed Conformer-LHUC, a Conformer model combined with Learning Hidden Unit Contributions (LHUCs) for speaker adaptation. Experiments demonstrated that Conformer-LHUC performed well in elderly speech recognition and showed potential in the clinical diagnosis and treatment of Alzheimer’s disease. Despite its strong performance in specific applications, the adaptability and generality of Conformer-LHUC in other scenarios have not yet been fully validated. Moreover, the inclusion of the LHUC mechanism increases model complexity and computational cost, which may limit its use in resource-constrained environments. Liu et al. [30] proposed a Mandarin audio-visual speech recognition model based on adaptive decision fusion and tone features, which combines audio and visual information to improve recognition performance. Experimental results showed that the model significantly enhanced the system’s robustness, particularly in noisy environments, compared to single-modal models. Similarly, another study [31] focused on emotion recognition using speech and visual modalities, introducing a multi-learning framework that effectively fuses information from both modalities to achieve precise emotion modeling. Although the multimodal model excels in noisy settings, its reliance on multimodal data limits its applicability in scenarios where visual information is unavailable.

While end-to-end models, characterized by their simplified architecture and the application of advanced deep learning algorithms, have substantially enhanced the accuracy of speech recognition, several challenges persist. Specifically, due to their reliance on deep learning neural networks, these models are prone to issues such as overfitting and the tendency to converge to local optima during training. Additionally, the large-scale nature of end-to-end models often involves a considerable number of parameters and intricate nonlinear relationships, which complicate the optimization process. Consequently, achieving a global optimum in such models remains a significant challenge.

Furthermore, this challenge is not confined to speech recognition alone; researchers employing deep learning models across various domains face similar issues. Zhang et al. [32] highlighted that while the Transformer model can achieve excellent performance in speech recognition tasks due to its complex and deep structure, the computational cost associated with its numerous parameters significantly increases, leading to difficulties in model training and a tendency to fall into local optima. Dongkuk Si and Chulhee Yun [33] discovered that even advanced sharpness-aware minimization techniques fail to fully converge to the optimal solution, implying that even with sophisticated optimization strategies, avoiding local optima remains challenging. Ohad Shamir [34] proposed a method that frames the optimization of unknown linear functions as a zero-sum game between two players, utilizing no-regret learners to compute strategies for each round in an attempt to address this problem. However, this approach still faces high computational complexity in practical applications and is limited in effectiveness when handling large-scale data, failing to completely resolve the issue of local optima. Zheng [35], analyzing the deep learning optimization problem from a probabilistic perspective, pointed out that the size of mini-batches can influence the range of parameter exploration, thereby affecting the model’s ability to avoid local minima. Although these new perspectives and methods offer some advancements, their optimization effects in practical applications are still less than ideal, especially when dealing with high-dimensional data and complex models, where the issue of local optima persists.

This suggests that, while technological advancements have simplified and enhanced the architecture of speech recognition models, particularly through the use of end-to-end approaches that streamline the training and computation processes, these models’ inherent reliance on neural networks gives rise to certain challenges. Specifically, during training, issues such as overfitting and convergence to local optima can arise, leading to poor generalization to unseen data and limited robustness in real-world applications. These limitations, in turn, hinder the broader adoption and effectiveness of speech recognition technology.

In recent years, many scholars have focused more on improving model architectures or innovating within specific application domains. As a result, the decoding process often relies on existing algorithms with little modification, leading to relatively few studies on decoding algorithm innovations. However, decoding remains a crucial part of speech recognition, as it is primarily responsible for converting the output of the acoustic model into text.

Some researchers have recently dedicated efforts to designing new algorithms to enhance decoding speed or accuracy, thereby improving the search efficiency of decoders. Regarding speed improvements, the Viterbi algorithm is a dynamic programming algorithm that can perform space-synchronous operations and search space pruning [36]. It is widely used in speech recognition to find the optimal state sequence in Markov models. However, in the context of high-dimensional data and complex models—especially in modern speech recognition scenarios that demand intricate models and deep learning techniques—the Viterbi algorithm can lead to high computational complexity, making it difficult to adapt to current application environments and methodologies. On the other hand, the asynchronous A* search algorithm is a novel heuristic approach that speeds up the process of finding the shortest path by parallel processing and avoiding the need to wait for all branches to fully expand [37]. This method can accelerate decoding to some extent, but it may encounter issues of uneven load distribution in practical applications, where some processing nodes become overloaded while others remain idle, ultimately affecting recognition accuracy.

In terms of accuracy improvements, Ren et al. explored the impact of different beam widths on decoding performance, discovering that a larger beam width leads to more accurate recognition at the cost of increased computational demands [38]. They also introduced a language model during the decoding phase, which further improved recognition accuracy. However, the computational cost of beam search grows exponentially with increasing beam width, significantly slowing down the decoding process. Additionally, incorporating a language model requires extra training and tuning, adding to the system’s complexity and making it challenging to implement in practical production environments. Zhang Yukai directly utilized the CTC (connectionist temporal classification) decoding function from the Keras backend, applying the best path method for decoding to enhance model recognition accuracy [39]. While CTC decoding performs well in handling variable-length inputs and outputs, it assumes that each label in the output sequence is independent, ignoring contextual information. This limitation can reduce its effectiveness in recognizing long sentences or conversations with complex contextual relationships. Xie designed an n-best re-scoring non-autoregressive decoding method, where the n-best results from CTC decoding are fed into an attention mechanism for weighted rescoring [40]. This approach ensures accurate decoding results while accelerating the process. However, this method may encounter issues with redundant results in the n-best list, leading to wasted computational resources and affecting the overall efficiency of recognition.

Some researchers have aimed to balance both speed and accuracy. Abdinabi et al. proposed an end-to-end approach that combines deep learning with Markov models and incorporates a CTC/attention network [41]. This method was applied to Uzbek speech recognition, where it improved recognition accuracy while reducing training time. However, in the decoding phase, they used a greedy algorithm, which, while effective at selecting the highest probability letter at each step, makes decisions based on local information alone, ignoring global constraints. As a result, this approach often struggles to find the global optimal solution. Li et al. designed a new network called ESAformer, which integrates a recurrent gated convolutional self-attention mechanism within the Transformer encoder layers [42]. This structure captures multi-order feature interactions while extracting global features, leading to improved recognition accuracy as validated on the Aishell-1, HKUST, and WSJ datasets. Yao et al. introduced innovations to the Conformer model structure, proposing a new model named Zipformer. This model employs a U-net-like structure in the encoding part, designs additional modules to improve the efficiency of calculating attention weights, utilizes a new normalization method called BiasNorm, and introduces new activation functions, SwooshR and SwooshL [43]. The model achieved excellent recognition results on the LibriSpeech, Aishell-1, and WenetSpeech datasets, demonstrating the feasibility of their approach. Tian et al. approached the problem from a different angle, classifying models based on their autoregressive nature. They then deeply integrated autoregressive and non-autoregressive models, combining the high recognition accuracy of one with the efficiency of the other [44]. This approach was validated on the Aishell-1 and Librispeech-960 h English databases, yielding promising experimental results. Cheng [45] designed a new model structure that integrates an acoustic encoder (wav2vec2.0) with a pre-trained language encoder (BERT) into an end-to-end ASR model. This model was tested on the CALLHOME corpus and demonstrated superior recognition performance compared to other end-to-end models. Al-Radhi et al. proposed an innovative approach that optimizes the decoding process using a continuous wavelet vocoder. By introducing a more efficient speech representation model, this method reduces computational complexity and offers a new perspective on balancing decoding speed and accuracy [6]. While this approach is primarily applied to speech synthesis, its principles provide valuable insights for optimizing the decoding process in continuous speech recognition, particularly in improving decoding efficiency. The method demonstrates considerable potential in real-time applications, especially in scenarios with high demands for both decoding speed and recognition accuracy. However, whether it can fully address the current challenges in decoding efficiency requires further research and validation.

Despite significant advancements in the accuracy and efficiency of speech recognition models, challenges remain in the decoding stage, with relatively few innovative studies addressing this issue. Traditional decoding methods, such as beam search, are computationally intensive, thereby affecting decoding speed. Although some recent algorithms have made progress in improving either speed or accuracy, achieving a balance between the two remains a formidable challenge. Consequently, optimizing decoding algorithms remains a critical and unresolved issue for enhancing the performance of speech recognition systems.

2.3. Proposed Method

In summary, the primary issues with the above methods are model overfitting, leading to local optima, and the need for improvement in decoding algorithms. This paper proposes the following enhancements:

• Model Architecture: The study adopts the advanced Conformer architecture and conducts preliminary experiments with three variant structures. A model weight fusion method is proposed, where during the convergence phase of training, the models’ weights are organically fused based on loss values. This approach addresses two key aspects: first, smoothing and reducing the sensitivity of specific training iterations to noise in the data, thereby preventing overfitting during training; second, combining the different strengths of the three networks to improve overall recognition accuracy.
• Decoding Algorithm: A new decoding algorithm, “Dynamic Adaptive Decoding”, is introduced. This method connects CTC and attention mechanisms using nonlinear functions to form a hybrid decoding approach. It optimizes the output by dynamically evaluating and selecting the superior results from both decoding methods, leading to more accurate speech recognition. Additionally, a penalty mechanism is introduced to regulate the reliance on a single decoding method by adjusting penalty coefficients, thereby enhancing the robustness of the decoding process.

Finally, experiments will be conducted on the LibriSpeech dataset. Due to its large scale (approximately 1000 h), the results will ensure generalizability, avoiding issues related to specialized domains or small dataset sizes, thereby validating the reasonableness and feasibility of the proposed methods.

3. Materials and Methods

This study will employ the advanced Conformer and its variant end-to-end model structures. On the one hand, a weighted fusion approach will be used to organically combine these models, enhancing the model’s generalization ability and stability. On the other hand, a novel adaptive dynamic re-scoring decoding algorithm is proposed, which ensures recognition accuracy while improving decoding efficiency. The overall process of the proposed methodology is illustrated in Figure 1.

Figure 1. Overall flowchart of the methodology.

3.1. Fundamental Theory

Feature extraction from audio data is a critical step in the preprocessing phase of speech recognition. It involves converting audio data into feature representations that can be utilized by deep learning models. This study employs the commonly used Mel-Frequency Cepstral Coefficients (MFCC) for feature extraction, as it effectively simulates human auditory characteristics, enhancing the efficiency of model training and improving generalization capabilities. The process is illustrated in Figure 2.

Figure 2. MFCC processing flow.

The model structure employed in this study is the Conformer network, a novel hybrid neural network architecture designed specifically for capturing both local and global dependencies in complex sequential data. The Conformer network ingeniously combines the local feature extraction capabilities of convolutional neural networks (CNNs) with the long-range dependency handling abilities of Transformers. By integrating convolutional layers into its structure, the Conformer is capable of capturing short-term phonetic features crucial for speech recognition tasks, while its self-attention mechanism allows it to understand and process the overall semantic structure of longer passages. This combination makes the Conformer particularly effective in tasks such as natural language processing and speech recognition, which require both detailed local observation and broad global understanding. The specific structure of the Conformer is illustrated in Figure 3.

Figure 3. Conformer network architecture.

Building on this, two variant structures are also employed in this study: Unified_Conformer and U2++_Conformer. These three architectures are similar in design. Unified_Conformer introduces improvements in the encoding layer by implementing a dynamic block training method and utilizing a causal convolution module to support streaming recognition, thereby enhancing training effectiveness. U2++_Conformer builds on the causal convolution by adding a right-to-left attention decoder in the decoder section, enabling bidirectional decoding and improving decoding accuracy. The subsequent experiments will focus on these three structures.

During the model training process, the calculation of the loss value is a hybrid of two types: CTC (1) and attention mechanism decoding (2). The structure is illustrated in Figure 4.

$$LCTC=-{\displaystyle {\sum }_{\mathrm{t}=1}^T\log P_{ctc}(y_t^{true} | X)}$$

(1)

$$L_{att}=-{\displaystyle {\sum }_{t=1}^T\log P_{att}(y_t^{ture} | y_{1:t-1},X)}$$

(2)

where $y_t^{ture}$ represents the true label information, $X$ represents the input audio feature sequence, and $y_{1:t-1}$ represents all target labels before the time step $t$.

Figure 4. Hybrid CTC/attention model.

By using $\lambda$ as a weighting coefficient, the two types of loss are combined to ultimately obtain the total loss value:

$$L_{total}=\lambda L_{CTC}+(1-\lambda )L_{att}$$

(3)

3.2. Model Fusion

Typically, after model training is completed, to ensure both accuracy and robustness, models from the better-performing training epochs are selected. The weight parameters of these selected models are then averaged using an arithmetic mean, resulting in a new model with improved stability and performance.

Figure 5 illustrates the training process curve of a model. By selecting the models $M=\{M_1,M_2,\dots ,M_S\}$ and corresponding loss values $L=\{L_1,L_2,\dots ,L_S\}$ from the convergence phase, the arithmetic averaging formula is as follows:

$$M_{ave}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^S{M}_i}}{S}}$$

(4)

where $M_{ave}$ is the newly obtained model after arithmetic averaging, and $S$ is the number of models.

Figure 5. Model training example.

Although arithmetic averaging is simple and easy to implement in model fusion, it has significant drawbacks when dealing with the following issues: First, arithmetic averaging combines the outputs or parameters of multiple models with equal weights without considering the generalization ability of each model. This means that even if some models have overfitted to their training data, their influence in the ensemble is the same as that of well-performing models. As a result, the effects of overfitting may be retained in the final ensemble model, reducing the accuracy of predictions on new data. Second, when independently trained models are stuck in different local optima within their parameter space, simple arithmetic averaging may cause these local optima to cancel each other out or dilute their effects, especially when these local optima are far apart. This averaging approach can potentially disrupt valuable solutions found by well-performing models within certain local regions.

To overcome these shortcomings, we propose a weighted averaging method. We select models from the stage where they are about to converge, calculate weights using loss values as evaluation criteria, and perform a weighted sum to obtain the final weighted average model. The formula for the weighted calculation is as follows:

$${M^{\prime }}_{ave}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^S{w}_i\cdot M_i}}{{\displaystyle {\sum }_{i=1}^S{w}_i}}}$$

(5)

$$w_i={\displaystyle \frac{1}{los{s}_i+\in }}$$

(6)

where ${M^{\prime }}_{ave}$ is the new model obtained using weighted averaging, $w_i$ is the weight of each model, $\in$ is a small constant (set to 1 × 10⁻⁶) to avoid division by zero errors, and $los{s}_i$ is the loss value of model $M_i$.

The above approach applies specifically to models with the same structure. When dealing with multiple different models, direct calculation using the entire models is not feasible due to structural differences. First, for the three different models, the model with the best performance, as evaluated by the loss value, is selected as the base model. When calculating across different models, only the structurally identical overlapping parts are weighted and averaged, while the non-overlapping parts are left unchanged. The specific formulas and pseudocode for this approach are as follows:

First, the selection of the base model is carried out by defining the model with the smallest loss value as the base model $M_b$:

$$M_b=M_{{\mathrm{argmin}}_{i}Los{s}_i} where i\in \{1,2,\dots ,\mathrm{s}\}$$

(7)

Next, for the specific structure within the models, the weights $\omega$ and biases $b$ of the $\mathit{\text{l-th}}$ layer in the base model are calculated as follows:

$${\omega }_l^M={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^S{w}_i\cdot I(l\in M_b\cap M_i)\cdot } {\omega }_l^i}{{\displaystyle {\sum }_{i=1}^S{w}_i\cdot I(p\in M_b\cap M_i)}}}$$

(8)

$$b_l^M={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^S{w}_i\cdot I(p\in M_b\cap M_i)\cdot } b_l^i}{{\displaystyle {\sum }_{i=1}^S{w}_i\cdot I(p\in M_b\cap M_i)}}}$$

(9)

where $I(l\in M_b\cap M_i)$ is an indicator function that equals 1 if $M_i$ shares common structural parameters in layer $l$ with the base model $M_b$, and 0 otherwise. ${\omega }_l^i$ and $b_l^i$ represent the weights and biases of layer $l$ in model $i$, while ${\omega }_l^M$ and $b_l^M$ denote the weights and biases of layer $l$ in the new model $M$.

Finally, these parameters from each layer are integrated and aggregated to obtain the final model ${M^{″}}_{ave}$:

$${M^{″}}_{ave}=swish(X_l\cdot {\omega }_l^M+b_l^M)$$

(10)

where $swish$ is an activation function used within the network.

In the weighted averaging fusion method, each model’s contribution is weighted based on its performance on validation dataset, ensuring that models with better performance and greater generalization ability have a larger influence on the final outcome. This approach not only effectively mitigates the impact of overfitting but also better addresses the issue of multiple models potentially converging to different local optima, thereby guiding the ensemble model towards a more effective global optimum. Through this weighted mechanism, the stability and generalization performance of the model can be significantly enhanced, leading to more reliable and consistent results across different datasets and tasks. The detailed model fusion method is provided in Algorithm 1.

Algorithm 1 Weighted Average Prediction

Require: ε: A small constant to prevent division by zero 1: Initialize Weighted_Sum as an empty dictionary 2: Initialize Total_Weight as an empty dictionary 3: for each model Mi in set of models M do 4:   Compute weight Wi = 1/(Loss of Mi + ε) 5:   for each parameter k in the intersection of keys in base model Mb and Mi do 6:     If k is not in Weighted_Sum 7:        Initialize Weighted_Sum[k] = 0 8:        Initialize Total_Weight[k] = 0 9:     Weighted_Sum[k] += Wi * value of parameter k in Mi 10:     Total_Weight[k] += Wi 11: end for 12: for each parameter k in Weighted_Sum do 13:   Y_weighted[k] = Weighted_Sum[k]/Total_Weight[k] 14: end for 15: Return Y_weighted

In our experiments, we applied the above-mentioned weighted averaging fusion method to combine three networks: the Conformer and its two variants. The final result is a new hybrid model, Mixformer. The specific fusion process is illustrated in Figure 6. This fusion approach allows us to effectively integrate the strengths of the three models, further improving the overall performance and generalization ability of the model.

Figure 6. Model fusion process.

3.3. Decoding Algorithms

The commonly used decoding methods include attention decoding, CTC greedy search, and CTC prefix beam search.

The attention decoding algorithm decodes the input sequence using an autoregressive search method, calculating context vectors and weights at each step while considering all previously generated words, thereby ensuring high accuracy. However, this method has high computational complexity and consumes significant resources. In contrast, the CTC greedy search method selects the most likely character at each step, which is efficient but less accurate in complex scenarios. The CTC prefix beam search improves upon greedy search by retaining multiple likely prefix paths at each step, calculating the total probability for each prefix to select the best path. Although this increases accuracy, it also consumes more memory and computational resources.

These traditional methods each have their strengths and weaknesses, but they often struggle to balance accuracy and computational efficiency when dealing with complex speech signals. To overcome these limitations and enhance the model’s generalization ability and decoding performance, we propose an adaptive attention fusion rescoring method (AAF rescoring). This approach combines nonlinear function connections and a penalty mechanism to rescore the outputs of decoders. The specific process is illustrated in Figure 7.

Figure 7. Adaptive attention fusion rescoring (AAF rescoring) decoding algorithm.

First, the CTC prefix beam search is used to perform an initial decoding of the speech input, generating potential candidate decoding results. Although CTC prefix beam search may lack precision when handling complex backgrounds, it provides an efficient initial decoding outcome that serves as a foundation for subsequent processing. Next, attention beam search is applied to further process the candidate results generated by the CTC. Nonlinear functions are employed to connect these candidate paths, resulting in richer decoding outputs. This method enhances the accuracy and flexibility of the decoding process while preserving global contextual information. Additionally, it significantly improves recognition accuracy and reduces the required computational resources. Finally, a penalty mechanism is introduced to rescore the newly generated decoding results. The penalty mechanism limits the weights to prevent the model from becoming overly dependent on the output of a single decoder, thereby enhancing the model’s robustness. The specific formula is as follows:

$$\begin{array}{l}{S}_{combined}^{AFF}=\sigma ({\omega }_r)\cdot \log P_{att}(y_t | y_{<t},x)+\sigma ({\omega }_c)\cdot \log P_{CTC}(y_t | x)\\ -\lambda \cdot (\sigma {({\omega }_r)}^2+\sigma {({\omega }_c)}^2)\end{array}$$

(11)

where $P_{att}(y_t | y_{<t},x)$ represents the output of the attention-based decoder, indicating the conditional probability of generating the current output given the previous outputs and the input $x$; $P_{CTC}(y_t | x)$ represents the output of the CTC decoder, indicating the probability of generating the current output given the entire sequence $x$; $\sigma ()$ denotes the nonlinear activation function, with the sigmoid function being used in this context. $\lambda$ is a regularization coefficient used to penalize excessively large weight values, limiting the model’s tendency to rely heavily on the output of a single decoder, thereby enhancing the model’s robustness.

In summary, the adaptive dynamic rescoring decoding algorithm proposed in this paper introduces improvements in three key ways: First, we effectively integrate the CTC and attention decoding methods, leveraging the efficiency of CTC and the high accuracy of attention through an adaptive dynamic rescoring mechanism. This combination significantly improves recognition accuracy and computational efficiency, allowing for more precise identification and analysis of features in diverse and uncertain audio signals. Second, the use of a nonlinear activation function enables the model to capture complex relationships within the data, enhancing its adaptability to diverse inputs. Third, the penalty mechanism reduces the model’s reliance on a single decoder by limiting weights, thus decreasing the risk of overfitting. These improvements collectively enable the model to achieve higher accuracy, stability, and efficiency in complex speech recognition tasks, meeting the demands of practical applications. The proposed adaptive dynamic rescoring algorithm not only facilitates more effective learning from complex data, improving the model’s robustness, but also helps mitigate the risk of overfitting associated with complex model structures.

4. Experiments

4.1. Experimental Setup

To validate the feasibility of the proposed model fusion and improved decoding algorithms, we implemented three different network architectures: Conformer, Unified_Conformer, and U2++_Conformer. The detailed experimental configurations for these models are presented in Table 1.

Table 1. Model parameter configurations.

Composition	Conformer	Unified_Conformer	U2++_Conformer
Encoder Params	Output Dim: 256	Output Dim: 256	Output Dim: 256
	Number of Blocks: 12	Number of Blocks: 12	Number of Blocks: 12
	Activation: Swish	Activation: Swish	Activation: Swish
Decoder Params	Number of Blocks: 6	Number of Blocks: 6	Number of Blocks: 3
Model Params	CTC Weight: 0.3	CTC Weight: 0.3	CTC Weight: 0.3
	LSM Weight: 0.1	LSM Weight: 0.1	LSM Weight: 0.1
	Reverse Weight: None	Reverse Weight: None	Reverse Weight: 0.3
Dataset Params	Max Length: 2000	Max Length: 2000	Max Length: 40960
	Min Length: 50	Min Length: 50	Min Length: 0
	Batch Size: 12	Batch Size: 12	Batch Size: 12
Max Training Epochs	100	100	100
Optimizer Params	Learning Rate: 0.004	Learning Rate: 0.001	Learning Rate: 0.001

Table 1 summarizes the key hyperparameter configurations used in the experiments, including the structure of the encoder and decoder, optimizer settings, and dataset parameters. Taking Conformer as an example, its encoder consists of 12 stacked layers, each with an output dimension of 256, utilizing the Swish activation function. The decoder comprises 6 stacked layers, with a CTC weight of 0.3, label smoothing weight of 0.1, and a batch size of 12. These parameters were derived from existing studies [46] and fine-tuned to align with specific task requirements. Building on this foundation, the Unified_Conformer and U2++_Conformer models introduce modifications to the model architecture and data processing strategies. Unified_Conformer employs a reduced learning rate (0.001) to enhance training stability. In contrast, U2++_Conformer reduces the number of decoder layers from 6 to 3 and incorporates a reverse sequence modeling weight (Reverse Weight = 0.3) to improve the capture of global features.

Additionally, during model training, the Adam optimizer was employed to dynamically adjust critical hyperparameters, thereby enhancing training efficiency and convergence [47].

Table 2 illustrates the unified network architecture of the three models, comprising an input layer, convolutional layers, positional encoding, Conformer blocks, and a fully connected layer. The input, represented as an 80-dimensional feature sequence, undergoes initial processing through 2D convolution to generate a 256-dimensional hidden state. This hidden state is then passed to the encoder, which is composed of 12 Conformer blocks. Each Conformer block includes feed-forward layers, self-attention mechanisms, and convolutional modules. Finally, the fully connected layer maps the encoded features to a 5002-dimensional output space. The 5002-dimensional output corresponds to the size of the model’s vocabulary, representing the total number of possible word or character categories.

Table 2. Model architecture.

Layer Name	Output Size	Input Size/Patch Size, Stride
Input	80-d	1000 × 80
Conv2D (input layer)	250 × 20 × 256	80 × T, 4 × 4, stride 2 (time and freq)
Positional encoding	250 × 256	250 × 256
Conformer block × 12	250 × 256	250 × 256
Feed-forward layer	250 × 256	250 × 256
Self-attention layer	250 × 256	250 × 256
CNN module	250 × 256	250 × 256
Fully connected layer	5002-d	250 × 256
Output	5002-d	250 × 5002

The hardware environment for the experiments includes an NVIDIA RTX 4070 GPU with 16.0 GB of RAM, running on a 64-bit Ubuntu 18.04 operating system. The software environment is Python 3.8.

4.2. Dataset Base

This study utilized the LibriSpeech dataset, a widely used large-scale corpus for speech recognition research. It comprises approximately 1000 h of English speech recordings from the public domain, encompassing diverse speech styles, accents, and topics. This dataset provides extensive resources for developing and evaluating speech recognition algorithms, ensuring the generalizability of experimental results.

In this study, the train-960 subset was used for model training, while dev-clean, dev-other, test-clean, and test-other were employed for validation and testing. The clean subsets include recordings characterized by clear speech, standardized pronunciation, and minimal background noise, primarily aimed at evaluating model performance under ideal conditions. In contrast, the other subsets represent recordings from more complex acoustic environments and are designed to assess the model’s robustness. These recordings are marked by distinctive features such as heavy accents, irregular pronunciations, rapid speaking rates, and low-quality audio caused by aging recording equipment or environmental factors. Additionally, they include various background noises, such as indoor reverberation, street sounds, and low-frequency environmental disturbances. Testing on these subsets, particularly dev-other and test-other, enables a comprehensive evaluation of the model’s adaptability and robustness across diverse speech scenarios, showcasing its ability to handle noisy and complex speech conditions effectively. Detailed information about the dataset is presented in Table 3.

Table 3. Dataset configuration.

Dataset	Subset	Audio Duration(h)	Word	Sampling Rate	Format
Training set	train-clean-100	100	5,185,647	16 kHz	flac
	train-clean-360	360	18,668,330
	train-clean-500	500	25,928,235
Validation set	dev-clean	5.4	53,167
	dev-other	5.3	52,183
Test set	dev-other	5.4	53,958
	test-other	5.1	50,961
Total		981.2	49,992,481

4.3. Performance Metrics

In speech recognition systems, evaluating model performance is crucial. To ensure that the experiments achieve the expected results, we use the word error rate ($WER$) as the evaluation metric. $WER$ measures the difference between the recognized words and the actual words by calculating the minimum number of operations required to substitute, delete, or insert words, and then dividing this by the total number of reference words. The specific formula is as follows:

$$WER={\displaystyle \frac{S_w+D_w+I_w}{N_w}}$$

(12)

where $S_w$ represents the number of word substitutions, $D_w$ represents the number of word deletions, $I_w$ represents the number of word insertions, and $N_w$ represents the total number of reference words. A lower $WER$ value indicates higher recognition accracy.

$Accuracy$ is another commonly used evaluation metric. The higher the accuracy, the better the model performs in recognizing speech. It can be calculated using the following formula:

$$Accuracy={\displaystyle \frac{N_E}{N_w}}$$

(13)

where $N_E$ represents the number of incorrectly recognized words, which is the sum of word substitutions, deletions, and insertions. Therefore, there is a corresponding relationship between $Accuracy$ and $WER$:

$$Accuracy={\displaystyle \frac{N_w-(S_w+D_w+I_w)}{N_w}}=1-WER$$

(14)

5. Results

5.1. Basic Experiment

We first trained the three base models—Conformer, Unified Conformer, and U2++ Conformer—on the LibriSpeech train-960 dataset and conducted basic experiments using three traditional decoding methods during the decoding phase. After the training was completed, we applied arithmetic averaging for an initial model fusion. The models were then tested and validated on the dev_clean, dev_other, test_clean, and test_other datasets. The changes in loss values and learning rates during the training process are shown in Figure 8 and Figure 9, respectively. The quantitative results from the testing and validation are presented in Table 3.

Figure 8. Loss value variation curve.

Figure 9. Learning rate variation curve.

Figure 8 and Figure 9 illustrate the changes in loss values and learning rates over 100 training epochs for the three models. Compared to the Conformer model, the Unified Conformer and U2++ Conformer models exhibit a more significant reduction in loss values and a smoother decline in learning rates. This improvement is attributed to the dynamic block training method and the enhancements from the causal convolution module, which have contributed to increased speech recognition accuracy. Additionally, although the U2++ Conformer shows a smooth decline in learning rates and the fastest reduction in loss values during the initial stages, its performance in later stages is not as strong as that of the Unified Conformer. This may be due to the increased model complexity introduced by the bidirectional decoding structure, which might have hindered the model’s ability to fully learn effective features within the training epochs, ultimately affecting its final performance.

Table 4 compares the recognition performance of three different network structures—Conformer, Unified Conformer, and U2++ Conformer—using three traditional decoding algorithms across two types of test and validation sets (clean and noisy environments).

Table 4. Basic test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Conformer (Arithmetic mean)	attention	7.10%	16.30%	7.23%	16.75%
	ctc_greedy_search	8.83%	18.46%	9.19%	18.77%
	ctc_prefix_beam_search	8.70%	18.39%	9.06%	18.73%
Unified_Conformer (Arithmetic mean)	attention	4.55%	10.12%	4.93%	10.31%
	ctc_greedy_search	4.68%	12.05%	4.88%	12.16%
	ctc_prefix_beam_search	4.63%	12.03%	4.84%	12.09%
U2++_Conformer (Arithmetic mean)	attention	4.41%	10.59%	4.87%	10.83%
	ctc_greedy_search	5.08%	12.38%	5.26%	12.59%
	ctc_prefix_beam_search	5.03%	12.34%	5.20%	12.48%

First, in terms of model methodology, Unified Conformer and U2++ Conformer show significant improvements in accuracy compared to the Conformer model across all dataset types, with WER decreasing by approximately 35% on a relative basis. However, when the environment shifts from clean to noisy, the WER for all models increases by about 125% relatively. This indicates that all three models still have limitations in handling noise, suggesting the need for further optimization to improve performance in complex environments.

Second, regarding decoding algorithms, the attention mechanism performs well on the clean environment validation set, with WERs of 7.10%, 4.55%, and 4.41% for Conformer, Unified Conformer, and U2++ Conformer, respectively. However, in noisy environments, the WERs increase to 16.30%, 10.12%, and 10.59%, respectively. Similarly, the recognition accuracy of both CTC greedy search and CTC Prefix Beam Search declines as the dataset environment transitions from clean to noisy.

In summary, while the recognition accuracy of Unified Conformer and U2++ Conformer shows improvement over Conformer in the basic experiments, there is still room for improvement in noisy environments. This reflects the inherent nature of these models as neural networks, which are prone to overfitting during training, and their robustness needs enhancement. Additionally, the current decoding algorithms exhibit suboptimal recognition accuracy when dealing with speech signals in complex environments. Therefore, there is a clear need for further optimization of both the model structures and decoding methods.

5.2. Model Fusion Experiment

To further optimize the models, we employed a weighted fusion approach. For each individual model, we conducted a comparative experiment between arithmetic averaging and weighted averaging under the same configuration environment. The comparison of the single model fusion test results for the three structures is presented in Table 5, Table 6 and Table 7.

Table 5. Conformer model fusion test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Conformer (Arithmetic mean)	attention	7.10%	16.30%	7.23%	16.75%
	ctc_greedy_search	8.83%	18.46%	9.19%	18.77%
	ctc_prefix_beam_search	8.70%	18.39%	9.06%	18.73%
Conformer (weighted mean)	attention	7.00%	16.26%	7.18%	16.72%
	ctc_greedy_search	8.83%	18.41%	9.16%	18.74%
	ctc_prefix_beam_search	8.69%	18.35%	9.06%	18.69%

Table 6. Unified Conformer model fusion test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Unified_Conformer (Arithmetic mean)	attention	4.55%	10.12%	4.93%	10.31%
	ctc_greedy_search	4.68%	12.05%	4.88%	12.16%
	ctc_prefix_beam_search	4.63%	12.03%	4.84%	12.09%
Unified_Conformer (weighted mean)	attention	4.56%	10.12%	4.93%	10.32%
	ctc_greedy_search	4.67%	12.05%	4.87%	12.17%
	ctc_prefix_beam_search	4.63%	12.02%	4.83%	12.10%

Table 7. U2++ Conformer model fusion test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
U2++_Conformer (Arithmetic mean)	attention	4.41%	10.59%	4.87%	10.83%
	ctc_greedy_search	5.08%	12.38%	5.26%	12.59%
	ctc_prefix_beam_search	5.03%	12.34%	5.20%	12.48%
U2++_Conformer (weighted mean)	attention	4.39%	10.54%	4.81%	10.79%
	ctc_greedy_search	5.05%	12.37%	5.27%	12.60%
	ctc_prefix_beam_search	5.01%	12.32%	5.20%	12.47%

Table 5, Table 6 and Table 7 present a comparison of recognition performance between arithmetic averaging fusion and weighted averaging fusion within the same model. The comparison reveals that using weighted fusion provides certain improvements across different data environments and decoding algorithms compared to arithmetic averaging fusion. For example, in the Conformer model with the attention mechanism decoding algorithm, the WER in the clean environment decreases from 7.10% to 7.00% on the Dev-Clean set, and from 7.23% to 7.18% on the Test-Clean set. In the noisy environment, the WER decreases from 16.30% to 16.26% on the Dev-Other set, and from 16.75% to 16.72% on the Test-Other set. Similar optimizations are also observed in the Unified Conformer and U2++ Conformer models.

These results indicate that model fusion using loss values as weights can effectively improve recognition accuracy, particularly in complex noisy environments. Additionally, this fusion method can mitigate overfitting issues during training, further enhancing the robustness of the model. Therefore, the proposed model fusion method not only optimizes the performance of individual models but also offers an effective strategy for addressing challenges in different environments. Subsequent experiments will apply weighted fusion to the three model structures.

Next, we continued with this fusion approach to combine the Conformer, Unified Conformer, and U2++ Conformer networks into a mixed model called Mixformer and conducted experiments. The results are presented in Table 8.

Table 8. Multi-model fusion test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Mixformer	attention	4.60%	10.12%	4.91%	10.22%
	ctc_greedy_search	4.58%	11.85%	4.85%	12.06%
	ctc_prefix_beam_search	4.54%	11.83%	4.83%	11.97%

Table 8 presents the recognition performance results after applying weighted fusion to the Conformer, Unified Conformer, and U2++ Conformer models. Comparing Table 7 with Table 5, Table 6 and Table 7, it is evident that the fused model outperforms the individual models in both clean and noisy environments, regardless of the decoding method used. Particularly in noisy environments, the WER of the fused model is significantly lower than that of the individual models under the same conditions. This demonstrates that the fused model, by integrating the structures and advantages of different models, not only improves recognition accuracy but also enhances robustness and generalization capabilities, leading to superior performance across various environments and decoding algorithms. Additionally, this approach effectively addresses issues such as local optima and overfitting that may arise during model training.

In summary, the Mixformer model, obtained by applying weighted fusion to the three Conformer models, significantly enhances the performance of the speech recognition system, demonstrating greater robustness and generalization ability. This indicates that the proposed model fusion method effectively resolves potential issues of local optima and overfitting during training, while also proving its significant effectiveness in improving recognition accuracy and handling complex environments.

5.3. Decoding Algorithm Improvement Experiment

For optimizing the decoding algorithm, we applied the proposed dynamic adaptive decoding algorithm and conducted experiments on the three models (after weighted fusion). The test results are presented in Table 9.

Table 9. Decoding algorithm innovation test results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Conformer	adaptive attention fusion rescoring	5.17%	12.35%	5.43%	12.36%
Unified_Conformer	adaptive attention fusion rescoring	4.35%	10.02%	4.73%	10.13%
U2++_Conformer	adaptive attention fusion rescoring	4.14%	10.20%	4.42%	10.08%

Table 9 presents the recognition performance of three models using the proposed decoding algorithm, while Table 10 compares it against traditional decoding methods, including attention, CTC greedy search, and CTC prefix beam search, highlighting the significant advantages of the adaptive attention fusion rescoring algorithm. This proposed algorithm dynamically integrates the efficiency of CTC with the global modeling capabilities of attention, effectively overcoming the limitations of traditional methods. Although attention decoding excels in capturing global dependencies, it struggles with complex alignments or noisy environments. Conversely, CTC decoding is computationally efficient but neglects contextual relationships, and both greedy search and prefix beam search often suffer from suboptimal accuracy in challenging scenarios.

Table 10. Comparison of decoding algorithms.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Conformer	attention	7.00%	16.26%	7.18%	16.72%
	ctc_greedy_search	8.83%	18.41%	9.16%	18.74%
	ctc_prefix_beam_search	8.69%	18.35%	9.06%	18.69%
	adaptive attention fusion rescoring	5.17%	12.35%	5.43%	12.36%
Unified_Conformer	attention	4.56%	10.12%	4.93%	10.32%
	ctc_greedy_search	4.67%	12.05%	4.87%	12.17%
	ctc_prefix_beam_search	4.63%	12.02%	4.83%	12.10%
	adaptive attention fusion rescoring	4.35%	10.02%	4.73%	10.13%
U2++_Conformer	attention	4.39%	10.54%	4.81%	10.79%
	ctc_greedy_search	5.05%	12.37%	5.27%	12.60%
	ctc_prefix_beam_search	5.01%	12.32%	5.20%	12.47%
	adaptive attention fusion rescoring	4.14%	10.20%	4.42%	10.08%

In contrast, the proposed decoding algorithm leverages a dynamic weighting mechanism to adaptively blend CTC’s efficiency with attention’s contextual understanding. This synergy enhances robustness in noisy conditions, delivering notable performance improvements. For instance, on noisy datasets such as Dev-Other and Test-Other, the proposed algorithm achieves significant reductions in WER. Specifically, on the Test-Other dataset with the U2++_Conformer model, WER decreased from 12.47% with CTC prefix beam search to 10.08%, a reduction of 2.39%. This demonstrates the algorithm’s ability to accurately capture complex, nonlinear features in speech signals.

In summary, the novel dynamic adaptive decoding method proposed in this paper effectively combines the strengths of both CTC and attention mechanisms, merging the advantages of both decoding algorithms. This approach improves decoding efficiency while ensuring recognition accuracy. By replacing linear functions with nonlinear ones, the method more effectively captures the complex nonlinear features in audio data. Additionally, by introducing a penalty function mechanism, it limits the model’s reliance on a single type of decoding output. Ultimately, this method significantly enhances the accuracy and robustness of speech recognition, providing new insights for the advancement of speech recognition technology and demonstrating its substantial effectiveness in practical applications.

5.4. Ablation Experiment

We combined the two aforementioned optimizations, using the Mixformer hybrid structure model and the dynamic adaptive decoding algorithm for the experiment. The contents of the previous experiments were summarized and compared in an ablation study. The experimental results are presented in Table 11 and Figure 10.

Table 11. Summary of ablation experiment results.

Method	Decoding Algorithm	WER(%)
		Dev-Clean	Dev-Other	Test-Clean	Test-Other
Conformer	attention	7.00%	16.26%	7.18%	16.72%
	ctc_greedy_search	8.83%	18.41%	9.16%	18.74%
	ctc_prefix_beam_search	8.69%	18.35%	9.06%	18.69%
Unified_Conformer	attention	4.56%	10.12%	4.93%	10.32%
	ctc_greedy_search	4.67%	12.05%	4.87%	12.17%
	ctc_prefix_beam_search	4.63%	12.02%	4.83%	12.10%
U2++_Conformer	attention	4.39%	10.54%	4.81%	10.79%
	ctc_greedy_search	5.05%	12.37%	5.27%	12.60%
	ctc_prefix_beam_search	5.01%	12.32%	5.20%	12.47%
Mixformer	attention	4.60%	10.12%	4.91%	10.22%
	ctc_greedy_search	4.58%	11.85%	4.85%	12.06%
	ctc_prefix_beam_search	4.54%	11.83%	4.83%	11.97%
Conformer	adaptive attention fusion rescoring	5.17%	12.35%	5.43%	12.36%
Unified_Conformer	adaptive attention fusion rescoring	4.35%	10.02%	4.73%	10.13%
U2++_Conformer	adaptive attention fusion rescoring	4.14%	10.20%	4.42%	10.08%
Mixformer	adaptive attention fusion rescoring	3.92%	8.94%	4.07%	9.31%

Figure 10. Summary of recognition results under different methods.

The above experimental results clearly demonstrate that each step of improvement significantly enhances recognition performance. First, the model fusion method (Mixformer) shows lower WER compared to individual models across both attention and other decoding algorithms, indicating its effectiveness in enhancing model robustness and generalization capabilities. Second, the dynamic adaptive decoding algorithm significantly reduces WER across various test environments compared to traditional decoding algorithms (attention, CTC greedy search, CTC prefix beam search), further improving recognition accuracy. Additionally, the ablation experiment confirms the effectiveness of each improvement step, highlighting the independent contributions of model fusion and decoding algorithm innovation to enhancing speech recognition performance, without mutual interference.

Ultimately, the combination of the model fusion and dynamic adaptive decoding algorithm yielded the best recognition performance across all test environments, with WERs of 3.92%, 8.94%, 4.07%, and 9.31% in the Dev-Clean, Dev-Other, Test-Clean, and Test-Other environments, respectively, validating the effectiveness and practicality of the proposed methods. This innovative approach offers new insights for the development of speech recognition technology, demonstrating significant advantages in improving recognition accuracy and handling complex environments.

Additionally, this study conducts a comprehensive comparison of the proposed Mixformer model with current state-of-the-art speech recognition models, as shown in Table 12 and Figure 11. The experimental results indicate that while DeepSpeech and Kaldi perform reasonably well on clean datasets such as Test-Clean, their performance significantly degrades in noisy environments, such as the Test-Other dataset, highlighting limitations in robustness. In contrast, models like HuBERT and Wav2Vec 2.0, which leverage deeper network architectures and pre-training techniques, achieve lower word error rates (WERs) across datasets, particularly excelling in clean data scenarios.

Table 12. Comparison of speech recognition accuracy using state-of-the-art models.

Method	WER(%)
	Dev-Clean	Dev-Other	Test-Clean	Test-Other
DeepSpeech	4.71%	12.6%	4.80%	13.0%
Kaldi	4.63%	12.1%	4.81%	12.7%
HuBERT	3.93%	9.35%	4.08%	10.5%
Wav2Vec 2.0	3.95%	9.25%	4.10%	9.82%
Mixformer (Our method)	3.92%	8.94%	4.07%	9.31%

Figure 11. Comparison of our proposed model with state-of-the-art models.

The proposed Mixformer model outperforms all benchmark models across all test datasets. Notably, it reduces the WER by approximately 0.4% compared to HuBERT on the Dev-Other and Test-Other datasets, demonstrating its enhanced robustness in challenging acoustic conditions. These findings suggest that Mixformer not only improves recognition accuracy but also provides better adaptability to complex environments, underscoring its effectiveness and potential in speech recognition applications.

6. Conclusions

As a popular research field, speech recognition has wide-ranging applications in various aspects of life. With the advent of deep learning, the latest end-to-end architectures have demonstrated high recognition efficiency and accuracy. However, overfitting and local optima are common challenges in training deep learning models. Additionally, existing decoding algorithms struggle to balance efficiency and accuracy during the decoding phase, and recognition accuracy often declines as the complexity of the dataset increases.

To address these issues, this paper proposes innovations in both model fusion and decoding algorithms. A fusion method based on training loss values combines models within and across structures near convergence, alleviating overfitting and local optima. Additionally, a dynamic adaptive rescoring algorithm integrates CTC and attention mechanisms, utilizing nonlinear functions and penalty mechanisms to reduce reliance on single decoding outputs, thereby improving robustness. Experimental results on the LibriSpeech dataset demonstrate that model fusion significantly enhances recognition accuracy, with cross-model fusion and the proposed decoding algorithm yielding further improvements. Furthermore, a comparative analysis with state-of-the-art models reveals that the proposed method demonstrates superior recognition accuracy, particularly on complex datasets, highlighting its robustness and adaptability in challenging acoustic environments.

7. Future Work

This study has several limitations concerning model fusion, dataset diversity, and hardware resources. First, the fusion strategies employed are confined to variants of the Conformer network. While these strategies effectively mitigate overfitting and improve recognition accuracy, they lack exploration of cross-architecture fusion, thereby narrowing the scope of investigation. Second, although the LibriSpeech dataset encompasses various noise types, the evaluation primarily focuses on its overall noisy subsets without conducting separate experiments for each specific noise category. This limits the ability to analyze the model’s robustness under distinct and diverse noise conditions, which are often encountered in real-world scenarios. Finally, the computational constraints of the laboratory server, with limited processing power, may have prolonged training times and restricted the optimization of model performance and precision.

Future work will address these limitations through the following improvements. First, a broader range of fusion strategies will be explored, integrating advanced architectures such as HuBERT and Wav2Vec to leverage their complementary strengths and develop more robust and generalizable models. Second, independent tests and analyses of each noise type will be conducted to thoroughly evaluate the model’s robustness in practical applications, thereby providing more comprehensive insights into its adaptability across various noise conditions. Lastly, upgrading the hardware platform to high-performance computing infrastructure, including multi-core and multi-GPU systems, could significantly enhance training efficiency and decoding speed, paving the way for further performance optimization.