AS-Split Conformer: A Stage-Wise Convolution–Attention Framework with Mamba Decoder for End-to-End Speech Recognition

Abstract

Automatic speech recognition (ASR) systems based on Conformer architectures achieve strong performance by jointly modeling local acoustic patterns and global contextual dependencies. However, their interleaved convolution–attention design leads to progressive entanglement of fine-grained acoustic features and global semantic representations, which weakens monotonic alignment in speech recognition and degrades performance in long utterances. To address this limitation, we propose an AS-Split Conformer–Mamba framework that decouples local and global modeling into two explicit stages. First, a stage-wise encoder is introduced, where a dedicated local modeling stage extracts phonetic-level acoustic features using SE-enhanced convolution, followed by a global modeling stage that captures long-range dependencies via multi-head self-attention and temporal convolution. Second, a Transition Fusion Block (TFB) is designed as an adaptive transition module that transforms local acoustic representations before they enter the global modeling stage. Third, intermediate CTC supervision is introduced to explicitly strengthen monotonic alignment at shallow representations. Finally, a hybrid Transformer–Mamba decoder is adopted, in which the Mamba block provides O(N) state-space computation within the replaced FFN sublayer while retaining Transformer attention mechanisms for acoustic–text alignment. Experiments conducted on AISHELL-1, THCHS-30, and ST-CMDS demonstrate that the proposed method achieves consistent improvements over strong baselines. On AISHELL-1, our model reduces Character Error Rate (CER) from 5.7% to 4.8% and Sentence Error Rate (SER) from 24.8% to 20.5%, while maintaining competitive computational efficiency.

1. Introduction

Automatic speech recognition (ASR) aims to automatically convert continuous speech signals into corresponding text sequences. It serves as a fundamental technology for intelligent human–computer interaction and supports applications such as meeting transcription, voice assistants, and in-vehicle speech systems. With the rapid advancement of deep learning, end-to-end ASR has become a dominant research paradigm. Its primary advantage is its ability to learn mappings directly from acoustic features to text sequences within a unified neural architecture, thereby avoiding complex procedures used in conventional hybrid systems, such as pronunciation-lexicon construction, frame-level alignment, and linguistic-rule engineering [1,2,3].

In the early development of end-to-end ASR [4], recurrent neural networks (RNNs) [5] were widely used for sequence modeling. Graves et al. proposed connectionist temporal classification (CTC), which enables end-to-end training without frame-level annotations by introducing a blank symbol and marginalizing over all possible monotonic alignment paths [6]. Subsequently, Chan et al. introduced the Listen, Attend and Spell (LAS) model, which incorporates an attention mechanism into an encoder–decoder framework to enable dynamic alignment between input speech and output text during decoding [7]. Kim et al. further proposed a joint CTC–attention optimization strategy that leverages the complementary strengths of both mechanisms to improve alignment stability and recognition accuracy [8,9].

Meanwhile, convolutional neural network (CNN)-based speech recognition models also achieved substantial progress. Deep Speech employed a hybrid architecture that combined deep convolutional and recurrent networks, achieving significant performance gains in both Mandarin and English ASR tasks [10]. Later, models such as wav2letter and time–depth separable (TDS) networks further demonstrated the effectiveness and efficiency of convolutional architectures for modeling local speech patterns [11,12].

In 2017, Vaswani et al. proposed the Transformer architecture, which replaces recurrent structures with multi-head self-attention and enables parallel modeling of dependencies across arbitrary positions within a sequence, achieving breakthrough performance in machine translation [13]. Subsequently, Speech-Transformer introduced Transformer-based modeling into speech recognition and demonstrated the effectiveness of purely attention-based ASR models. However, speech signals exhibit both long-range dependencies and strong local stationarity with temporal continuity. Pure self-attention lacks an inherent inductive bias toward local acoustic patterns and incurs high computational complexity when processing long sequences. Moreover, such models may overlook fine-grained local acoustic cues. Lightweight convolutional approaches can enhance local feature representation and effectively alleviate these limitations [14].

To simultaneously capture local and global information, Google introduced the Conformer architecture in 2020, which integrates depthwise separable convolution modules into the Transformer framework and adopts a Macaron-style encoder design [15]. By combining convolutional operations with self-attention, Conformer effectively captures both local acoustic patterns and long-range contextual dependencies. The model outperformed Transformer-based approaches on several benchmark datasets, including LibriSpeech and AISHELL-1, and consequently became one of the most widely adopted encoder architectures in end-to-end ASR systems [15].

Subsequent studies have sought to further improve the Conformer architecture. Branchformer improves feature representation by integrating self-attention and multilayer perceptron branches in parallel [16]. Squeezeformer reduces computational complexity through module compression and hierarchical reorganization strategies [17]. FastConformer improves inference efficiency for long-form speech by introducing linear local attention mechanisms [18,19]. Efficient Conformer further balances recognition accuracy and computational efficiency by combining progressive downsampling with efficient attention mechanisms, conceptually resembling the hierarchical compression strategy of Funnel-Transformer [20,21].

However, end-to-end ASR exhibits a strong monotonic alignment between speech frames and character outputs. Therefore, preserving fine-grained acoustic cues, including phoneme transitions, formant variations, and transient plosive characteristics, is particularly important. In the standard Conformer architecture, convolution modules and multi-head self-attention (MHSA) layers are alternately stacked within each encoder block. Although this design incorporates both local and global information at every layer, local acoustic representations become increasingly entangled with global contextual information as network depth increases [15,22]. Consequently, fine-grained acoustic information is gradually diluted and cannot be effectively preserved across layers. This phenomenon weakens low-level acoustic representations and adversely affects monotonic alignment during decoding, resulting in increased insertion and deletion errors, particularly in long utterances, variable-speaking-rate conditions, and noisy environments [14,23].

To address these limitations, we propose a local–global stage-wise Conformer framework with a Mamba-enhanced decoder for end-to-end Mandarin ASR. In the encoder, the proposed AS-Split Conformer decouples the conventional layer-wise alternating design into a stage-wise sequential architecture consisting of a local modeling stage, a Transition Fusion Block, and a global modeling stage. This design enables the model to first extract and accumulate short-term local acoustic features in shallow layers and then progressively integrate long-range contextual dependencies in deeper layers. A TFB is introduced to alleviate feature-distribution discontinuities between stages, while intermediate CTC supervision explicitly strengthens monotonic alignment learning in shallow representations.

Furthermore, we incorporate a Mamba state-space module into the Transformer decoder. Through its selective state-update mechanism, the Mamba module enhances long-range dependency modeling by replacing the feed-forward network in the Transformer decoder, reducing the computational cost of the replaced non-attention sublayer while maintaining Transformer-based acoustic–text alignment. This design is also motivated by the sequence modeling advantages demonstrated by Mamba-based speech applications [24].

Our principal contributions can be summarized as follows:

• A local-to-global stage-wise encoding paradigm that reorganizes the conventional layer-wise interleaved Conformer architecture into sequential acoustic refinement and contextual abstraction stages, preserving fine-grained acoustic information before global modeling.
• An adaptive Transition Fusion Block (TFB) that transforms local representations before global contextual modeling, ensuring smooth feature transition between stages.
• Intermediate CTC supervision that reinforces monotonic alignment at shallow representations, improving recognition stability.
• A Transformer–Mamba hybrid decoder that combines cross-attention alignment with efficient long-sequence modeling, where the Mamba block provides linear-complexity computation within the replaced non-attention sublayer.

2. Related Work

2.1. Connectionist Temporal Classification

CTC was proposed by Graves et al. [6]. By introducing a special blank symbol and marginalizing over all possible monotonic alignment paths, CTC enables end-to-end training without requiring frame-level annotations.

Let the input feature sequence be denoted by X and the target label sequence by Y:

$$X=\{x_1,x_2,\dots ,x_T\},Y=\{y_1,y_2,\dots ,y_U\}.$$

(1)

CTC defines the conditional probability of generating the target sequence Y given the input sequence X as the sum of probabilities over all valid alignment paths $\pi$ that correspond to Y:

$$P(Y\mid X)=\sum _{\pi \in {\mathcal{B}}^{-1}\left(Y\right)}{\displaystyle \prod _{t=1}^T}p({\pi }_t\mid X).$$

(2)

Here, $\mathcal{B}(·)$ denotes the collapsing operation that maps an alignment path $\pi$ to the target sequence Y by first merging consecutive repeated labels and then removing blank symbols. ${\mathcal{B}}^{-1}\left(Y\right)$ represents the set of all alignment paths that can be collapsed into the target sequence Y.

Model training is performed by minimizing the negative log-likelihood objective function:

$${\mathcal{L}}_{\mathrm{CTC}}=-\log P(Y\mid X).$$

(3)

Figure 1 illustrates an example of the CTC alignment process. Taking six input frames as an example, the model outputs the alignment path “a, a, $\epsilon$, b, $\epsilon$, b”.

Figure 1. Illustration of the CTC alignment process.

2.2. Squeeze-and-Excitation Block

The Squeeze-and-Excitation (SE) block performs channel-wise feature recalibration through two sequential stages, namely squeeze and excitation, thereby enhancing the network’s responsiveness to informative channel features [25]. This module serves as a core component of the SE-Conv unit employed in the local modeling stage of the proposed AS-Split Conformer architecture.

Squeeze stage: Global average pooling is applied to the feature map of each channel, compressing global spatial information into a compact channel descriptor:

$$z_c={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{h}_c\left(t\right).$$

(4)

Excitation stage: The importance weight of each channel is learned through a two-layer fully connected network. The first layer reduces the channel dimension from C to $C/r$, where r denotes the reduction ratio. After ReLU activation, the second layer restores the representation to the original channel dimension C. Finally, normalized channel weights are generated through a Sigmoid activation function:

$$\mathbf{s}=\sigma \left({\mathbf{W}}_2\delta \left({\mathbf{W}}_1\mathbf{z}\right)\right)$$

(5)

Here, $r=16$ denotes the channel reduction ratio, $\delta (·)$ denotes the ReLU activation function, and $\sigma (·)$ denotes the Sigmoid activation function.

Recalibration stage: The learned channel weights are multiplied channel-wise with the original feature maps to perform feature recalibration:

$${\widehat{h}}_c=s_c·h_c.$$

(6)

Through the above operations, the SE block explicitly models inter-channel dependencies. It enhances the contribution of informative features while suppressing irrelevant or noisy feature responses. Consequently, the representational capacity of the network is significantly improved, as shown in Figure 2.

Figure 2. Architecture of the Squeeze-and-Excitation (SE) block.

2.3. State-Space Models and Mamba

State-space models (SSMs) have recently emerged as efficient alternatives to Transformer-based architectures for long-sequence modeling. Unlike self-attention mechanisms, which compute pairwise interactions among all positions and introduce quadratic complexity with respect to sequence length, SSMs model sequential dependencies through latent state transitions and provide a more efficient framework for processing long contextual information. By parameterizing continuous-time dynamical systems and discretizing them into recurrent formulations, SSM-based architectures can capture long-range dependencies with linear computational complexity [26].

Recently, Mamba, a selective state-space model, has further improved the capability of traditional SSMs by introducing an input-dependent selective mechanism. Different from fixed state transition processes, Mamba dynamically adjusts state parameters according to the current input features, allowing the model to selectively retain or discard historical information. This selective state update mechanism enables efficient long-range sequence modeling while maintaining linear computational complexity, making Mamba a promising alternative to attention-based architectures for large-scale sequence processing [27].

Due to the inherent sequential characteristics of speech signals, Mamba and other SSM-based architectures have recently been investigated in speech processing and automatic speech recognition (ASR) tasks [28]. Existing studies can generally be divided into two major directions. The first direction focuses on incorporating Mamba modules into speech encoders to improve acoustic representation learning efficiency [28]. These approaches attempt to replace or complement attention-based acoustic modeling modules with selective state-space layers, aiming to reduce computational cost while maintaining the ability to capture long-range temporal dependencies in speech sequences [26]. Such encoder-oriented designs demonstrate the potential of SSMs for efficient modeling of long-duration speech signals, particularly in scenarios where conventional attention mechanisms suffer from increasing computational requirements.

The second research direction introduces Mamba modules into Transformer-based decoder or hybrid encoder–decoder architectures. Instead of completely replacing attention mechanisms, these methods preserve self-attention or cross-attention components for acoustic–text alignment and employ Mamba blocks for efficient sequential dependency modeling. Miyazaki et al. explored a hybrid Mamba decoder with cross-attention for end-to-end speech recognition, demonstrating that state-space modeling can be effectively integrated with Transformer-based alignment mechanisms and improve the efficiency of autoregressive decoding [24].

Although these studies demonstrate the effectiveness of Mamba-based architectures for speech sequence modeling, existing approaches mainly focus on improving computational efficiency through replacing or augmenting sequence modeling components. However, end-to-end ASR requires not only efficient long-context modeling but also accurate preservation of fine-grained acoustic information and stable monotonic alignment between speech frames and output tokens. Acoustic characteristics such as phoneme transitions, formant variations, and transient speech events are gradually abstracted during deep neural processing, and insufficient preservation of these local features may lead to alignment errors, especially in long utterances and variable speaking conditions.

Therefore, the interaction between local acoustic feature preservation and Mamba-based long-range sequence modeling remains insufficiently explored. In contrast to previous Mamba-based ASR studies that mainly optimize individual sequence modeling components, this work proposes an AS-Split Conformer–Mamba framework that explicitly separates local acoustic modeling and global contextual modeling in the encoder. Furthermore, a Transformer–Mamba hybrid decoder is introduced to combine the acoustic–text alignment capability of Transformer attention with the efficient sequential modeling capability of Mamba. This architecture provides a complementary solution by jointly addressing local acoustic representation preservation and long-context dependency modeling.

3. Materials and Methods

3.1. Datasets

Detailed statistics regarding the datasets are shown in Table 1.

Table 1. Dataset statistics.

Dataset	Duration (h)	Number of Speakers	Number of Utterances
AISHELL-1	178	400	141,600
THCHS-30	35	40	35,000
ST-CMDS	110	855	102,600

The detailed statistics of the datasets used in this study are presented in Table 1. Three publicly available Mandarin speech corpora, namely AISHELL-1, THCHS-30, and ST-CMDS, were employed for model training and evaluation. All datasets were partitioned into non-overlapping training, validation, and test sets.

Specifically, for AISHELL-1, we followed the official split: 120,098 utterances for training, 14,326 for validation, and 7176 for testing. For THCHS-30, we used 30,000 utterances for training, 2500 for validation, and 2500 for testing. For ST-CMDS, we used 92,340 utterances for training, 5130 for validation, and 5130 for testing.

AISHELL-1 [29], released by Beijing Shell Technology Co., Ltd., Beijing, China, contains Mandarin speech recordings collected from 400 speakers across different regions of China and covers multiple application domains, including finance, technology, and entertainment. It has become one of the most widely adopted benchmark datasets for end-to-end ASR research.

THCHS-30 [30], released by Tsinghua University, consists of approximately 35 h of Mandarin speech recorded from 40 speakers. Owing to its relatively clean recording conditions, it is frequently used for controlled speech recognition experiments.

ST-CMDS (ST Chinese Mandarin Dialect Speech) [31] contains approximately 110 h of standard Mandarin speech recordings collected from 855 speakers, providing substantial speaker diversity and variability.

In total, the three datasets comprise approximately 323 h of speech data and cover a wide range of recording environments, speaker characteristics, and speaking styles. For all experiments, 80-dimensional Fbank features were extracted from 16 kHz audio signals using a frame length of 25 ms and a frame shift of 10 ms. All recordings were stored in 16-bit mono WAV format, and global Cepstral Mean and Variance Normalization (CMVN) was applied. Model training and decoding were implemented based on the WeNet [32] and Kaldi [33] speech recognition toolkits.

3.2. Experimental Environment

All experiments were conducted on Ubuntu 22.04 LTS using PyTorch 2.2.0 with CUDA 12.1 and cuDNN 9.0. Model training and evaluation were performed on a single NVIDIA RTX 4090 GPU with 24 GB of memory, accompanied by an Intel Core i9-14900K CPU and 64 GB of RAM. The WeNet open-source speech recognition toolkit served as the underlying framework, upon which the proposed AS-Split Conformer–Mamba architecture was implemented.

For data preprocessing, all speech recordings were converted into 16 kHz, 16-bit mono WAV format. Eighty-dimensional Fbank features were extracted using a frame length of 25 ms and a frame shift of 10 ms. To improve robustness and generalization, Cepstral Mean and Variance Normalization (CMVN) and SpecAugment were applied. The SpecAugment strategy included both Time Masking and Frequency Masking operations.

During training, the Adam optimizer was adopted with an initial learning rate of $1\times {10}^{-3}$, together with a warm-up strategy and learning-rate decay schedule. The batch size was set to 32, and all models were trained for 100 epochs.

The encoder was based on the AS-Split Conformer architecture. The SE module employed a reduction ratio of 16, while the TCN module in the global stage consisted of three dilated residual convolution blocks with a kernel size of 3. Multi-head self-attention used four attention heads, and the hidden representation dimension was set to 256. The intermediate CTC supervision weight was configured as 0.3.

For decoding, a hybrid Transformer–Mamba architecture was adopted. The Mamba state dimension was set to 256. Beam Search with a beam width of 10 was used during inference, followed by Attention Rescoring to generate the final recognition results.

3.3. Model Architecture

The overall architecture of the proposed AS-Split Conformer–Mamba ASR model is shown in Figure 3. In contrast to the standard Conformer, which integrates convolution and self-attention within each layer, we adopt a stage-wise design that separates local and global modeling. The encoder is based on an AS-Split Conformer, while the decoder adopts a Transformer–Mamba hybrid structure. An intermediate CTC branch is introduced to improve alignment learning. The model consists of an input feature preprocessing module, a local modeling stage, a global modeling stage, a Transition Fusion Block (TFB), an intermediate CTC module, and a Transformer–Mamba decoder.

Figure 3. Overall architecture of the proposed AS-Split Conformer–Mamba model.

3.3.1. SE-Conv Based Local Modeling Module

In the proposed AS-Split Conformer encoder, the local modeling stage serves as the first processing phase, responsible for extracting fine-grained acoustic features from the input speech representations. Unlike the standard Conformer, where convolution and self-attention are interleaved at every layer, our design dedicates the initial $K=4$ layers exclusively to local feature extraction. This isolation ensures that short-term temporal patterns—such as phoneme transitions, formant variations, and transient acoustic events—are sufficiently captured before being integrated with global contextual information.

The local modeling stage is built upon a convolutional module enhanced with the Squeeze-and-Excitation (SE) mechanism (see Section 2.2). As illustrated in Figure 4, each SE-Conv layer consists of three sequential components: a depthwise separable convolution for local feature extraction, an SE block for channel-wise recalibration, and a residual connection for gradient flow preservation.

Figure 4. Architecture of the SE-Conv module in the local modeling stage. Each layer applies a depthwise separable convolution to extract local time–frequency patterns, followed by Squeeze-and-Excitation (SE) recalibration that generates channel-wise scaling factors. A residual connection is added to facilitate gradient flow. The output maintains the same temporal resolution and channel dimension as the input.

At each of the $K=4$ layers, the input feature sequence is first processed by a depthwise separable convolution with a kernel size of 3, which extracts local time–frequency patterns with low parameter cost:

$$U^{\left(l\right)}=\mathrm{Conv}\left(X^{(l-1)}\right),$$

(7)

where $X^{(l-1)}\in {\mathbb{R}}^{T\times C}$ denotes the input to the l-th layer, and the convolution operation is factorized as

$$\mathrm{Conv}\left(X\right)=\mathrm{Pointwise}\left(\mathrm{Depthwise}\left(X\right)\right),$$

(8)

with $\mathrm{Depthwise}(·)$ applying a single filter per input channel (preserving the channel dimension) and $\mathrm{Pointwise}(·)$ performing a $1\times 1$ convolution to project features to the desired channel dimension.

The convolved features $U^{\left(l\right)}$ are then passed through the SE block. As described in Section 2.2, the SE block generates channel-wise scaling factors ${\mathbf{s}}^{\left(l\right)}$ through squeeze and excitation stages. These factors recalibrate the convolutional outputs via element-wise multiplication:

$${\widehat{U}}^{\left(l\right)}={\mathbf{s}}^{\left(l\right)}\odot U^{\left(l\right)},$$

(9)

where ⊙ denotes channel-wise multiplication. This SE-enhanced convolution operates in a residual fashion:

$$X^{\left(l\right)}={\widehat{U}}^{\left(l\right)}+X^{(l-1)},$$

(10)

allowing gradient flow and enabling effective training of the local stage.

The SE mechanism selectively enhances features that are more relevant for phonetic discrimination while suppressing irrelevant or noisy components. This is particularly beneficial for Mandarin speech recognition, where fine spectral details carry critical phonetic distinctions.

After processing through all $K=4$ layers, the local stage produces a refined feature representation:

$$F_{local}={\mathrm{\Phi }}_{\mathrm{local}}\left(X\right)\in {\mathbb{R}}^{T\times C},$$

(11)

where ${\mathrm{\Phi }}_{\mathrm{local}}$ denotes the overall mapping function of the local modeling stage. This representation preserves the fine-grained acoustic details essential for accurate frame-to-phoneme alignment, and is subsequently passed to the Transition Fusion Block for integration with global contextual information.

3.3.2. MHSA-TCN Based Global Modeling Module

Although SE-Conv effectively extracts local acoustic features, its capacity for modeling long-range contextual information remains limited. To fully learn global semantic information and temporal positional relationships within the speech sequence, we construct a dual-branch MHSA–TCN structure for global modeling. The MHSA branch captures long-range dependencies, while the TCN branch models local temporal patterns. The outputs of both branches are fused at the feature level. Its overall structure is shown in Figure 5.

Figure 5. Detailed architecture of the parallel MHSA-TCN global modeling module.

Here, MHSA is used to capture long-range dependencies, with its computation expressed as

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V$$

(12)

where Q, K, and V represent the Query, Key, and Value matrices, respectively, and $d_k$ is the dimension of the Key vectors. The output of the multi-head attention is given by

$$F_{MHSA}=\mathrm{Concat}({head}_1,\dots ,{head}_h)W^O$$

(13)

where h is the number of attention heads. Simultaneously, to enhance the model’s perception of temporal positional information, a temporal convolutional network (TCN) is introduced alongside the attention branch. Inspired by the design in SE-Conformer [15], we adopt a TCN with dilated causal convolutions. The operation is defined as

$$F\left(s\right)=\left(X{\ast }_{d}f\right)\left(s\right)={\displaystyle \sum _{i=0}^{k-1}}f\left(i\right)X_{s-di}$$

(14)

where d is the dilation factor, k is the kernel size, and $f\left(i\right)$ are the convolution kernel parameters.

The TCN branch consists of three residual blocks with dilation rate pairs of $(1, 2)$, $(2, 4)$, and $(4, 8)$, respectively. Each residual block contains two causal convolution layers with kernel size 3, where the dilation rates of the two convolution layers are assigned according to the corresponding pair.

Specifically, each TCN residual block is composed of two stacked 1D convolutional layers with kernel size 3 and dilation rates $d_1$ and $d_2$, respectively. The transformation is defined as

$$F_{TCN}\left(X\right)=\mathrm{Conv}1{\mathrm{D}}_{d_2}\left(\mathrm{ReLU}\left(\mathrm{Conv}1{\mathrm{D}}_{d_1}\left(X\right)\right)\right).$$

(15)

Each residual block uses a kernel size of 3 and applies ReLU activation after each convolutional layer.

The progressively increasing receptive field can be expressed as

$$R=(k-1)d+1$$

(16)

Finally, the outputs of the MHSA and TCN are fused along the channel dimension. To maintain the constant hidden dimension $C=256$ for subsequent stages (consistent with the encoder configuration in Section 4.1), we apply an immediate linear projection. The fusion process is formulated as

$$F_{global}=\mathrm{Linear}\left({\mathrm{Concat}}_{\mathrm{channel}}(F_{MHSA},F_{TCN})\right),$$

(17)

where ${\mathrm{Concat}}_{\mathrm{channel}}(·)$ stacks the two features along the channel axis, yielding a tensor in ${\mathbb{R}}^{T\times 2C}$, and $\mathrm{Linear}(·)$ projects it back to ${\mathbb{R}}^{T\times C}$ via a trainable weight matrix $W_{proj}\in {\mathbb{R}}^{2C\times C}$. This yields a combined representation containing both semantic dependency and temporal positional information, while preserving the expected hidden dimensionality for subsequent decoder layers.

3.3.3. Transition Fusion Block

In the staged encoding strategy, the local-stage output $F_{local}$ captures fine-grained acoustic details but lacks global contextual information. Directly feeding $F_{local}$ into the global modeling stage would introduce distributional mismatches, leading to unstable gradient updates and compromising the semantic modeling capacity of deep networks. To address this, we design a Transition Fusion Block (TFB) that performs adaptive representation transformation before global contextual modeling.

As illustrated in Figure 6, the input to the TFB is the local-stage output $F_{local}\in {\mathbb{R}}^{T\times C}$. First, to obtain a sequence-level statistical descriptor, we apply global average pooling (GAP) along the temporal dimension to $F_{local}$, compressing it into a context descriptor $F_{ctx}\in {\mathbb{R}}^C$. The computation for the c-th channel is formulated as

$$F_{ctx}^{\left(c\right)}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{F}_{local}^{\left(c\right)}\left(t\right),$$

(18)

where T denotes the sequence length and c indexes the channel dimension.

Figure 6. Architecture of the Transition Fusion Block (TFB). The input $F_{local}$ is split into two paths: the Local Path preserves original features via an identity mapping followed by a linear transformation $W_l$, while the Context Compression Path extracts sequence-level statistics through temporal pooling (GAP), broadcast expansion, and another linear transformation $W_c$. A Sigmoid gate generates the fusion weight $\alpha$, and the final output $F_{fusion}$ is computed as $F_{fusion}=\alpha \odot F_{local}+(1-\alpha )\odot \mathrm{Expand}\left(F_{ctx}\right)$.

Subsequently, both $F_{local}$ and the broadcast version of $F_{ctx}$ are fed into two learnable linear transformation layers, $W_l$ and $W_c$, respectively. A fusion weight $\alpha$ is then generated via a Sigmoid gate:

$$\alpha =\sigma \left(W_l{F}_{local}+W_c·\mathrm{Expand}\left(F_{ctx}\right)\right),$$

(19)

where $\sigma (·)$ denotes the Sigmoid function and $\mathrm{Expand}(·)$ replicates the context descriptor $F_{ctx}\in {\mathbb{R}}^C$ to every time frame, yielding a tensor in ${\mathbb{R}}^{T\times C}$.

Finally, the TFB produces the output through a residual-style weighted fusion:

$$F_{fusion}=\alpha \odot F_{local}+(1-\alpha )\odot \mathrm{Expand}\left(F_{ctx}\right),$$

(20)

where ⊙ denotes element-wise multiplication.

Role of the TFB: It is important to clarify that the TFB does not directly combine outputs from the local and global stages. Instead, the context descriptor $F_{ctx}$ in Equation (19) is obtained by temporally pooling the local representation $F_{local}$ itself, serving as a sequence-level statistical summary rather than the output of the global modeling stage. Therefore, the TFB functions as an adaptive transition module that (1) extracts global statistical information from the current local representation, and (2) learns a gating mechanism to adaptively reweight local features before they enter the global modeling stage. This ensures that the global stage receives a well-prepared representation rather than raw local features.

This gating mechanism enables the model to dynamically adjust the weights according to the temporal characteristics of the current input. When $\alpha$ approaches 1, the network prioritizes preserving the original local acoustic details; when $\alpha$ approaches 0, the network enhances the guidance of global statistical information for the current frame. Through this smooth soft switching, the TFB effectively bridges the semantic gap between local and global feature distributions. It not only maintains the integrity of shallow acoustic features but also provides modulated and more discriminative initial representations for subsequent deep global modeling, thereby improving overall recognition performance.

3.3.4. Intermediate CTC Supervision

In the staged encoding architecture, the local modeling stage extracts fine-grained acoustic features but lacks direct sequence-level alignment supervision. Applying CTC loss only at the final encoder output forces shallow features to receive alignment signals indirectly through back-propagated gradients, resulting in unstable frame-to-phoneme mappings and overloading deeper layers with both feature abstraction and alignment optimization.

To address this, we introduce an intermediate CTC (InterCTC) auxiliary supervision branch immediately after the TFB. As shown in Figure 7, this branch taps off from $F_{fusion}\in {\mathbb{R}}^{T\times C}$, applies a linear projection to the vocabulary space, and computes a CTC loss against the target sequence. This placement is deliberate: local acoustic information has been fully extracted but not yet modulated by global context, making it a suitable intermediate position for providing alignment supervision to shallow representations.

Figure 7. Illustration of the intermediate CTC (InterCTC) supervision mechanism. The branch is attached after the TFB, providing explicit alignment supervision to the local modeling stage. It is active only during training and discarded at inference.

The InterCTC branch serves three purposes: (1) it provides explicit sequence-level alignment targets for the local stage, enabling SE-Conv modules to learn more discriminative acoustic representations; (2) its gradients directly reach shallow layers, alleviating gradient vanishing and accelerating convergence; (3) it acts as a regularizer by enforcing consistency between intermediate and final CTC predictions, reducing insertion and deletion errors.

Formally, given $F_{fusion}\in {\mathbb{R}}^{T\times C}$, the branch applies

$$F_{inter}=F_{fusion}{W}_{inter}+b_{inter},$$

(21)

where $W_{inter}\in {\mathbb{R}}^{C\times V}$, $b_{inter}\in {\mathbb{R}}^V$, and V is the vocabulary size. The intermediate CTC loss ${\mathcal{L}}_{\mathrm{InterCTC}}$ follows the standard CTC formulation (Section 2.1).

With InterCTC, the training objective extends from two terms to three. The original joint CTC–attention objective is

$$\mathcal{L}=\lambda {\mathcal{L}}_{\mathrm{CTC}}+(1-\lambda ){\mathcal{L}}_{\mathrm{Att}},$$

(22)

where $\lambda =0.4$ from baseline optimization (Section 4.2). With InterCTC incorporated,

$$\mathcal{L}={\lambda }_f{\mathcal{L}}_{\mathrm{CTC}}^{\left(\mathrm{final}\right)}+{\lambda }_i{\mathcal{L}}_{\mathrm{InterCTC}}+(1-{\lambda }_f-{\lambda }_i){\mathcal{L}}_{\mathrm{Att}},$$

(23)

where ${\lambda }_f=0.4$ and ${\lambda }_i=0.3$ as determined via sensitivity analysis, satisfying ${\lambda }_f+{\lambda }_i<1$ to preserve attention loss contribution.

The InterCTC branch is active only during training and discarded at inference, introducing no additional latency or overhead. This design ensures temporally well-aligned representations from the local stage before global contextual integration, improving alignment accuracy, convergence speed, and overall robustness.

3.3.5. Transformer–Mamba Decoder

In conventional end-to-end ASR systems, the decoder typically relies on the Transformer architecture with self-attention mechanisms. However, the quadratic complexity $\mathcal{O}\left(N^2\right)$ of self-attention poses significant challenges for long-form speech recognition [27].

To reduce the computational cost of the non-attention component of the Transformer decoder, we replace the position-wise FFN with a Mamba block while retaining masked self-attention and cross-attention. Mamba is a selective state-space model (SSM) that achieves $\mathcal{O}\left(N\right)$ complexity with respect to sequence length for its state-space computation [34]. Its core innovation is an input-dependent selective mechanism that dynamically adjusts state update parameters based on the current input, enabling the model to selectively retain or discard historical information according to the temporal dynamics of the speech signal. The state update process is formulated as

$$h_t={\overline{A}}_t{h}_{t-1}+{\overline{B}}_t{x}_t,y_t=C_t{h}_t,$$

(24)

where ${\overline{A}}_t$, ${\overline{B}}_t$, and $C_t$ are input-dependent dynamic parameters.

We adopt a hybrid Transformer–Mamba decoder architecture, as illustrated in Figure 8. The decoder operates autoregressively, generating one token at a time conditioned on the encoder output $F_{global}\in {\mathbb{R}}^{T\times C}$ and the previously generated tokens. At each decoding step, the current hidden state first passes through a masked self-attention layer to model dependencies among previous tokens while preventing future information leakage. The resulting representation then enters a cross-attention layer, which attends to the encoder output features to align acoustic and textual information.

Figure 8. Architecture of the Transformer–Mamba hybrid decoder. The decoder operates autoregressively: the current hidden state passes through masked self-attention and cross-attention layers, followed by a Mamba block that replaces the standard feed-forward network for efficient long-sequence modeling. The final output is projected to generate the next token distribution.

Specifically, we replace the position-wise feed-forward network (FFN) in each Transformer decoder layer with a Mamba block, while retaining the masked self-attention, cross-attention, residual connections, and layer normalization modules. The cross-attention output is subsequently fed into the Mamba block, which efficiently models long-range dependencies via the selective SSM. Finally, the Mamba output is projected to generate the next token probability distribution.

Computational Complexity Analysis: The overall computational complexity of the proposed decoder is composed of two parts: (1) the masked self-attention and cross-attention modules, which have $\mathcal{O}\left(N^2\right)$ complexity with respect to the sequence length N, and (2) the Mamba block, which achieves $\mathcal{O}\left(N\right)$ complexity through its selective state-space formulation. Therefore, while the entire decoder remains $\mathcal{O}\left(N^2\right)$ due to the retained attention modules, replacing the FFN with the Mamba block substantially reduces the “empirical computational cost (FLOPs)” in the non-attention sublayer. This may reduce the practical inference cost by lowering the FLOPs of the replaced FFN sublayers, as validated empirically.The encoder also employs multi-head self-attention with $\mathcal{O}\left(N^2\right)$ complexity for global context modeling.

More specifically, we retain the masked self-attention and cross-attention layers from the standard Transformer decoder to preserve the alignment capability with encoder features, while replacing the original feed-forward network with a Mamba block. The Mamba block consists of two key components: a linear projection layer that expands the input feature dimension, and a selective SSM layer that performs recurrent state updates. The SSM layer maintains a hidden state $h_t$ that evolves over time, enabling contextual information to be carried across long sequences. The input-dependent parameters allow the model to selectively retain or discard information based on the current input, providing linear-complexity sequence modeling within the replaced feed-forward sublayer through the selective SSM, while the retained self-attention and cross-attention modules continue to model acoustic–text alignment. Additionally, the Mamba block incorporates a SiLU activation function and a residual connection to facilitate gradient flow and training stability. This cascaded design ensures that the decoder benefits from both precise acoustic–text alignment via cross-attention and efficient long-context modeling via Mamba. The entire architecture is trained end-to-end jointly with the AS-Split Conformer encoder.

In summary, the proposed Transformer–Mamba hybrid decoder combines the alignment capability of Transformer cross-attention with the efficient long-sequence modeling power of Mamba, where the Mamba block provides linear complexity within its state-space computation while the attention modules retain $\mathcal{O}\left(N^2\right)$ complexity for alignment.

4. Experiments

4.1. Evaluation Metrics and Hyperparameters

To comprehensively evaluate the performance of our proposed AS-Split Conformer–Mamba model for speech recognition, we adopt the Character Error Rate (CER) and Sentence Error Rate (SER) as primary evaluation metrics. CER measures the character-level discrepancy between the recognition result and the ground-truth text, defined as

$$CER={\displaystyle \frac{S+D+I}{N}}\times 100\%$$

(25)

where S is the number of substitutions, D is the number of deletions, I is the number of insertions, and N is the total number of characters in the reference text.

SER evaluates the correctness of whole-sentence recognition, defined as

$$SER={\displaystyle \frac{\mathrm{Error}}{\mathrm{Total}}}\times 100\%$$

(26)

where Error is the number of sentences with at least one character error in the test set, and Total is the total number of sentences in the test set. CER reflects the model’s character-level recognition accuracy, while SER provides an assessment of the model’s holistic recognition capability. Lower values for both metrics indicate better model performance.

Our experiment was implemented based on the WeNet end-to-end speech recognition framework [32,35], using PyTorch 2.2.0 on Ubuntu 22.04 LTS. The hardware environment included an Intel Core i9-14900K processor, 64 GB of RAM, and an NVIDIA RTX 4090 GPU (24 GB VRAM). All speech data was uniformly converted to a single-channel WAV format with a 16 kHz sampling rate and 16-bit quantization. Input features were 80-dimensional Fbank features with a frame length of 25 ms and a frame shift of 10 ms, followed by CMVN normalization. To improve model generalization, SpecAugment [36] data augmentation (time and frequency masking) was applied during training [37].

The encoder adopts the proposed AS-Split Conformer architecture with a total of 12 layers, where the local modeling stage has $K=4$ layers, and the global modeling stage has $L=8$ layers. The SE block [25] is embedded in the convolutional modules of the local stage, with a reduction ratio of 16. The Transition Fusion Block (TFB) is placed between the two stages to mitigate abrupt changes in feature distribution. MHSA uses four attention heads with a hidden dimension of 256. The intermediate CTC auxiliary supervision weight was set to 0.3 to enhance the monotonic alignment capability of shallow representations.

The decoder adopts a Transformer–Mamba hybrid architecture, where the Mamba state dimension is 256, and the selective state-space module is used to enhance long-range historical context modeling. During inference, we used an attention rescoring decoding strategy with a beam search width of 10.

Implementation Details for Reproducibility: All experiments were conducted with the following configurations to ensure reproducibility, as detailed in Table 2.

Table 2. Implementation details for reproducibility.

Category	Details
Vocabulary	Official WeNet tokenizer: 4233 characters (AISHELL-1), 4196 (THCHS-30), 4185 (ST-CMDS)
Decoder Configuration	4 layers, hidden dimension 256, 4 attention heads
Subsampling Configuration	Two-layer Conv2d subsampling module with 3 × 3 kernels and stride (2,2) in each layer, resulting in an overall temporal downsampling factor of 4; output projected to 256-dimensional encoder representation
FFN Dimension	2048
Dropout	0.1
Label Smoothing	0.1
Gradient Clipping	Max norm 5.0
Weight Decay	1 × 10−6
Attention Rescoring Weight	0.5
Attention Rescoring Formulation	Weighted sum of CTC and attention probabilities with fixed rescoring weight of 0.5 $S(y\mid x)=0.5S_{\mathrm{CTC}}(y\mid x)+0.5S_{\mathrm{Att}}(y\mid x)$. Applied to N-best hypotheses generated by CTC beam search with beam size 10.
Mamba Configuration	$d_{\mathrm{state}}=256,d_{\mathrm{conv}}=4,\mathrm{expand}=2$
SpecAugment	Time masking: 10 masks of max length 50 steps; Frequency masking: 2 masks of max length 15 channels
Learning-Rate Schedule	Adam optimizer, $1\times {10}^{-3}$ initial LR, 5000 warm-up steps, Noam decay
Checkpoint Selection	Lowest CER on validation set
Random Seeds	42, 43, 44 (mean results reported)
Statistical Testing	Bootstrap test, 10,000 resamples, $p<0.05$
Hyperparameter Tuning	All hyperparameters tuned on validation set; test set used only for final evaluation

4.2. Baseline Model Preparation

To establish a baseline system for subsequent experiments, we first performed a CTC weight sensitivity experiment using the standard Conformer model on the AISHELL-1 dataset. Since most mainstream end-to-end ASR models adopt an Attention–CTC joint training strategy, the choice of CTC weight directly affects the model’s alignment ability and final recognition performance. To obtain the optimal baseline configuration, we trained models with different CTC weights and recorded the CER and SER on the validation set. The optimal CTC weight $(\lambda =0.4)$ was then selected based on validation performance, and the test set was used only for final evaluation.

To ensure a fair comparison, all baseline models (Conformer, Squeezeformer, Branchformer, and Efficient Conformer) were re-trained under identical experimental conditions to those used for our proposed model, including the same training data, batch size (32), number of epochs (100), optimizer (Adam with warm-up), and data augmentation (SpecAugment). The experimental results are shown in Table 3 [29,35].

Table 3. Baseline model performance under different CTC weights.

CTC Weight	CER (%)	SER (%)
0.1	7.2	28.9
0.2	7.0	27.2
0.3	6.6	26.1
0.4	5.7	24.8
0.5	6.4	25.9
0.6	6.8	27.3
0.7	7.1	28.0
0.8	7.4	29.1
0.9	7.8	30.2

The CTC weight was selected based on the validation-set CER. The test set was not used for hyperparameter tuning.

As shown in Table 3, the CTC weight significantly influences model performance. As the CTC weight increases from 0.1 to 0.4, both CER and SER decrease, reaching the best performance at 0.4 (CER 5.7%, SER 24.8%). Further increasing the CTC weight leads to performance degradation. This phenomenon occurs because the CTC loss provides an explicit monotonic alignment constraint, alleviating the training difficulty caused by length mismatches between speech and text sequences, thereby improving convergence speed and stability. However, CTC focuses primarily on local temporal alignment and cannot fully exploit global contextual information. In contrast, the attention mechanism models long-range dependencies and learns global semantic representations. When the CTC weight is too small, its auxiliary alignment effect is insufficient; when it is too large, the gradient signal for the attention branch is reduced, impairing the model’s ability to learn global context. Considering both CER and SER, we select the Conformer model with a CTC weight of 0.4 as the baseline system for subsequent experiments. This result is consistent with the findings of prior studies on Attention–CTC joint optimization, confirming that an appropriate CTC auxiliary loss effectively improves alignment and recognition performance.

5. Results

5.1. Model Recognition Performance

To validate the effectiveness of our proposed AS-Split Conformer–Mamba model, we conducted comprehensive experiments on Mandarin benchmarks and additionally evaluated the model’s applicability to an English ASR dataset. We selected the AISHELL-1 dataset as the primary experimental platform and compared our model against traditional statistical models (GMM-HMM), hybrid neural network models (LSTM-HMM [38]), the widely used Conformer [15], and several other recently proposed improved architectures. We also conducted additional experiments on THCHS-30 and ST-CMDS to verify generalization, and on LibriSpeech test-clean to provide an additional cross-lingual generalization evaluation.

The proposed model introduces a slightly lower number of parameters compared with the baseline Conformer (30.1 M vs. 30.6 M), indicating that the performance improvement mainly comes from architectural reorganization rather than parameter expansion.

As shown in Table 4, traditional GMM-HMM and LSTM-HMM models achieve relatively high CERs of 8.7% and 6.8%, respectively, due to their limited capacity to model complex acoustic patterns. Conformer [15] significantly lowers the CER to 5.7% by incorporating a convolutional module to enhance local modeling. Branchformer [16] achieves further improvements via a dual-branch structure that simultaneously models local and global information. Squeezeformer [17] reduces computational complexity through structural compression while maintaining performance. Although these models improve recognition performance to varying degrees, they still adopt an intralayer interleaving strategy for local and global information, failing to separate the two modeling stages explicitly.

Table 4. Comprehensive performance comparison on Mandarin datasets.

Model	# Params	AISHELL-1		THCHS-30		ST-CMDS
		CER	SER	CER	SER	CER	SER
Traditional/Hybrid Models
GMM-HMM	—	8.7	45.8	—	—	—	—
LSTM-HMM	—	6.8	39.1	—	—	—	—
End-to-End Models
Conformer	30.6 M	5.7	24.8	5.2	34.1	4.9	37.2
Squeezeformer	—	5.1	22.0	—	—	—	—
Branchformer	—	5.3	22.9	—	—	—	—
Proposed Model
AS-Split Conformer–Mamba	30.1 M	4.8 †	20.5 †	4.1 †	29.5 †	4.2 †	35.2 †

Notes: Bold text indicates the best performance in the corresponding column. CER = Character Error Rate; SER = Sentence Error Rate. ^† indicates statistically significant improvement over the corresponding Conformer baseline trained under identical experimental settings (p < 0.05, bootstrap test with 10,000 resamples). — indicates not reported. All results reported without external language models. Relative improvements over Conformer: AISHELL-1: −15.8% (CER), −17.3% (SER); THCHS-30: −21.2% (CER), −13.5% (SER); ST-CMDS: −14.3% (CER), −5.4% (SER). All baseline models were reproduced under identical AISHELL-1 settings, except those explicitly marked as reported from original publications.

Our proposed AS-Split Conformer explicitly decouples the encoding process into distinct local and global modeling stages, effectively preventing the over-dilution of local acoustic information in deeper layers. Combined with the TFB for inter-stage feature fusion and the Mamba decoder for enhanced long-range context modeling, our model achieves the best CER and SER scores across all Mandarin datasets. Compared to Squeezeformer, our method yields a relative CER reduction of approximately 5.9%. Against the standard Conformer, the relative reduction is 15.8%, demonstrating the efficacy of our approach. The improvements over the corresponding reimplemented Conformer baselines are statistically significant at $p<0.05$.

To evaluate the English ASR capability of our model, we conducted an additional experiment on the LibriSpeech test-clean dataset, a widely adopted benchmark for English ASR. Following standard practice, we used the train-clean-100 subset (100 h) for training. Our model achieves a Word Error Rate (WER) of 2.3% without external language models. This experiment is only intended to evaluate the applicability of the proposed architecture to English ASR. Since existing LibriSpeech results are reported under different training data scales, augmentation strategies, decoding configurations, and external language-model settings, we do not perform direct comparisons with previously published results. We note that a direct quantitative comparison with prior works is challenging due to varying training data subsets (e.g., 100 h vs. 960 h) and experimental settings. Therefore, this experiment is presented as an informative qualitative demonstration rather than a strictly controlled benchmark. For reproducibility, the LibriSpeech experiment follows the same optimization strategy and model architecture used in the Mandarin experiments, including the optimizer, learning-rate schedule, beam search decoding, and attention rescoring strategy. Only the dataset-specific vocabulary and training split are modified according to the official LibriSpeech train-clean-100 setting.

To further evaluate the contribution of each module to overall model performance, we conducted ablation experiments on the AISHELL-1 dataset. Each module was progressively added or replaced while keeping other configurations unchanged. The experimental results are shown in Table 5. For single-module ablations, each component was individually integrated into the baseline Conformer while maintaining the original encoder–decoder topology.

Table 5. Ablation study and module contribution analysis on the AISHELL-1 test set (mean ± std over 3 random seeds).

Model Configuration	# Params	CER (%)	SER (%)	ΔCER
Conformer (baseline)	30.6 M	5.7 ± 0.1	24.8 ± 0.2	—
+ SE module only	∼30.7 M	5.6 ± 0.1	24.3 ± 0.2	−1.8%
+ TCN branch only	∼30.7 M	5.5 ± 0.1	24.0 ± 0.3	−3.5%
+ TFB only	∼30.8 M	5.5 ± 0.1	23.7 ± 0.3	−3.5%
+ InterCTC only	∼30.6 M	5.5 ± 0.1	23.9 ± 0.2	−3.5%
+ Mamba only	∼29.4 M	5.3 ± 0.1	22.8 ± 0.3	−7.0%
+ AS-Split	31.8 M	5.2 ± 0.1	22.5 ± 0.2	−8.8%
+ AS-Split + InterCTC	31.8 M	5.0 ± 0.1	21.7 ± 0.2	−12.3%
+ AS-Split + InterCTC + TFB	31.8 M	5.0 ± 0.1	21.1 ± 0.2	−12.3%
AS-Split Conformer–Mamba (Full)	30.1 M	4.8 † ± 0.1	20.5 † ± 0.2	−15.8%

Notes: Bold text indicates the best performance in the corresponding column. InterCTC = intermediate CTC supervision; TFB = Transition Fusion Block; Δ = relative improvement over Conformer baseline. All ablation experiments were run with three random seeds (42, 43, 44). Results are reported as mean ± standard deviation. ^† indicates p < 0.05 vs. the corresponding Conformer baseline trained under identical settings (bootstrap test with 10,000 resamples). The full model achieves relative CER and SER reductions of 15.8% and 17.3%, respectively, compared with the 30.6 M parameter Conformer baseline. The full model contains 30.1 M parameters, which is 1.6% fewer than the Conformer baseline and 5.3% fewer than the 31.8 M parameter AS-Split configuration. The slight parameter reduction from AS-Split (31.8 M) to the full model (30.1 M) is due to the replacement of the Transformer decoder’s FFN with the Mamba block, which has a more compact parameter structure. The parameter difference between Conformer + Mamba and AS-Split Conformer–Mamba mainly originates from the additional encoder modules (SE-Conv, TFB, and InterCTC projection) introduced by AS-Split.

As shown in Table 5, the recognition performance improves progressively with the introduction of each module. The individual module ablations confirm that each component contributes positively: the SE module alone reduces CER by 1.8%, the TCN branch by 3.5%, the TFB by 3.5%, InterCTC by 3.5%, and Mamba alone by 7.0%. Adding the AS-Split structure further reduces CER to 5.2% and SER to 22.5%, demonstrating that separating local feature modeling from global semantic modeling helps mitigate the degradation of local acoustic information in deeper layers. Adding InterCTC to the AS-Split Conformer further lowers CER to 5.0% and SER to 21.7%, confirming that intermediate supervision continuously strengthens local acoustic information learning and provides more stable feature representations for the subsequent decoder. Introducing the TFB module then reduces SER to 21.1% (while CER remains at 5.0%), indicating that the feature transformation block effectively alleviates representation mismatches between the local and global modeling stages and improves information transfer efficiency. Finally, incorporating the Mamba decoder yields the best performance, with a CER of 4.8% and SER of 20.5%. Compared to the baseline Conformer, this represents a relative CER reduction of 15.8% and a relative SER reduction of 17.3%. These results suggest that the Mamba component contributes to performance improvement, possibly through more efficient sequential representation modeling. Notably, the full model has slightly fewer parameters (30.1 M) than the AS-Split configuration (31.8 M), demonstrating the parameter compactness achieved by replacing the Transformer FFN with the Mamba block within the complete architecture. Parameter counts of cumulative configurations include the corresponding auxiliary modules, while temporary training-only branches are excluded from inference parameters. Parameter counts are rounded to one decimal place.

To further evaluate the computational efficiency advantage of the Mamba module, we measured the parameter count, floating-point operations (FLOPs), and real-time factor (RTF) for different models. The results are shown in Table 6.

Table 6. Computational complexity and efficiency analysis.

Model	# Params	FLOPs (G)	RTF	Throughput	Param. Eff.	ΔFLOPs	ΔRTF
Conformer	30.6 M	63.4	0.081	27,714	0.186	—	—
AS-Split Conformer	31.8 M	67.9	0.087	26,181	0.164	+7.1%	+7.4%
Conformer + Mamba	29.4 M	51.3	0.064	36,141	0.180	−19.1%	−21.0%
AS-Split Conformer–Mamba	30.1 M	55.8	0.068	33,257	0.159	−12.0%	−16.0%

Notes: Bold text indicates the best performance in the corresponding column. FLOPs = floating-point operations per forward pass (batch 1, seq len 2048); RTF = real-time factor (lower is better); Throughput = tokens/s on NVIDIA RTX 4090. Param. Eff. = CER per million parameters (lower is better). Δ denotes the relative change with respect to the Conformer baseline. Positive values indicate increases and negative values indicate reductions. Efficiency measurements (FLOPs, RTF, and throughput) are averaged over 5 inference runs to reduce measurement fluctuations. Statistical significance markers are omitted for these deterministic metrics.

As shown in Table 6, the AS-Split module enhances acoustic feature representation by adding multi-scale feature branches, which incurs a modest increase in parameters and computation. Compared to Conformer, the AS-Split Conformer increases parameters from 30.6 M to 31.8 M, FLOPs from 63.4 G to 67.9 G, and RTF from 0.081 to 0.087. In contrast, introducing the Mamba module significantly reduces model complexity. The Conformer + Mamba model reduces parameters to 29.4 M (3.9% less than baseline), FLOPs to 51.3 G (a 19.1% reduction), and RTF from 0.081 to 0.064 (a 21.0% inference efficiency improvement). This demonstrates that Mamba, which replaces the position-wise feed-forward network (FFN) in the Transformer decoder with a selective state-space model, substantially lowers computational cost while maintaining long-range dependency modeling capability. For our complete AS-Split Conformer–Mamba model, although the AS-Split structure introduces a small number of extra parameters, the total parameter count remains at 30.1 M, nearly identical to the Conformer baseline. Moreover, FLOPs decrease to 55.8 G and RTF to 0.068, representing reductions of 12.0% and 16.0%, respectively, compared to the baseline. Combined with the ablation results in Table 5, the complete model achieves the best recognition performance while still being more efficient in inference than the baseline. These results suggest that replacing the Transformer FFN with the Mamba block provides a more efficient sequence modeling component, which contributes to improved recognition performance while maintaining lower computational cost and higher inference throughput.

To analyze the source of performance improvement, we performed a probe analysis on the feature representations and an error type analysis on the test set. The results are summarized in Table 7.

Table 7. Diagnostic analysis: Feature preservation and error reduction.

Metric	Conformer	AS-Split Conformer–Mamba	Δ
Probe Analysis (Local Acoustic Preservation)
Layer 3 Accuracy (%)	92.5	95.3 †	+2.8 pp
Layer 6 Accuracy (%)	91.8	94.9 †	+3.1 pp
Layer 9 Accuracy (%)	87.5	93.6 †	+6.1 pp
Layer 12 Accuracy (%)	82.9	92.8 †	+9.9 pp
Degradation Rate (pp/layer)	1.07	0.28	−0.79
Error Analysis
Insertion Errors (%)	0.83	0.52 †	−0.31 pp
Deletion Errors (%)	1.47	1.01 †	−0.46 pp
Substitution Errors (%)	3.42	3.30	−0.12 pp
Total Errors (%)	5.72	4.83 †	−0.89 pp

Notes: Bold text indicates the best performance in the corresponding column. Probe: Linear classifier predicting initial consonant category from frozen encoder features; higher accuracy = better local acoustic preservation. Error: Insertion = incorrectly added; Deletion = missed; Substitution = incorrectly recognized. pp = percentage points. ^† indicates p < 0.05 (bootstrap test, 10,000 resamples). AS-Split degrades at 0.28 pp/layer vs. 1.07 pp/layer for Conformer—73.8% better preservation. Insertion + deletion account for 86.5% of total error reduction.

Specifically, after training, we extracted hidden representations from the encoder’s layers 3, 6, 9, and 12, and froze the encoder parameters. We then constructed a linear probe classifier using a single-layer fully connected network to predict the initial consonant category of each speech frame. Initial consonants were selected as the probing target because they contain short-duration spectral transitions and fine temporal structures, making them highly sensitive indicators of local acoustic representation quality. A simple linear classifier’s accuracy on this task reflects how much local acoustic information is preserved in the frozen encoder features. Because the probe model is simple, its classification accuracy reflects the amount of local acoustic information preserved in the hidden representations. The results show that the two models have similar classification performance in shallow layers, both effectively extracting low-level acoustic features. However, as network depth increases, the probe accuracy of the standard Conformer steadily declines, from 92.5% at layer 3 to 82.9% at layer 12, indicating that local acoustic discriminative information in deep features gradually diminishes. In contrast, AS-Split Conformer maintains a high classification accuracy throughout the encoding process. Even at layer 12, its probe accuracy remains at 92.8%, significantly higher than that of the standard Conformer. This suggests that the proposed local modeling stage sufficiently preserves acoustic details. The AS-Split Conformer degrades at only 0.28 percentage points per layer, compared to 1.07 percentage points per layer for the standard Conformer—a 73.8% improvement in local acoustic information preservation.

We further counted the proportions of insertion errors, deletion errors, and substitution errors on the test set. Compared to the standard Conformer, our method reduces the insertion error rate by 37.35% and the deletion error rate by 31.29%, while the substitution error changes relatively little. This result indicates that the AS-Split structure effectively enhances the preservation of local acoustic boundary information, improving the boundary discrimination accuracy during CTC monotonic alignment, and thus significantly reduces insertion and deletion errors. Insertion and deletion errors together account for 86.5% of the total error reduction, suggesting that improved local feature preservation and alignment supervision are important contributors to the observed performance gains. Combined with the recognition results in Table 5, the improved local acoustic information preservation correlates well with the observed CER reduction, indicating that the AS-Split structure not only enhances feature representation quality but also provides clearer acoustic boundary information for CTC monotonic alignment, effectively reducing insertion and deletion errors and improving speech recognition performance.

5.2. Parameter Sensitivity

To further validate the rationale behind the key parameter settings in our proposed AS-Split Conformer–Mamba model, we conducted parameter sensitivity experiments on the AISHELL-1 dataset. During these experiments, all other network structures, training strategies, and decoding parameters were kept fixed, and only the parameter under analysis was varied. Specifically, we examined the impact of (1) the local–global stage split ratio, (2) the intermediate CTC auxiliary supervision weight, and (3) the Mamba state dimension on recognition performance, using CER and SER as evaluation metrics. All results were obtained on the validation set for parameter selection.

5.2.1. Analysis of Local–Global Stage Split Ratio

The AS-Split structure is the core improvement of our model. Unlike the standard Conformer, where convolutional and MHSA modules are interleaved within each layer, AS-Split divides the encoder into a local modeling stage (responsible for local acoustic feature extraction) and a global modeling stage (responsible for long-range context modeling). To study the effect of different stage split ratios, we kept the total number of encoder layers fixed at 12. We experimented with different $K/L$ configurations, where K is the number of layers in the local stage and L is the number of layers in the global stage. The results are shown in Table 8.

Table 8. Model validation-set performance under different K/L configurations without attention rescoring.

$\mathit{K}/\mathit{L}$ Configuration	CER (%)	SER (%)
2/10	5.3	36.4
4/8	5.0	35.1
6/6	5.1	35.7
8/4	5.2	36.3

The stage split ratio was selected based on the validation-set CER. The test set was not used for hyperparameter tuning.

As shown in Table 8, when the number of local-stage layers increases from two to four, CER drops from 5.3% to 5.0% and SER from 36.4% to 35.1%, indicating a clear improvement. This suggests that appropriately increasing the depth of the local modeling stage allows the convolutional modules to more fully extract local time–frequency features from the speech signal, providing more stable feature representations for subsequent global modeling. However, when the local stage is further increased to six or eight layers, performance begins to degrade. This is because, as the local stage occupies a larger proportion, the number of layers in the global stage correspondingly decreases, weakening the ability of the MHSA modules to model long-range contextual information. ASR relies not only on local acoustic information but also on cross-word and cross-sentence semantic context; excessive local modeling limits the model’s ability to learn global dependencies. We emphasize that the SER values reported in this sensitivity analysis are higher than the final test-set SER (20.5% in Table 4) because they are measured on the validation set using greedy decoding without attention rescoring. The purpose of this experiment is to compare the relative trends across different split ratios rather than to report absolute performance. The consistent CER and SER trends across configurations confirm the robustness of our stage-wise design.Considering the trade-off, $K=4$, $L=8$ achieves the best balance between local feature extraction and global context modeling, and this configuration was adopted as the optimal stage split for AS-Split Conformer.

5.2.2. Analysis of Intermediate CTC Auxiliary Supervision Weight

To enhance the alignment capability between shallow features and label sequences, we introduced an intermediate CTC auxiliary supervision branch at the output of the AS-Split module. This intermediate CTC provides additional supervision signals to the intermediate layers of the encoder, mitigating the difficulty of gradient propagation in deep networks. To analyze the effect of the auxiliary supervision weight, we varied the CTC weight coefficient $\lambda$ while keeping all other parameters fixed. The results are shown in Table 9.

Table 9. Model performance under different intermediate CTC weights.

$\mathit{\lambda }$	CER (%)	SER (%)
0.1	5.4	26.0
0.2	5.1	22.5
0.3	5.0	21.5
0.4	5.1	22.0
0.5	5.3	23.2

The intermediate CTC weight $\lambda$ was selected based on the validation-set CER. The test set was not used for hyperparameter tuning.

As shown in Table 9, as $\lambda$ increases from 0.1 to 0.3, model performance steadily improves. At $\lambda =0.3$, we obtain the lowest CER (5.0%) and SER (21.5%). An appropriate CTC weight strengthens the correspondence between shallow features and text labels, helping the model learn stable monotonic alignment paths and accelerating convergence. However, further increasing the CTC weight degrades performance. This is because the CTC loss focuses primarily on local alignment relationships, while the attention mechanism is more adept at learning global contextual information. When the CTC weight is too large, the model over-optimizes for local alignment, weakening the attention branch’s ability to learn long-range dependencies, thereby reducing overall recognition performance. Considering both convergence and final accuracy, we set $\lambda =0.3$ as the optimal intermediate CTC auxiliary supervision weight.

5.2.3. Analysis of Mamba State Dimension

The Mamba module uses a selective state-space model to model long-range dependencies. Its state dimension, which determines the storage capacity for historical information, is one of the most important hyperparameters affecting model performance. To study the impact of the state dimension, we varied the state size while keeping other parameters fixed. The results are shown in Table 10.

Table 10. Model performance under different Mamba state dimensions.

${\mathit{d}}_{\mathbf{state}}$	Params (M)	CER (%)
64	28.5	5.4
128	29.2	5.0
256	30.1	4.8
512	34.7	4.7

Bold text indicates the best performance in the corresponding column. The Mamba state dimension was selected based on the accuracy–efficiency trade-off on the validation set rather than CER alone. The test set was not used for hyperparameter tuning. Parameter counts were re-calculated using the same counting script for consistency.

As shown in Table 10, as the state dimension increases, recognition performance improves. Increasing from 64 to 256 reduces CER from 5.4% to 4.8%, indicating that a larger state space retains richer historical context and enhances long-range dependency modeling. However, further increasing to 512 yields only a minor performance gain (0.1% absolute CER reduction) while substantially increasing parameter count and computational cost. This suggests that beyond a certain point, the model already captures most of the relevant contextual information, and further increases bring diminishing returns. Therefore, considering the trade-off between recognition performance and computational efficiency, we selected $d_{\mathrm{state}}=256$ as the default configuration for the Mamba module.

As shown in Figure 9, the traditional Conformer model exhibits relatively slow loss reduction and maintains the highest loss throughout training, indicating limited feature learning efficiency and convergence speed. In contrast, the AS-Split Conformer, with its staged encoding structure, converges significantly faster, with a loss curve consistently below that of the baseline Conformer. This suggests that the local–global staged modeling mechanism more effectively extracts and preserves local acoustic features, thereby improving training efficiency. When the Mamba state-space module is introduced into the Conformer (Conformer–Mamba), the loss reduction rate further increases. This indicates that Mamba enhances long-range context modeling through its selective state update mechanism, allowing the model to learn sequence dependencies more rapidly and obtain better feature representations. The proposed AS-Split Conformer–Mamba achieves the fastest convergence and the lowest final loss, confirming the synergistic benefits of staged encoding and Mamba-based decoding.

Figure 9. Training loss curves of Conformer, AS-Split Conformer, Conformer–Mamba, and the proposed AS-Split Conformer–Mamba model on the AISHELL-1 training set over 100 epochs. The proposed model achieves the fastest convergence and lowest final loss.

6. Conclusions

In this paper, we propose an end-to-end speech recognition model based on the AS-Split Conformer–Mamba architecture to alleviate the limitations of the standard Conformer’s interleaved convolution–attention design, which tends to dilute fine-grained acoustic information in deeper layers. The encoder is explicitly decomposed into three sequential components: a local stage with Squeeze-and-Excitation (SE)-enhanced convolutions, a TFB that serves as an adaptive transition module, and a global stage with multi-head self-attention and temporal convolutional networks. This design effectively decouples local and global feature learning. An intermediate CTC supervision branch is further introduced to reinforce monotonic alignment in shallow layers, while the Transformer–Mamba decoder replaces the feed-forward network with a Mamba block that achieves linear complexity within its state-space computation, while the retained attention modules handle acoustic–text alignment.

Compared with the baseline Conformer, the proposed model consistently achieves superior recognition performance across all benchmark datasets. On AISHELL-1, the Character Error Rate (CER) was reduced from 5.7% to 4.8%, corresponding to a relative reduction of 15.8%. Similar gains were observed on THCHS-30 and ST-CMDS, confirming the strong generalization capability of the proposed approach. Ablation studies validated that each component—the stage-wise encoder, TFB, intermediate CTC, and Mamba decoder—contributes progressively to the overall performance. Linear probe analysis further revealed that the AS-Split structure preserves local acoustic information substantially better in deeper layers, which is consistent with the significant reductions in insertion and deletion errors.

In conclusion, the proposed framework effectively integrates local acoustic details with global semantic context, achieving superior recognition accuracy while maintaining inference efficiency. Future work will explore lightweight architectures, streaming recognition, self-supervised pre-training, and multilingual extension, potentially incorporating models such as wav2vec 2.0 [39] and HuBERT [40] to further enhance cross-domain adaptability.