SRC-IT2: Speech Rate-Controllable Mongolian Emotional Speech Synthesis Based on Improved Tacotron2

Abstract

To address the challenges of slow synthesis speed, unstable quality, limited emotional expressiveness, and the lack of controllable speaking rate in Mongolian emotional speech synthesis, this paper proposes a speech Rate-Controllable Mongolian emotional speech synthesis model based on improved Tacotron2 (SRC-IT2). First, an end-to-end Mongolian speech synthesis module is constructed based on an improved Tacotron2 framework, incorporating the unique linguistic characteristics of the Mongolian script. The front-end processing is optimized accordingly, and a G2P-Seq2Seq model is employed to achieve accurate grapheme-to-phoneme conversion for Mongolian characters. Next, on top of the end-to-end synthesis framework, a joint text-audio emotion analysis module is integrated to effectively learn and represent emotional style features specific to Mongolian speech. Finally, a style encoder and speaking rate control variable are embedded into the acoustic modeling process, further enhancing Tacotron2’s ability to dynamically adjust the speaking rate during emotional speech generation. Experimental results demonstrate that the proposed model produces more natural-sounding speech with improved emotional expressiveness and enables effective real-time control over speaking rate in Mongolian emotional speech synthesis.

1. Introduction

Speech synthesis [1], as one of the core technologies in the field of human-computer voice interaction, primarily aims to convert input text into natural and fluent speech. It has found wide-ranging applications in human communication. Speech synthesis technology is inherently interdisciplinary, combining phoneme conversion rules from linguistics, acoustic modeling principles from phonetics, digital encoding techniques from signal processing, and neural network architecture optimization from machine learning. This integration enables more intelligent and user-friendly voice interaction experiences [2].

Mongolian, as one of China’s minority languages, differs significantly from mainstream languages such as Mandarin Chinese. It exhibits unique characteristics in terms of expression style, prosody, and emotional coloration, which pose additional challenges for emotional speech synthesis. Moreover, emotional expression in Mongolian is highly nuanced and polysemous—variations in intonation, stress, and speech mood often convey rich emotional information. In recent years, notable progress has been made in Mongolian speech synthesis research. However, compared to mainstream languages like Mandarin, Mongolian speech synthesis remains a relatively underexplored area, with higher research difficulty and several unresolved technical challenges in emotional synthesis. Key research difficulties include how to effectively express emotions in synthesized Mongolian speech, how to control the speaking rate of emotional speech, and how to differentiate between various emotional styles through speech synthesis.

Therefore, the study of Mongolian emotional speech synthesis remains of significant importance. On one hand, such technology can play a vital role in education, cultural preservation, and language learning, enabling people in regions where Mongolian is the primary language to better acquire and use their native language. On the other hand, integrating Mongolian emotional speech synthesis into applications such as smart home systems and voice assistants can help address real-world needs in these regions, significantly enhancing quality of life and promoting the advancement of regional intelligent infrastructure.

To address the aforementioned challenges, this paper proposes a speech Rate-Controllable Mongolian emotional speech synthesis model based on improved Tacotron2 (SRC-IT2). The main contributions of this work are as follows:

• An end-to-end Mongolian speech synthesis model is constructed based on an improved Tacotron2 architecture. It incorporates the unique linguistic characteristics of Mongolian, optimizes the front-end processing module, and utilizes a G2P-Seq2Seq model to convert Mongolian characters into phonemes effectively.
• Building upon the end-to-end architecture, a joint emotion analysis module for both text and speech is embedded, allowing the model to learn optimal emotional style features specific to Mongolian.
• A style encoder and speech rate control variable are integrated into the acoustic modeling process, further optimizing the Tacotron2-based synthesis system to enable dynamic adjustment of speaking rate in Mongolian emotional speech generation.

2. Related Work

In recent years, with the rapid advancement of artificial intelligence technologies, deep learning has driven transformative progress across various domains. Speech synthesis techniques based on deep learning have increasingly attracted significant attention from both academia and industry.

2.1. Speech Synthesis

Jose Sotelo et al. proposed the Char2Wav model [3], which integrates all stages of speech synthesis into a unified end-to-end neural network framework, thereby avoiding the complex intermediate processes inherent in traditional speech synthesis approaches.

Microsoft introduced the FastSpeech model [4], which constructs a non-autoregressive generation framework based on a feed-forward Transformer architecture. By leveraging parallel computation, it significantly improves the generation speed of Mel-spectrograms, achieving a synthesis speed 38 times faster than baseline models. While FastSpeech made notable advancements in synthesis efficiency, it still suffered from issues such as Mel-spectrogram information loss and prolonged training time. To address these limitations, Microsoft Research proposed FastSpeech 2 [5], which improved upon its predecessor by eliminating the teacher-student framework and directly training the model with ground-truth Mel-spectrograms. Additionally, it introduced multiple prosodic features such as pitch and energy to enhance the model’s capacity for prosody modeling.

Microsoft Research Asia and the Azure Speech Team released the NaturalSpeech model [6], which enhanced the modeling of text priors while reducing the complexity of speech posteriors, significantly improving both naturalness and clarity of the synthesized speech.

Yao et al. proposed SR-TTS [7], a Transformer-based optimized architecture composed of an encoder, a prosody regulator, and a decoder. This model not only improves the quality and naturalness of synthesized speech but also accelerates the synthesis process and improves inference efficiency, outperforming existing approaches in both speech quality and expressiveness. In the same year, Guo et al. introduced FLY-TTS [8], a fast, lightweight, and high-quality end-to-end speech synthesis model. By replacing the decoder with a ConvNeXt module, the model size was significantly reduced. Furthermore, WavLM was utilized as a pretrained model for adversarial training, which effectively enhanced the quality and naturalness of the generated speech.

2.2. Emotional Speech Synthesis

Currently, mainstream approaches to emotional speech synthesis typically rely on large, emotion-specific datasets for each individual emotion category. Zhou et al. [9], for example, utilized the IEMOCAP dataset to train an emotion recognition model. They then extracted emotional parameters for each category using this model and employed a Wasserstein Generative Adversarial Network (W-GAN) to transform neutral speech into emotionally expressive utterances.

Another straightforward method for emotional speech synthesis involves training synthesis models on manually labeled emotional corpora, using emotion labels as conditional inputs to guide the generation of speech with the corresponding emotional expression [10]. However, this method often lacks the flexibility to adapt emotional expressiveness within the same emotion category based on different textual content. To overcome this limitation, Cai et al. [11] proposed an emotion-aware speech synthesis approach that does not require explicit emotion labels. Their method integrates transfer learning, an emotion recognition model, and a speech synthesis model to generate speech with specific emotional characteristics.

Li et al. [12] introduced DiCLET-TTS, a diffusion-based cross-lingual emotional transfer model that enables the transfer of emotional style from a source speaker to a target speaker, either within the same language or across different languages. Cho et al. [13] proposed EmoSphere-TTS, which controls the emotional style and intensity of synthesized speech using spherical emotion vectors. Without any manual annotations, it leverages spherical Cartesian transformation to simulate emotional dimensions. They also introduced a dual-conditional adversarial network, which improves the quality of synthesized speech by modeling diverse emotional attributes. Gudmalwar et al. [14] proposed VECL-TTS, a cross-lingual emotional speech synthesis model that supports both speaker identity and emotional style controllability. By extracting reference embeddings from the source language, VECL-TTS applies cross-lingual speech synthesis techniques to transfer both speaker characteristics and emotional expressions to the target language, thereby enhancing synthesis quality.

2.3. Mongolian Emotional Speech Synthesis

With the rapid advancement of intelligent Mongolian language processing technologies, Mongolian speech synthesis has reached a practical level of application, achieving high-quality speech generation. As a result, an increasing number of researchers are focusing on more natural, emotionally expressive, and highly expressive Mongolian emotional speech synthesis. However, current research in this area is still in its early stages.

Shan Yu [15] proposed a Mongolian emotional speech synthesis model based on an end-to-end synthesis framework incorporating emotion adaptation techniques. This model introduced an emotion embedding layer that transforms emotion labels into vector representations, which are then concatenated with the outputs of the acoustic model, thereby improving training efficiency. Huang Aihong [16] enhanced the emotional controllability of the Tacotron2 model by integrating a reference encoder and a variational autoencoder, further improving the model’s ability to capture and generate emotional variations in speech. Liang et al. [17] proposed a lightweight Mongolian speech synthesis model based on deep convolutional neural networks, which significantly improved the naturalness and quality of synthesized Mongolian speech. The model also reduced training time while maintaining high synthesis accuracy.

3. Methods

3.1. Overview

The overall architecture of the proposed SRC-IT2 is illustrated in Figure 1. First, an end-to-end Mongolian speech synthesis module is constructed based on an enhanced Tacotron2 framework. To accommodate the unique linguistic characteristics of the Mongolian language, the front-end processing pipeline is optimized, and a G2P-Seq2Seq model is incorporated to enable accurate grapheme-to-phoneme conversion for Mongolian script. Second, a text-audio emotion analysis module is integrated into the synthesis framework, allowing the model to learn and generate representative emotional style features specific to Mongolian. Finally, a style encoder and a speech rate control variable are introduced to further improve the acoustic modeling capabilities of Tacotron2. This enhancement enables dynamic control over the speaking rate of synthesized Mongolian emotional speech, thereby achieving controllable emotional speech synthesis with respect to speaking rate.

3.2. Mongolian Speech Synthesis Based on Improved Tacotron2

Speech synthesis is a technology that converts input text into corresponding speech output. The synthesis process primarily consists of three critical components: front-end processing, acoustic modeling, and audio synthesis. The front-end processing module is responsible for analyzing the input text and extracting relevant linguistic features. The acoustic model maps these linguistic features to acoustic representations, while the audio synthesis module synthesizes the final speech waveform from the predicted acoustic features.

3.2.1. Front-End Processing

Before converting Mongolian script into phonemes, the original Mongolian characters are first transliterated into their corresponding Latin representations. Subsequently, a G2P-Seq2Seq model is employed to transform the Latin characters into the corresponding phoneme sequences. The G2P-Seq2Seq model is a Transformer-based grapheme-to-phoneme conversion model. Compared with traditional sequence-to-sequence (Seq2Seq) models based on RNN or LSTM, the Transformer architecture leverages self-attention mechanisms to capture dependencies between input and output elements more efficiently and accurately. This enables the model to achieve higher performance in sequence transformation tasks such as grapheme-to-phoneme conversion.

Specifically, the input Latin character sequence X is first passed through an embedding layer to obtain the corresponding character embeddings E. The encoder processes E using a bidirectional LSTM, where the forward and backward hidden states are denoted as $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$, respectively. The final encoded representation is computed as:

$$h_i=concat\left(\overrightarrow{h_i},\overleftarrow{h_i}\right)$$

(1)

where $i\in \left[1,n\right]$, n denotes the length of the input Latin character sequence X, and $concat\left(·\right)$ denotes the concatenation operation.

At time step t, the decoder computes the attention weights ${\alpha }_{t,i}$ between the input sequence and the current decoding state via an attention mechanism.

$${\alpha }_{t,i}={\displaystyle \frac{exp\left(a\left(s_{t-1},h_i\right)\right)}{{\sum }_{j=1}^{n}exp\left(a\left(s_{t-1},h_j\right)\right)}}$$

(2)

Here, $a\left(s_{t-1},h_i\right)$ denotes the attention scoring function, where $s_{t-1}$ represents the decoder’s hidden state at the previous time step.

The context vector $c_t$ is computed as:

$$c_t=\sum _{i=1}^n{\alpha }_{t,i}·h_i$$

(3)

The G2P-Seq2Seq model adopts a Transformer architecture, which effectively avoids the issues of vanishing and exploding gradients commonly encountered in traditional recurrent neural networks. This significantly enhances both the training efficiency and accuracy of the model, thereby improving the naturalness and precision of speech synthesis.

3.2.2. Acoustic Modeling

In the proposed approach, acoustic modeling is conducted based on an improved Tacotron2 architecture. The spectrogram prediction network adopts an encoder–attention–decoder structure, which primarily aims to convert the input text sequence into a frame-level Mel-spectrogram representation.

Specifically, the input character sequence is first encoded into character embeddings E through an embedding layer. Then, local features are extracted using a three-layer convolutional module $Con{v}^3$, and the resulting features are passed through a bidirectional LSTM to generate the encoded sequence:

$$H=BiLSTM\left(Con{v}^3\left(E\right)\right)$$

(4)

The decoder predicts the Mel-spectrogram frames sequentially from the encoded sequence H, generating one frame at a time. The previously predicted spectrogram frame $\widehat{m_{t-1}}$ is first fed into a two-layer fully connected pre-net, and the output of the pre-net at time step t, denoted as $p_t$, is computed as:

$$p_t=ReLU\left(W_{p_2}·ReLU\left(W_{p_1}·\widehat{m_{t-1}}+b_{p_1}\right)+b_{p_2}\right)$$

(5)

where $W_{p_1}$ and $W_{p_2}$ are weight matrices, and $b_{p_1}$ and $b_{p_2}$ are bias terms.

Subsequently, $p_t$ is concatenated with the attention context vector, and the combined representation is passed through a unidirectional LSTM network. The resulting output is then concatenated again with the attention context vector and projected through a linear transformation to obtain the predicted spectrogram frame $\widehat{m_t}$:

$$\widehat{m_t}=W_m·\left[s_t;c_t\right]+b_m$$

(6)

Finally, the predicted spectrogram frame is passed through a post-net, which estimates a residual that is added to the initial spectrogram frame (before convolution), thereby refining the overall spectrogram reconstruction process and producing the final predicted Mel-spectrogram.

3.2.3. Audio Synthesis

HiFi-GAN is a vocoder based on Generative Adversarial Networks (GANs). Its generator enhances speech quality through an upsampling architecture combined with Multi-Receptive Field (MRF) modules, while the discriminator improves the ability to distinguish between real and synthesized audio using Multi-Scale Discriminators (MSD) and Multi-Period Discriminators (MPD).

As illustrated in Figure 2, the HiFi-GAN vocoder adopts a modular architecture for its generator, which primarily consists of two components: an upsampling structure and a MRF module. The generator performs upsampling of the Mel-spectrogram through one-dimensional transposed convolution layers, mapping low-temporal-resolution spectral features into high-resolution speech waveforms. Given the input Mel-spectrogram features $Mel$, after N layers of transposed convolution operations, the output speech waveform $\widehat{y}$ is obtained. The core mapping relationship can be expressed as:

$$\widehat{y}=Generator\left(Mel\right)=ConvTranspos{e}^N\left(Mel\right)$$

(7)

The MRF module integrates information from multiple receptive fields, facilitating multi-scale feature fusion. This allows the generator to capture hierarchical acoustic patterns more effectively, thus enhancing the quality and naturalness of the synthesized speech.

The discriminator of the HiFi-GAN vocoder, as illustrated in Figure 3, is primarily designed to distinguish whether the input audio originates from the real dataset or is synthesized by the generator. It comprises two components: the MSD and the MPD. The discriminator adopts a combined architecture of MSD and MPD, where adversarial training is achieved by minimizing the generator loss and maximizing the discriminator loss. The generator loss function ${\mathcal{L}}_G$ consists of the adversarial loss ${\mathcal{L}}_A$ and the feature matching loss ${\mathcal{L}}_{FM}$, formulated as follows:

$${\mathcal{L}}_G={\mathcal{L}}_A+{\lambda }_{FM}·{\mathcal{L}}_{FM}$$

(8)

Here, ${\lambda }_{FM}$ denotes the weighting coefficient of the feature matching loss, which is employed to balance the stability of adversarial training with the quality of the synthesized speech.

Traditional discriminators evaluate audio at a single fixed scale, whereas the MSD employs a multi-scale discrimination architecture that assesses audio waveforms across multiple temporal resolutions. This enables the capture of subtle variations within the audio, thereby enhancing the discriminator’s ability to differentiate between real and generated speech. The MPD focuses on modeling the periodic characteristics of the audio signal. By performing discrimination at various periodic scales, it helps the discriminator accurately identify the periodic structure inherent in natural speech, improving recognition accuracy and guiding the generator to produce more natural and coherent audio outputs.

3.3. Mongolian Emotional Speech Synthesis Based on Improved Tacotron2

3.3.1. Text Sentiment Analysis

In the textual emotion analysis module, XLNet is first utilized to obtain context-aware word embeddings that capture semantic dependencies within the input text. Subsequently, a Bidirectional Gated Recurrent Unit (BiGRU) is employed to extract global textual features. An attention mechanism is then applied to dynamically adjust the weights of the extracted features—assigning higher weights to features with greater emotional relevance and lower weights to less informative ones. Finally, a softmax classifier is used to classify the refined emotional features, yielding the final sentiment prediction. Figure 4 illustrates the architecture of the text emotion analysis module.

The XLNet model integrates the advantages of both autoregressive and autoencoding approaches, enabling it to capture richer contextual information and long-range dependencies within sequences. It incorporates two key pretraining tasks. The first is the permutation-based language modeling task, in which the input tokens are reordered according to all possible permutations. The model predicts each token based on these permutations, thereby learning to extract contextual information under various token orderings. The second is the application of the self-attention mechanism, which allows the model to effectively learn the dependencies between all tokens in the input sequence, thereby improving its ability to model context. Compared to other pretrained models, XLNet has demonstrated superior performance in tasks such as text classification and sentiment analysis.

The BiGRU extends the traditional GRU by processing the input sequence in both forward and backward directions. While the standard GRU captures only forward dependencies, BiGRU consists of two independent GRUs—one operating in the forward direction and the other in reverse—thereby leveraging both past and future contextual information for more comprehensive sequence modeling. The architecture of BiGRU is illustrated in Figure 5.

In BiGRU, each GRU unit utilizes an update gate and a reset gate to regulate the flow of information. The update gate determines how much of the current state should be updated, while the reset gate controls the extent to which the previous state should be discarded.

For a given input sequence $X=x_1,x_2,...,x_T$, the computation of the BiGRU is defined as follows:

$$\overrightarrow{h_t}=GRU\left(\overrightarrow{h_{t-1}},x_t\right)$$

(9)

$$\overleftarrow{h_t}=GRU\left(\overleftarrow{h_{t-1}},x_t\right)$$

(10)

Here, $\overrightarrow{h_t}$ and $\overleftarrow{h_t}$ denote the hidden states at time step i generated by the forward and backward GRU, respectively. The function $GRU\left(·\right)$ represents a nonlinear transformation applied to the input word vector, encoding it into the corresponding GRU hidden state, and $x_t$ is the $t-th$ element of the input sequence.

The hidden states from both directions are then concatenated to form the final hidden representation $h_t$. Finally, $h_t$ is passed through a fully connected layer to produce the output $y_t$.

$$y_t=Softmax\left(W{h}_t+b\right)$$

(11)

3.3.2. Audio Sentiment Analysis

In audio-based emotion analysis, emotional features are first extracted from the audio signal using an emotion encoder. These features are then passed to an emotion classifier, which performs the final classification to determine the corresponding emotional category.

As shown in Figure 6, the emotion encoder consists of a CNN followed by a BiLSTM layer, which collaboratively generate the emotional feature representations. The emotion classifier comprises a fully connected layer and a softmax layer, with the final emotional category predicted through the output of the softmax layer.

Figure 7 illustrates the structure of the BiLSTM network, which simultaneously processes sequential data in both forward and backward directions using forward and backward LSTM layers. This enables the model to capture bidirectional contextual information and learn more complex dependency structures. Such capabilities significantly enhance the model’s expressiveness, particularly in tasks involving long sequential data. In addition, the BiLSTM network demonstrates strong robustness, as it does not rely heavily on word embeddings or handcrafted features. Instead, it can effectively learn rich and informative feature representations directly from the data.

3.4. Speech Rate-Controllable Mongolian Emotional Speech Synthesis Based on Improved Tacotron2

Building upon the improved Tacotron2-based Mongolian emotional speech synthesis model, a style encoder and a speech rate control variable are incorporated to construct a speech Rate-Controllable Mongolian emotional speech synthesis model. The style encoder is primarily designed to extract acoustic features—such as pitch, intonation, and emotional attributes—from the Mel-spectrogram of emotional speech. This allows the model to retain other prosodic and acoustic characteristics while dynamically adjusting speech rate, thereby minimizing undesirable alterations to non-temporal attributes. The speech rate variable, on the other hand, is responsible for explicitly controlling the speaking rate of the synthesized speech, enabling the generation of both fast-paced and slow-paced emotional utterances. The enhanced model is thus capable of adjusting speaking rate dynamically while preserving the integrity of other speech attributes as much as possible.

3.4.1. Style Encoder

The style encoder consists of six convolutional layers, each followed by batch normalization and a ReLU activation function. A GRU layer is connected after the convolutional layers to further capture temporal dependencies in the encoded features.

As illustrated in Figure 8, the architecture of the style encoder takes the Mel-spectrogram of emotional speech as input. The input is passed through six convolutional layers to extract local features across different dimensions of the spectrogram. Subsequently, a GRU is employed to capture the temporal dependencies within the spectrogram sequence. The style encoder is capable of learning acoustic features such as pitch, intonation, and emotion types from the Mel-spectrogram. Batch Normalization (BN) is an optimization technique in deep learning that helps mitigate issues such as vanishing or exploding gradients during model training. By normalizing the mini-batch inputs to each layer, BN alleviates internal covariate shift, thereby improving the training stability and convergence speed of the model.

During the execution of BN, for a given input mini-batch x, the mean $\mu$ and variance ${\sigma }^2$ of each feature are first computed as follows:

$${\mu }_B={\displaystyle \frac{1}{m}}\sum _{i=1}^m{x}_i$$

(12)

$${\sigma }_B^2={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\left(x_i-{\mu }_B\right)}^2$$

(13)

Here, m denotes the mini-batch size, and $x_i$ represents the individual input data sample.

Subsequently, the input data are normalized using the computed mean and variance:

$$\widehat{x_i}={\displaystyle \frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+\epsilon }}}$$

(14)

where $\epsilon$ is a small constant introduced to prevent division by zero.

Finally, a linear transformation is applied:

$$y_i=\gamma \widehat{x_i}+\beta$$

(15)

where $\gamma$ is used to scale the normalized data, and $\beta$ is employed to shift the normalized data.

3.4.2. Speech Rate Variable

Common methods for measuring speech rate include Words Per Minute (WPM), Syllables Per Minute (SPM), and Words Per Silence (WPS). Among these, WPM is the most widely adopted metric, calculated by dividing the total number of words by the overall speech duration. In the field of phonetics, however, SPM is more frequently used because syllable counts offer a more fine-grained representation of speech structure. SPM is computed by dividing the total number of syllables by the total duration of the utterance. Subsequent studies have also introduced WPS as a relevant metric, which is calculated by dividing the number of words by the number of pauses. This metric has been shown to correlate strongly with a speaker’s rhythmic patterns and fluency. Nonetheless, all of the above methods have certain limitations when compared to phoneme-based approaches for estimating speech rate, especially in contexts where phonemes serve as the primary modeling unit.

SRC-IT2 takes phonemes as the model’s input. Therefore, it is more appropriate to compute the speech rate based on phoneme-level information. Inverse Mean Duration (IMD) is commonly employed as a measure of phoneme duration characteristics. A higher IMD value indicates a faster speaking rate, while a lower IMD value corresponds to slower speech. Accordingly, this study adopts IMD to quantify the speech rate. The calculation of speech rate using IMD is formulated as follows:

$$r=\lambda {\displaystyle \frac{S}{T}}$$

(16)

where $\lambda$ is the scaling factor for the speech rate, S denotes the number of phonemes in the text sequence, and T represents the total duration of the audio.

4. Experiment

4.1. Dataset

The dataset employed in this study was constructed by the Artificial Intelligence Laboratory of Inner Mongolia University of Technology. It comprises a speech corpus covering seven discrete emotional categories: ‘happy’, ‘angry’, ‘sad’, ‘surprise’, ‘fear’, ‘disgusted’, and ‘neutral’. For each emotional category, 300 utterances were recorded, resulting in a total duration of approximately 2.25 h.

Table 1. The duration range of speech of different emotional categories.

Emotions	The Shortest Duration (Seconds)	The Longest Duration (Seconds)	Total Duration (Hours)
Happy	2.46	9.52	0.37
Angry	1.93	5.67	0.24
Sad	2.90	9.71	0.43
Surprise	1.88	5.50	0.26
Fear	1.84	8.52	0.30
Disgusted	2.02	5.85	0.28
Neutral	3.90	8.17	0.37

presents the duration range of emotional speech samples across different categories.

Prior to model training, a Mongolian text front-end processing method was applied to convert Mongolian script into its corresponding Latin-based sequences. Each processed text sequence was then aligned with its corresponding emotional speech audio file. The resulting dataset was randomly divided into training, testing, and validation sets according to an 8:1:1 ratio, yielding 1680, 210, and 210 samples, respectively, for each subset.

4.2. Experimental Environment and Evaluation Index

4.2.1. Experimental Environment

The detailed experimental environment configuration is summarized in

Table 2. Experimental environment configuration.

Experimental Environment	Related Configuration
Operating system	Ubuntu 18.04.5 LTS
GPU	Nvidia Tesla P100
CPU	Intel® Xeon® Gold 6310 CPU @ 2.10GHz
Memory	64G
Python	3.9
CUDA	11.2

.

The hyperparameters are shown in

Table 3. Experimental hyperparameters.

Hyperparameters
Initial Learning Rate	1 × ${10}^{-4}$
Epochs	200
Optimizer	Adam
Batch Size	64

.

4.2.2. Evaluation Index

In this study, the Mean Opinion Score (MOS) metric was adopted to evaluate the quality of synthesized speech. Additionally, accuracy was used to assess the emotion recognition performance of speech rate perception.

The MOS is a widely used subjective evaluation method in speech synthesis. Therefore, this study employs MOS to conduct subjective assessments of the speech synthesis model. In the MOS evaluation, a group of listeners is invited to rate the synthesized speech samples on a five-point scale, where 1 indicates poor quality and 5 indicates excellent quality. Generally, a MOS score below 3 is considered unacceptable in terms of speech quality, while a score above 4 is regarded as indicating high-quality speech. Assuming M evaluators assess N synthesized speech samples, the MOS is calculated as follows:

$$MOS={\displaystyle \frac{1}{M}}{\sum }_{j=1}^M\left({\displaystyle \frac{{\sum }_{i=1}^{N}Scor{e}_{ij}}{N}}\right)$$

(17)

4.3. Sentiment Classification Experiment of Speech Speed Perception

As shown in

Table 4. The recognition accuracy statistics of emotional speech synthesized at different speech speeds.

Emotions	Fast	Medium	Slow
Happy	0.72	0.69	0.73
Angry	0.87	0.82	0.80
Sad	0.80	0.71	0.74
Surprise	0.86	0.79	0.85
Fear	0.84	0.77	0.82
Disgusted	0.74	0.62	0.69
Neutral	0.84	0.78	0.83
Average Accuracy	0.81	0.74	0.78

, the proposed model demonstrates a good ability to distinguish between the three speech rates. From the perspective of emotional categories, the recognition accuracies of ‘angry’, ‘surprise’, ‘fear’, and ‘neutral’ emotions at all three speech rates are higher than the average recognition accuracy for each corresponding rate. Specifically, the recognition accuracies for anger at fast and medium speech rates are the highest, reaching 0.87 and 0.82, respectively. The ‘surprise’ category achieves the highest recognition accuracy of 0.85 at the slow speech rate. Conversely, ‘disgusted’ yields the lowest recognition accuracies at medium and slow speech rates, at 0.62 and 0.69, respectively. The ‘happy’ emotion achieves its lowest recognition accuracy (0.72) at the fast speech rate. From the perspective of speech rate, the average recognition accuracy across all seven emotion categories is lowest at the medium speech rate (0.74), whereas the fast and slow rates yield higher average accuracies, 0.81 and 0.78, respectively. This indicates that fast and slow emotional speech samples are more distinguishable and perceptually salient than those at a medium speech rate.

Overall, the experimental results show slight variations in synthesis performance across different speech rates. The ‘angry’, ‘surprise’, ‘fear’, and ‘neutral’ categories demonstrate consistently strong synthesis quality at fast, medium, and slow rates. In contrast, ‘happy’, ‘sad’, and ‘disgusted’ show better synthesis performance at fast and slow rates compared to medium. These findings also suggest that the model exhibits varying learning capabilities across different speech rate conditions.

4.4. Emotional Speech Synthesis Quality Comparison Experiment

To evaluate the quality of emotional speech synthesis, a panel of 30 native Mongolian-speaking listeners was invited to serve as professional evaluators. A blind and independent listening test was conducted, in which participants rated the synthesized speech samples. The MOS was then calculated as the arithmetic mean of all valid scores. The evaluators independently rated audio samples synthesized by three baseline models—FastSpeech2 [5], Deep Voice3 [18], and VAE-Tacotron2 [19]—as well as the proposed SRC-IT2 model presented in this study.

The results presented in

Table 5. MOS results of different emotional speech synthesis models.

Emotions	FastSpeech2	Deep Voice3	VAE-Tacotron2	SRC-IT2
Happy	3.45 ± 0.10	3.58 ± 0.10	3.65 ± 0.10	3.67 ± 0.10
Angry	3.58 ± 0.10	3.69 ± 0.10	3.71 ± 0.10	3.83 ± 0.10
Sad	3.44 ± 0.10	3.60 ± 0.10	3.56 ± 0.10	3.63 ± 0.10
Surprise	3.60 ± 0.10	3.74 ± 0.10	3.64 ± 0.10	3.73 ± 0.10
Fear	3.67 ± 0.10	3.67 ± 0.10	3.58 ± 0.10	3.69 ± 0.10
Disgusted	3.42 ± 0.10	3.55 ± 0.10	3.51 ± 0.10	3.58 ± 0.10
Neutral	3.62 ± 0.10	3.72 ± 0.10	3.69 ± 0.10	3.77 ± 0.10
Average	3.54 ± 0.10	3.65 ± 0.10	3.62 ± 0.10	3.70 ± 0.10

indicate that the proposed SRC-IT2 model achieves an average MOS score of 3.70, slightly outperforming the other three baseline models. The VAE-Tacotron2 and Deep Voice3 models yield comparable MOS scores of 3.62 and 3.65, respectively, while FastSpeech2 demonstrates the lowest performance among the four models with an average MOS score of 3.54. Notably, FastSpeech2 underperforms particularly in the emotional categories of ‘happy’, ‘disgusted’, and ‘sad’.

The SRC-IT2 model achieves the highest MOS scores in multiple emotional categories. Specifically, it outperforms all baseline models in the ‘happy’, ‘angry’, ‘sad’, ‘fear’, ‘disgusted’, and ‘neutral’ categories. Only in the ‘surprise’ category does it score slightly lower than Deep Voice3. Overall, the results suggest that the SRC-IT2 model delivers superior performance in emotional speech synthesis compared to the three baseline models.

4.5. SRC-IT2 Model Ablation Experiment

4.5.1. Ablation Experiment of Emotional Module

To evaluate the impact of emotional embeddings on the SRC-IT2 model, ablation experiments were conducted and categorized into two groups: A and B. In Group A, the audio-based emotion analysis module was removed to assess the contribution of the text-based emotion analysis module to the overall model performance. In Group B, the text-based emotion analysis module was excluded to examine the influence of the audio-based emotion analysis module.

As shown in

Table 6. MOS results of the ablation experiment.

Emotions	A	B	SRC-IT2
Happy	3.51 ± 0.10	3.53 ± 0.10	3.67 ± 0.10
Angry	3.65 ± 0.10	3.64 ± 0.10	3.83 ± 0.10
Sad	3.58 ± 0.10	3.48 ± 0.10	3.63 ± 0.10
Surprise	3.64 ± 0.10	3.62 ± 0.10	3.73 ± 0.10
Fear	3.68 ± 0.10	3.72 ± 0.10	3.69 ± 0.10
Disgusted	3.57 ± 0.10	3.45 ± 0.10	3.58 ± 0.10
Neutral	3.71 ± 0.10	3.69 ± 0.10	3.77 ± 0.10
Average	3.62 ± 0.10	3.59 ± 0.10	3.70 ± 0.10

, the complete model (SRC-IT2) achieved the highest average MOS score of 3.70. The scores for Group A and Group B were slightly lower, at 3.62 and 3.59, respectively. These results indicate that incorporating both text-based and audio-based emotion analysis modules contributes to improved emotional speech synthesis quality, resulting in more natural and expressive synthesized speech.

4.5.2. Ablation Experiments at Different Speech Rates

To evaluate the impact of different speech rates on the model’s performance, an ablation study was conducted. One emotional speech category was selected from the seven synthesized emotion types for spectrogram-based analysis at varying speech rates. Figure 9 presents the Mel-spectrograms of synthesized speech with different speech rates under the ‘angry’ emotion condition.

As observed from the Mel-spectrograms, noticeable differences exist among the synthesized emotional speech at various speaking rates. In terms of audio duration, the fast speech lasts approximately 3.6 s, the normal-speed speech lasts around 4 s, and the slow speech extends to approximately 4.5 s, which aligns with natural variations in speech rate. From the perspective of spectrogram structure, the fast speech Mel-spectrogram appears more compact, with clearly visible high-frequency components; however, due to the accelerated rate, the representation of fine-grained details is less prominent compared to the normal and slow speeds. The normal-speed Mel-spectrogram exhibits a relatively stable and evenly distributed spectral pattern, effectively capturing subtle emotional nuances. In contrast, the slow-speed Mel-spectrogram shows a more relaxed structure with a longer temporal span and smoother spectral transitions, thereby enhancing the expressiveness of emotional content.

5. Conclusions

To address the challenges of limited Mongolian emotional speech corpora, slow synthesis speed, unstable quality, insufficient emotional expressiveness, and lack of controllable speech rate, this study proposes the SRC-IT2 model. Under the constraints of a limited emotional speech corpus, the model employs various optimizations to enhance the quality of synthesized Mongolian emotional speech while allowing dynamic adjustment of speaking rate. Firstly, an end-to-end Mongolian speech synthesis module is constructed based on an improved Tacotron2 framework. The front-end processing pipeline is optimized to account for the unique characteristics of Mongolian script, and a G2P-Seq2Seq is used for grapheme-to-phoneme conversion. Secondly, text and audio emotion analysis modules are integrated into the synthesis framework to learn and represent Mongolian emotional style features effectively. Finally, a style encoder and a speech rate control variable are introduced into the acoustic modeling pipeline, enabling the model to dynamically adjust speaking rate while adhering to emotional expression. Experiments demonstrate that the proposed SRC-IT2 model can generate high-quality, expressive Mongolian speech even with limited training data, while exhibiting controllable speech rate and robust overall performance.

In future work, we plan to collect a more diverse Mongolian emotional speech corpus and explore prosody-driven dynamic speaking rate adjustment strategies to further enhance the naturalness of speech rate modulation.