Joint Modeling and KAFusion Feature Fusion for Prosody-Controllable Speech Synthesis

Abstract

To address the limited expressiveness in current speech synthesis caused by coarse-grained prosody modeling and simplistic feature fusion strategies, a joint prosody modeling framework and a nonlinear fusion method named KAFusion are proposed, based on the Kolmogorov–Arnold (KA) representation theorem. The joint modeling integrates pitch and energy as prosodic priors with text encodings to jointly guide duration prediction, enabling explicit control over speech rate and tone. During feature fusion, KAFusion facilitates nonlinear interactions among features through its nested inner and outer functions. Information entropy serves as the quantitative metric, and both theoretical and experimental results demonstrate the fusion module’s efficacy in suppressing redundancy while preserving task-critical content. Evaluations on the AISHELL3 dataset show a 5.8% improvement in MOS over the baseline. Ablation studies further validate the effectiveness of the proposed components, where KAFusion achieves an output entropy of 3.47, which is 18.4% higher than that of linear fusion (2.93) and indicates richer information content.

1. Introduction

Text-to-speech (TTS) aims to convert text into highly natural and easily understandable natural speech [1]. As one of the core technologies in fields such as human–computer interaction, media entertainment, and AIGC (Artificial Intelligence-Generated Content), early TTS mainly relied on parameter synthesis or concatenation methods, where the naturalness and fluency of the synthesized speech were difficult to match with those of real human speech. In recent years, synthetic models based on neural networks have brought revolutionary improvements to the naturalness and expressiveness of speech.

WaveNet [2], a landmark advancement in TTS, introduced an end-to-end deep learning framework for raw waveform generation, significantly enhancing the quality of synthesized speech. Building upon this, VITS [3] achieved notable progress in speech naturalness and real-time synthesis performance. By integrating variational inference with adversarial training within a unified architecture, VITS has established itself as a dominant paradigm in modern TTS. Despite its strong overall performance, Liu et al. [4] observed that VITS employs a single-stage variational inference structure which treats the entire speech waveform as an optimization target. In this framework, the latent variables are primarily responsible for global acoustic reconstruction and do not explicitly model prosodic features at the phoneme or frame level. This results in insufficient disentanglement of prosody from the speech signal, making fine-grained prosodic control unattainable. As a critical paralinguistic cue for expressing emotion, emphasizing semantics, and conveying speaker-specific style, prosody is vital to speech naturalness. The lack of explicit prosodic modeling often leads to synthesized speech that sounds monotonous and over-smoothed, lacking dynamic intonation and rhythmic variation.

To synthesize natural and fluent speech, Tuomo et al. [5] proposed a prosody modeling and control method for single-speaker synthesis; Morrison et al. [6] introduced a deep autoregressive model that supports controllable and context-aware fundamental frequency (F0) generation. OpenVoice [7] adopted a fully decoupled architecture of timbre and style control to achieve fine-grained voice style control and tone replication. The Sony team [8] designed a neural vocoder that achieves fine control of acoustic features such as pitch and rhythm through parameter decomposition, which is disconnected from front-end text semantics. Although the dimensions of prosodic control have expanded from single to multiple, the fusion strategies predominantly rely on simple operations such as feature concatenation or linear addition. This often results in synthesized speech with rigid intonation variations, producing a “word-by-word recitation” effect. Cheng et al. [9,10] pointed out that intuitive prosodic features like pitch, pitch range, speech rate, and energy are key factors influencing the perception of speech prosody and speaker timbre. However, simple concatenation fusion strategies fail to adequately decouple prosodic features from speaker characteristics, limiting perceptual naturalness. For instance, although FastSpeech 2 [11] introduced variable information of duration and energy et al., it merely employed simple additive fusion, making the model highly sensitive to duration prediction errors and resulting in discontinuous speech synthesis with unnatural expression.

The aforementioned studies indicate that prosodic controllable speech synthesis faces two challenges: existing approaches regard different prosodic features as independent control streams, resulting in coarse granularity and poor coordination; and prevailing fusion methods predominantly rely on simplistic linear operations, which are inadequate for capturing the complex nonlinear relationships among prosodic features.

To address these issues, this paper proposes a joint modeling and KAFusion feature fusion method for prosodic controllable speech synthesis. The core innovations are as follows:

(1)

Within the BERT-VITS framework, an independent prosodic control module is constructed to jointly model pitch features, energy features, and content information. This integrated representation is then fed as a prior control signal into the duration predictor, enabling joint modeling and fine control of speech rate and tone.

(2)

The KAFusion module, based on the KA representation theorem, is introduced into the prosodic control module. It utilizes inner and outer functions to model nonlinear interactions among pitch, energy, and text features, generating a fused prosodic representation. Furthermore, a tone-adaptive T-KAFusion module is designed within the stochastic duration predictor to deeply integrate this representation with tone and speaker embedding, achieving collaborative modeling and dynamic adaptation of multidimensional prosody.

(3)

Information entropy is incorporated as an objective evaluation metric during the feature fusion stage. Both theoretical analysis and experimental validation are conducted to verify the model’s ability to retain task-relevant information while suppressing redundancy, thereby enabling quantitative assessment of fusion performance.

2. Background

2.1. Prosodic Modeling and Control

Current mainstream prosodic modeling methods commonly implement a loosely coupled architecture that independently controls prosodic features, including pitch, energy, and duration, by separating them from the speech signal. Spark-TTS [12] employs a single-stream large language model architecture that supports fine-grained adjustment of pitch and speech rate. It also introduces coarse-grained labels, such as gender, pitch level, and speech rate level, to achieve multilevel prosody generation combining both fine-grained and global control. However, this model still maintains a dual-path separation design at the input level, using semantic tokens and global tokens. During training, different encoders extract semantic and prosodic features, respectively, thus limiting deep interaction between the two types of information and constraining performance in highly coupled linguistic environments.

Seed-TTS [13] decouples prosodic attributes, including emotion, stress, and speech rate, from text content and timbre features via self-supervised speech representation. During instruction fine-tuning, only emotion labels or style hints are needed to generate speech with rich emotional expressions, including anger, surprise, and sadness, in a zero-shot manner. Nevertheless, the system relies on a language model to generate semantic tokens and a diffusion model to render acoustic details in a serial architecture. This design essentially adheres to a decouple-then-concatenate paradigm. In languages where semantics and prosody are deeply intertwined, this approach is prone to control conflicts and rhythm mismatches.

This loosely coupled architecture design performs well in multi-phoneme languages such as English, but it faces challenges in the Mandarin context, where tones, rhythms, and semantics are highly correlated. To address these limitations, the joint modeling method proposed in this paper realizes the collaborative learning of text semantics and multidimensional prosodic features within a unified representation space, aiming to enhance the naturalness and expressiveness of synthetic speech in complex language environments.

2.2. Feature Fusion

Feature fusion is the key to prosodic controllable TTS, and the fusion effect directly affects the naturalness and expressiveness of the synthesized speech. To enhance the fusion effect, some studies have attempted to introduce more complex fusion mechanisms. NaturalSpeech3 [14] decomposes speech waveforms into content, prosody, acoustic details, and timbre, and has advanced emotion control through a dual fusion mechanism of multi-attribute element-wise addition and diffusion cross-attention. Nevertheless, it has not fully resolved the challenge of modeling the coupling among multiple features. Dai et al. [15] proposed attention-based feature fusion and developed a multi-scale channel attention module for fusing features of different scales. Chen et al. [16] proposed a novel style control fusion module named HC-TSCM, achieving notable improvements in fusion performance. Although existing research has been promoting the performance development of feature fusion through continuous innovation, its core operations still mostly rely on empirical choices like addition or concatenation, lacking a theoretical framework to ensure its modeling ability for complex nonlinear relationships.

2.3. Kolmogorov–Arnold Representation Theorem

Kolmogorov–Arnold (KA) representation theorem [17] provides a theoretical basis for feature fusion. The theorem states that any multivariate continuous function with a bounded domain can be expressed as a two-layer superposition of a finite number of univariate continuous functions. The formula of the theorem is as follows:

$$\begin{array}{c}f\left(x_1,x_2,\dots ,x_n\right)={\displaystyle {\displaystyle \sum }_{q=1}^{2n+1}}{\mathsf{\Phi }}_q\left({\displaystyle {\displaystyle \sum }_{p=1}^n}{\varphi }_{p,q}\left(x_p\right)\right)\end{array}$$

(1)

where $f$ is an n-variate continuous function to be approximated; $x_i$ represents its inputs; ${\varphi }_{p,q}$ is an inner univariate function acting on the single input, $x_p$; ${\mathsf{\Phi }}_q$ is an outer univariate function that nonlinearly combines the summed inner outputs; and $p$ and $q$ index the inner and outer functions, respectively.

The theorem demonstrates that any multivariate continuous function can be decomposed into a two-layer nested structure and approximated by hierarchical compositions of a finite number of univariate nonlinear functions, thereby providing a structured decomposition framework for modeling multivariate functions. Our prior work [18] first instantiated this framework as a learnable two-layer feature fusion module KAF designed to address the fusion of multiple features, including RGB, frequency-domain, edge, and illumination-sensitive cues, for AI-generated image detection. This implementation employed learnable neural networks as universal function approximators; that work validated the effectiveness of this paradigm in feature fusion tasks.

For modeling the strong temporal dependencies and complex nonlinearity in speech signals, we propose KAFusion, a method grounded in the KA representation theorem. This method extends the KA representation theorem to a multilayer stacked composite function approximation architecture. Through a hierarchical fusion strategy of feature distribution alignment and deep interactive learning, it transforms the multivariable acoustic features fusion problem into a two-stage modeling process: the inner function, $\varphi \left(x\right)$, handles nonlinear transformation, while the outer function, $\mathsf{\Phi }\left(\mathrm{x}\right)$, performs nonlinear fusion across multiple information channels. This mechanism enables more effective modeling of the complex nonlinear relationships among prosodic features, providing a new technical path for constructing prosodic controllable speech synthesis systems with high expressiveness and strong generalization ability.

2.4. Feature Fusion Evaluation Based on Information Entropy

The current research on feature fusion lacks solid theoretical support and systematic quantitative evaluation methods. To establish a more reliable evaluation system, Xie P [19] proposed a theoretical framework based on information entropy for feature fusion, suggesting that effective fusion should significantly reduce system uncertainty. This framework posits that the conditional information fusion entropy, $H\left(Y|C_1{C}_2\right)$, of the fused output, $Y$, should satisfy the following condition:

$$\begin{array}{c}H\left(Y|C_1{C}_2\right)<H\left(Y|C_1\right)\end{array}, \begin{array}{c}H\left(Y|C_1{C}_2\right)<H\left(Y|C_2\right)\end{array}$$

(2)

It indicates that the prediction uncertainty of the fused system for the target variable should be lower than the conditional entropy under any single information source. Meanwhile, the mutual information between the input signal, $C_1$ and $C_2$, and the output, $Y$, is non-zero, that is, $I\left(Y,C_1\right)\ne 0$ and $I\left(Y,C_2\right)\ne 0$, to ensure that the fusion process retains valid information. This theory provides a quantifiable basis for evaluating the performance of the fusion method in terms of information retention and redundancy suppression.

Information entropy has also been employed to evaluate fusion effectiveness from the perspective of representation quality. Zhao et al. [20] found that genuine speech, originating from complex physical processes, exhibits high information entropy, whereas speech generated by generative models often shows significantly lower entropy. Based on this finding, they designed a frame-level latent information entropy detector (f-InfoED), which demonstrates strong generalization capability in audio deepfake detection by quantifying entropy differences. This achievement indicates that information entropy can serve as an evaluation metric to distinguish between high- and low-quality speech representations, thereby providing both a novel theoretical perspective and an experimental approach for validating the effectiveness of fusion methods.

3. CK-VITS

Figure 1 shows the overall architecture of the proposed model, termed CK-VITS, for training and inference. The controllable parameters are below, shown in Figure 1b. The prosody control module and the fusion module are introduced into the BERT-VITS architecture, aiming to achieve the best performance of prosodic controllable speech synthesis. The joint modeling part is marked in the figure. Additional parameter configurations for relevant modules are provided in Appendix B.2.

Figure 1. CK-VITS model architecture. (a) Train, (b) Inference.

The pre-trained BERT [21] language model is used to optimize Mandarin generation. Conditioned on the input text, it extracts pitch and energy from audio to train predictor modules, while extracting tone features is dedicated to training the tone-adaptive inner function of T-KAFusion. A speaker embedding is also incorporated to capture speaker characteristics. Control parameters are introduced to adjust prosodic attributes such as pitch, energy, and speech rate during synthesis. The text-to-acoustic module projects input text into acoustic feature, z, consistent with the implementation in the original VITS framework. The decoder is essentially based on the HiFiGAN [22] vocoder architecture.

3.1. Joint Modeling Method for Prosody Control

The core objective of Figure 2 is to achieve refined and decoupled control of the prosody in speech synthesis. The prosodic control module is responsible for extracting key prosodic features from the original audio and injecting them as prior conditions into training, while predicting pitch and energy, and optimizing the prediction accuracy through a loss function. Control parameters are then introduced to achieve precise control of pitch and energy.

Figure 2. Prosody control module.

3.1.1. Analysis and Extraction of Prosodic Features

The prosody of natural speech is cooperatively represented by multiple messages. The fundamental frequency directly reflects the tone variation, the energy feature indirectly reflects the rhythm and intensity of speech, and the text encoding defines the phoneme sequence and its grammatical and semantic functions. To achieve refined prosody modeling, the following features are extracted from the reference audio:

Pitch feature extraction: Continuous wavelet transform is adopted to perform multi-scale decomposition on the continuous pitch profile. Given continuous fundamental frequency contour, F0, the spectral representation, $W\left(\tau ,t\right)$, is obtained through continuous wavelet transform, CWT [23]:

$$\begin{array}{c}W\left(\tau ,t\right)={\tau }^{-{\displaystyle \frac{1}{2}}}\underset{-\infty }{\overset{+\infty }{{\displaystyle \int }}}{F}_0\left(x\right)\psi \left({\displaystyle \frac{x-t}{\tau }}\right)dx\end{array}$$

(3)

where $\psi$ denotes the Mexican-hat wavelet, and $F_0\left(x\right)$ represents the fundamental frequency (F0) value at position $x$, $\tau$, and $t$, referring to the scale and time parameters of the wavelet, respectively. By decomposing the F0 into 10 scale components, $W_i\left(t\right)$, a multi-resolution representation of the fundamental frequency is obtained. Finally, the F0 value of each frame is quantized on a logarithmic scale into 256 discrete levels, serving as the input to the predictor.

Energy feature extraction: The short-term energy is obtained by computing the L2 norm of the magnitude of the Mel spectrum for each frame. Let $x_n\left(m\right)$ denote the speech signal of the $n$-th frame. The energy is calculated as follows:

$$\begin{array}{c}{E}_n={\displaystyle {\displaystyle \sum }_{m=0}^{M-1}}{x}_n^2\left(m\right)\end{array}$$

(4)

where $M$ represents the frame length, and the energy features are also quantified on the same logarithmic scale to match the human ear’s perception characteristics of loudness.

3.1.2. Joint Modeling Method Based on Feature Fusion

Pitch and energy are crucial for the tone and emotion expression of speech, yet they are commonly modeled independently, lacking deep coupling with text semantics. This unidimensional control approach struggles to capture the coordinated changes among multiple prosodic features in natural speech, thereby limiting the expressiveness of synthetic speech. The issue becomes particularly pronounced in the Mandarin context, where tone, rhythm, and semantics are highly correlated, often leading to rigid rhythm and limited expressiveness in synthesized speech.

We propose a novel joint modeling framework that incorporates pitch and energy features as prosodic priors and deeply fuses them with text encoding (h_text) to jointly serve as input to the duration predictor. This design enables duration prediction to no longer rely solely on latent variable perturbations or textual information alone, instead subjecting it to multiple constraints derived from prosodic characteristics: rhythmic intensity governed by energy, pitch variation linked to fundamental frequency, and dynamics of speech rate. Through establishing an interaction pathway between pitch, energy, and text representations, the model achieves joint inference of tone, energy, and phoneme duration.

In contrast to conventional approaches that model pitch and duration independently, the proposed method extends the control space for duration prediction from unimodal text context to a multimodal integration of “pitch + energy + text encoding”. Users can explicitly influence the speech rate and tone performance in the synthesis result by adjusting the intuitive prosodic features. This design improves duration controllability while ensuring consistency among tone, rhythm, and linguistic content. The associated fusion module and tone adaptation mechanism will be detailed in the following section.

3.2. KAFusion

The KAFusion fusion module is shown in Figure 3a. This module is mainly composed of two cascaded KABLOCKS. In the task of speech synthesis, there are significant distribution differences and numerical scale mismatch issues among text, pitch, and energy features. Direct single fusion is prone to cause mutual suppression or information loss among features. KABLOCK1 is specifically responsible for feature distribution alignment. It maps each feature to a unified representation space through independent inner function transformation, solving the fundamental compatibility issue of whether they can work together. KABLOCK2, based on distributed alignment, focuses on mining deep nonlinear interaction patterns. By leveraging the stable foundation provided by first-layer fusion features and combining with the retained original feature pathways, it learns complex acoustic cooperative relationships, offering a new paradigm for complex feature fusion.

Figure 3. KAFusion module and inner function. (a) Architecture of the KAFusion and inner function components (the outer function shares a similar dual-layer convolution structure). (b) Detailed steps of tone extraction in the tone inner function: converting a toned pinyin sequence (Line 1) to phonemes with syllable boundaries ^ (Line 2), and then to a frame-level tone ID sequence where digits 1–4 (from pinyin) are mapped to vowels and 0 to initials, silence (sil), and boundaries.

For the design of the inner function, the previous work [18] proposed using the multilayer perceptron (MLP) based on the ReLU activation function instead of the traditional basis function, but it has limitations in capturing local temporal patterns in speech tasks. Considering that pitch and energy are typical time-series data with strong time dependence, convolution can automatically capture this local time-series pattern. The core architecture employs 1D convolutions with multiple kernel sizes to effectively capture local correlations in prosodic features.

In a single KABLOCK, each input feature is processed by the corresponding inner function to achieve feature alignment and initial fusion. The outer functions capture the complex nonlinear interactions among features. In addition, residual connections are used to maintain information integrity. The structure adaptively learns complex, nonlinear coupling relationships between prosodic features and text content in an interpretable way, enabling highly natural and controllable speech synthesis.

T-KAFusion

Prosodic features have a significant impact on the intelligibility of tonal language. Chen et al. [24] pointed out that in continuous speech, there is a significant co-articulation effect of Mandarin tones. When pronouncing a single character, the tone contour is relatively stable. However, in continuous speech flow, its contour changes significantly with the tone context, but it is almost indistinguishable in perception. To accurately model this phenomenon, a tone adaptive inner function was designed:

$$\begin{array}{c}{\phi }_{tone}\left(x,t\right)=x\cdot \left(1+{\alpha }_t\right)\end{array}$$

(5)

where ${\alpha }_t$ is the tone-related scaling factor.

T-KAFusion is a variant of KAFusion designed by introducing the ability of tone adaptation. Figure 3b shows the tone-adaptive inner function design of T-KAFusion. The core differences between T-KAFusion and standard KAFusion (Figure 3a) is that the former extracts the tone sequence from the input text. The special markers “sil” (silent segment), “sp” (pause), and “5” tone are uniformly encoded as 0. The tone ID sequence is strictly aligned with the phoneme sequence. Based on this sequence, the model maps it to continuous vectors through an embedding layer of dimension five (corresponding to four tone categories plus a neutral class), and then fine-tunes the text features in a fine-grained, adaptive manner via a gated network composed of 1D convolutions and a sigmoid activation function (1D convolutions is used for lightweight channel fusion, and sigmoid generates a soft gating factor in the (0, 1) interval), implemented in the residual form of Equation (5). This design enables the model to dynamically adjust the text feature representation according to the tonal context, effectively capture the tonal change patterns in continuous speech, and provide an accurate acoustic basis for subsequent prosodic control.

T-KAFusion deeply fuses the above fusion results with speech rate and speaker features to achieve high-order coupling modeling of tone undulations, timbre characteristics, and prosodic context, significantly enhancing the collaborative expression ability of multidimensional prosodic features.

3.3. Theoretical Analysis of Fusion Effect

To evaluate the fusion effect of the KAFusion module, this study adopts information entropy as the core evaluation index. Information entropy, as an essential measure of information uncertainty, is naturally suitable for evaluating the changes in information during the feature fusion process. The calculation formula of information entropy is as follows:

$$\begin{array}{c}H\left(Y\right)=-{\displaystyle {\displaystyle \sum }_i^n}p\left(x_i\right)ln\left(p\left(x_i\right)\right)\end{array}$$

(6)

where $p\left(x_i\right)$ represents the probability of a random event, $x_i$.

The relevant derivations and proofs are provided in Appendix A. Ideally, the following conclusion can be drawn: $H\left(Y\right)<H(X,P,E)$, where $X$, $P$, and $E$ are the capitalized first letter of the input feature, and $Y$ is the output of additive fusion. The conclusion indicates that the ideal fusion can achieve effective compression of information redundancy.

In preliminary experiments, the inner and outer functions are designed as different MLPs, and the inner functions are simplified as follows.

$$Y_{KA}=f\left(wx+b\right)=f\left(w_{1}X+w_{2}P+w_{3}E+b\right)$$

(7)

where $w={[w_1,w_2,w_3]}^T$ denotes the weight vector corresponding to the input, $b={\left[b\right]}^T$ represents the bias term, and $f\left(\cdot \right)$ is a nonlinear activation function. When the weight vector is set to $w={\left[{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}}\right]}^T$, the model simplifies to an additive fusion scheme: $Y={\displaystyle \frac{X+P+E}{3}}$. Consequently, $H\left(Y\right)\approx H\left(Y_{KA}\right)$ holds.

The inequality $H\left(Y\right)<H(X,P,E)$ indicates that the entropy of the fused feature, $Y$, is lower than the joint entropy of the original inputs, suggesting that $Y$ eliminates some redundant information through the fusion process. Practically, feature fusion often involves noise interference, making it entirely possible for $H\left(Y\right)$ to be not less than or even greater than $H\left(X,P,E\right)$. KAFusion employs learnable weight parameters, enabling the model to autonomously focus on high-variance features while suppressing redundant noise, thereby increasing the likelihood of achieving $H\left(Y_{KA}\right)>H\left(Y\right)$. The introduction of the ReLU nonlinear activation function effectively preserves high-confidence information by truncating negative noise values. Compared to linear additive fusion, KAFusion not only has stronger noise resistance but also introduces additional feature interactions through nonlinear transformation, increasing the output information volume while ensuring reliability.

KAFusion’s higher output entropy, $H\left(Y_{KA}\right)$, compared to additive fusion, $H\left(Y\right)$, superficially contradicts the expectation that fusion should reduce redundancy. In fact, this entropy increase stems from nonlinear retention of task-critical information, rather than noise injection. This indicates that while input redundancy is reduced, the rise in output entropy precisely reflects the enrichment of task-related information. This phenomenon is consistent with the conclusion of Zhao et al. [20] previously; that is, high information entropy is closely related to the authenticity and richness of data. This design embodies both redundancy reduction and information enhancement as two aspects of a unified process, as rigorously evaluated in Section 4.2.4.

3.4. Loss Functions

The core learning objectives of CK-VITS include fundamental reconstruction losses and specific constraints for prosodic modeling. The overall loss function is defined as follows:

$$L_{total}=L_{base}+L_{control}$$

(8)

The basic reconstruction loss, $L_{base}$, inherits from the loss function of VITS, combining a variational lower bound (ELBO) loss and the adversarial (GAN) loss, ensuring the overall naturalness and fidelity of the synthesized speech. For specific forms and optimization objectives, please refer to the original VITS paper. $L_{control}$ represents the proposed prosodic constraint loss, which is used to achieve the explicit modeling of pitch and energy. This loss is used to calculate the mean square error of pitch prediction and energy prediction, respectively, and is obtained through weighted summation:

$$L_{control}=\alpha L_{pitch}+\beta L_{energy}$$

(9)

Here, for $\alpha$ and $\beta$ to balance the loss of the weighted parameters, $L_{pitch}$ and $L_{energy}$ are calculated by the following formula:

$$\begin{array}{c}{L}_{pitch}=MSE({\tilde{P}}_{pitch},Y_{pitch})\end{array}, \begin{array}{c}{L}_{energy}=MSE({\tilde{P}}_{energy},Y_{energy})\end{array}$$

(10)

where $\begin{array}{c}\tilde{P}=Align(P,shape(Y))\end{array}$ indicates that the original prediction is aligned to the scale of the target tensor by applying interpolation and dimension expansion operations.

4. Experiments and Results

4.1. Experimental Setup

4.1.1. Experimental Platform

This experiment adopts the deep learning framework of PyTorch 1.11.0 and the Ubuntu 20.04 LTS operating system. The hardware uses the Intel(R) Xeon(R) Platinum 8352V processor with 16 virtual cpus and the Nvidia RTX 4090 graphics card, with 120 GB of memory and 24 GB of video memory, respectively.

4.1.2. Datasets

The training uses AISHELL3 [25] and BZNSYP, and it fully takes the training set and test set of AISHELL3 during training and inference. BZNSYP was selected for small-scale experiments such as parameter sensitivity analysis. The BZNSYP dataset was processed from a single speaker to nine speakers, and it was only used to adapt to the multi-speaker model.

The AISHELL3 dataset is a Mandarin speech synthesis dataset that was released by AISHELL in 2020, containing 218 speakers of different ages, genders, and dialect backgrounds. Each dataset contains full-bandwidth mono audio with a speech duration of 85 h, 88,035 sentences, and a sampling frequency of 44.1 kHz. It is mainly used in the research of multi-speaker speech synthesis and timbre cloning. BZNSYP is a single-speaker Mandarin dataset that was released by Biaobei Technology in 2019. It has a speech duration of approximately 12 h and 10,000 sentences, covering various types of text, such as daily conversations, news, poetry, and numbers. The sampling frequency is 48 kHz, and it is in mono. It is mainly used for training high-quality Mandarin female voice synthesis systems and speech cloning.

4.1.3. Data Preprocessing and Training Configuration

In the data preprocessing stage, the original speech is uniformly resampled to 16 kHz, and the 80-dimensional mel-spectrum is extracted. The frame length is set to 1024, and the frame shift to 256. The text front-end processing is based on the prosody of BERT training model; it converts the input text to a phoneme sequence, including its sound pronunciation, and other special phenomena are normalized during processing; and it reserves “sil” and “sp” prosody boundary markers. The AdamW optimizer is used in the training phase with an initial learning rate of 10⁻⁴, and the remaining hyperparameters follow the original configuration of VITS. The model training uses 174 speakers. The ablation experiment, entropy experiment, and sensitivity analysis experiment used BZNSYP to save on computational resources. All experiments were conducted under the same random seed.

4.2. Experimental Results and Analysis

4.2.1. Comparative Experiment

Construct a comparative model system covering different technical routes, as shown in Table 1, and use the Mean Opinion Score (MOS) to measure the subjective quality of synthetic speech. The root mean square error (F0 RMSE) evaluates the naturalness of the fundamental frequency. The lower the value, the better the performance. The real-time factor (RTF) reflects the system efficiency. The speaker encoder cosine similarity (SECS) is used to measure timbre similarity.

Table 1. Comprehensive model performance comparison.

Method	Dataset	MOS	SECS	F0RMSE	RTF	Conference/Journal
GT	-	4.28 ± 0.10	-	-	-	-
Accent-VITS	Self	3.91 ± 0.03 *	0.84	-	-	2023 NCMMSC
SponTTS	Self	4.02 ± 0.08 *	0.84	-	-	2024 ICASSP
CrossSpeech++	Multilingual Mixture	4.06 ± 0.09	0.76	-	-	2025 TASLP
OURS	WenetSpeech4TTS	-	0.92	-	-	2025 Interspeech
E2E-3M	BZNSYP	3.91 ± 0.11 *	0.83	-	-	2024 Acta Acustica
EMM-TTS	BZNSYP	4.12 ± 0.17	0.66	-	-	2025 Arxiv
VITS	BZNSYP	3.91 ± 0.12	0.87	0.678	0.31	-
CK-VITS	BZNSYP	4.11 ± 0.09	0.92	0.560	0.31	-
Fastspeech2	AISHELL3	3.75 ± 0.08	0.69	0.719	0.23	2020 Arxiv
Tacotron2	AISHELL3	3.84 ± 0.08	-	-	0.22	2018 ICASSP
VITS	AISHELL3	3.90 ± 0.07	0.78	0.625	0.28	2021 ICML
CK-VITS	AISHELL3	4.13 ± 0.05	0.98	0.518	0.33	-

* The original text is N-MOS. “-” indicates that the original text did not report this indicator or did not involve related tests. Models with full metrics were retrained in this study.

In early TTS models, Tacotron2 [26] and VITS exhibit weak prosody controllability, and FastSpeech2 achieves preliminary feature control, but its discrete modeling lacks coordination. Recent research has made significant progress in prosodic decoupling. For instance, Accent VITS [27], SponTTS [28], and CrossSpeech++ [29] have verified the effectiveness of decoupling strategies in a single dimension or a few dimensions, such as accent transfer, spontaneous speech synthesis, and cross-language timbre decoupling, respectively. OURS [30] achieves explicit monotonic alignment through phoneme position prediction, addressing the fundamental stability issue in prosodic modeling. The two-level (frame-level and phoneme-level) CVAE of E2E-3M [31] explicitly decouples prosody and achieves controllable cross-language style transfer through a prosody predictor. EMM-TTS [32] focuses on cross-language emotion synthesis and achieves higher-level decoupling and control of emotions and timbres through two-stage modeling. The different technical levels from basic prosodic modeling to advanced style controllability provide a solid foundation for a comprehensive assessment of the overall performance of this method.

In the comparative experiments on the same dataset, the comparative models were all adjusted to adapt to Mandarin-TTS. VITS is the baseline model used. MOS was conducted to evaluate 20 speech samples generated through multidimensional control of speech rate, pitch, energy, pauses, and tone. The test involved 17 native Mandarin speakers in a quiet room, using professional monitoring headphones. The presentation order of all samples was fully randomized. Listeners rated each sample on a 15 scale with 0.5-point increments. The results are reported as mean scores with 95% confidence intervals. CK-VITS outperforms traditional models in most objective indicators, while MOS is closer to the level of real Mandarin speech. Although its RTF is modestly higher than that of VITS (0.33 vs. 0.28), it is particularly outstanding in the SECS indicator.

4.2.2. Ablation Studies

The ablation study aims to provide fine-grained attribution analysis for performance improvement. It employs duration accuracy and F0 contour similarity as objective metrics to directly quantify the model’s capability in temporal control and intonation modeling, respectively. The calculation formulas are as follows:

$$\begin{array}{c}{T}_a=1-\min \left({\displaystyle \frac{|T_{ref}-T_{gen}|}{T_{ref}}},1.0\right)\end{array}$$

(11)

$$\begin{array}{c}F{0}_s=\max \left(0,\rho \left(F{0}_{ref},F{0}_{gen}\right)\right)\end{array}$$

(12)

where $ref$ represents the reference audio, and $gen$ represents the generated audio. Duration accuracy quantifies the consistency of the overall duration between the synthesized speech and the original speech, reflecting the model’s control ability over speech rate and rhythm. The F0 similarity is measured by calculating the Pearson correlation coefficient, $\rho$, of the fundamental frequency sequence to assess the degree of consistency between the two in terms of pitch variation patterns. The higher the value, the more natural the intonation of the synthesized speech.

Based on the convergence characteristics on the small-scale dataset (BZNSYP), evaluation was conducted at 50K steps, where the relative performance relationships between model variants had stabilized (detailed convergence analysis can be found in Appendix B.1). The UTMOS [33] no-reference automatic evaluation model (with a correlation > 0.9 with manual scoring) was adopted to replace subjective listening tests. The results are shown in Table 2. The first row (× × ×) indicates that no modules were added. The second row (√ × ×) indicates that a prosody control module was added, but the fusion method is a simple additive fusion, and so on.

Table 2. Ablation study results.

Method			UTMOS	F0_Similarity	Dur_Accuracy
Additive	KAFusion	T-KAFusion	GT 3.81	-	-
×	×	×	2.6825	0.647	0.892
√	×	×	2.8516	0.728	0.931
×	√	×	3.0062	0.758	0.941
√	×	√	2.9163	0.752	0.944
×	√	√	3.0197	0.918	0.951

The improvement of objective indicators and UTMOS scores verified the effectiveness of the joint modeling and KAFusion feature fusion method. The joint modeling mechanism of the prosodic control module significantly improved the fundamental frequency similarity (0.728) and duration accuracy (0.931). The nonlinear fusion of KAFusion and T-KAFusion enhances the model’s modeling ability for complex acoustic laws. The complete model, which includes all the proposed modules, achieved the best performance in both subjective and objective indicators, confirming the effectiveness of the joint modeling of pitch, energy, text, and tone and the feature fusion of KAFusion.

The entropy values of individual features were calculated using actual data, along with the entropy values after additive fusion and KAFusion. Differences in entropy across various training steps directly reflect shifts in the data distribution encountered during model training. Entropy is a dimensionless quantity measured in nats. Experiments were conducted to validate the effectiveness of information entropy as an evaluation metric for fusion performance, and the readily observable differences in feature fusion quality were visually analyzed, as shown in Figure 4.

Figure 4. Fusion entropy comparison.

The results of the entropy experiments demonstrate that KAFusion significantly surpasses traditional additive fusion in terms of information retention capability. Specifically, within an equivalent range of training steps, KAFusion achieves an average output entropy of 3.466, which is approximately 18.5% higher than the 2.928 average fusion entropy of additive fusion. As illustrated in Figure 4, the output entropy of KAFusion consistently exceeds that of additive fusion across all stages, indicating that KAFusion preserves richer informational features, thereby enhancing the expressiveness of subsequent speech synthesis.

Figure 5 illustrates the entropy distribution before feature fusion. Here, the additive fusion feature entropy reflects the state of the raw input features, serving as visual evidence for the feature distribution differences analyzed above. Conversely, the KAFusion feature entropy depicts features transformed by the inner function.

Figure 5. Feature entropy comparison.

The results show that the nonlinear transformation of the inner function effectively improves the fusion basis of multimodal features. The original feature, that is, the feature entropy of additive fusion in the figure, has a significant uneven information distribution, making it difficult for additive fusion to make balanced use of these features. After being processed by the inner function, the entropy values of each feature were significantly enhanced and recalibrated, especially for the pitch and energy features that originally had a lower information level, where the information content increased particularly significantly. This “align first, then fuse” strategy builds a more balanced and high-quality feature representation for the fusion of outer functions, rather than being dominated by a single dominant feature, thereby achieving more effective feature fusion.

To evaluate the redundancy reduction capacity of the fusion mechanism, the entropy of the original input features prior to fusion is defined as the joint entropy. Assuming pitch, energy, and text encoding features are mutually independent, the joint entropy is computed as the sum of their individual entropies. From steps 10K to 50K, the mean joint entropy is 4.535, significantly higher than the output entropy of either fusion method. Although actual features may exhibit interdependencies, resulting in an actual joint entropy lower than the summed value, comparative analysis of fusion entropy and input feature entropy reveals that KAFusion reduces the information volume by approximately 23%, whereas additive fusion exhibits a significantly higher compression rate of 35%. This demonstrates that KAFusion achieves more efficient redundancy control while preserving a greater proportion of essential information, highlighting its superior performance in maintaining information integrity during feature fusion.

4.2.3. Parameter Sensitivity Analysis

In the KAFusion module, the experiment compares the performance of two-layer and three-layer configurations for both MLP and CONV architectures. By transforming the structure and depth of the feature fusion module, the results show that the model performance is relatively sensitive to such structural changes. As shown in Table 3, the CONV model is significantly superior to MLP in terms of speech quality and spectral modeling, while MLP has more advantages in reasoning efficiency. Specifically, the CONV_3layer model performed best in temporal modeling (0.850) and spectral continuity (0.920), and its speech quality (0.720) was also the highest, but the inference delay (computational efficiency) was relatively large (2.010 ms). In contrast, the MLP_2layer model has the lowest latency (1.263 ms) and is suitable for real-time scenarios, but its speech quality (0.491) and time series-modeling ability (0.747) are relatively weak.

Table 3. Sensitivity analysis of performance to module architecture and depth.

Conv/MLP	Temporal_ Modeling	Spectral_ Continuity	Parameter_ Efficiency	Speech_ Quality	Computational Efficiency (ms) *	Fusion Entropy
conv_2layer	0.814	0.890	0.900	0.624	1.368	3.171
conv_3layer	0.850	0.920	0.900	0.720	2.010	3.258
mlp_2layer	0.747	0.750	0.900	0.491	1.263	3.188
mlp_3layer	0.780	0.750	0.900	0.543	1.486	2.846

* Computational efficiency (ms): Latency per forward pass (batch size = 8).

As shown in Figure 6b, our entropy heatmap further indicates that the information complexity of the CONV model is higher (fusion entropy > 3.0), which is consistent with its superior time series-modeling ability. As shown in Figure 6. Based on the above experimental results, this study ultimately selects the CONV structure as the inner function of KAFusion to generate more expressive speech. For application scenarios with high real-time requirements, the lightweight MLP structure remains an effective alternative solution. This work verified the effectiveness of the CONV architecture in prosodic modeling. Future research can build on this foundation to further explore more efficient and expressive inner function designs.

Figure 6. Visualization analysis. (a) Radar chart. (b) Entropy heatmap.

4.2.4. Redundancy Reduction and Information Increase

To assess how KAFusion balances redundancy reduction and information preservation, we analyze it through a suite of complementary information-theoretic measures:

• Output entropy, $H\left(Y\right)$, measures the overall information volume and diversity of the output features.
• Conditional entropy, $H\left(Y|X,P,E\right)$, measures the uncertainty that still exists in the output after all input sources are given, that is, the noise introduced by the fusion process.
• Mutual information, $I\left(Y;X\right),I\left(Y;P\right),I\left(Y;E\right)$, measures the amount of information shared between the output and each input source, that is, the degree of retention of task-related information.
• Redundancy reduction, $\mathrm{\Delta }R=H\left(X,P,E\right)-H(Y)$, measures the degree of compression of the input joint information set during the fusion process.

Figure 7a shows that KAFusion generates more abundant effective information output (output entropy increases by +0.517), while reasonably controlling noise (the increase in conditional entropy is smaller than that of output entropy). In Figure 7b, KAFusion achieves a more significant reduction in redundancy ($\mathrm{\Delta }R$ increases by +1.304), compressing the repetitive content between input sources. The key to unifying these two seemingly contradictory goals lies in the intelligent redistribution capability demonstrated by KAFusion in the feature utilization strategy, as shown in Figure 7c. Compared with additive fusion, KAFusion significantly enhanced the $I\left(Y;X\right)$ (from 0.224 to 1.018), while moderately strengthening the $I\left(Y;P\right)$. However, KAFusion instead reduced $I\left(Y;E\right)$ (from 1.793 to 0.883).

Figure 7. Entropy analysis over 100K training steps. (a) $H\left(Y\right)$ vs. $H\left(Y|X,P,E\right)$ (H_cond). (b) Redundancy reduction, $\mathrm{\Delta }R$. (c) Mutual information with each source. KAFusion shows higher output entropy, greater redundancy reduction, and reallocated source contributions. Arrows highlight mean improvements. The asterisk (*) above the connecting line indicates a statistically significant difference (p < 0.05) between Additive and KAFusion for that specific feature, based on paired t-tests.

This phenomenon is not a performance degradation but rather reflects KAFusion’s intelligent feature selection mechanism: (1) Energy features may contain a lot of noise or be highly redundant with other features. (2) For the target task, the text and pitch features are more discriminative. (3) KAFusion’s selective information extraction still increased the total mutual information from 3.627 to 3.709, demonstrating the high efficiency of KAFusion in information utilization. This means that the model better captures and retains fine-grained information related to prosody. Therefore, the entropy added by KAFusion is essentially the entropy of task-related information rather than noise or redundancy. This feature enables it to generate speech that is both natural and varied.

Thus, redundancy reduction serves as a global constraint (indicated by a larger $\mathrm{\Delta }R$), eliminating repetitive and irrelevant information across inputs, while information increase acts as a local optimization, selectively enhancing task-critical information (e.g., for prosody modeling) within this compressed space, thereby increasing the output’s effective diversity without violating the global constraint.

5. Conclusions

This paper proposes a joint modeling and KAFusion method for prosodic controllable speech synthesis. By constructing an independent prosodic control module and a nonlinear fusion mechanism based on the KA representation theorem, the collaborative modeling and independent regulation of multidimensional prosodic features such as pitch, energy, tone, and speech rate were achieved. The MOS and SECS metrics on the AISHELL3 test set reached 4.13 and 0.98, respectively. Ablation studies further validated the effectiveness of both the joint modeling and the KAFusion module. Information entropy was introduced as a theoretical metric to evaluate fusion performance, transforming the abstract feature interaction process into quantifiable information patterns. It should be noted that this study mainly focuses on verifying the feasibility of the fusion framework. Future work will systematically explore the better configuration of the inner and outer functions and activation functions of KAFusion to further improve the fusion performance.

Appendix B.1

On the small-scale BZNSYP dataset, models typically reach performance saturation within a relatively small number of training steps. The performance differences among various architectures become evident early in training, and prolonged training may lead to overfitting rather than substantial improvement. To verify the stability of the ablation studies, we supplemented an extended experiment to evaluate the performance of all models at steps 30K, 50K, 70K, 90K, and 100K.

Figure A1 demonstrates the stability of ablation conclusions across training progression. F0 similarity advantage (KAFusion vs. Baseline) stabilizes from 0.111 (50K) to 0.048 (100K); duration accuracy advantage reduces from 0.049 to 0.010. The ordering KAFusion > Additive > Baseline holds at all checkpoints (30K–100K steps) without reversal. The consistent performance ranking and stabilized relative gaps confirm that the qualitative conclusions from our ablation studies (i.e., the superiority of our proposed design over the baseline) are reliable.

Figure A1. Metric trends over 100K training steps. Data points are discrete; lines are shown for visual guidance only to indicate trends.

Figure A1. Metric trends over 100K training steps. Data points are discrete; lines are shown for visual guidance only to indicate trends.

Appendix B.2

• KAFusion module parameter configuration:

KAFusion adopts the following core configuration: The hidden dimension is set to 192. Each KABLOCK contains an inner transformation (three convolutional layers with kernel sizes of 5, 3, and 1, respectively) and an outer transformation (two convolutional layers with kernel sizes of 3 and 1, respectively). All convolutional layers use appropriate padding to maintain consistent input and output sequence lengths, with a fixed stride of 1. The module takes 63-dimensional text, pitch, and energy features as input and output 512-dimensional fused features.

Specific padding strategy:

(1)

Kernel size 5: padding = 2.

(2)

Kernel size 3: padding = 1.

(3)

Kernel size 1: padding = 0.

Regarding the number of inner functions:

The design follows a core principle: The number of inner functions is determined by the need for dimensional alignment of the input features. In this study, all input features have been preprocessed to the same dimension prior to fusion. Therefore, we employ a single, shared inner function to transform all features. This ensures efficient parameter utilization while satisfying the requirement for feature space consistency in fusion.

2.

Prosody control module configuration:

The prosody control module includes pitch and energy predictors, employing a two-layer convolutional structure (kernel size = 3; hidden dimension = 256), with padding = 1 in each layer to preserve sequence length. Each layer is followed by ReLU activation, layer normalization, and Dropout (p = 0.5). Pitch and energy features are discretized via quantization embedding layers (bucket size = 256), with differentiable quantization achieved during training using the Straight-Through Estimator. The control factors pitch_factor and energy_factor default to 1.0 and can be adjusted within the range [0.5, 2.0] for prosody control.

We quantified the computational load of the prosodic control module in the proposed model CK-VITS:

FLOPs analysis: The prosody control module requires 0.56 GFLOPs, contributing ~10% of the total model’s 5.6 GFLOPs.

Parameter analysis: It contains 3.256 M parameters (7.6% of total 43.032 M), demonstrating efficient design.