Using Denoising Diffusion Model for Predicting Global Style Tokens in an Expressive Text-to-Speech System

Abstract

Text-to-speech (TTS) systems based on neural networks have undergone a significant evolution, taking a step forward towards achieving human-like quality and expressiveness, which is crucial for applications such as social media content creation and voice interfaces for visually impaired individuals. An entire branch of research, known as Expressive Text-to-speech (ETTS), has emerged to address the so-called one-to-many mapping problem, which limits the naturalness of generated output. However, most ETTS systems applying explicit style modeling treat the prediction of prosodic features as a regressive, rather than generative, process and, consequently, do not capture prosodic diversity. We address this problem by proposing a novel technique for inference-time prediction of speaking-style features, which leverages a diffusion framework for sampling from a learned space of Global Style Tokens-based embeddings, which are then used to condition a neural TTS model. By incorporating the diffusion model, we can leverage its powerful modeling capabilities to learn the distribution of possible stylistic features and, during inference, sample them non-deterministically, which makes the generated speech more human-like by alleviating prosodic monotony across multiple sentences. Our system blends a regressive predictor with a diffusion-based generator to enable smooth control over the diversity of generated speech. Through quantitative and qualitative (human-centered) experiments, we demonstrated that our system generates expressive human speech with non-deterministic high-level prosodic features.

1. Introduction

Synthesizing human speech for manually chosen textual content has a variety of potential applications, ranging from generating social media content, such as advertisements and video materials, to systems involving human–computer interaction (HCI), including personal assistants for visually impaired individuals and automated health monitoring systems. Text-to-speech (TTS) systems aim to generate human-quality speech based on a textual prompt. The process can be formulated as follows:

$$TTS(l_1,\dots ,l_t)=a_1,\dots ,a_T$$

(1)

where $l_1,\dots ,l_t$ are the input linguistic features, e.g., characters, phonemes, or additional symbols, such as sentence breaks and part of speech (POS) tags; $a_1,\dots ,a_T$ is a sequence of acoustic features, such as real numbers describing a raw waveform, or a 2-dimensional mel-spectrogram.

1.1. State of the Art

In recent years, numerous approaches (e.g., [1]) have emerged that successfully generate speech resembling that produced by human articulatory organs. Some of the systems achieved true human quality; i.e., they produced output that is almost indistinguishable from natural speech. The systems above successfully leverage data-driven, Deep Learning-based architectures to find interrelationships between input data samples and generate speech in an end-to-end fashion. However, although neural networks enable the system to produce output without relying on curated, hand-engineered features or complex preprocessing pipelines, classical TTS systems suffer from the so-called one-to-many mapping problem.

The one-to-many mapping problem is a result of the inherent property of human speech, which is the fact that a particular textual content may be pronounced in an infinite number of ways, which differ from each other in the speaker’s voice, speaker-independent acoustic environment, and prosody, which encompasses the high-level properties of speech, such as rhythm. Neural network-based systems, which are trained to map input text to output acoustic features, lack access to additional parameters that govern expected speech expression and therefore tend to average the speaking style across the dataset they are trained on. This results in the generation of monotonous, fully deterministic speech, which impedes the creation of an illusion of interacting with a real human being, which, in turn, is crucial for ensuring the comfortable reception of generated information. To alleviate the problem, the so-called Expressive TTS (ETTS) approaches have emerged, which aim to provide the model with additional style information through either supervised style labels or unsupervised style embedding [2].

One of the most successful approaches to ETTS leverages the concept of the Global Style Tokens, introduced in [3], which consists of transforming the reference speech into a weighted sum of randomly initialized trainable vectors, which are expected to represent various aspects of the style in an unsupervised manner. The approach was observed to facilitate robust and accurate control of speaking style; however, it made the system dependent on an additional input, such as another speech sample, from which the style was supposed to be transferred. To address this problem, several works (e.g., [4]) attempted to train an additional model to predict the style embedding solely from the input text. This may be seen as a form of teacher forcing, where the model is trained to leverage an external source of information and, during inference, uses part of the predicted output as input.

1.2. Problem Formulation

Although the described approaches for predicting style embeddings directly from textual input were successful, they all suffer from the same problem: treating the prediction of style information as a regressive rather than a generative process. They do not attempt to learn the distribution of possible style features corresponding to a given textual content and instead merely provide the trained model with explicit “helper” style information to accelerate the convergence. An analogous technique was applied in FastSpeech 2 [5], where the acoustic model is provided with ground truth pitch and energy contours, which are subsequently predicted by a specialized part of the model. The approaches mentioned above improve the model’s performance by simplifying the difficult task of mapping high-dimensional tensors. In practice, this does not solve the one-to-many mapping problem but merely tackles its primary impact, which is a monotonous speaking style within each generated sentence. As a consequence, the above systems are likely to generate similar messages or parts thereof, invariably using the same style, making communication with them unnatural. Poor expressiveness in such cases does not reside in a lack of diverse prosody within a sentence, but rather in a monotonous articulation of certain linguistic structures across multiple sentences.

In this paper, we address the problem above and propose a TTS system, which models the speaking style as a GST-based numerical vector and predicts it using a mixture of a transformer-based model and a non-deterministic diffusion framework. The rationale behind the use of diffusion models is that they exhibit a powerful ability to learn complex probability distributions and generate diverse data samples through variational inference [6,7]. The baseline approaches, in turn, use simple regression, which we believe is a suboptimal way to learn a complex distribution of stylistic features.

The backbone of the system, which is modified FastSpeech [1], is conditioned on a style embedding, which is partially produced through a backward diffusion process, which should be treated as sampling from a modeled distribution rather than prediction. Through quantitative and qualitative (human-centered) experiments, we demonstrate that the proposed system not only produces speech of comparable quality to that generated by other known TTS systems but also facilitates the generation of speech samples with considerable diversity in high-level prosodic properties, such as pitch and duration. To our knowledge, this is the first time a diffusion framework combined with GST-based unsupervised style modeling has been used to tackle the problem of generating non-deterministic natural human speech.

To ensure the reproducibility of our work, we publish both the source code and data on our GitHub repository at https://github.com/WiktorProsowicz/ddpm-gst-speech-synthesis (accessed on 29 November 2025).

2. Related Work

Systems that process human speech aim to either reduce the distance between humans and computers, offering an acoustic human–computer interface in human–computer interaction, or ensure seamless communication between people, often involving technical solutions, such as noise reduction. Among various applications of speech processing in the area of human-centered computing, we consider the following to be related to our work: ref. [8], which aimed to ensure privacy during communication in shared spaces, such as libraries, through automatic recognition of inaudible speech and source separation. These solutions enabled seamless communication in environments that previously hindered it. Ref. [9] helps control a speech synthesis system by proposing a user-friendly interface that allows for high-level control of speaking-style features for social media content creation. Ref. [10] investigated the role of the gender of a voice assistant in the reception of the spoken information. Ref. [11] investigated the potential of extracting a style representation from a speech recording to guide a model generating talking heads with lip movement and facial expressions transferred from a reference video.

The evolution of classical TTS systems was driven mainly by the struggle to achieve output that would be indistinguishable from human-quality speech. It also involved striving to create a system capable of producing fast and robust output. A common taxonomy divides the available models into auto-regressive, involving [12,13,14,15,16], and parallel ones, such as [1,5,17]. The Tacotron 2 model consists of a combination of convolutional and recurrent neural networks and is trained to predict a single spectrogram frame given the previous ones. It uses a location-sensitive attention mechanism to learn alignment between the encoder and decoder outputs automatically. The FastSpeech and FastSpeech 2 models successfully produce high-quality, robust speech, achieving high inference speed. They leverage a transformer-based architecture and are therefore easily parallelizable. Instead of an attention mechanism, they use explicit duration prediction to fill the dimensionality gap between text and sound representations.

The Transformer architecture, proposed in [18], has been successfully applied across numerous domains due to its powerful language modeling capabilities, which can be leveraged in domains where data can be represented as sequences with complex multi-scale relationships. For example, it was observed to exhibit good performance in image processing (e.g., [19,20]), where it successfully finds relationships between image patches of varying proximity. Moreover, the architecture shows great performance in tasks derived from sequence-to-sequence translation. For example, in [21,22], the transformer architecture was used to generate sequences of quantized representations of waveform and style features, respectively. A research branch is devoted to improving the performance of the classical architecture. For example, ref. [23,24] strive to reduce the quadratic complexity of their attention mechanisms by proposing linearized equivalents.

One of the first attempts to model the speaking style as a numerical vector is [25], which introduced the concept of a reference encoder that passes the reference spectrogram through an information bottleneck, enabling unsupervised modeling of the style. This approach was then incorporated, in modified form, into numerous works, such as [26,27], which attempted to compress the reference audio into a latent representation that was subsequently used to condition the generation of the spectrogram. Learning the distribution of stylistic features in a latent space was also leveraged in style-conversion tasks, proposed in works, such as [28], which incorporated speaker-adaptive normalization layers, or [21], which transferred style and voice from a short audio recording using discrete representations from a neural codec model [29].

The method proposed in [3] builds upon the reference encoder and introduces Global Style Tokens, which may be considered an intermediate state between using a clean style embedding and Vector Quantization techniques. This became a standalone area of research in ETTS, and the concept of GST was leveraged in various works, such as [30,31,32], which involve, respectively, the use of semi-supervised emotion labels, graph neural networks-based conversational speech synthesis, and modeling hierarchical prosodic features.

The first successful attempt to predict GST solely from textual input, as mentioned in the literature, is [4], which proposes two methods for reconstructing the style embedding: predicting the vector of weights, which is subsequently multiplied by the pre-trained tokens, or predicting the style embedding vector directly. Ref. [33] trains the backbone model to use discrete word-level style variation (WSV) vectors and predicts them using both enhanced text representations from the model encoder and additional BERT [34] word representations. This work was followed by [35], which incorporates WSVs into a parallel acoustic model.

Modern generative algorithms have been successfully applied in expressive text-to-speech systems. In [36], the nature of the used diffusion algorithm influences the diversity of the low- and high-level properties of speech. The authors mention the possibility of tuning an additional temperature parameter, which modifies the variance in the backward diffusion and therefore increases the variety of pitch tracks in the generated speech. Ref. [37], which leverages a parallel flows-based framework, models the prediction of phoneme durations as a generative process by training an additional flow-based model, which increases the diversity of the produced output. In [38], the non-deterministic generation is ensured by using a Variational Autoencoder for end-to-end synthesis of waveform, conditioned on phoneme representations, upsampled using a stochastic duration predictor. Ref. [27] first encodes the reference mel-spectrogram into a fixed-size style vector and predicts it using a diffusion framework, which is capable of generating diverse style representations. In our work, we combine the powerful modeling capabilities of diffusion models with the boundedness of the style distribution ensured by the semi-clustering algorithm of the GST component, which helps control the amount of style-related information in the predicted style embedding.

It should be noted that Denoising Diffusion Probabilistic Models or their derivatives were successfully applied in numerous areas to model spatio-temporal relationships. To name a few, they were incorporated in image generation [39], image denoising [40], video generation [41], and waveform generation [42]. The powerful learning capabilities of diffusion models make them a universal approach to generative AI and place them between other standard techniques, such as VAEs [7] or GANs [43].

3. Materials and Methods

In this section, we introduce the system’s architecture. A simplified diagram showing the flow of the data is presented in Figure 1. We build upon an existing acoustic model, FastSpeech, which is a parallel feed-forward model. During training, we condition the generation of output spectrograms by adding a style embedding vector derived from the reference spectrogram to the backbone model’s encoder hidden states. During inference, the style embedding is created as a combination of deterministic and non-deterministic embeddings, generated by additional modules trained on the reference data and distilled from the acoustic model. The overall architecture is shown in Figure 2. We describe the system’s modules in detail in the following subsections.

3.1. GST FastSpeech

Both the encoder and decoder parts of the FastSpeech backbone, used in our system, consist of so-called Feed-Forward Transformer (FFT) blocks, which were inspired by [18] and introduce several small modifications, described in detail in [1]. Each block consists of a Multi-Head Self Attention Layer and a convolutional upscaling layer, each followed by layer normalization [44] and residual connections [45]. We also slightly modify the architecture of the modules by (a) leaving only one intermediate ReLU activation between the 1D convolutional layers and (b) applying dropout to the attention matrix in the Multi-Head Attention layer. Denote the hidden states of the encoder as ${\mathcal{H}}_{\mathcal{PHO}}=[h_1.h_2,\dots ,h_n]$, where n is the length of the input phonemes sequence.

Unlike the original work [3], we do not build our reference encoder upon convolutional layers, but instead use a linear projection pre-net, which extends the dimensionality of the reference spectrogram to the hidden size of the backbone model, followed by a stack of FFT blocks. The processed sequence of hidden states is then passed through a Multi-Head Attention module, where a trainable, randomly initialized vector is chosen as the query, referred to as pooling attention from this point forward.

The resulting vector is processed through a Dot-Product Attention module. Denote the reference embedding vector as $v_{RE}$ and the GST matrix as $A_{GST}$. Denote the resulting GST weights as

$$w_{GST}=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)=softmax({\displaystyle \frac{(v_{RE}{W}^Q+b^Q)\ast {(A_{GST}{W}^K+b^K)}^T}{\sqrt{d_k}}})$$

(2)

where $W^Q$, $W^K$, $b^Q$, and $b^K$ are the parameters used by the linear projection layers of the Dot-Product Attention module, and $d_k$ is the hidden size of the model. Denote the style embedding vector as

$$v_{SE}=w_{GST}V=w_{GST}(A_{GST}{W}^V+b^V)$$

(3)

where $W^V$ and $b^V$ are the linear projection parameters of the Dot-Product Attention module.

3.2. Denoising Diffusion Model for GST Prediction

Creating the non-deterministic part of the inference-time style embedding is realized through sampling from a learned conditional distribution of possible embeddings:

$$S{E}_{NDet}\sim p_{\theta }(x\mid y)$$

(4)

where x and y are continuous multi-dimensional random variables, which represent, respectively, the space of all possible style embeddings and tensors holding conditioning information, which in this case is a sequence of hidden linguistic representations.

As described in [6], learning the probability distribution with a diffusion framework consists of learning the intermediate conditional distributions between all adjacent steps of a diffusion process. Forward diffusion consists of T transformations applied to the data, formulated as follows:

$$\begin{array}{c}{x}^{\left(0\right)}\sim q\left(x^{\left(0\right)}\right)\\ \dots \\ x^{\left(t\right)}\sim q(x^{\left(t\right)}\mid x^{(t-1)})\\ \dots \\ x^{\left(T\right)}\sim q(x^{\left(T\right)}\mid x^{(T-1)})\end{array}$$

(5)

where $q(x^{\left(t\right)}\mid x^{(t-1)})=\mathcal{N}(x^{\left(t\right)};\sqrt{1-{\beta }_t}{x}^{(t-1)},\sqrt{{\beta }_t}I)$ is the distribution of the forward noising process with the chosen set of hyperparameters ${\beta }_1,\dots ,{\beta }_T$. With such a formulation of the noising process, it can be shown that, for sufficiently high T, the distribution of $x^{\left(T\right)}$ approaches $\mathcal{N}(x^{\left(T\right)};0,I)$ [46].

Since both the entire noising and denoising transformations are first-order Gaussian processes, the joint parametrized distribution of all backward diffusion steps is as follows:

$$p_{\theta }(x^{\left(0\right)},\dots ,x^{\left(T\right)}\mid y)=p_{\theta }(x^{\left(T\right)}\mid y)\prod _{t=1}^T{p}_{\theta }\left(x^{(t-1)}\right|x^{\left(t\right)},y;t-1)$$

(6)

which allows for breaking down the Evidence Lower Bound (ELBO) of the negative log-likelihood into independent elements, which are maximized through minimizing KL-Divergence between forward and backward conditional distributions at particular steps, which in turn, further simplifies to minimizing square distance between parametrized ${\mu }_{\theta }(x^{\left(t\right)},y,t-1)$ and mean value of the distribution $q(x^{(t-1)}\mid x^{\left(0\right)},x^{\left(t\right)})$.

Finally, after deriving $q(x^{\left(t\right)}\mid x^{\left(0\right)})=\left({\prod }_{t^{\prime }}^t\sqrt{1-{\beta }_{t^{\prime }}}\right)x^{\left(0\right)}+(1-{\prod }_{t^{\prime }}^t(1-{\beta }_{t^{\prime }}))ϵ$, where $ϵ\sim \mathcal{N}(0,1)$, the ultimate objective is optimized by minimizing the square distance between the noise, added to the “clean” data, and parametrized ${ϵ}_{\theta }(x^{\left(t\right)},y,t-1)$.

In conclusion, the data distribution is learned by training a neural network, which, based on the provided diffusion timestep information $t-1$, noisy data $x^{\left(t\right)}$, and the conditioning information, returns the estimated total noise, added to the clean data to create $x^{\left(t\right)}$. The predicted noise is then substituted into the formula for the mean value of the distribution $q(x^{(t-1)}\mid x^{\left(0\right)},x^{\left(t\right)})$, which is subsequently used to sample $x^{(t-1)}$.

3.3. Non-Deterministic GST Predictor

$S{E}_{NDet}$ is obtained as the result of the backward diffusion algorithm presented in Algorithm 1. The prediction of added noise in the diffusion framework is a regression problem, realized by a neural network, called Non-deterministic GST Predictor from this point forward. The network architecture is presented in Figure 3a.

Algorithm 1 Backward diffusion sampling in a diffusion framework

Require:  Trained noise prediction model ${\mathit{ϵ}}_{\theta }({\mathbf{x}}^{\left(t\right)},y,t)$, number of steps T, noise schedule ${\left\{{\beta }_t\right\}}_{t=1}^T$, conditioning information y.   1:  Compute ${\alpha }_t=1-{\beta }_t$,    ${\overline{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$   2:  Sample ${\mathbf{x}}^{\left(T\right)}\sim \mathcal{N}(0,\mathbf{I})$   3:  for $t=T$ down to 1 do   4:      ${\mu }_{t-1}={\displaystyle \frac{1}{\sqrt{{\alpha }_t}}}\left({\mathbf{x}}_t-{\displaystyle \frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}}{\mathit{ϵ}}_{\theta }({\mathbf{x}}^{\left(t\right)},y,t-1)\right)$   5:      if $t>1$ then   6:          Sample $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$   7:      else   8:          $\mathbf{z}\leftarrow \mathbf{0}$   9:      end if 10:      ${\mathbf{x}}^{(t-1)}={\mu }_{t-1}+\sqrt{{\beta }_t}\mathbf{z}$ 11:  end for 12:  return ${\mathbf{x}}_0$

Encoder: The encoder processes the linguistic information into a sequence of enriched representations, which is used to condition the generation of the output noise. Since the entire system operates on phoneme representation of textual data, it is unable to learn high-level semantic information, which would significantly aid in predicting the expected output. To address this problem, we enrich the phoneme-level input to the model with hidden representations taken from a pre-trained BERT model, which was inspired by [33], which shows that the use of pre-trained text representations improves the accuracy of predicting style features, as compared with baseline cases, such as [4], which incorporates only phoneme-level representations. Denote the enriched input as follows:

$${\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}}=cat({\mathcal{H}}_{\mathcal{PHO}};{\mathcal{H}}_{\mathcal{BERT}})$$

(7)

where $cat(·)$ is a concatenation operation, and ${\mathcal{H}}_{\mathcal{BERT}}$ is a sequence of token-level BERT hidden representations, first averaged and then stretched using token-to-word and word-to-phoneme mappings, respectively. A stack of FFT blocks processes the enriched linguistic information and then passes it through the post-encoder, which consists of a fully connected layer, lowering the dimensionality of the representations to the hidden size of the backbone model, followed by SiLU [47] activation and an additional dropout [48] layer.

Decoder: The decoder predicts Gaussian noise, used to create the noised data from the time step t, using the encoded linguistic information, the time step number $t-1$, and the noised data $x^{\left(t\right)}$. The integer $t-1$ is first encoded as a numerical vector using position encoding, as described in [18], which is then passed through a fully connected layer with SiLU activation. The noisy data from timestep $x^{\left(t\right)}$ is first stretched by a 1 × 1 convolutional pre-net to the number of internal channels and subsequently transformed by a stack of residual blocks. Each block contains three 1D convolution layers with SiLU activation, preceded by group normalization [49] and followed by a dropout regularization layer. The block’s input is passed through the first convolution and subsequently used as the query to a Multi-Head Attention module, which selects a combination of transformed elements of the conditioning information sequence for each of its channels. The attention output is added to the query and passed through the second convolution layer. This is followed by adding the encoded timestep, broadcast to match the intermediate output’s dimensionality. The final convolution and a residual connection complete the total block’s transformation. The output of the last block is passed through a pooling attention module, which reduces it to a single vector.

3.4. Deterministic GST Predictor

To enable diversity control of the generated speech, we decided to train an additional module that predicts the style embedding deterministically, referred to as the Deterministic GST predictor. Its role is to predict the GST weights $w_{GST}$, based solely on the enriched linguistic information ${\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}}$, which are used to obtain deterministic style embedding, denoted as $S{E}_{Det}$. This decision stems from the expectation that it will be much easier to predict the weights rather than the full embedding, given the significant difference in their dimensionalities. On the contrary, due to the stochastic nature of the non-deterministic GST predictor, we expect slight fluctuations in the style-embedding vector to do less harm to the system’s output than it would do if it were to generate the weights. The reason is that since some of the softmax-based weights are expected to be close to zero, every small absolute deviation would be a considerable relative change, which would then have a visible influence on the resulting style embedding.

The neural architecture of the model is presented in Figure 3b. It accepts ${\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}}$, described in Equation (7), as its input, which is first passed through a fully connected layer, reducing its dimensionality, followed by ReLU activation [50] and dropout. The encoded input is then processed through a stack of FFT blocks, pooling attention, and a post-net, with a similar architecture to the pre-net, which projects it to match the dimensionality of the GST weights vector.

3.5. Inference

During inference, the control of diversity of the produced speech is realized by obtaining a linear combination of the non-deterministic and deterministic style embeddings using a hyperparameter, which is denoted as $\lambda$. The deterministic embedding should be considered the baseline case, which is an averaged representation of the possible style characteristics corresponding to the given textual input. Blending it with the non-deterministic embedding with specific proportions allows for controllable exploration of multiple possible style representations.

Assume that we have the following: (a) a pre-trained backbone model with reference embedding and GST vectors; (b) two GST predictors, trained using the $({\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}},v_{SE},w_{GST})$ tuples with reference data taken from the backbone model. The inference process, given a set of input phonemes and BERT hidden representations, consists of the following steps:

• Process the input phonemes through the encoder of the backbone model and obtain ${\mathcal{H}}_{\mathcal{PHO}}$.
• Concatenate BERT representations with encoder’s output to obtain ${\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}}$.
• Calculate $S{E}_{NDet}$ using Algorithm 1, where the conditioning information y is ${\mathcal{H}}_{\mathcal{PHO}+\mathcal{BERT}}$.
• Calculate $w_{GST}^{\prime }$ using the Deterministic GST predictor and substitute it into the Equation (3) to obtain $S{E}_{Det}$.
• Calculate the final style embedding as $SE=\lambda S{E}_{Det}+(1-\lambda )S{E}_{NDet}$.
• Add the final style embedding to ${\mathcal{H}}_{\mathcal{PHO}}$ and pass it through the backbone model’s decoder to obtain the output spectrogram.
• Convert the output spectrogram to the output waveform using a vocoder.

The resulting waveform is the system’s output and does not require further processing.

4. Results

The two main questions that were to be answered through experiments are as follows: (1) How natural and expressive does the speech produced by the proposed system sound compared to other well-known TTS systems? (2) Does the $\lambda$ parameter of the proposed system allow us to manipulate the high-level properties of the produced speech?

To evaluate the system’s performance, we conducted two primary types of experiments: one examining numerical statistics derived from the generated speech samples and the other involving subjective evaluation by human raters. Each type comprises two experiments: the first compares various configurations of the proposed system with baseline models, whilst the latter evaluates the influence of the $\lambda$ parameter on the variety of features obtained from the generated samples.

The pivotal part of the experiments was the evaluation of the system by human raters, which we consider crucial for the following purpose: the addressed problem revolves around human-centered computing and, as such, should be rated within a subjective evaluation framework, which is bound to yield the most valuable results, as compared to assessing the system’s performance using less informative objective metrics. We conducted a subjective evaluation by presenting 15 people with a set of generated samples and summarizing the numerical scores they assigned to them.

4.1. Experimental Setup

The data set used for the experiments is LJSpeech [51], a publicly available data set consisting of 13,100 recordings of text samples from 7 non-fiction books. A native English-speaking female recorded all samples. The textual part of each data sample was normalized and transformed into a sequence of phonemes using the g2pE 2.1.0 tool [52]. The audio files were padded to 10 s and converted to 80-dimensional mel spectrograms. The phoneme durations were obtained using the Montreal Forced Alignment tool [53].

The model parameters are described in

Table 1. System’s parameters used for experiments.

Model’s Name	Component’s Name	Description of Parameter	Value of Parameter
Acoustic	Encoder/Decoder	Number of FFT blocks	6
		Number of filters in the FFT’s 1D Convolution	1536
	Duration predictor	Number of convolutional layers	4
	GST	Number of tokens	32
	Reference encoder	Number of blocks	5
		Number of filters in the FFT’s 1D Convolution	384
	Global	Dropout rate	0.1
		Hidden dimension	384
		Number of heads in FFT’s multi-head attention	4
Non-det. GST Predictor	Encoder	Number of blocks	16
		Number of heads in FFT’s multi-head attention	4
		Number of filters in the FFT’s 1D Convolution	1536
		Dropout rate	0.1
	Decoder	Number of blocks	10
		Number of 1D convolution channels	128
		Timestep embedding size	128
	Diffusion	Number of steps	1000
		Diffusion schedule	Linear [${\beta }_1=1\times {10}^{-4}$, ${\beta }_T=2\times {10}^{-5}$]
Det. GST Predictor	-	Number of blocks	2
		Dropout rate	0.2
		Hidden dimension	768
		Number heads in multi-head attention	16
		Number of filters in the FFT’s 1D Convolution	1536

. The hyperparameters of the backbone model (hidden dimension, number of blocks, configuration of attention) were directly adapted from the original FastSpeech paper [1]. The configuration of the GST module (number of tokens) and diffusion process (parameter schedule) were chosen empirically. In particular, selecting the number of tokens as 10, as in the original paper [3], resulted in a worse quality of the generated sound; therefore, we ultimately decided to set the value to 32.

The acoustic model was trained for 95 K steps (around eight hours). During the initial 80 K steps, the GST module was disabled, and the backbone model was conditioned directly on the Reference Encoder’s output. The GST module weights were subsequently trained for the next 10 K steps with the rest of the model frozen. Finally, the entire model was fine-tuned for the last 5000 steps. The base vanilla model was trained for 90 K steps. Training the GST Predictor took 400 K steps (about 18 h), with the deterministic part frozen after 40 K steps to prevent overfitting. The weights of all components of the model were trained using the Adam optimizer [54]. The components using FFT blocks had their learning rates adjusted according to the schedule described in [18], while the non-deterministic GST Predictor used a fixed learning rate of $2\times {10}^{-4}$.

For the baseline model, we chose Tacotron2 [13], an autoregressive model with a recurrent neural network architecture. The samples generated by it were transformed into a waveform using either the WaveRNN vocoder [55] or the Griffin–Lim algorithm [56]. We used models offered by PyTorch 2.5.1 pre-trained models registry.

The experiments were conducted on 100 test samples, randomly selected from the data set and not used during training. Additionally, several samples have been manually selected. For each configuration of the proposed system, we generated 15 outputs corresponding to their textual content, as shown in

Table 2. Textual contents of several samples from the test set chosen for evaluation of speech diversity.

Sample ID	Textual Content
LJ005-0008	They were followed by a crowd of reckless boys, who jeered at and insulted them.
LJ008-0184	Precautions had been taken by the erection of barriers, and the posting of placards at all the avenues to the Old Bailey, on which was printed.
LJ011-0095	He had prospered in early life, was a slop-seller on a large scale at Bury St. Edmunds, and a sugar-baker in the metropolis.
LJ015-0009	Cole’s difficulties increased more and more; warrant-holders came down upon him demanding to realize their goods.
LJ048-0261	Employees are strictly enjoined to refrain from the use of intoxicating liquor.
LJ050-0254	The Secret Service in the past has sometimes guarded its right to be acknowledged as the sole protector of the Chief Executive.

. Those were subsequently used to examine the variety of the system’s output. For all configurations of the system, we used a pre-trained HiFi-GAN vocoder [57] to obtain the waveform from the predicted spectrograms. Both training and inference were conducted on a single NVIDIA GeForce RTX 4070 Super GPU with 12 GB of VRAM.

For experiments, we compile several versions of the system, which differ in the choice of the $\lambda$ parameter. The chosen values are 0.2, 0.4, 0.6, 0.8, and 1.0, with the last corresponding to the fully deterministic system.

4.2. Speech Quality Evaluation with Objective Metrics

To thoroughly evaluate the system’s performance, we obtained objective metrics from the generated samples. First, we extracted the fundamental frequency (F0) contour from each sample. We aligned it with the one from the corresponding ground-truth sample using the Dynamic Time Warping (DTW) technique applied to the spectrograms. The durations were extracted from the MFA alignments. We subsequently calculated the following metrics: (a) Root Mean Square Error between the F0 contours (F0 RMSE). (b) Pearson correlation coefficient between the F0 contours (F0 Pearson). (c) Mel Cepstral Distortion (MCD), which is a measure of distance between high-level features extracted from the spectrogram, representing the properties of speech crucial for the recognition thereof. (d) Mean Relative Error between word durations (Duration MRE), which is the mean relative difference between the word durations of generated and ground truth samples. For each evaluated system, the values of each metric were averaged across all test samples.

The obtained results are presented in

Table 3. Mean values of F₀ RMSE, F₀ Pearson correlation and MCD, calculated over 100 test samples for ground-truth samples with HiFi-GAN Vocoder, baseline systems, and several configurations of the proposed system. The arrows next to the metric names refer to the ordering of the results in terms of their superiority (↓ indicates that the lowest value is the best, whereas ↑ prioritizes the highest value). The values marked in bold are the best among the compared setups.

System	F0 RMSE ↓	F0 Pearson ↑	MCD ↓	Duration MRE ↓
GT Vocoder	55.81	0.85	3.02	0.0202
Proposed Vanilla	110.83	0.476	6.62	0.1565
Proposed $\lambda$ = 0.2	108.69	0.484	6.63	0.1517
Proposed $\lambda$ = 0.4	106.10	0.505	6.60	0.1574
Proposed $\lambda$ = 0.6	108.65	0.488	6.52	0.1566
Proposed $\lambda$ = 0.8	107.52	0.497	6.50	0.1609
Proposed $\lambda$ = 1.0	108.54	0.490	6.54	0.1642
Tacotron2 + WaveRNN	106.49	0.496	7.22	0.1520
Tacotron2 + Griffin-Lim	108.56	0.477	27.24	0.1585

. We also included the metrics calculated for ground-truth spectrograms processed with the HiFi-GAN vocoder, enabling us to evaluate the bias introduced by the metric calculation.

4.3. Diversity of Expression Evaluated with Objective Metrics

The second experiment is designed to give us insight into the diversity of properties of generated speech. To this end, we obtained the same features as described in Section 4.2 from each of the 15 samples generated by each configuration of the system for the chosen textual prompts. We used the following metrics for the evaluation: (a) standard deviation of values belonging to the fundamental frequency contour; (b) standard deviation of the word durations. The metrics were calculated position-wise; that is, each text prompt yielded a sequence of standard deviation values (e.g., computed over all 15 instances of each word), which were subsequently averaged across all textual prompts.

The results are shown in

Table 4. Mean standard deviation of word duration and F0 calculated over 15 generated samples for several textual prompts chosen from the test data set.

System	F0	Word Duration
Proposed $\lambda =0.2$	58.07	0.0145
Proposed $\lambda =0.4$	52.82	0.0133
Proposed $\lambda =0.6$	49.31	0.0112
Proposed $\lambda =0.8$	42.27	0.0094

. The standard deviation of the duration of words for each word separately is shown in Figure 4.

4.4. Subjective Evaluation of Speech Quality and Diversity

We investigate the following hypotheses:

• H1: Speech generated by the proposed system is comparable to that produced by the baseline systems in terms of quality (lack of artifacts, quality of sound, lack of spelling errors).
• H2: Speech generated by the proposed system is comparable to that produced by the baseline systems in terms of expressiveness (natural intonation, consistent rhythm).
• H3: Manipulating the $\lambda$ hyperparameter of the system has visible influence on diversity of the high-level features of the produced speech (intonation, breaks, duration of words)

To assess the proposed system’s performance, we recruited 15 participants from our organization. The participants included adults in their twenties to their fifties. During the experiment, results were collected and stored anonymously; that is, each participant submitted a number of ratings without providing personal data.

Each participant was initially informed of all the definitions of the hypotheses H1–H3 and subsequently presented with 10 pairs of questions containing randomly sampled data. The first question contained a pair of recordings, one produced by the proposed system and the other by one of the baselines (Tacotron2 or Vanilla). The participant was asked to choose which recording sounds better in terms of (1) its quality and (2) its expressiveness. The second question contained another pair of recordings selected from all 15 outputs generated by a chosen system for a given data sample. The systems differed in their choice of the parameter $\lambda$, which had available values of 0.2, 0.6, and 1.0. The participant was asked to assign a numerical score of 0 to 2 to the recordings, where 0 indicated no difference in the high-level features, and 2 indicated a significant difference. The recordings for both questions were chosen from the sets described in Section 4.1. The participants were shown the textual input corresponding to the recordings they listened to.

The summarized results are shown in Figure 5a–c.

4.5. Case Study: Diversity of Word Durations

In addition to examining numerical metrics calculated across numerous test samples, we visualized the exact F0 contours obtained from each of the 15 samples generated for various system configurations. The extracted contours, together with the one obtained from the ground truth sample, for systems with the parameter $\lambda$ equal to 0.2 and 0.8 are plotted in Figure 6a,b, respectively.

5. Discussion

By evaluating the results with objective metrics, it is clear that the proposed system achieves the best performance across all metrics; however, for each metric, there is at least one configuration that scores worse than at least one baseline model.

The system with $\lambda =1.0$ never achieves the best score, which may indicate that the deterministic part of the GST Predictor is insufficient to model the distribution of style embeddings accurately. On the other hand, the system with $\lambda =0.2$ does not achieve the best score in each case, indicating that eager exploration of the distribution of possible embeddings corresponding to a given textual prompt leads to the generation of samples that are not close to the ground-truth ones. This, in turn, suggests two things: (1) the worthiness of performing subjective tests, which do not necessarily involve direct comparison with ground truth; (2) the worthiness of using advanced distribution modeling approaches in predicting style embedding (we listened to each generated sample and found that they do not contain significant flaws, which indicates that the style-embedding space contains multiple valid points, which allow the generation of good-quality speech).

In the evaluation of speech diversity, the standard deviation values are inversely correlated with the chosen values of the parameter $\lambda$. This indicates not only that treating the style-embedding prediction as a generative process has a real influence on the high-level properties of speech, but also that manipulating the system’s input parameters allows for smooth control over the diversity of the produced output. The influence of the control of diversity can be seen within a single sentence as well, as manipulating $\lambda$ has a clear impact on the diversity of the length of particular words.

The objective evaluation of different systems, presented in Figure 5a,b, shows that the proposed system outperforms the baselines both in terms of quality and expressiveness (respectively, H1 and H2). It is worth noting that a noticeable gap exists between the evaluated performance of the proposed and vanilla systems, about 80:20 and 75:25 for quality and expressiveness, respectively. Comparison with Tacotron2 also shows a slight advantage for the proposed system, with a 57:43 split in favor of quality and expressiveness. Figure 5c shows that there is a correlation between the choice of the $\lambda$ parameter and the difference assessed between the high-level features of compared generated recordings. Similarly to the results for the ground-truth recordings in Table 3, the system with $\lambda =1.0$ serves as a measure of the experiment’s inaccuracy. It should be used as a reference when assessing the remaining results.

It should be noted that incorporating the additional GST predictor components increases the inference latency. Obtaining a single second of waveform using the proposed method takes, on average, 206 milliseconds as compared to 98 milliseconds in the baseline setup. Although it means doubling the inference time, we do not consider this a serious drawback for the two following reasons: (1) the latency is still better than real time, which makes it suitable for application where the response time of the system is crucial; (2) the used components make our system parallel, which means the inference time does not grow rapidly with respect to the length of the input text, as in the autoregressive systems.

Both subjective and objective results indicate the following:

• The proposed method of explicit style modeling improves the quality of the generated speech, which proves its value in constructing robust TTS systems.
• The use of a diffusion framework in predicting the stylistic features has a visible influence on the diversity of high-level prosodic characteristics, such as pitch and word duration.
• Blending the regressive and diffusion-based predictors allows for smooth control of the produced speech’s prosody. As a result, by manipulating a single hyperparameter, we can decide whether, e.g., the pitch track will be centered near the average or deviate.

We consider the points above to prove that our system may be successfully applied in environments where natural and expressive speech generation is demanded, such as social media content creation or personal voice assistants. Our work reduces the gap between automatically generated and natural speech.

6. Conclusions

In this paper, we present our TTS system, which models speaking style by predicting GST-based style embedding in both deterministic and non-deterministic manners, using a transformer-based model and a diffusion model, respectively. The choice of architecture makes the system independent of any complex additional input; therefore, it lends itself better to real-time human–computer interfaces where the only available input data is of a single modality.

Through experiments involving the calculation of objective numerical metrics, we demonstrated that the system is capable of producing speech of comparable quality to that generated by another neural network-based TTS system. We also demonstrated that the system enables the generation of speech that is diverse with respect to its high-level prosodic features, such as fundamental frequency and word duration. By manipulating a single system’s input parameter, it is possible to adjust the level of diversity in the generated output.

In light of the statements above, we conclude that our proposed system enhances human–computer interaction by making the reception of the produced speech more natural and pleasant.

In this paper, we focus on presenting the contribution of our method to improving the expressiveness of the generated speech, both in terms of sentence-level diversity of prosodic features, as well as non-monotonous prosody between different generated sentences. We focused on a single-speaker single-sentence setup, which spans only a portion of the entire hierarchy of levels where the expressiveness can be improved, which we intend to explore in our future work. To name a few, the proposed method can be applied to a larger dataset of multi-speaker data with much more diverse prosody and speaker-specific characteristics. This would require using a stronger acoustic backbone, suited to large amounts of noisy data, and possibly a more powerful algorithm for learning probability distribution. Moreover, the demand for synthesizing long-form speech, e.g., audiobook paragraphs, influences us to think that our approach might be extended to such setups by incorporating a method of modeling inter-sentence style relationships, which would require using more complex style representations.