Speech-to-Speech Translation from Kazakh into Turkic Languages: Pretrained Speech Encoders, Turkic Speech Units, and Lessons from Negative Results

Abstract

Speech-to-speech translation (S2ST) among Turkic languages is rarely studied due to the scarcity of parallel speech corpora and the complex morphology. In this work, we develop low-resource S2ST systems translating from Kazakh to Turkish, Tatar, and Uzbek, and examine which architectural choices proved ineffective and why. Three architectures were trained on a synthetic parallel speech corpus: (i) a scratch-trained, morphology-aware Conformer encoder with a morpheme-level connectionist temporal classification (CTC) loss, a vector-quantized (VQ) codebook, and a semantic anchor; (ii) a pretrained Whisper encoder with a text decoder conditioned on the target language by feature-wise linear modulation (FiLM); and (iii) a text-mediated two-pass model that predicts Turkic speech units from the decoder states and converts them to speech with a Turkic unit vocoder. Text output was scored with BLEU and speech output with ASR-BLEU, that is, BLEU on automatic transcripts. The scratch-trained model and about ten variants scored ASR-BLEU 0. The Whisper-based speech-to-text model reached 19.49 BLEU on synthetic test speech and 3.56 on natural speech. The final speech-to-speech model reached 3.25 ASR-BLEU on about 1500 synthetic test utterances per language, 5.1 times the English-unit baseline, and 0.86 ASR-BLEU on natural speech. These scores show feasibility, not usable quality.

1. Introduction

Turkic languages are typologically and structurally related. This paper examines Kazakh, Tatar, Uzbek, and Turkish, which demonstrate similarities in agglutinativeness, word order, morphosyntactic dependencies, and several lexical and grammatical regularities. This creates an opportunity to build a unified cross-lingual architecture that leverages the shared structural features of Turkic languages, rather than training completely independent models for each language pair.

Despite rapid progress in speech recognition, machine translation, and neural speech synthesis, the issue of direct speech-to-speech translation between Turkic languages remains largely unexplored. Most modern speech-to-speech translation systems are currently geared towards high-resource languages such as English, Spanish, French, German, or Chinese. However, for Turkic languages—including Kazakh, Tatar, Uzbek, and Turkish—the number of dedicated models, corpora, and experimental studies remains notably limited. In particular, whilst preserving the semantics, acoustic naturalness, and distinctive features of the source language, there is a clear need for systems capable of performing direct S2ST translation without relying on text-to-text translation.

Speech-to-speech translation is mostly realized by a cascade architecture. First, the source speech is recognized by the Automatic Speech Recognition (ASR) system into text. Second, it is translated from the source language into the target language by using machine translation systems. Third, the translated text is converted into speech. This approach has several practical advantages. Each component can be trained separately, existing models can be used, and evaluation can be performed using independent metrics. However, cascade systems have significant drawbacks. Errors in the ASR stage are passed on to the machine translation module, whilst translation errors degrade the quality of the synthesized speech. This accumulation of errors is particularly problematic in low-resource, morphologically complex languages.

In recent years, significant advances in speech-to-speech translation have been made. These have been driven primarily by end-to-end neural network models, self-supervised speech learning methods, and multilingual models. However, most existing systems are only designed for resource-rich languages. Resource-poor, morphologically complex languages, such as Turkic languages, are often overlooked. In this work, we combine ideas from several approaches: direct speech-to-speech translation, morphology-aware language processing, discrete speech representations, and multilingual semantic embeddings. This section provides an overview of the necessary research for each of these directions.

This paper studies speech-to-speech translation from Kazakh into three target Turkic languages: Turkish, Tatar, and Uzbek. The starting point was MorphoTurkic-S2ST, a model trained from scratch that was intended to build a shared cross-lingual acoustic-semantic space reflecting the morphology of Turkic languages, with the target language selected by language conditioning. When this model failed, we diagnosed the causes and replaced its parts step by step with pretrained components, which led to a working text-mediated two-pass system.

The final system differs from a classical cascade of ASR, machine translation (MT), and text-to-speech (TTS) systems in that all stages belong to one jointly trained model: the hidden states of the text decoder, not only its output text, condition the speech-unit predictor. Because it still decodes text internally, it is a two-pass model in the sense of UnitY, not a text-free direct model.

The main innovations of the work are summarized in the following points:

• A reproducible negative result and its diagnosis. A morphology-aware architecture trained from scratch (a Conformer encoder with a morpheme-level connectionist temporal classification (CTC) head, a vector-quantized (VQ) codebook, and a semantic anchor), together with about ten variants, produced no usable translation in any experiment. In the ablation, only the morpheme CTC head improved the training objective; the VQ codebook had no measurable effect, and the semantic anchor made results worse. We therefore report these components as tested design options, not as validated contributions.
• Five specially designed control experiments on the scratch-trained model made exposure bias, over-smoothing, the form of the outputs, and a language-model shortcut unlikely causes of its failure. The most consistent explanation with these controls is the absence of a target-language representation that can be translated directly. This cause is inferred from the controls rather than measured directly. The exposure-bias gap reported later for the final pretrained two-pass system (Section 4.7) concerns a different model and does not contradict these controls.
• A working S2TT system on frozen Whisper. Relying on a pretrained, content-rich representation eliminates the need to train speech understanding from scratch under data-scarce conditions, and quality increases monotonically to 19.49 BiLingual Evaluation Understudy (BLEU) and 49.44 chrF++ on the synthetic test sample.
• A unified decoder with language-specific Feature-wise Linear Modulation (FiLM) conditioning: a single model translates into Turkish, Tatar, and Uzbek, with the target language specified by a control vector. This is a text-mediated two-pass design, in which text is predicted first and speech units second: the FiLM decoder’s text pass is not merely an auxiliary loss, but instead provides the conditioning signal to which the unit predictor cross-attend. It is therefore a required stage of inference rather than a discardable side output.
• The path to end-to-end speech synthesis and its solution: diagnosing failures in acoustic encoders and vocoders ‘from scratch’—a key transition to single-pass Turkic units (mHuBERT-147) with a Turkic-trained vocoder, which raised the end-to-end ASR-BLEU from 0 to 3.25 on the synthetic test sample. Absolute quality remains low (0.86 ASR-BLEU on natural speech), so the system shows technical feasibility rather than usable translation.
• In this study, we evaluated performance using precisely calculated and measured metrics. Specifically, we used the BLEU, chrF, chrF++, TER, and COMET metrics to assess the quality of the textual results. To evaluate the speech results, we used ASR-BLEU, ASR-chrF++, TER, and the character error rate (CER). To assess audio quality, we used the MOS, STOI, PESQ, and SI-SDR metrics of the torchaudio SQUIM system. Additionally, to assess speech naturalness, we conducted a short MOS evaluation with participants who were not informed about the system. To characterize the system’s real-time performance, we measured the RTF metric and the latency of its individual components. We considered uncalculated BLASER 2.0, SONAR-based semantic similarity, and human evaluation metrics for translation content preservation and comprehensibility separately. The application of these evaluation methods was identified as a prospective direction for future research (Section 4.2 and Section 6).

The work addresses three research questions. RQ1: Is a morphologically informed architecture trained from scratch sufficient for Kazakh-to-Turkic S2ST under low-resource conditions? Answer: No (Section 4.4). RQ2: What accounts for its failure—output representation, exposure bias, over-smoothing, or the absence of a transferable representation? Answer: Five controlled experiments make the first three explanations unlikely; the absence of a transferable representation is the explanation most consistent with the evidence, although it is inferred rather than measured directly (Section 4.4). RQ3: Which components are needed for non-zero end-to-end speech output? Answer: A pretrained representation, a content/rendering decomposition, and a Turkic-specific quantizer–vocoder stack (Section 4.5 and Section 4.6).

The article is organized as follows. Section 2 provides an overview of related work in four main areas: Turkic morphology in Natural Language Processing (NLP) and ASR tasks, direct speech-to-speech translation systems, the use of morphological analysis in neural models, and discrete speech representations and VQ codebooks in S2ST. Section 3 describes the data, pre-processing steps, evaluation metrics, and three architectures investigated in sequence: a morphologically aware model trained from scratch; a working S2TT system based on a frozen Whisper encoder and a FiLM decoder; and a final end-to-end Speech-to-Unit Translation (S2UT) system with Turkic speech units and a Turkic vocoder. Section 4 presents the experimental setup and results: an overall comparative evaluation of the development stages; S2TT results; end-to-end S2UT results; an analysis of negative experiments; and an ablation study. Section 5 discusses the results, their interpretation, the practical implications, and the identified limitations of the proposed approach. Finally, Section 6 summarizes the research and formulates the main conclusions regarding the feasibility of direct end-to-end speech-to-speech translation for low-resource Turkic languages.

2. Related Works

Recent advances in speech-to-speech translation have been driven by progress in end-to-end neural architectures, self-supervised speech representation learning, and multilingual modeling. However, most existing systems have been developed primarily for high-resource languages and do not explicitly address the challenges posed by morphologically rich, low-resource language families such as the Turkic languages. The proposed MorphoTurkic-S2ST architecture combines ideas from several research directions, including speech-to-speech translation, morphology-aware language processing, discrete speech representations, and multilingual semantic embedding models. This section reviews relevant studies in these areas and highlights their relevance to the proposed framework for direct speech translation between Turkic languages.

2.1. Turkic Morphology in NLP and ASR

Turkic languages have a rich morphological structure and belong to an agglutinative language family. A word can contain numerous suffixes defining grammatical categories such as case, number, tense, voice, and person. This, in turn, can lead to a high degree of variability in word forms and pose significant challenges for neural models based on a fixed vocabulary [1]. In neural machine translation (NMT), this problem has been extensively addressed using the Kazakh language as an example, where data sparsity and the prevalence of rare word forms often lead to incorrect generation of morphological sequences [2].

To reduce vocabulary sparsity, byte pair encoding (BPE) is widely used. However, studies of Kazakh-English and Kazakh-Russian translation have shown that standard BPE, applied without regard to morphological boundaries, performs worse with rare suffixal forms than morphologically motivated segmentation approaches [3]. Tukeev and Karibayeva [4] proposed the complete set of endings (CSE) for the Kazakh language as a linguistic resource for generating additional training examples, partially addressing the problem of the lack of parallel data.

In automatic speech recognition (ASR), multilingual training across Kazakh, Bashkir, Kyrgyz, Sakha, and Tatar has demonstrated significant reductions in word error rate (WER) by transferring knowledge of morphological patterns among closely related languages [5]. Orel et al. [6] further showed that Kazakh can serve as an effective pivot language for developing ASR systems for nine low-resource Turkic languages through cross-lingual transfer learning.

Despite these advances, existing studies primarily use morphological information in text-based NLP and ASR tasks. The integration of morphology-aware representations directly into end-to-end speech-to-speech translation models for Turkic languages remains underexplored.

2.2. Direct S2ST Systems: Translatotron, SeamlessM4T, and Related Architectures

The task of direct speech-to-speech translation (S2ST) aims to convert speech in a source language directly into speech in a target language within a single neural model, without relying on intermediate text representations. Cascade systems (ASR → MT → TTS) accumulate errors at each stage and are also unable to preserve paralinguistic features such as speaker identity and prosody [7].

Previous research has demonstrated that synthetic speech–translation pairs generated using pretrained Machine Translation (MT) and Text-to-Speech (TTS) models can substantially improve end-to-end speech translation systems, highlighting the importance of leveraging auxiliary linguistic knowledge when parallel speech translation data are scarce [8]. Translatotron 2 [9] addressed the main limitations of its predecessor by introducing a speech encoder, a linguistic decoder, a mel-spectrogram synthesizer, and a unified attention mechanism, achieving performance closer to cascade systems with improvements of up to +15.5 BLEU. Lee et al. [10] proposed a discrete unit-based approach called Speech-to-Unit Translation (S2UT). This approach converts target speech into discrete units using a self-learning HuBERT model, which is then predicted as a sequence of tokens. This paradigm allows speech translation even for languages without a standardized writing system.

SeamlessM4T [11] combines ASR, S2T, T2S, S2S, and T2T tasks for up to 100 languages in a single architecture. The model was trained on 1 million hours of speech data using w2v-BERT 2.0 and the SeamlessAlign corpus (406,000 h), achieving a 20% BLEU gain over the previous SOTA on FLEURS. Kazakh is included in the list of supported languages, but the morphological specificity of Turkic languages is not taken into account in the architecture.

UnitY [12] introduces a two-pass architecture that combines text generation with discrete unit prediction, enabling the use of text supervision in direct speech translation. The proposed model achieves higher translation accuracy and substantially faster decoding than single-pass approaches. MorphoTurkic-S2ST follows this same two-pass paradigm: the FiLM text decoder is not an auxiliary loss but the mechanism by which the single-stream unit predictor receives its conditioning signal, so translation quality and unit prediction quality are coupled through the same intermediate text pass. TranSpeech [13] addresses the problem of acoustic multimodality of discrete units using bilateral perturbations and non-regressive decoding, achieving 21x faster inference than autoregressive models.

Kozhirbayev and Islamgozhayev [14] presented the first cascade speech translation system specifically for the Kazakh language (Kazakh → Russian) based on the ISSAI corpus. A subsequent end-to-end comparison of the cascade and direct approaches [15] confirmed that direct systems demonstrate competitive results for low-resource languages, provided sufficient pretraining data is available.

Currently, modern direct S2ST architectures provide high translation efficiency and are typically optimized for general multilingual contexts. However, they do not demonstrate such efficiency for agglutinative language families, such as Turkic, with their specific morphological properties. This has stimulated research into morphologically oriented representations and segmentation methods that can better capture the internal structure of words in morphologically rich languages.

2.3. Morphological Analysis in Neural Models: Morfessor and Derivatives

Creutz and Lagus [16] introduced Morfessor, an unsupervised morphological segmentation model that learns a lexicon of morphs directly from raw text. The approach was shown to be particularly effective for morphologically rich languages, making it a valuable foundation for incorporating morphological information into language processing systems. Morfessor 2.0.6 [17] provides open-source Python 3.14 implementationand supports semi-supervised learning. Morfessor FlatCat [18] extends the base model by using Hidden Markov Models (HMM) to distinguish between suffixes, roots, and prefixes, improving segmentation accuracy for agglutinative languages. Cognate Morfessor [19] introduces a multilingual variant of the method that ensures consistent segmentation of cognates across related languages—a property particularly valuable for the Turkic family with its high level of lexical similarity.

Studies on the application of morphological segmentation in NMT for agglutinative languages [20] show that morphologically motivated tokenization consistently outperforms standard BPE on Turkish-English and Uyghur-Chinese pairs. The work [21] proposes a knowledge distillation method using an auxiliary morphological decoder: the main forward decoder is enhanced by a “reverse decoder” and a suffix information decoder, which improves the translation quality for Kazakh and Turkish. A similar approach—a dual encoder for contextual and morphological features—is applied in [22] to the translation of agglutinative languages into Chinese.

For agglutinative languages, morphology-aware segmentation techniques improve the linguistic representation. But they are mostly applied to symbolic representations of language, such as text or subword units, rather than acoustic speech inputs. This highlights the need for approaches that operate directly on speech signals including discrete representations based on vector quantization and codebook learning in speech-to-speech translation systems.

2.4. Discrete Speech Representations: VQ-VAE and Codebooks in S2ST Systems

The foundation of discrete speech representations is VQ-VAE [23], a variational autoencoder with vector quantization in which continuous hidden states are mapped to the nearest vectors from a finite codebook. In the S2ST task, VQ-VAE enables a compact representation that allows for language-neutral encoding. UWSpeech [24] applies VQ-VAE to the translation of unwritten languages, separating the linguistic content and acoustic characteristics of the voice in the latent space.

The self-supervised models HuBERT [25] and wav2vec 2.0 [26], together with k-means clustering, generate speech tokens used as target units in S2UT systems. EnCodec [27] introduces residual vector quantization (Residual VQ), a hierarchical scheme in which each codebook level refines the residual of the previous one, providing high-quality speech reconstruction with a compact representation.

A systematic review of direct S2ST systems [28] categorizes speech quantization approaches into three main groups: VQ-VAE-based quantizers optimized for spectrogram reconstruction, linear quantizers, and Transformer-based quantizers employing non-causal architectures. The review highlights the effectiveness of Transformer quantizers, particularly in high-resource settings, where they often achieve superior translation performance among discrete representation methods.

For cross-lingual S2ST, recent studies have explored the use of shared discrete speech units to capture phonetic regularities across multiple languages. However, most existing approaches learn language-independent or largely language-agnostic representations without explicitly modeling linguistic relationships between language families, highlighting the potential of incorporating typological and phonological similarities to improve representation learning in multilingual speech models.

3. Speech-to-Speech Translation Architecture of MorphoTurkic-S2ST

3.1. Data and Preprocessing

The study used the dataset_214k corpus, a four-way parallel speech corpus (kaz/tat/tur/uzb) containing 213,969 sentences in each language, aligned by sentence identifier. For the translation direction kaz→{tur, tat, uzb}, unified splits were used: training set—513,525 pairs, validation set—64,191 pairs, and test set—64,191 pairs. Sample identifiers are recorded and provided in the manifest files train_tur_kaz.tsv, valid_tur_kaz.tsv, and test_tur_kaz.tsv. All test-split results in this paper are computed on one fixed random sample of the test split, shared by all S2TT and end-to-end evaluations: 1484 Turkish, 1496 Tatar, and 1487 Uzbek utterances (seed 0). The audio data is in WAV format (mono, 16 kHz sampling frequency) and undergoes the same preprocessing: log-mel spectrogram calculation with 80 mel filters, STFT window size (n_fft, win_length) of 1024 and step (hop_length) of 256; the maximum sample duration is limited to 30.0 s; logarithm is taken using the formula log(clamp(mel, min = 1 × 10⁻⁵)) (natural logarithm); the data is cached in torch.float16 format.

The Kazakh texts were collected from the web and retain URL-level attribution in dataset_214k_metadata.csv: the top domains are kk.wikipedia.org (accessed on 5 September 2026) (20,541 sentences), adilet.zan.kz org (accessed on 5 September 2026) (legislative texts; 11,278), www.zharar.com org (accessed on 5 September 2026) (educational materials; 10,130), stud.kz (7309), egemen.kz org (accessed on 5 September 2026) (news; 4276), and adebiportal.kz org (accessed on 5 September 2026) (literary texts; 3414). The translations into the target languages were generated using a fine-tuned NLLB model [29]. The target speech was synthesized using Meta’s Massively Multilingual Speech (MMS) [30], with a separate model for each language (facebook/mms-tts-tur, facebook/mms-tts-tat, and facebook/mms-tts-uzb); each model is single-speaker, i.e., one speaker per language.

A check for overlap with the Kazakh FLORES-200 devtest set found no exact matches (0 out of 1012 sentences). We extended this check to near-duplicates. After normalization (Unicode NFKC, lowercasing, removal of punctuation), each evaluation sentence was compared with every unique training sentence by the Jaccard similarity of character 5 g sets. No FLORES-200 devtest sentence and none of the 349 FLEURS sentences used for the natural-speech evaluation has a training match above 0.32, and the Turkish and Tatar references show no training match of 0.5 or higher either. The synthetic test split is less clean, because the corpus was split randomly at the sentence level without text deduplication: 1.5% of its Kazakh sentences have an exact normalized duplicate in the training set and 2.2% have a near-duplicate with a similarity of 0.9 or higher (1.3% and 2.1% in the evaluation sample). This can raise the synthetic test scores slightly but does not affect the natural-speech results. FLEURS and FLORES were not used to train or fine-tune any of our models.

Because the target texts are machine translations, the models learn the wording of the system that produced them, including its errors, and the synthetic test references share that wording. BLEU on the synthetic test sample therefore rewards reproducing it, whereas the FLEURS evaluation uses human translations. Scores on the synthetic test sample are an optimistic estimate, and the natural-speech results are the better guide to generalization (Section Sources of the Synthetic-to-Natural Gap).

3.2. Overall Architecture

The methodology follows three experimental stages:

• A scratch-trained morphology aware S2ST architecture;
• A frozen Whisper-based S2TT system;
• A final S2UT system based on Turkic speech units and a Turkic-trained vocoder.

3.3. Scratch-Trained Morphology-Aware Model

The initial concept of the study was to construct a unified cross-lingual acoustic-semantic space combining three key components: a morphology-oriented Conformer encoder (Novelty A), a cross-lingual VQ codebook (Novelty B), and a semantic anchoring mechanism (Novelty C). The proposed architecture was envisioned to transform Kazakh speech into a common cross-lingual representation, from which it could then directly generate a speech signal in the target language within a single neural network, eliminating the need for separate recognition, translation, and speech synthesis modules. However, as shown in Section 5.1, this architecture failed to achieve acceptable translation quality, resulting in an ASR-BLEU score of 0. Despite this negative result, the model is described in full, as the analysis of the identified limitations and reasons for its failure served as the basis for the development of subsequent, practically applicable architecture. Figure 1 below diagrams this architecture in full.

Figure 1. Architecture of Approach 1 (trained from scratch): morphologically aware Conformer encoder with morpheme-based CTC head (Novelty A), cross-lingual VQ codebook (Novelty B), non-autoregressive Mel decoder with language-based FiLM and semantic LaBSE anchor (Novelty C).

Figure 1 shows the architecture, trained from scratch and taking morphology into account. It includes an acoustic front-end, a Conformer encoder with a morpheme-based CTC branch, a cross-lingual VQ codebook, a decoder with a FiLM condition, a semantic anchor, and a speech synthesis module. Each of these components is described in detail in the following subsections.

3.3.1. Acoustic Front-End and Subsampling

In the first stage, the input log-mel spectrogram is passed through an acoustic front-end with convolutional subsampling. The original representation has dimension (80, T), where T can be quite large, especially for long speech fragments.

To reduce computational load, a 4× convolutional compression is applied in time, after which the representation has dimensions (T/4, 512). The factor of 4× has been chosen to strike a balance between speed and quality. If the compression is weaker, there are still too many time steps, and self-attention in the Conformer encoder and the Transformer decoder becomes more computationally expensive. If the compression is stronger, important phonetic and prosodic details required for speech-to-speech translation may be lost. A 4× compression significantly shortens the sequence, whilst retaining sufficient acoustic information for subsequent morphologically aware encoding.

After subsampling, each step is represented by a 512-dimensional vector. This space is used as input to the Conformer encoder, which is designed to extract not only acoustic but also more abstract linguistic features of the original Kazakh speech.

3.3.2. Conformer Encoder

The first key novelty of MorphoTurkic-S2ST lies in the use of a morphologically aware Conformer encoder. The encoder consists of 12 layers with a hidden state dimension of d = 512 and 8 attention heads. The Conformer was chosen as the base architecture because it combines the advantages of self-attention and convolutional modules: self-attention allows for the modeling of long-term dependencies in speech sequences, whilst convolutional blocks effectively capture local phonetic and acoustic patterns.

Self-attention is structured in the standard way for Transformers. The input data H is transformed into three sets of vectors—queries (Q), keys (K), and values (V)—via trainable weight matrices (1):

$$\mathrm{Q}=\mathrm{H}{\mathrm{W}}^{\mathrm{Q}},\mathrm{K}=\mathrm{H}{\mathrm{W}}^{\mathrm{K}},\mathrm{V}=\mathrm{H}{\mathrm{W}}^{\mathrm{V}},$$

(1)

where $H$ is the input hidden representation, $Q$, $K$, and $V$ are the query, key, and value matrices, $W^Q$, $W^K$, and $W^V$ are trainable projection matrices.

Then, for each query, the model calculates how ‘similar’ it is to each key, and the sum of the values is weighted according to this similarity—this is how the model decides which parts of the input to pay more attention to (2):

$$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V,$$

(2)

where $d_k$ is the dimension of a single attention head, and $K^T$ is the transpose of the key matrix, used to compute query–key similarity scores in the self-attention mechanism.

The encoder uses 8 parallel attention heads, the results of which are concatenated and projected via the output matrix (3):

$$MultiHead\left(H\right)=Concat\left({head}_1,\dots ,{head}_8\right)W^O,$$

(3)

where ${head}_i$ denotes the output of the i-th attention head, and $W^O$ trainable output projection matrix of multi-headed attention.

The dimension of a single head is determined by dividing the total model dimension by the number of heads (4):

$$d_k=512/8=64,$$

(4)

To account for the agglutinative nature of Turkic languages, an auxiliary CTC head, trained on morpheme units, is added to the encoder. The morpheme dictionary is generated using Morfessor 2.0.6 and contains approximately 10,000 entries.

The morphemes provide clearer insight into the structure of Turkic words rather than word forms. In agglutinative languages such as Kazakh, Tatar, Uzbek, and Turkish, a word often consists of a root and several affixes, where affixes may add new grammatical or semantic meaning to the word.

The CTC head predicts sequences of morphemes instead of whole words. It estimates the probability of the correct morpheme sequence based on the encoder’s hidden representations. In doing so, CTC considers all possible alignments between acoustic features and morphemes (5):

$$P\left(y^{morph}|H_{enc}\right)={\sum }_{\pi ϵ{\mathcal{B}}^{-1}\left(y^{morph}\right)}{\displaystyle {\prod }_{t=1}^{T/4}}P\left({\pi }_t|H_{enc}\right),$$

(5)

where $y^{morph}$ is the target morpheme sequence, $H_{enc}$ output hidden representations of the Conformer encoder, π represents a valid CTC alignment path, ${\mathcal{B}}^{-1}\left(y^{morph}\right)$ is the set of all alignment paths corresponding to $y^{morph}$, ${\pi }_t$ the symbol or morpheme predicted at time step t;

In other words, the model sums the probabilities of all valid paths π that, after removing repetitions and empty tokens, yield the desired morpheme sequence. The CTC loss is calculated as the negative logarithm of this probability (6):

$$L_{CTC}=-logP\left(y^{morph}|H_{enc}\right),$$

(6)

where $L_{CTC}$ is the Connectionist Temporal Classification loss function.

It is important to note that this CTC branch is not used as the main ASR module for final speech recognition. It serves as an additional training mechanism that helps the encoder better preserve information about speech morphemic structure.

This approach helps the model rely less on acoustic features and better convey morphologically significant information across related Turkic languages.

3.3.3. Cross-Lingual VQ Codebook

The second key element of novelty is the use of a cross-lingual VQ codebook, which represents a common discrete intermediate space for the three target languages. After passing the Conformer encoder, the continuous latent representation of dimension (T/4, 512) is fed to a vector quantization module. The codebook contains 512 discrete vectors and is updated using an exponential moving average (EMA). The quantization procedure and its training objective are formalized below, where each encoder hidden state is assigned to the nearest codebook vector, replaced by the corresponding discrete representation, and optimized using the VQ reconstruction and commitment losses:

$$\mathsf{ℇ}=\left\{e_1,e_2,\dots ,e_{512}\right\},$$

(7)

$${\mathcal{z}}_t=\mathrm{a}\mathrm{r}\mathrm{g} \underset{k}{\min }{‖h_t-e_k‖}_2,$$

(8)

$$q_t=e_{{\mathcal{z}}_t},$$

(9)

$$Q_{VQ}=\left\{q_1,q_2,\dots ,q_{T/4}\right\}\in {\mathbb{R}}^{T/4\times 512},$$

(10)

$${\mathcal{L}}_{VQ}={\displaystyle \sum _{t=1}^{T/4}}{‖sg\left[h_t\right]-q_t‖}_2^2+\beta {‖h_t-sg\left[q\right]‖}_2^2,$$

(11)

where $\mathsf{ℇ}$ is the codebook containing 512 learnable code vectors, $h_t$ denotes the encoder hidden representation at time step t, ${\mathcal{z}}_t$ is the index of the nearest code vector, $q_t$ is the quantized representation, $Q_{VQ}$ is the sequence of quantized vectors, $sg$[⋅] denotes the stop-gradient operator, and β is the commitment loss coefficient.

The idea behind the common VQ codebook is to separate the high-level content of the source speech from the specifics of its linguistic implementation in the target languages. Since Turkish, Tatar, and Uzbek belong to the Turkic language group, they may share some acoustic-semantic and morphosyntactic regularities. Therefore, instead of training separate intermediate spaces for each language pair, we propose using a single codebook that forms a common discrete representation for all target languages.

A quantized memory of dimension (T/4, 512) is used as an interlingual intermediate representation, preserving semantically relevant information from the original Kazakh utterance without being bound to a specific target language. In a subsequent step, the Transformer decoder aggregates this representation with the target language embedding to generate a mel spectrogram in Turkish, Tatar, or Uzbek.

${\mathcal{L}}_{VQ}$ loss is used to stabilize the vector quantization process and align the encoder’s continuous representations with discrete codes. It helps prevent unstable training, where the model might ignore the codebook or use only a limited subset of available codes. Thus, the VQ codebook not only serves a compression function but also contributes to the structuring of the interlingual space.

3.3.4. FiLM-Conditioned Decoder

After the quantized memory is formed, the target mel spectrogram is generated using a Transformer decoder consisting of 12 layers. Each layer contains causal self-attention, which provides autoregressive modeling of previously generated acoustic features, and cross-attention, that attends to the VQ memory formed by the encoder (12):

$$CrossAttention=Attention\left(Q_{dec},K_{VQ},V_{VQ}\right),$$

(12)

where $Q_{dec}$ denotes the query vectors generated by the decoder, while $K_{VQ}$ and $V_{VQ}$ are the key and value vectors derived from the vector-quantized representations. The cross-attention mechanism enables the decoder to attend to the discrete linguistic representations extracted by the VQ module.

To enable speech generation in three different target languages, a language-conditioning mechanism is introduced into the decoder. Each target language l, where $l\in \left\{Turkish,Tatar,Uzbek\right\}$ is represented by a trainable language embedding (13):

$$e_l=Embedding\left(l\right),$$

(13)

where l—target language; $e_l$—the trainable embedding of target language l.

This embedding is used within the FiLM modulation framework to compute layer-specific scaling and shifting parameters. For the i-th decoder layer, the parameters are obtained as follows (14) and (15):

$${\gamma }_l^{\left(i\right)}=W_{\gamma }^{\left(i\right)}{e}_l,$$

(14)

$${\beta }_l^{\left(i\right)}=W_{\beta }^{\left(i\right)}{e}_l,$$

(15)

where, $W_{\gamma }^{\left(i\right)}$ and $W_{\beta }^{\left(i\right)}$ are learnable matrices that transform the language embedding el into parameters ${\gamma }_l^{\left(i\right)}$ and ${\beta }_l^{\left(i\right)}$.

The hidden representation of the decoder is then modulated by applying feature-wise scaling and shifting:

$${\tilde{H}}^{\left(i\right)}={\gamma }_l^{\left(i\right)}\odot H^{\left(i\right)}+{\beta }_l^{\left(i\right)},$$

(16)

where, i is the decoder layer number; $H^{\left(i\right)}$ is the hidden representation at the i-th decoder layer before FiLM modulation; ${\tilde{H}}^{\left(i\right)}$ is the hidden representation after language FiLM modulation; ${\gamma }_l^{\left(i\right)}$ controls the language-specific scaling of hidden activations, while ${\beta }_l^{\left(i\right)}$ provides the corresponding shift, ⊙ is the componentwise multiplication.

This allows the decoder to adapt the generation process to the phonetic, prosodic, and rhythmic properties of the selected target language.

This approach removes the need to train three independent decoders for Turkish, Tatar, and Uzbek. Instead, a single shared decoder is trained jointly across all target languages, while language-specific behavior is controlled through the embedding $e_l$ and FiLM modulation. This is especially useful in low-resource settings, where separate models for each language would increase the number of parameters and reduce the amount of effective training data per model.

In addition, a stop token is predicted at each decoding step to determine when generation should terminate:

$$p_t^{stop}=\sigma \left(W_s{h}_t+b_s\right),$$

(17)

where $W_s$ and $b_s$ are trainable parameters of the stop-token prediction layer, which maps the decoder hidden state hth_tht to a scalar stop probability. The sigmoid function $\sigma$ converts this scalar value into the probability $p_t^{stop}$ that generation should terminate at time step t, $h_t$ is the decoder hidden state at time step t, and $p_t^{stop}$ is the probability that generation should stop.

The synthesis process is terminated when $p_t^{stop}$ exceeds a predefined threshold. This mechanism supports stable generation of speech sequences with variable duration and helps prevent both premature termination and excessive continuation of the output speech.

3.3.5. Semantic Anchor

The third contribution of MorphoTurkic-S2ST is the introduction of a semantic anchor based on a frozen multilingual embedding model, such as SONAR or LaBSE. The main function of this component is to ensure semantic similarity between the source utterance and the generated target translation.

$$s_{text}^{\mathcal{l}}=f_{sem}\left(y_{text}^{\mathcal{l}}\right),$$

(18)

$$s_{model}=g\left(Q_{VQ}\right),$$

(19)

$$\cos \left(s_{model},s_{text}^{\mathcal{l}}\right)={\displaystyle \frac{s_{model}^T{s}_{text}^{\mathcal{l}}}{‖s_{model}‖‖s_{text}^{\mathcal{l}}‖}},$$

(20)

$${\mathcal{L}}_{sem}=1-\cos \left(s_{model}^T,s_{text}^{\mathcal{l}}\right),$$

(21)

In direct S2ST systems, there is a risk that the model may generate acoustically natural speech that partially distorts the original meaning. This problem is especially acute when working with low-resource languages, where the amount of parallel speech data is limited and end-to-end model training can be unstable. A semantic anchor helps mitigate this risk by placing an additional constraint on the model’s latent representation.

Formally, let ${\mathit{s}}_{model} ∊ {\mathbb{R}}^d$ be the latent representation obtained within the S2ST model, and ${\mathit{s}}_{anchor} ∊ {\mathbb{R}}^d$ be the embedding of the target text extracted by a frozen SONAR or LaBSE model. The semantic similarity between them is calculated as cosine similarity (22):

$$\mathrm{sim}\left({\mathit{s}}_{model},{\mathit{s}}_{anchor}\right)={\displaystyle \frac{{\mathit{s}}_{model}^{\mathsf{T}}{\mathit{s}}_{anchor}}{‖{\mathit{s}}_{model}‖‖{\mathit{s}}_{anchor}‖}},$$

(22)

where ${\mathit{s}}_{model}^{\mathsf{T}}{\mathit{s}}_{anchor}$ is the scalar product of two vectors, $‖{\mathit{s}}_{model}‖‖{\mathit{s}}_{anchor}‖$ which are their Euclidean norms, and d is the dimension of the embedding space (768 for LaBSE, 1024 for SONAR).

The corresponding component of the loss function minimizes the semantic discrepancy (23):

$${\mathcal{L}}_{sem}=1-\mathrm{sim}\left({\mathit{s}}_{model},{\mathit{s}}_{anchor}\right),$$

(23)

where the value ${\mathcal{L}}_{sem}=0$ corresponds to a complete semantic match of the representations, and ${\mathcal{L}}_{sem}=2$ corresponds to the maximum discrepancy (oppositely directed vectors). The higher the cosine similarity value, the better the model preserves the semantic content of the translation. Since the embedding model remains frozen, it serves as an external, stable reference point and does not adjust for the main S2ST model’s errors.

3.3.6. Synthesis Head and Vocoder

At the final stage, the generated mel spectrogram is passed to the High-Fidelity Generative Adversarial Network (HiFi-GAN) vocoder [31], which converts it into the target speech waveform. In the proposed architecture, the vocoder is conditioned on the target-language embedding rather than used as a fully language-independent universal module. Thus, the synthesis process takes into account the target language of the final speech—Turkish, Tatar, or Uzbek—as formalized (24):

$${\widehat{\mathit{y}}}_{audio}^{\mathcal{l}}=HiFiGAN\left({\widehat{\mathit{Y}}}_{mel}^{\mathcal{l}},{\mathit{e}}_{\mathcal{l}}\right),$$

(24)

where y is the generated speech signal in the target language, i.e., the resulting waveform; ${\widehat{\mathit{Y}}}_{mel}^{\mathcal{l}}$ is the predicted mel spectrogram obtained at the output of the NAR mel decoder; ${\mathit{e}}_{\mathcal{l}}$ is the target language embedding $\mathcal{l}$, where $\mathcal{l}$ can correspond to Turkish, Tatar, or Uzbek.

$HiFiGAN$()—the conditional HiFi-GAN vocoder that converts the mel spectrogram and language input into an audio signal.

It is important to adapt the vocoder for each language, because although Turkic languages are similar, their pronunciation, intonation, duration, and speech rhythm differ. If a single vocoder is used for all languages, these differences are lost, and the speech may not sound natural. By contrast, the conditional HiFi-GAN can preserve the shared mechanism while accounting for each language’s distinctive acoustic features to produce natural speech.

Vocoder adaptation across the three target languages can be performed jointly using language embeddings or lightweight language-specific adaptation layers. This approach enables more efficient use of the available multilingual data and avoids training three independent vocoders from scratch. As a result, the final system can generate more natural and language-specific speech while preserving the advantages of a unified S2ST architecture for Turkic languages.

3.3.7. Training Objective

A composite loss function combining several components is used to train the proposed MorphoTurkic-S2ST architecture. As the model addresses the task of direct speech-to-speech translation, training must simultaneously account for the acoustic quality of the target speech, the correctness of generation completion, the encoder’s morphological awareness, the stability of the discrete VQ representation, and the preservation of the semantic content of the translation. Therefore, the final loss function is defined as a weighted sum of several sub-criteria (25):

$${\mathcal{L}}_{total}={{\lambda }_{mel}\mathcal{L}}_{mel}+{\mathit{\lambda }}_{CTC}{\mathcal{L}}_{CTC}+{\mathit{\lambda }}_{VQ}{\mathcal{L}}_{VQ}+{\mathit{\lambda }}_{sem}{\mathcal{L}}_{sem}+{\mathit{\lambda }}_{stop}{\mathcal{L}}_{stop},$$

(25)

Here, ${\mathcal{L}}_{mel}$ is the main component and is responsible for the similarity between the predicted and ground-truth mel-spectrograms of the target speech. It guides the decoder to generate an acoustically correct representation, which the HiFi-GAN vocoder then uses to synthesize the audio signal.

Where ${\mathcal{L}}_{mel}$ is the fundamental acoustic component (L1-distance between the predicted and reference mel spectrograms).

The LCTC component ${\mathcal{L}}_{CTC}$ is used as auxiliary morpheme regularization for the Conformer encoder. It helps preserve the hidden representations of the morphemic structure of Kazakh speech, which is particularly important for agglutinative Turkic languages.

The ${\mathcal{L}}_{VQ}$ component is responsible for the stable training of the cross-lingual VQ codebook. It maps the encoder’s continuous outputs to discrete codes and prevents the codebook from being used in an unstable manner.

The ${\mathcal{L}}_{sem}$ component is linked to the semantic anchor and is used to preserve semantic proximity between the model’s hidden representation and the target text. This reduces the risk of semantic distortion during direct speech-to-speech translation.

Finally, ${\mathcal{L}}_{stop}$ trains the decoder to identify the end of the generated speech sequence correctly. The coefficients ${\mathit{\lambda }}_{CTC}$, ${\mathit{\lambda }}_{VQ}$, ${\mathit{\lambda }}_{sem}$, and ${\mathit{\lambda }}_{stop}$ regulate the contribution of the respective components to the overall training process.

3.4. Working S2TT Model with Frozen Whisper Encoder

Unlike the previously discussed Scratch-trained Morphology-aware Model, the working system does not train speech understanding from scratch. Instead, it relies on a pretrained Whisper encoder [32], which forms a semantically rich (content-rich) representation of the original speech signal. Translation and synthesis modules are trained on top of this representation, allowing the task to be divided into two levels: extracting content from speech and then generating the target acoustic signal. Figure 2 below diagrams the resulting working system.

Figure 2. The working system: translation of Kazakh speech into Turkish, Tatar, and Uzbek with a single jointly trained model, without a cascade. Frozen/ready blocks (Whisper encoder, vocoder) are shown in gray; those trained in this work (projection, text decoder with FiLM, synthesis head similar to Translatotron 2) are shown in green; and the work’s contribution is shown in purple.

As shown in Figure 2, the data flow in the working system goes through six main stages: an input waveform signal of Kazakh speech with a sampling frequency of 16 kHz; a frozen Whisper front-end, including log-mel feature extraction and a pretrained Whisper-small encoder; a trainable linear projection into the decoder’s memory; a FiLM Transformer decoder, which performs translation with language conditioning; a speech synthesis head in the style of Translatotron 2; and an external HiFi-GAN vocoder, which converts the mel spectrogram into a waveform of the target speech.

The main part of the task—understanding the source speech and transferring its content—is performed by the “Whisper memory + FiLM Transformer decoder” combination. Cross-attention extracts content from a pretrained speech representation, which already encodes semantic information. This eliminates the need for the synthesis head to perform translation and acoustic generation simultaneously: it receives prepared hidden states and converts them into a mel-spectrogram of the target speech. In the Scratch-trained Morphology-aware Model, by contrast, the mel decoder was forced to simultaneously perform content transfer and synthesize an acoustic signal based on a representation learned from scratch. Under low-resource agglutinative S2ST conditions, this scheme proved unstable: the model did not generate a translatable representation, and content transfer was not actually learned.

Formally, let $x\in {\mathbb{R}}^{T_x}$ be the original waveform signal in Kazakh, l be the target language, and $\widehat{y}$ is the synthesized target speech signal. The task of the system is to generate y^ in language l while preserving the content of the original signal x. Additionally, during training, the target text sequence $u=\left(u_1,\dots ,u_N\right)$ is used as an auxiliary signal but is not a link in the cascaded output.

First, the Whisper feature extractor converts the x waveform into log-mel features:

$$M={\varphi }_{mel}\left(x\right),M\in {\mathbb{R}}^{T_m\times 80},$$

(26)

$x$—original speech signal in Kazakh (waveform, 16 kHz). Log-mel $S80\times 3000$; Next, the frozen Whisper-small encoder generates a latent speech representation:

$$E=WhisperEnc\left(M\right),E\in {\mathbb{R}}^{T_e\times 768},$$

(27)

$E$—projected memory N $\times 512$.

Since subsequent modules use a latent dimension of 512, a trainable linear projection is applied:

$$H=E{W}_p+1b_p,W_p\in {\mathbb{R}}^{768\times 512},H\in {\mathbb{R}}^{T_e\times 512},$$

(28)

where $H$—encoder output Whisper $N\times d_w\left(N\approx 1500\right)$.

The autoregressive FiLM Transformer decoder performs translation. At each step t, the decoder uses previously generated tokens $u_{<t}$, memory H, and the target language embedding $e_l$:

$$h_t=Decoder\left(u_{<t},H,e_l\right),$$

(29)

distribution over the SentencePiece dictionary (2000 units):

$$p\left(u_t|u_{<t},x,l\right)=softmax\left(W_0{h}_t+b_0\right),$$

(30)

where $h_t$ represents the hidden states of the decoder.

Thus, the text branch is trained as an auxiliary S2T task, helping stabilize the semantic representation.

$$\left[{\gamma }_l;{\beta }_l\right]=W_f{e}_l+b_f,$$

(31)

Then, the activations of the $\mathcal{z}$ layer are modulated according to the following rule:

$$FiLM\left(\mathcal{z},l\right)={\gamma }_l\odot \mathcal{z}+{\beta }_l,$$

(32)

where ${\gamma }_l$ and ${\beta }_l$ are parameters dependent on the target language, and ⊙ denotes componentwise multiplication. This allows a single model to translate into multiple Turkic target languages without training a separate architecture for each language.

After the decoder’s hidden states $h_1,\dots ,h_N$ are generated, the speech synthesis head is used. The length predictor estimates the durations of hidden states:

$${\widehat{d}}_t=LengthPred\left(h_t,e_l\right),$$

(33)

Next, a non-autoregressive NAR mel decoder, based on FFT blocks and FiLM conditioning, generates the target mel spectrogram:

$$\widehat{S}=MelDec\left(\tilde{H},e_l\right),$$

(34)

Finally, an external HiFi-GAN vocoder converts the mel spectrogram into a waveform:

$$\widehat{y}=HiFiGAN\left(\widehat{S}\right),$$

(35)

The composite loss function combines the primary acoustic component, the auxiliary text target, and the adversarial term:

$$\mathcal{L}={\lambda }_{mel}{\mathcal{L}}_{mel}+{\lambda }_{stop}{\mathcal{L}}_{stop}+{\lambda }_{adv}{\mathcal{L}}_{adv},$$

(36)

Thus, the working system is a single jointly trained model, not an ASR/MT/TTS cascade. The text output is used only as an auxiliary training target, while the final speech is generated through the decoder’s hidden states, the synthesis head, and the HiFi-GAN vocoder. This architecture reduces the load on the acoustic decoder and allows the primary task of speech understanding to be transferred to the pretrained content-rich Whisper representation.

3.5. Final S2UT System with Turkic Speech Units

The final two-pass S2UT system translates speech into speech using an intermediate discrete representation of the target speech. As shown in Figure 3, the Kazakh speech input is first fed to the pretrained Whisper-large-v3 encoder, fine-tuned end-to-end during this stage. It extracts a stable and rich representation from it. Then, the FiLM Transformer decoder, which incorporates a target-language embedding, simultaneously solves the auxiliary S2T (speech-to-text) task and generates hidden states (h_t). These states are then used to predict speech units.

Figure 3. The final end-to-end S2UT system. The architecture includes a fine-tuned Whisper-large-v3 encoder, a FiLM Transformer decoder for S2T translation and hidden state generation (h_t), a single-stream Turkic unit predictor based on mHuBERT-147, layer 9, and a Turkic HiFi-GAN vocoder for unit-to-waveform conversion.

On top of the hidden states (h_t), a speech unit predictor is trained using cross-attention and scheduled sampling. It generates a sequence of single-stream Turkic discrete units obtained from mHuBERT-147, layer 9, and a k-means quantizer with k = 1000. The units are then fed to a Turkic HiFi-GAN vocoder trained on Turkic audio, which converts the unit sequence into a waveform. The final architecture thus splits the task into two layers: a content layer responsible for unit translation and prediction, and an acoustic layer responsible for speech rendering.

As shown in Figure 3, the final architecture divides the task into two layers: a content layer, responsible for translation and prediction of target units, and an acoustic layer, responsible for speech rendering. This division allows us to localize the system’s main bottleneck in unit prediction, rather than in the vocoder.

A critical component of the final system is the Turkic unit quantizer. Unlike English units, which can lose some phonemic information already during k-means clustering, the Turkic quantizer is trained on target Turkic audio using mHuBERT-147 with k = 1000. A cache of 60,000 single-stream units with a dedup ratio of 0.46–0.58 is used for training. The UnitHiFiGANGenerator vocoder with a vocabulary size of 1001 uses a warm-start convolutional stack from a pretrained SpeechBrain generator and then adapts to Turkic phonetics.

4. Experimental Setup and Results

4.1. Overall Comparison of Experimental Stages

Table 1 summarizes the overall comparison of the three experimental stages.

Table 1. Comparison of results across all experimental stages.

System	Main Components	Output Type	Vocoder	ASR-BLEU (BLEU 1)	Comment
Approach 1: scratch architecture	Conformer + CTC morpheme head + VQ codebook + semantic anchor	speech	HiFi-GAN	0	Failed to learn a translatable target representation
Approach 2: Whisper S2TT	Whisper encoder + FiLM text decoder	text	-	19.49	Successful speech-to-text translation
Approach 3: final S2UT system	Whisper-large-v3 + FiLM decoder + mHuBERT-147 units + Turkic vocoder	speech	Turkic HiFi-GAN	3.25	First non-zero end-to-end ASR-BLEU in this setup

¹ BLEU is used for text output (Approach 2); ASR-BLEU is used for speech output (Approaches 1 and 3).

The results show a clear progression from a fully trainable but unsuccessful morphology-aware architecture to a more robust system based on frozen pretrained speech representations. The scratch-trained model preserved some source-side acoustic content but failed to form a target-language representation suitable for translation. In contrast, the use of a frozen Whisper encoder provided a content-rich representation and enabled stable S2TT performance. Finally, replacing mel-based synthesis with Turkic speech units and a Turkic-trained vocoder made direct S2ST possible, although the absolute ASR-BLEU remains limited.

4.2. Evaluation Metrics

For speech-to-text (S2TT) and end-to-end (e2e) translation, a comprehensive set of metrics is used, with their limitations explicitly stated for agglutinative corpora:

• BLEU (sacreBLEU) [33]—the primary translation metric; for e2e—ASR-BLEU: synthesised speech is recognized using MMS-1b-all, and the hypothesis is compared with the reference text.
• chrF [34]—character-level metrics recommended for morphologically rich languages: they do not penalize a mismatch of a single morpheme as a mismatch of the entire word.
• COMET (Unbabel/wmt22-comet-da) [35]—a neural network-based, reference-based metric for translation quality.
• Translation Edit Rate (TER) [36] and character error rate (CER) are complementary measures (penalizing valid permutations and morphological variants); CER is preferable to WER in agglutinative languages.
• SQUIM (torchaudio) [37]—perceptual synthesis quality without a reference: SQUIM-SUBJECTIVE yields MOS, SQUIM-OBJECTIVE yields Short-Time Objective Intelligibility/Perceptual Evaluation of Speech Quality/Scale-Invariant Signal-to-Distortion Ratio (STOI/PESQ/SI-SDR).
• Performance—RTF (real-time factor) and component-wise latency.

A 95% bootstrap confidence interval (1000 resamples) is provided [38]. The following were not computed in this study and are left for future work: the S2ST metric BLASER 2.0, semantic similarity via SONAR, studio-condition MOS testing, and human judgments of meaning preservation and intelligibility. The human test that was run (Section 4.10) covers naturalness only.

ASR-BLEU is used as the main end-to-end metric because it is the standard metric in S2ST work [10,12] and makes our numbers comparable with that literature. It has a known weakness: the score mixes translation errors with errors of the recognizer that transcribes the output. We therefore report it together with three checks. The topline (MMS-ASR applied to the ground-truth target audio) shows how much the recognizer alone loses in each language. The text pass of the two-pass system is scored directly with BLEU and chrF++, which measures translation quality without any recognizer. ASR-chrF++ and CER are less sensitive than BLEU to single-morpheme recognition errors in agglutinative words.

4.3. Training Configuration and Hardware

To ensure reproducibility, the main training configurations used in the experiments are summarized in Table 2. The scratch-trained morphology-aware architecture was optimized using a composite objective, while the vocoder-related experiments used a separate adversarial training setup.

Table 2. Training hyperparameters used in the experiments.

Component	Parameter	Value
Scratch-trained S2ST architecture	Optimizer	AdamW
Scratch-trained S2ST architecture	Learning rate	3.0 × 10−5
Scratch-trained S2ST architecture	Betas	(0.9, 0.95)
Scratch-trained S2ST architecture	Weight decay	0.05
Scratch-trained S2ST architecture	LR schedule	linear warmup for 4000 steps, then plateau
Scratch-trained S2ST architecture	Batch size	12 per GPU
Scratch-trained S2ST architecture	Effective batch size	24 with DDP on 2 GPUs
Scratch-trained S2ST architecture	Precision	bf16 autocast
Scratch-trained S2ST architecture	Number of epochs	30
Scratch-trained S2ST architecture	Checkpoint frequency	every 5000 steps, with best checkpoint selected by validation loss
Scratch-trained S2ST architecture	Gradient clipping	1.0; reduced to 0.5 in unstable runs
Scratch-trained S2ST architecture	Loss weights	mel = 1.0, stop = 1.0, CTC = 0.3, VQ = 0.25, semantic anchor = 0.1
Vocoder training	Batch size per GPU	16
Vocoder training	Effective batch size	32 with DDP on 2 GPUs
Vocoder training	Audio segment length	8192 samples, corresponding to 0.512 s at 16 kHz
Vocoder training	Mel segment length	32 frames
Vocoder training	Segment sampling	random mel crop with synchronized audio crop; audio_offset = mel_offset × 256
Vocoder training	Learning rate	2.0 × 10−4 for both generator and discriminator
Vocoder training	Optimizer	AdamW, β = (0.8, 0.99), weight_decay = 10−2
Vocoder training	LR schedule	ExponentialLR, γ = 0.999
Vocoder training	Gradient clipping	5.0 by norm, applied separately to generator and discriminator
Vocoder training	Target number of steps	300,000
Vocoder training	Checkpoint frequency	every 20,000 steps
Vocoder training	Monitoring audio generation	every 2000 steps
Hardware	GPUs	RTX 5090 GPUs; DDP with NCCL backend for multi-GPU runs

4.4. Negative Results of the Scratch-Trained Architecture

The baseline architecture trained ‘from scratch’ and described in Section 3.3, achieves zero ASR-BLEU scores for all language pairs under consideration when using autoregressive inference. Since BLEU is a standard automated metric for assessing the quality of machine translation, based on the matching of n-grams with a reference translation [39], the result obtained indicates not a partial decline in quality, but the actual absence of recoverable translated content in the generated speech.

In teacher-forcing mode, the model receives the correct previous answers (ground truth data) as input during training. However, in real-world applications (when the model is used), such correct hints are not available—the model must rely solely on its own predictions. This difference gives rise to a problem known as exposure bias [40]: the model becomes accustomed to ideal training conditions and may perform less well in real-world conditions. In the experiments conducted, the ASR-BLEU score in teacher-forcing mode is close to the upper bound of the pipeline, defined using the reference mel-spectrogram input and Griffin–Lim reconstruction: tur—30.3, tat—5.9. Consequently, this mode does not measure the model’s ability to perform translation, but in fact confirms that the decoder can reproduce the given reference mel input. This is consistent with the nature of the Griffin–Lim algorithm, which reconstructs a signal from a spectrogram representation [41], but does not indicate the presence of a correct cross-linguistic mapping.

An ablation analysis of the novelty of attempt 1 using the val_mel metric—the decoder’s L1 error on masked segments—shows that a positive contribution is confirmed only for component A, which is associated with CTC morphemes. The CTC approach is designed for training on time-unlabeled sequences and allows the extraction of ordered character- or morpheme-based representations without explicitly aligning the input and output [42]. The ablation results are presented in Table 3.

Table 3. Ablation results.

Configuration	Components	Best val_mel	Effect
baseline	–	0.872	–
ctc_only	A (morphlogy)	0.727	−17% (working)
no_sonar	A + B	0.731	+0.5% (B neutral)
full	A + B + C	0.783	+7% (C counterproductive)

Thus, only Novelty A, i.e., the use of CTC morphemes, is confirmed. The VQ codebook does not provide a significant improvement over CTC-only, and the LaBSE semantic anchor proves counterproductive. This is particularly important, since LaBSE and SONAR are generally designed to construct interlingual sentence-level representations and to enable cross-lingual alignment [43,44]. However, in this architecture, external semantic anchoring does not lead to the formation of a translatable latent space; rather, it impairs masked mel reconstruction.

Five controlled conditions test the main alternative explanations for the null translation (Table 4).

Table 4. Five controlled conditions.

Configuration	Changed Component	Purpose	Criterion	Outcome
Baseline	Conformer encoder trained from scratch; mel and stop losses only	Reference for the ablation	val_mel	0.872
CTC only	+ morpheme CTC head (weight 0.3)	Hypothesis test: component A	val_mel	0.727
CTC + VQ	+ VQ codebook (weight 0.25)	Hypothesis test: component B	val_mel	0.731
Full	+ semantic anchor (weight 0.1)	Hypothesis test: component C	val_mel	0.783
Continuous mel (NAR)	autoregressive mel decoder replaced by a non-autoregressive one	Control: exposure bias	ASR-BLEU	0.0
Adversarial mel	+ mel discriminator on the NAR decoder	Control: over-smoothing	ASR-BLEU	0.0
Character text (S2TT)	character-level text target instead of mel	Control: continuous output	BLEU	about 0.1
Subword text (S2TT)	SentencePiece target (2000 units)	Exploratory variant	BLEU	0.02 to 0.09
Masked subword (S2TT)	50% of target tokens masked	Control: language-model shortcut	BLEU	0.05 to 0.14
Frozen Whisper-small encoder	pretrained encoder replaces the scratch-trained one	Intervention (not scratch-trained)	BLEU	4.67 in the first epoch, 13.20 after 247,000 steps

These results make it unlikely that the zero translation is caused only by exposure bias, over-smoothing, the continuous form of the acoustic output, or the language model’s use of a shortcut strategy. The remaining explanation, which these controls support but do not measure directly, is that a scratch-trained encoder benefits from the morpheme CTC head (Table 3), but it fails to generate a target representation suitable for translation. We therefore consider the lack of a shared, translatable latent representation in the encoder to be the most likely cause, not a proven single cause. The paragraph below lists the other explanations raised for this failure, the evidence that bears on each, and what remains untested.

Other explanations were not excluded by dedicated experiments, and we state the evidence for each. Loss-weight imbalance is unlikely: the text-only S2TT variants in Table 4 were trained with a single cross-entropy loss, without the mel, VQ, or anchor terms, and still scored near zero. Errors in the data pipeline or in source-target pairing are unlikely: the Whisper-based S2TT model was trained on the same manifests, with the same SentencePiece vocabulary and FiLM decoder layers, and reached 4.67 BLEU within its first epoch. This is indirect evidence, not a dedicated check. Optimization difficulty and insufficient training are not excluded: no learning-rate or loss-weight sweeps were run, and the best checkpoints of the text-only variants were reached after only 3000 to 4000 steps. Finally, the encoder explanation rests on an intervention: replacing the scratch-trained encoder with a pretrained one made the same decoder work. That intervention also changes the amount of pretraining data and whether the encoder is trained, so it supports the explanation without isolating it. All diagnostics in this section concern the scratch-trained model of Section 3.3. The exposure-bias analysis in Section 4.7 concerns a different model, the unit predictor of the final pretrained two-pass system. The two findings answer different questions: exposure bias does not explain why the scratch-trained model produced no translation, but it does limit the final system once translation works.

4.5. S2TT Results

Using the frozen Whisper-small encoder results in a steady increase in BLEU scores: from 0.06 to 4.67 at the zero epoch, and then to 13.20 after 247,000 training steps. Furthermore, fine-tuning the Whisper-large-v3 encoder allows us to overcome the quality limitation observed in the frozen-small mode and significantly improve the final performance of the S2TT model.

The primary S2TT test evaluation was performed using greedy decoding, with results reported separately for each target language in Table 5.

Table 5. S2TT results for each language.

Language	BLEU	chrF	chrF++	TER	COMET
tat	25.98	59.33	55.33	66.05	0.739
uzb	18.90	54.37	49.91	79.92	0.699
tur	13.59	46.94	43.07	87.86	0.694
macro	19.49	53.55	49.44	77.95	0.710

Given the morphological complexity of the Turkic languages considered in this study, the character-based metrics provide an important complementary perspective on translation quality. The macro-averaged chrF and chrF++ scores are 53.55 and 49.44, respectively, while the corresponding macro-average BLEU score is 19.49 (95% bootstrap CIs: Turkish [12.62, 14.68], Tatar [24.71, 27.31], Uzbek [17.65, 20.28]). The relatively narrow confidence intervals of approximately ±1–2 points indicate that the estimates are stable across the evaluated samples. The lower BLEU score relative to the earlier validation results may reflect the greater heterogeneity and difficulty of the test set and may also indicate partial adaptation to the validation distribution.

To further assess the generalization ability of the S2TT model beyond the synthetic TTS-based evaluation setting, an additional experiment was conducted using natural speech from the Kazakh test subset of FLEURS [45]. The evaluation set contained 856 real speech recordings corresponding to 349 unique sentences after deduplication. The FLEURS utterances were aligned with the FLORES-200 devtest set using sentence identifiers, and the corresponding human-produced Turkish, Tatar, and Uzbek reference translations were used for evaluation. We used the same 856 Kazakh recordings for all three target languages, so the source audio and sentences are identical for Turkish, Tatar, and Uzbek; only the target reference differs. Each of the 349 sentences has a human reference in all three target languages (FLORES-200 tur_Latn, tat_Cyrl, and uzn_Latn); we converted the Uzbek references to Cyrillic to match the Uzbek training targets. FLEURS provides about 2.5 recordings per sentence, read by volunteer speakers. We decoded all three languages with the same checkpoints, argmax decoding, and the same MMS-1b-all recognizer. The conditions are therefore matched across languages except for the recognizer, which is weakest for Tatar (Section 4.7); because FLEURS has no Tatar recordings, a recognizer topline could not be measured on this set.

On real speech data, the WS2TT_lv3 model obtained the following BLEU scores: Turkish—1.92, Tatar—4.92, Uzbek—3.84. The macro-average BLEU was 3.56, and the chrF++ average was 28.28. This score is approximately 5.5 times lower than the macro-average BLEU of 19.49 obtained on the synthetic dataset_214k test sample. The two test sets differ in more than acoustics, so this gap cannot be attributed to the acoustic domain alone; Section Sources of the Synthetic-to-Natural Gap separates the factors.

The full end-to-end pipeline (speech → units → vocoder → MMS-ASR), using the same S2UT_l9/UVOC_hifi checkpoint as in Table 6, was also evaluated on the same set. It yields ASR-BLEU scores of 0.93 for Turkish, 0.69 for Tatar, and 0.95 for Uzbek, with a mean of 0.86, which is nearly four times lower than the headline score of 3.25 on the synthetic test set. A degradation of similar magnitude for S2TT and the end-to-end pipeline suggests that the drop arises before the speech-generation stage, in the translation itself, rather than in a separate problem of the end-to-end component.

Table 6. End-to-end quality of the S2ST system on the random test sample (argmax decoding). Macro is the unweighted mean of the three languages; Topline is the ASR-BLEU of MMS-ASR on the ground-truth target audio; Retention is ASR-BLEU divided by Topline.

Language	ASR-BLEU	ASR-chrF++	TER	Topline Real, MMS	Retention
tat	2.87	32.29	89.7	15.09	19.0%
uzb	4.05	30.14	87.1	33.60	12.1%
tur	2.84	29.76	94.6	31.50	9.0%
macro	3.25	30.73	90.5	26.73	13.4%

The TER metric shows fairly high values, particularly for Turkish (TER = 87.86, Table 5); it is therefore presented as a less favorable but important additional measure of translation quality. At the same time, the COMET score of 0.710 (macro-average) indicates moderate translation quality and is consistent with results from the character-based metrics chrF and chrF++. Increase in BLEU in the S2TT task following fine-tuning of the Whisper-large-v3 encoder is shown in Figure 4.

4.6. S2UT End-to-End Results

The current S2TT system converts input speech into a text sequence; however, the ultimate aim of the project is to develop a system capable of direct speech-to-speech translation, i.e., producing a speech signal at the output. The transition from text output to end-to-end synthesis required a separate series of experiments. As a result of successive iterations, the end-to-end quality, measured by ASR-BLEU, increased from zero—a level typical of approximately 10 of the architectures tested—to 0.34, then to 0.61, and finally to 3.25.

The first negative result was linked to an attempt to predict discrete acoustic codes from text. In this approach, EnCodec/RVQ acoustic codes were used as an intermediate representation for subsequent sound reconstruction. However, argmax inference produced silence, whilst sampling produced fluent but semantically incorrect speech across all temperature values. This effect persisted even when reference tokens were provided. The prediction accuracy of the codes in free generation was ~0.10 per codebook, compared to ~0.30 in teacher forcing mode. Consequently, honest end-to-end generation proved unstable due to error accumulation and exposure bias drift.

Figure 4. Fine-tuning of the Whisper-large-v3 encoder in BLEU metrics.

Figure 4. Fine-tuning of the Whisper-large-v3 encoder in BLEU metrics.

The second negative result concerned training vocoders ‘from scratch’ on semantic units. Unlike acoustic codes, the semantic units obtained using HuBERT + k-means had lower entropy and were more easily predicted from the text. Nevertheless, vocoders trained from scratch exhibited a quality ceiling of approximately zero ASR-BLEU when using reference units. The main reason for this limitation was the over-smoothing effect in the mel-frequency representation, which led to a reduction in the intelligibility of the reconstructed speech.

To overcome this limitation, the pretrained neural vocoder, speechbrain/hifigan-hubert-k1000-LibriTTS, was utilized. Its application enabled the production of audibly intelligible resynthesis of Turkic audio from reference units. The quality ceiling increased to ~6.7, and the first end-to-end variant with recognizable speech output achieved an ASR-BLEU score of 0.34. This was the first configuration in this study whose speech output contained recoverable content; all previous configurations scored zero.

The third negative result was that the English-language units did not preserve Turkic phonetics with sufficient accuracy. Despite the improvement in speech intelligibility, the ASR-BLEU ceiling of 6.7 limited the system’s further development. To test this hypothesis, a pretrained 6-stream SpeechBrain vocoder was fine-tuned on ‘reference units—Turkic waveform’ pairs. However, the fine-tuning did not lead to an increase in the upper bound on quality: the ASR-BLEU ceiling of 6.76 was identical to that of the pretrained vocoder, mel_loss plateaued at around 1.69, and the discriminator showed signs of collapse.

The result obtained shows that the limitation is not due to the vocoder’s domain, but to the loss of phonetically relevant information at the quantization stage. Since k-means was trained on the English-language LibriSpeech corpus, Turkic phonemes were mapped to the nearest English cluster centers even before the vocoding stage. Consequently, subsequent fine-tuning of the renderer failed to recover the phonetic information lost during quantization. The training dynamics of the single-stream Turkic vocoder are shown in Figure 5.

Figure 5. Training of a single-stream Turkic vocoder: mel_loss decreases monotonically with a balanced adversarial signal, in contrast to the collapse observed during the fine-tuning of an English-language vocoder.

To address this limitation, the final architecture adopted single-stream Turkic units together with a dedicated Turkic vocoder. The analysis carried out suggests that the quantizer must be adapted to the phonetics of the target languages. To this end, k-means (k = 1000) was trained on mHuBERT-147 representations extracted from the 9th layer, using Turkic audio. A single-pass vocoder was then trained on ‘reference Turkic units—Turkic waveform’ pairs. During training, the mel_loss decreased monotonically from 3.48 to 1.0 with a balanced adversarial component (Figure 5).

The use of Turkic units significantly raised the system’s upper quality limit. The ceiling in the ‘reference Turkic units → vocoder → ASR’ mode increased from 6.76 to 15.67, i.e., by approximately 2.3 times. Furthermore, the predictor that converts text hidden representations into single-stream units achieved a teacher-forced unit accuracy of 0.70, whereas the 6-stream scheme achieved 0.49. As a result, the end-to-end quality of the system increased from 0.61 to 2.88–3.25 ASR-BLEU, corresponding to approximately a 4.7× improvement in a single iteration.

To determine whether the synthetic-to-real performance gap observed for S2TT also affects the complete speech-to-speech translation pipeline, the end-to-end system was evaluated on the same FLEURS-based natural-speech set. Using the same S2UT_l9/UVOC_hifi checkpoint as in Table 6 and the complete speech → units → vocoder → MMS-ASR evaluation pipeline, the system achieved ASR-BLEU scores of 0.93 for Turkish, 0.69 for Tatar, and 0.95 for Uzbek, yielding a macro-average of 0.86. This value is approximately 3.8 times lower than the corresponding ASR-BLEU score of 3.25 obtained on the synthetic test set.

The performance degradation observed in the end-to-end evaluation is consistent with the degradation in the S2TT model when switching from synthetic speech to natural speech. Because the S2TT and end-to-end scores drop by similar factors, the loss occurs mainly before unit prediction and vocoding. Which properties of the natural test set cause it is analyzed below.

Sources of the Synthetic-to-Natural Gap

The natural-speech set differs from the synthetic test sample in more than acoustics. Its sentences come from FLORES-200, which was translated from English Wikinews, Wikijunior, and Wikivoyage articles, while our corpus is Kazakh web text with a different domain mix. Its sentences are longer (17.8 vs. 13.9 Kazakh words on average) and contain more words that never occur in our training data (10.9% vs. 4.2% of running words, 17.4% vs. 12.3% of word types). Its references are human translations, while the synthetic references are machine translations of the kind the model was trained to reproduce. Its audio is read speech from volunteer speakers recorded on different devices, while the synthetic source audio comes from a single TTS voice. The drop from synthetic to natural speech therefore combines content, vocabulary, reference style, and acoustics. Speaker and recording effects could not be separated from each other, because speaker metadata were not part of our evaluation manifests.

4.7. Ablation Study

Final end-to-end results were computed on the random test sample described in Section 3.1 (1484 Turkish, 1496 Tatar, and 1487 Uzbek utterances; seed 0) with argmax decoding, a vocoder trained for 80,000 steps, and a unit predictor trained for 60,000 steps. The development-stage numbers quoted in Section 4.6 (0.34, 0.61, and 2.88 ASR-BLEU) and the first version of this evaluation used a file-order prefix of the test manifest (157 to 159 utterances per language). On that prefix the final system scored the same macro ASR-BLEU of 3.25. Per-language results are shown in Table 6.

On the random test sample, the final system reaches a macro ASR-BLEU of 3.25 with argmax decoding (per-language ASR-BLEU with 95% bootstrap CIs: Tatar 2.87 [2.50, 3.24], Uzbek 4.05 [3.60, 4.48], Turkish 2.84 [2.48, 3.18]). The macro-averaged ASR-chrF++ is 30.73. An ASR-chrF++ of about 31 together with an ASR-BLEU of about 3 means that the recognizer recovers part of the target content at the character level, while complete word sequences are rarely reproduced.

There is a major problem with this system’s end-to-end validation: it depends on the accuracy of the target language automatic recognition system (ASR). On the ground-truth target audio, the MMS recognizer reaches only 15.09 ASR-BLEU for Tatar, compared with 33.60 for Uzbek and 31.50 for Turkish. That is, the MMS system has a significant difficulty in understanding Tatar words. The text pass of the same model allows a rough separation of error sources. Scored without any recognizer, the text pass reaches 13.56, 26.02, and 18.92 BLEU for Turkish, Tatar, and Uzbek. The recognizer alone caps the score at 31.50, 15.09, and 33.60 even on ground-truth target audio, so for Tatar the evaluator ceiling is below the quality of the text pass. When the recognized system speech is compared with the model’s own text pass instead of the reference, it matches at only 8.97 to 12.35 BLEU (CER 31% to 42%). Most of the loss therefore arises between the text pass and the recognized speech, in unit prediction, vocoding, and recognition, rather than in translation. The low Tatar score reflects the recognizer as much as the translation system.

The architectural pivot was evaluated using the same test protocol against the e2e ASR-BLEU metric. A comparison of the original 6-stream scheme and the single-stream Turkic architecture is presented in Table 7.

Table 7. Comparison of S2ST system architectural configurations.

System	Units	Vocoder	E2E ASR-BLEU
6-stream SB	HuBERT-large with six layers and English-language k-means clustering	SB LibriTTS (English)	0.64 (argmax)
1-stream L9	mHuBERT-147 and Turkic k-means clustering	own Turkic HiFi-GAN	3.25 (argmax)

As shown in Table 7, the transition from the 6-stream SpeechBrain scheme to the single-stream Turkic architecture increased the end-to-end ASR-BLEU score from 0.64 to 3.25, i.e., by approximately 5.1 times. Because this comparison changes the units and the vocoder together, it does not by itself show which of the two is responsible; the controlled 2 × 2 experiment below addresses this question.

An assessment of the upper quality limit in the ‘reference units → vocoder → MMS’ mode also demonstrates the advantage of the Turkic configuration. The English-language 6-stream SB had a ceiling of 6.76, and its fine-tuning on Turkic audio did not improve the result. At the same time, the Turkic single-stream vocoder reached 9.08 after just 10k steps and 15.67 after 80k steps. Consequently, simply adapting the vocoder is insufficient: the units and the vocoder must be aligned with Turkic phonetics.

In contrast, the 1-stream L9 configuration is based on the ninth layer of mHuBERT-147 and Turkic k-means clustering, whilst a dedicated Turkic HiFi-GAN vocoder is used for synthesis. This architecture achieves an end-to-end ASR-BLEU score of 3.25 in argmax decoding mode. Thus, replacing the English-language discrete units and the vocoder with Turkic-oriented components results in approximately a 5.1-fold improvement in end-to-end ASR-BLEU.

In Table 8, both the unit type and the vocoder were changed at the same time. Therefore, an additional clean experiment was performed to see which had which effect separately. This is a 2 × 2 table experiment: English and Turkic units were tested in different pairs with English and Turkic vocoders. The scoring rules were left as before. The two previously unavailable cross-combinations were trained for 30,000 steps and evaluated using the same ground-truth-unit ceiling protocol (GT units → vocoder → MMS-ASR) used to obtain the existing ceiling scores of 6.76 and 15.67. For the cross-combinations, English units were used as a single stream of the 7th layer of the HuBERT-large network (i.e., as a single-stream surrogate). And to match the Turkic L9 units to the input of the English 6-stream vocoder, they were mapped to six input streams.

Table 8. Controlled 2 × 2 ablation of unit source and vocoder architecture (GT-units ceiling, ASR-BLEU).

	English Vocoder	Turkic Vocoder
English units	6.76	8.47 (new)
Turkic units	0.00 (new)	15.67

The results of 2 × 2 ablation showed that the unit representation and vocoder architecture are closely related. When using the Turkic vocoder with English units, the ASR-BLEU increased to 8.47. This was higher than the 6.76 in the original configuration (English units + English vocoder).

Meanwhile, by combining Turkic units with the English 6-stream vocoder, the ASR-BLEU was equal to 0.00. In this combination, the training log showed that the discriminator value was significantly reduced (D ≈ 0.01–0.04, instead of the usual 2–3) and the mel-L1 loss remained above 3.5–4.0. This indicates that the English vocoder could not adapt effectively to the Turkic units. The best result—15.67 ASR-BLEU—was obtained only when combining Turkic units with the Turkic vocoder. Therefore, the performance of the system cannot be attributed solely to the unit representation or the vocoder, but rather to their phonetic and architectural compatibility.

The 2 × 2 experiment has several limits. All four cells are ceilings with ground-truth units (GT units → vocoder → MMS-ASR) measured on 80 to 105 validation utterances per language, so they describe unit and vocoder pairs, not the final end-to-end system, and they were not recomputed on the random test sample. Each cell is a single training run without repeated seeds, so differences of a few ASR-BLEU points are not significant. Training lengths differ: the two new cross cells were trained for 30,000 steps, the Turkic pair for 80,000 steps (it already reached 9.08 after 10,000 steps), and the English pair is the pretrained SpeechBrain vocoder without further training. The English-unit cross cell uses a single HuBERT-large layer-7 stream as a surrogate for the original six streams, and the Turkic units were copied into six input slots to fit the English vocoder. All cells depend on the MMS recognizer, which is weakest for Tatar. The table therefore supports the compatibility conclusion qualitatively but does not give exact effect sizes for units and vocoder.

Once units and vocoder were matched, the main remaining obstacle in the final pretrained two-pass system was exposure bias in autoregressive unit prediction. This analysis concerns the unit predictor of the final system, not the scratch-trained model, for which Section 4.4 found exposure bias an unlikely cause of the zero translation.

Under teacher forcing, the accuracy of unit prediction was 0.811. When the model was self-generated, this figure dropped to 0.393, with an absolute difference of 0.418. These values were measured on 600 utterances (200 per language) held out from unit-predictor training.

However, the prediction of the chain length was much more stable: the ratio of the predicted length to the actual length was 1.000, and the Pearson correlation was 0.939.

Thus, the main limitation lies not in the vocoder or the length prediction, but in the autoregressive model’s inability to predict units accurately and consistently.

Overall, the ablation results demonstrate that linguistic and architectural compatibility among the unit representation, quantizer, and vocoder is critical for the proposed Turkic S2ST system. English-derived HuBERT representations combined with LibriTTS-oriented synthesis provide a substantially lower quality ceiling, while the controlled cross-combinations show that changing only one component does not reproduce the performance of the fully Turkic configuration. In particular, the Turkic vocoder remains functional with English units, whereas the English vocoder fails with Turkic units, revealing an asymmetric compatibility effect. The combination of mHuBERT-147 L9 representations, Turkic k-means quantization, and the dedicated Turkic HiFi-GAN vocoder therefore provides the most effective discrete speech representation and synthesis pathway among the evaluated configurations.

These comparisons do not isolate every component. Between the 6-stream and the single-stream systems, the Whisper-large-v3 encoder and the FiLM text decoder were the same frozen checkpoint, and both unit predictors used the same six-layer autoregressive architecture. The unit inventory, the number of streams, the vocoder, and the size and training length of the unit predictor changed together: the 6-stream predictor had 34.5M parameters and 100,000 training steps, the single-stream predictor 29.4M parameters and 60,000 steps. The 2 × 2 ceilings separate units from the vocoder only at the ceiling level, and we did not run ablations that vary the unit predictor or the text-decoder conditioning with everything else fixed. The improvement from 0.64 to 3.25 ASR-BLEU should therefore be read as the effect of the combined Turkic unit and vocoder stack, not as the contribution of a single component.

The perceptual quality of the synthesized speech was further assessed using SQUIM on a random subset of 300 test utterances per language drawn from the same test sample (seed 0). The results of the comparison between real audio, resynthesis using reference units, and the system output are presented in Table 9.

Table 9. Perceptual quality of synthesized speech according to SQUIM metrics.

Condition	MOS↑	STOI↑	PESQ↑	SI-SDR↑
Ground-truth target audio (MMS-TTS)	4.34	0.995	3.81	24.9
Reference units—vocoder	4.25	0.993	3.57	22.6
The output of the proposed system	4.21	0.995	3.66	23.6

↑ Higher values are better.

As shown in Table 9, the proposed system’s SQUIM-MOS is 4.21, close to the 4.34 of the ground-truth target audio. The STOI, PESQ, and SI-SDR values also remain close to those of the reference audio, which indicates that the vocoder adds little acoustic degradation. Consequently, the low end-to-end ASR-BLEU score of 3.25 is primarily due to content-level errors—that is, inaccurate unit prediction—rather than limitations of the vocoder.

The computational efficiency profile shows that processing a single utterance on a single RTX 5090 takes 1126 ms, corresponding to an RTF of ≈ 0.11, i.e., approximately 9.3 times faster than real time. The total model size is 746 million parameters. The main bottleneck in terms of speed is the autoregressive decoding of units, which accounts for 72% of the execution time, whilst the vocoder requires only 12 ms, or about 1% of the total latency.

4.8. Efficiency and Latency

Inference efficiency was evaluated on a single RTX 5090 GPU with batch size 1 and an average input duration of approximately 10.5 s. The final S2UT system achieved a total latency of 1126 ms per utterance, corresponding to a real-time factor of 0.107. Therefore, the system runs approximately 9.3 times faster than real time. This is shown in Table 10. Timing was measured on 30 test utterances after 3 warm-up utterances, with batch size 1 and fp32 precision, synchronizing the GPU before and after each stage. The timer covers all neural stages from the encoder to the vocoder output (Table 10); audio loading and log-mel feature extraction on the CPU are outside the timer and are not included in the RTF. The value comes from one timed pass; a repeated run on another day gave an RTF of 0.112. The cascade baseline was not timed, so we make no speed comparison between the two systems.

Table 10. Component-wise latency of the final S2UT system.

Component	Latency	Share of Total Latency
WS2TT encoder	41.5 ms	4%
Greedy text decoding	255.5 ms	23%
Hidden-state decoding	4.1 ms	<1%
Length predictor	0.7 ms	<1%
Autoregressive unit decoding	811.8 ms	72%
Vocoder	12.3 ms	1%
Total	1126 ms	100%

The main computational bottleneck is autoregressive unit decoding, which accounts for 72% of total inference time. In contrast, the vocoder is computationally inexpensive, requiring only 12.3 ms per utterance. The complete system contains approximately 746 million parameters, including 696M parameters in the WS2TT component, 32M in the unit predictor, and 18M in the vocoder. The measured throughput is approximately 0.9 utterances per second.

4.9. Cascade Baseline Comparison

As a reference point, we evaluated an off-the-shelf, zero-shot cascade. It consists of three parts: MMS-ASR, which converts Kazakh speech into text, MADLAD-400-10B, which translates text into another language, and MMS-TTS, which produces speech in the same language. Another MMS-ASR was used to check the output quality. None of these cascades were trained on our dataset_214k data. Another important point: the NLLB model may have been used to generate data. Therefore, MADLAD, not NLLB, was chosen for the cascade—otherwise we would have tested ourselves. Both—the cascade and the proposed direct system—were measured with the same test. The test used the same random sample as the end-to-end evaluation: 4467 Kazakh recordings (1484 Turkish, 1496 Tatar, and 1487 Uzbek target rows). The results of the cascade baseline are presented in Table 11. The confidence intervals of the results were obtained by bootstrapping 1000 times.

Table 11. Cascade baseline (MMS-ASR + MADLAD-400-10B + MMS-TTS) on the same test protocol.

Language	ASR-BLEU (Cascade)	95% CI	MT Text-BLEU
tur	2.13	[1.78, 2.48]	4.83
tat	1.15	[0.87, 1.40]	5.77
uzb	3.14	[2.76, 3.55]	6.98
macro	2.14	--	5.86

On the same random test sample, the two-pass system reached 3.25 macro ASR-BLEU (argmax) and the zero-shot cascade reached 2.14, a ratio of about 1.5. However, this difference cannot be considered as evidence of an intrinsic architectural advantage of the direct system. The proposed system was trained on dataset_214k and is therefore adapted to its lexical, syntactic, and acoustic characteristics, whereas the MMS-ASR, MADLAD-400-10B, and MMS-TTS components were used off the shelf and had not been adapted to this corpus. When qualitatively examining the output of the cascaded system, it was observed that the machine translation stage largely preserved the meaning. However, it often paraphrased the text, which resulted in a low BLEU score based on exact word matching. Despite preserving meaning, the BLEU score decreased due to the lack of exact n-gram matching. We therefore present the cascade as a reference against an unadapted baseline. The comparison shows where the two-pass system stands relative to off-the-shelf components on this corpus; it is not evidence that the two-pass design is better than a cascade adapted to the same data.

For context, in a recent study of Turkic speech-to-speech translation, Karibayeva et al. [46] compared a fine-tuned cascade (Whisper-medium recognition, GPT translation, and per-language speech synthesis) with a fine-tuned SeamlessM4T v2 model and reported an increase in average BLEU from 4.68 to 30.60 for their fine-tuned cascade. The languages, test sets, and metric pipelines differ from ours, so the numbers are not directly comparable, but they show that an in-domain fine-tuned cascade can be a much stronger baseline than the zero-shot cascade used here.

4.10. Human Evaluation

To complement the automatic metrics, a blinded listening test was run through the heval.law-ai.kz interface. Fifty clips were selected: 25 outputs of the proposed system and 25 ground-truth MMS-TTS target recordings, balanced by language and condition (Turkish 8 and 8, Tatar 8 and 8, Uzbek 9 and 9). Clips were presented in random order without system labels. Three native-speaker raters rated naturalness on a five-point MOS scale. One rater rated all 50 clips and the other two rated 45 and 28, so each clip received one to three ratings (2.46 on average). After removing 5 skipped items, 123 valid ratings remained. Means are computed over individual ratings, with 95% t-intervals over ratings. With three raters, an inter-rater agreement statistic would be unstable, so none is reported. Raters judged naturalness only; they did not judge intelligibility or whether the output preserved the meaning of the source sentence.

Table 12 reports MOS separately for each language and condition.

Table 12. Human MOS evaluation of speech naturalness (1 to 5 scale) by language and condition.

Language	Condition	Clips	Ratings	Mean MOS [95% CI]
tur	System	8	20	4.05 [3.61, 4.49]
tur	MMS-TTS	8	20	4.25 [3.88, 4.62]
tat	System	8	17	4.00 [3.45, 4.55]
tat	MMS-TTS	8	19	4.16 [3.76, 4.56]
uzb	System	9	25	3.56 [3.16, 3.96]
uzb	MMS-TTS	9	22	4.36 [4.04, 4.69]
All	System	25	62	3.84 [3.59, 4.09]
All	MMS-TTS	25	61	4.26 [4.06, 4.46]

When the two evaluation conditions were analyzed separately, the proposed system obtained a mean MOS of 3.84 (95% CI [3.59, 4.09]), whereas the MMS-TTS ground-truth target audio achieved 4.26 (95% CI [4.06, 4.46]). Thus, the human-evaluated naturalness gap between the two conditions was 0.42 MOS points. At the clip level (mean rating per clip), the system is lower by 0.41 points, with a 95% bootstrap CI of [0.14, 0.67] for the gap (10,000 resamples; Mann–Whitney U test, p = 0.013). The gap comes mostly from Uzbek (0.78 points, CI [0.41, 1.17], p = 0.004); for Turkish and Tatar the differences are small and not significant (p = 0.48 and p = 0.96). These tests treat clips as independent and do not model rater effects, so with three raters they are indicative only. For comparison, the automatic SQUIM-MOS evaluation produced scores of 4.21 for the proposed system and 4.34 for the ground-truth audio. The human scores are therefore both lower and more clearly separated than the corresponding SQUIM estimates. This difference is not contradictory. SQUIM-MOS is a metric that automatically assesses speech quality. Human listeners may be more sensitive to residual artifacts in naturalness, prosody, and synthesis than automatic metrics. Therefore, human evaluations complement, rather than replace, the SQUIM-MOS results.

5. Discussion

5.1. Revisiting the Research Questions

We now relate the empirical findings of Section 4 to the three research questions posed in the Introduction. RQ1—whether a morphologically informed architecture trained from scratch is sufficient for Kazakh-to-Turkic S2ST under low-resource conditions—is answered negatively: as shown in Section 4.4, none of the scratch-trained configurations in Table 4 produced usable translation (ASR-BLEU 0, text BLEU of 0.14 or less). RQ2—what accounts for this failure—is addressed by the five controlled diagnostic experiments in Section 4.4, which make exposure bias, over-smoothing, and acoustic-continuity issues unlikely causes of the scratch-trained model’s failure, and point to the absence of a transferable target-language representation as the most likely cause; this explanation is inferred from the controls rather than measured directly. RQ3, which components are needed for non-zero end-to-end speech output, is answered across Section 4.5 and Section 4.6: replacing the scratch-trained encoder with a frozen, pretrained Whisper representation enables a working S2TT system (19.49 BLEU); separating content generation from acoustic rendering; and replacing English speech units with Turkic-trained units (mHuBERT-147) and a Turkic HiFi-GAN vocoder together raise end-to-end quality from ASR-BLEU 0 to 3.25 (Section 4.6). Absolute quality remains low, so this answers a feasibility question rather than establishing usable translation.

5.2. Error Analysis

The results obtained highlight several important factors that influenced the method’s effectiveness. In the case of S2ST for resource-constrained agglutinative languages, especially when training is based primarily on synthetic data, it has been found that using pretrained, content-rich representations is not only a way to improve quality but also a necessary condition for the model to work at all. As shown in Section 4.4 and Section 4.5, an encoder trained from scratch cannot form a stable representation suitable for translation. At the same time, the frozen Whisper encoder produces a workable translation right from the start. This confirms that pretrained speech representations play a key role in such conditions.

The second important result relates to the ‘content versus render’ task separation principle. The synthesis head should primarily address the task of vocalizing an already-formed content representation, rather than performing translation and acoustic generation simultaneously. This separation reduces the complexity of training and allows the responsibilities of individual model components to be localized: the encoder is responsible for content transfer, whilst the synthesis module is responsible for its speech realization.

The third finding concerns the linguistic specifics of Turkic phonetics. A Turkic quantizer is critical for accurate modeling of the target speech. English-language units result in the loss of phonemes as early as the k-means clustering stage, and this loss cannot be recovered through subsequent retraining of the vocoder. In contrast, single-stream Turkic units significantly raise the upper limit on quality: according to the experimental results, they raise the achievable ceiling by approximately 2.3 times.

The analysis has revealed the system’s main weakness. It is neither the vocoder nor length prediction. It is unit prediction in free-running mode: when the unit predictor has to condition on its own previous outputs instead of the ground-truth units, its accuracy drops sharply. Two measurements support this: the system output has a SQUIM-MOS of 4.21, close to that of the ground-truth audio, and the exposure-bias gap of the unit predictor is 0.418 (Section 4.7). Consequently, to improve the system, the priority is to enhance the robustness of the autoregressive unit prediction.

One potential approach to address this issue is to use an enhanced scheduled sampling method, based on teacher forcing, to narrow the gap between training and the model’s text generation. Furthermore, to prevent gradual error accumulation, we propose using a separate duration predictor to combine units and determine their durations. Such an approach for processing recurring units (units) and using a separate duration predictor to determine their durations is proposed. This approach reduces the length of the sequences to be processed, thereby improving the model’s efficiency. Furthermore, for Turkish, where the quality of the textual results is relatively low, beam search could be an effective solution. However, because this method has not been empirically tested in this study, its potential effectiveness requires further investigation.

5.3. Limitations

Despite the improvements achieved in the proposed system, several limitations remain.

Firstly, the training and primary evaluation corpus is synthetic and based on TTS-generated speech, with alignment performed at the sentence level. This setting limits the achievable end-to-end quality and may not fully reflect the variability of natural speech, including differences in speakers, prosody, pronunciation, and recording conditions. The FLEURS evaluation and the control in Section Sources of the Synthetic-to-Natural Gap show how large this effect is.

While the macro-average S2TT BLEU score on the synthetic test sample was 19.49, in natural speech it decreased to 3.56. Meanwhile, the end-to-end ASR-BLEU score decreased from 3.25 to 0.86. As Section Sources of the Synthetic-to-Natural Gap shows, this gap combines differences in content and domain, sentence length and vocabulary, reference translations, and acoustics. Therefore, in the future, it will be necessary to train or adapt the system on natural parallel speech data.

Secondly, the reliability of ASR estimation varies across languages. For example, the MMS-1b-all ASR system performs relatively poorly in Tatar: even on the ground-truth Tatar target audio, the ASR-BLEU score is only 15.09, compared with 33.60 for Uzbek and 31.50 for Turkish. Low Tatar ASR-BLEU scores therefore reflect the limits of the recognizer as well as errors of the S2ST system.

Thirdly, the absolute end-to-end translation quality remains limited. The best result is ASR-BLEU = 3.25 on the synthetic test sample and 0.86 on natural speech, and the average retention relative to the ground-truth-audio topline is about 13.4%. Therefore, the proposed system should be regarded as a research prototype demonstrating the technical feasibility of Kazakh-to-Turkic speech-to-speech translation under the investigated low-resource conditions rather than as a production-ready system.

Fourthly, although the two-pass system outperforms the fully pretrained zero-shot cascade baseline under the same test protocol (3.25 vs. 2.14 macro ASR-BLEU), this comparison is not controlled for domain adaptation. The two-pass system was trained on dataset_214k, whereas the MMS-ASR → MADLAD-400-10B → MMS-TTS cascade was constructed entirely from off-the-shelf components without corpus-specific fine-tuning. Accordingly, the result only shows how the two-pass system compares with unadapted components on this corpus. It cannot separate the effect of the architecture from the effect of training on in-domain data. Fine-tuning the cascade components (ASR, MT, and TTS) on the same data would likely improve the cascade and could narrow or reverse the difference. A cascade fine-tuned on dataset_214k is the stronger baseline and is left for future work.

Finally, the perceptual evaluation is subject to limitations associated with the relatively small human evaluation sample. In addition to the automatic SQUIM-based assessment, a blinded human MOS evaluation was conducted using three native-speaker raters and 123 valid ratings across 50 clips. The proposed system achieved a mean MOS of 3.84, compared with 4.26 for the MMS-TTS ground-truth condition. Although these results suggest that the synthesized speech is reasonably natural, the small number of raters limits the statistical strength of the perceptual conclusions and prevents a robust analysis of inter-rater agreement. Raters judged naturalness only. Whether the spoken output preserves the meaning of the source sentence was not assessed by humans, so meaning preservation is supported only by automatic measures (BLEU, chrF++, and COMET of the text pass and ASR-chrF++ of the speech output). Future evaluation will therefore involve a larger and more diverse group of native speakers and a broader set of speech samples.

6. Conclusions

This paper presents the development path of a Kazakh-to-Turkic speech-to-speech translation prototype, including Kazakh, Turkish, Tatar, and Uzbek. Unlike most publications, which report only successful results, this article provides a structured account of a series of reproducible failures, the diagnosis of which enabled us to consistently identify their causes and formulate architectural solutions that led to a working system.

In the first phase, the MorphoTurkic-S2ST architecture was developed and trained from scratch, and then tested. It included a Conformer encoder with an additional morpheme-based CTC head, a cross-lingual VQ codebook, and a LaBSE semantic anchor. Although each component appeared reasonable individually, the system failed to learn a translation in any of the approximately 10 configurations: the resulting ASR-BLEU score was 0. Five control experiments on this model made other causes of the failure unlikely: exposure bias, over-smoothing, the continuous form of the acoustic output, and a language-model shortcut. The most likely cause is that the encoder, trained on limited synthetic data, did not form a target representation suitable for translation; we infer this from the controls rather than measure it directly. Under such conditions, content transfer between languages does not occur. The ablation study showed that only component A (CTC for morphemes, −17% compared to val_mel) contributed positively. Component B (VQ codebook) did not affect the result, while component C (LaBSE) actually worsened the quality.

In the second stage, a working speech-to-text translation system (S2TT) was built, based on the frozen, pretrained Whisper encoder. A key architectural solution was the separation of responsibilities: the encoder extracts a rich content representation, while a trainable text decoder with language-based FiLM conditioning performs translation. A single model translates speech into three target languages, defined by a control vector. Using the frozen Whisper-small model allowed us to achieve a non-zero BLEU value (4.67) from epoch zero, while further training with Whisper-large-v3 ensured a monotonic increase in quality to 19.49 BLEU (chrF++ 49.44, COMET 0.710) on the test set.

In the third stage, the system was brought to full end-to-end speech-to-speech translation. Individual experiments revealed the limitations of the three synthesis approaches. EnCodec/RVQ acoustic codes, when generated freely, produced silence or speech with incorrect meaning. Vocoders trained from scratch on semantic units reached a performance ceiling due to excessive smoothing and could not improve results.

English units (HuBERT-large, LibriSpeech k-means) lost the distinctive features of Turkic phonetics during the quantization stage, and subsequent vocoder training was unable to correct this. The solution was to use single-stream Turkic units (mHuBERT-147, layer 9, Turkic k-means) in combination with a HiFi-GAN vocoder trained on Turkic audio. This transition increased the performance ceiling in the “reference units → vocoder → ASR” mode from 6.76 to 15.67, and the end-to-end ASR-BLEU from 0.64 (6-stream English units) to 3.25 (1-stream Turkic units), a 5.1-fold improvement on the same random test sample (Table 7). At the character level, the speech output kept more of the content (ASR-chrF++ about 31), and human raters judged its naturalness at 3.84 on a five-point scale, below the 4.26 naturalness of the ground-truth audios. The perceptual quality of the synthesis closely approached that of the MMS-TTS-synthesized ground-truth audio (SQUIM-MOS 4.21 vs. 4.34), and the system throughput was 9.3× real-time (RTF 0.107).

Bottleneck analysis revealed that the main limitation of the system is not the vocoder quality or the accuracy of sequence length prediction (pred/gt ratio = 1.000, Pearson = 0.939), but the gap between teacher-forcing and free-running generation in autoregressive unit prediction: the accuracy in teacher-forcing mode is 0.811, while in free-running mode it drops to 0.393 (exposure-bias gap = 0.418). This indicates a priority direction for further work. This gap belongs to the final two-pass system and is separate from the diagnostics of the scratch-trained model.

The combined results allow us to formulate three methodological conclusions relevant for the development of S2ST systems for low-resource agglutinative languages. First, pretrained content representations are not simply a means of improving quality, but a necessary condition for system learnability in the face of a shortage of parallel speech data. Second, the principle of separating tasks “content-rendering” allows for localization of component responsibilities and increases training robustness. Third, the linguistic correspondence between the quantizer, units, and vocoder and the phonetics of the target languages is critical: the loss of phonological information during the quantization stage is not restored by further training of the synthetic modules.

Promising areas for further research include the use of reduced units with a duration predictor and enhanced scheduled sampling to reduce the exposure-bias gap; beam-search decoding for the Turkish language; training on real (not synthetic) parallel Turkic audio; the inclusion of specialized S2ST (BLASER 2.0) metrics, studio-based MOS scoring, human ratings of meaning preservation and intelligibility, and semantic similarity via SONAR; and the extension of the system to additional Turkic languages. A further direction is a text-free variant: predicting units directly from encoder hidden states without materializing text at inference, which would move the system from the current two-pass design toward a single-pass one; we leave this for future work, as it requires retraining the unit predictor on a different conditioning signal.