Skip to Content
You are currently on the new version of our website. Access the old version .
Applied SciencesApplied Sciences
  • Article
  • Open Access

11 February 2026

12 Pages

When Data Is Scarce: Training a Kazakh Speech Language Model from Discrete Units

and
Faculty of Information Technology and Artificial Intelligence, Al-Farabi Kazakh National University, Almaty 050040, Kazakhstan
*
Author to whom correspondence should be addressed.

Abstract

This research explores the development of a decoder-only speech language model (SLM) for Kazakh, a language currently characterized by limited computational resources. Our approach leverages discrete acoustic units synthesized from self-supervised speech representations. Specifically, we utilize a pretrained Wav2Vec 2.0 model to extract continuous latent features, which are then transformed into discrete semantic tokens via the k-means clustering algorithm. These tokens serve as the foundation for training a generative model designed to predict and maximize the likelihood of speech-unit sequences. To facilitate this study, we curated a specialized Kazakh speech corpus by synthesizing and refining multiple publicly available audio datasets. Given the constrained hardware resources available, we conducted large-scale feature extraction and tokenization to train the unit-based model. We evaluated the system’s efficacy using negative log-likelihood and perplexity metrics on independent test sets. The model captures Kazakh vowel harmony but struggles with long-range agglutinative chains. Key observations include the model’s high sensitivity to data quality, tokenization techniques, and specific training hyperparameters. Although constrained by data volume and training time relative to global benchmarks, the model successfully captures the underlying structural patterns in Kazakh speech. This work establishes a vital empirical baseline and suggests future improvements through refined unit discovery and integrated speech-text modeling.

1. Introduction

Recent advances in generative modeling have demonstrated that speech can be treated as a symbolic sequence in much the same way as text, enabling the direct application of language modeling techniques to acoustic data [1]. This paradigm shift has renewed interest in speech language models (SLMs), which aim to learn probability distributions over speech signals without relying on explicit text supervision. Such models are of particular importance for languages where annotated linguistic resources remain scarce, yet large quantities of raw speech data are increasingly available. A dominant approach to generative speech modeling relies on discretizing continuous acoustic representations into symbolic units, often referred to as semantic or speech tokens [2,3]. These units are typically obtained by clustering self-supervised speech representations, most commonly derived from HuBERT-style models, using algorithms such as k-means [4,5,6]. Discretization enables scalable training with standard language modeling objectives but remains the subject of ongoing debate. On the one hand, discrete units have been shown to capture linguistically meaningful structure and facilitate the unsupervised discovery of phonetic patterns [7,8]. On the other hand, several studies highlight their sensitivity to clustering granularity and their limited ability to encode long-range linguistic dependencies, particularly when evaluated solely through unit-level perplexity [9,10]. As a result, the interpretability and generalization of unit-based evaluation metrics remain contested. More recently, the Slamming framework has proposed initializing models from pre-trained text language models, rather than training them from random initialization [11,12,13]. This approach exploits the strong sequential inductive biases learned from large-scale text corpora, substantially reducing computational requirements and accelerating convergence. Existing evaluations, however, have primarily focused on high-resource or synthetic training regimes, leaving open questions about the feasibility of this Slam-style strategy for real-world, low-resource languages trained under explicitly constrained computational budgets. Kazakh provides a particularly informative test case. As an agglutinative language, Kazakh encodes grammatical relations through extended suffix chains, producing word forms with complex internal structure. This raises the question of whether sub-phonetic acoustic units, clustered primarily for similarity in latent space, can adequately capture the long-range dependencies required for modeling such morphology. At the same time, the availability of large-scale open corpora such as the ISSAI Kazakh Speech Corpus enables empirical investigation using diverse, naturally occurring speech data rather than curated or synthetic material [14,15]. In this work, we examine the practical feasibility of training a decoder-only generative speech-language model for Kazakh using discrete speech units and initialization from a pre-trained text-language model. We adopt SLM as the general model class and use Slamming as the designated initialization strategy. We trained a decoder-only Transformer language model operating on discrete speech units. Model weights were initialized from Qwen/Qwen2.5-0.5B, a 0.5B-parameter decoder-only text language model from the Qwen2.5 family [16]. Following the Slamming approach, the original token-embedding layer was replaced to match the discrete speech-unit vocabulary, while all other architectural components and pretrained parameters were left unchanged. All experiments are conducted under explicitly defined constraints: a single consumer-grade GPU, a fixed training budget, and publicly available data. Model performance is assessed using unit-level perplexity, which we treat as a relative diagnostic rather than a direct proxy for semantic competence, in line with recent critiques of perplexity-based evaluation for speech language models [17,18,19,20]. Our results show that, even under restricted computational conditions, the Slam-style training recipe can be successfully applied to Kazakh speech, yielding stable training dynamics and interpretable perplexity trends. These findings suggest that efficient generative speech modeling is attainable for morphologically rich, low-resource languages, while also underscoring the need for complementary evaluation methods beyond perplexity in future work.

2. Materials and Methods

The system pipeline transforms raw audio waveforms into discrete tokens via an XLSR-53 encoder, followed by k-means clustering and run-length encoding. This symbolic sequence is then modeled by a decoder-only Transformer initialized with weights from the Qwen2.5-0.5B text model.

2.1. Speech Data

Experiments were conducted using the ISSAI Kazakh Speech Corpus 2 (KSC2), an open Kazakh speech dataset released by the Institute of Smart Systems and Artificial Intelligence [14,15]. The corpus includes recordings from multiple domains, including broadcast speech, crowdsourced recordings, and text-to-speech material.
After filtering out corrupted and unreadable files, the final dataset contained approximately 627,000 audio files, totaling 26.7 h of speech. The high number of files relative to total duration reflects extensive segmentation in several corpus subsets, resulting in many very short utterances. Analysis of the segment distribution revealed a mean utterance duration of approximately 0.15 s. While this granularity provides a dense training signal for local acoustic transitions, it necessitates careful batching to avoid gradient instability. To address the apparent inconsistency between a high file count and throughput, we used dynamic padding in the data loader. The reported effective batch size of 1024 tokens (Section 3.2) refers to the total number of accumulated run-length encoded (RLE) pairs per step, rather than full utterances, ensuring consistent GPU memory utilization despite the high frequency of short audio segments. All audio was stored in FLAC format and resampled to 16 kHz during preprocessing. The full pipeline is shown in Figure 1.
Figure 1. Overview of the Kazakh SLAM-based speech language modeling pipeline.
All available training subsets were merged into a single pool. A development set of 31,391 utterances was created by random sampling at the file level, leaving 596,433 utterances for training. The dataset is publicly available via the Hugging Face Hub (issai/Kazakh_Speech_Corpus_2); no access restrictions apply.

2.2. Acoustic Feature Extraction and Unit Discovery

Continuous acoustic representations were extracted using the aismlv/wav2vec2-large-xlsr-kazakh checkpoint available on Hugging Face, based on the XLSR-53 architecture [21,22,23,24,25]. Audio waveforms were converted to mono, normalized, and passed through the model without fine-tuning.
Discrete speech units were obtained by clustering frame-level representations using the k-means algorithm, following established practice in generative spoken language modeling [1,17]. Frames were randomly sampled from the training split, with a maximum of one million frames to limit memory usage. Clustering was performed with 500 clusters using MiniBatch K-Means and a fixed random seed. The resulting centroids were saved and reused across all experiments.

2.3. Tokenization and Sequence Compression

Each speech utterance was converted into a sequence of discrete acoustic units by assigning every frame-level representation to its nearest k-means centroid. This nearest-centroid assignment produces a dense sequence of unit identifiers operating at the frame rate of the underlying self-supervised representation model.
Raw unit sequences obtained in this manner are typically long and highly redundant, as consecutive frames often map to the same centroid, especially in steady acoustic regions such as sustained vowels or silence. To mitigate this redundancy, run-length encoding (RLE) was applied to all unit sequences. Under this scheme, consecutive identical units were collapsed into (unit, duration) pairs, where the duration denotes the number of consecutive frames assigned to the same centroid. Specifically, duration information was modeled using discrete auxiliary tokens. For each symbolic unit u i , a duration index d i ∈ { 1 , … , D max } was appended, where D max denotes the maximum observed run length of identical centroids in the training data.
During training, the model’s output head was extended to perform joint multi-task sequence prediction:
P ( u i + 1 , d i + 1 ∣ u < i + 1 , d < i + 1 ) ,
and the total training objective was defined as the sum of the cross-entropy losses over both the unit tokens and their corresponding durations:
L = − ∑ i log P ( u i ∣ u < i , d < i ) + log P ( d i ∣ u ≤ i , d < i ) .
This formulation encourages the model to learn both the phonetic structure and the temporal rhythm of speech in an end-to-end manner (in practice, the two prediction terms are typically computed from a shared backbone with separate classification heads for units and durations).
This compression step reduced the effective sequence length by approximately one order of magnitude on average, substantially decreasing memory consumption and computational cost during training. At the same time, duration information preserves coarse temporal structure, allowing the model to distinguish between brief and sustained acoustic events without operating at the original frame resolution.
The resulting compressed sequences were serialized in JSONL format, with each line corresponding to a single utterance. These representations constituted the sole input modality used for training and evaluation. No textual transcripts, phoneme annotations, or linguistic supervision were employed at any stage of the pipeline, ensuring that all learned structure emerged exclusively from acoustic information.

2.4. Model Architecture and Initialization

The speech-language model employed in this study uses a decoder-only Transformer architecture that operates on sequences of discrete speech units. The model predicts the next unit (or run-length encoded unit–duration pair) autoregressively, using standard causal self-attention and position-wise feed-forward layers.
Rather than training the model from random initialization, all parameters were initialized from a pre-trained text-only language model, in line with the SLAM training paradigm [11]. This initialization strategy leverages the strong sequential inductive bias acquired during large-scale text pre-training, particularly in modeling long-range dependencies and hierarchical structure. Such priors are expected to transfer to speech token sequences, which, despite their acoustic origin, exhibit non-random ordering constraints analogous to textual syntax.
To adapt the model to the speech domain, the original text vocabulary and embedding layer were replaced with a new embedding matrix that maps to the discrete speech-unit inventory. The rest of the architecture stayed the same. We kept the attention blocks, normalization layers, and feed-forward networks exactly as they were. Positional encodings were reused directly, as unit sequences preserve a linear temporal ordering comparable to tokenized text.
No additional architectural changes, auxiliary losses, or modality-specific components were introduced. This design choice ensures that any observed modeling capability can be attributed primarily to the initialization strategy and the properties of the discrete speech representation, rather than to architectural specialization.

2.5. Training Procedure and Compute Budget

Training was performed on a single consumer-grade GPU with 24 GB of VRAM and 16 CPU cores. The primary experiment was constrained to approximately 4000 optimization steps, corresponding to roughly 34 h of wall-clock time.
Optimization used AdamW [26,27,28] with a learning rate of 2 × 10 − 4 , linear warmup over 5% of training steps, and gradient accumulation to achieve an effective batch size of 128 sequences.
Mixed-precision training was not used. Preliminary experiments indicated numerical instability when applying FP16/BF16 to highly compressed run-length encoded unit sequences. Full-precision training was therefore retained to ensure stable convergence under the selected batch-size and sequence-length regime.

2.6. Evaluation

Performance was evaluated using unit-level perplexity on the held-out development set. Perplexity measures the average negative log-likelihood of the observed unit sequences under the model and reflects how well the model predicts discrete acoustic tokens given their preceding context. In the present setting, perplexity is computed over sequences of run-length encoded speech units and therefore depends directly on the chosen discretization scheme, including the number of clusters and the compression strategy.
As a result, perplexity is interpreted as a relative indicator of model fit rather than an absolute measure of linguistic competence. Lower perplexity indicates improved internal consistency and predictive alignment with the training distribution, but it does not directly quantify semantic understanding or grammatical knowledge. This diagnostic use of perplexity is consistent with foundational SLM benchmarks in high-resource languages such as English and Hebrew, where initial model fit is established via unit-prediction likelihood before advancing to zero-shot semantic evaluations.
In this study, perplexity is therefore used primarily to compare training dynamics across checkpoints and to assess convergence behavior under constrained computational budgets. While useful for monitoring optimization progress, perplexity is complemented by qualitative analyses of generated unit sequences, and future work will incorporate downstream or zero-shot semantic evaluations to better assess the linguistic properties captured by the model [29,30,31].

3. Results

This section reports the empirical outcomes of training a decoder-only Speech Language Model on discrete speech units derived from the KSC2 corpus. The emphasis is placed on training dynamics, computational feasibility, and qualitative linguistic behavior rather than absolute benchmark performance.

3.1. Training Dynamics and Loss Behavior

Rather than a smooth monotonic decrease, the loss trajectory exhibited a characteristic two-phase behavior as shown in Figure 2. During the initial phase (steps 0–500), the training cross-entropy dropped sharply from approximately 3.0 to 2.6, forming a clear “elbow” in the loss curve. This suggests rapid alignment between the pre-trained text-language-model weights and the newly introduced acoustic token embeddings.
Figure 2. Representative training dynamics of the UnitLM model trained on KSC2 discrete speech tokens. The curve illustrates the evolution of the cross-entropy loss over 4000 optimization steps. Early rapid loss reduction is followed by gradual convergence, consistent with alignment of the text-initialized decoder to acoustic unit sequences.
After this point, loss reduction slowed considerably, with gradual improvements interspersed with short plateaus lasting 200–300 steps. These plateaus were more pronounced when batches were dominated by crowdsourced conversational speech, whereas broadcast and read speech segments continued to yield smoother decreases in loss. This suggests that the higher acoustic variability and inconsistent prosody of the crowdsourced subsets pose a greater modeling challenge than the broadcast and read-speech portions of the corpus.
At the end of training, the model converged to the following:
Final training loss: 2.31
Final validation loss: 2.48
The persistent gap between training and validation loss indicates mild underfitting rather than overfitting, which is expected given the limited number of training steps relative to the dataset size.

3.2. Computational Performance

Despite the constrained training regime, the pipeline remained computationally stable throughout the run. Approximately, the average throughput was as follows:
  • 27–32 s per optimization step;
  • Effective batch size: 1024 tokens (via gradient accumulation);
  • Total wall-clock training time: ∼35 h.
While this throughput is slower than reports based on professional GPUs (e.g., A5000 or A100), it confirms that meaningful SLM training is feasible on a single high-end consumer GPU without distributed training or synthetic data augmentation.
To validate the architecture’s effectiveness beyond perplexity, we monitored the gradient norm throughout the 4000 steps. The absence of gradient spikes or sudden loss diversions, even when processing heterogeneous crowdsourced data, confirms that the full-precision training regime and AdamW optimizer settings were sufficient to stabilize the 0.5B-parameter backbone under constrained compute. This stability across various corpus subsets suggests that the model is robust to the high acoustic variability inherent in the KSC2 dataset.

3.3. Token-Level Perplexity

Model performance was evaluated using token-level perplexity on the held-out validation split. Perplexity is computed as follows:
PPL val = exp − 1 N ∑ i = 1 N log P ( u i ∣ u < i )
where u i denotes a discrete acoustic unit obtained via k-means quantization.
The final validation perplexity achieved by the model was as follows:
PPL val = 10.12
Given the 500-cluster codebook and the exclusive use of speech-only supervision, this value is consistent with prior speech-language modeling studies using similar discretization strategies. In this context, perplexity serves as a relative indicator of model fit rather than an absolute measure of linguistic competence.

3.4. Qualitative Linguistic Observations

Although no downstream semantic evaluation was conducted, qualitative inspection of generated unit sequences revealed language-specific regularities. In particular, the model exhibited sensitivity to Kazakh vowel harmony, despite having no explicit access to phonemic or morphological annotations. Qualitative analysis indicates that the model captures Kazakh vowel harmony regularities despite being trained solely on discrete acoustic units. In particular, plural suffix-like unit patterns corresponding to -lar and -ler were generated conditionally on the vowel quality of the stem.
As shown in Figure 3, unit sequences corresponding to the back-vowel plural marker -lar were assigned higher probability following stems dominated by back vowels (e.g., bala), whereas front-vowel stems (e.g., äke) favored the front-vowel form -ler. These tendencies emerged consistently across speakers and recording conditions.
Figure 3. Next-token probability distribution illustrating emergent vowel harmony behavior. The model assigns higher probability to the back-vowel plural unit -lar following back-vowel stems and to the front-vowel form -ler following front-vowel stems, indicating sensitivity to non-local phonotactic structure.
This behavior suggests that the model has internalized non-local phonotactic dependencies that extend beyond immediate acoustic frames, despite the absence of explicit morphological or phonological supervision. In this context, discrete-unit language modeling appears sufficient to recover aspects of Kazakh agglutinative structure.
However, the model struggled with long agglutinative chains, often truncating or repeating unit sequences in highly inflected forms. This behavior highlights a known limitation of sub-phonetic discretization schemes, which tend to emphasize local acoustic similarity over long-range morphological structure.

3.5. Summary of Findings

The experimental results support the following conclusions:
  • Decoder-only speech language models initialized from text LLMs can be adapted to Kazakh using real-world speech data alone;
  • Stable convergence is achievable within a 36-h single-GPU budget, though training exhibits plateaus linked to acoustic variability;
  • Token-level perplexity reaches competitive values given the absence of text supervision or synthetic data;
  • The model captures linguistically meaningful patterns specific to Kazakh, including vowel harmony, while remaining limited in handling extended agglutinative morphology.
These findings demonstrate the feasibility of applying the SLAM paradigm to a low-resource, morphologically rich language under realistic computational constraints, while also revealing concrete limitations that motivate further investigation.

4. Discussion

This study evaluated the feasibility of training a decoder-only speech language model (SLM) for Kazakh under strict computational constraints. By utilizing the industrial-scale KSC2 corpus, we tested whether the SLAM paradigm, which initializes speech models with pretrained text Transformers, generalizes to a morphologically complex, low-resource language without requiring architectural modifications.
The applicability of the SLAM framework to Kazakh is rooted in the assumption that text-initialized Transformers possess universal sequential priors. While we focus on Kazakh, this methodology is intrinsically extensible to other morphologically rich, low-resource languages. The discrete-unit bottleneck allows the model to bypass language-specific phonetic labels, making it a viable strategy for any language where raw audio significantly outweighs annotated transcripts.
We did not run a complete set of component removal experiments against randomly initialized baselines. However, the pronounced bend (“elbow”) observed in the training loss curve (Figure 2) provides indirect support for the effectiveness of initializing from a pre-trained text language model. Previous work within the SLAM framework has shown that models trained from scratch on datasets of comparable scale typically exhibit an extended cold-start phase, characterized by slow early convergence and unstable optimization, and often fail to reach stable performance within a training budget comparable to the 34 h used in our experiments.
The primary objective of the present work is therefore not to quantify relative gains across alternative initialization strategies, but to assess the practical feasibility of applying text-initialized generative speech language modeling to Kazakh under explicitly constrained computational conditions.
The training efficiency we observed was heavily front-loaded. The "elbow" in the loss curve during early optimization suggests that text-derived weights provide a robust inductive bias for speech sequencing. This likely stems from the model’s existing grasp of linguistic logic, such as hierarchical structure and long-range dependencies, learned during text pretraining. This initialization allowed the model to map discrete acoustic tokens to an established latent space within a few hundred steps. Conversely, the plateauing observed in later stages suggests that the primary bottleneck is not sequence modeling itself, but rather the adaptation to fine-grained acoustic variability under a fixed compute budget.
Differences across corpus subsets further highlight the role of data characteristics. Validation loss was consistently higher for crowdsourced recordings than for broadcast-style segments, suggesting that increased speaker diversity, background noise, and prosodic inconsistency pose a greater modeling challenge than studio-quality speech. This observation suggests that the greater acoustic variability and inconsistent prosody of the crowdsourced subsets pose a greater modeling challenge than those in the broadcast data. From a practical standpoint, this points toward data-centric optimization strategies. For instance, a curriculum that begins with high-SNR broadcast speech and gradually introduces noisier crowdsourced recordings may yield more stable late-stage convergence without increasing overall training time.
From a linguistic perspective, qualitative analysis revealed that the model internalized aspects of Kazakh vowel harmony despite operating exclusively on discrete acoustic units. The emergence of suffix-like patterns conditioned on stem vowel quality indicates that the model learned dependencies spanning multiple frames and tokens, rather than relying solely on local acoustic correlations. This is notable given Kazakh’s agglutinative structure, where grammatical information is distributed across morpheme boundaries and not explicitly marked at the phoneme level. These findings suggest that discrete unit representations, while sub-phonetic in nature, can support the acquisition of morphophonological regularities when combined with a sufficiently expressive sequence model.
Evaluation remains a central limitation. While perplexity provides a useful proxy for model fit, its absolute value depends directly on the number of discrete units and therefore cannot be interpreted in isolation. In this context, perplexity serves as a relative indicator of model fit, enabling comparisons across training configurations and data subsets. Future work should incorporate task-based evaluations, such as zero-shot semantic plausibility tests or downstream speech understanding benchmarks, to better assess whether improvements in perplexity correspond to gains in linguistic competence.
Finally, this study underscores a broader implication for low-resource speech modeling. The results indicate that meaningful generative speech models can be trained without large-scale synthetic pipelines or multi-GPU infrastructure, provided that initialization and data selection are handled carefully. While the present model does not match the scale or performance of state-of-the-art multilingual systems, it demonstrates that SLAM-style training constitutes a viable entry point for investigating generative speech modeling in languages where both data and computational access remain limited.
These findings position SLAM not as a replacement for large-scale speech modeling efforts, but as a practical framework for probing the feasibility and limits of generative speech modeling in underrepresented languages under realistic constraints.

5. Conclusions

This study investigated the feasibility of training a decoder-only speech language model for Kazakh using discrete acoustic units under explicitly constrained computational conditions. By applying the SLAM training paradigm to the publicly available KSC2 corpus, we demonstrated that meaningful generative speech modeling is achievable for a morphologically complex, low-resource language without relying on synthetic data or large-scale distributed infrastructure.
Empirically, the results show that initializing a speech model from a pretrained text Transformer enables rapid early convergence. Most performance gains were realized within the first few thousand optimization steps, suggesting that pretrained text representations provide a robust inductive bias for sequential structure.
The experimental results demonstrate that the decoder-only architecture successfully identifies the underlying structural logic of Kazakh speech patterns without semantic supervision. Specifically, the model’s recovery of Kazakh vowel harmony as a complex phonotactic rule serves as empirical evidence that sub-phonetic discrete units support the emergence of non-local linguistic regularities. This acquisition of language-specific structural patterns, achieved entirely from raw acoustic data, proves that the SLAM initialization provides a sufficient inductive bias for modeling the morphophonology of agglutinative languages even under constrained computational budgets.
However, later-stage improvements were limited by acoustic variability, particularly within the crowdsourced subsets of the corpus. This highlights the critical role of data quality and curriculum design when operating under fixed training budgets.
Qualitative analysis further indicated that the model captured non-local linguistic regularities characteristic of Kazakh, such as vowel harmony patterns across suffix boundaries. These findings suggest that discrete unit representations are sufficient to support the emergence of morphophonological structure when paired with an expressive auto-regressive model, despite their sub-phonetic nature.
Several limitations remain. Evaluation was primarily based on perplexity, which serves as a relative indicator of model fit rather than a direct measure of linguistic competence. Additionally, the present study focused on a single backbone architecture and a fixed number of discrete units. Future work should extend this analysis to task-based semantic evaluations, alternative unit inventories, and curriculum strategies that better accommodate heterogeneous acoustic conditions.
In conclusion, this work demonstrates that SLAM-style speech modeling is a practical approach for underrepresented languages, using real-world data and modest computational resources. While the results do not yet match large-scale multilingual systems, the approach offers a viable framework for language-specific research in settings where both data and compute remain constrained.

Author Contributions

Methodology, B.K.; Validation, B.K.; Data curation, B.K.; Writing—review & editing, B.K.; Supervision, M.M. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the Ministry of Science and Higher Education of the Republic of Kazakhstan under grant number BR24993001, titled “Creation of a Large Language Model (LLM) to Support the Use of the Kazakh Language and Enhance Technological Progress”.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data presented in this study are openly available in the ISSAI repository at https://doi.org/10.48342/m90y-aj02, (accessed on 4 February 2026) and described in detail by Mussakhojayeva et al. [15]. The scripts for acoustic feature extraction, k-means unit discovery, and the SLAM training recipe used in this study are publicly available on GitHub at https://github.com/bauyrjanesq/kazakhslam, (accessed on 4 February 2026).

Acknowledgments

The authors would like to acknowledge Marek Kubis for his valuable academic discussions, methodological feedback, and expert guidance provided during the preparation and revision of this manuscript.

Conflicts of Interest

The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Lakhotia, K.; Del Rio, M.; Joulin, A.; Dupoux, E. Generative Spoken Language Modeling from Raw Audio. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Brno, Czech Republic, 30 August–3 September 2021; pp. 4593–4597. [Google Scholar]
  2. Chung, Y.-A.; Zhang, Y.; Han, W.; Chiu, C.-C.; Qin, J.; Pang, R.; Wu, Y. W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training. In Proceedings of the 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Cartagena, Colombia, 13–17 December 2021; pp. 2440–2444. [Google Scholar]
  3. Meister, C.; Cotterell, R. Language Model Evaluation Beyond Perplexity. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 5328–5339. [Google Scholar]
  4. Hsu, W.-N.; Bolte, B.; Tsai, Y.-H.H.; Lakhotia, K.; Salakhutdinov, R.; Mohamed, A. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Trans. Audio Speech Lang. Process. 2021, 29, 3451–3460. [Google Scholar] [CrossRef] [Scilit]
  5. Lloyd, S. Least squares quantization in PCM. IEEE Trans. Inf. Theory 1982, 28, 129–137. [Google Scholar] [CrossRef] [Scilit]
  6. Sculley, D. Web-scale k-means clustering. In Proceedings of the 19th International Conference on World Wide Web, Raleigh, NC, USA, 26–30 April 2010; pp. 1177–1178. [Google Scholar]
  7. Dunbar, E.; Algayres, R.; Benjumea, J.; Bernard, M.; Benjumea, J.; Cao, X.-N.; Miskic, L.; Dugrain, C.; Ondel, L.; Black, A.W.; et al. The Zero Resource Speech Challenge 2019: TTS without T. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Graz, Austria, 15–19 September 2019; pp. 1088–1092. [Google Scholar]
  8. Elmakies, A.; Abend, O.; Adi, Y. Unsupervised Speech Segmentation: A General Approach Using Speech Language Models. arXiv 2025, arXiv:2501.03711. [Google Scholar] [CrossRef] [Scilit]
  9. Polyak, A.; Adi, Y.; Copet, J.; Kharitonov, E.; Lakhotia, K.; Hsu, W.N.; Mohamed, A.; Dupoux, E. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Brno, Czech Republic, 30 August–3 September 2021; pp. 3615–3619. [Google Scholar]
  10. Kharitonov, E.; Lee, A.; Polyak, A.; Adi, Y.; Copet, J.; Lakhotia, K.; Nguyen, T.A.; Riviere, M.; Mohamed, A.; Dupoux, E.; et al. Text-Free Prosody-Aware Generative Spoken Language Modeling. In Proceedings of the International Conference on Learning Representations (ICLR), Online, 25–29 April 2022. [Google Scholar]
  11. Maimon, G.; Elmakies, A.; Adi, Y. Slamming: Training a Speech Language Model on One GPU in a Day. arXiv 2025, arXiv:2502.15814. [Google Scholar]
  12. Tang, S.; Yu, W.; Tan, C.; Chen, J.; Lu, Z.; Li, J.; Zhou, P.; Zhang, Y. SALMONN: Towards Generic Hearing Abilities for Large Language Models. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 7–11 May 2024. [Google Scholar]
  13. Maiti, S.; Peng, Y.; Choi, S.; Jung, J.-W.; Chang, X.; Watanabe, S. VOXTLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks. In ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); IEEE: Piscataway, NJ, USA, 2024. [Google Scholar] [CrossRef] [Scilit]
  14. Khassanov, Y.; Mussakhojayeva, S.; Mirzakhmetov, A.; Adiyev, A.; Nurpeiissov, M.; Varol, H.A. A Crowdsourced Open-Source Kazakh Speech Corpus. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Brno, Czech Republic, 30 August–3 September 2021; pp. 697–701. [Google Scholar]
  15. Mussakhojayeva, S.; Khassanov, Y.; Varol, H.A. KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus. In Proceedings of the 23rd Interspeech Conference, Incheon, Republic of Korea, 18–22 September 2022; pp. 1367–1371. [Google Scholar]
  16. Qwen; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; et al. Qwen2.5 Technical Report. arXiv 2025, arXiv:2412.15115. [Google Scholar] [CrossRef] [Scilit]
  17. Maimon, G.; Hassid, M.; Roth, A.; Adi, Y. Scaling Analysis of Interleaved Speech-Text Language Models. arXiv 2025, arXiv:2406.02398v1. [Google Scholar] [CrossRef] [Scilit]
  18. Nguyen, T.A.; Sagot, B.; Dupoux, E. Are Discrete Units Necessary for Spoken Language Modeling? IEEE J. Sel. Top. Signal Process. 2022, 16, 1415–1423. [Google Scholar] [CrossRef] [Scilit]
  19. Borsos, Z.; Marinier, R.; Vincent, D.; Kharitonov, E.; Pietquin, O.; Sharifi, M.; Roblek, D.; Teboul, O.; Grangier, D.; Tagliasacchi, M.; et al. AudioLM: A Language Modeling Approach to Audio Generation. IEEE/ACM Trans. Audio Speech Lang. Process. 2023, 31, 2522–2537. [Google Scholar] [CrossRef] [Scilit]
  20. Serrano, S.; Brumbaugh, Z.; Smith, N.A. Language Models: A Guide for the Perplexed. arXiv 2023, arXiv:2311.17301. [Google Scholar] [CrossRef] [Scilit]
  21. Conneau, A.; Baevski, A.; Collobert, R.; Mohamed, A.; Auli, M. Unsupervised Cross-Lingual Representation Learning for Speech Recognition. In Proceedings of the International Conference on Machine Learning (ICML), Virtual, 13–18 July 2020; pp. 2426–2435. [Google Scholar]
  22. Baevski, A.; Hsu, W.-N.; Xu, Q.; Babu, A.; Gu, J.; Auli, M. data2vec: A General Framework for Self-Supervised Learning in Speech, Vision and Language. In Proceedings of the 39th International Conference on Machine Learning (ICML), Baltimore, MD, USA, 17–23 July 2022; pp. 1298–1312. [Google Scholar]
  23. Baevski, A.; Zhou, Y.; Mohamed, A.; Auli, M. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Adv. Neural Inf. Process. Syst. (NeurIPS) 2020, 33, 12449–12460. [Google Scholar]
  24. Babu, A.; Wang, C.; Tjandra, A.; Gulati, K.; Metze, F.; Gu, J. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. arXiv 2021, arXiv:2111.09296. [Google Scholar]
  25. Chen, Y.; Zhang, H.; Yang, X.; Zhang, W.; Qu, D. Meta-Adaptable-Adapter: Efficient Adaptation of Self-Supervised Models for Low-Resource Speech Recognition. Neurocomputing 2024, 587, 128493. [Google Scholar] [CrossRef] [Scilit]
  26. Loshchilov, I.; Hutter, F. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France, 24–26 April 2017. [Google Scholar]
  27. Zhuang, J.; Tang, T.; Ding, Y.; Tatikonda, S.C.; Dvornek, N.; Papademetris, X.; Duncan, J. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. Adv. Neural Inf. Process. Syst. (NeurIPS) 2020, 33, 18795–18806. [Google Scholar]
  28. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4171–4186. [Google Scholar]
  29. Jelinek, F.; Mercer, R.L.; Bahl, L.R.; Baker, J.K. Perplexity—A measure of the difficulty of speech recognition. J. Acoust. Soc. Am. 1977, 62, S63. [Google Scholar] [CrossRef] [Scilit]
  30. LY Corporation; Carnegie Mellon University. Evaluating Self-Supervised Speech Models via Text-based LLMs. arXiv 2025, arXiv:2510.04463. [Google Scholar]
  31. Arisoy, E.; Menevse, M.U.; Manav, Y.; Ozgur, A. Evaluating Large Language Models in Data Generation for Low-Resource Scenarios: A Case Study on Question Answering. In Proceedings of the Interspeech 2025, Rotterdam, The Netherlands, 17–21 August 2025; pp. 17–21. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.