Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System

Abstract

This paper presents the first prototype of a speech-to-sign language translation system for Kazakh Sign Language (KRSL). The proposed pipeline integrates the NVIDIA FastConformer model for automatic speech recognition (ASR) in the Kazakh language and addresses the challenges of sign language translation in a low-resource setting. Unlike American or British Sign Languages, KRSL lacks publicly available datasets and established translation systems. The pipeline follows a multi-stage process: speech input is converted into text via ASR, segmented into phrases, matched with corresponding gestures, and visualized as sign language. System performance is evaluated using word error rate (WER) for ASR and accuracy metrics for speech-to-sign translation. This study also introduces the first KRSL dataset, consisting of 1200 manually recreated signs, including 95% static images and 5% dynamic gesture videos. To improve robustness under resource-constrained conditions, a Weighted Hybrid Similarity Score (WHSS)-based gesture matching method is proposed. Experimental results show that the FastConformer model achieves an average WER of 10.55%, with 7.8% for isolated words and 13.3% for full sentences. At the phrase level, the system achieves 92.1% accuracy for unigrams, 84.6% for bigrams, and 78.3% for trigrams. The complete pipeline reaches 85% accuracy for individual words and 70% for sentences, with an average latency of 310 ms. These results demonstrate the feasibility and effectiveness of the proposed system for supporting people with hearing and speech impairments in Kazakhstan.

1. Introduction

1.1. Problem Statement and Motivation

Hearing impairment affects a considerable portion of the population in Kazakhstan. According to the Ministry of Labor and Social Protection of the Population of the Republic of Kazakhstan, more than 30,000 people in Kazakhstan have hearing impairments [1]. Many of them cannot afford hearing aids, as the average salary in the country is 429,328 tenge, and the median salary is 302,584 tenge [2]. Even data on Kazakh Sign Language is difficult to obtain, and this area has hardly been researched. There is an undeniable communication barrier between people with hearing and speech impairments and hearing people. Sign language is a language like other languages but based on gestures rather than spoken language. Unfortunately, despite the rapid development of technology, the systems that already exist to overcome this barrier are still far from ideal and are not used in everyday life.

The lack of technological infrastructure for KRSL creates significant barriers in education, healthcare, and everyday communication. Unlike other well-studied sign languages, KRSL does not have large-scale video recordings or publicly available datasets. As a result, even basic tasks such as automatic gesture recognition and sign language translation remain unsolved. Ultimately, this research contributes to bridging the technological and social divide faced by people with hearing and speech impairments in Kazakhstan’s deaf community, enabling meaningful communication and promoting greater social integration.

1.2. Current State of Research

Kazakh Sign Language (KRSL) is low-resource sign language for which there is a lack of high-quality data and modern solutions. This contrasts with ASL, BSL, and other sign languages, for which there are open datasets, video recordings, and machine learning tools.

A review of peer-reviewed publications from 2021 to 2025 shows that existing research in Kazakhstan focuses mainly on sign language recognition, alphabet classification, or dataset construction.

While some studies address sign-to-speech or sign-to-text translation, no published work was found that implements an end-to-end speech-to-sign translation pipeline for Kazakh speech.

Therefore, this study represents the first reported attempt to bridge spoken Kazakh and Kazakh Sign Language within a unified technical framework.

1.3. Research Gap and Objectives

The lack of data for KRSL highlights a gap in sources. Although there are systems for other sign languages, similar approaches cannot be applied to KRSL due to linguistic differences.

Research objectives:

• To create the first structured digital data for Kazakh Sign Language based on a sign dictionary.
• To solve ASR noise problems.
• To design and implement a complete speech-to-sign translation pipeline for the Kazakh language.

1.4. Methodology Overview

The proposed system has a modular architecture that converts Kazakh speech into a sequence of gestures. First, an automatic speech recognition (ASR) model transcribes spoken Kazakh speech into text. The resulting text is then normalized and segmented using an n-gram-based approach with explicit handling of negative constructions. Each segment is matched to a customizable set of KRSL gesture data using a weighted hybrid similarity function that combines character-level, lexical, and semantic similarity measures. Finally, the system extracts the corresponding gesture as a static image or short video clip.

2. Related Works

2.1. Literature Review Methodology

Relevant works were collected from academic databases, including IEEE Xplore, ACM Digital Library, ScienceDirect, and arXiv. Keywords such as “sign language recognition,” “sign language translation,” “ASL dataset,” “BSL dataset,” “deep learning-based gesture recognition,” “pose-based sign language recognition,” and “low-resource sign language” were searched.

The selection criteria for inclusion in the review were as follows: (i) peer-reviewed journal articles and conference proceedings published between 2015 and 2025, to ensure relevance to contemporary deep learning methods; (ii) studies proposing new datasets, recognition or translation models, or systems for sign language processing; and (iii) works providing quantitative experimental evaluation.

2.2. Sign Language Datasets

The field of sign language recognition is currently being actively studied around the world, with progress driven by the availability of large datasets, whereas early systems were limited, which hindered model training and development.

One of the best datasets for working on sign language recognition is the WLASL dataset, presented by Lee et al. for American Sign Language (ASL), which covers over 2000 glosses [3]. The data was collected from various sources and carefully filtered. WLASL has become one of the most prominent datasets for word-level language recognition.

For British Sign Language (BSL), Albanie et al. proposed the BSL-1K dataset, which focuses on continuous sign language recognition under unconstrained conditions [4]. Unlike laboratory-recorded datasets, BSL-1K is derived from television broadcast footage, capturing natural signing variability, co-articulation, and background clutter. The dataset incorporates both visual hand cues and mouthing information, enabling multimodal modeling. Experimental results demonstrate that integrating mouthing cues significantly improves recognition accuracy.

Datasets for other sign languages, such as Chinese Sign Language (CSL) and German Sign Language (DGS), have also been developed. The RWTH-PHOENIX-Weather corpus is frequently used for real-time recognition and translation of German Sign Language [5]. However, these datasets were created for languages that have a strong infrastructure.

In contrast to previous works, KRSL does not have publicly available datasets. The lack of large, well-annotated datasets limits the applicability of previous works to KRSL and motivates the alternative approach to dataset construction adopted in this work.

2.3. Overview of Sign Language Translation Systems

2.3.1. Traditional and Early Deep Learning Approaches

Early sign language recognition systems relied on handcrafted features such as the Histogram of Oriented Gradients (HOG), SIFT, and frequency-domain descriptors. Temporal modeling was typically performed using Hidden Markov Models (HMMs), Conditional Random Fields, or Dynamic Time Warping (DTW), while classification was carried out using models such as Support Vector Machines (SVMs) [5,6,7]. These systems were generally evaluated on small datasets with limited vocabularies.

With the advent of deep learning, sign language recognition experienced significant performance improvements. Inspired by advances in action recognition, researchers began applying Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to model spatial and temporal information in sign language videos. CNN–LSTM architectures became a common design, enabling frame-level feature extraction followed by temporal sequence modeling [8,9].

2.3.2. Image Appearance-Based Methods

Image appearance-based methods focus on learning representations directly from raw video frames. Simonyan et al. introduced two-stream CNN architectures for action recognition, which inspired similar approaches in sign language recognition. Subsequent works employed CNN–LSTM and CNN–GRU architectures to capture temporal dynamics [10].

More recent approaches utilize 3D CNNs, such as C3D and I3D, which jointly model spatial and temporal information [11,12]. These architectures have been successfully adapted for sign language recognition, achieving strong performance on large datasets. Transformer-based models, including adaptations of BERT, have also been explored to capture long-range dependencies in sign sequences [13].

2.3.3. Pose-Based Methods

Pose-based works reduce background noise and visual variability by focusing on hu-man joint movements; they do not consider information about any pixels. Deep learning advances in human pose estimation have been developed at the highest level, enabling researchers to find accurate key points that explain the movements of the hands, forearms, and upper body, which are crucially important in sign language. One of the first deep learning-based pose estimation methods is DeepPose, proposed by Toshev and Segedi, which directly regresses human joint coordinates from images using Convolutional Neural Networks [14]. In a follow-up work by Thompson et al., key point estimation based on heatmaps combined with graphical models was presented, which significantly improved localization accuracy [15].

In the context of sign language recognition, pose sequences are typically modeled using Recurrent Neural Networks (RNNs) to capture temporal dependencies between consecutive frames. For example, Koller et al. used pose-based representations combined with temporal modeling to improve robustness to background noise and lighting changes [16].

Graph convolutional networks (GCNs) address the above limitations and were introduced to model spatial and temporal dependencies in pose sequences. Yan et al. proposed GCNs (ST-GCNs) for skeleton-based action recognition, which inspired new pose-based sign language recognition methods. Exploring this idea, Liu et al. and Zhang et al. applied graph architectures to sign language datasets, demonstrating improved recognition performance by explicitly modeling joint connectivity and motion dynamics [17,18].

2.3.4. Sign Language Translation Systems

Beyond recognition, several works have explored sign language translation. Fang et al. introduced DeepASL, one of the earliest systems capable of translating ASL gestures into text at both word and sentence levels using deep neural networks [19]. Subsequent studies have investigated video-based sign-to-text translation pipelines using encoder–decoder architectures and attention mechanisms [20,21]. Despite these advances, most translation systems focus on sign-to-text translation and rely on high-resource languages. Very few works address speech-to-sign translation, and none target low-resource sign languages such as KRSL. This gap motivates the retrieval-based, data-efficient approach proposed in this paper.

2.3.5. Recent Studies on Sign Language Processing

Research activity remains relatively limited with regard to sign languages with limited resources. In 2022, the KRSL-OnlineSchool corpus was presented, one of the first structured datasets of educational videos for Kazakh–Russian Sign Language. This work focused on corpus construction and linguistic annotation rather than automatic translation or multimodal pipelines [22].

Between 2023 and 2024, several studies were conducted in Kazakhstan on gesture recognition and the Kazakh Sign Language alphabet. These studies typically used Convolutional Neural Networks, recurrent architectures, or object detection-based pipelines to classify individual hand gestures or dactyl signs. Although these approaches showed promising classification accuracy, they were limited to gesture recognition tasks and did not address translation or speech input [23,24].

In 2025, recent studies have explored more sophisticated linguistic resources for Kazakh Sign Language. These include the construction of parallel corpora of texts and glosses, as well as rule-based methods for aligning texts and glosses to support future research in sign language translation [25]. Other works have proposed sign-to-speech translation systems in which recognized signs are converted into spoken Kazakh using sign recognition pipelines [26]. While these studies make valuable contributions to resource development and address accessibility issues, they work in the opposite direction and do not consider spoken speech as the primary input modality. A summary of related work on sign language datasets and translation systems is presented in Table 1.

Table 1. Summary of related work on sign language datasets and translation systems.

Work	Task	Key Results
Li et al. (WLASL)	ASL word-level recognition	Large-scale ASL benchmark
Albanie et al. (BSL-1K)	Continuous BSL recognition	Improved accuracy with mouthing cues
Koller et al. (RWTH-PHOENIX)	Continuous SL recognition/translation	Standard benchmark for DGS
Fang et al. (DeepASL)	ASL sign-to-text translation	High accuracy in controlled settings
Camgoz et al.	Neural SL translation	End-to-end sign-to-text translation
Tran et al. (C3D)	Video-based recognition	Spatio-temporal feature learning
Carreira & Zisserman (I3D)	Video recognition	State-of-the-art video features
Toshev & Szegedy (DeepPose)	Human pose estimation	Direct joint regression
Tompson et al.	Pose estimation	Accurate heatmap-based keypoints
Yan et al. (ST-GCN)	Skeleton-based recognition	Graph-based spatial–temporal modeling
Koller et al.	Pose-based SL recognition	Robust to background noise

3. Methodology

This section describes methods for developing a system for translating Kazakh speech into sign language. The methodology includes creating a set of sign language data and developing a modular translation system.

3.1. Dataset Construction

To address the lack of publicly available resources for Kazakh Sign Language (KRSL), we constructed a dedicated gesture dataset based on an official thematic sign language dictionary entitled “In the World of Gestures (Thematic Dictionary of Sign Language for Deaf Citizens of Kazakhstan)”. Each gesture was manually recreated according to the dictionary description to ensure linguistic consistency and semantic correctness.

The dataset creation process involved several steps. First, gestures were reproduced by trained signers in accordance with the dictionary definitions. Second, recordings were performed under controlled environmental conditions, including consistent background and lighting, to ensure visual clarity and minimize noise. Third, each gesture was saved as either a static image (PNG format) or a short video clip (MP4/MPEG format), depending on whether the gesture contained temporal motion. Finally, each gesture instance was manually annotated and assigned a corresponding lexical label.

The resulting dataset consists of 1200 gesture entries, including both static images and dynamic gesture videos. The main statistical characteristics of the constructed KRSL gesture dataset are summarized in Table 2.

Table 2. Statistical overview of the KRSL gesture dataset.

Attribute	Value
Total gesture entries	1200
Static images (PNG)	~95%
Dynamic video clips (MP4/MPEG)	~5%
Number of signers	2
Recording conditions	Controlled background and lighting
Lexical coverage	Basic pronouns, nouns, verbs, adjectives, selected phrase-signs
Non-manual markers (facial expression, head movement)	Not explicitly annotated
Continuous signing/co-articulation	Not modeled
Spatial grammar phenomena	Not represented

As shown in Table 2, the dataset predominantly contains static gesture images (~95%), with a smaller proportion of dynamic gesture videos (~5%). The recordings were collected from two signers under controlled recording conditions and cover fundamental lexical categories, including pronouns, nouns, verbs, adjectives, and selected phrase-level expressions.

However, it is important to note that certain linguistic aspects of natural sign language communication are not explicitly represented in the current dataset. In particular, non-manual markers such as facial expressions and head movements were not annotated, and continuous signing phenomena such as co-articulation and spatial grammar were not modeled. These limitations reflect the prototype-oriented design of the dataset and the current resource constraints in KRSL research.

This statistical overview improves transparency and provides a clearer understanding of the dataset scope, structure, and limitations, which is essential for interpreting the experimental results presented in this study.

Due to ethical and practical considerations, the KRSL gesture dataset has not been publicly released at this stage but can be provided for academic research upon reasonable request.

Representative examples of both static and dynamic gestures included in the dataset are shown in Figure 1.

Figure 1. Examples of static and dynamic hand gestures from the KRSL dataset.

3.2. Overview of the Proposed System

The system follows a sequential, modular architecture that translates spoken Kazakh into a sequence of sign gestures. The overall pipeline is illustrated in Figure 2 and consists of the following components.

Figure 2. Overview of the proposed speech-to-sign translation pipeline.

Each module is designed to compensate for the limitations of low-resource language settings and to ensure robustness against transcription noise and linguistic variability.

3.3. Speech Recognition (ASR Module)

To convert Kazakh speech into text, we use the FastConformer model from the NVIDIA Nemo framework. This model is one of the most accurate speech-to-text converters and has 1024 subtokens [27]. The processes are as follows:

• The audio signal is converted to a 16 kHz format.
• The FastConformer model predicts the sequence of tokens.
• The text is reconstructed using the decoding method.
• Text preprocessing and phrase segmentation are carried out.

3.4. Text Preprocessing and Phrase Segmentation

Kazakh is an agglutinative language in which words often contain multiple suffixes that express grammatical relations. As a result, a single spoken word may correspond to a phrase-level gesture in Kazakh Sign Language. This linguistic property creates challenges for direct word-to-gesture mapping and motivates the use of an n-gram-based segmentation strategy. The proposed segmentation module allows the system to detect both individual word gestures and phrase-level gestures (e.g., negative constructions such as “әдeмi eмec”). This leads to the creation of long words with several affixes. In Kazakh Sign Language, some gestures denote an entire phrase rather than individual words. Therefore, text processing is carried out in several stages:

• Text normalization:
  • Converting to lowercase;
  • Removing extra spaces.
• Tokenization:
  • The text is divided into separate words.
• Generation of n-grams:
  For example, the phrase “aғaш әдeмi eмec” generates:
  • Unigrams: aғaш, әдeмi, eмec;
  • Bigrams: әдeмi eмec;
  • Trigrams: aғaш әдeмi eмec.

3.5. Gesture Matching Module

After text segmentation, the system performs each version (word or n-gram) with gestures from the dataset. Since Kazakh Sign Language is low-resource, a Weighted Hybrid Similarity Score (WHSS) was developed for matching:

• Character-based (Levenshtein ratio) [28];
• Lexical (Jaccard similarity) [29];
• Semantic (cosine similarity between Sentence-BERT embeddings) [30].

3.5.1. Levenshtein Ratio

$$Lev\left(c,d\right)=1-{\displaystyle \frac{\mathrm{EditDistance}\left(c,d\right)}{\max \left(\left|c\right|,\left|d\right|\right)}},$$

(1)

EditDistance—the minimum number of operations (insertion, deletion, replacement), with a value in the range [0, 1], where 1 = exact match.

This allows for accurate comparison of:

“Agas” → “aғaш“;

“Ademies” → “әдeмi eмec”.

3.5.2. Jaccard Similarity

$$Jac\left(c,d\right)={\displaystyle \frac{\left|W_c\cap W_d\right|}{\left|W_c\cup W_d\right|}}$$

(2)

Used to detect phrase gestures (әдeмi eмec, жaқcы eмec):

W_c: set of candidate words;

W_d: set of dictionary gesture words.

3.5.3. Cosine Similarity

To determine the semantic similarity between an ASR candidate and a dictionary entry, we use the cosine similarity between their sentence embeddings. Given two embedding vectors E_c and E_d, the cosine similarity is defined as follows:

$$\cos \left(E_c,E_d\right)={\displaystyle \frac{E_c\cdot E_d}{\left|E_c\right|\left|E_d\right|}}$$

(3)

3.5.4. Weighted Hybrid Similarity Score (WHSS)

Formula (WHSS):

$$Score\left(c,d\right)=0.55\cdot Lev\left(c,d\right)+0.25\cdot Jac\left(c,d\right)+0.20\cdot \cos \left(E_c,E_d\right)$$

(4)

• c—candidate from ASR (word or n-gram);
• d—lexical unit from the dataset;
• “$Lev$” (c, d)—Levenshtein ratio;
• “$Jac$” (c, d)—Jaccard similarity;
• cos (E_c, E_d)—cosine similarity of SBERT embeddings.

Weight Selection and Normalization. The WHSS combines three complementary similarity measures—character-level robustness (Lev), phrase-level lexical overlap (Jac), and semantic consistency (cosine similarity of Sentence-BERT embeddings). The weights $\left(w_{lev}, w_{jac}, w_{cos}\right)$ are constrained to sum to one and were tuned on a held-out development subset to maximize retrieval accuracy under ASR noise. In our setting, ASR errors are dominated by character-level distortions (e.g., vowel substitutions and merged tokens); therefore the Levenshtein ratio is assigned the largest weight. Jaccard similarity is emphasized to better capture multiword constructions and negation patterns that frequently correspond to phrase-level gestures in the dictionary. The cosine similarity term provides semantic smoothing for morphological variation and near-synonymous expressions. The final weights $(0.55, 0.25, 0.20)$ were selected as the best-performing configuration on the development subset, and their robustness is confirmed via an ablation study and a sensitivity analysis reported in Section 4.4.

3.6. Summary of Architecture

Figure 3 shows the complete cycle of the system for translating Kazakh speech into sign language. The architecture consists of five parts: speech recognition (ASR), text processing, linguistic segmentation, hybrid gesture matching (WHSS), and gesture retrieval. The contribution of each similarity component and the robustness of the WHSS weights are empirically assessed in Section 4.4.

Figure 3. Complete system cycle.

The process begins with raw audio, which is fed into the ASR model (FastConformer). Using an algorithm, the model converts Kazakh speech into text, which may contain recognition noise typical of languages with limited resources.

The next step includes all data in segments being evaluated in the hybrid gesture matching module, which combines Levenshtein similarity, Jacard similarity, and SBERT-based cosine similarity into a weighted hybrid score. This ensures reliable matching even with ASR distortions, morphological variations, and verbose expressions.

The system then queries the gesture dataset, which contains static gesture images as well as short dynamic gesture videos. The gesture retrieval module selects the correct media asset associated with the matched gesture.

Finally, the selected gestures are assembled into a gesture output sequence, representing the visual translation of the speaker’s original utterance.

4. Results

This section presents the results of experiments with a system for translating Kazakh speech into sign language. We evaluate the system on three levels: (i) ASR transcription quality, (ii) gesture matching efficiency, and (iii) end-to-end translation quality.

4.1. Experimental Setup

To evaluate the Kazakh-to-sign language translation system, we created a dataset based on the World of Gestures (Thematic Dictionary of Sign Language for Deaf Citizens of Kazakhstan). Since there are no publicly available KRSL datasets, the dataset contains gestures related to five basic lexical categories that are widely used in everyday communication: pronouns, nouns, verbs, adjectives, and phrases (Table 3).

Table 3. Examples of lexical categories in the KRSL gesture dataset.

Pronouns	Nouns	Verbs	Adjectives	Phrase-Signs
мeн (I)	aғaш (tree)	кeлy (come)	үлкeн (big)	әдeмi eмec (not beautiful)
ceн (you)	cy (water)	бapy (go)	кiшi (small)	жaқcы eмec (not good)
oл (he/she)	кiтaп (book)	көpy (see)	жaқcы (good)
бiз (we)	бaлa (child)	aлy (take)	әдeмi (beautiful)

The evaluation focuses on four major components of the system: speech recognition quality, text segmentation accuracy, gesture matching performance, and overall end-to-end translation behavior. First, we assess the automatic speech recognition (ASR) module by computing the word error rate (WER) on a set of recorded Kazakh utterances spoken by different speakers under natural conditions. This allows us to measure how accurately the FastConformer model transcribes input audio and how ASR noise influences downstream processing.

To evaluate the effectiveness of the proposed Kazakh-to-sign language translation system, we conducted experiments on a test set consisting of 180 audio recordings (140 individual words and 40 complete sentences). The dataset included 95 static images of gestures and 14 short videos with gestures covering five lexical categories: pronouns, nouns, verbs, adjectives, and phrase-level expressions.

The proposed approach is a search-based system rather than a trainable deep learning model. Therefore, the test set was not used for training or parameter learning. The WHSS weights were determined using a small independent development subset that is entirely disjoint from the 180-audio evaluation set.

The individuals who performed the gestures during dataset construction are different from the speakers who recorded the audio used in the evaluation phase. Therefore, there is no overlap between the gesture signers and the evaluation speakers.

We first evaluated the FastConformer-ASR module using word error rate. Across all recordings, the ASR achieved:

• WER = 7.8% on isolated words;
• WER = 13.3% on full sentences;
• Overall average WER = 10.55%.

Most errors were due to vowel substitutions (ә/e, o/ө) and merged tokens (әдeмieмec instead of әдeмi eмec).

4.2. Text Segmentation Performance

To evaluate the n-gram segmentation and negation handling components, we com-pared predicted segments against manual annotations. The results are as follows:

• Uni-gram accuracy: 92.1%
• Bi-gram accuracy: 84.6%
• Tri-gram (phrase-level) accuracy: 78.3%

4.3. End-to-End Translation Accuracy

Assessment of the accuracy of the entire processing system by comparing predicted gesture sequences with reference sequences yielded:

• Word recognition accuracy from start to finish: 85.8%;
• Sentence recognition accuracy (all gestures recognized correctly): 70.5%;
• Average output delay: 310 ms per utterance.

Analysis shows that failures in end-to-end recognition are most often associated with the automatic speech recognition system combining words or incorrectly separating negative phrases.

4.4. WHSS Ablation and Weight Sensitivity

Since WHSS is a key component of the proposed pipeline, we evaluate (i) the individual contribution of each similarity term and (ii) the robustness of the weighting scheme. All experiments in this subsection use the same evaluation protocol described in Section 4.1, and we report the same end-to-end metrics as in Section 4.3.

4.4.1. Ablation Study

We first evaluate simplified variants of the matching function by removing one or more similarity terms. Specifically, we compare Lev only, Jac only, Cos only, Lev + Jac, Lev + Cos, Jac + Cos, and the full WHSS (Lev + Jac + Cos). This ablation isolates the effect of each term under low-resource ASR noise and phrase segmentation errors. The results are summarized in Table 4. The sensitivity of the system to different WHSS weight configurations is presented in Table 5. Overall, the full WHSS achieves the best performance, indicating that character-level robustness, phrase-level overlap, and semantic similarity provide complementary benefits.

Table 4. WHSS ablation results.

Matching Variant	Word Accuracy (%)	Sentence Accuracy (%)
Lev only	82.9	64.2
Jac only	61.5	38.0
Cos only	70.4	51.3
Lev + Jac	84.2	67.6
Lev + Cos	84.7	68.4
Jac + Cos	73.8	53.5
Full WHSS (Lev + Jac + Cos)	85.8	70.5

Table 5. Sensitivity of end-to-end performance to WHSS weights.

Lev	Jac	Cos	Word Accuracy (%)	Sentence Accuracy (%)
0.55	0.25	0.20	85.8	70.5
0.60	0.25	0.15	85.6	70.1
0.50	0.30	0.20	85.4	69.8
0.33	0.33	0.33	83.9	68.0

4.4.2. Weight Sensitivity Analysis

Next, we assess how sensitive the system is to the choice of weights $\left(w_{lev}, w_{jac}, w_{cos}\right)$, subject to $w_{lev}+w_{jac}+w_{cos} = 1$. We evaluate several representative weight configurations, including an equal-weight baseline and variants that downweigh or remove one component. Table 5 reports the results. Performance remains stable within a neighborhood of the selected weights, while configurations that remove the character-level term degrade performance the most, which is consistent with the observed prevalence of ASR character-level distortions in Kazakh speech (Section 4.1). These findings support the selected weighting scheme and confirm that the matching module is not overly brittle with respect to moderate weight changes.

5. Discussion

This section discusses the results of experiments in relation to the current state of the art and highlights the main advantages of the proposed approach. The discussion is based on the research objectives defined in the introduction and focuses on the effectiveness of the system under conditions of limited resources.

5.1. Comparison with the State of the Art

Recent research on sign language translation has primarily focused on deep learning architectures trained on large-scale video datasets. Examples include systems developed for American Sign Language (ASL) and British Sign Language (BSL), which rely on datasets such as WLASL, BSL-1K, and RWTH-PHOENIX-Weather. These approaches typically use Convolutional Neural Networks, Recurrent Neural Networks, or transformer-based models to learn continuous sign language representations from thousands of annotated video sequences.

In contrast, the proposed system addresses a fundamentally different scenario. Kazakh Sign Language (KRSL) is a low-resource language with no large, annotated video datasets. Therefore, instead of relying on end-to-end deep learning models, we adopt a search-based architecture that combines automatic speech recognition with hybrid similarity-based gesture retrieval. This design enables speech-to-sign translation in environments where large training corpora are unavailable.

There are large and successful models for translating speech into sign language, such as American Sign Language (ASL) and British Sign Language (BSL). These systems are mainly trained using large amounts of resources, where there are annotated datasets that can be trained using deep learning. The reported results indicate strong performance in high-resource settings.

Direct numerical comparison with large-scale end-to-end sign language translation systems (e.g., ASL or BSL models trained on continuous video corpora) is not straightforward due to fundamentally different assumptions and data availability. While existing systems rely on large annotated video datasets and model continuous signing phenomena, our approach addresses a low-resource setting through dictionary-level retrieval and modular processing. Therefore, the proposed system should be viewed as a data-efficient baseline enabling speech-to-gesture mapping in the absence of large KRSL corpora.

5.2. Effectiveness of the Proposed Methodology

The results of the experiments show that each component contributes to the result. The FastConformer-based ASR model provides sufficiently accurate transcriptions for subsequent processing, although ASR errors remain the main cause of failures. Our n-gram segmentation strategy has proven effective in detecting gestures at both the word and phrase levels. The gesture matching module, based on the proposed Weighted Hybrid Similarity Score (WHSS), plays an important role as it reduces ASR noise in this part. It combines similarity at the word level.

5.3. Answers to the Research Questions

Research demonstrates that it is possible to create speech-to-gesture translation systems for low-resource languages without access to large annotated video datasets. Second, experiments show that hybrid gesture alignment based on similarity can effectively bridge the gap between imperfect ASR output and dictionary-based gesture representation. Finally, the achieved end-to-end translation accuracy confirms that a search-based modular pipeline is a viable solution for KRSL under current resource constraints.

5.4. Limitations

Despite promising prototype-level results, several important limitations remain. First, the KRSL dataset is limited in size and predominantly consists of static images, which restricts the modeling of natural temporal dynamics in sign language.

In addition, the dataset does not explicitly represent non-manual markers such as facial expressions and head movements, which play a crucial linguistic role in sign languages. Spatial grammar and co-articulation effects are also not modeled. As a result, the system does not capture the full linguistic richness of natural KRSL discourse.

Consequently, the reported performance should be interpreted as evidence of feasibility in a limited vocabulary, controlled-setting scenario rather than as validation of a real-world continuous sign language translation system.

6. Conclusions

This article examined the problem of translating Kazakh speech into sign language with limited resources. Currently, there are no datasets or pre-trained models and translation systems available. The lack of technological solutions creates a significant communication barrier between people with hearing impairments and hearing people in Kazakhstan and limits access to education, healthcare, and other services. To address this issue, we proposed a working speech-to-sign language translation system developed specifically for resource-constrained environments. The methodology combines automatic speech recognition using the FastConformer model, text normalization, and n-gram-based linguistic segmentation. The KRSL gesture set was created manually based on a dictionary, and a Weighted Hybrid Similarity Score (WHSS) was used for reliable matching.

The results of the experiments show that the proposed system demonstrates reliable performance with an accuracy of over 70%. These results confirm that hybrid similarity-based retrieval can effectively compensate for ASR noise. Compared with recent deep learning-based sign language translation systems that require large annotated datasets, the proposed approach demonstrates that a modular retrieval-based architecture can provide a practical solution for low-resource sign languages such as KRSL.

However, there are some limitations. The gesture dataset is very limited in size. The system also depends on the results of automatic speech recognition, and errors that may occur at this stage can spread throughout the entire pipeline. These limitations indicate that this system should be viewed as an experimental version rather than a fully ready-to-use system, but it can be seen as the first step toward a system for translating the Kazakh language into gestures.

Future work will focus on expanding the KRSL dataset by adding fully dynamic gesture sequences, including temporal modeling and pose-based representations, as well as exploring avatar-based gesture synthesis. There are also plans to further improve the reliability of the automatic speech recognition system and integrate real-time translation functions. These enhancements are aimed at creating a scalable and deployment-ready assistive communication system for people with hearing and speech impairments in Kazakhstan.

Based on a review of the literature and experimental analysis, this study can be considered the first documented prototype of a speech-to-sign language translation system for the Kazakh language in peer-reviewed publications for 2021–2025. The proposed system establishes basic principles for future research in the field of Kazakh sign language translation and creates a foundation for scalable assistive communication technologies in Kazakhstan.

Given the current dataset characteristics—limited size and predominantly static gestures—the proposed system should be viewed as a prototype suitable for controlled and limited vocabulary scenarios. While the results demonstrate the feasibility of retrieval-based speech-to-gesture mapping for Kazakh, real-world deployment would require multi-signer dynamic corpora, modeling of non-manual markers, and support for continuous signing phenomena. The present study therefore establishes a technical foundation rather than a complete translation solution.