Tagalog Lip-Reading System Using 3D Convolutional Neural Network with Bidirectional Long Short-Term Memory ^†

Abstract

We present a Tagalog lip-reading system designed to enhance communication accessibility for individuals with hearing impairments. Existing lip-reading models focus on English and other major languages and cannot recognize Tagalog visual speech patterns. To address this gap, we implemented 3D Convolutional Neural Network combined with Bidirectional Long Short-Term Memory network, supported by a custom Tagalog dataset of common words. This architecture achieved an average character error rate of 10.09%, word error rate of 24.08%, and overall word accuracy of 76.27%, demonstrating promising recognition accuracy for Tagalog lip movements. By introducing the Tagalog-specific lip-reading framework, the potential of deep learning-based visual speech recognition was validated to support inclusive technologies, with applications in daily communication, education, and assistive tools for the Filipino deaf community.

1. Introduction

In the Philippines, around 7.5% of children and 14.7% of adults live with hearing impairments, which hamper their ability to communicate effectively, and where communication strategies such as lip-reading come into play [1,2]. Lip-reading systems rely solely on visual cues/movements made by the speaker’s lips, or visemes, to interpret and predict the speaker’s speech [3]. While lip movements are central, other facial features, such as jaw motion, cheek muscle activity, and eye expressions [4,5], provide complementary cues that enhance speech recognition accuracy. However, most lip-reading systems focus on major languages, such as English [6,7]. Hence, lip-reading systems are limited in applicability in countries such as the Philippines, particularly given its diverse set of dialects [8].

The dataset employed in this study was derived from the structure and parameters of the GRID Corpus, a widely used audiovisual resource for speech and lip-reading research. Each video clip in the GRID Corpus has a fixed duration of three seconds, recorded at 25 frames per second, yielding 75 frames per clip [6]. A resolution of 360 × 288 pixels was consistently maintained to ensure standardized visibility of facial and lip movements.

The model architecture adopted in this study was based on LipNet, the first end-to-end lip-reading framework [9]. LipNet introduced the integration of 3D-CNNs with Bidirectional Long Short-Term Memory (Bi-LSTM) layers, trained using Connectionist Temporal Classification (CTC) loss. This design enables the handling of variable-length input sequences without requiring explicit frame-to-character alignment. The 3D CNN extracts spatial and temporal features from consecutive frames, while Vi-LSTM captures contextual dependencies across time. The CTC loss subsequently aligns predicted outputs with target sequences. Subsequent studies have validated the effectiveness of this CNN–Bi-LSTM–CTC architecture for visual speech recognition [10,11].

By developing a lip-reading system for Tagalog, 3D-CNN with Bi-LSTM architecture is constructed to address the current absence of Tagalog-based lip-reading systems and datasets and enhance communication accessibility for the Filipino deaf community and contribute to the expansion of lip-reading research in the Philippines.

2. Materials and Methods

The system’s input is visual-only recordings of a person speaking. In addition to the study’s limitations, it focuses on common and basic Filipino words. 3D CNN pre-processes the video input (including video cropping and lip detection) and predicts the speech through the speaker’s lip movements. Bi-LSTM aids 3D CNN in training and tackles the vanishing gradient issue. After processing, the system outputs the predicted message of the speaker in text.

2.1. Model Architecture

The 3D-CNN layers process video frames as small 3D volumes, allowing the model to learn both spatial lip features and temporal motion. Unlike 2D CNNs that handle frames individually, 3D convolutions use 3 × 3 × 3 filters to capture subtle mouth movements such as opening and shifting between sounds. The extracted features are then passed to Bi-LSTM layers, which process the sequence forward and backward to understand the full context. Each Bi-LSTM layer has 128 hidden units with orthogonal initialization and a 0.5 dropout rate to ensure stable training and minimize overfitting. Finally, Dense and Softmax layers produce character probabilities for each frame, and the Connectionist Temporal Classification (CTC) loss aligns predictions with target sequences without exact frame-to-letter matching. The full architecture is summarized in

Table 1. 3D CNN and Bi-LSTM.

Layers	Outputs Shape
Input	(75, 40, 100, 1)
Conv3D	(75, 40, 100, 128)
MaxPool3D	(75, 20, 50, 128)
Conv3D	(75, 20, 50, 256)
MaxPool3D	(75, 10, 25, 256)
Conv3D	(75, 10, 25, 75)
MaxPool3D	(75, 5, 12, 75)
Flatten	(75, 4500)
Bi-LSTM	(75, 256)
Bi-LSTM	(75, 256)
Dense	(75, V)

.

2.2. Dataset Creation

The dataset centered around 74 random yet common Tagalog words used in day-to-day communication. Each set consisted of 80 lines, wherein there were three words per line that were randomized. Each video was three seconds long, shot with 25 frames per second, amounting to 75 frames per video, and with a resolution of 360 × 288. Transcription was done manually by matching the lip movement to the corresponding word the speaker spoke and logging the frame ranges to create the alignment files, as illustrated in Figure 1. A total of 35 sets were recorded, with 25 sets for training the system and 10 sets for testing. A total of four speakers were included in the dataset.

For the dataset, we applied video augmentation techniques to increase diversity and improve the model’s ability to generalize from limited data. Horizontal flipping was performed on all video samples, creating mirrored versions that helped the model recognize lip movements from both orientations. We adjusted brightness and contrast to generate lighter and darker variations, simulating real-world lighting differences. Each original video produced around five new augmented versions, expanding visual variability in lighting, contrast, and perspective.

Following this, we generated lighter and darker versions of every video to simulate real-world illumination differences that occur due to lighting direction, brightness levels, or camera exposure. Although normalization was later applied in preprocessing, brightness-based augmentation introduced meaningful pixel-level variability that normalization alone could not eliminate. This method made the model more robust to illumination changes and improved its ability to accurately predict speech under diverse visual conditions [12]. As a result, each original video sample produced augmented variants, including flipped, brightened, and darkened versions, which substantially increased the diversity and realism of the dataset and improved model adaptability and overall recognition performance (Figure 2).

2.3. Preprocessing and Feature Extraction

The preprocessing method adopts LipNet, focusing on precise mouth localization and consistent frame preparation for visual speech recognition [9]. Facial detection was performed using the Dlib library, with landmarks 48 to 67 used to isolate the mouth region. Each frame was converted to grayscale to emphasize motion over color, then cropped, resized, or padded to fit the model’s input size of 75 × 40 × 100 × 1. Normalization was applied to standardize pixel values and improve feature extraction (Figure 3).

2.4. System Performance Evaluation

We used character error rate (CER) and word error rate (WER) to evaluate the performance of the lip-reading system, both of which are standard metrics for assessing speech and visual speech recognition systems [13]. Both metrics are used to measure the accuracy of the system’s predictions by determining the minimum number of corrections required to bring it to ground-truth [9,14]. WER enables performance analysis at the sentence level, while CER provides analysis at the character level. Both metrics are computed using the Levenshtein distance algorithm, as shown in Equation (1), where N denotes the number of reference units (words or characters), S represents the number of substitutions, D the number of deletions, and I is the number of insertions required to align the predicted output with the ground truth [14].

$$ER={\displaystyle \frac{S+D+I}{N}}$$

(1)

Every prediction was compared to the ground-truth, wherein word-level performance was measured in terms of overall accuracy (A), precision (P), recall (R), and F1-score (Equation (2)).

$$\mathrm{A}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}, \mathrm{P}={\displaystyle \frac{TP}{TP+FP}}, \mathrm{R}={\displaystyle \frac{TP}{TP+FN}}, \mathrm{F}1=2 \ast {\displaystyle \frac{\mathrm{P}}{P+R}}$$

(2)

where TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

3. Results and Discussion

The system achieved an overall average CER and WER of 10.09 and 24.08% (

Table 2. Average CER and WER in testing.

Parameter	Set 60	Set 61	Set 62	Set 63	Set 64	Set 65	Set 66	Set 67	Set 68	Set 69
CER (%)	8.58	7.29	6.54	8.46	19.33	7.19	7.49	10.46	14.00	11.59
WER (%)	23.75	16.67	15.42	22.50	40.42	18.75	18.75	22.50	34.58	27.50

). These results suggest that minimal corrections are required to align the predictions with the ground-truth transcriptions. The low CER indicates that the system effectively distinguishes between different visemes at the character level. Meanwhile, the relatively higher WER indicates that the system can still accurately predict some words, but its performance decreases slightly when predicting words in a phrase or sentence.

Across the 2398 individual word pair predictions from 800 test videos, the system achieved an overall word accuracy of 76.27%. However, a substantial discrepancy was observed between accuracy and other word-level performance metrics, with a precision of 19.28%, a recall of 15.02%, and an F1-score of 16.68%. This discrepancy suggests the presence of class imbalance, wherein words occurring with higher frequency are more accurately predicted, while those with lower frequency reduce overall system performance. The imbalance is reflected in the system’s CER and WER (Table 2), as infrequent words contributed disproportionately to errors.

A limitation was also observed in prior lip-reading studies [8,15,16,17]. Misclassifications were particularly evident in word pairs with overlapping visemes, such as linggo–tao, tatlo–relo, and kailan–saan. Manual transcription confirmed these similarities, reinforcing that viseme overlap remains a fundamental challenge for lip-reading systems.

Despite these limitations, the system demonstrates the potential of the proposed Tagalog lip-reading framework. Its performance is consistent with results reported in comparable studies across other languages under similar constraints, where CER ranged from 30 to 45%, WER from 56 to 73%, and accuracy from 58 to 77% [6,15,16,17,18,19].

Training required four hours per epoch due to the large number of video frames and the computational complexity of the 3D CNN–Bi-LSTM model. The final three epochs were not recorded because of memory limitations. However, checkpointing preserved the latest weights and prevented data loss. The last recorded metrics were a training loss of 3.2946, validation loss of 0.7823, and a learning rate of 1.0 × 10⁻⁴. Although training ended prematurely, the model continued to converge effectively.

4. Conclusions and Recommendations

The presented Taglog lip-reading system achieved an overall accuracy of 76.27% with a CER and WER of 10.09 and 24.08%, respectively. The results provide a baseline for Tagalog lip-reading systems, as they are comparable to other studies in the same field. The system effectively captures fine-grained viseme patterns at the character level and maintains reasonable accuracy at the word level. Considering the limitations identified in the study, the dataset needs to be expanded by adding more words and speakers, and alternative methods need to be explored for building the model to address issues such as class imbalance, viseme similarity, long training times, and out-of-memory errors.