Innovative Filipino Sign Language Translation and Interpretation with MediaPipe ^†

Abstract

Filipino Sign Language (FSL) serves as a vital means of communication for the Deaf and hard-of-hearing in the Philippines. However, its societal use remains limited due to the scarcity of qualified interpreters and the general lack of FSL literacy among the population. Therefore, this study aims to address the gap between FSL development and automated FSL translation by employing machine learning and computer vision techniques. A model was trained using the FSL-105 dataset, which comprises video clips of gestures related to greetings and colors, and utilized MediaPipe for real-time detection of hand, face, and body landmarks. Through iterative training with transfer learning, the model’s performance improved from an initial accuracy of 80% to a final accuracy of 98.75%. The results demonstrate that the MediaPipe-based model can reliably interpret FSL gestures, positioning it as a potentially accessible assistive tool for the Deaf and hard of hearing community. This technology holds promise for applications in education, healthcare, and public service, offering new opportunities to promote the social inclusion of Filipino Deaf communities through more inclusive communication.

1. Introduction

Filipino Sign Language (FSL) is the sign language in the Philippines adopted by the government, following the Filipino Sign Language Act of 2018 [1]. This legislation re-establishes FSL as the central means of communication for the Filipino Deaf and hard-of-hearing population, thus making it the most appropriate method. FSL is a visual-spatial language that empowers the Deaf and hard-of-hearing community to articulate abstract concepts, feelings, and, more importantly, identity creation within that society. These people use this language for social inclusion to protect their culture and history and promote their integration into society’s functions. FSL is utilized across various contexts, including schools, workplaces, government offices, and public services, to ensure that the Deaf and hard-of-hearing people receive and can provide equal information as other people.

Deaf Filipinos continue to face significant communication challenges, as qualified interpreters are not always readily available, and overall societal literacy in FSL remains low. 1.78 million Filipinos have a hearing impairment, with most relying on FSL as their primary medium of communication [2]. However, limited resources and the scarcity of sign language interpreters restrict their ability to fully participate in society. These constraints highlight the need for new approaches, such as machine learning and computer vision, to expand access to FSL translation.

Previous research on sign language translation using computer vision frameworks, including MediaPipe, has demonstrated the feasibility of detecting hand gestures, facial expressions, and body movements to approximate translation systems, improving communication between Deaf and hearing individuals [3]. When applied to FSL, such automated tools could help break communication barriers and foster greater independence for Deaf Filipinos in their daily lives.

This study aims to develop a model capable of translating FLS gestures using MediaPipe for real-time interpretation and communication assistance. We collected and organized a dataset of Filipino Sign Language gestures, including common greetings and colors represented through video clips. The video data were preprocessed by extracting hand, body, and facial landmarks using MediaPipe. We implemented MediaPipe for the automatic detection and extraction of hand and body landmarks from sign language gestures. A machine learning model was designed and trained to leverage the extracted landmarks to recognize Filipino Sign Language gestures and translate them into corresponding text.

2. Literature Review

MediaPipe is an open-source framework developed by Google to support the creation of real-time systems based on computer vision and machine learning. It employs a flexible, componentized architecture in which applications are linked through interconnected computational graphs composed of “calculators” or estimators. This graph-based structure enables efficient data transfer between components, facilitating high-throughput execution of resource-intensive algorithms such as hand tracking, face detection, and pose estimation across diverse hardware platforms, including mobile and edge devices [4].

An advantage of MediaPipe lies in its scalability and low-latency performance. By decomposing processing tasks into modular components, the framework can integrate seamlessly with programmed elements and devices, ensuring reliable real-time operation even on resource-constrained platforms. This flexibility makes MediaPipe particularly suitable for applications requiring synchronous, low-latency processing, such as augmented reality, gesture recognition, and remote health monitoring [5,6,7].

For sign language recognition, MediaPipe’s 3D hand tracking capability is essential. The framework maps 21 key hand landmarks in three-dimensional space, enabling accurate interpretation of gesture positions, including depth information. Such precision is critical for distinguishing between similar signs, where hand shape, motion, and spatial orientation are defining parameters. Studies combining MediaPipe with convolutional neural networks (CNNs) have reported success rates exceeding 99.95% in recognizing American Sign Language (ASL) gestures, underscoring its potential for real-world applications in sign language translation and assistive communication technologies for the Deaf and hard-of-hearing community [8,9,10].

Beyond sign language translation, MediaPipe’s 3D tracking capabilities extend to broader accessibility initiatives. For example, real-time communication tools can leverage webcam input to capture hand gestures, apply MediaPipe to extract landmarks, and classify gestures using support vector machines (SVMs) or other machine learning models. Such systems enable gesture-to-text conversion, opening pathways to freely accessible technologies that enhance interaction with communication devices [11]. Through its multi-device compatibility and computational efficiency, MediaPipe provides a robust foundation for accessibility programs, advancing gesture detection and sign language translation to promote inclusive communication.

3. Methodology

Figure 1 illustrates the FSL translation system. The system employs the FSL dataset in video format as the foundation for model training and development. The dataset is first preprocessed using OpenCV to standardize video formatting, enhance image quality, and prepare the data for further analysis. The model, built with MediaPipe, focuses on the core task of recognizing and interpreting FSL gestures. Once gestures are detected, the system translates them into text and presents the output in a user-friendly format. This direct sign-to-text translation enhances accessibility and supports communication for FSL users.

Figure 1. Workflow of FSL translation system.

3.1. Data Collection

The dataset used in this study was sourced from FSL-105: A Dataset for Recognizing 105 Filipino Sign Language Videos, available on Mendeley Data. It contains video samples of 105 FSL gestures, with emphasis on basic signs such as greetings and colors, which align with the study’s objectives. While the dataset provides a structured format and clear gesture recordings for training, preprocessing was required to ensure uniform video size and frame rate within the MediaPipe framework for accurate recognition and interpretation of FSL signs.

3.2. Data Preprocessing

Figure 2 presents the preprocessing flowchart. Two categories were selected from the FSL dataset: Greetings and Colors, each comprising 10 labels of video clips. Preprocessing followed a detailed pipeline in which each frame was examined and processed individually. This involved segmenting videos into frame sequences and carefully annotating hand shapes, orientations, movements, and facial expressions to ensure accurate representation of FSL gestures.

Figure 2. Data preprocessing workflow.

3.3. Model Training and Development

Figure 3 illustrates the extraction of hand and facial keypoints using OpenCV 4.0.1. These key points were used to generate relevant features before splitting the dataset into training and testing sets. The two-category dataset contained 407 video clips across 20 labels, with each label consisting of 19–21 clips. The dataset was divided into the standard 80% training and 20% testing split. Gesture labels were mapped to numerical values, and the model was implemented using TensorFlow’s Keras library. A sequential architecture incorporating the Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) layers was employed to capture the temporal dynamics of FSL gestures. Dense and dropout layers were added to improve generalization and robustness.

Figure 3. Hand and face keypoints.

Figure 4 depicts the MediaPipe model and prediction flowchart. During training, preprocessed frames were passed through OpenCV and MediaPipe to identify keypoints. The trained model then processed these features to recognize and translate Filipino Sign Language gestures, producing text outputs that represent the corresponding signs.

Figure 4. MediaPipe model and FSL prediction flowchart.

4. Results and Discussion

The model employed an iterative transfer learning approach, with each version built directly upon the preceding one. Initially, three models were trained, beginning with a baseline accuracy of 80% after 2000 epochs. The weights and learned features from this model were transferred to a second model, which was further trained for 1000 epochs using refined parameters and additional data, yielding an improved precision of 93%. These results were subsequently integrated into the final prediction model, which underwent an additional 300 epochs of training and achieved a prediction accuracy of 98.75%.

4.1. Model Accuracy and Loss

Figure 5a illustrates the model accuracy over 300 epochs, showing convergence to a stable value of approximately 98%, indicating effective learning without overfitting. Figure 5b presents the training loss, which began at a relatively high error rate but decreased rapidly as training progressed, demonstrating that the model successfully adjusted its parameters and achieved a minimized and stable error rate.

Figure 5. System model: (a) Model accuracy graph; (b) Model loss graph.

4.2. Confusion Matrix

Figure 6 depicts the confusion matrix summarizing the model’s performance in classifying FSL gestures across two categories. Most gestures were correctly classified, reflecting high predictive accuracy, with only minimal misclassifications observed. The diagonal concentration of predictions indicates strong recognition of phrases such as walang anuman, kumusta, pula, and kahel. Occasional errors occurred, such as magandang hapon being misclassified as magandang gabi, likely due to similarities in hand landmark gestures. The color intensity of the matrix further confirms the model’s strong predictive performance, with only minor areas of confusion. The model achieved a precision score of 99.17% and an F1-score of 98.79%, underscoring its excellent performance. The high precision indicates that false positives were rare, while the strong F1-score reflects a balanced trade-off between precision and recall, confirming both accuracy and consistency in predictions.

Figure 6. Confusion Matrix.

4.3. Result and Visualization

Figure 7 illustrates the visualization of the trained model. As users perform gestures, a color bar appears to indicate the recognized FSL gesture, which is then translated directly into text. Correctly performed gestures are displayed in the blue navigation bar, providing immediate feedback and enhancing usability. This real-time visualization demonstrates the model’s capability to deliver prompt and interpretable outputs, allowing users to verify the correctness of recognized gestures instantly.

Figure 7. Translation model visualization.

The clear mapping between gesture input and textual output supports user confidence and facilitates intuitive interaction with the system. Furthermore, the interface design contributes to accessibility by offering a straightforward and responsive feedback mechanism, which is essential for practical applications in communication assistance. Overall, the visualization confirms the model’s effectiveness not only in recognition accuracy but also in delivering user-friendly and meaningful outputs.

5. Conclusions and Recommendations

We developed and evaluated a robust model for FSL translation and interpretation. Using the FSL-105 dataset and MediaPipe for real-time keypoint detection, the model was trained iteratively, with each stage building upon the learned features of the previous one. Accuracy improved from 80% to 93% after refinement and ultimately reached 98.75% in the final iteration. MediaPipe’s holistic framework, capable of detecting face, hand, and body landmarks, was instrumental in capturing subtle gestures related to greetings and colors, enabling precise recognition. The results demonstrate that iterative training combined with transfer learning can produce a reliable FSL recognition model. This translation tool has significant potential to bridge communication barriers and enhance accessibility for the Deaf and hard-of-hearing community in the Philippines.

The dataset needs to be expanded to include additional gesture categories, thereby increasing the model’s versatility for real-world applications. Introducing a short delay before input capture may further improve recognition accuracy, while the development of a user-friendly interface would enhance accessibility for both Deaf and hearing users. Nonetheless, limitations remain, as external factors such as lighting conditions, gesture speed, hand placement accuracy, and video quality may affect performance.