Development of Transactional Filipino Sign Language Recognition System Using MediaPipe and Gated Recurrent Units ^†

Abstract

Persistent communication barriers for the deaf and hard-of-hearing community in the Philippines are addressed in this study by developing a Filipino Sign Language Recognition (SLR) system. The system focuses on transactional signs commonly used in commercial environments such as markets and public facilities, thereby filling a gap left by existing SLR models. A vision-based approach was adopted, employing MediaPipe for landmark detection and Gated Recurrent Units for translating signs into text. To train the model, a custom dataset comprising 1065 video samples of 26 transactional signs was created, accounting for subtle variations in individual signing styles. The complete system was implemented on a Raspberry Pi 5 equipped with a webcam and touchscreen display. When evaluated on unseen data, the system achieved a recognition accuracy of 87%, demonstrating its potential for real-world applications in supporting commercial interactions for deaf and hard-of-hearing individuals.

1. Introduction

The Filipino Sign Language (FSL) was declared the national sign language of the Philippines through Republic Act No. 11106 [1]. Despite this recognition since 2018, communication barriers for the deaf and hard-of-hearing communities persist in official and commercial transactions [2]. In response, sign language recognition (SLR) systems are emerging as a technology-based solution to address the communication barriers faced by deaf and hard-of-hearing communities [3,4,5,6]. Comprehensive reviews of machine learning applications in SLR highlight the shift toward deep learning to improve gesture classification [7,8]. Many of these vision-based algorithms have been investigated to determine which are best suited for SLR systems [9,10]. Gated Recurrent Units (GRUs) are often preferred over Long Short-Term Memory (LSTM) networks due to lower computational complexity [11,12,13]. Recent studies on FSL recognition have explored various architectures, including InceptionV3 and CNN-LSTM hybrids, to capture spatial-temporal features [14,15].

With an accuracy of 95% during evaluation and 98% using the test dataset, the MediaPipe and LSTM combination in [9] demonstrated significant potential in dynamic sign language recognition. Gated Recurrent Units (GRUs) are preferred over Long Short-Term Memory (LSTM) networks for sequential data processing due to their similar performance, lower computational complexity, and quicker training times [11].

Moreover, Gated Recurrent Units are a type of recurrent neural network (RNN) designed for sequential data processing and serve as a tool against the vanishing gradient problem that faces common traditional RNNs. The update gate in GRU controls the percentage of information from the previous hidden state that is transmitted to the current hidden state [12,13]. Moreover, the reset gate governs the extent to which the previous hidden state is disregarded. Activation of the reset gate enables the network to effectively reset its internal memory, thereby allowing it to focus more intensely on the current input.

In this study, we used the MediaPipe Holistic algorithm for feature extraction, particularly hand landmarks, which are essential for recognizing both static and dynamic FSL gestures [14,15]. The holistic framework provides a comprehensive feature set comprising 543 landmarks, yielding 1629 features when the X, Y, and Z coordinates of each landmark are considered. Such granularity is critical for robust and nuanced FSL recognition [14]. The holistic pipeline encompasses face, hand, and pose landmark detection, isolating key points that serve as inputs for machine learning models used in classification.

Based on the results, we developed a transactional FSL recognition system using MediaPipe and GRUs. The system extracts key landmarks and translates dynamic transactional gestures into text. The system was implemented on a Raspberry Pi 5. Similar edge-computing setups using NVIDIA Jetson or Raspberry Pi have been utilized for real-time object detection and checkout systems [16]. Furthermore, vision-based recognition techniques have been successfully applied to related linguistic tasks, such as PIR-based gesture control [17], lip-reading for regional languages [18], and the translation of air-written Baybayin [19]. In addition, the system includes a webcam interface that uses MediaPipe for landmark detection and GRUs for sign identification. The system performance was evaluated using a multi-class confusion matrix and overall accuracy metrics. The system contributes to automated FSL recognition by expanding the dataset to include common transactional signs while accounting for natural variations in signing styles.

2. Materials and Methods

2.1. System Development

The system utilizes the input–process–output model. It takes the video feed of transactional signs as input (Figure 1). MediaPipe extracts the hand and pose landmark coordinates from the video frames, which are then fed as a time sequence to the GRU pipeline. The GRU processes the temporal dependencies to recognize the sign, with the final output being the text translation of the recognized gesture. The system was constructed using a Raspberry Pi 5 (Cytron Technologies, Bukit Mertajam, Malaysia), with a 1080p web camera connected through the Camera Serial Interface (CSI) port to supply the video input. The output is displayed on a 7-inch touchscreen monitor. Other components include a micro-Secure Digital card, a USB-C power supply, and a CSI cable. All components are housed in a 3D-printed case, with the Raspberry Pi 5 attached to the back of the monitor (Figure 2).

The developed FSL recognition model operates through the workflow illustrated in Figure 3. At system initialization, the webcam is activated to capture the video feed. The process relies on two primary modules: MediaPipe Landmarking and GRU Sign Identification. The MediaPipe Landmarking module extracts the spatial coordinates of landmarks from each video frame, providing detailed positional data. Once these landmarks are obtained, the GRU Sign Identification module analyzes the temporal progression of the gestures by tracking landmark movements across frames. This structured pipeline ensures that video frames are continuously captured and processed in real time, enabling accurate recognition of FSL gestures. The system then outputs the corresponding text translation of the recognized sign, with an average latency of approximately five seconds.

The MediaPipe processing module slices the three-second video feed into 75 frames. For each frame, a color format conversion is performed before MediaPipe executes advanced landmark detection. This entails locating and obtaining the 3D landmark coordinates for the left and right hands. To maintain a constant data structure, zero-padding is applied if a hand is not detected in any frame. The extracted hand landmark data from both hands are concatenated into a single array, producing a feature representation for that instant (Figure 4).

The processed landmarks data from MediaPipe are then directly fed into the GRU Model Inference Module, as shown in Figure 5. A reshaping operation aligns the data format with the GRU model’s expected input structure, which takes the stream of individual frame snapshots and arranges them to resemble the original video clip. By joining the frames, the GRU model analyzes the entire sequence to understand the temporal progression of the sign and identifies the specific gesture by choosing the sign with the highest assigned probability.

2.2. Experimental Setup

Figure 6a,b show how the prototype is used by prompting the signer via the touchscreen monitor to perform a gesture. The camera must visibly capture the user’s upper body and arm reach during the three-second recording window. The system then displays the text translation of the recognized sign below the video feed, along with the confidence score.

2.3. Data Gathering

We created a custom dataset for everyday transactional interactions in Filipino Sign Language. Twenty participants (aged 19 to 60) performed the signs, including two variations per sign to account for natural signing nuances. The final dataset consists of 1065 videos, split into an 80–20 ratio for model training and validation/testing, respectively.

3. Results

A multi-class confusion matrix is utilized to tabulate the number of correctly identified signs (TP), incorrectly identified signs (FP), correctly identified non-signs (TN), and incorrectly identified non-signs (FN). The accuracy of the model is calculated using Equation (1).

$$Accuracy={\displaystyle \frac{{\sum }_{n=1}^{26}{A}_{nn}}{{\sum }_{\begin{array}{c}i=1\\ j=1\end{array}}^{26}{A}_{ij}}}$$

(1)

The numerator takes the summation of the diagonal elements of the confusion matrix, each cell representing the count of samples correctly predicted for each sign (Figure 7). On the other hand, the denominator sums up all entries in the matrix, representing the total number of predictions made by the system.

The detailed test results, including the predicted outputs, actual outputs, and corresponding cases, are summarized in

Table 1. The system output.

Trial No.	Actual Output	Predicted Output	Results
1	Again	Again	Classified
2	Again	Again	Classified
3	Again	Again	Classified
4	Card	Card	Classified
5	Card	Card	Classified
…	…	…	…
78	Ten	Yes	Misclassified

. Out of seventy-eight trials, with three recorded clips per gesture, the system produced ten misclassifications. The resulting multi-class confusion matrix, shown in Figure 7, highlights key performance trends of the model.

Performance evaluation was conducted using the confusion matrix in conjunction with Equation (1), while Equation (2) was applied to compute overall accuracy by summing correct classifications for each gesture and dividing by the total number of trials. Based on this analysis, the model achieved an overall recognition accuracy of 87.18% in identifying transactional Filipino Sign Language gestures, demonstrating its effectiveness for practical applications.

$$Accuracy={\displaystyle \frac{0+0+1+1+2+2+2+3+\cdots +3}{78}}={\displaystyle \frac{68}{78}}$$

(2)

4. Discussion

Extended and distinct gesture sequences, such as the signs for Card and Coin, were consistently recognized by the prototype, highlighting its strength in identifying macro-level temporal distinctions in FSL. In contrast, the ten misclassified trials primarily involved numerical signs (Eight, Seven, Three, and Ten), with single instances of misclassification for Wait and Yes. These errors are attributed to the limited distinctiveness of landmark placement and temporal sequencing in numerical gestures compared to longer, more dynamic signs. This challenge is particularly evident in signs like Eight and Seven, which share nearly identical sequences and placements, differing only by one finger.

5. Conclusions

We developed an FSLR system capable of translating dynamic gestures from recorded clips into text. We implemented a system that recognizes 26 transactional signs. The system’s applicability was validated in public and commercial settings. The model achieved an overall recognition accuracy of 87.18%. While previous studies focused on basic signs or hand gestures under varied backgrounds [20], our work expands this to transactional contexts. Other recent efforts in facial and gesture recognition have achieved high precision by utilizing Keras-based models [21] and real-time inference pipelines [22]. The consistent and precise classification of extended gestures such as Card and Coin underscores the system’s potential for capturing macro-level temporal distinctions in FSL. However, the misclassification of numerical signs, particularly Seven and Eight, suggests that MediaPipe landmark extraction combined with GRU processing struggles to capture fine-grained differences in finger placement. Despite this limitation, a foundational transactional-focused set of signs for FSL recognition is established by addressing real-world commercial interactions—contrasting with prior studies that concentrated mainly on basic signs.

To improve recognition accuracy for similar gestures, feature extraction needs to be enhanced through targeted data augmentation and the expansion of the dataset with additional video samples. A diverse collection of the misclassified gestures is essential for training the model on subtle landmark differences and improving robustness in distinguishing closely related signs.