KamAI: A Basic Filipino Sign Language Recognition Mobile Application Using Deep Learning ^†

Abstract

KamAI is an Android-based mobile application designed to support the learning and recognition of basic Filipino Sign Language (FSL), the official sign language of the Deaf community in the Philippines. The system integrates Convolutional Neural Networks (CNNs) and Google MediaPipe to enable real-time gesture recognition for FSL letters, numbers, and common words. To evaluate the application’s overall quality and practical usability, User Acceptance Testing (UAT) was conducted using the ISO/IEC 25010 software quality framework. Six quality attributes, Functionality, Reliability, Compatibility, Usability, Efficiency, and Portability, were assessed using a five-point Likert scale. Results showed that all evaluated attributes received “Highly Acceptable” ratings, with mean scores ranging from 4.50 to 4.77. Functionality (4.73) and Portability (4.77) received the highest ratings, indicating accurate recognition performance and effective operation across various Android devices. These findings demonstrate strong user approval and confirm KamAI’s readiness for real-world deployment as a mobile assistive learning tool that promotes inclusive education and digital accessibility in the Philippine context.

1. Introduction

Members of the Deaf community continue to encounter significant communication barriers that limit their participation in education, employment, and everyday social interactions. These challenges are often linked to limited public understanding of sign language and the lack of accessible tools that support real-time communication [1]. In the Philippines, approximately 1.78 million Deaf individuals were recorded in the 2000 national census [2], highlighting the need for inclusive communication initiatives. The passage of Republic Act 11106 formally recognized Filipino Sign Language (FSL) as the national sign language of the Filipino Deaf community, affirming its importance not only as a communication system but also as a marker of cultural and linguistic identity.

Recent developments in artificial intelligence have contributed to improvements in sign language recognition systems. Prior studies have demonstrated encouraging results in recognizing Filipino Sign Language gestures using machine learning approaches [3,4,5,6]. While earlier systems have shown effectiveness in identifying hand gestures, challenges remain in ensuring consistent performance across different environments, lighting conditions, and user variations. In addition, maintaining reliability in mobile settings continues to be a concern.

The availability of lightweight hand-tracking technologies has enabled real-time gesture recognition on mobile devices [7]. When combined with learning-based models, these technologies allow applications to function directly on smartphones without requiring external hardware [8]. Mobile platforms are particularly suitable for assistive applications because of their portability and widespread use [9]. Previous mobile-based communication tools have demonstrated positive impacts on accessibility for Deaf users [10,11].

Despite these advancements, gesture recognition systems continue to face limitations related to variations in hand movement, complex backgrounds, and environmental inconsistencies [12]. Enhancing system reliability across diverse contexts requires continued refinement of learning models and careful attention to user-centered design principles [13,14,15]. Ensuring that assistive technologies are both functional and accessible is essential for meaningful adoption within the Deaf community.

In response to these needs, this study introduces KamAI Version 1.0, an Android-based mobile application designed to recognize basic Filipino Sign Language gestures in real time. The name KamAI combines the Filipino word kamay (hand) with Artificial Intelligence (AI), symbolizing the integration of accessibility and intelligent computing. By providing a mobile-first platform for recognizing FSL letters, numbers, and commonly used words, KamAI aims to support inclusive learning and improve communication accessibility in the Philippine context.

2. Materials and Method

This section describes the procedures undertaken to design, develop, and evaluate KamAI, an Android-based mobile application for real-time recognition of basic Filipino Sign Language (FSL). The study focused on integrating MediaPipe-based hand landmark detection with CNN-based gesture classification and refining the dataset, training configuration, and deployment pipeline to achieve reliable recognition for letters, numbers, and common words. The dataset sources, preprocessing steps, model training configurations, mobile implementation, and evaluation metrics are detailed in the following subsections to ensure reproducibility and clarity of the development process.

2.1. Dataset Collection and Composition

A dataset of over 29,000 images was compiled from publicly available repositories [16,17] and locally captured images collected with the assistance of FSL educators and Deaf participants. The dataset was organized into three categories aligned with KamAI’s recognition modes: FSL Letters (A–Z): 11,700 images, FSL Numbers: 16,500 images and FSL Words/Phrases: ~2000 images.

The dataset included variations in hand size, orientation, background complexity, and lighting conditions to enhance real-world generalization. Public datasets were curated to ensure alignment with Filipino Sign Language conventions, while locally collected images improved contextual and cultural accuracy. Data diversity was intentionally considered to reduce bias and improve robustness across different users and environmental conditions.

2.2. Image Preprocessing and Data Augmentation

Prior to model training, all collected images underwent preprocessing procedures, including image resizing, pixel value normalization, and label verification to ensure data consistency and quality. To enhance model robustness and improve generalization performance, several data augmentation techniques were applied. These included random rotation, horizontal flipping, brightness adjustment, and scaling.

These augmentation methods simulate real-world variations in hand sign appearance and help reduce overfitting during model training [18]. Figure 1 presents representative samples from the Filipino Sign Language (FSL) Letters dataset, Figure 2 presents representative samples from the FSL Numbers dataset, and Figure 3 presents representative samples from the FSL Words dataset. Together, these figures illustrate the diversity of the datasets in terms of hand orientation, finger positioning, lighting conditions, and background complexity, enabling the model to learn robust features for accurate real-world Filipino Sign Language recognition.

2.3. Hand Landmark Detection Pipeline

MediaPipe was used to extract 21 hand landmarks from the camera feed in real time [19]. These landmarks represent finger joints and wrist coordinates, which were transformed into structured numerical input features for the CNN classifier.

The landmark-based approach reduces dependency on raw image pixels and improves consistency across different backgrounds and lighting conditions.

2.4. CNN Model Architecture and Training Configuration

The researcher developed and trained the Convolutional Neural Network (CNN) models using TensorFlow 2.18.0 within a Jupyter Notebook 7.3 environment. The model architecture consisted of multiple convolutional and pooling layers, followed by fully connected dense layers and a final Softmax output layer for classification.

Cross-validation techniques were applied throughout the training process to prevent overfitting and improve the model’s ability to generalize to new data. Model performance was monitored using evaluation metrics including accuracy, precision, recall, and F1-score to optimize classification performance across gesture categories.

Once training was completed, the trained models were converted into TensorFlow Lite (.tflite) format to enable deployment within the Android-based KamAI mobile application.

2.5. Model Evaluation Metrics

Model performance was evaluated using a confusion matrix and standard multi-class classification metrics. The confusion matrix was used to analyze class-level prediction behavior and identify potential misclassification patterns across gesture categories.

To provide a comprehensive assessment of classification performance, the following metrics were computed: accuracy, precision, recall, and F1-score. Accuracy measured the overall proportion of correctly classified instances. Precision assessed the model’s ability to correctly predict positive classes, while recall evaluated its sensitivity in identifying actual gesture instances. The F1-score was calculated as the harmonic mean of precision and recall to provide a balanced measure of model performance, particularly in cases of class imbalance [20,21].

These metrics were computed separately for the letter, number, and word recognition models to ensure detailed performance analysis across gesture categories.

2.6. Acceptance Testing (UAT)

User Acceptance Testing (UAT) was conducted to evaluate the practical usability and software quality of the KamAI mobile application. The evaluation was guided by the ISO/IEC 25010 software quality model, an international standard for software product quality evaluation (ISO/IEC 25010, 2011). The standard provides a structured framework for assessing software systems across multiple quality characteristics.

Six quality attributes were evaluated in this study: functionality, reliability, compatibility, usability, efficiency, and portability. Participants assessed each attribute using a five-point Likert scale ranging from strongly disagree to strongly agree.

The UAT process aimed to determine whether the system met user expectations in terms of performance, ease of use, responsiveness, and cross-device compatibility. The collected responses were aggregated and analyzed to determine the overall acceptability of the application.

3. Results and Discussion

This section presents the results of the development, implementation, and evaluation of the KamAI mobile application. The findings are structured into three main components: (1) system implementation and functional features, (2) quantitative performance evaluation of the CNN-based recognition models, and (3) user acceptance evaluation based on ISO/IEC 25010 software quality standards. The results are analyzed to determine the system’s effectiveness in recognizing Filipino Sign Language (FSL) hand signs in real time and its practical usability as a mobile-based assistive learning tool.

3.1. CNN Model Architecture and Training Configuration System Implementation and Functional Features

KamAI was successfully developed using Android Studio, TensorFlow Lite, and Google’s MediaPipe framework. The application integrates real-time hand landmark detection with CNN-based gesture classification, enabling on-device recognition without requiring cloud processing. The system supports three primary recognition modes—letters, numbers, and words—allowing users to progressively learn Filipino Sign Language (FSL) gestures through structured interaction.

The user interface was designed to be intuitive and accessible for both Deaf and hearing users. Upon launching the application, a splash screen initializes system resources before transitioning to the home screen, where users are presented with a live camera feed for immediate gesture interaction. Recognized gestures are displayed in real time as corresponding text outputs, and users can easily switch between recognition modes through clearly labeled navigation controls. An additional about section provides categorized tutorial playlists to supplement learning. The overall application interface, including the splash screen, home screen, and mode selection interface, is shown in Figure 4.

The real-time performance of the developed system is illustrated through representative recognition outputs. Figure 5 presents sample recognition results for the FSL alphabet (Letters A–E), demonstrating the application’s capability to recognize alphabetic hand gestures in real time. Figure 6 shows representative recognition outputs for FSL numbers (6–10), highlighting the model’s ability to distinguish numerical hand signs with varying finger configurations. Figure 7 presents sample recognition outputs for commonly used FSL words, including Lolo, Kain, Ilan, and Lola, demonstrating the model’s capability to recognize conversational gestures under different environmental conditions.

3.2. CNN Model Performance Evaluation

The performance of the developed CNN-based recognition models was evaluated using standard multi-class classification metrics, including accuracy, precision, recall, and F1-score. A confusion matrix was also generated for each dataset to analyze class-level prediction behavior and identify misclassification patterns. These metrics provide a comprehensive assessment of the model’s ability to correctly classify Filipino Sign Language (FSL) hand signs across letters, numbers, and words. The confusion matrices for the FSL letter, number, and word recognition models are presented in Figure 8, Figure 9, and Figure 10, respectively.

The FSL letter recognition model achieved an accuracy of 88.38%, with a precision of 97%, recall of 87%, and F1-score of 90%. The high precision value indicates that when the model predicts a letter, it is highly likely to be correct. The relatively balanced recall and F1-score further confirm stable multi-class classification performance across the alphabet.

Despite strong overall performance, confusion matrix analysis revealed minor misclassifications between visually similar hand signs, particularly between “M” and “N.” These gestures share closely related finger positioning patterns, which can challenge CNN-based static image classifiers.

The FSL number recognition model achieved the highest overall accuracy at 91.41%, indicating strong classification performance across numerical gestures. However, while precision remained relatively high (89%), recall was significantly lower (21%), resulting in a reduced F1-score of 28%.

The lower recall suggests that although the model makes correct predictions when confident, it may fail to detect certain number gestures consistently. Confusion matrix analysis revealed misclassification patterns primarily among gestures with subtle finger differences, particularly between numbers such as “6,” “7,” and “8.”

The FSL word recognition model achieved an accuracy of 83.08%, with precision at 96%, recall at 88%, and F1-score at 91%. These results demonstrate the model’s capability to classify more complex gestures representing conversational words and phrases.

Unlike static alphabet gestures, word-level signs often involve more varied hand shapes and contextual positioning. The strong F1-score indicates balanced performance between precision and recall, confirming the robustness of the classification model in handling multi-class word recognition tasks.

3.3. Acceptance Testing (UAT) Results

To evaluate the practical usability and software quality of the KamAI mobile application, User Acceptance Testing (UAT) was conducted using the ISO/IEC 25010 software quality model. The evaluation covered six quality characteristics: functionality, reliability, compatibility, usability, efficiency, and portability. Participants assessed each criterion using a five-point Likert scale ranging from 1 (Highly Unacceptable) to 5 (Highly Acceptable).

The summary of the User Acceptance Testing results is presented in

Table 1. Summary of ISO/IEC 25010-based User Acceptance Testing results, showing the mean scores and descriptive interpretation of the evaluated software quality attributes.

Quality Attribute	Mean Score	Interpretation
Functionality	4.73	Highly Acceptable
Reliability	4.51	Highly Acceptable
Compatibility	4.71	Highly Acceptable
Usability	4.50	Highly Acceptable
Efficiency	4.59	Highly Acceptable
Portability	4.77	Highly Acceptable

, which shows the mean scores and corresponding descriptive interpretations for each evaluated software quality attribute based on the ISO/IEC 25010 framework.

All quality attributes achieved mean scores within the “Highly Acceptable” range (4.21–5.00), indicating strong overall user approval. Functionality (4.73), compatibility (4.71), and portability (4.77) received the highest ratings, confirming accurate gesture recognition and reliable operation across various Android devices. Reliability (4.51), efficiency (4.59), and usability (4.50) further demonstrate stable performance, resource efficiency, and an intuitive interface design. These results affirm that KamAI meets ISO/IEC 25010 quality standards and is suitable for practical mobile deployment.

4. Conclusions

The study demonstrates that integrating Convolutional Neural Networks (CNNs) with MediaPipe-based hand landmark detection significantly enhances real-time Filipino Sign Language (FSL) recognition on mobile devices. Rather than relying on cloud-based processing, the system focused on efficient on-device inference through TensorFlow Lite, ensuring accessibility and responsiveness in real-world conditions. While the recognition models performed strongly across letters, numbers, and words, certain visually similar gestures, particularly in numerical and alphabet categories, posed classification challenges due to subtle finger configuration differences. Despite these limitations, the developed approach achieved high precision and balanced F1-scores in letter and word recognition, confirming the effectiveness of CNN-based classification for static FSL gestures. The integration of structured recognition modes and real-time feedback further strengthened the application’s educational utility. User Acceptance Testing based on ISO/IEC 25010 standards indicated that the system met high levels of functionality, reliability, compatibility, usability, efficiency, and portability, reinforcing its readiness for practical deployment. These results highlight the value of lightweight deep learning architectures in mobile-based assistive technologies and demonstrate the system’s potential to support inclusive education and communication accessibility. The findings establish KamAI as a viable framework for further advancement in mobile sign language recognition systems.