Skip to Content
You are currently on the new version of our website. Access the old version .
Applied SciencesApplied Sciences
  • Article
  • Open Access

19 January 2026

13 Pages

Eye Gaze Detection Using a Hybrid Multimodal Deep Learning Model for Assistive Technology

,
,
,
and
Department of Applied Computer Science, The University of Douala, Douala P.O. Box 2701, Cameroon
*
Author to whom correspondence should be addressed.

Abstract

This paper presents a novel hybrid multimodal deep learning model for robust and real-time eye gaze estimation. Accurate gaze tracking is essential for advancing human–computer interaction (HCI) and assistive technologies, but existing methods often struggle with environmental variations, require extensive calibration, and are computationally intensive. Our proposed model, GazeNet-HM, addresses these limitations by synergistically fusing features from RGB, depth, and infrared (IR) imaging modalities. This multimodal approach allows the model to leverage complementary information: RGB provides rich texture, depth offers invariance to lighting and aids pose estimation, and IR ensures robust pupil detection. Furthermore, we introduce a personalized adaptation module that dynamically fine-tunes the model to individual users with minimal calibration data. To ensure practical deployment, we employ advanced model compression techniques, enabling real-time inference on resource-constrained embedded systems. Extensive evaluations on public datasets (MPIIGaze, EYEDIAP, Gaze360) and our collected M-Gaze dataset demonstrate that GazeNet-HM achieves state-of-the-art performance, reducing the mean angular error by up to 27.1% compared to leading unimodal methods. After model compression, the system achieves a real-time inference speed of 32 FPS on an embedded Jetson Xavier NX platform. Ablation studies confirm the contribution of each modality and component, highlighting the effectiveness of our holistic design.

1. Introduction

The evolution of Human–Computer Interaction (HCI) has consistently aimed to create more intuitive, efficient, and natural interfaces. While keyboards and mice remain dominant, they represent a bottleneck in this pursuit. This has catalyzed research into alternative input modalities, such as voice, gesture, and eye gaze. Among these, eye gaze is particularly compelling; it is a natural, fast, and often subconscious mechanism that can provide rich intent signals, making it a powerful tool for both assistive technologies (e.g., for individuals with motor disabilities) and mainstream applications (e.g., in augmented reality and driver monitoring).
Despite significant progress, accurate and robust eye gaze estimation in unconstrained, real-world environments remains a formidable challenge. Existing methods are often constrained by several key limitations:
  • Environmental Sensitivity: Appearance-based models, particularly those relying solely on RGB data, are highly susceptible to variations in illumination, head pose, and user distance from the sensor [1].
  • Calibration Burden: Many model-based approaches, especially those using geometric principles, require extensive and precise user-specific calibration, hindering usability and adoption [2].
  • Computational Intensity: State-of-the-art deep learning models achieve high accuracy but often at a computational cost that precludes real-time performance on resource-constrained devices like mobile phones or AR/VR headsets [3].
  • Lack of Generalization: Models trained on limited datasets often fail to generalize across diverse ethnicities, users with eyewear, or under dynamic lighting conditions [4].
In this paper, we propose a novel Hybrid Multimodal Deep Learning Model (GazeNet-HM) that directly and effectively addresses these challenges. Our work makes the following key contributions:
Novel Multimodal Architecture: We introduce a hybrid deep learning framework that synergistically fuses features from RGB, depth, and infrared (IR) imaging modalities. This fusion leverages the complementary strengths of each modality, a strategy underexplored in prior deep learning-based gaze estimation research. We describe our model as ‘hybrid’ because it combines: (1) model-based principles through the explicit use of geometric information from depth data and physiological features from IR data, with (2) appearance-based deep learning through the RGB stream. Furthermore, our architecture hybridizes different network designs (ResNet-18 for RGB/Depth and a custom CNN for IR) and fusion strategies, creating a comprehensive solution that leverages the strengths of multiple paradigms.
Enhanced Robustness and Personalization: Our model incorporates a self-adaptive user personalization layer that dynamically fine-tunes its parameters based on minimal user data, significantly reducing the calibration burden while improving accuracy and generalization. In this work, we define robustness as the model’s ability to maintain high accuracy despite variations in: (1) lighting conditions (from dim to backlit), (2) head pose (±30° yaw, ±15° pitch), and (3) user characteristics (including individuals wearing glasses). We quantitatively measure robustness through consistent performance across these challenging scenarios in our M-Gaze dataset and through cross-dataset evaluation.
Optimized for Real-Time Deployment: We employ state-of-the-art model compression techniques (including pruning and quantization) and efficient network design to achieve real-time inference speeds on embedded systems without a significant loss in performance, making our solution practical for consumer-grade devices.
Comprehensive Empirical Validation: We provide extensive ablation studies and benchmark our model against state-of-the-art methods on multiple public datasets (MPIIGaze [5], Gaze360 [6], EYEDIAP [7]), demonstrating superior performance in accuracy, robustness, and efficiency. We also introduce a new multimodal dataset, M-Gaze, to facilitate future research.
The remainder of this paper is structured as follows: Section 2 provides a focused review of related work. Section 3 details our proposed GazeNet-HM architecture. Section 4 describes the experimental setup. Section 5 presents and discusses the results. Finally, Section 6 concludes the paper and suggests future directions.

3. Proposed Methodology

3.1. System Overview and Data Acquisition

Our system utilizes a synchronized multimodal sensor array based on the Intel RealSense D455 depth camera. This device integrates an RGB sensor (global shutter, up to 1280 × 720 resolution at 30 FPS), an infrared stereo depth module, and an infrared projector. For our data acquisition, we used the synchronized RGB, depth, and IR (left imager) streams at a resolution of 640 × 480. The RGB images were processed in the sRGB color space. The data preprocessing pipeline includes temporal synchronization provided by the RealSense SDK, face detection using a modified Multi-Task Cascaded Convolutional Networks (MTCNN) [26], and region of interest (ROI) extraction for the eyes and face in all three modalities. The extracted patches were resized to 224 × 224 pixels. For the RGB stream, we applied normalization using ImageNet mean and standard deviation values. No additional gamma correction was applied. The depth and IR maps were normalized to a [0, 1] range based on the sensor’s maximum expected depth range and IR intensity, respectively. To improve generalization, data augmentation was applied, including random rotation (±10°), brightness jitter (for RGB, ±20% of the original value), and horizontal flipping.

3.2. GazeNet-HM Architecture

The proposed architecture is illustrated in Figure 1.
Figure 1. The GazeNet-HM architecture.
The model processes synchronized RGB, Depth, and IR patches through separate feature extraction backbones. Features are fused at multiple stages and processed by a personalized adaptive module before final gaze direction regression.
(Note: A high-resolution diagram would be inserted here, showing the three input streams, their respective CNN backbones, fusion layers, adaptive module, and regression output.)

3.2.1. Modality-Specific Feature Extractors

The first stage of GazeNet-HM consists of three separate convolutional neural networks (CNNs), each dedicated to processing a specific input modality. This design allows each backbone to learn specialized features optimized for its data type.
  • RGB Stream: The RGB input (a 3-channel eye/face crop) is processed by a lightweight ResNet-18 backbone [27], pre-trained on ImageNet. This stream is responsible for extracting rich texture and appearance-based features, such as eye shape, eyelid closure, and surrounding facial context. We reduce the initial convolutional layer kernel size to 3 × 3 and remove the final fully connected layer to output a 512-dimensional feature vector.
  • Depth Stream: The aligned depth map (1-channel) is processed by an identical, but separate, ResNet-18 architecture. The weights are initialized from the ImageNet-pre-trained model. During training, the depth stream backbone is fine-tuned independently from the RGB stream. This involves replacing the input layer to accept a single-channel depth map and then training the entire network on our gaze estimation task using the depth data. The Adam optimizer and the same learning rate schedule as the main model were used. This process allows the network to adapt its filters from recognizing natural image features to extracting geometric and structural features relevant from depth data, independent of the RGB modality. This stream learns features related to the geometric structure of the face and eye region, providing crucial invariance to absolute user distance and aiding in implicit head pose estimation.
  • IR Stream: The IR image (1-channel) is processed by a custom 6-layer CNN designed for high-contrast feature detection. This network, trained from scratch, consists of four convolutional layers with increasing filters (32, 64, 128, 256), each followed by Batch Normalization and ReLU activation, and two fully connected layers. Its purpose is to precisely localize the pupil center and corneal reflections (glints), providing robust, illumination-invariant cues for gaze direction.
  • We selected ResNet-18 as a backbone due to its proven effectiveness and strong balance between accuracy and computational cost. While lighter networks like MobileNet exist, our ablation studies found that ResNet-18 provided a more robust feature representation for the complex task of gaze estimation, leading to higher accuracy. Furthermore, our subsequent model compression pipeline (pruning and quantization) effectively reduces the ResNet-18 footprint to a level suitable for real-time embedded operation, making the choice of a slightly larger base architecture a worthwhile trade-off for performance.

3.2.2. Hierarchical Fusion Strategy

Instead of late fusion (averaging predictions), we employ an intermediate hierarchical fusion strategy to learn complex, non-linear correlations between modalities at a feature level.
  • Fusion Block 1 (Concatenation): The 512-D feature vectors from the RGB and Depth streams are concatenated with the 256-D feature vector from the IR stream, resulting in a 1280-D combined feature vector (512 + 512 + 256 = 1280). This preserves the unique information from each modality.
  • Fusion Block 2 (Feature Refinement): The F_combined vector is passed through a refinement module consisting of two 1 × 1 convolutional layers (with 1024 and 512 filters) interleaved with ReLU activation. The 1 × 1 convolutions act as cross-modal weighting filters, dynamically learning the importance of and relationships between features from different modalities. This outputs a refined 512-D feature vector (F_refined).

3.2.3. Personalized Adaptation Module

A key innovation of GazeNet-HM is a modular component that enables rapid personalization with minimal calibration data. This module adapts the model’s parameters based on a short, user-specific calibration sequence.
  • Calibration Procedure: For each new user, a short (~5 s) calibration sequence is recorded. During this sequence, the user is asked to follow a predefined set of 9 points (a 3 × 3 grid) displayed on the screen. The system records synchronized RGB, depth, and IR data while the user fixates on each point.
  • Input: The module takes two inputs:
    • The refined feature vector F_refined from the fusion network.
    • A user signature vector U Construction: This calibration sequence is processed by the main GazeNet-HM feature extractors and fusion network (Section 3.2.1 and Section 3.2.2), but with the final regression head removed. For each frame in the calibration sequence, a feature vector F_refined is extracted. These feature vectors, along with their corresponding known ground-truth gaze directions, are used as input to a small, separate encoder network. This encoder consists of two fully connected layers (128 → 64 units) with ReLU activation. It processes the sequence of features and gaze pairs to produce a compact, user-representative 64-dimensional user vector U. This vector U is designed to encode user-specific biases and characteristics, such as unique eye anatomy or systematic head pose tendencies.
  • Gating Mechanism: The adaptation is performed via a feature-wise modulation mechanism inspired by conditional normalization [28]. The user vector U is passed through two separate fully connected layers to generate scale (γ) and shift (β) parameters.
γ = Wγ × U + bγ
β = Wβ × U + bβ
with W β as “learnable weight matrices” and b γ and b β as “bias terms” in the gating mechanism.
These parameters are used to transform the refined feature vector:
F_personalized = γ ⊙ F_refined + β
where ⊙ denotes element-wise multiplication. This allows the model to adjust its feature responses to account for individual user characteristics like eye anatomy, glasses refraction, or typical head pose.

3.2.4. Output Regression Head

The personalized feature vector F_personalized is fed into the final regression head, which consists of two fully connected layers (512 → 128 units, then 128 → 2 units). The final layer outputs a 2-dimensional vector representing the gaze angles (yaw and pitch) in radians. The mean squared error (MSE) loss between the predicted and ground-truth angles is used to train the entire network end-to-end.

3.3. Model Optimization and Compression

To ensure real-time performance on embedded devices, we apply a structured pruning algorithm to each backbone network, removing filters with the lowest L1-norm. Subsequently, we employ post-training quantization to FP16 precision. This reduces the model size by over 60% and accelerates inference by 2.5× on our target hardware (NVIDIA Jetson Xavier NX) with negligible accuracy loss (<0.5°), as detailed in Section 5.3.
To provide comprehensive computational complexity analysis, we measured several key metrics for both the original and optimized models: Parameter Count (total trainable parameters in millions), Computational Cost (forward-pass FLOPs in GigaFLOPs), Model Size (memory footprint in MB), and Inference Speed (end-to-end processing time on Jetson Xavier NX).

4. Experimental Setup

4.1. Datasets and Metrics

We trained and evaluated GazeNet-HM on a combination of public datasets and our new dataset. It is important to note that the public datasets (MPIIGaze, EYEDIAP, Gaze360) primarily consist of RGB data. To evaluate the contribution of our full multimodal model (RGB + D + IR) and ensure a fair comparison with state-of-the-art methods, we conducted two types of evaluations:
  • Unimodal/Bimodal Benchmarking: For comparisons on public datasets (Table 1), we used only the RGB stream of GazeNet-HM against other RGB-only methods. This demonstrates the baseline performance of our architecture.
    Table 1. Comparison of mean angular error (MAE °) on the MPIIGaze and EYEDIAP datasets.
  • Full Multimodal Evaluation: The complete RGB + D + IR model and the ablation studies (Table 2) were exclusively trained and evaluated on our M-Gaze dataset, which provides all three synchronized modalities. This allows us to isolate and validate the performance gain from multimodal fusion.
    Table 2. Ablation study on the contribution of each modality and component on the M-Gaze dataset.
MPIIGaze [5]: A standard benchmark for within-dataset evaluation of appearance-based methods. Contains 213,659 images from 15 participants.
EYEDIAP [7]: Contains video recordings of 16 participants with varying head poses and using a screen target. Used for evaluating performance under motion.
Gaze360 [6]: A large-scale dataset with 172,000 images from 238 subjects under extreme head and gaze angles. Used for evaluating 3D gaze estimation robustness.
M-Gaze Dataset (Our Contribution): We introduce a new multimodal dataset to facilitate research in this area. Data was collected from 15 participants (10 male, 5 female, aged 22–35) using the Intel RealSense D455 sensor. Participants included individuals of African ethnicity and different body complexity, with 4 participants wearing prescription glasses. The collection environment featured 3 different lighting conditions: normal office lighting, dim light, and backlit. Participants were asked to perform varying head poses (frontal, ±30° yaw, ±15° pitch) while following a moving dot on the screen that covers a wide range of gaze angles. The dataset contains over 50,000 synchronized triplets of RGB, Depth, and IR frames, along with precise ground-truth gaze vectors. This dataset has been made publicly available to promote multimodal research and can be accessed at: https://github.com/everdzekov/GazeNet-HM (accessed on 26 October 2025). The repository contains the synchronized data, annotation files, and a detailed data sheet describing the collection protocol and demographics.
Primary Metric: Mean Angular Error (MAE) in degrees (°). The MAE is the average angular difference between the predicted gaze vector g p r e d and the ground truth gaze vector g g t , calculated as:
MAE = 1 N ∑ i = 1 N   a r c c o s ( g p r e d ( i ) ⋅ g g t ( i ) ∥ g p r e d ( i ) ∥   ∥ g g t ( i ) ∥ )
where N is the total number of samples and denotes the dot product.

4.2. Implementation Details

The model was implemented in PyTorch 1.13. We used weights pre-trained on ImageNet for the RGB and depth backbones. The IR stream was trained from scratch. We used the Adam optimizer with a learning rate of 1 × 10–4, decayed by a factor of 0.1 every 20 epochs. Training was performed on a single NVIDIA RTX 4090 GPU for 100 epochs with a batch size of 64. For the personalized module, a 5 s calibration sequence was used for each new user.

4.3. Training and Evaluation Protocol

For all experiments, we adhered to a strict subject-independent evaluation protocol. For the public datasets (MPIIGaze, EYEDIAP, Gaze360), we followed the standard leave-one-subject-out (LOSO) or the predefined train/test splits provided by the respective datasets. For our M-Gaze dataset, we performed a 5-fold cross-validation, where in each fold, data from 12 participants was used for training, 2 for validation, and 1 for testing. This ensures that the model is always evaluated on subjects whose data was never seen during training. The results reported are the average performance across all test folds or the official test sets.

5. Results and Discussion

5.1. Comparison with State-of-the-Art

We compare GazeNet-HM against leading methods, including foundational works and, most importantly, recent state-of-the-art models from 2022–2024. Results are shown in Table 1. For a fair comparison on public datasets, the RGB-only stream of our GazeNet-HM is compared against other RGB methods. The full multimodal model is evaluated on our M-Gaze dataset in the ablation study (Table 2).
Table 1 demonstrates that even the RGB-stream of our GazeNet-HM is highly competitive with the most recent state-of-the-art methods, outperforming GazeTR and DGaze on MPIIGaze and achieving comparable performance on EYEDIAP. This validates the strength of our underlying architecture and training methodology. The significant performance leap of the full multimodal model, as shown in the ablation study (Table 2), is therefore built upon a strong and modern baseline, making our claimed improvements over “leading unimodal methods” both accurate and relevant to the current research landscape.
Our model achieves state-of-the-art performance, reducing the error by 18.7% on MPIIGaze and 27.1% on EYEDIAP compared to the best unimodal methods. This conclusively demonstrates the advantage of our multimodal fusion approach, which effectively mitigates the weaknesses of individual modalities.

5.2. Ablation Studies

The ablation study was conducted on the M-Gaze dataset using the same 5-fold cross-validation protocol. We systematically removed or modified components from the full GazeNet-HM model to isolate their contributions:
  • RGB Only: Uses only the RGB stream to establish a baseline.
  • RGB + Depth: Adds depth modality to assess geometric information contribution.
  • RGB + IR: Adds IR modality to evaluate physiological feature benefits.
  • Full Model (RGB + D + IR): Our complete multimodal architecture.
  • Full Model + Personalization: Adds the adaptive module to measure personalization gains.
This structured approach clearly demonstrates how each modality and the personalization module incrementally improves performance.
Results are shown in Table 2.
The study validates the contribution of each component. Each additional modality reduces error, with IR providing the most significant boost due to its precise eye feature detection. The personalized adaptation module further reduces the error by 12.1%, highlighting its effectiveness in adapting to individual anthropomorphic differences.

5.3. Real-Time Performance Analysis

A critical goal was achieving real-time performance. We deployed our model on an NVIDIA Jetson Xavier NX, a common embedded platform for robotics and AR/VR. Results are shown in Table 3. The device was configured in 15 W mode and features a 384-core Volta GPU with 48 Tensor Cores and 8 GB of LPDDR4x memory.
Table 3. Performance and speed on an embedded platform (NVIDIA Jetson Xavier NX).
While the unoptimized version of GazeNet-HM is slower due to its higher complexity, our optimized model reduces inference time by 31% compared to the Full-Face baseline (31 ms vs. 45 ms) while simultaneously achieving a 29% lower error (3.4° vs. 4.8°) (Table 4). This demonstrates a superior accuracy-speed trade-off. Direct inference time comparisons for iTracker [29] and Gaze360 [6] on our embedded hardware are not available, as their original implementations and evaluations focused on desktop-grade GPUs and were not optimized for the Jetson platform. Our work specifically focuses on a deployable, efficient solution for resource-constrained devices.
Table 4. Computational complexity analysis of GazeNet-HM before and after optimization.
After compression and quantization, our model runs at 32 FPS with only a 0.1° increase in error, successfully meeting the requirement for real-time application in interactive systems.
The reported inference time of 31 ms (32 FPS) includes the complete processing pipeline: temporal synchronization, face detection using MTCNN, ROI extraction for all three modalities, and the forward pass of GazeNet-HM. This comprehensive measurement ensures that our real-time performance claims reflect practical deployment scenarios, not just isolated model inference.
To comprehensively address computational complexity, we present detailed metrics in Table 5. The original GazeNet-HM model contained 24.7 million parameters and required 3.8 GFLOPs per forward pass, resulting in a 98.5 MB model size. After applying structured pruning and FP16 quantization, we achieved a 60% reduction across all complexity metrics: parameters (9.9 M), FLOPs (1.5 GFLOPs), and model size (39.4 MB), with only a negligible 0.1° increase in MAE. This demonstrates that our optimization pipeline effectively reduces computational demands while maintaining high accuracy, making the model suitable for resource-constrained embedded systems.
Table 5. Comparison of mean angular error (MAE °) on the Gaze360 dataset.

5.4. Evaluation on Gaze360 Dataset

To evaluate performance under extreme gaze angles and large head poses, we benchmarked our model on the Gaze360 dataset. As this dataset lacks depth and IR modalities, we report the performance of our RGB stream. The results, shown in Table 5, demonstrate that our method maintains robust performance even in challenging scenarios, outperforming the original Gaze360 model.

6. Conclusions and Future Work

Limitations

While GazeNet-HM demonstrates strong performance, it is not without limitations. Firstly, the model relies on a synchronized multimodal sensor (Intel RealSense (Santa Clara, CA, USA)), which may not be available on all consumer devices (e.g., standard laptops or smartphones). Future work will explore domain adaptation techniques to allow a model trained on multimodal data to be applied using only an RGB camera. Secondly, although the personalized adaptation module reduces calibration, it does not eliminate it entirely. A zero-calibration system remains a challenging goal. Finally, the current model, while efficient, may still be too heavy for ultra-low-power microcontrollers, necessitating further research into neural architecture search for even more efficient backbone networks.
Finally, the current model, while efficient with 1.5 GFLOPs and 9.9 M parameters after optimization, may still be too heavy for ultra-low-power microcontrollers, necessitating further research into neural architecture search for even more efficient backbone networks.
In this paper, we presented GazeNet-HM, a novel hybrid multimodal deep learning model for robust and real-time eye gaze estimation. By innovatively fusing features from RGB, depth, and infrared modalities within a unified architecture, our model achieves superior accuracy and robustness against environmental variations. The incorporation of a personalized adaptation module significantly reduces the calibration burden and improves generalization. Furthermore, through careful model optimization, we demonstrated that high accuracy can be maintained even under strict computational constraints, enabling real-time deployment on embedded systems.
Extensive experiments on multiple benchmarks and detailed ablation studies confirm the effectiveness of each component of our proposed system. While this study provides a strong foundation, it also opens up several promising avenues for future research to further advance the field of practical gaze estimation:
1. Broader Dataset Analysis and Demographic Validation:
While we evaluated our model on several public datasets and our own M-Gaze dataset, future work will involve a more comprehensive analysis across a wider range of environmental conditions and, crucially, user demographics. We plan to curate a large-scale, public multimodal gaze dataset that explicitly includes diverse ethnicities, age groups, and users with visual impairments (e.g., wearing glasses or contact lenses). This will allow for a rigorous fairness audit of GazeNet-HM and similar models, ensuring that performance gains are equitable and not limited to a specific demographic subset. Analyzing performance across this spectrum will be essential for deploying truly inclusive and accessible gaze-controlled systems.
2. In-Depth User Studies in Real-World Scenarios:
To fully validate the model’s practicality, we intend to conduct extensive user studies with participants from diverse backgrounds. These studies will move beyond quantitative angular error metrics to evaluate qualitative aspects of the user experience, such as:
  • Learning Curve and Fatigue: How quickly do users adapt to a gaze-based interface powered by our model? Does the reduced calibration lessen cognitive load?
  • Effectiveness in Applications: Deploying the system in specific real-world tasks, such as controlling a smart home environment, typing on a virtual keyboard, or navigating a VR menu, to measure task completion time and error rate.
  • Long-Term Usability: Conducting longitudinal studies to assess the system’s stability and user satisfaction over extended periods of use.
3. Exploration of Additional Sensor Modalities:
The success of our multimodal fusion approach suggests that integrating additional, complementary sensors could push the boundaries of robustness even further. A key direction for future work is the investigation of thermal imaging as an additional modality. Thermal cameras are largely invariant to ambient lighting changes and can provide a clear physiological signature of the eye region based on body heat. Fusing thermal data with existing modalities could unlock unprecedented performance in extreme conditions, such as low-light scenarios, sudden illumination changes (e.g., car headlights at night), or even when the eye is partially occluded. We plan to explore novel fusion architectures capable of integrating this new thermal stream.
We believe this work not only pushes the boundaries of eye gaze recognition technology but also establishes a new benchmark for the creation of precise, adaptable, and ethically conscious gaze tracking systems. The conclusions and insights herein open up new avenues for future study and application, with potential impacts across various industries, from healthcare to human–computer interaction.

Author Contributions

Conceptualization, V.E.T.; Methodology, V.E.T. and W.P.B.; Software, V.E.T.; Investigation, V.E.T.; Writing—original draft, V.E.T. and W.P.B.; Writing—review & editing, V.E.T.; Visualization, V.E.T.; Supervision, N.W.A.V., M.N.J. and F.L.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Acknowledgments

The authors would like to thank the participants who volunteered their time for our user study. We are also grateful to the Department of Mathematics and Computer Science at the University of Douala for supporting this research. We acknowledge the use of open-source libraries OpenCV [31], PyTorch [32], and scikit-learn [33]. We also thank the reviewers for their insightful feedback.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bulling, A.; Ward, J.A.; Gellersen, H.; Tröster, G. Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 741–753. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  2. Hansen, D.W.; Ji, Q. In the Eye of the Beholder: A Survey of Models for Eyes and Gaze. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 32, 478–500. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Sugano, Y.; Matsushita, Y.; Sato, Y. Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 1821–1828. [Google Scholar]
  4. Wood, E.; Baltrusaitis, T.; Morency, L.P.; Robinson, P.; Bulling, A. Learning an appearance-based gaze estimator from one million synthesised images. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA), Charleston, SC, USA, 14–17 March 2016; pp. 131–138. [Google Scholar]
  5. Zhang, X.; Sugano, Y.; Fritz, M.; Bulling, A. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 4511–4520. [Google Scholar]
  6. Kellnhofer, P.; Recasens, A.; Stent, S.; Matusik, W.; Torralba, A. Gaze360: Physically Unconstrained Gaze Estimation in the Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6912–6921. [Google Scholar]
  7. Mora, K.A.F.; Odobez, J.-M. Geometric generative gaze estimation (G3E) for remote RGB-D cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 1773–1780. [Google Scholar]
  8. Young, L.; Sheena, D. Survey of eye movement recording methods. Behav. Res. Methods Instrum. 1975, 7, 397–429. [Google Scholar] [CrossRef] [Scilit]
  9. Duchowski, A.T. Eye Tracking Methodology: Theory and Practice, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2017. [Google Scholar]
  10. Park, S.; Zhang, X.; Bulling, A.; Hilliges, O. Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA), Warsaw, Poland, 14–17 June 2018; pp. 1–10. [Google Scholar]
  11. Smith, B.A.; Yin, Q.; Feiner, S.K.; Nayar, S.K. Gaze Locking: Passive Eye Contact Detection for Human-Object Interaction. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST), St. Andrews, UK, 8–11 October 2013; pp. 271–280. [Google Scholar]
  12. Hansen, D.W.; Pece, A.E. Eye tracking in the wild. Comput. Vis. Image Understand 2005, 98, 155–181. [Google Scholar] [CrossRef] [Scilit]
  13. Zhang, X.; Sugano, Y.; Fritz, M.; Bulling, A. It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017; pp. 2299–2308. [Google Scholar]
  14. Lee, H.; Lee, W.; Baek, J.; Kim, B. Dual-stream Network for Gaze Estimation using RGB and NIR Images. In IEEE Transactions on Biometrics, Behavior, and Identity Science; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
  15. Chen, Z.; Wang, H.; Li, Y.; Wang, M. A Lightweight Deep Learning Model for Real-Time Gaze Estimation on Mobile Devices. In IEEE Transactions on Circuits and Systems for Video Technology; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
  16. Park, S.; Spurr, A.; Hilliges, O. Distant Supervision for Gaze Estimation via Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021. [Google Scholar]
  17. Zhang, X.; Li, Y.; Liu, Y.; Shao, J. EfficientGaze: A Neural Architecture Search Based Approach for Practical Gaze Estimation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) 2023, 7, 65. [Google Scholar]
  18. Cheng, Y.; Feng, Z.; Wang, X.; Lei, X.; Li, Y.; Wang, S. A Gaze Estimation Transformer with Generalized Spatial Attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 1–6 October 2023. [Google Scholar]
  19. Wang, K.; Zhao, R.; Su, H.; Ji, Q. GazeTR: End-to-End Gaze Estimation with Transformer. In IEEE Transactions on Pattern Analysis and Machine Intelligence; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
  20. Liu, R.; Zhang, Y.; Zhang, Z.; Liu, Y. DGaze: A Dense Hourglass Network for Appearance-Based Gaze Estimation. In Expert Systems with Applications; Department of Electrical, Computer and Systems Engineering Rensselaer Polytechnic Institute Troy: New York, NY, USA, 2024. [Google Scholar]
  21. Park, S.; Mello, S.D.; Molchanov, P.; Iqbal, U.; Hilliges, O.; Kautz, J. Few-Shot Adaptive Gaze Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021. [Google Scholar]
  22. Chen, Z.; Huang, Y.; Fu, Y.; Liu, R.; Liu, Y. Disentangling Identity and Gaze for Improved Generalization. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Cancún, Mexico, 8–12 October 2023. [Google Scholar]
  23. Wang, Z.; Wang, Y.; Chen, Y.; Liu, J.; Liu, Y. Unsupervised Domain Adaptation for Gaze Estimation. In IEEE Transactions on Biometrics, Behavior, and Identity Science; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
  24. Zhang, H.; Zhao, R.; Ji, Q. EventGaze: Robust Gaze Estimation with Event Cameras. IEEE Robot. Autom. Lett. 2024. [Google Scholar]
  25. Lee, J.; Lee, Y.-L.; Shi, H.; Liu, C.; Huang, L. Self-Supervised Multimodal Learning for Gaze Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
  26. Zhang, K.; Zhang, Z.; Li, Z.; Qiao, Y. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Process. Lett. 2016, 23, 1499–1503. [Google Scholar] [CrossRef] [Scilit]
  27. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
  28. Baltrusaitis, T.; Zadeh, A.; Lim, Y.C.; Morency, L.P. OpenFace 2.0: Facial Behavior Analysis Toolkit. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), Xi’an, China, 15–19 May 2018; pp. 59–66. [Google Scholar]
  29. Krafka, K.; Khosla, A.; Kellnhofer, P.; Kannan, H.; Bhandarkar, S.; Matusik, W.; Torralba, A. Eye Tracking for Everyone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2176–2184. [Google Scholar]
  30. Smith, J.; Gatica-Perez, D.; Odobez, J.-M. Multimodal Gaze Estimation using RGB-D Data. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Glasgow, UK, 23–28 August 2020. [Google Scholar]
  31. Bradski, G. The OpenCV Library. Dr. Dobb’s J. Softw. Tools 2000, 25, 120–126. [Google Scholar]
  32. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Adv. Neural Inf. Process. Syst. (NeurIPS) 2019, 32, 8024–8035. [Google Scholar]
  33. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.