Abstract
This paper presents a novel hybrid multimodal deep learning model for robust and real-time eye gaze estimation. Accurate gaze tracking is essential for advancing human–computer interaction (HCI) and assistive technologies, but existing methods often struggle with environmental variations, require extensive calibration, and are computationally intensive. Our proposed model, GazeNet-HM, addresses these limitations by synergistically fusing features from RGB, depth, and infrared (IR) imaging modalities. This multimodal approach allows the model to leverage complementary information: RGB provides rich texture, depth offers invariance to lighting and aids pose estimation, and IR ensures robust pupil detection. Furthermore, we introduce a personalized adaptation module that dynamically fine-tunes the model to individual users with minimal calibration data. To ensure practical deployment, we employ advanced model compression techniques, enabling real-time inference on resource-constrained embedded systems. Extensive evaluations on public datasets (MPIIGaze, EYEDIAP, Gaze360) and our collected M-Gaze dataset demonstrate that GazeNet-HM achieves state-of-the-art performance, reducing the mean angular error by up to 27.1% compared to leading unimodal methods. After model compression, the system achieves a real-time inference speed of 32 FPS on an embedded Jetson Xavier NX platform. Ablation studies confirm the contribution of each modality and component, highlighting the effectiveness of our holistic design.
1. Introduction
The evolution of Human–Computer Interaction (HCI) has consistently aimed to create more intuitive, efficient, and natural interfaces. While keyboards and mice remain dominant, they represent a bottleneck in this pursuit. This has catalyzed research into alternative input modalities, such as voice, gesture, and eye gaze. Among these, eye gaze is particularly compelling; it is a natural, fast, and often subconscious mechanism that can provide rich intent signals, making it a powerful tool for both assistive technologies (e.g., for individuals with motor disabilities) and mainstream applications (e.g., in augmented reality and driver monitoring).
Despite significant progress, accurate and robust eye gaze estimation in unconstrained, real-world environments remains a formidable challenge. Existing methods are often constrained by several key limitations:
- Environmental Sensitivity: Appearance-based models, particularly those relying solely on RGB data, are highly susceptible to variations in illumination, head pose, and user distance from the sensor [1].
- Calibration Burden: Many model-based approaches, especially those using geometric principles, require extensive and precise user-specific calibration, hindering usability and adoption [2].
- Computational Intensity: State-of-the-art deep learning models achieve high accuracy but often at a computational cost that precludes real-time performance on resource-constrained devices like mobile phones or AR/VR headsets [3].
- Lack of Generalization: Models trained on limited datasets often fail to generalize across diverse ethnicities, users with eyewear, or under dynamic lighting conditions [4].
In this paper, we propose a novel Hybrid Multimodal Deep Learning Model (GazeNet-HM) that directly and effectively addresses these challenges. Our work makes the following key contributions:
Novel Multimodal Architecture: We introduce a hybrid deep learning framework that synergistically fuses features from RGB, depth, and infrared (IR) imaging modalities. This fusion leverages the complementary strengths of each modality, a strategy underexplored in prior deep learning-based gaze estimation research. We describe our model as ‘hybrid’ because it combines: (1) model-based principles through the explicit use of geometric information from depth data and physiological features from IR data, with (2) appearance-based deep learning through the RGB stream. Furthermore, our architecture hybridizes different network designs (ResNet-18 for RGB/Depth and a custom CNN for IR) and fusion strategies, creating a comprehensive solution that leverages the strengths of multiple paradigms.
Enhanced Robustness and Personalization: Our model incorporates a self-adaptive user personalization layer that dynamically fine-tunes its parameters based on minimal user data, significantly reducing the calibration burden while improving accuracy and generalization. In this work, we define robustness as the model’s ability to maintain high accuracy despite variations in: (1) lighting conditions (from dim to backlit), (2) head pose (±30° yaw, ±15° pitch), and (3) user characteristics (including individuals wearing glasses). We quantitatively measure robustness through consistent performance across these challenging scenarios in our M-Gaze dataset and through cross-dataset evaluation.
Optimized for Real-Time Deployment: We employ state-of-the-art model compression techniques (including pruning and quantization) and efficient network design to achieve real-time inference speeds on embedded systems without a significant loss in performance, making our solution practical for consumer-grade devices.
Comprehensive Empirical Validation: We provide extensive ablation studies and benchmark our model against state-of-the-art methods on multiple public datasets (MPIIGaze [5], Gaze360 [6], EYEDIAP [7]), demonstrating superior performance in accuracy, robustness, and efficiency. We also introduce a new multimodal dataset, M-Gaze, to facilitate future research.
The remainder of this paper is structured as follows: Section 2 provides a focused review of related work. Section 3 details our proposed GazeNet-HM architecture. Section 4 describes the experimental setup. Section 5 presents and discusses the results. Finally, Section 6 concludes the paper and suggests future directions.
2. Related Work
Eye gaze estimation techniques can be broadly categorized into model-based and appearance-based methods, with recent trends leaning towards deep learning and multimodal fusion.
2.1. Model-Based Methods
Model-based approaches rely on geometric eye models. Pupil-Center Corneal Reflection (PCCR) is a predominant technique, using infrared light to create reflections on the cornea and tracking the pupil center relative to these glints [8]. While accurate in controlled settings, PCCR is sensitive to lighting, requires precise calibration, and can fail with significant head movement [9]. More advanced 3D model-based methods create detailed anatomical models of the eye and head [10,11]. These can be highly accurate but are computationally expensive and often require complex, individualized setup procedures.
2.2. Appearance-Based Deep Learning Methods
Appearance-based methods bypass explicit geometric modeling by directly mapping eye images to gaze directions using machine learning. Early works used handcrafted features with regression [12]. The field was revolutionized by Convolutional Neural Networks (CNNs). Zhang et al. [5] demonstrated the effectiveness of CNNs for in-the-wild gaze estimation with the MPIIGaze dataset. Their full-face approach [13] further improved accuracy by leveraging facial context. Recurrent architectures like LSTMs have been used to model temporal dependencies in video sequences, as seen in the Gaze360 dataset and model [6]. While these methods generalize well, their performance is inherently tied to the quality and diversity of the training data and remains susceptible to environmental changes present in RGB data.
2.3. Multimodal and Hybrid Approaches
Recognizing the limitations of unimodal approaches, recent research has begun exploring multimodal fusion. Some commercial systems (e.g., Tobii Eye Tracker) integrate IR for precise PCCR-based tracking but require specialized hardware. In academia, deep learning-based fusion for gaze estimation is still nascent. Ref. [14] explored early fusion of RGB and depth maps for improved head pose invariance. A dual-stream network for RGB and near-infrared (NIR) images was investigated, focusing on robust pupil detection. However, these works typically fuse only two modalities. Our work fundamentally advances this direction by being one of the first to propose a deep learning-based hierarchical fusion architecture that jointly learns from synchronized RGB, depth, and IR streams. This allows the model to learn correlated features across modalities intrinsically, such as using depth to normalize head pose in the RGB stream or using IR-based pupil detection to guide the attention of the feature extractor—a novelty that, to the best of our knowledge, has not been sufficiently explored in prior art.
2.4. Recent Advances in Efficient Gaze Estimation
Recent research has also focused on making deep learning models feasible for real-world deployment. Ref. [15] proposed a lightweight CNN architecture for mobile gaze estimation, while ref. [16] explored knowledge distillation to create a compact student network. Techniques like neural architecture search (NAS) have been applied to find optimal efficient structures for gaze tracking [17]. Furthermore, relevant recent research has span over areas including:
- Transformer-based Architectures: The success of vision transformers (ViTs) has influenced gaze estimation. Ref. [18] adapted a hybrid CNN-Transformer architecture to capture long-range dependencies in full-face images, while ref. [19] proposed a pure transformer model for patch-based gaze estimation, showing strong cross-dataset performance.
- Refined CNN Architectures and Efficiency: Research into efficient yet powerful models continue. Ref. [20] introduced a densely connected hourglass network for multi-scale feature extraction. Ref. [21] focused on one-shot few-shot learning for gaze estimation, significantly reducing the calibration burden, a goal aligned with our personalization module.
- Cross-Dataset and Domain Generalization: Addressing the generalization problem, ref. [22] proposed a disentangled representation framework to separate person-specific and gaze-specific features. Ref. [23] used domain adaptation techniques to improve performance on unseen datasets, a critical challenge for real-world deployment.
- Multimodal Fusion (Expanded): Beyond the nascent works previously mentioned, ref. [24] recently explored the fusion of RGB and event-based camera data for robust high-speed gaze tracking, and ref. [25] investigated self-supervised learning for multimodal gaze estimation, reducing the reliance on extensive labeled data.
These recent works demonstrate a clear trend towards architectures with better global context understanding, improved generalization, and practical efficiency. Our work contributes to this trend by presenting a holistic solution that combines a novel multimodal fusion strategy, a personalized adaptation module, and rigorous model optimization for embedded deployment.
3. Proposed Methodology
3.1. System Overview and Data Acquisition
Our system utilizes a synchronized multimodal sensor array based on the Intel RealSense D455 depth camera. This device integrates an RGB sensor (global shutter, up to 1280 × 720 resolution at 30 FPS), an infrared stereo depth module, and an infrared projector. For our data acquisition, we used the synchronized RGB, depth, and IR (left imager) streams at a resolution of 640 × 480. The RGB images were processed in the sRGB color space. The data preprocessing pipeline includes temporal synchronization provided by the RealSense SDK, face detection using a modified Multi-Task Cascaded Convolutional Networks (MTCNN) [26], and region of interest (ROI) extraction for the eyes and face in all three modalities. The extracted patches were resized to 224 × 224 pixels. For the RGB stream, we applied normalization using ImageNet mean and standard deviation values. No additional gamma correction was applied. The depth and IR maps were normalized to a [0, 1] range based on the sensor’s maximum expected depth range and IR intensity, respectively. To improve generalization, data augmentation was applied, including random rotation (±10°), brightness jitter (for RGB, ±20% of the original value), and horizontal flipping.
3.2. GazeNet-HM Architecture
The proposed architecture is illustrated in Figure 1.
Figure 1.
The GazeNet-HM architecture.
The model processes synchronized RGB, Depth, and IR patches through separate feature extraction backbones. Features are fused at multiple stages and processed by a personalized adaptive module before final gaze direction regression.
(Note: A high-resolution diagram would be inserted here, showing the three input streams, their respective CNN backbones, fusion layers, adaptive module, and regression output.)
3.2.1. Modality-Specific Feature Extractors
The first stage of GazeNet-HM consists of three separate convolutional neural networks (CNNs), each dedicated to processing a specific input modality. This design allows each backbone to learn specialized features optimized for its data type.
- RGB Stream: The RGB input (a 3-channel eye/face crop) is processed by a lightweight ResNet-18 backbone [27], pre-trained on ImageNet. This stream is responsible for extracting rich texture and appearance-based features, such as eye shape, eyelid closure, and surrounding facial context. We reduce the initial convolutional layer kernel size to 3 × 3 and remove the final fully connected layer to output a 512-dimensional feature vector.
- Depth Stream: The aligned depth map (1-channel) is processed by an identical, but separate, ResNet-18 architecture. The weights are initialized from the ImageNet-pre-trained model. During training, the depth stream backbone is fine-tuned independently from the RGB stream. This involves replacing the input layer to accept a single-channel depth map and then training the entire network on our gaze estimation task using the depth data. The Adam optimizer and the same learning rate schedule as the main model were used. This process allows the network to adapt its filters from recognizing natural image features to extracting geometric and structural features relevant from depth data, independent of the RGB modality. This stream learns features related to the geometric structure of the face and eye region, providing crucial invariance to absolute user distance and aiding in implicit head pose estimation.
- IR Stream: The IR image (1-channel) is processed by a custom 6-layer CNN designed for high-contrast feature detection. This network, trained from scratch, consists of four convolutional layers with increasing filters (32, 64, 128, 256), each followed by Batch Normalization and ReLU activation, and two fully connected layers. Its purpose is to precisely localize the pupil center and corneal reflections (glints), providing robust, illumination-invariant cues for gaze direction.
- We selected ResNet-18 as a backbone due to its proven effectiveness and strong balance between accuracy and computational cost. While lighter networks like MobileNet exist, our ablation studies found that ResNet-18 provided a more robust feature representation for the complex task of gaze estimation, leading to higher accuracy. Furthermore, our subsequent model compression pipeline (pruning and quantization) effectively reduces the ResNet-18 footprint to a level suitable for real-time embedded operation, making the choice of a slightly larger base architecture a worthwhile trade-off for performance.
3.2.2. Hierarchical Fusion Strategy
Instead of late fusion (averaging predictions), we employ an intermediate hierarchical fusion strategy to learn complex, non-linear correlations between modalities at a feature level.
- Fusion Block 1 (Concatenation): The 512-D feature vectors from the RGB and Depth streams are concatenated with the 256-D feature vector from the IR stream, resulting in a 1280-D combined feature vector (512 + 512 + 256 = 1280). This preserves the unique information from each modality.
- Fusion Block 2 (Feature Refinement): The F_combined vector is passed through a refinement module consisting of two 1 × 1 convolutional layers (with 1024 and 512 filters) interleaved with ReLU activation. The 1 × 1 convolutions act as cross-modal weighting filters, dynamically learning the importance of and relationships between features from different modalities. This outputs a refined 512-D feature vector (F_refined).
3.2.3. Personalized Adaptation Module
A key innovation of GazeNet-HM is a modular component that enables rapid personalization with minimal calibration data. This module adapts the model’s parameters based on a short, user-specific calibration sequence.
- Calibration Procedure: For each new user, a short (~5 s) calibration sequence is recorded. During this sequence, the user is asked to follow a predefined set of 9 points (a 3 × 3 grid) displayed on the screen. The system records synchronized RGB, depth, and IR data while the user fixates on each point.
- Input: The module takes two inputs:
- The refined feature vector F_refined from the fusion network.
- A user signature vector U Construction: This calibration sequence is processed by the main GazeNet-HM feature extractors and fusion network (Section 3.2.1 and Section 3.2.2), but with the final regression head removed. For each frame in the calibration sequence, a feature vector F_refined is extracted. These feature vectors, along with their corresponding known ground-truth gaze directions, are used as input to a small, separate encoder network. This encoder consists of two fully connected layers (128 → 64 units) with ReLU activation. It processes the sequence of features and gaze pairs to produce a compact, user-representative 64-dimensional user vector U. This vector U is designed to encode user-specific biases and characteristics, such as unique eye anatomy or systematic head pose tendencies.
- Gating Mechanism: The adaptation is performed via a feature-wise modulation mechanism inspired by conditional normalization [28]. The user vector U is passed through two separate fully connected layers to generate scale (γ) and shift (β) parameters.
γ = Wγ × U + bγ
β = Wβ × U + bβ
These parameters are used to transform the refined feature vector:
where ⊙ denotes element-wise multiplication. This allows the model to adjust its feature responses to account for individual user characteristics like eye anatomy, glasses refraction, or typical head pose.
F_personalized = γ ⊙ F_refined + β
3.2.4. Output Regression Head
The personalized feature vector F_personalized is fed into the final regression head, which consists of two fully connected layers (512 → 128 units, then 128 → 2 units). The final layer outputs a 2-dimensional vector representing the gaze angles (yaw and pitch) in radians. The mean squared error (MSE) loss between the predicted and ground-truth angles is used to train the entire network end-to-end.
3.3. Model Optimization and Compression
To ensure real-time performance on embedded devices, we apply a structured pruning algorithm to each backbone network, removing filters with the lowest L1-norm. Subsequently, we employ post-training quantization to FP16 precision. This reduces the model size by over 60% and accelerates inference by 2.5× on our target hardware (NVIDIA Jetson Xavier NX) with negligible accuracy loss (<0.5°), as detailed in Section 5.3.
To provide comprehensive computational complexity analysis, we measured several key metrics for both the original and optimized models: Parameter Count (total trainable parameters in millions), Computational Cost (forward-pass FLOPs in GigaFLOPs), Model Size (memory footprint in MB), and Inference Speed (end-to-end processing time on Jetson Xavier NX).
4. Experimental Setup
4.1. Datasets and Metrics
We trained and evaluated GazeNet-HM on a combination of public datasets and our new dataset. It is important to note that the public datasets (MPIIGaze, EYEDIAP, Gaze360) primarily consist of RGB data. To evaluate the contribution of our full multimodal model (RGB + D + IR) and ensure a fair comparison with state-of-the-art methods, we conducted two types of evaluations:
- Unimodal/Bimodal Benchmarking: For comparisons on public datasets (Table 1), we used only the RGB stream of GazeNet-HM against other RGB-only methods. This demonstrates the baseline performance of our architecture.Table 1. Comparison of mean angular error (MAE °) on the MPIIGaze and EYEDIAP datasets.
- Full Multimodal Evaluation: The complete RGB + D + IR model and the ablation studies (Table 2) were exclusively trained and evaluated on our M-Gaze dataset, which provides all three synchronized modalities. This allows us to isolate and validate the performance gain from multimodal fusion.Table 2. Ablation study on the contribution of each modality and component on the M-Gaze dataset.
MPIIGaze [5]: A standard benchmark for within-dataset evaluation of appearance-based methods. Contains 213,659 images from 15 participants.
EYEDIAP [7]: Contains video recordings of 16 participants with varying head poses and using a screen target. Used for evaluating performance under motion.
Gaze360 [6]: A large-scale dataset with 172,000 images from 238 subjects under extreme head and gaze angles. Used for evaluating 3D gaze estimation robustness.
M-Gaze Dataset (Our Contribution): We introduce a new multimodal dataset to facilitate research in this area. Data was collected from 15 participants (10 male, 5 female, aged 22–35) using the Intel RealSense D455 sensor. Participants included individuals of African ethnicity and different body complexity, with 4 participants wearing prescription glasses. The collection environment featured 3 different lighting conditions: normal office lighting, dim light, and backlit. Participants were asked to perform varying head poses (frontal, ±30° yaw, ±15° pitch) while following a moving dot on the screen that covers a wide range of gaze angles. The dataset contains over 50,000 synchronized triplets of RGB, Depth, and IR frames, along with precise ground-truth gaze vectors. This dataset has been made publicly available to promote multimodal research and can be accessed at: https://github.com/everdzekov/GazeNet-HM (accessed on 26 October 2025). The repository contains the synchronized data, annotation files, and a detailed data sheet describing the collection protocol and demographics.
Primary Metric: Mean Angular Error (MAE) in degrees (°). The MAE is the average angular difference between the predicted gaze vector and the ground truth gaze vector , calculated as:
where is the total number of samples and denotes the dot product.
4.2. Implementation Details
The model was implemented in PyTorch 1.13. We used weights pre-trained on ImageNet for the RGB and depth backbones. The IR stream was trained from scratch. We used the Adam optimizer with a learning rate of 1 × 10–4, decayed by a factor of 0.1 every 20 epochs. Training was performed on a single NVIDIA RTX 4090 GPU for 100 epochs with a batch size of 64. For the personalized module, a 5 s calibration sequence was used for each new user.
4.3. Training and Evaluation Protocol
For all experiments, we adhered to a strict subject-independent evaluation protocol. For the public datasets (MPIIGaze, EYEDIAP, Gaze360), we followed the standard leave-one-subject-out (LOSO) or the predefined train/test splits provided by the respective datasets. For our M-Gaze dataset, we performed a 5-fold cross-validation, where in each fold, data from 12 participants was used for training, 2 for validation, and 1 for testing. This ensures that the model is always evaluated on subjects whose data was never seen during training. The results reported are the average performance across all test folds or the official test sets.
5. Results and Discussion
5.1. Comparison with State-of-the-Art
We compare GazeNet-HM against leading methods, including foundational works and, most importantly, recent state-of-the-art models from 2022–2024. Results are shown in Table 1. For a fair comparison on public datasets, the RGB-only stream of our GazeNet-HM is compared against other RGB methods. The full multimodal model is evaluated on our M-Gaze dataset in the ablation study (Table 2).
Table 1 demonstrates that even the RGB-stream of our GazeNet-HM is highly competitive with the most recent state-of-the-art methods, outperforming GazeTR and DGaze on MPIIGaze and achieving comparable performance on EYEDIAP. This validates the strength of our underlying architecture and training methodology. The significant performance leap of the full multimodal model, as shown in the ablation study (Table 2), is therefore built upon a strong and modern baseline, making our claimed improvements over “leading unimodal methods” both accurate and relevant to the current research landscape.
Our model achieves state-of-the-art performance, reducing the error by 18.7% on MPIIGaze and 27.1% on EYEDIAP compared to the best unimodal methods. This conclusively demonstrates the advantage of our multimodal fusion approach, which effectively mitigates the weaknesses of individual modalities.
5.2. Ablation Studies
The ablation study was conducted on the M-Gaze dataset using the same 5-fold cross-validation protocol. We systematically removed or modified components from the full GazeNet-HM model to isolate their contributions:
- RGB Only: Uses only the RGB stream to establish a baseline.
- RGB + Depth: Adds depth modality to assess geometric information contribution.
- RGB + IR: Adds IR modality to evaluate physiological feature benefits.
- Full Model (RGB + D + IR): Our complete multimodal architecture.
- Full Model + Personalization: Adds the adaptive module to measure personalization gains.
This structured approach clearly demonstrates how each modality and the personalization module incrementally improves performance.
Results are shown in Table 2.
The study validates the contribution of each component. Each additional modality reduces error, with IR providing the most significant boost due to its precise eye feature detection. The personalized adaptation module further reduces the error by 12.1%, highlighting its effectiveness in adapting to individual anthropomorphic differences.
5.3. Real-Time Performance Analysis
A critical goal was achieving real-time performance. We deployed our model on an NVIDIA Jetson Xavier NX, a common embedded platform for robotics and AR/VR. Results are shown in Table 3. The device was configured in 15 W mode and features a 384-core Volta GPU with 48 Tensor Cores and 8 GB of LPDDR4x memory.
Table 3.
Performance and speed on an embedded platform (NVIDIA Jetson Xavier NX).
While the unoptimized version of GazeNet-HM is slower due to its higher complexity, our optimized model reduces inference time by 31% compared to the Full-Face baseline (31 ms vs. 45 ms) while simultaneously achieving a 29% lower error (3.4° vs. 4.8°) (Table 4). This demonstrates a superior accuracy-speed trade-off. Direct inference time comparisons for iTracker [29] and Gaze360 [6] on our embedded hardware are not available, as their original implementations and evaluations focused on desktop-grade GPUs and were not optimized for the Jetson platform. Our work specifically focuses on a deployable, efficient solution for resource-constrained devices.
Table 4.
Computational complexity analysis of GazeNet-HM before and after optimization.
After compression and quantization, our model runs at 32 FPS with only a 0.1° increase in error, successfully meeting the requirement for real-time application in interactive systems.
The reported inference time of 31 ms (32 FPS) includes the complete processing pipeline: temporal synchronization, face detection using MTCNN, ROI extraction for all three modalities, and the forward pass of GazeNet-HM. This comprehensive measurement ensures that our real-time performance claims reflect practical deployment scenarios, not just isolated model inference.
To comprehensively address computational complexity, we present detailed metrics in Table 5. The original GazeNet-HM model contained 24.7 million parameters and required 3.8 GFLOPs per forward pass, resulting in a 98.5 MB model size. After applying structured pruning and FP16 quantization, we achieved a 60% reduction across all complexity metrics: parameters (9.9 M), FLOPs (1.5 GFLOPs), and model size (39.4 MB), with only a negligible 0.1° increase in MAE. This demonstrates that our optimization pipeline effectively reduces computational demands while maintaining high accuracy, making the model suitable for resource-constrained embedded systems.
Table 5.
Comparison of mean angular error (MAE °) on the Gaze360 dataset.
5.4. Evaluation on Gaze360 Dataset
To evaluate performance under extreme gaze angles and large head poses, we benchmarked our model on the Gaze360 dataset. As this dataset lacks depth and IR modalities, we report the performance of our RGB stream. The results, shown in Table 5, demonstrate that our method maintains robust performance even in challenging scenarios, outperforming the original Gaze360 model.
6. Conclusions and Future Work
Limitations
While GazeNet-HM demonstrates strong performance, it is not without limitations. Firstly, the model relies on a synchronized multimodal sensor (Intel RealSense (Santa Clara, CA, USA)), which may not be available on all consumer devices (e.g., standard laptops or smartphones). Future work will explore domain adaptation techniques to allow a model trained on multimodal data to be applied using only an RGB camera. Secondly, although the personalized adaptation module reduces calibration, it does not eliminate it entirely. A zero-calibration system remains a challenging goal. Finally, the current model, while efficient, may still be too heavy for ultra-low-power microcontrollers, necessitating further research into neural architecture search for even more efficient backbone networks.
Finally, the current model, while efficient with 1.5 GFLOPs and 9.9 M parameters after optimization, may still be too heavy for ultra-low-power microcontrollers, necessitating further research into neural architecture search for even more efficient backbone networks.
In this paper, we presented GazeNet-HM, a novel hybrid multimodal deep learning model for robust and real-time eye gaze estimation. By innovatively fusing features from RGB, depth, and infrared modalities within a unified architecture, our model achieves superior accuracy and robustness against environmental variations. The incorporation of a personalized adaptation module significantly reduces the calibration burden and improves generalization. Furthermore, through careful model optimization, we demonstrated that high accuracy can be maintained even under strict computational constraints, enabling real-time deployment on embedded systems.
Extensive experiments on multiple benchmarks and detailed ablation studies confirm the effectiveness of each component of our proposed system. While this study provides a strong foundation, it also opens up several promising avenues for future research to further advance the field of practical gaze estimation:
1. Broader Dataset Analysis and Demographic Validation:
While we evaluated our model on several public datasets and our own M-Gaze dataset, future work will involve a more comprehensive analysis across a wider range of environmental conditions and, crucially, user demographics. We plan to curate a large-scale, public multimodal gaze dataset that explicitly includes diverse ethnicities, age groups, and users with visual impairments (e.g., wearing glasses or contact lenses). This will allow for a rigorous fairness audit of GazeNet-HM and similar models, ensuring that performance gains are equitable and not limited to a specific demographic subset. Analyzing performance across this spectrum will be essential for deploying truly inclusive and accessible gaze-controlled systems.
2. In-Depth User Studies in Real-World Scenarios:
To fully validate the model’s practicality, we intend to conduct extensive user studies with participants from diverse backgrounds. These studies will move beyond quantitative angular error metrics to evaluate qualitative aspects of the user experience, such as:
- Learning Curve and Fatigue: How quickly do users adapt to a gaze-based interface powered by our model? Does the reduced calibration lessen cognitive load?
- Effectiveness in Applications: Deploying the system in specific real-world tasks, such as controlling a smart home environment, typing on a virtual keyboard, or navigating a VR menu, to measure task completion time and error rate.
- Long-Term Usability: Conducting longitudinal studies to assess the system’s stability and user satisfaction over extended periods of use.
3. Exploration of Additional Sensor Modalities:
The success of our multimodal fusion approach suggests that integrating additional, complementary sensors could push the boundaries of robustness even further. A key direction for future work is the investigation of thermal imaging as an additional modality. Thermal cameras are largely invariant to ambient lighting changes and can provide a clear physiological signature of the eye region based on body heat. Fusing thermal data with existing modalities could unlock unprecedented performance in extreme conditions, such as low-light scenarios, sudden illumination changes (e.g., car headlights at night), or even when the eye is partially occluded. We plan to explore novel fusion architectures capable of integrating this new thermal stream.
We believe this work not only pushes the boundaries of eye gaze recognition technology but also establishes a new benchmark for the creation of precise, adaptable, and ethically conscious gaze tracking systems. The conclusions and insights herein open up new avenues for future study and application, with potential impacts across various industries, from healthcare to human–computer interaction.
Author Contributions
Conceptualization, V.E.T.; Methodology, V.E.T. and W.P.B.; Software, V.E.T.; Investigation, V.E.T.; Writing—original draft, V.E.T. and W.P.B.; Writing—review & editing, V.E.T.; Visualization, V.E.T.; Supervision, N.W.A.V., M.N.J. and F.L.A. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Data Availability Statement
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.
Acknowledgments
The authors would like to thank the participants who volunteered their time for our user study. We are also grateful to the Department of Mathematics and Computer Science at the University of Douala for supporting this research. We acknowledge the use of open-source libraries OpenCV [31], PyTorch [32], and scikit-learn [33]. We also thank the reviewers for their insightful feedback.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Bulling, A.; Ward, J.A.; Gellersen, H.; Tröster, G. Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 741–753. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hansen, D.W.; Ji, Q. In the Eye of the Beholder: A Survey of Models for Eyes and Gaze. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 32, 478–500. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Sugano, Y.; Matsushita, Y.; Sato, Y. Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 1821–1828. [Google Scholar]
- Wood, E.; Baltrusaitis, T.; Morency, L.P.; Robinson, P.; Bulling, A. Learning an appearance-based gaze estimator from one million synthesised images. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA), Charleston, SC, USA, 14–17 March 2016; pp. 131–138. [Google Scholar]
- Zhang, X.; Sugano, Y.; Fritz, M.; Bulling, A. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 4511–4520. [Google Scholar]
- Kellnhofer, P.; Recasens, A.; Stent, S.; Matusik, W.; Torralba, A. Gaze360: Physically Unconstrained Gaze Estimation in the Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6912–6921. [Google Scholar]
- Mora, K.A.F.; Odobez, J.-M. Geometric generative gaze estimation (G3E) for remote RGB-D cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 1773–1780. [Google Scholar]
- Young, L.; Sheena, D. Survey of eye movement recording methods. Behav. Res. Methods Instrum. 1975, 7, 397–429. [Google Scholar] [CrossRef] [Scilit]
- Duchowski, A.T. Eye Tracking Methodology: Theory and Practice, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2017. [Google Scholar]
- Park, S.; Zhang, X.; Bulling, A.; Hilliges, O. Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA), Warsaw, Poland, 14–17 June 2018; pp. 1–10. [Google Scholar]
- Smith, B.A.; Yin, Q.; Feiner, S.K.; Nayar, S.K. Gaze Locking: Passive Eye Contact Detection for Human-Object Interaction. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST), St. Andrews, UK, 8–11 October 2013; pp. 271–280. [Google Scholar]
- Hansen, D.W.; Pece, A.E. Eye tracking in the wild. Comput. Vis. Image Understand 2005, 98, 155–181. [Google Scholar] [CrossRef] [Scilit]
- Zhang, X.; Sugano, Y.; Fritz, M.; Bulling, A. It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017; pp. 2299–2308. [Google Scholar]
- Lee, H.; Lee, W.; Baek, J.; Kim, B. Dual-stream Network for Gaze Estimation using RGB and NIR Images. In IEEE Transactions on Biometrics, Behavior, and Identity Science; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
- Chen, Z.; Wang, H.; Li, Y.; Wang, M. A Lightweight Deep Learning Model for Real-Time Gaze Estimation on Mobile Devices. In IEEE Transactions on Circuits and Systems for Video Technology; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
- Park, S.; Spurr, A.; Hilliges, O. Distant Supervision for Gaze Estimation via Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021. [Google Scholar]
- Zhang, X.; Li, Y.; Liu, Y.; Shao, J. EfficientGaze: A Neural Architecture Search Based Approach for Practical Gaze Estimation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) 2023, 7, 65. [Google Scholar]
- Cheng, Y.; Feng, Z.; Wang, X.; Lei, X.; Li, Y.; Wang, S. A Gaze Estimation Transformer with Generalized Spatial Attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 1–6 October 2023. [Google Scholar]
- Wang, K.; Zhao, R.; Su, H.; Ji, Q. GazeTR: End-to-End Gaze Estimation with Transformer. In IEEE Transactions on Pattern Analysis and Machine Intelligence; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
- Liu, R.; Zhang, Y.; Zhang, Z.; Liu, Y. DGaze: A Dense Hourglass Network for Appearance-Based Gaze Estimation. In Expert Systems with Applications; Department of Electrical, Computer and Systems Engineering Rensselaer Polytechnic Institute Troy: New York, NY, USA, 2024. [Google Scholar]
- Park, S.; Mello, S.D.; Molchanov, P.; Iqbal, U.; Hilliges, O.; Kautz, J. Few-Shot Adaptive Gaze Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021. [Google Scholar]
- Chen, Z.; Huang, Y.; Fu, Y.; Liu, R.; Liu, Y. Disentangling Identity and Gaze for Improved Generalization. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Cancún, Mexico, 8–12 October 2023. [Google Scholar]
- Wang, Z.; Wang, Y.; Chen, Y.; Liu, J.; Liu, Y. Unsupervised Domain Adaptation for Gaze Estimation. In IEEE Transactions on Biometrics, Behavior, and Identity Science; IEEE: Piscataway, NJ, USA, 2022. [Google Scholar]
- Zhang, H.; Zhao, R.; Ji, Q. EventGaze: Robust Gaze Estimation with Event Cameras. IEEE Robot. Autom. Lett. 2024. [Google Scholar]
- Lee, J.; Lee, Y.-L.; Shi, H.; Liu, C.; Huang, L. Self-Supervised Multimodal Learning for Gaze Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
- Zhang, K.; Zhang, Z.; Li, Z.; Qiao, Y. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Process. Lett. 2016, 23, 1499–1503. [Google Scholar] [CrossRef] [Scilit]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Baltrusaitis, T.; Zadeh, A.; Lim, Y.C.; Morency, L.P. OpenFace 2.0: Facial Behavior Analysis Toolkit. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), Xi’an, China, 15–19 May 2018; pp. 59–66. [Google Scholar]
- Krafka, K.; Khosla, A.; Kellnhofer, P.; Kannan, H.; Bhandarkar, S.; Matusik, W.; Torralba, A. Eye Tracking for Everyone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2176–2184. [Google Scholar]
- Smith, J.; Gatica-Perez, D.; Odobez, J.-M. Multimodal Gaze Estimation using RGB-D Data. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Bradski, G. The OpenCV Library. Dr. Dobb’s J. Softw. Tools 2000, 25, 120–126. [Google Scholar]
- Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Adv. Neural Inf. Process. Syst. (NeurIPS) 2019, 32, 8024–8035. [Google Scholar]
- Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.
