Landmark-Independent Head Pose Estimation Under Face Mask Occlusion Using Diffusion-Based Face Restoration

Abstract

Head pose estimation plays an important role in many real-world applications. However, the performance of existing methods degrades significantly when facial regions are occluded by face masks. To address this limitation, this paper proposes a two-stage framework that integrates diffusion-based face restoration with landmark-independent head pose estimation. In the proposed framework, face mask regions are first detected through segmentation, and the occluded facial regions are reconstructed using a pretrained Stable Diffusion model. The restored facial images are then fed into a deep learning-based head pose estimation model to predict the yaw, pitch, and roll angles. Experiments were conducted on a masked-face dataset constructed from AFLW2000 dataset by synthesizing realistic face-mask occlusions. The proposed framework achieves an average Mean Absolute Error (MAE) of 3.83° under mask occlusion, outperforming the landmark-based baseline and the evaluated landmark-independent head pose estimation methods. Furthermore, an ablation study shows that diffusion-based face restoration reduces the Average MAE from 5.43° to 3.83°, demonstrating its effectiveness in mitigating the information loss caused by facial occlusion and improving head pose estimation accuracy.

1. Introduction

Head pose estimation plays a critical role in a wide range of applications, including human–computer interaction, driver assistance systems, augmented and virtual reality (AR/VR), and healthcare. In human–computer interaction, it is commonly used to infer a user’s gaze direction and attention state [1,2]. In driver assistance systems, head pose estimation is essential for detecting driver distraction and drowsiness, thereby improving driving safety [3,4]. In AR/VR environments, real-time head pose estimation enables natural viewpoint control and enhances user immersion [5]. In healthcare, it has also been applied to patient posture assessment and rehabilitation support [6].

1.1. Landmark-Based Head Pose Estimation

Head pose estimation methods can be broadly classified into two categories: landmark-based approaches and deep learning-based approaches.

Landmark-based head pose estimation methods detect facial feature points (e.g., the eyes, nose, and mouth) and estimate head pose based on their geometric relationships [7,8]. Typically, head pose is computed by solving a Perspective-n-Point (PnP) problem, which establishes correspondences between 2D facial landmarks and a predefined 3D face model [9]. This approach originated from early studies in the 1990s that estimated head pose by aligning facial feature points with a 3D rigid head model [10,11]. Subsequently, research evolved from geometric estimation using a small number of 2D feature points to more sophisticated model-based methods incorporating detailed 3D head models [12]. The introduction of textured 3D face models further improved robustness to variations in illumination and head pose [13].

The number of facial landmarks used for head pose estimation has also increased substantially over time. Early methods relied on only a few to a dozen feature points [14], whereas later approaches employed several dozen landmarks, enabled by advances in statistical shape modeling, to represent facial geometry more accurately [15,16]. In parallel, pose estimation techniques evolved from simple 2D geometric approximations to the direct estimation of physically consistent 3D pose parameters.

Landmark-based methods have a well-established theoretical foundation and can achieve satisfactory performance with relatively small amounts of training data. In addition, the explicit correspondence between facial landmarks and pose estimation results makes these methods highly interpretable. However, their performance strongly depends on the accuracy of landmark detection. When facial regions are occluded, landmarks may be inaccurately detected or completely missing, and these errors can propagate directly to the subsequent pose estimation stage, resulting in substantial performance degradation [17].

1.2. Deep Learning-Based Face Orientation Estimation

Deep learning has demonstrated strong performance in various image processing and machine vision applications [18,19]. In particular, deep learning-based head pose estimation methods have been extensively studied since the 2010s. These approaches take facial images as input and directly estimate the Euler angles (yaw, pitch, and roll) representing head pose [20,21]. Unlike landmark-based methods, many deep learning-based approaches do not require explicit landmark detection and instead exploit appearance features extracted directly from facial images. Early studies primarily adopted regression-based networks to predict Euler angles from 2D facial images. For example, HopeNet, proposed by Ruiz et al. [20], combines classification and regression for head pose estimation, improving estimation stability and accuracy.

As the field has evolved, network architectures have become increasingly sophisticated. Deeper and more expressive architectures, such as residual networks (ResNet), have been introduced to provide stronger feature representation capabilities [22]. FSA-Net [21] introduces feature aggregation to effectively combine features at different levels for accurate head pose estimation, while WHENet [23] extends the estimation range to handle large head rotations. Furthermore, several studies have formulated head pose estimation as a hybrid classification–regression problem by discretizing continuous pose angles into classification bins and subsequently recovering continuous angle values [24]. This strategy can improve training stability and estimation accuracy on large-scale datasets.

Meanwhile, the estimation paradigm has gradually expanded from direct 2D pose regression to methods that explicitly exploit 3D facial information. While early deep learning-based approaches directly estimated Euler angles from 2D images, more recent methods estimate the parameters of a 3D Morphable Model (3DMM) [25] or employ multi-task learning frameworks that jointly estimate 3D facial shape and head pose [26]. These developments have extended head pose estimation beyond direct image-based regression toward approaches that model the underlying three-dimensional facial structure.

Because many deep learning-based methods do not rely on explicit landmark extraction, they are generally less sensitive to errors caused by inaccurate landmark detection and can achieve higher estimation accuracy than conventional landmark-based approaches [20]. However, when large portions of the face are occluded, essential visual information is lost. Consequently, even deep learning-based methods may struggle to extract reliable features, leading to performance degradation under severe occlusion [27].

1.3. Face Orientation Estimation Under Occlusion

In real-world environments, fully observable facial conditions are often unavailable. Face masks, sunglasses, hats, hands, and hair can partially obscure facial regions, making facial occlusion a common phenomenon in practical applications.

Facial occlusion can be broadly categorized into two types: temporary occlusion and persistent occlusion. Temporary occlusion occurs when parts of the body, such as hands or hair, briefly and dynamically cover the face. In contrast, persistent occlusion refers to situations in which specific facial regions are continuously obscured by objects such as face masks, eyeglasses, or sunglasses. In this study, we focus on persistent occlusion caused by face masks. Unlike temporary occlusion, persistent occlusion continuously obscures specific facial regions and can affect both landmark detection and visual feature extraction, leading to degradation in head pose estimation accuracy.

To address facial occlusion, various approaches have been proposed. These include methods that learn occlusion-robust feature representations through latent space regression [26] and methods that combine attention mechanisms with Capsule Networks to emphasize informative features under occlusion [28]. Although these approaches improve robustness to occlusion, they estimate head pose directly from partially occluded facial images without explicitly recovering the missing facial information. Consequently, the loss of visual information caused by occlusion remains a challenge.

Recent advances in image generation techniques, particularly generative adversarial networks (GANs) [29] and diffusion models [30], have enabled high-quality reconstruction of missing image content [31,32]. Based on these advances, restore-then-estimate approaches first reconstruct occluded facial regions and subsequently perform head pose estimation on the restored images. For example, previous work [33] reconstructed occluded facial regions using an image completion model and then estimated head pose using facial landmarks and geometric fitting. Although facial restoration can recover information lost due to occlusion, the subsequent head pose estimation stage still depends on accurate facial landmark localization. Therefore, landmark localization errors in the restored facial images can propagate to the final head pose estimation.

Motivated by these limitations, this study integrates diffusion-based face restoration with landmark-independent head pose estimation. The proposed framework differs from existing approaches in two main aspects. First, unlike existing restore-then-estimate approaches that rely on facial landmarks and geometric fitting after face restoration, the proposed framework estimates head pose from the restored facial images using a deep learning-based model without explicit landmark detection. Second, unlike landmark-independent head pose estimation methods that can be applied directly to occluded facial images, the proposed framework first reconstructs the facial information lost due to occlusion and then performs landmark-independent head pose estimation on the restored images. In this way, the proposed framework combines the recovery of facial information lost due to occlusion with the elimination of dependence on facial landmark detection.

The main contributions of this study are summarized as follows:

• Landmark-independent restore-then-estimate framework for head pose estimation under facial occlusion.

We propose a two-stage framework that combines diffusion-based recovery of occluded facial information with landmark-independent head pose estimation. Unlike existing restore-then-estimate approaches that rely on facial landmarks and geometric fitting after restoration, the proposed framework estimates head pose directly from restored facial images without explicit landmark detection. Moreover, unlike landmark-independent methods that estimate head pose directly from occluded facial images, the proposed framework first recovers the missing facial information before pose estimation. This design addresses both occlusion-induced information loss and the dependence on facial landmark localization.

• Systematic evaluation of the contributions of face restoration and landmark-independent pose estimation.

We systematically evaluate the proposed framework on a masked-face dataset constructed from a publicly available dataset while preserving the original head pose annotations. Comparisons with a landmark-based restore-then-estimate baseline and representative landmark-independent methods, including HopeNet, FSA-Net, and WHENet, are conducted to evaluate the effectiveness of landmark-independent estimation after restoration. In addition, a controlled comparison using the same head pose estimation model with and without diffusion-based restoration is performed to evaluate the contribution of diffusion-based face restoration. Qualitative experiments on real masked-face images are also conducted to examine the applicability of the proposed framework to real-world face mask occlusion.

The remainder of this paper is organized as follows. Section 2 describes the proposed framework and its constituent modules. Section 3 presents the experimental setup, results, and discussion. Finally, Section 4 concludes the paper and outlines directions for future research.

2. Proposed Method

In this study, we propose a two-stage framework that integrates face restoration with landmark-independent head pose estimation to improve estimation accuracy under facial occlusion. An overview of the proposed framework is shown in Figure 1. This study focuses specifically on persistent occlusion caused by face masks. However, the modular design allows the framework to be extended to other types of facial occlusion by adapting or replacing the corresponding components.

Figure 1. Overview of the proposed framework.

The input to the proposed framework is a facial image, regardless of whether the subject is wearing a face mask. First, a segmentation model is employed to detect and localize mask-induced occlusion within the facial region. Based on the segmentation result, the framework determines whether face mask occlusion is present. If a face mask is detected, a diffusion-based image inpainting model is applied to reconstruct the occluded facial regions and generate a restored facial image. Otherwise, the original image is directly forwarded to the subsequent head pose estimation module without restoration. Finally, either the restored image or the original image is fed into a landmark-independent deep learning-based head pose estimation model, which estimates the three Euler angles: yaw, pitch, and roll.

In the proposed framework, head pose estimation is performed after restoring the facial information lost due to occlusion. By reconstructing the occluded facial regions before pose estimation, the framework provides more complete facial information for subsequent feature extraction and head pose estimation. In addition, the use of a landmark-independent head pose estimation model eliminates the need for explicit facial landmark detection and reduces the influence of inaccurate landmark localization.

The proposed framework follows a modular architecture rather than an end-to-end training scheme, with face restoration and head pose estimation implemented as separate stages. This design allows the individual modules to be independently replaced or updated according to application requirements.

2.1. Mask Segmentation Model

Wearing a face mask occludes a large portion of the face, particularly the mouth and nose regions, thereby obscuring important facial features and degrading head pose estimation accuracy. To address this issue, the proposed framework first determines whether the input facial image contains a face mask and performs face restoration only when mask-induced occlusion is detected.

In this study, ABANet [34] is adopted for face mask segmentation. ABANet is specifically designed to segment face mask regions in facial images, enabling reliable detection of mask-induced occlusion. Furthermore, it produces pixel-level segmentation masks, which can be directly used for the subsequent image inpainting process.

As shown in Figure 2, ABANet adopts an encoder–decoder architecture augmented with a boundary-aware attention module and a hybrid loss function, enabling accurate delineation of mask boundaries. The network was trained on the Masked Face Segmentation Dataset (MFSD) [34], which contains 11,601 images with 12,758 masked faces annotated at the pixel level. Experimental results reported in [34] demonstrate that ABANet achieves high segmentation accuracy on the MFSD, making it well suited for the proposed face restoration framework.

Figure 2. Architecture of ABANet.

In this study, ABANet is applied to each input facial image to generate a segmentation map of the face mask region. The resulting segmentation map is converted into a binary mask, and the ratio of white pixels is calculated to determine whether a face mask is present. Specifically, the white pixel ratio is defined as the number of pixels classified as mask regions divided by the total number of pixels in the image. A face mask is considered to be present when the white pixel ratio exceeds a predefined threshold T. In this study, the threshold was set to T = 0.04, and its suitability was further evaluated through the sensitivity analysis presented in Section 3.5.2. When a face mask is detected, the binary mask is used as input to the subsequent diffusion-based face restoration module. Throughout this study, a publicly available pretrained ABANet model was used without additional training or fine-tuning.

2.2. Diffusion-Based Inpainting Model

Recent advances in image generation models, particularly Generative Adversarial Networks (GANs) [29] and diffusion models [30], have enabled the restoration of facial regions occluded by masks and other objects. Diffusion models generate images through a forward diffusion process, which gradually adds Gaussian noise to an image, and a reverse diffusion process, which progressively removes the noise to recover a clean image. During training, the model learns the probability distribution required to recover clean images from noisy observations, enabling high-quality image synthesis. Among diffusion models, Stable Diffusion [35] employs a latent diffusion architecture that performs the denoising process in a compressed latent space, enabling high-quality image generation and inpainting.

As illustrated in Figure 3, the proposed framework employs Stable Diffusion to restore facial regions occluded by a face mask. The binary mask generated by ABANet is used as the inpainting mask, and the masked facial region is reconstructed based on the visible facial information. During the restoration process, the input image is first encoded into a latent representation. Noise is added to the latent representation of the region to be restored, and the missing content is reconstructed through the reverse diffusion process conditioned on both the surrounding visual context and a text prompt. Consequently, Stable Diffusion uses the visible facial information to generate plausible facial structures in the occluded regions, including the nose, mouth, and jawline.

Figure 3. Architecture of the Stable Diffusion model used in this study.

Since high-quality pretrained Stable Diffusion models are publicly available, additional training is not required. Previous work by Guo et al. [33] demonstrated the effectiveness of a pretrained Stable Diffusion model for face restoration. Therefore, Stable Diffusion v1.5 [36] was adopted in this study to restore facial regions occluded by face masks before head pose estimation.

Since diffusion-based face restoration is inherently stochastic, different restored images can be generated from the same input image. The objective of this study is not to perfectly reproduce the subject’s original appearance, but rather to recover the facial structural information necessary for accurate head pose estimation. Consequently, although each restored image may differ in local facial details, such as the contours of the nose or mouth, the overall head pose should remain consistent. To reduce the influence of random variations introduced by the restoration process, multiple restored images can be generated for each input image, and head pose is estimated independently from each restored image. The resulting yaw, pitch, and roll estimates are then averaged to obtain the final head pose. The influence of the number of restored images and the stochastic nature of diffusion-based restoration is further investigated in Section 3.5.1 and Section 3.5.3, respectively.

2.3. Head Pose Estimation Model

For head pose estimation, we adopt SixDRepNet [37], a Convolutional Neural Networks (CNN)-based landmark-independent head pose estimation model, as shown in Figure 4. Unlike conventional methods that directly regress Euler angles, SixDRepNet estimates head pose using a 6D rotation representation of the 3D rotation matrix. This representation avoids gimbal lock and angular discontinuities associated with Euler angles, enabling continuous and stable rotation estimation [37].

Figure 4. Architecture of SixDRepNet.

SixDRepNet employs a ResNet backbone to extract facial features and estimate the 6D rotation representation. The network was trained on the 300W-LP dataset [25], a large-scale dataset containing diverse head pose variations. The results reported in [37] show that SixDRepNet achieves competitive head pose estimation accuracy on standard benchmark datasets.

In the proposed framework, the restored facial image is fed into SixDRepNet to estimate the head rotation matrix. The estimated rotation matrix is then converted into the corresponding Euler angles (yaw, pitch, and roll), which are used as the final head pose estimation results. Throughout this study, a publicly available pretrained SixDRepNet model was used without additional training or fine-tuning.

3. Experiments and Results

3.1. Experimental Setup

3.1.1. Dataset

In this study, the AFLW2000 dataset [25] is used as the basis for quantitative evaluation of head pose estimation under both non-occluded and mask-occluded conditions. AFLW2000 is a widely used benchmark constructed from a subset of the AFLW dataset [38]. It contains 2000 facial images with 3D facial annotations. Each image is annotated with head pose information obtained by fitting a 3D face model, providing the three Euler angles: yaw, pitch, and roll. The dataset includes images captured in unconstrained environments with diverse head poses, facial expressions, illumination conditions, and backgrounds, and has been widely used for evaluating head pose estimation methods [20,21,38].

Although several real masked-face datasets are publicly available, most do not provide continuous ground-truth head pose angles for yaw, pitch, and roll, which are required for MAE-based quantitative evaluation. Therefore, to quantitatively evaluate head pose estimation under face mask occlusion while preserving reliable head pose annotations, a corresponding masked-face dataset was constructed from AFLW2000 by synthetically adding face masks to the original images. The synthesized images retain the original AFLW2000 head pose annotations, enabling quantitative evaluation under mask occlusion.

Face mask synthesis was performed using MaskTheFace [39], which generates masked facial images based on facial landmark information. The tool estimates facial geometry from detected landmarks and overlays mask templates according to the position, scale, and orientation of the face. In this study, MaskTheFace was applied to all images in AFLW2000 to generate corresponding masked images. To increase the diversity of mask occlusion, mask templates were randomly selected from multiple types, including surgical masks, KN95 masks, and cloth masks with different colors and shapes. Examples of the original and synthesized masked images are shown in Figure 5.

Figure 5. Examples of original AFLW2000 images and corresponding images with synthesized face masks.

The original and synthesized masked AFLW2000 images are used to evaluate head pose estimation performance under non-occluded and mask-occluded conditions, respectively. Since the synthesized masked images retain the original head pose annotations, the influence of face mask occlusion can be quantitatively evaluated under consistent head pose conditions.

To further examine the applicability of the proposed framework to real face mask occlusion, real masked-face images from the MAFA dataset [40] are additionally used for qualitative evaluation. MAFA contains facial images with real face mask occlusions captured under diverse real-world conditions. However, because MAFA does not provide continuous ground-truth head pose angles for yaw, pitch, and roll, it is used only for qualitative evaluation in this study. Details of this evaluation are presented in Section 3.4.

3.1.2. Evaluation Metrics

In this study, Mean Absolute Error (MAE) is used to evaluate head pose estimation performance. Since head pose is represented by three Euler angles—yaw, pitch, and roll—the MAE is calculated separately for each angle, and the arithmetic mean of the three MAE values is reported as the Average MAE, which provides an overall measure of head pose estimation performance across the three pose angles. All MAE values are reported in degrees (°).

The error for each pose angle is defined as the absolute difference between the predicted value and the corresponding ground truth:

Error = |θ^pred − θ^gt|

(1)

where θ^pred and θ^gt denote the predicted and ground-truth pose angles, respectively.

The mean absolute error over the evaluation dataset is calculated as:

$$\mathrm{MAE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|{\theta }_i^{\mathit{pred}}-{\theta }_i^{\mathit{gt}}\right|$$

(2)

where N is the number of images used for evaluation.

3.1.3. Implementation Details

The proposed framework consists of mask segmentation, diffusion-based face restoration, and head pose estimation. The implementation details and preprocessing procedures for each stage are described below.

For mask segmentation, ABANet was used with its standard preprocessing procedure. Each input image was resized to 256 × 256 pixels and converted into a tensor before being fed into ABANet. No face cropping or landmark-based alignment was performed. The predicted segmentation map was first normalized to the range of 0–255 and then converted into a binary mask using a threshold of 120. The resulting binary mask was resized to the original image resolution using bilinear interpolation. For masked images, morphological dilation with a 3 × 3 kernel for five iterations was applied to enlarge the region used for inpainting region.

For face restoration, Stable Diffusion Inpainting v1.5 (stable-diffusion-inpainting) [36] was used. The original image and the dilated binary mask were converted to RGB format and resized to 512 × 512 pixels, without additional cropping or alignment. The text prompt was “face of people”, and no negative prompt was used. The PNDMScheduler was employed with 50 denoising steps and a guidance scale of 7.5. The random seed was not fixed. Inference was performed using FP16 precision, with the safety checker disabled.

For head pose estimation, SixDRepNet was used as the landmark-independent estimator. Each input image to SixDRepNet was resized so that its shorter side was 256 pixels and then center-cropped to 224 × 224 pixels. The image was normalized using the standard ImageNet mean and standard deviation. No landmark-based alignment was performed. SixDRepNet outputs a rotation matrix, which was converted into Euler angles representing yaw, pitch, and roll and expressed in degrees for evaluation.

All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU with 24 GB of memory. The software environment consisted of PyTorch 2.8.0+cu128, CUDA 12.8, and Diffusers 0.36.0.

3.2. Comparison with Head Pose Estimation Methods

3.2.1. Comparison with the Landmark-Based Method

We quantitatively compare the proposed method with the landmark-based head pose estimation method [33] using the synthesized masked-face dataset constructed from AFLW2000. The baseline first restores the occluded facial regions using a diffusion model and then estimates head pose by aligning detected facial landmarks with a predefined canonical face model using the Iterative Closest Point (ICP) algorithm [41,42]. In contrast, the proposed method estimates head pose directly from the restored facial images using a landmark-independent deep learning-based model. Therefore, this comparison evaluates the effectiveness of adopting landmark-independent head pose estimation after face restoration. The head pose estimation procedure following face restoration in the landmark-based method does not require model training. In contrast, the proposed method uses the publicly available pretrained SixDRepNet model, which was trained on the 300W-LP dataset [23]. For consistency with the experimental setup recommended for the landmark-based method, 25 restored images were generated for each masked input image for both methods. Both methods were evaluated using the same masked-face test dataset constructed from AFLW2000, evaluation metrics, and hardware environment.

Table 1 summarizes the estimation accuracy under three evaluation conditions: No Occlusion, Mask Occlusion, and Overall. No Occlusion corresponds to the original AFLW2000 images, without synthetic face mask occlusion. Mask Occlusion corresponds to the synthesized masked-face images, where the lower facial region is occluded by a face mask. Under this condition, the occluded facial regions are restored before head pose estimation. Overall reports the average performance across the No Occlusion and Mask Occlusion conditions.

Table 1. Comparison of head pose estimation performance between the landmark-based baseline and the proposed method.

Method	MAE	No Occlusion	Mask Occlusion	Overall
Baseline (Landmark-based)	yaw (°)	3.62	3.95	3.79
	pitch (°)	17.56	16.69	17.13
	roll (°)	3.52	3.63	3.58
	Average (°)	8.23	8.09	8.16
Proposed method (Deep learning-based)	yaw (°)	2.86	4.05	3.46
	pitch (°)	3.54	4.94	4.24
	roll (°)	1.93	2.51	2.22
	Average (°)	2.78	3.83	3.31

As shown in Table 1, the proposed method achieves substantially better overall performance than the landmark-based baseline, particularly for pitch and roll estimation. Under the Mask Occlusion condition, the proposed method achieves MAEs of 4.05°, 4.94°, and 2.51° for yaw, pitch, and roll, respectively. Although its yaw error is slightly higher than that of the landmark-based baseline (3.95°), the proposed method substantially reduces the pitch and roll errors, resulting in a lower Average MAE (3.83° vs. 8.09°). Overall, the proposed method reduces the Average MAE from 8.16° to 3.31°. These results demonstrate the advantage of using landmark-independent head pose estimation after face restoration.

A notable observation is that the landmark-based baseline exhibits relatively large pitch errors under both the No Occlusion (17.56°) and Mask Occlusion (16.69°) conditions. Although the pitch error is slightly lower under the Mask Occlusion condition, the difference is relatively small (0.87°) compared with the large absolute errors observed under both conditions. The consistently high pitch errors suggest that the main limitation is associated with the landmark-based estimation procedure itself rather than being primarily caused by mask occlusion. In this baseline, head pose is estimated by aligning detected facial landmarks with a single reference face using ICP. Therefore, the estimation may be sensitive to inter-subject variations in facial geometry and landmark positions, which may contribute to the large pitch errors and the small difference between the two experimental conditions.

3.2.2. Comparison with the Landmark- Independent Method

To evaluate the proposed framework against existing landmark-independent head pose estimation approaches, we compare it with three representative methods: HopeNet [20], FSA-Net [21], and WHENet [23]. HopeNet and FSA-Net were evaluated using both their publicly available pretrained models and models fine-tuned on the masked-face training data. WHENet was evaluated using the publicly available pretrained model because its training code was not available. The pretrained HopeNet and FSA-Net models were trained on the 300W-LP dataset, while the pretrained WHENet model was trained using 300W-LP together with the CMU Panoptic Dataset. Thus, although the original training conditions of these methods are not completely identical, the major compared models were trained using 300W-LP, which is also the training dataset used for the pretrained SixDRepNet model adopted in the proposed framework. Since the main objective of this study is to evaluate head pose estimation under face mask occlusion, HopeNet and FSA-Net were additionally fine-tuned using masked-face training data constructed from AFLW2000. The same mask synthesis procedure was used to construct the training and test data. This additional fine-tuning reduces the potential influence of differences in adaptation to mask occlusion and provides a more comprehensive comparison.

All compared methods were evaluated using the same masked-face test data, evaluation metrics, and hardware environment. MAE was calculated for yaw, pitch, and roll, and the Average MAE was calculated as the arithmetic mean of the three MAEs. For the proposed method, 25 restored images were generated for each masked input image, and the resulting yaw, pitch, and roll estimates were averaged to obtain the final head pose.

As shown in Table 2, the proposed method achieves the lowest Average MAE of 3.83° among the evaluated methods. Fine-tuning substantially improves the performance of both HopeNet and FSA-Net on the masked-face data; however, their Average MAEs remain 6.07° and 5.13°, respectively. In comparison, the proposed method achieves lower errors for yaw, pitch, and roll, resulting in the best overall performance. The proposed method also substantially outperforms the pretrained HopeNet, FSA-Net, and WHENet models across all three pose angles.

Table 2. Comparison with landmark-independent head pose estimation methods under mask occlusion.

Method	Fine-Tuning	Yaw (°)	Pitch (°)	Roll (°)	Average (°)
HopeNet	No	11.80	7.32	4.93	8.02
HopeNet	Yes	9.58	5.21	3.42	6.07
FSA-Net	No	11.07	7.33	5.04	7.81
FSA-Net	Yes	5.81	5.92	3.66	5.13
WHENet	No	19.25	10.84	7.71	12.60
Proposed method	No	4.05	4.94	2.51	3.83

These results demonstrate that the proposed restore-then-estimate framework achieves better head pose estimation performance under face mask occlusion than the evaluated landmark-independent methods. Notably, the proposed method also outperforms HopeNet and FSA-Net after these models are fine-tuned on the masked-face data, indicating the effectiveness of the proposed framework beyond directly applying or fine-tuning existing landmark-independent estimators on occluded facial images. The specific contribution of diffusion-based face restoration is further evaluated through the controlled experiment presented in Section 3.3.

3.3. Effect of Diffusion-Based Face Restoration

To evaluate the contribution of diffusion-based face restoration to head pose estimation, we conduct a controlled experiment using the same landmark-independent head pose estimator, SixDRepNet, with and without face restoration. In the proposed method, the occluded facial regions are first restored using Stable Diffusion before head pose estimation, whereas the method without restoration directly estimates head pose from the masked facial images. This comparison isolates the effect of diffusion-based face restoration while keeping the head pose estimation model unchanged.

Table 3 summarizes the MAEs under the Mask Occlusion condition. The proposed method with diffusion-based face restoration reduces the estimation errors across all three pose angles. Specifically, the MAE decreases from 6.15° to 4.05° for yaw, from 6.67° to 4.94° for pitch, and from 3.46° to 2.51° for roll. Consequently, the Average MAE is reduced from 5.43° to 3.83°.

Table 3. Effect of diffusion-based face restoration on head pose estimation performance under mask occlusion.

Method	MAE	Mask Occlusion
Proposed method (Diffusion + SixDRepNet)	yaw (°)	4.05
	pitch (°)	4.94
	roll (°)	2.51
	Average (°)	3.83
SixDRepNet without restoration	yaw (°)	6.15
	pitch (°)	6.67
	roll (°)	3.46
	Average (°)	5.43

These results demonstrate that the performance improvement is not solely attributable to the landmark-independent head pose estimator. When the same SixDRepNet estimator is used, restoring the occluded facial regions before pose estimation reduces the Average MAE by 1.60°, corresponding to an improvement of approximately 29.5%. This indicates that diffusion-based face restoration mitigates the information loss caused by mask occlusion and thereby improves subsequent head pose estimation accuracy.

3.4. Qualitative Evaluation

Representative head pose estimation results on the synthesized masked-face dataset constructed from AFLW2000 are shown in Figure 6. The figure compares the proposed method with the landmark-based baseline and three representative landmark-independent methods: HopeNet, FSA-Net, and WHENet. From top to bottom, the rows show the ground truth, landmark-based baseline, fine-tuned HopeNet, fine-tuned FSA-Net, pretrained WHENet, and the proposed method, respectively.

Figure 6. Representative head pose estimation results on the synthesized masked-face dataset constructed from AFLW2000. From top to bottom: ground-truth head pose, landmark-based baseline, fine-tuned HopeNet, fine-tuned FSA-Net, pretrained WHENet, and the proposed method.

As shown in Figure 6, the proposed method produces head pose estimates that are generally close to the ground-truth poses across different head orientations. In several cases, the proposed method provides more accurate pose estimates than the landmark-based baseline, while maintaining performance comparable to or better than the other landmark-independent methods. These qualitative observations are consistent with the quantitative results presented in Section 3.2.1 and Section 3.2.2, where the proposed method achieves the lowest Average MAE among the evaluated methods.

To further examine the applicability of the proposed framework to real-world face mask occlusion, we additionally conducted a qualitative evaluation using real masked-face images randomly selected from the MAFA test set. Unlike AFLW2000, MAFA does not provide continuous ground-truth yaw, pitch, and roll angles. Therefore, quantitative evaluation using MAE cannot be performed, and the MAFA experiment is used only for qualitative evaluation.

As shown in Figure 7, the proposed method provides visually reasonable head pose estimates across different real masked-face images. In several representative cases, particularly those shown in the sixth and seventh columns, the estimated head orientations produced by the proposed method appear more consistent with the observed facial orientations than those produced by the landmark-based baseline. In the third to fifth columns, the proposed method also shows qualitatively comparable or more reasonable head pose estimates than HopeNet, FSA-Net, and WHENet. Since ground-truth head pose angles are unavailable for these images, these results should be interpreted only as a qualitative assessment rather than as a quantitative accuracy comparison.

Figure 7. Qualitative comparison of head pose estimation on real masked-face images from the MAFA dataset. From top to bottom: input image, landmark-based baseline, fine-tuned HopeNet, fine-tuned FSA-Net, pretrained WHENet, and the proposed method.

For both Figure 6 and Figure 7, if a face mask was detected, 25 restored images were generated for each input image for the landmark-based baseline and the proposed method. Head pose was estimated independently from each restored image, and the resulting yaw, pitch, and roll estimates were averaged to obtain the final head pose shown in the figures.

3.5. Sensitivity and Computational Analysis

3.5.1. Number of Restored Images and Computational Cost

To investigate the effect of the number of restored images on head pose estimation accuracy and computational cost, we varied the number of images generated by Stable Diffusion for each masked input image. For each setting, the MAE for yaw, pitch, and roll was calculated, together with the Average MAE and the runtime required for diffusion-based face restoration. The results are summarized in Table 4.

Table 4. Effect of the number of restored images on head pose estimation accuracy and computational cost.

Number of Generated Images	Runtime [s]	MAE	Mask Occlusion
1	2.41	yaw (°)	4.11
		pitch (°)	4.85
		roll (°)	2.50
		Average (°)	3.82
3	7.31	yaw (°)	4.12
		pitch (°)	4.89
		roll (°)	2.49
		Average (°)	3.83
5	12.26	yaw (°)	4.05
		pitch (°)	4.96
		roll (°)	2.49
		Average (°)	3.84
10	24.48	yaw (°)	4.06
		pitch (°)	4.95
		roll (°)	2.54
		Average (°)	3.85
15	36.76	yaw (°)	4.01
		pitch (°)	4.93
		roll (°)	2.51
		Average (°)	3.81
20	49.00	yaw (°)	4.07
		pitch (°)	4.91
		roll (°)	2.51
		Average (°)	3.83
25	61.26	yaw (°)	4.05
		pitch (°)	4.94
		roll (°)	2.51
		Average (°)	3.83

As shown in Table 4, increasing the number of restored images does not lead to a monotonic improvement in head pose estimation accuracy. The Average MAE remains nearly unchanged across the evaluated settings. For example, using a single restored image achieves an Average MAE of 3.82°, while the lowest Average MAE of 3.81° is obtained using 15 restored images. This difference is negligible. In contrast, the computational cost increases substantially as the number of restored images increases.

These results indicate that generating multiple restored images does not provide a meaningful improvement in head pose estimation accuracy, despite substantially increasing the computational cost. Therefore, using a small number of restored images is preferable in practical applications, particularly when computational resources are limited or real-time performance is required.

3.5.2. Mask Detection Threshold

To investigate the sensitivity of mask detection to the threshold T, three threshold values, T = 0.01, 0.04, and 0.10, were evaluated. The true positive rate (TPR) for masked images and the true negative rate (TNR) for non-masked images were used to evaluate the mask detection performance. The results are summarized in Table 5.

Table 5. Sensitivity analysis of the mask detection threshold.

T	Ground Truth	Predicted Mask	Predicted No Mask	Detection Rate (%)
0.01	Mask	1504	5	99.67 (TPR)
	No Mask	1172	337	22.33 (TNR)
0.04	Mask	1488	21	98.61 (TPR)
	No Mask	824	685	45.39 (TNR)
0.1	Mask	626	883	41.48 (TPR)
	No Mask	424	1085	71.90 (TNR)

At T = 0.01, the TPR for masked images is high (99.67%), whereas the TNR for non-masked images is only 22.33%, indicating a high false-positive rate. Increasing the threshold to T = 0.10 improves the TNR to 71.90% but substantially reduces the TPR to 41.48%. At T = 0.04, the method maintains a high TPR of 98.61% while improving the TNR to 45.39% compared with T = 0.01. Therefore, T = 0.04 was selected as a reasonable trade-off between maintaining a high detection rate for masked faces and reducing false-positive detections for non-masked faces.

3.5.3. Stochastic Diffusion Restoration

Since diffusion-based restoration is stochastic, different random seeds may produce variations in the restored facial regions. To evaluate the influence of this stochasticity on head pose estimation, repeated experiments were conducted in which one restored image was generated for each input image in each run, with only the random seed varied across runs. Head pose was estimated using SixDRepNet for each restored image, and the mean and standard deviation of the MAE across different random seeds were calculated for yaw, pitch, and roll, as well as for the Average MAE.

Figure 8 shows representative restoration results generated using different random seeds. Although visual variations can be observed in the reconstructed facial regions, the overall facial structure and head orientation remain relatively consistent. The corresponding head pose estimation results are summarized in Table 6.

Figure 8. Qualitative examples of diffusion-based face restoration using different random seeds. The first and second columns show the original and masked images, respectively, while the remaining columns show restoration results generated using different random seeds.

Table 6. Sensitivity of head pose estimation to stochastic diffusion-based restoration.

	Yaw (°)	Pitch (°)	Roll (°)	Average (°)
Stable Diffusion	4.09 ± 0.086	4.93 ± 0.096	2.52 ± 0.081	3.85 ± 0.083

The relatively small standard deviations across different random seeds indicate that, although Stable Diffusion generates visually different restoration results, the resulting head pose estimation performance remains relatively stable. These results indicate that the stochastic variation introduced by diffusion-based restoration has a limited influence on subsequent head pose estimation performance.

3.6. Discussions

The present study focuses on persistent facial occlusion caused by face masks. Although the proposed framework demonstrates improved head pose estimation under mask occlusion, real-world facial occlusion can also be caused by hands, hair, scarves, eyeglasses, and other objects. Since the current segmentation component is specifically designed to detect mask-induced occlusion, extending the framework to other types of facial occlusion would require adapting or replacing the occlusion detection and segmentation component, as well as evaluating the framework using appropriate datasets with reliable head pose annotations.

In addition, this study employs Stable Diffusion v1.5 as the face restoration model and focuses on the integration of diffusion-based face restoration with landmark-independent head pose estimation rather than on a comparison of different restoration models. Different restoration models may produce different restoration characteristics, which may affect subsequent head pose estimation performance. Therefore, comparing alternative face restoration models and extending the framework to more diverse types of facial occlusion remain important directions for future work.

4. Conclusions

In this study, we proposed a two-stage framework that integrates diffusion-based face restoration with landmark-independent head pose estimation to improve head pose estimation under facial occlusion. Experimental results demonstrated that the proposed method reduced the overall Average MAE from 8.16° to 3.31° compared with the landmark-based baseline. Under mask occlusion, the proposed method achieved an Average MAE of 3.83°, outperforming the evaluated landmark-independent methods, including HopeNet, FSA-Net, and WHENet. Furthermore, the controlled experiment using the same SixDRepNet estimator showed that diffusion-based face restoration reduced the Average MAE from 5.43° to 3.83°, corresponding to an improvement of approximately 29.5%. These results demonstrate that diffusion-based face restoration and landmark-independent head pose estimation both contribute to the improved performance of the proposed framework by mitigating the information loss caused by occlusion and eliminating dependence on facial landmark detection, respectively.

Although the proposed framework demonstrates improved accuracy under face mask occlusion, the present study focuses specifically on mask-induced occlusion, and the current segmentation component is designed accordingly. Future work will extend the framework to other types of facial occlusion, such as hands, hair, scarves, and eyeglasses, by adapting or replacing the occlusion detection and segmentation component. In addition, Stable Diffusion v1.5 was used as the face restoration model in this study. Future work will compare alternative face restoration models to investigate their influence on subsequent head pose estimation performance.