Head-Movement-Robust Micro-Expression Detection Method via 3D Motion Correction and Transformers

Abstract

In micro-expression detection, head movements may seriously interfere with subtle and transient micro-expression signals, significantly limiting detection performance. Therefore, we propose a robust detection framework that integrates 3D motion correction and a Transformer network. This framework adopts a two-stage design. In the first stage, a depth-weighted optical flow method is proposed to decompose and suppress head motion in three-dimensional space and extract anti-interference temporal optical flow features. In the second stage, a Transformer-based encoder is used to model and classify feature sequences by leveraging its capability for global feature modeling integrated with positional information. The experimental results on the CASME3-PartC and MEVIEW datasets show that the proposed method achieves F1-scores of 0.151 and 0.326, respectively, outperforming existing methods and achieving the current optimal performance. Cross-dataset experiments further validated the good generalization ability of this method.

1. Introduction

Facial expressions are a primary non-verbal channel for conveying emotions and psychological states. Research indicates that over half of emotional information is transmitted through facial cues [1]. Based on duration and intensity, facial expressions can be categorized into two types: macro-expressions, which are longer in duration (0.5–4 s) and involve pronounced muscle movements; and micro-expressions, which are extremely brief (≤0.5 s), subtle, and involuntary. Micro-expressions are more indicative of genuine underlying emotions and thus hold significant value in fields such as lie detection and clinical diagnosis [2,3,4].

The analysis of micro-expressions typically involves two key steps: detection and recognition. The detection task aims to determine whether a micro-expression occurs in a continuous video and precisely localize its onset and offset intervals. The recognition task further classifies the localized micro-expression segment to identify its specific emotional category. Among these, micro-expression detection serves as the fundamental prerequisite for all subsequent analysis, posing particularly significant technical challenges. This is mainly due to the intrinsic characteristics of micro-expressions: extremely brief duration and subtle motion amplitude. However, in real-world, naturalistic scenarios, a long-overlooked yet critical challenge is that the subject’s spontaneous, large-amplitude rigid head movements (e.g., rotation, translation) can severely interfere with or even overwhelm the faint non-rigid expression signals, leading to a significant performance degradation for appearance-based methods [5].

To address the challenges in micro-expression detection, existing research has made significant progress. Early work primarily relied on handcrafted features, such as optical flow [6] and Local Binary Patterns on Three Orthogonal Planes (LBP-TOP) [7]. In recent years, deep learning methods have become mainstream. Researchers have employed CNN-based and 3D-CNN-based models, as well as recurrent networks, to learn spatiotemporal features from raw images or optical flow [8,9,10]. Furthermore, recent advances also explore the adaptation of large-scale vision-language foundation models (e.g., CLIP) for facial expression analysis [11]. Beyond per-clip recognition, recent work has also explored end-to-end temporal spotting/detection frameworks for long videos (e.g., two-stream spotting pipelines) and Transformer-style architectures [12]. However, these methods exhibit notable limitations in handling head movement interference: most studies only perform simple face alignment or cropping during preprocessing, failing to fundamentally separate and eliminate global rigid head motion. While some approaches attempt to utilize optical flow, they do not effectively distinguish between global optical flow dominated by head movements and local optical flow dominated by expression changes, resulting in features containing significant motion noise. Consequently, developing robust feature representations that can explicitly suppress head motion interference is key to improving the detection performance of micro-expressions in naturalistic scenarios.

To address the aforementioned issues, this paper proposes a two-stage micro-expression detection framework based on 3D motion correction and Transformer-based temporal modeling. The core of our method is to explicitly eliminate head movement interference during the feature extraction stage. Specifically, we first leverage head pose estimation technology to decompose the rotational component of head movement in 3D space. Building upon this, we introduce a depth-weighting mechanism to further separate the translational component of head movement. Subsequently, temporal smoothing is applied for noise reduction, resulting in clean sequential features focused on subtle facial muscle movements. Finally, the enhanced feature sequence is fed into a Transformer encoder to achieve robust micro-expression detection. Experimental validation on real-world datasets demonstrates that the proposed method effectively overcomes head movement interference, enhancing the accuracy and robustness of micro-expression detection. The main contributions of this work include:

(1)

We propose a 3D head-motion-correction-guided deep-weighted optical flow extraction method, which decomposes and suppresses global rigid head motion while leveraging depth information to enhance local optical flow weighting, thereby obtaining robust temporal representations of micro-expressions.

(2)

We constructed a Transformer-based sequential modeling and classification module. It leverages the model’s strong long-range dependency modeling capability to efficiently encode the preprocessed temporal optical flow features.

(3)

We conducted systematic experiments on real-scene micro-expression datasets with noticeable head movements. The results show that the proposed two-stage hybrid framework effectively overcomes head motion interference. It significantly outperforms existing mainstream methods in detection performance.

2. Related Work

Micro-expression detection methods mainly focus on capturing subtle facial muscle movements from videos. According to feature extraction methods, existing work can be roughly divided into manual feature-based methods, deep learning-based methods, and hybrid methods. However, in natural environments, intense head movements can seriously interfere with capturing subtle facial expressions, becoming a key challenge affecting the robustness of this method. Therefore, this section first reviews the specific research on head motion suppression, and then reviews three mainstream methods separately, analyzing their limitations in dealing with head motion interference.

2.1. Methods for Suppressing Head Movement Interference

Rigid head motion is a primary source of noise in natural-scene micro-expression analysis. Early work predominantly employed rigid registration or affine transformation based on facial landmarks (e.g., nose tip, eye corners) to achieve global alignment and stabilize the facial region. While such methods are effective for minor translation and rotation, they are insufficient for handling large pose variations or non-rigid facial deformations. To model head motion more accurately, researchers have introduced head pose estimation techniques. For instance, Zhang et al. [13] used the nasal region as a stable reference to compute the relative motion of other facial regions, thereby mitigating the impact of head movement. He et al. [14] further incorporated head pose estimation to model and compensate for head motion. While these methods perform holistic correction of head movement in 2D or 3D space, they typically do not explicitly separate it from the non-rigid motion produced by expressions at the feature level, leaving the corrected features potentially contaminated with residual motion noise.

2.2. Micro-Expression Detection Based on Handcrafted Features

Early research primarily relied on handcrafted features to capture subtle facial expression changes. Commonly used features include Local Binary Patterns (LBP) [15], Histogram of Oriented Gradients (HOG) [16], and Local Temporal Patterns [17]. Among these, optical flow has become a mainstream technique due to its ability to effectively represent motion. Related methods include distinguishing expressions using the continuity of optical flow [18], improved spatiotemporal integration approaches [19], and characterizations based on mean directional optical flow [20] or maximum difference in principal direction [21]. Although these handcrafted features are computationally efficient, they are insufficiently robust to the extremely subtle changes of micro-expressions. Moreover, the feature extraction process typically depends on simply aligned images, limiting performance under complex head movements.

2.3. Micro-Expression Detection Based on Deep Learning

In recent years, methods based on deep learning have gradually gained attention. For example, Reddy et al. [11] used a three-dimensional convolutional neural network to extract spatiotemporal features from the original image sequence. Lei et al. [22] used graph temporal convolutional networks to capture local muscle movements based on facial key points. Wei et al. [23] designed an attention-based network to focus on key regions. And Liu et al. [24] constructed an enhanced micro-expression recognition net-attention-based attention mechanism with distance correlation. However, these methods often rely on large-scale annotated data, and the limited size of micro-expression datasets limits the full exploitation of model performance. More importantly, these methods often directly take raw or simply aligned frame sequences as input, expecting the network to implicitly learn how to overcome head motion interference—which is a major challenge in situations where data is scarce.

2.4. Micro-Expression Detection Combining Manual Features and Deep Learning

Given the data scarcity issue, hybrid methods that combine handcrafted features with deep learning have gained attention. One line of work uses features such as LBP-TOP and HOG as the input [25]. Another mainstream approach takes optical flow as network input; for instance, Zhou et al. [26] computed the optical flow between the onset and apex frames and fed it into a dual-branch network for detection. While these methods leverage optical flow’s ability to encode motion, the optical flow used is typically computed from raw videos without in-depth purification specifically for head motion, resulting in input features that contain substantial motion noise.

The work in this paper falls under this hybrid paradigm and focuses on innovating the quality of its front-end feature optical flow. We inherit the idea of using head pose for motion correction [22] and make key advances: First, we propose to decompose and correct head rotation and translation in 3D space; second, we innovatively introduce a depth-weighting mechanism to enhance the corrected optical flow, more precisely highlighting local non-rigid motion. The resulting purified temporal optical flow features are then fed into a Transformer network for efficient modeling. This method aims to explicitly and accurately eliminate head motion interference at the front-end of feature extraction, providing a more robust input for the back-end deep learning model.

3. Methodology

In response to the interference of rigid head movements, this section proposes a complete micro-expression analysis framework from optical flow purification to feature learning. This framework decouples micro-expression movements with clear biomechanical significance from the original observation signals and then uses powerful sequence models to characterize and classify them. This method is based on theoretical decomposition and provides a formal definition of observed optical flow. The observed optical flow of pixel $\mathrm{p}=(\mathrm{x},\mathrm{y})$ in the image plane between adjacent frames is (${\mathsf{\mu }}_{\mathrm{o}\mathrm{b}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{o}\mathrm{b}\mathrm{s}}$), and its physical essence can be decomposed into the following:

$$\left\{\begin{array}{c}{\mathsf{\mu }}_{\mathrm{o}\mathrm{b}\mathrm{s}}={\mathsf{\mu }}_{\mathrm{r}\mathrm{o}\mathrm{t}}+{\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}+{\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}}+{\mathsf{\epsilon }}_{\mathsf{\mu }}\\ {\mathrm{v}}_{\mathrm{o}\mathrm{b}\mathrm{s}}={\mathrm{v}}_{\mathrm{r}\mathrm{o}\mathrm{t}}+{\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}+{\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}+{\mathsf{\epsilon }}_{\mathrm{v}}\end{array}\right.$$

(1)

Among these, ${\mathsf{\mu }}_{\mathrm{r}\mathrm{o}\mathrm{t}}$ and ${\mathrm{v}}_{\mathrm{r}\mathrm{o}\mathrm{t}}$ represent the theoretical optical flow components caused by the six-degree-of-freedom rotational motion of the head; ${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$ and ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$ represent the theoretical optical flow components caused by the translational motion of the head; ${\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$ and ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$ correspond to the optical flow changes generated by facial muscle movements; and ${\mathsf{\epsilon }}_{\mathsf{\mu }}$ and ${\mathsf{\epsilon }}_{\mathrm{v}}$ are the system noise terms.

To extract pure ${\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$ and ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$, this framework consists of two stages. The first stage adopts a hierarchical optical flow purification strategy (corresponding to Section 3.1, Section 3.3 and Section 3.4.1), which gradually removes physical interference in the video through progressive decoupling of rotation components, suppression of translation components, and noise attenuation operations, and finally outputs a purified micro-expression optical flow sequence. The second stage focuses on feature learning and classification (corresponding to Section 3.4.2), inputting the above optical flow sequence into a Transformer-based encoder BERT architecture for deep sequence modeling and classification tasks. The entire process is shown in Figure 1.

3.1. Head Pose Estimation Based on Depth Compensation

The stable and accurate estimation of head poses forms the foundation of the proposed micro-expression analysis framework. To effectively address the interference in micro-expression analysis caused by apparent image changes due to head rotation, the proposed method first performs real-time estimation of a robust 6-DoF head pose from the input video.

As shown in the upper part of Figure 1, the pose estimation network adopts a dual-branch (depth/RGB) architecture, extracting features from depth and RGB inputs separately. The extracted features are fused and then processed by a specialized module that employs Gram–Schmidt orthogonalization to directly regress the valid rotation matrix and translation vector. To provide this network with a stable and consistent input, the system first applies Gunnar Farnebäck’s polynomial expansion algorithm during the image preprocessing stage to compute a dense, per-pixel optical flow field between the reference frame and subsequent frames within a standardized region of interest (ROI). This algorithm assumes that the pixel motion in each small local region of the image can be approximated by a second-order polynomial, and the underlying local motion consistency assumption can be expressed as follows:

$${\mathrm{I}}_{\mathrm{x}}\mathrm{u}+{\mathrm{I}}_{\mathrm{y}}\mathrm{v}+{\mathrm{I}}_{\mathrm{t}}=0$$

(2)

This is a brightness constancy constraint, which is inherently satisfied by standard optical flow computation. However, in scenarios involving large-angle head rotations, this constraint significantly breaks down. Under such conditions, the high-amplitude motion caused by rotation completely overwhelms the weak signals of micro-expressions, causing the objective function in optical flow optimization to be dominated by the rotational component. As a result, the final output optical flow vectors fail to reflect the true local muscle movements [27].

To address the aforementioned issue, the system integrates the Deep6DHead model proposed in our previous work [28], which employs a depth-feature-enhanced, end-to-end approach to directly regress robust head pose. The core of this method lies in fusing RGB and depth information and incorporating anatomical constraints to overcome estimation biases caused by occlusion, illumination variations, and extreme angles, thereby providing stable head pose input for subsequent processing. As illustrated in Figure 1, its framework primarily comprises the following three innovations:

(1)

RGB-D Dual-Branch Feature Fusion: This module constructs a four-channel RGB-D input and adopts a dual-branch network architecture. The RGB branch uses RepVGG as the backbone to extract visual appearance features, while the depth branch employs a lightweight convolutional network to extract geometric features robust to illumination variations. To address multimodal feature alignment, the network incorporates a squeeze-and-excitation channel-attention mechanism in both branches. This mechanism adaptively enhances the feature responses of key anatomical regions that are highly discriminative for pose estimation (e.g., nasal bridge, eye sockets), achieving depth-guided intelligent feature fusion.

(2)

6D Rotation Representation and End-to-End Regression: The model adopts a continuous and unambiguous 6D rotation representation. The fused multimodal features are passed through fully connected layers to directly regress this 6D representation, and a Gram–Schmidt orthogonalization process is used to recover a strictly orthogonal rotation matrix. This design eliminates the reliance on intermediate facial landmark detection common in traditional methods, achieving efficient end-to-end regression from input to pose.

(3)

Nasal Bridge Normal Anatomical Constraint Loss: To enhance the geometric plausibility of rotation estimation, an anatomically driven orientation–constraint loss is introduced. This loss term leverages the relatively stable geometric prior of the human nasal bridge region under pose variations. By comparing the angle between the actual nasal bridge normal vector estimated from the depth map and the ideal normal vector obtained by transforming via the predicted rotation matrix, a constraint term ${\mathrm{L}}_{\mathrm{n}\mathrm{o}\mathrm{s}\mathrm{e}}$ is constructed. It is then weighted and combined with the standard geodesic loss ${\mathrm{L}}_{\mathrm{g}\mathrm{e}\mathrm{o}}$ to form the final rotation loss ${\mathrm{L}}_{\mathrm{r}\mathrm{o}\mathrm{t}}={\mathrm{L}}_{\mathrm{n}\mathrm{o}\mathrm{s}\mathrm{e}}+{\mathrm{L}}_{\mathrm{g}\mathrm{e}\mathrm{o}}$. This loss function forces the network’s predicted rotation directions to align with the head’s local anatomical structure, effectively suppressing pose deviations caused by noise or occlusion.

Experimental validation on multiple public datasets, including 300W-LP, BIWI, and AFLW2000, demonstrates that the Deep6DHead approach employed in this work achieves state-of-the-art performance on the roll axis (lowest error of 2.05°) with an overall mean absolute error of 3.45° [28]. By integrating this pose estimation method, the system can effectively decouple rigid rotational head motion, providing an accurate motion reference for subsequent micro-expression optical flow analysis.

3.2. Video Framing Strategy Based on Head Motion Dynamics

High-precision head pose estimation lays the foundation for subsequent analysis, while stable video segment partitioning is a crucial prerequisite for preventing micro-expression signals from being overwhelmed by motion interference. To this end, the system designs a frame-splitting strategy based on the kinematic characteristics of head motion, the input of which is directly derived from the robust 6-DoF pose parameters estimated by the Deep6DHead model in Section 3.1. The upper part of Figure 2 illustrates the complete workflow of this frame-splitting strategy, which primarily includes the calculation of head pose angular change rates, marking of steady-state candidate frames, sliding-window clustering, and non-continuity compensation as key steps.

The core workflow of the frame-splitting strategy is as follows: the system first calculates the rate of change of the head pose angles (yaw, pitch, roll) in real time. When this rate of change remains continuously below a preset threshold, it indicates that the head is in a relatively stable state, and the system marks the current frame as a “steady-state candidate frame.” Subsequently, a sliding-window mechanism is applied to dynamically cluster these consecutive candidate frames. The window length is configured as a temporal parameter T that is synchronized with the video frame rate, thereby generating stable video segments that exhibit spatiotemporal consistency.

To ensure robustness in scenarios disrupted by sudden head motions, the system incorporates a pose mutation detection mechanism. Once a sudden change in head pose is detected, the accumulation of the current window is terminated immediately, and a backtracking mechanism based on the historical buffer is activated to extract stable segments that satisfy the maximum effective duration.

Crucially, this strategy introduces a discontinuity compensation feature: when outputting a segmentation segment, the system actively steps back a fixed-length time window along the time axis in the negative direction (the length is set to 0.3 s based on the average duration of micro-expressions and converted to the corresponding number of frames according to the video frame rate). This mechanism effectively covers the risk of segmentation-point truncation caused by sudden head movements, ensuring that potential micro-expressions that happen to straddle the segmentation boundary are entirely preserved within a single continuous analysis segment, thereby fundamentally preventing the loss of micro-expression information across segments.

3.3. Depth-Weighted Optical Flow Compensation

Most current studies estimate global motion parameters based on optical flow in local regions such as the nose tip [29]. This approach inherently assumes that the face approximates a planar surface, making it difficult to accurately eliminate the optical flow components (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) induced by head translation when performing global motion compensation. The fundamental reason is that under a perspective projection model, the same head translation induces significantly different optical flows in the image plane across facial regions with varying depths. To accurately model the coupling relationship between translational motion and the optical flow field, depth information must be introduced.

In a monocular vision observation system, the observed optical flow (${\mathsf{\mu }}_{\mathrm{o}\mathrm{b}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{o}\mathrm{b}\mathrm{s}}$) at pixel point $\mathrm{p}(\mathrm{x},\mathrm{y})$ includes components induced by head translation. To separate them, it is first necessary to acquire the depth information $\mathrm{d}(\mathrm{x},\mathrm{y})$ corresponding to that pixel. Based on the depth map, the 3D coordinates of the pixel point in the camera coordinate system can be expressed as follows:

$$\mathbf{P}=\mathrm{d}(\mathrm{x},\mathrm{y}){\mathbf{K}}^{-1}\tilde{\mathbf{p}}$$

(3)

where K is the camera intrinsic matrix and $\tilde{\mathbf{p}}$ is the homogeneous coordinate of the pixel point. Combined with the translation vector t obtained from head pose estimation, the theoretical optical flow field caused solely by this head translation can be accurately derived through projective geometric relations:

$${\left(\begin{array}{c}{\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}\\ {\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}\end{array}\right)}^{\mathrm{\top }}=\mathsf{\pi }(\mathbf{K}[\mathbf{R}|\mathbf{t}]\mathbf{P})-\mathsf{\pi }(\mathbf{K}\mathbf{P})$$

(4)

where [$\mathbf{R}|\mathbf{t}$] is the rigid body transformation matrix and π (·) is the projection operator. This model achieves a physically accurate mapping from 3D translational motion to a 2D optical flow field.

However, uniformly subtracting (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) across the entire field introduces problems. In near-face regions (e.g., nasolabial folds, where d is small), subtle muscle movements (${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$, ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$) are prone to over-compensation, whereas in distant regions (e.g., cheek edges, where d is large), translational artifacts may be insufficiently suppressed. To address this, this study proposes a depth-weighted compensation function W(d) to achieve adaptive compensation:

$$\mathbf{W}(\mathrm{d})=\left[\begin{array}{cc}1-\mathsf{\alpha }{\mathrm{e}}^{-\mathsf{\gamma }\mathrm{d}}& 0\\ 0& 1-\mathsf{\beta }{\mathrm{e}}^{-\mathsf{\gamma }\mathrm{d}}\end{array}\right]$$

(5)

where $\mathsf{\alpha }=0.78$, $\mathsf{\beta }=0.83$ are compensation-intensity control parameters and γ is the depth attenuation coefficient [30]. The physical meaning of this weight is as follows: in near-camera regions (where d is small), ${\mathrm{e}}^{-\mathsf{\gamma }\mathrm{d}}\to 1$, causing the weight matrix element values to approach $(1-\mathsf{\alpha })$ or $\left(1-\mathsf{\beta }\right),$ thereby reducing the compensation strength applied to the theoretical translation optical flow (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) to protect micro-expression details in this region. In distant-camera regions (where d is large), ${\mathrm{e}}^{-\mathsf{\gamma }\mathrm{d}}\to 0$, causing the weight matrix element values to approach 1, thereby enhancing the compensation strength applied to the theoretical translation optical flow (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) to sufficiently suppress motion artifacts caused by head translation.

Finally, the elimination of the translation component (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) and the adaptive purification are achieved through the following operation:

$${\left(\begin{array}{c}{\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}}^{\prime }\\ {\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}^{\prime }\end{array}\right)}^{\mathrm{\top }}=\mathbf{W}(\mathrm{d})\odot \left[\begin{array}{c}{\left(\begin{array}{c}{\mathsf{\mu }}_{\mathrm{o}\mathrm{b}\mathrm{s}}\\ {\mathrm{v}}_{\mathrm{o}\mathrm{b}\mathrm{s}}\end{array}\right)}^{\mathrm{\top }}-{\left(\begin{array}{c}{\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}\\ {\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}\end{array}\right)}^{\mathrm{\top }}\end{array}\right]$$

(6)

where ⊙ denotes the Hadamard product. The core of this operation is to spatially modulate the theoretically derived translational optical flow field (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) using the depth-weighted matrix $\mathbf{W}\left(\mathrm{d}\right)$, and then subtract the result from the observed optical flow. By incorporating depth information, this method constructs a coupling model between rigid body translational motion and the optical flow field, achieving non-uniform and precise elimination of facial translation components, thereby laying the foundation for subsequent extraction of pure expression signals. The lower part of Figure 2 illustrates the workflow of the depth-weighted optical flow compensation described in this section, including 3D coordinate computation based on the depth map, derivation of the theoretical translational optical flow field, application of the depth-weighted compensation function, and the adaptive elimination of translation components.

3.4. Micro-Expression Detection Based on Transformer

3.4.1. Feature Preprocessing and Sequence Construction

Building upon the purified features obtained through the progressive processing in the preceding chapters, this section designs a BERT-based Transformer encoder architecture [31] for micro-expression detection, as illustrated in Figure 3.

For clarity, the logical order is: upstream removal of rotational and translational head motion (Section 3.1, Section 3.2 and Section 3.3), followed by the four numbered steps below—ROI definition and 3 × 3 spatial pooling, motion–magnitude summarization per grid, concatenation into a per-frame vector, and temporal smoothing with subject-level normalization—before the sequence enters the Transformer encoder.

Progressive elimination of interference components: The features input to this module underwent key interference stripping through the preceding processing steps:

• Rotation component elimination (Section 3.1): Through head pose estimation, the theoretical optical flow components ${\mathsf{\mu }}_{\mathrm{r}\mathrm{o}\mathrm{t}}$, ${\mathrm{v}}_{\mathrm{r}\mathrm{o}\mathrm{t}}$ induced by head rotational motion are calculated and subtracted.
• Translation component elimination (Section 3.3): Through the depth-weighted compensation model, the theoretical optical flow components (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) caused by head translational motion are adaptively eliminated. It is important to note that the core reference baseline for this compensation model is derived from the depth and optical flow information of the nose tip and its surrounding area. Therefore, in subsequent feature extraction, this nasal region is excluded to avoid re-inputting signals that have already been used in the compensation calculation as recognition features, thereby ensuring the independence and validity of the feature sources.

After the above two steps, the primary physical interference in the observed optical flow has been removed. The remaining signal mainly consists of the target expression optical flow (${\mathsf{\mu }}_{\mathrm{e}\mathrm{x}\mathrm{p}}$, ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$) and system noise (${\mathsf{\epsilon }}_{\mathsf{\mu }}$, ${\mathsf{\epsilon }}_{\mathrm{v}}$). From this point onward, processing shifts from global motion compensation to region-wise feature construction on (${\mathsf{\mu }}_{\mathrm{e}\mathrm{x}\mathrm{p}}$, ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$).

Spatial Feature Extraction and Regularization: To extract effective, biomechanically meaningful representations from the purified optical flow field and to standardize the data, the following operations are performed:

Key Region Delineation and Spatial Pooling: Based on the anatomical distribution of facial muscle action units, and after excluding the nasal reference region, 44 key analysis regions are delineated in the remaining facial area, each with an initial size of 21 × 21 pixels. To reduce spatial redundancy and enhance the robustness of local features, a non-overlapping 3 × 3 spatial pooling operation is applied to each ROI, down-sampling it to a 7 × 7 spatial grid. At this stage, each grid represents aggregated information from a 3 × 3 pixel block in the original ROI. In plain terms, this step fixes where to measure motion and reduces each ROI to a compact spatial grid.

Motion Magnitude Feature Calculation: For each of the aforementioned 7 × 7 spatial grids, the mean magnitude of expression optical flow across all pixel points within the grid is computed as its scalar feature. Specifically, for a pixel point included in grid G, its optical flow magnitude is $\mathrm{m}=\sqrt{{\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}}^2+{\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}^2}$; then, the feature value of the grid is ${\mathrm{f}}_{\mathrm{G}}={\displaystyle \frac{1}{\left|\mathrm{G}\right|}}\sum _{\mathrm{p}\in \mathrm{G}} {\mathrm{m}}_{\mathrm{p}}$. Thus, each 21 × 21 ROI is transformed into a 7 × 7 feature map, which is then flattened into a 49-dimensional feature vector, representing the spatial distribution pattern of optical flow motion intensity within that ROI. In plain terms, each grid cell is summarized by one motion strength value, and the cells are ordered into a 49-D vector for that ROI.

Feature Concatenation and Serialization: The 49-dimensional feature vectors corresponding to the 44 ROIs are concatenated, yielding a global representation vector ${\mathbf{f}}_{\mathrm{t}}\in {\mathbb{R}}^{2156}$ (44 × 49 = 2156 dimensions) for a single frame. In plain terms, all regional vectors are stacked side by side into a single frame-level descriptor.

Temporal Averaging, Down-Sampling, and Normalization: To smooth transient noise and unify the temporal scale, the feature sequence is non-overlappingly averaged using a five-frame window, producing aggregated features ${\mathrm{e}}_{\mathrm{k}}$. Subsequently, global Z-score normalization is applied to the aggregated features of all video clips at the subject level to eliminate inter-individual baseline differences, yielding a standardized sequence $\widehat{\mathbf{E}}=\{{\widehat{\mathbf{e}}}_1,{\widehat{\mathbf{e}}}_2,\dots ,{\widehat{\mathbf{e}}}_{\mathrm{T}}\}$, where T is the sequence length.

Finally, each video clip is represented as a normalized, uniformly dimensional high-dimensional spatiotemporal feature sequence $\widehat{\mathbf{E}}\in {\mathbb{R}}^{\mathrm{T}\times 2156}$. This sequence accurately encodes the intensity evolution information of non-rigid facial micro-expression movements across 44 key regions, each with 49 spatial positions, providing high-quality input for the subsequent Transformer model to perform long-range temporal dependency modeling. Its generation process can be formally summarized as follows:

$$\widehat{\mathbf{E}}=\mathrm{SubjectNorm}\left({\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}}_{5-\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}}\right(\mathrm{F}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\left({\mathrm{SpatialPool}}_{3\times 3}\right({\mathrm{R}\mathrm{O}\mathrm{I}}_{44}({\mathrm{u}}_{\mathrm{e}\mathrm{x}\mathrm{p}},{\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}})\left)\right)\left)\right)$$

(7)

3.4.2. Reasoning and Post-Processing Flow

During the inference stage, the model employs a BERT-based Transformer encoder architecture to process the temporal feature sequence. Specifically, the model takes the preprocessed temporal feature sequence (each sample is a 2156-dimensional feature vector, corresponding to a flattened 44 × 49 feature matrix) as an input, and uses an embedding layer to map the input dimension to the model’s internal dimension (${\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}=1024$). To preserve the temporal order information of the input features, the model adds a learnable positional encoding after the input embedding. This positional encoding is a learnable parameter matrix of the same length as the input sequence, and it is added element-wise to the embedded feature vectors, assigning a unique positional identifier to each position in the sequence.

In the self-attention mechanism of the Transformer encoder, each position in the sequence interacts via three vectors: Query (Q), Key (K), and Value (V). The Query vector is used to “query” information from other positions in the sequence, the Key vector is used to “match” the query request, and the Value vector carries the actual feature information. By computing the similarity matrix between Q and K and normalizing it into attention weights, the model can adaptively focus on other time steps in the sequence that are relevant to the current window, thereby capturing the temporal dependencies of micro-expressions. After feature extraction and information fusion through 12 layers of Transformer encoders, the model finally outputs the detection probability for each five-frame aggregated window and a global class probability distribution representing the entire segment.

To obtain the final micro-expression detection and classification results, a systematic post-processing pipeline is executed. First, the window-level label is determined based on the number of frames containing genuine expressions (micro- or macro-expressions) within each five-frame window: if the window contains at least three frames of genuine expressions, it is labeled as a positive sample (label 1); otherwise, it is labeled as a negative sample (label 0). This rule is designed to ensure robustness against transient noise. Subsequently, the positive windows output by the model are merged into several candidate expression intervals. For an objective evaluation of model performance, the Intersection over Union (IoU) is used as the criterion to determine whether a detection is successful [32,33]: a detection is considered correct only if the IoU value between the detected interval and the ground-truth expression interval on the time axis exceeds 0.5. Finally, based on the matching between all ground-truth intervals and detected intervals, precision, recall, and their harmonic mean (F1-score) are calculated, serving as the core metrics for measuring end-to-end detection performance.

This pipeline accomplishes a complete mapping from raw video to structured detection results. The Transformer architecture, through its multi-head self-attention mechanism, enables information interaction between any positions in the sequence, allowing the model to simultaneously focus on both local window features and global temporal patterns. Consequently, while avoiding error accumulation common in traditional cascade methods, the framework ensures robust and objective analysis of brief facial expression movements by employing clear window-label definitions and a strict IoU evaluation criterion.

4. Experiment

4.1. Dataset

The MEVIEW [34] database was released in 2017, and its video clips were collected from real scenes in non-laboratory environments, covering two types of triggering scenarios: card games and television interviews. This database contains a total of 40 videos, involving 39 micro-expression samples. It should be noted that some videos do not include micro-expressions, while others contain multiple micro-expressions. Each sample was labeled with seven emotion categories. A total of 16 participants were included in this database study. The video resolution is 1280 × 720 and the frame rate is 30 frames per second.

The CAS (ME)³ database [35] adopts the third-generation micro-expression induction paradigm to induce more ecologically valid micro-expressions by simulating crime scenes. This database contains 31 induction topics. During the simulated communication phase, each participant is questioned and the entire process is recorded for approximately 8 min. The video resolution is 1280 × 720 and the frame rate is 30 frames per second.

4.2. Experimental Setup

All videos were first processed using the Deep6DHead model [28] for head pose estimation, achieving inter-frame alignment and smoothing to generate stabilized face image sequences. Subsequently, the system employed a head-motion-dynamics-based frame-splitting strategy to segment the continuous video stream into candidate clips with stable spatiotemporal features.

For each candidate clip, the system first computed a per-pixel dense optical flow field using the Gunnar Farnebäck algorithm, followed by a progressive interference-elimination pipeline: the rotational component (${\mathsf{\mu }}_{\mathrm{r}\mathrm{o}\mathrm{t}}$, ${\mathrm{v}}_{\mathrm{r}\mathrm{o}\mathrm{t}}$) was directly calculated and removed via head pose estimation, and then the translational component (${\mathsf{\mu }}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$, ${\mathrm{v}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}}$) was adaptively eliminated using the depth-weighted compensation model (Section 3.3). Finally, a five-frame moving average and temporal normalization were applied to further reduce system noise, yielding a purified expression optical flow representation (${\mathsf{\mu }}_{\mathrm{e}\mathrm{x}\mathrm{p}}$, ${\mathrm{v}}_{\mathrm{e}\mathrm{x}\mathrm{p}}$).

Based on the purified optical flow field, the system delineated 44 key analysis regions (ROIs) in the remaining facial area after excluding the nasal reference region, following the anatomical distribution of facial muscle action units, with each region measuring 21 × 21 pixels. A non-overlapping 3 × 3 spatial pooling operation was applied to each ROI, down-sampling it to a 7 × 7 spatial grid, and the mean optical flow magnitude of all pixel points within the grid was calculated as the scalar feature. The feature vectors corresponding to the 44 ROIs were concatenated to obtain a global representation vector for a single frame. Finally, the feature sequence was non-overlappingly averaged using a five-frame window and normalized, yielding a standardized temporal feature sequence to serve as an input to the Transformer encoder.

The experiments were conducted on a server equipped with one Intel Xeon Platinum 8358 CPU and seven NVIDIA A40 GPUs. The Transformer encoder employed a 12-layer architecture with eight attention heads and a hidden dimension of 2048. The model was trained using the Adam optimizer [36], with the loss function set as Focal Loss [37]. Training was performed for 32 epochs, and a Leave-One-Subject-Out (LOSO) cross-validation strategy was adopted for intra-dataset performance evaluation.

Metric aggregation (LOSO): Precision, Recall, and F1 are each computed on the held-out subject per fold and then macro-averaged across folds (mean over subjects). Thus, the reported F1 is the mean of per-fold F1 scores (each fold uses the standard harmonic mean of that fold’s Precision and Recall), whereas the reported Precision and Recall are the means of the per-fold Precisions and Recalls. In general, $\mathrm{F}1=2\mathrm{P}\mathrm{R}/(\mathrm{P}+\mathrm{R})$, where P and R denote the reported macro-averages; the identity holds only in special cases (e.g., when per-fold metrics are nearly identical).

4.3. LOSO Cross-Validation Experiment

In the task of micro-expression recognition, to comprehensively evaluate the generalization ability and robustness of this method, we conducted experiments on two datasets: natural environment and controlled laboratory environment. All experiments adopted a one-subject cross-validation strategy. In each fold experiment, the training set and validation set are composed of samples from all subjects except one, and the same positive and negative sample sampling strategy is used (all positive samples and three times the number of negative samples). The test set includes all the samples of the subjects that have been left aside. The optimal learning rate is determined through grid search.

Natural environment dataset: This type of dataset is collected from unconstrained real-world scenarios and requires a higher generalization ability of algorithms. MEVIEW includes 40 participants. The optimal learning rate is 10⁻⁷. As shown in

Table 1. Comparison of experimental results on the MEVIEW dataset.

Method	Recall	Precision	F1-Score
Yang et al. [27]	0.230	0.321	0.268
Ours	0.304	0.351	0.326

, this method achieved an F1-score of 0.326, reaching the state-of-the-art level on this dataset. CASME3 (Part C) includes 31 subjects. The optimal learning rate is 5 × 10⁻⁶. As shown in

Table 2. Comparison of experimental results on the CASME3-PartC dataset.

Method	Recall	Precision	F1-Score
Yang et al. [27]	0.250	0.080	0.121
Ours	0.861	0.083	0.151

, this method achieved an F1-score of 0.151, reaching the state-of-the-art level on this dataset.

Laboratory environment dataset: This type of dataset is collected under controlled conditions and is a classic benchmark in the field of micro-expression research. CAS (ME) 2 [38] includes 22 subjects. The optimal learning rate is 10⁻⁸. As shown in

Table 3. Comparison of experimental results on the CAS (ME) 2 dataset.

Method	Recall	Precision	F1-score
MTSN (2022) [40]	0.342	0.385	0.362
Zhao et al. (2022) [41]	-	-	0.403
LGSNet (2023) [15]	0.367	0.630	0.464
Ours	0.405	0.563	0.471

, this method achieved an F1-score of 0.471, surpassing all comparison methods. SAMM-LV [39] includes 30 subjects. The optimal learning rate is 10⁻⁵. As shown in

Table 4. Comparison of experimental results on SAMM-LV dataset.

Method	Recall	Precision	F1-score
MTSN (2022) [40]	0.260	0.319	0.287
Zhao et al. (2022) [41]	-	-	0.386
LGSNet (2023) [15]	0.355	0.429	0.388
Ours	0.450	0.354	0.396

, this method achieved an F1-score of 0.396, surpassing all comparison methods.

4.4. Ablation Experiment

4.4.1. Deep Information Utilization Ablation Experiment

To investigate the impact of depth information on model performance, we compared two settings: using depth information and not using depth information. The settings for using depth information include: head pose estimation based on depth compensation (Section 3.1) and depth-weighted optical flow compensation (Section 3.3). The setting that does not use depth information adopts traditional methods, which only rely on RGB images for head pose estimation and optical flow compensation.

The depth information on the MEVIEW dataset was obtained through ablation experiments, and the results are shown in

Table 5. Experimental results of ablation using depth information on the MEVIEW dataset.

Method	Recall	Precision	F1-Score
Without depth information	0.259	0.168	0.175
With depth information	0.304	0.351	0.326

. When using depth information, the F1-score reached 0.362, significantly better than those without depth information.

The depth information on the CASME3-PartC dataset was obtained using ablation experiments, and the results are shown in

Table 6. Experimental results of ablation using depth information on the CASME3-PartC dataset.

Method	Recall	Precision	F1-Score
Without depth information	0.277	0.006	0.087
With depth information	0.861	0.083	0.151

. Similarly, when using depth information, the F1-score reached 0.151, which significantly outperforms those which did not use depth information.

Removing depth affects the pipeline in two concrete ways: (i) pose and alignment become less stable under illumination changes and partial occlusions (depth provides geometry that is less sensitive to appearance), which increases residual rigid-motion flow; and (ii) the depth-weighted translational suppression in Section 3.3 degenerates to a less adaptive compensation, so some large amplitude head translation leaks into the residual and can dominate the weak non-rigid expression flow. In practice, these failures manifest as more false positives on sudden head movements and missed detections when subtle events are masked by residual global motion.

4.4.2. Loss-Function-Type Ablation Experiment

To investigate the impact of different loss functions on model performance, we compared two loss functions: Cross-Entropy Loss and Focal Loss. The experimental results show that Focal Loss achieves the best performance on both datasets.

The ablation results for the loss function on the MEVIEW dataset are shown in

Table 7. Experimental results of loss-function-type ablation on the MEVIEW dataset.

Loss Function	Recall	Precision	F1-Score
Cross-Entropy Loss	0.104	0.300	0.287
Focal Loss	0.304	0.351	0.326

. When using Focal Loss, an F1-score of 0.326 was reached, which was significantly better than that obtained with Cross-Entropy Loss.

The ablation results for the loss function on the CASME3_PartC dataset are shown in

Table 8. Experimental results of loss-function-type ablation on the CASME3-PartC dataset.

Loss Function	Recall	Precision	F1-Score
Cross-Entropy Loss	0.651	0.035	0.111
Focal Loss	0.861	0.083	0.151

. Similarly, when using Focal Loss, an F1-score of 0.151 was achieved, representing the best performance.

Table 7 and Table 8 compare the default training objective with a cross-entropy baseline. Here, cross-entropy means binary cross-entropy (BCE) on the window level detection logits (positive vs. negative). Let $y_i\in \left\{\mathrm{0,1}\right\}$ be the label and $\widehat{{\mathrm{p}}_{\mathrm{i}}}\in (\mathrm{0,1})$ be the predicted probability of the positive class for the $\mathrm{i}-\mathrm{t}\mathrm{h}$ training sample among N samples. The baseline minimizes as follows:

$${\mathcal{L}}_{\mathrm{C}\mathrm{E}}=-{\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}} \left[{\mathrm{y}}_{\mathrm{i}}\mathrm{l}\mathrm{o}\mathrm{g}{\widehat{\mathrm{p}}}_{\mathrm{i}}+(1-{\mathrm{y}}_{\mathrm{i}})\mathrm{l}\mathrm{o}\mathrm{g}(1-{\widehat{\mathrm{p}}}_{\mathrm{i}})\right]$$

(8)

which is the standard Bernoulli negative log-likelihood and is equivalent to two-class SoftMax cross-entropy with a single sigmoid output. Focal Loss uses the same sigmoid parameterization but reweights each term as in Lin et al. to down-weight easy negatives, with the focusing parameter γ = 2 and inverse class frequency α = 0.25. Empirically, Focal Loss outperforms this BCE baseline on both in-the-wild sets.

4.4.3. Time-Series Window-Size Ablation Experiment

To investigate the impact of the time-averaged down-sampling of the window size on model performance, we compared three window settings: three frames, five frames, and eight frames. The experimental results indicate that the model achieves optimal performance on both datasets when the window size is five frames.

The experimental results of temporal window size ablation on the MEVIEW dataset are shown in

Table 9. Experimental results of temporal window-size ablation on the MEVIEW dataset.

Window Size	Recall	Precision	F1-Score
3 frames	0.197	0.314	0.279
5 frames	0.304	0.351	0.326
8 frames	0.270	0.293	0.306

. When the window size is five frames, the F1-score reaches 0.326, significantly better than the settings of three frames and eight frames.

The experimental results of temporal window-size ablation on the CASME3-PartC dataset are shown in

Table 10. Experimental results of temporal window-size ablation on the CASME3-PartC dataset.

Window Size	Recall	Precision	F1-Score
3 frames	0.357	0.069	0.139
5 frames	0.861	0.083	0.151
8 frames	0.755	0.063	0.146

. Similarly, when the window size is five frames, the F1-score reaches 0.151, indicating optimal performance.

4.5. Generalization Ability Verification

To validate the cross-dataset generalization capability of the framework, cross-dataset experiments were conducted. First, the model was trained on the MEVIEW dataset and tested on the CASME3_PartC dataset; then, it was trained on the CASME3_PartC dataset and tested on the MEVIEW dataset.

The experimental results show that when trained on MEVIEW and tested on CASME3_PartC, an F1-score of 0.134 is achieved; when trained on CASME3_PartC and tested on MEVIEW, an F1-score of 0.265 is obtained. Although the cross-dataset performance is lower than that in the within-dataset experiments, the results indicate that the proposed method still possesses a certain degree of cross-dataset generalization ability. This is primarily attributed to the effective suppression of head-movement interference by the depth-weighted optical flow compensation mechanism and the robust temporal feature modeling capability of the Transformer architecture.

5. Discussion

Following the Facial Action Coding System (FACS), Action Unit 9 (AU9, nose wrinkle) is relevant to expressions such as disgust. Our pipeline excludes a nasal reference region from the 44-analysis ROIs because depth and optical flow around the nose tip are already used to estimate the translational compensation baseline (Section 3.3), and reusing the same pixels as downstream features would couple calibration and recognition. (i) We acknowledge that if an AU9 dominant micro-expression is highly localized and concentrated mainly inside the excluded mask, sensitivity to that event could be reduced relative to a design that pools AU9 pixels directly. (ii) Future work could mitigate this tradeoff by using alternative compensation anchors that do not overlap with AU9 (e.g., non-AU9 stable references), or by adding an AU9-aware branch or dedicated ROI set, at the cost of a more complex system and additional validation.

6. Conclusions

To address the challenge of detecting weak and transient micro-expressions in natural scenes complicated by head movements, this paper proposes a robust detection framework that integrates 3D motion correction with a Transformer network. The method effectively suppresses interference by decomposing and compensating for head movements in 3D space via depth-weighted optical flow, and leverages the Transformer network to model the purified temporal features. The framework ultimately achieves state-of-the-art detection performance on the MEVIEW and CASME3_PartC datasets, significantly outperforming previous work from this research team, thus validating its effectiveness in complex real-world scenarios. Looking ahead, future work could be further advanced in multiple directions, such as multi-modal fusion, end-to-end optimization, improved cross-domain generalization, fine-grained attribute analysis, and more refined motion modeling, to drive the field toward more practical and robust applications.