Micro-Expression Spotting Based on a Short-Duration Prior and Multi-Stage Feature Extraction

Abstract

When micro-expressions are mixed with normal or macro-expressions, it becomes increasingly challenging to spot them in long videos. Aiming at the specific time prior of micro-expressions (MEs), an ME spotting network called AEM-Net (adaptive enhanced ME detection network) is proposed. This paper is an extension of the conference paper presented at the Chinese Conference on Biometric Recognition (CCBR). The network improves spotting performance in the following five aspects. Firstly, a multi-stage channel feature extraction module is constructed to extract the features at different depths. Then, an attention spatial-temporal module is leveraged to obtain salient and discriminative micro-expression segments while suppressing the generation of excessively long or short suggestions. Thirdly, a ME-NMS (non-maximum suppression) network is developed to reduce redundancy and decision errors. Fourthly, a multi-scale feature fusion module is introduced to fuse up-sampling features of high-level maps and fine-grained information, which obtains meaningful information on feature distribution and contributes to a good representation of MEs. Finally, two spotting mechanisms named anchor-based and anchor free were integrated to get final spotting. Extensive experiments were conducted on prevalent $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ and the SAMM-Long ME databases to evaluate the spotting performance. The results show that the AEM-Net achieves competitive performance, outperforming other state-of-the-art methods.

1. Introduction

Facial expressions, as alternative presentations of human emotion apart from speech, specifically their recognition, is a hot direction of affective computing. In addition to the macro-expressions (MaEs) that occur usually, micro-expressions can appear in the context of certain special situations. Micro-expressions (MEs), which arise in high-risk environments, are involuntary facial movements that occur spontaneously when an individual encounters certain emotions but try to suppress facial expressions [1]. Therefore, MEs can be widely applied in a series of fields. For example, when assessing the performance of a lecture session based on a student’s emotions, a transitory ME can reveal emotions normally hidden by the student. In business situations, a salesman can use ME to estimate a client’s real reaction during the showcasing of goods. Border control agents can detect unusual patterns of behavior by asking questions. The U.S. transportation security administration developed a spotting program, whereby airport staff are trained to monitor suspicious behaviors by analyzing MEs and signs in conversations [2]. In the medical field, especially in psychotherapy, doctors can use MEs as a guide to know how patients really feel [3]. Briefly, MEs analysis plays a vital role in analyzing people’s hidden emotions in various situations.

The duration of a ME is very short (generally no more than 500 ms) [4], and the intensities of involved muscle movements are subtle [5]. The automatic computing and analysis of ME is a new field in facial analysis, which began to arouse great interest from 2014 [6,7,8,9]. Generally speaking, facial ME analysis includes two tasks: recognition and spotting. The ME recognition has to do with determining the emotional states causing MEs, and the spotting is to locate the temporal positions of MEs in a video. Based on our survey, most ME researches focus on the recognition task, and ME spotting is given limited attention. This work aims to develop the ME spotting approach. For real-world applications, MEs, MaEs and no-expressions may exist simultaneously in long video sequences. Thus, the prerequisite for identifying categories of expression clips is to spot them accurately. As a video usually contains several expression clips, the purpose of the spotting is to predict the onset frame and offset frame, then keep the intersection over union (IoU) of ground truths (GTs) and proposals no less than the specified threshold. The principle of ME spotting with four scales is visualized in Figure 1. A clip of MaE or ME implies the complete process of certain emotion, which typically contains an onset frame, an apex frame and an offset frame. Based on this viewpoint, the onset frame is the starting point of an expression. The apex frame means the peak of facial muscles’ motions. The offset frame defines the point back to the initial state of the expression.

Recent advances in deep learning have witnessed widespread popularity in the recognition task, though limited efforts have been made for the spotting task, especially on long “untrimmed” videos. Owing to the similarity between temporal action spotting and object detection, many temporal action detection methods adopt a framework similar to object detection methods. One-stage detection of long videos was proposed firstly in SSAD [10], which detects the time boundaries and confidence scores of multiple categories directly. However, most long-video detection methods implement two-stage detectors, such as Faster RCNN, which will first obtain the proposals and then identify the category. Considering the low time cost of small ME data, this work concentrates on the two-stage framework for ME localization in long videos.

At present, ME spotting in long videos is still a worthwhile research topic with extraordinary attentions. However, several key challenges remain unresolved.

• The existing ME spotting samples are very limited; without abundant data, the deep neural network will fully “drain” the training data and over-absorb the information from the training set, resulting in performance degradation in the testing set.
• Due to the subtle movements of MEs, it is difficult to find MEs integrally in long video sequences; in other words, it is difficult to determine the boundaries of MEs.
• The temporal extent of expression varies dramatically compared to the sizes of objects in an image (from a fraction of 0.04 to 4 s). The significant distinction in duration also makes it very challenging to locate MEs and MaEs.

To improve the performance of ME spotting in long videos from the aforementioned three folds, we propose an adaptive enhanced micro-expression detection network based on a specific prior of ME. The main contributions of this paper are summarized as follows:

• We propose a multi-stage channel feature extraction module, named Prelayer, which is capable of fusing optical flow modality and RGB information and alleviating the problem of insufficient samples.
• A multi-scale network with the fusion of multiple scale features and an anchor-free-based mechanism was developed to locate each ME’s boundaries accurately.
• A new post-processing network for MEs, named ME-NMS (non-maximum suppression), is utilized to enhance the detection accuracy of some extremely short fragments.

This work is an extended version of our conference paper in the sixteenth Chinese Conference on Biometric Recognition (CCBR) [11], in which the multi-scale feature fusion network was shown to improve the spotting performance. Our code for this work is available at https://github.com/sijiacheng-0827/AEM-Net (accessed on 16 October 2022).

2. Related Work

In this section, we introduce the work related to the proposed spotting method. The related studies include action location, micro-expression spotting and non-maximum suppression.

2.1. Action Localization

The main purpose of action localization is to learn how to spot action instances in untrimmed videos, where boundaries and categories of action instances have been annotated. There are two kinds of action localization pipelines, the first being the one-stage pipeline, such as SSAD, and the other, the two-stage pipeline. Faster R-CNN is a typical example of a two-stage pipeline, which introduces a region proposal network (RPN) to generate proposals and also introduces an anchor box to cope with the changes in target shape. TAL [12] combined the two-stage pipeline with action localization and proposed an improved multi-scale-architecture, video-based action localization method which can accommodate extreme variation in action duration. While the aforementioned anchor-based methods severely rely on default anchors, which constrains the anchor point to be the center of the action instance. The anchor-based method limits the flexibility of dealing with action instances with drastic changes, especially for very short or long action instances. In order to address this problem, A2Net [13] uses an anchor-free module combined with the traditional anchor-based method for time-action positioning tasks, which assists action localization by temporal points and is capable of predicting the action instances whose durations are either extremely short or extremely long. In this paper, the mechanism proposed by A2Net is introduced into the localization of expression videos, so that very short micro-expression clips and very long macro-expression clips are localized precisely.

2.2. Micro-Expression Spotting

In micro-expression spotting, the duration of the micro-expression can be located by sliding the time window. Deep learning was used to perform ME localization for the first time in [14]. Their method uses 2D convolution to extract spatial features, 1D convolution to extract temporal features and the network module to generate clip proposals in the time domain. The clip proposal corresponds to the region proposal in the spatial domain. Moilanen et al. used local binary patterns (LBP) to extract feature differences (FD) between each frame of the video sequence to analyze the changes in facial motion and calculate the feature Chi-square ($X^2$) to generate the magnitude of the differences [15]. Davison et al. [16,17] spotted micro-expressions by histograms of oriented gradients. They represented all sequences detected within 100 frames as true positives, including blinks and fixations. Motion sequences that were detected but not encoded were classified as false positives. Patel et al. proposed a method to compute optical flow vectors over a small range and integrate them into the incorporated Spatio-temporal region to determine onset and offset times [18]. Tran et al. [19] constructed a multiscale benchmark based on a sliding-window multiscale assessment to fairly and better evaluate micro-expression identification methods. Wang et al. [20] implemented the same approach to spot micro-expressions in $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$. SOFTNet [21] proposed a multi-scale network model to predict a score, which captures the possibility of a frame in an expression interval by taking the location task as a regression problem and introducing pseudo-tags to promote learning efficient. Yifei Guo et al. proposed a ME spotting method [22] based on a magnitude and angle-combined optical flow feature, exploiting the angle information. LSSNet [23] compared the effect of localization frames or segments in the 2021 micro-expression contest. They also introduced a position-suppression module, which is similar to a non-local neural network [24]. It aims to suppress too-long and too-short proposals. The AEMNet proposed in this paper is an improvement on LSSNet, in which a deep feature adaptive enhancement module and a multi-level feature extraction module are integrated to focus on more subtle feature changes. Moreover, a feature pre-processing network named ME-NMS was designed, which combines the micro-expression prior to suppress the false detections and sets a suitable threshold for micro-expressions.

2.3. Non-Maximum Suppression

Non-maximum suppression (NMS) is an important part of the object detection process. NMS and its variants have been extraordinarily explored in recent years [25,26,27]. Specifically, the detection box is sorted by score, and then the box with the highest score is retained; other boxes whose overlap area with the box is larger than a certain proportion are deleted. If an object is within the preset overlap threshold, the object to be detected may not be detected. That is, when the two target boxes are close to each other, the box with a lower score will be deleted because the overlap area is too large. In most cases, there is no strong correlation between IoU and classification scores, and many boxes with high confidence in classification tags are not very accurate. In this regard, Navaneeth Bodlaand et al. proposed an approach, namely, Softnmsin, whose main function is to reduce confidence. This method greatly reduces the false detection rate for video clips. Seq-NMS proposed a modification of the post-processing phase that uses high-scoring object detection from nearby frames to boost scores of weaker detection within the same clip.

Inspired by these methods above, it is necessary to tune a post-processing network for ME-aware representation. More importantly, a new network namely, ME-NMS, was designed to add a prior of ME for good learning capacity.

3. Our Method

This section will introduce the AEM-Net in detail. First, the architecture of AEM-Net is outlined, and then the corresponding modules in the model are described.

3.1. Network Architecture

AEM-Net mainly consists of four parts: the multi-stage channel feature extraction module, multi-scale fused-feature localization network, position suppression and adaptive deep feature enhancement module and post-processing module (ME-NMS). The entire network structure is shown in Figure 2. Yu et al. [23] used the I3D model to extract micro-expression features in the LSSNet framework. The convolutional kernels and pooling kernels in the image classification network are extended from 2D to 3D, and the effective features can be extracted. However, the flow information and the RGB information extracted from the I3D model do not sufficiently interact. In order to extract richer and more robust features, a multi-stage channel feature extraction module named Prelayer was introduced. After obtaining the features that contain rich information, a sliding-window operation is performed on these feature sequences on four scales, in which the downsampling step of each scale is set to 2, and the feature maps of the next layer with larger receptive fields are upsampled and used to guide feature prediction in this layer. The number of frames contained in a micro-expression clip is often a relatively limited range, and the proposals obtained by the sliding-window operation may contain too many or too few frames, which will seriously interfere with the normal prediction and affect the detection performance. In order to suppress too-long and too-short proposals, and to make our network pay more attention to tiny motion features, a spatio-temporal attention module should be set up. The spatial attention mechanism is used to screen the proposals in a certain range, so as to improve the feature expression of crucial regions. The channel attention mechanism automatically obtains the importance of each feature channel by learning the network so that it adaptively strengthens the essential features and suppresses the non-important features. Finally, the ME-NMS module is proposed to search for the optimal candidate segment of the clip and suppress the false detection segment.

Given a video with MaEs or MEs or both, AEM-Net aims to locate all the MaE and ME instances appropriately. Suppose that all expressions $M\left(i\right)=\left\{f{\left(i\right)}_{on},f{\left(i\right)}_{off},c\left(i\right)\right\}$ in this video, where $f{\left(i\right)}_{on}$ and $f{\left(i\right)}_{off}$ represent onset and offset frames of the $i^{th}$ expression, respectively; $c\left(i\right)$ indicates the category label.

3.1.1. Multi-Stage Channel Feature Extraction

For a ME long video, LSSNet uses the dual-stream inflated 3D convolutional network (I3D) [28] to extract features from both the original images and the optical flow. On the one hand, the convolutional and pooling kernels in the image classification network are extended from 2D to 3D so as to learn spatio-temporal features by the 3D convolution network directly. On the other hand, the performance of ME spotting will be improved through the operation of fusing optical flow features. The features obtained through the I3D network contain some micro-expression information. However, the model simply concatenates the spatio-temporal and optical flow features. The insufficient fusion of them will lead to the loss of a lot of useful information and greatly increase the probability of negative samples. In order to extract richer and more robust features, a multi-stage channel feature extraction module was designed. The detailed structure of the module is shown in Figure 3.

Algorithm 1 ME-NMS loss calculating procedure.

Output: Correct process 1: Input: 2: $\beta =[b_1,\dots ,b_N],S=[s_1,\dots ,s_N],N_t,t,M$ 3: M means duration of $b_N$ 4: T is type of the set $t=1$ means macro-expression $t=2$ means micro expression 5: $\beta$ is the list of initial detection boxes 6: S contains corresponding detection scores 7: $N_t$ is the NMS threshold 8: Auxiliary variable: 9: K[ ] 10: K is the list to keep final detections after NMS 11: while$\beta \ne empty$do 12:  if $M(t=2)leqslant23$ then 13:   $K\leftarrow K\cup m;\beta \leftarrow \beta -b_m$ 14:   $S\leftarrow S-s_m$ 15:   $\mathbf{For}{b}_i\mathbf{in}\beta$ 16:    $\mathbf{if}IoU(b_m,b_i)\ge N_t$ 17:     $\beta =\beta -b_{i}S\leftarrow S\leftarrow s_i$ 18:  else 19:   $k\left[i\right]\to \beta \left[i\right]$ 20:  end if 21:  return K 22: end while

3.2. Modules

Specifically, three one-dimensional convolutional layers with different numbers of convolutional kernels were used to convolute the spatio-temporal and the optical flow features from the I3D model to obtain feature sequences with different channel numbers. The difference in the number of channels makes each feature sequence focus on different information. The parameter error of the convolutional layer will cause the offset of the estimated mean value, so a max pooling layer is needed to reduce the offset error and retain more facial texture information after each convolutional layer. Then, the feature sequences that pass through the max pooling operation are fused to obtain a richer feature expression.

3.2.1. Clip Proposal Network

In the field of object detection, researchers use segmentation algorithms to generate proposal regions. Micro-expression spotting is different from target detection. Figure 4 presents the effects of different lengths of clips using anchor-based and anchor-free methods, which were measured by MAP (mean average precision) [29].

For fragments with a duration between [2.5–4.9], the anchor-based detector head contributes to better results, and for fragments with a duration in the range of [0–2.5] or [4.9–6.9], the anchor-free detector head leads to better results. As the duration of the micro-expression sequence is generally in the range of [0–0.5] and the duration of the macro-expression sequence is generally in the range of [0.5–4], we combine the two detector heads as the detector in our method. The threshold was set as 0.1:0.1:0.7. The performance comparison focuses on the score under the threshold 0.5. It can be found that the anchor-free mechanism is more suitable for very short clips and the anchor-based mechanism is more suitable for medium-length clips. For the positioning of mixed-length clip sets, it is necessary to consider an effective combination of the two detectors. Hence, we follow the method of generating proposal regions in LSSNet. Specifically, anchor-based and anchor-free methods are employed in generating proposals. The detailed clip proposal network is shown in Figure 5.

The lengths of MaEs and MEs in the $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ datasets range from 3 to 117 frames. To be exact, the length of MEs range from 8 to 16 frames, and the lengths of MaEs range from 3 to 117. In view of this attribute of MaEs and MEs, it is necessary to ensure that the generated proposals are between 2 and 128. However, the generated feature sequence is too large when the proposal scale is set to two, leading to considerable computational complexity. Therefore, the case of scale two is discarded. In addition, five ratios [0.5, 0.75, 1, 1.5, 2] are set for the anchor of each scale, so we finally selected four scales from $2^3$ to $2^6$ as the anchor template. A convolutional layer with stride $si=2$ is used to reduce the temporal length at each hierarchy. Subsequently, the feature maps from the upper layer are up-sampled with the ratio of two. We also took a feature map from earlier in the network and merge it with our up-sampled features using the adding operator. This method enables our model to get more meaningful semantic information from the up-sampled features and finer-grained information from the earlier feature maps. In addition, several convolutional layers are embedded to eventually predict a similar tensor. Suppose the regression output of layer i is $l_i$, $r_i$ is the ratio of anchors, and the frame number of the sliding window is $s_w$. The number of anchors can be defined as follows:

$$n=r_i\times s_w/l_i$$

(1)

The anchor-based module selects the closest-matched anchor and regresses the boundary step by step. This mechanism enhances the capability to dispose of action instances with various lengths. Unlike the method of anchor-based, the proposed anchor-free module regresses the distances from the center of an action instance to its boundaries. Compared with previous prediction methods over action proposals, anchor-free regression is more flexible and robust, especially for action instances with extremely short or long durations. In more detail, the action instances are divided into several levels according to the actions’ temporal length. After mapping back, if the calculation result falls into the action’s scope, it is said to be the foreground point; otherwise, a background point. The anchor-free module predicts the classification score to determine whether a point falls into an action instance and its category label.

It can be found that the anchor-free module performs well for relatively short or long action instances, though the anchor-based module is more suitable for medium or long action instances. In essence, the reason for this phenomenon lies in the inherent flexibility of the anchor-free module and the stability of the classification and detection process adopted by the anchor-based module. As the duration of a micro-expression is short, the action is small, and there is a lot of interference in the facial expression. The duration of a macro-expression is longer and the action is larger. Thus, we use the detector combined with the anchor-based and the anchor-free modules. Combined with its detection ability, the weights of the two detectors in the detection are adjusted.

3.2.2. Spatio-Temporal Attention Block

In order to spot expression intervals of different duration (0.04–4 s) accurately, three down-sampling layers, that is, the convolution with four scales, were designed to comply with the intervals of expressions with a large duration span. However, it can lead to data redundancy and an increase in negative samples. The main reason is that the fewer down-sampling times are captured, the fewer scales and more proposals are generated. As a result, the attention to short-time duration samples is weaker. Thus, an attention module is introduced to alleviate this situation. Firstly, following the feature processing network, the clips in the dataset are restructured with some videos shorter than corresponding clips of MaE or some videos longer than the length of the whole video. Thus, the location-suppression module is built to suppress the too-long and too-short proposals. Secondly, the channel-wise feature recalibrating module is applied to focus on the adaptive local receptive field sizes of neurons. The detail of the channel attention block is shown in Figure 6. This will increase the attention of the network on short time samples and enhances the performance of ME spotting.

For the implementation of the position suppression part, assuming that the length of the sliding window is $s_w$; the sampling frequency of this sliding window is f; the strides of the current layer are $s_i$; and the input of sliding windows is $[bs,c_i,t_i]$, where $bs$ is the batch size, $c_i$ is the channel number and $t_i$ is the length of the current proposals.

$$t_i=\frac{s_w}{f\times s_i}$$

(2)

A position enhancer $e_{ij}$ is calculated by scale variation, which is activated with softmax as the weight of for different lengths of $t_i$.

$$e_{ij}={\phi }_{ij}\times {\rho }_{ij}$$

(3)

$$w_{ij}=Softmax\left(\frac{e_{ij}}{\sqrt{c_i}}\right)$$

(4)

where the size of position ${\rho }_{ij}$ is $[bs,s_i,t_i]$, the size of ${\phi }_{ij}$ is $[bs,t_i,1]$ and the number of ${\rho }_{ij}$ is $c_i$. The network structure is shown in Figure 7.

At the same time, to explore the relations among channels, the model proposes an architecture to recalibrate channel-wise feature responses by explicitly considering inter-dependencies between channels. Each weight value corresponds to each channel. It can present the correlation in channels by forming a bottleneck structure and outputting the same weights as the input features. Concretely, the global spatial information is first squeezed into the channel descriptors, which is average-pooled to 1, then recalibrated by $F_{ex}$, which contains two fully connected layers, and then the weight matrix is obtained. Finally, the weights are assigned to the sequence features. The weight of the channel is $W_{SE}$; $X_C$ is the original feature sequence.

$$W_{SE}=F_{ex}\left(F_{sq}\left(\left(X_C\right)\right)\right)$$

(5)

$$Z_C=W_{SE}\times X_C$$

(6)

$${\rho }_C=Z_C+X_C$$

(7)

$F_{sq}$ is the compression of the global spatial information into the channel descriptor. The global-average pooling is utilized to generate channel interaction. The operation is to squeeze the information in the time domain to a single point, which will take advantage of the information gathered in the compression operation. Then, a simple gating mechanism with a sigmoid activation $F_{ex}$ is followed. Layer normalization is used to balance the data distribution. In the end, the additional gain in performance is obtained by adding dropout in the convolutional layers.

$${\gamma }_{i{j}^{l-1}}={\gamma }_{i{j}^{l-1}}\ast r^{drop}$$

(8)

$$o_{ij}={\gamma }_{ij}^{\prime }\odot w_{ij}$$

(9)

3.2.3. ME-NMS Module

NMS is the abbreviation of non-maximum suppression. In traditional object detection, the closer the positions of the two boxes, the higher the score used for the benchmark. Specifically, the IoU (intersection of union) of two boxes is calculated by Equation (10).

$$IoU=\frac{a\cap b}{a\cup b}$$

(10)

If the IoU exceeds the threshold, it will suppress the box with a smaller score. As there is no need to output two similar boxes, the one with a large score is kept. In this paper, in order to reduce category decision errors, a ME-NMS module is proposed for ME spotting. In the post-processing process, we can set the box with the highest score as the candidate object and then sort the candidate boxes according to their scores from big to small; we intersect the current candidate segment with the union of the number n of first best candidates, and those with scores lower than the threshold $\theta$ are discarded ($\theta$ is set to 0.2). As a result, the false detection samples can be filtered out. As the micro-expression duration is extremely short and susceptible to interference, a threshold of ME should be set based on a short duration prior of ME. Through this operation, our multi-scalar localization process allows for a more accurate location of ME. Taking into account a video sequence, the anchor-based spotting module predicts anchors at each time position. By contrast, the anchor-free mechanism considers a temporal location $a_j(j\in \left\{0,\dots ,t-1\right\})$ on feature map $F_i$; the accumulated stride of the current layer is $s_i$. Therefore, we can map it back onto the input sequence:

$$J^{\prime }=s_i/2+j\times s_i$$

(11)

Since the length of micro-expressions in the $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ dataset ranges from 3 to 16, considering the mapping mechanism, a threshold was set for micro-expression localization during the post-processing process to filter candidate clips:

$$f{\left(i\right)}_{off}-f{\left(i\right)}_{on}=l_{max}+s_w/2=23$$

(12)

where $l_{max}$ is the number of frames in the longest micro-expression video clip and $s_w$ is the slider box size of layer 2. When the duration of the obtained candidate segment is larger than this threshold, the candidate segment will be taken as the final result. The algorithm implementation is elaborated in Algorithm 1.

4. Experiments

4.1. Database

$\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ [30] and SAMM-LV [31] were used for evaluating the proposed method. $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ is composed of 57 MEs and 300 MaEs, of which 57 ME samples are in 97 videos from 22 subjects with 30 frames percent second (FPS). Meanwhile, SAMM-LV includes 343 MaEs and 159 MEs in 147 long videos from 32 subjects with 200 FPS. Both the MaEs and MEs are labeled with corresponding types. Before extracting optical flow components, SAMM-LV was reconstructed by merging short videos and downsampling. Then, the features with 1024 dimensions were extracted from the optical flow information and raw images by I3D models with a frequency of eight and an interval of two. Figure 8 shows the partial samples in two databases.

4.2. Method

In our experiments, the TV-L1 optical flow operator was captured to extract the motion-related information for ME. Then, the I3D network, which is used to set fixed sliding windows, was constructed to extract the fixed-length features from the extracted optical flow and the original image. Finally, a multi-stage channel feature extraction module was added in front of the clip proposal network to make the network more sensitive to tiny motion features, and integrated the spatio-temporal information to alleviate the problem of insufficient samples. Three groups of comparative experiments were carried out to test the performance of AEM-net. To maintain fairness, we adopted the popular F1-score index for micro-expression spotting [32].

4.3. Loss Function Settings (Videos of Different Lengths)

In this paper, the detector exploits both the anchor-based and the anchor-free positioning modules. The classification loss generated by the previous module is $L_{cate}^{ab}$, the regression loss is $L_{Reg}^{ab}$, and the overlap loss is $L_{ol}^{ab}$. The classification loss generated by the latter module is $L_{cate}^{af}$; the regression loss is $L_{Reg}^{af}$. We combined the loss of the anchor-free module $L^{af}$ and the loss of the anchor-based module $L^{ab}$ to train AEMNet. The total loss L is defined by Equation (13)

$$L=\alpha L^{ab}+\gamma L^{af}$$

(13)

where $\alpha$ and $\gamma$ are the weights to balance these two kinds of losses. $L^{af}$ and $L^{ab}$ are provided as follows:

$$L^{af}=L_{cate}^{af}+\beta L_{Reg}^{af}$$

(14)

$$L^{ab}=L_{cate}^{ab}+{\beta }_1{L}_{Reg}^{ab}+{\beta }_2{L}_{ol}^{ab}$$

(15)

where $\beta$, ${\beta }_1$ and ${\beta }_2$ are weights to balance the three losses.

• Anchor-free classification loss: based on the classification scores $S^{af}(S_{ME},S_{MaE})$, AEMNet contributes the $L_{cate}^{af}$ by the standard cross-entropy loss.

  $$L_{cate}^{af}=-\frac{1}{N}\sum _{n=1}^N\sum _{i=0}^{C}log\left(\frac{exp\left(S_i^{af}\right)}{{\sum }_{j=0}^{C}exp\left(S_j^{af}\right)}\right)$$

  (16)
• The positioned time fragments are regressed to the start and end frames of micro-expressions and expressions $(r_s,r_e)$. The regression loss adopts the smooth $L1$ loss [33].

  $$L_{Reg}^{af}=-\frac{1}{N}\sum _{n=1}^N(smooth{L}_1(r_s,S^{\ast })+smooth{L}_1(r_e,e^{\ast }))$$

  (17)
• Anchor-based classification loss: $L_{cate}^{ab}$ is calculated via cross-entropy loss.

  $$L_{cate}^{ab}=-\frac{1}{N}\sum _{i=0}^N{y}_{i}log\left(y_i^{\ast }\right)+(1-y_i)log(1-y_i^{\ast })$$

  (18)
• Anchor-based overlap loss: The overlap $p_o$ is the calculation of intersection over union between the predicted instances and its closest ground truth; it is evaluated with mean-square error loss.

  $$L_o^{ab}=\frac{1}{N}\sum _{n=1}^N{(p_o-p_o^{\ast })}^2$$

  (19)
• Anchor-based regression loss: Given regression target $({\Delta }_C^{\ast },{\Delta }_W^{\ast })$ and the prediction $({\Delta }_C,{\Delta }_W)$, the anchor-based regression loss be defined as follows:

  $$L_{Reg}^{ab}=\frac{1}{N}\sum _{n=1}^N(smooth{L}_1({\Delta }_C,{\Delta }_C^{\ast })+smooth{L}_1({\Delta }_W,{\Delta }_W^{\ast }))$$

  (20)

4.4. Experimental Parameters and Configurations

All experiments were conducted on Pytorch 1.8.1. The initial learning rate was 0.0008, and the learning rate was decayed by a factor 0.1 with the interval of 30 epochs. The batch size was set to 32. The trade-off parameters were set as $\alpha =1.2,\gamma =0.8,$ $\beta =30,{\beta }_1=10,{\beta }_2=10$.

4.5. Results and Comparisons

The performance of AEM-Net is compared with those of some recent methods. The benchmarks were two public ME long-video datasets, $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ and SAMM-LV. The comparison’s results are listed in

Table 1. Comparison with other spotting methods.

Models	$\mathbf{CAS}{\left(\mathbf{ME}\right)}^2$			SAMM-LV
	MaE	ME	ALL	MaE	ME	ALL
Yap [34]	0.216	0.025	0.142	0.192	0.043	0.107
MESNet [35]	-	-	0.036	-	-	0.088
Zhang [36]	0.213	0.022	0.140	0.07	0.133	0.100
LSSNet [23]	0.368	0.050	0.314	0.314	0.191	0.286
AEM-Net	0.377	0.095	0.325	0.311	0.297	0.300
AEM-SE	0.315	0.104	0.286	0.260	0.315	0.264
AEM-F	0.293	0.129	0.289	0.355	0.303	0.313

Bold font indicates the best results.

, demonstrating how the AEM-Net approach greatly improves the performance of micro-expression localization. AEM-SE is a network in which the attention module of channel features is added to AEM-Net. AEM-F is a network in which the multiscale feature fusion operation is in a clip proposal module. In Table 1, it can be seen that by changing the attention to channel features, the network pays more attention to detailed features but pays less attention to structural features. The network has an improved ability to localize micro-expressions but less ability to localize macro-expressions. Structural information from large-receptive-field feature maps fused with fine-grained information enables more accurate ME localization.

Due to the characteristics of short duration and subtle motion in micro-expressions, the proposed method fuses the anchor-based and anchor-free localization mechanisms. We compared the method with the use of anchor-free and anchor-based methods on $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$. The results are shown in

Table 2. Different spotting methods used for micro-expression spotting.

	F1_Score_Macro	F1_Score_Micro	F1_Score
Anchor-free (Af)	0.219	0.000	0.194
Anchor-based (Ab)	0.341	0.060	0.295
Ab+Af	0.351	0.093	0.315

Bold font indicates the best results.

. It is experimentally concluded that the anchor-based+anchor-free method not only improves the accuracy of macro-expression localization, but also improves the accuracy of micro-expression localization, and the combination of these two detectors not only makes the localization of expression segments with different durations more accurate but also solves the problem of classes imbalance.

To further verify the contribution of the multi-stage channel feature extraction module (Prelayer), the ablation experiments for this module were conducted on $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$. The spotting results are listed in

Table 3. The performance when adding Prelayer.

	F1_Score_Macro	F1_Score_Micro	F1_Score
Prelayer	0.351	0.093	0.315
Non_pre	0.325	0.049	0.275

Bold font indicates the best results.

. As shown in Table 3, compared with a network without Prelayer(Non_pre), the localization performance is better. Therefore, it can be inferred that Prelayer can further optimize the feature sequences extracted from the I3D network by fusing feature-clustering information.

The impact of ME-NMS on network spotting performance is demonstrated in

Table 4. Performance of ME-NMS.

	F1_Score_Macro	F1_Score_Micro	F1_Score
NMS	0.351	0.086	0.312
ME-NMS	0.351	0.093	0.315
NMS-Nom	0.312	0.051	0.272

Bold font indicates the best results.

. The NMS method represents the same post-processing method without adding a threshold of MEs. NMS-Nom is our network using the traditional NMS algorithm. ME-NMS enables a certain enhancement in micro-expression spotting by setting a prior threshold that fits micro-expressions. However, there is no significant effect on macro-expression detection.

In forward propagation, the activation value of a certain neuron stops working with a certain probability. This makes the model more generalizable because it does not depend too much on certain local features. Following this fact, we did drop out of the output features and did a comparison experiment to explore the dropout rate to obtain optimal results. Specifically, we took (0:0.1:0.4) five groups of comparison experiments; the results are shown in Figure 9. The experimental results show that the best dropout rate is 0.1, which can suppress the overfitting problem of the spotting network.

In the evaluation phase, the top-threshold was used as the evaluation metric. Top-threshold is the top threshold (n%) predicted value of one video at a time for the leave-one-subject-out (LOSO) learning method, which was taken as the evaluation criterion. As the number of positive samples for a video is usually less than eight, we chose the threshold from 0.30 to 0.60 with an interval of 0.05 for testing. Each metric is compared for the $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ dataset and listed in

Table 5. F1-scoresof MaE and ME spotting on $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ using different top threshold values, one video at a time.

Threshold	0.3	0.35	0.4	0.45	0.5	0.55	0.6
MaE	0.367	0.356	0.362	0.377	0.365	0.354	0.342
ME	0.085	0.073	0.096	0.095	0.078	0.073	0.077
AU	0.321	0.325	0.307	0.325	0.313	0.311	0.295

Bold font indicates the best results.

. As shown in Table 5, the best result for the F1-score on $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ was a threshold of 0.45, which is also used for model training for SAMM-LV.

5. Conclusions

In this paper, a Prelayer model was developed for ME spatial-temporal feature fusion, aiming to improve the processing of micro-motion samples. This module combines the anchor-free mechanism with the anchor-based mechanism using fused multi-scale localization to accurately locate the onset and offset frames of expressions, avoiding prior biases regarding localization and duration. Moreover, the ME-NMS module is integrated to further reduce a priori-based false-positive micro-expression samples. In addition, channel attention and spatial attention jointly enhance the adaptive learning of key areas and deep features of the network. Comprehensive experiments demonstrated the effectiveness of our method. AEM-Net is capable of improving localized ME and MaE instances with various durations. It achieved improved performances on $\mathrm{CAS}{\left(\mathrm{ME}\right)}^2$ and SAMM-LV datasets. However, due to the brief and subtle nature of ME, the F1-score of our ME spotting is not satisfactory yet for real-world applications. Powerful feature learning and action localization in an end-to-end ME spotting framework are promising directions in future research. In addition, we will also study the bioinformatics principle of micro-expressions and further improve the performance of ME spotting.