Optical Flow Filtering-Based Micro-Expression Recognition Method

Abstract

The recognition accuracy of micro-expressions in the field of facial expressions is still understudied, as current research methods mainly focus on feature extraction and classification. Based on optical flow and decision thinking theory, we propose a novel micro-expression recognition method, which can filter low-quality micro-expression video clips. Determined by preset thresholds, we develop two optical flow filtering mechanisms: one based on two-branch decisions (OFF2BD) and the other based on three-way decisions (OFF3WD). In OFF2BD, which use the classical binary logic to classify images, and divide the images into positive or negative domain for further filtering. Differ from the OFF2BD, OFF3WD added boundary domain to delay to judge the motion quality of the images. In this way, the video clips with low degree of morphological change can be eliminated, so as to directly improve the quality of micro-expression features and recognition rate. From the experimental results, we verify the recognition accuracy of 61.57%, and 65.41% for CASMEII, and SMIC datasets, respectively. Through the comparative analysis, it shows that the scheme can effectively improve the recognition performance.

1. Introduction

Micro-expressions are always subtle and unconscious, which the duration of facial expression action is less than 0.2 s [1]. Micro-expressions can mine the true emotional state hidden in the human and demonstrates important application value in medical, polygraph, legal case and other fields. Therefore, micro-expressions recognition has gradually become a hot research issue. In early period, Ekman et al. [2] designed a Facial Action Coding System (FACS) based on the correlation between facial morphology changes to facilitate in-depth research in the field of it, and each of facial action coding is called an action unit (AU). After that, a lot of relative works have been researched on micro-expressions with the gradual maturity of artificial intelligence technology. However, the dataset of micro-expressions is difficult to collect because of the short duration and unconscious characteristics. Therefore, many micro-expression datasets have been constructed for the convenience of researchers for better in-depth research, such as SMIC [3], CASME II [4], SAMM [5], CAS(ME)2 [6], etc.

The main research idea of traditional micro-expression is to mine micro-expression information by extracting the features of the image combined with the classifier method. Pfister et al. [7] proposed an automatic micro-expression recognition system based on a temporal interpolation model and a spontaneous micro-expression corpus. This work is one of the earliest to realize automatic micro-expression recognition. Domestically, as representatives of micro-expression research and development, Fu et al. [8,9] made a lot of efforts to facilitate further research on micro-expression. With the gradual maturity of deep learning technology, researchers at home and abroad began to use deep learning methods [10,11,12] for micro-expression image coding and micro-expression recognition. Those methods used CNN and extended methods to replace the traditional texture feature extraction method for feature extraction and recognition.

According to the research, it can be found that the current research on micro-expression recognition at home and abroad is mainly related to the following two links: The first is how to extract high-quality texture features, and the second is to choose which classifier has a good classification effect. However, there are several problems with the current research thinking. 1. Traditional research methods mainly focus on the internal connection between the feature value in the images. Actually, the micro-expression is essentially the combined behavior of the morphological semantics of the various organs of the face, so that the influence of the morphological semantic quality of micro-expression is ignored. 2. Because of the small amount in the micro-expression dataset, the potential of deep learning method in the field of micro expression can’t be fully exploited, which also has low interpretability as a brute force method. In response to this problem, this paper introduces the decision-making thinking of rough set theory, which proposes an optical flow filtering method based on optical flow characteristics. This method eliminates video clips with low morphological change to improve the feature expression ability of the micro-expression to be recognized, thereby achieving the purpose of improving the performance of micro-expression recognition.

Therefore, the main contributions are as follows:

• From the morphological change perspective, we propose a novel image weighting method based on optical flow to assign semantic quality to images between video clips;
• Two optical flow filtering algorithms are proposed based on two-branch decisions and three-way decisions and a framework of micro-expression recognition, which establishes an interpretable and robust micro-expression recognition theory;
• Compared with the state-of-the-art solution, the experiment proves the superiority of the effectiveness and efficiency of this method.

The rest of the paper is organized as follows: Section 2 conduct research work on related technologies. Section 3 describes the details of our method. Section 4 reports and discusses the experimental results. Section 5 gives conclusion and future work.

2. Related Works

Compared with macro-expression, the micro-expression has the characteristics of short duration and low intensity of facial movements, which leads to greater challenges in the facial features of micro-expressions. For the research on micro-expressions, the current research can be summarized as end-to-end research mode, which mainly focus on pre-processing, diversified feature extraction, and joint classifiers to micro-expression recognition. Nowadays, more and more new research methods are proposed with the maturity of science and technology.

I. Face detection and alignment. Face detection is essential for many facial applications (such as facial recognition and facial expression analysis). Accurate facial recognition through face detection methods can effectively reduce the impact of noise on micro-expressions in human face. The current face detection methods mainly include: 1. Use feature points to align faces and determine key points in the face area; 2. Use deep learning methods for face detection.

Cootes et al. [13] proposed the active shape model (ASM), which uses a 68-point subjective shape model to locate the key points of the human face to obtain the key points of the face area. This is an early widely used method for aligning face parts. Through the improvement of ASM, Cootes et al. [14] further proposed the active appearance model (AAM), which not only includes the establishment of shape models, but also the establishment of texture models and hybrid models. Subsequently, Cootes et al. [15] proposed a constrained local model (CLM), which abandoned the global texture method of the AAM algorithm, and modeled the local area in the neighborhood of candidate matching feature points and Space position is constrained.

With the maturity of deep learning technology, deep learning is widely applied in different fields [16,17]. Yang et al. [18] proposed a cascaded structure to detect tiny faces in the complex environments. Zhang et al. [19] exploits the inherent correlation between the tasks of detection and alignment, and combined the CNN to proposed MTCNN. MTCNN utilizes a cascaded multi-task framework to boost up the performance of both face detection and alignment. In addition, fast R-CNN [20] or faster R-CNN [21] has also been applied to face detection, which has good improvement in detection speed.

II Feature extraction and derivation. Feature extraction has been a hot issue in the field of micro-expression research. The performance of micro-expression recognition is mainly reflected in the extraction of texture features according to current research. Ojala et al. [22] proposed a kind of binary vector encoding (LBP) of local pixels, and describes the texture information of the current image by generating a histogram. LBP generally used for feature extraction of static image. Expand based on LBP operator, Zhao et al. [23] proposed a dynamic spatiotemporal feature operator LBPTOP. Compared with LBP Operator, LBPTOP takes into account the influence of video spatiotemporal information, thus adding two new feature planes XT and YT of projection on T time axis. Based on the above two classic operators, a variety of feature extraction variant operators have been derived. Huang et al. [24] proposed a spatiotemporal Local Binary Pattern with Integral Projection (STLBP-IP), which uses a differential image-based integral projection method to obtain appearance and motion features in horizontal and vertical directions. Subsequently, Huang et al. [25] proposed a spatial and temporal complete local quantization module (STCLQP) for facial micro-expression analysis, which optimizes the efficiency of micro-expression recognition by extracting three levels of feature vectors of pixel coincidence, amplitude and direction. With the further research, the team [26] once again proposed the local binary mode based on the spatiotemporal radon transform. Guo et al. [27] proposed an extended local binary on the tri-orthogonal plane Value model (ELBPTOP). In order to improve the efficiency of dynamic feature extraction, the LBPSIP [28] (Local Binary Patterns with Six Intersection Points) operator is proposed through the optimization on the LBP-TOP operator. This method transforms the histogram features of the three planes of LBPTOP into a two-dimensional vector encoding between 6 points intersecting XY, XT and YT, and uses the generated histogram as the final dynamic texture feature. Guo et al. [29] proposed to replace the classic LBPTOP operator with CBPTOP operator for image feature extraction, and then use ELM for feature recognition of micro-expressions.

III Research on micro-expression framework. With the gradual maturity of feature operators, in order to break this solidified research model, researchers began to conduct in-depth research on the recognition framework of micro-expression. Jia et al. [30] used the LBP operator to extract the macro-information and the LBP-TOP operator to extract the micro-information to construct the conversion model of the micro expression macro and micro information. Yan et al. [31] restricted local model (CLM) algorithm combined with local binary mode (LBP) and optical flow (of) to identify and analyze the correlation between the frames of face region (ROIs). Wang et al. [32] combining the weights calculated by the cumulative optical flow with the spatiotemporal features extracted by LBPTOP obtained the weighted spatiotemporal features, and the weighted features are input into a support vector machine for micro-expression recognition. Wang et al. [33] convert a color video clip into a four-dimensional vector, where the first two dimensions are spatial information, the third dimension is temporal information, and the fourth dimension is color information. Liong et al. [34] extracted the features of the vertex frame by selecting the Apex frame and the offset frame image of each video and combining the bi-directional weighted optical flow (Bi-WOOF).

IV Application of decision thinking theory. The decision thinking in rough set theory is the core idea of two-branch decision and three-way decision. Two-branch decision emphasizes the right or wrong of things and is absolutely. However, the three-way decision is a decision-making model based on human cognition, which believes that people can immediately make judgments about things that are fully certain to accept or reject in the actual decision-making process. Additionally, for things that cannot be made immediately, people tend to postpone the judgment of the event. Yang et al. [35] applied the three-way decision in medical diagnosis. Liang et al. [36] introduced the method of two-branch decision and three-way decision into the field of clustering, and defined a cluster calculation method based on clustering.

In addition, there are analysis some state-of-the-art-review in the fields of micro-expression, as shown in

Table 1. Analysis of the state-of-the-art-review for micro-expression recognition.

Methods	Description	Advantage
STCLQP [25]	Extracts the features of sign, magnitude and orientation components information and fused.	For facial micro-expression appearance and motion feature extraction has a better effect.
TAOF [32]	The motion intensity is calculated by accumulating the neighboring optical flow in the time interval, and the feature are further weighted locally.	Enhanced the discrimination of the weighted features for micro-expression recognition.
BI-WOOF [34]	Created a new feature extractor, which to encode essential expressiveness of the apex frame.	Capable of representing the apexframe in a discriminative manner which emphasizes facial motion information at both bin and block levels.
FHOFO [37]	Construct suitable angular histograms from optical flow vector orientations and using histogram fuzzification to encode the temporal pattern.	Insensitive to change of illumination conditions among video clips and the extracted features are robust to the variation of expression intensities
RIMDS [38]	Explored the shallower-architecture and lower-resolution input data, shrinking model and input complexities simultaneously for the composite-database.	Achieved the superior performance compared to the micro-expression recognition approaches.
FMBH [39]	Combing both the horizontal and the vertical components of the differential of optical flow as inspired from the motion boundary histograms (MBH).	The unexpected motions, caused by residual mis-registration that appearsbetween images cropped from different frames, can be removed.
FDM [40]	Characterize the movements of a micro-expression in different granularity.	Provided an effective and intuitive understanding of human facial expressions.
STRCN [41]	A novel deep recurrent convolutional networks based micro-expression recognition approach and capturing the spatiotemporal deformations of micro-expression sequence.	Conquered the “limited and imbalanced training samples” problem and considered the spatiotemporal deformations.

. According to the above research results, this paper combines multiple decision-making methods with classic feature extraction methods to construct a new micro-expression research framework.

3. Proposed Methods

Our proposed method consists of three main phases: Weight generation phase, optical flows filtering phase and micro-expression recognition phase. The main framework of our method is shown in Figure 1.

3.1. Weight Generation Phase

The analysis of SMIC [3] and CASMEII [4] can be found that the video clips contains a large number of images with no expression and low morphological semantic change. The region of the expression is mainly focus on the beginning of Onset-Frame to the end of Offset-Frame, and the peak of semantics is reached in Apex-Frame. Therefore, the features extracted through low motion semantic images will result in a large amount of low-quality and imbalance data. In order to conquered the problem, this paper proposes a low motion semantic image filtering method based on optical flow and decision thinking theory, which is used for the experimental preprocessing of video clips.

A pre-processing step was needed to ensure that have common features because of the resolution and other external features of each dataset used in experiments are various. First, Multi-task Cascaded Convolutional Networks (MTCNN) [19] module was adopted to detect face areas from the input images. Second, it crops only facial area from the entire image to make them the same size and not affected by unnecessary parts such as hair or accessories. Additionally, the cropped face images are resized to a size of 300 × 250. In this manner, the variations of spatial appearance among different video clips are normalized. Subsequently, the dense optical flows [42] are extracted from the aligned video clips to represent motion information of each pixel in micro-expression video clips. The optical flows ${\xi }_{o_i}\left(x,y\right)$ are computed from each pair of neighboring images. When input t-frames video $V$= {$v_1,v_2,\dots ,v_t$} into the method, $t-1$ frames optical flows $O=\left\{o_1,o_2,\dots ,o_{t-1}\right\}$ will be obtained, Where $o_i$ is the optical flow image between $v_i$ and $v_{i+1}$. The optical flows ${\xi }_{o_i}\left(x,y\right)$ consist of horizontal displacements and vertical displacements. Therefore, the optical flows ${\xi }_{o_i}\left(x,y\right)$ can be expressed as:

$${\xi }_{o_i}\left(x,y\right)=\left(h_{o_i}\left(x,y\right),v_{o_i}\left(x,y\right)\right)$$

(1)

where $h_{o_i}\left(x,y\right)$ and $v_{o_i}\left(x,y\right)$ indicate the horizontal and vertical displacements respectively.

According to the optical flow of obtaining $t-1$ frames, for each image of optical flow, the Euclidean distance of the horizontal and vertical displacement of each pixel is obtained as the motion intensity of the current pixel. Then, the optical flow motion intensity $W_{o_i}$ of the entire optical flow image of $W\times H$ pixels can be calculated as:

$$W_{o_i}={\displaystyle \sum }_{\mathrm{x}=0}^W{\displaystyle \sum }_{\mathrm{y}=0}^H\sqrt{{\mathrm{h}}_{o_i}{\left(\mathrm{x},\mathrm{y}\right)}^2+{\mathrm{v}}_{o_i}{\left(\mathrm{x},\mathrm{y}\right)}^2}$$

(2)

where $o_i$ are the i-frame optical flow and $i=1,2,\dots ,t-1$, $W$ represents for optical flow $o_i$ width, $H$ represents for optical flow $o_i$ height.

For the optical flow $o_i$ in the set $O$, given a semantic quality metric ${\overline{W}}_{o_i}$ called the optical flow average motion intensity to measure the quality of $o_i$. ${\overline{W}}_{o_i}$ represents the average unit pixel semantic quality of the optical flow $o_i$ w.r.t the $v_i$ to $v_{i+1}$.The calculation formula as shown in Equation (3):

$${\overline{W}}_{o_i}=\frac{W_{o_i}}{W\times H}$$

(3)

where $W_{o_i}$ represents the motion intensity of i -frame optical flow, $W$ represents for optical flow $o_i$ width, $H$ represents for optical flow $o_i$ height.

Therefore, the motion intensity of optical flow can be measured by the average motion intensity value of optical flow in Equation (3). The average motion intensity reflects the strength of each group of neighboring video images. That is, the larger the average motion intensity of the optical flow, the larger the semantic quality between neighboring video frames. Namely, the smaller the average motion intensity of the optical flow, the smaller the semantic quality between neighboring video frames. Therefore, the average motion intensity of optical flow can be used to measure semantic quality information.

In order to better classify the semantic quality, the semantic quality of optical flow is evenly distributed in $\left[0,1\right]$ by defining a normalization weighting formula. That is to say, the larger the average motion intensity of the optical flow, the larger the normalized semantic weight value; on the contrary, the smaller the average motion intensity of the optical flow, the smaller the normalized semantic weight value. For each optical flow frame of the average motion intensity ${\overline{W}}_{o_i},$ the evaluation weighting of optical flow weighting filtering algorithm as shown in Equation (4).

$${\omega }_{o_i}=-e^{-\frac{{\overline{W}}_{o_i}}{\theta }}+1$$

(4)

where $\theta$ is a parameter to adjust the influence of the weighted over the index.

Therefore, all the video frames obtained the semantic quality weight information ${\omega }_{o_i}\left(i=1,2,\dots ,t-1\right)$ between neighboring video frames, the weight information represent the strength of the face area change between the neighboring video frames.

3.2. Optical Flows Filtering Phase

3.2.1. Optical Flows Filtering Algorithm Based on Two-Branch Decisions

The two-branch decisions have important application value in the choice of uncertainty about the data removal and retention. Therefore, determine the semantic quality of the video frames by defining the weight threshold $\delta \left(\delta \in \left(0,1\right)\right)$, which facilitates the classification of high-quality or low-quality semantic images. In solving uncertainty problems, classic binary logic embodies its important role. In distinguishing the motion intensity, the weight threshold is used to judge the semantic quality of the video frames, so that the optical flow weight between the video frames is divided into an object set $PO{S}_{\delta }\left(O\right)$ or $NE{G}_{\delta }\left(O\right)$.

When ${\omega }_{o_i}$≥ $\delta$, the optical flow frame $o_i$ is divided into the positive domain of the set ${\omega }_{o_i}$ and this domain is denoted as $PO{S}_{\delta }\left(O\right)$,recorded as $o_i\in PO{S}_{\delta }\left(O\right)$;

When ${\omega }_{o_i}$ < $\delta$, the optical flow frame $o_i$ is divided into the negative domain of the set ${\omega }_{o_i}$, and this domain is denoted as $NE{G}_{\delta }\left(O\right)$, recorded as $o_i\in NE{G}_{\delta }\left(O\right)$.

Through the above method, the optical flow frames with high semantic change can be distinguished from the optical flow frames with low semantic change, so that the high semantic quality optical flow frames in set $PO{S}_{\delta }\left(O\right)$, and the low semantic quality optical flow frames in set $NE{G}_{\delta }\left(O\right)$.

With the average motion intensity weight ${\omega }_{o_i}\left(i=1,2,\dots ,t-1\right)$ and threshold $\delta \left(\delta \in \left(0,1\right)\right)$, the motion intensity of the optical flow frame is divided into a set $PO{S}_{\delta }\left(O\right)$ and a set $NE{G}_{\delta }\left(O\right)$. The more essential meaning is to distinguish images with small morphological changes between neighboring video frames from pictures with large morphological changes. This method uses the image of the first frame as the semantic benchmark to mine the morphological semantic quality of time series. Corresponding to this, the video frames V can be divided as follows according to the optical flow sets $PO{S}_{\delta }\left(O\right)$ and $NE{G}_{\delta }\left(O\right)$ as follows:

When $o_i\in PO{S}_{\delta }\left(O\right)$, the video frame $v_{i+1}$ is divided into the high semantic quality domain, recorded as $v_{i+1}\in PO{S}_{\delta }\left(V\right)$;

When $o_i\in NE{G}_{\delta }\left(O\right)$, the video frame $v_{i+1}$ is divided into the low semantic quality domain, recorded as $v_{i+1}\in NE{G}_{\delta }\left(V\right)$;

At this point, any video frame $v_i\in V$ in the video frames set $V$ given semantic quality w.r.t $V$ and based on optical flow weighting of the image semantic quality filtering algorithm is completed, which lays a solid foundation for the establishment of the morphological semantic database in Section 4 below. The pipeline of OFF2BD algorithm is illustrated in Algorithm 1.

Algorithm 1 Optical Flows Filtering Algorithm Based on Two-Branch Decisions (OFF2BD)

Input: video frames $V$= {$v_1,v_2,\dots ,v_t$}, threshold δ Output: High-quality semantic image set $V$ after optical flow filtering 1. For $v_i=1,2,\dots ,\left|V\right|-1$/*Get initial optical flow set $O$ */; 2. For $v_j=2,3,\dots ,\left|V\right|$; 3. Generate optical flow $o_i$ for $v_i$ and $v_j$; 4. End for 5. End for 6. Obtain optical flow set $O$; 7. For $o_i=1,2,\dots ,\left|O\right|$/*Two-branch decisions iteration for optical flow in $O$ */; 8. Calculate the average optical flow motion intensity ${\overline{W}}_{o_i}$ of optical flow $o_i$ w.r.t. Formulae (2) and (3); 9. Calculate the optical flow weight ${\omega }_{o_i}$ based on the average optical flow motion intensity ${\overline{W}}_{o_i}$ w.r.t. formula 4; 10. If ${\omega }_{o_i}\ge \delta$ then; 11. put $o_i$ into $PO{S}_{\delta }\left(O\right)$; 12. Else; 13. put $o_i$ into $NE{G}_{\delta }\left(O\right)$; 14. End if; 15. End for; 16. Delete the optical flows set of $NE{G}_{\delta }\left(O\right)$ and update the optical flow set $O$ with $PO{S}_{\delta }\left(O\right)$; 17. Update video set $V$ according to the updated optical flow set $O$; 18. End for; 19. Output high-quality semantic image set $V.$

3.2.2. Optical Flows Filtering Algorithm Based on Three-Way Decisions

As the most important extension of the two-branch decisions, the three-way decisions [43] is an important granular calculation method derived from the rough set theory in machine learning, which is a new theory for solving uncertain problems proposed by the famous granular computing expert Yao [44] for the first time based on rough set theory and time research. Compared with the traditional two-branch decisions which requires the iterative incremental solution to solve the uncertainty of the problem, the three-way decision increases the delay behavior when the decision making is not accurate or cannot make accurate positive and negative decisions. The three-way decisions strategy not only reasonably explain the division behavior of the three decision domains (positive domain, negative domain and boundary domain) in the rough set theory, but also reduce the time consumption (abandoning incremental decision-making strategy (i.e., adding new knowledge constantly to make sure the uncertainty part turn to certain domain) in two-branch decisions.)

In dealing with uncertain problems, the three-way decisions method developed by the two-decision method is more suitable for human inertial thinking in solving problems. This approach is more representative of the problem-handling theory of contradictory relation and contrary relation problems. Therefore, in order to improve the semantic relationship between video frames and improve the quality of feature extraction, the quality thresholds $\alpha \left(\alpha \in \left(0,1\right)\right)$ and $\beta \left(\beta \in \left(0,1\right)\wedge \beta >\alpha \right)$ of the video frames are defined for optical flow filtering. And then, the optical flows can be divided into an object set $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$ or $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ or $NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$.

When ${\omega }_{o_i}$≥ $\beta$ the optical flow frame $o_i$ is divided into the positive domain of the set ${\omega }_{o_i}$ and this domain is denoted as $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$, recorded as $o_i\in PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$;

When $\alpha \le {\omega }_{o_i}<\beta$, the optical flow frame $o_i$ is divided into the positive domain of the set ${\omega }_{o_i}$ and this domain is denoted as $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$,recorded as $o_i\in BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$;

When ${\omega }_{o_i}<\alpha$, the optical flow frame $o_i$ is divided into the negative domain of the set ${\omega }_{o_i}$, and this domain is denoted as $NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$, recorded as $o_i\in NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$.

Three-way decisions methods for optical flow quality division are defined for $\left(\alpha ,\beta \right)$ threshold, wherein the $\left(\alpha ,\beta \right)$ are usually set to $0<\alpha <\beta <1$. Compared with the two-branch decisions filtering method, this method adds the uncertainty domain $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ as a buffer area for uncertain semantic quality processing. Then, the $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ domain and the $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$ domain are jointly used as the update domain, and iterative optical flow filtering analysis is performed until convergence is reached or a certain number of stops.

As the same time, the first frame of the image is used as a semantic reference combined with a convergent high-quality optical watershed to divide the collection of video frames. Corresponding to this, the video frames V can be divided as follows according to the optical flow set $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$, $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ and $NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$ as follows:

When $o_i\in PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$, the video frame $v_{i+1}$ is divided into the high semantic quality domain, recorded as $v_{i+1}\in PO{S}_{\left(\alpha ,\beta \right)}\left(V\right)$;

When $o_i\in BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$, the video frame $v_{i+1}$ is divided into the fuzzy semantic quality domain, recorded as $v_{i+1}\in BN{D}_{\left(\alpha ,\beta \right)}\left(V\right)$;

When $o_i\in NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$, the video frame $v_{i+1}$ is divided into the low semantic quality domain, recorded as $v_{i+1}\in NE{G}_{\left(\alpha ,\beta \right)}\left(V\right)$;

When a specified number of iterative filtering or convergence is achieved, the $PO{S}_{\left(\alpha ,\beta \right)}\left(V\right)$ and $\in BN{D}_{\left(\alpha ,\beta \right)}\left(V\right)$ video frames set are jointly updated to the video frames set of the target domain $V$, and the video frame is reordered to obtain a high-quality semantic sequence video combination. The pipeline of OFF3WD algorithm is illustrated in Algorithm 2.

Algorithm 2 Optical flows filtering algorithm based on three-way decision (OFF3WD)

Input: video frames $V$={ $v_1,v_2,\dots ,v_t$}, threshold $\alpha ,\beta$, iteration coefficient $\eta$; Output: High-quality semantic image set $V$ after optical flow filtering. 1. For $k=1,2,\dots ,\eta$/*Filtering iteration for video frames in set $V$ */; 2. For $v_i=1,2,\dots ,\left|V\right|-1$/*Get initial optical flow set $O$ */; 3. For $v_j=2,3,\dots ,\left|V\right|$; 4. Generate optical flow $o_i$ for $v_i$ and $v_j$; 5. End for 6. End for 7. Obtain optical flow set $O$; 8. For $o_i=1,2,\dots ,\left|O\right|$/*Three-way decision iteration for optical flow in $O$ */; 9. Calculate the average optical flow motion intensity ${\overline{W}}_{o_i}$ of optical flow $o_i$ w.r.t. Formulae (2) and (3); 10. Calculate the optical flow weight ${\omega }_{o_i}$ based on the average optical flow motion intensity ${\overline{W}}_{o_i}$ w.r.t. Formula (4); 11. If ${\omega }_{o_i}$≥ $\beta$ then; 12. put $o_i$ into $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$; 13. If $\alpha \le {\omega }_{o_i}<\beta$ then; 14. put $o_i$ into $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$; 15. Else; 16. put $o_i$ into $NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$; 17. End if; 18. When $\left|NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)\right|=0$ then;/*Reach the convergence condition*/ 19. Combine $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$ and $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ to update $O$; 20. Break; 21. Increase the $\alpha$ threshold naturally and delete the optical flows set of $NE{G}_{\left(\alpha ,\beta \right)}\left(O\right)$; 22. Combine $PO{S}_{\left(\alpha ,\beta \right)}\left(O\right)$ and $BN{D}_{\left(\alpha ,\beta \right)}\left(O\right)$ to update $O$; 23. Update video set $V$ according to the updated optical flow set $O$; 24. End for; 25. End for; 26. Obtain the video frame set $V$ according to the completed iteration condition or the converged optical flow set $O$; 27. Output high-quality semantic image set $V$.

3.3. Micro-Expression Recognition Phase

3.3.1. Feature Extraction

In the field of micro-expression recognition, Local Binary Pattern (LBP) [22] is a common symbiotic pattern method that encodes local pixels in an image. The original LBP operator is defined as within the window of 3 × 3, taking the center pixel of the window as the threshold, and comparing the gray value of the adjacent 8 pixels with the center pixel in a clockwise direction, if the surrounding pixel value is greater than or equal to the center pixel value, the position of the pixel is marked as 1, otherwise it is 0. In this way, 8 pixels in the neighborhood of 3 × 3 can be compared to produce 8-bit binary numbers (usually converted to decimal numbers, that is, LBP codes, a total of 256 kinds), that is, the LBP value of the pixel at the center of the window is obtained, and this value is used to Reflect the texture information of this area. Such a 3 × 3 picture can be transformed into 256-dimensional texture features. The formula is shown in Equation (5).

$$H\left(x_c,y_c\right)={\displaystyle \sum }_{p=0}^{p-1}{2}^{p}S\left(i_p-i_c\right)$$

(5)

$$S\left(x\right)=\{\begin{array}{cc}1& if\begin{array}{c}x\ge 0\end{array}\\ 0& else\end{array}$$

(6)

where $\left(x_c,y_c\right)$ represents the central pixel, $i_c$ is the gray value, $i_p$ is the gray value of the adjacent point, and $S\left(x\right)$ is a function that conforms to Equation (6). According to the above formula method, the binary extraction process of the face picture is shown in Figure 2.

In order to solve the dynamic feature extraction, based on the LBP method proposed a dynamic feature extraction method LBPTOP (Local Binary Pattern from Three Orthogonal Planes). LBPTOP solves the problem that LBP can only extract two-dimensional features, and can be applied to feature extraction in three dimensions. LBPTOP extracts LBP features in three dimensions from the spatial plane XY and the spatial–temporal planes XT and YT respectively, and then combines them in series to form spatial-temporal features.

We use LBPTOP, LBPSIP, STCLQP, STLBPIP feature descriptors to encode the optical flow filtered video to express temporal and spatial information. These feature descriptors are all improved based on the LBPTOP method, and feature enhancement is performed on the three planes of the spatial plane XY and the spatial–temporal plane XT, YT. Therefore, the feature histograms of the based on the two-branch decisions spatial–temporal video V obtained after the optical flow filtering can be expressed as:

$$\{\begin{array}{c}{V}_{\delta }=PO{S}_{\delta }\left(V\right)\\ H_{\delta }=\left[H_{V_{\delta }}^{XY},H_{V_{\delta }}^{XT},H_{V_{\delta }}^{YT}\right]\end{array}$$

(7)

where $V_{\delta }$ represents the set of high-quality semantic frame spatiotemporal sequences obtained after the convergence of the OFF2BD algorithm. $H_{\delta }$ are the functions to get the feature histograms from the $V_{\delta }$.

To further improve the morphological semantics quality of the spatial–temporal video $V$, the feature histograms based on OFF3WD can be expressed as:

$$\{\begin{array}{c}{V}_{\alpha ,\beta }=PO{S}_{\alpha ,\beta }\left(V\right)\cup BN{D}_{\alpha ,\beta }\left(V\right)\\ H_{\alpha ,\beta }=\left[H_{V_{\alpha ,\beta }}^{XY},H_{V_{\alpha ,\beta }}^{XT},H_{V_{\alpha ,\beta }}^{YT}\right]\end{array}$$

(8)

where $V_{\alpha ,\beta }$ represents the set of high-quality semantic frame spatiotemporal sequences obtained after the convergence of the OFF3WD algorithm. $H_{\alpha ,\beta }$ are the functions to get the feature histograms from the $V_{\alpha ,\beta }$.

3.3.2. Micro-Expression Recognition

For classic micro-expression feature recognition, Whitehill et al. [45] analyzed and compared it and found that SVM has relatively good recognition effect on micro-expression recognition compared with other traditional machine learning algorithms. FHOFO [37] uses linear SVM, 3point KNN and LDA to make correlation comparison among different micro-expression datasets. The experimental results show that compared with other classifiers, linear SVM has good generalization ability and stability. On the classification method, the leave-one-subject-out (LOSO) method was used to train and test the classifier. The LOSO-based validation is performed considering all images in the dataset. Given a dataset of n samples, the LOSO-based validation is performed with n iterations, such that in each iteration the classifier is trained with n-1 samples and tested on the remaining one sample. Using this strategy to evaluate the classification effect can more intuitively and accurately reflect the effect of the current experimental method. Therefore, this classification test method is currently used in most machine vision fields.

4. Experiments and Discussion

In this paper, The SMIC and CASME II will be used for optical flow filtering related experiments. All experiments in this paper are running on a PC of Windows 10, intel i7-8750H, 16G memory, python 3.7 environments, MATLAB 2016a and the python IDE environment is pyCharm professional 2019.3.

4.1. Datasets

The experiment selects two most popular spontaneous datasets, including CASME II and SMIC are used to evaluate the performance of the proposed method. The datasets statistics are summarized in

Table 2. A summary of the different features of the SMIC and CASME II datasets.

Feature	SMIC-HS	SMIC-VIS/NIR	CASME II
subjects	16	16	35
samples	164	71	247
Frames per sec.	100	25	200
Posed/spontaneous	Spontaneous	Spontaneous	Spontaneous
AU labels	no	no	yes
Emotion class	3(positive/negative/surprise)	3(positive/negative/surprise)	5(basic emotions)
Tagging	Emotion category	Emotion category	Emotion/FACS

.

SMIC [3]: SMIC was composed of 164 video clips of 16 subjects, which were divided into three different categories, such as Positive (51 samples), Negative (70 samples) and Surprise (43 samples). There are three versions of SMIC data: one is the high-speed camera (HS) of 100fps, the ordinary camera (VIS) version of 25 fps, and the near-infrared camera (NIR) version of 25 fps. HS cameras were used to record all data, while vis and NIR cameras were used to record data from the last eight subjects. Emotion classes are only based on participants’ self-report.

CASME II [4]: CASME II contains 247 micro-expression video clips from 26 subjects. All samples were recorded by high-speed cameras at 200 fps. The sampling resolution is 640 × 480 pixels. These samples were divided into five categories: Happiness (32 samples), Surprise (25 samples), Disgust (64 samples), Repression (27 samples) and Others (99 samples). Unlike SMIC, the AU label of CASME II follows the facial action coding system (FACS).

4.2. Experimental Setting

This section aims to verify the adaptability and performance of the method in this paper through relevant pairing and analysis experiments. MTCNN has good accuracy for face detection compared to traditional face alignment methods. Therefore, MTCNN is used for face positioning on the SMIC and CASME II data images, and the face region is cropped according to the final bounding box to obtain an average face area image size of 300 × 250.

Meanwhile, this article uses two optical flow filtering algorithms (OFF2BD, OFF3WD) to reduce the impact by low-quality semantic images. Through OFF2BD-LBPTOP, OFF3WD-LBPTOP, OFF2BD-LBPSIP, OFF3WD-LBPSIP, OFF2BD-STCLQP, OFF3WD-STCLQP, OFF2BD-STLBPIP, OFF3WD-STLBIP methods and classic micro-expressions recognition methods LBP-TOP, LBP-SIP, STCLQP, STLBP-IP for performance comparison. In order to verify the effectiveness of the optical flow filtering algorithm proposed in this paper on micro-expression recognition, several groups of experiments are designed as follows:

• Sensitivity of the parameter $\theta$ to normalization function $\omega$.
• Sensitivity of the threshold $\delta$ to OFF2BD.
• Sensitivity of the threshold $\left(\alpha ,\beta \right)$ to OFF3WD.
• The comparison of performance scores between refined algorithms (OFF2BD-LBPTOP, OFF3WD-LBPTOP, OFF2BD-LBPSIP, OFF3WD-LBPSIP, OFF2BD-STCLQP, OFF3WD-STCLQP, OFF2BD-STLBPIP, OFF3WD-STLBIP) and original algorithms (LBPTOP, LBPSIP, STCLQP, STLBPIP).
• The confusion matrix of recognition accuracies for each micro-expression on CASMEII and SMIC dataset.

Choices of Parameters $\theta$: The size of the cropped face image is equalized to 300 × 250 for SMIC and CASME II. A parameter $\theta$ is adopted in the computation of normalized weighting to adjust the influence of the magnitudes of optical flows over the index. Among then, through the image calculation of CSAMEII and SMIC the unit pixel magnitudes of each optical flow belong to the interval of $\left[3\times {10}^{-4},3.5\times {10}^{-3}\right]$. Parameter $\theta$ setting range is (0,1), and its relationship with the normalized weight can be represented by Figure 3. As shown in Figure 3, when setting $\theta$ within a certain range values, e.g., setting $\theta <0.005$, the Normalized Weight steady increase as the unit pixel magnitudes of optical flow increases and the difference between high-quality semantic optical flow weights and low-quality semantic weights is evenly expanded. When setting $\theta$ to larger values, the weight gap of high-quality semantic optical flow and that of low-quality semantic optical flow will be narrowed down, this will directly lead to the effect of the next optical flows weight filtering. Empirically, it is suggested that the parameter $\theta$ be set in the interval of $\left[5\times {10}^{-4},0.01\right]$. Whereas $\theta =0.001$, the weighted interval with a more uniform distribution will be obtained. Therefore, this paper chooses $\theta =0.001$ as the weight coefficient parameter of the experiment.

Performance Evaluation Index: Accuracy is a classic method to judge the performance of the model in machine learning. At present, most judgments on micro-expression recognition models are judged by recognition accuracy. The formula is as follows:

$$Acc=\frac{n^{*}}{n^{*}+m^{*}}$$

(9)

where $n^{*}$ is the number of correct micro-expressions predicted by the model, $m^{*}$ is the number of incorrect micro expressions predicted by the model, $n^{*}+m^{*}$ is the total number of micro expressions in the micro-expression dataset.

F1 score (F1 Score) is an indicator used to measure the accuracy of the binary classification model in statistics, and it also takes into account the precision rate $p$ and recall rate $r$ of the classification model. Therefore, by calculating the recall rate $r$ combined with the precision rate $p$ to further calculate F1-score. In the prediction of the $i$-th micro-expression class, the number $N_i^{TP}$ of true positive cases of the $i$-th micro-expression class that is predicted to be correct, the number $N_i^{FP}$ of false positive cases that the $i$-th micro-expression class is predicted to be incorrect and the number $N_i^{FN}$ of false negative cases that the non- $i$-th micro-expression class are predicted as $i$-th micro-expression class. The precision rate $p_i$ and the recall rate $r_i$ is calculated as follows:

$$p_i=\frac{N_i^{TP}}{N_i^{TP}+N_i^{FP}}$$

(10)

$$r_i=\frac{N_i^{TP}}{N_i^{TP}+N_i^{FN}}$$

(11)

where $N_i^{TP}+N_i^{FP}$ is the total number of the real $i$-th micro-expression class, $N_i^{TP}+N_i^{FN}$ is the number of $i$-th micro-expression class after prediction.

Calculation of micro-expression precision $p$ and recall rate $r$ of N ME classes, F1-score is defined as:

$$F=\frac{1}{N}{\displaystyle \sum }_{i=1}^N\frac{2p_i\times r_i}{p_i+r_i}$$

(12)

where $p_i$ and $r_i$ are the precision and recall of the $i$-th micro-expression class.

The values of the above Accuracy and F1-score performance evaluation indicators are in the range $\left[0,1\right]$, and the larger the indicator value, the better the recognition performance of the framework model for micro-expressions.

4.3. Experiments and Analysis

The LOSO method is used to the CASMEII and SMIC data are experimentally reproduced based on the LBP-TOP feature, and the selection of optimized relevant parameters value to further complete the parameter adjustment of the OFF2BD and OFF3BD algorithms. The video clips are divided to $N\times N\times 2$ blocks. The recognition results of micro-expressions are shown in

Table 3. The performance of different uniform LBP-TOP features on CASME II and SMIC.

$\left({\mathit{R}}_{\mathit{x}},{\mathit{R}}_{\mathit{y}},{\mathit{R}}_{\mathit{z}}\right)$	CASME II			SMIC
	$3\times 3$	$5\times 5$	$7\times 7$	$3\times 3$	$5\times 5$	$7\times 7$
	Acc (%)/F1-Score			Acc (%)/F1-Score
(1,1,2)	43.43/0.34	46.88/0.38	45.82/0.38	49.42/0.48	50.48/0.50	45.58/0.45
(1,1,3)	45.27/0.36	47.04/0.39	46.98/0.39	48.62/0.48	49.28/0.49	47.85/0.48
(1,1,4)	46.91/0.39	47.65/0.41	48.24/0.42	47.57/0.47	46.04/0.45	43.25/0.42
(2,2,2)	45.14/0.36	45.32/0.37	46.24/0.39	43.61/0.43	47.32/0.46	49.54/0.51
(2,2,3)	45.43/0.33	44.92/0.32	47.15/0.40	47.76/0.49	51.83/0.51	44.38/0.47
(2,2,4)	42.15/0.32	46.31/0.38	44.31/0.37	45.17/0.43	49.55/0.47	43.47/0.45
(3,3,2)	43.47/0.35	43.28/0.34	45.69/0.37	46.04/0.47	44.71/0.42	47.85/0.49
(3,3,3)	46.38/0.37	46.93/0.39	43.74/0.35	46.41/0.48	42.52/0.41	46.12/0.48
(3,3,4)	47.22/0.41	45.82/0.35	45.49/0.36	45.79/0.44	43.45/0.45	44.91/0.44

, the best accuracy and F1-score for the CASME II dataset is achieved at 7 $\times$7 blocks, the radius for the spatial plane (XY plane) and the radius for the spatial-temporal planes (XT and YT planes) is achieved at $\left(R_x,R_y,R_t\right)=\left(1,1,4\right)$. Whereas the best accuracy and F1-score for the SMIC is achieved at 5 $\times$5 blocks and $\left(R_x,R_y,R_t\right)=\left(2,2,3\right)$.

4.3.1. Sensitivity of the Threshold $\delta$ to OFF2BD

In this part, the dataset is divided into train and test sets multiple times with different specifications by CASME II and SMIC. Through $\theta$ the comparison of the micro-expression recognition improvement rate obtained by the various threshold $\delta$ of multiple models in different dataset, and the robustness and generalization of the model of the OFF2BD algorithm under different thresholds are verified by the curve comparison of the test set and train set maps. LBPTOP and OFF2BD-LBPTOP are used as the benchmark method to compare the experimental algorithm. The improvement rate of the improved algorithm and the baseline algorithm is used as the evaluation index. The two-dimensional curve graph of threshold and improvement rate is fitted by many experiments on the data under different thresholds.

In this experiment, the CASME II and SMIC data are divided into multiple different dimensions of the dataset, and the effect of fitting the training data and test data of each dimension can be obtained, as shown in Figure 4. From the overall perspective, the analysis of the sets of data shows that the test data has a similar performance curve to the train data. In other words, the OFF2BD algorithm model has good generalization ability. From a partial analysis, when the threshold is at $0.5\pm 0.1$, the optical flow processing of OFF2BD algorithm has a good performance improvement. Among them, SMIC peaked around 0.43, and CASME II peaked around 0.5. When the threshold is $\left(0.2,0.4\right)$, the micro-expression recognition performance gradually increases with the increase of the optical flow threshold, and when the threshold is greater than 0.55, the micro-expression recognition performance gradually decreases. Meanwhile, when the threshold is too high, as a large number of video clips are eliminated, some high-quality clips will be classified into negative domain, which will lead to a negative improvement rate values compared with the baseline. In summary, the OFF2BD algorithm further proves that the two-branch decision method is suitable for solving the uncertainty of image semantic quality.

4.3.2. Sensitivity of the Threshold $\left(\alpha ,\beta \right)$ to OFF3WD

Through the analysis of the experimental results of OFF2BD, it is found that the threshold between (0.2, 0.4) has a significant improvement, and the performance gradually decreases after 0.55. The uncertainty domain BND is mainly used to solve the uncertainty of low-quality and high-quality data, and is used as a buffer domain of data to further distinguish the judgment after combining. Based on the above experience, this experiment chooses the threshold $\alpha \in \left(0.2,0.4\right)$ and threshold $\beta \in \left(0.5,0.8\right)$ to experiment on the SMIC. Three-dimensional fitting is through the results generated by the OFF3WD algorithm, and the effect diagram is shown in Figure 5.

The $\alpha$ and $\beta$ increase naturally according to the specified step length, and the range of the uncertainty domain is adjusted to select the optical flow intensity quality. As shown in Figure 5, the recognition rate of ME under the OFF3WD algorithm changes when the threshold range changes. Among them, it is a better effect improvement when the value of $\alpha$ is around 0.35 and the value of $\beta$ is around 0.6. Additionally, the current threshold based on the OFF3WD algorithm also has a better improvement rate on CASME II ME recognition compared to the benchmark method.

4.3.3. Comparison with the State-of-the-Art Methods

LBPTOP was reproduced in this experiment and combined with the OFF2BD algorithm and the OFF3WD algorithm to obtain the ME recognition performance score. The experimental results are shown in Figure 6. The effects of OFF2BD-LBPTOP and OFF3WD-LBPTOP are significantly better than the traditional LBPTOP method on the test data. The accuracy has improved on the CASME II and SMIC about 2–3% by OFF2BD compared with the benchmark method. The OFF3WD algorithm has a more accurate judgment on the quality of optical flow semantics, which has increased by 0.5–2% compared with the OFF2BD algorithm. All in all, the experimental data preliminarily shows that the OFF2BD and OFF3WD algorithms can effectively mine the semantic information of micro-expressions and complete the image noise with low morphological semantic quality.

This section compares derivative algorithms (OFF2BD-LBPSIP, OFF3WD-LBPSIP, OFF2BD-STCLQP, OFF3WD-STCLQP, OFF2BD-STLBPIP, OFF3WD-STLBIP) and original algorithms (LBP-SIP, STCLQP, STLBP-IP) to further verify the improvement of the two filtering algorithms on the ME recognition. Two evaluation criteria (F1-score and Accuracy) were used to evaluate the performance scores. The experimental results are shown in

Table 4. Comparison with different ME recognition methods.

Method	CASME II		SMIC
	Acc (%)	F1-Score	Acc (%)	F1-Score
LBPTOP (reproduced)	48.24	0.42	51.83	0.51
LBPSIP [28]	49.73	0.47	47.42	0.46
STCLQP [25]	58.39	0.58	64.02	0.63
STCLBP-IP [24]	59.51	-	57.93	-
BI-WOOF [34]	58.46	0.60	61.29	0.61
FHOFO [37]	60.52	0.61	58.61	0.58
OFF2BD-LBPTOP	50.46	0.44	53.52	0.52
OFF2BD-LBPSIP	52.77	0.51	50.36	0.50
OFF2BD-STCLQP	60.87	0.62	64.84	0.63
OFF2BD-STCLBP-IP	61.57	-	59.18	-
OFF3WD-LBPTOP	51.68	0.46	54.35	0.54
OFF3WD-LBPSIP	53.45	0.51	51.57	0.52
OFF3WD-STCLQP	61.28	0.61	65.41	0.65
OFF3WD-STCLBP-IP	60.53	-	61.83	-

. The proposed method achieves comparable recognition performances with the state-of-the-art methods on SMIC and CASME II. Compared with LBPSIP, the accuracy of OFF2BD and OFF3WD On CASME II is increased from 49.73% to 52.77% and 53.45%, which is 3–4% higher, and the F1 score is improved from 0.47 to 0.51. The accuracy of SMIC was improved by 2.94% and 4.15%, and the F1 score was improved from 0.46 to 0.5. Compared with STCLQP and STCLBPIP, the accuracy of the two algorithms on CASME II is improved by 2.48%, 2.89% and 2.06%, 1.02% respectively, and 0.82%, 1.39% and 1.79%, 3.90% on SMIC.

Among the feature methods of the current experiment, the two algorithms in this paper have achieved the greatest recognition accuracy and F1 score on STCLQP features. Especially, after three-way decision optical filtering, the recognition accuracy of STCLQP reached 65.41% on SMIC and 61.28% on CASME II, which have higher accuracies and F1-score than the state-of-the-art methods such as BI-WOOF and FHOFO. Simultaneously, the micro-expression feature obtained by the algorithm also has a good improvement effect on the F1-score.

In order to further evaluate the performance of the method in this paper to identify different micro-expressions, we perform optical flow filtering algorithm application and performance testing on the STCLQP features, and use the LOSO cross-validation method to generate a confusion matrix with the best recognition accuracy. Figure 7 and Figure 8 show the confusion matrix of OFF2BD-STCLQP and OFF3WD-STCLQP on SMIC and CASMEII respectively.

On the task of micro-expression detection, STCLQP can easily detect most micro-expression videos from non-micro-expression videos, and the OFF2BD and OFF3WD algorithms select high-quality semantic images, which further deepens the feature expression ability of STCLQP. For the micro-expression recognition on the SMIC database, OFF2BD has a corresponding improvement in the recognition accuracy of “positive” and “negative” classes compared with STCLQP, while the recognition of “surprise” shows a stable trend. OFF3WD further enhances the expression of morphological and semantic features of the “negative” class and make the recognition accuracy of the “negative” category reach 0.7. On the CASME II database, the micro-expression categories have become more complex and diverse, and the CASME II database contains five categories. From the confusion matrix of the CASMEII data set, it is found that the recognition and expression ability of the two algorithms proposed in this paper on STCLQP has generally increased. It shows the best improvement in identifying “happy” and “disgust”, and there are also minor improvements in “depressed”, “surprise” and “other”. Among the five classes, “depression” and “disgust” are the most difficult to identify micro-expression, while the classes of “surprise”, “happiness” and “other” are easy to identify. The algorithm improves the morphological semantic information of different classes, enhances the expression ability of different classes of features, and reduces the probability that the samples of “happiness”, “surprise”, “disgust” and “depression” are wrongly classified as “other” micro expression classes.

5. Conclusions

We propose an optical flow filtering micro-expression recognition algorithm (OFF2BD and OFF3WD) based on two-branch decision and three-way decision. The algorithm integrates the three-way decision method of rough set theory and adopts the three-way division idea of uncertain affairs, and improves the judgment of POS, BND and NEG domain conditions. The algorithm divides the morphological semantic quality of temporal images by redefining the weight value of optical flow to obtain a more robust and high-quality video images collection. We have seen that these two algorithms have improved their effects on the classic feature extraction methods. For STCLQP features, the optimized recognition rate and F1 score of the test are achieved on the SMIC data set. Optical flow filtering strengthens the feature intensity information on the time axis and eliminates images with low morphological semantic quality.

For future work, we pay more attention to the changes brought about by the facial action areas, and combine the idea of the OFF3WD algorithm to conduct semantic mining on the XY axis and the time axis T to reduce the impact of noise in different dimensions. In addition, not only limited to the method test of different features, we are designing the connection between different classifiers with our algorithm.