A Novel Moving Object Detection Algorithm Based on Robust Image Feature Threshold Segmentation with Improved Optical Flow Estimation

Abstract

The detection of moving objects in images is a crucial research objective; however, several challenges, such as low accuracy, background fixing or moving, ‘ghost’ issues, and warping, exist in its execution. The majority of approaches operate with a fixed camera. This study proposes a robust feature threshold moving object identification and segmentation method with enhanced optical flow estimation to overcome these challenges. Unlike most optical flow Otsu segmentation for fixed cameras, a background feature threshold segmentation technique based on a combination of the Horn–Schunck (HS) and Lucas–Kanade (LK) optical flow methods is presented in this paper. This approach aims to obtain the segmentation of moving objects. First, the HS and LK optical flows with the image pyramid are integrated to establish the high-precision and anti-interference optical flow estimation equation. Next, the Delaunay triangulation is used to solve the motion occlusion problem. Finally, the proposed robust feature threshold segmentation method is applied to the optical flow field to attract the moving object, which is the. extracted from the Harris feature and the image background affine transformation model. The technique uses morphological image processing to create the final moving target foreground area. Experimental results verified that this method successfully detected and segmented objects with high accuracy when the camera was either fixed or moving.

1. Introduction

Image applications are ubiquitous, and the availability of various sensors and platforms such as satellites, mobile phones, and drones is constantly increasing. Indoor positioning, video surveillance, and other social applications have an increasing demand for image reliability and current information. Sensor systems for real-time indoor and outdoor mapping, such as stereo-RGB cameras, RGB-D depth cameras, and other sensors’ consumer markets, are constantly growing and low-cost sensors are ubiquitous, especially as the technology in smartphones and the Internet of Things grows increasingly advanced, which is a strong impetus for the development of real-time monitoring and mapping capabilities in indoor and outdoor environments. Moving object detection is an important aspect of real-time detection and mapping and includes detecting and extracting rigid or non-rigid moving objects in the complex background environments of an image sequence or video format. It has been one of the most prominent research areas of image vision and is frequently used in various fields such as video surveillance, assisted driving, scene analysis, target tracking, and target localization [1,2,3,4,5]. Moving objects are primarily detected using three methods: consecutive frame difference [6], background subtraction [7], and optical flow [8]. The consecutive frame difference method extracts moving objects by differencing two or more successive frames; however, it can only detect targets in a fixed backdrop because determining the target detection threshold is quite tricky. The problem of voids often accompanies the results; moreover, this method is easily disturbed by illumination variation and image noise [9,10,11]. The background subtraction method recovers the moving object by differentiating the background model from the current frame. Nevertheless, the algorithm’s accuracy is affected by the background modeling method, making it highly complicated. Moreover, this approach is deeply disturbed by the ‘ghosting’ problem [12,13,14]. The optical flow method detects moving objects by calculating the flow vector between each pixel in the current frame and the pixel of the same name in the next frame. Next, it thresholds the image’s optical flow field to identify moving objects [15,16,17]. Compared with the other two methods, the optical flow method requires no previous knowledge. It can directly estimate the moving object area and generate the motion parameters of moving objects. The optical flow method exhibits higher accuracy and robustness and has a better detection performance for targets with scale changes or deformations.

The optical flow method of moving object detection and extraction essentially entails thresholding, classifying, and morphologically processing the optical flow vector values of each image’s pixel in turn and then extracting the foreground region of the moving object. Adrian G. Bors et al. [18] were the first to apply the image optical flow estimation to the detection segmentation of moving objects. They used the median radial basis function neural network to divide the images into different regions, such as the still image and motion information, and fuse other motion regions of the same target to extract moving objects. Sasa Galic et al. [19] suggested object detection by optical flow and clustering for medical images. Moreover, they proposed the Euclidean norm of the optical flow vector and used it as the clustering features. However, their algorithm had low accuracy and a poor extraction effect with a moving background. Shui-gen Wei et al. [20] introduced the Otsu algorithm to the optical flow method of target detection. They first normalized the optical image flow as a grey image. Next, they used the Otsu algorithm to translate it into a binary image. The Otsu algorithm is the primary method of the optical flow method for motion target detection [21,22,23,24]. Their target extraction results were better under stationary background conditions but poorer results were obtained with a moving background. Furthermore, Sengar et al. [16] calculated the total optical flow by computing the optical flow between three consecutive frames and combining the HS optical flow vector fields between the current frame and the previous frame and between the current frame and the next frame. Then, they used Otsu thresholding to detect moving objects with high detection accuracy; however, the secondary optical flow estimation significantly increased the time consumption. Han et al. [25] used the homography matrix to construct the optical flow background model and RANSAC to solve and optimize the model. Next, they determined the adaptive threshold of optical flow according to the estimated background model. The optical flow threshold size can be adaptively adjusted with the change in the camera motion. However, the algorithm accuracy is affected by the background model, and the moving object extraction is disturbed by the ‘ghosting’ problem. Wei Sun et al. [26] determined the average amplitude value of optical flow as the threshold and put results into the HSV color space to mark and remove the shadow. However, the detected moving object still has hole problems so the method is only used on a rigid moving object.

Above all, the three issues in detecting moving objects by the optical flow algorithm should be resolved; meaning, first, the influence of illumination variation and the false detection of moving objects when there are illumination changes in the image sequence. Second, the ‘ghosting’ problem, due to the change in the position of the moving object which causes the background area to be covered by the region of the moving target in the previous frame while the background area appears in the next frame without being obscured and is mistakenly detected as the moving object area. Third, camera jittering or the movement that occurs when the camera may be non-fixed or dithered in acquiring image frames, which can change the image sequence background. Still, the commonly used optical flow moving object detection method is only used for a fixed background.

A moving object detection algorithm based on optical flow estimation and robust feature points was examined in this paper to address the problems of illumination variation, ‘ghosting’, dynamic texture interference, and camera jittering or movement. First, the algorithm predicts the image’s optical flow field. Then, according to the optical flow residual principle, the local optical flow calculation of the Lucas–Kanade (LK) optical flow method is integrated into the Horn–Schunck (HS) optical flow method to constitute the optical flow equation data item to estimate the optical flow vector. Next, the Delaunay triangulation is applied to solve the motion occlusion problem, establishing the high-precision optical flow estimation energy equation.

Furthermore, it is necessary to further threshold the optical flow vector field to determine the moving object’s foreground area. The Otsu algorithm is often applied in the threshold processing of the optical flow method of moving object detection because it has better detection results in a fixed background environment. However, the object segmentation results are poor when the camera is jittering or moving. To solve this problem, this study proposes an optical flow threshold algorithm based on robust feature points. The threshold is determined according to the Harris robust feature points of the background area instead of the traditional optical flow Otsu threshold method to extract the foreground area of moving targets. The proposed algorithm can then be applied in either condition: when the camera is moving or when it is fixed. Importantly, this algorithm offers more accurate detection and extraction results for rigid and non-rigid targets.

The main contributions are made as follows:

• We adopted a dense optical flow estimation algorithm that combines the HS pyramid large displacement optical flow method with the LK local optical flow method to improve the robustness of the algorithm and introduced the DF-β based on an edge-preserving strategy as the smoothing term of the optical flow energy equation to ensure edge integrity.
• We adopted the non-quadratic penalty function and Delaunay triangulation occlusion determination to reduce the inaccuracy of optical flow estimation caused by the ‘ghost’ problem and to improve the accuracy of moving object detection.
• We proposed a moving object detection algorithm based on optical flow thresholding of Harris robust feature points. According to the MSAC algorithm, the background model is calculated, and the background robust feature points are screened. The optical flow value of the Harris robust feature points is used as the threshold and can be applied to both a static or moving camera.

2. Materials and Methods

2.1. Algorithm Framework

Figure 1 illustrates the framework of the algorithm for detecting moving objects with advanced estimated optical flow robust feature threshold segmentation which is primarily comprised of two parts. First, the LK optical flow local calculation is combined with the HS dense optical flow estimation to formulate the energy equation. The image pyramid is then constructed for estimating a large displacement optical flow. Similarly, the Delaunay triangulation is used to determine the motion occlusion and iteratively compute the image optical flow field from up to down. Second, the Harris feature points are extracted. The affine transformation matrix of the background model between the current and next frames is then collected. Moreover, robust feature points are selected, and the optical flow value at the robust feature points is applied as the optical flow threshold to extract the moving object. Then, the binary image of the extracted moving object is processed by image morphology via expansion, corrosion, open operation, and closed operation.

2.2. Improved Optical Flow Estimation Method

2.2.1. Improved Optical Flow Estimation Energy Equation

The optical flow field estimation results directly affect the accuracy of moving object detection and extraction. To enhance the algorithm’s anti-interference of illumination variation and the robustness in the case of the rotation and warping of moving objects or background regions, the energy equation of optical flow estimation in this paper is designed to include two parts: data term $E_{Data}$ and smoothing term $E_{Smoot\mathrm{h}}$. The energy equation is expressed as follows: $E=E_{Data}+\alpha E_{Smoot\mathrm{h}}$.

The data term is designed to fuse image brightness conservation and first derivative gradient conservation:

$$E_{Data}=\underset{\mathsf{\Omega }}{\iint }\mathsf{\Psi }[{{\mathsf{\gamma }}_1\left(I\left(x+u,y+v,t+1\right)-I\left(x,y,t\right)\right)}^2+{\mathsf{\gamma }}_2{(\nabla I\left(x+u,y+v,t+1\right)-\nabla I\left(x,y,t\right))}^2]dxdy$$

(1)

where ${\mathsf{\gamma }}_1$ and ${\mathsf{\gamma }}_2$ are the coefficients of the grey and gradient term [27], respectively, $\mathsf{\Psi }\left(\mathrm{X}\right)=\sqrt{X^2+{\xi }^2},\mathrm{a}\mathrm{n}\mathrm{d} \xi$ is the non-square penalty function [28]. ξ is just for the numerical reason and can be set to a fixed value to ensure the Ψ can always be a convex function in the minimization process. Due to the small positive constant ξ, we set $\xi =0.001$.

The spatial gradient was defined as ${\nabla }_{3}I={[I_x{I}_y{I}_t]}^T$. The partial derivatives of the spatial gradient in the x and y directions are ${\nabla }_3{I}_x={[I_{xx}{I}_{xy}{I}_{xt}]}^T$ and ${\nabla }_3{I}_y={[I_{xy}{I}_{yy}{I}_{yt}]}^T$. $\omega ={[uv1]}^T$ represents the optical flow vector based on the assumption of brightness constancy: $[({{\nabla }_3{I}_x)}^T+({{\nabla }_3{I}_y)}^T]\omega =0$. The data term can be expressed as follows:

$$E_{Data}=\underset{\mathsf{\Omega }}{\iint }\mathsf{\Psi }\{{\omega }^T[{\mathsf{\gamma }}_1{{\nabla }_{3}I({\nabla }_{3}I)}^T+{\mathsf{\gamma }}_2({\nabla }_3{I}_x{({\nabla }_3{I}_x)}^T+{\nabla }_3{I}_y{({\nabla }_3{I}_y)}^T)]\mathsf{\omega }\}dxdy$$

(2)

The HS global optical flow method can determine the dense image optical flow field. However, because the data term of its energy equation is constructed based on the changes in brightness and gradient, the HS method has low robustness to image noise and dynamic texture such as shaking leaves, water, and fog. In contrast, the LK local optical flow estimation method exhibits higher anti-interference of those elements. Therefore, the local convolution kernel function $K_{\rho }$ of the LK method data term was introduced [29]. $K_{\rho }$ represents the Gaussian kernel function with a variance of $\rho$ for improving the robustness of noise and dynamic texture. Therefore, the data term fuses the HS and LK optical flow energy equations as follows:

$$E_{Data}={\iint }_{\mathsf{\Omega }}\mathsf{\Psi }\{{\omega }^T{K}_{\rho }\ast [{\mathsf{\gamma }}_1{{\nabla }_{3}I\left({\nabla }_{3}I\right)}^T+{\mathsf{\gamma }}_2({\nabla }_3{I}_x{\left({\nabla }_3{I}_x\right)}^T+{\nabla }_3{I}_y{({\nabla }_3{I}_y)}^T)]\mathsf{\omega }\}dxdy$$

(3)

The smoothing term is designed as the edge-preserving strategy $\mathrm{D}\mathrm{F}-\mathsf{\beta }$(decreasing scalar function) [30]. The smoothing term is defined as follows:

$$E_{Smooth}={\iint }_{\mathsf{\Omega }}\mathsf{\Psi }[{(e}^{-\lambda \left|\nabla I_t\right|}+\mathsf{\beta })({\left|\nabla u\right|}^2+{\left|\nabla v\right|}^2)]dxdy$$

(4)

$$\mathrm{s}.\mathrm{t}. \nabla u={(u_x{u}_y)}^T,\nabla v={(v_x{v}_y)}^T$$

2.2.2. Occlusion Determination Based on the Delaunay Triangulation

The motion occlusion problem, also known as ‘ghosting’, has always been difficult to solve in optical flow estimation and moving object detection. The movement of the objects between the previous and subsequent images causes the formation of a ‘ghost’ when the background area covered by the moving foreground in the last frame is exposed in the following frame. Figure 2 shows that from $I_t$ to $I_{t+1}$, the occluded point ${\mathrm{P}}_1$ in the $I_t$ image appears in $I_{t+1}$ because of the movement of foreground objects, whereas points ${\mathrm{P}}_2$ and ${\mathrm{P}}_3$ change from appearance to occlusion. From $I_{t+1}$ to $I_{t+2}$, the occluded point ${\mathrm{P}}_2$ changes from occlusion to appearance. All these points bring a false optical flow and lead to ‘ghosting’ problems. Two methods were used to solve this problem: the first is the general introduction of the punishment mechanism using the non-square penalty function. The second is to combine the Delaunay triangulation to ascertain occlusion decisions in the moving edge regions.

Only relying on the non-square penalty function cannot solve the occlusion problem well; the Delaunay [31] triangulation occlusion judgment was also incorporated to increase the accuracy of moving object extraction. The Delaunay structure and optical flow estimation proposed by Congxuan Zhang et al. [32] were applied to determine whether the point is a moving occlusion point. Because motion occlusion occurs in the edge region of the moving target, the occlusion is determined by detecting the brightness variations of the pixels’ triangular grid in the edge regions of the moving object. The optical flow estimation after occlusion processing is defined as ${\mathsf{\omega }}_{\mathsf{\Lambda }}$, and its calculating equation is defined as follows:

$${\mathsf{\omega }}_{\mathsf{\Lambda }}=\mathsf{\Lambda }\mathsf{\omega }$$

(5)

$$\mathrm{s}.\mathrm{t}. \mathsf{\Lambda }=\left\{\begin{array}{cc}1& \hfill \mathrm{normal}\\ 0& \hfill \mathrm{occlusion}\end{array}\right.$$

where $\mathsf{\omega }={(uv)}^T$ is the optical flow vector field and $\mathsf{\Lambda }$ represents the occlusion of each pixel. The Λ judgment method is divided into two steps:

The motion edge region information $R_{edge}$ in the image can be extracted according to the estimated image optical flow field. The calculation equation is expressed as follows:

$$R_{edge}=\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}(\omega ,Laplacian)$$

(6)

where the ‘edge’ represents the boundary extraction algorithm and the Laplacian edge detection algorithm is used.

According to the estimated image optical flow vector field, each pixel point position $B={(ij)}^T$ in $I_t$ was calculated and added to $\omega ={(uv)}^T$ at the same points to determine the corresponding point $I_{t+1}(i+uy+v)$ in $I_{t+1}$. Next, the grey difference $\Delta I$ of $I_{t+1}$ and $I_t$ of the estimated optical flow corresponding points were calculated. The calculation equation is defined as follows:

$$\Delta I=I_{t+1}\left(B+\omega \right)-I_t(B)$$

(7)

when $\Delta I=0$, the corresponding position of the continuous frames has a consistent grey level, indicating that point B is not occluded, $\mathsf{\Lambda }=1$.

When ΔI ≠ 0, the occlusion situation needs to be determined on a case-by-case basis. For the under-determined point in the $R_{edge}$ of $I_t$, the change in the brightness value of the triangle to which it belongs is calculated. Similarly, the Delaunay triangulation about the brightness difference map is constructed to determine the occlusion. Specifically, $B_1\left(i,j\right)$ is taken in $I_t$ along with the adjacent coordinate points $B_2\left(i+1,j\right)$ and $B_3(i,j+1)$, which constitutes the determination of the Delaunay triangulation. The Delaunay triangulation area can be calculated as follows:

$$\Delta I_{\Delta }={\phi }_1\Delta I_{B_1}+{\phi }_2\Delta I_{B_2}+{\phi }_3\Delta I_{B_3}$$

(8)

where ${\phi }_1$, ${\phi }_2,$ and ${\phi }_3$ represent the weights of three points $B_1$, $B_{2,}$ and $B_3$, respectively, because they are adjacent coordinate points: ${\phi }_1={\phi }_2={\phi }_3=1/3$. The greyscale difference was compared between the triangulation and the pixel points in a 25 × 25 neighborhood size around it to determine the occlusion of the point in the domain. The specific determination method is: if $\Delta I_B>\Delta I_{\Delta }$, the point is occluded and $\mathsf{\Lambda }=0$. If $\Delta I_B<\Delta I_{\Delta }$, the point is not occluded and $\mathsf{\Lambda }=1$.

The pseudo-code of improved optical flow estimation is as Algorithm 1:

Algorithm 1 Improved optical flow estimation

Input: $I_t$ and $I_{t+1}$ in image sequence Output: Optical flow estimation result ${\mathit{\omega }}_{\mathsf{\Lambda }}$ of $I_t$ at time t   Build the image pyramid $I_t^k$ and $I_{t+1}^k$ from bottom to up, which $k\in (\mathrm{1,2},\cdots n)$ QCalculate the initial value of the top-level optical flow ${\omega }_t^1$   for k = 2 to n do    The optical flow of current layer ${d\omega }_t^k$ is calculated according to $E=E_{Data}+\alpha E_{Smoot\mathrm{h}}$ Calculate the initial optical flow of next layer ${\omega }_t^{k+1}={\omega }_t^k+{d\omega }_t^k$   end   The optical flow of the current image $I_t$ is updated as $\omega ={\omega }_t^n$   Detecting the edge of optical flow ${\omega }_t^n$ as $R_{edge}=\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}(\omega ,sobel)$   Delaunay triangulation is established in the edge area $R_{edge}$   Calculating the grey difference: $\Delta I=I_{t+1}\left(B+\omega \right)-I_t(B)$   if $\Delta I=0$    $\mathsf{\Lambda }=1$   else    for i = 1 to K      Calculating the change in brightness of triangle $B_1^i$ point:      $\Delta {\mathrm{I}}_{\Delta }={\phi }_1\Delta I_{B_1}+{\phi }_2\Delta I_{B_2}+{\phi }_3\Delta I_{B_3}$      if $\Delta I_B>\Delta I_{\Delta }$       $\mathsf{\Lambda }=0$      else       $\mathsf{\Lambda }=1$      end end Estimating the optical flow after occlusion judgment: ${\omega }_{\mathsf{\Lambda }}=\mathsf{\Lambda }\omega$

Figure 3 shows the effects of occlusion determination on image optical flow estimation. The optical flow in the right side of the head and the bottom of the arm of the moving target is termed a ‘ghost’ due to motion occlusion. The optical flow in the ‘ghost region’ should be the same as the background region, which produces a false optical flow in the occlusion area. It has a wrong optical flow estimation. The optical flow estimation after occlusion removal can be corrected more after motion occlusion processing, thereby improving the accuracies of optical flow estimation and the subsequent motion target region extraction. The optical flow estimation map following occlusion removal can be viewed to be more corrected after motion occlusion processing, which increases the accuracies of optical flow estimation and subsequent motion target area extractions.

2.3. Moving Object Detection Based on Robust Feature Threshold Segmentation

Estimating the optical flow between successive frames can determine the optical flow vector field and compute the motion vector of each point. However, the position and state of the moving target in the image cannot be detected. The optical flow vector must be further processed to determine the specific moving target area. Hence, a moving object detection method based on the threshold segmentation of Harris robust background feature points was suggested to detect and segment moving object foreground regions in the stationary and moving backgrounds.

First, the Harris feature points in the current frame are extracted. Their positions in the next frame are calculated according to their corresponding optical flow values. Then, based on the principle that the image background conforms to the graphical affine transformation, the Harris feature points and their corresponding positions are combined in the next image. According to the M-estimators Sample Consensus (MSAC) algorithm [33], the feature points that do not belong to the background model and their optical flow values are excluded from solving the affine transformation homography matrix on the image background model. Robust Harris feature points are known as the Harris feature points of the background region. The robust feature points’ optical flow value is used as the threshold to segment the foreground region of the moving object; hence, the binary image of the foreground of the moving target is obtained. Finally, morphological image processing is used to deal with the issues of pores, discontinuities, and irregular moving edges in the binary image and create the foreground mask of the moving target regions.

The algorithm mainly includes three steps: (1) image optical flow vector field normalization processing; (2) optical flow thresholding processing based on Harris robust feature points; and (3) image morphological processing of the target foreground region.

(1) Image optical flow vector field normalization processing

Because the estimated optical flow vector field cannot be directly thresholded, it is necessary to normalize the optical flow vector field to transform the optical flow vector field of the image into a grey value image based on the optical flow.

First, the Euclidean norm of the optical flow vector value $\overrightarrow{\omega }(\mathrm{x},\mathrm{y})=[uv]$ at each pixel point is calculated and used as the optical flow grey value V$(x,y)$ at this point. The calculation formula is expressed as follows:

$$V\left(x,y\right)=\sqrt{{u(x,y)}^2+{v(x,y)}^2}$$

(9)

Next, the obtained image optical flow grey value V$(x,y)$ is normalized to determine the normalized grey image $V_{norm}$:

$$V_{norm}\left(x,y\right)=\frac{V\left(x,y\right)-V_{min}}{V_{max}-V_{min}}$$

(10)

$$\mathrm{s}.\mathrm{t}. V_{max}=\max (\sqrt{{u\left(x,y\right)}^2+{v\left(x,y\right)}^2}); V_{min}=\min (\sqrt{{u\left(x,y\right)}^2+{v\left(x,y\right)}^2})$$

(2) Optical flow thresholding processing based on Harris robust feature points

This study found that the feature points in the common scene are fixed no matter whether the camera is moving or stationary. Furthermore, due to the background movement caused by the camera movement, the position change of the feature points in the background area of the front and back frames conforms to the image affine transformation, whereas the relative position of the feature points in the moving object area changes in the front and back frames, which does not conform to the affine transformation. Therefore, an affine transformation matrix was built to solve the background model between the adjacent frames.

Because the optical flow estimate of the image pixel depicts a certain error, the optical flow estimate of the pixel point has a higher estimation accuracy at the image brightness change area and the image gradient change area. Moreover, the affine transformation matrix [25] calculation does not require the global pixel points and their optical flow values. Therefore, the feature points and their optical flow value were extracted. Next, the homography matrix $M$ of the affine transformation of the background region model was calculated according to the feature points and their optical flows. Because the background region of the previous frame image can be transformed into the background region of the next frame image by the affine transformation, the homography matrix $M$ of the affine transformation of the background model at the time of $t\to t+1$ should satisfy the following equation:

$$M_{t\to t+1}\ast {I_t}^B=I_{t+1}^B$$

(11)

where ${I_t}^B={\left(ij1\right)}^T$ represents each pixel in the image background area at time t; $I_{t+1}^B$ represents each pixel in the image background area at time $t+1$; and $M_{t\to t+1}$ represents the background model affine transformation matrix calculated from time $t$ to $t+1$.

According to the optical flow estimation, the corresponding points in the next frame image ${I_{t+1}}^B$ can be determined by adding each pixel ${(ij1)}^T$ in ${I_t}^B$ into its estimated optical flow value $\omega ={(uv0)}^T$ and is expressed as follows:

$${I_{t+1}}^B={I_t}^B+\omega$$

(12)

Because the affine transformation of the background region and the optical flow estimation of the same image are equal to the corresponding points in the next frame image ${I_{t+1}}^B$ (combining Equations (11) and (12)), the affine transformation matrix calculation with a background model of size $3\times 3$ can be determined as follows:

$$M_{t\to t+1}\ast {I_t}^B={I_t}^B+\omega$$

(13)

The MSAC algorithm was used to solve and optimize Equation (13) to exclude the feature points that do not belong to the background model because of the uncertainty of the location of the extracted feature points which may be located in the background area or the foreground area. To increase the computational efficiency, feature points and the corresponding optical flow estimation were calculated to calculate the affine transformation matrix on the optical flow of feature points. This process finally filters out the stable feature points in the background model area, known as robust feature points.

Figure 4 shows the outcomes of robust feature point extraction, wherein Figure 4a,b depict the current and the next frames, respectively. Figure 4c shows the feature points extracted by the features from the accelerated segment test (FAST) feature extraction algorithm [34] in the previous image. Figure 4d presents the FAST robust background feature points, which are selected from the FAST feature points by background model affine transformation matrix calculation and MSAC methods with their optical flow values. Figure 4e,f illustrate the speeded-up robust features (SURF) feature points [35] and the SURF robust background feature points, respectively. Figure 4g,h show the Harris feature [36] points and the Harris robust background feature points, respectively. The results demonstrate less feature information extracted by the FAST algorithm in Figure 4c. The final selected background robust feature points are concentrated in one area, which easily causes estimation errors in the calculated background model affine transformation matrix. It is difficult to screen out the correct background feature points. The SURF feature points in the moving target area are misinterpreted as the background feature points in Figure 4d. Figure 4e shows that the number of feature points extracted by SURF is too much, which increases the computational complexity of the background model matrix. The final extracted robust feature points also constitute the feature points in the moving target area (Figure 4f). Figure 4g,h demonstrate that the Harris feature extracts the feature points at the strong inflection points, which are more robust. The Harris feature points are quantity reasonable and distribution balanced. The screened robust feature points were shown in Figure 4h with no mistakes. The results revealed that the Harris robust background feature points are more accurate than FAST and SURF and are more suitable for estimating the background change model. All the robust background feature points screened exhibit the highest accuracy.

After determining the robust feature points, the moving object foreground area is segmented according to the optical flow value of the robust feature points, and the moving object area is retrieved. The maximum optical flow value of the robust feature points was used as the optical flow threshold for moving object extraction. The maximum grey value constant in the background feature points is used as a threshold, and the binary image of the foreground region of the moving target is extracted.

(3) Image morphological processing of the target foreground region

The rough extraction result of the moving object foreground area is established following the threshold processing of the image. The generated binary images contain many problems, such as salt and pepper noise, the discontinuity of the moving object extraction area, and an image hole in the foreground area due to the image noise and the optical flow estimation errors. Such concerns can be addressed by corrosion, expansion, open operation, closed operation, and other operations in image morphological processing.

Figure 5 shows the moving object extraction results. Figure 5a illustrates the normalization grey image of the image optical flow vector field. Figure 5b shows that the rough extraction result of the moving target is based on the optical flow’s grey value with the Harris robust feature points as the threshold. Figure 5c presents the morphological image processing results; the edge of the foreground region after morphological processing is smoother. Figure 5d shows the mask display for the extraction result. This paper’s moving target detection algorithm can accurately extract the moving target region.

The pseudo-code of moving object detection is as Algorithm 2:

Algorithm 2 Moving object detection

Input: Image $I_t$, $I_{t+1}$ and optical flow estimation $\omega$ at time t Output: moving objects area R  Calculating optical flow field $\overrightarrow{\omega }(x,y)=[uv]$ to optical flow grey value $V\left(x,y\right)$  Normalized grey image $V_{norm}$  Extracting Harris feature points of image $I_t$  Calculating corresponding points in image $I_{t+1}$: ${I_t}^B+\omega ={I_{t+1}}^B$  Calculating affine transformation matrix of background model: $M_{t\to t+1}\ast {I_t}^B={I_t}^B+\omega$  Calculating optical flow threshold of background robust feature points  Extracting binary image of moving objects region  Morphological processing

3. Experiment and Results

3.1. Experimental Environment and Evaluation Indicators

(1) Experimental environment

To verify the algorithm’s effectiveness, the Max Planck Institute Sintel Flow Dataset [37], a commonly used public dataset for image optical flow estimation, was applied to verify the accuracy of the proposed enhanced optical flow field estimation and moving object detection based on robust feature points. To reflect the algorithm’s accuracy, the optical flow estimation and comparison algorithms are LK [38], HS pyramid [39], Black & Anandan (B&A) [40], and Brox [41], among others. These are frequently used optical flow estimation methods. The optical flow threshold segmentation comparison algorithm is Otsu, which is the most commonly used algorithm for moving target detection based on optical flow.

(2) Evaluation indicators

In this paper, the accuracy of optical flow estimation [32] is evaluated by endpoint error (EPE) and average angular error (AAE), where EPE represents the Euclidean distance between the estimated optical flow value $(u,v)$ and the real optical flow value $(U,V)$. AEE represents the average angle between the estimated optical flow vector $\omega ={(u,v)}^T$ and the real optical flow vector $W={(U,V)}^T$. The calculation formulas of the two indexes are presented as follows:

$$\mathrm{E}\mathrm{P}\mathrm{E}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\sqrt{{(u_i-U_i)}^2+{(v_i-V_i)}^2}$$

(14)

$$\mathrm{A}\mathrm{A}\mathrm{E}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\mathrm{arc}\cos (\frac{u_i{U}_i+v_i{V}_i+k^2}{\sqrt{u_i^2+v_i^2+k^2}\sqrt{U_i^2+V_i^2+k^2}})$$

(15)

where $N$ represents the total number of image pixels; ${(u}_i,v_i)$ is the optical flow vector estimation value; $(U_i,V_i)$ is the real optical flow vector; and $k$ is the number of frames between the two frames used to calculate the optical flow. Because the true optical flow values provided by the public dataset are between the adjacent frames, the datasets used for validation in this paper are all calculated as the optical flow values between two adjacent frames; hence, in this paper, $k=1$.

In this paper, the performance test of the moving target detection algorithm primarily includes three indicators [42]: $E_{Pre}$ represents precision, $E_{Rec}$ represents recall, and $E_{FM}$ represents the F-measure. The calculation formulas of the three evaluation indexes are expressed as follows:

$$E_{Pre}=\frac{TP}{TP+FP}$$

(16)

$$E_{Rec}=\frac{TP}{TP+FN}$$

(17)

$$E_{FM}=\frac{2\ast E_{pre}\ast E_{rec}}{E_{pre}+E_{rec}}$$

(18)

$TP$ represents true positive, which is the number of correctly detected foreground pixels of the moving object detection results. $FP$ represents false positive, which is the number of pixels that mistakenly identify the background area as the moving object foreground area. Therefore, $TP+FP$ in Equation (16) represents the number of targets’ foreground pixels detected by the algorithm, and FN (false negative) represents the number of pixels in the moving object foreground that is not seen in the detection result and is marked as the background area. $TP+FN$ in Equation (17) represents the real number of pixels in the moving object foreground area.

3.2. Experiment and Analysis

Figure 6 shows the comparison of optical flow estimation. The left to right sides depict the sequence images of Alley2, Ambush4, and Bamboo2 in the MPI Sintel Flow Final dataset, and the moving objects are all non-rigid. The up-to-down areas of Figure 6 represent the current frame image, the next frame image, the true optical flow of the current frame, and the optical flow field calculated by LK, HS pyramid, B&A, Brox optical flow estimation, and the improved optical flow calculation. The optical flow estimation results reveal that the sparse optical flow value obtained by the LK method cannot determine the complete optical flow of the moving object area; hence, it is tricky to extract the moving target area of the optical flow threshold. The optical flow in the background (Bamboo2) or the moving target area cannot be distinguished in the Bamboo2 sequence images. The HS pyramid method produces more blurred results at the images’ edges, and there is a false optical flow detection in the background area in the+ Ambush4 sequence images. The B&A method has a better optical flow estimation result in the edge area than the HS pyramid method. However, the optical flow estimation error is too large when the optical flow value of the non-rigid moving target area is quite different, such as in the Ambush4 sequence. The Brox algorithm has apparent errors. The proposed improved optical flow estimation algorithm exhibits a higher accuracy and better edge retention. The optical flow effect is also better when the optical flow values have large differences in the moving object area. In addition, the HS and LK optical flows were combined to deal with the problem of an optical flow calculation error in dynamic texture regions because of the optical flow estimation. There was no problem with optical flow error detection in the background area of Ambush4 when the environment was complex and had fog and snow. The optical flow detection effect of the moving target showing in Bamboo2 is also better when the domain occludes it.

Figure 7 illustrates the EPE and AAE comparisons of the optical flow estimated results in Figure 6. It also shows that the two indexes of the optical flow estimation algorithm in this paper are lower than other algorithms for different image sequences with a higher estimated optical flow accuracy.

Figure 8 displays the segmentation results of moving target extraction corresponding to the different optical flow field estimations of each image frame in Figure 6. The robust feature threshold segmentation method was compared with the Otsu threshold method to prove its effectiveness and applicability. This method was also performed on different optical flow estimation methods such as LK, B&A, HS pyramidal, Brox, and this study’s results. The findings proved that this study’s robust feature threshold segmentation method could be applied to different optical flow estimation algorithms. Figure 8 presents the Otsu threshold moving object extraction results and Figure 8 represents the results of this study’s robust Harris feature threshold moving object extraction. According to the LK optical flow thresholding segmentation results of three image sequences, neither Otsu nor the proposed algorithm is suitable for sparse optical flow thresholds to extract moving objects. In the Alley2 sequence, the Otsu and this study’s threshold method failed because of the error of the optical flow estimation by the Brox algorithm. Otsu can only extract a part of the moving object, but this study’s method achieved better extraction results. This is because the optical flow amplitude of the head of the moving object is quite different from the body part, and the optical flow is also generated in the background. The Otsu can only segment a part of the foreground area with noticeable optical flow differences. This study’s optical flow estimation algorithm presented a higher accuracy and more accurate final extraction result. In the Bamboo2 sequence, the optical flow amplitude in the foreground area is significantly higher than in the background area due to the weak optical flow in the background area; hence, Otsu performs better in the Alley2 sequence. Because the optical flow amplitude of the moving target head region is significantly lower than other target regions, the Otsu fails to detect the head region completely in optical flow estimation results. In contrast, this study’s algorithm has a better segmentation. In the Bamboo2 sequence, some pieces of Bamboo2 are an occlusion obstacle as the background area generates optical flow, which should not be detected as the foreground area with the moving object, and the optical flow amplitude of the background area of the whole frame image is large. Those elements seriously interfere with the segmentation and extraction of the foreground area of the moving object. The threshold segmentation algorithm in this paper does not misinterpret the occlusion obstacle area in B&A and the enhanced optical flow estimation method presented in this study. In summary, the results of the robust feature threshold segmentation proposed in this study are significantly better than the commonly used Otsu threshold processing algorithm and more suitable than the Otsu threshold processing method for the HS pyramid, B&A, and Brox, which are commonly used optical flow estimation algorithms that can segment the moving target area more accurately in static or moving environments. Moreover, the enhanced optical flow estimation algorithms with the proposed robust feature threshold segmentation method in this study achieved better results.

Morphological image processing is required because many problems accompany the detected moving object foreground area. Figure 9 compares the morphological processing results of different optical flow algorithms with the proposed robust feature threshold segmentation method. The top-to-bottom regions of Figure 9 depict the current image, the real foreground area of the moving object, the morphological image operation of HS, B&A, Brox, and this study’s optical flow algorithm with threshold processing. The binary image of the foreground regions of the moving object extracted after morphological processing has no holes and irregular edges. The problem of small areas in the background area has also been addressed.

Figure 10 shows the performance test results of the moving object detection method obtained by the proposed algorithm and the HS pyramid, B&A, and Brox after thresholding and morphological operation processing in this paper. Figure 10a shows the accuracy rate. Figure 10b shows the recall rate. Figure 10c exhibits the all-around performance. The higher the value, the better the performance. Finally, the accuracy of the proposed algorithm is verified.

4. Conclusions

This paper proposes a robust feature threshold segmentation based on improved optical flow estimation to detect moving objects. Several experiments were conducted to prove the effectiveness of this study’s algorithm. This method exhibits obvious superiority compared with the representative methods based on optical flow as follows:

(1) The improved optical flow estimation method combined with HS and LK optical flow methods to reduce the effects of background noise and dynamic texture and improve the robustness and accuracy of optical flow estimation. Additionally, the Delaunay triangulation is introduced to reduce the inaccurate estimation of optical flow caused by the moving object ‘ghost’ problem. The algorithm can obtain high precision robust dense optical flow vector fields which can improve the accuracy and robustness of moving object detection.

(2) A robust feature threshold method was proposed, instead of the commonly used Otsu method, by extracting the feature points and calculating the affine transformation matrix of the background model. Then, the robust feature points were obtained which fulfilled the background model using the robust point optical flow value as a threshold to attract the moving object. Our robust feature threshold method can be used in both static and moving environments. Experimental results show that the threshold greatly improves the range of application scenes of moving object detection based on optical flow estimation.

Although this study has improved the moving object detection algorithm accuracy and application compared with the common moving object detection methods based on optical flow, there are still many aspects to be further studied. In the future, we will focus on the following research:

(1) This study did not identify different moving objects. Therefore, based on the research in this paper, using the motion optical flow information further, to conduct optical flow classification and moving behavior analysis, to achieve multi-objective detailed recognition will be the objective of future works.

(2) Robust feature threshold segmentation combined with an intensive optical flow estimation method based on deep learning will be considered as well as combining the Delaunay triangulation with the convolutional neural network to improve the efficiency of moving object detection in future work.