Refinement of Background-Subtraction Methods Based on Convolutional Neural Network Features for Dynamic Background

Abstract

Advancing the background-subtraction method in dynamic scenes is an ongoing timely goal for many researchers. Recently, background subtraction methods have been developed with deep convolutional features, which have improved their performance. However, most of these deep methods are supervised, only available for a certain scene, and have high computational cost. In contrast, the traditional background subtraction methods have low computational costs and can be applied to general scenes. Therefore, in this paper, we propose an unsupervised and concise method based on the features learned from a deep convolutional neural network to refine the traditional background subtraction methods. For the proposed method, the low-level features of an input image are extracted from the lower layer of a pretrained convolutional neural network, and the main features are retained to further establish the dynamic background model. The evaluation of the experiments on dynamic scenes demonstrates that the proposed method significantly improves the performance of traditional background subtraction methods.

1. Introduction

Background subtraction methods are widely applied in the realm of computer vision applications, for example, object tracking [1], moving object detection [2], intelligent video surveillance [3], and human–computer interaction [4]. Detailed overviews of background subtraction methods are available in [5,6]. In the past decade, several traditional background subtraction methods have been developed to segment foreground objects from the background in various video scenes. A parametric probabilistic background model, the Gaussian mixture model (GMM) [7], takes advantage of multiple normal distributions to fit the changes of pixels in order to deal with dynamic backgrounds, such as slow-moving objects, swaying trees, and water rippling. The Kernel density estimation (KDE) [8], a widely used nonparametric background method, estimates the distribution of pixels directly from the previously observed data without making any assumptions of the underlying distribution. Barnich et al. proposed using the visual background extractor (ViBe) method [9] for a sample-based background model. This method establishes the background model by aggregating previous input pixel information. The ViBe method adopts a stochastic renewal strategy for the phases of estimating and updating the background model and assumes that the information between adjacent pixels is transmitted [10]. Similarly, the pixel-based adaptive segmenter (PBAS) [11], which models the background by a history of recently observed pixel values, is also a nonparametric background modeling paradigm that introduces cybernetics to update threshold and background adaptively. The procedure for foreground detection by PBAS is similar to that by ViBe. Inspired by the low-cost and highly-efficient ViBe method, St-Charles et al. presented the local binary similarity segmenter (LOBSTER) [12] method for background subtraction, which uses a spatiotemporal binary similarity descriptor instead of relying solely on pixel intensities as its core component. The SuBSENSE method was proposed by St-Charles [13]. This method adopts the principle of sample consistency and feedback mechanism, and therefore the background model adapts to the diversity of a complex background. The fundamental idea of these traditional methods is as follows: First, establish a statistical background model with historical data. Subsequently, calculate the defined distance between a current pixel and the background. Once the pixel deviates from the background obviously, a foreground pixel is confirmed. Meanwhile, continue running the background maintenance to continuously update the background model. These methods have been widely used for detecting moving objects and they have achieved impressive results.

The robust principal component analysis (RPCA) is a popular method used in background models [14]. In models using this method, the structure of the video is decomposed into low-rank and sparse matrices, which provide a suitable framework to separate moving objects from the background. Bouwmans [15] reviewed various models based on RPCA. Recently, inspired by the impressive achievement of deep learning, some researchers applied deep neural networks for background subtraction [16]. Braham and Van Droogenbroeck [17] made use of convolutional neural networks (CNNs) to deal with the scene-specific problem of background subtraction, naming their model ConvNet, where the network architecture was inspired by the LeNet-5 network [18]. Their model consists of four phases: using a time median operation to extract gray background images from multiple initialization frames, generating the dataset for a specific scene, training the network, and background subtraction. In ConvNet, input image patches and corresponding background image patches are fed to the network to predict the probability of foreground pixels. Following this fundamental idea, Babaee et al. [19] trained a single CNN using various video scenes from the CD2014 dataset [20] to perform segmentation of the video sequences. Baustita et al. [21] applied a simple CNN for vehicle detection. Similarly, the input of the CNN is the observed patch image and the corresponding background patch image. The fully connected layer in the traditional convolutional network is replaced by the convolutional layer, which reduces the amount of computation and the architecture of fully convolutional networks (FCNs) can be trained end-to-end. Encouraged by the recent success of transfer learning and FCNs [22] for semantic segmentation, researchers leveraged FCNs to build background modeling. Zeng and Zhu [23] proposed a multiscale fully convolutional network architecture which utilized different layer features for background subtraction. Cinelli [24] varied different features of the ResNet [25] architectures to optimize them for background or foreground separation by exploring the advantages of FCNs. Yang et al. [26] also improved FCNs by using three atrous convolution branches with a different dilate to extract spatial information from different neighborhoods of pixels. Wang et al. presented a convolutional neural network based on a multiscale cascade [27]. In this network, a multiscale CNN model with cascade structure is adopted to model the correlation between adjacent pixels to enhance spatial correlation. The FgSegNet-M20 method [28] is based on a triplet CNN and a transposed convolutional neural network. This method uses the VGG-16 Net to embed an image in multiple scales into the feature space, and, in the decoder part, it uses a transposed convolutional network to learn a mapping from feature space to image space. Subsequent research resulted in the proposed FgSegNet-S and FgSegNet-V2. Li et al. [29] judged the location of targets in surveillance scenes using an adaptive deep neural network (ADCNN) method. Zhang et al. [30] designed a deep learned features based block-wise method with a binary spatiotemporal background model. Zhao et al. designed a deep pixel distribution learning model (DPDL) for background subtraction [31]. Lim et al. used an encoder–decoder-structured convolutional neural network for background subtraction [32]. Wang et al. used BGSNet-D to detect moving objects in the scenes where color information was not available [33]. Yu et al. [34] combined background subtraction and CNN for moving objects detection in pumping-unit scene.

The aforementioned deep neural network methods have contributed to background subtraction. However, it is evident that they have encountered several limitations. One limitation, for example, is that most of the learning algorithms are supervised methods that are merely used to deal with scene-specific situations. Therefore, for these methods, many historical video data are needed first to train the background model, and then the moving object is detected in the new video. Moreover, the trained model only detects the known targets in the category of the training set, which is an even more stringent limitation of these methods. When the unknown objects in the category appear in the current video scene, it produces incorrect judgments. Another limitation is that many of the algorithms have extremely high computational costs. For example, the deep convolutional neural network is in high demand as a computing resource, and often includes tens of millions of parameters; therefore, the computations required to train this network take a great deal of time and money. Some methods adopt an approach based on patch-wise, a method that segments an image into many patches, and then convolutes each patch separately. However, this approach generates a lot of redundancy. Some other methods adopt a fully convolutional network architecture in semantic segmentation for each pixel. Each pixel is divided into two categories: the foreground and the background. However, the computation load is still very large. Likewise, the storage space required by CNN is also relatively large. In comparison, a traditional background subtraction method (BGS) is computed with a lower computational load.

Table 1. An overview of background subtraction methods.

Categories	Methods, Authors	Features	Learning Types	Scene
traditional methods	GMM	hand-crafted	unsupervised	universal
	KDE
	ViBe
	PBAS
	LOBSTER
	SuBSENSE
	RPCA
CNN-based	ConvNet [17]	learned	supervised	specific
	Babaee [19]
	Baustita [21]
	Wang [27]
	FgSegNet [28]
	Li [29]
	Zhao [31]
	Lim [32]
	Wang [33]
Fully CNNs	Zeng [23,35]	learned	supervised	specific
	Cinelli [24]
	Yang [26]

shows an overview of these methods.

In this paper, a novel framework based on CNN is proposed to improve background subtraction. In the proposed method, the lower convolution layers are used to extract the general features of a video. These general features usually have multiple channels, and, in each channel, there is a lot of redundant information. A few of these channels contain the main information of the image, thus these main channels are applied to background subtraction, which not only reduces the amount of data processing, but also removes the redundant interference in the scene. Using a traditional background subtraction method to process the main features that are extracted by CNN further improves the performance of the background subtraction method. In comparison to the existing deep neural network methods of background subtraction, there are three advantages of the proposed method. First, the proposed method is a general unsupervised method, and there is no need to train models based on the historical data of specific scenarios. Second, the proposed method requires only a few calculations. Third, only one of the lower layers of the pretrained CNN is used to extract the general features of videos.

The remainder of this paper is organized as follows. Section 2 presents a detailed account of the proposed method. Section 3 describes the profound experiments on a dataset of surveillance videos that verified the validity and feasibility of the proposed method. Finally, Section 4 provides some conclusions.

2. The Proposed Method

The outline of the proposed method is illustrated in Figure 1. The input image I is fed into a pretrained convolution layer to obtain a series of convolution feature images F, and then a few feature images that are closest to the original image are selected to merge a new convolution feature image $\widehat{I}$. Subsequently, background subtraction methods are used to deal with $\widehat{I}$ and obtain the foreground image (FG).

It is well known that CNN has made great achievements in computer vision, image processing, and other fields. Jason et al. [36] showed that the features learned by the CNN are hierarchical. Basically, general features, such as color, edge, line, and so on, are extracted from the lower layers of the neural network. More advanced specific features are extracted from the deep layers; however, these specific features only present the scene features in the corresponding training set, and they cannot be used in other scenarios. On the contrary, the lower layers are used to extract the underlying common features of other scenes. Our framework was based on the VGG16 network [37] architecture trained on ImageNet. The first convolution layer, conv1_1, of the VGG16 is applied to extract the general features of the images. The detailed configuration parameters of the conv1_1 layer used in this study are shown in

Table 2. Parameter configuration of conv1_1 layer.

Layer	Input Size	Kernel	Stride	Padding	Output Size
conv1_1	W × H × 3	3 × 3	1	yes	W × H × 64

. The size of the input image was W × H × 3, where W is the width and H is the height of the image. The feature images with 64 channels were generated by conv1_1, and therefore the output size was W × H × 64.

The conv1_1 captured low-level information and retained higher spatial resolution. For example, the input image I was processed by the convolution layer conv1_1 of the VGG16. Then, the feature image (F) was obtained.

$$F=conv\left(I\right)$$

(1)

Conv is the convolution operation with a kernel size of 3 × 3 and the stride is 1. F contains 64 convolution feature images, and the resolution of each convolution feature image is the same as the input image. The feature images are shown in Figure 2. Our observations of these convolution feature images show that some features take the form of noise-like images, while other features reflect the major objects of the input image, such as a canoe and people. Based on these observations, the main convolution features that best represent the input image are reserved, while other features that have a low correlation with the input image are abandoned. Consequently, some irrelevant interference factors are removed from the input image. The performance of the background subtraction method is further refined by using the reserved convolution features instead of the corresponding input images.

To reduce the complexity of the computations, the two convolution feature images which have the strongest correlation with the input image are extracted to merge a new merging image. This new image is fed into the background subtraction method and replaces the input image. To evaluate the correlation strength between the convolution feature image and the input image, the value of the peak signal-to-noise ratio (PSNR) is calculated for the convolution feature image and the input image. The higher is the PSNR value, the stronger is the correlation between the convolution image and the input image.

$$PSNR=10·lo{g}_{10}\left(\frac{MA{X}_I^2}{MSE}\right)$$

(2)

where $MA{X}_I$ is the maximum pixel value of the input image and the data in this study are in the range of 0–1. $MA{X}_I$ is 1, and MSE is the mean square error between the grayscale of input image I and the convolution feature $F_k$.

$$MSE=\frac{1}{HW}{\displaystyle \sum }_{i=1}^H{\displaystyle \sum }_{j=1}^w{\Vert I\left(i,j\right)-F_k\left(i,j\right)\Vert }^2$$

(3)

where H and W are the height and width of the image, respectively; and k = 1, 2, …, 64 denotes the number of the convolution image.

The two feature images with the maximum values of PSNR are shown in Figure 3. The indices are the 32nd and 26th in Figure 2, respectively. In the same scene, the feature distributions extracted by the convolution layer for different video frames are the same. The ranking of the PSNR values which are obtained from one input frame are used as the indices of the convolution feature of the subsequent frames. Therefore, the indices of the two features with maximal PSNR only need to be computed once. Two convolution feature images of the maximum PSNR value are merged into a new image and the formula is as follows:

$$\widehat{I}=\frac{\widehat{F}-min(\widehat{F})}{max(\widehat{F})-min(\widehat{F})}$$

(4)

where $\widehat{F}=F_{max}+F_{sec}$, and $F_{max}$ and $F_{sec}$ are the two feature images with the maximum values of PSNR. Obviously, $\widehat{I}$ is normalized to a range 0–1.

Once the synthetic image is obtained, the foreground binary image (FG) is finally obtained by the traditional background subtraction method.

$$FG\left(i,j\right)=\{\begin{array}{c}1, if BGS\left(\widehat{I}\left(i,j\right)\right) is foreground\\ 0, if BGS\left(\widehat{I}\left(i,j\right)\right) is background\end{array}$$

(5)

where i and j are the position coordinates of the pixels; $i=1,2,\dots ,H; j=1,2,\dots ,W$; and BGS is a background subtraction method.

Algorithm 1 depicts the overall procedure of the proposed method.

Algorithm 1: The proposed method.

Input: $I_t$: input image of video, t = 1, 2, …, L, L is the frame number of the video; Output: $FG$: the foreground of the input image. $F\leftarrow conv\left(I_1\right)$. Calculate the PSNR values between $F$ and the grayscale image of $I_1$. Find the numbers $in{d}_{max}$ of $F_{max}$ and $in{d}_{sec}$ of $F_{sec}$. $t\leftarrow 2$ While $t\le L$ Do $F\leftarrow conv\left(I_t\right)$ $F_{max}=F\left(in{d}_{max}\right)$ and $F_{sec}=F\left(in{d}_{sec}\right)$ Calculate $\widehat{I}$ using (4) Obtain the foreground image $FG$ using Equation (5) $$t\leftarrow t+1$$ End While

3. Experiments

To verify the performance of the proposed method, the dynamic background videos from CDnet [20] were used to test the method. The CDnet is an expanded change detection benchmark dataset, which provides a realistic, diverse set of videos and covers a wide range of detection challenges. The main advantage of this dataset is that it provides the ground truths for all video frames. Therefore, the dataset can be used to quantitatively evaluate the performance of various background subtraction methods and rank them. The category of dynamic background contains six videos depicting outdoor scenes with strong background motion. The details of this category are presented in

Table 3. Dynamic background videos.

Videos	Size	Number of Frames	Dynamic Scenes
Boats	320 × 240	7999	water rippling
Canoe	320 × 240	1189	water rippling
Fountain01	432 × 288	1184	fountains
Fountain02	432 × 288	1499	fountains
Overpass	320 × 240	3000	waving trees
Fall	720 × 480	4000	waving trees

.

The performance of the background modeling methods was evaluated at the pixel level. The background modeling methods classified pixels into foreground or background. The following six metrics were used to evaluate the performance of the background modeling methods:

$$Recall=\frac{TP}{TP+FN}$$

$$Precision=\frac{TP}{TP+FP}$$

$$FPR=\frac{FP}{FP+TN}$$

$$FNR=\frac{FN}{TP+FN}$$

$$PWC=100\times \frac{FN+FP}{TP+FN+FP+TN}$$

$$F-measure=2\times \frac{Precision\times Recall}{Precision+Recall}$$

where TP is the number of correctly detected foreground pixels, TN is the number of correctly detected background pixels, FP is the number of background pixels that are incorrectly marked as the foreground pixel, and FN is the number of foreground pixels that are incorrectly marked as the background pixels. F-measure is the comprehensive evaluation index which represents the weighted harmonic mean and ranges between 0 and 1. Obviously, the higher are recall, precision and F-measure, the better is the performance, while the lower are FPR, FNR, and PWC, the better is the performance.

We applied the proposed framework to the following traditional background modeling methods: GMM, SuBSENSE, PBAS, KDE, LOBSTER, and ViBe. Details for the implementation of these traditional methods are available in the BGSlibrary [38], and, consequently, these implementations were used in this study. Figure 4 presents the refinements of the traditional background subtraction methods using the proposed method for foreground detection in dynamic scenes. It is noteworthy that the convolution feature methods (BGS_con) further suppressed the disturbance of dynamic background such as branch swaying, water ripple, and so on as compared with the corresponding background subtraction methods (BGS).

Comparisons of the performance between these background subtraction methods (BGS) and the corresponding convolution feature methods (BGS_con) are shown in

Table 4. Comparisons between BGS and BGS_con by the proposed method.

Category	Methods	Recall	FPR	FNR	PBC	Precision	F-Measure
dynamic background	GMM	0.7568	0.0440	0.2432	4.7859	0.2109	0.3062
	GMMcon	0.7683	0.0232	0.2317	2.7252	0.3484	0.4338
	SuBSENSE	0.7713	0.0006	0.2287	0.4084	0.8915	0.8132
	SuBSENSEcon	0.8228	0.0017	0.1772	0.3645	0.8228	0.8138
	PBAS	0.5634	0.0005	0.4366	0.7252	0.8787	0.6154
	PBAScon	0.6095	0.0005	0.3905	0.5800	0.8848	0.6712
	KDE	0.8562	0.0788	0.1438	7.8212	0.1062	0.1855
	KDEcon	0.4765	0.0041	0.5235	0.9584	0.5691	0.4884
	LOBSTER	0.7646	0.0189	0.2354	2.0795	0.5948	0.5682
	LOBSTERcon	0.6572	0.0047	0.3428	0.7922	0.7411	0.6367
	ViBe	0.5852	0.0100	0.4148	1.3884	0.4521	0.4733
	ViBecon	0.6020	0.0047	0.3980	0.8203	0.6237	0.5739

and illustrates that the convolution feature framework improves the performance of background subtraction methods to some extent.

To show the refinements of the traditional background subtraction methods by the proposed framework intuitively, the average values and standard deviations for F-measure, which are general international standards for binary classification, are presented in Figure 5. In the figure, the blue bars are the F-measure values of traditional background modeling methods while the red bars are the corresponding F-measure values of the convolution feature methods. The KDE method had the highest performance improvement, with an increase of more than 160%. GMM’s performance was improved by 42%, ViBe by 21%, LOBSTER by 12%, PBAS by 9%, and SuBSENSE by 0.1%. SuBSENSE, which is a state-of-the-art BGS method, had very high performance and surpassed most of the other unsupervised BGS methods. When the proposed method was applied, the performances of other background subtraction methods, with the exception of SuBSENSE, improved significantly.

To verify the ability of the proposed method, Neural Response Mixture (NeRM) [39] was selected for comparison. NeRM takes advantage of rich deep features extracted from the neural responses of an efficient, stochastically-formed deep neural network for constructing Gaussian mixture models to detect motion in a scene.

Table 5. Comparisons of improvement ratio of GMM.

Methods	Recall	FPR	FNR	PBC	Precision	F-Measure
NeRM [39]	50.0%	80.0%	−21.9%	74.5%	32.4%	40.0%
Proposed	1.5%	47.3%	4.7%	43.1%	65.2%	41.7%

shows the improvement ratio of GMM by NeRM and the proposed method respectively, in which the higher the ratio, the better the performance. The most representative F-measure evaluation metric of the proposed method is better.

For further evaluating the performance of the proposed method, shadow videos are used to test. Shadow is composed of six videos with both strong and soft moving and cast shadows [20].

Table 6. Shadow videos.

Videos	Size	Number of Frames	Dynamic Scenes
backdoor	320 × 240	2000	Shadow and illumination change
bungalows	360 × 240	1700
busStation	360 × 240	1250
copyMachine	720 × 480	3400
cubicle	352 × 240	7400
peopleInShade	380 × 244	1199

shows the details of this category. The refinement results are illustrated in Figure 6.

Table 7. Comparisons between BGS and BGS_con by the proposed method.

Category	Methods	Recall	FPR	FNR	PBC	Precision	F-Measure
shadow	GMM	0.7020	0.0124	0.2980	2.6666	0.7128	0.6862
	GMMcon	0.7190	0.0127	0.2810	2.7459	0.7497	0.7240
	SuBSENSE	0.9469	0.0081	0.0531	0.9960	0.8627	0.8998
	SuBSENSEcon	0.9091	0.0085	0.0909	1.1977	0.8661	0.8850
	PBAS	0.6917	0.0076	0.3083	2.1497	0.8487	0.7455
	PBAScon	0.7189	0.0079	0.2811	2.2203	0.8655	0.7729
	KDE	0.9269	0.0757	0.0731	7.6690	0.3913	0.5176
	KDEcon	0.6899	0.0122	0.3101	2.5481	0.7901	0.7275
	LOBSTER	0.8038	0.0063	0.1962	1.4937	0.9008	0.8452
	LOBSTERcon	0.8773	0.0070	0.1227	1.1703	0.8739	0.8709
	ViBe	0.6600	0.0064	0.3400	2.0278	0.8622	0.7397
	ViBecon	0.6944	0.0062	0.3056	1.9924	0.8877	0.7746

lists the quantitative comparison results. As shown in Table 7, GMM, KDE, PBAS, LOBSTER and ViBe methods were refined in a certain degree. Since SuBSENSE had high performance, this method was not improved.

Since the PSNR is simple and easy to realize, it was adopted to determine the feature images involved in the proposed method in our experiments. Structural similarity index measure (SSIM) is another well-known objective image quality metric. A simple analytical link exists between the PSNR and the SSIM and the PSNR is more sensitive to Gaussian noise than the SSIM [40]. The SSIM used to choose the convolution feature images was evaluated in the experiment as PSNR.

Table 8. Comparisons of PSNR and SSIM for GMM on dynamic scenes.

Videos	Methods	Recall	FPR	FNR	PBC	Precision	F-Measure
boats	GMM	0.5418	0.0627	0.4582	6.5735	0.0635	0.1136
	GMMPSNR	0.5563	0.0264	0.4437	2.9650	0.1417	0.2259
	GMMSSIM	0.3483	0.0272	0.6517	3.1088	0.0749	0.1232
canoe	GMM	0.5762	0.0673	0.4238	8.2353	0.2734	0.3708
	GMMPSNR	0.6212	0.0261	0.3788	4.1436	0.5190	0.5655
	GMMSSIM	0.4163	0.0474	0.5837	6.6359	0.2439	0.3076
fall	GMM	0.8427	0.0830	0.1573	8.4436	0.1638	0.2744
	GMMPSNR	0.8513	0.0553	0.1487	5.7039	0.2306	0.3629
	GMMSSIM	0.7391	0.0482	0.2609	5.1936	0.2168	0.3352
fountain01	GMM	0.8934	0.0274	0.1066	2.7464	0.0375	0.0721
	GMMPSNR	0.9096	0.0211	0.0904	2.1222	0.0506	0.0958
	GMMSSIM	0.7835	0.0168	0.2165	1.7016	0.0372	0.0710
fountain02	GMM	0.9162	0.0058	0.0838	0.6076	0.3173	0.4713
	GMMPSNR	0.9031	0.0026	0.0969	0.2856	0.5251	0.6641
	GMMSSIM	0.7211	0.0021	0.2789	0.2732	0.4212	0.5318
overpass	GMM	0.7704	0.0178	0.2296	2.1093	0.4100	0.5352
	GMMPSNR	0.7686	0.0077	0.2314	1.1307	0.6237	0.6886
	GMMSSIM	0.4666	0.0108	0.5334	1.7817	0.3694	0.4124

shows the comparison results. As can been observed, the PSNR metric was better than the SSIM.

The input images of dynamic scenes usually contain a large amount of redundant information that results in deterioration of background modeling performance while the original images of a video are used to model the background. The convolution of CNN, which was pretrained on ImageNet, extracted the underlying features of the input image. The dynamic redundancy features were abandoned, and the main features were merged into a new input image. On the basis of the new image, the background model, which was built by the BGS method, had better performance. In the proposed framework, only one lower layer convolution of CNN was used. Compared with a whole CNN network, which is often hundreds of megabytes in size, one convolution layer is only several megabytes. Therefore, the performance of the traditional background subtraction method is further improved without increasing computational load. As an example, for an input image of 320 × 240 in size, the memory required to extract the convolution feature from the input image by applying the conv1_1 layer of the VGG16 is 320 × 240 × 64 = 4.9 M, and the weight is 3 × 3 × 3 × 64 = 1728.

4. Conclusions

In this paper, we present an unsupervised, simple and universal framework that takes advantage of lower layer features of pretrained CNN for refinements of dynamic background subtraction methods. The framework extracts the low-level features of input images and merges the representation features of the input image into a new feature image. The background subtraction methods produce more accurate foreground object detection results by modeling the key features. To verify the performance of the proposed method, the dynamic background database was used to carry out experiments. Experiments showed that the proposed method significantly refined the traditional background subtraction methods for dynamic scenes.