Multi-Scale Adversarial Feature Learning for Saliency Detection

Abstract

Previous saliency detection methods usually focused on extracting powerful discriminative features to describe images with a complex background. Recently, the generative adversarial network (GAN) has shown a great ability in feature learning for synthesizing high quality natural images. Since the GAN shows a superior feature learning ability, we present a new multi-scale adversarial feature learning (MAFL) model for image saliency detection. In particular, we build this model, which is composed of two convolutional neural network (CNN) modules: the multi-scale G-network takes natural images as inputs and generates the corresponding synthetic saliency map, and we design a novel layer in the D-network, namely a correlation layer, which is used to determine whether one image is a synthetic saliency map or ground-truth saliency map. Quantitative and qualitative comparisons on several public datasets show the superiority of our approach.

1. Introduction

Saliency was originally defined in the field of neuroscience and psychology and later introduced into biology to stimulate the human visual attention mechanism. This term denotes that humans have the ability to find salient regions (regions of interest) in the image. Visual saliency has been incorporated into various computer vision tasks, such as image caption [1], video summarization [2], image retrieval [3], person re-identification [4], object detection [5], etc.

In recent years, most researchers have devoted themselves to the work of computer vision research. Therefore, saliency detection has become a well-known research topic. Due to the rapid development of artificial intelligence, it enables computers to automatically learn features to find a salient object. Existing methods based on saliency detection are mainly divided into two categories, bottom-up approaches and deep learning-based approaches.

Bottom-up saliency detection approaches show that people can focus on RoI (regions of interest) in an image. These methods tend to extract features such as color, intensity, texture and orientation for saliency detection. Cheng et al. [6] proposed a saliency detection method that extracts the color features of an image and performs contrast calculation. In [7], saliency detection was calculated by center-surround differences via extracting features such as color, texture and edges in a local context. There are other saliency detection methods [8,9,10,11] based on traditional hand-crafted features by exploring the complex textural feature difference between the background and the foreground, including textural uniqueness and structure descriptors. Although these bottom-up saliency detection methods can achieve good performance, it fails to capture the semantic information of images that have a complex background by simply using low-level features only. To incorporate semantic information, some saliency detection methods [12,13] employ global features and various priors, but the limited types of priors cannot thoroughly describe the diverse and rich semantic information in images.

In order to use high-level features to compute saliency values of images, some saliency detection approaches were proposed based on the convolutional neural network (CNN). The reason is that CNN shows powerful capabilities in automatically learning semantic features and has been widely used in computer vision, such as image classification and image segmentation. Some saliency detection methods based on deep learning techniques include: MDF [14], DISC [15], LEGS [16] and ELD [17]. Some representative methods include DCL [18], DS [19] and DSSOD [20]. All of the above methods perform image saliency detection by using the high-level features of the deep convolutional network and show superior performance compared to methods using only low-level features.

In recent years, the generative adversarial network (GAN) [21] has become a new research topic. The original GAN model was used to synthesize natural images. Functionally, GAN is composed of the G-network and the D-network. The G-network is generally used to generate fake images to deceive the D-network, and the D network is used to discriminate whether the images are generated or real images. Experiments show that the adversarial learning process can significantly enhance the performance of detection. Therefore, the adversarial features learned by the GAN model and used to generate synthesized images achieve satisfactory performance. Recently, some researchers have shown the potential applications of GAN in other fields of research, such as semantic segmentation [22] and object detection [23], but few methods have been proposed for image saliency detection. These methods include saliency detection by the conditional generative adversarial network (CGAN) [24] and supervised adversarial networks (SAN) for image saliency detection [25]. However, these methods all use a fully-supervised approach to learn the G-network and the D-network for saliency detection. In addition, some parameters are added during the training.

Different from previous work on saliency detection, we propose a new model named “multi-scale adversarial feature learning” (MAFL), which is built upon DCGAN (deep convolutional generative adversarial networks). We construct this model with the following considerations. At first, inspired by the strong ability of the adversarial features learned by GAN for image description, the adversarial features that are also devoted to image saliency are learned, and the synthesized image of the trained G-network becomes the target saliency map. That is to say, the aim of the traditional GAN model is to synthesize natural images, while our proposed MAFL model generates synthesized saliency maps. Secondly, considering that the scale problem is one of the most important cues that affects image saliency detection, we adjust the network structure of the G-network to make it compatible with the saliency detection task. Specifically, for each input image, different from the traditional G-network from which only the features from the top-level are output, the specially designed multi-scale G-network extracts features at multiple levels. Moreover, in order to capture subtle visual contrast on feature maps at each level, three extra convolution layers at each level of the G-network are successively added and performed to extract multi-scale deep contrast features. After obtaining deep contrast features at multiple scales for the input image, we fuse them as one feature vector, and the saliency map is obtained by passing the last soft-max layer. Thirdly, in order to enhance the discrimination ability, we apply a novel layer called the correlation layer to the D-network to discriminate the synthetic saliency map and the ground-truth saliency map accurately to the pixel-level. The detailed architecture of the proposed MAFL method is shown in Figure 1 and illustrated in Section 3.

To sum up, the main works we have done are as follows:

• We present a new multi-scale adversarial feature learning (MAFL) model based on the GAN model for image saliency detection. Instead of generating natural images, our MAFL model can directly obtain the image saliency map via the procedure of saliency map synthesis.
• We specially design a multi-scale G-network from which multiple scales of deep contrast features are extracted to incorporate the scale cue in image saliency detection.
• We provide a novel correlation layer in the D-network, by which the similarity between the synthetic saliency maps and the ground-truth saliency maps is calculated with the matching accuracy up to the per-pixel level.

The contents of each part of our paper are as follows. In Section 2, we analyze and discuss the traditional feature extraction methods and the CNN-based feature extraction methods for saliency detection. We describe the proposed MAFL model in detail in Section 3. In the next section, we make quantitative and qualitative comparisons on several public datasets, and compare our method to the other ten methods. Finally, we summarize the work and give our conclusion.

2. Related Work

We focus on analyzing and discussing the traditional feature extraction methods and the CNN-based feature extraction methods in the saliency detection.

Hand-crafted features: Most of the research work in saliency detection has been devoted to exploiting discriminative features [26,27,28,29,30,31]. Traditional hand-crafted features were the first type of features used. Thus, Itti et al. [32] first performed multi-channel feature extraction and then used the center-surround difference calculation to get feature maps on each feature dimension, followed by fusing these feature maps to obtain the aggregated saliency map. In [33], the author proposed a saliency detection method based on the frequency tuned for computing the saliency value of each pixel in an image. Ma et al. [34] employed color feature extraction in the local region of the image for image saliency estimation. Cheng et al. [6] proposed a method for extracting the color and region feature of an image to perform saliency detection. In [13], the author proposed a saliency measure based on feature uniqueness and spatial consistency. Jiang et al. [28] proposed to directly map region feature vectors to saliency values. Liu et al. [11] performed saliency detection by extracting local and global level features and using the conditional random field (CRF) to fuse these features. Although significant advances have been made, there are some pressing problems, especially when extracting hand-crafted features and performing heuristic fusion to obtain the final feature map. Firstly, the above-mentioned methods cannot detect targets in images with a cluttered background by extracting hand-crafted features. Secondly, these methods cannot detect salient objects with homogeneous highlight regions and accurate boundary information.

CNN-based deep features: Due to the wide application of artificial intelligence and deep learning techniques, a series of CNN-based saliency detection algorithms has appeared in the field of computer vision and significantly improved their performance. For example, Li et al. [14] presented a CNN-based saliency detection method to perform saliency detection by extracting multi-scale features. Zhao et al. [19] proposed a multi-task saliency detection method based on deep CNN, which performed saliency detection by extracting the semantic features of the object. Lee et al. [17] built a complete saliency detection model based on multiple CNN streams to extract multi-level features of the image. In [16], the authors provided a new method based on two CNN streams to extract features of local and global levels to perform saliency detection. In [35], the authors improved the traditional method by using the CNN framework to learn semantic features. Pan et al. [25] established a GAN-based supervised adversarial network model, which used the features of GAN to deal with image saliency detection. In our paper, we present a new multi-scale adversarial feature learning (MAFL) model for image saliency detection. In particular, we build this model, which is composed of two CNN modules: the multi-scale G-network takes natural images as inputs and generates the corresponding synthetic saliency map, and we design a novel layer in the D-network, namely a correlation layer, which is used to determine whether one image is a synthetic saliency map or ground-truth saliency map. Through experimental comparisons, we found that the performance of our approach has been greatly improved.

3. Our MAFL Model

Next, we focus on the structure and principle of the proposed MAFL model. Figure 1 shows the framework of our method. We proposed the saliency detection model, which consists of two deep neural networks, namely the generator network and the discriminator network, respectively. The multi-scale G-network takes natural images as inputs and generates the corresponding synthetic saliency map. In the D-network, we design a novel layer, namely a correlation layer, which is used to determine whether one image is a synthetic saliency map or a ground-truth saliency map. In this section, we will introduce them as follows.

3.1. Multi-Scale Generator Network

In our proposed MAFL model, we aim to design a multi-scale CNN to extract features that are discriminative enough to generate a realistic saliency map. We construct an end-to-end model with two considerations: (1) The designed network needs to be deep enough to generate a realistic saliency map. (2) The network is able to detect subtle visual contrast in the feature map at each level.

Compared with the GAN model, MAFL has a different structure for the G-network. The purpose of our design is to enable the GAN network to meet the task of image saliency detection. Some of our improvements to the G-network are mainly reflected in several aspects. Firstly, our proposed MAFL network consists of four pooling layers. For simplicity, the first two convolutional layers are tied together as $C_1$, and we call each of the pooling layers a level. Since from the first level to the last level, the receptive field is expanded gradually, feature maps generated at each level correspond to features at different scales. Thus, feature maps at all levels are used for fusion of the final saliency map (synthetic saliency map).

Secondly, in order to find more subtle visual contrast in the extracted features at each level, we add an extra three convolution layers after each of the first four max-pooling layers in the MAFL network. By expanding the receptive field of these feature maps, we expect to obtain more subtle features of the natural image. The layers of the parameters in the network are shown in Figure 2. However, after three extra layers of convolutions, the feature maps obtained at each level have different sizes. In order to make them have the same size, we set the step size of the first extra layer added at each level to 8, 4, 2 and 1, respectively. In this way, we obtain four scales of feature maps with subtle visual contrast refined. An example of features learned at different layers is shown in Figure 3. Figure 3a shows the original image, and Figure 3b describes the convolution features learned in the first layer, such as color, contrast, corner and edge features. Figure 3c,d present the output of the 3th and 4th layers, respectively. From these two figures, it can be seen that the salient parts in the feature maps at different levels are highlighted.

At last, in order to enhance the effect of feature learning, we fuse the feature maps at four scales. Then, the fused result is sent into a final convolution layer with a $1\times 1$ convolution kernel, one channel. The convolution is actually performed as a fusion step, and the result after convolution is our multi-scale deep feature map (synthetic saliency map) of the input natural image. Experiments demonstrate that our model and configuration can significantly enhance the stability and improve its performance.

3.2. Discriminator Network

In general, the D-network in the MAFL model has a similar structure as its counterpart in GAN. However, compared with GAN model, MAFL introduces a novel layer to further improve the saliency detection performance, i.e., correlation layer. In the D-network, we assume that the input is a ground-truth saliency map and its corresponding synthetic saliency map. During the forward procedure, we measure the similarity between the two saliency maps via a correlation layer. Here, denote two saliency maps $C_g$ and $C_s$ as the ground-truth saliency map and synthetic saliency map, respectively. The correlation layer is to compare each patch in $C_g$ with each patch in $C_s$, where $3\times 3$ is the patch size. Mathematically, for each location $s_1$ in $C_g$, we compute correlation $C(s_1,s_2)$ only in the neighborhood of $C_s$ centered at $s_2$, and the size of the neighborhood is set to $D=2d+1$, where $d=1$ represents the maximum displacement. The formulation comes to:

$$\begin{array}{c}\hfill C(s_1,s_2)=\sum _{d\in D}〈C_g(s_1+d),C_s(s_2+d)〉.\end{array}$$

(1)

Note that in Equation (1), different from general convolution, which is convolving data with a filter, we convolve data with other data. It is also worth mentioning that if this network is large enough, the similarity of two saliency map can be calculated accurately to the pixel-wise level. In this way, we obtain the similarity of two saliency maps. If the C value is large enough, it means that the synthetic saliency map generated by the G-network is more realistic. Meanwhile, it can also show that our proposed model is superior to other methods in performance.

4. Experiments

In this section, we validate our proposed MAFL method through experimental results. We first perform comparisons between the different methods on several benchmark datasets. Secondly, in order to further demonstrate the advantages of adding multi-scale GAN in the MAFL model, we perform comparisons by using single GAN. Finally, we replace the GAN network in the MAFL framework with a different network architecture, which demonstrates the advantages of using GAN in our model.

4.1. Dataset

In our proposed method, three public datasets are used to perform experiments.

THUR15K [36]: This dataset contains 6232 images and consists of five object categories. This dataset is unique because some of images contained in it do not have salient target.

MSRA-A [33]: This dataset contains 1000 images and their corresponding manual annotations [28].

PASCAL-S [37]: The dataset is derived from the PASCAL VOC2010 segmentation challenge dataset, which contains 850 images with multiple targets and a complex background.

4.2. Evaluation Criteria

To reflect the fairness of the comparison between the methods, we used three common evaluation criteria.

Precision-recall curve: Because the ROC [38] does not provide sufficient evaluation information, we use the P-R [12] curve for method comparison.

The process of obtaining the P-R curve is as follows: for a saliency map, we first binarize it using a threshold to obtain the corresponding binary mark (B); then compare the binary mask (B) with the corresponding ground-truth (G); finally, calculate the average precision and recall value in the whole dataset. The formula is as follows:

$$Precision=\frac{|G\bigcap B|}{\left|B\right|},Recall=\frac{|G\bigcap B|}{\left|G\right|}.$$

(2)

F-measure: When evaluating some methods, sometimes using the P-R curve does not really reflect the performance of the method. Therefore, we use the F-measure to comprehensively evaluate the methods. The formula for calculating the F-measure is as follows:

$$F_{\beta }=\frac{(1+{\beta }^2)\times Precision\times Recall}{{\beta }^2\times Precision+Recall}.$$

(3)

According to empirical evidence, we set ${\beta }^2$ to 0.3.

MAE: We use MAE (mean absolute error) to reduce the average error of TPR (true positive rate) and TNR (true negative rate), which can fit the relations between them. $S(x,y)$ and $G(x,y)$ denote the saliency map and ground-truth at the pixel $(x,y)$, respectively. Therefore, the formula for calculating MAE is as follows:

$$MAE=\frac{1}{W\times H}\sum _{x=1}^W\sum _{y=1}^H|S(x,y)-G(x,y)|.$$

(4)

4.3. Comparisons with Different Methods

In order to demonstrate the effectiveness of the proposed method, we compare our method with different methods on three benchmark datasets. We trained our proposed method on the THUR15K dataset, which contains 6232 images. To reflect the rationality of the training, we choose 5000 images for training and 1232 images for testing. We compare the proposed method with ten state-of-the-art approaches, including UCF [39], DCL [18], DS [19], KSR [40], SRM [41], NLDF [42], RST [43], ELD [17], SMD [44] and WSS [45]. In order to highlight significantly the advantages of our approach, we have made qualitative and quantitative comparisons of these methods.

Quantitative comparison: Figure 4 shows the visualization of the saliency maps obtained by our method and ten different methods. We can see from Figure 4 that our method can not only uniformly highlight the whole salient objects, but also preserves the details of the objects’ structure. Experiments on several datasets show that our method can produce excellent results in a variety of complex and challenging images. For example, there is a lower contrast between the target and background (Row 5); the image contains a cluttered background (Row 3); the image contains multiple salient objects (Rows 1, 2 and 5); and the objects are close to the border of the image (Row 4).

Qualitative comparison: We use the P-R curves to compare our method with the other ten methods. As can be seen from Figure 5, compared to the other ten approaches, our approach obtains the highest P-R curve on three benchmark datasets. The best performance is on the THUR15K dataset: our method achieves a precision value of 94.46%, which is better than the current best method (the precision value of the RST method is 92.54%) of 1.2%. Meanwhile, our method also achieves a 98.59% recall value, which significantly outperforms other methods.

At the same time, we also compare the proposed approach with other approaches using F-measure values on the datasets. Figure 6 shows the comparison of the different methods. It can be seen from Figure 6 that our approach obtains the highest F-measure value, which is significantly higher than other methods. Especially on the MSRA-A dataset, our method’s F-measure value increases from 0.862 to 0.8948 compared to the current best method (RST). On the THUR15K and PASCAL-S datasets, the F-measure of our proposed approach has also been greatly enhanced.

In order to further evaluate the generated synthetic saliency map, we use MAE as the evaluation metric, and Figure 7 shows the MAE values on the three datasets. We can see from Figure 7 that our method has achieved the smallest MAE value on all datasets. Compared with the current best performance approach (RST), our approach outperforms it. Especially on the PASCAL-S dataset, our method obtained the highest MAE value and successfully reduced the MAE value by 3.03%. Therefore, we can prove the effectiveness of the proposed method by this evaluation metric.

4.4. Performance Comparisons of Our MAFL Model with Multi-Scale and Single-Scale

To better show the superiority of the MAFL method with multi-scale feature extraction, comparisons are performed for our MAFL model with multi-scale GAN (MGAN) and single-scale GAN (SGAN). In our paper, MAFL with MGAN is our proposed model with four levels of output features, while the MAFL model with SGAN with only one level. Except for the different network architectures, all other settings of the two models are kept untouched. In this way, we can compare the experimental results of scale differences. We perform the experiment on the MAFL model with the MGAN configuration and the MAFL model with the SGAN configuration, respectively. Figure 8 shows the experimental results. We can see from Figure 8 that the result by using designed MAFL model with MGAN configuration is significantly better than MAFL with SGAN configuration.

4.5. Comparison of Experimental Results with Other Neural Networks

To further demonstrate the effectiveness of the proposed model with GAN, we replaced GAN in our MAFL architecture with other deep neural networks: GoogleNet, AlexNet and CNN. To further show the fairness of comparison, we compared MAFL model with different network structures on one dataset (MSRA-A). Except for the difference of the network architectures, other settings were kept the same. Figure 9 and

Table 1. The F-measure values for the three deep neural networks on the MSRA-A dataset.

Different Types of Neural Networks	F-Measure
$Ours$	0.832
$CNN$	0.685
$GoogleNet$	0.728
$AlexNet$	0.736

show that the MAFL model with multi-scale GAN performed significantly better than MAFL model with GoogleNet and AlexNet. In summary, by comparing the deep neural network models, we can see that the MAFL method had the advantage of GAN.

4.6. Comparison of Execution Time

In this subsection, in order to compare the advantages and disadvantages of each method more intuitively and clearly, we introduce another evaluation metric, that is the comparison of the execution time. By studying the performance of other methods, we found that many methods often achieve superior performance at the expense of time. However, our proposed method not only achieved excellent performance, it also had the shortest execution time. The execution times of different methods are shown in

Table 2. Comparison of the execution time (s) of each image on the MSRA-A dataset.

Method	UCF	DCL	DS	KSR	SRM	NLDF	RST	ELD	SMD	WSS	Ours
Times (s)	0.528	0.223	0.356	0.254	0.351	0.306	0.258	0.183	0.278	0.285	0.135

. From this table, we can see that our method achieves the shortest execution time.

5. Conclusions

In our paper, we propose a novel multi-scale adversarial feature learning (MAFL) approach for image saliency detection. The proposed saliency detection model consists of two CNN networks, which are the multi-scale generator network and the discriminator network. Firstly, we define a multi-scale G-network, which takes original images as inputs and generates corresponding synthetic saliency maps. Secondly, we design a novel layer in the D-network, namely a correlation layer, to compare the similarity between the synthetic saliency map and ground-truth saliency map. We further demonstrate that adversarial feature learning can significantly improve the performance. Experimental results on several public benchmark datasets demonstrate that the proposed method outperforms other existing methods.