Multi-Scale Pure Graphs with Multi-View Subspace Clustering for Salient Object Detection

Abstract

Salient object detection is a challenging task in the field of computer vision. The graph-based model has attracted lots of research attention and achieved remarkable progress in this task, which constructs graphs to formulate the intrinsic structure of any image. Nevertheless, the existing graph-based salient object detection methods still have certain limitations and face two major challenges: (1) Previous graphs are constructed by the Gaussian kernel, but they are often corrupted by original noise. (2) They fail to capture common representations and complementary diversity of multi-view features. Both of these degrade saliency performance. In this paper, we propose a novel method, called multi-scale pure graphs with multi-view subspace clustering for salient object detection. Its main contribution is a new, two-stage graph, constructed and constrained by multi-view subspace clustering with sparsity and low rank. One of the advantages is that the multi-scale pure graphs upgrade the saliency performance from the propagation of noise in the graph matrix. Another advantage is that the multi-scale pure graphs exploit consistency and complementary information among multi-view features, which can effectively boost the capability of the graphs. In addition, to verify the impact of the symmetry of the multi-scale pure graphs on the salient object detection performance, we compared the proposed two-stage graphs, which included cases considering the multi-scale pure graphs and those not considering the multi-scale pure graphs. The experimental results were derived using several RGB benchmark datasets and several state-of-the-art algorithms for comparison. The results demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of multiple standard evaluation metrics. This paper reveals that multi-view subspace clustering is beneficial in promoting graph-based saliency detection tasks.

1. Introduction

Saliency detection aims to seek the most interesting region or important object from a natural scene by simulating the visual attention mechanism. This is a hot topic in computer vision and widely applied in many computer vision tasks, such as image segmentation [1], image retrieval [2], image fusion [3], visual tracking [4], dynamic driving scenes [5], and others.

Existing saliency detection algorithms are used for either eye fixation prediction and salient object detection. Early works [6,7,8] focused on the former topic, predicting where human gazes focused on a given image. Afterward, developed algorithms [9,10,11,12] were mainly geared toward salient object detection, aiming to extract complete object information from RGB images. Salient object detection approaches include bottom-up models and top-down models. Bottom-up models [11,13,14] are stimuli-driven, without specific task guidance, employing low-level features such as colors, textures, orientations, and spatial distances to detect salient objects. Top-down models [15,16,17] are task-driven, requiring supervised learning with manually labeled ground truth, exploiting high-level information to better abstract salient objects.

It is worth mentioning that the bottom-up models [13,14] are pioneer, graph-based methods that have shown promising performance with simplicity and efficiency. These methods mainly contain two aspects, i.e., graph construction and foreground/background seed selection. For saliency detection, graph construction is a critical issue. Traditionally, graph models depend on the Gaussian kernel function to compute affinity matrices in a single-view feature (CIElab color). Background seeds follow the boundary prior (the boundary patches are mostly background). The single-view feature cannot contain rich information about an image. To address this problem, graph models [18,19,20,21] were further constructed on multi-view low-level features for salient object detection. However, the above methods have two limitations. On the one hand, these traditional graphs are constructed from the Gaussian kernel function and may often be corrupted by noise. On the other hand, they fail to efficiently exploit both the consistency and complementary intrinsic structure of multi-view features. Both degrade saliency performance (Figure 1).

To overcome the aforementioned limitations, on the basis of the existing graph models, we propose a novel, multi-scale pure graph with multi-view subspace clustering for salient object detection. It builds upon two-stage multi-scale graphs and follows the background to compute the saliency score of each superpixel. In detail, to capture the local structures of multi-view low-level features, we first utilized the Gaussian kernel function to calculate the traditional graph matrix. Secondly, to further depict the global structure and eliminate the noise of multi-view low-level features, multi-view low-rank sparse subspace clustering [22] was applied to generate a shared affinity matrix. Afterward, the traditional graph matrix and affinity matrix were decomposed by singular value decomposition (SVD). In order to exploit the consistency and complementary intrinsic structure of multi-view low-level features, a joint affinity graph matrix was learned from multi-view subspace clustering based on a low-rank representation with diversity regularization and a rank constraint [23]. Based on these graph matrices, we constructed two-stage multi-scale graphs for graph-based manifold ranking. The main contributions are summarized as follows:

1.

To depict the global structure and remove the noise of multi-view low-level features, multi-view low-rank sparse subspace clustering was applied to induce a shared affinity graph matrix.

2.

To further capture the consistency and complementary intrinsic structure of multi-view features, multi-view subspace clustering was explored based on a low-rank representation with diversity regularization.

3.

To clearly describe the local and global structure of multi-view low-level features, two-stage multi-scale pure graphs were constructed based on the above graph matrices.

4.

Extensive experiments demonstrate that our two-stage, multi-scale pure graphs consistently achieve better saliency performance than several state-of-the-art graph models on five benchmark datasets.

The rest of this paper is organized as follows. Section 2 describes the works related to the proposed method. Section 3 introduces the main aspects of our designed framework. Section 4 presents the experiments and comparisons. Section 5 gives a discussion of the proposed method. Finally, Section 6 concludes our work.

2. Related Work

In the early stage of graph-based saliency detection, in view of these representative graph-based saliency detection methods [14,24,25,26], researchers mainly adopted the CIElab color feature to compute the affinity matrix. However, these graph models only use the CIElab color space and are unable to present the rich diversity of salient objects and backgrounds. Even worse, traditional graphs fail to capture the global structure of an image.

In particular, the work that inspired our proposed method is that featuring graph-based manifold ranking (GMR) [14], which has attracted a lot of attention in salient object detection. In this method, an RGB image is mapped onto a graph $G=(V,E)$ with N nodes $\{v_1,v_2,...,v_N\}$ and a set of edges E. Traditionally, E is denoted by an affinity graph matrix $\mathbf{W}={\left[w_{ij}\right]}_{N\times N}$ and $w_{ij}$ is calculated by

$$w_{ij}=\left\{\begin{array}{ccc}\hfill e^{-{\displaystyle \frac{\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel }{{\sigma }^2}}}& \hfill & \hfill if j\in {\mathcal{N}}_i\\ \hfill 0& \hfill & \hfill otherwise\end{array}\right.$$

(1)

where ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ represent the mean scores of superpixels $v_i$ and $v_j$ in the feature space, respectively. $\sigma$ is a constant. ${\mathcal{N}}_i$ denotes the neighbors of superpixel $v_i$. From the graph matrix $\mathbf{W}={\left[w_{ij}\right]}_{N\times N}$, the corresponding degree matrix $\mathbf{D}=diag\{d_{11},...,d_{NN}\}$ is obtained, where $d_{ii}={\sum }_i^N{w}_{ij}$. Accordingly, let $\mathbf{y}={[y_1,y_2,..,y_N]}^T$ be a indication vector, in which $y_i=1$ if $x_i$ is a query and $y_i=0$ otherwise. The optimal ranking solution of queries is derived by solving the optimization formula:

$$\begin{array}{c}\hfill {\mathbf{f}}^{*}=arg\underset{f}{min}{\displaystyle \frac{1}{2}}(\sum _{i,j=1}^N{w}_{ij}\parallel {\displaystyle \frac{f_i}{\sqrt{d_{ii}}}}-{\displaystyle \frac{f_j}{\sqrt{d_{jj}}}}{\parallel }^2+\mu \sum _{i=1}^N\parallel f_i-y_i{\parallel }^2)\end{array}$$

(2)

where the first term $({\sum }_{i,j=1}^N{w}_{ij}\parallel {\displaystyle \frac{f_i}{\sqrt{d_{ii}}}}-{\displaystyle \frac{f_j}{\sqrt{d_{jj}}}}{\parallel }^2)$ represents the smoothness constraint. The second term $({\sum }_{i=1}^N\parallel f_i-y_i{\parallel }^2)$ represents the fitting constraint. The parameter $\mu$ controls the balance between these two terms. ${\mathbf{f}}^{*}$ is the ranking results. When setting the derivative of Equation (2) to be 0, the ranking results ${\mathbf{f}}^{*}$ is equal to

$$\begin{array}{c}\hfill {\mathbf{f}}^{*}={(\mathbf{D}-\alpha \mathbf{S})}^{-1}\mathbf{y}\end{array}$$

(3)

where $\mathbf{S}$ denotes a normalized Laplacian matrix, $\mathbf{S}={\mathbf{D}}^{-1/2}\mathbf{W}{\mathbf{D}}^{-1/2}$, and $\alpha ={\displaystyle \frac{1}{1+\mu }}=0.99$.

To suit salient object detection, an unnormalized Laplacian matrix is embeded into Equation (3) and then the final ranking result is generated as

$$\begin{array}{c}\hfill {\mathbf{f}}^{*}={(\mathbf{D}-\alpha \mathbf{W})}^{-1}\mathbf{y}\end{array}$$

(4)

The key contribution of graph-based manifold ranking is two-stage “superior” graphs. For a “superior” graph matrix $\mathbf{W}\in {\mathbb{R}}^{N\times N}$, each element $w_{ij}$ is an edge weight that reflects the similarity between two adjacent nodes/superpixels ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ of an input image. Inspired by this, researchers [18,20,27] extracted the multi-view appearance feature from an image and developed the corresponding graph-based approaches. They individually computed the traditional graph matrix in each appearance feature and achieved the final graph matrix through linear fusion or vector dot product. However, these approaches still cannot capture the global structure. Furthermore, their graphs are insufficient to possess consistency and complementary intrinsic structures from the multi-view features and do not obtain significantly enhanced saliency results. Benefiting from the high-level features of deep learning [28,29] that contain rich semantic concepts of salient objects, several improved graph-based algorithms [30,31,32,33] have been proposed. These methods construct traditional graph matrices by extracting the multi-view features containing high-level semantic information and appearance features. Since the high-level semantic information can effectively describe the salient objects, these algorithms achieve better saliency performance. However, the high-level semantic information needs plenty of time to train the sample datasets. Moreover, the pooling of convolutional neural networks seriously blurs the position information of salient objects. In recent years, with the superior performance of deep learning [34,35], convolutional neural networks [36,37] and attention-based graph models [38,39,40] have been developed to perform salient object detection and have achieved outstanding saliency results. But, they still have a common shortcoming whereby they require a huge amount of labeled data along with a GPU-enabled system. In this study, different from all the aforementioned salient object detection methods, our proposed method explicitly explores subspace clustering and constructs a robust graph model of the multi-view low-level features in saliency objects. Fortunately, compared with unsupervised graph-based saliency models with multi-view low-level features, our proposed method generates promising performance and breaks through the limitation of traditional graph models.

3. Methodology

A diagram of our proposed method is shown in Figure 2, and the details will be elaborated upon below.

3.1. Multi-View Feature Extraction

To effectively describe the difference between salient objects and background, our proposed method firstly extracts 64-dimension low-level features, as given in

Table 1. The details of the multi-view features.

Types	Feature Descriptions	Dim
Color features	The average RGB values of each superpixel	3
	The average LAB values of each superpixel	3
	The average HSV values of each superpixel	3
Texture features	The Gabor features of each superpixel [42]	36
	The steerable pyramids features of each superpixel [42]	12
	The average LBP features of each superpixel	1
	The QWLD features of each superpixel [43]	1
Saliency priors	The warm color prior map [44]	1
	The SR prior map [8]	1
	The dark channel prior maps [45]	3

. Secondly, SLIC [41] is carried out to over-segment the input image into N nonoverlapping superpixels $\mathbf{P}=\{P_1,P_2,\cdots ,P_N\}$. For each superpixel $P_i$, each view feature matrix is represented as ${\mathbf{X}}^{\left(v\right)}=\{{\mathbf{x}}_1^{\left(v\right)},{\mathbf{x}}_2^{\left(v\right)},\cdots ,{\mathbf{x}}_N^{\left(v\right)}\}\in {\mathbb{R}}^{d^v\times N}$, where $v=\{1,2,..,V\}$.

3.2. Multi-View Pure Graph Construction

This section includes the traditional graph matrix computation and joint affinity matrix learning with multi-view subspace clustering.

3.2.1. Adjacent Graph

Based on the suitability of the traditional graph matrix in salient object detection, the similarity between any two adjacent nodes $P_i$ and $P_j$ in multi-view features is calculated as follows:

$$w_{ij}^{\left(l\right)}=\left\{\begin{array}{ccc}\hfill e^{-{\displaystyle \frac{\parallel x_i^{\left(l\right)}-x_j^{\left(l\right)}{\parallel }^2}{{\sigma }^2}}}& \hfill & \hfill ifj\in {\mathcal{N}}_i\\ \hfill 0& \hfill & \hfill otherwise\end{array}\right.$$

(5)

$$w_{ij}^{\left(o\right)}=\left\{\begin{array}{ccc}\hfill e^{-{\displaystyle \frac{\parallel x_i^{\left(o\right)}-x_j^{\left(o\right)}{\parallel }^2}{{\sigma }^2}}}& \hfill & \hfill ifj\in {\mathcal{N}}_i\\ \hfill 0& \hfill & \hfill otherwise\end{array}\right.$$

(6)

where $x_i^{\left(l\right)}$ and $x_j^{\left(l\right)}$ represent the mean values of superpixels $P_i$ and $P_j$ in CIElab color features, respectively. $x_i^{\left(o\right)}$ and $x_j^{\left(o\right)}$ represent the mean values of superpixels $P_i$ and $P_j$ in other multi-view features, respectively. $\sigma$ is a constant parameter that controls the degree of similarity. To integrate these two graph matrices ${\mathbf{W}}^{\left(o\right)}={\left[w_{ij}^{\left(o\right)}\right]}_{N\times N}$ and ${\mathbf{W}}^{\left(l\right)}={\left[w_{ij}^{\left(l\right)}\right]}_{N\times N}$, the traditional graph matrix is generated by

$${\mathbf{W}}^{\left(T\right)}=\sqrt{{\left({\mathbf{W}}^{\left(l\right)}\right)}^2+{\left({\mathbf{W}}^{\left(o\right)}\right)}^2}$$

(7)

3.2.2. Affinity Graph Learning

The multi-view features ${\mathbf{X}}^{\left(v\right)}=\{{\mathbf{V}}^{\left(1\right)};{\mathbf{V}}^{\left(2\right)};\cdots ;{\mathbf{X}}^{\left(V\right)}\}\in {\mathbb{R}}^{d\times n}$ should share consensus information because different features represent an input image simultaneously. Firstly, to eliminate the confusion of noise, an affinity matrix shared among multi-views is learned with low-rank and sparsity constraints, then defined as

$$\begin{array}{cc}\hfill & \underset{{\mathbf{C}}^{\left(1\right)},{\mathbf{C}}^{\left(2\right)},...,{\mathbf{C}}^{\left(V\right)}}{min}=\sum _{v=1}^V({\beta }_1\parallel {\mathbf{C}}^{\left(v\right)}{\parallel }_{*}+{\beta }_2\parallel {\mathbf{C}}^{\left(v\right)}{\parallel }_1+{\lambda }^{\left(v\right)}\parallel C^{\left(v\right)}-{\mathbf{C}}^{*}{\parallel }_F^2)\hfill \\ & s.t.{\mathbf{X}}^v={\mathbf{X}}^v{\mathbf{C}}^v,diag\left({\mathbf{C}}^v\right)=\mathbf{0}.\hfill \end{array}$$

(8)

where ${\mathbf{C}}^{\left(v\right)}$ represents the low-rank representation matrix of the v-th feature, ${\mathbf{C}}^{*}$ represents the consensus low-rank representation matrix of all views, and the parameters ${\beta }_1,{\beta }_2$, and ${\lambda }^{\left(v\right)}$ control the trade-off between low-rank and sparsity terms, respectively. In this study, we set ${\beta }_1=0.1,{\beta }_2=1-{\beta }_1,{\lambda }^{\left(v\right)}=0.3,V=8$. The optimization problem of this objective function is solved with the method [22]. Then, the affinity matrix is obtained by

$$\begin{array}{c}\hfill {\mathbf{W}}^{\left(C\right)}={\displaystyle \frac{|{\mathbf{C}}^{*}|+|{\mathbf{C}}^{*}{|}^T}{2}}\end{array}$$

(9)

To better explore the local similarity information of the traditional graph matrix ${\mathbf{W}}^{\left(T\right)}$ and global similarity information of the affinity matrix ${\mathbf{W}}^{\left(C\right)}$, we utilize the Hadamard product to combine these two matrices, as follows:

$$\begin{array}{c}\hfill {\mathbf{W}}^{\left(TC\right)}={\widetilde{{\mathbf{W}}^{\left(C\right)}}}^T\circ {\mathbf{W}}^{\left(T\right)}\circ \widetilde{{\mathbf{W}}^{\left(C\right)}}+{\widetilde{{\mathbf{W}}^{\left(T\right)}}}^T\circ {\mathbf{W}}^{\left(C\right)}\circ \widetilde{{\mathbf{W}}^{\left(T\right)}}\end{array}$$

(10)

where ∘ represents the Hadamard product of $N\times N$ matrices. The Hadamard product can reduce the computational complexity of the matrix fusion and is insensitive to the saliency results of our proposed method. Regarding the matrices ${\mathbf{W}}^{\left(C\right)}$ and ${\mathbf{W}}^{\left(T\right)}$, the angular information of the principal directions of any two low-rank vectors probably extracted from the same subspace has a higher value than that of those extracted from different subspaces. Specifically, we calculate $\widetilde{{\mathbf{W}}^{\left(C\right)}}={\mathbf{U}}^{\left(C\right)}\Sigma {\left({\mathbf{V}}^{\left(C\right)}\right)}^T$ and $\widetilde{{\mathbf{W}}^{\left(T\right)}}={\mathbf{U}}^{\left(T\right)}\Sigma {\left({\mathbf{V}}^{\left(T\right)}\right)}^T$, which represent the SVDs of ${\mathbf{W}}^{\left(C\right)}$ and ${\mathbf{W}}^{\left(T\right)}$, respectively. Thus, ${\mathbf{W}}^{\left(TC\right)}$ with global clustering can be used to refine the neighbors of the traditional graph. Further, in order to sufficiently exploit consistency and complementary information among multi-view features, the graph regularization term and rank constraint are simultaneously mixed to formulate the objective model and then a joint affinity matrix is learned and obtained as follows:

$$\begin{array}{cc}\hfill & \underset{\widehat{\mathbf{Z}},{\mathbf{E}}^{\left(v\right)},\mathbf{w}}{min}\parallel \widehat{\mathbf{Z}}{\parallel }_{*}+\sum _{v=1}^V{w}^{\left(v\right)}{\parallel {\mathbf{E}}^{\left(v\right)}\parallel }_{2,1}+\lambda {\mathbf{w}}^T\mathbf{H}\mathbf{w}+\beta Tr(\widehat{\mathbf{Z}}{\mathbf{L}}_{\mathbf{s}}{\widehat{\mathbf{Z}}}^T)\hfill \\ \hfill & s.t.{\mathbf{X}}^{\left(v\right)}={\mathbf{X}}^{\left(v\right)}\widehat{\mathbf{Z}}+{\mathbf{E}}^{\left(v\right)},{\mathbf{w}}^T{\mathbf{1}}_V=1,rank\left({\mathbf{L}}_{\mathbf{s}}\right)=N-C,\hfill \end{array}$$

(11)

where $\widehat{\mathbf{Z}}$ is the joint representation matrix of the multi-view features. ${\mathbf{E}}^{\left(v\right)}$ is the representation residual matrix of the v-th view. ${\mathbf{L}}_s^{\left(TC\right)}$ is the Laplacian matrix of the graph matrix ${\mathbf{W}}^{\left(TC\right)}$. $\lambda$ and $\beta$ are two positive balance parameters, which are set to $\lambda =1$ and $\beta =1$. C represents different classes of the multiview data, setting $C=10$. $\mathbf{H}={\left[{\mathbf{H}}_{i,j}\right]}_{V\times V}\in {\mathbb{R}}^{V\times V}$ and ${\mathbf{H}}_{i,j}=Tr\left({\mathbf{T}}_i^T{\mathbf{T}}_j\right)$ represent the similarity between the i-th and j-th views, where ${\mathbf{T}}_i={\mathbf{D}}_i^{-1}{\mathbf{S}}_i(i=1,2,\dots ,V)$ is the probability transition matrix of the random walk of the i-th view. ${\mathbf{1}}_V\in {\mathbb{R}}^V$ is a vector and its elements are set as 1. Given a vector $\mathbf{w}\in {\mathbb{R}}_{+}^V$, a diversity regularization term is obtained as follows:

$$\begin{array}{c}\hfill \underset{\mathbf{w}\in {\mathbb{R}}_{+}^V}{min}\sum _{i,j=1}^V{w}_i{w}_j{\mathbf{H}}_{i,j}={\mathbf{w}}^T\mathbf{Hw}s.t.{\mathbf{w}}^T\mathbf{1}=1\end{array}$$

(12)

We adopt the augmented Lagrange multiplier with alternating direction minimizing (ALM-ADM) to optimize the objective function Equation (11). This objective function is solved via [23]. Accordingly, the joint affinity matrix is achieved by

$$\begin{array}{c}\hfill {\mathbf{W}}^{\left(Z\right)}={\displaystyle \frac{|\widehat{\mathbf{Z}}|+|\widehat{\mathbf{Z}}{|}^T}{2}}\end{array}$$

(13)

In order to compute saliency maps with homogeneity and integrity, ${\mathbf{W}}^{\left(Z\right)}$ is further normalized as follows:

$$\begin{array}{c}\hfill {\mathbf{W}}^{{\left(Z\right)}_{\ast }}={\left({\mathbf{D}}^{\left(Z\right)}\right)}^{-1}·{\mathbf{W}}^{\left(Z\right)}\end{array}$$

(14)

where ${\mathbf{D}}^{\left(Z\right)}$ is the degree matrix of the graph matrix ${\mathbf{W}}^{{\left(Z\right)}_{\ast }}$. On this basis, ${\mathbf{W}}^{{\left(Z\right)}_{\ast }}$ is applied to strengthen the above graph matrix. Accordingly, the graph matrix of first-stage graph-based manifold ranking is

$$\begin{array}{c}\hfill {\mathbf{W}}^{\left(1\right)}={\left({\mathbf{W}}^{{\left(Z\right)}_{*}}\right)}^T\circ {\mathbf{W}}^{\left(TC\right)}\circ {\mathbf{W}}^{{\left(Z\right)}_{\ast }}+{\eta }_1{\left({\mathbf{W}}^{{\left(Z\right)}_{\ast }}\right)}^T\circ {\mathbf{W}}^{\left(T\right)}\circ {\mathbf{W}}^{{\left(Z\right)}_{\ast }}\end{array}$$

(15)

Inspired by the background prior, the coarse saliency score is computed by propagating the background seeds $\mathbf{y}$ on the graph, which is formulated as

$$\begin{array}{c}\hfill {\mathbf{f}*}^{\left(1\right)}={({\mathbf{D}}^{\left(1\right)}-\alpha {\mathbf{W}}^{\left(1\right)})}^{-1}\mathbf{y}\end{array}$$

(16)

where $\mathbf{y}\in \{{\mathbf{y}}_{\mathbf{t}},{\mathbf{y}}_{\mathbf{d}},{\mathbf{y}}_{\mathbf{l}},{\mathbf{y}}_{\mathbf{r}}\}$, in which ${\mathbf{y}}_{\mathbf{t}},{\mathbf{y}}_{\mathbf{d}},{\mathbf{y}}_{\mathbf{l}}$, and ${\mathbf{y}}_{\mathbf{r}}$ represent the top, down, left, and right boundaries of an image, respectively. ${\mathbf{f}*}^{\left(1\right)}$ is the saliency result of the first-stage graph-based manifold ranking.

After segmenting ${\mathbf{f}*}^{\left(1\right)}$ by the threshold value $\kappa ·mean({\mathbf{f}*}^{\left(1\right)})$, the foreground seed ${\mathbf{y}}_{\mathbf{f}}$ is produced. Thereby, for the faithful propagation of foreground seed ${\mathbf{y}}_{\mathbf{f}}$, another “good” graph matrix is expected to be constructed. Considering the saliency ${\mathbf{f}*}^{\left(1\right)}$, a new graph matrix is computed as

$$\begin{array}{c}\hfill {\mathbf{W}}^{\left(2\right)}={\left({\mathbf{W}}^{\left(T\right)}\right)}^T\circ {\mathbf{W}}^{\left(f\right)}\circ {\mathbf{W}}^{\left(T\right)}+{\eta }_2{\left({\mathbf{W}}^{{\left(Z\right)}_{*}}\right)}^T\circ {\mathbf{W}}^{\left(f\right)}\circ {\mathbf{W}}^{{\left(Z\right)}_{*}}\end{array}$$

(17)

where ${\mathbf{W}}^{\left(f\right)}$ is the traditional graph matrix by using the saliency ${\mathbf{f}*}^{\left(1\right)}$. Finally, the saliency map is achieved by Algorithm 1:

$$\begin{array}{c}\hfill {\mathbf{f}*}^{\left(2\right)}={({\mathbf{D}}^{\left(2\right)}-\alpha {\mathbf{W}}^{\left(2\right)})}^{-1}{\mathbf{y}}_{\mathbf{f}}\end{array}$$

(18)

The saliency maps of the proposed framework are presented in Figure 3.

Computational Complexity: In Algorithm 1, two main computational complexities solve the objective functions (Equations (8) and (11)). Following [22], the computational complexity of solving the objective functions in Equation (8) is $O\left(TV{N}^3\right)$, where T is the number of iterations, V is the number of views, and N is the number of superpixels. In addition, the computational complexity of solving the objective functions in Equation (11) is $O(d_i^{2}N+d_i{N}^2+N^3)$ by following reference [23], where $d_i$ is the dimension of each view.

Algorithm 1 Multi-scale pure graphs with multi-view subspace clustering for salient object detection

Require: Multi-view features $\mathbf{X}$, background seeds $\mathbf{y}\in \{{\mathbf{y}}_{\mathbf{t}},{\mathbf{y}}_{\mathbf{d}},{\mathbf{y}}_{\mathbf{l}},{\mathbf{y}}_{\mathbf{r}}\}$. Ensure: Saliency result ${\mathbf{f}*}^{\left(2\right)}$.   1: Initially compute traditional graph matrices ${\mathbf{W}}^{\left(l\right)}={\left[w_{ij}^{\left(l\right)}\right]}_{N\times N}$ and ${\mathbf{W}}^{\left(o\right)}={\left[w_{ij}^{\left(o\right)}\right]}_{N\times N}$ by Equation (5) and Equation (6) respectively.   2: Fuse traditional graph matrices ${\mathbf{W}}^{\left(l\right)}={\left[w_{ij}^{\left(l\right)}\right]}_{N\times N}$ and ${\mathbf{W}}^{\left(o\right)}={\left[w_{ij}^{\left(o\right)}\right]}_{N\times N}$ to generate ${\mathbf{W}}^{\left(T\right)}$ by Equation (7).   3: Learn affinity graph matrix ${\mathbf{W}}^{\left(C\right)}$ by the objective function Equations (8) and (9).   4: Combine ${\mathbf{W}}^{\left(C\right)}$ and ${\mathbf{W}}^{\left(T\right)}$ and produce ${\mathbf{W}}^{\left(TC\right)}$ by Equation (10).   5: Achieve affinity graph matrix ${\mathbf{W}}^{\left(Z\right)}$ by the objective function Equations (11) and (13).   6: Normalize affinity graph matrix ${\mathbf{W}}^{\left(Z\right)}$ and obtain ${\mathbf{W}}^{{\left(Z\right)}_{*}}$ by Equation (14).   7: Construct first-stage graph matrix ${\mathbf{W}}^{\left(1\right)}$ by Equation (15).   8: Propagate background seeds $\mathbf{y}\in \{{\mathbf{y}}_{\mathbf{t}},{\mathbf{y}}_{\mathbf{d}},{\mathbf{y}}_{\mathbf{l}},{\mathbf{y}}_{\mathbf{r}}\}$ and compute coarse saliency map ${\mathbf{f}*}^{\left(1\right)}$ by Equation (16).   9: Construct second-stage graph matrix ${\mathbf{W}}^{\left(2\right)}$ by Equation (17). 10: Obtain foreground seeds by segmenting ${\mathbf{f}*}^{\left(1\right)}$. 11: Propagate foreground seeds ${\mathbf{y}}_{\mathbf{f}}$ and calculate fine saliency result ${\mathbf{f}*}^{\left(2\right)}$ by Equation (18).

4. Experimental Design and Analysis

In this section, we evaluate the saliency performance of the proposed method on five public RGB datasets, which provide qualitative and quantitative comparisons with ten state-of-the-art methods, and carry out some effectiveness analysis of ablation experiments.

4.1. Experimental Setup

4.1.1. Implementation Details

All extensive experiments were performed in MATLAB 2016b on an Intel(R) Core(TM) i5-3700K with CPU @ 3.40 GHz and 32.0 GB of RAM. In the proposed framework, the numbers of multi-scale superpixels were set as 200, 300, and 400.

4.1.2. Benchmark Dataset

To demonstrate the effectiveness of the proposed approach, we conducted extensive experiments on five public RGB saliency detection datasets, named ECSSD [46], SOD [47], DUTOMRON [46], HUK-IS [48], and SED2 [49].

ECSSD has 1000 images of complex scenes with meaningful and rich semantic concepts.

SOD contains 300 challenging images from the Berkeley segmentation dataset, in which most images have several objects, which is challenging for bottom-up models.

DUTOMRON contains 5168 images with structural complexity and at least one object in a scene.

HUK-IS contains 4447 images with multiple salient objects.

SED2 contains 100 images and each image has two salient objects.

4.1.3. Baseline Models

To show the superior saliency performance of the proposed method, we compared the proposed method with ten state-of-the-art methods, including DRFI [13], GMR [14], RBD [50], BSCA [26], SMD [42], DP2-LSG [51], RCRR [52], WMR [18], PDP [53], AME [54], and HCA [55].

GMR is the baseline of our method.

AME is a promising graph-based salient object detection approach among existing bottom-up models in which the graph is constructed by multi-view high-level features.

HCA is an improvement of BSCA, where BSCA is named single cellular automata. It utilizes multi-view high-level features and multi-view low-level features.

SMD is a saliency detection method based on low-rank matrix recovery theory that exploits multi-view low-level features.

DRFI is a deep learning approach that exploits multi-view low-level features to highlight salient regions and achieves remarkable saliency performance.

Others are representative graph-based methods.

4.1.4. Evaluation Metrics

To achieve a faithful evaluation of all experimental results, the comparison experiments relied on eight evaluation metrics, such as the precision–recall (PR) curve [14], S-measure [56], E-measure [57], F-measure ($F_m$), mean absolute error (MAE) [12], area under curve (AUC), overlap ratio (OR) [42], and weighted F-measure [58].

PR: in the precision–recall (PR) curve, the precision was the correct proportion of the detection results and the recall was the proportion of correct detection results in the ground truth (GT). They were defined as follows:

$$Precision={\displaystyle \frac{|M\cap GT|}{\left|M\right|}},Recall={\displaystyle \frac{|M\cap GT|}{\left|GT\right|}}$$

(19)

where M denotes the binarized map of a saliency map and GT represents the corresponding ground truth (GT).

S-measure was adopted to assess the detection performance and counted region-aware and object-aware structural similarity among the saliency map and the corresponding ground-truth map, which was computed by

$$S_{\beta }=\beta ·S_o+(1-\beta )·S_r$$

(20)

where $S_o$ and $S_r$ are the region-aware and object-aware structural similarity measures, respectively.

E-measure was an enhanced-alignment measure that integrated the local pixel score with the image-level mean score and computed the statistics for image levels and matching information of local pixels. It was defined as

$$E_{\xi }={\displaystyle \frac{1}{W\times H}}\sum _{i=1}^W\sum _{j=1}^H\theta \left(\xi \right)$$

(21)

where $\theta \left(\xi \right)$ denotes the enhanced-alignment matrix.

F-measure was used to obtain an overall measure for the saliency results and was formulated as

$$F_{\beta }={\displaystyle \frac{(1+{\beta }^2)·Precision·Recall}{{\beta }^2·Precision+Recall}}$$

(22)

${\beta }^2$ was set to 0.3 to strengthen the precision. Based on this, the weighted F-measure was computed by

$$F_{\beta }^{\omega }={\displaystyle \frac{(1+{\beta }^2)·Precisio{n}^{\omega }·Recal{l}^{\omega }}{{\beta }^2·Precisio{n}^{\omega }+Recal{l}^{\omega }}}$$

(23)

MAE was an assistant measure that calculated the average difference between the saliency map and the corresponding ground truth:

$$MAE={\displaystyle \frac{1}{W\times H}}\sum _{i=0}^{W-1}\sum _{j=0}^{H-1}|S(i,j)-GT(i,j)|$$

(24)

A lower computed value meant a better performance.

AUC was defined as the area under the ROC curve; the ROC curve was obtained by computing the true positive rate (TPR) and false positive rate (FPR) under different thresholds.

4.2. Comparison with State-of-the-Art Methods

This section comprehensively evaluates and analyzes both quantitative and qualitative comparisons of the experimental results for the ECSSD, SOD, DUTOMRON, HUK-IS, and SED2 datasets.

Figure 4 shows the PR curves of our proposed method and other compared methods. It is noteworthy that our method significantly outperformed other state-of-the-art methods for DUTOMRON, ECSSD, SOD, and HUK-IS, and our method was clearly better than the other baseline methods except for AME and HCA. The most important factor was that AME and HCA utilize deeper or learned semantic features, while our proposed method only employs multi-view low-level features. On SED2, our method also achieved a better result. Moreover,

Table 2. Quantitative comparisons on ECSSD in terms of S-measure, E-measure, F-measure, MAE, AUC, WF and OR scores.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
DRFI	0.752	0.816	0.733	0.164	0.833	0.584	0.542
GMR	0.689	0.774	0.689	0.189	0.790	0.520	0.493
RBD	0.689	0.787	0.676	0.189	0.781	0.525	0.513
BSCA	0.725	0.797	0.702	0.182	0.815	0.549	0.513
SMD	0.734	0.800	0.712	0.173	0.811	0.560	0.537
2LSG	0.702	0.786	0.703	0.181	0.795	0.541	0.510
RCRR	0.694	0.781	0.693	0.184	0.793	0.529	0.498
WMR	0.698	0.779	0.684	0.191	0.798	0.527	0.497
AME	0.775	0.824	0.789	0.168	0.832	0.628	0.586
HCA	0.707	0.825	0.778	0.119	0.781	0.616	0.674
Ours	0.756	0.821	0.756	0.145	0.816	0.614	0.603

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

,

Table 3. Quantitative comparisons on SOD in terms of S-measure, E-measure, F-measure, MAE, AUC, WF, and OR scores.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
DRFI	0.625	0.714	0.626	0.226	0.752	0.437	0.438
GMR	0.589	0.676	0.577	0.259	0.714	0.384	0.405
RBD	0.589	0.700	0.596	0.229	0.706	0.406	0.428
BSCA	0.622	0.692	0.582	0.252	0.738	0.396	0.432
SMD	0.632	0.702	0.606	0.234	0.732	0.378	0.411
2LSG	0.591	0.670	0.606	0.254	0.702	0.378	0.420
RCRR	0.590	0.672	0.574	0.256	0.714	0.529	0.498
WMR	0.591	0.672	0.558	0.266	0.717	0.356	0.409
AME	0.633	0.704	0.677	0.229	0.752	0.454	0.490
HCA	0.639	0.702	0.634	0.203	0.694	0.435	0.537
Ours	0.642	0.716	0.637	0.219	0.732	0.454	0.491

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

,

Table 4. Quantitative comparisons on DUTOMRON in terms of S-measure, E-measure, F-measure, MAE, AUC, WF, and OR scores.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
DRFI	0.696	0.738	0.623	0.155	0.857	0.451	0.408
GMR	0.645	0.723	0.527	0.197	0.781	0.419	0.379
RBD	0.681	0.720	0.528	0.144	0.814	0.432	0.428
BSCA	0.652	0.706	0.567	0.191	0.808	0.409	0.392
SMD	0.680	0.728	0.572	0.166	0.809	0.440	0.424
2LSG	0.664	0.741	0.573	0.177	0.795	0.494	0.406
RCRR	0.649	0.720	0.527	0.182	0.779	0.421	0.384
AME	0.613	0.713	0.692	0.271	0.841	0.425	0.283
HCA	0.671	0.701	0.539	0.156	0.776	0.438	0.475
Ours	0.704	0.753	0.589	0.144	0.815	0.491	0.464

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

,

Table 5. Quantitative comparisons on HUK-IS in terms of S-measure, E-measure, F-measure, MAE, AUC, WF, and OR scores.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
DRFI	0.735	0.831	0.738	0.148	0.849	0.571	0.498
GMR	0.674	0.792	0.661	0.175	0.794	0.501	0.456
RBD	0.707	0.812	0.677	0.143	0.810	0.538	0.516
BSCA	0.700	0.794	0.649	0.176	0.821	0.509	0.464
SMD	0.726	0.815	0.689	0.157	0.825	0.549	0.512
2LSG	0.692	0.807	0.663	0.166	0.808	0.539	0.479
RCRR	0.679	0.794	0.664	0.171	0.797	0.507	0.459
AME	0.765	0.860	0.772	0.137	0.845	0.636	0.573
HCA	0.743	0.824	0.740	0.113	0.786	0.581	0.628
Ours	0.745	0.829	0.725	0.132	0.826	0.592	0.567

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

and

Table 6. Quantitative comparisons on SED2 in terms of S-measure, E-measure, F-measure, MAE, AUC, WF, and OR scores.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
DRFI	0.766	0.810	0.731	0.130	0.828	0.613	0.637
GMR	0.688	0.807	0.727	0.184	0.728	0.541	0.570
RBD	0.751	0.830	0.780	0.130	0.776	0.598	0.641
BSCA	0.716	0.791	0.704	0.159	0.772	0.539	0.540
SMD	0.753	0.832	0.755	0.131	0.776	0.588	0.636
2LSG	0.707	0.817	0.747	0.161	0.744	0.579	0.591
RCRR	0.692	0.798	0.727	0.160	0.733	0.542	0.576
WMR	0.707	0.798	0.704	0.153	0.741	0.539	0.577
AME	0.701	0.765	0.698	0.156	0.745	0.507	0.548
Ours	0.746	0.834	0.763	0.136	0.770	0.608	0.637

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

present the quantitative comparison results, including the S-measure, E-measure, F-measure, MAE, AUC, WF, and OR matrices. Table 2, Table 3, Table 4 and Table 5 further demonstrate the superiority of our proposed method on ECSSD, SOD, HUK-IS, and DUTOMRON. As shown in Table 6, our method achieved considerable performance compared with the baseline methods.

Visual comparisons of saliency maps of different algorithms in typical scenarios are shown in Figure 5. Intuitively, our method could accurately extract the salient objects in complex scenes. Even if salient objects were extremely similar to the surroundings (Image 4, 5, 11, 12, 15), the proposed method could still generate better saliency performance. These examples obviously illustrate the effectiveness and robustness of the proposed method. In total, visual comparisons clearly demonstrate that our proposed method performed better than the state-of-the-art methods on challenging scenes.

4.3. Comparison of Ablation Experiments

In this subsection, we describe the related ablation experiments, including the graph model analysis and each stage of graph contribution.

In the first stage of graph-based manifold ranking, the key matter is the newly constructed graph ${\mathbf{W}}^{\left(1\right)}$. Based on this, we tested the proposed scheme with traditional multi-view graph ${\mathbf{W}}^T$ and new multi-view graph ${\mathbf{W}}^{\left(1\right)}$. From Figure 6a and

Table 7. Quantitative comparisons on ECSSD in terms of ablation experiments.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
GMR	0.689	0.774	0.689	0.189	0.790	0.520	0.493
Ours by ${\mathbf{W}}^T$	0.697	0.771	0.671	0.192	0.803	0.524	0.497
Ours	0.756	0.821	0.756	0.145	0.816	0.614	0.603

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

, we can observe that the new multi-view graph ${\mathbf{W}}^{\left(1\right)}$ performed significantly better than the traditional multi-view graph ${\mathbf{W}}^T$ and baseline method. Meanwhile, Figure 7 displays the superiority of the proposed method using visual comparisons. As for the contributions of the two-stage graphs, Figure 6b and

Table 8. Quantitative comparisons on ECSSD in terms of ablation experiments.

Methods	S-Measure ↑	E-Measure ↑	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
Stage 1 of ours with ${\mathbf{W}}^{\left(TC\right)}$	0.744	0.814	0.748	0.162	0.828	0.608	0.559
Stage 2 of ours with ${\mathbf{W}}^{\left(TC\right)}$	0.756	0.821	0.756	0.145	0.816	0.614	0.603
Stage 1 of ours without ${\mathbf{W}}^{\left(TC\right)}$	0.729	0.807	0.739	0.164	0.821	0.593	0.549
Stage 2 of ours without ${\mathbf{W}}^{\left(TC\right)}$	0.733	0.803	0.737	0.150	0.804	0.593	0.586

* Red indicates the first-best, green indicates the second-best, and blue indicates the third-best.

present the PR curves and other quantitative results. It can be seen that the affinity graph ${\mathbf{W}}^{\left(TC\right)}$ contributed to the two-stage graph-based manifold ranking. Table 8 quantitatively demonstrates the contribution of the proposed second-stage graph. Figure 8 displays visual comparisons, further showing the progress from coarse to fine.

Finally, we analyzed the influence of superpixels with different numbers of superpixels, as shown in Figure 9 and

Table 9. Quantitative comparisons of the different numbers of superpixels on the overall performance.

Superpixels	F-Measure ↑	MAE ↓	AUC ↑	OR ↑	WF ↑
N = 200	0.677	0.148	0.782	0.579	0.591
N = 300	0.677	0.144	0.805	0.605	0.602
N = 400	0.660	0.146	0.817	0.617	0.603
N = 500	0.673	0.150	0.824	0.613	0.594

. The best performance of the proposed method was obtained when the number of superpixels was set to $N=400$.

4.4. Extended Experiment on High-Resolution Dataset

To demonstrate the potential applicability of our proposed method in broader contexts, we provide quantitative comparisons on a high-resolution dataset. Most graph-based saliency detection methods mainly focus on natural images with low resolutions, such as 400 × 400 or smaller, which limits their potential applications. Therefore, we tested our method on the high-resolution dataset (HRSOD) and compared it with graph-based models (GMR, IDCL) and the PDP method. Figure 10 shows the PR curves of the experimental methods on the HRSOD dataset, showing that the proposed method achieved the best performance improvement. The proposed method demonstrates a powerful ability to be used in practical applications compared to the three traditional graph-based saliency detection models.

4.5. Running Time

The computational time of the proposed method was analyzed on the ECSSD benchmark dataset. The proposed method was implemented on MATLAB 2016b on the 13th Gen Intel(R) Core(TM) i7-13700K with CPU @ 3.40 GHz and 32.0 GB of RAM. It took approximately 5.79 s on average to generate the saliency map for experimental images of 400 × 267 pixels.

Table 10. Average runtime comparison of methods on ECSSD1000 dataset.

Methods	GMR	BSCA	DRFI	AME	HCA	Ours
Runtime(s)	0.21	0.29	2.93	5.11	1.59	2.83

shows the average time of our method and the baseline models. We found that our method was slower than the competitive methods but still outperformed them both considering the comprehensive evaluation performances. This was because the competitive models AME and HCA required a significant amount of time to train multi-view semantic features using neural networks.

5. Discussion

In this work, we explored graphs with low-level multi-view features and computed saliency scores by following the background prior. This could produce poor saliency results, as shown in Figure 11. For the challenging scenes, only low-level multi-view features caused an inability to identify salient objects. In a future study, we plan to exploit multi-view deep-level semantic information to construct a “good” graph matrix for hyperspectral images. In addition, it is worth mentioning that eye movement data can be formulated to describe clinically relevant regions in a medical image and potentially integrated into an artificial intelligence (AI) system for automatic diagnosis in medical imaging. Therefore, we will be committed to researching graph neural networks for saliency detection and applying them to chest X-rays.

6. Conclusions

In this paper, we proposed novel, multi-scale pure graphs with multi-view subspace clustering for salient object detection. This work presented two-stage graphs constrained by multi-view subspace clustering with sparsity and low rank and embedded them into manifold ranking to compute a saliency map. This could upgrade saliency performance using the propagation on the graph model that exploited consistency and complementary information among multi-view features. Both methods effectively boosted the capability of the graph model. To validate the promising performance of the proposed method in terms of standard evaluation indexes, experiments on several RGB benchmark datasets and comparisons with several state-of-the-art baseline methods were carried out. The superior saliency performance enhancement demonstrates the good generalization capability of our proposed saliency detection framework.