A Semi-Supervised Reduced-Space Method for Hyperspectral Imaging Segmentation

Abstract

The development of the hyperspectral remote sensor technology allows the acquisition of images with a very detailed spectral information for each pixel. Because of this, hyperspectral images (HSI) potentially possess larger capabilities in solving many scientific and practical problems in agriculture, biomedical, ecological, geological, hydrological studies. However, their analysis requires developing specialized and fast algorithms for data processing, due the high dimensionality of the data. In this work, we propose a new semi-supervised method for multilabel segmentation of HSI that combines a suitable linear discriminant analysis, a similarity index to compare different spectra, and a random walk based model with a direct label assignment. The user-marked regions are used for the projection of the original high-dimensional feature space to a lower dimensional space, such that the class separation is maximized. This allows to retain in an automatic way the most informative features, lightening the successive computational burden. The part of the random walk is related to a combinatorial Dirichlet problem involving a weighted graph, where the nodes are the projected pixel of the original HSI, and the positive weights depend on the distances between these nodes. We then assign to each pixel of the original image a probability quantifying the likelihood that the pixel (node) belongs to some subregion. The computation of the spectral distance involves both the coordinates in a features space of a pixel and of its neighbors. The final segmentation process is therefore reduced to a suitable optimization problem coupling the probabilities from the random walker computation, and the similarity with respect the initially labeled pixels. We discuss the properties of the new method with experimental results carried on benchmark images.

1. Introduction

Hyperspectral imaging systems have gained a great amount of attention from researchers in the past few years. The sensors of these systems allow the simultaneous acquisition of hundreds of spectral wavelengths for each image pixel. This detailed spectral information increases the possibility of more accurately discriminating objects, materials, or regions of interest. Furthermore, the fine spatial resolution of the sensors enables the analysis of small spatial structures in the image. The main property of the Hyperspectral images is the strong resolving power for fine spectra, then they have a wide range of applications in agriculture [1,2], food industry [3], geosciences [4,5], biomedical applications [6,7], document image processing [8], environment [9,10], and others. However, the analysis of HSI requires developing specialized methods and algorithms for data processing [11,12,13,14]. The main methods for the processing of the hyperspectral remote sensing images include image correction [11], noise reduction [15], dimensionality reduction [12], and classification [16,17,18,19].

In principle, the spectral information from the available hundreds of narrow bands collected by hyperspectral sensors can help discriminate among spectrally similar object pixels. Then, the accurate discrimination of different regions in the image is possible and the hyperspectral image classification is one of the most active part of the research in the hyperspectral field [18,20,21,22]. However, the HSI technology still faces a series of challenges, mainly including the following problems that need to be solved. The high dimensionality of the hyperspectral data, for example, the spectral reflectance values of hyperspectral images collected by airborne or space-borne imaging spectrometers, is up to hundreds of dimensions. Moreover, factors such as sensors, atmospheric conditions, surrounding environment, and composition and distribution of ground features affect the spatial variability of spectral information. The interference of noise (e.g., Poisson noise [23,24]) and background factors also seriously degrades the quality of the collected data and the corresponding classification accuracy of HSI. Finally, in practical applications, it is extremely difficult to obtain labeled samples to be used in the classification work of a hyperspectral image. The intrinsic properties of hyperspectral images need to be addressed specifically because conventional classification algorithms made for multispectral images do not adapt well to the analysis of hyperspectral images [25].

Several approaches have been proposed for classification of HSI. A subclass of classifiers is based on probabilistic approaches by using statistical tools to find the best class for a given pixel providing a probability of the pixel being a member of each of the possible classes. For instance, the multinomial logistic regression (MLR) classifier [26] supplies a degree of plausibility for such classes. In the sparse version of MLR a Laplacian prior to enforce sparsity is adopted which leads, with some computational limitations, to good generalization capabilities in HSI classification. More recently, an improved version of this classifier has also been proposed [27] using a subspace based method. The idea of applying subspace projection approach relies on the assumption that the samples within each class can approximately lie in a lower-dimensional subspace.

Due to their successful application in several problems of pattern recognition neural networks have also attracted many researchers in the field of the classification of hyperspectral images [28,29,30,31]. The main advantage of these approaches comes from the fact that neural networks do not need prior knowledge about the statistical distribution of the classes. They need the availability of feasible training techniques for nonlinearly separable data and their use has been affected by their training complexity as well as by the number of parameters that need to be tuned. Several neural network-based classification methods have been proposed in the literature that consider both supervised and unsupervised nonparametric approaches [32,33]. Recently, the extreme learning machine (ELM) algorithm has been successfully applied as nonlinear classifiers for hyperspectral data [34,35], and have shown remarkable efficiency in terms of accuracy and computational complexity. Some deep models have also been proposed for hyperspectral data feature extraction and classification [16]. The architecture design is the crucial part of a successful deep learning model together with the availability of an appropriate broad training set.

Another example of a supervised classification approach is support vector machines (SVMs). They have been widely used for the classification of hyperspectral data due to of their ability to handle high-dimensional data with a limited number of training samples [36]. To generalize the SVM for nonlinear classification problems, the so called kernel trick was introduced [37]. However, the sensitivity to the choice of the regularization parameters and the kernel parameters is the most important disadvantage of a kernel SVM. Other methods for HSI classification involve decision trees [38,39], random forests [40], and sparse representation classifiers with dictionary based generative models [41,42].

In this work we present a novel local/global method for semiautomatic multilabel classification of HSI. Semisupervised methods rely on limited information on the objects to recognize inside the data. Graph Clustering CNNs [43] consists of a two-stage clustering strategy in order to reduce the burden of graph convolution computation. The authors in [44] apply a coupled spatial-spectral approach for approximating the convolution on graphs. Autoencoder models are widely employed for HSI classification tasks (Ref. [45] and references therein). As a first step, we consider a preprocessing process using novel feature selection approaches in order to address the curse of dimensionality and reduce the redundancy of high-dimensional data. We adopt a linear discriminant analysis based on the labeled regions in order to project the high dimensional feature space to a lower dimensional subspace. Then we development a similarity index/distance between pixel reduced features of pixel using reduced features of pixel and involving pixels in a neighborhood. We point out that the so-called pixel based or spectral classifier only treats the HSI data as a list of spectral measurement without considering spatial relations of adjacent pixels, thus discarding important information. In a real image, neighboring pixels are related or correlated, both because imaging sensors acquire significant amount of energy from adjacent pixels and because homogeneous structures which are generally large compared to the size of a pixel frequently occurred in the image scene. This spatial contextual information should help for an accurate scene interpretation. Therefore, in order to improve classification results, spectral-spatial classification methods must be developed, which assign each image pixel to one class based on its own spectral values (the spectral information) and information extracted from its neighborhood (the spatial information). Then, the new similarity index includes the contextual spatial information provided in the HSI data considering features of adjacent pixels together. The same strategy was recently consider in [46] for the segmentation of color images.

Using the new similarity index we represent the image with an undirected graph where the set of vertices is the set of pixels present in the image, and the set of the edges consist of the pairs of neighboring pixels in the image. The weight of an edge can be represented by a function based on the difference between the features of each pixel. The vertices set can be partitioned into two sets: the “labeled vertices”, and the rest of the image pixels. In order to label the last vertices we develop a random walker approach which improves the quality of the image segmentation and involves only the minimization of a quadratic function. We remark that our method considers in a different nonlinear way two terms and it is not a simple thresholding step. In fact, the distances between features affect at the same time the similarity between labeled and unlabeled pixels and the construction of the graph for the random walk. We point out that we could consider the random walk method as a Laplacian-based manifold method with a solid theoretical background [47].

The remaining sections of the paper are organized as follows. Section 2 introduces the new method and the improved random walker segmentation algorithm. Moreover, we will also discuss a new definition of non-local distance between the features of the pixels. Section 3 is devoted to the numerical experiments. In this section the proposed method is tested on some benchmark images, in order to evaluate its performance. Following the findings of the case study, the conclusions are presented in the last Section 4 with a discussion about the properties and the possible developments of the approach.

2. A Spatial-Spectral Classifier Method for Hyperspectral Images

The detailed spectral information collected by the available hyperspectral sensors improves the capability of discriminating between different objects/sub-regions in an image by providing a division into classes with increased accuracy. This accurate capacity for discrimination makes hyperspectral data a valuable source of information to be fed to advanced classifiers. The output of the classification step is generally known as the classification map.

However, several particular problems should be considered in the classification task of hyperspectral data, among which are the following: the spatial variability of the spectral characteristics, the high number of spectral channels, the high cost of true sample labeling. In this paper we introduce a novel classification method in the semi-automatic approach framework. The general goal is to segment an image into two or more separate regions, each corresponding to a particular object (or the background), based on some user input. The proposed method reduces the interaction to the minimum, asking the user to just choose the object of interest by selecting a subregion inside it. The proposed approach does not attempt to estimate a specific model for a texture region, or an object, instead it tests for the homogeneity of a given feature pattern in a region: the features are the bands for each pixel in the image. In order to identify this homogeneity, the following assumptions about the image are made:

(a)

the image contains a set of approximately homogeneous regions;

(b)

spectral variables are also highly correlated that their dimension can be reduced without losing important information;

(c)

the features between two neighbouring regions are distinguishable.

The first assumption requires images that present several regions with similar features, and we require a noise level that allows to distinguish the different regions. See [48,49,50] for a discussion on noise affecting data. Based on the second assumption we include a dimension reduction as a first processing step. For classification, in the lower dimensional space, we expect that data are well separated. Then, in the low dimensional space nearby points or points on the same structure (cluster or manifold) are likely to have the same label. In our case, nearby points are those pixels with similar features. Assumption (a) refers to a local property of the image: in an homogeneous region there is an high probability that a random walker remains in such region, whilst it is unlikely that it can reach a far but similar region. On the other hand, Assumption (b) refers to a global property of the image: similar regions may share similar spectral signature even if they are distant.

The classification results could be improved by using the contextual spatial information provided in the HS data in addition to the spectral information. In order to capture the spatial variations of the pixel features we consider closest fixed neighborhoods and a distance which involves all the pixels in these neighborhoods. Finally, the classification map is obtained by combining a random walk based model [51,52,53] with a direct label assignment: both are computed using this distance. The resulting energy changes the energy related to the random walker approach and improves the quality of the image classification. Moreover, the algorithm involves only the minimization of a quadratic function. We point out that the graph-based methods have been paid attention because of their solid mathematical background [47], relationship with sparseness properties, kernel methods, model visualization, and reliable results in many areas.

2.1. Regularized Linear Discriminant Analysis

When dealing with hyperspectral image analysis, one usually has a large number of spectral features m and a relative small number of training marked pixels $N_T$, divided in $n_M$ marked regions. A simple linear discriminant analysis (LDA) would then result in an ill-posed problem. In fact, from a theoretical point of view, one should deal with an infinite dimension Hilbert space to be linearly and compactly projected onto a finite dimensional one (see [54]). From a computational point of view, different strategies have been tested to overcome the ill-posedness of the problem. In [55] it is shown that an efficient version of the regularized LDA (RLDA) proposed in [56] performs better in segmenting hyperspectral images than support vector machine (SVM) classifiers and other LDA-based classifiers, such as penalized LDA, orthogonal LDA, and uncorrelated LDA.

The aim of RLDA is to find a projection matrix $\widehat{G}\in {\mathbb{R}}^{m\times l}$ to reduce the high-dimensional feature $x\in {\mathbb{R}}^m$ to a lower dimensional vector $a={\widehat{G}}^{\top }x\in {\mathbb{R}}^l$, $l\ll m$. More precisely (see [55], Equation (10)),

$$\widehat{G}=\underset{G\in {\mathbb{R}}^{m\times l}}{argmax}\left\{\mathrm{trace}\left({\left(G^{\top }(S+\lambda I)G\right)}^{-1}{G}^{\top }{S}_{b}G\right)\right\},$$

(1)

where S is the total scatter matrix, $S_b$ is the between-class variance matrix, I is the identity matrix, and $\lambda$ is a regolarization parameter. Note that RLDA reduces to a classical LDA when $\lambda =0$.

The efficient Algorithm 1 for RLDA (see [55]) needs two singular values decompositions (SVD).

Algorithm 1 Efficient RLDA given in [55]

Require: $\lambda \ge 0$ Ensure: $\widehat{G}$ as in (1)  $[U,D,V]\leftarrow SVD\left(H\right)$                ▹$H=UD{V}^{\top }$  $r\leftarrow \mathrm{rank}\left(H\right)=\mathrm{rank}\left(D\right)$  $D_s\leftarrow D^2+\lambda I_r$  $[U_b,D_b,V_b]\leftarrow SVD\left(D_s^{-1/2}{U}^{\top }{H}_b\right)$    ▹$D_s^{-1/2}{U}^{\top }{H}_b=U_b{D}_b{V}_b^{\top }$  $\widehat{G}\leftarrow U{D}_s^{-1/2}{U}_b$

The first one computes the SVD of the normalized data matrix

$$H=\frac{1}{\sqrt{n_L}}[X_T-\mu 1^{\top }]\in {\mathbb{R}}^{m\times N_T},$$

where $X_T\in {\mathbb{R}}^{m\times N_T}$ is the matrix data of the marked pixels and $\mu \in {\mathbb{R}}^m$ is the feature mean vector of such data. The second SVD involves the normalized between-class data matrix

$$H_b=\frac{1}{\sqrt{N_T}}[\sqrt{n_1}({\mu }_1-\mu ),\sqrt{n_2}({\mu }_2-\mu ),\dots ,\sqrt{n_{n_M}}({\mu }_{n_M}-\mu )]\in {\mathbb{R}}^{m\times n_M},$$

where $n_j$ (resp. ${\mu }_j$) denotes the sample size (res. the feature mean vector) for each marked region indexed by $j\in 1,\dots ,n_M$. The challenging calculation of a $ϵ$-approximation of such SVDs may be faced with the new efficient randomized algorithms given in [57] (to this aim, Matlab introduced the function svdsketch since R2020b, that we decided to use).

Finally, the matrix $\widehat{G}$, output of the Algorithm 1, is the matrix that projects the $n_M$ marked regions in a subspace that optimizes the Fisher score in a robust way [55]. We expect that this low-dimensional subspace will carry the relevant information to segment the entire figure. Hence, the original image $X_0\in {\mathbb{R}}^{m\times n_P}$ is mapped to $Y={\widehat{G}}^{\top }{X}_0\in {\mathbb{R}}^{l\times n_P}$.

2.2. A Spectral/Spatial Similarity Measure

After reducing the data, it is necessary to define an appropriate metric to establish the similarity between pixels in the mapped image Y. Each pixel in $y_j$ in Y has $l\ll m$ features (projected bands) that represent a vector in the linear space ${\mathbb{R}}^l$. Instead, to focus on a pixel-wise similarity we fix a system of neighbourhoods with $N_b$ pixels, for each pixel, for example 8–neighborhoods, and a feature space, subset of ${\mathbb{R}}^l$. Finally we collect all the entries of the feature vectors in a single vector $f_j\in {\mathbb{R}}^{l·N_b}$. We fix a distance $d_{l·N_b}$ in the space ${\mathbb{R}}^{l·N_b}$. Then, for a couple of pixels $y_j$ and $y_i$, we define the similarity index $\mathcal{S}(y_j,y_i)={\left(d_{l·N_b}(f_j,f_i)+\epsilon \right)}^{-1}$, $\epsilon \ll 1$, that will be used in our experiments as a weight function, see (8) below. Moreover, the notion of similarity may be extended to a comparison between a pixel $y_j$ and a label $k\in \{1,\dots ,n_M\}$. In fact, the region marked by label k is formed by pixels that have their features, from which we may extract a “centroid feature” ${\widehat{f}}^{\left(k\right)}$, that best represents the labeled region. Then, for any pixel $y_i$ and any region k, the quantity

$$\mathcal{S}(i,k)={\left(d_{l·N_b}(f_i,{\widehat{f}}^{\left(k\right)})+\epsilon \right)}^{-1}$$

(2)

will represent the similarity between the pixel i and the region k. This index can be interpreted as the first step of a k-means algorithm, whose starting centroids are computed with the user-marked regions as seeds.

This similarity, or the distance, could be used in a clustering algorithm because it represents a measure and/or comparison between pixels. Moreover, this distance allows to see the feature of the pixel in relation to its neighbors. This non-local method is inspired by recent works in signal analysis [58,59,60].

Moreover, using information from the patches instead of single pixels induces a smoothing effect [61] which may be useful in presence of noise. Indeed, consider a pixel $y_j$: denote with $f_j^{*}\in {\mathbb{R}}^{l·N_b}$ its corresponding feature vector and with $f_j\sim f_j^{*}+\mathcal{N}(0,{\sigma }^2{\mathbf{I}}_{\mathbf{d}})$ its noisy version when Gaussian noise with zeros mean and covariance matrix ${\sigma }^2{\mathit{I}}_{\mathbf{d}}$ is considered, being ${\mathit{I}}_{\mathbf{d}}$ the identity matrix. Consider another pixel $y_i$ such that the intersection of the neighbourhoods of $y_j$ and $y_i$ is empty. We can give an estimation of the expected value $\mathrm{E}\left[d_{l·N_b}{(f_j,f_i)}^2\right]$ of the square euclidean distance as

$$\mathrm{E}\left[d_{l·N_b}{(f_j,f_i)}^2\right]={\parallel f_j^{*}-f_i^{*}\parallel }^2+2·(l·N_b){\sigma }^2=d_{l·N_b}{(f_j^{*},f_i^{*})}^2+(2l·N_b){\sigma }^2$$

A similar estimation can be given when the noise affecting the pixels is not addictive but signal-dependent: for the case of Poisson noise $f_j\sim \mathrm{Poiss}\left(f_j^{*}\right)$ one obtains

$$\mathrm{E}\left[d_{l·N_b}{(f_j,f_i)}^2\right]=\parallel f_j^{*}-f_i^{*}{\parallel }^2+|f_j^{*}+f_i^{*}{|}_1=d_{l·N_b}{(f_j^{*},f_i^{*})}^2+{\parallel f_j^{*}+f_i^{*}\parallel }_1$$

where ${\parallel ·\parallel }_1$ is the ${\ell }_1$ norm in ${\mathbb{R}}^{l·N_b}$ and $d_{l·N_b}(f_j^{*},f_i^{*})$ is the euclidean distance between the clean pixels. The above estimations are based on the fact that for a random variable X one has $\mathrm{E}\left[X^2\right]=\mathrm{E}{\left[X\right]}^2+{\sigma }^2\left(X\right)$, with ${\sigma }^2\left(X\right)$ the variance of X.

Figure 1 depicts the while procedure for features compression and similarity computation. The first row is devoted to explain with a small example how the reduction is pursued. Consider just three pixels and their spectral distribution, i.e., the red, blue and green curves. The RLDA projection compresses each distribution in just a 2D vector: having at disposal these reduced vectors, one can compute the relative distance among these.

The second row depicts the procedure for a larger image: the grid on the left represents the pixels of the image, while the curves correspond to the spectral distributions for (some of) these pixels. Suppose that the dimension of the image is $m\times n\times \ell$, where $m\times n$ is the dimension of the grid while ℓ refers to the number of recorded hyperspectral frequencies. The RDLA algorithm then compresses the distribution of each pixel in just two principal components ($PC1$ and $PC2$), providing a smaller data with size $m\times n\times 2$. Consider now two pixels, namely $P_1$ and $P_2$, and their 8–neighborhoods in each component: in the previous notation, this hence amounts to have $l=2$ and $N_b=8$. Then, the vectors $f_1$ and $f_2$ that collect all the feature values of $P_1$ and $P_2$ and relative neighborhoods, respectively, belong to ${\mathbb{R}}^{16}$: we can compute the similarity index as

$$\mathcal{S}(P_1,P_2)=\frac{1}{d_{16}(f_1,f_2)+\epsilon },\epsilon \ll 1$$

with $d_{16}$ being a distance in ${\mathbb{R}}^{16}$ (for example, the classical Euclidean distance).

2.3. The Random Walker Method

The random walker (RW) method (see [51]) is a segmentation method based on a “path distance”. It belongs to the classes of probabilistic methods, that allow to address several scientific tasks, such as optimization problems via Consensus-Based Optimization [62], Opinion Formation [63] or Particle Swarm Optimization [64].

More precisely, the RW method works as follows: in each image there are subregions of pixels which are marked with labels. Then, starting from a unmarked pixel, a random walk is performed along the whole image. Obviously, the subregions closer to the starting points are more likely to be first reached than the others. The random walk is then biased in order to promote those paths that involve similar pixels: a longer path with similar pixels may have a shorter “path distance” than a longer one that crosses non homogeneous regions.

More formally, the framework here involves an undirected graph $G=(V,E)$, where $V=\left\{v_i\right|v_i\mathrm{is}\mathrm{a}\mathrm{pixel}\mathrm{in}\mathrm{the}\mathrm{image}\}$ is the set of vertexes and E is the set of edges. Some pixels have been marked by the user, and will be denoted by $V_m$, the set of “marked vertices” in the graph. $V_m$ consists of $n_M$ marked regions. Define the set of labels for the marked vertices as a function $Q\left(v_j\right)=k$, $\forall v_j\in V_m$. The other “unmarked vertices” of the image are denoted by $V_u$, so that $V=V_m\bigcup V_u$. The set of edges E will contain all the pairs of pixels which are neighbors in the original image, and $e_{ij}$ will denote the edge linking the vertexes $v_i$ and $v_j$.

The weight of an edge $e_{ij}$ encodes node similarity, and is represented by a function $\omega (v_i,v_j)$. If $d(v_i,v_j)$ represents some distance between two pixels, then common choices of $\omega$ are

$$\omega (v_i,v_j)=e^{-\beta d{(v_i,v_j)}^2}\mathrm{or}\omega (v_i,v_j)=\frac{1}{\epsilon +\sigma d(v_i,v_j)}$$

where the value of the parameters $\beta ,\sigma ,\epsilon >0$ can be tuned accordingly. Starting form an unmarked vertex, a random walk is performed. At each step, when the walker is in the vertex $v_i$, the probability of reaching the neighbor $v_j$ depends on $\omega (v_i,v_j)$ proportionally: the highest values of the distance d will imply lower probabilities of reaching that neighbor. The algorithm then computes the probability of reaching any one of the labeled vertices belonging to one of the $n_M$ marked regions. Formally, for any vertex $v_i\in V_u$ and $k\in 1,\dots ,n_M$, we denote by $x_i^k$ this probability. For a labeled node $v_j\in V_m$ that is associated to the label $Q\left(v_j\right)\in \{1,\dots ,n_M\}$, we have $x_j^k={\delta }_{i,Q\left(v_i\right)}$. Image segmentation is then made on these probabilities, and the algorithm will tend to observe the weights of the image edges during the segmentation.

In [51], the computation of the probabilities $\{x_i^k,i\in V_u,k=1,\dots ,n_M\}$ are calculated by solving a sparse linear system of equations, that involves the graph Laplacian matrix L associated to $\omega$, that is defined as

$$L_{ij}=\left\{\begin{array}{cc}{\displaystyle \sum _{v_k\mathrm{adjacent}\mathrm{to}{v}_i}\omega (v_i,v_k),}\hfill & \mathrm{if}i=j\hfill \\ -\omega (v_i,v_j),\hfill & \mathrm{if}{v}_i\mathrm{and}{v}_j\mathrm{are}\mathrm{adjacent}\mathrm{nodes}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

In particular, for each label k, the probabilities ${\mathbf{x}}^k={\left(x_1^k,x_2^k,\cdots ,x_{\left|V\right|}^k\right)}^{\top }$ are found by solving the minimization of

$$\mathcal{E}\left({\mathbf{x}}^k\right)=\frac{1}{2}\sum _{(v_i,v_j)\in E}\omega (v_i,v_j){(x_i^k-x_j^k)}^2=\frac{1}{2}{{\mathbf{x}}^k}^T\mathbf{L}{\mathbf{x}}^k.$$

(3)

Note that, since $\mathbf{L}$ is positive semi-definite, the only critical points of E will be minima. In addition, since the corresponding continuous problem leads to the minimization of the Dirichlet integral via harmonic functions then the minimization problem (3) is also called combinatorial harmonic function (see [65]). The following problem

$${\mathbf{x}}_D^k=\underset{\mathbf{x}}{argmin}\mathcal{E}\left({\mathbf{x}}^k\right),k=1,\dots ,n_M$$

(4)

is also called combinatorial Dirichlet problem.

We recall that $V=V_m\bigcup V_u$ and $V_m\bigcap V_u=\varnothing$. Without loss of generality, it is assumed that the nodes in $\mathbf{L}$ and $\mathbf{x}$ are ordered so that marked nodes comes first and unmarked nodes after. Therefore, for each $k\in \{1,\dots ,n_M\}$, we may decompose the above formula (3) into

$$\mathcal{E}({\mathbf{x}}_m^k,{\mathbf{x}}_u^k)=\frac{1}{2}\left[{{\mathbf{x}}_m^k}^{\top },{{\mathbf{x}}_u^k}^{\top }\right]\left[\begin{array}{cc}{\mathbf{L}}_m& \mathbf{B}\\ {\mathbf{B}}^{\top }& {\mathbf{L}}_u\end{array}\right]\left[\begin{array}{c}{\mathbf{x}}_m^k\\ {\mathbf{x}}_u^k\end{array}\right],$$

(5)

where ${\mathbf{x}}_m^k$ and ${\mathbf{x}}_u^k$ correspond to the probabilities of the labeled and unlabeled nodes, respectively, while $\mathbf{B}$ represents the anti-diagonal blocks of the Laplacian matrix. The problem could be interpreted as an interpolation of missing data (the unmarked points), while we have defined some (numerical) values for a subset of the vertices (our labeled nodes).

Omitting the superscript k for the ease of notation, the Equation (5) reads

$$\mathcal{E}({\mathbf{x}}_m,{\mathbf{x}}_u)=\frac{1}{2}{\mathbf{x}}_m^{\top }{\mathbf{L}}_m{\mathbf{x}}_m+{\mathbf{x}}_u^{\top }{\mathbf{B}}^{\top }{\mathbf{x}}_m+\frac{1}{2}{\mathbf{x}}_u^T{\mathbf{L}}_u{\mathbf{x}}_u$$

and the unknowns are the entries of the vector ${\mathbf{x}}_u$. Differentiating E with respect to ${\mathbf{x}}_u$ and finding the critical point yields

$${\mathbf{L}}_u{\mathbf{x}}_u=-{\mathbf{B}}^{\top }{\mathbf{x}}_m$$

(6)

which is a system of linear equations with $|V_u|$ unknowns. If every connected component contains at least a labeled vertex, then the Equation (6) is non-singular. Define ${\mathbf{X}}_u=({\mathbf{x}}_u^1,{\mathbf{x}}_u^2,\dots ,{\mathbf{x}}_u^{n_M})$ and ${\mathbf{X}}_m=({\mathbf{x}}_m^1,{\mathbf{x}}_m^2,\dots ,{\mathbf{x}}_m^{n_M})$, then the solution to the combinatorial Dirichlet problems (4) may be found by solving

$${\mathbf{L}}_u{\mathbf{X}}_u=-{\mathbf{B}}^{\top }{\mathbf{X}}_m.$$

(7)

where ${\mathbf{X}}_m$ is the labeling matrix with values in $\{0,1\}$ such that ${\mathbf{X}}_m\mathbf{1}=\mathbf{1}$ and ${\mathbf{L}}_u$ is invertible. Thus, each unlabeled pixel $v_i$ gets $n_M$ probabilities $(x_i^1,x_i^2,\dots ,x_i^{n_M})$. Eventually, the label assigned to $v_i\in V_u$ corresponds to the index of maximum-by-rows in the solution of (7). For example, suppose that an image contains only $n_M=4$ marked regions. Consider just one pixel $v_i\in V_u$: the solution of (7) for this pixel reads as $(x_i^1,x_i^2,x_i^3,x_i^4)=(0.15,0.5,0.25,0.1)$: this means that a random walker starting from $v_i$ has a probability of reaching the first region equal to $0.15$, it has a probability of reaching the region labeled with $k=2$ of $0.5$ and to reach the the region marked as $k=3$ or $k=4$ with probability of $0.25$ or $0.1$, respectively. This pixel is then labeled with as belonging to the second region, since a random walker is more likely attracted from that region. This approach is adopted also in [53].

Finally, in the next section we will test a combination of RW probabilities and similarity index, where the weights in Equation (3) are chosen as

$$\omega (v_i,v_j)=\mathcal{S}(v_i,v_j)=\frac{1}{d_{l·N_b}(f_i,f_j)+\epsilon },\epsilon ={10}^{-3}.$$

(8)

2.4. A Local/Global Classification Method

Now, we consider the vertex labeling function, for simplicity we will consider labels represented by integers,

$$F_{\mathbf{L}}:V\to {\mathcal{S}}_{\mathbf{L}}=\{1,2,\dots ,n_M\},n_M\in \mathbb{N},n_M>1$$

which associates a label in a certain set to each vertex (pixel). Combining the RW approach, with the new distance defined above, and the new similarity measure, let $F_{\mathbf{L}}$ as

$$F_{\mathbf{L}}\left(v_i\right)=\underset{k\in {\mathcal{S}}_{\mathbf{L}}}{argmax}\left(\mathcal{S}{(i,k)}^{\alpha }{\left(x_i^k\right)}^{1-\alpha }\right)$$

(9)

where $\mathcal{S}(i,k)$ is given in (2), $x_i^k$ are solutions of (7), and $1\ge \alpha \ge 0$ is a parameter introduced for adding flexibility to the algorithm and to provide different weights to the two components of the labeling function. Due to the concavity of the logarithm function, and the positivity of $\mathcal{S}(i,k)$ and $x_i^k$, we can rewrite the labeling problem in an equivalent way as follows

$$F_{\mathbf{L}}\left(v_i\right)=\underset{k\in {\mathcal{S}}_{\mathbf{L}}}{argmax}\left(\alpha log\left(\mathcal{S}(i,k)\right)+\left(1-\alpha \right)log\left(x_i^k\right)\right).$$

(10)

The proposed method therefore can be summarized in Algorithm 2.

Algorithm 2 Hyperspectral Random Walk by Similarity Index algorithm (HyperRaWaSI)

Set the parameters $\alpha$, $\lambda$, the neighbor system and the similarity function. Acquisition of user-marked pixels. Compute the projection of the original image $X_0$, $Y={\widehat{G}}^{\top }{X}_0$, where ${\widehat{G}}^{\top }$ is given by Algorithm 1. Compute the global similarity index $\mathcal{S}(i,k)$ for any $v_i\in V_u$ and $v_k\in V_m$. Solve system (7) for $V_m$, where the Laplacian matrix uses the index computed at step 3. Evaluate the labeling function as in (10) for the image $X_0$.

Remark 1.

The two terms in the functional in (10) could be considered as a “fidelity term”, the $\alpha log\left(\mathcal{S}\right)$ part, and a regularizing term, the $(1-\alpha )log\left(x\right)$ part.

Remark 2.

The approach employed in our work is not simple post-processing: the similarity index with spatial information plays a role inside the classification decision. The optimization viewpoint depicted in Equation (10) raises a different framework from a post-processing one. Furthermore, the features distances affect at the same time the similarity between labeled and unlabeled pixels and on the construction of the graph for the part of the random walk.

Figure 2 provides a visual representation of the steps for Algorithm 2 and how they are related. The very first step is to acquire the user-marked region in the original image: this labels are the building blocks of the entire procedure, since

• they are employed to get the optimal projection of the hyperspectral image and hence the feature image;
• they are used to compute the centroids from the feature image;
• they represent the seeds for the random walker method.

The final segmentation is obtained by solving (10), which involves a convex combination of the similarity indexed image, obtained by the feature image and the centroids, and the output of the random walk method.

3. Results

In this section we present the numerical experiments done in order to assess the performance of the proposed algorithm. We used four different datasets, publicly available at http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes (last visited: 17 October 2021):

• University of Pavia: this image was taken by the German Aerospace Agency (DLR) using the airborne ROSIS (Reflective Optics System Imaging Spectrometer) sensor. The spatial dimensions of the slices are $610\times 340$ and the number of spectral band is 103: then the dataset size is $103\times$ 207,400. The ground resolution is 1.3 m and the spectral gamma is 430–860 nm. The number g of ground truth labels is 9: Asphalt, Meadows, Gravel, Trees, Painted metal sheets, Bare soil, Bitumen, Self-blocking bricks, and Shadow.
• Pavia Center: this image refers to the center of the city of Pavia, but some samples in this dataset contain no information: the spatial size is $1096\times 1096$ and it is then reduced to $1096\times 715$. The considered spectral bands are 102, leading to a final size of $102\times$ 783,640. The spectral bands lie in the interval 430–860 nm and the ground resolution is 1.3 m. The $g=9$ ground truth labels are Water, Trees, Asphalt, Self-Blocking Bricks, Bitumen, Tiles, Shadows, Meadows, Bare Soil.
• KSC: this image refers to the Kennedy Space Center, Florida (US) and it was acquired by the airborne AVIRIS (Airborne Visible/Infrared Imaging Spectrometer) NASA instrument. The spatial dimensions of the slices are $512\times 614$ and the number of the spectral bands is 176: the size of the dataset is hence $176\times$ 314,368. The ground resolution is 18 m and the spectral gamma is 400–2500 nm. The g ground truth labels are 13: Scrub, Willow swamp, CP hammock, CP/Oak, Slash pine, Oak/Broadleaf, Hardwood swamp, Graminoid marsh, Spartina marsh, Cattail marsh, Salt marsh, Mud flats, Water.
• Indian Pines: this image refers to the Indian Pines test site in North-Western Indiana, taken by AVIRIS sensor. The spatial dimensions are $145\times 145$ and the employed bands are 220: The dataset size is hence $220\times$ 21,025: the spectral gamma is $0.4$–$2.5$ nm. The g ground truth labels are 16: Alfalfa, Corn-notill, Corn-mintill, Corn, Grass-pasture, Grass-trees, Grass-pasture-mowed, Hay-windrowed, Oats, Soybean-notill, Soybean mintill, Soybean-clean, Wheat, Woods, Building-grass-trees-drives, Stone-trees-drives.
• Salinas HSI: this image refers to an area including agricultural fields in the Salinas Valley, California, acquired again by AVIRIS sensor. The spatial size is $512\times 217$ and the number of bands is 204: the dataset size is hence $204\times$ 111,104 the ground resolution is 3.7 m, while the spectral gamma is $0.4$–$2.5$ nm. The $g=16$ ground truth labels are: Broccoli gr. wds 1, Broccoli gr. wds, Fallow, Fallow rough plow, Fallow smooth, Stubble, Celery, Grapes untrained, Soil vineyard develop, Corn sen. gr. wds, Lettuce romaine 4 wk, Lettuce romaine 5 wk, Lettuce romaine 6 wk, Lettuce romaine 7 wk, Vineyard untrained, Vineyard vert. trellis.

The numerical experiments are organized as follows: the datasets Indian Pines, Pavia University and Salinas HSI are employed to assess the performance of the proposed algorithm, using suitable indexes and segmentation quality measurements. The datasets Pavia Center and KSC are used as real-world datasets: the labels contained in the ground truth are employed as seed for our segmentation technique.

All the experiments were carried on a laptop equipped with Linux 19.04, with an Intel(R) Core(TM) i5–8250U CPU (1.60 GHz), 16 GiB RAM memory (Intel, Santa Clara, CA, USA) and under MatLab R2020b environment (MathWorks, Natick, MA, USA). The code is available at https://github.com/AleBenfe/RaWaCs (last access: 17 October 2021).

3.1. Performance Measurements

This section makes use of the datasets Indian Pines, Pavia University, and Salinas HSI. As previously mentioned, the proposed method belongs to the class of semi-supervised techniques: the initial seeds, i.e., the user-marked regions, are in a lower number then the original ones, that are 16, 9, and 16 for Indian Pines, Pavia University, and Salinas HSI, respectively. We denote with $G_t$ the ground truth, which contains $N_G$ labels, and with ${\left\{g_i\right\}}_{i=1,\cdots ,g}$ the marked subsets. We select manually $n_M$ labels on the dataset: this will produce a segmentation $S_t$ result with $n_M$ marked regions ${\left\{s_j\right\}}_{j=1,\cdots ,n_M}$. In order to assess the performance when the ground truth contains a number g of labels greater than $n_M$, we make use of the following indexes.

• Rand Index (RI) [52]: this index measures if the two partitions of the image, namely $G_t$ and $S_t$, are coherent. For any couple of pixels $(p_1,p_2)$ in the ground truth $G_t$, the RI measures the coherence between the partitions: it checks if $p_1$ and $p_2$ belongs to the same subset $g\in G_t$ and at the same time they belongs to the same subset $s\in S_t$. It checks also if $p_1$ and $p_2$ belongs to the two different subsets $g_1,g_2\in G_t$ and at the same time they belongs to two different subsets $s_1,s_2\in S_t$. Denote with $n_s$ the number of couples that belongs to the same subset in $G_t$ and that belong to the same subset in $S_t$, while denote with $n_d$ the number of couples that do not belong to the same subset in $G_t$ and that do not belong to the same subset in $S_t$, then

  $$RI=\frac{n_s+n_d}{{\displaystyle \left(\genfrac{}{}{0pt}{}{n_G}{2}\right)}}$$

  where $n_G$ is the total number of pixels labeled in $G_t$ and ${\displaystyle \left(\genfrac{}{}{0pt}{}{n_G}{2}\right)}$ refers to the number of all possible couples. The best performances are obtained when the RI is close to 1.
• Overall Accuracy (OA): for each true label $i=1,\dots ,n_G$, we compute

  $${\displaystyle \widehat{\mathit{ı}}=arg\underset{j=1,\cdots ,n}{max}\{\#(g_i\cap s_j)\}.}$$
  This classifies the label i as belonging to the region of $s_{\widehat{\mathit{ı}}}\subseteq S_t$, marked with $\widehat{\mathit{ı}}$. The Overall Accuracy is defined as

  $$OA=\frac{{\displaystyle \sum _{i=1}^{N_g}\#(g_i\cap s_{\widehat{\mathit{ı}}})}}{{\displaystyle \sum _{i=1}^g\#\left(g_i\right)}}=\frac{{\displaystyle \sum _{i=1}^{N_g}\underset{j=1,\cdots ,n}{max}\{\#(g_i\cap s_j)\}}}{n_G},$$

  which measures the average number of pixels of the ground truth $G_t$ that are truly classified together. The best performances are obtained when the OA is close to 1.

The first experiment refers to the Indian Pines dataset: the original image is depicted in Figure 3a, the segmentation result in Figure 3b whilst the ground truth labels are in Figure 3b. These results are obtained by setting $\lambda =0.01$ and $\alpha =0.8$ in Algorithm 2. The Rand Index for this particular experiment is 0.8505, while the OA is 0.9029: this means that each user-marked label contains the most part of a region marked in the ground truth. This result is clearly shown in Figure 3d: the element in $(i,j)$ position shows the percentage of the pixels in ground truth label j which are marked as belonging to the user-marked label i.

The Pavia University and Salinas HSI datasets present very similar results, with the same setting for the parameters ($\alpha =0.8,\lambda =0.01$): the segmentation on the former provides a RI of 0.8060 and a OA of 0.9393, and the proposed procedure reaches a reliable performance on the latter too, with RI = 0.8704 and OA = 0.9696. These experiments show that the proposed procedure is able to properly cluster several regions, according to common hyperspectral properties, even when the initial seeds are selected via a visual inspection of some of the bands presented in the dataset. Figure 4 presents the results for the Pavia University and Salinas HSI datasets.

Table 1. Computational time in seconds.

Dataset	Size	Time
Pavia University	$103\times$ 207,400	1.08
Indian Pines	$220\times$ 21,025	0.13
Salinas HSI	$204\times$ 111,104	0.45

collects the computational times of the entire procedure (excluding the marking process) for each image: even for very large datasets such as Pavia University the proposed algorithm is very fast. We employed the MatLab function svdsketch which implements the RLDA procedure depicted in Algorithm 1.

We now assess the dependence of the performance on the parameters $\alpha$ and $\lambda$: the plots in Figure 5 depict the behavior of the Rand Index with respect to $\alpha$ in correspondence of several choice for $\lambda$, namely from $\lambda =1$ to $\lambda ={10}^{-6}$. These plots show that the proposed procedure is quite stable with respect to this settings, showing an increasing of the performance around $\alpha =0.8$. Moreover, the dimensionality reduction seems to be independent on the choice of $\lambda$. As already noted in [46], the best performances are achieved when $\alpha \ne 0$ and $\alpha \ne 1$, meaning that the coupling (in this case, via a convex combination) of the RW approach and of the similarity index is the optimal strategy with respect to selecting only one of the two methods.

3.2. Comparison with State–of–the–Art Methods

This section is devoted to compare the proposed strategy with state-of-the-art methods, namely K-MBC [21], FCM [66], FDPC [67], and GWEEN [68], when the true number of clusters is known. The seeds employed for our method are randomly chosen in the ground truth mask: for each marked region, two small squares with size 7 pixels are selected. See Figure 6 for a visual explanation of this process and for example of different square size.

Table 2. Comparison between methods.

	Salinas		Pavia		KSC
Method	O.A.	Purity	O.A.	Purity	O.A.	Purity
FCM	54.6	0.63	41.9	0.47	52.5	0.56
FDPC	62.2	0.71	44.2	0.54	47.6	0.64
GWEEN	65.3	0.77	47.9	0.62	49.8	0.69
K–MBC	76.5	0.93	65.9	0.91	58.2	0.74
HyperRaWaSI	93.8	0.94	91.1	0.90	97.8	0.98

presents the comparison between our method and the other ones using the Overall Accuracy and the Purity indexes as evaluations. The Purity index measures how much pure a cluster is, i.e., its tendency to contain a sole class. The performance indexes regarding the other methods are taken from ([21], Tables 4 and 5). The setting for the proposed procedure are $\alpha =0.93,0.92$ and $\alpha =0.91$ for Salinas, Pavia, and KSC datasets, respectively, while $\lambda =0.1$ for all 3 datasets.

The proposed procedure overcomes all the methods used in the comparison, both on overall accuracy and on purity indexes.

We test the performance of the strategy with respect to the size and shape of the user-marked regions. We consider the salinas and Pavia datasets, and we employ as user-marked regions two randomly chosen squares with size 3, 5, and 7 (see Figure 6). The results are depicted in

Table 3. Performance behavior with respect to the label size.

	Salinas		Pavia
Size	O.A.	Purity	O.A.	Purity
3	89.9	0.92	80.1	0.80
5	92.7	0.94	82.7	0.81
7	93.8	0.94	91.1	0.90

.

As one expects, the larger the marking size, the higher the performances: even with a very small marking ($3\times 3$ square), the overall accuracy is above 80% and the purity is greater than 90% in the case of salinas dataset, meaning that the recognized clusters contain mostly one class.

3.3. Segmentation as Atlas

In this section the Pavia Center and KSC datasets are segmented using the ground truth labels as seeds. The idea beyond this strategy relies on the concept of atlas (see [69]): an atlas is a collection of objects whose labels are known. The strategy consists in merging the images of the atlas with the one which has to be segmented, hence the resulting image in this way contains a labeled part (from the atlas) which allows the classification of the rest of the data.

We apply this strategy to the Pavia Center and KSC datasets: we adopted this approach since the subsets labeled in the ground truth data are low in number in both cases, despite the dimension of the images.

Figure 7 refers to the Pavia center dataset: in the left panel a colored image of the landscape is depicted, while the central panel shows the labels contained in the ground truth. The strategy consists then of using these labeled regions as an atlas, providing exact information for the classification, without requiring user intervention. The final result, obtained with $\alpha =0.9$ and $\lambda =0.2$ is given in the last panel of Figure 7: a visual inspection of the achieved segmentation suggests that the recognition of the various regions is remarkable and precise. Indeed, all the red roofs, together with their projected shadow, are recognized in a remarkable way. The trees are well segmented even if few markings of them are contained in the atlas. Note that there is an abrupt interruption inside the image, as previously mentioned in the description at the beginning of this section: despite this discontinuity, the proposed procedure does not suffer from the presence of such issue in the dataset, providing a segmentation that can take into account that the image is not continuous.

We apply the same strategy (using the ground truth as an atlas) to the KSC dataset, where the main aim is to segment the different type of vegetation. Figure 8 presents the visual representation of the segmentation result, obtained by setting $\alpha =0.97$ and $\lambda ={10}^{-4}$. The right part of the image present a reliable segmentation, since the different parts of terrain and vegetation are well recognized. On the other hand, the segmentation result on the left part classifies some buildings as part of vegetation or water, since the ground truth does not present any label referring to human constructions, such as buildings, bridges or streets. Figure 9 is devoted to showing the reason why some buildings are included in the Water label: indeed, the river contains several white parts which the segmentation process correctly includes in the Water label. On the other hand, there are several buildings that present a spectral signature (both in the visible and in the infrared region) very similar to these white regions inside the river. The picture shown in Figure 9 also better exposes the reliable performance on the right part of the image.

We finally report that the computational times for the segmentation of Pavia Center and KSC datasets, which present both large dimensions: $102\times$ 783,640 the former and $176\times$ 314,368 the latter. In both cases, the computational time is close to being negligible with respect to the size (see

Table 4. Computational time in seconds.

Dataset	Size	Time
Pavia University	$102\times$ 783,640	5.24
KSC	$176\times$ 314,368	3.31

).

4. Conclusions

In this work we proposed a new spectral/spatial semi-automatic classification method coupling a Regularized Linear Discriminant analysis for dimension reduction, a suitable similarity index, and a random walker approach. The final computational problem involves a new energy function obtained by a new definition of similarity/distance between pixel using a reduced features space and pixels in a neighborhood. The experimental results showed that the proposed approach is very robust with respect to the presence of noise and with good accuracy. In our approach we have the flexibility of a system of different neighborhoods for the calculation of the distance between pixels and in the construction of the graph corresponding to a given image.

Regarding the hyper-parameters $\lambda$ and $\alpha$, we observed that in the regularization process the choice of the $\lambda$ parameter does not seem to influence the numerical results even varying it by several orders of magnitude. The RW part encourages the selection of convex zones, consequently changing the $\alpha$ parameter in Equation (10) can give more weight to these sub-regions in the final classification. In a future paper we will consider appropriate learning methods for the optimal choice of these parameters for some classes of images.

From the computational point of view, the most expensive steps concern the calculation of the new distance between pixels and the preliminary reduction step (the computation of the projection operator ${\widehat{G}}^{\top }$, see Algorithm 1). However, we observe that these operations can be performed efficiently in parallel, for example with an appropriate implementation through the use of GPUs. Moreover, we adopted a fast state-of-the-art SVD algorithm. The computation of the probabilities of RW requires the numerical solution of linear systems which may be large, but sparse and well structured at the same time, consequently, efficient algorithms can be used. Then, our method is reliable and efficient also for high-definition images.

We point out that it is possible to introduce some set of suitable atlas through the projection matrix $\widehat{G}$, see Algorithm 1. We plan to explore the combination of atlas and our classifier using an adaptive approach in order to learn better weights to be employed in the features distance. Moreover, we will consider the possibility to replace some or all the labeled pixels with atlases well adapted to the specific image. Preliminary results were shown and discussed in Section 3.3. Further comparisons will be made with other semi-automatic methods, identifying suitable quality measures of the segmentation obtained.