Underwater Small Target Classification Using Sparse Multi-View Discriminant Analysis and the Invariant Scattering Transform

Abstract

Sonar automatic target recognition (ATR) systems suffer from complex acoustic scattering, background clutter, and waveguide effects that are ever-present in the ocean. Traditional signal processing techniques often struggle to distinguish targets when noise and complicated target geometries are introduced. Recent advancements in machine learning and wavelet theory offer promising directions for extracting informative features from sonar return data. This work introduces a feature extraction and dimensionality reduction technique using the invariant scattering transform and Sparse Multi-view Discriminant Analysis for identifying highly informative features in the PONDEX09/PONDEX10 datasets. The extracted features are used to train a support vector machine classifier that achieves an average classification accuracy of 97.3% using six unique targets.

1. Introduction

Automatic target recognition (ATR) in sonar systems has long been a focal point of naval research, driven by its critical role in naval operations. Sonar targets comprise a wide range of diverse objects, spanning small objects like fish and mines to large objects such as submarines and shipwrecks. Each target presents a unique challenge for achieving low false alarm detection and high classification accuracy. The complexity of the ocean environment, caused by a combination of surface reflections, marine life, and changing ocean dynamics, complicates the traditional signal processing techniques used for separating target features from background noise. This complexity is further compounded by unknown target geometries and unknown material compositions that introduce complex scattering processes.

Designing robust algorithms for extracting and selecting features from sonar returns is thus paramount for effective automatic target recognition. Additionally, the high cost and logistical challenges associated with data collection in ocean environments makes it difficult to build large datasets of sonar returns to train machine learning-based methods. Unlike domains such as Computer Vision, where large datasets are readily available, the sonar domain is generally data starved. This increases the difficulty of training deep neural networks for detection and classification, further emphasizing the need for designing efficient and robust ATR algorithms. The focus of this work is thus to design robust machine learning algorithms that work on limited data and can generalize well to unseen data.

2. Related Work

Building automated and robust sonar target recognition systems is crucial for the detection and classification of unexploded ordinances (UXOs). In earlier efforts, statistical signal processing techniques were used, which relied on simplified probability distribution assumptions to model the data [1,2]. With the increase in labeled data and computing power, there has been a notable shift towards data-driven models over traditional statistical approaches [3]. Supervised learning algorithms, such as support vector machines (SVMs) and neural networks (NNs), have risen to prominence due to their ability to generalize well to complex acoustic datasets when provided with sufficient data [3]. However, many data-driven classifiers suffer from the “curse of dimensionality”, which is often a problem in ATR because the dimensionality of the data often exceeds the number of samples available. To circumvent this issue, many algorithms include a feature extraction and dimensionality reduction step to construct a salient feature representation that provides separation between classes [4].

One of the less well-known dimensionality reduction techniques is Canonical Correlation Analysis (CCA), a statistical technique for finding linear relationships between highly correlated features between two sets (views) of multi-dimensional variables [5]. CCA uses these two views to find pairs of basis vectors that maximizes the correlation between these two views when projected onto the basis vectors [6]. CCA has been shown to be useful for finding correlations between two channel returns to improve mine detection in synthetic aperture sonar (SAS) images [7,8]. CCA has also been employed for extracting the dominant correlations between consecutive sonar ping returns [9,10,11]. The correlations extracted from consecutive sonar ping returns have been shown to significantly improve classification accuracy when the projections are used as training data instead of the raw returns [11]. Others have used CCA for detecting changes in aligned SAS images that were taken at different times [12].

While CCA is suitable for finding linear correlations between two sets of measurements, it is unable to find relationships between multiple views simultaneously. This could be problematic for tasks where multiple sonar returns are collected or multiple decision rules need to be considered. Williams et al. avoided a multi-view correlation computation by fusing multiple sonar acoustic color images into a single acoustic color image using a Gaussian weighting function [13]. A multi-view classification approach for an object scanned at different angles was formulated with a rule-based decision fusion strategy using a weighted sum of the individual classification decisions [14]. Others have used a multi-stage classification pruning process [15] and Bayesian Data Fusion schemes for seabed classification [16]. Deep convolutional networks have also be used to learn how to combine multiple views [17]. Recently, multi-view techniques such as Multi-View Discriminant Analysis [18] have been proposed for extending CCA to multiple views while also finding linear relationships between the views that maximizes separation between classes.

Included in many sonar target recognition systems is a feature extraction step that separates highly informative features in the sonar ping response. Localized transforms such as Scale-Invariant Feature Transform (SIFT) features and wavelets are commonly used to better capture instantaneous changes in a signal. In [19], SIFT features of simulated 3D submarine models projected onto a 2D image were used as a template library for a nearest neighbor classifier. In addition to SIFT features, another popular localized feature extraction technique in the sonar literature is decomposing a signal into a wavelet basis using the Gabor wavelet [20,21]. The 1D Gabor wavelet, given by

$$\begin{array}{c}\hfill \psi \left(t\right)={\displaystyle \frac{1}{{\left({\sigma }^2\pi \right)}^{\frac{1}{4}}}}exp\left({\displaystyle \frac{-t^2}{2{\sigma }^2}}\right)exp\left(i\eta t\right)\end{array}$$

(1)

possesses several interesting properties, such as being localized in both time (spatial) and frequency domains [22]. In this wavelet, $\sigma$ controls the width of the Gaussian envelope and $\eta$ controls the periodicity of the wavelet. Gabor wavelets have been used to form a compact dictionary that separates the components of a sonar pulse [23,24,25]. A recent paper by [26] developed a feature extraction pipeline that utilized a 2D Gabor wavelet filter to separate overlapping features in acoustic color magnitude images. Additionally, recent discoveries in deep learning have revealed that many of the deep convolutional neural networks, which attain state-of-the-art results on classification image tasks, fundamentally learn wavelet representations in their initial few layers [27]. This fact is bolstered by the work of Bruna and Mallet, which showed that state-of-the-art handwritten character recognition could be achieved using an invariant scattering transform [28]. This transform uses an alternating cascade of wavelet convolution operations and the nonlinear modulus operator on the complex-valued signal and possesses many desirable properties, such as being stable to deformations and additive noise. The wavelets used in the invariant scattering transform are not learned from data but instead are predefined Morlet wavelets. The invariant scattering transform has proven its efficacy in synthetic aperture radar (SAR) automatic target recognition, achieving an impressive average accuracy of 97.63% on benchmark datasets [29].

With the emergence of deep learning, a feature extraction step is no longer necessary, as an ample supply of data allows for robust feature representations to be learned in an end-to-end deep learning pipeline. Recently, convolutional neural networks (CNNs) have been used for object classification in synthetic aperture sonar (SAS) images using clever data augmentation strategies and adaptive weight initialization schemes [30,31]. Additionally, CNNs have been used for classifying different ship types based on ship trajectory images [32]. Other studies have focused on incorporating temporal attention into CNN-based underwater target recognition systems to extract nonlinear temporal features from ship-radiated acoustic data [33]. Furthermore, previous research has proposed transformer architectures for seabed sediment classification [34] and CNN-based classifiers with an additional attention mechanism for identifying different types of ship-radiated noise [35]. Self-organizing Feature Map neural networks have been used for seabed classification using acoustic backscattering strength data with classification accuracy in the 90% range [36]. CNNs have also been used for end-to-end object detection and have achieved impressive results using simple data augmentation methods [37,38]. When collecting SAS images, sometimes the same object can be viewed multiple items from varying perspectives. Williams, et al. proposed a technique for fusing multiple SAS views of an object into one view, followed by a neural network to classify the new view [39]. While neural networks have been proven useful for many sonar applications, it is often beneficial to add constraints to the model to limit the search space and prevent overfitting. It was shown in [40] that CNNs could achieve a classification accuracy rate of SAS images on par with a human expert by simply reducing the number of filters in each convolution block and increasing the depth of the network. Another study has shown that using small neural networks can reduce the risk of overfitting when classifying fish images in underwater environments, especially when the sample size is limited [41]. Transfer Learning has also been shown to improve classification accuracy of SAS images [42,43]. The idea of Transfer Learning is to start with a deep neural network, such as VGGnet [44] or GoogLeNet [45], that has been trained to classify generic images and retrain the network to classify domain-specific images, such as SAS images. Additionally, recent work has focused on designing wavelet convolution layers where only the parameters of the wavelet need to be learned [46]. This approach of only learning the parameters of wavelets is supported by a rigorous mathematical framework from Stéphane Mallat that shows CNNs learn the hierarchical structure of images in a way that resembles wavelets [47].

One of the final steps in many machine learning pipelines is a dimensionality reduction step that picks out features that are highly correlated with the class label. Some form of dimensionality reduction is usually required for high-dimensional data due to the “curse of dimensionality” [48]. Building a classifier that works in high dimensions is a non-trivial task because data points become equidistant from each other [48]. Luckily for many applications, the data points live on a lower-dimensional manifold, which means only a subset of the total features are relevant for classification [49]. Techniques that rely on Mutual Information for selecting features in SAS images have been shown to produce a compact subset of relevant features that improve classification performance [50]. One of the more well-known dimensionality reduction techniques is Principal Component Analysis (PCA) [51]. PCA is limited to finding only linear correlations in the data. Nonlinear extensions of PCA that utilize the “kernel-trick” of support vector machines have been proposed [52]. Kernel PCA requires the kernel to be chosen ahead of time, which often results in a brute-force search over a set of potential kernels. Recently, autoencoders have replaced Kernel PCA in popularity and have shown success in finding low-dimensional representations of sonar backscattering images [53].

When performing dimensionality reduction, it is often advantageous to use as few features as possible when projecting the data into a lower-dimensional space. Using fewer features can help improve the interpretability of a model and can improve the lower-dimensional representation by removing noisy features. The naive approach to finding a subset of features that minimizes some objective function is to search through all combinations of features for the optimal subset. This combinatorial approach quickly becomes intractable as the dimensionality of the data grows. While finding a sparse set of features that minimizes some objective may seem impossible, the past few decades have seen huge gains in efficient algorithms that promote sparsity in the solution by incorporating a penalty term. One of the more well-known sparse algorithms is the least absolute shrinkage and selection operator (LASSO), which adds an $L_1$ penalty to ordinary least squares [54]. Since LASSO tends to encourage sparsity in the solution, the resulting predictors are usually more interpretable since only a few of the values are non-zero. More recently, feature selection algorithms that use the $L_{2,1}$ norm given by

$$\begin{array}{c}\hfill {\parallel \mathbf{W}\parallel }_{2,1}=\sum _{i=1}^n\sqrt{\sum _{j=1}^m{w}_{ij}^2}\end{array}$$

(2)

where $\mathbf{W}\in {\mathbb{R}}^{n\times m}$ have been proposed [55]. The $L_{2,1}$ norm is preferable to the commonly used Frobenius norm because it is less sensitive to outliers and tends to promote sparsity along the rows of the matrix $\mathbf{W}$ in Equation (2). The $L_{2,1}$ norm has been used to improve the robustness of regression tasks [55] and Principal Component Analysis [56]. Additionally, the $L_{2,1}$ norm has been incorporated into eigenvalue problems for selecting sparse discriminative features from a dataset [57].

3. Key Contributions

This work presents a sparse multivariate, multi-view statistical technique that selects discriminating features from nonlinear wavelet coefficients in backscattering measurements, thus revealing salient target features partially hidden in background noise. The nonlinear wavelet coefficients are computed using the invariant scattering transform, which uses a cascade of wavelets and the nonlinear modulus operator to compute a representation that is stable to additive noise and deformations. The invariant scattering transform mimics convolutional neural networks in its use of convolutional filters and nonlinear functions, but does not require us to compute a non-convex optimization problem. Trying to solve non-convex problems on small samples sizes is prone to bad solutions that fall into a poor local minima. Removing the need for ill-posed optimization problems in favor of a more deterministic approach can lead to more robust algorithms and improve interpretability of the solution. Multiple sets of robust scattering coefficients are then used as input into our proposed Sparse Multi-view Discriminant Analysis method. Sparse Multi-view Discriminant Analysis is able to find discriminating features across multiple sets of variables, similar to how Canonical Correlation Analysis can extract linear correlations between two sets of variables and Linear Discriminant Analysis finds linear combinations of features that increase the separation between classes. Additionally, a sparsity penalty is added to the multi-view discriminant, such that noisy and uninformative features are removed from the solution. The sparse solution is computed using an iterative algorithm that rapidly converges to an optimal solution. The amount of sparsity can be tuned, such that high classification accuracy can be achieved with limited features. Using these robust features, a support vector machine is trained as the classifier because it solves an efficient convex optimization problem.

The aim of this work is to build upon the existing research on wavelets and deep learning, ensuring that the results are easily interpretable by practitioners and do not require extensive amounts of data to perform well. We introduce a method that extends previous approaches by incorporating wavelets and supervised learning techniques to produce interpretable classification decisions.

To summarize, the contributions in this work are as follows:

• Nonlinear feature extraction is performed on backscattering measurements using the invariant scattering transform. The resulting scattering coefficients are stable to noise and deformations, and they are easier to interpret than convolutional neural networks. Additionally, the wavelets used in the invariant scattering transform can effectively isolate instantaneous changes in the backscattering measurements that distinguish target classes.
• A multi-view discriminant method is used as a dimensionality reduction step to find linear combinations of the features across multiple sets of variables that increase the separation between classes in a shared lower-dimensional space.
• A sparsity penalty is added to the multi-view discriminant to select only the most informative features for classification. The iterative algorithm selects features using the sparsity-inducing $L_{2,1}$ norm, which is convex and does not require a brute-force combinatorial search over all subsets of features.

The remainder of this paper is organized as follows. The statistical techniques that are incorporated into the final algorithm and the PondEx dataset are described in Section 5. A description of the proposed algorithm is provided in Section 6. Section 7 discusses the experimental results. Concluding remarks are made in Section 8, along with directions for future work.

4. Notation

In the following mathematical derivations, lowercase letters denote scalars, bold lowercase letters denote vectors, and bold uppercase letters denote matrices. For a square matrix, $\mathbf{M}\in {\mathbb{R}}^{d\times d}$, we denote $tr\left(\mathbf{M}\right)$ as the trace operator. Given a labeled dataset with n samples, ${\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n$, where each sample, ${\mathbf{x}}_i\in {\mathbb{R}}^d$, has an associated class label, $y_i$, represented by a positive integer, we define c as the number of classes in the dataset and $c_i$ as the ith class label. The mean of the labeled dataset is given by $\mathit{\mu }={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\mathbf{x}}_i$. The mean of the ith class is denoted as ${\mathit{\mu }}_i={\displaystyle \frac{1}{n}}{\sum }_{j:y_j\in c_i}{\mathbf{x}}_j$, where $n_i$ is the number of samples in the ith class, $y_j\in c_i$ indicates that the jth sample belongs to the ith class, and the summation runs over all samples in the ith class.

For equations involving sparsity constraints, we use the scalar $\gamma >0$ as a user-defined parameter to adjust the sparsity of the solution. Similarly, $\beta >0$ is used to represent the regularization parameter for matrix inversion, denoted as ${(\mathbf{A}+\beta \mathbf{I})}^{-1}$, where $\mathbf{A}$ is a positive semi-definite matrix. We denote the convolution of a signal, $f\left(u\right)$, with a wavelet, $\psi \left(u\right)$, as $(f\ast \psi )\left(u\right)$. In some contexts, $\left(u\right)$ may be omitted for clarity.

For a multi-view dataset, where each labeled sample has multiple views, we use ${\mathbf{x}}_{jk}\in {\mathbb{R}}^d$ to denote the jth view for the kth sample. Assuming that each sample has m different views, the jth view for all n samples can be grouped into the set ${\mathbf{X}}_j=\{{\mathbf{x}}_{j1},{\mathbf{x}}_{j2},\cdots ,{\mathbf{x}}_{jn}\}$. Using the corresponding class labels $y_1,y_2,\cdots ,y_n$, the mean of the ith class for the jth view can be computed as ${\mathit{\mu }}_{ij}={\displaystyle \frac{1}{n_{ij}}}{\sum }_{k:y_k\in c_i}{\mathbf{x}}_{jk}$, where $n_{ij}$ is the number of samples in the ith class corresponding to the jth view. The mean across all views in the ith class is given as ${\mathit{\mu }}_i={\displaystyle \frac{1}{n_i}}{\sum }_{j=1}^m{n}_{ij}{\mathit{\mu }}_{ij}$, where $n_i={\sum }_{j=1}^m{n}_{ij}$ is the total number of samples in the ith class over all m views. The overall mean of the multi-view dataset is given as $\mathit{\mu }={\displaystyle \frac{1}{n}}{\sum }_{i=1}^c{n}_i{\mathit{\mu }}_i$, where $n={\sum }_{i=1}^c{n}_i$ is the total number of samples over all views of each class.

5. Materials and Methods

5.1. Data

To enhance the robustness of automatic target recognition systems for small targets, the acoustic scattering mechanics of simple elastic shapes have been rigorously studied [58,59,60]. Much of the research has focused on simulating the acoustic scattering of spherical shells in three different conditions: suspended in a water column, partially buried in sand, and completely buried in sand [58]. These simulation results are then compared with experimentally collected data to verify the theoretical models. For thin, air-filled spherical shells in a water column, antisymmetric $A_{0-}$ Lamb-type waves significantly influence the acoustic scattering amplitude [58]. As the frequency increases, detecting $A_{0-}$ waves becomes challenging due to radiation damping, which hinders the wave’s ability to traverse the shell’s perimeter and propagate back to the sensor [58].

In this work, we showcase the effectiveness of computational methods using well-understood experimental data that reflect the previously described scattering physics. However, a detailed analysis of the scattering physics for each individual small target is beyond the scope of this work. The experiments conducted in this paper utilized the PondEx09/PondEx10 datasets. PondEx09 and PondEx10 contain a variety of targets placed in a freshwater pond at the Naval Surface Warfare Center in Panama City. The datasets were collected using the rail system developed by the Applied Physics Laboratory at the University of Washington [61]. The rail system, shown in Figure 1, consisted of a mobile tower and a fixed sonar tower. Positioned on the rail system, the mobile tower contained multiple acoustic sources and receivers. The mobile tower traveled at a speed of 0.05 m/s, emitting a ping every 0.5 s. Transmissions from the movable tower consisted of a 6 ms LFM chirp centered at 16 kHz, with a bandwidth of 30 kHz. Two-dimensional “smile” images, like the one shown in Figure 2, were formed from the ping returns that were collected as the mobile tower moved along the rail system. Matched filtered backscattering measurements of the target were collected at aspect angles ranging from −80° to 80° in 20° increments and at distances of either 5 m or 10 m from the center of the rail system. Of the possible PondEx targets, we analyzed the following: an aluminum pipe 2 ft (0.61 m) long with a 1 ft (0.305 m) diameter, a solid aluminum cylinder 2 ft (0.61 m) long with a 1 ft (0.305 m) diameter, an aluminum replica of a 4 in. (10.16 cm) artillery shell, a steel replica of a 4 in. (10.16 cm) artillery shell, an 81 mm mortar, and rock #1. A breakdown of the sample size for each target class is provided in

Table 1. Number of samples per class in the PondEx09/PondEx10 datasets.

Target Shell	Number of Samples
Aluminum Pipe	9
Aluminum UXO Shell	9
Steel UXO Shell	9
Rock #1	27
Mortar	9
Solid Aluminum Cylinder	9

.

5.2. Canonical Correlation Analysis (CCA)

CCA is a statistical technique [62] that computes linear correlations between two sets of variables. The objective function of CCA aims to find a linear combination of the variables of each set that maximizes the correlation between the two sets. This is equivalent to minimizing the angle between the projected views. Specifically, let $\mathbf{X}\in {\mathbb{R}}^{p\times n}$ represent the first set of normalized, mean-centered variables, where each of the n columns in $\mathbf{X}$ corresponds to an observation and each of the p rows corresponds to a variable. Similarly, let $\mathbf{Y}\in {\mathbb{R}}^{q\times n}$ represent the second set of q normalized, mean-centered variables with the same number of observations. CCA seeks to find the coefficient vectors $\mathit{\alpha }\in {\mathbb{R}}^p$ and $\mathit{\beta }\in {\mathbb{R}}^q$, such that

$$\begin{array}{c}\hfill \rho ={\displaystyle \frac{{\mathit{\alpha }}^T{\mathbf{\Sigma }}_{\mathbf{XY}}\mathit{\beta }}{\sqrt{{\mathit{\alpha }}^T{\mathbf{\Sigma }}_{\mathbf{XX}}\mathit{\alpha }}\sqrt{{\mathit{\beta }}^T{\mathbf{\Sigma }}_{\mathbf{YY}}\mathit{\beta }}}}\end{array}$$

(3)

is maximized, where ${\mathbf{\Sigma }}_{\mathbf{XX}}$ and ${\mathbf{\Sigma }}_{\mathbf{YY}}$ are the covariance matrices of $\mathbf{X}$ and $\mathbf{Y}$, respectively. The matrix ${\mathbf{\Sigma }}_{\mathbf{XY}}$ is the cross covariance matrix between $\mathbf{X}$ and $\mathbf{Y}$. To guarantee that the projected variables have unit variance, the constraints

$$\begin{array}{cc}\hfill {\mathit{\alpha }}^T{\mathbf{\Sigma }}_{\mathbf{XX}}\mathit{\alpha }& =1\hfill \\ \hfill {\mathit{\beta }}^T{\mathbf{\Sigma }}_{\mathbf{YY}}\mathit{\beta }& =1\hfill \end{array}$$

(4)

are added. We can then formulate the Lagrangian as

$$\begin{array}{c}\hfill \mathcal{L}(\mathit{\alpha },\mathit{\beta },\lambda ,\mathit{\mu })={\mathit{\alpha }}^T{\mathbf{\Sigma }}_{\mathbf{XY}}\mathit{\beta }-\lambda ({\mathit{\alpha }}^T{\mathbf{\Sigma }}_{\mathbf{XX}}\mathit{\alpha }-1)-\mathit{\mu }({\mathit{\beta }}^T{\mathbf{\Sigma }}_{\mathbf{YY}}\mathit{\beta }-1)\end{array}$$

(5)

After solving for $\mathit{\alpha }$ and $\mathit{\beta }$ in Equation (5), the two views $\mathbf{X}$ and $\mathbf{Y}$ can be projected into a lower-dimensional space using the transformation $\mathbf{U}={\mathit{\alpha }}^T\mathbf{X}$ and $\mathbf{V}={\mathit{\beta }}^T\mathbf{Y}$.

5.3. Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis is a supervised statistical technique [63] that seeks to find a linear combination of features that maximally separates the means of each class while minimizing the spread of each class in the lower-dimensional space. The objective function of LDA is formulated as a Rayleigh quotient that maximizes the ratio of the between-class variance to the within-class variance.

Suppose we are given a labeled dataset of n column vectors ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n\in {\mathbb{R}}^d$, which have the corresponding class labels $y_1,y_2,\dots ,y_n$. LDA seeks to find a linear transformation matrix, $\mathbf{W}\in {\mathbb{R}}^{d\times k}$, such that the Fisher Criterion [62]

$$\begin{array}{c}\hfill J\left(\mathbf{W}\right)={\displaystyle \frac{{\mathbf{W}}^T{\mathbf{S}}_b\mathbf{W}}{{\mathbf{W}}^T{\mathbf{S}}_w\mathbf{W}}}\end{array}$$

(6)

is maximized, where ${\mathbf{S}}_b={\sum }_{i=1}^c{n}_i({\mathit{\mu }}_i-\mathit{\mu }){({\mathit{\mu }}_i-\mathit{\mu })}^T$ and ${\mathbf{S}}_w={\sum }_{i=1}^c{\sum }_{j:y_j\in c_i}{n}_i({\mathbf{x}}_j-{\mathit{\mu }}_i){({\mathbf{x}}_j-{\mathit{\mu }}_i)}^T$ are the between-class scatter matrix and the within-class scatter matrix, respectively. The expressions $\mathit{\mu }={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\mathbf{x}}_i$ and ${\mathit{\mu }}_i={\displaystyle \frac{1}{n_i}}{\sum }_{j:y_j\in c_i}{\mathbf{x}}_j$ are the overall mean of the dataset and the mean of the samples that belong to class $c_i$, respectively. Equation (6) can be rewritten as

$$\begin{array}{cc}\hfill \underset{{\mathbf{w}}_l}{max}& {\mathbf{w}}_l^T{\mathbf{S}}_b{\mathbf{w}}_l\forall l=0,1,\dots ,k\hfill \\ \hfill s.t.& {\mathbf{w}}_l^T{\mathbf{S}}_w{\mathbf{w}}_l=1\hfill \end{array}$$

(7)

We can then reformulate this constrained optimization problem as a non-constrained optimization problem using Lagrange multipliers

$$\begin{array}{c}\hfill \mathcal{L}(\mathbf{W},\mathbf{\Lambda })=tr[{\mathbf{W}}^T{\mathbf{S}}_b\mathbf{W}-\mathbf{\Lambda }({\mathbf{W}}^T{\mathbf{S}}_w\mathbf{W}-\mathbf{I})]\end{array}$$

(8)

where $tr(·)$ is the trace operator and $\mathbf{\Lambda }$ is the matrix containing the Lagrange multipliers. Taking the derivative of Equation (8) w.r.t. ${\mathbf{w}}_l$, we obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{w}}_l}}=0& \to {\mathbf{S}}_b{\mathbf{w}}_l=\lambda {\mathbf{S}}_w{\mathbf{w}}_l\forall l=0,1,\dots ,k\hfill \\ \hfill & \to {\mathbf{S}}_w^{-1}{\mathbf{S}}_b{\mathbf{w}}_l=\lambda {\mathbf{w}}_l\hfill \end{array}$$

(9)

which is a generalized eigenvalue. Thus, ${\mathbf{w}}_l$ is an eigenvector of ${\mathbf{S}}_w^{-1}{\mathbf{S}}_b$.

5.4. Sparse LDA

Although LDA is designed to select features from high-dimensional data that maximize the separation between classes, it has no mechanism for completely eliminating noisy features that do not improve classification accuracy. Decreasing the amplitudes of the coefficients in the LDA eigenvectors that correspond to these noisy features will result in better data representations, which in turn improves classification accuracy. Our approach for producing sparse LDA eigenvectors draws inspiration from the Unsupervised Discriminate Feature Selection algorithm proposed by [57].

We again assume that we are given a labeled dataset of n column vectors ${\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\in {\mathbb{R}}^d$, which have the corresponding class labels $y_1,y_2,\dots ,y_n$. Introducing sparsity into an optimization problem often involves minimizing a certain norm, such as the $L_1$ norm, of the vector or matrix under optimization. Maximizing Equation (6) while simultaneously minimizing a user-specified norm of matrix $\mathbf{W}$ is a computationally difficult problem to solve efficiently. To circumvent this issue, we instead choose to minimize

$$\begin{array}{c}\hfill J\left(\mathbf{W}\right)={\displaystyle \frac{{\mathbf{W}}^T{\mathbf{S}}_w\mathbf{W}}{{\mathbf{W}}^T{\mathbf{S}}_b\mathbf{W}}}\end{array}$$

(10)

which minimizes the spread of each class distribution in the lower-dimensional space while maximizing the separation between class means. The optimization in Equation (10) has the same effect as Equation (6). The problem with this approach is that ${\mathbf{S}}_b$ is usually low rank and therefore not invertible. This can be solved by adding a regularization term, $\beta \mathbf{I}$, to Equation (10)

$$\begin{array}{c}\hfill J\left(\mathbf{W}\right)={\displaystyle \frac{{\mathbf{W}}^T{\mathbf{S}}_w\mathbf{W}}{{\mathbf{W}}^T({\mathbf{S}}_b+\beta \mathbf{I})\mathbf{W}}}\end{array}$$

(11)

such that $\beta >0$. Next, to guarantee that we obtain an optimal solution, we would like to minimize a convex function. The Trace Ratio problem in Equation (11) is non-convex and cannot be solved efficiently [64]. To avoid this, we can multiply each of our training examples, ${\mathbf{x}}_i$, by the inverse square root of the regularized between-class scatter matrix, such that ${\tilde{\mathbf{x}}}_i={({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}{\mathbf{x}}_i$. This transformation rotates and stretches the data so that ${\mathbf{W}}^T({\mathbf{S}}_b+\beta \mathbf{I})\mathbf{W}=\mathbf{I}$. Therefore, the new optimization that we solve is

$$\begin{array}{cc}\hfill J\left(W\right)& =tr\left({\mathbf{W}}^T\mathbf{M}\mathbf{W}\right)\hfill \end{array}$$

(12)

where $\mathbf{M}={({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}{\mathbf{S}}_w{({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}$, since both ${({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}$ and ${\mathbf{S}}_w$ are positive semi-definite matrices. Now that we have an efficient way of minimizing Equation (11), we can can add a penalty to Equation (12) to promote sparsity. To accomplish this, we choose the $L_{2,1}$ norm, given by

$$\begin{array}{c}\hfill {\parallel \mathbf{W}\parallel }_{2,1}=\sum _{i=1}^n\sqrt{\sum _{j=1}^m{w}_{ij}^2}\end{array}$$

(13)

where n and m are the number of rows and columns in the matrix, respectively. As demonstrated in [55], the $L_{2,1}$ promotes sparsity along the rows of $\mathbf{W}$, which leads to robust feature selection by removing individual features from all of the projection vectors in $\mathbf{W}$. Therefore, the objective function for this sparse LDA algorithm can be written as

$$\begin{array}{cc}\hfill J\left(\mathbf{W}\right)& =tr\left[{\mathbf{W}}^T{({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}{\mathbf{S}}_w{({\mathbf{S}}_b+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}\mathbf{W}\right]+\gamma {\parallel \mathbf{W}\parallel }_{2,1}\hfill \end{array}$$

(14)

where $\gamma >0$ controls how sparse $\mathbf{W}$ should be. A proof of the convergence of Algorithm A1, which is identical to the one used in [57], is included in Appendix A for completeness.

5.5. Scattering Transform

The invariant scattering transform is a signal processing technique based on wavelet theory that extracts a robust representation of an input signal using a cascade of wavelet transforms and modulus operations [28]. This technique has been applied to various tasks, such as handwritten digit recognition [28] and speech classification [65]. The combination of wavelets and the modulus has the special property of reducing variability due to translations, rotations, and arbitrary scaling, as shown in [28]. Reducing variability among target classes is crucial for robust classification, especially when dealing with small sample sizes. Additionally, the invariant scattering transform is stable to deformations, which means that small variations or distortions in the input signal will only have minor effects on the scattering coefficients. As a result, classifiers that operate on the scattering coefficients will have an easier time generalizing to unseen samples.

We will briefly review the scattering transform for completeness. Wavelets can capture multi-resolution information by scaling and rotating the mother wavelet, $\psi \left(u\right)$:

$$\begin{array}{c}\hfill {\psi }_{\lambda }\left(u\right)=2^{-2j}\psi \left(2^{-j}{r}^{-1}u\right)\end{array}$$

(15)

where $\lambda$ parameterizes the scaling and rotation, j represents the change in scale, and r characterizes the rotation of the wavelet. The continuous wavelet transform, $W_{\lambda }$, of an input signal, $f\left(u\right)$, can be expressed as

$$\begin{array}{c}\hfill W_{\lambda }=(f\times {\psi }_{\lambda })\left(u\right)\end{array}$$

(16)

The wavelet transform is not translation invariant, but instead commutes with translation. To make the transformation invariant to translations, a nonlinearity must be introduced. To make the transformation stable to deformations and additive noise, the modulus operator is chosen as the nonlinearity. The wavelet typically used in the invariant scattering transform is the complex-valued 2D Morlet wavelet, given by

$$\begin{array}{c}\hfill \psi \left(u\right)=\alpha (e^{i\zeta (u_{x}cos\theta +u_{y}sin\theta )}-\tau )e^{-{\displaystyle \frac{(u_x^2+u_y^2)}{2{\sigma }^2}}}\end{array}$$

(17)

where $\sigma$ controls the drop-off of the Gaussian envelope, $\zeta$ controls the periodicity of the complex exponential, $\tau$ is adjusted such that $\int \psi \left(u\right)du=0$, and $\alpha$ is the normalization constant. The scattering coefficients of the signal, x, are then calculated using the $L_1$ norm

$$\begin{array}{c}\hfill \parallel f\ast {\psi }_{\lambda }{\parallel }_1=\int \left|(f\ast {\psi }_{\lambda })\left(u\right)\right|du\end{array}$$

(18)

While one may think that information is lost by discarding the phase of $f\ast {\psi }_{\lambda }\left(u\right)$ using the modulus, information is actually lost by integrating over the coefficients of $|(f\ast {\psi }_{\lambda })\left(u\right)|$. Integrating over $|(f\ast {\psi }_{\lambda })\left(u\right)|$ removes all the nonzero frequencies; however, these frequencies can be recovered by computing the wavelet coefficients of $\left|\right|f\ast {\psi }_{{\lambda }_1}|\ast {\psi }_{{\lambda }_2}|$, where ${\psi }_{{\lambda }_1}$ is the wavelet used in the first “layer” and ${\psi }_{{\lambda }_2}$ the wavelet in the second “layer” if we use the terminology commonly used in deep learning. We can continue this cascade of wavelets and modulus operators to compute a deeper representation.

To compute localized descriptors of the input signal, f, that are locally invariant to translation, a scaled spatial window function, ${\varphi }_{2^J}=2^{-2J}\varphi \left(2^{-J}u\right)$, is used to integrate the signal locally, where $2^J$ controls the scale of the window function. Let $p=({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_m)$ be the wavelet parameters used in the invariant scattering transform. The windowed scattering transform is defined as

$$\begin{array}{c}\hfill S_{p}f\left(u\right)=\left|\right|\left|\right|f\ast {\psi }_{{\lambda }_1}|\ast {\psi }_{{\lambda }_2}|\cdots |\ast {\psi }_{{\lambda }_m}|\ast {\varphi }_{2^J}\end{array}$$

(19)

Using this windowed scattering transform, we can build a scattering convolution network with m layer, as shown in Figure 3.

Each branch in Figure 3 from the j to the $j+1$ layer corresponds to the rotation, r, taken from angles in $[0,\pi )$. The scattering coefficients are sub-sampled after each layer in the network. The windowing function, $\varphi$, is also chosen to be a Gaussian.

It is important to note that, unlike conventional CNNs, the scattering convolutional network differs from the typical structure of a CNN. While conventional CNN architectures often reduce the input size through mechanisms like pooling layers and convolutional strides, this is not the case for scattering convolutional networks. Instead, scattering convolutional networks generate multiple views of the input image across various paths within the network. Each path yields scattering coefficient views that are appropriately down-sampled, ensuring that the total output dimensions from all of the scattering views remain within the same order of magnitude as the dimension of the input image.

5.6. Multi-View Discriminant Analysis (MvDA)

Multi-view Discriminant Analysis was originally proposed by Kan et. al. as a way of projecting m views into one shared lower-dimensional space by learning m projection vectors ${\mathbf{w}}_1,{\mathbf{w}}_2,\cdots ,{\mathbf{w}}_m$ [18]. MvDA can be considered as a statistical method that combines CCA and LDA and extends them to the multi-view case. MvDA finds a shared lower-dimensional space that maximizes the ratio between between-class variation and the within-class variation, similar to Equation (6).

To compute MvDA, we assume that we are given a labeled dataset $\{{\mathbf{x}}_{j1},{\mathbf{x}}_{j2},\cdots ,{\mathbf{x}}_{jn}\}\in {\mathbf{X}}_j$ that has the corresponding labels $y_1,y_2,\cdots ,y_n$, where ${\mathbf{X}}_j$ is the set of samples for the jth view and ${\mathbf{x}}_{jk}\in {\mathbb{R}}^{d_j}$ is the kth sample from the jth view. The samples from the jth view are projected into the lower-dimensional space using the linear transformation $f_j\left({\mathbf{x}}_{jk}\right)={\mathbf{w}}_j^T{\mathbf{x}}_{jk}$. In this shared lower-dimensional space, we seek to maximize the Fisher Criterion given by

$$\begin{array}{c}\hfill J\left(\tilde{\mathbf{W}}\right)={\displaystyle \frac{{\tilde{\mathbf{W}}}^T{\tilde{\mathbf{S}}}_b\tilde{\mathbf{W}}}{{\tilde{\mathbf{W}}}^T{\tilde{\mathbf{S}}}_w\tilde{\mathbf{W}}}}\end{array}$$

(20)

where $\tilde{\mathbf{W}}=\left[{\tilde{\mathbf{w}}}_1,{\tilde{\mathbf{w}}}_2,\cdots ,{\tilde{\mathbf{w}}}_p\right]$ contains the first p eigenvectors of ${\tilde{\mathbf{S}}}_w^{-1}{\tilde{\mathbf{S}}}_b$ and ${\tilde{\mathbf{w}}}_l=[{\mathbf{w}}_{l1}^T,{\mathbf{w}}_{l2}^T,$$\cdots ,{\mathbf{w}}_{lm}^T{]}^T$ is the lth eigenvector that contains the projection vectors corresponding to each view. The between-class scatter matrix, ${\tilde{\mathbf{S}}}_b$, and the within-class scatter matrix, ${\tilde{\mathbf{S}}}_w$, are given as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{S}}}_b& =\sum _{i=1}^c{n}_i({\mathit{\mu }}_i-\mathit{\mu }){({\mathit{\mu }}_i-\mathit{\mu })}^T\hfill \\ \hfill {\tilde{\mathbf{S}}}_w& =\sum _{i=1}^c\sum _{j=1}^m\sum _{k:y_k\in c_i}({\mathbf{x}}_{jk}-{\mathit{\mu }}_i){({\mathbf{x}}_{jk}-{\mathit{\mu }}_i)}^T\hfill \end{array}$$

(21)

where

$$\begin{array}{c}\hfill {\mathit{\mu }}_{ij}={\displaystyle \frac{1}{n_{ij}}}\sum _{k:y_k\in c_i}{\mathbf{x}}_{jk}\end{array}$$

(22)

is the mean over each sample of a single view in the ith class,

$$\begin{array}{cc}\hfill {\mathit{\mu }}_i& ={\displaystyle \frac{1}{n_i}}\sum _{j=1}^m\sum _{k:y_k\in c_i}{\mathbf{x}}_{jk}\hfill \\ \hfill & ={\displaystyle \frac{1}{n_i}}\sum _{j=1}^m{n}_{ij}{\mathit{\mu }}_{ij}\hfill \end{array}$$

(23)

is the mean over all views of each sample in the ith class, and

$$\begin{array}{cc}\hfill \mathit{\mu }& ={\displaystyle \frac{1}{n}}\sum _{i=1}^c\sum _{j=1}^m\sum _{k:y_k\in c_i}{\mathbf{x}}_{jk}\hfill \\ \hfill & ={\displaystyle \frac{1}{n}}\sum _{i=1}^c{n}_i{\mathit{\mu }}_i\hfill \end{array}$$

(24)

is the mean over all views of all samples. It should also be noted that $n_i={\sum }_{j=1}^m{n}_{ij}$ is the number of samples across all views that are in the ith class. Similarly, $n={\sum }_{i=1}^c{n}_i$ is the number of samples across all views of each class. The projection of the within-class scatter matrix can be reformatted in a more compact way as

$${\tilde{\mathbf{w}}}_l^T{\tilde{\mathbf{S}}}_w{\tilde{\mathbf{w}}}_l=[{\mathbf{w}}_{l1}^T,{\mathbf{w}}_{l2}^T,\cdots ,{\mathbf{w}}_{lm}^T]\left[\begin{array}{ccc}{\mathbf{B}}_{11}& \dots & {\mathbf{B}}_{1m}\\ ⋮& \ddots & ⋮\\ {\mathbf{B}}_{m1}& \dots & {\mathbf{B}}_{mm}\end{array}\right]\left[\begin{array}{c}{\mathbf{w}}_{l1}\\ {\mathbf{w}}_{l2}\\ \dots \\ {\mathbf{w}}_{lm}\end{array}\right]$$

(25)

where ${\mathbf{B}}_{jl}$ is defined as

$${\mathbf{B}}_{jl}=\left\{\begin{array}{cc}\sum _{i=1}^c\left[\sum _{k:y_k\in c_i}\left({\mathbf{x}}_{jk}{\mathbf{x}}_{jk}^T\right)-{\displaystyle \frac{n_{ij}^2}{n_i}}{\mathit{\mu }}_{ij}{\mathit{\mu }}_{ij}^T\right],\hfill & j=l\hfill \\ -\sum _{i=1}^c{\displaystyle \frac{n_{ij}{n}_{il}}{n_i}}{\mathit{\mu }}_{ij}{\mathit{\mu }}_{il}^T,\hfill & j\ne l\hfill \end{array}\right.$$

(26)

A proof of this result is provided in Appendix A. Similarly, the between-class scatter matrix can be rewritten as

$${\tilde{\mathbf{w}}}_l^T{\tilde{\mathbf{S}}}_b{\tilde{\mathbf{w}}}_l=[{\mathbf{w}}_{l1}^T,{\mathbf{w}}_{l2}^T,\cdots ,{\mathbf{w}}_{lm}^T][{\mathbf{D}}_{11}\left[\begin{array}{ccc}{\mathbf{D}}_{11}& \dots & {\mathbf{D}}_{1m}\\ ⋮& \ddots & ⋮\\ {\mathbf{D}}_{m1}& \dots & {\mathbf{D}}_{mm}\end{array}\right]\left[\begin{array}{c}{\mathbf{w}}_{l1}\\ {\mathbf{w}}_{l2}\\ \dots \\ {\mathbf{w}}_{lm}\end{array}\right]$$

(27)

where the block element ${\mathbf{D}}_{jl}$ is given by

$$\begin{array}{c}\hfill {\mathbf{D}}_{jl}=\left(\sum _{i=1}^c{\displaystyle \frac{n_{ij}{n}_{il}}{n_i}}{\mathit{\mu }}_{ij}{\mathit{\mu }}_{il}^T\right)-{\displaystyle \frac{1}{n}}\left(\sum _{i=1}^c{n}_{ij}{\mathit{\mu }}_{ij}\right)\left(\sum _{i=1}^c{n}_{il}{\mathit{\mu }}_{il}^T\right)\end{array}$$

(28)

Therefore, the Fisher Criterion in Equation (20) can be expressed as

$$\begin{array}{c}\hfill J\left(\tilde{\mathbf{W}}\right)={\displaystyle \frac{{\tilde{\mathbf{W}}}^T\mathbf{D}\tilde{\mathbf{W}}}{{\tilde{\mathbf{W}}}^T\mathbf{B}\tilde{\mathbf{W}}}}\end{array}$$

(29)

which can be solved as a generalized eigenvalue problem similar to the one solved by LDA. It is important to note that each pair of block diagonal elements ${\mathbf{B}}_{jj}$ and ${\mathbf{D}}_{jj}$ corresponds to a Fisher Criterion being computed on each of the individual views. These block diagonal elements learn intra-view discriminant information. Additionally, the off-diagonal block element pairs ${\mathbf{B}}_{jl}$ and ${\mathbf{D}}_{jl}$ compute inter-view discriminant information. This joint framework allows MvDA to learn discriminative relationships in the data across multiple views.

6. Technical Approach

The target classification pipeline employed in this paper uses a three-step approach described below to extract discriminative features and perform classification.

• Step 1: extract a robust representation of the PondEx data using an invariant scattering convolutional network described in Section 5.5.
• Step 2: project the extracted features into a lower-dimensional space using Sparse Multi-view Discriminant Analysis.
• Step 3: train an SVM classifier on the projected features.

A flow chart of the target classification pipeline is shown in Figure 4.

6.1. Step 1: Feature Extraction Using Scattering Network

We denote each individual real-valued “smile” image in the PONDEX dataset as ${\mathbf{x}}_j\in {\mathbb{R}}^{d_1\times d_2}$ with its corresponding class label $y_j$. The scattering convolutional network is applied to the entire “smile“ image, similar to how 2-D CNNs are used in image recognition tasks [44]. As shown in Figure 3, the scattering convolutional network does not reduce the dimensionality of the data after each layer like typical CNNs. Instead, a cascade of wavelet filters and nonlinear operators produce multiple representations of the input image. The image representation is sub-sampled after each wavelet layer, so the representation produced from a particular path in the scattering network is smaller than the original input image. However, the scattering network produces multiple views of the input image from the multiple paths. The resulting invariant scattering representation of the input has approximately the same order of magnitude of features as the original image. After applying the scattering network to the input image, ${\mathbf{x}}_j$, the output, ${\overline{\mathbf{x}}}_j=\mathrm{ScatNet}\left({\mathbf{x}}_j\right)\in {\mathbb{R}}^{m\times d_1^{\prime }\times d_2^{\prime }}$, has m views for the m different sets of parameters used in the network, and each view is of size $d_1^{\prime }\times d_2^{\prime }$, where $d_1^{\prime }<d_1$ and $d_2^{\prime }<d_2$. In the work, we used the Kymatio python package to compute the invariant scattering transform [66].

6.2. Step 2: Dimensionality Reduction

Prior to classification, the features extracted using the scattering network in Step 1 are projected onto a lower-dimensional subspace using MvDA. Each view in ${\overline{\mathbf{x}}}_j$ needs to be vectorized before using MvDA. We vectorize each view into a column vector using

$$\mathrm{vec}\left(\mathbf{x}\right)=\left[\begin{array}{c}{x}_{1,1}\\ ⋮\\ x_{d_1^{\prime },1}\\ x_{1,2}\\ ⋮\\ x_{d_1^{\prime },2}\\ x_{1,d_2^{\prime }}\\ ⋮\\ x_{d_1^{\prime },d_2^{\prime }}\end{array}\right]\in {\mathbb{R}}^{d_1^{\prime }{d}_2^{\prime }}$$

(30)

which stacks the columns of $\mathbf{x}$ in order. Using each training sample pair $({\tilde{\mathbf{x}}}_k,y_k)$, where ${\tilde{\mathbf{x}}}_{jk}=\mathrm{vec}\left({\overline{\mathbf{x}}}_{jk}\right)$ is the jth vectorized view of ${\tilde{\mathbf{x}}}_k$, we can construct the Multi-view Fisher Criterion in Equation (29) and solve for the eigenvectors that maximize Equation (29). After solving for the eigenvectors $\tilde{\mathbf{W}}$, the views are projected onto a subspace using the expression $f_l\left({\tilde{\mathbf{x}}}_k\right)={\sum }_{j=1}^m{\mathbf{w}}_{lj}^T{\tilde{\mathbf{x}}}_{jk}\forall l\in 1,2,\cdots ,p$. The resulting vector, $\mathbf{z}\in {\mathbb{R}}^p$, and its corresponding class label can then be used to train the classifier.

As discussed in Section 5.4, it is often beneficial to enforce some constraints on an optimization problem in order to prevent overfitting to noisy features. Since MvDA is an eigenvalue problem, we can enforce sparsity by modifying the objective function. If we instead minimize the expression

$$\begin{array}{c}\hfill J\left(\tilde{\mathbf{W}}\right)={\displaystyle \frac{{\tilde{\mathbf{W}}}^T\mathbf{B}\tilde{\mathbf{W}}}{{\tilde{\mathbf{W}}}^T\mathbf{D}\tilde{\mathbf{W}}}}\end{array}$$

(31)

We can then write the the objective function as Sparse MvDA as

$$\begin{array}{c}\hfill J\left(\tilde{\mathbf{W}}\right)=tr\left({\tilde{\mathbf{W}}}^T\mathbf{M}\tilde{\mathbf{W}}\right)+\gamma {\parallel \tilde{\mathbf{W}}\parallel }_{2,1}\end{array}$$

(32)

where $\mathbf{M}={(\mathbf{D}+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}\mathbf{B}{(\mathbf{D}+\beta \mathbf{I})}^{-{\displaystyle \frac{1}{2}}}$, $\gamma >0$ controls the sparsity of $\mathbf{W}$, and $\beta >0$ controls the amount of regularization added to $\mathbf{D}$. Similarly, after solving for the eigenvector $\tilde{\mathbf{W}}$, the views are projected onto a lower-dimensional space using the linear equation $f_l\left({\tilde{\mathbf{x}}}_k\right)={\sum }_{j=1}^m{\mathbf{w}}_{lj}^T{\tilde{\mathbf{x}}}_{jk}\forall l\in 1,2,\cdots ,p$. An SVM classifier is then trained on the robust feature vector.

6.3. Step 3: Classification

A support vector machine (SVM) was used for multi-class classification. SVMs solve a convex optimization problem and thus can be trained efficiently using custom solvers. In this work, we used the in the scikit-learn Python package [67]. SVMs are versatile classifiers because they can use the so-called “kernel trick” [68]. The kernel trick becomes beneficial when the data can be separated with a higher-order polynomial curve, but not a simple linear decision boundary. The kernel trick lifts the input data into a higher-dimensional feature space where a linear decision boundary can be drawn. We use the linear kernel to evaluate the features selected by Sparse MvDA. Additionally, if two classes overlap enough, where finding a linear decision boundary in the higher-dimensional feature space would cause overfitting, a regularization coefficient, C, can be introduced to the SVM to allow some of the data points in the training set to be misclassified. Allowing some data points in the training set to be misclassified can lead to better generalization on a holdout set. The SVM is trained on the feature vectors from PCA, MvDA, and Sparse MvDA using both the linear kernel and a small C value of 0.1. Additionally, we compare the results obtained using the invariant scattering transform with those obtained using acoustic color images. These acoustic color images are generated by applying a Blackman-windowed 8192-point Fourier Transform to each backscatter return at each cross-range position in the 2D PondEx “smile” image. The dimensionality of each image is reduced using PCA, MvDA, and Sparse MvDA individually, and each reduced image is then classified with a linear SVM with a C value of 0.1.

7. Results

7.1. Single-View Toy Example

In order to validate the design of our sparsity promoting technique, we first apply it to a toy problem where the answer is known and relatively few of the features are informative. In this example, three different class distributions are generated, which each contain 32 features, of which 4 are informative for classification. The samples for these four informative features are generated using $\mathbf{X}\sim \mathcal{N}({\mathit{\mu }}_k,\mathbf{C})$, and the parameters of the distributions are given as

$${\mathit{\mu }}_1=\left[\begin{array}{c}4\\ 4\\ 4\\ 4\end{array}\right],{\mathit{\mu }}_2=\left[\begin{array}{c}2\\ 2\\ 2\\ 0\end{array}\right],{\mathit{\mu }}_3=\left[\begin{array}{c}0\\ 0\\ 0\\ 2\end{array}\right]$$

(33)

$$\begin{array}{c}\hfill \mathbf{C}={\displaystyle \frac{3}{2}}{\mathbf{I}}_4-{\displaystyle \frac{1}{4}}\end{array}$$

(34)

where ${\mathit{\mu }}_k$ is the mean vector of each class and $\mathbf{C}$ is the covariance matrix for the informative features. The other 28 features are i.i.d taken from $\mathbf{X}\sim \mathcal{N}(0,1)$. This toy dataset serves as a good initial benchmark because it provides a controlled environment with a known ground truth, containing enough informative features to achieve robust classification results while also including a substantial number of uninformative features to effectively test the algorithm’s robustness to noisy, non-informative features.

A plot of these generated samples is shown in Figure 5. We compare both MvDA and Sparse MvDA when the number of views, m, equals 1. Having only one view means that we are effectively computing LDA and sparse LDA on the generated dataset. After minimizing the objective functions to solve for the eigenvectors, we keep the two eigenvectors corresponding to the two smallest eigenvalues. The pair of eigenvectors for MvDA and Sparse MvDA are shown in Figure 6 and  Figure 7, respectively. The eigenvector pairs in Figure 6 show that the ordinary MvDA is able to pick out the four informative features, but also fits to the noisy features that do not provide any discriminative information. We can see in Figure 6 that ordinary MvDA has overfit to the training data since the three classes are perfectly separable in the training dataset but have considerable overlap in the testing set. This is not the case for Sparse MvDA, where, in Figure 7, the coefficients in the eigenvectors that correspond to noisy features are basically zero. The projections of the data in Figure 7 show that, while the projected training set is not perfectly separable, the learned eigenvectors generalize significantly better to the test set than the eigenvectors learned from ordinary MvDA.

We evaluate Sparse MvDA against MvDA and PCA across 10 random shuffles of the dataset described above. Sparse MvDA uses the parameters $\beta ={10}^{-8}$ and $\gamma ={10}^{-1}$. We use the first two components of Sparse MvDA and MvDA to project the data, as the inherent rank of $\mathbf{D}$ in Equation (29) is determined by the three classes. For PCA, we use 14 principal components, which provide the best average classification results. After projecting the data into a lower-dimensional subspace, the data points are classified using a linear SVM with regularization parameter $C=0.1$. The results, shown in

Table 2. We evaluated the classification results across 10 random shuffles of the toy dataset shown in Figure 5. Each shuffle split the dataset into training and testing sets with a 60/40 ratio. PCA, MvDA, and Sparse MvDA were trained on the training data and used to project the data into a lower-dimensional space. Subsequently, a linear support vector machine (SVM) was employed to classify the projected data. The classification results presented below show the performance of each trained method on the test set.

Dimensionality Reduction	Accuracy
PCA	$83.67\pm 4.8\%$
MvDA	$93.0\pm 4.1\%$
Sparse MvDA	$94.0\pm 3.7\%$

, indicate that PCA performs the worst of the three methods. This is because PCA maximizes variance in the projection directions but does not necessarily enhance class separation, unlike MvDA and Sparse MvDA. MvDA and Sparse MvDA perform similarly, with Sparse MvDA slightly outperforming MvDA due to its added sparsity constraint, which makes it more resilient to overfitting from noisy features.

7.2. Classification Accuracy on PondEx Datasets

We use the mean classification accuracy to determine how well each method performed on the PondEx dataset. Each model we compare was trained on 10 random shuffles of the dataset, where the mean classification is calculated as

$$\begin{array}{c}\hfill \mathrm{accuracy}={\displaystyle \frac{1}{n}}\sum _i^n{\displaystyle \frac{{\mathbf{y}}_{\mathrm{true}}\cap {\mathbf{y}}_{\mathrm{pred}}}{{\mathbf{y}}_{\mathrm{true}}}}\end{array}$$

(35)

where ${\mathbf{y}}_{\mathrm{true}}$ is the set of ground truth labels for each sample in the test set and ${\mathbf{y}}_{\mathrm{pred}}$ is the set of predicted labels for each sample in the test set. The mean classification accuracies using PCA, MvDA, and Sparse MvDA trained on invariant scattering coefficients (an example of which is shown in Figure 8), alongside PCA, MvDA, and Sparse MvDA trained on acoustic color images, are listed in

Table 3. We evaluated the classification results across 10 random shuffles of the PondEx dataset, employing an 80/20 training/test split. The first three dimensionality reduction technique listed were trained on acoustic color images, as described in Section 6.3. For the last three techniques, the invariant scattering transform was first applied to extract features from PondEx images. Subsequently, the resulting invariant scattering coefficients were projected into a lower-dimensional space using one of the three techniques mentioned. Following dimensionality reduction, a linear support vector machine was employed to classify the projected data from all four listed methods.

Input Data	Dimensionality Reduction	Accuracy
	PCA	$87.3\pm 9.6\%$
Acoustic Color	MvDA	$91.5\pm 5.3\%$
	Sparse MvDA	$92.1\pm 1.6\%$
	PCA	$96.6\pm 4.5\%$
Scattering Coefficients	MvDA	$98.6\pm 2.6\%$
	Sparse MvDA	$97.3\pm 3.3\%$

. It is evident from Table 3 that the three dimensionality reduction techniques trained on the invariant scattering coefficients outperform the classification algorithms trained on the acoustic color images. Notably, both of the supervised methods (MvDA and Sparse MvDA) and the unsupervised method (PCA) show significant improvements in classification accuracy due to this feature extraction step. This boost in performance for the PCA is especially notable, showcasing the effectiveness of the invariant scattering transform. This advantage stems from the invariant scattering transform’s ability to differentiate target classes using a combination of localized wavelets and nonlinearities.

To investigate how altering the level of sparsity impacts the mean classification accuracy of Sparse MvDA, we experiment by varying the $\gamma$ parameter. Figure 9 and Figure 10 show classification results for $\gamma$ parameters taken from the set $\{{10}^{-8},{10}^{-7},{10}^{-6},{10}^{-5}\}$. Figure 9 shows mean classification accuracy trend lines for the four different $\gamma$ parameters using only the first 20 PCA components from each scattering coefficient view. The x-axis in Figure 9 shows the number of Sparse MvDA eigenvectors used to project the views into a lower-dimensional space. In Figure 9, a discernible trend emerges: as the number of eigenvectors increases, so does the mean classification accuracy. Moreover, it is worth noting that, when the number of eigenvectors is held constant, an increase in $\gamma$ tends to correspond with a decrease in mean classification accuracy. The cyan line, which corresponds to $\gamma ={10}^{-5}$, lags behind the other accuracy lines. This discrepancy is due to the $\gamma$ parameter being too strong and eliminating too many features in the first few eigenvectors. The cyan line does eventually converge to the same mean classification accuracy as the other three curves, but it takes a significant number of eigenvectors to do so. Additionally, the red and green curves corresponding to $\gamma ={10}^{-8}$ and $\gamma ={10}^{-7}$ reach their maximum mean classification accuracy with fewer eigenvectors and remain constant as the number of eigenvectors increases. This means that an ideal $\gamma$ that maximizes both the mean classification accuracy and the number of noisy features removed is somewhere in the range between ${10}^{-8}$ and ${10}^{-7}$.

Additionally, Figure 10 shows mean classification accuracy trend lines for the four different $\gamma$ parameters using the first 30 PCA components from each scattering coefficient view. We see that the trends for the four different $\gamma$ parameters are similar to the ones shown in Figure 9. The cyan curves lags behind the three other curves. The separation between the $\gamma ={10}^{-5}$ curve and the three other curves is more pronounced than the gap shown in Figure 9. Additionally, the cyan curve in Figure 10 never converges to the maximum mean classification accuracy achieved by the three other curves, like was seen in Figure 10. The cyan curve settles out at about 85% mean classification accuracy. Since the models trained in Figure 10 use 30 PCA components from each scattering coefficient view instead of 20 PCA components, the curve corresponding to $\gamma ={10}^{-5}$ in Figure 10 is eliminating too many features that are needed for classification. Similar to Figure 9, the three other curves take fewer eigenvectors to reach maximum classification accuracy and also achieve a higher classification accuracy.

To evaluate the effectiveness of the invariant scattering transform at extracting the nonlinear features in the PondEx dataset, we compare different classifiers for the classification of the six targets. Since our goal is to use only the invariant scattering transform to isolate nonlinear features, we limited our analysis to linear classifiers, given the small sample size of the PondEx dataset. Before applying each classifier, we project the data into a lower-dimensional subspace using one of the following methods: Linear Discriminant Analysis (LDA), Multi-view Discriminant Analysis (MvDA), or Sparse Multi-view Discriminant Analysis (Sparse MvDA). Following this projection, the samples are classified using either a linear support vector machines or linear Logistic Regression. The results are shown in

Table 4. We evaluated the classification results across 10 random shuffles of the PondEx dataset, employing an 80/20 training/test split. For all listed techniques, the invariant scattering transform was first applied to extract features from PondEx images. Subsequently, the resulting invariant scattering coefficients were projected into a lower-dimensional space using one of the three techniques mentioned. Following dimensionality reduction, either a linear support vector machine or linear Logistic Regression was employed to classify the projected data from all listed methods.

Classifier	Dimensionality Reduction	Accuracy
	LDA	$95.3\pm 5.9\%$
Logistic Regression	MvDA	$97.3\pm 4.4\%$
	Sparse MvDA	$96.6\pm 6.1\%$
	LDA	$95.3\pm 4.2\%$
Support Vector Machine	MvDA	$98.6\pm 2.6\%$
	Sparse MvDA	$97.3\pm 3.3\%$

. The two classifiers achieve similar classification accuracies across each of the three projection methods, with the SVM slightly outperforming Logistic Regression, which is typical when comparing the two methods.

To visualize the misclassification patterns, we present the classification results from Table 4 as confusion matrices in Figure 11. These confusion matrices are row normalized using the mean classification accuracy across 10 different splits. In the confusion matrix grid in Figure 11, the left column represents results using the SVM as the classifier, while the right column corresponds to Logistic Regression. The rows of the confusion matrix grid indicate the use of LDA, MvDA, or Sparse MvDA as the subspace projection method. A consistent misclassification across all confusion matrices is the mortar being incorrectly classified as rock #1. This misclassification is most pronounced when LDA is used for subspace projection. Due to the small sample size of the dataset, this misclassification is most likely caused by LDA, as well as MvDA and Sparse MvDA, not having enough samples to separate the distributions between the mortar and rock #1. Additionally, in some confusion matrices, the steel artillery shell (AL UXO Shell) is misclassified as the aluminum artillery shell (AL UXO Shell). Both the aluminium artillery shell and the steel artillery shell were originally machined from the same CAD drawing and thus have similar geometries [61], which could account for the occasional misclassification. Given the results presented in Table 4 and Figure 11, the SVM is preferable to Logistic Regression as the classifier. Additionally, while MvDA achieves the highest classification accuracy among the subspace projection methods, we can trade off a slight decrease in accuracy for greater interpretability by using Sparse MvDA.

To verify that Sparse MvDA is learning the correct features for discrimination, the magnitude of the coefficients of the eigenvector matrix $\tilde{\mathbf{W}}$ are plotted in Figure 12a–c for $\gamma$s taken from $\{{10}^{-5},{10}^{-7},{10}^{-8}\}$, respectively. Plotting the magnitude of the coefficients in $\tilde{\mathbf{W}}$ allows one to visually inspect how sparse the matrix $\tilde{\mathbf{W}}$ is. Figure 12a shows the magnitude coefficients for $\tilde{\mathbf{W}}$ when $\gamma ={10}^{-5}$. We can see in Figure 12a that the majority of the coefficients are nearly zero, with only a few non-zero coefficients that contain most of the energy. The color bar on the right of Figure 12a shows that coefficients that appear white have a value around 0.8. Since the eigenvectors are normalized to be unit length, this means that the coefficients that appear white contain the majority of the energy of the eigenvector. Additionally, the coefficients in Figure 12a are arranged such that horizontal strips are visible. As was previously mentioned, this is a byproduct of the row sparsity property of the $L_{2,1}$ norm. The strips are visible because many of the non-informative features have been removed.

Looking at Figure 12b,c, which show the magnitude coefficients for $\gamma ={10}^{-7}$ and $\gamma ={10}^{-8}$, respectively, we see that the horizontal strip pattern is again visible. Comparing Figure 12a with Figure 12b,c, it is apparent that the energy of the eigenvectors in Figure 12a is concentrated in a fewer number of coefficients. The energy of the coefficients in Figure 12b,c are spread out across a greater number of coefficients. This is to be expected since increasing the sparsity parameter, $\gamma$, promotes the removal of more features in $\tilde{\mathbf{W}}$. This also explains why we see the $\gamma ={10}^{-5}$ curve in Figure 9 and Figure 10 lagging behind the other curves. By increasing $\gamma$ to ${10}^{-5}$, the algorithm becomes excessively restrictive, forcing the removal of both noisy features and those critical for classification.

To achieve a suitable sparse representation, it is important to train the algorithm until it converges. A proof is provided in Appendix A, which shows the objective function in Equation (32) is a monotonically decreasing function. The minimization procedure in Algorithm A1 continues until the difference in the loss value between two successive iterations is less than a predefined threshold (we use ${10}^{-8}$). Figure 13a–c show the loss curves for Sparse MvDA for $\gamma$ values of ${10}^{-5}$, ${10}^{-7}$, and ${10}^{-8}$, respectively. All three curves converge to a minimum rapidly. The curve for $\gamma ={10}^{-5}$ takes the greatest number of iterations to converge (18 iterations), whereas the loss curve for $\gamma ={10}^{-8}$ in Figure 13c takes only 4 iterations to converge. Figure 13a takes more iterations because a stronger $L_{2,1}$ penalty is being used and thus requires more features to be eliminated for the $\tilde{\mathbf{W}}$ matrix.

8. Conclusions

In this work, we developed a multi-stage algorithm for extracting salient features and selecting the most discriminative features. Feature extraction was performed using the invariant scattering transform, which mimics the nonlinear feature extraction of convolutional neural networks without needing to solve a non-convex optimization problem. After feature extraction, the most discriminative features were selected using Multi-view Discriminant Analysis with a $L_{2,1}$ sparsity penalty. MvDA finds projection vectors that maximize the separation between class means when multiple sets of variables are projected into a shared lower-dimensional space. The additional $L_{2,1}$ sparsity penalty promotes row-wise sparsity of the projection matrix. Row-wise sparsity removes noisy features that negatively bias the projections of the data, which helps prevent overfitting. Using the discriminative features selected by Sparse MvDA, we achieved a mean classification accuracy of $97.3\pm 3.3\%$ over 10 random shuffles of the data. We also demonstrated that the projection matrix $\tilde{\mathbf{W}}$ learned by Sparse MvDA becomes row sparse due to the $L_{2,1}$ sparsity penalty. Visual inspection of the magnitude of the coefficients in $\tilde{\mathbf{W}}$ shows that many of the coefficients are zero, with the non-zero coefficients being the most discriminative features. We observe that the iterative algorithm used for finding the sparse solution converges rapidly, usually taking less than 10 iterations. Additionally, we showed that using the invariant scattering transform increases the classification accuracy of both supervised and unsupervised dimensionality reduction techniques when compared with the standard acoustic color representation of sonar backscattering data.

Future steps include extending Sparse MvDA to nonlinear dimensionality reduction using the kernel method. Kernel methods allow one to project data into a higher-dimensional feature space where linear dimensionality reduction can be performed. Kernel methods would be useful if the scattering coefficients can be separated by a higher-order polynomial but not a simple linear decision boundary. This extension would be relatively straightforward since the kernel trick maps data points ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ to higher dimensions via their inner product. We show in the paper that MvDA can be expressed as the inner product of the data points in the training set.