Mutual-Energy Inner Product Optimization Method for Constructing Feature Coordinates and Image Classification in Machine Learning

Abstract

As a key task in machine learning, data classification is essential to find a suitable coordinate system to represent the data features of different classes of samples. This paper proposes the mutual-energy inner product optimization method for constructing a feature coordinate system. First, by analyzing the solution space and eigenfunctions of the partial differential equations describing a non-uniform membrane, the mutual-energy inner product is defined. Second, by expressing the mutual-energy inner product as a series of eigenfunctions, it shows the significant advantage of enhancing low-frequency features and suppressing high-frequency noise, compared to the Euclidean inner product. And then, a mutual-energy inner product optimization model is built to extract the data features, and the convexity and concavity properties of its objective function are discussed. Next, by combining the finite element method, a stable and efficient sequential linearization algorithm is constructed to solve the optimization model. This algorithm only solves positive definite symmetric matrix equations and linear programming with a few constraints, and its vectorized implementation is discussed. Finally, the mutual-energy inner product optimization method is used to construct feature coordinates, and multi-class Gaussian classifiers are trained on the MINST training set. Good prediction results of the Gaussian classifiers are achieved on the MINST test set.

1. Introduction

In machine learning, data classification plays a very important role. Up to now, a large number of data classification methods have emerged and powered the development of machine learning and its practical applications in different domains [1], such as image detection [2], speech recognition [3], text understanding [4], disease diagnosis [5,6], and financial prediction [7].

Currently, popular data classification methods include the Support Vector Machine (SVM) [8,9], Decision Tree (DT) [10,11], Naive Bayes (NB) [12], K-Nearest Neighbors (KNN) [13], Random Forest (RF) [14], Deep Learning (DL) [15], and Deep Reinforcement Learning (DRL) [16]. SVM is based on optimization theory [17]. DL is implemented through a multilayer neural network under the guidance of optimization techniques, such as the stochastic gradient descent algorithm [18]. DRL combines DL with Reinforcement Learning, and it is effective in real-time scenarios [16]. The others fall in the title of statistical methods [19,20].

Many comparative studies are employed to evaluate these classification methods by analyzing their accuracies, time costs, stability, and sensitivity, as well as their advantages and disadvantages [21,22,23]. SVM is efficient when there is a clear margin of separation between the classes, but the choice of its kernel function is difficult, and it does not work with noisy datasets [23,24]. DL is developing rapidly, but its training is a very time-consuming process because a large number of parameters need to be optimized through the stochastic gradient descent algorithm. In addition, some hyperparameters in DL are set empirically, such as the number of layers in the neural network, the number of nodes in each layer, and the learning rate, resulting in high sensitivity in the performance, dependent on the hyperparameters and specific problems [25,26]. KNN and DT are easy to apply. However, KNN requires the calculation of the Euclidean distance between all the points, leading to a high computation cost. DT is unsuitable for continuous variables, and it has a problem of overfitting [23]. Other classical methods also obtained great successes [27,28]. In order to improve classification accuracy, an ensemble learning scheme, such as AdaBoost [29,30], Bagging [31], Stacking [32] and Gradient Boosting [33], is usually adopted to solve an intricate or large-scale problem [34,35].

Inspired by the ability of our brain to recognize the musical notes played by any musical instrument in a noisy environment, this paper proposes an optimization method for constructing feature coordinates for data classification by simulating a non-uniform membrane structure model. No matter how complex a musical instrument’s structure is, or how different its vibration patterns are, when we listen to a piece of music played by an instrument, our brain can extract the fundamental tone of its vibration at every moment, and can recognize the beautiful melody as time goes by. Mathematically, this can be clearly explained. The vibration of the musical instrument at every moment is adaptively expanded on its own eigenfunction system, and our brain can grasp the lowest eigenvalue and its eigenfunction components corresponding to the musical notes every moment, and enjoy the beautiful melody over time. In order to extract the data features from complex samples, we simulate the adaptively generating process of the eigenfunction coordinate system of a musical instrument and build the mapping from data features to the low-frequency subspace of the eigenfunction system. Through analyzing the solution space and the eigenfunctions of the partial differential equations describing the vibration of a non-uniform membrane, which is a simple musical instrument, the mutual-energy inner product is defined and is used to extract data features. The introduction of the mutual-energy inner product can not only avoid generating an eigenfunction system to reduce the computational complexity, but also can enhance the feature information and filter out data noise, furthermore, it can benefit the simplification of the data classifier training.

The full paper is divided into six sections. Section 1 briefly introduces popular data classification methods and the research background. Section 2 analyzes the solution space of the partial differential equations describing a non-uniform membrane, and defines the concept of the mutual-energy inner product. Section 3, by making use of the eigenvalues and the eigenfunctions of the non-uniform membrane vibration equations, the mutual-energy inner product is expressed as a series of eigenfunctions, and its potential in data classification is pointed out for enhancing feature information and filtering out data noise. Section 4 builds a mutual-energy inner product optimization model and discusses the convexity and concavity properties of its objective function. Section 5 designs a sequential linearization algorithm to solve the optimization model by combing the finite element method (FEM). Section 6, the mutual-energy inner product optimization method for constructing feature coordinates is applied to a 2-D image classification problem, and numerical examples are given in combination with Gaussian classifiers and the handwritten digit MINST dataset. Section 7, we summarize the full paper and introduce the future scope of the work.

2. Mutual-Energy Inner Product

Consider the linear partial differential equations

$$\{\begin{array}{ll}L\left[u\left(\overrightarrow{x}\right)\right]=f\left(\overrightarrow{x}\right)& \overrightarrow{x}\in \mathsf{\Omega }\\ l\left[u\left(\overrightarrow{x}\right)\right]=0& \overrightarrow{x}\in \mathsf{\Gamma }\end{array},$$

(1)

where $L\left[u\left(\overrightarrow{x}\right)\right]$ is a homogeneous linear self-adjoint differential operator; $f\left(\overrightarrow{x}\right)$ is a piecewise continuous function; $\mathsf{\Omega }$ is the domain of definition, with a boundary $\mathsf{\Gamma }$; and $l\left[u\left(\overrightarrow{x}\right)\right]$ is a homogeneous linear differential operator on the boundary $\mathsf{\Gamma }$, describing the Robin boundary condition.

$$\{\begin{array}{l}L\left[u\left(\overrightarrow{x}\right)\right]=-{\displaystyle \sum _{i=1}^n\frac{\partial }{\partial x_i}\left(p\left(\overrightarrow{x}\right)\frac{\partial u\left(\overrightarrow{x}\right)}{\partial x_i}\right)+q\left(\overrightarrow{x}\right)u\left(\overrightarrow{x}\right)}\\ l\left[u\left(\overrightarrow{x}\right)\right]=p\left(\overrightarrow{x}\right){\displaystyle \frac{\partial u\left(\overrightarrow{x}\right)}{\partial n}}+\sigma \left(\overrightarrow{x}\right)u\left(\overrightarrow{x}\right)\end{array}.$$

(2)

Expression (2) can be regarded as static equilibrium equations of a simple elastic structure, such as a 1-D string and a 2-D membrane, and can be expanded to an n-dimensional problem. For a 2-dimensional problem, $\mathsf{\Omega }$ is a domain occupied by a membrane with its boundary $\mathsf{\Gamma }$; $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ stand for the elastic modulus and distributed support elastic coefficient of the membrane, respectively; $\sigma \left(\overrightarrow{x}\right)$ is the support elastic coefficient on the boundary; $f\left(\overrightarrow{x}\right)$ is an external force acting on the membrane; $u\left(\overrightarrow{x}\right)$ is the deformations of the membrane due to $f\left(\overrightarrow{x}\right)$, and has a piecewise continuous first-order derivative; $\partial u/\partial n$ is the derivative of the deformations in the outward-pointing normal direction of $\mathsf{\Gamma }$. In this research, it is required that, $p\left(\overrightarrow{x}\right)$, $q\left(\overrightarrow{x}\right)$, $\sigma \left(\overrightarrow{x}\right)$ are piecewise continuous functions, and $p\left(\overrightarrow{x}\right)>0$, $q\left(\overrightarrow{x}\right)>0$, $\sigma \left(\overrightarrow{x}\right)\ge 0$.

A structure subjected to an external force $f\left(\overrightarrow{x}\right)$ will generate the deformation $u\left(\overrightarrow{x}\right)$, and its deformation energy $E\left[u\left(\overrightarrow{x}\right)\right]$ can be expressed as

$$E\left[u\left(\overrightarrow{x}\right)\right]={\displaystyle \frac{1}{2}}{\displaystyle {\int }_{\mathsf{\Omega }}f\left(\overrightarrow{x}\right)u\left(\overrightarrow{x}\right)d\mathsf{\Omega }}.$$

(3)

If the structure is simultaneously subjected to another external force $g\left(\overrightarrow{x}\right)$, then it will generate an additional deformation $v\left(\overrightarrow{x}\right)$. The total deformation $u\left(\overrightarrow{x}\right)+v\left(\overrightarrow{x}\right)$ satisfies the superposition principle due to the linearity of Expression (1). The deformation $v\left(\overrightarrow{x}\right)$ can cause additional work performed by $f\left(\overrightarrow{x}\right)$. Generally, the additional deformation energy $U\left[u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right]$ is called the mutual energy between $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ or the mutual work between $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$. The mutual energy describes the correlation of the two external forces, and can be expressed as

$$U\left[u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right]={\displaystyle {\int }_{\mathsf{\Omega }}f\left(\overrightarrow{x}\right)v\left(\overrightarrow{x}\right)d\mathsf{\Omega }}.$$

(4)

Substituting Expression (1) into Expression (4), by integrating by parts, we obtain

$$\begin{array}{ll}\hfill U\left[u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right]& ={\displaystyle {\int }_{\mathsf{\Omega }}\left({\displaystyle \sum _{i=1}^n\frac{\partial u\left(\overrightarrow{x}\right)}{\partial x_i}p\left(\overrightarrow{x}\right)\frac{\partial v\left(\overrightarrow{x}\right)}{\partial x_i}}+u\left(\overrightarrow{x}\right)q\left(\overrightarrow{x}\right)v\left(\overrightarrow{x}\right)\right)d\mathsf{\Omega }}\hfill \\ & +{\displaystyle {\int }_{\mathsf{\Gamma }}u\left(\overrightarrow{x}\right)\sigma \left(\overrightarrow{x}\right)v\left(\overrightarrow{x}\right)d\mathsf{\Gamma }}\hfill \end{array}.$$

(5)

Expression (5) is a bilinear functional. Comparing Expressions (3) and (4), we have

$$E\left[u\left(\overrightarrow{x}\right)\right]={\displaystyle \frac{1}{2}}U\left[u\left(\overrightarrow{x}\right),u\left(\overrightarrow{x}\right)\right].$$

(6)

Due to $p\left(\overrightarrow{x}\right)>0$, $q\left(\overrightarrow{x}\right)>0$, $\sigma \left(\overrightarrow{x}\right)\ge 0$, according to the Expressions (5) and (6), the mutual energy satisfies

$$U\left[u\left(\overrightarrow{x}\right),u\left(\overrightarrow{x}\right)\right]\ge 0\hspace{1em}\left(if\hspace{0.17em}only\hspace{0.17em}if\hspace{1em}u\left(\overrightarrow{x}\right)=0,\hspace{1em}U\left[u\left(\overrightarrow{x}\right),u\left(\overrightarrow{x}\right)\right]=0\right).$$

(7)

Expression (7) describes a simple physical phenomenon: when the elastic modulus of the structural material is positive, if the structure deforms, deformation energy is generated; otherwise, the deformation energy is zero.

Expression (5) also shows that the mutual energy is symmetrical and satisfies the commutative law. Combined with Expression (7), it can be inferred that the mutual energy satisfies the Cauchy–Schwarz inequality

$${\left(U\left[u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right]\right)}^2\le U\left[u\left(\overrightarrow{x}\right),u\left(\overrightarrow{x}\right)\right]·U\left[v\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right].$$

(8)

The Expressions (7) and (8) show that the mutual energy can be regarded as an inner product of the structural deformation functions. For simplicity, we use ${\langle u,v\rangle }_U$ and $\langle u,v\rangle$ to represent the mutual-energy inner product and the Euclidean inner product, respectively; that is,

$$\{\begin{array}{l}{\langle u,v\rangle }_U=U\left[u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right)\right]\\ \langle u,v\rangle ={\displaystyle {\int }_{\mathsf{\Omega }}u\left(\overrightarrow{x}\right)v\left(\overrightarrow{x}\right)d\mathsf{\Omega }}\end{array}.$$

(9)

We define ${\Vert u\Vert }_2$ as the norm derived from $\langle u,v\rangle$, and ${\Vert u\Vert }_U$ as the norm derived from ${\langle u,v\rangle }_U$. Based on Expression (6), ${\Vert u\Vert }_U$ satisfies

$${\Vert u\Vert }_U=\sqrt{2E\left[u\left(\overrightarrow{x}\right)\right]}.$$

(10)

${\Vert u\Vert }_U$ is proportional to the square root of the deformation energy, and is also the energy norm. According to the Cauchy–Schwarz inequality (8), ${\Vert u\Vert }_U$ satisfies the triangle inequality

$${\Vert u+v\Vert }_U\le {\Vert u\Vert }_U+{\Vert v\Vert }_U.$$

(11)

Based on Expression (1), when a structure is subjected to a piecewise continuous external force, its deformation function has piecewise continuous first-order derivatives on the domain $\mathsf{\Omega }$ and satisfies the boundary condition $l\left[u\left(\overrightarrow{x}\right)\right]=0$. The set of these deformation functions can span a space $V\left(\mathsf{\Omega }\right)$, which can be equipped either with the Euclidean inner product $\langle u,v\rangle$ or with the mutual-energy inner product ${\langle u,v\rangle }_U$.

In addition, applying the variational principle, Expression (1) can also be rewritten as the minimum energy principle expression

$$\underset{u\left(\overrightarrow{x}\right)}{\min }\hspace{1em}\pi \left[u\right]={\displaystyle \frac{1}{2}}{\langle u,u\rangle }_U-\langle f,u\rangle .$$

(12)

Here, the feasible domain of $u\left(\overrightarrow{x}\right)$ has piecewise continuous first-order derivatives on $\mathsf{\Omega }$, and does not need to satisfy homogeneous boundary conditions.

3. Signal Processing Property of Mutual-Energy Inner Product

The eigenequation of $L\left[u\left(\overrightarrow{x}\right)\right]$ can be written as

$$\{\begin{array}{ll}L\left[u\left(\overrightarrow{x}\right)\right]=\lambda u\left(\overrightarrow{x}\right)& \overrightarrow{x}\in \mathsf{\Omega }\\ p\left(\overrightarrow{x}\right){\displaystyle \frac{\partial u\left(\overrightarrow{x}\right)}{\partial n}}+\sigma \left(\overrightarrow{x}\right)u\left(\overrightarrow{x}\right)=0& \overrightarrow{x}\in \mathsf{\Gamma }\end{array}.$$

(13)

For Expression (13), its non-zero solutions $\phi \left(\overrightarrow{x}\right)$ and the corresponding coefficients $\lambda$ are called eigenfunctions and eigenvalues, respectively. These eigenfunctions and eigenvalues have the following properties due to $p\left(\overrightarrow{x}\right)>0$, $q\left(\overrightarrow{x}\right)>0$, $\sigma \left(\overrightarrow{x}\right)\ge 0$ [36].

(1)

Expression (13) has infinite eigenvalues ${\lambda }_n$ and eigenfunctions ${\phi }_n\left(\overrightarrow{x}\right)$, i.e., $n=1,2\dots \infty$. If all the eigenvalues are ranked like ${\lambda }_1\le {\lambda }_2\dots {\lambda }_n\le \dots$ then they satisfy ${\lambda }_1>0$ and $\underset{n\to \infty }{\lim }{\lambda }_n=+\infty$. Meanwhile, ${\lambda }_n$ has continuous dependence on $p\left(\overrightarrow{x}\right)$, $q\left(\overrightarrow{x}\right)$ and $\sigma \left(\overrightarrow{x}\right)$, and will increase with the increase in $p\left(\overrightarrow{x}\right)$, $q\left(\overrightarrow{x}\right)$ and $\sigma \left(\overrightarrow{x}\right)$.

(2)

Normalized eigenfunctions ${\phi }_n\left(\overrightarrow{x}\right)$ satisfy the orthogonality condition (14), and can form a set of orthogonal and complete basis functions to span the deformation function space $V\left(\mathsf{\Omega }\right)$.

$$\langle {\phi }_m,{\phi }_n\rangle =\{\begin{array}{c}1\hspace{1em}\hspace{1em}m=n\\ 0\hspace{1em}\hspace{1em}m\ne n\end{array}.$$

(14)

Therefore, the solutions of Expression (1) can be expressed by ${\phi }_n\left(\overrightarrow{x}\right)$. For $\forall u\left(\overrightarrow{x}\right)\in V\left(\mathsf{\Omega }\right)$, $u\left(\overrightarrow{x}\right)$ can be presented as a series of eigenfunctions satisfying absolute and uniform convergence, i.e.,

$$u\left(x\right)={\displaystyle \sum _{n=1}^{+\infty }{c}_n{\phi }_n\left(x\right)},\hspace{1em}{c}_n=\langle u,{\phi }_n\rangle .$$

(15)

Expression (15) has profound physical meaning. $\sqrt{{\lambda }_n}$ and ${\phi }_n\left(\overrightarrow{x}\right)$ are the $n-th$ order structural natural frequency and the $n-th$ order vibration mode. If $u\left(\overrightarrow{x}\right)$ is regarded as a vibration amplitude function, it can be decomposed into a superposition of the vibration modes at each order natural frequency, where the coefficient $c_n$ is the vibration magnitude at ${\phi }_n\left(\overrightarrow{x}\right)$. This is equivalent to spectral decomposition. Imagine such a scene. When we enjoy a piece of music, our brains constantly decompose the instantaneous vibration amplitude $u\left(\overrightarrow{x}\right)$ according to Expression (15), and meanwhile, perceive the vibration coefficients $c_n$ and mark them with $\sqrt{{\lambda }_n}$. For a musical instrument, $\sqrt{{\lambda }_1}$ is its fundamental frequency (tone) and the remaining eigenvalues are overtones. Different musical instruments have different vibration patterns, and their eigenfunctions $\left\{{\phi }_n\left(\overrightarrow{x}\right),n=1,2\dots \right\}$ are also different. However, after tuning the tone of the different musical instruments, the fundamental frequency of each note is consistent.

The eigenfunctions and eigenvalues satisfy Expression (13), so we have

$$L\left[{\phi }_n\left(\overrightarrow{x}\right)\right]={\lambda }_n{\phi }_n\left(\overrightarrow{x}\right).$$

(16)

Multiplying both sides of Expression (16) by ${\phi }_n\left(\overrightarrow{x}\right)$ and integrating by parts, we can yield

$${\langle {\phi }_n,{\phi }_m\rangle }_U=\{\begin{array}{ll}{\lambda }_n& m=n\\ 0& m\ne n\end{array}.$$

(17)

Expression (17) shows that the eigenfunctions also satisfy the orthogonal condition with respect to the mutual-energy inner product. So, these eigenfunctions $\left\{{\phi }_n\left(\overrightarrow{x}\right)/\sqrt{{\lambda }_n},\hspace{0.17em}n=1,2\dots \right\}$ can also be used as basis functions to span the mutual-energy inner product space $V\left(\mathsf{\Omega }\right)$.

Substituting Expression (15) into Expression (5) and applying Expression (17), we have

$${\langle u,u\rangle }_U={\displaystyle \sum _{n=1}^{\infty }{c}_n^2{\lambda }_n}.$$

(18)

If $u\left(\overrightarrow{x}\right)$ satisfies the normalization condition ${\Vert u\Vert }_2^2=1$ or ${\displaystyle \sum _{n=1}^{\infty }{c}_n^2}=1$, based on the Expressions (14) and (18), the eigenvalue ${\lambda }_n$ satisfies

$$\begin{array}{l}{\lambda }_n=\underset{u\in V\left(\mathsf{\Omega }\right)}{\min }{\langle u,u\rangle }_U\\ \hspace{1em}\hspace{1em}s.t.\\ \hspace{1em}\hspace{1em}\hspace{1em}\langle u,u\rangle =1\\ \hspace{1em}\hspace{1em}\hspace{1em}\langle u,{\phi }_i\rangle =0\hspace{1em}i=1,2\dots n-1\end{array},$$

(19)

where the optimal solution of $u\left(\overrightarrow{x}\right)$ is the eigenfunction ${\phi }_n\left(\overrightarrow{x}\right)$.

Similarly, the deformation $v\left(\overrightarrow{x}\right)$ caused by $g\left(\overrightarrow{x}\right)$ can be expressed as

$$v\left(\overrightarrow{x}\right)={\displaystyle \sum _{n=1}^{\infty }{d}_n{\phi }_n\left(\overrightarrow{x}\right)},$$

(20)

where $d_n$ is the amplitude coefficient and can be interpreted as the component of $v\left(\overrightarrow{x}\right)$ at the $n-th$ vibration mode ${\phi }_n\left(\overrightarrow{x}\right)$. Substituting Expression (20) into Expression (12) and using the orthogonal condition (17), we have

$$\underset{d_n,n=1,2\dots \infty }{\min }\hspace{1em}\pi \left[d_1,d_2\dots \right]={\displaystyle \sum _{n=1}^{\infty }\left(\frac{1}{2}{\lambda }_n{d}_n^2-g_n{d}_n\right)},$$

(21)

where the coefficient $g_n$ is the projection of $g\left(\overrightarrow{x}\right)$ on ${\phi }_n\left(\overrightarrow{x}\right)$ with respect to the Euclidean inner product

$$g_n=\langle g,{\phi }_n\rangle .$$

(22)

Enforcing the derivative of $\pi \left[d_1,d_2\dots \right]$ in Expression (21) with respect to $d_n$ to zero, we have

$$d_n={\displaystyle \frac{g_n}{{\lambda }_n}}\hspace{1em}\hspace{1em}n=1,2\dots \infty .$$

(23)

According to the series representation of $u\left(\overrightarrow{x}\right)$ in Expression (15), if $u\left(\overrightarrow{x}\right)$ is the deformation caused by $f\left(\overrightarrow{x}\right)$, the coefficient $c_n$ satisfies

$$c_n={\displaystyle \frac{f_n}{{\lambda }_n}}\hspace{1em},\hspace{1em}{f}_n=\langle f,{\phi }_n\rangle \hspace{1em}\hspace{1em}n=1,2\dots \infty .$$

(24)

Substituting the Expressions (15), (20), (23) and (24) into Expression (5) and using the orthogonal condition (17), we have

$${\langle u,v\rangle }_U={\displaystyle \sum _{n=1}^{\infty }\frac{f_n·g_n}{{\lambda }_n}}.$$

(25)

Generally speaking, the external force $f\left(\overrightarrow{x}\right)\notin V\left(\mathsf{\Omega }\right)$, because $f\left(\overrightarrow{x}\right)$ does not satisfy the homogeneous boundary conditions, i.e., $l\left[f\left(\overrightarrow{x}\right)\right]\ne 0$. In this case, ${\displaystyle \sum _{n=1}^{\infty }{f}_n{\phi }_n\left(\overrightarrow{x}\right)}$ is equal to the projection of $f\left(\overrightarrow{x}\right)$ on $V\left(\mathsf{\Omega }\right)$ or the optimal approximation of $f\left(\overrightarrow{x}\right)$ in $V\left(\mathsf{\Omega }\right)$. Of course, in order to make $f\left(\overrightarrow{x}\right)\in V\left(\mathsf{\Omega }\right)$, we may expand the design domain and simplify the boundary condition. For example, after expanding the design domain, we can set a fixed boundary and let $\sigma \left(\overrightarrow{x}\right)=\infty$, or set a mirror boundary and let $\sigma \left(\overrightarrow{x}\right)=0$. In these cases, $l\left[f\left(\overrightarrow{x}\right)\right]=0$ and $f\left(\overrightarrow{x}\right)={\displaystyle \sum _{n=1}^{\infty }{f}_n{\phi }_n\left(\overrightarrow{x}\right)}$. Then, applying the orthogonal condition (14) yields

$$\langle f,g\rangle ={\displaystyle \sum _{n=1}^{\infty }{f}_n·g_n}.$$

(26)

After $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$ are expressed as a superposition of the eigenfunctions of the operator $L$, through comparing the mutual-energy inner product ${\langle u,v\rangle }_U$ in Expression (25) and the Euclidean inner product $\langle f,g\rangle$ in Expression (26), it can be found that the mutual-energy inner product has the advantage of enhancing the low-frequency coordinate components ($\sqrt{{\lambda }_n}<1$) and suppressing the high-frequency coordinate components ($\sqrt{{\lambda }_n}>1$). In other words, if $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$ are regarded as signals, the mutual-energy inner product can augment the low-frequency eigenfunction components and filter out the high-frequency eigenfunction components of the signals, with the help of a structural model.

4. Mutual-Energy Inner Product Optimization Model for Feature Extraction

Assume that $D={\left\{\left(X^{\left(i\right)}\left(\overrightarrow{x}\right)\hspace{0.17em},\hspace{0.17em}{y}^{\left(i\right)}\right)\right\}}_{i=1}^N$ is a training dataset with $N$ samples, and each sample is represented as $X^{\left(i\right)}\left(\overrightarrow{x}\right)$, while $y^{\left(i\right)}$ represents the class labels. For example, the samples are divided into two classes, and $D$ includes two subsets $D_1={\left\{X^{\left(i\right)}\left(x\right)|\left(X^{\left(i\right)}\left(\overrightarrow{x}\right)\hspace{0.17em}\in D,\hspace{0.17em}{y}^{\left(i\right)}=1\right)\right\}}_{i=1}^{N_1}$ and $D_0={\left\{X^{\left(i\right)}\left(x\right)|\left(X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D\hspace{0.17em},\hspace{0.17em}{y}^{\left(i\right)}=0\right)\right\}}_{i=1}^{N_0}$, where $N=N_1+N_0$. Generally, the samples in different classes are assumed to be random variables, which are independent and have identical distributions.

We hope to find an appropriate feature coordinate system to represent $X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D$ and use fewer coordinate components to classify the samples. If there is no further information, we may select the means of the probability distribution of $D_1$ and $D_0$ as reference features. In order to design a feature extraction model, two points should be considered: one to enhance the feature information, and the other to suppress the effect of random noise. We resort to a structural model and use the mutual-energy inner product to extract the features. Its main idea is to map the data features to a low-frequency eigenfunction space of the structural model.

If $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$ are used to represent the means of the probability distribution of $D_1$ and $D_0$, respectively, their unbiased estimates can be written as

$$f\left(\overrightarrow{x}\right)={\displaystyle \frac{1}{N_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_1}{X}^{\left(i\right)}\left(\overrightarrow{x}\right)},\hspace{1em}g\left(\overrightarrow{x}\right)={\displaystyle \frac{1}{N_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_0}{X}^{\left(i\right)}\left(\overrightarrow{x}\right)}.$$

(27)

We regard $X^{\left(i\right)}\left(\overrightarrow{x}\right)$, $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$ as external forces acting on the structural model, and use $d^{\left(i\right)}\left(\overrightarrow{x}\right)$, $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ to represent their corresponding deformations, respectively. If we represent the selected reference feature in $V\left(\mathsf{\Omega }\right)$ as $\alpha \left(\overrightarrow{x}\right)$, we can use the mutual-energy inner product ${\langle d^{\left(i\right)},\alpha \rangle }_U$ to extract the feature coordinate component of $X^{\left(i\right)}\left(\overrightarrow{x}\right)$. In order to construct the feature extraction optimization model, we first select $u\left(\overrightarrow{x}\right)$ as the reference feature $\alpha \left(\overrightarrow{x}\right)$ and try to explore the physical meanings of the structural model when ${\langle d^{\left(i\right)},u\rangle }_U$ is the maximum, the minimum or equal to zero.

In order to enhance the feature information of the samples in $D_1$, a high statistical mean value ${\mu }_1$ should be given

$${\mu }_1={\displaystyle \frac{1}{N_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_1}{\langle d^{\left(i\right)},u\rangle }_U}={\langle {\displaystyle \frac{1}{N_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_1}{d}^{\left(i\right)}},u\rangle }_U={\langle u,u\rangle }_U,$$

(28)

with a primary objective

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\max }\hspace{1em}{\mu }_1={\langle u,u\rangle }_U.$$

(29)

In Expression (29), the mutual-energy inner product and deformations are functions of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$, and its physical meaning is not intuitive. So, next, we will conduct a quantitative analysis to reveal the structural characteristics hidden in Expression (29).

According to the minimum energy principle (12), if an optimal solution of $u\left(\overrightarrow{x}\right)$ is obtained, the derivative of the objective at the optimal solution in any direction $\delta u\left(\overrightarrow{x}\right)$ is zero, satisfying

$${\displaystyle \frac{d}{d\tau }}\pi \left[u\left(\overrightarrow{x}\right)+\tau ·\delta u\left(\overrightarrow{x}\right)\right]=0\hspace{1em}\hspace{1em}\forall \delta u\left(\overrightarrow{x}\right)\in C\left(\mathsf{\Omega }\right).$$

(30)

Through calculating Expression (30), we obtain the relationship between $u\left(\overrightarrow{x}\right)$ and $f\left(\overrightarrow{x}\right)$

$${\langle u,\delta u\rangle }_U-\langle f,\delta u\rangle =0\hspace{1em}\hspace{1em}\forall \delta u\left(\overrightarrow{x}\right)\in C\left(\mathsf{\Omega }\right).$$

(31)

Expression (31) is a structural static equilibrium equation, and is also a constraint on $u\left(\overrightarrow{x}\right)$ in optimization problem (29). In Expression (31), letting $\delta u\left(\overrightarrow{x}\right)=u\left(\overrightarrow{x}\right)$ yields

$${\langle u,u\rangle }_U=\langle f,u\rangle .$$

(32)

Substituting Expression (32) into Expression (12) yields the optimal value $\pi \left[u\right]$ of the objective

$$\pi \left[u\right]=-{\displaystyle \frac{1}{2}}{\langle u,u\rangle }_U.$$

(33)

Through substituting Expressions (12) and (33) into the optimization problem (29), Expression (29) is transformed into an unconstrained optimization problem

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right),u\left(\overrightarrow{x}\right)}{\max }\hspace{1em}{\mu }_1\left[u,p,q\right]=2\langle f,u\rangle -{\langle u,u\rangle }_U.$$

(34)

If $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ are given, ${\mu }_1\left[u,p,q\right]$ in Expression (34) is a quadratic and concave functional with respect to $u\left(\overrightarrow{x}\right)$, due to $p\left(\overrightarrow{x}\right)>0$ and $q\left(\overrightarrow{x}\right)>0$. If $u\left(\overrightarrow{x}\right)$ is given, ${\mu }_1\left[u,p,q\right]$ is a linear function with respect to $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$.Through using the Univariate Search Method to solve Expression (34), if $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ are given, the maximum value of ${\mu }_1\left[u,p,q\right]$ can be found by solving Expression (31) for $u\left(\overrightarrow{x}\right)$, and if $u\left(\overrightarrow{x}\right)$ is given, the maximum value of ${\mu }_1\left[u,p,q\right]$ will be reached on the lower bounds of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$. So, the lower bounds of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ must be larger than zero to ensure that Expression (29) has a finite optimal solution. In addition, the upper bounds of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ should also be constrained to avoid the trivial solution $u\left(\overrightarrow{x}\right)=0$. Therefore, when the optimization objective is to maximize the mutual-energy inner product, as shown in Expression (29), its optimal structural model would be the minimum stiffness structure, and the selected feature belongs to a low-frequency eigenfunction subspace. On the contrary, if the optimization objective is to minimize the mutual-energy inner product, the optimal structural model would be the maximum stiffness, and the selected feature would be mapped to a high-frequency eigenfunction subspace.

In addition, when using the mutual-energy inner product to extract feature information $f\left(\overrightarrow{x}\right)$ of the samples in $D_1$, the feature information $g\left(\overrightarrow{x}\right)$ of the samples in $D_0$ should be suppressed. So, a small statistical mean value ${\mu }_0$ is given

$${\mu }_0={\displaystyle \frac{1}{N_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_0}{\langle d^{\left(i\right)},u\rangle }_U}={\langle {\displaystyle \frac{1}{N_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_0}{d}^{\left(i\right)}},u\rangle }_U={\langle v,u\rangle }_U.$$

(35)

Here, we may set ${\mu }_0$ to be zero or even negative, and impose constraints on the structural model

$${\langle v,u\rangle }_U\le 0.$$

(36)

In Expression (31), setting $\delta u\left(\overrightarrow{x}\right)=v\left(\overrightarrow{x}\right)$ yields ${\langle u,v\rangle }_U=\langle f,v\rangle$. Replacing $f\left(\overrightarrow{x}\right)$ with $g\left(\overrightarrow{x}\right)$, and exchanging $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$, we have

$${\langle u,v\rangle }_U=\langle f,v\rangle =\langle g,u\rangle .$$

(37)

If Expression (36) satisfies ${\langle v,u\rangle }_U=0$, then $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ are required to be orthogonal with respect to the mutual-energy inner product. Although the means of the two classes of the samples are generally not orthogonal in the continuous function space $C\left(\mathsf{\Omega }\right)$, i.e., $\langle f,g\rangle \ne 0$, the orthogonality of $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ can be easily realized according to Expression (37). For example, if setting $p\left(\overrightarrow{x}\right)=0$ and dividing the domain $\mathsf{\Omega }$ into two sub-regions according to the same or opposite signs of $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$, we can adjust $q\left(\overrightarrow{x}\right)$ in the two sub-regions and control the positive and negative work performed by the external forces $g\left(\overrightarrow{x}\right)$ on the deformations $u\left(\overrightarrow{x}\right)$, so as to make the total work $\langle g,u\rangle$ in Expression (37) zero. According to Expression (25), this can also be understood as designing a structural model and adjusting its eigenfunctions and eigenvalues, so as to use these eigenvalues as weights to achieve the weighted orthogonality of $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$. Further, ${\langle v,u\rangle }_U\le 0$ can be regarded as the relaxation of the orthogonal constraints on the mutual-energy inner product, which can be realized by adjusting $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ to make $\langle g,u\rangle <0$. Geometrically, this means that the angle between $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ in the mutual-energy inner product space $V\left(\mathsf{\Omega }\right)$ is not an acute angle. If ${\mu }_0$ is required to be minimal

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\hspace{1em}{\mu }_0={\langle v,u\rangle }_U,$$

(38)

based on Expression (12), similar to the discussion on Expression (29), the optimization problem (38) can be transformed into an unconstrained form

$$\underset{u\left(\overrightarrow{x}\right),v\left(\overrightarrow{x}\right),p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\underset{z\left(\overrightarrow{x}\right)}{\max }\hspace{1em}{\mu }_0\left[z,u,v,p,q\right],$$

(39)

where $z\left(\overrightarrow{x}\right)\in C\left(\mathsf{\Omega }\right)$ is a slack variable introduced to relax the constraint, which is the constraint of the static equilibrium equation describing the structural deformation due to $f\left(\overrightarrow{x}\right)$ and $g\left(\overrightarrow{x}\right)$ acting on the structure simultaneously. The objective can be expressed as

$${\mu }_0\left[z,u,v,p,q\right]={\displaystyle \frac{1}{2}}{\langle u,u\rangle }_U+{\displaystyle \frac{1}{2}}{\langle v,v\rangle }_U-{\displaystyle \frac{1}{2}}{\langle z,z\rangle }_U-\langle f,u-z\rangle -\langle g,v-z\rangle .$$

(40)

Obviously, if $p\left(\overrightarrow{x}\right)>0$ and $q\left(\overrightarrow{x}\right)>0$ are given, ${\mu }_0\left[z,u,v,p,q\right]$ is a quadratic functional of $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$, and $z\left(\overrightarrow{x}\right)$. ${\mu }_0\left[z,u,v,p,q\right]$ is convex with respect to $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$, and is concave with respect to $z\left(\overrightarrow{x}\right)$. If $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$ and $z\left(\overrightarrow{x}\right)$ are given, ${\mu }_0\left[z,u,v,p,q\right]$ is linear with respect to $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$.

In order to design a feature coordinate to classify the samples in $D$, the objective is to maximize ${\mu }_1-{\mu }_0$ first. By combining the Expressions (28) and (35), the optimization objective can be expressed as

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\hspace{1em}{\mu }_0-{\mu }_1={\langle v-u,u\rangle }_U.$$

(41)

Then, to improve the classification accuracy, the distributions of the samples in $D_1$ and $D_0$ along the feature coordinate $u\left(\overrightarrow{x}\right)$ should also be considered, and their variances should be small. The variances of $D_1$ and $D_0$ are high-order functions of $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$, $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$, so putting them into the optimization objective function (41) will destroy its low-order characteristics.

In order to improve the computational efficiency, the sum of the absolute values of the sample deviations from the mean are used to replace the variances, and only some samples in $D_1$ and $D_0$ are selected for calculation. In the subset $D_1$, we only select $M_1$ samples $S_1={\left\{X^{\left(i\right)}\left(\overrightarrow{x}\right)|X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_1,\langle X^{\left(i\right)},u\rangle <{\mu }_1\right\}}_{i=0}^{M_1}$, whose components on $u\left(\overrightarrow{x}\right)$ are less than ${\mu }_1$, and calculate their mean absolute deviation ${\delta }_1$. In the subset $D_0$, we only select $M_0$ samples $S_0={\left\{X^{\left(i\right)}\left(\overrightarrow{x}\right)|X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_0,\langle X^{\left(i\right)},u\rangle >{\mu }_0\right\}}_{i=0}^{M_0}$, whose components on $u\left(\overrightarrow{x}\right)$ are larger than ${\mu }_0$, and calculate their mean absolute deviation ${\delta }_0$. ${\delta }_1$ and ${\delta }_0$ can be expressed as

$$\{\begin{array}{l}{\delta }_1={\displaystyle \frac{1}{M_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_1}\langle f-X^{\left(i\right)},u\rangle }={\langle u,u\rangle }_U-\langle {\displaystyle \frac{1}{M_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_1}{X}^{\left(i\right)}},u\rangle \\ {\delta }_0={\displaystyle \frac{1}{M_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_0}\langle X^{\left(i\right)}-g,u\rangle }=\langle {\displaystyle \frac{1}{M_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_0}{X}^{\left(i\right)}},u\rangle -{\langle v,u\rangle }_U\end{array}.$$

(42)

Through using Expressions (41) and (42), and considering the means and the mean absolute deviations of the samples, the optimization objective can be written as

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\hspace{1em}J\left[p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)\right]=\lambda \left({\mu }_0-{\mu }_1\right)+\left(1-\lambda \right)\left({\delta }_0+{\delta }_1\right),$$

(43)

where $\lambda$ is a weight variable, satisfying $0\le \lambda \le 1$. To simplify Expression (42), the auxiliary deformation function $w\left(\overrightarrow{x}\right)\in V\left(\mathsf{\Omega }\right)$ is defined as

$${\langle w,\delta w\rangle }_U-\langle h,\delta w\rangle =0\hspace{1em}\hspace{1em}\forall \delta w\left(\overrightarrow{x}\right)\in C\left(\mathsf{\Omega }\right),$$

(44)

where $h\left(\overrightarrow{x}\right)$ can be regarded as an external force corresponding to $w\left(\overrightarrow{x}\right)$, satisfying

$$h\left(\overrightarrow{x}\right)={\displaystyle \frac{1}{M_0}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_0}{X}^{\left(i\right)}\left(\overrightarrow{x}\right)}-{\displaystyle \frac{1}{M_1}}{\displaystyle \sum _{X^{\left(i\right)}\left(\overrightarrow{x}\right)\in S_1}{X}^{\left(i\right)}\left(\overrightarrow{x}\right)}.$$

(45)

By substituting Expressions (41), (42), (44) and (45) into Expression (43), the optimization objective is simplified as

$$\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\hspace{1em}J\left[p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)\right]={\langle c,u\rangle }_U.$$

(46)

Here, $c\left(\overrightarrow{x}\right)\in V\left(\mathsf{\Omega }\right)$ is a combination of the deformation functions, and can be expressed as

$$c\left(\overrightarrow{x}\right)=\left(1-2\lambda \right)\left(u\left(\overrightarrow{x}\right)-v\left(\overrightarrow{x}\right)\right)+\left(1-\lambda \right)w\left(\overrightarrow{x}\right).$$

(47)

In order to improve the generalization of the data classifier, regularizers should be added to the optimization model. Here, ${\Vert p\Vert }_1$ and ${\Vert q\Vert }_1$ stand for the 1-norms of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$, respectively, and are used as regularizers to avoid increasing the order of the optimization model. Meanwhile, these regularizers are treated as two constraints by directly setting the values of ${\Vert p\Vert }_1$ and ${\Vert q\Vert }_1$. Due to $p\left(\overrightarrow{x}\right)>0$ and $q\left(\overrightarrow{x}\right)>0$, ${\Vert p\Vert }_1$ and ${\Vert q\Vert }_1$ can be simply written as

$${\Vert p\left(\overrightarrow{x}\right)\Vert }_1={\displaystyle {\int }_{\mathsf{\Omega }}p\left(\overrightarrow{x}\right)d\mathsf{\Omega }\hspace{0.17em}},\hspace{1em}{\Vert q\left(\overrightarrow{x}\right)\Vert }_1={\displaystyle {\int }_{\mathsf{\Omega }}q\left(\overrightarrow{x}\right)d\mathsf{\Omega }}.$$

(48)

It should be noted that objective (46) is built by taking the mean $f\left(\overrightarrow{x}\right)$ of $D_1$ as the reference feature and selecting the deformation $u\left(\overrightarrow{x}\right)$ as the reference feature coordinate axis. If other deformation functions $\alpha \left(\overrightarrow{x}\right)$ are selected as the reference feature coordinate axis, the results are similar. For example, $\alpha \left(\overrightarrow{x}\right)$ can be set as $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$, $u\left(\overrightarrow{x}\right)-v\left(\overrightarrow{x}\right)$, or others. Through setting $\alpha \left(\overrightarrow{x}\right)$ as the reference feature coordinate axis, the optimization model can be summarized as

$$\begin{array}{l}\underset{p\left(\overrightarrow{x}\right),q\left(\overrightarrow{x}\right)}{\min }\hspace{1em}J\left[p,q\right]={\langle c,\alpha \rangle }_U\\ \hspace{1em}s.t.\\ \hspace{1em}\hspace{1em}{\langle u,\delta u\rangle }_U=\langle f,\delta u\rangle ,\hspace{1em}{\langle v,\delta v\rangle }_U=\langle g,\delta v\rangle ,\hspace{1em}{\langle w,\delta w\rangle }_U=\langle h,\delta w\rangle \\ \hspace{1em}\hspace{1em}{\langle u,v\rangle }_U\le 0\\ \hspace{1em}\hspace{1em}{\Vert p\Vert }_1=Tolp,\hspace{1em}{\Vert q\Vert }_1=Tolq\\ \hspace{1em}\hspace{1em}{p}_{\min }\le p\left(\overrightarrow{x}\right)\hspace{1em}{q}_{\min }\le q\left(\overrightarrow{x}\right)\end{array}.$$

(49)

Here, $\delta u$, $\delta v$ and $\delta w$ are arbitrary continuous functions on $\mathsf{\Omega }$; $p_{\min }>0$ and $q_{\min }>0$, are lower bounds of $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$; $Tolp$ and $Tolq$ are two constants; $f\left(\overrightarrow{x}\right)$, $g\left(\overrightarrow{x}\right)$, $h\left(\overrightarrow{x}\right)$ and $c\left(\overrightarrow{x}\right)$ are given in Expressions (27), (45) and (47). $S_1$ and $S_0$ should be determined according to the reference feature coordinate axis, and can be rewritten as

$$\begin{array}{l}{S}_1={\left\{X^{\left(i\right)}\left(\overrightarrow{x}\right)|X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_1,\langle X^{\left(i\right)},\alpha \rangle <\langle f,\alpha \rangle \right\}}_{i=0}^{M_1}\\ S_0={\left\{X^{\left(i\right)}\left(\overrightarrow{x}\right)|X^{\left(i\right)}\left(\overrightarrow{x}\right)\in D_0,\langle X^{\left(i\right)},\alpha \rangle >\langle g,\alpha \rangle \right\}}_{i=0}^{M_0}\end{array}.$$

(50)

5. Mutual-Energy Inner Product Feature Coordinate Optimization Algorithm

The EFM is used to solve the differential Equation (1) to realize the mapping from $f\left(\overrightarrow{x}\right)$, $g\left(\overrightarrow{x}\right)$, and $h\left(\overrightarrow{x}\right)$ to $u\left(\overrightarrow{x}\right)$, $v\left(\overrightarrow{x}\right)$, and $w\left(\overrightarrow{x}\right)$ in the optimization model (49). We divide the domain $\mathsf{\Omega }$ into $Ne$ elements ${\mathsf{\Omega }}_s^{\left(e\right)}$ $\left(e=1,2\dots Ne\right)$, and assume the $e-th$ element ${\mathsf{\Omega }}_s^{\left(e\right)}$ has $Nd$ nodes. For the $i-th$ $\left(i=1,2\dots Nd\right)$ node in ${\mathsf{\Omega }}_s^{\left(e\right)}$, its global coordinate in $\mathsf{\Omega }$, deformation value $u\left({\overrightarrow{x}}_i^{\left(e\right)}\right)$, and interpolation basis function are denoted as ${\overrightarrow{x}}_i^{\left(e\right)}$, $u_i^{\left(e\right)}$, $N_i\left(\overrightarrow{\xi }\right)$, respectively, where $\overrightarrow{\xi }\in R^n$, is the local coordinate of the element ${\mathsf{\Omega }}_s^{\left(e\right)}$. In this way, for an element, its global and local coordinate relationship ${\overrightarrow{x}}^{\left(e\right)}\left(\overrightarrow{\xi }\right)$ and the element deformation function $u^{\left(e\right)}\left(\overrightarrow{\xi }\right)$ can be expressed as [37]

$${\overrightarrow{x}}^{\left(e\right)}\left(\overrightarrow{\xi }\right)={\displaystyle \sum _{j=1}^{Nd}{N}_j}\left(\overrightarrow{\xi }\right){\overrightarrow{x}}_j^{\left(e\right)},\hspace{1em}{u}^{\left(e\right)}\left(\overrightarrow{\xi }\right)={\displaystyle \sum _{j=1}^{Nd}{N}_j}\left(\overrightarrow{\xi }\right)u_j^{\left(e\right)}.$$

(51)

It is assumed that $\overrightarrow{N}$ is an $Nd$-dimensional row vector with the $j-th$ component $N_j\left(\overrightarrow{\xi }\right)$; $L$ is an $n\times Nd$ matrix with the entry $L_{ij}=\partial N_j\left(\overrightarrow{\xi }\right)/\partial {\xi }_i$, where ${\xi }_i$ is the $i-th$ component of the local coordinate $\overrightarrow{\xi }$; and $X$ is an $Nd\times n$ matrix with the entry $X_{ij}=x_{ij}^{\left(e\right)}$, where $x_{ij}^{\left(e\right)}$ is the $j-th$ component of the element node coordinates ${\overrightarrow{x}}_i^{\left(e\right)}$. Applying Expression (51), the $n\times n$ Jacobi matrix $J$ for the transformation between the global and local coordinates, the deformation function $u^{\left(e\right)}\left(\overrightarrow{x}\right)$ and its $n$-dimensional gradient vector $\nabla u^{\left(e\right)}\left(\overrightarrow{x}\right)={\left[\begin{array}{cccc}\partial u^{\left(e\right)}/\partial x_1& \partial u^{\left(e\right)}/\partial x_2& \dots & \partial u^{\left(e\right)}/\partial x_n\end{array}\right]}^T$, can be expressed in the concise and compact form

$$u^{\left(e\right)}\left(\overrightarrow{x}\right)=\overrightarrow{N}·{\overrightarrow{u}}^{\left(e\right)},\hspace{1em}\hspace{0.17em}\nabla u^{\left(e\right)}\left(\overrightarrow{x}\right)=B·{\overrightarrow{u}}^{\left(e\right)},\hspace{1em}\hspace{0.17em}J=L·X,\hspace{0.17em}\hspace{1em}\hspace{0.17em}B=J^{-1}L,$$

(52)

where ${\overrightarrow{u}}^{\left(e\right)}={\left[\begin{array}{cccc}{u}_1^{\left(e\right)}& u_2^{\left(e\right)}& \dots & u_{Nd}^{\left(e\right)}\end{array}\right]}^T$ is a vector with the component $u_i^{\left(e\right)}$, which is the deformation value of the $i-th$ node in the $e-th$ element, and $B$ is an $n\times Nd$ matrix. In the optimization model (49), the design variables are $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$. We assume $p\left(\overrightarrow{x}\right)$ and $q\left(\overrightarrow{x}\right)$ in each element are constants $p_e$ and $q_e$. So, the design variables can be expressed as $\overrightarrow{p}={\left[\begin{array}{cccc}{p}_1& p_2& \dots & p_{Ne}\end{array}\right]}^T$ and $\overrightarrow{q}={\left[\begin{array}{cccc}{q}_1& q_2& \dots & q_{Ne}\end{array}\right]}^T$ in $\mathsf{\Omega }$.

Substituting Expression (52) into the mutual-energy expressions (5) and (9) yields

$${\langle u,u\rangle }_U={\displaystyle \sum _{e=1}^{Ne}{\overrightarrow{u}}^{\left(e\right)T}{K}_s^{\left(e\right)}{\overrightarrow{u}}^{\left(e\right)}},\hspace{1em}\hspace{1em}\hspace{0.17em}\langle f,u\rangle ={\displaystyle \sum _{e=1}^{Ne}{\overrightarrow{u}}^{\left(e\right)T}{\overrightarrow{f}}^{\left(e\right)}}.$$

(53)

Here, $K_s^{\left(e\right)}$ is an $Nd\times Nd$ element stiffness matrix, which is a positive semidefinite symmetric matrix and can be expressed as

$$K_s^{\left(e\right)}=p_e{K}_p^{\left(e\right)}+q_e{K}_q^{\left(e\right)}+K_{\sigma }^{\left(e\right)}.$$

(54)

In Expression (54), $K_s^{\left(e\right)}$ is a linear function of $p_e$ and $q_e$; $K_p^{\left(e\right)}$ and $K_q^{\left(e\right)}$ are corresponding coefficient matrices; and $K_{\sigma }^{\left(e\right)}$ is the contribution of the boundary constraint to the element stiffness matrix. If the element boundary does not overlap with the design domain boundary, then $K_{\sigma }^{\left(e\right)}=0$. Here, $K_p^{\left(e\right)}$, $K_q^{\left(e\right)}$, $K_{\sigma }^{\left(e\right)}$ can be calculated by

$$K_p^{\left(e\right)}={\displaystyle {\int }_{{\mathsf{\Omega }}_s^{\left(e\right)}}{B}^{T}Bd\mathsf{\Omega }\hspace{0.17em}},\hspace{1em}\hspace{0.17em}{K}_q^{\left(e\right)}={\displaystyle {\int }_{{\mathsf{\Omega }}_s^{\left(e\right)}}{\overrightarrow{N}}^T\overrightarrow{N}d\mathsf{\Omega }}\hspace{0.17em},\hspace{1em}\hspace{1em}{K}_{\sigma }^{\left(e\right)}={\displaystyle {\int }_{{\mathsf{\Gamma }}_s^{\left(e\right)}}\sigma \left(\overrightarrow{x}\right){\overrightarrow{N}}^T\overrightarrow{N}d\mathsf{\Omega }}.$$

(55)

In Expression (53), ${\overrightarrow{f}}^{\left(e\right)}$ is the equivalent node input vector, resulting from the equivalent action between the force $f\left(\overrightarrow{x}\right)$ on the element and the force ${\overrightarrow{f}}^{\left(e\right)}$ on the node, and satisfies

$${\overrightarrow{f}}^{\left(e\right)}={\displaystyle {\int }_{{\mathsf{\Omega }}_s^{\left(e\right)}}f\left(\overrightarrow{x}\right){\overrightarrow{N}}^{T}d\mathsf{\Omega }}.$$

(56)

It is assumed that the design domain $\mathsf{\Omega }$ comprises $M$ element nodes. We number these nodes globally, and use two $M$-dimension vectors $\overrightarrow{u}$ and $\overrightarrow{f}$ to denote the values of $u\left(\overrightarrow{x}\right)$ and $f\left(\overrightarrow{x}\right)$ at all the nodes. The components of $\overrightarrow{u}$ and $\overrightarrow{f}$ are $u_i$ and $f_i$, where the subscript $i$ is the global node number. The component $f_i$ can be calculated through Expression (56). Expression (56) is calculated for each element adjacent to the $i-th$ global node, and $f_i$ is the superposition of the element node corresponding to the $i-th$ global node.

Based on the relationship between the local and global node numbers, Expression (53) can be rewritten as

$${\langle u,u\rangle }_U={\overrightarrow{u}}^{T}K\overrightarrow{u}={\displaystyle \sum _{e=1}^{Ne}{\overrightarrow{u}}^{\left(e\right)T}{K}_s^{\left(e\right)}{\overrightarrow{u}}^{\left(e\right)}},\hspace{1em}\hspace{1em}\hspace{0.17em}\langle f,u\rangle ={\overrightarrow{u}}^T\overrightarrow{f}={\displaystyle \sum _{e=1}^{Ne}{\overrightarrow{u}}^{\left(e\right)T}{\overrightarrow{f}}^{\left(e\right)}},$$

(57)

where $K$ is the global stiffness matrix, an $M\times M$ positive definite symmetric matrix. Substituting Expression (57) into Expression (12) yields

$$\underset{\overrightarrow{u}}{\min }\hspace{1em}\pi \left[\overrightarrow{u}\right]={\displaystyle \frac{1}{2}}{\overrightarrow{u}}^{T}K\overrightarrow{u}-{\overrightarrow{u}}^T\overrightarrow{f}.$$

(58)

Based on Expression (58), the solution of the differential Equation (1) satisfies

$$K\overrightarrow{u}-\overrightarrow{f}=0.$$

(59)

Similarly, assume that the input of Expression (1) is $g\left(\overrightarrow{x}\right)$ and the corresponding solution is $v\left(\overrightarrow{x}\right)$; $\overrightarrow{v}$ is the global node vector corresponding to $v\left(\overrightarrow{x}\right)$ on $\mathsf{\Omega }$, and ${\overrightarrow{v}}^{\left(e\right)}$ is the element node vector corresponding to $v\left(\overrightarrow{x}\right)$ on ${\mathsf{\Omega }}_s^{\left(e\right)}$; and $\overrightarrow{g}$ is the equivalent node input vector corresponding to $g\left(\overrightarrow{x}\right)$. We have

$$K\overrightarrow{v}-\overrightarrow{g}=0.$$

(60)

Similarly to the derivation of Expression (57), through using Expressions (59) and (60), the mutual-energy expression of $u\left(\overrightarrow{x}\right)$ and $v\left(\overrightarrow{x}\right)$ can be derived

$${\langle u,v\rangle }_U={\overrightarrow{u}}^{T}K\overrightarrow{v}={\displaystyle \sum _{e=1}^{Ne}{\overrightarrow{u}}^{\left(e\right)T}{K}_s^{\left(e\right)}{\overrightarrow{v}}^{\left(e\right)}},\hspace{1em}\hspace{1em}\hspace{0.17em}{\langle u,v\rangle }_U={\overrightarrow{u}}^T\overrightarrow{f}={\overrightarrow{v}}^T\overrightarrow{g}.$$

(61)

In Expression (61), the first equation is used for model optimization, and the second equation is used for data classifier training and prediction, avoiding the need to solve for the Expressions (59) and (60).

After discretizing the design domain by finite elements, the differential Equation (1) is converted into a system of linear equations, and the mutual-energy definition (5) can be expressed by the matrix and vector product. In this way, the optimization model (49) can be rewritten in the vector form

$$\begin{array}{l}\underset{\overrightarrow{p},\overrightarrow{q}}{\min }\hspace{1em}J\left[\overrightarrow{p},\overleftarrow{q}\right]={\overrightarrow{c}}^{T}K\overrightarrow{\alpha }\\ \hspace{1em}s.t.\\ \hspace{1em}\hspace{1em}K\overrightarrow{u}=\overrightarrow{f},\hspace{1em}\hspace{1em}K\overrightarrow{v}=\overrightarrow{g},\hspace{1em}\hspace{1em}K\overrightarrow{w}=\overrightarrow{h}\\ \hspace{1em}\hspace{1em}G\left[\overrightarrow{p},\overleftarrow{q}\right]={\overrightarrow{u}}^{T}K\overrightarrow{v}\le 0\\ \hspace{1em}\hspace{1em}{\displaystyle \sum _{e=1}^{Ne}{p}_e}=Tolp,\hspace{1em}{\displaystyle \sum _{e=1}^{Ne}{q}_e}=Tolq\\ \hspace{1em}\hspace{1em}{p}_{\min }\le p_e,\hspace{1em}{q}_{\min }\le q_e,\hspace{1em}e=1,2\dots Ne\end{array}.$$

(62)

Here, $\overrightarrow{\alpha }$ is the finite element node vector corresponding to the selected reference feature coordinate, and can be the statistical features of the sample sets or their combination; for example,

$$\overrightarrow{\alpha }=\overrightarrow{u}\hspace{1em}or\hspace{1em}\overrightarrow{\alpha }=\overrightarrow{v}\hspace{1em}or\hspace{1em}\overrightarrow{\alpha }=\overrightarrow{u}-\overrightarrow{v}.$$

(63)

Meanwhile, $\overrightarrow{f}$, $\overrightarrow{g}$, and $\overrightarrow{h}$ are the finite element node vectors corresponding to the mean and deviation of the samples, and $\overrightarrow{c}$ is the temporary node vector generated by the mean and deviation. Expression (47) can be rewritten as

$$\overrightarrow{c}=\left(1-2\lambda \right)\left(\overrightarrow{u}-\overrightarrow{v}\right)+\left(1-\lambda \right)\overrightarrow{w}.$$

(64)

The significant advantage of the optimization model (62) is that $K$ is a positive definite symmetric matrix and is linear with respect to the design variables $\overrightarrow{p}$ and $\overrightarrow{q}$, and meanwhile, the coefficient matrices corresponding to the components of the design variables are positive semidefinite matrices, convenient for the algorithm design. Intermediate variables $\overrightarrow{u}$, $\overrightarrow{v}$, $\overrightarrow{w}$ are functions of the design variables and can be calculated by using the linear equations, and the optimization model (62) can be solved by the sequential linearization algorithm. The objective $J\left[\overrightarrow{p},\overleftarrow{q}\right]$ and the constraint $G\left[\overrightarrow{p},\overleftarrow{q}\right]$ are nonlinear, and their derivatives with respect to the design variables need to be calculated. The derivative of $G\left[\overrightarrow{p},\overleftarrow{q}\right]$ with respect to $p_e$ is

$${\displaystyle \frac{\partial G}{\partial p_e}}={\displaystyle \frac{\partial {\overrightarrow{u}}^T}{\partial p_e}}K\overrightarrow{v}+{\overrightarrow{u}}^T{\displaystyle \frac{\partial K}{\partial p_e}}\overrightarrow{v}+{\overrightarrow{u}}^{T}K{\displaystyle \frac{\partial \overrightarrow{v}}{\partial p_e}}\hspace{1em}\hspace{1em}e=1,2\dots Ne,$$

(65)

where ${\displaystyle \frac{\partial \overrightarrow{u}}{\partial p_e}}$ and ${\displaystyle \frac{\partial \overrightarrow{v}}{\partial p_e}}$ are determined by taking the derivative of $K\overrightarrow{u}=\overrightarrow{f}$ and $K\overrightarrow{v}=\overrightarrow{g}$ with respect to $p_e$

$${\displaystyle \frac{\partial K}{\partial p_e}}\overrightarrow{u}+K{\displaystyle \frac{\partial \overrightarrow{u}}{\partial p_e}}=0,\hspace{1em}\hspace{0.17em}{\displaystyle \frac{\partial K}{\partial p_e}}\overrightarrow{v}+K{\displaystyle \frac{\partial \overrightarrow{v}}{\partial p_e}}=0.$$

(66)

Substituting Expression (66) into Expression (44) yields

$${\displaystyle \frac{\partial G}{\partial p_e}}=-{\overrightarrow{u}}^T{\displaystyle \frac{\partial K}{\partial p_e}}\overrightarrow{v}.$$

(67)

Substituting Expression (54) into Expression (67) yields $\partial G/\partial p_e$. Similarly, $\partial G/\partial q_e$ can also be computed

$${\displaystyle \frac{\partial G}{\partial p_e}}=-{\overrightarrow{u}}^{\left(e\right)T}{K}_p^{\left(e\right)}{\overrightarrow{v}}^{\left(e\right)},\hspace{1em}{\displaystyle \frac{\partial G}{\partial q_e}}=-{\overrightarrow{u}}^{\left(e\right)T}{K}_q^{\left(e\right)}{\overrightarrow{v}}^{\left(e\right)}.$$

(68)

The Expressions (63) and (64) show that $\overrightarrow{c}$ and $\overrightarrow{\alpha }$ are linear combinations of $\overrightarrow{u}$, $\overrightarrow{v}$, and $\overrightarrow{w}$. According to the superposition principle, $\partial \overrightarrow{c}/\partial p_e$ and $\partial \overrightarrow{\alpha }/\partial p_e$ also satisfy equations similar to Expression (66), and have exactly the same derivation as Expression (68). So, we obtain

$${\displaystyle \frac{\partial J}{\partial p_e}}=-{\overrightarrow{c}}^{\left(e\right)T}{K}_p^{\left(e\right)}{\overrightarrow{\alpha }}^{\left(e\right)},\hspace{0.17em}\hspace{1em}{\displaystyle \frac{\partial J}{\partial q_e}}=-{\overrightarrow{c}}^{\left(e\right)T}{K}_q^{\left(e\right)}{\overrightarrow{\alpha }}^{\left(e\right)}.$$

(69)

Optimization Algorithm 1: Mutual-energy inner product feature coordinate optimization algorithm

Based on Expressions (68) and (69), the optimization model (62) can be solved by the sequential linearization algorithm. The algorithm steps are summarized as follows:

(1)

Use vectors to represent the sample data

Convert the sample data $X^{\left(i\right)}\left(\overrightarrow{x}\right)$ in the training subsets $D_1$ and $D_0$ into the finite element node vectors ${\overrightarrow{X}}^{\left(i\right)}\in R^M$. Based on Expression (70), first calculate the element node vectors ${\overrightarrow{X}}^{\left(i\right)\left(e\right)}\in R^{Nd}$, and then use them to assemble the global node vector ${\overrightarrow{X}}^{\left(i\right)}$.

$${\overrightarrow{X}}^{\left(i\right)\left(e\right)}={\displaystyle {\int }_{{\mathsf{\Omega }}_s^{\left(e\right)}}{X}^{\left(i\right)}\left(\overrightarrow{x}\right){\overrightarrow{N}}^{T}d\mathsf{\Omega }}\hspace{1em}\hspace{1em}e=1,2\dots Ne.$$

(70)

(2)

Set the optimization constants and initial values of the design variables

①

Set the optimization constants

Set $\lambda$, the weight of the mean and deviation, with the requirement $\lambda \in \left[0,1\right]$; set the total amount $Tolp$, $Tolq$ and the lower bounds $p_{\min }$, $q_{\min }$ of the design variables; set the moving limit $\mathsf{\Delta }{x}_{\max }$ of the design variables for the linear programming; set the design variable minimum increment ${\epsilon }_x$ and the objective function minimum increment ${\epsilon }_J$, which are used to determine if the optimization ends or not.

②

Set the initial values of the design variables

Set $p_e=p_e^{\left(0\right)}$, $q_e=q_e^{\left(0\right)}$ $\left(e=1,2\dots Ne\right)$. Generally, set $p_e^{\left(0\right)}=Tolp/M$, $q_e^{\left(0\right)}=Tolq/M$.

(3)

Calculate the current value of the objective function

①

Calculate the element stiffness matrices and assemble the global stiffness matrix

Based on Expressions (54) and (55), calculate the element stiffness matrices $K_s^{\left(e\right)}\hspace{0.17em}\left(e=1,2\dots Ne\right)$. The element stiffness matrix is linear with respect to $p_e$ and $q_e$, and the coefficient matrices are determined only by the element interpolation basis functions, so the calculation can be performed prior to the optimization to speed up the optimization process. Then, assemble the global stiffness matrix according to the node numbers. Since $K$ is a positive definite symmetric matrix, through performing Cholesky decomposition on it, we can have $K=L·L^T$, where $L$ is a lower triangular matrix.

②

Compute the mean vectors $\overrightarrow{u}$ and $\overrightarrow{v}$, and select the reference feature coordinate axis $\overrightarrow{\alpha }$

$$\{\begin{array}{c}\overrightarrow{u}=L^{-T}\left(L^{-1}\overrightarrow{f}\right)\\ \overrightarrow{f}={\displaystyle \frac{1}{N_1}}{\displaystyle \sum _{{\overrightarrow{X}}^{\left(i\right)}\in D_1}{\overrightarrow{X}}^{\left(i\right)}}\end{array}\hspace{0.17em}\hspace{1em},\hspace{1em}\{\begin{array}{c}\overrightarrow{v}=L^{-T}\left(L^{-1}\overrightarrow{g}\right)\\ \overrightarrow{g}={\displaystyle \frac{1}{N_0}}{\displaystyle \sum _{{\overrightarrow{X}}^{\left(i\right)}\in D_0}{\overrightarrow{X}}^{\left(i\right)}}\end{array},$$

(71)

where $\overrightarrow{f}$ and $\overrightarrow{g}$ represent the means of the sample data in $D_1$ and $D_0$; $N_1$ and $N_0$ are the sample numbers in $D_1$ and $D_0$; $\overrightarrow{\alpha }$ can be selected and calculated by Expression (63).

③

Compute the deviation vector $\overrightarrow{w}$ and the intermediate vector $\overrightarrow{c}$

$$\{\begin{array}{c}\overrightarrow{w}=L^{-T}\left(L^{-1}\overrightarrow{h}\right)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\\ \overrightarrow{h}={\displaystyle \frac{1}{M_0}}{\displaystyle \sum _{{\overrightarrow{X}}^{\left(i\right)}\in S_0}{\overrightarrow{X}}^{\left(i\right)}}-{\displaystyle \frac{1}{M_1}}{\displaystyle \sum _{{\overrightarrow{X}}^{\left(i\right)}\in S_1}{\overrightarrow{X}}^{\left(i\right)}}\end{array}\hspace{1em},\hspace{1em}\{\begin{array}{c}{S}_1={\left\{{\overrightarrow{X}}^{\left(i\right)}|{\overrightarrow{X}}^{\left(i\right)}\in D_1,\hspace{0.17em}{\overrightarrow{\alpha }}^T{\overrightarrow{X}}^{\left(i\right)}<{\mu }_1\right\}}_{i=1}^{M_1}\\ S_0={\left\{{\overrightarrow{X}}^{\left(i\right)}|{\overrightarrow{X}}^{\left(i\right)}\in D_0,\hspace{0.17em}{\overrightarrow{\alpha }}^T{\overrightarrow{X}}^{\left(i\right)}>{\mu }_0\right\}}_{i=1}^{M_0}\end{array},$$

(72)

where $\overrightarrow{h}$ is the deviation of the sample data and only the sample data in ${\mathrm{S}}_1$ and ${\mathrm{S}}_0$ are calculated. ${\mu }_1={\overrightarrow{\alpha }}^T\overrightarrow{f}$ and ${\mu }_0={\overrightarrow{\alpha }}^T\overrightarrow{g}$ represent the projections of the means of the sample data in $D_1$ and $D_0$ on $\overrightarrow{\alpha }$. After $\overrightarrow{u}$, $\overrightarrow{v}$, $\overrightarrow{w}$ are obtained, $\overrightarrow{c}$ can be obtained by Expression (64).

④

Calculate the current values of the objective function and the constraint

Based on the optimization model (62), the current values of $J\left[\overrightarrow{p},\overleftarrow{q}\right]$ and $G\left[\overrightarrow{p},\overleftarrow{q}\right]$ can be calculated by

$$J_0={\overrightarrow{c}}^{T}K\overrightarrow{\alpha }\hspace{1em},\hspace{1em}\hspace{1em}{G}_0={\overrightarrow{u}}^{T}K\overrightarrow{v}.$$

(73)

(4)

Calculate the gradient vectors of the objective function and the constraint

Apply Expressions (68) and (69) to calculate $\partial J/\partial p_e$, $\partial J/\partial q_e$, $\partial G/\partial p_e$ and $\partial G/\partial q_e$. Then, express them as the compact gradient vectors ${\nabla }_{\overrightarrow{p}}J$, ${\nabla }_{\overrightarrow{q}}J$, ${\nabla }_{\overrightarrow{p}}G$ and ${\nabla }_{\overrightarrow{q}}G$. Here, ${\nabla }_{\overrightarrow{p}}J$ is defined as ${\nabla }_{\overrightarrow{p}}J={\left[\begin{array}{cccc}\partial J/\partial p_1& \partial J/\partial p_2& \dots & \partial J/\partial p_M\end{array}\right]}^T$ and the other gradient vector definitions are similar. In Expressions (68) and (69), $K_p^{\left(e\right)}$ and $K_q^{\left(e\right)}$ are only determined by the element interpolation basis functions and are constant matrices independent of the design variables. So, $K_p^{\left(e\right)}$ and $K_q^{\left(e\right)}$ can be calculated prior to the optimization, and the gradient vectors of $J\left[\overrightarrow{p},\overleftarrow{q}\right]$ and $G\left[\overrightarrow{p},\overleftarrow{q}\right]$ can be achieved through the mapping relationship between the local and global node numbers.

(5)

Obtain increments of the design variables by solving the sequential linearization optimization model

①

Construct the sequential linearization optimization model

$$\begin{array}{l}\underset{{\overrightarrow{x}}_p,{\overrightarrow{x}}_q}{\min }\hspace{1em}J\left[{\overrightarrow{x}}_p,{\overrightarrow{x}}_q\right]=J_0+{\left({\nabla }_{\overrightarrow{p}}J\right)}^T{\overrightarrow{x}}_p+{\left({\nabla }_{\overrightarrow{q}}J\right)}^T{\overrightarrow{x}}_q\\ \hspace{1em}s.t.\\ \hspace{1em}\hspace{1em}G\left[{\overrightarrow{x}}_p,{\overrightarrow{x}}_q\right]=G_0+{\left({\nabla }_{\overrightarrow{p}}G\right)}^T{\overrightarrow{x}}_p+{\left({\nabla }_{\overrightarrow{q}}G\right)}^T{\overrightarrow{x}}_q\le 0\\ \hspace{1em}\hspace{1em}{\displaystyle \sum _{i=1}^{Ne}{x}_{p,i}}=Tol{x}_p,\hspace{1em}\hspace{1em}\hspace{1em}{\displaystyle \sum _{e=1}^{Ne}{x}_{q,i}}=Tol{x}_q\\ \hspace{1em}\hspace{1em}{x}_{p\min ,i}\le x_{p,i}\le \mathsf{\Delta }{x}_{\max },\hspace{1em}{x}_{q\min ,i}\le x_{q,i}\le \mathsf{\Delta }{x}_{\max }\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}i=1,2\dots Ne\end{array},$$

(74)

where the design variables $\overrightarrow{p}\in R^M$, $\overrightarrow{q}\in R^M$; ${\overrightarrow{x}}_p$ and ${\overrightarrow{x}}_q$ are increments of the design variables, and their $i-th$ components are $x_{p,i}$ and $x_{q,i}$; $Tol{x}_p$, $Tol{x}_q$, and $x_{p\min ,i}$, $x_{q\min ,i}$ can be calculated by

$$\{\begin{array}{c}Tol{x}_p=Tolp-{\displaystyle \sum _{i=1}^{Ne}{p}_i}\\ Tol{x}_q=Tolq-{\displaystyle \sum _{i=1}^{Ne}{q}_i}\end{array}\hspace{1em},\hspace{1em}\{\begin{array}{c}{x}_{p\min ,i}=\max \left(p_{\min }-p_i,-\mathsf{\Delta }{x}_{\max }\right)\\ x_{q\min ,i}=\max \left(q_{\min }-q_i,-\mathsf{\Delta }{x}_{\max }\right)\end{array}.$$

②

Solve the sequential linearization optimization model (74) to obtain ${\overrightarrow{x}}_p$ and ${\overrightarrow{x}}_q$

When solving Expression (74), slack variables are added to $G\left[{\overrightarrow{x}}_p,{\overrightarrow{x}}_q\right]\le 0$ to facilitate the initial feasible solution construction.

(6)

Determine whether to end the optimization iteration

①

Store the design variables, the objective function, and the constraint function of the previous step of the sequential linearization optimization.

Store the design variables ${\overrightarrow{p}}^{old}=\overrightarrow{p}$, ${\overrightarrow{q}}^{old}=\overrightarrow{q}$, the objective function value $J_0^{old}=J_0$, and the constraint function value $G_0^{old}=G_0$.

②

Update the design variables and the objective function value.

Let $\overrightarrow{p}=\overrightarrow{p}+{\overrightarrow{x}}_p$, $\overrightarrow{q}=\overrightarrow{q}+{\overrightarrow{x}}_q$, then execute step (3) to update the objective function value $J_0$.

③

Determine whether to end the iteration.

If $\left|J_0-J_0^{old}\right|\le {\epsilon }_J$ or $\max \left[{\Vert {\overrightarrow{x}}_p\Vert }_{\infty },{\Vert {\overrightarrow{x}}_p\Vert }_{\infty }\right]\le {\epsilon }_x$, then end the iteration. Otherwise, if $J_0<J_0^{old}$, go to step (4) to continue the iteration; if $J_0\ge J_0^{old}$, reduce the moving limits of the design variables by letting $\mathsf{\Delta }{x}_{\max }=\gamma ·\mathsf{\Delta }{x}_{\max }$; here, $\gamma =0.5\sim 0.85$, then go to step (5) to iteratively calculate the design variable increments ${\overrightarrow{x}}_p$ and ${\overrightarrow{x}}_q$.

6. Algorithm Implementation and Image Classifier

Image classification is used to determine if an image has certain given features and can be realized by algorithms for extracting the feature information of the image. Applying the mutual-energy inner product to extract the image features has the advantage of enhancing the feature information and suppressing other high-frequency noise. If we select multiple features of an image, we can design multiple mutual-energy inner products, and each mutual-energy inner product can be regarded as one feature coordinate of the image. Using multiple mutual-energy inner products to characterize an image is equivalent to using multiple feature coordinates to describe the image, or equivalent to representing the high-dimensional image in a low-dimensional space, reducing the dimensionality of image data.

This part will discuss the implementation of Optimization Algorithm 1 and its application in 2-D grayscale image classification. Assume that each sample $X^{\left(i\right)}\left(\overrightarrow{x}\right)$ in the training datasets $D_1$ and $D_0$ is a 2-D grayscale image; the domain $\mathsf{\Omega }$ occupied by the image is rectangular; each image is expressed by $n_1\times n_2$ pixels; and each pixel is a square with a side length of 1. In this case, $\overrightarrow{x}\in \mathsf{\Omega }\in R^2$ and $\mathsf{\Omega }=\left\{\left(x_1,x_2\right)|0\le x_1\le n_1,\hspace{0.17em}0\le x_2\le n_2\right\}$.

6.1. Vectorized Implementation of Optimization Algorithm 1

While using FEM to discretize the design domain, we regard each pixel as a finite element and divide the domain $\mathsf{\Omega }$ into $n_1\times n_2$ quadrilateral elements ${\mathsf{\Omega }}_s^{\left(e\right)}$, i.e., $e=1,\hspace{0.17em}2\dots Ne$ and $Ne=n_1\times n_2$. In $\mathsf{\Omega }$, the global element numbering uses column priority, where the upper left corner element is numbered 1 and the lower right corner element is numbered $Ne$. A planar quadrilateral element is used to interpolate the deformation functions. Each element has four nodes, so the total number of nodes is $M=\left(n_1+1\right)\times \left(n_2+1\right)$, and the total number of boundary nodes is $n_{\mathsf{\Gamma }}=2\times \left(n_1+n_2\right)$. The global node numbering also uses column priority, where the upper left corner node is numbered 1 and the lower right corner node is numbered $M$. The interpolation basis functions of the quadrilateral element are

$$N_i\left(\overrightarrow{\xi }\right)={\displaystyle \frac{1}{4}}\left(1+{\xi }_1^{\left(i\right)}{\xi }_1\right)\left(1+{\xi }_2^{\left(i\right)}{\xi }_2\right)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}i=1,\hspace{0.17em}2,\hspace{0.17em}3,\hspace{0.17em}4,$$

(75)

where the domain of the definition is square and is expressed as ${\mathsf{\Omega }}_{\xi }=\left\{\left({\xi }_1,{\xi }_2\right)|-1\le {\xi }_1\le 1,\hspace{0.17em}-1\le {\xi }_2\le 1\right\}$. The element nodes are four corner points of the quadrilateral. The node with the coordinate $\left(-1,-1\right)$ is numbered 1, in counter-clockwise order, and the other nodes with the coordinates $\left(+1,-1\right)$, $\left(+1,+1\right)$, $\left(-1,+1\right)$ are numbered 2, 3, and 4, respectively. The interpolation basis function $N_i\left(\overrightarrow{\xi }\right)$ corresponds to the $i-th$ node, where $\left({\xi }_1^{\left(i\right)},{\xi }_2^{\left(i\right)}\right)$ is the corresponding node coordinate. The mapping relationship between the element node numbers and the global node numbers can be described by an $Ne\times 4$ matrix $\mathsf{\Theta }$, and its $i-th$ row corresponds to the $i-th$ element. If ${\mathsf{\Theta }}_{i,j}$ denotes its entry at the $i-th$ row and the $j-th$ column, then ${\mathsf{\Theta }}_{i,1}$, ${\mathsf{\Theta }}_{i,2}$, ${\mathsf{\Theta }}_{i,3}$, ${\mathsf{\Theta }}_{i,4}$ are the global node numbers corresponding to the element node numbers 1, 2, 3 and 4 of the $i-th$ element. So, we have

$$\{\begin{array}{l}{\mathsf{\Theta }}_{i,1}=i+1+r\hspace{1em}\\ {\mathsf{\Theta }}_{i,2}={\mathsf{\Theta }}_{i,1}+n_1+1\end{array}\hspace{1em},\hspace{1em}\{\begin{array}{c}{\mathsf{\Theta }}_{i,3}={\mathsf{\Theta }}_{i,1}+n_1\\ {\mathsf{\Theta }}_{i,4}={\mathsf{\Theta }}_{i,1}-1\end{array},$$

(76)

where $r$ is a module when $i$ is divided by $n_1$. Since all the elements are same squares, the isoparametric transformation ${\overrightarrow{x}}^{\left(e\right)}\left(\overrightarrow{\xi }\right)$ in Expression (51) is actually a scaling transformation. Through substituting $N_i\left(\overrightarrow{\xi }\right)$ into Expressions (52) and (55), we can find that the coefficient matrices $K_p^{\left(e\right)}$ and $K_q^{\left(e\right)}$ are independent of the element node numbers. So, we use $K_p$ and $K_q$ to express $K_p^{\left(e\right)}$ and $K_q^{\left(e\right)}$, and calculate them directly by

$$K_p={\displaystyle \frac{1}{24}}\left[\begin{array}{cccc}4& -1& -2& -1\\ -1& 4& -1& -2\\ -2& -1& 4& -1\\ -1& -2& -1& 4\end{array}\right]\hspace{0.17em},\hspace{1em}{K}_q={\displaystyle \frac{1}{36}}\left[\begin{array}{cccc}4& 2& 1& 2\\ 2& 4& 2& 1\\ 1& 2& 4& 2\\ 2& 1& 2& 4\end{array}\right].$$

(77)

When a side of an element overlaps with the boundary of the domain $\mathsf{\Omega }$, the influence of the boundary conditions $l\left[u\left(\overrightarrow{x}\right)\right]=0$ in Expression (1) on $K_s^{\left(e\right)}$ should be considered, so a $4\times 4$ matrix $K_{\sigma }^{\left(e\right)}$ should be calculated. Assume that the $j-th$ side of the element overlaps with the boundary of $\mathsf{\Omega }$ and the entry in the $i-th$ row and $j-th$ column of $K_{\sigma }^{\left(e\right)}$ is $K_{\sigma i,j}^{\left(e\right)}$. Then, the non-zero entries in $K_{\sigma }^{\left(e\right)}$ can be calculated by

$$K_{\sigma j,j}^{\left(e\right)}=K_{\sigma \widehat{j},\widehat{j}}^{\left(e\right)}={\displaystyle \frac{2}{3}}{\sigma }_{j,\widehat{j}}^{\left(e\right)}\hspace{0.17em},\hspace{1em}{K}_{\sigma j,\widehat{j}}^{\left(e\right)}=K_{\sigma \widehat{j},j}^{\left(e\right)}={\displaystyle \frac{1}{3}}{\sigma }_{j,\widehat{j}}^{\left(e\right)}.$$

(78)

In Expression (78), the subscripts $j$ and $\widehat{j}$ stand for the starting and end points of the $j-th$ side of the element, where the starting point is the element node numbered $j$ and the end point is determined along the side in counterclockwise order; ${\sigma }_{j,\widehat{j}}^{\left(e\right)}$ is a constant, equal to the approximate value of $\sigma \left(\overrightarrow{x}\right)$ on the $j-th$ side. In this paper, we handle the influence of $l\left[u\left(\overrightarrow{x}\right)\right]=0$ on $K_s^{\left(e\right)}$ while assembling the global stiffness matrix. We just simply replace the subscripts $j$ and $\widehat{j}$ of $K_{\sigma }^{\left(e\right)}$ in Expression (78) with global node numbers, then directly use them to assemble the global stiffness matrix.

Because each element corresponds to a pixel, we can assume that its grayscale value is a constant $X_{gray}^{\left(i\right)\left(e\right)}$. In this way, a sample image $X^{\left(i\right)}\left(\overrightarrow{x}\right)$ can be expressed as ${\overrightarrow{X}}_{gray}^{\left(i\right)}={\left[X_{gray}^{(i)\left(1\right)}\hspace{0.17em}{X}_{gray}^{(i)\left(2\right)}\dots X_{gray}^{(i)\left(e\right)}\dots X_{gray}^{(i)\left(Ne\right)}\right]}^T$. Through substituting Expression (75) into Expression (70), the relationship between element node vectors and image grayscale values can be obtained

$${\overrightarrow{X}}^{\left(i\right)\left(e\right)}=X_{gray}^{\left(i\right)\left(e\right)}\overrightarrow{F},$$

(79)

where $\overrightarrow{F}=0.25\times {\left[\begin{array}{cccc}1& 1& 1& 1\end{array}\right]}^T$, which can be regarded as mapping coefficients from the image grayscale to the element node vector.

While using element stiffness matrices $K_s^{\left(e\right)}$ and element node vectors ${\overrightarrow{X}}^{\left(i\right)\left(e\right)}$ to assemble the global stiffness matrix $K$ and the global node vector ${\overrightarrow{X}}^{\left(i\right)}$, the functions for generating a sparse matrix in MATLAB R2020a or the Python 2.7 SciPy module can be used, and the input arguments include the row index vector, the column index vector, and the values of the non-zero entries. More importantly, these sparse matrix generation functions can sum the non-zero entries with the same indexes, which is consistent with the process of assembling $K$ and ${\overrightarrow{X}}^{\left(i\right)}$.

In order to convert the image grayscale vector ${\overrightarrow{X}}_{gray}^{\left(i\right)}$ to the global node vector ${\overrightarrow{X}}^{\left(i\right)}$, a $4\times Ne$ matrix $\overrightarrow{F}{\left({\overrightarrow{X}}_{gray}^{\left(i\right)}\right)}^{\mathrm{T}}$ should first be calculated, whose $e-th$ column corresponds to the element node vector ${\overrightarrow{X}}^{\left(i\right)\left(e\right)}$. Then, $\overrightarrow{F}{\left({\overrightarrow{X}}_{gray}^{\left(i\right)}\right)}^{\mathrm{T}}$ is converted to a $4Ne$-dimensional column vector ${\overrightarrow{V}}_X$ in column-major order. Obviously, if we divide the components of ${\overrightarrow{V}}_X$ into multiple groups in sequence and each group includes four components, then the $e-th$ group corresponds to the element node vector ${\overrightarrow{X}}^{\left(i\right)\left(e\right)}$. ${\overrightarrow{V}}_X$ can be calculated by

$${\overrightarrow{V}}_X=reshape\left(\overrightarrow{F}{\left({\overrightarrow{X}}_{gray}^{\left(i\right)}\right)}^{\mathrm{T}},\hspace{0.17em}4Ne,\hspace{0.17em}1\right),$$

(80)

where the function $reshape\left(A,m,n\right)$ can convert the dimension of the matrix $A$ into $m\times n$ while keeping the total number of the entries unchanged.

Through the mapping matrix $\mathsf{\Theta }$, the position indexes of components of ${\overrightarrow{V}}_X$ in the global node vector ${\overrightarrow{X}}^{\left(i\right)}$ can be obtained. We transpose the $Ne\times 4$ matrix $\mathsf{\Theta }$ to the $4\times Ne$ matrix ${\mathsf{\Theta }}^T$, whose $e-th$ column corresponds to global node numbers of the $e-th$ element, and then convert ${\mathsf{\Theta }}^T$ to a $4Ne$-dimensional column vector ${\overrightarrow{I}}_X$ in the column-major order. ${\overrightarrow{I}}_X$ can be figured out by

$${\overrightarrow{I}}_X=reshape\left({\mathsf{\Theta }}^T,4Ne,1\right).$$

(81)

${\overrightarrow{I}}_X$ is the row index vector for generating ${\overrightarrow{X}}^{\left(i\right)}$ by a sparse matrix generation function. Since ${\overrightarrow{X}}^{\left(i\right)}$ has only one column, we use ${\overrightarrow{J}}_X$ to denote a $4Ne$-dimensional column index vector and set all the components of ${\overrightarrow{J}}_X$ to 1. Through substituting ${\overrightarrow{V}}_X$, ${\overrightarrow{I}}_X$, ${\overrightarrow{J}}_X$ into the sparse matrix generation function, we can yield ${\overrightarrow{X}}^{\left(i\right)}$.

Similarly, the global stiffness matrix $K$ can be assembled by using the sparse matrix generation function. A vector ${\overrightarrow{V}}_K^{\left(pq\right)}$ related to $K_s^{\left(e\right)}$ should be first calculated by

$${\overrightarrow{V}}_K^{\left(pq\right)}=reshape\left(K_s^{\left(pq\right)},16Ne,1\right),\hspace{1em}{K}_s^{\left(pq\right)}=K_p\otimes {\overrightarrow{p}}^T+K_q\otimes {\overrightarrow{q}}^T,$$

(82)

where the operator $\otimes$ denotes the Kronecker product of the matrices; $K_s^{\left(pq\right)}$ is a $4\times 4Ne$ matrix; $\overrightarrow{p}$ and $\overrightarrow{q}$ are the design variables. If $K_s^{\left(pq\right)}$ is divided into multiple blocks from left to right and each block is a $4\times 4$ matrix, the $e-th$ block is the calculation result of the first two terms of $K_s^{\left(e\right)}$ in Expression (54), without including $K_{\sigma }^{\left(e\right)}$. Therefore, if ${\overrightarrow{V}}_K^{\left(pq\right)}$ are divided into multiple blocks in sequence and each block includes 16 components, the $e-th$ block will correspond to a 1-dimensional vector converted from the $e-th$ element stiffness matrix in the column priority. We set $\overrightarrow{1}={\left[\begin{array}{cccc}1& 1& 1& 1\end{array}\right]}^T$, and use ${\overrightarrow{I}}_K^{\left(pq\right)}$, ${\overrightarrow{J}}_K^{\left(pq\right)}$ to denote the row indexes and column indexes of the entries in the global stiffness matrix. Then, ${\overrightarrow{I}}_K^{\left(pq\right)}$, ${\overrightarrow{J}}_K^{\left(pq\right)}$ corresponding to the components of ${\overrightarrow{V}}_K^{\left(pq\right)}$ can be calculated by

$${\overrightarrow{I}}_K^{\left(pq\right)}=reshape\left({\mathsf{\Theta }}^T\otimes {\overrightarrow{1}}^T,16Ne,1\right),\hspace{1em}{\overrightarrow{J}}_K^{\left(pq\right)}=reshape\left({\mathsf{\Theta }}^T\otimes \overrightarrow{1},16Ne,1\right).$$

(83)

As mentioned above, the constraint on the design boundary $\mathsf{\Gamma }$ can generate additional stiffness $K_{\sigma }^{\left(e\right)}$ for the adjacent elements. If we regard an element side overlapping with $\mathsf{\Gamma }$ as a 2-node line element, then its stiffness matrix will be a $2\times 2$ matrix $K_{\mathsf{\Gamma }}^{\left(e\right)}$, which can be figured out by Expression (78). Similarly, these line element stiffness matrices can be assembled into the global stiffness matrix. While designing an image classifier based on the mutual-energy inner products, we set a fixed boundary for Expression (1), i.e., $u\left(\overrightarrow{x}\right)=0\hspace{0.17em}\hspace{0.17em}\left(\overrightarrow{x}\in \mathsf{\Gamma }\right)$. This boundary condition can be handled by adding a relatively large number ${\sigma }_0$ to the diagonal entries of $K$, where its diagonal entries correspond to the boundary node numbers. The sparse matrix generation function is used to implement this boundary condition. First, we set the dimension of the vector ${\overrightarrow{V}}_K^{\left({\sigma }_0\right)}$ as $n_{\mathsf{\Gamma }}$, which is the total number of the boundary nodes, and set all the components of ${\overrightarrow{V}}_K^{\left({\sigma }_0\right)}$ to ${\sigma }_0$. Meanwhile, we let the $n_{\mathsf{\Gamma }}$-dimensional row and column index vectors be the same, i.e., ${\overrightarrow{I}}_K^{\left({\sigma }_0\right)}={\overrightarrow{J}}_K^{\left({\sigma }_0\right)}$, and set their components to be the boundary node numbers. Finally, we combine ${\overrightarrow{V}}_K^{\left(pq\right)}$ and ${\overrightarrow{V}}_K^{\left({\sigma }_0\right)}$, ${\overrightarrow{I}}_K^{\left(pq\right)}$ and ${\overrightarrow{I}}_K^{\left({\sigma }_0\right)}$, ${\overrightarrow{J}}_K^{\left(pq\right)}$ and ${\overrightarrow{J}}_K^{\left({\sigma }_0\right)}$, respectively, and input them into the sparse matrix generation function to obtain $K$.

Based on Expressions (68) and (69), the gradients of the objective and the constraint can be efficiently obtained by using $\mathsf{\Theta }$. For example, if we have two $M$-dimensional global node vectors $\overrightarrow{c}$ and $\overrightarrow{\alpha }$, we can adopt fancy indexing to generate two $Ne\times 4$ matrices $N_c=\overrightarrow{c}\left(\mathsf{\Theta }\right)$ and $N_{\alpha }=\overrightarrow{\alpha }\left(\mathsf{\Theta }\right)$ whose $e-th$ rows correspond to the node vectors of the $e-th$ element. According to Expression (69), the objective function gradients ${\nabla }_{\overrightarrow{p}}J$, ${\nabla }_{\overrightarrow{q}}J$ can be calculated by

$${\nabla }_{\overrightarrow{p}}J=-sum\left(N_c{K}_p·\ast N_{\alpha },2\right),\hspace{1em}{\nabla }_{\overrightarrow{q}}J=-sum\left(N_c{K}_q·\ast N_{\alpha },2\right),$$

(84)

where $·\ast$ stands for multiplying the corresponding entries of the matrices, and $sum\left(A,2\right)$ is summing the rows of a matrix to obtain a column vector. Mathematically, Expression (84) can be written as ${\nabla }_{\overrightarrow{p}}J=diag\left(N_c{K}_p{N}_{\alpha }^T\right)$ and ${\nabla }_{\overrightarrow{p}}J=diag\left(N_c{K}_q{N}_{\alpha }^T\right)$, where the function $diag( )$ is used to extract the main diagonal entries from a square matrix. Similarly, the constraint function gradients ${\nabla }_{\overrightarrow{p}}G$, ${\nabla }_{\overrightarrow{q}}G$ can be calculated by replacing $\overrightarrow{c}$, $\overrightarrow{\alpha }$ with $\overrightarrow{u}$, $\overrightarrow{v}$.

6.2. Image Classifier

For a given training dataset $D$, in order to use Optimization Algorithm 1 to construct the mutual-energy inner product coordinate axes ${\overrightarrow{\alpha }}_m\hspace{0.17em}\left(m=1,2\dots N_{\alpha }\right)$, we select the subset $D_s$ of $D$ as the reference training set and select the mean of samples of the class “0” or the class “1” in $D_s$ or a combination of these means as the reference feature $\overrightarrow{\alpha }$. The subset $D_s$ is gradually generated as the coordinate ${\overrightarrow{\alpha }}_m$ is generated. Prior to generating the coordinate ${\overrightarrow{\alpha }}_{m+1}$, $m$ mutual-energy inner product coordinate axes ${\overrightarrow{\alpha }}_1,\hspace{0.17em}{\overrightarrow{\alpha }}_2\dots \hspace{0.17em}{\overrightarrow{\alpha }}_m$ have been generated and there are $m$ subsets $D_s^{\left(1\right)},\hspace{0.17em}{D}_s^{\left(2\right)}\dots D_s^{\left(m\right)}$. One of the $m$ subsets is selected as a subset $D_s$ to generate the coordinate ${\overrightarrow{\alpha }}_{m+1}$. In order to explain how the generation of new axes work, we use a set $S_T$ to manage the $m$ generated subsets, i.e., $S_T={\left\{D_s^{\left(1\right)},D_s^{\left(2\right)}\dots D_s^{\left(m\right)}\right\}}_1^m$. If $D_s^{\left(i\right)}$ has $M_0^{\left(i\right)}$ samples of the class “0” and $M_1^{\left(i\right)}$ samples of the class “1”, the subset $D_s^{\left(k\right)}$ in $S_T$ is taken as the reference training sample set $D_s$ to generate ${\overrightarrow{\alpha }}_{m+1}$ and its index $k$ satisfies

$$k=\underset{i}{\mathrm{argmax}}\hspace{0.17em}\hspace{0.17em}\min \left(M_0^{\left(i\right)},M_1^{\left(i\right)}\right).$$

(85)

After determining the subset $D_s$ and the reference feature $\overrightarrow{\alpha }$, ${\overrightarrow{\alpha }}_{m+1}$ can be obtained by Optimization Algorithm 1. Next, we divide $D_s$ into two subsets. First, for each sample ${\overrightarrow{X}}^{\left(i\right)}$ in $D_s$, we calculate its coordinate component $z_{m+1}^{\left(i\right)}$ on the axis ${\overrightarrow{\alpha }}_{m+1}$ by $z_{m+1}^{\left(i\right)}=\langle {\overrightarrow{\alpha }}_{m+1},{\overrightarrow{X}}^{\left(i\right)}\rangle$; we calculate ${\mu }_{0,m+1}$ and ${\mu }_{1,m+1}$, the means of samples of the class “0” and the class “1” and set a threshold $z_{m+1}^{th}=\left({\mu }_{0,m+1}+{\mu }_{1,m+1}\right)/2$. Second, according to $z_{m+1}^{\left(i\right)}$ and $z_{m+1}^{th}$, we divide $D_s$ into two subsets satisfying $D_{sI}=\left\{{\overrightarrow{X}}^{\left(i\right)}|\hspace{0.17em}{\overrightarrow{X}}^{\left(i\right)}\in D_s,\hspace{1em}{z}_{m+1}^{\left(i\right)}\le z_{m+1}^{th}\right\}$ and $D_{sII}=\left\{{\overrightarrow{X}}^{\left(i\right)}|\hspace{0.17em}{\overrightarrow{X}}^{\left(i\right)}\in D_s,\hspace{1em}{z}_{m+1}^{\left(i\right)}>z_{m+1}^{th}\right\}$. Finally, we add $D_{sI}$ and $D_{sII}$ into $S_T$, and delete $D_s^{\left(k\right)}$ from $S_T$. At this time, $S_T$ contains $m+1$ training sample subsets, and one of them will be selected to calculate the coordinate axis ${\overrightarrow{\alpha }}_{m+2}$.

The following summarizes the detailed steps of generating mutual-energy inner product feature coordinates.

Algorithm 2: Mutual-energy inner product feature coordinates generation

(1)

Let $m=0$ and $S_T=\left\{D\right\}$;

(2)

According to Expression (85), select $D_s$ in $S_T$ to generate the coordinate axis ${\overrightarrow{\alpha }}_{m+1}$ and delete $D_s^{\left(k\right)}$ from $S_T$;

(3)

Adopt Optimization Algorithm 1 to calculate ${\overrightarrow{\alpha }}_{m+1}$ based on the determined reference subset $D_s$ and the selected reference feature $\overrightarrow{\alpha }$;

(4)

For each sample ${\overrightarrow{X}}^{\left(i\right)}$ in $D_s$, calculate its coordinate components $z_{m+1}^{\left(i\right)}$ on the axis ${\overrightarrow{\alpha }}_{m+1}$, the means ${\mu }_{0,m+1}$ and ${\mu }_{1,m+1}$ of the class “0” and the class “1”, as well as the threshold $z_{m+1}^{th}=\left({\mu }_{0,m+1}+{\mu }_{1,m+1}\right)/2$;

(5)

According to $z_{m+1}^{\left(i\right)}$ and $z_{m+1}^{th}$, divide $D_s$ into two subsets $D_{sI}$ and $D_{sII}$, and add them into $S_T$;

(6)

Judge if $m<N_{\alpha }$, set $m=m+1$ and go to Step (2); otherwise, stop.

After generating mutual-energy inner product coordinate axes ${\overrightarrow{\alpha }}_m\hspace{0.17em}\left(m=1,2\dots N_{\alpha }\right)$ by Algorithm 2, the coordinate components $z_m^{\left(i\right)}\hspace{0.17em}\left(m=1,2\dots N_m\right)$ of each sample in $D$ can be calculated and are represented by a feature vector ${\overrightarrow{z}}^{\left(i\right)}={\left[\begin{array}{cccc}{z}_1^{\left(i\right)}& z_2^{\left(i\right)}& \dots & z_{N_m}^{\left(i\right)}\end{array}\right]}^T$. Based on ${\overrightarrow{z}}^{\left(i\right)}$, a simple Gaussian classifier is used to classify the images. We use $D_j$ to represent a training dataset comprising $M_j$ samples, where the subscript $j$ is the class index of the samples. A Gaussian classifier can be used to classify the samples into multiple classes. We use $y^{\left(i\right)}=j\hspace{0.17em}\left(j=0,1\dots C-1\right)$ to indicate the class of a sample and use $C$ to denote the total number of classes. In $D$, the probability of the class $y^{\left(i\right)}=j$ is

$$p\left(y^{\left(i\right)}=j\right)={\displaystyle \frac{M_j}{{\displaystyle \sum _{k=0}^{C-1}{M}_k}}}.$$

(86)

Furthermore, it is assumed that, for the samples in the same class, their feature vectors ${\overrightarrow{z}}^{\left(i\right)}$ follow the Gaussian distribution

$$p\left({\overrightarrow{z}}^{\left(i\right)}|y^{\left(i\right)}=j\right)\sim N\left({\overrightarrow{z}}^{\left(i\right)}|{\overrightarrow{\mu }}_j,{\sum }_j\right),$$

(87)

where ${\overrightarrow{\mu }}_j$ is the mean of ${\overrightarrow{z}}^{\left(i\right)}$; ${\sum }_j$ is the covariance matrix of ${\overrightarrow{z}}^{\left(i\right)}$; and the subscript $j$ corresponds to the class $y^{\left(i\right)}=j$. Using the training sample dataset $D_j$, their maximum likelihood estimates can be calculated by [38]

$${\overrightarrow{\mu }}_j={\displaystyle \frac{1}{M_j}}{\displaystyle \sum _{i=1}^{M_j}{\overrightarrow{z}}^{\left(i\right)}},\hspace{1em}{\sum }_j={\displaystyle \frac{1}{M_j}}{\displaystyle \sum _{i=1}^{M_j}\left({\overrightarrow{z}}^{\left(i\right)}-{\overrightarrow{\mu }}_j\right)}{\left({\overrightarrow{z}}^{\left(i\right)}-{\overrightarrow{\mu }}_j\right)}^T.$$

(88)

Here, $z^{\left(i\right)}\in D_j$. Based on Expressions (86) and (87), when giving the feature vector of a sample, the posterior probability of the sample belonging to the class $y=j$ is

$$p\left(y=j|\overrightarrow{z}\right)={\displaystyle \frac{e^{{\beta }_j\left(\overrightarrow{z}\right)}}{{\displaystyle \sum _{i=1}^C{e}^{{\beta }_i\left(\overrightarrow{z}\right)}}}},$$

(89)

where $p\left(y=j|\overrightarrow{z}\right)$ is the posterior probability, and ${\beta }_j\left(\overrightarrow{z}\right)$ can be expressed as

$$\{\begin{array}{l}{\beta }_j\left(\overrightarrow{z}\right)={\displaystyle \frac{1}{2}}{\overrightarrow{z}}^T{H}_j\overrightarrow{z}+{\overrightarrow{b}}_j^T\overrightarrow{z}+c_j\\ H_j=-{\Sigma }_j^{-1}\hspace{1em}\hspace{1em}{\overrightarrow{b}}_j={\Sigma }_j^{-1}{\overrightarrow{\mu }}_j\\ c_j=-{\displaystyle \frac{1}{2}}{\overrightarrow{\mu }}_j^T{\Sigma }_j^{-1}{\overrightarrow{\mu }}_j-{\displaystyle \frac{1}{2}}\ln \left(\left|{\Sigma }_j\right|\right)+\ln \left(p\left(y=j\right)\right)\end{array}.$$

(90)

Finally, the class of the sample is determined based on the posterior probability

$$y=\hspace{0.17em}\underset{j}{\arg \max } p\left(y=j|\overrightarrow{z}\right)\hspace{1em}j\in \left\{0,1\dots C-1\right\}.$$

(91)

6.3. Numerical Examples

The MNIST dataset has become one of the benchmark datasets in machine learning. It comprises 60,000 sample images in the training set and 10,000 sample images in the test set, and each one is a 28-by-28-pixel grayscale image of the handwritten digits 0–9. In this section, we will use the MNIST to design Gaussian image classifiers based on Optimization Algorithm 1.

Before designing Gaussian image classifiers, image preprocessing is conducted to align the image centroids and normalize the sample images. In Optimization Algorithm 1, the selected parameters are $\lambda =0.3$, $p_{\min }=q_{\min }={10}^{-3}$, $Tolp=Tolq=2.0$, ${\sigma }_0={10}^5$, $\mathsf{\Delta }{x}_{\max }=0.08$, ${\epsilon }_x=8\times {10}^{-4}$, and ${\epsilon }_J={10}^{-7}$.

6.3.1. Binary Gaussian Classifier: Identify Digits “0” and “1”

The MINST training set comprises 6742 samples “1” and 5923 samples “0”. We select the difference between the means of samples “1” and “0” as the reference feature, i.e., $\overrightarrow{\alpha }=\overrightarrow{u}-\overrightarrow{v}$. Optimization Algorithm 1 converges after 166 iterations. The means of samples “1” and “0”, the design variables, and the reference feature coordinate $\alpha \left(\overrightarrow{x}\right)$, are visualized in Figure 1, Figure 2 and Figure 3. Due to obvious differences in the mean feature, digits “0” and “1” can be identified using only one mutual-energy inner product coordinate $\overrightarrow{\alpha }$. Figure 4a shows the training sample distribution in accordance with the components on $\overrightarrow{\alpha }$. Figure 5a gives the Confusion Matrix of the classification results, where the horizontal and vertical axes correspond to the target class and the output class of the classifier, respectively. In the Confusion Matrix, the column on the far right shows the precision of all the examples predicted to belong to each class, and the row at the bottom shows the recall of all the examples belonging to each class; the entry in the bottom right shows the overall accuracy; the diagonal entries are the correctly classified numbers of digits “0” and “1” and the off-diagonal entries correspond to the wrong classifications. This binary Gaussian classifier on the training set achieves a very high overall accuracy of 99.66%, shown at the bottom right of the Confusion Matrix.

The binary Gaussian classifier is tested on the MINST test set, which comprises 1135 samples “1” and 980 samples “0”. The test results are visualized in Figure 4b and Figure 5b. Its overall accuracy can reach 99.91%, higher than that on the training set.

6.3.2. Binary Gaussian Classifier: Identify Digits “0” and “2”

The MINST training set comprises 5958 samples “2” and 5923 samples “0”, and the MINST test set comprises 1032 samples “2” and 980 samples “0”. Similarly to the previous classifier, the reference feature is also selected as $\overrightarrow{\alpha }=\overrightarrow{u}-\overrightarrow{v}$. The difference in the mean features of digits “2” and “0” is not as significant as that of digits “1” and “0”. If only one mutual-energy inner product coordinate is used for classification, the accuracy is only 96.72% on the training set and 97.81% on the test set. In order to improve the classification accuracy, we use Algorithm 2 to generate 60 mutual-energy inner product coordinates based on the training sample set and its subsets, and construct a 60-dimensional Gaussian classifier. The Confusion Matrices of the classification results are given in Figure 6a,b, showing an overall accuracy of 99.55% on the training set and a higher overall accuracy of 99.85% on the test set.

6.3.3. Binary Gaussian Classifier: Identify Digits “3” and “4”

The MINST training set comprises 6131 samples of “3” and 5842 samples of “4”, and the MINST test set comprises 1010 samples of “3” and 982 samples of “4”. Here, we select the means of samples “3” and “4” as reference features, i.e., $\overrightarrow{\alpha }=\overrightarrow{u}$ and $\overrightarrow{\alpha }=\overrightarrow{v}$, and then use Algorithm 2 to generate 50 mutual-energy inner product coordinates, respectively, finally forming 100 classification coordinates. Because these coordinates are not linearly independent, we use matrix singular value decomposition to construct a 50-dimensional Gaussian classifier. Figure 6c,d is the Confusion Matrices, showing an overall accuracy of 99.67% on the training set and a higher overall accuracy of 99.80% on the test set.

6.3.4. Multiclass Gaussian Classifier: Identify Digits “0”, “1”, “2”, “3” and “4”

In the training set, we select one digit from samples “0”, “1”, “2”, “3” and “4” as the first class and the other training samples of the five digits as the second class, and we take the two classes as the training sample set. Then, we select the difference between the means of the samples in the two classes as the reference feature, i.e., $\overrightarrow{\alpha }=\overrightarrow{u}-\overrightarrow{v}$, and use Algorithm 2 to generate 120 mutual-energy inner product coordinates. In this way, we construct 5 training sample sets and finally generate 600 coordinates. However, many of them are linearly dependent; in order to identify the digits “0”, “1”, “2”, “3” and “4”, we use matrix singular value decomposition to reduce its dimensions from 600 to 60 and construct a 60-dimensional multiclass Gaussian classifier. Figure 7 shows an overall accuracy of 98.22% on the training set and a higher overall accuracy of 98.83% on the test set.

7. Discussion

Based on the solution space of the partial differential equations describing the vibration of a non-uniform membrane, the concept of the mutual-energy inner product is defined. By expending the mutual-energy inner product as a superposition of the eigenfunctions of the partial differential equations, an important property is found: the mutual-energy inner product has the significant advantage of enhancing the low-frequency eigenfunction components and suppressing the high-frequency eigenfunction components, compared to the Euclidean inner product.

In data classification, if the reference data features of the samples belong to a low-frequency subspace of the set of the eigenfunctions, these data features can be extracted through the mutual-energy inner product, which can not only enhance feature information but also filter out high-frequency data noise. As a result, a mutual-energy inner product optimization model is built to extract the feature coordinates of the samples, which can enhance the data features, reduce the sample deviations, and regularize the design variables. We make use of the minimum energy principle to eliminate the constraints of the partial differential equations in the optimization model and obtain an unconstrained optimization objective function. The objective function is a quadratic functional, which is convex with respect to the variables that minimize the objective function, is concave with respect to the variables that maximize the objective function, and is linear with respect to the design variables. These properties facilitate the design of optimization algorithms.

FEM is used to discrete the design domain, and the design variables of each element are set as constants. Based on these finite elements, the gradients of the mutual-energy inner product relative to the element design variables are analyzed, and a sequential linearization algorithm is constructed to solve the mutual-energy inner product optimization model. Algorithm implementation only involves solving equations with the positive definite symmetric matrix when calculating the intermediate variables and only needs to handle a few constraints in the nested linear optimization module, guaranteeing the stability and effectiveness of the algorithm.

The mutual-energy inner product optimization model is applied to extract the feature coordinates of the sample images and construct a low-dimensional coordinate system to represent the sample images. Multiclass Gaussian classifiers are trained and tested to classify the 2-D images. Here, only the means of the training sample set and its subsets are selected as reference features in Optimization Algorithm 1, and the vectorized implementation of Optimization Algorithm 1 is discussed. Generating mutual-energy inner product coordinates via the optimization model and training or testing Gaussian classifiers are two independent steps. In training or testing Gaussian classifiers, calculating mutual-energy inner products can be converted into calculating the Euclidean inner products between the reference feature coordinates and the sample data, not adding computational complexity to the Gaussian classifiers.

In the MINST dataset, the mutual-energy inner product feature coordinate extraction method is used to train a 1-dimensional two-class Gaussian classifier, a 50-dimensional two-class Gaussian classifier, a 60-dimensional two-class Gaussian classifier, and a 60-dimensional five-class Gaussian classifier, and good prediction results are achieved. The feature coordinate extraction method achieves a higher overall accuracy on the test set than that on the training set, indicating that the classification model is experiencing underfitting. This shows large potential in the achievable accuracy of this method that has not yet been explored.

From the view of theory and algorithm, this feature extraction method is obviously different from the existing techniques in machine learning. Its limitation is the need of the given reference features in advance. In this paper, only the mean features of a sample dataset and its subsets are selected as the reference features to construct Gaussian classifiers. In the future, convolution operation can be adopted to construct other image reference features, such as image edge features, local features, textures [39], and multi-scale features, and these image features can be combined to generate a mutual-energy inner product feature coordinate system. In addition, other ensemble classifiers, such as Bagging and AdaBoost, can be introduced to improve the performances of the image classifiers. Meanwhile, the feasibility of applying the mutual-energy inner product optimization method to the neural network will also be explored.