Hyperparameter Optimization EM Algorithm via Bayesian Optimization and Relative Entropy

Abstract

Hyperparameter optimization (HPO), which is also called hyperparameter tuning, is a vital component of developing machine learning models. These parameters, which regulate the behavior of the machine learning algorithm and cannot be directly learned from the given training data, can significantly affect the performance of the model. In the context of relevance vector machine hyperparameter optimization, we have used zero-mean Gaussian weight priors to derive iterative equations through evidence function maximization. For a general Gaussian weight prior and Bayesian linear regression, we similarly derive iterative reestimation equations for hyperparameters through evidence function maximization. Subsequently, after using relative entropy and Bayesian optimization, the aforementioned non-closed-form reestimation equations can be partitioned into E and M steps, providing a clear mathematical and statistical explanation for the iterative reestimation equations of hyperparameters. The experimental result shows the effectiveness of the EM algorithm of hyperparameter optimization, and the algorithm also has the merit of fast convergence, except that the covariance of the posterior distribution is a singular matrix, which affects the increase in the likelihood.

1. Introduction

In machine learning, parameters are classified into two categories: model parameters, which are internal and configurable, and hyperparameters, which are external and cannot be estimated from data. Model parameters include the weights in a deep neural network, for example. Hyperparameters include batch size, the learning rate, and the number of hidden layers in a neural network [1]. Hyperparameter optimization (HPO) represents a pivotal aspect of machine learning model training, including the process of fine-tuning the hyperparameters of a model to improve its performance. Hyperparameters are parameters that are set before the learning process begins. The process of tuning these hyperparameters can significantly affect the model’s accuracy and generalization ability [2]. There are some main hyperparameter optimization methods as follows. Grid search: Grid search is the brute-force way of searching hyperparameters [3], with defined lower and higher bounds along with specific steps [4]. Grid search works based on the Cartesian product of the different sets of values, evaluates every configuration, and returns the combination with the best performance [5]. Grid search is a simple implementation that, however, can be highly inefficient for large search spaces due to its exhaustive nature. This problem is further compounded as data dimensionality increases. Random search: Random search involves randomly sampling hyperparameter combinations from a predefined search space. While less computationally intensive than grid search, it can often identify superior hyperparameter configurations because of its ability to explore the search space more efficiently [6,7]. Genetic algorithms: Genetic algorithms are inspired by the process of natural selection, whereby a population of candidate solutions (hyperparameter configurations) evolves over multiple generations [8,9]. They are useful for exploring large search spaces and can handle discrete and nonlinear search spaces effectively. Gradient-based optimization: Gradient-based optimization treats hyperparameters as continuous variables [10,11,12]. This approach is often used in neural network architectures and can be efficient for optimizing large-scale models. Bayesian optimization: Bayesian optimization employs a probabilistic model of the objective function (model performance) to identify the most promising hyperparameter configurations for evaluation. This approach is more efficient than grid and random search, particularly in the context of high-dimensional and noisy optimization problems [13,14,15,16]. Recently, automated machine learning (AutoML) platforms also have been applied to practical problems, which automate the entire machine learning pipeline, including data preprocessing, model selection, feature engineering, and hyperparameter optimization. They leverage various optimization techniques to find the best model configuration automatically and make machine learning more accessible to users with limited expertise in data science and machine learning by automating repetitive tasks and reducing the need for manual intervention. This allows experts to focus more on understanding the problem domain and interpreting the results rather than spending time on technical details [17,18,19,20]. Particle swarm optimization (PSO) is a popular algorithm for hyperparameter optimization recently. It is simple to implement and explores the global search space efficiently [21,22,23]. However, methods based on gradients, grid search, random search, genetic algorithms, and particle swarm optimization all lack rigorous mathematical explanation; they are more computational logic. Additionally, grid search, random search, and particle swarm optimization are all brute-force methods that are time-consuming and labor-intensive. Our proposed hyperparameter optimization EM algorithm, based on Bayesian optimization theory and relative entropy, has a strict mathematical derivation and explanation. The simulation results show that the algorithm has the advantage of fast convergence.

The relevance vector machine (RVM) represents a Bayesian sparse kernel technique that has been developed for the purpose of regression tasks. The RVM model for regression is linear, whose weight prior is Gaussian, defined by the following form [24]

$$p(\omega |\eta )={\displaystyle \prod _{i=1}^{M}N({\omega }_i|0,{\eta }_i^{-1})},$$

(1)

where ${\eta }_i$ is a hyperparameter and $\eta ={({\eta }_1,\dots ,{\eta }_M)}^T$.

In this paper, we will propose a Bayesian hyperparameter optimization in terms of a more general form of weight prior, defined as follows:

$$p(\omega |\eta ,\mu )={\displaystyle \prod _{i=1}^{M}N({\omega }_i|{\mu }_i,{\eta }_i^{-1})},$$

(2)

where ${\mu }_i$ is also a hyperparameter and $\mu ={({\mu }_1,\dots ,{\mu }_M)}^T$.

We designate weight prior (2) as the general Gaussian weight prior (GGWP). Initially, leveraging evidence function maximization and GGWP, we derive non-closed-form iteratively reestimation equations for hyperparameters. Subsequently, we partition the aforementioned non-closed-form reestimation equations into E step and M step, elucidating the hyperparameter reestimation equations mathematically and statistically.

2. Related Mathematical Knowledge

Lemma 1. 

${\displaystyle \frac{\partial }{\partial x}}\ln \left|M\right|=Tr(M^{-1}{\displaystyle \frac{\partial M}{\partial x}})$.

Lemma 2. 

${\displaystyle \frac{\partial }{\partial x}}(M^{-1})=-M^{-1}{\displaystyle \frac{\partial M}{\partial x}}{M}^{-1}$.

The proofs of both Lemmas 1 and 2 refer to [24,25,26,27]

Definition 1.

Let $\nabla$ be gradient and $h^T\nabla$ is the operator such that for $k\in N$,

$${(h^T\nabla )}^{k+1}f=(h^T\nabla ){(h^T\nabla )}^{k}f,$$

where ${(h^T\nabla )}^{0}f=f$.

The symbolic powers are referred to as operators that act upon a multivariate function with $n$ variables.

Lemma 3

(multivariate Taylor’s expansion). Let $f:B\left(x,r\right)\to ℝ$ be a multivariate function, where $B\left(x,r\right)\subseteq {ℝ}^k$ and $f(x)\in C\left(B\left(x,r\right)\right)$. For $h\in {ℝ}^k$  $\left(\parallel h\parallel <r\right)$, then there exists $\theta \in (0,1)$ such that 

$$f\left(x+h\right)={\displaystyle \sum }_{l=0}^{n-1}{\displaystyle \frac{1}{l!}}({(h^T\nabla )}^{l}f)(x)+{\displaystyle \frac{1}{n!}}({(h^T\nabla )}^{n}f)(x+\theta h).$$

Both Definition 1 and proof of Lemma 3 in detail refer to [28].

Corollary 1.

A quadratic function $f:B\left(w,r\right)\to ℝ$ is defined as follows:

$$f(x)=x^{T}Ax+x^{T}b+c,$$

where $c$ is a scalar constant, $b$ and $x={(x_1,\dots ,x_n)}^T$ are both n-dimensional column vectors as well as $A$ is a n-dimensional inverse and symmetric matrix. Then, at the stationary point $x_0={\displaystyle \frac{1}{2}}{A}^{-1}b$, we obtain the Taylor expansion of the function $f(x)$ as follows:

$$f(x)=f(x_0)+{\displaystyle \frac{1}{2}}{(x-x_0)}^{T}H(x-x_0),$$

where elements of the Hessian matrix $H$ are defined by

$$\left({\displaystyle \frac{\partial f(x)}{\partial x_i\partial x_j}}\right).$$

The proof of Corollary 1 refers to [29] in detail.

3. HPO via Maximization of the Evidence Function

3.1. Bayesian Linear Regression and Linear Basis Function Models

The linear model for regression is given with linear combinations of fixed nonlinear functions of the input variables as follows:

$$y(x,\omega )={\omega }_0+{\displaystyle \sum _{i=1}^{M-1}{\omega }_i{\psi }_i(x)},$$

(3)

where ${\omega }_0$ is called bias parameter and ${\psi }_i(x)$ is a basis function. For the sake of convenience, setting ${\psi }_0(x)=1$.

We rewrite (3) in matrix form [24]:

$$y(x,\omega )={\omega }^T\psi (x),$$

(4)

where $\psi (x)={({\psi }_0(x),\dots ,{\psi }_{M-1}(x))}^T$ and $\omega ={({\omega }_0,\dots ,{\omega }_{M-1})}^T$.

The target variable $t$ for regression is defined as the sum of a random noise $\epsilon$ and a deterministic function $y(x,\omega )$ as follows:

$$t=y(x,\omega )+\epsilon ,$$

(5)

where the noise $\epsilon$ is Gaussian, whose mean is zero and precision is $\lambda$ (inverse variance). So, we obtain

$$p(t|x,\omega ,\lambda )=N(t|y(x,\omega ),{\lambda }^{-1}).$$

(6)

Now, $X=(x_1,\dots ,x_N)$, called the input dataset, is now considered in conjunction with its target value vector $T=(t_1,\dots ,t_N)$. These data points are all drawn independently from the Gaussian distribution (6) based on our assumption. Thus, the likelihood function is obtained as follows:

$$p(T|X,\omega ,\lambda )={\displaystyle \prod _{i=1}^{N}N(t_i|{\omega }^T\psi (x_i),{\lambda }^{-1})}.$$

(7)

According to (7), we define corresponding conjugate prior as follows:

$$p(\omega )=N(\omega |m_0,C_0)$$

(8)

From (7) and (8), we can shortly obtain the following posterior distribution:

$$p(\omega |T,X,\lambda )=N(\omega |m_N,C_N),$$

(9)

where

$$m_N=C_N(C_0^{-1}{m}_0+\lambda {\Psi }^{T}t),$$

(10)

$$C_N^{-1}=C_0^{-1}+\lambda {\Psi }^T\Psi ,$$

(11)

$$\Psi =\left(\begin{array}{ccc}{\psi }_0(x_1)& \dots & {\psi }_{M-1}(x_1)\\ ⋮& \ddots & ⋮\\ {\psi }_0(x_N)& \dots & {\psi }_{M-1}(x_N)\end{array}\right).$$

(12)

The proofs of both (10) and (11) in detail refer to Appendix A in [29].

3.2. Evidence Approximation and Bayesian Model Comparison

Let us suppose we are presented with a series of models $\left\{M_k\right\}$, $k=1,\dots ,L$. We compare these models and then choose the optimal one according to a Bayesian perspective, aiming to mitigate the risk of overfitting commonly associated with maximum likelihood approaches. We express uncertainty by a prior probability $p(M_k)$. In the context of a training set, it is reasonable to assume that all models have the same prior probability. This assumption is consistent with the notion that, in practice, there should be no inherent preference for any specific models $L$. The objective is thus to assess the following posterior distribution given the dataset $D$,

$$p(M_k|D)\propto p(M_k)p(D|M_k).$$

(13)

The posterior distribution $p(M_k|D)$ represents the model evidence, also known as the marginal likelihood. This term is interpreted as a likelihood function [24].

In this section, we present a fully Bayesian treatment based on the introduction of priors over the hyperparameters $\eta$, $\lambda$ and $\mu$. The derivation of predictions can be accomplished by using the marginalization over these hyperparameters and the weight parameter. Nevertheless, it is difficult to completely marginalize all the variables in the analytical process. Consequently, we propose an approximation approach to identify these hyperparameters $\eta$, $\lambda$ and $\mu$ by the maximization of the marginal likelihood function after integrating over these parameters. In statistics, the method is commonly referred to as empirical Bayes [30,31], type 2 maximum likelihood [32], or generalized likelihood [33]. In the field of machine learning, it is frequently referred to as evidence approximation. [24,34]. Furthermore, a hyperprior over the parameters $\eta$, $\lambda$ and $\mu$ is introduced first. Subsequently, the following predictive distribution is obtained:

$$p(t|T)={\displaystyle \int {\displaystyle \int {\displaystyle \int {\displaystyle \int p(t|x,\omega ,\lambda )}}}}p(\eta ,\lambda ,\mu |T)p(\omega |T,\eta ,\lambda ,\mu )\mathrm{d}\eta d\lambda d\mu d\omega ,$$

(14)

where $p(t|x,\omega ,\lambda )$ is given according to (6) and $p(\omega |T,\eta ,\lambda ,\mu )$ is determined from (9).

If the posterior $p(\eta ,\lambda ,\mu |T)$ exhibits a sharp peak around $\widehat{\eta }$, $\widehat{\lambda }$ and $\widehat{\mu }$, the predictive distribution can be derived by marginalizing over $\omega$, where we set the hyperparameters to the value $\widehat{\eta }$, $\widehat{\lambda }$ and $\widehat{\mu }$. So, we obtain

$$p(t|T)\simeq p(t|T,\widehat{\eta },\widehat{\lambda },\widehat{\mu })={\displaystyle \int p(\omega |T,\widehat{\eta },\widehat{\lambda },\widehat{\mu })}p(t|x,\omega ,\widehat{\lambda })d\omega .$$

(15)

The posterior for the hyperparameters is expressed as

$$p(\eta ,\lambda ,\mu |T)\propto p(\eta ,\lambda ,\mu )p(T|\eta ,\lambda ,\mu ).$$

(16)

The values of $\widehat{\eta }$, $\widehat{\lambda }$, and $\widehat{\mu }$ can be derived by the maximization of the marginal likelihood function $p(T|\eta ,\lambda ,\mu )$ based on the evidence approximation, particularly when the prior is relatively flat. By continuously evaluating the marginal likelihood, it is possible to identify its maxima, which enables the values of these hyperparameters to be determined solely from the training dataset.

3.3. The Evidence Function Evaluation

From (7), we derive the following result:

$$\begin{array}{cc}p(T|X,\omega ,\lambda )\hfill & ={\displaystyle \prod _{i=1}^{N}N(t_i|{\omega }^T\psi (x_i),{\lambda }^{-1})}\hfill \\ & =\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}\exp (-{\displaystyle \frac{\lambda }{2}}{\displaystyle \sum _{i=1}^N{(t_i-{\omega }^T\psi (x_i))}^2)}\hfill \\ & =\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}\exp (-{\displaystyle \frac{\lambda }{2}}{‖T-\Psi \omega ‖}^2).\hfill \end{array}$$

(17)

From (2), the following result is obtained:

$$\begin{array}{cc}p(\omega |\eta ,\mu )\hfill & ={\displaystyle \prod _{i=1}^{M}N({\omega }_i|{\mu }_i,{\eta }_i^{-1})}\hfill \\ & ={\displaystyle \frac{\sqrt{{\displaystyle \prod _{i=1}^M{\eta }_i}}}{\sqrt{{(2\pi )}^M}}}\exp (-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^M{\eta }_i}{({\omega }_i-{\mu }_i)}^2)\hfill \\ & ={\displaystyle \frac{{\left|\Lambda \right|}^{\frac{1}{2}}}{\sqrt{{(2\pi )}^M}}}\exp (-{\displaystyle \frac{1}{2}}{(\omega -\mu )}^T\Lambda (\omega -\mu )).\hfill \end{array}$$

(18)

It can be demonstrated that the posterior with respect to weight parameter $\omega$ is Gaussian as follows:

$$p(\omega |T,X,\eta ,\lambda ,\mu )=N(\omega |m,C),$$

(19)

where

$$m={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}(\Lambda \mu +\lambda {\Psi }^{T}T),$$

(20)

$$C={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1},$$

(21)

where $\Lambda =diag({\eta }_i)$.

The proofs of both (20) and (21) are detailed in Appendix A.

The evidence approximation is utilized to evaluate the hyperparameters. We obtain the marginal likelihood function (evidence function) by integrating out the weight parameters as follows:

$$p(T|X,\eta ,\lambda ,\mu )={\displaystyle \int p(T|} X,\omega ,\lambda )p(\omega |\eta ,\mu )d\omega ,$$

(22)

Then, we obtain

$$p(T|X,\eta ,\lambda ,\mu )=\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}{\left|\Lambda \right|}^{{\displaystyle \frac{1}{2}}}{\left|C\right|}^{{\displaystyle \frac{1}{2}}}\exp (-G(m)),$$

(23)

where $G(m)={\displaystyle \frac{\lambda }{2}}{‖T-\Psi m‖}^2+{\displaystyle \frac{1}{2}}{(m-\mu )}^T\Lambda (m-\mu )$.

The proof of (23) is detailed in Appendix B.

In terms of (23), we take the logarithm of the marginal likelihood function, also called the log marginal likelihood function, and obtain the following result:

$$\ln p(T|X,\eta ,\lambda ,\mu )={\displaystyle \frac{N}{2}}\ln \lambda -G(m)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^M\ln }{\eta }_i+{\displaystyle \frac{1}{2}}\ln \left|C\right|-{\displaystyle \frac{N}{2}}\ln (2\pi ).$$

(24)

For convenience, we abbreviate the marginal likelihood function as the likelihood and the log marginal likelihood function as the log likelihood. Subsequently, the log likelihood will be maximized.

3.4. The Evidence Function Maximization

Now, $p(T|X,\eta ,\lambda ,\mu )$ is maximized with respect to ${\eta }_i$. From (24) and Lemma 1, we derive the following result:

$${\displaystyle \frac{\partial }{\partial {\eta }_i}}\ln \left|C\right|=Tr(C^{-1}{\displaystyle \frac{\partial C}{\partial {\eta }_i}})=Tr((\Lambda +\lambda {\Psi }^T\Psi ){\displaystyle \frac{\partial }{\partial {\eta }_i}}({(\Lambda +\lambda {\Psi }^T\Psi )}^{-1})),$$

(25)

By utilization of (21), (25), and Lemma 1, we get the following result:

$${\displaystyle \frac{\partial }{\partial {\eta }_i}}\ln \left|C\right|=-Tr(({\displaystyle \frac{\partial }{\partial {\eta }_i}}(\Lambda +\lambda {\Psi }^T\Psi ))C),$$

(26)

where we have used

$${\displaystyle \frac{\partial }{\partial {\eta }_i}}({(\Lambda +\lambda {\Psi }^T\Psi )}^{-1})=-{(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}{\displaystyle \frac{\partial }{\partial {\eta }_i}}((\Lambda +\lambda {\Psi }^T\Psi )){(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}.$$

By making use of (26), the following result was obtained

$${\displaystyle \frac{\partial }{\partial {\eta }_i}}\ln \left|C\right|=-C_{ii},$$

(27)

where $C_{ii}$ represents the ith component of the principal diagonal of the covariance matrix $C$.

The following result is obtained:

$${\displaystyle \frac{\partial G(m)}{\partial {\eta }_i}}={\displaystyle \frac{1}{2}}{m}_i^2,$$

(28)

By making use of (27) and (28), the stationary point is obtained regarding ${\alpha }_i$ such that

$${\displaystyle \frac{1}{2{\eta }_i}}-{\displaystyle \frac{1}{2}}{C}_{ii}-{\displaystyle \frac{1}{2}}{m}_i^2=0,$$

(29)

in which $m_i$ is the ith element of the matrix $m$.

From (29), we obtain the following result:

$${\eta }_i={\displaystyle \frac{1}{m_i^2+C_{ii}}}.$$

(30)

The derivative of $\ln \left|C\right|$ is obtained regarding $\lambda$ by applying Lemma 2 and (21) as follows:

$${\displaystyle \frac{\partial }{\partial \lambda }}\ln \left|C\right|=Tr(C^{-1}{\displaystyle \frac{\partial C}{\partial \lambda }})=Tr((\Lambda +\lambda {\Psi }^T\Psi ){\displaystyle \frac{\partial }{\partial \lambda }}({(\Lambda +\lambda {\Psi }^T\Psi )}^{-1})).$$

(31)

The application of Lemma 2 yields the following result.

$${\displaystyle \frac{\partial }{\partial \lambda }}\ln \left|C\right|=-Tr({\Psi }^T\Psi C).$$

(32)

We also get

$${\displaystyle \frac{\partial G(m)}{\partial \lambda }}={\displaystyle \frac{1}{2}}{‖T-\Psi m‖}^2.$$

(33)

From (32) and (33), the stationary point is obtained regarding $\lambda$ by satisfying the following conditions:

$${\displaystyle \frac{N}{\lambda }}-{‖T-\Psi m‖}^2-Tr({\Psi }^T\Psi C)=0.$$

(34)

From (34), we obtain

$$\lambda ={\displaystyle \frac{N}{{‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C)}}.$$

(35)

Let ${\displaystyle \frac{\partial G(m)}{\partial \mu }}=\Lambda \mu -\Lambda m=0$, we obtain

$$\mu =m.$$

(36)

From the above derivation, we know that (30), (35) and (36) are non-closed-form reestimation equations for $\eta$, $\lambda$ and $\mu$. In order to solve the problem, an iterative procedure is presented. Initially, after the mean and covariance are both computed using (20) and (21) by giving initial value $\eta$, $\lambda$ and $\mu$, respectively, we reestimate hyperparameters alternately by employing (30), (35) and (36). We again alternately employ (20) and (21) to recompute the mean and covariance, repeating this process until a suitable standard for convergence is met. Nevertheless, it remains unclear whether this approach is accurate, as outlined in the reestimate iterative procedure. To gain a clearer understanding of the iterative reestimation equations, we will turn to the EM algorithm to analyze them from both mathematical and statistical perspectives.

4. HPO EM Algorithm via Bayesian Optimization and Relative Entropy

The expectation maximization (EM) algorithm is a highly effective and sophisticated approach for identifying maximum likelihood solutions within probabilistic models that incorporate latent variables [35,36,37].

The EM algorithm was derived by first treating the weights as latent variables, a process that is inherently straightforward. The log marginal likelihood function is then obtained by the weight parameters marginalization over the joint distribution $p(T,\omega |X,\eta ,\lambda ,\mu )$

$$L=\ln p(T|X,\eta ,\lambda ,\mu )=\ln {\displaystyle \int p(T,\omega |X,\eta ,\lambda ,\mu )d\omega },$$

(37)

where $p(T,\omega |X,\eta ,\lambda ,\mu )$ is equal to the product of the prior $\omega$ and the likelihood given by the following result:

$$p(T,\omega |X,\eta ,\lambda ,\mu )=p(\omega |\eta ,\mu )p(T|X,\omega ,\lambda ).$$

(38)

From Jensen’s inequality, we obtain a lower bound on $L$ as follows:

$$\begin{array}{cc}L\hfill & =\ln {\displaystyle \int \frac{p(T,\omega |X,\eta ,\lambda ,\mu )}{v(\omega )}v(\omega )d\omega }\hfill \\ & \ge {\displaystyle \int \ln \frac{p(T,\omega |X,\eta ,\lambda ,\mu )}{v(\omega )}v(\omega )d\omega }\hfill \\ & =F(v,\eta ,\lambda ,\mu ,\theta ),\hfill \end{array}$$

where $v(\omega )$ is the variational probability distribution over the weights and $\theta$ denotes all other parameters. The EM algorithm is designed to maximize the log marginal likelihood function $L$ by using iterative maximization of the lower bound. For the E step, we maximize $F$ regarding the probability distribution $v(\omega )$ for fixed hyperparameters $\eta$, $\lambda$ and $\mu$. For the M step, we maximize $F$ regarding the hyperparameters $\eta$, $\lambda$ and $\mu$ for the fixed probability distribution $v(\omega )$. We rewrite the lower bound $F$ to enhance the understanding of the E step:

$$F=KL(v(\omega )||p(\omega |T,X,\eta ,\lambda ,\mu ))+L,$$

(39)

where $KL(v(\omega )||p(\omega |T,X,\eta ,\lambda ,\mu ))$ is the relative entropy, also called Kullback–Leibler (KL) divergence. The KL divergence is always greater than or equal to zero, which is zero only and only if the two distributions are equal. The E step corresponds thus to equating the posterior on $\omega$ with the distribution $v(\omega )$, which means $v(\omega )=p(\omega |T,X,\eta ,\lambda ,\mu )$ and $F=L$ [38]. Given that the posterior probability distribution is also Gaussian, the E step is reduced to the computation of its mean matrix $m$ and covariance matrix $C$ defined by (20) and (21), respectively.

To enforce the M step, $F$ is rewritten as a distinct form:

$$F={\displaystyle \int \ln p(T,\omega |X,\eta ,\lambda ,\mu )}v(\omega )d\omega -H(v(\omega )),$$

(40)

in which $H(v(\omega ))$ clearly is irrelevant to $\eta$, $\lambda$, and $\mu$ and is the entropy of $v(\omega )$.

In order to perform the M step, from (40), we obtain

$$\begin{array}{cc}F\hfill & ={\displaystyle \int \ln p(T,\omega |X,\eta ,\lambda ,\mu )v(\omega )}d\omega -H(v(\omega ))\hfill \\ & ={\displaystyle \int (\ln p(\omega |\eta ,\mu )}+\ln p(T|X,\omega ,\lambda ))v(\omega )d\omega -H(v(\omega ))\hfill \\ & ={\displaystyle \int \ln p(\omega |\eta ,\mu )}v(\omega )d\omega +{\displaystyle \int \ln p(T|X,\omega ,\lambda )}v(\omega )d\omega -H(v(\omega ))\hfill \end{array}$$

(41)

Finally, we obtain the following result:

$$\begin{array}{cc}F\hfill & ={\displaystyle \frac{1}{2}}\ln \left|\Lambda \right|-{\displaystyle \frac{M+N}{2}}\ln (2\pi )-{\displaystyle \frac{1}{2}}{m}^T\Lambda m-{\displaystyle \frac{1}{2}}Tr(\Lambda C)+{\mu }^{T}m-{\displaystyle \frac{1}{2}}{\mu }^T\mu \hfill \\ & +{\displaystyle \frac{N}{2}}\ln \lambda -{\displaystyle \frac{\lambda }{2}}({‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C))-H(v(\omega ))\hfill \end{array}$$

(42)

The proof of (42) is derived in detail in Appendix C.

The stationary point of (42) is obtained with ease regarding ${\eta }_i$, $\lambda$, and $\mu$ such that

$${\displaystyle \frac{\partial F}{\partial {\eta }_i}}={\displaystyle \frac{1}{2{\eta }_i}}-{\displaystyle \frac{1}{2}}(m_i^2+C_{ii})=0,$$

(43)

$${\displaystyle \frac{\partial F}{\partial \lambda }}={\displaystyle \frac{N}{2\lambda }}-{\displaystyle \frac{1}{2}}({‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C))=0,$$

(44)

$${\displaystyle \frac{\partial F}{\partial \mu }}=m-\mu =0.$$

(45)

So, the update laws are obtained as follows

$${\eta }_i^{new}={\displaystyle \frac{1}{m_i^2+C_{ii}}},$$

(46)

$${\lambda }^{new}={\displaystyle \frac{N}{{‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C)}},$$

(47)

$${\mu }^{new}=m$$

(48)

The goal of the update equations is to achieve the maximization of the log marginal likelihood by the proposed EM algorithm. This result is also the same as maximizing the evidence function.

Finally, let us provide an intuitive explanation of our proposed EM algorithm for hyperparameter optimization. Firstly, we generate initial random hyperparameters values $\eta$, $\lambda$, $\mu$, which allows us to “guess” the initial hyperparameter. Then, we compute(guess) $\eta$, $\lambda$, $\mu$ again by (46), (47), and (48). Subsequently, the log marginal likelihood is evaluated. By repeating this process, the algorithm alternates between refining the guesses and the log marginal likelihood gradually increases until it no longer changes—that is, once it has converged, the algorithm stops making further guesses for hyperparameters $\eta$, $\lambda$, $\mu$. This also implies that the hyperparameters have converged. Checking the convergence of the log likelihood function is easier to implement programmatically than checking the convergence of the hyperparameters. Different initial values for hyperparameters may lead to different local optima while the likelihood converges.

5. Experimental Set-Up

5.1. Synthetic Data

The parameter $w$ either is integrated out or is considered latent variables, so after sampling from a Gaussian distribution, its goal is to construct $\Psi$.

We firstly generate hyperparameters $\eta$, $\lambda$, $\mu$ and $X=\left\{x_1,\dots ,x_N\right\}$ randomly and choose a suitable linear basis function ${\psi }_j(x_n)$ to get $\Psi$ and $T$. Subsequently, the EM algorithm is applied in order to maximize the likelihood or the log likelihood. The procedure is presented in detail as Algorithm 1:

Algorithm 1: HPO EM algorithm for synthetic data
1. Generate randomly hyperparameters value $\eta$, $\lambda$, $\mu$ and the dataset $X=\left\{x_1,\dots ,x_N\right\}$
2. Choose a suitable linear basis function $\psi (x_n)$ to get an $N\times M$ matrix $\Psi$ by using (12).
3. Generate parameters value $\omega$ randomly sampled by (2).
4. Generate N-dimensional vector $\epsilon$ sampled randomly by the Gaussian distribution $N(0,{\lambda }^{-1})$ and then generate $T=(t_1,\dots ,t_N)$ by (4) and (5), respectively.
5. E step. Compute the mean $m$ and covariance $C$ using the current hyperparameter values.
$m={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}(\Lambda \mu +\lambda {\Psi }^{T}T)$,	(49)
$C={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}$.	(50)
6. M step. Estimate again the hyperparameters by employing the mean $m$ and covariance $C$ obtained by step 5 and the following update equations
${\eta }_i^{new}={\displaystyle \frac{1}{m_i^2+C_{ii}}}$,	(51)
${\lambda }^{new}={\displaystyle \frac{N}{{‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C)}}$,	(52)
${\mu }^{new}=m$.	(53)
7. Compute the likelihood function or the log likelihood function given by the following result:
$p(T|X,\eta ,\lambda ,\mu )=\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}{\left|\Lambda \right|}^{{\displaystyle \frac{1}{2}}}{\left|C\right|}^{{\displaystyle \frac{1}{2}}}\exp (-G(m))$
or
$\ln p(T|X,\eta ,\lambda ,\mu )={\displaystyle \frac{N}{2}}\ln \lambda -G(m)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^M\ln }{\eta }_i+{\displaystyle \frac{1}{2}}\ln \left|C\right|-{\displaystyle \frac{N}{2}}\ln (2\pi )$
and then determine the convergence of the hyperparameters or the likelihood. If convergence is not satisfied, back to step 5. If the likelihood or the log likelihood converges, then the algorithm’s computational complexity is $O(M)$.

We apply Algorithm 1 to optimize hyperparameters value $\eta$, $\lambda$, and $\mu$, where the linear basis function $\psi (x)$ is a Gaussian basis function. We have difficulty judging the convergence of hyperparameter values $\eta$, $\lambda$, and $\mu$, so we turn to judge the convergence of the likelihood or the log likelihood. It is reasonable because the convergence of hyperparameter values $\eta$, $\lambda$, and $\mu$ means the likelihood or the log likelihood keeps constant; in other words, the convergence of the likelihood or the log likelihood. This is much easier to implement programmatically.

From Figure 1, we see that after approximately 10 iterations, the likelihood converges, so we can draw the conclusion that the EM algorithm has the advantage of fast convergence, which comes from strict interpretation mathematically and statistically.

In the process of experimental set-up, the covariance $C$ of the posterior distribution defined by (50) may easily be a singular matrix, so we have to use a pseudo-inverse matrix to evaluate $C$, which may produce the imprecise result that affects the increase in the likelihood function as shown in Figure 2.

The singularity for $C$ arises from the initial value of the hyperparameter values $\eta$, $\lambda$, and $\mu$, the simplest and intuitive method is to randomly generate initial values of $\eta$, $\lambda$, and $\mu$ again and again when the singularity occurs until the convergence criterion is met like Figure 2.

Particle swarm optimization (PSO) is an intuitive and computationally efficient metaheuristic that is highly effective for hyperparameter optimization across a wide range of machine learning models. For the same likelihood function, we use PSO to optimize the hyperparameter values $\eta$, $\lambda$, and $\mu$, as shown in Figure 3. After 10,000 iterations, the likelihood function still has not converged. This highlights the advantage of our proposed method’s fast convergence, which is attributed to its rigorous mathematical foundation.

5.2. The Diabetes Dataset

The diabetes dataset contains data from 442 diabetes patients, with each patient having measurements for 10 baseline variables, including age, sex, body mass index, average blood pressure, and six blood serum measurements. In addition, each patient has a response variable that represents a quantitative measure of disease progression one year after the baseline. This dataset is commonly used for predicting the progression of the disease and is one of the most frequently used datasets in machine learning. It is particularly well suited for research on regression problems.

Firstly, from the diabetes dataset, we obtain the input dataset $X=\left\{x_1,\dots ,x_{442}\right\}$, a $8\times 442$ matrix and the target value $T=(t_1,\dots ,t_{442})$. We then generate hyperparameters $\eta$, $\lambda$, $\mu$ randomly and choose a suitable linear basis function ${\psi }_j(x_n)$ to get $\Psi$. Subsequently, the EM algorithm is applied in order to maximize the likelihood or the log likelihood. The procedure is presented in detail as Algorithm 2:

Algorithm 2: HPO EM algorithm for the diabetes dataset
1. Generate randomly hyperparameters value $\eta$, $\lambda$, $\mu$.
2. Choose a suitable linear basis function $\psi (x_n)$ to get an $8\times 442$ matrix $\Psi$ by using (12).
3. E step. Compute the mean $m$ and covariance $C$ using the current hyperparameter values.
$m={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}(\Lambda \mu +\lambda {\Psi }^{T}T),$	(54)
$C={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}.$	(55)
4. M step. Estimate again the hyperparameters by employing the mean $m$ and covariance $C$ obtained by step 3 and the following update equations
${\eta }_i^{new}={\displaystyle \frac{1}{m_i^2+C_{ii}}},$	(56)
${\lambda }^{new}={\displaystyle \frac{N}{{‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C)}},$	(57)
${\mu }^{new}=m.$	(58)
5. Compute the likelihood function or log likelihood function given by the following result:
$p(T|X,\eta ,\lambda ,\mu )=\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}{\left|\Lambda \right|}^{{\displaystyle \frac{1}{2}}}{\left|C\right|}^{{\displaystyle \frac{1}{2}}}\exp (-G(m))$
or
$\ln p(T|X,\eta ,\lambda ,\mu )={\displaystyle \frac{N}{2}}\ln \lambda -G(m)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^M\ln }{\eta }_i+{\displaystyle \frac{1}{2}}\ln \left|C\right|-{\displaystyle \frac{N}{2}}\ln (2\pi )$
and then determine the convergence of the hyperparameters or the likelihood. If convergence criterion is not satisfied, go back to step 3.

From Figure 4, we can also draw the same conclusion that the EM algorithm has the advantage of fast convergence for the famous dataset of machine learning after the first iteration.

6. Conclusions

In this paper, we present the general Gaussian weight prior rather than zero-mean Gaussian weight priors for the hyperparameter optimization in the field of machine learning. We firstly derive non-closed-form iterative reestimation equations for hyperparameters by evidence function maximization. Though we know how to optimize the hyperparameters, we can’t understand the iterative reestimation equations clearly. To better understand non-closed-form reestimation equations more clearly, we have to resort to the EM algorithm. After using Bayesian theory and optimization, the EM algorithm partitions the iterative reestimation equations into E and M steps, which provides a clear interpretation for the iterative reestimation equations both mathematically and statistically. The experimental result shows the effectiveness of the EM algorithm of hyperparameter optimization, and the EM algorithm also has the advantage of fast convergence, except that the variance $C$ of the posterior distribution defined by (50) is a singular matrix, which affects the increase in the likelihood.