Decomposed Gaussian Processes for Efficient Regression Models with Low Complexity

Abstract

In this paper, we address the challenges of inferring and learning from a substantial number of observations ($N\gg 1$) with a Gaussian process regression model. First, we propose a flexible construction of well-adapted covariances originally derived from specific differential operators. Second, we prove its convergence and show its low computational cost scaling as $\mathcal{O}\left(N{m}^2\right)$ for inference and $\mathcal{O}\left(m^3\right)$ for learning instead of $\mathcal{O}\left(N^3\right)$ for a canonical Gaussian process where $N\gg m$. Moreover, we develop an implementation that requires less memory $\mathcal{O}\left(m^2\right)$ instead of $\mathcal{O}\left(N^2\right)$. Finally, we demonstrate the effectiveness of the proposed method with simulation studies and experiments on real data. In addition, we conduct a comparative study with the aim of situating it in relation to certain cutting-edge methods.

1. Introduction

Gaussian processes are powerful and flexible statistical models that have gained significant popularity in different fields: signal processing, medical imaging, data science, machine learning, econometrics, shape analysis, etc. [1,2,3,4,5]. They provide a nonparametric approach for modeling complex relationships and uncertainty estimation in data [6]. The core idea of Gaussian processes is the assumption that any finite set of data points can be jointly modeled as a multivariate Gaussian distribution [7]. Rather than explicit formulations, Gaussian processes allow for the incorporation of prior knowledge and inference of a nonparametric function f that generates the Gaussian process for a set ${(t_i,y_i)}_{i=1}^N$ with $y_i=f\left(t_i\right)+{\tau }_i$; $t_i\in I\subset {\mathbb{R}}^d$ and noisy measurements $y_i\in \mathbb{R}$. If f is modeled with a Gaussian process prior, then it can be fully characterized by a mean $\mu$ and a covariance function k, satisfying

$$\begin{array}{ccc}\hfill \mu \left(t\right)& =& \mathbb{E}\left(f\left(t\right)\right);t\in I\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill k(t,s)& =& \mathbb{E}\left(\left(f\right(t)-\mu (t\left)\right)\left(f\right(s)-\mu (s\left)\right)\right);t,s\in I.\hfill \end{array}$$

(2)

The mean function is usually assumed to be zero ($\mu \left(t\right)=0$), whereas the covariance $k(t,s)$ provides the dependence between two inputs, t and s. Gaussian processes can be applied for various tasks, including regression [8], classification [9], and time series analysis [10]. In regression, Gaussian processes can capture complex and non-linear patterns in data, while, in classification, they enable probabilistic predictions and can handle imbalanced datasets [11]. Additionally, Gaussian processes have been successfully employed in optimization, experimental design, reinforcement learning, and more [12].

One of the key advantages of Gaussian processes is their ability to provide a rich characterization of uncertainty. This makes them particularly suitable for applications where robust uncertainty quantification is crucial, such as in decision-making processes or when dealing with limited or noisy data. Significant efforts have been dedicated to the development of asymptotically efficient or approximate computational methods for modeling with Gaussian processes. However, Gaussian processes may also suffer from some computational challenges. When the number of observations N increases, the computational complexity for inference and learning grows significantly and incurs an $\mathcal{O}\left(N^3\right)$ computational cost, which is unfeasible for many modern problems [13]. Another limitation of Gaussian processes is the memory scaling $\mathcal{O}\left(N^2\right)$ in a direct implementation. To address these issues, various approximations and scalable algorithms, such as sparse Gaussian processes [14,15] and variational inference [16], have been developed to make Gaussian processes applicable to larger datasets.

Certain approximations, as demonstrated in [17,18], involve reduced-rank Gaussian processes that rely on approximating the covariance function. For example, Ref. [19] addressed the computational challenge of working with large-scale datasets by approximating the covariance matrix, which is often required for computations involving kernel methods. In addition, Ref. [20] proposed an FFT-based method for stationary covariances as a technique that leverages the Fast Fourier Transform (FFT) to efficiently compute and manipulate covariance functions in the frequency domain. The link between state space models (SSMs) and Gaussian processes was explored in [21]. This could avoid the cubic complexity in time using Kalman filtering inference methods [22]. Recently, Ref. [23] presented a novel method for approximating covariance functions as an eigenfunction expansion of the Laplace operator defined on a compact domain. More recently, Ref. [24] introduced a reduced-rank algorithm for Gaussian process regression with a numerical scheme.

In this paper, we consider a specific Karhunen–Loève expansion of a Gaussian process with many advantages over other low-rank compression techniques [25]. Initially, we express a Gaussian process as a series of basis functions and random coefficients. By selecting a subset of the most vital basis functions based on the dominant eigenvalues, the rank of the Gaussian process can be reduced. This proves especially advantageous when working with extensive datasets as it can alleviate computational and storage demands. By scrutinizing the eigenvalues linked to the eigenfunctions, one can gauge each eigenfunction’s contribution to noise. This insight can be used for noise modeling, estimation, and separation. The Karhunen–Loève expansion offers a natural framework for model selection and regularization in Gaussian process modeling. By truncating the decomposition to a subset of significant eigenfunctions, we can prevent overfitting. This regularization has the potential to enhance the Gaussian process generalization capability and mitigate the influence of noise or irrelevant features.

In contrast to conventional Karhunen–Loève expansions where eigenfunctions are derived directly from the covariance function or the integral operator, our approach involves differential operators in which the corresponding orthogonal polynomials serve as eigenfunctions. This choice holds significant importance because polynomials are tailored to offer numerical stability and good conditioning, resulting in more precise and stable computations, particularly when dealing with rounding errors. Additionally, orthogonal polynomials frequently possess advantageous properties for integration and differentiation. These properties streamline efficient calculations involving interpolated functions, making them exceptionally valuable for applications necessitating complex computations. On the whole, decomposing Gaussian processes using orthogonal polynomials provides benefits such as numerical stability, quicker convergence, and accurate approximations [26]. However, their incorporation within the Gaussian process framework of the machine learning community has been virtually non-existent. The existing research predominantly revolves around the analysis of integral operators and numerical approximations to compute Karhunen–Loève expansions.

The paper is structured as follows. The first three sections review the necessary theoretical foundations: Section 2 introduces operators in Hilbert spaces, the expansion theorem, and the general Karhunen–Loève theorem; Section 3 focuses on the Gaussian process case; and Section 4 highlights the challenges of canonical Gaussian process regression. Section 5 explores low-complexity Gaussian processes and their computational advantages. Section 6 presents the proposed solutions for various differential operators using orthogonal polynomial bases. Finally, Section 7 discusses the experimental results, followed by a comprehensive discussion and conclusion in Section 8.

2. Operators on Hilbert Spaces

In this section, we recall and prove some useful results about linear compact symmetric operators on the particular ${\mathbb{L}}^2$ Hilbert space. More general results of any Hilbert space H are moved to Appendix A.

Let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ be a probability space, and let X and Y be second-order real-valued random variables, meaning $\mathbb{E}\left(X^2\right)<\infty$ and $\mathbb{E}\left(Y^2\right)<\infty$. By the Cauchy–Schwarz inequality, $\mathbb{E}\left(XY\right)<\infty$, allowing us to center the variables, assuming without loss of generality that they have zero mean. Thus, $\mathbb{E}\left(X^2\right)$ is the variance of X and $\mathbb{E}\left(XY\right)$ is the covariance of X and Y. The set of these random variables forms a Hilbert space with inner product $\mathbb{E}\left(XY\right)$ and norm ${\left|\right|X\left|\right|}_2=\sqrt{\mathbb{E}\left(X^2\right)}$, leading to mean square convergence: $X_n$ converges in norm to X if $\left|\right|X_n-X{\left|\right|}_2\to 0$; equivalently, $\mathbb{E}\left({(X_n-X)}^2\right)\to 0$ as $n\to \infty$. X and Y are orthogonal, written $X\perp Y$, if $\mathbb{E}\left(XY\right)=0$. If $X\perp Y$, then $\mathbb{E}{(X+Y)}^2=\mathbb{E}\left(X^2\right)+\mathbb{E}\left(Y^2\right)$. For mutually orthogonal variables $X_1,\dots ,X_n$, we have $\mathbb{E}{\left(X_1+\cdots +X_n\right)}^2=\mathbb{E}\left(X_1^2\right)+\cdots +\mathbb{E}\left(X_n^2\right)$.

Lemma 1.

Let $X={\left\{X_t\right\}}_t$, $t\in I\subset {\mathbb{R}}^d$ be a real-valued second-order random process with zero mean and covariance function $k(t,s):=\mathbb{E}\left(X_t{X}_s\right)$. Then, the covariance k is a symmetric non-negative definite function.

Proof. 

First, by definition of $k(t,s):=\mathbb{E}\left(X_t{X}_s\right)$, its symmetry is trivial. Next, it holds that, for all possible choices of $t_1,\dots ,t_N\in I$, $N\in {\mathbb{N}}^{\ast }$ and all possible functions $\varphi :I\to \mathbb{R}$, we have

$$\begin{array}{ccc}\hfill \sum _{i=1}^N\sum _{l=1}^N\varphi (t_i)k(t_i,t_l)\varphi (t_l)& =& \sum _{i=1}^N\sum _{l=1}^N\varphi (t_i)\mathbb{E}(X_{t_i}{X}_{t_l})\varphi (t_l),\hfill \\ & =& \mathbb{E}\left(\sum _{i=1}^N\sum _{l=1}^N\varphi (t_i)X_{t_i}{X}_{t_l}\varphi (t_l)\right),\hfill \\ & =& \mathbb{E}\left({\left(\sum _{i=1}^N\varphi (t_i)X_{t_i}\right)}^2\right)\ge 0.\hfill \end{array}$$

Thus, k is non-negative definite.    □

In this section, we are interested in the Hilbert space $H:={\mathbb{L}}^2(I,\rho )$, the space of all the real-valued Borel-measurable functions $\varphi$ on the interval $I\subset {\mathbb{R}}^d$ such that ${\int }_I{\varphi }^2\left(t\right)\rho \left(t\right)dt<\infty$ with a positive weight function $\rho \left(t\right)$, which inherits all proprieties from Appendix A. Consider the Hilbert–Schmidt integral operator $\mathcal{K}:{\mathbb{L}}^2(I,\rho )\mapsto {\mathbb{L}}^2(I,\rho )$, expressed as

$$\begin{array}{c}\hfill \left(\mathcal{K}\varphi \right)\left(t\right):={\int }_{I}k(t,s)\varphi \left(s\right)\rho \left(s\right)ds.\end{array}$$

(3)

Theorem 1 (Mercer’s Theorem).

Let $k:I\times I\to \mathbb{R}$ be continuous symmetric non-negative definite and let $\mathcal{K}$ be the corresponding Hilbert–Schmidt operator. Let ${\left\{{\varphi }_j\right\}}_j$ be an orthonormal basis for the space spanned by the eigenvectors corresponding to the non-zero eigenvalues of $\mathcal{K}$. If ${\varphi }_j$ is the eigenvector corresponding to the eigenvalue ${\lambda }_j$, then

$$k(t,s)=\sum _{j=1}^{\infty }{\lambda }_j{\varphi }_j\left(t\right){\varphi }_j\left(s\right),$$

(4)

where

(i) 

the series converges absolutely in both variables jointly.

(ii) 

the series converges to $k(t,s)$ uniformly in both variables jointly.

(iii) 

the series converges to $k(t,s)$ in ${\mathbb{L}}^2(I\times I,\rho )$.

Theorem 2 (Karhunen–Loève Theorem).

For a real-valued second-order random process $X={\left\{X_t\right\}}_t$ with zero mean and continuous covariance function $k(t,s)$ on $I\subset {\mathbb{R}}^d$, we can decompose each $X_t$ as

$$X_t=\sum _{j=1}^{\infty }{a}_j{\varphi }_j\left(t\right),$$

(5)

where

(i) 

${\left\{{\varphi }_j\right\}}_j$ are eigenfunctions of $\mathcal{K}$, which form an orthonormal basis, i.e., ${\int }_I{\varphi }_j\left(t\right){\varphi }_l\left(t\right)\rho \left(t\right)dt={\delta }_{jl}$, where ${\delta }_{jl}$ is the Kronecker’s delta.

(ii) 

$a_j={\int }_I{X}_t{\varphi }_j\left(t\right)\rho \left(t\right)dt$ is the coefficient given by the projection of $X_t$ onto the j-th deterministic element of the Karhunen–Loève basis in ${\mathbb{L}}^2(\mathrm{\Omega },\mathcal{F},\mathbb{P})$.

(iii) 

${\left\{a_j\right\}}_j$ are pairwise orthogonal random variables with zero mean and variance ${\lambda }_j$, corresponding to the eigenvalue of the eigenfunction ${\varphi }_j$.

Moreover, the series ${\sum }_{j=1}^{\infty }{a}_j{\varphi }_j\left(t\right)$ converges to $X_t$ in mean square uniformly for all $t\in I$.

Proof. 

We have $\mathbb{E}\left(a_j\right)=0$ and $\mathbb{E}\left(a_j{a}_l\right)={\lambda }_l{\delta }_{jl}$ due to orthonormality of ${\left\{{\varphi }_j\right\}}_j$. Thus, ${\left\{a_j\right\}}_j$ are pairwise orthogonal in ${\mathbb{L}}^2(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ and $v\left(a_j\right)={\lambda }_j$. To show the mean square convergence, let $S_t^m:={\sum }_{j=1}^m{a}_j{\varphi }_j\left(t\right)$ and ${\epsilon }_t^m:=S_t^m-X_t$. Then,

$$\begin{array}{ccc}\hfill \mathbb{E}\left({\left({\epsilon }_t^m\right)}^2\right)& =& \sum _{j=1}^m{\lambda }_j{\varphi }_j^2\left(t\right)-2\sum _{j=1}^m{\varphi }_j\left(t\right)\mathbb{E}\left(a_j{X}_t\right)+k(t,t),\hfill \\ & =& k(t,t)-\sum _{j=1}^m{\lambda }_j{\varphi }_j^2\left(t\right)\underset{m\to \infty }{\to }0,\hfill \end{array}$$

uniformly in $t\in I$ by Mercer’s theorem.    □

Definition 1.

The mean square error is defined as $\mathbb{E}\left({\left({\epsilon }_t^m\right)}^2\right)$. The mean integrated square error (MISE), denoted by ${\xi }_m$, is then given by ${\xi }^m:={\int }_I\mathbb{E}\left({\left({\epsilon }_t^m\right)}^2\right)\rho \left(t\right)dt$, representing the mean square error integrated over the basis ${\left\{{\varphi }_j\right\}}_j$ onto which $X_t$ is projected for every $t\in I$.

Proposition 1.

The MISE ${\xi }^m$ tends to 0 as $m\to \infty$.

Proof. 

The MISE of $S_t^m$ satisfies

$$\begin{array}{ccc}\hfill {\xi }^m& =& {\int }_I\mathbb{E}({({\epsilon }_t^m)}^2)\rho (t)dt,\hfill \\ & =& \mathbb{E}({\int }_I(\sum _{j=m+1}^{\infty }{a}_j{\varphi }_j{(t))}^2\rho (t)dt),\hfill \\ & =& \sum _{j=m+1}^{\infty }\mathbb{E}(a_j^2),{\int }_I{\varphi }_j{(t)}^2\rho (t)dt,\hfill \\ & =& \sum _{j=m+1}^{\infty }\mathbb{E}(a_j^2),\hfill \\ & =& \sum _{j=m+1}^{\infty }{\lambda }_j,\hfill \end{array}$$

which tends to 0 as $m\to \infty$ since ${\lambda }_j$ are absolutely summable.    □

Now, we highlight the crucial role of the Karhunen–Loève basis in minimizing the error incurred by truncating the expansion of $X_t$. By aligning the basis functions with the dominant modes of variation captured by the process, we achieve an optimal representation in terms of mean integrated square error.

Proposition 2.

The MISE ${\xi }^m$ is minimized if and only if ${\left\{{\varphi }_j\right\}}_j$ constitutes an orthonormalization of the eigenfunctions of the Fredholm equation

$$\left(\mathcal{K}\varphi \right)\left(t\right)=\lambda \varphi \left(t\right),$$

(6)

with ${\left\{{\varphi }_j\right\}}_j$ arranged to correspond to the eigenvalues ${\left\{{\lambda }_j\right\}}_j$ in decreasing magnitude: ${\lambda }_1>{\lambda }_2>\cdots >0$.

3. Expansion and Convergence of Gaussian Processes

A stochastic process $f={\left\{f\left(t\right)\right\}}_t$ is said to be a Gaussian process (GP) if for all positive integers $N\in {\mathbb{N}}^{\ast }$ and all choices of $t_1,\dots ,t_N$ the random variables $f\left(t_1\right),\dots ,f\left(t_N\right)$ form a Gaussian random vector, which means that they are jointly Gaussian. One of the main advantages of a GP is that it can be represented as a series expansion involving a complete set of deterministic basis functions with corresponding random Gaussian coefficients.

Theorem 3.

If $f={\left\{f\left(t\right)\right\}}_t$, $t\in I\subset {\mathbb{R}}^d$ is a zero-mean GP of covariance $k(t,s)$ denoted $f\left(t\right)\sim \mathcal{GP}(0,k(t,s\left)\right)$, then the Karhunen–Loève expansion projections $a_j$ are independent Gaussian random variables: $a_j\sim \mathcal{N}(0,{\lambda }_j)$.

Now, we provide a result that establishes important convergence properties, connecting mean square convergence, convergence in probability, and convergence almost surely for sequences of random variables.

Lemma 2.

1.  The mean square convergence of any sequence ${\left\{a_n\right\}}_n$ of real-valued random variables implies its convergence in probability.

2. 

If ${\left\{a_n\right\}}_n$ is a sequence of independent real-valued random variables, then the convergence of the series ${\sum }_{n=1}^{\infty }{a}_n$ in probability implies its convergence almost surely.

Corollary 1.

For each $t\in I$, ${\sum }_{j=1}^{\infty }{a}_j{\varphi }_j\left(t\right)$ converges to $f\left(t\right)$ almost surely.

Proof. 

Since ${\left\{a_j\right\}}_j$ are independent, ${\left\{a_j{\varphi }_j\left(t\right)\right\}}_j$ are also independent because the eigenfunctions are deterministic. Thus, Lemma 2 can be applied to the series of independent random variables ${\sum }_{j=1}^{\infty }{a}_j{\varphi }_j\left(t\right)$. This means that its convergence in probability implies almost sure convergence. Therefore, we only need to prove convergence in probability. This argument is straightforward because, by Lemma 2, convergence in mean square implies convergence in probability. The mean square convergence of the partial sums $f^m\left(t\right):={\sum }_{j=1}^m{a}_j{\varphi }_j\left(t\right)$ is a result of the Karhunen–Loève theorem.    □

Example 1.

1.  The Karhunen–Loève expansion of the Brownian motion on $I=[0,1]$ as a centered GP with covariance $k(t,s)=min(t,s)$ is given by

$$f\left(t\right)={\displaystyle \frac{2\sqrt{2}}{\pi }}\sum _{j=1}^{\infty }{\displaystyle \frac{a_j^{\ast }}{2j-1}}sin\left({\displaystyle \frac{(2j-1)\pi }{2}}t\right),$$

where $a_j^{\ast }={\displaystyle \frac{a_j}{\sqrt{{\lambda }_j}}}$ and ${\lambda }_j={\displaystyle \frac{4}{{(2j-1)}^2{\pi }^2}}$.

2. 

The Karhunen–Loève expansion of the Brownian bridge on $I=[0,1]$ as a centered GP with covariance $k(t,s)=min(t,s)-ts$ is given by

$$f\left(t\right)={\displaystyle \frac{\sqrt{2}}{\pi }}\sum _{j=1}^{\infty }{\displaystyle \frac{a_j^{\ast }}{j}}sin\left(j\pi t\right),$$

where $a_j^{\ast }=j\pi a_j$ and ${\lambda }_j={\displaystyle \frac{1}{j^2{\pi }^2}}$.

4. Ill-Conditioned Canonical Gaussian Process Regression

In a regression task, a nonparametric function f is assumed to be a realization of a stochastic GP prior, whereas the likelihood term holds from observations corrupted by a noise term according to the canonical form

$$\begin{array}{cc}\hfill y_i& =f\left(t_i\right)+{\tau }_i;i=1,\dots ,N,\hfill \\ \hfill f\left(t\right)& \sim \mathcal{GP}(0,k(t,s\left)\right),\hfill \end{array}$$

(7)

where ${\tau }_i\sim \mathcal{N}(0,{\sigma }_N^2)$ is a Gaussian noise. Given a training dataset $\mathcal{D}=(\mathbf{t},\mathbf{y})={(t_i,y_i)}_{i=1}^N$, the posterior distribution over $\mathbf{f}=f\left(\mathbf{t}\right)={(f\left(t_1\right),\dots ,f\left(t_N\right))}^{\top }$ is also Gaussian: $\mathbb{P}\left(\mathbf{f}\right|\mathcal{D})=\mathcal{N}(\mu ,\Sigma )$. From Bayes’ rule, we state that the mean and the covariance posterior are expressed as

$$\begin{array}{ccc}\hfill \mu & =& \mathbf{K}{(\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N)}^{-1}\mathbf{y},\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill \Sigma & =& {({\mathbf{K}}^{-1}+{\displaystyle \frac{1}{{\sigma }_N^2}}{\mathcal{I}}_N)}^{-1},\hfill \end{array}$$

(9)

where $\mathbf{K}={\left[k(t_i,t_j)\right]}_{i,j=1}^N$ is the prior covariance matrix and ${\mathcal{I}}_N$ is the $N\times N$ identity matrix. The predictive distribution at any test input $t_{★}$ can be computed in closed form as $f\left(t_{★}\right)|\mathcal{D},t_{★}\sim \mathcal{N}\left({\overline{f}}_{★},v\left(f_{★}\right)\right),$ with

$$\begin{array}{cc}\hfill {\overline{f}}_{★}& =\mathbf{k}{\left(t_{★}\right)}^{\top }{(\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N)}^{-1}\mathbf{y},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill v\left(f_{★}\right)& =k(t_{★},t_{★})-\mathbf{k}{\left(t_{★}\right)}^{\top }{(\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N)}^{-1}\mathbf{k}\left(t_{★}\right),\hfill \end{array}$$

(11)

where $\mathbf{k}\left(t_{★}\right)={\left[k(t_i,t_{★})\right]}_{i=1}^N$.

The covariance function k usually depends on a set of hyperparameters denoted ${\theta }_k$ that need to be estimated from the training dataset. The log marginal likelihood for GP regression serves as an indicator of the degree to which the selected model accurately captures the observed patterns. The log marginal likelihood is typically used for model selection and optimization. Let $\theta =({\theta }_k,{\sigma }_N^2)$ denote the set of all model parameters, and then the log marginal likelihood $log\mathbb{P}\left(\mathbf{y}\right|\mathbf{t},\theta )$ is given by

$$\begin{array}{c}\hfill l\left(\theta \right)=-{\displaystyle \frac{1}{2}}log|\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N|-{\displaystyle \frac{1}{2}}{\mathbf{y}}^{\top }{(\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N)}^{-1}\mathbf{y}-{\displaystyle \frac{N}{2}}log\left(2\pi \right).\end{array}$$

(12)

Here, $|.|$ denotes the determinant. The goal is to estimate $\theta$ that maximizes the log marginal likelihood. This can be achieved using different methods, such as gradient-based algorithm [27]. The weakness of inferring the posterior mean or the mean prediction or even learning the hyperparameters from the log marginal likelihood is the need to invert the $N\times N$ Gram matrix $\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N$. This operation costs $\mathcal{O}\left(N^3\right)$, which limits the applicability of standard GPs when the sample size N increases significantly. Furthermore, the memory requirements for GP regression scale with a computational complexity of $\mathcal{O}\left(N^2\right)$.

A covariance function $k(t,s)$ is said to be stationary (isotropic) if it is invariant to translation, i.e., a function of $\left|\right|t-s\left|\right|$ only. Two commonly used stationary covariance functions for GP regression are the Squared Exponential (SE) and Matérn-$\nu$ kernels defined by

$$\begin{array}{ccc}\hfill k(t,s)& =& {\sigma }^2{e}^{-{\epsilon }^2{\parallel t-s\parallel }^2};t,s\in \mathbb{R}\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill k(t,s)& =& {\sigma }^2{\displaystyle \frac{2^{1-\nu }}{\Gamma \left(\nu \right)}}{\left(\epsilon \sqrt{2\nu }\parallel t-s\parallel \right)}^{\nu }{K}_{\nu }\left(\epsilon \sqrt{2\nu }\parallel t-s\parallel \right);t,s\in \mathbb{R}\hfill \end{array}$$

(14)

respectively, where ${\sigma }^2$ is the variance parameter controlling the amplitude of the covariance, $\epsilon$ is the shape parameter, and $\nu =k+1/2$; $k\in \mathbb{N}$ is the half integer smoothness parameter controlling its differentiability. Here, $\Gamma$ is the gamma function and $K_{\nu }$ is the modified Bessel function of the second kind. Both the SE and Matérn covariance functions have hyperparameters that needs to be estimated from the data during the model training process. A GP with a Matérn-$\nu$ covariance is ${\displaystyle \lceil \nu \rceil -1}$ times differentiable in the mean-square sense. The SE covariance is the limit of Matérn-$\nu$ as the smoothness parameter $\nu$ approaches infinity. When choosing between the SE and Matérn covariance functions, it is often a matter of balancing the trade-off between modeling flexibility and computational complexity. The SE covariance function is simpler and more computationally efficient but may not capture complex patterns in data as well as the Matérn covariance function with an appropriate choice of smoothness parameter.

In order to compute (10), we also need to invert the $N\times N$ Gram matrix $\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N$. This task is impractical when the size sample N is large because inverting the matrix leads to $\mathcal{O}\left(N^2\right)$ memory and $\mathcal{O}\left(N^3\right)$ time complexities [28]. There are several methods to overcome this difficulty. For instance, the variational inference proceeds by introducing n inducing points and corresponding n inducing variables. The variational parameters of inducing variables are learned by minimizing the Kullback–Leibler divergence [16]. Picking $n\ll N$, the complexity reduces to $\mathcal{O}(N{n}^2+n^3)$ in prediction and $\mathcal{O}\left(n^2\right)$ in minimizing the Kullback–Leibler divergence. The computational complexity of conventional sparse Gaussian process (SGP) approximations typically scales as $\mathcal{O}\left(N{n}^2\right)$ in time for each step of evaluating the marginal likelihood [14]. The storage demand scales as $\mathcal{O}\left(Nn\right)$. This arises from the unavoidable cost of re-evaluating all results involving the basis functions at each step and the need to store the matrices required for these calculations.

5. Low-Complexity Gaussian Process Regression

In order to avoid the inversion of the $N\times N$ Gram matrix $\mathbf{K}+{\sigma }_N^2{\mathcal{I}}_N$, we use the approximation scheme presented in Section 3 and rewrite the GP with a truncated set of m basis functions. Hence, the truncated f at an arbitrary order $m\in {\mathbb{N}}^{\ast }$ is given by

$$\begin{array}{c}\hfill f^m\left(t\right):=\sum _{j=1}^m{a}_j{\varphi }_j\left(t\right),\end{array}$$

(15)

with an approximation error $e^m\left(t\right):={\sum }_{j=m+1}^{\infty }{a}_j{\varphi }_j\left(t\right)$. The canonical GP regression model (7) becomes

$$\begin{array}{cc}\hfill y_i& =f^m\left(t_i\right)+{\tau }_i;i=1,\dots ,N,\hfill \\ \hfill f^m\left(t\right)& \sim \mathcal{GP}(0,k^m(t,s)),\hfill \end{array}$$

(16)

with a covariance function approximated by $k^m(t,s)={\sum }_{j=1}^m{\lambda }_j{\varphi }_j\left(t\right){\varphi }_j\left(s\right)$. The convergence degree of the Mercer series in (4), that is, $k^m\underset{m\to \infty }{\to }k$ with $k(t,s)={\sum }_{j=1}^{\infty }{\lambda }_j{\varphi }_j\left(t\right){\varphi }_j\left(s\right),$ depends heavily on the eigenvalues and the differentiability of the covariance function. Ref. [29] showed that the speed of the uniform convergence varies in terms of the decay rate of eigenvalues and demonstrated that, for a $2\beta$ times differentiable covariance k, the truncated covariance $k^m$ approximates k as $\mathcal{O}({({\sum }_{j=m+1}^{\infty }{\lambda }_j)}^{{\displaystyle \frac{\beta }{\beta +1}}})$. For infinitely differentiable covariances, the latter is $\mathcal{O}({({\sum }_{j=m+1}^{\infty }{\lambda }_j)}^{1-ϵ})$ for any $ϵ>0$. To summarize, smoother covariance functions tend to exhibit faster convergence, while less smooth or non-differentiable covariance functions may exhibit slower or no convergence.

The resulting covariance falls into the class of reduced-rank approximations based on approximating the covariance matrix $\mathbf{K}$ with a matrix $\tilde{\mathbf{K}}={\left[k^m(t_i,t_j)\right]}_{i,j=1}^N=\mathbf{\Phi }\mathbf{\Gamma }{\mathbf{\Phi }}^{\top }$, where $\mathbf{\Gamma }$ is an $m\times m$ diagonal matrix with eigenvalues such that ${\mathbf{\Gamma }}_{jj}={\lambda }_j$ and $\mathbf{\Phi }$ is an $N\times m$ matrix with eigenfunctions such that ${\mathbf{\Phi }}_{ij}={\varphi }_j\left(t_i\right)$. Note that the approximate covariance matrix $\tilde{\mathbf{K}}$ becomes ill-conditioned when the ratio ${\lambda }_1/{\lambda }_m$ is large. This ill-conditioning occurs particularly when the observation points $t_i$ are too close to each other [30]. In practice, this can lead to significant numerical errors when inverting $\tilde{\mathbf{K}}$, resulting in unstable solutions, amplified errors in parameter estimation, and unreliable model predictions.

Now, we show how the approximated regression models make use of GP decomposition to achieve low complexity. We write down the expressions needed for inference and discuss the computational requirements. Applying the matrix inversion lemma [31], we rewrite the predictive distribution (10) and (11) as

$$\begin{array}{cc}\hfill {\overline{f}}_{★}& \approx {\varphi }_{★}^{\top }{({\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1})}^{-1}{\mathbf{\Phi }}^{\top }\mathbf{y},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill v\left(f_{★}\right)& \approx {\sigma }_N^2{\varphi }_{★}^{\top }{({\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1})}^{-1}{\varphi }_{★},\hfill \end{array}$$

(18)

where ${\varphi }_{★}$ is an m-dimensional vector with the j-th entry being ${\varphi }_j\left(t_{★}\right)$. When the number of observations is much higher than the number of required basis functions ($N\gg m$), the use of this approximation is advantageous. Thus, any prediction mean evaluation is dominated by the cost of constructing ${\mathbf{\Phi }}^{\top }\mathbf{\Phi }$, which means that the method has an overall asymptotic computational complexity of $\mathcal{O}\left(N{m}^2\right)$.

The approximate log marginal likelihood satisfies

$$\begin{array}{cc}\hfill l\left(\theta \right)\approx & -{\displaystyle \frac{1}{2}}log|\mathbf{\Phi }\mathbf{\Gamma }{\mathbf{\Phi }}^{\top }+{\sigma }_N^2{\mathcal{I}}_N|-{\displaystyle \frac{1}{2}}{\mathbf{y}}^{\top }{(\mathbf{\Phi }\mathbf{\Gamma }{\mathbf{\Phi }}^{\top }+{\sigma }_N^2{\mathcal{I}}_N)}^{-1}\mathbf{y}-{\displaystyle \frac{N}{2}}log\left(2\pi \right),\hfill \\ \hfill =& -{\displaystyle \frac{1}{2}}(N-m)log{\sigma }_N^2-{\displaystyle \frac{1}{2}}log|{\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1}|-{\displaystyle \frac{1}{2}}\sum _{j=1}^{m}log{\lambda }_j\hfill \\ \hfill & -{\displaystyle \frac{1}{2{\sigma }_N^2}}({\mathbf{y}}^{\top }\mathbf{y}-{\mathbf{y}}^{\top }\Phi {({\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1})}^{-1}{\mathbf{\Phi }}^{\top }\mathbf{y})-{\displaystyle \frac{N}{2}}log\left(2\pi \right).\hfill \end{array}$$

(19)

Consequently, evaluating the approximate log marginal likelihood has a complexity of $\mathcal{O}\left(m^3\right)$ needed to inverse the $m\times m$ matrix $\mathbf{M}={\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1}$. In practice, if the sample size N is large, it is preferable to store the result of ${\mathbf{\Phi }}^{\top }\mathbf{\Phi }$ in memory, leading to a memory requirement that scales as $\mathcal{O}\left(m^2\right)$. For efficient implementation, matrix-to-matrix multiplications can often be avoided, and the inversion of $\mathbf{M}$ can be performed using Cholesky factorization for numerical stability. This factorization ($\mathbf{L}{\mathbf{L}}^{\top }=\mathbf{M}$) can also be used to compute the term $log\left|\mathbf{M}\right|=2{\sum }_{j=1}^{m}log{\mathbf{L}}_{jj}$. Algorithm 1 outlines the main steps for estimating the hyperparameters of the low-complexity GP.

Algorithm 1 Gradient Ascent for Hyperparameter Learning.

Require:  Data $\mathcal{D}=(\mathbf{t},\mathbf{y})$, initial hyperparameters $\theta =({\theta }_k,{\sigma }_N^2)$, learning rate $\eta$, tolerance $ϵ$, maximum iterations T. Ensure:  Optimized hyperparameters ${\theta }^{\ast }$. 1: Initialize $\theta ={\left({\theta }_j\right)}_j$. 2: for $t=1$ to T do 3:     Construct $\mathbf{\Phi }$ and $\mathbf{\Gamma }$ depending on $\theta$. 4:     Compute $\mathbf{M}={\mathbf{\Phi }}^{\top }\mathbf{\Phi }+{\sigma }_N^2{\mathbf{\Gamma }}^{-1}$. 5:     Evaluate $\mathbf{L}=\mathrm{Cholesky}\left(\mathbf{M}\right)$. 6:     for each hyperparameter ${\theta }_j$ do 7:         Compute gradient $g_j=\mathrm{function}\left(\mathbf{M}\right)$. 8:         Update hyperparameter: ${\theta }_j\leftarrow {\theta }_j+\eta g_j$. 9:     end for 10:     if $\parallel \nabla l\left(\theta \right)\parallel <ϵ$ then 11:         break 12:     end if 13: end for 14: return optimized hyperparameters ${\theta }^{\ast }=({\theta }_k^{\ast },{\sigma }_N^{2,\ast })$.

6. Closed Solutions from Differential Operators

In this section, we highlight the connection between Hilbert–Schmidt integral operators in (3) and differential operators. Thus, we describe explicit solutions for the low complexity GP (LCGP) with covariances derived from differential operators. Unlike previous works where the eigen-decomposition is determined from the Mercer series or the integral operator directly, this paper focuses on constructing covariance functions that incorporate orthogonal polynomials as eigenfunctions. It is worth noting that polynomials can approximate a wide range of functions with various degrees of complexity. They can be adjusted to predict different data patterns and capture both linear and non-linear relationships [32].

The connection between a differential operator denoted $\mathcal{L}$ and the integral operator $\mathcal{K}$ has been largely used. We follow the same idea in [33,34,35] and define Green’s function G of the differential operator $\mathcal{L}$ as its “right inverse”, i.e.,

$$\begin{array}{c}\hfill \left(\mathcal{L}G\right)(t,s)=\delta (t-s);t,s\in I,\end{array}$$

(20)

where $\delta (.)$ denotes the Dirac delta function.

Theorem 4.

Let $k:I\times I\to \mathbb{R}$ be continuous symmetric non-negative definite and let $\mathcal{K}$ be the corresponding Hilbert–Schmidt operator. Let ${\left\{{\varphi }_j\right\}}_j$ be an orthonormal basis for the space spanned by the eigenfunctions corresponding to the non-zero eigenvalues of $\mathcal{K}$. If ${\varphi }_j$ is the eigenfunction associated with the eigenvalue ${\lambda }_j$ and the covariance function acts as a Green’s function of a differential operator $\mathcal{L}$, then the eigenvalues of $\mathcal{K}$ correspond to reciprocal eigenvalues of $\rho {\left(t\right)}^{-1}\mathcal{L}$, while the corresponding eigenfunctions are still the same.

Proof. 

We have

$$\begin{array}{ccc}\hfill {\lambda }_j\left(\mathcal{L}{\varphi }_j\right)\left(t\right)& =& \mathcal{L}\left(\mathcal{K}{\varphi }_j\right)\left(t\right),\hfill \\ & =& \mathcal{L}{\int }_{I}k(t,s){\varphi }_j\left(s\right)\rho \left(s\right)ds,\hfill \\ & =& {\int }_I\mathcal{L}k(t,s){\varphi }_j\left(s\right)\rho \left(s\right)ds,\hfill \\ & =& {\int }_I\delta (t-s){\varphi }_j\left(s\right)\rho \left(s\right)ds,\hfill \\ & =& {\varphi }_j\left(t\right)\rho \left(t\right).\hfill \end{array}$$

Finally, we obtain

$$\begin{array}{c}\hfill {\rho }^{-1}\left(t\right)\left(\mathcal{L}{\varphi }_j\right)\left(t\right)={\displaystyle \frac{1}{{\lambda }_j}}{\varphi }_j\left(t\right),\end{array}$$

which completes the proof. □

At this stage, we compute eigenvalues and eigenfunctions of $\rho {\left(t\right)}^{-1}\mathcal{L}$, from which we deduce the Mercer decomposition of $k(t,s)$ given in (4) replacing ${\lambda }_j$ by ${\gamma }_j={\displaystyle \frac{1}{{\lambda }_j}}$. We select a list of some interesting and useful differential operators that act on ${\mathbb{L}}^2(I,\rho )$, for example: Matérn, Legendre, Laguerre, Hermite, Chebyshev, and Jacobi, from which we explicitly find the corresponding eigen-decompositions.

Table 1. Various differential operators and their corresponding decompositions.

Operator	$\mathcal{L}$	I	$\mathit{\rho }$	${\mathit{\gamma }}_{\mathit{j}}$	${\mathit{\varphi }}_{\mathit{j}}\left(\mathit{t}\right)$	$\parallel {\mathit{\varphi }}_{\mathit{j}}{\parallel }_{{\mathbb{L}}^2}$	MISE
Matérn 1	${\sigma }^{-2}{\left(\epsilon -{\displaystyle \frac{d^2}{d{t}^2}}\right)}^{\alpha }$	$[0,1]$	1	${\sigma }^{-2}{\left(\epsilon +j^2{\pi }^2\right)}^{\alpha }$	$\sqrt{2}sin\left(j\pi t\right)$	1	${\sigma }^2{\sum }_{j=m+1}^{\infty }{\left(\epsilon +j^2{\pi }^2\right)}^{-\alpha }$
Legendre	$-(1-t^2){\displaystyle \frac{d^2}{d{t}^2}}+2t{\displaystyle \frac{d}{dt}}$	$[-1,1]$	1	$j(j+1)$	${\displaystyle \frac{1}{2^{j}j!}}{\displaystyle \frac{d^j}{d{t}^j}}{(t^2-1)}^j$	$\sqrt{{\displaystyle \frac{2}{2j+1}}}$	${\displaystyle \frac{1}{m+1}}$
Laguerre	${\left(t{\displaystyle \frac{d^2}{d{t}^2}}+(1-t){\displaystyle \frac{d}{dt}}\right)}^2$	$[0,\infty )$	$e^{-t}$	$j^2$	${\displaystyle \frac{e^t}{j!}}{\displaystyle \frac{d^j}{d{t}^j}}\left(e^{-t}{t}^j\right)$	1	${\displaystyle \frac{{\pi }^2}{6}}-{\sum }_{j=1}^m{\displaystyle \frac{1}{j^2}}$
Hermite	${\left({\displaystyle \frac{d^2}{d{t}^2}}-2t{\displaystyle \frac{d}{dt}}\right)}^2$	$\mathbb{R}$	$e^{-t^2}$	$4j^2$	${(-1)}^j{e}^{t^2}{\displaystyle \frac{d^j}{d{t}^j}}{e}^{-t^2}$	$\sqrt{\sqrt{\pi }{2}^{j}j!}$	${\displaystyle \frac{1}{4}}\left({\displaystyle \frac{{\pi }^2}{6}}-{\sum }_{j=1}^m{\displaystyle \frac{1}{j^2}}\right)$
Chebyshev	${\left(-(1-t^2){\displaystyle \frac{d^2}{d{t}^2}}+t{\displaystyle \frac{d}{dt}}\right)}^2$	$[-1,1]$	${\displaystyle \frac{1}{\sqrt{1-t^2}}}$	$j^2$	$cos(jarccost)$	$\sqrt{{\displaystyle \frac{\pi }{2}}}$	${\displaystyle \frac{{\pi }^4}{90}}-{\sum }_{j=1}^m{\displaystyle \frac{1}{j^4}}$
Jacobi	${\mathcal{L}}_J$	$[-1,1]$	${(1-t)}^{\alpha }{(1+t)}^{\beta }$	$j(j+\alpha +\beta +1)$	$J_j^{\alpha ,\beta }\left(t\right)$	see [36]	${\sum }_{j=m+1}^{\infty }{\displaystyle \frac{1}{j(j+\alpha +\beta +1)}}$

¹ Matérn hyperparameters: ${\sigma }^2$ for variance, $\epsilon$ for shape, and $\alpha$ for smoothness: $\alpha =\nu +1/2=k+1$; $k\in \mathbb{N}$.

summarizes each class of $\mathcal{L}$, the index set I, the weight function $\rho$, the eigenvalues ${\gamma }_j$, the eigenfunctions as polynomials ${\varphi }_j$, and the resulting MISE. Note that, for Laguerre, Hermite, and Chebyshev polynomials, the eigenvalues ${\gamma }_j$ of $\mathcal{L}$ were initially negatives. Therefore, we consider the iterated operator ${\mathcal{L}}^2:=\mathcal{L}\circ \mathcal{L}$ with squared eigenvalues and unchanged eigenfunctions. Further, for Legendre, Hermite, and Chebyshev, we state that $\parallel {\varphi }_j{\parallel }_{{\mathbb{L}}^2}\ne 1$, which means that ${\varphi }_j$ should be normalized to form an orthonormal basis. For Jacobi, the differential operator is ${\mathcal{L}}_J=(t^2-1){\displaystyle \frac{d^2}{d{t}^2}}+\left[\alpha -\beta +(\alpha +\beta +2)t\right]{\displaystyle \frac{d}{dt}}$ with parameters $\alpha ,\beta$ greater than $-1$. More details about the corresponding eigenfunctions $J_j^{\alpha ,\beta }$ and the ${\mathbb{L}}^2$-norm are provided in [36].

Figure 1 illustrates the behavior of the eigenvalues ${\lambda }_j={\displaystyle \frac{1}{{\gamma }_j}}$ of the integral operator $\mathcal{K}$ as the index j varies from 1 to 40. This is attributed to the smoothness of the true covariance as m is growing. Figure 2 visually depicts several GP realizations across various differential operator settings.

7. Experiments

In this section, we assess the effectiveness of the proposed LCGP by conducting evaluations on multiple datasets and reporting comparisons with some state-of-the-art methods. The comparative analysis will enable us to gain insights into the strengths and weaknesses of the proposed framework and determine its competitiveness. Here is an overview of the provided Python (version 3.12.4, packaged by Anaconda) code (https://github.com/anisfradi/Low-Complexity-Regression-Models-with-Gaussian-Process-Prior.git, accessed on 3 April 2025). The computations were executed on a computer with 125 GB of memory and a Xeon(R) W 2275 CPU @ 3.30 GHz. Throughout all experiments, we set the truncation parameter m to 25.

7.1. Simulations

In simulation study, we examine two functions: a parametric beta density function represented by $f\left(t\right)=\mathcal{B}\left(t\right|a=2,b=5)$ (Simulation 1) and a nonparametric quasi-periodic function satisfying $f\left(t\right)=tsin\left(10t\right)$ (Simulation 2). Both functions are defined on the unit interval $I=[0,1]$. For these experiments, we generated a total of 140 observations. Out of these, we allocated 40 observations for training and the remaining for testing. The input points $t_i$ are uniformly distributed on $[0,1]$. To introduce variability and simulate real-world conditions, each observed point was calculated as $y_i=f\left(t_i\right)+{\tau }_i$, where ${\tau }_i$ represents Gaussian noise drawn from $\mathcal{N}(0,{\sigma }_N^2=0.1)$. This procedure allows us to evaluate the performance of our models using noisy data.

In Figure 3, we show an illustration of predicting the true function from simulations. We observe that different types of polynomial eigenfunctions have distinct advantages in prediction with truncated GP. Matérn (MGP) and Legendre (LGP) are well suited for functions with exponential decay, making them suitable for decay processes. They also incorporate function values at data points, allowing for accurate predictions even with rapidly changing functions and effective in approximating functions with oscillatory behavior. Hermite (HGP) and Chebyshev (CGP) are accurate with more slowly varying processes. Jacobi (JGP) accurately fits functions passing through given data points. The performance of the Laguerre operator is comparatively weaker, leading to its exclusion from the experimental analysis. Figure 4 illustrates the uncertainty level of the proposed MGP when applied to the predicted data.

7.2. Real Data

In this part, we conduct a real study using two challenging datasets. The first dataset comprises more than $\mathrm{10,000}$ observations collected by the California Cooperative Oceanic Fisheries Investigations (CalCOFI) (https://www.kaggle.com/datasets/sohier/calcofi?resource=download, accessed on 3 April 2025). It investigates the ecological aspects surrounding the collapse of the sardine population of the coast of California, which is recognized as the longest and most comprehensive time series of oceanographic and larval fish data worldwide. It encompasses abundance data for over 250 fish species’ larvae, as well as larval length frequency data, egg abundance data for important commercial species, and oceanographic data. Data collected at depths up to 500 meters include temperature, salinity, oxygen, phosphate, silicate, nitrate and nitrite, chlorophyll, phytoplankton biodiversity, etc. In this experiment, we are specifically targeting climate change indicators on the California coast when we focus on data illustrating the temperature (°C) as function of the salinity (ppt). Some examples ($N=500$) are given in Figure 5 (left).

The second dataset used in this study pertains to Medical Cost Personal (mCP) (https://www.kaggle.com/datasets/mirichoi0218/insurance/discussion, accessed on 3 April 2025, which was sourced from demographic statistics provided by the US Census Bureau. It primarily focuses on the cost of treatment, which is influenced by various factors, including age, sex, body mass index (BMI), and smoking status. Specifically, this paper examines the relationship between treatment costs (charges in thousand dollars) and the BMI factor for both smokers and non-smokers. See some examples for smokers ($N=532$) and non-smokers ($N=137$) in Figure 5 (right). For both real datasets, a random split of $50\%$ is designated for training, with the remaining for testing. This strategy ensures a balanced distribution of data for model training and facilitates a comprehensive assessment of model performance. We compare our proposed LCGP with several baseline methods to determine if there are significant performance differences. The baseline methods include (i) the standard GP (std-GP) with a Matérn covariance function (as described in Section 4 and (ii) the sparse GP (SGP) [14]. The standard GP model was implemented in the scikit-learn library (https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html, accessed on 3 April 2025), and the sparse GP was implemented in GPy (https://nbviewer.org/github/SheffieldML/notebook/blob/master/GPy/sparse_gp_regression.ipynb, accessed on 3 April 2025). To evaluate the performance of the proposed methods, two commonly used metrics include the following:

• ISE: The integrated squared error as the average squared difference between the predicted function and the true function.
• R-squared: The coefficient of determination as a statistical measure used in regression analysis that represents the proportion of the variance in the dependent variable that is predictable from the independent variable.

Table 2. Results on the CalCOFI dataset. Optimal values across all methods are highlighted in red.

Method	ISE	R-Squared	Time (s)
MGP	1.5186	0.8588	0.0048
LGP	1.4988	0.8607	0.0033
HGP	1.5095	0.8597	0.0059
CGP	1.4991	0.8606	0.0022
JGP	1.4992	0.8606	0.0171
std-GP (standard)	1.4997	0.8606	0.2259
SGP (sparse)	1.5635	0.8547	0.1489

,

Table 3. Results on the mCP non-smoke dataset. Optimal values across all methods are highlighted in red.

Method	ISE	R-Squared	Time (s)
MGP	0.2556	−0.0029	0.0042
LGP	0.2568	−0.0076	0.0046
HGP	0.2578	−0.0117	0.0045
CGP	0.2572	−0.0091	0.0023
JGP	0.2570	−0.0086	0.0175
std-GP (standard)	0.2570	−0.0086	0.5880
SGP (sparse)	0.2569	−0.0083	0.1803

and

Table 4. Results on the mCP smoke dataset. Optimal values across all methods are highlighted in red.

Method	ISE	R-Squared	Time (s)
MGP	0.2152	0.7611	0.0018
LGP	0.1809	0.7992	0.0018
HGP	0.2518	0.7205	0.0020
CGP	0.1822	0.7977	0.0011
JGP	0.1836	0.7962	0.0139
std-GP (standard)	0.2018	0.7760	0.0288
SGP (sparse)	0.2227	0.7528	0.1696

present the prediction criteria obtained from real data and the computational time needed for this evaluation in seconds (s). Notably, Legendre exhibits slightly superior performance compared to other methods on CalCOFI and mCP smoker datasets, while Matérn emerges as the top performer on mCP non-smoker data. The number of inducing points is fixed to $n=10<m$ for the sparse GP. These results suggest that Legendre demonstrates greater adaptability in capturing a diverse array of patterns and structures within real data. Despite its equivalent complexity to other proposed models, the Chebyshev polynomial exhibits the shortest computational time. This efficiency can be attributed to a faster evaluation of the corresponding eigenfunctions. For further insights into the quality of predictions, refer to Figure 6 and Figure 7.

We also estimate the number of operations required for each approach. The sparse GP typically involves a computational complexity of $\mathcal{O}(N\times {10}^2)$, where N is the number of training points and $n=10$ the number of inducing points. This results in approximately $c_1\times N\times {10}^2$ operations, where $c_1$ is a constant depending on matrix operations. In contrast, our LCGP method operates with a complexity of $\mathcal{O}(N\times {25}^2)$, leading to $c_2\times N\times {25}^2$ operations, where $m=25$ is the number of basis functions and $c_2$ a corresponding constant.

To assess the robustness of our method, we have conducted a series of experiments on CalCOFI dataset by gradually increasing the sample size N to $\mathrm{12,000}$. The results shown in Figure 8 demonstrate that our methods are highly competitive in terms of accuracy: low ISE and high R-squared values. However, it is worth noting that the gap in computational time (measured in minutes) between the methods becomes significant.

8. Conclusions

In this paper, we have introduced a novel regression model with a Gaussian process prior. This nonparametric model is designed for inferring, predicting, and learning. Contrary to previous methods, the proposed one is derived from specific differential operators. We study and test different configurations with well adapted eigenfunctions’ bases, enabling straightforward implementations with closed-form expressions. In summary, a key advantage of our method is its ability to overcome the limitations associated with standard Gaussian processes. This includes reducing the computational cost to $\mathcal{O}\left(N{m}^2\right)$ for inference and $\mathcal{O}\left(m^3\right)$ for learning. We assess the effectiveness of our proposed model using a variety of simulated and real-world data. The experimental results and comparisons demonstrate its high accuracy, low computational overhead, and analytical simplicity in comparison to the existing methods.