Machine Learning in Least-Squares Monte Carlo Proxy Modeling of Life Insurance Companies

1. Introduction

The Solvency II directive of the European Parliament and European Council (2009) requires from insurance companies a derivation of the solvency capital requirement (SCR) using the full probability distributions of losses over a one-year period. Some life insurers comply with this requirement by setting up internal models. Other insurers opt for the much simpler standard formula, which enables an aggregation of the company’s exposures to single risks. Lacking an analytical valuation formula for the losses in a one-year period, life insurers with an internal model are supposed to utilize a Monte Carlo approach usually called nested simulations approach (Bauer et al. (2012)). In practice their cash-flow-projection (CFP) models need to be simulated several hundred thousand to several million times for a robust implementation of the nested simulations approach. But the insurers are currently far from being endowed with sufficient computational capacities to perform such expensive simulation tasks. By applying suitable approximation techniques like the least-squares Monte Carlo (LSMC) approach of Bauer and Ha (2015), the insurers are able to overcome these computational hurdles though. For example, they can implement the LSMC framework formalized by Krah et al. (2018) and applied by, for example, Bettels et al. (2014), to derive their full loss distributions. The central idea of this framework is to carry out a comparably small number of wisely chosen nested Monte Carlo simulations and to feed the simulation results into a supervised machine learning algorithm that translates the results into a proxy function of the insurer’s loss (output) with respect to the underlying risk factors (input).

Our starting point is the LSMC framework from Krah et al. (2018). In the following the same approach for the proxy derivation is assumed, we will only amend the calibration and validation steps. Therefore, we neither repeat the simulation setting nor the procedure for the full loss distribution forecast and SCR calculation here in detail. The purpose of this exposition is to introduce different machine learning methods that can be applied in the calibration step of the LSMC framework, to point out their similarities and differences and to compare their out-of-sample performances in the same slightly disguised real-world LSMC example already used in Krah et al. (2018).

We describe the data basis used for calibration and validation in Section 2.1, the structure of the calibration algorithm in Section 2.2 and our validation approach in Section 2.3. Our focus lies on out-of-sample performance rather than computational efficiency as the latter becomes only relevant if the former gives reason for it. We analyze a very realistic data basis with 15 risk factors and validate the proxy functions based on a very comprehensive and computationally expensive nested simulations test set comprising the SCR estimate.

The main idea of our approach is to combine different regression methods with an adaptive algorithm, in which the proxy functions are built up of basis functions in a stepwise fashion. In a four risk factor LSMC example, Teuguia et al. (2014) applied a full model approach, forward selection, backward elimination and a bidirectional approach as, for example, discussed in Hocking (1976) with orthogonal polynomial basis functions. They stated that only forward selection and the bidirectional approach were feasible when the number of risk factors or the polynomial degree exceeded 7, as then the resulting other models exploded. Life insurance companies covering a wide range of contracts in their portfolio are typically exposed to even more risk factors like, for example, 15. Complex business regulation frameworks such as those in Germany cause non-linear dependencies between risk factors and losses, which naturally lead to polynomials of higher degrees in the chosen proxy models. In these cases, even the standard forward selection and bidirectional approaches become infeasible as the sets of candidate terms from which the basis functions are chosen will explode then as well. We therefore follow the suggestion of Krah et al. (2018) to implement the so-called principle of marginality, an iteration-wise update technique of the set of candidate terms that lets the algorithm get along with comparably few carefully selected candidate terms.

Our main contribution is to identify, explain and illustrate a collection of regression methods and model selection criteria from the variety of regression design options that provide suitable proxy functions in the LSMC framework when applied in combination with the principle of marginality. After some general remarks in Section 3.1, we describe ordinary least-squares (OLS) regression in Section 3.2, generalized linear models (GLMs) by Nelder and Wedderburn (1972) in Section 3.3, generalized additive models (GAMs) by Hastie and Tibshirani (1986) and Hastie and Tibshirani (1990) in Section 3.4, feasible generalized least-squares (FGLS) regression in Section 3.5, multivariate adaptive regression splines (MARS) by Friedman (1991) in Section 3.6, and kernel regression by Watson (1964) and Nadaraya (1964) in Section 3.7. While some regression methods such as OLS and FGLS regression or GLMs can immediately be applied in conjunction with numerous model selection criteria such as Akaike information criterion (AIC), Bayesian information criterion (BIC), Mallow’s $C_P$ or generalized cross-validation (GCV), other regression methods such as GAMs, MARS, kernel, ridge or robust regression require well thought-through modifications thereof or work only with non-parametric alternatives such as k-fold or leave-one-out cross-validation. For adaptive approaches of FGLS, ridge and robust regression in life insurance proxy modeling, see also Hartmann (2015), Krah (2015) and Nikolić et al. (2017), respectively.

In the theory sections, we present the models with their assumptions, important properties and popular estimation algorithms and demonstrate how they can be embedded in the adaptive algorithm by proposing feasible implementation designs and combinable model selection criteria. While we shed light on the theoretical basic concepts of the models to lay the groundwork for the application and interpretation of the later following numerical experiments, we forego describing in detail technical enhancements or peculiarities of the involved algorithms and instead refer the interested reader to further sources. Additionally we provide the practicioners with R packages containing useful implementations of the presented regression routines. We complement the theory sections by corresponding empirical results in Section 4, throughout which we perform the same Monte Carlo approximation task to make the performance of the various methods comparable. We measure the approximation quality of the resulting proxy functions by means of aggregated validation figures on three out-of-sample test sets.

Conceivable alternatives to the entire adaptive algorithm are other typical machine learning techniques such as artificial neural networks (ANNs), decision tree learning or support vector machines. In particular, the classical feed forward networks proposed by Hejazi and Jackson (2017) and applied in various ways by Kopczyk (2018), Castellani et al. (2018), Born (2018) and Schelthoff (2019) were shown to capture the complex nature of CFP models well. A major challenge here is not only to find reliable hyperparameters such as the numbers of hidden layers and nodes in the network, batch size, weight initializer probability distribution, learning rate or activation functions but also the high dependence on the random seeds. We plan to contribute to this in a further publication which will be dedicated to hyperparameter search algorithms and stabilization methods such as ensemble methods. As an alternative to feed forward networks, Kazimov (2018) suggested to use radial basis function networks albeit so far none of the tested approaches performed better than the ordinary least squares regression in Krah et al. (2018).

In decision tree learning, random forests and tree-based gradient boosting machines were considered by Kopczyk (2018) and Schoenenwald (2019). While random forests were outperformed by feed forward networks but did better than the least absolute shrinkage and selection operator (LASSO) by Tibshirani (1996) in the example of the former author, they generally performed worse than the adaptive approaches by Krah et al. (2018) with OLS regression in numerous examples of the latter author. The gradient boosting machines, requiring more parameter tuning and thus being more versatile and demanding, came overall very close to the adaptive approaches.

Castellani et al. (2018) compared support vector regression (SVR) by Drucker et al. (1997) to ANNs and the adaptive approaches by Teuguia et al. (2014) in a seven risk factor example and found the performance of SVR placed somewhere inbetween the other two approaches with the ANNs getting closest to the nested simulations benchmark. As some further non-parametric approaches, Sell (2019) tested least-squares support-vector machines (LS-SVM) by Suykens and Vandewalle (1999) and shrunk additive least-squares approximations (SALSA) by Kandasamy and Yu (2016) in comparison to ANNs and the adaptive approaches by Krah et al. (2018) with OLS regression. In his examples, SALSA was able to beat the other two approaches whereas LS-SVM was left far behind. The analyzed machine learning alternatives have in common that they require at least to some degree a fine-tuning of some model hyperparameters. Since this is often a non-trivial but crucial task for generating suitable proxy functions, finding efficient and reliable search algorithms should become a subject of future research.

2. Calibration and Validation in the LSMC Framework

2.1. Fitting and Validation Points

2.1.1. Outer Scenarios and Inner Simulations

Our starting point is the LSMC approach (Krah et al. (2018)). LSMC proxy functions are calibrated conditional on the fitting points generated by the Monte Carlo simulations of the CFP model. Additional out-of-sample validation points serve as a mean for an assessment of the goodness-of-fit. The explaining variables of a proxy function are financial and actuarial risks the insurance company is exposed to. Examples for these risks are changes in interest rates, equity, credit, mortality, morbidity, lapse and expense levels over the one-year period. The dependent variable is an economic variable like the available capital, loss of available capital or best estimate of liabilites over the one-year period. Figure 1 plots the fitting values of an exemplary economic variable with respect to a financial risk factor. By an outer scenario we refer to a specific realized stress level combination of these risk factors over one year, and by an inner simulation to a stochastic path of an outer scenario in the CFP model under the given risk-neutral probability measure. Each outer scenario is assigned the probability weighted mean value of the economic variable over the corresponding inner simulations. In the LSMC context the fitting values are the mean values over only few inner simulations whereas the validation values are derived as the mean values over many inner simulations.

2.1.2. Different Trade-Off Requirements

According to the law of large numbers, this construction makes the validation values comparably stable while the fitting values are very volatile. Typically, the very limited fitting and validation simulation budgets are of similar sizes. Hence the few inner simulations in the case of the fitting points allow a great diversification among the outer scenarios whereas the many inner simulations in the case of the validation points let the validation values be quite close to their expectations but at the cost of only little diversification among the outer scenarios. These opposite ways to deal with the trade-off between the numbers of outer scenarios and inner simulations reflect the different requirements for the fitting and validation points in the LSMC approach. While the fitting scenarios should cover the domain of the real-world scenarios well to serve as a good regression basis, the validation values should approximate the expectations of the economic variable at the validation scenarios well to provide appropriate target values for the proxy functions.

2.2. Calibration Algorithm

2.2.1. Five Major Components

The calibration of the proxy function is performed by an adaptive algorithm that can be decomposed into the following five major components: (1) a set of allowed basis function types for the proxy function, (2) a regression method, (3) a model selection criterion, (4) a candidate term update principle, and (5) the number of steps per iteration and the directions of the algorithm. For illustration, we adopt the flowchart of the adaptive algorithm from Krah et al. (2018) and depict it in Figure 2. While components (1) and (5) enter the flowchart implicitly through the start proxy, candidate terms and the order of the processes and decisions in the chart, components (2), (3) and (4) are explicitly indicated through the labels “Regression”, “Model Selection Criterion” and “Get Candidate Terms”.

Let us briefly recapitulate the choice of components (1)–(5) from the successful applications of the adaptive algorithm in the insurance industry as described in Krah et al. (2018). As the function types for the basis functions (1), let only monomials be allowed. Let the regression method (2) be ordinary least-squares (OLS) regression and the model selection criterion (3) Akaike information criterion (AIC) from Akaike (1973). Let the set of candidate terms (4) be updated by the principle of marginality to which we will return in greater detail below. Lastly, when building up the proxy function iteratively, let the algorithm make only one step per iteration in the forward direction (5) meaning that in each iteration exactly one basis function is selected which cannot be removed anymore (adaptive forward stepwise selection).

2.2.2. Iterative Procedure

The algorithm starts in the upper left side of Figure 2 with the specification of the start proxy basis functions. We specify only the intercept so that the first regression ($k=0$) reduces to averaging over all fitting values. In order to harmonize the choices of OLS regression and AIC, we assume that the errors are normally distributed and homoscedastic because then the OLS estimator coincides with the maximum likelihood estimator. AIC is a relative measure for the goodness-of-fit of the proxy function and is defined as twice the negative of the maximum log-likelihood plus twice the number of degrees of freedom. The smaller the AIC score, the better the fit, and thus the trade-off between a too complex (overfitting) and too simple model (underfitting).

At the beginning of each iteration ($k=1,\dots ,K-1$), the set of candidate terms is updated by the principle of marginality which stipulates that a monomial basis function becomes a candidate if and only if all its derivatives are already included in the proxy function. The choice of a monomial basis is compatible to the principle of marginality. Using such a principle saves computational costs by selecting the basis functions conditionally on the current proxy function structure. In the first iteration ($k=1$), all linear monomials of the risk factors become candidates as their derivatives are constant values which are represented by the intercept.

The algorithm proceeds on the lower left side of the flowchart with a loop in which all candidate terms are separately added to the proxy function structure and tested with regard to their additional explanatory power. With each candidate, the fitting values are regressed against the fitting scenarios and the AIC score is calculated. If no candidate reduces the currently smallest AIC score, the algorithm terminates, and otherwise, the proxy function is updated by the one which reduces AIC most. Then the next iteration ($k+1$) begins with the update of the set of candidate terms, and so on. As long as no termination occurs, this procedure is repeated until the prespecified maximum number of terms $K_{\max }$ is reached.

2.3. Validation Figures

2.3.1. Validation Sets

Since it is the objective of this paper to propose suitable regression methods for the proxy function calibration in the LSMC framework, we introduce several validation figures serving as indicators for the approximation quality of the proxy functions. We measure the out-of-sample performance of each proxy function on three different validation sets by calculating five validation figures per set.

The three validation sets are a Sobol set, a nested simulations set and a capital region set. Unlike the Sobol set, the nested simulations and capital region sets do not serve as feasible validation sets in the LSMC routine as they become known only after evaluating the proxy function as explained below. Furthermore, they require massive computational capacities. Yet they can be regarded as the natural benchmark for the LSMC-based method and are thus very valuable for this analysis. Figure 3 plots the nested simulation values of an exemplary economic variable with respect to a financial risk factor. The Sobol set consists of, for example, between $L=15$ and $L=200$ Sobol validation points, of which the scenarios follow a Sobol sequence covering the fitting space uniformly. Thereby, the fitting space is the cube on which the outer fitting scenarios are defined. It has to cover the space of real-world scenarios used for the full loss distribution forecast sufficiently well. For interpretive reasons, sometimes the Sobol set is extended by points with, for example, one-dimensional risk scenarios or scenarios producing a risk capital close to the SCR ($=99.5\%$ value-at-risk) in previous risk capital calculations.

The nested simulations set comprises the, for example, $L=820$ to $L=6554$ validation points of which the scenarios correspond to the, for example, highest $2.5\%$ to $5\%$ losses from the full loss distribution forecast made by the proxy function that had been derived under the standard calibration algorithm choices described in Section 2.2. Like in the example of Chapter 5.2 in Krah et al. (2018), the order of these losses-which scenarios lead to which quantiles?following from the fourth and last step of the LSMC approach is very similar to the order following from the nested simulations approach. Therefore the scenarios of the nested simulations set are simply chosen by the order of the losses resulting from the LSMC approach. Several of these scenarios consist of stresses falling out of the fitting space. Compare Figure 1 and Figure 3 which depict fitting and nested simulation values from the same proxy modeling task with respect to the same risk factor. Severe outliers due to extreme stresses far outside of the fitting space should be excluded from the set. The capital region set is a subset of the nested simulations set containing the nested simulations SCR estimate, that is, the scenario leading to the $99.5\%$ loss, and the, for example, 64 losses above and below, which makes in total, for example, $L=129$ validation points.

2.3.2. Validation Figures

The five validation figures reported in our numerical experiments comprise two normalized mean absolute errors (MAEs), one with respect to the magnitude of the economic variable itself and one with respect to the magnitude of the corresponding market value of assets. They comprise further the mean error, that is, the mean of the residuals, as well as two validation figures based on the change of the economic variable from its base value (see the definition of the base value below): the normalized MAE with respect to the magnitude of the changes and the mean error of these changes. The smaller the normalized MAEs are, the better the proxy function approximates the economic variable. However, the validation values are afflicted with Monte Carlo errors so that the normalized MAEs serve only as meaningful indicators as long as the proxy functions do not become too precise. The means of the residuals should be possibly close to zero since they indicate systematic deviations of the proxy functions from the validation values. While the first three validation figues measure how well the proxy function reflects the economic variable in the CFP model, the latter two address the approximation effects on the SCR, compare Chapter 3.4.1 of Krah et al. (2018).

Let us write the absolute value as $\left|·\right|$ and let L denote the number of validation points. Then we can express the MAE of the proxy function $\widehat{f}\left(x^i\right)$ evaluated at the validation scenarios $x^i$ versus the validation values $y^i$ as $\frac{1}{L}{\sum }_{i=1}^L\left|y^i-\widehat{f}\left(x^i\right)\right|$. After normalizing the MAE with respect to the mean of the absolute values of the economic variable or the market value of assets, that is, $\frac{1}{L}{\sum }_{i=1}^L\left|d^i\right|$ with $d^i\in \left\{y^i,a^i\right\}$, we obtain the first two validation figures, that is,

$$\mathrm{mae}=\frac{{\sum }_{i=1}^L\left|y^i-\widehat{f}\left(x^i\right)\right|}{{\sum }_{i=1}^L\left|d^i\right|}.$$

(1)

In the following, we will refer to (1) with $d^i=y^i$ as the MAE with respect to the relative metric, and to (1) with $d^i=a^i$ as the MAE with respect to the asset metric. The mean of the residuals is given by

$$\mathrm{res}=\frac{1}{L}\sum _{i=1}^L\left(y^i-\widehat{f}\left(x^i\right)\right).$$

(2)

Let us refer by the base value $y^0$ to the validation value corresponding to the base scenario $x^0$ in which no risk factor has an effect on the economic variable. In analogy to (1) but only with respect to the relative metric, we introduce another normalized MAE by

$${\mathrm{mae}}^0=\frac{{\sum }_{i=1}^L\left|\left(y^i-y^0\right)-\left(\widehat{f}\left(x^i\right)-\widehat{f}\left(x^0\right)\right)\right|}{{\sum }_{i=1}^L\left|y^i-y^0\right|}.$$

(3)

The mean of the corresponding residuals is given by

$${\mathrm{res}}^0=\frac{1}{L}\sum _{i=1}^L\left(\left(y^i-y^0\right)-\left(\widehat{f}\left(x^i\right)-\widehat{f}\left(x^0\right)\right)\right).$$

(4)

In addition to these five validation figures, let us define the base residual which can be used as a substitute for (4) depending on personal taste. The base residual can easily be extracted from (2) and (4) by

$${\mathrm{res}}^{\mathrm{base}}=y^0-\widehat{f}\left(x^0\right)=\mathrm{res}-{\mathrm{res}}^0.$$

(5)

3. Machine Learning Regression Methods

3.1. General Remarks

As the main part of our work, we will compare various types of machine learning regression approaches for determining suitable proxy functions in the LSMC framework. The methods we present in this section range from ordinary and generalized least-squares regression variants over GLM and GAM approaches to multivariate adaptive regression splines and kernel regression approaches.

The performance of the newly derived proxy functions when applied to the described validation sets is one way of comparing the different methods. Another way consists of ensuring compatibility with the principle of marginality and utilizing a suitable model selection criterion such as AIC in order to be able to compare iteration-wise the candidate models inside the approaches.

We will in the following sections shortly introduce the different methods, collect some theoretical properties and then concentrate on aspects of their implementation. Their numerical performance on the different validation sets is the subject of Section 4.

Our aim in the calibration step below is to estimate the conditional expectation $Y\left(X\right)$ under the risk-neutral measure given an outer scenario X. In contrast to Krah et al. (2018)$Y\left(X\right)$ does not necessarily have to be the available capital but can instead be, for example, the best estimate of liabilites or the market value of assets. The D-dimensional fitting scenarios are always generated under the physical probability measure ${\mathbb{P}}^{\prime }$ on the fitting space which itself is a subspace of ${\mathbb{R}}^D$.

3.2. Ordinary Least-Squares (OLS) Regression

3.2.1. The Regression Model

In iteration $K-1$ of the adaptive forward stepwise algorithm (as given in Section 2.2), the OLS approximation consists of a linear combination of suitable linearly independent basis functions $e_k\left(X\right)\in L^2\left({\mathbb{R}}^D,\mathcal{B},{\mathbb{P}}^{\prime }\right),k=0,1,\dots ,K-1,$ that is,

$$Y\left(X\right)\stackrel{K<\infty }{\approx }f\left(X\right)=\sum _{k=0}^{K-1}{\beta }_k{e}_k\left(X\right).$$

(6)

We call $f\left(X\right)$ the predictor of $Y\left(X\right)$ or the systematic component.

With the fitting points $\left(x^i,y^i\right),i=1,\dots ,N,$ and uncorrelated errors ${ϵ}^i$ (the random components) having the same variance ${\sigma }^2>0$ (= homoscedastic errors), we obtain the classical linear regression model

$$y^i=\sum _{k=0}^{K-1}{\beta }_k{e}_k\left(x^i\right)+{ϵ}^i,$$

(7)

where $e_0\left(x^i\right)=1$ and ${\beta }_0$ is the intercept. Then, the ordinary least-squares (OLS) estimator ${\widehat{\mathit{\beta }}}_{\mathrm{OLS}}$ of the coefficients is given by

$${\widehat{\mathit{\beta }}}_{\mathrm{OLS}}=\underset{\mathit{\beta }\in {\mathbb{R}}^K}{arg\; min}\left\{\sum _{i=1}^N{\left(y^i-\sum _{k=0}^{K-1}{\beta }_k{e}_k\left(x^i\right)\right)}^2\right\}.$$

(8)

Using the notation $z_{ik}=e_k\left(x^i\right)$ the OLS problem is solved explicitly by

$${\widehat{\mathit{\beta }}}_{\mathrm{OLS}}={\left(Z^{T}Z\right)}^{-1}{Z}^T\mathbf{y}.$$

(9)

The proxy function $\widehat{f}\left(X\right)$ for the economic variable $Y\left(X\right)$ given an outer scenario X is

$$Y\left(X\right)\stackrel{K,N<\infty }{\approx }\widehat{f}\left(X\right)=\sum _{k=0}^{K-1}{\widehat{\beta }}_{\mathrm{OLS},k}{e}_k\left(X\right).$$

(10)

For a practical implementation see, for example, function lm($·$) in the R package stats of R Core Team (2018).

3.2.2. Gauss-Markov Theorem, ML Estimation and AIC

Under the assumptions of strict exogeneity $E\left[\mathit{ϵ}\mid Z\right]=\mathbf{0}$ (A1), a spherical error variance $V\left[\mathit{ϵ}\mid Z\right]={\sigma }^2{I}_N$ with $I_N$ the N-dimensional identity matrix (A2), and linearly independent basis functions (A3), we have (compare, for example, Hayashi (2000)):

• The OLS estimator is the best linear unbiased estimator (BLUE) of the coefficients in the classical linear regression model (7) (Gauss-Markov Theorem).
• If the errors $ϵ$ in (7) are in addition normally distributed (A4), then the OLS estimator and the maximum likelihood (ML) estimator of the coefficients coincide.
• Under Assumptions (A1)-(A4) the Akaike information criterion (AIC) has the form

  $$\begin{array}{c}\hfill \mathrm{AIC}=-2l\left({\widehat{\mathit{\beta }}}_{\mathrm{OLS}},{\widehat{\sigma }}^2\right)+2\left(K+1\right)=N\left(log\left(2\pi {\widehat{\sigma }}^2\right)+1\right)+2\left(K+1\right).\end{array}$$

  (11)

3.3. Generalized Linear Models (GLMs)

3.3.1. The Regression Model

The systematic component of a GLM (see Nelder and Wedderburn (1972) for its introduction) equals the linear predictor $\eta =f\left(X\right)$ of the model in (6). However, one uses a monotonic link function $g\left(·\right)$ that relates the economic variable $Y\left(X\right)$ to the linear predictor via

$$g(\underset{=\mu }{\underbrace{Y\left(X\right)}})\stackrel{K<\infty }{\approx }\underset{=\eta }{\underbrace{f\left(X\right)}}=\sum _{k=0}^{K-1}{\beta }_k{z}_k={\mathbf{z}}^T\mathit{\beta },$$

(12)

with $\mathbf{z}={\left(e_0\left(X\right),\dots ,e_{K-1}\left(X\right)\right)}^T$.

Of course, the choice of the link function $g(.)$ is a critical aspect. A possible motivation is a non-negativity requirement on $Y\left(X\right)$ that can be satisfied using $g\left(y\right)=ln\left(y\right)$. Further comments on choices of the link function are motivated below.

3.3.2. Canonical Link Function, GLM Estimation and IRLS Algorithm

While the normal distribution assumption for the random component allowed the derivation of nice properties in the linear model of the preceding section, the GLM considers random components with (conditional) distributions from the exponential family. Its canonical form with parameter $\theta$ is given by the density function

$$\pi (y\mid \theta ,\varphi )=exp\left(\frac{y\theta -b\left(\theta \right)}{a\left(\varphi \right)}+c(y,\varphi )\right),$$

(13)

where $a\left(\varphi \right)$, $b\left(\theta \right)$ and $c(y,\varphi )$ are specific functions. For example, a normally distributed economic variable with mean $\mu$ and variance ${\sigma }^2$ is given by $a\left(\varphi \right)=\varphi$, $b\left(\theta \right)=\frac{{\theta }^2}{2}$ and $c(y,\varphi )=-\frac{1}{2}\left(\frac{y^2}{{\sigma }^2}+log\left(2\pi {\sigma }^2\right)\right)$ with $\theta =\mu$ and $\varphi ={\sigma }^2$.

For a random variable Y with a distribution from the exponential family, we have

$$E\left(Y\right)=\mu =b^{\prime }\left(\theta \right),Var\left(Y\right)=b^{″}\left(\theta \right)a\left(\varphi \right)=:V\left[\mu \right]a\left(\varphi \right).$$

(14)

$a\left(\varphi \right)$ is called a dispersion parameter, $V[.]$ the variance function. We will in the following make the simplifying assumption $a\left({\varphi }^i\right)=\varphi$, $i=1,\dots ,N$ for a constant value of $\varphi$ (A5) and then obtain the ML estimator in the GLM from Equation (13) as

$${\widehat{\mathit{\beta }}}_{\mathrm{GLM}}=\underset{\mathit{\beta }\in {\mathbb{R}}^K}{arg\; max}\left\{\sum _{i=1}^N\left(\frac{y^i{\theta }^i-b\left({\theta }^i\right)}{\varphi }+c(y^i,\varphi )\right)\right\}.$$

(15)

Under (A5), there does in general not exist a closed-form solution for the GLM coefficient estimator (15). The resulting iterative method will be simplified for so-called canonical link functions $g\left(\mu \right)=\theta$ which due to relation (14) are given by

$$g\left(\mu \right)={\left(b^{\prime }\right)}^{-1}\left(\mu \right),$$

(16)

with $b(.)$ from the definition of the exponential family. Examples of pairs of canonical link functions and corresponding distributions are $g\left(\mu \right)=\mu$ and the normal, $g\left(\mu \right)=1/\mu$ and the gamma, and $g\left(\mu \right)=1/{\mu }^2$ and the inverse Gaussian distribution.

In Chapter 2.5, McCullagh and Nelder (1989) apply Fisher’s scoring method to obtain an approximation to the GLM estimator. Further, McCullagh and Nelder (1989) justify how Fisher’s scoring method can be cast in the form of the iteratively reweighted least squares (IRLS) algorithm. To state the IRLS algorithm in our context, we need some notation.

Let ${\widehat{\eta }}_{\left(t\right)}^i=\widehat{f}\left(x^i\right)$ be the estimate for the linear predictor evaluated at fitting scenario $x^i$, compare (12). Let ${\widehat{\mu }}_{\left(t\right)}^i=g^{-1}\left({\widehat{\eta }}_{\left(t\right)}^i\right)$ be the estimate for the economic variable, and $\frac{d\eta }{d\mu }\left({\widehat{\mu }}_{\left(t\right)}^i\right)=g^{\prime }\left({\widehat{\mu }}_{\left(t\right)}^i\right)$ the first derivative of the link function with respect to the economic variable evaluated at ${\widehat{\mu }}_{\left(t\right)}^i$. Furthermore, we introduce the weight matrix $W^{\left(t\right)}=\mathrm{diag}\left(w^1\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right),\dots ,w^N\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)\right)$ with components given by

$${\widehat{w}}^i\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)={\left(\frac{d\eta }{d\mu }\left({\widehat{\mu }}_{\left(t\right)}^i\right)\right)}^{-2}V{\left[{\widehat{\mu }}_{\left(t\right)}^i\right]}^{-1},$$

(17)

and $V\left[{\widehat{\mu }}_{\left(t\right)}^i\right]$ the variance function from above evaluated at ${\widehat{\mu }}_{\left(t\right)}^i$. Finally, we define $D^{\left(t\right)}=\mathrm{diag}(d_{\left(t\right)}^1,\dots ,d_{\left(t\right)}^N)$ with $d_{\left(t\right)}^i=g^{\prime }\left({\widehat{\mu }}_{\left(t\right)}^i\right)$ which allows us to formulate the IRLS algorithm for canonical link functions.

IRLS algorithm.

Perform the iterative approximation procedure below with an initialization of ${\widehat{\mu }}_{\left(0\right)}^i=y^i+0.1$ and ${\widehat{\eta }}_{\left(0\right)}^i=g\left({\widehat{\mu }}_{\left(0\right)}^i\right)$ as proposed by Dutang (2017) until convergence:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}& =& {\left(Z^T{W}^{\left(t\right)}Z\right)}^{-1}{Z}^T{W}^{\left(t\right)}{\widehat{\mathbf{s}}}^{\left(\mathit{t}\right)}\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right),\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{s}}}^{\left(\mathit{t}\right)}\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)& =& Z{\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}+D^{\left(t\right)}(y-{\widehat{\mu }}_t)\hfill \end{array}$$

(19)

After convergence, we set ${\widehat{\mathit{\beta }}}_{\mathrm{GLM}}={\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}$.

Green (1984) proposes to solve the system $\left(Z^T{W}^{\left(t\right)}Z\right){\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}=Z^T{W}^{\left(t\right)}{\widehat{\mathit{s}}}^{\left(\mathit{t}\right)}$ which is equivalent to (18) via a QR decomposition to increase numerical stability. For a practical implementation of GLMs using the IRLS algorithm, see, for example, function glm($·$) in R package stats of R Core Team (2018).

By inserting (17), (19) and the GLM estimator into (18) and by using (12), we obtain

$$\begin{array}{cc}\hfill {\widehat{\mathit{\beta }}}_{\mathrm{GLM}}& =\underset{\mathit{\beta }\in {\mathbb{R}}^K}{arg\; min}\left\{\sum _{i=1}^{N}V\left[{\widehat{\mu }}_{\mathrm{GLM}}^i\right]{\left(y^i-{\widehat{\mu }}_{\mathrm{GLM}}^i\right)}^2\right\},\hfill \end{array}$$

(20)

that is, the GLM estimator minimizes the squared sum of raw residuals scaled by the estimated individual variances of the economic variable.

The Pearson residuals are defined as the raw residuals divided by the estimated individual standard deviations, that is,

$${\widehat{ϵ}}^i=\frac{y^i-{\widehat{\mu }}_{\mathrm{GLM}}^i}{\sqrt{V\left[{\widehat{\mu }}_{\mathrm{GLM}}^i\right]}}.$$

(21)

3.3.3. AIC and Dispersion Estimation

Since AIC depends on the ML estimators, it is combinable with GLMs in the adaptive algorithm. Here, it has the form

$$\mathrm{AIC}=-2l\left({\widehat{\mathit{\beta }}}_{\mathrm{GLM}},\widehat{\varphi }\right)+2\left(K+p\right),$$

(22)

where K is the number of coefficients and p indicates the number of the additional model parameters associated with the distribution of the random component. For instance, in the normal model, we have $p=1$ due to the error variance/dispersion. A typical estimate of the dispersion in GLMs is the Pearson residual chi-squared statistic divided by $N-K$ as described by Zuur et al. (2009) and implemented, for example, in function glm($·$) belonging to R package stats, that is,

$$\widehat{\varphi }=\frac{1}{N-K}\sum _{i=1}^N{\left({\widehat{ϵ}}^i\right)}^2,$$

(23)

with ${\widehat{ϵ}}^i$ given by (21). Even though this is not the ML estimator, it is a good estimate because, if the model is specified correctly, the Pearson residual chi-squared statistic divided by the dispersion is asymptotically ${\chi }_{N-K}^2$ distributed and the expected value of a chi-squared distribution with $N-K$ degrees of freedom is $N-K$.

3.4. Generalized Additive Models (GAMs)

3.4.1. The Regression Model

Generalized additive models (GAMs) as introduced by Hastie and Tibshirani (1986) and Hastie and Tibshirani (1990) can be regarded as richly parameterized GLMs with smooth functions. While GAMs inherit from GLMs the random component (13) and the link function (12), they inherit from the additive models of Friedman and Stuetzle (1981) the linear predictor with the smooth functions. In the adaptive algorithm, we apply GAMs of the form

$$g(\underset{=\mu }{\underbrace{Y\left(X\right)}})\stackrel{K<\infty }{\approx }\underset{=\eta }{\underbrace{f\left(X\right)}}={\beta }_0+\sum _{k=1}^{K-1}{h}_k\left(z_k\right),$$

(24)

where $z_k=e_k\left(X\right)$, ${\beta }_0$ is the intercept and $h_k\left(·\right),k=1,\dots ,K-1,$ are the smooth functions to be estimated. In addition to the smooth functions, GAMs can also include simple linear terms of the basis functions as they appear in the linear predictor of GLMs. A smooth function $h_k\left(·\right)$ can be written as a basis expansion

$$h_k\left(z_k\right)=\sum _{j=1}^J{\beta }_{kj}{b}_{kj}\left(z_k\right),$$

(25)

with coefficients ${\beta }_{kj}$ and known basis functions $b_{kj}\left(z_k\right),j=1,\dots ,J,$ which should not be confused with their arguments, namely the first-order basis functions $z_k=e_k\left(X\right),k=0,\dots ,K-1$. The slightly adapted Figure 4 from Wood (2006) depicts an exemplary approximation of y by a GAM with a basis expansion in one dimension $z_k$ without an intercept. The solid colorful curves represent the pure basis functions $b_{kj}\left(z_k\right),j=1,\dots ,J,$ the dashed colorful curves show them after scaling with the coefficients ${\beta }_{kj}{b}_{kj}\left(z_k\right),j=1,\dots ,J,$ and the black curve is their sum (25).

Typical examples for basis functions are thin plate regression splines, duchon splines, cubic regression splines or Eilers and Marx style P-splines. See, for example, function gam($·$) in R package mgcv of Wood (2018) for a practical implementation of GAMs admitting these types of basis functions and using the PIRLS algorithm, which we present below.

In vector notation, we can write $\mathit{\beta }={\left({\beta }_0,{\mathit{\beta }}_1^T,\dots ,{\mathit{\beta }}_{K-1}^T\right)}^T$ with ${\mathit{\beta }}_k={\left({\beta }_{k1},\dots ,{\beta }_{kJ}\right)}^T$ and $\mathbf{a}={\left(1,{\mathbf{b}}_1{\left(z_1\right)}^T,\dots ,{\mathbf{b}}_{K-1}{\left(z_{K-1}\right)}^T\right)}^T$ with ${\mathbf{b}}_k\left(z_k\right)={\left(b_{k1}\left(z_k\right),\dots ,b_{kJ}\left(z_k\right)\right)}^T$, hence (24) becomes

$$g(\underset{=\mu }{\underbrace{Y\left(X\right)}})\stackrel{K<\infty }{\approx }\underset{=\eta }{\underbrace{f\left(X\right)}}={\mathbf{a}}^T\mathit{\beta }.$$

(26)

In order to make the smooth functions $h_k\left(·\right),k=1,\dots ,K-1,$ identifiable, identifiability constraints ${\sum }_{i=1}^N{h}_k\left(z_{ik}\right)=0$ with $z_{ik}=e_k\left(x^i\right)$ can be imposed. According to Wood (2006) this can be achieved by modification of the basis functions $b_{kj}\left(·\right)$ with one of them being lost.

3.4.2. Penalization and GAM Estimation via PIRLS Algorithm

Let the deviance corresponding to observation $y^i$ be $D^i\left(\mathit{\beta }\right)=2\left(l_{\mathrm{sat}}^i-l^i\left(\mathit{\beta },\varphi \right)\right)\varphi$ where $D^i\left(\mathit{\beta }\right)$ is independent of dispersion $\varphi$, where $l_{\mathrm{sat}}^i={max}_{{\mathit{\beta }}^i}{l}^i\left({\mathit{\beta }}^i,\varphi \right)$ is the saturated log-likelihood and $l^i\left(\mathit{\beta },\varphi \right)$ the log-likelihood. Then the model deviance can be written as $D\left(\mathit{\beta }\right)={\sum }_{i=1}^N{D}^i\left(\mathit{\beta }\right)$. It is a generalization of the residual sum of squares for ML estimation. For instance, in the normal model the unit deviance is ${\left(y^i-{\mu }^i\right)}^2$. For given smoothing parameters ${\lambda }_k>0,k=1,\dots ,K-1$, the GAM estimator ${\widehat{\mathit{\beta }}}_{\mathrm{GAM}}$ of the coefficients is defined as the minimizer of the penalized deviance

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}_{\mathrm{GAM}}=\underset{\mathit{\beta }\in {\mathbb{R}}^{(K-1)J+1}}{arg\; min}\left\{D\left(\mathit{\beta }\right)+\sum _{k=1}^{K-1}{\lambda }_k\int h_k^{″}{\left(z_k\right)}^2\mathrm{d}{z}_k\right\},\mathrm{where}\\ \hfill \int h_k^{″}{\left(z_k\right)}^2\mathrm{d}{z}_k={\mathit{\beta }}_k^T\left(\int {\mathbf{b}}_k^{″}\left(z_k\right){\mathbf{b}}_k^{″}{\left(z_k\right)}^T\mathrm{d}{z}_k\right){\mathit{\beta }}_k={\mathit{\beta }}_k^T{\mathcal{S}}_k{\mathit{\beta }}_k\end{array}$$

(27)

are the smoothing penalties. The smoothing parameters ${\lambda }_k$ control the trade-off between a too wiggly model (overfitting) and a too smooth model (underfitting). The larger the ${\lambda }_k$ values are, the more pronounced is the wiggliness of the basis functions reflected by their second derivatives in the minimization problem (27), and the higher is thus the penalty associated with the coefficients and the smoother is the estimated model.

A major advantage of the definition of GAMs via (24), (25), and (27) is its compatibility with information criteria and other model selection criteria such as generalized cross-validation. Besides, the resulting penalty matrix favors numerical stability in the PIRLS algorithm.

Since the saturated log-likelihood is a constant for a fixed distribution and set of fitting points, we can turn the minimization problem (27) into the maximization task of the penalized log-likelihood, that is,

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}_{\mathrm{GAM}}=\underset{\mathit{\beta }\in {\mathbb{R}}^{(K-1)J+1}}{arg\; max}\left\{l\left(\mathit{\beta },\varphi \right)-\frac{1}{2}\sum _{k=1}^{K-1}{\lambda }_k{\mathit{\beta }}_k^T{\mathcal{S}}_k{\mathit{\beta }}_k\right\}.\end{array}$$

(28)

Wood (2000) points out that Fisher’s scoring method can be cast in a penalized version of the iteratively reweighted least squares (PIRLS) algorithm when being used to approximate the GAM coefficient estimator (28). We formulate the PIRLS algorithm based on Marx and Eilers (1998) who indicate the iterative solution explicitly.

Let ${\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}$ now be the GAM coefficient approximation in iteration t. Then the vector of the dependent variable ${\widehat{\mathbf{s}}}^{\left(\mathit{t}\right)}={\left({\widehat{s}}^1\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right),\dots ,{\widehat{s}}^N\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)\right)}^T$ and the weight matrix given by $W^{\left(t\right)}=\mathrm{diag}\left(w^1\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right),\dots ,w^N\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)\right)$ have the same form as in the IRLS algorithm, see (19) and (17). Additionally, let $S=\mathrm{blockdiag}\left(0,{\lambda }_1{\mathcal{S}}_1,\dots ,{\lambda }_{K-1}{\mathcal{S}}_{K-1}\right)$ with $S_{11}=0$ belonging to the intercept be the penalty matrix.

PIRLS algorithm.

Perform the iterative approximation procedure below with initialization of ${\widehat{\mu }}_{\left(0\right)}^i=y^i+0.1$ and ${\widehat{\eta }}_{\left(0\right)}^i=g\left({\widehat{\mu }}_{\left(0\right)}^i\right)$ until convergence occurs:

$$\begin{array}{cc}\hfill {\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}& =\underset{\mathit{\beta }\in {\mathbb{R}}^{(K-1)J+1}}{arg\; min}\left\{\sum _{i=1}^N{w}^i{\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)}^{-1}{\left({\widehat{s}}^i\left({\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}\right)-{\beta }_0-\sum _{k=1}^{K-1}\sum _{j=1}^J{\beta }_{kj}{b}_{kj}\left(z_{ik}\right)\right)}^2+\sum _{k=1}^{K-1}{\lambda }_k{\mathit{\beta }}_k^T{\mathcal{S}}_k{\mathit{\beta }}_k\right\}\hfill \\ \hfill & ={\left(Z^T{W}^{\left(t\right)}Z+S\right)}^{-1}{Z}^T{W}^{\left(t\right)}{\widehat{\mathbf{s}}}^{\left(\mathit{t}\right)}.\hfill \end{array}$$

(29)

After convergence, we set ${\widehat{\mathit{\beta }}}_{\mathrm{GAM}}={\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}$.

3.4.3. Smoothing Parameter Selection, AIC and Stagewise Selection

The smoothing parameters ${\lambda }_k$ can be selected such that they minimize a suitable model selection criterion, for the sake of consistency, preferably the one used in the adaptive algorithm for basis function selection. The GAM estimator (28) does not exactly maximize the log-likelihood, therefore AIC has another form for GAMs than for GLMs. Hastie and Tibshirani (1990) propose a widely used version of AIC for GAMs, which uses effective degrees of freedom df in place of the number of coefficients $(K-1)J+1$. This is

$$\mathrm{AIC}=-2l\left({\widehat{\mathit{\beta }}}_{\mathrm{GAM}},\widehat{\varphi }\right)+2\left(\mathrm{df}+p\right),$$

(30)

where

$$\mathrm{df}=\mathrm{tr}\left({\left(I+S\right)}^{-1}I\right).$$

(31)

Note that $I+S=Z^{T}WZ+S$ is already approximately calculated in the PIRLS algorithm. For GAMs, an estimate of the dispersion $\widehat{\varphi }$ is obtained similarly to GLMs by (23). The parameter p is defined as in (22).

Another popular and effective smoothing parameter selection criterion invented by Craven and Wahba (1979) is generalized cross-validation (GCV), that is,

$$\mathrm{GCV}=\frac{ND\left({\widehat{\mathit{\beta }}}_{\mathrm{GAM}}\right)}{{\left(N-\mathrm{df}\right)}^2},$$

(32)

with the model deviance $D\left({\widehat{\mathit{\beta }}}_{\mathrm{GAM}}\right)$ evaluated at the GAM estimator and the effective degrees of freedom defined just like for AIC.

Note that the adaptive forward stepwise algorithm depicted in Figure 2 can become computationally infeasible with GAMs as opposed to, for example, GLMs. In iteration k, a GAM has $(K-1)J+1$ coefficients which need to be estimated while a GLM has only K coefficients. This difference in the estimation effort is increased further due to the iterative nature of the IRLS and PIRLS algorithms. Moreover, GAMs involve the task of optimal smoothing parameter selection. To deal with this aspect, Wood (2000), Wood et al. (2015) and Wood et al. (2017) have developed practical GAM fitting methods for large data sets. However, the suitable application of these methods in the adaptive algorithm is beyond the scope of our analysis, in particular as our focus is not on computational performance. Besides parallelizing the candidate loop on the lower left side of Figure 2, we achieve the necessary performance gains in GAMs by replacing the stepwise algorithm by a stagewise algorithm. This means that in each iteration, a predefined number L or proportion of candidate basis functions is selected simultaneously until a termination criterion is fulfilled. Thereby we select in one stage those basis functions which reduce the model selection criterion of our choice most when added separately to the current proxy function structure. When there are not at least as many basis functions as targeted, the algorithm shall be terminated after the ones which lead to a reduction in the model selection criterion have been selected.

3.5. Feasible Generalized Least-Squares (FGLS) Regression

3.5.1. The Regression Model

The regression model here equals the OLS case. However, we now let the errors have the covariance matrix $\mathsf{\Sigma }={\sigma }^2\mathsf{\Omega }$ where $\mathsf{\Omega }$ is positive definite and known and ${\sigma }^2>0$ is unknown. We transform the generalized regression model according to Hayashi (2000) to obtain a model (*) which satisfies Assumptions (A1), (A2) and (A3) of the classical linear regression model. For this, choose an invertible matrix H with ${\mathsf{\Omega }}^{-1}=H^{T}H$ which can, for example, be the Cholesky matrix. Then, the generalized response vector ${\mathbf{y}}^{*}$, design matrix $Z^{*}$ and error vector ${\mathit{ϵ}}^{*}$ are given by

$${\mathbf{y}}^{*}=H\mathbf{y},Z^{*}=HZ,{\mathit{ϵ}}^{*}={\mathbf{y}}^{*}-Z^{*}\mathit{\beta }=H\left(\mathbf{y}-Z\mathit{\beta }\right)=H\mathit{ϵ}.$$

(33)

In analogy to the OLS estimator, the generalized least-squares (GLS) estimator ${\widehat{\mathit{\beta }}}_{\mathrm{GLS}}$ of the coefficients is given as the minimizer of the generalized residual sum of squares, that is,

$${\widehat{\mathit{\beta }}}_{\mathrm{GLS}}=\underset{\mathit{\beta }\in {\mathbb{R}}^K}{arg\; min}\left\{\sum _{i=1}^N{\left({ϵ}^{*,i}\right)}^2\right\}.$$

(34)

The closed-form expression of the GLS estimator is

$${\widehat{\mathit{\beta }}}_{\mathrm{GLS}}={\left(Z^{*T}{Z}^{*}\right)}^{-1}{Z}^{*T}{\mathbf{y}}^{*}={\left(Z^T{\mathsf{\Omega }}^{-1}Z\right)}^{-1}{Z}^T{\mathsf{\Omega }}^{-1}\mathbf{y},$$

(35)

and the proxy function becomes

$$\widehat{f}\left(X\right)={\mathbf{z}}^T{\widehat{\mathit{\beta }}}_{\mathrm{GLS}},$$

(36)

where $\mathbf{z}={\left(e_0\left(X\right),\dots ,e_{K-1}\left(X\right)\right)}^T$. The scalar ${\sigma }^2$ can be estimated in analogy to OLS regression by $s_{\mathrm{GLS}}=\frac{1}{N-K}{\widehat{\mathit{ϵ}}}^{*T}{\widehat{\mathit{ϵ}}}^{*}$ where ${\widehat{\mathit{ϵ}}}^{*}={\mathbf{y}}^{*}-Z^{*}{\widehat{\mathit{\beta }}}_{\mathrm{GLS}}$ is the residual vector.

3.5.2. Gauss-Markov-Aitken Theorem and ML Estimation

Under the assumptions (A1), (A3), and a covariance matrix $\mathsf{\Sigma }={\sigma }^2\mathsf{\Omega }$ of which $\mathsf{\Omega }$ is positive definite and known (A6), we have:

• The GLS estimator is the BLUE of the coefficients in the generalized regression model (7) (Gauss-Markov-Aitken theorem).
• If in addition we have jointly normally distributed errors conditional on the fitting scenarios (A7) then the ML coefficient estimator coincides with the GLS estimator. Further, the ML estimator of the scalar ${\widehat{\sigma }}^2$ can be expressed as $\frac{N}{N-K}$ times $s_{\mathrm{GLS}}$.

As a consequence, given a known matrix $\mathsf{\Omega }$, we have a closed form solution for the GLS estimator that coincides with the ML estimator of the regression coefficients and the adaptive algorithm inside the LSMC approach goes through.

3.5.3. Unknown $\mathsf{\Omega }$ and FGLS Estimation via ML Algorithm

In the LSMC framework, $\mathsf{\Omega }$ is unknown. However, if a consistent estimator $\widehat{\mathsf{\Omega }}$ exists, we can apply feasible generalized least-squares (FGLS) regression, of which the estimator

$${\widehat{\mathit{\beta }}}_{\mathrm{FGLS}}={\left(Z^T{\widehat{\mathsf{\Omega }}}^{-1}Z\right)}^{-1}{Z}^T{\widehat{\mathsf{\Omega }}}^{-1}\mathbf{y}$$

(37)

has asymptotically the same properties as the GLS estimator (35).

With $\mathbf{z}={\left(e_0\left(X\right),\dots ,e_{K-1}\left(X\right)\right)}^T$ the FGLS proxy function is then given as

$$\widehat{f}\left(X\right)={\mathbf{z}}^T{\widehat{\mathit{\beta }}}_{\mathrm{FGLS}}.$$

(38)

For the estimation of $\mathsf{\Omega }$ we will in the following set ${\sigma }^2=1$ which can be done without loss of generality and consider $\mathsf{\Sigma }=\mathsf{\Omega }$. Furthermore, we assume in addition to (A1), (A3) and (A7) that the elements of the covariance matrix $\mathsf{\Sigma }$ are twice differentiable functions of parameters $\mathit{\alpha }={\left({\alpha }_0,\dots ,{\alpha }_{M-1}\right)}^T$ with $K+M\le N$. We then write $\mathsf{\Sigma }=\mathsf{\Sigma }\left(\mathit{\alpha }\right)$ (A8). The following result is the basis of the iterative ML algorithm for the regression coefficients and the variance matrix.

Theorem 1.

The generalized regression model (7) under Assumptions (A1), (A3), (A7) and (A8) has the following first-order ML conditions:

$$\begin{array}{cc}\hfill {\widehat{\mathit{\beta }}}_{\mathrm{ML}}& ={\left(Z^T{\widehat{\mathsf{\Sigma }}}^{-1}Z\right)}^{-1}{Z}^T{\widehat{\mathsf{\Sigma }}}^{-1}\mathbf{y},\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill \frac{\partial l}{\partial {\alpha }_m}& =\frac{1}{2}\mathrm{tr}{\left(\frac{\partial {\mathsf{\Sigma }}^{-1}}{\partial {\alpha }_m}\mathsf{\Sigma }\right)}_{\mathit{\alpha }={\widehat{\mathit{\alpha }}}_{\mathrm{ML}}}-\frac{1}{2}{\widehat{\mathit{ϵ}}}^T{\left(\frac{\partial {\mathsf{\Sigma }}^{-1}}{\partial {\alpha }_m}\right)}_{\mathit{\alpha }={\widehat{\mathit{\alpha }}}_{\mathrm{ML}}}\widehat{\mathit{ϵ}}=0,\hfill \end{array}$$

(40)

where $m=0,\dots ,M-1$, $\widehat{\mathsf{\Sigma }}=\mathsf{\Sigma }\left({\widehat{\mathit{\alpha }}}_{\mathrm{ML}}\right)$ and $\widehat{\mathit{ϵ}}=\mathbf{y}-Z{\widehat{\mathit{\beta }}}_{\mathrm{ML}}$.

The system in (39) and (40) is then solved iteratively (see, for example, Magnus (1978)). We start the procedure with ${\mathit{\beta }}^{\left(\mathbf{0}\right)}$ and then use PORT optimization routines as described in Gay (1990) and implemented in function nlminb($·$) belonging to R package stats of R Core Team (2018). In this iterative routine, ${\widehat{\mathit{\alpha }}}^{(\mathit{t}+\mathbf{1})}$ can be initialized, for example, by random numbers from the standard normal distribution.

ML algorithm.

Perform the following iterative approximation procedure with, for example, an initialization of ${\widehat{\mathit{\beta }}}^{\left(\mathbf{0}\right)}={\widehat{\mathit{\beta }}}_{\mathrm{OLS}}$ until convergence:

1. 

Calculate the residual vector ${\widehat{\mathit{ϵ}}}^{(\mathit{t}+\mathbf{1})}=\mathbf{y}-Z{\widehat{\mathit{\beta }}}^{\left(\mathit{t}\right)}$.

2. 

Substitute ${\widehat{\mathit{ϵ}}}^{(\mathit{t}+\mathbf{1})}$ into the M equations in M unknowns ${\alpha }_m$ given by (40) and solve them. If an explicit solution exists, set ${\widehat{\mathit{\alpha }}}^{(\mathit{t}+\mathbf{1})}=\mathit{\alpha }\left({\widehat{\mathit{ϵ}}}^{(\mathit{t}+\mathbf{1})}\right)$. Otherwise, select the maximum likelihood solution ${\widehat{\mathit{\alpha }}}^{(\mathit{t}+\mathbf{1})}$ iteratively, for example, by using PORT optimization routines.

3. 

Calculate

$$\begin{array}{cc}\hfill {\widehat{\mathsf{\Sigma }}}^{(t+1)}& =\mathsf{\Sigma }\left({\widehat{\mathit{\alpha }}}^{(\mathit{t}+\mathbf{1})}\right),\hfill \\ \hfill {\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}& ={\left(Z^T{\left({\widehat{\mathsf{\Sigma }}}^{(t+1)}\right)}^{-1}Z\right)}^{-1}{Z}^T{\left({\widehat{\mathsf{\Sigma }}}^{(t+1)}\right)}^{-1}\mathbf{y}.\hfill \end{array}$$

(41)

Continue with the next iteration.

After convergence, we set ${\widehat{\mathit{\beta }}}_{\mathrm{ML}}={\widehat{\mathit{\beta }}}^{(\mathit{t}+\mathbf{1})}$ and ${\widehat{\mathit{\alpha }}}_{\mathrm{ML}}={\widehat{\mathit{\alpha }}}^{(\mathit{t}+\mathbf{1})}$.

Theorem 5 of Magnus (1978) states that under some further regularity conditions the FGLS coefficient estimator can be derived as the ML coefficient estimator by the ML algorithm under Assumptions (A1), (A3), (A7) and (A8).

3.5.4. Heteroscedasticity, Variance Model Selection and AIC

Besides Assumption (A8) about the structure of the covariance matrix, we assume that the errors are uncorrelated with possibly different variances (= heteroscedastic errors), that is, $\mathsf{\Sigma }=\mathrm{diag}\left({\sigma }_1^2,\dots ,{\sigma }_N^2\right)$. We model each variance ${\sigma }_i^2,i=1,\dots ,N$, by a twice differentiable function in dependence of parameters $\mathit{\alpha }={\left({\alpha }_0,\dots ,{\alpha }_{M-1}\right)}^T$ and a suitable set of linearly independent basis functions $e_m\left(X\right)\in L^2\left({\mathbb{R}}^D,\mathcal{B},{\mathbb{P}}^{\prime }\right),m=0,1,\dots ,M-1$, with ${\mathbf{v}}^i={\left(e_0\left(x^i\right),\dots ,e_{M-1}\left(x^i\right)\right)}^T$, that is,

$${\sigma }_i^2={\sigma }^{2}V\left[\mathit{\alpha },{\mathbf{v}}^i\right],$$

(42)

where $V\left[\mathit{\alpha },{\mathbf{v}}^i\right]$ is referred to as the variance function in analogy to $V\left[\mu \right]$ for GLMs and GAMs. Without loss of generality, we set again ${\sigma }^2=1$.

Hartmann (2015) has already applied FGLS regression with different variance models in the LSMC framework. In her numerical examples, variance models with multiplicative heteroscedasticity led to the best performance of the proxy function in the validation. Therefore, we restrict our analyis on these kinds of structures, compare, for example, Harvey (1976), that is,

$$V\left[\mathit{\alpha },{\mathbf{v}}^i\right]=exp\left({\mathbf{v}}^{iT}\mathit{\alpha }\right).$$

(43)

Like the proxy function, the variance function (43) has to be calibrated to apply FGLS regression, which means that the variance function has to be composed of suitable basis functions. Again, such a composition can be found with the aid of a model selection criterion. We still choose AIC, but have to take care for the fact that in FGLS regression the covariance matrix now contains M unknown parameters instead of only one in the OLS case (the same variance for all observations). Under Assumption (A7), AIC is given as

$$\begin{array}{cc}\hfill \mathrm{AIC}& =-2l\left({\widehat{\mathit{\beta }}}_{\mathrm{FGLS}},\widehat{\mathsf{\Sigma }}\right)+2\left(K+M\right)\hfill \\ \hfill & =Nlog\left(2\pi \right)+log\left(det\widehat{\mathsf{\Sigma }}\right)+{\left(\mathbf{y}-Z{\widehat{\mathit{\beta }}}_{\mathrm{FGLS}}\right)}^T{\widehat{\mathsf{\Sigma }}}^{-1}\left(\mathbf{y}-Z{\widehat{\mathit{\beta }}}_{\mathrm{FGLS}}\right)+2\left(K+M\right).\hfill \end{array}$$

(44)

When using a variance model with multiplicative heteroscedasticity, AIC becomes

$$\mathrm{AIC}=Nlog\left(2\pi \right)+\left(\sum _{i=1}^N{\mathbf{v}}^{iT}\right)\widehat{\mathit{\alpha }}+\sum _{i=1}^{N}exp\left(-{\mathbf{v}}^{iT}\widehat{\mathit{\alpha }}\right){\left({\widehat{ϵ}}^i\right)}^2+2\left(K+M\right).$$

(45)

As an alternative or complement, the basis functions of the variance model can be selected with respect to their correlations with the final OLS residuals or based on graphical residual analysis.

For the final implementation of a variance model we use modified versions of two algorithms from Hartmann (2015). Our type I variant starts with the derivation of the proxy function by the standard adaptive OLS regression approach and then selects the variance model adaptively from the set of proxy basis functions of which the exponents sum up to at most two. The type II variant builds on the type I algorithm by taking the resulting variance model as given in its adaptive proxy basis function selection procedure with FGLS regression in each iteration.

Note further, that we should only apply FGLS regression as a substitute of OLS regression if heteroscedasticity prevails. This can be tested with the help of the Breusch-Pagan test of Breusch and Pagan (1979) for the following special structure of the variance function

$$V\left[\mathit{\alpha },{\mathbf{v}}^i\right]=h\left({\mathbf{v}}^{i,T}\mathit{\alpha }\right),$$

(46)

where the function $h\left(·\right)$ is twice differentiable and the first element of ${\mathbf{v}}^i$ is ${\mathrm{v}}_0^i=1$. Further, the assumption of normally distributed errors is made. We use it in the numerical computations to check if heteroscedasticity still prevails during the iteration procedure.

3.6. Multivariate Adaptive Regression Splines (MARS)

3.6.1. The Regression Model

The multivariate adaptive regression splines (MARS) were introduced by Friedman (1991). The classical MARS model is a form of the classical linear regression model (7) where the basis functions $e_k\left(x^i\right)$ are so-called hinge functions. Therefore, the theory of OLS regression applies in this context. GLMs (12) can also be applied in conjunction with MARS models. In this case we speak of generalized MARS models.

We describe the standard MARS algorithm in the LSMC routine according to Chapter 9.4 of Hastie et al. (2017). The building blocks of MARS proxy functions are reflected pairs of piecewise linear functions with knots t as depicted in Figure 5, that is,

$$\begin{array}{c}\hfill {\left(X_d-t\right)}_{+}=max\left(X_d-t,0\right),{\left(t-X_d\right)}_{+}=max\left(t-X_d,0\right),\end{array}$$

(47)

where the $X_d,d=1,\dots ,D$, represent the risk factors that together form the outer scenario $X={\left(X_1,\dots ,X_D\right)}^T$.

For each risk factor, reflected pairs with knots at each fitting scenario stress $x_d^i,i=1,\dots ,N$, are defined. All pairs are united in the following collection serving as the initial candidate basis function set of the MARS algorithm, that is,

$$C_1={\left\{{\left(X_d-t\right)}_{+},{\left(t-X_d\right)}_{+}\right\}}_{t\in \left\{x_d^1,x_d^2,\dots ,x_d^N\right\}\mid d=1,\dots ,D}.$$

(48)

We call the elements of $C_1$ hinge functions and consider them as functions $h\left(X\right)$ over the entire input space ${\mathbb{R}}^D$. $C_1$ contains in total $2DN$ basis functions.

The adaptive basis function selection algorithm now consists of two parts, the forward and the backward pass.

3.6.2. Adaptive Forward Stepwise Selection and Forward Pass

The forward pass of the MARS algorithm can be viewed as a variation of the adaptive forward stepwise algorithm depicted in Figure 2. The start proxy function consists only of the intercept, that is, $h_0\left(X\right)=1$. In the classical MARS model, the regression method of choice is the standard OLS regression approach with the estimator (8), where in each iteration a reflected pair of hinge functions is selected instead of $e_k\left(x^i\right)$. Similarly, the regression method of choice in the generalized MARS model is the IRLS algorithm (18). Let us denote the MARS coefficient estimator by ${\widehat{\mathit{\beta }}}_{\mathrm{MARS}}$. Note that the theory on AIC cannot be transferred without any adjustments since the notion of the degrees of freedom has to be reconsidered due to the knots in the hinge functions acting as additional degrees of freedom.

After each iteration, the set of candidate basis functions is extended by the products of the last two selected hinge functions with all hinge functions in $C_1$ that depend on risk factors of which the last two selected hinge functions do not depend on. Let the reflected pair selected in the first iteration ($k=1$) be

$$\begin{array}{c}\hfill h_1\left(X\right)={\left(X_{d_1}-t_1\right)}_{+},\\ \hfill h_2\left(X\right)={\left(t_1-X_{d_1}\right)}_{+}.\end{array}$$

(49)

Further, let $C_{1,-}=C_1\setminus \left\{h_1\left(X\right),h_2\left(X\right)\right\}$. Then, the set of candidate basis functions is updated at the beginning of the second iteration ($k=2$) such that

$$\begin{array}{cc}\hfill C_2=C_{1,-}& \cup {\left\{{\left(X_d-t\right)}_{+}{h}_1\left(X\right),{\left(t-X_d\right)}_{+}{h}_1\left(X\right)\right\}}_{t\in \left\{x_d^1,x_d^2,\dots ,x_d^N\right\}\mid d=1,\dots ,D,d\ne d_1}\hfill \\ \hfill & \cup {\left\{{\left(X_d-t\right)}_{+}{h}_2\left(X\right),{\left(t-X_d\right)}_{+}{h}_2\left(X\right)\right\}}_{t\in \left\{x_d^1,x_d^2,\dots ,x_d^N\right\}\mid d=1,\dots ,D,d\ne d_1}.\hfill \end{array}$$

(50)

The second set $C_2$ thus contains $2\left(DN-1\right)+4\left(D-1\right)N$ basis functions. Often, the order of interaction is limited to improve the interpretability of the proxy functions. Besides the maximum allowed number of terms, a minimum threshold for the decrease in the residual sum of squares can be employed as a termination criterion in the forward pass. Typically, the proxy functions generated in the forward pass overfit the data since model complexity is only penalized conservatively by stipulating a maximum number of basis functions and a minimum threshold.

3.6.3. Backward Pass and GCV

Due to the overfitting tendency of the proxy function generated in the forward pass, a backward pass is executed afterwards. Apart from the direction and slight differences, the backward pass is similar to the forward pass. In each iteration, the hinge function of which the removal causes the smallest increase in the residual sum of squares is removed and the backward model selection criterion for the resulting proxy function is evaluated. By this backward procedure, we generate the “best” proxy functions of each size in terms of the residual sum of squares. Out of all these best proxy functions, we finally select the one which minimizes the backward model selection criterion. As a result, the final proxy function will not only contain reflected pairs of hinge functions but also single hinge functions of which the complements have been removed. Optionally, the backward pass can also be omitted.

Let the number of basis functions in the MARS model be K and the number of knots be T. The standard choice for the backward model selection criterion is GCV defined as

$$\mathrm{GCV}=\frac{ND\left({\widehat{\mathit{\beta }}}_{\mathrm{MARS}}\right)}{{\left(N-\mathrm{df}\right)}^2},$$

(51)

with the effective degrees of freedom $\mathrm{df}=K+3T$.

An especially fast MARS algorithm was later developed by Friedman (1993) and is implemented, for example, in function earth($·$) of R package earth provided by Milborrow (2018).

3.7. Kernel Regression

3.7.1. The One-dimensional Regression Model

Kernel regression (which goes back to Nadaraya (1964) and Watson (1964)) is a type of locally weighted OLS regression where the weights vary with the input variable (the target scenario). We start with locally constant (LC) regression where for each $x_0\in \mathbb{R}$ the fixed univariate kernel with given bandwidth$\lambda >0$ be

$$K_{\lambda }\left(x_0,x^i\right)=D\left(\frac{\left|x^i-x_0\right|}{\lambda }\right),$$

(52)

where $D\left(·\right)$ denotes the specified kernel function. Solving the corresponding least squares problem

$${\widehat{\mathit{\beta }}}_{\mathrm{LC}}\left(x_0\right)=\underset{\mathit{\beta }\left(x_0\right)\in \mathbb{R}}{arg\; min}\left\{\sum _{i=1}^N{K}_{\lambda }\left(x_0,x^i\right){\left(y^i-{\beta }_0\left(x_0\right)\right)}^2\right\},$$

(53)

one obtains the Nadaraya-Watson kernel smoother as the kernel-weighted average at each $x_0$ over the fitting values $y^i$, that is,

$${\widehat{f}}_{\mathrm{LC}}\left(x_0\right)={\widehat{\beta }}_{\mathrm{LC}}\left(x_0\right)=\frac{{\sum }_{i=1}^N{K}_{\lambda }\left(x_0,x^i\right)y^i}{{\sum }_{i=1}^N{K}_{\lambda }\left(x_0,x^i\right)}.$$

(54)

Typical examples for the fixed kernel are the Epanechnikov (see the green shaded areas of Figure 6 inspired by Hastie et al. (2017)), tri-cube and uniform kernels or gaussian kernel. Note that a kernel smoother is continuous and varies over the domain of the target scenarios $x_0$, it needs to be estimated separately at all of them.

The bias at the boundaries of the domain of the LC kernel estimator (53) (see the left panel of Figure 6) is mainly eliminated by fitting locally linear functions instead of locally constant functions, see the right panel of Figure 6. At each target $x_0$, the LL kernel estimator is defined as the minimizer of the kernel-weighted residual sum of squares, that is,

$${\widehat{\mathit{\beta }}}_{\mathrm{LL}}\left(x_0\right)=\underset{\mathit{\beta }\left(x_0\right)\in {\mathbb{R}}^2}{arg\; min}\left\{\sum _{i=1}^N{K}_{\lambda }\left(x_0,x^i\right){\left(y^i-{\beta }_0\left(x_0\right)-{\beta }_1\left(x_0\right)x^i\right)}^2\right\},$$

(55)

with $\mathit{\beta }\left(x_0\right)={\left({\beta }_0\left(x_0\right),{\beta }_1\left(x_0\right)\right)}^T$. The proxy function at $x_0$ is given by

$${\widehat{f}}_{\mathrm{LL}}\left(x_0\right)={\widehat{\beta }}_{\mathrm{LL},0}\left(x_0\right)+{\widehat{\beta }}_{\mathrm{LL},1}\left(x_0\right)x_0.$$

(56)

Again the minimization problem (55) must be solved separately for all target scenarios so that the coefficients of the proxy function vary across their domain. For each target scenario $x_0$ a weighted least-squares (WLS) problem with weights $K_{\lambda }\left(x_0,x^i\right)$ has to be solved. Its solution is the WLS estimator

$${\widehat{\mathit{\beta }}}_{\mathrm{LL}}\left(x_0\right)={\left(Z^{T}W\left(x_0\right)Z\right)}^{-1}{Z}^{T}W\left(x_0\right)\mathbf{y},$$

(57)

with $\mathbf{y}$ the response vector, $W\left(x_0\right)=\mathrm{diag}\left(K_{\lambda }\left(x_0,x^1\right),\dots ,K_{\lambda }\left(x_0,x^N\right)\right)$ the weight matrix and Z the design matrix which contains row-wise the vectors ${\left(1,x^i\right)}^T$. We call H the hat matrix if $\widehat{\mathbf{y}}=H\mathbf{y}$ such that $\widehat{\mathbf{y}}={\left({\widehat{f}}_{\mathrm{LL}}\left(x^1\right),\dots ,{\widehat{f}}_{\mathrm{LL}}\left(x^N\right)\right)}^T$ contains the proxy function values at their target scenarios.

When we use proxy functions in LL regression that are composed of polynomial basis functions with exponents greater than one, we could also speak of local polynomial regression.

3.7.2. The Multidimensional Regression Model

We generalize LC regression to ${\mathbb{R}}^K$ by expressing the kernel with respect to the basis function vector $\mathbf{z}={\left(e_0\left(X\right),\dots ,e_{K-1}\left(X\right)\right)}^T$ following from the adaptive forward stepwise selection with OLS regression and small $K_{\max }$. At each target scenario vector ${\mathbf{z}}_0\in {\mathbb{R}}^K$ with elements $z_{0k}$, basis function vector ${\mathbf{z}}^i\in {\mathbb{R}}^K$ with elements $z_{ik}$ evaluated at fitting scenario $x^i$ and given bandwidth vector $\mathit{\lambda }={\left({\lambda }_0,\dots ,{\lambda }_{K-1}\right)}^T$, the multivariate kernel is defined as the product of univariate kernels, that is,

$$K_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right)=\prod _{k=0}^{K-1}D\left(\frac{\left|z_{ik}-z_{0k}\right|}{{\lambda }_k}\right).$$

(58)

The LC kernel estimator in ${\mathbb{R}}^K$ is defined at each ${\mathbf{z}}_0$ as

$${\widehat{f}}_{\mathrm{LC}}\left({\mathbf{z}}_0\right)={\widehat{\beta }}_{\mathrm{LC}}\left({\mathbf{z}}_0\right)=\frac{{\sum }_{i=1}^N{K}_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right)y^i}{{\sum }_{i=1}^N{K}_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right)}.$$

(59)

Since we let $e_0\left(X\right)$ represent the intercept so that $z_{i0}=z_{00}=1$, the corresponding univariate kernel $D\left(\frac{\left|z_{i0}-z_{00}\right|}{{\lambda }_0}\right)=D\left(0\right)$ is constant over all fitting points, thus cancels in (59) and can be omitted in (58).

The LL kernel estimator in ${\mathbb{R}}^K$ is given as the multidimensional analogue of (55) at each ${\mathbf{z}}_0$, that is,

$${\widehat{\mathit{\beta }}}_{\mathrm{LL}}\left({\mathbf{z}}_0\right)=\underset{\mathit{\beta }\left({\mathbf{z}}_0\right)\in {\mathbb{R}}^K}{arg\; min}\left\{\sum _{i=1}^N{K}_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right){\left(y^i-{\mathbf{z}}^{i,T}\mathit{\beta }\left({\mathbf{z}}_0\right)\right)}^2\right\},$$

(60)

with $\mathit{\beta }\left({\mathbf{z}}_0\right)={\left({\beta }_0\left({\mathbf{z}}_0\right),\dots ,{\beta }_{K-1}\left({\mathbf{z}}_0\right)\right)}^T$ and the proxy function at ${\mathbf{z}}_0$ is given by

$${\widehat{f}}_{\mathrm{LL}}\left({\mathbf{z}}_0\right)={\mathbf{z}}_0^T{\widehat{\mathit{\beta }}}_{\mathrm{LL}}\left({\mathbf{z}}_0\right).$$

(61)

The LL kernel estimator can again be computed by WLS regression, that is,

$${\widehat{\mathit{\beta }}}_{\mathrm{LL}}\left({\mathbf{z}}_0\right)={\left(Z^{T}W\left({\mathbf{z}}_0\right)Z\right)}^{-1}{Z}^{T}W\left({\mathbf{z}}_0\right)\mathbf{y},$$

(62)

where $W\left({\mathbf{z}}_0\right)=\mathrm{diag}\left(K_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^1\right),\dots ,K_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^N\right)\right)$ is the weight matrix and Z the design matrix containing row-wise the vectors ${\mathbf{z}}^{i,T}$. The hat matrix H satisfies $\widehat{\mathbf{y}}=H\mathbf{y}$ with $\widehat{\mathbf{y}}={\left({\widehat{f}}_{\mathrm{LL}}\left({\mathbf{z}}^1\right),\dots ,{\widehat{f}}_{\mathrm{LL}}\left({\mathbf{z}}^N\right)\right)}^T$ containing the proxy function values at their target scenario vectors.

3.7.3. Bandwidth Selection, AIC and LOO-CV

The bandwidths ${\lambda }_k$ in kernel regression can be selected similarly to the smoothing parameters in GAMs by minimization of a suitable model selection criterion. In fact, kernel smoothers can be interpreted as local non-parametric GLMs with identity link functions. More precisely, at each target scenario the kernel smoother can be viewed as a GLM (12) where the parametric weights $V\left[{\widehat{\mu }}_{\mathrm{GLM}}^i\right]$ in (20) are the non-parametric kernel weights $K_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right)$ in (60). Since GLMs are special cases of GAMs and the bandwidths in kernel regression can be understood as smoothing parameters, kernel smoothers and GAMs are sometimes lumped together in one category. If the numbers N of the fitting points and K of the basis functions are large, from a computational perspective it might be beneficial to perform bandwidth selection based on a reduced set of fitting points.

Hurvich et al. (1998) propose to select the bandwidths ${\lambda }_1,\dots ,{\lambda }_{K-1}$ based on an improved version of AIC which works in the context of non-parametric proxy functions that can be written as linear combinations of the observations. It has the form

$$\mathrm{AIC}=log\left({\widehat{\sigma }}^2\right)+\frac{1+\mathrm{tr}\left(H\right)/N}{1-\left(\mathrm{tr}\left(H\right)+2\right)/N},$$

(63)

where ${\widehat{\sigma }}^2=\frac{1}{N}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)$ and H is the hat matrix.

As an alternative, leave-one-out cross-validation (LOO-CV) is suggested by Li and Racine (2004) for bandwidth selection. Let us refer to

$${\widehat{\mathit{\beta }}}_{\mathrm{LL},-j}\left({\mathbf{z}}_0\right)=\underset{\mathit{\beta }\left({\mathbf{z}}_0\right)\in {\mathbb{R}}^K}{arg\; min}\left\{\sum _{i\ne j,i=1}^N{K}_{\mathit{\lambda }}\left({\mathbf{z}}_0,{\mathbf{z}}^i\right){\left(y^i-{\mathbf{z}}^{i,T}\mathit{\beta }\left({\mathbf{z}}_0\right)\right)}^2\right\}$$

(64)

as the leave-one-out LL kernel estimator and to ${\widehat{f}}_{\mathrm{LL},-j}\left({\mathbf{z}}_0\right)={\mathbf{z}}_0^T{\widehat{\mathit{\beta }}}_{\mathrm{LL},-j}\left({\mathbf{z}}_0\right)$ as the leave-one-out proxy function at ${\mathbf{z}}_0$. The objective of LOO-CV is to choose the bandwidths ${\lambda }_1,\dots ,{\lambda }_{K-1}$ which minimize

$$\mathrm{CV}=\frac{1}{N}\sum _{i=1}^N{\left(y^i-{\widehat{f}}_{\mathrm{LL},-i}\left({\mathbf{z}}_0\right)\right)}^2.$$

(65)

3.7.4. Adaptive Forward Stepwise OLS Selection

A practical implementation of kernel regression can be found, for example, via the combination of functions npreg($·$) and npregbw($·$) from R package np of Racine and Hayfield (2018).

In the other sections, basis function selection depends on the respective regression methods. Since the crucial process of bandwidth selection in kernel regression takes a very long time in the implementation of our choice, it would be infeasible to proceed here in the same way. Therefore, we derive the basis functions for LC and LL regression by adaptive forward stepwise selection based on OLS regression, by risk factor wise linear selection or a combination thereof. Thereby, we keep the maximum allowed number $K_{\max }$ of terms rather small as we aim to model the subtleties by kernel regression.

4. Numerical Experiments

4.1. General Remarks

4.1.1. Data Basis

In our slightly disguised real-world example, the life insurance company has a portfolio with a large proportion of traditional German annuity business. This choice was made in order to challenge the regression techniques since German traditional annuity business features high interest rate guarantees which may lead to large losses in low interest rate environments. We let the insurance company be exposed to $D=15$ relevant financial and actuarial risk factors. For the derivation of the fitting points, we run its CFP model conditional on $N=25,000$ fitting scenarios with each of these outer scenarios entailing two antithetic inner simulations. For a subset of the resulting fitting values of the best estimate of liabilities (BEL), see Figure 1, for summary statistics, the left column of Table 1, and for a histogram, the left panel of Figure 7.

The Sobol validation set is generated based on $L=51$ validation scenarios with 1000 inner simulations, comprising 26 Sobol scenarios, 15 one-dimensional risk scenarios, 1 base scenario and 9 scenarios that turned out to be capital region scenarios in the previous year risk capital calculations. The nested simulations set which is due to its high computational costs not available in the regular LSMC approach reflects the highest $5\%$ real-world losses and is based on $L=1638$ outer scenarios with respectively 4000 inner simulations. From the 1638 real-world scenarios, 14 exhibit extreme stresses far beyond the bounds of the fitting space and are therefore excluded from the analysis. For the remaining nested simulation values of BEL, see Figure 3, for summary statistics, the right column of Table 1, and for a histogram, the right panel of Figure 7. The capital region set consists of the $L=129$ nested simulations points which correspond to the nested simulations SCR estimate ($=99.5\%$ highest loss) and the 64 losses above and below ($=99.3\%$ to $99.7\%$ highest losses).

4.1.2. Validation Figures

We will output validation figure (1) with respect to the relative and asset metric, and additionally figures (2)–(4). While figures (3) and (4) are evaluated with respect to a base value resulting from 1000 inner simulations on the Sobol set, that is, ${\mathrm{v}.\mathrm{mae}}^0$, ${\mathrm{v}.\mathrm{res}}^0$, they are computed with respect to a base value resulting from $16,000$ inner simulations on the nested simulations set, that is, ${\mathrm{ns}.\mathrm{mae}}^0$, ${\mathrm{ns}.\mathrm{res}}^0$, and capital region set, that is, ${\mathrm{cr}.\mathrm{mae}}^0$, ${\mathrm{cr}.\mathrm{res}}^0$. The latter base value is supposed to be the more reliable validation value since it is the one associated with a lower standard error. Therefore it is worth noting here that figure ${\mathrm{v}.\mathrm{res}}^0$ can easily be transformed such that it is also evaluated with respect to the latter base value by subtracting from it the difference of 14 which the two different base values incur. We will not explicitly state the base residual (5) as it is just (2) minus (4).

4.1.3. Economic Variables

We derive the OLS proxy functions for two economic variables, namely for the best estimate of liabilities (BEL) and the available capital (AC) over a one-year risk horizon, that is, $Y\left(X\right)\in \left\{\mathrm{BEL}\left(X\right),\mathrm{AC}\left(X\right)\right\}$. Their approximation quality is assessed by validation figures (1) with respect to the relative and asset metric and (2). Essentially, AC is obtained as the market value of assets minus BEL, which means that AC reflects the negative behavior of BEL. Therefore, we will only derive BEL proxy functions with the other regression methods. The profit resulting from a certain risk constellation captured by an outer scenario X can be computed as $\mathrm{AC}\left(X\right)$ minus the base AC. Validation figures (3) and (4) address the approximation quality of this difference. Taking the negative of the profit yields the loss and evaluating the loss at all real-world scenarios the real-world loss distribution from which the SCR is derived as the $99.5\%$ value-at-risk. The out-of-sample performances of two different OLS proxy functions of BEL on the Sobol, nested simulations and capital region sets serve as the benchmark for the other regression methods.

4.1.4. Numerical Stability

Let us discuss the subject of numerical stability of QR decompositions in the OLS regression design under a monomial basis. If the weighting in the weighted least-squares problems associated with GLMs, heteroscedastic FGLS regression and kernel regression is good-natured, similar arguments apply as they can also be solved via QR decompositions according to Green (1984) where the weighting is just a scaling. However, the weighting itself raises additional numerical questions that need to be taken into consideration when making the regression design choices. In GLMs, these choices are the random component (13) and link function (12), in FGLS regression it is the functional form of the heteroscedatic variance model (42) and in kernel regression it is the kernel function (58). The following arguments do not apply to GAMs and MARS models as these are constructed out of spline functions, see (25) and (47), respectively. In GAMs, the penalty matrix increases numerical stability.

McLean (2014) justifies that from the perspective of numerical stability performing a QR decomposition on a monomial design matrix Z is asymptotically equivalent to using a Legendre design matrix $Z^{\prime }$ and transforming the resulting coefficient estimator into the monomial one. Under the assumption of an orthonormal basis, Weiß and Nikolić (2019) have derived an explicit upper bound for the condition number of non-diagonal matrix $\frac{1}{N}{\left(Z^{\prime }\right)}^T\left(Z^{\prime }\right)$ for $N<\infty$, where the factor $\frac{1}{N}$ is used for technical reasons. This upper bound increases in (1) the number of basis functions, (2) the Hardy-Krause variation of the basis, (3) the convergence constant of the low-discrepancy sequence, and (4) the outer scenario dimension. Our previously defined type of restriction setting controls aspect (1) through the specification of $K_{\max }$ and aspect (2) through the limitation of exponents $d_1{d}_2{d}_3$. Aspects (3) and (4) are beyond the scope of the calibration and validation steps of the LSMC framework and therefore left aside here.

4.1.5. Interpolation and Extrapolation

In the LSMC framework, let us refer by interpolation to prediction inside the fitting space and by extrapolation to prediction outside the fitting space. Runge (1901) found that high-degree polynomial interpolation at equidistant points can oscillate toward the ends of the interval with the approximation error getting worse the higher the degree is. In a least-squares problem, Runge’s phenomenon was shown by Dahlquist and Björck (1974) not to apply to polynomials of degree d fitted based on N equidistant points if the inequality $d<2\sqrt{N}$ holds. With N = 25,000 fitting points the inequality becomes $d<316$ so that we clearly do not have to impose any further restrictions in OLS, FGLS and kernel regression as well as in GLMs to keep this phenomenon under control. Splines as they occur in GAMs and MARS models do not suffer from this oscillation issue by construction.

Since Runge’s phenomenon concerns the ends of the interval and the real-world scenarios for the insurer’s full loss distribution forecast in the fourth step of the LSMC framework partly go beyond the fitting space, its scope comprises the extrapolation area as well. High-degree polynomial extrapolation can worsen the approximation error and play a crucial role if many real-world scenarios go far beyond the fitting space.

4.1.6. Principle of Parsimony

Another problem that can occur in an adaptive algorithm is overfitting. Burnham and Anderson (2002) state that overfitted models often have needlessly large sampling variances which means that their precision of the predictions is poorer than that of more parsimonious models which are also free of bias. In cases where AIC leads to overfitting, implementing restriction settings of the form $K_{\max }$ - $d_1{d}_2{d}_3$ becomes relevant for adhering to the principle of parsimony.

4.2. Ordinary Least-Squares (OLS) Regression

4.2.1. Settings

We build the OLS proxy functions (10) of $Y\left(X\right)\in \left\{\mathrm{BEL}\left(X\right),\mathrm{AC}\left(X\right)\right\}$ with respect to an outer scenario X out of monomial basis functions that can be written as $e_k\left(X\right)={\prod }_{l=1}^{15}{X}_l^{r_k^l}$ with $r_k^l\in {\mathbb{N}}_0$ so that each basis function can be represented by a 15-tuple $\left(r_k^1,\dots ,r_k^{15}\right)$. The final proxy function depends on the restrictions applied in the adaptive algorithm. The purpose of setting restrictions is to guarantee numerical stability, to keep the extrapolation behavior under control and the proxy functions parsimonious. In order to illustrate the impact of restrictions, we run the adaptive algorithm for BEL under two different restriction settings with the second one being so relaxed that it will not take effect in our example. Additionally, we run the adaptive algorithm under the first restriction setting for AC to give an example of how the behavior of BEL can transfer to AC. As the first ingredient of our restriction setting acts the maximum allowed number of terms $K_{\max }$. Furthermore, we limit the exponents in the monomial basis. Firstly we apply a uniform threshold to all exponents, that is, $r_k^l\le d_1$. Secondly we restrict the degree, that is, ${\sum }_{l=1}^{15}{r}_k^l\le d_2$. Thirdly we restrict the exponents in interaction basis functions, that is, if there are some $l_1\ne l_2$ with $r_k^{l_1},r_k^{l_2}>0$, we require $r_k^{l_1},r_k^{l_2}\le d_3$. Let us denote this type of restriction setting by $K_{\max }$ - $d_1{d}_2{d}_3$.

As the first and second restriction settings, we choose 150–443 and 300–886, respectively, motivated by Teuguia et al. (2014) who found in their LSMC example in Chapter 4 with four risk factors and 50,000 fitting scenarios entailing two inner simulations that the validation error computed based on 14 validation scenarios started to stabilize at degree 4 when using monomial or Legendre basis functions in different adaptive basis function selection procedures. Furthermore, they pointed out that the LSMC approach becomes infeasible for degrees higher than 12.

We apply R function lm($·$) implemented in R package stats of R Core Team (2018).

4.2.2. Results

Table A1. Ordinary least squares (OLS) proxy function of BEL derived under 150–443 in the adaptive algorithm with the final coefficients. Furthermore, Akaike information criterion (AIC) scores and out-of-sample mean absolute errors (MAEs) in % after each iteration.

k	${\mathit{r}}_{\mathit{k}}^1$	${\mathit{r}}_{\mathit{k}}^2$	${\mathit{r}}_{\mathit{k}}^3$	${\mathit{r}}_{\mathit{k}}^4$	${\mathit{r}}_{\mathit{k}}^5$	${\mathit{r}}_{\mathit{k}}^6$	${\mathit{r}}_{\mathit{k}}^7$	${\mathit{r}}_{\mathit{k}}^8$	${\mathit{r}}_{\mathit{k}}^9$	${\mathit{r}}_{\mathit{k}}^{10}$	${\mathit{r}}_{\mathit{k}}^{11}$	${\mathit{r}}_{\mathit{k}}^{12}$	${\mathit{r}}_{\mathit{k}}^{13}$	${\mathit{r}}_{\mathit{k}}^{14}$	${\mathit{r}}_{\mathit{k}}^{15}$	${\widehat{\mathit{\beta }}}_{\mathbf{OLS},\mathit{k}}$	AIC	v.mae	ns.mae	cr.mae
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	14,718.24	437,251	4.557	3.231	4.027
1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	7850.17	386,722	2.474	0.845	0.913
2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	−269.33	375,144	2.065	2.139	1.831
3	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	145.21	366,567	1.656	0.444	0.496
4	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	−5.36	358,894	1.647	1.006	0.556
5	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	434.04	355,732	1.635	0.853	0.469
6	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1753.4	354,318	1.679	0.956	0.374
7	0	0	0	0	0	0	0	2	0	0	0	0	0	0	0	19,145.78	349,759	1.234	0.491	0.628
8	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	33.33	347,796	0.999	0.34	0.594
9	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	868.25	346,444	0.912	0.357	0.602
10	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	30.59	345,045	0.839	0.389	0.650
11	1	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1.65	341,083	0.759	0.398	0.465
12	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	86.79	339,360	0.718	0.394	0.390
13	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	33.35	337,731	0.574	0.653	0.512
14	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	49.59	336,843	0.589	0.658	0.518
15	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	71.25	335,980	0.628	0.678	0.512
16	0	0	0	0	0	0	1	1	0	0	0	0	0	0	0	2667.92	335,351	0.609	0.671	0.503
17	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	96.43	334,876	0.579	0.701	0.545
18	1	0	0	0	0	0	0	1	0	0	0	0	0	0	1	−6.31	334,413	0.593	0.72	0.531
19	0	0	0	0	0	0	0	2	0	0	0	0	0	0	1	−47.09	333,904	0.562	0.621	0.474
20	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	48.93	333,447	0.565	0.597	0.454
21	1	0	0	0	0	0	0	2	0	0	0	0	0	0	0	−3,412.68	333,116	0.553	0.543	0.407
22	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0.02	332,806	0.562	0.478	0.358
23	2	0	0	0	0	0	0	0	0	0	0	0	0	0	1	−0.12	332,547	0.55	0.45	0.381
24	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	43.77	332,294	0.545	0.468	0.378
25	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	118.94	332,042	0.53	0.464	0.362
26	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	−1288.45	331,687	0.522	0.453	0.355
27	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	−44.72	331,405	0.525	0.444	0.343
28	0	0	0	0	0	0	0	3	0	0	0	0	0	0	0	−24,908.99	331,136	0.499	0.405	0.327
29	2	0	0	0	0	0	0	1	0	0	0	0	0	0	0	−86.88	330,562	0.504	0.348	0.268
30	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	0.55	330,361	0.518	0.418	0.264
31	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	77.26	330,163	0.512	0.443	0.272
32	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	24.78	329,988	0.508	0.443	0.264
33	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	14.33	329,834	0.477	0.491	0.286
34	0	1	0	0	0	0	0	0	0	0	0	0	0	0	1	−0.39	329,688	0.477	0.5	0.290
35	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	28.36	329,550	0.476	0.502	0.291
36	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	−370.92	329,442	0.472	0.499	0.288
37	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	−17.9	329,147	0.462	0.505	0.301
38	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	8574.53	329,043	0.472	0.518	0.3
39	0	0	0	0	0	1	0	1	0	0	0	0	0	0	1	−2.17	328,935	0.474	0.51	0.295
40	0	0	0	0	0	0	0	1	1	0	0	0	0	0	0	223.91	328,832	0.475	0.509	0.291
41	0	0	0	0	0	1	0	2	0	0	0	0	0	0	0	−1801.73	328,733	0.455	0.445	0.248
42	1	0	0	0	0	1	0	1	0	0	0	0	0	0	0	−102.1	327,927	0.372	0.345	0.237
43	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0.7	327,858	0.368	0.353	0.235
44	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0.56	327,792	0.366	0.352	0.233
45	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	−3034.32	327,729	0.365	0.356	0.228
46	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	−13,127.81	327,659	0.368	0.364	0.227
47	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	−17.54	327,603	0.368	0.366	0.226
48	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	−187.07	327,537	0.374	0.367	0.226
49	0	0	0	0	0	1	1	1	0	0	0	0	0	0	0	−300.54	327,483	0.369	0.367	0.230
50	1	0	0	0	0	1	0	0	0	0	0	0	0	0	1	−0.09	327,432	0.368	0.391	0.221
51	0	0	0	0	0	2	0	1	0	0	0	0	0	0	0	−60.84	327,382	0.359	0.39	0.228
52	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	−20.91	327,331	0.352	0.39	0.225
53	1	0	0	0	0	0	0	0	0	0	0	0	0	0	2	−0.0	327,287	0.346	0.377	0.206
54	0	0	0	0	0	0	0	1	0	0	0	0	0	0	2	−0.09	327,149	0.339	0.357	0.185
55	2	0	0	0	0	1	0	0	0	0	0	0	0	0	0	1.44	327,105	0.315	0.321	0.173
56	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1	−0.5	327,064	0.315	0.322	0.173
57	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	−6.06	327,025	0.322	0.317	0.175
58	0	0	0	0	0	0	1	2	0	0	0	0	0	0	0	−6,600.49	326,986	0.317	0.31	0.172
59	1	0	0	0	0	0	1	1	0	0	0	0	0	0	0	−407.57	326,823	0.308	0.302	0.183
60	0	0	1	0	0	0	0	2	0	0	0	0	0	0	0	3378.82	326,787	0.306	0.301	0.183
61	1	0	1	0	0	0	0	1	0	0	0	0	0	0	0	205.28	326,733	0.304	0.299	0.183
62	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	−18.73	326,700	0.306	0.299	0.182
63	0	0	1	0	0	1	0	1	0	0	0	0	0	0	0	175.39	326,668	0.304	0.296	0.182
64	0	0	0	0	0	0	0	0	0	0	0	1	0	0	1	−0.2	326,638	0.304	0.298	0.181
65	0	1	0	0	0	0	0	1	0	0	0	0	0	0	1	2.45	326,610	0.301	0.296	0.183
66	1	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0.11	326,572	0.297	0.299	0.180
67	2	0	0	0	0	1	0	1	0	0	0	0	0	0	0	−13.02	326,545	0.292	0.286	0.169
68	1	1	0	0	0	0	0	1	0	0	0	0	0	0	0	93.69	326,519	0.292	0.287	0.172
69	0	1	0	0	0	0	0	2	0	0	0	0	0	0	0	891.58	326,478	0.294	0.282	0.173
70	0	0	0	0	0	0	1	1	0	0	0	0	0	0	1	−6.21	326,453	0.291	0.281	0.175
71	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	−112.56	326,428	0.289	0.281	0.176
72	1	0	0	0	0	0	0	0	0	0	1	0	0	0	0	−5.27	326,398	0.284	0.282	0.173
73	1	0	0	0	0	0	0	3	0	0	0	0	0	0	0	1129.77	326,374	0.276	0.264	0.162
74	1	0	0	0	0	0	1	0	0	0	0	0	0	0	1	−0.29	326,352	0.272	0.266	0.158
75	1	0	0	0	0	0	0	1	1	0	0	0	0	0	0	−56.54	326,331	0.269	0.266	0.157
76	2	0	0	0	0	0	0	0	1	0	0	0	0	0	0	−3.02	326,313	0.271	0.266	0.155
77	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	−10.59	326,295	0.264	0.27	0.151
78	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	−6.99	326,278	0.264	0.275	0.153
79	1	0	0	0	0	2	0	0	0	0	0	0	0	0	0	−2.25	326,261	0.252	0.285	0.154
80	0	0	0	0	0	0	0	0	2	0	0	0	0	0	0	−14.77	326,245	0.263	0.309	0.157
81	2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	1.95	326,229	0.267	0.306	0.155
82	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	2248.54	326,214	0.266	0.307	0.156
83	0	0	0	0	0	0	0	3	0	0	0	0	0	0	1	−111.77	326,201	0.263	0.302	0.158
84	1	0	0	0	0	0	0	0	1	0	0	0	0	0	1	−0.11	326,187	0.262	0.302	0.157
85	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	−0.18	326,174	0.263	0.305	0.156
86	0	1	0	0	0	1	0	1	0	0	0	0	0	0	0	45.58	326,161	0.265	0.303	0.157
87	0	0	0	1	0	0	0	2	0	0	0	0	0	0	0	−83,291.89	326,149	0.267	0.308	0.156
88	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	−56.2	326,137	0.267	0.308	0.156
89	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0	−5.32	326,126	0.267	0.31	0.156
90	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	−10.87	326,116	0.267	0.313	0.158
91	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	−32.75	326,106	0.265	0.317	0.158
92	0	0	0	0	0	0	0	2	0	0	0	0	0	0	2	−0.09	326,097	0.265	0.308	0.151
93	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	10.87	326,089	0.265	0.308	0.151
94	1	0	0	0	0	1	1	1	0	0	0	0	0	0	0	−48.93	326,081	0.264	0.306	0.148
95	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	69.57	326,073	0.256	0.288	0.141
96	0	0	0	1	0	0	0	3	0	0	0	0	0	0	0	−542,688.19	326,066	0.256	0.289	0.141
97	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	10.44	326,058	0.248	0.275	0.136
98	0	0	0	0	0	0	0	1	1	0	0	0	0	0	1	−1.08	326,051	0.248	0.276	0.136
99	0	0	1	0	0	0	1	1	0	0	0	0	0	0	0	419.05	326,045	0.249	0.275	0.136
100	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	12.8	326,038	0.25	0.276	0.136
101	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	−3.94	326,033	0.25	0.276	0.136
102	1	0	0	0	0	0	0	2	0	0	0	0	0	0	1	−10.12	326,027	0.248	0.281	0.138
103	2	0	0	0	0	0	0	1	0	0	0	0	0	0	1	−0.36	326,017	0.244	0.283	0.135
104	0	0	1	0	0	0	0	1	0	0	0	0	0	0	1	1.74	326,012	0.244	0.282	0.136
105	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3	−0.0	326,006	0.242	0.268	0.132
106	2	0	0	0	0	0	1	0	0	0	0	0	0	0	0	−7.09	326,001	0.238	0.265	0.131
107	2	0	0	0	0	0	1	1	0	0	0	0	0	0	0	−109.46	325,982	0.238	0.263	0.129
108	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	−0.1	325,977	0.237	0.263	0.128
109	0	1	0	0	0	0	0	0	0	0	0	1	0	0	0	5.76	325,972	0.235	0.263	0.129
110	1	0	0	0	0	0	0	1	0	0	1	0	0	0	0	54.51	325,968	0.237	0.264	0.129
111	1	0	0	0	0	0	1	2	0	0	0	0	0	0	0	−1386.73	325,963	0.235	0.264	0.129
112	0	0	0	0	0	0	0	0	1	0	0	0	0	0	2	−0.0	325,959	0.237	0.265	0.13
113	0	1	0	0	0	0	0	0	1	0	0	0	0	0	1	0.11	325,955	0.235	0.265	0.13
114	0	1	0	0	0	1	0	0	0	0	0	0	0	0	1	0.05	325,951	0.234	0.266	0.13
115	1	0	1	0	0	1	0	0	0	0	0	0	0	0	0	4.3	325,948	0.236	0.265	0.127
116	1	0	0	0	0	2	0	1	0	0	0	0	0	0	0	−19.81	325,944	0.237	0.262	0.126
117	2	0	0	0	0	2	0	0	0	0	0	0	0	0	0	−0.87	325,938	0.241	0.267	0.124
118	0	1	0	0	0	1	0	1	0	0	0	0	0	0	1	−0.36	325,935	0.241	0.267	0.124
119	0	1	1	0	0	0	0	1	0	0	0	0	0	0	0	−80.29	325,931	0.241	0.267	0.125
120	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	−6.95	325,928	0.241	0.267	0.124
121	0	0	0	0	0	1	0	0	0	0	0	0	0	0	2	−0.0	325,925	0.243	0.259	0.121
122	0	0	0	0	0	0	0	2	0	0	1	0	0	0	0	436.56	325,923	0.241	0.259	0.121
123	0	0	0	0	0	2	0	0	0	0	0	0	0	0	1	−0.03	325,920	0.243	0.263	0.121
124	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	2.99	325,918	0.242	0.263	0.12
125	1	0	0	0	0	1	0	1	0	0	0	0	0	0	1	−0.59	325,916	0.241	0.261	0.119
126	2	0	0	0	0	1	0	0	0	0	0	0	0	0	1	−0.02	325,908	0.247	0.265	0.124
127	0	0	0	0	0	1	0	2	0	0	0	0	0	0	1	−4.66	325,902	0.249	0.279	0.123
128	0	0	0	0	0	0	1	3	0	0	0	0	0	0	0	−8179.68	325,900	0.249	0.28	0.124
129	0	0	0	0	0	1	0	3	0	0	0	0	0	0	0	691.4	325,898	0.249	0.28	0.123
130	1	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0.04	325,896	0.25	0.281	0.122
131	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	7.04	325,894	0.246	0.264	0.12
132	0	0	1	0	0	0	0	0	0	1	0	0	0	0	0	−27.72	325,892	0.247	0.264	0.119
133	2	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1.26	325,891	0.247	0.264	0.119
134	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	−2.67	325,889	0.249	0.265	0.118
135	1	0	0	0	0	1	0	0	0	0	0	1	0	0	0	1.53	325,887	0.25	0.266	0.119
136	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	−0.07	325,885	0.25	0.265	0.12
137	1	0	0	0	0	0	0	1	0	0	0	1	0	0	0	40.44	325,884	0.251	0.265	0.119
138	0	0	0	0	0	0	0	2	0	0	0	1	0	0	0	434.5	325,878	0.249	0.264	0.119
139	0	0	0	0	0	0	0	0	1	0	0	1	0	0	0	−5.99	325,877	0.248	0.264	0.119
140	0	0	0	0	0	0	0	0	2	0	0	1	0	0	0	14.64	325,873	0.246	0.263	0.12
141	0	0	0	0	0	2	0	2	0	0	0	0	0	0	0	−119.42	325,871	0.247	0.27	0.121
142	0	0	0	0	0	0	0	1	0	0	0	0	0	0	3	0.0	325,870	0.248	0.271	0.121
143	1	0	0	0	0	0	0	0	0	0	0	1	0	0	1	0.07	325,868	0.248	0.271	0.121
144	0	0	0	0	0	0	0	1	0	0	0	1	0	0	1	1.06	325,861	0.246	0.271	0.121
145	1	0	0	0	0	0	1	1	0	0	0	0	0	0	1	−0.74	325,859	0.247	0.271	0.121
146	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	−5.61	325,858	0.246	0.271	0.121
147	0	1	0	0	0	0	0	0	0	0	0	0	0	1	1	−0.08	325,857	0.247	0.27	0.121
148	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	−37.16	325,855	0.247	0.271	0.122
149	0	0	0	0	0	0	1	0	0	0	0	0	1	0	1	0.41	325,851	0.247	0.271	0.122
150	0	1	0	1	0	0	0	1	0	0	0	0	0	0	0	−7290.99	325,850	0.247	0.271	0.122

contains the final BEL proxy function derived under the first restriction setting 150–443 with the basis function representations and coefficients. Thereby reflect the rows the iterations of the adaptive algorithm and depict thus the sequence in which the basis functions are selected. Moreover, the iteration-wise AIC scores and out-of-sample MAEs (1) with respect to the relative metric in % on the Sobol, nested simulations and capital region sets are reported, that is, v.mae, ns.mae and cr.mae.

Table A2. OLS proxy function of available capital (AC) derived under 150–443 in the adaptive algorithm with the final coefficients. Furthermore, AIC scores and out-of-sample MAEs in % after each iteration.

k	${\mathit{r}}_{\mathit{k}}^1$	${\mathit{r}}_{\mathit{k}}^2$	${\mathit{r}}_{\mathit{k}}^3$	${\mathit{r}}_{\mathit{k}}^4$	${\mathit{r}}_{\mathit{k}}^5$	${\mathit{r}}_{\mathit{k}}^6$	${\mathit{r}}_{\mathit{k}}^7$	${\mathit{r}}_{\mathit{k}}^8$	${\mathit{r}}_{\mathit{k}}^9$	${\mathit{r}}_{\mathit{k}}^{10}$	${\mathit{r}}_{\mathit{k}}^{11}$	${\mathit{r}}_{\mathit{k}}^{12}$	${\mathit{r}}_{\mathit{k}}^{13}$	${\mathit{r}}_{\mathit{k}}^{14}$	${\mathit{r}}_{\mathit{k}}^{15}$	${\widehat{\mathit{\beta }}}_{\mathbf{OLS},\mathit{k}}$	AIC	v.mae	ns.mae	cr.mae
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	745.35	391,375	60.62	97.518	257.762
1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	5766.61	382,610	50.402	99.306	256.789
2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	272.75	367,667	35.285	38.124	99.902
3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	5.46	359,997	30.739	18.21	72.719
4	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	128.41	356,705	30.119	25.088	29.357
5	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	−1750.72	355,354	30.867	28.173	21.870
6	0	0	0	0	0	0	0	2	0	0	0	0	0	0	0	−19,127.27	351,002	22.942	14.948	44.668
7	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	−33.25	349,147	19.03	12.142	42.535
8	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	307.32	347,777	18.221	10.928	35.420
9	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	−868.05	346,423	16.662	11.527	35.941
10	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	−87.54	345,025	15.987	10.264	31.461
11	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	−30.51	343,570	14.858	11.187	34.502
12	1	0	0	0	0	0	0	0	0	0	0	0	0	0	1	−1.66	339,282	13.092	12.669	23.174
13	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	−33.33	337,648	10.427	20.976	30.402
14	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	−70.63	336,840	11.087	21.598	29.972
15	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	−41.37	336,120	11.436	21.764	30.408
16	0	0	0	0	0	0	1	1	0	0	0	0	0	0	0	−2666.44	335,495	11.088	21.543	29.890
17	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	−96.48	335,022	10.545	22.479	32.334
18	1	0	0	0	0	0	0	1	0	0	0	0	0	0	1	6.3	334,563	10.804	23.095	31.519
19	0	0	0	0	0	0	0	2	0	0	0	0	0	0	1	47.02	334,058	10.232	19.913	28.128
20	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	−48.77	333,610	10.292	19.163	26.995
21	1	0	0	0	0	0	0	2	0	0	0	0	0	0	0	3412.54	333,281	10.083	17.438	24.190
22	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2	−0.02	332,970	10.246	15.328	21.326
23	2	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0.12	332,714	10.02	14.436	22.671
24	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	−120.68	332,457	9.834	14.283	21.608
25	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	1287.63	332,108	9.725	13.969	21.273
26	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	44.71	331,832	9.755	13.661	20.501
27	0	0	0	0	0	0	0	3	0	0	0	0	0	0	0	24,899.66	331,569	9.275	12.462	19.873
28	2	0	0	0	0	0	0	1	0	0	0	0	0	0	0	87.04	331,004	9.292	10.757	17.022
29	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	−43.38	330,742	9.171	11.183	16.023
30	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	−0.55	330,543	9.444	13.409	15.766
31	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	−77.35	330,345	9.324	14.207	16.192
32	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	−25.2	330,161	9.246	14.203	15.692
33	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	−14.37	330,007	8.672	15.764	16.964
34	0	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0.39	329,859	8.682	16.031	17.223
35	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	−27.8	329,728	8.665	16.11	17.264
36	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	−8757.49	329,619	8.871	16.53	17.005
37	0	0	0	0	0	1	0	1	0	0	0	0	0	0	1	2.17	329,513	8.937	16.276	16.790
38	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	369.16	329,408	8.842	16.169	16.738
39	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	17.97	329,109	8.637	16.387	17.527
40	0	0	0	0	0	0	0	1	1	0	0	0	0	0	0	−222.55	329,008	8.656	16.359	17.271
41	0	0	0	0	0	1	0	2	0	0	0	0	0	0	0	1791.7	328,910	8.297	14.282	14.748
42	1	0	0	0	0	1	0	1	0	0	0	0	0	0	0	101.23	328,111	6.783	11.112	14.144
43	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	−0.7	328,041	6.713	11.355	14.013
44	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	−0.57	327,972	6.683	11.325	13.867
45	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	3083.05	327,905	6.654	11.456	13.595
46	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	12,863.79	327,837	6.7	11.721	13.5
47	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	17.78	327,780	6.71	11.777	13.450
48	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	190.46	327,711	6.824	11.818	13.468
49	0	0	0	0	0	1	1	1	0	0	0	0	0	0	0	300.76	327,657	6.724	11.793	13.716
50	1	0	0	0	0	1	0	0	0	0	0	0	0	0	1	0.09	327,607	6.718	12.565	13.182
51	0	0	0	0	0	2	0	1	0	0	0	0	0	0	0	60.83	327,557	6.543	12.533	13.558
52	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	20.91	327,507	6.415	12.53	13.394
53	1	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0.0	327,463	6.314	12.118	12.252
54	0	0	0	0	0	0	0	1	0	0	0	0	0	0	2	0.08	327,327	6.176	11.486	11.049
55	2	0	0	0	0	1	0	0	0	0	0	0	0	0	0	−1.46	327,284	5.751	10.339	10.295
56	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0.5	327,242	5.746	10.367	10.287
57	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	6.08	327,203	5.871	10.211	10.450
58	0	0	0	0	0	0	1	2	0	0	0	0	0	0	0	6593.98	327,165	5.78	9.973	10.274
59	1	0	0	0	0	0	1	1	0	0	0	0	0	0	0	406.73	327,003	5.618	9.722	10.897
60	0	0	1	0	0	0	0	2	0	0	0	0	0	0	0	−3,364.02	326,968	5.581	9.671	10.904
61	1	0	1	0	0	0	0	1	0	0	0	0	0	0	0	−204.12	326,914	5.542	9.626	10.921
62	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	18.9	326,881	5.588	9.611	10.837
63	0	0	1	0	0	1	0	1	0	0	0	0	0	0	0	−175.17	326,849	5.546	9.514	10.817
64	0	0	0	0	0	0	0	0	0	0	0	1	0	0	1	0.21	326,818	5.54	9.597	10.799
65	0	1	0	0	0	0	0	1	0	0	0	0	0	0	1	−2.44	326,791	5.494	9.532	10.896
66	1	1	0	0	0	0	0	0	0	0	0	0	0	0	1	−0.11	326,753	5.413	9.616	10.708
67	2	0	0	0	0	1	0	1	0	0	0	0	0	0	0	12.99	326,726	5.317	9.215	10.046
68	1	1	0	0	0	0	0	1	0	0	0	0	0	0	0	−93.57	326,700	5.329	9.255	10.231
69	0	1	0	0	0	0	0	2	0	0	0	0	0	0	0	−890.62	326,660	5.355	9.09	10.326
70	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	113.04	326,635	5.313	9.095	10.357
71	1	0	0	0	0	0	0	0	0	0	1	0	0	0	0	5.23	326,605	5.231	9.101	10.164
72	0	0	0	0	0	0	1	1	0	0	0	0	0	0	1	6.2	326,581	5.186	9.068	10.265
73	1	0	0	0	0	0	0	3	0	0	0	0	0	0	0	−1,133.83	326,556	5.034	8.488	9.647
74	1	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0.29	326,534	4.95	8.58	9.374
75	1	0	0	0	0	0	0	1	1	0	0	0	0	0	0	56.56	326,513	4.908	8.559	9.323
76	2	0	0	0	0	0	0	0	1	0	0	0	0	0	0	3.02	326,495	4.936	8.573	9.223
77	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	10.61	326,477	4.824	8.705	8.996
78	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	6.97	326,461	4.821	8.849	9.071
79	1	0	0	0	0	2	0	0	0	0	0	0	0	0	0	2.25	326,444	4.602	9.17	9.162
80	2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	−1.94	326,429	4.688	9.069	8.997
81	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	−2,257.4	326,414	4.676	9.099	9.070
82	0	0	0	0	0	0	0	0	2	0	0	0	0	0	0	14.06	326,399	4.853	9.831	9.278
83	1	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0.11	326,385	4.844	9.851	9.203
84	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0.18	326,372	4.861	9.935	9.174
85	0	0	0	0	0	0	0	3	0	0	0	0	0	0	1	111.58	326,358	4.796	9.769	9.270
86	0	1	0	0	0	1	0	1	0	0	0	0	0	0	0	−45.11	326,346	4.826	9.724	9.330
87	0	0	0	1	0	0	0	2	0	0	0	0	0	0	0	82,935.66	326,334	4.871	9.865	9.284
88	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	56.0	326,322	4.867	9.862	9.267
89	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0	5.35	326,311	4.857	9.938	9.258
90	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	10.88	326,301	4.87	10.043	9.414
91	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	32.81	326,291	4.833	10.156	9.394
92	1	0	0	0	0	1	1	1	0	0	0	0	0	0	0	48.96	326,283	4.812	10.085	9.185
93	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	−10.9	326,274	4.801	10.083	9.210
94	0	0	0	0	0	0	0	2	0	0	0	0	0	0	2	0.09	326,266	4.803	9.818	8.787
95	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	−69.45	326,258	4.659	9.25	8.413
96	0	0	0	1	0	0	0	3	0	0	0	0	0	0	0	543,840.26	326,251	4.663	9.269	8.393
97	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	−10.31	326,244	4.51	8.841	8.101
98	0	0	0	0	0	0	0	1	1	0	0	0	0	0	1	1.07	326,237	4.523	8.847	8.091
99	0	0	1	0	0	0	1	1	0	0	0	0	0	0	0	−417.88	326,231	4.531	8.84	8.101
100	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	−12.92	326,224	4.546	8.847	8.081
101	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	3.94	326,219	4.558	8.866	8.072
102	1	0	0	0	0	0	0	2	0	0	0	0	0	0	1	10.1	326,213	4.513	9.012	8.203
103	2	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0.36	326,204	4.453	9.084	8.035
104	0	0	1	0	0	0	0	1	0	0	0	0	0	0	1	−1.74	326,198	4.445	9.063	8.070
105	2	0	0	0	0	0	1	0	0	0	0	0	0	0	0	7.09	326,193	4.383	8.967	8.008
106	2	0	0	0	0	0	1	1	0	0	0	0	0	0	0	109.5	326,174	4.371	8.899	7.889
107	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3	0.0	326,169	4.332	8.454	7.669
108	0	1	0	0	0	0	0	0	0	0	0	1	0	0	0	−5.85	326,164	4.29	8.456	7.689
109	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0.1	326,159	4.282	8.457	7.657
110	1	0	0	0	0	0	0	1	0	0	1	0	0	0	0	−54.88	326,154	4.313	8.463	7.689
111	1	0	0	0	0	0	1	2	0	0	0	0	0	0	0	1380.74	326,150	4.291	8.489	7.7
112	0	0	0	0	0	0	0	0	1	0	0	0	0	0	2	0.0	326,146	4.315	8.498	7.751
113	0	1	0	0	0	0	0	0	1	0	0	0	0	0	1	−0.11	326,142	4.287	8.501	7.736
114	1	0	1	0	0	1	0	0	0	0	0	0	0	0	0	−4.3	326,138	4.32	8.461	7.558
115	0	1	0	0	0	1	0	0	0	0	0	0	0	0	1	−0.05	326,135	4.299	8.514	7.566
116	1	0	0	0	0	2	0	1	0	0	0	0	0	0	0	20.09	326,131	4.32	8.417	7.498
117	2	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0.87	326,125	4.393	8.561	7.371
118	0	1	0	0	0	1	0	1	0	0	0	0	0	0	1	0.36	326,122	4.389	8.564	7.409
119	0	1	1	0	0	0	0	1	0	0	0	0	0	0	0	79.51	326,118	4.394	8.56	7.411
120	0	0	0	0	0	1	0	0	0	0	0	0	0	0	2	0.0	326,115	4.43	8.304	7.187
121	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	6.91	326,113	4.42	8.305	7.176
122	0	0	0	0	0	0	0	2	0	0	1	0	0	0	0	−435.81	326,110	4.39	8.301	7.212
123	0	0	0	0	0	2	0	0	0	0	0	0	0	0	1	0.03	326,107	4.419	8.45	7.206
124	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	−2.99	326,105	4.407	8.434	7.163
125	1	0	0	0	0	1	0	1	0	0	0	0	0	0	1	0.59	326,103	4.394	8.366	7.095
126	2	0	0	0	0	1	0	0	0	0	0	0	0	0	1	0.02	326,096	4.502	8.499	7.382
127	0	0	0	0	0	1	0	2	0	0	0	0	0	0	1	4.66	326,089	4.543	8.962	7.340
128	0	0	0	0	0	1	0	3	0	0	0	0	0	0	0	−692.59	326,088	4.537	8.961	7.248
129	0	0	0	0	0	0	1	3	0	0	0	0	0	0	0	8097.7	326,086	4.539	8.995	7.316
130	1	0	0	0	0	0	0	0	0	0	0	0	1	0	1	−0.04	326,084	4.555	9.024	7.285
131	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	2.73	326,082	4.59	9.065	7.246
132	1	0	0	0	0	1	0	0	0	0	0	1	0	0	0	−1.53	326,080	4.612	9.097	7.280
133	2	0	0	0	0	0	0	0	0	0	1	0	0	0	0	−1.28	326,078	4.616	9.086	7.251
134	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	0.07	326,077	4.607	9.055	7.287
135	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	−6.96	326,075	4.533	8.527	7.230
136	0	0	1	0	0	0	0	0	0	1	0	0	0	0	0	27.74	326,073	4.556	8.52	7.115
137	0	0	0	0	0	2	0	2	0	0	0	0	0	0	0	122.08	326,071	4.571	8.746	7.171
138	0	0	0	0	0	0	0	0	1	0	0	1	0	0	0	6.0	326,070	4.556	8.745	7.190
139	0	0	0	0	0	0	0	0	2	0	0	1	0	0	0	−14.5	326,066	4.533	8.699	7.199
140	1	0	0	0	0	0	0	0	0	0	0	1	0	0	1	−0.07	326,064	4.532	8.722	7.227
141	0	0	0	0	0	0	0	1	0	0	0	1	0	0	1	−1.05	326,057	4.507	8.733	7.250
142	1	0	0	0	0	0	1	1	0	0	0	0	0	0	1	0.74	326,056	4.515	8.719	7.238
143	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	5.71	326,054	4.503	8.706	7.263
144	1	0	0	0	0	0	0	1	0	0	0	1	0	0	0	−39.87	326,053	4.499	8.715	7.244
145	0	0	0	0	0	0	0	2	0	0	0	1	0	0	0	−431.71	326,047	4.47	8.669	7.215
146	0	0	0	0	0	0	0	1	0	0	0	0	0	0	3	−0.0	326,046	4.488	8.698	7.207
147	0	1	0	0	0	0	0	0	0	0	0	0	0	1	1	0.08	326,045	4.494	8.694	7.223
148	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	37.33	326,043	4.496	8.703	7.236
149	0	0	0	0	0	0	1	0	0	0	0	0	1	0	1	−0.42	326,039	4.508	8.706	7.253
150	0	1	0	1	0	0	0	1	0	0	0	0	0	0	0	7224.25	326,038	4.512	8.712	7.265

contains the AC counterpart of the BEL proxy function derived under 150–443 and

Table A3. OLS proxy function of BEL derived under 300–886 in the adaptive algorithm with the final coefficients. Furthermore, AIC scores and out-of-sample MAEs in % after each iteration.

k	${\mathit{r}}_{\mathit{k}}^1$	${\mathit{r}}_{\mathit{k}}^2$	${\mathit{r}}_{\mathit{k}}^3$	${\mathit{r}}_{\mathit{k}}^4$	${\mathit{r}}_{\mathit{k}}^5$	${\mathit{r}}_{\mathit{k}}^6$	${\mathit{r}}_{\mathit{k}}^7$	${\mathit{r}}_{\mathit{k}}^8$	${\mathit{r}}_{\mathit{k}}^9$	${\mathit{r}}_{\mathit{k}}^{10}$	${\mathit{r}}_{\mathit{k}}^{11}$	${\mathit{r}}_{\mathit{k}}^{12}$	${\mathit{r}}_{\mathit{k}}^{13}$	${\mathit{r}}_{\mathit{k}}^{14}$	${\mathit{r}}_{\mathit{k}}^{15}$	${\widehat{\mathit{\beta }}}_{\mathbf{OLS},\mathit{k}}$	AIC	v.mae	ns.mae	cr.mae
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	14,689.75	437,251	4.557	3.231	4.027
1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	7990.98	386,722	2.474	0.845	0.913
2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	−274.24	375,144	2.065	2.139	1.831
3	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	145.73	366,567	1.656	0.444	0.496
4	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	−5.11	358,894	1.647	1.006	0.556
5	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	416.79	355,732	1.635	0.853	0.469
6	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	2332.91	354,318	1.679	0.956	0.374
7	0	0	0	0	0	0	0	2	0	0	0	0	0	0	0	24,914.36	349,759	1.234	0.491	0.628
8	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	49.42	347,796	0.999	0.34	0.594
9	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	859.49	346,444	0.912	0.357	0.602
10	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	29.5	345,045	0.839	0.389	0.65
11	1	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1.71	341,083	0.759	0.398	0.465
12	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	91.65	339,360	0.718	0.394	0.39
13	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	36.34	337,731	0.574	0.653	0.512
14	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	51.78	336,843	0.589	0.658	0.518
15	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	68.02	335,980	0.628	0.678	0.512
16	0	0	0	0	0	0	1	1	0	0	0	0	0	0	0	2661.47	335,351	0.609	0.671	0.503
17	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	109.14	334,876	0.579	0.701	0.545
18	1	0	0	0	0	0	0	1	0	0	0	0	0	0	1	−12.63	334,413	0.593	0.72	0.531
19	0	0	0	0	0	0	0	2	0	0	0	0	0	0	1	−114.48	333,904	0.562	0.621	0.474
20	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	35.4	333,447	0.565	0.597	0.454
21	1	0	0	0	0	0	0	2	0	0	0	0	0	0	0	−4570.15	333,116	0.553	0.543	0.407
22	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0.02	332,806	0.562	0.478	0.358
23	2	0	0	0	0	0	0	0	0	0	0	0	0	0	1	−0.26	332,547	0.55	0.45	0.381
24	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	47.17	332,294	0.545	0.468	0.378
25	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	123.47	332,042	0.53	0.464	0.362
26	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	−1,240.44	331,687	0.522	0.453	0.355
27	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	−43.82	331,405	0.525	0.444	0.343
28	0	0	0	0	0	0	0	3	0	0	0	0	0	0	0	−32,661.61	331,136	0.499	0.405	0.327
29	2	0	0	0	0	0	0	1	0	0	0	0	0	0	0	−140.9	330,562	0.504	0.348	0.268
30	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	0.56	330,361	0.518	0.418	0.264
31	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	87.33	330,163	0.512	0.443	0.272
32	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	25.31	329,988	0.508	0.443	0.264
33	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	14.22	329,834	0.477	0.491	0.286
34	0	1	0	0	0	0	0	0	0	0	0	0	0	0	1	−0.44	329,688	0.477	0.5	0.29
35	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	26.88	329,550	0.476	0.502	0.291
36	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	−391.81	329,442	0.472	0.499	0.288
37	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	−18.58	329,147	0.462	0.505	0.301
38	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	11,959.32	329,043	0.472	0.518	0.3
39	0	0	0	0	0	1	0	1	0	0	0	0	0	0	1	−2.15	328,935	0.474	0.51	0.295
40	0	0	0	0	0	0	0	1	1	0	0	0	0	0	0	228.32	328,832	0.475	0.509	0.291
41	0	0	0	0	0	1	0	2	0	0	0	0	0	0	0	−1938.37	328,733	0.455	0.445	0.248
42	1	0	0	0	0	1	0	1	0	0	0	0	0	0	0	−112.83	327,927	0.372	0.345	0.237
43	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0.71	327,858	0.368	0.353	0.235
44	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0.72	327,792	0.366	0.352	0.233
45	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	−4230.29	327,729	0.365	0.356	0.228
46	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	−10,720.3	327,659	0.368	0.364	0.227
47	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	−18.39	327,603	0.368	0.366	0.226
48	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	−212.78	327,537	0.374	0.367	0.226
49	0	0	0	0	0	1	1	1	0	0	0	0	0	0	0	−177.64	327,483	0.369	0.367	0.23
50	1	0	0	0	0	1	0	0	0	0	0	0	0	0	1	−0.09	327,432	0.368	0.391	0.221
51	0	0	0	0	0	2	0	1	0	0	0	0	0	0	0	−57.4	327,382	0.359	0.39	0.228
52	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	−23.55	327,331	0.352	0.39	0.225
53	1	0	0	0	0	0	0	0	0	0	0	0	0	0	2	−0.0	327,287	0.346	0.377	0.206
54	0	0	0	0	0	0	0	1	0	0	0	0	0	0	2	−0.08	327,149	0.339	0.357	0.185
55	2	0	0	0	0	1	0	0	0	0	0	0	0	0	0	1.15	327,105	0.315	0.321	0.173
56	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1	−0.65	327,064	0.315	0.322	0.173
57	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	−4.41	327,025	0.322	0.317	0.175
58	0	0	0	0	0	0	1	2	0	0	0	0	0	0	0	−6095.97	326,986	0.317	0.31	0.172
59	1	0	0	0	0	0	1	1	0	0	0	0	0	0	0	−332.88	326,823	0.308	0.302	0.183
60	0	0	1	0	0	0	0	2	0	0	0	0	0	0	0	3624.77	326,787	0.306	0.301	0.183
61	1	0	1	0	0	0	0	1	0	0	0	0	0	0	0	191.46	326,733	0.304	0.299	0.183
62	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	−17.49	326,700	0.306	0.299	0.182
63	0	0	1	0	0	1	0	1	0	0	0	0	0	0	0	183.68	326,668	0.304	0.296	0.182
64	0	0	0	0	0	0	0	0	0	0	0	1	0	0	1	−0.2	326,638	0.304	0.298	0.181
65	0	1	0	0	0	0	0	1	0	0	0	0	0	0	1	2.55	326,610	0.301	0.296	0.183
66	1	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0.13	326,572	0.297	0.299	0.18
67	2	0	0	0	0	1	0	1	0	0	0	0	0	0	0	−29.57	326,545	0.292	0.286	0.169
68	1	1	0	0	0	0	0	1	0	0	0	0	0	0	0	95.55	326,519	0.292	0.287	0.172
69	0	1	0	0	0	0	0	2	0	0	0	0	0	0	0	922.48	326,478	0.294	0.282	0.173
70	0	0	0	0	0	0	1	1	0	0	0	0	0	0	1	−6.22	326,453	0.291	0.281	0.175
71	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	−134.95	326,428	0.289	0.281	0.176
72	1	0	0	0	0	0	0	0	0	0	1	0	0	0	0	−4.47	326,398	0.284	0.282	0.173
73	1	0	0	0	0	0	0	3	0	0	0	0	0	0	0	−26,186.72	326,374	0.276	0.264	0.162
74	1	0	0	0	0	0	1	0	0	0	0	0	0	0	1	−0.29	326,352	0.272	0.266	0.158
75	1	0	0	0	0	0	0	1	1	0	0	0	0	0	0	−58.01	326,331	0.269	0.266	0.157
76	2	0	0	0	0	0	0	0	1	0	0	0	0	0	0	−3.11	326,313	0.271	0.266	0.155
77	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	−2.1	326,295	0.264	0.27	0.151
78	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	−8.73	326,278	0.264	0.275	0.153
79	1	0	0	0	0	2	0	0	0	0	0	0	0	0	0	−1.93	326,261	0.252	0.285	0.154
80	0	0	0	0	0	0	0	0	2	0	0	0	0	0	0	−14.9	326,245	0.263	0.309	0.157
81	2	1	0	0	0	0	0	0	0	0	0	0	0	0	0	−1.22	326,229	0.267	0.306	0.155
82	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	3341.29	326,214	0.266	0.307	0.156
83	0	0	0	0	0	0	0	3	0	0	0	0	0	0	1	−43.84	326,201	0.263	0.302	0.158
84	1	0	0	0	0	0	0	0	1	0	0	0	0	0	1	−0.12	326,187	0.262	0.302	0.157
85	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	−0.18	326,174	0.263	0.305	0.156
86	0	1	0	0	0	1	0	1	0	0	0	0	0	0	0	67.19	326,161	0.265	0.303	0.157
87	0	0	0	1	0	0	0	2	0	0	0	0	0	0	0	−432,954.98	326,149	0.267	0.308	0.156
88	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	−34.58	326,137	0.267	0.308	0.156
89	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0	−5.1	326,126	0.267	0.31	0.156
90	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	−10.78	326,116	0.267	0.313	0.158
91	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	−66.99	326,106	0.265	0.317	0.158
92	0	0	0	0	0	0	0	2	0	0	0	0	0	0	2	−0.09	326,097	0.265	0.308	0.151
93	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0.35	326,089	0.265	0.308	0.151
94	1	0	0	0	0	1	1	1	0	0	0	0	0	0	0	−93.83	326,081	0.264	0.306	0.148
95	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	70.45	326,073	0.256	0.288	0.141
96	0	0	0	1	0	0	0	3	0	0	0	0	0	0	0	−1,073,454.04	326,066	0.256	0.289	0.141
97	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	−21.59	326,058	0.248	0.275	0.136
98	0	0	0	0	0	0	0	1	1	0	0	0	0	0	1	−1.1	326,051	0.248	0.276	0.136
99	0	0	1	0	0	0	1	1	0	0	0	0	0	0	0	398.94	326,045	0.249	0.275	0.136
100	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	22.03	326,038	0.25	0.276	0.136
101	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	−4.12	326,033	0.25	0.276	0.136
102	1	0	0	0	0	0	0	2	0	0	0	0	0	0	1	1.3	326,027	0.248	0.281	0.138
103	2	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0.2	326,017	0.244	0.283	0.135
104	1	0	0	0	0	0	0	3	0	0	0	0	0	0	1	351.11	326,009	0.245	0.289	0.138
105	0	0	1	0	0	0	0	1	0	0	0	0	0	0	1	1.09	326,003	0.244	0.288	0.139
106	0	0	0	0	0	0	0	0	0	0	0	0	0	0	3	−0.0	325,997	0.242	0.274	0.136
107	2	0	0	0	0	0	1	0	0	0	0	0	0	0	0	−7.78	325,992	0.239	0.271	0.134
108	2	0	0	0	0	0	1	1	0	0	0	0	0	0	0	−126.28	325,973	0.238	0.269	0.132
109	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	−0.1	325,968	0.238	0.269	0.131
110	1	0	0	0	0	0	0	1	0	0	1	0	0	0	0	57.61	325,963	0.239	0.269	0.132
111	0	1	0	0	0	0	0	0	0	0	0	1	0	0	0	9.91	325,959	0.237	0.269	0.132
112	1	0	0	0	0	0	1	2	0	0	0	0	0	0	0	−1698.92	325,954	0.236	0.27	0.132
113	0	0	0	0	0	0	0	0	1	0	0	0	0	0	2	−0.01	325,950	0.237	0.27	0.133
114	0	1	0	0	0	0	0	0	1	0	0	0

the final BEL proxy function derived under the more relaxed restriction setting 300–886. Table A4 and Table A5 indicate respectively for the BEL and AC proxy functions derived under 150–443 the AIC scores and all five previously defined validation figures evaluated on the Sobol, nested simulations and capital region sets after each tenth iteration. Similarly, Table A6 reports these figures for the BEL proxy function derived under 300-886. Here the last row corresponds to the final iteration.

Lastly, we manipulate the validation values on all three validation sets twice insofar as we subtract respectively add pointwise $1.96$ times the standard errors from respectively to them (inspired by $95\%$ confidence interval of gaussian distribution). We then evaluate the validation figures for the final BEL proxy functions under both restriction settings on these manipulated sets of validation value estimates and depict them in Table A7 in order to assess the impact of the Monte Carlo error associated with the validation values.

4.2.3. Improvement by Relaxation

Table A1 and Table A2 state that the adaptive algorithm terminates under 150–443 for both BEL and AC when the maximum allowed number of terms is reached. This gives reason to relax the restriction setting to, for example, 300–886 which eventually lets the algorithm terminate due to no further reduction in the AIC score without hitting restrictions 886, compare Table A3 for BEL. In fact, only restrictions 224–464 are hit. Except for the already very small figures cr.mae, ${\mathrm{cr}.\mathrm{mae}}^a$ and cr.res all validation figures are further improved by the additional basis functions, see Table A4 and Table A6. The largest improvement takes place between iterations 180 and 190. The result that at maximum degrees 464 are selected is consistent with the result of Teuguia et al. (2014) who conclude in their numerical examples of Chapter 4 that under a monomial, Legendre or Laguerre basis the optimum degree is probably 4 or 5. Furthermore, Bauer and Ha (2015) derive a similar result in their one risk factor LSMC example of Chapter 6 when using $50,000$ fitting scenarios and Legendre, Hermite, Chebychev basis functions or eigenfunctions.

According to our Monte Carlo error impact assessment in Table A7, the slight deterioration at the end of the algorithm is not sufficient to indicate a slight overfitting tendency of AIC. Under the standard choices of the five major components, compare Section 2.2, the adaptive algorithm manages thus to provide a numerically stable and parsimonious proxy function even without a restriction setting. Here, allowing a priori unlimited degrees of freedom is thus beneficial to capturing the complex interactions in the CFP model.

4.2.4. Reduction of Bias

Overall, the systematic deviations indicated by the means of residuals (2) and (4) are reduced significantly on the three validation sets by the relaxation but not completely eliminated. For the 300–886 OLS residuals on the three sets, see the diamond-shaped residuals in Figure 8, Figure 9 and Figure 10, respectively. While the reduction of the bias comes along with the general improvement stated above, the remainder of the bias indicates that sample size is not sufficiently large or that the functional form is not flexible enough to replicate the complex interactions in CFP models. Note that if the functional form is correctly specified, Proposition 3.2 of Bauer and Ha (2015) states that if sample size is not sufficiently large, the AC proxy function will on average be positively biased in the tail reflecting the high losses and the BEL proxy function will thus be negatively biased there. Since Propositions 1 and 2 of Gordy and Juneja (2010) state that this result holds for the nested simulations estimators as well, the validation values of the nested simulations and capital region sets need to be more accurate in order to serve for bias detection in this case. For an illustration of such as bias, see Figures 5 and 6 of Bauer and Ha (2015). The bias in our one sample example is in the opposite systematic direction, which is an indication of insufficiency of polynomials. This is also consistent with the observations in the industry that the polynomials seem not to able to replicate the sudden changes in steepness of AC and BEL which are a consequence of regulation and complex management actions in the CFP models.

Unlike figures (1) and (2), figures (3) and (4) do not forgive a bad fit of the base value if the validation values are well approximated by a proxy function. Contrariwise, if a proxy function shows the same systematic deviation from the validation values and the base value, (3) and (4) will be close to zero whereas (1) and (2) will be not. The comparisons $\left|\mathrm{v}.\mathrm{res}\right|<\left|{\mathrm{v}.\mathrm{res}}^0\right|$, $\left|\mathrm{cr}.\mathrm{res}\right|<\left|{\mathrm{cr}.\mathrm{res}}^0\right|$ but $\left|\mathrm{ns}.\mathrm{res}\right|>\left|{\mathrm{ns}.\mathrm{res}}^0\right|$, holding under both restrictions settings, indicate that on the Sobol and capital region sets primarily the base value is not approximated well whereas on the nested simulations set not only the base value but also the validation values are missed. The MAEs capture this result, too, that is, $\mathrm{v}.\mathrm{mae},\mathrm{cr}.\mathrm{mae}<\mathrm{ns}.\mathrm{mae}$ but ${\mathrm{ns}.\mathrm{mae}}^0<{\mathrm{v}.\mathrm{mae}}^0,{\mathrm{cr}.\mathrm{mae}}^0$.

4.2.5. Relationship between BEL and AC

The MAEs with respect to the relative metric for BEL are much smaller than for AC since the two economic variables are subject to similar absolute fluctuations with, for example, in the base case BEL being approximately 20 times the size of AC. The similar absolute fluctuations are reflected by the iteration-wise very similar MAEs with respect to the asset metric of BEL and AC, compare ${\mathrm{v}.\mathrm{mae}}^a$, ${\mathrm{ns}.\mathrm{mae}}^a$ and ${\mathrm{cr}.\mathrm{mae}}^a$ given in % in Table A4 and Table A5. Furthermore, they manifest themselves in the iteration-wise opposing means of residuals v.res, ${\mathrm{v}.\mathrm{res}}^0$, ns.res and cr.res as well as in the similar-sized MAEs ${\mathrm{v}.\mathrm{mae}}^0$, ${\mathrm{ns}.\mathrm{mae}}^0$ and ${\mathrm{cr}.\mathrm{mae}}^0$.

4.3. Generalized Linear Models (GLMs)

4.3.1. Settings

We derive the GLMs (12) of BEL under restriction settings 150–443 and 300–886 which we also employed for the derivation of the OLS proxy functions. Thereby, we run each restriction setting with the canonical choices of random components for continuous (non-negative) response variables, that is, the gaussian, gamma and inverse gaussian distributions, compare McCullagh and Nelder (1989). In cases where the economic variable can also attain negative values (for example, AC), a suitable shift of the response values in a preceding step would be required. We combine each of the three random component choices with the commonly used identity, inverse and log link functions, that is, $g\left(\mu \right)\in \left\{\mathrm{id}\left(\mu \right),\frac{1}{\mu },log\left(\mu \right)\right\}$, compare Hastie and Pregibon (1992). In combination with the inverse gaussian random component, we consider additionally link function $\frac{1}{{\mu }^2}$. Further choices are conceivable but go beyond this first shot.

We take R function glm($·$) implemented in R package stats of R Core Team (2018).

4.3.2. Results

While Table A8, Table A9 and Table A10 display the AIC scores and five previously defined validation figures after each tenth iteration for the just mentioned combinations under 150–443, Table A11, Table A12 and Table A13 do so under 300-886 and include furthermore the final iterations. Table A14 gives an overview of the AIC scores and validation figures corresponding to all considered final GLMs and highlights in green and red respectively the best and worst values observed per figure.

4.3.3. Improvement by Relaxation

The OLS regression is the special case of a GLM with gaussian random component and identity link function which is why the first sections of Table A8 and Table A11 coincide respectively with Table A4 and Table A6. The adaptive algorithm terminates under 150–443 not only for this combination but also for all other ones when the maximum allowed number of terms is reached. Under 300–886 termination occurs due to no further reduction in the AIC score without hitting the restrictions-the different GLMs stop between 208–454 and 250–574.

For all GLMs except for the one with gamma random component and identity link, the AIC scores and eight most significant validation figures for measuring the approximation quality, namely leftmost figure v.mae to rightmost figure ns.res in the tables, are improved through the relaxation as can be seen in Table A14. For gamma random component with identity link, the deteriorations are negligible. Overall, figures ${\mathrm{ns}.\mathrm{mae}}^0$ and ${\mathrm{cr}.\mathrm{mae}}^0$ are deteriorated by at maximum $0.5\%$ points and figures ${\mathrm{ns}.\mathrm{res}}^0$ and ${\mathrm{cr}.\mathrm{res}}^0$ by at maximum 4 units. Figures cr.mae and ${\mathrm{cr}.\mathrm{mae}}^a$ are especially small under 150–443 so that slight deteriorations by at maximum $0.05\%$ points under 300-886 towards the levels of v.mae and ${\mathrm{v}.\mathrm{mae}}^a$ or ns.mae and ${\mathrm{ns}.\mathrm{mae}}^a$ are not surprising. Similar arguments apply to the acceptability of the maximum deterioration of cr.res by 13 to 17 units for inverse gaussian with $\frac{1}{{\mu }^2}$ link. We conclude that the more relaxed restriction setting 300–886 performs better than 150–443 for all GLMs in our numerical example. This result appears plausible in comparison with the OLS result from the previous section and hence also compared to the OLS results of Teuguia et al. (2014) and Bauer and Ha (2015).

AIC cannot be said to show an overfitting tendency according to Table A11, Table A12 and Table A13 and also Table A7 since the validation figures do not deteriorate in the late iterations more than they underly Monte Carlo fluctuations, compare the OLS interpretation. Using GLMs instead of OLS regression in the standard adaptive algorithm, compare Section 2.2, lets the algorithm thus maintain its property to yield numerically stable and parsimonious proxy functions even without restriction settings.

4.3.4. Reduction of Bias

According to Table A14, inverse gaussian with $\frac{1}{{\mu }^2}$ link shows the most significant decrease in v.mae by $-0.088\%$ points when moving from 150–443 to 300–886. Under 300–886 this combination even outperforms all other ones (highlighted in green) whereas under 150–443 it is vice versa (highlighted in red). Hence, the performance of a random component link combination under 150–443 does not generalize to 300–886. On the Sobol and nested simulations sets, the MAEs (1) are not only considerably lower for inverse gaussian with $\frac{1}{{\mu }^2}$ link than for all others but also the closest together even when the capital region set is included. This speaks for a great deal of consistency.

In fact, the systematic overestimation of $81\%$ of the points on the nested simulations set by inverse gaussian with $\frac{1}{{\mu }^2}$ link is certainly smaller than, for example, that of $89\%$ by gaussian with identity link but still very pronounced. On the capital region set, the overestimation rates for these two combinations are $41\%$ and $56\%$, respectively, meaning that here the bias is negligibe. Surprisingly, for most GLMs the bias is here smaller than for inverse gaussian with $\frac{1}{{\mu }^2}$ link but since this result does not generalize to the nested simulations set, we regard it as a chance event and do not question the rather mediocre performance of inverse gaussian with $\frac{1}{{\mu }^2}$ link here further. Interpreting the mean of residuals (2) provides similar insights.

In particular, for inverse gaussian $\frac{1}{{\mu }^2}$ link GLM the reduction of the bias comes along with the general improvement by the relaxation. The small remainder of the bias indicates not only that this GLM is a promising choice here but also that identifying suitable regression methods and functional forms is crucial to further improving the accuracy of the proxy function. For the residuals on the three sets, see the triangle-shaped residuals in Figure 8, Figure 9 and Figure 10, respectively.

4.3.5. Major and Minor Role of Link Function and Random Component

Apart from the just considered case, for all three random components, the relaxation to 300–886 yields the largest out-of-sample performance gains in terms of v.mae with identity link (between $-0.047\%$ and $-0.058\%$ points), closely followed by log link (between $-0.033\%$ and $-0.047\%$ points), and the least gains with inverse link (between $-0.017\%$ and $-0.020\%$ points). While with identity link the largest improvements before finalization take place for gaussian, gamma and inverse gaussian random components between iterations 180 to 190, 170 to 180, and 150 to 160, respectively, with log link they occur much sooner between iterations 120 to 130, 110 to 120, and 110 to 120, respectively, see Table A11, Table A12 and Table A13. As a result of this behavior, under 150–443 log link performs better than identity link for gaussian and inverse gaussian whereas under 300–886 it is vice versa. Inverse link always performs worse than identity and log links, in particular under 300–886.

Applying the same link with different random components does not bring much variation under 300–886 with gamma and inverse gaussian being slightly better than gaussian for all considered links though. A possible explanation is that the distribution of BEL is slightly skewed conditional on the outer scenarios. Thereby results the skewness in the inner simulations from an asymmetric profit sharing mechanism in the CFP model. While the policyholders are entitled to participate at the profits of an insurance company, see, for example, Mourik (2003), the company has to bear its losses fully by itself. Since gaussian performs only slightly worse than the skewed distributions, it should still be considered for practical reasons because it has a closed-form solution and a great deal of statistical theory has been developed for it, compare, for example, Dobson (2002). By conclusion, the choice of the link is more important than that of the random component so that trying alternative link functions might be beneficial.

4.4. Generalized Additive Models (GAMs)

4.4.1. Settings

For the derivation of the GAMs (26) of BEL, we apply only restriction settings $K_{\max }$-443 with $K_{\max }\le 150$ in the adaptive algorithm since we use smooth functions (25) constructed out of splines that may already have exponents greater than 1 to which the monomial first-order basis functions are raised. As the model selection criterion we take GCV (32) used by our chosen implementation by default. We vary different ingredients of GAMs while holding others fixed to carve out possible effects of these ingredients on the approximation quality of GAMs in adaptive algorithms and our application.

We rely on R function gam($·$) implemented in R package mgcv of Wood (2018).

4.4.2. Results

Table A15 contains the validation figures for GAMs with varying number of spline functions per smooth function, that is, $J\in \left\{4,5,8,10\right\}$, after each tenth and the finally selected smooth function. In the case of adaptive forward stepwise selection the iteration numbers coincide with the numbers of selected smooth functions. In contrast, table sections with adaptive forward stagewise selection results do not display the iteration numbers in the smooth function column k. In Table A16, we display the effective degrees of freedom, p-values and significance codes of each smooth function of the $J=4$ and $J=10$ GAMs from the previous table at stages $k\in \left\{50,100,150\right\}$. The p-values and significance codes are based on a test statistic of Marra and Wood (2012) having its foundations in the frequentist properties of Bayesian confidence intervals analyzed in Nychka (1988). Table A17 and Table A18 report the validation figures respectively for GAMs with numbers $J=5$ and $J=10$, where the types of the spline functions are varied. Thin plate regression splines, penalized cubic regression splines, duchon splines and Eilers and Marx style P-splines are considered. Thereafter, Table A19 and Table A20 display the validation figures respectively for GAMs with numbers $J=4$ and $J=8$ and different random component link function combinations. As in GLMs, we apply the gaussian, gamma and inverse gaussian distributions with identity, log, inverse and $\frac{1}{{\mu }^2}$ (only inverse gaussian) link functions.

Table A21 compares by means of two exemplary GAMs the effects of adaptive forward stagewise selection of length $L=5$ and adaptive forward stepwise selection. Last but not least, Table A22 contains a mixture of GAMs challenging the results which we will have deduced from the other GAM tables. Table A23 gives an overview of the validation figures corresponding to all derived final GAMs and highlights in green and red respectively the best and worst values observed per figure.

4.4.3. Efficiency and Performance Gains by Tailoring the Spline Function Number

Table A15 indicates that the MAEs (1) and (3) of the exemplary GAMs built up of thin plate regression splines with gaussian random component and identity link tend to increase with the number J of spline functions per dimension until $k=100$. Running more iterations reverses this behavior until $k=150$. Hence, as long as comparably few smooth functions have been selected in the adaptive algorithm fewer spline functions tend to yield better out-of-sample performances of the GAMs whereas many smooth functions tend to perform better with more spline functions. A possible explanation of this observation is that an omitted-variable bias due to too few smooth functions is aggravated here by an overfitting due to too many spline functions. For more details on an omitted-variable bias, see, for example, Pindyck and Rubinfeld (1998), and for the needlessly large sampling variances and thus low estimation precision of overfitted models, see, for example, Burnham and Anderson (2002). Differently, the absolute values of the means of residuals (2) and (4) tend to become smaller with increasing J regardless of k.

According to Table A16, the components of the effective degrees of freedom (31) associated with each smooth function tend to decrease for $J=4$ and $J=10$ slightly in k. This is plausible as the explanatory power of each additionally selected smooth term is expected to decline by trend in the adaptive algorithm. Conditional on $\mathrm{df}>1$, that is for proportions of at least $40\%$ of all smooth terms, the averages of the effective degrees of freedom belonging to $k\in \left\{50,100,150\right\}$ amount for $J=4$ and $J=10$ to $\left\{2.494,2.399,2.254\right\}$ and $\left\{5.366,4.530,4.424\right\}$, respectively. The values are by construction smaller than $J-1$ since one degree of freedom per smooth function is lost to the identifiability constraints. Hence, for at least $40\%$ of the smooth functions, on average $J=6$ is a reasonable choice to capture the CFP model properly while maintaining computational efficiency, compare Wood (2017). The other side of the coin here is that up to $60\%$ of the smooth functions are supposed to be replacable by simple linear terms without losing accuracy so that here tremendous efficiency gains can be realized by making the GAMs more parsimonious. Furthermore, setting J individually for each smooth function can help improve computational efficiency (if J should be set below average) and out-of-sample performance (if J should be set above average). However, such a tailored approach entails the challenge that the optimal J per smooth function is not stable across all k, compare row-wise the degrees of freedom in the table for $J=4$ and $J=10$.

4.4.4. Dependence of Best Spline Function Type

According to Table A17 and Table A18, the adaptive algorithm terminates only due to no further decrease in GCV when the GAMs are composed of duchon splines discussed in Duchon (1977). Whether GCV has an overfitting tendency here cannot be deduced from this example since only restriction settings with $K_{\max }\le 150$ are tested. The thin plate regression splines of Wood (2003) and penalized cubic regression splines of Wood (2017) perform similarly and significantly better than the duchon splines for both $J=5$ and $J=10$. For $J=5$ the Eilers and Marx style P-splines proposed by Eilers and Marx (1996) perform by far best when $K_{\max }=100$ smooth functions are allowed. However, for $J=10$ they are outperformed by both the thin plate regression splines and penalized cubic regression splines when between $K_{\max }=125$ and 150 smooth functions are allowed. This result illustrates well that the best choice of the spline function type varies with J and $K_{\max }$, meaning that it should be selected together with these parameters.

4.4.5. Minor Role of Link Function and Random Component

For GLMs, we have seen that varying the random component barely alters the validation results whereas varying the link function can make a noticeable impact. While this result mostly applies to the earlier compositions of GAMs as well, it certainly does not to the later ones. See for instance early composition $k=40$ in Table A19. Here identity link GAMs with gamma and inverse gaussian random components perform more similar to each other than identity and log link GAMs with gamma random component or identity and log link GAMs with inverse gaussian random component do. Log link GAMs with gamma and inverse gaussian random components show such a behavior as well. However identity link GAM with the less flexible gaussian random component (no skewness) does not show at all a behavior similar to that of identity link GAMs with gamma or inverse gaussian random components. Now see later compositions $k\in \left\{70,80\right\}$ to verify that all available GAMs in the table produce very similar validation results.

For another example see Table A20. For early composition $k=50$, identity link GAMs with gaussian and gamma random components behave very similar to each other just like log link GAMs with gaussian and gamma random components do. For later compositions $k\in \left\{100,110\right\}$, again all available GAMs produce very similar validation results. A possible explanation of this result is that the impact of the link function and random component decreases with the number of smooth functions as the latter take the modeling over. By conclusion, the choices of the random component and link function do not play a major role when the GAM is built up of many smooth functions.

4.4.6. Consistency of Results

Table A21 shows based on two exemplary GAMs constructed out of $J=8$ thin plate regression splines per dimension varying in the random component and link function that the adaptive forward stagewise selection of length $L=5$ and adaptive forward stepwise selection lead to very similar GAMs and validation results. As a result, stagewise selection should be preferred due to its considerable run time advantage. As we will see in the following, the run time can be further reduced without any drawbacks by dynamically selecting even more than 5 smooth functions per iteration.

The purpose of Table A22 is to challenge the hypotheses deduced above. Like Table A15, this table contains the results of GAMs with varying spline function number $J\in \left\{5,8,10\right\}$ and fixed spline function type. Instead of thin plate regression splines, now Eilers and Marx style P-splines are considered. Since adaptive forward stepwise and stagewise selection do not yield significant differences in the examples of Table A21, we do not expect that permutations thereof affect the results much here as well. This allows us to randomly assign three different adaptive forward selection approaches to the three exemplary proxy function derivation procedures. As one of these approaches, we choose a dynamic stagewise selection approach in which L is determined in each iteration as the proportion $0.25$ of the size of the candidate term set. Again we see that as long as only $k\in \left\{90,100\right\}$ smooth functions have been selected, $J=5$ performs better than $J=8$ and $J=8$ better than $J=10$. However, $k=150$ smooth functions are not sufficient this time for $J=10$ to catch up with the performance of $J=5$. The observed performance order is consistent with the hypotheses of a high stability of the GAMs with respect to the adaptive selection procedure and random component link function combination.

4.4.7. Potential of Improved Interaction Modeling

Table A23 presents as the most suitable GAM the one with highest allowed maximum number of smooth functions $K_{\max }=150$ and highest number of spline functions $J=10$ per dimension. The slight deterioration after $k=130$ reported by Table A15 indicates that at least one of the parameters is already comparably high. According to Table A16, there are a few smooth terms which might benefit from being composed of more than ten spline functions and increasing $K_{\max }$ might be helpful to capturing the interactions in the CFP model more appropriately, particularly in the light of the fact that the best GLM, having 250 basis functions, outperforms the best GAM on both the Sobol and nested simulations set, compare Table A14, with the best GAM showing a comparably low bias across the three validation sets though, see the dot-shaped residuals in Figure 8, Figure 9 and Figure 10, respectively. Variations in the random component link function combination and adaptive selection procedure are not expected to change the performance much. By conclusion, we recommend the fast gaussian identity link GAMs (several expressions in the PIRLS algorithm simplify) with tailored spline function numbers per smooth function and simple linear terms under stagewise selection approaches of suitable lengths $L\ge 5$ and more relaxed restriction settings where $K_{\max }>150$.

4.5. Feasible Generalized Least-Squares (FGLS) Regression

4.5.1. Settings

Like the OLS proxy functions and GLMs, we derive the FGLS proxy functions (38) under restriction settings 150–443 and 300–886. For the performance assessment of FGLS regression, we apply type I and II algorithms with variance models of different complexity, where type I results are obtained as a by-product of type II algorithm since the latter algorithm builds upon the former one. We control the complexity through the maximum allowed numbers of variance model terms $M_{\max }\in \left\{2;6;10;14;18;22\right\}$.

We combine R functions nlminb($·$) and lm($·$) implemented in R package stats of R Core Team (2018).

4.5.2. Results

Table A24 and Table A25 display respectively the adaptively selected FGLS variance models of BEL corresponding to maximum allowed numbers of terms $M_{\max }$ based on final 150–443 and 300–886 OLS proxy functions given in Table A1 and Table A3. For reasons of numerical stability and simplicity, only basis functions with exponents summing up to at max two are considered as candidates. Additionally, the AIC scores and MAEs with respect to the relative metric are reported in the tables. By construction, these results are also the type I algorithm outcomes. Table A26 and Table A27 summarize respectively under 150–443 and 300–886 all iteration-wise out-of-sample test results. The results of type II algorithm after each tenth and the final iteration of adaptive FGLS proxy function selection are respectively displayed by Table A28 and Table A29. Table A30 gives an overview of the AIC scores and validation figures corresponding to all final FGLS proxy functions and highlights as in the previous overview tables in green and red respectively the best and worst values observed per figure.

4.5.3. Consistency Gains by Variance Modeling

By looking at Table A24 and Table A25 we see similar out-of-sample performance patterns during adaptive variance model selection based on the basis function sets of 150–443 and 300–886 OLS proxy functions. In both cases, the p-values of Breusch-Pagan test indicate that heteroscedasticity is not eliminated but reduced when the variance models are extended, that is, when $M_{\max }$ is increased. In fact, in a more good-natured LSMC example Hartmann (2015) shows that a type I alike algorithm manages to fully eliminate heteroscedasticity. While the MAEs (1) barely change on the Sobol set, they decrease significantly on the nested simulations set and increase noticeably on the capital region set. Under 300–886 the effects are considerably smaller than under 150–443 since the capital region performance of 300–886 OLS proxy function is less extraordinarily good than that of 150–443 OLS proxy function. The three MAEs approach each other under both restriction settings. Hence the reductions in heteroscedasticity lead to consistency gains across the three validation sets.

Table A26 and Table A27 complete the just discussed picture. The remaining validation figures on the Sobol set improve through type I FGLS regression slightly compared to OLS regression. Like ns.mae, figure ns.res and the base residual improve a lot with increasing $M_{\max }$ under 150–443 and a little less under 300-886 but ${\mathrm{ns}.\mathrm{mae}}^0$ and ${\mathrm{ns}.\mathrm{res}}^0$ do not alter much as the aforementioned two figures cancel each other out here. On the capital region set, the figures deteriorate or remain comparably high in absolute values. The type I FGLS figures converge fast so that increasing $M_{\max }$ successively from 10 to 22 barely affects the out-of-sample performance anymore. As a result of heteroscedasticity modeling, the proxy functions are shifted such that overall approximation quality increases. Unfortunately, this does not guarantee an improvement in the relevant region for SCR estimation as our example illustrates well.

4.5.4. Monotonicity in Complexity

Let us address the type II FGLS results under 150-443 in Table A28 now. For $M_{\max }=2$, figures (3) and (4) are improved on all three validation sets significantly compared to OLS regression with the type I figures lying inbetween. The other validation figures are similar for OLS, type I and II FGLS regression, which traces the performance gains in (3) and (4) back to a better fit of the base value. For $M_{\max }=6$ to 22, the type II figures show the same effects as the type I ones but more pronouncedly, see the previous two paragraphs. These effects are by trend the more distinct the more complex the variance model becomes. The type II figures stabilize less than the type I ones because of the additional variability coming along with adaptive FGLS proxy function selection. Hartmann (2015) shows in terms of Sobol figures in her LSMC example that increasing the complexity while omitting only one regressor from the simpler variance model can deteriorate the out-of-sample performance dramatically. Intuitively, it is plausible that the FGLS validation figures are the farther from the OLS figures away the more elaborately heteroscedasticity is modeled.

Now let us relate the type II FGLS results under 300-886 in Table A29 to the other FGLS results. Under 300–886 for $M_{\max }=2$, figures (3) and (4) are already at a comparably good level with both OLS and type I FGLS regression so that they do not alter much or even deteriorate with type II FGLS regression. Like under 150–443 for $M_{\max }=6$ to 22, the type II figures show the effects of the type I ones more pronouncedly. Under both restriction settings, ns.mae and ns.res decrease thereby significantly. While this barely causes ${\mathrm{ns}.\mathrm{res}}^0$ to change under 150–443, it lets ${\mathrm{ns}.\mathrm{res}}^0$ increase in absolute values under 300–886. The slight improvements on the Sobol set and the deteriorations on the capital region set carry over to 300–886. When $M_{\max }$ is increased up to 22, the type II FGLS validation figures under 300–886 do not stop fluctuating. The variability entailed by adaptive FGLS proxy function selection intensifies thus through the relaxation of the restriction setting in this numerical example. According to Breusch-Pagan test, heteroscedasticity is neither eliminated by the type II algorithm here nor by a type II alike approach of Hartmann (2015) in her more good-natured example.

4.5.5. Improvement by Relaxation

Among all FGLS proxy functions listed in Table A30, we consider type II with $M_{\max }=14$ in variance model selection under 300–886 as the best performing one. Apart from nested simulations validation under type I algorithm, 300–886 performs better than 150–443. Since on the other hand type II algorithm performs better than type I algorithm under the respective restriction settings, 300–886 and type II algorithm are the most promising choices here. Differently $M_{\max }=14$ does not constitute a stable choice due to the high variability coming along with 300–886 and type II algorithm.

While all type I FGLS proxy functions are by definition composed of the same basis functions as the OLS proxy function, the compositions of type II FGLS proxy functions vary with $M_{\max }$ because of their renewed adaptive selection. Consequently, under 300–886 all type I FGLS proxy functions hit the same restrictions 224–464 as the OLS proxy function does, whereas the restrictions hit by type II FGLS proxy functions vary between 224–454 and 258–564. This variation is consistent with the OLS and GLM results from the previous sections and hence the OLS results of Teuguia et al. (2014) and Bauer and Ha (2015).

AIC does not have an overfitting tendency according to Table A26, Table A27, Table A28 and Table A29 as the validation figures do not deteriorate in the late iterations more than they underly Monte Carlo fluctuations, compare the OLS and GLM interpretations. Using FGLS instead of OLS regression in the standard adaptive algorithm, compare Section 2.2, lets the algorithm thus yield numerically stable and parsimonious proxy functions without restriction settings as well.

4.5.6. Reduction of Bias

The type II $M_{\max }=14$ FGLS proxy function under 300-886 reaches with 258 terms the highest observed number across all numerical experiments and not only outperforms all derived GLMs and GAMs in terms of combined Sobol and nested simulations validation, it also shows by far the smallest bias on these two validation sets and approximates the base value comparably well. This observation speaks for a high interaction complexity of the CFP model. The reduction of the bias comes again along with the general improvement by the relaxation. Given the fact that the capital region set presents the most extreme and challenging validation set in our analysis, the still mediocre performance here can be regarded as acceptable for now. Nevertheless, especially the bias on this set motivates the search for even more suitable regression methods and functional forms. For the residuals of the 300–886 FGLS proxy function on the three sets, see the x-shaped residuals in Figure 8, Figure 9 and Figure 10, respectively.

4.6. Multivariate Adaptive Regression Splines (MARS)

4.6.1. Settings

We undertake a two-step approach to identify suitable generalized MARS models out of numerous possibilities. In the first step, we vary several MARS ingredients over a wide range and obtain in this way a large number of different MARS models. To be more specific, we vary the maximum allowed number of terms $K_{\max }\in \left\{50,113,175,237,300\right\}$ and the minimum threshold for the decrease in the residual sum of squares $t_{\min }\in \left\{0,1.25,2.5,3.75,5\right\}·{10}^{-5}$ in the forward pass, the order of interaction $o\in \left\{3,4,5,6\right\}$, the pruning method $p\in \left\{‘\mathrm{n}’,‘\mathrm{b}’,‘\mathrm{f}’,‘\mathrm{s}’\right\}$ with $‘\mathrm{n}’=‘\mathrm{none}’$, $‘\mathrm{b}’=‘\mathrm{backward}’$, $‘\mathrm{f}’=‘\mathrm{forward}’$ and $‘\mathrm{s}’=‘\mathrm{seqrep}’$ in the backward pass, as well as the random component link function combination of the GLM extension. In addition to the 10 random component link function combinations applied in the numerical experiments of the GLMs, compare, for example, Table A14, we use poisson random component with identity, log and squareroot link functions. We work with the default fast MARS parameter $\mathrm{fast}.\mathrm{k}=20$ of our chosen implementation.

We use R function earth($·$) implemented in R package earth of Milborrow (2018).

4.6.2. Results

In total, these settings yield $4·5·5·4·13=5200$ MARS models with a lot of duplicates in our first step. We validate the $5200$ MARS models on the Sobol, nested simulations and capital region sets through evaluation of the five validation figures. Then we collect the five best performing MARS models in terms of each validation figure per set which gives us in total $5·5=25$ best performing models per first step validation set. Since the MAEs (1) with respect to the relative and asset metric entail the same best performing models, only $5·4=20$ of the collected models per first step set are potentially different. Based on the ingredients of each of these 20 MARS models per first step set, we define $5·5=25$ new sets of ingredients varying only with respect to $K_{\max }$ and $t_{\min }$ and derive the corresponding new but similar MARS models in the second step. As a result, we obtain in total $20·25=500$ new MARS models per first step set. Again, we assess their out-of-sample performances through evaluation of the five validation figures on the three validation sets. Out of the 500 new MARS models per first step set, we collect then the best performing ones in terms of each validation figure per second step set. Now this gives us in total $5·3=15$ best MARS models per first step set, or taking into account that the MAEs (1) with respect to the relative and asset metric entail once more the same best performing models, $4·3=12$ potentially different best models per first step set. In total, this makes $12·3=4·9=36$ best MARS models, which can be found in Table A31 sorted by first and second step validation sets.

4.6.3. Poor Interaction Modeling and Extrapolation

In Table A31, the out-of-sample performances of all MARS models derived in our two-step approach are sorted using the first step validation set as the primary and the second step validation set as the secondary sort key. Let us address the first step second step validation set combinations by the headlines in Table A31. By construction, the combinations $\mathrm{Sobol}{\mathrm{set}}^2$, $\mathrm{Nested}\mathrm{simulations}{\mathrm{set}}^2$ and $\mathrm{Capital}\mathrm{region}{\mathrm{set}}^2$ yield respectively the MARS models with the best validation figures (1)–(4) on the Sobol, nested simulations and capital region sets. See that in the table all corresponding diagonal elements are highlighted in green. But the best MAEs (1) and (3) are not even close to what OLS regression, GLMs, GAMs and FGLS regression achieve. Finding small residuals (2) and (4) regardless of the other validation figures is not sufficient. The performances on the nested simulations and capital region sets, comprising several scenarios beyond the fitting space, are especially poor. All these results indicate that MARS models do not seem very suitable for our application. Despite the possibility to select up to 300 basis functions, the MARS algorithm selects only at maximum 148 basis functions, which suggests that without any alterations, the algorithm is not able to capture the behavior of the CFP model properly, in particular extrapolation behavior is comparably poor.

The MARS model with the set of ingredients $K_{\max }=50$, $t_{\min }=0$, $o=4$, $p=‘\mathrm{b}’$, inverse gaussian random component and identity link function is selected as the best one six times out of 36, or once for each Sobol and nested simulations first step validation set combination. Furthermore, this model performs best in terms of ${\mathrm{v}.\mathrm{res}}^0$, ${\mathrm{ns}.\mathrm{mae}}^0$ and ${\mathrm{ns}.\mathrm{mae}}^a$. Since there is no other MARS model with a similar high occurrence and performance, we consider it the best performing and most stable one found in our two-step approach. For illustration of a MARS model, see this one in Table A32. The fact that this best MARS model performs worse than other ones in terms of several validation figures stresses the infeasibility of MARS models in this application.

4.6.4. Limitations

Table A31 suggests that, up to a certain upper limit, the higher the maximum allowed number of terms $K_{\max }$ the higher tends the performance on the Sobol set to be. However, this result does not generalize to the nested simulations and capital region sets. Since at maximum 148 basis functions are selected here even if up to 300 basis functions are allowed, extending the range of $K_{\max }$ in the first step of this numerical experiment would not affect the output in this regard. The threshold $t_{\min }$ is an instrument controlling the number of basis functions selected in the forward pass up to $K_{\max }$ which cannot be extended below zero, meaning that its variability has already been exhausted here as well. For the interaction order o similar considerations as for $K_{\max }$ apply. The pruning method p used in the backward pass does not play a large role compared to the other ingredients as it only helps reduce the set of selected basis functions. In terms of Sobol validation, inverse gaussian random component with identity link performs best, whereas in terms of nested simulations and capital region validation, inverse gaussian random component with any link or log link with gaussian or poisson random component perform best. We conclude that if there was a suitable MARS model for our application, our two-step approach would have found it.

4.7. Kernel Regression

4.7.1. Settings

We make a series of adjustments affecting either the structure or the derivation process of the multidimensional LC and LL proxy functions (59) and (61) to get as broad a picture of the potential of kernel regression in our application as possible. Our adjustments concern the kernel function and its order, the bandwidth selection criterion, the proportion of fitting points used for bandwidth selection, and the sets of basis functions of which the local proxy functions are composed of. Thereby we combine in various ways the gaussian, Epanechnikov and uniform kernels, orders $o\in \left\{2,4,6,8\right\}$, bandwidth selection criteria LOO-CV and AIC, and between 2500 (proportion $\mathrm{bw}=0.1$) and 25,000 (proportion $\mathrm{bw}=1$) fitting points for bandwidth selection.

We work with R functions npregbw($·$) and npreg($·$) implemented in R package np of Racine and Hayfield (2018).

4.7.2. Results

Furthermore, we alternate the four basis function sets contained in Table A33 and Table A34. The first two basis function sets with $K_{\max }\in \left\{16,27\right\}$ are derived by adaptive forward stepwise selection based on OLS regression, the third one with $K_{\max }=15$ by risk factor wise linear selection and the last one with $K_{\max }=22$ by a combination thereof. All combinations including their out-of-sample performances can be found in Table A35. Again, the best and worst values observed per validation figure are highlighted in green and red, respectively.

4.7.3. Poor Interaction Modeling and Extrapolation

We draw the following conclusions based on the validation results in Table A35. The comparisons of LC and LL regression applied with gaussian kernel and 16 basis functions or Epanechnikov kernel and 15 basis functions suggest that LL regression performs better than LC regression. However, even the best Sobol, nested simulations and capital region results of LL regression are still outperformed by OLS regression, GLMs, GAMs and FGLS regression. Possible explanations for this observation are that kernel regression is not able to model the interactions of the risk factors equally well with its few basis functions and that local regression approaches perform rather poorly close to and especially beyond the boundary of the fitting space because of the thinned out to missing data basis in this region. While the first explanation applies to all three validation sets, the latter one applies only to the nested simulations and capital region sets on which the validation figures are indeed worse than on the Sobol set. While LC regression produces interpretable results with the sets of 22 and 27 basis functions, the more complex LL regression does not in most cases.

4.7.4. Limitations

On the Sobol and capital region sets, both LC and LL regression show similar behaviors when relying on gaussian kernel and 16 basis functions compared to Epanechnikov kernel and 15 basis functions. But on the nested simulations set, gaussian kernel and 16 basis functions are the superior choices. Using a uniform kernel with LC regression deteriorates the out-of-sample performance. The results of LC regression indicate furthermore that an extension of the basis function sets from 15 to 27 only slightly affects the validation performance. With gaussian kernel switching from 16 to 27 basis functions barely has an impact and with Epanechnikov kernel only the nested simulations and capital region validation performance improve when using 27 as opposed to 15, 16 or 22 basis functions. While increasing the order of the gaussian or Epanechnikov kernel deteriorates the validation figures dramatically, for the uniform kernel the effects can go in both directions. AIC performs worse than LOO-CV when used for bandwidth selection of the gaussian kernel in LC regression. For LC regression, increasing the proportion of fitting points entering bandwidth selection improves all validation figures until a specific threshold is reached. But thereafter the nested simulations and capital region figures are deteriorated. For LL regression no such deterioration is observed.

Overall we do not see much potential in kernel regression for our practical example compared to most of the previously analyzed regression methods. Nonetheless in order to achieve comparably good kernel regression results, we consider LL regression more promising than LC regression due to the superior but still poor modeling close to and beyond the boundary of the fitting space. We would apply it with gaussian, Epanechnikov or other similar kernel functions. A high proportion of fitting points for bandwidth selection is recommended and it might be worth trying alternative comparably small basis function sets reflecting, for example, the risk factor interactions better than in our examples.

5. Conclusions

For high-dimensional variable selection applications such as the calibration step in the LSMC framework, we have presented various machine learning regression approaches ranging from ordinary and generalized least-squares regression variants over GLM and GAM approaches to multivariate adaptive regression splines and kernel regression approaches. At first we have justified the combinability of the ingredients of the regression routines such as the estimators and proposed model selection criteria in a theoretical discourse. Afterwards we have applied numerous configurations of these machine learning routines to the same slightly disguised real-world example in the LSMC framework. With the aid of different validation figures, we have analyzed the results, compared the out-of-sample performances and adviced to use certain routine designs.

In our slightly disguised real-world example and given LSMC setting, the adaptive OLS regression, GLM, GAM and FGLS regression algorithms turned out to be suitable machine learning methods for proxy modeling of life insurance companies with potential for both performance and computational efficiency gains by fine-tuning model hyperparameters and implementation designs. Differently, the MARS and kernel regression algorithms were not found to be convincing in our application. In order to study the robustness of our results, the approaches can be repeated in multiple other LSMC examples.

After all, none of our tested approaches was able to completely eliminate the bias observed in the validation figures and to yield consistent results across the three validation sets though. Investigations on whether these observations are systematic for the approaches, a result of the Monte Carlo error or a combination thereof help further narrow down the circle of recommended regression techniques. In order to assess the variance and bias of the proxy estimates conditional on an outer scenario, seed stability analyses in which the sets of fitting points are varied and convergence analyses in which sample size is increased need to be carried out. While such analyses would be computationally very costly, they would provide valuable insights into how to further improve approximation quality, that is, whether additional fitting points are necessary to reflect the underlying CFP model more accurately, whether more suitable functional forms and estimation assumptions are required for a more appropriate proxy modeling, or whether both aspects are relevant. Furthermore, one could deduce from such an analysis the sample sizes needed by the different regression algorithms to meet certain validation criteria. Since the generation of large sample sizes is currently computationally expensive for the industry, algorithms getting along with comparably few fitting points should be striven for.

Picking a suitable calibration algorithm is most important from the viewpoint of capturing the CFP model and hence the SCR appropriately. Therefore, if the bias observed in the validation figures indicates indeed issues with the functional forms of our approaches, doing further research on techniques not entailing such a bias or at least a smaller one is vital. On the one hand, one can fine-tune the approaches of this exposition and try different configurations thereof, and on the other hand, one can analyze further machine learning alternatives such as the ones mentioned in the introduction and already used in other LSMC applications. Ideally, various approaches like adaptive OLS regression, GLM, GAM and FGLS regression algorithms, artificial neural networks, tree-based methods and support vector machines would be fine-tuned and compared based on the same realistic and comprehensive data basis. Since the major challenges of machine learning calibration algorithms are hyperparameter selection and in some cases their dependence on randomness, future research should be dedicated to efficient hyperparameter search algorithms and stabilization methods such as ensemble methods.