Normal System in Laplace Expansion and Related Regression Modeling Problems

Abstract

This paper considers some innovative theoretical features and practical applications of the normal system of equations used for estimating parameters in multiple linear regression. The Laplace expansion of a determinant by cofactors and double Laplace expansion are employed for resolving the normal system. Additional features are described, including the ridge regularization applied directly to the normal system, geometric interpretation as a unique hyperplane through the points of special weighted means, Mahalanobis distances from observations to these means for the linear link functions, and multidimensional interpolation. The found properties are useful for a better understanding and interpretation of multiple regression, and the numerical examples demonstrate convenience and applicability of these tools in data modeling.

1. Introduction

Multiple regression is one of the most widely used tools of applied statistical modeling needed for solving various data fitting problems, analyzing predictors’ impact on the outcome variable, and making predictions. The construction of linear models is commonly performed by using the criterion of minimizing the total squared errors by the response variable, and such a model is known as the ordinary least squares (OLS) regression. Multiple textbooks, monographs, research papers, and internet sources are devoted to the OLS models and their applications in statistical modeling—see, for example [1,2,3,4,5,6,7,8,9,10].

The minimization of the OLS objective produces the so-called normal system of equations for the estimation of the model coefficients, and the current paper is devoted to the consideration of the normal system, as well as to its various properties and relations with several other theoretical and practical questions. It describes the classic Laplace expansion and its double form for presenting a determinant via the sum of the elements in a row, a column, or both, multiplied by their cofactors which allows the calculation of determinants via those of lower order [11,12,13]. The paper demonstrates that double Laplace expansion is a useful tool for presenting statistical modeling relations.

Another property obtained from the normal system of equations defines the geometric meaning of multiple linear regression as a hyperplane going through some special points [14,15]: a model with n parameters goes through n points of the weighted mean values. These mean values can be calculated from the data, and a normal system can be interpreted as equations of interpolation through the points of the weighted means [16,17]. Generally, the multivariate interpolation presents a difficult problem requiring grid approximations and other special techniques [18,19,20,21,22]; however, with the linear interpolation via the points of weighted means, the multivariate interpolation can be easily performed.

The normal system of equations can sometimes be based on an ill-conditioned matrix, with its determinant close to zero. An inversion of such a matrix can produce big values, inducting the inflated parameters of regression. In such cases, known as the effects of multicollinearity among the predictors, a regularization in forms of ridge regression, LASSO, elastic net, and other techniques [23,24,25,26,27,28,29,30,31] is applied. A penalizing function in these approaches is added to the OLS objective, which produces parameters of the model adjusted due to the applied restrictions. This paper considers the possibility of applying a penalizing function directly to the normal system of equations, which corresponds to the modified parameters of the model and has its interesting features.

The work also considers the estimation of the logistic regression, which is commonly built in the nonlinear modeling by the maximum likelihood criterion. As it is well-known, the generalized linear models (GLM) can be represented via the linear link functions [32,33,34]. Finding how to transform an outcome variable and obtain the linear link helps reduce a nonlinear to the linear model estimation. The paper describes the Mahalanobis distance known for its scaling multivariable observations for data clustering, segmentation, fusion, and other practical needs [35,36]. This distance is employed for measuring how far the observations are from the weighted means of the predictors used for interpolation by the normal system. A binary outcome variable can be weighted by these distances in relation to the centers of the weighted means of the predictors. The weighted means of the binary outcome can serve as the estimated proportions, which, in turn, can be used as the dependent variable values in the linear link function. Then, the parameters of the logistic regression can be found by solving the normal system of the linearized problem.

The paper is structured as follows. After Section 1 (Introduction), Section 2 describes the Laplace expansion and its double form, and Section 3 presents the relations of the multiple regression in terms of the Laplace expansion. Section 4 considers the geometrical properties of multiple regression as hyperplane interpolation via the points of weighted means. Section 5 describes ridge regularization of the normal system, and Section 6 defines Mahalanobis distances from the observations to the weighted means for building a linear link function of the nonlinear model. Section 7 illustrates the techniques by numerical examples and demonstrates that these approaches are useful in various data modeling problems. Finally, Section 8 summarizes the results.

2. Laplace Expansion and Its Double Form

To start with the main formulation for the Laplace expansion, consider a square matrix A of an n-th order, with elements a_jk, where j, k = 1, 2, …, n:

$$A=\left(\begin{array}{ccc}{ a}_{11}& \dots & { a}_{1n}\\ -& -& -\\ { a}_{1n}& \dots & { a}_{nn}\end{array}\right).$$

(1)

Due to the Laplace expansion [13], the determinant det(A) of the matrix (1) can be presented as sum of the elements of any row multiplied by cofactors of these elements, so with any jth row the determinant equals

$$\mathit{det}\left(A\right)={\sum }_{k=1}^n{\left(-1\right)}^{j+k}{a}_{jk}{m}_{jk},$$

(2)

where $m_{jk}$ are the minors or determinants of the (n − 1)th order submatrices obtained by removing the jth row and the kth columns from the original matrix (1). The signed minors ${\left(-1\right)}^{j+k}{m}_{jk}$ are the cofactors of the elements $a_{jk}$, and the expression (2) is called the cofactor expansion of determinant. Similarly, a determinant can be expanded as the sum of the elements of any column multiplied by their cofactors.

By the same pattern, a determinant can be expressed via the double sum of the product of elements in any row and column, weighted by their cofactors as well. Indeed, expanding the minors $m_{jk}$ in (2) by the elements in a jth row yields the expression for the dual expansion.

Let us consider such a formula in more detail. Suppose that matrix A (1) is extended to matrix B of the (n + 1)th order by the additional first row with elements x_k, additional first column with elements b_j, and an additional element y in the upper left corner:

$$B=\left(\begin{array}{cc}\begin{array}{ccc}y& { x}_1& { x}_2\\ { b}_1& { a}_{11}& a_{12}\end{array}& \begin{array}{cc}\dots & x_n\\ \dots & { a}_{1n}\end{array}\\ \begin{array}{ccc}{b}_2& { a}_{21}& { a}_{22}\\ -& -& -\\ b_n& { a}_{n1}& { a}_{n2}\end{array}& \begin{array}{cc}\dots & { a}_{2n}\\ -& -\\ \dots & { a}_{nn}\end{array}\end{array}\right).$$

(3)

The determinant of matrix B can be reduced to the following formula:

$$\mathit{det}\left(B\right)=det\left(A\right)y-{\sum }_{j=1}^n{\sum }_{k=1}^n{\left(-1\right)}^{j+k}{m}_{jk}{b}_j{x}_k.$$

(4)

This formula is expressed via the determinant of matrix A and the double sum of weighted products of the elements x_k in the first row and b_j in the first column. The weights in the sum (4) are defined by the minors $m_{jk}$ of matrix A, or, more exactly, by the signed minors ${\left(-1\right)}^{j+k}{m}_{jk}$, which are the cofactors of the elements a_jk of the matrix (1).

By assuming that matrix A is not singular, so its determinant differs from zero, it is possible to represent the expression (4) as

$$\mathit{det}\left(B\right)=det\left(A\right)\left(y-{\sum }_{j=1}^n{\sum }_{k=1}^n{\displaystyle \frac{{\left(-1\right)}^{j+k}{m}_{jk}}{\det \left(A\right)}}{b}_j{x}_k\right).$$

(5)

The elements of the inverted matrix A⁻¹ are defined by the cofactors of the transposed matrix divided by the determinant of the matrix, so we can rewrite the result (5) in the matrix form:

$$\mathit{det}\left(B\right)=det\left(A\right)(y-{\sum }_{j=1}^n{\sum }_{k=1}^n{\left(A^{-1}\right)}_{jk}{b}_j{x}_k)=det\left(A\right)\left(y-x^{\prime A^{-1}}b\right),$$

(6)

where x and b are vector columns with the elements x_k and b_j (3), respectively, and prime denotes transposition, so x’ is the vector row.

If the determinant (3) equals zero, det(B) = 0; then, with the non-singular matrix A, the result (6) yields the equation

$$y=x^{\prime A^{-1}}b.$$

(7)

We will return to this bilinear form in further consideration.

3. Multiple Linear Regression in Laplace Expansion

Let us briefly recall some main relations of regression modeling. The multiple linear regression for the dependent variable y and n independent variables x_k (k = 1, 2, …, n) can be written as

$$y_i=a_0+a_1{x}_{i1}+a_2{x}_{i2}+\cdots +a_n{x}_{in}+{\epsilon }_i,$$

(8)

where i denotes observations (i = 1, 2, …, N), the intercept a₀ and the coefficients of regression a_j are the unknown parameters of the model, and $\epsilon$ is the error term. The parameters are estimated by minimizing the sum of squared deviations, which is the OLS criterion:

$$S^2={\sum }_{i=1}^N{{\epsilon }_i}^2={\sum }_{i=1}^N{\left(y_i-a_0-a_1{x}_{i1}-\cdots -a_n{x}_{in}\right)}^2.$$

(9)

Equalizing the derivatives of the expression (9) by the unknown parameters to zero yields the so-called normal system:

$$\left\{\begin{array}{c}\sum _{i=1}^N{y}_i={ a}_{0}N+{ a}_1\sum _{i=1}^N{x}_{i1}+\dots +a_n\sum _{i=1}^N{x}_{in} \\ \sum _{i=1}^N{x_{i1}y}_i=a_0\sum _{i=1}^N{x}_{i1}+a_1\sum _{i=1}^N{x_{i1}}^2+\dots +a_n\sum _{i=1}^N{x}_{i1}{x}_{in}\\ -------------\\ \sum _{i=1}^N{x_{in}y}_i=a_0\sum _{i=1}^N{x}_{in}+a_1\sum _{i=1}^N{x}_{in}{x}_{i1}+\dots +a_n\sum _{i=1}^N{x_{in}}^2\end{array}\right..$$

(10)

Dividing the first equation in (10) by N defines the intercept in the regression model:

$$a_0=\overline{y}-a_1\overline{x_1}-\dots -a_n\overline{x_n}.$$

(11)

Substituting the intercept (11) into the other n Equation (10) reduces them to the following equation in matrix form:

$$C_{xx}a=c_{xy},$$

(12)

in which the matrix C_xx and vector c_xy of the nth order are defined as follows:

$$C_{xx}=X^{\prime }X, c_{xy}=X^{\prime }y.$$

(13)

In the relations (13), X denotes the matrix of N x n order of the centered observations by the predictors, y is the vector of the centered observations of the dependent variable, and X’ is the matrix transposed. The symmetric matrix C_xx and the vector c_xy in (12) and (13) are the sample covariance matrix of the predictors x among themselves and the vector of their covariances with the dependent variable y, respectively. Resolving Equation (12) for the vector a of the coefficients of regression yields the expression

$$a={C_{xx}^{-1}c}_{xy},$$

(14)

where $C_{xx}^{-1}$ is the inverted matrix. With the obtained parameters (14), the model (8) for the centered variables x and y can be presented as follows:

$$y=x^{\prime }a=x^{\prime }{C_{xx}^{-1}c}_{xy},$$

(15)

where x’ is the vector row of predictors and y is a value of the dependent variable predicted by the regression.

The obtained solution (15) coincides with Equation (7). Thus, if in the matrix (3) we use covariances with the vector b = c_xy, and the matrix A = C_xx, then the regression model corresponds to the following equation for the determinant of the matrix (3):

$$\left|\begin{array}{c} y { x}^{\prime }\\ { c}_{xy} C_{xx}\end{array}\right|=0,$$

(16)

which reduces to the solution (15), proving that the regression model can be expressed in terms of the Laplace expansion (7).

For an explicit example, let us use standardized variables centered and normalized by their standard deviations, when, in place of the covariances, we have correlations. The model with two predictors can be presented as follows:

$$y={\beta }_1{x}_1+{\beta }_2{x}_2,$$

(17)

where the parameters are so-called beta coefficients of the normalized equation. The matrix and vector of correlations (13) for the model (17) are

$$C_{xx}=\left(\begin{array}{c} 1 r_{12}\\ r_{12} 1\end{array}\right), c_{xy}=\left( \begin{array}{c}{r}_{1y}\\ r_{2y}\end{array}\right).$$

(18)

Then, relation (16) can be written explicitly as

$$\left|\begin{array}{ccc}y& x_1& { x}_2\\ r_{1y}& 1& r_{12}\\ r_{2y}& r_{12}& 1\end{array}\right|=0.$$

(19)

Applying the Laplace expansion (2) by the elements of the first row, which are the names of the variables, produces the equation

$$y\left|\begin{array}{cc}1& r_{12}\\ r_{12}& 1\end{array}\right|-x_1\left|\begin{array}{cc}{r}_{1y}& r_{12}\\ r_{2y}& 1\end{array}\right|+x_1\left|\begin{array}{cc}{r}_{1y}& 1\\ r_{2y}& r_{12}\end{array}\right|=0.$$

(20)

By finding determinants of the second order, we resolve Equation (20) for the variable y:

$$y={\displaystyle \frac{r_{1y}-r_{2y}{r}_{12}}{1-r_{12}^2}}{x}_1+{\displaystyle \frac{r_{2y}-r_{1y}{r}_{12}}{1-r_{12}^2}}{x}_2.$$

(21)

This is the simple pair regression (17) with beta coefficients. The Laplace expansion by the first column in (19) yields the equation

$$y={\displaystyle \frac{x_1-x_2{r}_{12}}{1-r_{12}^2}}{r}_{1y}+{\displaystyle \frac{x_2-x_1{r}_{12}}{1-r_{12}^2}}{r}_{y2},$$

(22)

which can also be regrouped to the solution (21). Substituting values of two predictors into the determinant (19) or into the explicit Equations (21) and (22) yields a predicted value of the dependent variable.

4. Multivariate Interpolation in Laplace Expansion

The Laplace expansion has a useful geometric interpretation. Consider a system of linear equations with the given coefficients a_jk and b_j, and variables x_k:

$$\left\{\begin{array}{c}{b}_1=a_{11}{x}_1+a_{12}{x}_2+\cdots +a_{1n}{x}_n \\ b_2=a_{21}{x}_1+a_{22}{x}_2+\cdots +a_{2n}{x}_n\\ --------\\ b_n=a_{n1}{x}_1+a_{n2}{x}_2+\cdots +a_{nn}{x}_n\end{array}\right..$$

(23)

This system geometrically corresponds to the hyperplane of the nth order going through n multidimensional points with the coordinates defined by the coefficients in each Equation (23), which are (b₁, a₁₁, a₁₂, …, a_1n), (b₂, a₂₁, a₂₂, …, a_2n), …, (b_n, a_n1, a_n2, …, a_nn). A practical way of finding the hyperplane going via these given points is known in analytic geometry [14,15]. It involves solving the following determinant equation:

$$\left|\begin{array}{cc}\begin{array}{ccc} y& x_1& x_2\\ { b}_1& a_{11}& a_{12}\end{array}& \begin{array}{cc}\dots & x_n\\ \dots & { a}_{1n}\end{array}\\ \begin{array}{ccc}{ b}_2& a_{21}& { a}_{22}\\ -& -& -\\ { b}_n& a_{n1}& { a}_{n2}\end{array}& \begin{array}{cc}\dots & { a}_{2n}\\ -& -\\ \dots & { a}_{nn}\end{array}\end{array}\right|=0.$$

(24)

The elements of this determinant match the matrix (3), and Equation (24) corresponds to expression (16). The first row of (24) contains the names of the variables, and the other rows contain parameters of the system (23). Substituting the names in the first row (24) by the values in any other row leads to the determinant with the same two rows. Such a determinant equals zero, so this hyperplane goes exactly via each of the multidimensional points.

The problem of multivariate interpolation becomes easily solvable by the linear interpolation in the approach (23) and (24): indeed, using the values of the predictors in a needed point in place of the names in (24) yields the value of the outcome. Expanding the determinant by the elements of the first row produces the equation of the hyperplane in the explicit formula for interpolation. The linear interpolation can be extended to the polynomial interpolation by many variables if, besides the linear items, we also include quadratic, mix-effects, higher powers of variables, and other kinds of polynomial or nonlinear items in the model. All such items can be denoted as new variables and used in determinant (24) for multivariable interpolation.

Let us return to the normal system (10) of the non-centered and non-standardized variables and show how it can be formulated in terms of multivariate interpolation through the set of special multidimensional points. The OLS model with 1 + n parameters of the intercept and coefficients of regression (9) can be presented as the hyperplane going through the 1 + n points of the weighted mean values of all variables. More specifically, assuming that the total of any x differs from zero, we divide each jth term in Equation (10) by the term with the intercept a₀, so this system reduces to the following one:

$$\left\{\begin{array}{c}{\displaystyle \frac{1}{N}}\sum _{i=1}^N{y}_i=a_0+{ a}_1{\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_{i1}+\dots +a_n{\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_{in} \\ \sum _{i=1}^N{\displaystyle \frac{x_{i1}}{\sum _{i=1}^N{x}_{i1}}}{y}_i=a_0{+a}_1\sum _{i=1}^N{\displaystyle \frac{x_{i1}}{\sum _{i=1}^N{x}_{i1}}}{x}_{i1}+\cdots +a_n\sum _{i=1}^N{\displaystyle \frac{x_{i1}}{\sum _{i=1}^N{x}_{i1}}}{x}_{in}\\ -------------\\ \sum _{i=1}^N{\displaystyle \frac{x_{in}}{\sum _{i=1}^N{x}_{in}}}{y}_i=a_0+a_1\sum _{i=1}^N{\displaystyle \frac{x_{in}}{\sum _{i=1}^N{x}_{in}}}{x}_{i1}+\cdots +a_n\sum _{i=1}^N{\displaystyle \frac{x_{in}}{\sum _{i=1}^N{x}_{in}}}{x}_{in}\end{array}\right..$$

(25)

The system (25) can be represented via the mean values, denoted by bars:

$$\begin{array}{c}\overline{y }=a_0+{ a}_1\overline{x_1}+\dots +a_n\overline{x_n}\\ \overline{y_{w1}}=a_0+a_1\overline{x_{1.w1}}+\cdots +a_n\overline{x_{n.w1}}\\ ---------\\ \overline{y_{wn}}=a_0+a_1\overline{x_{1.wn}}+\cdots +a_n\overline{x_{n.wn}}\end{array},$$

(26)

with the means and weighted means defined as

$$\left\{\begin{array}{c}\overline{y}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{y}_i , \overline{x_1}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_{i1} , \dots , \overline{x_n}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_{in} \\ \overline{y_{w1}}=\sum _{i=1}^N{w_{i1}y}_i, \overline{x_{1.w1}}=\sum _{i=1}^N{w_{i1}x}_{i1},\dots , \overline{x_{n.w1}}=\sum _{i=1}^N{w_{i1}x}_{in}\\ -------------\\ \overline{y_{wn}}=\sum _{i=1}^N{w_{in}y}_i, \overline{x_{1.wn}}=\sum _{i=1}^N{w_{in}x}_{i1}, \dots \overline{x_{n.wn}}=\sum _{i=1}^N{w_{in}x}_{in}\end{array}\right..$$

(27)

The weights of the ith observations in their total for each variable x_j are

$$w_{ij}={\displaystyle \frac{x_{ij}}{{\sum }_{i=1}^N{x}_{ij}}} , {\sum }_{i=1}^N{w}_{ij}=1,$$

(28)

with the totals for each jth variable equals one. Thus, $\overline{y_{wj}}$ in (26) and (27) are the mean values of y, and $\overline{x_{k.wj}}$ are the mean values of x_k, weighted by the jth set of weights built by x_j (j = 1, 2, …, n).

Consequently, the normal system (10) can be reduced to system (26), where each equation describes the hyperplane going through the corresponding (1 + n)-dimensional points defined in the rows (27). Therefore, linear regression (8) can be seen geometrically as a hyperplane going through the 1 + n points of the mean values (27) of variables weighted by each other variable. Such a hyperplane can be built similarly to the procedure described in (23) and (24), So, with the normal system (26), there is the determinant equation:

$$\left|\begin{array}{cc}\begin{array}{ccc}y& { x}_0& { x}_1\\ \overline{y }& 1& \overline{x_1}\end{array}& \begin{array}{cc}\dots & x_n\\ \dots & \overline{x_n}\end{array}\\ \begin{array}{ccc}\overline{y_{w1}}& 1& \overline{x_{1.w1}}\\ -& -& -\\ \overline{y_{wn}}& 1& \overline{x_{1.wn}}\end{array}& \begin{array}{cc}\dots & \overline{x_{n.w1}}\\ -& -\\ \dots & \overline{x_{n.wn}}\end{array}\end{array}\right|=0.$$

(29)

Unlike in (24), there is also a column with x₀ for the intercept, which corresponds to the variable identically equals one.

Expanding determinant (29) by the elements of the first row produces explicit formula for the hyperplane, which is also the regression model. For an explicit example of the model with one predictor, Equation (29) becomes

$$\left|\begin{array}{ccc}y& x_0& { x}_1\\ \overline{y }& 1& \overline{x_1}\\ \overline{y_{w1}}& 1& \overline{x_{1.w1}}\end{array}\right|=y\left|\begin{array}{cc}1& \overline{x_1}\\ 1& \overline{x_{1.w1}}\end{array}\right|-x_0\left|\begin{array}{cc}\overline{y }& \overline{x_1}\\ \overline{y_{w1}}& \overline{x_{1.w1}}\end{array}\right|+x_1\left|\begin{array}{cc}\overline{y }& 1\\ \overline{y_{w1}}& 1\end{array}\right|=0.$$

(30)

With identity x₀ = 1, finding the determinants of the second order and resolving Equation (30) for the dependent variable y, produces the following expression:

$$y={\displaystyle \frac{\overline{y} \overline{x_{1.w1}}-\overline{y_{w1}} \overline{x_1}}{\overline{x_{1.w1}}-\overline{x_1}}}+{\displaystyle \frac{\overline{y_{w1}}-\overline{y }}{\overline{x_{1.w1}}-\overline{x_1}}}{x}_1.$$

(31)

For only one predictor, x₁ can be simplified as x and w₁ as w, so the pair regression (31) can be reduced to

$$y={\displaystyle \frac{\overline{y} \overline{x_w}-\overline{y_w} \overline{x}}{\overline{x_w}-\overline{x}}}+{\displaystyle \frac{\overline{y_w}-\overline{y}}{\overline{x_w}-\overline{x}}}x=\left(\overline{y}-{\displaystyle \frac{\overline{y_w}-\overline{y}}{\overline{x_w}-\overline{x}}}\overline{x}\right)+{\displaystyle \frac{\overline{y_w}-\overline{y}}{\overline{x_w}-\overline{x}}}x=a_0+a_{1}x.$$

(32)

The regression line (32) goes via the two points of means and weighted means of the variables because the equality $x=\overline{x}$ yields $y=\overline{y}$, and equality $x=\overline{x_w}$ yields $y=\overline{y_w}$. The slope in (32) can be transformed to the expression

$$\begin{array}{ll}{a}_1={\displaystyle \frac{\overline{y_w}-\overline{y}}{\overline{x_w}-\overline{x}}}=& {\displaystyle \frac{\frac{\sum _{i=1}^N{y}_i{x}_i}{\sum _{i=1}^N{x}_i}-\overline{y}}{\frac{\sum _{i=1}^N{x_i}^2}{\sum _{i=1}^N{x}_i}-\overline{x}}}={\displaystyle \frac{\sum _{i=1}^N{y}_i{x}_i-N\overline{y} \overline{x}}{\sum _{i=1}^N{x_i}^2-N{\overline{x}}^2}}={\displaystyle \frac{\sum _{i=1}^N\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{\sum _{i=1}^N{\left(x_i-\overline{x}\right)}^2}}\\ & ={\displaystyle \frac{cov\left(x, y\right)}{var\left(x\right)}}\end{array}$$

(33)

which is the quotient of the sample covariance of x with y and the variance of the independent variable x, so it coincides with the regular definition of the slope for the pair regression. The intercept in (32) is

$$a_0=\overline{y}-{\displaystyle \frac{\overline{y_w}-\overline{y}}{\overline{x_w}-\overline{x}}}\overline{x}=\overline{y}-a_1\overline{x},$$

(34)

which corresponds to the relation (11) for the case of one predictor.

The normal system of equations considered above in (10), (25) and (26), or in the matrix form (12) can contain an ill-conditioned matrix with the determinant close to zero, so an inversion of such a matrix produces big values inducting the inflated parameters of regression. This situation requires imposing regularization with a penalizing function, which is commonly added to the OLS objective. However, as shown below, it is possible to apply a penalizing function directly to the normal system.

5. Regularization of the Normal System

For a clear exposition, let us start with the regular ridge regression (RR). The OLS criterion (9) in the matrix form for the standardized variables and with the added penalized function of the quadratic norm of standardized beta-coefficients is defined as

$$S^2=||y-X{\beta }_{RR}|{|}^2+k||{\beta }_{RR}|{|}^2=1-2{\beta }_{RR}^{\prime }{c}_{xy}+{\beta }_{RR}^{\prime }{C_{xx}\beta }_{RR}+k{\beta }_{RR}^{\prime }{\beta }_{RR},$$

(35)

where ${\beta }_{RR}$ denotes the vector of ridge regression parameters and k is a small positive ridge parameter. The notations (13) are used for the assumed normalized variables. Minimizing objective (35) by this vector yields the system of equations

$$(C_{xx}+kI){\beta }_{RR}=c_{xy},$$

(36)

where I is the identity matrix of the nth order. Equation (36) presents the normal system (12) with the added scalar matrix of the constant k. Resolving (36) yields

$${\beta }_{RR}=(C_{xx}+kI{)}^{-1}{c}_{xy},$$

(37)

which is the vector of the beta-coefficients for the RR model. The ridge regression solution (37) exists even for a singular correlation matrix C_xx, and it reduces to the OLS solution (14) when k reaches zero.

Instead of penalizing the OLS objective (35), it is possible to apply the ridge regularization item to the normal system (12) for new parameters:

$$\begin{array}{c}{NS}^2=||c_{xy}-C_{xx}{\beta }_{NS}|{|}^2+k||{\beta }_{NS}|{|}^2=c_{xy}^{\prime }{c}_{xy}-2{\beta }_{NS}^{\prime }{C_{xx}c}_{xy}+{\beta }_{NS}^{\prime }{C_{xx}^2\beta }_{NS}+\\ k{\beta }_{NS}^{\prime }{\beta }_{NS},\end{array}$$

(38)

where NS denotes the normal system. The derivative of objective (38) by the vector ${\beta }_{NS}$ produces the equation

$$\left(C_{xx}^2+kI\right){\beta }_{NS}={C_{xx}c}_{xy},$$

(39)

then, the ridge solution by the regularized normal system becomes as follows:

$${\beta }_{NS}=(C_{xx}^2+kI{)}^{-1}{C_{xx}c}_{xy}.$$

(40)

A similar solution corresponding to the parameter k = 1 was obtained for NS regularization by another criterion in [30] (note that there is a typo with the missed sign of inversion for the matrix in the formula (25) in that work).

Another useful step in building a regularized solution $\beta$ is the additional adjustment to the vector ${\beta }_{adj}$ by the constant term q, which improves the quality of the data fit:

$${\beta }_{adj}=q\beta .$$

(41)

For any obtained regularized solution of beta-coefficients $\beta$, the term q is defined as

$$q={\displaystyle \frac{{\beta }^{\prime }{c}_{xy}}{{\beta }^{\prime C_{xx}}\beta }}.$$

(42)

The maximum value of the coefficient of multiple determination, often used for the estimation of the regression quality of fit, with the adjustment (40) and (41), equals

$$R_{adj}^2={\displaystyle \frac{\left({\beta }^{\prime c_{xy}}\right)2}{{\beta }^{\prime C_{xx}}\beta }}.$$

(43)

More detail on this adjustment parameter is given in [30].

Substituting the ridge solutions (37) into (42) and using the obtained constant in (41) leads to the following adjusted ridge solution:

$${\beta }_{RR.adj}={\displaystyle \frac{c_{xy}^{\prime }(C_{xx}+kI{)}^{-1}{c}_{xy}}{c_{xy}^{\prime }(C_{xx}+kI{)}^{-2}{C_{xx}c}_{xy}}}(C_{xx}+kI{)}^{-1}{c}_{xy}.$$

(44)

Similarly, by using the normal system solution (40), we obtain the term (42) and the adjusted solution:

$${\beta }_{NS.adj}={\displaystyle \frac{c_{xy}^{\prime }(C_{xx}^2+kI{)}^{-1}{C_{xx}c}_{xy}}{c_{xy}^{\prime }(C_{xx}^2+kI{)}^{-2}{C_{xx}^{3}c}_{xy}}}(C_{xx}^2+kI{)}^{-1}{C_{xx}c}_{xy}.$$

(45)

Note that symmetrical matrices in the quotients (44) and (45) are commutative. The adjusted solutions (44) and (45) produce the maximum possible data fit by the vectors with structures (37) or (40), respectively.

6. Mahalanobis Distance to the Points of Means

The weighted mean values (27) can also be applied for facilitating estimations in some nonlinear models by their transformation to the corresponding linear link functions. Let us consider grouping the observations by the variables x_ij around the points (27) of the weighted means $\overline{x_{j.wk}}$ of the predictors. As a measure of distance from a multivariate observation to the points of mean values for the variables measured in different units, the Mahalanobis distance can be tried out. The n + 1 points of means correspond to the n + 1 centers with the coordinates given in all kth rows (27), without the mean values by the y variable. A distance from an ith point to the kth center can be defined by Mahalanobis distance as follows:

$$d_{ik}=\sqrt{(x_i-m_k{)}^{\prime }{C}_{xx}^{-1}\left(x_i-m_k\right)},$$

(46)

where $C_{xx}^{-1}$ is the inverted covariance matrix, x_i is the vector-column of the ith values by all x variables, and m_k is the vector column of the means by x-variables in each kth row in (27):

$$x_i={\left(x_{i1} , x_{i2 }, \dots ,x_{in}\right)}^{\prime }, m_k={\left( \overline{x_{1.wk}} , \overline{x_{2.wk}} , \dots , \overline{x_{n.wk}}\right)}^{\prime },$$

(47)

including the first row in (27) with the non-weighted mean values. A smaller d_ik (46) corresponds to a higher level of belonging of the ith observation to the kth cluster of mean values, so the weights of such a belonging can be defined by the reciprocal distances, 1/d_ik. However, such metrics cannot be used for the case d_ik = 0, which can happen in real data.

The weights for the correspondence of an ith observation x_i to a kth center m_k (47) can be better defined via the probability density function (pdf) of the multinormal distribution based on the Mahalanobis distance (46):

$$p\left(d_{ik}\right)={\displaystyle \frac{1}{\sqrt{\det \left({2\pi C}_{xx}\right)}}}\exp \left(-{\displaystyle \frac{d_{ik}^2}{2}}\right).$$

(48)

A smaller d_ik corresponds to a bigger pdf (48), and for d_ik = 0, the pdf reaches its maximum, so there is no singularity. It is convenient to use the weights v_ik of belonging of an ith observation to a kth group defined by the pdf values (48) normalized within each ith observation:

$$v_{ik}={\displaystyle \frac{p\left(d_{ik}\right)}{{\sum }_{k=0}^{n}p\left(d_{ik}\right)}}.$$

(49)

So, for each ith observation, the total of weights across the groups of means equals one, ${\sum }_{k=0}^n{v}_{ik}=1$. With the weights (49), the weighted means of the dependent variable can be found in relation to each of kth groups. Note that the group of k = 0 corresponds to the predictor x₀ with intercept in the first row in the Formulae (25)–(27).

Suppose the dependent variable y is presented by a binary variable with 0 and 1 values. With the weights (49), the weighted means for y are

$$\overline{y_k}={\sum }_{i=1}^N{v_{ik}y}_i.$$

(50)

The binary outcome model is commonly built in the form of the logistic regression by the maximum likelihood criterion. With this model, the probability in each ith point of observations can be found by the expression

$$p_i={\displaystyle \frac{\exp \left(b_0+b_1{x}_{i1}+\cdots +b_n{x}_{in}\right)}{1+\exp \left(b_0+b_1{x}_{i1}+\cdots +b_n{x}_{in}\right)}},$$

(51)

where b_j are the estimated logit regression parameters. The model (51) can be transformed into the linear link function:

$$ln{\displaystyle \frac{p_i}{1-p_i}}=b_0+b_1{x}_{i1}+\cdots +b_n{x}_{in}.$$

(52)

It is clear that the binary 0-or-1 values of the original outcome variable y_i cannot be used under the logarithm in (52). However, we can define the new dependent variable z at the left-hand side (52) via the proportions (50) as follows:

$$z_k=ln{\displaystyle \frac{\overline{y_k}}{1-\overline{y_k}}}.$$

(53)

Then, using the values (53) as the dependent variable in Equation (26), we solve this normal system of equations with respect to the parameters of logistic regression in the linearized form (52). The obtained parameters can serve in the logit Equation (51) for estimating probability at each observation point.

7. Numerical Examples

From the MASS package of the R software, the dataset “Cars 93” was taken for numerical examples. There are 93 observations on different car brands and models, measured by 27 numerical, ordinal, and nominal variables. The following eight numerical variables were used as predictors: x₁—Engine size (liters), x₂—Horsepower, x₃—Rev. per mile (engine revolutions per mile in highest gear), x₄—Fuel tank capacity (US gallons), x₅—Length (inches), x₆—Wheelbase (inches), x₇—Width (inches), x₈—Weight (pounds). As the dependent variable y, the Price (in USD 1000) is taken for the linear models with numerical outcome, and the binary variable Origin (USA and non-USA companies as 1 and 0 values, respectively) is taken for the logistic regression.

Table 1. Price modeling: correlations, OLS, and Ridge regression models.

Predictors		r	OLS	k = 0.2	k = 0.4	k = 0.6	k = 0.8	k = 1.0
x1	Engine size	0.597	0.229	0.138	0.125	0.120	0.117	0.115
x2	Horsepower	0.788	0.743	0.595	0.497	0.432	0.386	0.352
x3	Rev. per mile	−0.426	0.127	0.088	0.052	0.028	0.011	−0.001
x4	Fuel tank	0.619	0.083	0.130	0.143	0.144	0.143	0.140
x5	Length	0.504	0.138	0.089	0.075	0.071	0.071	0.071
x6	Wheelbase	0.501	0.282	0.123	0.091	0.081	0.077	0.075
x7	Width	0.456	−0.642	−0.276	−0.159	−0.100	−0.064	−0.040
x8	Weight	0.647	0.128	0.168	0.161	0.153	0.148	0.143
	R2		0.721	0.671	0.627	0.593	0.568	0.547
	q		1.000	1.154	1.217	1.259	1.294	1.326
R2adj			0.721	0.683	0.647	0.620	0.598	0.582

, in its first two numerical columns, presents the pair correlations r of the dependent variable of Price with the predictors and the beta-coefficients of the OLS linear regression (14) by normalized predictors.

The next five columns display the ridge regressions (37) adjusted by the relations (41) and (42) or (44) with different values of the ridge parameter k. The OLS model corresponds to the case k = 0, and all regressions show how their coefficients behave in profiling by the k parameter. The last three rows in Table 1 present the coefficient of multiple determination R², the parameter of adjustment q (42), and the coefficient R²_adj adjusted by this parameter. With bigger k, the quality of fit R² of the ridge regressions diminishes, although the adjustment parameter q improves it to R²_adj values.

Coefficients of the pairwise regressions of the price by each one predictor coincide with the pair correlations for the standardized data. Assuming that all the pairwise models have meaningful relation of the outcome with the predictors, we can notice that, in the OLS multiple regression, the parameter with x₃ becomes positive, and with x₇, it becomes negative, so of the opposite directions than observed in the pairwise relations. Such a change in sign in the coefficients corresponds to the effects of multicollinearity among the predictors in the multiple regression. Ridge regression makes the model less prone to collinearity impact, and by increasing the parameter k, we can reach the multiple ridge regression with the coefficients of same signs as in the pairwise models or in the correlations. However, as we see in Table 1, only with the ridge parameter growing to the rather big value of k = 1 can the coefficient with x₃ become negative, as in the correlation, but the coefficient with x₇ still cannot reach a positive value like in the pair relation.

A possibility to achieve the multiple regression coefficients of the signs coinciding with the pairwise models can be found using the ridge penalizing in the normal system itself, as described in the derivation (38)–(40).

Table 2. Price modeling: correlations, OLS, and Ridge Normal System models.

Predictors		r	OLS	k = 0.2	k = 0.4	k = 0.6	k = 0.8	k = 1.0
x1	Engine size	0.597	0.229	0.099	0.104	0.105	0.105	0.104
x2	Horsepower	0.788	0.743	0.437	0.336	0.279	0.243	0.218
x3	Rev. per mile	−0.426	0.127	0.039	−0.006	−0.028	−0.041	−0.050
x4	Fuel tank	0.619	0.083	0.184	0.168	0.154	0.144	0.137
x5	Length	0.504	0.138	0.025	0.033	0.043	0.051	0.056
x6	Wheelbase	0.501	0.282	0.013	0.025	0.037	0.045	0.052
x7	Width	0.456	−0.642	−0.023	0.027	0.047	0.059	0.066
x8	Weight	0.647	0.128	0.153	0.139	0.131	0.126	0.122
	R2		0.721	0.601	0.557	0.530	0.511	0.497
	q		1.000	1.128	1.135	1.131	1.126	1.122
R2adj			0.721	0.683	0.609	0.565	0.537	0.517

is built similarly to the previous table, starting with the correlations and the OLS model, but the next columns present the results of the ridge regularization for the normal system with the adjustment of (40)–(42) or (45).

The results of Table 2 demonstrate that, already with the ridge parameter, k = 0.4, the coefficients in multiple regressions all become of the same signs as the coefficients in the pairwise models or the correlations. It means that the ridge penalizing—not in the original OLS objective but in the normal system—presents a better way to obtain the interpretable solutions for multiple linear models.

Let us consider the Origin binary variable in the OLS modeling by the same eight predictors. Its solution can be reduced to the normal system of equations presented via the weighted mean values as it is described in relations (25)–(27).

Table 3. Origin modeling: Normal system for OLS regression via the weighted means.

	y	x0	x1	x2	x3	x4	x5	x6	x7	x8
x0	0.51613	1	2.6677	143.828	2332.2	16.665	183.20	103.95	69.376	3072.9
x1	0.59331	1	3.0668	158.578	2174.8	17.622	187.59	105.87	70.637	3264.7
x2	0.52938	1	2.9413	162.695	2224.8	17.505	186.10	105.14	70.254	3229.9
x3	0.46889	1	2.4877	137.206	2436.8	16.243	181.08	103.03	68.755	2981.6
x4	0.52813	1	2.8211	151.085	2273.3	17.303	185.17	104.95	69.964	3175.6
x5	0.53052	1	2.7316	146.103	2305.2	16.843	184.36	104.39	69.621	3110.4
x6	0.52498	1	2.7171	145.483	2311.7	16.826	183.98	104.39	69.574	3106.3
x7	0.52790	1	2.7162	145.647	2311.3	16.806	183.85	104.24	69.580	3100.7
x8	0.53669	1	2.8342	151.176	2262.9	17.221	185.44	105.08	70.004	3184.9

shows coefficients of the normal system given in the weighted mean values (26). The first column in Table 3 presents the weighted means of the dependent variable. The next column of the variable x₀ corresponds to the intercept. After that, the weighted means of the predictors are on the right-hand side. The values in each row define the nine multidimensional points through which the hyperplane of the linear regression (8) goes. Solving this normal system produces coefficients of the OLS regression model.

For the Origin binary outcome,

Table 4. Origin modeling: OLS, Logit, and Linear link by Mahalanobis distances.

	Predictors	OLS	Logit	Linear Link by Mahalanobis
x0	Intercept	−3.8091	−38.2030	−15.9008
x1	EngineSize	0.1663	3.7593	1.3595
x2	Horsepower	−0.0025	−0.0357	−0.0172
x3	Rev. per mile	−0.0003	−0.0015	0.0008
x4	Fuel tank capacity	−0.0414	−0.3962	−0.0564
x5	Length	0.0031	0.0288	0.0294
x6	Wheelbase	−0.0008	−0.0792	0.0194
x7	Width	0.0914	0.8695	0.1660
x8	Weight	−0.0004	−0.0043	−0.0016
R2		0.4217	0.4599	1.0000

in the first two numeric columns presents the non-standardized coefficients of the OLS and the logit regressions. The last row shows the coefficient of multiple determination R² for the OLS model and the pseudo-R² for the logit model (defined as one minus the quotient of the residual deviance by the null deviance). These models have a similar quality of fit and prediction.

The last column in Table 4 shows the estimation via the linear link by the Mahalanobis distance and weighted proportions described in the formulae (46)–(53). The R² in this case equals one, but it only indicates that this model corresponds to the interpolation via the points of the weighted means, so its residuals equal zero. The three solutions in Table 4 look different, but their correlations are very high. Correlating without the intercepts yields 0.9496 for OLS and Logit, 0.8996 for OLS and Linear-link, and 0.9906 for Logit and Linear-link. With the intercepts, these values become 0.9986, 0.9990, and 0.9998, respectively.

With these three models, we can estimate the fitted values for the outcome variable y and find correlations between them, which are 0.9119 between OLS and Logit, 0.9875 between OLS and Linear-link, and 0.9527 between the Logit and Linear-link models. Thus, these vectors of fitted values are highly correlated; however, some OLS predictions could occur to be negative or above one, while both Logit and Linear-link always produce meaningful probability values in the [0, 1] range. The predicted variables can also be dichotomized to 0 or 1 for the values below 0.5 and equal/higher than 0.5; then, it is possible to build the cross-tables of counts for the original versus the predicted values. The quality of prediction can be evaluated by the hit rate, which is the total of the correct 0 and 1 predictions of the dependent binary variable in the total number of observations. The hit rate reached by each of the three predictions is about 83%, so, in general, all these models are of a good quality of fit and prediction.

8. Summary

The work described several theoretical relations and practical applications based on the normal system of equations used for the estimation of parameters in multiple linear regression. The considered techniques further develop the approaches started in the works of [16,17,30,31,36]. Resolving the normal system and finding new relations are described in terms of the Laplace expansion of the determinant by its cofactors and double Laplace expansion. The new approaches include the ridge regularization of the normal system itself and the geometric interpretation of the normal system as a unique hyperplane going via the multivariate points of weighted means. The hyperplane definition in the determinant equation can be used for the multivariable interpolation of the linear or nonlinear functions. Mahalanobis distances from the observations to the weighted means are built to find the weighted proportions of the outcome binary variable and reduce the logistic regression to the linear link function and multidimensional interpolation by it. The numerical examples demonstrate the usefulness of the described tools in data modeling. Particularly, they show that the ridge penalizing not in the original OLS criterion but in the normal system occurs to be a better way of obtaining an interpretable solution for multiple linear models and that the logistic model can be substituted by the estimation via its linear link with the proportions for the binary outcome evaluated in the Mahalanobis distance approach. Future research can extend the Mahalanobis metrics to the generalizations like Bregman, Jensen, and other divergences [37,38]. The found properties are convenient for a better understanding and interpretation of multiple regression modeling.