Standardized Coefficients of Multiple Regression Beyond One and Multicollinearity Diagnostics

Abstract

Standardized coefficients of multiple regression, also known as beta coefficients, by absolute value are usually smaller than one, but sometimes they can exceed one. This effect had been studied mostly for models with two predictors, where it was explained by the high collinearity between them. The current paper considers multiple linear regression and defines the necessary and sufficient conditions for beta coefficients to exceed one. These conditions determine a measure of each predictor’s connection to the dependent variable in the model relative to the connection with other predictors. This criterion presents a new measure for diagnostics of the multicollinearity, which can be employed additionally to the commonly used variance inflation factor. A new interpretation is given to the meaning of the squared beta coefficients themselves. Numerical examples demonstrate these novel features. The obtained results are useful in applied regression analysis, helping practitioners and educators to understand and to explain the outcomes of regression modeling.

1. Introduction

Multiple linear regression is one of the most widely used tools for applied modeling, analysis, and prediction in various fields of statistical estimations. Classical works on regression analysis [1,2,3] have been developed in contemporary studies [4,5,6,7]. Various special techniques of statistical modeling and machine learning with their implementation in statistical software can be found in multiple sources [8,9,10,11,12,13,14]. Still, statistical models continue to reveal previously unnoticed or insufficiently studied intrinsic features and rarely observed but theoretically important properties.

One such phenomenon observed in multiple regression concerns the model coefficients. These coefficients estimate a change in the dependent variable due to the unit change in one of the independent variables subject to the other fixed predictors. If the response variable and predictors are standardized by centering with the mean values and normalizing by the standard deviations, the model’s parameters are called beta coefficients. Beta coefficients are dimensionless, so they can be compared between themselves to identify which of the predictors have a higher impact, positive or negative, on the dependent variable. In most practical cases, beta coefficients have values comparable with the coefficients of pair correlations of the variables. In the pairwise regression $y=\beta x$ of y by one predictor x, the coefficient of regression equals the correlation of the variables, $\beta =r_{yx}$. A correlation by absolute value cannot exceed one. It gives an impression, and practitioners routinely assume it, that beta coefficients in a model with many predictors belong to the same range from −1 to +1 as well, so the betas are less than one in absolute value, similarly to coefficients of correlation.

Nevertheless, sometimes a rare event occurs when a beta coefficient happens to be beyond the [−1, 1] interval. This effect was first described by American statistician and political scientist John Deegan, Jr. [15], who studied it for the model with two predictors and explained it by a high collinearity between predictors. Then renowned Swedish statistician Karl Jöreskog [16] described this possibility in an example of two predictors, and in both these works it was mentioned that such an event can happen in multiple regressions with more variables as well. Although this effect has been known already for dozens of years, questions about it have been raised periodically and discussed in internet blogs. For example, it is considered in relation to multicollinearity, suppression effects, and structural equation modeling [17,18].

The current paper extends the consideration of this effect for the general case of n predictors in multiple regression and describes the necessary and sufficient conditions when some standardized coefficients in absolute value can exceed one. Such coefficients identify the key drivers with a maximum magnitude of impact of the unit change in predictors leading to the increase above one in the outcome. These conditions can be expressed as the criterion of importance of each particular predictor in the model taken relatively to its connection to all other predictors.

This criterion can serve other aims as well; for example, it presents a new measure useful for diagnostics of multicollinearity. The multicollinearity can have detrimental effects in the analysis of individual predictors’ impact on the outcome variable: parameter estimates could fluctuate wildly with a negligible change in the sample, they could have signs opposite to signs of easily understood pair correlations, and theoretically important variables could have insignificant coefficients, so it is important to have a reliable tool for the identification of multicollinearity. Starting with the seminal work [19] on ridge regression for mitigating multicollinearity effects, various techniques have been developed, including the least absolute shrinkage and selection operator, or LASSO [20,21]; elastic net regression [22]; and least angle regression, or LARS [23]. Other regularization methods are also known [24,25,26,27]; for example, those based on orthonormal matrix decompositions and the Shapley value tool from cooperative game theory [28,29]. Penalizing approaches for generalized linear models have been investigated and applied as well [30,31,32].

The considered criterion of the squared beta coefficients could be employed as an additional tool for the commonly used variance inflation factor (VIF). It can even be more adequate than the VIF because it takes into account the two-fold relations of each predictor to the outcome of the model and to the set of all independent variables. The obtained results give a novel interpretation for squared beta coefficients: they correspond to increments in the coefficient of multiple determination due to the inclusion of each particular predictor relative to the level of its connection to other predictors. Thus, the squared beta coefficients define the importance of predictors in a model. Numerical examples demonstrate these features, which are useful in practical regression analysis, helping in the interpretation of regression output for various applied problems.

The paper is organized as follows: Section 2 presents formulae and theorems on the conditions of occurrence of the studied phenomenon, Section 3 discusses some simulation examples for models with two predictors, and Section 4 presents numerical examples for multiple regressions. Section 5 summarizes, and the Appendix A proves a needed formula.

2. Derivation of the Needed Criteria

Consider a dataset with n variables x_j (j = 1, 2, …, n) and N observations x_ij by each of them (i = 1, 2, …, N), so the data can be presented in the matrix X of N-by-n order, with N > n. The variables x_j can be centered and normalized by the standard deviations, then the product X’X (the prime denotes transposition) defines the sample correlation matrix, which is denoted as C. Let us briefly recall some formulae of regression modeling. For the assumed multivariate normal distribution with variables which have zero mean values and the unit standard deviations, the mathematical expectation of x₁, conditional to the other variables fixed, can be presented by the linear dependence

$$x_1={\beta }_{12.}{x}_2+{\beta }_{13.}{x}_3+\cdots +{\beta }_{1n.}{x}_n.$$

(1)

It is the linear regression of x₁ by other variables, with beta coefficients written in Yule’s notation, where the dot means all other variables. The coefficients in (1) are defined as follows [3]:

$${\beta }_{1j.}= {\displaystyle \frac{{-C}^{1j}}{C^{11}}},$$

(2)

where $C^{jj}$ denote the elements of the inverse $C^{-1}$ correlation matrix of all variables. In the form (2), the coefficients of regression were considered in [16] where is a typo of the missed sign.

It is clear by the quotient (2) that if by absolute value the numerator is bigger than the denominator, the beta coefficient has a value beyond the interval [−1, 1]. To answer the question why it can happen, let us look at the coefficient of partial correlation between the dependent variable x₁ and the predictor x_j with the other fixed predictors, and express it via the beta coefficient (2):

$${\rho }_{1j.}= {\displaystyle \frac{{-C}^{1j}}{{(C^{11}{C}^{jj})}^{1/2}}}={\beta }_{1j.}{\left({\displaystyle \frac{C^{11}}{C^{jj}}}\right)}^{1/2}.$$

(3)

All correlations, including partial correlations, by absolute value are less than one, so

$$|{\rho }_{1j.}|={|\beta }_{1j.}|{\left({\displaystyle \frac{C^{11}}{C^{jj}}}\right)}^{1/2}\le 1.$$

(4)

From (4), the following inequality can be written:

$${|\beta }_{1j.}|\le {\left({\displaystyle \frac{C^{jj}}{C^{11}}}\right)}^{1/2}.$$

(5)

As it is known in regression analysis, the diagonal elements of the inverse correlation matrix can be expressed as follows:

$$C^{jj}= {\displaystyle \frac{1}{1-R_{j.}^2}},$$

(6)

where $R_{j.}^2$ denotes the so-called coefficient of multiple determination which is a characteristic of the quality of fit for each linear dependence of a jth variable by all other variables. For example, $R_{1.}^2$shows the quality of fit for the model (1), and a general $R_{j.}^2$corresponds to the quality of fit in the model

$$x_j={\beta }_{j1.}{x}_1+{\beta }_{j2.}{x}_2+\cdots +{\beta }_{j,j-1.}{x}_{j-1}+{\beta }_{j,j+1.}{x}_{j+1}+\cdots +{\beta }_{jn.}{x}_n.$$

(7)

Substituting the relations (6) into the Formula (5) transforms it into the expression:

$${|\beta }_{1j.}|\le {\left({\displaystyle \frac{1-R_{1.}^2}{1-R_{j.}^2}}\right)}^{1/2}.$$

(8)

If a value at the right-hand side (8) is less than one, the beta coefficient by absolute value is smaller than one. However, consider the opposite case of the right-hand side in (8) being bigger than one; that is, $1-R_{1.}^2\ge 1-R_{j.}^2$, or simply

$$R_{j.}^2\ge R_{1.}^2.$$

(9)

With the inequality (9), the right-hand side in (8) is more than one, $1+∆$ with a constant $∆>0$, so in this case the beta coefficient ${|\beta }_{1j.}|<1+∆$. Thus, beta can be less or more than one. Then the following theorem can be formulated.

Theorem 1

(on the necessary condition). In the regression model (1) of x₁ by the other variables, a coefficient ${\beta }_{1j.}$ with a jth variable can lay beyond the [−1, 1] interval,

$${|\beta }_{1j.}|\ge 1,$$

(10)

if the inequality (9) for the coefficients of multiple determination is satisfied, so by the quality of fit the model (7) of the jth variable by all the others outperforms the model (1). The condition (9) is necessary, but not sufficient for fulfillment of the inequality (10).

Proof of Theorem 1.

The proof of this theorem consists in the given derivation (1)–(10). □

For practical estimations, by the inverse correlation matrix $C^{-1}$we have all the needed values $C^{jj}$ and R_j.² to identify which specific models have a possibility to contain beta coefficients beyond the [−1, 1] interval.

Let us describe some additional properties of the regression models which permit us to find a sufficient condition for beta coefficients to exceed one. For this aim, consider the model (1) with one jth predictor excluded:

$$x_1={\beta }_{12.}{x}_2+\cdots +{\beta }_{1,j-1.}{x}_{j-1}+{\beta }_{1,j+1.}{x}_{j+1}+\cdots +{\beta }_{1n.}{x}_n.$$

(11)

The corresponding coefficient of multiple determination can be denoted as $R_{1.(-j)}^2$ which indicates that the predictors’ set misses the jth variable.

There exists an important relation connecting coefficients of multiple determination in the whole model (1) and the model (11) with one variable excluded:

$$R_{1.}^2=R_{1.(-j)}^2+{\beta }_{1j.}^2(1-R_{j.(-1)}^2).$$

(12)

This formula shows that if, to add a jth variable to the model (11), the coefficient of multiple determination $R_{1.(-j)}^2$ grows to the value $R_{1.}^2$ for the model (1) with more parameters. Due to (12), this increment is proportional to the squared beta coefficient ${\beta }_{1j.}^2$ of the model (1), multiplied by $1-R_{j.(-1)}^2$, where the value $R_{j.(-1)}^2$ is nothing else but the coefficient of multiple determination of the following model:

$$x_j={\beta }_{j2.}{x}_2+\cdots +{\beta }_{j,j-1.}{x}_{j-1}+{\beta }_{j,j+1.}{x}_{j+1}+\cdots +{\beta }_{jn.}{x}_n.$$

(13)

This is the regression model of the jth variable by other predictors, similar to (7) but without the first variable x₁. The derivation of the relation (12) is given in Appendix A, and more detail on it can be found in [33,34].

Solving the relation (12) with respect to the beta coefficient squared produces the expression:

$${\beta }_{1j.}^2={\displaystyle \frac{R_{1.}^2-R_{1.(-j)}^2}{1-R_{j.(-1)}^2}}.$$

(14)

This expression has an interesting interpretation. Its numerator shows that, for different predictors x_j (j = 2, 3, …, n), the bigger the increment $R_{1.}^2-R_{1.(-j)}^2$ due to the inclusion of this predictor in the model, the more useful this predictor is for an explanation of fit of the outcome variable in the whole model (1). Concerning the denominator in (14), it is defined by the distance of $R_{j.(-1)}^2$ to its maximum possible value one. The coefficient $R_{j.(-1)}^2$ evaluates the connection of the predictor x_j with all other predictors in the model (13), so it is a measure of multicollinearity and can also be expressed via the variance inflation factor defined as

$${VIF}_j=1/(1-R_{j.\left(-1\right)}^2).$$

(15)

Note that, unlike in (6), where the whole matrix of n variables is used, the VIF values in (15) are obtained by the inverse correlation matrix of n − 1 variables without x₁, so only for the variables serving as predictors in (1). Therefore, the numerator and denominator in (14) present the measures of connection of each predictor x_j with the dependent variable x₁, and with the other predictors, respectively. Then we can formulate the following theorem.

Theorem 2

(on the necessary and sufficient condition). In the regression model (1) of x₁ by the other variables, a coefficient ${\beta }_{1j.}$ lays beyond the [−1, 1] interval, if and only if the following inequality for the coefficients of multiple determination is satisfied:

$$R_{1.}^2-R_{1.(-j)}^2\ge 1-R_{j.(-1)}^2.$$

(16)

Proof of Theorem 2.

To prove Theorem 2, we notice that the condition (16), due to the equality (14), defines the coefficients ${\beta }_{1j.}^2\ge 1$. Therefore, the coefficient ${\beta }_{1j.}$ is laying beyond the [−1, 1] interval if the predictor x_j produces the increment in the coefficient of multiple determination for the model (1) which is bigger than the connection of x_j with the other predictors. The condition (16) is necessary and sufficient for fulfillment of the inequality (10). □

Corollary 1.

Recalling that the coefficient of multiple determination in a model is connected with the residual sum of squares S² by the Pythagorean relation $R^2=1-S^2$, we can represent the inequality (16) via the corresponding residual sums as follows:

$$S_{1.(-j)}^2-S_{1.}^2\ge S_{j.(-1)}^2.$$

(17)

Then Theorem 2 can be expressed in terms of the residual sums: when the increment in the residual sums in the reduced model (11) and full model (1), without and with the jth predictor, is bigger than the residual sum in the model (13) of this predictor by the other predictors, the coefficient ${\beta }_{1j.}$ by absolute value exceeds one.

The criterion (14) can also be taken as a new measure for diagnostics of the multicollinearity. Commonly, VIF values (15) above several units are considered as indicating the presence of multicollinearity, and above ten or so as a strong multicollinearity. Substituting expression (15) into the Formula (14) yields the relation:

$${\beta }_{1j.}^2=(R_{1.}^2-R_{1.(-j)}^2){VIF}_j.$$

(18)

Thus, the squared beta coefficients are proportional to VIF terms corrected by the increment in the coefficient of multiple determination due to the inclusion of each predictor in the model. The values (18) can be used additionally to VIF values because they take into account not only the relations of each predictor to the set of other independent variables, but also their importance in increasing the fit for the full model (1).

The result (14) produces a novel interpretation of beta coefficients themselves: each jth beta squared is defined by the quotient where the numerator equals the increment of the coefficient of multiple determination due to the inclusion of this predictor, and the denominator equals the distance from the coefficient of multiple determination of this predictor by other predictors to the maximum possible value one. The ranks of squared beta coefficients can be used to define the order of importance of predictors in the model.

Let us make several additional remarks. The obtained relations can be expressed in terms of cofactors C_jk of jkth elements in the correlation matrix which are proportional to elements $C^{jk}$ of the inverse matrix $C^{-1}$because $C^{jk}=C_{jk}/\mathrm{d}\mathrm{e}\mathrm{t}\left(C\right)$, with det(C) as determinant of matrix. The connection of a predictor to the dependent variable on one side and to predictors on the other side was described with the so-called efficiency index in [35].

It is interesting to note that the expression (12) can be seen in relation to the t-statistics for regression parameters. The squared standard error ${{\sigma }^2(\beta }_{1j.})$ of regression coefficients in the model (1) is defined via the residual variance $S_{1.}^2/df$ of the regression fit (where df = N − n − 1 denotes degrees of freedom) and the diagonal element of the inverse matrix of predictors, or the VIF (15) as their product:

$${{\sigma }^2(\beta }_{1j.})={\displaystyle \frac{S_{1.}^2}{df}}{VIF}_j={\displaystyle \frac{{1-R}_{1.}^2}{df(1-R_{j.(-1)}^2)}}.$$

(19)

Then the squared t-value for each coefficient of regression (1) can be written as:

$${t^2(\beta }_{1j.})= {\displaystyle \frac{{\beta }_{1j.}^2}{{{\sigma }^2(\beta }_{1j.})}}= {\displaystyle \frac{{\beta }_{1j.}^2(1-R_{j.\left(-1\right)}^2)}{{1-R}_{1.}^2}}df.$$

(20)

The numerator in quotient (20) can be expressed by the relation (12), then this equation reduces to the following one:

$${t^2(\beta }_{1j.})= {\displaystyle \frac{R_{1.}^2-R_{1.(-j)}^2}{{1-R}_{1.}^2}}df={\displaystyle \frac{S_{1.(-j)}^2-S_{1.}^2}{S_{1.}^2}}df.$$

(21)

Therefore, the squared t-value is proportional to the relative change in the residual sums of squares of the reduced model without the jth predictor and the full model, that is also proportional to the partial F-statistics of regressors, which is often applied in stepwise regression modeling for the elimination or inclusion of insignificant or significant regressors, respectively.

3. Examples for Models with Two Predictors

Numerical examples of beta coefficients exceeding one in models with two predictors are given in [15,16,17,18]. Standardized regression by two predictors can be written in the simple notations as

$$y={\beta }_1{x}_1+{\beta }_2{x}_2.$$

(22)

These regression coefficients can be expressed via three correlation coefficients r_y1, r_y2, and r₁₂ between the variables in explicit form:

$${\beta }_1={\displaystyle \frac{r_{y1}-r_{y2}{r}_{12}}{1-r_{12}^2}},\hspace{1em}{\beta }_2={\displaystyle \frac{r_{y2}-r_{y1}{r}_{12}}{1-r_{12}^2}}.$$

(23)

When r₁₂ reaches 1 or −1, the coefficients (23) become inflated, so by absolute value they could exceed one. To study this effect in more detail, consider the matrix of correlations of all three variables. It should be a non-negative definite matrix, so its determinant d can be presented in the following inequality:

$$d= \left|\begin{array}{c}1\hspace{1em}{r}_{12}\hspace{1em}{r}_{y1}\\ r_{12}\hspace{1em}1\hspace{1em}{r}_{y2}\\ r_{y1}\hspace{1em}{r}_{y2}\hspace{1em}1\end{array}\right|=1+2r_{y1}{r}_{y2}{r}_{12}-r_{y1}^2-r_{y2}^2-r_{12}^2\ge 0.$$

(24)

For two given correlations, for example, r₁₂ and r_y2, the inequality (24) can be written as

$$d=(1-r_{12}^2)(1-r_{y2}^2)-(r_{y1}-r_{12}{r}_{y2}{)}^2\ge 0,$$

(25)

then the third correlation r_y1 has values only within the range:

$$r_{y2}{r}_{12}-\sqrt{(1-r_{y2}^2)(1-r_{12}^2)}\hspace{0.33em}\le \hspace{0.33em}{r}_{y1}\hspace{0.33em}\le \hspace{0.33em}{r}_{y2}{r}_{12}+\sqrt{(1-r_{y2}^2)(1-r_{12}^2).}$$

(26)

Numerical runs for beta coefficients (23), within the set of r_y1 and r_y2 in the wide range of their values and for some given values of r₁₂, are presented in Table 1 and Table 2.

Table 1. Feasible solutions for ${\beta }_1$ when r₁₂ = 0.7.

	ry2
ry1	−0.99	−0.90	−0.70	−0.50	−0.30	−0.10	0.00	0.10	0.30	0.50	0.70	0.90	0.99
−0.99			−0.98
−0.90		−0.53	−0.80	−1.08
−0.70	−0.01	−0.14	−0.41	−0.69	−0.96	−1.24	−1.37
−0.50		0.25	−0.02	−0.29	−0.57	−0.84	−0.98	−1.12
−0.30			0.37	0.10	−0.18	−0.45	−0.59	−0.73	−1.00
−0.10			0.76	0.49	0.22	−0.06	−0.20	−0.33	−0.61	−0.88
0.00			0.96	0.69	0.41	0.14	0.00	−0.14	−0.41	−0.69	−0.96
0.10				0.88	0.61	0.33	0.20	0.06	−0.22	−0.49	−0.76
0.30					1.00	0.73	0.59	0.45	0.18	−0.10	−0.37
0.50						1.12	0.98	0.84	0.57	0.29	0.02	−0.25
0.70							1.37	1.24	0.96	0.69	0.41	0.14	0.01
0.90										1.08	0.80	0.53
0.99											0.98

Table 2. Feasible solutions for ${\beta }_1$ when r₁₂ = 0.9.

	ry2
ry1	−0.99	−0.90	−0.70	−0.50	−0.30	−0.10	0.00	0.10	0.30	0.50	0.70	0.90	0.99
−0.99		−0.95
−0.90	−0.05	−0.47	−1.42
−0.70		0.58	−0.37	−1.32
−0.50			0.68	−0.26	−1.21	−2.16
−0.30				0.79	−0.16	−1.11	−1.58	−2.05
−0.10				1.84	0.89	−0.05	−0.53	−1.00	−1.95
0.00					1.42	0.47	0.00	−0.47	−1.42
0.10					1.95	1.00	0.53	0.05	−0.89	−1.84
0.30						2.05	1.58	1.11	0.16	−0.79
0.50								2.16	1.21	0.26	−0.68
0.70										1.32	0.37	−0.58
0.90											1.42	0.47	0.05
0.99												0.95

The values r_y1 and r_y2 correspond to the tables’ rows and columns, respectively. Table 1 and Table 2 present the values for coefficients ${\beta }_1$, while the other coefficient ${\beta }_2$ can be found in the transposed location. Table 1 shows the result for $r_{12}=0.7$, and Table 2 for $r_{12}=0.9$. The beta coefficients beyond the [−1, 1] interval are marked in bold font, and we can see that this event is not so rare: in Table 1, such coefficients appear in about 13% cases, and in Table 2 in 40% of cases of the feasible coefficients.

For several more examples, the value $r_{12}=0.99$, with $r_{y2}=0$ and $r_{y1}=\pm 0.1$, yield coefficients ${\beta }_1=\pm 5.03$, respectively, while for $r_{y1}=0$ and $r_{y2}=\pm 0.1$ the coefficients are ${\beta }_1=\pm 4.97$, so about five times exceeding one. For the negative values r₁₂ such tables can be obtained by the symmetrical reflection of Table 1 or Table 2 around the vertical line going via a central 0 value. Similar tables, although for a different aim, were considered in [36], where more cases of the beta coefficients beyond 1 can be found.

4. Numerical Examples with Multiple Regressions

For multiple regression by real data, the dataset “Cars 93” on American and foreign cars is taken from the MASS library in the R software (MASS R package [Documentation]|R PACKAGES, version 7.3-65). There are 93 observations measured by multiple characteristics from which the following 11 numerical variables were used: x₁—Engine size (liters), x₂—Horsepower, x₃—RPM (revolutions per minute at maximum horsepower), x₄—Rev.per.mile (engine revolutions per mile in highest gear), x₅—Fuel tank capacity (US gallons), x₆—Passenger places (number of persons), x₇—Length (inches), x₈—Wheelbase (inches), x₉—Width (inches), x₁₀—Turn.circle (U-turn space, feet), x₁₁—Weight (pounds). The pair correlations matrix of these variables is shown in Table 3. The bottom line of Table 3 shows the coefficients of multiple determination R_j.² of each jth variable by all the other variables in the full model (7).

Table 3. Pair correlations matrix and R_j.².

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10	x11
x1	1.00	0.73	−0.55	−0.82	0.76	0.37	0.78	0.73	0.87	0.78	0.85
x2	0.73	1.00	0.04	−0.60	0.71	0.01	0.55	0.49	0.64	0.56	0.74
x3	−0.55	0.04	1.00	0.49	−0.33	−0.47	−0.44	−0.47	−0.54	−0.51	−0.43
x4	−0.82	−0.60	0.49	1.00	−0.61	−0.33	−0.69	−0.64	−0.78	−0.73	−0.74
x5	0.76	0.71	−0.33	−0.61	1.00	0.47	0.69	0.76	0.80	0.67	0.89
x6	0.37	0.01	−0.47	−0.33	0.47	1.00	0.49	0.69	0.49	0.45	0.55
x7	0.78	0.55	−0.44	−0.69	0.69	0.49	1.00	0.82	0.82	0.74	0.81
x8	0.73	0.49	−0.47	−0.64	0.76	0.69	0.82	1.00	0.81	0.72	0.87
x9	0.87	0.64	−0.54	−0.78	0.80	0.49	0.82	0.81	1.00	0.82	0.87
x10	0.78	0.56	−0.51	−0.73	0.67	0.45	0.74	0.72	0.82	1.00	0.78
x11	0.85	0.74	−0.43	−0.74	0.89	0.55	0.81	0.87	0.87	0.78	1.00
Rj.2	0.909	0.921	0.786	0.721	0.822	0.710	0.777	0.868	0.867	0.716	0.947

The inequality (9) can be applied to find which regressions can, in principle, have the coefficients beyond the [−1, 1] interval. For example, the model of the variable x₁ by all the others has $R_{1.}^2=0.909$, so only $R_{2.}^2$ for x₂ and $R_{11.}^2$ for x₁₁ are above it, and their coefficients could satisfy inequality (10). However, the criterion (9) is just the necessary not the sufficient condition, and all parameters of this model by absolute value occurred to be smaller than one. In general, for n ordered values of the multiple determination, there are n(n + 1)/2 pairs with one of them bigger than another one, so for n = 11 there are 55 such possibilities. Actually, relation (10) is a rare event, and among all possible regressions by this data it was observed only twice.

One such model is the regression of x₃ by the other variables, which is presented in Table 4.

Table 4. Model for x₃ (RPM), various coefficients R², and criterion for beta $\left|{\beta }_{3j.}\right|>1$.

		The Full Model (7)		Reduced Models (11)	Predictors’ Models (13)	Criterion (14)	The Stepwise Model from (7)
Predictors		beta ${\beta }_{3j.}$	t-value	R23.(−j)	R2j.(−3)	${\beta }_{3j.}^2$	beta ${\beta }_{3j.}$	t-value
Engine size	x1	−0.959	−7.2612	0.6492	0.8507	0.9197	−0.9972	−8.4463
Horsepower	x2	1.2623	10.9342	0.4752	0.8046	1.5933	1.2723	11.3282
Rev. per mile	x4	0.0277	0.2873	0.7863	0.7207	0.0008
Fuel tank	x5	0.0153	0.1268	0.7864	0.8221	0.0002
Passengers	x6	0.1877	2.0305	0.7758	0.6953	0.0352	0.1891	2.0770
Length	x7	0.1773	1.6695	0.7792	0.7692	0.0314	0.1650	1.5857
Wheelbase	x8	0.2228	1.613	0.7797	0.8635	0.0496	0.2235	1.6409
Width	x9	−0.2398	−1.7426	0.7786	0.8625	0.0575	−0.2802	−2.2052
Turn circle	x10	−0.089	−0.9343	0.7842	0.7127	0.0079
Weight	x11	−0.7054	−3.4159	0.7561	0.9389	0.4975	−0.7126	−3.9333
	R3.2	0.7865					0.7835

The first two numeric columns show the coefficients, their t-statistics, and the coefficient of multiple determination R_3.² obtained in the regular modeling for regression (7) with x₃ as the dependent variable. Due to the criterion (9) (with x₃ as the dependent variable in place of x₁ there) and by the last row in Table 3, we see that six variables have a higher R_j.² than R_3.² = 0.7865, so there could be six coefficients $\left|{\beta }_{3j.}\right| > 1$. However, we should check these possibilities by the sufficient criterion (16), and the values needed for it are given in the next two columns of Table 4: multiple determinations R²_3.(−j) in the reduced models (11) and R²_j.(−3) in the models among the predictors (13). The quotient of the corresponding values defines the squared coefficients ${\beta }_{3j.}^2$ by the relation (14), which is shown in the next column of Table 4. We see that there is just one such value ${\beta }_{32.}^2$ above one, so the coefficient of regression x₃ with the predictor x₂ is bigger than one. The square root of these values in the last column are equal to the absolute value of the beta coefficients given in the first column of Table 4, where ${\beta }_{32.}=1.2623>1$. The values of beta squared ${\beta }_{3j.}^2$can be used for ordering the predictors by their importance in the models, and the most useful variables are x₂, x₁, and x₁₁. The same variables are identified as the most significant by the t-statistics in Table 4.

The last two columns of Table 4 present beta coefficients and their t-statistics obtained in the stepwise backward selection procedure from the full model (7), performed with the function step(initial_model, direction = “backward”) in the R software. Judging by R_3.² = 0.7835, the quality of the reduced model is close to the quality of the original model. Three predictors with lesser t-values have been eliminated, and the remaining variables have coefficients similar to those in the full model. Particularly, the parameter ${\beta }_{32.}=1.2723$ is also above one.

Another such model is the regression of x₆ shown in Table 5. It is organized similarly to Table 4: the columns contain coefficients of regression, their t-statistics, and the coefficient of multiple determination R_6.². By the criterion (9) (for x₆ as the dependent variable) and by the last row of Table 3, we see that all ten variables have a higher R_j.² than R_6.² = 0.786, so they could all be coefficients $\left|{\beta }_{6j.}\right|>1$. However, using the next two columns of Table 5 for checking these possibilities by the sufficient criterion (16), we see that ${\beta }_{6j.}^2>1$ only for one coefficient of regression with the predictor x₂. The square root of the values in the last column yields the absolute value of the beta coefficients in the first column of Table 5. In this model the coefficient ${\beta }_{62.}=-1.0277<-1$. The beta squared ${\beta }_{6j.}^2$can be used for ordering the predictors by their importance in this model, and the most valuable variables are x₂, x₁₁, and x₈, which correspond to the most significant predictors by the t-statistics in Table 5 as well.

Table 5. Model for x₆ (passengers), various coefficients R², and criterion for beta $\left|{\beta }_{6j.}\right| > 1$.

		The Full Model (7)		Reduced Models (11)	Predictors’ Models (13)	Criterion (14)	The Stepwise Model from (7)
Predictors		beta ${\beta }_{6j.}$	t-value	R26.(−j)	R2j.(−6)	${\beta }_{6j.}^2$	Beta ${\beta }_{6j.}$	t-value
Engine size	x1	0.2646	1.3557	0.7034	0.9071	0.0700	0.2697	1.4922
Horsepower	x2	−1.0277	−5.7787	0.5918	0.8881	1.0561	−1.0074	−5.8367
RPM	x3	0.2550	2.0305	0.6953	0.7758	0.0650	0.2421	2.0300
Rev. per mile	x4	0.0247	0.2194	0.7097	0.7208	0.0006
Fuel tank	x5	0.0952	0.6769	0.7083	0.8211	0.0091
Length	x7	−0.1746	−1.4031	0.7029	0.7715	0.0305	−0.1667	−1.4315
Wheelbase	x8	0.3930	2.4930	0.6879	0.8576	0.1544	0.4062	2.6486
Width	x9	0.0448	0.2743	0.7096	0.8673	0.0020
Turn circle	x10	0.0471	0.4227	0.7093	0.7151	0.0022
Weight	x11	0.8534	3.5652	0.6649	0.9383	0.7283	0.9535	4.7503
R6.2		0.7099					0.7067

Similarly to the previous table, the last two columns in Table 5 show beta coefficients and their t-statistics also obtained in the stepwise backward selection procedure from the initial model (7). The quality of fit R_6.² = 0.7067 of the reduced regression is close to the quality in the initial model. Four predictors with lesser t-values have been eliminated, and the remaining variables have coefficients similar to those in the full model. Note that the parameter ${\beta }_{62.}=-1.0074$ is still below minus one.

It is interesting to mention that in both models, by Table 4 and Table 5, x₂—Horsepower is the most important predictor by the beta coefficients and by the t-statistics. An increase in Horsepower by one standard deviation, controlling for all other predictors, leads to the effect above a standard deviation in these outcome variables. It is the enhance-synergism effect discussed, for example, in [37]. We can also note that the values of multiple determination in the columns 3 and 4 of these tables can be used in the Formulas (19)–(21) for producing the standard errors and t-values for the regression parameters.

5. Summary

The paper considered an interesting feature of multiple linear regression that some of its beta coefficients can exceed the interval [−1, 1]. Such coefficients show a big change in the dependent variable due to the unit change in predictors, so these specific variables play a role as key drivers in the outcome of the regression model. The necessary and sufficient conditions for such big beta coefficients are derived and various features obtained. The results suggest a novel interpretation for beta coefficients: each squared coefficient is defined by a quotient where, in the numerator, there is an increment of coefficient of multiple determination due to the inclusion of this particular predictor into the model, and the denominator presents a distance from the coefficient of multiple determination of this predictor by other predictors to the maximum possible value of this connection.

The obtained results help to perform the multicollinearity diagnostics in high-dimensional models. The ranks of the squared beta coefficients can serve to order the predictors by their importance in the model. Numerical examples illustrate this approach and demonstrate that it can be useful in the interpretation of the model and in solving various practical regression problems. The described properties of multiple regressions can be useful for educational purposes as well.