Estimating Forward-Looking Stock Correlations from Risk Factors

Abstract

This study provides fully mathematically and economically feasible solutions to estimating implied correlation matrices in equity markets. Factor analysis is combined with option data to receive ex ante beliefs for cross-sectional correlations. Necessary conditions for implied correlation matrices to be realistic, both in a mathematical and in an economical sense, are developed. An evaluation of existing models reveals that none can comply with the developed conditions consistently. This study overcomes this pitfall and provides two estimation models via exploiting the underlying factor structure of returns. The first solution reformulates the task into a constrained nearest correlation matrix problem. This method can be used either as a stand-alone instrument or as a repair tool to re-establish the feasibility of another model’s estimate. One of these properties is matrix invertibility, which is especially valuable for portfolio optimization tasks. The second solution transforms common risk factors into an implied correlation matrix. The solutions are evaluated upon empirical experiments of S&P 100 and S&P 500 data. They turn out to require modest computational power and comply with the developed constraints. Thus, they provide practitioners with a reliable method to estimate realistic implied correlation matrices.

1. Introduction

Stock markets are forward looking by nature, and properly assessing future developments can be a decisive competitive advantage for any market participant. The option implied correlation matrix is an important tool, as it embodies the ex ante aggregated expectation about forward-looking dependencies between securities [1,2]. Sufficient knowledge about this matrix is a key requirement for pricing basket/index options [2,3], but is also important for other purposes, such as factor-based asset pricing [4,5], forecasting [1,6,7,8], trading [8,9,10,11], managing risks [1,12,13], or understanding market behavior [14,15]. Using the implied correlation matrix can be advantageous over using historical data for the above tasks in practice. The implied correlation matrix contains forward-looking market expectations and investors’ current perception of risk [16]. Hence, it can be used to make predictions about the future at no additional assumptions. Using historic data for future predictions however, comes with the assumption that patterns in the past are also decisive for the future. Empirical analyses show that implied estimates can be a better proxy for future realized volatility [17,18] and future returns [19]. It is also documented that mean-variance efficient portfolios constructed from implied correlations outperform those constructed from past data [10,20]. In a broader context, research shows that the beyond second moment characteristics of implied data also have rich information content (e.g., [21]) and that besides options, other instruments and channels can also be used to extract forward-looking information (e.g., [22,23]). Given its superiority of timeliness and richer information content [24], there is a broad interest in solving the cross-sectional implied correlation puzzle. The goal of this paper is to provide a fully feasible solution to that puzzle.

Despite the broad interest, yet finding a realistic solution to the puzzle is a challenging task [25]. The main difficulty of solving for implied correlations is that there are far fewer options traded than there are unknown correlation pairs within the matrix. Hence, the problem is underdetermined, and multiple solutions exist in general. Current models solve the problem but, as it is shown later, they cannot guarantee to meet the important mathematical properties or impose unrealistic economic assumptions. In other words, while they deliver numerically correct results, those results can be rather useless from a practical standpoint.

The purpose of this paper is to overcome this shortfall in a two-stage process. Its first contribution is to answer the question on what requirements the implied correlation estimates must fulfill to be mathematically and economically feasible. In doing so, it identifies the shortcomings of existing models and argues that none of these models can be used as a reliable and realistic tool by investors to estimate implied correlation matrices. The second contribution of this paper is that it validates the hypothesis that fully feasible solutions can be generated via using the factor structure of returns. It proposes two alternative estimation methods to the problem at hand. Both solutions are based on the idea that a well-conditioned correlation matrix is composed of a set of factors. The first solution is quantitatively motivated. It computes the nearest implied correlation matrix with respect to a pre-specified target that can be freely chosen. Hence, it can be used as a stand-alone method or to re-establish the feasibility and invertibility of any other model’s estimate. This feature makes the quantitative solution especially attractive for applications in portfolio optimization. The second solution is economically motivated. It discusses the link between common risk factors (such as CAPM [26] or Fama-French [27,28]) and the corresponding expected correlation matrix. This closes a gap between factor-based asset pricing and ex ante beliefs. An empirical experiment shows that both solutions are computationally efficient and can be used reliably in practice.

As already discussed, multiple plausible solutions of the correlation matrix puzzle work numerically but can be rather useless from a mathematical and economic standpoint. To filter the pool of potential solutions, feasibility constraints were developed and grouped into two categories. More precisely, for a market covering n securities, a correlation matrix $C\in {\mathbb{R}}^{n\times n}$ satisfies mathematical feasibility if the following criteria are met:

(i)

It is symmetric;

(ii)

All its elements lie within the $[-1,1]$ interval;

(iii)

It has unit diagonal;

(iv)

It is positive semi-definite (psd).

Economic feasibility is fulfilled if the following criteria are met:

(v)

The matrix is ex ante free of arbitrage;

(vi)

The values have a realistic structure.

While the mathematical conditions are straightforward, the economic ones need a little more clarification. Ex ante arbitrage freeness (v) is interpreted in the following sense. Within a financial market, there are typically options traded on individual firms, but also for the market index directly. Hence, a minimum criterion for C to meet condition (v) is that it properly aggregates the prices of the constituents to match the observable price of the basket option (cp. [1,2]):

$$\begin{array}{c}\hfill (\mathrm{v}):{\sigma }_m^2=w^{\prime }\sigma C\sigma w\end{array}$$

(1)

with ${\sigma }_m^2\in {\mathbb{R}}_{+}$ as the implied variance of the market portfolio, $w\in {\mathbb{R}}^n$ the corresponding weight vector and $\sigma \in {\mathbb{R}}_{+}^{n\times n}$ a diagonal matrix of firm implied volatilities. If this condition does not hold, an arbitrageur can earn riskless profits via simultaneously buying (selling) the index (constituent) options or vice versa. Such profit opportunities should not exist following common economic sense because within the moment that such a trading opportunity emerges, it should be vanished by the traders who exploit it. Condition (vi) is also taken from the existing literature (such as [29]). Its loose definition implies that, in a real economy, the pair-wise correlations $C_{i\ne j}$ are heterogeneous across stock pairs $i,j\in \{1,\dots ,n\}$. A more narrow definition is linked to the thousands of publications in finance that decompose returns into factor loadings (see [30,31]). Given this large body of literature, a realistic structure of implied correlations can not only be expected to be heterogeneous, but also to originate from factor loadings.

To come up with a solution that matches all six conditions (i)–(vi), C is generated from an inherent k-factor structure [32,33]. In its final form, the factor structure is derived as

$$\begin{array}{c}\hfill C(X)=J\circ X{X}^{\prime }+I\end{array}$$

(2)

having $X\in {\mathbb{R}}^{n\times k}$ as factor loadings, ∘ the Hadamard product, I the identity matrix, and $J:={\mathbf{1}}_{n\times n}-I$ with ${\mathbf{1}}_{n\times n}$ as the unity matrix of dimension as indicated in the subscript. This structured representation of the implied correlation matrix has the advantage that controlling for the above requirements (i)–(vi) is easy, which allows this paper to provide fully realistic solutions.

The reading is set up as follows. Section 2 provides a comparison of existing models and identifies their violations of mathematical/economical feasibility. Section 3 discusses the factor model and the proposed solutions to the implied correlation puzzle. In Section 4, the empirical implementation upon data of the S&P 100 and S&P 500 is evaluated. Section 5 concludes. Throughout the reading, the matrix- and Hadamard products are displayed in a fashion like $J\circ X{X}^{\prime }$. The corresponding multiplication order should be read as $J\circ (X{X}^{\prime })$ and not as $(J\circ X)X^{\prime }$ (i.e., ‘matrix-before-Hadamard product’). The brackets are dropped to increase readability.

2. Discussion on Existing Models

From a literature review, four distinct concepts for solving the implied correlation matrix are identified. Each concept is presented shortly and evaluated against the feasibility constraints developed above. Before doing so, it must be highlighted that, generally, in quantitative finance, it is often more convenient to work with implied volatilities directly instead of option prices (cp. [34]). This also applies for implied correlations. Given non-flat implied volatility surfaces, a common convention is to use options of same strike and maturity [1,2,9]. The existing concepts for solving implied correlation matrices are given as follows.   

• Equi-Correlations

The literature on implied correlations was initialized by [1] in 2005. This model is not only cited in most of the subsequent research in that direction, but it is also used as the fundamental concept behind the CBOE implied correlation index. The idea of the method is as follows. Since for a given point in time, the weights w and the forward-looking measures ${\sigma }_m^2$ and $\sigma$ are available (or can be estimated), and each off-diagonal element within C can be set to the same scalar $\overline{c}$ to match the equilibrium of Equation (1). The equi-correlation $\overline{c}$ is then easily computed by

$$\begin{array}{c}\hfill \overline{C}:=\overline{c}J+I:{\sigma }_m^2\equiv w^{\prime }\sigma \overline{C}\sigma w\Rightarrow \overline{c}=\frac{{\sigma }_m^2-w^{\prime }{\sigma }^{2}w}{w^{\prime }\sigma J\sigma w}\end{array}$$

(3)

(cp. [5]). From the proof of [33] (Lem. 2.1) follows, that $\overline{C}$ will be psd if $-1/(n-1)\le \overline{c}\le 1$, which is also sufficient for mathematical feasibility. Within an economy, securities are typically, on average, positively correlated (the CBOE implied correlation index is also thoroughly positive), so this should not be much of an issue empirically. However, it is unrealistic that each correlation pair inside C takes on the same value, and hence $\overline{C}$ is seen as economically unfeasible and violates condition (vi) (cp. [9,29]). Additionally, $\overline{C}$ clearly rejects the existence of a (risk-)factor structure (also discussed in [5]), being in contradiction with empirical asset pricing literature. In case the market index also has options traded upon sub-indices (e.g., S&P 500 sector indices), it is unlikely that the equi-correlation matrix will match them. Hence, this is a violation of condition (v). Due to those reasons, the equi-correlation approach is more frequently used as an index of average diversification possibilities rather than as a correlation matrix per se (e.g., [14]).   

• Local Equi-Correlations

The equilibrium (Equation (1)) violation from existing sub-index options can be easily resolved by introducing local equi-correlations, where the correlations are averaged out within sub-squares of the matrix. This idea is mentioned in [35], who use options on the S&P 500 market index and 10 S&P 500 sector sub-indices. Herein, always the average correlation within a sub-index is computed, before computing the global equi-correlations outside the sub-portfolio. For a sub-portfolio p, Equation (1) must hold for its weights $w_p$ and volatility ${\sigma }_p$. Local equi-correlations of different ps are always centered alongside the diagonal of C. This raises the issue that large parts of C cannot be covered and obviously, with a handful of sub-indices and a very large number of unknown correlation pairs, the resolving correlation matrix will still give a very blurred and unrealistic picture. For example, if $n=500$, there are 124,750 unknown correlation pairs. With 10 sub-indices, there are only 11 distinct estimates. For this definition of local equi-correlations, condition (iv) does not hold, as the resulting matrix is not necessarily psd. Additionally, as argued in [5], the known factor structure of security markets (condition (vi)) will still be ignored. The work of [5] further demonstrates how the factor structure can be resolved from calibrating also off-diagonal local equi-correlations. However, also this refinement yields a very blurred estimate of the implied correlation matrix. Hence, as the local equi-correlation method only slightly improves precision, compared to the global equi-correlation, it still causes unrealistic estimates of C and is thus economically not feasible.

• Adjusted Ex Post

This method combines (local) equi-correlations with backward-looking estimates to approximate the forward-looking implied correlation matrix [9,35]. For the discussion of this model, it has to be kept in mind that due to the stochastic nature of variances and correlations, option prices are documented to carry risk premia for variance (VRP) and for correlation (CRP; see [36,37,38,39]). Let $\mathbb{P}$ denote investor expectations under the physical and $\mathbb{Q}$ under the risk-neutral probability measure. Option implied volatilities are known to be $\mathbb{Q}$-measured. The physically expected correlation matrix can be denoted by $C_{\mathbb{P}}$. Then, the difference between implied market and aggregated individual volatilities can be used to express the volatility quoted ex ante CRP as $CRP={\sigma }_m^2-w^{\prime }\sigma C_{\mathbb{P}}\sigma w$ (cp. [6,37]). Building on this idea, A can be defined as a backward-looking estimated correlation matrix; in the example of [9] it is simply the 1 year historically realized return correlations, but A can also be chosen from a more sophisticated model (e.g., incorporating mean-reversion [35]). For the model of [9] (indicated by the superscript $AEP1$), two crucial assumptions are obligatory. First, they introduce that the backward-looking A is equivalent to the forward-looking $\mathbb{P}$ matrix, $A\equiv C_{\mathbb{P}}$. While investors use backward-looking information to form their beliefs, there is no fundamental reasoning why this should hold, as A does not carry information on the market outlook (see, e.g., [1]). The second assumption is that the correlation risk premium enters into the matrix in the specific form of

$$\begin{array}{c}\hfill C_{\mathbb{Q}}^{AEP1}:=C_{\mathbb{P}}-\alpha ({\mathbf{1}}_{n\times n}-C_{\mathbb{P}})\end{array}$$

(4)

with $\alpha$ as a scalar calibrating for the correlation risk premium and $C_{\mathbb{Q}}^{AEP1}$ as their estimate of the implied correlation matrix. The two assumptions come at a mathematical convenience. Let $\widehat{\alpha }:=-\alpha$, then by rearranging terms, the equation can be brought into the form $C_{\mathbb{Q}}^{AEP1}=\widehat{\alpha }{\mathbf{1}}_{n\times n}+(1-\widehat{\alpha })C_{\mathbb{P}}$, known in the literature as the ’weighted average correlation matrix’ [40]. Since both ${\mathbf{1}}_{n\times n}$ and $C_{\mathbb{P}}$ are psd, and the sum of two psd matrices is also psd, the mathematical feasibility of $C_{\mathbb{Q}}^{AEP1}$ holds for $\alpha \in (-1,0]$, which is fulfilled when $CRP>0$. Empirically, however, $\alpha$ is likely to fall outside that range, and this method does not comply with condition (iv) in general. For example, on monthly S&P 100 data from 1996 to 2020 with A as the one year historically realized correlation matrix, it can be observed that $CRP>0$ in 156 and $CRP<0$ in 144 of the 300 monthly estimates, thus the requirement did not hold for 48% of the time. This rejection of mathematical feasibility is also the main critique stated in [29], who provide a workaround for the $CRP<0$ cases. In this model, whenever $\alpha >0$ the matrix ${\mathbf{1}}_{n\times n}$ of Equation (4) is replaced by the ’equi-correlation lower-bound’, L defined as $\forall i\ne j:L_{ij}=-1/(n-1)$ and unit diagonal. Here, L is simply the smallest possible psd equi-correlation matrix (see the discussion on equi-correlations above). Following the modification of [29] (indicated by $AEP2$), the implied correlation matrix is computed by

$$\begin{array}{c}\hfill C_{\mathbb{Q}}^{AEP2}=\left\{\begin{array}{c}\widehat{\alpha }{\mathbf{1}}_{n\times n}+(1-\widehat{\alpha })C_{\mathbb{P}},\mathrm{for}CRP\ge 0\Rightarrow \widehat{\alpha }\ge 0\hfill \\ \widehat{\alpha }L+(1-\widehat{\alpha })C_{\mathbb{P}},\mathrm{for}\widehat{\alpha }<0\hfill \end{array}\right.\end{array}$$

(5)

According to [9], consistent investor preferences require that all $\mathbb{P}$-correlation pairs are scaled in the same direction, under general risk aversion; this means up if $CRP>0$ and down when $CRP<0$. This consistency clearly holds in [9,29] for the $\widehat{\alpha }\ge 0$ cases. Analyzing the workaround of [29] in greater detail, one recognizes that off-diagonal entries inside L are slightly negative and close to zero (e.g., −0.01 for $n=100$ and −0.002 for $n=500$). As a result, for $\widehat{\alpha }<0$, almost every negative $\mathbb{P}$-correlation pair will be up-scaled, while positive ones will be down-scaled at the $\mathbb{P}$-$\mathbb{Q}$ transformation. Therefore, the [29] workaround repairs the mathematical feasibility of the basic adjusted ex post model, but at the same time causes an inconsistent implementation of the correlation risk premium, thus violating condition (v). This yields into a rejection of economical feasibility, leaving an insufficient solution of the implied correlation matrix behind. While mathematical/economical flaws still persist, compared to the other concepts in place, the adjusted ex post method seems to be the most realistic one. The nearest implied correlation algorithm as introduced in Section 3 can be applied upon these models to repair feasibility.

• Skewness Approach

Of note, Refs. [10,12] use a CAPM-like model, introduce economic assumptions to cancel out mathematical relationships, and estimate implied correlations between a stock and the market portfolio by combining option-implied volatilities with risk-neutral skewness. Since CAPM is a factor model, the estimates correspond to X of Equation (2), and their approach can thus be used for a solution to the implied correlation matrix. However, this model does not coerce with market conditions (Equation (1)), nor does it stick to the boundaries stated in condition (ii). So, mathematical and economical feasibilities are ignored. On the other hand, the psd condition is easily met when combined with the factor model of Equation (2).

Table 1. The evaluation of existing models with respect to the developed requirements for realistic implied correlation matrices. A check mark ✓ indicates that the model is compliant with the condition. If it does not guarantee to comply, then it is marked with a ✗. A realistic solution has to fulfill all constraints (i)–(vi).

	Mathematical				Economical
	(i)	(ii)	(iii)	(iv)	(v)	(vi)
	Symmetric	|Cij| ≤ 1	Unit Diag.	Pos. Semi-Def.	Arbitrage-Free	Hetero. Struct.
Model:
Equi-Correlations	✓	✓	✓	✓	✗	✗
Local Equi-Correlations	✓	✓	✓	✗	✓	✗
Adj. Ex Post of [9], $AEP1$	✓	✓	✓	✗	✓	✓
Adj. Ex Post of [29], $AEP2$	✓	✓	✓	✓	✗	✓
Skewness Approach	✓	✗	✓	✓	✗	✓

summarizes the assessment of existing models with respect to the constraints developed in Section 1. It can be observed that none of the presented models meet all feasibility constraints. It appears that while most models handle mathematical feasibility relatively well, they do so at the expense of being economically justifiable. From the results of Table 1, it can be hypothesized that all drawbacks can be overcome when an implied correlation matrix is estimated from factors. This hypothesis is tested in the following section.

3. Solutions from Factor Structures

This section develops the two proposed approaches to estimate a mathematically and economically feasible correlation matrix. The section starts with a discussion on how correlation matrices can be estimated from factors. Thereafter, quantitative and economic estimation approaches are introduced that both ensure meeting the mathematical and economical feasibility constraints.

Generally, implied volatilities are used as input parameters, which can be estimated in various ways (see [34] for a detailed discussion). Dependent on what kind of implied volatilities to use (e.g., centered vs. directly parameterized [41]), the methodology is not limited to Pearson-type correlation matrices and can also be used within more sophisticated option pricing models, adjusting for non-normal distributions. An example for the case of a multivariate variance-gamma process can be found in Appendix A.

The correlation structure that is typical for financial markets can be expressed with the multi-factor copula model described by [32]. This model is used in a nearest correlation matrix context by [33]. The factor generating core of the model follows a simple but intuitive definition, that is

$$\begin{array}{c}\hfill \xi =X\eta +Fϵ\end{array}$$

(6)

where $\xi \in {\mathbb{R}}^n$ describes a random vector, $F\in {\mathbb{R}}^{n\times n}$ a diagonal matrix and $\eta \in {\mathbb{R}}^k$ corresponds to the factor’s magnitude. All three vectors, $\xi$, $\eta$ and $ϵ\in {\mathbb{R}}^n$ are defined to have zero mean and unit variance; $\eta$ and $ϵ$ are orthogonal,

$$\begin{array}{c}\hfill E\left[\xi \right]=E\left[\eta \right]=E\left[ϵ\right]=0,var(\xi )=var(\eta )=var(ϵ)=1,cov(\eta ,ϵ)=0\end{array}$$

(7)

From this, it follows that

$$\begin{array}{c}\hfill cov(\xi )=E\left[\xi {\xi }^{\prime }\right]=X{X}^{\prime }+F^2\end{array}$$

(8)

Since $\xi$ has unit variance, $cov(\xi )$ turns out to be a correlation matrix with the boundaries of

$$\begin{array}{c}\hfill \forall i=\{1,\dots ,n\}:\sum _{d=1}^k{X}_{i,d}^2+F_{ii}^2=1\Rightarrow \sum _{d=1}^k{X}_{i,d}^2\le 1\end{array}$$

(9)

such that X is necessarily limited to $[-1,1]$. From the above equation, it follows that every $X_{i,d}^2$ corresponds to the goodness-of-fit of stock i explained by the the risk factor d, and $F_{ii}^2$ to the unsystematic correlation, which cannot be explained by the given set of factors. From the notation, it follows that $X_{i,d}$ itself can be interpreted as the correlation of a stock i to a risk factor d (e.g., $X_{i,m}=corr(i,m)$ in CAPM). $F^2$ itself is of less importance for further modeling, as it can be easily computed once a feasible solution to X is found. Therefore, the factor-structured correlation matrix of Equation (8) can be equivalently rewritten in a form where $F^2$ is suppressed and X remains the only unknown. This reformulation is used in the specific context as the factor-structured implied correlation matrix $C(X)$, which is now denoted by Equation (2). Other works in the literature (e.g., [33]) write $C(X)=X{X}^{\prime }-diag(X{X}^{\prime })+I$ with $diag(X{X}^{\prime })$ as the diagonal matrix of $X{X}^{\prime }$. This alternative definition yields the same $C(X)$; however, given the cleaner notation of Equation (2) and its superiority in terms of computation, as J can be pre-computed (thus not iterated) while $diag(X{X}^{\prime })$ cannot, Equation (2) is preferred in this paper. Below, it is shown how Equation (2) generates a well-conditioned correlation matrix.

Theorem 1.

The matrix $C(X)$ is a mathematically feasible correlation matrix whenever

$$\begin{array}{c}\hfill X\in \Omega :=\{X\in {\mathbb{R}}^{n\times k}:\sum _{d=1}^k{X}_{i,d}^2\le 1,\forall i=\{1,\dots ,n\}\}\end{array}$$

(10)

Proof. 

Since $X{X}^{\prime }$, J and I are symmetric, it follows that $C(X)$ is symmetric and condition (i) is guaranteed. From $X\in \Omega$ it follows that condition (ii) holds. As for condition (iii), $X{X}^{\prime }\circ J$ has zero diagonal such that addition by I guarantees unit diagonal. $C(X)$’s off-diagonal elements are generated from $X{X}^{\prime }$, which is positive semi-definite by construction. Since $diag(C(X))={\mathbf{1}}_{n\times 1}\ge diag(X{X}^{\prime })$, it follows that $C(X)$ is positive semi-definite given that $X\in \Omega$. Therefore, conditions (i)–(iv) are met.    □

In empirical applications, it is unrealistic that asset returns are perfectly (anti-)correlated with risk factors. Therefore, excluding −1 and 1 from the factor loadings, X, is not only more appropriate, but it will also come with the practical convenience that it guarantees $C(X)$ to be non-singular (invertible). This is practical for portfolio optimizations where the inverse of the correlation matrix can be used for closed-form solutions. An example of such an optimization task is the computation of the global minimum variance portfolio.

Lemma 1.

The correlation matrix $C(\stackrel{\overline{¯}}{X})$ is invertible if its factor loadings $\stackrel{\overline{¯}}{X}$ are not perfectly (anti-)correlated,

$$\begin{array}{c}\hfill \stackrel{\overline{¯}}{X}\in \stackrel{\overline{¯}}{\Omega }:=\{\stackrel{\overline{¯}}{X}\in {\mathbb{R}}^{n\times k}:\sum _{d=1}^k{\stackrel{\overline{¯}}{X}}_{i,d}^2<1,\forall i=\{1,\dots ,n\}\}\end{array}$$

(11)

Proof. 

For the one-factor case of $k=1$, ref. [33] (Cor. 3.3) showed that non-singularity holds if at most one entry in $\left|X\right|$ is 1. Hence, $C(\stackrel{\overline{¯}}{X})$ subject to $k=1$ is guaranteed to be non-singular. The behavior for $k>1$ is similar in the sense that at most, one entry per column of $\left|X\right|$ is allowed to take on 1. This condition becomes more obvious when looking at the two-factor case $k=2$. In this case, X is structured as

$$\begin{array}{c}\hfill X=\left(\begin{array}{cc}y& u\\ z& 0\end{array}\right)\Rightarrow X{X}^{\prime }=\left(\begin{array}{cc}y{y}^{\prime }+u{u}^{\prime }& y{z}^{\prime }\\ z{y}^{\prime }& z{z}^{\prime }\end{array}\right)\end{array}$$

(12)

where $y,u$ represent vectors of dimension $m=n-q$, and z is of dimension q. The vectors have the characteristics of $y,u\in (-1,1)$ and $\left|z\right|={\mathbf{1}}_{q\times 1}$. Hence, the first column of $\left|X\right|$ has q many entries equal to 1. When $X\in \Omega$, it follows that all X entries in the same row as z take on zero (if not z itself). This means that the columns of X can be arbitrarily exchanged, but it will not affect $X{X}^{\prime }$. Further, from the permutation argument of [33] (Cor. 3.3), it is known that the position of z does not matter. The upper-left corner of the correlation matrix can be abbreviated by $C_1(X)=(y{y}^{\prime }+u{u}^{\prime })\circ J_{m\times m}+I_{m\times m}$, which represents a feasible correlation matrix itself. Thus, $C(X)$ is alternatively represented by

$$\begin{array}{c}\hfill C(X)=\left(\begin{array}{cc}{C}_1(X)& y{z}^{\prime }\\ z{y}^{\prime }& z{z}^{\prime }\end{array}\right)\end{array}$$

(13)

for which [33] (Cor. 3.3) showed to be non-singular if $X\in \Omega$ and $q\le 1$. Therefore, when using $\stackrel{\overline{¯}}{X}$ instead, it follows that $q=0$ and $C(\stackrel{\overline{¯}}{X})$ are guaranteed to be invertible. Since adding additional factors $k>2$ will enter into $C_1(\stackrel{\overline{¯}}{X})$ only, it does not change the conclusion.    □

For economic feasibility Equation (2) needs to be combined with the equilibrium conditions of Equation (1). Therefore, the options traded on the market- and its sub-indices restrict possible values of X. These restrictions are referenced with ’market-constraints’ from now on. Given $n_c$ many market-constraints, the general solution to the factor-structured implied correlation matrix now evolves as

$$\begin{array}{c}\hfill C(X)\mathrm{subject}\mathrm{to}\left\{\begin{array}{c}X\in \Omega \hfill \\ {\sigma }_j^2=w_j^{\prime }\sigma C(X)\sigma w_j,\forall j=\{1,\dots ,n_c\}\hfill \end{array}\right.\end{array}$$

(14)

From this definition, it follows that $\Omega$ is a closed convex set of real numbers. This set can be expressed as inequality constraints, and the market constraints are of the equality type. From the equality constraints, it follows that the feasible set is a subset (hypersurface if $n_c=1$) within $\Omega$, denoted by $\dot{\Omega }$ in the subsequent. For optimization purposes, it is convenient to formulate the constraints as follows.

Definition 1.

Let $h(X)$ describe the inequality constraints,

$$\begin{array}{c}\hfill h(X):={\mathbf{1}}_{n\times 1}-(X\circ X){\mathbf{1}}_{k\times 1},h(X):{\mathbb{R}}^{n\times k}\to {\mathbb{R}}^n\end{array}$$

(15)

such that $h(X)\ge {\mathbf{0}}_{n\times 1}$ guarantees that $X\in \Omega$, which is necessary and sufficient for mathematical feasibility.

In practical terms, when one wants to explicitly ensure that the implied correlation matrix is invertible, the easiest way to do so is by introducing a very slim tolerance $\mathtt{tol}$ for $h(X)\ge {\mathtt{tol}}_{n\times 1}$.

Definition 2.

Let $g(X)$ define the vector of market constraints, which simply stacks Equation (1) for every (sub-) index that has option contracts traded upon,

$$\begin{array}{c}\hfill \forall j=\{2,\dots ,n_c\}:g(X)=\left(\begin{array}{c}{w}^{\prime }\sigma C(X)\sigma w\\ w_j^{\prime }\sigma C(X)\sigma w_j\\ ⋮\end{array}\right)-\left(\begin{array}{c}{\sigma }_m^2\\ {\sigma }_j^2\\ ⋮\end{array}\right),g(X):={\mathbb{R}}^{n\times k}\to {\mathbb{R}}^{n_c}\end{array}$$

(16)

with $j=1$ reserved for the market portfolio m. Hence $g(X)={\mathbf{0}}_{n_c\times 1}$ is necessary for economic feasibility.

The equality constraint is necessary, but not sufficient for economic feasibility. The best example is the (local) equi-correlation model, which meets this requirement but yields an unrealistic implied correlation matrix, such that economic feasibility is not met.

In general, the number of unknown correlation pairs of a correlation matrix $C(X)$ as in Equation (14) is $n(n-1)/2$, and the number of observable implied volatilities is $n+n_c$. Empirically, it is given that $n+n_c\ll n(n-1)/2$ almost surely, and hence, many different numerical solutions to X exist. To come up with a reasonable choice of X, this paper argues in favor of two approaches: first, a purely quantitative approach computing the feasible implied correlation matrix that is nearest to some pre-specified target; second, an economic approach using factor-based asset pricing models, assuming that $\mathbb{P}$ expected factor loadings can be estimated.

3.1. Quantitative Approach: Computing Nearest Implied

This solution is based on the case when there exists an educated guess about the forward-looking correlation matrix. For example, this educated guess could be derived from historically realized correlations, from a GARCH forecast, or from a not fully feasible estimate of the discussed models in Section 2. As before, this educated guess is denoted by A, but different to [9,29], the assumption that $A\equiv C_{\mathbb{P}}$ is not required. Such an educated guess allows to search for a $C(X)$ that is as similar as possible to A, but satisfies all market constraints in order to be considered a feasible implied correlation matrix.

3.1.1. Formulating the Problem

Finding the most similar of a target matrix is known in the mathematical literature as the nearest correlation matrix problem (e.g., [33]). Similarity between the generated $C(X)$ and the target matrix A can be quantified by the squared Frobenius norm between them,

$$\begin{array}{c}\hfill f(X)={∥C(X)-A∥}_F^2,f(X):{\mathbb{R}}^{n\times k}\to {\mathbb{R}}_{+}\end{array}$$

(17)

which, when introducing $\widehat{A}=A-I$, can also be written as

$$\begin{array}{c}\hfill f(X)={∥J\circ X{X}^{\prime }-\widehat{A}∥}_F^2\end{array}$$

(18)

Lemma 2.

The gradient of $f(X)$ is

$$\begin{array}{c}\hfill {\nabla }_{X}f=4(J\circ X{X}^{\prime }-\widehat{A})X,{\nabla }_{X}f\in {\mathbb{R}}^{n\times k}\end{array}$$

(19)

Proof. 

For simplicity, let $M:=J\circ X{X}^{\prime }-\widehat{A}$ and the Frobenius product (trace operator) be denoted by the colon symbol ‘:’, such that $tr(M^{\prime }M)=M:M$. The problem can then be written as $f(X)={∥M∥}_F^2=M:M$ and its differential is thus $df=2M:dM$. The differential of M itself is $dM=J\circ d(X{X}^{\prime })$, and $d(X{X}^{\prime })=dX{X}^{\prime }+Xd{X}^{\prime }$. Both: and ∘ are mutually commutative operators, and hence $M:J\circ d(X{X}^{\prime })=M\circ J:d(X{X}^{\prime })$. Therefore, substituting back in yields

$$\begin{array}{c}\hfill df=2J\circ M:(dX{X}^{\prime }+Xd{X}^{\prime })=2(J\circ M+J^{\prime }\circ M^{\prime }):dX{X}^{\prime },\end{array}$$

(20)

since J and M are symmetric, this reduces to $df=4J\circ MX:dX$, and 

$$\begin{array}{c}\hfill {\nabla }_{X}f=\frac{\partial f}{\partial X}=4J\circ MX\end{array}$$

(21)

follows. As $J\circ J=J$ and $J\circ \widehat{A}=\widehat{A}$, plugging back in M gives Equation (19). This result is a reduced-form equivalent to the gradient as found in [33], who derived it as ${\nabla }_{X}f=4(X{X}^{\prime }-diag(X{X}^{\prime })-\widehat{A})X$.    □

Popular optimization methods such as sequential quadratic programming (SQP) build on the Lagrangian function, which for the problem at hand can be formulated as

$$\begin{array}{c}\hfill \mathcal{L}(X,\lambda ,\kappa )=f(x)+{\lambda }^{\prime }g(x)+{\kappa }^{\prime }h(x))\end{array}$$

(22)

with $\lambda \in {\mathbb{R}}^{n_c}$ and $\kappa \in {\mathbb{R}}^n$ representing the Lagrangian multipliers of the market constraints introduced in Section 3. Working with the Lagrangian is probably the most common practice, and hence the respective gradients are also reported below.

Lemma 3.

The gradient of ${\lambda }^{\prime }g(X)$ with respect to X is

$$\begin{array}{c}\hfill {\nabla }_X({\lambda }^{\prime }g(X))=2\sigma \left(J\circ \sum _{j=1}^{n_c}{\lambda }_j{w}_j{w}_j^{\prime }\right)\sigma X\end{array}$$

(23)

Proof. 

${\lambda }^{\prime }g(X)$ can be written as ${\sum }_{j=1}^{n_c}{\lambda }_j{g}_j(X)$. One market constraint $g_j(X)$, can be rearranged into the form $g_j(X)=w_j^{\prime }\sigma (J\circ X{X}^{\prime })\sigma w_j+w_j^{\prime }{\sigma }^2{w}_j-{\sigma }_j^2$. To abbreviate notation, let $\widehat{M}:=J\circ X{X}^{\prime }$ and $B:=\sigma w_j{w}_j^{\prime }\sigma$, both being symmetric matrices. For the gradient ${\nabla }_X{g}_j$, all non-X terms cancel out, so the focus is on $w_j^{\prime }\sigma \widehat{M}\sigma w_j$, which can be expressed as the matrix trace $tr(B\widehat{M})$. The differential of $g_j$ is thus reduced to $d{g}_j=tr(Bd\widehat{M})=B:d\widehat{M}$. From the proof of Lemma 2 it follows that $d\widehat{M}=J\circ (dX{X}^{\prime }+Xd{X}^{\prime })$. Therefore, the differential writes

$$\begin{array}{c}\hfill d{g}_j=B\circ J:(dX{X}^{\prime }+Xd{X}^{\prime })=2(B\circ J):dX{X}^{\prime }=2(B\circ J)X:dX\end{array}$$

(24)

hence the gradient of $g_j$ is given as ${\nabla }_X{g}_j=2(B\circ J)X$. Here, $B\circ J$ equals $\sigma (J\circ w_j{w}_j^{\prime })\sigma$, so substituting back in and multiplying by ${\lambda }_j$ gives

$$\begin{array}{c}\hfill {\lambda }_j{\nabla }_X{g}_j=2{\lambda }_j\sigma (J\circ w_j{w}_j^{\prime })\sigma X\end{array}$$

(25)

thus, the lemma evolves from ${\nabla }_X({\lambda }^{\prime }g(X))={\sum }_{j=1}^{n_c}{\lambda }_j{\nabla }_X{g}_j$.    □

Lemma 4.

Let $D_{\kappa }$ denote the diagonal matrix of κ, the gradient of ${\kappa }^{\prime }h(X)$ can then be written as

$$\begin{array}{c}\hfill {\nabla }_X({\kappa }^{\prime }h(X))=-2D_{\kappa }X\end{array}$$

(26)

Proof. 

The term ${\kappa }^{\prime }h(X)$ can alternatively be written as ${\kappa }^{\prime }{\mathbf{1}}_{n\times 1}-{\kappa }^{\prime }{X}^{\circ 2}{\mathbf{1}}_{k\times 1}$, where $X^{\circ 2}$ describes that each element in X is squared (Hadamard quadratic). Thus, $\partial ({\kappa }^{\prime }h(X))/\partial x_{id}=-2{\kappa }_i{x}_{id}$, which for $i\in \{1,\dots ,n\},d\in \{1,\dots ,k\}$ yields the result above.    □

3.1.2. Numerical Method

The nearest implied correlation matrix can now be attained by the following optimization:

$$\begin{array}{c}\hfill X=\underset{X}{arg\; min}\mathcal{L}(X,\lambda ,\kappa )\end{array}$$

(27)

A performance comparison of numerical methods for a general nearest correlation matrix problem (i.e., without equality constraints $g(X)$) was presented by [33]. They recommend the spectral projected gradient (SPGM) as the most efficient and reliable method for this task (compared algorithms include ‘alternating directions’, ‘principal factors method’, ‘spectral projected gradient’ and Newton-based methods; see [33] for details). Based on this finding, the SPGM is also used as the main algorithm within this study for solving the implied correlation matrix puzzle. A brief comparison to a SQP-based solver is included in Section 4.

First, a brief elaboration on the SPGM algorithm and expansions that are relevant for applications in this paper are presented in this paragraph. SPGM was initially introduced by [42] and is used in many different studies. The study of [43] summarizes a long list of corresponding works and the applications to financial data include [25,33]. A detailed discussion on the algorithm can be found in [43,44]. Technically, the SPGM comes with three advantages when compared to other potential candidates. First, the method ’only’ requires the gradient—known for this task—but not the Hessian matrix. Second, the non-linear constraints are guaranteed to be satisfied and third, the method is ensured to converge toward the optimum (cp. [42]). In a nutshell, the method minimizes a continuous differentiable function on a nonempty closed convex set via projecting possible values of X back onto the feasible set. Consequently, a key requirement is that projections on the feasible set can be made at low computational effort. If so, then the method provides a very efficient way of handling the constraints. For the application at hand, handling the equality constraint alongside $\Omega$ can become tricky. This issue is resolved by combining SPGM with a more general inexact restoration framework (e.g., [45] for a discussion). The IR-SPGM was introduced by [46] who also provided an in-depth explanation of the algorithm, thus the details will not be repeated here. Roughly speaking, the algorithm can be broken down into two phases: a restoration phase, projecting X back onto the feasible set, and an optimization phase, computing the step size.

For the particular application at hand, greater attention has to be paid on the projection function. In case of this paper, $\dot{\Omega }$ defines the feasible region that satisfies both $\Omega$ and the market constraints. As mentioned by [46], for some X outside $\dot{\Omega }$, the projection function $P_{\dot{\Omega }}(X)$ should be defined such that $P_{\dot{\Omega }}(X)\approx {arg\; min}_{\dot{X}}{\parallel X-\dot{X}\parallel }_2$ where $\dot{X}\in \dot{\Omega }$. This means that the projection function should find a point in $\dot{\Omega }$ that is approximately the closest to X (inexact restoration). Obviously, $P_{\dot{\Omega }}(X)$ is an orthogonal projection if an $\dot{X}\perp X$ exists, which is not necessarily the case, given the upper boundaries of $\dot{X}$ (Equation (9)). Figure 1 visualizes the projection and feasible set on a simplified example.

For the inexact restoration framework, the projection is split up into two functions, a projection onto the inequality constraint $P_{\Omega }(·)$ and one onto the equality constraint $P_E(·)$. As for $P_{\Omega }(·)$, [33] already discussed that this projection is easily carried out by replacing every row i of X which exceeds ${\sum }_{d=1}^k{X}_{i,d}^2>1$ by $X_i/\parallel X_i\parallel$. Hence, the projection on $\Omega$ comes at very low computational effort.

For the $P_E(·)$ projection, given the market constraint ${\sigma }_m^2$ and defining $X_E$ to be any point that fulfills $g(X_E)=0$, the Lagrangian can be formulated as

$$\begin{array}{c}\hfill {\mathcal{L}}_E(X_E,{\lambda }_E)={\parallel X_E-X\parallel }_F^2-{\lambda }_E[v^{\prime }(X_E{X}_E^{\prime }\circ J+I)v-{\sigma }_m^2]\end{array}$$

(28)

To simplify notation, introduce $v=\sigma w$. With the results from above, the gradient of ${\mathcal{L}}_E$ can now be written as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\nabla }_{X_E}{\mathcal{L}}_E& =2(X_E-X)-2{\lambda }_E(v{v}^{\prime }\circ J)X_E\mathrm{and}\hfill \\ \hfill {\nabla }_{{\lambda }_E}{\mathcal{L}}_E& ={\sigma }_m^2-v^{\prime }(X_E{X}_E^{\prime }\circ J+I)v\hfill \end{array}\end{array}$$

(29)

To minimize the Lagrangian function, both gradients are set to zero, ${\nabla }_{X_E}{\mathcal{L}}_E={\nabla }_{{\lambda }_E}{\mathcal{L}}_E=0$. Therefore, from rearranging terms of ${\nabla }_{X_E}{\mathcal{L}}_E$, $X_E$ can be defined as

$$\begin{array}{c}\hfill X_E={(I-{\lambda }_{E}v{v}^{\prime }\circ J)}^{-1}X\end{array}$$

(30)

The inverse of ${(I-{\lambda }_{E}v{v}^{\prime }\circ J)}^{-1}$ can be expressed following the expansion procedure of [47]. Since in a real economy the elements of $v{v}^{\prime }\circ J$ are generally smaller than 1 and close to 0, higher-order terms, such as ${(v{v}^{\prime }\circ J)}^2$, converge toward zero very fast. If this was not the case, the implied volatilities could be time-scaled down, for example, from yearly to daily such that $\forall i:v_i<1$. Consequently, the inverse is efficiently approximated by

$$\begin{array}{c}\hfill {(I-{\lambda }_{E}v{v}^{\prime }\circ J)}^{-1}\approx I+{\lambda }_{E}v{v}^{\prime }\circ J\Rightarrow P_E(X):X_E\approx X+{\lambda }_E(v{v}^{\prime }\circ J)X\end{array}$$

(31)

Plugging back into the equality constraint ${\nabla }_{{\lambda }_E}{\mathcal{L}}_E=0$ and rearranging terms, ${\lambda }_E$ can then be solved from a simple quadratic equation with two solutions,

$$\begin{array}{c}\hfill {\lambda }_{E,\pm }=\frac{{\lambda }_{E,1}\pm \sqrt{{\lambda }_{E,1}^2-4{\lambda }_{E,0}{\lambda }_{E,2}}}{2{\lambda }_{E,0}},\mathrm{with}\left\{\begin{array}{c}{\lambda }_{E,0}=v^{\prime }[{(v{v}^{\prime }\circ J)}^2\circ J]v\hfill \\ {\lambda }_{E,1}=2v^{\prime }[X{X}^{\prime }(v{v}^{\prime }\circ J)\circ J]v\hfill \\ {\lambda }_{E,2}=v^{\prime }[X{X}^{\prime }\circ J+I]v-{\sigma }_m^2\hfill \end{array}\right.\end{array}$$

(32)

Having estimated both solutions of ${\lambda }_{E,\pm }$ and inserting the results into Equation (31), it is easy to attain whether $X_{E,+}$ or $X_{E,-}$ is closer to X. Hence, it evolves that also the projection $P_E(X)$ is inexpensive to compute. In case ${\lambda }_E$ is complex, taking only its realistic part is considered to be sufficient, given the inexact restoration framework.

$\dot{X}$ that fulfills both $\dot{X}\in \Omega$ and $g(\dot{X})=0$ can now be found following a simple alternating projection algorithm that proceeds as follows. It starts with the initial projection on the market-constraint $P_E(X)$. If $P_E(X)\in \Omega$, then $\dot{X}$ has already been found. Otherwise, it locks in whether ${\lambda }_{E,+}$ or ${\lambda }_{E,-}$ was used and takes only the upper or lower solution subsequently. This is useful because, therefore, the market-constraint becomes convex, and alternating the projections $P_{\Omega }(·)$ and $P_E(·)$ converges toward the optimal solution $\dot{X}$. The alternating procedure can be stopped if a certain tolerance level for $g(·)$ has been reached. With the defined projection procedure, the optimization under IR-SPGM can now be carried out following the guidelines of [46].

3.2. Economic Approach

As an alternative to the above quantitative approach, a set of expected risk factors can also be utilized to estimate the implied correlation matrix. The factor exposures can be computed either from a statistical (e.g., principal component analysis) or from an economic routine (such as CAPM [26], Fama-French [27,28]). As for the latter, X corresponds to the correlation between stocks and the risk factors. From the literature on factor analysis, it is known that if X can be found such that $X{X}^{\prime }$ has a unit diagonal, then a correlation matrix is fully explained by its reduced structure X. It is also known that if $k=n$, then a correlation matrix can be fully described by a set of factors (something that is known, e.g., from eigenvalue decomposition). So, either way, a (psd) correlation matrix C is an aggregate with X as the underlying structure. In the models of [9,29], the correlation risk premium is incorporated, stock pair-wise. Different to that, this paper assumes that investors use factor-based asset pricing models and thereby quote the risk-premium on the factor level.

Whenever the implied correlation matrix is described by a one-factor model, $k=1$, the weighting idea of [9] can be applied to the factor loading level to achieve the $\mathbb{P}-\mathbb{Q}$ transformation. Taking a one-factor model is no oversimplification, because in financial markets, it is documented (e.g., [48]) that the first principal component already explains a very large portion of the realized correlation matrix. It is also reported that the first principal component is typically interpreted as CAPM’s market portfolio. In the sense of [9,29], it can be assumed that the $\mathbb{P}$ expected correlations to risk factors can be estimated on a reasonable basis. Next, following their concept, $X_{\mathbb{P}}$ is weighted against ${\mathbf{1}}_{n\times k}$ if $CRP>0$. In case $CRP<0$, then the weighting is made toward the lower boundary, which can now be chosen as $-{\mathbf{1}}_{n\times k}$. Hence, investors are modeled to quote the risk-premium on the factor exposure rather than on the single stock level. Considering the diversification effects, the mark-up to risk factors seems more realistic from an economic point of view. Furthermore, the factor-level implementation will not run into an economically inconsistent-scaling as in [29] and there is also no psd problem as in [9]. When introducing the sign of the correlation risk premium as $s:=\mathrm{sign}(CRP)$ that indicates up- or down-scaling, the transformation can then, similar to Equation (4), be written as

$$\begin{array}{c}\hfill X_{\mathbb{Q}}=X_{\mathbb{P}}+\tilde{\alpha }{X}_{\Delta },\mathrm{with}{X}_{\Delta }=s{\mathbf{1}}_{n\times k}-X_{\mathbb{P}}\end{array}$$

(33)

It generally holds that $\tilde{\alpha }\in [0,1]$ and the implied correlation matrix is given by Equation (2) using $X_{\mathbb{Q}}$. To compute $\tilde{\alpha }$, one has to match the market constraint in Equation (1), from which it follows that $\tilde{\alpha }$ can be explicitly solved for by

$$\begin{array}{c}\hfill \tilde{\alpha }=\frac{-{\sigma }_{\mathbb{P},\Delta }^2+s\sqrt{{\sigma }_{\mathbb{P},\Delta }^4-{\sigma }_{\Delta }^2({\sigma }_{\mathbb{P}}^2-{\sigma }_m^2)}}{{\sigma }_{\Delta }^2}\mathrm{with}\left\{\begin{array}{cc}{\sigma }_{\mathbb{P}}^2\hfill & =w^{\prime }\sigma (X_{\mathbb{P}}{X}_{\mathbb{P}}^{\prime }\circ J+I)\sigma w\hfill \\ {\sigma }_{\Delta }^2\hfill & =w^{\prime }\sigma (X_{\Delta }{X}_{\Delta }^{\prime }\circ J)\sigma w\hfill \\ {\sigma }_{\mathbb{P},\Delta }^2\hfill & =w^{\prime }\sigma (X_{\mathbb{P}}{X}_{\Delta }^{\prime }\circ J)\sigma w\hfill \end{array}\right.\end{array}$$

(34)

For multiple market constraints, one can either calibrate $X_{\mathbb{P}}$ and obtain a unique $\tilde{\alpha }$, or first solve for various $\tilde{\alpha }$ on sub-index levels, and then on the market index level. The above quadratic equation obviously gives two values of $\tilde{\alpha }$, given that ‘$+s$’ can be replaced by ‘±’. Since $\tilde{\alpha }$ is associated to an economic interpretation, the upper value is taken in the case of $CRP>0$ (up-scaling), and the lower value if $CRP<0$ (down-scaling); hence, ‘±’ is substituted by ‘$+s$’ to implement this mechanism. In a multi-factor pricing model, the high multicolinearity among factors could potentially violate $X_{\mathbb{P}}\in \Omega$. This issue can be easily resolved by first computing the correlations between stocks and factors and then orthogonalizing the set of vectors (e.g., via the Gram–Schmidt process).

The procedure of computing the implied correlation matrix from pricing factors can now be summarized as follows. First, the physically expected correlations between stocks and risk factors have to be computed and denoted in a set of vectors as $X_{\mathbb{P}}$. If necessary, $X_{\mathbb{P}}$ has to be orthogonalized. Thereafter, the weighting scalar $\tilde{\alpha }$ has to be calculated according to Equation (34). Lastly, $X_{\mathbb{Q}}$ has to be computed from Equation (33), and the implied correlation matrix then evolves from Equation (2).

4. Empirical Experiment

A brief empirical experiment for the developed models was conducted to evaluate their implementation difficulty and computational efficiency, using data of the S&P 100 and S&P 500 index. The experiment focuses on the monthly at-the-money (ATM) Call option implied volatilities with a target maturity of one month, that were derived from OptionMetrics. It is important to know that OptionMetrics follows a three-dimensional kernel regression for interpolating the option surface (see [49]); a discussion on this topic can be found in [50]. Computations were carried out at the beginning of each month, starting in 1 January 1996 to 2 December 2020, thus giving 300 implied correlation matrices per time-series. The ATM level was chosen due to three reasons: first, options are typically most liquid around the ATM level [51]; second, the ATM level is less sensitive to model misspecification (compared to out-of-money; [52]); third, ATM Call prices are close to ATM Puts. Return (daily) and market value (monthly) data were derived from the CRSP (Center for Research in Security Prices), the lists of index constituents are from Compustat. All computations were executed under an Intel^® Core™ i5-8250U CPU with 1.60 GHz, using the statistical programming software R.

The experiment was executed using the following model specifications. As for the optimization algorithms, the variance tolerance for the market constraint was set at 1 $\times {10}^{-6}$. The stopping criteria was set to a marginal improvement in the objective function of 1 $\times {10}^{-4}$. A modest mark-up of $\left|X\right|\le 1-1\times {10}^{-8}$ was also introduced to ensure that estimates were invertible. The IR-SPGM was implemented as described by [46] (Algorithm 2.1) except that a monotone line search strategy was used. The non-monotone line search strategy runs additional sub-routines of projections to speed up convergence. However, since the projection function here was a potential alternating procedure on its own, the monotone line search was found to be slightly faster than the non-monotone one. Two types of target matrices (A) were used. First, simple Pearson’s correlation matrices from 12-month historically realized returns. Second, a mean-reverting matrix with the entries $\widehat{\rho }(i,j)={\theta }_{ij}\rho (i,j)+(1-{\theta }_{ij})\overline{\rho }(i,j)$, using 9 months of historically realized returns for $\rho (i,j)$ and the mean-correlation between i and j over the total time horizon for $\overline{\rho }(i,j)$. The reversion speed ${\theta }_{ij}$ was randomly drawn from a uniform distribution between 0 and 0.4 to bring in some noise into the target matrix. Generally, the two S&P indices are re-balanced on a quarterly basis, and hence there is no guarantee that the target matrices per se are psd  as firms leave and enter the indices within the estimation horizon. For the non-psd cases, the quantitative approach also served as a repairing tool.

As for the starting value $X^{(0)}$, a modified version of [33] was used. For the target matrix A, let e be the set of eigenvectors and $\iota$ the corresponding eigenvalues. Then, for each column $d=\{1,\dots ,k\}$ of $X^{(0)}$, the starting values were computed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \forall d& =\{1,\dots ,k\}:X_d^{(0)}={\varsigma }_d{e}_d\hfill \\ & \mathrm{where}{\varsigma }_d=min\left\{\sqrt{\frac{({\iota }_d-1){\parallel e_d\parallel }_2^2}{k\parallel e_d{\parallel }_2^4-k{\sum }_{i=1}^n{e}_{d,i}^4}},\frac{1}{\sqrt{k}{max}_i\left|e_{d,i}\right|}\right\}\hfill \end{array}\end{array}$$

(35)

This starting value is identical with the one proposed by [33] in case $k=1$, but differs when the number of factors is higher. Within the empirical experiments of the $k>1$ cases, the reduction in the objective function was found to be larger under the modified than under $X^{(0)}$ of [33]. Hence, the modified starting value was preferred.

The empirical experiment was split into three parts, which are summarized in

Table 2. Summary statistics of computing nearest (quantitative approach, Panels A–B) and risk-factor structured (economic approach, Panel C) implied correlation matrices on monthly option data. As for the quantitative approach, it turns out that computations were carried out in a small amount of time, converging towards the optimal solution within few iterations. The nearest factor-structured matrix can thus be used either as a stand-alone estimate, or as a tool to repair positive semi-definiteness and invertibility. Using the economic approach, implied as well as common risk factors from models, such as CAPM or Fama–French, can be used to estimate the implied correlation matrix.

	k	A	tavg	tsd	fnavg	fnsd	|v.tol|avg	|v.tol|max	iteravg	itersd	Index
Panel A:			hist. matrix
SP100	1	hist.	0.051	0.024	159.9	172.8	5.1 × ${10}^{-11}$	1.5 × ${10}^{-8}$	3.037	1.491	SP100
SP100	3	hist.	0.106	0.050	111.0	168.3	1.3 $\times {10}^{-8}$	6.2$\times {10}^{-7}$	4.819	2.137	SP100
SP100	5	hist.	0.140	0.077	102.7	166.9	1.9$\times {10}^{-8}$	9.9$\times {10}^{-7}$	4.990	1.977	SP100
			mean-reverting matrix
SP100	1	m.r.	0.055	0.027	158.0	158.2	2.0$\times {10}^{-11}$	2.5$\times {10}^{-9}$	3.064	1.438	SP100
SP100	3	m.r.	0.107	0.047	113.3	173.2	3.9$\times {10}^{-8}$	9.6$\times {10}^{-7}$	4.708	1.790	SP100
SP100	5	m.r.	0.149	0.082	104.6	172.5	2.4$\times {10}^{-8}$	8.9$\times {10}^{-7}$	5.054	1.989	SP100
Panel B:
SP500	1	hist.	5.347	2.311	2847.9	4395.0	1.5$\times {10}^{-11}$	1.3$\times {10}^{-10}$	5.003	1.892	SP500
Repaired $C_{\mathbb{Q}}^{AEP1}$	15	${\mathrm{m}.\mathrm{r}.}_{\mathbb{Q}}$	0.223	0.070	16.41	6.775	3.1$\times {10}^{-8}$	9.7$\times {10}^{-7}$	7.184	2.351	SP100
SQP	1	hist.	0.337	0.098	161.4	166.8	3.5$\times {10}^{-7}$	9.6$\times {10}^{-7}$	3.517	0.721	SP100
Panel C:									$\tilde{\alpha }$${}_{\mathrm{avg}}$	$\tilde{\alpha }$${}_{\mathrm{sd}}$
CAPM	1	skew${}_{\mathbb{Q}}$	0.018	0.014	4925.6	4548.0	5.0$\times {10}^{-17}$	4.4$\times {10}^{-16}$	0.099	0.083	SP500
CAPM	1	hist.	0.017	0.018	5108.5	5858.9	3.0$\times {10}^{-17}$	4.7$\times {10}^{-16}$	0.138	0.133	SP500
Fama-Fr.3	3	hist.	0.019	0.018	4376.6	4740.8	3.3$\times {10}^{-17}$	2.5$\times {10}^{-16}$	0.091	0.070	SP500
FF3+Mom.	4	hist.	0.019	0.019	4167.1	4381.3	3.3$\times {10}^{-17}$	3.1$\times {10}^{-16}$	0.082	0.060	SP500
Fama-Fr.5	5	hist.	0.019	0.024	4125.3	4509.1	3.6$\times {10}^{-17}$	3.9$\times {10}^{-16}$	0.076	0.055	SP500

, Panels A–C. The columns of Table 2 describe the number of risk factors (k), the target matrix (A), the mean and standard deviation of computation time (t), the optimized objective function (fn), the absolute realization of the variance tolerance (|v.tol|), the number of outer iterations (iter), and the index data used for the calculation (index). Panels A and B show the results of the quantitative approach, computing the nearest implied correlation matrices. Panel C displays the results of the economic approach.

In Panel A, the nearest implied correlation matrix was computed using IR-SPGM for the S&P 100 under different settings with respect to the number of risk factors and the two different target matrices. Overall, the patterns between the historical- and the mean-reverting matrices look very similar. The average computation speed was very fast and is comparable to the non-equality-constrained results of [33]. The computational speed was probably caused by the fast convergence within few outer iterations—around 3 to 5 on average—and the simplicity of the projection. As it is unlikely that the target matrix perfectly equaled the hidden true implied correlation matrix, the objective function was not expected to reach zero. The results show that on the one hand, an increasing number of risk factors indeed reduced the final objective function, and hence improved the estimation accuracy. On the other hand, non-surprisingly, a larger k came at higher computational effort, as it multiplied the model’s number of variables. With a look on the variance tolerance, the IR-SPGM algorithm had no difficulties to stay inside the feasible region.

Panel B displays the results for the nearest implied correlation matrix method applied to three alternative settings. The first line shows the results on S&P 500 data. Using the S&P 500 increased the number of unknown correlation pairs from 4950 ($n=100$) to 124,750. With the increase in the number of stocks the computation time rose over-proportionally. This was probably caused by the larger number of iterations needed to converge toward the optimum (mean 3.037 vs. 5.003). The optimized objective functions can be compared when dividing by $n^2-n$ (off-diagonal entries), which is 0.016 for the S&P 100 and 0.011 for the S&P 500 index. With a look on the variance tolerance, the S&P 500 computations stuck more strictly to the first market constraint with a maximum deviation of 1.3 $\times {10}^{-10}$. Therefore, while the computation time over-proportionally increased from the S&P 100 to the 500 index, the mean fn per matrix-entry and also the realized variance tolerance were remarkably smaller. Hence, stopping criteria and variance tolerances can potentially be relaxed the more constituents the index holds. The second line of Panel B reports results on the S&P 100, where the target matrix was chosen from mean-reverting correlations, converted into an implied correlation matrix following the adjusted ex post model of [9] (Equation (4)). Hence, the target matrix here already was an implied correlation matrix that fulfilled the market constraint, but did not stick to mathematical feasibility. In this setting, only the monthly estimates of non-psd target matrices were taken such that the quantitative factor approach was used to repair the estimates under the [9]-model. This included 64 of the 300 monthly matrices. The number of risk factors was set higher here to achieve a better fit, and with a look on fn, it can be observed that the objective function was indeed substantially smaller at this application. Hence, the factor model qualifies as a repair tool for existing implied correlation models. Within the third line of Panel B, the IR-SPGM algorithm was replaced by a SQP routine for the S&P 100 data. The algorithm used was taken from the Rsolnp package (see [53,54]). The SQP solver served as a reference to cross-validate whether the IR-SPGM method was implemented correctly. The results can thus directly be compared to the first line of Panel A. It can be observed that the optimized objective functions were very similar between the SQP and IR-SPGM algorithm, which confirms correct implementation of the IR-SPGM routine. Comparing computation times between them, SQP does not seem to be competitive, which was also found in the general case of [33]. This finding thus motivates the usage of IR-SPGM.

In Panel C of Table 2 the economic approach was implemented on the S&P 500 index with respect to implied market exposure [12], CAPM (market factor), the Fama–French three-factor model [27], the extension for the momentum factor (FF3+Mom.) and the Fama–French five-factor model [28]. For the implied market exposure, the risk-neutral skewness was calculated based on [55] using the public code library by Grigory Vilkov (https://doi.org/10.17605/OSF.IO/Z2486, accessed on 23 February 2022). The return data for the remaining factor portfolios were derived directly from K.R. French’s data library (http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html, accessed on 7 April 2021). Following [12], the purely forward-looking correlation between stock i and the market m was computed as $X_{\mathbb{P}}={(ske{w}_i/ske{w}_m)}^{1/3}$. To approximate $X_{\mathbb{P}}$ for the remaining four methods, 12-month realized correlations between stocks and the respective risk factors were used. Note that $X_{\mathbb{P}}$ of the implied market exposure factor solely relied on option data and was thus purely forward looking. $X_{\mathbb{P}}$ of the other factors, however, had a hybrid form that took historical data to assess implied future co-movement. Next, to ensure that $X_{\mathbb{P}}\in \Omega$ for estimations with $k>1$, the factors were orthogonalized via the Gram–Schmidt process. On the one hand, since the economic approach does not iterate and consists only of one projection, its computation was very inexpensive. On the other hand, its estimates deviated more strongly from the target matrix than those of the quantitative approach—irrespective of the number of factors employed. Comparing the results within the economic approach, it can be observed that for hybrid estimations, the larger the number of risk factors, the smaller the fn_avg. The factor solely relying on implied data resulted by far in the highest fn_avg. However, similarity to the target matrix plays an subordinate role here. Similar to that of fn_avg, a reduction pattern can also be seen for $\tilde{\alpha }$ among hybrid models. In Section 3.2, $\tilde{\alpha }$ was defined as the weight of the boundary and $(1-\tilde{\alpha })$ as the weight of $X_{\mathbb{P}}$ inside the risk-neutral factor correlations $X_{\mathbb{Q}}$. This means that the larger the number of risk factors, the less modification was required to match the observed implied market variance in this empirical test. Thus, a larger k is likely to explain the hidden (true) implied correlation matrix better. This time, comparing the purely forward looking measure to the hybrid estimations shows that when implied data were used, less modification was required. Generally, as $\tilde{\alpha }$${}_{\mathrm{avg}}$ was close to zero across all five economic models, it can be concluded that correlation risk premia enter modestly in the factor-structured implied correlation matrix framework.

5. Concluding Remarks

Having an idea about future diversification possibilities requires knowledge about future correlations. Identifying such is a challenging task, as backward-looking information will never capture events that are expected to happen in the future. In contrast, option-implied volatilities are known to carry information about the market outlook. Hence, they are used by academics and practitioners alike to obtain forward-looking perspectives. Computing implied correlation matrices, however, is an intricate puzzle since it is a highly under-determined problem. While there already exists a strain of literature that provides estimation methods for such matrices, this paper shows that these approaches either fail to ensure important mathematical characteristics or discard economic rational grounds. In short, a fully feasible solution to the puzzle has not yet been found.

This paper provides two solutions to the problem by exploiting the commonly accepted assumption that returns stem from factor risk exposure. The first approach is quantitatively motivated and computes the nearest implied correlation matrix subject to a pre-specified target, that can be freely chosen. This method turns out to be a useful tool for repairing (implied) correlation matrix estimates, moreover ensuring their invertibility. It can also be used as a stand-alone estimate, coming at a minimum of assumptions. With the second economically motivated approach, the paper demonstrates how expected risk-factor loadings (or betas) can be translated into an ex ante correlation matrix. As long as one has an educated assumption about either the implied correlation matrix or efficient factor loadings, both approaches provide a fully rational solution to the implied correlation puzzle. Thus, the hypothesis that the shortfalls of existing models can be overcome with factor-based solutions can be accepted. Furthermore, an empirical application of the two approaches on monthly option data of the S&P 100 and S&P 500 (1996–2020) shows that the implementation of the two proposed solutions is easy and computationally efficient.

The findings of this paper have strong implications for practitioners and literature alike. With the current models, investors either risk losing important mathematical properties or have to abandon economic rational grounds. The two provided methodologies are not only able to overcome these shortcomings, but also turn out to be handy regarding practical implementation. Thus, this paper makes an important contribution to the literature, as it solves a long-lasting problem without compromise. With the broad interest in estimating future-oriented co-movements, the presented approaches can find multiple applications in all areas of finance. These include asset pricing, market forecasting, portfolio optimization, and risk management. While this paper focuses more on the theory of implied correlation estimation, future research should evaluate the performance of the presented solutions in an investment context.