When a Correlation Matrix is not a Correlation Matrix: the Nearest Correlation Matrix Problem
Estimating how individual assets are moving together is an important part of many financial applications^{1} and the most commonly used measure for this is the Pearson correlation.
Unfortunately, for a variety of reasons, what sometimes appears to be a correlation matrix is actually not a valid correlation matrix, which might prevent algorithms using such a matrix in input from providing meaningful results (or from working altogether).
In this blog post, I will describe a possible solution to repair an invalid correlation matrix, which is implemented in Portfolio Optimizer.
But first things first: let’s start by defining what is a valid correlation matrix.
Mathematical definition of a correlation matrix
Let $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be a square matrix.
$C$ is a correlation matrix if and only if
 $C$ is symmetric: $C {}^t = C$
 $C$ is unit diagonal: $C_{i,i} = 1$, $i=1..n$
 $C$ is positive semidefinite: $C \geqslant 0$
Now, while it is possible that a matrix fails to be a correlation matrix because it is numerically nonsymmetric (e.g. correlation of assets A and B equal to 0.95 v.s. correlation of assets B and A equal to 0.9500001), or because it is numerically non unitdiagonal (e.g. 1.0000001 instead of 1 on its diagonal), the most common reason is because it is non positive semidefinite, that is, it possesses a least one negative eigenvalue.
Below are a couple of practical situations where a correlation matrix might not be positive semidefinite.
Examples of broken correlation matrices due to loss of positive semidefiniteness
Missing data when computing correlations
When values are missing in series of data^{2}, computing Pearson correlations using the pairwise deletion method for handling missing values might result in a non positive semidefinite correlation matrix.
For example, the following series of data contain eight observations for three variables A, B and C, with missing values
A  B  C 

1  2  2 
10  n/a  2 
1  5  10 
1  3  5 
2  5  n/a 
4  4  3 
The associated correlation matrix, as computed for instance by Google Sheets, is
$\begin{bmatrix} 1 & 0.2647058824 & 0.4905403962 \newline 0.2647058824 & 1 & 0.7980238751 \newline 0.4905403962 & 0.7980238751 & 1 \end{bmatrix}$
and is not a valid correlation matrix because it smallest eigenvalue is $\approx 0.068 < 0$.
Asynchronous data when computing correlations
When values are not observed synchronously in series of data^{3}, computing Pearson correlations without correcting for asynchronicity will systematically result in downward biased correlations^{4}.
Other estimators of correlations, like the HayashiYoshida estimator^{5}, allow to overcome this bias, but the resulting correlation matrix is then not guaranteed to be positive semidefinite.
Altered individual correlations in a valid correlation matrix
Starting from a valid correlation matrix, it is sometimes desired to alter one or several individual correlations^{6}, which might result in the loss of the semidefinite positiveness of the altered correlation matrix.
For example, the following correlation matrix describing the correlations between three assets is a valid correlation matrix
$\begin{bmatrix} 1 & 0.5 & 0.9 \newline 0.5 & 1 & 0.7 \newline 0.9 & 0.7 & 1 \end{bmatrix}$
but if the correlation between assets 2 and 3 is altered from 0.7 to 0.9
$\begin{bmatrix} 1 & 0.5 & 0.9 \newline 0.5 & 1 & 0.9 \newline 0.9 & 0.9 & 1 \end{bmatrix}$
then the resulting matrix is not a valid correlation matrix anymore because its smallest eigenvalue becomes $\approx 0.047 < 0$.
Usage of robust estimators of correlations
Kendall’s tau or Spearman’s rho are examples of robust estimators of correlations^{7}.
While such robust estimators of correlations are less sensitive to outliers than the Pearson correlation, the resulting (Fisher consistent) robust correlation matrix is not guaranteed to be positive semidefinite^{8}.
Limited numerical precision
Because of the limited precision of floatingpoint arithmetic, roundoff errors might be present in the coefficients of a correlation matrix, which might create small negative eigenvalues.
For example, the matrix
$\begin{bmatrix} 1 & 0 & 0 \newline 0 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$
is a valid correlation matrix, but the matrix
$\begin{bmatrix} 1 & 0 & 0.000001 \newline 0.000001 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$
is not due to its smallest eigenvalue being $\approx 5 \times 10^{13} < 0$.
In other words, a mere difference of $10^{6}$ in a correlation coefficient might transform a valid correlation matrix into an invalid one!
Fixing a broken correlation matrix
Mathematical formulation of the problem
In all the situations described above, the invalid correlation matrix can be considered as an approximate correlation matrix which must somehow be fixed so that it recovers its positive semidefiniteness.
One possibility to do so is to search for a valid correlation matrix close enough to this approximate correlation matrix, with close enough to be defined.
The mathematical formulation of this problem, known as the nearest correlation matrix problem^{9}, is the following.
Let
 $A \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be an approximate correlation matrix
 $S_n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right) \text{ such that } X {}^t = X \}$
 $U_n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right) \text{ such that } X {}^t = X \text{ and } x_{ii} = 1, i = 1,…,n \}$
The nearest correlation matrix $C$ to the matrix $A$, in the Frobenius norm, is the solution of the optimization problem
\[\displaylines{ C = \operatorname{argmin} \left\Vert X  A \right\Vert_F \text{ s.t. } X \in S_n \cap U_n }\]With $S_n$ and $U_n$ both being nonempty convex sets with a nonempty intersection, the optimization problem above always has a (unique) solution, so that it is always possible to compute the nearest correlation matrix to an approximate correlation matrix, however badly broken this approximate correlation matrix may be!
Computational solution of the problem
There are two main algorithmic approaches to solve the nearest correlation matrix problem
 Using an alternating projections method with Dykstra’s correction^{9}
 Using a Newton method^{10}
The Newton method is the best performing method, with quadratic convergence v.s. linear convergence for the alternating projections method, but the alternating projections method is simpler to implement and is more flexible when additional constraints need to be added on the computed correlation matrix.
Portfolio Optimizer implements a variation of the alternating projections method ensuring that the computed nearest correlation matrix is always strictly definite positive.
For more information, the corresponding Portfolio Optimizer API endpoint is /assets/correlation/matrix/nearest
.
Below are a couple of examples of Portfolio Optimizer usage to compute nearest correlation matrices.
Examples
Higham’s illustrative correlation matrix
As a first example, let’s take the following approximate correlation matrix, used by Higham^{9} to illustrate his alternating projections method
$H = \begin{bmatrix} 1 & 1 & 0 \newline 1 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$
The nearest correlation matrix to the matrix $H$ is^{9}
$C_H \approx \begin{bmatrix} 1 & 0.7607 & 0.1573 \newline 0.7607 & 1 & 0.7607 \newline 0.1573 & 0.7607 & 1 \end{bmatrix}$
with $\left\Vert C_H  H \right\Vert_F \approx 0.5278$ and the minimum eigenvalue of $C_H$ equal to 0 (that is, $C_H$ is semidefinite positive).
Using Portfolio Optimizer, the nearest correlation matrix to the matrix $H$ is
{}
$C_{PO} \approx \begin{bmatrix} 1 & 0.7606 & 0.1573 \newline 0.7606 & 1 & 0.7606 \newline 0.1573 & 0.7606 & 1 \end{bmatrix}$
with $\left\Vert C_{PO}  H \right\Vert_F \approx 0.5279$ and the minimum eigenvalue of $C_{PO}$ equal to $10^{4}$ (that is, $C_{PO}$ is strictly definite positive).
For all practical purposes, the two matrices $C_H$ and $C_{PO}$ are identical, but because $C_{PO}$ is strictly definite positive, it is probably to be preferred as input to numerical algorithms requiring a valid correlation matrix.
Finger’s currencies correlation matrix
As a second example, let’s take the Finger’s correlation matrix^{11}, which represents the correlations between three nonAsian and four Asian currencies
$F = \begin{bmatrix} 1.0 & 0.18 & −0.13 & −0.26 & 0.19 & −0.25 & −0.12 \newline 0.18 & 1.0 & 0.22 & −0.14 & 0.31 & 0.16 & 0.09 \newline −0.13 & 0.22 & 1.0 & 0.060 & −0.080 & 0.040 & 0.04 \newline −0.26 & −0.14 & 0.06 & 1.0 & −0.21 & 0.14 & −0.15 \newline 0.19 & 0.31 & −0.08 & −0.21 & 1.0 & 0.22 & 0.10 \newline −0.25 & 0.16 & 0.04 & 0.14 & 0.22 & 1.0 & 0.07 \newline −0.12 & 0.09 & 0.04 & −0.15 & 0.10 & 0.07 & 1.0 \end{bmatrix}$
where
 $F(1:3,1:3)$ contains the correlations between the nonAsian currencies
 $F(4:7,4:7)$ contains the correlations between the Asian currencies
In order to stresstest the behavior of an investment portfolio made of all these currencies, the correlations between the four Asian currencies can be set to a constant correlation of 0.85
$G = \begin{bmatrix} 1.0 & 0.18 & −0.13 & −0.26 & 0.19 & −0.25 & −0.12 \newline 0.18 & 1.0 & 0.22 & −0.14 & 0.31 & 0.16 & 0.09 \newline −0.13 & 0.22 & 1.0 & 0.06 & −0.08 & 0.040 & 0.04 \newline −0.26 & −0.14 & 0.06 & 1.0 & 0.85 & 0.85 & 0.85 \newline 0.19 & 0.31 & −0.08 & 0.85 & 1.0 & 0.85 & 0.85 \newline −0.25 & 0.16 & 0.04 & 0.85 & 0.85 & 1.0 & 0.85 \newline −0.12 & 0.09 & 0.04 & 0.85 & 0.85 & 0.85 & 1.0 \end{bmatrix}$
The altered correlation matrix $G$ is not a valid correlation matrix anymore, because its smallest eigenvalue is $\approx −0.038 < 0$.
Computing the nearest correlation matrix to $G$ gives
$C_G \approx \begin{bmatrix} 1 & 0.1839 & 0.1318 & 0.2514 & 0.1784 & 0.2479 & 0.1191 \newline 0.1839 & 1 & 0.2182 & 0.1315 & 0.2986 & 0.162 & 0.0909 \newline 0.1318 & 0.2182 & 1 & 0.0561 & 0.0747 & 0.039 & 0.0396 \newline 0.2514 & 0.1315 & 0.0561 & 1 & 0.8245 & 0.8546 & 0.8521 \newline 0.1784 & 0.2986 & 0.0747 & 0.8245 & 1 & 0.8438 & 0.8472 \newline 0.2479 & 0.162 & 0.039 & 0.8546 & 0.8438 & 1 & 0.8505 \newline 0.1191 & 0.0909 & 0.0396 & 0.8521 & 0.8472 & 0.8505 & 1 \end{bmatrix}$
The correlations between the Asian currencies in $C_G$ are close to their desired common target of 0.85, so that this is all good, but unfortunately, the correlations between the nonAsian currencies have changed, which is not desirable in the stresstesting scenario envisaged.
This is a situation where the nearest correlation matrix to an approximate correlation matrix needs to be computed while leaving some original correlations unchanged.
Using Portfolio Optimizer, the computed nearest correlation matrix to the matrix $G$ while keeping the correlations in $G(1:3,1:3)$ unchanged is
{}
$C_{G}^{'} \approx \begin{bmatrix} 1 & 0.18 & 0.13 & 0.2512 & 0.1781 & 0.2479 & 0.119 \newline 0.18 & 1 & 0.22 & 0.1314 & 0.2983 & 0.1621 & 0.0909 \newline 0.13 & 0.22 & 1 & 0.056 & 0.0745 & 0.039 & 0.0396 \newline 0.2512 & 0.1314 & 0.056 & 1 & 0.8241 & 0.8546 & 0.8521 \newline 0.1781 & 0.2983 & 0.0745 & 0.8241 & 1 & 0.8437 & 0.8472 \newline 0.2479 & 0.1621 & 0.039 & 0.8546 & 0.8437 & 1 & 0.8505 \newline 0.119 & 0.0909 & 0.0396 & 0.8521 & 0.8472 & 0.8505 & 1 \end{bmatrix}$
The correlations between the Asian currencies in $C_{G}^{'}$ are again close to the desired common target of 0.85, but this time the correlations between the nonAsian currencies have not changed!
These examples conclude this blog post.
–

For example, investment portfolios optimization (meanvariance optimization, riskbased optimization…), risk management (value at risk calculation…), financial models calibration, hedging strategies design, etc. ↩

Due to stock markets being closed for bank holidays (which are not the same in all countries), due to a stock quotation being suspended for whatever reason, etc. ↩

The typical example is time series of index returns for US, European and Japanese Equity Indices (S&P 500, STOXX Europe 600, Topix). ↩

Clayton, Michael A., Correlation Estimates from Asynchronously Observed Series, Available at SSRN ↩

Hayashi, T. and Yoshida, N. (2005), On covariance estimation of nonsynchronously observed diffusion processes. Bernoulli 11 (2), 359–379 ↩

For example, to analyze the impact of different correlation scenarios (decorrelations…) on the profits and losses of a portfolio (a.k.a. stress testing). ↩

Other ones are described in Shevlyakov, G. and Smirnov, P. (2011) Robust Estimation of the Correlation Coefficient: An Attempt of Survey. Austrian Journal of Statistics, 40, 147156.. ↩

Zhao, Tuo, et al. Positive Semidefinite RankBased Correlation Matrix Estimation With Application to Semiparametric Graph Estimation. Journal of Computational and Graphical Statistics, vol. 23, no. 4, 2014, pp. 895–922. JSTOR ↩

Nicholas J. Higham, Computing the Nearest Correlation Matrix—A Problem from Finance, IMA J. Numer. Anal. 22, 329–343, 2002. ↩ ↩^{2} ↩^{3} ↩^{4}

Borsdorf, Rudiger, A Newton Algorithm for the Nearest Correlation Matrix, 2007 ↩

C. Finger. A methodology to stress correlations. RiskMetrics Monitor, pages 3–11, 1997. ↩