When a Correlation Matrix is not a Correlation Matrix: the Nearest Correlation Matrix Problem

9 minute read

Estimating how individual assets are moving together is an important part of many financial applications1 and the most commonly used measure for this is the Pearson correlation.

Unfortunately, for a variety of reasons, what sometimes appears to be a correlation matrix is actually not a valid correlation matrix, which might prevent algorithms using such a matrix in input from providing meaningful results (or from working altogether).

In this blog post, I will describe a possible solution to repair an invalid correlation matrix, which is implemented in Portfolio Optimizer.

But first things first: let’s start by defining what is a valid correlation matrix.

Mathematical definition of a correlation matrix

Let $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be a square matrix.

$C$ is a correlation matrix if and only if

  • $C$ is symmetric: $C {}^t = C$
  • $C$ is unit diagonal: $C_{i,i} = 1$, $i=1..n$
  • $C$ is positive semi-definite: $C \geqslant 0$

Now, while it is possible that a matrix fails to be a correlation matrix because it is numerically non-symmetric (e.g. correlation of assets A and B equal to 0.95 v.s. correlation of assets B and A equal to 0.9500001), or because it is numerically non unit-diagonal (e.g. 1.0000001 instead of 1 on its diagonal), the most common reason is because it is non positive semi-definite, that is, it possesses a least one negative eigenvalue.

Below are a couple of practical situations where a correlation matrix might not be positive semi-definite.

Examples of broken correlation matrices due to loss of positive semi-definiteness

Missing data when computing correlations

When values are missing in series of data2, computing Pearson correlations using the pairwise deletion method for handling missing values might result in a non positive semi-definite correlation matrix.

For example, the following series of data contain eight observations for three variables A, B and C, with missing values

A B C
1 2 2
10 n/a 2
1 5 10
1 3 5
2 5 n/a
4 4 3

The associated correlation matrix, as computed for instance by Google Sheets, is

$\begin{bmatrix} 1 & 0.2647058824 & -0.4905403962 \newline 0.2647058824 & 1 & 0.7980238751 \newline -0.4905403962 & 0.7980238751 & 1 \end{bmatrix}$

and is not a valid correlation matrix because it smallest eigenvalue is $\approx -0.068 < 0$.

Asynchronous data when computing correlations

When values are not observed synchronously in series of data3, computing Pearson correlations without correcting for asynchronicity will systematically result in downward biased correlations4.

Other estimators of correlations, like the Hayashi-Yoshida estimator5, allow to overcome this bias, but the resulting correlation matrix is then not guaranteed to be positive semi-definite.

Altered individual correlations in a valid correlation matrix

Starting from a valid correlation matrix, it is sometimes desired to alter one or several individual correlations6, which might result in the loss of the semi-definite positiveness of the altered correlation matrix.

For example, the following correlation matrix describing the correlations between three assets is a valid correlation matrix

$\begin{bmatrix} 1 & 0.5 & 0.9 \newline 0.5 & 1 & 0.7 \newline 0.9 & 0.7 & 1 \end{bmatrix}$

but if the correlation between assets 2 and 3 is altered from 0.7 to 0.9

$\begin{bmatrix} 1 & 0.5 & 0.9 \newline 0.5 & 1 & 0.9 \newline 0.9 & 0.9 & 1 \end{bmatrix}$

then the resulting matrix is not a valid correlation matrix anymore because its smallest eigenvalue becomes $\approx -0.047 < 0$.

Usage of robust estimators of correlations

Kendall’s tau or Spearman’s rho are examples of robust estimators of correlations7.

While such robust estimators of correlations are less sensitive to outliers than the Pearson correlation, the resulting (Fisher consistent) robust correlation matrix is not guaranteed to be positive semi-definite8.

Limited numerical precision

Because of the limited precision of floating-point arithmetic, round-off errors might be present in the coefficients of a correlation matrix, which might create small negative eigenvalues.

For example, the matrix

$\begin{bmatrix} 1 & 0 & 0 \newline 0 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$

is a valid correlation matrix, but the matrix

$\begin{bmatrix} 1 & 0 & -0.000001 \newline -0.000001 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$

is not due to its smallest eigenvalue being $\approx -5 \times 10^{-13} < 0$.

In other words, a mere difference of $10^{-6}$ in a correlation coefficient might transform a valid correlation matrix into an invalid one!

Fixing a broken correlation matrix

Mathematical formulation of the problem

In all the situations described above, the invalid correlation matrix can be considered as an approximate correlation matrix which must somehow be fixed so that it recovers its positive semi-definiteness.

One possibility to do so is to search for a valid correlation matrix close enough to this approximate correlation matrix, with close enough to be defined.

The mathematical formulation of this problem, known as the nearest correlation matrix problem9, is the following.

Let

  • $A \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be an approximate correlation matrix
  • $S_n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right) \text{ such that } X {}^t = X \}$
  • $U_n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right) \text{ such that } X {}^t = X \text{ and } x_{ii} = 1, i = 1,…,n \}$

The nearest correlation matrix $C$ to the matrix $A$, in the Frobenius norm, is the solution of the optimization problem

\[\displaylines{ C = \operatorname{argmin} \left\Vert X - A \right\Vert_F \text{ s.t. } X \in S_n \cap U_n }\]

With $S_n$ and $U_n$ both being non-empty convex sets with a non-empty intersection, the optimization problem above always has a (unique) solution, so that it is always possible to compute the nearest correlation matrix to an approximate correlation matrix, however badly broken this approximate correlation matrix may be!

Computational solution of the problem

There are two main algorithmic approaches to solve the nearest correlation matrix problem

The Newton method is the best performing method, with quadratic convergence v.s. linear convergence for the alternating projections method, but the alternating projections method is simpler to implement and is more flexible when additional constraints need to be added on the computed correlation matrix.

Implementation in Portfolio Optimizer

Portfolio Optimizer implements a variation of the alternating projections method ensuring that the computed nearest correlation matrix is always strictly definite positive.

For more information, the corresponding Portfolio Optimizer endpoint is /assets/correlation/matrix/nearest.

Examples

Below are a couple of examples of Portfolio Optimizer usage to compute nearest correlation matrices.

Higham’s illustrative correlation matrix

As a first example, let’s take the following approximate correlation matrix, used by Higham9 to illustrate his alternating projections method

$H = \begin{bmatrix} 1 & 1 & 0 \newline 1 & 1 & 1 \newline 0 & 1 & 1 \end{bmatrix}$

The nearest correlation matrix to the matrix $H$ is9

$C_H \approx \begin{bmatrix} 1 & 0.7607 & 0.1573 \newline 0.7607 & 1 & 0.7607 \newline 0.1573 & 0.7607 & 1 \end{bmatrix}$

with $\left\Vert C_H - H \right\Vert_F \approx 0.5278$ and the minimum eigenvalue of $C_H$ equal to 0 (that is, $C_H$ is semi-definite positive).

Using Portfolio Optimizer, the nearest correlation matrix to the matrix $H$ is

{}

Finger’s currencies correlation matrix

As a second example, let’s take the Finger’s correlation matrix11, which represents the correlations between three non-Asian and four Asian currencies

$F = \begin{bmatrix} 1.0 & 0.18 & −0.13 & −0.26 & 0.19 & −0.25 & −0.12 \newline 0.18 & 1.0 & 0.22 & −0.14 & 0.31 & 0.16 & 0.09 \newline −0.13 & 0.22 & 1.0 & 0.060 & −0.080 & 0.040 & 0.04 \newline −0.26 & −0.14 & 0.06 & 1.0 & −0.21 & 0.14 & −0.15 \newline 0.19 & 0.31 & −0.08 & −0.21 & 1.0 & 0.22 & 0.10 \newline −0.25 & 0.16 & 0.04 & 0.14 & 0.22 & 1.0 & 0.07 \newline −0.12 & 0.09 & 0.04 & −0.15 & 0.10 & 0.07 & 1.0 \end{bmatrix}$

where

  • $F(1:3,1:3)$ contains the correlations between the non-Asian currencies
  • $F(4:7,4:7)$ contains the correlations between the Asian currencies

In order to stress-test the behavior of an investment portfolio made of all these currencies, the correlations between the four Asian currencies can be set to a constant correlation of 0.85

$G = \begin{bmatrix} 1.0 & 0.18 & −0.13 & −0.26 & 0.19 & −0.25 & −0.12 \newline 0.18 & 1.0 & 0.22 & −0.14 & 0.31 & 0.16 & 0.09 \newline −0.13 & 0.22 & 1.0 & 0.06 & −0.08 & 0.040 & 0.04 \newline −0.26 & −0.14 & 0.06 & 1.0 & 0.85 & 0.85 & 0.85 \newline 0.19 & 0.31 & −0.08 & 0.85 & 1.0 & 0.85 & 0.85 \newline −0.25 & 0.16 & 0.04 & 0.85 & 0.85 & 1.0 & 0.85 \newline −0.12 & 0.09 & 0.04 & 0.85 & 0.85 & 0.85 & 1.0 \end{bmatrix}$

The altered correlation matrix $G$ is not a valid correlation matrix anymore, because its smallest eigenvalue is $\approx −0.038 < 0$.

Computing the nearest correlation matrix to $G$ gives

$C_G \approx \begin{bmatrix} 1 & 0.1839 & -0.1318 & -0.2514 & 0.1784 & -0.2479 & -0.1191 \newline 0.1839 & 1 & 0.2182 & -0.1315 & 0.2986 & 0.162 & 0.0909 \newline -0.1318 & 0.2182 & 1 & 0.0561 & -0.0747 & 0.039 & 0.0396 \newline -0.2514 & -0.1315 & 0.0561 & 1 & 0.8245 & 0.8546 & 0.8521 \newline 0.1784 & 0.2986 & -0.0747 & 0.8245 & 1 & 0.8438 & 0.8472 \newline -0.2479 & 0.162 & 0.039 & 0.8546 & 0.8438 & 1 & 0.8505 \newline -0.1191 & 0.0909 & 0.0396 & 0.8521 & 0.8472 & 0.8505 & 1 \end{bmatrix}$

The correlations between the Asian currencies in $C_G$ are close to their desired common target of 0.85, so that this is all good, but unfortunately, the correlations between the non-Asian currencies have changed, which is not desirable in the stress-testing scenario envisaged.

This is a situation where the nearest correlation matrix to an approximate correlation matrix needs to be computed while leaving some original correlations unchanged.

Using Portfolio Optimizer, the computed nearest correlation matrix to the matrix $G$ while keeping the correlations in $G(1:3,1:3)$ unchanged is

{}


These examples conclude this blog post.

–

  1. For example, investment portfolios optimization (mean-variance optimization, risk-based optimization…), risk management (value at risk calculation…), financial models calibration, hedging strategies design, etc. ↩

  2. Due to stock markets being closed for bank holidays (which are not the same in all countries), due to a stock quotation being suspended for whatever reason, etc. ↩

  3. The typical example is time series of index returns for US, European and Japanese Equity Indices (S&P 500, STOXX Europe 600, Topix). ↩

  4. Clayton, Michael A., Correlation Estimates from Asynchronously Observed Series, Available at SSRN ↩

  5. Hayashi, T. and Yoshida, N. (2005), On covariance estimation of non-synchronously observed diffusion processes. Bernoulli 11 (2), 359–379 ↩

  6. For example, to analyze the impact of different correlation scenarios (de-correlations…) on the profits and losses of a portfolio (a.k.a. stress testing). ↩

  7. Other ones are described in Shevlyakov, G. and Smirnov, P. (2011) Robust Estimation of the Correlation Coefficient: An Attempt of Survey. Austrian Journal of Statistics, 40, 147-156.. ↩

  8. Zhao, Tuo, et al. Positive Semidefinite Rank-Based Correlation Matrix Estimation With Application to Semiparametric Graph Estimation. Journal of Computational and Graphical Statistics, vol. 23, no. 4, 2014, pp. 895–922. JSTOR ↩

  9. Nicholas J. Higham, Computing the Nearest Correlation Matrix—A Problem from Finance, IMA J. Numer. Anal. 22, 329–343, 2002. ↩ ↩2 ↩3 ↩4

  10. Borsdorf, Rudiger, A Newton Algorithm for the Nearest Correlation Matrix, 2007 ↩

  11. C. Finger. A methodology to stress correlations. RiskMetrics Monitor, pages 3–11, 1997. ↩