Portfolio Optimizer

The Market Rank Indicator: Measuring Financial Risk, Part 3

2026-03-04T00:00:00-06:00

In the previous post of this series on measuring financial risk, I described the absorption ratio, a measure of financial market fragility based on principal components analysis, introduced in Kritzman et al.¹.

In this new blog post, I will describe another measure of financial distress called the market rank indicator (MRI), this time related to the notion of condition number² of a matrix, introduced in Figini et al.².

As an example of usage, I will show how to use the market rank indicator to dynamically scale the market exposure of a portfolio of U.S. equities.

The market rank indicator

Definition

Let be:

$n$, the number of assets
$T$, the number of time periods, with $T > n$
$X \in \mathbb{R}^{T \times n}$, the matrix of the asset arithmetic or logarithmic returns for each of the $T$ time periods
$\sigma_1,…,\sigma_n$, the singular values of $X$ ordered such that $0 < \sigma_1 \leq … \leq \sigma_n$
$1\leq k \leq n$, the number of singular values $\lambda_1,…,\lambda_k$ to retain in the computation of the market rank indicator

The market rank indicator $\text{MRI}$ of the assets is defined² as the ratio between the largest singular value and the geometric mean of the $k$ smallest singular values of the matrix $X$², that is

\[\text{MRI}_k = \frac{ \sigma_n }{ \left( \prod_{i=1}^k \sigma_i \right)^{1/k} }\]

Alternative definition

Let be:

$n$, the number of assets
$T$, the number of time periods, with $T > n$
$X \in \mathbb{R}^{T \times n}$, the matrix of the asset arithmetic or logarithmic returns for each of the $T$ time periods
$\Sigma = \frac{1}{T} X {}^t X \in \mathcal{M}(\mathbb{R}^{n \times n})$, the asset returns covariance matrix
$\lambda_1,…,\lambda_n$, the eigenvalues of $\Sigma$ ordered such that $0 < \lambda_1 \leq … \leq \lambda_n$

Thanks to the relationship $\lambda_i = \frac{\sigma_i^2}{T}$, $i=1..n$, between the eigenvalues of $\Sigma$ and the singular values of $X$, the market rank indicator can alternatively be defined as the square root of the ratio between the largest eigenvalue and the geometric mean of the $k$ smallest eigenvalues of the matrix $\Sigma$, that is

\[\text{MRI}_k = \sqrt{ \frac{ \lambda_n }{ \left( \prod_{i=1}^k \lambda_i \right)^{1/k} } }\]

Generalized definition

In the two previous sub-sections, it is assumed that $T > n$.

This is to guarantee that the singular values of the matrix $X$ or that the eigenvalues of the matrix $\Sigma$ are not null.

In case $T \leq n$, the definition of the market rank indicator needs to be adapted, which is simply done by redefining [it] on the first $T$ [non zero] singular values².

Rationale

Figini et al.² highlights that what contains valuable information on market synchronization in a principal components analysis on asset returns is not only the strength of the first principal component, but also the weakness of the last components².

This observation leads to the proposal of the market rank indicator as a generalization of the condition number $\kappa(X)$ of the matrix $X$², with $\text{MRI}_1 = \kappa(X)$ as a special case.

Interpretation

Figini et al.² proposes both a mathematical and a financial interpretation of the market rank indicator:

Mathematically, the market rank indicator measures the difficulty to span $\mathbb{R}^n$ with the $n$ columns of $X$².

Indeed, for a given $1\leq k \leq n$, a large value of $\text{MRI}_k$ indicates that […] the columns of $X$ are a basis for an $(n − k)$-dimensional space².
Financially, the market rank indicator may be interpreted as a measure of distance of the asset return series from a market where the number of independent assets is reduced².

Indeed, again for a given $1\leq k \leq n$, a large value of $\text{MRI}_k$ indicates that $k$ dimensions/assets out of $n$ are not very useful in diversifying a portfolio².

Theoretical and empirical properties

Figini et al.² establishes several theoretical properties of the market rank indicator, among which:

Property 1: For any $1 \leq k \leq n$, $ 1 \leq \text{MRI}_k \leq \text{MRI}_1 $.

Property 2: For a given matrix $X$, $\text{MRI}_k$ is non-increasing in $1 \leq k \leq n$.

In addition, Figini et al.² empirically demonstrates that the market rank indicator is a suitable early warning system for future turbulence on the market² by showing³ that a large value of [that indicator] signals that the market will experience worse performances in the near future and that the probability of a large and negative return increases².

How to determine the number of singular values to retain?

When computing the market rank indicator, Figini et al.² suggests to use a number of singular values of about 1/3th the number of assets²:

The value for $k$ is set after a careful sensitivity analysis: an extensive comparison of the results obtained under different parameter settings showed that the best choice for $k$ is around one third of $n$.

To be noted, though, that depending on the specific context at hand, the number of singular values to retain could also be chosen using the numerical optimization method, assuming for example $k$ stochastic².

Comparison with other measures of financial risk

Comparison with the absorption ratio

Figini et al.² notes that critical market conditions² are characterized by two connected but not perfectly equivalent² phenomena:

An increase in the weight of the first principal components²
A reduction in the weight of the last components²

The absorption ration focuses on the quantity of variability explained by the largest components², that is, it focuses on the first phenomenon.

The market rank indicator, on the other hand, focuses on the second phenomenon², which theoretically makes it a perfect complement to the absorption ratio as a measure of financial risk.

Comparison with the covariance matrix effective rank

Familiar readers might remember that in a previous blog post, I discussed the effective rank⁴ of a covariance matrix, which is a real-valued extension of its rank allowing to measure the dimensionality of the associated universe of assets.

It turns out - as its name suggests - that the market rank indicator is very similar to the effective rank.

Indeed, when there is an increase in the co-movement of the asset returns², the vectors of the return time series become closer², which leads to a reduction in the diversification opportunities or, from a numerical point of view, a reduction in the market “dimension”² which is also captured by the effective rank.

As an illustration, Figure 1 compares both indicators when applied to the 11 Vanguard U.S. sector ETFs⁵ over the period 31th October 2005 - 30th January 2026, using the same methodology as described in the Exemple of usage section.

Figure 1. Market rank indicator v.s. effective rank, 11 Vanguard U.S. sector ETFs, 31th October 2005 - 30th January 2026.

On Figure 1, it is clearly visible that the market rank indicator and the effective rank move in a near perfect opposite direction to each other, which is confirmed numerically by a -84% correlation between these two indicators.

Still, the inverse relationship between the market rank indicator and the effective rank is not perfect⁶, so that they might be considered as two similar-but-different indicators.

Horse race for Value-at-Risk forecasting

Figini et al.² compares the empirical performances of the market rank indicator to those of three other measures of financial risk (the absorption ratio, the turbulence index and the average correlation) when forecasting the level of the 20-day ahead Value-at-Risk, and shows that it is the indicator which displays the best performances².

In more details, Figini et al.²:

First notes that in general, a good systemic risk indicator would allow to well discriminate between the regular behavior of the market and periods of distress² because intuitively, the VaR (or other risk indicators) should be low when the indicator is low and large when the indicator is large².
Then proceeds to empirically test this assumption in the case of the S&P 500 index, the STOXX Europe 600 index and the DAX index and concludes that:

Although each indicator presents a degree of forecasting power, the MRI is the indicator which displays the best performances for the three markets.

In fact, [as can be seen in Figure 2] not only the VaR associated to the highest percentile interval is the largest, but also the difference between the VaR associated to the highest interval and the one associated to the lowest interval present the largest outcomes as well […].

This shows that the MRI can best discriminate between regular and turbulent periods, with respect to other systemic risk indicators.

Figure 2. Future 20-day S&P 500 Value-at-Risk at 5% level v.s. quintiles of various systemic risk indicators, 02nd January 1992 - 01st July 2011. Source: Figini et al.

Implementation in Portfolio Optimizer

Portfolio Optimizer implements the computation of the market rank indicator through the endpoint /assets/analysis/market-rank-indicator, with the default number of singular values to retain suggested in Figini et al.².

Example of usage - Scaling the market exposure of a portfolio of U.S. equities

I will now illustrate the performances of the market rank indicator in a trading strategy desgined to capture the idea that exposure to stocks should be decreased when the market rank indicator is increasing.

For this:

I will work within the universe of U.S. equities represented by the SPY ETF and the 11 Vanguard U.S. sector ETFs⁵.
I will rely on daily return data⁷ over the period 31th October 2005 - 30th January 2026.
I will use the following trading strategy at the end of each month:
- Compute the sample covariance matrix of the 11 Vanguard U.S. sector ETFs over the past month, using the ETF daily arithmetic returns.
- Compute the market rank indicator associated to that covariance matrix, retaining 3 eigenvalues.
- Determine how elevated is this market rank indicator relative to its past 12-month history on a scale $s$ from 0% to 100%, thanks to a percentile rank.
- If $s$ is greater than 75% (i.e., relatively elevated recent market rank indicator), allocate the portfolio to cash (with 0% interest) else allocate the portfolio to U.S. equities (SPY ETF); in both cases, hold that portfolio for the next month.

The equity curve associated to that market rank indicator-based trading stategy is depicted in Figure 3.

Figure 3. MRI-based trading strategy v.s. buy and hold, SPY ETF, 31th October 2005 - 30th January 2026.

From Figure 3, the market rank indicator-based trading stategy does a good job in avoiding some serious drawdowns when compared to the buy and hold strategy, but unfortunately lags in terms of total return.

Figures:

Portfolio Management Strategy	Average Exposure	CAGR	Annualized Sharpe Ratio	Maximum (Monthly) Drawdown
Buy and hold	100%	10.9%	0.78	51%
MRI-based	68%	8.2%	0.79	33%

For comparison, Figure 4 additionally depicts the equity curve associated to a similar strategy, this time based on the absorption ratio of the 11 Vanguard U.S. sector ETFs⁸.

Figure 4. MRI-based and AR-based trading strategies v.s. buy and hold, SPY ETF, 31th October 2005 - 30th January 2026.

With figures:

Portfolio Management Strategy	Average Exposure	CAGR	Annualized Sharpe Ratio	Maximum (Monthly) Drawdown
Buy and hold	100%	10.9%	0.78	51%
MRI-based	68%	8.2%	0.79	33%
AR-based	69%	10.6%	1.03	21%

From Figure 4, it seems that that the absorption ratio is superior⁹ to the market rank indicator in terms of predictive performances for the specific universe of assets and over the specific period considered here.

Incidentally, that result is at odds with the findings of Figini et al.², so that the interested reader might want to dig deeper…

Conclusion

Together with the turbulence index and the absorption ratio¹⁰, the market rank indicator is a simple early warning indicator² for dangerous periods on financial markets.

Although it did not particularly shine v.s. the absorption ratio in the specific example studied in this blog post, the market rank indicator is definitely an additional tool to integrate² so as to improve [one’s] warning system².

Waiting for the analysis of the other measures of financial risk, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Mark Kritzman, Yuanzhen Li, Sebastien Page and Roberto Rigobon, Principal Components as a Measure of Systemic Risk, The Journal of Portfolio Management Summer 2011, 37 (4) 112-126. ↩
See Silvia Figini, Mario Maggi, Pierpaolo Uberti, The market rank indicator to detect financial distress, Econometrics and Statistics, Volume 14, 2020, Pages 63-73. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³ ↩³⁴ ↩³⁵ ↩³⁶ ↩³⁷ ↩³⁸ ↩³⁹ ↩⁴⁰
Using returns of the 10 sectors composing the S&P 500 index. ↩
See Olivier Roy and Martin Vetterli, The effective rank: A measure of effective dimensionality, 15th European Signal Processing Conference, 2007. ↩
These are “VOX”, “VCR”, “VDC”, “VDE”, “VFH”, “VHT”, “VIS”, “VGT”, “VAW”, “VNQ”, “VPU”, c.f. https://investor.vanguard.com/investment-products/etfs/sector-etfs. ↩ ↩²
Most probably due to the fact that the number of eigenvalues used by each indicator is different. ↩
(Adjusted) prices of the ETFs have have been retrieved using Tiingo. ↩
The absorption ratio-based trading strategy is exactly the same as the market rank indicator-based trading strategy, except that the absorption ratio is computed instead of the market rank indicator; that’s the only change. ↩
The two indicators are agreeing most of the time - their correlation is 85%; as a side note, it could be interesting to understand what happens when they are disagreeing… ↩
As well as the effective rank; I cannot insist too much on that one, though, because - to this date - I did not analyze its usefulness as a measure of financial risk on the blog. ↩

More Bootstrap Simulations with Portfolio Optimizer: the Autoregressive Online Bootstrap

2026-02-01T00:00:00-06:00

In a previous article, I described several classical bootstrap techniques — i.i.d. bootstrap, circular block bootstrap, and stationary block bootstrap — and showed how the stationary block bootstrap could be used to simulate future price paths for financial assets by following the methodology of Anarkulova et al.¹.

In this blog post, I will detail another bootstrap technique called the autoregressive online bootstrap² and introduced in Palm and Nagler², that is best described as a multiplier bootstrap³ coupled with an autoregressive sequence of weights specifically chosen to make it useable with streaming time series data.

As an example of usage, I will simulate alternative price histories for the SPY and TLT ETFs, which are ETFs representative of the US stock market and of the long-term US Treasury bonds.

Mathematical preliminaries

Let $X_1, …, X_n$⁴, $n \geq 1$ be a sample of data observed from a population.

Limitations of classical bootstrap techniques in an online setting

In a typical⁵ online setting, the length $n$ of the sample of data $X_1, …, X_n$ is growing infinitely, with new events [that] are observed at the moment they occur².

In such a setting, the classical bootstrap techniques described in the previous blog post on bootstrap simulations require all data observed so far² at any given point in time.

Indeed:

The i.i.d. bootstrap, by definition, requires keeping track of the entire observed sample $ {X_1, …, X_n} $².
The circular and the stationary block bootstrap require all blocks […] to increase in size with $n$², so that to compute the bootstrap in practice, the entire data set $ {X_1, …, X_n} $ needs to be kept in memory and processed fully, every time the block size changes².

Depending on the domain of application, this can be a serious computational limitation when:

n is large
n is moderately large but the underlying $n$ observations require a lot of computer memory to be stored
n is moderately large but the computation time is limited (e.g. real-time applications)

The multiplier bootstrap

The multiplier bootstrap³ is a general class of bootstrapping schemes based on perturbations of the original observations with suitable weights².

In other words, compared to classical bootstrap techniques, the multiplier bootstrap replaces the idea of “randomly resampling observations” with the idea of “randomly reweighting observations”, which enables it to be applied in an online setting.

For i.i.d. data $X_1, …, X_n$, it is defined as follows:

Let $V_1, …, V_n$ be $n$ i.i.d. random variables of unit mean/variance.
Let $ \bar V_i = \frac{1}{i} \sum_{j=1}^{i} V_j $ the running mean of these random variables, which can be computed recursively through the formula
\[\bar V_i = \frac{(i-1) \bar V_{i-1} + V_i }{i}\]
The multiplier bootstrap samples $ X_1^*,…,X_n^* $ are then defined as
\[X_i^* = \frac{V_i}{\bar V_i} X_i\]

The autoregressive online bootstrap

Methodology

The autoregressive online bootstrap² is a specific instance of the multiplier bootstrap that generates a sequence of random weights evolving according to an autoregressive process centered around 1:

\[V_i = 1 + \rho_i \left( V_{i-1} - 1 \right) + \sqrt{1 - \rho_i^2} \zeta_i, i=1..n\]

, with:

$V_0 = 0$
$\rho_i = 1 - i^{-\beta}$, $ 0 < \beta < \frac{1}{2}$
$\zeta_i \sim \mathcal N(0,1), i=1..n$

The associated autoregressive online bootstrap samples $ X_1^*,…,X_n^* $ are then defined as

\[X_i^* = \frac{V_i}{\bar V_i} X_i\]

Rationale

Palm and Nagler² notes that for the multiplier bootstrap to remain valid for time series:

The dependencies between weights $V_i$ and $V_j$ must increase with the sample size $n$, but at the same time remain almost independent when the time gap $|i-j|$ is sufficiently large compared to $n$².
A scaling of the weights by their arithmetic mean is also necessary²

This is what they propose with the autoregressive online bootstrap technique.

Properties

The main result of Palm and Nagler² is that under mild conditions on the original data $X_1, …, X_n$, the autoregressive online bootstrap is a consistent resampling scheme for the mean and for any continously differentiable transformation of the mean⁶ of an univariate or a multivariate time series.

How to select the parameter $\beta$?

The parameter $\beta$ controls the behaviour of the autoregressive online bootstrap samples in a way similar to how the block length controls the behaviour of the block bootstrap samples in classical block bootstrap techniques.

Intuitively:

A small value of $\beta$ leads to slowly changing weights and thus to long stretches of bootstrap samples that are consistent with the original observations, although up- or down-weighted.

This is similar in spirit to having a long block size in a block bootstrap technique.
On the contrary, a large value of $\beta$ leads to quickly changing weights and thus to rapidly varying bootstrap samples.

This is similar in spirit to having a short block size in a block bootstrap technique.

Palm and Nagler² demonstrates that $\beta_{opt} = \sqrt{2} - 1$ allows for an optimal bias-variance trade-off², so that unless there is a specific need to play with this parameter, using that value should be the default choice.

Practical performances

Palm and Nagler² compares the practical performances of the autoregressive online bootstrap to an i.i.d. multiplier bootstrap and to a block bootstrap technique called the moving average block bootstrap⁷.

In particular, Palm and Nagler² shows that:

The two time series bootstraps achieve approximately correct coverage in all [studied] scenarios, even in the presence of nonlinear [AR(2)-GARCH(1,1)] dependencies² corresponding to a stochastic volatility process.
The autoregressive online bootstrap allows for cheap online updates - in constant time, as illustrated in Figure 1 - contrary to the moving average block bootstrap.

Figure 1. Computation time per online update of 200 bootstrap samples as the bootstrap techniques progress through a stream of 2000 samples. Source: Palm and Nagler.
The autoregressive online bootstrap has a slightly higher variance compared to the moving average block bootstrap, which is the cost [to] pay for its computational advantage².

Caveats

One important limitation of the autoregressive online bootstrap is that the weights $V_i$,$i=1..n$ can be negative.

For this reason, in finance, logarithmic asset returns should then be prefered to either asset prices (since they could become negative) or arithmetic asset returns (singe they could become < -100%).

Implementations

Implementation in Portfolio Optimizer

Portfolio Optimizer implements the autoregressive online bootstrap through the endpoint /assets/returns/simulation/bootstrap/online.

Implementation elsewhere

An implementation of the autoregressive online bootstrap in Python is available at https://github.com/nicolaipalm/online-bootstrap-implementation by its authors.

Example of usage - Simulation of alternative price histories for ETFs

As an example of usage, I propose to use the autoregressive online bootstrap to generate alternative price histories for the SPY and TLT ETFs and compute a couple of associated descriptive statistics⁸.

Alternative price histories for the SPY ETF

Figure 2 illustrates 10 synthetic price histories generated by applying the autoregressive online bootstrap to the logarithmic returns of the SPY ETF over the period 1st January 2025 - 31th December 2025⁹.

Figure 2. Alternative price histories for the SPY ETF, autoregressive online bootstrap, 1st January 2025 - 31th December 2025.

On Figure 2, it is visible that the autoregressive online bootstrap allows for a wide variety of scenarios to be generated.

This is confirmed in Figure 3, which depicts the distributions of the first four moments of the logarithmic returns of 1000 synthetic price histories generated by applying the autoregressive online bootstrap to the logarithmic returns of the SPY ETF over the period 1st January 2025 - 31th December 2025⁹.

Figure 3. Distributions of the first four moments of 1000 alternative price histories for the SPY ETF, autoregressive online bootstrap, 1st January 2025 - 31th December 2025.

Figure 3 empirically demonstrates two points:

It confirms the mean-preservation property of the autoregressive online bootstrap.

As a side note, in case this is a problem for the usage at hand, it is always possible to alter the mean of the generated scenarios in a perfectly controlled way, c.f. a previous blog post.
It shows that the autoregressive online bootstrap can generate scenarios with a wildly different standard deviation/skewness/kurtosis compared to the original one.

Here, to be noted that some scenarios might seem implausible, with a kurtosis > 100 for example.

Nevertheless, implausible does not equal impossible¹⁰, so that depending on the usage at hand, such scenarios could be eliminated or put aside for specific stress-testing.

Figure 4 illustrates the variety of scenarios in another way, by looking at the correlation between the logarithmic returns of the SPY ETF and of its 1000 alternatives, which approximatively ranges from -0.4 to 1.

Figure 4. Distribution of the correlation between the SPY ETF price history and 1000 alternative price histories, autoregressive online bootstrap, 1st January 2025 - 31th December 2025.

Alternative price histories for the SPY and TLT ETFs

Figure 5 illustrates the behaviour of the autoregressive online boostrap in terms of bivariate correlation when applied to the logarithmic returns of both the SPY and TLT ETFs over the period 1st January 2025 - 31th December 2025⁹.

Figure 5. Distribution of the correlation between the two components of 1000 alternative price histories for the SPY and TLT ETFs, autoregressive online bootstrap, 1st January 2025 - 31th December 2025.

From Figure 5, it appears that the whole range of admissible correlations $[-1,1]$ is achievable through the autoregressive online bootstrap, with the majority of scenarios falling in the interval $[-0.25, 0.50]$.

Conclusion

The autoregressive online bootstrap introduced in Palm and Nagler² and detailled in this blog post provides an alternative to classical block bootstrap techniques for time series that is specifically taylored to streaming data.

Don’t hesitate to experiment with it and evaluate whether it could replace your current boostrap methodology!

As usual, feel also free to connect with me on LinkedIn or to follow me on Twitter.

–

See Anarkulova, Aizhan and Cederburg, Scott and O’Doherty, Michael S., The Long-Horizon Returns of Stocks, Bonds, and Bills: Evidence from a Broad Sample of Developed Markets (November 15, 2021). ↩
See Nicolai Palm, Thomas Nagler; An Online Bootstrap for Time Series; Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:190-198. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰
See Van Der Vaart, A. W. and Wellner, J. A. Weak convergence. In Weak convergence and empirical processes, pp. 16–28. Springer, 1996. ↩ ↩²
The observations $X_1, …, X_n$ can be observations of random variables or of random vectors. ↩
It is also possible that for computational reasons - like a huge $n$ - a complete data set is available from the start², but is processed sequentially or in batches². ↩
Like the variance, c.f. Palm and Nagler². ↩
See Buhlmann, P. L. (1993). The blockwise bootstrap in time series and empirical processes. PhD thesis, ETH Zurich. ↩
In practice, the generation of such alternative price histories would typically be integrated into a backtesting engine. ↩
(Adjusted) prices of the SPY and TLT ETFs have have been retrieved using Tiingo. ↩ ↩² ↩³
Who could have predicted the price action of gold and silver on 30th January 2026? ↩

Completing a Correlation Matrix: Another Problem from Finance

2026-01-06T00:00:00-06:00

The previous post of this series on mathematical problems related to correlation matrices introduced the nearest correlation matrix problem¹, which consists in determining the closest² valid correlation matrix to an approximate correlation matrix

In this blog post, I will now describe the correlation matrix completion problem, which consist in filling in the missing coefficients of a partially specified correlation matrix in order to produce a valid correlation matrix.

As noted in Dreyer et al.³, such matrices often arise in financial applications when the number of stochastic variables becomes large or when several smaller models are combined in a larger model³.

After a couple of reminders on correlation matrices, I will detail the mathematical formulation of the correlation matrix completion problem, discuss its exact and approximate solutions and I will finally illustrate how this problem naturally appears when working with capital market assumptions from financial institutions.

Mathematical preliminaries

Let $n$ be a number of assets.

Correlation matrices

Let $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be a square matrix.

$C$ is a correlation matrix if and only if

$C$ is symmetric: $C {}^t = C$
$C$ is unit diagonal: $C_{i,i} = 1$, $i=1..n$
$C$ is positive semi-definite, that is, $x {}^t C x \geqslant 0, \forall x \in \mathbb{R}^n $

The sets of correlation matrices

Let be the following convex sets:

$\mathcal{S}^n_{+} = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $X {}^t = X$ and $X$ is positive semi-definite $\}$
$\mathcal{S}^n_{++} = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $X {}^t = X$ and $X$ is positive definite $\}$
$\mathcal{E}^n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $X {}^t = X$ and $x_{ii} = 1, i = 1,…,n \}$

Then:

The set of correlation matrices is the convex compact set $\mathcal{S}^n_{+} \cap \mathcal{E}^n$.
The set of invertible correlation matrices is the open bounded convex set $\mathcal{S}^n_{++} \cap \mathcal{E}^n$.

As a geometrical side note, the set of vectorized correlation matrices defines a subset of the unit hypercube in $\mathbb{R}^n$ called the elliptope, c.f. for example the website Convex Optimization.

Partial correlation matrices completion

A partial correlation matrix $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ is a symmetric unit diagonal matrix⁴ whose coefficients $c_{ij} = c_{ji}$ are not all specified.

A partial correlation matrix is said to be partial positive (semi-)definite when its specified principal minors are positive (semi-)definite.

A positive semi-definite completion - or simply a completion - of a partial correlation matrix $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ is a correlation matrix $C^* \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $c_{ij}^{*} = c_{ij}$ whenever the coefficient $c_{ij}$ is specified in $C$.

A positive definite completion of a partial correlation matrix $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ is a positive definite correlation matrix $C^* \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $c_{ij}^{*} = c_{ij}$ whenever the coefficient $c_{ij}$ is specified in $C$.

Undirected graph associated to a correlation matrix

Let $C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ be a partial correlation matrix.

It is possible⁵ to associate an undirected graph $G = \left( V, E \right)$ to $C$, defined as follows:

$V = \{ 1, 2, …, n \}$ is the set of vertices
$E$ is the set of edges, with $(i,j) \in E$ whenever the coefficient $c_{ij}$ is specified, with $i \ne j$.

The graph $G$ is then said to be chordal if every cycle of length greater than or equal to 4 has a chord, which is an edge that is not part of the cycle but connects two vertices of the cycle⁶.

As a reminder, a cycle in G is a sequence of $k \geq 3$ pairwise distinct vertices $\left( v_1, …, v_s \right)$ such that $\left( v_1, v_2 \right)$, $\left( v_2, v_3 \right)$, …, $\left( v_{k-1} v_k \right)$, $\left( v_k v_1 \right)$ $\in E$, with $k$ called the length of the cycle.

The correlation matrix completion problem

Problem formulation

Let be:

$C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a partial correlation matrix.
$\mathcal{N}$ the index set of the specified off-diagonal elements of $C$.
$\mathcal{U}^n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $x_{ij}=c_{ij}$ for $(i,j) \in \mathcal{N} \}$

Then:

The correlation matrix completion problem is the problem of finding a correlation matrix $C^* \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ that completes the matrix $C$, that is, finding $C^* \in \mathcal{S}^n_{+} \cap \mathcal{U}^n$.
The positive definite correlation matrix completion problem is the problem of finding a positive definite correlation matrix $C^* \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ that completes the matrix $C$, that is, finding $C^* \in \mathcal{S}^n_{++} \cap \mathcal{U}^n$.

Existence of solutions

Contrary to the nearest correlation matrix problem, the correlation matrix completion problem does not always admit a solution.

Indeed, an already necessary condition for a completion to exist is for the partial correlation matrix to be partial positive semi-definite⁵.

So, for example, the following partial correlation matrix does not admit any completion:

\[C_1 = \begin{pmatrix} 1 & -1 & -1 & -1 \\ -1 & 1 & -1 & -1 \\ -1 & -1 & 1 & . \\ -1 & -1 & . & 1 \end{pmatrix}\]

Beyond this necessary condition, Fiedler⁷, Grone et al.⁵ and Smith⁸ all establish the following result⁹:

A partial positive semi-definite correlation matrix is completable regardless of the values of the specified correlations if and only if the undirected graph associated to these specified correlations is chordal.

To be noted that when that graph is not chordal, nothing can be said in general because the existence of a completion then depends on the exact values of the specified correlations.

As an illustration:

Let $C_2 \in \mathcal{M} \left( \mathbb{R}^{4 \times 4} \right)$ be the following partial positive definite correlation matrix:
\[C_2 = \begin{pmatrix} 1 & 1 & . & 1 \\ 1 & 1 & 1 & . \\ . & 1 & 1 & . \\ 1 & . & . & 1 \end{pmatrix}\]
The undirected graph associated to that partial correlation matrix is the graph with the set of vertives $V = \{ 1, 2, 3, 4 \}$ and the set of edges $G = \{ (1,2), (1,4), (2,3) \}$, depicted in Figure 1.

Figure 1. Undirected graph associated to the partial correlation matrix C2.

From Figure 1, that graph is chordal, because there are no cycles of length >= 4.

Consequently, the partial correlation matrix $C_2$ admit a positive definite completion and the existence of that completion does actually even not depend on the values of the specified correlations (here, all 1s).
Let now $C_3 \in \mathcal{M} \left( \mathbb{R}^{4 \times 4} \right)$ be the following partial positive semi-definite correlation matrix:
\[C_3 = \begin{pmatrix} 1 & 1 & . & 0 \\ 1 & 1 & 1 & . \\ . & 1 & 1 & 1 \\ 0 & . & 1 & 1 \end{pmatrix}\]
The undirected graph associated to that partial correlation matrix is the graph with the set of vertives $V = \{ 1, 2, 3, 4 \}$ and the set of edges $G = \{ (1,2), (1,4), (2,3), (3,4) \}$, depicted in Figure 2.

Figure 2. Undirected graph associated to the partial correlation matrix C3.

From Figure 2, that graph is not chordal, because there is only one cycle of length 4 and that cycle does not have any chord.

Consequently, the partial correlation matrix $C_3$ might or might not admit a completion, nothing can be said at this stage.

For the interested reader, a couple of other theoretical results can be found in Fiedler⁷, which also notes that the general [completion] problem […] seems to be difficult⁷.

Unicity of solutions

Again contrary to the nearest correlation matrix problem, the correlation matrix completion problem does not generally admit a unique solution when one exists.

Indeed, Grone et al.⁵ establishes that the set of all positive semi-definite completions of a partial correlation matrix is in general a convex compact set and not a singleton.

This result can be illustrated with one and two missing correlations:

One missing correlation

The partial correlation matrix $C_4 = \begin{pmatrix} 1 & . \\ . & 1 \end{pmatrix} $ has one missing correlation.

By introducing the variable $x$ representing that missing correlation, a completion is a valid completion if and only if it has a positive or null determinant, that is, $\det(x) = 1 - x^2 \geq 0$.

That condition being equivalent to the condition $x \in [-1,1]$, all correlation matrices of the form $ C^*_4(x) = \begin{pmatrix} 1 & x \\ x & 1 \end{pmatrix} $ with $x \in [-1,1]$ are completions of $C_4$.
Two missing correlations

The partial correlation matrix $C_5 \in \mathcal{M} \left( \mathbb{R}^{5 \times 5} \right)$ represented in Figure 3 has two missing correlations.

Figure 3. Partial correlation matrix with 2 missing correlations. Source: Georgescu et al.

Because all the specified sub-matrices of $C_5$ are positive semi-definite, a completion is again a valid completion if and only if it has a positive or null determinant.

By introducing the variables $x$ and $y$ representing the two missing correlations, the determinant of $C_5$ can be factored as $ \det(x,y) $ $ \approx$ $ −0.18 \left( 0.67x^2 +0.75y^2 −xy −0.32x−0.24y+0.17 \right)$.

Non-negativity of that determinant is then equivalent to the condition
\[0.67x^2 +0.75y^2 −xy −0.32x−0.24y+0.17 \leq 0\]
, which defines the 2-dimensional ellipse (and its interior) depicted in Figure 4.

Figure 4. Feasible region for Georgescu et al.'s partial correlation matrix completions.

All correlation matrices $C_5^{*}(x,y) \in \mathcal{M} \left( \mathbb{R}^{5 \times 5} \right)$ with the same correlations as $C_4$, a correlation $c^{*}_{34}$ $=$ $x$ and a correlation $c^{*}_{35}$ $=$ $y$ - with $x,y$ satisfying the above relationship - are thus completions of $C_5$.

The existence of infinitely many completions¹⁰ naturally leads to trying to find a best-estimate completion in some sense⁶, which does exist and is called the maximum determinant completion⁶.

The maximum determinant correlation matrix completion problem

Fiedler⁷ originally demonstrated that if a positive definite partial correlation matrix admits a completion, then, there is a unique matrix in the (nonempty) class of all positive definite completions […] that has maximum determinant⁸.

More recently, by introducing a generalized determinant that gives the determinant of the nonsingular part of [a] matrix¹¹, Dreyer¹¹ established a similar result for positive semi-definite partial correlation matrices.

That completion, called the maximum determinant completion - or the Max-Det completion - has several intesting theoretical properties, which makes it an ideal candidate for guaranteeing the unicity of the completion of a correlation matrix.

Problem formulation

Let be:

$C \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a partial correlation matrix.
$\mathcal{N}$ the index set of the specified off-diagonal elements of $C$.
$\mathcal{U}^n = \{ X \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ such that $x_{ij}=c_{ij}$ for $(i,j) \in \mathcal{N} \}$

The maximum determinant correlation matrix completion problem can be cast as a generalization of a semidefinite programming problem:

\[C^* = \operatorname{argmax} \log ( \det (X) ) \text{ s.t. } X \in \mathcal{S}^n_{+} \cap \mathcal{E}^n \cap \mathcal{U}^n\]

Assuming that a solution exist, c.f. the previous section, Grone et al.⁵ and Olvera Astivia¹² show using standard results from mathematical optimization theory that it is necessarily unique.

Mathematical properties of the solution

Georgescu et al.⁶ summarizes the main properties of the maximum determinant correlation matrix completion:

For the multivariate normal model, it maximizes the entropy of the distribution described by the matrix

van der Schans and Boer¹³ comments this property as follows:

Since the already specified [correlations] imply a dependence between variables between which no dependence is specified and dependence reduces the amount of uncertainty in a system, the intuitive interpretation of entropy maximization is not introducing more dependence than is already implied by the already specified [correlations].
Again for the multivariate normal model, it maximizes the likelihood of the correlation matrix
It corresponds to the analytic centre of the feasible region described by the positive semi-definiteness constraints⁶

In other words, for a given partial correlation matrix, its maximum determinant completion lies as “deep” as possible¹⁴ inside the set of all its positive definite completions.

Computation of the solution

The original algorithmic approach to solving determinant maximization problems with matrix constraints has been to use interior point methods¹⁵.

More recently, dual projected gradient methods have been developed that are better able to scale with the dimensionality of the problem¹⁶.

Nevertheless, the drawbacks of these algorithms, from an application point of view, are that they are difficult to implement […], do not always converge if inconsistent starting [correlations] are specified and, finally, global optimization is too slow and memory consuming for large matrices¹³.

More on “inconsistent starting correlations” later.

For these reasons, it is tempting to [simply] set the missing [correlations] to zero³, but this approach has several shortcomings³, among which the fact that forcing unspecified correlations to zero¹⁷ might not be ideal for financial applications where most assets are correlated to each other.

To illustrate this point, Figure 5 represents a partial correlation matrix with a block of missing correlations.

Figure 5. Partial correlation matrix with a block of missing correlations. Source: Georgescu et al.

The maximum determinant completion of the block of missing correlations is⁶

\[E_1 = \begin{pmatrix} 0.1000 & 0.1500 & 0.0500 & 0.0750 \\ 0.2400 & 0.3600 & 0.1200 & 0.1800 \\ 0.2200 & 0.3300 & 0.1100 & 0.1650 \\ 0.2600 & 0.3900 & 0.1300 & 0.1950 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

, while the completion of the same block of missing correlations obtained after setting them to zero and computing the nearest correlation matrix to the resulting full approximate correlation matrix is⁶

\[E_2 = \begin{pmatrix} 0.0022 & 0.0084 & 0.0004 & 0.0035 \\ 0.0003 & 0.0011 & 0.0001 & 0.0005 \\ 0.0025 & 0.0098 & 0.0005 & 0.0040 \\ 0.0042 & 0.0164 & 0.0008 & 0.0067 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

Comparing the two sub-matrices $E_1$ and $E_2$, it is clear that the completed correlations of $E_2$ are much closer to zero than those of $E_1$, thus imposing an additional soft constraint of “closeness to zero” to the correlation matrix completion problem.

Reverting to simple heuristics for the correlation matrix completion problem might thus be potentially dangerous, depending on the context.

Hopefully, van der Schans and Boer¹³ proposes an heuristic which does not suffer from these shortcomings and that:

Is fast, which makes it suitable for applications in which the computation time is important¹³
Tries to not introduce more dependence between variables than is implied by the initially specified [correlations]¹³, contrary to the “setting missing correlations to zero” heuristic
Additionally corrects for inconsistencies in the already specified [correlations]¹³, but again, more on this later
Empirically produces completions with an average correlation difference with the maxdet completion [that is] reasonable¹³

A drawback of this heuristic, though, is that it depends on the ordering of the rows and columns of the [correlation] matrix, i.e. the heuristic will yield a different result if first rows and columns of [the matrix] are interchanged before starting the procedure¹³, which might or might not be acceptable in practice.

As a side note, explicit solutions of the maximum determinant correlation matrix completion problem are known in specific cases, for example for correlation matrices that follow an L-shaped, block diagonal pattern which is a common structure in insurance problems¹², c.f. Georgescu et al.⁶

The infeasible maximum determinant correlation matrix completion problem

In the previous section, a solution to the maximum determinant correlation matrix completion problem was assumed to exist.

There are at least two practical problems with this assumption:

A partial correlation matrix might theoretically admit a completion, but numerical round-off errors might prevent an algorithm from finding it

An example of such an ill-behaved matrix can be found in Glunt et al.¹⁸, but it is a partial covariance matrix and not a partial correlation matrix…

Building a similar example using a partial correlation matrix is certainly doable, but I failed to do so quickly, so, no example here.
More typically, the initially specified [correlations] can be inconsistent in the sense that no valid completion exists¹³

Here, the partial correlation matrix $C_1$ perfectly illustrates this point, even though it is a rather extreme example.

In both cases, simply failing to compute a solution might be unacceptable (e.g. for downstream pipelines), so that something must be done.

Unfortunately, the currently existing literature does not focus on algorithms that both complete partially specified matrices and also correct for inconsistencies¹³.

In other words, there is no well-established formulation of what could be called the infeasible maximum determinant correlation matrix completion problem.

At minimum, what can be said is that any solution to this new problem must involve a trade-off between

The closeness to the initial partial correlation matrix, for example in terms of the Frobenius distance
The value of the determinant

Incidentally, this is exactly how the heuristic algorithm of van der Schans and Boer¹³ is working: the specified [correlations] are adjusted as little as possible¹³ and the introduced extra (conditional) dependence between variables is as little as possible¹³.

When applied to the partial correlation matrix $C_1$, van der Schans and Boer’s algorithm¹³ gives

\[C^*_1 = \begin{pmatrix} 1 & -0.99 & -0.07 & -0.07 \\ -0.99 & 1 & -0.07 & -0.07 \\ -0.07 & -0.07 & 1 & 0.98 \\ -0.07 & -0.07 & 0.98 & 1 \end{pmatrix}\]

As expected, the completed correlation matrix $C^*_1$ is very far from the partial correlation matrix $C_1$, with only one correlation close to its initial value of -1 (-0.99).

Again, this is an extreme infeasible example, but it empirically demonstrates that adjusting the initially specified correlations, even as little as possible¹³, can lead to a completed correlation matrix that does not ressemble at all the initial partial correlation matrix.

Implementation in Portfolio Optimizer

Portfolio Optimizer implements two methods to complete a partial correlation matrix through the endpoint /assets/correlation/matrix/completed:

One proprietary method that guarantees to find the maximum determinant completion if it exists or that guarantees to minimally adjust - in terms of Frobenius distance - the partially specified correlation matrix so that a maximum determinant completion exists
The heuristic method of van der Schans and Boer¹³, with additional tweaks to improve its numerical robustness

For comparison with van der Schans and Boer’s algorithm¹³, the proprietary method of Portfolio Optimizer applied to the partial correlation matrix $C_1$ gives:

\[C^{**}_1 = \begin{pmatrix} 1 & -0.30 & -0.59 & -0.59 \\ -0.30 & 1 & -0.59 & -0.59 \\ -0.59 & -0.59 & 1 & 1 \\ -0.59 & -0.59 & 1 & 1 \end{pmatrix}\]

Comparing the two completed matrices $C_1^{*}$ and $C_1^{**}$, it appears that the matrix $C_1^{**}$ is much¹⁹ closer to the partial correlation matrix $C_1$ than the matrix $C_1^{*}$, which is consistent with the expected behaviour of Portfolio Optimizer.

Example of usage - Completing partial correlation matrices from financial institutions

Major financial institutions regularly provide forecasts of future risk/return characteristics for broad asset classes over the next 5 to 20 years, called (Long Term) Capital Market Assumptions (LTCMA).

In addition to the future expected volatility, the risk forecasts sometimes also include future expected correlation matrices, which is for example the case with J.P. Morgan.

Unfortunately, these correlation matrices are typically partially specified.

In addition, even if they were fully specified, combining capital market assumptions from several financial institutions would render them partial due to all institutions not covering the same asset classes…

So, as an example of usage, I propose to compute the maximum determinant completion of the partial correlation matrix provided by Blackrock as part of their November 2025 5-year capital market assumptions and represented in Figure 6.

Figure 6. Five-year expected future partial correlations between major asset classes, NZD currency, 13th November 2025. Source: Blackrock.

The resulting maximum determinant completed correlation matrix is displayed in Figure 7.

Figure 7. Five-year expected future Max-Det completed correlations between major asset classes, NZD currency, 13th November 2025.

I would like to mention that such an example of usage was inspired by a LinkedIn post from Peter Urbani, who proposes to complete Blackrock’s partial correlation matrix into the correlation matrix displayed in Figure 8 thanks to a 2-factor model built from the fully specified global equities/global bonds correlations.

Figure 8. Five-year expected future 2-factor model completed correlations between major asset classes, NZD currency, 13th November 2025. Source: Peter Urbani.

Comparing the “Max-Det” completed correlation matrix with Peter Urbani’s “2-factor model” completed correlation matrix , both matrices are perfectly identical²⁰.

As a closing (fun) remark, Olvera Astivia¹² highlights that having half of the entries in a correlation matrix missing [can be] considered a rather extreme condition¹².

What could then be said about the initial Blackrock’s partial correlation matrix that has around 82% of missing entries?!

Conclusion

With this first blog post of 2026, my hope is that you have added to your quantitative toolbox a useful methodology to obtain, in a mathematically principled way, values implied in correlational structure of the data, even if said data has not (or cannot) be obtained¹².

In any cases, for more mathematics of correlation matrices, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

Nicholas J. Higham, Computing the Nearest Correlation Matrix—A Problem from Finance, IMA J. Numer. Anal. 22, 329–343, 2002.. ↩
In terms of the Frobenius distance. ↩
Olaf Dreyer, Horst Kohler, Thomas Streuer, Completing correlation matrices, arXiv. ↩ ↩² ↩³ ↩⁴
That is, belonging to the set $\mathcal{E}^n$. ↩
R. Grone, C.R. Johnson, E. Sa, H. Wolkowicz, Positive definite completions of partial Hermitian matrices, Linear Algebra Appl. 58 (1984) 109–124. ↩ ↩² ↩³ ↩⁴ ↩⁵
Georgescu DI, Higham NJ, Peters GW. 2018 Explicit solutions to correlation matrix completion problems, with an application to risk management and insurance. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸
Fiedler, M. Matrix Inequalities. Numer. Math. 9, 109–119 (1966). ↩ ↩² ↩³ ↩⁴
Ronald L. Smith, The positive definite completion problem revisited, Linear Algebra and its Applications, Volume 429, Issue 7, 2008,Pages 1442-1452. ↩ ↩²
One interesting feature of that result is that the existence of a correlation matrix completion - which seems quite algebric in nature - is actually equivalent to a “visual” condition. ↩
When one exists. ↩
Olaf Dreyer, Matrix completion and semidefinite matrices, arXiv. ↩ ↩²
Oscar L. Olvera Astivia (2021) A Note on the General Solution to Completing Partially Specified Correlation Matrices, Measurement: Interdisciplinary Research and Perspectives, 19:2, 115-123. ↩ ↩² ↩³ ↩⁴ ↩⁵
See van der Schans, Martin and Boer, Alex, A Heuristic for Completing Covariance And Correlation Matrices (March 14, 2013). Technical Working Paper 2014-01 November 2014. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷
The maximum determinant correlation matrix completion maximizes the product of distances to the defining hyperplanes⁶. ↩
See Vandenberghe, Lieven and Boyd, Stephen and Wu, Shao-Po, Determinant Maximization with Linear Matrix Inequality Constraints, SIAM Journal on Matrix Analysis and Applications, Volume 19, Number 2, Pages 499-533. ↩
See Nakagaki, T., Fukuda, M., Kim, S. et al. A dual spectral projected gradient method for log-determinant semidefinite problems. Comput Optim Appl 76, 33–68 (2020). ↩
Or close to zero in case the nearest correlation matrix to the resulting approximate correlation matrix is computed. ↩
W. Glunt, T.L. Hayden, Charles R. Johnson, P. Tarazaga, Positive definite completions and determinant maximization, Linear Algebra and its Applications, Volume 288, 1999, Pages 1-10. ↩
The Frobenius distance between $C_1$ and $C^*_1$ is $\approx 2.630$, while the Frobenius distance between $C_1$ and $C^{**}_1$ is $\approx 1.575$. ↩
Incidentally, this allows to conclude - due to the entropy maximization property of the maximum determinant correlation matrix completion under a multivariate normal model - that the 2-factor model used by Peter Urbani is a vanilla multivariate normal model. ↩

Covariance Matrix Forecasting: Average Oracle Method

2025-12-10T00:00:00-06:00

Continuing this series on covariance matrix forecasting (c.f. here and there for the previous posts), I will now describe a relatively recent¹ data-driven, model-free, way to [forecast] covariance [and correlation] matrices of time-varying systems² rooted in random matrix theory.

This method - introduced in Bongiorno et al.² and called Average Oracle - consists in replacing the eigenvalues of a (noisy) estimate of a time-varying covariance matrix by time-independant eigenvalues that encode the average influence of the future on present eigenvalues².

In this blog post, I will describe that method and, as now usual in this series, I will illustrate its empirical performances in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio.

Mathematical preliminaries

Some of these sub-sections contain reminders from a previous blog post.

Dynamic covariance and correlation matrices

Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (logarithmic) return process of these assets over a time period $t$ (a day, a week, a month..) over which their mean return vector $\mu_t \in \mathbb{R}^n$ is supposed to be null.

Then:

The asset covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ over the time period $t$ is defined as $\Sigma_t = \mathbb{E} \left[ r_t r_t {}^t \right]$.

That matrix is called³ the population (or true) covariance matrix of the asset returns over the time period $t$.
The asset correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ over the time period $t$ is defined as the correlation matrix $ C_t = V_t^{-1} \Sigma_t V_t^{-1} $ associated to the covariance matrix $\Sigma_t$, where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset standard deviations.

Let now be $T$ time periods $t = 1..T$.

Then:

The averaged⁴ covariance matrix $\Sigma_{1:T} \in \mathcal{M}(\mathbb{R}^{n \times n})$ over the $T$ time periods $t = 1..T$ is defined as $ \Sigma_{1:T} = \frac{1}{T} \sum_{t=1}^{T} \Sigma_{t} $.

In case the return process is time-invariant, the averaged covariance matrix $\Sigma_{1:T}$ is equal to the (constant) population covariance matrix $\left( \Sigma = \right) \Sigma_t, t = 1..T$.
The averaged⁴ correlation matrix $C_{1:T} \in \mathcal{M}(\mathbb{R}^{n \times n})$ over the $T$ time periods $t = 1..T$ is defined as $ C_{1:T} = \frac{1}{T} \sum_{t=1}^{T} C_{t} $.

In case the return process is time-invariant, the averaged correlation matrix $C_{1:T}$ is equal to the (constant) correlation matrix $\left( C = \right) = C_t, t = 1..T$.
The pseudo-averaged⁵ correlation matrix $C_{p, 1:T} \in \mathcal{M}(\mathbb{R}^{n \times n})$ over the $T$ time periods $t = 1..T$ is defined as the correlation matrix associated to the averaged covariance matrix $\Sigma_{1:T}$.

In case the return process is time-invariant, the pseudo-averaged correlation matrix $C_{p, 1:T} $ is equal to the averaged correlation matrix $C_{1:T}$, but in general, due to time-varying asset standard deviations, the two correlation matrices are different.

Dynamic covariance and correlation matrices sample estimators

In practice, the asset return process $r_t$ is usually not known and the only available information is the vectors of realized asset returns $ \tilde{r}_1,…, \tilde{r}_T \in \mathbb{R}^n$ for $T$ time periods.

From each of these vectors:

The classical way to estimate the covariances is to compute the empirical (or sample) covariance matrix thanks to Pearson estimator³ defined as $\tilde{\Sigma}_t = \tilde{r}_t \tilde{r}_t {}^t $ over each time period $t = 1..T$.

Here, the outer product of the realized asset returns $ \tilde{r}_t \tilde{r}_t {}^t $ over the time period $t$ is called a covariance estimate $\tilde{\Sigma}_t$ - or covariance proxy⁶ - for the (unobserved) asset returns covariance matrix over that time period.
The empirical (or sample) correlation matrix over each time period $t = 1..T$ is defined as the correlation matrix $ \tilde{C}_t $ associated to the covariance matrix $\tilde{\Sigma}_t$.
The empirical averaged covariance matrix over the $T$ time periods $t = 1..T$ is typically estimated from the realized asset returns $ \tilde{r}_1,…, \tilde{r}_T \in \mathbb{R}^n$ thanks to the averaged Pearson estimator defined as $\tilde{\Sigma}_{1:T} = \frac{1}{T} \sum_{t=1}^{T} \tilde{r}_t \tilde{r}_t {}^t $.
The empirical averaged correlation matrix over the $T$ time periods $t = 1..T$ is defined as $ \tilde{C}_{1:T} = \frac{1}{T} \sum_{t=1}^{T} \tilde{C}_{t} $.
The empirical pseudo-averaged correlation matrix over the $T$ time periods $t = 1..T$ is defined as the correlation matrix $ \tilde{C}_{p, 1:T}$ associated to the empirical averaged covariance matrix $\tilde{\Sigma}_{1:T}$.

Covariance and correlation matrices rotationally invariant estimators

Rotationally invariant estimators

As already mentioned in a previous blog post on correlation matrices denoising, the estimation of empirical covariance and correlation matrices in finance is affected by noise, in the form of measurement error, due in part to the short length of the time series of asset returns typically used in their computation.

Indeed:

Constructing a well-diversified portfolio requires many assets⁷, that is, a big $n$.
In contrast, rapid shifts in financial market dependencies can only be captured by short calibration windows⁷ for estimating asset correlations, that is, a small $T$.

This situation leads to an aspect ratio $q = \frac{n}{T}$ of empirical matrices either close to 1 - or even worse, much greater than 1 -, which is catastrophic from an estimation perspective, c.f. the above blog post and references therein.

Hopefully, numerous techniques have been developed to improve the estimation of noisy covariance [or correlation] matrices⁷.

Several of these techniques, like the eigenvalue clipping method⁸, involve a specific class of matrix estimators - known as Rotationally Invariant Estimators (RIE) or Orthogonally Invariant Estimators (OIE) - that leaves the eigenvectors of the empirical matrices untouched while altering their eigenvalues.

In other words, an RIE estimator $\Xi \left( \tilde{\Sigma}_t \right) \in \mathcal{M}(\mathbb{R}^{n \times n})$ of a true covariance or correlation matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ obtained from its empirical counterpart $\tilde{\Sigma}_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ has the general form

\[\Xi \left( \tilde{\Sigma}_t \right) = \tilde{V}_t \Lambda_t \tilde{V}_t {}^t\]

, where:

$\Lambda_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is a diagonal matrix of well-chosen² eigenvalues.
$\tilde{V}_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the matrix of eigenvectors of $\tilde{\Sigma}_t$, defined through the spectral decomposition $ \tilde{\Sigma}_t = \tilde{V}_t \tilde{\Lambda}_t \tilde{V}_t {}^t $ with $ \tilde{\Lambda}_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ the diagonal matrix of eigenvalues of $\tilde{\Sigma}_t$.

Bun et al.³ explains the underlying rationale as follows:

The true matrix [$\Sigma_t$] is unknown and we do not have any particular insights on its components (the eigenvectors).

Therefore we would like our estimator [$\Xi \left( \tilde{\Sigma}_t \right)$] to be constructed in a rotationally invariant way from the noisy observation [$\tilde{\Sigma}_t$] that we have.

In simple terms, this means that there is no privileged direction in the $n$-dimensional space that would allow one to bias the eigenvectors of the estimator [$\Xi \left( \tilde{\Sigma}_t \right)$] in some special directions.

More formally, the estimator construction must obey: $ \Omega \Xi \left( \tilde{\Sigma}_t \right) \Omega {}^t$ $=$ $\Xi \left( \Omega \tilde{\Sigma}_t \Omega {}^t \right)$ for any rotation matrix $\Omega \in \mathcal{M}(\mathbb{R}^{n \times n})$.

Any estimator satisfying [that equation] will be referred to as a Rotational Invariant Estimator (RIE).

In this case, it turns out that the eigenvectors of the estimator [$\Xi \left( \tilde{\Sigma}_t \right)$] have to be the same as those of the noisy matrix [$\tilde{\Sigma}_t$].

Rotationally invariant oracle estimator

Bun et al.³ shows that the optimal RIE estimator of the unknown matrix $\Sigma_t$ in terms of Frobenius norm is the RIE estimator whose diagonal matrix of eigenvalues $\Lambda_O$ satisfy

\[\Lambda_O = \text{diag} \left( \tilde{V}_t {}^t \Sigma_t \tilde{V}_t \right)\]

That estimator is sometimes called the oracle estimator because it depends explicitly on the knowledge of the true signal [$\Sigma_t$]³ and so is not directly usable in practice.

Remarkably, [though], asymptotically optimal RIEs that converge to the oracle estimator can be obtained without the knowledge of the true covariance; however, such estimators require that: i) the ground truth does not change, ii) the data matrix is very large, and iii) the data has at least finite fourth moments².

Those conditions, and especially the first one, are definitely not satisfied by asset returns, which leads to suboptimal estimators in practice…

As a side note, and maybe contrary to intuition, the optimal eigenvalues $\Lambda_O$ are NOT equal to the eigenvalues of $\Sigma_t$, because this would result in a spectrum that is too wide³.

The Average Oracle covariance matrix forecasting method

Forecasting formulas

Let be:

$n$ be the number of assets in a universe of assets
$\tilde{r}_t \tilde{r}_t {}^t$, $t=1..T$ the outer products of the observed asset returns over each of $T$ past periods
$1 \leq h_{in} \ll T$ a chosen number of past periods
$\mathcal{I}_{cal} = [t_{cal}, T - h_{next}]$ a long calibration window, with $t_{cal}$ chosen so that $t_{cal} \geq h_{in} - 1$ and $| \mathcal{I}_{cal} | \gg h_{in}$

Asset returns averaged covariance matrix

The Average Oracle covariance matrix forecasting model estimates the asset returns averaged covariance matrix $\hat{\Sigma}_{T+1:T+h_{next}}$ over the next $h_{next} \geq 1$ periods as follows²:

Choose a number of random time periods $n_B \geq 1$ to generate.
For $b = 1..n_B$ do
- Select uniformly at random with replacement a time period $t^{(b)} \in \mathcal{I}_{cal}$
- Compute the “past” averaged asset returns covariance matrix $ \tilde{\Sigma}^{(b)}_{in} $ on the train window $\mathcal{I}^{(b)}_{in} = [t^{(b)} - h_{in} + 1, t^{(b)}]$, defined by $\tilde{\Sigma}^{(b)}_{in} = \tilde{\Sigma}_{t^{(b)} - h_{in} + 1:t^{(b)}} = \frac{1}{h_{in}} \sum_{t=t^{(b)} - h_{in} + 1}^{t^{(b)}} \tilde{r}_t \tilde{r}_t {}^t $ and its associated correlation matrix $\tilde{C}^{(b)}_{in} = \tilde{C}_{p, t^{(b)} - h_{in} + 1:t^{(b)}}$ whose spectral decomposition is given by $\tilde{C}^{(b)}_{in} = \tilde{V}^{(b)}_{in} \tilde{\Lambda}^{(b)}_{in} \tilde{V}^{(b)}_{in} {}^t $.
- Compute the “future” averaged asset returns covariance matrix $ \tilde{\Sigma}^{(b)}_{next} $ on the test window $\mathcal{I}^{(b)}_{next} = [t^{(b)} + 1, t^{(b)} + h_{next}]$, defined by $ \tilde{\Sigma}^{(b)}_{next} = \tilde{\Sigma}_{t^{(b)} + 1:t^{(b)} + h_{next}} = \frac{1}{h_{next}} \sum_{t=t^{(b)} + 1}^{t^{(b)} + h_{next}} \tilde{r}_t \tilde{r}_t {}^t $ and its associated correlation matrix $\tilde{C}^{(b)}_{next} = \tilde{C}_{p, t^{(b)} + 1:t^{(b)} + h_{next}}$.
- Compute the diagonal matrix of oracle eigenvalues $\tilde{\Lambda}_O^{(b)} = \text{diag} \left( \tilde{V}^{(b)}_{in} {}^t \tilde{C}^{(b)}_{next} \tilde{V}^{(b)}_{in} \right) $.
Compute the diagonal matrix of Average Oracle eigenvalues $\tilde{\Lambda}_{AO} = \frac{1}{n_B} \sum_{b=1}^{n_B} \tilde{\Lambda}_O^{(b)} $.
Compute the most recent “past” averaged asset returns covariance matrix $ \tilde{\Sigma}_{in} $ on the window $\mathcal{I}_{in} = [T - h_{in} + 1, T]$, defined by $ \tilde{\Sigma}_{in} = \tilde{\Sigma}_{T - h_{in} + 1:T} = \frac{1}{h_{in}} \sum_{t=T - h_{in} + 1}^{T} \tilde{r}_t \tilde{r}_t {}^t $ and its associated correlation matrix $\tilde{C}_{in} = \tilde{C}_{p, T - h_{in} + 1:T}$ whose spectral decomposition is given by $\tilde{C}_{in} = \tilde{V}_{in} \tilde{\Lambda}_{in} \tilde{V}_{in} {}^t $.
Compute $\hat{\Sigma}_{T+1:T+h_{next}} = D_{in} \tilde{V}_{in} \tilde{\Lambda}_{AO} \tilde{V}_{in} {}^t D_{in} $, where $D_{in} \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the standard deviations $\sqrt{ \left( \tilde{\Sigma}_{in} \right)_ii }, i=1..n$.

For more visual clarity, Figure 1 illustrates that process.

Figure 1. Illustration of the Average Oracle eigenvalues computation process, which uses different windows included in a long calibration window. Source: Adapted from Bongiorno et al.

Asset returns averaged and pseudo-averaged correlation matrix

The Average Oracle covariance matrix forecasting model does not easily⁹ allow to estimate the asset returns averaged correlation matrix $\hat{C}_{T+1:T+h_{next}}$, because it does not rely on the estimation of the individual covariance matrices $\hat{\Sigma}_{T+1}$, $…$, $\hat{\Sigma}_{T+h_{next}}$.

The asset returns pseudo-averaged correlation matrix $\hat{C}_{p, T+1:T+h_{next}}$ over the next $h_{next}$ periods, though, corresponds to the correlation matrix associated to the averaged covariance matrix $\hat{\Sigma}_{T+1:T+h_{next}}$.

Rationale

The Average Oracle covariance matrix forecasting method captures the average transition from two consecutive time windows² by averaging [oracle eigenvalues], rank-wise, over many randomly selected consecutive intervals taken from a long calibration window².

That covariance matrix forecasting method thus tackles the evolution of [asset returns] dependencies with a time-invariant eigenvalue cleaning scheme².

As noted in Bongiorno et al.²:

This is a zeroth order approximation, as the fluctuations of the optimal eigenvalue matrix around $\tilde{\Lambda}_{AO}$ sometimes most probably contain valuable additional information (as may do those of the eigenvectors). Nevertheless, this approximation is a powerful filtering tool and is easily computed from data without any modeling assumptions about the underlying system.

Performances

From a practical perspective, Bongiorno et al.² and Bongiorno and Challet¹⁰ empirically demonstrate that the Average Oracle covariance matrix forecasting method is more performant than the current state-of-the-art (and complex) methods, Dynamic Conditional Covariance coupled to Non-Linear Shrinkage (DCC+NLS)¹⁰, both in terms of Frobenius distance - as highlighted in Figure 2 - and in terms of four key portfolio metrics: Sharpe ratio, turnover, gross leverage, and diversification¹⁰.

Figure 2. Average Frobenius distance between the forecasted and the out-of-sample covariance matrices of n = 100 U.S. stocks as a function of the number of past periods $h_{in}$. Source: Adapted from Bongiorno et al.

These performances are commented as follows in Bongiorno et al.²:

The fact that the Average Oracle is a better estimator for time-evolving covariance matrices most often implies that the most recent information contained in the sample eigenvalues is less relevant (and more noisy) than the AO ones that focus on the average transition.

Thus, the advantage of the Average Oracle is precisely that it captures some part of the average dynamics that is discarded by the assumption of a constant true covariance matrix [made in the DCC+NLS method].

Implementation details

How to choose the number of past periods $h_{in}$?

Through extensive simulations, Bongiorno et al.² concludes that the Average Oracle eigenvalues $\tilde{\Lambda}_{AO}$ mainly depend on the number of assets $n$ and¹¹ on the number of past periods $h_{in}$ over which to compute the averaged asset returns covariance matrix.

A natural question - unfortunately neither answered in Bongiorno et al.² nor in Bongiorno and Challet¹⁰ - is then how to choose the value of $h_{in}$ in order to obtain the best forecasting performances?

One possible answer, that relies on the interpretation of the Average Oracle covariance matrix forecasting method as a covariance matrix cleaning scheme², is to select $h_{in}$ so as to maximize the forecasting performances of a simple moving average covariance matrix forecasting model with a window size equal to $h_{in}$ for the considered value of $h_{next}$.

How to choose the number of random time periods $n_B$?

The number of random time periods $n_B$ must be high enough to ensure that the Average Oracle eigenvalues are stable enough - in particular when $h_{next}$ is small - because by reducing [$h_{next}$], the estimation becomes noisier and thus requires more train and test windows […] to yield average eigenvalues with the same level of precision².

Two examples¹²:

Bongiorno et al.² uses $n_B = 10 000$ together with $h_{in} = h_{next} = 252$.
Bongiorno and Challet¹⁰ uses $n_B = 10 000$ together with $h_{in} \in \lbrace 240, 1200 \rbrace$ and $h_{next} \in \lbrace 5, 20 \rbrace$.

To be noted, though, that depending on the length of the calibration window $\mathcal{I}_{cal}$ and/or on the number of assets $n$, the time periods do not need to be generated at random - they can perfectly be generated deterministically so as to cover the whole calibration window.

How to enforce a proper ordering of the Average Oracle eigenvalues?

Bongiorno et al.² stresses that the columns of the eigenvectors [$\tilde{V}^{(b)}_{in}$ and $\tilde{V}_{in}$] must always follow the same the eigenvalue ordering convention².

Nevertheless, despite enforcing such a convention, the order of the resulting Average Oracle eigenvalues $\tilde{\Lambda}_{AO}$ is not necessarily preserved due to the finite size of the sample¹³.

This may be an unwanted feature within a rotational invariant assumption¹³, since there is no reason a priori to expect that it is optimal to modify the order of the eigenvalues, that is to say, the variance associated with the principal components¹³.

Bun et al.¹³ proposes two solutions to this problem:

Sort the resulting eigenvalues
Perform an isotonic regression on the resulting eigenvalues

In the context the cross-validated eigenvalues cleaning scheme described in Reigneron et al.¹⁴, the impact of using an isotonic regression is depicted in Figure 3.

Figure 3. Raw, cross-validated and isotonic eigenvalues eigenvalues as a function of in-sample eigenvalues. Source: Reigneron et al.

Implementation in Portfolio Optimizer

Portfolio Optimizer implements the Average Oracle covariance and correlation matrix forecasting models through the endpoints /assets/covariance/matrix/forecast/average-oracle and /assets/correlation/matrix/forecast/average-oracle.

These endpoints support the 2 covariance proxies below:

Squared (close-to-close) returns
Demeaned squared (close-to-close) returns

These endpoints also:

Implement an isotonic regression correction step for the Average Oracle eigenvalues.
Allow to automatically determine the number of past periods $h_{in}$, using a proprietary procedure.
Allow to either generate a given number of time periods $n_B$ uniformly at random within the calibration window $\mathcal{I}_{cal}$ or use all the time periods available within that window.

Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs

As an example of usage, I propose to evaluate the empirical performances of the Average Oracle covariance matrix forecating model within the framework of the previous blog bost, whose aim is to forecast monthly covariance and correlation matrices for a portfolio of 10 ETFs representative¹⁵ of misc. asset classes:

U.S. stocks (SPY ETF)
European stocks (EZU ETF)
Japanese stocks (EWJ ETF)
Emerging markets stocks (EEM ETF)
U.S. REITs (VNQ ETF)
International REITs (RWX ETF)
U.S. 7-10 year Treasuries (IEF ETF)
U.S. 20+ year Treasuries (TLT ETF)
Commodities (DBC ETF)
Gold (GLD ETF)

Results - Covariance matrix forecasting

Results over the period 31st January 2008 - 31st July 2023¹⁶ for covariance matrices are the following¹⁷:

Covariance matrix model	Covariance matrix MSE
SMA, window size of all the previous months (historical average model)	9.59 $10^{-6}$
SMA, window size of the previous year	9.08 $10^{-6}$
Average Oracle, optimal¹⁸ $h_{in}$	6.77 $10^{-6}$
EWMA, optimal¹⁸ $\lambda$	6.52 $10^{-6}$
IEWMA, optimal¹⁸ $\left(\lambda_{vol},\lambda_{cor}\right)$	6.16 $10^{-6}$
SMA, window size of the previous month (random walk model)	6.06 $10^{-6}$

Within this specific evaluation framework, the Average Oracle covariance matrix forecasting model¹⁹ unfortunately does not seem to exhibit improved performances v.s. much simplier models, like the EWMA covariance matrix forecasting model.

Results - Correlation matrix forecasting

Results over the period 31st January 2008 - 31st July 2023¹⁶ for the correlation matrices associated to the covariance matrices of the previous sub-section are the following¹⁷:

Covariance matrix model	Correlation matrix MSE
SMA, window size of the previous month (random walk model)	8.19
SMA, window size of all the previous months (historical average model)	8.10
Average Oracle, optimal¹⁸ $h_{in}$	6.59
SMA, window size of the previous year	6.50
EWMA, optimal¹⁸ $\lambda$	5.87
IEWMA, optimal¹⁸ $\left(\lambda_{vol},\lambda_{cor}\right)$	5.70

Here again, the Average Oracle model¹⁹ does not seem to particularly shine v.s. simplier models…

Comments

Results from the previous sub-sections seem contradictory to those obtained in Bongiorno et al.².

However, as noted in Tan and Zohren¹, this is probably due to the fact that at relatively small dimensions, a covariance estimator may benefit more from picking up more recent time series variations¹ like the EWMA and IEWMA covariance estimators.

In other words, the Average Oracle is certainly a good choice when the dimension of the problem becomes very large³ but is probably not competitive otherwise.

Conclusion

The Average Oracle is a covariance and correlation matrix forecasting method very different in spirit from the moving average-based methods already described in the previous posts of this series.

Unfortunately, the empirical performances of the Average Oracle method in terms of covariance and correlation matrix forecasting do not seem to improve over the much simplier EWMA and IEWMA methods, at least under the specific asset allocation context described in the previous section.

To be noted, though, that this conclusion might be different with other Oracle-based covariance matrix forecasting methods, like those described in Tan and Zohren¹ or in Reigneron et al.¹⁴…

Anyway, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Vincent Tan, Stefan Zohren, Estimation of Large Financial Covariances: A Cross-Validation Approach, The Journal of Portfolio Management February 2025, 51 (4) 83-95. ↩ ↩² ↩³ ↩⁴
See Bongiorno, C., Challet, D. and Loeper, G., Filtering time-dependent covariance matrices using time-independent eigenvalues. J. Stat. Mech.: Theory and Experiment, 2023, 2, 023402. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰
See J. Bun, R. Allez, J. -P. Bouchaud and M. Potters, Rotational Invariant Estimator for General Noisy Matrices, IEEE Transactions on Information Theory, vol. 62, no. 12, pp. 7475-7490, Dec. 2016. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
See Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426. ↩ ↩²
For the lack of a better term. ↩
See Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg. ↩
See Christian Bongiorno and Lamia Lamrani, Quantifying the information lost in optimal covariance matrix cleaning, Physica A: Statistical Mechanics and its Applications, 657, 130225, 2025. ↩ ↩² ↩³
See Laurent Laloux, Pierre Cizeau, Jean-Philippe Bouchaud, and Marc Potters, Noise Dressing of Financial Correlation Matrices, Phys. Rev. Lett. 83, 1467. ↩
At least directly; by varying $h_{next}$, it is possible - but cumbersome - to compute the individual covariance matrices $\hat{\Sigma}_{T+1}$, $…$, $\hat{\Sigma}_{T+h_{next}}$ and deduce the averaged correlation matrix $\hat{C}_{T+1:T+h_{next}}$. ↩
See Bongiorno, C., & Challet, D. (2024). Covariance matrix filtering and portfolio optimisation: the average oracle vs non-linear shrinkage and all the variants of DCC-NLS. Quantitative Finance, 24(9), 1227–1234. ↩ ↩² ↩³ ↩⁴ ↩⁵
And interestingly, not so much on the value of $h_{next}$. ↩
In both cases, the 10 000 time periods random selection also incorporates a random selection of U.S. stocks within each time window. ↩
See Bun, J., Bouchaud, J. P., & Potters, M., Overlaps between eigenvectors of correlated random matrices. Physical Review E, 98(5), 052145 (2018). ↩ ↩² ↩³ ↩⁴
See Pierre-Alain Reigneron, Vincent Nguyen, Stefano Ciliberti, Philip Seager, Jean-Philippe Bouchaud, Agnostic Allocation Portfolios: A Sweet Spot in the Risk-Based Jungle?, The Journal of Portfolio Management March 2020, 46 (4) 22-38. ↩ ↩²
These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer²⁰. ↩
(Adjusted) daily prices have have been retrieved using Tiingo. ↩ ↩²
Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. ↩ ↩²
As determined by Portfolio Optimizer at the end of every month using all the available asset returns history up to that point in time; thus, there is no look-ahead bias. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Results using a couple of fixed value for $h_{in}$ (21,…) are worse and so not presented here. ↩ ↩²
See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩

Value at Risk: Univariate Estimation Methods

2025-10-28T00:00:00-05:00

Value-at-Risk (VaR) is one of the most commonly used risk measures in the financial industry¹ in part thanks to its simplicity - because VaR reduces the market risk associated with any portfolio to just one number² - and in part due to regulatory requirements (Basel market risk frameworks³⁴, SEC Rule 18f-4⁵…).

Nevertheless, when it comes to actual computations, the above definition is by no means constructive¹ and accurately estimating VaR is a very challenging statistical problem² for which several methods have been developed.

In this blog post, I will describe some of the most well-known univariate VaR estimation methods, ranging from non-parametric methods based on empirical quantiles to semi-parametric methods involving kernel smoothing or extreme value theory and to parametric methods relying on distributional assumptions.

Value-at-Risk

Definition

The Value-at-Risk of a portfolio of financial instruments corresponds to the maximum potential change in value of [that portfolio] with a given probability over a certain horizon².

More formally, the Value-at-Risk $VaR_{\alpha}$ of a portfolio over a time horizon $T$ (1 day, 10 days, 20 days…) and at a confidence level $\alpha$% $\in ]0,1[$ (95%, 97.5%, 99%…) can be defined⁶ as the opposite⁷ of the $1 - \alpha$ quantile of the portfolio return distribution over the time horizon $T$

\[\text{VaR}_{\alpha} = - \inf_{x} \left\{x \in \mathbb{R}, P(X \leq x) \geq 1 - \alpha \right\}\]

, where $X$ is a random variable representing the portfolio return over the time horizon $T$.

This formula is also equivalent⁸ to

\[\text{VaR}_{\alpha} = - F_X^{-1}(1 - \alpha)\]

, where $F_X^{-1}$ is the inverse cumulative distribution function, also called the quantile function, of the random variable $X$.

Graphically, this definition is illustrated in:

Figure 1, for a continuous portfolio return distribution at a generic confidence level $\alpha$% and over a generic horizon.

Figure 1. Graphical illustration of a portfolio VaR as a quantile of its continuous return distribution. Source: Adapted from Yamai and Yoshiba.
Figure 2, for a discrete portfolio return distribution at a confidence level $\alpha$ = 99% and over a 1-month horizon, which is commented as follows in Jorion⁹:

We need to find the loss that will not be exceeded in 99% of cases, or such that 1% of [the 624] observations - that is, 6 out of 624 occurrences - are lower. From Figure [2], this number is about -3.6%, [resulting in a portfolio VaR of 3.6%].

Figure 2. Graphical illustration of a portfolio VaR as a quantile of its discrete monthly return distribution, $\alpha$ = 99%. Source: Jorion.

Univariate v.s. multivariate Value-at-Risk

A portfolio can be considered as both:

An asset in itself, with its own return distribution.
A weighted collection of individual assets, each with their own return distribution.

This raises the question of whether first to aggregate profit and loss data and proceed with a univariate [VaR] model for the aggregate, or to start with disaggregate data¹⁰ and proceed with a multivariate VaR model from the disaggregated data.

In this blog post, I will only discuss univariate¹¹ VaR models - originally suggested by Zangari¹² as simple and effective approach[es] for calculating Value-at-Risk¹² - in which portfolio returns are considered as a univariate time series without reference to the portfolio constituents¹³.

Indeed, since the goal of VaR is to measure the market risk of a portfolio, it seems reasonable to model the portfolio return series directly¹².

Arithmetic returns v.s. logarithmic returns in Value-at-Risk calculations

In VaR calculations, it is usually prefered, for a variety of reasons, to work with logarithmic returns rather than arithmetic (simple, linear) ones¹³, c.f. Ballotta¹³ and Jorion⁹ for more details.

In that case, though, because investors are primarily interested in simple returns¹⁴, the logarithmic VaR $\text{VaR}_{\alpha}^{(l)} $ needs to be converted into an arithmetic VaR $\text{VaR}_{\alpha}^{(a)} $.

Thanks to the definition of VaR as a quantile of the portfolio return distribution and the relationship between arithmetic and logarithmic returns, this is easily be done through the formula

\[\text{VaR}_{\alpha}^{(a)} = 1 - \exp \left( - \text{VaR}_{\alpha}^{(l)} - 1 \right)\]

While not frequently mentioned in the litterature¹⁵, it is important to be aware of this subtlety.

History of Value-at-Risk

Searching for the best means to represent the risk exposure of a financial institution’s trading portfolio in a single number¹⁶ is a quest that folklore attributes the inception of to Dennis Weatherstone at J.P. Morgan [in the late 1980s], who was looking for a way to convey meaningful risk exposure information to the financial institution’s board without the need for significant technical expertise on the part of the board members¹⁶.

It is then Till Guldimann, head of global research at J.P. Morgan at that time, who designed what would come to be known as the J.P. Morgan’s daily VaR report¹⁷ and who thus can be viewed as the creator of the term Value-at-Risk⁹.

The interested reader is referred to Holton¹⁸ for an historical perspective on Value-at-Risk, in which the origins of VaR as a measure of risk are even traced back as far as 1922 to capital requirements the New York Stock Exchange imposed on member firms¹⁸.

Value-at-Risk estimation

When a sample of portfolio returns over a given time horizon $r_1,…,r_n$ is available - like in ex post analysis -, the Value-at-Risk of that portfolio over the same horizon¹⁹ at a confidence level $\alpha \in ]0,1[$ is a textbook example of VaR calculation as the opposite of the $(1 - \alpha)$% quantile of the empirical return distribution $r_1,…,r_n$.

Problem is, the discrete nature of the extreme returns of interest makes it difficult to accurately compute that quantile, as explained in Danielsson and de Vries²⁰:

In the interior, the empirical sampling distribution is very dense, with adjacent observations very close to each other. As a result the sampling distribution is very smooth in the interior and is the mean squared error consistent estimate of the true distribution.

The closer one gets to the extremes, the longer the interval between adjacent returns becomes. This can be seen in [Figure 3] where the 7 largest and smallest returns on the stocks in the sample portfolio and SP-500 Index for 10 years are listed. These extreme observations are typically the most important for VaR analysis, however since these values are clearly discrete, the VaR will also be discrete, and hence be either underpredicted or overpredicted.

Figure 3. Extreme daily returns for select U.S. stocks and S&P 500, 1987-1996. Source: Danielsson and de Vries.

In other words, the quantile corresponding to the estimation of the Value-at-Risk […] rather depends on the realizations of the [portfolio returns] than on their probability distribution¹, so that the Value-at-Risk calculated with a quantile of the empirical distribution will be highly unstable, especially when considering a Value-at-Risk with a high confidence level with only few available data¹.

This is why VaR estimation is a very challenging statistical problem², sharing many similarities with the problem of estimating the frequency and/or severity of extreme events in other domains, like floods frequency estimation²¹ in hydrology.

In order to compute a statistical estimator of a portfolio Value-at-Risk, three main approaches exist:

Non-parametric approaches, that do not make any specific distributional assumptions on the portfolio return distribution and whose VaR estimators do not depend on any auxiliary parameter.
Semi-parametric approaches, that do not make any specific distributional assumptions on the portfolio return distribution but whose VaR estimators depend on one or several auxiliary parameters.
Parametric approaches, that make a specific distributional assumption on the portfolio return distribution and whose VaR estimators depend on one or several auxiliary parameters.

To be noted that non-parametric and semi-parametric approaches might still make distributional assumptions, in particular for convergence proofs - like assuming that returns are independent and identically distributed (i.i.d.) -, but these assumptions are then generic in nature, contrary to parametric approaches which assume a very specific return distribution, like a Gaussian distribution, which is one of the most widely applied parametric probability distribution²² in finance.

Non-parametric and semi-parametric Value-at-Risk estimation

Chen and Yong Tang²³ notes that non-parametric and semi-parametric VaR estimators have the advantages of (i) being free of distributional assumptions […] while being able to capture fat-tail and asymmetry distribution of returns automatically; and (ii) imposing much weaker assumptions on the dynamics of the return process and allowing data “speak for themselves”²³.

Empirical quantile of the portfolio return distribution

A well-known estimator of the $(1 - \alpha)$% quantile of any probability distribution is the empirical $(1 - \alpha)$% quantile of that distribution, which relies on order statistics.

In the context of VaR estimation, the underlying idea is explained in Dowd²⁴:

If we have a sample of $n$ profit and loss (P/L) observations, we can regard each observation as giving an estimate of VaR at an implied probability level. For example, if $n$ = 100, we can take the 5% VaR as the negative of the sixth²⁵ smallest P/L observation, the 1% VaR as the negative of the second-smallest, and so on.

This leads to the empirical portfolio VaR estimator, defined²⁶ as the opposite of the $n (1 - \alpha) + 1$-th highest portfolio return²⁷

\[\text{VaR}_{\alpha} = -r_{\left( n (1 - \alpha) + 1 \right)}\]

, where $r_{(1)} \leq r_{(2)} \leq … \leq r_{(n-1)} \leq r_{(n)}$ are the order statistics of the portfolio returns.

Now, due to the discrete nature of the portfolio return distribution, there is little chance that $n (1 - \alpha) + 1$ is an integer.

In that case, two²⁶ possible choices are:

Either to define the opposite of the $\lfloor n (1 - \alpha) \rfloor + 1$-th highest portfolio return²⁷ as the empirical portfolio VaR estimator²⁴
\[\text{VaR}_{\alpha} = - r_{\left( \lfloor n (1 - \alpha) \rfloor + 1 \right)}\]
Or to define a linear interpolation²⁸ between the opposite of the $\lfloor (n+1) \left( 1 - \alpha \right) \rfloor $-th and $ \lfloor (n+1) \left( 1 - \alpha \right) \rfloor + 1$-th highest portfolio returns²⁷ as the empirical portfolio VaR estimator²⁹³⁰¹³
\[\text{VaR}_{\alpha} = - \left( 1 - \gamma \right) r_{\left( \lfloor (n+1) \left( 1 - \alpha \right) \rfloor \right)} - \gamma r_{\left( \lfloor (n+1) \left( 1 - \alpha \right) \rfloor + 1 \right)}\]
, with $\gamma$ $= (n+1) \left( 1 - \alpha \right)$ $- \lfloor (n+1) \left( 1 - \alpha \right) \rfloor$.

An interesting property of the resulting portfolio VaR estimator is that it is consistent in the presence of weak dependence³¹ between portfolio returns, c.f. Chen and Yong Tang²³.

In terms of drawbacks, the two major limitations of the empirical portfolio VaR estimator are that:

It only takes into account a small part of the information contained in the [portfolio returns] distribution function¹ - that is, at most two returns - which is highly inefficient, especially when the number of portfolio returns is already relatively small.
It cannot generate any information about the tail of the return distribution beyond the smallest sample observation¹⁶, which might lead to severly underestimate the true risk of the portfolio.

Kernel-smoothed quantile of the portfolio return distribution

Another way to account for the information available in the empirical […] distribution¹ than using the empirical quantile estimator discussed in the previous sub-section is to use the kernel-smoothed quantile estimator introduced in Gourieroux et al.³².

Kernel-smoothing is a methodology belonging to statistics and probability theory that can be thought of as a way of generalizing a histogram constructed with the sample data¹⁶, as illustrated in Figure 4.

Figure 4. Histogram v.s. kernel-smoothed density for the same sample of data. Source: Wikipedia.

On Figure 4, where a histogram results in a density that is piecewise constant, a kernel[-smoothed} approximation results in a smooth density²³.

Coming back to the estimator of Gourieroux et al.³², it is defined as the $(1 - \alpha)$% quantile of a kernel-smoothed approximation of the portfolio return distribution, which essentially results in a weighted average of the order statistics around [$r_{\left( \lfloor n (1 - \alpha) \rfloor + 1 \right)}$] rather than […] a single order statistic²³ or a linear interpolation between two order statistics.

From a practical perspective, that VaR estimator is computed as follows:

Select a kernel function $K$, usually²³ taken as a symmetric³³ probability density function.

The theoretical optimal³⁴ choice for such a kernel function is the Epanechnikov kernel, defined as $ K(u) = \frac{3}{4} \left( 1 - u^2 \right) I_{|u| \leq 1}$.

However, the litterature suggests that the form of the kernel has little effect on the [accuracy] of the [kernel-smoothed approximation of the return distribution]³², mainly because:
- The theoretical framework used to establish the optimality of the Epanechnikov kernel relies on large asymptotics, moreover in a debatable way³⁵.
- The performances³⁶ of other commonly used kernel functions are anyway very close to those of the Epanechnikov kernel³⁷.
So, in applications, the most common²³³² choice for a kernel function is rather the Gaussian kernel, defined as $ K(u) = \frac{1}{\sqrt{2 \pi }} e^{- \frac{u^2}{2} }$.

As a side note, using a kernel function to approximate the portfolio return distribution might look like a parametric approach to VaR estimation in disguise, but Butler and Schachter¹⁶ explains why this is not the case:

Note that use of a normal or Gaussian kernel estimator does not make the ultimate estimation of the VaR parametric.

As the sample size grows, the net sum of all the smoothed points approaches the true [portfolio return distribution], whatever that may be, irrespective of the method of smoothing the data. This is because the influence of each point becomes arbitrarily small as the sample size grows, so the choice of kernel imposes no restrictions on the results.
Select a kernel bandwidth parameter $h > 0$ for the kernel function.

Gourieroux et al.³² describes that parameter as follows:

The bandwidth parameter controls the range of data points that will be used to estimate the distribution.

A small bandwidth results in a rough distribution that does not improve appreciably on the original data, while a large bandwidth over-smoothes the density curve and erases the underlying structure.

This latter point is illustrated in Figure 5 and Figure 6.

Figure 5. Influence of the bandwidth parameter on the kernel-smoothed approximation of a normal mixture distribution (dashed) from n = 1000 observations. Source: Adapted from Wand and Jones.

Figure 6. Dynamic influence of the bandwidth parameter on the kernel-smoothed approximation of a normal distribution. Source: KDEpy.

On Figure 5, it is clearly visible that:
- a) The estimate of the normal mixture distribution is very rough³⁷.
  
  This corresponds to a too small bandwidth parameter $h$ that undersmoothes the observations.
- b) The estimate of the normal mixture distribution smoothes away its bimodality structure.
  
  This corresponds to a too big bandwidth parameter $h$ that oversmoothes the observations.
- c) The estimate of the normal mixture distribution is not overly noisy, yet the essential structure of the underlying density has been recovered³⁷.
  
  This corresponds to an adequate bandwidth parameter $h$.
On Figure 6, the situation is the same as in Figure 5, except that the bandwidth parameter $h$ is being dynamically increased from 0 to ~20.

Figure 5 and Figure 6 empirically demonstrate that the choice of the bandwidth is of crucial importance³⁸, although a difficult task, especially when smoothing the tails of underlying distributions with possible data scarcity³⁸.

The interested reader is refered to Wand and Jones³⁷, Tsybakov³⁵ and Cheng and Sun³⁹ for the description of several methods to choose the optimal bandwidth for a kernel function.
Compute the kernel-smoothed portfolio VaR estimator as
\[\text{VaR}_{\alpha} = - \hat{F}^{-1}(1 - \alpha)\]
, where $ \hat{F}^{-1}(1 - \alpha) $ is the solution of the quantile equation
\[\hat{F}(x) = \int_{-\infty}^x \hat{f}(u) \, du = 1 - \alpha\]
with $ \hat{f}(x) = \frac{1}{n h} \sum_{i=1}^n K \left( \frac{x - r_i}{h} \right) $.

That part is typically done with a numerical algorithm, like the Gauss–Newton algorithm mentioned in Gourieroux et al.³².

Two important positive results on the kernel-smoothed VaR estimator are established in Chen and Yong Tang²³:

Theoretically, it is consistent in the presence of weak dependence between portfolio returns.
Empirically, it produces more precise estimates⁴⁰ than those obtained with the empirical VaR estimator - especially when the number of observations is small - which can translate to a large amount in financial terms²³.

Similar results - in a non-financial context - are reported in Cheng and Sun³⁹:

It turns out that kernel smoothed quantile estimators, with no matter which bandwidth selection method used, are more efficient than the empirical quantile estimator in most situations.

And when sample size is relatively small, kernel smoothed estimators are especially more efficient than the empirical quantile estimator.

In other words, the extra effort of smoothing pays of at the end²³!

The major limitation of the kernel-smoothed VaR estimator, though, is that if the selected kernel function does not reflect the tail features of the true portfolio return distribution, some problems may arise when the quantile to be estimated requires an extrapolation […] far beyond the range of observed data⁴¹.

In the words of Danielsson and de Vries²⁰:

Almost all kernels are estimated with the entire data set, with interior observations dominating the kernel estimation.

While even the most careful kernel estimation will provide good estimates for the interior, there is no reason to believe that the kernel will describe the tails adequately.

Tail bumpiness is a common problem in kernel estimation.

So, while the kernel-smoothed VaR estimator is capable of tail extrapolation - contrary to the empirical VaR estimator - that capability should be used with extreme caution⁴².

A proper portfolio VaR estimator when tail extrapolation is needed thus remains elusive at this stage.

Extrapolated empirical quantile of the portfolio return distribution

In order to solve the problem of tail extrapolation while retaining the simplicity of the empirical quantile estimator, Hutson³⁰ proposes to extend the linearly interpolated quantile function into a tail extrapolation quantile function³⁵ that allows for non-parametric extrapolation beyond the observed data³⁰.

In terms of VaR estimation, Hutson’s work translates into the following extrapolated empirical portfolio VaR estimator:

For $0 < \left( 1 - \alpha \right) \leq \frac{1}{n+1}$
\[\text{VaR}_{\alpha} = - r_{(1)} - \left( r_{(2)} - r_{(1)} \right) \log \left( (n+1) \left( 1 - \alpha \right) \right)\]
For $\left( 1 - \alpha \right) \in ]\frac{1}{n+1} , \frac{n}{n+1}[$, $ \text{VaR}_{\alpha} $ is defined as the standard empirical portfolio VaR estimator
For $\frac{n}{n+1} < \left( 1 - \alpha \right) < 1$
\[\text{VaR}_{\alpha} = - r_{(n)} + \left( r_{(n)} - r_{(n-1)} \right) \log \left( (n+1) \alpha \right)\]

Hutson³⁰ establishes the consistency of his quantile estimator for i.i.d. observations and empirically demonstrates using misc. theoretical distributions that it fits well to the ideal sample for all distributions for ideal samples as small as $n = 10$³⁰.

Unfortunately for financial applications, Hutson³⁰ also notes that his quantile estimator appears to be [unable] to completely capture the tail behavior of heavy-tailed⁴³ distributions such as Cauchy³⁰.

This is confirmed by Banfi et al.⁴¹, who finds that Hutson’s method provides competitive results when light-tailed distributions are of interest⁴¹ but generates large biases in the case of heavy-tailed distributions⁴¹.

Extreme value theory-based quantile of the portfolio return distribution

Another possibility to improve [the] tail extrapolation⁴¹ properties of the empirical quantile estimator is to rely on extreme value theory (EVT), which is a domain of statistics concerned with the study of the asymptotical distribution of extreme events, that is to say events which are rare in frequency and huge with respect to the majority of observations⁴⁴.

Indeed, because VaR only deals with extreme quantiles of the distribution⁴⁴, EVT sounds like a natural framework for providing more reliable VaR estimates than the usual ones, given that [it] directly concentrates on the tails of the distribution, thus avoiding a major flaw of [other] approaches whose estimates are somehow biased by the credit they give to the central part of the distribution, thus underestimating extremes and outliers, which are exactly what one is interested in when calculating VaR⁴⁴.

Two preliminary remarks before proceeding:

One notational

The EVT litterature⁴⁵ typically focuses on the upper tail of distributions, so that the portfolio returns $r_1,…,r_n$ need to be replaced by their opposites $r^{‘}_1 = -r_1$,…,$r^{‘}_n=-r_n$.
One important for applying EVT results in finance

Most existing EVT methods require [i.i.d.] observations, whereas financial time series exhibit obvious serial dependence features such as volatility clustering⁴⁴.

It turns out that this issue has been addressed in works dealing with weak serial dependence [and] the main message from these studies is that usual EVT methods are still valid⁴⁴.

For example, the Hill estimator discussed below has originally been derived under the assumptions of i.i.d. observations⁴⁶, but it has also been proved to be usable with weakly dependent observations⁴⁷; the same applies⁴⁸ to the Weissman quantile estimator also discussed below.

In addition, even though the [EVT] estimators obtained may be less accurate and neglecting this fact could lead to inadequate resolutions in order to cope with the risk of occurrence of extreme events⁴⁴, they are consistent and unbiased in the presence of higher moment dependence¹⁴ and it is even possible to explicitly model extreme dependence using the [notion of] extremal index¹⁴.

With these remarks in mind, EVT offers two main methods¹⁴ to model the upper tail of the portfolio return distribution $r^{‘}_1$,…,$r^{‘}_n$ and compute an EVT-based portfolio VaR estimator:

A fully parametric method based on the generalized Pareto distribution (GPD)
A semi-parametric method based on the Hill (or similar) estimator

Generalized Pareto distribution-based method

This method consists in fitting a generalized Pareto distribution to the upper tail of the portfolio return cumulative distribution function (c.d.f.) and computing its $\alpha$% quantile.

This is for example done in the seminal paper of McNeil and Frey⁴⁹, in which such a distribution is fitted to the returns⁵⁰ of different financial instruments.

The theoretical justification for this method lies in the Pickands-Balkema-De Hann’s theorem, which states that EVT holds sufficiently far out in the tails such that we can obtain the distribution not only of the maxima but also of other extremely large observations¹⁴.

In practice, fitting a GPD to the upper tail of the portfolio return distribution is a two-step process:

First, a threshold $r^{‘}_{(n - k)}, k \geq 1$ beyond which returns should be considered as belonging to the upper tail needs to be selected.

This threshold corresponds to the location parameter $u \in \mathbb{R}$ of the GPD.

Unfortunately, the choice of how many of the $k$-largest observations should be considered extreme is not straightforward⁴¹ and is actually a central issue to any application of EVT⁴⁴.

Indeed, as detailled in de Haan et al.⁵¹:

Theoretically, the statistical properties of EVT-based estimators are established for $k$ such that $k \to \infty$ and $k/n \to 0$ as $n \to \infty$.

In applications with a finite sample size, it is necessary to investigate how to choose the number of high observations used in estimation.

For financial practitioners, two difficulties arise: firstly, there is no straightforward procedure for the selection; secondly, the performance of the EVT estimators is rather sensitive to this choice.

More specifically, there is a bias–variance tradeoff: with a low level of $k$, the estimation variance is at a high level which may not be acceptable for the application; by increasing $k$, i.e., using progressively more data, the variance is reduced, but at the cost of an increasing bias.

The literature offers some guidance on how to choose an adequate cut-off between the central part and the upper tail of the return distribution but that choice remains notoriously difficult in general, c.f. Benito et al.⁵² for a review.

Fortunately, in the specific context of VaR estimation, there is a large set of thresholds that provide similar GPD quantiles estimators and as a consequence similar market risk measures⁵³ - from about the 80th percentile of observations to the 95th percentile of observations⁵³ - so that the researchers and practitioners should not focus excessively on the threshold choice⁵³.

Figure 7 and Figure 8 illustrate this point with a GPD fitted to the lower tail of daily percentage returns of Deutsche mark/British pound (DEM/GBP) exchange rates from 3rd January 1984 to 31st December 1991⁵⁴, using two different thresholds:
- Figure 7 - A threshold $u \approx -1.2292$, corresponding to ~2% of the observations.
  
  Figure 7. Lower tail of daily percentage returns of Deutsche mark/British pound (DEM/GBP) exchange rates, GDP fit with threshold u = -1.2292, 3rd January 1984 to 31st December 1991.
- Figure 8 - A threshold $u \approx -0.2683$, corresponding to ~20% of the observations.
  
  Figure 8. Lower tail of daily percentage returns of Deutsche mark/British pound (DEM/GBP) exchange rates, GDP fit with threshold u = -0.2683, 3rd January 1984 to 31st December 1991.
From Figure 7 and Figure 8, both thresholds lead to an equally good extreme lower tail GPD fit, which is confirmed numerically by goodness-of-fit measures.

Consequently, in order to keep the threshold selection step as simple as possible for VaR estimation, the early suggestion of DuMouchel⁵⁵ to use the 90th percentile of observations seems a very good starting point.
Second, the shape parameter $\xi \in \mathbb{R}$ and the scale parameter $\sigma > 0$ of the GPD need to be estimated.

This is usually done through likelihood maximization, but other procedures are described in the litterature (method of moments⁵⁶, maximization of goodness-of-fit estimators⁵⁷, etc.).

Once the parameters of the GPD have been determined, the EVT/GPD-based portfolio VaR estimator⁴⁹ is defined as the $\alpha$% quantile of that GPD through the formula

\[\text{VaR}^{\text{GPD}}_{\alpha} = \begin{cases} r^{'}_{(n - k)} + \frac{\hat{\sigma}}{\hat{\xi}} \left( \left( \frac{k}{T ( 1 - \alpha)} \right)^\xi -1 \right) &\text{if } \hat{\xi} \ne 0 \\ r^{'}_{(n - k)} - \hat{\sigma} \ln \frac{T ( 1 - \alpha)}{k} &\text{if } \hat{\xi} = 0 \\ \end{cases}\]

, where:

$\hat{\xi}$ is an estimator of the shape parameter $\xi$ of the GPD.
$\hat{\sigma}$ is an estimator of the scale parameter $\sigma$ of the GPD.

Figure 9, taken from Danielsson ¹⁴, illustrates the near perfect fit that can be obtained when such a method is applied to the upper and lower tails of the daily S&P 500 returns over the period 1970-2009.

Figure 9. Upper and lower tails of daily S&P 500 returns fitted with an EVT-estimated distribution v.s. a normal distribution, 1970-2009. Source: Danielsson.

Hill estimator-based method

Under the assumption that the portfolio return distribution belongs⁵⁸ to the generic family of heavy-tailed distributions, this method consists in deriving an asymptotic estimator of its $\alpha$% quantile.

This is for example done in Danielsson and de Vries⁵⁹ and in Drees⁴⁸.

This method is justified by stylized facts⁶⁰ of asset returns, as explained in Danielsson and de Vries²⁰:

[…] because we know that financial return data are heavy tailed distributed, one can rely on a limit expansion for the tail behavior that is shared by all heavy tailed distributions. The importance of the central limit law for extremes is similar to the importance of the central limit law, i.e. one does not have to choose a particular parametric distribution.

Under that generic assumption, it can be demonstrated⁶¹ that the upper tail of the portfolio returns decays as a power function multiplied by a slowly varying function, that is

\[1 - F(x) = x^{-\gamma} L(x), x > 0\]

, where:

$F$ is the c.d.f. of the (opposite of the) portfolio returns.
$\gamma = \frac{1}{\xi} > 0$ is the tail index⁶² of $F$, with $\xi$ the same shape parameter as in the GPD method.
$L$ is a slowly varying function in a sense defined in Rocco⁴⁴.

From this asymptotic behaviour, the EVT/Weissman-based portfolio VaR estimator is defined as the $\alpha$% Weissman⁶³ quantile estimator of an heavy-tailed distribution through the formula

\[\text{VaR}^{\text{WM}}_{\alpha} = r^{'}_{(n - k)} \left( \frac{k}{n \left( 1 - \alpha \right) } \right)^{\frac{1}{\hat{\gamma}}}\]

, where⁶⁴:

$\hat{\gamma}$ is an estimator of the tail index $\gamma$.

The most frequently employed estimator of the tail index is by far the Hill estimator⁴⁴ introduced in Hill⁴⁶ and defined conditionally on $k$ as
\[\hat{\gamma} = \frac{1}{k} \sum_{i=1}^k \ln \frac{r^{'}_{(n - i + 1)}}{r^{'}_{(n - k)}}\]
The interested reader is referred to Fedotenkov⁶⁵, in which more than one hundred tail index estimators proposed in the literature are reviewed.
$k$ is the number of observations $r^{‘}_{(n - k + 1)}$, …, $r^{‘}_{(n)}$ that should be considered extreme.

Here, and contrary to the GPD method, the threshold index $k$ has a huge influence on the Weissman quantile estimator and thus on the EVT/Weissman-based VaR estimator.

As an illustration, Figure 10 depicts the daily VaR of the Dow Jones Industrial Average index at a confidence level $\alpha = 99.9$%⁶⁶ as a function of $k$, when estimated by the EVT/Weissman-based VaR estimator.

Figure 10. Impact of the threshold index k between the central part and the upper tail of the Dow Jones Industrial Average daily return distribution, Weissman quantile estimator of VaR 99.9%, 1980-2010. Source: de Haan et al.

On Figure 10, three distinct ranges of values for the index $k$ are visible:
- An initial range of values for $k \in [1, 150]$ that results in a bumpy VaR estimate, although on this specific figure the bumpyness is not that pronounced.
  
  This is the “high variance” region of the estimator.
- A second range of values for $k \in [150, 300]$ that results in a flat-ish VaR estimate.
  
  This is the “optimal bias-variance” region of the estimator, in which the exact value of $k$ is not important.
  
  Obtaining an estimate of $k$ belonging to that region is the ultimate goal.
- A third range of values for $k \geq 300$ that results in a diverging VaR estimate.
  
  This is the “high bias” region of the estimator.
Hopefully, this problem can be solved by using a bias-corrected Weissman quantile estimator.

For example, Figure 11 depicts the daily VaR of the Dow Jones Industrial Average index at a confidence level $\alpha = 99.9$% as a function of $k$, when estimated by the EVT/unbiased Weissman-based VaR estimator introduced in de Haan et al.⁵¹.

Figure 11. Impact of the threshold index k between the central part and the upper tail of the Dow Jones Industrial Average daily return distribution, unbiased Weissman quantile estimator of VaR 99.9%, 1980-2010. Source: de Haan et al.

Comparing Figure 11 to Figure 10, the improvement in stability of the quantile estimator is striking.

One important limitation of this method compared to the GPD-based method is that depending on the exact financial data used (financial instrument, time period, returns frequency…), the stylized heavy tailedness nature of return distributions might be violated, with return distributions being thin-tailed or short-tailed⁶⁷ instead of heavy-tailed, c.f. Longin and Solnik⁶⁷ and Drees⁴⁸.

Indicentally, the two lower tail GPD fits depicted in Figure 7 and Figure 8 both correspond to a short-tailed distribution since $\hat{\xi} \approx -0.2304$ on Figure 7 and $\hat{\xi} \approx -0.021$ on Figure 8.

Practical performances

In terms of practical performances, Danielsson¹⁴ highlights that EVT-based VaR estimation delivers good probability–quantile estimates where EVT holds¹⁴, but that there are no rules that tell us when [EVT] becomes inaccurate¹⁴.

Indeed, it depends on the underlying distribution of the data. In some cases, it may be accurate up to 1% or even 5%, while in other cases it is not reliable even up to 0.1%¹⁴.

In addition, the accuracy of EVT also depends on the selected threshold.

This is illustrated in Figure 12, which is identical to Figure 7 except that the GPD fit has been graphically extended to the right of the threshold $u \approx -1.2292$.

Figure 12. Lower tail of daily percentage returns of Deutsche mark/British pound (DEM/GBP) exchange rates, GDP fit with threshold u = -1.2292 extended, 3rd January 1984 to 31st December 1991.

From Figure 7 and Figure 8, both GPT fits seem usable below $u \approx -1.2292$.

But while the GPD fit of Figure 8 is valid up to $u \approx -0.2683$, Figure 12 makes it clear that the GPD fit of Figure 7 is completely off above $u \approx -1.2292$.

Other estimators

It is out of scope of this blog post to list all non-parametric and semi-parametric portfolio VaR estimators discussed in the litterature, but I would like to finish this section by mentionning estimators based on smoothing splines.

An example of such an estimator is described in Shaker-Akhtekhane and Poorabbas⁶⁸, in which it is empirically demonstrated to outperform common historical, parametric, and kernel-based methods⁶⁸ when applied to the S&P500 index at VaR confidence levels of $\alpha = 95$% and $\alpha = 99$%.

Misc. points of attention for such estimators:

Monotonicity constraints need to be imposed on smoothing splines in order to properly approximate the c.d.f. of the portfolio returns, as highlighted in Wood⁶⁹.

These are not mentioned in Shaker-Akhtekhane and Poorabbas⁶⁸, but are required in practice.
Similar to kernel-smoothing, a smoothing parameter needs to be selected.

The same kind of problems arise, with solutions that are very close in spirit like generalized cross-validation, c.f. Wood⁶⁹.
Again similar to kernel-smoothing, special care must be taken when extrapolating beyond the range of observed values.

In particular, the estimator described in Shaker-Akhtekhane and Poorabbas⁶⁸ cannot extrapolate beyond the smallest and the highest portfolio returns due to the splines constraints associated with these two points.

Parametric Value-at-Risk estimation

Contrary to the non-parametric and semi-parametric approaches, parametric - also called analytical - approaches make the assumption⁷⁰ that the whole portfolio return distribution can be described by a parametric distribution $F_{\theta}$ whose parameters $\theta$ need to be estimated from observations.

The $(1 - \alpha)$% quantile of the portfolio return distribution is then simply obtained by inverting that parametric distribution, which leads to the definition of the parametric portfolio VaR estimator as

\[\text{VaR}_{\alpha} = - F_{\theta}^{-1} (1 - \alpha)\]

Parametric approaches thus replace the problem of accurately computing the quantile of an empirical distribution of observations by the problem of choosing a parametric distribution that best fits these observations.

A couple of examples:

If the portfolio returns are assumed to be distributed according to a Gaussian distribution, the associated portfolio VaR is called Gaussian Value-at-Risk (GVaR) and is computed through the formula⁷¹
\[\text{GVaR}_{\alpha} (X) = - \mu - \sigma z_{1 - \alpha}\]
, where:
- The location parameter $\mu$ and the scale parameter $\sigma$ are usually⁷² estimated by their sample counterparts.
- $z_{1 - \alpha}$ is the $1 - \alpha$ quantile of the standard normal distribution.
This is the assumption made in the RiskMetrics model⁷³.
If the portfolio returns are assumed to be distributed according to an heavy-tailed distribution, several distributions can be used:
- A Cornish-Fisher distribution, whose associated portfolio VaR is known as Modified Value-at-Risk
- A Gaussian mixture distribution
- …
  
  Here, as a side note, even though it is known that financial time series usually exhibit skewed and fat-tailed distributions⁶⁰, there is no complete agreement on what distribution could fit them best⁴⁴, so that finding the best distribution to use is as much art as science…

Parametric VaR estimation approaches might be very successful, especially with long risk horizons (months, years…) and/or with not too extreme confidence levels (90%, 95%…).

Nevertheless, they might also not be able to provide an adequate description of the whole range of data, resulting in a good fit of the body but a non accurate description of the tails⁴¹, leading to biases in VaR estimates.

Implementation in Portfolio Optimizer

Portfolio Optimizer supports different portfolio Value-at-Risk estimation methods, c.f. the documentation:

Empirical portfolio VaR estimation
Extrapolated empirical portfolio VaR estimation
Kernel-smoothed empirical portfolio VaR estimation

For that estimation method, the kernel bandwidth parameter $h$ is automatically computed using a proprietary variation of the improved Sheather and Jones rule described in Botev et al.⁷⁴
EVT-based portfolio VaR estimation (both GPD-based and Hill estimator-based)

For these estimation methods:
- The number of extreme observations $k$ is automatically computed using a proprietary variation of the goodness-of-fit procedure described in El-Aroui and Diebolt⁷⁵.
- The estimated GPD parameters are unbiased through the formulas described in Giles et al.⁷⁶.
- The estimated Hill and Weissman estimators are unbiased through the formulas described in de Haan et al.⁵¹ and corrected in Chavez-Demoulin and Guillou⁷⁷.
Parametric portfolio VaR estimation

The supported parametric distributions are:
- The Gaussian distribution
- The Gaussian mixture distribution
- The Cornish-Fisher distribution

Conclusion

This blog post described some of the most well-known methods for univariate Value-at-Risk estimation.

Thanks to these, it is possible to analyze the past behaviour of a financial portfolio, but their real interest lies in univariate Value-at-Risk forecasting, which will be the subject of the next blog post in this series.

Stay tuned!

Meanwhile, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Gueant, O., Computing the Value at Risk of a Portfolio: Academic literature and Practionners’ response, EMMA, Working Paper. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
See Manganelli, Simone and Engle, Robert F., Value at Risk Models in Finance (August 2001). ECB Working Paper No. 75. ↩ ↩² ↩³ ↩⁴
Basel II requires⁷⁸ to calculate market risk capital requirements using VaR at a 99% confidence level over a 10-day horizon. ↩
Basel III requires⁷⁹ internal backtesting procedures based on VaR and Stress VaR (VaR applied to a market stress period). ↩
The SEC Rule 18f-4 requires companies to calculate daily VaR at a 99% confidence level over a 20-day horizon and using at least 3 years of historical data; it also requires companies to backtest their VaR models daily over a 1-day horizon. ↩
See Stoyan V. Stoyanov, Svetlozar T. Rachev, Frank J. Fabozzi, Sensitivity of portfolio VaR and CVaR to portfolio return characteristics, Working paper. ↩
C.f. Dowd²⁴, the VaR is the negative of the relevant P/L observation because P/L is positive for profitable outcomes and negative for losses, and the VaR is the maximum likely loss (rather than profit) at the specified probability²⁴, so that VaR is a positive percentage; to be noted, though, that VaR can be negative when no loss is incurred within the confidence level, in which case it is meaningless; c.f. Daníelsson¹⁴. ↩
This is the case when the portfolio return cumulative distribution function is strictly increasing and continuous; otherwise, a similar formula is still valid, with $F_X^{-1}$ the generalized inverse distribution function of $X$, but these subtleties - important in mathematical proofs and in numerical implementations - are out of scope of this blog post. ↩
See Jorion, P. (2007). Value at risk: The new benchmark for managing financial risk. New York, NY: McGraw-Hill. ↩ ↩² ↩³
See Keith Kuester, Stefan Mittnik, Marc S. Paolella, Value-at-Risk Prediction: A Comparison of Alternative Strategies, Journal of Financial Econometrics, Volume 4, Issue 1, Winter 2006, Pages 53–89. ↩
Also called top-down VaR models¹³ or portfolio aggregation-based models. ↩
See Zangari, Peter, 1997, Streamlining the market risk measurement process, RiskMetrics Monitor, 1, 29–35. ↩ ↩² ↩³
See Ballotta, L. ORCID: 0000-0002-2059-6281 and Fusai, G. ORCID: 0000-0001-9215-2586 (2017). A Gentle Introduction to Value at Risk. ↩ ↩² ↩³ ↩⁴ ↩⁵
See Jon Danielsson, Financial Risk Forecasting: The Theory and Practice of Forecasting Market Risk, with Implementation in R and Matlab, Wiley 2011. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹
Danielsson¹⁴ comes to my mind. ↩
See J. S. Butler & Barry Schachter, 1996. Improving Value-At-Risk Estimates By Combining Kernel Estimation With Historical Simulation, Finance 9605001, University Library of Munich, Germany. ↩ ↩² ↩³ ↩⁴ ↩⁵
That report was required by 4.15pm and originally became known as the 4.15 report. ↩
See Glyn A. Holton, (2002), History of Value-at-Risk: 1922-1998, Method and Hist of Econ Thought, University Library of Munich, Germany. ↩ ↩²
The two horizons might be different, but this is out of scope of this blog post. ↩
See Danielsson, Jon, and Casper G. De Vries. Value-at-Risk and Extreme Returns. Annales d’Économie et de Statistique, no. 60, 2000, pp. 239–70. ↩ ↩² ↩³
See Lall, U., Y. Moon, and K. Bosworth (1993), Kernel flood frequency estimators: Bandwidth selection and kernel choice, Water Resour. Res.,29(4), 1003–1015. ↩
See RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition. ↩
See Song Xi Chen, Cheng Yong Tang, Nonparametric Inference of Value-at-Risk for Dependent Financial Returns, Journal of Financial Econometrics, Volume 3, Issue 2, Spring 2005, Pages 227–255. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰
See Kevin Dowd, Estimating VaR with Order Statistics, The Journal of Derivatives Spring 2001, 8 (3) 23-30 ↩ ↩² ↩³ ↩⁴
To be noted that Dowd²⁴ proposes to take the sixth observation as [the] 5% VaR because we want 5% of the probability mass to lie to the left of [the] VaR²⁴, but other auhors propose to use the fifth observation instead¹³²⁹. ↩
Many other choices - at least 9 - are possible though, c.f. Hyndman and Fan⁸⁰; ultimately, what is needed is an estimator of the $(1 - \alpha)$% quantile of the empirical portfolio return distribution. ↩ ↩²
For $ 1 - \alpha \in ]\frac{1}{n+1} , \frac{n}{n+1}[$, that is, no extrapolation is possible. ↩ ↩² ↩³
Such a linearly interpolated quantile estimator has been known since at least Parzen⁸¹. ↩
See Peter Hall and Andrew Rieck. (2001). Improving Coverage Accuracy of Nonparametric Prediction Intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 63(4), 717–725. ↩ ↩²
See Hutson, A.D. A Semi-Parametric Quantile Function Estimator for Use in Bootstrap Estimation Procedures. Statistics and Computing 12, 331–338 (2002). ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
To be noted that weak dependence is a kind of misnomer, because this type of dependence actually covers a very broad range of time series models! ↩
See Gourieroux, C., Scaillet, O. and Laurent, J.P. (2000). Sensitivity analysis of Values at Risk. Journal of Empirical Finance, 7, 225-245. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Asymmetric kernel functions also exist, c.f. for example Abadir and Lawford⁸². ↩
In a mean squared-error sense. ↩
See Alexandre B. Tsybakov. 2008. Introduction to Nonparametric Estimation (1st. ed.). Springer Publishing Company, Incorporated. ↩ ↩² ↩³
The performance of a kernel estimator $K$ is defined in terms of the ratio of sample sizes necessary to obtain the same minimum asymptotic mean integrated squared error (for a given [function $f$ that is being kernel-smoothed] when using $K$ as when using [the Epanechnikov kernel]³⁷. ↩
See Wand, M.P., & Jones, M.C. (1994). Kernel Smoothing (1st ed.). Chapman and Hall/CRC. ↩ ↩² ↩³ ↩⁴ ↩⁵
See Keming Yu & Abdallah K. Ally & Shanchao Yang & David J. Hand, Kernel quantile based estimation of expected shortfall, Journal of Risk, Journal of Risk. ↩ ↩²
See Cheng, M.-Y. and S. Sun (2006). Bandwidth selection for kernel quantile estimation. Journal of the Chinese Statistical Association 44 (3), 271–295. ↩ ↩²
Yet, Chen and Yong Tang²³ notes that the reduction in RMSE is not very large for large samples²³, confirming the theoretical results that the reduction is of second order only²³. ↩
See Banfi, F., Cazzaniga, G. & De Michele, C. Nonparametric extrapolation of extreme quantiles: a comparison study. Stoch Environ Res Risk Assess 36, 1579–1596 (2022). ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Potential solutions to this problem might be to use 1) a data-driven mixture of a Gaussian and a Cauchy kernel as proposed in Banfi et al.⁴¹ or 2) a preliminary transformation of the data with a Champernowne distribution as proposed in Buch-Kromann et al.⁸³. ↩
The family of heavy-tailed distributions encompasses all the fat-tailed distributions enocuntered in finance like the Student-t distribution and more. ↩
See Rocco, Marco, Extreme Value Theory for Finance: A Survey (February 3, 2012). Bank of Italy Occasional Paper No. 99. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰
See Danielsson, J., de Vries, C., 1997. Tail index and quantile estimation with very high frequency data. Journal of Empirical Finance 4, 241–257. ↩
See Hill, B.M. (1975) A Simple General Approach to Inference About the Tail of a Distribution. Annals of Statistics, 3, 1163-1174. ↩ ↩²
See B. Karima and B. Youcef, Asymptotic Normality of Hill’s Estimator under Weak Dependence, Statistical Methodologies. IntechOpen, Feb. 26, 2020. ↩
See Drees, Holger. Extreme Quantile Estimation for Dependent Data, with Applications to Finance. Bernoulli, vol. 9, no. 4, 2003, pp. 617–57. ↩ ↩² ↩³
See Alexander J. McNeil, Rudiger Frey, Estimation of tail-related risk measures for heteroscedastic financial time series: an extreme value approach, Journal of Empirical Finance, Volume 7, Issues 3–4, 2000, Pages 271-300. ↩ ↩²
More precisely, to the standardized residuals of an AR(1)-GARCH(1,1) model of these asset returns; the rationale is that excesses over threshold should be more justified with AR(1)-GARCH(1,1) standardized residuals than with raw asset returns. ↩
See de Haan, L., Mercadier, C. & Zhou, C. Adapting extreme value statistics to financial time series: dealing with bias and serial dependence. Finance Stoch 20, 321–354 (2016). ↩ ↩² ↩³
See Benito S, Lopez-Martín C, Navarro MA. Assessing the importance of the choice threshold in quantifying market risk under the POT approach (EVT). Risk Manag. 2023;25(1). ↩
See Benito, S., Lopez-Martín, C. & Navarro, M.A. Assessing the importance of the choice threshold in quantifying market risk under the POT approach (EVT). Risk Manag 25, 6 (2023). ↩ ↩² ↩³
See Bollerslev, T., and Ghysels, E. (1996). Periodic Autoregressive Conditional Heteroskedasticity. Journal of Business & Economic Statistics, 14, 139–151. ↩
See William H. DuMouchel. “Estimating the Stable Index α in Order to Measure Tail Thickness: A Critique.” Ann. Statist. 11 (4) 1019 - 1031, December, 1983. ↩
See J. R. M. Hosking and J. R. Wallis, Parameter and Quantile Estimation for the Generalized Pareto Distribution, Technometrics, Vol. 29, No. 3 (Aug., 1987), pp. 339-349. ↩
See Alberto Luceno, Fitting the generalized Pareto distribution to data using maximum goodness-of-fit estimators, Computational Statistics & Data Analysis, Volume 51, Issue 2, 2006, Pages 904-917. ↩
In EVT terms, the true portfolio return distribution is assumed to be in the Fréchet domain of attraction. ↩
See Danielsson J., de Vries C. G. (1997). Beyond the Sample: Extreme Quantile and Probability Estimation, Mimeo, Tinbergen Institute Rotterdam. ↩
See R. Cont (2001) Empirical properties of asset returns: stylized facts and statistical issues, Quantitative Finance, 1:2, 223-236. ↩ ↩²
The cumulative distribution function of the portfolio returns is in the maximum domain of attraction of a Fréchet-type extreme value distribution if and only if $1 - F$ has that form, c.f. Rocco⁴⁴. ↩
The tail index is also known as the extreme value index. ↩
See Weissman, I.: Estimation of parameters and large quantiles based on the k largest observations. J. Am. Stat. Assoc. 73, 812–815 (1978). ↩
The formula for the EVT/Weissman-based portfolio VaR estimator is the one in Nieto and Ruiz⁸⁴ and is slightly different from the one in Danielsson and de Vries⁵⁹, c.f. Danielsson¹⁴ in which a general threshold $u$ is used instead of $r^’_{(\hat{k})}$ or $r^’_{(\hat{k}+1)}$. ↩
See Fedotenkov, I. (2020). A Review of More than One Hundred Pareto-Tail Index Estimators. Statistica, 80(3), 245–299. ↩
As required under Basel III framework⁸⁵. ↩
See Longin, F.M., and B. Solnik (1997). Correlation structure of international equity markets during extremely volatile periods. Working Paper 97-039, ESSEC, Cergy-Pontoise, France. ↩ ↩²
See Saeed Shaker-Akhtekhane & Solmaz Poorabbas, 2023. Value-at-Risk Estimation Using an Interpolated Distribution of Financial Returns Series, Journal of Applied Finance & Banking, SCIENPRESS Ltd, vol. 13(1), pages 1-6. ↩ ↩² ↩³ ↩⁴
See Wood, S. N., Monotonic Smoothing Splines Fitted by Cross Validation, 1994, SIAM Journal on Scientific Computing, 1126-1133, 15, 5. ↩ ↩²
In addition, with multivariate VaR models, parametric methods also needs to us[e] approximations of the pricing formulas of each [non simple] asset in the portfolio¹, leading to methods like Delta-Normal or Delta-Gamma-(Theta-)Normal based on Taylor expansions of the assets pricing formulas, c.f. Gueant¹. ↩
See Boudt, Kris and Peterson, Brian G. and Croux, Christophe, Estimation and Decomposition of Downside Risk for Portfolios with Non-Normal Returns (October 31, 2007). Journal of Risk, Vol. 11, No. 2, pp. 79-103, 2008. ↩
See Martin, R. Douglas and Arora, Rohit, Inefficiency of Modified VaR and ES. ↩
Gueant¹ notes that the RiskMetrics model for the distribution of the evolution of the risk factors is based on the assumption that log-returns of prices (or variations in the case of interest rates) are independent across time and normally distributed, when appropriately scaled by an appropriate measure of volatility¹. ↩
See Z. I. Botev. J. F. Grotowski. D. P. Kroese. “Kernel density estimation via diffusion.” Ann. Statist. 38 (5) 2916 - 2957, October 2010. ↩
See Mhamed-Ali El-Aroui, Jean Diebolt, On the use of the peaks over thresholds method for estimating out-of-sample quantiles, Computational Statistics & Data Analysis, Volume 39, Issue 4, 2002, Pages 453-475. ↩
See David E. Giles, Hui Feng & Ryan T. Godwin (2016) Bias-corrected maximum likelihood estimation of the parameters of the generalized Pareto distribution, Communications in Statistics - Theory and Methods, 45:8, 2465-2483. ↩
See Valerie Chavez-Demoulin, Armelle Guillou, Extreme quantile estimation for beta-mixing time series and applications, Insurance: Mathematics and Economics, Volume 83, 2018, Pages 59-74. ↩
See Basel Committee on Banking Supervision. Revisions to the Basel II market risk framework (Updated as of 31 December 2010). 2011. ↩
See Basel Committee on Banking Supervision. Fundamental review of the trading book.. ↩
See Hyndman, R. J., & Fan, Y. (1996). Sample Quantiles in Statistical Packages. The American Statistician, 50(4), 361–365. ↩
See Parzen E (1979), Nonparametric statistical data modeling, J Am Stat Assoc 74(365):105–121. ↩
See Karim M Abadir, Steve Lawford, Optimal asymmetric kernels, Economics Letters, Volume 83, Issue 1, 2004, Pages 61-68. ↩
See Buch-Kromann, Tine and Nielsen, Jens Perch and Guillen, Montserrat and Bolancé, Catalina, Kernel Density Estimation for Heavy-Tailed Distributions Using the Champernowne Transformation (January 2005). ↩
See Maria Rosa Nieto, Esther Ruiz, Frontiers in VaR forecasting and backtesting, International Journal of Forecasting, Volume 32, Issue 2, 2016, Pages 475-501. ↩
Basel III requires⁷⁹ that the VaR measures for risks on both trading and banking books must be calculated at a 99.9% confidence level. ↩

Supervised Portfolios: A Supervised Machine Learning Approach to Portfolio Optimization

2025-06-07T00:00:00-05:00

Standard portfolio allocation algorithms like Markowitz mean-variance optimization or Choueffati diversification ratio optimization usually take in input asset information (expected returns, estimated covariance matrix…) as well investor constraints and preferences (maximum asset weights, risk aversion…) to produce in output portfolio weights satisfying a selected mathematical objective like the maximization of the portfolio Sharpe ratio or Diversification ratio.

Chevalier et al.¹ introduces a non-standard portfolio allocation framework - represented in Figure 1 - under which the same input is first used to “learn” in-sample optimized portfolio weights in a supervised training phase and then used to produce out-of-sample optimized portfolio weights in an inference phase.

Figure 1. Standard v.s. supervised portfolio allocation framework. Source: Adapted from Chevalier et al.

In this blog post, I will provide some details about that framework when used with the $k$-nearest neighbors supervised machine learning algorithm, which is an idea originally proposed in Varadi and Teed²³.

As an example of usage, I will compare the performances of a $k$-nearest neighbors supervised portfolio with those of a “direct” mean-variance portfolio in the context of a monthly tactical asset allocation strategy for a 2-asset class portfolio made of U.S. equities and U.S. Treasury bonds.

Mathematical preliminaries

Supervised machine learning algorithms

Let $\left( X_1, Y_1 \right)$, …, $\left( X_n, Y_n \right)$ be $n$ pairs of data points in⁴ $\mathbb{R}^m \times \mathbb{R}$, $m \geq 1$⁵, where:

Each data point $X_1, …, X_n$ represents an object - like the pixels of an image - and is called a feature vector
Each data point $Y_1,…,Y_n$ represents a characteristic of its associated object - like what kind of animal is depicted in an image (discrete characteristic) or the angle of the rotation between a rotated image and its original version (continuous characteristic) - and is called a label

Given a feature vector $x \in \mathbb{R}^m$, the aim of a supervised machine learning algorithm is then to estimate the “most appropriate” label associated to $x$ - $\hat{y} \in \mathbb{R}$ - thanks to the information contained in the training dataset $\left( X_1, Y_1 \right)$, …, $\left( X_n, Y_n \right)$.

$k$-nearest neighbors regression algorithm

Let $d$ be a distance metric⁶ on $\mathbb{R}^m$, like the standard Euclidean distance.

The $k$-nearest neighbor ($k$-NN) regression algorithm is an early⁷ [supervised] machine learning algorithm⁸ that uses the “neighborhood” of a feature vector in order to estimate its label.

In more details, let $\left( X_{(i)}(x), Y_{(i)}(x) \right)$, $i=1..n$ denotes the $i$-th training data point closest to $x$ among all the training data points $\left( X_1, Y_1 \right)$, …, $\left( X_n, Y_n \right)$ such that the distance of each training data point to $x$ satisfies $d \left(x, X_{(1)}(x) \right)$ $\leq … \leq$ $d \left(x, X_{(n)}(x) \right)$.

By definition, the $k$-NN estimate for the label associated to $x$ is then⁹ the uniformly or non-uniformly weighted average label of the $k \in \{ 1,…,n \}$ nearest neighbors $Y_{(1)}(x)$,…,$Y_{(j)}(x)$

\[\hat{y} = \frac{1}{k} \sum_{i=1}^k Y_{(i)}(x)\]

\[\hat{y} = \sum_{i=1}^k w_i Y_{(i)}(x)\]

, where $w_i \geq 0$ is the weight associated to the $i$-th nearest neighbor $Y_{(i)}(x)$ and all the weights $w_i$, $i=1..k$ sum to one, that is, $\sum_{j=1}^k w_k$.

For illustration purposes, the process of selecting the 2 nearest neighbors $X_{(1)}(x)$ and $X_{(2)}(x)$ of a data point $x$ in $\mathbb{R}^2$ is outlined in Figure 2.

Figure 2. Example of $k$-NN nearest neighbors selection process in m = 2 dimensions, with n = 3 training data points and k = 2 nearest neighbors.

Notes:

It additionally exists the $k$-NN classification algorithm, which is a variant of the $k$-NN regression algorithm where the label space is not $\mathbb{R}$ but a finite subset of $\mathbb{N}$.

Theoretical guarantees

Since the seminal paper of Cover and Hart¹⁰ - proving under mild conditions that the $k$-NN classification algorithm achieves an error rate that is at most twice the best error rate achievable⁸ -, several convergence results have been established for $k$-NN methods.

For example, under an asymptotic regime where the number of training data points $n$ and the number of nearest neighbors $k$ both go to infinity, it has been demonstrated⁹ that the $k$-NN regression algorithm is able to learn any functional relationship of the form $Y_i = f \left( X_i \right) + \epsilon_i$, $i=1..n$, where $f$ is an unknown function and $\epsilon_i$ represents additive noise.

As another example, this time under a finite¹¹ sample regime, Jiang¹² derives the first sup-norm finite-sample [“convergence”] result¹² for the $k$-NN regression algorithm and shows that it achieves a maximum error rate that is equal to the best maximum error rate achievable up to logarithmic factors¹², with high probability.

In addition to these convergence results, $k$-NN methods also exhibit interesting properties w.r.t. the dimensionality of the feature space $\mathbb{R}^m$.

For example, while the curse of dimensionality forces non-parametric methods such as $k$-NN to require an exponential-in-dimension sample complexity¹², the $k$-NN regression algorithm actually adapts to the local intrinsic dimension without any modifications to the procedure or data¹².

In other words, if the feature vectors belong to $\mathbb{R}^m$ but have a “true” dimensionality equal to $\mathbb{R}^p, p < m$, then the $k$-NN regression algorithm will [behave] as if it were in the lower dimensional space [of dimension $p$] and independent of the ambient dimension [$m$]¹².

Further properties of $k$-NN methods can be found in Chen and Shah⁸ and in Biau and Devroye⁹.

Practical performances

Like all supervised machine learning algorithms, the practical performances of the $k$-NN regression algorithm heavily depend on the problem at hand.

Yet, in general, it often yields competitive results [v.s. other more complex algorithms like neural networks], and in certain domains, when cleverly combined with prior knowledge, it has significantly advanced the state-of-the-art¹³.

Beyond these competitive performances, Chen and Shah⁸ also highlights other important practical aspects of $k$-NN methods that contributed to their empirical success over the years⁸:

Their flexibility in choosing a problem-specific definition of “near” through a custom distance metric¹⁴
Their computational efficiency, which has enabled these methods to scale to massive datasets (“big data”)⁸ thanks to approaches like approximate nearest neighbor search¹⁵ or random projections¹⁶
Their non-parametric nature, in that they make very few assumptions on the underlying model for the data⁸
Their ease of interpretability, since they provide evidence for their predictions by exhibiting the nearest neighbors found⁸

$k$-NN based supervised portfolios

Supervised portfolios

Chevalier et al.¹ describes an asset allocation strategy that engineers optimal weights before feeding them to a supervised learning algorithm¹, represented in the lower part of Figure 1.

Given a training dataset of past [financial] observations¹ like past asset returns, past macroeconomic indicators, etc., it proceeds as follows:

For any relevant date¹⁷ $t=t_1,…$ in the training dataset
- Compute optimal (in-sample) future portfolio weights $w_{t+1}$ over a (also in-sample) desired future horizon¹⁸, using a selected portfolio optimization algorithm with financial observations up to the time $t+1$
  
  These optimal future portfolio weights are the labels $Y_t$, $t=t_1,…$, of the training data points.
  
  To be noted that by lagging the data, we can use the in-sample future realized returns to compute all the [returns-based] estimates¹ required by the portfolio optimization algorithm like the expected asset returns, the asset covariance matrix, etc. This allows to be forward-looking in the training sample, while at the same time avoiding any look-ahead bias¹.
  
  During this step, constraints can of course be added in order to satisfy targets and policies¹.
- Compute a chosen set of predictors supposed to be linked to the in-sample future portfolio weights $w_{t+1}$, using financial observations up to the time $t$
  
  These predictors are the feature vectors $X_t$, $t=t_1,…$, of the training data points.
Train and tune a supervised machine learning algorithm using the training data points $\left( X_t, Y_t \right)$, $t=t_1,…$.

Once the training phase is completed, the supervised portfolio allocation algorithm is ready to be used with test data¹⁹.

For any relevant (out-of-sample) test date $t’=t’_1,…$
- Compute the set of predictors chosen during the training phase, using financial observations up to the time $t’$
  
  These predictors are the test feature vectors $x_{t’}$, $t’=t’_1,…$.
- Provide that set of predictors as an input test feature vector to the supervised machine learning algorithm to receive in output the estimated optimal portfolio weights $\hat{w}_{t’+1}$ over the (out-of-sample) future horizon
  
  These estimated optimal portfolio weights are the estimated labels $\hat{y}_{t’}$, $t’=t’_1,…$.
  
  Here, depending on the exact supervised machine learning algorithm, the estimated portfolio weights $\hat{w}_{t’+1}$ might not satisfy the portfolio constraints²⁰ imposed in the training phase, in which case a post-processing phase would be required.

The portfolio allocation framework of Chevalier et al.¹ described above allows the algorithm to learn from past time series of in-sample optimal weights and to infer the best weights from variables such as past performance, risk, and proxies of the macro-economic outlook¹.

This contrasts with the standard practice of directly forecasting the input of a portfolio optimization algorithm, making that framework rather original.

In terms of empirical performances, Chevalier et al.¹ finds that predicting the optimal weights directly instead of the traditional two step approach leads to more stable portfolios with statistically better risk-adjusted performance measures¹ when using mean-variance optimization as the selected portfolio optimization algorithm and gradient boosting decision trees as the selected supervised machine learning algorithm²¹.

Some of these risk-adjusted performance measures are displayed in Figure 3 in the case of 4 asset classes²², for the 3 horizons of predicted returns and the 3 risk aversion levels used in Chevalier et al.¹.

Figure 3. Performances of supervised portfolios v.s. direct mean-variance optimized portfolios, 4 asset classes. Source: Adapted from Chevalier et al.

Notes:

Additional information can be found in the follow-up paper Chevalier et al.²³ and in a video of Thomas Raffinot for QuantMinds International.

$k$-NN-based supervised portfolios

Theoretically, the supervised machine learning model used in the portfolio allocation framework of Chevalier et al.¹ is trained to learn the following model¹:

\[w_{t+1} = g_t \left(X_t \right) + \epsilon_{t+1}\]

, where:

$X_t $ is the feature vector made of the chosen set of predictors computed at time $t$
$w_{t+1}$ is the vector of optimal portfolio weights over the desired future horizon $t+1$
$g$ is an unknown function

Because such a model describes a functional relationship compatible with a $k$-NN regression algorithm, it is reasonable to think about using that algorithm as the supervised machine learning algorithm in the above framework.

Enter $k$-NN-based supervised portfolios, a portfolio allocation framework originally introduced in Varadi and Teed² as follows:

This naturally leads us down the path of creating algorithms that can learn from past data and evolve over time to change the method for creating portfolio allocations.

The simplest and most intuitive machine-learning algorithm is the K-Nearest Neighbor method ($k$-NN) […, which] is a form of “case-based” reasoning. That is, it learns from examples that are similar to current situation by looking at the past [and says: “what happened historically when I saw patterns that are close to the current pattern?”].

It shares a lot in common with how human beings make decisions. When portfolio managers talk about having 20 years of experience, they are really saying that they have a large inventory of past “case studies” in memory to make superior decisions about the current environment.

As a side note, Varadi and Teed² is not the first paper to apply a $k$-NN regression algorithm to the problem of portfolio allocation, c.f. for example Gyorfi and al.²⁴ in the setting of online portfolio selection, but Varadi and Teed² is - to my knowledge - the first paper about the same “kind” of supervised portfolios as in Chevalier et al.¹.

A couple of practical advantages of $k$-NN-based supervised portfolios v.s. for example “gradient boosting decision trees”-based supervised portfolios as used in Chevalier et al.¹ are:

The simplicity of the training

Since nearest neighbor methods are lazy learners, there is strictly speaking no real training phase.
The simplicity of the tuning

There can be no tuning at all if no “advanced” technique (automated features selection, distance learning…) is used.
The guarantee that (convex) portfolio constraints learned during the training phase are satisfied during the test phase

In $k$-NN regression²⁵, the estimate for the label associated to a test point is a convex combination of that point nearest neighbors.

As a consequence, the estimated portfolio weights $\hat{w}_{t’+1}$ are guaranteed²⁵ to satisfy any learned convex portfolio constraints, thereby avoiding any post-processing that could degrade the “quality” of the estimated weights.
The ease of interpretability

Due to algorithm aversion, Chevalier et al.²³ highlights the need to be able to transform a black box nonlinear predictive algorithm [like gradient boosting decision trees] into a simple combination of rules²³ in order to make it interpretable for humans.

With a $k$-NN regression algorithm, which is one of the most transparent supervised machine learning algorithm in existence, that step is probably not useful²⁶.

In terms of empirical performances, Varadi and Teed² concludes that $k$-NN-based supervised portfolios consistently outperformed [vanially maximum Sharpe ratio portfolios] on both heterogeneous and homogenous data sets on a risk-adjusted basis², with the $k$-NN-based approach [exhibiting] a Sharpe ratio [… up to] over 30% higher than [the direct maximum Sharpe ratio approach]².

Average performance measures for the $k$-NN-based supervised portfolios in Varadi and Teed² are reported in Figure 4.

Figure 4. Performances of $k$-NN-based supervised portfolios v.s. direct mean-variance optimized portfolios. Source: Adapted from Varadi and Teed.

Implementing $k$-NN-based supervised portfolios

Features selection

Biau and Devroye⁹ describes features selection as:

[…] the process of choosing relevant components of the [feature] vector $X$ for use in model construction.

There are many potential benefits of such an operation: facilitating data visualization and data understanding, reducing the measurement and storage requirements, decreasing training and utilization times, and defying the curse of dimensionality to improve prediction performance.

, and provides some rules of thumb that should be followed⁹:

Noisy measurements, that is, components that are independent of $Y$, should be avoided⁹, especially because nearest neighbor methods are extremely sensitive to the features used²⁷
Adding a component that is a function of other components is useless⁹

Beyond these generic rules, and although it has been an active research area in the statistics, machine learning, and data mining communities¹, features selection is unfortunately strongly problem-dependent.

In the context of supervised portfolios, Chevalier et al.¹ and Varadi and Teed² both propose to use:

Past asset returns over different horizons²⁸ so as to assess momentum and reversals¹
Past asset volatilities over different horizons²⁸, to approximate asset-specific risk¹

Varadi and Teed² additionally proposes to include past asset correlations over different horizons²⁸ to ensure that [the] $k$-NN algorithm [doesn’t] have access to any information that the [direct mean-variance optimization] [doesn’t] have, but merely use it differently².

Chevalier et al.¹, building on stocks asset pricing litterature, does not suggest to include other returns-based indicator than past asset returns and volatilities but suggests instead to include various macroeconomic indicators (yield curve, VIX…).

Features scaling

Typical distance metrics²⁹ used with nearest neighbor methods like the Euclidean distance are said to be scale variant, meaning that the definition of a nearest neighbor is influenced by the relative and absolute scale of the different features.

For example, when using the Euclidean distance with features such as a person’s height and a person’s age:

The height feature disproportionally infuences the definition of a neighbor if the height feature is measured in millimeters and age in years
The age feature disproportionally infuences the definition of a neighbor if the height feature is measured in meters and age in days

For this reason, features are usually scaled to a similar range before being provided in input to a $k$-NN algorithm³⁰, which is a pre-processing step called features scaling.

A couple of techniques for features scaling are described in Arora et al.³¹:

Min-max scaling, which scales all the values of a feature $\left( X_i \right)_j$, $j \in \{ 1,…,m \}$, $i=1..n$ to a given interval - like $[0,1]$ -, based on the minimum and the maximum values of that feature:
\[\left( X_i \right)_j' = \frac{\left( X_i \right)_j - \min_j \left( X_i \right)_j }{\max_j \left( X_i \right)_j - \min_j \left( X_i \right)_j }, i=1..n\]
Standardization, also called z-score normalization, which transforms all the values of a feature $\left( X_i \right)_j$, $j \in \{ 1,…,m \}$ , $i=1..n$ into values that are approximatively standardly normally distributed:
\[\left( X_i \right)_j' = \frac{ \left( X_i \right)_j - \overline{\left( X_i \right)_j}}{ \sigma_{\left( X_i \right)_j} }\]

In the context of supervised portfolios, additional techniques are described in Chevalier et al.¹:

Quantile normal transformation for a “time series”-like feature, which standardizes the time-series into quantile and then map the values to a normal distribution¹

It is important to note that at any given date, the quantiles should be computed using information up to that date only to avoid forward looking leakage¹.

In addition, a lookback window over which to compute the quantiles should be chosen, with possible impacts on the performances of the supervised machine learning algorithm.
Cross sectional normalization for a regular feature, which scales the cross sectional values between 0 and 1 using the empirical cumulative distribution function¹

At any given date, this normalization can be performed fully in the cross-section at that date if there are enough assets or in the cross-section at that date using information up to that date to compute the empirical cumulative distribution function.

In the latter case, c.f. the previous point.
Hyperbolic tangent function ($\tanh$) scaling for labels, in order to center [them] and make them more comparable by taming outliers¹:
\[Y' = 0.5 \tanh{\left( 0.01 \frac{Y − \overline{Y}}{ \sigma_Y } \right) }\]
Naturally, the reverse transformation is performed after the prediction to transform back the labels into its original values¹.

Finally, in the specific context of $k$-NN-based supervised portfolios, 2 additional techniques are described in Varadi and Teed², that are variations of the techniques of Chevalier et al.¹.

Distance metric selection

As already mentioned in the previous sub-section, the distance metric used with a nearest neighbor method influences the definition of a nearest neighbor due to its scale variant or scale invariant nature.

But that’s not all, because different distance metrics behave differently with regards to outliers, to noise, to the dimension of the feature space, etc. On top of that, the chosen distance metric is sometimes not a proper metric⁶…

So, what to do in the specific context of $k$-NN-based supervised portfolios?

From the empirical results in Varadi and Teed², the Euclidean distance seems to be a good choice as long as the chosen predictors are properly scaled.

From the empirical results later in this blog post, a little known distance metric called the Hassanat distance³² also seems to be a good choice and additionally does not require³³ the chosen predictors to be scaled because it is scale invariant³⁴.

That distance - noted $HasD(x,y)$ - is defined between two vectors $x = \left(x_1,…,x_m\right)$ and $y = \left(y_1,…,y_m\right)$ as follows:

\[HasD(x,y) = \sum_{i=1}^m D(x_i,y_i)\]

, with

\[D(x_i,y_i) = \begin{cases} 1 - \frac{1 + \min(x_i,y_i)}{1 + max(x_i,y_i)}, &\text{if } \min(x_i,y_i) \geq 0 \\ 1 - \frac{1}{1 + \max(x_i,y_i) - min(x_i,y_i) } &\text{if } \min(x_i,y_i) < 0 \end{cases}\]

Figure 5 illustrates the 1-dimensional Hassanat distance $HasD(0,n)$ with $n \in [-10,10]$.

Figure 5. Representation of the 1-dimensional Hassanat distance between the points 0 and n. Source: Abu Alfeilat et al.

As a side note, the Hassanat distance has been empirically demonstrated to perform the best when applied on most data sets comparing with the other tested distances³⁵ in Abu Alfeilat et al.³⁵, which compares the performances of 54 distance metrics used in $k$-NN classification.

How to select the number of nearest neighbors?

Together with the distance metric $d$, the number of nearest neighbors $k$ is the other hyperparameter that has to be selected in nearest neighbor methods.

Varadi and Teed² explains:

The choice of the number of nearest matches (or neighbors) is the $k$ in $k$-NN.

This is an important variable that allows the benefit of allowing the use the ability to trade-off accuracy versus reliability. Choosing a value for $k$ that is too high will lead to matches that are not appropriate to the current case. Choosing a value that is too low will lead to exact matches but poor generalizability and high sensitivity to noise.

The optimal value for K that maximizes out-of-sample forecast accuracy will vary depending on the data and the features chosen.

In practice, the number of nearest neighbors $k$ […] [is] usually selected via cross-validation or more simply data splitting⁸ and the selected [value] minimizes an objective function which is often the Root Mean Square Error (RMSE) or sometimes the Mean Absolute Error (MAE)³⁶.

That being said, Guegan and Huck³⁶ cautions about that practice by highlighting that:

The estimation of $k$ via in sample predictions leads to choose high values, near or on the border of one has tabulated because the RMSE is a decreasing function of the number of neighbors³⁶
A high value for the number of nearest neighbors is an erroneous usage of the [method] because the neighbors are thus not near the pattern they should mimic³⁶, leading to (useless) forecasts very close to the mean of the sample³⁶.

Another direction is to adaptively choose the number of nearest neighbors $k$ […] depending on the test feature vector⁸.

For example, Anava and Levy³⁷ proposes solving an optimization problem to adaptively choose what $k$ to use for [a given feature vector] in an approach called $k^*$-NN⁸.

In the specific context of $k$-NN-based supervised portfolios, and again to avoid choosing an explicit number of nearest neighbors, Varadi and Teed² suggests to select a range³⁸ of $k$’s to make [the] selection base more robust to potential changes in an “optimal” $k$ selection².

Surprisingly, it turns out that this method is an ensemble method similar in spirit to the method described in Hassanat et al.³⁹ for $k$-NN classification, which consists in using a base $k$-NN classifier with $k=1,2,…,\lfloor \sqrt{n} \rfloor$ and to combine the $\lfloor \sqrt{n} \rfloor$ classification results using inverse logarithmic weights.

Misc. remarks

Importance of training dataset diversity

Asymptotic convergence results for the $k$-NN regression algorithm guarantee that by increasing the amount of [training] data, […] the error probability gets arbitrarily close to the optimum for every training sequence⁹.

But the amount of data available for training $k$-NN-based supervised portfolios is not infinite and might even in some cases be extremely limited⁴⁰.

In that case, there is a high risk that the training data is “unevenly balanced” in the feature space, a situation illustrated in Figure 6 in the case of a univariate feature whose underlying distribution is Gaussian.

Figure 6. Univariate $k$-NN regression with low accuracy, Gaussian feature distribution, far away training data. Source: Chen and Shah.

From Figure 6, it is clear that such a lack of training data - or more precisely, such a lack of diversity in the training data - would force the $k$-NN regression algorithm to use far away nearest neighbors, which would severely degrade the quality of the forecasted portfolio weights.

So, particular attention must be paid to the size and the diversity of the training dataset when using $k$-NN-based supervised portfolios, with for example ad-hoc procedures used whenever needed to simulate past asset returns for assets without return histories (here) or to extend return histories for assets with shorter return histories than others (here).

Avoiding the curse of dimensionality

The number of features selected by Varadi and Teed² grows quadratically with the number of assets.

At some point⁴¹, the underlying $k$-NN regression algorithm will then inevitably face issues due to:

Distance concentration, which is the tendency of distances between all pairs of points in high-dimensional data to become almost equal⁴²
Poor discrimination of the nearest and farthest points for a given test point, which is an issue on top of the distance concentration problem, c.f. Beyer et al.⁴³
Hubness⁴², defined as the emergence of points called hubs which appear overly similar to many others
…

In addition, the higher the number of features selected, the more training data is required to learn enough combinations of these different features, which further compounds the problem mentioned in the previous sub-section…

All in all, that approach is not scalable but hopefully, a solution is also proposed in Varadi and Teed²:

[…] to explore multi-asset portfolios [without introducing the problem of dimensionality with too-large a feature space], we took the average weight of each security from a single-pair run, and averaged them across all pair runs.

While this proposal may look like an ad-hoc workaround, it actually corresponds to an ensemble method that has been empirically shown to be effective for $k$-NN classification in high dimension in both:

Domeniconi and Yan⁴⁴, with a deterministic selection of features as in Varadi and Teed²
Bay⁴⁵, with a random²⁷ selection of features

The underlying idea of that ensemble method is to exploit [the] instability of $k$-NN classifiers with respect to different choices of features to generate an effective and diverse set of NN classifiers with possibly uncorrelated errors⁴⁴.

Implementations

Implementation in Portfolio Optimizer

Portfolio Optimizer supports $k$-NN-based supervised portfolios through the endpoint /portfolios/optimization/supervised/nearest-neighbors-based.

This endpoint supports 2 different distance metrics:

The Euclidean distance matrix
The Hassanat distance metric (default)

As for the selection of the number of nearest neighbors, this endpoint supports:

A manually-defined number of nearest neighbors
A dynamically-determined number of nearest neighbors together with their individual weights through:
- The $k$-NN ensemble method of Hassanat et al.³⁹
- A proprietary variation of the $k^*$-NN method of Anava and Levy³⁷ (default)

Implementation elsewhere

Chevalier et al.¹ kindly provides a Python code to experiment with “gradient boosting decision trees”-based supervised portfolios.

Example of usage - Learning maximum Sharpe ratio portfolios

Because most portfolio allocation decisions for active portfolio managers revolve around the optimal allocation between stocks and bonds², I propose to reproduce the results of Varadi and Teed² in the case of a 2-asset class portfolio made of:

U.S. equities, represented by the SPY ETF
U.S. long-term Treasury bonds, represented by the TLT ETF

Methodology

Varadi and Teed² follows the general procedure of Chevalier et al.¹ to train a $k$-NN-based supervised portfolio allocation algorithm for learning portfolio weights maximizing the Sharpe ratio.

For this, and without entering into the details:

The selected features are asset returns, standard deviations and correlations over different²⁸ past lookback periods, scaled through a specific normal distribution standardization
The selected distance metric is the standard Euclidean distance
The selected number of nearest neighbors is not a single value but a range of values related to the size of the training dataset³⁸
The relevant initial training dates are the 2000 daily dates present in Varadi and Teed²’s dataset from 4/13/1976 minus 2000 days to 4/12/1976
The relevant subsequent training dates and test dates are all the daily dates present in Varadi and Teed²’s dataset from 4/13/1976 to 12/31/13

To be noted that the training data is used in a rolling window manner over a 2000-day lookback.
The future horizon over which maximum Sharpe ratio portfolio weights are learned during the training phase and evaluated during the test phase is a 20-day horizon

On my side:

The selected features will be:
- Past 12-month asset arithmetic returns, cross-sectionally normalized using the procedure described in Almgren and Neil⁴⁶
- Future aggregated asset covariances forecasted over the next month using an exponentially weighted moving average covariance matrix forecasting model with daily squared (close-to-close) returns
The selected distance metric will be the Hassanat distance

This avoids the need for further features scaling.
The selected number of nearest neighbors will be:
- 1
- 10
- Dynamically determined with their individual weights by:
  - The $k$-NN ensemble method of Hassanat et al.³⁹
  - The $k^*$-NN method of Anava and Levy³⁷
The relevant initial training dates will be all month-end dates present in a SPY/TLT ETFs-like training dataset from 1st January 1979 to 30 November 2003

Due to the relatively recent inception dates of both the SPY ETF (22nd January 1993) and the TLT ETF (22th July 2002), it is required to use proxies to extend the returns history of these assets:
- The daily U.S. market returns $Mkt$ provided in the Fama and French data library, as a proxy for the SPY ETF daily returns
- The simulated daily returns associated to the daily FRED 30-Year Treasury Constant Maturity Rates, as a proxy for the TLT ETF daily returns
With these, the earliest date for which daily SPY/TLT ETFs-like returns are available is 16th February 1977; adding 1 year of data for computing the past 12-month returns gives 16th February 1978; rounded to 1st January 1979.
The relevant subsequent training dates and test dates will be all month-end dates present in the SPY/TLT ETFs test dataset from 1st January 2004 to 28th February 2025⁴⁷

The earliest date for which daily SPY/TLT ETFs returns are available is 29th July 2002; adding 1 year of data for computing the past 12-month returns gives 29th July 2003; rounded to 1st January 2004.

To be noted that the training data is used in an expanding window manner.

As a consequence, the training dataset is made of 299 data points on 1st January 2004, expanding up to 552 data points on 28th February 2025 when the last forecast is made.

This is in stark constrast with Varadi and Teed²’s training dataset which 1) contains 2000 data points and 2) is not expanding but is being rolled forward to keep the algorithm more robust to market changes in feature relevance².

As mentionned in a previous section, such a difference in quantity and in “local” diversity of the training dataset might impact my results v.s. those of Varadi and Teed².
The future horizon over which maximum Sharpe ratio portfolio weights are learned during the training phase and evaluated during the test phase is a 1-month horizon at daily level
The risk free rate is set to 0% when computing maximum Sharpe ratio portfolio weights
The cash portion of the different SPY/TLT portfolios - if any - is allocated to U.S. short-term Treasury bonds, represented by the SHY ETF

Results

Figure 7 compares the standard direct approach for maximizing the Sharpe ratio to the $k$-NN-based supervised portfolios approach, with the 4 choices of nearest neighbors proposed in the previous sub-section.

In both cases, like in Varadi and Teed², the same features are used in input of the two algorithms.

Figure 7. MSR portfolio v.s. $k$-NN-based learned MSR portfolios, SPY/TLT ETFs, 1st January 2004 - 31th March 2025.

Summmary statistics:

Portfolio	CAGR	Average Exposure	Annualized Sharpe Ratio	Maximum (Monthly) Drawdown
Maximum Sharpe ratio (MSR)	~5.6%	~61%	~0.73	~14.4%
$k$-NN learned MSR, $k=1$	~6.4%	~49%	~0.86	~14.4%
$k$-NN learned MSR, $k=10$	~5.2%	~51%	~0.99	~15.8%
$k$-NN learned MSR, $k=kEnsemble$	~5.4%	~51%	~1.01	~15.1%
$k$-NN learned MSR, $k=k^*$	~6.8%	~49%	~1.00	~14.0%

Comments

A couple of comments are in order:

Consistent with Varadi and Teed², the results demonstrate that the [$k$-NN-based supervised portfolio allocation] approach tends to outperform [the direct] MVO portfolio allocation [approach] on a risk-adjusted basis², with a Sharpe ratio ~18%-38% higher.

This is quite interesting to highlight since the objective of the direct approach is supposed to be the maximization of the portfolio Sharpe ratio!
The average exposure of the MSR portfolio is ~61% v.s. a relatively much lower exposure of ~50% for all the $k$-NN learned MSR portfolios

The Sharpe ratio of all the $k$-NN learned MSR portfolios being higher than that of the MSR portfolio, it implies that the changes in exposure are pretty well “timed”.
The $k$-NN learned MSR portfolios with $k=10$ and $k=kEnsemble$ are nearly identical

This is confirmed by examining the underlying asset weights (not shown here).

The $k$-NN ensemble portfolio has the advantage of not requiring to choose a specific value for $k$, though, and should definitely be prefered.
The $k$-NN learned MSR portfolios with $k=1$ and $k=k^*$ are close in terms of raw performances, but not in terms of Sharpe ratio

A closer look (not detailled here) reveals that this is because the $k$-NN learned MSR portfolios with $k=k^*$ regulary selects only 1 nearest neighbor when the other neighbors are too “far away” but also regularly selects many more neighbors when the other neighbors are “close enough”.

I interpret this as an empirical demonstration of the ability of the $k^*$-NN method of Anava and Levy³⁷ to adaptively choose the number of nearest neighbors $k$ […] depending on the test feature vector⁸.
The maximum drawdowns are comparable across all portfolios

This shows that the $k$-NN learned MSR portfolios, despite their attractive risk-adjusted performancess, are not able to magically avoid “dramatic” events.

Another layer of risk management, better return predictors, or both, is probably needed for that.
The winner of this horse race is the $k$-NN learned MSR portfolios with $k=k^*$, but this comes at a price in terms of turnover v.s. the $k$-NN learned MSR portfolios with $k=kEnsemble$

Also consistent with Varadi and Teed², the asset weights of the $k$-NN learned MSR portfolio with $k=kEnsemble$ (and with $k=10$) are relatively stable and on average similar to an equal weight portfolio, while those of the MSR portfolio show considerable noise and turnover².

This is visible on the portfolio transition maps displayed in Figures 8 and 9.

Figure 8. $k$-NN-based learned MSR portfolio, $k=kEnsemble$, SPY/TLT/SHY ETFs allocations through time, 1st January 2004 - 31th March 2025.

Figure 9. MSR portfolio, SPY/TLT/SHY allocations through time, 1st January 2004 - 31th March 2025.

For Varadi and Teed², this demonstrates the general uncertainty of the portfolio indicator inputs in aggregate² and that $k$-NN learned MSR portfolio with $k=kEnsemble$ manages to dynamically balance this uncertainty over time and shift more towards a probabilistic allocation that did not overweight or over-react to poor information².

This statement is slightly less applicable to the $k$-NN learned MSR portfolio with $k=k^*$, because its better raw performances are explained by a more aggressive allocation, resulting in a much higher turnover, as can be seen by comparing Figure 8 to Figure 10.

Figure 10. $k$-NN-based learned MSR portfolio, $k=k^*$, SPY/TLT/SHY ETFs allocations through time, 1st January 2004 - 31th March 2025.

Conclusion

Exactly like in Varadi and Teed², and despite the differences in implementation and in the size of the training dataset⁴⁸:

The results of this section shows that a traditional mean-variance/Markowitz/MPT framework under-performs [a $k$-NN-based supervised portfolio allocation] framework in terms of maximizing the Sharpe ratio²
The data further implies that traditional MPT makes far too many trades and takes on too many extreme positions as a function of how it is supposed to generate portfolio weights²

Varadi and Teed² provides the following explanation:

This occurs because the inputs - especially the returns - are very noisy and may also demonstrate non-linear or counter-intuitive relationships. In contrast, by learning how the inputs map historically to optimal portfolios at the asset level, the resulting [$k$-NN-based supervised portfolios] allocations drift in a more stable manner over time.

Final conclusion

Supervised portfolios as introduced in Chevalier et al.¹ are able learn from past time series of in-sample optimal weights¹ and to infer the best weights from variables such as past performance, risk, and proxies of the macro-economic outlook¹.

In this blog post, I empirically demonstrated that this capability allows one of their simplest embodiement - $k$-NN-based supervised portfolios - to outperform a traditional mean-variance framework that seeks to maximize the Sharpe ratio of a portfolio, which independently confirms the prior results of Varadi and Teed².

To keep discovering non-standard portfolio allocation frameworks, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Chevalier, G., Coqueret, G., & Raffinot, T. (2022). Supervised portfolios. Quantitative Finance, 22(12), 2275–2295. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³
See David Varadi, Jason Teed, Adaptive Portfolio Allocations, NAAIM paper. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³ ↩³⁴ ↩³⁵ ↩³⁶ ↩³⁷ ↩³⁸ ↩³⁹ ↩⁴⁰
Varadi and Teed² has been submitted to the 2014 NAAIM annual white paper competition known as the NAAIM Founders Award. ↩
To be noted that the data points could belong to a more generic space than $\mathbb{R}^m \times \mathbb{R}$. ↩
$\mathbb{R}^m$ is usually called the feature space. ↩
In practice, $d$ might not necessarily be a proper metric; for example, it might not satisfy the triangle inequality property like the cosine “distance”. ↩ ↩²
For an historical perspective on the $k$-NN algorithm going beyond the usual technical report from Fix and Hodges⁴⁹ and the seminal paper from Cover and Hart¹⁰, the interested reader is refered to Chen and Shah¹², which mentions that the $k$-NN classification algorithm was already mentioned in a text from the early 11th century. ↩
See George H. Chen; Devavrat Shah, Explaining the Success of Nearest Neighbor Methods in Prediction , now, 2018. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²
See Gerard Biau, Luc Devroye, Lectures on the Nearest Neighbor Method, Springer Series in the Data Sciences. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸
See Cover, T. M. and P. E. Hart (1967). “Nearest neighbor pattern classification”. IEEE Transactions on Information Theory. ↩ ↩²
Although $n$ must be sufficiently large in order for there to exist a $k$ that satisfies the conditions¹² required by the main theorem of Jiang¹². ↩
See Jiang, H. (2019). Non-Asymptotic Uniform Rates of Consistency for $k$-NN Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3999-4006. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹
See S. Sun and R. Huang, “An adaptive k-nearest neighbor algorithm,” 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery, Yantai, China, 2010, pp. 91-94. ↩
See for example P.Y. Simard, Y. LeCun and J. Decker, “Efficient pattern recognition using a new transformation distance,” In Advances in Neural Information Processing Systems, vol. 6, 1993, pp. 50-58, in which the Euclidean distance between images of handwritten digits is replaced by an ad-hoc distance invariant with respect to geometric transformations of such images (rotation, translation, scaling, etc.). ↩
See Har-Peled, S., P. Indyk, and R. Motwani (2012). “Approximate Nearest eighbor: Towards Removing the Curse of Dimensionality.” Theory of Computing. ↩
See Kleinberg, J. M. (1997). “Two algorithms for nearest-neighbor search in high dimensions”. In: Symposium on Theory of Computing. ↩
For example, the end of each month for learning a monthly asset allocation strategy. ↩
A day, a week, a month, etc. ↩
Also called inference data, that is, data not “seen” during the training phase. ↩
Like budget constraints, asset weights constraints, asset group constraints, portfolio exposure constraints, etc. ↩
Chevalier et al.¹ notes that these empirical results still hold when replacing boosted trees by simple regressions¹. ↩
Developed equities, emerging equities, global corporate bonds, global government bonds. ↩
See Chevalier, Guillaume, Coqueret, Guillaume, Raffinot, Thomas, Interpretable Supervised Portfolios, The Journal of Financial Data Science Spring 2024, 6 (2) 10-34. ↩ ↩² ↩³
See L. Gyorfi, F. Udina, and H. Walk. Nonparametric nearest neighbor based empirical portfolio selection strategies. Statistics & Decisions, International Mathematical Journal for Stochastic Methods and Models, 26(2):145–157, 2008. ↩
Unless a very specific variation of $k$-NN regression is used. ↩ ↩²
At least from an algorithm aversion perspective. Nevertheless, there can be other benefits, c.f. Chevalier et al.²³. ↩
Similar to random subspace optimization. ↩ ↩²
1 month, 2 months, 3 months, 6 months and 12 months. ↩ ↩² ↩³ ↩⁴
See Avivit Levy, B. Riva Shalom, Michal Chalamish, A Guide to Similarity Measures, arXiv for a very long list of distance metrics. ↩
Or more generally, to most supervised machine learning algorithms. ↩
See Ishan Arora and Namit Khanduja and Mayank Bansal, Effect of Distance Metric and Feature Scaling on KNN Algorithm while Classifying X-rays, RIF, 2022. ↩
See Hassanat, A.B., 2014. Dimensionality Invariant Similarity Measure. Journal of American Science, 10(8), pp.221-26. ↩
To be noted that features scaling might still be performed to try to improve the performances of the $k$-NN regression algorithm. ↩
Other properties of the Hassanat distance are for example robustness to noise and linear growth with the dimension of the feature space, c.f. Abu Alfeilat et al.³⁵. ↩
See Abu Alfeilat HA, Hassanat ABA, Lasassmeh O, Tarawneh AS, Alhasanat MB, Eyal Salman HS, Prasath VBS. Effects of Distance Measure Choice on K-Nearest Neighbor Classifier Performance: A Review. Big Data. 2019 Dec;7(4):221-248. ↩ ↩² ↩³
See Guegan, D. and Huck, N. (2005). On the Use of Nearest Neighbors in Finance. Finance, . 26(2), 67-86. ↩ ↩² ↩³ ↩⁴ ↩⁵
See Oren Anava, Kfir Levy, k*-Nearest Neighbors: From Global to Local, Advances in Neural Information Processing Systems 29 (NIPS 2016). ↩ ↩² ↩³ ↩⁴
In more details, Varadi and Teed² chooses the $k$’s in percentages of the size of the training space, which were 5%, 10%, 15% and 20% resulting essentially in a weighted average of the top instances². ↩ ↩²
See Hassanat, A.B., Mohammad Ali Abbadi, Ghada Awad Altarawneh, Ahmad Ali Alhasanat, 2014. Solving the Problem of the K Parameter in the KNN Classifier Using an Ensemble Learning Approach. International Journal of Computer Science and Information Security, 12(8), pp.33-39. ↩ ↩² ↩³
For example, due to the limited price history of some assets or due to the length of the desired horizon over which optimal portfolio weights need to be computed. ↩
Possibly for as few as 10-15 dimensions⁴³! ↩
See Radovanovic, Milo and Nanopoulos, Alexandros and Ivanovic, Mirjana}, Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data, Journal of Machine Learning Research 11 (2010) 2487-2531. ↩ ↩²
See Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U. (1999). When Is “Nearest Neighbor” Meaningful?. In: Beeri, C., Buneman, P. (eds) Database Theory — ICDT’99. ICDT 1999. Lecture Notes in Computer Science, vol 1540. Springer, Berlin, Heidelberg. ↩ ↩²
See Domeniconi, C., & Yan, B. (2004). Nearest neighbor ensemble. In Pattern recognition, international conference on, Vol. 1 (pp. 228–231). Los Alamitos, CA, USA: IEEE Computer Society. ↩ ↩²
See Bay S.D. Nearest neighbor classification from multiple feature subsets Intelligent Data Analysis, 3 (1999), pp. 191-209. ↩
See Almgren, Robert and Chriss, Neil A., Optimal Portfolios from Ordering Information (December 2004). ↩
(Adjusted) daily prices have have been retrieved using Tiingo. ↩
This empirically confirms that, desptite their dependency on the size of the training dataset, nearest neighbor methods can learn from a small set of examples⁴⁵. ↩
See Discriminatory analysis, nonparametric discrimination: Consistency properties”. Technical report, USAF School of Aviation Medicine. ↩

Correlation-Based Clustering: Spectral Clustering Methods

2025-05-01T00:00:00-05:00

Clustering consists in trying to identify groups of “similar behavior”¹ - called clusters - from a dataset, according to some chosen characteristics.

An example of such a characteristic in finance is the correlation coefficient between two time series of asset returns, whose usage to partition a universe of assets into groups of “close” and “distant” assets thanks to a hierarchical clustering method was originally² proposed in Mantegna³.

In this blog post, I will describe two correlation-based clustering methods belonging to the family of spectral clustering methods: the Blockbuster method introduced in Brownlees et al.⁴ and the SPONGE method introduced in Cucuringu et al.⁵.

As examples of usage, I will discuss 1) how to automatically group U.S. stocks together without relying on external information like industry classification and 2) how to identify risk-on and risk-off assets within a U.S. centric universe of assets.

Mathematical preliminaries

Let $x_1$, …, $x_n$ be $n$ points⁶ in $\mathbb{R}^m, m \geq 1$ to be partitioned into $k \geq 2$ subsets.

Spectral clustering

Spectral clustering, like other approaches to clustering - geometric approaches such as $k$-means , density-based approaches such as DBSCAN… - initially relies on pairwise similarities $s(x_i, x_j)$ between points $i,j=1..n$ with $s$ some similarity function which is symmetric and non-negative¹.

Once the corresponding similarity matrix $S_{ij} = s(x_i, x_j)$, $i,j=1…n$, has been computed, a spectral clustering method then usually follows a three-step process:

Compute an affinity matrix $A \in \mathbb{R}^{n \times n}$ from the similarity matrix $S$

The affinity matrix $A$ corresponds to the adjacency matrix of an underlying graph whose vertices represent the points $x_1$, …, $x_n$ and whose edges model the local⁷ neighborhood relationships between the data points¹.
Compute a matrix $L \in \mathbb{R}^{n \times n}$ derived from the affinity matrix $A$

Due to the relationship between spectral clustering and graph theory¹, the matrix $L$ is typically called a Laplacian matrix.
Compute a matrix $Y \in \mathbb{R}^{n \times k}$ derived from the eigenvectors corresponding to the $k$ smallest (or sometimes largest) eigenvalues of the matrix $L$ and cluster its $n$ rows $y_1$, …, $y_n$ using the $k$-means algorihm

The result of that clustering represents the desired clustering of the original $n$ points $x_1$, …, $x_n$.

It should be emphasized that there is nothing principled about using the $k$-means algorithm in this step¹, c.f. for example Huang et al.⁸ in which spectral rotations are used instead of $k$-means, but one can argue that at least the Euclidean distance between the [rows of the matrix $Y$] is a meaningful quantity to look at¹.

The Ng-Jordan-Weiss spectral clustering method

Different ways of computing the affinity matrix $A$, the Laplacian matrix $L$ or the matrix $Y$ lead to different spectral clustering methods⁹.

One popular spectral clustering method is the Ng-Jordan-Weiss (NJW) method¹⁰, detailed in Figure 1 taken from Ng et al.¹⁰ where $\sigma^2$ is a scaling parameter that controls how rapidly the affinity $A_{ij}$ falls off with the distance between $s_i$ and $s_j$¹⁰.

Figure 1. Ng-Jordan-Weiss spectral clustering method. Source: Ng et al.

Rationale behind spectral clustering

At first sight, [spectral clustering] seem to make little sense. Since we run $k$-means [on points $y_1$, …, $y_n$], why not just apply $k$-means directly to the [original points $x_1$, …, $x_n$]¹⁰?

A visual justification of spectral clustering is provided in Figure 2 adpated from Ng et al.¹⁰, which displays data points in $\mathbb{R}^2$ forming two circles (e) and their associated spectral representation in $\mathbb{R}^2$ using the NJW spectral clustering method (h).

Figure 2. Data points forming two circles, 2D plane and spectral plane. Source: Ng et al.

On Figure 2, it is clearly visible that the transformed data points (h) form two well-separated convex clusters in the spectral plane, which is the ideal situation for the $k$-means algorithm.

More generally¹¹, it can be demonstrated that the change of representation from the original data points $x_1$, …, $x_n$ to the embedded data points $y_1$, …, $y_n$ enhances the cluster-properties in the data, so that clusters can be trivially detected [by the $k$-means clustering algorithm] in the new representation¹.

Correlation-based spectral clustering

Let $x_1$, …, $x_n$ be $n$ variables whose pairwise correlations¹² $\rho_{ij}$, $i,j=1…n$, have been assembled in a correlation matrix $C \in \mathbb{R}^{n \times n}$.

Because correlation is a measure of dependency, a legitimate question is whether it is possible to use spectral clustering to partition these variables w.r.t. their correlations? Ideally, we would like to have highly correlated variables grouped together in the same clusters, with low or even negative correlations between the clusters themselves.

Problem is, as noted in Mantegna³, the correlation coefficient of a pair of [variables] cannot be used as a distance [or as a similarity] between the two [variables] because it does not fulfill the three axioms that define a metric³, which a priori excludes its direct usage in a similarity matrix or in an affinity matrix.

That being said, a¹³ metric¹⁴ can be defined using as distance a function of the correlation coefficient³ - like the distance $ d_{ij} = \sqrt{ 2 \left( 1 - \rho_{ij} \right) } $ -, which allows to indirectly use any spectral clustering method¹⁵ with correlations.

But what if we insist to work directly with correlations? In this case, a couple of specific spectral clustering methods have been developped, among which:

The Blockbuster spectral clustering method of Brownlees et al.⁴
The SPONGE spectral clustering method of Cucuringu et al.⁵

The Blockbuster spectral clustering method

Brownlees et al.⁴ introduces a network model¹⁶ under which large panels of time series are partitioned into latent groups such that correlation is higher within groups than between them⁴ and proposes an algorithm relying on the eigenvectors of the sample covariance matrix¹⁷ $\Sigma \in \mathbb{R}^{n \times n}$ to detect these groups.

As can be seen in Figure 3 taken from Brownlees et al.⁴, this algorithm - called Blockbuster - is surprisingly very close to the NJW spectral clustering method.

Figure 3. Blockbuster algorithm. Source: Brownlees et al.

Brownlees et al.⁴ establishes that the Blockbuster algorithm consistently detects the [groups] when the number of observations $T$ and the dimension of the panel $n$ are sufficiently¹⁸ large⁴, as long as a couple of assumptions are satisfied, like:

$T \geq n$, with the more fat-tailed and dependent the data are, the larger $T$ has to be⁴
The precision matrix $\Sigma^{-1}$ exists and contains only non-positive entries (or its fraction of positive entries is appropriately controlled¹⁹)

An important feature of the Blockbuster algorithm is that it allows one to detect the [groups] without estimating the network structure of the data⁴. In other words, while the Blockbuster algorithm is a spectral clustering method, nowhere is an affinity matrix computed!

The black magic at work here is that the underlying network model of Brownlees et al.⁴ additionally assumes that the (symmetric normalized) Laplacian matrix $L$ is a function of the precision matrix $\Sigma^{-1}$ through

\[L = \frac{\sigma^2}{\phi} \Sigma^{-1} - \frac{1}{\phi} I_n\]

, where:

$\sigma^2$ is a network variance parameter that does not influence the detection of the groups
$\phi$ is a network dependence parameter that does not influence the detection of the groups
$I_n \in \mathbb{R}^{n \times n}$ is the identity matrix of order $n$

This formula clarifies the otherwise mysterious connection²⁰ between the Blockbuster algorithm and the NJW method.

The SPONGE spectral clustering method

Cucuringu et al.⁵ extends the spectral clustering framework described in the previous section to the case of a signed affinity matrix $A$ whose underlying complete weighted graph represents the variables $x_1$, …, $x_n$ and their pairwise correlations.

Figure 4 taken from Jin et al.²¹ illustrates the idea of Cucuringu et al.⁵, which is to minimize the number of violations in the constructed partition, with a violation, as in this figure, is when there are negative edges in a cluster and positive edges across clusters²¹.

Figure 4. Illustration of the idea behind the SPONGE clustering algorithm - minimizing the number of violations in the constructed partition. Source: Jin et al.

More formally, Cucuringu et al.⁵ proposes a three-step algorithm²² - called SPONGE (Signed Positive Over Negative Generalized Eigenproblem) - that aims to find a partition [of the underlying graph] into k clusters such that most edges within clusters are positive, and most edges across clusters are negative⁵:

Decompose the adjacency matrix as $A = A^+ - A^-$, with
- $A^+ \in \mathbb{R}^{n \times n}$, with $A^+_{ij} = A_{ij}$ if $A_{ij} \geq 0$ and $A^+_{ij} = 0$ if $A_{ij} < 0$
- $A^- \in \mathbb{R}^{n \times n}$, with $A^-_{ij} = A_{ij}$ if $A_{ij} \leq 0$ and $A^-_{ij} = 0$ if $A_{ij} > 0$
Compute the (unnormalized) positive and negative Laplacian matrices $L^+$ and $L^-$, with
- $L^+ \in \mathbb{R}^{n \times n}$, with $L^+ = D^+ - A^+$ and $D^+ \in \mathbb{R}^{n \times n}$ a diagonal matrix satisfying $D^+_{ii} = \sum_{j=1}^n A^+_{ij}$
- $L^- \in \mathbb{R}^{n \times n}$, with $L^- = D^- - A^-$ and $D^- \in \mathbb{R}^{n \times n}$ a diagonal matrix satisfying $D^-_{ii} = \sum_{j=1}^n A^-_{ij}$
Compute the matrix $Y \in \mathbb{R}^{n \times k}$ made of the eigenvectors corresponding to the $k$ smallest eigenvalues of the matrix $\left( L^- + \tau^+ D^+ \right)^{-1/2} \left( L^+ + \tau^- D^- \right) \left( L^- + \tau^+ D^+ \right)^{-1/2}$, where $\tau^+ > 0$ and $\tau^- > 0$ are regularization parameters²³, and cluster its $n$ rows using the $k$-means algorihm

Cucuringu et al.⁵ establishes the consistency²⁴ of the SPONGE algorithm in the case of $k = 2$ equally-sized clusters, provided $\tau^-$ is sufficiently small²⁵ compared to $\tau^+$⁵ and the number of variables $n$ is sufficiently large²⁶ for a clustering to be recoverable⁵.

In subsequent work, Cucuringu et al.²⁷ establish the consistency²⁴ of a variant of the SPONGE algorithm - called symmetric²⁸ SPONGE - in the case of $k \geq 2$ unequal-sized clusters when $n$ is large enough²⁷.

How to choose the number of clusters in correlation-based spectral clustering?

Choosing the number $k$ of clusters is a general problem for all clustering algorithms, and a variety of more or less successful methods have been devised for this problem¹.

Correlation-based spectral clustering being 1) a clustering method 2) based on the spectrum of a matrix derived from 3) a correlation matrix, there are at least three families of methods which can be used to determine the “optimal” number of clusters:

Generic methods
Specific methods taylored to spectral clustering
Specific methods taylored to correlation-based clustering or to correlation-based factor analysis

However, even if it is mathematically satisfying to find such an optimal number, it is important to keep in mind that just because we find an [optimal] partition […] does not preclude the possibility of other good partitions²⁹.

Indeed, as Wang and Rohe²⁹ puts it:

We must disabuse ourselves of the notion of “the correct partition”. Instead, there are several “reasonable partitions” some of these clusterings might be consistent with one another (as might be imagined in a hierarchical clustering), others might not be consistent.

Generic methods

Any black box method to select an optimal number of clusters in a clustering algorithm - like the silhouette index³⁰ or the gap statistic³¹ - can be used in correlation-based spectral clustering.

On an opinionated note, though, the the elbow criterion should not be used, for reasons detailled in Schubert³².

Specific methods taylored to spectral clustering

In the context of spectral clustering, several specific methods for determining the optimal number of clusters have been proposed.

The most well-known of these methods is the eigengap heuristic¹, whose aim is to choose the number $k$ such that all eigenvalues $\lambda_1$,…,$\lambda_k$ [of the Laplacian matrix] are very small, but $\lambda_{k+1}$ is relatively large¹. In practice, this method works well if the clusters in the data are very well pronounced¹. Nevertheless, the more noisy or overlapping the clusters are, the less effective is this heuristic¹, which is obviously a problem for financial applications³³. Furthermore, just because there is a gap, it doesn’t mean that the rest of the eigenvectors are noise²⁹.

Another popular method is the method used in the self-tuning spectral clustering algorithm of Zelnik-Manor and Perona³⁴, which selects the optimal number of clusters as the number allowing to “best” align the (rows of the) eigenvectors of the Laplacian matrix with the vectors of the canonical basis of $\mathbb{R}^{k}$.

Specific methods taylored to correlation-based clustering or to correlation-based factor analysis

Correlation-based clustering, whether spectral or not, revolves around a very special object - a correlation matrix -, whose properties can be used to find the optimal number of clusters.

Random matrix theory-based methods

From random matrix theory, the distribution of the eigenvalues of a large random correlation matrix follows a particular distribution - called the Marchenko-Pastur distribution - independent³⁵ of the underlying observations.

That distribution involves a threshold $\lambda_{+} \geq 0$ beyond which it is not expected to find any eigenvalue in a random correlation matrix.

Laloux et al.³⁶ uses this threshold to define a correlation matrix denoising procedure called the eigenvalues clipping method, c.f. a previous blog post.

In the context at hand, Jin et al.²¹ proposes to determine the optimal number of clusters as the number of eigenvalues of the correlation matrix that exceed [this threshold], which are the eigenvalues associated with dominant factors or patterns in the [original data]²¹.

To illustrate this method, Figure 5, taken from Laloux et al.³⁶, depicts two different Marchenko-Pastur distributions fitted to a correlation matrix of³⁷ 406 stocks belonging to the S&P 500 index.

Figure 5. Smoothed density of the eigenvalues of the correlation matrix of 406 assets belonging to the S&P 500 and two fitted Marchenko-Pastur distributions, 1991 – 1996. Source: Laloux et al.

From this figure, the number of clusters corresponding to the method of Jin et al.²¹ when applied to the dotted Marchenko-Pastur distribution would be around 15.

Non-linear shrinkage-based methods

De Nard³⁸ notes that non-linear shrinkage [of the sample covariance matrix] pushes the sample eigenvalues toward their closest and most numerous neighbors, thus toward (local) cluster³⁸.

As a consequence, it should be possible to use the information from non-linear shrinkage — the number of jumps in the shrunk eigenvalues — or directly the distribution of the sample eigenvalues — the number of sample eigenvalue clusters — to obtain³⁸ the optimal number of clusters.

This method is illustrated in Figure 6 taken from de Nard³⁸, on which 2 clusters are visible.

Figure 6. Distribution of the sample eigenvalues of a covariance matrix and optimal number of groups corresponding to the centroids of that distribution (0.8 and 2). Source: de Nard.

From a practical perspective, de Nard³⁸ suggests to use 1D KDE clustering in order to determine the number of sample eigenvalue clusters.

Factor analysis-based methods

Although the ultimate goal of cluster analysis and factor analysis is different, the underlying logic of both techniques is dimension reduction (i.e., summarizing information on multiple variables into just a few variables)³⁹.

Based on this similarity, de Nard³⁸ discusses the eigenvalue ratio estimator⁴⁰ of Ahn and Horenstein⁴⁰ that consists in maximizing the ratio of two adjacent eigenvalues [of the sample correlation matrix⁴¹] to determine the number of factors (here clusters)³⁸ in economic or financial data⁴⁰.

Unfortunately, the eigenvalue ratio estimator often cannot identify any cluster [in a large universe of U.S. stocks] and sets $k = 1$³⁸, which corresponds to the situation described in Ahn and Horenstein⁴⁰ where one factor [- here, the market factor -] has extremely strong explanatory power for response variables⁴⁰. In such a situation, the growth ratio estimator⁴⁰ of Ahn and Horenstein⁴⁰ - this time maximizing the ratio of the logarithmic growth rate of two consecutive eigenvalues to determine the number of factors - should be used instead, because it empirically appears to be able to mitigate the effect of the dominant factor⁴⁰.

Yet another estimator of the number of common factors is the adjusted correlation thresholding estimator⁴² of Fan et al.⁴², which determines the number of common factors as the number of eigenvalues greater than 1 of the population correlation matrix […], taking into account the sampling variabilities and biases of top sample eigenvalues⁴².

Implementation in Portfolio Optimizer

Portfolio Optimizer implements correlation-based spectral clustering through the endpoint /assets/clustering/spectral/correlation-based.

This endpoint supports three different methods:

The Blockbuster spectral clustering method
The SPONGE spectral clustering method
The symmetric SPONGE spectral clustering method (default)

As for the number of clusters to use, this endpoint supports:

A manually-defined number of clusters
An automatically-determined number of clusters, computed through a proprietary variation of Horn’s parallel analysis method⁴³ (default)

As a side note, Horn’s parallel analysis method seems to be little known in finance, but numerous studies [in psychology] have consistently shown that [it] is the most nearly accurate methodology for determining the number of factors to retain in an exploratory factor analysis⁴⁴.

Examples of usage

Automated clustering of U.S. stocks

De Nard³⁸ and Jin et al.²¹ both study the automated clustering of a dynamic universe of U.S. stocks through correlation-based⁴⁵ spectral clustering:

De Nard³⁸, in the context of covariance matrix shrinkage

It uses the Blockbuster method, together with a 1D KDE clustering method to automatically determine the number of clusters.
Jin et al.²¹, in the context of the construction of statistical arbitrage portfolios

It uses the SPONGE and SPONGE symmetric methods, together with a Marchenko-Pastur distribution-based method⁴⁶ to automatically determine the number of clusters.

An important remark at this point.

When clustering stocks, it is possible to rely on what de Nard³⁸ calls external information³⁸, for example industry classifications like:

Nevertheless, such an external information may fail to create (a valid number of) homogeneous groups³⁸.

In the words of de Nard³⁸:

For example, if we cluster the covariance matrix into groups of financial and nonfinancial firms, arguably, there will be some nonfinancial firm(s) that has (have) some stock characteristics more similar to the financial stocks as to the non-financial stocks and is (are) therefore misclassified. Especially in large dimension one would expect a few of such misclassifications in both directions. To overcome this misclassification problem and to really create homogeneous groups, [a data-driven procedure should be used instead].

Results of de Nard³⁸ and Jin et al.²¹ are the following:

Within a universe of 500 stocks over the period January 1998 to December 2017, de Nard³⁸ finds that the number of clusters is limited to only 1 or 2 about 75% of the time.
Within a universe of ~600 stocks over the period January 2000 to December 2022, Jin et al.²¹ finds that the number of clusters is relatively stable around 19, even though it tends to drop during financial hardships of the United States²¹ down to a minimum of 8, as displayed in Figure 7 adapted from Jin et al.²¹.

Figure 7. Evolution of the number of clusters found by a Marchenko-Pastur distribution-based method, January 2000 - December 2022. Source: Jin et al.

These diverging results on closely related data sets show that different methods to determine the number of clusters to use in clustering analysis might be completely at odds with one another⁴⁷.

But these results also show that the “optimal” number of clusters - whatever it is - is not constant over time and that methods that dynamically determine [it] can capture changes in market dynamics, especially when there is significant downside risks in the market²¹. Incidentally, this is another argument in favor of not relying on (slowly evolving) external information to determine the number of clusters to use.

Identification of risk-on/risk-off assets in a U.S.-centric universe of assets

In April 2025, “risk on/risk off” has been making the headlines as a phrase describing investment and asset price behavior⁴⁸.

Lee⁴⁸ summarizes the underlying meaning as follows⁴⁸:

[…] risk on/risk off generally refers to an investment environment in which asset price behavior is largely driven by how risk appetite advances or retreats over time, usually in a synchronized way at a faster than normal pace across global regions and assets.

And describes it in more details as follows⁴⁸:

Depending on the environment, investors tend to buy or sell risky assets across the board, paying less attention to the unique characteristics of these assets. Volatilities and, most noticeably, correlations of assets that are perceived as risky will jump, particularly during risk-off periods, inspiring comments such as “correlations go to one” during a crisis. Assets, such as U.S. Treasury bonds, and currencies, such as the Japanese yen, tend to move in the opposite direction of risky assets and are generally perceived as the safer assets to hold in the event of a flight to safety.

In this sub-section, I propose to study the capability of correlation-based spectral clustering to identify risk-on/risk-off assets in a U.S.-centric universe of assets.

For this:

I will use as a universe of assets 13 ETFs representative⁴⁹ of misc. asset classes:
- U.S. stocks (SPY ETF)
- European stocks (EZU ETF)
- Japanese stocks (EWJ ETF)
- Emerging markets stocks (EEM ETF)
- U.S. REITs (VNQ ETF)
- International REITs (RWX ETF)
- Cash (SHY ETF)
- U.S. 7-10 year Treasuries (IEF ETF)
- U.S. 20+ year Treasuries (TLT ETF)
- U.S. Investment Grade Corporate Bonds(LQD ETF)
- U.S. High Yield Corporate Bonds (HYG ETF)
- Commodities (DBC ETF)
- Gold (GLD ETF)
I will compute the correlation matrix of the daily returns of these assets over the period 1st April 2025 - 30th April 2025⁵⁰

Figure 8 displays that correlation matrix using a planar t-SNE representation of its rows, considered as points in $\mathbb{R}^{13}$.

Figure 8. t-SNE representation of the asset correlation matrix, April 2025.

To be noted that the t-SNE representation of Figure 8 is a bit misleading in the context of spectral clustering, because spectral clustering does not operate in the t-SNE plane. Still, it is an helpful representation to give a sense of asset “closeness” in terms of correlations.
I will cluster these assets using the symmetric SPONGE spectral clustering method with $k = 2$ and $k = 3$ clusters, the rationale being that:
- All the assets clustered together with Cash should correspond to risk-off assets ($k = 2,3$)
- All the assets clustered together with U.S. stocks should correspond to risk-on assets ($k = 2,3$)
- All the assets clustered in the remaining cluster should correspond to a another category to be determined ($k = 3$ only)

Figures 9 and 10 show the resulting clusterings.

Figure 9. Symmetric SPONGE clustering of the asset correlation matrix, two clusters, April 2025.

Figure 10. Symmetric SPONGE clustering of the asset correlation matrix, three clusters, April 2025.

From these figures, and over the considered period:

Intermediate-term U.S. Treasuries is found to be a risk-off asset when $k = 2$
Gold is found to be a risk-off asset when $k = 3$, with Intermediate-term and Long-term U.S. Treasuries now moved into their own cluster

The evolution of the classification of Gold as a risk-off asset is particularly interesting⁵¹ and also serves to illustrate one of the limitations of the t-SNE representation of the asset correlation matrix.

Indeed, from that representation, it makes no sense that Cash and Gold could belong to the same cluster since they are complete opposites in the t-SNE plane! But in reality, if the t-SNE plane is diagonally “folded”, Cash and Gold truly are close neighbors…

Conclusion

This (abruptly?) concludes this already too long overview of correlation-based spectral clustering.

Waiting for the next blog post on correlation-based clustering, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See von Luxburg, U. A tutorial on spectral clustering. Stat Comput 17, 395–416 (2007). ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²
See R. N. Mantegna, H. E. Stanley, Introduction to econophysics: correlations and complexity in finance, Cambridge university press, 1999. ↩
See Mantegna, R.N. Hierarchical structure in financial markets. Eur. Phys. J. B 11, 193–197 (1999). ↩ ↩² ↩³ ↩⁴
See Brownlees, C., Guðmundsson, G. S., & Lugosi, G. (2020). Community Detection in Partial Correlation Network Models. Journal of Business & Economic Statistics, 40(1), 216–226. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰
See Mihai Cucuringu, Peter Davies, Aldo Glielmo, and Hemant Tyagi. SPONGE: A generalized eigenproblem for clustering signed networks. In Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1088–1098. PMLR, 2019. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹
$x_1, …, x_n$ can be more general “objects”, as long as a distance between these objects is defined. ↩
As opposed to their global relationships that already available from the similarity matrix $S$. ↩
See Jin Huang, Feiping Nie, and Heng Huang. 2013. Spectral rotation versus K-means in spectral clustering. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI’13). AAAI Press, 431–437. ↩
The similarity function $s$ is also very important in spectral clustering, although ultimately, the choice of the similarity function depends on the domain the data comes from, and no general advice can be given¹. ↩
See Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. 2001. On spectral clustering: analysis and an algorithm. In Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic (NIPS’01). MIT Press, Cambridge, MA, USA, 849–856. ↩ ↩² ↩³ ↩⁴ ↩⁵
Thanks to the properties of the Laplacian matrix $L$. ↩
Typically the Pearson correlation, but other measures like the Spearman correlation or the Gerber correlation can also be used as long as the resulting correlation matrix $C$ is a valid correlation matrix. ↩
It is actually possible to define a whole family of metrics as function of the Pearson or Spearman correlation coefficient, c.f. van Dongen and Enright⁵². ↩
Such correlation-based metrics are also used elsewere in finance, for example in de Prado’s⁵³ Hierarchical Risk Parity portfolio optimization algorithm. ↩
To be noted that depending on the exact spectral clustering method used, it might be required to first convert that distance to a similarity measure. ↩
Called the Generalised Stochastic Block Model in Brownlees et al.⁴, which is an extension of the vanilla stochastic bloc model. ↩
Or of the sample correlation matrix, since a correlation matrix is a covariance matrix of a special kind. ↩
Both the simulation study and the empirical application in Brownlees et al.⁴ show that the Blockbuster algorithm already performs satisfactorily⁴ with $n = 50$, depending on the strength of the correlations; “sufficiently” large is thus not necessarily “very” large, that is, the Blockbuster algorithm seems to be well-behaved in finite sample. ↩
The definition of “appropriately controlled” is left for future research in Brownlees et al.⁴. ↩
In particular, this formula explains why the $k$ “largest” eigenvectors of the matrix $\Sigma$ are extracted by the Blockbuster algorithm - it is because they correspond to the $k$ “smallest” eigenvectors of the matrix $\Sigma^{-1}$ (and of the matrix $L$). ↩
See Qi Jin, Mihai Cucuringu, and Álvaro Cartea. 2023. Correlation Matrix Clustering for Statistical Arbitrage Portfolios. In Proceedings of the Fourth ACM International Conference on AI in Finance (ICAIF ‘23). Association for Computing Machinery, New York, NY, USA, 557–564. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²
A Python implementation of the SPONGE and symmetric SPONGE algorithms is available at https://github.com/alan-turing-institute/signet. ↩
These two regularization parameters aim to promote clusterizations that avoid small-sized clusters²¹. ↩
Under a signed stochastic bloc model⁵, which is an extension of the vanilla stochastic bloc model. ↩ ↩²
In numerical experiments, Cucuringu et al.⁵ uses $\tau^+$ = $\tau^-$ = 1, which nearly always falls within the region of maximum recovery when it is present⁵. ↩
In numerical experiments, Cucuringu et al.⁵ uses $n$ ranging from 50 to 11259. ↩
See Mihai Cucuringu, Apoorv Vikram Singh, Deborah Sulem, and Hemant Tyagi. 2021. Regularized spectral methods for clustering signed networks. Journal of Machine Learning Research 22, 264 (2021), 1–79. ↩ ↩²
The symmetric SPONGE algorithm uses symmetric positive and negative Laplacian matrices $L^+_{sym} = \left( D^+ \right)^{-1/2} L^+ \left( D^+ \right)^{-1/2}$ and $L^-_{sym} = \left( D^- \right)^{-1/2} L^- \left( D^- \right)^{-1/2}$ instead of unnormalized ones and the $k$ smallest eigenvalues of the matrix $\left( L^-_{sym} + \tau^+ I_n \right)^{-1/2} \left( L^+_{sym} + \tau^- I_n \right) \left( L^-_{sym} + \tau^+ I_n \right)^{-1/2}$, c.f. Cucuringu et al.⁵. ↩
See Song Wang. Karl Rohe. “Discussion of “Coauthorship and citation networks for statisticians”.” Ann. Appl. Stat. 10 (4) 1820 - 1826, December 2016. ↩ ↩² ↩³
See Peter J. Rousseeuw (1987). Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis. Computational and Applied Mathematics. 20: 53–65. ↩
See Tibshirani, R., Walther, G., and Hastie, T. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 2 (2001), 411–423. ↩
See Erich Schubert. 2023. Stop using the elbow criterion for k-means and how to choose the number of clusters instead. SIGKDD Explor. Newsl. 25, 1 (June 2023), 36–42. ↩
For example, de Nard³⁸ finds that for the estimation problem of asset return covariance matrices, the eigengap heuristic often cannot identify any cluster and sets $k=1$³⁸. ↩
See Zelnik-Manor, L. and P. Perona (2004). Self-tuning spectral clustering. In Advances in neural information processing systems, pp. 1601–1608. ↩
Under suitables technical assumptions. ↩
See Laurent Laloux, Pierre Cizeau, Jean-Philippe Bouchaud, and Marc Potters, Noise Dressing of Financial Correlation Matrices, Phys. Rev. Lett. 83, 1467. ↩ ↩²
The returns of. ↩
See Gianluca de Nard, Oops! I Shrunk the Sample Covariance Matrix Again: Blockbuster Meets Shrinkage, Journal of Financial Econometrics, Volume 20, Issue 4, Fall 2022, Pages 569–611. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸
See Hedwig Hofstetter, Elise Dusseldorp, Pepijn van Empelen, Theo W.G.M. Paulussen, A primer on the use of cluster analysis or factor analysis to assess co-occurrence of risk behaviors, Preventive Medicine, Volume 67, 2014, Pages 141-146. ↩
See Ahn, S., and A. Horenstein. 2013. Eigenvalue Ratio Test for the Number of Factors. Econometrica 80: 1203–1227. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸
The eigenvalue ratio estimator and the growth ratio estimator of Ahn and Horenstein⁴⁰ both rely on the covariance matrix, of which the correlation matrix is a special kind. In addition, for reasons detailled in Fan et al.⁴², using the covariance matrix to determine the number of factors to retain in a factor analysis is generally a bad idea. ↩
See Fan, J., Guo, J., & Zheng, S. (2020). Estimating Number of Factors by Adjusted Eigenvalues Thresholding. Journal of the American Statistical Association, 117(538), 852–861. ↩ ↩² ↩³ ↩⁴
See Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. ↩
See Glorfeld, L. W. (1995). An Improvement on Horn’s Parallel Analysis Methodology for Selecting the Correct Number of Factors to Retain. Educational and Psychological Measurement, 55(3), 377-393. ↩
Due to the context of the paper, de Nard³⁸ uses covariance-based spectral clustering. ↩
Jin et al.²¹ also uses another method to determine the number of clusters, based on the percentage of variance explained by selecting the top $k$ eigenvalues of the asset correlation matrix. ↩
To be noted, though, that Jin et al.²¹’s methodology to compute the asset correlation matrix is different from de Nard³⁸’s, especially in that Jin et al.²¹ consider residual asset returns and not raw asset returns. Such a difference could - and actually, should - influence the computation of the optimal number of clusters, whatever the method used. Nevertheless, my personal experience with de Nard³⁸’s 1D KDE clustering method is that it is anyway too sensitive to its parameters (kernel and bandwidth) to be confidently used in practice. ↩
See Lee, Wai, Risk On/Risk Off, The Journal of Portfolio Management Spring 2012, 38 (3) 28-39. ↩ ↩² ↩³ ↩⁴
10 of these ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer⁵⁴. ↩
(Adjusted) prices have have been retrieved using Tiingo. ↩
My own interpretation is that if one would prefer to avoid investing in Intermediate-term U.S. Treasuries, Gold would then represents the “closest” risk-off asset. ↩
See Stijn van Dongen, Anton J. Enright, Metric distances derived from cosine similarity and Pearson and Spearman correlations, arXiv. ↩
Lopez de Prado, M. (2016). Building diversified portfolios that outperform out-of-sample. Journal of Portfolio Management, 42(4), 59–69. ↩
See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩

Volatility Forecasting: HExp Model

2025-03-09T00:00:00-06:00

In this series on volatility forecasting, I previously detailed the Heterogeneous AutoRegressive (HAR) volatility forecasting model that has become the workhorse of the volatility forecasting literature¹ since its introduction by Corsi².

I will now describe an extension of that model due to Bollerslev et al.³, called the Heterogeneous Exponential (HExp) volatility forecasting model, in which the lagged HAR volatility components are exponentially - rather than arithmetically - averaged.

In addition, I will also discuss the panel-based estimation procedure for the HExp and the HAR model parameters proposed in Bollerslev et al.³, which is empirically demonstrated³ to improve the out-of-sample forecasting performances of these two volatility forecasting models when compared to the standard² individual asset-based procedure.

Finally, I will illustrate the practical performances of the HExp volatility forecasting model and its panel-based parameters estimation procedure in the context of monthly volatility forecasting for various ETFs.

Mathematical preliminaries (reminders)

This section contains reminders from the first blog post of this series.

Volatility modelling and volatility proxies

Let $r_t$ be the logarithmic return of an asset over a time period $t$ (a day, a week, a month..), over which its (conditional) mean return is supposed to be null.

Then:

The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $

From this definition, the squared return $r_t^2$ of an asset is a (noisy⁴) variance estimator - or variance proxy⁴ - for that asset variance over the considered time period.

Another example of an asset variance proxy is the Parkinson range of an asset.

Yet another example of an asset variance proxy, this time over a specific time period $t$ of one day, is the daily realized variance $RV_t$, which is defined as the sum of the asset squared intraday returns sampled at a high frequency (1 minutes, 5 minutes, 15 minutes…).

The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$.
The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $

The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$.

Weighted moving average volatility forecasting model

Boudoukh et al.⁵ shows that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with

\[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\]

, where:

$k$, with $1 \leq k \leq T$, is the size of the moving average, possibly time-dependent
$w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well

The original HAR volatility forecasting model

The HAR volatility forecasting model is an additive cascade model of different volatility components² subject to economically meaningful restrictions².

Under that model, an asset next day’s daily realized variance $RV_{T+1}$ is forecasted through the formula⁴:

\[\hat{RV}_{T+1} = \beta + \beta_d RV_{T} + \beta_w RV_{T}^w + \beta_m RV_{T}^m\]

, where:

$\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$
$RV_T$ is the asset daily realized variance at time $T$
$RV_T^w = \frac{1}{5} \sum_{i=1}^5 RV_{T-i+1}$ is the asset weekly realized variance at time $T$
$RV_T^m = \frac{1}{22} \sum_{i=1}^{22} RV_{T-i+1}$ is the asset monthly realized variance at time $T$
$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined

The HExp volatility forecasting model

Discontinuity of the HAR volatility forecasting model

Under the HAR volatility forecasting model, forecasted future volatilities depends on the past volatilities in a way that is [dis]continuous […] in the lag lengths³ due to the presence of simple moving averages, which might lead to potential variance estimation issues³.

As noted by Bollerslev et al.³:

The stepwise nature of the volatility factors employed in the HAR models, imply that the forecasts from the models are subject to potentially abrupt changes as an unusually large/small daily lagged [realized variance] drops out of the sums for the longer-horizon lagged volatility factors.

Figure 1, adapted from Bollerslev et al.³, illustrates the lag coefficients implied by the regression coefficients³ of the HAR model together with those of a 21-day simple moving average model.

Figure 1. Lag coefficients of the HAR and of the 21-day simple moving average volatility forecasting models. Source: Bollerslev et al.

The discontinuity of the HAR model at the 1-day, 5-day and 21-day lags⁶ is apparent, similar in spirit to the discontinuity of the 21-day simple moving average model at the 21-day lag.

The original HExp volatility forecasting model

In order to avoid the stepwise changes inherent in the forecast from the HAR component-type structure³, Bollerslev et al.³ proposes to replace the simple moving averages appearing in the HAR model by exponentially weighted moving averages.

Under the resulting volatility forecasting model - denoted the Heterogenous Exponential realized volatility model (HExp for short)³ - an asset next day’s daily realized variance $RV_{T+1}$ is forecasted through the formula³⁷

\[\hat{RV}_{T+1} = \beta + \beta_d ExpVP_T^{\lambda(1)} + \beta_w ExpVP_T^{\lambda(5)} + \beta_m ExpVP_T^{\lambda(25)} + \beta_h ExpVP_T^{\lambda(125)}\]

, where:

$\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$
$ExpVP_T^{\lambda(CoM)}$ $=$ $\sum_{i=1}^T \frac{e^{-i \lambda(CoM)}}{\sum_{j=1}^T e^{-j \lambda(CoM)}} RV_{T+1-i} $
$\lambda \left(CoM\right)$ $=$ $\log \left( 1 + \frac{1}{CoM} \right)$, with CoM standing for center-of-mass³
$RV_i$ is the asset daily realized variance at time $i$, $i=1..T$
$\beta$, $\beta_d$, $\beta_w$, $\beta_m$ and $\beta_h$ are the HExp model parameters, to be determined

To be noted that each center-of-mass used in the HExp model (1, 5, 25 and 125) effectively summarizes the “average” horizon of the lagged realized volatilities that it uses³ and that they all have been chosen in Bollerslev et al.³ so as to “span” the universe of past [realized variance]’s in a way that is both parsimonious and “smooth”³.

Speaking of smoothness, Figure 2, again adapted from Bollerslev et al.³, compares the lag coefficients implied by the regression coefficients³ of the HAR model with those of the HExp model.

Figure 2. Lag coefficients of the HAR and of the HExp volatility forecasting models. Source: Bollerslev et al.

The continuous nature of the HExp volatility forecasting model is clearly visible.

In terms of practical performances, the HExp model perform[s] well in out-of-sample risk forecasting³ and is even slightly more performant than the HAR model in terms of $r$-squared, as can be seen on Figure 3 adapted from Bollerslev et al.³.

Figure 3. Out-of-sample $r$-squared of the HAR model v.s. the HExp model for predicting the 20-day future realized volatility of several assets and for different methods of parameters estimation (Ind, Panel, Mega, that will be discussed in the next section). Source: Bollerslev et al.

Realized variance v.s. generic variance proxy

The original HExp model described in the previous subsection relies on a very specific asset variance proxy - the realized variance of an asset - over a very specific time period - a day - for its definition.

Similarly to the HAR model, it is possible to replace the daily realized variance by any generic daily variance estimator like daily squared returns⁸ or any daily range-based variance estimator (Parkinson, Garman-Klass, Rogers-Satchell…).

This leads to the generic HExp volatility forecasting model, under which an asset next days’s conditional variance $\sigma_{T+1}^2$ is forecasted through the formula

\[\hat{\sigma}_{T+1}^2 = \beta + \beta_d ExpVP_T^{\lambda(1)} + \beta_w ExpVP_T^{\lambda(5)} + \beta_m ExpVP_T^{\lambda(25)} + \beta_h ExpVP_T^{\lambda(125)}\]

, where:

$\hat{\sigma}_{T+1}^2$ is the forecast at time $T$ of the asset next day’s conditional variance $\sigma_{T+1}^2$
$ExpVP_T^{\lambda(CoM)} = \sum_{i=1}^T \frac{e^{-i \lambda(CoM)}}{\sum_{j=1}^T e^{-j \lambda(CoM)}} \tilde{\sigma}^2_{T+1-i} $
$\lambda \left(CoM\right) = \log \left( 1 + \frac{1}{CoM} \right) $
$\tilde{\sigma}^2_{i}$ is the asset daily variance estimator at time $i$, $i=1..T$
$\beta$, $\beta_d$, $\beta_w$, $\beta_m$ and $\beta_h$ are the HExp model parameters, to be determined

Relationship with the generic weighted moving average model

From its definition, it is not too difficult to see that the HExp volatility forecasting model is a specific kind of weighted moving average volatility forecasting model, with:

$w_0 = \beta$
$w_1 = \beta_d \frac{e^{- \lambda(1)}}{\sum_{j=1}^T e^{-j \lambda(1)}}$ $+$ $\beta_w \frac{e^{- \lambda(5)}}{\sum_{j=1}^T e^{-j \lambda(5)}}$ $+$ $\beta_m \frac{e^{- \lambda(25)}}{\sum_{j=1}^T e^{-j \lambda(25)}}$ $+$ $\beta_h \frac{e^{- \lambda(125)}}{\sum_{j=1}^T e^{-j \lambda(125)}} $
$w_2 = …$

Volatility forecasting formulas

Under an HExp volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes:

To estimate an asset next day’s volatility:
\[\hat{\sigma}_{T+1} = \sqrt{ \beta + \beta_d ExpVP_T^{\lambda(1)} + \beta_w ExpVP_T^{\lambda(5)} + \beta_m ExpVP_T^{\lambda(25)} + \beta_h ExpVP_T^{\lambda(125)} }\]
To estimate an asset next $h$-day’s ahead volatility⁹, $h \geq 2$, using an indirect¹ multi-step ahead forecast scheme:
\[\hat{\sigma}_{T+h} = \sqrt{ \beta + \beta_d ExpVP_{T+h-1}^{\lambda(1)} + \beta_w ExpVP_{T+h-1}^{\lambda(5)} + \beta_m ExpVP_{T+h-1}^{\lambda(25)} + \beta_h ExpVP_{T+h-1}^{\lambda(125)} }\]

, where:

$ExpVP_{T+h-1}^{\lambda(CoM)}$ $=$ $\sum_{i=1}^T \frac{e^{-i \lambda(CoM)}}{\sum_{j=1}^{T+h-1} e^{-j \lambda(CoM)}} \tilde{\sigma}^2_{T+1-i} $ $+$ $\sum_{i=1}^{h-1} \frac{e^{-i \lambda(CoM)}}{\sum_{j=1}^{T+h-1} e^{-j \lambda(CoM)}} \hat{\sigma}^2_{T+h-i} $
To estimate an asset aggregated volatility⁹ over the next $h$ days:
\[\hat{\sigma}_{T+1:T+h} = \sqrt{ \sum_{i=1}^{h} \hat{\sigma}^2_{T+i} }\]

Estimating the HExp model parameters

Individual estimation

As for the HAR model, the easiest way to estimate the HExp model parameters is by applying simple linear regression² on an asset-by-asset basis³, in which case the asset-specific ordinary least squares (OLS) estimator of the parameters $\beta$, $\beta_d$, $\beta_w$, $\beta_m$ and $\beta_h$ at time $T$ is the solution of the minimization problem⁴

\[\argmin_{ \left( \beta, \beta_d, \beta_w, \beta_m, \beta_h \right) \in \mathbb{R}^{5}} \sum_{t=1}^T \left( \tilde{\sigma}_{t}^2 - \beta - \beta_d ExpVP_{t-1}^{\lambda(1)} - \beta_w ExpVP_{t-1}^{\lambda(5)} - \beta_m ExpVP_{t-1}^{\lambda(25)} - \beta_h ExpVP_{t-1}^{\lambda(125)} \right)^2\]

Alternatively, following Clements and Preve⁴ and Clements et al.¹, more complex asset-specific least squares estimators than OLS can be used to try to improve forecast performances (weighted least squares estimators (WLS), robust least squares estimators (RLS)…)

Panel-based estimation

Bollerslev et al.³ establishes that the dynamics of realized volatility are common across many different financial assets.

This is illustrated in Figure 4, directly taken from Bollerslev et al.³, which depicts the unconditional distributions of daily normalized realized volatilities for different asset classes.

Figure 4. Normalized unconditional daily realized variance distributions for misc. asset classes. Source: Bollerslev et al.

From this figure, volatility indeed seems to behave similarly across asset classes³ and Bollerslev et al.³ proposes to exploit these strong similarities in the distributions of the volatilities across and within asset classes³ by using panel regression techniques that force the [HExp model parameters] to be the same within and across different asset classes³.

In more details, Bollerslev et al.³ reformulates the generic HExp volatility forecasting model as follows:

\[\hat{\sigma}_{T+1}^2 = \tilde{\sigma}_{T}^{2, LR} + \beta_d^P \left( ExpVP_T^{\lambda(1)} - \tilde{\sigma}_{T}^{2, LR} \right) + \beta_w^P \left( ExpVP_T^{\lambda(5)} - \tilde{\sigma}_{T}^{2, LR} \right) + \beta_m^P \left( ExpVP_T^{\lambda(25)} - \tilde{\sigma}_{T}^{2, LR} \right) + \beta_h^P \left( ExpVP_T^{\lambda(125)} - \tilde{\sigma}_{T}^{2, LR} \right)\]

, where:

$ \tilde{\sigma}_{T}^{2, LR}$ is a long-run volatility factor, equal to the expanding sample mean of [the asset daily variance estimator] from the start of the sample up until day $T$³.
$\beta_d^P$, $\beta_w^P$, $\beta_m^P$ and $\beta_h^P$ are the HExp “panel” model parameters, to be determined

Such a reformulation - called centering³ in Bollerslev et al.³ - eliminat[es] the level of the [asset] volatility³ from the HExp model and enables¹⁰ the parameters $\beta_d^P$, $\beta_w^P$, $\beta_m^P$ and $\beta_h^P$ to be estimated simultaneously for all assets by panel regression techniques that add power by exploiting the similarities in the cross-asset risk characteristics³.

Additionally¹¹, that specific reformulation ensures that the iterated long-run forecasts from the model constructed on day $T$ converges to this day $T$ estimate of the “unconditional” volatility³.

From a practical perspective, Figure 3 shows that estimating the HExp parameters¹² through panel-based estimation (lines Panel and Mega) leads to much better performances v.s. their individual estimation (lines Ind).

Implementation in Portfolio Optimizer

Portfolio Optimizer implements the HExp volatility forecasting model - together with all the extensions of its predecessor (the insanity filter described in Clements and Preve⁴, the log transformation…) - through the endpoint /assets/volatility/forecast/hexp.

This endpoint supports the 4 variance proxies below:

Squared close-to-close returns
Demeaned squared close-to-close returns
The Parkinson range
The jump-adjusted Parkinson range

This endpoint also supports:

Individual and panel-based estimation of the HExp model parameters.
Using up to 5 centers-of-mass for the variance proxies, the default ones being 1, 5 and 25¹³.

Example of usage - Volatility forecasting at monthly level for various ETFs

As an example of usage, I propose to enrich the results of the previous blog post, in which monthly forecasts produced by different volatility models¹⁴ are compared - using Mincer-Zarnowitz¹⁵ regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative¹⁶ of misc. asset classes:

U.S. stocks (SPY ETF)
European stocks (EZU ETF)
Japanese stocks (EWJ ETF)
Emerging markets stocks (EEM ETF)
U.S. REITs (VNQ ETF)
International REITs (RWX ETF)
U.S. 7-10 year Treasuries (IEF ETF)
U.S. 20+ year Treasuries (TLT ETF)
Commodities (DBC ETF)
Gold (GLD ETF)

Individual estimation

Averaged results for all ETFs/regression models over each ETF price history¹⁷ are the following¹⁸, when adding the HExp volatility forecasting model and its log variation¹⁹:

Volatility model	Variance proxy	$\bar{\alpha}$	$\bar{\beta}$	$\bar{R^2}$
EWMA, optimal $\lambda$	Squared close-to-close returns	4.7%	0.73	45%
HAR	Squared close-to-close returns	-0.7%	0.95	46%
HAR (log)	Squared close-to-close returns	0.5%	0.62	40%
HExp	Squared close-to-close returns	-0.7%	0.93	48%
HExp (log)	Squared close-to-close returns	2.1%	0.57	42%
EWMA, optimal $\lambda$	Parkinson range	4.3%	1.06	48%
HAR	Parkinson range	0.1%	1.25	44%
HAR (log)	Parkinson range	1.9%	1.22	50%
HExp	Parkinson range	-0.5%	1.29	47%
HExp (log)	Parkinson range	2.0%	1.21	51%
EWMA, optimal $\lambda$	Jump-adjusted Parkinson range	4.0%	0.76	45%
HAR	Jump-adjusted Parkinson range	-1.4%	0.99	47%
HAR (log)	Jump-adjusted Parkinson range	0.9%	0.92	51%
HExp	Jump-adjusted Parkinson range	-1.5%	0.98	49%
HExp (log)	Jump-adjusted Parkinson range	1.1%	0.91	52%

Panel-based estimation

Averaged results for all ETFs/regression models over the common ETF price history²⁰ are the following¹⁸:

When using the EWMA, HAR and HExp volatility forecasting models with an asset-specific parameters estimation procedure (for reference):

Volatility model	Variance proxy	$\bar{\alpha}$	$\bar{\beta}$	$\bar{R^2}$
EWMA, optimal $\lambda$ (individual est.)	Squared close-to-close returns	5%	0.72	43%
HAR (individual est.)	Squared close-to-close returns	-0.2%	0.89	46%
HExp (individual est.)	Squared close-to-close returns	0.02%	0.87	47%
EWMA, optimal $\lambda$ (individual est.)	Parkinson range	4.8%	1.02	45%
HAR (individual est.)	Parkinson range	0.7%	1.17	46%
HExp (individual est.)	Parkinson range	1.2%	1.15	47%
EWMA, optimal $\lambda$ (individual est.)	Jump-adjusted Parkinson range	4.5%	0.73	43%
HAR (individual est.)	Jump-adjusted Parkinson range	-0.9%	0.94	46%
HExp (individual est.)	Jump-adjusted Parkinson range	-0.4%	0.90	46%

When using the HAR and HExp volatility forecasting models with a panel-based parameters estimation procedure comparable to the Mega procedure described in Bollerslev et al.³:

Volatility model	Variance proxy	$\bar{\alpha}$	$\bar{\beta}$	$\bar{R^2}$
HAR (panel est.)	Squared close-to-close returns	2.2%	0.76	47%
HExp (panel est.)	Squared close-to-close returns	1.1%	0.80	47%
HAR (panel est.)	Parkinson range	3.1%	1.11	50%
HExp (panel est.)	Parkinson range	3.6%	1.08	50%
HAR (panel est.)	Jump-adjusted Parkinson range	0.09%	0.88	46%
HExp (panel est.)	Jump-adjusted Parkinson range	-0.07%	0.89	48%

Comments

From the results of the two previous subsections, it is possible to make the following comments:

Consistent with Bollerslev et al.³, the HExp model is uniformly better than the HAR model in terms of r-squared.
Contrary to Bollerslev et al.³, the panel-based estimation procedure does not seem to dramatically improve the HAR/HExp models performances, except when the Parkinson range is used as a daily variance proxy.

Comparing lines #1,2,5,6 with lines #3,4 suggests to perform the same test with (high frequency) realized variances in order to confirm that this is due to the “quality” of the daily variance proxy used.

Conclusion

This blog post empirically confirmed that the HExp volatility forecasting model of Bollerslev et al.³ belongs to the category of the state-of-the-art dynamic [risk models]³ published in the litterature.

This blog post also concludes this series on volatility forecasting by weighted moving average models, at least until I find a better such model than the HExp model.

Waiting for that to happen or for a blog post on volatility forecasting by a non-weighted moving average model, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Clements, Adam and Preve, Daniel P. A. and Tee, Clarence, Harvesting the HAR-X Volatility Model. ↩ ↩² ↩³
See Fulvio Corsi, A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics, Volume 7, Issue 2, Spring 2009, Pages 174–196. ↩ ↩² ↩³ ↩⁴ ↩⁵
See Tim Bollerslev, Benjamin Hood, John Huss, Lasse Heje Pedersen, Risk Everywhere: Modeling and Managing Volatility, The Review of Financial Studies, Volume 31, Issue 7, July 2018, Pages 2729–2773. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³ ↩³⁴ ↩³⁵ ↩³⁶ ↩³⁷ ↩³⁸ ↩³⁹
See Adam Clements, Daniel P.A. Preve, A Practical Guide to harnessing the HAR volatility model, Journal of Banking & Finance, Volume 133, 2021. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
See Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71. ↩
These lags correspond to the daily, weekly and monthly realized variance components of the original HAR model. ↩
In Bollerslev et al.³, the calculation of $ExpVP_T^{\lambda(CoM)}$ uses only the first 500 lags (i.e., is truncated to $T=500$) because the influence of the remaining lags is numerically immaterial³. ↩
Bollerslev et al.³ discusses the impact of replacing realized volatilities by daily squared returns in the HExp model. ↩
See Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22. ↩ ↩²
Eliminating the level of the asset volatility is a pre-requisite in order to use panel regression techniques because the very different volatility levels for different asset classes means that³ 1) it is unreasonable to force the $\beta$ intercepts to be the same³ for all assets and 2) it is necessary to ensure that all [the remaining] parameters are “scale-free” in the sense that they do not depend on the level of risk³. ↩
Thanks to that reformulation, the coefficients $\beta_d$, $\beta_w$, $\beta_m$ and $\beta_h$ are also free (i.e., need not sum to one)³, which allows an easy fitting through OLS. ↩
As a side note, the centering reformulation described in Bollerslev et al.³ is also applicable to the HAR volatility forecasting model and is done in Bollerslev et al.³. ↩
Contrary to Bollerslev et al.³, the center-of-mass 125 is not included by default in the HExp model as implemented in Portfolio Optimizer; this choice was made to make the default HExp model implementation directly comparable with the default HAR model implementation. ↩
Using Portfolio Optimizer. ↩
See Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations. ↩
These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer²¹. ↩
The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. ↩
For all models, I used an expanding window for the volatility forecast computation. ↩ ↩²
The log HExp model is similar in spirit to the log HAR model described in the previous blog post. ↩
The common starting price history of the ETFs is 31 July 2007 and their common ending price history is 31 August 2023. ↩
See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩

The Mathematics of Portfolio Return: Simple Return, Money-Weighted Return and Time-Weighted Return

2025-02-05T00:00:00-06:00

Whether we manage our own investment assets or choose to hire others to manage the assets on our behalf we are keen to know how well our […] portfolio of assets is performing¹ and the calculation of portfolio return is the first step in [that] performance measurement process¹.

Now, while the matter of measuring the rate of return of [a portfolio] appears, on the surface, to be simple enough², the presence of external cash flows - either contributions to the portfolio or withdrawals from the portfolio - leads to the definition of several rate of returns, with no single rate of return measure [being] appropriate for every purpose².

In this blog post, strongly inspired by the book Practical Portfolio Performance Measurement and Attribution² from Carl Bacon, I will describe the two main methods of portfolio return calculation in the presence of external cash flows.

As an example of usage, I will illustrate the investor gap in the case of a MSCI World ETF, that is, I will show that the returns of that ETF are different from the returns achieved by the average investor in that ETF.

Notes:

A fully functional Google sheet corresponding to this post is available here

External v.s. internal cash flows

In the context of portfolio return calculation, two types of cash flows are usually distinguished:

External cash flows, corresponding to any new money added to or taken from [a] portfolio, whether in the form of cash or other assets¹
Internal cash flows, corresponding to transactions funded from within [a] portfolio¹ like dividend and coupon payments, free shares attributed by companies, positions rebalancing, etc.

In other words, external cash flows impact the valuation of a portfolio other than by the growth (positive or negative) of the funds invested in that portfolio while internal cash flows have no impact on the portfolio valuation.

Portfolio return in the absence of external cash flows

Let be:

$T$, the number of observations of the value of a portfolio over a time period $1..T$, with $t_1 > t_2 > … > t_T$ the $T$ observation times
$V_1$ the value of the portfolio at the initial observation time $t_1$
$V_T$ the value of the portfolio at the final observation time $t_T$

When no external cash flow occurred over a time period, the return $r_p$ of a portfolio over that period is defined as the change of the portfolio value relative to its beginning value³.

Mathematically, using the notations above, this gives:

\[r_p = \frac{V_T - V_1}{V_1} = \frac{V_T}{V_1} - 1\]

$r_p$ is called the simple rate of return or the arithmetic rate of return of the portfolio over the time period $1..T$.

To be noted that, by definition, the arithmetic return of a portfolio over a time period is supposed to take into account any internal cash flow out of the portfolio constituents⁴ (dividends for stocks, coupon payments for bonds…); for this reason, the arithmetic return of a portfolio is also sometimes called the total (arithmetic) return of the portfolio⁵.

Portfolio return in the presence of external cash flows

Let now be:

$T$, the number of observations of the value of a portfolio over a time period $1..T$, with $t_1 > t_2 > … > t_T$ the $T$ observation times
$V_i$ the value of the portfolio at the observation time $t_i$, $i=1..T$
$C_i$ the value of a potential external cash flow ($C_i > 0$ for a contribution, $C_i < 0$ for a withdrawal) at the observation time $t_i$, $i=1..T$, assumed to occur immediately after the observation time $t_i$ (i.e., the cash flow $C_i$ is assumed to be excluded from the portfolio value $V_i$)

When an external cash flow occurred over a time period, the cash flow itself will contribute to the [portfolio] valuation¹, so that the calculation of [the portfolio return] […] must compensate for the fact that the increase in market value is not entirely due to investment gain[/loss] during the period⁶.

Since at least 1968¹, there are two main measures of such a compensated portfolio return:

The portfolio money-weighted return (MWR), which integrates the timing and the amount of the external cash flows in the portfolio return, leading to a measure of portfolio return that includes the impacts of both the decisions to contribute money in (resp. withdraw money from) the portfolio and the decisions about asset allocation and security selection⁶.
The portfolio time-weighted return (TWR), which eliminates the impact of the external cash flows from the portfolio return, leading to a measure of portfolio return that isolates the decisions about asset allocation and security selection⁶.

Money-weighted return

The money-weighted return of a portfolio is a performance statistic reflecting how much money was earned during the measurement period⁶ and is thus representative of the return an investor actually experiences⁶.

As such, the money-weighted return is influenced by both:

The decisions made by the [portfolio] manager [- who can be the investor herself] - to allocate assets and select securities within the portfolio⁶
The timing of [the investor’s] decisions to contribute to or withdraw money from [the portfolio]⁶, with the different cases being summarized in Figure 1, taken from Feibel⁶.

Figure 1. Impact of external cash flows on money-weighted portfolio performances. Source: Feibel.

Calculation

To borrow a methodology used throughout finance¹, the money-weighted return of a portfolio corresponds to the internal rate of return (IRR) of that portfolio, defined as the rate of return which reconciles the beginning market value and additional cash flows into the portfolio to the ending market value⁶.

Internal rate of return method

Mathematically, the calculation of the (annualized) money-weighted return $r_{mw,irr}$ of a portfolio over a time period through the IRR method is done by solving what is usually called the IRR equation⁷.

Using the notations of this section, that equation is:

\[V_1 - \sum_{i=1}^{T} \frac{C_i}{ \left(1 + r_{mw,irr}\right)^{yearfrac(t_i, t_1)}} - \frac{V_T}{ \left(1 + r_{mw,irr}\right)^{yearfrac(t_T, t_1)}} = 0\]

, where $yearfrac$ is the number of (fractional) years between two dates using a given day count convention.

Numerically, the IRR equation is usually solved by an iterative algorithm, like the Newton-Raphson method or similar other methods.

Modified Dietz method

The calculation of $r_{mw,irr}$ described in the previous sub-section was a problem when computer CPU time was very expensive and needed to be conserved¹, which led to the development of various IRR estimation techniques that did not require [an] iterative algorithm¹.

One of these approximations - the modified Dietz method - is a first-order [closed-form] linear approximation to the IRR¹ method and is the most common⁸ way of calculating periodic investment returns¹.

Mathematically, the calculation of the (total) money-weighted return $r_{mw,md}$ of a portfolio over a time period through the modified Dietz method is done by incorporating “weighted” external cash flows into the formula of the simple rate of return.

Using the notations of this section, this gives:

\[r_{mw,md} = \frac{V_T - V_1 - \sum_{i=1}^T C_i}{V_1 + \sum_{i=1}^T \left( 1 - \frac{yearfrac(t_1, t_i)}{yearfrac(t_1, t_T)} \right) C_i }\]

, where the terms $ 1 - \frac{yearfrac(t_1, t_i)}{yearfrac(t_1, t_T)} $, $i=1..T$, are used to weight the [external cash] flows by the number of days they were available for investment during the period¹.

Caveats

The two money-weighted return calculation methods discussed in the previous sub-sections have certain limitations to be aware of:

The IRR method provides no theoretical guarantee in general⁹ as to the existence or the unicity of an internal rate of return.

Indeed, the IRR equation might not have any solution or might have several solutions¹⁰.

In addition, even when the IRR equation has a unique solution, it might be numerically very hard to find…
The modified Dietz method has its own issues, for example the accuracy of the result [being] dependent on relatively small capital flows, low volatility and frequent valuations¹¹.

In particular, a very important point of attention is that the modified Dietz [method] is less useful as an approximation to [IRR] over longer time periods¹.

When to use it?

Calculating the money-weighted return of a portfolio allows to answer the question

How much did a specific investor’s portfolio grow, thanks to both its underlying strategy and its pattern of external cash flows?

Thus, the money-weighted return of a portfolio is an appropriate measure of portfolio performance when it is desired to:

Compute the rate of return earned on a specific portfolio
Compare the rate of return earned on a specific portfolio to another rate (rate of inflation, rate of return earned on another portfolio¹², rate of return earned on an alternative investment like real estate or private equity…)
Analyse the pattern of external cash flows for a specific portfolio¹³

Time-weighted return

The time-weighted return of a portfolio is a form of total return that measures the performance […] for the complete measurement period⁶ in a way that fully eliminates the timing effect that external portfolio cash flows have⁶.

So, contrary to its money-weighted counterpart, the time-weighted return of a portfolio measures only the effects of the market and manager decision⁶.

As a side note, the time-weighted return of a portfolio is the only well-behaved rate of return that is not influenced by contributions or withdrawals², c.f. for example Gray and Dewar².

Calculation

The calculation of the time-weighted return of a portfolio over a time period is easily done through the unit price or unitised method¹, described in Bacon¹:

A standardised unit price or “net asset value” price is calculated immediately before each external cash flow by dividing the market value by the number of units previously allocated. Units are then added or subtracted (bought or sold) in the portfolio at the unit price corresponding to the time of the cash flow […].

Mathematically, using the notations of this section, let be $U_i$, $i=1..T$, corresponding to the following transformation¹⁴ of $V_i$, $i=1..T$:

\[U_1 = 1 \newline U_{i+1} = U_i \frac{V_{i+1}}{V_i + C_i}, i=1..T-1\]

The (total) time-weighted return $r_{tw}$ of the portfolio over the time period $1..T$ is then defined as the simple rate of return of the transformed portfolio values $U_i$, $i=1..T$:

\[r_{tw} = \frac{U_T - U_1}{U_1} = \frac{U_T}{U_1} - 1\]

Caveats

The time-weighted return has two main limitations to be aware of:

One is practical, with Feibel⁶ noting that there is […] a potentially significant hurdle to implementing this method: the time-weighted return methodology requires valuation of the portfolio before each cash flow⁶.

For an individual investor, and depending on the exact tools used, this might or might not be a real issue though.
One is methological, with the time-weighted return of a portfolio being sometimes positive while the overall portfolio is at a loss¹⁵!

Bacon¹ and Febeil⁶ provide examples of such cases, the bottom line being that it is important [for the portfolio manager] to perform well in the […] period[s] when the majority of client money is invested¹ with the time-weighted return calculation.

When to use it?

Calculating the time-weighted return of a portfolio allows to answer the question

How much did a specific investor’s portfolio grow thanks to its underlying strategy (asset allocation, exposure…)?.

The time-weighted return of a portfolio is thus an appropriate measure of portfolio performance when it is desired to compar[e] the performance of [a portfolio with] different asset managers […] and with benchmark indexes¹.

Hybrid return

Bacon¹ notes that in practice, many asset managers use neither true time-weighted nor money-weighted calculations exclusively but rather a hybrid combination of both¹, some of which are depicted in Figure 2, taken from Bacon¹.

Figure 2. The evolution of performance returns methodologies. Source: Bacon.

Money-weighted v.s. time-weighted return

To conclude this section, Figure 3, taken from Feibel⁶, summarizes the respective properties of money-weighted and time-weighted returns.

Figure 3. Properties of money-weighted and time-weighted returns. Source: Feibel.

To that summary, I would add that for an individual investor, comparing the money-weighted return and the time-weighted return of a portfolio allows to determine whether the timing of contributions and withdrawals has been successful ($r_{mw} \geq r_{tw}$) or not ($r_{mw} \leq r_{tw}$).

In the latter case, it might be interesting to review¹⁶ the external cash flows timing strategy (panic sells, fomo buys…).

Implementation in Portfolio Optimizer

Portfolio Optimizer allows to:

Compute the money-weighted and the time-weighted return of a portfolio over a time period, through the endpoints /portfolios/analysis/return/money-weighted and /portfolios/analysis/return/time-weighted.
Transform portfolio values impacted by external cash flows into money-weighted or time-weighted portfolio values directly usable to compute portfolio performance indicators (return, Sharpe ratio, maximum drawdown…), through the endpoints /portfolios/transformation/money-weighted and /portfolios/transformation/time-weighted.

Examples of usage

I propose to illustrate the differences between money-weighted returns and time-weighted returns through two examples:

Comparing a dollar cost averaging investment strategy in an ETF to a lump sum investment strategy in the same ETF
Comparing the returns of an ETF to the returns of an “average” investor in that ETF

Dollar cost averaging v.s. lump sum investing in an MSCI World ETF

Let’s suppose that we would like to compare two investment strategies in the Amundi MSCI World UCITS ETF - EUR (C) over the period 29/12/2023 - 31/12/2024:

Dollar cost averaging (DCA) investing
- Portfolio creation - 1000€ on 29/12/2023
- Portfolio contributions - 100€ at each month’s end
Lump sum (LS) investing
- Portfolio creation - 2200€ on 29/12/2023
- Portfolio contributions - n/a

In both cases, the amount of external portfolio cash flows is the same¹⁷ - equal to 2200€ - but the patterns of these cash flows are different, leading to different portfolio returns:

Portfolio return measure	Total return (%)
Simple return (DCA)	161.42%
Money-weighted return (DCA)	25.45%
Time-weighted return (DCA)	26.33%
Simple return (LS)	26.33%
Money-weighted return (LS)	26.33%
Time-weighted return (LS)	26.33%

Notes:

Detailed calculations are available in the Google Sheet associated to this blog post.

The table above empirically confirms that:

In the absence of external cash flows, all portfolio return measures discussed in this blog post become equal (lines #4-#6)
In the presence of external cash flows
- The simple return is not an appropriate measure of portfolio return (line #1)
  
  Despite this, and at the date of publication of this blog post, there are still some financial services using that measure to compute portfolio returns in the presence of external cash flows, like Trading 212, c.f. the associated discussion on their forum.
  
  Usually, it is because of data availability problems like difficulties in identifying external cash flows, but sometimes it is simply because providing accurate portfolio performance measures is not a priority…
- The money-weighted return is impacted by the timing of external cash flows (line #2)
  
  Here, there is a ~-1% difference between the portfolio money-weighted and time-weighted returns, mainly due to the ill-timed¹⁸ contribution made on 28/03/2024.
  
  Such a difference is relatively insignificant over one year, but depending on the pattern of external cash flows, it could be much worse.
  
  For example, if a unique contribution of 1200€ on 28/03/2024 was made instead of monthly contributions of 100€, the associated portfolio money-weighted return would decrease to 22.59%, representing this time a ~-4% difference!
- The time-weighted return is not impacted by external cash flows¹⁹ and is akin to a “lump sum investing”-equivalent return (line #3 is equal to lines #4-#6)

Investment returns v.s. average investor returns in an MSCI World ETF

Financial studies regularly show that investors tend to lag actual fund returns across a variety of asset classes, leading to what is usually called the investor gap²⁰²¹.

For example:

The Dalbar’s Quantitative Analysis of Investor Behavior (QAIB) annual study measure[s] the effects of investor decisions to buy, sell and switch into and out of funds over short and long-term timeframes²² and the results consistently show that the average investor earns less – in many cases, much less – than mutual fund performance reports would suggest²².

For the U.S., Figure 4, adapted from Dalbar²², depicts the returns of U.S. funds v.s. the returns of the average investor in U.S. funds over the period 1994-2023.

Figure 4. U.S. funds returns v.s. average U.S. funds investor returns (30-year returns, 1994-2023). Source: Dalbar.

The observed performance differencial of ~2% per year creates dramatic cumulative effects over such a long period²²!
The Morningstar’s Mind the Gap annual study compares funds’ dollar-weighted returns [- that is, funds’ returns as experienced by their investors -] with their time-weighted returns to see how large the gap, or difference, has been over time²³ and analyses where investors succeeded in capturing most of their funds’ returns²³.

Figure 5, directly taken from Morningstar²³, illustrates that gap by U.S. funds category over the 10 years ended 31th December 2023.

Figure 5. Investor return gaps by U.S. category group (10-year returns, 2013-2023). Source: Morningstar.

In this sub-section, I propose to apply the same methodology as these studies to the Amundi MSCI World UCITS ETF - EUR (C) over the period 30/04/2018 - 31/12/2024.

In more details:

I will use the month-end asset data [(AUM)] compared with the underlying total return to estimate a net inflow or outflow for [each] month²³, using Morningstar’s approach to estimating funds’ monthly net flows²⁴:

The [net] cash flow estimate for a month $C_t$ is the difference in the beginning $NAV_{t-1}$ and ending $NAV_t$ total net assets that cannot be explained by the monthly total return $r_t$, that is, $ C_t = NAV_t - NAV_{t-1} \left( 1 + r_t \right) $.
Once all the monthly cash flows are available for the period²³, I will calculate the associated money-weighted return - which correspond to the average investor return in the ETF - and I will then compare it to the ETF money-weighted return²⁵ - which corresponds to the ETF return itself.

Notes:

Detailed calculations are available in the Google Sheet associated to this blog post.

Results are the following:

Portfolio return measure	Annualized return (%)
Money-weighted return (AUM)	14.15%
Time-weighted return (AUM)	12.74%

The table above confirms the presence of an investor gap for the considered MSCI World ETF, but for once, to the benefit (of around ~1.5% yearly) of the average investor!

This is at odds with Morningstar’s findings²³ that the gap between index ETFs’ dollar-weighted returns and their total return […] was quite a bit wider than the gap for index open-end funds²³, but of couse, the analysis of this sub-section has no statistical significance compared to that of Morningstar.

Or, maybe investors in the Amundi MSCI World UCITS ETF have specific timing abilities, who knows!

Conclusion

As mentioned in the introduction, the calculation of portfolio return is the first step in the performance measurement process¹.

Once properly done, the next step is to know if the return is good or bad. In other words, [it is needed] to evaluate performance (risk and return), against an appropriate benchmark¹, which can for example be random portfolios.

For more discussions on portfolio performances, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

See Carl Bacon, Practical Portfolio Performance Measurement and Attribution, Third edition. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³
See Kenneth B. Gray, Jr. and Robert B. K. Dewar, Axiomatic Characterization of the Time-Weighted Rate of Return, Management Science, Vol. 18, No. 2, Application Series (Oct., 1971), pp. B32-B35. ↩ ↩² ↩³ ↩⁴ ↩⁵
See W. Marty, Portfolio Analytics. An Introduction to Return and RiskMeasurement, Springer Texts in Business and Economics (2nd edition), Springer Berlin, 2015.. ↩
Internal cash flows our of the portfolio is sometimes refered to as income¹⁶ from the portfolio constituents. ↩
In contrast with the price return of the portfolio, which would only take into account the price evolution of the portfolio constituents. ↩
See Bruce J. Feibel, Investment Performance Measurement. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷
As noted in Bacon¹, the formal definition of the IRR is the discount rate that makes the net present value equal to zero in a discounted cash flow analysis¹; that equation is precisely the equation of the net present value in the specific case of a portfolio of assets. ↩
For example, the modified Dietz method is used in Interactive Brokers PortfolioAnalyst product to compute the money-weighted return of the investor’s portfolio. ↩
C.f. for example the documentation of the Excel’s function XIRR. ↩
Although Bacon¹ notes that you need to work very hard, with quite extreme data, to generate suitable examples¹ because multiple solutions are extremely unlikely to be experienced in practice¹. ↩
See Hager DP. Measurement of Pension Fund Investment Performance. Journal of the Staple Inn Actuarial Students’ Society. 1980;24:33-64. ↩
For Bacon¹, though, the money-weighted return is unique and not really comparable with other portfolios enjoying a different pattern of cash flow¹; on this, I kindly disagree, because as an individual investor, I will for example certainly be pleased to have a higher money-weighted return than that of my father/brother in law! ↩
For example when testing for investor’s skill in timing external cash flows or when comparing dollar-cost averaging (DCA) investment strategies (weekly, monthly, quarterly…). ↩
That transformation is in effect a normalised market value¹ transformation of the portfolio. ↩
Or conversely, with the time-weighted return of a portfolio being sometimes negative while the overall portfolio is at a gain!! ↩
For example, by switching to a dollar cost averaging strategy. ↩
The last contribution of 100€ made on 31/12/2024 is excluded from this count, since it is invested at the end of the considered period. ↩
From 28/03/2024 to 30/04/2024, the MSCI World ETF under consideration returned -2.74%. ↩
The time-weighted return is also equal to the NAV return. ↩
To be noted that the investor gap as discussed here is not related to fund fees; for example, Morningstar²³ didn’t find a strong link between fees and investor return gaps in the study²³. ↩
Also known as the behaviour gap. ↩
See Dalbar’s 2024 QAIB Report. ↩ ↩² ↩³ ↩⁴
See Morningstar’s Mind the Gap 2024 Report. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹
See Morningstar’s Estimated Net Cash Flow Methodology. ↩
As with most funds, ETF returns correspond to time-weighted returns. ↩

Covariance Matrix Forecasting: Iterated Exponentially Weighted Moving Average Model

2024-10-28T00:00:00-05:00

In the previous post of this series on covariance matrix forecasting, I reviewed both the simple and the exponentially weighted moving average covariance matrix forecasting models, which are straightforward extensions of their respective univariate volatility forecasting models to a multivariate setting.

With these reference models established, we can now delve into more sophisticated approaches for forecasting covariance matrices¹.

In this blog post, I will describe the iterated exponentially weighted moving average (IEWMA) model that has recently² been introduced in Johansson et al.³ and I will illustrate its empirical performances in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio.

Mathematical preliminaries

Covariance matrix modelling and covariance proxies (reminders)

This sub-section contains reminders from a previous blog post.

Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (unknown) (logarithmic) return process of these assets over a time period $t$ (a day, a week, a month..), over which the (conditional) mean return vector $\mu_t \in \mathbb{R}^n$ of these assets is supposed to be null.

Then:

$r_t$ can be expressed as⁴ $r_t = \epsilon_t$, with $\epsilon_t \in \mathbb{R}^n$ an unpredictable error term, often referred to as a vector of “shocks” or as a vector of “random disturbances”⁴, over the time period $t$
The asset (conditional) covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ over that time period $t$ is defined as $\Sigma_t = \mathbb{E} \left[ r_t r_t {}^t \right]$

From this definition, the outer product of the realized asset returns $ \tilde{r}_t \tilde{r}_t {}^t $ over a time period $t$ (a day, a week, a month..) is a covariance estimate $\tilde{\Sigma}_t$ - or covariance proxy⁵ - for the (unknown) asset returns covariance matrix over the considered time period $t$.
The asset (conditional) correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is defined as $ C_t = V_t^{-1} \Sigma_t V_t^{-1} $, where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset (conditional) standard deviations.

Correlation matrix modelling

In order to clarify the relation between conditional correlations and conditional variances⁶, Engle⁶ proposes to write the [asset] returns as the conditional standard deviation times the standardized disturbance⁶, that is

\[r_{i,t} = \epsilon_{i,t} = \sigma_{i,t} \varepsilon_{i,t}, i=1..n\]

, with:

$\sigma_{i,t} = \sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] } $
$\varepsilon_{i,t}$ a standardized disturbance that has mean zero and variance one⁶

This way, the conditional correlation [between asset $i$ and asset $j$ becomes equal to] the conditional covariance between the standardized disturbances [$\varepsilon_{i,t}$ and $\varepsilon_{j,t}$]⁶

\[\rho_{ij,t} = \frac{\mathbb{E} \left[ r_{i,t} r_{j,t} \right]}{\sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] \mathbb{E} \left[ r_{j,t}^2 \right] }} = \mathbb{E} \left[ \varepsilon_{i,t} \varepsilon_{j,t} \right]\]

The iterated exponentially weighted moving average covariance matrix forecasting model

The IEWMA covariance matrix forecasting model³ is a two-step model that:

Uses an EWMA volatility forecasting model with squared asset returns as variance proxies in order to forecast asset volatilities
Uses an EWMA correlation matrix forecasting model with outer products of (EWMA-)volatility-standardized asset returns as covariance matrix proxies in order to forecast asset correlations

Johansson et al.³ highlights that the IEWMA model was originally⁷ proposed in Engle⁶ as an efficient alternative to the DCC-GARCH⁶ predictor, although he did not refer to it as IEWMA⁶.

In other words, the IEWMA covariance matrix forecasting model bridges the gap between the simple EWMA model and the more complex DCC-GARCH model.

Forecasting formulas

Let be:

$n$ be the number of assets in a universe of assets
$r_t r_t {}^t$, $t=1..T$ the outer products of the realized asset returns over each of $T$ past periods
A decay factor $\lambda_{vol} \in [0, 1]$
A decay factor $\lambda_{cor} \in [0, 1]$

Next period’s asset returns covariance/correlation matrix

The IEWMA covariance matrix forecasting model estimates the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$ as follows³:

For each asset $i=1..n$ in the universe of assets
- Forecast the asset one-period-ahead variances $\hat{\sigma}^2_{i,2}$, …, $\hat{\sigma}^2_{i,T+1}$ using an EWMA volatility forecasting model with decay factor $\lambda_{vol}$ and squared asset returns $r_{i,t}^2$, $t=1..T$, as variance proxies
- Compute the volatility-standardized asset returns $\tilde{r}_{i,2},…,\tilde{r}_{i,T}$ defined as the asset returns standardized by their one-period-ahead EWMA volatility forecasts
  \[\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}, t = 2..T\]
Compute the diagonal matrix of the assets next period’s forecasted volatilities $V_{T+1}$
\[V_{T+1} = \begin{pmatrix} \hat{\sigma}_{1,T+1} & 0 & ... & 0 \\ 0 & \hat{\sigma}_{2,T+1} & ... & 0 \\ ... & ... & ... & ... \\ 0 & 0 & ... & \hat{\sigma}_{n,T+1} \end{pmatrix}\]
Forecast the next period’s asset returns correlation matrix $\hat{C}_{T+1}$ using an EWMA covariance matrix forecasting model with decay factor $\lambda_{cor}$ and outer products of volatility-standardized asset returns $\tilde{r_t} \tilde{r_t} {}^t$, $t=2..T$, as covariance matrix proxies
Compute the next period’s forecasted asset returns covariance matrix $\hat{\Sigma}_{T+1}$
\[\hat{\Sigma}_{T+1} = V_{T+1} \hat{C}_{T+1} V_{T+1}\]

Next $h$-period’s ahead asset returns covariance/correlation matrix

The IEWMA covariance matrix forecasting model estimates the next $h$-period’s ahead asset returns covariance matrix $\hat{\Sigma}_{T+h}$ and correlation matrix $\hat{C}_{T+h}$, $h \geq 2$, by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$.

Indeed, due to the properties of the EWMA volatility and covariance matrix forecasting models⁸, we have:

$V_{T+h} = V_{T+1}$, $h \geq 2$
$\hat{C}_{T+h} = \hat{C}_{T+1}$, $h \geq 2$

So that $\hat{\Sigma}_{T+h} = \hat{\Sigma}_{T+1}$, $h \geq 2$.

Averaged asset returns covariance/correlation matrix over the next $h$ periods

The IEWMA covariance matrix forecasting model estimates the averaged⁹ asset returns covariance matrix $\hat{\Sigma}_{T+1:T+h}$ and correlation matrix $\hat{C}_{T+1:T+h}$ over the next $h$ periods, $h \geq 2$, by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$.

Indeed, from the previous sub-section, we have:

$ \hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i} = \hat{\Sigma}_{T+1} $, $h \geq 2$
$ \hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i} = \hat{C}_{T+1} $, $h \geq 2$

Rationale

The rationale behind the IEWMA covariance matrix forecasting model is twofold:

Separate volatility forecasting from correlation matrix forecasting while using the same baseline forecasting model for internal consistency.

This first idea is well known among practitioners, c.f. for example Menchero and Li¹⁰ which describes the usage of different EWMA half-lives for estimating volatilities and correlations¹¹.
Use volatility-standardized asset returns instead of raw asset returns for correlation matrix forecasting.

This second idea, detailled for example in Engle⁶, originates from the fact that conditional correlations between asset returns are equal to conditional covariances between standardized disturbances, c.f. the previous section.

So, given an estimator for the (unobservable) standardized disturbances $\epsilon_{i,t} = \frac{r_{i,t}}{\sigma_{i,t}}$, $i=1..n$, $t=1..T$, it is possible to estimate the conditional correlations between asset returns.

An example of such estimator is the volatility-standardized asset returns $\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}$, with the drawback that the quality of the correlation forecasts is then influenced by the quality of the volatility forecasts.

And unfortunately, because it is well known that […] asset returns […] fat tails are typically reduced but not eliminated when returns are standardized by volatilities estimated from popular [volatility forecasting] models¹² and because the use of correlation as a measure of dependence can be misdealing in the case of (conditionally) non-Gaussian returns¹³, we should not expect any magic here…

How to choose the decay factors?

Due to its relationship with the vanilla EWMA volatility and covariance matrix forecasting models, there are two main procedures to choose the decay factors $\lambda_{vol}$ and $\lambda_{cor}$ of an IEWMA covariance matrix forecasting model:

Using recommended values from the EWMA models literature (0.94, 0.97…).

On this, Johansson et al.³ notes that empirical studies on real return data confirm that choosing a faster volatility half-life than correlation half-life yields better estimates⁶ and uses the following pairs of decay factors $\left(\lambda_{vol}, \lambda_{cor}\right)$ in their experimental setup:
- Short term - $\left(0.870,0.933\right)$, $\left(0.933,0.967\right)$
- Medium term - $\left(0.967,0.989\right)$, $\left(0.989,0.994\right)$, $\left(0.994,0.997\right)$
- Long term - $\left(0.997,0.998\right)$, $\left(0.998,0.999\right)$
Determining the optimal values w.r.t. the forecast horizon $h$, for example through the minimization of the root mean square error (RMSE) between the forecasted covariance matrix over the desired horizon and the observed covariance matrix over that horizon¹⁴.

In practice, because there are two decay factors to choose, this can be done two ways:
- Either consider the two decay factors as a two independent univariate parameters $\lambda_{vol} \in [0,1]$ and $\lambda_{cor} \in [0,1]$.
  
  This choice is justified by the original desire to separate volatility forecasting from correlation matrix forecasting.
- Or consider the two decay factors as a single multivariate parameter $\left(\lambda_{vol}, \lambda_{cor}\right) \in [0,1]^2$.
  
  This choice is justified by the observed dependency of the correlation forecasts on the volatility-standardized asset returns.

Extensions of the iterated exponentially weighted moving average covariance matrix forecasting model

Asset-specific volatility decay factors

The IEWMA covariance matrix forecasting model uses univariate EWMA models in order to forecast asset volatilities, all these models sharing the same decay factor $\lambda_{vol}$.

Having an identical decay factor for all assets is parsimonious, but is somewhat at odds with the DCC-GARCH model of Engle⁶ which uses asset-specific univariate GARCH models - that is, each with its own asset-specific parameters - in order to forecast asset volatilities.

So, one natural extension of the IEWMA model is to allow asset-specific univariate EWMA models - each with its own asset-specific decay factor $\lambda_{i,vol}$, $i=1..n$ - in order to forecast asset volatilities.

Linear combination of IEWMA covariance matrix forecasting models

Another interesting covariance matrix forecasting model is introduced in Johansson et al.³ as the combined multiple iterated exponentially weighted moving average (CM-IEWMA) model, which consist in a time-varying linear combination of individual IEWMA models, each with its own pair of fixed decay factors.

As explained in Johansson et al.³:

The CM-IEWMA predictor is constructed from a modest number of IEWMA predictors, with different pairs of half-lives, which are combined using dynamically varying weights that are based on recent performance.

The rationale behind the CM-IEWMA model is that different pairs of half-lives may work better for different market conditions³, with short half-lives [typically performing] better in volatile markets [and] long half-lives [performing] better for calm markets where conditions are changing slowly³.

This behaviour is illustrated in Figure 1, taken from Johansson et al.³, which shows the evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks.

Figure 1. Evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks, 4th January 2010 - 30th December 2022. Source: Johansson et al.

From Figure 1, it is visible that although substantial weight is put on the slower (longer halflife) IEWMAs most years³, the CM-IEWMA model still adapts the weights depending on market conditions³.

The interested reader is referenced to Johansson et al.³ for all the technicalities of the CM-IEWMA model, and in particular for the details about the computation of the dynamically varying weights of the individual IEWMA models through the resolution of a convex optimization problem¹⁵.

A last important remark to conclude this sub-section - the CM-IEWMA model is actually a special case of [a more general] dynamically weighted prediction [model]³, so that the same weighting logic can be applied to any combination of covariance matrix forecasting models.

Implementations

Implementation in Portfolio Optimizer

Portfolio Optimizer implements the IEWMA covariance and correlation matrix forecasting model through the endpoints /assets/covariance/matrix/forecast/ewma/iterated and /assets/correlation/matrix/forecast/ewma/iterated.

These endpoints support the 2 covariance proxies below:

Squared (close-to-close) returns
Demeaned squared (close-to-close) returns

These endpoints also allow:

To use asset-specific univariate EWMA models in order to forecast asset volatilities
To automatically determine the optimal value of their parameters (the decay factors $\lambda_{vol}$ and $\lambda_{cor}$) using a proprietary procedure.

To be noted that Portfolio Optimizer does not provide any implementation of the CM-IEWMA model¹⁶, but c.f. the next sub-section.

Implementation elsewhere

Johansson et al.³ kindly provides an open source Python implementation of:

The IEWMA covariance matrix forecasting model
The CM-IEWMA covariance matrix forecasting model
The general “covariance matrix forecasting models” combination model

at https://github.com/cvxgrp/cov_pred_finance.

I definitely encourage anyone interested in the CM-IEMWA model to play with this code!

Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs

As an example of usage, I propose to evaluate the empirical performances of the IEWMA covariance matrix forecating model within the framework of the previous blog bost, whose aim is to forecast monthly covariance and correlation matrices for a portfolio of 10 ETFs representative¹⁷ of misc. asset classes:

U.S. stocks (SPY ETF)
European stocks (EZU ETF)
Japanese stocks (EWJ ETF)
Emerging markets stocks (EEM ETF)
U.S. REITs (VNQ ETF)
International REITs (RWX ETF)
U.S. 7-10 year Treasuries (IEF ETF)
U.S. 20+ year Treasuries (TLT ETF)
Commodities (DBC ETF)
Gold (GLD ETF)

Results - Covariance matrix forecasting

Results over the period 31st January 2008 - 31st July 2023¹⁸ for covariance matrices are the following¹⁹:

Covariance matrix model	Covariance matrix MSE
SMA, window size of all the previous months (historical average model)	9.59 $10^{-6}$
SMA, window size of the previous year	9.08 $10^{-6}$
EWMA, optimal²⁰ $\lambda$	6.52 $10^{-6}$
EWMA, $\lambda = 0.97$	6.37 $10^{-6}$
IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$	6.35 $10^{-6}$
IEWMA, optimal²⁰ $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$	6.33 $10^{-6}$
IEWMA, optimal²⁰ $\left(\lambda_{vol},\lambda_{cor}\right)$	6.16 $10^{-6}$
SMA, window size of the previous month (random walk model)	6.06 $10^{-6}$
EWMA, $\lambda = 0.94$	5.78 $10^{-6}$
IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$	5.78 $10^{-6}$

Within this specific evaluation framework, the IEWMA covariance matrix forecasting model unfortunately does not seem to improve upon the previous models despite the added complexity.

It is also noteworthy that the IEWMA model with asset-specific univariate EWMA models for volatility (line #6) does not exhibit better performances than the vanilla IEWMA model (line #7) when using automatically determined parameters²¹.

Results - Correlation matrix forecasting

Results over the period 31st January 2008 - 31st July 2023¹⁸ for the correlation matrices associated to the covariance matrices of the previous sub-section are the following¹⁹:

Covariance matrix model	Correlation matrix MSE
SMA, window size of the previous month (random walk model)	8.19
SMA, window size of all the previous months (historical average model)	8.10
EWMA, $\lambda = 0.94$	7.67
SMA, window size of the previous year	6.50
EWMA, $\lambda = 0.97$	6.36
EWMA, optimal²⁰ $\lambda$	5.87
IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$	5.85
IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$	5.85
IEWMA, optimal²⁰ $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$	5.72
IEWMA, optimal²⁰ $\left(\lambda_{vol},\lambda_{cor}\right)$	5.70

This time, the IEWMA model - and especially the two IEWMA models using automatically determined parameters (lines #9-#10) - does exhibit slightly better performances than all the previous models.

Nevertheless, the improvement over the previously best-performing model (line #6 - the EWMA model with automatically determined parameter) is not that impressive…

So, the idea of removing the impact of volatility on asset returns in order to better estimate asset correlations seems to have some merits, but the EWMA volatility forecasting model might be insufficient to fully exploit this idea.

Also, here again, the performances of the IEWMA model with asset-specific univariate EWMA models for volatility are again strictly worse than the performances of the vanilla IEWMA model²¹.

Conclusion

One of the main characteristics of the IEWMA covariance matrix forecasting model of Johansson et al.³ is to forecast asset correlations using (EWMA-)volatility-standardized asset returns instead of raw asset returns, so that the impact of asset volatilities on their correlations is (tentatively) minimized.

Unfortunately, the empirical performances of that model in terms of correlation matrix forecasting are not that different from those of the EWMA model, which raises the question of whether improving the volatility-standardization might lead to better correlation forecasts.

This will be the subject of a future blog post in this series.

As usual, feel free to connect with me on LinkedIn or to follow me on Twitter.

–

ChatGPT-generated, as can be seen by the signature word “delve” :-) ! ↩
At the date of the initial publication of this blog post. ↩
See Kasper Johansson, Mehmet G. Ogut, Markus Pelger, Thomas Schmelzer and Stephen Boyd (2023), A Simple Method for Predicting Covariance Matrices of Financial Returns, Foundations and Trends in Econometrics: Vol. 12: No. 4, pp 324-407. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶
See Valeriy Zakamulin, A Test of Covariance-Matrix Forecasting Methods, The Journal of Portfolio Management Spring 2015, 41 (3) 97-108. ↩ ↩²
See Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg. ↩
See Engle, R. (2002). Dynamic Conditional Correlation. Journal of Business & Economic Statistics. 20(3): 339–350. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹
That being said, I find that what Engle⁶ describes is closer to an iterated $n$-univariate GARCH volatility forecasting models/EWMA correlation matrix forecasting model than to the iterated EWMA forecasting model of Johansson et al.³… ↩
C.f. the associated blog posts here and there. ↩
See Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426. ↩
See Menchero, Jose and Peng Li. Correlation Shrinkage: Implications for Risk Forecasting, Journal of Investment Management (2020). ↩
Digging a little deeper, the old MAC1 and MAC2 Bloomberg multi-asset risk models were using an half-life of 26 weeks for estimating volatilities and an half-life of 52 weeks for estimating correlations. ↩
See Andersen, Torben G., Tim Bollerslev, Francis X. Diebold, and Paul Labys, 2000, Exchange Rate Returns Standardized by Realized Volatility are (Nearly) Gaussian, Multinational Finance Journal 4, 159-179.. ↩
See Bahram Pesaran, M. Hashem Pesaran, Conditional volatility and correlations of weekly returns and the VaR analysis of 2008 stock market crash, Economic Modelling, Volume 27, Issue 6, 2010, Pages 1398-1416. ↩
See RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition. ↩
The maximization of the average log-likelihood of the combined [covariance matrix] prediction over [a trailing number of periods]³. ↩
This is because my own tests did not highlight any strong improvement in terms of forecasting ability v.s. the IEWMA model when used with an optimal pair of decay factors. ↩
These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer²². ↩
(Adjusted) daily prices have have been retrieved using Tiingo. ↩ ↩²
Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. ↩ ↩²
The optimal decay factors ($\lambda$, $\lambda_{vol}$, $\lambda_{cor}$) are computed at the end of every month using all the available asset returns history up to that point in time, as implemented in Portfolio Optimizer; thus, there is no look-ahead bias. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
The difference between the two models is probably just noise, which, at worst, implies that the added complexity of the IEWMA model with asset-specific univariate EWMA models is useless in practice. ↩ ↩²
See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩