Jekyll2024-10-28T10:21:28-05:00https://portfoliooptimizer.io/feed.xmlPortfolio OptimizerPortfolio Optimizer is a Web API democratizing the access to the Nobel Prize-winning science of portfolio optimization.Roman R.Covariance Matrix Forecasting: Iterated Exponentially Weighted Moving Average Model2024-10-28T00:00:00-05:002024-10-28T00:00:00-05:00https://portfoliooptimizer.io/blog/covariance-matrix-forecasting-iterated-exponentially-weighted-moving-average-model<p>In the <a href="/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models/">previous post</a> of this series on covariance matrix forecasting, I reviewed both the simple and the exponentially weighted moving average covariance matrix forecasting models,
which are straightforward extensions of <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">their respective univariate volatility forecasting models</a> to a multivariate setting.</p>
<p>With these reference models established, <em>we can now delve into more sophisticated approaches for forecasting covariance matrices</em><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">1</a></sup>.</p>
<p>In this blog post, I will describe the <em>iterated exponentially weighted moving average (IEWMA) model</em> that has recently<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup> been introduced in Johansson et al.<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>
and I will illustrate its empirical performances in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="covariance-matrix-modelling-and-covariance-proxies-reminders">Covariance matrix modelling and covariance proxies (reminders)</h3>
<p>This sub-section contains reminders from a <a href="/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a>.</p>
<p>Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>)
return of these assets over a time period $t$ (a day, a week, a month..), over which the (conditional) mean return vector $\mu_t \in \mathbb{R}^n$ of these assets is supposed to be null.</p>
<p>Then:</p>
<ul>
<li>
<p>$r_t$ can be expressed as<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">4</a></sup> $r_t = \epsilon_t$, with $\epsilon_t \in \mathbb{R}^n$ an unpredictable error term, often referred to as a vector of “shocks” or as a vector of “random disturbances”<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">4</a></sup>, over the time period $t$</p>
</li>
<li>
<p>The asset (conditional) covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is defined as $\Sigma_t = \mathbb{E} \left[ r_t r_t {}^t \right]$</p>
<p>From this definition, the <a href="https://en.wikipedia.org/wiki/Outer_product">outer product</a> of the asset returns $ r_t r_t {}^t $ is a covariance estimate - or covariance proxy<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">5</a></sup> - for the asset returns covariance matrix over the considered time period $t$.</p>
</li>
<li>
<p>The asset (conditional) correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is defined as $ C_t = V_t^{-1} \Sigma_t V_t^{-1} $, where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset (conditional) standard deviations.</p>
</li>
</ul>
<h3 id="correlation-matrix-modelling">Correlation matrix modelling</h3>
<p>In order to <em>clarify the relation between conditional correlations and conditional variances</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup>, Engle<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> proposes to <em>write the [asset] returns as the conditional standard deviation
times the standardized disturbance</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup>, that is</p>
\[r_{i,t} = \epsilon_{i,t} = \sigma_{i,t} \varepsilon_{i,t}, i=1..n\]
<p>, with:</p>
<ul>
<li>$\sigma_{i,t} = \sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] } $</li>
<li>$\varepsilon_{i,t}$ a <em>standardized disturbance that has mean zero and variance one</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup></li>
</ul>
<p>This way, <em>the conditional correlation [between asset $i$ and asset $j$ becomes equal to] the conditional covariance between the standardized disturbances [$\varepsilon_{i,t}$ and $\varepsilon_{j,t}$]</em><sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup></p>
\[\rho_{ij,t} = \frac{\mathbb{E} \left[ r_{i,t} r_{j,t} \right]}{\sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] \mathbb{E} \left[ r_{j,t}^2 \right] }} = \mathbb{E} \left[ \varepsilon_{i,t} \varepsilon_{j,t} \right]\]
<h2 id="the-iterated-exponentially-weighted-moving-average-covariance-matrix-forecasting-model">The iterated exponentially weighted moving average covariance matrix forecasting model</h2>
<p>The IEWMA covariance matrix forecasting model<sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> is a two-step model that:</p>
<ol>
<li>Uses <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">an EWMA volatility forecasting model</a> with squared asset returns as variance proxies in order to forecast asset volatilities</li>
<li>Uses <a href="/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models/">an EWMA correlation matrix forecasting model</a> with outer products of (EWMA-)volatility-standardized asset returns as covariance matrix proxies in order to forecast asset correlations</li>
</ol>
<p>Johansson et al.<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> highlights that the IEWMA model was originally<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup> proposed in Engle<sup id="fnref:2:5" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> <em>as an efficient alternative to the DCC-GARCH<sup id="fnref:2:6" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> predictor, although he did not refer to it as IEWMA</em><sup id="fnref:2:7" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup>.</p>
<p>In other words, the IEWMA covariance matrix forecasting model bridges the gap between the simple EWMA model and the more complex DCC-GARCH model.</p>
<h3 id="forecasting-formulas">Forecasting formulas</h3>
<p>Let be:</p>
<ul>
<li>$n$ be the number of assets in a universe of assets</li>
<li>$r_t r_t {}^t$, $t=1..T$ the outer products of the asset returns over each of $T$ past periods</li>
<li>A decay factor $\lambda_{vol} \in [0, 1]$</li>
<li>A decay factor $\lambda_{cor} \in [0, 1]$</li>
</ul>
<h4 id="next-periods-asset-returns-covariancecorrelation-matrix">Next period’s asset returns covariance/correlation matrix</h4>
<p>The IEWMA covariance matrix forecasting model estimates the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$ as follows<sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>:</p>
<ul>
<li>For each asset $i=1..n$ in the universe of assets
<ul>
<li>Forecast the asset one-period-ahead variances $\hat{\sigma}^2_{i,2}$, …, $\hat{\sigma}^2_{i,T+1}$ using an EWMA volatility forecasting model with decay factor $\lambda_{vol}$ and squared asset returns $r_{i,t}^2$, $t=1..T$, as variance proxies</li>
<li>
<p>Compute the volatility-standardized asset returns $\tilde{r}_{i,2},…,\tilde{r}_{i,T}$ defined as the asset returns standardized by their one-period-ahead EWMA volatility forecasts</p>
\[\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}, t = 2..T\]
</li>
</ul>
</li>
<li>
<p>Compute the diagonal matrix of the assets next period’s forecasted volatilities $V_{T+1}$</p>
\[V_{T+1} = \begin{pmatrix}
\hat{\sigma}_{1,T+1} & 0 & ... & 0 \\
0 & \hat{\sigma}_{2,T+1} & ... & 0 \\
... & ... & ... & ... \\
0 & 0 & ... & \hat{\sigma}_{n,T+1}
\end{pmatrix}\]
</li>
<li>Forecast the next period’s asset returns correlation matrix $\hat{C}_{T+1}$ using an EWMA covariance matrix forecasting model with decay factor $\lambda_{cor}$ and outer products of volatility-standardized asset returns $\tilde{r_t} \tilde{r_t} {}^t$, $t=2..T$, as covariance matrix proxies</li>
<li>
<p>Compute the next period’s forecasted asset returns covariance matrix $\hat{\Sigma}_{T+1}$</p>
\[\hat{\Sigma}_{T+1} = V_{T+1} \hat{C}_{T+1} V_{T+1}\]
</li>
</ul>
<h4 id="next-h-periods-ahead-asset-returns-covariancecorrelation-matrix">Next $h$-period’s ahead asset returns covariance/correlation matrix</h4>
<p>The IEWMA covariance matrix forecasting model estimates the next $h$-period’s ahead asset returns covariance matrix $\hat{\Sigma}_{T+h}$ and correlation matrix $\hat{C}_{T+h}$, $h \geq 2$,
by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$.</p>
<p>Indeed, due to the properties of the EWMA volatility and covariance matrix forecasting models<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">8</a></sup>, we have:</p>
<ul>
<li>$V_{T+h} = V_{T+1}$, $h \geq 2$</li>
<li>$\hat{C}_{T+h} = \hat{C}_{T+1}$, $h \geq 2$</li>
</ul>
<p>So that $\hat{\Sigma}_{T+h} = \hat{\Sigma}_{T+1}$, $h \geq 2$.</p>
<h4 id="averaged-asset-returns-covariancecorrelation-matrix-over-the-next-h-periods">Averaged asset returns covariance/correlation matrix over the next $h$ periods</h4>
<p>The IEWMA covariance matrix forecasting model estimates the averaged<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup> asset returns covariance matrix $\hat{\Sigma}_{T+1:T+h}$ and correlation matrix $\hat{C}_{T+1:T+h}$ over the next $h$ periods,
$h \geq 2$, by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$.</p>
<p>Indeed, from the previous sub-section, we have:</p>
<ul>
<li>$ \hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i} = \hat{\Sigma}_{T+1} $, $h \geq 2$</li>
<li>$ \hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i} = \hat{C}_{T+1} $, $h \geq 2$</li>
</ul>
<h3 id="rationale">Rationale</h3>
<p>The rationale behind the IEWMA covariance matrix forecasting model is twofold:</p>
<ul>
<li>
<p>Separate volatility forecasting from correlation matrix forecasting while using the same baseline forecasting model for internal consistency.</p>
<p>This first idea is well known among practitioners, c.f. for example Menchero and Li<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">10</a></sup> which describes the usage of different EWMA half-lives for estimating volatilities and correlations<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>.</p>
</li>
<li>
<p>Use volatility-standardized asset returns instead of raw asset returns for correlation matrix forecasting.</p>
<p>This second idea, detailled for example in Engle<sup id="fnref:2:8" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup>, originates from the fact that conditional correlations between asset returns are equal to conditional covariances between standardized disturbances, c.f. the previous section.</p>
<p>So, given an estimator for the (unobservable) standardized disturbances $\epsilon_{i,t} = \frac{r_{i,t}}{\sigma_{i,t}}$, $i=1..n$, $t=1..T$, it is possible to estimate the conditional correlations between asset returns.</p>
<p>An example of such estimator is the volatility-standardized asset returns $\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}$, with the drawback that the quality of the correlation forecasts is then influenced by the quality of the volatility forecasts.</p>
<p>And unfortunately, because <em>it is well known that […] asset returns […] fat tails are typically reduced but not eliminated when returns are standardized by volatilities estimated from popular [volatility forecasting] models</em><sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">12</a></sup> and because <em>the use of correlation as a measure of dependence can be misdealing in the case of (conditionally) non-Gaussian returns</em><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">13</a></sup>, we should not expect any magic here…</p>
</li>
</ul>
<h3 id="how-to-choose-the-decay-factors">How to choose the decay factors?</h3>
<p>Due to its relationship with the vanilla EWMA volatility and covariance matrix forecasting models, there are two main procedures to choose the decay factors
$\lambda_{vol}$ and $\lambda_{cor}$ of an IEWMA covariance matrix forecasting model:</p>
<ul>
<li>
<p>Using recommended values from the EWMA models literature (0.94, 0.97…).</p>
<p>On this, Johansson et al.<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> notes that <em>empirical studies on real return data confirm that choosing a faster volatility half-life than correlation half-life yields better estimates</em><sup id="fnref:2:9" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> and uses the following pairs of decay factors $\left(\lambda_{vol}, \lambda_{cor}\right)$ in their experimental setup:</p>
<ul>
<li>Short term - $\left(0.870,0.933\right)$, $\left(0.933,0.967\right)$</li>
<li>Medium term - $\left(0.967,0.989\right)$, $\left(0.989,0.994\right)$, $\left(0.994,0.997\right)$</li>
<li>Long term - $\left(0.997,0.998\right)$, $\left(0.998,0.999\right)$</li>
</ul>
</li>
<li>
<p>Determining the optimal values w.r.t. the forecast horizon $h$, for example through the minimization of the <a href="https://en.wikipedia.org/wiki/Root-mean-square_deviation">root mean square error (RMSE)</a> between the forecasted covariance matrix over the desired horizon and the observed covariance matrix over that horizon<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>.</p>
<p>In practice, because there are two decay factors to choose, this can be done two ways:</p>
<ul>
<li>
<p>Either consider the two decay factors as a two independent univariate parameters $\lambda_{vol} \in [0,1]$ and $\lambda_{cor} \in [0,1]$.</p>
<p>This choice is justified by the original desire to separate volatility forecasting from correlation matrix forecasting.</p>
</li>
<li>
<p>Or consider the two decay factors as a single multivariate parameter $\left(\lambda_{vol}, \lambda_{cor}\right) \in [0,1]^2$.</p>
<p>This choice is justified by the observed dependency of the correlation forecasts on the volatility-standardized asset returns.</p>
</li>
</ul>
</li>
</ul>
<h2 id="extensions-of-the-iterated-exponentially-weighted-moving-average-covariance-matrix-forecasting-model">Extensions of the iterated exponentially weighted moving average covariance matrix forecasting model</h2>
<h3 id="asset-specific-volatility-decay-factors">Asset-specific volatility decay factors</h3>
<p>The IEWMA covariance matrix forecasting model uses univariate EWMA models in order to forecast asset volatilities, all these models sharing the same decay factor $\lambda_{vol}$.</p>
<p>Having an identical decay factor for all assets is parsimonious, but is somewhat at odds with the DCC-GARCH model of Engle<sup id="fnref:2:10" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> which uses asset-specific univariate GARCH models -
that is, each with its own asset-specific parameters - in order to forecast asset volatilities.</p>
<p>So, one natural extension of the IEWMA model is to allow asset-specific univariate EWMA models - each with its own asset-specific decay factor $\lambda_{i,vol}$, $i=1..n$ -
in order to forecast asset volatilities.</p>
<h3 id="linear-combination-of-iewma-covariance-matrix-forecasting-models">Linear combination of IEWMA covariance matrix forecasting models</h3>
<p>Another interesting covariance matrix forecasting model is introduced in Johansson et al.<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> as the <em>combined multiple iterated exponentially weighted moving average
(CM-IEWMA) model</em>, which consist in a time-varying linear combination of individual IEWMA models, each with its own pair of fixed decay factors.</p>
<p>As explained in Johansson et al.<sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>:</p>
<blockquote>
<p>The CM-IEWMA predictor is constructed from a modest number of IEWMA predictors, with different pairs of half-lives, which are combined using dynamically varying weights that are based on recent performance.</p>
</blockquote>
<p>The rationale behind the CM-IEWMA model is that <em>different pairs of half-lives may work better for different market conditions</em><sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, with <em>short half-lives [typically performing] better
in volatile markets [and] long half-lives [performing] better for calm markets where conditions are changing slowly</em><sup id="fnref:3:8" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>This behaviour is illustrated in Figure 1, taken from Johansson et al.<sup id="fnref:3:9" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, which shows the evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model
applied to a universe of U.S. stocks.</p>
<figure>
<a href="/assets/images/blog/covariance-matrix-forecasting-iewma-cm-iewma-weights-evolution.png"><img src="/assets/images/blog/covariance-matrix-forecasting-iewma-cm-iewma-weights-evolution-thumb.png" alt="Evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks, 4th January 2010 - 30th December 2022." /></a>
<figcaption>Figure 1. Evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks, 4th January 2010 - 30th December 2022. Source: Johansson et al.</figcaption>
</figure>
<p>From Figure 1, it is visible that although <em>substantial weight is put on the slower (longer halflife) IEWMAs most years</em><sup id="fnref:3:10" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, the CM-IEWMA model still
<em>adapts the weights depending on market conditions</em><sup id="fnref:3:11" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>The interested reader is referenced to Johansson et al.<sup id="fnref:3:12" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> for all the technicalities of the CM-IEWMA model, and in particular for the details about the computation of the dynamically varying
weights of the individual IEWMA models through the resolution of a convex optimization problem<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">15</a></sup>.</p>
<p>A last important remark to conclude this sub-section - the CM-IEWMA model is actually <em>a special case of [a more general] dynamically weighted prediction [model]</em><sup id="fnref:3:13" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, so that the same
weighting logic can be applied to any combination of covariance matrix forecasting models.</p>
<h2 id="implementations">Implementations</h2>
<h3 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h3>
<p><strong>Portfolio Optimizer</strong> implements the IEWMA covariance and correlation matrix forecasting model through the endpoints <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/covariance/matrix/forecast/iewma</code></a> and <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/correlation/matrix/forecast/iewma</code></a>.</p>
<p>These endpoints support the 2 covariance proxies below:</p>
<ul>
<li>Squared (close-to-close) returns</li>
<li>Demeaned squared (close-to-close) returns</li>
</ul>
<p>These endpoints also allow:</p>
<ul>
<li>To use asset-specific univariate EWMA models in order to forecast asset volatilities</li>
<li>To automatically determine the optimal value of their parameters (the decay factors $\lambda_{vol}$ and $\lambda_{cor}$) using a proprietary procedure.</li>
</ul>
<p>To be noted that <strong>Portfolio Optimizer</strong> does not provide any implementation of the CM-IEWMA model<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">16</a></sup>, but c.f. the next sub-section.</p>
<h3 id="implementation-elsewhere">Implementation elsewhere</h3>
<p>Johansson et al.<sup id="fnref:3:14" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> kindly provides an open source Python implementation of:</p>
<ul>
<li>The IEWMA covariance matrix forecasting model</li>
<li>The CM-IEWMA covariance matrix forecasting model</li>
<li>The general “covariance matrix forecasting models” combination model</li>
</ul>
<p>at <a href="https://github.com/cvxgrp/cov_pred_finance">https://github.com/cvxgrp/cov_pred_finance</a>.</p>
<p>I definitely encourage anyone interested in the CM-IEMWA model to play with this code!</p>
<h2 id="example-of-usage---covariance-matrix-forecasting-at-monthly-level-for-a-portfolio-of-various-etfs">Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs</h2>
<p>As an example of usage, I propose to evaluate the empirical performances of the IEWMA covariance matrix forecating model within the framework of <a href="/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models/">the previous blog bost</a>, whose aim is
to forecast monthly covariance and correlation matrices for a portfolio of 10 ETFs representative<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">17</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<h3 id="results---covariance-matrix-forecasting">Results - Covariance matrix forecasting</h3>
<p>Results over the period 31st January 2008 - 31st July 2023<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">18</a></sup> for covariance matrices are the following<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>:</p>
<table>
<thead>
<tr>
<th>Covariance matrix model</th>
<th>Covariance matrix MSE</th>
</tr>
</thead>
<tbody>
<tr>
<td>SMA, window size of all the previous months (historical average model)</td>
<td>9.59 $10^{-6}$</td>
</tr>
<tr>
<td>SMA, window size of the previous year</td>
<td>9.08 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, optimal<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\lambda$</td>
<td>6.52 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.97$</td>
<td>6.37 $10^{-6}$</td>
</tr>
<tr>
<td><strong>IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$</strong></td>
<td><strong>6.35 $10^{-6}$</strong></td>
</tr>
<tr>
<td><strong>IEWMA, optimal<sup id="fnref:24:1" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$</strong></td>
<td><strong>6.33 $10^{-6}$</strong></td>
</tr>
<tr>
<td><strong>IEWMA, optimal<sup id="fnref:24:2" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\left(\lambda_{vol},\lambda_{cor}\right)$</strong></td>
<td><strong>6.16 $10^{-6}$</strong></td>
</tr>
<tr>
<td>SMA, window size of the previous month (random walk model)</td>
<td>6.06 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.94$</td>
<td>5.78 $10^{-6}$</td>
</tr>
<tr>
<td><strong>IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$</strong></td>
<td><strong>5.78 $10^{-6}$</strong></td>
</tr>
</tbody>
</table>
<p>Within this specific evaluation framework, the IEWMA covariance matrix forecasting model unfortunately does not seem to improve upon the previous models despite the added complexity.</p>
<p>It is also noteworthy that the IEWMA model with asset-specific univariate EWMA models for volatility (line #6) does not exhibit better performances than the vanilla
IEWMA model (line #7) when using automatically determined parameters<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">21</a></sup>.</p>
<h3 id="results---correlation-matrix-forecasting">Results - Correlation matrix forecasting</h3>
<p>Results over the period 31st January 2008 - 31st July 2023<sup id="fnref:22:1" role="doc-noteref"><a href="#fn:22" class="footnote">18</a></sup> for the correlation matrices associated to the covariance matrices of the previous sub-section are the following<sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>:</p>
<table>
<thead>
<tr>
<th>Covariance matrix model</th>
<th>Correlation matrix MSE</th>
</tr>
</thead>
<tbody>
<tr>
<td>SMA, window size of the previous month (random walk model)</td>
<td>8.19</td>
</tr>
<tr>
<td>SMA, window size of all the previous months (historical average model)</td>
<td>8.10</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.94$</td>
<td>7.67</td>
</tr>
<tr>
<td>SMA, window size of the previous year</td>
<td>6.50</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.97$</td>
<td>6.36</td>
</tr>
<tr>
<td>EWMA, optimal<sup id="fnref:24:3" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\lambda$</td>
<td>5.87</td>
</tr>
<tr>
<td><strong>IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$</strong></td>
<td><strong>5.85</strong></td>
</tr>
<tr>
<td><strong>IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$</strong></td>
<td><strong>5.85</strong></td>
</tr>
<tr>
<td><strong>IEWMA, optimal<sup id="fnref:24:4" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$</strong></td>
<td><strong>5.72</strong></td>
</tr>
<tr>
<td><strong>IEWMA, optimal<sup id="fnref:24:5" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> $\left(\lambda_{vol},\lambda_{cor}\right)$</strong></td>
<td><strong>5.70</strong></td>
</tr>
</tbody>
</table>
<p>This time, the IEWMA model - and especially the two IEWMA models using automatically determined parameters (lines #9-#10) - does exhibit slightly better performances than all the previous models.</p>
<p>Nevertheless, the improvement over the previously best-performing model (line #6 - the EWMA model with automatically determined parameter) is not that impressive…</p>
<p>So, the idea of removing the impact of volatility on asset returns in order to better estimate asset correlations seems to have some merits, but the EWMA volatility forecasting model
might be insufficient to fully exploit this idea.</p>
<p>Also, here again, the performances of the IEWMA model with asset-specific univariate EWMA models for volatility are again strictly worse than the performances of the vanilla IEWMA model<sup id="fnref:19:1" role="doc-noteref"><a href="#fn:19" class="footnote">21</a></sup>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>One of the main characteristics of the IEWMA covariance matrix forecasting model of Johansson et al.<sup id="fnref:3:15" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> is to forecast asset correlations using (EWMA-)volatility-standardized asset returns
instead of raw asset returns, so that the impact of asset volatilities on their correlations is (tentatively) minimized.</p>
<p>Unfortunately, the empirical performances of that model in terms of correlation matrix forecasting are not that different from those of the EWMA model, which raises the question of whether
improving the volatility-standardization might lead to better correlation forecasts.</p>
<p>This will be the subject of a future blog post in this series.</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:15" role="doc-endnote">
<p>ChatGPT-generated, as can be seen by the signature word “delve” :-) ! <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>At the date of the initial publication of this blog post. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.nowpublishers.com/article/Details/ECO-047">Kasper Johansson, Mehmet G. Ogut, Markus Pelger, Thomas Schmelzer and Stephen Boyd (2023), A Simple Method for Predicting Covariance Matrices of Financial Returns, Foundations and Trends in Econometrics: Vol. 12: No. 4, pp 324-407</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:3:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:3:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:3:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:3:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:3:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:3:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:3:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a> <a href="#fnref:3:14" class="reversefootnote" role="doc-backlink">↩<sup>15</sup></a> <a href="#fnref:3:15" class="reversefootnote" role="doc-backlink">↩<sup>16</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/41/3/97">Valeriy Zakamulin, A Test of Covariance-Matrix Forecasting Methods, The Journal of Portfolio Management Spring 2015, 41 (3) 97-108</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-3-540-71297-8_36">Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1198/073500102288618487">Engle, R. (2002). Dynamic Conditional Correlation. Journal of Business & Economic Statistics. 20(3): 339–350</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:2:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:2:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:2:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:2:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:2:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:2:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>That being said, I find that what Engle<sup id="fnref:2:11" role="doc-noteref"><a href="#fn:2" class="footnote">6</a></sup> describes is closer to an iterated $n$-univariate GARCH volatility forecasting models/EWMA correlation matrix forecasting model than to the iterated EWMA forecasting model of Johansson et al.<sup id="fnref:3:16" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>… <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>C.f. the associated blog posts <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">here</a> and <a href="/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models/">there</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S0378426622000267">Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://joim.com/downloads/correlation-shrinkage-implications-for-risk-forecasting/">Menchero, Jose and Peng Li. Correlation Shrinkage: Implications for Risk Forecasting, Journal of Investment Management (2020)</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>Digging a little deeper, the old MAC1 and MAC2 Bloomberg multi-asset risk models were using an half-life of 26 weeks for estimating volatilities and an half-life of 52 weeks for estimating correlations. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://www.mfsociety.org/modules/modDashboard/uploadFiles/journals/googleScholar/684.html">Andersen, Torben G., Tim Bollerslev, Francis X. Diebold, and Paul Labys, 2000, Exchange Rate Returns Standardized by Realized Volatility are (Nearly) Gaussian, Multinational Finance Journal 4, 159-179.</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0264999310001392">Bahram Pesaran, M. Hashem Pesaran, Conditional volatility and correlations of weekly returns and the VaR analysis of 2008 stock market crash, Economic Modelling, Volume 27, Issue 6, 2010, Pages 1398-1416</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.msci.com/documents/10199/5915b101-4206-4ba0-aee2-3449d5c7e95a">RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>The maximization of <em>the average log-likelihood of the combined [covariance matrix] prediction over [a trailing number of periods]</em><sup id="fnref:3:17" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>This is because my own tests did not highlight any strong improvement in terms of forecasting ability v.s. the IEWMA model when used with an optimal pair of decay factors. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">22</a></sup>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>(Adjusted) daily prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:22:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:23:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>The optimal decay factors ($\lambda$, $\lambda_{vol}$, $\lambda_{cor}$) are computed at the end of every month using all the available asset returns history up to that point in time, as implemented in <strong>Portfolio Optimizer</strong>; thus, there is no look-ahead bias. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:24:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:24:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:24:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:24:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:24:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>The difference between the two models is probably just noise, which, at worst, implies that the added complexity of the IEWMA model with asset-specific univariate EWMA models is useless in practice. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:19:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.In the previous post of this series on covariance matrix forecasting, I reviewed both the simple and the exponentially weighted moving average covariance matrix forecasting models, which are straightforward extensions of their respective univariate volatility forecasting models to a multivariate setting. With these reference models established, we can now delve into more sophisticated approaches for forecasting covariance matrices1. In this blog post, I will describe the iterated exponentially weighted moving average (IEWMA) model that has recently2 been introduced in Johansson et al.3 and I will illustrate its empirical performances in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio. Mathematical preliminaries Covariance matrix modelling and covariance proxies (reminders) This sub-section contains reminders from a previous blog post. Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (logarithmic) return of these assets over a time period $t$ (a day, a week, a month..), over which the (conditional) mean return vector $\mu_t \in \mathbb{R}^n$ of these assets is supposed to be null. Then: $r_t$ can be expressed as4 $r_t = \epsilon_t$, with $\epsilon_t \in \mathbb{R}^n$ an unpredictable error term, often referred to as a vector of “shocks” or as a vector of “random disturbances”4, over the time period $t$ The asset (conditional) covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is defined as $\Sigma_t = \mathbb{E} \left[ r_t r_t {}^t \right]$ From this definition, the outer product of the asset returns $ r_t r_t {}^t $ is a covariance estimate - or covariance proxy5 - for the asset returns covariance matrix over the considered time period $t$. The asset (conditional) correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is defined as $ C_t = V_t^{-1} \Sigma_t V_t^{-1} $, where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset (conditional) standard deviations. Correlation matrix modelling In order to clarify the relation between conditional correlations and conditional variances6, Engle6 proposes to write the [asset] returns as the conditional standard deviation times the standardized disturbance6, that is \[r_{i,t} = \epsilon_{i,t} = \sigma_{i,t} \varepsilon_{i,t}, i=1..n\] , with: $\sigma_{i,t} = \sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] } $ $\varepsilon_{i,t}$ a standardized disturbance that has mean zero and variance one6 This way, the conditional correlation [between asset $i$ and asset $j$ becomes equal to] the conditional covariance between the standardized disturbances [$\varepsilon_{i,t}$ and $\varepsilon_{j,t}$]6 \[\rho_{ij,t} = \frac{\mathbb{E} \left[ r_{i,t} r_{j,t} \right]}{\sqrt{ \mathbb{E} \left[ r_{i,t}^2 \right] \mathbb{E} \left[ r_{j,t}^2 \right] }} = \mathbb{E} \left[ \varepsilon_{i,t} \varepsilon_{j,t} \right]\] The iterated exponentially weighted moving average covariance matrix forecasting model The IEWMA covariance matrix forecasting model3 is a two-step model that: Uses an EWMA volatility forecasting model with squared asset returns as variance proxies in order to forecast asset volatilities Uses an EWMA correlation matrix forecasting model with outer products of (EWMA-)volatility-standardized asset returns as covariance matrix proxies in order to forecast asset correlations Johansson et al.3 highlights that the IEWMA model was originally7 proposed in Engle6 as an efficient alternative to the DCC-GARCH6 predictor, although he did not refer to it as IEWMA6. In other words, the IEWMA covariance matrix forecasting model bridges the gap between the simple EWMA model and the more complex DCC-GARCH model. Forecasting formulas Let be: $n$ be the number of assets in a universe of assets $r_t r_t {}^t$, $t=1..T$ the outer products of the asset returns over each of $T$ past periods A decay factor $\lambda_{vol} \in [0, 1]$ A decay factor $\lambda_{cor} \in [0, 1]$ Next period’s asset returns covariance/correlation matrix The IEWMA covariance matrix forecasting model estimates the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$ as follows3: For each asset $i=1..n$ in the universe of assets Forecast the asset one-period-ahead variances $\hat{\sigma}^2_{i,2}$, …, $\hat{\sigma}^2_{i,T+1}$ using an EWMA volatility forecasting model with decay factor $\lambda_{vol}$ and squared asset returns $r_{i,t}^2$, $t=1..T$, as variance proxies Compute the volatility-standardized asset returns $\tilde{r}_{i,2},…,\tilde{r}_{i,T}$ defined as the asset returns standardized by their one-period-ahead EWMA volatility forecasts \[\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}, t = 2..T\] Compute the diagonal matrix of the assets next period’s forecasted volatilities $V_{T+1}$ \[V_{T+1} = \begin{pmatrix} \hat{\sigma}_{1,T+1} & 0 & ... & 0 \\ 0 & \hat{\sigma}_{2,T+1} & ... & 0 \\ ... & ... & ... & ... \\ 0 & 0 & ... & \hat{\sigma}_{n,T+1} \end{pmatrix}\] Forecast the next period’s asset returns correlation matrix $\hat{C}_{T+1}$ using an EWMA covariance matrix forecasting model with decay factor $\lambda_{cor}$ and outer products of volatility-standardized asset returns $\tilde{r_t} \tilde{r_t} {}^t$, $t=2..T$, as covariance matrix proxies Compute the next period’s forecasted asset returns covariance matrix $\hat{\Sigma}_{T+1}$ \[\hat{\Sigma}_{T+1} = V_{T+1} \hat{C}_{T+1} V_{T+1}\] Next $h$-period’s ahead asset returns covariance/correlation matrix The IEWMA covariance matrix forecasting model estimates the next $h$-period’s ahead asset returns covariance matrix $\hat{\Sigma}_{T+h}$ and correlation matrix $\hat{C}_{T+h}$, $h \geq 2$, by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$. Indeed, due to the properties of the EWMA volatility and covariance matrix forecasting models8, we have: $V_{T+h} = V_{T+1}$, $h \geq 2$ $\hat{C}_{T+h} = \hat{C}_{T+1}$, $h \geq 2$ So that $\hat{\Sigma}_{T+h} = \hat{\Sigma}_{T+1}$, $h \geq 2$. Averaged asset returns covariance/correlation matrix over the next $h$ periods The IEWMA covariance matrix forecasting model estimates the averaged9 asset returns covariance matrix $\hat{\Sigma}_{T+1:T+h}$ and correlation matrix $\hat{C}_{T+1:T+h}$ over the next $h$ periods, $h \geq 2$, by the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$ and correlation matrix $\hat{C}_{T+1}$. Indeed, from the previous sub-section, we have: $ \hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i} = \hat{\Sigma}_{T+1} $, $h \geq 2$ $ \hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i} = \hat{C}_{T+1} $, $h \geq 2$ Rationale The rationale behind the IEWMA covariance matrix forecasting model is twofold: Separate volatility forecasting from correlation matrix forecasting while using the same baseline forecasting model for internal consistency. This first idea is well known among practitioners, c.f. for example Menchero and Li10 which describes the usage of different EWMA half-lives for estimating volatilities and correlations11. Use volatility-standardized asset returns instead of raw asset returns for correlation matrix forecasting. This second idea, detailled for example in Engle6, originates from the fact that conditional correlations between asset returns are equal to conditional covariances between standardized disturbances, c.f. the previous section. So, given an estimator for the (unobservable) standardized disturbances $\epsilon_{i,t} = \frac{r_{i,t}}{\sigma_{i,t}}$, $i=1..n$, $t=1..T$, it is possible to estimate the conditional correlations between asset returns. An example of such estimator is the volatility-standardized asset returns $\tilde{r}_{i,t} = \frac{r_{i,t}}{\hat{\sigma}_{i,t}}$, with the drawback that the quality of the correlation forecasts is then influenced by the quality of the volatility forecasts. And unfortunately, because it is well known that […] asset returns […] fat tails are typically reduced but not eliminated when returns are standardized by volatilities estimated from popular [volatility forecasting] models12 and because the use of correlation as a measure of dependence can be misdealing in the case of (conditionally) non-Gaussian returns13, we should not expect any magic here… How to choose the decay factors? Due to its relationship with the vanilla EWMA volatility and covariance matrix forecasting models, there are two main procedures to choose the decay factors $\lambda_{vol}$ and $\lambda_{cor}$ of an IEWMA covariance matrix forecasting model: Using recommended values from the EWMA models literature (0.94, 0.97…). On this, Johansson et al.3 notes that empirical studies on real return data confirm that choosing a faster volatility half-life than correlation half-life yields better estimates6 and uses the following pairs of decay factors $\left(\lambda_{vol}, \lambda_{cor}\right)$ in their experimental setup: Short term - $\left(0.870,0.933\right)$, $\left(0.933,0.967\right)$ Medium term - $\left(0.967,0.989\right)$, $\left(0.989,0.994\right)$, $\left(0.994,0.997\right)$ Long term - $\left(0.997,0.998\right)$, $\left(0.998,0.999\right)$ Determining the optimal values w.r.t. the forecast horizon $h$, for example through the minimization of the root mean square error (RMSE) between the forecasted covariance matrix over the desired horizon and the observed covariance matrix over that horizon14. In practice, because there are two decay factors to choose, this can be done two ways: Either consider the two decay factors as a two independent univariate parameters $\lambda_{vol} \in [0,1]$ and $\lambda_{cor} \in [0,1]$. This choice is justified by the original desire to separate volatility forecasting from correlation matrix forecasting. Or consider the two decay factors as a single multivariate parameter $\left(\lambda_{vol}, \lambda_{cor}\right) \in [0,1]^2$. This choice is justified by the observed dependency of the correlation forecasts on the volatility-standardized asset returns. Extensions of the iterated exponentially weighted moving average covariance matrix forecasting model Asset-specific volatility decay factors The IEWMA covariance matrix forecasting model uses univariate EWMA models in order to forecast asset volatilities, all these models sharing the same decay factor $\lambda_{vol}$. Having an identical decay factor for all assets is parsimonious, but is somewhat at odds with the DCC-GARCH model of Engle6 which uses asset-specific univariate GARCH models - that is, each with its own asset-specific parameters - in order to forecast asset volatilities. So, one natural extension of the IEWMA model is to allow asset-specific univariate EWMA models - each with its own asset-specific decay factor $\lambda_{i,vol}$, $i=1..n$ - in order to forecast asset volatilities. Linear combination of IEWMA covariance matrix forecasting models Another interesting covariance matrix forecasting model is introduced in Johansson et al.3 as the combined multiple iterated exponentially weighted moving average (CM-IEWMA) model, which consist in a time-varying linear combination of individual IEWMA models, each with its own pair of fixed decay factors. As explained in Johansson et al.3: The CM-IEWMA predictor is constructed from a modest number of IEWMA predictors, with different pairs of half-lives, which are combined using dynamically varying weights that are based on recent performance. The rationale behind the CM-IEWMA model is that different pairs of half-lives may work better for different market conditions3, with short half-lives [typically performing] better in volatile markets [and] long half-lives [performing] better for calm markets where conditions are changing slowly3. This behaviour is illustrated in Figure 1, taken from Johansson et al.3, which shows the evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks. Figure 1. Evolution of the weights of a 5-IEWMA CM-IEWMA covariance matrix forecasting model applied to a universe of U.S. stocks, 4th January 2010 - 30th December 2022. Source: Johansson et al. From Figure 1, it is visible that although substantial weight is put on the slower (longer halflife) IEWMAs most years3, the CM-IEWMA model still adapts the weights depending on market conditions3. The interested reader is referenced to Johansson et al.3 for all the technicalities of the CM-IEWMA model, and in particular for the details about the computation of the dynamically varying weights of the individual IEWMA models through the resolution of a convex optimization problem15. A last important remark to conclude this sub-section - the CM-IEWMA model is actually a special case of [a more general] dynamically weighted prediction [model]3, so that the same weighting logic can be applied to any combination of covariance matrix forecasting models. Implementations Implementation in Portfolio Optimizer Portfolio Optimizer implements the IEWMA covariance and correlation matrix forecasting model through the endpoints /assets/covariance/matrix/forecast/iewma and /assets/correlation/matrix/forecast/iewma. These endpoints support the 2 covariance proxies below: Squared (close-to-close) returns Demeaned squared (close-to-close) returns These endpoints also allow: To use asset-specific univariate EWMA models in order to forecast asset volatilities To automatically determine the optimal value of their parameters (the decay factors $\lambda_{vol}$ and $\lambda_{cor}$) using a proprietary procedure. To be noted that Portfolio Optimizer does not provide any implementation of the CM-IEWMA model16, but c.f. the next sub-section. Implementation elsewhere Johansson et al.3 kindly provides an open source Python implementation of: The IEWMA covariance matrix forecasting model The CM-IEWMA covariance matrix forecasting model The general “covariance matrix forecasting models” combination model at https://github.com/cvxgrp/cov_pred_finance. I definitely encourage anyone interested in the CM-IEMWA model to play with this code! Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs As an example of usage, I propose to evaluate the empirical performances of the IEWMA covariance matrix forecating model within the framework of the previous blog bost, whose aim is to forecast monthly covariance and correlation matrices for a portfolio of 10 ETFs representative17 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Results - Covariance matrix forecasting Results over the period 31st January 2008 - 31st July 202318 for covariance matrices are the following19: Covariance matrix model Covariance matrix MSE SMA, window size of all the previous months (historical average model) 9.59 $10^{-6}$ SMA, window size of the previous year 9.08 $10^{-6}$ EWMA, optimal20 $\lambda$ 6.52 $10^{-6}$ EWMA, $\lambda = 0.97$ 6.37 $10^{-6}$ IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$ 6.35 $10^{-6}$ IEWMA, optimal20 $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$ 6.33 $10^{-6}$ IEWMA, optimal20 $\left(\lambda_{vol},\lambda_{cor}\right)$ 6.16 $10^{-6}$ SMA, window size of the previous month (random walk model) 6.06 $10^{-6}$ EWMA, $\lambda = 0.94$ 5.78 $10^{-6}$ IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$ 5.78 $10^{-6}$ Within this specific evaluation framework, the IEWMA covariance matrix forecasting model unfortunately does not seem to improve upon the previous models despite the added complexity. It is also noteworthy that the IEWMA model with asset-specific univariate EWMA models for volatility (line #6) does not exhibit better performances than the vanilla IEWMA model (line #7) when using automatically determined parameters21. Results - Correlation matrix forecasting Results over the period 31st January 2008 - 31st July 202318 for the correlation matrices associated to the covariance matrices of the previous sub-section are the following19: Covariance matrix model Correlation matrix MSE SMA, window size of the previous month (random walk model) 8.19 SMA, window size of all the previous months (historical average model) 8.10 EWMA, $\lambda = 0.94$ 7.67 SMA, window size of the previous year 6.50 EWMA, $\lambda = 0.97$ 6.36 EWMA, optimal20 $\lambda$ 5.87 IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.94,0.97\right)$ 5.85 IEWMA, $\left(\lambda_{vol},\lambda_{cor}\right) = \left(0.97,0.99\right)$ 5.85 IEWMA, optimal20 $\lambda_{i,vol}$, $i=1..n$ and $\lambda_{cor}$ 5.72 IEWMA, optimal20 $\left(\lambda_{vol},\lambda_{cor}\right)$ 5.70 This time, the IEWMA model - and especially the two IEWMA models using automatically determined parameters (lines #9-#10) - does exhibit slightly better performances than all the previous models. Nevertheless, the improvement over the previously best-performing model (line #6 - the EWMA model with automatically determined parameter) is not that impressive… So, the idea of removing the impact of volatility on asset returns in order to better estimate asset correlations seems to have some merits, but the EWMA volatility forecasting model might be insufficient to fully exploit this idea. Also, here again, the performances of the IEWMA model with asset-specific univariate EWMA models for volatility are again strictly worse than the performances of the vanilla IEWMA model21. Conclusion One of the main characteristics of the IEWMA covariance matrix forecasting model of Johansson et al.3 is to forecast asset correlations using (EWMA-)volatility-standardized asset returns instead of raw asset returns, so that the impact of asset volatilities on their correlations is (tentatively) minimized. Unfortunately, the empirical performances of that model in terms of correlation matrix forecasting are not that different from those of the EWMA model, which raises the question of whether improving the volatility-standardization might lead to better correlation forecasts. This will be the subject of a future blog post in this series. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – ChatGPT-generated, as can be seen by the signature word “delve” :-) ! ↩ At the date of the initial publication of this blog post. ↩ See Kasper Johansson, Mehmet G. Ogut, Markus Pelger, Thomas Schmelzer and Stephen Boyd (2023), A Simple Method for Predicting Covariance Matrices of Financial Returns, Foundations and Trends in Econometrics: Vol. 12: No. 4, pp 324-407. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 See Valeriy Zakamulin, A Test of Covariance-Matrix Forecasting Methods, The Journal of Portfolio Management Spring 2015, 41 (3) 97-108. ↩ ↩2 See Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg. ↩ See Engle, R. (2002). Dynamic Conditional Correlation. Journal of Business & Economic Statistics. 20(3): 339–350. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 That being said, I find that what Engle6 describes is closer to an iterated $n$-univariate GARCH volatility forecasting models/EWMA correlation matrix forecasting model than to the iterated EWMA forecasting model of Johansson et al.3… ↩ C.f. the associated blog posts here and there. ↩ See Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426. ↩ See Menchero, Jose and Peng Li. Correlation Shrinkage: Implications for Risk Forecasting, Journal of Investment Management (2020). ↩ Digging a little deeper, the old MAC1 and MAC2 Bloomberg multi-asset risk models were using an half-life of 26 weeks for estimating volatilities and an half-life of 52 weeks for estimating correlations. ↩ See Andersen, Torben G., Tim Bollerslev, Francis X. Diebold, and Paul Labys, 2000, Exchange Rate Returns Standardized by Realized Volatility are (Nearly) Gaussian, Multinational Finance Journal 4, 159-179.. ↩ See Bahram Pesaran, M. Hashem Pesaran, Conditional volatility and correlations of weekly returns and the VaR analysis of 2008 stock market crash, Economic Modelling, Volume 27, Issue 6, 2010, Pages 1398-1416. ↩ See RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition. ↩ The maximization of the average log-likelihood of the combined [covariance matrix] prediction over [a trailing number of periods]3. ↩ This is because my own tests did not highlight any strong improvement in terms of forecasting ability v.s. the IEWMA model when used with an optimal pair of decay factors. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer22. ↩ (Adjusted) daily prices have have been retrieved using Tiingo. ↩ ↩2 Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. ↩ ↩2 The optimal decay factors ($\lambda$, $\lambda_{vol}$, $\lambda_{cor}$) are computed at the end of every month using all the available asset returns history up to that point in time, as implemented in Portfolio Optimizer; thus, there is no look-ahead bias. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 The difference between the two models is probably just noise, which, at worst, implies that the added complexity of the IEWMA model with asset-specific univariate EWMA models is useless in practice. ↩ ↩2 See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩From Volatility Forecasting to Covariance Matrix Forecasting: The Return of Simple and Exponentially Weighted Moving Average Models2024-10-12T00:00:00-05:002024-10-12T00:00:00-05:00https://portfoliooptimizer.io/blog/from-volatility-forecasting-to-covariance-matrix-forecasting-the-return-of-simple-and-exponentially-weighted-moving-average-models<p>In the <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">initial post</a> of the series on volatility forecasting, I described the simple and the exponentially
weighted moving average forecasting models, that are both easy to understand and relatively performant in practice.</p>
<p>Beyond (univariate) volatility forecasting, these two models are also widely used in (multivariate) covariance matrix forecasting<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup><sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">3</a></sup>, for the very same reasons.</p>
<p>In this blog post, I will detail the simple and exponentially weighted moving average covariance matrix forecasting models and I will illustrate their empirical performances
in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="covariance-matrix-modelling">Covariance matrix modelling</h3>
<p>Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>)
return of these assets over a time period $t$.</p>
<p>In all generality, $r_t$ can be expressed as<sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup></p>
\[r_t = \mu_t + \epsilon_t\]
<p>, where:</p>
<ul>
<li>$\mu_t \in \mathbb{R}^n$, $\mu_t = \mathbb{E} \left[ r_t \right]$, is a predictable quantity representing the vector of the (conditional) assets mean return over the time period $t$</li>
<li>$\epsilon_t \in \mathbb{R}^n$, $\epsilon_t = r_t - \mathbb{E} \left[ r_t \right]$, is an unpredictable error term, often referred to as a vector of “shocks” or as a vector of “random disturbances”<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>, over the time period $t$</li>
</ul>
<p>The asset (conditional) covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is then defined by<sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup></p>
\[\begin{aligned}
\Sigma_t &= Cov \left[ r_t \right] \\
&= \mathbb{E} \left[ \left( r_t - \mu_t \right) \left( r_t - \mu_t \right) {}^t \right] \\
&= \mathbb{E} \left[ r_t r_t {}^t \right] - \mathbb{E} \left[ r_t \right] \mathbb{E} \left[ r_t \right] {}^t \\
&= \mathbb{E} \left[ r_t r_t {}^t \right] - \mu_t \mu_t {}^t \\
\end{aligned}\]
<p>From this general model for asset returns, it is possible to derive different models for the asset covariance matrix depending on working assumptions.</p>
<p>In this blog post, the main working assumption will be that the assets mean return over any of the considered time periods $t$ is zero, that is, $\mu_t = 0_n$.</p>
<p>From <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">a previous blog post</a>, such an assumption is routinely made when working with daily stock returns
but is also empirically justified when working with lower frequency data on a wide range of assets.</p>
<p>Indeed, as highlighted by Johansson et al.<sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup></p>
<blockquote>
<p>[…] the mean [$\mu_t$] is small enough […] for most daily, weekly, or monthly stock, bond, and futures returns, factor returns, and index returns.</p>
</blockquote>
<h3 id="covariance-proxies">Covariance proxies</h3>
<p>Under the working assumption of the previous sub-section, the covariance matrix $\Sigma_t$ is then equal<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">4</a></sup> to the second moment $\mathbb{E} \left[ r_t r_t {}^t \right]$.</p>
<p>As a consequence, the <a href="https://en.wikipedia.org/wiki/Outer_product">outer product</a> of the asset returns $ r_t r_t {}^t $ over a time period $t$ (a day, a week, a month..)
is a covariance estimate $\tilde{\Sigma}_t$ - or <em>covariance proxy</em><sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">5</a></sup> - for the asset returns covariance matrix over the considered time period.</p>
<p>To be noted that other covariance proxies exist, like realized covariances<sup id="fnref:9:1" role="doc-noteref"><a href="#fn:9" class="footnote">5</a></sup> or asset regularized returns<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">6</a></sup>, but they will not be discussed in this blog post.</p>
<h3 id="correlation-matrix-modelling">Correlation matrix modelling</h3>
<p>The asset (conditional) correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is related to the asset covariance matrix $\Sigma_t$ through the standard formula<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">7</a></sup>:</p>
\[C_t = V_t^{-1} \Sigma_t V_t^{-1}\]
<p>, where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset standard deviations</p>
\[V_t = \begin{pmatrix}
\sqrt {\left(\Sigma_t\right)_{1,1}} & 0 & ... & 0 \\
0 & \sqrt {\left(\Sigma_t\right)_{2,2}} & ... & 0 \\
... & ... & ... & ... \\
0 & 0 & ... & \sqrt {\left(\Sigma_t\right)_{n,n}}
\end{pmatrix}\]
<h2 id="the-simple-moving-average-covariance-matrix-forecasting-model">The simple moving average covariance matrix forecasting model</h2>
<p>The simple moving average (SMA) covariance matrix forecasting model<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">8</a></sup>, also known as the rolling historical covariance matrix forecasting model<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>, uses
<em>an equally weighted moving average [of past covariance proxies $\tilde{\Sigma}_{t}$, $t=1..T$] calculated on a […] data window [of fixed size $1 \leq k \leq T$]
that is rolled over time</em><sup id="fnref:4:5" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup> in order to forecast the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$.</p>
<h3 id="forecasting-formulas">Forecasting formulas</h3>
<p>Under a simple moving average covariance matrix forecasting model, forecasting formulas are:</p>
<ul>
<li>
<p>To estimate the next period’s asset returns covariance/correlation matrix:</p>
\[\hat{\Sigma}_{T+1} = \frac{1}{k} \sum_{i=1}^{k} \tilde{\Sigma}_{T+1-i}\]
\[\hat{C}_{T+1} = \hat{V}_{T+1}^{-1} \hat{\Sigma}_{T+1} \hat{V}_{T+1}^{-1}\]
</li>
<li>
<p>To estimate the next $h$-period’s ahead asset returns covariance/correlation matrix, $h \geq 2$:</p>
\[\hat{\Sigma}_{T+h} = \frac{1}{k} \left( \sum_{i=1}^{k-h+1} \tilde{\Sigma}_{T+1-i} + \sum_{i=1}^{h-1} \hat{\Sigma}_{T+h-i} \right)\]
\[\hat{C}_{T+h} = \hat{V}_{T+h}^{-1} \hat{\Sigma}_{T+h} \hat{V}_{T+h}^{-1}\]
</li>
<li>
<p>To estimate the averaged asset returns covariance/correlation matrix<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">6</a></sup> over the next $h$ periods:</p>
\[\hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i}\]
\[\hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i}\]
</li>
</ul>
<h3 id="specific-cases">Specific cases</h3>
<p>The simple moving average covariance matrix forecasting model encompasses two specific models:</p>
<ul>
<li>
<p>The random walk model, which corresponds to $k = 1$.</p>
<p>Under this model, the forecast of the next period’s asset returns covariance matrix is the current period’s asset returns covariance matrix.</p>
</li>
<li>
<p>The historical average model, which corresponds to $k = T$.</p>
<p>Under this model, the forecast of the next period’s asset returns covariance matrix is the long term average of the past periods’ asset returns covariance matrix.</p>
</li>
</ul>
<h3 id="how-to-choose-the-window-size">How to choose the window size?</h3>
<p>Like for its univariate couterpart, selecting the “best” window size $k$ of a simple moving average covariance matrix forecasting model is a problem in itself, c.f.
<a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the associated blog post</a>.</p>
<p>In addition, and this time specific to the multivariate nature of that forecasting model, the <em>rolling window covariance estimate is not full rank</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup> when $k < n$,
so that specific post-processing might need to be implemented in order to ensure that covariance matrix forecasts are positive definite<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">9</a></sup>.</p>
<h2 id="the-exponentially-weighted-moving-average-covariance-matrix-forecasting-model">The exponentially weighted moving average covariance matrix forecasting model</h2>
<p>Menchero and Morozov<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">10</a></sup> notes that:</p>
<blockquote>
<p>If return distributions were stationary, then using the maximum sample size and equally weighting every observation [- that is, using an historical average forecasting model, c.f. the previous section -] would minimize sampling error and hence produce the most accurate forecasts. Return distributions, however, are not stationary. Events that occurred 10 years ago have little to do with current [ones]. Therefore, to reflect current market conditions, we must give more weight to recent observations.</p>
</blockquote>
<p>This remark leads to the introduction of the exponentially weighted moving average (EWMA) covariance matrix forecasting model<sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">8</a></sup>, which, thanks to a decay factor $\lambda \in [0, 1]$,
gives $\lambda$-exponentially less emphasis to distant past covariance proxies v.s. more recent ones in order to forecast the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">11</a></sup>.</p>
<p>Incidentally, the exponentially weighted moving average covariance matrix forecasting model is <em>perhaps the most widely used […] model among practitioners</em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">12</a></sup>, in particular due<br />
to <em>its use in the RiskMetrics VaR software of J.P. Morgan</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">12</a></sup>.</p>
<h3 id="forecasting-formulas-1">Forecasting formulas</h3>
<p>Under an exponentially weighted moving average covariance matrix forecasting model, forecasting formulas are:</p>
<ul>
<li>
<p>To estimate the next period’s asset returns covariance/correlation matrix:</p>
\[\hat{\Sigma}_{T+1} = \frac{1 - \lambda}{1 - \lambda^{T}} \sum_{i=1}^{T} \lambda^{T-i} \tilde{\Sigma}_{i}\]
\[\hat{C}_{T+1} = \hat{V}_{T+1}^{-1} \hat{\Sigma}_{T+1} \hat{V}_{T+1}^{-1}\]
</li>
<li>
<p>To estimate the next $h$-period’s ahead asset returns covariance/correlation matrix, $h \geq 2$:</p>
\[\hat{\Sigma}_{T+h} = \hat{\Sigma}_{T+1}\]
\[\hat{C}_{T+h} = \hat{C}_{T+1}\]
<p>This result means that covariance and correlation matrix forecasts beyond the (immediate) next period are all equal to the covariance and correlation matrix forecast for that next period,
in a kind of random walk model way, and is a known limitation of this model when multi-period ahead forecasts are required.</p>
</li>
<li>
<p>To estimate the averaged asset returns covariance/correlation matrix<sup id="fnref:11:2" role="doc-noteref"><a href="#fn:11" class="footnote">6</a></sup> over the next $h$ periods:</p>
\[\hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i} = \hat{\Sigma}_{T+1}\]
\[\hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i} = \hat{C}_{T+1}\]
</li>
</ul>
<h3 id="how-to-choose-the-decay-factor">How to choose the decay factor?</h3>
<p>Like its univariate couterpart, there are essentially two<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">13</a></sup> procedures to choose the decay factor $\lambda$ of an exponentially weighted moving average covariance matrix forecasting model:</p>
<ul>
<li>
<p>Using recommended values from the literature (0.94, 0.97…).</p>
</li>
<li>
<p>Determining the optimal value w.r.t. the forecast horizon $h$, for example through the minimization of the <a href="https://en.wikipedia.org/wiki/Root-mean-square_deviation">root mean square error (RMSE)</a> between the forecasted covariance matrix over the desired horizon and the observed covariance matrix over that horizon<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">14</a></sup>.</p>
</li>
</ul>
<p>C.f. the <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the associated blog post</a> for more details.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements:</p>
<ul>
<li>The simple moving average covariance and correlation matrix forecasting model through the endpoints <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/covariance/matrix/forecast/sma</code></a> and <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/correlation/matrix/forecast/sma</code></a></li>
<li>The exponentially weighted moving average covariance and correlation matrix forecasting model through the endpoints <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/covariance/matrix/ewma</code></a> and <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/correlation/matrix/forecast/ewma</code></a></li>
</ul>
<p>All these endpoints support the 2 covariance proxies below:</p>
<ul>
<li>Squared (close-to-close) returns</li>
<li>Demeaned squared (close-to-close) returns</li>
</ul>
<p>The second group of endpoints allows to automatically determine the optimal value of its parameter (the decay factor $\lambda$) using a proprietary variation of the procedures described in
the RiskMetrics technical document<sup id="fnref:18:1" role="doc-noteref"><a href="#fn:18" class="footnote">14</a></sup>.</p>
<h2 id="example-of-usage---covariance-matrix-forecasting-at-monthly-level-for-a-portfolio-of-various-etfs">Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs</h2>
<p>As an example of usage, I propose to evaluate the empirical performances of the simple and exponentially weighted moving average covariance matrix forecasting models in the context
of a portfolio of 10 ETFs representative<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">15</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<h3 id="methodology">Methodology</h3>
<p>At the end of each month, I will compare the (averaged) covariance matrix forecasts produced by the simple
and exponentially weighted moving average covariance matrix forecasting models<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">16</a></sup> at a one-month horizon to the next month’s empirical covariance matrix.</p>
<p>The performance criterion considered will be the <a href="https://en.wikipedia.org/wiki/Mean_squared_error">mean squared error (MSE)</a> between the forecasted and the observed covariance matrices,
which is a direct<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">17</a></sup> method to evaluate the out-of-sample forecast accuracy of a covariance matrix forecasting model, c.f. for example in Johansson et al.<sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup></p>
<h3 id="results">Results</h3>
<p>Results over the period 31st January 2008 - 31st July 2023<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">18</a></sup> are the following<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>, with a graphical illustration on Figure 1:</p>
<table>
<thead>
<tr>
<th>Covariance matrix model</th>
<th>Covariance matrix MSE</th>
</tr>
</thead>
<tbody>
<tr>
<td>SMA, window size of all the previous months (historical average model)</td>
<td>9.59 $10^{-6}$</td>
</tr>
<tr>
<td>SMA, window size of the previous year</td>
<td>9.08 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup></td>
<td>6.52 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.97$</td>
<td>6.37 $10^{-6}$</td>
</tr>
<tr>
<td>SMA, window size of the previous month (random walk model)</td>
<td>6.06 $10^{-6}$</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.94$</td>
<td>5.78 $10^{-6}$</td>
</tr>
</tbody>
</table>
<figure>
<a href="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-covariance-matrix-mse.png"><img src="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-covariance-matrix-mse.png" alt="Next month's covariance matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023." /></a>
<figcaption>Figure 1. Next month's covariance matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023.</figcaption>
</figure>
<p>A couple of general remarks from Figure 1:</p>
<ul>
<li>Both covariance matrix forecasting models seem to do an excellent job at forecasting next month’s empirical covariance matrix; unfortunately, the very low MSEs are misleading - more on this in the next sub-section.</li>
<li>The exponentially weighted moving average covariance matrix forecasting model is generally more performant than the simple moving average covariance matrix forecasting model - provided a proper decay factor is used - which is consistent with the higher reactivity of that model.</li>
<li>There is a penalty to pay in terms of MSE for automatically determining the optimal decay factor of the exponentially weighted moving average covariance matrix forecasting model v.s. using a pre-defined value.</li>
</ul>
<h3 id="results-take-2">Results, take 2</h3>
<p>Because a covariance matrix depends on both asset variances and correlations, let’s have a look at the MSE<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">21</a></sup> between
the forecasted and the observed correlation matrices associated to the covariance matrices of the previous sub-section, with a graphical illustration on Figure 2:</p>
<table>
<thead>
<tr>
<th>Covariance matrix model</th>
<th>Correlation matrix MSE</th>
</tr>
</thead>
<tbody>
<tr>
<td>SMA, window size of the previous month (random walk model)</td>
<td>8.19</td>
</tr>
<tr>
<td>SMA, window size of all the previous months (historical average model)</td>
<td>8.10</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.94$</td>
<td>7.67</td>
</tr>
<tr>
<td>SMA, window size of the previous year</td>
<td>6.50</td>
</tr>
<tr>
<td>EWMA, $\lambda = 0.97$</td>
<td>6.36</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$<sup id="fnref:24:1" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup></td>
<td>5.87</td>
</tr>
</tbody>
</table>
<figure>
<a href="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-correlation-matrix-mse.png"><img src="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-correlation-matrix-mse.png" alt="Next month's correlation matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023." /></a>
<figcaption>Figure 2. Next month's correlation matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023.</figcaption>
</figure>
<p>It appears that the correlation matrix forecasts are actually pretty bad, or at the very least, not as good as the covariance matrix forecasts MSEs have led us to believe in the previous sub-section!</p>
<p>The explanation is that the covariance matrix forecasts MSEs are artificially deflated due to the difference in scale<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">22</a></sup> between asset returns variances and correlations, so that they are
more indicative of the quality of the variances forecasts than of the correlations forecasts…</p>
<p>Question is then, how bad are the correlation matrix forecasts?</p>
<p>Figure 3<sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">21</a></sup> illustrates the MSEs obtained when forecasting the next month’s correlation matrix by the observed next month’s correlation matrix <a href="/blog/correlation-matrix-stress-testing-random-perturbations-of-a-correlation-matrix/">randomly perturbed by a given noise level</a>.</p>
<figure>
<a href="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-correlation-matrix-perturbation-vs-mse.png"><img src="/assets/images/blog/covariance-matrix-forecasting-sma-ewma-correlation-matrix-perturbation-vs-mse.png" alt="Next month's correlation matrix perturbation level v.s. resulting MSE, 10-ETF universe, 31st January 2008 - 31st July 2023." /></a>
<figcaption>Figure 3. Next month's correlation matrix perturbation level v.s. resulting MSE, 10-ETF universe, 31st January 2008 - 31st July 2023.</figcaption>
</figure>
<p>Figure 3 is interpreted as follows:</p>
<ul>
<li>When all the correlation coefficients of the forecasted next month’s correlation matrix are equal to those of the observed next month’s correlation matrix (perturbation level equal to 0), the resulting MSE is 0.</li>
<li>When all the correlation coefficients of the forecasted next month’s correlation matrix are within $\pm 1.4$ from those of the observed next month’s correlation matrix (perturbation level equal to 1.4), the resulting MSE is ~35.</li>
</ul>
<p>Reverse-reading Figure 3, it is thus possible to conclude that:</p>
<ul>
<li>The correlation coefficients forecasted by the worst performing covariance forecasting model (MSE ~8.19) are on average within $\pm 0.60$ of the observed correlation coefficients.</li>
<li>The correlation coefficients forecasted by the best performing covariance forecasting model (MSE ~5.87) are on average within $\pm 0.50$ of the observed correlation coefficients.</li>
</ul>
<p>So, all in all, the correlation matrix forecasts produced by the simple and exponentially weighted moving average covariance matrix forecasting models are not that bad, but an average $\pm 0.50$ or $\pm 0.60$
uncertainty around the forecasted correlations would definitely gain to be improved.</p>
<p>As a final remark here, one should always be wary of covariance matrix forecasting results reported in terms of “covariance” MSEs…</p>
<h2 id="conclusion">Conclusion</h2>
<p>This first blog post on covariance matrix forecasting introduced two baseline models, which are direct extensions of volatility forecasting models.</p>
<p>The observed discrepencies between those models’ covariance and correlation matrix forecasting performances lead to wonder whether it is possible to improve these models
by separating the covariance matrix forecasting process into two steps - a first volatility forecasting step and a second correlation matrix forecasting step.</p>
<p>This will be the subject of the next blog post in this series.</p>
<p>Meanwhile, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.nowpublishers.com/article/Details/ECO-047">Kasper Johansson, Mehmet G. Ogut, Markus Pelger, Thomas Schmelzer and Stephen Boyd (2023), A Simple Method for Predicting Covariance Matrices of Financial Returns, Foundations and Trends in Econometrics: Vol. 12: No. 4, pp 324-407</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/41/3/97">Valeriy Zakamulin, A Test of Covariance-Matrix Forecasting Methods, The Journal of Portfolio Management Spring 2015, 41 (3) 97-108</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:4:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=4854248">Guijarro-Ordonez, Jorge and van Beek, Misha and Dhaliwal, Amandeep, A Simple Responsive Covariance Matrix Forecaster for Multiple Horizons and Asset Classes</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Or at least, should be well approximated by that second moment… <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-3-540-71297-8_36">Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:9:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S0378426622000267">Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:11:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://en.wikipedia.org/wiki/Covariance_matrix#Inverse_of_the_covariance_matrix">Wikipedia, Covariance matrix</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>This covariance matrix forecasting model is a direct extension of <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">its volatility forecasting model counterpart</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>In all generality, specific post-processing might anyway need to be implemented to ensure that covariance matrix forecasts are well-conditioned, c.f. Johansson et al.<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup> <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/41/3/84">Jose Menchero, Andrei Morozov, Improving Risk Forecasts Through Cross-Sectional Observations, The Journal of Portfolio Management Spring 2015, 41 (3) 84-96</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>The exponentially weighted moving average covariance matrix forecasting model is <em>a special case of the diagonal vec multivariate GARCH model of Engle and Kroner<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">23</a></sup>, and corresponds to an integrated GARCH model with no constant vector</em><sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">12</a></sup>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0169207009000466">Richard D.F. Harris, Fatih Yilmaz, Estimation of the conditional variance–covariance matrix of returns using the intraday range, International Journal of Forecasting, Volume 26, Issue 1, 2010, Pages 180-194</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>Other procedures are described in the RiskMetrics technical document<sup id="fnref:18:2" role="doc-noteref"><a href="#fn:18" class="footnote">14</a></sup>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.msci.com/documents/10199/5915b101-4206-4ba0-aee2-3449d5c7e95a">RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:18:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:18:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">24</a></sup>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>As implemented in <strong>Portfolio Optimizer</strong>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>Direct methods <em>use a proxy for the true covariance matrix to evaluate the predictor, while indirect methods use the covariance predictor on tasks of interest, such as portfolio construction or portfolio tracking</em><sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup>. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>(Adjusted) daily prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>The optimal decay factor $\lambda$ is computed at the end of every month using all the available asset returns history up to that point in time, as implemented in <strong>Portfolio Optimizer</strong>; thus, there is no look-ahead bias. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:24:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>The maximum perturbation level on Figure 3 is limited to 1.4, because that level correponds to the MSE also reached by forecasting next month’s correlation matrix by a random correlation matrix; so, there is no practical interest in going beyond that level up to the theoretical limit of 2. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>Especially for the variance of daily asset returns, as is the case here. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://www.cambridge.org/core/journals/econometric-theory/article/abs/multivariate-simultaneous-generalized-arch/45D6E128F2EFA55BD28955093A1CE849">Engle RF, Kroner KF. Multivariate Simultaneous Generalized ARCH. Econometric Theory. 1995;11(1):122-150</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.In the initial post of the series on volatility forecasting, I described the simple and the exponentially weighted moving average forecasting models, that are both easy to understand and relatively performant in practice. Beyond (univariate) volatility forecasting, these two models are also widely used in (multivariate) covariance matrix forecasting123, for the very same reasons. In this blog post, I will detail the simple and exponentially weighted moving average covariance matrix forecasting models and I will illustrate their empirical performances in the context of monthly covariance matrix forecasting for a multi-asset class ETF portfolio. Mathematical preliminaries Covariance matrix modelling Let $n$ be the number of assets in a universe of assets and $r_t \in \mathbb{R}^n$ be the vector of the (logarithmic) return of these assets over a time period $t$. In all generality, $r_t$ can be expressed as2 \[r_t = \mu_t + \epsilon_t\] , where: $\mu_t \in \mathbb{R}^n$, $\mu_t = \mathbb{E} \left[ r_t \right]$, is a predictable quantity representing the vector of the (conditional) assets mean return over the time period $t$ $\epsilon_t \in \mathbb{R}^n$, $\epsilon_t = r_t - \mathbb{E} \left[ r_t \right]$, is an unpredictable error term, often referred to as a vector of “shocks” or as a vector of “random disturbances”2, over the time period $t$ The asset (conditional) covariance matrix $\Sigma_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is then defined by1 \[\begin{aligned} \Sigma_t &= Cov \left[ r_t \right] \\ &= \mathbb{E} \left[ \left( r_t - \mu_t \right) \left( r_t - \mu_t \right) {}^t \right] \\ &= \mathbb{E} \left[ r_t r_t {}^t \right] - \mathbb{E} \left[ r_t \right] \mathbb{E} \left[ r_t \right] {}^t \\ &= \mathbb{E} \left[ r_t r_t {}^t \right] - \mu_t \mu_t {}^t \\ \end{aligned}\] From this general model for asset returns, it is possible to derive different models for the asset covariance matrix depending on working assumptions. In this blog post, the main working assumption will be that the assets mean return over any of the considered time periods $t$ is zero, that is, $\mu_t = 0_n$. From a previous blog post, such an assumption is routinely made when working with daily stock returns but is also empirically justified when working with lower frequency data on a wide range of assets. Indeed, as highlighted by Johansson et al.2 […] the mean [$\mu_t$] is small enough […] for most daily, weekly, or monthly stock, bond, and futures returns, factor returns, and index returns. Covariance proxies Under the working assumption of the previous sub-section, the covariance matrix $\Sigma_t$ is then equal4 to the second moment $\mathbb{E} \left[ r_t r_t {}^t \right]$. As a consequence, the outer product of the asset returns $ r_t r_t {}^t $ over a time period $t$ (a day, a week, a month..) is a covariance estimate $\tilde{\Sigma}_t$ - or covariance proxy5 - for the asset returns covariance matrix over the considered time period. To be noted that other covariance proxies exist, like realized covariances5 or asset regularized returns6, but they will not be discussed in this blog post. Correlation matrix modelling The asset (conditional) correlation matrix $C_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is related to the asset covariance matrix $\Sigma_t$ through the standard formula7: \[C_t = V_t^{-1} \Sigma_t V_t^{-1}\] , where $V_t \in \mathcal{M}(\mathbb{R}^{n \times n})$ is the diagonal matrix of the asset standard deviations \[V_t = \begin{pmatrix} \sqrt {\left(\Sigma_t\right)_{1,1}} & 0 & ... & 0 \\ 0 & \sqrt {\left(\Sigma_t\right)_{2,2}} & ... & 0 \\ ... & ... & ... & ... \\ 0 & 0 & ... & \sqrt {\left(\Sigma_t\right)_{n,n}} \end{pmatrix}\] The simple moving average covariance matrix forecasting model The simple moving average (SMA) covariance matrix forecasting model8, also known as the rolling historical covariance matrix forecasting model2, uses an equally weighted moving average [of past covariance proxies $\tilde{\Sigma}_{t}$, $t=1..T$] calculated on a […] data window [of fixed size $1 \leq k \leq T$] that is rolled over time2 in order to forecast the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$. Forecasting formulas Under a simple moving average covariance matrix forecasting model, forecasting formulas are: To estimate the next period’s asset returns covariance/correlation matrix: \[\hat{\Sigma}_{T+1} = \frac{1}{k} \sum_{i=1}^{k} \tilde{\Sigma}_{T+1-i}\] \[\hat{C}_{T+1} = \hat{V}_{T+1}^{-1} \hat{\Sigma}_{T+1} \hat{V}_{T+1}^{-1}\] To estimate the next $h$-period’s ahead asset returns covariance/correlation matrix, $h \geq 2$: \[\hat{\Sigma}_{T+h} = \frac{1}{k} \left( \sum_{i=1}^{k-h+1} \tilde{\Sigma}_{T+1-i} + \sum_{i=1}^{h-1} \hat{\Sigma}_{T+h-i} \right)\] \[\hat{C}_{T+h} = \hat{V}_{T+h}^{-1} \hat{\Sigma}_{T+h} \hat{V}_{T+h}^{-1}\] To estimate the averaged asset returns covariance/correlation matrix6 over the next $h$ periods: \[\hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i}\] \[\hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i}\] Specific cases The simple moving average covariance matrix forecasting model encompasses two specific models: The random walk model, which corresponds to $k = 1$. Under this model, the forecast of the next period’s asset returns covariance matrix is the current period’s asset returns covariance matrix. The historical average model, which corresponds to $k = T$. Under this model, the forecast of the next period’s asset returns covariance matrix is the long term average of the past periods’ asset returns covariance matrix. How to choose the window size? Like for its univariate couterpart, selecting the “best” window size $k$ of a simple moving average covariance matrix forecasting model is a problem in itself, c.f. the associated blog post. In addition, and this time specific to the multivariate nature of that forecasting model, the rolling window covariance estimate is not full rank1 when $k < n$, so that specific post-processing might need to be implemented in order to ensure that covariance matrix forecasts are positive definite9. The exponentially weighted moving average covariance matrix forecasting model Menchero and Morozov10 notes that: If return distributions were stationary, then using the maximum sample size and equally weighting every observation [- that is, using an historical average forecasting model, c.f. the previous section -] would minimize sampling error and hence produce the most accurate forecasts. Return distributions, however, are not stationary. Events that occurred 10 years ago have little to do with current [ones]. Therefore, to reflect current market conditions, we must give more weight to recent observations. This remark leads to the introduction of the exponentially weighted moving average (EWMA) covariance matrix forecasting model8, which, thanks to a decay factor $\lambda \in [0, 1]$, gives $\lambda$-exponentially less emphasis to distant past covariance proxies v.s. more recent ones in order to forecast the next period’s asset returns covariance matrix $\hat{\Sigma}_{T+1}$11. Incidentally, the exponentially weighted moving average covariance matrix forecasting model is perhaps the most widely used […] model among practitioners12, in particular due to its use in the RiskMetrics VaR software of J.P. Morgan12. Forecasting formulas Under an exponentially weighted moving average covariance matrix forecasting model, forecasting formulas are: To estimate the next period’s asset returns covariance/correlation matrix: \[\hat{\Sigma}_{T+1} = \frac{1 - \lambda}{1 - \lambda^{T}} \sum_{i=1}^{T} \lambda^{T-i} \tilde{\Sigma}_{i}\] \[\hat{C}_{T+1} = \hat{V}_{T+1}^{-1} \hat{\Sigma}_{T+1} \hat{V}_{T+1}^{-1}\] To estimate the next $h$-period’s ahead asset returns covariance/correlation matrix, $h \geq 2$: \[\hat{\Sigma}_{T+h} = \hat{\Sigma}_{T+1}\] \[\hat{C}_{T+h} = \hat{C}_{T+1}\] This result means that covariance and correlation matrix forecasts beyond the (immediate) next period are all equal to the covariance and correlation matrix forecast for that next period, in a kind of random walk model way, and is a known limitation of this model when multi-period ahead forecasts are required. To estimate the averaged asset returns covariance/correlation matrix6 over the next $h$ periods: \[\hat{\Sigma}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{\Sigma}_{T+i} = \hat{\Sigma}_{T+1}\] \[\hat{C}_{T+1:T+h} = \frac{1}{h} \sum_{i=1}^{h} \hat{C}_{T+i} = \hat{C}_{T+1}\] How to choose the decay factor? Like its univariate couterpart, there are essentially two13 procedures to choose the decay factor $\lambda$ of an exponentially weighted moving average covariance matrix forecasting model: Using recommended values from the literature (0.94, 0.97…). Determining the optimal value w.r.t. the forecast horizon $h$, for example through the minimization of the root mean square error (RMSE) between the forecasted covariance matrix over the desired horizon and the observed covariance matrix over that horizon14. C.f. the the associated blog post for more details. Implementation in Portfolio Optimizer Portfolio Optimizer implements: The simple moving average covariance and correlation matrix forecasting model through the endpoints /assets/covariance/matrix/forecast/sma and /assets/correlation/matrix/forecast/sma The exponentially weighted moving average covariance and correlation matrix forecasting model through the endpoints /assets/covariance/matrix/ewma and /assets/correlation/matrix/forecast/ewma All these endpoints support the 2 covariance proxies below: Squared (close-to-close) returns Demeaned squared (close-to-close) returns The second group of endpoints allows to automatically determine the optimal value of its parameter (the decay factor $\lambda$) using a proprietary variation of the procedures described in the RiskMetrics technical document14. Example of usage - Covariance matrix forecasting at monthly level for a portfolio of various ETFs As an example of usage, I propose to evaluate the empirical performances of the simple and exponentially weighted moving average covariance matrix forecasting models in the context of a portfolio of 10 ETFs representative15 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Methodology At the end of each month, I will compare the (averaged) covariance matrix forecasts produced by the simple and exponentially weighted moving average covariance matrix forecasting models16 at a one-month horizon to the next month’s empirical covariance matrix. The performance criterion considered will be the mean squared error (MSE) between the forecasted and the observed covariance matrices, which is a direct17 method to evaluate the out-of-sample forecast accuracy of a covariance matrix forecasting model, c.f. for example in Johansson et al.1 Results Results over the period 31st January 2008 - 31st July 202318 are the following19, with a graphical illustration on Figure 1: Covariance matrix model Covariance matrix MSE SMA, window size of all the previous months (historical average model) 9.59 $10^{-6}$ SMA, window size of the previous year 9.08 $10^{-6}$ EWMA, optimal $\lambda$20 6.52 $10^{-6}$ EWMA, $\lambda = 0.97$ 6.37 $10^{-6}$ SMA, window size of the previous month (random walk model) 6.06 $10^{-6}$ EWMA, $\lambda = 0.94$ 5.78 $10^{-6}$ Figure 1. Next month's covariance matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023. A couple of general remarks from Figure 1: Both covariance matrix forecasting models seem to do an excellent job at forecasting next month’s empirical covariance matrix; unfortunately, the very low MSEs are misleading - more on this in the next sub-section. The exponentially weighted moving average covariance matrix forecasting model is generally more performant than the simple moving average covariance matrix forecasting model - provided a proper decay factor is used - which is consistent with the higher reactivity of that model. There is a penalty to pay in terms of MSE for automatically determining the optimal decay factor of the exponentially weighted moving average covariance matrix forecasting model v.s. using a pre-defined value. Results, take 2 Because a covariance matrix depends on both asset variances and correlations, let’s have a look at the MSE21 between the forecasted and the observed correlation matrices associated to the covariance matrices of the previous sub-section, with a graphical illustration on Figure 2: Covariance matrix model Correlation matrix MSE SMA, window size of the previous month (random walk model) 8.19 SMA, window size of all the previous months (historical average model) 8.10 EWMA, $\lambda = 0.94$ 7.67 SMA, window size of the previous year 6.50 EWMA, $\lambda = 0.97$ 6.36 EWMA, optimal $\lambda$20 5.87 Figure 2. Next month's correlation matrix forecasting model MSE, 10-ETF universe, 31st January 2008 - 31st July 2023. It appears that the correlation matrix forecasts are actually pretty bad, or at the very least, not as good as the covariance matrix forecasts MSEs have led us to believe in the previous sub-section! The explanation is that the covariance matrix forecasts MSEs are artificially deflated due to the difference in scale22 between asset returns variances and correlations, so that they are more indicative of the quality of the variances forecasts than of the correlations forecasts… Question is then, how bad are the correlation matrix forecasts? Figure 321 illustrates the MSEs obtained when forecasting the next month’s correlation matrix by the observed next month’s correlation matrix randomly perturbed by a given noise level. Figure 3. Next month's correlation matrix perturbation level v.s. resulting MSE, 10-ETF universe, 31st January 2008 - 31st July 2023. Figure 3 is interpreted as follows: When all the correlation coefficients of the forecasted next month’s correlation matrix are equal to those of the observed next month’s correlation matrix (perturbation level equal to 0), the resulting MSE is 0. When all the correlation coefficients of the forecasted next month’s correlation matrix are within $\pm 1.4$ from those of the observed next month’s correlation matrix (perturbation level equal to 1.4), the resulting MSE is ~35. Reverse-reading Figure 3, it is thus possible to conclude that: The correlation coefficients forecasted by the worst performing covariance forecasting model (MSE ~8.19) are on average within $\pm 0.60$ of the observed correlation coefficients. The correlation coefficients forecasted by the best performing covariance forecasting model (MSE ~5.87) are on average within $\pm 0.50$ of the observed correlation coefficients. So, all in all, the correlation matrix forecasts produced by the simple and exponentially weighted moving average covariance matrix forecasting models are not that bad, but an average $\pm 0.50$ or $\pm 0.60$ uncertainty around the forecasted correlations would definitely gain to be improved. As a final remark here, one should always be wary of covariance matrix forecasting results reported in terms of “covariance” MSEs… Conclusion This first blog post on covariance matrix forecasting introduced two baseline models, which are direct extensions of volatility forecasting models. The observed discrepencies between those models’ covariance and correlation matrix forecasting performances lead to wonder whether it is possible to improve these models by separating the covariance matrix forecasting process into two steps - a first volatility forecasting step and a second correlation matrix forecasting step. This will be the subject of the next blog post in this series. Meanwhile, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Kasper Johansson, Mehmet G. Ogut, Markus Pelger, Thomas Schmelzer and Stephen Boyd (2023), A Simple Method for Predicting Covariance Matrices of Financial Returns, Foundations and Trends in Econometrics: Vol. 12: No. 4, pp 324-407. ↩ ↩2 ↩3 ↩4 See Valeriy Zakamulin, A Test of Covariance-Matrix Forecasting Methods, The Journal of Portfolio Management Spring 2015, 41 (3) 97-108. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 See Guijarro-Ordonez, Jorge and van Beek, Misha and Dhaliwal, Amandeep, A Simple Responsive Covariance Matrix Forecaster for Multiple Horizons and Asset Classes. ↩ Or at least, should be well approximated by that second moment… ↩ See Patton, A.J., Sheppard, K. (2009). Evaluating Volatility and Correlation Forecasts. In: Mikosch, T., Kreiß, JP., Davis, R., Andersen, T. (eds) Handbook of Financial Time Series. Springer, Berlin, Heidelberg. ↩ ↩2 See Gianluca De Nard, Robert F. Engle, Olivier Ledoit, Michael Wolf, Large dynamic covariance matrices: Enhancements based on intraday data, Journal of Banking & Finance, Volume 138, 2022, 106426. ↩ ↩2 ↩3 See Wikipedia, Covariance matrix. ↩ This covariance matrix forecasting model is a direct extension of its volatility forecasting model counterpart. ↩ ↩2 In all generality, specific post-processing might anyway need to be implemented to ensure that covariance matrix forecasts are well-conditioned, c.f. Johansson et al.1 ↩ See Jose Menchero, Andrei Morozov, Improving Risk Forecasts Through Cross-Sectional Observations, The Journal of Portfolio Management Spring 2015, 41 (3) 84-96. ↩ The exponentially weighted moving average covariance matrix forecasting model is a special case of the diagonal vec multivariate GARCH model of Engle and Kroner23, and corresponds to an integrated GARCH model with no constant vector12. ↩ See Richard D.F. Harris, Fatih Yilmaz, Estimation of the conditional variance–covariance matrix of returns using the intraday range, International Journal of Forecasting, Volume 26, Issue 1, 2010, Pages 180-194. ↩ ↩2 ↩3 Other procedures are described in the RiskMetrics technical document14. ↩ See RiskMetrics. Technical Document, J.P.Morgan/Reuters, New York, 1996. Fourth Edition. ↩ ↩2 ↩3 These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer24. ↩ As implemented in Portfolio Optimizer. ↩ Direct methods use a proxy for the true covariance matrix to evaluate the predictor, while indirect methods use the covariance predictor on tasks of interest, such as portfolio construction or portfolio tracking1. ↩ (Adjusted) daily prices have have been retrieved using Tiingo. ↩ Using the outer product of asset returns - assuming a mean return of 0 - as covariance proxy, and using an expanding historical window of asset returns. ↩ The optimal decay factor $\lambda$ is computed at the end of every month using all the available asset returns history up to that point in time, as implemented in Portfolio Optimizer; thus, there is no look-ahead bias. ↩ ↩2 The maximum perturbation level on Figure 3 is limited to 1.4, because that level correponds to the MSE also reached by forecasting next month’s correlation matrix by a random correlation matrix; so, there is no practical interest in going beyond that level up to the theoretical limit of 2. ↩ ↩2 Especially for the variance of daily asset returns, as is the case here. ↩ See Engle RF, Kroner KF. Multivariate Simultaneous Generalized ARCH. Econometric Theory. 1995;11(1):122-150. ↩ See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Capital Market Assumptions: Combining Institutions’ Forecasts for Improved Accuracy2024-07-10T00:00:00-05:002024-07-10T00:00:00-05:00https://portfoliooptimizer.io/blog/capital-market-assumptions-combining-institutions-forecasts-for-improved-accuracy<p>Capital market assumptions<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> (CMAs) are forecasts of future risk/return characteristics for broad asset classes over the next 5 to 20 years
produced by <em>leading investment managers, consultants and advisors</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>These forecasts are <em>well-reasoned, analytically rigorous assumptions about uncertain future market movements</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> and are used
<em>almost universally among institutional investors</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, for example as inputs to their <a href="https://www.investopedia.com/terms/s/strategicassetallocation.asp">stategical asset allocation</a><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In this blog post, I will show that combining capital market assumptions might be preferable to using any institution-specific estimates
and I will describe a couple of practical data issues encountered when doing so.</p>
<h2 id="the-variability-of-capital-market-assumptions">The variability of capital market assumptions</h2>
<p>Each year since 2010, people at <a href="https://www.horizonactuarial.com/">Horizon Actuarial</a> <em>ask different investment firms to provide their capital market assumptions</em><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, and release an associated
survey.</p>
<p>Figure 1 - taken from the 2023 edition of that survey<sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> - clearly shows <em>significant differences in expected returns and standard deviations among investment advisors</em><sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-horizon-actuarial-large.png"><img src="/assets/images/blog/capital-markert-assumptions-horizon-actuarial.png" alt="Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial." /></a>
<figcaption>Figure 1. Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial.</figcaption>
</figure>
<p>Of course, variability should come as no surprise, because different institutions use different forecasting methodologies and/or have different views on the future<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup>…</p>
<p>Nevertheless, such a high variability is definitely an issue for an investor since it is impossible to identify a priori the institution that will produce the most accurate forecasts
of future returns, standard deviations and correlations.</p>
<p>The risk of relying on a “bad” institution for an investor is therefore very real…</p>
<h2 id="combining-capital-market-assumptions">Combining capital market assumptions</h2>
<h3 id="the-general-forecast-combination-problem">The general forecast combination problem</h3>
<p>The specific problem described in the previous section is an instance of a more general problem known as <em>the forecast combination problem</em>, introduced as follows by Timmermann<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>:</p>
<blockquote>
<p>Faced with multiple forecasts of the same variable, an issue that immediately arises is how best to exploit information in the individual forecasts. In particular, should a single dominant forecast be identified or should a combination of the underlying forecasts be used to produce a pooled summary measure?</p>
</blockquote>
<p>From the forecasting literature, a pragmatic solution to this problem is to average all individual forecasts, which leads to an equal-weighting forecast combination scheme.</p>
<p>While guaranteed to degrade the forecast accuracy v.s. using the “best” individual forecast<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">7</a></sup>, such a forecast combination scheme has repeatedly been found to
<em>outperform sophisticated adaptive combination methods in empirical applications</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, thus setting <em>a benchmark that has proved surprisingly difficult to beat</em><sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>As an interesting side note for readers of this blog, it turns out that the forecast combination problem is closely related to the mean-variance portfolio optimization problem<sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<h3 id="averaging-capital-market-assumptions">Averaging capital market assumptions</h3>
<p>From a theoretical perspective, an equal-weighted forecast combination scheme is well-suited to the specific context of capital market assumptions.</p>
<p>Indeed, given the <em>significant effort, resources and intellectual capital [put] into the production of [capital market assumptions]</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, it is reasonable to assume that the
<em>relative size of the variance of the forecast errors associated with [the] different forecasting methods</em><sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> are all similar, which in theory favors an equal-weighting forecast combination scheme<sup id="fnref:6:5" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>From a practical perspective, an equal-weighted forecast combination scheme is easy to compute and does not introduce any parameter estimation errors, which
<em>are known to be a serious problem for many combination techniques</em><sup id="fnref:6:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>As for the forecast accuracy of that scheme, it is possible to have an idea of it thanks to Figure 2 - taken from Sebastian<sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> -, which compares:</p>
<ul>
<li>The min-max range of 10-year capital market assumptions produced by 19 investment firms for misc. asset classes in 2013</li>
<li>The associated averaged 10-year capital market assumptions</li>
<li>The subsequent realized 10-year returns</li>
</ul>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-sebastian-large.png"><img src="/assets/images/blog/capital-markert-assumptions-sebastian.png" alt="2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian." /></a>
<figcaption>Figure 2. 2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian.</figcaption>
</figure>
<p>From Figure 2, it is visible that:</p>
<ul>
<li>
<p>Using averaged capital market assumptions v.s. firm-specific estimates prevented<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">8</a></sup> severe over/under-predictions for a couple of asset classes, like a ~3% over-prediction for Emerging Markets equities, which is great.</p>
</li>
<li>
<p><em>In the majority of cases, the average capital market assumption [is] an unreliable guide to the future return</em><sup id="fnref:2:5" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, which is not so great, but which ultimately depends on the accuracy and/or bias of the individual capital market assumptions and not on the forecast combination scheme itself<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">9</a></sup>.</p>
<p>That being said, a sample size of 1 observation per asset class is anyway maybe a little too small for reaching definite conclusions…</p>
</li>
</ul>
<h2 id="capital-market-assumptions-data-preparation">Capital market assumptions data preparation</h2>
<p>Before being able to average capital market assumptions originating from different institutions, it is first required to collect and normalize these forecasts.</p>
<p>Below are examples of typical data preparation steps for doing so.</p>
<h3 id="collecting-a-sufficient-number-of-forecasts">Collecting a sufficient number of forecasts</h3>
<p>Figure 3 - taken from an <a href="https://ecrresearch.com/">ECR Research</a> report<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">10</a></sup> - illustrates the variety of forecast horizons for which a representative sample of leading investment firms provide
their capital market assumptions.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-ecrresearch-large.png"><img src="/assets/images/blog/capital-markert-assumptions-ecrresearch.png" alt="Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research." /></a>
<figcaption>Figure 3. Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research.</figcaption>
</figure>
<p>From Figure 3, and in oder to combine as many distinct capital market assumptions as possible, it seems a good idea to focus on collecting forecasts for either a 5-year or a 10-year horizon, with a
preference for a 10-year horizon.</p>
<h3 id="reconciliating-asset-classes-across-institutions">Reconciliating asset classes across institutions</h3>
<p>Because <em>not all investment advisors use the same asset classes when developing their capital market assumptions</em><sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, reconciliating different asset classes across institutions is mandatory and
requires to <em>exercice judgment in classifying [institutions] capital market assumptions into a standard set of asset</em><sup id="fnref:2:6" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>As a side note, there are also sometimes inconsistencies in the name of asset classes.</p>
<p>For example, under the asset class label <em>Developed Equities</em>, different institutions might mean:</p>
<ul>
<li>An asset class represented by the <em>MSCI World Index</em>, which would be the standard convention</li>
<li>An asset class represented by the <em>MSCI World ex-US Index</em></li>
<li>An asset class represnted by the <em>MSCI EAFE Index</em></li>
</ul>
<h3 id="converting-expected-returns">Converting expected returns</h3>
<h4 id="converting-real-expected-returns-to-nominal-expected-returns">Converting real expected returns to nominal expected returns</h4>
<p>The vast majority of institutions provide expected returns in nominal terms, but a couple of them provide expected returns in <a href="https://www.investopedia.com/terms/r/realrateofreturn.asp">real terms</a>.</p>
<p>In order to compare apples to apples, it is necessary to convert these real returns into nominal returns by adding local expected inflation<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>, that is:</p>
<ul>
<li>U.S. expected inflation, for U.S. asset classes (<em>U.S. Equities</em>, <em>U.S. Bonds</em>…) as well as usually for commodities</li>
<li>Eurozone expected inflation<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">12</a></sup>, for euro-denominated asset classes (<em>E.M.U. Equities</em>, <em>EUR Bonds</em>…)</li>
<li>Emerging countries inflation<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">12</a></sup>, for <em>Emerging Equities</em></li>
<li>…</li>
</ul>
<h4 id="converting-expected-returns-currencies">Converting expected returns currencies</h4>
<p>Depending on the institution and on the asset class, expected returns can be provided in:</p>
<ul>
<li>USD, or another investable currency like EUR</li>
<li>USD EUR-hedged, or another investable currency pair hedged like EUR JPY-hedged</li>
<li>Local currency</li>
<li>Local currency investable currency-hedged, or investable currency local currency-hedged or whatever weird combination</li>
</ul>
<p>In order to again compare apples to apples, it is necessary to convert all the currencies appearing in the different expected returns into a unique investable currency.</p>
<h5 id="converting-expected-returns-in-an-investable-currency-to-expected-returns-in-another-investable-currency">Converting expected returns in an investable currency to expected returns in another investable currency</h5>
<p>The most usual scenario is the need to convert expected returns initially provided in an investable currency like USD into another investable currency like EUR.</p>
<p>This convertion - <em>ignoring the small geometric interaction</em><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup> - is done by adding the expected foreign exchange return to the expected returns<sup id="fnref:13:2" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>.</p>
<h5 id="converting-investable-currency-hedged-expected-returns-to-investable-currency-unhedged-expected-returns">Converting investable currency-hedged expected returns to investable currency-unhedged expected returns</h5>
<p>Expected returns for “foreign” fixed income asset classes are frequently provided as “foreign currency”-hedged<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">13</a></sup>.</p>
<p>The convertion from “foreign currency”-hedged expected returns to “foreign currency”-unhedged expected returns is done by <em>correcting for the expected nominal risk-free rate differential</em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>, that is,
by adding the difference between the foreign and the local cash returns.</p>
<h5 id="converting-expected-returns-in-local-currency-to-expected-returns-in-an-investable-currency">Converting expected returns in local currency to expected returns in an investable currency</h5>
<p>Many institutions provide expected returns in local currency to <em>avoid the issue of [determining] what an asset’s expected return will be from a particular investor’s currency perspective</em><sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">13</a></sup>.</p>
<p>Ultimately, though, an investor needs to make a currency decision, and expected returns in local currency must be converted into expected returns in an investable currency like USD or EUR.</p>
<p>For single-currency asset classes, like <em>U.S. Equities</em> or <em>Eurozone Government Bonds</em>, this is not an issue, c.f. the previous sub-sections.</p>
<p>For multi-currency asset classes, like <em>Emerging Market Equities</em> or <em>Global Government Bonds</em>, the situation is more complex because a proper conversion would at the very least require assumptions on:</p>
<ul>
<li>The expected currency weights in the asset class at the forecast horizon</li>
<li>The expected foreign exchange return at the forecast horizon, for all currencies</li>
</ul>
<p>To be noted that ignoring this complexity is probably not an option, as illustrated in Figure 4 - taken from the factsheet of the <em>MSCI Emerging Markets Index</em> -,
which shows a ~3% difference in annualized returns over the last 10 years between the “local currency” and the “USD” flavors of the <em>MSCI Emerging Markets Index</em>.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-msci-large.png"><img src="/assets/images/blog/capital-markert-assumptions-msci.png" alt="Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI." /></a>
<figcaption>Figure 4. Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI.</figcaption>
</figure>
<p>Between those two extremes, finding a pragmatic solution depending on the exact asset class is the best thing to do!</p>
<h5 id="other-convertions">Other convertions</h5>
<p>The need for other weird currency convertions might be encountered, at which point a balance must be struck between:</p>
<ul>
<li>Adding new forecasts, possibly unique, to the list of collected capital market assumptions</li>
<li>Adding complexity to the capital market assumptions collection process</li>
</ul>
<h4 id="converting-geometric-expected-returns-to-arithmetic-expected-returns">Converting geometric expected returns to arithmetic expected returns</h4>
<p>Nearly all institutions produce <em>estimates of geometric (compound) asset class returns</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>, while <em>standard mean-variance portfolio optimization<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">15</a></sup> requires
expected returns to be provided in arithmetic rather than in geometric terms</em><sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>.</p>
<p>In order to approximately convert geometric expected returns to arithmetic expected returns, it suffices to add half of the expected asset class variance<sup id="fnref:13:3" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>.</p>
<p>As a down-to-earth comment on why is that, I cannot resist sharing what people at <a href="https://www.verusinvestments.com/">Verus</a> wrote in the 2024 edition of their capital market assumptions:</p>
<blockquote>
<p>This is the industry standard approach, but requires a complex explanation only a heavy quant could love, so we have chosen not to provide further details in this document […]</p>
</blockquote>
<h3 id="converting-expected-standard-deviationscorrelations">Converting expected standard deviations/correlations</h3>
<p>Similar to expected returns, expected standard deviations and correlations should also be converted into an investor’s chosen currency.</p>
<p>Unfortunately, this conversion is far more complex to perform than that of expected returns, so that - unless specified otherwise by the originating institution -
it is probably best to leave the original expected standard deviations and correlations untouched.</p>
<h2 id="implementations">Implementations</h2>
<h3 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h3>
<p><strong>Portfolio Optimizer</strong> allows to retrieve minimum, maximum and averaged 10-year capital market assumptions (nominal arithmetic and geometric returns, standard deviations, correlations)
from leading financial institutions and for selected asset classes:</p>
<ul>
<li>In EUR, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/markets/capital-assumptions/eur</code></a></li>
<li>In USD, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/markets/capital-assumptions/usd</code></a></li>
</ul>
<p>A couple of remarks:</p>
<ul>
<li>
<p>The list of financial institutions from which capital market assumptions have been sourced is provided in output of these endpoints.</p>
</li>
<li>
<p>The supported asset classes, together with a representative benchmark, are the following:</p>
<table>
<thead>
<tr>
<th>Asset Class</th>
<th>Asset Class Label in <strong>Portfolio Optimizer</strong></th>
<th>Representative Benchmark</th>
</tr>
</thead>
<tbody>
<tr>
<td>U.S. Cash</td>
<td>cash-us</td>
<td>J.P. Morgan Cash Index USD</td>
</tr>
<tr>
<td>Eurozone Cash</td>
<td>cash-eurozone</td>
<td>J.P. Morgan Cash Index Euro Currency</td>
</tr>
<tr>
<td>U.S. Government Bonds</td>
<td>governmentBonds-us</td>
<td>Bloomberg U.S. Treasury Index</td>
</tr>
<tr>
<td>Eurozone Government Bonds</td>
<td>governmentBonds-eurozone</td>
<td>Bloomberg Euro Aggregate Treasury Index</td>
</tr>
<tr>
<td>U.S. Investment Grade Corporate Bonds</td>
<td>corporateBonds-us</td>
<td>ICE BofA US Corporate Index</td>
</tr>
<tr>
<td>Eurozone Investment Grade Corporate Bonds</td>
<td>corporateBonds-eurozone</td>
<td>ICE BofA Euro Corporate Index</td>
</tr>
<tr>
<td>U.S. Equities</td>
<td>equities-us</td>
<td>MSCI USA Index</td>
</tr>
<tr>
<td>Eurozone Equities</td>
<td>equities-eurozone</td>
<td>MSCI EMU Index</td>
</tr>
<tr>
<td>Developed Markets Equities</td>
<td>equities-developedMarkets</td>
<td>MSCI Developed Markets Index</td>
</tr>
<tr>
<td>Emerging Markets Equities</td>
<td>equities-emergingMarkets</td>
<td>MSCI Emerging Markets Index</td>
</tr>
<tr>
<td>Commodities</td>
<td>commodities-all</td>
<td>Bloomberg Commodity Index</td>
</tr>
<tr>
<td>Gold</td>
<td>gold-all</td>
<td>Gold Spot Price</td>
</tr>
</tbody>
</table>
</li>
<li>
<p>The expected correlation matrices provided in output of these endpoints are not guaranteed to be complete nor valid.</p>
<ul>
<li>In case a correlation matrix is complete but invalid, it is possible to compute its nearest valid correlation matrix thanks to the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/correlation/matrix/nearest</code></a></li>
<li>In case a correlation matrix is incomplete, <strong>Portfolio Optimizer</strong> provides no solution for now but stay tuned!</li>
</ul>
</li>
</ul>
<p>Last, if you would like a specific currency or a specific asset class to be integrated into <strong>Portfolio Optimizer</strong>, feel free to <a href="/contact">reach out</a>.</p>
<h3 id="implementations-elsewhere">Implementations elsewhere</h3>
<p>I am aware of other capital market assumptions “averaging” services as described in this blog post:</p>
<ul>
<li><strong><a href="https://www.horizonactuarial.com/survey-of-capital-market-assumptions">Horizon Actuarial</a></strong> yearly survey of capital market assumptions</li>
<li><strong><a href="https://www.peresec.com/insights/2024-long-term-capital-market-assumptions-summary/">Peresec</a></strong> interactive summary of long-term capital market assumptions</li>
<li><strong><a href="https://ecrresearch.com/">ECR Research</a></strong> strategic asset allocation quarterly reports</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>This concludes this first blog post on capital market assumptions.</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Or capital market expectations (CMEs). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4184885">Sebastian, Michael D., The Accuracy and Use of Capital Market Assumptions</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:2:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:2:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.researchaffiliates.com/content/dam/ra/documents/asset-allocation/aai-methodology-0823.pdf">Research Affiliates Capital Market Expectations Methodology, 05.31.2023</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://www.horizonactuarial.com/_files/ugd/f76a4b_1057ff4efa7244d6bb7b1a8fb88236e6.pdf">Horizon Actuarial Services LLC, Survey of Capital Market Assumptions, 2023 Edition</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Additionally, Horizon Actuarial<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> notes that different institutions publish their capital market assumptions at different dates. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S1574070605010049">Allan Timmermann, Chapter 4 Forecast Combinations, Editor(s): G. Elliott, C.W.J. Granger, A. Timmermann, Handbook of Economic Forecasting, Elsevier, Volume 1, 2006, Pages 135-196</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:6:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:6:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>Again, impossible to identify ex-ante. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Of course, Figure 2 also shows that using averaged capital market assumptions prevented to use the best firm-specific estimates, but as discussed in the previous section, it is impossible to identify these ex-ante, so that’s not a real drawback. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>One could think of using an optimal-weighting forecast combination scheme, with for example the objective of minimizing the out-of-sample forecast error, but due to the small sample size over which the optimal weights would need to be estimated, such a scheme would probably lead to poor performance v.s. an equal-weighting scheme, as is usual in the forecasting litterature, c.f. Timmermann<sup id="fnref:6:7" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://ecrresearch.com/sites/default/files/research_downloads/Expected%20Returns%20Model%20Methodology_EXR.pdf">ECR Research, Expected Returns Model</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.monash.edu/business/mcfs/our-research/all-projects/investment-strategy/capital-market-assumptions">Lai Hoang, Ummul Ruthbah, Long-Term Capital Market Assumptions Public Equities</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:13:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:13:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>With or without an appropriate weighting per country representative of the asset class weighting per country, which quickly becomes a mess to manage… <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://www.bnpparibas-am.com/en-us/portfolio-perspectives/long-term-asset-allocation-the-slow-return-to-normal/">BNP Paribas AM, Long-Term Asset Allocation - The slow return to normal, Portfolio perspectives, White paper</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.invesco.com/us/en/insights/long-term-capital-market-assumptions-outlook.html">Invesco, 2024 long-term capital market assumptions</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>The same remark also applies to other portfolio allocation or optimization techniques, like Monte Carlo simulations. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Capital market assumptions1 (CMAs) are forecasts of future risk/return characteristics for broad asset classes over the next 5 to 20 years produced by leading investment managers, consultants and advisors2. These forecasts are well-reasoned, analytically rigorous assumptions about uncertain future market movements2 and are used almost universally among institutional investors2, for example as inputs to their stategical asset allocation3. In this blog post, I will show that combining capital market assumptions might be preferable to using any institution-specific estimates and I will describe a couple of practical data issues encountered when doing so. The variability of capital market assumptions Each year since 2010, people at Horizon Actuarial ask different investment firms to provide their capital market assumptions4, and release an associated survey. Figure 1 - taken from the 2023 edition of that survey4 - clearly shows significant differences in expected returns and standard deviations among investment advisors4. Figure 1. Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial. Of course, variability should come as no surprise, because different institutions use different forecasting methodologies and/or have different views on the future5… Nevertheless, such a high variability is definitely an issue for an investor since it is impossible to identify a priori the institution that will produce the most accurate forecasts of future returns, standard deviations and correlations. The risk of relying on a “bad” institution for an investor is therefore very real… Combining capital market assumptions The general forecast combination problem The specific problem described in the previous section is an instance of a more general problem known as the forecast combination problem, introduced as follows by Timmermann6: Faced with multiple forecasts of the same variable, an issue that immediately arises is how best to exploit information in the individual forecasts. In particular, should a single dominant forecast be identified or should a combination of the underlying forecasts be used to produce a pooled summary measure? From the forecasting literature, a pragmatic solution to this problem is to average all individual forecasts, which leads to an equal-weighting forecast combination scheme. While guaranteed to degrade the forecast accuracy v.s. using the “best” individual forecast7, such a forecast combination scheme has repeatedly been found to outperform sophisticated adaptive combination methods in empirical applications6, thus setting a benchmark that has proved surprisingly difficult to beat6. As an interesting side note for readers of this blog, it turns out that the forecast combination problem is closely related to the mean-variance portfolio optimization problem6. Averaging capital market assumptions From a theoretical perspective, an equal-weighted forecast combination scheme is well-suited to the specific context of capital market assumptions. Indeed, given the significant effort, resources and intellectual capital [put] into the production of [capital market assumptions]2, it is reasonable to assume that the relative size of the variance of the forecast errors associated with [the] different forecasting methods6 are all similar, which in theory favors an equal-weighting forecast combination scheme6. From a practical perspective, an equal-weighted forecast combination scheme is easy to compute and does not introduce any parameter estimation errors, which are known to be a serious problem for many combination techniques6. As for the forecast accuracy of that scheme, it is possible to have an idea of it thanks to Figure 2 - taken from Sebastian2 -, which compares: The min-max range of 10-year capital market assumptions produced by 19 investment firms for misc. asset classes in 2013 The associated averaged 10-year capital market assumptions The subsequent realized 10-year returns Figure 2. 2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian. From Figure 2, it is visible that: Using averaged capital market assumptions v.s. firm-specific estimates prevented8 severe over/under-predictions for a couple of asset classes, like a ~3% over-prediction for Emerging Markets equities, which is great. In the majority of cases, the average capital market assumption [is] an unreliable guide to the future return2, which is not so great, but which ultimately depends on the accuracy and/or bias of the individual capital market assumptions and not on the forecast combination scheme itself9. That being said, a sample size of 1 observation per asset class is anyway maybe a little too small for reaching definite conclusions… Capital market assumptions data preparation Before being able to average capital market assumptions originating from different institutions, it is first required to collect and normalize these forecasts. Below are examples of typical data preparation steps for doing so. Collecting a sufficient number of forecasts Figure 3 - taken from an ECR Research report10 - illustrates the variety of forecast horizons for which a representative sample of leading investment firms provide their capital market assumptions. Figure 3. Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research. From Figure 3, and in oder to combine as many distinct capital market assumptions as possible, it seems a good idea to focus on collecting forecasts for either a 5-year or a 10-year horizon, with a preference for a 10-year horizon. Reconciliating asset classes across institutions Because not all investment advisors use the same asset classes when developing their capital market assumptions4, reconciliating different asset classes across institutions is mandatory and requires to exercice judgment in classifying [institutions] capital market assumptions into a standard set of asset2. As a side note, there are also sometimes inconsistencies in the name of asset classes. For example, under the asset class label Developed Equities, different institutions might mean: An asset class represented by the MSCI World Index, which would be the standard convention An asset class represented by the MSCI World ex-US Index An asset class represnted by the MSCI EAFE Index Converting expected returns Converting real expected returns to nominal expected returns The vast majority of institutions provide expected returns in nominal terms, but a couple of them provide expected returns in real terms. In order to compare apples to apples, it is necessary to convert these real returns into nominal returns by adding local expected inflation11, that is: U.S. expected inflation, for U.S. asset classes (U.S. Equities, U.S. Bonds…) as well as usually for commodities Eurozone expected inflation12, for euro-denominated asset classes (E.M.U. Equities, EUR Bonds…) Emerging countries inflation12, for Emerging Equities … Converting expected returns currencies Depending on the institution and on the asset class, expected returns can be provided in: USD, or another investable currency like EUR USD EUR-hedged, or another investable currency pair hedged like EUR JPY-hedged Local currency Local currency investable currency-hedged, or investable currency local currency-hedged or whatever weird combination In order to again compare apples to apples, it is necessary to convert all the currencies appearing in the different expected returns into a unique investable currency. Converting expected returns in an investable currency to expected returns in another investable currency The most usual scenario is the need to convert expected returns initially provided in an investable currency like USD into another investable currency like EUR. This convertion - ignoring the small geometric interaction11 - is done by adding the expected foreign exchange return to the expected returns11. Converting investable currency-hedged expected returns to investable currency-unhedged expected returns Expected returns for “foreign” fixed income asset classes are frequently provided as “foreign currency”-hedged13. The convertion from “foreign currency”-hedged expected returns to “foreign currency”-unhedged expected returns is done by correcting for the expected nominal risk-free rate differential14, that is, by adding the difference between the foreign and the local cash returns. Converting expected returns in local currency to expected returns in an investable currency Many institutions provide expected returns in local currency to avoid the issue of [determining] what an asset’s expected return will be from a particular investor’s currency perspective13. Ultimately, though, an investor needs to make a currency decision, and expected returns in local currency must be converted into expected returns in an investable currency like USD or EUR. For single-currency asset classes, like U.S. Equities or Eurozone Government Bonds, this is not an issue, c.f. the previous sub-sections. For multi-currency asset classes, like Emerging Market Equities or Global Government Bonds, the situation is more complex because a proper conversion would at the very least require assumptions on: The expected currency weights in the asset class at the forecast horizon The expected foreign exchange return at the forecast horizon, for all currencies To be noted that ignoring this complexity is probably not an option, as illustrated in Figure 4 - taken from the factsheet of the MSCI Emerging Markets Index -, which shows a ~3% difference in annualized returns over the last 10 years between the “local currency” and the “USD” flavors of the MSCI Emerging Markets Index. Figure 4. Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI. Between those two extremes, finding a pragmatic solution depending on the exact asset class is the best thing to do! Other convertions The need for other weird currency convertions might be encountered, at which point a balance must be struck between: Adding new forecasts, possibly unique, to the list of collected capital market assumptions Adding complexity to the capital market assumptions collection process Converting geometric expected returns to arithmetic expected returns Nearly all institutions produce estimates of geometric (compound) asset class returns14, while standard mean-variance portfolio optimization15 requires expected returns to be provided in arithmetic rather than in geometric terms14. In order to approximately convert geometric expected returns to arithmetic expected returns, it suffices to add half of the expected asset class variance11. As a down-to-earth comment on why is that, I cannot resist sharing what people at Verus wrote in the 2024 edition of their capital market assumptions: This is the industry standard approach, but requires a complex explanation only a heavy quant could love, so we have chosen not to provide further details in this document […] Converting expected standard deviations/correlations Similar to expected returns, expected standard deviations and correlations should also be converted into an investor’s chosen currency. Unfortunately, this conversion is far more complex to perform than that of expected returns, so that - unless specified otherwise by the originating institution - it is probably best to leave the original expected standard deviations and correlations untouched. Implementations Implementation in Portfolio Optimizer Portfolio Optimizer allows to retrieve minimum, maximum and averaged 10-year capital market assumptions (nominal arithmetic and geometric returns, standard deviations, correlations) from leading financial institutions and for selected asset classes: In EUR, through the endpoint /markets/capital-assumptions/eur In USD, through the endpoint /markets/capital-assumptions/usd A couple of remarks: The list of financial institutions from which capital market assumptions have been sourced is provided in output of these endpoints. The supported asset classes, together with a representative benchmark, are the following: Asset Class Asset Class Label in Portfolio Optimizer Representative Benchmark U.S. Cash cash-us J.P. Morgan Cash Index USD Eurozone Cash cash-eurozone J.P. Morgan Cash Index Euro Currency U.S. Government Bonds governmentBonds-us Bloomberg U.S. Treasury Index Eurozone Government Bonds governmentBonds-eurozone Bloomberg Euro Aggregate Treasury Index U.S. Investment Grade Corporate Bonds corporateBonds-us ICE BofA US Corporate Index Eurozone Investment Grade Corporate Bonds corporateBonds-eurozone ICE BofA Euro Corporate Index U.S. Equities equities-us MSCI USA Index Eurozone Equities equities-eurozone MSCI EMU Index Developed Markets Equities equities-developedMarkets MSCI Developed Markets Index Emerging Markets Equities equities-emergingMarkets MSCI Emerging Markets Index Commodities commodities-all Bloomberg Commodity Index Gold gold-all Gold Spot Price The expected correlation matrices provided in output of these endpoints are not guaranteed to be complete nor valid. In case a correlation matrix is complete but invalid, it is possible to compute its nearest valid correlation matrix thanks to the endpoint /assets/correlation/matrix/nearest In case a correlation matrix is incomplete, Portfolio Optimizer provides no solution for now but stay tuned! Last, if you would like a specific currency or a specific asset class to be integrated into Portfolio Optimizer, feel free to reach out. Implementations elsewhere I am aware of other capital market assumptions “averaging” services as described in this blog post: Horizon Actuarial yearly survey of capital market assumptions Peresec interactive summary of long-term capital market assumptions ECR Research strategic asset allocation quarterly reports Conclusion This concludes this first blog post on capital market assumptions. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – Or capital market expectations (CMEs). ↩ See Sebastian, Michael D., The Accuracy and Use of Capital Market Assumptions. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 See Research Affiliates Capital Market Expectations Methodology, 05.31.2023. ↩ See Horizon Actuarial Services LLC, Survey of Capital Market Assumptions, 2023 Edition. ↩ ↩2 ↩3 ↩4 Additionally, Horizon Actuarial4 notes that different institutions publish their capital market assumptions at different dates. ↩ See Allan Timmermann, Chapter 4 Forecast Combinations, Editor(s): G. Elliott, C.W.J. Granger, A. Timmermann, Handbook of Economic Forecasting, Elsevier, Volume 1, 2006, Pages 135-196. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 Again, impossible to identify ex-ante. ↩ Of course, Figure 2 also shows that using averaged capital market assumptions prevented to use the best firm-specific estimates, but as discussed in the previous section, it is impossible to identify these ex-ante, so that’s not a real drawback. ↩ One could think of using an optimal-weighting forecast combination scheme, with for example the objective of minimizing the out-of-sample forecast error, but due to the small sample size over which the optimal weights would need to be estimated, such a scheme would probably lead to poor performance v.s. an equal-weighting scheme, as is usual in the forecasting litterature, c.f. Timmermann6. ↩ See ECR Research, Expected Returns Model. ↩ See Lai Hoang, Ummul Ruthbah, Long-Term Capital Market Assumptions Public Equities. ↩ ↩2 ↩3 ↩4 With or without an appropriate weighting per country representative of the asset class weighting per country, which quickly becomes a mess to manage… ↩ ↩2 See BNP Paribas AM, Long-Term Asset Allocation - The slow return to normal, Portfolio perspectives, White paper. ↩ ↩2 See Invesco, 2024 long-term capital market assumptions. ↩ ↩2 ↩3 The same remark also applies to other portfolio allocation or optimization techniques, like Monte Carlo simulations. ↩Volatility Forecasting: HAR Model2024-06-25T00:00:00-05:002024-06-25T00:00:00-05:00https://portfoliooptimizer.io/blog/volatility-forecasting-har-model<p>Among the different members of the family of volatility forecasting models by weighted moving average<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> like
<a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the simple and the exponentially weighted moving average models</a> or
<a href="/blog/volatility-forecasting-garch11-model/">the GARCH(1,1) model</a>, the <em>Heterogeneous AutoRegressive (HAR) model</em> introduced by Corsi<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>
<em>has become the workhorse of the volatility forecasting literature</em><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> <em>on account of its simplicity and generally good forecasting performance</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>.</p>
<p>In this blog post, strongly inspired by the paper <em>A Practical Guide to harnessing the HAR volatility model</em> from Clements and Preve<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, I will describe the HAR volatility forecasting model
together with some important implementation details and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs.</p>
<h2 id="mathematical-preliminaries-reminders">Mathematical preliminaries (reminders)</h2>
<p>This section contains reminders from a <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a>.</p>
<h3 id="volatility-modelling-and-volatility-proxies">Volatility modelling and volatility proxies</h3>
<p>Let $r_t$ be the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>) return of an asset over a time period $t$ (a day, a week, a month..),
over which its (conditional) mean return is supposed to be null.</p>
<p>Then:</p>
<ul>
<li>
<p>The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $</p>
<p>From this definition, the squared return $r_t^2$ of an asset is a (noisy<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>) <em>variance estimator</em> - or <em>variance proxy</em><sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> - for that asset variance over the considered time period.</p>
<p>Another example of an asset variance proxy is <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">the Parkinson range</a> of an asset.</p>
<p>Yet another example of an asset variance proxy, this time over a specific time period $t$ of one day, is the <em>daily realized variance</em> $RV_t$ of an asset, which is defined as the sum of the asset squared intraday returns sampled at a high frequency (1 minutes, 5 minutes, 15 minutes…).</p>
<p>The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$.</p>
</li>
<li>
<p>The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $</p>
<p>The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$.</p>
</li>
</ul>
<h3 id="weighted-moving-average-volatility-forecasting-model">Weighted moving average volatility forecasting model</h3>
<p>Boudoukh et al.<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> show that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s
variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with</p>
\[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\]
<p>, where:</p>
<ul>
<li>$1 \leq k \leq T$ is the size of the moving average, possibly time-dependent</li>
<li>$w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well</li>
</ul>
<h2 id="the-har-volatility-forecasting-model">The HAR volatility forecasting model</h2>
<h3 id="the-original-har-model">The original HAR model</h3>
<p>Due to the limitations of the GARCH model in reproducing <em>the main empirical features of financial returns (long memory, fat tails, and self-similarity)</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, Corsi<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>
proposes to use <em>an additive cascade model of different volatility components each of which is generated by the actions of different types of market participants</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>.</p>
<p>As detailled by Corsi<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>:</p>
<blockquote>
<p>The main idea is that agents with different time horizons perceive, react to, and cause different types of volatility components. Simplifying a bit, we can identify three primary volatility components: the short-term traders with daily or higher trading frequency, the medium-term investors who typically rebalance their positions weekly, and the long-term agents with a characteristic time of one or more months.</p>
</blockquote>
<p>Under this volatility forecasting model called the <em>Heterogeneous AutoRegressive model (HAR)</em><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">5</a></sup>, an asset next day’s daily realized variance $RV_{T+1}$ is modeled as
an <a href="https://en.wikipedia.org/wiki/Autoregressive_model">AR(22) process</a> subject to <em>economically meaningful restrictions</em><sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup> on its parameters, which results in the formula<sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></p>
\[\hat{RV}_{T+1} = \beta + \beta_d RV_{T} + \beta_w RV_{T}^w + \beta_m RV_{T}^m\]
<p>, where:</p>
<ul>
<li>$\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$</li>
<li>$RV_T$ is the asset daily realized variance at time $T$</li>
<li>$RV_T^w = \frac{1}{5} \sum_{i=1}^5 RV_{T-i+1}$ is the asset weekly realized variance at time $T$</li>
<li>$RV_T^m = \frac{1}{22} \sum_{i=1}^{22} RV_{T-i+1}$ is the asset monthly realized variance at time $T$</li>
<li>$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined</li>
</ul>
<p>In terms of practical performances, and <em>in spite of its simplicity […], the [HAR] model is able to reproduce the same volatility persistence observed in the empirical data as well as many
of the other main stylized facts of financial data</em><sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, which makes it a very accurate volatility forecasting model.</p>
<h3 id="realized-variance-vs-generic-variance-proxy">Realized variance v.s. generic variance proxy</h3>
<p>The original HAR model described in the previous subsection relies on a very specific asset variance proxy - the realized variance of an asset - over a very specific time period - a day - for its definition.</p>
<p>Some papers (Lyocsa et al.<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">6</a></sup>, Lyocsa et al.<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">7</a></sup>, Clements et al.<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>, …) propose to replace the (high-frequency) daily realized variance by a (low-frequency) daily range-based variance estimator<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>
like:</p>
<ul>
<li>The square of the Parkinson volatility estimator</li>
<li>The square of the Garman-Klass volatility estimator</li>
<li>The square of the Rogers-Satchell volatility estimator</li>
</ul>
<p>Going one step further, it is possible to replace the daily realized variance by any generic daily variance estimator.</p>
<p>This leads to the generic HAR volatility forecasting model, under which an asset next days’s conditional variance $\sigma_{T+1}^2$ is modeled as
<em>a linear function of [its previous day’s] daily, weekly and monthly [conditional variance] components</em><sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, with the following daily variance forecasting formula</p>
\[\hat{\sigma}_{T+1}^2 = \beta + \beta_d \tilde{\sigma}^2_{T} + \beta_w \tilde{\sigma}^{2,w}_{T} + \beta_m \tilde{\sigma}^{2,m}_{T}\]
<p>, where:</p>
<ul>
<li>$\hat{\sigma}_{T+1}^2$ is the forecast at time $T$ of the asset next day’s conditional variance $\sigma_{T+1}^2$</li>
<li>$\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$</li>
<li>$\tilde{\sigma}^{2,w}_{T} = \frac{1}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$</li>
<li>$\tilde{\sigma}^{2,m}_{T} = \frac{1}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$</li>
<li>$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined</li>
</ul>
<p>Going another step further, it is also possible to replace the baseline daily time period by any desired time period (weekly, biweekly, monthly, quarterly…), but
given the theoretical foundations of the HAR model, this also requires to replace the weekly and the monthly variance estimators $\tilde{\sigma}^{2,w}_{T}$ and $\tilde{\sigma}^{2,m}_{T}$
by appropriate variance estimators.</p>
<h3 id="relationship-with-the-generic-weighted-moving-average-model">Relationship with the generic weighted moving average model</h3>
<p>From its definition, it is easy to see that the HAR volatility forecasting model is a specific kind of weighted moving average volatility forecasting model, with:</p>
<ul>
<li>$w_0 = \beta$</li>
<li>$w_1 = \beta_d + \frac{1}{5} \beta_w + \frac{1}{22} \beta_m$</li>
<li>$w_i = \frac{1}{5} \beta_w + \frac{1}{22} \beta_m, i = 2..5$</li>
<li>$w_i = \frac{1}{22} \beta_m, i = 6..22$</li>
<li>$w_i = 0$, $i \geq 23$, discarding all the past variance proxies beyond the $22$-th from the model</li>
</ul>
<h3 id="volatility-forecasting-formulas">Volatility forecasting formulas</h3>
<p>Under an HAR volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes:</p>
<ul>
<li>
<p>To estimate an asset next day’s volatility:</p>
\[\hat{\sigma}_{T+1} = \sqrt{ \beta + \beta_d \tilde{\sigma}^2_{T} + \frac{\beta_w}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i} + \frac{\beta_m}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i} }\]
</li>
<li>
<p>To estimate an asset next $h$-day’s ahead volatility<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">9</a></sup>, $h \geq 2$:</p>
\[\hat{\sigma}_{T+h} = \sqrt{ \beta + \beta_d \hat{\sigma}_{T+h-1}^2 + \frac{\beta_w}{5} \left( \sum_{i=1}^{5-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) + \frac{\beta_m}{22} \left( \sum_{i=1}^{22-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) }\]
</li>
<li>
<p>To estimate an asset aggregated volatility<sup id="fnref:16:1" role="doc-noteref"><a href="#fn:16" class="footnote">9</a></sup> over the next $h$ days:</p>
\[\hat{\sigma}_{T+1:T+h} = \sqrt{ \sum_{i=1}^{h} \hat{\sigma}^2_{T+i} }\]
</li>
</ul>
<blockquote>
<p><strong><em>Note:</em></strong><br />
Clements and Preve<sup id="fnref:6:5" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> extensively discuss whether to use an indirect or a direct multi-step ahead forecast scheme for estimating an asset next $h$-day’s ahead volatility under an HAR volatility forecasting model:</p>
<ul>
<li>In an indirect scheme<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">10</a></sup>, which is the scheme described above, the asset next $h$-day’s ahead volatility is estimated indirectly, by a recursive application of the HAR model formula for the asset next days’s conditional variance.</li>
<li>In a direct scheme, the asset next $h$-day’s ahead volatility is estimated directly, by replacing the asset next days’s conditional variance on the left-hand side of the HAR model formula with the asset next $h$-day’s ahead conditional variance.</li>
</ul>
<p>Clements and Preve<sup id="fnref:6:6" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> argue in particular that <em>“direct forecasts are easy to compute and more robust to model misspecification compared to indirect forecasts”</em><sup id="fnref:6:7" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, but at the same time, they show that this robustness does not always translate into more accurate forecasts.</p>
</blockquote>
<h2 id="harnessing-the-har-volatility-forecasting-model">Harnessing the HAR volatility forecasting model</h2>
<h3 id="har-model-parameters-estimation">HAR model parameters estimation</h3>
<h4 id="ordinary-least-squares-estimators">Ordinary least squares estimators</h4>
<p>Corsi<sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup> writes that it is possible to <em>easily estimate [the HAR model] parameters by applying simple linear regression</em><sup id="fnref:3:8" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, in which case
the <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS) estimator</a> of the parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ at time $T \geq 23$ is the solution of the minimization problem<sup id="fnref:6:8" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></p>
\[\argmin_{ \left( \beta, \beta_d, \beta_w, \beta_m \right) \in \mathbb{R}^{4}} \sum_{t=23}^T \left( \tilde{\sigma}_{t}^2 - \beta - \beta_d \tilde{\sigma}^2_{t-1} - \beta_w \tilde{\sigma}^{2,w}_{t-1} - \beta_m \tilde{\sigma}^{2,m}_{t-1} \right)^2\]
<h4 id="other-least-squares-estimators">Other least squares estimators</h4>
<p>Nevertheless, Clements and Preve<sup id="fnref:6:9" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> warn <em>that given the stylized facts of [volatility estimators] (such as spikes/outliers, conditional heteroskedasticity, and non-Gaussianity)
and well-known properties of OLS, this […] should be far from ideal</em><sup id="fnref:6:10" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<p>Indeed, as detailled in Patton and Sheppard<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>:</p>
<blockquote>
<p>Because the dependent variable in all of our regressions is a volatility measure, estimation by OLS has the unfortunate feature that the resulting estimates focus primarily on fitting periods of high variance and place little weight on more tranquil periods. This is an important drawback in our applications, as the level of variance changes substantially across our sample period and the level of the variance and the volatility in the error are known to have a positive relationship.</p>
</blockquote>
<p>Instead of OLS, Clements and Preve<sup id="fnref:6:11" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> and Clements et al.<sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> suggest to use other least squares estimators:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Weighted_least_squares">Weighted least squares estimators (WLS)</a>, with for example the weighting schemes described in Clements and Preve<sup id="fnref:6:12" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></li>
<li><a href="https://en.wikipedia.org/wiki/Robust_regression">Robust least squares estimators (RLS)</a>, with for example <a href="https://en.wikipedia.org/wiki/Redescending_M-estimator#Examples">Tukey’s biweight loss function</a></li>
<li><a href="https://en.wikipedia.org/wiki/Regularized_least_squares">Regularized least squares estimators (RRLS)</a> (ridge regression, LASSO regression, elastic net regression…), with the cross-validation procedure for the associated hyperparameters described in Clements et al.<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>.</li>
</ul>
<h4 id="expanding-vs-rolling-window-estimation-procedure">Expanding v.s. rolling window estimation procedure</h4>
<p>Clements and Preve<sup id="fnref:6:13" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> empirically demonstrate that the HAR model parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are time-varying, as illustrated in Figure 1.</p>
<figure>
<a href="/assets/images/blog/volatility-forecasting-har-beta-variability-clements-preve.png"><img src="/assets/images/blog/volatility-forecasting-har-beta-variability-clements-preve.png" alt="Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve." /></a>
<figcaption>Figure 1. Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve.</figcaption>
</figure>
<p>To deal with this non-stationarity, or more generally to <em>deal with parameter drift that is difficult to model explicitly</em><sup id="fnref:6:14" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, the standard<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> approach in the litterature is to
use a rolling window procedure<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup> for the least squares estimation of the HAR model parameters.</p>
<h3 id="insanity-filter">Insanity filter</h3>
<p>The HAR volatility forecasting model <em>may on rare occasions generate implausibly large or small forecasts</em><sup id="fnref:6:15" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, because <em>no restrictions on the parameters [$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$]
are imposed</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>In particular, it has been noted<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">14</a></sup> that <em>forecasts are occasionally negative</em><sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>In order to correct this behaviour, Clements and Preve<sup id="fnref:6:16" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> propose to implement an <em>insanity filter</em><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">15</a></sup> ensuring that <em>any forecast greater than the maximum, or less than the minimum,
of [an asset next days’s conditional variance] observed in the estimation period is replaced by the sample average over that period</em><sup id="fnref:6:17" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<p>Two important remarks on such a filter:</p>
<ul>
<li>It seems that <em>the particular choice of insanity filter is not important; what matters is to eliminate unrealistic forecasts</em><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">16</a></sup>.</li>
<li>The presence of an insanity filter is all the more important when an indirect multi-step ahead forecast scheme is used for estimating an asset next $h$-day’s ahead volatility, because <em>an unreasonable [forecast] is most likely to generate another unreasonable forecast</em><sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">16</a></sup>.</li>
</ul>
<h3 id="variance-proxies-transformations">Variance proxies transformations</h3>
<p>Clements and Preve<sup id="fnref:6:18" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> study three different <a href="https://encyclopediaofmath.org/wiki/Box-Cox_transformation">Box-Cox transformations</a> of an asset daily realized variance that can be used
in the HAR volatility forecasting model instead of that asset “raw” daily realized variance:</p>
<ul>
<li>The logarithmic transformation (log)</li>
<li>The quartic root transformation (qr)</li>
<li>The square root transformation (sqr)</li>
</ul>
<p>It is visible on Figure 2 that these transformations <em>appear useful for reducing skewness, and hence the possible effect of outliers and potential heteroskedasticity in the realized variance series</em><sup id="fnref:6:19" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<figure>
<a href="/assets/images/blog/volatility-forecasting-har-transformations-clements-preve.png"><img src="/assets/images/blog/volatility-forecasting-har-transformations-clements-preve.png" alt="S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve." /></a>
<figcaption>Figure 2. S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve.</figcaption>
</figure>
<p>Of particular interest is the logarithmic transformation<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">17</a></sup>, which in addition to good practical performances and closed form bias-corrected expressions for variance forecasts<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup>,
also guarantees that the generated forecasts are always positive<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">19</a></sup>.</p>
<h3 id="non-overlapping-daily-weekly-and-monthly-variance-proxies">Non-overlapping daily, weekly and monthly variance proxies</h3>
<p>Corsi et al.<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup> propose a slightly different parametrization of the HAR model compared to Corsi<sup id="fnref:3:9" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, where:</p>
<ul>
<li>$\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$</li>
<li>$\tilde{\sigma}^{2,w}_{T} = \frac{1}{4} \sum_{i=2}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$, non-overlapping with the asset daily variance estimator $\tilde{\sigma}^2_{T}$</li>
<li>$\tilde{\sigma}^{2,m}_{T} = \frac{1}{17} \sum_{i=6}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$, non-overlapping with either the asset daily variance estimator $\tilde{\sigma}^2_{T}$ nor with the asset weekly variance estimator $\tilde{\sigma}^{2,w}_{T}$</li>
</ul>
<p>Such a re-parametrization <em>does not imply any loss of information compared to the original [HAR model], since it relies only on a different rearrangement of the terms</em><sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup>, but allows an easiest
interpretation of the HAR model parameters, c.f. Patton and Sheppard<sup id="fnref:14:3" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>Incidentally, this non-overlapping formulation of the HAR volatility forecasting model has been found to generate better forecasts by practitioners<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">21</a></sup>.</p>
<h3 id="other-lag-indexes-for-the-variance-proxies">Other lag indexes for the variance proxies</h3>
<p>The standard lag indexes for the variance proxies in the HAR volatility forecasting model are 1, 5 and 22, each corresponding to a different volatility component in Corsi’s underlying additive cascade model.</p>
<p>As mentioned in Corsi<sup id="fnref:3:10" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, <em>more components could easily be added to the additive cascade of partial volatilities</em><sup id="fnref:3:11" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, which is done for example in Lyocsa et al.<sup id="fnref:25:1" role="doc-noteref"><a href="#fn:25" class="footnote">7</a></sup>.</p>
<p>In that case, <em>denoting $l$ and $h$, respectively, the lowest and highest frequency in the cascade</em><sup id="fnref:3:12" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, an asset next days’s conditional variance is modeled as an AR($\frac{l}{h}$) process
<em>reparameterized in a parsimonious way by imposing economically meaningful restrictions</em><sup id="fnref:3:13" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the HAR volatility forecasting model - augmented with the insanity filter described in Clements and Preve<sup id="fnref:6:20" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> - through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/volatility/forecast/har</code></a>.</p>
<p>This endpoint supports the 4 variance proxies below:</p>
<ul>
<li>Squared close-to-close returns</li>
<li>Demeaned squared close-to-close returns</li>
<li>The Parkinson range</li>
<li>The jump-adjusted Parkinson range</li>
</ul>
<p>This endpoint also supports:</p>
<ul>
<li>
<p>Transforming the input variance proxies into log variance proxies before estimating the HAR model parameters; the associated variance forecasts are then unbiased thanks to the formulas established in Buccheri and Corsi<sup id="fnref:19:1" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup>.</p>
</li>
<li>Estimating the HAR model parameters through 3 different least squares procedures:
<ul>
<li>Ordinary least squares</li>
<li>Weighted least squares, using the inverse of the variance proxies as weights</li>
<li>Robust least squares, using Tukey’s biweight loss function</li>
</ul>
</li>
<li>Using up to 5 lag indexes for the variance proxies</li>
</ul>
<h2 id="example-of-usage---volatility-forecasting-at-monthly-level-for-various-etfs">Example of usage - Volatility forecasting at monthly level for various ETFs</h2>
<p>As an example of usage, I propose to enrich the results of <a href="/blog/volatility-forecasting-garch11-model/">the previous blog post</a>, in which
monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">22</a></sup> regressions - to the next month’s close-to-close observed volatility
for 10 ETFs representative<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">23</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<h3 id="vanilla-har-volatility-forecasting-model">Vanilla HAR volatility forecasting model</h3>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">24</a></sup> are the following<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">25</a></sup>, when using the vanilla HAR volatility forecasting model:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td>Random walk</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.66</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.68</td>
<td>46%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Squared close-to-close returns</td>
<td>4.7%</td>
<td>0.73</td>
<td>45%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Squared close-to-close returns</td>
<td>-1.3%</td>
<td>0.98</td>
<td>43%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-0.7%</strong></td>
<td><strong>0.95</strong></td>
<td><strong>46%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Parkinson range</td>
<td>5.6%</td>
<td>0.94</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Parkinson range</td>
<td>5.1%</td>
<td>1.00</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Parkinson range</td>
<td>4.3%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Parkinson range</td>
<td>2.7%</td>
<td>1.18</td>
<td>47%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>0.1%</strong></td>
<td><strong>1.25</strong></td>
<td><strong>44%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.9%</td>
<td>0.70</td>
<td>45%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Jump-adjusted Parkinson range</td>
<td>5.1%</td>
<td>0.71</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.0%</td>
<td>0.76</td>
<td>45%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-1.0%</td>
<td>1.00</td>
<td>45%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.4%</strong></td>
<td><strong>0.99</strong></td>
<td><strong>47%</strong></td>
</tr>
</tbody>
</table>
<h3 id="alternative-har-volatility-forecasting-models">Alternative HAR volatility forecasting models</h3>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">24</a></sup> are the following<sup id="fnref:28:1" role="doc-noteref"><a href="#fn:28" class="footnote">25</a></sup>, when using different variations of the vanilla HAR volatility forecasting model:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-0.7%</strong></td>
<td><strong>0.95</strong></td>
<td><strong>46%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Squared close-to-close returns</td>
<td>3.1%</td>
<td>0.71</td>
<td>34%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Squared close-to-close returns</td>
<td>-12.3%</td>
<td>2.50</td>
<td>26%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Squared close-to-close returns</td>
<td>0.5%</td>
<td>0.62</td>
<td>40%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Squared close-to-close returns</td>
<td>10%</td>
<td>0.26</td>
<td>13%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Squared close-to-close returns</td>
<td>0.5%</td>
<td>0.53</td>
<td>40%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>0.1%</strong></td>
<td><strong>1.25</strong></td>
<td><strong>44%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Parkinson range</td>
<td>0.1%</td>
<td>1.19</td>
<td>44%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Parkinson range</td>
<td>-4.2%</td>
<td>2.20</td>
<td>40%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Parkinson range</td>
<td>1.9%</td>
<td>1.22</td>
<td>50%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Parkinson range</td>
<td>-0.6%</td>
<td>1.47</td>
<td>47%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Parkinson range</td>
<td>2.2%</td>
<td>1.22</td>
<td>50%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.4%</strong></td>
<td><strong>0.99</strong></td>
<td><strong>47%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-4.2%</td>
<td>0.92</td>
<td>46%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-6.6%</td>
<td>1.76</td>
<td>41%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Jump-adjusted Parkinson range</td>
<td>0.9%</td>
<td>0.92</td>
<td>51%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-0.8%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>1.2%</td>
<td>0.92</td>
<td>51%</td>
</tr>
</tbody>
</table>
<h3 id="comments">Comments</h3>
<p>From the results of the two previous subsections, it is possible to make the following comments:</p>
<ul>
<li>When using squared returns as a variance proxy, the vanilla HAR model is the best volatility forecasting model among all the alternative HAR models as well as among all the models already studied in this series.</li>
<li>When using the Parkinson range or the jump-adjusted Parkinson range as variance proxies, the log HAR model exhibits the highest r-squared among all the models already studied in this series.</li>
<li>When using the jump-adjusted Parkinson range as a variance proxy, the log HAR model is the best volatility forecasting model in this series (relatively low bias, highest r-squared).</li>
<li>When using alternative least square estimators, the forecasts quality generally degrades</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The previous section empirically demonstrated that the HAR volatility forecasting model, despite its relationship with realized variance, is still accurate when used with
range-based variance estimators at a long forecasting horizon, in line with Lyocsa et al.<sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">6</a></sup></p>
<p>In that context, the (log) HAR model is also the most accurate volatility forecasting model studied so far in this series on volatility forecasting
by weighted moving average models!</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijderiv/4/3/63">Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article-abstract/7/2/174/856522">Fulvio Corsi, A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics, Volume 7, Issue 2, Spring 2009, Pages 174–196</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:3:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:3:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:3:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:3:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:3:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:3:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:3:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4733597">Clements, Adam and Preve, Daniel P. A. and Tee, Clarence, Harvesting the HAR-X Volatility Model</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0378426621002417">Adam Clements, Daniel P.A. Preve, A Practical Guide to harnessing the HAR volatility model, Journal of Banking & Finance, Volume 133, 2021</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:6:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:6:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:6:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:6:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:6:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:6:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:6:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:6:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:6:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a> <a href="#fnref:6:14" class="reversefootnote" role="doc-backlink">↩<sup>15</sup></a> <a href="#fnref:6:15" class="reversefootnote" role="doc-backlink">↩<sup>16</sup></a> <a href="#fnref:6:16" class="reversefootnote" role="doc-backlink">↩<sup>17</sup></a> <a href="#fnref:6:17" class="reversefootnote" role="doc-backlink">↩<sup>18</sup></a> <a href="#fnref:6:18" class="reversefootnote" role="doc-backlink">↩<sup>19</sup></a> <a href="#fnref:6:19" class="reversefootnote" role="doc-backlink">↩<sup>20</sup></a> <a href="#fnref:6:20" class="reversefootnote" role="doc-backlink">↩<sup>21</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>This additive cascade model is autoregressive in (daily) realized variance and combines realized variance over different horizons (daily, weekly, monthly), hence its name. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0169207020301874">Stefan Lyocsa, Peter Molnar, Tomas Vyrost, Stock market volatility forecasting: Do we need high-frequency data?, International Journal of Forecasting, Volume 37, Issue 3, 2021, Pages 1092-1110</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1544612320315907">Stefan Lyocsa, Tomas Plihal, Tomas Vyrost, FX market volatility modelling: Can we use low-frequency data?, Finance Research Letters, Volume 40, 2021, 101776</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:25:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>These three volatility estimators are described in details in <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">a previous blog post</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://centaur.reading.ac.uk/21316/">Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:16:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>Also called iterated scheme. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://direct.mit.edu/rest/article-abstract/97/3/683/58249/Good-Volatility-Bad-Volatility-Signed-Jumps-and?redirectedFrom=fulltext">Andrew J. Patton, Kevin Sheppard; Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility. The Review of Economics and Statistics 2015; 97 (3): 683–697</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:14:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>Some papers like Buccheri and Corsi<sup id="fnref:19:2" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup> describe how to explicitely model the HAR model parameters as time-varying, but in that case, those parameters are not anymore obtained through a simple least squares regression… <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>The length of the rolling window is typically 1000 days, corresponding approximately to 1000 days of trading; more details on the rolling window procedure can be found in Clements et al.<sup id="fnref:4:5" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> and a numerical comparison in terms of out-of-sample volatility forecasts between an expanding and a rolling window procedure can be found in Clements and Preve<sup id="fnref:6:21" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>And I confirm from practical experience. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>Such a filter has apparently a long history in the forecasting litterature, dating back at least to Swanson and White<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">26</a></sup> in the context of neural-network models for interest rates forecasting. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://repositorio.fgv.br/items/0bedff75-288e-47c1-a9fb-7c455bd5c533">Cunha, Ronan, Kock, Anders Bredahl and Pereira, Pedro L. Valls, Forecasting large covariance matrices: comparing autometrics and LASSOVAR</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Already hinted at in Corsi<sup id="fnref:3:14" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article-abstract/19/4/614/5553055?redirectedFrom=fulltext">Giuseppe Buccheri, Fulvio Corsi, HARK the SHARK: Realized Volatility Modeling with Measurement Errors and Nonlinear Dependencies, Journal of Financial Econometrics, Volume 19, Issue 4, Fall 2021, Pages 614–649</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:19:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:19:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>Which helps limiting the need for an insanity filter. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X12001808">Fulvio Corsi, Nicola Fusari, Davide La Vecchia, Realizing smiles: Options pricing with realized volatility, Journal of Financial Economics, Volume 107, Issue 2, 2013, Pages 284-304</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:23:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://saltfinancial.com/static/uploads/2021/05/The%20Laymans%20Guide%20to%20Volatility%20Forecasting.pdf">Salt Financial, The Layman’s Guide to Volatility Forecasting: Predicting the Future, One Day at a Time, Research Note</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://econpapers.repec.org/bookchap/nbrnberch/1214.htm">Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">27</a></sup>. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>For all models, I used an expanding window for the volatility forecast computation. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:28:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1392186">Swanson, N. R. and H. White (1995). A model-selection approach to assessing the information in the term structure using linear models and artificial neural networks. Journal of Business & Economic Statistics 13 (3), 265–275</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Among the different members of the family of volatility forecasting models by weighted moving average1 like the simple and the exponentially weighted moving average models or the GARCH(1,1) model, the Heterogeneous AutoRegressive (HAR) model introduced by Corsi2 has become the workhorse of the volatility forecasting literature3 on account of its simplicity and generally good forecasting performance3. In this blog post, strongly inspired by the paper A Practical Guide to harnessing the HAR volatility model from Clements and Preve4, I will describe the HAR volatility forecasting model together with some important implementation details and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs. Mathematical preliminaries (reminders) This section contains reminders from a previous blog post. Volatility modelling and volatility proxies Let $r_t$ be the (logarithmic) return of an asset over a time period $t$ (a day, a week, a month..), over which its (conditional) mean return is supposed to be null. Then: The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $ From this definition, the squared return $r_t^2$ of an asset is a (noisy4) variance estimator - or variance proxy4 - for that asset variance over the considered time period. Another example of an asset variance proxy is the Parkinson range of an asset. Yet another example of an asset variance proxy, this time over a specific time period $t$ of one day, is the daily realized variance $RV_t$ of an asset, which is defined as the sum of the asset squared intraday returns sampled at a high frequency (1 minutes, 5 minutes, 15 minutes…). The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$. The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $ The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$. Weighted moving average volatility forecasting model Boudoukh et al.1 show that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with \[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\] , where: $1 \leq k \leq T$ is the size of the moving average, possibly time-dependent $w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well The HAR volatility forecasting model The original HAR model Due to the limitations of the GARCH model in reproducing the main empirical features of financial returns (long memory, fat tails, and self-similarity)2, Corsi2 proposes to use an additive cascade model of different volatility components each of which is generated by the actions of different types of market participants2. As detailled by Corsi2: The main idea is that agents with different time horizons perceive, react to, and cause different types of volatility components. Simplifying a bit, we can identify three primary volatility components: the short-term traders with daily or higher trading frequency, the medium-term investors who typically rebalance their positions weekly, and the long-term agents with a characteristic time of one or more months. Under this volatility forecasting model called the Heterogeneous AutoRegressive model (HAR)5, an asset next day’s daily realized variance $RV_{T+1}$ is modeled as an AR(22) process subject to economically meaningful restrictions2 on its parameters, which results in the formula4 \[\hat{RV}_{T+1} = \beta + \beta_d RV_{T} + \beta_w RV_{T}^w + \beta_m RV_{T}^m\] , where: $\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$ $RV_T$ is the asset daily realized variance at time $T$ $RV_T^w = \frac{1}{5} \sum_{i=1}^5 RV_{T-i+1}$ is the asset weekly realized variance at time $T$ $RV_T^m = \frac{1}{22} \sum_{i=1}^{22} RV_{T-i+1}$ is the asset monthly realized variance at time $T$ $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined In terms of practical performances, and in spite of its simplicity […], the [HAR] model is able to reproduce the same volatility persistence observed in the empirical data as well as many of the other main stylized facts of financial data2, which makes it a very accurate volatility forecasting model. Realized variance v.s. generic variance proxy The original HAR model described in the previous subsection relies on a very specific asset variance proxy - the realized variance of an asset - over a very specific time period - a day - for its definition. Some papers (Lyocsa et al.6, Lyocsa et al.7, Clements et al.3, …) propose to replace the (high-frequency) daily realized variance by a (low-frequency) daily range-based variance estimator8 like: The square of the Parkinson volatility estimator The square of the Garman-Klass volatility estimator The square of the Rogers-Satchell volatility estimator Going one step further, it is possible to replace the daily realized variance by any generic daily variance estimator. This leads to the generic HAR volatility forecasting model, under which an asset next days’s conditional variance $\sigma_{T+1}^2$ is modeled as a linear function of [its previous day’s] daily, weekly and monthly [conditional variance] components4, with the following daily variance forecasting formula \[\hat{\sigma}_{T+1}^2 = \beta + \beta_d \tilde{\sigma}^2_{T} + \beta_w \tilde{\sigma}^{2,w}_{T} + \beta_m \tilde{\sigma}^{2,m}_{T}\] , where: $\hat{\sigma}_{T+1}^2$ is the forecast at time $T$ of the asset next day’s conditional variance $\sigma_{T+1}^2$ $\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$ $\tilde{\sigma}^{2,w}_{T} = \frac{1}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$ $\tilde{\sigma}^{2,m}_{T} = \frac{1}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$ $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined Going another step further, it is also possible to replace the baseline daily time period by any desired time period (weekly, biweekly, monthly, quarterly…), but given the theoretical foundations of the HAR model, this also requires to replace the weekly and the monthly variance estimators $\tilde{\sigma}^{2,w}_{T}$ and $\tilde{\sigma}^{2,m}_{T}$ by appropriate variance estimators. Relationship with the generic weighted moving average model From its definition, it is easy to see that the HAR volatility forecasting model is a specific kind of weighted moving average volatility forecasting model, with: $w_0 = \beta$ $w_1 = \beta_d + \frac{1}{5} \beta_w + \frac{1}{22} \beta_m$ $w_i = \frac{1}{5} \beta_w + \frac{1}{22} \beta_m, i = 2..5$ $w_i = \frac{1}{22} \beta_m, i = 6..22$ $w_i = 0$, $i \geq 23$, discarding all the past variance proxies beyond the $22$-th from the model Volatility forecasting formulas Under an HAR volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes: To estimate an asset next day’s volatility: \[\hat{\sigma}_{T+1} = \sqrt{ \beta + \beta_d \tilde{\sigma}^2_{T} + \frac{\beta_w}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i} + \frac{\beta_m}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i} }\] To estimate an asset next $h$-day’s ahead volatility9, $h \geq 2$: \[\hat{\sigma}_{T+h} = \sqrt{ \beta + \beta_d \hat{\sigma}_{T+h-1}^2 + \frac{\beta_w}{5} \left( \sum_{i=1}^{5-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) + \frac{\beta_m}{22} \left( \sum_{i=1}^{22-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) }\] To estimate an asset aggregated volatility9 over the next $h$ days: \[\hat{\sigma}_{T+1:T+h} = \sqrt{ \sum_{i=1}^{h} \hat{\sigma}^2_{T+i} }\] Note: Clements and Preve4 extensively discuss whether to use an indirect or a direct multi-step ahead forecast scheme for estimating an asset next $h$-day’s ahead volatility under an HAR volatility forecasting model: In an indirect scheme10, which is the scheme described above, the asset next $h$-day’s ahead volatility is estimated indirectly, by a recursive application of the HAR model formula for the asset next days’s conditional variance. In a direct scheme, the asset next $h$-day’s ahead volatility is estimated directly, by replacing the asset next days’s conditional variance on the left-hand side of the HAR model formula with the asset next $h$-day’s ahead conditional variance. Clements and Preve4 argue in particular that “direct forecasts are easy to compute and more robust to model misspecification compared to indirect forecasts”4, but at the same time, they show that this robustness does not always translate into more accurate forecasts. Harnessing the HAR volatility forecasting model HAR model parameters estimation Ordinary least squares estimators Corsi2 writes that it is possible to easily estimate [the HAR model] parameters by applying simple linear regression2, in which case the ordinary least squares (OLS) estimator of the parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ at time $T \geq 23$ is the solution of the minimization problem4 \[\argmin_{ \left( \beta, \beta_d, \beta_w, \beta_m \right) \in \mathbb{R}^{4}} \sum_{t=23}^T \left( \tilde{\sigma}_{t}^2 - \beta - \beta_d \tilde{\sigma}^2_{t-1} - \beta_w \tilde{\sigma}^{2,w}_{t-1} - \beta_m \tilde{\sigma}^{2,m}_{t-1} \right)^2\] Other least squares estimators Nevertheless, Clements and Preve4 warn that given the stylized facts of [volatility estimators] (such as spikes/outliers, conditional heteroskedasticity, and non-Gaussianity) and well-known properties of OLS, this […] should be far from ideal4. Indeed, as detailled in Patton and Sheppard11: Because the dependent variable in all of our regressions is a volatility measure, estimation by OLS has the unfortunate feature that the resulting estimates focus primarily on fitting periods of high variance and place little weight on more tranquil periods. This is an important drawback in our applications, as the level of variance changes substantially across our sample period and the level of the variance and the volatility in the error are known to have a positive relationship. Instead of OLS, Clements and Preve4 and Clements et al.3 suggest to use other least squares estimators: Weighted least squares estimators (WLS), with for example the weighting schemes described in Clements and Preve4 Robust least squares estimators (RLS), with for example Tukey’s biweight loss function Regularized least squares estimators (RRLS) (ridge regression, LASSO regression, elastic net regression…), with the cross-validation procedure for the associated hyperparameters described in Clements et al.3. Expanding v.s. rolling window estimation procedure Clements and Preve4 empirically demonstrate that the HAR model parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are time-varying, as illustrated in Figure 1. Figure 1. Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve. To deal with this non-stationarity, or more generally to deal with parameter drift that is difficult to model explicitly4, the standard12 approach in the litterature is to use a rolling window procedure13 for the least squares estimation of the HAR model parameters. Insanity filter The HAR volatility forecasting model may on rare occasions generate implausibly large or small forecasts4, because no restrictions on the parameters [$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$] are imposed11. In particular, it has been noted14 that forecasts are occasionally negative11. In order to correct this behaviour, Clements and Preve4 propose to implement an insanity filter15 ensuring that any forecast greater than the maximum, or less than the minimum, of [an asset next days’s conditional variance] observed in the estimation period is replaced by the sample average over that period4. Two important remarks on such a filter: It seems that the particular choice of insanity filter is not important; what matters is to eliminate unrealistic forecasts16. The presence of an insanity filter is all the more important when an indirect multi-step ahead forecast scheme is used for estimating an asset next $h$-day’s ahead volatility, because an unreasonable [forecast] is most likely to generate another unreasonable forecast16. Variance proxies transformations Clements and Preve4 study three different Box-Cox transformations of an asset daily realized variance that can be used in the HAR volatility forecasting model instead of that asset “raw” daily realized variance: The logarithmic transformation (log) The quartic root transformation (qr) The square root transformation (sqr) It is visible on Figure 2 that these transformations appear useful for reducing skewness, and hence the possible effect of outliers and potential heteroskedasticity in the realized variance series4. Figure 2. S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve. Of particular interest is the logarithmic transformation17, which in addition to good practical performances and closed form bias-corrected expressions for variance forecasts18, also guarantees that the generated forecasts are always positive19. Non-overlapping daily, weekly and monthly variance proxies Corsi et al.20 propose a slightly different parametrization of the HAR model compared to Corsi2, where: $\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$ $\tilde{\sigma}^{2,w}_{T} = \frac{1}{4} \sum_{i=2}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$, non-overlapping with the asset daily variance estimator $\tilde{\sigma}^2_{T}$ $\tilde{\sigma}^{2,m}_{T} = \frac{1}{17} \sum_{i=6}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$, non-overlapping with either the asset daily variance estimator $\tilde{\sigma}^2_{T}$ nor with the asset weekly variance estimator $\tilde{\sigma}^{2,w}_{T}$ Such a re-parametrization does not imply any loss of information compared to the original [HAR model], since it relies only on a different rearrangement of the terms20, but allows an easiest interpretation of the HAR model parameters, c.f. Patton and Sheppard11. Incidentally, this non-overlapping formulation of the HAR volatility forecasting model has been found to generate better forecasts by practitioners21. Other lag indexes for the variance proxies The standard lag indexes for the variance proxies in the HAR volatility forecasting model are 1, 5 and 22, each corresponding to a different volatility component in Corsi’s underlying additive cascade model. As mentioned in Corsi2, more components could easily be added to the additive cascade of partial volatilities2, which is done for example in Lyocsa et al.7. In that case, denoting $l$ and $h$, respectively, the lowest and highest frequency in the cascade2, an asset next days’s conditional variance is modeled as an AR($\frac{l}{h}$) process reparameterized in a parsimonious way by imposing economically meaningful restrictions2. Implementation in Portfolio Optimizer Portfolio Optimizer implements the HAR volatility forecasting model - augmented with the insanity filter described in Clements and Preve4 - through the endpoint /assets/volatility/forecast/har. This endpoint supports the 4 variance proxies below: Squared close-to-close returns Demeaned squared close-to-close returns The Parkinson range The jump-adjusted Parkinson range This endpoint also supports: Transforming the input variance proxies into log variance proxies before estimating the HAR model parameters; the associated variance forecasts are then unbiased thanks to the formulas established in Buccheri and Corsi18. Estimating the HAR model parameters through 3 different least squares procedures: Ordinary least squares Weighted least squares, using the inverse of the variance proxies as weights Robust least squares, using Tukey’s biweight loss function Using up to 5 lag indexes for the variance proxies Example of usage - Volatility forecasting at monthly level for various ETFs As an example of usage, I propose to enrich the results of the previous blog post, in which monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz22 regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative23 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Vanilla HAR volatility forecasting model Averaged results for all ETFs/regression models over each ETF price history24 are the following25, when using the vanilla HAR volatility forecasting model: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ Random walk Squared close-to-close returns 5.8% 0.66 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Squared close-to-close returns 5.8% 0.68 46% EWMA, optimal $\lambda$ Squared close-to-close returns 4.7% 0.73 45% GARCH(1,1) Squared close-to-close returns -1.3% 0.98 43% HAR Squared close-to-close returns -0.7% 0.95 46% Random walk Parkinson range 5.6% 0.94 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Parkinson range 5.1% 1.00 47% EWMA, optimal $\lambda$ Parkinson range 4.3% 1.06 48% GARCH(1,1) Parkinson range 2.7% 1.18 47% HAR Parkinson range 0.1% 1.25 44% Random walk Jump-adjusted Parkinson range 4.9% 0.70 45% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Jump-adjusted Parkinson range 5.1% 0.71 47% EWMA, optimal $\lambda$ Jump-adjusted Parkinson range 4.0% 0.76 45% GARCH(1,1) Jump-adjusted Parkinson range -1.0% 1.00 45% HAR Jump-adjusted Parkinson range -1.4% 0.99 47% Alternative HAR volatility forecasting models Averaged results for all ETFs/regression models over each ETF price history24 are the following25, when using different variations of the vanilla HAR volatility forecasting model: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ HAR Squared close-to-close returns -0.7% 0.95 46% HAR (weighted least squares) Squared close-to-close returns 3.1% 0.71 34% HAR (robust least squares) Squared close-to-close returns -12.3% 2.50 26% HAR (log) Squared close-to-close returns 0.5% 0.62 40% HAR (log, weighted least squares) Squared close-to-close returns 10% 0.26 13% HAR (log, robust least squares) Squared close-to-close returns 0.5% 0.53 40% HAR Parkinson range 0.1% 1.25 44% HAR (weighted least squares) Parkinson range 0.1% 1.19 44% HAR (robust least squares) Parkinson range -4.2% 2.20 40% HAR (log) Parkinson range 1.9% 1.22 50% HAR (log, weighted least squares) Parkinson range -0.6% 1.47 47% HAR (log, robust least squares) Parkinson range 2.2% 1.22 50% HAR Jump-adjusted Parkinson range -1.4% 0.99 47% HAR (weighted least squares) Jump-adjusted Parkinson range -4.2% 0.92 46% HAR (robust least squares) Jump-adjusted Parkinson range -6.6% 1.76 41% HAR (log) Jump-adjusted Parkinson range 0.9% 0.92 51% HAR (log, weighted least squares) Jump-adjusted Parkinson range -0.8% 1.06 48% HAR (log, robust least squares) Jump-adjusted Parkinson range 1.2% 0.92 51% Comments From the results of the two previous subsections, it is possible to make the following comments: When using squared returns as a variance proxy, the vanilla HAR model is the best volatility forecasting model among all the alternative HAR models as well as among all the models already studied in this series. When using the Parkinson range or the jump-adjusted Parkinson range as variance proxies, the log HAR model exhibits the highest r-squared among all the models already studied in this series. When using the jump-adjusted Parkinson range as a variance proxy, the log HAR model is the best volatility forecasting model in this series (relatively low bias, highest r-squared). When using alternative least square estimators, the forecasts quality generally degrades Conclusion The previous section empirically demonstrated that the HAR volatility forecasting model, despite its relationship with realized variance, is still accurate when used with range-based variance estimators at a long forecasting horizon, in line with Lyocsa et al.6 In that context, the (log) HAR model is also the most accurate volatility forecasting model studied so far in this series on volatility forecasting by weighted moving average models! As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71. ↩ ↩2 See Fulvio Corsi, A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics, Volume 7, Issue 2, Spring 2009, Pages 174–196. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 See Clements, Adam and Preve, Daniel P. A. and Tee, Clarence, Harvesting the HAR-X Volatility Model. ↩ ↩2 ↩3 ↩4 ↩5 See Adam Clements, Daniel P.A. Preve, A Practical Guide to harnessing the HAR volatility model, Journal of Banking & Finance, Volume 133, 2021. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 This additive cascade model is autoregressive in (daily) realized variance and combines realized variance over different horizons (daily, weekly, monthly), hence its name. ↩ See Stefan Lyocsa, Peter Molnar, Tomas Vyrost, Stock market volatility forecasting: Do we need high-frequency data?, International Journal of Forecasting, Volume 37, Issue 3, 2021, Pages 1092-1110. ↩ ↩2 See Stefan Lyocsa, Tomas Plihal, Tomas Vyrost, FX market volatility modelling: Can we use low-frequency data?, Finance Research Letters, Volume 40, 2021, 101776. ↩ ↩2 These three volatility estimators are described in details in a previous blog post. ↩ See Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22. ↩ ↩2 Also called iterated scheme. ↩ See Andrew J. Patton, Kevin Sheppard; Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility. The Review of Economics and Statistics 2015; 97 (3): 683–697. ↩ ↩2 ↩3 ↩4 Some papers like Buccheri and Corsi18 describe how to explicitely model the HAR model parameters as time-varying, but in that case, those parameters are not anymore obtained through a simple least squares regression… ↩ The length of the rolling window is typically 1000 days, corresponding approximately to 1000 days of trading; more details on the rolling window procedure can be found in Clements et al.3 and a numerical comparison in terms of out-of-sample volatility forecasts between an expanding and a rolling window procedure can be found in Clements and Preve4. ↩ And I confirm from practical experience. ↩ Such a filter has apparently a long history in the forecasting litterature, dating back at least to Swanson and White26 in the context of neural-network models for interest rates forecasting. ↩ See Cunha, Ronan, Kock, Anders Bredahl and Pereira, Pedro L. Valls, Forecasting large covariance matrices: comparing autometrics and LASSOVAR. ↩ ↩2 Already hinted at in Corsi2. ↩ See Giuseppe Buccheri, Fulvio Corsi, HARK the SHARK: Realized Volatility Modeling with Measurement Errors and Nonlinear Dependencies, Journal of Financial Econometrics, Volume 19, Issue 4, Fall 2021, Pages 614–649. ↩ ↩2 ↩3 Which helps limiting the need for an insanity filter. ↩ See Fulvio Corsi, Nicola Fusari, Davide La Vecchia, Realizing smiles: Options pricing with realized volatility, Journal of Financial Economics, Volume 107, Issue 2, 2013, Pages 284-304. ↩ ↩2 See Salt Financial, The Layman’s Guide to Volatility Forecasting: Predicting the Future, One Day at a Time, Research Note. ↩ See Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer27. ↩ The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. ↩ ↩2 For all models, I used an expanding window for the volatility forecast computation. ↩ ↩2 See Swanson, N. R. and H. White (1995). A model-selection approach to assessing the information in the term structure using linear models and artificial neural networks. Journal of Business & Economic Statistics 13 (3), 265–275. ↩ See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Combating Volatility Laundering: Unsmoothing Artificially Smoothed Returns2024-06-01T00:00:00-05:002024-06-01T00:00:00-05:00https://portfoliooptimizer.io/blog/combating-volatility-laundering-unsmoothing-artificially-smoothed-returns<h2 id="introduction">Introduction</h2>
<p>It is common knowledge that <em>returns to hedge funds and other alternative investments [like private equity or real estate] are often highly serially correlated</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>This results in apparently smooth returns that <em>have artificially lower volatilities and covariations with other asset classes</em><sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>, which in turn <em>bias [portfolio] allocations toward the smoothed asset classes</em><sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>In this blog post, I will detail two simple unsmoothing procedures designed to help with these problems.</p>
<p>As examples of usage, I will show that these procedures allow a fairer comparison between private and public U.S. real estate investments and I will
illustrate the impact of smoothed v.s. unsmoothed returns on the mean-variance efficient frontier associated to a universe made of both public and private investments.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>A Google Sheet corresponding to this post is available <a href="https://docs.google.com/spreadsheets/d/1s8sA5B8t2Hpxwxe8Tb7X983kPlW7elk96ZwxvLJ2NxA/edit?usp=sharing">here</a>.</li>
</ul>
</blockquote>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="autocorrelation-of-a-time-series">Autocorrelation of a time series</h3>
<p>Let be:</p>
<ul>
<li>$x_1,…,x_T$ the values of a time series observed over $T$ time periods</li>
<li>$\bar{x} = \frac{1}{T} \sum_{i=1}^T x_i$ the arithmetic average of that time series</li>
</ul>
<p>The <em>autocovariance function</em> - or <em>serial covariance function</em> - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">3</a></sup>:</p>
\[\gamma_n = \frac{1}{T} \sum_{i=\max(1,-n)}^{\min(T-n,T)} \left( x_{i+n} - \bar{x} \right) \left( x_i - \bar{x} \right)\]
<p>The <em>autocorrelation function</em> - or <em>serial correlation function</em> - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">3</a></sup>:</p>
\[\rho_n = \frac{\gamma_n}{\gamma_0}\]
<p>These functions represent the covariance and the correlation between the time series $x_{1+n},…,x_T$ and a delayed copy of itself $x_1,…,x_{T-n}$, hence their name.</p>
<h3 id="autocorrelation-of-financial-asset-returns-and-volatility-laundering">Autocorrelation of financial asset returns and volatility laundering</h3>
<p>In financial asset returns, it has been noted that both positive and negative autocorrelation <em>might be present simultaneously at different horizons</em><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">4</a></sup>.</p>
<p>Nevertheless, for both theoretical and practical reasons, any significant and persistent autocorrelation should quickly be arbitraged away<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">5</a></sup>.</p>
<p>It is thus quite remarkable that returns on several alternative assets (art<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">6</a></sup>, collectible stamps<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">7</a></sup>, hedge funds<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, private equity<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">8</a></sup>, private infrastructure<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">9</a></sup>, private real estate<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>, wine<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">11</a></sup>…)
have been found to be significantly (positively) autocorrelated.</p>
<p>Such an autocorrelation is nevertheless spurious - a <em>purely mathematical artifact</em><sup id="fnref:17:1" role="doc-noteref"><a href="#fn:17" class="footnote">9</a></sup> - and originates from the characteristics of these alternative assets:</p>
<ul>
<li>Illiquidity (infrequent trading, infrequent mark-to-market valuations…)</li>
<li>Smoothing of the underlying “true” economic returns</li>
<li>Lagging reporting and/or appraisal process</li>
<li>Fees structure</li>
<li>…</li>
</ul>
<p>Indeed, as explained<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> by Getmansky et al.<sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>:</p>
<blockquote>
<p>In such cases, the <em>reported</em> returns of [these assets] will appear to be smoother than true <em>economic</em> returns (returns that fully reflect all available market information […]) and this, in turn, will […] yield positive serial return correlation.</p>
</blockquote>
<p>Unfortunately, even if artificial, the presence of autocorrelation in these assets’ returns <em>invalidates traditional measures of risk and performance</em><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup> <em>such as volatility, Sharpe ratio,
correlation, and market-beta estimates</em><sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> and ultimately <em>presents the risks of [these] assets as more attractive than they are</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">11</a></sup>, especially when compared to typical public alternatives (stocks, REITs…).</p>
<p>Incidentally, the <em>smoothing service</em><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">14</a></sup> offered by these alternative assets has been named <em>volatility laundering</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">15</a></sup><sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">16</a></sup> by <a href="https://www.linkedin.com/in/cliff-asness/">Cliff Asness</a>, the founder of
<a href="https://www.aqr.com/">AQR Capital Management</a>.</p>
<h2 id="unsmoothing-artificially-smoothed-financial-asset-returns">Unsmoothing artificially smoothed financial asset returns</h2>
<p>In order to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">17</a></sup>, one possibility is to unsmooth (or desmooth) those smoothed returns
so as to recover the <em>true underlying (unobservable) returns</em><sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>.</p>
<p>In other words, given a time series $r_1,…,r_T$ of artificially smoothed financial asset returns, we would like to determine the “hidden” underlying unsmoothed time series $r_1^*,…,r_T^*$.</p>
<p>Now, of course, <em>the nature and interpretation of such an [unsmoothed] series depends on the assumptions that are imbedded in the smoothing model and unsmoothing procedure</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>.</p>
<p>The standard approach <em>is to rely on the assumption that the underlying true returns are uncorrelated across time, consistent with the classical hypothesis of weak-form informational efficiency in asset markets</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>.</p>
<p>Based on this assumption, several different unsmoothing procedures have been proposed in the litterature<sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup><sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">20</a></sup>, the two simplest ones being:</p>
<ul>
<li>The procedure of Geltner<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup> (or of Fisher-Geltner-Webb<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>), able to approximately remove the first-order autocorrelation from a time series</li>
<li>The procedure of Okunev and White<sup id="fnref:8:1" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>, an extension of Geltner’s procedure able to approximately remove the autocorrelation of any order from a time series</li>
</ul>
<h3 id="geltners-procedure">Geltner’s procedure</h3>
<p>The smoothing model of Geltner<sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup> assumes that the reported (smoothed) return at time $t$ $r_t$ is a linear combination of the true economic (unsmoothed) return at time $t$ $r_t^*$ and the one-period lagged reported
return $r_{t-1}$:</p>
\[r_t = \left( 1 - \alpha \right) r_t^* + \alpha r_{t-1}, t=2..T\]
<p>, with $\alpha \in \left[ 0,1 \right]$ an unknown parameter to be determined.</p>
<p>After some manipulations, this leads to the following formula for the time series $r_1^*,…,r_T^*$:</p>
\[r_1^* = r_1\]
\[r_t^* = \frac{r_t - c r_{t-1}}{1 - c}, t=2..T\]
<p>, with $c = \rho_1$ the first-order autocorrelation of the time series $r_1,…,r_T$.</p>
<p>It can be shown<sup id="fnref:8:2" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> that the unsmoothed time series has:</p>
<ul>
<li>An arithmetic mean equal to that of the original time series</li>
<li>Its first-order autocorrelation approximately null</li>
</ul>
<h3 id="okunev-and-whites-procedure">Okunev and White’s procedure</h3>
<p>One <em>difficulty with [Geltner’s procedure] is that it is only strictly correct for <a href="https://en.wikipedia.org/wiki/Autoregressive_model">an AR(1) process</a> and it only acts to remove first order autocorrelation</em><sup id="fnref:8:3" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</p>
<p>This led Okunev and White<sup id="fnref:8:4" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> to propose an extension of Geltner’s smoothing model by assuming that the reported return at time $t$ $r_t$ is a linear combination of the true economic
return at time $t$ $r_t^*$ and the lagged reported returns $r_{t-1}, r_{t-2}, …, r_{t-m}$:</p>
\[r_t = \left( 1 - \alpha \right) r_t^* + \sum_{i=1}^m \beta_i r_{t-i}, t=m+1..T\]
<p>, with:</p>
<ul>
<li>$m \geq 1$ the number of lagged reported returns to consider in order to determine the true economic returns, usually taken equal to 1 or 2 in practical applications</li>
<li>$\beta_i \in \left[ 0,1 \right]$, $i=1..m$</li>
<li>$\alpha = \sum_{i=1}^m \beta_i$</li>
</ul>
<p>Under that model, Okunev and White<sup id="fnref:8:5" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> describes how to approximately remove the $m$-th order autocorrelation from the time series $r_1,…,r_T$:</p>
<ol>
<li>
<p>Define $r_{0,t} = r_t$, $t=1..T$</p>
</li>
<li>
<p>Remove the first-order autocorrelation from the time series $r_{0,1},…,r_{0,T}$ by defining the time series $r_{1,1},…,r_{1,T}$</p>
\[r_{1,1} = r_{0,1}\]
\[r_{1,t} = \frac{r_{0,t} - c_1 r_{0,t-1}}{1 - c_1}, t=2..T\]
<p>, with:</p>
<ul>
<li>$c_1 = \frac{\left( 1+a_{0,2} \right) \pm \sqrt{\left( 1+a_{0,2} \right)^2 - 4a_{0,1}^2}}{2a_{0,1}}$, choosen so that $\left| c_1 \right| \leq 1$</li>
<li>$a_{0,1} = \rho_1$ the first-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$</li>
<li>$a_{0,2} = \rho_2$ the second-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$</li>
</ul>
</li>
<li>
<p>For $i=2..m$, remove the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ by defining the time series $r_{i,1},…,r_{i,T}$</p>
\[r_{i,t} = r_{i-1,t}, t=1..i\]
\[r_{i,t} = \frac{r_{i-1,t} - c_i r_{i-1,t-1}}{1 - c_i}, t=i+1..T\]
<p>, with:</p>
<ul>
<li>$c_i = \frac{\left( 1+a_{i-1,2i} \right) \pm \sqrt{\left( 1+a_{i-1,2i} \right)^2 - 4a_{i-1,i}^2}}{2a_{i-1,i}}$, choosen so that $\left| c_i \right| \leq 1$</li>
<li>$a_{i-1,i}$ the $i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$</li>
<li>$a_{i-1,2i}$ the $2i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$</li>
</ul>
</li>
</ol>
<p>It can be shown<sup id="fnref:8:6" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> that the unsmoothed time series has:</p>
<ul>
<li>An arithmetic mean equal to that of the original time series</li>
<li>Its first $m$-th order autocorrelations approximately null</li>
</ul>
<blockquote>
<p><strong><em>Notes on Okunev and White’s procedure:</em></strong></p>
<ul>
<li>It is iterative by nature, because removing the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ alters its $i-1$-th order autocorrelation; steps 1, 2 and 3 thus need to be applied iteratively, <em>until the first $m$ autocorrelations are sufficiently close to zero</em><sup id="fnref:8:7" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</li>
<li>It allows, more generally, to set the first $m$-th order autocorrelations of the time series $r_1,…,r_T$ to any desired values, under certain mathematical conditions.</li>
<li>Its influence on the skewness, the kurtosis and the <a href="/blog/corrected-cornish-fisher-expansion-improving-the-accuracy-of-modified-value-at-risk/">modified Value-At-Risk</a> of the unsmoothed returns has been studied<sup id="fnref:22:1" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> in Gallais-Hamonno and Nguyen-Thi-Thanh<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup>.</li>
</ul>
</blockquote>
<h3 id="misc-remarks">Misc. remarks</h3>
<p>A couple of miscealenous remarks on the two unsmoothing procedures described in the previous sub-sections:</p>
<ul>
<li>Both procedures should preferably be used with smoothed logarithmic<sup id="fnref:7:2" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> asset returns rather than smoothed arithmetic asset returns because unsmoothing arithmetic returns might result in returns that are less than -100%<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">22</a></sup>.</li>
<li>Both procedure require some numerical attention when determining the unsmoothing coefficients $c$ or $c_1,…,c_m$.</li>
<li>Okunev and White’s procedure better nullifies the first order autocorrelation of a time series of smoothed asset returns than Geltner’s procedure.</li>
</ul>
<h2 id="adjusting-the-performance-statistics-of-artificially-smoothed-financial-asset-returns">Adjusting the performance statistics of artificially smoothed financial asset returns</h2>
<p>Another possibility to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed is to adjust traditional performance statistics<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">23</a></sup><br />
<em>to control for the spurious serial correlation in [those] returns</em><sup id="fnref:8:8" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</p>
<p>Describing such adjustments is out of scope of this blog post, but the interested reader can for example refer to:</p>
<ul>
<li>Lo<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">24</a></sup>, in which the “square root of time” rule typically used to annualize a monthly Sharpe Ratio is shown to be invalid in the presence of autocorrelation and in which a more appropriate adjustment method is determined.</li>
<li>Bailey and de Prado<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">25</a></sup>, in which the impact of first-order autocorrelation on maximum drawdown and time under water is analyzed.</li>
</ul>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements both Geltner’s and Okunev and White’s procedures through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/unsmoothed</code></a>.</p>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="comparing-private-and-public-real-estate-investments">Comparing private and public real estate investments</h3>
<p>In the blog post <em><a href="https://insights.finominal.com/research-the-case-against-private-markets/">The Case Against Private Markets</a></em>, <a href="https://www.linkedin.com/in/nicolas-rabener-caia-a345353a">Nicolas Rabener</a>
from <a href="https://finominal.com/">Finominal</a> highlights the <em>discrepancies between private and public market valuations</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">26</a></sup> in the case of U.S. real estate.</p>
<p>In more details, he compares the Blackstone Real Estate Income Trust <a href="https://www.breit.com/">BREIT</a> - a private real estate investment - together with the
<a href="https://institutional.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=FREL">Fidelity MSCI Real Estate Index ETF (FREL)</a> - a publicly traded real estate investment.</p>
<p>Rabener then shows that:</p>
<ul>
<li>The volatility of BREIT is absurdly low compared to that of the FREL ETF</li>
<li>The Sharpe Ratio of BREIT is thus absurdly high compared to that of the FREL ETF</li>
</ul>
<p>Problem is, those two investments are exposed to the same underlying economic drivers<sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">26</a></sup>, so that such differences do not really make sense…</p>
<p>I propose to revisit Rabener’s results, this time comparing BREIT with the <a href="https://investor.vanguard.com/investment-products/etfs/profile/vnq">Vanguard Real Estate Index Fund ETF (VNQ)</a>, both depicted in Figure 1<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">27</a></sup>.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-breit-vs-vnq.png"><img src="/assets/images/blog/combating-volatility-laundering-breit-vs-vnq.png" alt="Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024." /></a>
<figcaption>Figure 1. Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024.</figcaption>
</figure>
<p>Some statistics<sup id="fnref:41" role="doc-noteref"><a href="#fn:41" class="footnote">28</a></sup> for these two funds:</p>
<table>
<thead>
<tr>
<th>Instrument</th>
<th>Annualized Mean</th>
<th>Annualized Standard Deviation</th>
<th>Annualized Sharpe Ratio</th>
<th>First-order autocorrelation</th>
</tr>
</thead>
<tbody>
<tr>
<td>BREIT</td>
<td>10.10%</td>
<td>4.58%</td>
<td>2.20</td>
<td>0.42</td>
</tr>
<tr>
<td>VNQ ETF</td>
<td>6.26%</td>
<td>18.68%</td>
<td>0.34</td>
<td>-0.13</td>
</tr>
</tbody>
</table>
<p>From Figure 1 and the associated statistics (e.g., one-fourth of the volatility of its public counterpart, strong first-order autocorrelation), BREIT is a textbook case of volatility laundering that we can combat
thanks to Geltner’s procedure<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup>!</p>
<p>Figure 2 depicts the resulting unsmoothed BREIT together with the original BREIT and the VNQ ETF.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-breit-unsmoothed-vs-vnq.png"><img src="/assets/images/blog/combating-volatility-laundering-breit-unsmoothed-vs-vnq.png" alt="Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024." /></a>
<figcaption>Figure 2. Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024.</figcaption>
</figure>
<p>Some additional statistics for the unsmoothed BREIT:</p>
<table>
<thead>
<tr>
<th>Instrument</th>
<th>Annualized Mean</th>
<th>Annualized Standard Deviation</th>
<th>Annualized Sharpe Ratio</th>
<th>First-order autocorrelation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Unsmoothed BREIT</td>
<td>10.28%</td>
<td>6.85%</td>
<td>1.50</td>
<td>-0.04</td>
</tr>
</tbody>
</table>
<p>While Figure 2 is hardly distinguishable from Figure 1, the associated statistics are clear and consistent with the litterature:</p>
<ul>
<li>Unsmoothing the returns of BREIT increases its volatility by nearly 50%</li>
<li>Unsmoothing the returns of BREIT also decreases its annualized Sharpe Ratio by approximately 30%</li>
</ul>
<p>This small exercice empirically validates that unsmoothing private asset returns makes them <em>look and feel more like a public market proxy</em><sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">30</a></sup>.</p>
<p>To be noted that the unsmoothed BREIT still seems a little bit too smooth when compared to the VNQ ETF.</p>
<p>More on this later.</p>
<h3 id="analysing-hedge-funds-returns">Analysing hedge funds returns</h3>
<p>Brooks and Kat<sup id="fnref:31:1" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>, as well as many other authors<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:8:9" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup><sup id="fnref:20:1" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup><sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">31</a></sup>, observes that hedge funds exhibit significant levels of first - and sometimes second or higher - order autocorrelation,
which <em>has important consequences for investors</em><sup id="fnref:31:2" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>.</p>
<p>In particular, and thanks to the unsmoothing procedures previously described:</p>
<ul>
<li>
<p>Gallais-Hamonno and Nguyen-Thi-Thanh<sup id="fnref:20:2" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup> notes that</p>
<blockquote>
<p>[…] the hedge fund performance measured by traditional ratios considerably decreases after unsmoothing, decrease which is on average 20%, even 25% according to the method and to the performance ratio used.</p>
</blockquote>
<blockquote>
<p>[…] 20% of [hedge funds] have a change in rankings. Moreover, there are several cases of strong “down-grading” and of strong “over-grading” […]</p>
</blockquote>
</li>
<li>
<p>Okunev and White<sup id="fnref:8:10" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> shows that</p>
<blockquote>
<p>After removing the autocorrelations from returns, we find increases in risk of between 60 and 100 percent for many of the individual indices.</p>
</blockquote>
</li>
</ul>
<p>So, when analysing hedge funds returns, a pre-requisite is to always use an unsmoothing procedure!</p>
<h3 id="mitigating-covariance-misspecification-in-asset-allocation">Mitigating covariance misspecification in asset allocation</h3>
<p>In the context of a multi-asset portfolio, Peterson and Grier<sup id="fnref:7:3" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> shows through a <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo</a> <a href="https://en.wikipedia.org/wiki/Modern_portfolio_theory">mean-variance</a>
experiment that <em>the naive covariance estimates [relying on smoothed asset returns] generate significant weight bias, undesirable allocative tilts, and higher portfolio risk</em><sup id="fnref:7:4" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>Similar in spirit to that paper, I propose to illustrate the impact of smoothed returns on mean-variance analysis.</p>
<p>For this:</p>
<ul>
<li>I consider a U.S.-centric universe of assets made of five <a href="https://www.preqin.com/">Preqin</a> private investment indexes and their <a href="https://www.preqin.com/help-center/articles/view/public-market-equivalents">public markets equivalents</a>:
<ul>
<li>Private Equity and S&P 500 index</li>
<li>Venture Capital and Russell 2000 index</li>
<li>Private Debt and an High Yield Bonds index<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">32</a></sup></li>
<li>Private Real Estate and MSCI U.S. Real Estate index</li>
<li>
<p>Private Infrastructure and S&P Infrastructure index</p>
<p>Figure 3 displays the (quarterly) evolution of all these indexes over the period 31 December 2007 - 31 December 2023.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-preqin-vs-pme.png"><img src="/assets/images/blog/combating-volatility-laundering-preqin-vs-pme.png" alt="Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 3. Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
</li>
</ul>
</li>
<li>Within that universe, I then compute two mean-variance efficient frontiers over the period 31 December 2007 - 31 December 2023, using in-sample annualized mean-variance inputs<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">33</a></sup>:
<ul>
<li>A “raw” efficient frontier, using reported returns for all indexes</li>
<li>
<p>An “unsmoothed” efficient frontier, using unsmoothed<sup id="fnref:37" role="doc-noteref"><a href="#fn:37" class="footnote">34</a></sup> reported returns for the private investment indexes and reported returns for their public markets equivalents</p>
<p>Figure 4 represents these two efficient frontiers in the mean-variance plane, together with the different indexes of the considered universe.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-efficient-frontiers.png"><img src="/assets/images/blog/combating-volatility-laundering-efficient-frontiers.png" alt="Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 4. Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
<p>From Figure 4, and in accordance with Peterson and Grier<sup id="fnref:7:5" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>:</p>
<ul>
<li>The standard deviations of the private investment indexes increase significantly when their reported returns are unsmoothed.</li>
<li><em>The difference in the locations of the two frontiers is largely<sup id="fnref:38" role="doc-noteref"><a href="#fn:38" class="footnote">35</a></sup> a result of the underestimated standard deviations of the smoothed [private investment indexes]</em><sup id="fnref:7:6" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</li>
</ul>
<p>This confirms that <em>the naive covariance matrix underestimates risk and overestimates returns relative to the more efficient revised covariance matrix</em><sup id="fnref:7:7" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> and that unsmoothing is necessary to <em>remove potential biases and improve efficiency by maximizing the information content of [the mean-variance input] estimates</em><sup id="fnref:7:8" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>One important remark, though, is that the unsmoothing procedure used only partially resolved the issues related to smoothed asset returns.</p>
<p>Indeed, the transition maps<sup id="fnref:39" role="doc-noteref"><a href="#fn:39" class="footnote">36</a></sup> of the two efficient frontiers (Figure 5) shows that unsmoothed mean-variance efficient portfolios are still heavily concentrated in private assets…</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-efficient-frontiers-transition-maps.png"><img src="/assets/images/blog/combating-volatility-laundering-efficient-frontiers-transition-maps.png" alt="Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 5. Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
<p>So, to paraphrase <a href="https://www.linkedin.com/in/trym-riksen-6580952/">Trym Riksen</a>, private assets break the Markovitz’s model, and do so even when their “specificities” are accounted for!</p>
<p>More on this right now.</p>
</li>
</ul>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>A proper conclusion is certainly that <em>in general, [private asset returns] series must be unsmoothed before analysis</em><sup id="fnref:7:9" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> to avoid introducing any unwanted bias.</p>
<p>Nevertheless, the two unsmoothing procedures described in this blog post are not entirely satisfying because the resulting unsmoothed returns are yet too smooth…</p>
<p>Next in this series on combating volatility laundering, I will describe another unsmoothing procedure, this time able to take into account<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">37</a></sup> the economic relationship between
a private asset and its public market equivalent, which might improve its practical performances.</p>
<p>In order to smoothly stay up-to-date with I am doing, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X04000698">Mila Getmansky, Andrew W. Lo, Igor Makarov, An econometric model of serial correlation and illiquidity in hedge fund returns, Journal of Financial Economics, Volume 74, Issue 3, 2004, Pages 529-609</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/4480761">Steven P. Peterson and John T. Grier, Covariance Misspecification in Asset Allocation, Financial Analysts Journal, Vol. 62, No. 4 (Jul. - Aug., 2006), pp. 76-85</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:7:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:7:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:7:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:7:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:7:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:7:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:7:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:7:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://link.springer.com/book/10.1007/978-0-387-21706-2">Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article/19/1/39/6124729">On the Autocorrelation of the Stock Market, Ian Martin, Journal of Financial Econometrics, Volume 19, Issue 1, Winter 2021, Pages 39–52</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://www.worldscientific.com/doi/abs/10.1142/9789814566926_0002">Samuelson P. A. 1965. Proof That Properly Anticipated Prices Fluctuate Randomly. Industrial Management Review 6: 41–49</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p><a href="https://www.pm-research.com/content/iijaltinv/10/4/64">Campbell, R., 2008. Art as a financial investment. Journal of Alternative Investments 10, 64–81</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X10002941">Dimson, Elroy, and Christophe Spaenjers. 2011. Ex post: The investment performance of collectible stamps. Journal of Financial Economics 100:443–458</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://onlinelibrary.wiley.com/doi/10.1111/jofi.12688">Ang, A., Chen, B., Goetzmann, W. N., Phalippou, L., 2018. Estimating private equity returns from limited partner cash flows. The Journal of Finance 73, 1751–1783</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.cembenchmarking.com/ri/insight/16">Alexander D. Beath, PhD and Chris Flynn, CFA, ASSET ALLOCATION, COST OF INVESTING AND PERFORMANCE OF EUROPEAN DB PENSION FUNDS: THE IMPACT OF REAL ESTATE, CEM Benchmarking Inc.</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:17:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/BF00161933">Geltner, David, 1991, Smoothing in Appraisal-Based Returns, Journal of Real Estate Finance and Economics, Vol.4, p.327-345</a>, <a href="https://www.jstor.org/stable/44095449">Geltner, David, 1993, Estimating Market Values from Appraised Values without Assuming an Efficient Market, Journal of Real Estate Research, Vol.8, p.325-345</a> and <a href="https://link.springer.com/article/10.1007/BF01099972">Fisher, J. D., Geltner, D. M., & Webb, R. B. (1994). Value indices of commercial real estate: A comparison of index construction methods. The Journal of Realv Estate Finance and Economics, 9 (2), 137–164</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=4775496">Verdickt, Gertjan, Volatility Laundering: On the Feasibility of Wine Investment Funds (March 28, 2024)</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>In the context of hedge funds. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:22:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://academic.oup.com/rfs/advance-article-abstract/doi/10.1093/rfs/hhae006/7625074">Spencer J Couts, Andrei S Gonçalves, Andrea Rossi, Unsmoothing Returns of Illiquid Funds, The Review of Financial Studies, 2024</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/49/4/21">Antti Ilmanen, Investing in Interesting Times, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 21 - 35</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://www.aqr.com/Insights/Perspectives/Volatility-Laundering">Cliff Asness, Volatility Laundering, Perspective</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>In the context of private equity, c.f. the podcast <a href="https://twiii.podbean.com/e/private-equity-and-the-game-of-volatility-laundering/">Private Equity and the Game of ”Volatility Laundering”</a> from <a href="https://twiii.podbean.com/">This Week in Intelligent Investing</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Or <em>volatility laundered</em>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijaltinv/5/2/26">Brooks, C. and H. Kat, The statistical properties of Hedge Fund index returns and their implications for investors, The Journal of Alternative Investments, Fall 2002, 5 (2) 26-44</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:31:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:31:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijfixinc/16/2/46">Geoff Loudon, John Okunev, Derek White, Hedge Fund Risk Factors and the Value at Risk of Fixed Income Trading Strategies, The Journal of Fixed Income, Fall 2006, 16 (2) 46-61</a> and <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=460641">Okunev, John and White, Derek, Hedge Fund Risk Factors and Value at Risk of Credit Trading Strategies (October 2003)</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:8:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:8:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:8:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:8:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:8:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:8:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:8:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:8:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:8:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:8:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1059944">Shilling, James D. Measurement Error in FRC/NCREIF Returns on Real Estate. Southern Economic Journal, vol. 60, no. 1, 1993, pp. 210–19</a>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>See <a href="https://shs.hal.science/halshs-00184470">Georges Gallais-Hamonno, Huyen Nguyen-Thi-Thanh. The necessity to correct hedge fund returns: empirical evidence and correction method. 2007</a>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:20:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>For example, use the time series [-0.94, -0.96, -0.98, -0.95]. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>Called i.i.d.-based in de Prado<sup id="fnref:24:1" role="doc-noteref"><a href="#fn:24" class="footnote">25</a></sup>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.2469/faj.v58.n4.2453">Lo, A. W., 2002. The statistics of sharpe ratios. Financial Analysts Journal 58, No. 4, 36–52</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2201302">Bailey, David H. and Lopez de Prado, Marcos, Stop-Outs Under Serial Correlation and ‘The Triple Penance Rule’</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:24:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://insights.finominal.com/research-the-case-against-private-markets/">The Case Against Private Markets</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:41" role="doc-endnote">
<p>Due to the first-order autocorrelation in BREIT returns, the naive annualization of the volatility and of the Sharpe Ratio that I have done is incorrect, c.f. Lo<sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">24</a></sup>; this is done on purpose, though. <a href="#fnref:41" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>Okunev and White’s procedure gives nearly identical results. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>See <a href="https://www.venn.twosigma.com/insights/desmoothing-201">Bingxu Chen & Christopher Carrano, Two Easy Steps to Take Your Private Asset Returns Public, Vnn by Two Sigma</a>. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/28/1/6">Clifford S Asness, Robert J Krail, John M Liew, Do Hedge Funds Hedge?, The Journal of Portfolio Management, Fall 2001, 28 (1) 6-19</a>. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>The associated index is the Markit iBoxx USD Liquid High Yield Index; comparison of Private Debt to both Investment Grade and High Yield bonds is discussed in Boni and Manigart<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">38</a></sup>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>In-sample asset mean returns and in-sample asset returns covariance matrix, using asset returns over the period 31 March 2008 - 31 December 2023. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:37" role="doc-endnote">
<p>Reported returns for the private investment indexes were unsmoothed using the Okunev and White’s procedure of order 1; results are similar with Geltner’s procedure, but since I already used Geltner’s procedure in the context of real-estate, I decided to play with its extension. <a href="#fnref:37" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:38" role="doc-endnote">
<p>To be noted that the in-sample asset mean returns are numerically different between smoothed and unsmoothed returns, but this does not explain the differences between the two frontiers. <a href="#fnref:38" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:39" role="doc-endnote">
<p>The transition map of the efficient frontier is a stacked area plot of the composition, in terms of asset weights, of the efficient portfolios belonging to the efficient frontier. <a href="#fnref:39" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>That being said, a variant of Geltner’s method allow to use a public market equivalent of a private asset to “inject” volatility in the unsmoothed returns, c.f. Geltner<sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=3932484">Boni, Pascal and Manigart, Sophie, Private Debt Fund Returns, Persistence, and Market Conditions</a>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Introduction It is common knowledge that returns to hedge funds and other alternative investments [like private equity or real estate] are often highly serially correlated1. This results in apparently smooth returns that have artificially lower volatilities and covariations with other asset classes2, which in turn bias [portfolio] allocations toward the smoothed asset classes2. In this blog post, I will detail two simple unsmoothing procedures designed to help with these problems. As examples of usage, I will show that these procedures allow a fairer comparison between private and public U.S. real estate investments and I will illustrate the impact of smoothed v.s. unsmoothed returns on the mean-variance efficient frontier associated to a universe made of both public and private investments. Notes: A Google Sheet corresponding to this post is available here. Mathematical preliminaries Autocorrelation of a time series Let be: $x_1,…,x_T$ the values of a time series observed over $T$ time periods $\bar{x} = \frac{1}{T} \sum_{i=1}^T x_i$ the arithmetic average of that time series The autocovariance function - or serial covariance function - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as3: \[\gamma_n = \frac{1}{T} \sum_{i=\max(1,-n)}^{\min(T-n,T)} \left( x_{i+n} - \bar{x} \right) \left( x_i - \bar{x} \right)\] The autocorrelation function - or serial correlation function - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as3: \[\rho_n = \frac{\gamma_n}{\gamma_0}\] These functions represent the covariance and the correlation between the time series $x_{1+n},…,x_T$ and a delayed copy of itself $x_1,…,x_{T-n}$, hence their name. Autocorrelation of financial asset returns and volatility laundering In financial asset returns, it has been noted that both positive and negative autocorrelation might be present simultaneously at different horizons4. Nevertheless, for both theoretical and practical reasons, any significant and persistent autocorrelation should quickly be arbitraged away5. It is thus quite remarkable that returns on several alternative assets (art6, collectible stamps7, hedge funds1, private equity8, private infrastructure9, private real estate10, wine11…) have been found to be significantly (positively) autocorrelated. Such an autocorrelation is nevertheless spurious - a purely mathematical artifact9 - and originates from the characteristics of these alternative assets: Illiquidity (infrequent trading, infrequent mark-to-market valuations…) Smoothing of the underlying “true” economic returns Lagging reporting and/or appraisal process Fees structure … Indeed, as explained12 by Getmansky et al.1: In such cases, the reported returns of [these assets] will appear to be smoother than true economic returns (returns that fully reflect all available market information […]) and this, in turn, will […] yield positive serial return correlation. Unfortunately, even if artificial, the presence of autocorrelation in these assets’ returns invalidates traditional measures of risk and performance13 such as volatility, Sharpe ratio, correlation, and market-beta estimates1 and ultimately presents the risks of [these] assets as more attractive than they are11, especially when compared to typical public alternatives (stocks, REITs…). Incidentally, the smoothing service14 offered by these alternative assets has been named volatility laundering1516 by Cliff Asness, the founder of AQR Capital Management. Unsmoothing artificially smoothed financial asset returns In order to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed17, one possibility is to unsmooth (or desmooth) those smoothed returns so as to recover the true underlying (unobservable) returns18. In other words, given a time series $r_1,…,r_T$ of artificially smoothed financial asset returns, we would like to determine the “hidden” underlying unsmoothed time series $r_1^*,…,r_T^*$. Now, of course, the nature and interpretation of such an [unsmoothed] series depends on the assumptions that are imbedded in the smoothing model and unsmoothing procedure10. The standard approach is to rely on the assumption that the underlying true returns are uncorrelated across time, consistent with the classical hypothesis of weak-form informational efficiency in asset markets10. Based on this assumption, several different unsmoothing procedures have been proposed in the litterature101913120, the two simplest ones being: The procedure of Geltner10 (or of Fisher-Geltner-Webb10), able to approximately remove the first-order autocorrelation from a time series The procedure of Okunev and White19, an extension of Geltner’s procedure able to approximately remove the autocorrelation of any order from a time series Geltner’s procedure The smoothing model of Geltner10 assumes that the reported (smoothed) return at time $t$ $r_t$ is a linear combination of the true economic (unsmoothed) return at time $t$ $r_t^*$ and the one-period lagged reported return $r_{t-1}$: \[r_t = \left( 1 - \alpha \right) r_t^* + \alpha r_{t-1}, t=2..T\] , with $\alpha \in \left[ 0,1 \right]$ an unknown parameter to be determined. After some manipulations, this leads to the following formula for the time series $r_1^*,…,r_T^*$: \[r_1^* = r_1\] \[r_t^* = \frac{r_t - c r_{t-1}}{1 - c}, t=2..T\] , with $c = \rho_1$ the first-order autocorrelation of the time series $r_1,…,r_T$. It can be shown19 that the unsmoothed time series has: An arithmetic mean equal to that of the original time series Its first-order autocorrelation approximately null Okunev and White’s procedure One difficulty with [Geltner’s procedure] is that it is only strictly correct for an AR(1) process and it only acts to remove first order autocorrelation19. This led Okunev and White19 to propose an extension of Geltner’s smoothing model by assuming that the reported return at time $t$ $r_t$ is a linear combination of the true economic return at time $t$ $r_t^*$ and the lagged reported returns $r_{t-1}, r_{t-2}, …, r_{t-m}$: \[r_t = \left( 1 - \alpha \right) r_t^* + \sum_{i=1}^m \beta_i r_{t-i}, t=m+1..T\] , with: $m \geq 1$ the number of lagged reported returns to consider in order to determine the true economic returns, usually taken equal to 1 or 2 in practical applications $\beta_i \in \left[ 0,1 \right]$, $i=1..m$ $\alpha = \sum_{i=1}^m \beta_i$ Under that model, Okunev and White19 describes how to approximately remove the $m$-th order autocorrelation from the time series $r_1,…,r_T$: Define $r_{0,t} = r_t$, $t=1..T$ Remove the first-order autocorrelation from the time series $r_{0,1},…,r_{0,T}$ by defining the time series $r_{1,1},…,r_{1,T}$ \[r_{1,1} = r_{0,1}\] \[r_{1,t} = \frac{r_{0,t} - c_1 r_{0,t-1}}{1 - c_1}, t=2..T\] , with: $c_1 = \frac{\left( 1+a_{0,2} \right) \pm \sqrt{\left( 1+a_{0,2} \right)^2 - 4a_{0,1}^2}}{2a_{0,1}}$, choosen so that $\left| c_1 \right| \leq 1$ $a_{0,1} = \rho_1$ the first-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$ $a_{0,2} = \rho_2$ the second-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$ For $i=2..m$, remove the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ by defining the time series $r_{i,1},…,r_{i,T}$ \[r_{i,t} = r_{i-1,t}, t=1..i\] \[r_{i,t} = \frac{r_{i-1,t} - c_i r_{i-1,t-1}}{1 - c_i}, t=i+1..T\] , with: $c_i = \frac{\left( 1+a_{i-1,2i} \right) \pm \sqrt{\left( 1+a_{i-1,2i} \right)^2 - 4a_{i-1,i}^2}}{2a_{i-1,i}}$, choosen so that $\left| c_i \right| \leq 1$ $a_{i-1,i}$ the $i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$ $a_{i-1,2i}$ the $2i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$ It can be shown19 that the unsmoothed time series has: An arithmetic mean equal to that of the original time series Its first $m$-th order autocorrelations approximately null Notes on Okunev and White’s procedure: It is iterative by nature, because removing the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ alters its $i-1$-th order autocorrelation; steps 1, 2 and 3 thus need to be applied iteratively, until the first $m$ autocorrelations are sufficiently close to zero19. It allows, more generally, to set the first $m$-th order autocorrelations of the time series $r_1,…,r_T$ to any desired values, under certain mathematical conditions. Its influence on the skewness, the kurtosis and the modified Value-At-Risk of the unsmoothed returns has been studied12 in Gallais-Hamonno and Nguyen-Thi-Thanh21. Misc. remarks A couple of miscealenous remarks on the two unsmoothing procedures described in the previous sub-sections: Both procedures should preferably be used with smoothed logarithmic2 asset returns rather than smoothed arithmetic asset returns because unsmoothing arithmetic returns might result in returns that are less than -100%22. Both procedure require some numerical attention when determining the unsmoothing coefficients $c$ or $c_1,…,c_m$. Okunev and White’s procedure better nullifies the first order autocorrelation of a time series of smoothed asset returns than Geltner’s procedure. Adjusting the performance statistics of artificially smoothed financial asset returns Another possibility to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed is to adjust traditional performance statistics23 to control for the spurious serial correlation in [those] returns19. Describing such adjustments is out of scope of this blog post, but the interested reader can for example refer to: Lo24, in which the “square root of time” rule typically used to annualize a monthly Sharpe Ratio is shown to be invalid in the presence of autocorrelation and in which a more appropriate adjustment method is determined. Bailey and de Prado25, in which the impact of first-order autocorrelation on maximum drawdown and time under water is analyzed. Implementation in Portfolio Optimizer Portfolio Optimizer implements both Geltner’s and Okunev and White’s procedures through the endpoint /assets/returns/unsmoothed. Examples of usage Comparing private and public real estate investments In the blog post The Case Against Private Markets, Nicolas Rabener from Finominal highlights the discrepancies between private and public market valuations26 in the case of U.S. real estate. In more details, he compares the Blackstone Real Estate Income Trust BREIT - a private real estate investment - together with the Fidelity MSCI Real Estate Index ETF (FREL) - a publicly traded real estate investment. Rabener then shows that: The volatility of BREIT is absurdly low compared to that of the FREL ETF The Sharpe Ratio of BREIT is thus absurdly high compared to that of the FREL ETF Problem is, those two investments are exposed to the same underlying economic drivers26, so that such differences do not really make sense… I propose to revisit Rabener’s results, this time comparing BREIT with the Vanguard Real Estate Index Fund ETF (VNQ), both depicted in Figure 127. Figure 1. Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024. Some statistics28 for these two funds: Instrument Annualized Mean Annualized Standard Deviation Annualized Sharpe Ratio First-order autocorrelation BREIT 10.10% 4.58% 2.20 0.42 VNQ ETF 6.26% 18.68% 0.34 -0.13 From Figure 1 and the associated statistics (e.g., one-fourth of the volatility of its public counterpart, strong first-order autocorrelation), BREIT is a textbook case of volatility laundering that we can combat thanks to Geltner’s procedure29! Figure 2 depicts the resulting unsmoothed BREIT together with the original BREIT and the VNQ ETF. Figure 2. Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024. Some additional statistics for the unsmoothed BREIT: Instrument Annualized Mean Annualized Standard Deviation Annualized Sharpe Ratio First-order autocorrelation Unsmoothed BREIT 10.28% 6.85% 1.50 -0.04 While Figure 2 is hardly distinguishable from Figure 1, the associated statistics are clear and consistent with the litterature: Unsmoothing the returns of BREIT increases its volatility by nearly 50% Unsmoothing the returns of BREIT also decreases its annualized Sharpe Ratio by approximately 30% This small exercice empirically validates that unsmoothing private asset returns makes them look and feel more like a public market proxy30. To be noted that the unsmoothed BREIT still seems a little bit too smooth when compared to the VNQ ETF. More on this later. Analysing hedge funds returns Brooks and Kat18, as well as many other authors1192131, observes that hedge funds exhibit significant levels of first - and sometimes second or higher - order autocorrelation, which has important consequences for investors18. In particular, and thanks to the unsmoothing procedures previously described: Gallais-Hamonno and Nguyen-Thi-Thanh21 notes that […] the hedge fund performance measured by traditional ratios considerably decreases after unsmoothing, decrease which is on average 20%, even 25% according to the method and to the performance ratio used. […] 20% of [hedge funds] have a change in rankings. Moreover, there are several cases of strong “down-grading” and of strong “over-grading” […] Okunev and White19 shows that After removing the autocorrelations from returns, we find increases in risk of between 60 and 100 percent for many of the individual indices. So, when analysing hedge funds returns, a pre-requisite is to always use an unsmoothing procedure! Mitigating covariance misspecification in asset allocation In the context of a multi-asset portfolio, Peterson and Grier2 shows through a Monte Carlo mean-variance experiment that the naive covariance estimates [relying on smoothed asset returns] generate significant weight bias, undesirable allocative tilts, and higher portfolio risk2. Similar in spirit to that paper, I propose to illustrate the impact of smoothed returns on mean-variance analysis. For this: I consider a U.S.-centric universe of assets made of five Preqin private investment indexes and their public markets equivalents: Private Equity and S&P 500 index Venture Capital and Russell 2000 index Private Debt and an High Yield Bonds index32 Private Real Estate and MSCI U.S. Real Estate index Private Infrastructure and S&P Infrastructure index Figure 3 displays the (quarterly) evolution of all these indexes over the period 31 December 2007 - 31 December 2023. Figure 3. Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023. Within that universe, I then compute two mean-variance efficient frontiers over the period 31 December 2007 - 31 December 2023, using in-sample annualized mean-variance inputs33: A “raw” efficient frontier, using reported returns for all indexes An “unsmoothed” efficient frontier, using unsmoothed34 reported returns for the private investment indexes and reported returns for their public markets equivalents Figure 4 represents these two efficient frontiers in the mean-variance plane, together with the different indexes of the considered universe. Figure 4. Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023. From Figure 4, and in accordance with Peterson and Grier2: The standard deviations of the private investment indexes increase significantly when their reported returns are unsmoothed. The difference in the locations of the two frontiers is largely35 a result of the underestimated standard deviations of the smoothed [private investment indexes]2. This confirms that the naive covariance matrix underestimates risk and overestimates returns relative to the more efficient revised covariance matrix2 and that unsmoothing is necessary to remove potential biases and improve efficiency by maximizing the information content of [the mean-variance input] estimates2. One important remark, though, is that the unsmoothing procedure used only partially resolved the issues related to smoothed asset returns. Indeed, the transition maps36 of the two efficient frontiers (Figure 5) shows that unsmoothed mean-variance efficient portfolios are still heavily concentrated in private assets… Figure 5. Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023. So, to paraphrase Trym Riksen, private assets break the Markovitz’s model, and do so even when their “specificities” are accounted for! More on this right now. Conclusion A proper conclusion is certainly that in general, [private asset returns] series must be unsmoothed before analysis2 to avoid introducing any unwanted bias. Nevertheless, the two unsmoothing procedures described in this blog post are not entirely satisfying because the resulting unsmoothed returns are yet too smooth… Next in this series on combating volatility laundering, I will describe another unsmoothing procedure, this time able to take into account37 the economic relationship between a private asset and its public market equivalent, which might improve its practical performances. In order to smoothly stay up-to-date with I am doing, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Mila Getmansky, Andrew W. Lo, Igor Makarov, An econometric model of serial correlation and illiquidity in hedge fund returns, Journal of Financial Economics, Volume 74, Issue 3, 2004, Pages 529-609. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 See Steven P. Peterson and John T. Grier, Covariance Misspecification in Asset Allocation, Financial Analysts Journal, Vol. 62, No. 4 (Jul. - Aug., 2006), pp. 76-85. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 See Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag. ↩ ↩2 See On the Autocorrelation of the Stock Market, Ian Martin, Journal of Financial Econometrics, Volume 19, Issue 1, Winter 2021, Pages 39–52. ↩ See Samuelson P. A. 1965. Proof That Properly Anticipated Prices Fluctuate Randomly. Industrial Management Review 6: 41–49. ↩ Campbell, R., 2008. Art as a financial investment. Journal of Alternative Investments 10, 64–81. ↩ See Dimson, Elroy, and Christophe Spaenjers. 2011. Ex post: The investment performance of collectible stamps. Journal of Financial Economics 100:443–458. ↩ See Ang, A., Chen, B., Goetzmann, W. N., Phalippou, L., 2018. Estimating private equity returns from limited partner cash flows. The Journal of Finance 73, 1751–1783. ↩ See Alexander D. Beath, PhD and Chris Flynn, CFA, ASSET ALLOCATION, COST OF INVESTING AND PERFORMANCE OF EUROPEAN DB PENSION FUNDS: THE IMPACT OF REAL ESTATE, CEM Benchmarking Inc.. ↩ ↩2 See Geltner, David, 1991, Smoothing in Appraisal-Based Returns, Journal of Real Estate Finance and Economics, Vol.4, p.327-345, Geltner, David, 1993, Estimating Market Values from Appraised Values without Assuming an Efficient Market, Journal of Real Estate Research, Vol.8, p.325-345 and Fisher, J. D., Geltner, D. M., & Webb, R. B. (1994). Value indices of commercial real estate: A comparison of index construction methods. The Journal of Realv Estate Finance and Economics, 9 (2), 137–164. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 See Verdickt, Gertjan, Volatility Laundering: On the Feasibility of Wine Investment Funds (March 28, 2024). ↩ ↩2 In the context of hedge funds. ↩ ↩2 See Spencer J Couts, Andrei S Gonçalves, Andrea Rossi, Unsmoothing Returns of Illiquid Funds, The Review of Financial Studies, 2024. ↩ ↩2 See Antti Ilmanen, Investing in Interesting Times, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 21 - 35. ↩ See Cliff Asness, Volatility Laundering, Perspective. ↩ In the context of private equity, c.f. the podcast Private Equity and the Game of ”Volatility Laundering” from This Week in Intelligent Investing. ↩ Or volatility laundered. ↩ See Brooks, C. and H. Kat, The statistical properties of Hedge Fund index returns and their implications for investors, The Journal of Alternative Investments, Fall 2002, 5 (2) 26-44. ↩ ↩2 ↩3 See Geoff Loudon, John Okunev, Derek White, Hedge Fund Risk Factors and the Value at Risk of Fixed Income Trading Strategies, The Journal of Fixed Income, Fall 2006, 16 (2) 46-61 and Okunev, John and White, Derek, Hedge Fund Risk Factors and Value at Risk of Credit Trading Strategies (October 2003). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 See Shilling, James D. Measurement Error in FRC/NCREIF Returns on Real Estate. Southern Economic Journal, vol. 60, no. 1, 1993, pp. 210–19. ↩ See Georges Gallais-Hamonno, Huyen Nguyen-Thi-Thanh. The necessity to correct hedge fund returns: empirical evidence and correction method. 2007. ↩ ↩2 ↩3 For example, use the time series [-0.94, -0.96, -0.98, -0.95]. ↩ Called i.i.d.-based in de Prado25. ↩ See Lo, A. W., 2002. The statistics of sharpe ratios. Financial Analysts Journal 58, No. 4, 36–52. ↩ See Bailey, David H. and Lopez de Prado, Marcos, Stop-Outs Under Serial Correlation and ‘The Triple Penance Rule’. ↩ ↩2 See The Case Against Private Markets. ↩ ↩2 (Adjusted) prices have have been retrieved using Tiingo. ↩ Due to the first-order autocorrelation in BREIT returns, the naive annualization of the volatility and of the Sharpe Ratio that I have done is incorrect, c.f. Lo24; this is done on purpose, though. ↩ Okunev and White’s procedure gives nearly identical results. ↩ See Bingxu Chen & Christopher Carrano, Two Easy Steps to Take Your Private Asset Returns Public, Vnn by Two Sigma. ↩ See Clifford S Asness, Robert J Krail, John M Liew, Do Hedge Funds Hedge?, The Journal of Portfolio Management, Fall 2001, 28 (1) 6-19. ↩ The associated index is the Markit iBoxx USD Liquid High Yield Index; comparison of Private Debt to both Investment Grade and High Yield bonds is discussed in Boni and Manigart38. ↩ In-sample asset mean returns and in-sample asset returns covariance matrix, using asset returns over the period 31 March 2008 - 31 December 2023. ↩ Reported returns for the private investment indexes were unsmoothed using the Okunev and White’s procedure of order 1; results are similar with Geltner’s procedure, but since I already used Geltner’s procedure in the context of real-estate, I decided to play with its extension. ↩ To be noted that the in-sample asset mean returns are numerically different between smoothed and unsmoothed returns, but this does not explain the differences between the two frontiers. ↩ The transition map of the efficient frontier is a stacked area plot of the composition, in terms of asset weights, of the efficient portfolios belonging to the efficient frontier. ↩ That being said, a variant of Geltner’s method allow to use a public market equivalent of a private asset to “inject” volatility in the unsmoothed returns, c.f. Geltner10. ↩ See Boni, Pascal and Manigart, Sophie, Private Debt Fund Returns, Persistence, and Market Conditions. ↩Bootstrap Simulations with Exact Sample Mean Vector and Sample Covariance Matrix2024-05-05T00:00:00-05:002024-05-05T00:00:00-05:00https://portfoliooptimizer.io/blog/bootstrap-simulations-with-exact-sample-mean-vector-and-sample-covariance-matrix<p><a href="https://en.wikipedia.org/wiki/Bootstrapping_(statistics)">Bootstrapping</a> is a statistical method which consists in sampling with replacement from an original data set
to compute the distribution of a desired <a href="https://en.wikipedia.org/wiki/Statistic">statistic</a>, with <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">plenty of possible variations</a>
depending on the exact context (non-dependent data, dependent data…).</p>
<p>Because <em>bootstrap methods are not, in general, based on any particular assumption on the distribution of the data, they are well suited for the analysis of [financial] returns</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, in which case
the original data set typically consists of historical return time series<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>.</p>
<p>Unfortunately, bootstrap simulations <em>have the disadvantage that only previously recorded values can be simulated</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, so that they cannot <em>explicitly reflect the investors’ expectations</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In this blog post, inspired by <a href="https://www.linkedin.com/in/robert-stuart-carver/">Rob Carver</a>’s piece
<em><a href="https://qoppac.blogspot.com/2023/12/portfolio-optimisation-uncertainty.html">Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots</a></em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup>, I will describe a way to incorporate exact views
on expected returns, standard deviations and correlations into bootstrapping so that the resulting simulations both <em>preserve the salient characteristics of asset returns</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>
and also match forward-looking risk<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">5</a></sup> and return assumptions.</p>
<p>As an example of application, I will revisit the methodology described in the research note <em>Considering the Past and the Future in Asset Simulation</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> from <a href="https://www.troweprice.com/en/us">T. Rowe Price</a>
and incorporate the full range of capital market assumptions into <em>a systematic modelling process to simulate potential asset returns for portfolio construction</em><sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<p>Let be:</p>
<ul>
<li>$n$, the number of assets in a universe of assets</li>
<li>$T$, the number of time periods</li>
<li>$R \in \mathcal{M}(\mathbb{R}^{T \times n})$, with $R_{t,i}$ the return<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">6</a></sup> of the asset $i=1..n$ for the time period $t=1..T$, representing a scenario (historical, simulated…) for the temporal evolution of the asset returns</li>
<li>$\mu \in \mathbb{R}^{n}$, the vector of the arithmetic averages of the asset returns $R$</li>
<li>$\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the covariance matrix of the asset returns $R$, supposed to be invertible to avoid numerical subtleties</li>
</ul>
<h3 id="moment-matching">Moment-matching</h3>
<p>Let be:</p>
<ul>
<li>$\bar{\mu} \in \mathbb{R}^{n}$ a vector</li>
<li>$\bar{\Sigma} \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a <a href="https://en.wikipedia.org/wiki/Definite_matrix">positive-definite matrix</a></li>
</ul>
<p>Suppose that we would like the first two empirical moments of the asset returns $R$ - that is, $\mu$ and $\Sigma$ - to take the values $\bar{\mu}$ and $\bar{\Sigma}$.</p>
<p>How to proceed?</p>
<p>This problem - called <em>moment-matching by twisting scenarios</em> in <a href="https://www.linkedin.com/in/attilio-meucci-72580013/">Attilio Meucci</a>’s <a href="https://www.arpm.co/">Advanced Risk and Portfolio Management</a> website -
consists in transforming the original asset returns $R$ into modified asset returns $\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n})$, so that:</p>
<ul>
<li>$ \frac{1}{T} \sum_{t=1}^{T} \tilde{R}_{t,i} = \bar{\mu}_i $, $i=1..n$</li>
<li>$ \frac{1}{T} \sum_{k=1}^T \left( \tilde{R}_{k,i} - \bar{\mu}_i \right) \left( \tilde{R}_{k,j} - \bar{\mu}_j \right) = \bar{\Sigma}_{i,j} $, $i=1..n$, $j=1..n$</li>
</ul>
<h3 id="moment-matching-with-n1-shift-and-rescaling-procedure">Moment-matching with $n=1$, shift and rescaling procedure</h3>
<p>When $n=1$, a procedure to compute the modified asset returns $\tilde{R}$ from the original asset returns $R$ is discussed in Boyle et al.<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup>.</p>
<p>It involves the following <em>intuitive shift/rescaling</em><sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> of the original asset returns:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu} + \frac{\bar{\sigma}}{\sigma} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu \right)\]
<p>, with:</p>
<ul>
<li>$\sigma = \sqrt{\Sigma_{1,1}}$, the (sample) standard deviation of the original asset returns</li>
<li>$\bar{\sigma} = \sqrt{\bar{\Sigma}_{1,1}}$, the desired (sample) standard deviation of the modified asset returns</li>
</ul>
<h3 id="moment-matching-with-n-geq-2">Moment-matching with $n \geq 2$</h3>
<h4 id="multivariate-shift-and-rescaling-procedure">Multivariate shift and rescaling procedure</h4>
<p>When $n \geq 2$, Kaut and Lium<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> hints at a multivariate generalization of the procedure described in the previous sub-section, which becomes <a href="https://en.wikipedia.org/wiki/Affine_transformation">an affine transformation</a>
of the original asset returns $R$:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \bar{\Sigma}^{\frac{1}{2}} \Sigma^{-\frac{1}{2}} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\]
<p>, with:</p>
<ul>
<li>$\bar{\Sigma}^{\frac{1}{2}}$, <a href="https://en.wikipedia.org/wiki/Square_root_of_a_matrix#Positive_semidefinite_matrices">the unique positive semidefinite square root</a> of $\bar{\Sigma}$</li>
<li>$\Sigma^{-\frac{1}{2}}$, the inverse of the unique positive semidefinite square root of $\Sigma$</li>
</ul>
<p>To be noted that if only the average asset returns and/or the asset standard deviations need to be altered - that is, if the asset correlations should be left unchanged -, the generic formula above
becomes:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + Diag \left( \frac{\bar{\sigma}}{\sigma} \right) \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\]
<p>, with</p>
\[Diag \left( \frac{\bar{\sigma}}{\sigma} \right) = \begin{pmatrix}
\frac{\bar{\sigma}_1}{\sigma_1} & 0 & ... & 0 \\
0 & \frac{\bar{\sigma}_2}{\sigma_2} & ... & 0 \\
... & ... & ... & ... \\
0 & 0 & ... & \frac{\bar{\sigma}_n}{\sigma_n}
\end{pmatrix}\]
<p>, which is exactly the one-dimensional procedure described in the previous sub-section, applied independently to each of the $n$ assets.</p>
<h4 id="minimum-correction-second-moment-matching-procedure">Minimum-correction second-moment-matching procedure</h4>
<p>One problem with the multivariate shift and rescaling procedure described in the previous sub-section is that it offers no guarantees regarding the “distance” between the original asset returns and the modified asset returns.</p>
<p>In other words, it is theoretically possible that the modified asset returns obtained through this procedure bear no resemblance at all with the original asset returns,
which would render them useless for the purpose of <em>maintain[ing] important information from [the original asset returns]</em><sup id="fnref:2:5" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>,</p>
<p>To solve this problem, <em>a natural approach is to aim for a minimal correction $\tilde{R} - R$ to avoid introducing numerical artifacts</em><sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup>.</p>
<p>Such an approach to minimally distorting the original asset returns is described in Lin and Lermusiaux<sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> under the name <em>minimum-correction second-moment-matching</em><sup id="fnref:7:2" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup>, with the following main result<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">11</a></sup>:</p>
\[\argmin_{\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n}), \frac{1}{n}\tilde{R}^t R = \bar{\Sigma}} \left\Vert R - \tilde{R} \right\Vert^2_F = R A_*\]
<p>, with $A_*$ defined by</p>
\[A_* = { \bar{\Sigma}^{\frac{1}{2}} }^t \left( { \bar{\Sigma}^{\frac{1}{2}} } \Sigma { \bar{\Sigma}^{\frac{1}{2}} }^t \right)^{-\frac{1}{2}} { \bar{\Sigma}^{\frac{1}{2}} }\]
<p>This leads to the mapping:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right) A_*\]
<h4 id="comparison-of-the-two-procedures">Comparison of the two procedures</h4>
<p>Figure 1 and Figure 2 highlight, over the period 04rd January 2010 - 31th December 2020<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup>, the typical differences between the two procedures described in the previous sub-sections when applied to a universe of 10 ETFs
representative<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">13</a></sup> of misc. asset classes.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-comparison.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-comparison.png" alt="Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 1. Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-tlt-comparison.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-tlt-comparison.png" alt="Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 2. Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>On Figure 1 and Figure 2:</p>
<ul>
<li>The curve in blue corresponds to the original asset returns</li>
<li>The two curves in green and orange correspond to modified asset returns so that the average returns and the standard deviations of the asset returns are left unchanged but their correlations are altered, using
<ul>
<li>The multivariate shift and rescaling procedure (green curve)</li>
<li>The minimum correction procedure (orange curve)</li>
</ul>
</li>
</ul>
<p>From these figures, it is clear that the curve corresponding to the minimum correction procedure is “closer” to the original curve than the curve corresponding to the multivariate shift and rescaling procedure.</p>
<p>This is confirmed numerically, with the “minimally corrected” asset returns being about 10% closer to the original asset returns than the “shifted and scaled” asset returns:</p>
<ul>
<li>$\left\Vert R - \tilde{R}_{shift-rescaling} \right\Vert_F \approx 1.30$</li>
<li>$\left\Vert R - \tilde{R}_{min-correction} \right\Vert_F \approx 1.17$</li>
</ul>
<h3 id="misc-remarks">Misc. remarks</h3>
<p>A couple of remarks on what precedes:</p>
<ul>
<li>
<p>The underlying distribution of the original asset returns - whatever its nature - is usually not preserved by the different moment-matching procedures.</p>
<p>This is discussed in Boyle et al.<sup id="fnref:5:1" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup> for the one-dimensional case and in Kaut and Lium<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> for the general multi-dimensional case.</p>
</li>
<li>
<p>The multivariate formulas for computing $\tilde{R}$ provided in the previous sub-sections are not the most generic ones.</p>
<p>In particular, Kaut and Lium<sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> and Lin and Lermusiaux<sup id="fnref:7:3" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> emphasize that the matrix square roots appearing in those formulas need not be the unique positive semidefinite square roots.</p>
</li>
</ul>
<h2 id="bootstrap-simulations-with-exact-sample-mean-vector-and-sample-covariance-matrix">Bootstrap simulations with exact sample mean vector and sample covariance matrix</h2>
<h3 id="bootstrap-simulations">Bootstrap simulations…</h3>
<p>Carver<sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup> gives a great summary of why bootstrap simulations are so useful in finance:</p>
<blockquote>
<p>Bootstrapping is particularly potent in the field of financial data because we only have one set of data: history. We can’t run experiments to get more data. Bootstrapping allows us to create ‘alternative histories’ that have the same basic character as our actual history, but aren’t quite the same. Apart from generating completely random data […], there isn’t really much else we can do.</p>
</blockquote>
<p>When applied to historical asset returns, bootstrapping creates simulated asset returns that differ from their historical counterparts but that nevertheless
<em>maintain important information gleaned from history</em><sup id="fnref:2:6" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>Now, for various reasons, it might be useful to be able to tweak the first two empirical moments of those simulated asset returns:</p>
<ul>
<li>
<p>To incorporate forward-looking estimates of mean asset returns based on current market valuation indicators like <a href="/blog/the-single-greatest-predictor-of-future-stock-market-returns-ten-years-after/">the U.S. AIAE indicator</a> for U.S. stocks or <a href="/blog/the-bogle-model-for-bonds-predicting-the-returns-of-constant-maturity-government-bond-etfs/">the BSRM/B model</a> for U.S. bonds.</p>
</li>
<li>
<p>To incorporate forward-looking estimates of asset correlations, like an estimate of the future stock-bond correlation based on the framework described in Czasonis et al.<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> or, more recently, in Molenaar et al.<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup></p>
</li>
<li>
<p>To simulate correlation scenarios incompatible with the historical asset returns, like unprecedented correlation breakdowns<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">16</a></sup>.</p>
</li>
</ul>
<h3 id="-with-exact-sample-mean-vector-and-sample-covariance-matrix">… with exact sample mean vector and sample covariance matrix</h3>
<p>For this, one possibility is to use the moment-matching procedures described in the previous section, and in particular the minimum-correction second-moment-matching procedure.</p>
<p>Indeed:</p>
<ul>
<li>
<p>Although these procedures do not preserve the distribution of the simulated asset returns, it is not a problem in the context of bootstrapping because - contrary to <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">a Monte Carlo simulation</a> - there is (usually) no assumption made about that distribution.</p>
</li>
<li>
<p>Additionally, the “minimum alteration” property of the minimum-correction second-moment-matching procedure gives some assurance that, even if the distribution of the simulated asset returns is altered by this procedure, it should be kind of “minimally” altered.</p>
</li>
</ul>
<p>In practice, a moment-matching procedure can be used in two different ways with bootstrap simulations:</p>
<ul>
<li>For tweaking the historical asset returns</li>
<li>For tweaking the simulated asset returns</li>
</ul>
<h4 id="tweaking-the-historical-asset-returns">Tweaking the historical asset returns</h4>
<p>This is the method described for example in Carver<sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup> or in Wu and Walsh<sup id="fnref:2:7" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>It consists in:</p>
<ul>
<li>
<p>Transforming the historical asset returns into modified historical asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix).</p>
</li>
<li>
<p>Bootstrapping these modified historical assets returns in order to simulate asset returns.</p>
</li>
</ul>
<p>Figure 3 illustrates this method applied to a universe of U.S. stocks (SPY ETF) and bonds (TLT ETF) over the period 04rd January 2010 - 31th December 2020<sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup> when the historical correlation of about
$-0.5$ between stocks and bonds is altered to about $0.5$, which is more representative of the pre-2000 period, c.f. Brixton et al.<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">17</a></sup>.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-history.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-history.png" alt="Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 3. Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>One of the advantages of this method is that the computationally costly part - the moment-matching procedure - needs to be performed only once.</p>
<p>Nevertheless, with this method, the simulated asset returns will exhibit a varying sample mean vector and sample covariance matrix, which might be undesirable.</p>
<h4 id="tweaking-the-simulated-asset-returns">Tweaking the simulated asset returns</h4>
<p>This method consists in:</p>
<ul>
<li>
<p>Bootstrapping the historical assets returns in order to simulate asset returns.</p>
</li>
<li>
<p>Transforming the simulated asset returns into modified simulated asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix).</p>
</li>
</ul>
<p>Figure 4 illustrates this method in the same context as in Figure 3.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-simulations.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-simulations.png" alt="Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 4. Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>With this method, the simulated asset returns will exhibit a sample mean vector and a sample covariance matrix that exactly match the desired target mean vector and target covariance matrix.</p>
<p>Unfortunately, one of the drawbacks of this method is that it is computationally heavy, because every time a bootstrap simulation is generated, the moment-matching procedure must be applied to the associated simulated asset
returns.</p>
<h3 id="comparison-with-entropy-pooling">Comparison with entropy pooling</h3>
<p>Readers familiar with a procedure called <em>entropy pooling</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup> - that also allows to <em>express views on features of the market, such as expectation or correlations, regardless of the market distribution</em><sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">19</a></sup> - might wonder about
the relationship between that procedure and the moment-matched bootstrapping procedure described in this section.</p>
<p>Two preliminary remarks:</p>
<ul>
<li>
<p>Bootstrapping, especially its variations designed for time-series, introduces time-dependency in the sampled asset returns.</p>
<p>This makes moment-matched bootstrapping closer in spirit to time-dependent entropy pooling - as described in van der Schans<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup>, which applies the entropy pooling <em>computational approach in a time-dependent setting with sample paths, called scenarios, instead of sample points</em><sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup> - than to vanilla entropy pooling.</p>
</li>
<li>
<p>Time-dependent entropy pooling <em>incorporates the [views] by assigning weights to the scenarios</em><sup id="fnref:27:2" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup> within a scenario-probability setting, while moment-matched bootstrapping operates on one scenario at a time, without any reference to a probability distribution.</p>
</li>
</ul>
<p>With that in mind, the relationship between entropy pooling and moment-matched bootstrapping would be the following:</p>
<ul>
<li>Time-dependent entropy pooling works by <em>tweaking the relative probabilities of the scenarios without affecting the scenarios themselves</em><sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup></li>
<li>Moment-matched bootstrapping works by tweaking the scenarios without affecting the relative probabilities themselves, since in that case, <em>the user associates full probability to one single scenario</em><sup id="fnref:26:2" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup></li>
</ul>
<p>That being said, entropy pooling is a much more generic<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">21</a></sup> framework for imposing views than moment-matched bootstrapping, so that the comparison ends here.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the minimum-correction second-moment-matching procedure described in the previous sections through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/moment-matched</code></a>.</p>
<p>Together with the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/simulation/bootstrap</code></a>, this allows to use any of the two methods described in the previous section.</p>
<h2 id="example-of-application---extending-an-historically-informed-forward-looking-simulation-framework">Example of application - extending an historically informed, forward-looking simulation framework</h2>
<h3 id="t-rowe-prices-simulation-framework">T. Rowe Price’s simulation framework</h3>
<p>Wu and Walsh<sup id="fnref:2:8" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> describes a simulation-based portfolio construction framework that <em>embed both long-term historical asset behaviours and investor expectations of future performance when simulating asset returns</em><sup id="fnref:2:9" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In details, this <em>historical informed</em><sup id="fnref:2:10" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> framework is a three-step process that:</p>
<ul>
<li>
<p>Backfills missing historical asset returns for assets whose return histories differ in length, using the relationships observed between all assets over their common returns history while accounting for the associated estimation error.</p>
<p>As a side note, this step uses the procedure described in a <a href="/blog/managing-missing-asset-returns-in-portfolio-analysis-and-optimization-backfilling-through-residuals-recycling/">previous blog post</a>.</p>
</li>
<li>
<p>Tweaks the resulting historical asset returns, taking into account <em>user-defined return expectations</em><sup id="fnref:2:11" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>This step tweaks the historical asset returns thanks to the univariate shift procedure<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">22</a></sup> described in the previous sections, using T. Rowe Price’s own expected returns as input.</p>
<p>Figure 5, nearly identical<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">23</a></sup> to Figure 8 from Wu and Walsh<sup id="fnref:2:12" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, depicts the result of that procedure when applied to the SPY ETF over the period December 2010 - December 2021 with a target expected return of 4.9% per year<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">24</a></sup>.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-troweprice-reproduction.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-troweprice-reproduction.png" alt="12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021." /></a>
<figcaption>Figure 5. 12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021.</figcaption>
</figure>
</li>
<li>
<p>Simulates asset returns from the modified historical asset returns, ensuring to <em>retain the actual pattern of […] asset movements in each simulated scenario</em><sup id="fnref:2:13" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>This step relies on what seems to be a rolling <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">block-bootstrap</a> in order <em>to recognise extreme tail risk occurrences in actual historical context</em><sup id="fnref:2:14" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
</li>
</ul>
<p>Ultimately, T. Rowe Price’s framework can be used <em>to demonstrate how [a multi-asset portfolio] would have performed in different market conditions reflective of historical experience,
while incorporating the investor’s expectations for the future</em><sup id="fnref:2:15" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<h3 id="incorporating-all-capital-market-assumptions-into-t-rowe-prices-simulation-framework">Incorporating all capital market assumptions into T. Rowe Price’s simulation framework</h3>
<p>I propose to extend T. Rowe Price’s framework to be able to take into account:</p>
<ul>
<li>User-defined return expectations</li>
<li>User-defined standard deviation expectations</li>
<li>User-defined correlation expectations</li>
</ul>
<p>Because these three quantities usually represent the full range of capital market assumptions published by financial institutions<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">25</a></sup>, the resulting simulation framework will become even more powerful while retaining
its original simplicity.</p>
<p>In line with this blog post, this is simply done by integrating the minimum-correction second-moment-matching procedure described in the previous sections into the second step of T. Rowe Price’s framework.</p>
<h2 id="conclusion">Conclusion</h2>
<p>A perfect summary of the technique described in this blog post is given by Carver<sup id="fnref:14:3" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup>:</p>
<blockquote>
<p>[…] it basically allows us to use forward looking estimates for the first two moments (and first co-moment - correlation) of the distribution, whilst using actual data for the higher moments (skew, kurtosis and so on) and co-moments (co-skew, co-kurtosis etc). In a sense it’s sort of a blend of a parameterised monte-carlo and a non parameterised bootstrap.</p>
</blockquote>
<p>For more fun with bootstrap, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://onlinelibrary.wiley.com/doi/10.1111/1467-6419.00170">Esther Ruiz and Lorenzo Pascual. Bootstrapping Financial Time Series. Journal of Economic Surveys, 2002, vol. 16, issue 3, 271-300</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Like historical asset returns or historical factor returns. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.troweprice.com/financial-intermediary/uk/en/thinking/articles/2022/q4/considering-the-past-and-the-future-in-asset-simulation.html">Wu, Walsh, Considering the Past and the Future in Asset Simulation, T. Rowe Price, Investment Insights, November 2022</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:2:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:2:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:2:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:2:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:2:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:2:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:2:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:2:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:2:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a> <a href="#fnref:2:14" class="reversefootnote" role="doc-backlink">↩<sup>15</sup></a> <a href="#fnref:2:15" class="reversefootnote" role="doc-backlink">↩<sup>16</sup></a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://qoppac.blogspot.com/2023/12/portfolio-optimisation-uncertainty.html">Rob Carver, This Blog is Systematic, Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots. Ho, ho, ho</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:14:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>With risk defined in terms of volatility. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Arithmetic, logarithmic… <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://dl.acm.org/doi/10.1145/224401.224465">Boyle P, M Broadie and P Glasserman, 1995, Recent advances in simulation for security pricing, Proceedings of the 1995 Winter Simulation Conference, pages 212–219</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:5:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1415699">Meucci, Attilio, Simulations with Exact Means and Covariances (June 7, 2009)</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://eudml.org/doc/262201">Kaut, Michal, and Lium, Arnt-Gunnar. “Scenario generation with distribution functions and correlations.” Kybernetika 50.6 (2014): 1049-1064</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s00211-021-01178-8">Lin, J., Lermusiaux, P.F.J. Minimum-correction second-moment-matching: theory, algorithms and applications. Numer. Math. 147, 611–650 (2021)</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:7:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:7:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>To be noted that Theorem 2.1 from Lin and Lermusiaux<sup id="fnref:7:4" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> requires a couple of numerical assumptions - that $m \geq n$ and that $R$ has full column rank. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">26</a></sup>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/47/3/67">Megan Czasonis, Mark Kritzman, David Turkington, The Stock-Bond Correlation, The Journal of Portfolio Management, February 2021, 47 (3) 67-76</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4514947">Molenaar, Roderick and Senechal, Edouard and Swinkels, Laurens and Wang, Zhenping, Empirical evidence on the stock-bond correlation (February 9, 2023)</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>On this, see also the blog posts <a href="/blog/correlation-matrix-stress-testing-shrinkage-toward-an-equicorrelation-matrix/">here</a>, <a href="/blog/correlation-matrix-stress-testing-shrinkage-toward-the-lower-and-upper-bounds-of-a-correlation-matrix/">here</a> and <a href="/blog/correlation-matrix-stress-testing-random-perturbations-of-a-correlation-matrix/">here</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/49/4/64">A Changing Stock–Bond Correlation: Drivers and Implications, Alfie Brixton, Jordan Brooks, Pete Hecht, Antti Ilmanen, Thomas Maloney, Nicholas McQuinn, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 64 - 80</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1213325">Meucci, Attilio, Fully Flexible Views: Theory and Practice (August 8, 2008). Fully Flexible Views: Theory and Practice, Risk, Vol. 21, No. 10, pp. 97-102, October 2008</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:26:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2583612">Meucci, Attilio and Nicolosi, Marco, Dynamic Portfolio Management with Views at Multiple Horizons (April 16, 2015). Applied Mathematics and Computation, Volume 274, 1 February 2016, Pages 495-518</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s10614-018-9824-7">van der Schans, M. Entropy Pooling with Discrete Weights in a Time-Dependent Setting. Comput Econ 53, 1633–1647 (2019)</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:27:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>Interestingly though, with (time-dependent) entropy pooling, <em>when scenarios of two variables always move along with each other, we cannot impose a negative correlation</em><sup id="fnref:27:3" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup>, while this situation is not an issue for moment-matched bootstrapping! <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>There is no alteration of asset standard deviations or of correlations described in Wu and Walsh<sup id="fnref:2:16" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, so that the multivariate shift and rescaling procedure actually becomes a simple univariate shift procedure… <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>Returns before shifting in Figure 5 seem slightly higher than their counterparts in Figure 8 from Wu and Walsh<sup id="fnref:2:17" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>; this might be due to different data sources used. As a side note, Figure 8 from Wu and Walsh<sup id="fnref:2:18" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> has inverted legends for the two thick lines. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>Which corresponds to the 2022 T. Rowe Price five-year capital market assumptions / expected returns for U.S. Large Cap<sup id="fnref:25:1" role="doc-noteref"><a href="#fn:25" class="footnote">25</a></sup>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://www.troweprice.com/financial-intermediary/us/en/insights/articles/2022/q1/capital-market-assumptions-five-year-perspective.asset-allocation.html">T. Rowe Price, Capital Market Assumptions Five-Year Perspective</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:25:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Bootstrapping is a statistical method which consists in sampling with replacement from an original data set to compute the distribution of a desired statistic, with plenty of possible variations depending on the exact context (non-dependent data, dependent data…). Because bootstrap methods are not, in general, based on any particular assumption on the distribution of the data, they are well suited for the analysis of [financial] returns1, in which case the original data set typically consists of historical return time series2. Unfortunately, bootstrap simulations have the disadvantage that only previously recorded values can be simulated3, so that they cannot explicitly reflect the investors’ expectations3. In this blog post, inspired by Rob Carver’s piece Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots4, I will describe a way to incorporate exact views on expected returns, standard deviations and correlations into bootstrapping so that the resulting simulations both preserve the salient characteristics of asset returns3 and also match forward-looking risk5 and return assumptions. As an example of application, I will revisit the methodology described in the research note Considering the Past and the Future in Asset Simulation3 from T. Rowe Price and incorporate the full range of capital market assumptions into a systematic modelling process to simulate potential asset returns for portfolio construction3. Mathematical preliminaries Let be: $n$, the number of assets in a universe of assets $T$, the number of time periods $R \in \mathcal{M}(\mathbb{R}^{T \times n})$, with $R_{t,i}$ the return6 of the asset $i=1..n$ for the time period $t=1..T$, representing a scenario (historical, simulated…) for the temporal evolution of the asset returns $\mu \in \mathbb{R}^{n}$, the vector of the arithmetic averages of the asset returns $R$ $\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the covariance matrix of the asset returns $R$, supposed to be invertible to avoid numerical subtleties Moment-matching Let be: $\bar{\mu} \in \mathbb{R}^{n}$ a vector $\bar{\Sigma} \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a positive-definite matrix Suppose that we would like the first two empirical moments of the asset returns $R$ - that is, $\mu$ and $\Sigma$ - to take the values $\bar{\mu}$ and $\bar{\Sigma}$. How to proceed? This problem - called moment-matching by twisting scenarios in Attilio Meucci’s Advanced Risk and Portfolio Management website - consists in transforming the original asset returns $R$ into modified asset returns $\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n})$, so that: $ \frac{1}{T} \sum_{t=1}^{T} \tilde{R}_{t,i} = \bar{\mu}_i $, $i=1..n$ $ \frac{1}{T} \sum_{k=1}^T \left( \tilde{R}_{k,i} - \bar{\mu}_i \right) \left( \tilde{R}_{k,j} - \bar{\mu}_j \right) = \bar{\Sigma}_{i,j} $, $i=1..n$, $j=1..n$ Moment-matching with $n=1$, shift and rescaling procedure When $n=1$, a procedure to compute the modified asset returns $\tilde{R}$ from the original asset returns $R$ is discussed in Boyle et al.7. It involves the following intuitive shift/rescaling8 of the original asset returns: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu} + \frac{\bar{\sigma}}{\sigma} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu \right)\] , with: $\sigma = \sqrt{\Sigma_{1,1}}$, the (sample) standard deviation of the original asset returns $\bar{\sigma} = \sqrt{\bar{\Sigma}_{1,1}}$, the desired (sample) standard deviation of the modified asset returns Moment-matching with $n \geq 2$ Multivariate shift and rescaling procedure When $n \geq 2$, Kaut and Lium9 hints at a multivariate generalization of the procedure described in the previous sub-section, which becomes an affine transformation of the original asset returns $R$: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \bar{\Sigma}^{\frac{1}{2}} \Sigma^{-\frac{1}{2}} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\] , with: $\bar{\Sigma}^{\frac{1}{2}}$, the unique positive semidefinite square root of $\bar{\Sigma}$ $\Sigma^{-\frac{1}{2}}$, the inverse of the unique positive semidefinite square root of $\Sigma$ To be noted that if only the average asset returns and/or the asset standard deviations need to be altered - that is, if the asset correlations should be left unchanged -, the generic formula above becomes: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + Diag \left( \frac{\bar{\sigma}}{\sigma} \right) \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\] , with \[Diag \left( \frac{\bar{\sigma}}{\sigma} \right) = \begin{pmatrix} \frac{\bar{\sigma}_1}{\sigma_1} & 0 & ... & 0 \\ 0 & \frac{\bar{\sigma}_2}{\sigma_2} & ... & 0 \\ ... & ... & ... & ... \\ 0 & 0 & ... & \frac{\bar{\sigma}_n}{\sigma_n} \end{pmatrix}\] , which is exactly the one-dimensional procedure described in the previous sub-section, applied independently to each of the $n$ assets. Minimum-correction second-moment-matching procedure One problem with the multivariate shift and rescaling procedure described in the previous sub-section is that it offers no guarantees regarding the “distance” between the original asset returns and the modified asset returns. In other words, it is theoretically possible that the modified asset returns obtained through this procedure bear no resemblance at all with the original asset returns, which would render them useless for the purpose of maintain[ing] important information from [the original asset returns]3, To solve this problem, a natural approach is to aim for a minimal correction $\tilde{R} - R$ to avoid introducing numerical artifacts10. Such an approach to minimally distorting the original asset returns is described in Lin and Lermusiaux10 under the name minimum-correction second-moment-matching10, with the following main result11: \[\argmin_{\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n}), \frac{1}{n}\tilde{R}^t R = \bar{\Sigma}} \left\Vert R - \tilde{R} \right\Vert^2_F = R A_*\] , with $A_*$ defined by \[A_* = { \bar{\Sigma}^{\frac{1}{2}} }^t \left( { \bar{\Sigma}^{\frac{1}{2}} } \Sigma { \bar{\Sigma}^{\frac{1}{2}} }^t \right)^{-\frac{1}{2}} { \bar{\Sigma}^{\frac{1}{2}} }\] This leads to the mapping: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right) A_*\] Comparison of the two procedures Figure 1 and Figure 2 highlight, over the period 04rd January 2010 - 31th December 202012, the typical differences between the two procedures described in the previous sub-sections when applied to a universe of 10 ETFs representative13 of misc. asset classes. Figure 1. Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020. Figure 2. Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020. On Figure 1 and Figure 2: The curve in blue corresponds to the original asset returns The two curves in green and orange correspond to modified asset returns so that the average returns and the standard deviations of the asset returns are left unchanged but their correlations are altered, using The multivariate shift and rescaling procedure (green curve) The minimum correction procedure (orange curve) From these figures, it is clear that the curve corresponding to the minimum correction procedure is “closer” to the original curve than the curve corresponding to the multivariate shift and rescaling procedure. This is confirmed numerically, with the “minimally corrected” asset returns being about 10% closer to the original asset returns than the “shifted and scaled” asset returns: $\left\Vert R - \tilde{R}_{shift-rescaling} \right\Vert_F \approx 1.30$ $\left\Vert R - \tilde{R}_{min-correction} \right\Vert_F \approx 1.17$ Misc. remarks A couple of remarks on what precedes: The underlying distribution of the original asset returns - whatever its nature - is usually not preserved by the different moment-matching procedures. This is discussed in Boyle et al.7 for the one-dimensional case and in Kaut and Lium9 for the general multi-dimensional case. The multivariate formulas for computing $\tilde{R}$ provided in the previous sub-sections are not the most generic ones. In particular, Kaut and Lium9 and Lin and Lermusiaux10 emphasize that the matrix square roots appearing in those formulas need not be the unique positive semidefinite square roots. Bootstrap simulations with exact sample mean vector and sample covariance matrix Bootstrap simulations… Carver4 gives a great summary of why bootstrap simulations are so useful in finance: Bootstrapping is particularly potent in the field of financial data because we only have one set of data: history. We can’t run experiments to get more data. Bootstrapping allows us to create ‘alternative histories’ that have the same basic character as our actual history, but aren’t quite the same. Apart from generating completely random data […], there isn’t really much else we can do. When applied to historical asset returns, bootstrapping creates simulated asset returns that differ from their historical counterparts but that nevertheless maintain important information gleaned from history3. Now, for various reasons, it might be useful to be able to tweak the first two empirical moments of those simulated asset returns: To incorporate forward-looking estimates of mean asset returns based on current market valuation indicators like the U.S. AIAE indicator for U.S. stocks or the BSRM/B model for U.S. bonds. To incorporate forward-looking estimates of asset correlations, like an estimate of the future stock-bond correlation based on the framework described in Czasonis et al.14 or, more recently, in Molenaar et al.15 To simulate correlation scenarios incompatible with the historical asset returns, like unprecedented correlation breakdowns16. … with exact sample mean vector and sample covariance matrix For this, one possibility is to use the moment-matching procedures described in the previous section, and in particular the minimum-correction second-moment-matching procedure. Indeed: Although these procedures do not preserve the distribution of the simulated asset returns, it is not a problem in the context of bootstrapping because - contrary to a Monte Carlo simulation - there is (usually) no assumption made about that distribution. Additionally, the “minimum alteration” property of the minimum-correction second-moment-matching procedure gives some assurance that, even if the distribution of the simulated asset returns is altered by this procedure, it should be kind of “minimally” altered. In practice, a moment-matching procedure can be used in two different ways with bootstrap simulations: For tweaking the historical asset returns For tweaking the simulated asset returns Tweaking the historical asset returns This is the method described for example in Carver4 or in Wu and Walsh3. It consists in: Transforming the historical asset returns into modified historical asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix). Bootstrapping these modified historical assets returns in order to simulate asset returns. Figure 3 illustrates this method applied to a universe of U.S. stocks (SPY ETF) and bonds (TLT ETF) over the period 04rd January 2010 - 31th December 202012 when the historical correlation of about $-0.5$ between stocks and bonds is altered to about $0.5$, which is more representative of the pre-2000 period, c.f. Brixton et al.17. Figure 3. Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020. One of the advantages of this method is that the computationally costly part - the moment-matching procedure - needs to be performed only once. Nevertheless, with this method, the simulated asset returns will exhibit a varying sample mean vector and sample covariance matrix, which might be undesirable. Tweaking the simulated asset returns This method consists in: Bootstrapping the historical assets returns in order to simulate asset returns. Transforming the simulated asset returns into modified simulated asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix). Figure 4 illustrates this method in the same context as in Figure 3. Figure 4. Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020. With this method, the simulated asset returns will exhibit a sample mean vector and a sample covariance matrix that exactly match the desired target mean vector and target covariance matrix. Unfortunately, one of the drawbacks of this method is that it is computationally heavy, because every time a bootstrap simulation is generated, the moment-matching procedure must be applied to the associated simulated asset returns. Comparison with entropy pooling Readers familiar with a procedure called entropy pooling18 - that also allows to express views on features of the market, such as expectation or correlations, regardless of the market distribution19 - might wonder about the relationship between that procedure and the moment-matched bootstrapping procedure described in this section. Two preliminary remarks: Bootstrapping, especially its variations designed for time-series, introduces time-dependency in the sampled asset returns. This makes moment-matched bootstrapping closer in spirit to time-dependent entropy pooling - as described in van der Schans20, which applies the entropy pooling computational approach in a time-dependent setting with sample paths, called scenarios, instead of sample points20 - than to vanilla entropy pooling. Time-dependent entropy pooling incorporates the [views] by assigning weights to the scenarios20 within a scenario-probability setting, while moment-matched bootstrapping operates on one scenario at a time, without any reference to a probability distribution. With that in mind, the relationship between entropy pooling and moment-matched bootstrapping would be the following: Time-dependent entropy pooling works by tweaking the relative probabilities of the scenarios without affecting the scenarios themselves18 Moment-matched bootstrapping works by tweaking the scenarios without affecting the relative probabilities themselves, since in that case, the user associates full probability to one single scenario18 That being said, entropy pooling is a much more generic21 framework for imposing views than moment-matched bootstrapping, so that the comparison ends here. Implementation in Portfolio Optimizer Portfolio Optimizer implements the minimum-correction second-moment-matching procedure described in the previous sections through the endpoint /assets/returns/moment-matched. Together with the endpoint /assets/returns/simulation/bootstrap, this allows to use any of the two methods described in the previous section. Example of application - extending an historically informed, forward-looking simulation framework T. Rowe Price’s simulation framework Wu and Walsh3 describes a simulation-based portfolio construction framework that embed both long-term historical asset behaviours and investor expectations of future performance when simulating asset returns3. In details, this historical informed3 framework is a three-step process that: Backfills missing historical asset returns for assets whose return histories differ in length, using the relationships observed between all assets over their common returns history while accounting for the associated estimation error. As a side note, this step uses the procedure described in a previous blog post. Tweaks the resulting historical asset returns, taking into account user-defined return expectations3. This step tweaks the historical asset returns thanks to the univariate shift procedure22 described in the previous sections, using T. Rowe Price’s own expected returns as input. Figure 5, nearly identical23 to Figure 8 from Wu and Walsh3, depicts the result of that procedure when applied to the SPY ETF over the period December 2010 - December 2021 with a target expected return of 4.9% per year24. Figure 5. 12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021. Simulates asset returns from the modified historical asset returns, ensuring to retain the actual pattern of […] asset movements in each simulated scenario3. This step relies on what seems to be a rolling block-bootstrap in order to recognise extreme tail risk occurrences in actual historical context3. Ultimately, T. Rowe Price’s framework can be used to demonstrate how [a multi-asset portfolio] would have performed in different market conditions reflective of historical experience, while incorporating the investor’s expectations for the future3. Incorporating all capital market assumptions into T. Rowe Price’s simulation framework I propose to extend T. Rowe Price’s framework to be able to take into account: User-defined return expectations User-defined standard deviation expectations User-defined correlation expectations Because these three quantities usually represent the full range of capital market assumptions published by financial institutions25, the resulting simulation framework will become even more powerful while retaining its original simplicity. In line with this blog post, this is simply done by integrating the minimum-correction second-moment-matching procedure described in the previous sections into the second step of T. Rowe Price’s framework. Conclusion A perfect summary of the technique described in this blog post is given by Carver4: […] it basically allows us to use forward looking estimates for the first two moments (and first co-moment - correlation) of the distribution, whilst using actual data for the higher moments (skew, kurtosis and so on) and co-moments (co-skew, co-kurtosis etc). In a sense it’s sort of a blend of a parameterised monte-carlo and a non parameterised bootstrap. For more fun with bootstrap, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Esther Ruiz and Lorenzo Pascual. Bootstrapping Financial Time Series. Journal of Economic Surveys, 2002, vol. 16, issue 3, 271-300. ↩ Like historical asset returns or historical factor returns. ↩ See Wu, Walsh, Considering the Past and the Future in Asset Simulation, T. Rowe Price, Investment Insights, November 2022. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 See Rob Carver, This Blog is Systematic, Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots. Ho, ho, ho. ↩ ↩2 ↩3 ↩4 With risk defined in terms of volatility. ↩ Arithmetic, logarithmic… ↩ See Boyle P, M Broadie and P Glasserman, 1995, Recent advances in simulation for security pricing, Proceedings of the 1995 Winter Simulation Conference, pages 212–219. ↩ ↩2 See Meucci, Attilio, Simulations with Exact Means and Covariances (June 7, 2009). ↩ See Kaut, Michal, and Lium, Arnt-Gunnar. “Scenario generation with distribution functions and correlations.” Kybernetika 50.6 (2014): 1049-1064. ↩ ↩2 ↩3 See Lin, J., Lermusiaux, P.F.J. Minimum-correction second-moment-matching: theory, algorithms and applications. Numer. Math. 147, 611–650 (2021). ↩ ↩2 ↩3 ↩4 To be noted that Theorem 2.1 from Lin and Lermusiaux10 requires a couple of numerical assumptions - that $m \geq n$ and that $R$ has full column rank. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ ↩2 These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer26. ↩ See Megan Czasonis, Mark Kritzman, David Turkington, The Stock-Bond Correlation, The Journal of Portfolio Management, February 2021, 47 (3) 67-76. ↩ See Molenaar, Roderick and Senechal, Edouard and Swinkels, Laurens and Wang, Zhenping, Empirical evidence on the stock-bond correlation (February 9, 2023). ↩ On this, see also the blog posts here, here and here. ↩ See A Changing Stock–Bond Correlation: Drivers and Implications, Alfie Brixton, Jordan Brooks, Pete Hecht, Antti Ilmanen, Thomas Maloney, Nicholas McQuinn, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 64 - 80. ↩ See Meucci, Attilio, Fully Flexible Views: Theory and Practice (August 8, 2008). Fully Flexible Views: Theory and Practice, Risk, Vol. 21, No. 10, pp. 97-102, October 2008. ↩ ↩2 ↩3 See Meucci, Attilio and Nicolosi, Marco, Dynamic Portfolio Management with Views at Multiple Horizons (April 16, 2015). Applied Mathematics and Computation, Volume 274, 1 February 2016, Pages 495-518. ↩ See van der Schans, M. Entropy Pooling with Discrete Weights in a Time-Dependent Setting. Comput Econ 53, 1633–1647 (2019). ↩ ↩2 ↩3 Interestingly though, with (time-dependent) entropy pooling, when scenarios of two variables always move along with each other, we cannot impose a negative correlation20, while this situation is not an issue for moment-matched bootstrapping! ↩ There is no alteration of asset standard deviations or of correlations described in Wu and Walsh3, so that the multivariate shift and rescaling procedure actually becomes a simple univariate shift procedure… ↩ Returns before shifting in Figure 5 seem slightly higher than their counterparts in Figure 8 from Wu and Walsh3; this might be due to different data sources used. As a side note, Figure 8 from Wu and Walsh3 has inverted legends for the two thick lines. ↩ Which corresponds to the 2022 T. Rowe Price five-year capital market assumptions / expected returns for U.S. Large Cap25. ↩ See T. Rowe Price, Capital Market Assumptions Five-Year Perspective. ↩ ↩2 See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Cluster Risk Parity: Equalizing Risk Contributions Between and Within Asset Classes2024-04-24T00:00:00-05:002024-04-24T00:00:00-05:00https://portfoliooptimizer.io/blog/cluster-risk-parity-equalizing-risk-contributions-between-and-within-asset-classes<p>The <em>equal risk contribution (ERC)</em> portfolio, introduced in Maillard et al.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, is a portfolio aiming to <em>equalize the risk contributions from [its] different components</em><sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>Empirically, the ERC portfolio has been found to <em>be a middle-ground alternative</em><sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> to an equally weighted portfolio and a minimum variance portfolio, <em>balanced in risk and in weights</em><sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, which exhibits
interesting performances in terms of risk-adjusted returns<sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>One issue with the ERC portfolio, though, it that it is <em>highly dependent upon the structure of the universe of assets considered</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> and tends to distribute risk unevenly in terms of asset classes whenever
the number of assets per asset class differs.</p>
<p>In this short blog post, I will describe an extension of the ERC portfolio proposed by Varadi<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> and Kapler<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> designed to solve this problem.</p>
<h2 id="the-equal-risk-contribution-portfolio">The equal risk contribution portfolio</h2>
<p>Let be:</p>
<ul>
<li>$n$, the number of assets in a universe of assets</li>
<li>$w = (w_1,…,w_n) \in {\mathbb{R}^+}^{n}$, the asset weights of a long-only portfolio invested in the considered universe of assets</li>
<li>$\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the asset covariance matrix</li>
<li>$\sigma_{i,j}$, the covariance between assets $i$ and $j$, $i=1..n$, $j=1..n$</li>
<li>$\sigma_1^2,…,\sigma_n^2 \in {\mathbb{R}^+}^{n}$, the asset variances</li>
<li>$\sigma(w) = \sqrt{ w^t \Sigma w }$, the portfolio standard deviation/volatility</li>
</ul>
<h3 id="definitions">Definitions</h3>
<h4 id="risk-contributions">Risk contributions</h4>
<p>The <em>marginal contribution to risk</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup> - or <em>marginal risk contribution</em> - $MCTR_i(w)$ of the asset $i, i=1..n$, to the portfolio risk $\sigma(w)$ is defined as:</p>
\[MCTR_i(w) = \frac{\partial \sigma(w)}{\partial w_i} = \frac{w_i \sigma_{i}^2 + \Sigma_{i=1, i\ne j}^n w_j \sigma_{i,j}}{\sigma(w)}\]
<p>As noted in Maillard et al.<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, <em>the adjective marginal qualifies the fact that those quantities give the change in volatility of the portfolio induced by a small increase in the weight of one component</em><sup id="fnref:1:6" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>,
all other weights staying the same.</p>
<p>The <em>total contribution to risk</em> - or <em>total risk contribution</em> - $TCTR_i(w)$ of the $i$-th asset, $i=1..n$, to the portfolio risk $\sigma(w)$ is defined as:</p>
\[TCTR_i(w) = w_i MCTR_i(w)\]
<h4 id="the-erc-portfolio">The ERC portfolio</h4>
<p>The ERC portfolio is defined as <em>the risk-balanced portfolio such that the total risk contribution is the same for all assets in the portfolio</em><sup id="fnref:1:7" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>More formally, the ERC portfolio is the long-only portfolio whose asset weights $w^* \in [0,1]^{n}$ satisfy:</p>
<ul>
<li>$\sum_{i=1}^{n} w_i = 1$</li>
<li>$TCTR_i(w) = TCTR_j(w)$, for all $i$, $j$, $i=1..n$, $j=1..n$</li>
</ul>
<h3 id="sensitivity-of-the-erc-portfolio-to-the-universe-of-assets">Sensitivity of the ERC portfolio to the universe of assets</h3>
<p>Choueifaty et al.<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> defines <em>a set of basic properties that an unbiased, agnostic portfolio construction process should respect, based on common sense and reasonable economic grounds</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>I propose to illustrate one of these properties, named <em>duplication invariance</em>, by comparing two portfolios invested in U.S. stocks (SPY ETF) and U.S. bonds (TLT ETF):</p>
<ul>
<li>The minimum variance (MV) portfolio</li>
<li>The ERC portfolio</li>
</ul>
<p>Using asset data over the period 03rd January 2023 - 29th December 2023<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, the associated portfolios are displayed in Figure 1:</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-vs-mv.png"><img src="/assets/images/blog/cluster-risk-parity-erc-vs-mv.png" alt="ERC v.s. MV portfolio weights, SPY-TLT universe." /></a>
<figcaption>Figure 1. ERC v.s. MV portfolio weights, SPY-TLT universe.</figcaption>
</figure>
<p>Something interesting to note in Figure 1 is that the ERC portfolio appears closer to an equally weighted portfolio than the MV portfolio, which is a general property discussed in Maillard et al.<sup id="fnref:1:8" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>Now, let’s suppose that the TLT ETF is somehow duplicated into a TLT’ ETF over the same period of time.</p>
<p>What would be the impact of this asset duplication on the above portfolios?</p>
<p>Answer is provided in Figure 2.</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-vs-mv-duplicated.png"><img src="/assets/images/blog/cluster-risk-parity-erc-vs-mv-duplicated.png" alt="Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe." /></a>
<figcaption>Figure 2. Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe.</figcaption>
</figure>
<p>It is visible on Figure 2 that:</p>
<ul>
<li>
<p>The weights of the ERC portfolio in the original assets are (severely) impacted by the duplication of TLT ETF.</p>
<p>The ERC portfolio is said be <em>non duplication invariant</em>.</p>
</li>
<li>
<p>The weights of the MV portfolio in the original assets are left unchanged<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">7</a></sup> by the duplication of the TLT ETF.</p>
<p>The MV portfolio is said to be <em>duplication invariant</em>.</p>
</li>
</ul>
<p>This lack of duplication invariance means that ERC portfolios are generally <em>biased toward assets with multiple representations</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, so that <em>the asset universe is an important factor when considering [ERC] portfolios</em><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">8</a></sup>.</p>
<p>As an example, for a portfolio invested in bonds and equities, <em>the number […] of bond versus equity will dramatically affect the risk contribution of the [ERC] portfolio from each category</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>Indeed, as detailed in Roncalli and Weisang<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">8</a></sup>:</p>
<blockquote>
<p>If the universe includes 5 equity indices and 5 bond indices, then the ERC portfolio will be well balanced between equity and bond in terms of risk. Conversely, if the universe includes 7 equity indices and 3 bond indices, the equity risk of the ERC portfolio represents then 70% of the portfolio’s total risk, a solution very unbalanced between equity and bond risks.</p>
</blockquote>
<p>Figure 3 illustrates this imbalance within the SPY-TLT universe, with the TLT duplicated 5 times (resp. 3 times) and the SPY ETF duplicated 5 times (resp. 7 times).</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-tctr-spy-tlt-duplicated.png"><img src="/assets/images/blog/cluster-risk-parity-tctr-spy-tlt-duplicated.png" alt="Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe." /></a>
<figcaption>Figure 3. Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe.</figcaption>
</figure>
<p>Interestingly, such a behavior is not in violation of the mathematical objective of the ERC portfolio<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">9</a></sup>, but is nevertheless in clear violation of the ERC portfolio <em>philosophy based on diversification</em><sup id="fnref:1:9" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>Asset weights in Figure 1 and Figure 2 might seem surprising at first sight (higher weights for the SPY ETF v.s. the TLT ETF), but 2023 has been a particular year for U.S. Treasuries… For reference, the exact (co)variances of the SPY and TLT ETFs are provided in Figure 5.</li>
<li>For more discussion about duplication invariance, see for example Gava and Turc<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">10</a></sup>, which introduces a quantitative measure of the sensitivity of a portfolio to duplication.</li>
</ul>
</blockquote>
<h2 id="the-cluster-risk-parity-portfolio">The cluster risk parity portfolio</h2>
<h3 id="description">Description</h3>
<p>One possible solution to the problem of the sensitivity of the ERC portfolio discussed in the previous section is a careful selection of the universe of assets.</p>
<p>Another possible solution is to slightly alter the ERC portfolio construction process by taking into account both the asset and the asset class level, which is the solution proposed by Varadi<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> and Kapler<sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>
under the name <em>cluster risk parity (CRP)</em> portfolio.</p>
<p>In details, the CRP portfolio is computed through a three-step process, detailed below.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>Another cluster risk parity portfolio is described in the <a href="https://quantpedia.com/">Quantpedia</a> series <em><a href="https://quantpedia.com/introduction-to-clustering-methods-in-portfolio-management-part-3/">Introduction to Clustering Methods In Portfolio Management</a></em>, but the two portfolio construction methodologies are not the same.</li>
</ul>
</blockquote>
<h4 id="step-1---partitioning-the-universe-of-assets-into-clusters">Step 1 - Partitioning the universe of assets into clusters</h4>
<p>The first step consists in using a <a href="https://en.wikipedia.org/wiki/Cluster_analysis">clustering algorithm</a> to group similar assets together within the considered universe of assets.</p>
<p>This results in $k$ clusters<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">11</a></sup> $\mathcal{G}_1,…,\mathcal{G}_k$, $1 \leq k \leq n$, each corresponding to an asset class automatically determined by the clustering algorithm.</p>
<p>The purpose of this step is <em>to avoid the need for artificial or manual grouping</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> of the assets and enables <em>the [CRP] portfolio to adapt to changes […] without having to run a lot of ad hoc analysis
and having to make continual adjustments</em><sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>Using the same <a href="https://en.wikipedia.org/wiki/Hierarchical_clustering">hierarchical clustering algorithm</a> as the one used in the
<em><a href="/blog/beyond-hierarchical-risk-parity-hierarchical-clustering-based-risk-parity">Hierarchical Clustering-Based Risk Parity</a></em> algorithm and using asset data over the period 03rd January 2023 - 29th December 2023<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>,
Figure 4 illustrates this first step for a universe of 10 ETFs representative<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">12</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-hierarchical-clustering.png"><img src="/assets/images/blog/cluster-risk-parity-hierarchical-clustering.png" alt="Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe." /></a>
<figcaption>Figure 4. Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe.</figcaption>
</figure>
<p>Figure 4 shows that 3 clusters/asset classes have been automatically determined by the hierarchical clustering algorithm over the studied period, materialized by the thick black dendrogram cut:</p>
<ul>
<li>An equity cluster $\mathcal{G}_1$ made of the equity ETFs SPY, EZU, EWJ, EEM and of the real estate ETFs VNQ, RWX</li>
<li>A bond cluster $\mathcal{G}_2$ made of the bond ETFs IEF, TLT</li>
<li>A commodity cluster $\mathcal{G}_3$ made of the commodity ETF DBC and of the gold ETF GLD</li>
</ul>
<h4 id="step-2---assets-weights-computation-within-each-cluster">Step 2 - Assets weights computation within each cluster</h4>
<p>The second step consists in computing the ERC portfolio within each cluster $\mathcal{G}_i$, $i=1..k$.</p>
<p>This results in $k$ vectors of asset weights $w_{i}^* \in [0,1]^{n}$, $i=1..k$.</p>
<p>Using as input the covariance matrix $\Sigma$ provided in Figure 5<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">13</a></sup>, the ERC portfolio computed within each cluster $\mathcal{G}_1$, $\mathcal{G}_2$ and $\mathcal{G}_3$ defined
in the previous sub-section is displayed in Figure 6.</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-asset-covariance-matrix.png"><img src="/assets/images/blog/cluster-risk-parity-asset-covariance-matrix.png" alt="Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe." /></a>
<figcaption>Figure 5. Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-portfolio-within-clusters.png"><img src="/assets/images/blog/cluster-risk-parity-erc-portfolio-within-clusters.png" alt="Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe." /></a>
<figcaption>Figure 6. Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe.</figcaption>
</figure>
<h4 id="step-3---assets-weights-computation-across-all-clusters-and-final-assets-weights-computation">Step 3 - Assets weights computation across all clusters and final assets weights computation</h4>
<p>The third and last step consists in computing the ERC portfolio across all clusters $\mathcal{G}_i$, $i=1..k$.</p>
<p>For this:</p>
<ul>
<li>
<p>Because the $k$ clusters $\mathcal{G}_1,…,\mathcal{G}_k$ correspond to $k$ risk factors, whose loadings matrix $A \in \mathcal{M}(\mathbb{R}^{n \times k})$ - in the original asset weights - is $ A = \begin{pmatrix} w_1^* & w_2^* & … & w_k^* \end{pmatrix} $, the cluster covariance matrix $\Sigma_c \in \mathcal{M}(\mathbb{R}^{k \times k})$ is equal<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">14</a></sup> to $ A^t \Sigma A $.</p>
</li>
<li>
<p>Thanks to that cluster covariance matrix, it is possible to compute the ERC portfolio across all clusters.</p>
<p>This results in a vector of cluster weights $w_c^* \in [0,1]^{k}$.</p>
</li>
<li>
<p>Then, all remaining is to convert the vector of cluster weights $w_c^*$ into the final vector of asset weights $w^* \in [0,1]^n$ through the formula $w^* = A w_c^*$.</p>
</li>
</ul>
<p>Continuing the example from the previous sub-section:</p>
<ul>
<li>
<p>The loading matrix $A$, deduced from Figure 6, is:</p>
\[A = \begin{pmatrix}
0.19 & 0 & 0 \\
0.16 & 0 & 0 \\
0.19 & 0 & 0 \\
0.17 & 0 & 0 \\
0.13 & 0 & 0 \\
0.16 & 0 & 0 \\
0 & 0.66 & 0 \\
0 & 0.34 & 0 \\
0 & 0 & 0.48 \\
0 & 0 & 0.52 \\
\end{pmatrix}\]
</li>
<li>
<p>The cluster covariance matrix $\Sigma_c$ is:</p>
\[\Sigma_c = A^t \Sigma A = \begin{pmatrix}
0.000063 & 0.000014 & 0.000016 \\
0.000014 & 0.000053 & 0.000011 \\
0.000016 & 0.000011 & 0.000047 \\
\end{pmatrix}\]
</li>
<li>
<p>The vector of cluster weights of the across-cluster ERC portfolio is:</p>
\[w_c^* = \begin{pmatrix} 0.30 \\ 0.34 \\ 0.36 \end{pmatrix}\]
</li>
<li>
<p>The vector of asset weights of the CRP portfolio, $w^* = A w_c^*$, is displayed in Figure 7:</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-crp-portfolio.png"><img src="/assets/images/blog/cluster-risk-parity-crp-portfolio.png" alt="Example of CRP portfolio computation, final portfolio weights, 10-ETF universe." /></a>
<figcaption>Figure 7. Example of CRP portfolio computation, final portfolio weights, 10-ETF universe.</figcaption>
</figure>
</li>
</ul>
<h3 id="practical-performances">Practical performances</h3>
<p>The CRP portfolio has been empirically benchmarked against several other portfolios within different multi-asset classes universes<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">15</a></sup> and its performances, especially in terms of risk-adjusted returns,
have been found to be excellent.</p>
<p>On this aspect, Varadi<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> concludes that:</p>
<blockquote>
<p>The [CRP portfolio] is perhaps the most robust method of passive portfolio allocation, and it also produces the best risk-adjusted returns without relying as much on the low-volatility factor or bond/fixed income performance.</p>
</blockquote>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> allows to easily compute the CRP portfolio through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/optimization/equal-risk-contributions/clustering-based</code></a>, with optional support
for minimum and maximum asset weights constraints.</p>
<p>For maximum flexibility, the partitioning of the considered universe of assets into clusters is expected to be provided in input to that endpoint, which can be done using either:</p>
<ul>
<li>Any clustering algorithm available on the user side</li>
<li>Any clustering algorithm provided by <strong>Portfolio Optimizer</strong>:
<ul>
<li>The <em><a href="https://cssanalytics.wordpress.com/2013/11/26/fast-threshold-clustering-algorithm-ftca/">Fast Threshold Clustering Algorithm (FTCA)</a></em>, available through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/clustering/ftca</code></a></li>
<li>The hierarchical clustering algorithm used in the <a href="/blog/beyond-hierarchical-risk-parity-hierarchical-clustering-based-risk-parity">Hierarchical Clustering-Based Risk Parity</a> algorithm, available through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/clustering/hierarchical</code></a></li>
</ul>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>This concludes the description of the CRP portfolio, a little known but <em>excellent framework to consider for a robust risk parity approach</em><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">15</a></sup>.</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://doi.org/10.3905/jpm.2010.36.4.060">Maillard, S., Roncalli, T., Teiletche, J.: The properties of equally weighted risk contribution portfolios. J. Portf. Manag. 36, 60–70 (2010)</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:1:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:1:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:1:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:1:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.risk.net/journal-of-investment-strategies/2255764/properties-of-the-most-diversified-portfolio">Choueifaty, Y., Tristan Froidure, T., Reynier, J. Properties of the Most Diversified Portfolio. Journal of Investment Strategies, Vol.2(2), Spring 2013, pp.49-70</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://cssanalytics.wordpress.com/2013/01/03/cluster-risk-parity/">David Varadi, CSSA, Cluster Risk Parity</a> and <a href="https://cssanalytics.wordpress.com/2013/01/06/cluster-risk-parity-crp-versus-risk-parity-rp-and-equal-risk-contribution-erc/">David Varadi, CSSA, Cluster Risk Parity (CRP) versus Risk Parity (RP) and Equal Risk Contribution (ERC)</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://systematicinvestor.wordpress.com/2013/02/12/cluster-portfolio-allocation/">Michael Kapler, Systematic Investor Blog, Cluster Portfolio Allocation</a> and <a href="https://systematicinvestor.wordpress.com/2013/03/05/cluster-risk-parity-back-test/">Michael Kapler, Systematic Investor Blog, Cluster Risk Parity back-test</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>When the risk measure is defined as the portfolio volatility; other risk measures can be used instead, c.f. Maillard et al.<sup id="fnref:1:10" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>This property is established in all generality in Maillard et al.<sup id="fnref:1:11" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2155159">Roncalli, Thierry and Weisang, Guillaume, Risk Parity Portfolios with Risk Factors</a> and <a href="https://www.tandfonline.com/doi/full/10.1080/14697688.2015.1046907">Roncalli & G. Weisang, Risk parity portfolios with risk factors, Quantitative Finance</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Per construction of the ERC portfolio, ll assets present in that portfolio are guaranteed to have the same total risk contributions. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.mdpi.com/1099-4300/24/11/1631">Gava J, Turc J. The Properties of Alpha Risk Parity Portfolios. Entropy. 2022; 24(11):1631</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Assumed to form <a href="https://en.wikipedia.org/wiki/Partition_of_a_set">a partition</a> of the universe of assets. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">16</a></sup>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>This covariance matrix is the empirical asset covariance matrix computed from the arithmetic returns of the 10 assets over the period 04th January 2023 - 29th December 2023. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>This is a direct application of the formula for the covariance matrix of a linear transformation of a random vector, c.f. for example <a href="https://en.wikipedia.org/wiki/Covariance_matrix">Wikipedia, Covariance matrix</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://investresolve.com/dynamic-asset-allocation-for-practitioners-part-5-robust-risk-parity/">ReSolve Asset Management, Dynamic asset allocation for practitioners part 5: robust risk parity</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.The equal risk contribution (ERC) portfolio, introduced in Maillard et al.1, is a portfolio aiming to equalize the risk contributions from [its] different components1. Empirically, the ERC portfolio has been found to be a middle-ground alternative1 to an equally weighted portfolio and a minimum variance portfolio, balanced in risk and in weights1, which exhibits interesting performances in terms of risk-adjusted returns1. One issue with the ERC portfolio, though, it that it is highly dependent upon the structure of the universe of assets considered2 and tends to distribute risk unevenly in terms of asset classes whenever the number of assets per asset class differs. In this short blog post, I will describe an extension of the ERC portfolio proposed by Varadi3 and Kapler4 designed to solve this problem. The equal risk contribution portfolio Let be: $n$, the number of assets in a universe of assets $w = (w_1,…,w_n) \in {\mathbb{R}^+}^{n}$, the asset weights of a long-only portfolio invested in the considered universe of assets $\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the asset covariance matrix $\sigma_{i,j}$, the covariance between assets $i$ and $j$, $i=1..n$, $j=1..n$ $\sigma_1^2,…,\sigma_n^2 \in {\mathbb{R}^+}^{n}$, the asset variances $\sigma(w) = \sqrt{ w^t \Sigma w }$, the portfolio standard deviation/volatility Definitions Risk contributions The marginal contribution to risk5 - or marginal risk contribution - $MCTR_i(w)$ of the asset $i, i=1..n$, to the portfolio risk $\sigma(w)$ is defined as: \[MCTR_i(w) = \frac{\partial \sigma(w)}{\partial w_i} = \frac{w_i \sigma_{i}^2 + \Sigma_{i=1, i\ne j}^n w_j \sigma_{i,j}}{\sigma(w)}\] As noted in Maillard et al.1, the adjective marginal qualifies the fact that those quantities give the change in volatility of the portfolio induced by a small increase in the weight of one component1, all other weights staying the same. The total contribution to risk - or total risk contribution - $TCTR_i(w)$ of the $i$-th asset, $i=1..n$, to the portfolio risk $\sigma(w)$ is defined as: \[TCTR_i(w) = w_i MCTR_i(w)\] The ERC portfolio The ERC portfolio is defined as the risk-balanced portfolio such that the total risk contribution is the same for all assets in the portfolio1. More formally, the ERC portfolio is the long-only portfolio whose asset weights $w^* \in [0,1]^{n}$ satisfy: $\sum_{i=1}^{n} w_i = 1$ $TCTR_i(w) = TCTR_j(w)$, for all $i$, $j$, $i=1..n$, $j=1..n$ Sensitivity of the ERC portfolio to the universe of assets Choueifaty et al.2 defines a set of basic properties that an unbiased, agnostic portfolio construction process should respect, based on common sense and reasonable economic grounds2. I propose to illustrate one of these properties, named duplication invariance, by comparing two portfolios invested in U.S. stocks (SPY ETF) and U.S. bonds (TLT ETF): The minimum variance (MV) portfolio The ERC portfolio Using asset data over the period 03rd January 2023 - 29th December 20236, the associated portfolios are displayed in Figure 1: Figure 1. ERC v.s. MV portfolio weights, SPY-TLT universe. Something interesting to note in Figure 1 is that the ERC portfolio appears closer to an equally weighted portfolio than the MV portfolio, which is a general property discussed in Maillard et al.1. Now, let’s suppose that the TLT ETF is somehow duplicated into a TLT’ ETF over the same period of time. What would be the impact of this asset duplication on the above portfolios? Answer is provided in Figure 2. Figure 2. Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe. It is visible on Figure 2 that: The weights of the ERC portfolio in the original assets are (severely) impacted by the duplication of TLT ETF. The ERC portfolio is said be non duplication invariant. The weights of the MV portfolio in the original assets are left unchanged7 by the duplication of the TLT ETF. The MV portfolio is said to be duplication invariant. This lack of duplication invariance means that ERC portfolios are generally biased toward assets with multiple representations2, so that the asset universe is an important factor when considering [ERC] portfolios8. As an example, for a portfolio invested in bonds and equities, the number […] of bond versus equity will dramatically affect the risk contribution of the [ERC] portfolio from each category3. Indeed, as detailed in Roncalli and Weisang8: If the universe includes 5 equity indices and 5 bond indices, then the ERC portfolio will be well balanced between equity and bond in terms of risk. Conversely, if the universe includes 7 equity indices and 3 bond indices, the equity risk of the ERC portfolio represents then 70% of the portfolio’s total risk, a solution very unbalanced between equity and bond risks. Figure 3 illustrates this imbalance within the SPY-TLT universe, with the TLT duplicated 5 times (resp. 3 times) and the SPY ETF duplicated 5 times (resp. 7 times). Figure 3. Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe. Interestingly, such a behavior is not in violation of the mathematical objective of the ERC portfolio9, but is nevertheless in clear violation of the ERC portfolio philosophy based on diversification1. Notes: Asset weights in Figure 1 and Figure 2 might seem surprising at first sight (higher weights for the SPY ETF v.s. the TLT ETF), but 2023 has been a particular year for U.S. Treasuries… For reference, the exact (co)variances of the SPY and TLT ETFs are provided in Figure 5. For more discussion about duplication invariance, see for example Gava and Turc10, which introduces a quantitative measure of the sensitivity of a portfolio to duplication. The cluster risk parity portfolio Description One possible solution to the problem of the sensitivity of the ERC portfolio discussed in the previous section is a careful selection of the universe of assets. Another possible solution is to slightly alter the ERC portfolio construction process by taking into account both the asset and the asset class level, which is the solution proposed by Varadi3 and Kapler4 under the name cluster risk parity (CRP) portfolio. In details, the CRP portfolio is computed through a three-step process, detailed below. Notes: Another cluster risk parity portfolio is described in the Quantpedia series Introduction to Clustering Methods In Portfolio Management, but the two portfolio construction methodologies are not the same. Step 1 - Partitioning the universe of assets into clusters The first step consists in using a clustering algorithm to group similar assets together within the considered universe of assets. This results in $k$ clusters11 $\mathcal{G}_1,…,\mathcal{G}_k$, $1 \leq k \leq n$, each corresponding to an asset class automatically determined by the clustering algorithm. The purpose of this step is to avoid the need for artificial or manual grouping3 of the assets and enables the [CRP] portfolio to adapt to changes […] without having to run a lot of ad hoc analysis and having to make continual adjustments3. Using the same hierarchical clustering algorithm as the one used in the Hierarchical Clustering-Based Risk Parity algorithm and using asset data over the period 03rd January 2023 - 29th December 20236, Figure 4 illustrates this first step for a universe of 10 ETFs representative12 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Figure 4. Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe. Figure 4 shows that 3 clusters/asset classes have been automatically determined by the hierarchical clustering algorithm over the studied period, materialized by the thick black dendrogram cut: An equity cluster $\mathcal{G}_1$ made of the equity ETFs SPY, EZU, EWJ, EEM and of the real estate ETFs VNQ, RWX A bond cluster $\mathcal{G}_2$ made of the bond ETFs IEF, TLT A commodity cluster $\mathcal{G}_3$ made of the commodity ETF DBC and of the gold ETF GLD Step 2 - Assets weights computation within each cluster The second step consists in computing the ERC portfolio within each cluster $\mathcal{G}_i$, $i=1..k$. This results in $k$ vectors of asset weights $w_{i}^* \in [0,1]^{n}$, $i=1..k$. Using as input the covariance matrix $\Sigma$ provided in Figure 513, the ERC portfolio computed within each cluster $\mathcal{G}_1$, $\mathcal{G}_2$ and $\mathcal{G}_3$ defined in the previous sub-section is displayed in Figure 6. Figure 5. Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe. Figure 6. Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe. Step 3 - Assets weights computation across all clusters and final assets weights computation The third and last step consists in computing the ERC portfolio across all clusters $\mathcal{G}_i$, $i=1..k$. For this: Because the $k$ clusters $\mathcal{G}_1,…,\mathcal{G}_k$ correspond to $k$ risk factors, whose loadings matrix $A \in \mathcal{M}(\mathbb{R}^{n \times k})$ - in the original asset weights - is $ A = \begin{pmatrix} w_1^* & w_2^* & … & w_k^* \end{pmatrix} $, the cluster covariance matrix $\Sigma_c \in \mathcal{M}(\mathbb{R}^{k \times k})$ is equal14 to $ A^t \Sigma A $. Thanks to that cluster covariance matrix, it is possible to compute the ERC portfolio across all clusters. This results in a vector of cluster weights $w_c^* \in [0,1]^{k}$. Then, all remaining is to convert the vector of cluster weights $w_c^*$ into the final vector of asset weights $w^* \in [0,1]^n$ through the formula $w^* = A w_c^*$. Continuing the example from the previous sub-section: The loading matrix $A$, deduced from Figure 6, is: \[A = \begin{pmatrix} 0.19 & 0 & 0 \\ 0.16 & 0 & 0 \\ 0.19 & 0 & 0 \\ 0.17 & 0 & 0 \\ 0.13 & 0 & 0 \\ 0.16 & 0 & 0 \\ 0 & 0.66 & 0 \\ 0 & 0.34 & 0 \\ 0 & 0 & 0.48 \\ 0 & 0 & 0.52 \\ \end{pmatrix}\] The cluster covariance matrix $\Sigma_c$ is: \[\Sigma_c = A^t \Sigma A = \begin{pmatrix} 0.000063 & 0.000014 & 0.000016 \\ 0.000014 & 0.000053 & 0.000011 \\ 0.000016 & 0.000011 & 0.000047 \\ \end{pmatrix}\] The vector of cluster weights of the across-cluster ERC portfolio is: \[w_c^* = \begin{pmatrix} 0.30 \\ 0.34 \\ 0.36 \end{pmatrix}\] The vector of asset weights of the CRP portfolio, $w^* = A w_c^*$, is displayed in Figure 7: Figure 7. Example of CRP portfolio computation, final portfolio weights, 10-ETF universe. Practical performances The CRP portfolio has been empirically benchmarked against several other portfolios within different multi-asset classes universes415 and its performances, especially in terms of risk-adjusted returns, have been found to be excellent. On this aspect, Varadi3 concludes that: The [CRP portfolio] is perhaps the most robust method of passive portfolio allocation, and it also produces the best risk-adjusted returns without relying as much on the low-volatility factor or bond/fixed income performance. Implementation in Portfolio Optimizer Portfolio Optimizer allows to easily compute the CRP portfolio through the endpoint /portfolios/optimization/equal-risk-contributions/clustering-based, with optional support for minimum and maximum asset weights constraints. For maximum flexibility, the partitioning of the considered universe of assets into clusters is expected to be provided in input to that endpoint, which can be done using either: Any clustering algorithm available on the user side Any clustering algorithm provided by Portfolio Optimizer: The Fast Threshold Clustering Algorithm (FTCA), available through the endpoint /assets/clustering/ftca The hierarchical clustering algorithm used in the Hierarchical Clustering-Based Risk Parity algorithm, available through the endpoint /assets/clustering/hierarchical Conclusion This concludes the description of the CRP portfolio, a little known but excellent framework to consider for a robust risk parity approach15. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Maillard, S., Roncalli, T., Teiletche, J.: The properties of equally weighted risk contribution portfolios. J. Portf. Manag. 36, 60–70 (2010). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 See Choueifaty, Y., Tristan Froidure, T., Reynier, J. Properties of the Most Diversified Portfolio. Journal of Investment Strategies, Vol.2(2), Spring 2013, pp.49-70. ↩ ↩2 ↩3 ↩4 See David Varadi, CSSA, Cluster Risk Parity and David Varadi, CSSA, Cluster Risk Parity (CRP) versus Risk Parity (RP) and Equal Risk Contribution (ERC). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 See Michael Kapler, Systematic Investor Blog, Cluster Portfolio Allocation and Michael Kapler, Systematic Investor Blog, Cluster Risk Parity back-test. ↩ ↩2 ↩3 When the risk measure is defined as the portfolio volatility; other risk measures can be used instead, c.f. Maillard et al.1. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ ↩2 This property is established in all generality in Maillard et al.1. ↩ See Roncalli, Thierry and Weisang, Guillaume, Risk Parity Portfolios with Risk Factors and Roncalli & G. Weisang, Risk parity portfolios with risk factors, Quantitative Finance. ↩ ↩2 Per construction of the ERC portfolio, ll assets present in that portfolio are guaranteed to have the same total risk contributions. ↩ See Gava J, Turc J. The Properties of Alpha Risk Parity Portfolios. Entropy. 2022; 24(11):1631. ↩ Assumed to form a partition of the universe of assets. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer16. ↩ This covariance matrix is the empirical asset covariance matrix computed from the arithmetic returns of the 10 assets over the period 04th January 2023 - 29th December 2023. ↩ This is a direct application of the formula for the covariance matrix of a linear transformation of a random vector, c.f. for example Wikipedia, Covariance matrix. ↩ See ReSolve Asset Management, Dynamic asset allocation for practitioners part 5: robust risk parity. ↩ ↩2 See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Volatility Forecasting: GARCH(1,1) Model2024-03-11T00:00:00-05:002024-03-11T00:00:00-05:00https://portfoliooptimizer.io/blog/volatility-forecasting-garch11-model<p>In the <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous post</a> of this series on volatility forecasting, I described the simple and the exponentially
weighted moving average volatility forecasting models.</p>
<p>In particular, I showed that these two models belong to the generic family of weighted moving average volatility forecasting models<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup>, whose members
represent the volatility of an asset as a <a href="https://en.wikipedia.org/wiki/Moving_average">weighted moving average</a> of its past squared returns<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>.</p>
<p>Another member of this family is the <em><a href="https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity">Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) model</a></em>,
<em>widely used in financial time series modelling and implemented in most statistics and econometric software packages</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In this blog post, I will detail the <em>simplest but often very useful</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">4</a></sup> GARCH(1,1) volatility forecasting model and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs.</p>
<h2 id="mathematical-preliminaries-reminders">Mathematical preliminaries (reminders)</h2>
<p>This section contains reminders from a <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a>.</p>
<h3 id="volatility-modelling-and-volatility-proxies">Volatility modelling and volatility proxies</h3>
<p>Let $r_t$ be the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>) return of an asset over a time period $t$ (a day, a week, a month..),
over which its (conditional) mean return is supposed to be null.</p>
<p>Then:</p>
<ul>
<li>
<p>The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $</p>
<p>From this definition, the squared return $r_t^2$ of an asset is a (noisy<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">5</a></sup>) <em>variance estimator</em> - or <em>variance proxy</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">5</a></sup> - for that asset variance over the considered time period.</p>
<p>Another example of an asset variance proxy is <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">the Parkinson range</a> of an asset.</p>
<p>The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$.</p>
</li>
<li>
<p>The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $</p>
<p>The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$.</p>
</li>
</ul>
<h3 id="weighted-moving-average-volatility-forecasting-model">Weighted moving average volatility forecasting model</h3>
<p>Boudoukh et al.<sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup> shows that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as
a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with</p>
\[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\]
<p>, where:</p>
<ul>
<li>$1 \leq k \leq T$ is the size of the moving average, possibly time-dependent</li>
<li>$w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well</li>
</ul>
<h2 id="garch11-volatility-forecasting-model">GARCH(1,1) volatility forecasting model</h2>
<h3 id="the-garchpq-model">The GARCH(p,q) model</h3>
<h4 id="definition">Definition</h4>
<p>Bollerslev<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">4</a></sup>’s GARCH model is a generalization of Engle’s <a href="https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity">ARCH</a> <a href="https://en.wikipedia.org/wiki/Econometric_model">econometric model</a>
which captures the time-varying nature of the (conditional) variance of certain time series like asset returns.</p>
<p>Under a GARCH(p,q) model, an asset next period’s conditional variance $\sigma_{T+1}^2$ is modeled as recursive linear function of its own $p$ lagged conditional variances $\sigma_{T}^2, \sigma_{T-1}^2…$ and of its $q$
lagged squared returns $r_{T}^2, r_{T-1}^2…$, which leads to the formula</p>
\[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j r_{T+1-i}^2\]
<p>, where:</p>
<ul>
<li>The parameters $\omega$, $\alpha_j$, $j=1..q$ and $\beta_i$, $i=1..p$ are non-negative and subject to various inequality constraints depending on working assumptions<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">6</a></sup></li>
<li>The initial conditional variance $\hat{\sigma}_1^2$ is usually taken equal to $r_1^2$, but c.f. Pelagatti and Lisi<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">7</a></sup> for a thorough discussion about this subject</li>
</ul>
<h4 id="squared-returns-vs-generic-variance-proxy">Squared returns v.s. generic variance proxy</h4>
<p>Molnar<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> notes that <em>in GARCH type of models, demeaned squared returns serve as a way to calculate innovations to the volatility</em><sup id="fnref:9:1" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> so that <em>replacing the squared returns by more
precise volatility estimates will produce better GARCH models, regarding both in-sample fit and out-of-sample forecasting performance</em><sup id="fnref:9:2" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<p>Molnar<sup id="fnref:9:3" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> then proposes to modify the GARCH(p,q) model for the estimation of an asset next period’s conditional variance $\sigma_{T+1}^2$ as follows</p>
\[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j \tilde{\sigma}_{T+1-i}^2\]
<p>, where $\tilde{\sigma}^2_t$, $t=1..T$ are the asset past periods’ variance proxies.</p>
<p>To be noted that replacing squared returns by less noisy variance proxies is already discussed at length in the
<a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a> in the case of the simple and the exponentially weighted moving average volatility forecasting models.</p>
<h3 id="the-garch11-model">The GARCH(1,1) model</h3>
<h4 id="definition-1">Definition</h4>
<p>Because the GARCH(1,1) model <em>works surprisingly well in comparison with much more complex [GARCH] models</em><sup id="fnref:9:4" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>, it is usually the main GARCH model used in practice.</p>
<p>Under this model, the generic GARCH formula for the estimate of an asset next period’s conditional variance can be re-parametrized as follows</p>
\[\hat{\sigma}_{T+1}^2 = \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2\]
<p>, where:</p>
<ul>
<li>$\alpha$, $\beta$ and $\gamma$ are positive parameters summing to one</li>
<li>$\tilde{\sigma}^2$ is a strictly positive parameter, corresponding to the asset unconditional variance<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup></li>
</ul>
<p>The GARCH(1,1) model thus estimates an asset next period’s conditional variance $\hat{\sigma}_{T+1}^2$ as a weighted average<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">10</a></sup> of three different variance estimators:</p>
<ul>
<li>A long-term variance estimator $\tilde{\sigma}^2$</li>
<li>A short-term variance estimator $\tilde{\sigma}^2_{T}$</li>
<li>The current GARCH(1,1) variance estimator $\hat{\sigma}_{T}^2$</li>
</ul>
<p>and the weights $\alpha$, $\beta$ and $\gamma$ determine the speed with which the model adapts to short-term variance v.s. reverts to its long-term variance.</p>
<h4 id="relationship-with-the-generic-weighted-moving-average-model">Relationship with the generic weighted moving average model</h4>
<p>By developing the recursive definition of the GARCH(1,1) model, it is possible to see that it is a specific kind of weighted moving average volatility forecasting model, with:</p>
<ul>
<li>$k = T$</li>
<li>$w_0 = \gamma \sum_{k=0}^{T-1} \beta^k$</li>
<li>$w_1 = \alpha$, $w_2 = \alpha \beta$, …, $w_{T-1} = \alpha \beta^{T-2}$, $w_T = \alpha \beta^{T-1}$, that is, exponentially decreasing weights emphasizing recent past variance proxies v.s. more distant ones in the model, exactly like in the exponentially weighted moving average volatility forecasting model<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup></li>
</ul>
<h4 id="volatility-forecasting-formulas">Volatility forecasting formulas</h4>
<p>Under a GARCH(1,1) volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes:</p>
<ul>
<li>
<p>To estimate an asset next period’s volatility:</p>
\[\hat{\sigma}_{T+1} = \sqrt{ \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2 }\]
</li>
<li>
<p>To estimate an asset next $h$-period’s ahead volatility<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">12</a></sup>, $h \geq 2$:</p>
\[\hat{\sigma}_{T+h} = \sqrt{ \tilde{\sigma}^2 + \left( \alpha + \beta \right)^{h-1} \left( \hat{\sigma}_{T+1} - \tilde{\sigma}^2 \right) }\]
</li>
<li>
<p>To estimate an asset aggregated volatility<sup id="fnref:16:1" role="doc-noteref"><a href="#fn:16" class="footnote">12</a></sup> over the next $h$ periods:</p>
\[\hat{\sigma}_{T+1:T+h} = \sqrt{h} \hat{\sigma}_{T+1}\]
</li>
</ul>
<h4 id="how-to-determine-the-parameters-of-a-garch11-model">How to determine the parameters of a GARCH(1,1) model?</h4>
<p>The parameters of a GARCH(1,1) model - either $\omega$, $\alpha$ and $\beta$ or $\alpha$, $\beta$, $\gamma$ and $\tilde{\sigma}^2$ - are typically determined by
<a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation">maximum likelihood estimation (MLE)</a> with a Gaussian<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">13</a></sup> or Student’s $t$ assumption for the distribution of the innovations.</p>
<p>A note of caution, though.</p>
<p>There are plenty of software packages able to do this estimation, but the underlying optimization problem <em>has been documented to be numerically
difficult and prone to error</em><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> due to <em>a one dimensional manifold in the parameter space where the likelihood function is large and almost constant</em><sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup>, which tends to “trap” numerical algorithms.</p>
<p>Possible remediations have been suggested in Zumbach<sup id="fnref:15:2" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> and in Kristensen and Linton<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup>, like reformulating the optimization problem in an alternative parameter space or using
a closed-form estimator for the GARCH(1,1) parameters that does not rely on any numerical optimization procedure, but unfortunately, these remediations are not sufficient due to the
problematic<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">16</a></sup> finite sample behavior of the maximum likelihood estimates…</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the GARCH(1,1) volatility forecasting model through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/volatility/forecast/garch</code></a>.</p>
<p>This endpoint supports the 4 variance proxies below:</p>
<ul>
<li>Squared close-to-close returns</li>
<li>Demeaned squared close-to-close returns</li>
<li>The Parkinson range</li>
<li>The jump-adjusted Parkinson range</li>
</ul>
<p>Internally, this endpoint:</p>
<ul>
<li>Assumes that the asset unconditional variance $\tilde{\sigma}^2$ is equal to its long-term average value $\frac{1}{T} \sum_{t=1}^{T} \tilde{\sigma}^2_t$</li>
<li>Automatically determines the optimal value of the GARCH(1,1) parameters $\alpha$, $\beta$ and $\gamma$ using a proprietary numerical optimization procedure</li>
</ul>
<h2 id="example-of-usage---volatility-forecasting-at-monthly-level-for-various-etfs">Example of usage - Volatility forecasting at monthly level for various ETFs</h2>
<p>As an example of usage, I propose to enrich the results of <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the previous blog post</a>, in which
monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">17</a></sup> regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">18</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">19</a></sup> are the following<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup>:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td>Random walk</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.66</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.68</td>
<td>46%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Squared close-to-close returns</td>
<td>4.7%</td>
<td>0.73</td>
<td>45%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-1.3%</strong></td>
<td><strong>0.98</strong></td>
<td><strong>43%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Parkinson range</td>
<td>5.6%</td>
<td>0.94</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Parkinson range</td>
<td>5.1%</td>
<td>1.00</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Parkinson range</td>
<td>4.3%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>2.7%</strong></td>
<td><strong>1.18</strong></td>
<td><strong>47%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.9%</td>
<td>0.70</td>
<td>45%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Jump-adjusted Parkinson range</td>
<td>5.1%</td>
<td>0.71</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.0%</td>
<td>0.76</td>
<td>45%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.0%</strong></td>
<td><strong>1.00</strong></td>
<td><strong>45%</strong></td>
</tr>
</tbody>
</table>
<p>From these, it is possible to conclude the following:</p>
<ul>
<li>The two GARCH(1,1) models using improved variance proxies produce volatility forecasts with better r-squared than the GARCH(1,1) model using squared returns (lines #8 and #12 v.s. line #4), which is in agreement with Molnar<sup id="fnref:9:5" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup></li>
<li>The two GARCH(1,1) models using variance proxies that integrate close prices produce nearly unbiased forecasts (lines #4 and #12), which, together with their relatively high r-squared, makes them volatility forecasting models to recommend in these cases</li>
<li>The GARCH(1,1) using the Parkinson range as variance proxy produces the most biased forecasts (line #8), which makes it a volatility forecasting model to avoid in this case</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The GARCH(1,1) volatility forecasting model exhibits good practical performances for a wide range of assets, as empirically demonstrated in the previous section.</p>
<p>Nevertheless, because this model is still <em>unable to describe certain aspects often found in financial data</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, many different extensions have been proposed over the years like AGARCH, EGARCH, QGARCH, TGARCH<sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>….</p>
<p>I will not discuss these further, though, and next in this series dedicated to volatility forecasting, I will detail a model that was initially developed for use with high frequency data.</p>
<p>Meanwhile, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijderiv/4/3/63">Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Or more generally, of a weighted moving average of one of its past variance proxies. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://math.berkeley.edu/~btw/thesis4.pdf">Brandon Williams, GARCH(1,1) models, B. Sc. Thesis, 15. Juli 2011</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>See <a href="">Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307–327</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S030440761000076X">Andrew J. Patton, Volatility forecast comparison using imperfect volatility proxies, Journal of Econometrics, Volume 160, Issue 1, 2011, Pages 246-256</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1391681">Daniel B. Nelson and Charles Q. Cao, Inequality Constraints in the Univariate GARCH Model, Journal of Business & Economic Statistics, Vol. 10, No. 2 (Apr., 1992), pp. 229-235</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.researchgate.net/publication/237530561_VARIANCE_INITIALISATION_IN_GARCH_ESTIMATION">Pelagatti, M., Lisi, F. (2009). Variance initialisation in GARCH estimation. In Paganoni, A.M., Sangalli, L.M., Secchi, P., Vantini, S. (eds.), S.Co. 2009 Sixth Conference Complex Data Modeling and Computationally Intensive Statistical Methods for Estimation and Prediction, Maggioli Editore, Milan</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="http://dx.doi.org/10.1080/00036846.2016.1170929">Peter Molnar (2016): High-low range in GARCH models of stock return volatility, Applied Economics</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:9:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:9:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:9:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:9:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:9:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>Also called the asset long-term variance. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>More precisely, a convex combination. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>Which is not surprising since <em>in fact, exponential smoothing is a constrained version of GARCH (1,1)</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup>, without mean-reversion. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://centaur.reading.ac.uk/21316/">Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:16:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>In which case, the Gaussian MLE is usually considered as a <a href="https://en.wikipedia.org/wiki/Quasi-maximum_likelihood_estimate">quasi-maximum likelihood estimate</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-1-4615-4389-3_8">Zumbach, G. (2000). The Pitfalls in Fitting Garch(1,1) Processes. In: Dunis, C.L. (eds) Advances in Quantitative Asset Management. Studies in Computational Finance, vol 1. Springer, Boston, MA</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:15:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/4093228">Dennis Kristensen and Oliver Linton, A Closed-Form Estimator for the GARCH(1,1) Model, Econometric Theory, Vol. 22, No. 2 (Apr., 2006), pp. 323-337</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See for example <a href="https://ntguardian.wordpress.com/2017/11/02/problems-estimating-garch-parameters-r/">here</a> and <a href="https://ntguardian.wordpress.com/2019/01/28/problems-estimating-garch-parameters-r-part-2-rugarch/">there</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>See <a href="https://econpapers.repec.org/bookchap/nbrnberch/1214.htm">Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations</a>. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">21</a></sup>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>For all models, I used an expanding window for the volatility forecast computation. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.In the previous post of this series on volatility forecasting, I described the simple and the exponentially weighted moving average volatility forecasting models. In particular, I showed that these two models belong to the generic family of weighted moving average volatility forecasting models1, whose members represent the volatility of an asset as a weighted moving average of its past squared returns2. Another member of this family is the Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) model, widely used in financial time series modelling and implemented in most statistics and econometric software packages3. In this blog post, I will detail the simplest but often very useful4 GARCH(1,1) volatility forecasting model and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs. Mathematical preliminaries (reminders) This section contains reminders from a previous blog post. Volatility modelling and volatility proxies Let $r_t$ be the (logarithmic) return of an asset over a time period $t$ (a day, a week, a month..), over which its (conditional) mean return is supposed to be null. Then: The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $ From this definition, the squared return $r_t^2$ of an asset is a (noisy5) variance estimator - or variance proxy5 - for that asset variance over the considered time period. Another example of an asset variance proxy is the Parkinson range of an asset. The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$. The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $ The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$. Weighted moving average volatility forecasting model Boudoukh et al.1 shows that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with \[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\] , where: $1 \leq k \leq T$ is the size of the moving average, possibly time-dependent $w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well GARCH(1,1) volatility forecasting model The GARCH(p,q) model Definition Bollerslev4’s GARCH model is a generalization of Engle’s ARCH econometric model which captures the time-varying nature of the (conditional) variance of certain time series like asset returns. Under a GARCH(p,q) model, an asset next period’s conditional variance $\sigma_{T+1}^2$ is modeled as recursive linear function of its own $p$ lagged conditional variances $\sigma_{T}^2, \sigma_{T-1}^2…$ and of its $q$ lagged squared returns $r_{T}^2, r_{T-1}^2…$, which leads to the formula \[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j r_{T+1-i}^2\] , where: The parameters $\omega$, $\alpha_j$, $j=1..q$ and $\beta_i$, $i=1..p$ are non-negative and subject to various inequality constraints depending on working assumptions6 The initial conditional variance $\hat{\sigma}_1^2$ is usually taken equal to $r_1^2$, but c.f. Pelagatti and Lisi7 for a thorough discussion about this subject Squared returns v.s. generic variance proxy Molnar8 notes that in GARCH type of models, demeaned squared returns serve as a way to calculate innovations to the volatility8 so that replacing the squared returns by more precise volatility estimates will produce better GARCH models, regarding both in-sample fit and out-of-sample forecasting performance8. Molnar8 then proposes to modify the GARCH(p,q) model for the estimation of an asset next period’s conditional variance $\sigma_{T+1}^2$ as follows \[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j \tilde{\sigma}_{T+1-i}^2\] , where $\tilde{\sigma}^2_t$, $t=1..T$ are the asset past periods’ variance proxies. To be noted that replacing squared returns by less noisy variance proxies is already discussed at length in the previous blog post in the case of the simple and the exponentially weighted moving average volatility forecasting models. The GARCH(1,1) model Definition Because the GARCH(1,1) model works surprisingly well in comparison with much more complex [GARCH] models8, it is usually the main GARCH model used in practice. Under this model, the generic GARCH formula for the estimate of an asset next period’s conditional variance can be re-parametrized as follows \[\hat{\sigma}_{T+1}^2 = \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2\] , where: $\alpha$, $\beta$ and $\gamma$ are positive parameters summing to one $\tilde{\sigma}^2$ is a strictly positive parameter, corresponding to the asset unconditional variance9 The GARCH(1,1) model thus estimates an asset next period’s conditional variance $\hat{\sigma}_{T+1}^2$ as a weighted average10 of three different variance estimators: A long-term variance estimator $\tilde{\sigma}^2$ A short-term variance estimator $\tilde{\sigma}^2_{T}$ The current GARCH(1,1) variance estimator $\hat{\sigma}_{T}^2$ and the weights $\alpha$, $\beta$ and $\gamma$ determine the speed with which the model adapts to short-term variance v.s. reverts to its long-term variance. Relationship with the generic weighted moving average model By developing the recursive definition of the GARCH(1,1) model, it is possible to see that it is a specific kind of weighted moving average volatility forecasting model, with: $k = T$ $w_0 = \gamma \sum_{k=0}^{T-1} \beta^k$ $w_1 = \alpha$, $w_2 = \alpha \beta$, …, $w_{T-1} = \alpha \beta^{T-2}$, $w_T = \alpha \beta^{T-1}$, that is, exponentially decreasing weights emphasizing recent past variance proxies v.s. more distant ones in the model, exactly like in the exponentially weighted moving average volatility forecasting model11 Volatility forecasting formulas Under a GARCH(1,1) volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes: To estimate an asset next period’s volatility: \[\hat{\sigma}_{T+1} = \sqrt{ \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2 }\] To estimate an asset next $h$-period’s ahead volatility12, $h \geq 2$: \[\hat{\sigma}_{T+h} = \sqrt{ \tilde{\sigma}^2 + \left( \alpha + \beta \right)^{h-1} \left( \hat{\sigma}_{T+1} - \tilde{\sigma}^2 \right) }\] To estimate an asset aggregated volatility12 over the next $h$ periods: \[\hat{\sigma}_{T+1:T+h} = \sqrt{h} \hat{\sigma}_{T+1}\] How to determine the parameters of a GARCH(1,1) model? The parameters of a GARCH(1,1) model - either $\omega$, $\alpha$ and $\beta$ or $\alpha$, $\beta$, $\gamma$ and $\tilde{\sigma}^2$ - are typically determined by maximum likelihood estimation (MLE) with a Gaussian13 or Student’s $t$ assumption for the distribution of the innovations. A note of caution, though. There are plenty of software packages able to do this estimation, but the underlying optimization problem has been documented to be numerically difficult and prone to error14 due to a one dimensional manifold in the parameter space where the likelihood function is large and almost constant14, which tends to “trap” numerical algorithms. Possible remediations have been suggested in Zumbach14 and in Kristensen and Linton15, like reformulating the optimization problem in an alternative parameter space or using a closed-form estimator for the GARCH(1,1) parameters that does not rely on any numerical optimization procedure, but unfortunately, these remediations are not sufficient due to the problematic16 finite sample behavior of the maximum likelihood estimates… Implementation in Portfolio Optimizer Portfolio Optimizer implements the GARCH(1,1) volatility forecasting model through the endpoint /assets/volatility/forecast/garch. This endpoint supports the 4 variance proxies below: Squared close-to-close returns Demeaned squared close-to-close returns The Parkinson range The jump-adjusted Parkinson range Internally, this endpoint: Assumes that the asset unconditional variance $\tilde{\sigma}^2$ is equal to its long-term average value $\frac{1}{T} \sum_{t=1}^{T} \tilde{\sigma}^2_t$ Automatically determines the optimal value of the GARCH(1,1) parameters $\alpha$, $\beta$ and $\gamma$ using a proprietary numerical optimization procedure Example of usage - Volatility forecasting at monthly level for various ETFs As an example of usage, I propose to enrich the results of the previous blog post, in which monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz17 regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative18 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Averaged results for all ETFs/regression models over each ETF price history19 are the following20: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ Random walk Squared close-to-close returns 5.8% 0.66 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Squared close-to-close returns 5.8% 0.68 46% EWMA, optimal $\lambda$ Squared close-to-close returns 4.7% 0.73 45% GARCH(1,1) Squared close-to-close returns -1.3% 0.98 43% Random walk Parkinson range 5.6% 0.94 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Parkinson range 5.1% 1.00 47% EWMA, optimal $\lambda$ Parkinson range 4.3% 1.06 48% GARCH(1,1) Parkinson range 2.7% 1.18 47% Random walk Jump-adjusted Parkinson range 4.9% 0.70 45% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Jump-adjusted Parkinson range 5.1% 0.71 47% EWMA, optimal $\lambda$ Jump-adjusted Parkinson range 4.0% 0.76 45% GARCH(1,1) Jump-adjusted Parkinson range -1.0% 1.00 45% From these, it is possible to conclude the following: The two GARCH(1,1) models using improved variance proxies produce volatility forecasts with better r-squared than the GARCH(1,1) model using squared returns (lines #8 and #12 v.s. line #4), which is in agreement with Molnar8 The two GARCH(1,1) models using variance proxies that integrate close prices produce nearly unbiased forecasts (lines #4 and #12), which, together with their relatively high r-squared, makes them volatility forecasting models to recommend in these cases The GARCH(1,1) using the Parkinson range as variance proxy produces the most biased forecasts (line #8), which makes it a volatility forecasting model to avoid in this case Conclusion The GARCH(1,1) volatility forecasting model exhibits good practical performances for a wide range of assets, as empirically demonstrated in the previous section. Nevertheless, because this model is still unable to describe certain aspects often found in financial data3, many different extensions have been proposed over the years like AGARCH, EGARCH, QGARCH, TGARCH3…. I will not discuss these further, though, and next in this series dedicated to volatility forecasting, I will detail a model that was initially developed for use with high frequency data. Meanwhile, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71. ↩ ↩2 Or more generally, of a weighted moving average of one of its past variance proxies. ↩ See Brandon Williams, GARCH(1,1) models, B. Sc. Thesis, 15. Juli 2011. ↩ ↩2 ↩3 See Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307–327. ↩ ↩2 See Andrew J. Patton, Volatility forecast comparison using imperfect volatility proxies, Journal of Econometrics, Volume 160, Issue 1, 2011, Pages 246-256. ↩ ↩2 See Daniel B. Nelson and Charles Q. Cao, Inequality Constraints in the Univariate GARCH Model, Journal of Business & Economic Statistics, Vol. 10, No. 2 (Apr., 1992), pp. 229-235. ↩ See Pelagatti, M., Lisi, F. (2009). Variance initialisation in GARCH estimation. In Paganoni, A.M., Sangalli, L.M., Secchi, P., Vantini, S. (eds.), S.Co. 2009 Sixth Conference Complex Data Modeling and Computationally Intensive Statistical Methods for Estimation and Prediction, Maggioli Editore, Milan. ↩ See Peter Molnar (2016): High-low range in GARCH models of stock return volatility, Applied Economics. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 Also called the asset long-term variance. ↩ More precisely, a convex combination. ↩ Which is not surprising since in fact, exponential smoothing is a constrained version of GARCH (1,1)1, without mean-reversion. ↩ See Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22. ↩ ↩2 In which case, the Gaussian MLE is usually considered as a quasi-maximum likelihood estimate. ↩ See Zumbach, G. (2000). The Pitfalls in Fitting Garch(1,1) Processes. In: Dunis, C.L. (eds) Advances in Quantitative Asset Management. Studies in Computational Finance, vol 1. Springer, Boston, MA. ↩ ↩2 ↩3 See Dennis Kristensen and Oliver Linton, A Closed-Form Estimator for the GARCH(1,1) Model, Econometric Theory, Vol. 22, No. 2 (Apr., 2006), pp. 323-337. ↩ See for example here and there. ↩ See Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer21. ↩ The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. ↩ For all models, I used an expanding window for the volatility forecast computation. ↩ See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Random Portfolio Benchmarking: Simulation-based Performance Evaluation in Finance2024-02-06T00:00:00-06:002024-02-06T00:00:00-06:00https://portfoliooptimizer.io/blog/random-portfolio-benchmarking-simulation-based-performance-evaluation-in-finance<p>As noted in Surz<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>, <em>the question “Is [a mutual fund’s]<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">2</a></sup> performance good?” can only be answered relative to something</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>, typically by comparing that fund to a benchmark like a financial index or to a peer group.</p>
<p>Unfortunately, these two methodologies are not without issues. For example, it is very difficult to create an index <em>captur[ing] the essence of the people, process, and philosophy behind an investment product</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> and peer groups have well-known biases<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>
(classification bias, survivorship bias…).</p>
<p>In a series of papers (Surz<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, Surz<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>, Surz<sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>…), Ronald J. Surz proposes an <em>innovation that combines the better aspects of both [methodologies] while eliminating their undesirable properties</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>. This innovation
consists in evaluating a fund against the fund manager’s <em>true opportunity set</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, defined as the set of <em>all of the possible portfolios that the manager could have conceivably
held following his unique investment process</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>.</p>
<p>In practice, the fund manager’s opportunity set is approximated by the simulation of thousands of random portfolios in the same universe of assets as the one of the fund manager and satisfying the same constraints (long-only,
long-short…) and rules (portfolio rebalancing rules…) as those of the fund manager. Then, because these portfolios do not exhibit any particular skill<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, they can be <em>used as the control group to test [the]
fund manager skill</em><sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, thus allowing to apply <em>modern statistics to the problem of performance evaluation</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>.</p>
<p>In this blog post, I will describe how to generate random portfolios, detail Surz’s original methodology as well as some of its variations and illustrate the usage of random portfolios with a couple of examples like
the creation of synthetic benchmarks or the evaluation and monitoring of trading strategies.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="definition">Definition</h3>
<p>Let be:</p>
<ul>
<li>$n$ the number of assets in a universe of assets</li>
<li>$\mathcal{C} \subset \mathbb{R}^{n}$ a subset of $ \mathbb{R}^{n}$ representing the constraints imposed<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">7</a></sup> on a fund manager when investing in that universe of assets, for example:
<ul>
<li>Short sale constraints</li>
<li>Concentration constraints (assets, sectors, industries…)</li>
<li>Leverage constraints</li>
<li>Cardinality constraints (i.e., minimum or maximum number of assets constraints)</li>
<li>Portfolio volatility constraints</li>
<li>Portfolio tracking error constraints</li>
<li>…</li>
</ul>
</li>
</ul>
<p>Then, a <em>(constrained) random portfolio</em> in that universe of assets is a vector of portfolio weights $w \in \mathbb{R}^{n}$ generated at random over the set $\mathcal{C}$.</p>
<h3 id="generation-at-random-vs-generation-uniformly-at-random">Generation at random v.s. generation uniformly at random</h3>
<p>In the context of performance evaluation, as in Surz’s methodology, it is theoretically preferable that random portfolios are generated <em>uniformly</em> at random over the set $\mathcal{C}$, so as not to introduce any biases<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>
which would otherwise defeat the purpose of using these portfolios as an unbiased control group<sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>Figure 1 and Figure 2 illustrate the difference between random portfolios generated at random v.s. uniformly at random in the case of a three-asset universe subject to long-only
and full investment constraints<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup>:</p>
<ul>
<li>In Figure 1, the random portfolios are visibly concentrated in the middle of the standard 2-simplex in $\mathbb{R}^3$<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup> - these random portfolios are NOT generated uniformly at random</li>
<li>In Figure 2, the random portfolios seem to be “well spread” over the standard 2-simplex in $\mathbb{R}^3$<sup id="fnref:11:2" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup> - these random portfolios are generated uniformly at random</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-non-uniform-random-generation-shaw.png"><img src="/assets/images/blog/random-portfolio-benchmarking-non-uniform-random-generation-shaw.png" alt="Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw." /></a>
<figcaption>Figure 1. Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-uniform-random-generation-shaw.png"><img src="/assets/images/blog/random-portfolio-benchmarking-uniform-random-generation-shaw.png" alt="Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw." /></a>
<figcaption>Figure 2. Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw.</figcaption>
</figure>
<p>One important remark, though, is that real-life portfolios are usually binding on at least one of their constraints, so that generating random portfolios biased toward the boundary of the geometrical
object associated to the constraints set $\mathcal{C}$ might not be a real problem in practice<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>.</p>
<h3 id="generation-of-random-portfolios-over-the-standard-simplex">Generation of random portfolios over the standard simplex</h3>
<p>When the constraints imposed on a portfolio are 1) a full investment constraint and 2) a long-only constraint, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, w_i \geq 0, i = 1..n\}\]
<p>and the geometrical object associated to that subset is called<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">11</a></sup> a <a href="https://en.wikipedia.org/wiki/Simplex#Standard_simplex">standard simplex</a>, already illustrated in Figure 1 and Figure 2 with $n = 3$.</p>
<p>Several algorithms exist to generate points uniformly at random over a standard simplex<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup>, among which:</p>
<ul>
<li>An algorithm based on differences of sorted <a href="https://en.wikipedia.org/wiki/Continuous_uniform_distribution">uniform random variables</a></li>
<li>An algorithm based on normalized <a href="https://en.wikipedia.org/wiki/Exponential_distribution">unit exponential random variables</a></li>
</ul>
<h3 id="generation-of-random-portfolios-over-the-restricted-standard-simplex">Generation of random portfolios over the restricted standard simplex</h3>
<p>When the constraints imposed on a portfolio are 1) a full investment constraint, 2) a long-only constraint and 3) minimum/maximum asset weights constraints, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, 1 \geq u_i \geq w_i \geq l_i \geq 0, i = 1..n\}\]
<p>, where:</p>
<ul>
<li>$l_1, …, l_n$ represent minimum asset weights constraints</li>
<li>$u_1, …, u_n$ represent maximum asset weights constraints</li>
</ul>
<p>The geometrical object associated to that subset is a “restricted” standard simplex, as illustrated in Figure 3, with:</p>
<ul>
<li>$n = 3$</li>
<li>$ 0.7 \geq w_1 \geq 0.1$</li>
<li>$ 0.8 \geq w_2 \geq 0 $</li>
<li>$ 0.6 \geq w_3 \geq 0.1 $</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-restricted-simplex-borkowski-piepel.png"><img src="/assets/images/blog/random-portfolio-benchmarking-restricted-simplex-borkowski-piepel.png" alt="Example of restricted standard simplex, three-asset universe. Source: Piepel." /></a>
<figcaption>Figure 3. Example of restricted standard simplex, three-asset universe. Source: Piepel.</figcaption>
</figure>
<p>Generating points uniformly at random over a restricted standard simplex is much more complex than generating points uniformly at random over a standard simplex.</p>
<p>Hopefully, this problem has been studied at least since the 1990’s by people working in the statistical domain of the <a href="https://en.wikipedia.org/wiki/Design_of_experiments">design of experiments</a>,
and an algorithm based on the <a href="https://en.wikipedia.org/wiki/Conditional_probability_distribution">conditional distribution method</a> has been published<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">13</a></sup> in 2000.</p>
<h3 id="generation-of-random-portfolios-over-a-convex-polytope">Generation of random portfolios over a convex polytope</h3>
<p>When the constraints imposed on a portfolio consist in generic linear constraints, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } A_e w = b_e, A_i w \leq b_i \}\]
<p>, where:</p>
<ul>
<li>$A_e \in \mathbb{R}^{n_e \times n}$ and $b_e \in \mathbb{R}^{n_e}$, $n_e \geq 1$, represent linear equality constraints</li>
<li>$A_i \in \mathbb{R}^{n_i \times n}$ and $b_i \in \mathbb{R}^{n_i}$, $n_i \geq 1$, represent linear inequality constraints</li>
</ul>
<p>The geometrical object associated to that subset is called a <a href="https://en.wikipedia.org/wiki/Convex_polytope">convex polytope</a>, illustrated in Figure 4, with:</p>
<ul>
<li>$n = 3$</li>
<li>$\sum_{i=1}^n w_i = 1$</li>
<li>$1 \geq w_i \geq 0, i = 1..n$</li>
<li>$w_1 > w_2$</li>
<li>$2w_3 > w_2 > 0.5w_3$</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-general-simplex-tervonen.png"><img src="/assets/images/blog/random-portfolio-benchmarking-general-simplex-tervonen.png" alt="Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al." /></a>
<figcaption>Figure 4. Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al.</figcaption>
</figure>
<p>As one can guess, generating points uniformly at random over a convex polytope is another level higher in terms of complexity, and while some algorithms exist to do so<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">14</a></sup>, they are impractical in high dimension.</p>
<p>From the literature, what is possible to achieve instead is to generate points <em>asymptotically</em> uniformly at random, using <a href="https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo">Markov chain Monte Carlo (MCMC)</a> algorithms
like the Hit-And-Run algorithm<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">15</a></sup>.</p>
<h3 id="generation-of-random-portfolios-over-a-generic-constraint-set">Generation of random portfolios over a generic constraint set</h3>
<p>When the constraints imposed on a portfolio are generic, that is, quadratic (volatility or tracking error constraints) and/or non-convex (threshold constraints) and/or integer (maximum number of assets constraints,
round lot constraints), the final boss has arrived.</p>
<p>With such constraints, the only reasonable<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">16</a></sup> approach in the literature seems to recast the problem of generating random points over the constraint set $C$ as a (penalized) optimization problem and use
<a href="https://en.wikipedia.org/wiki/Genetic_algorithm">genetic algorithms</a> to solve it<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>.</p>
<p>The underlying idea is to randomly<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">17</a></sup> minimize an objective function $f$ essentially made of penalties - the higher the distance of a point $w \in \mathbb{R}^{n}$ from the constraint set $C$, the higher the value of the objective function $f(w)$ -,
so that at optimum, a random point satisfying all the constraints<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup> is found.</p>
<p>Of course, the points generated this way are not generated uniformly at random, but when the constraints are fully generic, that requirement should probably be dropped altogether as mentioned in Dawson and Young<sup id="fnref:9:1" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<h2 id="investment-funds-performance-evaluation-with-random-portfolios">Investment fund’s performance evaluation with random portfolios</h2>
<h3 id="rationale">Rationale</h3>
<p>Evaluating the performance of an investment fund is actually a <a href="https://en.wikipedia.org/wiki/Statistical_hypothesis_testing">statistical hypothesis test</a> in disguise<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>, in which:</p>
<ul>
<li>The null hypothesis is <em>The investment fund’s exhibit no particular “performance” over the considered time period</em></li>
<li>The <a href="https://en.wikipedia.org/wiki/Test_statistic">test statistic</a> is a quantitative measure of “performance” over the considered time period (e.g. annualized return, holding period return, risk-adjusted return…)</li>
<li>The (empirical) distribution of the test statistic under the null hypothesis is computed from a sample made of either
<ul>
<li>One observation - the investment fund’s benchmark</li>
<li>A small number of observations - the investment fund’s peer group</li>
<li>Any desired number of observations - random portfolios generated from the investment fund’s universe of assets and obeying to the fund’s constraints and rules</li>
</ul>
</li>
</ul>
<p>From this perspective, using a benchmark, a peer group or random portfolios for performance evaluation <em>is essentially a choice between sampling approaches</em><sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>Still, random portfolios represent a more rigorous approach to performance evaluation than the two other alternatives, for various reasons highlighted in different papers<sup id="fnref:4:5" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup><sup id="fnref:9:2" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup><sup id="fnref:10:2" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> and summarized by Dawson<sup id="fnref:9:3" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> as follows</p>
<blockquote>
<p>A set of uniformly distributed, stochastically generated, portfolios that by construction incorporate no investment strategy, bias or skill form an effective control set for any portfolio measurement metric. This allows the true value of a strategy or model to be identified. They also provide a mechanism to differentiate between effects due to “market conditions” and effects due to either the management of a portfolio, or the constraints the management is obliged to work within.</p>
</blockquote>
<h3 id="surzs-methodology">Surz’s methodology</h3>
<p>Using random portfolios to evaluate an investment fund’s performance over a considered time period is a two-step <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo</a> simulation process<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">20</a></sup>:</p>
<ol>
<li>
<p>Modeling step</p>
<p>Identify the fund’s main characteristics.</p>
</li>
<li>
<p>Computational step</p>
</li>
</ol>
<ul>
<li>
<p>Random portfolios simulation</p>
<p>Generate random portfolios compatible with the identified fund’s main characteristics and simulate their evolution through the considered time period.</p>
</li>
<li>
<p>Performance evaluation</p>
<p>Determine the level of statistical significance of the fund’s performance over the considered time period using the previously simulated random portfolios.</p>
</li>
</ul>
<h4 id="step-1---identifying-the-funds-main-characteristics">Step 1 - Identifying the fund’s main characteristics</h4>
<p>Dawson<sup id="fnref:9:4" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> note that <em>an investment strategy is essentially a set of constraints [and rules] that are carefully constructed to (hopefully) remove, in aggregate, poor performing regions of the solution space</em><sup id="fnref:9:5" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>. Thus,
it is very important to identify as precisely as possible the main characteristics of the fund, even though <em>this […] information is not always available, as only the manager knows [it] exactly</em><sup id="fnref:23:2" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>.</p>
<p>From a practical perspective, these characteristics need to be provided as<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">21</a></sup>:</p>
<ul>
<li>A universe of $n$ assets modeling the fund’s universe of assets</li>
<li>A constraint set $\mathcal{C} \subset \mathbb{R}^{n}$ modeling the fund’s constraints</li>
<li>An ensemble of decision rules modeling<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">22</a></sup> the fund’s rebalancing rules</li>
</ul>
<p>For examples of this step in different contexts, c.f. Kothari and Warner<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">23</a></sup> and Surz<sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<h4 id="step-2a---simulating-random-portfolios">Step 2a - Simulating random portfolios</h4>
<p>Thanks to the previous step, it becomes possible to simulate thousands of random portfolios that the fund’s manager could have potentially held over the considered time period.</p>
<p>The only difficulty at this point is computational, due to the impact of the constraint set on the algorithmic machinery required to generate portfolio weights at random.</p>
<p>As a side note, and <em>for a fair comparison [with the fund under analysis], transaction costs, as well as all other kinds of costs, must be considered</em><sup id="fnref:10:3" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup> when simulating random portfolios.</p>
<h4 id="step-2b---evaluating-the-funds-performance">Step 2b - Evaluating the fund’s performance</h4>
<p>The ranking of the fund’s performance against the performance of all the simulated random portfolios provides a direct measure of the statistical significance of that fund’s performance<sup id="fnref:4:6" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>Indeed, under the null hypothesis that the investment fund’s exhibit no particular performance over the considered time period, <a href="https://en.wikipedia.org/wiki/P-value">the $p$-value</a> of the statistical hypothesis test
mentioned at the beginning of this section is defined as<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup></p>
\[p = \frac{n_x + 1}{N + 1}\]
<p>, where:</p>
<ul>
<li>$n_x$ is the the number of random portfolios whose performance over the considered time period is as extreme or more extreme than that of the fund under evaluation</li>
<li>$N$ is the number of simulated random portfolios</li>
</ul>
<p>This idea is illustrated in Figure 5, which depicts:</p>
<ul>
<li>The distribution of the holding period return of 1000 random portfolios made of stocks belonging to the S&P 500 and subject to misc. turnover and cardinality constraints (solid line)</li>
<li>The 95th percentile of that distribution (dashed line)</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-performance-evaluation-stein.png"><img src="/assets/images/blog/random-portfolio-benchmarking-performance-evaluation-stein.png" alt="Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein." /></a>
<figcaption>Figure 5. Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein.</figcaption>
</figure>
<p>From this figure, if we were to evaluate the performances of a fund <em>trading these securities and operating under constraints similar to those simulated, [it] would have to obtain a return equal of at least 80%
during the 6 year period for the null of no particular performance to be rejected</em><sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> at a 95% confidence level.</p>
<h3 id="variations-on-surzs-methodology">Variations on Surz’s methodology</h3>
<p>Burns<sup id="fnref:10:4" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup> and Lisi<sup id="fnref:23:3" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> both extend Surz’s methodology from one time period to several time periods - which allows <em>to consider the persistence of results over time</em><sup id="fnref:23:4" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> - by describing different ways of
combining individual $p$-values<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>. As a by-product,
Burns<sup id="fnref:10:5" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:26:2" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup> empirically demonstrates that Stouffer’s method<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">27</a></sup> for combining $p$-values should be preferred to <a href="https://en.wikipedia.org/wiki/Fisher%27s_method">Fisher’s</a> when evaluating a fund’s performance
over multiple time periods.</p>
<p>Stein<sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to replace Surz’s methodology by testing <em>whether the distribution of returns of the [fund under evaluation] is
<a href="https://en.wikipedia.org/wiki/Stochastic_ordering">stochastically greater</a> than that of [a] chosen percentile random fund</em> using the <a href="https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test">Mann–Whitney <em>U</em> test</a>.</p>
<h3 id="caveats">Caveats</h3>
<p>Surz<sup id="fnref:4:7" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> warns that <em>there are many ways to implement a [Monte Carlo simulation] approach [with random portfolios], some better than others, and some worse</em><sup id="fnref:4:8" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>In particular, Surz<sup id="fnref:4:9" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> argues that, when selecting a number of assets at random to satisfy portfolio cardinality constraints, it is required to use <em>value-weighted sampling, so that the probability of choosing a given [asset] is proportionate to its outstanding
capitalization</em><sup id="fnref:4:10" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>. Otherwise, some <em>macroeconomic [in]consistency</em><sup id="fnref:4:11" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> would be introduced by the Monte Carlo simulation process, which would ultimately bias the end results. This point is confirmed for both
U.S. and global stocks in Arnott et al.<sup id="fnref:28:1" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>, who show that random portfolios <em>introduce, often unintentionally, value and small cap tilts</em><sup id="fnref:28:2" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>. Nevertheless, other authors<sup id="fnref:23:5" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> argue on the contrary that
using equally-weighted sampling is more representative of the behavior of an “unskilled” manager, which is exactly what random portfolios are supposed to model!</p>
<p>More generally, random portfolios might have “unfair” characteristics v.s. the fund under evaluation (higher volatility, higher beta, lower turnover…), which must either be acknowledged or controlled for depending on the exact circumstances.</p>
<h2 id="implementations">Implementations</h2>
<h3 id="implementation-in-portfolio-optimizer">Implementation in <strong>Portfolio Optimizer</strong></h3>
<p>Through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/simulation/random</code></a>, <strong>Portfolio Optimizer</strong> allows to:</p>
<ul>
<li>Generate the weights of a random portfolio subject to general linear inequality constraints</li>
</ul>
<p>Through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/simulation/evolution/random</code></a>, <strong>Portfolio Optimizer</strong> allows to:</p>
<ul>
<li>Simulate the evolution of a random portfolio subject to general linear inequality constraints over a specified time period, with 3 different portfolio rebalancing strategies:
<ul>
<li>Buy and hold</li>
<li>Rebalancing toward the weights of the initial random portfolio</li>
<li>Rebalancing toward the weights of a new random portfolio</li>
</ul>
</li>
</ul>
<h3 id="implementations-elsewhere">Implementations elsewhere</h3>
<p>I am aware of two commercial implementations of random portfolios for performance evaluation as described in this blog post:</p>
<ul>
<li><strong><a href="https://www.portfolioprobe.com/">Portfolio Probe</a></strong>, from Patrick Burns<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">28</a></sup> of <a href="https://www.burns-stat.com/">Burns Statistics</a></li>
<li><strong><a href="https://ppca-inc.com/PODs/pipods.htm">PIPODs</a></strong>, from Ronald Surz<sup id="fnref:21:1" role="doc-noteref"><a href="#fn:21" class="footnote">28</a></sup> of <a href="https://ppca-inc.com/">PPCA Inc.</a>, which seems discontinued</li>
</ul>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="hedge-funds-performance-evaluation">Hedge funds performance evaluation</h3>
<p>The performances of hedge funds are notoriously difficult to evaluate due to the lack of both proper benchmarks and homogeneous peer groups<sup id="fnref:32:1" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>Random portfolios offer a solution to this problem <em>because they can reflect the unique specifications of each individual hedge fund</em><sup id="fnref:32:2" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>More details can be found in Surz<sup id="fnref:32:3" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup> or in Molyboga and Ahelec<sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">29</a></sup>.</p>
<h3 id="investment-strategy-returns-dispersion-analysis">Investment strategy returns dispersion analysis</h3>
<p>Kritzman and Page<sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup> uses random portfolios to compare the relative importance of different investment strategies (investment at global asset class level, at individual stock level, etc.)
and concludes that<sup id="fnref:31:1" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup></p>
<blockquote>
<p>Contrary to perceived doctrine, dispersion around average performance arising from security selection is substantially greater than dispersion around average performance arising from all other investment choices. Moreover, asset allocation, widely considered the most important investment choice, produces the least dispersion; thus, from a normative perspective it is the least important investment choice.</p>
</blockquote>
<p>Figure 6, which quantifies that dispersion around average performance, <em>shows the extent to which a talented investor (top 25th or 5th percentile) could have improved upon average performance by engaging in various
investment choices across a global universe. It also shows how far below average an unlucky investor (bottom 75th or 95th percentile) could have performed, depending on the choice of investment discretion</em><sup id="fnref:31:2" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup>.</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-returns-dispersion-kritzman.png"><img src="/assets/images/blog/random-portfolio-benchmarking-returns-dispersion-kritzman.png" alt="5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page." /></a>
<figcaption>Figure 6. 5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page.</figcaption>
</figure>
<p>While this conclusion has been criticized<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">31</a></sup>, the underlying methodology - returns dispersion analysis - provides valuable insight to investors in that it allows to understand
<em>the potential impact of [any] investment choice, irrespective of investment behavior</em><sup id="fnref:31:3" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup>.</p>
<p>For example, let’s suppose a French investor would like to passively invest in an MSCI World ETF, but is worried by both<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">32</a></sup>:</p>
<ul>
<li>The massive ~70% weight of the United States in that index</li>
<li>The ridiculous ~3% weight of his home country in that index</li>
</ul>
<p>Thus, this investor would rather like to invest in an MSCI World ETF “tilted” away from the United States and toward France, although he has no precise idea about how to implement such a tilt.</p>
<p>Here, returns dispersion analysis through random portfolios can help our investor understand the impact on performances of his proposed tactical deviation from the baseline investment strategy, at least historically<sup id="fnref:38" role="doc-noteref"><a href="#fn:38" class="footnote">33</a></sup>.</p>
<p>For this, Figure 7 depicts the range of annualized returns achieved over the period 31 December 2008 - 29 December 2023 by 10000 random portfolios invested in the 23 countries of the MSCI World index when<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">34</a></sup>:</p>
<ul>
<li>The weights of the United States and France are made randomly varying between 0% and 100%</li>
<li>The weights of all the other countries are kept constant v.s. their reference weights</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-msci-world-france-tilt.png"><img src="/assets/images/blog/random-portfolio-benchmarking-msci-world-france-tilt.png" alt="Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023." /></a>
<figcaption>Figure 7. Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023.</figcaption>
</figure>
<p>It is clear from from Figure 7 that deviating from the MSCI World index by under-weighting the United States and over-weighting France has been a relatively bad strategy over the considered period,
with a median annualized return of ~8.30% compared to an annualized return of ~11.30% for the MSCI World<sup id="fnref:37" role="doc-noteref"><a href="#fn:37" class="footnote">35</a></sup>!</p>
<p>Whether history will repeat itself remains to be seen, but thanks to this historical returns dispersion analysis, our investor is at least informed to take an educated decision re. his proposed investment strategy.</p>
<h3 id="synthetic-benchmark-construction">Synthetic benchmark construction</h3>
<p>Stein<sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to use random portfolios in order to construct a “synthetic” benchmark, representative of all possible investment strategies within a given universe
of assets and subject to a given set of constraints and rules.</p>
<p>In more details, Stein<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to construct such a benchmark directly from <em>the time series of returns of a [well chosen] single random portfolio</em><sup id="fnref:1:6" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, which is typically the median<sup id="fnref:4:12" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> random portfolio
w.r.t. a chosen performance measure.</p>
<p>A similar idea is discussed in Lisi<sup id="fnref:23:6" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>, in which such a benchmark is this time constructed from the cross-sectional returns of the random portfolios.</p>
<p>Figure 8 illustrates Lisi’s methodology; on it, the green line represents a synthetic benchmark made of the 95th percentile<sup id="fnref:45" role="doc-noteref"><a href="#fn:45" class="footnote">36</a></sup> of the distribution of the random portfolios holding period returns at each time $t$</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-lisi.png"><img src="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-lisi.png" alt="Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi." /></a>
<figcaption>Figure 8. Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi.</figcaption>
</figure>
<p>The possibility to create synthetic benchmarks is particularly useful when evaluating the performances of tactical asset allocation (TAA) strategies,
because their allocation might completely change from one period to another, making their comparison with a static benchmark - like a buy and hold portfolio or a 60/40 stock-bond portfolio - difficult to justify <em>a priori</em>.</p>
<p>For example, Figure 9 compares over the period 30th November 2006 - 31th January 2024<sup id="fnref:42" role="doc-noteref"><a href="#fn:42" class="footnote">37</a></sup>:</p>
<ul>
<li>The <em>Global Tactical Asset Allocation (GTAA)</em><sup id="fnref:39" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup> strategy of <a href="https://mebfaber.com/">Mebane Faber</a>, which invests monthly - depending on the quantitative rules described in Faber<sup id="fnref:39:1" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup> - within a five-asset universe made of:
<ul>
<li>U.S. Equities, represented by the SPY ETF</li>
<li>International Equities, represented by the EFA ETF</li>
<li>Intermediate-Term U.S. Treasury Bonds, represented by the IEF ETF</li>
<li>U.S. Real Estate, represented by the VNQ ETF</li>
<li>Global Commodities, represented by the DBC ETF</li>
</ul>
</li>
<li>An (equal-weighted) buy and hold portfolio within Faber’s five-asset universe, which is the benchmark of the <em>GTAA</em> strategy proposed in Faber<sup id="fnref:39:2" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup></li>
<li>A synthetic benchmark <em>à la Lisi<sup id="fnref:23:7" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup></em> constructed from the median cross-sectional holding period returns of 25000 random portfolios simulated<sup id="fnref:41" role="doc-noteref"><a href="#fn:41" class="footnote">39</a></sup> within Faber’s five-asset universe</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-gtta.png"><img src="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-gtta.png" alt="Global Tactical Asset Allocation (GTAA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024." /></a>
<figcaption>Figure 9. Global Tactical Asset Allocation (GTAA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024.</figcaption>
</figure>
<p>In Figure 9, the buy and portfolio and the synthetic benchmark both appears to behave quite similarly<sup id="fnref:48" role="doc-noteref"><a href="#fn:48" class="footnote">40</a></sup>, but the synthetic benchmark should theoretically be preferred for performance comparison purposes
because it better reflects the dynamics of the <em>GTAA</em> strategy (varying asset weights, varying exposure) while guaranteeing the complete absence of skill<sup id="fnref:40" role="doc-noteref"><a href="#fn:40" class="footnote">41</a></sup>.</p>
<h3 id="trading-strategy-evaluation">Trading strategy evaluation</h3>
<p>Random portfolios also find applications in evaluating (quantitative) trading strategies.</p>
<p>One of these applications, described in Dawson<sup id="fnref:9:6" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>, consists in evaluating the significance of an alpha signal through the comparison of two different samples of random portfolios:</p>
<blockquote>
<p>It is […] very difficult to show conclusively the effect that a model, strategy […] has on the real investment process. […] The ideal solution would be to generate a set of portfolios, constrained in the same way as the portfolio(s) built with the theory or model under investigation, only without the information produced by the theory. It would then be possible to compare the distributions of the portfolio characteristics with and without the added information from the new theory, giving strong statistical evidence of the effects of the new information.</p>
</blockquote>
<p>As a practical illustration, let’s come back on Faber’s <em>GTAA</em> strategy introduced in the previous subsection and compare:</p>
<ul>
<li>A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested<sup id="fnref:44" role="doc-noteref"><a href="#fn:44" class="footnote">42</a></sup> in all the assets selected for investment by the <em>GTAA</em> strategy rules</li>
<li>A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested<sup id="fnref:44:1" role="doc-noteref"><a href="#fn:44" class="footnote">42</a></sup> in all the assets NOT selected for investment by the <em>GTAA</em> strategy rules</li>
</ul>
<p>The resulting ranges of annualized returns over the period 30th November 2006 - 31th January 2024 are displayed in Figure 10, which highlights a clear under-performance of the second sample of random portfolios.</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-evaluation-gtta.png"><img src="/assets/images/blog/random-portfolio-benchmarking-evaluation-gtta.png" alt="Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTAA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024." /></a>
<figcaption>Figure 10. Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTAA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024.</figcaption>
</figure>
<p>This kind of under-performance is exactly what is expected under the hypothesis that the <em>GTAA</em> strategy asset selection rules correspond to a true alpha signal.</p>
<p>Indeed, as Arnott et al.<sup id="fnref:28:3" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup> puts it:</p>
<blockquote>
<p>In inverting the strategies, we tacitly examine whether these strategies outperform because they are predicated on meaningful investment theses and deep insights on capital markets, or for reasons unrelated to the investment theses. If the investment beliefs are the source of outperformance, then contradicting those beliefs should lead to underperformance.</p>
</blockquote>
<h3 id="trading-strategy-monitoring">Trading strategy monitoring</h3>
<p>Going beyond trading strategy evaluation, random portfolios can also be used to monitor how the performances of a trading strategy differ between in-sample and out-of-sample periods.</p>
<p>I propose to illustrate the associated process with another tactical asset allocation strategy, the <em>Global Equities Momentum (GEM)</em><sup id="fnref:46" role="doc-noteref"><a href="#fn:46" class="footnote">43</a></sup> strategy of <a href="https://dualmomentum.net/">Gary Antonacci</a>, which invests monthly -
depending on the quantitative rules described in Antonacci<sup id="fnref:46:1" role="doc-noteref"><a href="#fn:46" class="footnote">43</a></sup> - in one asset among a three-asset universe made of:</p>
<ul>
<li>U.S. Equities, represented by the S&P 500 Index</li>
<li>International Equities, represented by the MSCI ACWI ex-USA Index</li>
<li>U.S. Bonds, represented by the Barclays Capital US Aggregate Bond Index</li>
</ul>
<p>Because this strategy was published in 2013, let’s first check <em>GEM</em> performances during the in-sample period 1989-2012 (<a href="https://docs.google.com/spreadsheets/d/1jCLJNg8gHs_e1snq7wAuTP03ZUK_b1rUCXUto0_JRHU/edit?usp=sharing">Google Sheet</a>).</p>
<p>Figure 11 depicts the equity curves of a couple of random portfolios simulated<sup id="fnref:47" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe (in solid) v.s. the GEM equity curve (in dashed) over that period.</p>
<figure>
<a href="/assets/images/blog/gem-random-portfolios.png"><img src="/assets/images/blog/gem-random-portfolios.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012" /></a>
<figcaption>Figure 11. Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012.</figcaption>
</figure>
<p>For the real thing, Figure 12 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated<sup id="fnref:47:1" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe over that period, with the
red line corresponding to the <em>GEM</em> Sharpe Ratio.</p>
<figure>
<a href="/assets/images/blog/gem-sharpe-ratios-in-sample.png"><img src="/assets/images/blog/gem-sharpe-ratios-in-sample.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012" /></a>
<figcaption>Figure 12. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012.</figcaption>
</figure>
<p>Pretty amazing, as the <em>GEM</em> Sharpe Ratio is among the best obtainable Sharpe Ratios over the in-sample period!</p>
<p>Let’s now check <em>GEM</em> performances during the out-of-sample period 2013-October 2020 (<a href="https://docs.google.com/spreadsheets/d/1mzus6uIV9JGwELAO_1sAtv1gB9iAX3FQd5LJJ5P61Go/edit?usp=sharing">Google Sheet</a>).</p>
<p>Figure 13 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated<sup id="fnref:47:2" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe over that new period, with the
red line again corresponding to the <em>GEM</em> Sharpe Ratio.</p>
<figure>
<a href="/assets/images/blog/gem-sharpe-ratios-out-of-sample.png"><img src="/assets/images/blog/gem-sharpe-ratios-out-of-sample.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020" /></a>
<figcaption>Figure 13. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020.</figcaption>
</figure>
<p>Pretty blah this time, with the <em>GEM</em> Sharpe Ratio roughly comparable to the median random portfolio Sharpe Ratio, which by definition exhibits no particular skill…</p>
<p>Such a difference in the <em>GEM</em> Sharpe Ratio <em>relative to its simulated peer group</em><sup id="fnref:4:13" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> between the in-sample period and the out-of-sample period is puzzling, but analyzing a particular tactical asset
allocation strategy is out of scope of this blog post, so that I need to leave the “why” question unanswered.</p>
<p>The same process can be applied to any trading strategy in order to detect a potential shift in that strategy’s performances, so, don’t hesitate to abuse the computing power available with today’s computers!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Dawson<sup id="fnref:9:7" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> notes that <em>[Monte Carlo analysis] is not a tool that has been readily applied to the investment process in the past, due to the perceived complexity of the problem</em><sup id="fnref:9:8" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<p>Through this blog post, I hope to have demonstrated that random portfolios are not necessarily complex to use, and come with many benefits for performance evaluation.</p>
<p>Feel free to randomly reach out <a href="https://www.linkedin.com/in/roman-rubsamen/">on LinkedIn</a> or <a href="https://twitter.com/portfoliooptim">on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://jpm.pm-research.com/content/32/4/54">Surz, Ronald, A Fresh Look at Investment Performance Evaluation: Unifying Best Practices to Improve Timeliness and Reliability, Journal of Portfolio Management, Vol. 32, No. 4, Summer 2006, pp 54-65</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:4:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:4:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:4:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:4:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:4:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:4:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:4:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:4:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:4:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Or a trading strategy, or whatever. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijinvest/3/2/36">Surz, R. J. 1994. Portfolio opportunity distributions: an innovation in performance evaluation. The Journal of Investing, 3(2): 36-41</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://ppca-inc.com/pdf/accurate-benchmarking.pdf">Surz, Ron. Accurate Benchmarking is Gone But Not Forgotten: The Imperative Need to Get Back to Basics, Journal of Performance Measurement, Vol. 11, No. 3, Spring, pp 34-43</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/10293523.2014.11082564">Roberto Stein, Not fooled by randomness: Using random portfolios to analyse investment funds, Investment Analysts Journal, 43:79, 1-15</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:1:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>By construction. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>These <em>can be imposed by the firm that offers the funds, for example in terms of the prospectus and investment goals, or self-imposed trading behavior that the manager maintains over his career</em><sup id="fnref:1:7" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>; these can also be imposed by regulatory bodies or stock exchanges. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/B9780750654487500082">Dawson, R. and R. Young: 2003, Near-uniformly Distributed, Stochastically Generated Portfolios. In: S. Satchell and A. Scowcroft (eds.): Advances in Portfolio Construction and Implementation. Butterworth–Heinemann</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:9:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:9:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:9:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:9:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:9:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:9:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:9:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:9:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>The $3$-dimensional geometrical object associated to the subset of $\mathbb{R}^3$ modeling long-only and full investment constraints is <a href="https://en.wikipedia.org/wiki/Simplex#Standard_simplex">the standard 2-simplex in $\mathbb{R}^3$</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:11:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/3-540-36626-1_11">Burns, P. (2007). Random Portfolios for Performance Measurement. In: Kontoghiorghes, E.J., Gatu, C. (eds) Optimisation, Econometric and Financial Analysis. Advances in Computational Management Science, vol 9. Springer, Berlin, Heidelberg</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:10:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:10:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:10:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:10:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>Such an object is also called a unit simplex or the probabilistic simplex. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s10479-009-0567-7">Onn, S., Weissman, I. Generating uniform random vectors over a simplex with implications to the volume of a certain polytope and to multivariate extremes. Ann Oper Res 189, 331–342 (2011)</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0167715299000954?via%3Dihub">Kai-Tai Fang, Zhen-Hai Yang, On uniform design of experiments with restricted mixtures and generation of uniform distribution on some domains, Statistics & Probability Letters, Volume 46, Issue 2, 2000, Pages 113-120</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/03610918408812382">Paul A. Rubin (1984) Generating random points in a polytope, Communications in Statistics - Simulation and Computation, 13:3, 375-396</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221714005396">Gert van Valkenhoef, Tommi Tervonen, Douwe Postmus, Notes on “Hit-And-Run enables efficient weight generation for simulation-based multiple criteria decision analysis”, European Journal of Operational Research, Volume 239, Issue 3, 2014, Pages 865-867</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>The <a href="https://en.wikipedia.org/wiki/Rejection_sampling">rejection method</a> is not a reasonable approach because <em>the probability of a portfolio being accepted is generally extremely small when realistic constraints are in place</em><sup id="fnref:10:6" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Due to the nature of genetic algorithms. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>Within a given numerical tolerance. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/14697688.2010.513337">Francesco Lisi (2011) Dicing with the market: randomized procedures for evaluation of mutual funds, Quantitative Finance, 11:2, 163-172</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:23:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:23:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:23:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:23:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:23:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:23:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:23:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>Some authors like Kritzman and Page<sup id="fnref:31:4" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup> consider the usage of random portfolios to be a bootstrap simulation process, and not a Monte Carlo one. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>To be noted that the universe of assets, constraints and rules might perfectly be time-dependent; for example, at a given point in time, the universe of assets might be completely different from that at an earlier or later point in time. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>The frontier between the universe of assets and the rebalancing rules might not always be perfectly clear; at heart, the rebalancing rules must model the trading behaviour of the fund. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=75871">Kothari, S.P. and Warner, Jerold B., Evaluating Mutual Fund Performance (August 1997)</a>. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijwealthmgmt/7/4/78">Ronald J. Surz, Testing the Hypothesis “Hedge Fund Performance Is Good”, The Journal of Wealth Management, Spring 2005, 7 (4) 78-83</a>. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:32:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:32:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:32:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=630123">Burns, Patrick J., Performance Measurement Via Random Portfolios (December 2, 2004)</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:26:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/39/4/91">Robert D. Arnott, Jason Hsu, Vitali Kalesnik, Phil Tindall, The Surprising Alpha From Malkiel’s Monkey and Upside-Down Strategies, The Journal of Portfolio Management, Summer 2013, 39 (4) 91-105</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:28:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:28:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:28:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://doi.org/10.1093/biomet/asx076">N A Heard, P Rubin-Delanchy, Choosing between methods of combining p-values, Biometrika, Volume 105, Issue 1, March 2018, Pages 239–246</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>I have no affiliation. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:21:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>See <a href="https://doi.org/10.1057/jam.2016.3">Molyboga, M., Ahelec, C. A simulation-based methodology for evaluating hedge fund investments. J Asset Manag 17, 434–452 (2016)</a>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/29/4/11">Kritzman, Mark and Sébastien Page (2003), The Hierarchy of Investment Choice, Journal of Portfolio Management 29, number 4, pages 11-23.</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:31:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:31:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:31:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:31:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/31/1/118">Staub, R. (2004). The Hierarchy of Investment Choice. The Journal of Portfolio Management, 31(1), 118–123</a>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>C.f. <a href="https://www.msci.com/documents/10199/178e6643-6ae6-47b9-82be-e1fc565ededb">the MSCI World index factsheet</a>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:38" role="doc-endnote">
<p>If future asset prices are available, for example thanks to <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">a bootstrap simulation</a>, nothing prevents a returns dispersion analysis to integrate them. <a href="#fnref:38" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>In more details, the methodology is as follows: 1) Gross USD monthly price data for all the 23 countries represented in the MSCI World index has been collected from <a href="https://www.msci.com/end-of-day-data-country">the MSCI website</a>, 2) The MSCI World <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">tracking portfolio</a> - exhibiting a nearly null tracking error - has been computed over the period 31 December 2008 - 29 December 2023, which gives reference weights for the 23 countries represented in the MSCI World (e.g., United States ~50% and France ~5%), 3) Using <strong>Portfolio Optimizer</strong>, the evolution of 10000 random portfolios has been simulated over the period 31 December 2008 - 29 December 2023, these portfolios being a) constrained so that all country weights are positive, sum to one and all country weights except these for the United States and France are kept constant v.s. their reference weights and b) monthly rebalanced toward random portfolios in order to encompass any possible tilting, 4) The annualized return of each of these 10000 random portfolios has been computed. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:37" role="doc-endnote">
<p>Which, in addition, is near the top of the achievable annualized returns! <a href="#fnref:37" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:45" role="doc-endnote">
<p>To be noted that in practice, this 95th quantile would probably need to be replaced by the 50th quantile because <em>the benchmark return [should] always ranks median</em><sup id="fnref:4:14" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>. <a href="#fnref:45" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:42" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:42" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:39" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=962461">Faber, Meb, A Quantitative Approach to Tactical Asset Allocation (February 1, 2013). The Journal of Wealth Management, Spring 2007</a>. <a href="#fnref:39" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:39:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:39:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:41" role="doc-endnote">
<p>In more details, using <strong>Portfolio Optimizer</strong>, the evolution of 25000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a random exposure between 0% and 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. <a href="#fnref:41" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:48" role="doc-endnote">
<p>Which justifies <em>a posteriori</em> the use of the buy and hold portfolio as a benchmark. <a href="#fnref:48" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:40" role="doc-endnote">
<p>As a side note, Figure 9 highlights that an equal-weighted buy and hold portfolio is a tough benchmark to beat, c.f. DeMiguel et al.<sup id="fnref:43" role="doc-noteref"><a href="#fn:43" class="footnote">45</a></sup>! <a href="#fnref:40" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:44" role="doc-endnote">
<p>Long-only and fully invested. <a href="#fnref:44" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:44:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:46" role="doc-endnote">
<p>Gary Antonacci, Dual Momentum Investing: An Innovative Strategy for Higher Returns With Lower Risk <a href="#fnref:46" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:46:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:47" role="doc-endnote">
<p>In more details, using <strong>Portfolio Optimizer</strong>, the evolution of 10000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. <a href="#fnref:47" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:47:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:47:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:43" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=1376199">DeMiguel, Victor and Garlappi, Lorenzo and Uppal, Raman, Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (May 2009). The Review of Financial Studies, Vol. 22, Issue 5, pp. 1915-1953, 2009</a>. <a href="#fnref:43" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.As noted in Surz1, the question “Is [a mutual fund’s]2 performance good?” can only be answered relative to something1, typically by comparing that fund to a benchmark like a financial index or to a peer group. Unfortunately, these two methodologies are not without issues. For example, it is very difficult to create an index captur[ing] the essence of the people, process, and philosophy behind an investment product3 and peer groups have well-known biases1 (classification bias, survivorship bias…). In a series of papers (Surz3, Surz4, Surz1…), Ronald J. Surz proposes an innovation that combines the better aspects of both [methodologies] while eliminating their undesirable properties3. This innovation consists in evaluating a fund against the fund manager’s true opportunity set5, defined as the set of all of the possible portfolios that the manager could have conceivably held following his unique investment process4. In practice, the fund manager’s opportunity set is approximated by the simulation of thousands of random portfolios in the same universe of assets as the one of the fund manager and satisfying the same constraints (long-only, long-short…) and rules (portfolio rebalancing rules…) as those of the fund manager. Then, because these portfolios do not exhibit any particular skill6, they can be used as the control group to test [the] fund manager skill5, thus allowing to apply modern statistics to the problem of performance evaluation4. In this blog post, I will describe how to generate random portfolios, detail Surz’s original methodology as well as some of its variations and illustrate the usage of random portfolios with a couple of examples like the creation of synthetic benchmarks or the evaluation and monitoring of trading strategies. Mathematical preliminaries Definition Let be: $n$ the number of assets in a universe of assets $\mathcal{C} \subset \mathbb{R}^{n}$ a subset of $ \mathbb{R}^{n}$ representing the constraints imposed7 on a fund manager when investing in that universe of assets, for example: Short sale constraints Concentration constraints (assets, sectors, industries…) Leverage constraints Cardinality constraints (i.e., minimum or maximum number of assets constraints) Portfolio volatility constraints Portfolio tracking error constraints … Then, a (constrained) random portfolio in that universe of assets is a vector of portfolio weights $w \in \mathbb{R}^{n}$ generated at random over the set $\mathcal{C}$. Generation at random v.s. generation uniformly at random In the context of performance evaluation, as in Surz’s methodology, it is theoretically preferable that random portfolios are generated uniformly at random over the set $\mathcal{C}$, so as not to introduce any biases8 which would otherwise defeat the purpose of using these portfolios as an unbiased control group3. Figure 1 and Figure 2 illustrate the difference between random portfolios generated at random v.s. uniformly at random in the case of a three-asset universe subject to long-only and full investment constraints9: In Figure 1, the random portfolios are visibly concentrated in the middle of the standard 2-simplex in $\mathbb{R}^3$9 - these random portfolios are NOT generated uniformly at random In Figure 2, the random portfolios seem to be “well spread” over the standard 2-simplex in $\mathbb{R}^3$9 - these random portfolios are generated uniformly at random Figure 1. Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw. Figure 2. Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw. One important remark, though, is that real-life portfolios are usually binding on at least one of their constraints, so that generating random portfolios biased toward the boundary of the geometrical object associated to the constraints set $\mathcal{C}$ might not be a real problem in practice10. Generation of random portfolios over the standard simplex When the constraints imposed on a portfolio are 1) a full investment constraint and 2) a long-only constraint, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, w_i \geq 0, i = 1..n\}\] and the geometrical object associated to that subset is called11 a standard simplex, already illustrated in Figure 1 and Figure 2 with $n = 3$. Several algorithms exist to generate points uniformly at random over a standard simplex12, among which: An algorithm based on differences of sorted uniform random variables An algorithm based on normalized unit exponential random variables Generation of random portfolios over the restricted standard simplex When the constraints imposed on a portfolio are 1) a full investment constraint, 2) a long-only constraint and 3) minimum/maximum asset weights constraints, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, 1 \geq u_i \geq w_i \geq l_i \geq 0, i = 1..n\}\] , where: $l_1, …, l_n$ represent minimum asset weights constraints $u_1, …, u_n$ represent maximum asset weights constraints The geometrical object associated to that subset is a “restricted” standard simplex, as illustrated in Figure 3, with: $n = 3$ $ 0.7 \geq w_1 \geq 0.1$ $ 0.8 \geq w_2 \geq 0 $ $ 0.6 \geq w_3 \geq 0.1 $ Figure 3. Example of restricted standard simplex, three-asset universe. Source: Piepel. Generating points uniformly at random over a restricted standard simplex is much more complex than generating points uniformly at random over a standard simplex. Hopefully, this problem has been studied at least since the 1990’s by people working in the statistical domain of the design of experiments, and an algorithm based on the conditional distribution method has been published13 in 2000. Generation of random portfolios over a convex polytope When the constraints imposed on a portfolio consist in generic linear constraints, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } A_e w = b_e, A_i w \leq b_i \}\] , where: $A_e \in \mathbb{R}^{n_e \times n}$ and $b_e \in \mathbb{R}^{n_e}$, $n_e \geq 1$, represent linear equality constraints $A_i \in \mathbb{R}^{n_i \times n}$ and $b_i \in \mathbb{R}^{n_i}$, $n_i \geq 1$, represent linear inequality constraints The geometrical object associated to that subset is called a convex polytope, illustrated in Figure 4, with: $n = 3$ $\sum_{i=1}^n w_i = 1$ $1 \geq w_i \geq 0, i = 1..n$ $w_1 > w_2$ $2w_3 > w_2 > 0.5w_3$ Figure 4. Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al. As one can guess, generating points uniformly at random over a convex polytope is another level higher in terms of complexity, and while some algorithms exist to do so14, they are impractical in high dimension. From the literature, what is possible to achieve instead is to generate points asymptotically uniformly at random, using Markov chain Monte Carlo (MCMC) algorithms like the Hit-And-Run algorithm15. Generation of random portfolios over a generic constraint set When the constraints imposed on a portfolio are generic, that is, quadratic (volatility or tracking error constraints) and/or non-convex (threshold constraints) and/or integer (maximum number of assets constraints, round lot constraints), the final boss has arrived. With such constraints, the only reasonable16 approach in the literature seems to recast the problem of generating random points over the constraint set $C$ as a (penalized) optimization problem and use genetic algorithms to solve it10. The underlying idea is to randomly17 minimize an objective function $f$ essentially made of penalties - the higher the distance of a point $w \in \mathbb{R}^{n}$ from the constraint set $C$, the higher the value of the objective function $f(w)$ -, so that at optimum, a random point satisfying all the constraints18 is found. Of course, the points generated this way are not generated uniformly at random, but when the constraints are fully generic, that requirement should probably be dropped altogether as mentioned in Dawson and Young8. Investment fund’s performance evaluation with random portfolios Rationale Evaluating the performance of an investment fund is actually a statistical hypothesis test in disguise19, in which: The null hypothesis is The investment fund’s exhibit no particular “performance” over the considered time period The test statistic is a quantitative measure of “performance” over the considered time period (e.g. annualized return, holding period return, risk-adjusted return…) The (empirical) distribution of the test statistic under the null hypothesis is computed from a sample made of either One observation - the investment fund’s benchmark A small number of observations - the investment fund’s peer group Any desired number of observations - random portfolios generated from the investment fund’s universe of assets and obeying to the fund’s constraints and rules From this perspective, using a benchmark, a peer group or random portfolios for performance evaluation is essentially a choice between sampling approaches1. Still, random portfolios represent a more rigorous approach to performance evaluation than the two other alternatives, for various reasons highlighted in different papers181019 and summarized by Dawson8 as follows A set of uniformly distributed, stochastically generated, portfolios that by construction incorporate no investment strategy, bias or skill form an effective control set for any portfolio measurement metric. This allows the true value of a strategy or model to be identified. They also provide a mechanism to differentiate between effects due to “market conditions” and effects due to either the management of a portfolio, or the constraints the management is obliged to work within. Surz’s methodology Using random portfolios to evaluate an investment fund’s performance over a considered time period is a two-step Monte Carlo simulation process20: Modeling step Identify the fund’s main characteristics. Computational step Random portfolios simulation Generate random portfolios compatible with the identified fund’s main characteristics and simulate their evolution through the considered time period. Performance evaluation Determine the level of statistical significance of the fund’s performance over the considered time period using the previously simulated random portfolios. Step 1 - Identifying the fund’s main characteristics Dawson8 note that an investment strategy is essentially a set of constraints [and rules] that are carefully constructed to (hopefully) remove, in aggregate, poor performing regions of the solution space8. Thus, it is very important to identify as precisely as possible the main characteristics of the fund, even though this […] information is not always available, as only the manager knows [it] exactly19. From a practical perspective, these characteristics need to be provided as21: A universe of $n$ assets modeling the fund’s universe of assets A constraint set $\mathcal{C} \subset \mathbb{R}^{n}$ modeling the fund’s constraints An ensemble of decision rules modeling22 the fund’s rebalancing rules For examples of this step in different contexts, c.f. Kothari and Warner23 and Surz24. Step 2a - Simulating random portfolios Thanks to the previous step, it becomes possible to simulate thousands of random portfolios that the fund’s manager could have potentially held over the considered time period. The only difficulty at this point is computational, due to the impact of the constraint set on the algorithmic machinery required to generate portfolio weights at random. As a side note, and for a fair comparison [with the fund under analysis], transaction costs, as well as all other kinds of costs, must be considered10 when simulating random portfolios. Step 2b - Evaluating the fund’s performance The ranking of the fund’s performance against the performance of all the simulated random portfolios provides a direct measure of the statistical significance of that fund’s performance1. Indeed, under the null hypothesis that the investment fund’s exhibit no particular performance over the considered time period, the $p$-value of the statistical hypothesis test mentioned at the beginning of this section is defined as25 \[p = \frac{n_x + 1}{N + 1}\] , where: $n_x$ is the the number of random portfolios whose performance over the considered time period is as extreme or more extreme than that of the fund under evaluation $N$ is the number of simulated random portfolios This idea is illustrated in Figure 5, which depicts: The distribution of the holding period return of 1000 random portfolios made of stocks belonging to the S&P 500 and subject to misc. turnover and cardinality constraints (solid line) The 95th percentile of that distribution (dashed line) Figure 5. Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein. From this figure, if we were to evaluate the performances of a fund trading these securities and operating under constraints similar to those simulated, [it] would have to obtain a return equal of at least 80% during the 6 year period for the null of no particular performance to be rejected5 at a 95% confidence level. Variations on Surz’s methodology Burns1025 and Lisi19 both extend Surz’s methodology from one time period to several time periods - which allows to consider the persistence of results over time19 - by describing different ways of combining individual $p$-values26. As a by-product, Burns1025 empirically demonstrates that Stouffer’s method27 for combining $p$-values should be preferred to Fisher’s when evaluating a fund’s performance over multiple time periods. Stein5 proposes to replace Surz’s methodology by testing whether the distribution of returns of the [fund under evaluation] is stochastically greater than that of [a] chosen percentile random fund using the Mann–Whitney U test. Caveats Surz1 warns that there are many ways to implement a [Monte Carlo simulation] approach [with random portfolios], some better than others, and some worse1. In particular, Surz1 argues that, when selecting a number of assets at random to satisfy portfolio cardinality constraints, it is required to use value-weighted sampling, so that the probability of choosing a given [asset] is proportionate to its outstanding capitalization1. Otherwise, some macroeconomic [in]consistency1 would be introduced by the Monte Carlo simulation process, which would ultimately bias the end results. This point is confirmed for both U.S. and global stocks in Arnott et al.26, who show that random portfolios introduce, often unintentionally, value and small cap tilts26. Nevertheless, other authors19 argue on the contrary that using equally-weighted sampling is more representative of the behavior of an “unskilled” manager, which is exactly what random portfolios are supposed to model! More generally, random portfolios might have “unfair” characteristics v.s. the fund under evaluation (higher volatility, higher beta, lower turnover…), which must either be acknowledged or controlled for depending on the exact circumstances. Implementations Implementation in Portfolio Optimizer Through the endpoint /portfolios/simulation/random, Portfolio Optimizer allows to: Generate the weights of a random portfolio subject to general linear inequality constraints Through the endpoint /portfolios/simulation/evolution/random, Portfolio Optimizer allows to: Simulate the evolution of a random portfolio subject to general linear inequality constraints over a specified time period, with 3 different portfolio rebalancing strategies: Buy and hold Rebalancing toward the weights of the initial random portfolio Rebalancing toward the weights of a new random portfolio Implementations elsewhere I am aware of two commercial implementations of random portfolios for performance evaluation as described in this blog post: Portfolio Probe, from Patrick Burns28 of Burns Statistics PIPODs, from Ronald Surz28 of PPCA Inc., which seems discontinued Examples of usage Hedge funds performance evaluation The performances of hedge funds are notoriously difficult to evaluate due to the lack of both proper benchmarks and homogeneous peer groups24. Random portfolios offer a solution to this problem because they can reflect the unique specifications of each individual hedge fund24. More details can be found in Surz24 or in Molyboga and Ahelec29. Investment strategy returns dispersion analysis Kritzman and Page30 uses random portfolios to compare the relative importance of different investment strategies (investment at global asset class level, at individual stock level, etc.) and concludes that30 Contrary to perceived doctrine, dispersion around average performance arising from security selection is substantially greater than dispersion around average performance arising from all other investment choices. Moreover, asset allocation, widely considered the most important investment choice, produces the least dispersion; thus, from a normative perspective it is the least important investment choice. Figure 6, which quantifies that dispersion around average performance, shows the extent to which a talented investor (top 25th or 5th percentile) could have improved upon average performance by engaging in various investment choices across a global universe. It also shows how far below average an unlucky investor (bottom 75th or 95th percentile) could have performed, depending on the choice of investment discretion30. Figure 6. 5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page. While this conclusion has been criticized31, the underlying methodology - returns dispersion analysis - provides valuable insight to investors in that it allows to understand the potential impact of [any] investment choice, irrespective of investment behavior30. For example, let’s suppose a French investor would like to passively invest in an MSCI World ETF, but is worried by both32: The massive ~70% weight of the United States in that index The ridiculous ~3% weight of his home country in that index Thus, this investor would rather like to invest in an MSCI World ETF “tilted” away from the United States and toward France, although he has no precise idea about how to implement such a tilt. Here, returns dispersion analysis through random portfolios can help our investor understand the impact on performances of his proposed tactical deviation from the baseline investment strategy, at least historically33. For this, Figure 7 depicts the range of annualized returns achieved over the period 31 December 2008 - 29 December 2023 by 10000 random portfolios invested in the 23 countries of the MSCI World index when34: The weights of the United States and France are made randomly varying between 0% and 100% The weights of all the other countries are kept constant v.s. their reference weights Figure 7. Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023. It is clear from from Figure 7 that deviating from the MSCI World index by under-weighting the United States and over-weighting France has been a relatively bad strategy over the considered period, with a median annualized return of ~8.30% compared to an annualized return of ~11.30% for the MSCI World35! Whether history will repeat itself remains to be seen, but thanks to this historical returns dispersion analysis, our investor is at least informed to take an educated decision re. his proposed investment strategy. Synthetic benchmark construction Stein5 proposes to use random portfolios in order to construct a “synthetic” benchmark, representative of all possible investment strategies within a given universe of assets and subject to a given set of constraints and rules. In more details, Stein5 proposes to construct such a benchmark directly from the time series of returns of a [well chosen] single random portfolio5, which is typically the median1 random portfolio w.r.t. a chosen performance measure. A similar idea is discussed in Lisi19, in which such a benchmark is this time constructed from the cross-sectional returns of the random portfolios. Figure 8 illustrates Lisi’s methodology; on it, the green line represents a synthetic benchmark made of the 95th percentile36 of the distribution of the random portfolios holding period returns at each time $t$ Figure 8. Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi. The possibility to create synthetic benchmarks is particularly useful when evaluating the performances of tactical asset allocation (TAA) strategies, because their allocation might completely change from one period to another, making their comparison with a static benchmark - like a buy and hold portfolio or a 60/40 stock-bond portfolio - difficult to justify a priori. For example, Figure 9 compares over the period 30th November 2006 - 31th January 202437: The Global Tactical Asset Allocation (GTAA)38 strategy of Mebane Faber, which invests monthly - depending on the quantitative rules described in Faber38 - within a five-asset universe made of: U.S. Equities, represented by the SPY ETF International Equities, represented by the EFA ETF Intermediate-Term U.S. Treasury Bonds, represented by the IEF ETF U.S. Real Estate, represented by the VNQ ETF Global Commodities, represented by the DBC ETF An (equal-weighted) buy and hold portfolio within Faber’s five-asset universe, which is the benchmark of the GTAA strategy proposed in Faber38 A synthetic benchmark à la Lisi19 constructed from the median cross-sectional holding period returns of 25000 random portfolios simulated39 within Faber’s five-asset universe Figure 9. Global Tactical Asset Allocation (GTAA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024. In Figure 9, the buy and portfolio and the synthetic benchmark both appears to behave quite similarly40, but the synthetic benchmark should theoretically be preferred for performance comparison purposes because it better reflects the dynamics of the GTAA strategy (varying asset weights, varying exposure) while guaranteeing the complete absence of skill41. Trading strategy evaluation Random portfolios also find applications in evaluating (quantitative) trading strategies. One of these applications, described in Dawson8, consists in evaluating the significance of an alpha signal through the comparison of two different samples of random portfolios: It is […] very difficult to show conclusively the effect that a model, strategy […] has on the real investment process. […] The ideal solution would be to generate a set of portfolios, constrained in the same way as the portfolio(s) built with the theory or model under investigation, only without the information produced by the theory. It would then be possible to compare the distributions of the portfolio characteristics with and without the added information from the new theory, giving strong statistical evidence of the effects of the new information. As a practical illustration, let’s come back on Faber’s GTAA strategy introduced in the previous subsection and compare: A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested42 in all the assets selected for investment by the GTAA strategy rules A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested42 in all the assets NOT selected for investment by the GTAA strategy rules The resulting ranges of annualized returns over the period 30th November 2006 - 31th January 2024 are displayed in Figure 10, which highlights a clear under-performance of the second sample of random portfolios. Figure 10. Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTAA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024. This kind of under-performance is exactly what is expected under the hypothesis that the GTAA strategy asset selection rules correspond to a true alpha signal. Indeed, as Arnott et al.26 puts it: In inverting the strategies, we tacitly examine whether these strategies outperform because they are predicated on meaningful investment theses and deep insights on capital markets, or for reasons unrelated to the investment theses. If the investment beliefs are the source of outperformance, then contradicting those beliefs should lead to underperformance. Trading strategy monitoring Going beyond trading strategy evaluation, random portfolios can also be used to monitor how the performances of a trading strategy differ between in-sample and out-of-sample periods. I propose to illustrate the associated process with another tactical asset allocation strategy, the Global Equities Momentum (GEM)43 strategy of Gary Antonacci, which invests monthly - depending on the quantitative rules described in Antonacci43 - in one asset among a three-asset universe made of: U.S. Equities, represented by the S&P 500 Index International Equities, represented by the MSCI ACWI ex-USA Index U.S. Bonds, represented by the Barclays Capital US Aggregate Bond Index Because this strategy was published in 2013, let’s first check GEM performances during the in-sample period 1989-2012 (Google Sheet). Figure 11 depicts the equity curves of a couple of random portfolios simulated44 within Antonacci’s three-asset universe (in solid) v.s. the GEM equity curve (in dashed) over that period. Figure 11. Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012. For the real thing, Figure 12 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated44 within Antonacci’s three-asset universe over that period, with the red line corresponding to the GEM Sharpe Ratio. Figure 12. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012. Pretty amazing, as the GEM Sharpe Ratio is among the best obtainable Sharpe Ratios over the in-sample period! Let’s now check GEM performances during the out-of-sample period 2013-October 2020 (Google Sheet). Figure 13 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated44 within Antonacci’s three-asset universe over that new period, with the red line again corresponding to the GEM Sharpe Ratio. Figure 13. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020. Pretty blah this time, with the GEM Sharpe Ratio roughly comparable to the median random portfolio Sharpe Ratio, which by definition exhibits no particular skill… Such a difference in the GEM Sharpe Ratio relative to its simulated peer group1 between the in-sample period and the out-of-sample period is puzzling, but analyzing a particular tactical asset allocation strategy is out of scope of this blog post, so that I need to leave the “why” question unanswered. The same process can be applied to any trading strategy in order to detect a potential shift in that strategy’s performances, so, don’t hesitate to abuse the computing power available with today’s computers! Conclusion Dawson8 notes that [Monte Carlo analysis] is not a tool that has been readily applied to the investment process in the past, due to the perceived complexity of the problem8. Through this blog post, I hope to have demonstrated that random portfolios are not necessarily complex to use, and come with many benefits for performance evaluation. Feel free to randomly reach out on LinkedIn or on Twitter. – See Surz, Ronald, A Fresh Look at Investment Performance Evaluation: Unifying Best Practices to Improve Timeliness and Reliability, Journal of Portfolio Management, Vol. 32, No. 4, Summer 2006, pp 54-65. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 Or a trading strategy, or whatever. ↩ See Surz, R. J. 1994. Portfolio opportunity distributions: an innovation in performance evaluation. The Journal of Investing, 3(2): 36-41. ↩ ↩2 ↩3 ↩4 See Surz, Ron. Accurate Benchmarking is Gone But Not Forgotten: The Imperative Need to Get Back to Basics, Journal of Performance Measurement, Vol. 11, No. 3, Spring, pp 34-43. ↩ ↩2 ↩3 See Roberto Stein, Not fooled by randomness: Using random portfolios to analyse investment funds, Investment Analysts Journal, 43:79, 1-15. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 By construction. ↩ These can be imposed by the firm that offers the funds, for example in terms of the prospectus and investment goals, or self-imposed trading behavior that the manager maintains over his career5; these can also be imposed by regulatory bodies or stock exchanges. ↩ See Dawson, R. and R. Young: 2003, Near-uniformly Distributed, Stochastically Generated Portfolios. In: S. Satchell and A. Scowcroft (eds.): Advances in Portfolio Construction and Implementation. Butterworth–Heinemann. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 The $3$-dimensional geometrical object associated to the subset of $\mathbb{R}^3$ modeling long-only and full investment constraints is the standard 2-simplex in $\mathbb{R}^3$. ↩ ↩2 ↩3 See Burns, P. (2007). Random Portfolios for Performance Measurement. In: Kontoghiorghes, E.J., Gatu, C. (eds) Optimisation, Econometric and Financial Analysis. Advances in Computational Management Science, vol 9. Springer, Berlin, Heidelberg. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 Such an object is also called a unit simplex or the probabilistic simplex. ↩ See Onn, S., Weissman, I. Generating uniform random vectors over a simplex with implications to the volume of a certain polytope and to multivariate extremes. Ann Oper Res 189, 331–342 (2011). ↩ See Kai-Tai Fang, Zhen-Hai Yang, On uniform design of experiments with restricted mixtures and generation of uniform distribution on some domains, Statistics & Probability Letters, Volume 46, Issue 2, 2000, Pages 113-120. ↩ See Paul A. Rubin (1984) Generating random points in a polytope, Communications in Statistics - Simulation and Computation, 13:3, 375-396. ↩ See Gert van Valkenhoef, Tommi Tervonen, Douwe Postmus, Notes on “Hit-And-Run enables efficient weight generation for simulation-based multiple criteria decision analysis”, European Journal of Operational Research, Volume 239, Issue 3, 2014, Pages 865-867. ↩ The rejection method is not a reasonable approach because the probability of a portfolio being accepted is generally extremely small when realistic constraints are in place10. ↩ Due to the nature of genetic algorithms. ↩ Within a given numerical tolerance. ↩ See Francesco Lisi (2011) Dicing with the market: randomized procedures for evaluation of mutual funds, Quantitative Finance, 11:2, 163-172. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 Some authors like Kritzman and Page30 consider the usage of random portfolios to be a bootstrap simulation process, and not a Monte Carlo one. ↩ To be noted that the universe of assets, constraints and rules might perfectly be time-dependent; for example, at a given point in time, the universe of assets might be completely different from that at an earlier or later point in time. ↩ The frontier between the universe of assets and the rebalancing rules might not always be perfectly clear; at heart, the rebalancing rules must model the trading behaviour of the fund. ↩ See Kothari, S.P. and Warner, Jerold B., Evaluating Mutual Fund Performance (August 1997). ↩ See Ronald J. Surz, Testing the Hypothesis “Hedge Fund Performance Is Good”, The Journal of Wealth Management, Spring 2005, 7 (4) 78-83. ↩ ↩2 ↩3 ↩4 See Burns, Patrick J., Performance Measurement Via Random Portfolios (December 2, 2004). ↩ ↩2 ↩3 See Robert D. Arnott, Jason Hsu, Vitali Kalesnik, Phil Tindall, The Surprising Alpha From Malkiel’s Monkey and Upside-Down Strategies, The Journal of Portfolio Management, Summer 2013, 39 (4) 91-105. ↩ ↩2 ↩3 ↩4 See N A Heard, P Rubin-Delanchy, Choosing between methods of combining p-values, Biometrika, Volume 105, Issue 1, March 2018, Pages 239–246. ↩ I have no affiliation. ↩ ↩2 See Molyboga, M., Ahelec, C. A simulation-based methodology for evaluating hedge fund investments. J Asset Manag 17, 434–452 (2016). ↩ See Kritzman, Mark and Sébastien Page (2003), The Hierarchy of Investment Choice, Journal of Portfolio Management 29, number 4, pages 11-23.. ↩ ↩2 ↩3 ↩4 ↩5 See Staub, R. (2004). The Hierarchy of Investment Choice. The Journal of Portfolio Management, 31(1), 118–123. ↩ C.f. the MSCI World index factsheet. ↩ If future asset prices are available, for example thanks to a bootstrap simulation, nothing prevents a returns dispersion analysis to integrate them. ↩ In more details, the methodology is as follows: 1) Gross USD monthly price data for all the 23 countries represented in the MSCI World index has been collected from the MSCI website, 2) The MSCI World tracking portfolio - exhibiting a nearly null tracking error - has been computed over the period 31 December 2008 - 29 December 2023, which gives reference weights for the 23 countries represented in the MSCI World (e.g., United States ~50% and France ~5%), 3) Using Portfolio Optimizer, the evolution of 10000 random portfolios has been simulated over the period 31 December 2008 - 29 December 2023, these portfolios being a) constrained so that all country weights are positive, sum to one and all country weights except these for the United States and France are kept constant v.s. their reference weights and b) monthly rebalanced toward random portfolios in order to encompass any possible tilting, 4) The annualized return of each of these 10000 random portfolios has been computed. ↩ Which, in addition, is near the top of the achievable annualized returns! ↩ To be noted that in practice, this 95th quantile would probably need to be replaced by the 50th quantile because the benchmark return [should] always ranks median1. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ See Faber, Meb, A Quantitative Approach to Tactical Asset Allocation (February 1, 2013). The Journal of Wealth Management, Spring 2007. ↩ ↩2 ↩3 In more details, using Portfolio Optimizer, the evolution of 25000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a random exposure between 0% and 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. ↩ Which justifies a posteriori the use of the buy and hold portfolio as a benchmark. ↩ As a side note, Figure 9 highlights that an equal-weighted buy and hold portfolio is a tough benchmark to beat, c.f. DeMiguel et al.45! ↩ Long-only and fully invested. ↩ ↩2 Gary Antonacci, Dual Momentum Investing: An Innovative Strategy for Higher Returns With Lower Risk ↩ ↩2 In more details, using Portfolio Optimizer, the evolution of 10000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. ↩ ↩2 ↩3 See DeMiguel, Victor and Garlappi, Lorenzo and Uppal, Raman, Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (May 2009). The Review of Financial Studies, Vol. 22, Issue 5, pp. 1915-1953, 2009. ↩Sparse Index Tracking: Limiting the Number of Assets in an Index Tracking Portfolio2024-01-08T00:00:00-06:002024-01-08T00:00:00-06:00https://portfoliooptimizer.io/blog/sparse-index-tracking-limiting-the-number-of-assets-in-an-index-tracking-portfolio<p>In the <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">previous post</a>, I introduced the <em>index tracking problem</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, which consists
in finding a portfolio that tracks as closely as possible<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup> a given financial market index.</p>
<p>Because such a portfolio might contain any number of assets, with for example an <a href="https://en.wikipedia.org/wiki/S%26P_500">S&P 500</a> tracking portfolio possibly containing ~500 stocks, <em>it is [sometimes desirable]
that the tracking portfolio consists of a small number of assets</em><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> in order to <em>simplify the execution, avoid small and illiquid positions, and large transaction costs</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In other words, it is sometimes desirable to impose a constraint on the maximum number of assets contained in an index tracking portfolio, which leads to an extension of the regular index tracking problem called the <em>sparse index tracking problem</em><sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">4</a></sup>.</p>
<p>In this new post, I will describe the mathematics of the sparse index tracking problem and I will detail a few examples of usage like:</p>
<ul>
<li>Replicating a fund of funds to reduce fees while keeping the number of directly-invested funds manageable</li>
<li>Replicating the S&P 500 with only ~50 stocks as an (extreme) illustration of the <em>optimized sampling</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup> ETF methodology</li>
</ul>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="the-general-sparse-index-tracking-optimization-problem">The general sparse index tracking optimization problem</h3>
<p>Let be:</p>
<ul>
<li>$T$, a number of time periods</li>
<li>$r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods</li>
<li>$n$, the number of assets in the universe of the sparse index tracking portfolio</li>
<li>$X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods</li>
<li>$w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets</li>
</ul>
<p>The vector of the sparse index tracking portfolio weights $w^*$ is then the<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> solution to the optimization problem which consists in minimizing a tracking error measure
$f \left( X w - r_{idx} \right)$ - with $f$ some <a href="https://en.wikipedia.org/wiki/Loss_function">loss function</a> - subject to additional constraints like full investment constraint, no short sale constraint, etc. and
a cardinality constraint<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup></p>
\[w^* = \operatorname{argmin} f \left( X w - r_{idx} \right) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\]
<p>, where:</p>
<ul>
<li>$\lVert . \rVert_0$ is the zero “norm”<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">7</a></sup> of a vector, that is, the number of non-zero elements in that vector</li>
<li>$n_{max}$ is the maximum number of assets with non-zero weights desired in the sparse index tracking portfolio</li>
</ul>
<p>From a computational perspective, and whatever the exact definition of the loss function $f$, it is known that <em>when a cardinality constraint restricting the number of stocks is introduced, the problem of optimizing the composition
of a portfolio tends to become <a href="https://en.wikipedia.org/wiki/NP-hardness">NP-hard</a></em><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup>, which <em>means that exact solutions to instances of realistic sizes are computationally intractable, and thus inexact solution methods are the
only practical ones</em><sup id="fnref:8:1" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup>.</p>
<p>Standard quadratic optimization methods guaranteeing an optimal solution are thus unfortunately not usable to solve the sparse index tracking optimization problem<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">9</a></sup>…</p>
<h3 id="heuristics-for-solving-the-general-sparse-index-tracking-optimization-problem">Heuristics for solving the general sparse index tracking optimization problem</h3>
<p>While an exhaustive enumeration of all possible methods for solving the sparse index tracking optimization problem is out of scope of this post, most of them<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">10</a></sup> seem to fall into one of three rough categories.</p>
<h4 id="combinatorial-optimization-methods">Combinatorial optimization methods</h4>
<p>The cardinality constraint present in the sparse index tracking problem makes it a combinatorial problem, so that standard combinatorial optimization methods (continuous relaxation of a mixed integer programming formulation<sup id="fnref:8:2" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">11</a></sup>,
genetic algorithms<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">12</a></sup><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup>…) can be used to find an approximate solution.</p>
<p>As a side note, thanks to the relationship of the sparse index tracking problem with combinatorial optimization and hence with <a href="https://en.wikipedia.org/wiki/Operations_research">operations research</a>,
it is possible to find datasets for different index tracking problems - ranging from a universe of 31 assets to a universe of 2151 assets - in the <a href="http://people.brunel.ac.uk/~mastjjb/jeb/orlib/indtrackinfo.html">OR-Library</a>.</p>
<h4 id="signal-processing-methods">Signal processing methods</h4>
<p>The sparse index tracking problem in finance is closely related to a similar problem in signal processing called <a href="https://en.wikipedia.org/wiki/Compressed_sensing">compressed sensing</a>,
for which specific algorithms have been developed over the years, like the iterative hard thresholding algorithm<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup> or the compressive sampling matching pursuit (CoSaMP) algorithm<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup>.</p>
<p>To be noted, though, that all signal processing methods that rely on the <em>least absolute shrinkage and selection operator (<a href="https://en.wikipedia.org/wiki/Lasso_(statistics)">LASSO</a>)</em>, which consists in approximating
the $l_0$-norm constraint by a $l_1$-norm constraint, are not applicable to the sparse index tracking problem. Indeed, when full investment and no-short sales constraints are imposed<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">16</a></sup>,
we have, for any portfolio weights vector $w$</p>
\[\lVert w \rVert_1 = \sum_{i=1}^n \left| w_i \right| = \sum_{i=1}^n w_i = 1\]
<p>, so that <em>the</em> $l_1$<em>-norm reduces to a constant and is therefore irrelevant</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<h4 id="non-linear-optimization-methods">Non-linear optimization methods</h4>
<p>At the crossroads with signal processing methods, generic non-linear optimization methods taking into account cardinality constraints have also been developed over the years:</p>
<ul>
<li>
<p>Methods that directly integrate cardinality constraints, like projected gradient methods<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">17</a></sup><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">18</a></sup> or coordinate descent methods<sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">18</a></sup></p>
</li>
<li>
<p>Methods that indirectly integrate cardinality constraints by either:</p>
<ul>
<li>Relaxing the $l_0$-norm constraint into an “easier” constraint, like the entropic lower bound algorithm of Jacquet et al.<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">19</a></sup></li>
<li>Reformulating the $l_0$-norm constraint as a $l_0$-norm penalty integrated in the tracking error measure<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">20</a></sup> and replacing that $l_0$-norm penalty with an “easier” penalty, like the $q$-norm approximation algorithm of Jansen and van Dijk<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">21</a></sup> or the <a href="https://en.wikipedia.org/wiki/MM_algorithm">majorization-minimization</a> algorithm of Benidis et al.<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup></li>
</ul>
</li>
</ul>
<h3 id="the-sparse-index-trackingempirical-tracking-error-optimization-problem">The sparse index tracking/empirical tracking error optimization problem</h3>
<p>Like in the <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">previous post of this series</a>, this post will use the empirical tracking error as the preferred tracking error measure.</p>
<p>In this case, the general sparse index tracking optimization problem becomes</p>
\[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\]
<p>, which, as noted by Benidis et al.<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, is nothing else than a constrained sparse regression problem:</p>
<blockquote>
<p>the sparse index tracking problem is similar to many sparsity formulations in the signal processing area in the sense that it is a regression problem with some sparsity requirements</p>
</blockquote>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> allows to compute an approximate solution to the sparse index tracking optimization problem under the empirical tracking error measure through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/replication/index-tracking/sparse</code></a>.</p>
<p>In addition, <strong>Portfolio Optimizer</strong> allows to impose misc. constraints like minimum and maximum asset weights or minimum and maximum group weights constraints<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">22</a></sup>.</p>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="replicating-the-sp-500-by-extreme-optimized-sampling">Replicating the S&P 500 by (extreme) optimized sampling</h3>
<p>One of the most immediate application of the sparse index tracking problem is the ability to replicate a financial market index without investing into all its constituents, which is desirable for the reasons explained in the introduction of this post.</p>
<p>In order to illustrate this, I propose to partially replicate one of the numerical experiments in Benidis et al.<sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In details, using the S&P 500 data provided in <a href="https://cran.r-project.org/web/packages/sparseIndexTracking/index.html">the R package <em>sparseIndexTracking</em></a> - which consists in the daily returns of the S&P 500 index and of 386 of its constituents
over the period from 1st January 2010 to 31th December 2010 - I will solve<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following 80 sparse index tracking problems:</p>
<ul>
<li><strong>Index</strong> - S&P 500</li>
<li><strong>Tracking assets</strong> - 386 stocks included in the S&P 500</li>
<li><strong>Maximum number of assets</strong> - Varying from 20 to 100</li>
<li><strong>Misc. constraints</strong> - Maximum asset weight constraint of 5%</li>
</ul>
<p>The empirical tracking error of the resulting 80 sparse S&P 500 tracking portfolios, as well as - for reference - the empirical tracking error of the regular S&P 500 tracking portfolio<sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>, is depicted in Figure 1.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-ete-sp500.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-ete-sp500.png" alt="Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks" /></a>
<figcaption>Figure 1. Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks</figcaption>
</figure>
<p>From this figure, it is visible that a sparse tracking portfolio made of ~100 stocks allows to track the S&P 500 with the same level of tracking error as a regular tracking portfolio made of much more stocks<sup id="fnref:32:1" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>But it is possible to do better!</p>
<p>A sparse tracking portfolio made of only ~50 stocks is actually sufficient, as illustrated in Figure 2.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-growth-sp500.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-growth-sp500.png" alt="Sparse S&P 500 55-stock tracking portfolios over the period 01 January 2010 - 31 December 2010, performances comparison with S&P 500" /></a>
<figcaption>Figure 2. Sparse 55-stock S&P 500 tracking portfolio over the period 01 January 2010 - 31 December 2010, performances comparison with the S&P 500</figcaption>
</figure>
<p>Of course, such a small portfolio might be less “diversified”<sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">25</a></sup> than its 100-stock counterpart, but the main takeaway is that a portfolio made of ~50-100 stocks adequately replicates the S&P 500, which opens the door of
<em>direct investing</em><sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">26</a></sup> to DIY investors.</p>
<h3 id="automatically-determining-the-proper-composite-benchmark-for-a-mutual-fund">Automatically determining the proper composite benchmark for a mutual fund</h3>
<p>In <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">the previous post of this series</a>, the index tracking machinery was shown to allow to easily and automatically construct a benchmark for any mutual fund.</p>
<p>By extension, the sparse index tracking machinery also allows to do the same, with an important advantage for multi-asset mutual funds - the automated construction of the best composite benchmark with a given number of constituents.</p>
<h3 id="replicating-a-fund-of-funds-to-reduce-fees">Replicating a fund of funds to reduce fees</h3>
<p>As a last example of usage, I propose to build upon <a href="https://www.linkedin.com/posts/finominal_whats-the-likelihood-that-all-22-funds-in-activity-7112822889518616576-lEu4?utm_source=share&utm_medium=member_desktop">a LinkedIn post</a>
from <a href="https://finominal.com/">Finominal</a>, in which the following question is asked:</p>
<blockquote>
<p>What’s the likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value?</p>
</blockquote>
<p>A bit of context first.</p>
<p>The <em><a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-investor-growth-income-fund-a-4812c2858">JPMorgan Investor Growth & Income Fund</a></em> is a fund of funds (FoF) whose <em>main investment strategy is to invest in other J.P. Morgan Funds</em><sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">27</a></sup>.
As of 31 December 2023, it is invested in 25 such other J.P. Morgan funds (<a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-core-bond-fund-r6-4812c0100"><em>JPMorgan Core Bond R6</em></a>,
<a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-us-equity-fund-r6-48121l817"><em>JPMorgan US Equity R6</em></a>…).</p>
<p>One problem with FoF is that they usually have higher fees than traditional mutual funds <em>because they include the management fees charged by the underlying funds</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">28</a></sup>.</p>
<p>The <em>JPMorgan Investor Growth & Income Fund</em> is no different, with a total expense ratio ranging from 1.47% to 0.72%, the latter being for its institutional class share (ticker ONGFX).</p>
<p>Inspired by the results from <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">the previous post of this series</a>, it is possible to reduce those fees by investing directly in all the underlying J.P. Morgan funds
in proportion determined by solving<sup id="fnref:35:1" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following regular index tracking problem:</p>
<ul>
<li><strong>Index</strong> - The <em>JPMorgan Investor Growth & Income Fund</em> (ONGFX)</li>
<li><strong>Tracking assets</strong> - 24 over the 25 J.P. Morgan funds held by the <em>JPMorgan Investor Growth & Income Fund</em> as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup></li>
</ul>
<p>Using price data<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">30</a></sup> for the period 01 January 2023 - 31 December 2023, this gives the regular tracking portfolio whose weights are displayed in Figure 3 and whose performances are compared with the <em>JPMorgan Investor Growth & Income Fund</em> in Figure 4.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-weights-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-weights-jpm.png" alt="JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund" /></a>
<figcaption>Figure 3. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-growth-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-growth-jpm.png" alt="JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition" /></a>
<figcaption>Figure 4. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund</figcaption>
</figure>
<p>This tracking portfolio has:</p>
<ul>
<li>A practically null tracking error v.s. the <em>JPMorgan Investor Growth & Income Fund</em> (Figure 4)</li>
<li>An expense ratio of ~0.43%<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">31</a></sup> v.s. of at least 0.72% for the <em>JPMorgan Investor Growth & Income Fund</em></li>
</ul>
<p>So, this tracking portfolio is an interesting alternative to the <em>JPMorgan Investor Growth & Income Fund</em>, but there is a catch - it is invested in a total of 22 J.P. Morgan funds (Figure 3), which might be cumbersome to manage…</p>
<p>An even better tracking portfolio would have a smaller number of directly-invested funds.</p>
<p>Does such a unicorn exist?</p>
<p>Figures 5 and 6 answer this question, by solving<sup id="fnref:35:2" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following sequence of sparse index tracking problems over the period 01 January 2023 - 31 December 2023:</p>
<ul>
<li><strong>Index</strong> - The <em>JPMorgan Investor Growth & Income Fund</em> (ONGFX)</li>
<li><strong>Tracking assets</strong> - The 24 non-restricted J.P. Morgan funds held by the <em>JPMorgan Investor Growth & Income Fund</em> as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar<sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup></li>
<li><strong>Maximum number of assets</strong> - Varying from 1 to 24</li>
</ul>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-growth-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-growth-jpm.png" alt="Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund" /></a>
<figcaption>Figure 5. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-fees-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-fees-jpm.png" alt="Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees" /></a>
<figcaption>Figure 6. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees</figcaption>
</figure>
<p>From these two figures, the sweet spot for the number of directly-invested funds in terms of tracking error/fees/manageability seems to be ~6-8.</p>
<p>Coming back to the initial question from Finominal, this analysis confirms that there is <em>basically zero likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value</em>, because these can be drastically reduced<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">32</a></sup>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>That’s it for the first blog post of 2024!</p>
<p>Next in this series, I will discuss another extension of the regular index tracking problem, aiming this time at tracking a financial index with “dynamic” asset weights over the considered $T$ time periods.</p>
<p>As usual, feel free to connect with me <a href="https://www.linkedin.com/in/roman-rubsamen/">on LinkedIn</a> or to follow me <a href="https://twitter.com/portfoliooptim">on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Also called the <em>benchmark replication problem</em>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>In terms of a given tracking error measure, like the <em>empirical tracking error</em><sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, also called the <em>mean squared tracking error</em>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://ieeexplore.ieee.org/document/8384194">Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:3:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>Also called the <em>partial benchmark replication problem</em>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/34/2/143">Kees van Montfort, Elout Visser, Laurens Fijn van Draat, Index Tracking by Means of Optimized Sampling, The Journal of Portfolio Management Winter 2008, 34 (2) 143-152</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Or a solution, in case multiple solutions exist. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>The zero “norm” is not a real norm. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0305054817302265">Purity Mutunge, Dag Haugland, Minimizing the tracking error of cardinality constrained portfolios, Computers & Operations Research, Volume 90, 2018, Pages 33-41</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:8:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:8:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>To be noted that in the specific case of a quadratic loss function, Mutunge and Haugland<sup id="fnref:8:3" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup> establish that the sparse index tracking optimization problem is actually <a href="https://en.wikipedia.org/wiki/Strong_NP-completeness">strongly NP-hard</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>Excluding simple heuristics like selecting the largest assets according to a given criteria. For example, <em>a widely used method, especially for a market capitalization weighted index, is to select the largest K assets according to their market capitalizations</em><sup id="fnref:3:8" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>; as another example, greedy or reverse greedy algorithms can be used on top of a non-sparse formulation of the index tracking problem to include or exclude assets one by one<sup id="fnref:19:1" role="doc-noteref"><a href="#fn:19" class="footnote">19</a></sup>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S037722170800283X">N.A. Canakgoz, J.E. Beasley, Mixed-integer programming approaches for index tracking and enhanced indexation, European Journal of Operational Research, Volume 196, Issue 1, 2009, Pages 384-399</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221702004253">J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-1-4757-5226-7_1">Gilli, M., Kellezi, E. (2002). The Threshold Accepting Heuristic for Index Tracking. In: Pardalos, P.M., Tsitsiringos, V.K. (eds) Financial Engineering, E-commerce and Supply Chain. Applied Optimization, vol 70. Springer, Boston, MA</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1063520309000384">T. Blumensath and M. Davies, Iterative hard thresholding for compressed sensing, Appl. Comput. Harmon. Anal., 27 (2009), pp. 265–274.</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1063520308000638">Deanna Needell and Joel A Tropp. Cosamp: iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301–321, 2009</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Which is usually the case in practice… <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s11425-016-9124-0">Xu, F., Dai, Y., Zhao, Z. et al. Efficient projected gradient methods for cardinality constrained optimization. Sci. China Math. 62, 245–268 (2019).</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://epubs.siam.org/doi/abs/10.1137/120869778?journalCode=sjope8">Amir Beck, Yonina C. Eldar, Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms, 2013, SIAM Journal on Optimization, 1480-1509, 23, 3</a> <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See <a href="https://inria.hal.science/hal-03874638/">Quentin Jacquet, Agnes Bialecki, Laurent El Ghaoui, Stéphane Gaubert, Riadh Zorgati. Entropic Lower Bound of Cardinality for Sparse Optimization. 2022.</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:19:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>It is not trivial to show that such a reformulation is equivalent; c.f. Jansen and van Dijk<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">21</a></sup> for a detailed proof. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/28/2/33">Roel Jansen, Ronald van Dijk, Optimal Benchmark Tracking with Small Portfolios, The Journal of Portfolio Management, Winter 2002, 28 (2) 33-39</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>The possibility to include minimum and maximum group weights constraints is important in practice, because it allows for example to set an upper limit on fees or to (try to) impose sector neutrality in the computed portfolio, a characteristic often overlooked c.f. for example Che et al.<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">33</a></sup> <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>Using <strong>Portfolio Optimizer</strong>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:35:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:35:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>The non-sparse S&P 500 tracking portfolio, subject to the same maximum asset weight constraint of 5% as the sparse S&P 500 tracking portfolios, is made of 286 stocks. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:32:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>For example, subject to sector concentration bias, as highlighted in Che et al.<sup id="fnref:22:1" role="doc-noteref"><a href="#fn:22" class="footnote">33</a></sup>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>Direct investing solutions are usually provided by asset management firms (<a href="https://www.fidelity.com/learning-center/trading-investing/direct-indexing">Fidelity</a>, <a href="https://frec.com/">Frec</a>…). In the case of a stock market index, investing directly in stocks composing that stock market index, as opposed to indirectly through an ETF, has several advantages (null expense ratio, possibility to do tax-loss harvesting…). <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>C.f. the <a href="https://am.jpmorgan.com/JPMorgan/TADF/4812C2858/SP?site=JPMorgan">JPMorgan Investor Growth & Income Fund prospectus</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>C.f. <a href="https://en.wikipedia.org/wiki/Fund_of_funds">Wikipedia</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>C.f. <a href="https://www.morningstar.com/funds/xnas/ONGFX/quote">Morningstar</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>Computation made using the expense ratios reported by Morningstar for the different J.P. Morgan funds, you will have to trust me on this one. To also be noted that the fees associated to the potential rebalancing of the tracking portfolio are not taken into account and would need to be properly minimized in real life. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>In view of Figure 5 and Figure 6, I would take the <em>returns increased</em> part of <a href="https://www.linkedin.com/posts/finominal_whats-the-likelihood-that-all-22-funds-in-activity-7112822889518616576-lEu4?utm_source=share&utm_medium=member_desktop">Finominal’s LinkedIn post</a> with a grain of salt, although it is certain that returns will increase thanks to lowering the fees. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>See <a href="https://doi.org/10.3390/math10152645">Che, Y.; Chen, S.; Liu, X. Sparse Index Tracking Portfolio with Sector Neutrality. Mathematics 2022, 10, 2645.</a>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:22:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
</ol>
</div>Roman R.In the previous post, I introduced the index tracking problem1, which consists in finding a portfolio that tracks as closely as possible2 a given financial market index. Because such a portfolio might contain any number of assets, with for example an S&P 500 tracking portfolio possibly containing ~500 stocks, it is [sometimes desirable] that the tracking portfolio consists of a small number of assets3 in order to simplify the execution, avoid small and illiquid positions, and large transaction costs3. In other words, it is sometimes desirable to impose a constraint on the maximum number of assets contained in an index tracking portfolio, which leads to an extension of the regular index tracking problem called the sparse index tracking problem4. In this new post, I will describe the mathematics of the sparse index tracking problem and I will detail a few examples of usage like: Replicating a fund of funds to reduce fees while keeping the number of directly-invested funds manageable Replicating the S&P 500 with only ~50 stocks as an (extreme) illustration of the optimized sampling5 ETF methodology Mathematical preliminaries The general sparse index tracking optimization problem Let be: $T$, a number of time periods $r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods $n$, the number of assets in the universe of the sparse index tracking portfolio $X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods $w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets The vector of the sparse index tracking portfolio weights $w^*$ is then the6 solution to the optimization problem which consists in minimizing a tracking error measure $f \left( X w - r_{idx} \right)$ - with $f$ some loss function - subject to additional constraints like full investment constraint, no short sale constraint, etc. and a cardinality constraint3 \[w^* = \operatorname{argmin} f \left( X w - r_{idx} \right) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\] , where: $\lVert . \rVert_0$ is the zero “norm”7 of a vector, that is, the number of non-zero elements in that vector $n_{max}$ is the maximum number of assets with non-zero weights desired in the sparse index tracking portfolio From a computational perspective, and whatever the exact definition of the loss function $f$, it is known that when a cardinality constraint restricting the number of stocks is introduced, the problem of optimizing the composition of a portfolio tends to become NP-hard8, which means that exact solutions to instances of realistic sizes are computationally intractable, and thus inexact solution methods are the only practical ones8. Standard quadratic optimization methods guaranteeing an optimal solution are thus unfortunately not usable to solve the sparse index tracking optimization problem9… Heuristics for solving the general sparse index tracking optimization problem While an exhaustive enumeration of all possible methods for solving the sparse index tracking optimization problem is out of scope of this post, most of them10 seem to fall into one of three rough categories. Combinatorial optimization methods The cardinality constraint present in the sparse index tracking problem makes it a combinatorial problem, so that standard combinatorial optimization methods (continuous relaxation of a mixed integer programming formulation811, genetic algorithms1213…) can be used to find an approximate solution. As a side note, thanks to the relationship of the sparse index tracking problem with combinatorial optimization and hence with operations research, it is possible to find datasets for different index tracking problems - ranging from a universe of 31 assets to a universe of 2151 assets - in the OR-Library. Signal processing methods The sparse index tracking problem in finance is closely related to a similar problem in signal processing called compressed sensing, for which specific algorithms have been developed over the years, like the iterative hard thresholding algorithm14 or the compressive sampling matching pursuit (CoSaMP) algorithm15. To be noted, though, that all signal processing methods that rely on the least absolute shrinkage and selection operator (LASSO), which consists in approximating the $l_0$-norm constraint by a $l_1$-norm constraint, are not applicable to the sparse index tracking problem. Indeed, when full investment and no-short sales constraints are imposed16, we have, for any portfolio weights vector $w$ \[\lVert w \rVert_1 = \sum_{i=1}^n \left| w_i \right| = \sum_{i=1}^n w_i = 1\] , so that the $l_1$-norm reduces to a constant and is therefore irrelevant3. Non-linear optimization methods At the crossroads with signal processing methods, generic non-linear optimization methods taking into account cardinality constraints have also been developed over the years: Methods that directly integrate cardinality constraints, like projected gradient methods1718 or coordinate descent methods18 Methods that indirectly integrate cardinality constraints by either: Relaxing the $l_0$-norm constraint into an “easier” constraint, like the entropic lower bound algorithm of Jacquet et al.19 Reformulating the $l_0$-norm constraint as a $l_0$-norm penalty integrated in the tracking error measure20 and replacing that $l_0$-norm penalty with an “easier” penalty, like the $q$-norm approximation algorithm of Jansen and van Dijk21 or the majorization-minimization algorithm of Benidis et al.3 The sparse index tracking/empirical tracking error optimization problem Like in the previous post of this series, this post will use the empirical tracking error as the preferred tracking error measure. In this case, the general sparse index tracking optimization problem becomes \[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\] , which, as noted by Benidis et al.3, is nothing else than a constrained sparse regression problem: the sparse index tracking problem is similar to many sparsity formulations in the signal processing area in the sense that it is a regression problem with some sparsity requirements Implementation in Portfolio Optimizer Portfolio Optimizer allows to compute an approximate solution to the sparse index tracking optimization problem under the empirical tracking error measure through the endpoint /portfolios/replication/index-tracking/sparse. In addition, Portfolio Optimizer allows to impose misc. constraints like minimum and maximum asset weights or minimum and maximum group weights constraints22. Examples of usage Replicating the S&P 500 by (extreme) optimized sampling One of the most immediate application of the sparse index tracking problem is the ability to replicate a financial market index without investing into all its constituents, which is desirable for the reasons explained in the introduction of this post. In order to illustrate this, I propose to partially replicate one of the numerical experiments in Benidis et al.3. In details, using the S&P 500 data provided in the R package sparseIndexTracking - which consists in the daily returns of the S&P 500 index and of 386 of its constituents over the period from 1st January 2010 to 31th December 2010 - I will solve23 the following 80 sparse index tracking problems: Index - S&P 500 Tracking assets - 386 stocks included in the S&P 500 Maximum number of assets - Varying from 20 to 100 Misc. constraints - Maximum asset weight constraint of 5% The empirical tracking error of the resulting 80 sparse S&P 500 tracking portfolios, as well as - for reference - the empirical tracking error of the regular S&P 500 tracking portfolio24, is depicted in Figure 1. Figure 1. Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks From this figure, it is visible that a sparse tracking portfolio made of ~100 stocks allows to track the S&P 500 with the same level of tracking error as a regular tracking portfolio made of much more stocks24. But it is possible to do better! A sparse tracking portfolio made of only ~50 stocks is actually sufficient, as illustrated in Figure 2. Figure 2. Sparse 55-stock S&P 500 tracking portfolio over the period 01 January 2010 - 31 December 2010, performances comparison with the S&P 500 Of course, such a small portfolio might be less “diversified”25 than its 100-stock counterpart, but the main takeaway is that a portfolio made of ~50-100 stocks adequately replicates the S&P 500, which opens the door of direct investing26 to DIY investors. Automatically determining the proper composite benchmark for a mutual fund In the previous post of this series, the index tracking machinery was shown to allow to easily and automatically construct a benchmark for any mutual fund. By extension, the sparse index tracking machinery also allows to do the same, with an important advantage for multi-asset mutual funds - the automated construction of the best composite benchmark with a given number of constituents. Replicating a fund of funds to reduce fees As a last example of usage, I propose to build upon a LinkedIn post from Finominal, in which the following question is asked: What’s the likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value? A bit of context first. The JPMorgan Investor Growth & Income Fund is a fund of funds (FoF) whose main investment strategy is to invest in other J.P. Morgan Funds27. As of 31 December 2023, it is invested in 25 such other J.P. Morgan funds (JPMorgan Core Bond R6, JPMorgan US Equity R6…). One problem with FoF is that they usually have higher fees than traditional mutual funds because they include the management fees charged by the underlying funds28. The JPMorgan Investor Growth & Income Fund is no different, with a total expense ratio ranging from 1.47% to 0.72%, the latter being for its institutional class share (ticker ONGFX). Inspired by the results from the previous post of this series, it is possible to reduce those fees by investing directly in all the underlying J.P. Morgan funds in proportion determined by solving23 the following regular index tracking problem: Index - The JPMorgan Investor Growth & Income Fund (ONGFX) Tracking assets - 24 over the 25 J.P. Morgan funds held by the JPMorgan Investor Growth & Income Fund as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar29 Using price data30 for the period 01 January 2023 - 31 December 2023, this gives the regular tracking portfolio whose weights are displayed in Figure 3 and whose performances are compared with the JPMorgan Investor Growth & Income Fund in Figure 4. Figure 3. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition Figure 4. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund This tracking portfolio has: A practically null tracking error v.s. the JPMorgan Investor Growth & Income Fund (Figure 4) An expense ratio of ~0.43%31 v.s. of at least 0.72% for the JPMorgan Investor Growth & Income Fund So, this tracking portfolio is an interesting alternative to the JPMorgan Investor Growth & Income Fund, but there is a catch - it is invested in a total of 22 J.P. Morgan funds (Figure 3), which might be cumbersome to manage… An even better tracking portfolio would have a smaller number of directly-invested funds. Does such a unicorn exist? Figures 5 and 6 answer this question, by solving23 the following sequence of sparse index tracking problems over the period 01 January 2023 - 31 December 2023: Index - The JPMorgan Investor Growth & Income Fund (ONGFX) Tracking assets - The 24 non-restricted J.P. Morgan funds held by the JPMorgan Investor Growth & Income Fund as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar29 Maximum number of assets - Varying from 1 to 24 Figure 5. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund Figure 6. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees From these two figures, the sweet spot for the number of directly-invested funds in terms of tracking error/fees/manageability seems to be ~6-8. Coming back to the initial question from Finominal, this analysis confirms that there is basically zero likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value, because these can be drastically reduced32. Conclusion That’s it for the first blog post of 2024! Next in this series, I will discuss another extension of the regular index tracking problem, aiming this time at tracking a financial index with “dynamic” asset weights over the considered $T$ time periods. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – Also called the benchmark replication problem. ↩ In terms of a given tracking error measure, like the empirical tracking error3, also called the mean squared tracking error. ↩ See Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 Also called the partial benchmark replication problem. ↩ See Kees van Montfort, Elout Visser, Laurens Fijn van Draat, Index Tracking by Means of Optimized Sampling, The Journal of Portfolio Management Winter 2008, 34 (2) 143-152. ↩ Or a solution, in case multiple solutions exist. ↩ The zero “norm” is not a real norm. ↩ See Purity Mutunge, Dag Haugland, Minimizing the tracking error of cardinality constrained portfolios, Computers & Operations Research, Volume 90, 2018, Pages 33-41. ↩ ↩2 ↩3 To be noted that in the specific case of a quadratic loss function, Mutunge and Haugland8 establish that the sparse index tracking optimization problem is actually strongly NP-hard. ↩ Excluding simple heuristics like selecting the largest assets according to a given criteria. For example, a widely used method, especially for a market capitalization weighted index, is to select the largest K assets according to their market capitalizations3; as another example, greedy or reverse greedy algorithms can be used on top of a non-sparse formulation of the index tracking problem to include or exclude assets one by one19. ↩ See N.A. Canakgoz, J.E. Beasley, Mixed-integer programming approaches for index tracking and enhanced indexation, European Journal of Operational Research, Volume 196, Issue 1, 2009, Pages 384-399. ↩ See J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643. ↩ See Gilli, M., Kellezi, E. (2002). The Threshold Accepting Heuristic for Index Tracking. In: Pardalos, P.M., Tsitsiringos, V.K. (eds) Financial Engineering, E-commerce and Supply Chain. Applied Optimization, vol 70. Springer, Boston, MA. ↩ See T. Blumensath and M. Davies, Iterative hard thresholding for compressed sensing, Appl. Comput. Harmon. Anal., 27 (2009), pp. 265–274.. ↩ See Deanna Needell and Joel A Tropp. Cosamp: iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301–321, 2009. ↩ Which is usually the case in practice… ↩ See Xu, F., Dai, Y., Zhao, Z. et al. Efficient projected gradient methods for cardinality constrained optimization. Sci. China Math. 62, 245–268 (2019).. ↩ See Amir Beck, Yonina C. Eldar, Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms, 2013, SIAM Journal on Optimization, 1480-1509, 23, 3 ↩ ↩2 See Quentin Jacquet, Agnes Bialecki, Laurent El Ghaoui, Stéphane Gaubert, Riadh Zorgati. Entropic Lower Bound of Cardinality for Sparse Optimization. 2022.. ↩ ↩2 It is not trivial to show that such a reformulation is equivalent; c.f. Jansen and van Dijk21 for a detailed proof. ↩ See Roel Jansen, Ronald van Dijk, Optimal Benchmark Tracking with Small Portfolios, The Journal of Portfolio Management, Winter 2002, 28 (2) 33-39. ↩ ↩2 The possibility to include minimum and maximum group weights constraints is important in practice, because it allows for example to set an upper limit on fees or to (try to) impose sector neutrality in the computed portfolio, a characteristic often overlooked c.f. for example Che et al.33 ↩ Using Portfolio Optimizer. ↩ ↩2 ↩3 The non-sparse S&P 500 tracking portfolio, subject to the same maximum asset weight constraint of 5% as the sparse S&P 500 tracking portfolios, is made of 286 stocks. ↩ ↩2 For example, subject to sector concentration bias, as highlighted in Che et al.33. ↩ Direct investing solutions are usually provided by asset management firms (Fidelity, Frec…). In the case of a stock market index, investing directly in stocks composing that stock market index, as opposed to indirectly through an ETF, has several advantages (null expense ratio, possibility to do tax-loss harvesting…). ↩ C.f. the JPMorgan Investor Growth & Income Fund prospectus. ↩ C.f. Wikipedia. ↩ C.f. Morningstar. ↩ ↩2 (Adjusted) prices have have been retrieved using Tiingo. ↩ Computation made using the expense ratios reported by Morningstar for the different J.P. Morgan funds, you will have to trust me on this one. To also be noted that the fees associated to the potential rebalancing of the tracking portfolio are not taken into account and would need to be properly minimized in real life. ↩ In view of Figure 5 and Figure 6, I would take the returns increased part of Finominal’s LinkedIn post with a grain of salt, although it is certain that returns will increase thanks to lowering the fees. ↩ See Che, Y.; Chen, S.; Liu, X. Sparse Index Tracking Portfolio with Sector Neutrality. Mathematics 2022, 10, 2645.. ↩ ↩2