Jekyll2024-07-10T08:57:05-05:00https://portfoliooptimizer.io/feed.xmlPortfolio OptimizerPortfolio Optimizer is a Web API democratizing the access to the Nobel Prize-winning science of portfolio optimization.Roman R.Capital Market Assumptions: Combining Institutions’ Forecasts for Improved Accuracy2024-07-10T00:00:00-05:002024-07-10T00:00:00-05:00https://portfoliooptimizer.io/blog/capital-market-assumptions-combining-institutions-forecasts-for-improved-accuracy<p>Capital market assumptions<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> (CMAs) are forecasts of future risk/return characteristics for broad asset classes over the next 5 to 20 years
produced by <em>leading investment managers, consultants and advisors</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>These forecasts are <em>well-reasoned, analytically rigorous assumptions about uncertain future market movements</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> and are used
<em>almost universally among institutional investors</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, for example as inputs to their <a href="https://www.investopedia.com/terms/s/strategicassetallocation.asp">stategical asset allocation</a><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In this blog post, I will show that combining capital market assumptions might be preferable to using any institution-specific estimates
and I will describe a couple of practical data issues encountered when doing so.</p>
<h2 id="the-variability-of-capital-market-assumptions">The variability of capital market assumptions</h2>
<p>Each year since 2010, people at <a href="https://www.horizonactuarial.com/">Horizon Actuarial</a> <em>ask different investment firms to provide their capital market assumptions</em><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, and release an associated
survey.</p>
<p>Figure 1 - taken from the 2023 edition of that survey<sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> - clearly shows <em>significant differences in expected returns and standard deviations among investment advisors</em><sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-horizon-actuarial-large.png"><img src="/assets/images/blog/capital-markert-assumptions-horizon-actuarial.png" alt="Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial." /></a>
<figcaption>Figure 1. Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial.</figcaption>
</figure>
<p>Of course, variability should come as no surprise, because different institutions use different forecasting methodologies and/or have different views on the future<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup>…</p>
<p>Nevertheless, such a high variability is definitely an issue for an investor since it is impossible to identify a priori the institution that will produce the most accurate forecasts
of future returns, standard deviations and correlations.</p>
<p>The risk of relying on a “bad” institution for an investor is therefore very real…</p>
<h2 id="combining-capital-market-assumptions">Combining capital market assumptions</h2>
<h3 id="the-general-forecast-combination-problem">The general forecast combination problem</h3>
<p>The specific problem described in the previous section is an instance of a more general problem known as <em>the forecast combination problem</em>, introduced as follows by Timmermann<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>:</p>
<blockquote>
<p>Faced with multiple forecasts of the same variable, an issue that immediately arises is how best to exploit information in the individual forecasts. In particular, should a single dominant forecast be identified or should a combination of the underlying forecasts be used to produce a pooled summary measure?</p>
</blockquote>
<p>From the forecasting literature, a pragmatic solution to this problem is to average all individual forecasts, which leads to an equal-weighting forecast combination scheme.</p>
<p>While guaranteed to degrade the forecast accuracy v.s. using the “best” individual forecast<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">7</a></sup>, such a forecast combination scheme has repeatedly been found to
<em>outperform sophisticated adaptive combination methods in empirical applications</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, thus setting <em>a benchmark that has proved surprisingly difficult to beat</em><sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>As an interesting side note for readers of this blog, it turns out that the forecast combination problem is closely related to the mean-variance portfolio optimization problem<sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<h3 id="averaging-capital-market-assumptions">Averaging capital market assumptions</h3>
<p>From a theoretical perspective, an equal-weighted forecast combination scheme is well-suited to the specific context of capital market assumptions.</p>
<p>Indeed, given the <em>significant effort, resources and intellectual capital [put] into the production of [capital market assumptions]</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, it is reasonable to assume that the
<em>relative size of the variance of the forecast errors associated with [the] different forecasting methods</em><sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> are all similar, which in theory favors an equal-weighting forecast combination scheme<sup id="fnref:6:5" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>From a practical perspective, an equal-weighted forecast combination scheme is easy to compute and does not introduce any parameter estimation errors, which
<em>are known to be a serious problem for many combination techniques</em><sup id="fnref:6:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>As for the forecast accuracy of that scheme, it is possible to have an idea of it thanks to Figure 2 - taken from Sebastian<sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> -, which compares:</p>
<ul>
<li>The min-max range of 10-year capital market assumptions produced by 19 investment firms for misc. asset classes in 2013</li>
<li>The associated averaged 10-year capital market assumptions</li>
<li>The subsequent realized 10-year returns</li>
</ul>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-sebastian-large.png"><img src="/assets/images/blog/capital-markert-assumptions-sebastian.png" alt="2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian." /></a>
<figcaption>Figure 2. 2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian.</figcaption>
</figure>
<p>From Figure 2, it is visible that:</p>
<ul>
<li>
<p>Using averaged capital market assumptions v.s. firm-specific estimates prevented<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">8</a></sup> severe over/under-predictions for a couple of asset classes, like a ~3% over-prediction for Emerging Markets equities, which is great.</p>
</li>
<li>
<p><em>In the majority of cases, the average capital market assumption [is] an unreliable guide to the future return</em><sup id="fnref:2:5" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, which is not so great, but which ultimately depends on the accuracy and/or bias of the individual capital market assumptions and not on the forecast combination scheme itself<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">9</a></sup>.</p>
<p>That being said, a sample size of 1 observation per asset class is anyway maybe a little too small for reaching definite conclusions…</p>
</li>
</ul>
<h2 id="capital-market-assumptions-data-preparation">Capital market assumptions data preparation</h2>
<p>Before being able to average capital market assumptions originating from different institutions, it is first required to collect and normalize these forecasts.</p>
<p>Below are examples of typical data preparation steps for doing so.</p>
<h3 id="collecting-a-sufficient-number-of-forecasts">Collecting a sufficient number of forecasts</h3>
<p>Figure 3 - taken from an <a href="https://ecrresearch.com/">ECR Research</a> report<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">10</a></sup> - illustrates the variety of forecast horizons for which a representative sample of leading investment firms provide
their capital market assumptions.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-ecrresearch-large.png"><img src="/assets/images/blog/capital-markert-assumptions-ecrresearch.png" alt="Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research." /></a>
<figcaption>Figure 3. Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research.</figcaption>
</figure>
<p>From Figure 3, and in oder to combine as many distinct capital market assumptions as possible, it seems a good idea to focus on collecting forecasts for either a 5-year or a 10-year horizon, with a
preference for a 10-year horizon.</p>
<h3 id="reconciliating-asset-classes-across-institutions">Reconciliating asset classes across institutions</h3>
<p>Because <em>not all investment advisors use the same asset classes when developing their capital market assumptions</em><sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, reconciliating different asset classes across institutions is mandatory and
requires to <em>exercice judgment in classifying [institutions] capital market assumptions into a standard set of asset</em><sup id="fnref:2:6" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>As a side note, there are also sometimes inconsistencies in the name of asset classes.</p>
<p>For example, under the asset class label <em>Developed Equities</em>, different institutions might mean:</p>
<ul>
<li>An asset class represented by the <em>MSCI World Index</em>, which would be the standard convention</li>
<li>An asset class represented by the <em>MSCI World ex-US Index</em></li>
<li>An asset class represnted by the <em>MSCI EAFE Index</em></li>
</ul>
<h3 id="converting-expected-returns">Converting expected returns</h3>
<h4 id="converting-real-expected-returns-to-nominal-expected-returns">Converting real expected returns to nominal expected returns</h4>
<p>The vast majority of institutions provide expected returns in nominal terms, but a couple of them provide expected returns in <a href="https://www.investopedia.com/terms/r/realrateofreturn.asp">real terms</a>.</p>
<p>In order to compare apples to apples, it is necessary to convert these real returns into nominal returns by adding local expected inflation<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>, that is:</p>
<ul>
<li>U.S. expected inflation, for U.S. asset classes (<em>U.S. Equities</em>, <em>U.S. Bonds</em>…) as well as usually for commodities</li>
<li>Eurozone expected inflation<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">12</a></sup>, for euro-denominated asset classes (<em>E.M.U. Equities</em>, <em>EUR Bonds</em>…)</li>
<li>Emerging countries inflation<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">12</a></sup>, for <em>Emerging Equities</em></li>
<li>…</li>
</ul>
<h4 id="converting-expected-returns-currencies">Converting expected returns currencies</h4>
<p>Depending on the institution and on the asset class, expected returns can be provided in:</p>
<ul>
<li>USD, or another investable currency like EUR</li>
<li>USD EUR-hedged, or another investable currency pair hedged like EUR JPY-hedged</li>
<li>Local currency</li>
<li>Local currency investable currency-hedged, or investable currency local currency-hedged or whatever weird combination</li>
</ul>
<p>In order to again compare apples to apples, it is necessary to convert all the currencies appearing in the different expected returns into a unique investable currency.</p>
<h5 id="converting-expected-returns-in-an-investable-currency-to-expected-returns-in-another-investable-currency">Converting expected returns in an investable currency to expected returns in another investable currency</h5>
<p>The most usual scenario is the need to convert expected returns initially provided in an investable currency like USD into another investable currency like EUR.</p>
<p>This convertion - <em>ignoring the small geometric interaction</em><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup> - is done by adding the expected foreign exchange return to the expected returns<sup id="fnref:13:2" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>.</p>
<h5 id="converting-investable-currency-hedged-expected-returns-to-investable-currency-unhedged-expected-returns">Converting investable currency-hedged expected returns to investable currency-unhedged expected returns</h5>
<p>Expected returns for “foreign” fixed income asset classes are frequently provided as “foreign currency”-hedged<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">13</a></sup>.</p>
<p>The convertion from “foreign currency”-hedged expected returns to “foreign currency”-unhedged expected returns is done by <em>correcting for the expected nominal risk-free rate differential</em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>, that is,
by adding the difference between the foreign and the local cash returns.</p>
<h5 id="converting-expected-returns-in-local-currency-to-expected-returns-in-an-investable-currency">Converting expected returns in local currency to expected returns in an investable currency</h5>
<p>Many institutions provide expected returns in local currency to <em>avoid the issue of [determining] what an asset’s expected return will be from a particular investor’s currency perspective</em><sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">13</a></sup>.</p>
<p>Ultimately, though, an investor needs to make a currency decision, and expected returns in local currency must be converted into expected returns in an investable currency like USD or EUR.</p>
<p>For single-currency asset classes, like <em>U.S. Equities</em> or <em>Eurozone Government Bonds</em>, this is not an issue, c.f. the previous sub-sections.</p>
<p>For multi-currency asset classes, like <em>Emerging Market Equities</em> or <em>Global Government Bonds</em>, the situation is more complex because a proper conversion would at the very least require assumptions on:</p>
<ul>
<li>The expected currency weights in the asset class at the forecast horizon</li>
<li>The expected foreign exchange return at the forecast horizon, for all currencies</li>
</ul>
<p>To be noted that ignoring this complexity is probably not an option, as illustrated in Figure 4 - taken from the factsheet of the <em>MSCI Emerging Markets Index</em> -,
which shows a ~3% difference in annualized returns over the last 10 years between the “local currency” and the “USD” flavors of the <em>MSCI Emerging Markets Index</em>.</p>
<figure>
<a href="/assets/images/blog/capital-markert-assumptions-msci-large.png"><img src="/assets/images/blog/capital-markert-assumptions-msci.png" alt="Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI." /></a>
<figcaption>Figure 4. Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI.</figcaption>
</figure>
<p>Between those two extremes, finding a pragmatic solution depending on the exact asset class is the best thing to do!</p>
<h5 id="other-convertions">Other convertions</h5>
<p>The need for other weird currency convertions might be encountered, at which point a balance must be struck between:</p>
<ul>
<li>Adding new forecasts, possibly unique, to the list of collected capital market assumptions</li>
<li>Adding complexity to the capital market assumptions collection process</li>
</ul>
<h4 id="converting-geometric-expected-returns-to-arithmetic-expected-returns">Converting geometric expected returns to arithmetic expected returns</h4>
<p>Nearly all institutions produce <em>estimates of geometric (compound) asset class returns</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>, while <em>standard mean-variance portfolio optimization<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">15</a></sup> requires
expected returns to be provided in arithmetic rather than in geometric terms</em><sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">14</a></sup>.</p>
<p>In order to approximately convert geometric expected returns to arithmetic expected returns, it suffices to add half of the expected asset class variance<sup id="fnref:13:3" role="doc-noteref"><a href="#fn:13" class="footnote">11</a></sup>.</p>
<p>As a down-to-earth comment on why is that, I cannot resist sharing what people at <a href="https://www.verusinvestments.com/">Verus</a> wrote in the 2024 edition of their capital market assumptions:</p>
<blockquote>
<p>This is the industry standard approach, but requires a complex explanation only a heavy quant could love, so we have chosen not to provide further details in this document […]</p>
</blockquote>
<h3 id="converting-expected-standard-deviationscorrelations">Converting expected standard deviations/correlations</h3>
<p>Similar to expected returns, expected standard deviations and correlations should also be converted into an investor’s chosen currency.</p>
<p>Unfortunately, this conversion is far more complex to perform than that of expected returns, so that - unless specified otherwise by the originating institution -
it is probably best to leave the original expected standard deviations and correlations untouched.</p>
<h2 id="implementations">Implementations</h2>
<h3 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h3>
<p><strong>Portfolio Optimizer</strong> allows to retrieve minimum, maximum and averaged 10-year capital market assumptions (nominal arithmetic and geometric returns, standard deviations, correlations)
from leading financial institutions and for selected asset classes:</p>
<ul>
<li>In EUR, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/markets/capital-assumptions/eur</code></a></li>
<li>In USD, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/markets/capital-assumptions/usd</code></a></li>
</ul>
<p>A couple of remarks:</p>
<ul>
<li>
<p>The list of financial institutions from which capital market assumptions have been sourced is provided in output of these endpoints.</p>
</li>
<li>
<p>The supported asset classes, together with a representative benchmark, are the following:</p>
<table>
<thead>
<tr>
<th>Asset Class</th>
<th>Asset Class Label in <strong>Portfolio Optimizer</strong></th>
<th>Representative Benchmark</th>
</tr>
</thead>
<tbody>
<tr>
<td>U.S. Cash</td>
<td>cash-us</td>
<td>J.P. Morgan Cash Index USD</td>
</tr>
<tr>
<td>Eurozone Cash</td>
<td>cash-eurozone</td>
<td>J.P. Morgan Cash Index Euro Currency</td>
</tr>
<tr>
<td>U.S. Government Bonds</td>
<td>governmentBonds-us</td>
<td>Bloomberg U.S. Treasury Index</td>
</tr>
<tr>
<td>Eurozone Government Bonds</td>
<td>governmentBonds-eurozone</td>
<td>Bloomberg Euro Aggregate Treasury Index</td>
</tr>
<tr>
<td>U.S. Investment Grade Corporate Bonds</td>
<td>corporateBonds-us</td>
<td>ICE BofA US Corporate Index</td>
</tr>
<tr>
<td>Eurozone Investment Grade Corporate Bonds</td>
<td>corporateBonds-eurozone</td>
<td>ICE BofA Euro Corporate Index</td>
</tr>
<tr>
<td>U.S. Equities</td>
<td>equities-us</td>
<td>MSCI USA Index</td>
</tr>
<tr>
<td>Eurozone Equities</td>
<td>equities-eurozone</td>
<td>MSCI EMU Index</td>
</tr>
<tr>
<td>Developed Markets Equities</td>
<td>equities-developedMarkets</td>
<td>MSCI Developed Markets Index</td>
</tr>
<tr>
<td>Emerging Markets Equities</td>
<td>equities-emergingMarkets</td>
<td>MSCI Emerging Markets Index</td>
</tr>
<tr>
<td>Commodities</td>
<td>commodities-all</td>
<td>Bloomberg Commodity Index</td>
</tr>
<tr>
<td>Gold</td>
<td>gold-all</td>
<td>Gold Spot Price</td>
</tr>
</tbody>
</table>
</li>
<li>
<p>The expected correlation matrices provided in output of these endpoints are not guaranteed to be complete nor valid.</p>
<ul>
<li>In case a correlation matrix is complete but invalid, it is possible to compute its nearest valid correlation matrix thanks to the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/correlation/matrix/nearest</code></a></li>
<li>In case a correlation matrix is incomplete, <strong>Portfolio Optimizer</strong> provides no solution for now but stay tuned!</li>
</ul>
</li>
</ul>
<p>Last, if you would like a specific currency or a specific asset class to be integrated into <strong>Portfolio Optimizer</strong>, feel free to <a href="/contact">reach out</a>.</p>
<h3 id="implementations-elsewhere">Implementations elsewhere</h3>
<p>I am aware of other capital market assumptions “averaging” services as described in this blog post:</p>
<ul>
<li><strong><a href="https://www.horizonactuarial.com/survey-of-capital-market-assumptions">Horizon Actuarial</a></strong> yearly survey of capital market assumptions</li>
<li><strong><a href="https://www.peresec.com/insights/2024-long-term-capital-market-assumptions-summary/">Peresec</a></strong> interactive summary of long-term capital market assumptions</li>
<li><strong><a href="https://ecrresearch.com/">ECR Research</a></strong> strategic asset allocation quarterly reports</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>This concludes this first blog post on capital market assumptions.</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Or capital market expectations (CMEs). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4184885">Sebastian, Michael D., The Accuracy and Use of Capital Market Assumptions</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:2:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:2:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.researchaffiliates.com/content/dam/ra/documents/asset-allocation/aai-methodology-0823.pdf">Research Affiliates Capital Market Expectations Methodology, 05.31.2023</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://www.horizonactuarial.com/_files/ugd/f76a4b_1057ff4efa7244d6bb7b1a8fb88236e6.pdf">Horizon Actuarial Services LLC, Survey of Capital Market Assumptions, 2023 Edition</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Additionally, Horizon Actuarial<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> notes that different institutions publish their capital market assumptions at different dates. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S1574070605010049">Allan Timmermann, Chapter 4 Forecast Combinations, Editor(s): G. Elliott, C.W.J. Granger, A. Timmermann, Handbook of Economic Forecasting, Elsevier, Volume 1, 2006, Pages 135-196</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:6:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:6:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>Again, impossible to identify ex-ante. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Of course, Figure 2 also shows that using averaged capital market assumptions prevented to use the best firm-specific estimates, but as discussed in the previous section, it is impossible to identify these ex-ante, so that’s not a real drawback. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>One could think of using an optimal-weighting forecast combination scheme, with for example the objective of minimizing the out-of-sample forecast error, but due to the small sample size over which the optimal weights would need to be estimated, such a scheme would probably lead to poor performance v.s. an equal-weighting scheme, as is usual in the forecasting litterature, c.f. Timmermann<sup id="fnref:6:7" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://ecrresearch.com/sites/default/files/research_downloads/Expected%20Returns%20Model%20Methodology_EXR.pdf">ECR Research, Expected Returns Model</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.monash.edu/business/mcfs/our-research/all-projects/investment-strategy/capital-market-assumptions">Lai Hoang, Ummul Ruthbah, Long-Term Capital Market Assumptions Public Equities</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:13:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:13:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>With or without an appropriate weighting per country representative of the asset class weighting per country, which quickly becomes a mess to manage… <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://www.bnpparibas-am.com/en-us/portfolio-perspectives/long-term-asset-allocation-the-slow-return-to-normal/">BNP Paribas AM, Long-Term Asset Allocation - The slow return to normal, Portfolio perspectives, White paper</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.invesco.com/us/en/insights/long-term-capital-market-assumptions-outlook.html">Invesco, 2024 long-term capital market assumptions</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>The same remark also applies to other portfolio allocation or optimization techniques, like Monte Carlo simulations. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Capital market assumptions1 (CMAs) are forecasts of future risk/return characteristics for broad asset classes over the next 5 to 20 years produced by leading investment managers, consultants and advisors2. These forecasts are well-reasoned, analytically rigorous assumptions about uncertain future market movements2 and are used almost universally among institutional investors2, for example as inputs to their stategical asset allocation3. In this blog post, I will show that combining capital market assumptions might be preferable to using any institution-specific estimates and I will describe a couple of practical data issues encountered when doing so. The variability of capital market assumptions Each year since 2010, people at Horizon Actuarial ask different investment firms to provide their capital market assumptions4, and release an associated survey. Figure 1 - taken from the 2023 edition of that survey4 - clearly shows significant differences in expected returns and standard deviations among investment advisors4. Figure 1. Distribution of expected returns and standard deviations for misc. asset classes over the next 10 years, Source: Horizon Actuarial. Of course, variability should come as no surprise, because different institutions use different forecasting methodologies and/or have different views on the future5… Nevertheless, such a high variability is definitely an issue for an investor since it is impossible to identify a priori the institution that will produce the most accurate forecasts of future returns, standard deviations and correlations. The risk of relying on a “bad” institution for an investor is therefore very real… Combining capital market assumptions The general forecast combination problem The specific problem described in the previous section is an instance of a more general problem known as the forecast combination problem, introduced as follows by Timmermann6: Faced with multiple forecasts of the same variable, an issue that immediately arises is how best to exploit information in the individual forecasts. In particular, should a single dominant forecast be identified or should a combination of the underlying forecasts be used to produce a pooled summary measure? From the forecasting literature, a pragmatic solution to this problem is to average all individual forecasts, which leads to an equal-weighting forecast combination scheme. While guaranteed to degrade the forecast accuracy v.s. using the “best” individual forecast7, such a forecast combination scheme has repeatedly been found to outperform sophisticated adaptive combination methods in empirical applications6, thus setting a benchmark that has proved surprisingly difficult to beat6. As an interesting side note for readers of this blog, it turns out that the forecast combination problem is closely related to the mean-variance portfolio optimization problem6. Averaging capital market assumptions From a theoretical perspective, an equal-weighted forecast combination scheme is well-suited to the specific context of capital market assumptions. Indeed, given the significant effort, resources and intellectual capital [put] into the production of [capital market assumptions]2, it is reasonable to assume that the relative size of the variance of the forecast errors associated with [the] different forecasting methods6 are all similar, which in theory favors an equal-weighting forecast combination scheme6. From a practical perspective, an equal-weighted forecast combination scheme is easy to compute and does not introduce any parameter estimation errors, which are known to be a serious problem for many combination techniques6. As for the forecast accuracy of that scheme, it is possible to have an idea of it thanks to Figure 2 - taken from Sebastian2 -, which compares: The min-max range of 10-year capital market assumptions produced by 19 investment firms for misc. asset classes in 2013 The associated averaged 10-year capital market assumptions The subsequent realized 10-year returns Figure 2. 2013 10-year capital market assumptions v.s. subsequent 10-year returns, Source: Sebastian. From Figure 2, it is visible that: Using averaged capital market assumptions v.s. firm-specific estimates prevented8 severe over/under-predictions for a couple of asset classes, like a ~3% over-prediction for Emerging Markets equities, which is great. In the majority of cases, the average capital market assumption [is] an unreliable guide to the future return2, which is not so great, but which ultimately depends on the accuracy and/or bias of the individual capital market assumptions and not on the forecast combination scheme itself9. That being said, a sample size of 1 observation per asset class is anyway maybe a little too small for reaching definite conclusions… Capital market assumptions data preparation Before being able to average capital market assumptions originating from different institutions, it is first required to collect and normalize these forecasts. Below are examples of typical data preparation steps for doing so. Collecting a sufficient number of forecasts Figure 3 - taken from an ECR Research report10 - illustrates the variety of forecast horizons for which a representative sample of leading investment firms provide their capital market assumptions. Figure 3. Representative frequency of investment firms capital market assumptions forecast horizons, Source: ECR Research. From Figure 3, and in oder to combine as many distinct capital market assumptions as possible, it seems a good idea to focus on collecting forecasts for either a 5-year or a 10-year horizon, with a preference for a 10-year horizon. Reconciliating asset classes across institutions Because not all investment advisors use the same asset classes when developing their capital market assumptions4, reconciliating different asset classes across institutions is mandatory and requires to exercice judgment in classifying [institutions] capital market assumptions into a standard set of asset2. As a side note, there are also sometimes inconsistencies in the name of asset classes. For example, under the asset class label Developed Equities, different institutions might mean: An asset class represented by the MSCI World Index, which would be the standard convention An asset class represented by the MSCI World ex-US Index An asset class represnted by the MSCI EAFE Index Converting expected returns Converting real expected returns to nominal expected returns The vast majority of institutions provide expected returns in nominal terms, but a couple of them provide expected returns in real terms. In order to compare apples to apples, it is necessary to convert these real returns into nominal returns by adding local expected inflation11, that is: U.S. expected inflation, for U.S. asset classes (U.S. Equities, U.S. Bonds…) as well as usually for commodities Eurozone expected inflation12, for euro-denominated asset classes (E.M.U. Equities, EUR Bonds…) Emerging countries inflation12, for Emerging Equities … Converting expected returns currencies Depending on the institution and on the asset class, expected returns can be provided in: USD, or another investable currency like EUR USD EUR-hedged, or another investable currency pair hedged like EUR JPY-hedged Local currency Local currency investable currency-hedged, or investable currency local currency-hedged or whatever weird combination In order to again compare apples to apples, it is necessary to convert all the currencies appearing in the different expected returns into a unique investable currency. Converting expected returns in an investable currency to expected returns in another investable currency The most usual scenario is the need to convert expected returns initially provided in an investable currency like USD into another investable currency like EUR. This convertion - ignoring the small geometric interaction11 - is done by adding the expected foreign exchange return to the expected returns11. Converting investable currency-hedged expected returns to investable currency-unhedged expected returns Expected returns for “foreign” fixed income asset classes are frequently provided as “foreign currency”-hedged13. The convertion from “foreign currency”-hedged expected returns to “foreign currency”-unhedged expected returns is done by correcting for the expected nominal risk-free rate differential14, that is, by adding the difference between the foreign and the local cash returns. Converting expected returns in local currency to expected returns in an investable currency Many institutions provide expected returns in local currency to avoid the issue of [determining] what an asset’s expected return will be from a particular investor’s currency perspective13. Ultimately, though, an investor needs to make a currency decision, and expected returns in local currency must be converted into expected returns in an investable currency like USD or EUR. For single-currency asset classes, like U.S. Equities or Eurozone Government Bonds, this is not an issue, c.f. the previous sub-sections. For multi-currency asset classes, like Emerging Market Equities or Global Government Bonds, the situation is more complex because a proper conversion would at the very least require assumptions on: The expected currency weights in the asset class at the forecast horizon The expected foreign exchange return at the forecast horizon, for all currencies To be noted that ignoring this complexity is probably not an option, as illustrated in Figure 4 - taken from the factsheet of the MSCI Emerging Markets Index -, which shows a ~3% difference in annualized returns over the last 10 years between the “local currency” and the “USD” flavors of the MSCI Emerging Markets Index. Figure 4. Cumulative MSCI Emerging Markets Net Returns Index performance, USD v.s. local currency (v.s. hedged to USD), May 2009 - May 2024, Source: MSCI. Between those two extremes, finding a pragmatic solution depending on the exact asset class is the best thing to do! Other convertions The need for other weird currency convertions might be encountered, at which point a balance must be struck between: Adding new forecasts, possibly unique, to the list of collected capital market assumptions Adding complexity to the capital market assumptions collection process Converting geometric expected returns to arithmetic expected returns Nearly all institutions produce estimates of geometric (compound) asset class returns14, while standard mean-variance portfolio optimization15 requires expected returns to be provided in arithmetic rather than in geometric terms14. In order to approximately convert geometric expected returns to arithmetic expected returns, it suffices to add half of the expected asset class variance11. As a down-to-earth comment on why is that, I cannot resist sharing what people at Verus wrote in the 2024 edition of their capital market assumptions: This is the industry standard approach, but requires a complex explanation only a heavy quant could love, so we have chosen not to provide further details in this document […] Converting expected standard deviations/correlations Similar to expected returns, expected standard deviations and correlations should also be converted into an investor’s chosen currency. Unfortunately, this conversion is far more complex to perform than that of expected returns, so that - unless specified otherwise by the originating institution - it is probably best to leave the original expected standard deviations and correlations untouched. Implementations Implementation in Portfolio Optimizer Portfolio Optimizer allows to retrieve minimum, maximum and averaged 10-year capital market assumptions (nominal arithmetic and geometric returns, standard deviations, correlations) from leading financial institutions and for selected asset classes: In EUR, through the endpoint /markets/capital-assumptions/eur In USD, through the endpoint /markets/capital-assumptions/usd A couple of remarks: The list of financial institutions from which capital market assumptions have been sourced is provided in output of these endpoints. The supported asset classes, together with a representative benchmark, are the following: Asset Class Asset Class Label in Portfolio Optimizer Representative Benchmark U.S. Cash cash-us J.P. Morgan Cash Index USD Eurozone Cash cash-eurozone J.P. Morgan Cash Index Euro Currency U.S. Government Bonds governmentBonds-us Bloomberg U.S. Treasury Index Eurozone Government Bonds governmentBonds-eurozone Bloomberg Euro Aggregate Treasury Index U.S. Investment Grade Corporate Bonds corporateBonds-us ICE BofA US Corporate Index Eurozone Investment Grade Corporate Bonds corporateBonds-eurozone ICE BofA Euro Corporate Index U.S. Equities equities-us MSCI USA Index Eurozone Equities equities-eurozone MSCI EMU Index Developed Markets Equities equities-developedMarkets MSCI Developed Markets Index Emerging Markets Equities equities-emergingMarkets MSCI Emerging Markets Index Commodities commodities-all Bloomberg Commodity Index Gold gold-all Gold Spot Price The expected correlation matrices provided in output of these endpoints are not guaranteed to be complete nor valid. In case a correlation matrix is complete but invalid, it is possible to compute its nearest valid correlation matrix thanks to the endpoint /assets/correlation/matrix/nearest In case a correlation matrix is incomplete, Portfolio Optimizer provides no solution for now but stay tuned! Last, if you would like a specific currency or a specific asset class to be integrated into Portfolio Optimizer, feel free to reach out. Implementations elsewhere I am aware of other capital market assumptions “averaging” services as described in this blog post: Horizon Actuarial yearly survey of capital market assumptions Peresec interactive summary of long-term capital market assumptions ECR Research strategic asset allocation quarterly reports Conclusion This concludes this first blog post on capital market assumptions. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – Or capital market expectations (CMEs). ↩ See Sebastian, Michael D., The Accuracy and Use of Capital Market Assumptions. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 See Research Affiliates Capital Market Expectations Methodology, 05.31.2023. ↩ See Horizon Actuarial Services LLC, Survey of Capital Market Assumptions, 2023 Edition. ↩ ↩2 ↩3 ↩4 Additionally, Horizon Actuarial4 notes that different institutions publish their capital market assumptions at different dates. ↩ See Allan Timmermann, Chapter 4 Forecast Combinations, Editor(s): G. Elliott, C.W.J. Granger, A. Timmermann, Handbook of Economic Forecasting, Elsevier, Volume 1, 2006, Pages 135-196. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 Again, impossible to identify ex-ante. ↩ Of course, Figure 2 also shows that using averaged capital market assumptions prevented to use the best firm-specific estimates, but as discussed in the previous section, it is impossible to identify these ex-ante, so that’s not a real drawback. ↩ One could think of using an optimal-weighting forecast combination scheme, with for example the objective of minimizing the out-of-sample forecast error, but due to the small sample size over which the optimal weights would need to be estimated, such a scheme would probably lead to poor performance v.s. an equal-weighting scheme, as is usual in the forecasting litterature, c.f. Timmermann6. ↩ See ECR Research, Expected Returns Model. ↩ See Lai Hoang, Ummul Ruthbah, Long-Term Capital Market Assumptions Public Equities. ↩ ↩2 ↩3 ↩4 With or without an appropriate weighting per country representative of the asset class weighting per country, which quickly becomes a mess to manage… ↩ ↩2 See BNP Paribas AM, Long-Term Asset Allocation - The slow return to normal, Portfolio perspectives, White paper. ↩ ↩2 See Invesco, 2024 long-term capital market assumptions. ↩ ↩2 ↩3 The same remark also applies to other portfolio allocation or optimization techniques, like Monte Carlo simulations. ↩Volatility Forecasting: HAR Model2024-06-25T00:00:00-05:002024-06-25T00:00:00-05:00https://portfoliooptimizer.io/blog/volatility-forecasting-har-model<p>Among the different members of the family of volatility forecasting models by weighted moving average<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> like
<a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the simple and the exponentially weighted moving average models</a> or
<a href="/blog/volatility-forecasting-garch11-model/">the GARCH(1,1) model</a>, the <em>Heterogeneous AutoRegressive (HAR) model</em> introduced by Corsi<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>
<em>has become the workhorse of the volatility forecasting literature</em><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> <em>on account of its simplicity and generally good forecasting performance</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>.</p>
<p>In this blog post, strongly inspired by the paper <em>A Practical Guide to harnessing the HAR volatility model</em> from Clements and Preve<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, I will describe the HAR volatility forecasting model
together with some important implementation details and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs.</p>
<h2 id="mathematical-preliminaries-reminders">Mathematical preliminaries (reminders)</h2>
<p>This section contains reminders from a <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a>.</p>
<h3 id="volatility-modelling-and-volatility-proxies">Volatility modelling and volatility proxies</h3>
<p>Let $r_t$ be the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>) return of an asset over a time period $t$ (a day, a week, a month..),
over which its (conditional) mean return is supposed to be null.</p>
<p>Then:</p>
<ul>
<li>
<p>The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $</p>
<p>From this definition, the squared return $r_t^2$ of an asset is a (noisy<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>) <em>variance estimator</em> - or <em>variance proxy</em><sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> - for that asset variance over the considered time period.</p>
<p>Another example of an asset variance proxy is <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">the Parkinson range</a> of an asset.</p>
<p>Yet another example of an asset variance proxy, this time over a specific time period $t$ of one day, is the <em>daily realized variance</em> $RV_t$ of an asset, which is defined as the sum of the asset squared intraday returns sampled at a high frequency (1 minutes, 5 minutes, 15 minutes…).</p>
<p>The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$.</p>
</li>
<li>
<p>The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $</p>
<p>The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$.</p>
</li>
</ul>
<h3 id="weighted-moving-average-volatility-forecasting-model">Weighted moving average volatility forecasting model</h3>
<p>Boudoukh et al.<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> show that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s
variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with</p>
\[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\]
<p>, where:</p>
<ul>
<li>$1 \leq k \leq T$ is the size of the moving average, possibly time-dependent</li>
<li>$w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well</li>
</ul>
<h2 id="the-har-volatility-forecasting-model">The HAR volatility forecasting model</h2>
<h3 id="the-original-har-model">The original HAR model</h3>
<p>Due to the limitations of the GARCH model in reproducing <em>the main empirical features of financial returns (long memory, fat tails, and self-similarity)</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, Corsi<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>
proposes to use <em>an additive cascade model of different volatility components each of which is generated by the actions of different types of market participants</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>.</p>
<p>As detailled by Corsi<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>:</p>
<blockquote>
<p>The main idea is that agents with different time horizons perceive, react to, and cause different types of volatility components. Simplifying a bit, we can identify three primary volatility components: the short-term traders with daily or higher trading frequency, the medium-term investors who typically rebalance their positions weekly, and the long-term agents with a characteristic time of one or more months.</p>
</blockquote>
<p>Under this volatility forecasting model called the <em>Heterogeneous AutoRegressive model (HAR)</em><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">5</a></sup>, an asset next day’s daily realized variance $RV_{T+1}$ is modeled as
an <a href="https://en.wikipedia.org/wiki/Autoregressive_model">AR(22) process</a> subject to <em>economically meaningful restrictions</em><sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup> on its parameters, which results in the formula<sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></p>
\[\hat{RV}_{T+1} = \beta + \beta_d RV_{T} + \beta_w RV_{T}^w + \beta_m RV_{T}^m\]
<p>, where:</p>
<ul>
<li>$\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$</li>
<li>$RV_T$ is the asset daily realized variance at time $T$</li>
<li>$RV_T^w = \frac{1}{5} \sum_{i=1}^5 RV_{T-i+1}$ is the asset weekly realized variance at time $T$</li>
<li>$RV_T^m = \frac{1}{22} \sum_{i=1}^{22} RV_{T-i+1}$ is the asset monthly realized variance at time $T$</li>
<li>$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined</li>
</ul>
<p>In terms of practical performances, and <em>in spite of its simplicity […], the [HAR] model is able to reproduce the same volatility persistence observed in the empirical data as well as many
of the other main stylized facts of financial data</em><sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, which makes it a very accurate volatility forecasting model.</p>
<h3 id="realized-variance-vs-generic-variance-proxy">Realized variance v.s. generic variance proxy</h3>
<p>The original HAR model described in the previous subsection relies on a very specific asset variance proxy - the realized variance of an asset - over a very specific time period - a day - for its definition.</p>
<p>Some papers (Lyocsa et al.<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">6</a></sup>, Lyocsa et al.<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">7</a></sup>, Clements et al.<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>, …) propose to replace the (high-frequency) daily realized variance by a (low-frequency) daily range-based variance estimator<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>
like:</p>
<ul>
<li>The square of the Parkinson volatility estimator</li>
<li>The square of the Garman-Klass volatility estimator</li>
<li>The square of the Rogers-Satchell volatility estimator</li>
</ul>
<p>Going one step further, it is possible to replace the daily realized variance by any generic daily variance estimator.</p>
<p>This leads to the generic HAR volatility forecasting model, under which an asset next days’s conditional variance $\sigma_{T+1}^2$ is modeled as
<em>a linear function of [its previous day’s] daily, weekly and monthly [conditional variance] components</em><sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, with the following daily variance forecasting formula</p>
\[\hat{\sigma}_{T+1}^2 = \beta + \beta_d \tilde{\sigma}^2_{T} + \beta_w \tilde{\sigma}^{2,w}_{T} + \beta_m \tilde{\sigma}^{2,m}_{T}\]
<p>, where:</p>
<ul>
<li>$\hat{\sigma}_{T+1}^2$ is the forecast at time $T$ of the asset next day’s conditional variance $\sigma_{T+1}^2$</li>
<li>$\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$</li>
<li>$\tilde{\sigma}^{2,w}_{T} = \frac{1}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$</li>
<li>$\tilde{\sigma}^{2,m}_{T} = \frac{1}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$</li>
<li>$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined</li>
</ul>
<p>Going another step further, it is also possible to replace the baseline daily time period by any desired time period (weekly, biweekly, monthly, quarterly…), but
given the theoretical foundations of the HAR model, this also requires to replace the weekly and the monthly variance estimators $\tilde{\sigma}^{2,w}_{T}$ and $\tilde{\sigma}^{2,m}_{T}$
by appropriate variance estimators.</p>
<h3 id="relationship-with-the-generic-weighted-moving-average-model">Relationship with the generic weighted moving average model</h3>
<p>From its definition, it is easy to see that the HAR volatility forecasting model is a specific kind of weighted moving average volatility forecasting model, with:</p>
<ul>
<li>$w_0 = \beta$</li>
<li>$w_1 = \beta_d + \frac{1}{5} \beta_w + \frac{1}{22} \beta_m$</li>
<li>$w_i = \frac{1}{5} \beta_w + \frac{1}{22} \beta_m, i = 2..5$</li>
<li>$w_i = \frac{1}{22} \beta_m, i = 6..22$</li>
<li>$w_i = 0$, $i \geq 23$, discarding all the past variance proxies beyond the $22$-th from the model</li>
</ul>
<h3 id="volatility-forecasting-formulas">Volatility forecasting formulas</h3>
<p>Under an HAR volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes:</p>
<ul>
<li>
<p>To estimate an asset next day’s volatility:</p>
\[\hat{\sigma}_{T+1} = \sqrt{ \beta + \beta_d \tilde{\sigma}^2_{T} + \frac{\beta_w}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i} + \frac{\beta_m}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i} }\]
</li>
<li>
<p>To estimate an asset next $h$-day’s ahead volatility<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">9</a></sup>, $h \geq 2$:</p>
\[\hat{\sigma}_{T+h} = \sqrt{ \beta + \beta_d \hat{\sigma}_{T+h-1}^2 + \frac{\beta_w}{5} \left( \sum_{i=1}^{5-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) + \frac{\beta_m}{22} \left( \sum_{i=1}^{22-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) }\]
</li>
<li>
<p>To estimate an asset aggregated volatility<sup id="fnref:16:1" role="doc-noteref"><a href="#fn:16" class="footnote">9</a></sup> over the next $h$ days:</p>
\[\hat{\sigma}_{T+1:T+h} = \sqrt{ \sum_{i=1}^{h} \hat{\sigma}^2_{T+i} }\]
</li>
</ul>
<blockquote>
<p><strong><em>Note:</em></strong><br />
Clements and Preve<sup id="fnref:6:5" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> extensively discuss whether to use an indirect or a direct multi-step ahead forecast scheme for estimating an asset next $h$-day’s ahead volatility under an HAR volatility forecasting model:</p>
<ul>
<li>In an indirect scheme<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">10</a></sup>, which is the scheme described above, the asset next $h$-day’s ahead volatility is estimated indirectly, by a recursive application of the HAR model formula for the asset next days’s conditional variance.</li>
<li>In a direct scheme, the asset next $h$-day’s ahead volatility is estimated directly, by replacing the asset next days’s conditional variance on the left-hand side of the HAR model formula with the asset next $h$-day’s ahead conditional variance.</li>
</ul>
<p>Clements and Preve<sup id="fnref:6:6" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> argue in particular that <em>“direct forecasts are easy to compute and more robust to model misspecification compared to indirect forecasts”</em><sup id="fnref:6:7" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, but at the same time, they show that this robustness does not always translate into more accurate forecasts.</p>
</blockquote>
<h2 id="harnessing-the-har-volatility-forecasting-model">Harnessing the HAR volatility forecasting model</h2>
<h3 id="har-model-parameters-estimation">HAR model parameters estimation</h3>
<h4 id="ordinary-least-squares-estimators">Ordinary least squares estimators</h4>
<p>Corsi<sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup> writes that it is possible to <em>easily estimate [the HAR model] parameters by applying simple linear regression</em><sup id="fnref:3:8" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, in which case
the <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS) estimator</a> of the parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ at time $T \geq 23$ is the solution of the minimization problem<sup id="fnref:6:8" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></p>
\[\argmin_{ \left( \beta, \beta_d, \beta_w, \beta_m \right) \in \mathbb{R}^{4}} \sum_{t=23}^T \left( \tilde{\sigma}_{t}^2 - \beta - \beta_d \tilde{\sigma}^2_{t-1} - \beta_w \tilde{\sigma}^{2,w}_{t-1} - \beta_m \tilde{\sigma}^{2,m}_{t-1} \right)^2\]
<h4 id="other-least-squares-estimators">Other least squares estimators</h4>
<p>Nevertheless, Clements and Preve<sup id="fnref:6:9" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> warn <em>that given the stylized facts of [volatility estimators] (such as spikes/outliers, conditional heteroskedasticity, and non-Gaussianity)
and well-known properties of OLS, this […] should be far from ideal</em><sup id="fnref:6:10" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<p>Indeed, as detailled in Patton and Sheppard<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>:</p>
<blockquote>
<p>Because the dependent variable in all of our regressions is a volatility measure, estimation by OLS has the unfortunate feature that the resulting estimates focus primarily on fitting periods of high variance and place little weight on more tranquil periods. This is an important drawback in our applications, as the level of variance changes substantially across our sample period and the level of the variance and the volatility in the error are known to have a positive relationship.</p>
</blockquote>
<p>Instead of OLS, Clements and Preve<sup id="fnref:6:11" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> and Clements et al.<sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> suggest to use other least squares estimators:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Weighted_least_squares">Weighted least squares estimators (WLS)</a>, with for example the weighting schemes described in Clements and Preve<sup id="fnref:6:12" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup></li>
<li><a href="https://en.wikipedia.org/wiki/Robust_regression">Robust least squares estimators (RLS)</a>, with for example <a href="https://en.wikipedia.org/wiki/Redescending_M-estimator#Examples">Tukey’s biweight loss function</a></li>
<li><a href="https://en.wikipedia.org/wiki/Regularized_least_squares">Regularized least squares estimators (RRLS)</a> (ridge regression, LASSO regression, elastic net regression…), with the cross-validation procedure for the associated hyperparameters described in Clements et al.<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup>.</li>
</ul>
<h4 id="expanding-vs-rolling-window-estimation-procedure">Expanding v.s. rolling window estimation procedure</h4>
<p>Clements and Preve<sup id="fnref:6:13" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> empirically demonstrate that the HAR model parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are time-varying, as illustrated in Figure 1.</p>
<figure>
<a href="/assets/images/blog/volatility-forecasting-har-beta-variability-clements-preve.png"><img src="/assets/images/blog/volatility-forecasting-har-beta-variability-clements-preve.png" alt="Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve." /></a>
<figcaption>Figure 1. Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve.</figcaption>
</figure>
<p>To deal with this non-stationarity, or more generally to <em>deal with parameter drift that is difficult to model explicitly</em><sup id="fnref:6:14" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, the standard<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> approach in the litterature is to
use a rolling window procedure<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup> for the least squares estimation of the HAR model parameters.</p>
<h3 id="insanity-filter">Insanity filter</h3>
<p>The HAR volatility forecasting model <em>may on rare occasions generate implausibly large or small forecasts</em><sup id="fnref:6:15" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>, because <em>no restrictions on the parameters [$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$]
are imposed</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>In particular, it has been noted<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">14</a></sup> that <em>forecasts are occasionally negative</em><sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>In order to correct this behaviour, Clements and Preve<sup id="fnref:6:16" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> propose to implement an <em>insanity filter</em><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">15</a></sup> ensuring that <em>any forecast greater than the maximum, or less than the minimum,
of [an asset next days’s conditional variance] observed in the estimation period is replaced by the sample average over that period</em><sup id="fnref:6:17" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<p>Two important remarks on such a filter:</p>
<ul>
<li>It seems that <em>the particular choice of insanity filter is not important; what matters is to eliminate unrealistic forecasts</em><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">16</a></sup>.</li>
<li>The presence of an insanity filter is all the more important when an indirect multi-step ahead forecast scheme is used for estimating an asset next $h$-day’s ahead volatility, because <em>an unreasonable [forecast] is most likely to generate another unreasonable forecast</em><sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">16</a></sup>.</li>
</ul>
<h3 id="variance-proxies-transformations">Variance proxies transformations</h3>
<p>Clements and Preve<sup id="fnref:6:18" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> study three different <a href="https://encyclopediaofmath.org/wiki/Box-Cox_transformation">Box-Cox transformations</a> of an asset daily realized variance that can be used
in the HAR volatility forecasting model instead of that asset “raw” daily realized variance:</p>
<ul>
<li>The logarithmic transformation (log)</li>
<li>The quartic root transformation (qr)</li>
<li>The square root transformation (sqr)</li>
</ul>
<p>It is visible on Figure 2 that these transformations <em>appear useful for reducing skewness, and hence the possible effect of outliers and potential heteroskedasticity in the realized variance series</em><sup id="fnref:6:19" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>.</p>
<figure>
<a href="/assets/images/blog/volatility-forecasting-har-transformations-clements-preve.png"><img src="/assets/images/blog/volatility-forecasting-har-transformations-clements-preve.png" alt="S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve." /></a>
<figcaption>Figure 2. S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve.</figcaption>
</figure>
<p>Of particular interest is the logarithmic transformation<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">17</a></sup>, which in addition to good practical performances and closed form bias-corrected expressions for variance forecasts<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup>,
also guarantees that the generated forecasts are always positive<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">19</a></sup>.</p>
<h3 id="non-overlapping-daily-weekly-and-monthly-variance-proxies">Non-overlapping daily, weekly and monthly variance proxies</h3>
<p>Corsi et al.<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup> propose a slightly different parametrization of the HAR model compared to Corsi<sup id="fnref:3:9" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, where:</p>
<ul>
<li>$\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$</li>
<li>$\tilde{\sigma}^{2,w}_{T} = \frac{1}{4} \sum_{i=2}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$, non-overlapping with the asset daily variance estimator $\tilde{\sigma}^2_{T}$</li>
<li>$\tilde{\sigma}^{2,m}_{T} = \frac{1}{17} \sum_{i=6}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$, non-overlapping with either the asset daily variance estimator $\tilde{\sigma}^2_{T}$ nor with the asset weekly variance estimator $\tilde{\sigma}^{2,w}_{T}$</li>
</ul>
<p>Such a re-parametrization <em>does not imply any loss of information compared to the original [HAR model], since it relies only on a different rearrangement of the terms</em><sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup>, but allows an easiest
interpretation of the HAR model parameters, c.f. Patton and Sheppard<sup id="fnref:14:3" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup>.</p>
<p>Incidentally, this non-overlapping formulation of the HAR volatility forecasting model has been found to generate better forecasts by practitioners<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">21</a></sup>.</p>
<h3 id="other-lag-indexes-for-the-variance-proxies">Other lag indexes for the variance proxies</h3>
<p>The standard lag indexes for the variance proxies in the HAR volatility forecasting model are 1, 5 and 22, each corresponding to a different volatility component in Corsi’s underlying additive cascade model.</p>
<p>As mentioned in Corsi<sup id="fnref:3:10" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, <em>more components could easily be added to the additive cascade of partial volatilities</em><sup id="fnref:3:11" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, which is done for example in Lyocsa et al.<sup id="fnref:25:1" role="doc-noteref"><a href="#fn:25" class="footnote">7</a></sup>.</p>
<p>In that case, <em>denoting $l$ and $h$, respectively, the lowest and highest frequency in the cascade</em><sup id="fnref:3:12" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>, an asset next days’s conditional variance is modeled as an AR($\frac{l}{h}$) process
<em>reparameterized in a parsimonious way by imposing economically meaningful restrictions</em><sup id="fnref:3:13" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the HAR volatility forecasting model - augmented with the insanity filter described in Clements and Preve<sup id="fnref:6:20" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup> - through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/volatility/forecast/har</code></a>.</p>
<p>This endpoint supports the 4 variance proxies below:</p>
<ul>
<li>Squared close-to-close returns</li>
<li>Demeaned squared close-to-close returns</li>
<li>The Parkinson range</li>
<li>The jump-adjusted Parkinson range</li>
</ul>
<p>This endpoint also supports:</p>
<ul>
<li>
<p>Transforming the input variance proxies into log variance proxies before estimating the HAR model parameters; the associated variance forecasts are then unbiased thanks to the formulas established in Buccheri and Corsi<sup id="fnref:19:1" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup>.</p>
</li>
<li>Estimating the HAR model parameters through 3 different least squares procedures:
<ul>
<li>Ordinary least squares</li>
<li>Weighted least squares, using the inverse of the variance proxies as weights</li>
<li>Robust least squares, using Tukey’s biweight loss function</li>
</ul>
</li>
<li>Using up to 5 lag indexes for the variance proxies</li>
</ul>
<h2 id="example-of-usage---volatility-forecasting-at-monthly-level-for-various-etfs">Example of usage - Volatility forecasting at monthly level for various ETFs</h2>
<p>As an example of usage, I propose to enrich the results of <a href="/blog/volatility-forecasting-garch11-model/">the previous blog post</a>, in which
monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">22</a></sup> regressions - to the next month’s close-to-close observed volatility
for 10 ETFs representative<sup id="fnref:20:1" role="doc-noteref"><a href="#fn:20" class="footnote">17</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<h3 id="vanilla-har-volatility-forecasting-model">Vanilla HAR volatility forecasting model</h3>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">23</a></sup> are the following<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">24</a></sup>, when using the vanilla HAR volatility forecasting model:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td>Random walk</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.66</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.68</td>
<td>46%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Squared close-to-close returns</td>
<td>4.7%</td>
<td>0.73</td>
<td>45%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Squared close-to-close returns</td>
<td>-1.3%</td>
<td>0.98</td>
<td>43%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-0.7%</strong></td>
<td><strong>0.95</strong></td>
<td><strong>46%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Parkinson range</td>
<td>5.6%</td>
<td>0.94</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Parkinson range</td>
<td>5.1%</td>
<td>1.00</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Parkinson range</td>
<td>4.3%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Parkinson range</td>
<td>2.7%</td>
<td>1.18</td>
<td>47%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>0.1%</strong></td>
<td><strong>1.25</strong></td>
<td><strong>44%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.9%</td>
<td>0.70</td>
<td>45%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Jump-adjusted Parkinson range</td>
<td>5.1%</td>
<td>0.71</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.0%</td>
<td>0.76</td>
<td>45%</td>
</tr>
<tr>
<td>GARCH(1,1)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-1.0%</td>
<td>1.00</td>
<td>45%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.4%</strong></td>
<td><strong>0.99</strong></td>
<td><strong>47%</strong></td>
</tr>
</tbody>
</table>
<h3 id="alternative-har-volatility-forecasting-models">Alternative HAR volatility forecasting models</h3>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">23</a></sup> are the following<sup id="fnref:28:1" role="doc-noteref"><a href="#fn:28" class="footnote">24</a></sup>, when using different variations of the vanilla HAR volatility forecasting model:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-0.7%</strong></td>
<td><strong>0.95</strong></td>
<td><strong>46%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Squared close-to-close returns</td>
<td>3.1%</td>
<td>0.71</td>
<td>34%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Squared close-to-close returns</td>
<td>-12.3%</td>
<td>2.50</td>
<td>26%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Squared close-to-close returns</td>
<td>0.5%</td>
<td>0.62</td>
<td>40%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Squared close-to-close returns</td>
<td>10%</td>
<td>0.26</td>
<td>13%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Squared close-to-close returns</td>
<td>0.5%</td>
<td>0.53</td>
<td>40%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>0.1%</strong></td>
<td><strong>1.25</strong></td>
<td><strong>44%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Parkinson range</td>
<td>0.1%</td>
<td>1.19</td>
<td>44%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Parkinson range</td>
<td>-4.2%</td>
<td>2.20</td>
<td>40%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Parkinson range</td>
<td>1.9%</td>
<td>1.22</td>
<td>50%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Parkinson range</td>
<td>-0.6%</td>
<td>1.47</td>
<td>47%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Parkinson range</td>
<td>2.2%</td>
<td>1.22</td>
<td>50%</td>
</tr>
<tr>
<td><strong>HAR</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.4%</strong></td>
<td><strong>0.99</strong></td>
<td><strong>47%</strong></td>
</tr>
<tr>
<td>HAR (weighted least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-4.2%</td>
<td>0.92</td>
<td>46%</td>
</tr>
<tr>
<td>HAR (robust least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-6.6%</td>
<td>1.76</td>
<td>41%</td>
</tr>
<tr>
<td>HAR (log)</td>
<td>Jump-adjusted Parkinson range</td>
<td>0.9%</td>
<td>0.92</td>
<td>51%</td>
</tr>
<tr>
<td>HAR (log, weighted least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>-0.8%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td>HAR (log, robust least squares)</td>
<td>Jump-adjusted Parkinson range</td>
<td>1.2%</td>
<td>0.92</td>
<td>51%</td>
</tr>
</tbody>
</table>
<h3 id="comments">Comments</h3>
<p>From the results of the two previous subsections, it is possible to make the following comments:</p>
<ul>
<li>When using squared returns as a variance proxy, the vanilla HAR model is the best volatility forecasting model among all the alternative HAR models as well as among all the models already studied in this series.</li>
<li>When using the Parkinson range or the jump-adjusted Parkinson range as variance proxies, the log HAR model exhibits the highest r-squared among all the models already studied in this series.</li>
<li>When using the jump-adjusted Parkinson range as a variance proxy, the log HAR model is the best volatility forecasting model in this series (relatively low bias, highest r-squared).</li>
<li>When using alternative least square estimators, the forecasts quality generally degrades</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The previous section empirically demonstrated that the HAR volatility forecasting model, despite its native relationship with realized variance, is still accurate when used with
range-based variance estimators at a long forecasting horizon, in line with Lyocsa et al.<sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">6</a></sup></p>
<p>In that context, the (log) HAR model is also the most accurate volatility forecasting model studied so far in this series on volatility forecasting
by weighted moving average models!</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijderiv/4/3/63">Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article-abstract/7/2/174/856522">Fulvio Corsi, A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics, Volume 7, Issue 2, Spring 2009, Pages 174–196</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:3:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:3:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:3:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:3:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:3:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:3:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:3:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4733597">Clements, Adam and Preve, Daniel P. A. and Tee, Clarence, Harvesting the HAR-X Volatility Model</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0378426621002417">Adam Clements, Daniel P.A. Preve, A Practical Guide to harnessing the HAR volatility model, Journal of Banking & Finance, Volume 133, 2021</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:6:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:6:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:6:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:6:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:6:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:6:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:6:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:6:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:6:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a> <a href="#fnref:6:14" class="reversefootnote" role="doc-backlink">↩<sup>15</sup></a> <a href="#fnref:6:15" class="reversefootnote" role="doc-backlink">↩<sup>16</sup></a> <a href="#fnref:6:16" class="reversefootnote" role="doc-backlink">↩<sup>17</sup></a> <a href="#fnref:6:17" class="reversefootnote" role="doc-backlink">↩<sup>18</sup></a> <a href="#fnref:6:18" class="reversefootnote" role="doc-backlink">↩<sup>19</sup></a> <a href="#fnref:6:19" class="reversefootnote" role="doc-backlink">↩<sup>20</sup></a> <a href="#fnref:6:20" class="reversefootnote" role="doc-backlink">↩<sup>21</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>This additive cascade model is autoregressive in (daily) realized variance and combines realized variance over different horizons (daily, weekly, monthly), hence its name. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0169207020301874">Stefan Lyocsa, Peter Molnar, Tomas Vyrost, Stock market volatility forecasting: Do we need high-frequency data?, International Journal of Forecasting, Volume 37, Issue 3, 2021, Pages 1092-1110</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1544612320315907">Stefan Lyocsa, Tomas Plihal, Tomas Vyrost, FX market volatility modelling: Can we use low-frequency data?, Finance Research Letters, Volume 40, 2021, 101776</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:25:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>These three volatility estimators are described in details in <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">a previous blog post</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://centaur.reading.ac.uk/21316/">Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:16:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>Also called iterated scheme. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://direct.mit.edu/rest/article-abstract/97/3/683/58249/Good-Volatility-Bad-Volatility-Signed-Jumps-and?redirectedFrom=fulltext">Andrew J. Patton, Kevin Sheppard; Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility. The Review of Economics and Statistics 2015; 97 (3): 683–697</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:14:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>Some papers like Buccheri and Corsi<sup id="fnref:19:2" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup> describe how to explicitely model the HAR model parameters as time-varying, but in that case, those parameters are not anymore obtained through a simple least squares regression… <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>The length of the rolling window is typically 1000 days, corresponding approximately to 1000 days of trading; more details on the rolling window procedure can be found in Clements et al.<sup id="fnref:4:5" role="doc-noteref"><a href="#fn:4" class="footnote">3</a></sup> and a numerical comparison in terms of out-of-sample volatility forecasts between an expanding and a rolling window procedure can be found in Clements and Preve<sup id="fnref:6:21" role="doc-noteref"><a href="#fn:6" class="footnote">4</a></sup>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>And I confirm from practical experience. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>Such a filter has apparently a long history in the forecasting litterature, dating back at least to Swanson and White<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">25</a></sup> in the context of neural-network models for interest rates forecasting. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://repositorio.fgv.br/items/0bedff75-288e-47c1-a9fb-7c455bd5c533">Cunha, Ronan, Kock, Anders Bredahl and Pereira, Pedro L. Valls, Forecasting large covariance matrices: comparing autometrics and LASSOVAR</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Already hinted at in Corsi<sup id="fnref:3:14" role="doc-noteref"><a href="#fn:3" class="footnote">2</a></sup>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article-abstract/19/4/614/5553055?redirectedFrom=fulltext">Giuseppe Buccheri, Fulvio Corsi, HARK the SHARK: Realized Volatility Modeling with Measurement Errors and Nonlinear Dependencies, Journal of Financial Econometrics, Volume 19, Issue 4, Fall 2021, Pages 614–649</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:19:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:19:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>Which helps limiting the need for an insanity filter. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X12001808">Fulvio Corsi, Nicola Fusari, Davide La Vecchia, Realizing smiles: Options pricing with realized volatility, Journal of Financial Economics, Volume 107, Issue 2, 2013, Pages 284-304</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:23:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://saltfinancial.com/static/uploads/2021/05/The%20Laymans%20Guide%20to%20Volatility%20Forecasting.pdf">Salt Financial, The Layman’s Guide to Volatility Forecasting: Predicting the Future, One Day at a Time, Research Note</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://econpapers.repec.org/bookchap/nbrnberch/1214.htm">Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>For all models, I used an expanding window for the volatility forecast computation. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:28:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1392186">Swanson, N. R. and H. White (1995). A model-selection approach to assessing the information in the term structure using linear models and artificial neural networks. Journal of Business & Economic Statistics 13 (3), 265–275</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Among the different members of the family of volatility forecasting models by weighted moving average1 like the simple and the exponentially weighted moving average models or the GARCH(1,1) model, the Heterogeneous AutoRegressive (HAR) model introduced by Corsi2 has become the workhorse of the volatility forecasting literature3 on account of its simplicity and generally good forecasting performance3. In this blog post, strongly inspired by the paper A Practical Guide to harnessing the HAR volatility model from Clements and Preve4, I will describe the HAR volatility forecasting model together with some important implementation details and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs. Mathematical preliminaries (reminders) This section contains reminders from a previous blog post. Volatility modelling and volatility proxies Let $r_t$ be the (logarithmic) return of an asset over a time period $t$ (a day, a week, a month..), over which its (conditional) mean return is supposed to be null. Then: The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $ From this definition, the squared return $r_t^2$ of an asset is a (noisy4) variance estimator - or variance proxy4 - for that asset variance over the considered time period. Another example of an asset variance proxy is the Parkinson range of an asset. Yet another example of an asset variance proxy, this time over a specific time period $t$ of one day, is the daily realized variance $RV_t$ of an asset, which is defined as the sum of the asset squared intraday returns sampled at a high frequency (1 minutes, 5 minutes, 15 minutes…). The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$. The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $ The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$. Weighted moving average volatility forecasting model Boudoukh et al.1 show that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with \[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\] , where: $1 \leq k \leq T$ is the size of the moving average, possibly time-dependent $w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well The HAR volatility forecasting model The original HAR model Due to the limitations of the GARCH model in reproducing the main empirical features of financial returns (long memory, fat tails, and self-similarity)2, Corsi2 proposes to use an additive cascade model of different volatility components each of which is generated by the actions of different types of market participants2. As detailled by Corsi2: The main idea is that agents with different time horizons perceive, react to, and cause different types of volatility components. Simplifying a bit, we can identify three primary volatility components: the short-term traders with daily or higher trading frequency, the medium-term investors who typically rebalance their positions weekly, and the long-term agents with a characteristic time of one or more months. Under this volatility forecasting model called the Heterogeneous AutoRegressive model (HAR)5, an asset next day’s daily realized variance $RV_{T+1}$ is modeled as an AR(22) process subject to economically meaningful restrictions2 on its parameters, which results in the formula4 \[\hat{RV}_{T+1} = \beta + \beta_d RV_{T} + \beta_w RV_{T}^w + \beta_m RV_{T}^m\] , where: $\hat{RV}_{T+1}$ is the forecast at time $T$ of the asset next day’s daily realized variance $RV_{T+1}$ $RV_T$ is the asset daily realized variance at time $T$ $RV_T^w = \frac{1}{5} \sum_{i=1}^5 RV_{T-i+1}$ is the asset weekly realized variance at time $T$ $RV_T^m = \frac{1}{22} \sum_{i=1}^{22} RV_{T-i+1}$ is the asset monthly realized variance at time $T$ $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined In terms of practical performances, and in spite of its simplicity […], the [HAR] model is able to reproduce the same volatility persistence observed in the empirical data as well as many of the other main stylized facts of financial data2, which makes it a very accurate volatility forecasting model. Realized variance v.s. generic variance proxy The original HAR model described in the previous subsection relies on a very specific asset variance proxy - the realized variance of an asset - over a very specific time period - a day - for its definition. Some papers (Lyocsa et al.6, Lyocsa et al.7, Clements et al.3, …) propose to replace the (high-frequency) daily realized variance by a (low-frequency) daily range-based variance estimator8 like: The square of the Parkinson volatility estimator The square of the Garman-Klass volatility estimator The square of the Rogers-Satchell volatility estimator Going one step further, it is possible to replace the daily realized variance by any generic daily variance estimator. This leads to the generic HAR volatility forecasting model, under which an asset next days’s conditional variance $\sigma_{T+1}^2$ is modeled as a linear function of [its previous day’s] daily, weekly and monthly [conditional variance] components4, with the following daily variance forecasting formula \[\hat{\sigma}_{T+1}^2 = \beta + \beta_d \tilde{\sigma}^2_{T} + \beta_w \tilde{\sigma}^{2,w}_{T} + \beta_m \tilde{\sigma}^{2,m}_{T}\] , where: $\hat{\sigma}_{T+1}^2$ is the forecast at time $T$ of the asset next day’s conditional variance $\sigma_{T+1}^2$ $\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$ $\tilde{\sigma}^{2,w}_{T} = \frac{1}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$ $\tilde{\sigma}^{2,m}_{T} = \frac{1}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$ $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are the HAR model parameters, to be determined Going another step further, it is also possible to replace the baseline daily time period by any desired time period (weekly, biweekly, monthly, quarterly…), but given the theoretical foundations of the HAR model, this also requires to replace the weekly and the monthly variance estimators $\tilde{\sigma}^{2,w}_{T}$ and $\tilde{\sigma}^{2,m}_{T}$ by appropriate variance estimators. Relationship with the generic weighted moving average model From its definition, it is easy to see that the HAR volatility forecasting model is a specific kind of weighted moving average volatility forecasting model, with: $w_0 = \beta$ $w_1 = \beta_d + \frac{1}{5} \beta_w + \frac{1}{22} \beta_m$ $w_i = \frac{1}{5} \beta_w + \frac{1}{22} \beta_m, i = 2..5$ $w_i = \frac{1}{22} \beta_m, i = 6..22$ $w_i = 0$, $i \geq 23$, discarding all the past variance proxies beyond the $22$-th from the model Volatility forecasting formulas Under an HAR volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes: To estimate an asset next day’s volatility: \[\hat{\sigma}_{T+1} = \sqrt{ \beta + \beta_d \tilde{\sigma}^2_{T} + \frac{\beta_w}{5} \sum_{i=1}^5 \tilde{\sigma}^2_{T+1-i} + \frac{\beta_m}{22} \sum_{i=1}^{22} \tilde{\sigma}^2_{T+1-i} }\] To estimate an asset next $h$-day’s ahead volatility9, $h \geq 2$: \[\hat{\sigma}_{T+h} = \sqrt{ \beta + \beta_d \hat{\sigma}_{T+h-1}^2 + \frac{\beta_w}{5} \left( \sum_{i=1}^{5-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) + \frac{\beta_m}{22} \left( \sum_{i=1}^{22-h+1} \tilde{\sigma}^2_{T+1-i} + \sum_{i=1}^{h-1} \hat{\sigma}^2_{T+h-i} \right) }\] To estimate an asset aggregated volatility9 over the next $h$ days: \[\hat{\sigma}_{T+1:T+h} = \sqrt{ \sum_{i=1}^{h} \hat{\sigma}^2_{T+i} }\] Note: Clements and Preve4 extensively discuss whether to use an indirect or a direct multi-step ahead forecast scheme for estimating an asset next $h$-day’s ahead volatility under an HAR volatility forecasting model: In an indirect scheme10, which is the scheme described above, the asset next $h$-day’s ahead volatility is estimated indirectly, by a recursive application of the HAR model formula for the asset next days’s conditional variance. In a direct scheme, the asset next $h$-day’s ahead volatility is estimated directly, by replacing the asset next days’s conditional variance on the left-hand side of the HAR model formula with the asset next $h$-day’s ahead conditional variance. Clements and Preve4 argue in particular that “direct forecasts are easy to compute and more robust to model misspecification compared to indirect forecasts”4, but at the same time, they show that this robustness does not always translate into more accurate forecasts. Harnessing the HAR volatility forecasting model HAR model parameters estimation Ordinary least squares estimators Corsi2 writes that it is possible to easily estimate [the HAR model] parameters by applying simple linear regression2, in which case the ordinary least squares (OLS) estimator of the parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ at time $T \geq 23$ is the solution of the minimization problem4 \[\argmin_{ \left( \beta, \beta_d, \beta_w, \beta_m \right) \in \mathbb{R}^{4}} \sum_{t=23}^T \left( \tilde{\sigma}_{t}^2 - \beta - \beta_d \tilde{\sigma}^2_{t-1} - \beta_w \tilde{\sigma}^{2,w}_{t-1} - \beta_m \tilde{\sigma}^{2,m}_{t-1} \right)^2\] Other least squares estimators Nevertheless, Clements and Preve4 warn that given the stylized facts of [volatility estimators] (such as spikes/outliers, conditional heteroskedasticity, and non-Gaussianity) and well-known properties of OLS, this […] should be far from ideal4. Indeed, as detailled in Patton and Sheppard11: Because the dependent variable in all of our regressions is a volatility measure, estimation by OLS has the unfortunate feature that the resulting estimates focus primarily on fitting periods of high variance and place little weight on more tranquil periods. This is an important drawback in our applications, as the level of variance changes substantially across our sample period and the level of the variance and the volatility in the error are known to have a positive relationship. Instead of OLS, Clements and Preve4 and Clements et al.3 suggest to use other least squares estimators: Weighted least squares estimators (WLS), with for example the weighting schemes described in Clements and Preve4 Robust least squares estimators (RLS), with for example Tukey’s biweight loss function Regularized least squares estimators (RRLS) (ridge regression, LASSO regression, elastic net regression…), with the cross-validation procedure for the associated hyperparameters described in Clements et al.3. Expanding v.s. rolling window estimation procedure Clements and Preve4 empirically demonstrate that the HAR model parameters $\beta$, $\beta_d$, $\beta_w$ and $\beta_m$ are time-varying, as illustrated in Figure 1. Figure 1. Evolution of the HAR model weekly beta coefficient estimated through misc. least squares estimators (solid black - OLS, dashed colors - WLS and RLS) applied on a 1000-day rolling window, S&P 500 index daily realized variance, 6th April 2001 - 29th August 2013, Source: Clements and Preve. To deal with this non-stationarity, or more generally to deal with parameter drift that is difficult to model explicitly4, the standard12 approach in the litterature is to use a rolling window procedure13 for the least squares estimation of the HAR model parameters. Insanity filter The HAR volatility forecasting model may on rare occasions generate implausibly large or small forecasts4, because no restrictions on the parameters [$\beta$, $\beta_d$, $\beta_w$ and $\beta_m$] are imposed11. In particular, it has been noted14 that forecasts are occasionally negative11. In order to correct this behaviour, Clements and Preve4 propose to implement an insanity filter15 ensuring that any forecast greater than the maximum, or less than the minimum, of [an asset next days’s conditional variance] observed in the estimation period is replaced by the sample average over that period4. Two important remarks on such a filter: It seems that the particular choice of insanity filter is not important; what matters is to eliminate unrealistic forecasts16. The presence of an insanity filter is all the more important when an indirect multi-step ahead forecast scheme is used for estimating an asset next $h$-day’s ahead volatility, because an unreasonable [forecast] is most likely to generate another unreasonable forecast16. Variance proxies transformations Clements and Preve4 study three different Box-Cox transformations of an asset daily realized variance that can be used in the HAR volatility forecasting model instead of that asset “raw” daily realized variance: The logarithmic transformation (log) The quartic root transformation (qr) The square root transformation (sqr) It is visible on Figure 2 that these transformations appear useful for reducing skewness, and hence the possible effect of outliers and potential heteroskedasticity in the realized variance series4. Figure 2. S&P 500 index daily realized variance, raw v.s. transformed, 6th April 2001 - 29th August 2013, Source: Clements and Preve. Of particular interest is the logarithmic transformation17, which in addition to good practical performances and closed form bias-corrected expressions for variance forecasts18, also guarantees that the generated forecasts are always positive19. Non-overlapping daily, weekly and monthly variance proxies Corsi et al.20 propose a slightly different parametrization of the HAR model compared to Corsi2, where: $\tilde{\sigma}^2_{T}, \tilde{\sigma}^2_{T-1},…,\tilde{\sigma}^2_{T-21}$ are the asset daily variance estimators over each of the previous 22 days at times $T$, $T-1$, …, $T-21$ $\tilde{\sigma}^{2,w}_{T} = \frac{1}{4} \sum_{i=2}^5 \tilde{\sigma}^2_{T+1-i}$ is the asset weekly variance estimator at time $T$, non-overlapping with the asset daily variance estimator $\tilde{\sigma}^2_{T}$ $\tilde{\sigma}^{2,m}_{T} = \frac{1}{17} \sum_{i=6}^{22} \tilde{\sigma}^2_{T+1-i}$ is the asset monthly variance estimator at time $T$, non-overlapping with either the asset daily variance estimator $\tilde{\sigma}^2_{T}$ nor with the asset weekly variance estimator $\tilde{\sigma}^{2,w}_{T}$ Such a re-parametrization does not imply any loss of information compared to the original [HAR model], since it relies only on a different rearrangement of the terms20, but allows an easiest interpretation of the HAR model parameters, c.f. Patton and Sheppard11. Incidentally, this non-overlapping formulation of the HAR volatility forecasting model has been found to generate better forecasts by practitioners21. Other lag indexes for the variance proxies The standard lag indexes for the variance proxies in the HAR volatility forecasting model are 1, 5 and 22, each corresponding to a different volatility component in Corsi’s underlying additive cascade model. As mentioned in Corsi2, more components could easily be added to the additive cascade of partial volatilities2, which is done for example in Lyocsa et al.7. In that case, denoting $l$ and $h$, respectively, the lowest and highest frequency in the cascade2, an asset next days’s conditional variance is modeled as an AR($\frac{l}{h}$) process reparameterized in a parsimonious way by imposing economically meaningful restrictions2. Implementation in Portfolio Optimizer Portfolio Optimizer implements the HAR volatility forecasting model - augmented with the insanity filter described in Clements and Preve4 - through the endpoint /assets/volatility/forecast/har. This endpoint supports the 4 variance proxies below: Squared close-to-close returns Demeaned squared close-to-close returns The Parkinson range The jump-adjusted Parkinson range This endpoint also supports: Transforming the input variance proxies into log variance proxies before estimating the HAR model parameters; the associated variance forecasts are then unbiased thanks to the formulas established in Buccheri and Corsi18. Estimating the HAR model parameters through 3 different least squares procedures: Ordinary least squares Weighted least squares, using the inverse of the variance proxies as weights Robust least squares, using Tukey’s biweight loss function Using up to 5 lag indexes for the variance proxies Example of usage - Volatility forecasting at monthly level for various ETFs As an example of usage, I propose to enrich the results of the previous blog post, in which monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz22 regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative17 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Vanilla HAR volatility forecasting model Averaged results for all ETFs/regression models over each ETF price history23 are the following24, when using the vanilla HAR volatility forecasting model: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ Random walk Squared close-to-close returns 5.8% 0.66 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Squared close-to-close returns 5.8% 0.68 46% EWMA, optimal $\lambda$ Squared close-to-close returns 4.7% 0.73 45% GARCH(1,1) Squared close-to-close returns -1.3% 0.98 43% HAR Squared close-to-close returns -0.7% 0.95 46% Random walk Parkinson range 5.6% 0.94 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Parkinson range 5.1% 1.00 47% EWMA, optimal $\lambda$ Parkinson range 4.3% 1.06 48% GARCH(1,1) Parkinson range 2.7% 1.18 47% HAR Parkinson range 0.1% 1.25 44% Random walk Jump-adjusted Parkinson range 4.9% 0.70 45% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Jump-adjusted Parkinson range 5.1% 0.71 47% EWMA, optimal $\lambda$ Jump-adjusted Parkinson range 4.0% 0.76 45% GARCH(1,1) Jump-adjusted Parkinson range -1.0% 1.00 45% HAR Jump-adjusted Parkinson range -1.4% 0.99 47% Alternative HAR volatility forecasting models Averaged results for all ETFs/regression models over each ETF price history23 are the following24, when using different variations of the vanilla HAR volatility forecasting model: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ HAR Squared close-to-close returns -0.7% 0.95 46% HAR (weighted least squares) Squared close-to-close returns 3.1% 0.71 34% HAR (robust least squares) Squared close-to-close returns -12.3% 2.50 26% HAR (log) Squared close-to-close returns 0.5% 0.62 40% HAR (log, weighted least squares) Squared close-to-close returns 10% 0.26 13% HAR (log, robust least squares) Squared close-to-close returns 0.5% 0.53 40% HAR Parkinson range 0.1% 1.25 44% HAR (weighted least squares) Parkinson range 0.1% 1.19 44% HAR (robust least squares) Parkinson range -4.2% 2.20 40% HAR (log) Parkinson range 1.9% 1.22 50% HAR (log, weighted least squares) Parkinson range -0.6% 1.47 47% HAR (log, robust least squares) Parkinson range 2.2% 1.22 50% HAR Jump-adjusted Parkinson range -1.4% 0.99 47% HAR (weighted least squares) Jump-adjusted Parkinson range -4.2% 0.92 46% HAR (robust least squares) Jump-adjusted Parkinson range -6.6% 1.76 41% HAR (log) Jump-adjusted Parkinson range 0.9% 0.92 51% HAR (log, weighted least squares) Jump-adjusted Parkinson range -0.8% 1.06 48% HAR (log, robust least squares) Jump-adjusted Parkinson range 1.2% 0.92 51% Comments From the results of the two previous subsections, it is possible to make the following comments: When using squared returns as a variance proxy, the vanilla HAR model is the best volatility forecasting model among all the alternative HAR models as well as among all the models already studied in this series. When using the Parkinson range or the jump-adjusted Parkinson range as variance proxies, the log HAR model exhibits the highest r-squared among all the models already studied in this series. When using the jump-adjusted Parkinson range as a variance proxy, the log HAR model is the best volatility forecasting model in this series (relatively low bias, highest r-squared). When using alternative least square estimators, the forecasts quality generally degrades Conclusion The previous section empirically demonstrated that the HAR volatility forecasting model, despite its native relationship with realized variance, is still accurate when used with range-based variance estimators at a long forecasting horizon, in line with Lyocsa et al.6 In that context, the (log) HAR model is also the most accurate volatility forecasting model studied so far in this series on volatility forecasting by weighted moving average models! As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71. ↩ ↩2 See Fulvio Corsi, A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics, Volume 7, Issue 2, Spring 2009, Pages 174–196. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 See Clements, Adam and Preve, Daniel P. A. and Tee, Clarence, Harvesting the HAR-X Volatility Model. ↩ ↩2 ↩3 ↩4 ↩5 See Adam Clements, Daniel P.A. Preve, A Practical Guide to harnessing the HAR volatility model, Journal of Banking & Finance, Volume 133, 2021. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 This additive cascade model is autoregressive in (daily) realized variance and combines realized variance over different horizons (daily, weekly, monthly), hence its name. ↩ See Stefan Lyocsa, Peter Molnar, Tomas Vyrost, Stock market volatility forecasting: Do we need high-frequency data?, International Journal of Forecasting, Volume 37, Issue 3, 2021, Pages 1092-1110. ↩ ↩2 See Stefan Lyocsa, Tomas Plihal, Tomas Vyrost, FX market volatility modelling: Can we use low-frequency data?, Finance Research Letters, Volume 40, 2021, 101776. ↩ ↩2 These three volatility estimators are described in details in a previous blog post. ↩ See Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22. ↩ ↩2 Also called iterated scheme. ↩ See Andrew J. Patton, Kevin Sheppard; Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility. The Review of Economics and Statistics 2015; 97 (3): 683–697. ↩ ↩2 ↩3 ↩4 Some papers like Buccheri and Corsi18 describe how to explicitely model the HAR model parameters as time-varying, but in that case, those parameters are not anymore obtained through a simple least squares regression… ↩ The length of the rolling window is typically 1000 days, corresponding approximately to 1000 days of trading; more details on the rolling window procedure can be found in Clements et al.3 and a numerical comparison in terms of out-of-sample volatility forecasts between an expanding and a rolling window procedure can be found in Clements and Preve4. ↩ And I confirm from practical experience. ↩ Such a filter has apparently a long history in the forecasting litterature, dating back at least to Swanson and White25 in the context of neural-network models for interest rates forecasting. ↩ See Cunha, Ronan, Kock, Anders Bredahl and Pereira, Pedro L. Valls, Forecasting large covariance matrices: comparing autometrics and LASSOVAR. ↩ ↩2 Already hinted at in Corsi2. ↩ ↩2 See Giuseppe Buccheri, Fulvio Corsi, HARK the SHARK: Realized Volatility Modeling with Measurement Errors and Nonlinear Dependencies, Journal of Financial Econometrics, Volume 19, Issue 4, Fall 2021, Pages 614–649. ↩ ↩2 ↩3 Which helps limiting the need for an insanity filter. ↩ See Fulvio Corsi, Nicola Fusari, Davide La Vecchia, Realizing smiles: Options pricing with realized volatility, Journal of Financial Economics, Volume 107, Issue 2, 2013, Pages 284-304. ↩ ↩2 See Salt Financial, The Layman’s Guide to Volatility Forecasting: Predicting the Future, One Day at a Time, Research Note. ↩ See Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations. ↩ The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. ↩ ↩2 For all models, I used an expanding window for the volatility forecast computation. ↩ ↩2 See Swanson, N. R. and H. White (1995). A model-selection approach to assessing the information in the term structure using linear models and artificial neural networks. Journal of Business & Economic Statistics 13 (3), 265–275. ↩Combating Volatility Laundering: Unsmoothing Artificially Smoothed Returns2024-06-01T00:00:00-05:002024-06-01T00:00:00-05:00https://portfoliooptimizer.io/blog/combating-volatility-laundering-unsmoothing-artificially-smoothed-returns<h2 id="introduction">Introduction</h2>
<p>It is common knowledge that <em>returns to hedge funds and other alternative investments [like private equity or real estate] are often highly serially correlated</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>This results in apparently smooth returns that <em>have artificially lower volatilities and covariations with other asset classes</em><sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>, which in turn <em>bias [portfolio] allocations toward the smoothed asset classes</em><sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>In this blog post, I will detail two simple unsmoothing procedures designed to help with these problems.</p>
<p>As examples of usage, I will show that these procedures allow a fairer comparison between private and public U.S. real estate investments and I will
illustrate the impact of smoothed v.s. unsmoothed returns on the mean-variance efficient frontier associated to a universe made of both public and private investments.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>A Google Sheet corresponding to this post is available <a href="https://docs.google.com/spreadsheets/d/1s8sA5B8t2Hpxwxe8Tb7X983kPlW7elk96ZwxvLJ2NxA/edit?usp=sharing">here</a>.</li>
</ul>
</blockquote>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="autocorrelation-of-a-time-series">Autocorrelation of a time series</h3>
<p>Let be:</p>
<ul>
<li>$x_1,…,x_T$ the values of a time series observed over $T$ time periods</li>
<li>$\bar{x} = \frac{1}{T} \sum_{i=1}^T x_i$ the arithmetic average of that time series</li>
</ul>
<p>The <em>autocovariance function</em> - or <em>serial covariance function</em> - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">3</a></sup>:</p>
\[\gamma_n = \frac{1}{T} \sum_{i=\max(1,-n)}^{\min(T-n,T)} \left( x_{i+n} - \bar{x} \right) \left( x_i - \bar{x} \right)\]
<p>The <em>autocorrelation function</em> - or <em>serial correlation function</em> - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">3</a></sup>:</p>
\[\rho_n = \frac{\gamma_n}{\gamma_0}\]
<p>These functions represent the covariance and the correlation between the time series $x_{1+n},…,x_T$ and a delayed copy of itself $x_1,…,x_{T-n}$, hence their name.</p>
<h3 id="autocorrelation-of-financial-asset-returns-and-volatility-laundering">Autocorrelation of financial asset returns and volatility laundering</h3>
<p>In financial asset returns, it has been noted that both positive and negative autocorrelation <em>might be present simultaneously at different horizons</em><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">4</a></sup>.</p>
<p>Nevertheless, for both theoretical and practical reasons, any significant and persistent autocorrelation should quickly be arbitraged away<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">5</a></sup>.</p>
<p>It is thus quite remarkable that returns on several alternative assets (art<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">6</a></sup>, collectible stamps<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">7</a></sup>, hedge funds<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, private equity<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">8</a></sup>, private infrastructure<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">9</a></sup>, private real estate<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>, wine<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">11</a></sup>…)
have been found to be significantly (positively) autocorrelated.</p>
<p>Such an autocorrelation is nevertheless spurious - a <em>purely mathematical artifact</em><sup id="fnref:17:1" role="doc-noteref"><a href="#fn:17" class="footnote">9</a></sup> - and originates from the characteristics of these alternative assets:</p>
<ul>
<li>Illiquidity (infrequent trading, infrequent mark-to-market valuations…)</li>
<li>Smoothing of the underlying “true” economic returns</li>
<li>Lagging reporting and/or appraisal process</li>
<li>Fees structure</li>
<li>…</li>
</ul>
<p>Indeed, as explained<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> by Getmansky et al.<sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>:</p>
<blockquote>
<p>In such cases, the <em>reported</em> returns of [these assets] will appear to be smoother than true <em>economic</em> returns (returns that fully reflect all available market information […]) and this, in turn, will […] yield positive serial return correlation.</p>
</blockquote>
<p>Unfortunately, even if artificial, the presence of autocorrelation in these assets’ returns <em>invalidates traditional measures of risk and performance</em><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup> <em>such as volatility, Sharpe ratio,
correlation, and market-beta estimates</em><sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> and ultimately <em>presents the risks of [these] assets as more attractive than they are</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">11</a></sup>, especially when compared to typical public alternatives (stocks, REITs…).</p>
<p>Incidentally, the <em>smoothing service</em><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">14</a></sup> offered by these alternative assets has been named <em>volatility laundering</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">15</a></sup><sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">16</a></sup> by <a href="https://www.linkedin.com/in/cliff-asness/">Cliff Asness</a>, the founder of
<a href="https://www.aqr.com/">AQR Capital Management</a>.</p>
<h2 id="unsmoothing-artificially-smoothed-financial-asset-returns">Unsmoothing artificially smoothed financial asset returns</h2>
<p>In order to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">17</a></sup>, one possibility is to unsmooth (or desmooth) those smoothed returns
so as to recover the <em>true underlying (unobservable) returns</em><sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>.</p>
<p>In other words, given a time series $r_1,…,r_T$ of artificially smoothed financial asset returns, we would like to determine the “hidden” underlying unsmoothed time series $r_1^*,…,r_T^*$.</p>
<p>Now, of course, <em>the nature and interpretation of such an [unsmoothed] series depends on the assumptions that are imbedded in the smoothing model and unsmoothing procedure</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>.</p>
<p>The standard approach <em>is to rely on the assumption that the underlying true returns are uncorrelated across time, consistent with the classical hypothesis of weak-form informational efficiency in asset markets</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>.</p>
<p>Based on this assumption, several different unsmoothing procedures have been proposed in the litterature<sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup><sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">20</a></sup>, the two simplest ones being:</p>
<ul>
<li>The procedure of Geltner<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup> (or of Fisher-Geltner-Webb<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>), able to approximately remove the first-order autocorrelation from a time series</li>
<li>The procedure of Okunev and White<sup id="fnref:8:1" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>, an extension of Geltner’s procedure able to approximately remove the autocorrelation of any order from a time series</li>
</ul>
<h3 id="geltners-procedure">Geltner’s procedure</h3>
<p>The smoothing model of Geltner<sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup> assumes that the reported (smoothed) return at time $t$ $r_t$ is a linear combination of the true economic (unsmoothed) return at time $t$ $r_t^*$ and the one-period lagged reported
return $r_{t-1}$:</p>
\[r_t = \left( 1 - \alpha \right) r_t^* + \alpha r_{t-1}, t=2..T\]
<p>, with $\alpha \in \left[ 0,1 \right]$ an unknown parameter to be determined.</p>
<p>After some manipulations, this leads to the following formula for the time series $r_1^*,…,r_T^*$:</p>
\[r_1^* = r_1\]
\[r_t^* = \frac{r_t - c r_{t-1}}{1 - c}, t=2..T\]
<p>, with $c = \rho_1$ the first-order autocorrelation of the time series $r_1,…,r_T$.</p>
<p>It can be shown<sup id="fnref:8:2" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> that the unsmoothed time series has:</p>
<ul>
<li>An arithmetic mean equal to that of the original time series</li>
<li>Its first-order autocorrelation approximately null</li>
</ul>
<h3 id="okunev-and-whites-procedure">Okunev and White’s procedure</h3>
<p>One <em>difficulty with [Geltner’s procedure] is that it is only strictly correct for <a href="https://en.wikipedia.org/wiki/Autoregressive_model">an AR(1) process</a> and it only acts to remove first order autocorrelation</em><sup id="fnref:8:3" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</p>
<p>This led Okunev and White<sup id="fnref:8:4" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> to propose an extension of Geltner’s smoothing model by assuming that the reported return at time $t$ $r_t$ is a linear combination of the true economic
return at time $t$ $r_t^*$ and the lagged reported returns $r_{t-1}, r_{t-2}, …, r_{t-m}$:</p>
\[r_t = \left( 1 - \alpha \right) r_t^* + \sum_{i=1}^m \beta_i r_{t-i}, t=m+1..T\]
<p>, with:</p>
<ul>
<li>$m \geq 1$ the number of lagged reported returns to consider in order to determine the true economic returns, usually taken equal to 1 or 2 in practical applications</li>
<li>$\beta_i \in \left[ 0,1 \right]$, $i=1..m$</li>
<li>$\alpha = \sum_{i=1}^m \beta_i$</li>
</ul>
<p>Under that model, Okunev and White<sup id="fnref:8:5" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> describes how to approximately remove the $m$-th order autocorrelation from the time series $r_1,…,r_T$:</p>
<ol>
<li>
<p>Define $r_{0,t} = r_t$, $t=1..T$</p>
</li>
<li>
<p>Remove the first-order autocorrelation from the time series $r_{0,1},…,r_{0,T}$ by defining the time series $r_{1,1},…,r_{1,T}$</p>
\[r_{1,1} = r_{0,1}\]
\[r_{1,t} = \frac{r_{0,t} - c_1 r_{0,t-1}}{1 - c_1}, t=2..T\]
<p>, with:</p>
<ul>
<li>$c_1 = \frac{\left( 1+a_{0,2} \right) \pm \sqrt{\left( 1+a_{0,2} \right)^2 - 4a_{0,1}^2}}{2a_{0,1}}$, choosen so that $\left| c_1 \right| \leq 1$</li>
<li>$a_{0,1} = \rho_1$ the first-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$</li>
<li>$a_{0,2} = \rho_2$ the second-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$</li>
</ul>
</li>
<li>
<p>For $i=2..m$, remove the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ by defining the time series $r_{i,1},…,r_{i,T}$</p>
\[r_{i,t} = r_{i-1,t}, t=1..i\]
\[r_{i,t} = \frac{r_{i-1,t} - c_i r_{i-1,t-1}}{1 - c_i}, t=i+1..T\]
<p>, with:</p>
<ul>
<li>$c_i = \frac{\left( 1+a_{i-1,2i} \right) \pm \sqrt{\left( 1+a_{i-1,2i} \right)^2 - 4a_{i-1,i}^2}}{2a_{i-1,i}}$, choosen so that $\left| c_i \right| \leq 1$</li>
<li>$a_{i-1,i}$ the $i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$</li>
<li>$a_{i-1,2i}$ the $2i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$</li>
</ul>
</li>
</ol>
<p>It can be shown<sup id="fnref:8:6" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> that the unsmoothed time series has:</p>
<ul>
<li>An arithmetic mean equal to that of the original time series</li>
<li>Its first $m$-th order autocorrelations approximately null</li>
</ul>
<blockquote>
<p><strong><em>Notes on Okunev and White’s procedure:</em></strong></p>
<ul>
<li>It is iterative by nature, because removing the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ alters its $i-1$-th order autocorrelation; steps 1, 2 and 3 thus need to be applied iteratively, <em>until the first $m$ autocorrelations are sufficiently close to zero</em><sup id="fnref:8:7" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</li>
<li>It allows, more generally, to set the first $m$-th order autocorrelations of the time series $r_1,…,r_T$ to any desired values, under certain mathematical conditions.</li>
<li>Its influence on the skewness, the kurtosis and the <a href="/blog/corrected-cornish-fisher-expansion-improving-the-accuracy-of-modified-value-at-risk/">modified Value-At-Risk</a> of the unsmoothed returns has been studied<sup id="fnref:22:1" role="doc-noteref"><a href="#fn:22" class="footnote">12</a></sup> in Gallais-Hamonno and Nguyen-Thi-Thanh<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup>.</li>
</ul>
</blockquote>
<h3 id="misc-remarks">Misc. remarks</h3>
<p>A couple of miscealenous remarks on the two unsmoothing procedures described in the previous sub-sections:</p>
<ul>
<li>Both procedures should preferably be used with smoothed logarithmic<sup id="fnref:7:2" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> asset returns rather than smoothed arithmetic asset returns because unsmoothing arithmetic returns might result in returns that are less than -100%<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">22</a></sup>.</li>
<li>Both procedure require some numerical attention when determining the unsmoothing coefficients $c$ or $c_1,…,c_m$.</li>
<li>Okunev and White’s procedure better nullifies the first order autocorrelation of a time series of smoothed asset returns than Geltner’s procedure.</li>
</ul>
<h2 id="adjusting-the-performance-statistics-of-artificially-smoothed-financial-asset-returns">Adjusting the performance statistics of artificially smoothed financial asset returns</h2>
<p>Another possibility to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed is to adjust traditional performance statistics<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">23</a></sup><br />
<em>to control for the spurious serial correlation in [those] returns</em><sup id="fnref:8:8" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup>.</p>
<p>Describing such adjustments is out of scope of this blog post, but the interested reader can for example refer to:</p>
<ul>
<li>Lo<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">24</a></sup>, in which the “square root of time” rule typically used to annualize a monthly Sharpe Ratio is shown to be invalid in the presence of autocorrelation and in which a more appropriate adjustment method is determined.</li>
<li>Bailey and de Prado<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">25</a></sup>, in which the impact of first-order autocorrelation on maximum drawdown and time under water is analyzed.</li>
</ul>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements both Geltner’s and Okunev and White’s procedures through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/unsmoothed</code></a>.</p>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="comparing-private-and-public-real-estate-investments">Comparing private and public real estate investments</h3>
<p>In the blog post <em><a href="https://insights.finominal.com/research-the-case-against-private-markets/">The Case Against Private Markets</a></em>, <a href="https://www.linkedin.com/in/nicolas-rabener-caia-a345353a">Nicolas Rabener</a>
from <a href="https://finominal.com/">Finominal</a> highlights the <em>discrepancies between private and public market valuations</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">26</a></sup> in the case of U.S. real estate.</p>
<p>In more details, he compares the Blackstone Real Estate Income Trust <a href="https://www.breit.com/">BREIT</a> - a private real estate investment - together with the
<a href="https://institutional.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=FREL">Fidelity MSCI Real Estate Index ETF (FREL)</a> - a publicly traded real estate investment.</p>
<p>Rabener then shows that:</p>
<ul>
<li>The volatility of BREIT is absurdly low compared to that of the FREL ETF</li>
<li>The Sharpe Ratio of BREIT is thus absurdly high compared to that of the FREL ETF</li>
</ul>
<p>Problem is, those two investments are exposed to the same underlying economic drivers<sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">26</a></sup>, so that such differences do not really make sense…</p>
<p>I propose to revisit Rabener’s results, this time comparing BREIT with the <a href="https://investor.vanguard.com/investment-products/etfs/profile/vnq">Vanguard Real Estate Index Fund ETF (VNQ)</a>, both depicted in Figure 1<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">27</a></sup>.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-breit-vs-vnq.png"><img src="/assets/images/blog/combating-volatility-laundering-breit-vs-vnq.png" alt="Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024." /></a>
<figcaption>Figure 1. Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024.</figcaption>
</figure>
<p>Some statistics<sup id="fnref:41" role="doc-noteref"><a href="#fn:41" class="footnote">28</a></sup> for these two funds:</p>
<table>
<thead>
<tr>
<th>Instrument</th>
<th>Annualized Mean</th>
<th>Annualized Standard Deviation</th>
<th>Annualized Sharpe Ratio</th>
<th>First-order autocorrelation</th>
</tr>
</thead>
<tbody>
<tr>
<td>BREIT</td>
<td>10.10%</td>
<td>4.58%</td>
<td>2.20</td>
<td>0.42</td>
</tr>
<tr>
<td>VNQ ETF</td>
<td>6.26%</td>
<td>18.68%</td>
<td>0.34</td>
<td>-0.13</td>
</tr>
</tbody>
</table>
<p>From Figure 1 and the associated statistics (e.g., one-fourth of the volatility of its public counterpart, strong first-order autocorrelation), BREIT is a textbook case of volatility laundering that we can combat
thanks to Geltner’s procedure<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup>!</p>
<p>Figure 2 depicts the resulting unsmoothed BREIT together with the original BREIT and the VNQ ETF.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-breit-unsmoothed-vs-vnq.png"><img src="/assets/images/blog/combating-volatility-laundering-breit-unsmoothed-vs-vnq.png" alt="Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024." /></a>
<figcaption>Figure 2. Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024.</figcaption>
</figure>
<p>Some additional statistics for the unsmoothed BREIT:</p>
<table>
<thead>
<tr>
<th>Instrument</th>
<th>Annualized Mean</th>
<th>Annualized Standard Deviation</th>
<th>Annualized Sharpe Ratio</th>
<th>First-order autocorrelation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Unsmoothed BREIT</td>
<td>10.28%</td>
<td>6.85%</td>
<td>1.50</td>
<td>-0.04</td>
</tr>
</tbody>
</table>
<p>While Figure 2 is hardly distinguishable from Figure 1, the associated statistics are clear and consistent with the litterature:</p>
<ul>
<li>Unsmoothing the returns of BREIT increases its volatility by nearly 50%</li>
<li>Unsmoothing the returns of BREIT also decreases its annualized Sharpe Ratio by approximately 30%</li>
</ul>
<p>This small exercice empirically validates that unsmoothing private asset returns makes them <em>look and feel more like a public market proxy</em><sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">30</a></sup>.</p>
<p>To be noted that the unsmoothed BREIT still seems a little bit too smooth when compared to the VNQ ETF.</p>
<p>More on this later.</p>
<h3 id="analysing-hedge-funds-returns">Analysing hedge funds returns</h3>
<p>Brooks and Kat<sup id="fnref:31:1" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>, as well as many other authors<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:8:9" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup><sup id="fnref:20:1" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup><sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">31</a></sup>, observes that hedge funds exhibit significant levels of first - and sometimes second or higher - order autocorrelation,
which <em>has important consequences for investors</em><sup id="fnref:31:2" role="doc-noteref"><a href="#fn:31" class="footnote">18</a></sup>.</p>
<p>In particular, and thanks to the unsmoothing procedures previously described:</p>
<ul>
<li>
<p>Gallais-Hamonno and Nguyen-Thi-Thanh<sup id="fnref:20:2" role="doc-noteref"><a href="#fn:20" class="footnote">21</a></sup> notes that</p>
<blockquote>
<p>[…] the hedge fund performance measured by traditional ratios considerably decreases after unsmoothing, decrease which is on average 20%, even 25% according to the method and to the performance ratio used.</p>
</blockquote>
<blockquote>
<p>[…] 20% of [hedge funds] have a change in rankings. Moreover, there are several cases of strong “down-grading” and of strong “over-grading” […]</p>
</blockquote>
</li>
<li>
<p>Okunev and White<sup id="fnref:8:10" role="doc-noteref"><a href="#fn:8" class="footnote">19</a></sup> shows that</p>
<blockquote>
<p>After removing the autocorrelations from returns, we find increases in risk of between 60 and 100 percent for many of the individual indices.</p>
</blockquote>
</li>
</ul>
<p>So, when analysing hedge funds returns, a pre-requisite is to always use an unsmoothing procedure!</p>
<h3 id="mitigating-covariance-misspecification-in-asset-allocation">Mitigating covariance misspecification in asset allocation</h3>
<p>In the context of a multi-asset portfolio, Peterson and Grier<sup id="fnref:7:3" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> shows through a <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo</a> <a href="https://en.wikipedia.org/wiki/Modern_portfolio_theory">mean-variance</a>
experiment that <em>the naive covariance estimates [relying on smoothed asset returns] generate significant weight bias, undesirable allocative tilts, and higher portfolio risk</em><sup id="fnref:7:4" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>Similar in spirit to that paper, I propose to illustrate the impact of smoothed returns on mean-variance analysis.</p>
<p>For this:</p>
<ul>
<li>I consider a U.S.-centric universe of assets made of five <a href="https://www.preqin.com/">Preqin</a> private investment indexes and their <a href="https://www.preqin.com/help-center/articles/view/public-market-equivalents">public markets equivalents</a>:
<ul>
<li>Private Equity and S&P 500 index</li>
<li>Venture Capital and Russell 2000 index</li>
<li>Private Debt and an High Yield Bonds index<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">32</a></sup></li>
<li>Private Real Estate and MSCI U.S. Real Estate index</li>
<li>
<p>Private Infrastructure and S&P Infrastructure index</p>
<p>Figure 3 displays the (quarterly) evolution of all these indexes over the period 31 December 2007 - 31 December 2023.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-preqin-vs-pme.png"><img src="/assets/images/blog/combating-volatility-laundering-preqin-vs-pme.png" alt="Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 3. Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
</li>
</ul>
</li>
<li>Within that universe, I then compute two mean-variance efficient frontiers over the period 31 December 2007 - 31 December 2023, using in-sample annualized mean-variance inputs<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">33</a></sup>:
<ul>
<li>A “raw” efficient frontier, using reported returns for all indexes</li>
<li>
<p>An “unsmoothed” efficient frontier, using unsmoothed<sup id="fnref:37" role="doc-noteref"><a href="#fn:37" class="footnote">34</a></sup> reported returns for the private investment indexes and reported returns for their public markets equivalents</p>
<p>Figure 4 represents these two efficient frontiers in the mean-variance plane, together with the different indexes of the considered universe.</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-efficient-frontiers.png"><img src="/assets/images/blog/combating-volatility-laundering-efficient-frontiers.png" alt="Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 4. Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
<p>From Figure 4, and in accordance with Peterson and Grier<sup id="fnref:7:5" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>:</p>
<ul>
<li>The standard deviations of the private investment indexes increase significantly when their reported returns are unsmoothed.</li>
<li><em>The difference in the locations of the two frontiers is largely<sup id="fnref:38" role="doc-noteref"><a href="#fn:38" class="footnote">35</a></sup> a result of the underestimated standard deviations of the smoothed [private investment indexes]</em><sup id="fnref:7:6" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</li>
</ul>
<p>This confirms that <em>the naive covariance matrix underestimates risk and overestimates returns relative to the more efficient revised covariance matrix</em><sup id="fnref:7:7" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> and that unsmoothing is necessary to <em>remove potential biases and improve efficiency by maximizing the information content of [the mean-variance input] estimates</em><sup id="fnref:7:8" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup>.</p>
<p>One important remark, though, is that the unsmoothing procedure used only partially resolved the issues related to smoothed asset returns.</p>
<p>Indeed, the transition maps<sup id="fnref:39" role="doc-noteref"><a href="#fn:39" class="footnote">36</a></sup> of the two efficient frontiers (Figure 5) shows that unsmoothed mean-variance efficient portfolios are still heavily concentrated in private assets…</p>
<figure>
<a href="/assets/images/blog/combating-volatility-laundering-efficient-frontiers-transition-maps.png"><img src="/assets/images/blog/combating-volatility-laundering-efficient-frontiers-transition-maps.png" alt="Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023." /></a>
<figcaption>Figure 5. Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023.</figcaption>
</figure>
<p>So, to paraphrase <a href="https://www.linkedin.com/in/trym-riksen-6580952/">Trym Riksen</a>, private assets break the Markovitz’s model, and do so even when their “specificities” are accounted for!</p>
<p>More on this right now.</p>
</li>
</ul>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>A proper conclusion is certainly that <em>in general, [private asset returns] series must be unsmoothed before analysis</em><sup id="fnref:7:9" role="doc-noteref"><a href="#fn:7" class="footnote">2</a></sup> to avoid introducing any unwanted bias.</p>
<p>Nevertheless, the two unsmoothing procedures described in this blog post are not entirely satisfying because the resulting unsmoothed returns are yet too smooth…</p>
<p>Next in this series on combating volatility laundering, I will describe another unsmoothing procedure, this time able to take into account<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">37</a></sup> the economic relationship between
a private asset and its public market equivalent, which might improve its practical performances.</p>
<p>In order to smoothly stay up-to-date with I am doing, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X04000698">Mila Getmansky, Andrew W. Lo, Igor Makarov, An econometric model of serial correlation and illiquidity in hedge fund returns, Journal of Financial Economics, Volume 74, Issue 3, 2004, Pages 529-609</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/4480761">Steven P. Peterson and John T. Grier, Covariance Misspecification in Asset Allocation, Financial Analysts Journal, Vol. 62, No. 4 (Jul. - Aug., 2006), pp. 76-85</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:7:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:7:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:7:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:7:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:7:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:7:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:7:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:7:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://link.springer.com/book/10.1007/978-0-387-21706-2">Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://academic.oup.com/jfec/article/19/1/39/6124729">On the Autocorrelation of the Stock Market, Ian Martin, Journal of Financial Econometrics, Volume 19, Issue 1, Winter 2021, Pages 39–52</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://www.worldscientific.com/doi/abs/10.1142/9789814566926_0002">Samuelson P. A. 1965. Proof That Properly Anticipated Prices Fluctuate Randomly. Industrial Management Review 6: 41–49</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p><a href="https://www.pm-research.com/content/iijaltinv/10/4/64">Campbell, R., 2008. Art as a financial investment. Journal of Alternative Investments 10, 64–81</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0304405X10002941">Dimson, Elroy, and Christophe Spaenjers. 2011. Ex post: The investment performance of collectible stamps. Journal of Financial Economics 100:443–458</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://onlinelibrary.wiley.com/doi/10.1111/jofi.12688">Ang, A., Chen, B., Goetzmann, W. N., Phalippou, L., 2018. Estimating private equity returns from limited partner cash flows. The Journal of Finance 73, 1751–1783</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.cembenchmarking.com/ri/insight/16">Alexander D. Beath, PhD and Chris Flynn, CFA, ASSET ALLOCATION, COST OF INVESTING AND PERFORMANCE OF EUROPEAN DB PENSION FUNDS: THE IMPACT OF REAL ESTATE, CEM Benchmarking Inc.</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:17:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/BF00161933">Geltner, David, 1991, Smoothing in Appraisal-Based Returns, Journal of Real Estate Finance and Economics, Vol.4, p.327-345</a>, <a href="https://www.jstor.org/stable/44095449">Geltner, David, 1993, Estimating Market Values from Appraised Values without Assuming an Efficient Market, Journal of Real Estate Research, Vol.8, p.325-345</a> and <a href="https://link.springer.com/article/10.1007/BF01099972">Fisher, J. D., Geltner, D. M., & Webb, R. B. (1994). Value indices of commercial real estate: A comparison of index construction methods. The Journal of Realv Estate Finance and Economics, 9 (2), 137–164</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=4775496">Verdickt, Gertjan, Volatility Laundering: On the Feasibility of Wine Investment Funds (March 28, 2024)</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>In the context of hedge funds. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:22:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://academic.oup.com/rfs/advance-article-abstract/doi/10.1093/rfs/hhae006/7625074">Spencer J Couts, Andrei S Gonçalves, Andrea Rossi, Unsmoothing Returns of Illiquid Funds, The Review of Financial Studies, 2024</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/49/4/21">Antti Ilmanen, Investing in Interesting Times, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 21 - 35</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://www.aqr.com/Insights/Perspectives/Volatility-Laundering">Cliff Asness, Volatility Laundering, Perspective</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>In the context of private equity, c.f. the podcast <a href="https://twiii.podbean.com/e/private-equity-and-the-game-of-volatility-laundering/">Private Equity and the Game of ”Volatility Laundering”</a> from <a href="https://twiii.podbean.com/">This Week in Intelligent Investing</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Or <em>volatility laundered</em>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijaltinv/5/2/26">Brooks, C. and H. Kat, The statistical properties of Hedge Fund index returns and their implications for investors, The Journal of Alternative Investments, Fall 2002, 5 (2) 26-44</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:31:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:31:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijfixinc/16/2/46">Geoff Loudon, John Okunev, Derek White, Hedge Fund Risk Factors and the Value at Risk of Fixed Income Trading Strategies, The Journal of Fixed Income, Fall 2006, 16 (2) 46-61</a> and <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=460641">Okunev, John and White, Derek, Hedge Fund Risk Factors and Value at Risk of Credit Trading Strategies (October 2003)</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:8:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:8:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:8:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:8:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:8:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:8:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:8:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:8:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:8:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:8:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1059944">Shilling, James D. Measurement Error in FRC/NCREIF Returns on Real Estate. Southern Economic Journal, vol. 60, no. 1, 1993, pp. 210–19</a>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>See <a href="https://shs.hal.science/halshs-00184470">Georges Gallais-Hamonno, Huyen Nguyen-Thi-Thanh. The necessity to correct hedge fund returns: empirical evidence and correction method. 2007</a>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:20:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>For example, use the time series [-0.94, -0.96, -0.98, -0.95]. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>Called i.i.d.-based in de Prado<sup id="fnref:24:1" role="doc-noteref"><a href="#fn:24" class="footnote">25</a></sup>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.2469/faj.v58.n4.2453">Lo, A. W., 2002. The statistics of sharpe ratios. Financial Analysts Journal 58, No. 4, 36–52</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2201302">Bailey, David H. and Lopez de Prado, Marcos, Stop-Outs Under Serial Correlation and ‘The Triple Penance Rule’</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:24:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://insights.finominal.com/research-the-case-against-private-markets/">The Case Against Private Markets</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:41" role="doc-endnote">
<p>Due to the first-order autocorrelation in BREIT returns, the naive annualization of the volatility and of the Sharpe Ratio that I have done is incorrect, c.f. Lo<sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">24</a></sup>; this is done on purpose, though. <a href="#fnref:41" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>Okunev and White’s procedure gives nearly identical results. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>See <a href="https://www.venn.twosigma.com/insights/desmoothing-201">Bingxu Chen & Christopher Carrano, Two Easy Steps to Take Your Private Asset Returns Public, Vnn by Two Sigma</a>. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/28/1/6">Clifford S Asness, Robert J Krail, John M Liew, Do Hedge Funds Hedge?, The Journal of Portfolio Management, Fall 2001, 28 (1) 6-19</a>. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>The associated index is the Markit iBoxx USD Liquid High Yield Index; comparison of Private Debt to both Investment Grade and High Yield bonds is discussed in Boni and Manigart<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">38</a></sup>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>In-sample asset mean returns and in-sample asset returns covariance matrix, using asset returns over the period 31 March 2008 - 31 December 2023. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:37" role="doc-endnote">
<p>Reported returns for the private investment indexes were unsmoothed using the Okunev and White’s procedure of order 1; results are similar with Geltner’s procedure, but since I already used Geltner’s procedure in the context of real-estate, I decided to play with its extension. <a href="#fnref:37" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:38" role="doc-endnote">
<p>To be noted that the in-sample asset mean returns are numerically different between smoothed and unsmoothed returns, but this does not explain the differences between the two frontiers. <a href="#fnref:38" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:39" role="doc-endnote">
<p>The transition map of the efficient frontier is a stacked area plot of the composition, in terms of asset weights, of the efficient portfolios belonging to the efficient frontier. <a href="#fnref:39" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>That being said, a variant of Geltner’s method allow to use a public market equivalent of a private asset to “inject” volatility in the unsmoothed returns, c.f. Geltner<sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">10</a></sup>. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=3932484">Boni, Pascal and Manigart, Sophie, Private Debt Fund Returns, Persistence, and Market Conditions</a>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Introduction It is common knowledge that returns to hedge funds and other alternative investments [like private equity or real estate] are often highly serially correlated1. This results in apparently smooth returns that have artificially lower volatilities and covariations with other asset classes2, which in turn bias [portfolio] allocations toward the smoothed asset classes2. In this blog post, I will detail two simple unsmoothing procedures designed to help with these problems. As examples of usage, I will show that these procedures allow a fairer comparison between private and public U.S. real estate investments and I will illustrate the impact of smoothed v.s. unsmoothed returns on the mean-variance efficient frontier associated to a universe made of both public and private investments. Notes: A Google Sheet corresponding to this post is available here. Mathematical preliminaries Autocorrelation of a time series Let be: $x_1,…,x_T$ the values of a time series observed over $T$ time periods $\bar{x} = \frac{1}{T} \sum_{i=1}^T x_i$ the arithmetic average of that time series The autocovariance function - or serial covariance function - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as3: \[\gamma_n = \frac{1}{T} \sum_{i=\max(1,-n)}^{\min(T-n,T)} \left( x_{i+n} - \bar{x} \right) \left( x_i - \bar{x} \right)\] The autocorrelation function - or serial correlation function - of order $n \in \mathbb{N}$ of the time series $x_1,…,x_T$ is defined as3: \[\rho_n = \frac{\gamma_n}{\gamma_0}\] These functions represent the covariance and the correlation between the time series $x_{1+n},…,x_T$ and a delayed copy of itself $x_1,…,x_{T-n}$, hence their name. Autocorrelation of financial asset returns and volatility laundering In financial asset returns, it has been noted that both positive and negative autocorrelation might be present simultaneously at different horizons4. Nevertheless, for both theoretical and practical reasons, any significant and persistent autocorrelation should quickly be arbitraged away5. It is thus quite remarkable that returns on several alternative assets (art6, collectible stamps7, hedge funds1, private equity8, private infrastructure9, private real estate10, wine11…) have been found to be significantly (positively) autocorrelated. Such an autocorrelation is nevertheless spurious - a purely mathematical artifact9 - and originates from the characteristics of these alternative assets: Illiquidity (infrequent trading, infrequent mark-to-market valuations…) Smoothing of the underlying “true” economic returns Lagging reporting and/or appraisal process Fees structure … Indeed, as explained12 by Getmansky et al.1: In such cases, the reported returns of [these assets] will appear to be smoother than true economic returns (returns that fully reflect all available market information […]) and this, in turn, will […] yield positive serial return correlation. Unfortunately, even if artificial, the presence of autocorrelation in these assets’ returns invalidates traditional measures of risk and performance13 such as volatility, Sharpe ratio, correlation, and market-beta estimates1 and ultimately presents the risks of [these] assets as more attractive than they are11, especially when compared to typical public alternatives (stocks, REITs…). Incidentally, the smoothing service14 offered by these alternative assets has been named volatility laundering1516 by Cliff Asness, the founder of AQR Capital Management. Unsmoothing artificially smoothed financial asset returns In order to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed17, one possibility is to unsmooth (or desmooth) those smoothed returns so as to recover the true underlying (unobservable) returns18. In other words, given a time series $r_1,…,r_T$ of artificially smoothed financial asset returns, we would like to determine the “hidden” underlying unsmoothed time series $r_1^*,…,r_T^*$. Now, of course, the nature and interpretation of such an [unsmoothed] series depends on the assumptions that are imbedded in the smoothing model and unsmoothing procedure10. The standard approach is to rely on the assumption that the underlying true returns are uncorrelated across time, consistent with the classical hypothesis of weak-form informational efficiency in asset markets10. Based on this assumption, several different unsmoothing procedures have been proposed in the litterature101913120, the two simplest ones being: The procedure of Geltner10 (or of Fisher-Geltner-Webb10), able to approximately remove the first-order autocorrelation from a time series The procedure of Okunev and White19, an extension of Geltner’s procedure able to approximately remove the autocorrelation of any order from a time series Geltner’s procedure The smoothing model of Geltner10 assumes that the reported (smoothed) return at time $t$ $r_t$ is a linear combination of the true economic (unsmoothed) return at time $t$ $r_t^*$ and the one-period lagged reported return $r_{t-1}$: \[r_t = \left( 1 - \alpha \right) r_t^* + \alpha r_{t-1}, t=2..T\] , with $\alpha \in \left[ 0,1 \right]$ an unknown parameter to be determined. After some manipulations, this leads to the following formula for the time series $r_1^*,…,r_T^*$: \[r_1^* = r_1\] \[r_t^* = \frac{r_t - c r_{t-1}}{1 - c}, t=2..T\] , with $c = \rho_1$ the first-order autocorrelation of the time series $r_1,…,r_T$. It can be shown19 that the unsmoothed time series has: An arithmetic mean equal to that of the original time series Its first-order autocorrelation approximately null Okunev and White’s procedure One difficulty with [Geltner’s procedure] is that it is only strictly correct for an AR(1) process and it only acts to remove first order autocorrelation19. This led Okunev and White19 to propose an extension of Geltner’s smoothing model by assuming that the reported return at time $t$ $r_t$ is a linear combination of the true economic return at time $t$ $r_t^*$ and the lagged reported returns $r_{t-1}, r_{t-2}, …, r_{t-m}$: \[r_t = \left( 1 - \alpha \right) r_t^* + \sum_{i=1}^m \beta_i r_{t-i}, t=m+1..T\] , with: $m \geq 1$ the number of lagged reported returns to consider in order to determine the true economic returns, usually taken equal to 1 or 2 in practical applications $\beta_i \in \left[ 0,1 \right]$, $i=1..m$ $\alpha = \sum_{i=1}^m \beta_i$ Under that model, Okunev and White19 describes how to approximately remove the $m$-th order autocorrelation from the time series $r_1,…,r_T$: Define $r_{0,t} = r_t$, $t=1..T$ Remove the first-order autocorrelation from the time series $r_{0,1},…,r_{0,T}$ by defining the time series $r_{1,1},…,r_{1,T}$ \[r_{1,1} = r_{0,1}\] \[r_{1,t} = \frac{r_{0,t} - c_1 r_{0,t-1}}{1 - c_1}, t=2..T\] , with: $c_1 = \frac{\left( 1+a_{0,2} \right) \pm \sqrt{\left( 1+a_{0,2} \right)^2 - 4a_{0,1}^2}}{2a_{0,1}}$, choosen so that $\left| c_1 \right| \leq 1$ $a_{0,1} = \rho_1$ the first-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$ $a_{0,2} = \rho_2$ the second-order autocorrelation of the time series $r_{0,1},…,r_{0,T}$ For $i=2..m$, remove the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ by defining the time series $r_{i,1},…,r_{i,T}$ \[r_{i,t} = r_{i-1,t}, t=1..i\] \[r_{i,t} = \frac{r_{i-1,t} - c_i r_{i-1,t-1}}{1 - c_i}, t=i+1..T\] , with: $c_i = \frac{\left( 1+a_{i-1,2i} \right) \pm \sqrt{\left( 1+a_{i-1,2i} \right)^2 - 4a_{i-1,i}^2}}{2a_{i-1,i}}$, choosen so that $\left| c_i \right| \leq 1$ $a_{i-1,i}$ the $i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$ $a_{i-1,2i}$ the $2i$-th order autocorrelation of the time series $r_{i-1,1},…,r_{i-1,T}$ It can be shown19 that the unsmoothed time series has: An arithmetic mean equal to that of the original time series Its first $m$-th order autocorrelations approximately null Notes on Okunev and White’s procedure: It is iterative by nature, because removing the $i$-th order autocorrelation from the time series $r_{i-1,1},…,r_{i-1,T}$ alters its $i-1$-th order autocorrelation; steps 1, 2 and 3 thus need to be applied iteratively, until the first $m$ autocorrelations are sufficiently close to zero19. It allows, more generally, to set the first $m$-th order autocorrelations of the time series $r_1,…,r_T$ to any desired values, under certain mathematical conditions. Its influence on the skewness, the kurtosis and the modified Value-At-Risk of the unsmoothed returns has been studied12 in Gallais-Hamonno and Nguyen-Thi-Thanh21. Misc. remarks A couple of miscealenous remarks on the two unsmoothing procedures described in the previous sub-sections: Both procedures should preferably be used with smoothed logarithmic2 asset returns rather than smoothed arithmetic asset returns because unsmoothing arithmetic returns might result in returns that are less than -100%22. Both procedure require some numerical attention when determining the unsmoothing coefficients $c$ or $c_1,…,c_m$. Okunev and White’s procedure better nullifies the first order autocorrelation of a time series of smoothed asset returns than Geltner’s procedure. Adjusting the performance statistics of artificially smoothed financial asset returns Another possibility to provide a more accurate picture of the risks associated to assets whose returns have been artificially smoothed is to adjust traditional performance statistics23 to control for the spurious serial correlation in [those] returns19. Describing such adjustments is out of scope of this blog post, but the interested reader can for example refer to: Lo24, in which the “square root of time” rule typically used to annualize a monthly Sharpe Ratio is shown to be invalid in the presence of autocorrelation and in which a more appropriate adjustment method is determined. Bailey and de Prado25, in which the impact of first-order autocorrelation on maximum drawdown and time under water is analyzed. Implementation in Portfolio Optimizer Portfolio Optimizer implements both Geltner’s and Okunev and White’s procedures through the endpoint /assets/returns/unsmoothed. Examples of usage Comparing private and public real estate investments In the blog post The Case Against Private Markets, Nicolas Rabener from Finominal highlights the discrepancies between private and public market valuations26 in the case of U.S. real estate. In more details, he compares the Blackstone Real Estate Income Trust BREIT - a private real estate investment - together with the Fidelity MSCI Real Estate Index ETF (FREL) - a publicly traded real estate investment. Rabener then shows that: The volatility of BREIT is absurdly low compared to that of the FREL ETF The Sharpe Ratio of BREIT is thus absurdly high compared to that of the FREL ETF Problem is, those two investments are exposed to the same underlying economic drivers26, so that such differences do not really make sense… I propose to revisit Rabener’s results, this time comparing BREIT with the Vanguard Real Estate Index Fund ETF (VNQ), both depicted in Figure 127. Figure 1. Private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024. Some statistics28 for these two funds: Instrument Annualized Mean Annualized Standard Deviation Annualized Sharpe Ratio First-order autocorrelation BREIT 10.10% 4.58% 2.20 0.42 VNQ ETF 6.26% 18.68% 0.34 -0.13 From Figure 1 and the associated statistics (e.g., one-fourth of the volatility of its public counterpart, strong first-order autocorrelation), BREIT is a textbook case of volatility laundering that we can combat thanks to Geltner’s procedure29! Figure 2 depicts the resulting unsmoothed BREIT together with the original BREIT and the VNQ ETF. Figure 2. Smoothed and unsmoothed private real estate (BREIT) v.s. public real estate (VNQ ETF), January 2017 - March 2024. Some additional statistics for the unsmoothed BREIT: Instrument Annualized Mean Annualized Standard Deviation Annualized Sharpe Ratio First-order autocorrelation Unsmoothed BREIT 10.28% 6.85% 1.50 -0.04 While Figure 2 is hardly distinguishable from Figure 1, the associated statistics are clear and consistent with the litterature: Unsmoothing the returns of BREIT increases its volatility by nearly 50% Unsmoothing the returns of BREIT also decreases its annualized Sharpe Ratio by approximately 30% This small exercice empirically validates that unsmoothing private asset returns makes them look and feel more like a public market proxy30. To be noted that the unsmoothed BREIT still seems a little bit too smooth when compared to the VNQ ETF. More on this later. Analysing hedge funds returns Brooks and Kat18, as well as many other authors1192131, observes that hedge funds exhibit significant levels of first - and sometimes second or higher - order autocorrelation, which has important consequences for investors18. In particular, and thanks to the unsmoothing procedures previously described: Gallais-Hamonno and Nguyen-Thi-Thanh21 notes that […] the hedge fund performance measured by traditional ratios considerably decreases after unsmoothing, decrease which is on average 20%, even 25% according to the method and to the performance ratio used. […] 20% of [hedge funds] have a change in rankings. Moreover, there are several cases of strong “down-grading” and of strong “over-grading” […] Okunev and White19 shows that After removing the autocorrelations from returns, we find increases in risk of between 60 and 100 percent for many of the individual indices. So, when analysing hedge funds returns, a pre-requisite is to always use an unsmoothing procedure! Mitigating covariance misspecification in asset allocation In the context of a multi-asset portfolio, Peterson and Grier2 shows through a Monte Carlo mean-variance experiment that the naive covariance estimates [relying on smoothed asset returns] generate significant weight bias, undesirable allocative tilts, and higher portfolio risk2. Similar in spirit to that paper, I propose to illustrate the impact of smoothed returns on mean-variance analysis. For this: I consider a U.S.-centric universe of assets made of five Preqin private investment indexes and their public markets equivalents: Private Equity and S&P 500 index Venture Capital and Russell 2000 index Private Debt and an High Yield Bonds index32 Private Real Estate and MSCI U.S. Real Estate index Private Infrastructure and S&P Infrastructure index Figure 3 displays the (quarterly) evolution of all these indexes over the period 31 December 2007 - 31 December 2023. Figure 3. Private investment indexes v.s. public markets equivalents, 31 December 2007 - 31 December 2023. Within that universe, I then compute two mean-variance efficient frontiers over the period 31 December 2007 - 31 December 2023, using in-sample annualized mean-variance inputs33: A “raw” efficient frontier, using reported returns for all indexes An “unsmoothed” efficient frontier, using unsmoothed34 reported returns for the private investment indexes and reported returns for their public markets equivalents Figure 4 represents these two efficient frontiers in the mean-variance plane, together with the different indexes of the considered universe. Figure 4. Efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023. From Figure 4, and in accordance with Peterson and Grier2: The standard deviations of the private investment indexes increase significantly when their reported returns are unsmoothed. The difference in the locations of the two frontiers is largely35 a result of the underestimated standard deviations of the smoothed [private investment indexes]2. This confirms that the naive covariance matrix underestimates risk and overestimates returns relative to the more efficient revised covariance matrix2 and that unsmoothing is necessary to remove potential biases and improve efficiency by maximizing the information content of [the mean-variance input] estimates2. One important remark, though, is that the unsmoothing procedure used only partially resolved the issues related to smoothed asset returns. Indeed, the transition maps36 of the two efficient frontiers (Figure 5) shows that unsmoothed mean-variance efficient portfolios are still heavily concentrated in private assets… Figure 5. Transition maps of efficient frontiers for raw v.s. unsmoothed private investment indexes, 31 December 2007 - 31 December 2023. So, to paraphrase Trym Riksen, private assets break the Markovitz’s model, and do so even when their “specificities” are accounted for! More on this right now. Conclusion A proper conclusion is certainly that in general, [private asset returns] series must be unsmoothed before analysis2 to avoid introducing any unwanted bias. Nevertheless, the two unsmoothing procedures described in this blog post are not entirely satisfying because the resulting unsmoothed returns are yet too smooth… Next in this series on combating volatility laundering, I will describe another unsmoothing procedure, this time able to take into account37 the economic relationship between a private asset and its public market equivalent, which might improve its practical performances. In order to smoothly stay up-to-date with I am doing, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Mila Getmansky, Andrew W. Lo, Igor Makarov, An econometric model of serial correlation and illiquidity in hedge fund returns, Journal of Financial Economics, Volume 74, Issue 3, 2004, Pages 529-609. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 See Steven P. Peterson and John T. Grier, Covariance Misspecification in Asset Allocation, Financial Analysts Journal, Vol. 62, No. 4 (Jul. - Aug., 2006), pp. 76-85. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 See Venables, W. N. and Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer-Verlag. ↩ ↩2 See On the Autocorrelation of the Stock Market, Ian Martin, Journal of Financial Econometrics, Volume 19, Issue 1, Winter 2021, Pages 39–52. ↩ See Samuelson P. A. 1965. Proof That Properly Anticipated Prices Fluctuate Randomly. Industrial Management Review 6: 41–49. ↩ Campbell, R., 2008. Art as a financial investment. Journal of Alternative Investments 10, 64–81. ↩ See Dimson, Elroy, and Christophe Spaenjers. 2011. Ex post: The investment performance of collectible stamps. Journal of Financial Economics 100:443–458. ↩ See Ang, A., Chen, B., Goetzmann, W. N., Phalippou, L., 2018. Estimating private equity returns from limited partner cash flows. The Journal of Finance 73, 1751–1783. ↩ See Alexander D. Beath, PhD and Chris Flynn, CFA, ASSET ALLOCATION, COST OF INVESTING AND PERFORMANCE OF EUROPEAN DB PENSION FUNDS: THE IMPACT OF REAL ESTATE, CEM Benchmarking Inc.. ↩ ↩2 See Geltner, David, 1991, Smoothing in Appraisal-Based Returns, Journal of Real Estate Finance and Economics, Vol.4, p.327-345, Geltner, David, 1993, Estimating Market Values from Appraised Values without Assuming an Efficient Market, Journal of Real Estate Research, Vol.8, p.325-345 and Fisher, J. D., Geltner, D. M., & Webb, R. B. (1994). Value indices of commercial real estate: A comparison of index construction methods. The Journal of Realv Estate Finance and Economics, 9 (2), 137–164. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 See Verdickt, Gertjan, Volatility Laundering: On the Feasibility of Wine Investment Funds (March 28, 2024). ↩ ↩2 In the context of hedge funds. ↩ ↩2 See Spencer J Couts, Andrei S Gonçalves, Andrea Rossi, Unsmoothing Returns of Illiquid Funds, The Review of Financial Studies, 2024. ↩ ↩2 See Antti Ilmanen, Investing in Interesting Times, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 21 - 35. ↩ See Cliff Asness, Volatility Laundering, Perspective. ↩ In the context of private equity, c.f. the podcast Private Equity and the Game of ”Volatility Laundering” from This Week in Intelligent Investing. ↩ Or volatility laundered. ↩ See Brooks, C. and H. Kat, The statistical properties of Hedge Fund index returns and their implications for investors, The Journal of Alternative Investments, Fall 2002, 5 (2) 26-44. ↩ ↩2 ↩3 See Geoff Loudon, John Okunev, Derek White, Hedge Fund Risk Factors and the Value at Risk of Fixed Income Trading Strategies, The Journal of Fixed Income, Fall 2006, 16 (2) 46-61 and Okunev, John and White, Derek, Hedge Fund Risk Factors and Value at Risk of Credit Trading Strategies (October 2003). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 See Shilling, James D. Measurement Error in FRC/NCREIF Returns on Real Estate. Southern Economic Journal, vol. 60, no. 1, 1993, pp. 210–19. ↩ See Georges Gallais-Hamonno, Huyen Nguyen-Thi-Thanh. The necessity to correct hedge fund returns: empirical evidence and correction method. 2007. ↩ ↩2 ↩3 For example, use the time series [-0.94, -0.96, -0.98, -0.95]. ↩ Called i.i.d.-based in de Prado25. ↩ See Lo, A. W., 2002. The statistics of sharpe ratios. Financial Analysts Journal 58, No. 4, 36–52. ↩ See Bailey, David H. and Lopez de Prado, Marcos, Stop-Outs Under Serial Correlation and ‘The Triple Penance Rule’. ↩ ↩2 See The Case Against Private Markets. ↩ ↩2 (Adjusted) prices have have been retrieved using Tiingo. ↩ Due to the first-order autocorrelation in BREIT returns, the naive annualization of the volatility and of the Sharpe Ratio that I have done is incorrect, c.f. Lo24; this is done on purpose, though. ↩ Okunev and White’s procedure gives nearly identical results. ↩ See Bingxu Chen & Christopher Carrano, Two Easy Steps to Take Your Private Asset Returns Public, Vnn by Two Sigma. ↩ See Clifford S Asness, Robert J Krail, John M Liew, Do Hedge Funds Hedge?, The Journal of Portfolio Management, Fall 2001, 28 (1) 6-19. ↩ The associated index is the Markit iBoxx USD Liquid High Yield Index; comparison of Private Debt to both Investment Grade and High Yield bonds is discussed in Boni and Manigart38. ↩ In-sample asset mean returns and in-sample asset returns covariance matrix, using asset returns over the period 31 March 2008 - 31 December 2023. ↩ Reported returns for the private investment indexes were unsmoothed using the Okunev and White’s procedure of order 1; results are similar with Geltner’s procedure, but since I already used Geltner’s procedure in the context of real-estate, I decided to play with its extension. ↩ To be noted that the in-sample asset mean returns are numerically different between smoothed and unsmoothed returns, but this does not explain the differences between the two frontiers. ↩ The transition map of the efficient frontier is a stacked area plot of the composition, in terms of asset weights, of the efficient portfolios belonging to the efficient frontier. ↩ That being said, a variant of Geltner’s method allow to use a public market equivalent of a private asset to “inject” volatility in the unsmoothed returns, c.f. Geltner10. ↩ See Boni, Pascal and Manigart, Sophie, Private Debt Fund Returns, Persistence, and Market Conditions. ↩Bootstrap Simulations with Exact Sample Mean Vector and Sample Covariance Matrix2024-05-05T00:00:00-05:002024-05-05T00:00:00-05:00https://portfoliooptimizer.io/blog/bootstrap-simulations-with-exact-sample-mean-vector-and-sample-covariance-matrix<p><a href="https://en.wikipedia.org/wiki/Bootstrapping_(statistics)">Bootstrapping</a> is a statistical method which consists in sampling with replacement from an original data set
to compute the distribution of a desired <a href="https://en.wikipedia.org/wiki/Statistic">statistic</a>, with <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">plenty of possible variations</a>
depending on the exact context (non-dependent data, dependent data…).</p>
<p>Because <em>bootstrap methods are not, in general, based on any particular assumption on the distribution of the data, they are well suited for the analysis of [financial] returns</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, in which case
the original data set typically consists of historical return time series<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>.</p>
<p>Unfortunately, bootstrap simulations <em>have the disadvantage that only previously recorded values can be simulated</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, so that they cannot <em>explicitly reflect the investors’ expectations</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In this blog post, inspired by <a href="https://www.linkedin.com/in/robert-stuart-carver/">Rob Carver</a>’s piece
<em><a href="https://qoppac.blogspot.com/2023/12/portfolio-optimisation-uncertainty.html">Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots</a></em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup>, I will describe a way to incorporate exact views
on expected returns, standard deviations and correlations into bootstrapping so that the resulting simulations both <em>preserve the salient characteristics of asset returns</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>
and also match forward-looking risk<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">5</a></sup> and return assumptions.</p>
<p>As an example of application, I will revisit the methodology described in the research note <em>Considering the Past and the Future in Asset Simulation</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> from <a href="https://www.troweprice.com/en/us">T. Rowe Price</a>
and incorporate the full range of capital market assumptions into <em>a systematic modelling process to simulate potential asset returns for portfolio construction</em><sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<p>Let be:</p>
<ul>
<li>$n$, the number of assets in a universe of assets</li>
<li>$T$, the number of time periods</li>
<li>$R \in \mathcal{M}(\mathbb{R}^{T \times n})$, with $R_{t,i}$ the return<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">6</a></sup> of the asset $i=1..n$ for the time period $t=1..T$, representing a scenario (historical, simulated…) for the temporal evolution of the asset returns</li>
<li>$\mu \in \mathbb{R}^{n}$, the vector of the arithmetic averages of the asset returns $R$</li>
<li>$\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the covariance matrix of the asset returns $R$, supposed to be invertible to avoid numerical subtleties</li>
</ul>
<h3 id="moment-matching">Moment-matching</h3>
<p>Let be:</p>
<ul>
<li>$\bar{\mu} \in \mathbb{R}^{n}$ a vector</li>
<li>$\bar{\Sigma} \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a <a href="https://en.wikipedia.org/wiki/Definite_matrix">positive-definite matrix</a></li>
</ul>
<p>Suppose that we would like the first two empirical moments of the asset returns $R$ - that is, $\mu$ and $\Sigma$ - to take the values $\bar{\mu}$ and $\bar{\Sigma}$.</p>
<p>How to proceed?</p>
<p>This problem - called <em>moment-matching by twisting scenarios</em> in <a href="https://www.linkedin.com/in/attilio-meucci-72580013/">Attilio Meucci</a>’s <a href="https://www.arpm.co/">Advanced Risk and Portfolio Management</a> website -
consists in transforming the original asset returns $R$ into modified asset returns $\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n})$, so that:</p>
<ul>
<li>$ \frac{1}{T} \sum_{t=1}^{T} \tilde{R}_{t,i} = \bar{\mu}_i $, $i=1..n$</li>
<li>$ \frac{1}{T} \sum_{k=1}^T \left( \tilde{R}_{k,i} - \bar{\mu}_i \right) \left( \tilde{R}_{k,j} - \bar{\mu}_j \right) = \bar{\Sigma}_{i,j} $, $i=1..n$, $j=1..n$</li>
</ul>
<h3 id="moment-matching-with-n1-shift-and-rescaling-procedure">Moment-matching with $n=1$, shift and rescaling procedure</h3>
<p>When $n=1$, a procedure to compute the modified asset returns $\tilde{R}$ from the original asset returns $R$ is discussed in Boyle et al.<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup>.</p>
<p>It involves the following <em>intuitive shift/rescaling</em><sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> of the original asset returns:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu} + \frac{\bar{\sigma}}{\sigma} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu \right)\]
<p>, with:</p>
<ul>
<li>$\sigma = \sqrt{\Sigma_{1,1}}$, the (sample) standard deviation of the original asset returns</li>
<li>$\bar{\sigma} = \sqrt{\bar{\Sigma}_{1,1}}$, the desired (sample) standard deviation of the modified asset returns</li>
</ul>
<h3 id="moment-matching-with-n-geq-2">Moment-matching with $n \geq 2$</h3>
<h4 id="multivariate-shift-and-rescaling-procedure">Multivariate shift and rescaling procedure</h4>
<p>When $n \geq 2$, Kaut and Lium<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> hints at a multivariate generalization of the procedure described in the previous sub-section, which becomes <a href="https://en.wikipedia.org/wiki/Affine_transformation">an affine transformation</a>
of the original asset returns $R$:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \bar{\Sigma}^{\frac{1}{2}} \Sigma^{-\frac{1}{2}} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\]
<p>, with:</p>
<ul>
<li>$\bar{\Sigma}^{\frac{1}{2}}$, <a href="https://en.wikipedia.org/wiki/Square_root_of_a_matrix#Positive_semidefinite_matrices">the unique positive semidefinite square root</a> of $\bar{\Sigma}$</li>
<li>$\Sigma^{-\frac{1}{2}}$, the inverse of the unique positive semidefinite square root of $\Sigma$</li>
</ul>
<p>To be noted that if only the average asset returns and/or the asset standard deviations need to be altered - that is, if the asset correlations should be left unchanged -, the generic formula above
becomes:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + Diag \left( \frac{\bar{\sigma}}{\sigma} \right) \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\]
<p>, with</p>
\[Diag \left( \frac{\bar{\sigma}}{\sigma} \right) = \begin{pmatrix}
\frac{\bar{\sigma}_1}{\sigma_1} & 0 & ... & 0 \\
0 & \frac{\bar{\sigma}_2}{\sigma_2} & ... & 0 \\
... & ... & ... & ... \\
0 & 0 & ... & \frac{\bar{\sigma}_n}{\sigma_n}
\end{pmatrix}\]
<p>, which is exactly the one-dimensional procedure described in the previous sub-section, applied independently to each of the $n$ assets.</p>
<h4 id="minimum-correction-second-moment-matching-procedure">Minimum-correction second-moment-matching procedure</h4>
<p>One problem with the multivariate shift and rescaling procedure described in the previous sub-section is that it offers no guarantees regarding the “distance” between the original asset returns and the modified asset returns.</p>
<p>In other words, it is theoretically possible that the modified asset returns obtained through this procedure bear no resemblance at all with the original asset returns,
which would render them useless for the purpose of <em>maintain[ing] important information from [the original asset returns]</em><sup id="fnref:2:5" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>,</p>
<p>To solve this problem, <em>a natural approach is to aim for a minimal correction $\tilde{R} - R$ to avoid introducing numerical artifacts</em><sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup>.</p>
<p>Such an approach to minimally distorting the original asset returns is described in Lin and Lermusiaux<sup id="fnref:7:1" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> under the name <em>minimum-correction second-moment-matching</em><sup id="fnref:7:2" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup>, with the following main result<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">11</a></sup>:</p>
\[\argmin_{\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n}), \frac{1}{n}\tilde{R}^t R = \bar{\Sigma}} \left\Vert R - \tilde{R} \right\Vert^2_F = R A_*\]
<p>, with $A_*$ defined by</p>
\[A_* = { \bar{\Sigma}^{\frac{1}{2}} }^t \left( { \bar{\Sigma}^{\frac{1}{2}} } \Sigma { \bar{\Sigma}^{\frac{1}{2}} }^t \right)^{-\frac{1}{2}} { \bar{\Sigma}^{\frac{1}{2}} }\]
<p>This leads to the mapping:</p>
\[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right) A_*\]
<h4 id="comparison-of-the-two-procedures">Comparison of the two procedures</h4>
<p>Figure 1 and Figure 2 highlight, over the period 04rd January 2010 - 31th December 2020<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup>, the typical differences between the two procedures described in the previous sub-sections when applied to a universe of 10 ETFs
representative<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">13</a></sup> of misc. asset classes.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-comparison.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-comparison.png" alt="Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 1. Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-tlt-comparison.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-tlt-comparison.png" alt="Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 2. Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>On Figure 1 and Figure 2:</p>
<ul>
<li>The curve in blue corresponds to the original asset returns</li>
<li>The two curves in green and orange correspond to modified asset returns so that the average returns and the standard deviations of the asset returns are left unchanged but their correlations are altered, using
<ul>
<li>The multivariate shift and rescaling procedure (green curve)</li>
<li>The minimum correction procedure (orange curve)</li>
</ul>
</li>
</ul>
<p>From these figures, it is clear that the curve corresponding to the minimum correction procedure is “closer” to the original curve than the curve corresponding to the multivariate shift and rescaling procedure.</p>
<p>This is confirmed numerically, with the “minimally corrected” asset returns being about 10% closer to the original asset returns than the “shifted and scaled” asset returns:</p>
<ul>
<li>$\left\Vert R - \tilde{R}_{shift-rescaling} \right\Vert_F \approx 1.30$</li>
<li>$\left\Vert R - \tilde{R}_{min-correction} \right\Vert_F \approx 1.17$</li>
</ul>
<h3 id="misc-remarks">Misc. remarks</h3>
<p>A couple of remarks on what precedes:</p>
<ul>
<li>
<p>The underlying distribution of the original asset returns - whatever its nature - is usually not preserved by the different moment-matching procedures.</p>
<p>This is discussed in Boyle et al.<sup id="fnref:5:1" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup> for the one-dimensional case and in Kaut and Lium<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> for the general multi-dimensional case.</p>
</li>
<li>
<p>The multivariate formulas for computing $\tilde{R}$ provided in the previous sub-sections are not the most generic ones.</p>
<p>In particular, Kaut and Lium<sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> and Lin and Lermusiaux<sup id="fnref:7:3" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> emphasize that the matrix square roots appearing in those formulas need not be the unique positive semidefinite square roots.</p>
</li>
</ul>
<h2 id="bootstrap-simulations-with-exact-sample-mean-vector-and-sample-covariance-matrix">Bootstrap simulations with exact sample mean vector and sample covariance matrix</h2>
<h3 id="bootstrap-simulations">Bootstrap simulations…</h3>
<p>Carver<sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup> gives a great summary of why bootstrap simulations are so useful in finance:</p>
<blockquote>
<p>Bootstrapping is particularly potent in the field of financial data because we only have one set of data: history. We can’t run experiments to get more data. Bootstrapping allows us to create ‘alternative histories’ that have the same basic character as our actual history, but aren’t quite the same. Apart from generating completely random data […], there isn’t really much else we can do.</p>
</blockquote>
<p>When applied to historical asset returns, bootstrapping creates simulated asset returns that differ from their historical counterparts but that nevertheless
<em>maintain important information gleaned from history</em><sup id="fnref:2:6" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>Now, for various reasons, it might be useful to be able to tweak the first two empirical moments of those simulated asset returns:</p>
<ul>
<li>
<p>To incorporate forward-looking estimates of mean asset returns based on current market valuation indicators like <a href="/blog/the-single-greatest-predictor-of-future-stock-market-returns-ten-years-after/">the U.S. AIAE indicator</a> for U.S. stocks or <a href="/blog/the-bogle-model-for-bonds-predicting-the-returns-of-constant-maturity-government-bond-etfs/">the BSRM/B model</a> for U.S. bonds.</p>
</li>
<li>
<p>To incorporate forward-looking estimates of asset correlations, like an estimate of the future stock-bond correlation based on the framework described in Czasonis et al.<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> or, more recently, in Molenaar et al.<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup></p>
</li>
<li>
<p>To simulate correlation scenarios incompatible with the historical asset returns, like unprecedented correlation breakdowns<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">16</a></sup>.</p>
</li>
</ul>
<h3 id="-with-exact-sample-mean-vector-and-sample-covariance-matrix">… with exact sample mean vector and sample covariance matrix</h3>
<p>For this, one possibility is to use the moment-matching procedures described in the previous section, and in particular the minimum-correction second-moment-matching procedure.</p>
<p>Indeed:</p>
<ul>
<li>
<p>Although these procedures do not preserve the distribution of the simulated asset returns, it is not a problem in the context of bootstrapping because - contrary to <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">a Monte Carlo simulation</a> - there is (usually) no assumption made about that distribution.</p>
</li>
<li>
<p>Additionally, the “minimum alteration” property of the minimum-correction second-moment-matching procedure gives some assurance that, even if the distribution of the simulated asset returns is altered by this procedure, it should be kind of “minimally” altered.</p>
</li>
</ul>
<p>In practice, a moment-matching procedure can be used in two different ways with bootstrap simulations:</p>
<ul>
<li>For tweaking the historical asset returns</li>
<li>For tweaking the simulated asset returns</li>
</ul>
<h4 id="tweaking-the-historical-asset-returns">Tweaking the historical asset returns</h4>
<p>This is the method described for example in Carver<sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup> or in Wu and Walsh<sup id="fnref:2:7" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>It consists in:</p>
<ul>
<li>
<p>Transforming the historical asset returns into modified historical asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix).</p>
</li>
<li>
<p>Bootstrapping these modified historical assets returns in order to simulate asset returns.</p>
</li>
</ul>
<p>Figure 3 illustrates this method applied to a universe of U.S. stocks (SPY ETF) and bonds (TLT ETF) over the period 04rd January 2010 - 31th December 2020<sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup> when the historical correlation of about
$-0.5$ between stocks and bonds is altered to about $0.5$, which is more representative of the pre-2000 period, c.f. Brixton et al.<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">17</a></sup>.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-history.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-history.png" alt="Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 3. Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>One of the advantages of this method is that the computationally costly part - the moment-matching procedure - needs to be performed only once.</p>
<p>Nevertheless, with this method, the simulated asset returns will exhibit a varying sample mean vector and sample covariance matrix, which might be undesirable.</p>
<h4 id="tweaking-the-simulated-asset-returns">Tweaking the simulated asset returns</h4>
<p>This method consists in:</p>
<ul>
<li>
<p>Bootstrapping the historical assets returns in order to simulate asset returns.</p>
</li>
<li>
<p>Transforming the simulated asset returns into modified simulated asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix).</p>
</li>
</ul>
<p>Figure 4 illustrates this method in the same context as in Figure 3.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-simulations.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-tlt-tweaking-simulations.png" alt="Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020." /></a>
<figcaption>Figure 4. Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020.</figcaption>
</figure>
<p>With this method, the simulated asset returns will exhibit a sample mean vector and a sample covariance matrix that exactly match the desired target mean vector and target covariance matrix.</p>
<p>Unfortunately, one of the drawbacks of this method is that it is computationally heavy, because every time a bootstrap simulation is generated, the moment-matching procedure must be applied to the associated simulated asset
returns.</p>
<h3 id="comparison-with-entropy-pooling">Comparison with entropy pooling</h3>
<p>Readers familiar with a procedure called <em>entropy pooling</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup> - that also allows to <em>express views on features of the market, such as expectation or correlations, regardless of the market distribution</em><sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">19</a></sup> - might wonder about
the relationship between that procedure and the moment-matched bootstrapping procedure described in this section.</p>
<p>Two preliminary remarks:</p>
<ul>
<li>
<p>Bootstrapping, especially its variations designed for time-series, introduces time-dependency in the sampled asset returns.</p>
<p>This makes moment-matched bootstrapping closer in spirit to time-dependent entropy pooling - as described in van der Schans<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup>, which applies the entropy pooling <em>computational approach in a time-dependent setting with sample paths, called scenarios, instead of sample points</em><sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup> - than to vanilla entropy pooling.</p>
</li>
<li>
<p>Time-dependent entropy pooling <em>incorporates the [views] by assigning weights to the scenarios</em><sup id="fnref:27:2" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup> within a scenario-probability setting, while moment-matched bootstrapping operates on one scenario at a time, without any reference to a probability distribution.</p>
</li>
</ul>
<p>With that in mind, the relationship between entropy pooling and moment-matched bootstrapping would be the following:</p>
<ul>
<li>Time-dependent entropy pooling works by <em>tweaking the relative probabilities of the scenarios without affecting the scenarios themselves</em><sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup></li>
<li>Moment-matched bootstrapping works by tweaking the scenarios without affecting the relative probabilities themselves, since in that case, <em>the user associates full probability to one single scenario</em><sup id="fnref:26:2" role="doc-noteref"><a href="#fn:26" class="footnote">18</a></sup></li>
</ul>
<p>That being said, entropy pooling is a much more generic<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">21</a></sup> framework for imposing views than moment-matched bootstrapping, so that the comparison ends here.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the minimum-correction second-moment-matching procedure described in the previous sections through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/moment-matched</code></a>.</p>
<p>Together with the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/returns/simulation/bootstrap</code></a>, this allows to use any of the two methods described in the previous section.</p>
<h2 id="example-of-application---extending-an-historically-informed-forward-looking-simulation-framework">Example of application - extending an historically informed, forward-looking simulation framework</h2>
<h3 id="t-rowe-prices-simulation-framework">T. Rowe Price’s simulation framework</h3>
<p>Wu and Walsh<sup id="fnref:2:8" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> describes a simulation-based portfolio construction framework that <em>embed both long-term historical asset behaviours and investor expectations of future performance when simulating asset returns</em><sup id="fnref:2:9" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In details, this <em>historical informed</em><sup id="fnref:2:10" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> framework is a three-step process that:</p>
<ul>
<li>
<p>Backfills missing historical asset returns for assets whose return histories differ in length, using the relationships observed between all assets over their common returns history while accounting for the associated estimation error.</p>
<p>As a side note, this step uses the procedure described in a <a href="/blog/managing-missing-asset-returns-in-portfolio-analysis-and-optimization-backfilling-through-residuals-recycling/">previous blog post</a>.</p>
</li>
<li>
<p>Tweaks the resulting historical asset returns, taking into account <em>user-defined return expectations</em><sup id="fnref:2:11" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>This step tweaks the historical asset returns thanks to the univariate shift procedure<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">22</a></sup> described in the previous sections, using T. Rowe Price’s own expected returns as input.</p>
<p>Figure 5, nearly identical<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">23</a></sup> to Figure 8 from Wu and Walsh<sup id="fnref:2:12" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, depicts the result of that procedure when applied to the SPY ETF over the period December 2010 - December 2021 with a target expected return of 4.9% per year<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">24</a></sup>.</p>
<figure>
<a href="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-troweprice-reproduction.png"><img src="/assets/images/blog/bootstrap-simulation-second-moment-matching-spy-troweprice-reproduction.png" alt="12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021." /></a>
<figcaption>Figure 5. 12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021.</figcaption>
</figure>
</li>
<li>
<p>Simulates asset returns from the modified historical asset returns, ensuring to <em>retain the actual pattern of […] asset movements in each simulated scenario</em><sup id="fnref:2:13" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>This step relies on what seems to be a rolling <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">block-bootstrap</a> in order <em>to recognise extreme tail risk occurrences in actual historical context</em><sup id="fnref:2:14" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
</li>
</ul>
<p>Ultimately, T. Rowe Price’s framework can be used <em>to demonstrate how [a multi-asset portfolio] would have performed in different market conditions reflective of historical experience,
while incorporating the investor’s expectations for the future</em><sup id="fnref:2:15" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<h3 id="incorporating-all-capital-market-assumptions-into-t-rowe-prices-simulation-framework">Incorporating all capital market assumptions into T. Rowe Price’s simulation framework</h3>
<p>I propose to extend T. Rowe Price’s framework to be able to take into account:</p>
<ul>
<li>User-defined return expectations</li>
<li>User-defined standard deviation expectations</li>
<li>User-defined correlation expectations</li>
</ul>
<p>Because these three quantities usually represent the full range of capital market assumptions published by financial institutions<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">25</a></sup>, the resulting simulation framework will become even more powerful while retaining
its original simplicity.</p>
<p>In line with this blog post, this is simply done by integrating the minimum-correction second-moment-matching procedure described in the previous sections into the second step of T. Rowe Price’s framework.</p>
<h2 id="conclusion">Conclusion</h2>
<p>A perfect summary of the technique described in this blog post is given by Carver<sup id="fnref:14:3" role="doc-noteref"><a href="#fn:14" class="footnote">4</a></sup>:</p>
<blockquote>
<p>[…] it basically allows us to use forward looking estimates for the first two moments (and first co-moment - correlation) of the distribution, whilst using actual data for the higher moments (skew, kurtosis and so on) and co-moments (co-skew, co-kurtosis etc). In a sense it’s sort of a blend of a parameterised monte-carlo and a non parameterised bootstrap.</p>
</blockquote>
<p>For more fun with bootstrap, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://onlinelibrary.wiley.com/doi/10.1111/1467-6419.00170">Esther Ruiz and Lorenzo Pascual. Bootstrapping Financial Time Series. Journal of Economic Surveys, 2002, vol. 16, issue 3, 271-300</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Like historical asset returns or historical factor returns. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.troweprice.com/financial-intermediary/uk/en/thinking/articles/2022/q4/considering-the-past-and-the-future-in-asset-simulation.html">Wu, Walsh, Considering the Past and the Future in Asset Simulation, T. Rowe Price, Investment Insights, November 2022</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:2:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:2:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:2:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:2:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:2:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:2:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:2:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:2:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:2:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a> <a href="#fnref:2:14" class="reversefootnote" role="doc-backlink">↩<sup>15</sup></a> <a href="#fnref:2:15" class="reversefootnote" role="doc-backlink">↩<sup>16</sup></a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://qoppac.blogspot.com/2023/12/portfolio-optimisation-uncertainty.html">Rob Carver, This Blog is Systematic, Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots. Ho, ho, ho</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:14:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>With risk defined in terms of volatility. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Arithmetic, logarithmic… <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://dl.acm.org/doi/10.1145/224401.224465">Boyle P, M Broadie and P Glasserman, 1995, Recent advances in simulation for security pricing, Proceedings of the 1995 Winter Simulation Conference, pages 212–219</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:5:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1415699">Meucci, Attilio, Simulations with Exact Means and Covariances (June 7, 2009)</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://eudml.org/doc/262201">Kaut, Michal, and Lium, Arnt-Gunnar. “Scenario generation with distribution functions and correlations.” Kybernetika 50.6 (2014): 1049-1064</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s00211-021-01178-8">Lin, J., Lermusiaux, P.F.J. Minimum-correction second-moment-matching: theory, algorithms and applications. Numer. Math. 147, 611–650 (2021)</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:7:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:7:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>To be noted that Theorem 2.1 from Lin and Lermusiaux<sup id="fnref:7:4" role="doc-noteref"><a href="#fn:7" class="footnote">10</a></sup> requires a couple of numerical assumptions - that $m \geq n$ and that $R$ has full column rank. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">26</a></sup>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/47/3/67">Megan Czasonis, Mark Kritzman, David Turkington, The Stock-Bond Correlation, The Journal of Portfolio Management, February 2021, 47 (3) 67-76</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4514947">Molenaar, Roderick and Senechal, Edouard and Swinkels, Laurens and Wang, Zhenping, Empirical evidence on the stock-bond correlation (February 9, 2023)</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>On this, see also the blog posts <a href="/blog/correlation-matrix-stress-testing-shrinkage-toward-an-equicorrelation-matrix/">here</a>, <a href="/blog/correlation-matrix-stress-testing-shrinkage-toward-the-lower-and-upper-bounds-of-a-correlation-matrix/">here</a> and <a href="/blog/correlation-matrix-stress-testing-random-perturbations-of-a-correlation-matrix/">here</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/49/4/64">A Changing Stock–Bond Correlation: Drivers and Implications, Alfie Brixton, Jordan Brooks, Pete Hecht, Antti Ilmanen, Thomas Maloney, Nicholas McQuinn, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 64 - 80</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1213325">Meucci, Attilio, Fully Flexible Views: Theory and Practice (August 8, 2008). Fully Flexible Views: Theory and Practice, Risk, Vol. 21, No. 10, pp. 97-102, October 2008</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:26:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2583612">Meucci, Attilio and Nicolosi, Marco, Dynamic Portfolio Management with Views at Multiple Horizons (April 16, 2015). Applied Mathematics and Computation, Volume 274, 1 February 2016, Pages 495-518</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s10614-018-9824-7">van der Schans, M. Entropy Pooling with Discrete Weights in a Time-Dependent Setting. Comput Econ 53, 1633–1647 (2019)</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:27:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>Interestingly though, with (time-dependent) entropy pooling, <em>when scenarios of two variables always move along with each other, we cannot impose a negative correlation</em><sup id="fnref:27:3" role="doc-noteref"><a href="#fn:27" class="footnote">20</a></sup>, while this situation is not an issue for moment-matched bootstrapping! <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>There is no alteration of asset standard deviations or of correlations described in Wu and Walsh<sup id="fnref:2:16" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, so that the multivariate shift and rescaling procedure actually becomes a simple univariate shift procedure… <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>Returns before shifting in Figure 5 seem slightly higher than their counterparts in Figure 8 from Wu and Walsh<sup id="fnref:2:17" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>; this might be due to different data sources used. As a side note, Figure 8 from Wu and Walsh<sup id="fnref:2:18" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> has inverted legends for the two thick lines. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>Which corresponds to the 2022 T. Rowe Price five-year capital market assumptions / expected returns for U.S. Large Cap<sup id="fnref:25:1" role="doc-noteref"><a href="#fn:25" class="footnote">25</a></sup>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://www.troweprice.com/financial-intermediary/us/en/insights/articles/2022/q1/capital-market-assumptions-five-year-perspective.asset-allocation.html">T. Rowe Price, Capital Market Assumptions Five-Year Perspective</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:25:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.Bootstrapping is a statistical method which consists in sampling with replacement from an original data set to compute the distribution of a desired statistic, with plenty of possible variations depending on the exact context (non-dependent data, dependent data…). Because bootstrap methods are not, in general, based on any particular assumption on the distribution of the data, they are well suited for the analysis of [financial] returns1, in which case the original data set typically consists of historical return time series2. Unfortunately, bootstrap simulations have the disadvantage that only previously recorded values can be simulated3, so that they cannot explicitly reflect the investors’ expectations3. In this blog post, inspired by Rob Carver’s piece Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots4, I will describe a way to incorporate exact views on expected returns, standard deviations and correlations into bootstrapping so that the resulting simulations both preserve the salient characteristics of asset returns3 and also match forward-looking risk5 and return assumptions. As an example of application, I will revisit the methodology described in the research note Considering the Past and the Future in Asset Simulation3 from T. Rowe Price and incorporate the full range of capital market assumptions into a systematic modelling process to simulate potential asset returns for portfolio construction3. Mathematical preliminaries Let be: $n$, the number of assets in a universe of assets $T$, the number of time periods $R \in \mathcal{M}(\mathbb{R}^{T \times n})$, with $R_{t,i}$ the return6 of the asset $i=1..n$ for the time period $t=1..T$, representing a scenario (historical, simulated…) for the temporal evolution of the asset returns $\mu \in \mathbb{R}^{n}$, the vector of the arithmetic averages of the asset returns $R$ $\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the covariance matrix of the asset returns $R$, supposed to be invertible to avoid numerical subtleties Moment-matching Let be: $\bar{\mu} \in \mathbb{R}^{n}$ a vector $\bar{\Sigma} \in \mathcal{M} \left( \mathbb{R}^{n \times n} \right)$ a positive-definite matrix Suppose that we would like the first two empirical moments of the asset returns $R$ - that is, $\mu$ and $\Sigma$ - to take the values $\bar{\mu}$ and $\bar{\Sigma}$. How to proceed? This problem - called moment-matching by twisting scenarios in Attilio Meucci’s Advanced Risk and Portfolio Management website - consists in transforming the original asset returns $R$ into modified asset returns $\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n})$, so that: $ \frac{1}{T} \sum_{t=1}^{T} \tilde{R}_{t,i} = \bar{\mu}_i $, $i=1..n$ $ \frac{1}{T} \sum_{k=1}^T \left( \tilde{R}_{k,i} - \bar{\mu}_i \right) \left( \tilde{R}_{k,j} - \bar{\mu}_j \right) = \bar{\Sigma}_{i,j} $, $i=1..n$, $j=1..n$ Moment-matching with $n=1$, shift and rescaling procedure When $n=1$, a procedure to compute the modified asset returns $\tilde{R}$ from the original asset returns $R$ is discussed in Boyle et al.7. It involves the following intuitive shift/rescaling8 of the original asset returns: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu} + \frac{\bar{\sigma}}{\sigma} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu \right)\] , with: $\sigma = \sqrt{\Sigma_{1,1}}$, the (sample) standard deviation of the original asset returns $\bar{\sigma} = \sqrt{\bar{\Sigma}_{1,1}}$, the desired (sample) standard deviation of the modified asset returns Moment-matching with $n \geq 2$ Multivariate shift and rescaling procedure When $n \geq 2$, Kaut and Lium9 hints at a multivariate generalization of the procedure described in the previous sub-section, which becomes an affine transformation of the original asset returns $R$: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \bar{\Sigma}^{\frac{1}{2}} \Sigma^{-\frac{1}{2}} \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\] , with: $\bar{\Sigma}^{\frac{1}{2}}$, the unique positive semidefinite square root of $\bar{\Sigma}$ $\Sigma^{-\frac{1}{2}}$, the inverse of the unique positive semidefinite square root of $\Sigma$ To be noted that if only the average asset returns and/or the asset standard deviations need to be altered - that is, if the asset correlations should be left unchanged -, the generic formula above becomes: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + Diag \left( \frac{\bar{\sigma}}{\sigma} \right) \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right)\] , with \[Diag \left( \frac{\bar{\sigma}}{\sigma} \right) = \begin{pmatrix} \frac{\bar{\sigma}_1}{\sigma_1} & 0 & ... & 0 \\ 0 & \frac{\bar{\sigma}_2}{\sigma_2} & ... & 0 \\ ... & ... & ... & ... \\ 0 & 0 & ... & \frac{\bar{\sigma}_n}{\sigma_n} \end{pmatrix}\] , which is exactly the one-dimensional procedure described in the previous sub-section, applied independently to each of the $n$ assets. Minimum-correction second-moment-matching procedure One problem with the multivariate shift and rescaling procedure described in the previous sub-section is that it offers no guarantees regarding the “distance” between the original asset returns and the modified asset returns. In other words, it is theoretically possible that the modified asset returns obtained through this procedure bear no resemblance at all with the original asset returns, which would render them useless for the purpose of maintain[ing] important information from [the original asset returns]3, To solve this problem, a natural approach is to aim for a minimal correction $\tilde{R} - R$ to avoid introducing numerical artifacts10. Such an approach to minimally distorting the original asset returns is described in Lin and Lermusiaux10 under the name minimum-correction second-moment-matching10, with the following main result11: \[\argmin_{\tilde{R} \in \mathcal{M}(\mathbb{R}^{T \times n}), \frac{1}{n}\tilde{R}^t R = \bar{\Sigma}} \left\Vert R - \tilde{R} \right\Vert^2_F = R A_*\] , with $A_*$ defined by \[A_* = { \bar{\Sigma}^{\frac{1}{2}} }^t \left( { \bar{\Sigma}^{\frac{1}{2}} } \Sigma { \bar{\Sigma}^{\frac{1}{2}} }^t \right)^{-\frac{1}{2}} { \bar{\Sigma}^{\frac{1}{2}} }\] This leads to the mapping: \[\tilde{R} = \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \bar{\mu}^t + \left( R - \begin{pmatrix} 1 \\ 1 \\ ... \\ 1 \end{pmatrix} \mu^t \right) A_*\] Comparison of the two procedures Figure 1 and Figure 2 highlight, over the period 04rd January 2010 - 31th December 202012, the typical differences between the two procedures described in the previous sub-sections when applied to a universe of 10 ETFs representative13 of misc. asset classes. Figure 1. Comparison of moment matching procedures in a 10 ETF universe, SPY ETF, 04rd January 2010 - 31th December 2020. Figure 2. Comparison of moment matching procedures in 10 ETF universe, TLT ETF, 04rd January 2010 - 31th December 2020. On Figure 1 and Figure 2: The curve in blue corresponds to the original asset returns The two curves in green and orange correspond to modified asset returns so that the average returns and the standard deviations of the asset returns are left unchanged but their correlations are altered, using The multivariate shift and rescaling procedure (green curve) The minimum correction procedure (orange curve) From these figures, it is clear that the curve corresponding to the minimum correction procedure is “closer” to the original curve than the curve corresponding to the multivariate shift and rescaling procedure. This is confirmed numerically, with the “minimally corrected” asset returns being about 10% closer to the original asset returns than the “shifted and scaled” asset returns: $\left\Vert R - \tilde{R}_{shift-rescaling} \right\Vert_F \approx 1.30$ $\left\Vert R - \tilde{R}_{min-correction} \right\Vert_F \approx 1.17$ Misc. remarks A couple of remarks on what precedes: The underlying distribution of the original asset returns - whatever its nature - is usually not preserved by the different moment-matching procedures. This is discussed in Boyle et al.7 for the one-dimensional case and in Kaut and Lium9 for the general multi-dimensional case. The multivariate formulas for computing $\tilde{R}$ provided in the previous sub-sections are not the most generic ones. In particular, Kaut and Lium9 and Lin and Lermusiaux10 emphasize that the matrix square roots appearing in those formulas need not be the unique positive semidefinite square roots. Bootstrap simulations with exact sample mean vector and sample covariance matrix Bootstrap simulations… Carver4 gives a great summary of why bootstrap simulations are so useful in finance: Bootstrapping is particularly potent in the field of financial data because we only have one set of data: history. We can’t run experiments to get more data. Bootstrapping allows us to create ‘alternative histories’ that have the same basic character as our actual history, but aren’t quite the same. Apart from generating completely random data […], there isn’t really much else we can do. When applied to historical asset returns, bootstrapping creates simulated asset returns that differ from their historical counterparts but that nevertheless maintain important information gleaned from history3. Now, for various reasons, it might be useful to be able to tweak the first two empirical moments of those simulated asset returns: To incorporate forward-looking estimates of mean asset returns based on current market valuation indicators like the U.S. AIAE indicator for U.S. stocks or the BSRM/B model for U.S. bonds. To incorporate forward-looking estimates of asset correlations, like an estimate of the future stock-bond correlation based on the framework described in Czasonis et al.14 or, more recently, in Molenaar et al.15 To simulate correlation scenarios incompatible with the historical asset returns, like unprecedented correlation breakdowns16. … with exact sample mean vector and sample covariance matrix For this, one possibility is to use the moment-matching procedures described in the previous section, and in particular the minimum-correction second-moment-matching procedure. Indeed: Although these procedures do not preserve the distribution of the simulated asset returns, it is not a problem in the context of bootstrapping because - contrary to a Monte Carlo simulation - there is (usually) no assumption made about that distribution. Additionally, the “minimum alteration” property of the minimum-correction second-moment-matching procedure gives some assurance that, even if the distribution of the simulated asset returns is altered by this procedure, it should be kind of “minimally” altered. In practice, a moment-matching procedure can be used in two different ways with bootstrap simulations: For tweaking the historical asset returns For tweaking the simulated asset returns Tweaking the historical asset returns This is the method described for example in Carver4 or in Wu and Walsh3. It consists in: Transforming the historical asset returns into modified historical asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix). Bootstrapping these modified historical assets returns in order to simulate asset returns. Figure 3 illustrates this method applied to a universe of U.S. stocks (SPY ETF) and bonds (TLT ETF) over the period 04rd January 2010 - 31th December 202012 when the historical correlation of about $-0.5$ between stocks and bonds is altered to about $0.5$, which is more representative of the pre-2000 period, c.f. Brixton et al.17. Figure 3. Simulated SPY/TLT ETF returns obtained from bootstrapping moment-matched historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020. One of the advantages of this method is that the computationally costly part - the moment-matching procedure - needs to be performed only once. Nevertheless, with this method, the simulated asset returns will exhibit a varying sample mean vector and sample covariance matrix, which might be undesirable. Tweaking the simulated asset returns This method consists in: Bootstrapping the historical assets returns in order to simulate asset returns. Transforming the simulated asset returns into modified simulated asset returns whose sample mean vector (resp. sample covariance matrix) is equal to a desired target mean vector (resp. target covariance matrix). Figure 4 illustrates this method in the same context as in Figure 3. Figure 4. Moment-matched simulated SPY/TLT ETF returns obtained from bootstrapping historical SPY/TLT ETF returns, 04rd January 2010 - 31th December 2020. With this method, the simulated asset returns will exhibit a sample mean vector and a sample covariance matrix that exactly match the desired target mean vector and target covariance matrix. Unfortunately, one of the drawbacks of this method is that it is computationally heavy, because every time a bootstrap simulation is generated, the moment-matching procedure must be applied to the associated simulated asset returns. Comparison with entropy pooling Readers familiar with a procedure called entropy pooling18 - that also allows to express views on features of the market, such as expectation or correlations, regardless of the market distribution19 - might wonder about the relationship between that procedure and the moment-matched bootstrapping procedure described in this section. Two preliminary remarks: Bootstrapping, especially its variations designed for time-series, introduces time-dependency in the sampled asset returns. This makes moment-matched bootstrapping closer in spirit to time-dependent entropy pooling - as described in van der Schans20, which applies the entropy pooling computational approach in a time-dependent setting with sample paths, called scenarios, instead of sample points20 - than to vanilla entropy pooling. Time-dependent entropy pooling incorporates the [views] by assigning weights to the scenarios20 within a scenario-probability setting, while moment-matched bootstrapping operates on one scenario at a time, without any reference to a probability distribution. With that in mind, the relationship between entropy pooling and moment-matched bootstrapping would be the following: Time-dependent entropy pooling works by tweaking the relative probabilities of the scenarios without affecting the scenarios themselves18 Moment-matched bootstrapping works by tweaking the scenarios without affecting the relative probabilities themselves, since in that case, the user associates full probability to one single scenario18 That being said, entropy pooling is a much more generic21 framework for imposing views than moment-matched bootstrapping, so that the comparison ends here. Implementation in Portfolio Optimizer Portfolio Optimizer implements the minimum-correction second-moment-matching procedure described in the previous sections through the endpoint /assets/returns/moment-matched. Together with the endpoint /assets/returns/simulation/bootstrap, this allows to use any of the two methods described in the previous section. Example of application - extending an historically informed, forward-looking simulation framework T. Rowe Price’s simulation framework Wu and Walsh3 describes a simulation-based portfolio construction framework that embed both long-term historical asset behaviours and investor expectations of future performance when simulating asset returns3. In details, this historical informed3 framework is a three-step process that: Backfills missing historical asset returns for assets whose return histories differ in length, using the relationships observed between all assets over their common returns history while accounting for the associated estimation error. As a side note, this step uses the procedure described in a previous blog post. Tweaks the resulting historical asset returns, taking into account user-defined return expectations3. This step tweaks the historical asset returns thanks to the univariate shift procedure22 described in the previous sections, using T. Rowe Price’s own expected returns as input. Figure 5, nearly identical23 to Figure 8 from Wu and Walsh3, depicts the result of that procedure when applied to the SPY ETF over the period December 2010 - December 2021 with a target expected return of 4.9% per year24. Figure 5. 12-month rolling SPY ETF returns before and after sample mean moment-matching, December 2010 - December 2021. Simulates asset returns from the modified historical asset returns, ensuring to retain the actual pattern of […] asset movements in each simulated scenario3. This step relies on what seems to be a rolling block-bootstrap in order to recognise extreme tail risk occurrences in actual historical context3. Ultimately, T. Rowe Price’s framework can be used to demonstrate how [a multi-asset portfolio] would have performed in different market conditions reflective of historical experience, while incorporating the investor’s expectations for the future3. Incorporating all capital market assumptions into T. Rowe Price’s simulation framework I propose to extend T. Rowe Price’s framework to be able to take into account: User-defined return expectations User-defined standard deviation expectations User-defined correlation expectations Because these three quantities usually represent the full range of capital market assumptions published by financial institutions25, the resulting simulation framework will become even more powerful while retaining its original simplicity. In line with this blog post, this is simply done by integrating the minimum-correction second-moment-matching procedure described in the previous sections into the second step of T. Rowe Price’s framework. Conclusion A perfect summary of the technique described in this blog post is given by Carver4: […] it basically allows us to use forward looking estimates for the first two moments (and first co-moment - correlation) of the distribution, whilst using actual data for the higher moments (skew, kurtosis and so on) and co-moments (co-skew, co-kurtosis etc). In a sense it’s sort of a blend of a parameterised monte-carlo and a non parameterised bootstrap. For more fun with bootstrap, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Esther Ruiz and Lorenzo Pascual. Bootstrapping Financial Time Series. Journal of Economic Surveys, 2002, vol. 16, issue 3, 271-300. ↩ Like historical asset returns or historical factor returns. ↩ See Wu, Walsh, Considering the Past and the Future in Asset Simulation, T. Rowe Price, Investment Insights, November 2022. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 See Rob Carver, This Blog is Systematic, Portfolio optimisation, uncertainty, bootstrapping, and some pretty plots. Ho, ho, ho. ↩ ↩2 ↩3 ↩4 With risk defined in terms of volatility. ↩ Arithmetic, logarithmic… ↩ See Boyle P, M Broadie and P Glasserman, 1995, Recent advances in simulation for security pricing, Proceedings of the 1995 Winter Simulation Conference, pages 212–219. ↩ ↩2 See Meucci, Attilio, Simulations with Exact Means and Covariances (June 7, 2009). ↩ See Kaut, Michal, and Lium, Arnt-Gunnar. “Scenario generation with distribution functions and correlations.” Kybernetika 50.6 (2014): 1049-1064. ↩ ↩2 ↩3 See Lin, J., Lermusiaux, P.F.J. Minimum-correction second-moment-matching: theory, algorithms and applications. Numer. Math. 147, 611–650 (2021). ↩ ↩2 ↩3 ↩4 To be noted that Theorem 2.1 from Lin and Lermusiaux10 requires a couple of numerical assumptions - that $m \geq n$ and that $R$ has full column rank. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ ↩2 These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer26. ↩ See Megan Czasonis, Mark Kritzman, David Turkington, The Stock-Bond Correlation, The Journal of Portfolio Management, February 2021, 47 (3) 67-76. ↩ See Molenaar, Roderick and Senechal, Edouard and Swinkels, Laurens and Wang, Zhenping, Empirical evidence on the stock-bond correlation (February 9, 2023). ↩ On this, see also the blog posts here, here and here. ↩ See A Changing Stock–Bond Correlation: Drivers and Implications, Alfie Brixton, Jordan Brooks, Pete Hecht, Antti Ilmanen, Thomas Maloney, Nicholas McQuinn, The Journal of Portfolio Management, Multi-Asset Special Issue 2023, 49 (4) 64 - 80. ↩ See Meucci, Attilio, Fully Flexible Views: Theory and Practice (August 8, 2008). Fully Flexible Views: Theory and Practice, Risk, Vol. 21, No. 10, pp. 97-102, October 2008. ↩ ↩2 ↩3 See Meucci, Attilio and Nicolosi, Marco, Dynamic Portfolio Management with Views at Multiple Horizons (April 16, 2015). Applied Mathematics and Computation, Volume 274, 1 February 2016, Pages 495-518. ↩ See van der Schans, M. Entropy Pooling with Discrete Weights in a Time-Dependent Setting. Comput Econ 53, 1633–1647 (2019). ↩ ↩2 ↩3 Interestingly though, with (time-dependent) entropy pooling, when scenarios of two variables always move along with each other, we cannot impose a negative correlation20, while this situation is not an issue for moment-matched bootstrapping! ↩ There is no alteration of asset standard deviations or of correlations described in Wu and Walsh3, so that the multivariate shift and rescaling procedure actually becomes a simple univariate shift procedure… ↩ Returns before shifting in Figure 5 seem slightly higher than their counterparts in Figure 8 from Wu and Walsh3; this might be due to different data sources used. As a side note, Figure 8 from Wu and Walsh3 has inverted legends for the two thick lines. ↩ Which corresponds to the 2022 T. Rowe Price five-year capital market assumptions / expected returns for U.S. Large Cap25. ↩ See T. Rowe Price, Capital Market Assumptions Five-Year Perspective. ↩ ↩2 See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Cluster Risk Parity: Equalizing Risk Contributions Between and Within Asset Classes2024-04-24T00:00:00-05:002024-04-24T00:00:00-05:00https://portfoliooptimizer.io/blog/cluster-risk-parity-equalizing-risk-contributions-between-and-within-asset-classes<p>The <em>equal risk contribution (ERC)</em> portfolio, introduced in Maillard et al.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, is a portfolio aiming to <em>equalize the risk contributions from [its] different components</em><sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>Empirically, the ERC portfolio has been found to <em>be a middle-ground alternative</em><sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> to an equally weighted portfolio and a minimum variance portfolio, <em>balanced in risk and in weights</em><sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, which exhibits
interesting performances in terms of risk-adjusted returns<sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>One issue with the ERC portfolio, though, it that it is <em>highly dependent upon the structure of the universe of assets considered</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> and tends to distribute risk unevenly in terms of asset classes whenever
the number of assets per asset class differs.</p>
<p>In this short blog post, I will describe an extension of the ERC portfolio proposed by Varadi<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> and Kapler<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> designed to solve this problem.</p>
<h2 id="the-equal-risk-contribution-portfolio">The equal risk contribution portfolio</h2>
<p>Let be:</p>
<ul>
<li>$n$, the number of assets in a universe of assets</li>
<li>$w = (w_1,…,w_n) \in {\mathbb{R}^+}^{n}$, the asset weights of a long-only portfolio invested in the considered universe of assets</li>
<li>$\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the asset covariance matrix</li>
<li>$\sigma_{i,j}$, the covariance between assets $i$ and $j$, $i=1..n$, $j=1..n$</li>
<li>$\sigma_1^2,…,\sigma_n^2 \in {\mathbb{R}^+}^{n}$, the asset variances</li>
<li>$\sigma(w) = \sqrt{ w^t \Sigma w }$, the portfolio standard deviation/volatility</li>
</ul>
<h3 id="definitions">Definitions</h3>
<h4 id="risk-contributions">Risk contributions</h4>
<p>The <em>marginal contribution to risk</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup> - or <em>marginal risk contribution</em> - $MCTR_i(w)$ of the asset $i, i=1..n$, to the portfolio risk $\sigma(w)$ is defined as:</p>
\[MCTR_i(w) = \frac{\partial \sigma(w)}{\partial w_i} = \frac{w_i \sigma_{i}^2 + \Sigma_{i=1, i\ne j}^n w_j \sigma_{i,j}}{\sigma(w)}\]
<p>As noted in Maillard et al.<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, <em>the adjective marginal qualifies the fact that those quantities give the change in volatility of the portfolio induced by a small increase in the weight of one component</em><sup id="fnref:1:6" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>,
all other weights staying the same.</p>
<p>The <em>total contribution to risk</em> - or <em>total risk contribution</em> - $TCTR_i(w)$ of the $i$-th asset, $i=1..n$, to the portfolio risk $\sigma(w)$ is defined as:</p>
\[TCTR_i(w) = w_i MCTR_i(w)\]
<h4 id="the-erc-portfolio">The ERC portfolio</h4>
<p>The ERC portfolio is defined as <em>the risk-balanced portfolio such that the total risk contribution is the same for all assets in the portfolio</em><sup id="fnref:1:7" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>More formally, the ERC portfolio is the long-only portfolio whose asset weights $w^* \in [0,1]^{n}$ satisfy:</p>
<ul>
<li>$\sum_{i=1}^{n} w_i = 1$</li>
<li>$TCTR_i(w) = TCTR_j(w)$, for all $i$, $j$, $i=1..n$, $j=1..n$</li>
</ul>
<h3 id="sensitivity-of-the-erc-portfolio-to-the-universe-of-assets">Sensitivity of the ERC portfolio to the universe of assets</h3>
<p>Choueifaty et al.<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup> defines <em>a set of basic properties that an unbiased, agnostic portfolio construction process should respect, based on common sense and reasonable economic grounds</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>I propose to illustrate one of these properties, named <em>duplication invariance</em>, by comparing two portfolios invested in U.S. stocks (SPY ETF) and U.S. bonds (TLT ETF):</p>
<ul>
<li>The minimum variance (MV) portfolio</li>
<li>The ERC portfolio</li>
</ul>
<p>Using asset data over the period 03rd January 2023 - 29th December 2023<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, the associated portfolios are displayed in Figure 1:</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-vs-mv.png"><img src="/assets/images/blog/cluster-risk-parity-erc-vs-mv.png" alt="ERC v.s. MV portfolio weights, SPY-TLT universe." /></a>
<figcaption>Figure 1. ERC v.s. MV portfolio weights, SPY-TLT universe.</figcaption>
</figure>
<p>Something interesting to note in Figure 1 is that the ERC portfolio appears closer to an equally weighted portfolio than the MV portfolio, which is a general property discussed in Maillard et al.<sup id="fnref:1:8" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<p>Now, let’s suppose that the TLT ETF is somehow duplicated into a TLT’ ETF over the same period of time.</p>
<p>What would be the impact of this asset duplication on the above portfolios?</p>
<p>Answer is provided in Figure 2.</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-vs-mv-duplicated.png"><img src="/assets/images/blog/cluster-risk-parity-erc-vs-mv-duplicated.png" alt="Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe." /></a>
<figcaption>Figure 2. Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe.</figcaption>
</figure>
<p>It is visible on Figure 2 that:</p>
<ul>
<li>
<p>The weights of the ERC portfolio in the original assets are (severely) impacted by the duplication of TLT ETF.</p>
<p>The ERC portfolio is said be <em>non duplication invariant</em>.</p>
</li>
<li>
<p>The weights of the MV portfolio in the original assets are left unchanged<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">7</a></sup> by the duplication of the TLT ETF.</p>
<p>The MV portfolio is said to be <em>duplication invariant</em>.</p>
</li>
</ul>
<p>This lack of duplication invariance means that ERC portfolios are generally <em>biased toward assets with multiple representations</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">2</a></sup>, so that <em>the asset universe is an important factor when considering [ERC] portfolios</em><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">8</a></sup>.</p>
<p>As an example, for a portfolio invested in bonds and equities, <em>the number […] of bond versus equity will dramatically affect the risk contribution of the [ERC] portfolio from each category</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>Indeed, as detailed in Roncalli and Weisang<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">8</a></sup>:</p>
<blockquote>
<p>If the universe includes 5 equity indices and 5 bond indices, then the ERC portfolio will be well balanced between equity and bond in terms of risk. Conversely, if the universe includes 7 equity indices and 3 bond indices, the equity risk of the ERC portfolio represents then 70% of the portfolio’s total risk, a solution very unbalanced between equity and bond risks.</p>
</blockquote>
<p>Figure 3 illustrates this imbalance within the SPY-TLT universe, with the TLT duplicated 5 times (resp. 3 times) and the SPY ETF duplicated 5 times (resp. 7 times).</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-tctr-spy-tlt-duplicated.png"><img src="/assets/images/blog/cluster-risk-parity-tctr-spy-tlt-duplicated.png" alt="Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe." /></a>
<figcaption>Figure 3. Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe.</figcaption>
</figure>
<p>Interestingly, such a behavior is not in violation of the mathematical objective of the ERC portfolio<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">9</a></sup>, but is nevertheless in clear violation of the ERC portfolio <em>philosophy based on diversification</em><sup id="fnref:1:9" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>Asset weights in Figure 1 and Figure 2 might seem surprising at first sight (higher weights for the SPY ETF v.s. the TLT ETF), but 2023 has been a particular year for U.S. Treasuries… For reference, the exact (co)variances of the SPY and TLT ETFs are provided in Figure 5.</li>
<li>For more discussion about duplication invariance, see for example Gava and Turc<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">10</a></sup>, which introduces a quantitative measure of the sensitivity of a portfolio to duplication.</li>
</ul>
</blockquote>
<h2 id="the-cluster-risk-parity-portfolio">The cluster risk parity portfolio</h2>
<h3 id="description">Description</h3>
<p>One possible solution to the problem of the sensitivity of the ERC portfolio discussed in the previous section is a careful selection of the universe of assets.</p>
<p>Another possible solution is to slightly alter the ERC portfolio construction process by taking into account both the asset and the asset class level, which is the solution proposed by Varadi<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> and Kapler<sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>
under the name <em>cluster risk parity (CRP)</em> portfolio.</p>
<p>In details, the CRP portfolio is computed through a three-step process, detailed below.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>Another cluster risk parity portfolio is described in the <a href="https://quantpedia.com/">Quantpedia</a> series <em><a href="https://quantpedia.com/introduction-to-clustering-methods-in-portfolio-management-part-3/">Introduction to Clustering Methods In Portfolio Management</a></em>, but the two portfolio construction methodologies are not the same.</li>
</ul>
</blockquote>
<h4 id="step-1---partitioning-the-universe-of-assets-into-clusters">Step 1 - Partitioning the universe of assets into clusters</h4>
<p>The first step consists in using a <a href="https://en.wikipedia.org/wiki/Cluster_analysis">clustering algorithm</a> to group similar assets together within the considered universe of assets.</p>
<p>This results in $k$ clusters<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">11</a></sup> $\mathcal{G}_1,…,\mathcal{G}_k$, $1 \leq k \leq n$, each corresponding to an asset class automatically determined by the clustering algorithm.</p>
<p>The purpose of this step is <em>to avoid the need for artificial or manual grouping</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> of the assets and enables <em>the [CRP] portfolio to adapt to changes […] without having to run a lot of ad hoc analysis
and having to make continual adjustments</em><sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>Using the same <a href="https://en.wikipedia.org/wiki/Hierarchical_clustering">hierarchical clustering algorithm</a> as the one used in the
<em><a href="/blog/beyond-hierarchical-risk-parity-hierarchical-clustering-based-risk-parity">Hierarchical Clustering-Based Risk Parity</a></em> algorithm and using asset data over the period 03rd January 2023 - 29th December 2023<sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>,
Figure 4 illustrates this first step for a universe of 10 ETFs representative<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">12</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-hierarchical-clustering.png"><img src="/assets/images/blog/cluster-risk-parity-hierarchical-clustering.png" alt="Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe." /></a>
<figcaption>Figure 4. Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe.</figcaption>
</figure>
<p>Figure 4 shows that 3 clusters/asset classes have been automatically determined by the hierarchical clustering algorithm over the studied period, materialized by the thick black dendrogram cut:</p>
<ul>
<li>An equity cluster $\mathcal{G}_1$ made of the equity ETFs SPY, EZU, EWJ, EEM and of the real estate ETFs VNQ, RWX</li>
<li>A bond cluster $\mathcal{G}_2$ made of the bond ETFs IEF, TLT</li>
<li>A commodity cluster $\mathcal{G}_3$ made of the commodity ETF DBC and of the gold ETF GLD</li>
</ul>
<h4 id="step-2---assets-weights-computation-within-each-cluster">Step 2 - Assets weights computation within each cluster</h4>
<p>The second step consists in computing the ERC portfolio within each cluster $\mathcal{G}_i$, $i=1..k$.</p>
<p>This results in $k$ vectors of asset weights $w_{i}^* \in [0,1]^{n}$, $i=1..k$.</p>
<p>Using as input the covariance matrix $\Sigma$ provided in Figure 5<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">13</a></sup>, the ERC portfolio computed within each cluster $\mathcal{G}_1$, $\mathcal{G}_2$ and $\mathcal{G}_3$ defined
in the previous sub-section is displayed in Figure 6.</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-asset-covariance-matrix.png"><img src="/assets/images/blog/cluster-risk-parity-asset-covariance-matrix.png" alt="Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe." /></a>
<figcaption>Figure 5. Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-erc-portfolio-within-clusters.png"><img src="/assets/images/blog/cluster-risk-parity-erc-portfolio-within-clusters.png" alt="Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe." /></a>
<figcaption>Figure 6. Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe.</figcaption>
</figure>
<h4 id="step-3---assets-weights-computation-across-all-clusters-and-final-assets-weights-computation">Step 3 - Assets weights computation across all clusters and final assets weights computation</h4>
<p>The third and last step consists in computing the ERC portfolio across all clusters $\mathcal{G}_i$, $i=1..k$.</p>
<p>For this:</p>
<ul>
<li>
<p>Because the $k$ clusters $\mathcal{G}_1,…,\mathcal{G}_k$ correspond to $k$ risk factors, whose loadings matrix $A \in \mathcal{M}(\mathbb{R}^{n \times k})$ - in the original asset weights - is $ A = \begin{pmatrix} w_1^* & w_2^* & … & w_k^* \end{pmatrix} $, the cluster covariance matrix $\Sigma_c \in \mathcal{M}(\mathbb{R}^{k \times k})$ is equal<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">14</a></sup> to $ A^t \Sigma A $.</p>
</li>
<li>
<p>Thanks to that cluster covariance matrix, it is possible to compute the ERC portfolio across all clusters.</p>
<p>This results in a vector of cluster weights $w_c^* \in [0,1]^{k}$.</p>
</li>
<li>
<p>Then, all remaining is to convert the vector of cluster weights $w_c^*$ into the final vector of asset weights $w^* \in [0,1]^n$ through the formula $w^* = A w_c^*$.</p>
</li>
</ul>
<p>Continuing the example from the previous sub-section:</p>
<ul>
<li>
<p>The loading matrix $A$, deduced from Figure 6, is:</p>
\[A = \begin{pmatrix}
0.19 & 0 & 0 \\
0.16 & 0 & 0 \\
0.19 & 0 & 0 \\
0.17 & 0 & 0 \\
0.13 & 0 & 0 \\
0.16 & 0 & 0 \\
0 & 0.66 & 0 \\
0 & 0.34 & 0 \\
0 & 0 & 0.48 \\
0 & 0 & 0.52 \\
\end{pmatrix}\]
</li>
<li>
<p>The cluster covariance matrix $\Sigma_c$ is:</p>
\[\Sigma_c = A^t \Sigma A = \begin{pmatrix}
0.000063 & 0.000014 & 0.000016 \\
0.000014 & 0.000053 & 0.000011 \\
0.000016 & 0.000011 & 0.000047 \\
\end{pmatrix}\]
</li>
<li>
<p>The vector of cluster weights of the across-cluster ERC portfolio is:</p>
\[w_c^* = \begin{pmatrix} 0.30 \\ 0.34 \\ 0.36 \end{pmatrix}\]
</li>
<li>
<p>The vector of asset weights of the CRP portfolio, $w^* = A w_c^*$, is displayed in Figure 7:</p>
<figure>
<a href="/assets/images/blog/cluster-risk-parity-crp-portfolio.png"><img src="/assets/images/blog/cluster-risk-parity-crp-portfolio.png" alt="Example of CRP portfolio computation, final portfolio weights, 10-ETF universe." /></a>
<figcaption>Figure 7. Example of CRP portfolio computation, final portfolio weights, 10-ETF universe.</figcaption>
</figure>
</li>
</ul>
<h3 id="practical-performances">Practical performances</h3>
<p>The CRP portfolio has been empirically benchmarked against several other portfolios within different multi-asset classes universes<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">15</a></sup> and its performances, especially in terms of risk-adjusted returns,
have been found to be excellent.</p>
<p>On this aspect, Varadi<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> concludes that:</p>
<blockquote>
<p>The [CRP portfolio] is perhaps the most robust method of passive portfolio allocation, and it also produces the best risk-adjusted returns without relying as much on the low-volatility factor or bond/fixed income performance.</p>
</blockquote>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> allows to easily compute the CRP portfolio through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/optimization/equal-risk-contributions/clustering-based</code></a>, with optional support
for minimum and maximum asset weights constraints.</p>
<p>For maximum flexibility, the partitioning of the considered universe of assets into clusters is expected to be provided in input to that endpoint, which can be done using either:</p>
<ul>
<li>Any clustering algorithm available on the user side</li>
<li>Any clustering algorithm provided by <strong>Portfolio Optimizer</strong>:
<ul>
<li>The <em><a href="https://cssanalytics.wordpress.com/2013/11/26/fast-threshold-clustering-algorithm-ftca/">Fast Threshold Clustering Algorithm (FTCA)</a></em>, available through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/clustering/ftca</code></a></li>
<li>The hierarchical clustering algorithm used in the <a href="/blog/beyond-hierarchical-risk-parity-hierarchical-clustering-based-risk-parity">Hierarchical Clustering-Based Risk Parity</a> algorithm, available through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/clustering/hierarchical</code></a></li>
</ul>
</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>This concludes the description of the CRP portfolio, a little known but <em>excellent framework to consider for a robust risk parity approach</em><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">15</a></sup>.</p>
<p>As usual, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://doi.org/10.3905/jpm.2010.36.4.060">Maillard, S., Roncalli, T., Teiletche, J.: The properties of equally weighted risk contribution portfolios. J. Portf. Manag. 36, 60–70 (2010)</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:1:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:1:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:1:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:1:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.risk.net/journal-of-investment-strategies/2255764/properties-of-the-most-diversified-portfolio">Choueifaty, Y., Tristan Froidure, T., Reynier, J. Properties of the Most Diversified Portfolio. Journal of Investment Strategies, Vol.2(2), Spring 2013, pp.49-70</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://cssanalytics.wordpress.com/2013/01/03/cluster-risk-parity/">David Varadi, CSSA, Cluster Risk Parity</a> and <a href="https://cssanalytics.wordpress.com/2013/01/06/cluster-risk-parity-crp-versus-risk-parity-rp-and-equal-risk-contribution-erc/">David Varadi, CSSA, Cluster Risk Parity (CRP) versus Risk Parity (RP) and Equal Risk Contribution (ERC)</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://systematicinvestor.wordpress.com/2013/02/12/cluster-portfolio-allocation/">Michael Kapler, Systematic Investor Blog, Cluster Portfolio Allocation</a> and <a href="https://systematicinvestor.wordpress.com/2013/03/05/cluster-risk-parity-back-test/">Michael Kapler, Systematic Investor Blog, Cluster Risk Parity back-test</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>When the risk measure is defined as the portfolio volatility; other risk measures can be used instead, c.f. Maillard et al.<sup id="fnref:1:10" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>This property is established in all generality in Maillard et al.<sup id="fnref:1:11" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2155159">Roncalli, Thierry and Weisang, Guillaume, Risk Parity Portfolios with Risk Factors</a> and <a href="https://www.tandfonline.com/doi/full/10.1080/14697688.2015.1046907">Roncalli & G. Weisang, Risk parity portfolios with risk factors, Quantitative Finance</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Per construction of the ERC portfolio, ll assets present in that portfolio are guaranteed to have the same total risk contributions. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.mdpi.com/1099-4300/24/11/1631">Gava J, Turc J. The Properties of Alpha Risk Parity Portfolios. Entropy. 2022; 24(11):1631</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Assumed to form <a href="https://en.wikipedia.org/wiki/Partition_of_a_set">a partition</a> of the universe of assets. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">16</a></sup>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>This covariance matrix is the empirical asset covariance matrix computed from the arithmetic returns of the 10 assets over the period 04th January 2023 - 29th December 2023. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>This is a direct application of the formula for the covariance matrix of a linear transformation of a random vector, c.f. for example <a href="https://en.wikipedia.org/wiki/Covariance_matrix">Wikipedia, Covariance matrix</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://investresolve.com/dynamic-asset-allocation-for-practitioners-part-5-robust-risk-parity/">ReSolve Asset Management, Dynamic asset allocation for practitioners part 5: robust risk parity</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.The equal risk contribution (ERC) portfolio, introduced in Maillard et al.1, is a portfolio aiming to equalize the risk contributions from [its] different components1. Empirically, the ERC portfolio has been found to be a middle-ground alternative1 to an equally weighted portfolio and a minimum variance portfolio, balanced in risk and in weights1, which exhibits interesting performances in terms of risk-adjusted returns1. One issue with the ERC portfolio, though, it that it is highly dependent upon the structure of the universe of assets considered2 and tends to distribute risk unevenly in terms of asset classes whenever the number of assets per asset class differs. In this short blog post, I will describe an extension of the ERC portfolio proposed by Varadi3 and Kapler4 designed to solve this problem. The equal risk contribution portfolio Let be: $n$, the number of assets in a universe of assets $w = (w_1,…,w_n) \in {\mathbb{R}^+}^{n}$, the asset weights of a long-only portfolio invested in the considered universe of assets $\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the asset covariance matrix $\sigma_{i,j}$, the covariance between assets $i$ and $j$, $i=1..n$, $j=1..n$ $\sigma_1^2,…,\sigma_n^2 \in {\mathbb{R}^+}^{n}$, the asset variances $\sigma(w) = \sqrt{ w^t \Sigma w }$, the portfolio standard deviation/volatility Definitions Risk contributions The marginal contribution to risk5 - or marginal risk contribution - $MCTR_i(w)$ of the asset $i, i=1..n$, to the portfolio risk $\sigma(w)$ is defined as: \[MCTR_i(w) = \frac{\partial \sigma(w)}{\partial w_i} = \frac{w_i \sigma_{i}^2 + \Sigma_{i=1, i\ne j}^n w_j \sigma_{i,j}}{\sigma(w)}\] As noted in Maillard et al.1, the adjective marginal qualifies the fact that those quantities give the change in volatility of the portfolio induced by a small increase in the weight of one component1, all other weights staying the same. The total contribution to risk - or total risk contribution - $TCTR_i(w)$ of the $i$-th asset, $i=1..n$, to the portfolio risk $\sigma(w)$ is defined as: \[TCTR_i(w) = w_i MCTR_i(w)\] The ERC portfolio The ERC portfolio is defined as the risk-balanced portfolio such that the total risk contribution is the same for all assets in the portfolio1. More formally, the ERC portfolio is the long-only portfolio whose asset weights $w^* \in [0,1]^{n}$ satisfy: $\sum_{i=1}^{n} w_i = 1$ $TCTR_i(w) = TCTR_j(w)$, for all $i$, $j$, $i=1..n$, $j=1..n$ Sensitivity of the ERC portfolio to the universe of assets Choueifaty et al.2 defines a set of basic properties that an unbiased, agnostic portfolio construction process should respect, based on common sense and reasonable economic grounds2. I propose to illustrate one of these properties, named duplication invariance, by comparing two portfolios invested in U.S. stocks (SPY ETF) and U.S. bonds (TLT ETF): The minimum variance (MV) portfolio The ERC portfolio Using asset data over the period 03rd January 2023 - 29th December 20236, the associated portfolios are displayed in Figure 1: Figure 1. ERC v.s. MV portfolio weights, SPY-TLT universe. Something interesting to note in Figure 1 is that the ERC portfolio appears closer to an equally weighted portfolio than the MV portfolio, which is a general property discussed in Maillard et al.1. Now, let’s suppose that the TLT ETF is somehow duplicated into a TLT’ ETF over the same period of time. What would be the impact of this asset duplication on the above portfolios? Answer is provided in Figure 2. Figure 2. Impact of TLT ETF duplication on ERC v.s. MV portfolio weights, SPY-TLT universe. It is visible on Figure 2 that: The weights of the ERC portfolio in the original assets are (severely) impacted by the duplication of TLT ETF. The ERC portfolio is said be non duplication invariant. The weights of the MV portfolio in the original assets are left unchanged7 by the duplication of the TLT ETF. The MV portfolio is said to be duplication invariant. This lack of duplication invariance means that ERC portfolios are generally biased toward assets with multiple representations2, so that the asset universe is an important factor when considering [ERC] portfolios8. As an example, for a portfolio invested in bonds and equities, the number […] of bond versus equity will dramatically affect the risk contribution of the [ERC] portfolio from each category3. Indeed, as detailed in Roncalli and Weisang8: If the universe includes 5 equity indices and 5 bond indices, then the ERC portfolio will be well balanced between equity and bond in terms of risk. Conversely, if the universe includes 7 equity indices and 3 bond indices, the equity risk of the ERC portfolio represents then 70% of the portfolio’s total risk, a solution very unbalanced between equity and bond risks. Figure 3 illustrates this imbalance within the SPY-TLT universe, with the TLT duplicated 5 times (resp. 3 times) and the SPY ETF duplicated 5 times (resp. 7 times). Figure 3. Impact of SPY and TLT ETFs duplication on ERC portfolio total contributions to risk, assets v.s. asset classes, SPY-TLT universe. Interestingly, such a behavior is not in violation of the mathematical objective of the ERC portfolio9, but is nevertheless in clear violation of the ERC portfolio philosophy based on diversification1. Notes: Asset weights in Figure 1 and Figure 2 might seem surprising at first sight (higher weights for the SPY ETF v.s. the TLT ETF), but 2023 has been a particular year for U.S. Treasuries… For reference, the exact (co)variances of the SPY and TLT ETFs are provided in Figure 5. For more discussion about duplication invariance, see for example Gava and Turc10, which introduces a quantitative measure of the sensitivity of a portfolio to duplication. The cluster risk parity portfolio Description One possible solution to the problem of the sensitivity of the ERC portfolio discussed in the previous section is a careful selection of the universe of assets. Another possible solution is to slightly alter the ERC portfolio construction process by taking into account both the asset and the asset class level, which is the solution proposed by Varadi3 and Kapler4 under the name cluster risk parity (CRP) portfolio. In details, the CRP portfolio is computed through a three-step process, detailed below. Notes: Another cluster risk parity portfolio is described in the Quantpedia series Introduction to Clustering Methods In Portfolio Management, but the two portfolio construction methodologies are not the same. Step 1 - Partitioning the universe of assets into clusters The first step consists in using a clustering algorithm to group similar assets together within the considered universe of assets. This results in $k$ clusters11 $\mathcal{G}_1,…,\mathcal{G}_k$, $1 \leq k \leq n$, each corresponding to an asset class automatically determined by the clustering algorithm. The purpose of this step is to avoid the need for artificial or manual grouping3 of the assets and enables the [CRP] portfolio to adapt to changes […] without having to run a lot of ad hoc analysis and having to make continual adjustments3. Using the same hierarchical clustering algorithm as the one used in the Hierarchical Clustering-Based Risk Parity algorithm and using asset data over the period 03rd January 2023 - 29th December 20236, Figure 4 illustrates this first step for a universe of 10 ETFs representative12 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Figure 4. Example of hierarchical clustering algorithm with automatic determination of the optimal number of clusters, 10-ETF universe. Figure 4 shows that 3 clusters/asset classes have been automatically determined by the hierarchical clustering algorithm over the studied period, materialized by the thick black dendrogram cut: An equity cluster $\mathcal{G}_1$ made of the equity ETFs SPY, EZU, EWJ, EEM and of the real estate ETFs VNQ, RWX A bond cluster $\mathcal{G}_2$ made of the bond ETFs IEF, TLT A commodity cluster $\mathcal{G}_3$ made of the commodity ETF DBC and of the gold ETF GLD Step 2 - Assets weights computation within each cluster The second step consists in computing the ERC portfolio within each cluster $\mathcal{G}_i$, $i=1..k$. This results in $k$ vectors of asset weights $w_{i}^* \in [0,1]^{n}$, $i=1..k$. Using as input the covariance matrix $\Sigma$ provided in Figure 513, the ERC portfolio computed within each cluster $\mathcal{G}_1$, $\mathcal{G}_2$ and $\mathcal{G}_3$ defined in the previous sub-section is displayed in Figure 6. Figure 5. Example of CRP portfolio computation, asset covariance matrix, 10-ETF universe. Figure 6. Example of CRP portfolio computation, within-cluster ERC portfolios, 10-ETF universe. Step 3 - Assets weights computation across all clusters and final assets weights computation The third and last step consists in computing the ERC portfolio across all clusters $\mathcal{G}_i$, $i=1..k$. For this: Because the $k$ clusters $\mathcal{G}_1,…,\mathcal{G}_k$ correspond to $k$ risk factors, whose loadings matrix $A \in \mathcal{M}(\mathbb{R}^{n \times k})$ - in the original asset weights - is $ A = \begin{pmatrix} w_1^* & w_2^* & … & w_k^* \end{pmatrix} $, the cluster covariance matrix $\Sigma_c \in \mathcal{M}(\mathbb{R}^{k \times k})$ is equal14 to $ A^t \Sigma A $. Thanks to that cluster covariance matrix, it is possible to compute the ERC portfolio across all clusters. This results in a vector of cluster weights $w_c^* \in [0,1]^{k}$. Then, all remaining is to convert the vector of cluster weights $w_c^*$ into the final vector of asset weights $w^* \in [0,1]^n$ through the formula $w^* = A w_c^*$. Continuing the example from the previous sub-section: The loading matrix $A$, deduced from Figure 6, is: \[A = \begin{pmatrix} 0.19 & 0 & 0 \\ 0.16 & 0 & 0 \\ 0.19 & 0 & 0 \\ 0.17 & 0 & 0 \\ 0.13 & 0 & 0 \\ 0.16 & 0 & 0 \\ 0 & 0.66 & 0 \\ 0 & 0.34 & 0 \\ 0 & 0 & 0.48 \\ 0 & 0 & 0.52 \\ \end{pmatrix}\] The cluster covariance matrix $\Sigma_c$ is: \[\Sigma_c = A^t \Sigma A = \begin{pmatrix} 0.000063 & 0.000014 & 0.000016 \\ 0.000014 & 0.000053 & 0.000011 \\ 0.000016 & 0.000011 & 0.000047 \\ \end{pmatrix}\] The vector of cluster weights of the across-cluster ERC portfolio is: \[w_c^* = \begin{pmatrix} 0.30 \\ 0.34 \\ 0.36 \end{pmatrix}\] The vector of asset weights of the CRP portfolio, $w^* = A w_c^*$, is displayed in Figure 7: Figure 7. Example of CRP portfolio computation, final portfolio weights, 10-ETF universe. Practical performances The CRP portfolio has been empirically benchmarked against several other portfolios within different multi-asset classes universes415 and its performances, especially in terms of risk-adjusted returns, have been found to be excellent. On this aspect, Varadi3 concludes that: The [CRP portfolio] is perhaps the most robust method of passive portfolio allocation, and it also produces the best risk-adjusted returns without relying as much on the low-volatility factor or bond/fixed income performance. Implementation in Portfolio Optimizer Portfolio Optimizer allows to easily compute the CRP portfolio through the endpoint /portfolios/optimization/equal-risk-contributions/clustering-based, with optional support for minimum and maximum asset weights constraints. For maximum flexibility, the partitioning of the considered universe of assets into clusters is expected to be provided in input to that endpoint, which can be done using either: Any clustering algorithm available on the user side Any clustering algorithm provided by Portfolio Optimizer: The Fast Threshold Clustering Algorithm (FTCA), available through the endpoint /assets/clustering/ftca The hierarchical clustering algorithm used in the Hierarchical Clustering-Based Risk Parity algorithm, available through the endpoint /assets/clustering/hierarchical Conclusion This concludes the description of the CRP portfolio, a little known but excellent framework to consider for a robust risk parity approach15. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Maillard, S., Roncalli, T., Teiletche, J.: The properties of equally weighted risk contribution portfolios. J. Portf. Manag. 36, 60–70 (2010). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 See Choueifaty, Y., Tristan Froidure, T., Reynier, J. Properties of the Most Diversified Portfolio. Journal of Investment Strategies, Vol.2(2), Spring 2013, pp.49-70. ↩ ↩2 ↩3 ↩4 See David Varadi, CSSA, Cluster Risk Parity and David Varadi, CSSA, Cluster Risk Parity (CRP) versus Risk Parity (RP) and Equal Risk Contribution (ERC). ↩ ↩2 ↩3 ↩4 ↩5 ↩6 See Michael Kapler, Systematic Investor Blog, Cluster Portfolio Allocation and Michael Kapler, Systematic Investor Blog, Cluster Risk Parity back-test. ↩ ↩2 ↩3 When the risk measure is defined as the portfolio volatility; other risk measures can be used instead, c.f. Maillard et al.1. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ ↩2 This property is established in all generality in Maillard et al.1. ↩ See Roncalli, Thierry and Weisang, Guillaume, Risk Parity Portfolios with Risk Factors and Roncalli & G. Weisang, Risk parity portfolios with risk factors, Quantitative Finance. ↩ ↩2 Per construction of the ERC portfolio, ll assets present in that portfolio are guaranteed to have the same total risk contributions. ↩ See Gava J, Turc J. The Properties of Alpha Risk Parity Portfolios. Entropy. 2022; 24(11):1631. ↩ Assumed to form a partition of the universe of assets. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer16. ↩ This covariance matrix is the empirical asset covariance matrix computed from the arithmetic returns of the 10 assets over the period 04th January 2023 - 29th December 2023. ↩ This is a direct application of the formula for the covariance matrix of a linear transformation of a random vector, c.f. for example Wikipedia, Covariance matrix. ↩ See ReSolve Asset Management, Dynamic asset allocation for practitioners part 5: robust risk parity. ↩ ↩2 See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Volatility Forecasting: GARCH(1,1) Model2024-03-11T00:00:00-05:002024-03-11T00:00:00-05:00https://portfoliooptimizer.io/blog/volatility-forecasting-garch11-model<p>In the <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous post</a> of this series on volatility forecasting, I described the simple and the exponentially
weighted moving average volatility forecasting models.</p>
<p>In particular, I showed that these two models belong to the generic family of weighted moving average volatility forecasting models<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup>, whose members
represent the volatility of an asset as a <a href="https://en.wikipedia.org/wiki/Moving_average">weighted moving average</a> of its past squared returns<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup>.</p>
<p>Another member of this family is the <em><a href="https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity">Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) model</a></em>,
<em>widely used in financial time series modelling and implemented in most statistics and econometric software packages</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>In this blog post, I will detail the <em>simplest but often very useful</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">4</a></sup> GARCH(1,1) volatility forecasting model and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs.</p>
<h2 id="mathematical-preliminaries-reminders">Mathematical preliminaries (reminders)</h2>
<p>This section contains reminders from a <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a>.</p>
<h3 id="volatility-modelling-and-volatility-proxies">Volatility modelling and volatility proxies</h3>
<p>Let $r_t$ be the (<a href="https://en.wikipedia.org/wiki/Rate_of_return#Logarithmic_or_continuously_compounded_return">logarithmic</a>) return of an asset over a time period $t$ (a day, a week, a month..),
over which its (conditional) mean return is supposed to be null.</p>
<p>Then:</p>
<ul>
<li>
<p>The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $</p>
<p>From this definition, the squared return $r_t^2$ of an asset is a (noisy<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">5</a></sup>) <em>variance estimator</em> - or <em>variance proxy</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">5</a></sup> - for that asset variance over the considered time period.</p>
<p>Another example of an asset variance proxy is <a href="/blog/range-based-volatility-estimators-overview-and-examples-of-usage/">the Parkinson range</a> of an asset.</p>
<p>The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$.</p>
</li>
<li>
<p>The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $</p>
<p>The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$.</p>
</li>
</ul>
<h3 id="weighted-moving-average-volatility-forecasting-model">Weighted moving average volatility forecasting model</h3>
<p>Boudoukh et al.<sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup> shows that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as
a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with</p>
\[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\]
<p>, where:</p>
<ul>
<li>$1 \leq k \leq T$ is the size of the moving average, possibly time-dependent</li>
<li>$w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well</li>
</ul>
<h2 id="garch11-volatility-forecasting-model">GARCH(1,1) volatility forecasting model</h2>
<h3 id="the-garchpq-model">The GARCH(p,q) model</h3>
<h4 id="definition">Definition</h4>
<p>Bollerslev<sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">4</a></sup>’s GARCH model is a generalization of Engle’s <a href="https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity">ARCH</a> <a href="https://en.wikipedia.org/wiki/Econometric_model">econometric model</a>
which captures the time-varying nature of the (conditional) variance of certain time series like asset returns.</p>
<p>Under a GARCH(p,q) model, an asset next period’s conditional variance $\sigma_{T+1}^2$ is modeled as recursive linear function of its own $p$ lagged conditional variances $\sigma_{T}^2, \sigma_{T-1}^2…$ and of its $q$
lagged squared returns $r_{T}^2, r_{T-1}^2…$, which leads to the formula</p>
\[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j r_{T+1-i}^2\]
<p>, where:</p>
<ul>
<li>The parameters $\omega$, $\alpha_j$, $j=1..q$ and $\beta_i$, $i=1..p$ are non-negative and subject to various inequality constraints depending on working assumptions<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">6</a></sup></li>
<li>The initial conditional variance $\hat{\sigma}_1^2$ is usually taken equal to $r_1^2$, but c.f. Pelagatti and Lisi<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">7</a></sup> for a thorough discussion about this subject</li>
</ul>
<h4 id="squared-returns-vs-generic-variance-proxy">Squared returns v.s. generic variance proxy</h4>
<p>Molnar<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> notes that <em>in GARCH type of models, demeaned squared returns serve as a way to calculate innovations to the volatility</em><sup id="fnref:9:1" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> so that <em>replacing the squared returns by more
precise volatility estimates will produce better GARCH models, regarding both in-sample fit and out-of-sample forecasting performance</em><sup id="fnref:9:2" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<p>Molnar<sup id="fnref:9:3" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> then proposes to modify the GARCH(p,q) model for the estimation of an asset next period’s conditional variance $\sigma_{T+1}^2$ as follows</p>
\[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j \tilde{\sigma}_{T+1-i}^2\]
<p>, where $\tilde{\sigma}^2_t$, $t=1..T$ are the asset past periods’ variance proxies.</p>
<p>To be noted that replacing squared returns by less noisy variance proxies is already discussed at length in the
<a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">previous blog post</a> in the case of the simple and the exponentially weighted moving average volatility forecasting models.</p>
<h3 id="the-garch11-model">The GARCH(1,1) model</h3>
<h4 id="definition-1">Definition</h4>
<p>Because the GARCH(1,1) model <em>works surprisingly well in comparison with much more complex [GARCH] models</em><sup id="fnref:9:4" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>, it is usually the main GARCH model used in practice.</p>
<p>Under this model, the generic GARCH formula for the estimate of an asset next period’s conditional variance can be re-parametrized as follows</p>
\[\hat{\sigma}_{T+1}^2 = \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2\]
<p>, where:</p>
<ul>
<li>$\alpha$, $\beta$ and $\gamma$ are positive parameters summing to one</li>
<li>$\tilde{\sigma}^2$ is a strictly positive parameter, corresponding to the asset unconditional variance<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup></li>
</ul>
<p>The GARCH(1,1) model thus estimates an asset next period’s conditional variance $\hat{\sigma}_{T+1}^2$ as a weighted average<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">10</a></sup> of three different variance estimators:</p>
<ul>
<li>A long-term variance estimator $\tilde{\sigma}^2$</li>
<li>A short-term variance estimator $\tilde{\sigma}^2_{T}$</li>
<li>The current GARCH(1,1) variance estimator $\hat{\sigma}_{T}^2$</li>
</ul>
<p>and the weights $\alpha$, $\beta$ and $\gamma$ determine the speed with which the model adapts to short-term variance v.s. reverts to its long-term variance.</p>
<h4 id="relationship-with-the-generic-weighted-moving-average-model">Relationship with the generic weighted moving average model</h4>
<p>By developing the recursive definition of the GARCH(1,1) model, it is possible to see that it is a specific kind of weighted moving average volatility forecasting model, with:</p>
<ul>
<li>$k = T$</li>
<li>$w_0 = \gamma \sum_{k=0}^{T-1} \beta^k$</li>
<li>$w_1 = \alpha$, $w_2 = \alpha \beta$, …, $w_{T-1} = \alpha \beta^{T-2}$, $w_T = \alpha \beta^{T-1}$, that is, exponentially decreasing weights emphasizing recent past variance proxies v.s. more distant ones in the model, exactly like in the exponentially weighted moving average volatility forecasting model<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">11</a></sup></li>
</ul>
<h4 id="volatility-forecasting-formulas">Volatility forecasting formulas</h4>
<p>Under a GARCH(1,1) volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes:</p>
<ul>
<li>
<p>To estimate an asset next period’s volatility:</p>
\[\hat{\sigma}_{T+1} = \sqrt{ \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2 }\]
</li>
<li>
<p>To estimate an asset next $h$-period’s ahead volatility<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">12</a></sup>, $h \geq 2$:</p>
\[\hat{\sigma}_{T+h} = \sqrt{ \tilde{\sigma}^2 + \left( \alpha + \beta \right)^{h-1} \left( \hat{\sigma}_{T+1} - \tilde{\sigma}^2 \right) }\]
</li>
<li>
<p>To estimate an asset aggregated volatility<sup id="fnref:16:1" role="doc-noteref"><a href="#fn:16" class="footnote">12</a></sup> over the next $h$ periods:</p>
\[\hat{\sigma}_{T+1:T+h} = \sqrt{h} \hat{\sigma}_{T+1}\]
</li>
</ul>
<h4 id="how-to-determine-the-parameters-of-a-garch11-model">How to determine the parameters of a GARCH(1,1) model?</h4>
<p>The parameters of a GARCH(1,1) model - either $\omega$, $\alpha$ and $\beta$ or $\alpha$, $\beta$, $\gamma$ and $\tilde{\sigma}^2$ - are typically determined by
<a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation">maximum likelihood estimation (MLE)</a> with a Gaussian<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">13</a></sup> or Student’s $t$ assumption for the distribution of the innovations.</p>
<p>A note of caution, though.</p>
<p>There are plenty of software packages able to do this estimation, but the underlying optimization problem <em>has been documented to be numerically
difficult and prone to error</em><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> due to <em>a one dimensional manifold in the parameter space where the likelihood function is large and almost constant</em><sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup>, which tends to “trap” numerical algorithms.</p>
<p>Possible remediations have been suggested in Zumbach<sup id="fnref:15:2" role="doc-noteref"><a href="#fn:15" class="footnote">14</a></sup> and in Kristensen and Linton<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup>, like reformulating the optimization problem in an alternative parameter space or using
a closed-form estimator for the GARCH(1,1) parameters that does not rely on any numerical optimization procedure, but unfortunately, these remediations are not sufficient due to the
problematic<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">16</a></sup> finite sample behavior of the maximum likelihood estimates…</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the GARCH(1,1) volatility forecasting model through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/assets/volatility/forecast/garch</code></a>.</p>
<p>This endpoint supports the 4 variance proxies below:</p>
<ul>
<li>Squared close-to-close returns</li>
<li>Demeaned squared close-to-close returns</li>
<li>The Parkinson range</li>
<li>The jump-adjusted Parkinson range</li>
</ul>
<p>Internally, this endpoint:</p>
<ul>
<li>Assumes that the asset unconditional variance $\tilde{\sigma}^2$ is equal to its long-term average value $\frac{1}{T} \sum_{t=1}^{T} \tilde{\sigma}^2_t$</li>
<li>Automatically determines the optimal value of the GARCH(1,1) parameters $\alpha$, $\beta$ and $\gamma$ using a proprietary numerical optimization procedure</li>
</ul>
<h2 id="example-of-usage---volatility-forecasting-at-monthly-level-for-various-etfs">Example of usage - Volatility forecasting at monthly level for various ETFs</h2>
<p>As an example of usage, I propose to enrich the results of <a href="/blog/volatility-forecasting-simple-and-exponentially-weighted-moving-average-models/">the previous blog post</a>, in which
monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">17</a></sup> regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">18</a></sup> of misc. asset classes:</p>
<ul>
<li>U.S. stocks (SPY ETF)</li>
<li>European stocks (EZU ETF)</li>
<li>Japanese stocks (EWJ ETF)</li>
<li>Emerging markets stocks (EEM ETF)</li>
<li>U.S. REITs (VNQ ETF)</li>
<li>International REITs (RWX ETF)</li>
<li>U.S. 7-10 year Treasuries (IEF ETF)</li>
<li>U.S. 20+ year Treasuries (TLT ETF)</li>
<li>Commodities (DBC ETF)</li>
<li>Gold (GLD ETF)</li>
</ul>
<p>Averaged results for all ETFs/regression models over each ETF price history<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">19</a></sup> are the following<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">20</a></sup>:</p>
<table>
<thead>
<tr>
<th>Volatility model</th>
<th>Variance proxy</th>
<th>$\bar{\alpha}$</th>
<th>$\bar{\beta}$</th>
<th>$\bar{R^2}$</th>
</tr>
</thead>
<tbody>
<tr>
<td>Random walk</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.66</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Squared close-to-close returns</td>
<td>5.8%</td>
<td>0.68</td>
<td>46%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Squared close-to-close returns</td>
<td>4.7%</td>
<td>0.73</td>
<td>45%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Squared close-to-close returns</strong></td>
<td><strong>-1.3%</strong></td>
<td><strong>0.98</strong></td>
<td><strong>43%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Parkinson range</td>
<td>5.6%</td>
<td>0.94</td>
<td>44%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Parkinson range</td>
<td>5.1%</td>
<td>1.00</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Parkinson range</td>
<td>4.3%</td>
<td>1.06</td>
<td>48%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Parkinson range</strong></td>
<td><strong>2.7%</strong></td>
<td><strong>1.18</strong></td>
<td><strong>47%</strong></td>
</tr>
<tr>
<td>Random walk</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.9%</td>
<td>0.70</td>
<td>45%</td>
</tr>
<tr>
<td>SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days</td>
<td>Jump-adjusted Parkinson range</td>
<td>5.1%</td>
<td>0.71</td>
<td>47%</td>
</tr>
<tr>
<td>EWMA, optimal $\lambda$</td>
<td>Jump-adjusted Parkinson range</td>
<td>4.0%</td>
<td>0.76</td>
<td>45%</td>
</tr>
<tr>
<td><strong>GARCH(1,1)</strong></td>
<td><strong>Jump-adjusted Parkinson range</strong></td>
<td><strong>-1.0%</strong></td>
<td><strong>1.00</strong></td>
<td><strong>45%</strong></td>
</tr>
</tbody>
</table>
<p>From these, it is possible to conclude the following:</p>
<ul>
<li>The two GARCH(1,1) models using improved variance proxies produce volatility forecasts with better r-squared than the GARCH(1,1) model using squared returns (lines #8 and #12 v.s. line #4), which is in agreement with Molnar<sup id="fnref:9:5" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup></li>
<li>The two GARCH(1,1) models using variance proxies that integrate close prices produce nearly unbiased forecasts (lines #4 and #12), which, together with their relatively high r-squared, makes them volatility forecasting models to recommend in these cases</li>
<li>The GARCH(1,1) using the Parkinson range as variance proxy produces the most biased forecasts (line #8), which makes it a volatility forecasting model to avoid in this case</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>The GARCH(1,1) volatility forecasting model exhibits good practical performances for a wide range of assets, as empirically demonstrated in the previous section.</p>
<p>Nevertheless, because this model is still <em>unable to describe certain aspects often found in financial data</em><sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, many different extensions have been proposed over the years like AGARCH, EGARCH, QGARCH, TGARCH<sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>….</p>
<p>I will not discuss these further, though, and next in this series dedicated to volatility forecasting, I will detail a model that was initially developed for use with high frequency data.</p>
<p>Meanwhile, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or to <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijderiv/4/3/63">Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Or more generally, of a weighted moving average of one of its past variance proxies. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://math.berkeley.edu/~btw/thesis4.pdf">Brandon Williams, GARCH(1,1) models, B. Sc. Thesis, 15. Juli 2011</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>See <a href="">Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307–327</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S030440761000076X">Andrew J. Patton, Volatility forecast comparison using imperfect volatility proxies, Journal of Econometrics, Volume 160, Issue 1, 2011, Pages 246-256</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/1391681">Daniel B. Nelson and Charles Q. Cao, Inequality Constraints in the Univariate GARCH Model, Journal of Business & Economic Statistics, Vol. 10, No. 2 (Apr., 1992), pp. 229-235</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.researchgate.net/publication/237530561_VARIANCE_INITIALISATION_IN_GARCH_ESTIMATION">Pelagatti, M., Lisi, F. (2009). Variance initialisation in GARCH estimation. In Paganoni, A.M., Sangalli, L.M., Secchi, P., Vantini, S. (eds.), S.Co. 2009 Sixth Conference Complex Data Modeling and Computationally Intensive Statistical Methods for Estimation and Prediction, Maggioli Editore, Milan</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="http://dx.doi.org/10.1080/00036846.2016.1170929">Peter Molnar (2016): High-low range in GARCH models of stock return volatility, Applied Economics</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:9:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:9:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:9:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:9:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:9:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>Also called the asset long-term variance. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>More precisely, a convex combination. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>Which is not surprising since <em>in fact, exponential smoothing is a constrained version of GARCH (1,1)</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">1</a></sup>, without mean-reversion. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://centaur.reading.ac.uk/21316/">Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:16:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>In which case, the Gaussian MLE is usually considered as a <a href="https://en.wikipedia.org/wiki/Quasi-maximum_likelihood_estimate">quasi-maximum likelihood estimate</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-1-4615-4389-3_8">Zumbach, G. (2000). The Pitfalls in Fitting Garch(1,1) Processes. In: Dunis, C.L. (eds) Advances in Quantitative Asset Management. Studies in Computational Finance, vol 1. Springer, Boston, MA</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:15:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/4093228">Dennis Kristensen and Oliver Linton, A Closed-Form Estimator for the GARCH(1,1) Model, Econometric Theory, Vol. 22, No. 2 (Apr., 2006), pp. 323-337</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See for example <a href="https://ntguardian.wordpress.com/2017/11/02/problems-estimating-garch-parameters-r/">here</a> and <a href="https://ntguardian.wordpress.com/2019/01/28/problems-estimating-garch-parameters-r-part-2-rugarch/">there</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>See <a href="https://econpapers.repec.org/bookchap/nbrnberch/1214.htm">Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations</a>. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>These ETFs are used in the <em>Adaptative Asset Allocation</em> strategy from <a href="https://investresolve.com/">ReSolve Asset Management</a>, described in the paper <em>Adaptive Asset Allocation: A Primer</em><sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">21</a></sup>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>For all models, I used an expanding window for the volatility forecast computation. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2328254">Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.In the previous post of this series on volatility forecasting, I described the simple and the exponentially weighted moving average volatility forecasting models. In particular, I showed that these two models belong to the generic family of weighted moving average volatility forecasting models1, whose members represent the volatility of an asset as a weighted moving average of its past squared returns2. Another member of this family is the Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) model, widely used in financial time series modelling and implemented in most statistics and econometric software packages3. In this blog post, I will detail the simplest but often very useful4 GARCH(1,1) volatility forecasting model and I will illustrate its practical performances in the context of monthly volatility forecasting for various ETFs. Mathematical preliminaries (reminders) This section contains reminders from a previous blog post. Volatility modelling and volatility proxies Let $r_t$ be the (logarithmic) return of an asset over a time period $t$ (a day, a week, a month..), over which its (conditional) mean return is supposed to be null. Then: The asset (conditional) variance is defined as $ \sigma_t^2 = \mathbb{E} \left[ r_t^2 \right] $ From this definition, the squared return $r_t^2$ of an asset is a (noisy5) variance estimator - or variance proxy5 - for that asset variance over the considered time period. Another example of an asset variance proxy is the Parkinson range of an asset. The generic notation for an asset variance proxy in this blog post is $\tilde{\sigma}_t^2$. The asset (conditional) volatility is defined as $ \sigma_t = \sqrt { \sigma_t^2 } $ The generic notation for an asset volatility proxy in this blog post is $\tilde{\sigma}_t$. Weighted moving average volatility forecasting model Boudoukh et al.1 shows that many seemingly different methods of volatility forecasting actually share the same underlying representation of the estimate of an asset next period’s variance $\hat{\sigma}_{T+1}^2$ as a weighted moving average of that asset past periods’ variance proxies $\tilde{\sigma}^2_t$, $t=1..T$, with \[\hat{\sigma}_{T+1}^2 = w_0 + \sum_{i=1}^{k} w_i \tilde{\sigma}^2_{T+1-i}\] , where: $1 \leq k \leq T$ is the size of the moving average, possibly time-dependent $w_i, i=0..k$ are the weights of the moving average, possibly time-dependent as well GARCH(1,1) volatility forecasting model The GARCH(p,q) model Definition Bollerslev4’s GARCH model is a generalization of Engle’s ARCH econometric model which captures the time-varying nature of the (conditional) variance of certain time series like asset returns. Under a GARCH(p,q) model, an asset next period’s conditional variance $\sigma_{T+1}^2$ is modeled as recursive linear function of its own $p$ lagged conditional variances $\sigma_{T}^2, \sigma_{T-1}^2…$ and of its $q$ lagged squared returns $r_{T}^2, r_{T-1}^2…$, which leads to the formula \[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j r_{T+1-i}^2\] , where: The parameters $\omega$, $\alpha_j$, $j=1..q$ and $\beta_i$, $i=1..p$ are non-negative and subject to various inequality constraints depending on working assumptions6 The initial conditional variance $\hat{\sigma}_1^2$ is usually taken equal to $r_1^2$, but c.f. Pelagatti and Lisi7 for a thorough discussion about this subject Squared returns v.s. generic variance proxy Molnar8 notes that in GARCH type of models, demeaned squared returns serve as a way to calculate innovations to the volatility8 so that replacing the squared returns by more precise volatility estimates will produce better GARCH models, regarding both in-sample fit and out-of-sample forecasting performance8. Molnar8 then proposes to modify the GARCH(p,q) model for the estimation of an asset next period’s conditional variance $\sigma_{T+1}^2$ as follows \[\hat{\sigma}_{T+1}^2 = \omega + \sum_{i=1}^p \beta_i \hat{\sigma}_{T+1-i}^2+ \sum_{j=1}^q \alpha_j \tilde{\sigma}_{T+1-i}^2\] , where $\tilde{\sigma}^2_t$, $t=1..T$ are the asset past periods’ variance proxies. To be noted that replacing squared returns by less noisy variance proxies is already discussed at length in the previous blog post in the case of the simple and the exponentially weighted moving average volatility forecasting models. The GARCH(1,1) model Definition Because the GARCH(1,1) model works surprisingly well in comparison with much more complex [GARCH] models8, it is usually the main GARCH model used in practice. Under this model, the generic GARCH formula for the estimate of an asset next period’s conditional variance can be re-parametrized as follows \[\hat{\sigma}_{T+1}^2 = \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2\] , where: $\alpha$, $\beta$ and $\gamma$ are positive parameters summing to one $\tilde{\sigma}^2$ is a strictly positive parameter, corresponding to the asset unconditional variance9 The GARCH(1,1) model thus estimates an asset next period’s conditional variance $\hat{\sigma}_{T+1}^2$ as a weighted average10 of three different variance estimators: A long-term variance estimator $\tilde{\sigma}^2$ A short-term variance estimator $\tilde{\sigma}^2_{T}$ The current GARCH(1,1) variance estimator $\hat{\sigma}_{T}^2$ and the weights $\alpha$, $\beta$ and $\gamma$ determine the speed with which the model adapts to short-term variance v.s. reverts to its long-term variance. Relationship with the generic weighted moving average model By developing the recursive definition of the GARCH(1,1) model, it is possible to see that it is a specific kind of weighted moving average volatility forecasting model, with: $k = T$ $w_0 = \gamma \sum_{k=0}^{T-1} \beta^k$ $w_1 = \alpha$, $w_2 = \alpha \beta$, …, $w_{T-1} = \alpha \beta^{T-2}$, $w_T = \alpha \beta^{T-1}$, that is, exponentially decreasing weights emphasizing recent past variance proxies v.s. more distant ones in the model, exactly like in the exponentially weighted moving average volatility forecasting model11 Volatility forecasting formulas Under a GARCH(1,1) volatility forecasting model, the generic weighted moving average volatility forecasting formula becomes: To estimate an asset next period’s volatility: \[\hat{\sigma}_{T+1} = \sqrt{ \gamma \tilde{\sigma}^2 + \alpha \tilde{\sigma}^2_{T} + \beta \hat{\sigma}_{T}^2 }\] To estimate an asset next $h$-period’s ahead volatility12, $h \geq 2$: \[\hat{\sigma}_{T+h} = \sqrt{ \tilde{\sigma}^2 + \left( \alpha + \beta \right)^{h-1} \left( \hat{\sigma}_{T+1} - \tilde{\sigma}^2 \right) }\] To estimate an asset aggregated volatility12 over the next $h$ periods: \[\hat{\sigma}_{T+1:T+h} = \sqrt{h} \hat{\sigma}_{T+1}\] How to determine the parameters of a GARCH(1,1) model? The parameters of a GARCH(1,1) model - either $\omega$, $\alpha$ and $\beta$ or $\alpha$, $\beta$, $\gamma$ and $\tilde{\sigma}^2$ - are typically determined by maximum likelihood estimation (MLE) with a Gaussian13 or Student’s $t$ assumption for the distribution of the innovations. A note of caution, though. There are plenty of software packages able to do this estimation, but the underlying optimization problem has been documented to be numerically difficult and prone to error14 due to a one dimensional manifold in the parameter space where the likelihood function is large and almost constant14, which tends to “trap” numerical algorithms. Possible remediations have been suggested in Zumbach14 and in Kristensen and Linton15, like reformulating the optimization problem in an alternative parameter space or using a closed-form estimator for the GARCH(1,1) parameters that does not rely on any numerical optimization procedure, but unfortunately, these remediations are not sufficient due to the problematic16 finite sample behavior of the maximum likelihood estimates… Implementation in Portfolio Optimizer Portfolio Optimizer implements the GARCH(1,1) volatility forecasting model through the endpoint /assets/volatility/forecast/garch. This endpoint supports the 4 variance proxies below: Squared close-to-close returns Demeaned squared close-to-close returns The Parkinson range The jump-adjusted Parkinson range Internally, this endpoint: Assumes that the asset unconditional variance $\tilde{\sigma}^2$ is equal to its long-term average value $\frac{1}{T} \sum_{t=1}^{T} \tilde{\sigma}^2_t$ Automatically determines the optimal value of the GARCH(1,1) parameters $\alpha$, $\beta$ and $\gamma$ using a proprietary numerical optimization procedure Example of usage - Volatility forecasting at monthly level for various ETFs As an example of usage, I propose to enrich the results of the previous blog post, in which monthly forecasts produced by different volatility models are compared - using Mincer-Zarnowitz17 regressions - to the next month’s close-to-close observed volatility for 10 ETFs representative18 of misc. asset classes: U.S. stocks (SPY ETF) European stocks (EZU ETF) Japanese stocks (EWJ ETF) Emerging markets stocks (EEM ETF) U.S. REITs (VNQ ETF) International REITs (RWX ETF) U.S. 7-10 year Treasuries (IEF ETF) U.S. 20+ year Treasuries (TLT ETF) Commodities (DBC ETF) Gold (GLD ETF) Averaged results for all ETFs/regression models over each ETF price history19 are the following20: Volatility model Variance proxy $\bar{\alpha}$ $\bar{\beta}$ $\bar{R^2}$ Random walk Squared close-to-close returns 5.8% 0.66 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Squared close-to-close returns 5.8% 0.68 46% EWMA, optimal $\lambda$ Squared close-to-close returns 4.7% 0.73 45% GARCH(1,1) Squared close-to-close returns -1.3% 0.98 43% Random walk Parkinson range 5.6% 0.94 44% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Parkinson range 5.1% 1.00 47% EWMA, optimal $\lambda$ Parkinson range 4.3% 1.06 48% GARCH(1,1) Parkinson range 2.7% 1.18 47% Random walk Jump-adjusted Parkinson range 4.9% 0.70 45% SMA, optimal $k \in \left[ 1, 5, 10, 15, 20 \right]$ days Jump-adjusted Parkinson range 5.1% 0.71 47% EWMA, optimal $\lambda$ Jump-adjusted Parkinson range 4.0% 0.76 45% GARCH(1,1) Jump-adjusted Parkinson range -1.0% 1.00 45% From these, it is possible to conclude the following: The two GARCH(1,1) models using improved variance proxies produce volatility forecasts with better r-squared than the GARCH(1,1) model using squared returns (lines #8 and #12 v.s. line #4), which is in agreement with Molnar8 The two GARCH(1,1) models using variance proxies that integrate close prices produce nearly unbiased forecasts (lines #4 and #12), which, together with their relatively high r-squared, makes them volatility forecasting models to recommend in these cases The GARCH(1,1) using the Parkinson range as variance proxy produces the most biased forecasts (line #8), which makes it a volatility forecasting model to avoid in this case Conclusion The GARCH(1,1) volatility forecasting model exhibits good practical performances for a wide range of assets, as empirically demonstrated in the previous section. Nevertheless, because this model is still unable to describe certain aspects often found in financial data3, many different extensions have been proposed over the years like AGARCH, EGARCH, QGARCH, TGARCH3…. I will not discuss these further, though, and next in this series dedicated to volatility forecasting, I will detail a model that was initially developed for use with high frequency data. Meanwhile, feel free to connect with me on LinkedIn or to follow me on Twitter. – See Boudoukh, J., Richardson, M., & Whitelaw, R.F. (1997). Investigation of a class of volatility estimators, Journal of Derivatives, 4 Spring, 63-71. ↩ ↩2 Or more generally, of a weighted moving average of one of its past variance proxies. ↩ See Brandon Williams, GARCH(1,1) models, B. Sc. Thesis, 15. Juli 2011. ↩ ↩2 ↩3 See Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307–327. ↩ ↩2 See Andrew J. Patton, Volatility forecast comparison using imperfect volatility proxies, Journal of Econometrics, Volume 160, Issue 1, 2011, Pages 246-256. ↩ ↩2 See Daniel B. Nelson and Charles Q. Cao, Inequality Constraints in the Univariate GARCH Model, Journal of Business & Economic Statistics, Vol. 10, No. 2 (Apr., 1992), pp. 229-235. ↩ See Pelagatti, M., Lisi, F. (2009). Variance initialisation in GARCH estimation. In Paganoni, A.M., Sangalli, L.M., Secchi, P., Vantini, S. (eds.), S.Co. 2009 Sixth Conference Complex Data Modeling and Computationally Intensive Statistical Methods for Estimation and Prediction, Maggioli Editore, Milan. ↩ See Peter Molnar (2016): High-low range in GARCH models of stock return volatility, Applied Economics. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 Also called the asset long-term variance. ↩ More precisely, a convex combination. ↩ Which is not surprising since in fact, exponential smoothing is a constrained version of GARCH (1,1)1, without mean-reversion. ↩ See Brooks, Chris and Persand, Gitanjali (2003) Volatility forecasting for risk management. Journal of Forecasting, 22(1). pp. 1-22. ↩ ↩2 In which case, the Gaussian MLE is usually considered as a quasi-maximum likelihood estimate. ↩ See Zumbach, G. (2000). The Pitfalls in Fitting Garch(1,1) Processes. In: Dunis, C.L. (eds) Advances in Quantitative Asset Management. Studies in Computational Finance, vol 1. Springer, Boston, MA. ↩ ↩2 ↩3 See Dennis Kristensen and Oliver Linton, A Closed-Form Estimator for the GARCH(1,1) Model, Econometric Theory, Vol. 22, No. 2 (Apr., 2006), pp. 323-337. ↩ See for example here and there. ↩ See Mincer, J. and V. Zarnowitz (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic Forecasts and Expectations. ↩ These ETFs are used in the Adaptative Asset Allocation strategy from ReSolve Asset Management, described in the paper Adaptive Asset Allocation: A Primer21. ↩ The common ending price history of all the ETFs is 31 August 2023, but there is no common starting price history, as all ETFs started trading on different dates. ↩ For all models, I used an expanding window for the volatility forecast computation. ↩ See Butler, Adam and Philbrick, Mike and Gordillo, Rodrigo and Varadi, David, Adaptive Asset Allocation: A Primer. ↩Random Portfolio Benchmarking: Simulation-based Performance Evaluation in Finance2024-02-06T00:00:00-06:002024-02-06T00:00:00-06:00https://portfoliooptimizer.io/blog/random-portfolio-benchmarking-simulation-based-performance-evaluation-in-finance<p>As noted in Surz<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>, <em>the question “Is [a mutual fund’s]<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">2</a></sup> performance good?” can only be answered relative to something</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>, typically by comparing that fund to a benchmark like a financial index or to a peer group.</p>
<p>Unfortunately, these two methodologies are not without issues. For example, it is very difficult to create an index <em>captur[ing] the essence of the people, process, and philosophy behind an investment product</em><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup> and peer groups have well-known biases<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>
(classification bias, survivorship bias…).</p>
<p>In a series of papers (Surz<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>, Surz<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>, Surz<sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>…), Ronald J. Surz proposes an <em>innovation that combines the better aspects of both [methodologies] while eliminating their undesirable properties</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>. This innovation
consists in evaluating a fund against the fund manager’s <em>true opportunity set</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, defined as the set of <em>all of the possible portfolios that the manager could have conceivably
held following his unique investment process</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>.</p>
<p>In practice, the fund manager’s opportunity set is approximated by the simulation of thousands of random portfolios in the same universe of assets as the one of the fund manager and satisfying the same constraints (long-only,
long-short…) and rules (portfolio rebalancing rules…) as those of the fund manager. Then, because these portfolios do not exhibit any particular skill<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, they can be <em>used as the control group to test [the]
fund manager skill</em><sup id="fnref:1:1" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, thus allowing to apply <em>modern statistics to the problem of performance evaluation</em><sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">4</a></sup>.</p>
<p>In this blog post, I will describe how to generate random portfolios, detail Surz’s original methodology as well as some of its variations and illustrate the usage of random portfolios with a couple of examples like
the creation of synthetic benchmarks or the evaluation and monitoring of trading strategies.</p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="definition">Definition</h3>
<p>Let be:</p>
<ul>
<li>$n$ the number of assets in a universe of assets</li>
<li>$\mathcal{C} \subset \mathbb{R}^{n}$ a subset of $ \mathbb{R}^{n}$ representing the constraints imposed<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">7</a></sup> on a fund manager when investing in that universe of assets, for example:
<ul>
<li>Short sale constraints</li>
<li>Concentration constraints (assets, sectors, industries…)</li>
<li>Leverage constraints</li>
<li>Cardinality constraints (i.e., minimum or maximum number of assets constraints)</li>
<li>Portfolio volatility constraints</li>
<li>Portfolio tracking error constraints</li>
<li>…</li>
</ul>
</li>
</ul>
<p>Then, a <em>(constrained) random portfolio</em> in that universe of assets is a vector of portfolio weights $w \in \mathbb{R}^{n}$ generated at random over the set $\mathcal{C}$.</p>
<h3 id="generation-at-random-vs-generation-uniformly-at-random">Generation at random v.s. generation uniformly at random</h3>
<p>In the context of performance evaluation, as in Surz’s methodology, it is theoretically preferable that random portfolios are generated <em>uniformly</em> at random over the set $\mathcal{C}$, so as not to introduce any biases<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>
which would otherwise defeat the purpose of using these portfolios as an unbiased control group<sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">3</a></sup>.</p>
<p>Figure 1 and Figure 2 illustrate the difference between random portfolios generated at random v.s. uniformly at random in the case of a three-asset universe subject to long-only
and full investment constraints<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup>:</p>
<ul>
<li>In Figure 1, the random portfolios are visibly concentrated in the middle of the standard 2-simplex in $\mathbb{R}^3$<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup> - these random portfolios are NOT generated uniformly at random</li>
<li>In Figure 2, the random portfolios seem to be “well spread” over the standard 2-simplex in $\mathbb{R}^3$<sup id="fnref:11:2" role="doc-noteref"><a href="#fn:11" class="footnote">9</a></sup> - these random portfolios are generated uniformly at random</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-non-uniform-random-generation-shaw.png"><img src="/assets/images/blog/random-portfolio-benchmarking-non-uniform-random-generation-shaw.png" alt="Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw." /></a>
<figcaption>Figure 1. Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-uniform-random-generation-shaw.png"><img src="/assets/images/blog/random-portfolio-benchmarking-uniform-random-generation-shaw.png" alt="Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw." /></a>
<figcaption>Figure 2. Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw.</figcaption>
</figure>
<p>One important remark, though, is that real-life portfolios are usually binding on at least one of their constraints, so that generating random portfolios biased toward the boundary of the geometrical
object associated to the constraints set $\mathcal{C}$ might not be a real problem in practice<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>.</p>
<h3 id="generation-of-random-portfolios-over-the-standard-simplex">Generation of random portfolios over the standard simplex</h3>
<p>When the constraints imposed on a portfolio are 1) a full investment constraint and 2) a long-only constraint, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, w_i \geq 0, i = 1..n\}\]
<p>and the geometrical object associated to that subset is called<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">11</a></sup> a <a href="https://en.wikipedia.org/wiki/Simplex#Standard_simplex">standard simplex</a>, already illustrated in Figure 1 and Figure 2 with $n = 3$.</p>
<p>Several algorithms exist to generate points uniformly at random over a standard simplex<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup>, among which:</p>
<ul>
<li>An algorithm based on differences of sorted <a href="https://en.wikipedia.org/wiki/Continuous_uniform_distribution">uniform random variables</a></li>
<li>An algorithm based on normalized <a href="https://en.wikipedia.org/wiki/Exponential_distribution">unit exponential random variables</a></li>
</ul>
<h3 id="generation-of-random-portfolios-over-the-restricted-standard-simplex">Generation of random portfolios over the restricted standard simplex</h3>
<p>When the constraints imposed on a portfolio are 1) a full investment constraint, 2) a long-only constraint and 3) minimum/maximum asset weights constraints, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, 1 \geq u_i \geq w_i \geq l_i \geq 0, i = 1..n\}\]
<p>, where:</p>
<ul>
<li>$l_1, …, l_n$ represent minimum asset weights constraints</li>
<li>$u_1, …, u_n$ represent maximum asset weights constraints</li>
</ul>
<p>The geometrical object associated to that subset is a “restricted” standard simplex, as illustrated in Figure 3, with:</p>
<ul>
<li>$n = 3$</li>
<li>$ 0.7 \geq w_1 \geq 0.1$</li>
<li>$ 0.8 \geq w_2 \geq 0 $</li>
<li>$ 0.6 \geq w_3 \geq 0.1 $</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-restricted-simplex-borkowski-piepel.png"><img src="/assets/images/blog/random-portfolio-benchmarking-restricted-simplex-borkowski-piepel.png" alt="Example of restricted standard simplex, three-asset universe. Source: Piepel." /></a>
<figcaption>Figure 3. Example of restricted standard simplex, three-asset universe. Source: Piepel.</figcaption>
</figure>
<p>Generating points uniformly at random over a restricted standard simplex is much more complex than generating points uniformly at random over a standard simplex.</p>
<p>Hopefully, this problem has been studied at least since the 1990’s by people working in the statistical domain of the <a href="https://en.wikipedia.org/wiki/Design_of_experiments">design of experiments</a>,
and an algorithm based on the <a href="https://en.wikipedia.org/wiki/Conditional_probability_distribution">conditional distribution method</a> has been published<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">13</a></sup> in 2000.</p>
<h3 id="generation-of-random-portfolios-over-a-convex-polytope">Generation of random portfolios over a convex polytope</h3>
<p>When the constraints imposed on a portfolio consist in generic linear constraints, the subset $C$ is then equal to</p>
\[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } A_e w = b_e, A_i w \leq b_i \}\]
<p>, where:</p>
<ul>
<li>$A_e \in \mathbb{R}^{n_e \times n}$ and $b_e \in \mathbb{R}^{n_e}$, $n_e \geq 1$, represent linear equality constraints</li>
<li>$A_i \in \mathbb{R}^{n_i \times n}$ and $b_i \in \mathbb{R}^{n_i}$, $n_i \geq 1$, represent linear inequality constraints</li>
</ul>
<p>The geometrical object associated to that subset is called a <a href="https://en.wikipedia.org/wiki/Convex_polytope">convex polytope</a>, illustrated in Figure 4, with:</p>
<ul>
<li>$n = 3$</li>
<li>$\sum_{i=1}^n w_i = 1$</li>
<li>$1 \geq w_i \geq 0, i = 1..n$</li>
<li>$w_1 > w_2$</li>
<li>$2w_3 > w_2 > 0.5w_3$</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-general-simplex-tervonen.png"><img src="/assets/images/blog/random-portfolio-benchmarking-general-simplex-tervonen.png" alt="Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al." /></a>
<figcaption>Figure 4. Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al.</figcaption>
</figure>
<p>As one can guess, generating points uniformly at random over a convex polytope is another level higher in terms of complexity, and while some algorithms exist to do so<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">14</a></sup>, they are impractical in high dimension.</p>
<p>From the literature, what is possible to achieve instead is to generate points <em>asymptotically</em> uniformly at random, using <a href="https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo">Markov chain Monte Carlo (MCMC)</a> algorithms
like the Hit-And-Run algorithm<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">15</a></sup>.</p>
<h3 id="generation-of-random-portfolios-over-a-generic-constraint-set">Generation of random portfolios over a generic constraint set</h3>
<p>When the constraints imposed on a portfolio are generic, that is, quadratic (volatility or tracking error constraints) and/or non-convex (threshold constraints) and/or integer (maximum number of assets constraints,
round lot constraints), the final boss has arrived.</p>
<p>With such constraints, the only reasonable<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">16</a></sup> approach in the literature seems to recast the problem of generating random points over the constraint set $C$ as a (penalized) optimization problem and use
<a href="https://en.wikipedia.org/wiki/Genetic_algorithm">genetic algorithms</a> to solve it<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>.</p>
<p>The underlying idea is to randomly<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">17</a></sup> minimize an objective function $f$ essentially made of penalties - the higher the distance of a point $w \in \mathbb{R}^{n}$ from the constraint set $C$, the higher the value of the objective function $f(w)$ -,
so that at optimum, a random point satisfying all the constraints<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">18</a></sup> is found.</p>
<p>Of course, the points generated this way are not generated uniformly at random, but when the constraints are fully generic, that requirement should probably be dropped altogether as mentioned in Dawson and Young<sup id="fnref:9:1" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<h2 id="investment-funds-performance-evaluation-with-random-portfolios">Investment fund’s performance evaluation with random portfolios</h2>
<h3 id="rationale">Rationale</h3>
<p>Evaluating the performance of an investment fund is actually a <a href="https://en.wikipedia.org/wiki/Statistical_hypothesis_testing">statistical hypothesis test</a> in disguise<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>, in which:</p>
<ul>
<li>The null hypothesis is <em>The investment fund’s exhibit no particular “performance” over the considered time period</em></li>
<li>The <a href="https://en.wikipedia.org/wiki/Test_statistic">test statistic</a> is a quantitative measure of “performance” over the considered time period (e.g. annualized return, holding period return, risk-adjusted return…)</li>
<li>The (empirical) distribution of the test statistic under the null hypothesis is computed from a sample made of either
<ul>
<li>One observation - the investment fund’s benchmark</li>
<li>A small number of observations - the investment fund’s peer group</li>
<li>Any desired number of observations - random portfolios generated from the investment fund’s universe of assets and obeying to the fund’s constraints and rules</li>
</ul>
</li>
</ul>
<p>From this perspective, using a benchmark, a peer group or random portfolios for performance evaluation <em>is essentially a choice between sampling approaches</em><sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>Still, random portfolios represent a more rigorous approach to performance evaluation than the two other alternatives, for various reasons highlighted in different papers<sup id="fnref:4:5" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup><sup id="fnref:9:2" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup><sup id="fnref:10:2" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:23:1" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> and summarized by Dawson<sup id="fnref:9:3" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> as follows</p>
<blockquote>
<p>A set of uniformly distributed, stochastically generated, portfolios that by construction incorporate no investment strategy, bias or skill form an effective control set for any portfolio measurement metric. This allows the true value of a strategy or model to be identified. They also provide a mechanism to differentiate between effects due to “market conditions” and effects due to either the management of a portfolio, or the constraints the management is obliged to work within.</p>
</blockquote>
<h3 id="surzs-methodology">Surz’s methodology</h3>
<p>Using random portfolios to evaluate an investment fund’s performance over a considered time period is a two-step <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo</a> simulation process<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">20</a></sup>:</p>
<ol>
<li>
<p>Modeling step</p>
<p>Identify the fund’s main characteristics.</p>
</li>
<li>
<p>Computational step</p>
</li>
</ol>
<ul>
<li>
<p>Random portfolios simulation</p>
<p>Generate random portfolios compatible with the identified fund’s main characteristics and simulate their evolution through the considered time period.</p>
</li>
<li>
<p>Performance evaluation</p>
<p>Determine the level of statistical significance of the fund’s performance over the considered time period using the previously simulated random portfolios.</p>
</li>
</ul>
<h4 id="step-1---identifying-the-funds-main-characteristics">Step 1 - Identifying the fund’s main characteristics</h4>
<p>Dawson<sup id="fnref:9:4" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> note that <em>an investment strategy is essentially a set of constraints [and rules] that are carefully constructed to (hopefully) remove, in aggregate, poor performing regions of the solution space</em><sup id="fnref:9:5" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>. Thus,
it is very important to identify as precisely as possible the main characteristics of the fund, even though <em>this […] information is not always available, as only the manager knows [it] exactly</em><sup id="fnref:23:2" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>.</p>
<p>From a practical perspective, these characteristics need to be provided as<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">21</a></sup>:</p>
<ul>
<li>A universe of $n$ assets modeling the fund’s universe of assets</li>
<li>A constraint set $\mathcal{C} \subset \mathbb{R}^{n}$ modeling the fund’s constraints</li>
<li>An ensemble of decision rules modeling<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">22</a></sup> the fund’s rebalancing rules</li>
</ul>
<p>For examples of this step in different contexts, c.f. Kothari and Warner<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">23</a></sup> and Surz<sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<h4 id="step-2a---simulating-random-portfolios">Step 2a - Simulating random portfolios</h4>
<p>Thanks to the previous step, it becomes possible to simulate thousands of random portfolios that the fund’s manager could have potentially held over the considered time period.</p>
<p>The only difficulty at this point is computational, due to the impact of the constraint set on the algorithmic machinery required to generate portfolio weights at random.</p>
<p>As a side note, and <em>for a fair comparison [with the fund under analysis], transaction costs, as well as all other kinds of costs, must be considered</em><sup id="fnref:10:3" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup> when simulating random portfolios.</p>
<h4 id="step-2b---evaluating-the-funds-performance">Step 2b - Evaluating the fund’s performance</h4>
<p>The ranking of the fund’s performance against the performance of all the simulated random portfolios provides a direct measure of the statistical significance of that fund’s performance<sup id="fnref:4:6" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>Indeed, under the null hypothesis that the investment fund’s exhibit no particular performance over the considered time period, <a href="https://en.wikipedia.org/wiki/P-value">the $p$-value</a> of the statistical hypothesis test
mentioned at the beginning of this section is defined as<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup></p>
\[p = \frac{n_x + 1}{N + 1}\]
<p>, where:</p>
<ul>
<li>$n_x$ is the the number of random portfolios whose performance over the considered time period is as extreme or more extreme than that of the fund under evaluation</li>
<li>$N$ is the number of simulated random portfolios</li>
</ul>
<p>This idea is illustrated in Figure 5, which depicts:</p>
<ul>
<li>The distribution of the holding period return of 1000 random portfolios made of stocks belonging to the S&P 500 and subject to misc. turnover and cardinality constraints (solid line)</li>
<li>The 95th percentile of that distribution (dashed line)</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-performance-evaluation-stein.png"><img src="/assets/images/blog/random-portfolio-benchmarking-performance-evaluation-stein.png" alt="Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein." /></a>
<figcaption>Figure 5. Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein.</figcaption>
</figure>
<p>From this figure, if we were to evaluate the performances of a fund <em>trading these securities and operating under constraints similar to those simulated, [it] would have to obtain a return equal of at least 80%
during the 6 year period for the null of no particular performance to be rejected</em><sup id="fnref:1:2" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> at a 95% confidence level.</p>
<h3 id="variations-on-surzs-methodology">Variations on Surz’s methodology</h3>
<p>Burns<sup id="fnref:10:4" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup> and Lisi<sup id="fnref:23:3" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> both extend Surz’s methodology from one time period to several time periods - which allows <em>to consider the persistence of results over time</em><sup id="fnref:23:4" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> - by describing different ways of
combining individual $p$-values<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>. As a by-product,
Burns<sup id="fnref:10:5" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup><sup id="fnref:26:2" role="doc-noteref"><a href="#fn:26" class="footnote">25</a></sup> empirically demonstrates that Stouffer’s method<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">27</a></sup> for combining $p$-values should be preferred to <a href="https://en.wikipedia.org/wiki/Fisher%27s_method">Fisher’s</a> when evaluating a fund’s performance
over multiple time periods.</p>
<p>Stein<sup id="fnref:1:3" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to replace Surz’s methodology by testing <em>whether the distribution of returns of the [fund under evaluation] is
<a href="https://en.wikipedia.org/wiki/Stochastic_ordering">stochastically greater</a> than that of [a] chosen percentile random fund</em> using the <a href="https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test">Mann–Whitney <em>U</em> test</a>.</p>
<h3 id="caveats">Caveats</h3>
<p>Surz<sup id="fnref:4:7" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> warns that <em>there are many ways to implement a [Monte Carlo simulation] approach [with random portfolios], some better than others, and some worse</em><sup id="fnref:4:8" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>.</p>
<p>In particular, Surz<sup id="fnref:4:9" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> argues that, when selecting a number of assets at random to satisfy portfolio cardinality constraints, it is required to use <em>value-weighted sampling, so that the probability of choosing a given [asset] is proportionate to its outstanding
capitalization</em><sup id="fnref:4:10" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>. Otherwise, some <em>macroeconomic [in]consistency</em><sup id="fnref:4:11" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> would be introduced by the Monte Carlo simulation process, which would ultimately bias the end results. This point is confirmed for both
U.S. and global stocks in Arnott et al.<sup id="fnref:28:1" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>, who show that random portfolios <em>introduce, often unintentionally, value and small cap tilts</em><sup id="fnref:28:2" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup>. Nevertheless, other authors<sup id="fnref:23:5" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup> argue on the contrary that
using equally-weighted sampling is more representative of the behavior of an “unskilled” manager, which is exactly what random portfolios are supposed to model!</p>
<p>More generally, random portfolios might have “unfair” characteristics v.s. the fund under evaluation (higher volatility, higher beta, lower turnover…), which must either be acknowledged or controlled for depending on the exact circumstances.</p>
<h2 id="implementations">Implementations</h2>
<h3 id="implementation-in-portfolio-optimizer">Implementation in <strong>Portfolio Optimizer</strong></h3>
<p>Through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/simulation/random</code></a>, <strong>Portfolio Optimizer</strong> allows to:</p>
<ul>
<li>Generate the weights of a random portfolio subject to general linear inequality constraints</li>
</ul>
<p>Through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/simulation/evolution/random</code></a>, <strong>Portfolio Optimizer</strong> allows to:</p>
<ul>
<li>Simulate the evolution of a random portfolio subject to general linear inequality constraints over a specified time period, with 3 different portfolio rebalancing strategies:
<ul>
<li>Buy and hold</li>
<li>Rebalancing toward the weights of the initial random portfolio</li>
<li>Rebalancing toward the weights of a new random portfolio</li>
</ul>
</li>
</ul>
<h3 id="implementations-elsewhere">Implementations elsewhere</h3>
<p>I am aware of two commercial implementations of random portfolios for performance evaluation as described in this blog post:</p>
<ul>
<li><strong><a href="https://www.portfolioprobe.com/">Portfolio Probe</a></strong>, from Patrick Burns<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">28</a></sup> of <a href="https://www.burns-stat.com/">Burns Statistics</a></li>
<li><strong><a href="https://ppca-inc.com/PODs/pipods.htm">PIPODs</a></strong>, from Ronald Surz<sup id="fnref:21:1" role="doc-noteref"><a href="#fn:21" class="footnote">28</a></sup> of <a href="https://ppca-inc.com/">PPCA Inc.</a>, which seems discontinued</li>
</ul>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="hedge-funds-performance-evaluation">Hedge funds performance evaluation</h3>
<p>The performances of hedge funds are notoriously difficult to evaluate due to the lack of both proper benchmarks and homogeneous peer groups<sup id="fnref:32:1" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>Random portfolios offer a solution to this problem <em>because they can reflect the unique specifications of each individual hedge fund</em><sup id="fnref:32:2" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>More details can be found in Surz<sup id="fnref:32:3" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup> or in Molyboga and Ahelec<sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">29</a></sup>.</p>
<h3 id="investment-strategy-returns-dispersion-analysis">Investment strategy returns dispersion analysis</h3>
<p>Kritzman and Page<sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup> uses random portfolios to compare the relative importance of different investment strategies (investment at global asset class level, at individual stock level, etc.)
and concludes that<sup id="fnref:31:1" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup></p>
<blockquote>
<p>Contrary to perceived doctrine, dispersion around average performance arising from security selection is substantially greater than dispersion around average performance arising from all other investment choices. Moreover, asset allocation, widely considered the most important investment choice, produces the least dispersion; thus, from a normative perspective it is the least important investment choice.</p>
</blockquote>
<p>Figure 6, which quantifies that dispersion around average performance, <em>shows the extent to which a talented investor (top 25th or 5th percentile) could have improved upon average performance by engaging in various
investment choices across a global universe. It also shows how far below average an unlucky investor (bottom 75th or 95th percentile) could have performed, depending on the choice of investment discretion</em><sup id="fnref:31:2" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup>.</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-returns-dispersion-kritzman.png"><img src="/assets/images/blog/random-portfolio-benchmarking-returns-dispersion-kritzman.png" alt="5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page." /></a>
<figcaption>Figure 6. 5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page.</figcaption>
</figure>
<p>While this conclusion has been criticized<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">31</a></sup>, the underlying methodology - returns dispersion analysis - provides valuable insight to investors in that it allows to understand
<em>the potential impact of [any] investment choice, irrespective of investment behavior</em><sup id="fnref:31:3" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup>.</p>
<p>For example, let’s suppose a French investor would like to passively invest in an MSCI World ETF, but is worried by both<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">32</a></sup>:</p>
<ul>
<li>The massive ~70% weight of the United States in that index</li>
<li>The ridiculous ~3% weight of his home country in that index</li>
</ul>
<p>Thus, this investor would rather like to invest in an MSCI World ETF “tilted” away from the United States and toward France, although he has no precise idea about how to implement such a tilt.</p>
<p>Here, returns dispersion analysis through random portfolios can help our investor understand the impact on performances of his proposed tactical deviation from the baseline investment strategy, at least historically<sup id="fnref:38" role="doc-noteref"><a href="#fn:38" class="footnote">33</a></sup>.</p>
<p>For this, Figure 7 depicts the range of annualized returns achieved over the period 31 December 2008 - 29 December 2023 by 10000 random portfolios invested in the 23 countries of the MSCI World index when<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">34</a></sup>:</p>
<ul>
<li>The weights of the United States and France are made randomly varying between 0% and 100%</li>
<li>The weights of all the other countries are kept constant v.s. their reference weights</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-msci-world-france-tilt.png"><img src="/assets/images/blog/random-portfolio-benchmarking-msci-world-france-tilt.png" alt="Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023." /></a>
<figcaption>Figure 7. Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023.</figcaption>
</figure>
<p>It is clear from from Figure 7 that deviating from the MSCI World index by under-weighting the United States and over-weighting France has been a relatively bad strategy over the considered period,
with a median annualized return of ~8.30% compared to an annualized return of ~11.30% for the MSCI World<sup id="fnref:37" role="doc-noteref"><a href="#fn:37" class="footnote">35</a></sup>!</p>
<p>Whether history will repeat itself remains to be seen, but thanks to this historical returns dispersion analysis, our investor is at least informed to take an educated decision re. his proposed investment strategy.</p>
<h3 id="synthetic-benchmark-construction">Synthetic benchmark construction</h3>
<p>Stein<sup id="fnref:1:4" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to use random portfolios in order to construct a “synthetic” benchmark, representative of all possible investment strategies within a given universe
of assets and subject to a given set of constraints and rules.</p>
<p>In more details, Stein<sup id="fnref:1:5" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup> proposes to construct such a benchmark directly from <em>the time series of returns of a [well chosen] single random portfolio</em><sup id="fnref:1:6" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>, which is typically the median<sup id="fnref:4:12" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> random portfolio
w.r.t. a chosen performance measure.</p>
<p>A similar idea is discussed in Lisi<sup id="fnref:23:6" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup>, in which such a benchmark is this time constructed from the cross-sectional returns of the random portfolios.</p>
<p>Figure 8 illustrates Lisi’s methodology; on it, the green line represents a synthetic benchmark made of the 95th percentile<sup id="fnref:45" role="doc-noteref"><a href="#fn:45" class="footnote">36</a></sup> of the distribution of the random portfolios holding period returns at each time $t$</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-lisi.png"><img src="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-lisi.png" alt="Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi." /></a>
<figcaption>Figure 8. Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi.</figcaption>
</figure>
<p>The possibility to create synthetic benchmarks is particularly useful when evaluating the performances of tactical asset allocation (TAA) strategies,
because their allocation might completely change from one period to another, making their comparison with a static benchmark - like a buy and hold portfolio or a 60/40 stock-bond portfolio - difficult to justify <em>a priori</em>.</p>
<p>For example, Figure 9 compares over the period 30th November 2006 - 31th January 2024<sup id="fnref:42" role="doc-noteref"><a href="#fn:42" class="footnote">37</a></sup>:</p>
<ul>
<li>The <em>Global Tactical Asset Allocation (GTTA)</em><sup id="fnref:39" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup> strategy of <a href="https://mebfaber.com/">Mebane Faber</a>, which invests monthly - depending on the quantitative rules described in Faber<sup id="fnref:39:1" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup> - within a five-asset universe made of:
<ul>
<li>U.S. Equities, represented by the SPY ETF</li>
<li>International Equities, represented by the EFA ETF</li>
<li>Intermediate-Term U.S. Treasury Bonds, represented by the IEF ETF</li>
<li>U.S. Real Estate, represented by the VNQ ETF</li>
<li>Global Commodities, represented by the DBC ETF</li>
</ul>
</li>
<li>An (equal-weighted) buy and hold portfolio within Faber’s five-asset universe, which is the benchmark of the <em>GTTA</em> strategy proposed in Faber<sup id="fnref:39:2" role="doc-noteref"><a href="#fn:39" class="footnote">38</a></sup></li>
<li>A synthetic benchmark <em>à la Lisi<sup id="fnref:23:7" role="doc-noteref"><a href="#fn:23" class="footnote">19</a></sup></em> constructed from the median cross-sectional holding period returns of 25000 random portfolios simulated<sup id="fnref:41" role="doc-noteref"><a href="#fn:41" class="footnote">39</a></sup> within Faber’s five-asset universe</li>
</ul>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-gtta.png"><img src="/assets/images/blog/random-portfolio-benchmarking-synthetic-benchmark-gtta.png" alt="Global Tactical Asset Allocation (GTTA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024." /></a>
<figcaption>Figure 9. Global Tactical Asset Allocation (GTTA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024.</figcaption>
</figure>
<p>In Figure 9, the buy and portfolio and the synthetic benchmark both appears to behave quite similarly<sup id="fnref:48" role="doc-noteref"><a href="#fn:48" class="footnote">40</a></sup>, but the synthetic benchmark should theoretically be preferred for performance comparison purposes
because it better reflects the dynamics of the <em>GTTA</em> strategy (varying asset weights, varying exposure) while guaranteeing the complete absence of skill<sup id="fnref:40" role="doc-noteref"><a href="#fn:40" class="footnote">41</a></sup>.</p>
<h3 id="trading-strategy-evaluation">Trading strategy evaluation</h3>
<p>Random portfolios also find applications in evaluating (quantitative) trading strategies.</p>
<p>One of these applications, described in Dawson<sup id="fnref:9:6" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>, consists in evaluating the significance of an alpha signal through the comparison of two different samples of random portfolios:</p>
<blockquote>
<p>It is […] very difficult to show conclusively the effect that a model, strategy […] has on the real investment process. […] The ideal solution would be to generate a set of portfolios, constrained in the same way as the portfolio(s) built with the theory or model under investigation, only without the information produced by the theory. It would then be possible to compare the distributions of the portfolio characteristics with and without the added information from the new theory, giving strong statistical evidence of the effects of the new information.</p>
</blockquote>
<p>As a practical illustration, let’s come back on Faber’s <em>GTTA</em> strategy introduced in the previous subsection and compare:</p>
<ul>
<li>A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested<sup id="fnref:44" role="doc-noteref"><a href="#fn:44" class="footnote">42</a></sup> in all the assets selected for investment by the <em>GTTA</em> strategy rules</li>
<li>A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested<sup id="fnref:44:1" role="doc-noteref"><a href="#fn:44" class="footnote">42</a></sup> in all the assets NOT selected for investment by the <em>GTTA</em> strategy rules</li>
</ul>
<p>The resulting ranges of annualized returns over the period 30th November 2006 - 31th January 2024 are displayed in Figure 10, which highlights a clear under-performance of the second sample of random portfolios.</p>
<figure>
<a href="/assets/images/blog/random-portfolio-benchmarking-evaluation-gtta.png"><img src="/assets/images/blog/random-portfolio-benchmarking-evaluation-gtta.png" alt="Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTTA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024." /></a>
<figcaption>Figure 10. Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTTA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024.</figcaption>
</figure>
<p>This kind of under-performance is exactly what is expected under the hypothesis that the <em>GTTA</em> strategy asset selection rules correspond to a true alpha signal.</p>
<p>Indeed, as Arnott et al.<sup id="fnref:28:3" role="doc-noteref"><a href="#fn:28" class="footnote">26</a></sup> puts it:</p>
<blockquote>
<p>In inverting the strategies, we tacitly examine whether these strategies outperform because they are predicated on meaningful investment theses and deep insights on capital markets, or for reasons unrelated to the investment theses. If the investment beliefs are the source of outperformance, then contradicting those beliefs should lead to underperformance.</p>
</blockquote>
<h3 id="trading-strategy-monitoring">Trading strategy monitoring</h3>
<p>Going beyond trading strategy evaluation, random portfolios can also be used to monitor how the performances of a trading strategy differ between in-sample and out-of-sample periods.</p>
<p>I propose to illustrate the associated process with another tactical asset allocation strategy, the <em>Global Equities Momentum (GEM)</em><sup id="fnref:46" role="doc-noteref"><a href="#fn:46" class="footnote">43</a></sup> strategy of <a href="https://dualmomentum.net/">Gary Antonacci</a>, which invests monthly -
depending on the quantitative rules described in Antonacci<sup id="fnref:46:1" role="doc-noteref"><a href="#fn:46" class="footnote">43</a></sup> - in one asset among a three-asset universe made of:</p>
<ul>
<li>U.S. Equities, represented by the S&P 500 Index</li>
<li>International Equities, represented by the MSCI ACWI ex-USA Index</li>
<li>U.S. Bonds, represented by the Barclays Capital US Aggregate Bond Index</li>
</ul>
<p>Because this strategy was published in 2013, let’s first check <em>GEM</em> performances during the in-sample period 1989-2012 (<a href="https://docs.google.com/spreadsheets/d/1jCLJNg8gHs_e1snq7wAuTP03ZUK_b1rUCXUto0_JRHU/edit?usp=sharing">Google Sheet</a>).</p>
<p>Figure 11 depicts the equity curves of a couple of random portfolios simulated<sup id="fnref:47" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe (in solid) v.s. the GEM equity curve (in dashed) over that period.</p>
<figure>
<a href="/assets/images/blog/gem-random-portfolios.png"><img src="/assets/images/blog/gem-random-portfolios.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012" /></a>
<figcaption>Figure 11. Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012.</figcaption>
</figure>
<p>For the real thing, Figure 12 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated<sup id="fnref:47:1" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe over that period, with the
red line corresponding to the <em>GEM</em> Sharpe Ratio.</p>
<figure>
<a href="/assets/images/blog/gem-sharpe-ratios-in-sample.png"><img src="/assets/images/blog/gem-sharpe-ratios-in-sample.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012" /></a>
<figcaption>Figure 12. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012.</figcaption>
</figure>
<p>Pretty amazing, as the <em>GEM</em> Sharpe Ratio is among the best obtainable Sharpe Ratios over the in-sample period!</p>
<p>Let’s now check <em>GEM</em> performances during the out-of-sample period 2013-October 2020 (<a href="https://docs.google.com/spreadsheets/d/1mzus6uIV9JGwELAO_1sAtv1gB9iAX3FQd5LJJ5P61Go/edit?usp=sharing">Google Sheet</a>).</p>
<p>Figure 13 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated<sup id="fnref:47:2" role="doc-noteref"><a href="#fn:47" class="footnote">44</a></sup> within Antonacci’s three-asset universe over that new period, with the
red line again corresponding to the <em>GEM</em> Sharpe Ratio.</p>
<figure>
<a href="/assets/images/blog/gem-sharpe-ratios-out-of-sample.png"><img src="/assets/images/blog/gem-sharpe-ratios-out-of-sample.png" alt="Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020" /></a>
<figcaption>Figure 13. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020.</figcaption>
</figure>
<p>Pretty blah this time, with the <em>GEM</em> Sharpe Ratio roughly comparable to the median random portfolio Sharpe Ratio, which by definition exhibits no particular skill…</p>
<p>Such a difference in the <em>GEM</em> Sharpe Ratio <em>relative to its simulated peer group</em><sup id="fnref:4:13" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup> between the in-sample period and the out-of-sample period is puzzling, but analyzing a particular tactical asset
allocation strategy is out of scope of this blog post, so that I need to leave the “why” question unanswered.</p>
<p>The same process can be applied to any trading strategy in order to detect a potential shift in that strategy’s performances, so, don’t hesitate to abuse the computing power available with today’s computers!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Dawson<sup id="fnref:9:7" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup> notes that <em>[Monte Carlo analysis] is not a tool that has been readily applied to the investment process in the past, due to the perceived complexity of the problem</em><sup id="fnref:9:8" role="doc-noteref"><a href="#fn:9" class="footnote">8</a></sup>.</p>
<p>Through this blog post, I hope to have demonstrated that random portfolios are not necessarily complex to use, and come with many benefits for performance evaluation.</p>
<p>Feel free to randomly reach out <a href="https://www.linkedin.com/in/roman-rubsamen/">on LinkedIn</a> or <a href="https://twitter.com/portfoliooptim">on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://jpm.pm-research.com/content/32/4/54">Surz, Ronald, A Fresh Look at Investment Performance Evaluation: Unifying Best Practices to Improve Timeliness and Reliability, Journal of Portfolio Management, Vol. 32, No. 4, Summer 2006, pp 54-65</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:4:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:4:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:4:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:4:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:4:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a> <a href="#fnref:4:10" class="reversefootnote" role="doc-backlink">↩<sup>11</sup></a> <a href="#fnref:4:11" class="reversefootnote" role="doc-backlink">↩<sup>12</sup></a> <a href="#fnref:4:12" class="reversefootnote" role="doc-backlink">↩<sup>13</sup></a> <a href="#fnref:4:13" class="reversefootnote" role="doc-backlink">↩<sup>14</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Or a trading strategy, or whatever. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijinvest/3/2/36">Surz, R. J. 1994. Portfolio opportunity distributions: an innovation in performance evaluation. The Journal of Investing, 3(2): 36-41</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://ppca-inc.com/pdf/accurate-benchmarking.pdf">Surz, Ron. Accurate Benchmarking is Gone But Not Forgotten: The Imperative Need to Get Back to Basics, Journal of Performance Measurement, Vol. 11, No. 3, Spring, pp 34-43</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/10293523.2014.11082564">Roberto Stein, Not fooled by randomness: Using random portfolios to analyse investment funds, Investment Analysts Journal, 43:79, 1-15</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:1:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:1:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:1:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>By construction. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>These <em>can be imposed by the firm that offers the funds, for example in terms of the prospectus and investment goals, or self-imposed trading behavior that the manager maintains over his career</em><sup id="fnref:1:7" role="doc-noteref"><a href="#fn:1" class="footnote">5</a></sup>; these can also be imposed by regulatory bodies or stock exchanges. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/B9780750654487500082">Dawson, R. and R. Young: 2003, Near-uniformly Distributed, Stochastically Generated Portfolios. In: S. Satchell and A. Scowcroft (eds.): Advances in Portfolio Construction and Implementation. Butterworth–Heinemann</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:9:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:9:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:9:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:9:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:9:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:9:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:9:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:9:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>The $3$-dimensional geometrical object associated to the subset of $\mathbb{R}^3$ modeling long-only and full investment constraints is <a href="https://en.wikipedia.org/wiki/Simplex#Standard_simplex">the standard 2-simplex in $\mathbb{R}^3$</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:11:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/3-540-36626-1_11">Burns, P. (2007). Random Portfolios for Performance Measurement. In: Kontoghiorghes, E.J., Gatu, C. (eds) Optimisation, Econometric and Financial Analysis. Advances in Computational Management Science, vol 9. Springer, Berlin, Heidelberg</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:10:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:10:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:10:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:10:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>Such an object is also called a unit simplex or the probabilistic simplex. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s10479-009-0567-7">Onn, S., Weissman, I. Generating uniform random vectors over a simplex with implications to the volume of a certain polytope and to multivariate extremes. Ann Oper Res 189, 331–342 (2011)</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0167715299000954?via%3Dihub">Kai-Tai Fang, Zhen-Hai Yang, On uniform design of experiments with restricted mixtures and generation of uniform distribution on some domains, Statistics & Probability Letters, Volume 46, Issue 2, 2000, Pages 113-120</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/03610918408812382">Paul A. Rubin (1984) Generating random points in a polytope, Communications in Statistics - Simulation and Computation, 13:3, 375-396</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221714005396">Gert van Valkenhoef, Tommi Tervonen, Douwe Postmus, Notes on “Hit-And-Run enables efficient weight generation for simulation-based multiple criteria decision analysis”, European Journal of Operational Research, Volume 239, Issue 3, 2014, Pages 865-867</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>The <a href="https://en.wikipedia.org/wiki/Rejection_sampling">rejection method</a> is not a reasonable approach because <em>the probability of a portfolio being accepted is generally extremely small when realistic constraints are in place</em><sup id="fnref:10:6" role="doc-noteref"><a href="#fn:10" class="footnote">10</a></sup>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Due to the nature of genetic algorithms. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>Within a given numerical tolerance. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/14697688.2010.513337">Francesco Lisi (2011) Dicing with the market: randomized procedures for evaluation of mutual funds, Quantitative Finance, 11:2, 163-172</a>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:23:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:23:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:23:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:23:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:23:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:23:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:23:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>Some authors like Kritzman and Page<sup id="fnref:31:4" role="doc-noteref"><a href="#fn:31" class="footnote">30</a></sup> consider the usage of random portfolios to be a bootstrap simulation process, and not a Monte Carlo one. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>To be noted that the universe of assets, constraints and rules might perfectly be time-dependent; for example, at a given point in time, the universe of assets might be completely different from that at an earlier or later point in time. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>The frontier between the universe of assets and the rebalancing rules might not always be perfectly clear; at heart, the rebalancing rules must model the trading behaviour of the fund. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=75871">Kothari, S.P. and Warner, Jerold B., Evaluating Mutual Fund Performance (August 1997)</a>. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijwealthmgmt/7/4/78">Ronald J. Surz, Testing the Hypothesis “Hedge Fund Performance Is Good”, The Journal of Wealth Management, Spring 2005, 7 (4) 78-83</a>. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:32:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:32:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:32:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=630123">Burns, Patrick J., Performance Measurement Via Random Portfolios (December 2, 2004)</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:26:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/39/4/91">Robert D. Arnott, Jason Hsu, Vitali Kalesnik, Phil Tindall, The Surprising Alpha From Malkiel’s Monkey and Upside-Down Strategies, The Journal of Portfolio Management, Summer 2013, 39 (4) 91-105</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:28:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:28:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:28:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://doi.org/10.1093/biomet/asx076">N A Heard, P Rubin-Delanchy, Choosing between methods of combining p-values, Biometrika, Volume 105, Issue 1, March 2018, Pages 239–246</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>I have no affiliation. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:21:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>See <a href="https://doi.org/10.1057/jam.2016.3">Molyboga, M., Ahelec, C. A simulation-based methodology for evaluating hedge fund investments. J Asset Manag 17, 434–452 (2016)</a>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/29/4/11">Kritzman, Mark and Sébastien Page (2003), The Hierarchy of Investment Choice, Journal of Portfolio Management 29, number 4, pages 11-23.</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:31:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:31:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:31:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:31:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/31/1/118">Staub, R. (2004). The Hierarchy of Investment Choice. The Journal of Portfolio Management, 31(1), 118–123</a>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>C.f. <a href="https://www.msci.com/documents/10199/178e6643-6ae6-47b9-82be-e1fc565ededb">the MSCI World index factsheet</a>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:38" role="doc-endnote">
<p>If future asset prices are available, for example thanks to <a href="/blog/bootstrap-simulation-with-portfolio-optimizer-usage-for-financial-planning/">a bootstrap simulation</a>, nothing prevents a returns dispersion analysis to integrate them. <a href="#fnref:38" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>In more details, the methodology is as follows: 1) Gross USD monthly price data for all the 23 countries represented in the MSCI World index has been collected from <a href="https://www.msci.com/end-of-day-data-country">the MSCI website</a>, 2) The MSCI World <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">tracking portfolio</a> - exhibiting a nearly null tracking error - has been computed over the period 31 December 2008 - 29 December 2023, which gives reference weights for the 23 countries represented in the MSCI World (e.g., United States ~50% and France ~5%), 3) Using <strong>Portfolio Optimizer</strong>, the evolution of 10000 random portfolios has been simulated over the period 31 December 2008 - 29 December 2023, these portfolios being a) constrained so that all country weights are positive, sum to one and all country weights except these for the United States and France are kept constant v.s. their reference weights and b) monthly rebalanced toward random portfolios in order to encompass any possible tilting, 4) The annualized return of each of these 10000 random portfolios has been computed. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:37" role="doc-endnote">
<p>Which, in addition, is near the top of the achievable annualized returns! <a href="#fnref:37" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:45" role="doc-endnote">
<p>To be noted that in practice, this 95th quantile would probably need to be replaced by the 50th quantile because <em>the benchmark return [should] always ranks median</em><sup id="fnref:4:14" role="doc-noteref"><a href="#fn:4" class="footnote">1</a></sup>. <a href="#fnref:45" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:42" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:42" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:39" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=962461">Faber, Meb, A Quantitative Approach to Tactical Asset Allocation (February 1, 2013). The Journal of Wealth Management, Spring 2007</a>. <a href="#fnref:39" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:39:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:39:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:41" role="doc-endnote">
<p>In more details, using <strong>Portfolio Optimizer</strong>, the evolution of 25000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a random exposure between 0% and 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. <a href="#fnref:41" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:48" role="doc-endnote">
<p>Which justifies <em>a posteriori</em> the use of the buy and hold portfolio as a benchmark. <a href="#fnref:48" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:40" role="doc-endnote">
<p>As a side note, Figure 9 highlights that an equal-weighted buy and hold portfolio is a tough benchmark to beat, c.f. DeMiguel et al.<sup id="fnref:43" role="doc-noteref"><a href="#fn:43" class="footnote">45</a></sup>! <a href="#fnref:40" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:44" role="doc-endnote">
<p>Long-only and fully invested. <a href="#fnref:44" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:44:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:46" role="doc-endnote">
<p>Gary Antonacci, Dual Momentum Investing: An Innovative Strategy for Higher Returns With Lower Risk <a href="#fnref:46" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:46:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:47" role="doc-endnote">
<p>In more details, using <strong>Portfolio Optimizer</strong>, the evolution of 10000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. <a href="#fnref:47" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:47:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:47:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:43" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=1376199">DeMiguel, Victor and Garlappi, Lorenzo and Uppal, Raman, Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (May 2009). The Review of Financial Studies, Vol. 22, Issue 5, pp. 1915-1953, 2009</a>. <a href="#fnref:43" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.As noted in Surz1, the question “Is [a mutual fund’s]2 performance good?” can only be answered relative to something1, typically by comparing that fund to a benchmark like a financial index or to a peer group. Unfortunately, these two methodologies are not without issues. For example, it is very difficult to create an index captur[ing] the essence of the people, process, and philosophy behind an investment product3 and peer groups have well-known biases1 (classification bias, survivorship bias…). In a series of papers (Surz3, Surz4, Surz1…), Ronald J. Surz proposes an innovation that combines the better aspects of both [methodologies] while eliminating their undesirable properties3. This innovation consists in evaluating a fund against the fund manager’s true opportunity set5, defined as the set of all of the possible portfolios that the manager could have conceivably held following his unique investment process4. In practice, the fund manager’s opportunity set is approximated by the simulation of thousands of random portfolios in the same universe of assets as the one of the fund manager and satisfying the same constraints (long-only, long-short…) and rules (portfolio rebalancing rules…) as those of the fund manager. Then, because these portfolios do not exhibit any particular skill6, they can be used as the control group to test [the] fund manager skill5, thus allowing to apply modern statistics to the problem of performance evaluation4. In this blog post, I will describe how to generate random portfolios, detail Surz’s original methodology as well as some of its variations and illustrate the usage of random portfolios with a couple of examples like the creation of synthetic benchmarks or the evaluation and monitoring of trading strategies. Mathematical preliminaries Definition Let be: $n$ the number of assets in a universe of assets $\mathcal{C} \subset \mathbb{R}^{n}$ a subset of $ \mathbb{R}^{n}$ representing the constraints imposed7 on a fund manager when investing in that universe of assets, for example: Short sale constraints Concentration constraints (assets, sectors, industries…) Leverage constraints Cardinality constraints (i.e., minimum or maximum number of assets constraints) Portfolio volatility constraints Portfolio tracking error constraints … Then, a (constrained) random portfolio in that universe of assets is a vector of portfolio weights $w \in \mathbb{R}^{n}$ generated at random over the set $\mathcal{C}$. Generation at random v.s. generation uniformly at random In the context of performance evaluation, as in Surz’s methodology, it is theoretically preferable that random portfolios are generated uniformly at random over the set $\mathcal{C}$, so as not to introduce any biases8 which would otherwise defeat the purpose of using these portfolios as an unbiased control group3. Figure 1 and Figure 2 illustrate the difference between random portfolios generated at random v.s. uniformly at random in the case of a three-asset universe subject to long-only and full investment constraints9: In Figure 1, the random portfolios are visibly concentrated in the middle of the standard 2-simplex in $\mathbb{R}^3$9 - these random portfolios are NOT generated uniformly at random In Figure 2, the random portfolios seem to be “well spread” over the standard 2-simplex in $\mathbb{R}^3$9 - these random portfolios are generated uniformly at random Figure 1. Random portfolios not generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw. Figure 2. Random portfolios generated uniformly at random over a standard simplex, three-asset universe. Source: Shaw. One important remark, though, is that real-life portfolios are usually binding on at least one of their constraints, so that generating random portfolios biased toward the boundary of the geometrical object associated to the constraints set $\mathcal{C}$ might not be a real problem in practice10. Generation of random portfolios over the standard simplex When the constraints imposed on a portfolio are 1) a full investment constraint and 2) a long-only constraint, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, w_i \geq 0, i = 1..n\}\] and the geometrical object associated to that subset is called11 a standard simplex, already illustrated in Figure 1 and Figure 2 with $n = 3$. Several algorithms exist to generate points uniformly at random over a standard simplex12, among which: An algorithm based on differences of sorted uniform random variables An algorithm based on normalized unit exponential random variables Generation of random portfolios over the restricted standard simplex When the constraints imposed on a portfolio are 1) a full investment constraint, 2) a long-only constraint and 3) minimum/maximum asset weights constraints, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } \sum_{i=1}^n w_i = 1, 1 \geq u_i \geq w_i \geq l_i \geq 0, i = 1..n\}\] , where: $l_1, …, l_n$ represent minimum asset weights constraints $u_1, …, u_n$ represent maximum asset weights constraints The geometrical object associated to that subset is a “restricted” standard simplex, as illustrated in Figure 3, with: $n = 3$ $ 0.7 \geq w_1 \geq 0.1$ $ 0.8 \geq w_2 \geq 0 $ $ 0.6 \geq w_3 \geq 0.1 $ Figure 3. Example of restricted standard simplex, three-asset universe. Source: Piepel. Generating points uniformly at random over a restricted standard simplex is much more complex than generating points uniformly at random over a standard simplex. Hopefully, this problem has been studied at least since the 1990’s by people working in the statistical domain of the design of experiments, and an algorithm based on the conditional distribution method has been published13 in 2000. Generation of random portfolios over a convex polytope When the constraints imposed on a portfolio consist in generic linear constraints, the subset $C$ is then equal to \[C = \{w \in \mathbb{R}^{n} \textrm{ s.t. } A_e w = b_e, A_i w \leq b_i \}\] , where: $A_e \in \mathbb{R}^{n_e \times n}$ and $b_e \in \mathbb{R}^{n_e}$, $n_e \geq 1$, represent linear equality constraints $A_i \in \mathbb{R}^{n_i \times n}$ and $b_i \in \mathbb{R}^{n_i}$, $n_i \geq 1$, represent linear inequality constraints The geometrical object associated to that subset is called a convex polytope, illustrated in Figure 4, with: $n = 3$ $\sum_{i=1}^n w_i = 1$ $1 \geq w_i \geq 0, i = 1..n$ $w_1 > w_2$ $2w_3 > w_2 > 0.5w_3$ Figure 4. Example of standard simplex with additional linear inequality constraints, three-asset universe. Source: Tervonen et al. As one can guess, generating points uniformly at random over a convex polytope is another level higher in terms of complexity, and while some algorithms exist to do so14, they are impractical in high dimension. From the literature, what is possible to achieve instead is to generate points asymptotically uniformly at random, using Markov chain Monte Carlo (MCMC) algorithms like the Hit-And-Run algorithm15. Generation of random portfolios over a generic constraint set When the constraints imposed on a portfolio are generic, that is, quadratic (volatility or tracking error constraints) and/or non-convex (threshold constraints) and/or integer (maximum number of assets constraints, round lot constraints), the final boss has arrived. With such constraints, the only reasonable16 approach in the literature seems to recast the problem of generating random points over the constraint set $C$ as a (penalized) optimization problem and use genetic algorithms to solve it10. The underlying idea is to randomly17 minimize an objective function $f$ essentially made of penalties - the higher the distance of a point $w \in \mathbb{R}^{n}$ from the constraint set $C$, the higher the value of the objective function $f(w)$ -, so that at optimum, a random point satisfying all the constraints18 is found. Of course, the points generated this way are not generated uniformly at random, but when the constraints are fully generic, that requirement should probably be dropped altogether as mentioned in Dawson and Young8. Investment fund’s performance evaluation with random portfolios Rationale Evaluating the performance of an investment fund is actually a statistical hypothesis test in disguise19, in which: The null hypothesis is The investment fund’s exhibit no particular “performance” over the considered time period The test statistic is a quantitative measure of “performance” over the considered time period (e.g. annualized return, holding period return, risk-adjusted return…) The (empirical) distribution of the test statistic under the null hypothesis is computed from a sample made of either One observation - the investment fund’s benchmark A small number of observations - the investment fund’s peer group Any desired number of observations - random portfolios generated from the investment fund’s universe of assets and obeying to the fund’s constraints and rules From this perspective, using a benchmark, a peer group or random portfolios for performance evaluation is essentially a choice between sampling approaches1. Still, random portfolios represent a more rigorous approach to performance evaluation than the two other alternatives, for various reasons highlighted in different papers181019 and summarized by Dawson8 as follows A set of uniformly distributed, stochastically generated, portfolios that by construction incorporate no investment strategy, bias or skill form an effective control set for any portfolio measurement metric. This allows the true value of a strategy or model to be identified. They also provide a mechanism to differentiate between effects due to “market conditions” and effects due to either the management of a portfolio, or the constraints the management is obliged to work within. Surz’s methodology Using random portfolios to evaluate an investment fund’s performance over a considered time period is a two-step Monte Carlo simulation process20: Modeling step Identify the fund’s main characteristics. Computational step Random portfolios simulation Generate random portfolios compatible with the identified fund’s main characteristics and simulate their evolution through the considered time period. Performance evaluation Determine the level of statistical significance of the fund’s performance over the considered time period using the previously simulated random portfolios. Step 1 - Identifying the fund’s main characteristics Dawson8 note that an investment strategy is essentially a set of constraints [and rules] that are carefully constructed to (hopefully) remove, in aggregate, poor performing regions of the solution space8. Thus, it is very important to identify as precisely as possible the main characteristics of the fund, even though this […] information is not always available, as only the manager knows [it] exactly19. From a practical perspective, these characteristics need to be provided as21: A universe of $n$ assets modeling the fund’s universe of assets A constraint set $\mathcal{C} \subset \mathbb{R}^{n}$ modeling the fund’s constraints An ensemble of decision rules modeling22 the fund’s rebalancing rules For examples of this step in different contexts, c.f. Kothari and Warner23 and Surz24. Step 2a - Simulating random portfolios Thanks to the previous step, it becomes possible to simulate thousands of random portfolios that the fund’s manager could have potentially held over the considered time period. The only difficulty at this point is computational, due to the impact of the constraint set on the algorithmic machinery required to generate portfolio weights at random. As a side note, and for a fair comparison [with the fund under analysis], transaction costs, as well as all other kinds of costs, must be considered10 when simulating random portfolios. Step 2b - Evaluating the fund’s performance The ranking of the fund’s performance against the performance of all the simulated random portfolios provides a direct measure of the statistical significance of that fund’s performance1. Indeed, under the null hypothesis that the investment fund’s exhibit no particular performance over the considered time period, the $p$-value of the statistical hypothesis test mentioned at the beginning of this section is defined as25 \[p = \frac{n_x + 1}{N + 1}\] , where: $n_x$ is the the number of random portfolios whose performance over the considered time period is as extreme or more extreme than that of the fund under evaluation $N$ is the number of simulated random portfolios This idea is illustrated in Figure 5, which depicts: The distribution of the holding period return of 1000 random portfolios made of stocks belonging to the S&P 500 and subject to misc. turnover and cardinality constraints (solid line) The 95th percentile of that distribution (dashed line) Figure 5. Distribution of the holding period return of random portfolios simulated from stocks belonging to the S&P 500, 2005 - 2010. Source: Stein. From this figure, if we were to evaluate the performances of a fund trading these securities and operating under constraints similar to those simulated, [it] would have to obtain a return equal of at least 80% during the 6 year period for the null of no particular performance to be rejected5 at a 95% confidence level. Variations on Surz’s methodology Burns1025 and Lisi19 both extend Surz’s methodology from one time period to several time periods - which allows to consider the persistence of results over time19 - by describing different ways of combining individual $p$-values26. As a by-product, Burns1025 empirically demonstrates that Stouffer’s method27 for combining $p$-values should be preferred to Fisher’s when evaluating a fund’s performance over multiple time periods. Stein5 proposes to replace Surz’s methodology by testing whether the distribution of returns of the [fund under evaluation] is stochastically greater than that of [a] chosen percentile random fund using the Mann–Whitney U test. Caveats Surz1 warns that there are many ways to implement a [Monte Carlo simulation] approach [with random portfolios], some better than others, and some worse1. In particular, Surz1 argues that, when selecting a number of assets at random to satisfy portfolio cardinality constraints, it is required to use value-weighted sampling, so that the probability of choosing a given [asset] is proportionate to its outstanding capitalization1. Otherwise, some macroeconomic [in]consistency1 would be introduced by the Monte Carlo simulation process, which would ultimately bias the end results. This point is confirmed for both U.S. and global stocks in Arnott et al.26, who show that random portfolios introduce, often unintentionally, value and small cap tilts26. Nevertheless, other authors19 argue on the contrary that using equally-weighted sampling is more representative of the behavior of an “unskilled” manager, which is exactly what random portfolios are supposed to model! More generally, random portfolios might have “unfair” characteristics v.s. the fund under evaluation (higher volatility, higher beta, lower turnover…), which must either be acknowledged or controlled for depending on the exact circumstances. Implementations Implementation in Portfolio Optimizer Through the endpoint /portfolios/simulation/random, Portfolio Optimizer allows to: Generate the weights of a random portfolio subject to general linear inequality constraints Through the endpoint /portfolios/simulation/evolution/random, Portfolio Optimizer allows to: Simulate the evolution of a random portfolio subject to general linear inequality constraints over a specified time period, with 3 different portfolio rebalancing strategies: Buy and hold Rebalancing toward the weights of the initial random portfolio Rebalancing toward the weights of a new random portfolio Implementations elsewhere I am aware of two commercial implementations of random portfolios for performance evaluation as described in this blog post: Portfolio Probe, from Patrick Burns28 of Burns Statistics PIPODs, from Ronald Surz28 of PPCA Inc., which seems discontinued Examples of usage Hedge funds performance evaluation The performances of hedge funds are notoriously difficult to evaluate due to the lack of both proper benchmarks and homogeneous peer groups24. Random portfolios offer a solution to this problem because they can reflect the unique specifications of each individual hedge fund24. More details can be found in Surz24 or in Molyboga and Ahelec29. Investment strategy returns dispersion analysis Kritzman and Page30 uses random portfolios to compare the relative importance of different investment strategies (investment at global asset class level, at individual stock level, etc.) and concludes that30 Contrary to perceived doctrine, dispersion around average performance arising from security selection is substantially greater than dispersion around average performance arising from all other investment choices. Moreover, asset allocation, widely considered the most important investment choice, produces the least dispersion; thus, from a normative perspective it is the least important investment choice. Figure 6, which quantifies that dispersion around average performance, shows the extent to which a talented investor (top 25th or 5th percentile) could have improved upon average performance by engaging in various investment choices across a global universe. It also shows how far below average an unlucky investor (bottom 75th or 95th percentile) could have performed, depending on the choice of investment discretion30. Figure 6. 5th, 25th, 75th, and 95th percentile of the annualized difference from average performance of random portfolios simulated from misc. global asset classes, 1987-2001. Source: Kritzman and Page. While this conclusion has been criticized31, the underlying methodology - returns dispersion analysis - provides valuable insight to investors in that it allows to understand the potential impact of [any] investment choice, irrespective of investment behavior30. For example, let’s suppose a French investor would like to passively invest in an MSCI World ETF, but is worried by both32: The massive ~70% weight of the United States in that index The ridiculous ~3% weight of his home country in that index Thus, this investor would rather like to invest in an MSCI World ETF “tilted” away from the United States and toward France, although he has no precise idea about how to implement such a tilt. Here, returns dispersion analysis through random portfolios can help our investor understand the impact on performances of his proposed tactical deviation from the baseline investment strategy, at least historically33. For this, Figure 7 depicts the range of annualized returns achieved over the period 31 December 2008 - 29 December 2023 by 10000 random portfolios invested in the 23 countries of the MSCI World index when34: The weights of the United States and France are made randomly varying between 0% and 100% The weights of all the other countries are kept constant v.s. their reference weights Figure 7. Quartiles of the annualized returns of random portfolios from countries included in the MSCI World tilted away from the U.S. and toward France, 31 December 2008 - 29 December 2023. It is clear from from Figure 7 that deviating from the MSCI World index by under-weighting the United States and over-weighting France has been a relatively bad strategy over the considered period, with a median annualized return of ~8.30% compared to an annualized return of ~11.30% for the MSCI World35! Whether history will repeat itself remains to be seen, but thanks to this historical returns dispersion analysis, our investor is at least informed to take an educated decision re. his proposed investment strategy. Synthetic benchmark construction Stein5 proposes to use random portfolios in order to construct a “synthetic” benchmark, representative of all possible investment strategies within a given universe of assets and subject to a given set of constraints and rules. In more details, Stein5 proposes to construct such a benchmark directly from the time series of returns of a [well chosen] single random portfolio5, which is typically the median1 random portfolio w.r.t. a chosen performance measure. A similar idea is discussed in Lisi19, in which such a benchmark is this time constructed from the cross-sectional returns of the random portfolios. Figure 8 illustrates Lisi’s methodology; on it, the green line represents a synthetic benchmark made of the 95th percentile36 of the distribution of the random portfolios holding period returns at each time $t$ Figure 8. Time series of the 95th percentile of cross-sectional holding period returns of random portfolios. Source: Lisi. The possibility to create synthetic benchmarks is particularly useful when evaluating the performances of tactical asset allocation (TAA) strategies, because their allocation might completely change from one period to another, making their comparison with a static benchmark - like a buy and hold portfolio or a 60/40 stock-bond portfolio - difficult to justify a priori. For example, Figure 9 compares over the period 30th November 2006 - 31th January 202437: The Global Tactical Asset Allocation (GTTA)38 strategy of Mebane Faber, which invests monthly - depending on the quantitative rules described in Faber38 - within a five-asset universe made of: U.S. Equities, represented by the SPY ETF International Equities, represented by the EFA ETF Intermediate-Term U.S. Treasury Bonds, represented by the IEF ETF U.S. Real Estate, represented by the VNQ ETF Global Commodities, represented by the DBC ETF An (equal-weighted) buy and hold portfolio within Faber’s five-asset universe, which is the benchmark of the GTTA strategy proposed in Faber38 A synthetic benchmark à la Lisi19 constructed from the median cross-sectional holding period returns of 25000 random portfolios simulated39 within Faber’s five-asset universe Figure 9. Global Tactical Asset Allocation (GTTA) strategy v.s. misc. benchmarks, 30th November 2006 - 31th January 2024. In Figure 9, the buy and portfolio and the synthetic benchmark both appears to behave quite similarly40, but the synthetic benchmark should theoretically be preferred for performance comparison purposes because it better reflects the dynamics of the GTTA strategy (varying asset weights, varying exposure) while guaranteeing the complete absence of skill41. Trading strategy evaluation Random portfolios also find applications in evaluating (quantitative) trading strategies. One of these applications, described in Dawson8, consists in evaluating the significance of an alpha signal through the comparison of two different samples of random portfolios: It is […] very difficult to show conclusively the effect that a model, strategy […] has on the real investment process. […] The ideal solution would be to generate a set of portfolios, constrained in the same way as the portfolio(s) built with the theory or model under investigation, only without the information produced by the theory. It would then be possible to compare the distributions of the portfolio characteristics with and without the added information from the new theory, giving strong statistical evidence of the effects of the new information. As a practical illustration, let’s come back on Faber’s GTTA strategy introduced in the previous subsection and compare: A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested42 in all the assets selected for investment by the GTTA strategy rules A sample of 10000 random portfolios simulated within Faber’s five-asset universe, rebalanced each month toward a random portfolio invested42 in all the assets NOT selected for investment by the GTTA strategy rules The resulting ranges of annualized returns over the period 30th November 2006 - 31th January 2024 are displayed in Figure 10, which highlights a clear under-performance of the second sample of random portfolios. Figure 10. Quartiles of the annualized returns of the Global Tactical Asset Allocation (GTTA) strategy original asset selection rules v.s. inverted asset selection rules using random portfolios for weighting the selected assets, 30th November 2006 - 31th January 2024. This kind of under-performance is exactly what is expected under the hypothesis that the GTTA strategy asset selection rules correspond to a true alpha signal. Indeed, as Arnott et al.26 puts it: In inverting the strategies, we tacitly examine whether these strategies outperform because they are predicated on meaningful investment theses and deep insights on capital markets, or for reasons unrelated to the investment theses. If the investment beliefs are the source of outperformance, then contradicting those beliefs should lead to underperformance. Trading strategy monitoring Going beyond trading strategy evaluation, random portfolios can also be used to monitor how the performances of a trading strategy differ between in-sample and out-of-sample periods. I propose to illustrate the associated process with another tactical asset allocation strategy, the Global Equities Momentum (GEM)43 strategy of Gary Antonacci, which invests monthly - depending on the quantitative rules described in Antonacci43 - in one asset among a three-asset universe made of: U.S. Equities, represented by the S&P 500 Index International Equities, represented by the MSCI ACWI ex-USA Index U.S. Bonds, represented by the Barclays Capital US Aggregate Bond Index Because this strategy was published in 2013, let’s first check GEM performances during the in-sample period 1989-2012 (Google Sheet). Figure 11 depicts the equity curves of a couple of random portfolios simulated44 within Antonacci’s three-asset universe (in solid) v.s. the GEM equity curve (in dashed) over that period. Figure 11. Global Equities Momentum (GEM) strategy v.s. random portfolios, in-sample period 1989-2012. For the real thing, Figure 12 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated44 within Antonacci’s three-asset universe over that period, with the red line corresponding to the GEM Sharpe Ratio. Figure 12. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, in-sample period 1989-2012. Pretty amazing, as the GEM Sharpe Ratio is among the best obtainable Sharpe Ratios over the in-sample period! Let’s now check GEM performances during the out-of-sample period 2013-October 2020 (Google Sheet). Figure 13 depicts the Sharpe Ratio distribution of 10000 random portfolios simulated44 within Antonacci’s three-asset universe over that new period, with the red line again corresponding to the GEM Sharpe Ratio. Figure 13. Global Equities Momentum (GEM) strategy v.s. random portfolios, distribution of Sharpe Ratios, out-of-sample period 2013-October 2020. Pretty blah this time, with the GEM Sharpe Ratio roughly comparable to the median random portfolio Sharpe Ratio, which by definition exhibits no particular skill… Such a difference in the GEM Sharpe Ratio relative to its simulated peer group1 between the in-sample period and the out-of-sample period is puzzling, but analyzing a particular tactical asset allocation strategy is out of scope of this blog post, so that I need to leave the “why” question unanswered. The same process can be applied to any trading strategy in order to detect a potential shift in that strategy’s performances, so, don’t hesitate to abuse the computing power available with today’s computers! Conclusion Dawson8 notes that [Monte Carlo analysis] is not a tool that has been readily applied to the investment process in the past, due to the perceived complexity of the problem8. Through this blog post, I hope to have demonstrated that random portfolios are not necessarily complex to use, and come with many benefits for performance evaluation. Feel free to randomly reach out on LinkedIn or on Twitter. – See Surz, Ronald, A Fresh Look at Investment Performance Evaluation: Unifying Best Practices to Improve Timeliness and Reliability, Journal of Portfolio Management, Vol. 32, No. 4, Summer 2006, pp 54-65. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 Or a trading strategy, or whatever. ↩ See Surz, R. J. 1994. Portfolio opportunity distributions: an innovation in performance evaluation. The Journal of Investing, 3(2): 36-41. ↩ ↩2 ↩3 ↩4 See Surz, Ron. Accurate Benchmarking is Gone But Not Forgotten: The Imperative Need to Get Back to Basics, Journal of Performance Measurement, Vol. 11, No. 3, Spring, pp 34-43. ↩ ↩2 ↩3 See Roberto Stein, Not fooled by randomness: Using random portfolios to analyse investment funds, Investment Analysts Journal, 43:79, 1-15. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 By construction. ↩ These can be imposed by the firm that offers the funds, for example in terms of the prospectus and investment goals, or self-imposed trading behavior that the manager maintains over his career5; these can also be imposed by regulatory bodies or stock exchanges. ↩ See Dawson, R. and R. Young: 2003, Near-uniformly Distributed, Stochastically Generated Portfolios. In: S. Satchell and A. Scowcroft (eds.): Advances in Portfolio Construction and Implementation. Butterworth–Heinemann. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 The $3$-dimensional geometrical object associated to the subset of $\mathbb{R}^3$ modeling long-only and full investment constraints is the standard 2-simplex in $\mathbb{R}^3$. ↩ ↩2 ↩3 See Burns, P. (2007). Random Portfolios for Performance Measurement. In: Kontoghiorghes, E.J., Gatu, C. (eds) Optimisation, Econometric and Financial Analysis. Advances in Computational Management Science, vol 9. Springer, Berlin, Heidelberg. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 Such an object is also called a unit simplex or the probabilistic simplex. ↩ See Onn, S., Weissman, I. Generating uniform random vectors over a simplex with implications to the volume of a certain polytope and to multivariate extremes. Ann Oper Res 189, 331–342 (2011). ↩ See Kai-Tai Fang, Zhen-Hai Yang, On uniform design of experiments with restricted mixtures and generation of uniform distribution on some domains, Statistics & Probability Letters, Volume 46, Issue 2, 2000, Pages 113-120. ↩ See Paul A. Rubin (1984) Generating random points in a polytope, Communications in Statistics - Simulation and Computation, 13:3, 375-396. ↩ See Gert van Valkenhoef, Tommi Tervonen, Douwe Postmus, Notes on “Hit-And-Run enables efficient weight generation for simulation-based multiple criteria decision analysis”, European Journal of Operational Research, Volume 239, Issue 3, 2014, Pages 865-867. ↩ The rejection method is not a reasonable approach because the probability of a portfolio being accepted is generally extremely small when realistic constraints are in place10. ↩ Due to the nature of genetic algorithms. ↩ Within a given numerical tolerance. ↩ See Francesco Lisi (2011) Dicing with the market: randomized procedures for evaluation of mutual funds, Quantitative Finance, 11:2, 163-172. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 Some authors like Kritzman and Page30 consider the usage of random portfolios to be a bootstrap simulation process, and not a Monte Carlo one. ↩ To be noted that the universe of assets, constraints and rules might perfectly be time-dependent; for example, at a given point in time, the universe of assets might be completely different from that at an earlier or later point in time. ↩ The frontier between the universe of assets and the rebalancing rules might not always be perfectly clear; at heart, the rebalancing rules must model the trading behaviour of the fund. ↩ See Kothari, S.P. and Warner, Jerold B., Evaluating Mutual Fund Performance (August 1997). ↩ See Ronald J. Surz, Testing the Hypothesis “Hedge Fund Performance Is Good”, The Journal of Wealth Management, Spring 2005, 7 (4) 78-83. ↩ ↩2 ↩3 ↩4 See Burns, Patrick J., Performance Measurement Via Random Portfolios (December 2, 2004). ↩ ↩2 ↩3 See Robert D. Arnott, Jason Hsu, Vitali Kalesnik, Phil Tindall, The Surprising Alpha From Malkiel’s Monkey and Upside-Down Strategies, The Journal of Portfolio Management, Summer 2013, 39 (4) 91-105. ↩ ↩2 ↩3 ↩4 See N A Heard, P Rubin-Delanchy, Choosing between methods of combining p-values, Biometrika, Volume 105, Issue 1, March 2018, Pages 239–246. ↩ I have no affiliation. ↩ ↩2 See Molyboga, M., Ahelec, C. A simulation-based methodology for evaluating hedge fund investments. J Asset Manag 17, 434–452 (2016). ↩ See Kritzman, Mark and Sébastien Page (2003), The Hierarchy of Investment Choice, Journal of Portfolio Management 29, number 4, pages 11-23.. ↩ ↩2 ↩3 ↩4 ↩5 See Staub, R. (2004). The Hierarchy of Investment Choice. The Journal of Portfolio Management, 31(1), 118–123. ↩ C.f. the MSCI World index factsheet. ↩ If future asset prices are available, for example thanks to a bootstrap simulation, nothing prevents a returns dispersion analysis to integrate them. ↩ In more details, the methodology is as follows: 1) Gross USD monthly price data for all the 23 countries represented in the MSCI World index has been collected from the MSCI website, 2) The MSCI World tracking portfolio - exhibiting a nearly null tracking error - has been computed over the period 31 December 2008 - 29 December 2023, which gives reference weights for the 23 countries represented in the MSCI World (e.g., United States ~50% and France ~5%), 3) Using Portfolio Optimizer, the evolution of 10000 random portfolios has been simulated over the period 31 December 2008 - 29 December 2023, these portfolios being a) constrained so that all country weights are positive, sum to one and all country weights except these for the United States and France are kept constant v.s. their reference weights and b) monthly rebalanced toward random portfolios in order to encompass any possible tilting, 4) The annualized return of each of these 10000 random portfolios has been computed. ↩ Which, in addition, is near the top of the achievable annualized returns! ↩ To be noted that in practice, this 95th quantile would probably need to be replaced by the 50th quantile because the benchmark return [should] always ranks median1. ↩ (Adjusted) prices have have been retrieved using Tiingo. ↩ See Faber, Meb, A Quantitative Approach to Tactical Asset Allocation (February 1, 2013). The Journal of Wealth Management, Spring 2007. ↩ ↩2 ↩3 In more details, using Portfolio Optimizer, the evolution of 25000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a random exposure between 0% and 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. ↩ Which justifies a posteriori the use of the buy and hold portfolio as a benchmark. ↩ As a side note, Figure 9 highlights that an equal-weighted buy and hold portfolio is a tough benchmark to beat, c.f. DeMiguel et al.45! ↩ Long-only and fully invested. ↩ ↩2 Gary Antonacci, Dual Momentum Investing: An Innovative Strategy for Higher Returns With Lower Risk ↩ ↩2 In more details, using Portfolio Optimizer, the evolution of 10000 random portfolios has been simulated over the considered period, these portfolios being a) constrained so that all weights are positive and sum to a 100% and b) monthly rebalanced toward random portfolios in order to encompass any possible tactical allocation. ↩ ↩2 ↩3 See DeMiguel, Victor and Garlappi, Lorenzo and Uppal, Raman, Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (May 2009). The Review of Financial Studies, Vol. 22, Issue 5, pp. 1915-1953, 2009. ↩Sparse Index Tracking: Limiting the Number of Assets in an Index Tracking Portfolio2024-01-08T00:00:00-06:002024-01-08T00:00:00-06:00https://portfoliooptimizer.io/blog/sparse-index-tracking-limiting-the-number-of-assets-in-an-index-tracking-portfolio<p>In the <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">previous post</a>, I introduced the <em>index tracking problem</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup>, which consists
in finding a portfolio that tracks as closely as possible<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup> a given financial market index.</p>
<p>Because such a portfolio might contain any number of assets, with for example an <a href="https://en.wikipedia.org/wiki/S%26P_500">S&P 500</a> tracking portfolio possibly containing ~500 stocks, <em>it is [sometimes desirable]
that the tracking portfolio consists of a small number of assets</em><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup> in order to <em>simplify the execution, avoid small and illiquid positions, and large transaction costs</em><sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In other words, it is sometimes desirable to impose a constraint on the maximum number of assets contained in an index tracking portfolio, which leads to an extension of the regular index tracking problem called the <em>sparse index tracking problem</em><sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">4</a></sup>.</p>
<p>In this new post, I will describe the mathematics of the sparse index tracking problem and I will detail a few examples of usage like:</p>
<ul>
<li>Replicating a fund of funds to reduce fees while keeping the number of directly-invested funds manageable</li>
<li>Replicating the S&P 500 with only ~50 stocks as an (extreme) illustration of the <em>optimized sampling</em><sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">5</a></sup> ETF methodology</li>
</ul>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="the-general-sparse-index-tracking-optimization-problem">The general sparse index tracking optimization problem</h3>
<p>Let be:</p>
<ul>
<li>$T$, a number of time periods</li>
<li>$r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods</li>
<li>$n$, the number of assets in the universe of the sparse index tracking portfolio</li>
<li>$X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods</li>
<li>$w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets</li>
</ul>
<p>The vector of the sparse index tracking portfolio weights $w^*$ is then the<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> solution to the optimization problem which consists in minimizing a tracking error measure
$f \left( X w - r_{idx} \right)$ - with $f$ some <a href="https://en.wikipedia.org/wiki/Loss_function">loss function</a> - subject to additional constraints like full investment constraint, no short sale constraint, etc. and
a cardinality constraint<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup></p>
\[w^* = \operatorname{argmin} f \left( X w - r_{idx} \right) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\]
<p>, where:</p>
<ul>
<li>$\lVert . \rVert_0$ is the zero “norm”<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">7</a></sup> of a vector, that is, the number of non-zero elements in that vector</li>
<li>$n_{max}$ is the maximum number of assets with non-zero weights desired in the sparse index tracking portfolio</li>
</ul>
<p>From a computational perspective, and whatever the exact definition of the loss function $f$, it is known that <em>when a cardinality constraint restricting the number of stocks is introduced, the problem of optimizing the composition
of a portfolio tends to become <a href="https://en.wikipedia.org/wiki/NP-hardness">NP-hard</a></em><sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup>, which <em>means that exact solutions to instances of realistic sizes are computationally intractable, and thus inexact solution methods are the
only practical ones</em><sup id="fnref:8:1" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup>.</p>
<p>Standard quadratic optimization methods guaranteeing an optimal solution are thus unfortunately not usable to solve the sparse index tracking optimization problem<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">9</a></sup>…</p>
<h3 id="heuristics-for-solving-the-general-sparse-index-tracking-optimization-problem">Heuristics for solving the general sparse index tracking optimization problem</h3>
<p>While an exhaustive enumeration of all possible methods for solving the sparse index tracking optimization problem is out of scope of this post, most of them<sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">10</a></sup> seem to fall into one of three rough categories.</p>
<h4 id="combinatorial-optimization-methods">Combinatorial optimization methods</h4>
<p>The cardinality constraint present in the sparse index tracking problem makes it a combinatorial problem, so that standard combinatorial optimization methods (continuous relaxation of a mixed integer programming formulation<sup id="fnref:8:2" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup><sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">11</a></sup>,
genetic algorithms<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">12</a></sup><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">13</a></sup>…) can be used to find an approximate solution.</p>
<p>As a side note, thanks to the relationship of the sparse index tracking problem with combinatorial optimization and hence with <a href="https://en.wikipedia.org/wiki/Operations_research">operations research</a>,
it is possible to find datasets for different index tracking problems - ranging from a universe of 31 assets to a universe of 2151 assets - in the <a href="http://people.brunel.ac.uk/~mastjjb/jeb/orlib/indtrackinfo.html">OR-Library</a>.</p>
<h4 id="signal-processing-methods">Signal processing methods</h4>
<p>The sparse index tracking problem in finance is closely related to a similar problem in signal processing called <a href="https://en.wikipedia.org/wiki/Compressed_sensing">compressed sensing</a>,
for which specific algorithms have been developed over the years, like the iterative hard thresholding algorithm<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup> or the compressive sampling matching pursuit (CoSaMP) algorithm<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">15</a></sup>.</p>
<p>To be noted, though, that all signal processing methods that rely on the <em>least absolute shrinkage and selection operator (<a href="https://en.wikipedia.org/wiki/Lasso_(statistics)">LASSO</a>)</em>, which consists in approximating
the $l_0$-norm constraint by a $l_1$-norm constraint, are not applicable to the sparse index tracking problem. Indeed, when full investment and no-short sales constraints are imposed<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">16</a></sup>,
we have, for any portfolio weights vector $w$</p>
\[\lVert w \rVert_1 = \sum_{i=1}^n \left| w_i \right| = \sum_{i=1}^n w_i = 1\]
<p>, so that <em>the</em> $l_1$<em>-norm reduces to a constant and is therefore irrelevant</em><sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<h4 id="non-linear-optimization-methods">Non-linear optimization methods</h4>
<p>At the crossroads with signal processing methods, generic non-linear optimization methods taking into account cardinality constraints have also been developed over the years:</p>
<ul>
<li>
<p>Methods that directly integrate cardinality constraints, like projected gradient methods<sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">17</a></sup><sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">18</a></sup> or coordinate descent methods<sup id="fnref:15:1" role="doc-noteref"><a href="#fn:15" class="footnote">18</a></sup></p>
</li>
<li>
<p>Methods that indirectly integrate cardinality constraints by either:</p>
<ul>
<li>Relaxing the $l_0$-norm constraint into an “easier” constraint, like the entropic lower bound algorithm of Jacquet et al.<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">19</a></sup></li>
<li>Reformulating the $l_0$-norm constraint as a $l_0$-norm penalty integrated in the tracking error measure<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">20</a></sup> and replacing that $l_0$-norm penalty with an “easier” penalty, like the $q$-norm approximation algorithm of Jansen and van Dijk<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">21</a></sup> or the <a href="https://en.wikipedia.org/wiki/MM_algorithm">majorization-minimization</a> algorithm of Benidis et al.<sup id="fnref:3:4" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup></li>
</ul>
</li>
</ul>
<h3 id="the-sparse-index-trackingempirical-tracking-error-optimization-problem">The sparse index tracking/empirical tracking error optimization problem</h3>
<p>Like in the <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">previous post of this series</a>, this post will use the empirical tracking error as the preferred tracking error measure.</p>
<p>In this case, the general sparse index tracking optimization problem becomes</p>
\[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\]
<p>, which, as noted by Benidis et al.<sup id="fnref:3:5" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, is nothing else than a constrained sparse regression problem:</p>
<blockquote>
<p>the sparse index tracking problem is similar to many sparsity formulations in the signal processing area in the sense that it is a regression problem with some sparsity requirements</p>
</blockquote>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> allows to compute an approximate solution to the sparse index tracking optimization problem under the empirical tracking error measure through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/replication/index-tracking/sparse</code></a>.</p>
<p>In addition, <strong>Portfolio Optimizer</strong> allows to impose misc. constraints like minimum and maximum asset weights or minimum and maximum group weights constraints<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">22</a></sup>.</p>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="replicating-the-sp-500-by-extreme-optimized-sampling">Replicating the S&P 500 by (extreme) optimized sampling</h3>
<p>One of the most immediate application of the sparse index tracking problem is the ability to replicate a financial market index without investing into all its constituents, which is desirable for the reasons explained in the introduction of this post.</p>
<p>In order to illustrate this, I propose to partially replicate one of the numerical experiments in Benidis et al.<sup id="fnref:3:6" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>In details, using the S&P 500 data provided in <a href="https://cran.r-project.org/web/packages/sparseIndexTracking/index.html">the R package <em>sparseIndexTracking</em></a> - which consists in the daily returns of the S&P 500 index and of 386 of its constituents
over the period from 1st January 2010 to 31th December 2010 - I will solve<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following 80 sparse index tracking problems:</p>
<ul>
<li><strong>Index</strong> - S&P 500</li>
<li><strong>Tracking assets</strong> - 386 stocks included in the S&P 500</li>
<li><strong>Maximum number of assets</strong> - Varying from 20 to 100</li>
<li><strong>Misc. constraints</strong> - Maximum asset weight constraint of 5%</li>
</ul>
<p>The empirical tracking error of the resulting 80 sparse S&P 500 tracking portfolios, as well as - for reference - the empirical tracking error of the regular S&P 500 tracking portfolio<sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>, is depicted in Figure 1.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-ete-sp500.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-ete-sp500.png" alt="Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks" /></a>
<figcaption>Figure 1. Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks</figcaption>
</figure>
<p>From this figure, it is visible that a sparse tracking portfolio made of ~100 stocks allows to track the S&P 500 with the same level of tracking error as a regular tracking portfolio made of much more stocks<sup id="fnref:32:1" role="doc-noteref"><a href="#fn:32" class="footnote">24</a></sup>.</p>
<p>But it is possible to do better!</p>
<p>A sparse tracking portfolio made of only ~50 stocks is actually sufficient, as illustrated in Figure 2.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-growth-sp500.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-growth-sp500.png" alt="Sparse S&P 500 55-stock tracking portfolios over the period 01 January 2010 - 31 December 2010, performances comparison with S&P 500" /></a>
<figcaption>Figure 2. Sparse 55-stock S&P 500 tracking portfolio over the period 01 January 2010 - 31 December 2010, performances comparison with the S&P 500</figcaption>
</figure>
<p>Of course, such a small portfolio might be less “diversified”<sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">25</a></sup> than its 100-stock counterpart, but the main takeaway is that a portfolio made of ~50-100 stocks adequately replicates the S&P 500, which opens the door of
<em>direct investing</em><sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">26</a></sup> to DIY investors.</p>
<h3 id="automatically-determining-the-proper-composite-benchmark-for-a-mutual-fund">Automatically determining the proper composite benchmark for a mutual fund</h3>
<p>In <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">the previous post of this series</a>, the index tracking machinery was shown to allow to easily and automatically construct a benchmark for any mutual fund.</p>
<p>By extension, the sparse index tracking machinery also allows to do the same, with an important advantage for multi-asset mutual funds - the automated construction of the best composite benchmark with a given number of constituents.</p>
<h3 id="replicating-a-fund-of-funds-to-reduce-fees">Replicating a fund of funds to reduce fees</h3>
<p>As a last example of usage, I propose to build upon <a href="https://www.linkedin.com/posts/finominal_whats-the-likelihood-that-all-22-funds-in-activity-7112822889518616576-lEu4?utm_source=share&utm_medium=member_desktop">a LinkedIn post</a>
from <a href="https://finominal.com/">Finominal</a>, in which the following question is asked:</p>
<blockquote>
<p>What’s the likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value?</p>
</blockquote>
<p>A bit of context first.</p>
<p>The <em><a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-investor-growth-income-fund-a-4812c2858">JPMorgan Investor Growth & Income Fund</a></em> is a fund of funds (FoF) whose <em>main investment strategy is to invest in other J.P. Morgan Funds</em><sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">27</a></sup>.
As of 31 December 2023, it is invested in 25 such other J.P. Morgan funds (<a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-core-bond-fund-r6-4812c0100"><em>JPMorgan Core Bond R6</em></a>,
<a href="https://am.jpmorgan.com/us/en/asset-management/adv/products/jpmorgan-us-equity-fund-r6-48121l817"><em>JPMorgan US Equity R6</em></a>…).</p>
<p>One problem with FoF is that they usually have higher fees than traditional mutual funds <em>because they include the management fees charged by the underlying funds</em><sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">28</a></sup>.</p>
<p>The <em>JPMorgan Investor Growth & Income Fund</em> is no different, with a total expense ratio ranging from 1.47% to 0.72%, the latter being for its institutional class share (ticker ONGFX).</p>
<p>Inspired by the results from <a href="/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more/">the previous post of this series</a>, it is possible to reduce those fees by investing directly in all the underlying J.P. Morgan funds
in proportion determined by solving<sup id="fnref:35:1" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following regular index tracking problem:</p>
<ul>
<li><strong>Index</strong> - The <em>JPMorgan Investor Growth & Income Fund</em> (ONGFX)</li>
<li><strong>Tracking assets</strong> - 24 over the 25 J.P. Morgan funds held by the <em>JPMorgan Investor Growth & Income Fund</em> as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar<sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup></li>
</ul>
<p>Using price data<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">30</a></sup> for the period 01 January 2023 - 31 December 2023, this gives the regular tracking portfolio whose weights are displayed in Figure 3 and whose performances are compared with the <em>JPMorgan Investor Growth & Income Fund</em> in Figure 4.</p>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-weights-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-weights-jpm.png" alt="JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund" /></a>
<figcaption>Figure 3. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-growth-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-full-replication-growth-jpm.png" alt="JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition" /></a>
<figcaption>Figure 4. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund</figcaption>
</figure>
<p>This tracking portfolio has:</p>
<ul>
<li>A practically null tracking error v.s. the <em>JPMorgan Investor Growth & Income Fund</em> (Figure 4)</li>
<li>An expense ratio of ~0.43%<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">31</a></sup> v.s. of at least 0.72% for the <em>JPMorgan Investor Growth & Income Fund</em></li>
</ul>
<p>So, this tracking portfolio is an interesting alternative to the <em>JPMorgan Investor Growth & Income Fund</em>, but there is a catch - it is invested in a total of 22 J.P. Morgan funds (Figure 3), which might be cumbersome to manage…</p>
<p>An even better tracking portfolio would have a smaller number of directly-invested funds.</p>
<p>Does such a unicorn exist?</p>
<p>Figures 5 and 6 answer this question, by solving<sup id="fnref:35:2" role="doc-noteref"><a href="#fn:35" class="footnote">23</a></sup> the following sequence of sparse index tracking problems over the period 01 January 2023 - 31 December 2023:</p>
<ul>
<li><strong>Index</strong> - The <em>JPMorgan Investor Growth & Income Fund</em> (ONGFX)</li>
<li><strong>Tracking assets</strong> - The 24 non-restricted J.P. Morgan funds held by the <em>JPMorgan Investor Growth & Income Fund</em> as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar<sup id="fnref:27:1" role="doc-noteref"><a href="#fn:27" class="footnote">29</a></sup></li>
<li><strong>Maximum number of assets</strong> - Varying from 1 to 24</li>
</ul>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-growth-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-growth-jpm.png" alt="Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund" /></a>
<figcaption>Figure 5. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/sparse-index-tracking-portfolio-fees-jpm.png"><img src="/assets/images/blog/sparse-index-tracking-portfolio-fees-jpm.png" alt="Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees" /></a>
<figcaption>Figure 6. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees</figcaption>
</figure>
<p>From these two figures, the sweet spot for the number of directly-invested funds in terms of tracking error/fees/manageability seems to be ~6-8.</p>
<p>Coming back to the initial question from Finominal, this analysis confirms that there is <em>basically zero likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value</em>, because these can be drastically reduced<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">32</a></sup>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>That’s it for the first blog post of 2024!</p>
<p>Next in this series, I will discuss another extension of the regular index tracking problem, aiming this time at tracking a financial index with “dynamic” asset weights over the considered $T$ time periods.</p>
<p>As usual, feel free to connect with me <a href="https://www.linkedin.com/in/roman-rubsamen/">on LinkedIn</a> or to follow me <a href="https://twitter.com/portfoliooptim">on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Also called the <em>benchmark replication problem</em>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>In terms of a given tracking error measure, like the <em>empirical tracking error</em><sup id="fnref:3:7" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>, also called the <em>mean squared tracking error</em>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://ieeexplore.ieee.org/document/8384194">Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:3:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:3:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:3:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:3:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>Also called the <em>partial benchmark replication problem</em>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/34/2/143">Kees van Montfort, Elout Visser, Laurens Fijn van Draat, Index Tracking by Means of Optimized Sampling, The Journal of Portfolio Management Winter 2008, 34 (2) 143-152</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Or a solution, in case multiple solutions exist. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>The zero “norm” is not a real norm. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0305054817302265">Purity Mutunge, Dag Haugland, Minimizing the tracking error of cardinality constrained portfolios, Computers & Operations Research, Volume 90, 2018, Pages 33-41</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:8:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:8:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>To be noted that in the specific case of a quadratic loss function, Mutunge and Haugland<sup id="fnref:8:3" role="doc-noteref"><a href="#fn:8" class="footnote">8</a></sup> establish that the sparse index tracking optimization problem is actually <a href="https://en.wikipedia.org/wiki/Strong_NP-completeness">strongly NP-hard</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>Excluding simple heuristics like selecting the largest assets according to a given criteria. For example, <em>a widely used method, especially for a market capitalization weighted index, is to select the largest K assets according to their market capitalizations</em><sup id="fnref:3:8" role="doc-noteref"><a href="#fn:3" class="footnote">3</a></sup>; as another example, greedy or reverse greedy algorithms can be used on top of a non-sparse formulation of the index tracking problem to include or exclude assets one by one<sup id="fnref:19:1" role="doc-noteref"><a href="#fn:19" class="footnote">19</a></sup>. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S037722170800283X">N.A. Canakgoz, J.E. Beasley, Mixed-integer programming approaches for index tracking and enhanced indexation, European Journal of Operational Research, Volume 196, Issue 1, 2009, Pages 384-399</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221702004253">J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-1-4757-5226-7_1">Gilli, M., Kellezi, E. (2002). The Threshold Accepting Heuristic for Index Tracking. In: Pardalos, P.M., Tsitsiringos, V.K. (eds) Financial Engineering, E-commerce and Supply Chain. Applied Optimization, vol 70. Springer, Boston, MA</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1063520309000384">T. Blumensath and M. Davies, Iterative hard thresholding for compressed sensing, Appl. Comput. Harmon. Anal., 27 (2009), pp. 265–274.</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S1063520308000638">Deanna Needell and Joel A Tropp. Cosamp: iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301–321, 2009</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>Which is usually the case in practice… <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/s11425-016-9124-0">Xu, F., Dai, Y., Zhao, Z. et al. Efficient projected gradient methods for cardinality constrained optimization. Sci. China Math. 62, 245–268 (2019).</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://epubs.siam.org/doi/abs/10.1137/120869778?journalCode=sjope8">Amir Beck, Yonina C. Eldar, Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms, 2013, SIAM Journal on Optimization, 1480-1509, 23, 3</a> <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:15:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See <a href="https://inria.hal.science/hal-03874638/">Quentin Jacquet, Agnes Bialecki, Laurent El Ghaoui, Stéphane Gaubert, Riadh Zorgati. Entropic Lower Bound of Cardinality for Sparse Optimization. 2022.</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:19:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>It is not trivial to show that such a reformulation is equivalent; c.f. Jansen and van Dijk<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">21</a></sup> for a detailed proof. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/28/2/33">Roel Jansen, Ronald van Dijk, Optimal Benchmark Tracking with Small Portfolios, The Journal of Portfolio Management, Winter 2002, 28 (2) 33-39</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>The possibility to include minimum and maximum group weights constraints is important in practice, because it allows for example to set an upper limit on fees or to (try to) impose sector neutrality in the computed portfolio, a characteristic often overlooked c.f. for example Che et al.<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">33</a></sup> <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>Using <strong>Portfolio Optimizer</strong>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:35:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:35:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>The non-sparse S&P 500 tracking portfolio, subject to the same maximum asset weight constraint of 5% as the sparse S&P 500 tracking portfolios, is made of 286 stocks. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:32:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>For example, subject to sector concentration bias, as highlighted in Che et al.<sup id="fnref:22:1" role="doc-noteref"><a href="#fn:22" class="footnote">33</a></sup>. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>Direct investing solutions are usually provided by asset management firms (<a href="https://www.fidelity.com/learning-center/trading-investing/direct-indexing">Fidelity</a>, <a href="https://frec.com/">Frec</a>…). In the case of a stock market index, investing directly in stocks composing that stock market index, as opposed to indirectly through an ETF, has several advantages (null expense ratio, possibility to do tax-loss harvesting…). <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>C.f. the <a href="https://am.jpmorgan.com/JPMorgan/TADF/4812C2858/SP?site=JPMorgan">JPMorgan Investor Growth & Income Fund prospectus</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>C.f. <a href="https://en.wikipedia.org/wiki/Fund_of_funds">Wikipedia</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>C.f. <a href="https://www.morningstar.com/funds/xnas/ONGFX/quote">Morningstar</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:27:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>(Adjusted) prices have have been retrieved using <a href="https://api.tiingo.com/">Tiingo</a>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>Computation made using the expense ratios reported by Morningstar for the different J.P. Morgan funds, you will have to trust me on this one. To also be noted that the fees associated to the potential rebalancing of the tracking portfolio are not taken into account and would need to be properly minimized in real life. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>In view of Figure 5 and Figure 6, I would take the <em>returns increased</em> part of <a href="https://www.linkedin.com/posts/finominal_whats-the-likelihood-that-all-22-funds-in-activity-7112822889518616576-lEu4?utm_source=share&utm_medium=member_desktop">Finominal’s LinkedIn post</a> with a grain of salt, although it is certain that returns will increase thanks to lowering the fees. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>See <a href="https://doi.org/10.3390/math10152645">Che, Y.; Chen, S.; Liu, X. Sparse Index Tracking Portfolio with Sector Neutrality. Mathematics 2022, 10, 2645.</a>. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:22:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
</ol>
</div>Roman R.In the previous post, I introduced the index tracking problem1, which consists in finding a portfolio that tracks as closely as possible2 a given financial market index. Because such a portfolio might contain any number of assets, with for example an S&P 500 tracking portfolio possibly containing ~500 stocks, it is [sometimes desirable] that the tracking portfolio consists of a small number of assets3 in order to simplify the execution, avoid small and illiquid positions, and large transaction costs3. In other words, it is sometimes desirable to impose a constraint on the maximum number of assets contained in an index tracking portfolio, which leads to an extension of the regular index tracking problem called the sparse index tracking problem4. In this new post, I will describe the mathematics of the sparse index tracking problem and I will detail a few examples of usage like: Replicating a fund of funds to reduce fees while keeping the number of directly-invested funds manageable Replicating the S&P 500 with only ~50 stocks as an (extreme) illustration of the optimized sampling5 ETF methodology Mathematical preliminaries The general sparse index tracking optimization problem Let be: $T$, a number of time periods $r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods $n$, the number of assets in the universe of the sparse index tracking portfolio $X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods $w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets The vector of the sparse index tracking portfolio weights $w^*$ is then the6 solution to the optimization problem which consists in minimizing a tracking error measure $f \left( X w - r_{idx} \right)$ - with $f$ some loss function - subject to additional constraints like full investment constraint, no short sale constraint, etc. and a cardinality constraint3 \[w^* = \operatorname{argmin} f \left( X w - r_{idx} \right) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\] , where: $\lVert . \rVert_0$ is the zero “norm”7 of a vector, that is, the number of non-zero elements in that vector $n_{max}$ is the maximum number of assets with non-zero weights desired in the sparse index tracking portfolio From a computational perspective, and whatever the exact definition of the loss function $f$, it is known that when a cardinality constraint restricting the number of stocks is introduced, the problem of optimizing the composition of a portfolio tends to become NP-hard8, which means that exact solutions to instances of realistic sizes are computationally intractable, and thus inexact solution methods are the only practical ones8. Standard quadratic optimization methods guaranteeing an optimal solution are thus unfortunately not usable to solve the sparse index tracking optimization problem9… Heuristics for solving the general sparse index tracking optimization problem While an exhaustive enumeration of all possible methods for solving the sparse index tracking optimization problem is out of scope of this post, most of them10 seem to fall into one of three rough categories. Combinatorial optimization methods The cardinality constraint present in the sparse index tracking problem makes it a combinatorial problem, so that standard combinatorial optimization methods (continuous relaxation of a mixed integer programming formulation811, genetic algorithms1213…) can be used to find an approximate solution. As a side note, thanks to the relationship of the sparse index tracking problem with combinatorial optimization and hence with operations research, it is possible to find datasets for different index tracking problems - ranging from a universe of 31 assets to a universe of 2151 assets - in the OR-Library. Signal processing methods The sparse index tracking problem in finance is closely related to a similar problem in signal processing called compressed sensing, for which specific algorithms have been developed over the years, like the iterative hard thresholding algorithm14 or the compressive sampling matching pursuit (CoSaMP) algorithm15. To be noted, though, that all signal processing methods that rely on the least absolute shrinkage and selection operator (LASSO), which consists in approximating the $l_0$-norm constraint by a $l_1$-norm constraint, are not applicable to the sparse index tracking problem. Indeed, when full investment and no-short sales constraints are imposed16, we have, for any portfolio weights vector $w$ \[\lVert w \rVert_1 = \sum_{i=1}^n \left| w_i \right| = \sum_{i=1}^n w_i = 1\] , so that the $l_1$-norm reduces to a constant and is therefore irrelevant3. Non-linear optimization methods At the crossroads with signal processing methods, generic non-linear optimization methods taking into account cardinality constraints have also been developed over the years: Methods that directly integrate cardinality constraints, like projected gradient methods1718 or coordinate descent methods18 Methods that indirectly integrate cardinality constraints by either: Relaxing the $l_0$-norm constraint into an “easier” constraint, like the entropic lower bound algorithm of Jacquet et al.19 Reformulating the $l_0$-norm constraint as a $l_0$-norm penalty integrated in the tracking error measure20 and replacing that $l_0$-norm penalty with an “easier” penalty, like the $q$-norm approximation algorithm of Jansen and van Dijk21 or the majorization-minimization algorithm of Benidis et al.3 The sparse index tracking/empirical tracking error optimization problem Like in the previous post of this series, this post will use the empirical tracking error as the preferred tracking error measure. In this case, the general sparse index tracking optimization problem becomes \[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \newline \lVert w \rVert_0 \le n_{max} \end{cases}\] , which, as noted by Benidis et al.3, is nothing else than a constrained sparse regression problem: the sparse index tracking problem is similar to many sparsity formulations in the signal processing area in the sense that it is a regression problem with some sparsity requirements Implementation in Portfolio Optimizer Portfolio Optimizer allows to compute an approximate solution to the sparse index tracking optimization problem under the empirical tracking error measure through the endpoint /portfolios/replication/index-tracking/sparse. In addition, Portfolio Optimizer allows to impose misc. constraints like minimum and maximum asset weights or minimum and maximum group weights constraints22. Examples of usage Replicating the S&P 500 by (extreme) optimized sampling One of the most immediate application of the sparse index tracking problem is the ability to replicate a financial market index without investing into all its constituents, which is desirable for the reasons explained in the introduction of this post. In order to illustrate this, I propose to partially replicate one of the numerical experiments in Benidis et al.3. In details, using the S&P 500 data provided in the R package sparseIndexTracking - which consists in the daily returns of the S&P 500 index and of 386 of its constituents over the period from 1st January 2010 to 31th December 2010 - I will solve23 the following 80 sparse index tracking problems: Index - S&P 500 Tracking assets - 386 stocks included in the S&P 500 Maximum number of assets - Varying from 20 to 100 Misc. constraints - Maximum asset weight constraint of 5% The empirical tracking error of the resulting 80 sparse S&P 500 tracking portfolios, as well as - for reference - the empirical tracking error of the regular S&P 500 tracking portfolio24, is depicted in Figure 1. Figure 1. Sparse S&P 500 tracking portfolios over the period 01 January 2010 - 31 December 2010, empirical tracking error v.s. maximum number of stocks From this figure, it is visible that a sparse tracking portfolio made of ~100 stocks allows to track the S&P 500 with the same level of tracking error as a regular tracking portfolio made of much more stocks24. But it is possible to do better! A sparse tracking portfolio made of only ~50 stocks is actually sufficient, as illustrated in Figure 2. Figure 2. Sparse 55-stock S&P 500 tracking portfolio over the period 01 January 2010 - 31 December 2010, performances comparison with the S&P 500 Of course, such a small portfolio might be less “diversified”25 than its 100-stock counterpart, but the main takeaway is that a portfolio made of ~50-100 stocks adequately replicates the S&P 500, which opens the door of direct investing26 to DIY investors. Automatically determining the proper composite benchmark for a mutual fund In the previous post of this series, the index tracking machinery was shown to allow to easily and automatically construct a benchmark for any mutual fund. By extension, the sparse index tracking machinery also allows to do the same, with an important advantage for multi-asset mutual funds - the automated construction of the best composite benchmark with a given number of constituents. Replicating a fund of funds to reduce fees As a last example of usage, I propose to build upon a LinkedIn post from Finominal, in which the following question is asked: What’s the likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value? A bit of context first. The JPMorgan Investor Growth & Income Fund is a fund of funds (FoF) whose main investment strategy is to invest in other J.P. Morgan Funds27. As of 31 December 2023, it is invested in 25 such other J.P. Morgan funds (JPMorgan Core Bond R6, JPMorgan US Equity R6…). One problem with FoF is that they usually have higher fees than traditional mutual funds because they include the management fees charged by the underlying funds28. The JPMorgan Investor Growth & Income Fund is no different, with a total expense ratio ranging from 1.47% to 0.72%, the latter being for its institutional class share (ticker ONGFX). Inspired by the results from the previous post of this series, it is possible to reduce those fees by investing directly in all the underlying J.P. Morgan funds in proportion determined by solving23 the following regular index tracking problem: Index - The JPMorgan Investor Growth & Income Fund (ONGFX) Tracking assets - 24 over the 25 J.P. Morgan funds held by the JPMorgan Investor Growth & Income Fund as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar29 Using price data30 for the period 01 January 2023 - 31 December 2023, this gives the regular tracking portfolio whose weights are displayed in Figure 3 and whose performances are compared with the JPMorgan Investor Growth & Income Fund in Figure 4. Figure 3. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, composition Figure 4. JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund This tracking portfolio has: A practically null tracking error v.s. the JPMorgan Investor Growth & Income Fund (Figure 4) An expense ratio of ~0.43%31 v.s. of at least 0.72% for the JPMorgan Investor Growth & Income Fund So, this tracking portfolio is an interesting alternative to the JPMorgan Investor Growth & Income Fund, but there is a catch - it is invested in a total of 22 J.P. Morgan funds (Figure 3), which might be cumbersome to manage… An even better tracking portfolio would have a smaller number of directly-invested funds. Does such a unicorn exist? Figures 5 and 6 answer this question, by solving23 the following sequence of sparse index tracking problems over the period 01 January 2023 - 31 December 2023: Index - The JPMorgan Investor Growth & Income Fund (ONGFX) Tracking assets - The 24 non-restricted J.P. Morgan funds held by the JPMorgan Investor Growth & Income Fund as of 31 December 2023, knowing that the 25th fund is marked as restricted for investment by Morningstar29 Maximum number of assets - Varying from 1 to 24 Figure 5. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, performances comparison with JPMorgan Investor Growth & Income Fund Figure 6. Sparse JPMorgan Investor Growth & Income Fund tracking portfolio over the period 01 January 2023 - 31 December 2023, fees From these two figures, the sweet spot for the number of directly-invested funds in terms of tracking error/fees/manageability seems to be ~6-8. Coming back to the initial question from Finominal, this analysis confirms that there is basically zero likelihood that all […] funds in JP’s Investor Growth & Income Fund contribute value, because these can be drastically reduced32. Conclusion That’s it for the first blog post of 2024! Next in this series, I will discuss another extension of the regular index tracking problem, aiming this time at tracking a financial index with “dynamic” asset weights over the considered $T$ time periods. As usual, feel free to connect with me on LinkedIn or to follow me on Twitter. – Also called the benchmark replication problem. ↩ In terms of a given tracking error measure, like the empirical tracking error3, also called the mean squared tracking error. ↩ See Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 Also called the partial benchmark replication problem. ↩ See Kees van Montfort, Elout Visser, Laurens Fijn van Draat, Index Tracking by Means of Optimized Sampling, The Journal of Portfolio Management Winter 2008, 34 (2) 143-152. ↩ Or a solution, in case multiple solutions exist. ↩ The zero “norm” is not a real norm. ↩ See Purity Mutunge, Dag Haugland, Minimizing the tracking error of cardinality constrained portfolios, Computers & Operations Research, Volume 90, 2018, Pages 33-41. ↩ ↩2 ↩3 To be noted that in the specific case of a quadratic loss function, Mutunge and Haugland8 establish that the sparse index tracking optimization problem is actually strongly NP-hard. ↩ Excluding simple heuristics like selecting the largest assets according to a given criteria. For example, a widely used method, especially for a market capitalization weighted index, is to select the largest K assets according to their market capitalizations3; as another example, greedy or reverse greedy algorithms can be used on top of a non-sparse formulation of the index tracking problem to include or exclude assets one by one19. ↩ See N.A. Canakgoz, J.E. Beasley, Mixed-integer programming approaches for index tracking and enhanced indexation, European Journal of Operational Research, Volume 196, Issue 1, 2009, Pages 384-399. ↩ See J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643. ↩ See Gilli, M., Kellezi, E. (2002). The Threshold Accepting Heuristic for Index Tracking. In: Pardalos, P.M., Tsitsiringos, V.K. (eds) Financial Engineering, E-commerce and Supply Chain. Applied Optimization, vol 70. Springer, Boston, MA. ↩ See T. Blumensath and M. Davies, Iterative hard thresholding for compressed sensing, Appl. Comput. Harmon. Anal., 27 (2009), pp. 265–274.. ↩ See Deanna Needell and Joel A Tropp. Cosamp: iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301–321, 2009. ↩ Which is usually the case in practice… ↩ See Xu, F., Dai, Y., Zhao, Z. et al. Efficient projected gradient methods for cardinality constrained optimization. Sci. China Math. 62, 245–268 (2019).. ↩ See Amir Beck, Yonina C. Eldar, Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms, 2013, SIAM Journal on Optimization, 1480-1509, 23, 3 ↩ ↩2 See Quentin Jacquet, Agnes Bialecki, Laurent El Ghaoui, Stéphane Gaubert, Riadh Zorgati. Entropic Lower Bound of Cardinality for Sparse Optimization. 2022.. ↩ ↩2 It is not trivial to show that such a reformulation is equivalent; c.f. Jansen and van Dijk21 for a detailed proof. ↩ See Roel Jansen, Ronald van Dijk, Optimal Benchmark Tracking with Small Portfolios, The Journal of Portfolio Management, Winter 2002, 28 (2) 33-39. ↩ ↩2 The possibility to include minimum and maximum group weights constraints is important in practice, because it allows for example to set an upper limit on fees or to (try to) impose sector neutrality in the computed portfolio, a characteristic often overlooked c.f. for example Che et al.33 ↩ Using Portfolio Optimizer. ↩ ↩2 ↩3 The non-sparse S&P 500 tracking portfolio, subject to the same maximum asset weight constraint of 5% as the sparse S&P 500 tracking portfolios, is made of 286 stocks. ↩ ↩2 For example, subject to sector concentration bias, as highlighted in Che et al.33. ↩ Direct investing solutions are usually provided by asset management firms (Fidelity, Frec…). In the case of a stock market index, investing directly in stocks composing that stock market index, as opposed to indirectly through an ETF, has several advantages (null expense ratio, possibility to do tax-loss harvesting…). ↩ C.f. the JPMorgan Investor Growth & Income Fund prospectus. ↩ C.f. Wikipedia. ↩ C.f. Morningstar. ↩ ↩2 (Adjusted) prices have have been retrieved using Tiingo. ↩ Computation made using the expense ratios reported by Morningstar for the different J.P. Morgan funds, you will have to trust me on this one. To also be noted that the fees associated to the potential rebalancing of the tracking portfolio are not taken into account and would need to be properly minimized in real life. ↩ In view of Figure 5 and Figure 6, I would take the returns increased part of Finominal’s LinkedIn post with a grain of salt, although it is certain that returns will increase thanks to lowering the fees. ↩ See Che, Y.; Chen, S.; Liu, X. Sparse Index Tracking Portfolio with Sector Neutrality. Mathematics 2022, 10, 2645.. ↩ ↩2Beyond Modified Value-at-Risk: Application of Gaussian Mixtures to the Computation of Value-at-Risk2023-12-18T00:00:00-06:002023-12-18T00:00:00-06:00https://portfoliooptimizer.io/blog/beyond-modified-value-at-risk-application-of-gaussian-mixtures-to-the-computation-of-value-at-risk<p>In <a href="/blog/corrected-cornish-fisher-expansion-improving-the-accuracy-of-modified-value-at-risk/">a previous post</a>, I described a parametric approach to computing Value-at-Risk (VaR) -
called <em>modified VaR</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup><sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">2</a></sup> - that adjusts Gaussian VaR for asymmetry and fat tails present in financial asset returns<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">3</a></sup> thanks to the usage of a
<a href="https://en.wikipedia.org/wiki/Cornish%E2%80%93Fisher_expansion">Cornish–Fisher expansion</a>.</p>
<p>Modified VaR, when properly used<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">4</a></sup>, provides accurate estimates of the VaR for a wide range of non-normal portfolio return distributions. Unfortunately, for mathematical reasons, it cannot be computed for
an even wider range of such distributions<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">5</a></sup>, which poses an issue in practice.</p>
<p>In this blog post, I will describe another parametric approach to computing VaR - called <em>Gaussian mixture Value-at-Risk</em> for the lack of a better name - which
this time adjusts Gaussian VaR for higher moments thanks to the usage of a Gaussian mixture distribution and is free of any computational restriction.</p>
<p>After providing the formula for the Gaussian mixture VaR and explaining how to fit a Gaussian mixture distribution to an empirical return distribution, I will show how to compute the Value-at-Risk of Bitcoin
under the Gaussian mixture model described in the <a href="https://www.blackrock.com/">BlackRock</a> paper <em>Asset Allocation with Crypto: Application of Preferences for Positive Skewness</em> from Ang et al.<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup></p>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="univariate-gaussian-mixture-distribution">Univariate Gaussian mixture distribution</h3>
<p>A random variable $X$ is said to follow a univariate Gaussian mixture distribution, written as $X \sim \mathcal{GM} \left( \left( \mu_i, \sigma_i, p_i \right)_{i=1}^k \right) $,
if its cumulative distribution function (c.d.f.) $F_X$ is of the form<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">7</a></sup></p>
\[F_X(x) = \sum_{i=1}^k p_i \Phi \left( \frac{x - \mu_j}{\sigma_j} \right)\]
<p>, where:</p>
<ul>
<li>$k \geq 1$ is the (integer) number of mixture components</li>
<li>$p_i \in \left[ 0,1 \right]$, $i=1..k$, are the probabilities of the mixture components, with $\sum_{i=1}^kp_i = 1$</li>
<li>$\mu_i \in \mathbb{R}$, $i=1..k$, are the means of the mixture components</li>
<li>$\sigma_i \gt 0$, $i=1..k$, are the standard deviations of the mixture components</li>
<li>$\Phi$ is the c.d.f. of the standard normal distribution</li>
</ul>
<h3 id="reminder-on-value-at-risk">Reminder on Value-at-Risk</h3>
<p>The (percentage) VaR of a portfolio of financial assets corresponds to the percentage of portfolio wealth that can be lost over a certain time horizon and with a certain probability.</p>
<p>Formally, the VaR $VaR_{\alpha}$ of a portfolio over a time horizon $T$ (1 day, 10 days…) and at a confidence level $\alpha$% $\in ]0,1[$ (95%, 97.5%, 99%…) can be defined as<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">8</a></sup></p>
\[\text{VaR}_{\alpha} (X) = - F_X^{-1}(1 - \alpha)\]
<p>, where:</p>
<ul>
<li>$X$ is a random variable representing the portfolio return over the time horizon $T$</li>
<li>$F_X^{-1}$ is the inverse c.d.f. - also called the <a href="https://en.wikipedia.org/wiki/Quantile_function">quantile function</a> - of the random variable $X$</li>
</ul>
<h2 id="gaussian-mixture-value-at-risk">Gaussian mixture Value-at-Risk</h2>
<h3 id="definition">Definition</h3>
<p>When the return distribution of a portfolio is approximated by a Gaussian mixture distribution, that is, when $X \sim \mathcal{GM} \left( \left( \mu_i, \sigma_i, p_i \right)_{i=1}^k \right) $ with $X$ a
random variable representing the portfolio return over a given time horizon $T$, the resulting parametric VaR can be called <em>Gaussian mixture Value-at-Risk</em> (GmVaR).</p>
<p>Mathematically, the Gaussian mixture VaR over the chosen time horizon $T$ and at a confidence level $\alpha$% $GmVaR_{\alpha}$ is implicitly defined by the following equation<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">9</a></sup>,
obtained by mixing together the formula for the VaR of a portfolio and the formula for the c.d.f. of a Gaussian mixture distribution</p>
\[\sum_{i=1}^k p_i \Phi \left( - \frac{ GmVaR_{\alpha} + \mu_i }{\sigma_i} \right) = 1 - \alpha\]
<h3 id="computation">Computation</h3>
<p>Because the formula of the previous subsection is implicit in $GmVaR_{\alpha}$, the effective computation of the Gaussian mixture VaR must involve a numerical procedure such as:</p>
<ul>
<li>A non-linear least-squares minimization procedure<sup id="fnref:10:1" role="doc-noteref"><a href="#fn:10" class="footnote">9</a></sup></li>
<li>A bisection procedure<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">10</a></sup></li>
<li>Or more generally, a procedure to compute a quantile function from a closed-form c.d.f.</li>
</ul>
<h3 id="rationale">Rationale</h3>
<p>It has been observed since the 1970s<sup id="fnref:12" role="doc-noteref"><a href="#fn:12" class="footnote">11</a></sup> that Gaussian mixture distributions adequately approximate the unconditional distributions of financial asset returns. For example,
Kon<sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">12</a></sup> shows that two, three and four-component Gaussian mixture distributions can be used to model the daily returns of 3 U.S. stock market indexes and of 30 Dow Jones stocks. Similarly,
Cuevas-Covarrubias et al.<sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">13</a></sup> shows how two-component Gaussian mixture distributions exhibit <em>a natural capacity to fit leptokurtic distributions</em><sup id="fnref:17:1" role="doc-noteref"><a href="#fn:17" class="footnote">13</a></sup> using the daily returns of 3 Mexican stocks.</p>
<p>Compared to alternative distributions, Gaussian mixture distributions <em>offer a versatile combination of precision and simplicity</em><sup id="fnref:17:2" role="doc-noteref"><a href="#fn:17" class="footnote">13</a></sup> for modeling asset returns. Indeed,
Gaussian mixture distributions are both flexible enough to handle arbitrary asset return distributions<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup> and simple enough to be numerically tractable<sup id="fnref:16:1" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup>. In contrast,
<em>[alternative distributions] tend to suffer from one of two evils: either they are too restrictive in the variety of p.d.f. shapes that can be achieved,
or not restrictive enough, in the sense that they have too many degrees of freedom for calibration to be feasible</em><sup id="fnref:16:2" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup>.</p>
<p>As for applications of Gaussian mixture distributions to VaR, Hull and White<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">15</a></sup> used a two-component Gaussian mixture distribution more than 25 years ago in order to illustrate their non-normal VaR computation methodology.
More recent papers include for example Saissi Hassani and Dionne<sup id="fnref:10:2" role="doc-noteref"><a href="#fn:10" class="footnote">9</a></sup> who performs VaR backtests for several parametric models, among which two and three-component Gaussian mixture models.</p>
<p>The Gaussian mixture VaR is thus a well-known extension of the Gaussian VaR when dealing with real-life asset returns.</p>
<h2 id="fitting-a-univariate-gaussian-mixture-distribution-to-an-empirical-portfolio-return-distribution">Fitting a univariate Gaussian mixture distribution to an empirical portfolio return distribution</h2>
<p>Let $r_1,…,r_T \in \mathbb{R}$ be the returns of a portfolio observed over $T$ time periods.</p>
<p>In order to compute the Gaussian mixture VaR of this portfolio using the formula of the previous section, it is first required to approximate the empirical distribution of the returns $r_1,…,r_T$ by a
Gaussian mixture distribution.</p>
<p>This is usually done by determining the “best” <a href="https://en.wikipedia.org/wiki/Estimator">statistical estimators</a> for the unknown parameters $k$ and $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$ of that
Gaussian mixture distribution, with “best” to be defined.</p>
<h3 id="how-to-determine-the-number-of-mixture-components">How to determine the number of mixture components?</h3>
<h4 id="manually">Manually</h4>
<p>Because it is possible to <em>interpret the components of a Gaussian mixture return distribution as market regimes, with a latent variable that represents the active regime and a return distribution that is Gaussian, given the regime</em><sup id="fnref:8:1" role="doc-noteref"><a href="#fn:8" class="footnote">7</a></sup>,
the number of mixture components $k$ can be chosen manually.</p>
<p>For example:</p>
<ul>
<li>Choosing $k = 2$ means defining two market regimes in the portfolio returns - a <em>normal</em> regime and a <em>rare events</em> regime, sometimes called a <em>distressed regime</em><sup id="fnref:16:3" role="doc-noteref"><a href="#fn:16" class="footnote">14</a></sup> or a <em>bliss regime</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup></li>
<li>Choosing $k = 3$ means defining three market regimes in the portfolio returns - the typical <em>bear</em>, <em>neutral</em> and <em>bull</em> market regimes<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">16</a></sup></li>
</ul>
<h4 id="automatically">Automatically</h4>
<p>The number of mixture components $k$ can also be chosen automatically through two general families of methods:</p>
<ul>
<li>
<p>Methods that determine the number of mixture components independently of the remaining mixture parameters</p>
<p>Methods from this family typically iterate over a number of potential values for $k$, determine the remaining parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$, and evaluate how well the resulting $k$-component Gaussian mixture distribution approximates the portfolio return distribution.</p>
<p>The chosen number of mixture components is then the value $k^*$, among these potential values for $k$, that allows to best approximate the portfolio return distribution<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">17</a></sup>.</p>
<p>Examples from this family include the <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test">Kolmogorov-Smirnov test</a><sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">18</a></sup> or the more common Bayesian Information Criterion (BIC)<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">19</a></sup>.</p>
</li>
<li>
<p>Methods that determine the number of mixture components together with the remaining mixture parameters</p>
<p>Methods from this family determine all the parameters of the Gaussian mixture distribution at once.</p>
<p>Examples from this family include modified Expectation-Maximization algorithms<sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">20</a></sup> or ad-hoc procedures<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">21</a></sup>.</p>
</li>
</ul>
<h3 id="how-to-determine-the-remaining-mixture-parameters">How to determine the remaining mixture parameters?</h3>
<p>In this subsection, the number of mixture components $k$ is supposed to be known<sup id="fnref:30" role="doc-noteref"><a href="#fn:30" class="footnote">22</a></sup>.</p>
<h4 id="likelihood-maximization">Likelihood maximization</h4>
<p>Sensible values for the unknown mixture parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$, are maximum-likelihood estimates, defined as the choice of parameters
$\left(p_i^*, \mu_i^*, \sigma_i^* \right)$, $i=1..k$, that globally<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">23</a></sup> maximizes the log-likelihood function $L$ associated to the returns $r_1,…,r_T$ and given by</p>
\[L \left( p_1, \mu_1, \sigma_1,..., p_k, \mu_k, \sigma_k \right) = \sum_{i=1}^T \ln \left[ \sum_{j=1}^k p_j \frac{1}{\sqrt{2 \pi} \sigma_j} \exp \left( - \frac{1}{2} \left( \frac{ x_i - \mu_j}{ \sigma_j} \right )^2 \right) \right]\]
<p>Numerically, several procedures can be used to compute these maximum-likelihood estimates, like Newton’s method<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">24</a></sup>, Fisher’s method of scoring<sup id="fnref:29:1" role="doc-noteref"><a href="#fn:29" class="footnote">24</a></sup> or the Expectation-Maximization (EM) algorithm<sup id="fnref:25:1" role="doc-noteref"><a href="#fn:25" class="footnote">19</a></sup> which is actually <em>the
method of choice for learning mixtures of Gaussians</em><sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">25</a></sup><sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">26</a></sup>.</p>
<p>Unfortunately, due to the complexity<sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">27</a></sup> of the log-likelihood function $L$, none of these procedures is guaranteed to produce “true” maximum-likelihood estimates<sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">28</a></sup>. Even worse, the behavior of all these procedures
is known to <em>be highly dependent from [the initial guess of the unknown parameters] and it may fail as a result of degeneracies</em><sup id="fnref:32:1" role="doc-noteref"><a href="#fn:32" class="footnote">28</a></sup>, which led practitioners to
implement misc. numerical tweaks, like running these procedures <em>several times with different, randomly chosen starting points</em><sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">29</a></sup>, in order to <em>avoid [obtaining] a poor approximation to the true maximum</em><sup id="fnref:35:1" role="doc-noteref"><a href="#fn:35" class="footnote">29</a></sup>.</p>
<h4 id="turbulence-partitioning-of-returns">Turbulence partitioning of returns</h4>
<p>Although <em>in most applications, parameters of a [Gaussian] mixture model are estimated by maximizing the likelihood</em><sup id="fnref:35:2" role="doc-noteref"><a href="#fn:35" class="footnote">29</a></sup>, there are other procedures to determine the remaining parameters of a Gaussian mixture distribution
(minimum distance estimation<sup id="fnref:41" role="doc-noteref"><a href="#fn:41" class="footnote">30</a></sup>…) which have <em>often been show to be more robust to departures from underlying assumptions than is maximum likelihood</em><sup id="fnref:41:1" role="doc-noteref"><a href="#fn:41" class="footnote">30</a></sup>.</p>
<p>While an exhaustive review of these alternative procedures is out of scope of this post, I would still like to mention a procedure described in
<a href="/blog/the-turbulence-index-regime-based-partitioning-of-asset-returns/">another post</a> in the the context of a two-component Gaussian mixture distribution.</p>
<p>This procedure is based on a measure of statistical unusualness of asset returns popularized by Kritzman and Li<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">31</a></sup> called the <em>Turbulence Index</em>, and can easily be generalized to a $k$-component Gaussian mixture distribution as follows:</p>
<ul>
<li>Compute the turbulence index values $d(r_t)$, $t=1..T$, associated to the portfolio returns<sup id="fnref:37" role="doc-noteref"><a href="#fn:37" class="footnote">32</a></sup></li>
<li>Partition the portfolio returns into $k$ partitions based on their turbulence index values, either through manual percentage-based thresholding<sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">33</a></sup> or through automated clustering (1D $k$-means<sup id="fnref:38" role="doc-noteref"><a href="#fn:38" class="footnote">34</a></sup>…).</li>
<li>For each of these partitions $P_i$,$i=1..k$
<ul>
<li>Define the estimate of the $i$-th Gaussian mixture component probability $p_i^{**}$ as the proportion of the portfolio returns belonging to $P_i$</li>
<li>Define the estimate of the $i$-th Gaussian mixture component mean $\mu_i^{**}$ as the mean of the portfolio returns belonging to $P_i$</li>
<li>Define the estimate of the $i$-th Gaussian mixture component standard deviation $\sigma_i^{**}$ as the standard deviation of the portfolio returns belonging to $P_i$</li>
</ul>
</li>
</ul>
<p>Compared to likelihood maximization, this procedure is:</p>
<ul>
<li>Easy to implement, with no local optima and no convergence issue to worry about</li>
<li>Easy to interpret, with a one-to-one relationship between partitions and market regimes</li>
</ul>
<p>In addition, as illustrated in <a href="/blog/the-turbulence-index-regime-based-partitioning-of-asset-returns/">the aforementioned post</a>, the quality of the resulting parameters estimates is on par with that of
the maximum-likelihood parameters estimates, at least in the case of a two-asset universe made of U.S. stocks and U.S. Treasuries.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements the computation of a portfolio Gaussian mixture Value-at-Risk through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/analysis/value-at-risk/gaussian/mixture</code></a>.</p>
<p>This endpoint:</p>
<ul>
<li>Fits a univariate Gaussian mixture distribution to an empirical portfolio (logarithmic) return distribution, using either
<ul>
<li>A variation of the Expectation-Maximization algorithm described in the previous section</li>
<li>The turbulence partitioning procedure also described in the previous section, relying on either a percentage-based thresholding of turbulence index values or a 1D $k$-means clustering of turbulence index values</li>
</ul>
</li>
<li>Computes the associated Gaussian mixture Value-at-Risk, using a variation of the bisection procedure</li>
</ul>
<p>To be noted that it is possible to let <strong>Portfolio Optimizer</strong> automatically determine the best number of Gaussian mixture components<sup id="fnref:46" role="doc-noteref"><a href="#fn:46" class="footnote">35</a></sup>.</p>
<h2 id="example-of-usage---computing-the-value-at-risk-of-bitcoin">Example of usage - Computing the Value-at-Risk of Bitcoin</h2>
<p>In their paper, Ang et al.<sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> empirically demonstrate that <em>the very positively skewed returns [of Bitcoin] are well
captured by a [two-component] mixture of Normals distribution</em><sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>. They then use this result to suggest that <em>given a small probability regime where Bitcoin may generate high returns, investors with
preferences for positive skewness of portfolio returns should have portfolio exposure to Bitcoin</em><sup id="fnref:39" role="doc-noteref"><a href="#fn:39" class="footnote">36</a></sup>.</p>
<p>As an example of usage of the Gaussian mixture VaR, I propose to compute the VaR of Bitcoin under such a two-component Gaussian mixture model.</p>
<h3 id="bitcoin-returns-analysis">Bitcoin returns analysis</h3>
<p>I propose to start by comparing the Bitcoin price data used in Ang et al.<sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> to the Bitcoin price data used in this post:</p>
<ul>
<li>
<p>Figures 1<sup id="fnref:42" role="doc-noteref"><a href="#fn:42" class="footnote">37</a></sup> and 2 compare the cumulative returns of Bitcoin over the period 31 August 2010 - 31 December 2021</p>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-cumulative-returns-ang.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-cumulative-returns-ang.png" alt="Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al." /></a>
<figcaption>Figure 1. Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-cumulative-returns.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-cumulative-returns.png" alt="Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, this post's data." /></a>
<figcaption>Figure 2. Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, this post's data.</figcaption>
</figure>
</li>
<li>
<p>Figures 3<sup id="fnref:42:1" role="doc-noteref"><a href="#fn:42" class="footnote">37</a></sup> and 4 compare the annualized mean, standard deviation and Sharpe ratio of Bitcoin monthly (log) returns over the same period</p>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-moments-ang.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-moments-ang.png" alt="Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al." /></a>
<figcaption>Figure 3. Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-moments.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-moments.png" alt="Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, this post's data." /></a>
<figcaption>Figure 4. Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, this post's data.</figcaption>
</figure>
</li>
</ul>
<p>From these figures, it is clear that the Bitcoin data used in Ang et al.<sup id="fnref:6:5" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup> differ from the Bitcoin data used in this post (Figure 3 v.s. Figure 4), although not that much (Figure 1 v.s. Figure 2).</p>
<p>To be noted that this is not surprising, because there is no official price for Bitcoin due to its decentralized nature.</p>
<h3 id="fitting-a-gaussian-mixture-model-to-bitcoin-returns">Fitting a Gaussian mixture model to Bitcoin returns</h3>
<p>Similar to Ang et al.<sup id="fnref:6:6" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>, I now propose to fit a two-component Gaussian mixture model to the Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021.</p>
<h4 id="likelihood-maximization-1">Likelihood maximization</h4>
<p>The EM algorithm, as implemented by the <a href="https://scikit-learn.org/">scikit-learn</a> method <a href="https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html"><code class="language-plaintext highlighter-rouge">sklearn.mixture.GaussianMixture</code></a>
with default parameters, leads to three different maximum likelihood estimates<sup id="fnref:47" role="doc-noteref"><a href="#fn:47" class="footnote">38</a></sup></p>
<ul>
<li>$\left(p_1^1, \mu_1^1, \sigma_1^1 \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^1, \mu_2^1, \sigma_2^1 \right)$ = $\left( 0.04, 14.98, 0.89 \right)$</li>
<li>$\left(p_1^2, \mu_1^2, \sigma_1^2 \right)$ = $\left( 0.91, 0.54, 0.82 \right)$ and $\left(p_2^2, \mu_2^2, \sigma_2^2 \right)$ = $\left( 0.09, 7.45, 2.03 \right)$</li>
<li>$\left(p_1^3, \mu_1^3, \sigma_1^3 \right)$ = $\left( 0.91, 0.55, 0.82 \right)$ and $\left(p_2^3, \mu_2^3, \sigma_2^3 \right)$ = $\left( 0.09, 7.81, 2.03 \right)$</li>
</ul>
<p>This situation confirms that an <em>important drawback of EM is that its solution can highly depend on its starting position and consequently produce sub-optimal maximum likelihood estimates</em><sup id="fnref:35:3" role="doc-noteref"><a href="#fn:35" class="footnote">29</a></sup>!</p>
<p>Fortunately, increasing the number of random initializations performed in the <code class="language-plaintext highlighter-rouge">sklearn.mixture.GaussianMixture</code> method leads to the optimal maximum likelihood estimates,
but somebody unaware of the numerical properties of the EM algorithm might conclude too quickly that <em>the mixture of Normals can be estimated <strong>easily</strong> by maximum likelihood or EM algorithms</em><sup id="fnref:6:7" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>.</p>
<p>These “true” maximum likelihood estimates are $ \left(p_1^*, \mu_1^*, \sigma_1^* \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^*, \mu_2^*, \sigma_2^* \right)$ = $\left( 0.04, 14.98, 0.89 \right)$.</p>
<h4 id="turbulence-partitioning">Turbulence partitioning</h4>
<p>The turbulence partitioning procedure used with an exact 1D $k$-means clustering leads to the same<sup id="fnref:43" role="doc-noteref"><a href="#fn:43" class="footnote">39</a></sup> estimates as the “true” maximum likelihood estimates,
which empirically demonstrates the validity of this procedure to fit the parameters of a univariate Gaussian mixture distribution.</p>
<p>As a reminder, these estimates are $ \left(p_1^{**}, \mu_1^{**}, \sigma_1^{**} \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^{**}, \mu_2^{**}, \sigma_2^{**} \right)$ = $\left( 0.04, 14.98, 0.89 \right)$.</p>
<p>One of the advantages of the turbulence partitioning procedure is that it allows to easily explain how the Gaussian mixture parameters have been estimated:</p>
<ul>
<li>
<p>$p_1^{**}$ (resp. $p_2^{**}$) is the proportion of Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method</p>
<p>Graphically, thanks to Figure 5 which displays the 137 turbulence index values associated to the Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021:</p>
<ul>
<li>
<p>$p_1^{**}$ is the proportion of Bitcoin returns whose turbulence index values are below the red horizontal line, that is, $p_1^{**} = \frac{132}{137} \approx 0.96$</p>
</li>
<li>
<p>$p_2^{**}$ is the proportion of Bitcoin returns whose turbulence index values are above the red horizontal line, that is, $p_2^{**} = \frac{5}{137} \approx 0.04$</p>
</li>
</ul>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-turbulence-index.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-turbulence-index.png" alt="Bitcoin monthly turbulence index values, 31 August 2010 - 31 December 2021." /></a>
<figcaption>Figure 5. Bitcoin monthly turbulence index values, 31 August 2010 - 31 December 2021.</figcaption>
</figure>
</li>
<li>
<p>$\mu_1^{**}$ (resp. $\mu_2^{**}$) is the mean of the Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method</p>
</li>
<li>
<p>$\sigma_1^{**}$ (resp. $\sigma_2^{**}$) is the standard deviation of the Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method</p>
</li>
</ul>
<p>In terms of market regimes, the first (resp. second) Gaussian mixture component models the behavior of Bitcoin during a <em>normal</em> (resp. <em>rare events</em>) market regime.</p>
<p>As a side note, the turbulence partitioning procedure highlights that only 5 returns have been used to compute the parameters estimates $\left(p_2^{**}, \mu_2^{**}, \sigma_2^{**} \right)$, which
raises the question of the statistical significance of both these estimates<sup id="fnref:44" role="doc-noteref"><a href="#fn:44" class="footnote">40</a></sup> and the maximum likelihood estimates $\left(p_2^*, \mu_2^*, \sigma_2^* \right)$ since they are equal…</p>
<h4 id="evaluation">Evaluation</h4>
<p>Figure 6 and Figure 7 compare the theoretical distribution of the estimated two-component Gaussian mixture distribution with the empirical distribution of the monthly (log) Bitcoin returns.</p>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-cdf-vs-gmm-cdf.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-cdf-vs-gmm-cdf.png" alt="Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., 31 August 2010 - 31 December 2021." /></a>
<figcaption>Figure 6. Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., 31 August 2010 - 31 December 2021.</figcaption>
</figure>
<figure>
<a href="/assets/images/blog/beyond-modified-var-bitcoin-cdf-vs-gmm-cdf-left-tail.png"><img src="/assets/images/blog/beyond-modified-var-bitcoin-cdf-vs-gmm-cdf-left-tail.png" alt="Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., left tail, 31 August 2010 - 31 December 2021." /></a>
<figcaption>Figure 7. Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., left tail, 31 August 2010 - 31 December 2021.</figcaption>
</figure>
<p>From these figures, the estimated mixture model seems to adequately capture the main characteristics of the Bitcoin returns, except part of their left tail behavior, which is confirmed more quantitatively
by the Kolmogorov-Smirnov goodness of fit test<sup id="fnref:45" role="doc-noteref"><a href="#fn:45" class="footnote">41</a></sup>.</p>
<h3 id="bitcoin-returns-value-at-risk">Bitcoin returns Value-at-Risk</h3>
<p>I finally propose to compute the Value-at-Risk of Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021, using two methods:</p>
<ul>
<li>The <em>historical Value-at-Risk</em> (HVaR), introduced in <a href="/blog/the-turbulence-index-regime-based-partitioning-of-asset-returns/">another post</a></li>
<li>The Gaussian mixture Value-at-Risk, introduced in this blog post</li>
</ul>
<p>Results for various confidence levels, all visible on Figure 7, are provided below:</p>
<table>
<thead>
<tr>
<th>Confidence level $\alpha$</th>
<th>$\text{HVaR}_{\alpha}$</th>
<th>$\text{GmVaR}_{\alpha}$</th>
</tr>
</thead>
<tbody>
<tr>
<td>95%</td>
<td>42.02%</td>
<td>34.21%</td>
</tr>
<tr>
<td>97.5%</td>
<td>45.84%</td>
<td>41.97%</td>
</tr>
<tr>
<td>99%</td>
<td>47.22%</td>
<td>50.96%</td>
</tr>
<tr>
<td>99.5%</td>
<td>49.86%</td>
<td>57.08%</td>
</tr>
<tr>
<td>99.9%</td>
<td>49.86%</td>
<td>69.69%</td>
</tr>
</tbody>
</table>
<h2 id="conclusion">Conclusion</h2>
<p>Aim of this post was to describe an extension of Gaussian Value-at-Risk, second in order of complexity after modified Value-At-Risk, in case the later cannot be computed.</p>
<p>This being done, I wish you Season’s Greetings and a Happy New Year!</p>
<p>Waiting for 2024, feel free to <a href="https://www.linkedin.com/in/roman-rubsamen/">connect with me on LinkedIn</a> or <a href="https://twitter.com/portfoliooptim">follow me on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>See <a href="">Zangari, P. (1996). A VaR methodology for portfolios that include options. RiskMetrics Monitor First Quarter, 4–12</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Or <em>Cornish-Fisher VaR</em>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>In this post, all returns are assumed to be logarithmic returns. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1997178">Maillard, Didier, A User’s Guide to the Cornish Fisher Expansion</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>All portfolio return distribution for which the skewness and the (excess) kurtosis are outside of the domain of validity of the Cornish-Fisher expansion underlying modified VaR, c.f. Maillard<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">4</a></sup>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijaltinv/25/4/7">Ang, Andrew, Morris, Tom, Savi, Raffaele, Asset Allocation with Crypto: Application of Preferences for Positive Skewness, The Journal of Alternative Investments, Spring 2023, 25 (4) 7-28</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:6:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:6:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:6:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>See <a href="https://doi.org/10.1007/s11081-023-09814-y">Luxenberg, E., Boyd, S. Portfolio construction with Gaussian mixture returns and exponential utility via convex optimization. Optim Eng (2023).</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:8:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Some theoretical subtleties are detailed in <a href="/blog/corrected-cornish-fisher-expansion-improving-the-accuracy-of-modified-value-at-risk/">a previous post</a>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3766511">Saissi Hassani, Samir and Dionne, Georges, The New International Regulation of Market Risk: Roles of VaR and CVaR in Model Validation (January 12, 2021)</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:10:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>See <a href="https://arxiv.org/abs/2202.10721">Benjamin Bruder, Nazar Kostyuchyk, Thierry Roncalli, Risk Parity Portfolios with Skewness Risk: An Application to Factor Investing and Alternative Risk Premia, arXiv</a>. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>See <a href="http://www.jstor.org/stable/2330751">Joyce A. Hall, Wade Brorsen and Scott H. Irwin, The Distribution of Futures Prices: A Test of the Stable Paretian and Mixture of Normals Hypotheses, The Journal of Financial and Quantitative Analysis, Vol. 24, No. 1 (Mar., 1989), pp.105-116</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-6261.1984.tb03865.x">Kon S (1984) Models of stock returns-a comparison. J Financ 39(1):147–16</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>See <a href="https://www.discuss.wmie.uz.zgora.pl/ps/source/publish/view_pdf.php?doi=10.7151/dmps.1190">C. Cuevas-Covarrubias, J. Inigo-Martinez and R. Jimenez-Padilla, Gaussian mixtures and financial returns, Discussiones Mathematicae, Probability and Statistics 37 (2017) 101–122</a>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:17:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:17:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221706006242">Ian Buckley, David Saunders, Luis Seco, Portfolio optimization when asset returns have the Gaussian mixture distribution, European Journal of Operational Research, Volume 185, Issue 3, 2008, Pages 1434-1461</a>. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:16:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:16:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:16:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijderiv/5/3/9">J. Hull, A. White, Value at risk when daily changes in market variables are not normally distributed, Journal of Derivatives 5 (3) (1998) 9–19</a>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>See <a href="https://arxiv.org/abs/2006.11383">Yuantong Li, Qi Ma, Sujit K. Ghosh, A Non-Iterative Quantile Change Detection Method in Mixture Model with Heavy-Tailed Components, arXiv</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>The selected number of mixture components sometimes also takes into account both the quality of the approximation of the portfolio return distribution and the potential for overfitting. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/pii/S0895717703900227">Ming-Heng Zhang, Qian-Sheng Cheng, Gaussian mixture modelling to detect random walks in capital markets, Mathematical and Computer Modelling, Volume 38, Issues 5–6, 2003, Pages 503-508</a>. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://ieeexplore.ieee.org/abstract/document/8046036/">J. Worms and S. Touati, “Modelling Program’s Performance with Gaussian Mixtures for Parametric Statistics,” in IEEE Transactions on Multi-Scale Computing Systems, vol. 4, no. 3, pp. 383-395, 1 July-Sept. 2018</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:25:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://www3.stat.sinica.edu.tw/statistica/J27N1/J27N17/J27N17.html">Tao Huang, Heng Peng and Kun Zhang, Model selection for Gaussian mixture models, Statistica Sinica 27 (2017), 147-169</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p><a href="https://dl.acm.org/doi/10.1145/3394486.3403240">Yuantong Li, Qi Ma, and Sujit K. Ghosh. 2020. A Non-Iterative Quantile Change Detection Method in Mixture Model with Heavy-Tailed Components. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ‘20). Association for Computing Machinery, New York, NY, USA, 1888–1898.</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:30" role="doc-endnote">
<p>Typically, because it is determined independently of the remaining parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$. <a href="#fnref:30" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>To avoid theoretical issues, some authors prefer to work with non-unique local maximizers, c.f. for example Peters and Walker<sup id="fnref:29:2" role="doc-noteref"><a href="#fn:29" class="footnote">24</a></sup>. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>See <a href="http://www.jstor.org/stable/2100676">B. Charles Peters, Jr. and Homer F. Walker, An Iterative Procedure for Obtaining Maximum-Likelihood Estimates of the Parameters for a Mixture of Normal Distributions, SIAM Journal on Applied Mathematics, Vol. 35, No. 2 (Sep., 1978), pp. 362-378</a>. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:29:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:29:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://arxiv.org/abs/1301.3850">Sanjoy Dasgupta, Leonard Schulman, A Two-round Variant of EM for Gaussian Mixtures, arXiv</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>This is because the estimation of the parameters of a Gaussian mixture distribution can be formulated as a missing data problem<sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">42</a></sup>, for which the EM algorithm has been specifically designed. <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>In particular, it is non convex. <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>See <a href="https://inria.hal.science/hal-01113242/document">Jean-Patrick Baudry, Gilles Celeux. EM for mixtures - Initialization requires special care. 2015. hal-01113242</a>. <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:32:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0167947302001639">Christophe Biernacki, Gilles Celeux, Gérard Govaert, Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models, Computational Statistics & Data Analysis, Volume 41, Issues 3–4, 2003, Pages 561-575</a>. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:35:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:35:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:35:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:41" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.1080/01621459.1984.10478085">Wayne A. Woodward , William C. Parr , William R. Schucany & Hildegard Lindsey (1984) A Comparison of Minimum Distance and Maximum Likelihood Estimation of a Mixture Proportion, Journal of the American Statistical Association, 79:387, 590-598</a>. <a href="#fnref:41" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:41:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>See <a href="https://www.tandfonline.com/doi/abs/10.2469/faj.v66.n5.3">M. Kritzman, Y. Li, Skulls, Financial Turbulence, and Risk Management,Financial Analysts Journal, Volume 66, Number 5, Pages 30-41, Year 2010</a>. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:37" role="doc-endnote">
<p>Or the underlying multivariate asset returns, if available. <a href="#fnref:37" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>See <a href="https://www.jstor.org/stable/4480169">George Chow, Jacquier, E., Kritzman, M., & Kenneth Lowry. (1999). Optimal Portfolios in Good Times and Bad. Financial Analysts Journal, 55(3), 65–73.</a>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:38" role="doc-endnote">
<p>To be noted that in 1D, the $k$-means algorithm can be computed exactly, see for example <a href="https://arxiv.org/abs/1701.07204">Allan Gronlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, Mingzhou Song, Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D, arXiv</a>. <a href="#fnref:38" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:46" role="doc-endnote">
<p>A proprietary methodology is used for that, except when the turbulence partitioning procedure is used together with a percentage-based thresholding of turbulence index values, in which case the number of Gaussian mixture components is equal<sup id="fnref:40" role="doc-noteref"><a href="#fn:40" class="footnote">43</a></sup> to the number of provided thresholds plus one. <a href="#fnref:46" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:39" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4217841">Sepp, Artur, Optimal Allocation to Cryptocurrencies in Diversified Portfolios (December 31, 2022). Risk Magazine, October 2023, 1-6</a>. <a href="#fnref:39" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:42" role="doc-endnote">
<p>Adapted from Ang et al.<sup id="fnref:6:8" role="doc-noteref"><a href="#fn:6" class="footnote">6</a></sup>. <a href="#fnref:42" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:42:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:47" role="doc-endnote">
<p>Annualized. <a href="#fnref:47" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:43" role="doc-endnote">
<p>You will need to trust me on this one, as long as <strong>Portfolio Optimizer</strong> does not expose an endpoint to fit a Gaussian mixture distribution to an empirical return distribution. <a href="#fnref:43" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:44" role="doc-endnote">
<p>See <a href="https://digitalcommons.unl.edu/usnavyresearch/34">Greenwood, Joseph A. and Sandomire, Marion M., “Sample Size Required For Estimating The Standard Deviation as a Percent of Its True Value” (1950). U.S. Navy Research. 34.</a>. <a href="#fnref:44" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:45" role="doc-endnote">
<p>The $p$-values are so ridiculously high (> 0.95) that even <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Test_with_estimated_parameters">taking into account the fact that the parameters of the two-component Gaussian mixture distribution have been estimated</a>, the conclusion remains the same. <a href="#fnref:45" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>See <a href="https://www.econstor.eu/bitstream/10419/22198/1/24_tk_gm_skn.pdf">McLachlan, Geoffrey J.; Krishnan, Thriyambakam; Ng, See Ket (2004) : The EM Algorithm, Papers, No. 2004,24, Humboldt-Universität zu Berlin, Center for Applied Statistics and Economics (CASE), Berlin</a>. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:40" role="doc-endnote">
<p>Except when there are empty partitions. <a href="#fnref:40" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.In a previous post, I described a parametric approach to computing Value-at-Risk (VaR) - called modified VaR12 - that adjusts Gaussian VaR for asymmetry and fat tails present in financial asset returns3 thanks to the usage of a Cornish–Fisher expansion. Modified VaR, when properly used4, provides accurate estimates of the VaR for a wide range of non-normal portfolio return distributions. Unfortunately, for mathematical reasons, it cannot be computed for an even wider range of such distributions5, which poses an issue in practice. In this blog post, I will describe another parametric approach to computing VaR - called Gaussian mixture Value-at-Risk for the lack of a better name - which this time adjusts Gaussian VaR for higher moments thanks to the usage of a Gaussian mixture distribution and is free of any computational restriction. After providing the formula for the Gaussian mixture VaR and explaining how to fit a Gaussian mixture distribution to an empirical return distribution, I will show how to compute the Value-at-Risk of Bitcoin under the Gaussian mixture model described in the BlackRock paper Asset Allocation with Crypto: Application of Preferences for Positive Skewness from Ang et al.6 Mathematical preliminaries Univariate Gaussian mixture distribution A random variable $X$ is said to follow a univariate Gaussian mixture distribution, written as $X \sim \mathcal{GM} \left( \left( \mu_i, \sigma_i, p_i \right)_{i=1}^k \right) $, if its cumulative distribution function (c.d.f.) $F_X$ is of the form7 \[F_X(x) = \sum_{i=1}^k p_i \Phi \left( \frac{x - \mu_j}{\sigma_j} \right)\] , where: $k \geq 1$ is the (integer) number of mixture components $p_i \in \left[ 0,1 \right]$, $i=1..k$, are the probabilities of the mixture components, with $\sum_{i=1}^kp_i = 1$ $\mu_i \in \mathbb{R}$, $i=1..k$, are the means of the mixture components $\sigma_i \gt 0$, $i=1..k$, are the standard deviations of the mixture components $\Phi$ is the c.d.f. of the standard normal distribution Reminder on Value-at-Risk The (percentage) VaR of a portfolio of financial assets corresponds to the percentage of portfolio wealth that can be lost over a certain time horizon and with a certain probability. Formally, the VaR $VaR_{\alpha}$ of a portfolio over a time horizon $T$ (1 day, 10 days…) and at a confidence level $\alpha$% $\in ]0,1[$ (95%, 97.5%, 99%…) can be defined as8 \[\text{VaR}_{\alpha} (X) = - F_X^{-1}(1 - \alpha)\] , where: $X$ is a random variable representing the portfolio return over the time horizon $T$ $F_X^{-1}$ is the inverse c.d.f. - also called the quantile function - of the random variable $X$ Gaussian mixture Value-at-Risk Definition When the return distribution of a portfolio is approximated by a Gaussian mixture distribution, that is, when $X \sim \mathcal{GM} \left( \left( \mu_i, \sigma_i, p_i \right)_{i=1}^k \right) $ with $X$ a random variable representing the portfolio return over a given time horizon $T$, the resulting parametric VaR can be called Gaussian mixture Value-at-Risk (GmVaR). Mathematically, the Gaussian mixture VaR over the chosen time horizon $T$ and at a confidence level $\alpha$% $GmVaR_{\alpha}$ is implicitly defined by the following equation9, obtained by mixing together the formula for the VaR of a portfolio and the formula for the c.d.f. of a Gaussian mixture distribution \[\sum_{i=1}^k p_i \Phi \left( - \frac{ GmVaR_{\alpha} + \mu_i }{\sigma_i} \right) = 1 - \alpha\] Computation Because the formula of the previous subsection is implicit in $GmVaR_{\alpha}$, the effective computation of the Gaussian mixture VaR must involve a numerical procedure such as: A non-linear least-squares minimization procedure9 A bisection procedure10 Or more generally, a procedure to compute a quantile function from a closed-form c.d.f. Rationale It has been observed since the 1970s11 that Gaussian mixture distributions adequately approximate the unconditional distributions of financial asset returns. For example, Kon12 shows that two, three and four-component Gaussian mixture distributions can be used to model the daily returns of 3 U.S. stock market indexes and of 30 Dow Jones stocks. Similarly, Cuevas-Covarrubias et al.13 shows how two-component Gaussian mixture distributions exhibit a natural capacity to fit leptokurtic distributions13 using the daily returns of 3 Mexican stocks. Compared to alternative distributions, Gaussian mixture distributions offer a versatile combination of precision and simplicity13 for modeling asset returns. Indeed, Gaussian mixture distributions are both flexible enough to handle arbitrary asset return distributions14 and simple enough to be numerically tractable14. In contrast, [alternative distributions] tend to suffer from one of two evils: either they are too restrictive in the variety of p.d.f. shapes that can be achieved, or not restrictive enough, in the sense that they have too many degrees of freedom for calibration to be feasible14. As for applications of Gaussian mixture distributions to VaR, Hull and White15 used a two-component Gaussian mixture distribution more than 25 years ago in order to illustrate their non-normal VaR computation methodology. More recent papers include for example Saissi Hassani and Dionne9 who performs VaR backtests for several parametric models, among which two and three-component Gaussian mixture models. The Gaussian mixture VaR is thus a well-known extension of the Gaussian VaR when dealing with real-life asset returns. Fitting a univariate Gaussian mixture distribution to an empirical portfolio return distribution Let $r_1,…,r_T \in \mathbb{R}$ be the returns of a portfolio observed over $T$ time periods. In order to compute the Gaussian mixture VaR of this portfolio using the formula of the previous section, it is first required to approximate the empirical distribution of the returns $r_1,…,r_T$ by a Gaussian mixture distribution. This is usually done by determining the “best” statistical estimators for the unknown parameters $k$ and $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$ of that Gaussian mixture distribution, with “best” to be defined. How to determine the number of mixture components? Manually Because it is possible to interpret the components of a Gaussian mixture return distribution as market regimes, with a latent variable that represents the active regime and a return distribution that is Gaussian, given the regime7, the number of mixture components $k$ can be chosen manually. For example: Choosing $k = 2$ means defining two market regimes in the portfolio returns - a normal regime and a rare events regime, sometimes called a distressed regime14 or a bliss regime6 Choosing $k = 3$ means defining three market regimes in the portfolio returns - the typical bear, neutral and bull market regimes16 Automatically The number of mixture components $k$ can also be chosen automatically through two general families of methods: Methods that determine the number of mixture components independently of the remaining mixture parameters Methods from this family typically iterate over a number of potential values for $k$, determine the remaining parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$, and evaluate how well the resulting $k$-component Gaussian mixture distribution approximates the portfolio return distribution. The chosen number of mixture components is then the value $k^*$, among these potential values for $k$, that allows to best approximate the portfolio return distribution17. Examples from this family include the Kolmogorov-Smirnov test18 or the more common Bayesian Information Criterion (BIC)19. Methods that determine the number of mixture components together with the remaining mixture parameters Methods from this family determine all the parameters of the Gaussian mixture distribution at once. Examples from this family include modified Expectation-Maximization algorithms20 or ad-hoc procedures21. How to determine the remaining mixture parameters? In this subsection, the number of mixture components $k$ is supposed to be known22. Likelihood maximization Sensible values for the unknown mixture parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$, are maximum-likelihood estimates, defined as the choice of parameters $\left(p_i^*, \mu_i^*, \sigma_i^* \right)$, $i=1..k$, that globally23 maximizes the log-likelihood function $L$ associated to the returns $r_1,…,r_T$ and given by \[L \left( p_1, \mu_1, \sigma_1,..., p_k, \mu_k, \sigma_k \right) = \sum_{i=1}^T \ln \left[ \sum_{j=1}^k p_j \frac{1}{\sqrt{2 \pi} \sigma_j} \exp \left( - \frac{1}{2} \left( \frac{ x_i - \mu_j}{ \sigma_j} \right )^2 \right) \right]\] Numerically, several procedures can be used to compute these maximum-likelihood estimates, like Newton’s method24, Fisher’s method of scoring24 or the Expectation-Maximization (EM) algorithm19 which is actually the method of choice for learning mixtures of Gaussians2526. Unfortunately, due to the complexity27 of the log-likelihood function $L$, none of these procedures is guaranteed to produce “true” maximum-likelihood estimates28. Even worse, the behavior of all these procedures is known to be highly dependent from [the initial guess of the unknown parameters] and it may fail as a result of degeneracies28, which led practitioners to implement misc. numerical tweaks, like running these procedures several times with different, randomly chosen starting points29, in order to avoid [obtaining] a poor approximation to the true maximum29. Turbulence partitioning of returns Although in most applications, parameters of a [Gaussian] mixture model are estimated by maximizing the likelihood29, there are other procedures to determine the remaining parameters of a Gaussian mixture distribution (minimum distance estimation30…) which have often been show to be more robust to departures from underlying assumptions than is maximum likelihood30. While an exhaustive review of these alternative procedures is out of scope of this post, I would still like to mention a procedure described in another post in the the context of a two-component Gaussian mixture distribution. This procedure is based on a measure of statistical unusualness of asset returns popularized by Kritzman and Li31 called the Turbulence Index, and can easily be generalized to a $k$-component Gaussian mixture distribution as follows: Compute the turbulence index values $d(r_t)$, $t=1..T$, associated to the portfolio returns32 Partition the portfolio returns into $k$ partitions based on their turbulence index values, either through manual percentage-based thresholding33 or through automated clustering (1D $k$-means34…). For each of these partitions $P_i$,$i=1..k$ Define the estimate of the $i$-th Gaussian mixture component probability $p_i^{**}$ as the proportion of the portfolio returns belonging to $P_i$ Define the estimate of the $i$-th Gaussian mixture component mean $\mu_i^{**}$ as the mean of the portfolio returns belonging to $P_i$ Define the estimate of the $i$-th Gaussian mixture component standard deviation $\sigma_i^{**}$ as the standard deviation of the portfolio returns belonging to $P_i$ Compared to likelihood maximization, this procedure is: Easy to implement, with no local optima and no convergence issue to worry about Easy to interpret, with a one-to-one relationship between partitions and market regimes In addition, as illustrated in the aforementioned post, the quality of the resulting parameters estimates is on par with that of the maximum-likelihood parameters estimates, at least in the case of a two-asset universe made of U.S. stocks and U.S. Treasuries. Implementation in Portfolio Optimizer Portfolio Optimizer implements the computation of a portfolio Gaussian mixture Value-at-Risk through the endpoint /portfolios/analysis/value-at-risk/gaussian/mixture. This endpoint: Fits a univariate Gaussian mixture distribution to an empirical portfolio (logarithmic) return distribution, using either A variation of the Expectation-Maximization algorithm described in the previous section The turbulence partitioning procedure also described in the previous section, relying on either a percentage-based thresholding of turbulence index values or a 1D $k$-means clustering of turbulence index values Computes the associated Gaussian mixture Value-at-Risk, using a variation of the bisection procedure To be noted that it is possible to let Portfolio Optimizer automatically determine the best number of Gaussian mixture components35. Example of usage - Computing the Value-at-Risk of Bitcoin In their paper, Ang et al.6 empirically demonstrate that the very positively skewed returns [of Bitcoin] are well captured by a [two-component] mixture of Normals distribution6. They then use this result to suggest that given a small probability regime where Bitcoin may generate high returns, investors with preferences for positive skewness of portfolio returns should have portfolio exposure to Bitcoin36. As an example of usage of the Gaussian mixture VaR, I propose to compute the VaR of Bitcoin under such a two-component Gaussian mixture model. Bitcoin returns analysis I propose to start by comparing the Bitcoin price data used in Ang et al.6 to the Bitcoin price data used in this post: Figures 137 and 2 compare the cumulative returns of Bitcoin over the period 31 August 2010 - 31 December 2021 Figure 1. Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al. Figure 2. Cumulative Bitcoin returns over the period 31 August 2010 - 31 December 2021, this post's data. Figures 337 and 4 compare the annualized mean, standard deviation and Sharpe ratio of Bitcoin monthly (log) returns over the same period Figure 3. Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, Ang et al.'s data. Source: Ang et al. Figure 4. Summary statistics of Bitcoin monthly returns over the period 31 August 2010 - 31 December 2021, this post's data. From these figures, it is clear that the Bitcoin data used in Ang et al.6 differ from the Bitcoin data used in this post (Figure 3 v.s. Figure 4), although not that much (Figure 1 v.s. Figure 2). To be noted that this is not surprising, because there is no official price for Bitcoin due to its decentralized nature. Fitting a Gaussian mixture model to Bitcoin returns Similar to Ang et al.6, I now propose to fit a two-component Gaussian mixture model to the Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021. Likelihood maximization The EM algorithm, as implemented by the scikit-learn method sklearn.mixture.GaussianMixture with default parameters, leads to three different maximum likelihood estimates38 $\left(p_1^1, \mu_1^1, \sigma_1^1 \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^1, \mu_2^1, \sigma_2^1 \right)$ = $\left( 0.04, 14.98, 0.89 \right)$ $\left(p_1^2, \mu_1^2, \sigma_1^2 \right)$ = $\left( 0.91, 0.54, 0.82 \right)$ and $\left(p_2^2, \mu_2^2, \sigma_2^2 \right)$ = $\left( 0.09, 7.45, 2.03 \right)$ $\left(p_1^3, \mu_1^3, \sigma_1^3 \right)$ = $\left( 0.91, 0.55, 0.82 \right)$ and $\left(p_2^3, \mu_2^3, \sigma_2^3 \right)$ = $\left( 0.09, 7.81, 2.03 \right)$ This situation confirms that an important drawback of EM is that its solution can highly depend on its starting position and consequently produce sub-optimal maximum likelihood estimates29! Fortunately, increasing the number of random initializations performed in the sklearn.mixture.GaussianMixture method leads to the optimal maximum likelihood estimates, but somebody unaware of the numerical properties of the EM algorithm might conclude too quickly that the mixture of Normals can be estimated easily by maximum likelihood or EM algorithms6. These “true” maximum likelihood estimates are $ \left(p_1^*, \mu_1^*, \sigma_1^* \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^*, \mu_2^*, \sigma_2^* \right)$ = $\left( 0.04, 14.98, 0.89 \right)$. Turbulence partitioning The turbulence partitioning procedure used with an exact 1D $k$-means clustering leads to the same39 estimates as the “true” maximum likelihood estimates, which empirically demonstrates the validity of this procedure to fit the parameters of a univariate Gaussian mixture distribution. As a reminder, these estimates are $ \left(p_1^{**}, \mu_1^{**}, \sigma_1^{**} \right)$ = $\left( 0.96, 0.66, 0.84 \right)$ and $\left(p_2^{**}, \mu_2^{**}, \sigma_2^{**} \right)$ = $\left( 0.04, 14.98, 0.89 \right)$. One of the advantages of the turbulence partitioning procedure is that it allows to easily explain how the Gaussian mixture parameters have been estimated: $p_1^{**}$ (resp. $p_2^{**}$) is the proportion of Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method Graphically, thanks to Figure 5 which displays the 137 turbulence index values associated to the Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021: $p_1^{**}$ is the proportion of Bitcoin returns whose turbulence index values are below the red horizontal line, that is, $p_1^{**} = \frac{132}{137} \approx 0.96$ $p_2^{**}$ is the proportion of Bitcoin returns whose turbulence index values are above the red horizontal line, that is, $p_2^{**} = \frac{5}{137} \approx 0.04$ Figure 5. Bitcoin monthly turbulence index values, 31 August 2010 - 31 December 2021. $\mu_1^{**}$ (resp. $\mu_2^{**}$) is the mean of the Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method $\sigma_1^{**}$ (resp. $\sigma_2^{**}$) is the standard deviation of the Bitcoin returns whose turbulence index values are classified as “normal” (resp. “outliers”) by the 1D $k$-means clustering method In terms of market regimes, the first (resp. second) Gaussian mixture component models the behavior of Bitcoin during a normal (resp. rare events) market regime. As a side note, the turbulence partitioning procedure highlights that only 5 returns have been used to compute the parameters estimates $\left(p_2^{**}, \mu_2^{**}, \sigma_2^{**} \right)$, which raises the question of the statistical significance of both these estimates40 and the maximum likelihood estimates $\left(p_2^*, \mu_2^*, \sigma_2^* \right)$ since they are equal… Evaluation Figure 6 and Figure 7 compare the theoretical distribution of the estimated two-component Gaussian mixture distribution with the empirical distribution of the monthly (log) Bitcoin returns. Figure 6. Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., 31 August 2010 - 31 December 2021. Figure 7. Bitcoin monthly returns, empirical c.d.f. v.s. two-component Gaussian mixture c.d.f., left tail, 31 August 2010 - 31 December 2021. From these figures, the estimated mixture model seems to adequately capture the main characteristics of the Bitcoin returns, except part of their left tail behavior, which is confirmed more quantitatively by the Kolmogorov-Smirnov goodness of fit test41. Bitcoin returns Value-at-Risk I finally propose to compute the Value-at-Risk of Bitcoin monthly (log) returns over the period 31 August 2010 - 31 December 2021, using two methods: The historical Value-at-Risk (HVaR), introduced in another post The Gaussian mixture Value-at-Risk, introduced in this blog post Results for various confidence levels, all visible on Figure 7, are provided below: Confidence level $\alpha$ $\text{HVaR}_{\alpha}$ $\text{GmVaR}_{\alpha}$ 95% 42.02% 34.21% 97.5% 45.84% 41.97% 99% 47.22% 50.96% 99.5% 49.86% 57.08% 99.9% 49.86% 69.69% Conclusion Aim of this post was to describe an extension of Gaussian Value-at-Risk, second in order of complexity after modified Value-At-Risk, in case the later cannot be computed. This being done, I wish you Season’s Greetings and a Happy New Year! Waiting for 2024, feel free to connect with me on LinkedIn or follow me on Twitter. – See Zangari, P. (1996). A VaR methodology for portfolios that include options. RiskMetrics Monitor First Quarter, 4–12. ↩ Or Cornish-Fisher VaR. ↩ In this post, all returns are assumed to be logarithmic returns. ↩ See Maillard, Didier, A User’s Guide to the Cornish Fisher Expansion. ↩ All portfolio return distribution for which the skewness and the (excess) kurtosis are outside of the domain of validity of the Cornish-Fisher expansion underlying modified VaR, c.f. Maillard4. ↩ See Ang, Andrew, Morris, Tom, Savi, Raffaele, Asset Allocation with Crypto: Application of Preferences for Positive Skewness, The Journal of Alternative Investments, Spring 2023, 25 (4) 7-28. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 See Luxenberg, E., Boyd, S. Portfolio construction with Gaussian mixture returns and exponential utility via convex optimization. Optim Eng (2023).. ↩ ↩2 Some theoretical subtleties are detailed in a previous post. ↩ See Saissi Hassani, Samir and Dionne, Georges, The New International Regulation of Market Risk: Roles of VaR and CVaR in Model Validation (January 12, 2021). ↩ ↩2 ↩3 See Benjamin Bruder, Nazar Kostyuchyk, Thierry Roncalli, Risk Parity Portfolios with Skewness Risk: An Application to Factor Investing and Alternative Risk Premia, arXiv. ↩ See Joyce A. Hall, Wade Brorsen and Scott H. Irwin, The Distribution of Futures Prices: A Test of the Stable Paretian and Mixture of Normals Hypotheses, The Journal of Financial and Quantitative Analysis, Vol. 24, No. 1 (Mar., 1989), pp.105-116. ↩ See Kon S (1984) Models of stock returns-a comparison. J Financ 39(1):147–16. ↩ See C. Cuevas-Covarrubias, J. Inigo-Martinez and R. Jimenez-Padilla, Gaussian mixtures and financial returns, Discussiones Mathematicae, Probability and Statistics 37 (2017) 101–122. ↩ ↩2 ↩3 See Ian Buckley, David Saunders, Luis Seco, Portfolio optimization when asset returns have the Gaussian mixture distribution, European Journal of Operational Research, Volume 185, Issue 3, 2008, Pages 1434-1461. ↩ ↩2 ↩3 ↩4 See J. Hull, A. White, Value at risk when daily changes in market variables are not normally distributed, Journal of Derivatives 5 (3) (1998) 9–19. ↩ See Yuantong Li, Qi Ma, Sujit K. Ghosh, A Non-Iterative Quantile Change Detection Method in Mixture Model with Heavy-Tailed Components, arXiv. ↩ The selected number of mixture components sometimes also takes into account both the quality of the approximation of the portfolio return distribution and the potential for overfitting. ↩ See Ming-Heng Zhang, Qian-Sheng Cheng, Gaussian mixture modelling to detect random walks in capital markets, Mathematical and Computer Modelling, Volume 38, Issues 5–6, 2003, Pages 503-508. ↩ See J. Worms and S. Touati, “Modelling Program’s Performance with Gaussian Mixtures for Parametric Statistics,” in IEEE Transactions on Multi-Scale Computing Systems, vol. 4, no. 3, pp. 383-395, 1 July-Sept. 2018. ↩ ↩2 See Tao Huang, Heng Peng and Kun Zhang, Model selection for Gaussian mixture models, Statistica Sinica 27 (2017), 147-169. ↩ Yuantong Li, Qi Ma, and Sujit K. Ghosh. 2020. A Non-Iterative Quantile Change Detection Method in Mixture Model with Heavy-Tailed Components. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ‘20). Association for Computing Machinery, New York, NY, USA, 1888–1898.. ↩ Typically, because it is determined independently of the remaining parameters $\left(p_i, \mu_i, \sigma_i \right)$, $i=1..k$. ↩ To avoid theoretical issues, some authors prefer to work with non-unique local maximizers, c.f. for example Peters and Walker24. ↩ See B. Charles Peters, Jr. and Homer F. Walker, An Iterative Procedure for Obtaining Maximum-Likelihood Estimates of the Parameters for a Mixture of Normal Distributions, SIAM Journal on Applied Mathematics, Vol. 35, No. 2 (Sep., 1978), pp. 362-378. ↩ ↩2 ↩3 See Sanjoy Dasgupta, Leonard Schulman, A Two-round Variant of EM for Gaussian Mixtures, arXiv. ↩ This is because the estimation of the parameters of a Gaussian mixture distribution can be formulated as a missing data problem42, for which the EM algorithm has been specifically designed. ↩ In particular, it is non convex. ↩ See Jean-Patrick Baudry, Gilles Celeux. EM for mixtures - Initialization requires special care. 2015. hal-01113242. ↩ ↩2 See Christophe Biernacki, Gilles Celeux, Gérard Govaert, Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models, Computational Statistics & Data Analysis, Volume 41, Issues 3–4, 2003, Pages 561-575. ↩ ↩2 ↩3 ↩4 See Wayne A. Woodward , William C. Parr , William R. Schucany & Hildegard Lindsey (1984) A Comparison of Minimum Distance and Maximum Likelihood Estimation of a Mixture Proportion, Journal of the American Statistical Association, 79:387, 590-598. ↩ ↩2 See M. Kritzman, Y. Li, Skulls, Financial Turbulence, and Risk Management,Financial Analysts Journal, Volume 66, Number 5, Pages 30-41, Year 2010. ↩ Or the underlying multivariate asset returns, if available. ↩ See George Chow, Jacquier, E., Kritzman, M., & Kenneth Lowry. (1999). Optimal Portfolios in Good Times and Bad. Financial Analysts Journal, 55(3), 65–73.. ↩ To be noted that in 1D, the $k$-means algorithm can be computed exactly, see for example Allan Gronlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, Mingzhou Song, Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D, arXiv. ↩ A proprietary methodology is used for that, except when the turbulence partitioning procedure is used together with a percentage-based thresholding of turbulence index values, in which case the number of Gaussian mixture components is equal43 to the number of provided thresholds plus one. ↩ See Sepp, Artur, Optimal Allocation to Cryptocurrencies in Diversified Portfolios (December 31, 2022). Risk Magazine, October 2023, 1-6. ↩ Adapted from Ang et al.6. ↩ ↩2 Annualized. ↩ You will need to trust me on this one, as long as Portfolio Optimizer does not expose an endpoint to fit a Gaussian mixture distribution to an empirical return distribution. ↩ See Greenwood, Joseph A. and Sandomire, Marion M., “Sample Size Required For Estimating The Standard Deviation as a Percent of Its True Value” (1950). U.S. Navy Research. 34.. ↩ The $p$-values are so ridiculously high (> 0.95) that even taking into account the fact that the parameters of the two-component Gaussian mixture distribution have been estimated, the conclusion remains the same. ↩ See McLachlan, Geoffrey J.; Krishnan, Thriyambakam; Ng, See Ket (2004) : The EM Algorithm, Papers, No. 2004,24, Humboldt-Universität zu Berlin, Center for Applied Statistics and Economics (CASE), Berlin. ↩ Except when there are empty partitions. ↩Index Tracking: Reproducing the Performance of a Financial Market Index (and more)2023-10-31T00:00:00-05:002023-10-31T00:00:00-05:00https://portfoliooptimizer.io/blog/index-tracking-reproducing-the-performance-of-a-financial-market-index-and-more<p>An <em>index tracking portfolio</em><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup> is a portfolio designed to track as closely<sup id="fnref:28" role="doc-noteref"><a href="#fn:28" class="footnote">2</a></sup> as possible a financial market index when its exact replication<sup id="fnref:29" role="doc-noteref"><a href="#fn:29" class="footnote">3</a></sup> is either impractical or impossible due to various reasons<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>
(transaction costs, liquidity issues, licensing requirements…).</p>
<p>In this blog post, after reviewing the underlying mathematics described in Hallerbach<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote">5</a></sup>, I will review a couple of applications of index tracking beyond “pure” index tracking, like helping financial advisors to
reduce their customers investment fees or minimizing the sensitivity of mean-variance optimization to estimation error.</p>
<blockquote>
<p><strong><em>Notes:</em></strong></p>
<ul>
<li>A Google Sheet corresponding to this post is available <a href="https://docs.google.com/spreadsheets/d/1U431Zp3viXzJurK0lexV3Oi1iT4P3MrVpHgKRq5IUeU/edit?usp=sharing">here</a></li>
</ul>
</blockquote>
<h2 id="mathematical-preliminaries">Mathematical preliminaries</h2>
<h3 id="the-general-index-tracking-optimization-problem">The general index tracking optimization problem</h3>
<p>Let be:</p>
<ul>
<li>$T$, a number of time periods</li>
<li>$r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods</li>
<li>$r_{tracking} = \left( r_{tracking, 1}, …, r_{tracking, T} \right) \in \mathcal{R}^{T}$, the vector of the (yet to be computed) index tracking portfolio arithmetic returns over each of the $T$ time periods</li>
</ul>
<p>Then:</p>
<ul>
<li>
<p>For a given time period $t$, the difference between the index return $r_{idx, t}$ and the index tracking portfolio return $r_{tracking, t}$ is defined as the <em>tracking error</em> $TE_t$:</p>
\[TE_t = r_{tracking, t} - r_{idx, t}\]
</li>
<li>
<p>For all time periods $t=1..T$, the vector of the tracking errors $TE_t$ is defined as the <em>tracking error vector</em> $TE \in \mathcal{R}^{T}$:</p>
\[TE = \left( r_{tracking, 1} - r_{idx, 1}, ..., r_{tracking, T} - r_{idx, T}\right)\]
</li>
</ul>
<p>Now, let be:</p>
<ul>
<li>$n$, the number of assets in the universe of the index tracking portfolio</li>
<li>$w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets</li>
<li>$r_i = \left( r_{i, 1}, …, r_{i, T} \right) \in \mathcal{R}^{T}$, the vector of the asset $i$ arithmetic returns over each of the $T$ time periods, $i = 1..n$</li>
</ul>
<p>Because arithmetic returns can be aggregated across assets, we have for each time period $t$:</p>
\[r_{tracking, t} \left( w \right) = \sum_{i=1}^n w_i r_{i, t}\]
<p>So that:</p>
\[TE_t \left( w \right) = \sum_{i=1}^n w_i r_{i, t} - r_{idx, t}\]
<p>and:</p>
\[TE \left( w \right) = X w - r_{idx}\]
<p>, with $X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods.</p>
<p>The vector of the index tracking portfolio weights $w^<em>$ is then the<sup id="fnref:31" role="doc-noteref"><a href="#fn:31" class="footnote">6</a></sup> solution to the general index tracking optimization problem which consists in minimizing some <a href="https://en.wikipedia.org/wiki/Loss_function">loss function</a>
$f$ of the tracking error vector $TE$ - called *tracking error measure</em> - subject to additional constraints like full investment constraint, no short sale constraint, etc.<sup id="fnref:2:1" role="doc-noteref"><a href="#fn:2" class="footnote">5</a></sup></p>
\[w^* = \operatorname{argmin} f(TE \left( w \right) ) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \end{cases}\]
<h3 id="tracking-error-measures">Tracking error measures</h3>
<p>A common tracking error measure is the <em>empirical tracking error (ETE)</em><sup id="fnref:4:1" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, also called the <em>mean squared tracking error (MSTE)</em>, defined as the sum of the squared tracking errors $TE_t$, $t=1..T$</p>
\[ETE(w) = \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2\]
<p>or, equivalently, the <em>root mean square tracking error (RMSTE)</em><sup id="fnref:2:2" role="doc-noteref"><a href="#fn:2" class="footnote">5</a></sup>, defined by</p>
\[RMSTE(w) = \sqrt{\frac{1}{T} \lVert X w - r_{idx} \rVert_2^2}\]
<p>Another common tracking error measure is the <em>tracking error variance (TEV)</em><sup id="fnref:2:3" role="doc-noteref"><a href="#fn:2" class="footnote">5</a></sup>, defined as the variance of the tracking errors $TE_t$, $t=1..T$</p>
\[TEV(w) = Var( X w - r_b )\]
<p>However, as noted in Beasley et al.<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote">7</a></sup>, the usage of the tracking error variance is problematic due to its insensitivity to bias in the tracking errors $TE_t$. Indeed, if the index tracking portfolio
returns $r_{tracking}$ are systematically higher or lower than the index returns $r_{idx}$<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote">8</a></sup>, the tracking error variance is null although the index tracking portfolio systematically deviates
from its index!</p>
<p>This limitation is illustrated in Figure 1, adapted from Rossbach and Karlow<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup>, in which it is clearly visible that <em>portfolio 1 has a worse tracking quality than portfolio 2 but the TEV of portfolio 1
is equal to zero</em><sup id="fnref:6:1" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup>.</p>
<figure>
<a href="/assets/images/blog/index-tracking-tev-limitation.png"><img src="/assets/images/blog/index-tracking-tev-limitation.png" alt="Insensitivity of the tracking error variance to bias. Source: Rossbach and Karlow." /></a>
<figcaption>Figure 1. Insensitivity of the tracking error variance to bias. Source: Rossbach and Karlow.</figcaption>
</figure>
<p>In addition to the empirical tracking error and to the tracking error variance, other tracking error measures also exist like the mean absolute deviation of the tracking errors<sup id="fnref:6:2" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup>,
the correlation between the index tracking portfolio returns $r_{tracking}$ and the index returns $r_{idx}$<sup id="fnref:2:4" role="doc-noteref"><a href="#fn:2" class="footnote">5</a></sup>, the downside risk<sup id="fnref:4:2" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>, etc., but they will not be detailed in this blog post.</p>
<p>In all cases, it is important to note that different tracking error measures lead to different solutions to the general index tracking optimization problem - that is, different index tracking portfolios -
with different properties<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote">10</a></sup>.</p>
<h3 id="the-index-trackingempirical-tracking-error-optimization-problem">The index tracking/empirical tracking error optimization problem</h3>
<p>Due to the limitation of the tracking error variance highlighted in the previous sub-section, this blog post will use the empirical tracking error as the preferred tracking error measure<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote">11</a></sup>.</p>
<p>In this case, the general index tracking optimization problem becomes a constrained regression problem in which the tracking errors $TE_t$, $t = 1..T$ represent the error terms:</p>
\[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \end{cases}\]
<p>From this perspective, and <em>although index tracking is driven from the financial industry, it is in fact a pure signal processing problem</em><sup id="fnref:4:3" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup> that can be solved with standard quadratic optimization algorithms.</p>
<h3 id="caveats">Caveats</h3>
<p>The index tracking optimization problem as presented in the previous sub-sections relies on a couple of assumptions that are best to keep in mind:</p>
<ul>
<li>
<p>The<sup id="fnref:31:1" role="doc-noteref"><a href="#fn:31" class="footnote">6</a></sup> solution to the index tracking optimization problem is the portfolio with the lowest in-sample tracking error measure, while out-of-sample performances are ultimately what matters</p>
<p>Rossbach and Karlow<sup id="fnref:6:3" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup> study the stability of standard tracking error measures, including the empirical tracking error and the tracking error variance, and conclude that<sup id="fnref:6:4" role="doc-noteref"><a href="#fn:6" class="footnote">9</a></sup></p>
<blockquote>
<p>The results indicate a poor stability for every of the […] traditional measures. Furthermore, the relation to the ex post tracking quality is weak.</p>
</blockquote>
<p>Nevertheless, this problem can be attenuated by regularly re-computing an index tracking portfolio thanks to a rolling window of recent past returns, c.f. Benidis<sup id="fnref:4:4" role="doc-noteref"><a href="#fn:4" class="footnote">4</a></sup>.</p>
</li>
<li>
<p>The<sup id="fnref:31:2" role="doc-noteref"><a href="#fn:31" class="footnote">6</a></sup> solution to the index tracking optimization problem has fixed asset weights, while a certain level of drift is always present in practice</p>
<p>To make the underlying optimization problem tractable, the index tracking portfolio is supposed to be constantly rebalanced at each in-sample time period $t=1..T$.</p>
<p>This assumption is likely to be violated in practice, so that evaluating the <em>ex-post</em> impact of other rebalancing strategies (buy and hold, etc.) is important.</p>
</li>
</ul>
<h3 id="relationship-with-sharpes-returns-based-style-analysis">Relationship with Sharpe’s returns-based style analysis</h3>
<p><a href="https://en.wikipedia.org/wiki/Returns-based_style_analysis"><em>Returns-based style analysis (RBSA)</em></a> is a methodology initially introduced by Sharpe<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote">12</a></sup><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote">13</a></sup> to evaluate the composition
of a mutual fund in absence of detailed holdings information.</p>
<p>In details, returns-based style analysis consists in determining the combination of major asset classes - represented by market indices - which most closely replicate the actual performances
of a mutual fund over a given time period.</p>
<p>Unsurprisingly, the underlying mathematical formulation is actually equivalent to that of the index tracking/tracking error variance optimization problem<sup id="fnref:3:1" role="doc-noteref"><a href="#fn:3" class="footnote">13</a></sup>, with a specific vocabulary:</p>
<ul>
<li>The index tracking portfolio is called the mutual fund <em>style benchmark</em></li>
<li>The index tracking portfolio weights vector $w$ is called the mutual fund <em>effective asset mix</em></li>
<li>For a given time period $t$
<ul>
<li>The index tracking portfolio return $r_{tracking, t}$ is interpreted as the mutual fund return attributable to <em>style</em>, that is, the mutual fund return originating from its passive exposure to the different asset classes</li>
<li>The tracking error $TE_t$ is interpreted as the mutual fund return attributable to <em>selection</em>, that is, the mutual fund return originating from the skill of the portfolio manager(s)</li>
</ul>
</li>
</ul>
<h3 id="relationship-with-factor-analysis">Relationship with factor analysis</h3>
<p>Sharpe<sup id="fnref:3:2" role="doc-noteref"><a href="#fn:3" class="footnote">13</a></sup> highlights that returns-based style analysis is a specific type of <a href="https://en.wikipedia.org/wiki/Factor_analysis">factor analysis</a></p>
<blockquote>
<p>An asset class factor model can be considered a special case of the generic type. In such a model each factor represents the return on an asset class, and the sensitivities ([beta_i] values) are required to sum to 1 (100%).</p>
</blockquote>
<p>Similarly, from the previous sub-section, index tracking can also be considered as a member of the more general family of methods for portfolio factor analysis.</p>
<h2 id="implementation-in-portfolio-optimizer">Implementation in Portfolio Optimizer</h2>
<p><strong>Portfolio Optimizer</strong> implements three functionalities related to index tracking as presented in the previous section:</p>
<ul>
<li>The computation of the empirical tracking error between a portfolio and an index, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/analysis/empirical-tracking-error</code></a></li>
<li>The computation of the tracking error variance between a portfolio and an index, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/analysis/tracking-error-variance</code></a></li>
<li>The computation of the index tracking portfolio solution to the index tracking optimization problem under the empirical tracking error measure, through the endpoint <a href="https://docs.portfoliooptimizer.io/"><code class="language-plaintext highlighter-rouge">/portfolios/replication/index-tracking</code></a></li>
</ul>
<h2 id="examples-of-usage">Examples of usage</h2>
<h3 id="tracking-an-index-when-no-etf-is-available">Tracking an index when no ETF is available</h3>
<p>I propose to illustrate the “vanilla” usage of an index tracking portfolio when no ETF associated to an index is available due to regulatory reasons.</p>
<p>For the context, residents of France have access to a tax-efficient investment wrapper called <em><a href="https://www.hsbc.fr/en-fr/investissement/bourse-opcvm/produits/plan-epargne-actions/">Plan d’Epargne en Actions (PEA)</a></em>.</p>
<p>A limitation of this investment wrapper, though, is that non-European financial assets cannot be sheltered under it - like global ETFs (MSCI World ETFs, etc.) - unless they have specifically been issued as PEA-compatible.</p>
<p>For example:</p>
<ul>
<li>Investing in an ETF tracking <a href="https://www.msci.com/documents/10199/178e6643-6ae6-47b9-82be-e1fc565ededb">the MSCI World Index</a> is allowed within a PEA thanks for example to the <a href="https://www.amundietf.fr/fr/professionnels/produits/equity/lyxor-pea-monde-msci-world-ucits-etf-capi/fr0011869353"><em>Lyxor PEA Monde (MSCI World) ETF Capi</em> ETF</a><sup id="fnref:32" role="doc-noteref"><a href="#fn:32" class="footnote">14</a></sup></li>
<li>Similarly, investing in an ETF tracking <a href="https://www.msci.com/www/fact-sheet/msci-emerging-markets-index/07149641">the MSCI Emerging Markets (EM) Index</a> is allowed within a PEA thanks for example to the <a href="https://www.amundietf.fr/fr/professionnels/produits/equity/amundi-pea-msci-emerging-emea-esg-leaders-ucits-etf-acc/fr0011440478"><em>Amundi PEA MSCI Emg EMEA ESG Lead UE Acc</em> ETF</a><sup id="fnref:33" role="doc-noteref"><a href="#fn:33" class="footnote">15</a></sup></li>
<li>Yet, investing in an ETF tracking <a href="https://www.msci.com/www/fact-sheet/msci-acwi/06846097">the MSCI All Country World (ACWI) Index</a> is NOT allowed within a PEA, because no PEA-compatible ETF which tracks that index has been issued<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote">16</a></sup></li>
</ul>
<p>Now, for the sake of the exercise, let’s suppose a French DIY investor would like to track the MSCI ACWI Index within her PEA.</p>
<p>As mentioned above, this cannot be done directly since there is no PEA-compatible ETF which tracks this index.</p>
<p>Nevertheless, this can be done indirectly<sup id="fnref:35" role="doc-noteref"><a href="#fn:35" class="footnote">17</a></sup> through an adequate combination of a PEA-compatible MSCI World ETF and a PEA-compatible MSCI EM ETF!</p>
<p>Indeed, Figure 2 compares the evolution, over the period 31 December 2019 - 30 September 2023, of:</p>
<ul>
<li>The PEA-incompatible <a href="https://www.amundietf.fr/fr/professionnels/produits/equity/lyxor-msci-all-country-world-ucits-etf-acc-eur/lu1829220216"><em>Lyxor MSCI All Country World UCITS ETF - Acc</em></a><sup id="fnref:34" role="doc-noteref"><a href="#fn:34" class="footnote">18</a></sup></li>
<li>The PEA-compatible portfolio invested 86% in the <em>Lyxor PEA Monde (MSCI World) ETF Capi</em> ETF and 14% in the <em>Amundi PEA MSCI Emg EMEA ESG Lead UE Acc</em> ETF</li>
</ul>
<figure>
<a href="/assets/images/blog/index-tracking-pea-etfs.png"><img src="/assets/images/blog/index-tracking-pea-etfs.png" alt="PEA-incompatible MSCI ACWI ETF v.s. PEA-compatible MSCI ACWI Index tracking portfolio, 31 December 2019 - 30 September 2023" /></a>
<figcaption>Figure 2. PEA-incompatible MSCI ACWI ETF v.s. PEA-compatible MSCI ACWI Index tracking portfolio, 31 December 2019 - 30 September 2023</figcaption>
</figure>
<p>On this figure, it is visible that the tracking error between the two portfolios is almost null, which confirms that building a custom MSCI ACWI ETF inside a PEA is perfectly feasible<sup id="fnref:16" role="doc-noteref"><a href="#fn:16" class="footnote">19</a></sup>!</p>
<p>Question is, how have the weights of this PEA-compatible MSCI ACWI Index tracking portfolio been computed?</p>
<p>Well, it turns out that the problem at hand is an index tracking problem, with:</p>
<ul>
<li><strong>Index</strong> - The PEA-incompatible <a href="https://www.amundietf.fr/fr/professionnels/produits/equity/lyxor-msci-all-country-world-ucits-etf-acc-eur/lu1829220216"><em>Lyxor MSCI All Country World UCITS ETF - Acc</em></a><sup id="fnref:34:1" role="doc-noteref"><a href="#fn:34" class="footnote">18</a></sup></li>
<li><strong>Tracking assets</strong> - The two PEA-compatible ETFs <em>Lyxor PEA Monde (MSCI World) ETF Capi</em> and <em>Amundi PEA MSCI Emg EMEA ESG Lead UE Acc</em></li>
</ul>
<p>Thus, using the monthly returns of these ETFs over the period 31 December 2019 - 30 September 2023, it is possible to compute the associated index tracking portfolio, which results in the PEA-compatible portfolio above.</p>
<h3 id="reducing-the-fees-of-a-mutual-funds-portfolio">Reducing the fees of a mutual funds portfolio</h3>
<p>In the book <em>Cloning Wall Street</em><sup id="fnref:14" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>, <a href="https://www.jonathanwallentine.com/">Jonathan Wallentine</a> details how financial advisors can use index tracking portfolios - called <em>replicating portfolios</em><sup id="fnref:14:1" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup> -
to reduce portfolio fees for their clients.</p>
<p>As an example from that book<sup id="fnref:23" role="doc-noteref"><a href="#fn:23" class="footnote">21</a></sup>, let’s suppose you are a financial advisor living in 2016 and a prospective client named Jessica comes to see you in order to decrease her mutual funds portfolio annual fees.</p>
<p>Jessica’s portfolio is made of the mutual funds listed<sup id="fnref:15" role="doc-noteref"><a href="#fn:15" class="footnote">22</a></sup> in Figure 3, taken from Wallentine<sup id="fnref:14:2" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>.</p>
<figure>
<a href="/assets/images/blog/index-tracking-cloning-wall-street-mutual-funds.png"><img src="/assets/images/blog/index-tracking-cloning-wall-street-mutual-funds.png" alt="Jessica's portfolio constituents. Source: Wallentine." /></a>
<figcaption>Figure 3. Jessica's portfolio constituents. Source: Wallentine.</figcaption>
</figure>
<p>After some thinking<sup id="fnref:14:3" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>, you conclude that Jessica’s mutual funds portfolio could probably be replaced by a portfolio made of the following low-cost ETFs:</p>
<ul>
<li>U.S. short term Treasuries (SHY ETF)</li>
<li>U.S. aggregate bonds (AGG ETF)</li>
<li>U.S. big caps stocks (SPY ETF)</li>
<li>U.S. small caps stocks (IWM ETF)</li>
<li>International stocks (EFA ETF)</li>
</ul>
<p>Problem is, you don’t have a clue about the proportion to invest in each of these ETFs to properly replace Jessica’s portfolio…</p>
<p>What to do?</p>
<p>Here again, the problem at hand is actually an index tracking problem in disguise, with:</p>
<ul>
<li><strong>Index</strong> - Jessica’s mutual funds portfolio</li>
<li><strong>Tracking assets</strong> - The five selected ETFs SHY, AGG, SPY, IWM and EFA</li>
</ul>
<p>Thus, using the monthly returns of Jessica’s portfolio and of these five ETFs over the past period 2011 - 2015, it is possible to compute<sup id="fnref:14:4" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup> Jessica’s tracking portfolio weights:</p>
<ul>
<li>5% in the SHY ETF</li>
<li>13% in the AGG ETF</li>
<li>54% in the SPY ETF</li>
<li>6% in the IWM ETF</li>
<li>22% in the EFA ETF</li>
</ul>
<p>Figure 4, adapted from Wallentine<sup id="fnref:14:5" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>, depicts the evolution of Jessica’s original portfolio and of Jessica’s tracking portfolio.</p>
<figure>
<a href="/assets/images/blog/index-tracking-cloning-wall-street-mutual-funds-replication.png"><img src="/assets/images/blog/index-tracking-cloning-wall-street-mutual-funds-replication.png" alt="Jessica's original portfolio v.s. Jessica's tracking portfolio, 2011 - 2015. Source: Wallentine." /></a>
<figcaption>Figure 4. Jessica's original portfolio v.s. Jessica's tracking portfolio, 2011 - 2015. Source: Wallentine.</figcaption>
</figure>
<p>In addition to confirming a very small tacking error between the two portfolios, Figure 4 also highlights that Jessica’s tracking portfolio actually <em>outperformed Jessica’s portfolio by 70bps annually</em><sup id="fnref:14:6" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup><sup id="fnref:17" role="doc-noteref"><a href="#fn:17" class="footnote">23</a></sup>.</p>
<p>Now, to come back on Jessica’s original request:</p>
<ul>
<li>Jessica’s mutual funds portfolio has an average expense ratio of 1.13%, which leads to total annual fees of around $11,623<sup id="fnref:14:7" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup></li>
<li>Your alternative low-cost ETFs portfolio has an average expense ratio of 0.06%, which leads to total annual fees of around $617<sup id="fnref:14:8" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup></li>
</ul>
<p>In other words, your alternative portfolio - which is for all intent and purposes equivalent to Jessica’s current portfolio - leads to a 95% decrease in annual fees<sup id="fnref:18" role="doc-noteref"><a href="#fn:18" class="footnote">24</a></sup>.</p>
<p>Jessica is delighted!</p>
<p>To conclude on this example, <em>the results outlined in this [sub-]section should give readers confidence that [an index tracking portfolio] can successfully
replicate a variety of different mutual fund portfolios from large fund companies</em><sup id="fnref:14:9" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>.</p>
<h3 id="automatically-determining-the-proper-benchmark-for-a-mutual-fund">Automatically determining the proper benchmark for a mutual fund</h3>
<p>In his 1992 paper, Sharpe notes that<sup id="fnref:3:3" role="doc-noteref"><a href="#fn:3" class="footnote">13</a></sup></p>
<blockquote>
<p>Style analysis provides a natural method for constructing benchmarks [for performance measurement]. The return obtained by a fund in each month can be compared with the return on a mix of asset classes with the same estimated style, where the style is estimated prior to the month in question.</p>
</blockquote>
<p>Due to the one-to-one relationship between Sharpe’s style analysis and index tracking, the index tracking machinery then allows to easily and automatically construct a benchmark for any mutual fund<sup id="fnref:19" role="doc-noteref"><a href="#fn:19" class="footnote">25</a></sup>.</p>
<p>As an illustration, I propose to revisit <a href="https://twitter.com/Finominally/status/1694360191527977299">a tweet from Nicolas Rabener</a> - the founder and CEO of <a href="https://finominal.com/">Finominal</a> -
in which an interesting discrepancy is reported between:</p>
<ul>
<li>The <a href="https://www.seic.com/asset-management/enhanced-factor-etfs/sei-enhanced-low-volatility-us-large-cap-etf-selv">SEI Enhanced Low Volatility U.S. Large Cap (SELV) ETF</a></li>
<li>The self-reported benchmark<sup id="fnref:21" role="doc-noteref"><a href="#fn:21" class="footnote">26</a></sup> for this ETF - the Russell 1000 Index</li>
</ul>
<p>For this, Figure 5 compares, over the period 01 January 2023 - 26 October 2023, the evolution of:</p>
<ul>
<li>The SELV ETF</li>
<li>The <a href="https://www.ishares.com/us/products/239707/ishares-russell-1000-etf">iShares Russell 1000 (IWB) ETF</a>, representative of the self-reported benchmark for the SELV ETF - the Russel 1000 Index</li>
<li>The <a href="https://www.ishares.com/us/products/239695/">iShares MSCI USA Min Vol Factor (USMV) ETF</a>, representative of the benchmark for the SELV ETF automatically determined by <a href="https://finominal.com/">Finominal</a> - the MSCI USA Minimum Volatility Index</li>
<li>A portfolio made of 13% IWB ETF / 87% USMV ETF, representative of the benchmark for the SELV ETF automatically determined by <strong>Portfolio Optimizer</strong> using
<ul>
<li><strong>Index</strong> - The SELV ETF</li>
<li><strong>Tracking assets</strong> - The IWB and the USMV ETFs</li>
</ul>
</li>
</ul>
<figure>
<a href="/assets/images/blog/index-tracking-selv.png"><img src="/assets/images/blog/index-tracking-selv.png" alt="SELV ETF v.s. misc. other ETFs and optimal benchmark, 01 January 2023 - 26 October 2023." /></a>
<figcaption>Figure 5. SELV ETF v.s. misc. other ETFs and optimal benchmark, 01 January 2023 - 26 October 2023.</figcaption>
</figure>
<p>Figure 5 empirically demonstrates that the Russell 1000 Index is a very bad choice of benchmark for the SELV ETF and that a more realistic benchmark would consist either of:</p>
<ul>
<li>The MSCI USA Minimum Volatility Index, if only one index can be chosen</li>
<li>The portfolio made of 13% Russel 1000 Index / 87% MSCI USA Minimum Volatility Index, if a composite index can be chosen<sup id="fnref:22" role="doc-noteref"><a href="#fn:22" class="footnote">27</a></sup></li>
</ul>
<p>These two alternative benchmarks are consistent with the SELV ETF fact sheet, which states that<sup id="fnref:21:1" role="doc-noteref"><a href="#fn:21" class="footnote">26</a></sup></p>
<blockquote>
<p>The SEI Enhanced Low Volatility U.S. Large Cap ETF seeks to provide long-term capital appreciation by investing primarily in U.S. common stocks, while aiming to experience lower volatility compared to the broad U.S. large cap equity market.</p>
</blockquote>
<p>Coincidentally, Nicolas Rabener recently<sup id="fnref:11:1" role="doc-noteref"><a href="#fn:11" class="footnote">16</a></sup> blogged about different metrics that can help selecting an optimal benchmark for a fund in his article
<a href="https://insights.finominal.com/research-determining-the-optimal-benchmark-for-funds/"><em>Determining the Optimal Benchmark for Funds</em></a>. Among these metrics, the tracking error is found to be the best performing one and
<em>results in nine selections [out of ten] that seem reasonable</em><sup id="fnref:20" role="doc-noteref"><a href="#fn:20" class="footnote">28</a></sup>.</p>
<p>As a side note, I wholeheartedly recommend readers interested in the topic of automatic benchmark selection to dig into the associated <a href="https://insights.finominal.com/">Finominal’s research</a> blog posts.</p>
<h3 id="minimizing-the-sensitivity-of-mean-variance-optimization-to-estimation-error">Minimizing the sensitivity of mean-variance optimization to estimation error</h3>
<p>A stylized fact of Markowitz’s mean-variance framework is the sensitivity of efficient portfolios to estimation error in asset returns and (co)variances<sup id="fnref:25" role="doc-noteref"><a href="#fn:25" class="footnote">29</a></sup>,
which explains why efficient portfolios are sometimes labeled <em>estimation-error maximizers</em><sup id="fnref:24" role="doc-noteref"><a href="#fn:24" class="footnote">30</a></sup>.</p>
<p>This issue is thoroughly analyzed, both theoretically and empirically, by Kinlaw et al.<sup id="fnref:26" role="doc-noteref"><a href="#fn:26" class="footnote">31</a></sup> who conclude that</p>
<blockquote>
<p>[mean-variance optimization] only appears to be hypersensitive when asset classes are close substitutes for each other, because the optimal weights may shift significantly in response to input errors.</p>
</blockquote>
<p>In other words, Kinlaw et al.<sup id="fnref:26:1" role="doc-noteref"><a href="#fn:26" class="footnote">31</a></sup> establish that it is possible to minimize the sensitivity of mean-variance optimization to estimation error by ensuring that assets are as “distinct” as possible.</p>
<p>A practical method for doing so is proposed in David Berns’ book <em>Modern Asset Allocation for Wealth Management</em><sup id="fnref:13" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup>.</p>
<p>In details, Berns’ methodology is a two-step procedure allowing to determine whether an asset in a portfolio is redundant with the other assets also present in that portfolio:</p>
<ul>
<li>Step 1 - Compute an index tracking portfolio - called the <em>mimicking portfolio</em><sup id="fnref:13:1" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup> - using:
<ul>
<li><strong>Index</strong> - The selected asset</li>
<li><strong>Tracking assets</strong> - The other assets present in the portfolio</li>
</ul>
</li>
<li>
<p>Step 2 - Assess the redundancy of the selected asset by <em>looking at the tracking error [variance] between the mimicking portfolio and the asset</em><sup id="fnref:13:2" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup>, because<sup id="fnref:13:3" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup></p>
<blockquote>
<p>If the mimicking portfolio closely resembles [the selected] asset, then the [tracking error variance] will be low, and [the selected] asset is redundant and should be avoided to avoid estimation error sensitivity. However, if the [tracking error variance] for [the selected] asset is high, then [that] asset clearly showcases distinctness in one or more moments and can potentially be a valuable asset to consider for the portfolio.</p>
</blockquote>
</li>
</ul>
<p>In practice, Berns<sup id="fnref:13:4" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup> suggests to use a tracking error variance of around 2% as <em>a reasonable level below which to start flagging assets for potential removal due to redundancy</em><sup id="fnref:13:5" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup>.</p>
<p>Berns<sup id="fnref:13:6" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup> illustrates his proposed methodology using a portfolio made of intermediate-term Treasuries, intermediate-term investment grade bonds, and high-yield bonds.
The mimicking portfolio for each asset is computed from the two other assets and the tracking error variance - called <em>mimicking
portfolio tracking error (MPTE)</em> - between the mimicking portfolio and the asset under consideration is computed as in Figure 6, adapted from Berns<sup id="fnref:13:7" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup>:</p>
<figure>
<a href="/assets/images/blog/index-tracking-bonds-mpte.png"><img src="/assets/images/blog/index-tracking-bonds-mpte.png" alt="Tracking error variance of fixed income mutual funds v.s. associated tracking portfolios, January 1994 - February 2019. Source: Berns." /></a>
<figcaption>Figure 6. Tracking error variance of fixed income mutual funds v.s. associated tracking portfolios, January 1994 - February 2019. Source: Berns.</figcaption>
</figure>
<p>For this specific portfolio, intermediate-term investment grade bonds are flagged as the most redundant asset for reasons detailed in Berns<sup id="fnref:13:8" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup>, but there is no real redundancy between the three assets<sup id="fnref:36" role="doc-noteref"><a href="#fn:36" class="footnote">33</a></sup>.</p>
<p>To be noted that other approaches for the problem of mean-variance optimization sensitivity to estimation error exist - like the computation of
<a href="/blog/mean-variance-optimization-in-practice-subset-resampling-based-efficient-portfolios/">subset resampling-based efficient portfolios</a> or the usage of the <em>nested clustered optimization algorithm (NCO)</em><sup id="fnref:27" role="doc-noteref"><a href="#fn:27" class="footnote">34</a></sup> -
but these alternatives are usually much more mathematically involved than Berns’ <em>modern yet practical</em><sup id="fnref:13:9" role="doc-noteref"><a href="#fn:13" class="footnote">32</a></sup> approach.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Index tracking is a framework which deserves a place in one’s quantitative arsenal thanks to its versatility, as hopefully shown in this blog post.</p>
<p>Next in this series, I will discuss extensions of the basic index tracking problem, like for example methods to construct an index tracking portfolio with a limit on the number of assets it contains.</p>
<p>In the meantime, feel free to track me <a href="https://www.linkedin.com/in/roman-rubsamen/">on LinkedIn</a> or <a href="https://twitter.com/portfoliooptim">on Twitter</a>.</p>
<p>–</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Also called a <em>benchmark replicating portfolio</em>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p>The term <em>closely</em> is loosely defined at this point. <a href="#fnref:28" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p>From its constituent weights. <a href="#fnref:29" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>See <a href="https://ieeexplore.ieee.org/document/8384194">Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:4:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:4:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:4:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:4:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>See <a href="https://link.springer.com/chapter/10.1007/978-3-642-86706-4_7">Hallerbach, W.G. (1994). Index Tracking: Some Techniques and Results. In: Peccati, L., Virén, M. (eds) Financial Modelling. Contributions to Management Science. Physica-Verlag HD</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:2:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:2:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:2:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:2:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:31" role="doc-endnote">
<p>Or a solution, in case multiple solutions exist. <a href="#fnref:31" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:31:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:31:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0377221702004253">J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>That is, if $r_{tracking, t} = r_{idx, t} + c$, $t=1..T$, with $c$ a constant. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>See <a href="https://econpapers.repec.org/paper/zbwfsfmwp/164.htm">Rossbach, Peter and Karlow, Denis, (2011), The stability of traditional measures of index tracking quality, No 164, Frankfurt School - Working Paper Series, Frankfurt School of Finance and Management.</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:6:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:6:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:6:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:6:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>See <a href="https://www.sciencedirect.com/science/article/abs/pii/S0378426698000764">Markus Rudolf, Hans-Jurgen Wolter, Heinz Zimmermann, A linear model for tracking error minimization, Journal of Banking & Finance, Volume 23, Issue 1, 1999, Pages 85-103</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>The empirical tracking error is not without issues, though; for example, it penalizes similarly both positive and negative deviations from the index. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>See Sharpe, William F, Determining a Fund’s Effective Asset Mix. Investment Management Review, December 1988, pp. 59-69. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>See <a href="https://www.pm-research.com/content/iijpormgmt/18/2/7">William F. Sharpe, Asset allocation, Management style and performance measurement, The Journal of Portfolio Management, Winter 1992, 18 (2) 7-19</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:3:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:3:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:3:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:32" role="doc-endnote">
<p>FR0011869353 <a href="#fnref:32" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:33" role="doc-endnote">
<p>FR0011440478 <a href="#fnref:33" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>At the date of the initial publication of this blog post. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:11:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:35" role="doc-endnote">
<p>The rationale is that by construction, all constituents of the MSCI ACWI Index belong either to the MSCI World Index or to the MSCI EM Index. <a href="#fnref:35" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:34" role="doc-endnote">
<p>LU1829220216 <a href="#fnref:34" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:34:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p>Of course, this example of usage is very close to a toy example, but it already illustrates the potential of index tracking based on returns data. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p>See <a href="https://www.jonathanwallentine.com/">Jonathan Wallentine, Cloning Wall Street, Actuarial Management Company</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:14:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:14:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:14:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:14:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:14:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:14:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:14:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:14:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:14:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p>Further details are provided in Wallentine<sup id="fnref:14:10" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>. <a href="#fnref:23" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p>Some of these mutual funds do not exist anymore at the date of publication of this post. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p>As noted in Wallentine<sup id="fnref:14:11" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>, <em>is likely the result of high mutual fund expense ratios in her portfolio (i.e. it is not due to any secret alpha generated by [the index tracking portfolio itself]</em><sup id="fnref:14:12" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>. <a href="#fnref:17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p>One caveat, though, is that the index tracking portfolio needs to be purchased and then regularly re-balanced; also, as mentioned in Wallentine<sup id="fnref:14:13" role="doc-noteref"><a href="#fn:14" class="footnote">20</a></sup>, your management fees as advisor should of course be be taken into account. <a href="#fnref:18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p>Or more generally, a benchmarka for any given asset. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p>C.f. <a href="https://seietfs.filepoint.live/assets/pdfs/SELV_FactSheet.pdf">SELV Fact Sheet / 31th July 2023</a>. <a href="#fnref:21" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:21:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p>While not visible on Figure 5, the portfolio made of 13% IWB ETF / 87% USMV ETF has a lower empirical tracking error against the SELV ETF than the IWB ETF. <a href="#fnref:22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p>See <a href="https://insights.finominal.com/research-determining-the-optimal-benchmark-for-funds/">Determining the Optimal Benchmark for Funds</a>. <a href="#fnref:20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>See <a href="https://link.springer.com/article/10.1007/BF02282040">Broadie, M. Computing efficient frontiers using estimated parameters. Ann Oper Res 45, 21–58 (1993)</a>. <a href="#fnref:25" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p>See <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2387669">Michaud, Richard O., The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal? (1989). Financial Analysts Journal, 1989</a>. <a href="#fnref:24" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>See <a href="https://www.wiley.com/en-br/Asset+Allocation:+From+Theory+to+Practice+and+Beyond-p-9781119817710">William Kinlaw, Mark P. Kritzman, David Turkington, Asset Allocation: From Theory to Practice and Beyond, Wiley Finance</a>. <a href="#fnref:26" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:26:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>See <a href="https://www.wiley.com/en-us/Modern+Asset+Allocation+for+Wealth+Management-p-9781119566946">David M. Berns, Modern Asset Allocation for Wealth Management, Wiley Finance, 2020</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:13:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:13:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:13:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a> <a href="#fnref:13:5" class="reversefootnote" role="doc-backlink">↩<sup>6</sup></a> <a href="#fnref:13:6" class="reversefootnote" role="doc-backlink">↩<sup>7</sup></a> <a href="#fnref:13:7" class="reversefootnote" role="doc-backlink">↩<sup>8</sup></a> <a href="#fnref:13:8" class="reversefootnote" role="doc-backlink">↩<sup>9</sup></a> <a href="#fnref:13:9" class="reversefootnote" role="doc-backlink">↩<sup>10</sup></a></p>
</li>
<li id="fn:36" role="doc-endnote">
<p>And no asset has a mimicking portfolio tracking error greater than 2%. <a href="#fnref:36" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>See <a href="https://ssrn.com/abstract=3469961">Lopez de Prado, Marcos, A Robust Estimator of the Efficient Frontier (October 15, 2016)</a>. <a href="#fnref:27" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Roman R.An index tracking portfolio1 is a portfolio designed to track as closely2 as possible a financial market index when its exact replication3 is either impractical or impossible due to various reasons4 (transaction costs, liquidity issues, licensing requirements…). In this blog post, after reviewing the underlying mathematics described in Hallerbach5, I will review a couple of applications of index tracking beyond “pure” index tracking, like helping financial advisors to reduce their customers investment fees or minimizing the sensitivity of mean-variance optimization to estimation error. Notes: A Google Sheet corresponding to this post is available here Mathematical preliminaries The general index tracking optimization problem Let be: $T$, a number of time periods $r_{idx} = \left( r_{idx, 1}, …, r_{idx, T} \right) \in \mathcal{R}^{T}$, the vector of the index arithmetic returns over each of the $T$ time periods $r_{tracking} = \left( r_{tracking, 1}, …, r_{tracking, T} \right) \in \mathcal{R}^{T}$, the vector of the (yet to be computed) index tracking portfolio arithmetic returns over each of the $T$ time periods Then: For a given time period $t$, the difference between the index return $r_{idx, t}$ and the index tracking portfolio return $r_{tracking, t}$ is defined as the tracking error $TE_t$: \[TE_t = r_{tracking, t} - r_{idx, t}\] For all time periods $t=1..T$, the vector of the tracking errors $TE_t$ is defined as the tracking error vector $TE \in \mathcal{R}^{T}$: \[TE = \left( r_{tracking, 1} - r_{idx, 1}, ..., r_{tracking, T} - r_{idx, T}\right)\] Now, let be: $n$, the number of assets in the universe of the index tracking portfolio $w = \left( w_1,…,w_n \right) \in \mathcal{R}^{n} $ the vector of the weights of a portfolio in each of the $n$ assets $r_i = \left( r_{i, 1}, …, r_{i, T} \right) \in \mathcal{R}^{T}$, the vector of the asset $i$ arithmetic returns over each of the $T$ time periods, $i = 1..n$ Because arithmetic returns can be aggregated across assets, we have for each time period $t$: \[r_{tracking, t} \left( w \right) = \sum_{i=1}^n w_i r_{i, t}\] So that: \[TE_t \left( w \right) = \sum_{i=1}^n w_i r_{i, t} - r_{idx, t}\] and: \[TE \left( w \right) = X w - r_{idx}\] , with $X \in \mathcal{R}^{T \times n}$ the matrix of the $n$ assets arithmetic returns over each of the $T$ time periods. The vector of the index tracking portfolio weights $w^$ is then the6 solution to the general index tracking optimization problem which consists in minimizing some loss function $f$ of the tracking error vector $TE$ - called *tracking error measure - subject to additional constraints like full investment constraint, no short sale constraint, etc.5 \[w^* = \operatorname{argmin} f(TE \left( w \right) ) \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \end{cases}\] Tracking error measures A common tracking error measure is the empirical tracking error (ETE)4, also called the mean squared tracking error (MSTE), defined as the sum of the squared tracking errors $TE_t$, $t=1..T$ \[ETE(w) = \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2\] or, equivalently, the root mean square tracking error (RMSTE)5, defined by \[RMSTE(w) = \sqrt{\frac{1}{T} \lVert X w - r_{idx} \rVert_2^2}\] Another common tracking error measure is the tracking error variance (TEV)5, defined as the variance of the tracking errors $TE_t$, $t=1..T$ \[TEV(w) = Var( X w - r_b )\] However, as noted in Beasley et al.7, the usage of the tracking error variance is problematic due to its insensitivity to bias in the tracking errors $TE_t$. Indeed, if the index tracking portfolio returns $r_{tracking}$ are systematically higher or lower than the index returns $r_{idx}$8, the tracking error variance is null although the index tracking portfolio systematically deviates from its index! This limitation is illustrated in Figure 1, adapted from Rossbach and Karlow9, in which it is clearly visible that portfolio 1 has a worse tracking quality than portfolio 2 but the TEV of portfolio 1 is equal to zero9. Figure 1. Insensitivity of the tracking error variance to bias. Source: Rossbach and Karlow. In addition to the empirical tracking error and to the tracking error variance, other tracking error measures also exist like the mean absolute deviation of the tracking errors9, the correlation between the index tracking portfolio returns $r_{tracking}$ and the index returns $r_{idx}$5, the downside risk4, etc., but they will not be detailed in this blog post. In all cases, it is important to note that different tracking error measures lead to different solutions to the general index tracking optimization problem - that is, different index tracking portfolios - with different properties10. The index tracking/empirical tracking error optimization problem Due to the limitation of the tracking error variance highlighted in the previous sub-section, this blog post will use the empirical tracking error as the preferred tracking error measure11. In this case, the general index tracking optimization problem becomes a constrained regression problem in which the tracking errors $TE_t$, $t = 1..T$ represent the error terms: \[w^* = \operatorname{argmin} \frac{1}{T} \lVert X w - r_{idx} \rVert_2^2 \newline \textrm{s.t. } \begin{cases} \sum_{i=1}^{n} w_i = 1 \newline 0 \leqslant w_i \leqslant 1, i = 1..n \newline ... \end{cases}\] From this perspective, and although index tracking is driven from the financial industry, it is in fact a pure signal processing problem4 that can be solved with standard quadratic optimization algorithms. Caveats The index tracking optimization problem as presented in the previous sub-sections relies on a couple of assumptions that are best to keep in mind: The6 solution to the index tracking optimization problem is the portfolio with the lowest in-sample tracking error measure, while out-of-sample performances are ultimately what matters Rossbach and Karlow9 study the stability of standard tracking error measures, including the empirical tracking error and the tracking error variance, and conclude that9 The results indicate a poor stability for every of the […] traditional measures. Furthermore, the relation to the ex post tracking quality is weak. Nevertheless, this problem can be attenuated by regularly re-computing an index tracking portfolio thanks to a rolling window of recent past returns, c.f. Benidis4. The6 solution to the index tracking optimization problem has fixed asset weights, while a certain level of drift is always present in practice To make the underlying optimization problem tractable, the index tracking portfolio is supposed to be constantly rebalanced at each in-sample time period $t=1..T$. This assumption is likely to be violated in practice, so that evaluating the ex-post impact of other rebalancing strategies (buy and hold, etc.) is important. Relationship with Sharpe’s returns-based style analysis Returns-based style analysis (RBSA) is a methodology initially introduced by Sharpe1213 to evaluate the composition of a mutual fund in absence of detailed holdings information. In details, returns-based style analysis consists in determining the combination of major asset classes - represented by market indices - which most closely replicate the actual performances of a mutual fund over a given time period. Unsurprisingly, the underlying mathematical formulation is actually equivalent to that of the index tracking/tracking error variance optimization problem13, with a specific vocabulary: The index tracking portfolio is called the mutual fund style benchmark The index tracking portfolio weights vector $w$ is called the mutual fund effective asset mix For a given time period $t$ The index tracking portfolio return $r_{tracking, t}$ is interpreted as the mutual fund return attributable to style, that is, the mutual fund return originating from its passive exposure to the different asset classes The tracking error $TE_t$ is interpreted as the mutual fund return attributable to selection, that is, the mutual fund return originating from the skill of the portfolio manager(s) Relationship with factor analysis Sharpe13 highlights that returns-based style analysis is a specific type of factor analysis An asset class factor model can be considered a special case of the generic type. In such a model each factor represents the return on an asset class, and the sensitivities ([beta_i] values) are required to sum to 1 (100%). Similarly, from the previous sub-section, index tracking can also be considered as a member of the more general family of methods for portfolio factor analysis. Implementation in Portfolio Optimizer Portfolio Optimizer implements three functionalities related to index tracking as presented in the previous section: The computation of the empirical tracking error between a portfolio and an index, through the endpoint /portfolios/analysis/empirical-tracking-error The computation of the tracking error variance between a portfolio and an index, through the endpoint /portfolios/analysis/tracking-error-variance The computation of the index tracking portfolio solution to the index tracking optimization problem under the empirical tracking error measure, through the endpoint /portfolios/replication/index-tracking Examples of usage Tracking an index when no ETF is available I propose to illustrate the “vanilla” usage of an index tracking portfolio when no ETF associated to an index is available due to regulatory reasons. For the context, residents of France have access to a tax-efficient investment wrapper called Plan d’Epargne en Actions (PEA). A limitation of this investment wrapper, though, is that non-European financial assets cannot be sheltered under it - like global ETFs (MSCI World ETFs, etc.) - unless they have specifically been issued as PEA-compatible. For example: Investing in an ETF tracking the MSCI World Index is allowed within a PEA thanks for example to the Lyxor PEA Monde (MSCI World) ETF Capi ETF14 Similarly, investing in an ETF tracking the MSCI Emerging Markets (EM) Index is allowed within a PEA thanks for example to the Amundi PEA MSCI Emg EMEA ESG Lead UE Acc ETF15 Yet, investing in an ETF tracking the MSCI All Country World (ACWI) Index is NOT allowed within a PEA, because no PEA-compatible ETF which tracks that index has been issued16 Now, for the sake of the exercise, let’s suppose a French DIY investor would like to track the MSCI ACWI Index within her PEA. As mentioned above, this cannot be done directly since there is no PEA-compatible ETF which tracks this index. Nevertheless, this can be done indirectly17 through an adequate combination of a PEA-compatible MSCI World ETF and a PEA-compatible MSCI EM ETF! Indeed, Figure 2 compares the evolution, over the period 31 December 2019 - 30 September 2023, of: The PEA-incompatible Lyxor MSCI All Country World UCITS ETF - Acc18 The PEA-compatible portfolio invested 86% in the Lyxor PEA Monde (MSCI World) ETF Capi ETF and 14% in the Amundi PEA MSCI Emg EMEA ESG Lead UE Acc ETF Figure 2. PEA-incompatible MSCI ACWI ETF v.s. PEA-compatible MSCI ACWI Index tracking portfolio, 31 December 2019 - 30 September 2023 On this figure, it is visible that the tracking error between the two portfolios is almost null, which confirms that building a custom MSCI ACWI ETF inside a PEA is perfectly feasible19! Question is, how have the weights of this PEA-compatible MSCI ACWI Index tracking portfolio been computed? Well, it turns out that the problem at hand is an index tracking problem, with: Index - The PEA-incompatible Lyxor MSCI All Country World UCITS ETF - Acc18 Tracking assets - The two PEA-compatible ETFs Lyxor PEA Monde (MSCI World) ETF Capi and Amundi PEA MSCI Emg EMEA ESG Lead UE Acc Thus, using the monthly returns of these ETFs over the period 31 December 2019 - 30 September 2023, it is possible to compute the associated index tracking portfolio, which results in the PEA-compatible portfolio above. Reducing the fees of a mutual funds portfolio In the book Cloning Wall Street20, Jonathan Wallentine details how financial advisors can use index tracking portfolios - called replicating portfolios20 - to reduce portfolio fees for their clients. As an example from that book21, let’s suppose you are a financial advisor living in 2016 and a prospective client named Jessica comes to see you in order to decrease her mutual funds portfolio annual fees. Jessica’s portfolio is made of the mutual funds listed22 in Figure 3, taken from Wallentine20. Figure 3. Jessica's portfolio constituents. Source: Wallentine. After some thinking20, you conclude that Jessica’s mutual funds portfolio could probably be replaced by a portfolio made of the following low-cost ETFs: U.S. short term Treasuries (SHY ETF) U.S. aggregate bonds (AGG ETF) U.S. big caps stocks (SPY ETF) U.S. small caps stocks (IWM ETF) International stocks (EFA ETF) Problem is, you don’t have a clue about the proportion to invest in each of these ETFs to properly replace Jessica’s portfolio… What to do? Here again, the problem at hand is actually an index tracking problem in disguise, with: Index - Jessica’s mutual funds portfolio Tracking assets - The five selected ETFs SHY, AGG, SPY, IWM and EFA Thus, using the monthly returns of Jessica’s portfolio and of these five ETFs over the past period 2011 - 2015, it is possible to compute20 Jessica’s tracking portfolio weights: 5% in the SHY ETF 13% in the AGG ETF 54% in the SPY ETF 6% in the IWM ETF 22% in the EFA ETF Figure 4, adapted from Wallentine20, depicts the evolution of Jessica’s original portfolio and of Jessica’s tracking portfolio. Figure 4. Jessica's original portfolio v.s. Jessica's tracking portfolio, 2011 - 2015. Source: Wallentine. In addition to confirming a very small tacking error between the two portfolios, Figure 4 also highlights that Jessica’s tracking portfolio actually outperformed Jessica’s portfolio by 70bps annually2023. Now, to come back on Jessica’s original request: Jessica’s mutual funds portfolio has an average expense ratio of 1.13%, which leads to total annual fees of around $11,62320 Your alternative low-cost ETFs portfolio has an average expense ratio of 0.06%, which leads to total annual fees of around $61720 In other words, your alternative portfolio - which is for all intent and purposes equivalent to Jessica’s current portfolio - leads to a 95% decrease in annual fees24. Jessica is delighted! To conclude on this example, the results outlined in this [sub-]section should give readers confidence that [an index tracking portfolio] can successfully replicate a variety of different mutual fund portfolios from large fund companies20. Automatically determining the proper benchmark for a mutual fund In his 1992 paper, Sharpe notes that13 Style analysis provides a natural method for constructing benchmarks [for performance measurement]. The return obtained by a fund in each month can be compared with the return on a mix of asset classes with the same estimated style, where the style is estimated prior to the month in question. Due to the one-to-one relationship between Sharpe’s style analysis and index tracking, the index tracking machinery then allows to easily and automatically construct a benchmark for any mutual fund25. As an illustration, I propose to revisit a tweet from Nicolas Rabener - the founder and CEO of Finominal - in which an interesting discrepancy is reported between: The SEI Enhanced Low Volatility U.S. Large Cap (SELV) ETF The self-reported benchmark26 for this ETF - the Russell 1000 Index For this, Figure 5 compares, over the period 01 January 2023 - 26 October 2023, the evolution of: The SELV ETF The iShares Russell 1000 (IWB) ETF, representative of the self-reported benchmark for the SELV ETF - the Russel 1000 Index The iShares MSCI USA Min Vol Factor (USMV) ETF, representative of the benchmark for the SELV ETF automatically determined by Finominal - the MSCI USA Minimum Volatility Index A portfolio made of 13% IWB ETF / 87% USMV ETF, representative of the benchmark for the SELV ETF automatically determined by Portfolio Optimizer using Index - The SELV ETF Tracking assets - The IWB and the USMV ETFs Figure 5. SELV ETF v.s. misc. other ETFs and optimal benchmark, 01 January 2023 - 26 October 2023. Figure 5 empirically demonstrates that the Russell 1000 Index is a very bad choice of benchmark for the SELV ETF and that a more realistic benchmark would consist either of: The MSCI USA Minimum Volatility Index, if only one index can be chosen The portfolio made of 13% Russel 1000 Index / 87% MSCI USA Minimum Volatility Index, if a composite index can be chosen27 These two alternative benchmarks are consistent with the SELV ETF fact sheet, which states that26 The SEI Enhanced Low Volatility U.S. Large Cap ETF seeks to provide long-term capital appreciation by investing primarily in U.S. common stocks, while aiming to experience lower volatility compared to the broad U.S. large cap equity market. Coincidentally, Nicolas Rabener recently16 blogged about different metrics that can help selecting an optimal benchmark for a fund in his article Determining the Optimal Benchmark for Funds. Among these metrics, the tracking error is found to be the best performing one and results in nine selections [out of ten] that seem reasonable28. As a side note, I wholeheartedly recommend readers interested in the topic of automatic benchmark selection to dig into the associated Finominal’s research blog posts. Minimizing the sensitivity of mean-variance optimization to estimation error A stylized fact of Markowitz’s mean-variance framework is the sensitivity of efficient portfolios to estimation error in asset returns and (co)variances29, which explains why efficient portfolios are sometimes labeled estimation-error maximizers30. This issue is thoroughly analyzed, both theoretically and empirically, by Kinlaw et al.31 who conclude that [mean-variance optimization] only appears to be hypersensitive when asset classes are close substitutes for each other, because the optimal weights may shift significantly in response to input errors. In other words, Kinlaw et al.31 establish that it is possible to minimize the sensitivity of mean-variance optimization to estimation error by ensuring that assets are as “distinct” as possible. A practical method for doing so is proposed in David Berns’ book Modern Asset Allocation for Wealth Management32. In details, Berns’ methodology is a two-step procedure allowing to determine whether an asset in a portfolio is redundant with the other assets also present in that portfolio: Step 1 - Compute an index tracking portfolio - called the mimicking portfolio32 - using: Index - The selected asset Tracking assets - The other assets present in the portfolio Step 2 - Assess the redundancy of the selected asset by looking at the tracking error [variance] between the mimicking portfolio and the asset32, because32 If the mimicking portfolio closely resembles [the selected] asset, then the [tracking error variance] will be low, and [the selected] asset is redundant and should be avoided to avoid estimation error sensitivity. However, if the [tracking error variance] for [the selected] asset is high, then [that] asset clearly showcases distinctness in one or more moments and can potentially be a valuable asset to consider for the portfolio. In practice, Berns32 suggests to use a tracking error variance of around 2% as a reasonable level below which to start flagging assets for potential removal due to redundancy32. Berns32 illustrates his proposed methodology using a portfolio made of intermediate-term Treasuries, intermediate-term investment grade bonds, and high-yield bonds. The mimicking portfolio for each asset is computed from the two other assets and the tracking error variance - called mimicking portfolio tracking error (MPTE) - between the mimicking portfolio and the asset under consideration is computed as in Figure 6, adapted from Berns32: Figure 6. Tracking error variance of fixed income mutual funds v.s. associated tracking portfolios, January 1994 - February 2019. Source: Berns. For this specific portfolio, intermediate-term investment grade bonds are flagged as the most redundant asset for reasons detailed in Berns32, but there is no real redundancy between the three assets33. To be noted that other approaches for the problem of mean-variance optimization sensitivity to estimation error exist - like the computation of subset resampling-based efficient portfolios or the usage of the nested clustered optimization algorithm (NCO)34 - but these alternatives are usually much more mathematically involved than Berns’ modern yet practical32 approach. Conclusion Index tracking is a framework which deserves a place in one’s quantitative arsenal thanks to its versatility, as hopefully shown in this blog post. Next in this series, I will discuss extensions of the basic index tracking problem, like for example methods to construct an index tracking portfolio with a limit on the number of assets it contains. In the meantime, feel free to track me on LinkedIn or on Twitter. – Also called a benchmark replicating portfolio. ↩ The term closely is loosely defined at this point. ↩ From its constituent weights. ↩ See Konstantinos Benidis; Yiyong Feng; Daniel P. Palomar, Optimization Methods for Financial Index Tracking: From Theory to Practice , now, 2018.. ↩ ↩2 ↩3 ↩4 ↩5 See Hallerbach, W.G. (1994). Index Tracking: Some Techniques and Results. In: Peccati, L., Virén, M. (eds) Financial Modelling. Contributions to Management Science. Physica-Verlag HD. ↩ ↩2 ↩3 ↩4 ↩5 Or a solution, in case multiple solutions exist. ↩ ↩2 ↩3 See J.E. Beasley, N. Meade, T.-J. Chang, An evolutionary heuristic for the index tracking problem, European Journal of Operational Research, Volume 148, Issue 3, 2003, Pages 621-643. ↩ That is, if $r_{tracking, t} = r_{idx, t} + c$, $t=1..T$, with $c$ a constant. ↩ See Rossbach, Peter and Karlow, Denis, (2011), The stability of traditional measures of index tracking quality, No 164, Frankfurt School - Working Paper Series, Frankfurt School of Finance and Management.. ↩ ↩2 ↩3 ↩4 ↩5 See Markus Rudolf, Hans-Jurgen Wolter, Heinz Zimmermann, A linear model for tracking error minimization, Journal of Banking & Finance, Volume 23, Issue 1, 1999, Pages 85-103. ↩ The empirical tracking error is not without issues, though; for example, it penalizes similarly both positive and negative deviations from the index. ↩ See Sharpe, William F, Determining a Fund’s Effective Asset Mix. Investment Management Review, December 1988, pp. 59-69. ↩ See William F. Sharpe, Asset allocation, Management style and performance measurement, The Journal of Portfolio Management, Winter 1992, 18 (2) 7-19. ↩ ↩2 ↩3 ↩4 FR0011869353 ↩ FR0011440478 ↩ At the date of the initial publication of this blog post. ↩ ↩2 The rationale is that by construction, all constituents of the MSCI ACWI Index belong either to the MSCI World Index or to the MSCI EM Index. ↩ LU1829220216 ↩ ↩2 Of course, this example of usage is very close to a toy example, but it already illustrates the potential of index tracking based on returns data. ↩ See Jonathan Wallentine, Cloning Wall Street, Actuarial Management Company. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 Further details are provided in Wallentine20. ↩ Some of these mutual funds do not exist anymore at the date of publication of this post. ↩ As noted in Wallentine20, is likely the result of high mutual fund expense ratios in her portfolio (i.e. it is not due to any secret alpha generated by [the index tracking portfolio itself]20. ↩ One caveat, though, is that the index tracking portfolio needs to be purchased and then regularly re-balanced; also, as mentioned in Wallentine20, your management fees as advisor should of course be be taken into account. ↩ Or more generally, a benchmarka for any given asset. ↩ C.f. SELV Fact Sheet / 31th July 2023. ↩ ↩2 While not visible on Figure 5, the portfolio made of 13% IWB ETF / 87% USMV ETF has a lower empirical tracking error against the SELV ETF than the IWB ETF. ↩ See Determining the Optimal Benchmark for Funds. ↩ See Broadie, M. Computing efficient frontiers using estimated parameters. Ann Oper Res 45, 21–58 (1993). ↩ See Michaud, Richard O., The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal? (1989). Financial Analysts Journal, 1989. ↩ See William Kinlaw, Mark P. Kritzman, David Turkington, Asset Allocation: From Theory to Practice and Beyond, Wiley Finance. ↩ ↩2 See David M. Berns, Modern Asset Allocation for Wealth Management, Wiley Finance, 2020. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 And no asset has a mimicking portfolio tracking error greater than 2%. ↩ See Lopez de Prado, Marcos, A Robust Estimator of the Efficient Frontier (October 15, 2016). ↩