What Are the Primary Difficulties in Backtesting Expected Shortfall Models?

Concept

The Illusion of a Single Number

The transition from Value at Risk (VaR) to Expected Shortfall (ES) within regulatory frameworks like the Fundamental Review of the Trading Book (FRTB) was driven by a clear mandate ▴ to capture the nature of tail risk that VaR systematically overlooks. VaR identifies the threshold of a potential loss at a specific confidence level, but it offers no information about the severity of losses that exceed this point. ES, defined as the conditional expectation of loss given that the loss is beyond the VaR level, was designed to address this precise deficiency by averaging the losses in the tail. This provides a more comprehensive view of potential downside risk.

The core challenge in backtesting ES arises from this very improvement. Backtesting VaR is a comparatively direct process of counting the frequency of breaches, or exceedances, over a given period. An ES forecast, conversely, is an average of unobserved events in the tail of a distribution, making its validation a far more complex statistical problem. The difficulties are not merely procedural; they are deeply rooted in the mathematical properties of the risk measure itself.

Elicitability and the Backtesting Dilemma

A central issue complicating the backtesting of ES is the mathematical property of elicitability. A risk measure is considered elicitable if it is the solution to minimizing the expected value of a specific scoring function. VaR possesses this property, which allows for direct comparison and ranking of different VaR models. Gneiting’s 2011 finding that ES is not elicitable introduced a significant hurdle, suggesting that robust, direct backtesting of ES might be inherently more challenging than for VaR.

This lack of elicitability means there is no straightforward scoring function that can be used to objectively assess the accuracy of an ES forecast in the same way one can for a VaR forecast. Consequently, backtesting procedures for ES must rely on indirect methods, often by decomposing ES into a series of VaR forecasts at various confidence levels or by developing more complex statistical tests that do not depend on elicitability. This introduces layers of modeling assumptions and potential for error that are absent in the simpler VaR backtesting frameworks.

The fundamental challenge of backtesting Expected Shortfall stems from its nature as an average of tail losses, a characteristic that makes it statistically robust but difficult to validate directly.

Data Scarcity in the Tail

Another primary difficulty in backtesting ES models is the inherent scarcity of data related to extreme market events. ES is specifically designed to measure risk in the tail of the profit and loss (P&L) distribution. These tail events, by their very definition, occur infrequently. A robust backtest requires a sufficient number of observations to make statistically significant conclusions.

When backtesting a 97.5% ES, as mandated by FRTB, the model is concerned with events that happen only 2.5% of the time. Over a standard one-year backtesting period of approximately 250 trading days, this translates to only a handful of expected exceedances. This small sample size makes it extremely difficult to distinguish between a genuinely flawed model and one that has simply been unlucky. The statistical power of any backtest is severely diminished with such a limited number of data points, increasing the probability of both failing to identify a bad model (a Type II error) and incorrectly penalizing a good model (a Type I error). This data sparsity is a practical constraint that affects all ES backtesting methodologies, regardless of their theoretical sophistication.

Strategy

Decomposition as a Strategic Imperative

Given the non-elicitable nature of Expected Shortfall, a primary strategic approach to backtesting involves decomposing the ES forecast into components that are themselves elicitable. This strategy circumvents the core mathematical problem by transforming the validation of a single, complex measure into the validation of multiple, simpler ones. The most common application of this principle is to view ES as an integral of VaRs across different confidence levels. By backtesting a series of VaR models at various high quantiles (e.g.

97.5%, 98%, 98.5%, 99%), an institution can build a comprehensive picture of the model’s performance in the tail of the distribution. A model that consistently performs well across this spectrum of VaR levels is likely to produce a reliable ES forecast. This multinomial testing framework, where the number of exceedances at each VaR level is simultaneously evaluated, offers greater statistical power than a simple binomial test at a single VaR level. It allows risk managers to identify not just whether the model is underestimating risk, but also where in the tail the miscalibration is most severe.

Comparative Analysis of Backtesting Approaches

The strategic decision of which backtesting methodology to employ involves a trade-off between statistical power, implementation complexity, and diagnostic usefulness. The table below outlines the key characteristics of several prominent approaches.

The Overlapping Nature of Observations

A significant strategic challenge, particularly under the FRTB framework, is the use of overlapping time series for P&L data. The regulations require banks to calculate ES based on a 10-day liquidity horizon. However, to generate a sufficiently long time series for backtesting, these 10-day P&L figures are often calculated on a daily basis, creating overlapping observations. For example, the 10-day P&L calculated on Tuesday shares nine days of data with the 10-day P&L calculated on Monday.

This overlapping introduces significant serial correlation, or autocorrelation, into the backtesting data. Standard backtesting techniques, which assume that observations are independent and identically distributed (i.i.d.), become unreliable in the presence of such autocorrelation. The presence of serial correlation can distort the results of backtests, making a model appear more accurate than it actually is by smoothing out the volatility of the P&L series. Addressing this requires the use of more sophisticated statistical methods, such as the Hansen-Hodrick correction or block bootstrap techniques, to adjust the standard errors of the test statistics for the presence of autocorrelation. The failure to account for this overlapping data structure can lead to a dangerously misplaced sense of confidence in a risk model’s performance.

Effective backtesting strategy requires decomposing the complex ES measure into a series of verifiable VaR-level tests, thereby building a mosaic of evidence about the model’s tail performance.

Model Risk and the Problem of Pro-Cyclicality

A broader strategic difficulty in backtesting ES models is managing the inherent model risk and the potential for pro-cyclicality. Backtesting is, by its nature, a historical exercise. It evaluates a model’s performance based on past data. A model that performs well during a period of low volatility may fail spectacularly during a market crisis.

This is the essence of model risk ▴ the danger that a model is only a good fit for a specific historical regime. Furthermore, the regulatory consequences of backtesting failures can introduce pro-cyclicality into the financial system. Under FRTB, a model that fails its backtest is subject to a capital multiplier, increasing the bank’s capital requirements. This penalty is most likely to be triggered during a period of high market stress, precisely when the bank’s capital is most constrained.

This can force the bank to reduce its risk-taking and lending activities, exacerbating the economic downturn. A strategic approach to backtesting must therefore go beyond a simple pass/fail assessment. It should involve rigorous stress testing and scenario analysis to understand how the model performs under a wide range of market conditions, not just those observed in the recent past. This forward-looking perspective is essential for mitigating model risk and avoiding the unintended consequences of a purely backward-looking validation process.

Execution

Operationalizing the Acerbi-Szekely Test

One of the more robust direct backtests for ES that has emerged is the methodology proposed by Acerbi and Szekely. This test is designed to assess the accuracy of ES forecasts without being confounded by the measure’s lack of elicitability. The core of the test is a statistic, denoted as Z, which compares the realized P&L on days when a VaR breach occurs to the forecasted ES for those days.

The test is constructed to have an expected value of zero under the null hypothesis that the ES model is correctly specified. A negative value of the test statistic indicates that the model is, on average, underestimating the magnitude of tail losses.

The test statistic is calculated as follows:

1. Identify VaR Breaches ▴ Over the backtesting period (e.g. 250 days), identify all days where the actual loss exceeded the forecasted 97.5% VaR. Let N be the total number of such breaches.
2. Calculate Individual Test Statistics ▴ For each day t that a breach occurred, calculate the ratio of the realized loss (L_t) to the forecasted Expected Shortfall (ES_t) for that day. Sum these values.
3. Normalize The Statistic ▴ The final test statistic Z is calculated by summing the ratios from the previous step and normalizing by the number of breaches and the forecasted VaR. The formula is ▴ Z = Σ (L_t / ES_t) / N

A value of Z significantly less than 1 suggests that the model’s ES forecasts are systematically lower than the realized losses during tail events, indicating a model failure. The critical value for the test can be determined through simulation or by reference to pre-computed tables.

Hypothetical Backtest Execution Data

The following table provides a simplified example of the data required to execute the Acerbi-Szekely backtest over a 10-day period with a 97.5% VaR and ES. In this scenario, two VaR breaches occur.

In this example, N=2. The sum of the L_t / ES_t values is 1.037 + 1.167 = 2.204. The test statistic Z would be 2.204 / 2 = 1.102. This value would then be compared to the appropriate critical value to determine if the model passes or fails the backtest.

Managing Data Infrastructure and Model Governance

The execution of a robust ES backtesting program requires a sophisticated data and technology infrastructure. The system must be capable of storing and processing vast quantities of historical market and P&L data. This includes:

• A Centralized Data Repository ▴ A data warehouse or data lake that can store granular historical P&L vectors at the individual trade or position level. This allows for the re-aggregation of P&L under different hypothetical scenarios.
• A High-Performance Risk Engine ▴ The risk engine must be capable of running daily VaR and ES calculations across the entire trading book in a timely manner. It must also support the simulation-based approaches required for some of the more advanced backtesting methodologies.
• A Model Governance Framework ▴ A clear and well-documented model governance process is essential. This includes policies for model validation, performance monitoring, and periodic recalibration. All backtesting results, including any exceptions and the remedial actions taken, must be meticulously documented for both internal audit and regulatory review.

Successful execution of ES backtesting hinges on a robust technological infrastructure and a disciplined model governance framework capable of handling the statistical complexities and data intensity of the process.

A Case Study in Backtesting Failure

Consider a regional bank that implemented a new ES model for its trading book based on a historical simulation approach with a relatively short look-back period of one year. For the first two years of operation, the model passed its VaR backtests with flying colors, experiencing a number of breaches consistent with the 97.5% confidence level. The bank’s risk management committee grew confident in the model’s performance.

In the third year, a sudden geopolitical event triggered a sharp, albeit brief, market downturn. The bank’s trading book, which was heavily exposed to credit spreads, experienced several days of unprecedented losses. The ES model, which had been calibrated on a period of relative market calm, failed to predict the magnitude of these losses. The realized losses on the breach days were, on average, 30% higher than the forecasted ES.

An application of the Acerbi-Szekely test yielded a Z-statistic that was deep in the rejection region, indicating a catastrophic model failure. The subsequent regulatory review resulted in a significant capital add-on, forcing the bank to curtail its trading activities. The post-mortem analysis revealed that the short look-back period had made the model insensitive to tail events, and the reliance on a simple VaR backtest had created a false sense of security. This case underscores the necessity of employing direct ES backtests and conducting rigorous stress testing to supplement standard backtesting procedures.

Reflection

Beyond a Passing Grade

The intricate challenges of backtesting Expected Shortfall models compel a fundamental re-evaluation of what it means to validate a risk system. The process transcends a mere regulatory check-box exercise. It becomes a deep, diagnostic exploration of a model’s capacity to perform under duress. The statistical hurdles, from non-elicitability to data scarcity, are not inconveniences to be engineered around; they are signals that demand a more sophisticated and humble approach to risk modeling.

An operational framework that relies solely on the output of a single backtest, however robust, is brittle. A resilient system, conversely, integrates the quantitative results of multiple, diverse backtests with qualitative oversight and forward-looking stress scenarios. The ultimate objective is not simply to achieve a passing score on a historical test. The goal is to cultivate a profound, systemic understanding of a model’s limitations and breaking points before they manifest in a live environment. This perspective transforms backtesting from a retrospective audit into a proactive instrument of institutional resilience.