The objective of this blog post is to outline a methodology that could be used by the Federal Reserve to assess the severity of supervisory stress scenarios. The Federal Reserve could use this or a similar methodology to design scenarios that meet a severity standard that they would announce after receiving public comment. Establishing such a standard would improve transparency and accountability and reduce excessive variability in a bank’s capital requirements over time. We are not proposing that scenarios be made more or less severe than they are currently, but simply that the Federal Reserve announce and stick to a measurable standard.
Federal Reserve officials have suggested that the Federal Reserve might provide notice and seek comment on its stress test scenarios which we support as good policy and legally required.1 However, such a process would be largely meaningless if there were no established regulatory standard against which to measure the proposed scenario; comment would be uninformed and appeal impossible. Thus, we believe that the Federal Reserve should, prior to developing its 2019 CCAR scenario, propose and finalize a rule specifying what that scenario is designed to achieve. The Federal Reserve has already outlined such a standard through guidance, but as this note and others demonstrate, it has not adhered to it in practice. Even if the Federal Reserve were simply to propose and re-adopt that existing standard, doing so in combination with a methodology broadly similar to the one proposed here to evaluate scenarios against that standard would materially enhance transparency and accountability in its stress testing.
We propose that scenario severity be measured as the probability that economic conditions will be worse for banks than the conditions assumed in the scenario. In our previous work, we have mostly focused on the path for the unemployment rate even though the scenario includes many other macroeconomic variables such as house prices and various interest rates in large part because, in the case of the unemployment rate, “worse” always corresponds to ”higher (see here, here, here, and here).” Judging severity using the unemployment rate alone requires answering the relatively straightforward statistical question “What is the probability the unemployment rate will be higher than what is assumed in the stress scenario?”
However, defining “worse” for a multiple variable scenario is more complicated. For example, which is worse, a modest increase in the unemployment rate and a sharp drop in home prices or a sharp increase in the unemployment rate and a modest drop in home prices? To map multiple macroeconomic variables into a single metric, we develop a model of aggregate bank performance and estimate the sensitivity of performance to macroeconomic conditions. We define a scenario (that is, a set of macroeconomic conditions) as “worse” than another scenario (such as the Fed’s stress test scenario) if it leads to greater bank losses and therefore a larger decline in banks’ regulatory capital ratios.
To determine the severity of the scenario, we estimate the probability that economic conditions will turn out worse than those of the scenario. To calculate that probability, we simulate a model of the macro economy 5 million times and calculate the fraction of times conditions lead to a larger decline in regulatory capital ratios.
Using this measure, we find that the 2017 and especially the 2018 stress test scenarios were significantly more severe than the Great Recession. The Federal Reserve has established in Regulation YY that it intends to design its stress test scenarios to mimic conditions that have characterized severe post-war U.S. recessions; consequently, our results suggest the neither the 2017 nor the 2018 standard met the Federal Reserve’s own announced standard.2 We encourage the Federal Reserve to evaluate its scenarios in the manner proposed here using its own models.
A model of bank performance
The first step in the procedure is developing and estimating a simple model of aggregate bank performance. The model is estimated using data from the FR Y-9C regulatory reports on banks’ return-on assets (ROA, the ratio of banks’ net income to total assets) between the first quarter of 1990 and the third quarter of 2017, and macroeconomic data published by the Federal Reserve. The model for bank ROA includes three key variables that are included in the Fed’s supervisory scenarios—the unemployment rate, the house price index and the corporate bond BBB credit spread.3
The chart below depicts the actual impact of the 2007-2009 financial crisis on ROA accumulated over the 9-quarter stress test horizon used in the United States as well as the impact of the stress scenario included in the two most recent CCAR exercises. Accumulated ROA (profits or losses) over the stress test horizon measures the projected impact of profits on bank capital. The lower the level of cumulative ROA over a given test horizon, the larger is the projected reduction in common equity tier 1 capital under the stress scenario. During the 2007-2009 financial crisis, cumulative ROA reached a trough of -2.6 percent in the fourth quarter of 2008. That is, during the financial crisis banks’ losses pulled down their equity-to-asset ratios in aggregate by a bit over 2½ percentage points. Using the model, cumulative ROA is projected to have reached a trough of -4.1 percent under CCAR 2017 (grey line) and -6.0 percent under CCAR 2018 (red line).
Thus, the troughs in projected cumulative ROA achieved using the macroeconomic shocks of the previous two stress test scenarios are considerably below the realized value of cumulative ROA registered during the 2007-2009 crisis. These results are consistent with our earlier research, which shows that the severely adverse scenario in CCAR is much more severe than the conditions that prevailed in the 2007-2009 recession. The advantage of this approach is that we can map the overall impact of scenario severity on bank profitability allowing for a more holistic evaluation of scenario severity. Moreover, the CCAR 2018 severely adverse scenario is significantly more stressful than the 2017 severely adverse scenario. We didn’t include cumulative ROA under the CCAR 2016 severely adverse scenario in the chart, because the projections are similar to those obtained under CCAR 2017. As a result, the degree of severity of the 2018 severely adverse scenario rose significantly relative to 2017 severely adverse scenario, which appears to be excessive even under the supervisors’ objective to make scenarios countercyclical.
Estimating the severity of a scenario
The next step is to estimate the probability that economic conditions “worse” than a specific scenario will occur, conditional on the current economic situation. To do so, we use a standard macroeconomic model, namely, a three-variable vector autoregression with two lags, to generate many hypothetical paths (5 million) for the macroeconomic variables included in the ROA model. We then fit the estimated model for ROA using the simulated macroeconomic outcome to derive projected cumulative ROA associated with that simulation.
To evaluate the severity of any given macroeconomic scenario, we fit the model for ROA to the conditions in that scenario and calculate the associated level of cumulative ROA over the 9-quarter horizon. The probability that conditions will be worse than that scenario is defined as the fraction of simulations of the macroeconomic model that give rise to projected cumulative ROA that falls below the level estimated for the scenario being assessed. We propose using that probability as a metric for the “severity” of that scenario.
For example, we estimate that economic conditions worse than the CCAR 2017 and CCAR 2018 stress test scenarios have a 0.19 and 0.02 basis point probability of occurring. As a point of reference, economic conditions worse than the 2007-2009 financial crisis have a 3 basis point probability of occurring. Thus, the scenarios are considerably more severe than the great recession.
We recognize that these probabilities are all implausibly low. For example, an event with a 3 basis point chance of occurring would have had a 50 percent chance of occurring once since Rome destroyed Carthage. However, as described in the next section, what matters for implementation of the methodology is not the precise level of the probabilities, but rather the relative probabilities of different scenarios.
An example: Using the last financial crisis as a guideline
The Federal Reserve could, for example, announce that it intended for its stress test scenario to generate declines in banks’ capital ratios greater than, but similar to, those experienced during the crisis. Specifically, the Federal Reserve could propose that projected cumulative ROA under a stress test scenario be greater than but about equal to the cumulative ROA experienced during the financial crisis, which, as noted above, was -2.6 percent. An alternative approach would be to set as a guideline that the scenario be close to but more severe than the conditions experienced during the crisis; this equals 3 basis points under our results, but the precise probability would depend on the model. We prefer a guideline based on actual cumulative ROA (-2.6 percent) for three reasons:
1. It is more conservative. Because the fitted values of our model for ROA are smoother than the realized values (as is true for all models), the macroeconomic conditions of the crisis correspond to a fitted cumulative ROA of -1.6 percent, above the realized value of -2.6 percent. Such conservatism would allow for the possibility that conditions in the next crisis became worse than those experienced during the last crisis, and bank performance becoming more sensitive to a deterioration in the economy, an outcome consistent with there being no government intervention to support the financial system. Moreover, because our model is estimated without the counterfactual event “crisis with no government intervention” the coefficients indicate less sensitivity of bank performance to macroeconomic outcomes than would otherwise be the case. That reduced sensitivity results in the guideline being tougher because it takes worse economic outcomes to generate the target level of losses.
2. It is countercyclical. When times are good, the probability of reaching the -2.6 percent threshold is lower compared to when economic conditions are bad. Therefore, specifying scenario guidelines in terms of an ROA target instead of a probability, generates scenarios that are more severe when times are good and less severe when times are bad.
3. It is simpler. The guideline only requires a model for ROA while the probability approach requires a model for the macroeconomy and simulations.
The charts just above compare the three key macroeconomic series included in the ROA model consistent with our scenario guideline (in orange) against the CCAR 2018 scenario (red) and the 2007-2009 financial crisis (purple). There are, of course, an infinite number of scenarios that generate cumulative ROA of about -2.6 percent, but in practice, most of the Monte Carlo simulations of that severity are similar to the “guideline” scenario plotted above. In the hypothetical guideline scenario, the increase in the unemployment rate as well as the decline in the house price index are less extreme than the shocks assumed in CCAR 2018. For example, the unemployment rate peaks at 10 percent in CCAR 2018, while it peaks at 8.7 percent in the hypothetical scenario. In addition, the Federal Reserve’s scenarios are made significantly more severe by front-loading all the stress early in the 9-quarter projection. Similarly, the decline of the house price index in CCAR 2018 is significantly more severe than the one generated using our methodology. The nominal house price index declined 25 percent in CCAR 2018, while it fell 5 percent under our proposed scenario. On the other hand, the path of the corporate BBB spread is similar under the Federal Reserve’s CCAR 2018 scenario and the series obtained using our hypothetical procedure. In both scenarios, the increase in the bond spread is more severe than the financial crisis.
In practice, if the Federal Reserve were to use this approach to design a scenario that slightly exceeded bank losses in the crisis, it could, of course, adjust each of the paths for the three variables as long as the combined severity was unaffected. There is nothing special about the precise scenario we have provided as an illustration. In the crisis, for example, house prices fell by more, but the BBB spread and the unemployment rate increased more slowly.
Extensions
There are various ways to extend this proposal. First, the model that relates banks’ profits and capital to macroeconomic shocks can be made more granular along the lines of Hirtle et al (2015). This would allow for the inclusion of additional macroeconomic series, such as the term spread, the stock market index, etc. Second, it would be straightforward to extend this approach to assess the coherence between the supervisory macroeconomic scenarios and the global market shock.
1 See Vice Chairman Quarles recent testimony.
2 See https://www.federalreserve.gov/bankinforeg/bcreg20131107a1.pdf.
3 All variables enter in the model contemporaneously and the unemployment rate as well as the corporate bond spread enter the regression in levels while the house price index is included using its quarterly growth rate. All coefficients have economically intuitive signs, are statistically significant at conventional levels, and the overall fit of the model is about 75 percent. Note that we tried to include the lagged value of ROA as an explanatory variable, but the coefficient was not statistically different from zero so it was excluded from the model.
Disclaimer: The views expressed in this post are those of the author(s) and do not necessarily reflect the position of the Bank Policy Institute or its membership, and are not intended to be, and should not be construed as, legal advice of any kind.
