In this note, we document undesirable variability in the Federal Reserve’s supervisory stress tests and the resulting uncertainty for banks in capital planning. We then identify some of the sources of that volatility and recommend ways to decrease it, all consistent with enhancing the accuracy of the tests.
BACKGROUND: STRESS TESTS AND THE VARIABILITY IN CAPITAL REQUIREMENTS THEY PRODUCE
The stress tests as a component of regulatory capital. The post-crisis regulatory capital regime is quite complex. Currently, the largest banks are subject to 18 capital requirements. Of these, eight must be met each quarter (Basel III, or the non-stressed requirements), and the remaining 10 are based on projections under hypothesized stress conditions (stressed or stress test requirements). There are actually five requirements in the stress tests, but since the Fed presents each firm with two different stress scenarios—adverse and severely adverse—the combined number of stressed requirements is 10.
In addition, the Basel III and stressed capital requirements have both risk-based and non-risk-based requirements. The risk-based requirements apply either standardized or internal model-based risk-weights to different types of assets.[1] For example, the risk-weight of a loan to a corporation generally under the standardized approach is set at 100 percent, while a Treasury security is assigned a risk-weight of zero. In contrast, non-risk-based requirements assume a uniform risk-weight of 100 percent for any asset.
The relative importance of stress tests in determining bank capital. For the banks that participate in the stress tests, a question often asked is, which of the 18 regulatory capital ratios is the most likely to bind for each bank?
As shown in the bar chart, the proportion of banks for which the stress test capital requirements are binding (the sum of the dark and light orange bars) consistently exceeds more than half but varies widely year to year. For instance, this jumped from 55 percent of banks in 2017 to almost 98 percent of banks in 2018. The chart also illustrates the volatility in the type of capital requirement that is binding over time. Non-risk-based requirements in the stress tests were binding for more than 70 percent of the banks in 2018, compared with less than 20 percent in 2015.
In sum, stress tests appear to be binding frequently enough to be a primary determinant of how much capital a bank chooses to hold, based on its projection of how it will perform in the upcoming stress tests.
SOURCES OF VARIATION IN STRESS TEST RESULTS
We have identified three primary sources of volatility for banks in projecting the results of future supervisory stress tests:
- Annual changes in the stress scenarios, which we documented in earlier research here.
- Periodic changes to the (non-public) models that the Fed uses to calculate stress losses.
- Excessive sensitivity of the projections generated by the Fed models to changes in the stress scenario—in other words, how a change in the stress scenario maps into projected losses and revenues under the Fed models.
Factors two and three might be manageable for banks if the supervisory models were known. Opacity of the supervisory models—or not knowing how a given model maps a change in supervisory scenarios or in bank risk exposures into stressed losses and capital – makes it impossible for banks to anticipate and mitigate the excessive volatility arising from those two factors.
One major factor that produces variability in stress test results would be the actual changes in bank exposures. This variability is altogether appropriate and intended. However, the three sources of volatility listed above appear spurious and undesirable. (Note that variation attributable to changes to the stress scenario could be seen as appropriate if the stress scenario were probabilistic or based on unfolding economic conditions—that is, if it were designed to reflect stress from a risk that the Fed were predicting to occur over the coming stress period. But the Fed has avowed that the stress scenarios are not probabilistic in design, and we are aware of no evidence to show that they have been probabilistic in effect.)
Problems of spurious volatility and resulting uncertainty will be exacerbated if the Federal Reserve, as it has stated is its intent, adopts a stress capital buffer (equal to prior stress losses) to replace the capital conservation buffer in non-stressed capital requirements for banks. In the current CCAR process, there is flexibility to adjust dividends and share buybacks by taking a mulligan. It will be harder to avoid the impact of stress test surprises on bank capital plans if the stress capital buffer is added to the non-stressed capital requirements. Banks will no longer have the option of adjusting their capital plans to offset downward surprises in the Fed’s projections.[2]
QUANTIFYING THE RELATIVE SOURCES OF SPURIOUS VOLATILITY AND UNCERTAINTY
Given the opacity of the Fed models, it is difficult to determine how much variability arises from each of the first three factors, but the following data give us some insight.
Identifying volatility derived from supervisory model changes and model sensitivity. We can control for variability in stress scenarios by comparing the annual projections generated using supervisory models with the projections using the banks’ own models, because both sets of projections use the same stress scenarios.
The scatterplot presents this comparison with respect to year-over-year variability of projected declines in bank CET1 ratios from the 2013–2019 annual stress tests. For a given bank and year, each point represents a comparison between the change relative to the previous year in the outcome (capital ratio decline) projected by the bank’s own models, and the change relative to the previous year in the outcome projected by the supervisory models. The x-axis measures the year-over-year change in the capital decline projected by the supervisory models, and the y-axis the year-over-year change in the capital decline projected by a bank’s models.
Because this analysis compares year-to-year variability in projected outcomes between banks’ own models and supervisory ones, where both are inputting the same supervisory stress scenario each year (the severely adverse scenario), the impact of scenario variability on stress projections is neutralized. Therefore, the scatterplot illustrates two other sources of uncertainty in supervisory models described earlier: the annual changes to supervisory models, and the sensitivity of the supervisory models to changes in stress scenarios.
Two main conclusions can be drawn from the scatterplot.[3] First, the Fed and banks’ own projections often disagree on the year-over-year change in capital requirements. Namely, the correlation between changes in capital requirements using banks’ own projections and supervisory projections is about 25 percent.[4] Also, as shown by the red dots in the upper left/lower right quadrants, the projections generated under supervisory models often indicate a decrease/increase in stress test severity, while those generated under banks’ own models indicate the opposite. Second, changes in capital requirements are, on average, about twice as volatile using supervisory models than the banks’ own models, because the slope of the regression line that fits all the dots in the scatterplot is less than one-half.
In sum, while annual changes to the stress scenarios clearly produce considerable volatility in results from year to year (and thereby yield significant uncertainty in projecting future capital requirements and needs), the other two components also play a large role. Meanwhile, less opacity of supervisory models would enable banks and the public to better anticipate the impact of model changes and excess sensitivity of supervisory projections to changes in stress scenarios.
The role of model sensitivity as a source of spurious volatility. We can also control for the variability in periodic model changes and assess the importance of model sensitivity to changes in stress scenarios by comparing the Fed’s own projections of pre-provision net revenue (PPNR) in 2018 and 2019 with those of banks.
PPNR models—which do not receive as much attention as loan loss or trading and counterparty loss projections—play a key role in determining bank performance under stress. The PPNR chart shows that there were significant changes in the Fed’s PPNR projections relative to those of banks between 2018 and 2019. We will show that those changes were likely driven by excessive sensitivity of the projections generated by Fed models to changes in the supervisory scenarios.
Each dot in the PPNR chart represents the difference between banks’ own projections of PPNR and those of the supervisors, as a percentage of total assets (at the start of the stress tests) for the two most recent stress testing cycles. The observations are ranked, starting from the banks that forecast the lowest revenues relative to the supervisory projections (to the left), to the ones that project the highest revenues relative to the supervisory projections (to the right). The first takeaway is that almost all banks project lower revenues under stress than the Fed did; the dots fall below the horizontal axis. The second is that there was a material change in the supervisory PPNR projections in 2019, shown in an upward shift in dots between 2018 and 2019.
Had the Fed’s projections maintained the same gap with banks’ own projections in 2019, as was the case in 2018, projected PPNR under the Fed models would have been $68 billion higher over the 9-quarter planning horizon (i.e., a 20-percent increase in PPNR). To offer some perspective, dividend payouts over the 9-quarter planning horizon under DFAST were about $87 billion. Thus, the change in PPNR projections in 2019 is material.
This variability was most likely not driven by revisions to the parameters of PPNR models. The Fed stated in its March 2019 release (three months before the June release of the supervisory stress tests results) that “the re-estimation [of the PPNR model] is expected to result in a small decrease in aggregate PPNR forecasts due to weaker PPNR performance in the most recent year, particularly for net interest income” (italics added). Thus, the material decline in the Fed’s PPNR projections relative to those of banks in the 2019 stress testing cycle was likely due to an excess sensitivity of the projections generated by Fed models to changes in the supervisory scenarios.
Evidence suggests that changes in supervisory PPNR projections—whether rising or falling—are not well founded. Til Schuermann and co-authors published a paper in 2014 highlighting the pitfalls of using data from regulatory reports to generate projections of PPNR under stress, as the Fed does. They concluded that reliance on the limited level of granular data available on regulatory reports poses considerable challenges. Namely, relying on highly aggregate data leads to inaccurate bank-specific projections of stressed revenues, because it obscures important differences in banks’ revenue sensitivities to the stress scenarios. As a result, Schuermann and colleagues concluded that a bank-specific approach was preferable.
RECOMMENDATIONS
In summary, we suggest several recommendations to improve supervisory stress tests, reducing spurious volatility and enhancing certainty in capital planning.
Improved disclosure. The unpredictability of supervisory stress test projections that arises from model changes can be resolved by more detailed disclosure. In particular:
- After changes in the supervisory models are adopted, the Fed should calculate and disclose the impact of model changes on the relevant projections using the prior-year stress scenario.
- The Fed should present a decomposition of the year-to-year variability in stress test projections in relation to changes in stress scenarios, supervisory models, and bank portfolios. This decomposition would allow supervisors, investors, and the public to better understand what the results of the stress tests say about the financial condition of the banks. For instance, if a bank’s results are worse in one year compared against the previous year, how much of that result is due to a more stringent scenario or to a supervisory model change versus an increase in the bank’s risk-taking? This approach would require the Fed to generate more than one set of projections each year, but it shouldn’t be excessively burdensome.
- For both legal and policy reasons, as we have argued before, the Federal Reserve should make its models transparent and subject them to notice and comment rulemaking.[5]
Reducing sensitivity of the Fed’s models to changes in supervisory stress scenarios. The data on PPNR projections demonstrate the excessive sensitivity of the projections generated by the supervisory models to changes in the supervisory scenarios. We recommend that the Fed either rely on banks’ own approaches to model certain PPNR subcomponents—similar to what the Fed already does in terms of modeling the losses associated with the global market shock—or, alternatively, use an approach more tailored to each BHC. This would contain a more granular segmentation of revenues across the different business activities and include the largest and most volatile revenue items, along the lines suggested by Schuermann and coauthors.
CONCLUSION
In summary, the opacity of supervisory models, on top of the variability of model projections, leads to uncertainty as to what level of capital is appropriate for banks to hold. Model projections are characterized by variability and unpredictability, because supervisory models are periodically modified and projections are highly sensitive to changes in the stress scenarios. In turn, uncertainty and volatility of capital requirements can cause banks to pull away from lending to households and businesses and even reduce the willingness of banks to provide funding for financial market activities that carry very little risk.
[1] The stress tests use only the standardized risk-weights.
[2] In the SCB proposal, increases in stress losses would be translated into a higher SCB shortly after the release of the stress tests results. However, it is possible that in the revised SCB proposal, increases in stress losses would take effect after a lag, while decreases in stress losses would take effect immediately.
[3] We redid the analysis over time and confirmed that the divergence between banks’ own and supervisory projections persists.
[4] We excluded Santander Holdings USA in 2015 because this single observation has an outsized impact on the correlation between changes in capital requirements using banks’ own projections and supervisory projections. Prior to 2015, Santander Holdings USA held a much larger concentration of subprime auto loans in its portfolio which generated very large capital declines under stress.
[5] See The Clearing House, Letter re: Stress Testing Transparency Proposals, at 13–14 (January 22, 2018) (link). In March 2019, the Federal Reserve released enhanced descriptions of the models it uses to project regulatory capital ratios in the stress tests, as well as ranges of loss rates produced by its models for corporate loans and credit cards. While useful, the information being released is not enough to reduce the unpredictability of stress test outcomes.
