In this research note, we present an overview of important methodological challenges associated with climate change stress testing for banks. While significant and important progress has been made in developing climate scenario analysis, the methodological challenges collectively make the results of climate stress testing extremely subjective and highly variable. Despite these challenges, the underlying analysis and research could gather valuable information to help shape climate policy.
The challenges of climate stress testing are best understood by comparing them with the existing regime for macroeconomic stress testing, which is generally considered a reasonable approach to assessing bank resiliency and formulating capital requirements. Stress testing for climate change is starkly different from existing macro stress testing and given data and methodology challenges likely to be less reliable. First, the lack of historical data creates important challenges in modeling the interactions between climate, the macroeconomy, and the financial sector, which are necessary requirements in designing plausible and coherent scenarios. Second, climate stress testing attempts to measure outcomes over a much longer time horizon—30 to 50 years rather than nine quarters for macroeconomic stress testing. Third, models that generally relate credit losses to climate risk scenarios require large amounts of information about future counterparty behavior over a long time horizon. Fourth, climate stress tests generally assume that banks take no actions to hedge or reduce exposures to climate risks over that horizon. While macroeconomic stress testing has a similar assumption regarding hedging, and therefore may produce some error over a nine-quarter horizon, this assumption, however, becomes deeply counterfactual over a period of decades. Finally, there is also uncertainty associated with market participants’ actions on equilibrium prices and aggregate outcomes.
At this point, given the evolving nature of climate sensitivity analysis, such analysis appears better suited to alerting regulators and banks to such risks, and ensuring that those risks are considered in establishing regulatory policy and bank business strategies.
Overview
Multiple central banks and bank regulators wish to add climate risk scenarios to their stress testing frameworks. Recently, the Network for Greening the Financial System (NGFS) published its Guide to Climate Scenario Analysis for Central Banks and Supervisors, along with a set of reference scenarios. This summer, the French Prudential Supervisor Authority (ACPR) released initial scenarios and main assumptions for its climate exercise, as well as a more technical companion paper that proposes a framework to build scenarios needed to assess climate-related risks. In December 2019, the Bank of England (BoE) published a discussion paper proposing a framework to assess the exposures of U.K. banks to climate-related risks. The BoE plans to release more details about the framework later this year and launch the exercise in mid-2021. In the United States, Chair Jerome H. Powell indicated earlier this year that the Federal Reserve is also trying to develop a strategy to assess the implications of climate change for the U.S. financial system and the economy, though the Federal Reserve has thus far declined to join the NGFS. In September of this year, the Commodity Futures Trading Commission Subcommittee on Climate recommended that banks and regulators undertake climate risk stress testing like those outlined in the NGFS’s guide.
Past Regulatory Stress Testing
Macroeconomic stress testing of bank capital levels is a key supervisory tool that was developed during the 2007–2009 financial crisis. At the time, there was significant uncertainty about the overall level of bank losses. Stress tests emerged as a solution to assess the condition of bank balance sheets and instill confidence in the sector. They also helped recapitalize the U.S. banking sector quickly. During the post-crisis period, stress tests have been the Federal Reserve’s core tool for assessing bank resilience. In March 2020, the Fed converted its annual CCAR stress test into a stress capital buffer rule, which uses the results of the stress tests to make the static capital framework more risk sensitive and forward looking.
The focus on regulatory capital in macroeconomic stress tests requires the modeling of loan losses and other subcomponents of bank profits over a planning horizon.[1] The length of that horizon is set over a period where forecasts can be made with some confidence, with the U.S. choosing a nine-quarter horizon. Obviously, a longer horizon comes with increasing uncertainty about the accuracy of stress projections. In addition, macroeconomic stress tests calculate the effect of macroeconomic scenarios on the creditworthiness of bank borrowers and bank revenues. They do so based on statistical models estimated using datasets that span past economic downturns, and which continue to be back-tested as more data emerge. Therefore, there is a limited role for the use of judgment in the derivation of profit and loss projections under macroeconomic stress conditions.[2]
First, little is known about the interactions between climate and macro-financial variables, which are important inputs in the development of scenarios that are coherent and plausible. Second, stress testing for climate change thus far has chosen a much longer time horizon, which makes the static bank balance sheet assumption implausible. Third, the estimation of credit risk models needed to generate loss projections relies on a limited set of datapoints and has no near-term potential for back-testing. Furthermore, because the loss projections rely so heavily on the judgment of experts, validating the projections is nearly impossible.
Contrasts with Climate Stress Testing
Stress testing for climate change departs from the standard process of macroeconomic stress tests in several important ways. The post discusses the following five main challenges in greater detail:
- The mapping between climate change scenarios and macroeconomic and financial outcomes;
- An extremely long (30- to 50-year) stress time horizon for climate risks, which contrasts with current horizons used for modeling credit risk, market risk, and operational risk;
- A lack of historical data on the relationship between climate risk and credit losses, in contrast to voluminous historical data on the effect of changes in unemployment, GDP, and other macroeconomic factors on bank performance;
- A large degree of uncertainty driven by the simplifying assumption that banks would not change their behavior over the time horizon if the presumed stress in fact plays out—that is, that banks would not reduce or hedge their exposure to those companies over time; and
- Uncertainty around “second-round” effects to account for the impact of the actions of market participants on equilibrium prices and on their behavior. Unfortunately, the research in this area is still highly theoretical and not ready for practical applications.
While these challenges have been acknowledged by central banks, the overwhelming scope of the task in rectifying these deficiencies has received limited analysis. Collectively, these shortcomings with stress testing for climate risk seem likely to make such tests unreliable as a measure of banks’ financial risk from climate change. Significant further research and testing should precede any effort to make decisions about capital adequacy or other issues based on the results of climate-related stress tests. This is not to say however that scenario analysis is not useful for informing policymakers and banks about possible strategic decisions that must be made over longer time horizons.
Table 1: Summary of Challenges in Stress testing of Bank Balance Sheets for Climate Change
Challenges to the Planning Horizon of Stress Scenarios
The planning horizon in the U.S. supervisory stress tests is nine quarters. The EBA and U.K. stress tests have somewhat longer planning horizons: three and five years, respectively. The Federal Reserve requires banks to maintain capital ratios above regulatory minimums over nine quarters. However, at the end of the planning horizon, a bank needs to show it has enough allowance for credit losses to cover expected losses for one more year. This means that the effective planning horizon of the U.S. stress tests is a bit more than three years, and more in line with the EBA stress tests.
The planning horizon of the U.S. supervisory stress tests follows the Supervisory Capital Assessment Program, which was conducted in the first half of 2009. At that time, the Federal Reserve justified the choice of nine quarters to strike the balance between being able to capture most of the losses on loans originated while lending standards were loose with the need to have a reasonable degree of confidence in the projections of losses and revenues over the planning horizon. Projection uncertainty rises as the horizon lengthens because models are imprecise, the assumptions about the macroeconomy depart further and further from realizations, and the bank itself will take actions over the intervening period in response to developments not captured by its current condition. The Federal Reserve thus concluded that making projections for loan losses and revenues beyond two years was too uncertain to be a useful indicator of bank soundness.
Climate change and policy actions designed to reach the Paris Accord target will occur slowly over several decades. Shorter-term scenarios therefore do not allow those risks to materialize. As a result, the ACPR and the BoE are proposing to include scenarios with a planning horizon of 30 years. The scenarios start in 2020 and extend through 2050. In addition, one of the scenarios proposed by the BoE frontloads material risks estimated to occur between 2050 and 2080 (30 to 50 years from now) to the period before 2050, by adjusting the path of the variables included in the scenario. Banks will be required to use the scenarios to produce projections for expected losses every five years.
The NGFS goes even further and recommends that supervisors adopt a 50-year planning horizon in its scenarios, offering flexibility in the frequency of the analysis. Namely, climate-related risks would be assessed every year, every five years, every 10 years, etc. By contrast, the climate change scenarios of the Dutch National Bank (DNB) have a planning horizon of five years to reduce the uncertainty in the assessment of climate-related risks and more precisely evaluate the impact on regulatory capital. Risks are assessed every year in the DNB’s climate change stress tests.
The choice of an extremely long planning horizon in climate change stress tests creates significant challenges in designing and executing the exercise. Obviously, a longer horizon allows for climate changes to become more material; however, it also leads to more uncertainty about how such changes will affect banks. For longer time-horizon projections to be useful, a greater body of historical data or greater model validation would be needed, and the latter has yet to occur—as we discuss next.
Challenges in the Design of Stress Scenarios
The design of scenarios in macroeconomic stress tests typically replicates the aggregate macroeconomic environment during a prior stress episode, like the 2007–2009 financial crisis and resulting recession. The variables included in the stress scenario are known to be important drivers of bank net income, including pre-provision net revenue and credit losses on loans and securities. Importantly, there are voluminous amounts of historical data to estimate the correlations between the variables included in the stress scenarios and bank performance under stress. There is also a tight correlation between the macroeconomic and financial variables that can be inferred from prior downturns and crisis periods.
Analysts generally agree that climate change stress tests should have multiple scenarios to capture the uncertainty around the transition to a low-carbon economy. Specifically, multiple scenarios are needed to capture the risks associated with climate change and potential mitigation from government policy and technological developments. For instance, the timing of the implementation of policies that favor renewables and the availability of carbon dioxide removal technologies is unknown, and the multiplicity of scenarios offers results using different sets of assumptions. Also, some scenarios consider a decline in investor and consumer confidence due to a lack of government action in implementing new policies.
Each climate change scenario typically includes both physical and transition variables, which are then translated into key macroeconomic and financial variables using macroeconometric models. The physical variables include temperature pathways, frequency and severity of extreme weather events, population longevity, and agricultural productivity. The transitional variables include the paths for carbon prices, emissions, commodity and energy prices, and energy mix. The macroeconomic and financial variables are the standard series included in the stress tests (unemployment, real GDP, house price index, bond yields, etc.). The mapping between climate change variables and macro-financial variables is integral to the design of climate change scenarios. However, because the economic consequences of climate change are new, there are currently no data that calibrate the dynamic interactions between the macroeconomy, the financial system, climate change, and environmental policies.
In summary, considerably more work needs to be done to marry science-based climate models to financial ones. It is critical that banks, central banks and supervisors partner with the climate modelers to ensure that expertise is shared across the disciplines.
The models used to translate the stress scenarios to bank-specific losses and income are key elements of macroeconomic stress tests. The main building blocks include various models used to project pre-tax net income, namely those used to forecast pre-provision net revenue, provisions for credit losses, losses on market risk exposures, losses from the default of a large counterparty, credit losses on securities, and losses due to operational-risk events. To assess climate-related risks, one approach has been to modify the models used to project credit losses in stress tests, so we first describe how those models are applied in macroeconomic stress tests.
To project loan losses for loans accruing on the books of banks, the Fed estimates the probability of default (PD), loss given default (LGD), and exposure at default (EAD) for each quarter of the stress horizon. The PD estimates the probability a loan defaults in each quarter, the LGD represents the share of the exposure the lender will not be able to recover after the borrower default, and the EAD estimates the size of the exposure at the time of the default. The loan’s expected loss in each quarter is equal to the product of these three components.
Lack of Historical Data and Model Challenges
In macroeconomic stress tests, for each component used to estimate expected loss—e.g., exposures to corporate obligors—there is a quantitative model that uses borrower and loan characteristics, as well as the economic conditions defined in the scenario to project that component. The Fed has therefore developed PD, LGD, and EAD models for each loan portfolio that uniquely map loan characteristics and stress scenarios to projections of quarterly expected losses on these loans over the planning horizon of the stress test scenario. In addition, there is no role for judgment being applied in the mapping between the inputs and loss projections, and these models have been validated by Fed staff not directly involved in the stress tests.[3]
For climate change stress testing, banks are required to model the impact of the scenarios on expected loss for corporate, household, and government exposures. Importantly, if banks are to effectively assess borrower-level risks, the climate change scenarios need to include a granular sectoral dimension. This is because the results of the exercise significantly depend on the emissions of a particular sector or the ability of that sector to develop alternative carbon-free technologies. In addition, a key challenge in climate change stress tests is the lack of historical data on climate risks and credit losses. This inevitably means that banks need to make speculative judgments on the impact of climate risks on the expected loss of each borrower or group of borrowers in a given sector, which will yield varying results.
Under the U.K. and ACPR stress tests, banks must estimate changes in the probability of default of their borrowers by performing a counterparty-level analysis due to the lack of historical data. The estimation requires projecting the balance sheet and income statement of each counterparty over the planning horizon. Those projections are then used to calculate the key variables used as input to determine the credit rating of a borrower (e.g., corporate leverage, CAPEX, etc.). Banks presumably then could rely on the business-as-usual mapping between credit ratings and probabilities of default to estimate the impact of climate risk on the probability of default of each borrower.
Counterparty-level projections require a large amount of information about the future actions of the borrower in light of climate risk. Banks must rely on information available on corporate statements as part of the Task Force on Climate related Financial Disclosures, but such information is only typically available for large counterparties.[4] Even then, this information is imprecise, and the analysis depends heavily on judgment since there are no historical data to use in adjusting the risk parameters of smaller obligors.
The U.K. stress tests also require banks to estimate the impact of physical and transition risks on property values, loan-to-values, and probabilities of default on mortgage and unsecured loans to households over a 30-year period. First, they are to presume that the occurrence of extreme weather events leads to a decline in the value of properties exposed to them. (Consistent with its assumption that banks will not hedge any climate risks, the test also assumes no increase in the value of properties not exposed to such events—so effectively an ahistorical 30-year secular decline in property values.) The decline in property values is then presumed to cause loan-to-values (by definition) and probabilities of default to rise. The increase in the probabilities of default should increase banks’ expected losses on mortgage loans and possibly unsecured credit, although the latter is less clear.[5] The impact of extreme weather events depends significantly on the location of the property, the property’s value, the amount of the loan, and the time to maturity. The ACPR stress tests acknowledge that the quantification of higher carbon prices on the behavior of households is too challenging to model. Instead, the tests follow the macro-stress test approach by using the path of the macroeconomic variables included in the scenario to calculate expected loss.
Several other challenges are portfolio specific and, for the most part, are associated with the estimation of physical risks. For corporate exposures, banks are required to assess the ability of businesses to move their operations to areas less subject to extreme weather events. This is a time-consuming and difficult exercise when corporations have physical assets in many geographical locations. A similar issue arises with the estimation of physical losses for household exposures. Banks are required to develop ways to access large databases containing granular information about the location of homes of borrowers to assess the probability of a home being in an area more exposed to extreme weather events. This challenge also applies to the measurement of physical risks for exposures to governments and municipalities. Another important issue arises with the ability of banks to incorporate the effect of insurance markets in the quantification of losses. Assuming that insurance policies for mortgages and other physical assets will disappear is extreme, because insurance premiums can be repriced annually, and the impact of climate change on the frequency of extreme weather events is gradual.
Another challenge is the double-counting of climate change impacts on asset prices and credit losses. Several market prices (home prices, bond prices, etc.) and banks’ own risk assessments already incorporate climate risk into a business-as-usual scenario. Two important examples that are well documented in the academic literature include the price of homes vulnerable to the rise of sea levels, and the price of municipal bonds. Bernstein, Gustafson, and Lewis (2019) find that homes exposed to projected sea-level rises face a 7-percent discount relative to similar homes unlikely to experience a flood over the next 50 years.[6] Also, Painter (2020) finds that counties more likely to be affected by climate change already pay higher underwriting fees and yields to issue long-term municipal bonds.[7] However, the approaches used to assess physical and transitions risks assume that market prices and the creditworthiness of the borrower do not already consider the impact of climate risk.
Challenges in Bank Behavior During the Planning Horizon
The last important assumption relevant to climate change stress tests is the evolution of bank balance sheets over the planning horizon. Since 2014, the Federal Reserve has constructed its own projections of risk-weighted assets and total assets, under the assumption that credit supply does not contract under a nine-quarter stress test. This assumption was driven by a desire to avoid procyclicality, because it prevents banks from assuming they will reduce the supply of credit to firms and households in a recession to slow down the decline in regulatory capital ratios under stress. In practice, this assumption overstates capital needs, because loan demand contracts sharply during economic downturns. So, by assuming that balance sheets remain constant under stress, the Fed is seeking to guarantee that banks will be able to continue to lend in a downturn and help support economic activity.[8]
Most climate change stress tests also assume a constant balance sheet, with the exception of the exercise conducted by ACPR, which allows for a dynamic balance sheet between 2025 and 2050. Assuming that banks are unable to change their portfolios over a 30-year period is highly unrealistic. It would be akin to assuming that, at the onset of the Industrial Revolution, banks would have continued to lend only to companies that relied solely on hand production over the next several decades and would not have allocated their portfolios to companies using the new manufacturing processes. To make the exercise more realistic, some supervisors allow banks to submit a follow-up round of submissions considering changes in bank strategies in response to the scenario, both in terms of mitigating climate change risks as well as pursuing new opportunities. These are general equilibrium effects that reflect the actions that all the different types of agents in the economy have on equilibrium prices and rates of return. Climate change is not just about risks, but it also creates an opportunity for banks to lend to borrowers more likely to benefit from a transition to a low-carbon environment, or those borrowers nimble in managing the transition.
Consequently, allowing for the modeling of second-round effects in stress testing for climate change is extremely important, because it is counterfactual to assume banks would keep a static portfolio over a long horizon. However, this is a daunting task that sits on top of all the challenges discussed in this note, including the lack of data, challenges in scenario design, and the highly uncertain model projections. It is already quite challenging to model second-round effects in macroeconomic stress tests where banks and supervisors have much more data, more experience in scenario design, and models with demonstrated reliability under stress conditions.[9]
Concluding Thoughts
Climate related risks are significant, especially if governments do not enact policies to curb carbon emissions. In this post, we have highlighted some key challenges in stress testing for climate change: the planning horizon, scenario design, lack of data, and heavy reliance on professional judgment. Therefore, it seems premature to add new climate change scenarios to the macroeconomic stress tests, and even more so to link climate stress test to capital requirements. Doing so risks severely undermining the rigor and therefore the credibility of those requirements, and—given the multiplicity of non-data-based judgments required—politicizing them. Nonetheless, it is important that research continues in this area and that banks continue to make advancements in their internal scenario analysis. Such efforts will help advance the testing methodologies through the publication of relevant work as well as through more focused case studies on the impact of physical and transition risks on bank portfolios.
Lastly, although there are serious and potentially unsurmountable challenges to using climate stress tests to set capital requirements, the underlying analysis and research will gather valuable information for public policy on climate change. And while the costs of policies to slow climate change are immediate and relatively easy to see, the costs of not acting are harder to measure. The analysis currently envisioned for climate stress tests could help shine a brighter light on the costs of delay.
[1] Banks with significant trading operations are also required to recognize trading and counterparty mark-to-market losses based on shocks to tens of thousands of risk factors included in the global market shock.
[2] That said, there are also challenges around using the results of macro stress tests to set banks’ capital requirements. For instance, in the current crisis, it is very difficult to evaluate the effect of government stimulus on bank losses. The Federal Reserve is being careful in trying to avoid adjusting bank capital requirements during the crisis in part for this reason.
[3] Also, while there is no judgment used in calculating the results of each component, there is considerable judgment in the choice of the type of models to estimate and which variables to include as risk drivers.
[4] The objective of the TFCD is to help companies develop consistent climate-related financial risk disclosures for their stakeholders.
[5] See Gallagher, Justin, and Daniel Hartley, “Household finance after a natural disaster: The case of Hurricane Katrina,” American Economic Journal: Economic Policy, Vol. 9, No. 3, August 2017 (pp. 199–228).
[6] See Bernstein, Asaf, Matthew Gustafson, and Ryan Lewis, “Disaster on the horizon: The price effect of sea level rise,” Journal of Financial Economics, Vol. 134, Issue 2, November 2019 (pp. 253–272).
[7] See Painter, Marcus, “An inconvenient cost: The effects of climate change on municipal bonds,” Journal of Financial Economics, Vol. 135, Issue 2, February 2020 (pp. 468–482).
[8] However, this assumption raises the de facto capital requirement on the loans assumed counterfactually to remain constant. Because those higher requirements will also hold in a recession when capital becomes scarcer and more expensive, the assumption of a static balance sheet is pro-, not counter-cyclical, because it reduces credit supply in downturns.
[9] Moreover, the Fed, at least, has been reluctant to allow banks to offset projected losses with speculative plans, because if it did so the stress tests would become even more judgmental and contentious. In the case of climate stress tests, such offsets become even more judgmental while at the same time critical if the tests are meant to reflect reality.
