Each major central bank has for some years published a financial stability report to inform policy and shape bank examination. Those reports seek to forecast emerging financial stability risks. In this note, we use an artificial intelligence engine to conduct a historical review of reports from the Federal Reserve, European Central Bank and Bank of England to determine their forecasting ability; we then compare those forecasts to current market expectations as reported in the financial press concurrently or before those forecasts were made — effectively, forecasting the central bank’s forecasting alpha.
Our results show that the financial stability reports produce zero or negative alpha — that is, they consistently fail to identify subsequent financial stability problems that are not already recognized by the market.
We note (with some surprise) that this effort is the first of which we are aware that attempts to assess the value of these reports. While central banks describe their reports as monitoring and assessing vulnerabilities, they are in fact forecasting risks, and using those forecasts in their supervision. When a bank assesses its risk, it examines risk measures such as value at risk or changes in credit ratings, because these assessments are relevant in predicting the likelihood of adverse events. Central banks similarly look at risk measures such as the level of asset prices relative to economic fundamentals or leverage within the financial sector to gauge financial stability risk. Supervisors expect banks to review and validate the predictive utility of their risk reports, but central banks have not — publicly at least — appraised their own financial stability reports.
We publish our methodology, which involves a series of prompts that involve the use of judgment and are open to question. We hope that others in both academia and the public sector will replicate and vary this methodology to determine whether the same conclusion is reached.
If further analysis validates the findings of this maiden effort, then central banks should consider discontinuing production of these financial stability reports and instead use AI/ML methods to reflect market sentiment, and they should discontinue entirely the use of financial stability projections to inform the focus of bank examination. If regulators wish to gauge market sentiment on financial stability risk, AI should be an efficient tool. However, until it demonstrates some predictive ability, we believe the central focus of bank examination should be ensuring the adequacy of capital and liquidity and the firm’s ability to manage interest rate and major operational risks, rather than achieving some “macroprudential goal.”
Building a Simple Risk Management AI Methodology
In order for an LLM to analyze a financial stability report’s effectiveness, it is necessary to compare and contrast dozens of reports and read millions of words. However, the amount of text an LLM can analyze is limited by the size of its context window, and even the most advanced models are currently limited to approximately 750,000 words.[1] For an LLM to analyze all the reports at once, those reports must simultaneously fit in the LLM’s context window. As Table 1 makes clear, none of the central banks’ financial stability reports can fit into current LLMs’ context windows.
Table 1
One simpler strategy is to have the LLM read each report one by one, taking notes on the most important points. Then, a summary of all the reports is created. That summary, which is an order of magnitude smaller than the original reports as shown in Table 1, can be loaded into the LLM’s context window and analyzed. Thus, we develop a two-step process for each central bank’s reports. In the first step, we download each report, either in PDF or HTML form. We then upload each report to the ChatGPT 5.4 API and have it create a thorough summary of the report, guided by a custom prompt that focuses the model on the issues we seek to analyze. We repeat that process for each report until we have a note file for each report. For the ECB, for example, we would have 44 note files corresponding to the 44 financial stability reports.
In the second step, we combine the note files into a single file that contains notes for all the reports. We then check if the note file is too large for the LLM’s context window. If it is, we summarize it further. Finally, we upload the cumulative note file to ChatGPT 5.4[2] and have the model analyze all the reports, guided by a custom prompt. Figure 1 illustrates the process.
Figure 1
Prompts
Note-taking prompt
We must provide a prompt to the LLM to guide it at the note-taking stage. Once the reports are properly summarized, we must create a report effectiveness prompt to synthesize the summary notes into an overall view of the reports across time. The prompts contain the criteria we used to judge the effectiveness of the series of financial stability reports.
In constructing the note-taking prompt, we asked the LLM to do the following:
- first, record any discussion of risks rising or falling so we understand what risks were emphasized and de-emphasized in each report;
- second, for each risk view, identify the justification for its conclusion and the strength of the evidence offered;
- third, identify any relative ranking of the risks, in the view of the central bank staff;
- fourth, identify any predictions that risks would rise or fall, or whether certain market events would occur, to measure each report’s ability to “see around the corner”; and
- fifth, identify any acknowledgment that risks that the report raised were already being managed by financial institutions.
We also added the requirement that the LLM support its points with actual quotes from the report along with the page numbers on which the quotes could be found. Although LLMs hallucinate less than they used to, verification of an LLM’s output is still an extremely important step in using them. The requirement for quotes and page numbers forces the model to focus on objective evidence that can be found in the reports rather than what it memorized in its training. Quotes and page numbers also allow independent verification of the model’s analysis. The note-taking prompt can be viewed in Appendix A.
Report effectiveness prompt
Once the notes are taken for each report, the notes are combined into a file that contains relevant data on all the reports from their inception to the present. As Table 1 shows, the note-taking step reduced the size of the information considerably. For example, the over six million words in the BOE reports were reduced to a 260,000-word set of notes that was uploaded to ChatGPT 5.4 for further analysis. We then formulated a report effectiveness prompt that asks for all cases in which 1) the central bank made a prediction about a risk that was realized; 2) the central bank made a prediction about a risk that was not realized; and 3) the central bank failed to anticipate a risk that turned out to be important. The prompt was motivated by the observation that risk managers will assess the effectiveness of risk reports that predict the likelihood of events by examining whether in fact the events actually occurred or not, and, very importantly, whether adverse events materialized that were not highlighted sufficiently. The prompt we used to analyze the reports can be examined in Appendix A.
LLM Analysis of All Financial Stability Reports
The model gave an extensive report for each central bank. Appendix B reports tables that summarize the results of the final LLM analysis that combined the notes for each central bank. In the tables, we note that in every case a central bank’s forecast of a risk was correct, that risk was already a widespread expectation discussed by the financial press, analysts or economists. In other words, the financial stability reports forecast successfully only very high probability events. For example, in late 2008, the ECB highlighted bank funding costs and their potential to worsen the financial stress on banks. But the bank funding cost issue was a well-known high probability risk since borrowing costs for all borrowers had skyrocketed, as, for example, covered by the Wall Street Journal in September 2008.[3] The ECB and the BOE also predicted in late 2011 that sovereign stress could exacerbate bank stress, but that risk was reported by CNBC in early 2011.[4] The Fed frequently highlighted the risks from commercial real estate after Covid in its financial stability reports, but that risk was already widely known and reported — for example, by the Financial Times in August 2020.[5] Risk management alpha would be shown if the reports highlighted a risk as being significant or higher probability when it was not already widely expected to be by the market. The financial stability reports did not show any evidence of alpha.
The second category in the tables are risks that were forecast to be significant but did not turn out to be significant — at least not yet. The central banks tend to follow a consistent pattern in this category. Risks highlighted tend to be macroeconomic imbalances, assets that appear to be overvalued relative to some posited level of economic fundamentals, and new and emerging risks, such as Y2K, climate risk and AI. These risks are generally also widely understood by financial institutions and markets.
The third category in the tables — risks that were not highlighted to be significant but turned out to be — provide evidence of zero or negative alpha in the risk reports. For example, neither the ECB nor BOE reports appreciated the risks of structured credit and subprime contagion prior to the Global Financial Crisis. The Fed financial stability reports missed the acceleration of inflation after COVID as well as the vulnerabilities leading to collapse of Silicon Valley Bank.
Conclusions
The results indicate that the financial stability of the major Western central banks reports produce zero or negative alpha — that is, they consistently fail to identify subsequent financial stability problems that are not already recognized by the market, and they miss important risks that are also missed by the market.
It should not be surprising that the central bank financial stability reports historically miss the most important market risks faced by financial institutions. The efficient markets hypothesis, one of the most well-tested theories in financial economics, shows that to a close approximation all information available to the market is very rapidly incorporated into market prices, making predictions about future market events very difficult.
Assuming this assessment of their track record is confirmed, financial stability reports should not be an input into bank supervision, and they should be discontinued and replaced by AI and machine learning techniques that assess market expectations and sentiment to inform policymakers.
Indeed, the methodology we have developed could more generally be used by regulatory supervisors and bank risk management departments to analyze and improve their processes. For example, an agency could analyze how effective regulation of banks has been over time by having an LLM read and analyze the body of examination reports. The LLM could help surface biases and blind spots and uncover examiner mandates that produced a misallocation of bank risk management resources. We would strongly urge that any such effort be done with as much transparency as possible, redacting individual firm information, but allowing outsiders to validate the methodology.
The use of AI in the financial services industry has opened up exciting new risk management opportunities for regulators and banks. The ability to audit a process that has a substantial written record is just one example. However, the use of this new technology must be carefully managed to avoid misleading results. The result of any LLM analysis is highly dependent on the prompts given to the model. Small changes in prompts could substantially affect the conclusions. The prompting process itself must be carefully controlled, if not risk managed. Moreover, the results of any LLM analysis must be capable of being independently validated and must in fact be validated. New prompt risk management and validation techniques will need to be developed for that purpose.
Appendix A:
Note-taking prompt:
Take concise but detailed notes on this Financial Stability Review.
Please include:
- Summary of any discussion of risks rising or falling with short quotes illustrating the discussion.
- Any reasons or analysis supporting the inclusion of the risk as rising or falling along with short quotes illustrating the point.
- Any discussion ranking the risks in terms of relative importance and reasons for doing so. Include short quotes where the discussion occurred.
- Any predictions of risks occurring over some future horizon along with short quotes illustrating those predictions.
- Any analysis of whether the risks were already controlled by financial institutions and short quotes supporting the points.
- Anything that could be useful later when synthesizing across documents.
- Include page numbers where each quote occurred.
- Name the set of notes after the file name of the pdf appended with “notes.”
Return structured notes in markdown format.
Report effectiveness prompt
Read the notes that I have uploaded and make a list of all cases in which 1) A risk was predicted or forecast to be significant for the banking system and the prediction later turned out to be correct; 2) a risk was predicted or forecast to be significant for the banking system and subsequently did not turn out to be correct; and 3) a risk that was not predicted or forecast to be significant but subsequently turned out to be significant for the banking system.
Appendix B
LLM Analysis of European Central Bank Financial Stability Reports 2004-2026
LLM Analysis of Federal Reserve Financial Stability Reports 2018-2026
LLM Analysis of Bank of England Financial Stability Reports 1996-2026
[1] GPT 5.4’s context window is 1.05 million tokens. A token is a word or part of a word. Using the rule of thumb that a word is ¾ of a token, the context window is approximately 750,000 words.
[2] We chose to use GPT 5.4 since it was the cheapest model that could accomplish the task. We do not need the most recent frontier model, such as GPT 5.5 or Claude Opus 4.8, for a summarization task. GPT 5.4 and Claude Sonnet 4.6 have similar capabilities. GPT 5.4 costs $2.50 per million input tokens and $15 per million output tokens; Claude Sonnet 4.6 costs $3 per million input tokens and $15 per million output tokens. The analysis is dominated by input tokens and GPT 5.4 is cheaper.
[3] “Mounting Fears Shake World Markets As Banking Giants Rush to Raise Capital,” Wall Street Journal, September 18, 2008 available at https://s.wsj.net/public/resources/documents/Crisis.091808.A1.pdf
[4] “European Sovereign Debt: The Crisis that Isn’t Going Away,” CNBC, January 8, 2011, available at https://www.cnbc.com/2011/01/08/european-sovereign-debt-crisis-that-isnt-going-away.html
[5] “Covid-crippled property markets ring the alarm for banks,” Financial Times, August 2020 available at https://www.ft.com/content/aaf192ac-dc94-4509-8f24-5831a32e7aa2
