The global financial crisis has been a watershed for the global economy. Its implications for growth, distribution, and public finances have been severe. Structural changes have affected both the real economy and the financial sector. Reforms of the financial regulatory system have not only been a consequence, but also a driver of these trends. The G20 countries undertook an internationally coordinated overhaul of the financial regulatory landscape. It has been the broad objective of these reforms to reduce the probability and impact of future financial crises.

The crisis has particularly challenged the perception that the stability of individual institutions automatically guarantees the stability of the entire financial system. Macroprudential now complements microprudential supervision. The purpose of macroprudential policy is to ensure that the financial system can perform its role for the real economy, even in times of crises or distress.

In the following, I will briefly review the origins of financial crises and the importance of taking a systemic view. I will then review the reform agenda of the past 10 years. As the implementation of key reform elements is well underway, the Financial Stability Board (FSB) has started a structured approach towards ex post evaluation as an integral part of a policy cycle. Ex post policy evaluation assesses whether reforms have achieved their objectives and whether there have been major unintended side effects. I will conclude by discussing elements of an evaluation process which ensure independence and transparency.

1 What causes financial crises?

A stable financial system is consistently in a position to absorb both financial and real economic shocks, especially when confronted with unforeseen events, in stress situations and in periods of structural adjustment. A resilient financial system is able to absorb losses from unexpected developments, it prevents contagion and adverse feedback effects.

Financial stability is at risk if systemic risks emerge. At the heart of systemic risks are externalities: Systemic risk may occur, for instance, when the distress of one or more market participants jeopardises the functioning of the entire system. This may be the case when the distressed market participant is very large or closely interlinked with other market players.

Interconnectedness can be a channel through which unexpected adverse developments are transmitted to the financial system as a whole, thus impairing its stability. Market participants can be connected directly through contractual relationship such as on the interbank market. Besides this, indirect channels of contagion may exist. Many market participants may conduct similar transactions or hold similar types of assets. In such a situation, investors may interpret negative news as signals that other market participants are adversely affected as well. Hence, systemic risks also exist if a large number of small market participants is exposed to similar risks or risks that are closely correlated with each other.

In the presence of systemic risk externalities, the optimal level of risk taken by financial institutions from a private perspective is thus higher than the socially efficient level of risk. Owners of financial institutions may reap the potential benefits of their decisions in full, but bear the potential risks only partially if they can expect to be bailed out by the government bailout in the event of distress.

This excessive risk taking implies costs for society: if risks materialize, governments may be pressured to bail out banks, thus imposing costs on taxpayers. Regulatory reforms aimed at internalizing these externalities comprise higher loss-absorbing capacity and provisions to facilitate the orderly resolution of banks. These reforms have the potential to increase social welfare. However, they may come at a net private cost for the affected institutions: they will forgo the net private benefit (profits) associated with their excessive risk taking, they may incur costs related to compliance with new regulations, and their funding may be more costly as it must provide greater capacity to absorb losses. Through these channels, changes in regulations affect the incentives for banks to take on risks.

2 What happened since the financial crisis?

Capital flow/default cycles have been around since at least 1800 – if not before. Technology has changed, the height of humans has changed, and fashions have changed. Yet the ability of governments and investors to delude themselves, giving rise to periodic bouts of euphoria that usually end in tears, seems to have remained a constant. (Kaminsky and Rogoff 2008, p. 53)[1].

Is this time different? Have policymakers learned from the recent financial crisis? One key insight of the financial crisis has been that supervision needs to go beyond the individual institution and take a system-wide perspective. Macroprudential policy aims to mitigate systemic risks, i.e. risks to the financial system as a whole. It builds on microprudential regulations which are concerned with the stability of individual financial institutions. The need for a macroprudential perspective is a lesson learned from the global financial crisis: even though individual financial institutions may seem stable from a microprudential perspective, the financial system may be highly vulnerable to shocks, and it may amplify these shocks.

Since 2009, the G20 has implemented a wide range of policies which aim at reducing the probability and the effects of future financial crises (Figure 1). These fall into four areas:[2] A first set of policies aims at enhancing the resilience of the financial sector with regard to adverse shocks. These policies include higher capital and liquidity requirements for all banks and in particular for systemically important banks. More resilient financial institutions can better withstand negative shocks and are less likely to amplify them. This mitigates negative repercussions on the real economy.

Figure 1: Post-Crisis Financial Sector Reforms

One key insight of the financial crisis has been that large, complex, and connected financial institutions can be particularly systemic and “too big to fail”. Large and systemically important financial institutions are thus required to fulfil additional capital requirements, and these banks are also in the focus of new resolution policies. Requirements for their total loss-absorbing capacity (TLAC)[3] are intended to ensure that systemically important banks have issued a sufficient amount of bail-inable debt and can be resolved without recourse to taxpayer money, while maintaining their vital economic functions. Recovery and resolution planning identifies critical functions that can be ring-fenced and shielded from the negative consequences of adverse scenarios. Finally, enhanced supervision of systemically important banks includes higher supervisory expectations for risk management functions, risk governance, and internal controls.

Additional policies aim at making derivatives markets safer and more transparent. They include requirements on trade reporting, central clearing, platform trading, margins and higher capital requirements for non-centrally cleared derivatives. Their broader aim is to mitigate systemic risk, contagion, and to avoid market abuse. A final set of policies aims at transforming shadow banking into resilient market-based finance. These policies include new regulations for money market funds and securitizations. Their broad purpose is to mitigate systemic risks associated with non-bank financial intermediation.

3 What are the effects of the reforms?[4]

Many post-crisis financial regulatory reforms have entered uncharted territory: reliable assessments of reform effects and knowledge about transmission channels were lacking. Reforms had to be implemented in an environment characterized by a high degree of uncertainty. Hence, a structured policy cycle is needed to reduce the risks of ineffective policies, miscalibrated policy instruments, and unintended side effects. Such a cycle involves four steps (Figure 2):

1. defining policy objectives,
2. choosing intermediate objectives and appropriate indicators,
3. linking instruments to indicators through ex ante impact assessments, and
4. analysing policy effects through ex post evaluations.

Figure 2: Stylized representation of a macroprudential policy process

In a first step, the policy objective(s) of macroprudential policy need to be specified. Macroprudential authorities use different definitions of the policy objective, but they all aim at reducing systemic risk arising from externalities. Ideally, the policy objective is derived from an analytical framework which links performance of the financial system to the functioning of the real economy.

In a second step, intermediate objectives need to be specified and appropriate indicators chosen. Intermediate objectives are linked to the drivers of systemic risk, such as leverage, risk-taking incentives, interconnectedness, or exposure to common shocks. In many cases, it will not be possible to specify a direct, simple, and linear relationship between intermediate objectives and financial stability. For example, the link between the interconnectedness of financial institutions and the probability of financial contagion is non-linear.[5] Intermediate objectives can thus be state-dependent as their relationship to financial stability depends on the structure and state of the financial system and real economy.

In a third step, the activation or recalibration of policy instruments that address evolving vulnerabilities and financial stability risks needs to be considered. The decision on whether and how to activate policy measures should be based on a structured process of ex ante policy evaluation. This provides information about the relative contribution of policy instruments reducing systemic risk. Trade-offs between stability and other policy objectives can be taken into account when performing such an analysis and when calibrating instruments.

In a fourth step, and once sufficient time has elapsed, the effects of the instruments need to be assessed in an ex post evaluation. This step provides information about the effectiveness of the measure(s) taken, about intended or unintended side effects, and it also serves as an input into a possible recalibration of the policy instruments.

Ultimately, policy evaluation provides insights beyond the assessment of specific policy instruments by answering the question of “how to get there”. Indeed, it can also help address the question of “where to go” by providing information about the appropriateness of the specific policy objectives and targets, which may be revised if needed. This is particularly important given that macroprudential policy is a relatively new area and given that experience with the above policy cycle is limited.

Evaluation of financial sector policies: The approach of the Financial Stability Board

The Financial Stability Board (FSB) has recently developed a framework to guide its own evaluation work related to the G20 post-crisis reforms of financial sector regulation.[6] According to the FSB Framework, evaluation methods need to answer three questions:

• Did the reform “cause” an outcome (attribution)?
• Did the reform have similar effects across relevant markets, states of the world, or jurisdictions and regions (heterogeneity)?
• Did the reform achieve its overall objective (aggregation/general equilibrium)?

Analytical tools vary in their ability to answer these questions. Which evaluation approach to choose depends on their relevance to the objectives of the evaluation, their feasibility and replicability for implementation across countries, and their costs in terms of resource inputs and data requirements. As shown by the first FSB evaluations, a robust evaluation strategy combines different (both qualitative and quantitative) analytical approaches.

In addition to fleshing out analytical concepts, the FSB framework contains a number of arrangements concerning the governance of evaluation work. For instance, the framework recommends that the evaluation process should include means for engagement with external stakeholders. These may include the publication of information on forthcoming evaluation projects, public consultations on evaluation reports before publication, and the publication of evaluation reports. Workshops and roundtables of FSB members with academia and participants from the industry and the civil society are means of interaction.

The first two evaluations under the framework have been delivered to the G20-Summit in November 2018:[7]

• The first project investigates the extent to which post-crisis reforms incentivized the central clearing of OTC derivatives. It has been the aim of these reforms to make derivatives markets both, safer and more transparent. Evaluation results suggest that reforms are providing the intended incentives for central clearing, especially for the most systemic market participants. 
• The second project evaluates the effect of reforms on the financing of infrastructure. It concluded that effects of the reforms on infrastructure finance were of second order importance relative to other factors such as the macro-financial environment, fiscal and monetary policies. The analyses do not point to a significant effect on the volume and cost of infrastructure finance, while there are some indications that reforms may have shortened the average maturities of infrastructure loans granted by global systemically important banks.

More projects are underway. An evaluation of reform effects on SME financing will be delivered to the G20 Summit this year. The respective consultation report was published recently.[8] According to this report, the evaluation does not identify persistent negative effects of the G20 regulatory reforms on SME financing overall. At the same time, the adjustment patterns in the data are not inconsistent with stronger effects of the reforms on those banks for which the new rules on capital regulation have been binding the most.

A new project which has been started in 2019 focuses on a core objective of the reforms. This project will assess the extent to which the too-big-to fail (TBTF) reforms have addressed the systemic and moral hazard risks associated with systemically important financial institutions:[9]

• The project will analyse whether the reforms were effective in reducing the perceived probability of the impact of failure of systemically important institutions – for instance, by assessing changes in estimates of implicit funding subsidies and analysing progress in the resolvability of systemically important banks. 
• It will provide answers to the question of whether reforms are leading to changes in bank behaviour, for instance by attributing changes in business models and risk taking that reflect the internalization of the negative systemic risk externalities to relevant reforms. 
• It will examine the broader effects of the reforms, including the functioning of the financial system, market fragmentation, cost and availability of financing to the real economy. A key issue will be to focus on the achievement of intended outcomes but also analyse potential (positive or negative) unintended consequences.

The FSB, in line with its aim to engage with a broad range of stakeholders, has invited feedback on this project, including supporting evidence of reform effects, by June 21, 2019. To put the evaluation on a solid analytical basis, academic advisors are working in close collaboration with the group on methodological issues.[10] Stakeholders will have further opportunities to interact with the FSB and contribute to this project: as the project progresses, workshops and roundtables will provide an opportunity for externals to engage with the FSB evaluation team. Lastly, the FSB plans to publish a consultation report in June 2020.

4 Independence of policy evaluations [11]

Credible policy evaluation obviously requires independence. Independent and objective assessments are needed to gain an unbiased picture of the effects of reforms. There are several avenues leading to this goal. External experts can be involved who have a mandate for independent evaluations and who have no stake in the formulation of policies. These can be existing institutions such as research institutes or policy advisory bodies. Independence of internal evaluations can be enhanced through institutional arrangements such as peer reviews or independence from policy groups.

Transparency is another precondition for a credible regulatory process. Policy evaluation means being transparent about the goals of regulatory policies and what these policies have actually achieved. Repositories of evaluation studies can improve transparency by making it easier and less costly for (internal) evaluators and external stakeholders to keep track of evaluation work. Other fields, such as development economics or medicine, often rely on repositories to structure the available evidence.[12] The Bank for International Settlements (BIS) recently launched an online repository of studies on the effects of financial regulations.[13]  The repository is an interactive tool, allowing the user to select and visualize findings from multiple studies. Beyond this, it also allows users to submit their own studies.

Replicability is an important element of transparency. Replicability ensures that policy decisions are based on robust evidence. This requires transparency about the models being used, access to data, and documentation of how studies are set up. Replication studies repeat an original analysis using different data and are thus especially useful for policy work. Academic study often focuses on specific novel results, which may limit the external validity of findings in a different setting. Evidence-based policy decisions require information on whether a particular finding is robust with regard to changing the underlying setting. Hence, there is a potential tension between the novelty and uniqueness of findings that are appreciated in academia and the need to rely on sufficiently robust results as a basis for policy decisions. Enhanced appreciation of replication studies in academic work and enhanced capacity of policy institutions to conduct their own replication studies can help relaxing this tension. 

Finally, an active outreach strategy should also be part of an independent, transparent evaluation process. Stakeholders should be given the opportunity to give input and voice their concerns. This interaction can take place via public consultations of reports, through workshops or conferences, or by appointing external academic advisors. All this will ultimately improve the quality of evaluations.

Footnotes:

1. Reinhart, Carmen M. and Kenneth Rogoff (2018), “This time is different: a panoramic view of eight centuries of financial crisis”, NBER Working Paper 13882, March 2018.
2. For more information: https://www.fsb.org/2018/11/implementation-and-effects-of-the-g20-financial-regulatory-reforms-fourth-annual-report/
3. A related policy measure in the European context are Minimum Requirements for own funds and Eligible Liabilities (MREL).
4. This section draws on Buch, Claudia M., Edgar Vogel, Benjamin Weigert (2018), “Evaluating macroprudential policies”, European Systemic Risk Board Working Paper No 76, Frankfurt a. M.
5. Allen, Franklin and Douglas Gale (2000), “Financial Contagion”, Journal of Political Economy, Vol. 108(1).
6. Financial Stability Board (2017), “Framework for Post-Implementation Evaluation of the Effects of the G20 Financial Regulatory Reforms”, Basel, https://www.fsb.org/2017/07/framework-for-post-implementation-evaluation-of-the-effects-of-the-g20-financial-regulatory-reforms/
7. Consultative and final reports are available at: https://www.fsb.org/work-of-the-fsb/implementation-monitoring/effects-of-reforms/
8. https://www.fsb.org/2019/06/fsb-publishes-consultation-on-sme-financing-evaluation/
9. Financial Stability Board (2011), “Policy Measures to Address Systemically Important Financial Institutions”, November 2011. For details see the terms of reference (https://www.fsb.org/2019/05/evaluation-of-too-big-to-fail-reforms-summary-terms-of-reference/). The final report is scheduled for the G20 Presidency in 2020.
10. See the call for nominations for more details (https://www.fsb.org/2019/02/call-for-nominations-appointment-of-academic-advisors-for-the-fsb-evaluation-of-too-big-to-fail-reforms/).
11. See also “Evaluating Financial Sector Reforms: A Joint Task for Academia and Policymakers”, Claudia M. Buch, remarks prepared for the panel discussion “Improving Financial Resilience” at the T20 Summit “Global Solutions”, Berlin, May 30, 2017.
12. Examples for repositories from other policy areas are the McMaster Health Forum (https://www.mcmasterforum.org/) or the 3ie initiative (https://www.3ieimpact.org/) in development economics.
13. See https://stats.bis.org/frame/ and Boissay, F., C. Cantú, S. Claessens, and A. Villegas (2019), “Impact of financial regulations: insights from an online repository of studies”, BIS Quarterly Review, March 2019.