We have a panel on basically on the policy dimension of macroprudential stress testing. We have Diane Perret from the University of Luxembourg, Beverly Hilton here to Vice President for Research Fed New York, Jesus Saorina who is the Director General Financial

Stability, Irregulation and Resolution, Banco de España, Martin Cihak who is the Head of Stress Testing in the IMF, and I'm Carmelo Sallejo, Head of the Stress Testing Division in the ECB in Financial Stability.

We tried with this panel, we'll try to cover lots of ground, so these are going to be high level short interventions, we'll try to keep some time for questions and answers at the end. I think we'll have, Diane will start with her views on what exactly is the measure

you want to look at when you have vulnerability when you're doing a macroprudential stress test, and then Beverly will give us some views on in what direction we should be developing stress testing to make this an effective policy tool.

I will give some perspective on how actually this can be used to calibrate macroprudential policies. Ben Jesus will share some views on how to coordinate the macroprudential and the microprudential stress test, and then finally Martin will tell us about communication

challenges of exercises which are complicated and especially when you move into the policy domain, the translation is not always immediate, and you've heard today also how important communication is and transparency in stress testing, so this is really an overarching

argument topic. And now without further ado, I'll give Diane the pointer, and I guess it's on forever. Is this one working? Yes. Well first of all, let me thank you for inviting me to sit on the panel of such a distinguished

panelist. I feel very impressed here. So here, what I want to talk about today, so you asked me my view on what's the right measure of vulnerability of a bank, and I think if there was one, we would be out

of business, right? So probably there's no such a thing as a right measure. I think probably you want to use multiple measures, as much information as you can have.

I'm a non-metrician by training, so the information set should increase to inform your view. But why do we need so many measures, and why do we need a stress test, why do we need supervision? In the end, I think what I wanted to talk today is about this trilogy that we have

between, when we talk about supervision, we have this interaction with capital requirements and regulatory arbitrage, so if there was no rules, there would not be room for checking the rules and having supervisors that make sure that these rules, these requirements

are implemented, and if there was no regulatory arbitrage, probably rules would be enough, and we would be also out of business. So I want to talk about this equilibrium that we have between capital requirements, supervision, and regulatory arbitrage, and see a little bit how, when you move one,

what happens to the other two. And my way of thinking of stress test is, stress test to me is the emblematic example where capital requirements and supervision are intertwined.

And what we've seen in most countries, in the US, in Europe, is that with the implementation of stress test, you have both higher capital requirements and enhanced supervision, at the same time, for the same group of banks, the largest banks.

So why do we think that supervision is important is that we actually saw that this was, this still plays a role in the US, that in identifying this direct effect of stress test supervision was actually what was working and reducing risk-taking incentives

at banks, and that is because without doing that, we would not actually have seen any effect on risk-taking. So not disentangling between supervision, capital requirement, and the effect that

stress test have on capital requirement would not have been able, we didn't see any effect whatsoever of stress test on bank risk-taking. So why is that is because, well, there is an effect of capital requirement. So banks are actually responding to capital requirement, there is no separation between

capital structure and investment decision, as I also explained in my discussion. And, well, this irrelevance of Modigliani-Miller for banks doesn't only mean that with conventional wisdom we think you increase capital, banks should be safer, right?

Here banks also respond to that, and they could actually take more risk, such that by taking on more risk, you actually end up with banks that are riskier. And why would they take more risk is because, well, we have in mind this increase in cost

of funding that comes when equity is costlier than that, which could very well be the case for banks, knowing that there is deposit insurance and banks don't fully pay for the deposit insurance. So with this increase in cost of funding, banks could optimally respond by taking on more risk, could be the opposite channels, more skin in the game that would prevail.

But what actually we found in the data is that banks tend to respond to increases in capital requirement by taking on more risk in the short run. And this is something that actually regulation is based on.

Basel capital requirements is based on this principle that you should ask for more capital for the riskier exposures in order to prevent these banks to actually each time take more risk when they have higher capital requirements. So the idea since even Basel I was to link capital requirement to asset riskiness through regulatory risk weights.

Now if this is included in capital requirements, why do we again, why do we need supervision if capital requirements already reflect asset riskiness? And I mean, then I go back to the evidence on regulatory arbitrage. So there is, so even if capital requirements reflect risk, they don't fully reflect risk

or they don't fully reflect risk in a dynamic manner. And banks are going to always shift their portfolio towards the most underestimated risk weight class, asset class. So for example, mortgage-backed securities during financial crisis, sovereign bonds during the European sovereign debt crisis.

And this is where I think market measures of risk or additional measures that the regulator is not necessarily using to assess the banks on a day-to-day basis can be useful to flag those episodes where regulatory arbitrage is very important.

So why are market measures of risk interesting and useful here is because by definition they are not subject to regulatory arbitrage. You might say they might be wrong, they might have over many other caveats, but at least they are not subject to regulatory arbitrage, so they will reflect a ranking

of bank riskiness that might not be the same ranking as bank riskiness according to regulatory risk weights. And I think when this can be informative is when you have actually a negative correlation. When the market participants think that the riskiest banks are actually the banks that

have the lowest risk weights and have the lowest capital requirements, then that can tell you something about whether risk weights correctly reflect risk and whether there is regulatory arbitrage. It was the case in 2011 for the 2011 stress test in Europe where we saw a very negative

correlation with market risk weights, market implied risk weights, coming from the fact that many banks actually were loaded on risky sovereign bonds. But you see that in all the stress tests this has continued to be a negative correlation.

Sometimes it's because of low market to book ratios, sometimes it's because of other reasons, but this is a red flag, right? Once you have this flag then you're like, why is this so? Let's dig a little bit deeper in the data and try to figure out what's going on. Also in more recent research we saw that the more stringent capital requirement the

bank face depends on risk weights, the more banks are going to respond to increases in capital requirements by taking on more risk. There is this evidence all over the place there. Not only that banks are doing regulatory arbitrage, but they are also doing regulatory arbitrage

in stress tests. So this is where supervision has the most important roles. When capital requirements are high this supervision can play an important role.

Before the crisis we know that there was a lot of regulatory arbitrage because there was a lot of opportunities to do so. You had only four different buckets, risk buckets, loosely defined risk weights and the Basel one, so banks could very well go for the riskiest assets within a risk weight class.

Now there are less opportunities to do regulatory arbitrage, you have more precise risk weights, you have also more stringent capital requirements, but still we find a very important role for qualitative supervision, so I'm going to just leave it with open questions and

say, well, maybe the opportunities have decreased, but maybe the incentives have changed too to engage in regulatory arbitrage. And that's maybe where we want to think about what's the role of supervision when banks capital requirements are high versus low, whether supervision and capital requirements

or high capital requirements are substitutes. It seems that there was a little bit of this discussion at least in the U.S. with the Financial Choice Act to see high capital requirements as a substitute to stress test supervision. I don't think that's going to go through, but I'm saying this has been on the table at some point and these are questions to think about.

Thank you. Thanks a lot. Sorry, we'll move on. Let's see. Thank you. While the slides come up, I'll say everything I'm going to say today is from the perspective of the U.S. stress test, which is what I know about, but at the end I'm going to try to draw some lessons that I think apply more broadly.

I hope apply more broadly. Great. I have to start with a disclaimer. We'll move through this. I was going to start off by telling you all about the details of the U.S. stress tests. That's been fortunately covered already. The key thing for my remarks to pull out of this is that the way we calculate the stress tests is they're calculated for individual banks and they're part of a broader program,

the CCAR program, which has implications for the individual banks that are in the program. Let me spend just a moment here because I don't think this has been discussed quite as much today, so what are the key elements of the stress tests that we do for CCAR and

for the Dodd-Frank Act, which is the stress test, which is what DFAST stands for? The macro scenario that gets fed in gets more stringent as the economy improves. That's sort of built into the way the scenario is designed. The unemployment rate must go up by a certain amount and must hit a certain level. So as the unemployment rate has fallen in the U.S., the amount by which the unemployment

rate increases in the scenario has gone way up. There are similar or in a different vein the way it's done, but similar ideas for what happens to housing prices and other asset prices. They will tend to fall more as they've been running above trend.

Another thing to know is that we do these series of what you would call top-down models, meaning that we do the estimates, there's also bottom-up modeling as part of the CCAR program. Today, everything I'm going to talk about is what the Fed does, our top-down models. So basically those results are we have our own models, the models are the same for

everybody, but we get a lot of very, very granular data from the banks. And so same model, same macro scenario, but the differences are in the data that feeds in that comes from the banks. So I thought it was maybe helpful to take just a moment to think about if we're talking

about macroprudential stress tests, what do we mean by macroprudential? Other people have talked about the distinction I'm going to make. Here I'm certainly not the first, but I think there's two different ways of thinking about what macroprudential means. One is from kind of a structural view that's about identifying important nodes in the system.

Those institutions where if something happens, the negative externalities associated with distress or failure are the most severe and pervasive in the system. And when you think about the policy implications of that, it has something to do with strengthening the potential requirements and supervision at those systemically important nodes in

the banking or financial system. And then there's the view of macroprudential that I think has been the focus of what we've mostly talked about today, which is the cyclical view, which is how are risks of financial stability changing over time, understanding the cycles in credit, asset prices, leverage, liquidity, all the things that we've been talking about today.

And there the policy implications are something about lessening the probability and or the consequences of a turning of the cycle. So in playing those two different thoughts on to what we actually do in the U.S. stress tests, as has already been talked about, what we do is actually project regulatory

capital ratios for the banks in the stress tests, you know, sort of nine quarters forward under the scenario, very much regulatory capital and accounting-based calculations.

But what I really want to focus on here is how the stress tests are calculated. So the stress tests are calculated for each of the banks individually on a standalone approach. There's a lot of attention to projecting out the different pieces of net income and then how that impacts capital. But basically, given the scenario, given the models, given the individual bank data, what

happens to bank A in the calculations doesn't affect what happens at bank B or vice versa, except as Don Cone said, the general scenario is one in which things would be pretty bad for banks in general.

So in thinking about the structural macroprudential elements, you know, why do we come to do this bank-by-bank calculations? And I think it's important to remember how the results are used, you know, dating back to the first time we did stress tests in the middle of the crisis in 2009 and now

is embedded in CCAR. The results of the stress tests have consequences for the individual banks, both supervisory consequences in terms of the supervisory observations that they will get and then potentially consequences for their ability to increase their dividends and repurchases to shareholders.

So because of the sort of microprudential implications of the stress tests, the push in the modeling and thinking about it, I would argue, has been towards sort of accuracy and precision at the firm level. Make the models, even though the models are the same for everybody, get more and

more detailed data, have the models, you know, reflect particular kinds of lending that different banks are doing so that you really get the numbers, given the scenario, as correct as possible at the individual banks. I would say this, you know, arguably addresses the structural macroprudential concerns that

we talked about, that I talked about a minute ago. The primary cyclical element are the design of the scenario, the scenarios, which as I said are designed to be more severe when times are good, but as has already been

talked about by Don and others, these counter cyclical elements compete against improvements in asset quality. There's a little bit of a horse race there, and Don's work with Nellie Lang suggests that the horse race is, you know, maybe a tie or that the asset quality, the improvement asset quality is winning, and most of the sort of counter cyclical elements of the

stress test has to do with the fact that dividends and sharer purchases, which figure into the ultimate capital ratios, have gone up and increased pretty steadily during the expansion. So you know, what kind of macroprudential insights do we get out of the stress tests? I mean, I think there's some limits that it's important to acknowledge.

First of all, these are capital stress tests and don't directly capture liquidity, runs fire sale risks of the type we just were hearing about. The large U.S. banks are separate, are subject to a separate liquidity stress test as part of the CELAR program, but those stress tests are separate from the capital ones.

They aren't linked together. The standalone approach means that all the linkages you might like to know about, bank to bank, banks to non-banks, the financial sector back to the real economy, we don't really capture in these stress tests, except, again, from the fact that the scenario is really, really bad and sort of already presumes that there's a lot of trouble in the economy

and the banking sector. And again, relevant to some of what we just heard, the really complex models and the high level of granular data, loan level in many cases, mean that generating the projections

is really sort of resource and labor intensive, and it's hard to do more than a handful of scenarios raising questions about whether all the risks are being captured, will the vulnerabilities at all the banks be identified. So what does this imply for some of the design choices?

This is what I was actually asked to speak about finally on the last slide here. So I think some of the things to think about is where should the complexity be. In this microprudential setting, as I've said, we drove a lot of complexity to get the numbers right at a detailed level for individual banks. Maybe if the concern is the system as a whole, the complexity should be capturing

how what happens in one part of the system affects another part of the system. So thinking about the purpose of the tests, where do you want to put the complexity in the modeling and the data collection. In terms of data collection, maybe there's a role for more sort of high frequency interday

transactions that really capture the way funding moves during the day and over the course of days between institutions. And then of course there's the data, sort of the low hanging fruit to mention, the data from the non-banks and from what in the U.S. is a very important unregulated sector.

So how complete a picture do we get by looking at the banking sector alone? And then as has already been talked about, the ability to do many scenarios, not just a handful. So I think that's my... Thanks a lot.

Okay, so a few thoughts about how one can use macroprudential stress test to calibrate

macroprudential policies. So basically what is a macroprudential stress test? In the end, you're projecting along the most likely path of your variables, that's a baseline scenario. And then you have a scenario where you have severe shocks, let's say, people say severe

but plausible, that would be the adverse scenario. And ideally you have interactions, so institutions react to these shocks and maybe interact with each other and that feeds back into the economy. So that's a macroprudential stress test. And how do you calibrate? On the other hand, when you think of calibrating a macroprudential policy, how do you look

at that? Well, a standard approach to policy making is to do cost benefit analysis. Now in the specific field of macroprudential policy, there's some challenges because costs are relatively straightforward. We're talking about solvency measures, so increase in capital or borrower-based measures.

But anyway, usually what this does, this slows down the economy one way or another. If it's a capital measure, you're restricting a bit the supplier credit. Because you're doing it in normal times, you have to measure the impact along your business scenario. So you have a little bit of a slowdown in the economy. But what are the benefits?

It's unclear what the benefits are, or there's many ways of looking at it. One way of looking at it, of course, first of all, the benefits should be in some sort of comparable unit. If you say that the benefits are something that is not expressed in the same unit, then it becomes very difficult to compare. So what people have been done for a while is to do some sort of macro analog of the

micro perspective and an expected loss approach. So you say, well, if banks are more resilient, there's a lower probability of a crisis and or a lower cost of a crisis, and therefore, you have a lower expected loss. And then you compare these two magnitudes, the output you lose because you put the

policy versus the output you gain in expectation because you have averted a crisis. You've made it less expensive. Now, the problem here beyond the fact that macro is probably not exactly an analog of micro, but without getting into philosophical issues, measuring that is very difficult.

First of all, because crisis events are few and they tend to be highly correlated, they're at country level, so you're comparing different countries, and these are small samples. And so how meaningful is it if you have in your sample and you are a European country to have a crisis in Mexico?

It's meaningful to an extent, but it's not incredibly meaningful all the time, especially if the crisis in Mexico was 30 years ago, which is what you have in your samples. And then when you estimate, it's not unclear what the cost of a crisis is. For some people, it's a loss of output in a few years following the crisis. For others, it's a change in growth rate.

Now, if you express a change in growth rate in GDP levels, you basically have an infinite amount. So you have estimates of the cost of crisis that can go into the hundreds of points of GDPs. And so it's a bit complicated to put all this together because you have such a wide variation in what your measurement of benefits is, that when you compare it to these costs which

are small fractions of GDP, then it becomes very complicated. Now, what could be an alternative approach to look at the benefits of macroprudential policy which relies less on rare events? And basically, the idea here is to consider that the macroprudential policy, what it

does, it makes your financial system more resilient, not in terms of avoiding a crisis, which is an extreme event, but in reducing the probability and the depth of a severe downturn in the economy. Now, that's something that happens a lot more. You have more observations.

It's a more, let's say, run-of-the-mill concept that you can see what is the impact of having banks in better shape, or financial system in better shape when you have a severe downturn. And now, where does stress testing come in? Well, stress testing basically, if you think of it, already provides you with infrastructure

to compare these things because you can see what is the cost of introducing a macroprudential measure by taking your stress testing infrastructure and going through the baseline without any macroprudential measure, so that's just projecting things as they are, versus by introducing your macroprudential measure, which means

that if you allow banks to react to that in your model, you will have that you're imposing an increase in capital requirements, and therefore banks, the cost of, the weighted average cost of credit will increase a little bit, and therefore you will have a little bit of a slowdown, and that can be done very easily with the models

we have. And that's your cost. And as for the benefits, well, the adverse scenario is a severe but plausible downturn, so it's something that is not entirely fully-fledged systemic crisis, but it's still severe enough that it makes sense to want a resilient system because there should be benefits there, and there again you see what the path of the economy

would be without the measure compared to with the measure, and hopefully if the measure works, you have introduced, for example, a capital buffer that the banks can use, and then they will be deleveraging less by using the buffer than they would have without that buffer, and therefore your path of the economy will be slightly less bad than it would have been otherwise.

And then you have, in this system, you can directly compare your costs and your benefits. You have estimated it with the same model. You've done it for your own country. You didn't need to do cross-country. You have more observations. You have measures that are consistent with each other, and that can give you

a perspective on how to calibrate your policies. Now, there is also an issue of what is the value to you as a policymaker of a unit of output in good times versus a unit of output in bad times, but we won't get there. Let's just assume risk neutrality and they have the same value, but they don't need to be. Now, what would be an extension of this consistent approach?

Instead of having one adverse scenario, like Beverly was saying, it would be interesting to have many different scenarios because with many different scenarios, you can cover a wider variety of risks. So, if your technology allows you to have multiple scenarios, in the limit, you can basically model a whole distribution of outcomes.

The more scenarios you have, the more you have the whole shape of outcomes, and then you can see how the distribution changes with and without your macroprudential measure. And that would be a robust and a complete assessment of the impact of your measure. And you would be confident, you could actually also see in what type of

scenarios your measure isn't working very well. So, this is basically, if you want, an approach that is equivalent to growth at risk but in the policy space. Growth at risk has been thought of as a way to identify risks, how

they materialize. So, something like loser financial conditions today translate into a different shape of the distribution of output tomorrow, and here the idea is the same. An introduction of a policy measure today will change the shape of the distribution of output tomorrow. So, here you can see what this would look like.

The blue line is your baseline, and then you introduce your measure, and you have that blue pocket. That's because the economy has slowed down a bit. Now, it could be the case that after that it picks up again. So, you look at these pockets and you compare them. And what is your benefit is that the lower line, that's your path of output

in the adverse scenario. And you can see the red dotted line is where you would be if you had your policy measure. And the gain is a green shaded area. And this is just for two cases, but imagine this with the whole distribution, and you can just compare the distributions. So, what are the benefits of this approach compared to other ways of looking

at macroprudential policies? Well, compared to standard macro models, here in the stress testing framework, you take into account the heterogeneity of institutions in your sample. Now, sometimes it doesn't matter, but most of the time it matters.

There's a big difference in having all banks that have a management buffer of two percent, say, versus half of your system being very, very close to their thresholds and half of the system having a lot of extra capital. The reaction would be different, but that you can only capture if you have a model that goes at the institution level.

Now, compared to the standard growth at risk model, what you need for policies, you need to understand the transmission channels. At the ECB, we have developed a model that is semi-structural for the macro part, and this allows us to understand the transmission rather than the reduced form

where you're just correlating things. So, in the policy space, it pays off to have some structure. If you move to multiple scenarios, you can also experiment with all sorts of environments and assumptions about your parameters, and in the end, this kind of approach gives you lots of internally consistent outcomes.

You can look at growth at risk or at capital at risk, if you want to call it, credit at risk, whatever you want. You can decide what is a distribution that's relevant for you and look at it in this space. Now, of course, what are the cons of this? Well, this requires large investment, and it's quite complicated to run.

As Beverly was saying before, if you put in all this also interactions among institutions and so on, and you don't limit yourself to banks, but put the whole financial system, this can quickly become completely unmanageable. In Europe, we are relatively lucky because banks are still quite central, so if we confine ourselves to banks, we're not too wrong,

but that's not necessarily the case everywhere. So, the bottom line here is that the macroprudential stress test framework is really, if you look at it as a tool and not as an end in itself, it's really an actual candidate for policy evaluation because it provides estimates of outcomes in a likely and in a bad outcome,

which is pretty much what you want in financial stability, because what you care about is the left hand of the distribution. Conceptually, this can be mapped into growth at risk, which is a nice, elegant, and consistent intellectual framework to look at things, which is probably more satisfying

than the ad hoc macro analog to micro-reasoning. And of course, there's technical challenges that include the integration of system-wide considerations and then also communicating that you have decided on policy measures with this kind of concept. It's not necessarily straightforward,

but I guess Martin will give us some perspectives on communication. And with this, I pass the ball to Jesus. Thank you very much, Carmelo. Let me start by thanking the ECB, and in particular you, for inviting me to this panel. It's really a pleasure and an honor to be here.

If there is a price in this conference for the laziest guy, this is for me because, as Carmelo mentioned, I have now a slight, not a single PowerPoint slide. I'm not the odd one out just only for that. I'm only the odd one out because this is a conference

basically about the frontier of stress tests. All of you work in that frontier. My remarks focus on three points that are far, far, far away from that frontier of the knowledge. These three remarks are very simple.

No complexity. I think that the whole program of the conference has been, to me, too complex. Probably I'm getting too old for this business. But still I think that they may be useful for focusing your analysis.

These three remarks come from more than 15 years of stress testing practical experience, starting with 2005-2006 FSAP-IMF stress test, a very nice exercise. Then it came the 2011-2012 FSAP stress test,

then it came the Troikalet stress test, and I still have some friends from that exercise. John is not here. Matthias is at the end of the room, Edward. Then it came the 2014 ECB or SSM stress testing,

and then it came our annual stress test of all our banks. So I hope that these three remarks are useful for you. The first one has to do with stress testing in bad times.

They can be extremely useful to enhance financial stability and to recover confidence in the banking sector. They can also be useless. I think that the stress test in the U.S. in 2009, also the Spanish banks stress test in 2012, are perfect examples of extremely successful stress tests

to regain confidence and to regain financial stability. This comment, you are not going to like it. The key determinant of the success of a stress test in bad times for me is the existence of a backstop.

It doesn't matter whether you have a very nice scenario, one scenario, 400 scenarios, 20,000 scenarios, whether you have very good models, whether you have very good methodologies. In bad times, what you need to have a successful stress test

is a backstop. If you don't have a backstop, then the stress test becomes much more complex. I know that this is obvious, but I think it's important not to forget it. My second comment has to do with top-down versus bottom-up stress tests.

So basically I know that stress tests can be very complex. Some of the papers have shown this, but for me the stress test conceptually is relatively simple. It's basically you have a starting point of exposures, capital levels, and profitability.

You have scenarios, and you have an engine to transform these scenarios or to apply these scenarios over the initial exposures and then get the final capital depletion. So the key variable here is who controls the engine of this exercise,

conceptually relatively simple. Top-down for me means that the control is in the hands of the authorities. Bottom-up stress test means control by the banks. And if the control is by the banks,

then you have a huge amount of resources devoted to kind of make this bottom-up stress test converge with the idea that the supervisor may have. So to me, clearly, the top-down is something that needs to be reflected carefully

because it means that, like in the U.S., the control of the engine is in the hands of the competent authorities. And I think that this deserves some thoughtful discussion. And finally, the top-down can be very granular.

We have a modest stress testing tool in Spain. It's called Flesbi. Some of Mattias know it very well because it was something that we developed to get out of the program in 2013. We used it, and it was very successful.

We have keep increasing and developing it. And basically, it's a top-down stress test with granular information. And there's no complexity there. Basically, you get loan-by-loan information, collateral-by-collateral information linked to those loans, appraisal value for each of the collateral you have,

and then the haircut from the appraisal value to the real final price. And this is a very simple way of doing stress testing top-down for 90, 95 significant institutions that mainly have credit risk

and that if there is a problem, it would be a credit risk problem. So I think that this top-down approach needs some reflection, and it's not that difficult. It's true that it may consume some resources, but the other approach also is very intensive in resources. And then finally, and coming to the initial Carmelo proposal

on the micro, macro prudential usage of stress testing, usually stress tests, as Beverly mentioned, are performed bang-by-bang. So if you have bang-by-bang results,

I think that there is immediate micro prudential usage. And here, to link very tightly the results of the stress test with the capital requirements, I think that needs also careful reflection because you may end up in a place that is not satisfactory

all the years that you run this exercise. I think that the stress test is a tool to set capital requirements among other potential tools and qualitative assessment that may be needed to set the P2G, for instance. I also, a firm believer after my experience

that the stress test cannot be performed without micro prudential supervisors. We may have a very nice top-down model, but you need the interaction with the micro guys, the guys on the other building down the river or up the river. I don't know which way it flows.

And I think that this is an important part because you need a reality check of your models with the joint supervisory teams that know whether these models are or should be properly applied or not. Double check your results on the projections on the P&L,

projections of credit risk, and so on. And of course, there may be a micro prudential usage for stress testing. To me, this macro prudential usage is to set the stance of macro prudential policy, maybe regarding the CCYB, to set the stance.

It's not easy to link this directly to a direct macro prudential tool. I think that we need to be humble on what we can achieve with this stress test. And finally, and I think that Rochelle's paper had a very good insight, we need to reflect about whether the scenarios should

or should not be counter cyclical. I mean, stress testing banking sector in bad times probably needs a less stringent scenario that in good times. And this is very important because this is a fundamental way

of setting this stance of the macro prudential policy. We also have comments on the governance, but I think that the governance is much easier in bad times than in good times. Thank you. And now to Martin.

Thank you. My name is Martin Schihek. I lead the division in the IMF that runs the financial sector assessments, including stress tests. And it's a pleasure to see so many people listening and talking about stress testing for a whole day.

When we started this 20 years ago, it was a very small exercise and has obviously grown up. And in those early years, when the EF-SUB started about 20 years ago, there was no question about publication or communication of stress tests. In fact, the first 12 assessments we've done,

you can never see them. We agreed not to ever publish them. So since then, the debate has moved on quite dramatically. And now quite often we see the opposite requests for technical assistance or capacity building on communications of results of stress testing and financial stability reporting. And we often tend to push back and say,

this is all very nice, but first make sure that you have a release of solid macro prudential stress testing for your own sake before you go out and publish. And so this chart is from an earlier paper on communication where we sort of put out these pros and cons.

Here I focus, this is the macro prudential stress testing, so it's the kind of ex-ante stress testing as opposed to the crisis management stress testing, which is a bit more tricky. But anyway, even for those stress tests, there are many, many arguments you can make to sort of be cautious. And one again that comes to mind on the macro prudential

is this sort of potential for confusion that again I think we've seen mentioned a couple of times. You sort of, in some ways we are victims of the success of the stress testing because it has proliferated into many different uses. So we have the micro and the macro, bottom up, top down.

And so you have the communication challenge when you come out with a new stress test to explain, well, this is now a purely macro prudential stress test. It's not the one that's just micro with some micro overlay. So that creates a national challenge. The obvious one I mentioned here is this idea

that applies mostly to the micro, which is the attempts to game the tests, and after a while you have this sort of false sense of confidence that you have these very granular micro stress tests, but in a way you may be missing a whole lot of topics that come with interconnectedness. So anyway, I think there are many examples,

and as I'm not picking up on these two, but I think that we can go to also to IMF reports that oftentimes we have missed in our stress tests some risks that were incoming, but these are some examples that obviously didn't turn out the way that these stress tests were by modeling.

And you of course may know that in the Aslan case, it was a very traditional sort of solvency-based stress test and indeed the banks looked very solid and it did not really have the liquidity in it. And again, the Latvia test didn't expect that the growth of GDP in the right next quarter will be minus 18 year on year.

So this is an older paper that I used. This was back in 2006 where we looked at the communications of financial stability in FSRs, and we have updated it now. And it's another new methodology. I stole it from an earlier paper that looked at monetary policy communication and then it looked at dispersion of inflation forecasts.

And what we found is that it's really the quality of communication that matters. And so we've done a redo of this paper to just look on stress testing in FSRs. And again, we following this earlier paper, we sort of have these three measures. One is the clarity and the consistency of reporting and the coverage.

And here when we talk about clarity, it means it's not just you're clear what the outcome is, but you're also clear on what the assumptions are and what the data are, and do you put out the underlying metadata on the website. So we ran several sets of probit regressions looking at probit crisis.

First, just looking at the reporting itself. Do you report or not? But then you also look at the quality of the reporting. Of course, you have to control for the fact that the countries that start to report are more likely to be confident that their system is stable. There was also a period when we saw the stability reports popping up.

So we had to incorporate this Heckman equation to correct for the selection bias. And the results were actually quite striking in the sense that it's the high quality or low quality of the reporting that makes the difference. And again, it's the fact how much do you disclose

on the metadata of the exercise. And again, it's a bit different from the sort of pure crisis management exercise. Publication has no link whatsoever. Whether you publish the FSR or the results of stress test or not has no statistically significant effect.

We were prohibited from publishing the whole list of the FSRs, but at least for the ones that you see here, this is from an IMF working paper, you see that the quality varies quite widely and it's still leaves much to be desired. In this case, the maximum is four for these four components.

And why is that? Well, especially when it comes to the macroprudential stress test, the challenges are quite enormous as we've seen. So this is kind of our standard toolkit. I think that you heard about top-down and bottom-up, each has these components. This is all very nice and dandy,

but then once it comes to the microprudential stress test, ideally we'd like to measure the feedback effect going back from the stress test to the scenario design. Again, this is something that's really challenging first to do, and then to explain why the results attained this way may be quite different from the ones

that you just get in the microprudential stress test. And so what I'm talking about here is reusing the micro data bank by bank data to feed back into the macro scenario design. And we have really done it only on a limited basis and using structural of our models. And again, this is something that's still a work in progress.

So try communicating this, leaving aside all these other issues we discussed earlier today. I think there was an interesting discussion earlier on the building of reputation. And again, once you're trying to build a reputation with microprudential, it's of course even harder

because we're not talking about pass and fail of individual banks. This is about sort of coming up with a bottom line assessment for the system as a whole. So this has been quite practically a challenge, and that's why most of the published stress tests are in a way a simplified version that really gets into the pass and fail and looks at how many banks

will be below a certain threshold. Of course, the challenge of truly microprudential stress tests is that you need to take into account not only the microprudential, but also the other elements of the financial stability assessment. So it's the oversight framework and the safety nets you have in place.

So here's just an example from a Caribbean country that we've done an assessment recently. Again, one, you have very different types of shocking to consider. The issue is when you're doing microprudential stress tests, you're not trying to look at just the cycle, but also you're looking at shocks that hit from, let's say, the U.S. mainland.

Also, you have some natural disasters that may hit. And then you're trying to interpret the severity of the shocks, and when you look at the results, is 22% of capital legacy ratio sufficient? Well, you need to know more about what kind of regulatory and supervisory framework

you have in place. And again, I think this is one sort of simplified way that this particular exercise was presented. I mentioned we've been doing these assessments for 20 years now. So on the occasion of this anniversary, we've been reviewing the stress test

and the whole FSAP. So the review will be finished by summer. Actually, today there's a paper that you can download on the IMF stress testing. But we did a survey of country authorities around the world about what they see as the main priority. Interestingly, the two areas that came most strongly

were interconnectedness, including cross-border, and then it was the non-banks because you need to add up non-banks and market-based finance. So when you add those up, those were important. And then we also had quite intense interest in these emerging topics, like cyber stress testing,

which we've done so far once in Singapore. Work on fintech stress testing again once and then push for more analysis on climate. And some of these new topics make the challenges of communicating stress tests even larger because, of course, with the transition risk on climate,

you're not dealing with a stress test that's more counter-cyclical but something that's more like a structural change in the economy happening over decades. So again, it's a very different type of exercise. We have started in this vein.

We have an ongoing one on Norway. So this is something that's quite exciting. Anyway, back to the topic. Again, I think we are on communication. I think it's important to make this distinction between the experimental exercises we talked about and sort of the basic exercises that Jesus has highlighted

and oftentimes can be quite insightful because they are more transparent and easier to explain as long as you're very clear about the caveats or the assumptions that you're making. Thank you. So, well, let's have questions.

And maybe if you have in mind someone who should answer specifically your question, then... Actually, I would have three questions and let's say maybe we don't cover... I mean, you don't cover all of them, but... So the first one would be for Jesus. Like, you made this point about top-down,

let's say, against bottom-up, clarifying that for you top-down, it means that the machine is in the hands of the policy authority, let's say. And, I mean, if I understand correctly, I understood that you kind of think that the top-down approach, like, let's say, has some advantages

with respect to the bottom-up. I mean, I would tend to share this view, but my question would be now where you think that kind of a top-down exercise could be really conducted in the SSM, let's say, Euro area context, where basically, I mean, let's say, I mean, banks are much more heterogeneous

than maybe in only one country. Basically, we have different business models. The number of banks is much higher than, for example, in the U.S., the ones that are covered in the ADFAST or CCAR. And then, basically, on top also, there is the issue that despite, I mean, the amount of data that we have available is kind of growing and improving year by year, basically.

Still, I mean, we don't have such granular data as, for example, that were used in these papers today from the colleagues from the Fed. So, I mean, in principle, what I mean is that I would tend to agree with your view, but I would like to hear your feedback, basically, whether you think it to be something doable at the current stage, basically, in the SSM context. And then maybe, like, if I might ask

a question to Beverly as well. As regards, you kind of pointed out to some elements that the macroprudential stress test should feature, and I totally agree with those. Something like interconnectedness, et cetera, let's say.

I would understand also that at the moment, like, for example, the fast CCAR do not cover some of these elements because maybe they are more useful for macroprudential purposes. I was wondering whether in the Fed you are planning to conduct, or you are already conducting some similar exercises,

like something like more macroprudential exercises in parallel, or if you are planning to do something like that in the future. And then, maybe, okay, I will have also a fair question, but maybe let's see if...

Okay, I also have a question for Ms. Sifl and Mr. Saorina, regarding, so you mentioned that, of course, one has to weigh the complexity and where to put the complexity in how to design the stress test. Also that it involves a lot of granular data

and a lot of resources in working on this data. So I would like to ask you what you think about how, also regarding the design of the stress test, where the competitive advantage lies on the side of the supervisor and the regulator to work on the data that you have available

in forms of the top-down approach can, of course, being the regulator, having the information on all the banks simultaneously, produce much more information that a bank can produce on its own, and what you think about how to factor that into the design of stress test.

Okay, we'll take one more question, and then we answer this round, then we'll see if there's a second round. Thanks to everyone for the very interesting presentations. One remark to Jesus. You might be accused of laziness with the slides, not with conceptual laziness. That was a very forceful and clear presentation.

Thank you. I have one, I want to go back a bit to Diane's presentation. So you were talking about arbitrage, regulatory arbitrage in connection to stress test. So it was more about playing with risk weights, playing with the allocation, the type of asset.

So let's imagine, and the stress tests are mostly conducted with quarterly data, like the one top-down, very granular quarterly data. So let's imagine you solve that problem. You solve that link. There is still the issue of regulatory arbitrage that takes place within the quarter, and this can be very large, like a canonical example, the repo window dressing by some European banks. This reverberates through the entire financial system.

It's seen in the Fed's balance sheet. It's seen in FX swap bases in currencies which are not the euro. So how should one think about this, and does it maybe change or increase the role of one of the corners of your triangle, namely the supervisors?

Thank you. Maybe you want to start. Jesus and Diane. All right. So the first question was about whether the Fed is working on other forms of stress testing that build in some of the interlinkages. The answer, I think, is yes. I mean, there's a pretty active research community

across the entire Federal Reserve system thinking about and building those models. I don't want to speak for the staff or the Board of Governors. I'm in no position to do so, being at the New York Fed, but there has recently begun to release a financial stability report twice a year that looks at, among other things,

interconnectedness, leverage, liquidity, across a bunch of sectors including banks and non-banks so that there is analysis that is done that goes into looking at some of those other factors We at the New York Fed, in our Liberty Street economic blog series,

released the results of some models that looked at fire sales. I think Fernando, who is sitting over there, is going to talk about some of the modeling that was underneath some of that you'll hear tomorrow. So we're not, at least at the New York Fed, there are not things that are set.

in the same supervisory consequences that CCAR is, because that is a supervisory program. But there's a lot of work and analysis that's going on. I'm sorry, what was the second, I didn't have something to write down. The second question was just. Where to put complexity, the advantage of the supervisor.

Oh, the advantage of the supervisor has. I think the way I would answer that is that it's not so much about whether from the top down or bottom up. They both, at least in the CCAR program, involve a lot of complexity. A big part of what is done in the supervisory review

as part of the CCAR program is looking at the models that the banks use to do their own stress testing. And there's a lot of complexity in both ways. I think the one advantage of the top down, the way the Fed has done it, is it's the same models for everybody. So you know you're getting an apples to apples comparison.

Same model, same scenario. And with the differences varying based on the very granular input data. So that in a sense, the exact same loan held at two different banks should generate the same kind of losses at the two different banks.

And so it's that, it's ability to have a common framework where you can say, well this is everybody together. What does this look like for the system? Even if the system consists of just adding up the pieces. I think that's maybe the advantage, one of the important advantages of having top down estimates.

Thank you, Cosimo. Well the first obvious answer is call the consultants. They will organize you the top down exercise without any problem. Now, as they did in the past, right? So more seriously, I think that one way

you point to the right problem of too many banks, significant amount of banks. But maybe this is, there is a scope for a true cooperation with national authorities that may have models locally applied and then you check the consistency. But in let's say enhanced cooperation

with national competent authorities, I think that it's possible to run this exercise. And then time is on your side because an accredit is building. And there are other national authorities that have credit register with different coverage.

But you also can explore this credit register together with the national authorities. So frankly, I don't think that this is a big problem. It is true that you need to have this as a medium term objective. This is not for tomorrow. Cannot be implemented tomorrow, neither next year or in the coming two years.

But the important thing is whether you agree on the direction of travel or not. For me, this is a fundamental question. Who controls the exercise? If I may ask your colleague next to you,

I think that supervisors have a significant advantage. They can ask for the granular data. I mean, on my right is an example of a supervisor that asks the granular data, long by long, right? So collateral by collateral. This is something that it is not for tomorrow, but I think it's a significant potential ability

that the supervisors have in micro, the macro, both of them. And you need to develop modeling teams and so on. And the cooperation with the national authorities, I think that could be really very, very, very useful. And if I may also, although the question was not for me,

but listen, you are not gonna be never happy with this line of reasoning. Because you say, end of year data, they arbitrage. Then you ask for quarterly data. Then monthly data, then weekly, then daily,

and then intraday data. You are not gonna be happy. The answer is very easy. Send the JSTs to the bank. Thank you. You see what I mean? So about the regulatory arbitrage within the quarter or window dressing, I would call this window dressing.

I'm well aware of that because I've been working also on U.S. money market funds data. And money market funds in the U.S. have to report on monthly basis and European banks report on a quarterly basis. So if you look at investments of U.S. money market funds in European banks, it's very funny to see the seasonality.

So, and why do we have this a bit less in the U.S. is because the U.S. banks have to also report average capital ratios over the quarter. So, and that's supposed that the supervisor is collecting data at a more frequent basis

than the quarterly level. So there are some data actually at the monthly level. That makes a lot of sense. Well, market measures can also be a way because market measures are available on daily basis or even more frequently.

So if you want to compare with market measures, you will also have that dimension. But also in general, asking supervisors, asking for more and more granular data at the single exposure name, at the borrower name level, you might also think that asking for more frequent data

might be, of course there's a cost for that, but that might be also, there might be some value in that. Of course, with this multiplicity of constraints that we impose on banks, they are more and more innovative in the way of doing a regulatory arbitrage. So recently, when I was talking with the Swiss supervisor,

they were telling me that there's also regulatory arbitrage when banks have to report the exposures. So there is, the more you impose constraints and the more severe the constraints are, the more, the little, the room for maneuvers of banks

the more they're going to be innovative, of course, in the way they do arbitrage. Thank you. Other questions?

So thank you to the panel very much. I thought it was really interesting discussion. I had some, I'm not exactly sure who my question is directed at because most of the time when you're talking about stress tests, there were more sort of stress tests where the scenarios feature very sort of acute business and credit cycle downturn type scenarios. But I guess I had a question to do with,

so everyone who works on financial stability talks a lot about low for long, low for long interest rates. And so I suppose my question is, on your thoughts for thinking about more of these types of like very slow burn type scenarios as opposed to more acute type scenarios

and the challenges, usefulness of using stress tests to actually look at these sorts of scenarios, like low for long type scenarios. You know, what were your thoughts on that? Well, I guess I can answer this one because the EBA exercise

that is going to be performed in 2020, the scenario was published a week ago and it is a low for long scenario. So there is the usual downturn and drop in asset prices but there is also a shift downwards and an inversion of the yield curve.

Now, I don't know whether you call, with the three years you call that low for long enough. So maybe there could be a case to try and project that even further. But let's say that this year the ESRB, which is the body that chooses the relevant risk, decided that this was worth exploring.

The current framework is over three years. I think from a top-down perspective, since we will have the starting points, we could extend that for longer. Of course, the question is the more you extend it, the more you would have to model what banks would do because the EBA exercise is a static exercise.

So already it's a bit of a stretch to say that nothing happens over three years. But if you were to wonder what happens if we have another 10 years of low interest rates, then I think this is something that in the current framework in Europe where banks are not asked to provide their reactions,

this is something we would have to do here as a sort of top-down projection where we do have models that project banks' reactions to such a scenario. And indeed, it would be interesting to try that. But we're going in that direction. Okay, that's the short answer to the question. Other questions?

Okay, well then, thanks a lot to everyone. And we'll see you tomorrow morning.