Good afternoon ladies and gentlemen and welcome to today's webinar on model risk, the emerging challenges and opportunities brought to you by Risk.net in association with SaaS. My name is Aaron Walner, country and editor of Risk.net and I'm joined today by a mixture of industry experts. Let's find the full range from solution providers, industry associations to bank risk managers. From SaaS, we have David Asemani, solution lead for model risk management at the firm, joining him is John Hill, head of the New York chapter of the International Model Risk Management Association.
While from HSBC, we have Keith Garber, the head of independent model review at the bank. The role of models was closely scrutinized by its regulators and the aftermath of the financial crisis and there's been an ever-increasing focus on this aspect of the banking sector since. The ECB's targeted review of internal models, or trim for sure, is now in full swing, meaning firms on both sides of the landscape are facing increasing regulatory oversight of the models they use. Additionally, banks are grappling with issues such as the interconnectedness of using multiple models and how machine learning can best be used to improve model outputs.
The ultimate aim, of course, is to improve these models by banks and today's discussion will look not just at the challenges, but also the opportunities to increase focus on model risk management will bring to the banking sector. While financial risk models may be facing you, which have used, they've been around for some time, so before we get to the details of today's discussion, I'd like to bring David in to give us an overview. David, where do we get to where we are today with financial risk models and what do you see as the main emerging risks posed to this sector? Thanks, Aaron.
So from my perspective, again, seeing, again, talking to so many organizations around model risk, that there tends to be this, you know, the number of models within organizations has been growing exponentially. And in addition, there, you know, the complexity of these models, how these models are connected to each other, and then from a regulatory perspective, again, there's more scrutiny across the modeling landscape within organizations. All of those things kind of are adding to a system, and from many organizations and outdated system, you know, we're kind of the tools that are being used to manage this complex set of models, you know, maybe spreadsheets or SharePoint sites or EGRC type solutions. So kind of with some of those tools that, again, have been, you know, maybe we're capable of managing, you know, model risk previously.
The stress on those systems with these kind of emerging, new emerging risks, additional models, et cetera, has really resulted in a lot of teams being stretched very thin, and, you know, not able to, from my perspective, at least, not able to provide the amount of care required for those, you know, for the set of models within organization. Okay, in terms of, you're saying teams are being stretched in because of the number of regulatory requirements that they require to meet? Yeah, I think there's a couple different reasons, right? So, I mean, in general, organizations are looking to become more analytical, right?
And that's increased the number of models. So, you know, previously, right, there was maybe an individual that made a decision on a loan application, right? Today, that's all being done by models, you know, trading. Again, something that maybe was done by individuals that now, again, there are algorithms and models that are used for that.
So, again, a push to become more analytical within organizations has resulted in more models. You have regulatory exercises, like stress testing, at first nine, et cetera, and those activities themselves have increased the amount of models across the organization. So, again, there is the push from a regulatory perspective as far as new models required to meet those activities. But then on the model risk management side, there is also kind of that expectation that's been set to have a full inventory, to understand how models are used, to understand when models start to degrade in performance.
Again, so there's that expectation from that side as well. And then add on top of that some of the complexities around machine learning, where you have the business kind of seeing opportunities, or really feeling that they to be successful and to compete. They need to start using some of these new techniques, better utilizing some of the data they have available. So, there's a push from the business side to use some of those additional machine learning models.
And then again, on the model risk management team, those teams are ultimately responsible for both understanding and mitigating the model risk across an organization. So, kind of having pressures from both sides. So, again, when looking at the growth in financial models and some of the emerging risks, I think kind of again just to summarize the increase in models, the increase in complexities associated with models. And then again having both the talent and the systems to decipher and recognize model risk throughout an organization.
I don't know if you just kind of expand on your point about inventory risk. Is that simply the multiplicity of models around and managing models in your inventory? Is that what you meant by that? Yeah, and John Hill will definitely speak a little bit more about inventory risk.
But think of having, you know, you're required to understand, you know, every model within your organization. You're also kind of responsible to within your organization, you know, what is the definition of a model, right? How do you classify something as a model? And again, the regulations kind of have some, you know, some guidelines around that.
But again, organizations themselves often have their own understanding and definition. So, not only then do you have to understand where these models are. And again, a lot of times you may have models that are just popping up through an organization and not knowing that they're there. Okay, so that's a form of kind of inventory risk, right?
So do I, first of all, do I know where my models are? And then on the second side of that is how are those models kind of interconnected, right? A lot of times you will not have a single model in isolation, right? It's more of a combination of models and maybe even different data processes and human intervention.
And so it's more of kind of a process where models are linked together. And so understanding how those connections are made is another form of inventory risk. So those are just kind of two examples. But yes, kind of having a full understanding of that inventory, where the models are within an organization, and then how they're kind of linked together and used is a massive source of model risk within organizations.
Okay, thank you, David. Well, John, I think David perfectly set you up for the next point. Would you like to kind of expand on the inventory risk issue and what are the challenges emerging risks that you see? Sure.
I'd also like to expand a little bit on David's point, why the number of models seems to be increasing each year is not just because of increased regulatory requirements for stress tests like CRCSOL and the fundamental review of the trading book. It's because of the line between what is a model and what is a non-model, seems to get pushed each year towards the non-model into the spectrum. So that almost each year, regulatory direction is to include more and more software implementations that were not formally considered to be subject to validation, which have to come into the validation mandate of SR117, which is something I'll talk about a little bit later. A good example of that would be anti-money monitoring models, which from most points of view, from any quantitative one of you are not really models at all.
They're transaction monitoring systems, but because of the high risk involved and reputation and financial risk involved with money laundering activities, a very clear direction was given by the Federal Reserve Board. I think three or four years ago that these models also had to be included in the validation domain. So regulatory expansion of definition of what should be included in the validation also contributes to that increase. Now, to get to the topic of inventory risk is something I've been interested in and had worked on recently.
There are two particular problems that I say outstanding problems with having a complete and accurate inventory, which is a requirement clearly stated in SR11-7 and another document produced by the Federal Reserve Board called 1518. And it requires a complete understanding of both upstream and downstream dependencies of models. And that's an area that tends to be incomplete because the model developers should have a complete understanding of their upstream dependencies, but very often they don't know who all the downstream users are. So it can be actually a fairly difficult attestation process to develop a complete understanding of model dependencies.
And even with upstream dependencies, in my experience at three different leading tier banks, model owners who are responsible for defining and identifying those upstream dependencies will trace their input data. They know the data that goes into their model. They have to know that. They'll trace them up to the first upstream model.
And very often they'll stop right there because they don't own that upstream model. They're not responsible for it. They tend not to pursue the question of does that upstream model have another second generation of upstream dependencies. So in my experience, I found that the upstream dependencies provided by developers were also incomplete and that they only covered the first generation above stream dependencies.
Another area of weakness, and this leads to a lot of inventory risk. And by inventory risk, I mean incomplete or inaccurate inventories are what are called EUC or end user controlled models. This is something that David already referred to. They're often spreadsheets.
They could be SaaS programs. They could be R programs. But they're models that are generated on individual users' workstations as a trader or someone involved in risk analytics. But they exist only on a single workstation.
And very often the problem of discovering these models is quite extensive because very often the people who develop the models may not be aware of the requirements for validation and for submitting them into inventory. The most typical pushback I've seen is it's just a spreadsheet. Therefore, it's not a model. That's actually not correct because plenty of spreadsheets can satisfy even the very narrow definition of what a model is.
So EUC is a particular challenge at every bank. And I talked to some third party vendors who provide tools for finding these EUC models. And they said you would be astonished by the large number of EUC models that exist in the top-tier banks that aren't in inventory because of just a lack of communication or a lack of incentive to enter into inventory. So these are the two aspects of inventory risk that Tamir most pronounced, dependencies between models.
And here we really think of models in terms of a model ecosystem. If we take that perspective of them, models do not exist standalone, not function standalone typically, but they are nodes in an ecosystem of models within the firm's inventory. Then it leads to a natural understanding that there are relationships between the nodes of the ecosystem that need to be mapped out and understood. And that is a very clear requirement, particularly for these CCAR exams.
The regulators and bank examiners in CCAR make it very clear they want to complete exclamation of all the upstream dependencies for CCAR models. So those are my concerns about inventory management. The problem is, when I put this question to represent the major banks and I say, are you sure you have a complete and full inventory, I very rarely get the answer. Yes.
Usually you'll say, well, we think we know most of our models, but we can't be sure we know that all of them. And that's the current state of affairs. And I think this is an opportunity for automation to find other ways of discovering models that are in use within a firm and relationships between those models. Thank you, John.
Keith, perhaps, do you agree with John and David in terms of the points they made? I mean, is inventory risk a significant issue for you or do you have other concerns? I'm actually agree. Of course, I think nothing in this world is back and white, including as we were just hearing the model definition.
I'm relatively new to which we see just having joined last year. And I thought with you that it's the world's most complex banking institution, and I'm waiting for someone to tell me or convince me otherwise. So I think then that we probably have the industry's biggest challenge on understanding the full model landscape and aspects of the inventory risk. And I'm pretty confident actually that we do have, just through fairly conventional processes, a very complete model inventory.
And I guess the point about inventory risk is that it's about risk. So I would assert that it's relatively easy to find the biggest and most risky and most material models in the institution, and therefore to capture most of the model risk in your inventory. There will be small cases, edge cases, you know, widespread sheets and things that might meet your definition. And they may be harder to find completely, but I think that probably in terms of the quantum of risk, it's not so hard for firms to discover all the models that they have.
And especially with, you know, as processes mature, see if the virus has come in, or it has come in, eventually you can be very confident. So I think it's about completeness. Of course, the question about the model, non-model boundary, is one that I've been interested in for a while because I've always made the point that it shouldn't matter whether something is a model or not. And this is something that comes out, I think in 11.7, you know, it's clear from a business risk perspective and from discussions with supervisors, is that we want to avoid a difference between an artificial line that this is a model.
This is not where your model gets lots of attention, independent review and great governance. And then your thing that is not quite making the line gets ignored. So then of course, you know, something the organization may have an incentive to argue all day about whether they're saying it's a model or not. So of course, a good practice there is to have a continuum in your governance processes that covers calculations and analytics from models to non-models.
And of course, you know, I think what we've seen, and yes, if you want to talk about the boundary shifting, it's really model governance being a bit of a victim of its own success. Supervisors can see how effective all our efforts on model governance is. And so they want to bring more things into that scope like, you know, to my long-term things or more algo trading things and, you know, more AI things. So that's a bit of a ramble.
But you can see then that I don't think we have a great deal of inventory risk from a risk perspective, at least. Okay. David, I mean, from your perspective, do you have anything to add to the points that Keith and John made? You know, it's always good hearing experts like Keith and John kind of gives, you know, I know I always learn a lot just kind of, you know, just joining the conversation.
So the one additional comment that I, you know, again from my perspective, some of the conversations that I've had is, you know, with this growth in financial markets models, one of the groups that really has struggled is the audit team. And, you know, kind of think of an audit from an auditor's perspective, right? I mean, they're kind of coming in and, you know, they're responsible again for making sure that the policies are followed correctly. So one of the things that, you know, one of the struggles that I have seen is, you know, something simple like model documentation.
And a lot of times organizations, you know, they do not have that central repository or kind of template standards for the documents. And a lot of times it's not, you know, a lot of the work is actually chasing different individuals to kind of collect the information that's required, you know, from a documentation perspective. Another area that I see is kind of performance monitoring. So again, answering the question, you know, how well has this model done in the past?
Will you know, when will you know if the model starts to grade, how is it compared to the benchmarks? These are things in a lot of organizations that, again, the information may be dispersed and may even be with certain individuals. So from an auditor's perspective, again, kind of chasing down and information is not easy. So with, again, that expansion in models, having the consistency to have a, you know, a single location or source where all the documents are kept, having kind of templates that can be used.
So you do have that consistency across groups, across individuals. And then also on the performance monitoring. So have ways to automate that and then have the resultant information in a single, you know, single location. And then also have the ability to kind of trigger alerts and notifications when those models start to perform or degrade.
So I think those are all kind of things that I have seen in a lot of these conversations that is a struggle within organizations and areas where, you know, organizations can take steps forward to kind of mitigate some of that risk. Okay. One of the points you made earlier, David, was about kind of the risk model teams being stretched. And John, I just kind of want to get your take on that.
Is this kind of stretching of resources, is that purely just a kind of regulatory creep or other issues behind perhaps the sort of the increased burden on the kind of financial risk modeling team? I would mention two typical trends that result in what I view is the resources in model risk management chronically being stretched. One is that senior management who allocate funds are reluctant to provide funding for areas that are not profit centers. Model risk management is a regulatory requirement.
It's also should be a business requirement, but it doesn't per save making money for the firm. And secondly, as I mentioned many times, there's this model creep phenomenon going on that both the requirements for rigorous model risk management increase, seem to increase each year. Every year that I've been in a bank exam at leading to your banks, the regulators would say, you're still not there yet. We expect to see improvement each year in the goal of full compliance with F.R.
11.7, which is the current Bible for model validation and model risk management. So there's this constant pressure for doing more work and yet seeing a management very often at firms will not be aware of the requirements and the amount that are necessary. So I've said this many times to my colleagues and it's true. It was not a single year in my years at three different leading tier banks that my team did not have to fall back on short-term consultants from the top four accounting firms, such as EMY or Deloitte or KPMG, because we would get into a crunch for some validation, pressure, some new requirement from the regulators.
And always we were told we'll bring on some consultants, this is just a one-off affair. And that happened year after year after year. In one firm, I was actually asked by the regulators to tally up the amount of money that had been spent on consultants from my team. And even I was shocked when I added up all the numbers, just over two years at this other firm.
I had a team of perhaps 12 quants working for me, risk model validation. And when I looked at the amount that we had spent on external consultants, it came over $5 million over the space of two years just from my team. And when you look at that economy, it just doesn't make any sense. We could have hired full-time employees for a lot less money probably because we were using overseas resources, which are less expensive.
But I never won that argument with management. I never was able to convince them it's more sensible to have a fully staffed team. Even if they're not fully employed the whole time, we put them on secondary projects. Then they bring in consultants on an ad hoc basis.
There's no better example of that than in 2015 when Morgan Stanley was required to do a resubmission of a C-car example because of a conditional pass. I was working at Morgan Stanley at the time. And at one point, in order, this was failure is not an option. A resubmission over $100 million was invested just in bringing on consultants to make sure we would get through that challenge.
And it succeeded, but it was admittedly expensive. So those are the chronic problems with staffing that senior managers don't see the immediate benefit because it really requires far-sightedness looking down the path and seeing that these requirements and inventories continue to grow and requirements are going to continue to grow. So that is my explanation of why I think we're chronically under, the teams tend to be understaffed, may not be true at every firm, but I think it's a general truth in the industry. And I should point out now that the Federal Reserve has finally announced that US banks will only be doing the C-car distress test now once every two years, which will actually be interpreted as a need for fewer model risk managers at many firms possibly.
So there's a second that's going to impact the staffing issue. So that's my view on it. Okay. So that is actually going to make an excuse for management to kind of reduce the focus on the business.
I think given what you just said, we've had a bunch of really interesting questions coming from the audience. And there's one I want to put to the panelists here, and it relates very much to the point that you just made. And it's talking about the risk model review team, and we've had a question come in. And then someone who just asked, how do you prioritize?
I mean, if you've talked about your multiple number of models, do you have a kind of a system to prioritize which models you assess and which are more important? I mean, what advice would you have to someone do them with that? That's the whole panel. Who would like to take that question from the audience?
Sure. I'll start with that one. Every firm today is going through the process of risk-curing their models, and that's actually a requirement, again, of the regulatory guidance on model risk management. So most firms will end up classifying their models as high and medium or low risk, with much more attention and greater being given to the iris models, and at least being given the lowest models.
So that's how the prioritization is done. High risk models, you get the most attention, and probably don't need to say this, but all, for instance, models used with the secret stress test are by default high risk. So they get a great deal more attention than businesses' usual models would get, for instance. Okay.
And Keith, I think that is something that you obviously looked at. I mean, you're the overall head of model review, and you're looking at the sort of, you mentioned the large number of models with HSBC. Do you find the prioritization of reviewing those models? Is that significant issue?
And do you have any advice on the best way to approach that? It's certainly important. As John mentioned, every firm will have various ways to stratify and clear the model population as a first step to determining what the governance and validation approach should be. Although maybe that's not quite the same as prioritization.
It's more about deciding what approach to take for each model. But of course, looking across a complex banking group, you also have to recognize that prioritization, there may be different perspectives, and there may be a global perspective and a local perspective. It could be that in a branch or some part of a large group, there is a model which is a huge attention to the local manager and practical supervisors' local business risk needed, of course, from a global perspective, it might look small. So as the global model currency team, you have to be sympathetic to the local view of importance and try to manage those conflicting prioritizations accordingly.
Of course, it can be difficult to get that balance right, especially if resources are scarce or all time this time. And then my view and advice is that the best thing to do is to be transparent with management. This is what we can do. It's what we can't do if we need to do more than the defunding gap or re-paratization, what's it that you want us not to do in order to do this other thing.
So charity is transparency. That's my advice. Fantastic. We have a number of really interesting questions coming from the audience.
In terms of I'll try and bring some into the discussion, and also there will be some time at the end of the webinar to do some Q&A. So please keep sending your questions in. I will put those to the panel. I'd like to just kind of move on to the third bullet point in terms of it's come up a few times so far in terms of SR-11-7.
It's been the biggest move and most significant move forward in the kind of tremendous follow-down. I mean, John, I'd like to kind of bring you on this question. I mean, twice then has 11-7 improved standards. I mean, how significant has that been?
Well, first of all, we need to start with the original preceding document, which was issued by the OCC in the year 2000. And it was known as a 2000-16. And that is a document that actually set the bar for model validation and modelness management at all conforming banks. And it really changed the playing field.
I was involved in model validation a few years later, and I could tell that model risk management and model validators were given a great deal more authority by this document, 2000-16, because it specified for the very first time that model developers were required to provide documentation that was sufficiently detailed to allow complete replication by a qualified individual of that model without reference to the developers or the development code. That's actually a critical criteria because it said prior to that model developers were very hazard about documenting their models. And this set of standards for model documentation that really made the job of validation much more precise. So, move the clock forward 11 years.
This document was superseded by an expanded version issued jointly by the upper beginning of the OCC called SR11-7. The 11 stands for the year 2011 and 7, the bulletin number for that year. And just on the side, I don't think I've ever met anyone who actually knows what SR stands for. So I'll throw that out.
Does anyone on this call know what SR stands for? Well, here's a way. Now, you can dazzle your management because it stands for supervision and regulation. So, that's the FRB's code.
You have to do it this way. Survision and regulatory and 11-7, it was issued. And it's an expanded version of that document referred to as 2016. It went from 9 pages to 21 pages and it expanded a radial on the concept of modelless management and set the bar in the playing field for all U.S.
banks. But it was such a complete and comprehensive document, and I said a very effective document, that it has become the de facto standard for almost all European and British banks. It's constantly referenced by banks abroad as well as Asian banks. And so far to this day, I don't think any European or other regulatory agency has produced a document that could compete with 11-7 for comprehensiveness.
Among other things, it also defines the role of internal audit and overseeing a modelless management, which was not a subject that was covered in the original 2016. It also introduced a concept of stress testing models, which is critically important now. But again, the original document didn't talk about testing models to their limit. So that's a bit of the history.
11-7 is the Bible, certainly in the United States, the standard against which all of us management is impaired. And the regulatory exams examiners will always give an assessment of how far along a firm is in complying with 11-7. They said that I think every major bank that I know of, and probably second and third tier banks as well, had gone through a process we call GAP analysis, which is comparing our modelless management practice to every paragraph. And that's our 11-7 to make sure every single point is covered every firm I've been involved with and that I know about has gone through what we call the 11-7 GAP analysis.
So that's the history of the document. It's 21 pages. It's a very good read. And it still is what I would say is instructive, but not prescriptive.
But this document tells what the regulators expect banks to do to comply with modelless management. It doesn't tell them how to do it. And that's a critical distinction. It's up to banks to find the best ways to achieve the goals that are laid out in 11-7.
And by the way, 11-7 does allow for a risk-weighted approach to validation, which allows firms to allocate their resources preferentially to the high risk models and with less resources being allocated to the lower risk models. And so that's my quick summary that everybody knows what SR stands for. I was going to take a stab at standard regulation. That was my best guess.
But yeah, it's quite close. When I have to say that, 21 pages is kind of from regulatory standards, that's kind of a remarkably brief. Keith, perhaps I could bring you in at this point. From your perspective, do you agree with John's point about SR 11-7 being the kind of the bible of financial model risk management?
It's certainly been very influential and it's been interesting to be in the industry and see the spread of similar approaches to model governance gradually across the globe. So if the question is had it helped to improve modeling and model governance standards, I think the answer is definitely yes. It's also interesting. I think that spread is continuing.
We've seen other regulations coming out, which are very much in line, SSA 18 in the UK, in Canada, et cetera. And I think the interesting thing about TRIM is it's actually quite different. And so the ECB has taken much more of a rules-based approach, rather than a principle-based approach, as John was describing, the TRIM requirements include very specific named tests to be performed on certain kinds of models. So that's a bit of a shock, I think, to us all to adapt to a much more rules-based approach than a principles-based approach.
Maybe that's something about the degree to which the ECB has been satisfied with the principles-based success in meeting the principles by firms who are following a principles-based approach. David, from your perspective, you've already kind of a global overview. Where do you sit there on the kind of the relative importance of that? And do you think it's significant that the ECB has made to move away from the principles to a more of a rules-based approach with TRIM?
One of the things that we have seen over here at SAS, initially there was a big interest in clients, especially here in the US, that were looking to satisfy all the requirements around SR-117. And so that was, in many ways, the driver for us in our inner tool to make sure that we completely can kind of satisfy the requirements of those laid out in SR-117. So we did have a lot of interest in the US. I would say over the last two or three years, we have seen also kind of a very big increase in Europe.
And I think a lot of that has to do with kind of the regulations, TRIM being one of them. But just again, kind of in some ways, the groundwork that was laid for SR-117 has kind of also influenced from a global perspective. So we have seen a lot of interest in model risk management or increased focus in model risk management over, I would say, again, the last two years, year and a half. And again, TRIM is part of that, but not the only kind of driver.
And we're also, at this point, seeing that interest also kind of shift over into Asia as well. And again, that is actually just starting to happen. So again, a lot of it is driven by the regulations. But I think one of the other kind of things outside of kind of the standards and regulations has been just the general acknowledgement that some of the new techniques that are being used, like machine learning, is introducing kind of, introducing risks, model risk.
Whether that be a regulatory perspective or financial perspective or a reputation. So again, I think there is that general understanding that this is a discipline that continues to evolve to meet the different kind of obstacles or challenges that are out there. I think regulation is a big part of it. Over the last year we have seen, and again, John kind of mentioned some of the regulations stress testing maybe kind of not as firm as it had been.
But in the model risk management space, again, we are seeing a lot of kind of reaction to kind of, again, that bar that has been raised as far as a regulatory perspective on model risk management. And that has resulted in a flurry of activity for SaaS. So again, I think from a general regulatory perspective in model risk management, again, the expectations are that will continue to rise and has been rising. And then on the business side, I think there is that kind of general understanding and expectation that maybe some of the new techniques, especially in machine learning, are introducing some complexities that organizations are trying to do their best to both understand and mitigate.
That's perfect timing because we've just had a very interesting question coming from a European bank on the issue of machine learning. And specifically, I'll bring that question into the Q&A side of things. But John, I know from your side that machine learning is an area that you've been focusing on. I mean, just in terms of general financial risk modelling, how significant is the addition of machine learning to this financial risk model?
I mean, from an industry perspective. Sure. I go to annual risk conferences, which will always have some stream on model risk. And I can say with authority that four to five years ago, I never heard to talk about using AI and machine learning in the model risk management space.
At a recent conference in San Francisco, fully half the presentations run various aspects of machine learning and AI. So it's really an idea of this time that's come. It was just to say that 10 years ago that I didn't think we would see self-driving cars in our lifetimes, and now 10 years later, self-driving cars are running over pedestrians out in California. So the regular progress is made with machine learning.
Fortunately, when there is applied to finance, the only thing that could possibly get lost is money. But there's a great, and definitely it varies from firm to firm, how much investment is being made. There's one outstanding leading firm that I think has really set the bar for employing machine learning techniques in the model validation model risk management space. And there are many different areas I think that machine learning can be used now in machine learning models to offload some of the most tedious and manual parts of model validation.
I used to say that I did my first model validation myself back in 2003, and up until 2018, I could say that the process was completely static. It really hadn't changed. We had made incremental improvements. We had their tools for replicating the models, but it was still a very manual process of retrieving acquiring the model documentation, reviewing the documentation, making sure all the checkpoints are developing benchmarks to test the model against going to all those designing tests, sweet tests for independent testing, and then running up the validation.
Some of those parts of the validation process can be taken over by machine learning. For instance, there are language parsing machine learning programs now that do a very good job of analyzing content and readability of documents. They could be used to do a first pass on review of documentation, which is a requirement for any validation document to read the quality of the model documentation. Also reviewing input data and assessing the quality of the input data and even correcting errors that may occur, recognizing principles of extreme outliers as idiosyncratic errors or filling in back-filling missing data.
These are all things that could be performed to call it data grooming or data cleansing. It can be turned over to machine learning programs, and then of course the generation of benchmark programs. One of our leaders will spend a lot of time developing benchmarks, which are alternative programs, as a way of sanity checking the production model with their validating. But their machine learning programs are very good at generating benchmarks based on input data.
The conundrum, and there's a conundrum here as well, because both first and second line of defense developers and validators are now introducing AI machine learning programs for pricing models for risk models. There's a conundrum that has one validate a machine learning model, because one of the notorious issues with machine learning is that the models tend to be opaque. It's not very often not clear what is actually going on inside the model. And I would compare it to the vendor problem, the problem of validating a vendor model.
And by the way, SR117 makes it very clear that the firm uses vendor models. Those vendor models are subject to the same validation requirements as an internally developed model. And if it's a good responsible vendor, they will provide evidence that the model has been validated to 11 set of standards. But very often, vendors won't do that.
And then it's up to the client to validate a vendor model. That's a black box problem. How do you validate something that the vendor won't tell you what's going on inside the box? And what you do, if you're benchmark or you replicate doesn't agree, the same problem with machine learning.
So there's an issue there of opacity and lack of transparency in the machine learning models. Secondly, machine learning models have a huge appetite for immense amounts of data. It's going to be difficult for a human reviewer to stay on top of all the different dimensions of data that a machine learning model might be able to consume in order to produce a desired result. Nevertheless, this is the trend.
I call it a major disruption. The combination of machine learning and big data is coming. And it's already being employed at some firms, some leading firms. But it's going to fundamentally change the way modelist managers are performing their job.
And I would say over the next five years, we're going to see a major C change in the way that we perform modelist management and machine learning and that big data are going to play a major role in convergence between modelist management and those two other disciplines, which is finally coming. So that's my view on machine learning. That is the future. And I recommend everybody involve the modelist management to learn the confluent in these concepts.
That's a machine learning of the future. David, do you agree? Yes, we're seeing, again, this has been one of the drivers from our perspective in modelist management. And John kind of laid out kind of both sides of this.
So on one side, we see machine learning and AI being a great tool that kind of helps with the overall validation and model risk management. So whether it's using the tools and something that we've been playing with or working on is the ability to do some auto documentation, right, in which their tools that can kind of reduce some of the repetitive kind of low complexity documentation that happens, but takes a lot of time within an organization, feature selection within models or other tools, benchmark comparisons, where you're able to use some of these machine learning techniques to kind of improve your overall modelist management framework. On the other side, you have your task with kind of governing these machine learning models. And when you kind of just take, again, for most of these techniques, in some ways the risk shifts from the model itself to the data.
And the way I like to kind of describe it is that model is going to tell the stories within that data. So if that data changes to a certain degree, right, if it's significantly different than the data that was used to train the model, or again, the data has changed dramatically very quickly. There is a chance that that model is going to run into some serious problems. So again, there's the shift in kind of governing those models properly, kind of shifts to the data side.
We've talked about interconnectedness of models, but there's also having a complete understanding of what data is flowing into what models, and perhaps even having some measurements on that data, right? How many gaps is it? What is the stability of the data, et cetera? And understanding it from that perspective is a form and kind of blurs the line between data governance and model governance in some ways.
And then on the other side of perhaps that black box is understanding the results. And so more time, I know, again, maybe about two years ago, we built our system to kind of automate the performance monitoring of models. And when we were doing so, kind of the requirements where models have performed, performance monitoring perhaps once a quarter. But what we quickly found out, and I'm glad that we built it this way, was that this is not something that is something you can do on a quarterly basis for some of these models, especially on the reinforced learning side, you need the ability to kind of monitor these models as frequently as possible.
And it could be back testing, could be performance, how well the models are doing, projected versus actual, or it could actually be the projected values versus the projected values of a simpler statistical fine-tuned model and use that as a benchmark, and to do those metrics that way. So again, we've kind of really seen an area where a lot of the spreadsheets and manual work needed on the performance measurement side has kind of greatly kind of that has shifted, and now there needs to be this done in an automated way. Okay, just at this point, I mean, we're talking about the whole machine learning side of things, and then it's something, hello? One example.
So there's a sound going on the balance. Yes, so Keith, in terms of your perspective, you're from the industry, machine learning is it from the future, or is machine learning the future, or is it the obvious? It's the present, of course, you know, it's been a great deal of expectation for the adoption of machine learning techniques in finance, and we really are seeing that. Also, we're getting close to the end of the hour, so I went to speak a lot on this, but I'll just come with a point.
One is, of course, that it's all about the fit between the solution and the machine learning solution and the performance statement. So, not all aspects of the financial modelling are, you know, including big data analysis for machine learning techniques, so there's a two-to-one point. And then, actually, I think it's alluded to just now, that the emphasis or the focus for governance around these techniques is, you know, it includes certain data, but also has to do the control framework around the algorithm and the way it works. So, what the game is a bit like, I'll go create a circuit breaker, or what are the controls to prevent the machine going haywire, or, you know, behaving in a way that isn't successful in life use.
So, just a couple of points there. And also, of course, they're all focused on interpretability of machine learning tools, but we are seeing the vendors of the systems now beginning to really focus on that and providing more insight into, you know, allowing us to peer under the hurt and see how the algorithms are working, trying that. That problem, instead of recognising, is being addressed already for us. Okay.
So, in terms of that, the algo-classifying machine learning algorithms as models, that's being dealt with. The interpretability question, the tools are appearing to allow us to sort of interpret the algorithms to see what they're doing, how they're behaving, how they're fitting. It's not such a black box as has been feared in all cases. Okay.
Okay. Well, just moving on to the last few minutes, we've had some of the excellent questions coming from the audience. I'd like to put some of those to the panel themselves. I'm just trying to kind of go through some of those in terms of, it's quite, when someone from the, I'm guessing it's a European bank, but the question is, is trim superseding for European banks, is trim now performing the same role for European banks that SR-11 has performed with US banks?
Is that change happening? Is there anyone from the panel? I'm Keith, you're from a European bank. I mean, perhaps you could take the outcome on that.
After a large international bank grant deals with multiple supervisors and jurisdictions, so, you know, we have been, of course, over the last few years, focusing on our model governments and ensuring that it meets all the expectations of all our supervisors. But as I mentioned a few minutes ago, I think the difference with trim compared to the others is that it is less principles-based than, and more, more rules-based. So, we are having to think about it more developing our approaches to make sure that we really explicitly meet those rules and can sort of tick them off rather than having a discussion around why we think the approach that we're taking is adequate from a resource perspective, which is really what our other supervisors are more focused on. Okay.
A very interesting question coming from Australia, and I think it relates back to what sort of panelists were saying earlier on about the issue of how stretched some of the risk modeling departments are and perhaps some of the kind of staffing issues, the wrong word, but perhaps, and a few of people who have been overworked. So, some of this comes from Australia asking, with FRTB and risk-free rate changes, both involving new models, do you see a need for more quants and developers and validators over the next three years? I mean, I'm not saying that that needs will be met, but is there any need in the industry for more quants and developers and validators to deal with these requirements, or is it just a question that firms are going to have to do with what they've got? David, perhaps you could take that.
So, I think the easy question is yes. Again, with some of the things that we've spoken here, with those additional requirements from the new regulations, the additional models, machine learning models, there is going to be a higher demand and need for that type of skill set, and it's something that we're seeing today. When you look across some of the job boards, et cetera, you see there is always a need, an increasing need for those that have the analytical skills that can do this type of work. Again, those individuals are in high demand and other as well.
Absolutely, I think that will be one of the challenges moving forward is having the appropriate talent to deliver what we've been talking about here with these increased challenges. I'll paraphrase a popular book by saying very often, at least in my experience, it takes an MRA, which is a citation issued by US regulators saying, you need to fix something. My team actually once got the only MRA I ever loved, which was from the regulators saying, my team was understaffed. For the first time in my career, my management came to me and said, how many staff do you need to meet this challenge?
I said, I think, too additional. This was for consultants, actually, too additional to do it. They said, you'll take five. I was actually ordered to take more staff than I'd asked for because there was an MRA moving in the background, and sometimes it takes that incentive to motivate the management.
Okay. All right. So, did you make a good use of all five members of staff? Did you need them in the end?
Well, sure. Since we were always planning to understand, I could put them on projects that weren't directly related to the immediate challenge as well. But it was short-term. It was for three or four months.
But my view is that management will always respond to regulatory requirements very promptly, and that loosens its purse strings better than any other incentive that I discovered. Yeah. I think that makes a total sense. We're coming up very close to the end of the one-arm webinar.
I just wanted a very quick final point. David, we've discussed a lot of issues here, whether it's kind of machine learning, what's happening with regulation, and in terms of classifying model risk. I mean, what is the one takeaway that listeners should focus on in terms of improving their model risk management? What's the most important issue to look at?
Brief answer, please. Yeah. I mean, I would say this, that the organizations, again, that I've been working with or have had conversations with that have the champions within an organization that are investing not only in spending time looking at what they can do today, but also the challenges that are moving forward. So to me, I always love these type of conversations.
I think in general, it kind of attracts those champions that are really looking to kind of move forward in this discipline. So again, take away from me is this is something that is changing every year and actually more frequently than that. And so being part of the conversation, asking questions, what are some things that you think you should be able to answer today within your organization that you cannot, making things simpler and standardizing where possible. These are all kind of concepts and having that driving force that kind of leads forward.
That's where you have successful organizations from a model risk management perspective. Well, thank you, David. We just come up to the end of time. And thank you very much for panelists for input.
And thank you to all the people that have sent questions in. And my apologies for those of you that we didn't have the time to answer. Thank you for interacting. And I'm sure I'm sure some of the questions were answered in some of the other panelists answers.
So I think just to kind of finish off, I think David's point about finding a champion and looking towards the future is critical to understanding the direction of financial risk model management. And it's clearly something that's going to become more significant for the banking sector over the coming years. I suspect this is a conversation that we're going to come back to. But for today, that's the end of our webinar.
So thank you to David, to John and to Keith and to SaaS for supporting this. My name is Aaron Walner. I'm contrary ahead of the risk.net. And I look forward to speaking to you on the next webinar.
Thank you. And good afternoon.