Welcome back to the New Age of Risk Analytics. This episode will be the first in our two part series on the topic of model risk management. In this episode, we speak with model risk management expert, Sonia Sodi. Hello, and welcome again to the New Age of Risk Analytics podcast.
This is Delisa White, and I'm talking with Sonia Sodi. We're talking today about model risk management. Sonia just gave an excellent presentation at the SAS Global Forum. We'd like to share some of the insights from that presentation.
So Sonia has a long experience. We just wanted to clarify that you're speaking from the breadth of your experience and from your own personal point of view and not for the organization for which you currently work. I can explain why it's so especially because when you are in model risk management business, it can be a bit tricky because there is regulations and other things, compliance parts. So that's why I refrain from representing any of my current or previous employees.
It's all my experience. And I think that's most important is to share the knowledge and get to the subject to help the subject grow. So that's what I am here for. Today, you were talking about model risk management.
And our audience comes from all elements of risk management. And they come from a variety of industries as well. So and very often, they move in and out of risk and the business lines. So sometimes somebody moves into the risk arena and they come from an ERM background or an operational risk background.
And then suddenly they're tasked with overseeing elements that include model risk management for the benefit of those people who are moving into a sphere where this has to become part of their consciousness. Can we talk a little bit about what do you mean when you say a model? How does that differ from just a tool? Yeah, what is a model?
I think what is a model is to let's keep it very simple rather than very complicated. When you are using data to create an outcome and when there is an element of uncertainty, when there is an element of estimation, when you estimate something, when you predict something, that and then it is a repeated estimation based on a underlying that's a model. Because that to me is the definition of model. Of course, SR117 has a definition also that is in line with this that if you have an input and you have an output and you're using quantitative methods to do the processing to generate the output, that's a model and these quantitative methods are mainly statistical or mathematical theories that can help you to estimate and or predict what is this prediction used for?
It is used for basically portfolio optimization, prediction of profit and loss, capital, reserves that you need to keep. That is what is usage but there are more usage of the models, it can be used for various other decision making within the bank. I think that was probably a follow-up question that you wanted to ask, is it? Yes, so please go ahead.
So it is basically that when the models, what the models are used for, models should be now considered as a game changer for the transformation of the financial world, not just banking world, because the models can help to take better decisions if you have developed them well, if you have developed them to make decision making more prudent and help to create revenue enhancements and reduce your risk, that is what is the usage of models. Absolutely, can you eliminate it a little bit for our audience whose focus is gonna be on risk? Although, I think risk in general is evolving to be not just a risk management, but also opportunity and value add optimization. But what are the ways that models really approach reducing the risk for a financial institution?
Yeah, I think there are various parts, where the models are currently being used and I see more avenues where models will be used in future. Let's take an example of where models are currently being used to assess the adequacy of the capital requirements. When you assess the adequacy of a capital requirement for a portfolio, then eventually you are in giving a decision, how much capital you should underline what folio should have. And thereby, if you are helping to have the portfolio optimization and take a necessary decision with regard to a conscious decision, I would say, even a calculated decision, rather than what should be there.
So they have reduced the risk that you have from a particular model. Thank you, thank you very, very much for that. Now, what is when people talk about model risk, they're talking specifically about the risks that are introduced by the use of models. So can you illuminate for us what those are?
I think it is not just the use of the model, but there are various sources of model risk. And one could be the data that has been used for the model itself. There are underlying uncertainties in the methodologies that have been used for the model. Example, it could be a standard error of regression, which we all know has certain uncertainties introduced due to that.
And also, a model has been, like you said, it was designed for something, but it has been used for a different purpose. Yes, it can happen if it has not been documented well, that this model was to be used for, let's say, and I used a cartoon actually on my slide for that, that if a model has been designed for early estimates and you started using it for monthly estimates, those kind of things. Yeah, it can happen, you know? So it's better to always have a clear documentation, clear purpose of your model's well-evidenced.
Thank you, thank you very much. How do you manage the model risk? What are some of the approaches that you found in your experience that really help manage those potential elements? Well, I think the most important thing about managing model risk industry is in the evolutionary phase where you really go to an aggregate level at organization to manage model risk.
To reach that level, you really need to set up a good model risk management framework enabled by a model risk management platform, which helps you to ensure that the policies and the framework that you have established to manage the model risk are actually adhered across the organization. And that's the basic foundation you need to do before you start managing model risk. I think the industry is still learning on that subject, and it has to be evolved, but it has to be throughout the lifecycle of the models. How does this intersect with regulators and regulations?
Are there regulations around model risk management? There aren't. I think both in states and in Europe, there are regulations. ECB came up with a general topics chapter last year, and they had to hold the institutions to implement a model risk management framework, which covers policy, register, and inventory guidelines for qualitative aspects of model risk, how you are going to measure that, and the methodologies for assessment and measurement of model risk.
So there are regulations, and it's now time institutions have to cater for these regulations, including defining procedures for model risk management reporting. Okay, can you give us an example of what some of the negative consequences could be of failure in model risk management? Is there a use case that you can give us? Well, there is.
But, yeah, indeed, of course, if you fail to manage model risk, you can easily come up with a model which was used while it was designed for something else, or in a very different convocation, I can say that what could happen is that you have incorrect outcome of the capital requirement, and you thereby take a wrong decision, which can put your financial institution at a great risk. Now, this kind of scenario, where you have taken a wrong decision, while the whole purpose of the models was to help you to take the right decision, that's why you need risk management. Absolutely. Can you give us an example of where this has gone wrong?
A gone, and also the fines that they need to pay. That's a known story where a model was not used or the purpose it was designed for and resulted in, as I said, a use of model for different purpose, and thereby there was a marketing, or was, you know, how do I say this, that it's not about finger pointing that it went wrong, but that resulted in misrepresentation towards the clients, and thereby they had to pay fines to the regulator. Now, these kind of situations should be avoided. Absolutely, absolutely.
Now, what are the components of a model risk management framework? Well, a model risk management components are basically you need to have policies and documents, which are applicable for your organization. You can always have policies which make sense for your business that you do in. You need to then have a clear identification and assessment of models because you need to identify.
You need to have a clear definition of how you have identified, slash classified these models, because depending on the level of classification, you are also defining the level of workflow, slash lifecycle management, or the MRM activities that are applicable for that model. Therefore, you need to have a good clear workflow and lifecycle management defined. And then we, as I mentioned, that you need to have a model inventory, which is also called by the regulator as model register, which captures a model attributes, model information besides other things linked to a model, which include the findings, the action plans, and any model document that is relevant per workflow stage. Last but not least, an important component of MRM is the model risk dashboard, because you need to have the reporting that how the models have inherent risk or the residual risk, and how does that aggregate for the overall organization?
Right, and you were talking in your presentation a little bit about, you know, a lot of people just want to use Excel spreadsheets, and we just put them on Google Sheets, and can you speak to like, why this involves a little bit more complexity? Well, it involves more complexity indeed, because an Excel sheet does not allow you to manage each workflow step. It does not allow you that if suppose a model has been created, then all the documentation of that model is also saved in that Excel, including the data snapshots, because if your data snapshots have not been saved at a good place, which is easily traceable and linked to the model version that you have used, you were never able to prove the model outcome in case the audit here comes to you and wants you to say, how did you come up with that capital requirement when you don't, you can't even replicate your model outcome? Now, Excel doesn't allow you to do that, and of course, I think you need to maintain model stakeholders, you need to maintain model usages, you need to maintain model documents, you need to maintain model data, you need to maintain findings and actions that are linked to that model.
Now, think about the 360 degree or three-dimensional, four-dimensional, five-dimensional things linked to a model, how are you able to ensure that there is performance when you have thousands of models in an organization? Come on, let's be pragmatic here, it's just impossible to have that performance from Excel or access or any of those kind of tools. We need more power here, and while we are talking models, I want to be developed by AI and machine learning, and we use an Excel to maintain information about the models. What are we talking about here?
Yeah, exactly, exactly. So you use more powerful software to intersect, and a lot of people come at this from different software backgrounds, some are using different software for these purposes. You spoke a little bit in your presentation about the integration of all of these tools. Yeah, so that was an answer to a question, because what usually happens is that these models are not always developed in SAS, some of the latest other languages like R and Python are also being used for model development.
Now, the question was, can we also integrate other environments which are used for creating models to provide the model specific information to SAS MRM? And my answer was yes, of course, SAS MRM can also be linked to SAS Enterprise Guide if the models have been developed in SAS, so that the model developer doesn't need to enter all the information about the model again, to SAS modelist management, all the attributes that SAS MRM needs from the Enterprise Guide, there's some basic model information or additional information can be provided. That linkage can be established. At the same time, if there is any other tool or any other interface that is being used for R or Python models, it is possible to link them to SAS MRM.
You can have a simple XML interface to provide the information that SAS MRM needs, so as to ensure that there is no rework at a model developer slash anybody to enter the same information. Another question that I thought was really interesting is if you could explain the difference between internal validation departments and MRM? Yeah, because I think this is, as I said, this is also a constant discussion currently, because people think, oh, model validation or internal review is already checking the model for its fit for purpose or whether it's a good model or not, then what is model risk management going to do? And the answer that I gave was, the validation only looks at the information that is provided by model development.
They don't look at the entire workflow that is linked to a model, as a model risk management needs to look at the entire process or link to a model starting from the conception of a model to the development in validation, implementation, and till the time the model is retired. Now, validation is one step within the whole model lifecycle. So MRM has a, I would say, a overarching role to the entire model lifecycle as compared to model validation. Okay, is there anything else that you think that our audience should know?
Like I said, they're not all modelers, they come from all aspects of risk. What do you think is important for our audience to know about model risk management? What I want to say to the audience is that model risk management is not rocket science. It's not very, it's basic things that you need to cater for when you are developing anything.
Now, it's just that our guys, they find developing a model more flashy and in-thing or something, and they forget that doing these basic, how do you say overall, it is a kind of bureaucracy, I would not deny that, but if you don't do that, your hard work in developing the model is lost and it loses the rationale for which you want you to build that model. So what I would want to say to the audience is that the modelers come from a highly educated, most of them are PhDs or, you know, and then how could, and I'm also learning this, how could this basic workflow around the models has not been established in the industry. It's very, it's a, I don't know how did that happen, but it is time that we should gather ourselves to ensure that models are managed properly, models are recorded well, the life cycle is maintained well, we get a more organized how we manage our models to help ourselves and to help our organizations. You know, sometimes I wonder, I think that the analogy that I would use is sometimes when people are focused on the calculus, they forget about the arithmetic.
Yeah, exactly, exactly. That's the thing, you know, is there so much focus into the calculus and they just forget about the arithmetic and it is so basic things, you know, even in your house, let me put it this way, you would want to keep your things structured so that you can find them when you need them. This is not very difficult. If you keep things disorganized, you will never be able to find them.
We all know that. And if you use a fork as a hairbrush, then you're bound to come out to some trouble. So, you know, certain drugs are used for off label purposes. If you use an aspirin as a bandaid, it's not going to work.
If you use a model for the wrong mat. I like your analogy that if you use aspirin for outdated aspirin, then it's causing you more harm than your, you know, the usage that it was built for. Now, who is responsible for that? Is the person who developed aspirin or is the user who did not check whether the aspirin was outdated?
Now, how do you know that the aspirin was outdated is because it had that information captured somewhere. So you need to provide that information to help your users to use your model correctly. And then that information gets lost during change of hands within various departments within the organization. So it is where if you are passionate about your models, then please be passionate to also keep the information on the usage of your models so that it also passes through.
And then it is used for the purpose that it was developed for. I love the analogy. Thank you. That is really great advice.
And I will leave you with that. Thank you so much for your time. I really appreciate you talking with us today. And same here.
I really appreciate you having me. Thank you. Thank you.