I'm your host, Paul Marshall, and we also have our co-host here, Mr. Dan Featherly. Hey, Dan. Hey, Paul.
How are you doing? Good, good. Happy to have you back here. And of course, our esteemed guest, Taka Arriga.
How are you doing there? Thank you, Paul. Thank you, Dan, for having me. How awesome.
Well, welcome back. Actually, I'm not sure if you've been on this podcast, but I feel like you have, so honorarily you've been here before. Actually, do you mind just reminding folks what your day job is there so they kind of know what's going on? Sure.
I'm the Chief Data Scientist and the Director of Innovation Lab for the Carbon Accountability Office. And what that really means is, you know, we're in the business of exploring the impact of emerging technology and how GAO might conduct our evaluation, but also explore the kind of use cases that are appropriate in an oversight context. And so I've never heard all day. All right.
So this will be more of a free-flowing podcast, the ones that I like the most, but we wanted to generally speak about, you know, AI from an ERM perspective or just risk perspective in general. To get your thoughts, opinions, and, you know, we'll share ours and we can go back and forth and debate, I guess. But the first topic really was, you know, what do you see as some of the risks from AI being used for government accountability work? And it's a very, I'm trying to make it not too general a question, but there's a lot of applications now out on the market and experimental things.
There's ChatGPT, there's all kinds of AI that can, you know, do anything, write songs, do whatever. I mean, all that's not going to be government accountability work, but there's definitely tools that can help you do really good OCR, you know, intelligent character recognition too. You know, they can read any document and maybe extract things to put in your financial system, et cetera, things like that. And some people are using it, some agencies are, and some are kind of worried about it.
I don't know, let's start talking about, what are your thoughts, just generally from those kind of tools, some of the risks that you envision? Yeah, and I think that's a great framing. When it comes to artificial intelligence, in my view, they're usually disconnected conversation, right? We talk about transformation, we talk about modernization, we talk about productivity gains, and I think all of those are true.
You know, with evidence of generative AI, we're now able to do things that are categorically in the domain of science fiction only a couple of years ago. We can generate, you know, I'll put this in quotation mark, artwork by simply describing them. We can generate functioning software code by not having even taken a single day of programming. So there are wondrous possibilities, but at the end of the day for the federal government, we are concerned with risks that are, for example, governance.
You know, how do we decide whether we buy versus build AI capabilities? Do we have sufficient sort of volume of visual-literate workforce to undertake these inherently governmental functions? So for example, understanding outputs of AI, be able to sort of technically implement and direct the work development of AI capabilities. Regulatory compliance.
We often talk about, there are a lot of legislative agenda around AI that's happening in Senate and in the House. But, you know, AI or no AI, which you're still not violating, for example, the Civil Rights Act, the Voting Rights Act. So there's some legacy compliance posture that are necessary even in the AI world. Cyber security is another risk.
We often talk about zero-trust of human beings. I'm not sure we spend enough time talking about zero-trust of algorithm. And then there are all sorts of, perhaps, you know, more mundane kind of operations and maintenance kind of issues associated with AI. Do we have the capacity to routinely check on the performance of these models as they continue to sort of drift?
And last but not the least, I think this is a new development in the AI world. There are now affirmative requirements around equity and ethics on use of AI. I don't specifically recall when laptop computers came online that anybody was talking about equity of laptop computers. But I think rightfully so with the evidence of artificial intelligence proliferating.
There are a lot of conversations around how do we use AI in a way that's responsible and accountable. Right. So yeah, a lot of things. I have a lot of questions on these.
Dan, I'm here too. Yeah. I mean, you just threw the boatload at us right there. I think you covered everything in a few sentences right there.
So my question is, where do you guys start? Like when you're attacking AI? What's your perspective on bite-sized apples? You know, where do you attack?
Because it's impacting everything. Yeah, that's a great question. You know, AI could be sometimes scary, right? People view it as sort of a black box.
You don't really understand the underlying performance of that. You don't know what goes into it. You don't know what comes out of it. But my view, and you know, to answer your point, if we sort of unpack AI into its component elements.
So for example, in any AI implementation, you likely will encounter some sort of machine learning models underneath it. Whether that's your deep learning model or a large language model. There are very specific functions that those models are supposed to do. Whether it's image recognition.
Paul, you mentioned OCR. You know, whether that's sort of generating images, for example. So those are the, there's modeling aspects to AI. And then there's data elements to AI, right?
So we often talk about AI being biased. I actually think what we really need is that the underlying data that we use to train these models have some sort of skew or historical inequity. And so if we sort of specifically focus on the data elements of AI, we can probably more readily address some of these sort of inequity issues that may be baked into it. There is a CX element to AI because you need to have some sort of front-end user interface to interact with users of these systems.
So this is where hallucination comes in. This is where some of these toxicity may come in. So how do we sort of address that from a user experience perspective? From cybersecurity, from operational, you know, I think we already have a lot of good tools in addressing those functions.
And then I think we have a lot of good ideas around how we can continue to augment the digital literacy of the federal workforce. So in my view, if you unpack them into those component elements, these issues, while daunting, I think become more manageable. And we oftentimes already have existing risk management framework to tackle elements of these issues. It doesn't mean that we can just transplant, for example, IT governance into AI governance.
But it could serve as a starting point as we address AI-specific challenges. Yeah, I mean, I would think, you know, obviously we're not going to just jump from where we are today straight into AI doing everything. That's ridiculous. So we have to, you know, I guess we would just take every, okay, here's a new tool.
Here's what it says it does. Here's why we think it would be a benefit. Now let's closely analyze and carefully look at what it does. You know, have these human-in-the-loop things.
Have this, you know, test it out. I think that's a great point. It's important for us to take a step back. There's so much hype around AI.
It's almost like people are walking around the hallways with AI hammer to say, who has a nail for me to sort of develop something involving AI. My suggestion would always be try to focus on the problem that we're looking to solve. And perhaps more often than not, we end up arriving at a simpler solution that may not involve the complexity of AI, right? And, you know, I'll just use GA as an example.
We certainly have a lot of sort of kind of analytical challenges that we're looking to address, but not all of them are appropriate in AI context. So we do go through a lot of sort of work in evaluating what is the question that we're being asked to resolve. And if AI is part of that solution, great. The second part is, you know, we often talk about human-in-the-loop, which in general, I agree with that idea.
But that human-in-the-loop has to be digital-literate in terms of understanding how AI behaves. Because if you put my daughter, who happens to be five years old, in that loop, I'm not sure she would necessarily be that effective in sort of looking for, for example, hallucination or even answers that are perhaps inappropriate in the context of carrying out federal agency missions. Yeah, I think one of the things that can feel scarier, you know, you don't have control of the situation is, the chat GBC is a great example because everybody's looking at a plane with it. You know, you ask any question, it gives you these answers based on who knows what.
It goes on the internet or something. I mean, I know the folks that are prototyping using it are trying to make it a much more closed system. So, you know, you can only answer my questions based on this document I dropped in, right? And then even when you answer it, you must give me a citation of exactly where you found that answer in this document.
That seems like a nice way to kind of, you know, slowly step into the capabilities of that software without just letting it go wild, right? Yeah, that's absolutely right. I think the question that you briefly mentioned is absolutely the concern that we have. Who knows what, right?
There's so much excitement out there, you know, from a GEO perspective, we certainly don't want people to just jump into a third-party tool like ChatGV to say, oh, let's see if I can pre-write an audit report. It's wholly inappropriate for us to be uploading, for example, non-public sensitive information even for the purpose of driving productivity. So, you know, we don't know the totality of all the sort of appropriate use cases even for GAO. As a matter of fact, back in January of this year, we issued a report identifying at least 1200 use cases across all corners of the federal government.
But most of them are still in early stages of development. That tells me that there are likely more ideas that are being generated. And again, using GAO as an example, based on our generative AI prototype with a use case of supporting knowledge management, now we have at least 30 additional ideas that have been submitted to us in terms of possible use of something like generative AI. And so we don't specifically know what the upper bounds of those use cases are, and I think we certainly don't have all the answers around what those use cases may look like.
But the important part is how do we enable that safe and sort of secure way of exploring those use cases without introducing unnecessary risk? And so the way that GAO has tackled, at least from a generative AI perspective, is that we have deployed a large language foundation model inside our secure cloud environment with a goal of controlling what comes in, but more importantly, what leaves our boundary. And that establishes a sort of sandbox environment where we can appropriately experiment on, for example, retrieval augmented generation. How do we point these models to trusted GAO bodies of data so that we can mitigate some of the effects of these models being trained on all the internet, for example?
Are you kind of comparing those between, okay, you have the safe GAO data versus what else is out there? Are you kind of running them in parallel to kind of see what potential harm could come or what potential good could come? Eventually. At this point, we're looking primarily at, for example, using GAO published report as a primary source of that knowledge management.
Right. Because people often ask, has GAO done in the topic of X? Sure. The answer is yes, right?
But there's no really easy way to compile information like what are the open recommendations or the key findings. And so we're pointing our large language model to specifically on GAO published report. But GAO also doesn't have all the answers. There may be a specific question that requires us to combine GAO input as well as maybe additional sort of data that we need to get from the internet.
I'll give you an example. Out of the box, many of these foundation models actually doesn't know what the concept of current date is. Because current date is as of the date that the model has been trained. Could be 2022, could be 2023.
So if you're asking, you know, what is the age of somebody, it's very important for the model to know what is today's date so that they can calculate the correct answer. That may be an instance that we need to make an internet API call to say, help me understand what today's date is so that I can appropriately make that calculation. And so that's an example, perhaps a silly example, where data needs to be sort of combined in a way to generate the right answer. But more importantly, if that is not possible, we don't want these models to close and make some up.
You know, that goes back to the question too of weird, you know, just me, I don't know anything about programming these things, how they code them or any of that stuff. So I mean, I would have never thought that it didn't know anything about what the actual current date really means, right? That would be a simple one in my mind. Right.
It seems like that would be awesome. It can do all these other amazing things, but it doesn't seem to understand what current date is. I mean, that's surprising to me. So again, that's another one of those risks, just assuming that, you know, what these things can do.
I mean, do you feel like anybody looking to really implement these tools needs somebody that really understands how they work? I mean, whatever that may be. I mean, whoever that may be. Or just kind of like experimenting and figuring it out, you know?
I don't know. Yeah. I think understanding how these AI models are supposed to work is important. Right?
We often talk about hallucination as a bad outcome. But hallucination is what the models are supposed to do. As part of this operation, say, if I don't have that information, let me give the best attempt at it. Because if you sort of unpack what a large language model in a very simplistic way, it's only predicting what is the next likely word it's supposed to be.
So a phrase like roses are, you know, red, violets are, in a human sort of context, we know the answer should be blue. And that's, you know, again, in an oversimplified way, that's what a large language model is trying to predict is that we think blue is the most likely word. But based on that prediction of the next likely word, it can do wondrous things. And sometimes we conflate those, you know, ability for chat to be the past bar exam with cognition and reasoning.
Those two are sort of different concepts. So it's important for us to understand the behavior of the underlying model, the sources in which these models are trained from, the kind of parameters, the way that we might be able to tune the behavior of these models. So for example, you can dial down on the level of creativity. You can dial up and down on the level of toxicity.
So depending on the use cases that you may have, you may want to say, don't be creative. Just tell me the bottom line. Right? Or if you're trying to write the first draft of a report, maybe you want some creativity built into the style, the tone, to the way that you might want to draft the narrative.
So yeah, I absolutely agree. I think effective, accountable, responsible implementation of AI begins with digital literate workforce, again, for us to undertake these apparently governmental functions. So when you're looking kind of at the broader landscape, not just federal, but you look at certain, like, leading corporations to draw ideas from, do you partner with other entities? Like, what's the relationship there for you?
Yeah, I think partnership is critical. So for example, you know, GAO could have developed our own custom large language model from scratch, but that would have cost hundreds of millions of dollars to thousands of people. It was just not realistically practical. So partnering with, you know, cloud service provider, for example, even some of these language provider, large language model provider, to understand how do we explore the different feature?
How do we understand analysis alternative? How do we secure these things? How do we, even on operational pieces, how do we not incur unnecessary costs through our exploration process? Because, you know, training these models are often very sort of cost-intensive, resource-intensive exercise.
We want to make sure that we can do it responsibly, but also within the practical means of our agency. So I think partnership is very important. And oftentimes, you know, in a federal agency perspective, we have specific focus around equity and ethics, right? That may not necessarily always be the case.
So I think it's a bilateral dialogue to understand the kind of things that are important to federal agencies. And then, you know, oftentimes in our conversation, we end up pushing the boundaries of some of these commercially available tools that they take back to the engineering team to get a rate on. So I think it's very much, you know, partnership is the right framing of it, the bilateral conversation. So I want to get back to the equity ethics because, you know, obviously it's a pretty big topic with AI.
And, you know, I've seen some of the AI forums and government forums and that's almost always the topic or one of the main topics that are discussed. So I'm just, you know, because I guess I'm more in the financial world, right? And I think ethics, I'm like, I don't know, like what's ethical about an invoice or paying an invoice? I don't know.
So obviously, I mean, this is ERM. We're talking GAO. We're talking much more than just finance. Obviously, I can think of many scenarios where ethics could be a problem, you know, reviewing a resume or profiling a person for whatever reason.
But I'm just curious, have you come across things in the financial world that you think or even audit world where AI might show some bias? You know, I don't know. Have you seen anything? Yeah.
I mean, I've talked to a lot of agencies out there. I'll use maybe privacy as an example, right? AI could be privacy enhancing, but AI could also be privacy encroaching. So for an agency, that deals with a lot of sort of PII, financial information.
Not protecting that privacy, I would argue, is probably not an ethical use of that AI capabilities. Definitely not compliance either, right? Yeah. And I think with AI, the ability to basically combine different data sets to even en masse, that's what is supposedly anonymous data, the risk is there, right?
So making sure that those assessments are done, you know, to the sort of risk management conversation. Understand the exposure and then understand the kind of mitigating control that you can put in place so that you can figure out what that difference looks like, what additional procedure needs to be applied. And so I think clarity of that risk is important. Same thing around disparate impacts.
We know there are a lot of quality challenges when it comes to federal government data. And, you know, AI in particular have a tendency to accelerate those disparate impacts. And I'll use sort of maybe hiring or even loan . adjudication as part of the use cases, you know, many of these historical data because AI by definition learns from historical patterns, but if the historical patterns themselves include discriminations or, you know, different sort of disparate impact already built in, of course the output will end up perpetuating those behaviors.
Or even making it worse. Yeah, if it's just using historical information to make new decisions, that's a problem. So, yeah, I mean, I guess I was thinking too, like maybe fraud detection. Was it just going to, like, pick on certain demographics and things like that to assume that these are more fraudulent potential people versus anybody else?
I can see that. Yeah, absolutely. And we, GL has done work on use of facial recognition in law enforcement, right? If those kind of tools are being used to support, for example, crime prevention, well, is it really based on historical crime data that you end up perpetuating more policing activity in underserved community?
And I don't think that's the outcome that we necessarily are looking for. So there are issues around that where I think understanding risk exposure is important, not just always celebrating the potential benefits of AI. Right. So, yeah, our clock is ticking really fast today, which is awesome.
We're going to have to have a part two one day. So I did have a couple other questions on here. These might be hard to answer, so we'll see how it goes, but... I'll give it a try.
That's right. He's got ChatGPT open. He seems to know everything. That's right.
Between him and ChatGPT, we're good. You know, I kind of wanted to get your opinion. Again, you've talked to many agencies. You've been studying this for a while.
Two-parter here. Are there risks in using AI that you kind of expect that were going to pop up, and yet you found that they really were? Those were justified concerns, and maybe some that, oh, you know what? It really wasn't.
I really haven't seen evidence of this being a problem. Starting with that. Yeah. One area that we're starting to see is in the domain of copyrights.
So within GAO, we certainly publish report, and part of that report publishing process we usually rely on imagery and graphics. We can see the need to maybe use something like generative AI to help us create a customized graphic and try to explain complex topics. But as you might suspect, the provenance of copyright is not always clear. Matter of fact, there are a lot of lawsuits out there that are basically claiming copyright infringement on proprietary data.
Making sure that we can take advantage of the generative AI capability is important. The second, I think, is ultimate impacts to the workforce. When we talk about AI, there's usually sort of Terminator, SkyNet type of, oh, you're here to replace us kind of conversation. I personally don't subscribe to that narrative.
I think AI is another form of technology that we will learn. We will learn to adapt. But will there be impact? Absolutely.
Do we know the extent of those impacts? I'm not sure we have the full pictures. But I think that's part of going back to the risk management conversation is maybe we don't have all the answer, but at least from a governance perspective, maybe there are certain things that we agree that we won't cross. So, for example, at GAO, could we, in theory, come up with a generative AI solution meant to substitute professional judgments of our analysts, in theory, right, that's possible, but we choose not to do so.
So another element of our decision-making process is just because we could doesn't mean we should. Right. So, yeah, and I guess the other one, you know, this is a fun one, more speculative, but, you know, there's that concept of, you know, there's a known risk, there's an unknown risk, or even things that, you know, unknown, unknown, things we wouldn't even think to think about. I mean, have any of these questions popped up in your head when you've been studying this stuff?
Or just, you know, things that you kind of, these are probably risks, but what are things that, I don't know, just speculating that we can't even imagine because we're not there yet or whatever reason. Yeah, I mean, as humans, we tend to like certainties, right, we want all the risks filled out. But from AI perspective, you know, AI is meant to evolve over time, for good or for ill. And part of it is expecting behaviors that maybe we didn't specifically account for.
And so building that flexibility to say, do we have processing in place to monitor the health and the behavior of these models, or are we treating AI deployment like a software deployment, where once it's deployed, out of sight, out of mind, we're done, we move on to the next project. So I think those kind of processes need to be baked in and integrated across organizations. And setting up broad guardrails. That's right, yeah.
So it's not just a data science kind of concern, it's also, you know, your CIO concern, is your chief risk officer concern, is your legal concern. You know, in the GAO's AI accountability framework, we specifically talk about treating AI implementation as a team source. So that helps to sort of bring different perspective, but also helps to diffuse the risk mitigation. So it's not just all data scientists are not responsible, not only to create a sort of performing code, but also act as an ethicist and equity specialist, which, you know, there's that inherent conflict because I think most data scientists are not trained with those ethical considerations in mind, at least not at the technical level.
Yeah, something I think about or worry about, I mean, I do feel that government should, I mean, we need to take advantage of the risks. I mean, the benefits, but technical, take advantage of the risks. No, I mean, ERN's about risks and benefits, right? Like, you know, there's got to be so much we can benefit from using this technology.
Are we just, are we going to be hampered by not understanding it, being kind of scared about it or trying to overanalyze it or over, I don't know, guard rail it? It's easy to far down the road to get the toothpaste back. Well, then that's the other side of it, right. So, I don't know, cost-benefit analysis in your mind.
What are your thoughts on that? Yeah, I think this is one of the reasons why chief AI officers are being appointed to address that kind of consideration for, you know, specific agencies. And I think you're absolutely right. You know, not just within the federal government, but also our state and local partners and our international sort of partners as well.
There's usually a sense of urgency in terms of how do we imagine potential use cases of AI, but making sure those performance can coexist with the kind of accountability desires that we have as a democratic nation. And so, you know, we spend a lot of time talking at various levels of sort of public sector entity to make sure that we can continue to exchange lessons learned, success stories, but also making sure that we are, to Dan's point, having those conversations with industry partners to say, you know, the federal government also would like to take advantage of AI, but we have a very specific consideration around responsible and accountable performance of that and making sure that those services and products are baked in to support the governmental sort of function that we need to undertake. Yeah, I think it's crazy. This stuff changes so fast.
I mean, you're trying to ATO version 1. They're already on version 10. Now there's a thousand more things that are much better or whatever. It's just, you know, what can you do?
But part of it is also understanding fit for purpose, right? So we talk about a large language model and some of these models have 70, 80 billion parameters. Think about Excel spreadsheet with 70 billion columns. But we're starting to see existence of medium language model and small language model, like fit for purpose application of these tools.
Because in some use cases, you may not need a large language model. It's cost-intensive. It's difficult to manage. You know, the risks are different in order of magnitude.
And so, you know, I think understanding the problem to begin with and then what are the right tools and capabilities that we can apply to solve that problem is, I think, an important consideration. Well, that's a good example. I mean, yeah, we've been using Excel for how long now? It's been around for decades, but we're still using it.
Just, yeah, we'll be using ChatGPT version 3 for the next 10 years or 20. Who knows, right? Yeah, and I think that underscores the need for more digital literacy training. Because if you think about Excel, it's very reproducible.
Excel spreadsheet, they don't change. It's zeros and ones and truth and false. But AI is meant to give you a probabilistic output. And the answer you get on Tuesday is maybe different than the answer you get on Wednesday.
And so how do you deal with that evolving dynamic of the system behavior? I think fundamentally we need to shift away from this sort of Excel mentality. And so this is why I think some of these governance structure makes sense from adjacent areas. But we need to start thinking about AI-specific risk and how do we address them with the necessary speed and flexibility.
Well, I was going to say, I think, at least for the financial world, the reason we use Excel so much is because maybe the current system can't answer our question. You know, like, oh, what's that anomalous transaction? I'm going to pull a report and try to, you know, weave through it and figure it out. Well, maybe, I mean, hopefully the AI can do that for us.
Or, you know, all the things these Excels are doing, you have better tools that can actually perform that for you. Yeah, better and simpler. And more accurate and simpler, whatever it may be. But, you know, those conversations are starting.
So, for example, could AI be used in a way that more appropriately calculates risk reserve or estimate risk that maybe as a human being we don't have all the visibilities into? But at the end of the day, you know, investors and government entities rely on financial statements. So there's always that inherent conflict around stability, certainty versus evolution, and maybe a little bit of a black box approach to things. But ultimately, I think AI is here to stay, right?
It's so incumbent upon us to how do we adapt our perspective, you know, even in the financials and management domain, to understand how AI could be beneficial in an appropriate way or how AI could be detrimental in a way that maybe we're not intending to do. Right. All right. Well, we're pretty much there.
Is there anything you want to mention about GAO use cases or do you want to wrap it up here? Yeah, I mean, I'll just highlight, you know, we often talk about accountable AI and responsible implementations of AI, but what does that really mean? So GAO, back in 2021, we issued our AI accountability framework, and it's really the blueprint for our auditors to follow as we evaluate implementations of AI. We're looking for evidence of those deliberations.
We're looking for evidence of those risk management processes. We're not trying to be prescriptive to say AI should be used in certain ways, but we do believe trust and verify is an important element of that. I talked about it and it didn't happen. That's right.
All right. Well, Taka, Dan, thanks again. This was a fun podcast. I appreciate you making your way out here today.
Absolutely. Thanks for having me, Taka. All right. And that was Risk Chats.
Signing off for Dan and Taka, myself, Paul Marshall.