Skip to content
Exploring AI Agents S3E17 episode artwork

May 9, 2024 32 min

Exploring AI Agents S3E17

from Invisible Machines podcast by UX Magazine Hosted by Invisible Machines

As the second wave of AI-led disruption takes shape, we're seeing and hearing a lot more about AI agents. Analogous to the Intelligent Digital Workers (IDWs) described in Age of Invisible Machines, AI Agents are more complex that they appear and many organizations are discovering that they aren't structured to allow these agents to perform meaningful work. Robb Wilson (CEO and co-founder of OneReach.ai) and Josh Tyson explore the real nature of AI agents, what kinds of technology ecosystems they require, and how organizations can give them real agency. This practical episode also imagines what the proliferation of AI agents means for the future of software. The bestselling book that explores AI agents, or intelligent digital workers (IDWs) at far greater length is Age of Invisible Machines: https://onereach.ai/invisiblemachines/ Learn more about how to make AI agents, and play with a playground of AI skills: https://onereach.ai/ #ai #aiagents #hyperautomation #conversationalai #podcast #aipodcast #ux #uxdesign #productmanagement #aiproductmanager #aiproducts #aiproductivity #invisiblemachines #googleai #llm #generativeai

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Exploring AI Agents S3E17

0:00 32:27
of matches

Transcript Auto-generated, may contain errors

Greetings. I'm Josh Tyson, co-author of Age of Invisible Machines, the first best-selling book about conversational AI. Age of Invisible Machines explores the learnings of industry veteran and OneReach.ai CEO Rob Wilson. Each week, Rob and I meet up to continue the conversation we started in the pages of our book. This week on the Invisible Machines podcast, we're going to talk all about AI agents. What are they? What kinds of technology ecosystems are needed for AI agents to become real force multipliers? And what will the proliferation of AI agents mean for the future of software? In Age of Invisible Machines...

Rob and I refer to AI agents as intelligent digital workers or IDWs. When organizations create the right infrastructure, IDWs can revolutionize the way teams use their time, sharing information across standardized communication channels and automating routine tasks so that they can work on higher level problem solving. IDWs are the manifestation of a diligent strategy and a shitload of work. And on this episode, Rob and I will demystify them by answering questions drawn from conversations with business and technology leaders from a variety of global enterprises. This practical conversation.

will give you a new perspective on what it takes for AI agents to start doing real work. Invisible Machines is produced in partnership with OneReach.ai. Their Generative Studio X platform is the only orchestration platform that's been named a leader by every major analyst group. Enterprises and other organizations are already using GSX to create technology ecosystems where they can grow their own digital teammates called intelligent digital workers. These IDWs can be set to work with hundreds of customizable skills utilizing GSX's no-code building tools. Head to OneReach.ai to test drive an IDW and experience the next phase of Generative AI.

All right. Well, so, you know, back in April at the Google Cloud Next 24 events, their opening keynote featured a lot of demonstrations of AI agents, as they're calling them. Yeah, I was excited to see that the dialogue is starting. Yeah, right. We've been building these things for years. In our book, we call them intelligent digital workers. Right. They're very similar to what AI agents represent, but I would say, you know, in some ways they take it a step further.

We actually have some questions on hand here that have come from partners, colleagues, the community, customers. And we thought it might be fun to kind of run through some of these questions, especially in this moment, because people have a lot of questions. Maybe chief among them, what is an AI agent? One of the most difficult challenges, I think, getting people to wrap their head around it is that it is such a new way of thinking. It's a complete paradigm shift. And if you're trying to map it as a puzzle piece into an existing paradigm, which is our software paradigm today.

you move to a world in which software is being written on the fly, which means it's being written, tested, deployed instantaneously. It is a completely different way of thinking. And agents enable this concept of just-in-time software because what you're essentially doing or able to do with a machine now is instead of giving it very specific instructions, which to any programmer, you know, the first thing you learn when you take an intro to programming class is how to break down tasks. for the machine into really small pieces. In order to provide a machine with instructions, explicit instructions on how to perform a task, you have to give it very granular pieces and you have to break down every if and then statement and run scenarios in your head as a developer and then build out instructions on how you want the machine to handle each scenario. And the interesting thing about that is you don't actually necessarily almost ever give the machine an objective. It doesn't, it never knows the objective. When we think about

building and training machines to do tasks through agents, we're actually talking about now just giving it an objective and letting it work out all of that instruction set. And when you look at the amount of time and effort that goes into creating that instruction set, it's massive. It's the majority of the cost of building software is in the fact that we take an instruction approach to it, instructing it piece by piece and testing all those instructions and scenarios out. So this is, it's just fundamental when you think that all of this. It could be where a machine could be instructed by an objective. Now, the anatomy of an agent is usually three pieces if you want to deploy it successfully. And so one is, yes, an objective. Two is constraints. You want to make sure that going out the window is not an option. And then the third evolution of this will become less and less necessary is some form of instruction as well. If we want them to do things by our instruction, in other words, we want it to do it our way, not its way.

we're going to have to provide it with our way. And so you'll see this component of, hey, I want the machine to manage PTO requests from my employees, but here is our organizational rules on how much PTO and who can request PTO and what the approval process is for PTO and not allow the machine to just decide what the policy should be on the fly. How do we get from where we are, where all of our software is based on very, very brittle instruction sets? given to machines to this new world. It's not like overnight, we're just going to scrap all of our old software and every agents are going to run everything. We need a roadmap, a bridge to go from where we are to agents. And I think that's the part that is confusing people because they're based on where we are and they're thinking, okay, here's how passwords are reset today. How will this happen with agents considering the fact that all my passwords are stored in a database somewhere? And you think, well,

What we have is this interim moment where agents will operate our old software and orchestrate our old software until our old software is no longer necessary. Yeah, it just kind of gets dissolved. So we get a lot of questions about, you know, when should I use agents? How hard are they to use? I think, you know, even in the Google demonstration, you can see that it's kind of moving towards what you're describing, that objective-based software, where, you know, the objective is to look at this photo of this person in a band and find me the same shirt they're wearing so that I could buy it. But what you're describing is something a little more profound, right? Because it's not just like these single objectives. What about like the question of where you run them, which is sort of a strange, broad question. I feel like it touches on some of the things we talk about.

in the book, specifically in the second edition that we're working on. Just the idea of, like, there's bolt-on solutions out in the marketplace that have emerged. There's sort of these point solutions, which, you know, I think the Google AI agents feel more like they're in that vein. And then there's this need for something a little more sophisticated, a platform, or we've also called it an extensible cognitive architecture, where instead of having, you know, different agents that can do a few different things, you have agents that can access a shared library of skills, that can access, you know, data that's been specifically set up so that it can be searched. semantically and meaning and intent can be discovered. There's all these different things that need to be happening inside of an ecosystem for an agent to actually have agency. So where do you look when you want to start running something like that? Yeah. I think this is a pretty big question because when you go to play with agents, just creating an agent is relatively simple, but trying to put it into an ecosystem where there's communication between agents and where there's monitoring and security and all of the other considerations that you have to have when running these things at scale.

and in production. So I think it's pretty well established that a microservices architecture is fairly ideal because what you have is that composability where they can spawn and scale and reproduce. And the just-in-time sort of compiling nature of it allows you to generate code and run it on the fly. But there's a whole apparatus that you need outside of just this. microservices architecture, which is event-based systems and threading and state and memory. So there's a lot of components that you need to contextualize around these services that are getting either created dynamically or statically.

to help manage and orchestrate them. And to really understand the architecture you need, you need to understand some of the patterns that agents can run in. I think a lot of people, and what I just explained was a single agent pattern. You have an agent that reviews your inbox, and those are relatively easy from an architecture standpoint because they're sort of a standalone app. But when you get into the multi-agent approach, which is really where things get exciting, then you start talking about apps, not just performing different skills or functions and then communicating with each other, but you have apps.

or agents that are performing the same function together, but from different perspectives. In other words, one agent might suggest a course of action for resetting your password, but then you may have another agent that reviews that course of action and decides if it's a viable course of action, or guesses whether that course of action would make the end user happy or unhappy based on the possible outcomes. Or another agent might just evaluate the risks. of doing it against the original agent that's proposing it. And so you have different patterns of how multiple agent threads could work together, either antagonistically, cooperatively is another way where they're working together to help. And then iteratively, where it's just going, is there a better way or a faster way to perform this than the way you just suggested? Well, maybe if you did it this way.

So you have these different patterns and ways that you can swarm agents. And once you understand these patterns, you realize you need some sort of event-based and thread-based and state system above all of this that can keep track of a common memory and a common state across all of these agents and also facilitate the communication between agents. And so this microservice architecture with an event-based system and memory and state, all of those things become critical to creating a multi-agent architecture and environment. We get asked a lot, how hard are AI agents to use? That feels like a bit of a misleading question because a demo like Google showed, I think makes it seem a little simpler than it actually is. It depends on the inherent complexity of the challenge, right? I think there's three ways to look at how hard they are to implement. One is the inherent complexity of the problem itself. So if you're talking about an inherently complicated scenario, that's going to be harder. The next thing is how much you

you want the LLM to follow the path that you set for it, how much you want it to follow your instructions and do things the way you want it to operate versus the way it can determine using its own general intelligence, let's call it. That adds a factor of complexity to the whole thing. Then I think another piece is, I think this is a piece that doesn't get talked about enough, is templating. So this idea that you have some form of symbolic code or templates that the LLM is sort of filling in. if you're beginning with a template, like if blank, then blank, and you let the LLM fill in the blanks. So if you have a template for password reset that has sort of a structure to it, like call it a scaffolding, and then your job is to just provide context, but it keeps sort of like providing train tracks to an LLM. If there's train tracks already laid, and your job is to just provide sort of context around that so that it can operate in your specific environment, then that can be really, really simple. So I think we'll see.

lots and lots of these templates being an approach, which is like a hybrid approach to agents on rails, we could call it, right? It's probably where we're gonna see a lot of this stuff move fast, but if you're building it from scratch and it's complicated, inherently, those are gonna be much more challenging, especially from a testing standpoint. Testing is the key. Building, it becomes easy, but testing becomes a whole nother story. So what's that relationship look like? Because there's another question we get a lot, like, how do you test these things? Like in our book, we talk a lot about being more agile and agile, like that it's really important to start propping things up and knocking them over and how quick that testing phase can be. And also how it's kind of an ongoing thing. It's not like a finite period of development. So what does that look like? What does the testing entail for an AI agent? Yeah, I think that thing's the most challenging part. A lot of times what you save in development time can cost you exponentially in testing. And this is one of the things you really need to consider because, wow, it's so fast.

That scenario I just gave of managing an inbox based on critical issues, now you have to test that thing to make sure it works. And it's less predictable on what the outputs are. If you've created a bunch of keywords to detect whether or not a severe outage has happened, it's much easier to predict and test that because you now know it either contains those keywords or not. You can predict what will happen based on certain emails. When you're running an LM, you don't know. And so now, you know, testing becomes this job of creating a corpus of fake or example emails from the past, running them all through, and then taking a look to see if they got it right and making iterations. And instead of just being able to, you know, fix one sample that didn't work and know that the rest will continue to work like in a symbolic way, you have to regression test all of them because that one change that you made.

to capture that one email may have affected its ability to capture all the rest. So the regression testing becomes absolutely key. Where I think we still win, though, is that once you start utilizing these agents to test themselves, we'll start to mitigate that challenge. But a lot of people dive into this, especially if you have a scenario where you're the developer and someone else has to test. We'll see the scenario where you dive into it, try something very quickly, hand it off, and then exponential time on testing.

Yeah. And LLMs have sort of introduced a new wrinkle in terms of safety and security. And the fact that we anthropomorphize them so heavily, we are quick to believe that they can just do anything. And sort of the open-ended nature of a chat GPT window speaks to that too. You can ask it to do anything. You could ask it any question. And even if it gives you the wrong answer, you might believe it because it sounds very convincing. So it almost feels like that is somehow baked into the question of where should someone start with AI agents? Because it feels like an important thing to know, even going into a project involving LLMs in particular, is what they aren't capable of, like what their limitations are. Yeah. There's two paths.

First, it's to just build one so that you're familiar, kind of make yourself familiar with the tool because it's just a tool in toolbox. It's not a solution unto itself. It's a technique for using LLMs. It's not a technology unto itself. And so getting familiar with the technique, getting comfortable with it so that you get a feel for it. And so that is just about practicing a little bit. And that's organizationally and that's personally and on a team basis. The second thing is to use it kind of in a sense when symbolic is no longer viable.

whether it's time, budget, or both, and kind of saying, okay, we tried to do this quickly just using simple code, and it's not feasible. And so now agents can make things feasible. I still think agents should only be used when they're absolutely needed because there's a cost, both performance-wise, resource-wise, even environment-wise, electricity and things like that, that just... Say, you know, if symbolic works, if simple works, use simple. Keep it simple, right? When you can. So I think symbolic is always going to be the simple way until it's not. And as soon as trying to do it using symbolic code becomes complicated, then you start to look to agents to solve the problem. It may even be an iteration. A lot of people...

don't realize like you can start simply without an agent and then iterate into it. But one of the things that agents really benefit from is real world interactions for testing. So there's an interesting concept of sort of starting a skill more symbolically where your real goal is not to fully automate tasks, but more to just collect data and input from end users so that you can then build agents and run simulations. to test the agent before you deploy it. So it's interesting as like an evolution of a skill to say, start simplistically taking your skill from basic. We talk about this in the book, going from basic to wise and saying like most basic skills can be done without agents. And then as you evolve that skill to becoming more wise, to do more capable things, you start to evolve its architecture going from symbolic to hybrid to maybe almost fully agent based. I think those who are, who are thinking about this from a

top-down versus bottom-up approach, create an architecture to plug them into first, are the ones who are going to really benefit the fastest from agents. Yeah, and so then the trick is communicating the value of that architecture to the stakeholders, which is an abstraction in a sense. Yeah, I think to the people more connected, more practitioners that understand how agents work, and that becomes easier, it's as they try to communicate up the chain. to get permission to make these investments, they have a hard time communicating the value of having an infrastructure, investing into infrastructure, just like a city has to invest in infrastructure before that turns into tax revenue. It's the same idea, convincing senior management that infrastructure is critical and important. So I think infrastructure is the magic word here.

start building an infrastructure for agents to work within um and what would that infrastructure entail and look like uh and you know it'd be great if you didn't need to worry about that and you could plug it in later but the problem is it's really a you know gut and replace everything if you don't consider an infrastructure from the beginning yeah because you're uh like the easiest way to waste a whole lot of money is to try and cram these technologies into that old software development paradigm that you're describing at the beginning yeah and i think that's the paradigm we're in so most companies think well that's how we work we have a bunch of apps Lots and lots of apps. We have a problem with those apps talking to each other. We accept that. That's just the way it is. We even evaluate most of our new software we buy based on its ability to talk to other apps. But we're rarely satisfied with the degree in which that happens. And so I guess we'll buy our agents and architect our agents in the same framework as our apps, right? The problem won't be that you have a bunch of apps and agents running around that don't talk to each other.

That's not the real threat as much as that's going to be chaotic as it is today. The real threat is that your competitors or other companies aren't. They're going to have an ecosystem. Someone's going to figure out that these things talking to each other is more important than the speed in which you can implement a single agent. And that short-term win of automating password reset today without investing in an architecture to plug it into and making sure that it's going to be able to talk to other agents, even though they get the short-term win, the long-term.

you know, we'll be a losing proposition. So if everybody introduces these things in the same paradigm as we introduce apps today and don't consider how they might talk to each other, I guess there's no harm. But if companies, and I believe there will be smarter companies that get it, that's where they'll just, there'll be a force multiplier effect that other companies won't be able to keep up with. Once an organization within a vertical hits that, yeah, hits that ecosystem mile marker, anyone who hasn't at least taken a few steps in that direction is going to be hopelessly behind. Yeah.

Yeah, it's an interesting paradigm because the leaders in different categories, you find they're either willing to take risks because they have capital available to try new things. So they do. Or you see the opposite, which is they're so confident in the way they're doing things that they're not willing to entertain doing things a different way. It's sort of like the way we're doing things works, let's not mess with it. And then you have the incumbents that are trying to break through and trying to leapfrog. And so the question is, but they have less capital to take risks.

So the question is, in each category, which one prevails? Is it the incumbent that's more motivated and sort of jumps in and takes the risk, or is it going to be the leaders? And I think that will play out possibly different within each vertical and each industry. I think the most impactful thing is that you can talk about leapfrog moments, but it's a lot easier to say than do. Because of the nature of this approach to software, eventually replacing all of the software we've built, all the software that exists in companies, all the interfaces.

that we use and the backend software and how it works, it truly is a leapfrog moment. It really is an opportunity similar to the internet where an offline company that's maybe trailing decides that they're going to go all in online, like an Amazon, right? And truly leapfrog the Walmarts and others of the world because... they just go all in on this new paradigm and this new paradigm is so dramatically different that it really offers that opportunity to leapfrog versus just words. Yeah, it feels like the companies that get this right and have this ecosystem in place will be almost unrecognizable next to the organizations that don't in the way that like, you know, like Netflix and Blockbuster feel a little more the same than the kind of divide we're talking about here. And I think people are just, they have a hard time recognizing that the ground is shifting on the consumer side, on the end user side, right? Because...

Just the other day, I was in Google, and I was trying to determine if a restaurant was open, and it offered me the option to call the restaurant on my behalf and check to see if they were open. I pressed the button, sure. Behind the scenes, it made a couple attempts to call. It didn't get an answer, but you could imagine that had a human answered, likely a human would have answered, and said it would have asked for hours, it would have... I got an answer from the human using some form of LLM approach and then returned back to me and said, you know, either they are or they aren't open. And it's not hard to see that this is not the fax machine scenario, right? Where you had to have another fax machine on the other side to answer a human answer. And so it became practical. But then at what point is that restaurant getting so many calls from bots asking hours that it wants a bot to answer?

the phone. And so you just see this just becoming exponentially adopted. Even if businesses are slow to pick up, will end users be slow to pick up? I don't think so. And so I think there's a concept where the businesses are watching each other based on the adoption, but forgetting that, hey, there's a new consumer in town and that new consumer and user is an agent.

All right. Well, thanks again for joining this ongoing conversation about conversational AI. Be sure to subscribe to Invisible Machines wherever you get your podcasts. And if you feel so inclined, please leave us a review on the Apple Podcast app. You can also watch these conversations on the Invisible Machines YouTube channel. This podcast is produced for UX Magazine in partnership with OneReach.ai. Over the past five years, our team of nearly 200 engineers, scientists, experience designers, anthropologists, and linguists have been developing Generative Studio X, an award-winning platform that has the lone distinction of being named a leader by every major analyst group.

GSX is a complete environment for hosting, creating, analyzing, and scaling your own digital teammates called Intelligent Digital Workers. For an interactive demo of IDWs in action and to learn more about the GSX platform, head to OneReach.ai. This podcast would not be possible without the hard work and dedication of executive producer Elias Parker and producer Kate Timchenko. Our video and audio editor is Michael Lipinov, and we rely on support from the marketing team at OneReach.ai, including Allison Harshberger, Anastasia Nechitalio, and Vera Pekotko. Thanks again, and we look forward to connecting with you next week right here on Invisible Machines.

Frequently asked questions

How long is this episode of Invisible Machines podcast by UX Magazine?

This episode is 32 minutes long.

When was this Invisible Machines podcast by UX Magazine episode published?

This episode was published on May 9, 2024.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Invisible Machines podcast by UX Magazine episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.