LLM powered systems continue to move steadily into production, but this process is presenting teams with challenges that traditional software practices don't commonly encounter. Models and agents are non-deterministic systems, which makes it difficult to test changes, reason about failures, and confidently ship updates. This has created the need for new evaluation tooling designed specifically around the properties of LLMs. Comet is a platform with roots and MLOps that has evolved to support teams building modern LLM powered applications. The company recently launched OPIC, which is an open source platform focused on evaluation, optimization, and observability for LLM agents.
Together, the tools aim to bring the rigor of traditional engineering and ML workflows to the rapidly evolving world of agent-based systems by treating prompts, tools, and workflows as optimizable components that can be evaluated and improved over time. Gideon Mendels is the co-founder and CEO of Comet. He previously worked at Google on hate speech and deception detection, and he founded Group Wise, which trained and deployed MLP models processing billions of chats. In this episode, Gideon joins Kevin Vol to discuss how agent development sits between software engineering and ML. YE-val is the missing foundation for most AI teams, prompt optimization as a search problem, and the future for continuously improving agents in production.
Kevin Vol, or K-ball, is the vice president of engineering at MENTO and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI and action discussion group through Latinspace. Check out the show notes to follow K-ball on Twitter or LinkedIn or visit his website, kball.llc. Gideon, welcome to the show. Yeah, Kevin, thanks for having me.
I'm a big fan of the podcast, so I was looking forward for this one. Yeah, I'm excited. I'm with you. So can you give a little bit about your background, how you ended up at Comet, and then some of what Comet is about? Absolutely.
So I originally started as an software engineer, kind of moved throughout the staff in the first kind of few years. And then about 10, 12 years ago, I shifted to working on machine learning. I was a grad student, and then I went to Google. Funny enough, I worked on language models. This is 2016, so they weren't large, no, we're very good.
These are pretty transformer days. Unfortunately, LCMs, if anyone still knows what that is. And someone coming from a software engineer background where we take a lot of pride of how we build software. Obviously, a lot of that's changing right now. I'm sure we'll talk about it.
A lot of pride of how we build software, the tools that we use, and then joining an ML team with amazing, very, very smart and talented people. But just seeing how the whole thing is kind of like a little bit like the Wild West. It was very, very challenging. We worked on hate speech detection and YouTube comments. If you remember the YouTube comments section back in the day, I think someone called it, the worst place on the internet.
So we had a hard time getting these models to work. And from that point, I was like, look, we had data, we had compute, we had smart people, we still couldn't do it. What is it? And it's not considered necessarily a hard ML problem. And I realized it's just kind of like I run the process of how you drive these projects.
And I called my co-founder here a comment, who worked on another startup building ML models. I was like, hey, you remember making fun of my ML workloads and how everything is stitched together? So I'm at Google and it's exactly the same thing, just a massive scale. So that's really how we got started. You know, this is 2017-18.
And we started with, you know, specifically what my team and myself needed back then at Google, which was around model experiment tracking. You know, you're trying a bunch of these models. There's all these moving pieces, hyper parameters, data set versioning, all these results. And it's really hard to know that you're making progress to understand what you're doing next. Collaborating is completely out of the question because no one has access to anything.
So we started with that. And then over the years, we kind of expanded that side of the platform. So, you know, data set versioning, model registries, model monitoring and such. And then about two years ago or so, obviously, like, it was quite a big shift in the industry. And we started seeing, you know, a lot of our customers and users started telling us, hey, you know, for this use case, we're not going to train a model anymore.
We're going to try to build it on top of, you know, OpenAI API. But it's still like very similar because we're testing all these different stuff and we still want to use comment for that. How can you help us? And at first we built, you know, we started ads on features and such to help with that. But eventually we realized, okay, there's a lot of similarities, but there's also enough differences not to try to bake it into like a slightly different workflow.
In September 2024, we launched Opik, OPIK, which is our open source product focus on team building agents, any type of LLM powered applications, really focusing on the end to end from like early dev through kind of this deployment process to production and specifically, you know, things around visibility and evaluation and optimization of these agents automatically. But yeah, it's been a fun ride. You know, we power some amazing AI teams and Uber, Netflix, Etsy, Shopify, Autodesk. We have roughly 150,000 engineers over the world using our products, great adoption of the open source friends. So yeah, it's been a fun ride so far.
Well, let's dig into a few pieces of that. I definitely remember like MLOps has been a term for a while. There's just all this operational stuff that is hard and different. And as you highlight like now in the LLM world, it's everybody's getting to play with this. In fact, everybody's having to be in this space and start to deal with non-determinism and how do you deal with data flows and all of this different stuff.
But it's also a little bit different. So let's maybe, can we dial in what are the particular characteristics of development with agents that were different enough that you said, okay, this is a whole new product. This is a thing we need to build. Yeah. Yeah, that's a great point.
I mean, when you're training a model, first of all, like you typically have some kind of a training data set and the algorithms are mostly a commodity, right? Like, you know, everyone's mostly using the same algorithm. So you do spend some time maybe like changing some stuff, but like it's mostly out of the box. And the majority of the time you spend on figuring out what is the right data set or what variation, you know, you want to augment it in different ways, the model hyper parameters, how to formulate the business problem as a machine learning problem, which is not trivial in many cases. And then you know, you get these like massive binaries on the other side of like the model weights and how do you do retraining with so at that level, right, like the majority of our users and customers consume the LLM as an API.
It could be like that they're deploying it and it's open source, but majority of them use the commercial ones. So you're no longer in control over the weights whatsoever. You have like three production hyper, like, you know, temperature stuff like that that you can play with, but it's mostly static, right? Like sure, you know, opening, I will release a new version, you know, every quarter or so, but it's mostly static. And what you do control is a builder, whether it's a simple LLM workflow or a full on agent are slightly different things, right?
You control the system prompt, which is, you know, the equivalent of the weights, you know, if you train the model, you control tool calls, context, you know, vector DBs, all these, but the similarity is like the reality is like you have all these variables, you system problem configuration of your chunking, you know, the tool call descriptions, you got all these variables and you're trying to find a combination that gives you the best results. So from that perspective, it's quite similar in the day to day, it tends to look quite different. So we had a lot of experience in this workflow, but you know, SDKs that UI, all those things tend to look quite different. So that was kind of the main motivation to like separate the two. That makes sense.
So looking at now that development problem, right, you have these different pieces that you can coordinate and definitely one of the things that I see teams struggling to come up to speed with or a lot of engineers at least is just grappling with non determinism. If you're coming from machine learning background, that may be old hat, right? Machine learning has always been statistical and dealing. But if you're coming from a traditional software background, which now more and more people are and still having to deal with this stuff, non determinism is weird. That's scary.
We used to try to get rid of all that. So like what changes in the process need to happen? You got kind of multiple levels of non determinism. Maybe some of them are not officially non determinism, right? Obviously you got the LLM technically or supposed to be able to get deterministic output if you set the temperature to zero technically.
But you know, it's actually interesting right up on why they're not deterministic. And it's just, it's actually because of how they're deployed with a mix of experts. Because at the end of the day, it's a combination of matrix multiplications that should be deterministic if they input is the same. So that's, but that's digressing. So you got that.
I think, you know, to your point, like what is like a slightly different concept to software engineers and much more familiar with for data scientists or people training models is software, we write these unit tests. And at some point, we have like a pretty good coverage. Of course, something could have a production in another edge case will add a unit test. But like you have this level of confidence that this new version that you build is not going to break everything right before you deploy. And with these agents, how do you translate that concept?
Right? Because like you can't, first of all, in the small details, you can't do string matching and stuff because obviously you can have the exact, or semantically the same output, but from a string content completely different, right? So you start off to think, how do I compare or do assertions between two of these outputs? Right? So that's a hard problem.
But also like this concept of generalization, right, which is very common in ML is like, how do I write a test suite that gives me enough coverage and variance so I have confidence that this thing is working, right? So I think that's a concept that's relatively new for a lot of people who haven't trained models before. And then that's the reality is like this building these agents is somewhere in between software engineering and ML. It's definitely not pure ML. It's definitely not pure software engineering.
But there's like a lot of learnings from both of these paradigms that you could bring together and actually shift stuff that works. Like that's the reality. It's hard to get the stuff right. But when you get it right, it's magical. I think we're all experiencing that on a day to day basis.
Yeah. So let's talk about some of those primitives. And one in particular that comes up a lot and it's talked about a lot, but then also when I talk to teams, everybody's like, yeah, we need that, but we're not really sure what it means is e-vals. So how do you think about e-vals? E-vals, if you put in the work, they are extremely powerful, useful, and kind of give you that level of certainty.
Now I completely agree with you that most teams don't do it, right? And I have a very strong thesis on why that's the case. But e-vals is really kind of mapping this concept of like a test suite to your application, right? So, okay, I have the most basic form, a list of inputs. I have a list of the expected output.
Now it's not a simple assertion. You typically want to have some way to measure the distance between the agents answer and expect it out, but you could do it with deterministic metrics, like blue score, stuff like that, or you could do it with Ellen's a judge. But at the end, you get a score. Hey, this past has failed. This is your overall score across the board.
So I think they're extremely powerful. From a customer base that wanted to put in the work and invested in adding those, they're definitely like on the more successful ones and getting these agents to work well in production. And I think the reason not that many people do them is because it's extremely hard to construct these data sets. It's painfully hard for various reasons. First of all, often the person that's building the application is not an expert on what's the right answer, right?
If I'm a software engineer and I'm building a tool for the sales team or the HR team, I do not know what is the right PTO policy in Norway for an employee that's been in the company for two and a half years, right? So you kind of have a domain knowledge gap, which is hard. In addition, like, you know, if you're doing this kind of end-to-end testing, completely monastic, what happens in the process, which is one way of doing it, that's maybe fine. If you're trying to test something that does take you in a kind of which tool it should have called constructing these like graph traversals and compare, it's really hard work. It's extremely challenging.
And look, there's a lot of ways we and others like help with that. Like we have UIs for subject matter experts to clearly like annotate stuff and add things. There's attempts, which I'm not a big believer in to solve this with synthetic data. But I think that the solution is a product solution or like a product approach, right? So the reality is like whether you build e-vuls or not, you put the stuff in production and some point someone comes complaining, right?
Hey, this user tried to do it again, it's completely wrong answer or this blew up and so on. And so when you're not owning this, whether you're an engineer or a PM, you go ahead and you try to fix it, right? Because that's what we do. And what we're spending a lot of time on is like, how can I take this activity that you're going to do anyway and use that to bootstrap the e-vul data set? So I'll give you a specific example.
Let's say we take that example with an HR chatbot, that user asks about PTO policy in Norway, there's some context injected with like their tenure and all those kind of things. HR comes and say, hey, that gives, that's the wrong answer. So you would go in there and you would say, hey, if the employee, like you write in free to exit employee is located in this country and it has this kind of tenure, like you should look at this document or the answer should be this. So people will do it anyway and might go and change the system prompt. But like what we're trying to do is like actually build this like test suite based on that.
So now you have a new test sample with this input in this output with an assertion that says the answer should be X or it should be greater than certain years. And then the next time you try to iterate on your agent, whatever you're changing dial in the system prompt, the tool calls, you're going to run through this e-vul suite automatically. So it's a product solution, right? It's not like a algorithmic solution to this problem. Turn speaking to users and customers, I think that's the right solution.
And we're spending a lot of time really nailing that workflow, you know whether it's UI first or it's like terminal first and there's all these questions because I do believe e-vul, they're very powerful. One of the things that I find myself wondering here is and you said something about like, oh, are you doing this end to end or are you doing sub pieces and diving in? And what you described in terms of a product solution potentially supports either. But like how do you think about the different levels at which an e-val makes sense? Coming back to the software world, are there relative equivalents to unit tests versus integration tests versus system tests or what have you?
Yeah, absolutely. What I typically recommend customers and teams to start with like the end to end the system tests, right? Like you're not able to test every single scenario that way because, you know, the side effects and all those kind of things. But that's typically the easiest data set to compile and it does gives you tons of value. Another thing we're seeing a lot of people do is, you know, when you start adding these tools, a very, very common failure mode is it's not calling the right tool.
Even though if your tool is perfect, if it's not calling the right one, like it's not going to work. So we see people build data sets that are essentially it's like a classification problem. Like given a certain context, you know, user context, previous message history, anything that came from previous tool calls, like basically the entire graphics execution up to this point, did it call the right tool. That's like relatively easy. It's not the right product to help you, you know, you don't have to manually create the graph context.
That's relatively easy to generate. And it also provides like relatively good results. And then of course, you know, a certain tool could have an L1 call in it or it could be a complete sub-agent. Often you want to test that in isolation as well. So I think it does kind of correlate to the software engineering testing concepts that we're familiar with at a high level, at least.
So far we've kind of talked about it using it. I think your example was, Oh, this edge case came in. I got a complaint or whatever. My expert looks at this and says you should do this. And that to me, that kind of maps, I'm going to keep mapping to software engineering.
That's like a regression test, right? We had a failure. So we're introducing a test to make sure that we don't and then we go and introduce a fix. What other types of use do you see? Is there an equivalent to, for example, test driven development?
Is there an equivalent to other types of sort of process level utilization of tests for these emails? The approach I talked about introducing these, like regression tests is mostly a way to try to bootstrap that data set in the most user friendly without requiring you to do a lot of work. But when you put together like a true evaluation data set where the subject matter expert, not necessarily based on production regressions, then at that point, like you can do something very similar to test driven development, right? Like you get one shot to the system prompt, just write two sentences, put something together really quickly, which is easy today, right? Test it on a data set.
And then, okay, it's failing these samples and these samples, I need to add another tool maybe. I need to see that. Transparency, like I don't see a lot. I think we're going that way. I'm not going to say that I see a lot of teams do that, but this space is moving so fast.
It's interesting because it reminds me of, are you familiar with DSPY? Yeah, of course. So DSPY is a very researcher slash developer centric version of this. I'm going to describe the intent of what this thing should do and go. It's extremely not productized user friendly.
And I think at least the last time I looked, it was very optimized for like single inference types of tasks. When you're talking about at the agent level and thinking about this, what does that kind of loop look like? And how do you think about like, for example, this may walk through your example of how a particular case my agent is complex, it has different tools. It's got a sub agent. It can call.
It can go down this. And I want to like something at the very top level, my eval is failing. How do I step through and figure out the right places to fix or debug this? Does that also involve evals or tooling? Like, how do you see people doing this?
Yeah. So the going workflow is like, you know, you identify the failure and then you use the UI, like trace field, which really shows you very clear breakdown at what happened in every step, every tool call, every other lamp call, every function call internally, right? Typically, what we see is people just go through that, you know, okay, and then okay, this is the step it's failed or what? Right. So for example, if the output from the rack database, the vector DB was bad, then clearly you're not going to expect the right answer.
If that was good, but that I'm still like provided. So it really allows you to pinpoint where the issue is. And that's super powerful, super powerful. But I think like where we are as an industry today, I think we're going to look back a couple maybe, I don't know, in this space, maybe not a couple of years, maybe a couple of months, but we're going to look back and say, hey, like that didn't make sense that we did that so manually, right? Like, you know, if you go back to the early days of neural nets, people manually tried to put the weights like on the neurons, like manual, this is before kind of SGT and that kind of stuff.
And now you look at it as a con. I mean, obviously they have an intuition that the architecture could work, but the reality is like everything we're talking about is a search problem. Like you have a system with a bunch of variables and parameters and configurations. And if you have an evil test suite, you have a certain score of how well you're doing against it. And we are searching for hopefully the global maxima for this search problem, right?
So let's look at it as a search problem as an optimization problem. So this is something we're spending a lot of time thinking about and we shipped a bunch of stuff on the product and I truly feel that like we're not completely there yet, but I truly feel that's like how things are going to be in the future where we're going to stop doing this like manual inspection of a trace to try to figure out where it failed. We'll have this flywheel of a, you know, evil suite that continues to grow over time. And then, you know, a nightly optimization process that tries to find a new global maxima. Let's talk about what that might mean, right?
Cause like, I think a mental model, we're now moving mental models in a lot of ways, right? We've been taking this mental model. Okay, this is a unit test. What do I do if a unit test is failing? I'm going to debug, which is kind of a manual process or maybe a coding age and assisted process, but it's still like pinpoint to this thing a thread going down based on a failure and trying to correct it.
And what you're describing now is something quite different from that and much more algorithmic and broad facing. So let's watch out. What do you mean when you say, Hey, this is actually, this is an optimization problem. I completely agree, right? Like LMS kind of because of their like fuzzy nature, lend more to this versus like unit test that are going by nature.
Do they pass or do they fail? Right? So if we know what are the variables that we control, which agents is typically assistant prompt tool descriptions, you know, if you have a rep step, like chunking strategy, all that kind of stuff, you know, how many values are returning from vector to be like, you got all these values. Let's call them hyper parameters. Okay.
And you have scored the other side of it with this tuple of these values for this variables, I'm getting 80%. Right? And I'm not getting into how you compute it, but like we can get to that in a while. Like the most naive approach is like, let's brute force search all the possible values to try to get the best result and putting kind of compute constraints aside. Like you would eventually find the best results.
The space of possible system prompts is very large. Yeah, yeah. It's like not actually feasible. Like if you think about it, you know, as a problem, like you would eventually find that. And then you say, okay, you know, let's kind of walk through what they look like.
And then you're like, okay, there's like random search. Okay. Let's randomly select, you know, the user would find your other ranges that I think makes sense. That's randomly sort of. And then you've got methods from ML like Bayesian search, which are a little bit more informed than how to kind of search that space.
But then with these LMS as part of the search or the optimization algorithms, you can do things that are a lot more robust and a lot more kind of compute efficient and kind of searching through all the possibilities. So especially for things like system prompts, which is, you know, you can random searches. It's got to be quite challenging. You know, there's obviously a lot of work on this topic. I think the leading lab across the stand for there's like, Jepa is like a very well known algorithm in space.
We built a couple of our own as well. But the idea is, okay, so I have a system at the configuration that tool bowl. I ran through all the inputs from my evaluation suite. I got a result. Now I need to come up with new candidates for these variables.
So think of let's talk about system prompts. So you can use an LOM in that process to look at the ones that failed. So you say, okay, here are the 10 samples that fail in my data set. Let's look at the tree, like the trace of what happened where it failed, what, you know, what common failure modes. And let's suggest a new system prompt candidate.
It's a suggestion to candidate that we think might fix it. You put this candidate in, you rerun the whole process. Am I doing better? Am I doing worse? And so on and so on.
Now these algorithms get more sophisticated and you're just like evolutionary approach, whereas, you know, suggesting multiple candidates and kind of trying to emerge the best ones. We can get into it if you want. But the reality is, if you have a good evil, they said they work extremely well. Like it's really cool to see. And the best part that this is my favorite part is, you know, an ML, especially an LOM.
But generally, ML, you need at least thousands of samples with these algorithms, like even 20, 30 samples will take you a very, very long way. So it's extremely powerful. Happy to give you a ransom kind of public. I would love actually. Let's talk about some examples of that and like mapping to because like my brain immediately goes to, okay, we have an LOM in the loop.
We're running LOM evals and all this. Like those token costs are going to rack up. So like what's the order of magnitude we're talking about there? What's the length of time this tends to take for a loop or to converge in all those different pieces? Yeah.
So I'll give you kind of one, let's call it a relatively simple, but a real life example, right? Not a 21. So land chain, obviously a very popular library has baked in the library a system prompt that is used when you're using an LOM that doesn't have structured outputs, which is a lot of the kind of smaller open source ones. And it basically asks the LOM, hey, return your response at hearing the specific JSON schema, right? Because you need to parse it into stuff afterwards.
And we just saw from our user base, like how often it's failing. Then we said, okay, let's try to look at this as an optimization problem. We created a simple data set of JSON schemas essentially, and then I started an output with like random values that adheres to that schema. And then we ran it through the original system prompt they had in layer chain. And I think we got about 12% pass, which is we're just making sure that the schema matches are simple, deterministic metric.
And then we ran it through the optimizer. And within two iterations, we got it from 12% to 96%. It was that powerful. Now we're talking LOM costs, it was like less than $1. It was like a frontier model, but it was less than $1.
Like I'm very cheap. We opened the PR to the land chain team. I got committed the same day. They were like, thank you so much. This was painful.
So it works very, very well. Yeah, we have a couple of customers using it as well. We're also software, like you know, software evaluators. But yeah, it truly depends on how much do you believe that metric on your evaluation data set. Fascinating.
Data, your evaluation can be deterministic, you know, very regular. I think LOM is judges where I start to wonder is like, okay, I have an LOM is judge. I have a subject matter expert that's inserted a bunch of things about why, you know, how they would evaluate, et cetera. One, it's a little fuzzier. Two, it's probably a little more fragile.
And three, it's expensive to run a lot of times. Yeah. Yeah, I don't disagree, right? Like, in no way I'm saying like this is like perfect, right? But I'm comparing it to the status quo, which is like, yeah.
And I think that's like a big reason why we're not seeing more of these agents out there. There's obviously a few that are extremely popular in common. Like, I would argue to expect to see more out there, right? Is because of kind of what is the status quo, which is, you know, this kind of like more vite checking, which is essentially, I'll test a few inputs. I'll see what it looks good.
Let's go to the introduction, right? And one argument I always kind of think about when people talk about cost is like, you know, if you look at the cost per token curve over the last three years, it does keep dropping tremendously. Yeah. Remindously, right? But yeah, it does, you know, incur additional costs.
There's no argument about that. Let's talk a little bit about what is the life cycle around these then because I think one off costs or one time costs are very different than recurring everyday costs, things like that. So let's say you're going all in. You're doing an optimization, an optimization, an optimization, you have your Evolve Suite. One, how frequently are you building up to that suite?
Two, what are the situations in which you're running it? And three, how often do you do this? Like optimization pass over all of your different agents, et cetera. So I think there's like where we are today as an industry and I think where we're going to be in six, 12 months. So what we see from our user and customer base is people don't kind of run these optimization every day or every week or maybe not even every month, right?
Like they spend the time to build the evaluation suite and then they do it in dev maybe a couple of times to get to like a version that's kind of good enough. So that goes back to the challenge of like their evaluation suite is not growing as much as you'd expect, right? So which is the bottom like for everything here. I think that's where we are today. I do think that if you fast forward six, 12, 18 months, I think what it should look like is a lot more what ML teams or pipelines used to look like.
All of my ML customers retrain their models on a daily, weekly, monthly basis, right? Obviously some of these algorithms are completely online so they, you know, they update as you use them. So I think it will look something like that, right? Like we have an agent in production. We are getting feedback, whether it's from user feedback, LMs, the judge evaluators to flag, you know, different edge cases, behaviors and so on, you either have or don't have a human in the loop to kind of like improve that data.
It goes into the evaluation data set. And then yeah, once a day, once a week, you're re-optimized, retrain the agent because I completely get the cost side, but like, you know, the fundamental equation with ML is like more data equals better model. Yeah. But these agents is like, you know, you ship this thing, whether you have a one user using or a billion, it's stale. Like it would not get better unless you do something about it.
So I think if we are able to like close that loop where we can learn from production data, I think this guy's a limit. So I'm very excited. I'm fortunate. To me, it's one of the most exciting problems out there. So let's dig in then.
One, is anybody doing this today in terms of being able to learn live, like, I think this is like in V-Vote Learning, right? Like it's happening in the flow of what's going on. Are any of your customers doing that? Like how does that work or what are the barriers that are keeping people from doing it? With this type of optimization, you know, RL stuff that obviously, you know, people do unpalacy and stuff like that.
But I don't think any of my customers are at the point where it's completely like autonomous, you know, in the loop yet. I would also say like we just launched this optimization open source SDK about six months ago, five months ago. I think we've been able to do this for the next couple of days for this entire category. Sorry, remind me the second part of your question. What are the barriers?
I mean, I could think of a few in terms of immediately I go to like, okay, what about privacy, right? Like, where is this EVL suite living? If it's using production data, is there cross contamination? Like how do you deal with that? But what are the barriers that you're seeing?
Yeah, there's a lot of it, right? So first of all, you know, what kind of what we talked about is like when you add stuff to the simulations, then it's like the ground truth. It needs to be right. So you can't just put a bunch of noise or garbage in there. So like, is it human in the loop?
Do you have a really good way to do that? So that's kind of one fundamental issue. Second one is kind of like the operation side of it, right? Like, where do you store these data sets? Like how do you split between different users, customers, deployments, like, you know, all these operational problems, which are real problems.
And then I think the algorithm still have a lot more room to improve. Like, all this work is like super nascent, super exciting, but like it's very early days. And then the other part is like from a deployment model, how do you test this stuff? Right? So for example, right?
Like, you know, one way people do it is like they kind of bake the system prompt in their repo as like, you know, some configuration or whatever, yaml file or even in the source code, they manage it like classic version control. You got a new prompt. You want to test, you redeploy the entire application, but then you prompt. That's, you know, one way to do that. But like, if you think about it, like you could test, we talked about evaluation, sweet testing, but a lot of teams, AB tests these things in production, which is, you know, or can't read that or whatever, but can't read deployments.
But it's like such a small component, typically that like, does it require full on redeployment? And when you think about AB testing platforms, you know, you kind of inject the framework in your code and, you know, if test this, do that and so on, or display the spot in future flags, all that stuff. So one of the approach we're looking into is like, hey, maybe when your application spins up, it fetches the prod prompt from, you know, our platform, for example, right? And then when you have a new candidate that you want to test because you ran an evil sweet or you manually went and changed something, the next time it'll make this inference, it will just get the new prompt. And that suddenly opens up like this whole new world of like, hey, I can correlate production performance to my evil test week performance, right?
Like in production, you can often look at the downstream actions that they actually book the flight, right? For example, yeah. So, you know, it kind of opens up a lot of interesting use cases, which is, I think if you tell, like a lot of software engineers tell it, hey, like your application is dependent on a third party to fetch like a critical configuration. Well, it's a fascinating question, right? Because prompts in some ways, like a mental model I sometimes use is LLMs are like this big virtual machine.
And you're throwing in a combination of code and data that is text. Your system prompt is code. Any user input is code, but it's also kind of data, right? It's also kind of like, hey, we're using this to shape anything coming back from a tool call. Is some combination of this is data or this is code, this is instructions to this virtual machine to keep going.
So how do you manage it? Do you manage it like code in a repository? Like I can make arguments for that. It's important to know the version control and like, this is shaping the core behavior of my system. Or do you manage it like data?
Because like also it's got a different lifecycle and maybe you're optimizing it or changing it or tweaking it or all these other like, it's a really interesting question. Absolutely. And the coolest slash, the scariest part of this is like often like when you confront these problems, like there's like some teams or some companies that like have been doing this for a few years, right? They're like so much ahead that like you can kind of ask or read their blog posts, right? They're like, okay, they might have seen, but here it's like, it's so new that even the leading edge deans are like figuring out stuff as we go.
Totally. I was having this conversation with an engineer. He's like, isn't there an established solution for this? I'm like, are you kidding? This whole field is a year old?
I'm not going to establish anything. Absolutely. We've seen a little bit of it with like the MLOps world, but again, then we kind of lean we have like these early customers that were doing an online production for 10 years. So we lean a lot of them, but it's funny. A lot of like the solutions came from some weird constraints.
I'll give you an example, right? Like so when you think about how you manage model binaries, right? Like they have their versions, they're tied to a production version because inference code needs to match. You got all this kind of moving pieces. And the biggest limiting factor was like get LFS, which wasn't good slash fast enough, right?
So never really made it into like repos. Because like I would argue it should probably live in a repo. I mean, there's coupling. I mean, this is another thing, right? Like some prompts, you have code that is handling that, that is parsing it.
There's not a part of the prompt. So there's coupling between your code and your prompt as code. Yeah. And most teams on that front like use like a model registry, which is like a wrapper around object store, right? And does that coupling exactly that you're referring to, but it also allows you to do these like hot switches in production, right?
So that's actually quite powerful in the ML world. A lot of people use that. Like you don't have to redeploy the entire application unless of course the inference kind of code had to change, but it gives a lot of flexibility because often the team is building the models are not the team deploying the software, right? It's like, so yeah, it's really fun. So are you seeing that metaphor working out in the ML world where you have essentially like a registry for this prompt that has whatever associated code and you say, like, okay, this prompt, probably you have an eval that explains like what is the type of data that can come out of this so that my coupled registry is able to deal with it.
The first approach we took because we weren't sure is like, yeah, you can do it either way, right? Like you can kind of take the prompt through CIC, like you can do it a lot of ways like you decide what you want to do, right? Like we weren't sure what's the right approach, but now, you know, about a year and a half in and seeing a lot of teams doing it, that's definitely the approach we're taking. And it's not just a prompt. What we're building is essentially like a configuration manager, which is has all the prompts has all the other kind of variables that impact the agent.
And then a product manager, which is often the person that owns these agents can go and change stuff, right? You need to be able to revert to older version. You need to have some kind of process to like, you don't accidentally click about it and break production. You need to make sure that like your API is as the configuration registry, like meet a certain SLA, right? Because other people are dependent on it.
But then you can start doing all this cool stuff. You can start doing overnight optimization. You can start doing canary deployments of agents, versions and so on. So I do think that's how things are going to shape out. And again, we have all flavors in the product and it's clear that it's getting there.
And I think it's predominantly driven by the fact that how involved product managers are versus just a pure side. You're smiling. That makes a lot of sense. But yeah, so engineering builds it and kind of throws it to the PMT and they need to manage this entire thing. I'm curious, right?
And this is literally a problem that I'm tackling in my day job right now. It's like, how do you navigate non-technical stakeholders doing prompt optimizations, doing these things and the coupling between all the different pieces. So what you're doing is you've got this prompts configuration system. It has versioning and release gates and all this other, but it has within it a prompt, some amount of configuration. Are there dependency graphs?
It's like this prompt needs to run and it needs to have access to this type of data injected or these types of tools. Is there an SLA for like, this is outputting structured data? Is there an SDK that is matching? How does all the pieces fit together? Yeah, yeah.
We're still figuring out some stuff. But generally speaking, so you have an SDK where you manage to commit or write problems or read problems, right? And it kind of supports the templating and all the stuff that you want to do there. So for example, you'd be like, oh, big dot, get prompt and then prompt name dot latest, for example, right? Something like that.
And then double fetch it when that API or the function call happens. But it works when your agent architecture is somehow stable, but it could have multiple prompts that are definitely dependent on each other. So it's not like this prompt. It's called a blueprint. It's all together.
It might have 10 prompts and have a bunch of configuration, which I'll lend you using or all these different things that you want to control. And then because you're fetching the configuration with some calls invoke, you can change it in the UI very easily and get a good result. And you can respond to production incidents really quickly. So I think it's a very powerful approach. Yeah, that's super cool.
Looking forward then, what is the edge that you're working on? What are the things that you see coming in the next, as you said earlier, times feel compressed, right? I don't try to look at it multiple years anymore. But what is coming in the next few weeks or months, right? And specifically that, Opek front or just my prediction?
Well, let's start with Opek. But I think you have a window into this space that I'd be curious to go and look at also beyond Opek. So I think our friends, we're spending a bunch of the stuff we talked about. How do we help teams bootstrap evil recretions? It's very easily.
And an unethical person can do it. So I think that's the biggest blocker at the moment for everything. So if we figure that out, I think a lot of people will be much more successful with their agents. It's like the first block and then really this concept of blueprints, this concept that you can run up and station, get a new blueprint or variation, test it in production on a certain percentage of your traffic. So that all of that is deep in the pipeline and it should be coming soon.
All going to be open source, like everything we do. So that's that. And then I think once we start getting customers and users in this, like what you call in vivo, like in production, I think we'll probably learn a bunch more. I don't know how to predict how that would look like yet. So that's maybe a few more months in the future.
I mean, generally on the industry, I think first of all, it's clear that these models are still getting better. Everyone who used Cloudcode somewhere in the December time, something changed, right? Like it's. Opus four, five, possibly. Yeah, probably.
But there's been versions, but like there's like something more substantial of how good and opus is like huge part, but I think Cloudcode is a harness is like extremely impressive. Like it's such a good job there. Well, and that is one of the things that's really interesting to me here is like the models are incredible and they do a lot of things and they're continuing to go. But there's still tremendous nuance into how to build an effective harness and put those pieces together. And what I'm hearing from you is like the eval needs to be capturing all of those pieces and looking at all of those.
You can't look at a prompt and a model in isolation. Yeah. And the sweetbench benchmark is not actually a good because that's only the model usually, right? So there's a lot there, right? Like on, and you know, one of the things everyone were talking about vector database like a year ago or two years ago and Cloudcode like I think they tested embeddings at some point, but like it grips most of the stuff and it does it in a really smart way.
So it doesn't kind of blow up the context and like it works so well, right? So I think that's like the other thing that I'm starting to see more and more is a lot of teams, you know, when started hitting challenges with their agent, you know, as engineers were like, okay, let's start building some structure around it. Like, you know, we kind of have these like user flows and if you run this flow, inject this context and if you're in that flow and inject that context, but I think the trend is like, you actually want to give the models more freedom versus more constraint, which is interesting. It's really interesting. And I think it's probably problem dependent, right?
Like I see examples where it's like, it depends on how flexible is the problem you're attacking, right? I have a well-constrated problem where I know what good looks like. Great. Let me lock everything down, get a reproducible pipeline, a bunch of small, tightly scoped steps. Awesome.
I have a general purpose problem. This person's coding, who the heck knows in what the heck language? I can't lock that down. I need to be able to flow with all those different changes. Yeah.
And I think it's like, you know, this is something I'm asking myself a lot, right? So, you know, all these companies are building agents, right? Can you go to this product and then pops up in the UI on the right side and chat to where agent and all that kind of stuff. And if we're going to a world where everyone runs their multiple or whatever, you know, it's called whatever the name is today, like hopefully one that is less of a self-hacking pathway. Yeah.
Yeah. I was like, I said it off and I was like, you want to send him connection to your email? I was like, no. I said, let me not only have the supply chain hack of NPM packages, but like this thing is actively going out. I mean, we're talking about prompts as code.
Let me go out and ask randos on the internet what code I should be running on your machine that you've given me all these permissions. Yeah. So, you know, I think people are like privacy and security oriented like can sandbox this pretty well. But I guess like to the point is like, what is going to be the interface? Like, is this if everyone's just going to expose an MCP and you're going to just use your agent to call it?
And if so, how much do you control out of the harness, you know, right? Like it starts becoming like a different interaction or like you will as a company, you will have your own agent people type in your chat window. And you know, I'm thinking a lot about like, what's the role of UIs? Like we spent everyone has been software industry. We spent so much effort on like UIs and UX and like all of those things.
But like, is the future where like every UI is on demand, like generated by your agent to just show you what you need to see right now? And I don't know, right? I don't have answers to all of these, but like exciting times to be building. There's something there that might be worth digging into when we talk about like agent optimization and how to do this. So we talk about UIs.
I think one of the really interesting things that LMS allow you to do in a UI is you can build kind of more intense based interactions, right? So traditional software is very imperative. I am going to click on this thing and drag it over here. I'm going to go this way. What have you?
Even if you don't have a full on chat bot, which I think we over index on chat bots when we talk about LMS, but like you have something that can interpret fuzzy direction, whether it's voice driven or language or even like I think can still be some amount of gesture or other different things, but it's able to kind of make inferences based on incomplete information and do the imperative pieces for you. And that just opens this tremendous opportunity in terms of streamlining people's experiences. Even if your core product has nothing to do with it all of them. Yeah, I completely agree. I think the question is like, okay, like I have my application code and I write all these all in workflows to try to kind of determine that kind of stuff.
But like, what do you think about like a chat bot session and doesn't actually have to be exposed to the user like, you know, text is like great for something horrible for other things, right? Like, I'm kind of thinking like you eyes will need to be in sync with your chat session, right? So like the UI, it's kind of like, you see the UI keeps changing based on what the context of the session is. And for some things you're going to go and type it, some things I really like talk about data tables. Like I really rather have a couple buttons to filter and sort them and look at things, but they have to be in sync into your point and then you can start doing all these things that like you can infer implicitly about what the user is trying to do like so much better than like, oh, that was the user flow.
We thought about when we did this product meeting. Now you have an interesting eval question too, right? How do I eval that this UI that got generated on the fly and I have seen some fascinating experiments with like generating, you know, generative UI as you can do really interesting things. Some of them are terrible. Some of them are great.
But like, what does that eval look like? Yeah, I don't have all the answers, right? Like the reality is obviously it's a very different modality, but like it'd be similar to like what we're seeing today where you're going to have some form that's human review provided the feedback and then you're going to try to align your or optimize your LM as a judge to try to be as close as possible to the human evaluator, right? Like, and that's like a continuous thing. It's not scalable to have humans review everything.
LM as a judge by itself is introduced so much more fuzziness and noise, so it's not helpful. So I think we'll have to be somewhere there to make this stuff work. But yeah, indeed, like that, it gets hard. This is like a slightly different use case, but like there are all these evaluation datasets for browser use models, which are different but not completely, right? Like, yes, the UI is not generated, but like the state is determined based on the actions that the web user agent did.
So I think they do it on the DOM level, to be honest. Well, you could do something interesting with that. And like one of the models I've seen for doing this is like, essentially you have UI layer that is controlled by like reactor Redux or something like that. And then you have like a JS sandbox and you let your LM just like, you'll load code into the sandbox and all it can do is issue Redux things, right? But that's enough to run your UI layer.
And so like you've got a sandbox that's safe, it's not messing with stuff, but then you end up with React code, which you could test it in any of the React testing libraries or it does well down to DOM level and you could do DOM level stuff. That's interesting. Yeah, I've been seeing that. You should have like a pop later. I'd love to see that.
Like I've seen some attempts. There's like a library of forgot the name of it. And then there's some attempts to sync sessions between like some, you know, agent session and a UI. There's a bunch of stuff of like in chat UI components that try to do that. I don't know what the right modality is, but clearly like where we have tons of chat bots, tons of UI products and like at the moment, it's just like a widget at the right side.
Like I feel like it's going to change a lot. So we're getting closer to the end of our time. Well, so let's maybe close with this, right? You're seeing a lot of teams building agents grappling with these problems. Do you have like a set of advice or guidelines or like things you would say like, hey, you're tackling agents, there is no established, you know, your old best practices, but here's what I'm seeing in the field.
Like you should be doing these things. The first one is like, there's tens, if not hundreds of like agent frameworks out there, 80% of the people I see are not using anyone. Like you can spend, you know, months in like testing all these different frameworks and they do add value. I'm not like diminishing them, but the reality is some of the most successful ones out there are kind of home-brewed vanilla builds. So I wouldn't spend too much effort on that.
My next piece of advice is like, and it's kind of annoying. It's kind of like you go to the doctor and tell you, oh, you have to work out more, right? It's like, you know the answer, but a lot of people struggle with like doing or eat healthier and all those kind of things. But like spend some time on building like a very small evaluation data set, 20 samples. Like just 20.
Like it will pay off big time, right? Big time. And then you can yolo the whole thing, you know, vibe code your agent. Like you'll be so much more successful. The other thing is like when we talk a little bit about it, like I don't typically suggest to worry too much about costs early on, mostly because they tend to go down by roughly 90% year over year by design.
And it's like a little bit of premature optimization to use the best model, use a frontier model, make it work first. And then you can figure out can you kind of make this work with like a cheaper, smaller model. And then just generally right, like if you're online, you're online, you're on Twitter, it seems like everyone figured this out and everyone has like hundreds of, you know, Fortune 500 as well as say we have 10,000 employees or actual ages. The reality is like everyone in the industry is trying to figure this out and it's hard for everyone, including you know, open AI. They just put a great post on their data agent and you can see they're like struggling with the same challenges like all of us.
So don't feel that pressure. There's like everyone figured it out that you did it. So I would say that's an unnatural.