All right, Boris, we're so excited to have you here, the creator of Flotcode. Thank you. It's great to be here. Fresh off the press, you guys just shipped Opus 5 yesterday.
Yes. And it seems that model performance keeps accelerating. You guys got, took Arc-80i 3 to 30%, which is incredible. Yes.
And for context, before the best score was in the low single-day desert low teens, right? What can Opus 5 do now that it couldn't versus a previous version? Yeah, there's a lot that goes into every new model. And there's a lot of new capabilities that we teach and get the model to do.
Whenever you do model training, you try to teach a whole bunch of different things. And most often, it doesn't work. But some subset of the things the model does learn. And sometimes it also surprises you.
It has these skills, it has abilities that you actually didn't really teach it, but it just kind of run. For five, one example of something that does that I think no other model has done is it runs for a very long period of time. And especially when you combine Opus 5 with automotive, it's just incredible. It can go for days, weeks, months at a time.
It just won't stop. You don't even need to use scaffolding. So you don't need sacheco. You don't need all this other stuff.
It'll just go because it knows it needs to do the task. Another thing that I'm really excited about, and I'm going to start, I think, to talk about a little bit more, but it's kind of surprising because it's such a new capability, is the model does not seem to be prompt injectable anymore. That's kind of injectable. It's crazy.
People have talked about this lethal trifecta for a long time. And this really affects this kind of harness design, that agent sign, and product design. Because if the model reads some instruction on the internet, that's like do X and Y and Z, and also delete everything on the users in here. A year ago, the model would have just done it.
But nowadays, Opus does not. And this has actually been the case since Opus 4.7, 4.8, Sonnet 5 has been quite good at this table, as I could add it. But Opus 5 just hits a new frontier on this. So essentially, if you combine a well-aligned model, so this is essentially three years of research into alignment, with a prompt injection classifier, which we run for all traffic.
And what this is doing is based on crystal is mechanistic interpretability work, where it's literally we're looking at neurons in the model's brain that light up when prompt injection happens. So the model won't even tell you, but we can actually see those neurons, and we can figure out integrals that are happening. And then you combine that with the automotive classifier. And with these three layers, we just cannot demonstrate prompt injection anymore.
And we've hired security researchers. We've done like a renting. We've done competitions. No one can demonstrate it.
And so I'm actually quite curious if people are able to. And that'll be actually a really amazing signal for the research. So if anyone in this room prompt injects a clock, Opus 5, you won't get a special prize before, as maybe. Now talking about a prompt injection, the other side of the coin is now the system prompt.
Let's talk a bit about the new release. You actually deleted over 80% of the system prompt from the clock code. Tell us more about that. I think something that a lot of people might not realize is cloud code as a product and as a parent is just always changing.
We're always adding stuff. We're always deleting stuff. Every time that a new model comes out, we delete a bunch of the system prompt, change a bunch of the system prompt. We change the set of tools all the time.
We change the prompts for the tools all the time. And the reason is every model is very different. So something that you did for one model maybe three months ago, it just might not translate at all to the next model. And so one thing about Opus 5 is it's just really intelligent.
And a lot of the stuff in the system prompt was correcting for these behaviors that a model should have known, but it didn't. Now Opus 5 just does it. So yeah, we deleted 80% of the system prompt. You can actually try deleting the rest of it too.
So when you run cloud code, you can just do dash dash system prompt and set whatever system prompt you want if you want to experiment with it. And another thing that you can try is simple mode. So this is actually this undocumented feature. If you do cloud code simple equals 1, like this environment variable, and then you run cloud, it'll delete all the system prompts, including from the tools.
And we actually use this as a sort of ablation to figure out is the prompt useful. And what's interesting is that the model is actually a little bit more intelligent without these prompts. That's something that we've been finding. But when you use cloud code as a product, you do actually want some of these prompts because it helps you use the product, and it helps them put the product behave and the model behave in the way that you would want when you're using it as a person.
I think the thing that's really fascinating in this era of building, basically, you'll build the best harness in the world for cloud. And that's cloud code. From what I'm hearing, for every model released, you basically delete all of the code base, delete all of the prompt and start from scratch every time. That in the whole world would have been not something started with a done for the product.
It's like pressing the lead every six months for everything. That's right. So to be fair, we don't delete the entire code base, but we do delete a lot. So every time there's a new model, we try in research, it's called this ablation.
And so what this means is you delete the entire system prompts, and then you bring it back line by line to figure out what is the impact of each individual line. It's sort of like an eval. And you can kind of like evaluate it, and the ablation essentially is an eval, but you delete things to figure out the impact. And yeah, we do the same thing for tools.
We unchip tools all the time. We, you know, delete code in the harness all the time. If you look at actually the code that's in the quad code harness today, almost all of it is about safety and permissions and static analysis. And there's a bunch of UI code.
And we've actually unchipped a lot of the other code already. Do you think this way of building a agentic product and harness, and basically doing ablation every time with a new model release to everyone in this room that's building AI products, basically do that, be comfortable and brave to test delete? 100%. Yeah, and for people that aren't building agentic products but you're using quad code, every six months, delete your bottom D, delete your skills, delete your hooks, see what the model does and it might surprise you.
And actually for Opus 5, this is something we really do recommend is just try deleting all of these things because the model might really just not need all those instructions that you needed for past models. Let's talk a bit about how then you build this new prompt. When there's a new model release, like for everyone in the room, everyone wants to try Opus 5 and they're gonna press delete on their system prompt. How do they go about rebuilding the system prompt?
How do you set up your environment? So you do it with kind of piece by piece. So the first step is you delete. The next step is you use it.
And you don't wanna guess what's the instruction that the model needs because you might not predict it correctly. The thing that you wanna do is you wanna run it. And if it's like a custom agentic product that you're building, you wanna run the product, you wanna see where it fails with the model, you wanna see what it does well. If you're using quad code, you wanna see where it does what we're gonna put this or maybe where it stumbles over the architecture or it stumbles over something else.
And only when you see it repeatedly stumble on the same thing, that's when you add it back. But you don't wanna do it too early because remember like the model is gonna read this instruction every single time you use it. So you really wanna make sure that the model needs this instruction. I think this is sort of the crazy thing about building on models is just so different than all the engineering that I have every time.
Like in the past when you build on systems, you build these like big, beautiful systems and you really think about the system design upfront, you have like a big suite of unit tests, you think about everything. And like a re-architecture is a big project. It sometimes it takes months. I've worked on re-architecture products at big companies that take years.
And the model is not like that. It's the way to think about it is almost like a living creature, like as something more organic. It's a thing where every model generation it behaves differently. It has a slightly different personality.
And you have to take the time to get to know it and then adjust the harness based on that. And I think it's just very much like an empirical and kind of scientific thing. You have to take a very scientific mindset to it where you try something, you see the result and then you iterate based on that. If you're building in this world right now, what then becomes stable are you vast?
Something that you keep from the previous models and keep using them in each new model release? We do until we max out the evolve. So that's what it took for everyone. So code and system from you have, if you want to build at the bleeding edge and have the most capability for models, you've got to delete those.
But you vast are constant and keep appending to them basically. Yeah, you keep appending. What happens is, I actually wouldn't even go this far to be honest. I think evals, they outlived the harness a little bit, but not that much.
Like an eval might live from maybe one to three model generations, but nowadays we're on exponential. The model is improving so quickly, very often we just saturate the eval and then we have to throw it away and we have to come up with a new eval. And this is just part of the process. And again, it's about being empirical.
You have to use the product, you have to use the model, you have to see where it struggles. And then based on that, that's the eval set that you should build. I think one one term I heard you describe how to build the best agent products on top of a plot is this concept of a on hobbling plot. And tell us more about what that means.
Yeah, so hobbling is this idea in research that the model is doing something and you're just getting in the way. There's this kind of way of thinking about it that I really like, it's very useful when you're building product. It's called product overhang. And the idea is, the model is able to do all sorts of things with today's models, not a future model, but today's model that we have not yet realized.
And there are so many capabilities the model has like this that people are not aware of. And this is like the ability to maybe use a particular tool, use a particular language, solve a particular kind of problem, do things a particular kind of way that we thought was kind of beyond the model's capability. And there's this overhang, because the model can do this at every given model generation. But there is often not a product that lets the model do this and lets it express this kind of ability to do this.
And on the flip side, often what happens is the product gets in the way. And it's getting in the way that we call this hobbling and then not eliciting the correct behavior from the model, we call this product overhang. So it's kind of like two sides of the same thing. One example of this was the original product code.
When I first started working on it, this was like a year and a half, two years ago, something like that. This was like Sonic 3.5. At the time, that was an incredible coding model. That was like the best coding model that exists.
Nowadays, it's a pretty terrible coding model by moderate standards. But I think that was like the first great coding model that we built as a product. And at the time, if you looked at the coding products at the time, what were they doing? They were doing like single line autocomplete.
They were doing sometimes multi line autocomplete. That was sort of a new idea. They were doing chat. So you can talk to the agent, but it wasn't right access.
You can only read. You could ask about the code base. And so the feeling was that there wasn't really a product that was fully eliciting the model's capability to write entire functions at a time, entire files at a time. At the time, it wasn't entire features.
We weren't there yet, but probably entire files. That was the level of capability at the time. And so the deal with quad code was, all right, we think the model can probably do this. What if we get rid of all this capital thing and just give the model the simplest possible harness so it can write an entire file at a time and build an entire feature?
And that was kind of it. That was the product overhang at the time. The model was capable of doing something and everything was just kind of getting in the way. I think that nowadays with modern models, there's so much product overhang that I'm not saying startups capture.
And I think there's people thinking about these problems, but there's just a huge amount of opportunity to elicit these behaviors in the model that are just amazing and interesting and commercially valuable. I think this is such a special insight for everyone here in the room. Basically, all of you could create the next quad code if you figure out how to unhobble the models. Because that's effectively the best story of quad code.
You unhobble 23.5. Because all of the previous iterations were still getting the model very rigid and idealized. And clock code was one of the first instances that gave it just a full terminal axis. And that then created this amazing product just that keep going.
So let's talk about one or some areas of how should future founders here think about unhobbling, cloud, and fixing this product overhang. So there's a couple of things that we'll think about. One is you should give the model slightly harder tasks than what you think you can do. I think a really common mistake that I see is people are using quad code.
They're using quad code. They just give it way over-wee specific instructions. They're like, I want you to do this, but I want you to do it in this way, this way, you must do one, and two, and three, and four. And from other models, that's actually really not the way to do it.
You want to go a little bit higher level. You want to describe the task. You want to describe the guardrails. You want to describe the exit criteria.
And then just go with the model cook. And come back in a little bit. And I think it'll surprise you. And again, this is just not something that would have worked six months ago, but it does work today.
Can you give some examples of these challenging task or capabilities that people should explore that it can do now, that it could end six months ago? Yeah. So, OK, one example is the model can now rewrite essentially any code base from one language to a different language. It's just sort of crazy.
Like, if this work that would have taken just like a very long time as an engineer, and now the model is quite fast at it. So one example of this is quad code is built on the BUN JavaScript runtime. It's an open source JavaScript runtime. It's an alternative to Node.js.
It's kind of a faster node. BUN was written in Zig. Zig is a systems programming language. It's kind of like C.
It's very well-level. One of the problems with Zig is you have to manually manage memory. And so it's quite easy to run into situations where there's memory weeks and other memory management issues. And so one thing that the BUN team was doing is they were having quad fuzz the code base and trying to simulate and trigger memory weeks.
And we were doing this for a long period of time. They were able to find a memory week. It was sort of like a case at a time. And that was kind of the capability of the model at the time was doing this fuzzing.
And then at some point, Jared on the team was like, OK, let's just rewrite it. Maybe the model can do this. And I think this is one of these test problems that he threw at the model with every new model generation and starting with Fable, the model started to be able to do it. And so I think all this pipe could do it as well.
And so what he did was essentially he defined a test suite. The nice thing about BUN is it's very, very well-tested. There's a big test suite in BUN. There's a big test suite in Node.js.
So it's easy to know if you did the right thing. And he had the model rewrite it from Zig to Rust. It was one prompt. It was a dynamic workflow.
And a dynamic workflow was our feature in quad code that essentially let you orchestrate dozens, hundred thousands of agents to do work productively. And it ran for 11 days. And it rewrote the entire code base. And this was one shot.
It was one shot with, no, it wasn't one shot. There was steering. There was steering. But previous models just couldn't do this.
Even with a steering. It just wouldn't have been possible. Just 11 days. Oh my god.
This could have taken in the past. Even with the best of the years. What was the ones years? Definitely over here.
Over here. This was over 100,000. Like JavaScript sometimes were really complicated. There's a lot of stuff in there.
And yeah, it works. This is in production now. This is what quad code uses now when you're running it. So this is kind of one example.
I would give a second example of a product overhang. And so this is like a practical use case where there's a problem you're solving. It's like a business problem and engineering problem or product problem. And you should just keep throwing the latest model out to see if it'll just do it.
Because even if a previous model didn't, then one might. I think the second way to think about it is experiment. And just give yourself freedom to play with a model and do creative things. Often it'll surprise you.
So something that's actually been really popular at an internally that's been kind of viral within the last couple of weeks, is someone figured out that you can give Opus 5 OpenCD. And you can have a drop. And so something you can do is you can ask Opus, hey, use OpenCD to draw this image. And it's actually quite good.
It can do portraits. It can draw animals. It can do landscapes. And we didn't train the model to draw.
It's just like this solicitation gap. If you ask it to do it the right way, you can just do it. And we discovered this kind of accidentally, just by playing around and trying creative things that didn't have direct commercial applications. But it's just kind of interesting.
And my hypothesis is there's probably dozens, hundreds of opportunities, like this with the models of today, that no one has yet realized. And the big area of research for this is basically model elicitation. Becoming really good at figuring out all these capabilities and asking the model to do the right thing. Yes.
How do people get better at that? And effectively, how do people get better at prompt engineering? Do people still need to do a lot of prompt engineering? Changing as well.
Tell us about where this is going. Yeah. I remember a year ago, one of the most popular job of things was prompt engineering. And then it kind of changed.
And then I think they can like context engineer. So there's these kind of waves of it. I think these will kind of like come and go. I think the skill nowadays is less about prompt engineering.
And more about figuring out how do you give quite a hard task that seems a little bit too hard. And then how do you make it possible for a lot to verify its work along the way? And the verification, I think, is probably the single most important thing that people do not get right, fortunately. One example of this is people were, we have this desktop after cloud.
And it's built using electron. We've made it quite fast. So now it's like a pretty awesome experience. Six months ago, it was like a sluggish.
And it wasn't very reliable. Now it's pretty awesome. And I think that most of the team uses. As an experiment, though, I wanted to see, what would it feel like if it was native?
And so what I did is I started a cloud tag session. And cloud tag is just, it's a new product we have. It's just called running in Slack. My first question was, hey, you have access to a Mac OS runner on GitHub.
And it said no. And then I hope to run it. So it was able to start a Mac virtual machine using GitHub. And then my second question is, I created this empty code base that was a cloud desktop app rewritten in Swift.
And I asked you to access this code base. I said no. And then I gave it access. And I was like, OK, great.
Now I have access. And then I was like, OK, now I want you to do is, I want you to rewrite the electron app in Swift. I want you to run the electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel. Compare it to the Swift version.
Don't stop until you're done. And that was your prompt, basically. That was my prompt. And how long did this take to run?
It's still running. When did you start it? It's been over two weeks. So it's like working days, 15 days.
Yeah. So I don't know if anyone in the audience has gotten a clock to run a task for more than two weeks. I don't know. Raise your hand.
Anyone in the audience. Oh. All right. Some?
This is like one of these. This is about a recitation. So this is really one of those examples where the model can do it today. You just have to let it do it.
And you don't need the fancy stuff. You don't need slash loop. Please help. But really all you need is give the model the task.
Give it a way to verify the output of its work so it doesn't get stuck. And then it will just go. And actually, in this case, it's a live blog. So what it did is it created a Slack channel internally and it started disposing screenshots every few minutes of its progress.
Wow. So the prompt sound is so simple. I mean, everyone here could do it. And I guess what is separating the people here that can become the top 1% clock or users?
How can people learn to use clock or like Boris? Maybe like, don't listen to the LinkedIn influencers. Don't listen to it. You're on the sweet Twitter.
This is the thing about the model. I think everyone's looking for a good one. We were tricked to do it. It's just like, that doesn't exist.
There's nothing like that. The way the model works is you have to approach empirically, you have to give it a task that's too hard. You have to give it the tools to verify the work. Like you would yourself.
Like you would if you were doing the task. You have to see where it struggles. And then you have to fix that. Either with better prompting or with a skill.
Or if the model's missing context, give it a MCP. So it can pull in the context that it needs. That's kind of it. It sounds very simple.
I think people tend to overthink it a little bit. I think people tend to over-engineer. Because I think in a lot of ways, let me build systems in the past. That's the way you have to do it.
So when I look at engineers that have been coding for a long time, for years or for decades, this is a really, really common failure mode is trying to over-specify. And it's trying to be overly specific. Get the model to do the task exactly the way that you would have done it. And that's just not the way the model works.
But I think a lot of people are kind of un-learning this. And it's a journey to un-burn it. And it's a journey to figure out how do you treat this thing like you would a co-worker. I think that's the level of intelligence that is at now.
And as far as this, let's go deeper into this task. It's still running two weeks. Since you launched it, like, two weeks ago. How many agents did it spawn?
No, I'm not sure. I can ask quite. And then I can get back to you. Oh, yes.
Thousands, tens of thousands. Has anyone in the audience had a prompt to do the models that spawn more than 1,000 agents? I think this is another of the tips. The best cloud users are able to spawn tasks that are really providing a lot of leverage, like thousands of agents.
Yes. How do you do that? There's a few different ways to do it. The easiest way is dynamic workflows.
To use dynamic workflows is a fairly new feature in Cloud Code. And all you have to say is use a workflow. That's it. And the cloud will destroy the dynamic workflow.
What a dynamic workflow is essentially we have the bond run time. We use bond as a sandbox, and we sort of have virtual machine within bond. And we let Cloud start a lot of agents and work to straight them. And it doesn't just do one agent.
It doesn't just do like 10 parallel agents. What it might do is, let's say a task is like rewrite the code base, or do really in-depth data analysis, or some really complicated data, or maybe build a very complex feature that takes multiple stages and maybe dozens of forward quests. And so what it's going to do is it's going to start a bunch of agents to do the first pass. Based on that, it might do a second step, where it has another set of agents that verify the work, or that summarize the work.
Then it might do a third stage, where it'll fan out again. So it'll kind of productively orchestrate a bunch of different agents. So my background is functional programming. And so the way that we design this is essentially an algebra for agents.
So there's a way to run agents in sequence. There's a way to run agents in parallel. And Cloud has different tools in order to orchestrate these agents inside of the sandbox to use tokens efficiently to do really, really complex work. It's kind of cool in something that just hasn't really been written about a lot.
But this is actually a new form of test-time compute. When we talk about the scaling laws, and we talk about the model getting more intelligent over time, historically, it's been a function of the size of the neural net, the amount of training data, and the number of blocks that you put into the training. And then recently, we also added test-time compute. So this is essentially a fancy researcher way of saying, how many tokens does it generate?
And now, dynamic workflows are essentially a new way to orchestrate test-time compute. And it's a new way to kind of really, really ramp up the amount of test-time compute that you use to do a really hard test. So this is all very one way to say this is one way to launch thousands of agents in a way that is productive and efficient. A second way to do it is loops and routines.
Loop is essentially a cron job that's running locally for a bug. Retin is the same thing, but it's running the bug in the cloud. So you can close your route up. And this is slightly different.
Because for a dynamic workflow, it's one task and you break it up into chunks. For loops and routines, it's one task that is repetitive. That doesn't sure context, but it might share memory. And you kind of do this over and over.
You can do it every hour, every five minutes, every day. And so the thing that we started doing is we actually have quad maintaining itself now. And the way we do this is we have a Slack channel where we just have to start a bunch of different routines to maintain its own codebase. And we actually do this for the CLI, for the Android app, for the desktop app.
And for example, one routine is clean up that code. This is a single prompt. It's like one sentence. Quad runs this every day.
It'll look for a dead code across all the codebases using static and dynamic analysis. We didn't prompt that. It just kind of figured it out. And it'll put up a work list every day to do with the dead code.
Another example is shipping experiments that should go out. So the experiment's already out to 100%. It'll delete it from the codebase. And it'll just ship it.
Another one is writing tests for areas of the codebase that need test coverage. Another one is the leading test that don't need to be there. Because another kind of useless test added by older models or added by people at some point. One that I really love is this, I forget what we call it.
I think we call it abstraction police. An idea is there are often in a big codebase there's kind of the same abstraction. And it appears multiple times. And if you kind of went to, maybe you should just be the same abstraction.
But kind of over time for whatever reason, you rebuild it multiple ways in different parts of the codebase. So Quad kind of goes out every day across all our codebases. It finds these nearly duplicated abstractions and it unifies them. And so now we have every day, maybe 20 or 30 of these routines.
It's running across all of our codebases. And it's not totally there yet. But we're on the path to fully automating the maintenance of our apps by doing this. And this is, again, hundreds of agents running every day.
Sometimes thousands of agents every day. It's doing the work of dozens or hundreds of engineers. This is what it used to take to do this kind of work. And this means that engineers can do the thing they actually want to do, which is shipping products, and talk to users and do stuff that's actually called.
I guess it makes a conclusion from this. What you have mentioned in the past that basically coding is solved, right? You have mentioned this. I'm curious now that effectively everyone can write software.
What separates the exceptional builders from the rest? What are the qualities now that everyone can code? I would give one caveat. So coding is solved for the kind of coding that I do.
It's not software everywhere. There's still code bases that are super deep systems code bases where quad-stool struggles, distributed systems where quad-stool struggles, there's really kind of in the weeds you either have a vision, something is off-vipix or something. But it's still not perfect. Like, oh, this is a big leap in vision and computer use, but it's still not perfect.
But I'm not sure if you're here. Raise your hand if 100% of your code is written using agents. You don't write any code by hand anymore. It's pretty good.
How about more than 50%. It's like we have best hands. I think it's getting there. So it's getting to this to being solved for more and more kinds of code.
And that's kind of cool. When I think about the people that are the best at using quad, I think there's a certain mindset that you can bring that's really effective. And it's really about being empirical. So forget all the things that you learned about past models.
Forget everything that you learned about computer science theory in class. Look at the model, try to do a task, see where it struggles, and then based on that adjust. So it's just like very much become, it's not a theoretical science, it's become an empirical science. So I think people that are really good at this, that are really good at forgetting their priors, letting go of this idea that didn't work before and just being open to trying it again.
This is the kind of skill that's very, very successful now. Now my last question is given everything that we talk about, if there's someone here that's studying CS and you learned the program before this era of coding, what should students still learn the hard way, like the old way? So for me, I learned computer science practically. I learned it by teaching myself the code in order to solve problems.
Whenever I was doing this, I was doing it to solve a particular problem that I had. So I actually first learned the code on TI-83 calculators. It was back in middle school. I ended up actually writing a guide on the internet for programming TI-83 calculators.
It's still up on the internet somewhere. And it was basic. That was my first language. And I learned how to program on the calculators so I could just get better at my math test by cheating on the test.
So it was about something practical. Like to me as a middle schooler, that was the most practical thing I could think of. And I ended up getting good grades and I got this little serial cable to give the programs to my classmates and they got really good grades. And then the math got a little bit harder.
It wasn't something that I could solve in basic anymore. And then I went from this algebra solver that was written in basic. And I had to solve harder problems. And once we got into calculus, I had to run assemblies so I could write a better solver, a cheaper on the test.
Now that it was tough. And so for me, programming has always been very practical. And I think this is always my advice for people in school is wearing not just a computer science. This is intellectually fascinating.
And it's really, really interesting to know. But we're not able to apply it. And often this is about building startups. It's about building products.
It's about developing your own design sense, developing your business sense, learning how to do data science, learning how to talk to users. There are all these other skills. And when you combine it with computer science and engineering, that's where it becomes really, really valuable. So those are the hard skills that I would still be doing by hand.
So if I'm hearing and summarizing, start with making something you want first for yourself. And then level up and make something people want. Yes. And we just have one last special announcement for us.
One last thing. Yeah. So for everyone here today, you are getting max 20x. Pretty good.
So look for a code in your email. And I can't wait to see what you built. So I'm curious. Someone in this room should be building something that runs, hopefully, multiple months and thousands of agents.
Now that you have the account to do it. And with that, thank you so much, Morris. Thank you.