Open-Weight AI Models

Open-weight models are AI systems whose trained parameters are publicly released, which allows developers to run, fine-tune, and deploy them independently, rather than accessing them only through a hosted API. While closed-weight models from companies like OpenAI or Anthropic are delivered as managed services, Open-weight models give organizations direct control over how the models are deployed and used. Importantly, the performance of these models is steadily improving, and they've become credible alternatives for production workloads, with advantages in customization and data privacy. Fireworks AI is building a platform focused on serving and customizing Open-weight models at scale. The platform includes optimized inference infrastructure, multi-hardware support across NVIDIA and AMD, and reinforcement fine-tuning capabilities.

Beni Chen is a co-founder of Fireworks AI. In this episode, he joins Gregorvan to discuss his path from Metas ML infrastructure teams to co-founding Fireworks AI, why Open-weight models are becoming increasingly competitive, how custom kernels and speculative decoding improve performance, reinforcement fine-tuning, and much more. Gregorvan is a security-focused technologist, having previously been a CTO across cybersecurity, cyber-insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn. Hello, and welcome to software engineering daily.

My guest today is Beni Chen. Welcome, Beni. Thanks for having me. Yeah, great to have you here. So we're going to be talking all about Fireworks AI, which is a company that I believe you co-founded, I think, three and a half years ago.

Is that right? Yeah. Nice. Yeah. So before we dive into Fireworks AI, and I think, especially today, this is quite pertinent to sort of where maybe Fireworks AI came from, you spent a lot of time at Meta and in their ML team, which means you were doing things with ML probably way before many of us really had it on our radar.

But what was your path, what's been your path to, like, through software engineering and especially the Meta phase as well? Yeah, that's definitely a... It's been a while, so I mean, what the journey was like in the beginning. So in the very beginning, I joined as a software engineer in 2014 on the Inter-18, where most of, I would say, the non-recommendation system makes remin started. Early on, it was sort of like decision trees for different fraud behaviors.

And then also on the team, we started doing, like, image classifiers for different rules for advertising. And then come 2016, switched over to the ads infrastructure, so worked on supporting the recommendation system models. And then in 2017, I think the Facebook leadership back then started thinking about having a ASIC in-house, that's more like Google's TPU. So we started collaborating with Intel back then on a ASIC for recommendation systems. So there's anything about like doing something that's not cool before everyone remembers, definitely doing ASICs in 2017 isn't necessarily cool.

It was an interesting project for sure. And I always tell people that the ASICs I worked on supporting back then was 17 watts. So if you look at the new NVIDIA GPUs, that's like 1,000 watts. Some of the peripherals on that chip is more than 17 watts. But it was a very small chip, it's like how many years ago?

Almost, nine years ago, nine years ago this one. So it's like time really flies and things really change up quickly in Bay Area, I guess. But yeah, I worked on supporting that chip for about two years. And then fast forward NVIDIA started shipping 8100s. That's when everyone realizes, hey, I guess ASICs are not going to be as good as 8100s.

And I worked on supporting the PyTorch enablement for all the ads models in 2019, 2020, like around the pandemic time. But yeah, like NVIDIA also wasn't a huge company back then. So I probably should have just loaded up on a video stock rather than doing anything else. And after that, it was sort of like all the NVIDIA GPUs for another two years, until we decided that it was time to start something new and then started working on Fireworks. Yeah, nice.

Because I was going to ask, I guess, why leave meta, I guess. But everyone's always got their own reasons for doing their tenure at one of the big tech and then starting their own thing. But was there a defining moment for you? Was it just like the time is now? Yeah, to be frank, I probably could have premeditated more.

I probably could have think of through more analytically. At the same time, I do think AI infrastructure will take off. We in fact started before chat GPT came out. So we couldn't time the better. We started maybe like five, six months before chat GPT was shipped.

But yeah, it's not so much that there's any particular trigger. But definitely being a metaphor about E is, I mean, all good show must end. So for me, I think it was also maybe it's a little bit more risk-prone than some of my friends. I do think taking home a risk is a good thing. So I probably didn't think it through as hard as I should have in retrospect.

Yeah, but I mean, that's where some of the best things come from. So yeah, that's probably why we're sitting here today speaking about Fireworks AI, which I think, yeah, let's get on to what you did next. So let's just sort of start a super high level. What is Fireworks AI? Fireworks is a platform that serves and trains open source models.

And we mostly stuck to that mission since chat GPT shipped till today. And to be frank, our play hasn't been super sophisticated for our customers. At the same time, the work itself is very complicated. And a lot of our customers appreciate the support we're able to give them. Specifically, a lot of our customers are very either AI-native or they are very AI-leaning enterprises who are looking to either offer products that's based on language models or looking for automating large part of the organization through large language models.

We're here to help them customize the open source models and then scale them up. A lot of startups will work with also start with different key models. And as they scale, they needed to make sure that their unit economics are good. And we're here to help customize open source models so they can reduce their total cost of ownership for their customers and start making money. In general, I really appreciate a lot of our customers support for having trust in such a young company versus other big clouds and other offerings out there.

And yeah, we're here to help. Nice. And I think usually we kind of later in episodes go into stories, I guess, but I think one that's maybe useful to pull out now to help set the scene for listeners is Cursor. So I believe that you guys do work with Cursor. I'm taking a guess that this is like, for example, when, well, I think code completion is like one area that you guys supply to Cursor is like, can you just talk a little bit about that?

Because I think that's going to help set the scene for the rest of the episode just in terms of what does Fars actually do and for whom. Yeah. Yeah, honestly, the Cursor people are amazing and we're mostly here to help support them. I think early on, a lot of work around supporting Cursor was their custom models like tapped and editing models. And we helped design solutions that would work very fast with their model in a cost-effective way.

I think one thing that we publicized early on was the Fars supply model. We serve for Cursor, which require a lot of special support for speculative decoding in order to support the model properly with fast decoding. So in those settings inside the editor, you want to edit a very large file in one go. A lot of times there's a lot of nuances on how to serve those kind of models. And we work with them to set up a sort of like a dedicated algorithm so that we can serve those models very, very cost effectively.

Yeah. And I think like recently Cursor also published a blog around how to do like online learning with the attack models. Yeah, they are like very sophisticated team. Yeah. No, I mean, I was a fan of Cursor for a long time, I think, since it almost came about.

Unfortunately, I don't do as much hands-on coding today as I did even 18 months ago. But that's just a function of where I've gone in life. But yeah, Cursor still loves Cursor products. Still use it if I do code. So let's kind of walk through some of the features if you like, of how Fireworks' inference infra actually operates.

I mean, just to kind of give some scale idea here, probably should mention to this at the start, but you guys processed something like 13 trillion tokens a day. Is that right? Yeah. Yeah. And I think I'm probably not here to share the latest numbers, but at the same time, I think the 13 trillion number was larger than the Gemini and the opening high number they shared for the APIs.

I think Gemini was like 10, 11 trillion kilo peak. And we've been growing quite a bit since we shared a number last time as well. But yeah, open source models are very strong. Definitely it was a leap of faith from us to focus on open source models. At the same time, I'm like very pleasantly surprised on how strong the open source model has been.

Getting to a point where it is price competitive against close source models. This year, it just in the beginning of this year is amazing. I do think a lot of open source models last year shows promising benchmarks scores, but are not competitive end-to-end. I think what really helped set this year is when Open Claw came out, which one, people realized, wow, these models are amazing. Two, wow, these models are expensive.

I had a friend who set up Open Claw and he said, yeah, this is so good. At the same time, I don't know why one message cost me a million tokens. Just send my message on Telegram and then a million tokens gone. And he doesn't even know where it went. He just sees the money going away.

And two, a lot of the open source models now offered are very competitive in a real open core setting. Well, you don't necessarily want the most expensive model just to buck your restaurant. You probably just want something normal. Can't book your restaurant, but with one point of the price. So yeah, I think there's a lot of competition heating up.

And five works here is here to support the open source model as much as we can. And having it be fair to say that because you came from Meta ultimately, and they had been working, well, the things I'm trying to look at, like the timeline here, which is Lama. And was that really a thing internally when you were still at Meta? Or did that only really come out? Or like, what are you already working on something Lama related before you left?

And that got your kind of brain jogging on these open models or not really? Maybe the closest answer is kind of, so when I was working at Meta, the Lama or like the large language model program wasn't really that well funded. So it was different from where OpenAI was doing the Yolo run for like GB4, and with like, I forgot like thousands of 8100s, there was no, nowhere near as much of a commitment in Meta back then as of now. But I worked a lot on recommendation system models that had transformers. And there was a clear sign where you just throw more compute at the problem and the model just gets better.

The RUI for those increased compute may not be worth it at the same time, there was a clear friend. So while people were talking about scaling models for language models, I guess we were sort of like seeing the scaling law for recommendation system models in real time as well. So the migration between the tiny 17-watt P6 into the 200, 300-watt, 3100, there was like a clear payoff for that. And then keeping on migrating to 8100, there was also a clear payoff for that as well. So yeah, I guess I was working in related fields and seeing the scaling law working but in a different way.

At the same time, I think the commitment or like the belief in open source models right now seems obvious, but I think it was definitely contrary to three years ago. People forget the models we had back then were OPT, Lama 1 and Falcon. And if those models could hold a three-term conversation, that was already amazing. And people also forget three years ago there was no function calling. And it's like we also worked on open sourcing function calling models, which wasn't really a straightforward thing back then.

So yeah, I would say it wasn't straightforward at all to say, hey, Lama is here and it's here to say, in fact, it's not here to say anymore. But sort of like the belief we had, because we worked on open source software for so long and sort of like to believe that the models are more like software rather than some kind of hardware kind of setup, I think that sort of leap of pace was important for us and I think carried a lot of weight even in today. Yeah, and I realize we're sticking a lot on models generally, but I think this is quite interesting just for a second. I mean, I think just to be clear, a lot of these open, well, first of all, just to clarify, you mentioned open source models. I mean, is there a distinction here open weight?

Like are you using interchangeably with open weight or? Yeah. I am using it interchangeably. And for all the people out there who really understand the distinction, I think they will hate me for these terms interchangeably. I'm more of an attitude where like, I don't really make those distinctions when the goodies are still flowing to be honest, as long as the final artifact is produced, I think it's good.

But I have huge respect to both like the Omo team and the Nemo Tron team. The Omo team is like L and AI, they have funding from NFS as well as I think other sources. And they really publish all the results as well as the other intermediate artifacts. I think that's like a really good foundation of work for everyone working at field. And then I think a video also is really committing to keep pushing open source models.

And they also publish what they train and code or their recipes to benefit the community. Yeah, absolutely. So yeah, I guess to keep some of our listeners happy, when we talk about open weight models, which is a lot of, I guess, what far X is working with. If I look at most of those names, or especially on the far X platform site, but I think just known generally, they're mostly coming, would it be fair to say from Chinese companies and Chinese tech? Is that good question? Yeah. Yeah.

So I mean, I'm someone who's based in Asia. I fully appreciate just the amazing talent and innovation that comes out of this part of the world. And I think it is helpful to maybe just talk through, have you hit any bumps with the fact that you're a US company and then pushing models like this from that part of the world? Or I'm just curious like how that looks because it's something that also gets talked a lot about maybe in the news even, you know, especially when DeepSeek landed, if you want to call it that. And I think all that's kind of settled down to some respect.

But what was your thinking around all that when putting these models into fireworks? Yeah, I think that's a good question. So I would say 90% of our customers don't really care about the origin of the model, especially if they're doing fine tuning, because they will be sort of like running these models in very specific environments. For example, for coding environment, often you don't really care about the political preference for these models because you're just writing reacts, you know, like it doesn't really matter as much. And then definitely for certain customers who are more like consumer facing, they have much more aware of the origin of the model.

And we are more constrained to serve things like e-motron, GPS, these kind of American models. I do think fireworks here is here to help support our customers. And we don't really judge why are they making these distinctions. We just here to help support them as much as we can. Mistro people also do a great job.

And they also publish various raw models and sometimes also sort of mistro models. But yeah, like I would say like one thing I keep chatting with my colleagues on for like, it's like a year-old and GPS is surprisingly competitive today amongst all the American models. So huge respect for open AI as well for open sourcing the model back then. Yeah, the new models coming out every day from NVIDIA, from Ommo. Those are also getting more competitive.

And I do think at the end of the day, given like the same data set and the same model size, the results will converge. And then the data set should converge as people start sharing more and more of those intermediate artifacts and recipes. So I bet the American models will catch up this year very quickly. I honestly feel like this is a little secret out there today. I feel like there's so many conversations where like people think they have alpha and then they sort of exchange notes and be like, oh, I didn't realize some other people are doing the same thing.

But yeah, I do think like a lot of things will converge and American models will be very competitive. Yeah. And I guess as like, Fire says a platform, which we're going to get into in more detail in a second. But I guess part of your offering is to help your customers understand which model would suit their case best. Because I think at least to me, when I was doing a bit more hands-on, top ones ago, I think trying to pick through like which model why even the, okay, it's kind of the same model by once like, you know, 4B and 1's 16B or whatever, you know, like this is, I'm sure where you're actually able to give more direction and input to your customers, given the size of them.

And like, this isn't just a, you know, you can't kind of make the correct wrong decision on these things. Yeah, that's a good question. How we talk with our customer on when to use which model, depending on the use case quite a bit. I would say for probably a third of the use case, our customer is more sophisticated than us. They have run the evaluations in house.

They explicitly come to us and be like, this is the model you need to serve. Don't argue with me. I will pay you for this, this is serve this model. And then one third is more somewhere in the middle where they kind of know they're going to pick between two or three models. They mostly need some judgment call on the cost of serving on like whether the scales on our platform.

So they run the evaluation locally and then they just need to know the cost or the scalability of the setup. And then the last one third is where the customer really looking for us for advice on which model to use and which use case. And in those cases, we share our evaluation results internally and try to paint a full picture on like, hey, this model is much better coding. This model is much more malleable and a good starting point for reinforcement learning. And then just show them the data point and give them the judgment call.

To answer that question more concretely, for example, a lot of choices on one you want, depending on what you want to do. For example, like, Kimi is a very big model. So it's very easy to do a reinforcement learning on. Thank you. Easy as in like infrastructure, it's difficult, but it's easy to push results out of big model rather than trying to like a small model and try to get the small model to be smarter.

At the same time, the evaluation initially maybe was. So then it is not going to be obvious to our customers. Hey, like, hey, Kimi is a much better starting point than a lot of other models. For example, when some of our customers are just looking to serve open source models and not customize, then like, for example, if it's like a coding use case, then GOM and minimax both are great. And then just depends on the cost of thickness, depending on the use case.

So we help try to paint a full picture to our customers and still rely on them to decide what's the best for them. Yeah. No, it's very interesting. I maybe hadn't appreciated what you said. So towards the beginning, a good chunk of people come and just say, this is the model already figured out, just run it for me.

So let's get on to the just run it for me a bit. So you have quite advanced info as it has to be running running all this. I believe you have things called or like a piece of the puzzle called fire attention, for example, like maybe could you walk us through what is that? Let's just kind of start there and we can go through some of these interesting bits of the fireworks stack. Yeah, we set up our own kernel and house for a few reasons.

One, a lot of kernels in the open source world are not numerically pressing in the sense that it will work. It may not work in the way that you expected or it may not be working because of the reason you think it's working. So this boy is new kernels coming out every which way we try to make sure our customer when they use fireworks to serve the model. They don't think about all the complexity for like all these weird kernels. They just understand that we provide the best trade-off between quality and speed.

We'll push as much as we can on speed, but not compromise on quality. And a lot of our customers appreciate that. That's sort of like the first reason why we set the kernels in house. The second reason is we have a very, I would say, expensive but important commitment to do multi-hardware. So we work with AMD team quite a bit to make sure we can properly support their hardware.

And not everyone is willing to sync the time and effort into supporting different hardware. So yeah, we set up those AMD kernels in house as well just to make sure that we can serve on different hardware and try to find the best price for our customers. And also maybe the last reason I can go through on these internal fire attention kernels is because we do a lot of reinforcement learning workload, making sure that the training inference consistency is minimized is very, very important. You see a lot of fancy algorithm. It did publish every day.

There's like so many variants of GRPO at this point. I bet if you put two random letters in front of it, there's probably a paper for it. I would not know where it came from, but there's probably a paper on it. But all three letters for certain people. From our observation, those algorithms are important, but not as important as controlling the numerics and making sure the numerics are lying across training inference.

So you see a lot of headlines on, hey, like it's like a reinforcement learning stack where we use like trainer from one stack and inference from another stack. And then I get you just work for a big model from our experiences far from it. We really take it seriously and make sure that we align the numerics across training inference for those kernels. That's why we have to have those in-house and that's why we have to spend all the sweat and tears. So then our customer don't have to think twice and be like, hey, like did these people do their job and is this hard run not going out because the numerics sucks.

We try to really make sure our customer trusts us to make those happen. Yeah. And then something called speculative decoding, which again, maybe context-wise, I believe cursor built their fast apply feature on this API. Could you maybe just like speak to that? Yeah, speculative decoding, I would say at this point is a pretty well-known concept for many serving stack.

It's a set up where you have a small model that try to guess which token the big model would like and just ask the big model, do you like these tokens? Yes, let's spit them out at the same time. There's a lot of interesting research in this area. I think recently there's like using different model architectures to do the speculation. We also spend a lot of time doing those research in-house.

Yeah, speculative decoding is definitely also workload dependent and making sure that we can train different form of speculative decoding models while the data is coming in to make sure that we keep up with the change in distribution in our customer's data. Those are also very important as well. I think one other thing that people often don't appreciate enough is that training a e-model is often like training a large-language model, or like a training a small-language model. In the sense that data quality is important, training infrastructure is important, dev efficiency on this stack is important. There's a lot of nuances on how we can support our customer with good speculative decoding models, and we spend a lot of time and effort on that, making sure that when they bring an open source model that they find you, we can quickly train a really good speculative model for them.

Yeah, because I was going to ask the idea of the effectiveness of draft model and the target model, and I guess you're hinting that the target model is often actually the customers fine-tuned model, and far as is needing to pair that speculative draft model. Absolutely. A lot of our customers are very sophisticated. They may have most of the things like 70-80% away there, and we are here to deploy the model and often part of scaling the model is to train the speculator for them as well. Yeah, nice. And then you have something called 3D Fire Optimizer. So yeah, I'll just let you take that one.

Yeah, I think it's an interesting name when we came up with the concept. Sounds like something I find in gaming. So yeah, that's quite interesting. I think conceptually it's very straightforward though. It's a database of all the previous performance optimization results we have, as well as predicted results of what the customer workload will be, because there's so many dimension to do tradeoffs.

For example, like workload patterns, hardware types, cache rate, many, many different variants to deploy the same workload, and it's important for us to be able to scale the engagements for our customers in an automated way. So we have a database for these performance optimization techniques that our automation can make use of and make sure that we can get back to our customer and answer very quickly. Yeah, oftentimes, how to set up the workload correctly is like 50% of the work. And 3D Optimizer is an in-house stack that helps us get there. Nice. And then when you have mentioned hardware, I think, especially in relation to the Fire Attention kernel, you are a sort of hardware agnostic, if you like, shot and by agnostic.

I mean, we're talking basically two providers because that's kind of what it boils down to. But yeah, I mean, what is your thought process apart? Like, I mean, let's assume the cost comes into it somewhere, but cost plus what else sort of is what you're looking at when you're deciding in video and AMD, basically, which I believe you run both. And just to be clear, both are investors. So I don't want to...

Gotcha. We love NVIDIA and work a lot with NVIDIA just to be clear. I do think when we run your business, we make sure we maximize our customer value. And in the world, we maximize customer value. It's definitely a little bit weird to be locked into one hardware vendor.

And I really think this year is honestly not so much about loyalty to NVIDIA and whatnot, because NVIDIA is already producing chips and its assets again. It's just that everyone else is buying them up. So practically speaking as well for multi hardware vendor, oftentimes it's all about supply chain reliability, making sure that you can actually buy the cards when you have the money. And at any point in time, there may not be NVIDIA cards available and you have to buy AMD cards. Available as in like available at reasonable price, because of course, if you are willing to pay like there's always someone willing to sell you, just at the premium will be very, very high.

And honestly, while I was working at NVIDIA, because I worked on P6, I was also in conversation with all the procurement process. Of course, I'm not the one negotiating a contract, but because I was on HasTeam, I was providing all the inputs on how many CPUs and GPUs we need. And often in those, even very early on in those conversations, it was AMD and Intel for CPUs back then. And it was important that even just to have to do a supplier strategy that works surprisingly well. So in case where people think two suppliers may or may not be enough, oftentimes it is enough.

It would be great when you have three suppliers, that's really when you get to benefit. But even having more than one is very, very helpful. And that's ingrained into how I think about this process and that's why I'm multi-hardware is so important for us. And I do want to be honest about it, that there is a lot of work. And there's always trade-off between working with AMD for five attention versus working on other stuff that we can serve out customer better.

And at the same time, we really believe that the AMD investment will pay off. Yeah, I think it's really helpful. I'm sure for us to hear just sort of actually one of the things that you have to consider when running something like this. And obviously investors come into it as well. I think that's what people forget about it as well.

It's the most companies in the space have investors and that will often have a bearing on some direction at some stage as well. So I think that's really helpful. Let's maybe sort of move on to EVALS. I mean, most of our list of ways should be somewhat familiar with the concept of EVALS and why it's important to anything, especially running your own or wanting to fine tune your own models, but even just picking between models as well. And I think at least as a company, you've kind of made the statement which other people agree with, which is one of the biggest barriers to this whole AI ROI problem or mystery to some people.

For those just ROI return on investment. So is the money being put into your AI within a company? Are you guessing something that effectively is larger than the investment you've been? Very simple. And Fireworks is saying the biggest barrier to that isn't cost as such, but it is defined good.

What is it that you think good is? So, yeah, I mean, could you just walk us through what does Fireworks do in this area? Do you help your customers do EVALS or just walk us through kind of what that looks like? Yeah, Fireworks investment on the EVAL is I would say like 70% on the infrastructure side and 30% on the consulting side. So the 70% on infrastructure, we have a project open source project called EVAL protocol, where we help people author EVALS for reinforcement learning.

Oftentimes when I ask customers whether they have EVALS and recommend them to write EVALS and whatnot, my pitch often is even if this engagement falls through, the worst outcome is that you'll have better EVALS. Which hopefully isn't a bad outcome because there's so much innovation coming out of these front-tier labs, close source models even, how do you decide which one to use? It's like every day is something new. Okay, every day is overstatement, but like every week is something new. And this is even like the most straightforward case.

Let's say you love EVALS and you want to use XS model. How do you know when it's appropriate? Like you can't even pay EVALS unless you have the EVALS. So it's very important, I think, for a lot of customers to realize that these are assets and Fireworks is here to help you build up those assets. And as soon as people have EVALS, the gap between using that EVAL to evaluate model to the gap to use those EVALS to train in reinforcement learning, to train your own model, is very, very small.

So it's very helpful for our customer to first be able to pick different models and then second, once they are comfortable, start training new models on Fireworks with reinforcement learning through EVAL protocol, it's something that we've seen repeatedly happen over and over again. At the end of the day, if you know how to evaluate your model, you have all the power. You get to decide which supplier to use in what setting and when. And that power is very, very important to a lot of our customers. It's just that some of them haven't realized how important that power is.

The other part is also we work with some of the large customers to help them set up the EVAL as well. Because in certain settings that it's not practical to leave our customer hanging and ask them to author the EVAL themselves, we have a lot of no-house in-house as well after all these engagements and we help our customers really use our no-house and author the EVAL themselves as well. And you have an EVAL framework that has actually been open source, is that right? Yeah, it's called EVAL protocol. It is focused on helping people author evaluations for reinforcement learning settings.

So you focus on writing the evaluation itself and then Fireworks can help take care of the rest for doing row-outs on inference, passing those row-outs to the trainer and making sure that the reinforcement learning graph goes up into the right. It also helps with observability for like, which traces went wrong and why and what did the model trip on? It is often understated on how much people focus on sort of like the fancy algorithm inside of RL, but in practice a lot of times the observability is very, very important. Making sure that you have a open source SDK that help people hook into any part of the infrastructure however they want and they need their assurance because unless they can hold the code themselves, like they don't want to touch it.

Making sure that all the observability stuff can be set up so they can see the environment's behavior at any point in time. Because oftentimes the RL problems are coming from the environment itself, like something broke in the environment and you want to fix the environment to continue on training. So those things are all very, very important and eval protocol helps you with all that. Yeah, and I encourage anyone interested in any of these topics. You guys have a really good blog and something was published on their traces are all you need to practice to rank LLMs.

So this is in reference to production trace data and being sort of a better signal, say than benchmark suites and like there's probably a cap somewhere so I think just maybe if you briefly unpack what is that about? Yeah, I think when we talk with our customers a lot of them feel that it's intimidating to all the evaluations. A lot of them didn't start out as machine learning engineers but that's the beauty of this wave of innovation. A lot of people coming in from product backgrounds who has the ability to clearly articulate what is good and what is bad for their customers. And oftentimes I feel like that's under appreciated in a lot of settings because honestly if you can't clearly articulate what is good and what is bad, you're 90% of the way there.

The only delta is then now you need to take traces from your production workload and run a language model through those traces with your articulation. And oftentimes I think in a lot of settings like our customers tend to have more elaborate setup for the evaluations and what we find is honestly if you can't articulate it, you're 90% of the way there. There are definitely special cases where for example we have an example of SVG agent where you still need to render the output through a Chrome server to be able to get the screenshot to even use your articulation of what is a good SVG. At the same time, honestly like Opus 4.6 you say the word, it comes out the other end. I don't even know what happened in between but it works.

Then the only important bit is can you pull your traces out of your production system and then can you articulate what is bad. So we're going to move on probably so the final, fairly, meaty topic is just on reinforcement fine tuning, something that I believe again, Fireworks helps people figure out. So supervised fine tuning has been around a while. So what does reinforcement fine tuning like? Why does that change things like fundamentally, which is a mind-attending of it's a huge shift.

It's not just a small incremental change, it's kind of a step change in this area. Absolutely, absolutely. Yeah. Maybe I can start with how it changes the industry and specifically how it changes fireworks. I would say reinforcement learning is a new lever that the industry found while the pre-training sort of like a free writing slowdown.

You definitely see a lot of articulation from like Dario saying like, hey, like anthropic does not see improvement slowing down and whatnot and pre-training is still giving a lot of gains. At the same time, my current understanding is that you need sort of like an exponential amount of compute to get the straight line going. And at some point the money aspect will kick in, like reality will have to kick in. Even if part of reality doesn't kick in, electricity part of reality will kick in. So there are limits to how far you can push these ideas.

And reinforcement learning was a new paradigm that the industry found to keep the scaling going. And even for relatively small models, if you can push RRL, you can get really good results out of the model. And I think the other thing that people don't tend to talk about for reinforcement learning is that it really consolidates your evaluation as an asset. Meaning that the same evaluation on the same environment can be used across different generations of model without significant changes. This is different from SFDE datasets where depending on what's needed for this model, you may need like more supervised functioning data to sort of like push the model to a certain direction.

And that is definitely important work that a lot of frontier labs people are pushing in every day to make sure they curate better and better supervised functioning dataset. At the same time, when the models are really good, you have to throw certain things out, just to even like not confuse the model. Whereas for like asking the model to do an Excel spreadsheet is always going to be valuable, no matter where you start. Because you always want to make sure that it knows how to read an Excel spreadsheet and knows how to manipulate a spreadsheet. So those assets are sort of more enduring for these settings.

And that's why I think so many frontier labs are investing so much money into these environments. Specifically for fireworks, it is very important because it now finally unlocks the customization loop from a software engineer directly to a tune model. Previously to do supervised functioning, the conversation often goes like, oh, do you have a team of MLEs in house who knows how to work with daylablers? Because these MLEs need to have some experience managing labelers and making sure that they can sort of clearly communicate with another set of human that they never worked with before. What is good?

What is bad? And what I'm looking for. And it also takes a few iterations because oftentimes these data labeling companies will have to assign you certain set of people repeatedly just so that they don't lose the context. And so you need to have quality control on the other end, making sure that the supervised data set, the supervised functioning data set are consistent. And it will continue to be what you're looking for at the 1000th label.

That is a very tedious process, a very difficult process for many, many people. And the process also breaks down when it's long context. Because honestly, as a certain context, I really don't think I even understand what's going on. Like it probably takes me like an hour just to read through all the conversations to figure out where we're wrong for me to edit the conversation to make it work correctly. Whereas for reinforcement learning, as long as you have a product manager who can articulate what is going on bad, they will be able to author a language model as a judge snippet and then send it to fireworks and be like, hey, teach my model this, right?

Everyone else is out of the loop. And then we can sort of bootstrap this very, very quickly. And as the coding model gets better and better, I really think that it is a lot more we can push out of these coding models to get reinforcement learning more automated. I think that would be like an interesting topic to dig into more this year. Yeah.

And I believe it was for Sal used as a use RFT with you guys and that sort of come out with, you know, putting numbers on it, something along the lines of 40x faster code fixing and with better outputs. So that's like sort of interesting use case for RFT. Yeah, absolutely. Yeah. And we have a lot of smaller customers and you know, who benefits from RFT as well.

I would say, for example, the cell has like a very, very strong engineering and producting. Those things are exceptionally well fit to do reinforcement learning because we're just with like two or three people, they can totally align what's good or what's bad and they can just go. Nice. Yeah, that's a really kind of powerful image there. So we're sort of starting to wrap up, but something I did want to touch on is this competitor as a vendor problem that we've seen in a play out over the last, I guess, 18 months, especially.

So you know, some examples on SED news, which is our monthly podcast that myself and Sean Falconer do we've got a few of these. That's quite interesting. So the fact that anthropic effect, we cut out windsurf and then, you know, opening eye launching codecs, which computer cursor, you've then got things like Google search, you know, versus perplexity in some respects. So if you're the vendor, like how are you thinking about where this risk for you potentially lies or doesn't even so, yeah. I honestly think we are such a small player in this market and the market is so early still that I don't know if the competitive pressure matters yet.

Yeah, I keep telling my colleagues, like, if we don't do single digit percent of what NVIDIA is doing, I feel like we're doing something really wrong. And we're not there yet. So well, NVIDIA is I forgot what the exact number is like 400 billion this year. Something like that. Yeah.

Yeah. I would say like probably before, like we do a few percent of what NVIDIA is doing. I really don't think that we will run into those hard constraints on like competitive vendors and whatnot. I do think I can be more helpful from this answer in the sense that like when our customers are thinking about fireworks or what they're looking for specifically. And honestly, I think at the end of the day, it's mostly about trust.

Trusting that we get the numerics right so they don't have to figure out all the details, trusting that we got all the serving details right so that function calls can happen properly, a constraint generation happens properly. The trust on that we set up our reinforcement learning correctly so they don't have to like do the numerical debugging themselves. And I don't think a lot of those are competitor related in the sense that a lot of our customers are making honestly a lot of money and they just want to make sure that we handle those complexities for them. I do think in the fast evolving field, maybe I should think more about the competitors landscaping well now but I really don't think it matters as much yet.

And it's more about helping our customers make as much money as they can at this stage. Yeah. Exactly. And you know, you work with some very big names. So it's sort of anyone potentially smaller coming along and that's a trust piece.

I think that's where you've got some of the bigger names that kind of back here at the stage in terms of if you're able to work with the cursors and the for cells that surely says something. So yeah. We're coming to time but I think for anyone out there, a developer or anyone maybe slightly slightly high level because I see versus business is kind of the same thing but some of these are not hands on keyboard. But I mean if someone wants to get sort of started with fireworks, like what's kind of the best path there. Install open core hook up Kimmy or fireworks with open core and honestly Peter is amazing.

I watched some of these podcasts like the author for open core. Yeah. He's just joined open AI at least a couple days ago before this recording. Like I don't know how much he's paid but he definitely deserves it. It is surprising honestly how fast everything is moving still.

As soon as you expect things to slow down maybe a little bit like all the models come out in the last two weeks, right? And then like open core gets their fame to acquisition in like a few weeks. It's the things are not slowing down. No. And I do think that for anyone who's listening any amount of effort in this area I think will pay off in anything.

It's like if it's not open core and like I don't know why code something because I really think a lot of the difference that I'm seeing for some of customers just that they are two month early. And the fact that they are two month early makes the whole difference. Yeah. Yeah. So yeah, I mean, thank you so much for coming on today, Benny.

Is there anywhere that you personally are on like the people can follow you? Are you on X or anything like that or? I guess I'm old. I would say Twitter. Yeah.

Nice. I would still say Twitter. I'm just the number time I hear X formerly known as Twitter. I'm like, we're going to pick one that way. Cool.

So you're on Twitter. What's your handle on Twitter? Funny can. Oh, cool. Okay.

Funny Chen. Nice. Awesome. Well, yeah. Thank you so much for coming on.

I've learned a lot as well as yes, it's kind of really awesome to see a company in this space and maybe you weren't analytical enough about when you left Meta and started this. But I mean, this sounds amazing and you've obviously done a huge amount of work in this space that someone else hasn't done. So I think that's that says already. So yeah. Thank you for having me.

Thank you. Thanks a lot. Okay. I hope we get the catch up again in the future. Thank you.

Thank you.