NATS and the CNCF kerfuffle (Interview)

Well, friends, this is your favorite podcast, The Change Log. Yes, we talk to the hackers, the leaders and those building the future of software. Today, we're joined by Derek Collison. Derek is the co-founder and CEO of Synadia. He's also the creator of Nats. For the uninitiated, Nats is a high performance open source messaging system designed for cloud native distributed systems. And today, Derek walks us through what it took to create Nats, where it came from, how they eventually decided to form a company around Nats called Synadia to offer this to enterprises and more. Derek also shares the behind the scenes of the recent kerfuffle between Nats and CNCF.

A massive thank you to our friends over at Fly.io, our partners, our sponsors, our friends, and the home of changelog.com. You can launch your app in five minutes at Fly.io. Okay, let's nuts. Well, friends, I'm here with Terrence Lee talking about what's coming for the next generation of Heroku. They're calling this next gen Fur. Terrence, one of the biggest moves for Fur in this next generation of Heroku. It's being built on open standards and cloud native. What can you share about this journey? If you look at the last...

half a decade or so, like there's been a lot that's changed in the industry. A lot of the 12 factorisms that have been popularized and are well accepted, even outside the Ruby community, are things that are, I think, table stakes for building modern applications, right? And so being able to take all those things from kind of 10, 14 years ago, being able to revisit and be like, okay, we help popularize a lot of these things. We now don't need to be our own island of this stuff. And it's just better to be part of the broader ecosystem. Like you said, since Heroku's existence, there's been people who've been trying to rebuild Heroku. I feel like there's a good Kelsey quote where we can stop trying to rebuild Heroku. It's like people keep trying to like build their own version of Heroku, like internally at their own company, let alone the public offerings out there. I mean, I feel like...

Heroku's been the gold standard. Yeah, I mean, I think it's the gold standard because there's a thing that Heroku's hit, this piece of magic around developer experience, but giving you enough flexibility and power to do what you need to do. Okay, so part of FUR and this next generation of Heroku is adding support for .NET. What can you share about that? Why .NET and why now? I think if you look at .NET over the last decade, it's changed a lot. .NET is known for being this Windows-only platform. You have WinForms, use it to build.

Windows stuff, double IS, and it's moved well beyond that over the last decade. You can build .NET on Linux, on Mac. There's this whole cross-platform open source ecosystem, and it's become this juggernaut of an ecosystem around it. And we've gotten this ask to support .NET for a long time. It isn't a new ask. And regardless of our support of it, people have been running .NET on Heroku in production today. There's been a mono-built pack since the early days when you couldn't run .NET on Linux, and now with .NET Core, the fact that it's cross-platform, this .NET Core built pack that people are using to run their apps on Heroku. The kind of shift now is to take it from that to a first-class instance. And so what that means for Heroku is we have this languages team. We're now staffing.

someone to basically live, breathe, and eat being a .NET person, right? Someone from the community that we've plucked to be this person to provide that day zero support for the language and runtimes that you expect and what we have for all of our languages, right? To answer your support and deal with all those things when you open support tickets on Heroku and kind of all the documentation that you expect for having qualities of language support in the platform. In addition to that, one of the things that it means to be first class is that when we are building out new features and things is now one of the languages as part of this ecosystem that we're going to test and make sure runs smoothly, right? So you can get this kind of end-to-end experience. You can go to Dev Center.

There's a .NET icon to find all the .NET documentation, take your app, create a new Heroku app, run, get push Heroku main, and you're off to the races. So with the coming release of Fur and this next generation of Heroku, .NET is officially a first class language on the platform, dedicated support, dedicated documentation, all the things. If you haven't yet, go to heroku.com slash changelog podcast and get excited about what's to come for Heroku. Once again, heroku.com slash changelog podcast. Today we're joined by Derek Collison from Nats and Cinedia. Derek, you were on one of my favorite episodes of GoTime years ago. A long time ago. Challenges of distributed messaging systems. Yes, GoTime is no longer being produced, but the feed is still out there. The episodes are still out there. Please go listen to that. I learned so much from you about these things. So welcome to the ChangeLog. I appreciate you guys having me on. Thank you. Literally years in the making. Jared, you remember going to QCon a while back? I do. The circus. That's right. The circus in Chicago. A couple years back. We met the Cinedia folks and the Nats folks. I think it was...

not quite early days for Sinedia, but like it was early-ish for announcements, maybe even marketing. Derek, you can kind of speak to this, but remember that circus and meeting those folks? I do. Yeah. I don't know if I was there. What year was it? Because Sinedia was formed at the end of 2017, right? On the heels of Epsera. But I don't remember. I don't think I was at the one in Chicago. Many years after that. 2023? Okay. So not too far back yet. Yeah. Until in 2023. Yeah, you weren't there, but we had a few friends there that were listeners as well, and we became to know each other. So it's been nice knowing you all. I've known about Nats for a very long time.

When did NAS start? I mean, it's been around forever now, right? Yeah, NAS was kicked off as part of a system that I was building at the time at VMware called Cloud Foundry, right? Which was my spin on Heroku for the enterprise back in, you know, 2010, I guess, you know. And at the time, you know, I had come off of time at Google, spent about eight years there. But prior to that was at a company called Tipco, which before that was called Technicron. And we had been designing lots of messaging systems that then were used to scale up distributed systems for customers and clients. So I just had a natural way of...

building systems like command and control, telemetry, eventing, using these types of constructs. But TIBCO was closed source, right? And so we couldn't use that. So we reached out to Alexis Richardson, who was running a company that was kind of in charge of Rabbit and Cube, which at the time was kind of a... you know, the major open source player from a messaging construct. And, you know, Rabbit and Few is still part of, I think, VMware slash Broadcom now, part of Cloud Foundry or whatever that has turned into be. But at the time, it was more of an enterprise type of technology. What I mean by that is that if you ask it to do something, it would bend over backwards trying to do it for you, even at the detriment of everyone else. And, you know, we had some challenges there which led to the creation of NAS, which was, you know...

Kind of like a dial tone and think of it like the electricity in your house or your condo or whatever. It's, you know, I can plug in a really bad blender and I might blow out something in my own place, but hopefully it doesn't take out the electric grid, you know, for the whole neighborhood. And so Nats was born on a very simple principle of just do a couple of things very, very well. And yeah, it was the proverbial weekend project. So I left on a Friday and Monday, you know, we plugged it in. And believe it or not, applications that were written for that initial version, you know, the protocol that speaks between clients and servers still run today.

You know, so that you can just fire up something from the cloud boundary days and just run it against a modern NAT server and it just works. So we've tried very, very hard and we're not perfect by any stretch, but we tried really, really hard to make sure we were always backward compatible with the protocol. Because we knew the protocol, even though it was designed very, very quickly and only a couple of days, that was kind of the really big thing from a technology standpoint there. I'm not a distributed systems person, but NATS feels a lot to me like tail scale, but a different layer. Is that how you would describe it to somebody who doesn't?

build distributed systems or have sort of the depth and knowledge you might have? Yeah, I'm a huge fan of TailScale and, you know, the folks that were either at Google or involved in the Go programming language that are over there. But it's an overlay on top of the constructs that we still know, which is IP-based addressing. Almost everything is point-to-point. Most, I don't know about TailScale, but most cloud providers, right, trump UDP broadcasts or multicasts. And so it's, you know, a point-to-point location-dependent type of a system that now has surfaced up over a very secure overlay with WireGuard.

What NASA was trying to do was kind of change the notion of what we call intelligent connectivity. And specifically when, you know, folks that are listening say, what does that mean? fairly simple from an abstract standpoint, which is everything is location independent, right? And so one might say, oh, well, things are kind of location independent today, but I would argue that there's a lot of unnatural apps going on below the covers to get that to appear that way. Load balancers, GSLB, any cast DNS tricks, right? So the biggest first one is location independence. And the second one is instead of one-to-one request reply is the dominant pattern, HTTP, that's all you can kind of do, SANS, SSE, and WebSockets. We are end-to-end and both push and pull.

And the push becomes very, very interesting for certain use cases where instead of keep asking, hey, have you updated your temperature? Have you updated your temperature? Just tell me whenever you update your temperature, right? And for scalability distributed systems, just those primitives from the connective layer become pretty powerful, right, at scale. So NAS really took off, right? I mean, a huge adoption. Can you tell a story? Yeah, so, I mean, NAS was built specifically to power those subsystems in Cloud Foundry, right? And I had no desire to do anything more than that. And so it was, for at least probably the first 18 months of its lifetime, specifically geared toward just supporting what we needed in Cloud Foundry from command and control, telemetry, eventing, things like that.

When we started the company AppSera, which was trying to do a redo of Cloud Foundry on some of the things I felt I didn't spend enough time and energy on, security, governance, federation, all these different types of things, we rewrote NAT in Go. NAT was originally written in Ruby. That's what Cloud Foundry was written in. Still love the language. Not great to deploy things into production that are Ruby-based, although I know DHH has done an amazing job of trying to simplify that. But back in the day, it was...

There was a lot of moving pieces to deploy the same. And we were starting to hit some performance issues with very, very large systems needing more performance there. So we had, at least at the time, we were looking at either node. JS, JavaScript, or Go, which was very brand new. I think when I started playing with it, it was 0.52. And I felt that the speed advantage with the concurrency, the IO mechanisms that they had, but mostly the fact that you could build pretty much a static binary, meaning shipping and production was just move one binary, was the direction that we took. And it was a great direction. I mean, I think Go has grown up and there's people that like it and people maybe that don't like it, which is totally fair. But we've gotten a tremendous amount of knowledge out of it. And at the time, looking back, it was the absolute right decision. Now, going forward, we're looking at massive, massive extreme scales that some of our...

customers are pushing us on, we're starting to bump into some things in Go that we need to address. And whenever I'm in a language and we start to do unnatural acts to get around certain things within the language, that's when we kind of say, all right, let's take a step back and are we at another inflection point? And we're not there right now, but with the introduction of Jetstream, which is the data persistence mechanism that Nats uses that was added on about four, four and a half, five years ago, I guess, we're starting to see a lot of friction with the way Go does. Disc I.O. and things like that. And so we're trying to figure out what makes the most sense as customers are saying, hey, remember when I said a million was good? Then I said 10 million was good. Now I need a billion. You know what I mean? And, you know, could you do that in the next two years type of stuff? And those are great problems to have because they're hard and they're fun to work on. But it challenges you to think outside the box sometimes. What are you looking at?

You know, I don't do it anymore. I'm sorry to say that I dropped the ball, but I used to learn a new language every year, you know, and I would just write an ask client for it to learn it. And so I learned Rust and there's certain things in Rust that were really, really attractive. But to be frank, I just didn't enjoy programming it. When you were doing a low-level library, if you were consuming other things and building like an upper-level app that was kind of a composite of lots of different things together, especially the packages or crates were really, really well done. It felt really, really nice and clean, but I was at a low-level library trying to do some things and we got it and it's one of our most popular clients.

I like, again, I'm older probably than the majority of the demographic that's listening to the podcast, but I like languages that I can hold most of it in my head. You know, I've never been a big IDE person. I still program in Emacs. I do use AI and I ask, you know, Claude or ChatGBT questions and things like that, but I don't have that co-pilot wind server code mentality yet, but I do like Zig quite a bit. But Zig right now, you know.

Andrew was trying to take on some really gnarly problems, which I have the utmost respect for, especially the colorless async. And that kind of tied in a little bit to event loop structures, and they kind of got pulled out. We need some sort of direction there before we finalize a zig client. But I did write one. I have one that kind of uses threads and does some interesting things.

And then Joran from Tiger View. I've known him for a while. Full disclosure, I'm a minor investor in their company. Love distributed. All things distributed, I think, was their conference. And really been watching them and Bunn and, of course, Mitchell with Ghosty on what's possible. But each one of those event loops and IOU ring interactions is all kind of a little bit snowflake. Not exactly. But I think Andrew is now thinking through a way to...

Uplift the abstraction of IO that could tie into an event loop as something similar to allocators in a language where you have to pass it into a package, right, or to a constructor. Say, hey, if you need to use IO, please use this essentially interface, right? And so I'm really interested to see where that lands. But I like Zig. Believe it or not, I like Mojo, even though I'm not a Python person. I'm a huge fan of Chris Lattner's and what he's trying to do there. I think it is impressive. But Zig's probably number one right now on my mind. But again, we need a little bit more direction, not from the server perspective, but if we wanted to do a client on.

How do we interrupt between all the different event loops? You know, whether they reincarnate the one that they have in the language, but then Mitchell's with XEB, I think. And then of course, Bund has one that's, I think, a pull from Tiger Beetles and Tiger Beetle has one. So that'll be good when we get there. Well, completely different problem space, but you like Chris Lattner. The Ladybird folks are probably picking up Swift from C++ to Swift. Have you considered Swift? I haven't. We built a Swift client. I wasn't involved in it. Again, I dropped the ball on my learn a new language every year. I actually did program a little bit Swift. You know, I think the interesting part about where Zig is, is that, you know.

We can control direct IO, meaning we could go through IU Ring on Linux if it's, you know, they're applicable or whatever. And it allows us also to control more precise layouts of the data structures and, of course, memory, right? So anyone who's been programming Go for a long time and has a tremendous amount of investment in a big code base, right, you start to see things pop up. It's like, oh, avoid JSON at all costs because the built-in JSON marshallers, you know, user reflection can use lots of memory. The network IO and the disk IO subsystems in Go are very different. And the disk IO subsystems are pretty much...

you know, a native thread, you know, that kicks back into your Go routine that says, okay, it completed, the right completed or whatever. And I think what we need is a lot more fine control over those things. So what I've been looking at is the adoption or the ability to introduce IO Uring in a meaningful state. I think Zig's done a pretty good job there. Maybe not within the language per se, but with, you know, folks that use it, you know, Tiger Beetle and Bun come to mind. And so that one's kind of a big one. The data structure layout, the garbage collection and taking control over disk IO.

Right. Which, you know, Jetstream, we've got folks that are coming in and says, oh, yeah, I need to do a million stored messages per second, replicated three ways, and then an async mirror over in a different region and a different cloud provider type stuff. So we can abuse the disk, you know, the storage subsystems pretty, pretty well. And we might be getting close to kind of everything we can accomplish with Go. I'm not saying that that's definitive, but it's starting to feel that way as, you know, we're starting to do not unnatural acts, but we're starting to do a lot of things inside of the server to get around how Go actually marries concurrency with this guy.

So when you said some of your customers need like a billion, is that like message passes per second? Like what does that mean a billion? A billion what? Some need, you know, a lot of messages per second. Nats has always been pretty performant. Last time I checked, just raw speed, a single server can do about 20 million per second. You know, messages sustained. All of our clients can't do that, right? They have more of the logic, but a raw just, you know, show it in there. But when we say billions of things, it's usually stored messages and then subject cardinality. So in event sourcing, event streaming, right? Every single message has a unique subject. And what we found was, is that people are starting to treat.

the stream, you know, of messages. And we have materialized views like key value and object store on top of them, but they're starting to treat them like a subject-based addressing lookup thing, meaning that they use subjects all the time to get messages out or filter messages or do some other operation. And so you can imagine, you know, trying to figure out an index that looks like a traditional data structure that anyone who's, you know, done a comp side, you know, 200, 300 level class would understand, but overlay subject semantics with wildcards and partial wildcards is a little bit tricky. That makes sense. Yeah. Like a giant hash table in the sky that also.

It has some very gnarly semantics that you can use. Yeah, and to be honest, it's called the S tree, and it's an adaptive radix tree with path compression and expansion, but then it has an overlay where we can say, hey, you just stored a billion subjects in here, hopefully in an efficient way. Pick out and call me back on everything that matches foo.star. In other words, we're a wild card. And so we have that, but that lives in memory. And so if you have 10 million messages but only 1,000 subjects, you're fine. You'll never even see this. But we're starting to see people where they've got billions upon billions of messages and everyone has a unique subject.

Next thing you know, your server's taking up 30 gig just for the indexing on the subject-based addressing layer. So that's one of the big challenges that the team is looking into solving for 212, the next version of the server. Well, it's cool that you still have your head down in the technical trenches of this project because it seems like from what I've seen from you all and your name signed on things recently, it's all been higher-level business, maybe technical legalese, not technical technicals. But it's cool that you're still close to the middle of this project and not just the business guy now. Yeah, I mean, I...

I think I definitely wear multiple hats, obviously. And as of late, I've been more concerned about upper-level discussions around Synadia and NADS and the CNCF, obviously, which we can talk about, but also the business side of Synadia. But I identify with builders. I feel that, you know, working this hard, you want to get to a place where you get to do the stuff you'd like to do at least part of the time. Right. And I still love doing that. You know, am I as good as other folks in the company? Probably not anymore, but I still love to reason about things, think hard about things, you know, tackle really, really hard problems. That's kind of my relaxing motion in terms of work, if that makes sense. So if, if Nats was roughly around 2010 ish era, you really didn't plan to do anything with it beyond its original case. At what point did you say, well, you know what, we should be a company. We should create Cineadine. That was 2017 based on what you said before. How did that?

How did you go from there in 2010 to there in 2017? Yeah, it's an interesting story. So one of the things that folks might not know is that when we were doing Cloud Foundry, VMware had never released anything that was open source. And my pitch to Paul Maritz at the time, who was the CEO who was running VMware, I said, I really think we'll have more bang for the buck here if this is released as open source. And so when I didn't ask, I said, I'm not going to keep this closed. So I actually just opened it as MIT, right? And then we did...

I can't remember what license we put on Cloud Foundry, but it was open source, which was a really big deal. When we went to AppSara, we did the rewrite and go. And we started to contemplate whether or not this could be something bigger than just an OSS project. I mean, it was powering all of the AppSara stuff. It's similar to Cloud Foundry, right? Telemetry, command and control, and eventing.

And so as we looked at Apsara, and Apsara was doing a commercial version of Cloud Foundry++ to be very, very simplistic about it. But as soon as Kubernetes came out, you know, certain things Kubernetes did that people liked, certain things that the Apsara Continuum platform did, they liked, you know. So it wasn't a technical battle per se. It was, where's the consumer bias going? And Kubernetes, because no individual stakeholder needed to make direct money off of it, drove the consumer bias to zero. And so our ACB started declining. And so I called it a mercy board meeting and said, hey, you know, I see the writing on the wall and all I can tell you for the 30 plus million dollars left in the bank over the next four years is that we can tread water, but I don't think it'll change the outcome. And then I think pulling it into Ericsson and doing something constructive with some of the internal projects that I know they're working on made better sense. At that point in time, there were certain people that were going to leave.

certain people that were going to stay with Ericsson. And then there was Nats, the team that was working on Nats in the middle. And at the time, Ericsson said, we don't necessarily have any need for Nats. It's open source. We can use it. But if you want to do something with it, we can. And a bunch of folks internal, prior to that decision-making process, had already kind of approached me saying, hey, we might be able to build a business out of this under the assumptions that distributed systems will continue to have more moving pieces, but more importantly, that these pieces are going to start to be stretched out. So stretched out across regions, across multiple cloud providers, but more importantly, out to this at the time, new thing called Edge. And the bet that we made was that

Edge would dominate interaction models and would dwarf cloud within probably a decade, and that cloud's not going anywhere. Don't misread that, but that cloud would become the new mainframe, but all the interactions would be at the edge, and the rules are very, very different. So as many rules changed as we went from data centers into cloud, and we had the Roku guys, Adam Wright, and the 12-factor apps, and then, of course, cloud-native architecture, of course, then the CNCF. All of those things, that step function from data centers to that, I believe, will be even bigger stepping from cloud into edge, where it's not batteries included. You can't bring all the stuff that the cloud is offering, and you have to kind of...

Not think differently about the what, meaning you're still drawing the same triangles and squares on a whiteboard as you're architecting, but the how would be radically different. And so that's when we made a leap to create a stadium at the end of 2017, based around a foundation that modern distributed systems will look very different than they do today. And they'll be pushed very hard in terms of dynamic agility. They can change all the time. Again, I probably further along career-wise than some folks on the call, but it used to be that when a system was set up like this in the 90s, early 90s, no one touched it.

I mean, I can walk into any financial institution on Wall Street and I could go into their server and point exactly where our stuff was running. And it never deviated from that, right? And that's very different than what I think a lot of the audience probably thinks now, where it's like, I don't know, somewhere in AWS East too, you know what I mean? But I have no clue what machine it's running on. And so our thought process and our priors were that that type of step change is going to accelerate with Edge and that the current tools of doing everything with like HTTP or HTTP derivatives just simply won't be able to keep up in our opinion. How has that bet played out? Very, very well. What I mean by that is that

In the 90s with Tipco, I was on the engineering side. I wasn't in an executive role or had any type of position around pricing or anything like that. But just from an engineering perspective, we cared deeply about interacting with these customers and such like that. When we watched what was happening with Edge and how fast it was moving and how different the rules were. At Tipco, we had to educate the market. And I remember how painful that was. A lot of the sales cycles were 80% education. We need to explain to you why we think you are going to have a problem in the near future that we think we might only be able to solve. In this world, what was good for Cineadia was people were coming to us going, hey, I tried to pull Kubernetes and JVM stuff like Kafka and put it into an edge thing, and it just didn't work for me. And I need to think differently about it. Can you help us? And so when we started Cineadia...

We want to start a company based on if you're running a production and it's critical to what you're doing, which luckily our stuff is critical to every customer that we have. There's no coffee breaks. If it goes down, they will light my phone up. That was the opportunity we were looking for in terms of starting a company. And we didn't have to educate the market. They were already in pain to start the company and that they would want a commercial agreement with us to wring our neck if something went wrong. And I think that's a really good way to start a company. It's not the best way to scale it. So you need a backup plan as you evolve. But to start it, I think that's a really good place for folks to start, especially if it's based around OSS.

Well, friends, I'm here with David Hsu, CEO of Retool. David, I want to talk about awareness beyond Silicon Valley. Retool has a great presence and a great awareness inside Silicon Valley. But what about beyond? What's really cool is I think we've done a really good job of building awareness inside of Silicon Valley. And so when you look at customers that use Retool, pretty much every big company in Silicon Valley, about 1,000 people today, now uses Retool and builds internal apps.

by Retool. So that's really awesome. And I'm really proud of the progress we've made there. But I think the larger opportunity for us, actually, it's outside of something about. When you think about, for example, the Kroger's of the world, the Coca-Cola's of the world, many of them are customers already today. But I think we haven't done as good as a job building awareness, if you will, around the developers and all these companies. And that, to me, is where the opportunity lies. Because so much of these companies run on software. Software is so important. If you think about Coca-Cola, for example, Coca-Cola's not really gotten any cheaper to manufacture in the last 10 or 20 years. Instead, the reason why Coca-Cola's doing well as a company is because they are getting more productive by better software. And so every company needs to become a software company, and Retool lets you do that.

So Coke is a big company, but the principle rings true. Become more efficient by using better software. There you go. Well, if you're beyond Silicon Valley, raise your hand. We want to hear from you. I want to tell David we've reached people beyond Silicon Valley that we're raising the awareness of what Retool is and what Retool does and some big announcements coming soon here on ChangeLot, which is cool. Well, if you haven't yet, go to Retool.com, get a demo, try it out for free. All that good stuff. Again, Retool.com.

So OSS, should we open the can of worms around OSS? Most recently, there's been a, I don't know what you call it, a skirmish, a disagreement. A kerfuffle. A kerfuffle between CNCF and yourself as an idea, specifically around NATS and its involvement in CNCF and potential relicense. Of course, we've been here many times in the world of open source and business and licensing. It's something that everyone's trying to figure out. And some people have or have not, and then other people think they have and then they have not. And then things change and people are offended. And so there's been a lot of, I don't know, maybe drama is just the best word around the business and the project.

Maybe start with the CNCF and Nats' relationship to it prior to the most recent blog posts and press releases. Yeah, so what the audience might not know is that the CNCF was kicked off at a large meeting at the Switch Data Center in Las Vegas, which I attended. And I was actually part of the founding governing board. So a lot of people might think, oh, Derek's role with the Nats project and Nats being a project in the CNCF. That's kind of the totality of it. But that's not the case, right? And even early on, we were looking at...

what value CNCF could bring to the ecosystem from a consumption perspective, the users to the people that are the companies that want to utilize that. And then of course the project themselves. And then for me, it was like the companies that are driving said projects. In the case of something like Kubernetes, where it's, you know, all of the big boys, so to speak, there's not a lot that the CNCF needs to do that, right? It's kind of taken on a life of its own and it's marching down a path.

What we started to realize is that as time goes by, because this was, I think, over a decade ago, that both the CNCF and projects can evolve. And where we were at was looking at the question, are we the best fit for CNCF? And no fault of CNCFs per se, but it just felt pretty clear to us that certain projects were at this tier and other projects were way down here, and we kind of were in the latter category. In no way were we attempting to re-license the whole code base. I mean, one, you can't do it because once it's released, it's AP2, it's there, that will never change. But the CNCF in early discussions, because we said, hey, would you be open to letting the project leave if the landing spot made sense?

We could do a joint statement where it's like, no, it's landing in this new foundation made up of lots of, you know, big users and customers or production uses of the NATS ecosystem, commitment to AP2, XYZ. And during those early conversations, which started in February, we were asked, hey, are you considering a license change? And we're very well aware that we can fork and do a scenario version, enterprise server, and XYZ. And so we said, you know, we're considering it just for the server, you know, that we might do like a BSL. And at the time, the BSL...

For us, it wasn't translated well out into the media. But the reason that we were considering it was we felt that it was the best show of commitment to both OSS and to our customers. And I'll explain what I mean by that. The BSL has usage clauses that you have to define when you release the software. And one of them is there's a period of time after which it converts back to whatever language you pick, which we would pick AP2. And it doesn't matter what Synedia does, the Linux Foundation does, CNCF does, that is a legal contract that once we release, let's say, a server with the BSL license after two years, it's converted to AP2, even if we don't actually update the copyrights or update the license or whatever. And so we felt that from an OSS ecosystem. And again, we've gotten lots of additional information, lots of additional feedback, most of it constructive, some of it not.

But, you know, and we've been listening, but our initial thought process as we were going down this path was, hey, this signals really strongly that we're not going to hold anything back. It will always kind of revert back to generic drug prices, right? Some of the good parts of what the patent system originally was trying to do early on. I'm not claiming that the patent system is necessarily good these days. It's a necessary evil, right? In addition, if you would imagine a two-year BSL window, and let's say Jared wanted to be a customer of Synadia, right? And, you know, customers are worried about vendor lock-in, price gouging, all those sort of things. You could say...

I went into a two-year contract, so I have predictable pricing. And at the end of that, that all stuff that I'm paying for converts to AP2, and I can now use it for free. Or I can say, oh, I like the next version too. I want to use that and re-up, right, and do that. So I'm not saying that it...

transpired the way we wanted it to. But I think people's thought process that we're just evil and we're just trying to price gouge and be greedy and stuff like that, I don't feel that that necessarily is correct. At least as we were discussing internally, we care deeply about our commitment to open source. But we were starting to see a very disturbing trend where customers that everyone would recognize, they're in the Fortune 50, that are using NATS to power production-level services or functions or products, not only had never reached out for any type of commercial... agree with us, but actually have policies that say, if you're in the CNCF and you're incubating and graduating, we're not, we're not paying for it, period. And to me, I thought that that was at least worth the discussion, you know, and it might not be a discussion between myself or Sanadia or the CNCF, but the broader ecosystem, I think, needs to look at, do we want open source projects that drive value, that can be critical, that are driven by a small group of individuals or a company, right, that's trying to make, you know, business out of said value. And, you know, Sanadia has lots of different revenue buckets, so I don't want listeners to think that this is our only drive.

But I was at least concerned enough to say, are we signaling to the ecosystem, the broader ecosystem outside of us, that we only want projects that look like Kubernetes? We don't want any projects that are being driven by a single company. And I thought that was worth a discussion, for sure. How much time before the blog post that I think broke the news, so to speak, from the CNCF directly, how much behind the scenes conversation was going on to sort of illustrate your points and your concerns? Like, was there a meeting? Okay, so this was a surprise? Yeah, so we were not...

We had gotten to a point where we were trying to assert the fact that, you know, we wanted them to consider letting us leave and they were stuck on some of the trademark stuff, right? And so for the listeners, you know, who haven't been in this, when you're doing negotiations behind the scenes, you posture up and you do all kinds of stuff and then you figure out when you can't move the stakes anymore and then you make a decision, right? And our decision was always going to be, hey, if they dig in, even if our legal team say that we could potentially win this. we have to consider the damage to the ecosystem and whether or not it's worth it. And we had already probably decided that they were just going to dig their heels in and say, nope, you absolutely can't do that. If you want to go down that, it'll have to be a legal recourse that we were going to say, okay, we'll figure out something else. But I can't speak on their behalf and I'm not trying to. And again, I'm extremely encouraged by my talks with the Linux Foundation and some of the ongoing stuff with the CNCF and that mutual statement. Again, everybody's had a hand in that pie, so it kind of gets diluted down. But I can tell you from my individual discussions with folks, especially at the Linux Foundation, I'm very encouraged with them saying,

OK, let's see if we can do better here, you know, which is all I really kind of wanted at the time. But I did feel that, you know, we could be a good steward. And if we could say, hey, here's what we're committing to. And that is commensurate with kind of what the CNCF, you know, is doing around certain projects. I thought it was worth a discussion to have. Now, a lot of people don't agree with that and have very, you know, emotional responses to that. And although I don't necessarily agree with it, I understand that people will.

fight very hard for consumption being zero, but production is never zero. Someone has to pay for it, right? I mean, you know, unless it's just something, whether it's time or money or both. And so we were concerned enough to say, hey, let's have a discussion about that. The main concern, I think, if I understood correctly, around even making this a concern for you was the fact that Cinedia was having challenges securing certain contracts because of its incubation status of NAT, which Cinedia is built on top of as the commercialized version of it. Is that the summary of like your initial concern? No, I mean, we didn't.

and that's a public record, and it was mostly around diversity of the contributors to the server, right? So GitHub has all the stats so you can look at all the people that have contributed to it. And I didn't agree that that was super important for the success of a project long-term. Right. So I'm not saying the CNCF is wrong and I'm right. I'm just saying we had a differing opinion of what successful projects look like. And, you know, there's some that I've driven in my career that now have been around for 30 plus years. Two of them have been around for 20 plus years. So I kind of get what it is, but it doesn't have to only look like Kubernetes. Right. In my opinion. And so that was kind of the first one where I was like, you know, the CNCF is evolving where they bring in lots of different projects and it's a daunting task of them having to do all that. And hats off to them on that.

But as you watched how things moved through the channels and got towards graduation, there was some friction on, does the project look like what CNCF wants it to? And again, totally the right to do that. And then also I was noticing little perturbations in companies doing strange things to try to drive a business model where they were the main contributor, right? Their company was the main contributor driving said project, you know, outside of that. And we were watching that pretty closely. We don't get too many people worrying about the CNCF or our graduation status. Every once in a while we do. It's like, hey, why haven't you graduated? And we just say, oh, because, you know, the CNCF's graduation criteria looked like this. This is how kind of the server looks like.

And it usually is that quick of a conversation, like, you know, 15, 20 seconds max with customers every once in a while. I think I've gotten two of those in the last six months, you know, and it was just a, hey, why haven't you graduated? So it's not that per se. It was as the CNCF evolves, are all projects good fits and is the only answer. You have to fork it away and then we'll kill it.

I don't feel like that should be the only answer. How about that? Right. There's so much brand equity in the Nats name. Obviously, you founded it. You created it. I created the Nats name driving home from VMware right on Friday. I remember. And, you know, everyone laughs and, you know, the marketing came up with a funny term. But, you know, I remember just shaking my head driving home going, jeez, I don't have to create another TIPCO server. I was like, not another TIPCO server. I just kept saying that to myself driving home. But I said, I feel like I have to. Well, at the end, not another TIPCO server is exactly how it got named. Yeah. Yeah. So when you first joined or you started incubating, is that the?

entry-level term there with NATS. Were you aware of the CNCF's desire or were there requirements like this project will move towards a non-single corporate backer model or whatever language would fit? Like, did you know coming into it that they wanted you to kind of change the format or the org structure of NATS? I don't think... I don't think that was discussed at that level early on. So, you know, we were at the switch kickoff meeting. We saw some of the early projects. They wanted us to be one of the earliest projects. I was like hesitant. So I said, I want to be part of this, but I want to wait and see. And Google was doing similar things early on. They're like, we just want to kind of see how this is going to kind of pan out. But I think it was more from a sustainability. How many people are using it? What type of impact does it have both within the CNCI but then externally? We're more of what I remember very early on of how do we classify a graduated project? Now, as the TOC, that's the Technical Oversight Committee, right, keeps changing and new people come into that, they started changing, you know.

So instead of kind of a vote and a gut instinct, you know, on a vote type stuff, they started formalizing some of the things that they felt needed to gate between sandbox to incubating to graduate. There might be some more levels in there. I'm not sure. And so at the time, though, we weren't aware of that. It was only kind of really solidified when we actually went through the graduation process. And again, that's all online, the back and forth between us and the TOC on trying to graduate. So was your failure to graduate then?

a shock to you? Or did you see it coming? I did not see it coming. And it took us a while. And again, it's all public record. It took us a while to figure out what's going on here. And it was specifically about diversity of code contributions to core pieces of the project.

And again, CNCF, you know, I think they've done an amazing job with all of the different pieces that are flying at them. But if you look back historically, you could see certain projects that were like, well, how did that graduate? You know what I mean? That looks a lot more similar to these types of projects than others. And I think they've slowly cleaned that up and kind of solidified that process and those criteria. It was just, I didn't agree with the criteria. You know, and again, I'm not saying I'm right or wrong by any stretch, but I did fundamentally disagree that not all projects- You must think you're right then, right? Like if you disagree fundamentally, then you got to think you're right.

Well, in certain projects, right? And so, you know, could Kubernetes have been driven only by Google and gotten to where it was? No, but it has a very different role of kind of greasing the skids between cloud provider mobility. That was kind of some of the original witness around Diane and the folks that built it, at least from me talking with Diane back in the day, of why they were doing it. But I don't think saying that that project fits means that a project like Nats doesn't fit. Right, but they disagree with that. Correct. And they're the bosses of the CNCF, so they get final say on that. Now, when your conversations began back in February.

this is when you realized you were not going to graduate or you'd already kind of known it for a while. Like you had decided, well, we're not going to be a good fit anymore, right? Yeah, that was the graduation I think was in 2020 as well as the trademark fight. So this was more of a, hey, we just think we're not a good fit anymore and let's figure out if there's a way where both the CNCF and Cineadium can message to the ecosystem that this is okay, but as long as there's a landing spot that holds true to what the CNCF wants, commitment to open source, things like that, would that be possible? And obviously the answer is no, but I still think it was worth a conversation and that's where we went. So in February,

You wanted to kind of slip out the backjack. It wasn't going to work out. And then those conversations began. And then they asked you if you were thinking about a relicense. And you said, yes, we're thinking about BSL. And then there's probably some more conversations between there. But then they decided to publish the blog post or what I'm missing in the interim between. So I think, you know.

we were very clear our commitment to AP2 and OSS and that it would be a fork that we put BSL work into that then would, again, in our opinion, would signal a commitment to OSS because no matter what we did or said after that, two years, which was the number we had picked internally as we were going through that process, it would convert over. And so I was saying that the licensing stuff is kind of separate. You know, we're talking specifically about the open source nature of NATS and whether or not it's a good fit and if it can land somewhere else with both the blessing from the CNCF and the commitment from Cineadio that allows them to feel comfortable doing that. They then started getting to the point where we were starting to figure out when the stakes would move or not move. Again, you know, just, and it's totally fine, you know, negotiating types of stuff. And they put a hard stake in the ground. Nothing will ever leave the CNCF. We'll kill it instead. Nats cannot leave the CNCF. Nothing can leave, ever. Nothing, ever, including Nats. Correct. And that was my question. It was like, has it ever happened before? Right, right, right. No, it hasn't. The details behind the scenes here. Okay, so it's like Hotel California. Is that the one? You can check it out. Right, and again, I think that, you know.

That's totally fine that that's their position. I still felt it warranted a conversation of, because that doesn't feel right to me. So if you look at like Grafana, right, had an interesting thing with Cortex and they were going to start driving more stuff, but it wasn't necessarily compatible with the CNCF, but they were still committed to the open source nature, you know, of what they were trying to do, but it wasn't compatible. And so the CNCF said nothing ever leaves, so you can fork it and then abandon this one. And that's what they had to do.

And again, I'm not saying they're right or wrong, but when they purport that's a success story, I talk to other people and it doesn't feel as much of a success story, right? Mimir started and that's what almost everyone on the Grafana side uses. And I think Cortex is mostly supported by AWS so that they can run it and not have to pay anything to Grafana, right? Again, the consumption of zero is a sense of entitlement that I also think is kind of interesting, but production side is never zero, right? Someone's got to pay. Yeah, it's a hard position to be in whenever, and almost knowing this, I think maybe this even exposes a different scenario where

Maybe you were not fully aware, like I was asking, were you aware of X, Y and Z when you first joined the CNCF, you incubated that whole process? And I think over time you get, you know, more and more privy to certain details that are truths about what it truly means to join the CNCF. I'm not saying anything negative or bad, but not being able to leave, like coming in the door, not being able to leave with the thing you think about over a weekend and you name. is challenging because if you don't pass the bar of incubation and graduate and you're not a, I don't know what it means to not graduate. Does that mean you're like not? What does it mean to graduate even? Like what does that mean? I don't get all of it. Does that mean you're not valuable? Does that, I don't understand the graduation status is either. I'm sure there's a lot of nuance too. Yeah, there is. And even after our TOC process around graduation where we did not graduate, there was a slight change that said incubating or graduated projects are considered mature and production ready. But within the grand scheme of things, someone from the outside, a user, right, who's saying, I'm just looking for some software that might solve my problem. You know, they see graduated as a higher level than incubating.

And so they're like, why haven't you graduated? Because you've been here for a while. You started at incubating, you know, back in 2018, I think is when we officially joined, if I recall correctly. I might be off on that one. And so, yeah, I wanted a conversation. We had spent, you know, and I'm not saying that CNCF has not also worked to promote NATs, you know, maybe not as much as Kubernetes and things like that in KubeConware, but they did do a part there.

we lived and breathed this 7x24x365 for almost 15 years now, right? We've put, you know, everything we can, and not just myself, it's just, you know, all the folks that were at Absera and then, of course, now at Sanedia. And I'm not saying that what we thought was necessarily correct per se, but I find it interesting when people go, oh, you really don't care about the project, you know what I mean? Or you're just trying to do a money grab and gouge people for money at this time. That's a little unfortunate. I get it. You know, and again, I know this whole topic is very emotional and I can appreciate that. But that's about as far from the truth as you can.

possibly get, you know, from, from what we were really trying to do. And again, if you, if you looked at some of these companies and, you know, we talked to other customers of ours, big, big customers and said, Oh, you know, these people that are kind of on the same level of you and the fortune 50, they refuse to pay. And when that kind of came up, they actually kind of changed their tone and said, Ooh, so we could actually get access to stuff that we're helping fund by having a commercial relationship with Stadia for a period of time, two years before, you know, one conversation, they call them the freeloaders. And he's like, it's not Bob and Vinny's pizza shop out of Venice. It's, you know, people on the fortune 50 that have the means to pay. And this isn't a little internal developer tool. It's a,

production-level critical service, they actually went, oh, yeah, we actually kind of like that idea. And so from a part of an ecosystem standpoint, those conversations, we had lots of them because we lost control of the narrative, right? The CNCF wanted to all of a sudden move things into the court of public opinion. That's totally their right. We didn't have a heads up, though, so my phone started blowing up when that happened. But business as usual, we kept doing commits, PRs, and we actually did a release the very next day, a full-blown server release, which is a really big deal. It takes a lot of effort for us to kind of do all those. So we've been trying to single to the right folks looking at the right pieces of data that...

nope, there's a disagreement here. We're going to try to work it out, but we're still committed to what we've been doing all along. You mentioned some of the emotional responses online, and I read a few of them. one of which, which I'll summarize because I use more flowery language than I'll allow myself. But maybe you have a chance to answer this particular outsider perspective, which if you're just watching from the sidelines, I can't remember if we hit record yet or not, we start talking about armchair quarterbacks, but you're just sitting in your armchair quarterbacking. And what you see is Synadia brings NATS into the CNCF as an incubating project, gets all the benefits, even though it sounds like you were getting all the benefits that you maybe thought you would get because Kubernetes maybe sucks a lot of the air out of the room.

Gets all the benefits, you know, the conferences, whatever, you know, the clout that you get of being a CNCF project and then hangs out there for a while, decides they want to relicense and take their ball and go home. And that makes people upset. What do you say to that sentiment? Like, why is that wrong if it is? The reaction is not wrong. What I have an issue with is the conflation of facts within the reporting, right? And so it's easy to draw that at conclusions. I don't get upset at people being very upset at us and more so directed directly at me. The question is, is how much energy do we want to put into trying to?

make sure the facts are correct right the whole licensing and leaving the cncf were conflated when they never should have been because they were never conflated in my initial conversations with unrelated conversation they just asked me and said hey because this is a problem we're seeing this and we don't know what to do about it yet but we wanted to know i'm like yeah because i said we're starting to see you know said three letter you know companies use our stuff um and get a lot of value and contribute nothing back and i'll be very transparent yes contributions comes in a lot of ways but for A company like Cinedia, a commercial agreement and funds is the only one we really care about. Yes, we care about issues and PRs and all that other stuff. But, you know, just being very, very transparent, they don't keep the lights on. And again, Cinedia has plenty of money and we have lots of revenue opportunities and we're driving lots of those. But at the time I was like, this to me feels wrong. You know, it just doesn't feel correct that the consumer bias is being driven to zero for companies that it makes sense for them, right? Because if you sell a piece of hardware or service, it's kind of nice to have a guarantee that all the software you're going to use to augment that will always be free and keeps your margins where they are.

I just think as an ecosystem, we're evolving. Things are changing. You know, CNCF has changed a lot, you know, that we still need to kind of think through some of these and figure out what do we really want as a global ecosystem around this? And charity is not a business model, right? So the whole notion of, oh, you can do donations, that's not a business model. And so...

What this lesson has taught me is that the biggest challenge, obviously, is in a company that develops OSS software that wants to make a business off of that software, specifically as software, not necessarily as a service. Service changes the consumer bias. So if it's something physical or somebody's running something for you, your natural bias is, oh, I have to pay for that. But if I give you a piece of software that we spent over 30-some million dollars developing over the last decade or so, you start to see this notion of entitlement. Oh, I deserve this for free, and I don't have to give anything back. And we even have some people that just bark orders at us, like, you need to do this in the system. And we're like...

Okay, you know, that's interesting. You're not a customer. We don't even know if you use it. And everyone has these problems, right? All the OSS projects have these problems. But for what we were looking at, I was like, that doesn't feel right to me. And I wanted to reach out and open a dialogue and have conversations that would hopefully spur, you know, some constructive dialogue. But just for the record, you know, we never were abandoning open source. We were never going to blanket relicense everything. That's just simply...

conflation of facts in the media and then people reacting to them. And then that being amplified, you know, quite a bit over the social media channels. So that's the, I have no problem with people being emotional about it. But the, the way it was presented and the fact that we, we not only lost control of the narrative, but then all of these articles never reached out to us or me at any point in time and said, Hey, what's your side of the story? You know what I mean? I was like, that's interesting. I even have one now where someone pointed me to a YouTube video where the guy just, you know.

F-bombing me and trashing me, you know, left and right. And I'm like, I don't know who this person is. I've never met them. They never talked to us, you know. And I asked the whole team, I said, has anyone interacted with this person? They're like, nope. You know, they're just, you know, a YouTuber that, you know. That's how you get the views, you know. Yeah. And I get that. But at the same time, you know, I kind of shake my head. I'm like, okay. No, it's not. And again, if as an ecosystem, we say to ourselves, we don't want open source projects that drive mission critical functionality that are driven by a single company. I don't agree with that. But if as a global ecosystem, we say that, but I want people to think really hard about what that really, really means.

Well, friends, building multi-agent software is hard. Agent to agent and agent to tool communication is still the wild, wild west. So how do you achieve accuracy and consistency in non-deterministic agent applications? That's where the agency, A-G-N-T-C-Y, comes in. The agency is an open source collective building the internet of agents. And what is the internet of agents?

It's a collaboration layer where AI agents can communicate, discover each other, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. You can now build with other engineers who care about high-quality multi-agent software. Visit agency.org and add your support. That's A-G-N-T-C-Y dot org.

If you had your way, like if this went without a fight and you were able to do what you're trying to do, what were you really trying to do with the project? If things went your way, this fight didn't happen and you could leave the hotel, so to speak, what would you have done? What was the goal? So earlier on, we were talking, I think, about I don't know if we were being recorded or not on programming languages, you know. And so what I was looking at originally was if we created a foundation that was, you know, had a representative, but also had representatives, all the major.

users of Nats and customers of Synadia in a 501 3C or something like that, that had a commitment to AP2, right? Open source, you know, XYZ. That's probably what we would have done, but it was simply just for the betterment of Nats. And everyone that sat on the board, similar to kind of like the TOC, although the TOC is more of, hey, we'll get involved if you need to move from sandbox to incubating to graduation or if something's off the rails. Otherwise, we let projects manage themselves. This would be more of bringing in, you know, a lot of our top customers, you know, Walmarts of the world, you know, things like that to have a seat at this and be able to help direct where the project would go.

And right now in the CNCF, that's just not a full-on reality. That's more limiting, more governed by other folks. Yeah, couldn't you do that inside the CNCF? We could, meaning we could. I mean, right now we still have the problem. But then you graduate, wouldn't you? I mean, you did that. Then they graduate you. No, the graduation criteria that we failed, and I don't know if it's still the same, I have to check with that, was the code contributions in the server were not diverse. Okay, so it's not like governance. It's actually code contributions. That was what tripped us up.

in terms of trying to graduate in 2020. Yeah, that seems to be off. Like, that doesn't make sense. I mean, it's more measurable. But if you established even a TOC of your own, right, or TSC, like a technical steering committee comprised of Sanedia folk plus your largest customers, right? Yeah. And they didn't write the code. Like, who cares who writes the code? If they're helping vote on the direction of Nats, I would think that would be a diverse...

open source project. Yeah. You know, I would tend to agree, but again, I'm not trying to say, oh, the CNCS is wrong or try to speak on their behalf. And they aren't here to represent themselves. Right, right. We also have one side of the story. We're not journalists, we're just podcasters, we're here to talk, but worth saying they're not here to defend their side of any of this. Yeah, but what happened to us with us not controlling the narrative, I want to be very sensitive to the fact that they do have differing opinions. And again, I'm extremely encouraged by our talks and how they're like, okay, yeah, let's see if we can do better here from a, you know, graduation flow potentially, but also...

I don't know if it's this notion of, I don't know the answer and I don't think the CNCF knows the answer either. But like, for example, Modular, I don't know if you saw, they just did a license change as well. They're getting more and more permissive, right? Because they want broader adoption. And it's AP2, which is great. Modular, they do Mojo, that's Chris Labner. But what the interesting thing was is that they say, if you use it in production, you just have to let us use your logo in the use case. And I was like...

That would be good. I would like that. Because we have so many companies. I mean, if you go to Naps.io and look at who uses us, those people opt in. They send us and say, hey, please put our logo on there. But there's so many more that use us and don't want that or actually say, no, you can't use our logo whatsoever on that. So even something like that to kind of go, oh, we can see Acme Company is actually using that. So we know Acme Company. That's really cool. That would help a company like Snedio, right? That's that trust factor. And so I shot that over to the folks at Lynx Foundation and just were like, hey, would you guys ever consider something like this? I don't think they will, but I'm trying to.

do my best and operate in good faith and say, okay, if we want to make this better, I'm going to tell you things from my perspective that could make it better for a project that looks like Nats and companies like Sniggy that are driving that. And we'll see how that goes, right? I don't expect anything material to change radically, but at least they're open to the conversations now and are starting to be receptive to, okay, we see that there's some challenges here that we might be able to improve. So Modular setting that up, effectively Apache 2 with an additional constraint on it, isn't that then technically like a proprietary license now? Like they've customized...

a license and now it's not exactly Apache 2 anymore. It's like Apache 2 with this other thing, which is very similar to like what Mende did with Llama, which is like, it's pretty much open source except for this one. Now their constraint was much more strict than we want to put your logo on our website. It was like, if you have 10 billion users, you can't use it or something like that. But I think, I mean, you raise a good point. My, I'm not a lawyer and don't pretend to be, but I think in that notion, the license strictly governs what you can and can't do with the code, meaning the code itself.

So for example, even within the CNCF, right, the code is AP2, but the CNCF has very other regulations around how you use the trademark and how you can use the logo and how you can, you know, do the products and use the naming. So I think what Modger is doing is fine. You know, it's outside of that pure, what can I do with the actual code? But what they're saying is, is that if you're using a production, then we, you know, you're giving us the right to use a logo. I think those are separate, but I understand it feels like it could be sticky because the difference between a trademark versus a copyright versus a code license are all nuanced in how they kind of interplay, for sure.

Reminds me of something else that I read. And this was just, again, armchair commentary. So I'm not sure if this is true or not. But when you become a CNCF project, you typically will actually turn over IP to the CNCF, right? Like the actual intellectual property becomes part of the trademark licenses. But yeah. And with NATS, did that happen?

What happened with the trademark was kind of interesting. And again, it wasn't necessarily stated. And I tried to correct some of the facts and I got pitchforked even worse. But essentially, when we did the original kickoff for the CNCF at the switch, this is my recollection, was that you only legally transfer those upon graduation. And I remember Google kind of being a little nervous that Kubernetes started directly as a graduated project, that they would have to figure out all of that stuff with all of their lawyers at XYZ. And so that was in my head. Now, to the CNCFs,

you know, to benefit them, I think they quickly changed that once Kubernetes graduated. And they said, nope, as soon as you're in, you know, you essentially have to transfer or at least once you move past Sandbox. And we joined as an incubating project in 2018. So the rules had shifted from what my recollection from the original kickoff meeting was. But besides that, the thing that, to be honest with you, frustrated me, you know what I mean? Just being transparent. But again, I like the conversations that we're having with the Linux Foundation and CNCF. But as we were trying to graduate and we had just joined, all of a sudden Major League Baseball came and said, hey, you know, we don't like your trademark.

because it conflicts with the Nationals baseball team out of Washington. And so I was like, well, that kind of is silly. And anyone who's done patents from your listeners know that there's a little bit of a game to that as well, the whole patent and trademark and all this other stuff. But on trademarks, I was like, I can't imagine someone... being able to argue correctly to confuse a baseball team with a piece of software. Yeah, different domains. I think that's spurious. Yeah, so I approached the CNCF and said, hey, we got this opposition. We want to fight. I asked our legal team. They think it's going to cost around this much. And the CNCF at the time was not willing to take that fight on and suggested that we just change the trademark and that if we do that and make it easier on them, that they would give us a keynote slot so that we could announce the name change. But I always went to the name. I cared deeply about the name and the logo. For obvious reasons. Yeah, because remember, this had been gone. It was named and we kind of started to have a logo for almost eight years before we officially joined the CNCF, right? So it had...

a probably longer lifespan than people realized prior to CNCF. And so we fought it, right? And on our own dime as a small startup that was only, you know, a year old, had just gotten seed funding, you know, and we successfully defended it. And so upon completion of the successful defense, the trademark was awarded to Cinedia, right, with the deal. And so...

At the time, when I was talking to CNCF, they were like, well, we own the trademark. I go, well, technically you don't. Now, this will get listeners frustrated and upset. And I get that. And I'm totally empathetic to why you would get upset. And I'm not asking for empathy by any stretch. But I was very frustrated that something that they said they cared so deeply about, they threw away in one email and said, oh, no, you guys figure it out. And then to say, oh, but congratulations, you won. Now give it to us. That also rubbed me the wrong way.

But technically what the CNCF was saying is, no, you know, the commitment was you were going to give it to us. And so I definitely understand their position as well. And we are committed to that, right? So in part of our agreement, we're going to transfer everything over. The lawyers are working out the legal docs and things like that. But that was kind of my frustration, you know, to be transparent. And then they never asked for it after we successfully won that. And so, again, the way things were conflated and released out was, I think, very targeted, you know, in terms of, you know, trying this in the court of public opinion.

when anyone, a project, maybe this one in particular, but any project, decides to take that project, donate it to CNCF and do all the things you did, incubate, et cetera. Is there ever a point where you sign a specific agreement that's clear? So I feel like, Jared, you've heard me say this before, clarity and expectation. Like if you have clarity on an agreement, it's the whole point of an agreement. It's less about binding you, but more like, here's the line we both stand on. This is where we both agree. Was there any clarity or is there a signature scenario where you're signing and it's clear and this will happen and there's a series of events? Was that, did you ever sign something? We signed something to get into QCon in 2018, which had a line that says,

If you have a project in the CNCF, you promise, I'm making some up, but I think it's online. You promise to abide by all the rules for CNCF projects. That link, I still can't even get into that link today. And so I think there's an opportunity here to improve that process. And also, I would imagine the listeners are very well aware of those emails that say, our terms have changed or terms of service have changed. And you get all these emails and most of us don't even read those. I never saw any of that. I'm not saying they weren't sent, but I never saw any of that to say, oh, hey, what you think, Derek, from the kickoff meeting, we changed it. And now it's this. So I think there's an opportunity to make that a lot more clear. But most people who are coming from...

in opposition to me, say that's always been the case. That should have been crystal clear to you that you donate trademarks and XYZ. And I understand that because they see the CNCF as it is of all. They didn't see it when it was first getting kicked off. And so I think they're totally right in the assertion that I should have known better, but I still think there's an opportunity to make it a lot clearer. Like even now, I can't remember if it's as soon as you get part of the CNCF or once you move past Sandbox. I think if Sandbox, you can take it back. If you don't get past Sandbox, I think, but I could be wrong there. And so even with all of this going on, it's still not necessarily clear what...

all of the rules of engagement are, but I just see that as an opportunity to improve for everyone, if that makes sense. Yeah. Well, like in that case there, if the link is broken or there's some sort of like language that's compressed, there's a whole other document that's compressed in like four words, essentially, based on what you just said. And so those are usually addendums to an agreement. You know, this is probably signed tons of agreements in your life. Those are usually addendums and that's usually compacted into a single product. This is the agreement I'm signing. And if there's terms beyond the agreement I'm signing, they're attached as an addendum to kind of, to help everyone safeguard themselves from scenarios potentially like this.

OK, so my second question is this is it sounds like y'all have an agreement. The blog post is out there. It sounds like kumbaya to some degree. Maybe not. Is any of this conversation you're having with us right now going to get you in trouble more so? I don't know because you're being very clear. Yeah, I'm trying to be very good reasons, but you're also being very clear. And it could be very clearly wrong. Clear, not wrongly in terms of truth, but more like, hey, come on now. Don't say that in public on a podcast that gets listened to by a lot of developers in our world. Yeah, I hope not because.

We're entering into this in good faith, and I've tried to make sure to interject that I'm extremely encouraged by the discussions that we have committed to transfer the trademarks over. We actually did finalize early this morning, which I'm happy to talk about, you know, our actual direction in terms of whether we're going to do a fork or what commitment we're going to do to the server, X, Y, Z. And so we're operating in good faith, and I would imagine they are as well. And it's a trust but verify, you know what I mean, type situation. But I'm very encouraged by them being open and receptive to some of the challenges that companies like Cinedio or projects like Nats might have. And hopefully there's some meeting of the minds around.

pass forward that might level the playing ground, if that makes sense. But I would imagine, you know, there's going to be some people that are going to, you know, pitchfork again, and that's totally their right. But that's, you know, that's how things went down, and that's kind of where we are right now. And I still feel deeply about the NAS project and where it's going. I mean, one of the other things that listeners might not understand is there's different, in my opinion, there's different personas to projects that are kind of on maintenance mode, meaning we'll fix bugs, and if there's other stuff, the features are pretty much complete.

NATS and the server are definitely not in that camp, right? We've got, you know, over two years worth of plans and still the majority of our R&D budget goes into stuff that sits in the NATSIO ecosystem, right? The part of that open source project that sits inside the CNC from all of the clients, obviously the server, a lot of tools, you know, Kubernetes, Soundcharts, SystemD, all kinds of things like that. We're still probably spending, you know, 70 plus percent of our R&D budget just on that. And that hasn't changed, right? And we don't plan on slowing that down. It was, hey, is there an opportunity for us to recoup some of those costs for people that care the most about some of those crazy enterprise features, massive scale, security, whatever? And that's kind of how we started to go down the path of, hey, what if we did a commercial fork, but made it BSL? Again, it didn't land well because of the narrative, but our thought internally was that to us signaled the biggest commitment to OSS and to our customers trying to avoid vendor locking or price scouting.

But it was in a vacuum. And so once we got additional information, we realized that we probably would have to reevaluate that thought process. So how would the commercial fork with BSL be different than OpenCore? I'm not understanding that. We would be on... BSL also is source available. It's not an OSI-compliant license. So NAS would be the entire NAS project? Just a server. We would put all of the effort into that fork, which meant that once it was released, then the two-year window started. And so then that would flow into the AP2 codebase that's discovered by the CNCF. That was the original thought process. Now, we're not going to do that. What's wrong with OpenCore, though? It seems like you could just...

accomplish the same goals without relicensing. You just have, you know, you want your billion transactions or messages per second, then we've got an add-on for you or whatever it is. Yeah, you're absolutely right. And what we've heard from the ecosystem is, is we like that model because that's what's familiar to us. What I try to point out to people is, is that commercial forks, it's totally up to Sanadia and maybe that's totally fine. And people are saying it's fine, but again, I don't think.

folks think all the way through some of these things sometimes. It's at our discretion what we put in there, and it's also our discretion that we don't have any commitment to put it back into the open source community whatsoever. So let's say there's something, Jared, that you really, really wanted, and you're an avid user and stuff, and Canadian sides, oh, we're only going to put it in the enterprise version. Now, of course, you could try to do your own version of it in the OSS side, but at the time, and again, this was in a vacuum, and now we have additional information, but I was arguing hard that the VSL would signal a better commitment to saying everything will become AP2. After two years, no matter what we put in there, it will become that way. And so that was kind of our original thinking. We knew that was going to be a fork. The question on whether or not we could use the NATS name, who it was, you know, part of where we lived in the CNCF or if it came out and who technically owned that trademark. But we already knew, we knew that that was what we were, we could do. It's just if it's commercial, we have no, you know, no guarantees to either our customers from vendor lock-in and price gouging or the OSS ecosystem. And so at the time I was like, ooh, you know, I feel this is a better signaling mechanism, but we needed to control the narrative. Meaning once we actually did decide, and that's the other thing that folks might.

Not, no, we hadn't finalized our decision when the can of worms got opened up. And so everyone's like, you got to tell us what you're doing. I'm like, we don't even know what we're doing yet. We were just still having internal discussions on that because it was totally separate subject from the relationship with CNCF. Well, do you know what you're going to do now or are you still trying to figure it out? No, we've been meeting every day since the whole debacle started. And so we already had gone down a path of doing composition by us in ADIP. So what I mean by that is, is that a lot of customers came to us and said, hey, we want to take control over AuthN and AuthC.

We have a pretty rich authenticity system, but they go, it's not enough. We need something else. And you can imagine a world where we said, oh, we're going to do a scenario server and we'll put all that stuff in there somehow. Instead, and this is already done, this code was done probably a couple of years back, we did an off-callout mechanism, meaning the AP2 server, you can say, hey, you're no longer in charge. something over there is in charge, right? And then we, Synadia, could build lots of IP around for Authent, XYZ, OBA, integrations, things like that. And so we already were going down that path. So our decision now is we will not do a VSL stuff. We know we have the right to reserve the right to do a commercial fork, but I think what we're going to do is we're going to post, and we can do this in the show notes, everything we've committed to go into the AP2 server without any delay. The VSL would have had a delay on there, and that we're going to expand our commercial offerings through composition, meaning you're running the server, but now all of a sudden it's got security appliance, which we've been working on and some really interesting stuff. Think of it like...

a laugh for Nats plus a whole bunch of other stuff. And so we're going to go down that path. Well, I can tell you there's so much nuance in this history, nuance, good intentions, not a lot of bad intention necessarily, but maybe some confusion, some lack of clarity. OMG, please fix that stuff. When a subject matter is very emotional, right? It puts your brain into fight or flight, which means that you have to make decisions very quickly on imperfect information. That's how we all survive, right? That's why, you know, we're all here because we ran away from the lion instead of going, oh, look, a cat, let's pet it, you know, type stuff. So I'm not being critical of folks doing that because I understand why they're doing that.

But hopefully they'll understand that they definitely had imperfect information and not even close to a complete story. And it's not their job to understand all of that. But at some point, hopefully they go, oh, there's a lot more to this than we might have realized. I'm not saying we're right or wrong. And I'm not doing that at all. I'm just saying it's worth a dialogue and there's a lot more pieces to the pie that folks might not know. And to the credit, I think of the potential pitchfork folks or the folks in the crowd, not in the arena, is that it seems to be almost a, not literally a daily occurrence, but pretty frequently these days, something happening around a license change or rug pull. It's so much that it's become a thing we've said here. Rug pull, not cool. That is our phrase. Okay. Rug pull, not cool is totally not cool. And that's an additional one, Jerry. We can put that one on a t-shirt.

You know, I think they're used to and we're used to hearing this and you think immediately negativity. You really think this is, you know, nefarious. There's, you know, something going on here. And so it's almost as if there's a level of PTSD in the community. And I think, you know, as purveyors of and participants of enthusiasts of open source software, they have a right to that in a way. But also there's so much nuance here that.

they have to let some of the details percolate. And it's not always clear. So, and you're right. This podcast is why we have you on the show. Like, like I said, this is yours. And I'm making this. We're gonna have you on the podcast anyways. We've been talking on LinkedIn about getting you on the pod. And then this happened. I'm like, well, Derek, can you talk? Are you able to pod? I mean, I don't know if you can even pod. And you're like, yeah, I'm cool. And so we had you on the pod. And that's why we do this. So we give that clarity. Yeah. And I knew the road pool stuff. I've been listening and I know like Joran was talking about it and others been talking about it. And so I'm totally aware. And what my hope is, is that while we're focusing on, you know, license change and road pools that we up level to have a dialogue of.

Why is this happening? What are we as an ecosystem failing at? And what can we improve upon? I said earlier on in the podcast, charity is not a business model. Well, until you're a company that is self-sustainable, you're not just running on VC dollars. That's still kind of a charity model. So in other words, right now, our customers are funding a lot of the development, but the VCs who have funded Sanedia are funding a lot of the development that everyone feels entitled to for free. And so my hope and...

My sincere hope is that we can up-level and have a dialogue about why do we think this is happening and are there things that we could potentially improve from an ecosystem perspective that could do that? And again, I'm not a fan of this, but one might be, no, we don't want any projects that run critical infrastructure to be from just a bunch of folks or a single company. I don't think that's right, but that could be one answer. Or, hey, if you're a big, well-known Fortune 100 company and you're using something in critical production, give back. You know, it kind of like your rug pull thing. It's mine is, you know, no pay, not okay. You know, because again, production side is never zero. Someone has to put in time and money to produce this. But on the consumption side, right, we're getting to everyone expects it to be for free. And that eventually has to be resolved somehow. Now for projects like Kubernetes, it's resolved because the Googles and Microsofts and, you know, Amazons of the world, right, are motivated to put stuff in and they have other revenue streams that can offset the cost of participating in the software. That's totally fine. I'm not saying that that's wrong.

I'm just wanting the dialogue of saying, is that the only model that we really think will survive? And I don't, I don't want a world that looks like that myself. Yeah. Well, good for you for sticking up, I suppose, for yourself or trying some attempt and having this clarity. I know the narrative hasn't always been your favorite, but I appreciate you coming on the show. Even if it gets you in some trouble, I sure hope it doesn't. Just come on here and being clear. And, you know, honestly, I think.

Not that it's here at the end, but all along, anyone from the CNCF who represents, you know, sandboxing, incubation, graduation and the clarity there and maybe some of the things we brought up here, which was the agreement you signed and the terms that you alluded to and the lack of clarity there. I'm happy to and we're happy to have anybody on this. I think it's good for everyone to have some of that clarity. So if you want to have something on the show and talk about the details, what it takes to join the CNCF, what those are, we would be more than happy to have that conversation in detail. Yeah, and I think that's great. I really do. Like I said, my hope.

for all of this is that, you know, we can have an open dialogue about, are there things that we can prove upon, you know, outside of Sanity, outside of CNCF, but just as a broader spectrum. And I think there's some interesting questions that should be discussed on, you know, how we want to go about that and what role the CNCF and the Lynx Foundation plays, what role, you know, retainers play. Because one of the other things that we got a lot of pushback from is I feel deeply that the lifeblood of a project is in the maintainers. It's, yes, people want to concentrate on license, what foundation are you part of, all that other stuff. And I think they do play a role. But at the end of the day, it's the maintainers. And if all of a sudden the maintainers withdraw their support from a project.

I think a lot of projects that have a tremendous amount of investment and are not just a simple weekend toy, if that makes sense, saying, oh, we'll just get some maintainers and they'll spend a weekend and then they'll know how to program, let's say, a net server. I think that's maybe a little short-sighted. We have someone who's awesome internally that just got promoted and is now running the server team. They graciously took that from me so that I can do other things. That person came in and was under our tutelage, and specifically under my tutelage, and it's still talking about eight months or so. So we were paying them.

and we were tutoring them, and most of the server's infrastructure was built by myself and someone named Mavon, and we were tutoring this person. And it still took about eight months, almost a year, to become like, oh, I can go anywhere in the server code and be effective, right, be productive. And so when you get statements that, oh, fork it, you guys go do your own thing, and then we'll... you know, archive it or try to find new maintainers. Again, I want folks to kind of really think through that and say, is that the best outcome for the ecosystem? I don't think it necessarily is, but someone from the CNCF might say, yes, that's absolutely the way it's supposed to work. But what people are looking at to the CNCF around stability for projects and longevity, you know, if the maintainers leave, right, there's going to be some level of disruption for sure. Depends on the project, but there's, you know, it's not going to be zero. And in some cases, you know, in our point of view, which people might disagree, that's...

thrives when Synedia thrives, right? Because we're driving all of that stuff. At least that's my opinion. And again, not all projects look like that, and that's totally fine as well. But that's another one where I was like, that doesn't feel like it should be the only answer, right? There could be a dialogue of how do we better support these types of projects? What if they're not necessarily a good fit as a foundation evolves? What options are there for doing that? And again, I think...

prior to the, the, the dustup, but also going forward. And again, we'll release a statement next week, probably about whenever the podcast goes out around commitment. You know, most people might not understand, you know, if our total R and D budget 70% is going into open source and it's funded by the VCs who funds media and it's funded by our customers. And the ones that can pay that aren't paying that are like, give me, give me, give me, you know, I'm entitled to this, you know, Hey. get off your ass and change this thing for me, which we see every once in a while, like GitHub issues. That's kind of what I'm like, maybe we need to figure out a way to have them change their perspectives a little bit. I'm not saying, you know, I don't want hobbyists to be able to use our stuff for free or embed it all over the place, right? My original tagline for NAS was connect everything, meaning it had to run everywhere and was openly accessible. But again, it's that production problem, which is non-zero. It's like, how do we solve that problem for different levels of projects and open source? For sure.

And I don't have all the answers, as you can tell, but I do believe. I mean, I think the important thing is you have a lot of thoughts. You have a lot of care. And that's the thing you have to really come back to. You got a lot of care for what you're building, what you've built, you know, the community, the spirit of open source, your customers, your employees, your investors. I mean, there's so much things I'm sure that you're holding that we're not privy to that we can definitely see that you care. But it's cool having you on this show talking about this stuff and even more cool that you're a listener. And you know, road goals are not cool. I know. And, you know, no pay, not okay. That's right. No pay, not okay. That's right. Represent both sides. That's right. Okay. Get it together. Thank you so much.

I appreciate it. Thank you guys. And, uh, you know, congrats on all the success. Really, really cool to see. Oh man. Thanks, Derek.

So my takeaway from this conversation specifically around the kerfuffle with the CNCF and how that all played out is that there's a lot of nuance in these things. There's a lot of detail hidden behind the scenes, a lot of time spent, emotion, investment, time, energy, and all the things to really complicate the relationships in and around open source and ultimately any commercial open source companies that spin up around open source. It's not just black and white. There's a lot of details. And this conversation really helped me see Derek's side of things. Now, with that being said, of course, the conversation is open.

we would happily have anyone from the CNCF discuss with us the incubation process, what it takes to graduate, the clarity they offer to anyone donating or assigning IP or copyright or a project to the CNCF, because there's a lot to discover there. There's a lot of detail in there. And again, it comes back to that nuance. So if you listen to this from the CNCF, reach out to us, editors at changelog.com. So a side note, I suppose, for next week for our loyal listeners and those who listen to the end of the show that you listen right now, yes, you. We're out next week. We're in Seattle at Microsoft Build. I'm not sure.

what the publishing schedule will be. We're heading into summer. It's the last week of school, and I feel like everything needs to be done right now. With that being said, if you're going to be at Microsoft Build, make sure you say hi. Hop in Zulip. Go to changelog.com slash community. Sign up. It is free to join. At mention us or whatever on the socials. DM us. Email us. All the things. Whatever. Just say hello. Of course, a big thank you to our friends at Heroku for sponsoring this episode. The new generation of Heroku is so exciting. Heroku.com slash changelog podcast. Our friends over at Retool. Retool.com slash changelog. And of course, the Beat Freak in residence.

Brake. Master. Cylinder. OK, friends, that's it. The show's done. We'll see you.