What is the scale of a company from an engineer? When you add that up, we have more than a trillion events that we're capturing every day between consumer interactions, things that are happening across products and services that support decision-making. Live was a big launch last year. We had a lot of learnings from Paul Tyson because it was such a large demand. I've often mentioned- It was a world largest, right?
65 million concurrent streams. Watching that took up, I think one of our biggest ever days of sign-ups. There were probably 100 people on site. I was sitting in a room with maybe 30 or 40, both engineers and data scientists. We had our laptops and makeshift screens sitting there.
When I think about where we were for Paul Tyson, I joke with people like I lost 10 years of my life in that one night. We don't have formal performance reviews, which is probably the first unusual thing. So the way we approach it in Netflix is- Netflix needs to introduction, but its scale can still surprise many people. But what is it like to work at a streaming company as a software engineer? I sat down with Netflix's T2 Elizabeth Stone to get more details.
In today's conversation we cover, the unique engineering challenges that Netflix, including your learnings from three years of Netflix live. Netflix is entering principles and why Elizabeth's favorite is you're to learn. How Netflix has no performance reviews and what they do instead. How the company uses AI tools and why animal detection and analysis is a great use case that they found for them, and many more details. If you're interested in understanding more about how Netflix works as a software engineer and what it takes to do well in the kind of environment they operate in, then this episode is for you.
This podcast episode was presented by Stavsik, the unified platform for flags, analytics, experiments, and more. Check out the show notes below to learn more about them and our other seasoned sponsor. So Elizabeth, welcome to the podcast. Thank you. Thank you for having me and welcome to Netflix.
It is so nice to be here at Netflix. I'm sitting in a director chair. It has the Netflix on it, so it's inside of Netflix offices, which truly feels special. And I just remind you that Netflix is an entertainment company at its core. Yeah, a lot of magic happens here.
Even in the tech teams, we take things like video very seriously. A lot of people know Netflix, of course, from the video offerings, the films and movies. Behind the scenes, what is the scale of the company from an engineer? How can I make sense of how large this operation is? It's probably larger than people realize.
Very often I'll get questions from people in my personal life saying, well, how many engineers can it really take to build the Netflix product? So first of all, it takes quite a few. When you think about how do you make the tech work so well, that it's basically seamless, in some ways ideally invisible, because members just get to enjoy their experience. But then we also as a tech team build tools and products for studio productions, our advertising tech stock, we build a lot of the developer platform capabilities and launch capabilities for games. I think about anything related to commerce, plans, pricing, payments, partnerships, those are all things that are supported by the tech team.
So when you add that up, we have more than a trillion events that we're capturing every day between consumer interactions, things that are happening across products and services that support decision making. So it's quite a global enterprise at this point. Some of the things that you mentioned, I feel are somewhat typical at large companies, for example, building payments, building ads, building things. But you mentioned things that I haven't heard in any other company, which is for example, building custom software for your production studio. What are some of the parts of the software that you're building that?
It might be pretty unique to Netflix that you might not find elsewhere as an engineer. Yeah, it's actually very much part of our superpower that we've been able to bring technology to entertainment or one of the biggest studios in the world. And we have an advantage in thinking about what are some of the problems we could uniquely solve for those productions. So good examples would be things like our media production suite, which took something that was a fairly antiquated, slow and expensive way for media files to travel across creative teams around the world and really modernize what that looks like. So that if you have a production that's shooting somewhere in Europe and you have someone sitting in Los Angeles who's going to review the daily clips and footage, they're able to have those media files travel around the world, provide notes or input on that of those notes travel back and be ready for another day of production just the next day.
So things like that media production suite or other tools that allow us to monitor how are we progressing on some of the productions is something that's very novel from Netflix. We also have a big presence through Scanline and I-Line, which is a visual effects studio that was an acquisition a few years ago that does really cutting edge research and technology for things that affect how we do data capture visual effects, different ways to think about strategies that bring life to productions that wouldn't be easy just based on standard camera technology. In terms of the engineering work behind that, what are some of the challenges here? Because what I'm hearing is you're saying, you know, like movie, all this stuff to me, what the rings of bell is like, it'll be like large amounts of data probably.
I assume the latency might be interesting challenges. It's unbelievable scale. When you think about hundreds or thousands of productions that are in progress at any given moment in time. So of course, all those productions, you have media files, which are also especially large, complex and difficult to move. So think about scale, think about cost of both storage, compute and travel of that data.
Latency in some cases, you know, it depends on the use case. For some cases, it's capturing media that is going to be reviewed in a way that can be the next day. But for other things, we've got media traveling for things like live productions that has to be essentially instantaneous. The other thing I would say about some of the unique challenges in that space is the level of quality that we're trying to bring. So when you think about some of the challenges to think about how do you create very high quality images, videos, whether that's the content itself or the promotion of that content on the service, there's a lot of engineering challenges that come to meeting that type of bar.
The other thing that's familiar externally, which I'm sure you've heard about is open connector content delivery network, which is extremely unique that Netflix has that. It was a big bet that we made more than 10 years ago to build our own content delivery network. And the scale of that often surprises people. So 6,000 locations around the world, more than 175 countries. And that actually allows us to place local files for film, TV, games that you're going to play so that there can be very low latency and very high quality for members no matter where they click play.
Basically, these are server locations at 6,000 different locations in cities, what not where you have, it's like your edge network, right? That's right. And we integrate with internet service providers to actually get the content to when someone clicks play on their phone, on their TV, on their laptop, that actually gets the content through that last mile to the member of the consumer. And then when you're in sign-up, you get to be exposed to some level of this detail, which I guess most engineers have other places wouldn't be used to see them provider and it will be a black box, but you're building this thing, right? We're building this thing.
And it's been an incredible head start as we think about new content types. So when we started to go into live, into games, especially cloud or streaming games, as we think about just the breadth of our film and TV offering, Open Connect has been a huge strategic advantage and we're extending that to be able to deliver different types of content. The other thing that's unique is Open Connect as a content delivery or edge network is sort of the end of a very long integrated life cycle that content moves. So I mentioned that Media Production Suite. That's on a studio production.
Files are being transferred for review, quality, making sure that we're aligning with the creative vision. Once a title is ready to launch, that flows through other pipelines that we think about. Do we have the promotional assets? Are we ready to give great recommendations to the right audiences? How do we encode those files so that they're actually ready to be transmitted through Open Connect as a content delivery network?
When you think about that, sometimes we lightly call it pitch to play. There's an element of engineering all the way along that life cycle, which is unusual because at many other companies, they haven't built that end-to-end pipeline themselves as Netflix has over time. And what is pitch to play? So think about from the moment a title is pitched that someone in the content team greenlights, yes, we're going to develop and produce this title. There's data science teams.
There's engineering teams that help support those decisions on programming. Then there's tech teams that help support the creation of that content, the promotion of that content, the recommendations and ultimately the delivery of that. So tech basically underlies that whole life cycle. Wow. So this sounds like a lot more work.
Usually when I hear of the work of a pipeline, we would think of a CI CD pipeline. And I think we're very familiar with that. You have your code reviews, test run and that kind of goes on. But what I understand is just a lot bigger. Imagine that CI CD pipeline times thousands because it's actually going to follow an entire bring content to life for members cycle.
This is a great time to mention linear, our season sponsor. After all, linear was born partially thanks to the pain points that happened during company scaling up. The idea for linear came about when their founders were going through hypergo phases at Airbnb, Coinbase and Uber. As you'd expect with real scale, these companies started to slow down. What you said day started taking weeks and sometimes even months, not because people work less hard, but because there were a lot more moving parts that needed to be coordinated.
Whenever you're adapting to scale, you often pick up new workflows, processes, just things you need to do. Over time, this creates real workflow depth. Software engineers often get hit the hardest here, having to check five boxes and three labels when creating an issue just so someone's dashboard populates correctly, we've all been there. If this accumulation of steps that slows orgs down, and frustrates engineering teams. For companies that have made this wish to linear like OpenAI Coinbase scale, the move has been like a hard reset on all this process depth.
The results are striking. Scale, for example, cut their bug resolution time in half by switching to linear. If you're curious about making the switch a more switch for a re-than you think. Linear has native imported for Jira, GitHub issues as Santa, you can even run them side by side during transition. The team is also happy to work with you to run a four to six week pilot all on side you're existing tool just to prove the impact.
Check out linear.app.slash switch, they have a migration guide that walks you through the entire process. So that's the opposite episode. And now my first association seeing this longer pipeline is it sounds like it would be rigid, but of course, the math book is moving really fast. What does it look like from a software in your perspective, from an engineering team's perspective, getting a project done? How are these typically done?
Is it based on some sort of following you on the schedule or is it a lot more elastic? This is probably where the uniqueness of Netflix's culture comes in to play. So a lot of the way that our engineering systems, products and tools were built was highly driven by individual contributors thinking about how to build those systems. So innovation is really driven from within the teams rather than top down. There's a lot of autonomy and local judgment in how we build things.
That has allowed us to build this end to end view of how to deliver content in a way that we think delivers the best quality most efficiently and allows us to play with the puzzle pieces of that as we have new needs that come up. So like I mentioned, we had many of those things in place for video on demand, film and TV. When we went into live, engineers needed to rethink how are we going to have to change how we think about how we're delivering content given the requirements of live. They were able to start with what we've already built but also have a lot of their own decision-making on how to evolve all of our systems and products to be able to deliver new content types. So over time, the way Netflix has been built has been very driven by engineers within the teams rather than top down overarching, let me draw the architecture for you and now let's build it in that direction, which has both advantages but also things that we've had to evolve towards over time because as the company becomes much bigger, scale becomes more of a challenge.
We want to make sure that we're building things in a way that support that so it's not like it's static and that elasticity to use your word is something that has allowed us to actually engineer for what Netflix requires today versus what it requires 10 years ago. Can we talk about specifically live because live was a big launch last year, I remember one of the big boxing match between Jake Paul and Mike Tyson was a huge event. Can you give us a little insight of how that project started, how engineering teams got involved, like was it a small team, what was it, multiple teams, I'm assuming there must have been multiple teams working together and what was your process of getting this to launch again, was it overly planned out, was it just Yolo something in between?
Yeah, right, Yolo. Not quite Yolo. Live title was a Crick-Chris Rock special, I believe in March 2023. That was our first time bringing live to Netflix members and started what was a very intense period of if I take through to that Paul Tyson match, that was November 2024. So you think about that as basically 18 months from our first ever outing on live to the largest streamed event ever, which is what Paul Tyson ended up being.
The way that came to life was with urgency, a lot of scrappiness and like I mentioned, engineers making it happen. So typically we would set a goal saying, okay, so we've got Paul Tyson scheduled, originally it was scheduled for July 2024. It was rescheduled because of Tyson's health to November 2024, which gave us a few more months. But picture teams from Open Connect encoding, our content, production and promotion teams, our discovery teams, thinking who are the right people to lean in here and help bring this to life, but they self-organize, they develop their own roadmaps, they think about who needs to be on point for what things, what are some of the systems that we need to make sure are actually resilient enough for live.
It was an incredibly tight timeline and to end, not to mention that we had a lot of learnings from Paul Tyson because it was such a large event. Yeah. The world's largest, right? 65 million concurrent streams. Watching that tick up, I think one of our biggest ever days of signups, so we were watching the signups go through the roof that day, then watching, I think by the time we were in some of the first couple of undercard fights, we'd already exceeded our expectations for how big the fight would be.
The energy in the launch room here in Los Gatos was palpable, that you could feel excitement, nervousness, like very real time problem solving by engineers because we'd never seen no one had ever seen scale like that. So you get like your real time figuring out how to deliver something in real time. I've never been prouder of the team for figuring out what are the levers we need to pull to keep this as stable as possible. With those learnings in November 2024, we had about five weeks to be ready for two American NFL football games on Christmas day, where the bar is very high to deliver well for members and for fans. And so the team immediately took the learnings from Paul Tyson to say, how do we build greater resilience?
How do we think about how we're going to direct content if we end up bandwidth constrained in some markets? How can we really optimize by using some of our quality levers for what that experience is going to be? And those NFL games ended up being flawless. That from Chris Rock to a love is blind failure to Paul Tyson with lots of learnings at that scale to NFL and now weekly WWE, not to mention lots of other big events. That was all driven by teams on the ground being relentless and saying, how do we do this well?
What are the problems we need to solve? And the accountability for learn fast doesn't mean we're not going to have failures, but when we have failures, learn fast, iterate and deliver ever better. That's really where the beauty of the Netflix culture comes into play. And I've watched the same thing happen, standing up our own ads tech stack, being able to deliver games, watching our new TV UI. Each of those was a group of engineers coming up with what's the best way to bring this to life.
You mentioned the control room, but I think most people have not had this experience. It's a live event. Of course, you can tweak things. Can you explain to me what was the control room like? I assume it must have been a bunch of dashboards on all sorts of metrics.
Right? Was it a little bit like that? Yeah. So even our dashboards were brand new. So they were not built for that event, the engineering team put it together.
The data science and engineering team collectively put together a set of dashboards that would give us visibility into some core quality of experience metrics. So things like time to render, app start timing, rebuffer rates. It was the rebuffers that we started to see amp up during Paul Tyson. There were probably 100 people on site. I was sitting in a room with maybe 30 or 40, both engineers and data scientists.
We had our laptops and makeshift screens sitting there. Everyone was hard-lined into internet so that we weren't risking anything with the Wi-Fi. We had VPN backups. If anything was to go wrong, we had a launch commander with a headset dialed in talking to people in the production truck, but it was new. So it wasn't streamlined.
It wasn't perfect. It wasn't like the fancy launch rooms. I imagine a lot of live productions typically have. And so we're watching metrics. Something started flashing in red, causing you to draw attention to it.
We would create makeshift Google meet rooms. So small groups of people could triage. We had people who were on the hook for being informed captains or decision makers. So if we have an issue with Open Connect, if we have an issue with playback, if we have an issue with discovery, are people actually able to find a title? There were people that we had in basically the launch plan.
Imagine I think the document ended up being 40 or 50 pages of if-then statements. If this thing happens, then what? It was new for us. When I think about where we were for Paul Tyson, I joke with people like I lost 10 years of my life in that one night because it was so stressful and there's so little, like there's no hands-on keyboard thing I can do to help. I'm there to support the team, to trust the team in making decisions.
When I look now at how we're doing NFL games or the Canelo Crawford fight or WWE, it's much more sophisticated in the sense of the resiliency we've built, the metrics and dashboards, the visibility we have into what's happening. That was very human driven when we were first coming out of the gate, which was a lot of where the learnings came from. But I say to the team, how often do you get to work at a mature company, but build something truly from scratch like this and think about all the things that might go wrong and how to be prepared for them and then say very common cool and your pressure if something happens. So what strikes me isn't very interesting about the story. As you mentioned that the team had a bunch of preparation, you mentioned in four-year or fifty pages of the NELs, which sounds way more detailed than I've seen most launches but you usually have a launch plan but again, it was a conflict launch so you prepared.
Even so, they were hiccups both with Love is Blind, both with the Jake Paul Mat. Can you tell me how the team handled the aftermath? Again, it's pretty common to have a blameless post-mortems, but I'm more curious on how formal this process is, how less formal driven by people stepping up or, again, do you have more rigid process around this or it's just getting together and people do go on fix things. Like you said, it's not uncommon to have a blameless post-mortems retro, so we have that for sure. It's much more interesting to talk about the learnings from something than to overly focus on who did what thing wrong.
There's not much that we actually gained from that. It's not a rigid process. It happens very organically and it's led by the people who are close to the work itself and feel tons of accountability for doing reflections on both what went well and what can we do better. If I take Paul Tyson as a good example, he was a pretty complicated set of emotions and couple of days afterwards. We're celebrating the biggest ever live stream event.
Way bigger than we ever could have hoped for. We're celebrating that we work at a company that takes such a big swing. We're celebrating that we didn't collapse when we got to 20 million, 30 million, 40 million concurrent streams. If someone said to me, you're going to have 65 million concurrent streams, I would have said, this is not going to go well. And yet, yes, we have hiccups.
We always want to deliver a great member experience. I would like to say I woke up the next morning. I was awake the whole night thinking about what do we do next? We only have five weeks till NFL. I woke up through a set of memos where the team had already written down.
Here's what we observed. Here's what we think we can improve. Here's some of the things we should immediately prioritize. So some of that was how do we direct traffic when we get congested? What were our algorithms doing to think about directing traffic in that moment versus what do we want them to do if we end up congested?
And thinking about what are ways that we can gracefully fall back or degrade when we're under that type of dress? There was nothing in that event that we could have created before seeing what happens to the system's life. And so just knowing I woke up thinking all my like, we're going to have to do this. But it was already there in the document that the team feels such accountability to how do we do this better that we don't really require a process to say, now we should do a post-mortem. Now we should develop reflections on this.
The team owns that very directly. We have language in our culture memo about being unusually responsible. That's really the talent on the team. It comes with high-down talent density. It comes with treating people like adults where they get a lot of autonomy and making decisions and then they own the outcomes and they have a mindset, which is there was a lot that was exciting about that.
There's a lot we can do better. Let's go do better now. So I'm there to help provide input to ask some questions so that I understand and I can represent it accurately. But we'd almost never require a leader to say, now we must do the following thing because it's so driven by the teams. I have to ask this question, but I'm sensing the answer to it already.
When it comes to engineering culture, at a lot of companies, you go into a company as a new engineer and you ask around saying, hey, what are the processes I need to follow because a lot of companies, there's a mandatory code review. If you launch a feature, you might need to use a feature flag, let's say on the mobile app, you need to have certain sign-offs from people. So there's a bunch of CICD always needs to pass. You cannot override it. In the Netflix engineering team, how much of these things are put down at a global level?
Or at least to follow it, teams can decide and do the side versus just based on the judgment of the engineering team or the engineering themselves? Yeah. A lot is left to the engineering team and engineering themselves. So even for new or early career talent, one of the things we've evolved towards, so even if I think going back over many years when Netflix introduced chaos monkey as a concept, the idea of an individual engineer having responsibility to understand how and when their system will break and how they're going to be resilient, detect that and recover quickly was just a core part of the culture and something that we continue to maintain. A lot of the where we were a few years ago, let me talk about kind of pre and post-live is probably the useful threshold.
Pre-live, there were a lot of ways to take smart risks with video on demand because we had many years under our belt of understanding when something breaks, how are we going to fix that? And it was left to teams to think about the extent of testing and resilience that they needed to have and great on-call teams and support teams for when something does go wrong. When we introduced live, there's a different threshold because you have to watch it live. There's no such thing as Netflix is going to be down temporarily while we address something where we were taking a smart risk. That was scary at first, but as we started to introduce what are the guardrails we need to have to do this safely, it was things like introducing, especially for tier zero or tier one applications that were in the critical path for live, a higher threshold for what's the testing that you're doing to make sure that your system is ready for the duresset may be under a live event.
We will share those guidelines that come from our central engineering team and it gives people an opportunity to have less process because they're able to say, if I pass these guidelines if I've done this testing, I don't need to be in a quiet period, for example, during a live event or we've done end-to-end testing so we know those system dependencies very deeply and we're able to prepare for the what-ifs something goes wrong. But that's not a very structured process, like a code review or you must check these boxes or some type of gating function for code being actually deployed. We do have quiet periods during the end of your holidays. I think that's pretty common. We'll have some rules of the road around live events to make sure we don't take unnecessary risk, but we are constantly finding ways to like how do we reduce the quiet periods?
How do we leave a lot of judgment to the teams and then they're accountable for anything that goes wrong with their service? Do I sense it correctly that instead of the process, what you focus on is let's say the impact of the system. The tiering system tier zero, I guess the most important one is tier one and it goes down. The tools for example, quiet periods are other ways that the teams can then use to manage risk. That's right.
A lot of those were new introductions once we entered live. The teams were doing that very much on their own with their own judgment and accountability. Live, we ended up in a situation where we had to be more structured about it because it was new because it was higher risk because you must watch it live and because live is something that touches so many teams. Going back to our conversation around our content pipeline, if you think about live from the camera to the production truck to the origin or cloud to then be able to get to our content delivery network, there's a lot of systems that need to talk to one another in real time for a live event. There was a new set of things that could go wrong.
Until we were very competent that we knew those connection points, we wanted to introduce more guidelines for how to think about that. That's when we introduced the tiering system. It's when we introduced some things for which services or systems should think about being part of the quiet period or not. But we've already dialed a lot of those things down from when we first started with live because our preference is to not have to have so many constraints because it slows down too many teams to be able to make their own improvements and innovations that have nothing to do with live along the way. And we don't want to actually slow down other parts of the business in favor of just one priority area.
Elizabeth was just talking about how above a certain scale, Netflix had to put some basic guardrails in place for hybrid systems. But they kept being worried about introducing too many constraints because that was slow teams down. This problem of wanting both reliability and speed is not unique to Netflix. But the best companies figure out what the right tools are to enable the right culture, whether it's continuous development or experimentation for its approaches. These cultural values need the right tooling and infrastructure.
And this is where StackZig, our presenting sponsor, comes in. StackZig built a unified platform that enables both cultures, both continuous shipping and experimentation. Data flags, unless you ship continuously with confidence, roll out to 10% of users, catch issues early, roll back instantly if needed. Built in experimentation, it means every role that automatically becomes a learning opportunity with proper statistical analysis showing you exactly how features impact your metrics. And because it's all in one platform with the same product data, analytics, session replays everything, teams across your organization can collaborate and make data different decisions.
Companies like Notion went from single digit experiments per quarter to over 300 experiments with StackZig. They shipped over 600 features behind feature flags, moving fast while protecting its metric progression. Microsoft, Atlash and Mbrex use StackZig for the same reason is the infrastructure that enables both speed and reliability at scale. Speaking of scale, StackZig processes trailings of events per day. So whether you're a startup or building an open AI scale, the platform grows with you.
You can integrate it into your existing product data stack easily. If you're interested in building a culture of continuous development and experimentation, go to StackZig.com slash Pragmatic. They have a generous freeze here, a $50,000 starter program and affordable enterprise plans. Just tell them the Pragmatic engineers send you. With this, let's get back to the conversation about Netflix's engineering culture.
So when people and engineers will be listening or watching this recording, a lot of them will just be nodding like, yeah, wow, I'd love to work at a place where we can make these decisions and decide how much risk we take and the process we set. One reason that you can do this, I know for a fact because I know engineers at Netflix is you have a very high bar per talent. And you've always had that from the very beginning. In fact, for the first 25 years of Netflix, as I said, correct, the only software engineering level used to be a senior software engineer. Can you talk about how you go about hiring, what this bar is?
And in your experience, how did Netflix manage to? Is the only company that for again, such a decades only have this one level? There was no other things that, and this was just this high bar. How did it work? I continue to be amazed by the talent density of Netflix.
I almost didn't believe it before I joined a little over five years ago of, yeah, I'll believe it when I see it. I think the way to think about talent density at Netflix is a lot of the aspects of our culture, including talent density, are a means to excellence in our work. So none of them are the end games. They're saying like, no rules, no process or high talent density or context, no control. Any of the things that we're talking about are key elements of getting to a group of people who strive to do the best possible work they can.
Being able to get through so much of Netflix's history without the complexity of levels or rules or process helped to signify to people we're expecting a lot of you. I find it a very human thing when someone says, I'm expecting a lot of you, that people step up and do the best work they can. So in some ways, it builds upon itself to say, we have high talent density. We expect excellence. You have a lot of autonomy, but also a lot of accountability, that the best people will thrive in that situation.
They're not distracted by a lot of the things that you could surround that with. They know that the bar's high and they want to meet that bar. All the people I work with feel that way. You don't need to tell them what to do. They lean in to try to do the best possible work.
Being that over time, especially as we've scaled, as a company is a challenge. So anytime a team grows from 100 to 1,000 to 1,000, you have to think about like, what's the scaffolding you put around that to make sure we can maintain the spirit of what that culture was. So things that have changed over time, we don't still just have a single level. So as we think about growing as an engineering or as a tech organization more broadly, not every role requires somebody who has 10, 15, 20 years of experience. Some roles are a great match for someone who's newly out of college or has a couple years of work experience, but you would want to think about the expectations for that person, the compensation for that person being different.
So we did start to build some scaffolding around levels, not to say now we want to have so much structure that that's sort of suffocating. We want to maintain all the great things about a lot of independence and accountability, but we didn't even have vocabulary to talk about how might I construct a team to have a broader array of talents. So that's one of the things we've changed over the last couple of years. We've also had to think about one of the things you get as you introduce levels are things like IC or people management pathways. What are the expectations that you have at each level?
Some of that is about skills. Every company is going to have that reflected. But a big part of what we have in our pathways and our ways of talking are the cultural things. So do you uplift other people around you? Do you deliver a lot of excellence and accountability in your work?
And we hold a high bar for people meeting those things. That's selflessness. That's good judgment. That's thinking about what's best for Netflix. Some of our engineering principles are things like building for the future teams who are going to thank you for the work that you did today, which means don't take shortcuts.
Build high quality durable products. Things like think globally, act locally, meaning think about the broader ramifications across the tech organization or Netflix, even as you make your local decision. And maybe my personal favorite, which is year to learn. It's a nice, mimefied raise of like be curious. You know, think about like, I might think about the right problem in the right way.
That's how we mean high talent density to say like, those are our ways of working and we expect that from everyone, no matter what your level is. And you have to watch out for the incentives that things create. You know, it's very easy to say, I'm going to do the thing that I think puts me in a better position rather than my team or the company. And we really try to discourage that and really celebrate when people do the thing that's selfless or does the less glamorous or less visible work as long as it's better for everyone else. I find that when people behave that way, it continues to attract and retain the best talent.
You mentioned that you're trying to not have things that distract people. Now when I was a manager, one thing that did distract me every six months on the dog. Yes, performance to you season. And as we just called the perf, it was one month of my life, or especially at the end of the year, even longer. Yeah.
And whenever I talk with fellow injury managers, I just got up with a friend and he was saying, like, oh, yeah, perf is coming up. So I won't be able to help you in this period. How do you go about this necessary evil or just necessary process of performance management? Because I understand it's very different. And really, it's this is very publicly known, the keeper test, which which Netflix shares on the website as well.
How does this play into it if it does it all? Yeah. We don't have formal performance reviews, which is probably the first unusual thing. So when you think about other companies spending that time to talk through each person or assign a rating for whether they meet, exceed, I've seen that at other companies do, we don't do it that way. But we do carefully think about feedback, performance expectations, all the things that would be done to keep our test, which I'm happy to talk through.
So the way we approach it at Netflix is first trying to get to something that looks like continuous, timely, candid feedback, easier said than done. It requires trust. It requires deep relationships to be able to give someone in the moment, very candid feedback. It could be here's the thing you did great. It's not always a negative or constructive thing and to be able to receive that type of feedback.
If we're living in Netflix culture well, that's something that would be familiar and comfortable every day of the year. So you're not having to wait for a certain performance review or feedback cycle in order to hear how you're doing or be able to provide that type of input to others. We do as kind of a safety net have an annual 360 process where I would request feedback from a bunch of people I work with. I get requests from a bunch of people. But that's something that you're having a direct conversation with the individual about feedback.
It's something I would review with my manager to say, here's some of the themes that I heard, some of the things I'm going to work on. So there's an opportunity to think about what is my performance, how are people perceiving our working relationship and my contributions, but it's not structured as an evaluation. It's structured in the context of feedback that helps people improve. And then separately, once a year, we go through both compensation review, which is a reflection in ways of what's my level of impact, what skills have I gained, what are my contributions to the company. So you naturally talk about performance as part of thinking about someone's personal top of market, which is our compensation philosophy.
So it comes up as a conversation there where managers really think about for each person on their team, how do I think about the compensation that reflects this person's value to Netflix and value in the market. So that has a performance flavor to it, but is not a performance review. And then a couple of years, a time's a year, we evaluate promotions. So in that case, we're a group of people who might be up for a promotion from level five to a level six, we would collect feedback that helps us make that decision. So if I look across the feedback, continuous in the 360 cycle, compensation review and promotion evaluations, we get quite a few touch points where people are hearing how they're doing, but it feels more constructive and actionable than the performance review structure that other companies have.
What this requires us to do well is still a lot of manager attention and judgment. And it's not a manager in isolation being able to say, I think you're meeting the keeper test, or I think this should be your compensation. We do have structure around that so that managers are accountable for the decisions that they're making on their team. So if I think about my role in meeting the tech organization, I review collectively, who's getting promoted? How many people are getting promoted?
What are the themes coming up in 360 feedback? Where's compensation landing across the teams? So it's a little bit of checks and balances because we do leave so much to a relatively unstructured process. And we try to provide a lot of support to managers when they are making keeper test decisions. So for context, that's asking the question of, you know, is this person really meeting expectations for the role and what the business requires?
There's a keeper test that a manager might ask themselves and have that conversation with members of their team. But honestly, there's a keeper test that goes from members of their team to their managers. Do I want to stay? Am I excited about the work that I'm doing? Is my manager giving me growth and development, helping to guide my work in good ways?
You know, I want to make sure it's clear that that's then a we're all kind of accountable for that instead of it just being like a manager makes decisions for their teams. But the keeper test and asking yourself that question is a good way to make sure that we're all accountable for holding high talent density bar in our team. It's also a good way for someone on my team to ask me, Hey, how am I doing? You know, is there any feedback that maybe I haven't heard that I should know about to make sure I'm meeting expectations? And ideally we do that in a way that feels like normal course of business.
So the keeper test is unusual. The fact that you don't do performance reviews is unusual or at least not a structured cadence for someone listening. They might think, well, that sounds real stressful. However, I looked at data from Signal Fire. They had this chart with the retention of tech companies in the talent bar.
So like kind of higher talent bar gear and retention and Netflix on the top, top corner of all companies, which means that they're based on the data they have high engineering talent is the least likely to leave ad Netflix across companies are comparable. My question to you, why do you think people leave companies and why are they saying? Yeah, I'm glad that here we were like on the upper right, but we have to earn we have to earn that over time. I personally find that people leave when they're not getting the challenges and the fulfillment that they would like to get from their work, or they don't feel like they're adequately recognized for the contributions that they're making.
I don't know that any company can guarantee that everyone loves their job and feels perfectly recognized every day. But we get a lot of kind of at vats on that for people to feel like I'm solving really hard, interesting problems. I have a lot of agency on autonomy on how I solve those problems. I don't feel constrained by a lot of rules or process. We don't have a top down command and control culture that really narrows people's contributions.
And we expect a lot of that responsibility for both the successes and the failures. A lot of people love that environment. We fight really hard to maintain that type of environment so that people are excited to stay. It doesn't mean that our retention is 100%. People get great opportunities other places.
And I actually think it's good for them to take it because we're not saying like we expect you to be at Netflix forever. We do want people to be excited here. Think they're doing the best work of their life. Sometimes they get opportunities elsewhere, but hopefully that's a positive experience they've had in terms of the work and the culture that they experienced at Netflix. I also think managers and leaders have a big role to play in why people say in setting a vision, in setting a clear strategy, in making tough, timely decisions so that people can do their best work.
And in my experience, sometimes people will decide to stay or leave based on that overall sense of like, I feel really inspired by the direction we're taking. That I think Netflix has had a lot to offer, especially with some of our newer vets and things that we're building from scratch and new experiences that we're bringing to studios or advertisers or members. So hopefully that also builds some of the enthusiasm. And to finish my thought, I think people stay when they're impressed by the talent around them. This is where talent density builds on itself.
You know, if you really hold people to high bar, great talent is much more likely to want to stay. So this is another reason to make sure we do a really good job with that. So one of the very exciting things these days, of course, is AI and AI tools both building with them, but also using as engineers. In your experience inside Netflix, how are the engineering teams using these AI tools for their own work? How are their experimenting with them?
What is working? What is maybe not a great fit? Yeah, it's a huge area focus for us right now, but with a lot of intention and pragmatism of where these tools are actually helpful versus I love that. They're not. And again, the hope is that we identify those places where we actually get higher quality, more impact for the business versus things that are lower quality or just about cost reduction.
That's really not interesting to us. So across any technical application, but including JNI, we're looking for the thing that is meaningfully advancing our impact for the company. So for engineers, we are experimenting with a set of coding assistance. The way we approach it is to provide a lot of different tools to the teams. So they are able to explore, experiment, decide which tools meet their needs, start to learn what works better for some use cases or some applications than others.
We're trying to create space so that people actually have the time to do that. As we all know, there's quite a learning curve when you're thinking about like, I'm going to change how I write code, how I document, how I think about making decisions, especially for the people who are very accomplished in their roles, changing your way of working can be kind of jarring. And we have tight timelines and big ambitions here. So there's not a lot of free time running around. Let me figure out how to use this new technology.
So we're doing things like both enabling the tools. We have some weeks where we let people just be focused on, let me try a new project. Let me experiment with something new that gives people a little bit of space. And then we're collecting tons of feedback from across the team around which tools are actually useful, which do we want to graduate to pave paths, where are the areas where we actually see the most impact. A lot of that is self-reported.
You know, it's teams that are experimenting. We have what we call JNI champions throughout the business. And so they're able to kind of help teams troubleshoot, understand what's available, but also feedback to central teams, what's working and what's not. So we can continue to advance how we're approaching this. And I do think we're doing a good job being pragmatic and thinking about not feeling like it needs to be like, JNI is a silver bullet for everything that engineering or technical teams are doing.
I think we want to be more surgical about where the impact comes from. And in some ways that reflects the same strategy we're taking for member facing use cases or creator facing use cases where we're trying to figure out, like, where do we actually get a better experience and higher quality and experiment with a ton of different capabilities, which also has implications for what's our infrastructure strategy, what's our overall strategy of like, how do we give access to a lot of different options so we can experiment and then think about where's the market solving this well, where should we build something in house? It's very likely that for a lot of our tech productivity, the market is solving those problems very well. There's not really a big advantage to us building tools in-house, but we still want to be kind of choosy in which market tools we actually leverage.
You said you're getting a lot of feedback already. You know, people, teams, organizations are sharing what's working, what's not. Are you seeing some areas where these tools, specifically, coding assistants, agenting tools are maybe a little bit more helpful. May that be greenfield things, migrations, prototyping or some other areas? You named a couple of them very well.
So maybe starting at the end there, prototyping is a lot faster. And actually, that's a place where when you think about the cross-functional teams across engineers, data scientists, product managers, designers, we're hoping that we can actually bootstrap things very quickly. I have an idea. Let's visualize that idea. Let's quickly throw together a set of code that would help to bring this to life.
That's not necessarily something we would productize or consider production ready code, but that's okay because it helps teams advance and innovate and kind of workshop ideas very quickly. And then, you know, as you probably know, in your listeners know, there's a lot of what can be tedious work. And it's not always the actual coding work, where it feels like it's the biggest time commitment. It can be accessing knowledge about how systems work. It can be documenting code.
It can be thinking about big migrations that we've had on our plate that we can actually automate much of that work. And there's also things around detecting issues. So anomaly detection, response, being able to do deep dives of issues. We're finding that there's a lot of promise for gen AI tools in that space, which helps us with some of our resilience and just general best practices and health as an engineering organization. If we're able to use gen AI tools in those spaces, prototyping, documentation, migrations, detection and response, it leaves a lot of time for the more innovative work.
So how do we think about architectures and systems and products that we're building to deliver business impact? So then hopefully we can actually get more impact for engineers because they're able to actually leverage some of the tools or to agent experiences to minimize the time spent on some of the less impactful activities. But it's really a portfolio of work. And I would say again, it's not a silver bullet in any of those spaces. I would say it's come a long way since some of the tools we first started experimenting with a couple of years ago, which, um, let's just say they didn't meet the quality bar that we really Yeah, I know it's, it's been a big change on this.
One thing that's you need to know, it's an affix is you mentioned how for a long time Netflix only hire senior software engineer, senior above software engineers about two years ago or so a few years ago, you've now started to hire earlier career software engineers. Can you tell me on how that has changed the culture at Netflix, what you've learned by hiring and what is your strategy and are you planning to keep bringing in new grads or interns or early career folks? Or are you planning to do because a lot of companies these days are single? Let's just go with seniors for now, at least, especially until we figure out this whole AI thing. Yeah, we've had a great experience with new grads and early career talent and also our internship program, which you mentioned, we were starting from a very different place than a lot of other tech companies.
So when you look at the distribution of levels or talent at some of the other larger tech companies, they had in some cases 30, 40, 50 percent, what I'll call level three, level four engineers. So when you think about a new technology shift or the work that those companies need to do now, I understand why they might need a different distribution of talent. We were starting at 0% in most cases. Yeah, which is why, yeah, so we had mostly a level five and above population. So we had a huge opportunity to complement the team we had with earlier career talent, who brings new skills, new perspectives, great energy to the teams.
And with a technology shift right now with JNI, a lot of native AI familiarity. So when you think about somebody who's graduated from school in the last few years, they're very accustomed to using AI in whether it's developing products, writing code, thinking about solving data problems. So it's actually a useful way to bring new skills and perspectives to the team. I think we will absolutely maintain that investment in earlier career talent because it's been so additive in different parts of the business. I also think it everything in its right proportion.
Like we also, there's plenty of problems where we need extremely senior talent. So at the same time, I would say I'm pushing for us to also think about how do we add more staff, principal, distinguished engineers and scientists to the team, because that's also a place. You know, you think about that tale of the distribution being a really important place to maintain strong talent. So we're investing in both of those tales of the distribution. I love it because I usually hear companies talk about one or the other, but not both.
And I guess that's something that the people here one day will hopefully be there. Another way to think about it is building talent from within. So we hope that a lot of the early career talent joining has a great experience, develop skills and impact here and becomes those more senior technical talent over time. And our most senior technical talent have to be great role models there, too. So we're doing more of that internally in that talent development than we would have done five or 10 years ago.
And I would say it's been a huge boost to the team. One of the most surprising things I've learned about Netflix just very recently is how much you invest in open source. This might sound a bit silly because we know Chaos Monkey is very famous. In fact, everyone that Netflix is known for that one. But again, in a recent report, I looked at all the companies and what percentage of engineers and the working on open source and Netflix again was at the very highest bar.
This publication estimated that about one in five engineers work on open source projects. Sure enough, I go through open source page. It's just so, so much open source. Can you tell me why and how and since when is Netflix doing so much open source? And why do we not know about this?
This was new to me. Perhaps we should be talking about it more. So this is a good opportunity to start. You know, we were talking earlier about the engineering culture and the sense of a talent density, which often comes for a passion to contribute to the broader technical community. So the people at Netflix care deeply about the quality of their work and advancing innovation more generally.
For some things, it's Netflix specific innovation and it's important we keep that IP as a competitive advantage. But for many, it's something that helps to actually drive broader industry innovation, which also benefits Netflix over time. So if I can give you one example among the list of places where we've been very involved with internally and externally, it's in the encoding space and driving a ton of innovation. Video encoding, we've now won, I believe, nine Emmys for these contributions. I always used to associate Emmys, you know, just with TV and the red carpet, but we've won a lot of technical and engineering Emmys at this point, specifically on video encoding work.
So as one example, that helps to contribute incredibly to quality and efficiency of our ability to encode our titles and deliver them. So Netflix gets an immediate benefit by improving the technology in that space. But we are also a founding member of the Open Media Alliance, which is an industry community that pushes for open advancement of encoding technology. If we're able to inspire that work, Netflix actually also benefits because the whole industry up levels and we think about integrations with different technologies that we might do over time with everyone helping to push the bar. A statistic I like to cite is when you look at the catalog now of Netflix content, think about how much bigger the catalog is than when we were first starting with originals.
I believe we now require 60% less bandwidth, 60% fewer bit rates for same or better quality with a much bigger catalog. That comes from our media encoding innovation and having a whole industry that's pushing that benefits anyone who's in the entertainment space and definitely benefits consumers and our members. So that's a good example where it starts from an open source contribution. Netflix doesn't lose anything, only gain something by contributing to the broader innovation landscape. And I am a strong proponent of talking more about the innovation we're driving.
So things like different blog posts that show up in our tech blog, for example, we're talking more about what it took to bring live to life as one example. I just think is a great contribution to the broader community. This is one of the reasons I love software engineering because I feel contributing to the open and sharing things. It lifts the tide for everyone. Yeah, I believe so.
Yeah, we definitely benefit. And we are trying to drive better outcomes overall, especially with the real member focus. And so a lot of the technology we're building is able to do that. So it's closing. Netflix sounds like a very different and special place compared even across all of the larger tech companies or even the innovative companies.
What would your advice be for a new start software engineer starting Netflix? How can they succeed in this environment and how can they grow up to the expectations of a place like this? Curiosity, curiosity, curiosity. When people ask me, like, what's the Netflix value that most resonates with me? And I most love to see across the team its curiosity asking questions, questioning whether we're solving the right problems in the right way, just because you're new to Netflix or earlier in the career doesn't mean you're not going to be the source of innovation.
If anything, it's great ideas come from everywhere. And that starts with just being curious, open minded, experiment, explore, take smart risks, try to reduce that, that kind of voice in your head that is fearful of exploring something new or taking that risk. And I think when people join Netflix and they approach it with that type of curious mindset, they're already set up for success. I would also say mean on other people. So we have great, talented Netflix and they are all more than happy to help other people be successful.
So don't shy away from finding a mentor asking somebody, why does this work this way? Can you give me more of the history of this? Can you help me understand which business problem we're solving and why? It's another flavor of curiosity, but it's also about the broader community and really leveraging that at Netflix. Oh, also, but thank you.
This was very, very interesting. And I've learned a lot. Great. Really happy to be here. And I was happy.
It worked out to have this conversation. Thank you. Thanks. One of the most interesting learnings for me about Netflix was just how most open source they contribute to and how about one of five engineers involved in open source work. The other one was how performance management is really lightweight and tries to be truly continuous.
Both of these things feel like they're quite different from how most other big tech companies operate. I previously did a deep dive on how Netflix's engineering levels changed from the single senior level to the new five levels. Check out this the pragmatic engineer deep dive in the show notes link below, as well as deep dive on the engineering culture of other big tech companies like meta Amazon and Google. If you enjoyed this podcast, please subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating for the show.
Thanks and see you in the next one.